Process Management and Monitoring Notebook - page 27 of 74

First PagePrevious PageNext PageLast PageTable of ContentsSearch

Date and Author(s)

Interfaces for Checkpoint/Restart

Interfaces for Checkpoint/Restart and related mechanisms

Interfaces for Checkpoint/Restart and related mechanisms.

I would like to see calls which specify the INTENT in a way that "send SIGSTOP" or "send SIGCHECKPOINT" cannot. Specifically I advocate the following calls.

There are some things not explicit in the above descriptions which we need to consider. Please add anymore you think of, or comment on the existing list.
  1. Validating a RESTART or MIGRATE call
    PROBLEM: We desire a mechanism by which the Scheduler can determine if a RESTART or MIGRATE will succeed with a given nodeList - a mechanism which is lighter weight than trying and failing. In terms of implementation, this could be done primarily by matching information pulled from the checkpoint file(s) against the node configuration database.
    SOLUTION 1 (depricated): I suggest "dryRun" option to both the RESTART and MIGRATE calls (like -n in make) which would return status information telling if the operation is certain to fail. This result is not intended to be a guarantee that the call would succeed, but cases which are certain to fail could be eliminated inexpensively.
    SOLUTION 2: An solution which was more popular on the 2001.12.12 conf was to have a mechanism for determining the dependencies/requirements of a checkpoint, so the scheduler could select nodes what met these requirements. Here is a suggestion:
  2. Utilizing application-level support
    PROBLEM: When the application implements its own checkpoint files, these are typically much more efficient than any mechanism supplied by the runtime environment. Therefore we wish to use the application's own mechanisms when available.
    SOLUTION: We need a way to communicate the application's capabilities to the Process Manager (PM). I suggest that these capabilities be an optional argument to the CreateProcess call. This information would originate from the user's batch request. The argument(s) would also need to express the mechanism, such as "send SIGUSR1".
  3. Variable level of support present in runtime environments
    PROBLEM: The Scheduler needs to know which of these calls a given PM supports, on a per-node basis. It seems necessary that CHECKPOINT, RESTART and MIGRATE should be made optional features to cover many existing systems. We need a more polite way to let the Scheduler know this than simply by failing the first time it requests one of these actions.
    SOLUTION: We will need a call that lets the Scheduler query the features supported by the PM, on a per-node basis.
  4. Variable level of resource reclaimation in SUSPEND implementations
    PROBLEM: Almost any system can support the most basic version of SUSPEND/RESUME using SIGSTOP and SIGCONT. However, these will not free the resources consumed by the job, such as virtual memory space and file descriptors. Because the job which is preempting the present job may need these resources (especially the memory), it seems that the Scheduler would need to know what resources will and will not be released when a job is suspended. This could vary between nodes.
    SOLUTION: Resource reclaimation support could be advertised along with the list of supported features, described above.