man cr_restart (1): restarts a process, process group, or session from a checkpoint file.

SYNOPSIS

cr_restart [options] [checkpoint_file]

DESCRIPTION

cr_restart restarts a process (or set of processes) from a checkpoint file created with cr_checkpoint(1).

A restarted process has all of the attributes they had at checkpoint time, including its process id. If any needed resources cannot be attained for the processes in a checkpoint file (ex: a pid is in use), cr_restart will fail. If a process group or session is restarted, all parent/child relations and pipes, etc., between the processes in the checkpoint will be correctly restored.

If the stdin/stdout/stderr of any restarted process was directed to a terminal at checkpoint time, it is redirected to the controlling terminal of the cr_restart program.

The current working directory of a restarted process is the same as when it was checkpointed, regardless of where the context file is located, or where cr_restart is invoked.

The cr_restart process becomes the parent of the 'eldest' process in any restarted job. This means that getppid(2) may return a different value to the eldest process after restart. When the eldest restarted process exits (or dies from a signal), cr_restart will exit with the same error code (or kill itself with the same signal), so it is largely invisible (it is necessary to keep cr_restart `in-between' your shell and restarted processes, however, as most Unix shells get quite confused if they observe their children changing process ids).

Signals

By default restarted processes begin to run after the restart is complete. Alternatively, you may specify that they be stopped (via --stop), or terminated/aborted/killed (via --term, --abort, or --kill). This is done by sending the appropriate signal to every process that is part of the restart. If the processes were stopped at the time the checkpoint was requested, then --cont may be used to send SIGCONT to all processes after the restart is completed.

Error handling

By default cr_restart will block until the restarted process has completed, and will exit with the same exit value as the restarted process (even if the restarted process died with a fatal signal). This can make it nearly impossible to determine if a non-zero exit from cr_restart is due to a failure to restart, or is the exit code of a correctly restarted process. The simple approach of looking for 'Restart failed:' is not reliable. Therefore, the --run-on-* family of flags are available to supply alternative (or supplementary) error handling. When any of the --run-on-* flags is passed, a hook is installed for the given category of failure (or success), as defined below. When an error (or success) is detected and a corresponding hook is installed, the hook is run via the system(3) function. If the exit code of the hook is non-zero, then cr_restart returns this value, suppressing any error message that would otherwise be generated. If no hook is installed, the hook is an empty string, or if the hook returns an exit code of zero, then an explanatory error message is printed and an exit code related to the errno value at the time of failure is returned.

--run-on-success='cmd': Runs the given command as soon as the restarted process(es) are known to be running. If the return value of 'cmd' is non-zero, this also results in cr_restart terminating without waiting on termination of the restarted process(es).
--run-on-fail-args='cmd': Runs the given command if the arguments are invalid. This includes the case in which the given context file is missing or unreadable.
--run-on-fail-temp='cmd': Runs the given command if a "temporary" failure is detected. This includes the case of a required pid being in use.
--run-on-fail-perm='cmd': Runs the given command if a "permanent" failure is detected. This is most commonly due to a corrupted context file.
--run-on-fail-env='cmd': Runs the given command if an "environmental" failure is detected. This includes when files required for restarting are missing or inaccessible.
--run-on-failure='cmd': This installs the given command for all of the --run-on-fail-* hooks.

File relocation

By default, files and directories are saved `by reference', storing their full pathname in the context file. This includes files associated with a process via open(2) and/or mmap(2) and directories associated via opendir(3) or as the current working directory. Use of --relocate oldpath=newpath allows remapping of such paths to new locations at restart-time.

When parsing the --relocate argument the sequences `\=' and `\\' are interpreted as `=' and `\', respectively, to allow for paths that contain the `=' character. The `\' character is not special in any other context. (Note that command shells also have special treatment of `\' and you may therefore need quotes or additional `\' characters to pass the argument you intend.)

When file or directory associations are restored, the oldpath is compared to the saved fullpath of each file or directory. If it matches the leading components of the path, the matching portion is replaced by the value of newpath. Note that oldpath must match entire path components, and only leading components. Therefore an oldpath of /tmp/foo will match /tmp/foo or /tmp/foo/1, but will not match to /tmp/fooz (not matching the full component fooz) or to /var/tmp/foo (not matching the leading component /var.)

It is important to be aware the the saved fullpaths in a context file are the canonical paths. Therefore the oldpath you provide must also be a canonical path, though the newpath doesn't need to be. For instance, if /tmp is a symbolic link to /var/tmp, then if your application opens the file /tmp/work/1234 the path stored in the context file will be /var/tmp/work/1234. Therefore,

: --relocate /tmp/work=/tmp/play

would not work as desired, but either of the following would:

: --relocate /var/tmp/work=/tmp/play
--relocate /var/tmp/work=/var/tmp/play

If the --relocate option is passed multiple times, all are applied to restored file or directory associations, but only the first match is applied to any given path. Currently a maximum of 16 relocations is supported.

PID and related identifiers

By default, processes are restarted with the same pid and thread id (as returned by getpid(2), and gettid(2) respectively). This default ensures that processes and threads that signal each other and processes that wait on children will continue to function correctly. However, this prevents restarting concurrent instances of the same context file.

By default, the process group and session (as returned by getpgrp(2), and getsid(2)) are set to those of the cr_restart program. This ensures that job control via the requester's session leader (typically a login shell) will continue to function correctly. However, this interferes with any job control or process group signaling that may be take place among the restarted processes.

There are options to individually control whether the pid, process group and session are restored to their saved values or assume new values (the process group and session inherited from cr_restart and a fresh pid obtained from fork(2)). There is no separate control for the thread ids, as they must always follow the same policy as the pid. The following describes each option, along with outlining some of the risks associated with the non-default ones:

--restore-pid: (default) This causes pid and thread ids to be restored to their saved values.
--no-restore-pid: This causes pid and thread ids to assume new values. Any multi-threaded process has the possibility of using functions like tkill(2) which will not behave as desired if the thread ids are not restored. Similarly, any multi-process application may make use of kill(2) or waitpid(2), among others, that require restored pids for correct operation. It is also worth noting that many versions of glibc will cache the result of getpid(), which may result in calls after restore returning the original value, even though the pid was changed by the restart.
--restore-pgid: This causes the process group ids to be restored to their saved values. This is required for correct operation of any multi-process application that may perform signal or wait operations on process groups (as by passing a negative pid value to kill(2) or waitpid(2), among others), or which uses process groups for POSIX job control operations. This is NOT the default behavior because restoring the process group ids will prevent job control by the requester's shell (or other controlling process).
--no-restore-pgid: (default) This causes the restarted processes to join the process group of the cr_restart process.
--restore-sid: This causes the session ids to be restored to their saved values. This is required, for instance, for systems that are performing batch accounting based on the session id.
--no-restore-sid: (default) This causes the restarted processes to join the session of the cr_restart process.

Note that use of --restore-pgid or --restore-sid will produce an error in the case that the required identifiers are in use in the system. This includes the possibility that they conflict the the process group or session of cr_restart.

OPTIONS

General options:

-?, --help: print this help message.
-v, --version: print version information.
-q, --quiet: suppress error/warning messages to stderr.

Options for source location of the checkpoint:

-d, --dir DIR: checkpoint read from directory DIR, with one 'context.ID' file per process (unimplemented).
-f, --file FILE: checkpoint read from FILE.
-F, --fd FD: checkpoint read from an open file descriptor.
: Options in this group are mutually exclusive. If no option is given from this group, the default is to take the final argument as FILE.

Options for signal sent to process(es) after restart:

--run: no signal sent: continue execution (default).
-S, --signal NUM: signal NUM sent to all processes/threads.
--stop: SIGSTOP sent to all processes.
--term: SIGTERM sent to all processes.
--abort: SIGABRT sent to all processes.
--kill: SIGKILL sent to all processes.
--cont: SIGCONT sent to all processes.
: Options in this group are mutually exclusive. If more than one is given then only the last will be honored.

Options for checkpoints of restarted process(es):

--omit-maybe: use a heuristic to omit cr_restart from checkpoints (default)
--omit-always: always omit cr_restart from checkpoints
--omit-never: never omit cr_restart from checkpoints

Options for alternate error handling:

--run-on-success='cmd': run the given command on success
--run-on-fail-args='cmd': run the given command invalid arguments
--run-on-fail-temp='cmd': run the given command on 'temporary' failure
--run-on-fail-env='cmd': run the given command on 'environmental' failure
--run-on-fail-perm='cmd': run the given command on 'permanent' failure
--run-on-failure='cmd': run the given command on any failure

Options for relocation:

--relocate OLDPATH=NEWPATH: map paths of files and directories to new locations by prefix replacement.

Options for restoring pid, process group and session ids

--restore-pid: restore pids to saved values (default).
--no-restore-pid: restart with new pids.
--restore-pgid: restore pgid to saved values.
--no-restore-pgid: restart with new pgids (default).
--restore-sid: restore sid to saved values.
--no-restore-sid: restart with new sids (default).
: Options in each restore/no-restore pair are mutually exclusive. If both are given then only the last will be honored.

Options for kernel log messages (default is --kmsg-error):

--kmsg-none: don't report any kernel messages.
--kmsg-error: on restart failure, report on stderr any kernel messages associated with the restart request.
--kmsg-warning: report on stderr any kernel messages associated with the restart request, regardless of success or failure. Messages generated in the absence of failure are considered to be warnings.
: Options in this group are mutually exclusive. If more than one is given then only the last will be honored. Note that --quiet suppresses all stderr output, including these messages.

AUTHORS

Jason Duell, Paul Hargrove, and Eric Roman, Lawrence Berkeley National Laboratory.

REPORTING BUGS

Bug reports may be filed on the web at http://mantis.lbl.gov/bugzilla.