IMPLEMENTING OFFLINE SYSTEM UPDATES
This man page describes how to implement "offline" system updates with systemd. By "offline" OS updates we mean package installations and updates that are run with the system booted into a special system update mode, in order to avoid problems related to conflicts of libraries and services that are currently running with those on disk. This document is inspired by this m[blue]GNOME design whiteboardm[][1].
The logic:
- 1. The package manager prepares system updates by downloading all (RPM or DEB or whatever) packages to update off-line in a special directory /var/lib/system-update (or another directory of the package/upgrade manager's choice).
- 2. When the user OK'ed the update, the symlink /system-update is created that points to /var/lib/system-update (or wherever the directory with the upgrade files is located) and the system is rebooted. This symlink is in the root directory, since we need to check for it very early at boot, at a time where /var is not available yet.
- 3. Very early in the new boot systemd-update-generator(8) checks whether /system-update exists. If so, it (temporarily and for this boot only) redirects (i.e. symlinks) default.target to system-update.target, a special target that is pulls in the base system (i.e. sysinit.target, so that all file systems are mounted but little else) and the system update units.
- 4. The system now continues to boot into default.target, and thus into system-update.target. This target pulls in the system update unit, which starts the system update script after all file systems have been mounted.
- 5. As the first step, the update script should check if the /system-update symlink points to the the location used by that update script. In case it does not exists or points to a different location, the script must exit without error. It is possible for multiple update services to be installed, and for multiple update scripts to be launched in parallel, and only the one that corresponds to the tool that created the symlink before reboot should perform any actions. It is unsafe to run multiple updates in parallel.
- 6. The update script should now do its job. If applicable and possible, it should create a file system snapshot, then install all packages. After completion (regardless whether the update succeeded or failed) the machine must be rebooted, for example by calling systemctl reboot. In addition, on failure the script should revert to the old file system snapshot (without the symlink).
- 7. The system is rebooted. Since the /system-update symlink is gone, the generator won't redirect default.target after reboot and the system now boots into the default target again.
RECOMMENDATIONS
- 1. To make things a bit more robust we recommend hooking the update script into system-update.target via a .wants/ symlink in the distribution package, rather than depending on systemctl enable in the postinst scriptlets of your package. More specifically, for your update script create a .service file, without [Install] section, and then add a symlink like /lib/systemd/system-update.target.wants/foobar.service → ../foobar.service to your package.
- 2. Make sure to remove the /system-update symlink as early as possible in the update script to avoid reboot loops in case the update fails.
- 3. Use FailureAction=reboot in the service file for your update script to ensure that a reboot is automatically triggered if the update fails. FailureAction= makes sure that the specified unit is activated if your script exits uncleanly (by non-zero error code, or signal/coredump). If your script succeeds you should trigger the reboot in your own code, for example by invoking logind's Reboot() call or calling systemct reboot. See m[blue]logind dbus APIm[][2] for details.
- 4. The update service should declare DefaultDependencies=false, and pull in any services it requires explicitly.
NOTES
- 1.
- GNOME design whiteboard
- 2.
- logind dbus API
- 3.
-
Implementing Offline System Updates