Description

Ladder creates a SecureApt repository to migrate production devices from one release milestone to the next. The repository contains all binary packages which would be installed to upgrade the target package of the specified release, including base packages. Source packages are not included as this would make the final tarball much larger than necessary. Sources should remain available via the main repositories.

For the purposes of "ladder", the bare installation / rootfs should be considered to always precede the first software release. Subsequent steps can then be based on the tarball of the previous milestone.

Note that if using "multistrap" or a foreign architecture "debootstrap", ensure that the rootfs inside the tarball is configured and repacked before being used with "ladder". i.e. use the production tarball rather than the build system tarball.

Ladder checks the installed package list from the production tarball for that release, calculates the packages needed to migrate to the specified milestone and prepares a repository containing those packages, including all dependencies.

If the specified package list and the specified milestone are NOT contiguous, errors can result if some of the contained packages need to migrate between data formats. For most cases, create a ladder step for each software release and upgrade devices in the same sequence. "ladder" steps can be chained by modifying the update scripts.

Config files

Ladder configuration files live in /etc/ladder.d/ and need to be named after the release described. e.g. /etc/ladder.d/internal.conf.

A minimal file to upgrade to Debian sid could look like:

 [sid]
 suite=unstable
 location=http://ftp.uk.debian.org/debian
 targetpackage=apt

A more comprehensive config file could look like:

 [internal]
 suite=interim
 codename=milestone
 branch=software_release_4
 key=0xDEADBEEF
 keyringdir=keys
 location=copy:///srv/repo
 rootpackage=libfoo3
 targetpackage=metapackage
 extrapackages=bar baz other
 updatescript=true
 mountpoint=/media/ladder

(It is possible to list more than one package, as a space separated list. Commas or other markers will not be parsed by apt.)

The section name (e.g. internal in the example above) is used as the milestone name, which can differ from the suite name and the branch name.

For more information on the key and keyringdir options, see the section on SecureApt below.

Requirements

The rootfs is expected to carry some existing apt sources, the location specified in the config file should be the one additional source which provides the updated packages and the expectation is that this will have a different suite name to the suite configured in the rootfs. If the location and suite are the same, "apt" will print messages about duplicate source lists but these messages can be ignored.

In order for apt to calculate the packages needed for the update, all repositories which are enabled in the rootfs tarball including the location specified in the config file must be accessible on the machine running "ladder".

Deployment of ladder tarballs

The final tarball contains an example apt source showing the syntax which would be suitable for use with the packaged repository. The full path will need to be specified in the final sources list file. e.g.

 deb copy:///milestone suite main

May need to be modified to:

 deb copy:///media/usb0/milestone suite main

The example source is packaged as ladder.list in the tarball.

The key should normally already be part of a keyring package and installed on the devices. If not, an exported copy of the public key is also included in the tarball which can be included into the device keyring using "apt-key" (which needs to be run as root):

 apt-key add /path/milestone/ladder.gpg

Some scripting / programming support will be needed to make this process seamless on-device, in particular to provide the knowledge of the actual sequence of milestone names, but this is beyond the scope of "ladder", if only because the ladder tarball needs to be unpacked first.

If the system is set with some standard apt sources already, the upgrade will need to only allow "apt-get" to see the ladder repository (because the normal network connection isn't available, so the update would fail). To do this, use apt command line options to reset the location of the SourceList and SourceParts:

 apt-get -o Dir::Etc::SourceList=ladder.list -o Dir::Etc::SourceParts=./dir/ update

(./dir/ should be an empty directory - or a directory containing empty .list files and nothing else.)

The only requirements to use the ladder tarball are to create the relevant source list file, ensure the key is available and then call apt-get update; apt-get upgrade. There is no need for perl, reprepro or anything else used by "ladder" itself.

Example update script

If the configuration file includes the "updatescript" option an example script will be included, listing the value of the "rootpackage" option to be removed. If the "mountpoint" option is set, the DIR variable will be set in the example script as well. (You may need to invest time in a "udev" rule as part of your rootfs to get a known mount point but such rules are beyond the scope of this documentation.)

 #!/bin/sh
 
 set -e
 DIR=/media/ladder
 ROOTPKG=
 CONFIG=-y -o Dir::Etc::SourceList=${DIR}/ladder.list -o Dir::Etc::SourceParts=${DIR}/list.d/
 apt-key add ${DIR}/pubkey.asc
 apt-get ${CONFIG} update
 if [ -n "$ROOTPKG" ]; then
    apt-get ${CONFIG} --purge autoremove $ROOTPKG
 fi
 apt-get ${CONFIG} dist-upgrade
 apt-get ${CONFIG} autoclean

SecureApt

Signing a ladder step repository requires that the secret key is usable without a passphrase and that the secret key is accessible to the root user, either directly or via sudo.

As with anything related to GnuPG, protecting the secret key is the sole responsibility of the key owner. It is recommended that ladder steps are only created in a secure environment comparable with that used to generate the keys. The same requirements apply to the machines which use the secret key to sign the internal milestone repositories, so it may be appropriate to create ladder steps on those machines.

Specifying a keyring directory and key ID

If "keyringdir" is used, the specified directory must contain the public and secret keyrings which contain the specified "key". "ladder" will then make both the secret key and public key accessible to the root user using a temporary keyring in /var/lib/ladder/keys. Only the key available in /var/lib/ladder/keys will be available to the repository signing process. "ladder" only needs to be able to read the secret and public keyrings of the keyringdir specified. Ensure that the secret key is available - without a passphrase - or the repository will not be signed.

Using just a key ID

If "keyringdir" is not used, the user must ensure that the key is available to the root user as ladder requires sudo/root to be able to use apt. Ensure that the specified secret key is available - without a passphrase - to the root user or the repository will not be signed.

 sudo gpg --list-secret-key KEYID

Using "keyringdir" is generally the easiest option.

If the key is not available, the repository simply won't be signed and devices would need to pass the AllowUnauthenticated option to "apt-get" when using the ladder repository. ladder does not add the unauthenticated option to generated upgrade scripts! You can tell a SecureApt repository by the presence of the Release.gpg file.

It is possible to auto-generate GnuPG keys but "ladder" does not support this currently. The main problem is entropy - generating a new GnuPG (or SSH) key requires a lot of entropy, especially as default key lengths increase. It is a lot easier to ensure high entropy when the key generation process is interactive.

Keyring packages are recommended

With careful planning, the security of the step upgrades can be much improved by modifying the update scripts to not add the signing key using "apt-key add" but instead to provide a keyring package in the rootfs itself which contains the public key which will be used to sign the next milestone. This is how Debian arranges keys - the release of milestone A is not made until the key which will be used to sign milestone B has the corresponding public key already included in the keyring package in milestone A.

Such keyring packages themselves need to be in the milestone repository because then the keyring package itself is protected by SecureApt.

Note that keyring packages will make it harder to use the downgrade solution explained below, hence the need for planning.

Key expiries

Key expiry dates will complicate "ladder" usage, especially if downgrades are to be available. If a device was released on milestone D and needs to be downgraded to milestone B, you will have problems if the key used to sign milestone B has since expired. Equally, repairing or servicing a device running milestone B becomes problematic if the key for milestone B has expired whilst the device was in use.

Avoid using expiry dates on keys unless you are very, very confident that a particular milestone will not be in use after a certain date.

Key management

Keys can be revoked but this relies on the devices which need to verify that key being able to download the revocation certificate and then to still have a usable key available for the upgrade. Consider revoking the key for milestone B in the version of the keyring package released with milestone D (milestone C still needs it to be able to upgrade). This allows keys to be revoked on-device but still be usable should it become necessary to repair, service or downgrade.

If a key is compromised, then unless the keyring package in any one milestone still includes a usable key, there may be no way of securely upgrading devices without manually adding a replacement key. Take care of your secret keys.

Steps and milestones

Ladder - as with Debian - only works forwards. Downgrades are not supported. If the rootfs tarball contains an existing apt source which contains packages NEWER than the requested milestone, then the packages downloaded will be for the existing apt source, not the milestone. Check the output with the "-n|--dry-run" option.

However, judicious use of the "rootpackage" option can assist with limited downgrades - especially when the software being downgraded is under your own control. The generated updatescript can use "apt-get --purge autoremove" on the root package. Specifying a core library or special platform dependency package here can allow the rootfs to be returned to a pristine state. The required milestone can then be installed as if from a clean base. This is not quite the same as an explicit downgrade but is a much more reliable mechanism as it provides the equivalent rootfs to when the original milestone was created.

For these reasons, always keep a copy of the original clean rootfs which has no complicating apt sources.

If your root package is a shared library, you can specify multiple root packages in the config file so that all released SONAME versions are removed. Use only spaces to separate packages in the config file.

If you are using keyring packages, ensure that a suitable keyring package is available to the ladder step which purges the root package. To be able to upgrade to the end milestone from a purged rootfs, the keyring package first needs to be upgraded to include the key used to sign the end milestone (although the upgraded keyring package is free to include revoked copies of intermediary keys, if appropriate).

Output

Ladder works in the /var/lib/ladder directory, unpacking the tarball into ./rootfs and creating the repository in a directory named after the milestone.

Results will be /var/lib/ladder/ladder-$name.tgz

Support

"ladder" was written with a specific purpose in mind but is available in Debian in the hope it will be useful for other situations as well. If there are specific situations where "ladder" could be extended to be more useful for others, let me know using the Debian bug tracking system: bugs.debian.org/ladder.

Note that "reprepro" already has snapshot support which is not the same as a "ladder" of milestones. Snapshots include full sources and ancillary packages which are not needed on-device and are intended for build systems and developer use - ladder milestones are intended to provide a small repository which can be used on machines after production.