man rsbackup (1): rsync-based backup utility

SYNOPSIS

rsbackup [OPTIONS] [--] [SELECTOR...]
rsbackup --retire [OPTIONS] [--] [SELECTOR...]
rsbackup --retire-device [OPTIONS] [--] DEVICE...

DESCRIPTION

Backs up files from one or more (remote) destinations to a single backup storage directory, preserving their contents, layout, ownership, permissions, timestamps and hardlink structure.

Incremental backups are achieved by hard-linking identical files within successive backups of the same files.

OPTIONS

Action Options

At least one of these options must be specified. When multiple actions are specified, they are executed in the order shown below.

--backup, -b: Make a backup of the selected volumes. At most one backup of a given volume will be made per day.
--retire-device: Retire the named devices. Retiring a device only means deleting the records of it. Files on the device itself are not touched.
: If the device is still listed in the configuration file then you will be asked whether you really want to retire it; you can suppress this check with the --force option.
--retire: Retire the named hosts and volumes. Retiring a volume means deleting any available backups for the volume and the records of them. Records corresponding to backups on unavailable devices are not removed.
: If you just want to remove backup records for retired volumes but want to keep the backups, you should either manually remove the records (see SCHEMA below), or rename it within the volume.
: If the volume is still listed in the configuration file then you will be asked whether you really want to retire it; you can suppress this check with the --force option.
--prune, -p: Prune old backups of selected volumes. See PRUNING below.
--prune-incomplete,-P: Prune incomplete backups of selected volumes. Any backups that failed before completion will be removed.
--html PATH, -H PATH: Write an HTML report to PATH. The report covers all volumes, not just selected ones. PATH can be - to write to standard output.
--text PATH, -T PATH: Write a plain text report to PATH. The report covers all volumes, not just selected ones. PATH can be - to write to standard output.
--email ADDRESS, -e ADDRESS: Email a report to ADDRESS. The contents is equivalent to the output of --text and --html.
--dump-config: Writes the parsed configuration file to standard output. Must not be combined with any other action option.
: With --verbose, the configuration file is annotated with descriptive comments.

General Options

--config PATH, -c PATH: The path to the configuration file. The default is /etc/rsbackup/config.
--store PATH, -s PATH: Specify the destination directory to back up to. Using this option (possibly more than once) is equivalent to removing the store directives from the configuration file and replacing them with the paths give in --store options.
: This option implicitly enables the --warn-store option.
--verbose, -v: Enable verbose mode. Various messages will be displayed to report progress and the rsync --quiet option is suppressed.
--dry-run, -n: Enable dry-run mode. Commands will be displayed but nothing will actually be done.
--force, -f: Suppress checks made when retiring devices and volumes.
--wait, -w: Waits rather than giving up if another copy of rsbackup is running.
--database, -D PATH: Override the path to the backup database.
--help, -h: Display a usage message.
--version, -V: Display the version number.

Report Verbosity

--logs VERBOSITY

Controls which logfiles for a given volume/device pair to include in the report. The possible values of VERBOSITY are:

all: Includes all nonempty logfiles, even if the backup succeeded.
errors: Includes all error logfiles.
recent: Includes only the most recent error logfile.
latest: Includes only the latest logfile, even if the backup succeeded.
failed: Includes only the most recent logfile but only if that attempt failed. This is the default.

Warning Options

--warn-unknown: Display warnings for unknown devices, hosts and volumes. (Warnings will always be included in the report, this refers to runtime error output.)
--warn-store: Display warnings for unsuitable store directories and unavailable devices.
--warn-unreachable: Display warnings for unreachable hosts.
--no-warn-partial: Suppress warnings for rsync "partial transfer" diagnostics (which are on by default).
--warn-all, -W: Enable all --warn- options.
--no-errors: Suppress display of errors from rsync.

Volume Selection

The list of selectors on the command line determines what subset of the known volumes are backed up, pruned or retired. The following selectors are possible:

HOST: Select all volumes for the host.
HOST:VOLUME: Select the volume.
-HOST: Deselect all volumes for the host.
-HOST:VOLUME: Deselect the volume.
*: Select all volumes.

If no hosts or volumes are specified on the command line then all volumes are selected for backing up or pruning. For retiring, you must explicitly select hosts or volumes to retire and only positive selections are possible.

CONFIGURATION FILE

The config file contains global directives and a series of host stanzas. Each host stanze in turn contains host directives and volume stanzas. Although it is not enforced it is suggested that host and volume stanzas are indented.

Comments are introduced by an initial "#".

Command arguments may be quoted, using "double quotes". Quotes and backslashes within quoted strings are escaped with backslashes.

Global Directives

Global directives control some general aspect of the program.

colors GOOD BAD: The colors used to represent good states (a recent backup) and bad states (no sufficiently recent backup).
: GOOD and BAD are integer values representing RGB triples. It is most convenient to write them in hex, e.g. as 0xRRGGBB. For example, black is 0x000000, red is 0xFF0000, and so on.
device DEVICE: Names a device. This can be used multiple times. The store must have a file called STORE/device-id which contains a known device name. Backups will only be made to known devices.
: When a device is lost or destroyed, remove its device entry and use the --prune-unknown option to delete records of backups on it.
: Device names may contain letters, digits, dots and underscores.
include PATH: Include another file as part of the configuration. If PATH is a directory then the files within it are included (excluding dotfiles, backup and recovery files).
keep-prune-logs DAYS: The number of days to keep records of pruned backups for. The default is 31.
lock PATH: Enable locking. If this directive is present then PATH will be used as a lockfile for operations that change anything (--backup, --prune, etc).
: The lock is made by opening PATH and calling flock(2) on it with LOCK_EX.
logs PATH: The directory to store logfiles and backup records. The default is /var/log/backup.
post-access-hook COMMAND...: A command to execute after all backup and prune operations. This is executed only once per invocation of rsbackup. A backup is still considered to have succeeded even if the post-access hook fails (i.e. exits nonzero). See HOOKS below.
pre-access-hook COMMAND...: A command to execute before anything that accesses any backup devices (i.e. backup and prune operations). This is executed only once per invocation of rsbackup and if it fails (i.e. exits nonzero) then rsbackup terminates immediately. See HOOKS below.
public true|false: If true, backups are public. Normally backups must only be accessible by the calling user. This option suppresses the check.
report-prune-logs DAYS: The number of days of pruning logs to put in hte report. The default is 3.
sendmail PATH: The path to the executable to use for sending email. The default is platform-dependent but typically /usr/sbin/sendmail. The executable should support the -t, -oee, -oi and -odb options.
store PATH: A path at which a backup device may be mounted. This can be used multiple times.
store-pattern PATTERN: A glob(7) pattern matching paths at which a backup device may be mounted. This can be used multiple times.
stylesheet PATH: The path to the stylesheet to use in the HTML report. If this is absent then a built-in default stylesheet is used.

Inheritable Directives

Inheritable directives control an aspect of one or more backups. They can be specified at the global level or in a host or volume stanza (see below). If one appears in multiple places then volume settings override host settings and host settings override global settings.

hook-timeout SECONDS: How long to wait before concluding a hook has hung, in seconds. The default is 0, which means to wait indefinitely.
max-age DAYS: The maximum age of the most recent backup before you feel uncomfortable. The default is 3, meaning that if a volume hasn't been backed up in the last 3 days it will have red ink in the HTML report.
min-backups COUNT: The minimum number of backups for each volume to keep on each store, when pruning. The default is 1.
: (This is an alias for prune-parameter min-backups and will be removed in a future version.)
post-backup-hook COMMAND...: A command to execute after finishing a backup, or after it failed. A backup is still considered to have succeeded even if the post-backup hook fails (exits nonzero). See HOOKS below.
pre-backup-hook COMMAND...: A command to execute before starting a backup. If this hook fails (i.e. exits nonzero) then the backup is not made and the post-backup hook will not be run. See HOOKS below.
: This hook can override the source path for the backup by writing a new source path to standard output.
prune-age DAYS: The age at which a backup may be pruned. The default is 366, meaning a backup will never be pruned until it is at least a whole year old.
: (This is an alias for prune-parameter prune-age and will be removed in a future version.)
prune-parameter NAME VALUE: Set a parameter for the pruning policy. See PRUNING below.
prune-parameter --remove NAME: Remove a parameter for pruning policy.
prune-policy NAME: The pruning policy to use. See PRUNING below.
rsync-timeout SECONDS: How long to wait before concluding rsync has hung, in seconds. The default is 0, which means to wait indefinitely.
ssh-timeout SECONDS: How long to wait before concluding a host is down, in seconds. The default is 60.

Host Directives

A host stanza is started by a host directive.

host HOST: Introduce a host stanza. The name is used for the backup directory for this host.

The following directives, and volume stanzas (see below), can appear in a host stanza:

always-up true|false: If true, the host is expected to always be available. If it is not then a warning will be issued when making a backup if it is not. Failed attempts to make a backup will also be recorded as failures for always-up hosts (normally hosts that cannot be reached are silently skipped).
devices PATTERN: A glob(3) pattern restricting the devices that this host will be backed up to.
: Note that only backup creation honors this restriction. Pruning and retiring do not.
hostname HOSTNAME: The SSH hostname for this host. The default is the name from the host stanza.
: The hostname localhost is treated specially: it is assumed to always be identical to the local system, so files will be read from the local filesystem.
priority INTEGER: The priority of this host. Hosts are backed up in descending priority order. The default priority is 0.
user USERNAME: The SSH username for this host. The default is not to supply a username.

In addition, inheritable directives can appear in a host stanza, and override any appearance of them at the global level.

Conventionally the contents of a host stanza are indented.

Remote hosts are accessed by SSH. The user rsbackup runs as must be able to connect to the remote host (and without a password being entered if it is to be run from a cron job or similar).

Volume Directives

A volume stanza is started by a volume directive.

volume VOLUME PATH: Introduce a volume stanza. The name is used for the backup directory for this volume. The path is the absolute path on the host.

The following directives can appear in a volume stanza:

check-file PATH: Checks that PATH exists before backing up the volume. PATH may be either an absolute path or a relative path (to the root of the volume). It need not be inside the volume though the usual use would be to check for a file which is always present there.
: This check is done before executing the pre-backup-hook, so it applies to the real path to the volume, not the rewritten path.
check-mounted true|false: If true, checks that the volume's path is a mount point before backing up the volume.
: This check is done before executing the pre-backup-hook, so it applies to the real path to the volume, not the rewritten path.
: Note that if multiple check- options are used, all checks must pass for the volume to be backed up.
exclude PATTERN: An exclusion for this volume. The pattern is passed to the rsync --exclude option. This directive may appear multiple times per volume.
: See the rsync man page for full details.
traverse true|false: If true, traverse mount points. This suppresses the rsync --one-file-system option.

In addition, inheritable directives can appear in a volume stanza, and override any appearance of them at the host or global level.

Conventionally the contents of a volume stanza are indented.

PRUNING

This is process of removing old backups (using the --prune option). The pruning policy used to determine which backups to remove is set with the inheritable prune-policy directive, and parameters to the policy set via the prune-parameter directive.

The available policies are listed below. The default policy is age.

age

This policy deletes backups older than a minimum age, provided a minimum number of backups on a device remain available. The following pruning parameters are supported:

min-backups: The minimum number of backups of the volume to maintain on the device. Pruning will never cause the number of backups to fall below this value. The default (and minimum) is 1.
prune-age: The age after backups become eligible for pruning, in days. Only backups more than this many days old will be pruned. The default is 366 and the minimum is 1.

For backwards compatibility, these values can also be set using the directives of the same name. This will be disabled in a future version.

decay

This policy thins out backups older than a minimum age, using a configurable decay pattern that arranges to keep a declining number of backups with age. The following pruning parameters are supported:

decay-start: The age after backups become eligible for pruning, in days. Only backups more than this many days old will be pruned. The default is 1 and the minimum is 1.
decay-limit: The age after which backups are always pruned, in days. Backups older than this will always be pruned unless this would leave no backups at all. The default is 366 and the minimum is 1.
decay-scale: The scale at which the decay window is expanded. The default is 2 and the minimum is 2.
decay-window: The size of the decay window. The default is 1 and the minimum is 1.

exec

This policy executes a subprogram with parameters and additional information supplied in the environment.

The following parameters are supported:

path: The path to the subprogram to execute.

Any additional parameters are supplied to the subprogram via environment variables, prefixed with PRUNE_. Additionally the following environment variables are set:

PRUNE_DEVICE: The name of the device containing the backup.
PRUNE_HOST: The name of the host.
PRUNE_ONDEVICE: The list of backups on the device, by age in days. This list excludes any that have already been scheduled for pruning, and includes the backup under consideration (i.e. the value of BACKUP_AGE will appear in this list).
PRUNE_TOTAL: The total number of backups of this volume on any device. Note that it does not include backups on other devices that have just been selected for pruning by another call to the subprogram.
PRUNE_VOLUME: The name of the volume.

These environment variables all override any parameters with clashing names.

The output should be a list of backups to prune, one per line (in any order). Each line should contain the age in days of the backup to prune (i.e. the same value as appeared in PRUNE_ONDEVICE), followed by a colon, followed by the reason that this backup is to be pruned.

As a convenience, if the argument to prune-policy starts with / then the exec policy is chosen with the policy name as the path parameter.

never

This policy never deletes any backups.

HOOKS

A hook is a command executed by rsbackup just before or just after some action. The command is passed directly to execvp(3); to use a shell command, therefore, either wrap it in a script or invoke the shell with the -c option.

All hooks are run in --dry-run mode. Hook scripts must honor RSBACKUP_ACT which will be set to false in this mode and true otherwise.

Access Hooks

Access hooks are executed (once) before doing anything that will access backup devices (even just to read them).

The following environment variables are set when an access hook is executed:

RSBACKUP_ACT: Set to false in --dry-run mode and true otherwise.
RSBACKUP_DEVICES: A space-separated list of known device names.
RSBACKUP_HOOK: The name of the hook (i.e. pre-access-hook, etc). This allows a single hook script to serve as the implementation for multiple hooks.

Backup Hooks

Backup hooks are executed just before or just after a backup is made.

The following environment variables are set when a backup hook is executed:

RSBACKUP_ACT: Set to false in --dry-run mode and true otherwise.
RSBACKUP_DEVICE: The target device name for the backup.
: Note that this may be removed in a future version.
RSBACKUP_HOOK: The name of the hook (i.e. pre-backup-hook, etc). This allows a single hook script to serve as the implementation for multiple hooks.
RSBACKUP_HOST: The name of the host.
RSBACKUP_SSH_HOSTNAME: The SSH hostname of the host.
: Recall that rsbackup treats the hostname localhost specially. If the hook also needs to do so then it must duplicate this logic.
RSBACKUP_SSH_TARGET: The SSH hostname and username combined for passing to ssh(1).
: This will be username@hostname or just hostname depending on whether a SSH username was set.
RSBACKUP_SSH_USERNAME: The SSH username of the host. If no SSH username was set, this variable will not be set.
RSBACKUP_STATUS: (Only for post-backup-hook). Either ok or failed.
RSBACKUP_STORE: The path to the store directory where the device is mounted.
RSBACKUP_VOLUME: The name of the volume.
RSBACKUP_VOLUME_PATH: The path to the volume.

The error output from backup hooks is stored in the same backup record as the output from rsync.

NOTE: The current behavior is that the pre/post backup hooks are run separately for each backup. In a future version, they may be run only once for all backups of a given volume, in which case RSBACKUP_DEVICE will no longer be set.

See rsbackup-snapshot-hook(1) for a hook program that can be used to back up from Linux LVM snapshots.

BACKUP LIFECYCLE

Adding A New Host

To add a new host create a host entry for it in the configuration file.

To back up the local host, specify hostname localhost. Otherwise you can usually omit hostname.

You may want to set host-wide values for prune-age, max-age and min-backups.

A host with no volumes has no effect.

Adding A New Volume

To add a new volume create a volume entry for it in the relevant host section of the configuration file.

Add exclude options to skip files you don't want to back up. This might include temporary files and the contents of "trash" directories.

If the volume contains mount points, and you want to back up the contents of the subsiduary filesystems, then be sure to include the traverse option.

You may want to set per-volume values for prune-age, max-age and min-backups.

Adding A New Device

To add a new device, format and mount it and create a device-id file in its top-level directory. Add a device entry for it in the configuration file and a store entry mentioning its usual mount point.

Under normal circumstances you should make sure that the backup filesystem is owned by root and mode 0700.

Making Backups

To backup up all available volumes to all available devices:

rsbackup --backup

You will probably want to automate this. To only back up a limited set of volumes specify selection arguments on the command line.

Pruning Backups

To prune old backups:

rsbackup --prune --prune-incomplete

You will probably want to automate this.

An "incomplete backup" occurs when a backup of a volume fails or is interrupted before completion. They are not immediately deleted because rsync may be able to use the files already transferred to save effort on subsequent backups on the same day, or (if there are no complete backups to use for this purpose) later days.

Retiring A Host

Retiring a host means removing all backups for it. The suggested approach is to remove configuration for it and then use rsbackup --retire HOST to remove its backups too. You can do this the other way around but you will be prompted to check you really meant to remove backups for a host still listed in the configuration file.

If any of the backups for the host are on a retired device you should retire that device first.

Retiring A Volume

Retiring a volume means removing all backups for it. It is almost the same as retiring a whole host but the command is rsbackup --retire HOST:VOLUME.

You can retire multiple hosts and volumes in a single command.

Retiring A Device

Retiring a device just means removing the records for it. Use rsbackup --retire-device DEVICE to do this. The contents of the device are not modified; if you want that you must do it manually.

You can retire multiple devices in a single command.

RESTORING

Restore costs extra l-)

Manual Restore

The backup has the same layout, permissions etc as the original system, so it's perfectly possible to simply copy files from a backup directory to their proper location.

Be careful to get file ownership right. The backup is stored with the same numeric user and group ID as the original system used.

Until a backup is completed, or while one is being pruned, a corresponding .incomplete file will exist. Check for such a file before restoring any given backup.

Restoring With rsync

Supposing that host chymax has a volume called users in which user home directories are backed up, and user rjk wants their entire home directory to be restored, an example restore command might be:

rsync -aSHz --numeric-ids /store/chymax/users/2010-04-01/rjk/. chymax:~rjk/.

You could add the --delete option if you wanted to restore to exactly the status quo ante, or at the opposite extreme --existing if you only wanted to restore files that had been deleted.

You might prefer to rsync back into a staging area and then pick files out manually.

Restoring with tar

You could tar up a backup directory (or a subset of it) and then untar it on the target. Remember to use the --numeric-owner option to tar.

STORE VALIDITY

A store may be in the following states:

available: The store can be used for a backup.
unavailable: The store cannot be used for a backup. Normally this does not generate an error but --warn-store
bad: The store cannot be used for a backup. This always generates an error message, but does not prevent backups to other stores taking place.
fatally broken: The store cannot be used for a backup. The program will be terminated.

The states are recognized using the following tests (in this order):

If the store path does not exist, the store is bad.
If the store does not have a device-id file then it is unavailable. If it has one but reading it raises an error then it is bad.
If the store's device-id file contains an unknown device name then it is bad.
If the store's device-id file names the same device as some other store then it is fatally broken.
If the store is not owned by root then it is bad. This check can be overridden with the public directive.
If the store can be read or written by group or world then it is bad. This check can be overridden with the public directive.

FILES

/etc/rsbackup/config: Configuration file.
LOGS/backups.db: The backup records. See SCHEMA below.
STORE/HOST/VOLUME/YYYY-MM-DD: One backup for a volume.
STORE/HOST/VOLUME/YYYY-MM-DD.incomplete: Flag file for an incomplete backup.

SCHEMA

backups.db is a SQLite database. It contains a single table with the following definition:

CREATE TABLE backup (
  host TEXT,
  volume TEXT,
  device TEXT,
  id TEXT,
  time INTEGER,
  pruned INTEGER,
  rc INTEGER,
  status INTEGER,
  log BLOB,
  PRIMARY KEY (host,volume,device,id)
)

Each row represents a completed backup. The meanings of the fields are as follows:

host: The name of the host the backup was taken from.
volume: The name of the volume the backup was taken from.
device: The name of the device the backup was written to.
id: The unique identifier for the backup. Currently this is the date the backup was made, in the format YYYY-MM-DD but this may be changed in the future.
time: The time that the backup was started, as a time_t.
pruned: The time that backup pruning started (if it is underway) or finished (if it is complete), as a time_t.
rc: The exit status of the backup process. 0 means success.
status: Status of this backup. See below.
log: The log output of rsync(1) and hooks. If the backup status is pruning or pruned (see below) then this contains the reason for the pruning.

Possible status values are:

0: Unknown status. Not normally seen.
1: Internally this means the backup is underway. If seen externally after rsbackup terminates it means the backup is incomplete.
2: Backup is complete.
3: Backup has failed.
4: Pruning has started.
5: Pruning has completed.

rsbackup is not designed with concurrent access to this table in mind. Therefore it is recommended that you only modify its contents when the program is not running.

HISTORICAL BEHAVIOR

Older versions of rsbackup stored the logs for each backup in a separate file. If such files are encountered then rsbackup will automatically populate backups.db from them and then delete them.

Older versions of rsbackup logged pruning information to a pruning logfile. These files will be deleted at the same rate as records of pruned backups in the database. They are not included in the report.

AUTHOR

Richard Kettlewell <[email protected]>