Basic configuration on any host to backup

Stopping and starting processes: ssd-backup

If you want to create a snapshot of all backed up filesystems together and not of each filesystem separately, you need to stop all processes that write to the backed up locations, remount filesystems temporarily read-only and then create snapshots. After the snapshots are created, the filesystems can be remounted back read-write and the processes can be started again. This task of stopping and starting daemons or processes is performed by the ssd-backup script that is part of the Tar-LVM suite.

Let's now look at the ssd-backup configuration. This script is configured by default using the file /usr/local/etc/ssd-backup.conf.

Supported directives are concisely described directly in the configuration file.

The nosysv or nosystemd directives can disable the System V or Systemd support and can thus allow usage of this script even if no service or systemctl binary is found.

The stopstart directives define all services to be stopped in the ssd-backup stop mode or started in its start mode. The services are stopped in the specified order if they're running and started in the reverse order depending on their initial status. They also allow to specify an optional PID file and/or process name to identify running services if the operating system itself doesn't provide this information for certain service.

The kill directive allows to kill certain processes based on their names. And the survruser entries allow to specify all real usernames whose processes shouldn't be killed if the -u argument is used. This argument instructs ssd-backup to kill all user processes, i.e. all non-root processes that are not excluded by survruser.

The stopcomm and startcomm directives can be used to execute arbitrary commands at the end of the stop mode or at the beginning of the start mode. They can be therefore used for any non-standard operation, e.g. remounting filesystems read-only or read-write, cleanup of some locations, sending a message to users or anything else that's scriptable.

See the ssd-status help for more information and complete list of its arguments.

It's wise to confirm that ssd-backup is configured properly before continuing. Simply by invoking a command sequence that's similar to the following one. But be prepared for short downtime of your services.

All the ssd-backup and mount commands should succeed, of course, if the configuration is correct. If you need to identify the processes that are using files on certain filesystem, try the lsof or fuser commands.

Creating and removing snapshots, backing up: tar-lvm

The snapshot of the writable filesystems backed up from inside of a running operating system can be created by remounting the filesystems read-only, creating LVM snapshots and remounting the filesystems back to their original state, mostly often back read-write. Or by remounting the writable filesystems read-only during the whole backup process. However, the downtime of your services is much longer in the latter case.

All these operations are performed by the tar-lvm script. Its pre mode remounts the filesystems read-only, creates LVM snapshots are remounts LVM filesystems back read-write. Its run mode creates the tar backups and the post mode remounts the non-LVM filesystems back to their initial state.

The tar-lvm script is configured in the file /usr/local/etc/tar-lvm/tar-lvm.conf which has the following syntax.

Supported directives are concisely described directly in the configuration file.

The lvsnapsuffix entry defines the suffix that is appended to the names of the LVM volumes to get the corresponding LVM snapshot volume names.

The noacls entry can disable support for POSIX ACLs. This becomes important on older systems whose GNU tar version doesn't support the --acls option yet.

All other directives, i.e. the fs and lv items specify the filesystems to backup. The fs directive refers to non-LVM devices with filesystems that are remounted read-only during the whole backup process. The device can be specified either by device path or by its UUID.

The lv lines refer to LVM logical volumes by their names and volume groups they belong to. Because new snapshot logical volume is created for each logical volume to backup in the tar-lvm pre mode, its size must be specified as a certain number of percents of its origin.

Furhtermode, both fs and lv entries can contain more optional arguments that list directories or files that should be excluded from the backup. See the lv entry for the home filesystem above for an example. The path should always start with a dot and is relative to the filesystem root directory.

To confirm the validity of new configuration, the following command sequence should succeed. Be prepared for short downtime of your services again if you're using LVM or for longer downtime comprising the whole backup process if your read-write filesystems are not on LVM.

Automating the backup

Wrapper script for one host backup: tar-lvm-one

The tar-lvm-one wrapper script must be configured to simplify the automation of a backup of a specific host. This script invokes the commands described and configured earlier and also mounts and unmounts the backup device in between.

This wrapper script can be configured either on each host separately in local configuration files or in a shared configuration file located on one host denoted as allhost in the local configuration. The syntax of both files is identical and the directives specified in the local configuration file override the shared directives. The shared configuration doesn't have to be used. This is the case if no allhost directive is present in the local file. However, the local configuration file cannot be omitted and must specify all needed directives locally or at least the allhost directive pointing to the host with shared configuration.

Let's now look at an example configuration shared among more hosts from one allhost. The local configuration must then contain the allhost directive on each host. All other entries are optional and become useful only if the shared configuration should be overriden by something more specific on certain host.

The local configuration file /usr/local/etc/tar-lvm/tar-lvm-one.local.conf must contain at least the following part in our example.

In this case, the shared configuration file /usr/local/etc/tar-lvm/tar-lvm-one.shared.conf must define all other directives.

The nosshfs entry supresses usage of the remote SSHFS backup filesystem and also the check that the sshfs and fusermount binaries are present. It should be therefore used on systems without these binaries installed.

As mentioned earlier, the backup device can be either a local device defined by its name or UUID or a remote SSHFS filesystem. All possibilities are specified by the dev line depending on its syntax.

If the backup filesystem is located at a local device and if it is encrypted by LUKS, the dm line is required. It instructs tar-lvm-one to ask for password and decrypt the device before it is mounted.

The mntdir entry is a mount point of the backup filesystem, i.e. a directory to which the backup filesystem is mounted during the backup.

And the rootdir line specifies a path to an existing directory on the backup filesystem where the backups and logs should be stored.

There's one more optional entry in the configuration file and that's the deferssdstart flag. If it is set to true, it instructs the script to defer ssd-backup start as the last operation. The default behaviour differs, ssd-backup start is invoked immediately after the LVM snapshots are created, i.e. before the tar backup itself. This becomes handy if some read-write filesystem is not located on LVM and if no snapshot can be thus created.

The tar-lvm-one configuration and backup can be tested as follows:

Choosing the triggering model: centralized or distributed

There're two ways how to trigger the backups on all hosts: centralized or distributed. The first centralized way must be used if you need to backup whole physical machine with separate backups of its KVM virtual machines to a device that is connected directly to the physical machine. The reason is that it needs to attach and detach the backup device to the virtuals. However, if you want to backup just via SSH with the exception of the machine the device is connected to, you can choose whether to use the distributed or centralized triggering model.

The main difference between both models even if using just a SSHFS storage is that the centralized way is managed by the tar-lvm-all script which controls the whole process, the backups are triggered sequentially and there's a maximal number of parallel backups that can run at once. The distributed way is managed and controlled on each host separately, backups are usually triggered from the Cron scheduler at a specific time and they all run independently of each other. Both models use the tar-lvm-one wrapper script. This script is invoked by tar-lvm-all in the former case, i.e. in the centralized model, and directly from Cron in the latter case, i.e. in the distributed model.

In fact, both models can be combined and used at once, but there's one significant limitation. If tar-lvm-all attaches the backup device to some of its virtuals that store the backups directly to this device, the distributed model shouldn't be mixed up with the centralized one unless it uses different backup device. The backup device can be mounted just by one machine at a time if using the direct access to avoid data corruption.

Centralized triggering model: tar-lvm-all

If you chose the centralized model of triggering the backups, let's proceed with configuration of the tar-lvm-all script. This script must run on the physical machine the backup device is connected to if this machine or its KVM virtuals do not want to use SSHFS, but the more efficient direct access method to access the backup device. Otherwise if all hosts to backup store their backups remotely using SSHFS, it can be run on any host - even on a host that's not going to be backed up.

The tar-lvm-all configuration file is located at /usr/local/etc/tar-lvm/tar-lvm-all.conf and it looks as follows.

If at least one KVM virtual machine is backed up to a local device connected to the physical machine and not to a remote SSHFS filesystem, both pmdev and vmdev entries must be defined. The pmdev entry defines the backup device on the physical machine that should be connected as the vmdev device to the virtual machines. However, both entries are required only if at least one vm entry is present.

Another optional entry is the passprompt flag. It instructs tar-lvm-all to ask for password that should be passed to tar-lvm-one if the backup device is encrypted.

The pmbackup flag determines whether to backup the physical machine (i.e. the machine the script runs at) or not. If it is set to true, root@localhost must be accessible from root@localhost by SSH public key authentication, because the backup is performed in the same way as in the case of backing up other host. Just the backup device doesn't have to be attached.

The most important entries are the vm and host directives that specify the nodes to backup. More specifically, the virtual machines with direct access to the backup device and hosts to backup remotely via SSH. Their names are especially important by the vm lines because they are equal to the names of the KVM virtual machines. They can be also used to refer to DNS hostnames if they belong to the resolver search domain, but if the names are not resolvable, an optional colon can be used to append FQDN, IP address or another resolvable hostname entry.

There's one more entry that influences the way how the remote SSHFS backups are triggered - the parhostnum directive. It specifies maximal number of hosts to backup in parallel and if this number of hosts being backed up is reached, all reamining hosts must wait.

The remaining lines configure email notification and they're mostly self-explanatory. The only important thing to take into account that may not be obvious is the fact that if no smtpserver directive is specified, then local MTA is used to deliver emails. If it is specified, emails are delivered directly via SMTP to the specified mail server. However, that's not the preferred solution. If the delivery fails, there's neither a way how to inform about the error nor any later repeated delivery attempt. The error output can get lost easily.

If you want to test the backup for all machines and hosts, simply invoke tar-lvm-all on the host that manages the backup process centrally.

Distributed triggering model: tar-lvm-one

If you chose the distributed model of triggering the backups, there's no need to configure any other tool, because tar-lvm-one is used for this purpose and this script was already configured earlier.