Backup and Restore

Table of Contents

1. Back up Uyuni
2. Administering the Database with smdba
3. Database Backup with smdba
- 3.1. Perform a Manual Database Backup
- 3.2. Scheduling Automatic Backups
4. Restore from Backup
5. Archive Log Settings
6. Retrieve an Overview of Occupied Database Space
7. Move the Database
8. Recover From a Crashed Root Partition
9. Database Connection Information

Back up your Uyuni installation regularly, to prevent data loss. Because Uyuni relies on a database as well as the installed program and configurations, it is important to back up all components of your installation. This chapter contains information on the files you need to back up, and introduces the smdba tool to manage database backups. It also contains information about restoring from your backups in the case of a system failure.

Regardless of the backup method you use, you must have available at least three times the amount of space your current installation uses. Running out of space can result in backups failing, so check this often.

1. Back up Uyuni

The most comprehensive method for backing up your Uyuni installation is to back up the relevant files and directories. This can save you time in administering your backup, and can be faster to reinstall and re-synchronize in the case of failure. However, this method requires significant disk space and could take a long time to perform the backup.

If you want to back up only the required files and directories, use the following list. To make this process simpler, and more comprehensive, we recommend backing up the entire /etc and /root directories, not just the ones specified here. Some files only exist if you are actually using the related SUSE Manager feature.

/etc/cobbler/
/etc/dhcp.conf
/etc/fstab and any ISO mount points you require.

If your UUID has changed, ensure you have updated the fstab entries accordingly.
/etc/rhn/
/etc/salt
/etc/sudoers
/etc/sysconfig/rhn/
/root/.gnupg/
/root/.ssh

This file exists if you are using an SSH tunnel or SSH push. You also need to have saved a copy of the id-susemanager key.
/root/ssl-build/
/srv/formula_metadata
/srv/pillar
/srv/salt
/srv/susemanager
/srv/tftpboot/
/srv/www/cobbler
/srv/www/htdocs/pub/
/srv/www/os-images
/var/cache/rhn
/var/cache/salt
/var/lib/cobbler/
/var/lib/cobbler/templates/ (before version 4.0 it is /var/lib/rhn/kickstarts/)
/var/lib/Kiwi
/var/lib/rhn/
/var/run/pgsql/
/var/lib/salt/
/var/run/salt/
/var/spacewalk/
Any directories containing custom data such as scripts, Kickstart or AutoYaST profiles, and custom RPMs.

You also need to back up your database, which you can do with the smdba tool.

Procedure: Restoring from a Manual Backup

Re-deploy the Uyuni Server with mgradm install podman.
Re-synchronize your Uyuni repositories using either the Uyuni Web UI, or with the mgr-sync tool at the command prompt. You can choose to re-register your product, or skip the registration and SSL certificate generation sections.
Re-install the /var/lib/containers/storage/volumes/root/ssl-build/rhn-org-httpd-ssl-key-pair-MACHINE_NAME-VER-REL.noarch.rpm package.
Schedule the re-creation of search indexes next time the rhn-search service is started. This command produces only debug messages, it does not produce error messages:
```
rhn-search cleanindex
```
Check whether you need to restore /var/lib/containers/storage/volumes/var-spacewalk/_data/packages/. If /var/lib/containers/storage/volumes/var-spacewalk/_data/packages/ was not in your backup, you need to restore it. If the source repository is available, you can restore `/var/lib/containers/storage/volumes/var-spacewalk/_data/packages/ with a complete channel synchronization:
```
mgr-sync refresh --refresh-channels
```

2. Administering the Database with smdba

The smdba tool is used for managing a local PostgreSQL database. It allows you to back up and restore your database, and manage backups. It can also be used to check the status of your database, and perform administration tasks, such as restarting. For more information about the smdba tool, see [reference:cli-smdba].

The smdba tool works with local PostgreSQL databases only, it does not work with remotely accessed databases, or Oracle databases.

To access a shell inside the Server container, run mgrctl term on the container host. From there, one can run the CLI tools as usual.

The smdba tool requires sudo access, to execute system changes. Ensure you have enabled sudo access for the admin user before you begin, by checking the /etc/sudoers file for this line:

admin   ALL=(postgres) /usr/bin/smdba

Check the runtime status of your database with:

smdba db-status

This command returns either online or offline, for example:

Checking database core...       online

Starting and stopping the database can be performed with:

smdba db-start

And:

smdba db-stop

3. Database Backup with smdba

The smdba tool performs a continuous archiving backup. This backup method combines a log of every change made to the database during the current session, with a series of more traditional backup files. When a crash occurs, the database state is first restored from the most recent backup file on disk, then the log of the current session is replayed exactly, to bring the database back to a current state. A continuous archiving backup with smdba is performed with the database running, so there is no need for downtime.

This method of backing up is stable and generally creates consistent snapshots, however it can take up a lot of storage space. Ensure you have at least three times the current database size of space available for backups. You can check your current database size by navigating to /var/lib/pgsql/ and running df -h.

The smdba tool also manages your archives, keeping only the most recent backup, and the current archive of logs. The log files can only be a maximum file size of 16 MB, so a new log file is created when the files reach this size. Every time you create a new backup, previous backups are purged to release disk space. We recommend you use systemd timer on host system to schedule your smdba backups to ensure that your storage is managed effectively, and you always have a backup ready in case of failure.

3.1. Perform a Manual Database Backup

To access a shell inside the Server container, run mgrctl term on the container host. From there, run command line tools as usual.

The smdba tool can be run in the container shell. We recommend you run a manual database backup immediately after installation, or if you have made any significant changes to your configuration.

When smdba is run for the first time, or if you have changed the location of the backup, it needs to restart your database before performing the archive. This results in a small amount of downtime. Regular database backups do not require any downtime.

Procedure: Performing a Manual Database Backup

Allocate permanent storage space for your backup. This example uses a directory located at /var/spacewalk/. This becomes a permanent target for your backup, so ensure it remains accessible by your server at all times.
In the container terminal, enter:
```
mgrctl term
```
In your backup location, create a directory for the backup:

As root:
```
install -d -o postgres -g postgres -m 700 /var/spacewalk/db-backup
```
Ensure you have the correct permissions set on the backup location:
```
chown postgres:postgres /var/spacewalk/db-backup
```
To create a backup for the first time, run the smdba backup-hot command with the enable option set. This creates the backup in the specified directory, and, if necessary, restart the database:
```
smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup
```
This command produces debug messages and finishes sucessfully with the output:
```
INFO: Finished
```
Check that the backup files exist in the /var/spacewalk/db-backup directory, to ensure that your backup has been successful.

3.2. Scheduling Automatic Backups

You do not need to shut down your system to perform a database backup with smdba. However, because it is a large operation, database performance can slow down while the backup is running. We recommend you schedule regular database backups for a low-traffic period, to minimize disruption.

Ensure you have at least three times the current database size of space available for backups. You can check your current database size by running:

mgrctl exec "df -h /var/lib/pgsql"

Procedure: Creating a directory for the backup (do it only once)

In the container terminal, enter:
```
mgrctl term
```
In your backup location, create a directory for the backup. As root, enter:
```
install -d -o postgres -g postgres -m 700 /var/spacewalk/db-backup
```
Ensure you have the correct permissions set on the backup location:
```
chown postgres:postgres /var/spacewalk/db-backup
```

Procedure: Scheduling Automatic Backups

Create a system unit file at /etc/systemd/system/server-db-backup.service with the content:

[Unit]
Description="Server backup service"

[Service]
ExecStart=mgrctl exec "smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup"

Create a systemd timer file at /etc/systemd/system/server-db-backup.timer with the content:

[Unit]
Description="Run server database backup relative to activation time"

[Timer]
OnCalendar=Mo..Sun *-*-* 2:00:00
Unit=server-db-backup.service

[Install]
WantedBy=multi-user.target

Check that all configuration are correct:

systemd-analyze verify /etc/systemd/system/server-db-backup.*

Enable the timer:

systemctl enable --now server-db-backup.timer

4. Restore from Backup

The smdba tool can be used to restore from backup in the case of failure.

Procedure: Restoring from Backup

In the container terminal, enter:
```
mgrctl term
```
Shut down the database:
```
smdba db-stop
```
Start the restore process and wait for it to complete:
```
smdba backup-restore start
```
Restart the database:
```
smdba db-start
```
Check if there are differences between the RPMs and the database.
```
spacewalk-data-fsck
```

5. Archive Log Settings

Archive logging allows the database management tool smdba to perform hot backups. In Uyuni with an embedded database, archive logging is enabled by default.

PostgreSQL maintains a limited number of archive logs. Using the default configuration, approximately 64 files with a size of 16 MiB are stored.

Creating a user and syncing the channels:

SLES12-SP2-Pool-x86_64
SLES12-SP2-Updates-x86_64
SLE-Manager-Tools12-Pool-x86_64-SP2
SLE-Manager-Tools12-Updates-x86_64-SP2

PostgreSQL generates an additional roughly 1 GB of data. So it is important to think about a backup strategy and create a backups in a regular way.

Archive logs are stored at /var/lib/pgsql/data/pg_xlog/ (postgresql).

6. Retrieve an Overview of Occupied Database Space

Database administrators may use the subcommand space-overview to get a report about occupied table spaces, for example:

smdba space-overview

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH

Tablespace  | Size (Mb) | Avail (Mb) | Use %
------------+-----------+------------+------
postgres    | 7         | 49168      | 0.013
susemanager | 776       | 48399      | 1.602

The smdba command is available for PostgreSQL. For a more detailed report, use the space-tables subcommand. It lists the table and its size, for example:

smdba space-tables

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH

Table                                 | Size
--------------------------------------+-----------
public.all_primary_keys               | 0 bytes
public.all_tab_columns                | 0 bytes
public.allserverkeywordsincereboot    | 0 bytes
public.dblink_pkey_results            | 0 bytes
public.dual                           | 8192 bytes
public.evr_t                          | 0 bytes
public.log                            | 32 kB
...

7. Move the Database

It is possible to move the database to another location. For example, if database storage space is running low.

Procedure: Moving the Database

The default storage location for Uyuni is /var/lib/pgsql/. If you would like to move it, for example to /storage/postgres/, proceed as follows.
At the command prompt, as root, stop the running database:
```
rcpostgresql stop
```
Shut down the running spacewalk services:
```
spacewalk-service stop
```
Copy the current working directory structure with cp using the -a, --archive option. For example:
```
cp --archive /var/lib/pgsql/ /storage/postgres/
```
This command copies the contents of /var/lib/pgsql/ to /storage/postgres/pgsql/.

The contents of the /var/lib/pgsql directory needs to remain the same, otherwise the Uyuni database may malfunction. You also should ensure that there is enough available disk space.
Mount the new database directory:
```
mount /storage/postgres/pgsql
```
Make sure ownership is postgres:postgres and not root:root by changing to the new directory and running these commands:
```
cd /storage/postgres/pgsql/
ls -l
```
Outputs:
```
total 8
drwxr-x---  4 postgres postgres   47 Jun  2 14:35 ./
```
Add the new database mount location to your servers fstab by editing etc/fstab.
Start the database with:
```
rcpostgresql start
```
Start the spacewalk services:
```
spacewalk-service start
```

8. Recover From a Crashed Root Partition

If your root partition has crashed, you can restart the Uyuni Server with some additional steps. This section assumes you have setup your server using separate partitions for the database and for channels, mounted at /var/lib/pgsql and /var/spacewalk/.

After a new installation of a system most users and groups get different IDs. Most backup systems store the names instead of the IDs and will restore the files with the correct ownership and permissions. But if you mount existing partitions, you must align the ownership to the new system.

Procedure: Recovering from a Crashed Root Partition

Install Uyuni. Do not mount the /var/spacewalk and /var/lib/pgsql partitions. Wait for the installation to complete before going on to the next step.
Shut down the database:
```
rcpostgresql stop
```
Shut down the services:
```
spacewalk-service stop
```
Mount /var/spacewalk and /var/lib/pgsql partitions.
Restore the directories listed in Back up Uyuni.
Start the database:
```
rcpostgresql start
```
Start the spacewalk services:
```
spacewalk-service start
```

Uyuni should now operate normally without loss of your database or synchronized channels.

9. Database Connection Information

You can set information for connecting to the Uyuni database by adding or editing these variable in /etc/rhn/rhn.conf:

db_backend = postgresql
db_user = susemanager
db_password = susemanager
db_name = susemanager
db_host = localhost
db_port = 5432
db_ssl_enabled =