Release notes for Uyuni Server

Repository synchronization has been optimized to perform faster than in previous versions. This applies to if the synchronization is triggered in the WebUI, or from the command prompt using the spacewalk-repo-sync command. It also applies whether the synchronization is invoked manually, or automatically as part of product or custom channel synchronizations. The performance improvement is up six times faster than previous versions, but the improvement depends mostly on your hardware setup, especially the number of CPUs, and how many packages are being synchronized.

IMPORTANT: This requires a vendor change for the package python3-psycopg2-2.8.4-2.1.uyuni.x86_64.

After running zypper update you will need force the vendor change with

Then update again again, so the spacewalk-backend subpackages are updated:

For Uyuni 2020.06 python3-psycopg2-2.8.4 should be already part of openSUSE Leap 15.1 and we will provide instructions use the openSUSE version again.

Custom info key-value pairs defined in image profiles are now passed to the Docker build command as build arguments. They can be accessed in Dockerfiles using the ARG command.

After a Service Pack migration dry-run, if the result is a success you will get a "Run migration" button in the event history to retrieve the "dry-run" settings and confirm the migration with these settings.

Scheduling recurring actions allows you to manage schedules for automated recurring highstate execution on client, group, and organization level depending on the frequency you choose.

This is useful, for example, to apply highstates on a regular schedule and ensure configurations are enforced.

For more information, see the Administration Guide.

Before this release, only password authentication was available for bootstrapping Salt clients from the Server.

Now SSH private key authentication is available, including use of a passphrase on the private key. For Uyuni 2020.04 this is only available from the API. It will be made available from the WebUI in a future release.

For security reasons, the private key is stored at the Uyuni Server only for the bootstrap procedure, and removed after bootstrapping is complete. The private key must be provided for each bootstrap.

The new method bootstrapWithPrivateSshKey in the namespace system is documented in the API Documentation.

You can use this example by adjusting the client, keyfile, passphrase, MANAGER_URL, MANAGER_LOGIN and MANAGER_PASSWORD according to your environment:

The content lifecycle project page in the WebUI now has improved feedback messages about module filters, including missing or conflicting modules, and dependency resolution problems. The messages are in the form of errors that require the user to fix configurations, or warnings about potential problems.

Database schema upgrades are now applied automatically during services startup, so there is no need to call spacewalk-schema-upgrade manually. A security mechanism has been implemented that prevents Uyuni Services from starting if the schema upgrade has failed.

When this occurs:

Uyuni is designed by default to work on small and medium scale installations.

For installations with more than 1000 clients per Uyuni Server, adequate hardware sizing and parameter tuning must be performed, and the new guide provides information about how to do it.

Keep in mind there is no hard maximum number of supported systems. Many factors can affect how many clients can reliably be used in a particular installation. Factors can include which features are used, and how the hardware and systems are configured.

The Uyuni Hub announced for 2020.03 has now documentation available as part of the Large Deployments Guide (section Multiple Servers with Hub).

This is a draft release, so please provide feedback using the Resources menu in the online documentation

This new draft guide shows you the fastest way to get Uyuni up and running in a public cloud. It includes instructions for Amazon Web Services, Microsoft Azure, and Google Cloud Engine.

This is a draft release, so please provide feedback using the Resources menu in the online documentation

CaaSP specific Grafana dashboards have been integrated and can be deployed via the UI.

The new version of the Prometheus formula allows configuring federation and pulling relevant metrics from Prometheus instances to provide a global monitoring view.

Note that suitable recording rules have to be configured on the Prometheus instances (for example at CaaSP Prometheus instances).

For more information about Prometheus federation, check the official documentation.

A default set of alerting rules have been added to monitor the Prometheus instances themselves (meta-monitoring) and the availability of configured targets. The rules can be disabled in the WebUI.

We now provide these Prometheus exporters as packages for CentOS8 x86_64 (compatible also with similar systems such as RHEL8):

All the changes can be found at the changelog for the package, or at https://github.com/prometheus/node_exporter/releases/tag/v0.18.0 and https://github.com/prometheus/node_exporter/releases/tag/v0.18.1

Keep in mind this new version includes some breaking changes:

Until now users could list the storage pools, which is where the virtual machines disks are stored. Storage pools are where virtual machine disks are stored. In previous versions, you could only list the pools. With this update, you can create, edit, start, stop, refresh, and delete storage pools. This is available from the WebUI, or through Salt states.

We now offer Debian client tools that allow for easy onboarding of Debian as salt minions, as well as running spacecmd from them.

Check the Client Configuration Guide for information about how to configure Uyuni Server to work with Debian clients.

For now the following architectures are supported: x86_64, aarch64, armv7l, i586

We plan to continue improving Debian support in the future, including support for ppc64le and s390x Debian 10 clients.

Nodes in a SUSE Container as a Service Platforms should be patched, rebooted, etc following CaaSP recommendations to avoid breaking cluster availability and software compability.

In Uyuni 2020.03, we have introduced node locking and action filtering to prevent uninteded operations.

You can enable or disable the system lock using the System Lock formula. When the system lock is disabled, all operations are permitted.

In Uyuni 4.0.1, we introduced virtual host gatherers for Amazon Web Services, Microsoft Azure and Google Cloud Engine. With these gatherers, our subscription matcher gained the ability to also include virtual machines running on the cloud in its calculations.

We have now enhanced the subscription matcher to exclude pay-as-you-go (PAYG) instances. Those do not require a subscription, as the agreement between the Cloud Service Provider and the Customer covers them.

A bootstrap repository contains packages for installing Salt on clients, as well as the required packages for registering Salt or traditional clients during bootstrapping.

In Uyuni 2020.01 and earlier, bootstrap repository creation was a manual step, by using the mgr-create-bootstrap-repo tool.

In Uyuni 2020.03, bootstrap repositories are automatically created and regenerated on the Uyuni Server after a product is synchronized (i. e. all mandatory channels are fully mirrored).

More details, including how to revert to manual invokation, are available from the Client Configuration Guide.

Enable provisioning API with Salt and bootstrap entitled systems. Previously, this only worked for traditional clients.

You can schedule automated recurring highstate actions for Salt clients.

Recurring highstate actions apply the highstate to clients on a specified schedule. You can apply recurring action to individual clients, to all clients in a system group, or to an entire organization. The Recurring Actions section in the Administration Guide contains all the details for this feature.

More improvements in regards to automation will be coming in subsequent releases of Uyuni: maintenance windows and patch automation.

RHEL, SLES ES, CentOS and Oracle Linux 8 appstreams can now be mixed and converted to flat repositories using a new type of CLM filter.

Each Uyuni release and maintenance update brings better Ubuntu support. In Uyuni 2020.03, we have include two small but valuable improvements:

Yomi (yet one more installer) is a Salt-based installer for SUSE and openSUSE operating systems.

In Uyuni, Yomi can be used as part of provisioning new clients, as an alternative to AutoYaST. Yomi consists of two components:

Detailed information on how to use Yomi is available from the Salt Guide.

Yomi is work in progress and more operating systems and features will be added in coming releases.

The Uyuni Hub is a new multi-server architecture we are introducing as a technology preview in Uyuni 2020.03.

Multiple Uyuni Servers can be managed from a single Hub node. The Hub is a Salt master itself and the managed Uyuni Server servers are both a minion (to the hub) and a master (to their own minions).

The Hub covers a number of use cases, such as:

The Hub comprises a number of components that we will be releasing and enhancing in the future. The first component of the Hub we are now introducing as a Technology Preview is the Hub XML-RPC API, which provides an extended version of the Uyuni Server XML-RPC API, targeted for the multi-server case.

Installation and usage

Install Uyuni Server and then install the hub-xmlrpc-api package. That Uyuni Server is now the Hub Server.

Configuration of hub-xmlrpc-api is specified in a JSON file like the following:

Set the HUB_CONFIG_FILE environment variable to point to the configuration file. hub-xmlrpc-api is a daemon, currently to be launched from the command line.

Once running, you can connect to the hub-xmlrpc-api at port 8888 via any XMLRPC compliant client libraries (see examples below).

API endpoints, namespaces and examples

Details about usage with Python script examples are available at the Uyuni project site: https://github.com/uyuni-project/hub-xmlrpc-api

In Uyuni 2020.01 and earlier, the spacewalk-utils package contained a mix tested and untested tools.

In Uyuni 2020.03, we have split spacewalk-utils in two packages:

Tools in spacewalk-utils-extras are valuable but they are so specific, or require additional customization for each user, that it is not possible for us to test for every use case. If you were using these scripts in spacewalk-utils in Uyuni 2020.01 or earlier, you will need to install spacewalk-utils-extras in Uyuni 2020.03.

The dhcp formula, branch network formula and pxe formula have been updated to support booting EFI terminals (systems) via HTTP in addition to TFTP.

On public cloud providers, the subscription matcher will identify pay-as-you-go instances, whose subscription is provided by the Cloud Service Provider, and will not ask for additional subscriptions.

Also, stackable subscriptions with the same parameters will be aggregated.

Uyuni supports Single Sign-On authentication by implementing the Security Assertion Markup Language (SAML) 2 protocol. This feature, introduced in 4.0.2 as a Technology Preview, is now declared stable

Uyuni must be reconfigured to use the IdP as the source of authentication and post-login mapped users must be already created before enabling SSO.

For more on configuring SSO, see the Authentication Methods chapter in the Administration guide.

In an effort to provide our web UI users with a smoother navigation, we have implemented large parts of the user interface as a single page application.

This enhancement was started in Uyuni 2020.01 as an opt-in feature and now becomes the default in Uyuni 2020.03

It is no longer necessary to have Python 3 on RHEL8 systems for the onboarding to work. With this enhancement, even plain-text RHEL machines can be onboarded directly.

Uyuni is now changing from X.Y version format to YYYY.MM format, and the URLs for the repositories remove the X.Y part.

This will allow easier releases, no need to change URLs at all in the future, and less confussion regarding the relationship between Uyuni and SUSE Manager (Uyuni is always ahead).

Because of the version format change, you need to adapt your zypper repository at the server before updating.

If you followed the instructions for installation, this command will do it for you:

sed -i -e 's/Uyuni-Server-4.0-POOL-x86_64-Media1/Uyuni-Server-POOL-x86_64-Media1/' /etc/zypp/repos.d/uyuni-server-stable.repo

Otherwise, find the Uyuni Server Stable repository and replace:

baseurl=https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-4.0-POOL-x86_64-Media1/

with:

baseurl=https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/

If you are currently syncing Uyuni Proxy 4.0 (usually because you have proxies), you need to:

Most users will not require this unless, but if you have the Uyuni Server 4.0 channel at your server:

CentOS 8, Red Hat Enterprise Linux 8 and SUSE Linux Enterprise Server Expanded Support 8 are now supported clients as Salt minions. The traditional stack will not be supported on these operating systems.

With the new application streams concept introduced in these operating systems, you will need to import both the BaseOS and the AppStream directories from the ISO image for the bootstrap repository to be created correctly. If the AppStream directory is not imported, you will receive an error about missing Python 3 packages.

AppStream awareness in the UI and Content Lifecycle Management will be available in an upcoming version of Uyuni.

This version of Uyuni includes formulas to install Prometheus and Grafana, and makes the Apache exporter available for Ubuntu 18.04, CentOS6, CentOS7 and Proxy.

Additionally, self-monitoring capabilities have been implemented in the Admin Monitoring UI.

SUSE Package Hub is now supported on the Server, since the problems with the search that were caused by PackageHub-provided packages have been solved.

If you were using Package Hub as a source of packages for you clients, it is recommended that you re-generate all package metadata. The reason for this is in the Package Hub repositories there may exist multiple packages with the same NEVRA but different checksums. This might result in checksum errors when repositories are used on the clients as Uyuni randomly selected any of those packages. After this update, Uyuni will generate the checksum into the package path to ensure the right package is used. If you use also Uyuni Proxy please update all of them before you re-generate the metadata.

The Formulas with Forms screen has an enhanced layout that folds vertically instead of nesting deep inside, making if cleaner. Besides this, validators are now possible in formulas using the JEXL expression language.

The cpu-mitigations-formula is now installed by default.

The Retail branch network formula now works all SUSE and openSUSE based distros, using SuSEfirewall or firewalld as appropriate.

In Uyuni 4.0.2 we introduced Content Lifecycle Management with a filter to exclude packages and patches based on their name. Feedback for this feature was very positive and many proposals for enhancement were received.

In this release, we are introducing a lot of new possibilities for Content Lifecycle Management:

We have documented two typical use cases: a monthly patch cycle and live patching.

More enhancements to Content Lifecycle Management will come in future releases of Uyuni.

With each release of Uyuni, we continue to enhance our Debian and Ubuntu support.

Uyuni 2020.01 greatly improves our compatibility thanks to:

A new set of client tool packages now includes Prometheus exporters for more clients: CentOS 6, CentOS 7, RHEL 6, RHEL 7, SLES ES 6, SLES ES 7 and Ubuntu 18.04. Both the Prometheus node exporter and the PostgreSQL exporter are provided for those operating systems. The prometheus-exporters-formula formula makes easy to deploy them.

We’ve added new types of Virtual Host Managers in order to gather virtual instances from Public Cloud providers. Azure, AWS and Google Cloud are now supported, in addition to the existing VMware and generic (file-based, manually-maintained, useful for any cloud provider) gatherer modules.

Creating VHM to gather virtual instances from the Public Cloud will enable the subscription matcher to match "1-2 virtual machines" subscriptions for those instances that are running on the same Public Cloud zone.

Please take into account the following considerations in this version. They will be addressed in upcoming versions of Uyuni:

Some users have hit in the past a database corruption problem when PostgreSQL ran out of space.

In order to prevent that from happening in the future, we have added a diskchecker to Uyuni Server. This feature will send a warning mail when the most common and important Uyuni directories are below 10% of free disk space, and will shut down the Uyuni Server when those directories are below 5% of free disk space.

This new feature is only enabled by defult in new installations. For existing installations, the administrator can enable the tool manually after updating to the latest maintenance update by running:

systemctl --quiet enable spacewalk-diskcheck.timer

systemctl start spacewalk-diskcheck.timer

Full details on the parameterization of this new feature are available in the Managing disk space documentation page.

In an effort to provide our web UI users with a smoother navigation, we have implemented large parts of the user interface as a single page application.

This feature is optional for this release and is disabled by default. To enable it, users can now add web.spa.enable = true to /etc/rhn/rhn.conf, and then restart Tomcat.

Grafana is a tool for data visualization, monitoring, and analysis. It is used to create dashboards with panels representing specific metrics over a set period of time. Grafana is commonly used together with Prometheus, but also supports other data sources such as ElasticSearch, MySQL, PostgreSQL, and Influx DB.

This version of Uyuni includes Grafana in the client tools repositories. An Uyuni Grafana dashboard is provided as an example.

Monitoring section of the Administration Guide contains full detail on how to configure Grafana together with Uyuni.

Prometheus is a monitoring tool used to record real-time metrics in a time-series database. Metrics are collected using HTTP pulls, allowing for higher performance and scalability.

We have updated the Prometheus package with a new version that include a built-in service discovery mechanism that will allow users to more easily configure monitoring on their Uyuni systems.

Previously, after configuring the exporters on managed clients, users had to manually configure their Prometheus servers to start scrapping metrics from those systems. With this update, it will be possible to use a "service discovery" mechanism that will automate this part of the configuration. The configuration options are simple: it is only required to provide a Uyuni Server URL and valid API credentials.

Under the hood, what this mechanism does is letting Prometheus poll the Uyuni API, asking for a list of systems that have monitoring enabled, and automatically configuring Prometheus to collect metrics from those systems.

In this version, the autodiscovery functionality is provided as a Technology Preview.

More information about configuring Prometheus can be found in the Monitoring section of the Administration Guide.

CPU mitigations have been introduced to improve security on CPUs affected by vulnerabilities such as Meltdown and Spectre. The mitigations are available in SUSE Linux Enterprise 12 SP3 and later in the cpu-mitigations-formula package, which is not installed by default.

The new CPU Mitigation formula allows you to control which mitigations are enabled.

The Uyuni documentation has received improvements in all of the books, with small clarifications and enhancements all around: content lifecycle management filters, public cloud, JeOS, formulas, etc

Of particular interest for users with large installations will be the new Large Scale Deployment and Salt Tuning sections in the Salt Guide. Given that modifying advanced parameters can cause catastrophic failure, we recommend making a backup and being conservative doing changes.

Additionally, the search functionality in the documentation now works offline.

The Multi-Arch and Pre-depends headers are now supported for .deb repositories, hence avoiding installation problems that could arise in some cases when deploying packages from the UI.

Also, Ubuntu and Debian channels now come preconfigured in spacewalk-common-channels. The Debian CDN is used to provide the best mirror at each moment. For Ubuntu, you may want to replace the default mirror with a closer geo-mirror.

Keep in mind SUSE does not provide support for the spacewalk-common-channels tool form the spacewalk-utils package.

The Virtual Host Manager functionality has been extended to support SUSE Container as a Service Platform v4.

You can register each CaaSP node to Uyuni using the same method as you would a Salt client. After doing this, you will be able to see the patch level status of each node, perform configuration management on the nodes and assign channels o clusters.

We strongly recommend to check the documentation on the scope and extent of the CaaSPv4 integration in Uyuni: https://www.uyuni-project.org/uyuni-docs/suse-manager/client-configuration/vhm-caasp.html

Upcoming versions of Uyuni will enhance CaaSP integration.

4.0.2 is now based on openSUSE Leap 15.1, so a base OS system is required.

To help administrators with the migration, a new script is provided by the susemanager package at /usr/lib/susemanager/bin/server-migrator.sh

Then, update susemanager package only:

And finally run the script:

After the migration is complete, you will be requested to reboot your server

Uyuni Server 4.0.2 works with SUSE Uyuni Proxy 4.0.1.

When upgrading, upgrade the Server first, followed by the Proxies.

Salt has been upgraded to the 2019.2.0 release.

We intend to regularly upgrade Salt to more recent versions.

For more detail about changes in your manually-created Salt states, see the Salt upstream release notes 2019.2.0.

The base system was upgraded to openSUSE 15.1.

As a result, all code was ported to run with Python 3 and OpenJDK 11.

We now include packages for the latest version of Prometheus, as well as self-monitoring capabilities for Uyuni.

Prometheus is a monitoring tool that is used to record real-time metrics in a time-series database.

For more information about Prometheus, see the Administration Guide

The content lifecycle management feature allows you to clone software channels through a lifecycle of several environments. You are able to create content projects, select a custom set of software channels as sources, and promote software channels through a pre-defined lifecycle of environments.

You can define filters to exclude specific packages and patches. More filters will be added in a later release.

Once you have selected your sources you can build the selected set which will populate the first environment. After the first environment is built, you can promote it through the environment lifecycle to the next environment in the loop. You can see the status of the build at any time throughout the process.

The result of the build, and the content of every environment, is a channel tree made of cloned software channels of the selected sources, to which systems can be assigned.

The existing virtualization features have been enhanced for Salt-based systems. This is a technology preview and will require an additional Virtualization Management entitlement. Pricing will be announced soon.

Salt-based virtualization host systems can also create virtual machines using a pre-built disk image.

These features have been added:

In this release, we have reorganized our documentation and updated our tooling to make it clearer where information is, and make it easier for you to find the content you need, when you need it.

The Salt Remote Command Page log now every command executed in a separate logfile (/var/log/rhn/rhn_salt_remote_commands.log). In addition to this, an entry in the System History is generated for every minion where the command was executed.

openSUSE Leap 15.1 and SLE15 SP1 can now be managed.

openSUSE Leap 42.3 is now End of Life since July 1st, as announced at the openSUSE Mailing lists

While the repositories for Leap 42.3 are still available, no support is provided aymore.

Any action scheduled on multiple Salt minions has now an upper limit on the number of systems that will process it simultaneously. This is referred to as batch size in Salt jargon, and defaults to 100 minions.

Please check the documentation for performance considerations in large installations (more than 1000 minions).

In the past information about product channels were shipped via maintenance updates. Now these information will be downloaded from SUSE Customer Center (SCC) like the other product and repository information.

In case of using the fromdir configuration with SMT or RMT, please check if they support already downloading this file. You can get the file with the following command:

Using SLES 12 SP4 as the base OS for an image build host is now supported.

Also building SLES 12 SP4 OS Images is supported.

This update contains a new backend to communicate with the SUSE Customer Center (SCC). This requires to run a mgr-sync refresh at the end of the update procedure.

The whole update procedure:

In case of Inter Server Sync (ISS) the master needs to be updated first, then the slave.

This change show products like they are setup in the SUSE Customer Center. As a consequence of this some older products show no architecture anymore and mirror all available architectures when such a product is selected for mirroring.

With this change also some invalid product combinations were removed. Please check /var/log/rhn/rhn_web_ui.log for error messages. Invalid channels can be removed using spacewalk-remove-channel command.

Due to the changes in the backend for communicating with SCC corresponding XMLRPC API has changed:

Deprecated calls:

New call:

For a refresh the XMLRPC API should be called in the following order:

Management of Ubuntu clients is now supported. We provide a repository with salt packages that can easily be added with spacewalk-common-channels or manually.

The following new features were added:

However, the root user on Ubuntu is disabled by default, so in order to use bootstrapping, you will require an existing user with sudo privileges for Python.

Channel authentication tokens are valid by default for about 1 year. The renew of tokens happens automatically some time before they expire but they are not deployed automatically to the clients.

As the renew happens mostly without noticing by the administrator that behavior has changed to autodeploy renewed tokens to the clients automatically.

This old behavior can be preserved by setting

in /etc/rhn/rhn.conf and restarting the services.

In case of a token renew without autodeployment enabled a log message will inform the administrator about it.

A new option --force was added to spacecmd softwarechannel_regenerateyumcache to force a regeneration of the metadata files.

It is now possible to set a custom protocol, host and path for minions to download packages at installation time. This will override the default setting of the Uyuni Server or Uyuni Proxy used at registration time.

Uyuni supports Single Sign-On authentication by implementing the Security Assertion Markup Language (SAML) 2 protocol. Mandatory requirement: an already existing and configured SAML Identity Service Provider (IdP). Uyuni must be reconfigured to use the IdP as the source of authentication and post-login mapped users must be already created before enabling SSO.

For more on configuring SSO, see the Administration Guide

A new version of the PostgreSQL database is available in openSUSE Leap 42.3 and can be used for Uyuni Server.

New installations of Uyuni Server based on openSUSE Leap 42.3 will automatically pick up this version.

PostgreSQL 10 needs a new version of smdba to initiate backups. This version is part of Uyuni Server 4.0.1.

You should have an up-to-date database backup before attempting the migration.

Existing installations of Uyuni Server will need to run

to migrate from PostgreSQL 9.6 to PostgreSQL 10

Your Uyuni Server installation will not be accessible during the migration.

Note The migration will create a copy of the database under /var/lib/pgsql and thus needs sufficient disk space to hold two copies (9.6 and 10) of the database.

Since it does a full copy of the database, it also needs considerable time depending on the size of the database and the IO speed of the storage.

If your system is scarce on disk space you can do an fast, in-place migration by running

The fast migration usually only takes minutes and no additional disk space. However, in case of failure you need to restore the database from a backup.

This wiki page contains additional information about the database migration.

spacecmd, the command line access to the Uyuni API, has been adapted to support state channels (aka Salt Minion config channels) with the following changes:

Please use the help functionality of spacecmd for detailed option descriptions for each mentioned call.

Functions softwarechannel_mergepackages and softwarechannel_errata_merge to merge packages and errata through spacecmd were added.

Added:

Removed:

openSUSE Leap 15.0, openSUSE Leap 42.3, SLE12, SLE15, CentOS6 and CentOS7 are now verified to bootstrap as both salt minions and traditional clients.