docs/doc/source/backup/kubernetes/backing-up-starlingx-system-data.rst

Back Up System Data Overview

A system data backup of the system captures core system information needed to restore a fully operational cluster. See Run Ansible Backup Playbook Locally on the Controller <running-ansible-backup-playbook-locally-on-the-controller> or Run Ansible Backup Playbook Remotely <running-ansible-backup-playbook-remotely> for information on how to backup up a standalone system or a System Controller.

In a environment, see Backup a Subcloud/Group of Subclouds using DCManager CLI <backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42> for how to remotely backup a subcloud from the System Controller.

Note

Backup archives should be stored in a secured (offsite) location.

Contents of System Backup

The following content is included in the backup:

• All platform configuration data required to fully restore the system to a working state following the platform restore procedure.
  • Platform and Kubernetes databases.
  • Platform configuration files.
  • Platform certificates and keys.
• Home directory for the sysadmin user and all user accounts.
• End-user container images in registry.local; that is, any images other than system and application images. system and application images are re-pulled from their original source, and (optional) external registries during the restore procedure.
• Distributed Cloud Vault (Central System Controller only).

The following content is excluded from the backup:

• Application data on Ceph clusters.
• Modifications manually made to the file systems, such as configuration changes on the /etc directory. After a restore operation has been completed, these modifications must be reapplied.
• Home directories and passwords of local user accounts. They must be backed up manually by the sysadmin.
• The /root directory. Use the sysadmin account instead when root access is needed.

Note

Ceph data may be retained when restoring to the same servers and cluster.

System Backup Size

Consider the following for backup size:

• The base size of a platform system backup sizes range from 10MB to 30MB, depending on the size of the system and deployment. systems are typically 20MB or less.
• Backup of user home directories can cause the backup archive to be very large and is limited to 2GB or less.
• Total backup size of the System Controller should be below 100MB when using centralized backup and restore of subclouds.
• Container images are large and will only be backed up locally to avoid large image archives being transferred for each subcloud. Container images that are not present on the system may be pulled as part of platform and application deployment, or restored separately to the local registry (registry.local).

System Backup Filesystem Usage

The following filesystems are used during the backup operations of the system for both local and centralized backup.

Staging Storage

The host filesystem used to stage temporary files during backup operations. The filesystem may also be used to temporarily store final backup images prior to transfer, if the filesystem is sufficiently sized to store the backup archives.

Host filesystem name: backup

System path: /opt/backups

Default size: 25GB

For more information on how to modify the host filesystem sizes see Resize Filesystems on a Host <resizing-filesystems-on-a-host>.

Local Storage

The host filesystem used to store backup files in a protected partition which does not get wiped during system reinstallation. The protected local backup partition is typically used by systems where there is no redundant filesystem storage and it is the default for local backups.

Note

The filesystem is shared with system release pre-staging and needs to be sized for both pre-staging installation media and backup archives.

System Path: /opt/platform-backup/backups

Default Size: 30GB

Centralized Storage

The Vault filesystem is used to store backup archives when using centralized backup and restore of subclouds. The filesystem size must be increased to accommodate subcloud backup archive storage. A separate backup archive is stored per subcloud and release, and therefore, must be sized to accommodate all backups.

System path: /opt/dc-vault/backups/<subcloud-name>/<release-version>

Default size: 15GB

Note

The filesystem is shared for subcloud deployment and management and must be sized to store subcloud deployment files (subcloud configuration, ISO images and subcloud staging files).

For more information on how to modify the controller filesystem sizes see Storage on Controller Hosts <controller-hosts-storage-on-controller-hosts>.

Distributed Cloud Centralized Backups

A subcloud's system data and optionally container images (from registry.local) can be backed up using DCManager CLI command line interface. The subcloud's system backup data can either be stored locally on the subcloud or on the System Controller. The subcloud's container image backup (from registry.local) can only be stored locally on the subcloud to avoid overloading the central storage and the network with large amount of data transfer and redundant storage of images in a central location.

For more information on the CLI operation of the centralized backup capability see Backup a Subcloud/Group of Subclouds using DCManager CLI <backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42>.

For more information on DCManager - Subcloud Backup API see Subcloud Backups.

Execution Time for System Backups

• The time to execute system backups is approximately 3-4 minutes for an idle system.
• Centralized backups may require additional time for network transfer for larger backups.
• Subcloud backups may be initiated and monitored from the DCManager CLI or API, including parallel backups.
• A minor alarm (210.001) "System Backup in progress" is raised while backing up an individual system.
• Systems with at least 4 platform cores will have much faster execution times.

Recommended Backup and Retention Policies

• Use of the centralized backup mechanism of the System Controllers to perform and store the backups of subclouds. Standalone systems and System Controllers backups can be performed locally or remotely, and the archive must be stored off the system.

• Backups are not allowed till the system is healthy (this excludes non-management affecting alarms). However, a new parameter -e ignore_health=true can be added in the ansible playbook to ignore system health and force the backup to proceed.

  Warning

  Using the -e ignore_health=true option should be avoided unless it is required. Restoring an unhealthy backup will result in system issues.

• All backups are done during off-peak hours (i.e. maintenance window).

  • Weekly backups should be performed under normal steady state conditions to ensure the system can be restored to a fully operational state.
  • Nightly backups are the exception and should only be performed in periods of significant reconfiguration to the system such as during large/mass rollout (addition of subclouds), upgrade cycle of multiple sites, or disaster recovery rehoming of subclouds.

• Backups should be performed prior to performing maintenance operations or applying configuration changes to the platform or hosted applications.

• The retention period of backups should be approximately one month.

  • Since Kubernetes is an intent-based system, the most recent backup is the most important.

Run Ansible Backup Playbook Locally on the Controller <running-ansible-backup-playbook-locally-on-the-controller>

Run Ansible Backup Playbook Remotely <running-ansible-backup-playbook-remotely>