diff --git a/doc/source/_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest b/doc/source/_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest new file mode 100644 index 000000000..c0239da0d --- /dev/null +++ b/doc/source/_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest @@ -0,0 +1,11 @@ +.. short-desc-begin +.. short-desc-end + +.. images-id-begin +.. images-id-end + +.. note-begin +.. note-end + +.. warning-begin +.. warning-end diff --git a/doc/source/dist_cloud/kubernetes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rst b/doc/source/dist_cloud/kubernetes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rst new file mode 100644 index 000000000..a3ad1d3ef --- /dev/null +++ b/doc/source/dist_cloud/kubernetes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rst @@ -0,0 +1,581 @@ +.. _deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b: + +======================================================== +Deploy, Restore and Manage Subclouds of Previous Release +======================================================== + +.. only:: starlingx + +Starting with |prod| |v_r8| and subsequent releases, you can now install, +restore, and manage subclouds of the previous software release. This capability +can be used to revert to the previous release when an unforeseen issue is +encountered during a subcloud upgrade or deployment. + +.. only:: partner + + .. include:: /_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest + :start-after: short-desc-begin + :end-before: short-desc-end + +.. note:: + + This capability is not available for |prod| |v_r7|. If the system + controller is on |prod| |v_r8|, subclouds can be deployed or restored to + either |prod| |v_r8| or |prod| |v_r6|. + +.. contents:: |minitoc| + :local: + :depth: 2 + +.. rubric:: |context| + +This section describes the steps for deploying, restoring, and managing +subclouds of the previous release. + +**Operations Supported for Subclouds of Previous Release** + +- New subcloud deployment + +- Subcloud re-install + +- Subcloud backup removal + +- Subcloud restore from backup + +- Subcloud prestage (single subcloud) + +- Orchestrated subcloud prestage (multiple subclouds) + +- Subcloud rehome + +- Deploy files upload/show + +.. note:: + + After deployment, subclouds running the previous release can be managed, + unmanaged, audited, updated, upgraded, or deleted in the same way as + subclouds running the current release. + +**Operations with Limited Support for Subclouds of Previous Release** + +- Subcloud Backup + + **Limitation:** The option to exclude the ``/opt/patching`` directory from + the backup, which helps to reduce the backup size, does not apply to + subclouds operating on |prod| |v_r6|. For additional details on the list of + configurable system backup parameters and to identify the specific + parameter(s) that do not apply to |prod| |v_r6| subclouds, see + :ref:`running-ansible-backup-playbook-locally-on-the-controller`. + +.. warning:: + + Ensure that the RECONCILED state of each host is **true** before creating a + subcloud backup. + + .. only:: starlingx + + You cannot create a backup for a subcloud in a failed state when + running an upgrade from |prod| |v_r6| to |prod| |v_r8|. + + .. only:: partner + + .. include:: /_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest + :start-after: warning-begin + :end-before: warning-end + + Use the following command to verify the RECONCILED state. + + .. code-block:: + + kubectl get hosts -n deployment + +- Subcloud Error Reporting + + **Limitation:** In the event of a failure during subcloud install, bootstrap, + deployment, backup, restore, or upgrade, you can use the + :command:`dcmanager subcloud errors` command to obtain detailed information + about the failure. + + The error reporting for backup and restore failures may vary depending on + the release version. For example, in the case of a backup failure during a + subcloud upgrade due to insufficient disk space, a |prod| |v_r8| subcloud + will generate a generic error report stating, ``Failed to generate upgrade + data. Please check sysinv.log on the subcloud for details``. The same + failure on |prod| |v_r8| subcloud would give a more detailed error report + such as, ``Not enough free space in /opt/platform-backup. It has + 2010876KiB. It needs at least 3558912KiB``. + +**Operations Not Supported for Subclouds of Previous Release** + +- Orchestrated subcloud kubernetes upgrade + +- Orchestrated subcloud firmware update + +- Orchestrated subcloud patching + +- Admin network configuration + + +.. rubric:: |prereq| + +To support subcloud install or subcloud restore to either the current (N) or +previous (N-1) release, ensure that the following prerequisites are met: + +- The N load (pre-patched ISO) containing this capability must be imported to + the system controller using the ``--active`` option. The ISO imported via + ``load-import --active`` should be at the same patch level as the system + controller. This requirement ensures that the subcloud boot image is + aligned with the patch level of the load to be installed on the subcloud. + + + .. note:: + + Each time a new patch is applied on the system controller, it is + necessary to re-import the new pre-patched ISO containing the same + patch(es). + + +- The N-1 load (pre-patched ISO) must be imported to the system controller + using the ``--inactive`` option. This load must be upgradable to the + current active release. + + .. warning:: + + A subcloud running the N-1 release cannot be patched using patch + orchestration. To deploy a subcloud with the latest patch of N-1 + release, the N-1 release ISO containing the latest patch must be + imported using the following command: + + .. code-block:: none + + ~(keystone_admin)]$ sw-patch –os-region-name SystemController load-import --inactive .iso .sig + +- The central registry (or source registry configured for the subclouds) must + contain platform and applications container images of both N and N-1 + releases. + +- Subcloud install values, bootstrap values, and deploy config files must be + compatible with the release the subcloud is being deployed to. + + .. warning:: + + Since orchestrated Kubernetes upgrades are not supported for subclouds + running the previous release, these subclouds must be deployed with the + highest Kubernetes version for that release. Do not set the + ``kubernetes_version`` value in subcloud ``bootstrap-values.yaml`` when + deploying a new N-1 subcloud. If this value is not the highest + Kubernetes version, the subcloud add request will be rejected. + + .. warning:: + + If a subcloud was initially deployed in one release and re-deployed to + a different release, ensure that ``persistent_size`` value in the + subcloud install values is equal to or greater than ``persistent_size`` + specified in the previous deployment. Failing to do so may result in a + subcloud installation failure, as the persistent + ``/opt/platform-backup`` partition cannot be reduced in size. + +- If the system controller was installed with default settings, it will use + the latest Kubernetes version for the release it's running. To support + upgrading the subcloud, ensure that the system controller has locally + stored copies (caches) of container images for all the previous versions of + Kubernetes relevant to the platform release. For example, if the system + controlle r is installed with Kubernetes 1.24, we need to ensure that all + the system images related to Kubernetes 1.21, 1.22, and 1.23 are stored + locally (cached) on the system controller for use when the subcloud tries + to upgrade to those versions. + + To obtain more information about the list of images, see the corresponding + pre-build ISO and docker images section for |prod| |v_r6| and |v_r8|. + +.. only:: starlingx + + - `The pre-built ISO (CentOS and Debian) and Docker images for StarlingX r6 `_. + + - `The pre-built ISO (Debian) for StarlingX r8 `_. + +.. only:: partner + + .. include:: /_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest + :start-after: images-id-begin + :end-before: images-id-end + +Import an Inactive Load +----------------------- + +An inactive load can be imported using the ``--inactive`` option with the +``load-import`` command to allow the import of the previous release load on the +system controller. + +.. note:: + + Ensure that the imported ISO is at the same patch level as the system + controller. To check the system controller patch level of N and/or N-1 + release, use the :command:`sw-patch query` command. + +.. rubric:: |proc| + +#. Transfer the pre-patched ISO of the previous release and its signature file + to the system controller. + +#. Run the following command. + + .. code-block:: none + + ~(keystone_admin)]$ system --os-region-name SystemController load-import --inactive --local .iso .sig + +.. only:: starlingx + + .. note:: + + ``load-import --inactive`` is not rejected when importing a + non-upgradable load. The inactive load must be a pre-patched ISO with + |prod| |v_r6| or above. + +.. only:: partner + + .. include:: /_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest + :start-after: note-begin + :end-before: note-end + +Upload Deploy/Prestage files +---------------------------- + +Use the :command:`dcmanager subcloud-deploy upload` command with the +``--release`` option to upload deploy or prestage artifacts for a specific +release version. + +.. rubric:: |proc| + +#. To upload deploy files use :command:`dcmanager subcloud-deploy upload` + command with ``--deploy-playbook``, ``--deploy-chart`` and + ``--deploy-overrides`` options. The following command uploads deploy files + for |prod| |v_r8|: + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-deploy upload + --deploy-playbook + --deploy-chart + --deploy-overrides + --release + + Where, is the |prod| |v_r8| version. + + .. note:: + + The deploy files for N-1 release are automatically uploaded when + running the :command:`system --os-region-name SystemController + load-import --inactive` command. It is not mandatory to upload deploy + files for the N release. If the deploy files are not uploaded, the + software will use the default deploy files included in the active load. + +#. To upload prestage images list use :command:`dcmanager subcloud-deploy + upload` command with ``--prestage-images`` option. The following command + uploads the images list for |prod| |v_r6|: + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-deploy upload --prestage-images --release + + Where, is the |prod| |v_r6| version. + + +#. View uploaded files for a particular release. + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-deploy show + --release + +.. note:: + + If the ``--release`` option is not specified, the deploy/images list files + are uploaded or shown for the current release. + + +Deploy a New Subcloud +--------------------- + +Use the :command:`dcmanager subcloud add` command with the ``--release`` option +to deploy a subcloud with the specified release version. + +To initiate the deployment of a subcloud running |prod| release |v_r6|, ensure +that the required |prod| |v_r6| deploy files are uploaded, and that the +specific subcloud configurations for |prod| |v_r6| are properly prepared and +run the following command: + +.. note:: + + If controller-1 of the system controller is the active controller, make + sure that `/var/www/pages/feed/rel-` exists before deploying a + |prod| |v_r6| subcloud. Use the :command:`scp` command to copy the entire + directory from controller-0 to controller-1. + +.. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud add --bootstrap-address + --bootstrap-address + --bootstrap-values /home/sysadmin/subcloud1-bootstrap-values.yaml + --install-values /home/sysadmin/subcloud1-install-values.yaml + --bmc-password + --deploy-config /home/sysadmin/subcloud1-deployment-config.yaml + --sysadmin-password + --release + +Where, is the |prod| |v_r6| version. + +.. note:: + + If the ``--release`` option is not specified, the subcloud will be deployed + to the current release of the system controller. + +.. warning:: + + If the value of `software_version` specified in the subcloud + `install-values.yaml` does not align with the ``--release`` value in the + :command:`subcloud add` command, the operation will fail. It is recommended + to exclude `software_version` from the subcloud's ``install-values.yaml`` + configuration to prevent potential conflicts. + +Restore a Subcloud +------------------ + +.. note:: + + If a subcloud needs to be reverted to the previous release after an upgrade + failure, it is recommended to take a backup of the subcloud before the + upgrade using the :command:`dcmanager subcloud-backup create` command. For + additional information, see + :ref:`backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42`. + +To restore a subcloud from a backup of the previous release, ensure that the +pre-patched ISO has been imported into the DC vault and use the +:command:`dcmanager subcloud-backup restore` command with the ``--release`` +option. + +.. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud --with-install --sysadmin-password --release + +Where, is the |prod| |v_r6| version. + +For additional information about the restore, refer to "Restore a single subcloud" +section in +:ref:`restore-a-subcloud-group-of-subclouds-from-backup-data-using-dcmanager-cli-f10c1b63a95e`. + +.. note:: + + If the ``--release`` option is not specified, the subcloud will be + installed to the current release and restored from a backup of the current + release. + +.. warning:: + + The restore operation will fail if the backup is either non-existent or + incompatible with the release version to which it is being restored. + + +Delete a Subcloud Backup +------------------------ + +To delete a subcloud backup of a particular release, use the +:command:`dcmanager subcloud-backup delete` command with the release number +specified. For more information, see +:ref:`delete-subcloud-backup-data-using-dcmanager-cli-9cabe48bc4fd`. + + +- To delete |prod| |v_r8| backup data from central storage, use the following + command. + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-backup delete --subcloud + + Where, is the |prod| |v_r8| version. + + +- To delete |prod| |v_r6| backup data from local storage, use the following + command. + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-backup delete --local-only --subcloud --sysadmin-password + + Where, is the |prod| |v_r6| version. + + +Prestage a Subcloud +------------------- + +Use the :command:`dcmanager subcloud prestage` command with the ``--release`` +option to prestage a subcloud for the specified software release. A subcloud +can be prestaged with data from either the current or the previous release, +irrespective of its current software release version. + +The format of prestaged container images depends on the software release. For +|prod| |v_r6| release (CentOS) in the production environment, images are +created in the docker image archive format. However, starting from |prod| +|v_r8| and beyond (Debian), images are uploaded to the local registry, +and a snapshot of the local registry is captured. This format significantly +enhances the speed and efficiency of restoration during subcloud upgrades, +re-deployment, or restore processes. + + +Prestage a Subcloud with Current Release Data +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before prestaging the subcloud(s) with N release data, be sure to complete one +or both of the following steps: + +#. Import the pre-patched ISO for the N release as an active load using the + following command: + + .. code-block:: none + + ~(keystone_admin)]$ system –os-region-name SystemController load-import --active --local + + +#. |optional| Upload the N release images list by using the following command: + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-deploy upload --prestage-images + + +.. table:: Table 1. Container Images Prestage Behavior for |prod| |v_r8| and higher + :widths: auto + + +----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------+ + | Subcloud Software Version | N Release Images List Available | N Release Images List Not Available | + +==============================================+=================================================================================================================================================================================================================================================================================================+========================================================================================+ + | |prod| |v_r7| | The container images specified in the list are downloaded from the configured source(s) and pushed to the subcloud local registry. A snapshot of the local registry is then taken and saved in the prestage directory. Finally, downloaded images are removed from the subcloud local registry. | Images prestage is skipped. | + +----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------+ + | |prod| |v_r8| | The container images specified in the list are downloaded from the configured source(s) and pushed to the subcloud local registry. A snapshot of the local registry is then taken and saved in the prestage directory. | A snapshot of the subcloud local registry is taken and saved in the prestage directory.| + +----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------+ + +.. note:: + + It is recommended to repeat subcloud prestaging for the current release + after a new patch of the current release has been applied on the system + controller. This requirement ensures that the subcloud boot image is + aligned with the patch level of the load to be installed on the subcloud + using prestaged data. + + +Prestage a Subcloud with N-1 Release Data +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before prestaging the subcloud(s) with N-1 release data, be sure to complete +one or both of the following steps: + +#. Import the pre-patched ISO for the N-1 release as an inactive load using + the following command: + + .. code-block:: none + + ~(keystone_admin)]$ system --os-region-name SystemController load-import --inactive --local + +#. |optional| Upload the N-1 release images list by using the following + command: + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud-deploy upload --prestage-images --release + +.. table:: Table 2. Container Images Prestage Behavior for |prod| |v_r6| + :widths: auto + + +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ + | Subcloud Software Version | N-1 Release Images List Available | N-1 Release Images List Not Available | + +==============================================+==================================================================================================================================================================================+=============================================================================================+ + | |prod| |v_r6| | The container images specified in the list are downloaded from the configured source(s). Subsequently, Docker image bundles are generated and saved in the prestage directory. | Image bundles are generated from local registry content and saved in the Prestage directory.| + +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ + | |prod| |v_r8| | The container images specified in the list are downloaded from the configured source(s). Subsequently, Docker image bundles are generated and saved in the prestage directory. | Images prestage is skipped. | + +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ + +- To prestage a single subcloud, use :command:`dcmanager subcloud prestage` + command. The following command prestages a single subcloud with |prod| + |v_r6| data: + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud prestage --release --sysadmin-password + + Where, is the |prod| |v_r6| version. + + +- To prestage a group of subclouds use :command:`dcmanager prestage-strategy` + related commands. The following commands prestages a group of subclouds + with |prod| |v_r8| data: + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager prestage-strategy create --release --sysadmin-password --group + + ~(keystone_admin)]$ dcmanager prestage-strategy apply + + Where, is the |prod| |v_r8| version. + +Rehome a Subcloud +----------------- + +N-1 subclouds can be migrated to another system controller running N software +release. Use the :command:`dcmanager subcloud add --migrate` command with the +``--release`` option to rehome a subcloud. It is not possible to migrate a +subcloud running N software release to a system controller running N-1 software +release. + +Rehome |prod| |v_r6| Subcloud to |prod| |v_r8| System Controller +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To migrate |prod| |v_r6| subcloud from system controller A to system controller +B running |prod| |v_r8|, ensure that the |prod| |v_r6| pre-patched ISO has been +imported to System Controller B as an inactive load, and follow the |prod| +|v_r6| subcloud rehome procedure as follows: + +#. Unmanage the subcloud from the previous system controller. + +#. Update the admin password on the subcloud to match the new system + controller, if required. + +#. Run the :command:`dcmanager subcloud add` command on the new System + Controller with the ``--migrate`` and ``--release`` options to initiate the + migration: + + .. code-block:: none + + ~(keystone_admin)]$ dcmanager subcloud add --migrate + --bootstrap-address + --bootstrap-values /home/sysadmin/subcloud1-bootstrap-values.yaml + --release + + Where, is the |prod| |v_r6| version. + +#. On the subcloud, lock/unlock the subcloud controller(s) to enable the new + configuration. + +#. Use the :command:`dcmanager subcloud list` command to check the subcloud's + status. Ensure that the subcloud is online and fully operational before + proceeding to manage the subcloud. + +#. On the new system controller, set the subcloud to ``managed`` and wait for + it to sync. + +#. Delete the subcloud from the previous system controller. + +For more detailed information, refer to the `Rehome-a-Subcloud +`_ +topic in the |prod| |v_r6| documentation. + +Rehome |prod| |v_r8| Subcloud to |prod| |v_r8| System Controller +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For more detailed information, see :ref:`rehoming-a-subcloud`. + +.. note:: + + You can add the ``--release `` option to the :command:`dcmanager + subcloud add --migrate` command but it is not mandatory. + + Where, is the |prod| |v_r8| version. diff --git a/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst b/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst index 8cb05739d..a6fce71a6 100644 --- a/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst +++ b/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst @@ -86,6 +86,16 @@ Manage Subcloud Groups delete-subcloud-groups-22a7c65e66d7 ochestration-strategy-using-subcloud-groups + +-------------------------------------------------------- +Deploy, Restore and Manage Subclouds of Previous Release +-------------------------------------------------------- + +.. toctree:: + :maxdepth: 1 + + deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b + ------------------------- Update (Patch) management ------------------------- diff --git a/doc/source/dist_cloud/kubernetes/restore-a-subcloud-group-of-subclouds-from-backup-data-using-dcmanager-cli-f10c1b63a95e.rst b/doc/source/dist_cloud/kubernetes/restore-a-subcloud-group-of-subclouds-from-backup-data-using-dcmanager-cli-f10c1b63a95e.rst index 3790e2985..eccdcac09 100644 --- a/doc/source/dist_cloud/kubernetes/restore-a-subcloud-group-of-subclouds-from-backup-data-using-dcmanager-cli-f10c1b63a95e.rst +++ b/doc/source/dist_cloud/kubernetes/restore-a-subcloud-group-of-subclouds-from-backup-data-using-dcmanager-cli-f10c1b63a95e.rst @@ -92,6 +92,7 @@ To restore images from a custom backup file on the subcloud using ``--restore-values `` option, the registry_backup_filename parameter must be set in ``restore_values`` yaml file. +.. _restoresubcloud: Restore a single subcloud -------------------------