This commit includes new unit tests for subcloud_manager.py,
covering new test cases in deploy, add, delete, update, compose,
backup and restore, redeploy, backup, prestage and migrate
operations.
Test plan:
1) PASS: Run tox py39, pylint and pep8 envs and
verify that they are all passing.
2) PASS: Check 'tox -e cover' command output.
Coverage increased from 61% to 70%
Depends-On: https://review.opendev.org/c/starlingx/distcloud/+/913989
Story: 2007082
Task: 49618
Change-Id: I05604a4940eac62b311dd8476498965a0f021be0
Signed-off-by: Swapna Gorre <swapna.gorre@windriver.com>
Improves unit test coverage for dcmanager's sw_update_strategy
API from 76% to 98%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49652
Change-Id: I1d403abcaf503cccfed2b8242703f29fbb26844f
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's subcloud_group
API from 77% to 98%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49641
Change-Id: I4f8a87379c770409590ed84b0d2cd50c06110199
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's phased_subcloud_deploy
API from 81% to 99%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49319
Change-Id: Ie649e35dd17796e1cd4e7e7b452b32c7c2b5cd9f
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's subcloud_backup
API from 81% to 98%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49581
Change-Id: I7cb6b661dd75a9d92f31113ff7ba5a887e489abc
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's subcloud_deploy
API from 85% to 99%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49578
Change-Id: Ie8aa1feb9f5f1f7e9d73b76112d2f2a6db528e6a
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's subcloud_peer_group API
from 50% to 99%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49320
Change-Id: If327a7bbef984fa37d26e27e1d4994a09a97dce8
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's peer_group_association API
from 65% to 99%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49318
Change-Id: I547a3bebeb3b88dab11800c51ee382b210f38830
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Classes are created for logical structuring and
referenced with its respective object.
Test plan:
1) PASS: Run tox py39, pylint and pep8 envs and
verify that they are all passing.
2) PASS: Check 'tox -e cover' command output.
Depends-On: https://review.opendev.org/c/starlingx/distcloud/+/913988
Story: 2007082
Task: 49618
Change-Id: I60fde3af0c34bff0334fc6578a7d7652d36ecb70
Signed-off-by: Swapna Gorre <swapna.gorre@windriver.com>
Leverage mock methods from base.py
Test plan:
1) PASS: Run tox py39, pylint and pep8 envs and
verify that they are all passing.
2) PASS: Check 'tox -e cover' command output.
Story: 2007082
Task: 49618
Change-Id: If86b586933930749c7a0a7776fa397b6a7587ce0
Signed-off-by: Swapna Gorre <swapna.gorre@windriver.com>
Improves unit test coverage for dcmanager's orchestrator/states/software
functionality from 77% to 100%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2010676
Task: 49313
Change-Id: Id3de824406df4d4c6c6504aa07e95dc457e7c2df
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's orchestrator/states/software/cache
functionality from 63% to 99%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2010676
Task: 49314
Change-Id: I7cb03f657489e8b1523472f4c63de173c936de95
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's orchestrator/states/prestage
functionality from 78% to 99%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49310
Change-Id: If041d9aede8f93b99b8cc1001f99088c1a1c77be
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves unit test coverage for dcmanager's orchestrator/states/firmware
functionality from 70% to 100%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49312
Change-Id: I00895fb7616d0c3eeb54f623a8a1ccd223169bd9
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves UT coverage for dcmanager's manager/service.py
from 72% to 92%
Test plan:
1) PASS: Run tox py39, pylint and pep8 envs and
verify that they are all passing.
2) PASS: 'tox -e cover' command output is 92%
Story: 2007082
Task: 49605
Change-Id: I72a361bf89d3df32f71e78e16e107b183974c53e
Signed-off-by: Swapna Gorre <swapna.gorre@windriver.com>
If the subcloud rehome_data contains an incorrect bootstrap-address in
site A and the user migrates the corresponding peer group to site B,
the migration would fail. Subsequently, it will have the 'rehome-failed'
deploy-status in site B and 'rehome-pending' deploy-status in site A.
Then the user won't be able to update the bootstrap-address in either
site due to the following restrictions:
a) Primary site (site A) is not the current leader of the peer group;
b) Update in non-primary site (site B) is not allowed.
To fix this issue, the following changes are made:
1. In the non-primary site, if the subcloud deploy-status is
rehome-failed and the primary site is unavailable, updating
the bootstrap-values and bootstrap-address will be allowed, and the PGA
will be marked as out-of-sync.
2. Modify audit to automatically sync the rehome_data from non-primary
site to primary site if subcloud in the non-primary site is managed and
online and the PGA is out-of-sync.
Additional fix for the system_leader_id issue: When migrating SPG from
one site to another, if all of the subclouds rehome fail, the leader id
of the SPG in the target site has already been updated to the target
site's UUID. However, in the source site, the leader id is not updated
to the target UUID. The fix ensures that regardless of the migration's
success, only if the migration completes, the leader id in both sites
should be updated to the target UUID.
Test plan:
Pre-Steps: 1. Create the system peer from Site A to Site B
2. Create System peer from Site B to Site A
3. Create the subcloud peer group in the Site A
4. Add a subcloud with an incorrect bootstrap-address
to the peer group
5. Create peer group association to associate system peer
and subcloud peer group - Site A
6. Check current sync status in sites A and B. Verify
they are 'in-sync'.
7. Run migration for the subcloud peer group from Site B.
8. Verify 'rehome-failed' deploy-status in both sites.
PASS: Verify that the bootstrap-address can be updated in site B when
site A is down, and the PGA sync status is set to out-of-sync
in site B. Also, verify that the audit will sync the rehome_data
to site A and change back the PGA to in-sync once the reattempt of
migration is successful and site A is up.
PASS: Verify that the bootstrap-values and bootstrap-address are
the only fields that can be updated in site B when site A is down.
PASS: Verify that the update of bootstrap-address was rejected in site B
when site A is up.
PASS: Verify that even if all of the subclouds in an SPG experience
rehome failures, the system_leader_id in both sites is updated to
the target's UUID.
PASS: Verify that when site A is always online or recovered during
the migration to site B, the subcloud deploy_status in both sites
is "rehome-failed" after the migration completes. In this
scenario, site A can migrate the subcloud back, even though it's
still failed. However, after correcting the bootstrap-address in
site A, the reattempt of migration in site A succeeds.
Closes-Bug: 2057981
Change-Id: I999dbf035e29950fd823e9cdb087160ce40fd4ca
Signed-off-by: lzhu1 <li.zhu@windriver.com>
This commit changes the bootstrap address parameter from
"bootstrap-address" to "bootstrap_address" during the subcloud update
call made during the PGA sync operation. This fixes the issue where
the bootstrap_address was not being updated on the peer site, as the
subcloud update API expects the "bootstrap_address" parameter, with
an underscore.
Test Plan:
1. PASS - Create a peer group association and let it do the initial
sync. Modify the bootstrap-address of the subcloud using
the subcloud update command and then run the PGA sync
command. Verify that the rehome_data of the secondary
subcloud was updated with the new address.
Closes-Bug: 2057973
Change-Id: Ib5786a56c90f771b940e740bc095ebc8168d2830
Signed-off-by: Gustavo Herzmann <gustavo.herzmann@windriver.com>
The current message prints the error_msg.keys() directly, which returns
a dict_keys object. This commit fix the SPA sync_message by casting it
to a list.
This commit also fixes a log message where the subcloud name and peer
name order was inverted.
Test Plan:
1. PASS - Try to sync a PGA that has the rehome-failed state in the
non-primary site, causing the sync to fail. Verify that the
sync_message prints the subcloud list without including the
'dict_keys' string;
2. PASS - Introduce an error during the _delete_subcloud() function
and then try to delete the peer group association. Verify
that the sync_message prints the subcloud list without
including the 'dict_keys' string;
3. PASS - During SPG migration, verify that the modified log message
prints the subcloud name and peer site name in the correct
order.
Closes-Bug: 2057934
Change-Id: Idfdc2cc1731a51c6098a06863b2469c3085aa813
Signed-off-by: Gustavo Herzmann <gustavo.herzmann@windriver.com>
This commit adds a new rehome_data semantic check when attempting to
update which peer group a subcloud is part of. If rehome_data
is not already present, the request payload must contain both the
bootstrap-values and bootstrap-address; otherwise, the request will
be aborted.
Additionally, this commit updates the rehome_data during the subcloud
rename operation, guarenteeing that name is up-to-date.
Test Plan:
1. PASS - Attempt to add a subcloud with no rehome_data to a peer
group under the following conditions and verify that it
fails:
- Without passing bootstrap-address and bootstrap-values
- Passing only the bootstrap-address
- Passing only the bootstrap-values
2. PASS - Add a subcloud with rehome_data to a peer group and verify
that the operation succeeds regardless of the presence of
bootstrap-address and bootstrap-values.
3. PASS - Rename a subcloud with rehome_data and verify that the
rehome_data name field is updated to the new name.
4. PASS - Rename a subcloud without rehome_data and verify that the
rename operation still works.
5. PASS - Migrate a renamed subcloud back and forth and verify that
the migration completes successfully.
Closes-Bug: 2055883
Closes-Bug: 2056796
Change-Id: I4403dc50062db07a0de24e04139e3af8087c546f
Signed-off-by: Gustavo Herzmann <gustavo.herzmann@windriver.com>
Log subcloud health output for quick diagnosis. With this change,
the user would not need to log into the subcloud to check
health output. Sometimes, the health condition is resolved by
the time the user runs the system health-query command in the
subcloud. As the result, more time would be required to determine
what caused the failed health check in the first place.
Test Plan:
- Verify successful subcloud backup with the change.
- Induce a health condition (e.g. management affecting
alarm). Verify that subcloud backup request is rejected
and the subcloud health output is captured in
dcmanager-api.log
Closes-Bug: 2056721
Change-Id: I32fea354f9cf594ea45d412359a9090e7b1bfb83
Signed-off-by: Tee Ngo <tee.ngo@windriver.com>
Fix the reported issue:
The PGA fails to transition from 'in-sync' to 'out-of-sync' after
updating subclouds.
This commit includes the following changes:
1. Updates the SPG sync_status to 'out-of-sync' after performing any of
the following operations and provides an informative message to
the operator:
a) Adding a subcloud to the SPG
b) Removing a subcloud from the SPG
c) Updating a subcloud in the SPG
2. Ensures that updates on SPG attributes, such as name,
max_subcloud_rehoming, and group_state, are automatically
propagated to the peer site.
Test plan:
Pre-Steps: 1. Create the system peer from Site A to Site B
2. Create System peer from Site B to Site A
3. Create the subcloud peer group in the Site A
4. Add subcloud(s) to the peer group
5. Create peer group association to associate system peer
and subcloud peer group - Site A
6. Check current sync status on Sites A and B. Verify
they are 'in-sync'.
PASS: Verify 'out-of-sync' on both sites after running any of
the following operations on site A, the primary and leader site:
1. Adding subcloud to the SPG.
2. Removing subcloud from the SPG.
3. Updating any field of a subcloud in the SPG, such as bootstrap
address, bootstrap values, install values, etc.
PASS: Repeat the above operations while site B is down and verify that
PGA sync_status is set to "failed".
PASS: Verify that SPG attribute updates are accepted if peer site is up
and the updates are successfully synced to the peer site.
PASS: Verify that SPG attribute updates are rejected if the peer site
is down.
PASS: Verify that if the subcloud does not belong to any peer group, or
if it is part of a peer group but the peer group is not associated
with any peer yet, updating the subcloud would not result in an
"informative" message and no attempt to update the PGA sync_status
should occur.
Closes-Bug: 2054123
Closes-Bug: 2054124
Change-Id: I9f0e44e34c7db5d60d211b70e839606d0361cf83
Signed-off-by: lzhu1 <li.zhu@windriver.com>
Improves unit test coverage for dcmanager's alarm_manager and notifications APIs
from 73% and 85%, respectively, to 100%.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49317
Change-Id: I8fd1a60ded61ca5dc8edcd8f4297fca974ba1e24
Signed-off-by: rlima <Raphael.Lima@windriver.com>
This commit addresses the issue where the primary site's PGA status
remains 'in-sync' even after the secondary site becomes unreachable.
With this fix, the PGA status will be updated to 'unknown' upon the
secondary site's failure. Additionally, the status will transition to
'in-sync' once the secondary site is operational again.
If there are any changes in the association while the secondary site is
down, the PGA status will be set to failed. The sync status will
transition to "out-of-sync" upon secondary site recovery.
In this commit, the audit thread in the primary site will also update
PGA sync_status. If the primary site is down and the SPG is migrated to
secondary site, upon primary site recovery, its audit thread will update
the PGA sync_status on both sites accordingly.
Finally, the commit prevents the peergroup to from being updated in the
secondary site.
Test Case:
1. PASS - Shutdown of site2 (secondary site) results in the
synchronization status of the peer group association
transitioning from 'in-sync' to 'unknown'.
2. PASS - Restoration of site2 (secondary site) leads to the
synchronization status of the peer group association on
the primary site changing to 'in-sync', and the peer
group association status on site2 also reflects 'in-sync'.
3. PASS - While secondary is is offline, execute some operations which
result in PGA sync_status being set to "failed". Recover
secondary site and verify that the PGA sync_status is set to
out-of-sync on both sites.
4. PASS - Verify that updating peer group on secondary site is
disallowed.
5. PASS - Shut down the primary site, migrate the SPG to secondary site.
Restore the primary site while migration is in progress. Verify
that the PGA sync_status is out-of-sync. Verify that PGA
sync_status is set to in-sync shortly after the migration is
complete.
Closes-Bug: 2055030
Change-Id: I67f4200118621205c539b24eb764e3cc5acf12c0
Signed-off-by: Zhang Rong(Jon) <rong.zhang@windriver.com>
Since USM is still a work-in-progress feature, errors could occur,
causing the software audit to be out-of-sync, which also results in
the subcloud being out of sync.
This commit changes the behavior of the software audit to always report as in-sync, even if issues are found with the software service and the USM endpoints.
Known issues:
- Story = 2010676
- Task = 49666: Add support for https in USM software client
- Task = 49668: Copy GA metadata file to USM location in the Subcloud
Test Plan:
PASS - Manage a subcloud and check if the software audit is
in-sync even if the software service is disabled.
- Other audits in-sync.
- Subcloud in-sync.
Story: 2010676
Task: 49598
Change-Id: I596219f1fa4c42e6be7984f06618349c54f9448c
Signed-off-by: Hugo Brito <hugo.brito@windriver.com>
This commit will automatically delete the subcloud that was removed
from SPG on the peer site after the peer site migrated the SGP.
Conduct syntax checks for managing, unmanaging, deleting, and updating
subclouds, add or remove subcloud from SPG.
Specifically, If the secondary site leads the current SPG, the
subcloud can be removed from the SPG only when the primary site is
unavailable.
Test Plan:
PASS - Shutdown site1, migrate SPG to site2, and verify the status of
subclouds upon site1 recovery. As anticipated, the subclouds
transitioned to the secondary status.
PASS - Shutdown site1, migrate SPG to site2, then remove a subcloud
from the SPG on site2. Upon site1 recovery, verify the subclouds'
status. As expected, the subcloud was deleted in site1
corresponding to the removal of the subcloud from the SPG in
site2.
PASS - Shutdown site1, migrate SPG to site2, then remove a subcloud from
the SPG on site2 and subsequently delete it. Check the status of
subclouds upon site1 recovery. As expected, the subcloud was
deleted in site1 corresponding to the removal of the subcloud
from the SPG in site2.
PASS - Removal subcloud from the SPG in the primary site when it is the
current leader was successful as expected.
PASS - Keep site1 online, migrate SPG to site2, remove a subcloud from
SPG in site2 should not be allowed if site1 is online.
PASS - Add a subcloud that is offline/unmanaged to an SPG failed in
the primary site as expected.
PASS - Manage/unmanage subcloud failed when it associated to SPG.
PASS - Update subcloud successful when it associated to SPG in the
primary site and the SPG is the current leader.
PASS - Update subcloud failed when it associated to SPG in the secondary
site as expected.
Closes-bug: 2052415
Closes-bug: 2052584
Signed-off-by: Zhang Rong(Jon) <rong.zhang@windriver.com>
Change-Id: I210e2865228d166d7f5a5b26015ab07b4d09db47
Today, the software audit is being conducted alongside the patch audit (load/patching). However, because the software audit uses a different RestAPI, any failures in it also result in patch audit failures. This commit creates a new software audit to separate it from the patch audit.
Note that the patch audit will be removed later on.
TEST PLAN:
PASS: Manage a subcloud and get all audits in-sync.
- Software, load, and patching audits working properly.
PASS: Test with Software RestAPI disabled/broken.
- Load and patching audits working properly
- Software audit unknown status.
Story: 2010676
Task: 49598
Change-Id: I38c00bfdf4d86d56e1e656892f7de32206755865
Signed-off-by: Hugo Brito <hugo.brito@windriver.com>
Update dcmanager's mixins to use the base structure for API tests
and validate errors through pecan's abort method.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49374
Change-Id: I28fb72a6767d80cdc04092d69afb572708039d8a
Signed-off-by: rlima <Raphael.Lima@windriver.com>
In this commit, the dcmanager subcloud show command was updated
to display region_name as default.
Test Plan:
PASS: run dcmanager subcloud show <subcloud name> and verify that
region name is displayed
PASS: run dcmanager subcloud show <subcloud name> -d and verify that
region name is displayed
Closes-bug: 2054803
Change-Id: I4af4be72e2d09ad5ef0c36cd36ecc4294caf81b1
Signed-off-by: Christopher de Oliveira Souza <Christopher.DeOliveiraSouza@windriver.com>
Restrict subcloud's secondary sites (the peer site where
subcloud in SPG can be migrated to) from reconfiguring
the subcloud. This is necessary because subcloud reconfig
may update the OpenStack endpoints/routes/network IPs and impact
the ability to migrate the subcloud back to the primary site.
- Introduced a check in "deploy configure" to ensure that
the target subcloud is managed by its primary site. The site
status is inferred from the subcloud's peer group priority on the site.
Test Plan:
Setup a DC system with GR configuration (2 sites + subclouds) and
verify the "subcloud deploy config" command with the following arrangements:
1. PASS: For a subcloud that's not part of protection group
(not in any SPG / not part of GR),
ensure that the operation is not restricted.
2. PASS: Add subcloud to SPG and keep it managed by primary site,
ensure that the operation is not restricted.
3. PASS: Add system peer, create SPG association with a secondary
site, and migrate the subcloud from #2. Verify that "deploy
config" is restricted and it fails with following error
message: "Subcloud must be managed by its primary site"
4. PASS: Using the subcloud from #3, migrate it back to the primary
site. Attempt to deploy config and ensure that the operation
is not restricted.
Closes-Bug: 2054161
Change-Id: I7e986771a32882bfad917c85439e6f6b99dfe173
Signed-off-by: Salman Rana <salman.rana@windriver.com>
Improves unit test coverage for dcmanager's orchestrator/states/kube functionality.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49309
Change-Id: I14a884e884b78aadb21d7276b49aada70aceb974
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Improves the unit test coverage for dcmanager's orchestrator/states/kube_rootca
functionality.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49306
Change-Id: If43ff65e0b225c044f8f15ef7e97ecb8397e75a0
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Update the base structure to be used in tests for controllers
and updates the DCManagerTestCase to include common mocks
between test classes.
Test plan:
All of the tests were created taking into account the
output of 'tox -c tox.ini -e cover' command
Story: 2007082
Task: 49366
Change-Id: I0f2ea4e7eb300940e365e59221bfdb0c27ddb197
Signed-off-by: rlima <Raphael.Lima@windriver.com>
Swapna Gorre