When hosts are in 'updating' state, the kube rootca
update orchestration should consider those hosts to be
in-progress, rather than attempt the same command, which
immediately fails the strategy.
A second scenario is that if a host is rebooted while
initiating a kube rootca updating action, it will not complete
and will block orchestration. The only way to resume is
to abort that rootca update.
The compatability changes in this review are:
- The removal of the 'force' option for rootca 'patch' REST
API calls pertaining to the 'complete' and 'abort' operations.
The logic changes in this review are:
- If a host is already 'updated' when the step is invoked,
it will be considered successful.
- If a host is 'updating', orchestration will wait for it
to complete. If the host is stalled, the step will fail based
on the step timeout. If the host was updated longer than a
specific duration, it will immediately be considered timed out.
- If a host step fails, the rootca update is aborted,
not just the orchestration.
Known limitation:
VIM orchestration updates the hosts sorted by their name.
If a system has been manually updated, and then VIM orchestration
is used to resume it, it will fail if the hosts cannot be updated
in the expected order.
TEST PLAN:
PASS: Run a kube-rootca-update orchestration (2 controllers)
PASS: Abort a kube-rootca-update orchestration after first step and
verify update is not aborted.
PASS: Create a valid kube-rootca-update orchestration in a system
with an existing 'started' update.
PASS: Reboot second host before it is sent its 'updating' request.
The host state moves to 'updating' but never completes.
Verify orchestration fails (step times out after 10 minutes) and
the update is aborted.
PASS: Run kube-rootca-update orchestration over an aborted update.
This will create a new update which can be run.
PASS: Abort orchestration during the 'pods' step.
This completes the pods step, and aborts the orchestration but will
not abort the update.
PASS: Manually start an update and generate a cert.
Create an orchestration.
Manually update controller-1 and let it complete.
Start the orchestration.
Orchestration should succeed (skips cleanly over controller-1)
Story: 2009665
Task: 44058
Depends-On: https://review.opendev.org/c/starlingx/config/+/819020
Signed-off-by: albailey <Al.Bailey@windriver.com>
Change-Id: If5da6fb7648a6c07438b41449e7acc724f71e0f7
60738b4cdb