48978d804d
Performing a forced reboot of the active controller sometimes results in a second reboot of that controller. The cause of the second reboot was due to its reported uptime in the first mtcAlive message, following the reboot, as greater than 10 minutes. Maintenance has a long standing graceful recovery threshold of 10 minutes. Meaning that if a host looses heartbeat and enters Graceful Recovery, if the uptime value extracted from the first mtcAlive message following the recovery of that host exceeds 10 minutes, then maintenance interprets that the host did not reboot. If a host goes absent for longer than this threshold then for reasons not limited to security, maintenance declares the host as 'failed' and force re-enables it through a reboot. With the introduction of containers and addition of new features over the last few releases, boot times on some servers are approaching the 10 minute threshold and in this case exceeded the threshold. The primary fix in this update is to increase this long standing threshold to 15 minutes to account for evolution of the product. During the debug of this issue a few other related undesirable behaviors related to Graceful Recovery were observed with the following additional changes implemented. - Remove hbsAgent process restart in ha service management failover failure recovery handling. This change is in the ha git with a loose dependency placed on this update. Reason: https://review.opendev.org/c/starlingx/ha/+/788299 - Prevent the hbsAgent from sending heartbeat clear events to maintenance in response to a heartbeat stop command. Reason: Maintenance receiving these clear events while in Graceful Recovery causes it to pop out of graceful recovery only to re-enter as a retry and therefore needlessly consumes one (of a max of 5) retry count. - Prevent successful Graceful Recovery until all heartbeat monitored networks recover. Reason: If heartbeat of one network, say cluster recovers but another (management) does not then its possible the max Graceful Recovery Retries could be reached quite quickly, while one network recovered but the other may not have, causing maintenance to fail the host and force a full enable with reboot. - Extend the wait for the hbsClient ready event in the graceful recovery handler timout from 1 minute to worker config timeout. Reason: To give the worker config time to complete before force starting the recovery handler's heartbeat soak. - Add Graceful Recovery Wait state recovery over process restart. Reason: Avoid double reboot of Gracefully Recovering host over SM service bounce. - Add requirement for a valid out-of-band mtce flags value before declaring configuration error in the subfunction enable handler. Reason: rebooting the active controller can sometimes result in a falsely reported configation error due to the subfunction enable handler interpreting a zero value as a configuration error. - Add uptime to all Graceful Recovery 'Connectivity Recovered' logs. Reason: To assist log analysis and issue debug Test Plan: PASS: Verify handling active controller reboot cases: AIO DC, AIO DX, Standard, and Storage PASS: Verify Graceful Recovery Wait behavior cases: with and without timeout, with and without bmc cases: uptime > 15 mins and 10 < uptime < 15 mins PASS: Verify Graceful Recovery continuation over mtcAgent restart cases: peer controller, compute, MNFA 4 computes PASS: Verify AIO DX and DC active controller reboot to standby takeover that up for less than 15 minutes. Regression: PASS: Verify MNFA feature ; 4 computes in 8 node Storage system PASS: Verify cluster network only heartbeat loss handling cases: worker and standby controller in all systems. PASS: Verify Dead Office Recovery (DOR) cases: AIO DC, AIO DX, Standard, Storage PASS: Verify system installations cases: AIO SX/DC/DX and 8 node Storage system PASS: Verify heartbeat and graceful recovery of both 'standby controller' and worker nodes in AIO Plus. PASS: Verify logging and no coredumps over all of testing PASS: Verify no missing or stuck alarms over all of testing Change-Id: I3d16d8627b7e838faf931a3c2039a6babf2a79ef Closes-Bug: 1922584 Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com> |
||
|---|---|---|
| api-ref/source | ||
| bsp-files | ||
| devstack | ||
| doc | ||
| installer | ||
| kickstart | ||
| mtce | ||
| mtce-common | ||
| mtce-compute | ||
| mtce-control | ||
| mtce-storage | ||
| releasenotes | ||
| tools/rvmc/centos | ||
| .gitignore | ||
| .gitreview | ||
| .zuul.yaml | ||
| CONTRIBUTORS.wrs | ||
| LICENSE | ||
| README.rst | ||
| centos_build_layer.cfg | ||
| centos_iso_image.inc | ||
| centos_pkg_dirs | ||
| centos_stable_docker_images.inc | ||
| pylint.rc | ||
| test-requirements.txt | ||
| tox.ini | ||
README.rst
metal
StarlingX Bare Metal Management