From 7da4eb945fd64ecddbe04f90695e471cf52f58c6 Mon Sep 17 00:00:00 2001 From: Eric MacDonald Date: Tue, 3 Jul 2018 12:40:09 -0400 Subject: [PATCH] Enable host heartbeat in add handler when not in DOR mode Two Node System: VMs did not switch to ERROR state after host reboot A logically failed (rebooted) active controller is not being administratively failed by maintenance. As a result the host's offline availability state is not reported to the VIM and the VMs on that (rebooted) All-in-one host are not evacuated. This issue only applies to two node systems because of how the heartbeat enable of an All-in-one host needs to be held off until its compute manifests apply in the DOR case so as to avoid maintenance failing the peer controller over a DOR. The challange in maintenance is to distinguish between this spontaneous failure and a DOR. For All-in-one hosts, DOR mode is active for a whopping 600 seconds ; long enough to account for both sets of manifests to apply. It's that long delay that is making this silent fault stand out so obviously. This update uses 'active DOR mode' to decide whether or not to enable a host's heartbeat in the add handler. To better handle early active controller failure the qualifier for DOR mode was reduced from 20 to 15 minutes. Meaning that maintenance DOR mode is activated if its host up time is less than 15 minutes ; rather than 20 as it was before this update. Note that normally the active controller starts maintenance with an uptime of 5-7 minutes. Story: 2002995 Task: 23009 Change-Id: I749aefef45b9db6e86a2c6b81d131ebeccc68926 Signed-off-by: David Sullivan --- mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeCtrl.cpp | 2 +- mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeHdlrs.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeCtrl.cpp b/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeCtrl.cpp index 7e67f50f..6e34d367 100755 --- a/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeCtrl.cpp +++ b/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeCtrl.cpp @@ -1248,7 +1248,7 @@ void daemon_service_run ( void ) } #endif - if ( ts.tv_sec < MTC_MINS_20 ) + if ( ts.tv_sec < MTC_MINS_15 ) { /* CPE DOR window is much greater in CPE since heartbeat * cannot start until the inactive CPE has run both manifests */ diff --git a/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeHdlrs.cpp b/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeHdlrs.cpp index a5316f34..1a3d1c7e 100755 --- a/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeHdlrs.cpp +++ b/mtce-common/cgts-mtce-common-1.0/maintenance/mtcNodeHdlrs.cpp @@ -5607,7 +5607,7 @@ int nodeLinkClass::add_handler ( struct nodeLinkClass::node * node_ptr ) if ( NOT_THIS_HOST ) { if (( LARGE_SYSTEM ) || - (( CPE_SYSTEM ) && ( is_controller(node_ptr) == false ))) + (( CPE_SYSTEM ) && ( this->dor_mode_active == false ))) { send_hbs_command ( node_ptr->hostname, MTC_CMD_START_HOST ); }