Introduce failover FSM to handle communication failure between
controllers.
Failover FSM has 4 states:
Normal: when system running with full redundency
Fail Pending: communication failure occured
Failed: the controller is determined as failure. Its peer will
assume service
Survived: the controller is determined as survivor. Its peer has
failed
The controllers are in one of the below possible state pairs:
normal/normal, fail-pending/fail-pending, failed/survived
A failed controller will not resume responsbility before the
system restores its full redundency (normal/normal)
A survivor will not fail before the system restores its
full redundency (normal/normal)
Future implementation may allow an administrator to force
a failed controller become active, to manually recover
(with possiblity of losing data), should the survivor is
no longer capable to provide service.
Story: 2003577
Task: 26404
Change-Id: I51635e9e60b6fb6bad89e06c9f08d3f28e21db82
Signed-off-by: Bin Qian <bin.qian@windriver.com>
edc8a56472