starlingx
/
config
Code Issues Proposed changes

Fix failed pods not being detected by rootca health check 

On the health check prior to rootca update, there was a bug that
prevented CrashLoopBackoff pods being detected as unhealthy. This is
because the pods are in phase "Running", but the status of the
container itself is "ready: false". This commit adds an additional
check to "Running" pods so if any container inside it is not ready,
the pod will be deemed unhealthy.

Test plan:
  - PASS: Attempt to perform a rootca update with a pod in
          CrashloopBackoff state. Verify the update is not possible
          and the health check fails with the pod being show as
          unhealthy is "system health-query-kube-upgrade --rootca"
  - PASS: Verify the rootca update is possible if no pods are in
          CrashloopBackoff state.

Closes-Bug: 2057779

Change-Id: I115b6621df11516db2279fe6bc96452d27975c50
Signed-off-by: Victor Romano <victor.gluzromano@windriver.com>
This commit is contained in:
Victor Romano 2024-03-13 09:06:49 -03:00
parent ca30ba6056
commit d807f868d6
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
sysinv/sysinv/sysinv/sysinv/common/health.py
View File

@ -264,6 +264,13 @@ class Health(object):
if pod.status.phase not in ['Pending', 'Running', 'Succeeded']:
# Add it to the failed list as it's not ready/completed/pending
fail_pod_list.append((pod.metadata.name, pod.metadata.namespace))
elif pod.status.phase == 'Running':
for container_status in pod.status.container_statuses:
if container_status.ready is not True:
# Pod has running status but it's not ready
fail_pod_list.append((pod.metadata.name,
pod.metadata.namespace))
break
success = not fail_pod_list
return success, fail_pod_list