This update is to prevent nodes from crashing while powering
off during graceful shutdown (or reboot). This improves timing
and shutdown of containerd.service.
The containerd shutdown script stops all containers via
'crictl stop' with 5 second timeout, followed by stop all
pods via 'crictl stopp'. This cleans up lingering /pause
sandbox containers.
This modifies the arguments to xargs and crictl to let xargs
deal with parallelism instead of batching to crictl.
crictl appears to do the stop operations serially.
The number stop in parallel is engineered to 10.
Engineering the number of stop in parallel in relation to
shutdown timings under stress load will be addressed in a
subsequent update. The engineering TC should align with
customer requirements.
When testing containerd shutdown under the stress of multiple
pods writing to a shared PersistentVolume, even the new parallel
shutdown code is not sufficient to complete the shutdown within
the default 90-second timeout. Additional changes will be needed
to enable clean shutdown under those circumstances.
Partial-Bug: 2043069
Test plan:
- PASS - build-image, install and boot up on AIO-SX
- PASS - perform reboot and verify /var/log/daemon.log
has new k8s-container-cleanup.sh logs
for 'Stopping all pods' and 'Stopping all containers',
and that drbd stops after containerd.
- FAIL - verify containerd shutdown works under stress with
the new parallel stop pods parameter NPAR=10.
The stress load uses ReadWriteMany PVC, and multiple
pods, each writing to the shared PVC.
Change-Id: Ibfc0a474a40344a629b3f0780449906a9c6b03ba
Signed-off-by: Jim Gauld <James.Gauld@windriver.com>
05bbc77057