05bbc77057
This update is to prevent nodes from crashing while powering
off during graceful shutdown (or reboot). This improves timing
and shutdown of containerd.service.
The containerd shutdown script stops all containers via
'crictl stop' with 5 second timeout, followed by stop all
pods via 'crictl stopp'. This cleans up lingering /pause
sandbox containers.
This modifies the arguments to xargs and crictl to let xargs
deal with parallelism instead of batching to crictl.
crictl appears to do the stop operations serially.
The number stop in parallel is engineered to 10.
Engineering the number of stop in parallel in relation to
shutdown timings under stress load will be addressed in a
subsequent update. The engineering TC should align with
customer requirements.
When testing containerd shutdown under the stress of multiple
pods writing to a shared PersistentVolume, even the new parallel
shutdown code is not sufficient to complete the shutdown within
the default 90-second timeout. Additional changes will be needed
to enable clean shutdown under those circumstances.
Partial-Bug: 2043069
Test plan:
- PASS - build-image, install and boot up on AIO-SX
- PASS - perform reboot and verify /var/log/daemon.log
has new k8s-container-cleanup.sh logs
for 'Stopping all pods' and 'Stopping all containers',
and that drbd stops after containerd.
- FAIL - verify containerd shutdown works under stress with
the new parallel stop pods parameter NPAR=10.
The stress load uses ReadWriteMany PVC, and multiple
pods, each writing to the shared PVC.
Change-Id: Ibfc0a474a40344a629b3f0780449906a9c6b03ba
Signed-off-by: Jim Gauld <James.Gauld@windriver.com>
|
||
|---|---|---|
| .. | ||
| deb_folder | ||
| files | ||
| README.txt | ||
| meta_data.yaml | ||
README.txt
The containerd.service file and config.toml were pulled in from the upstream Debian package containerd_1.6.20~ds1-1+b1_amd64.deb downloaded from http://ftp.ca.debian.org/debian/pool/main/c/containerd/ The config.toml file is identical to what we were using previously with the older version of containerd, and is unchanged in the newer version of the package. It will get overwritten by ansible/puppet anyways during system bringup. The containerd.service file is identical to the version from the containerd github source tag "v1.6.21" except that the containerd binary is in /usr/bin/ instead of /usr/local/bin. The only difference from what we had before is that LimitNOFILE is now set to "infinity" to align with both Debian and containerd upstream. The binaries that get pulled in at build time are from the containerd upstream binary release containerd-1.6.21-linux-amd64.tar.gz downloaded from https://github.com/containerd/containerd/releases/tag/v1.6.21 The rationale for using the upstream binaries rather than the Debian "bookworm" package is that the Debian package requires a lot of other dependencies including newer glibc and python3, which would be too intrusive for our purposes.