diff --git a/doc/source/specs/stx-3.0/approved/containerization-2006145-kata-containers-integration.rst b/doc/source/specs/stx-3.0/approved/containerization-2006145-kata-containers-integration.rst new file mode 100644 index 0000000..aba2f6f --- /dev/null +++ b/doc/source/specs/stx-3.0/approved/containerization-2006145-kata-containers-integration.rst @@ -0,0 +1,322 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. http://creativecommons.org/licenses/by/3.0/legalcode + +======================================== +Integrate Kata Containers into StarlingX +======================================== + +Storyboard: https://storyboard.openstack.org/#!/story/2006145 + +This story integrates Kata Containers into StarlingX. So that StarlingX could +support container run with kata container runtime. + +Problem description +=================== +Kata Containers is an open source community working to build a secure +container runtime with lightweight virtual machines that feel and perform +like containers, but provide stronger workload isolation using hardware +virtualization technology as a second layer of defense.[0] + +This story try to integrate Kata Containers into StarlingX, to address +customer's concern for the security of container. + +Due to there is still performance gap between Kata Containers and conventional +containers [1], and Kubernetes cluster system cannot completely run on Kata +Containers [2], Kata Containers is not used in default unless explicitly +requirement is found in the Kubernetes's pod configuration file. Or in other +words, regardless Kata Containers is enabled or not by this story, containers +currently we have in StarlingX will not use kata container runtime in default, +unless extra change is made. The method to declare the requirement is shared in +"Proposed change" section. + +Use Cases +--------- +With Kata Containers supported in StarlingX, System developers, testers, +operators, administrators could choose container runtime to use when run +a container image, based on their needs. Such as if there is security concern +for the pod/container, it could be selected to run with kata container. For the +default case, runc will be selected as default low-level container runtime, +which is the default low-level runtime for Kubernetes. + +Proposed change +=============== +Kubernetes is the only container orchestration tool in StarlingX. To integrate +Kata Containers into StarlingX, Kuberentes will be configured to support kata +container runtime. + +For Kubernetes, there are two methods to select kata container runtime. +1. Use annotation. Add "io.kubernetes.cri.untrusted-workload" annotation in +pod file.[3] +2. Use Runtime Class.[4] +1st method is simpler, but it is deprecated. 2nd method is recommended. +Runtime Class is supported in Kubernetes since v1.14, and current Kubernetes +version in StarlingX master is v1.15. So use Runtime Class to select kata +container runtime is recommended. + +There are 3 major container runtime in kubernetes: docker/cri-o/containerd.[5] +Docker runtime in Kubernetes supports runc only, it cannot be configured to +support kata runtime. So in order to use kata container runtime, Kubernetes +has to switch to use containerd or cri-o. + +Per Kata Containers [6]: "As of Kata Containers 1.5, using shimv2 with +containerd 1.2.0 or above is the preferred way to run Kata Containers with +Kubernetes (see the howto). The CRI-O will catch up soon." So I choose +containerd as the new container runtime for Kubernetes to support Kata +Containers. + +So we have two solutions here now for Kata Containers support in Kubernetes. +Solution 1: +A global flag will be added in ansible playbook local override file +(localhost.yml) to let deployer choose whether Kata Containers support is +enabled or not in StarlingX. +If flag is not set or set as False, Kata Containers support is disabled, +Kubernetes uses docker as container runtime. The flow will be like below: +kubelet -> docker -> runc -> container image +If flag is set as True, Kata Containers support is enabled, Kubernetes uses +containerd as container runtime. For container doesn't require kata container, +the flow will be like below: +kubelet -> containerd -> runc -> container image +For container requires kata container, the follow will be like below: +kubelet -> containerd -> kata container runtime -> container image +Advantage: +For user doesn't use Kata Containers, system will act the same as previous. +Code implementation will not affect current design. So it could be implemented +in master directly piece by piece. +Disadvantage: +User cannot enable Kata Containers support after deployment. +Code implementation needs consider both docker and containerd case. + +Solution 2: +No choice for user, and Kubernetes uses containerd as the container runtime. +For container doesn't require kata container, the flow will be like below: +kubelet -> containerd -> runc -> container image +For container requires kata container, the flow will be like below: +kubelet -> containerd -> kata container runtime -> container image +Advantage: +User always could use kata container. +Code implementation will be simpler since just need consider containerd case. +Disadvantage: +Docker tools cannot be used for container deployed by Kubernetes. User should +use kubectl to operate container or new tool like crictl/ctr. +There is a cutover from docker to containerd. Code need be fully verified +before merge. + +To keep code clean and easy for maintenance, Solution 2 will be choosen for +this feature implementation. +More details for Solution 2 implementation is shared in "Work Items" section. + +Alternatives +------------ + +CRI-O should be another option to enable Kata Containers support in Kubernetes. +Containerd is selected instead of CRI-O, because Kata Containers prefer it.[5] + +Data model impact +----------------- + +None. This story does not change any existing data models. + +REST API impact +--------------- + +None. This story does not change any existing REST APIs. + +Security impact +--------------- + +There is no sensitive data touch for this feature, except containerd need +touch certificate/key for local secure registry "registry.local". +Kata Containers could help enhance system's security by "Runs in a dedicated +kernel, providing isolation of network, I/O and memory and can utilize +hardware-enforced isolation with virtualization VT extensions." [0] + +Other end user impact +--------------------- + +With this feature implemented, users are able to run containers with +kata-runtime. +The major change for this feature is Kubernetes switch to use containerd as +container runtime instead of docker. This change is transparent if users access +the container with kubectl. Users will find container created by Kubernetes +cannot be accessd by docker CLI. Users need use other CLI like crictl. + +Performance Impact +------------------ + +For containers doesn't use kata-runtime, the performance should be the same as +previous. +For containers use kata-runtime, some performance drop is expected. Per Kata +Containers website:"Delivers consistent performance as standard Linux +containers; increased isolation with the performance tax of standard virtual +machines."[0] +Also there is a report of "I/O performance of Kata Containers". [1] + +Other deployer impact +--------------------- + +There is no change for the StarlingX deploy process. +In node installation stage, a few RPMs for Kata Containers will be installed +automatically in each node in order to support Kata Containers. +In node configuration stage, docker and Kubernetes will be auto configured to +support kata container runtime. +After deployment, users need to update application tarball file to select kata +runtime if want to use kata-runtime. Current application tarball will still +work with runc runtime, just the same as before. + +Developer impact +---------------- + +The major change for this feature is, Kubernetes will switch to use containerd +as container runtime. So any code implemented as docker specific in Kubernetes +need switch to use containerd. + +Upgrade impact +-------------- + +This feature will introduce several new rpms for kata container. Kubernetes may +be initialized with another container runtime during upgrade. Per my test, +container runtime selection is per node. It means it is independent for the +container runtime for each node. So this feature implementation should not +affect upgrade. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Shuicheng Lin + Yi Wang + +Repos Impacted +-------------- + +* stx-integ - add containerd configuration files +* stx-config - add containerd confguration +* stx-ansible-playbook - configure Kubernetes to use containerd +* stx-root - add kata container rpms to ISO +* stx-metal - add containerd partition +* stx-tools - add kata container repo and rpms + +Work Items +---------- + +Add kata container +* Add kata container rpms to ISO and install to nodes + + There is repo for Kata Containers rpms on CentOS 7. [7] + kata-runtime rpm will be added to ISO explicitly, and other rpms will be + added as dependency automatically. + Kata Containers uses qemu as the hypervisor. Kata also could be configured + to use Firecracker, which is out of scope of this story, and could be + handled later if there is request. + +Support kata container in Kubernetes +* Configure containerd/kubelet service + + Containerd binary is already installed in StarlingX as part of docker. + A package with containerd related configuration file will be added. File + with parameter need be configured dynamically will be configured through + puppet. File doesn't need be updated will be installed by package. + +* Switch kubeadm to use containerd + + This will be done in ansible playbook for controller-0, and puppet + configuration for controller-1, both by kubeadm init cmd. For worker node, + it will be configured to use containerd by kubeadm join cmd. + +* Configure containerd/docker to share same partition for container storage + + By default, docker and containerd has different path for container image + storage. /var/lib/docker for docker, and /var/lib/containerd for containerd. + This is already a 30G lvm partition(/var/run/docker) created for docker + currently. This partition is mainly for save container images and container + runtime ephemeral data. With current StarlingX system, there are two + container images armada and tiller pulled by docker directly, and 2 + application platform-integ-apps and stx-openstack pulled by Kubernetes. + Armada and tiller images consume 550 Mb disk, while 2 applications consume + 17G disk. + To avoid create 1 more lvm partition for containerd. Containerd will be + configured to use /var/lib/docker path also. And all container images will + be pulled by containerd in default. But users are still able to pull and run + container with docker. + +* Add crictl as the CLI for containerd + + crictl is a CLI for CRI-compatible container runtimes. It allows the CRI + runtime developers to debug their runtime without needing to set up + Kubernetes components.[9] It could help debug containerd related issues. + There is a rpm for this tool from OpenSuSE repo with latest version, but it + is configured to contact with cri-o in default, need update configure file to + contact with containerd. + +* Configure containerd to support insecure registry/local registry/proxy etc + + This is to replace the docker specific function used in Kubernetes. + Containerd also need be configured to support upper feature. + Docker SDK is used in sysinv to pull container image, and push it to local + registry. Containerd doesn't have similar python SDK. We need re-write the + code to call containerd or CLI to achieve the same function. + +Besides upper work items, there are other work items maybe need added during +implementation of this story. Please check the story link in the beginning of +this spec to get the full work item list. + +Dependencies +============ + +This specification depends upon the open source upstream: +Kata Containers/containerd +https://github.com/kata-containers/kata-containers +https://github.com/containerd/containerd + +Testing +======= + +This feature can be tested in a fully provisioned StarlingX system + +* if kata container support flag is not set, or set as False, the system + should perform as previous. +* if kata container support flag is set as True, the system should also + pass deploy/sanity test, and containers managed by Kubernetes should be + managed by containerd also, and use runc as low-level runtime in default. + With kata container required container, it should be run with kata runtime. + +To check whether the running container is with kata runtime or not, we could +login into the container, and use "uname" cmd to check the kernel version. +Host's kernel version is 3.10 while kata's kernel version should be 4.x. + +Documentation Impact +==================== + +A new page should be added to show how to enable Kata Containers support, and +how to explicitly require kata container runtime in Kubernetes. This feature +will add a new container runtime containered for Kubernetes. Any docker +specific operation in Kubernetes may be updated to include containerd +operation. + +References +========== +[0]: https://katacontainers.io/ +[1]: https://www.stackhpc.com/kata-io-1.html +[2]: https://github.com/kata-containers/runtime/issues/1853 +[3]: https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-an-untrusted-pod-using-kata-containers +[4]: https://kubernetes.io/docs/concepts/containers/runtime-class/ +[5]: https://kubernetes.io/docs/setup/production-environment/container-runtimes/ +[6]: https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-a-cri-implementation +[7]: http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/CentOS_7/ +[8]: https://github.com/kata-containers/documentation/blob/master/install/docker/centos-docker-install.md +[9]: https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - stx-3.0 + - Introduced