diff --git a/doc/source/specs/stx-8.0/approved/security-2010310-apparmor-support.rst b/doc/source/specs/stx-8.0/approved/security-2010310-apparmor-support.rst new file mode 100644 index 0000000..d322b87 --- /dev/null +++ b/doc/source/specs/stx-8.0/approved/security-2010310-apparmor-support.rst @@ -0,0 +1,357 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. http://creativecommons.org/licenses/by/3.0/legalcode + +======================================== +StarlingX AppArmor Support +======================================== + +Storyboard: +https://storyboard.openstack.org/#!/story/2010310 + +This specification describes the introduction of AppArmor into the +StarlingX solution, initially for the purposes of running end user's +hosted containers under AppArmor profiles. + +AppArmor is a Mandatory Access Control (MAC) system built on Linux's +LSM (Linux Security Modules) interface. In practice, the kernel queries +AppArmor before each system call to know whether the process is authorized +to do the given operation. Through this mechanism, AppArmor confines programs +to a limited set of resources. + +AppArmor applies a set of rules (known as a “profile”) on each program. The +profile applied by the kernel depends on the installation path of the program +being executed, the rules applied do not depend on the user. All users face +the same set of rules when they are executing the same program but traditional +user permissions still apply and might result in different behavior. + +AppArmor profiles contain a list of access control rules on resources that each +program can make use of. Each profile can be loaded either in enforcing or +complaining mode. The former enforces the policy and reports violation attempts, +while the latter does not enforce the policy but still logs the system calls +that would have been denied. + +AppArmor helps administrators in running a more secure kubernetes deployment by +restricting what containers/pods are allowed to do, and/or provide better +auditing through system logs.The access needed by a container/pod is configured +through profiles tuned to allow the access such as Linux capabilities, network +access, file permissions, etc. + + + +Problem description +=================== +Need a mechanism for restricting application pods run by the end users. AppArmor does +this by allowing the system administrator to restrict programs' capabilities +with per-program profiles.Then Kubernetes' container.apparmor.security.beta.kubernetes.io +annotation can be used in a pod spec to assign an AppArmor profile to a pod. + +Management of AppArmor profiles across all the kubernetes hosts of a StarlingX kubernetes +cluster should be simple for end user. + +There are concerns over the system performance degradation when enabling the +AppArmor. However, some end users want it enabled for security reasons, and are willing +to accept the system performance degradation. + + +Use Cases +--------- + +AppArmor can be optionally enabled on a per host basis: + +* Default recommendation will be that AppArmor is enabled on all hosts +* However, if required (e.g. to avoid performance impact on some cards), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled. + +If AppArmor is enabled on host(s) of a StarlingX system, a user will be able to run selected user's hosted application pods under a selected AppArmor profile by specifying the AppArmor profile in the pod's spec. + +**NOTE** Support for AppArmor profiles on the StarlingX Platform entities (both host processes and containers) themselves, is left to a future release. + +Proposed change +=============== + +This change provides a way to enable/disable on a particular host. + +* Default recommendation will be that AppArmor is enabled on all hosts +* However, if required (e.g. to avoid performance impact on some hosts), user could choose + which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) + **NOTE** In this case, a user would need to use labels and node-selector in pod spec to ensure the pod + is only scheduled on nodes with AppArmor enabled. + +In order apply the profiles to a particular pod, the profiles need to be available to the host machine +where pod is launched. Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator) +provides AppArmor profile management (i.e. loading/unloading) across k8s nodes. SPO defines an AppArmor Profile CRD, such +that end users' can define AppArmor profiles for SPO to manage. + +SPO will be integrated as a system application using FluxCD manifest which utilizes helm chart that +deploys a controller/operator for centrally managing the AppArmor Profile resources, a webhook for validation of AppArmor +Profile resources and a daemonset for loading/unloading the AppArmor Profiles. The DaemonSet configuration creates an SPO +pod/container on each host in the StarlingX cluster. +Creating a DaemonSet configuration for AppArmor container ensures that SPO will run on all kubernetes nodes, +such as controllers, all-in-one controllers and worker nodes. + +Once an AppArmor profile is loaded to the K8S nodes, it can be applied to a particular pod using annotations. +| example: container.apparmor.security.beta.kubernetes.io/:localhost/. Please refer +https://kubernetes.io/docs/tutorials/security/apparmor/#example + +Alternatives +------------ + +In order apply the profiles to a particular pod, the profiles need to be available to the host +machine where pod is launched. There are several solutions available and below alternatives were evaluated: + +* security-profiles-operator +* apparmor-loader +* kube_apparmor_Manager +* Manually copying and loading profiles on each host (DEV Options) + +Below is a comparision table + +.. list-table:: Alternatives to load AppArmor profiles in K8s cluster + :widths: 10 20 30 20 20 + :header-rows: 1 + + * - + - AppArmor Loader + - Security Profile Operator(SPO) + - AppArmor_Kube_Manager + - Manually Loading Profiles (DEV options) + * - Community Type + - K8S + - K8S + - Private, sysdiglab + - NA + * - Approach + - Daemon Set + - Daemon Set + - SSH based copy + - Manual copy (SSH based) + * - Developement Type + - POC + - Controlled as Alpha, beta release + - Not sure + - NA + * - Active + - Last updated 4 years back + - Very Much Active + - Last updated 2 years back + - NA + * - Release Frequency + - NA + - Quarterly basis + - NA + - NA + * - Latest Version + - No version control + - V0.4.3 + - V0.0.5 + - NA + * - Last Release Data + - + - Jun 07, 2022 + - Apr 04, 2020 + - NA + * - Roadmap + - No + - Yes, KEP and Roadmap in place + - No + - NA + * - Github Link + - https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader + - https://github.com/kubernetes-sigs/security-profiles-operator + - https://github.com/sysdiglabs/kube-apparmor-manager + - NA + * - Functionality + - Only Load AppArmor profiles on all worker nodes in k8s cluster + - Out-of-tree Kubernetes enhancement | Supports SELinux, seccomp, AppArmor | AppArmor profiles load and unload | Feature enabling and disabling using feature gate | Supplementary features – Metrics , service monitoring, log enricher, | Debugging features- CPU and Memory profile endpoints + - Load AppArmor profiles on all worker nodes in k8s cluster | Sync feature to cater profile change synchronize across worker nodes + - Copy the profiles on each host using SSH | Load the profile on each host using aa-enforce, aa-complain, or apparmor_parser | Unload the profiles on each host using aa-disable or apparmer_parser + + +Security Profile Operator (SPO) is the best option available and will continue our effort in incorporating it in StarlingX based on below advantages. + +* Better community support +* Feature rich +* Roadmap in place + +There are concerns about the stability of AppArmor support in SPO and the basic functionality of deploying AppArmor +profiles to various nodes is not working at this point. We will continue our effort to +track SPO fixes to apparmor functionality. + +* Retest updates to SPO and provide feedback to contributor quickly +* Offer any assistance we can to SPO contributor +* Identify any missing/desired functionality (e.g. complain vs enforce) + +For now, we will proceed in parallel with + +* using DEV option of linux AppArmor tools (aa commands, https://gitlab.com/apparmor/apparmor/-/wikis/ManPages) to load apparmor profiles on hosts AND +* work closely in testing and providing feedback (and possibly changes) to SPO project + + +Data model impact +----------------- + +A host level attribute "apparmor" would be added to the host table. +This can contain two possible values "enabled" or "disabled" + +REST API impact +--------------- +AppArmor can be enabled or disabled at runtime on a per-host basis. + +* Below cli command(s) would be used to enable or disable AppArmor in kernel for a particular host + * system host-lock + * system host-update apparmor=enabled/disabled **Example:** system host-update controller-0 apparmor=enabled + * system host-unlock + + The above command(s) would enable the AppArmor in kernel with NO services (on host or containerized) + in the WRCP/StarlingX Solution running under an apparmor profile + **NOTE** will explore whether default profiles enabled in Debian Bullseye can be used as default + when AppArmor is enabled + +* Changes in the below API to reflect AppArmor. | + GET /v1/ihosts/​{host_id}​ | + Description: Shows detailed information about a specific host | + INPUT: | + host_id : csapi:UUID: part of user-info | + Output: | + "apparmor": "enabled" or "apparmor": "disabled" would be added to the output json | + +* Addition of a new attribute 'apparmor' to the below API: + +| PATCH /v1/ihosts/​{host_id}​ +| Apparmor attribute can be modified. Support would be added for apparmor (xsd:string) +| INPUT: +| host_id : csapi:UUID: part of user-info +| Example: [ +| { +| "path":"/apparmor", +| "value":"enabled", +| "op":"replace" +| } +| ] +| Output: +| "apparmor": "enabled" or "apparmor": "disabled" would be added to the output json +| **NOTE**: Changing the 'apparmor' attribute, will ONLY be allowed when host is locked. + + +Security impact +--------------- + +The AppArmor helps in running a more secure kubernetes deployment by restricting what +containers/pods are allowed to do. User should be able to run selected user's hosted +application pods under a selected AppArmor profile. AppArmor enforces the policy defined +in the profile as well as reports policy violation attempts. + + +Other end user impact +--------------------- + +* If apparmor is NOT enabled (disabled by default), then there is NO impact on end user +* If apparmor is used, + * there is admin user impact to enable apparmor on hosts and load apparmor profiles on host and + * there is general user impact that they 'can' run hosted containers under apparmor profiles for enhanced security. + +Performance Impact +------------------ + +AppArmor is known to have a performance impact when running. There will be a performance impact characterization +(StarlingX Infrastructure Operations and Datapaths) of enabling AppArmor in the StarlingX which involved getting data with: + +* apparmor disabled +* apparmor enabled but NO apparmor profile applied to datapath app pod +* apparmor enabled and apparmor profile applied to datapath app pod + + +Other deployer impact +--------------------- + +Developer impact +---------------- + +None. + +Upgrade impact +-------------- + +A new host level attribute "apparmor" would be added to the host table. There should not be any impact on backup and restore, upgrade, and rollback. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: +* Jagatguru Prasad Mishra (jmishra) + +Other contributors: +* Sirin Shaikh (sshaikh) +* Kirti Singh (ksingh) +* Rahul Roshan Kachchap (rkachchap) + + +Repos Impacted +-------------- + +* starlingx/config and starlingx/stx-puppet repo would be modified to add API/CLI support for AppArmor. +* A new repo would be created for security-profiles-operator. + +Work Items +---------- + +* Integration of SPO as a system application +* CLI, RESTAPI, DB update and config update backend +* Performance impact testing +* End-to-end integration testing + +Dependencies +============ + +* Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator) + +Testing +======= + +Testing will involve: + +* Unit test cases for the new code introduced +* Installing/enabling the SPO system app +* verifying the enabling/disabling of Apparmor at host level using API and CLI +* Configuring Apparmor profiles across hosts +* Configuring Kubernetes Pods with Apparmor profiles +* Verifying that a policy violation by a container is logged in file "/var/log/syslog". +* Backup and Restore testing +* Upgrades testing +* DC testing + + +Documentation Impact +==================== + +Some information that is useful to be documented will be: + +* enabling Apparmor per host +* Configuring Apparmor profiles across hosts +* Configuring Kubernetes Pods with Apparmor profiles with example to do simple test +* Guidelines on how to write/develop apparmor profiles for your containers using aa-genprof + +References +========== + +* https://www.appvia.io/blog/how-to-use-the-security-profiles-operator +* https://medium.com/information-and-technology/so-what-is-apparmor-64d7ae211ed +* https://github.com/kubernetes-sigs/security-profiles-operator +* https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader +* https://github.com/sysdiglabs/kube-apparmor-manager +* https://kubernetes.io/docs/tutorials/security/apparmor/#example + + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - STX-8.0 + - Introduced