358 lines
14 KiB
ReStructuredText
358 lines
14 KiB
ReStructuredText
..
|
||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||
License. http://creativecommons.org/licenses/by/3.0/legalcode
|
||
|
||
========================================
|
||
StarlingX AppArmor Support
|
||
========================================
|
||
|
||
Storyboard:
|
||
https://storyboard.openstack.org/#!/story/2010310
|
||
|
||
This specification describes the introduction of AppArmor into the
|
||
StarlingX solution, initially for the purposes of running end user's
|
||
hosted containers under AppArmor profiles.
|
||
|
||
AppArmor is a Mandatory Access Control (MAC) system built on Linux's
|
||
LSM (Linux Security Modules) interface. In practice, the kernel queries
|
||
AppArmor before each system call to know whether the process is authorized
|
||
to do the given operation. Through this mechanism, AppArmor confines programs
|
||
to a limited set of resources.
|
||
|
||
AppArmor applies a set of rules (known as a “profile”) on each program. The
|
||
profile applied by the kernel depends on the installation path of the program
|
||
being executed, the rules applied do not depend on the user. All users face
|
||
the same set of rules when they are executing the same program but traditional
|
||
user permissions still apply and might result in different behavior.
|
||
|
||
AppArmor profiles contain a list of access control rules on resources that each
|
||
program can make use of. Each profile can be loaded either in enforcing or
|
||
complaining mode. The former enforces the policy and reports violation attempts,
|
||
while the latter does not enforce the policy but still logs the system calls
|
||
that would have been denied.
|
||
|
||
AppArmor helps administrators in running a more secure kubernetes deployment by
|
||
restricting what containers/pods are allowed to do, and/or provide better
|
||
auditing through system logs.The access needed by a container/pod is configured
|
||
through profiles tuned to allow the access such as Linux capabilities, network
|
||
access, file permissions, etc.
|
||
|
||
|
||
|
||
Problem description
|
||
===================
|
||
Need a mechanism for restricting application pods run by the end users. AppArmor does
|
||
this by allowing the system administrator to restrict programs' capabilities
|
||
with per-program profiles.Then Kubernetes' container.apparmor.security.beta.kubernetes.io
|
||
annotation can be used in a pod spec to assign an AppArmor profile to a pod.
|
||
|
||
Management of AppArmor profiles across all the kubernetes hosts of a StarlingX kubernetes
|
||
cluster should be simple for end user.
|
||
|
||
There are concerns over the system performance degradation when enabling the
|
||
AppArmor. However, some end users want it enabled for security reasons, and are willing
|
||
to accept the system performance degradation.
|
||
|
||
|
||
Use Cases
|
||
---------
|
||
|
||
AppArmor can be optionally enabled on a per host basis:
|
||
|
||
* Default recommendation will be that AppArmor is enabled on all hosts
|
||
* However, if required (e.g. to avoid performance impact on some cards), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled.
|
||
|
||
If AppArmor is enabled on host(s) of a StarlingX system, a user will be able to run selected user's hosted application pods under a selected AppArmor profile by specifying the AppArmor profile in the pod's spec.
|
||
|
||
**NOTE** Support for AppArmor profiles on the StarlingX Platform entities (both host processes and containers) themselves, is left to a future release.
|
||
|
||
Proposed change
|
||
===============
|
||
|
||
This change provides a way to enable/disable on a particular host.
|
||
|
||
* Default recommendation will be that AppArmor is enabled on all hosts
|
||
* However, if required (e.g. to avoid performance impact on some hosts), user could choose
|
||
which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System)
|
||
**NOTE** In this case, a user would need to use labels and node-selector in pod spec to ensure the pod
|
||
is only scheduled on nodes with AppArmor enabled.
|
||
|
||
In order apply the profiles to a particular pod, the profiles need to be available to the host machine
|
||
where pod is launched. Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator)
|
||
provides AppArmor profile management (i.e. loading/unloading) across k8s nodes. SPO defines an AppArmor Profile CRD, such
|
||
that end users' can define AppArmor profiles for SPO to manage.
|
||
|
||
SPO will be integrated as a system application using FluxCD manifest which utilizes helm chart that
|
||
deploys a controller/operator for centrally managing the AppArmor Profile resources, a webhook for validation of AppArmor
|
||
Profile resources and a daemonset for loading/unloading the AppArmor Profiles. The DaemonSet configuration creates an SPO
|
||
pod/container on each host in the StarlingX cluster.
|
||
Creating a DaemonSet configuration for AppArmor container ensures that SPO will run on all kubernetes nodes,
|
||
such as controllers, all-in-one controllers and worker nodes.
|
||
|
||
Once an AppArmor profile is loaded to the K8S nodes, it can be applied to a particular pod using annotations.
|
||
| example: container.apparmor.security.beta.kubernetes.io/<pod_name>:localhost/<profile_ref>. Please refer
|
||
https://kubernetes.io/docs/tutorials/security/apparmor/#example
|
||
|
||
Alternatives
|
||
------------
|
||
|
||
In order apply the profiles to a particular pod, the profiles need to be available to the host
|
||
machine where pod is launched. There are several solutions available and below alternatives were evaluated:
|
||
|
||
* security-profiles-operator
|
||
* apparmor-loader
|
||
* kube_apparmor_Manager
|
||
* Manually copying and loading profiles on each host (DEV Options)
|
||
|
||
Below is a comparision table
|
||
|
||
.. list-table:: Alternatives to load AppArmor profiles in K8s cluster
|
||
:widths: 10 20 30 20 20
|
||
:header-rows: 1
|
||
|
||
* -
|
||
- AppArmor Loader
|
||
- Security Profile Operator(SPO)
|
||
- AppArmor_Kube_Manager
|
||
- Manually Loading Profiles (DEV options)
|
||
* - Community Type
|
||
- K8S
|
||
- K8S
|
||
- Private, sysdiglab
|
||
- NA
|
||
* - Approach
|
||
- Daemon Set
|
||
- Daemon Set
|
||
- SSH based copy
|
||
- Manual copy (SSH based)
|
||
* - Developement Type
|
||
- POC
|
||
- Controlled as Alpha, beta release
|
||
- Not sure
|
||
- NA
|
||
* - Active
|
||
- Last updated 4 years back
|
||
- Very Much Active
|
||
- Last updated 2 years back
|
||
- NA
|
||
* - Release Frequency
|
||
- NA
|
||
- Quarterly basis
|
||
- NA
|
||
- NA
|
||
* - Latest Version
|
||
- No version control
|
||
- V0.4.3
|
||
- V0.0.5
|
||
- NA
|
||
* - Last Release Data
|
||
-
|
||
- Jun 07, 2022
|
||
- Apr 04, 2020
|
||
- NA
|
||
* - Roadmap
|
||
- No
|
||
- Yes, KEP and Roadmap in place
|
||
- No
|
||
- NA
|
||
* - Github Link
|
||
- https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader
|
||
- https://github.com/kubernetes-sigs/security-profiles-operator
|
||
- https://github.com/sysdiglabs/kube-apparmor-manager
|
||
- NA
|
||
* - Functionality
|
||
- Only Load AppArmor profiles on all worker nodes in k8s cluster
|
||
- Out-of-tree Kubernetes enhancement | Supports SELinux, seccomp, AppArmor | AppArmor profiles load and unload | Feature enabling and disabling using feature gate | Supplementary features – Metrics , service monitoring, log enricher, | Debugging features- CPU and Memory profile endpoints
|
||
- Load AppArmor profiles on all worker nodes in k8s cluster | Sync feature to cater profile change synchronize across worker nodes
|
||
- Copy the profiles on each host using SSH | Load the profile on each host using aa-enforce, aa-complain, or apparmor_parser | Unload the profiles on each host using aa-disable or apparmer_parser
|
||
|
||
|
||
Security Profile Operator (SPO) is the best option available and will continue our effort in incorporating it in StarlingX based on below advantages.
|
||
|
||
* Better community support
|
||
* Feature rich
|
||
* Roadmap in place
|
||
|
||
There are concerns about the stability of AppArmor support in SPO and the basic functionality of deploying AppArmor
|
||
profiles to various nodes is not working at this point. We will continue our effort to
|
||
track SPO fixes to apparmor functionality.
|
||
|
||
* Retest updates to SPO and provide feedback to contributor quickly
|
||
* Offer any assistance we can to SPO contributor
|
||
* Identify any missing/desired functionality (e.g. complain vs enforce)
|
||
|
||
For now, we will proceed in parallel with
|
||
|
||
* using DEV option of linux AppArmor tools (aa commands, https://gitlab.com/apparmor/apparmor/-/wikis/ManPages) to load apparmor profiles on hosts AND
|
||
* work closely in testing and providing feedback (and possibly changes) to SPO project
|
||
|
||
|
||
Data model impact
|
||
-----------------
|
||
|
||
A host level attribute "apparmor" would be added to the host table.
|
||
This can contain two possible values "enabled" or "disabled"
|
||
|
||
REST API impact
|
||
---------------
|
||
AppArmor can be enabled or disabled at runtime on a per-host basis.
|
||
|
||
* Below cli command(s) would be used to enable or disable AppArmor in kernel for a particular host
|
||
* system host-lock <hostname>
|
||
* system host-update <hostname> apparmor=enabled/disabled **Example:** system host-update controller-0 apparmor=enabled
|
||
* system host-unlock <hostname>
|
||
|
||
The above command(s) would enable the AppArmor in kernel with NO services (on host or containerized)
|
||
in the WRCP/StarlingX Solution running under an apparmor profile
|
||
**NOTE** will explore whether default profiles enabled in Debian Bullseye can be used as default
|
||
when AppArmor is enabled
|
||
|
||
* Changes in the below API to reflect AppArmor. |
|
||
GET /v1/ihosts/{host_id} |
|
||
Description: Shows detailed information about a specific host |
|
||
INPUT: |
|
||
host_id : csapi:UUID: part of user-info |
|
||
Output: |
|
||
"apparmor": "enabled" or "apparmor": "disabled" would be added to the output json |
|
||
|
||
* Addition of a new attribute 'apparmor' to the below API:
|
||
|
||
| PATCH /v1/ihosts/{host_id}
|
||
| Apparmor attribute can be modified. Support would be added for apparmor (xsd:string)
|
||
| INPUT:
|
||
| host_id : csapi:UUID: part of user-info
|
||
| Example: [
|
||
| {
|
||
| "path":"/apparmor",
|
||
| "value":"enabled",
|
||
| "op":"replace"
|
||
| }
|
||
| ]
|
||
| Output:
|
||
| "apparmor": "enabled" or "apparmor": "disabled" would be added to the output json
|
||
| **NOTE**: Changing the 'apparmor' attribute, will ONLY be allowed when host is locked.
|
||
|
||
|
||
Security impact
|
||
---------------
|
||
|
||
The AppArmor helps in running a more secure kubernetes deployment by restricting what
|
||
containers/pods are allowed to do. User should be able to run selected user's hosted
|
||
application pods under a selected AppArmor profile. AppArmor enforces the policy defined
|
||
in the profile as well as reports policy violation attempts.
|
||
|
||
|
||
Other end user impact
|
||
---------------------
|
||
|
||
* If apparmor is NOT enabled (disabled by default), then there is NO impact on end user
|
||
* If apparmor is used,
|
||
* there is admin user impact to enable apparmor on hosts and load apparmor profiles on host and
|
||
* there is general user impact that they 'can' run hosted containers under apparmor profiles for enhanced security.
|
||
|
||
Performance Impact
|
||
------------------
|
||
|
||
AppArmor is known to have a performance impact when running. There will be a performance impact characterization
|
||
(StarlingX Infrastructure Operations and Datapaths) of enabling AppArmor in the StarlingX which involved getting data with:
|
||
|
||
* apparmor disabled
|
||
* apparmor enabled but NO apparmor profile applied to datapath app pod
|
||
* apparmor enabled and apparmor profile applied to datapath app pod
|
||
|
||
|
||
Other deployer impact
|
||
---------------------
|
||
|
||
Developer impact
|
||
----------------
|
||
|
||
None.
|
||
|
||
Upgrade impact
|
||
--------------
|
||
|
||
A new host level attribute "apparmor" would be added to the host table. There should not be any impact on backup and restore, upgrade, and rollback.
|
||
|
||
Implementation
|
||
==============
|
||
|
||
Assignee(s)
|
||
-----------
|
||
|
||
Primary assignee:
|
||
* Jagatguru Prasad Mishra (jmishra)
|
||
|
||
Other contributors:
|
||
* Sirin Shaikh (sshaikh)
|
||
* Kirti Singh (ksingh)
|
||
* Rahul Roshan Kachchap (rkachchap)
|
||
|
||
|
||
Repos Impacted
|
||
--------------
|
||
|
||
* starlingx/config and starlingx/stx-puppet repo would be modified to add API/CLI support for AppArmor.
|
||
* A new repo would be created for security-profiles-operator.
|
||
|
||
Work Items
|
||
----------
|
||
|
||
* Integration of SPO as a system application
|
||
* CLI, RESTAPI, DB update and config update backend
|
||
* Performance impact testing
|
||
* End-to-end integration testing
|
||
|
||
Dependencies
|
||
============
|
||
|
||
* Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator)
|
||
|
||
Testing
|
||
=======
|
||
|
||
Testing will involve:
|
||
|
||
* Unit test cases for the new code introduced
|
||
* Installing/enabling the SPO system app
|
||
* verifying the enabling/disabling of Apparmor at host level using API and CLI
|
||
* Configuring Apparmor profiles across hosts
|
||
* Configuring Kubernetes Pods with Apparmor profiles
|
||
* Verifying that a policy violation by a container is logged in file "/var/log/syslog".
|
||
* Backup and Restore testing
|
||
* Upgrades testing
|
||
* DC testing
|
||
|
||
|
||
Documentation Impact
|
||
====================
|
||
|
||
Some information that is useful to be documented will be:
|
||
|
||
* enabling Apparmor per host
|
||
* Configuring Apparmor profiles across hosts
|
||
* Configuring Kubernetes Pods with Apparmor profiles with example to do simple test
|
||
* Guidelines on how to write/develop apparmor profiles for your containers using aa-genprof
|
||
|
||
References
|
||
==========
|
||
|
||
* https://www.appvia.io/blog/how-to-use-the-security-profiles-operator
|
||
* https://medium.com/information-and-technology/so-what-is-apparmor-64d7ae211ed
|
||
* https://github.com/kubernetes-sigs/security-profiles-operator
|
||
* https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader
|
||
* https://github.com/sysdiglabs/kube-apparmor-manager
|
||
* https://kubernetes.io/docs/tutorials/security/apparmor/#example
|
||
|
||
|
||
History
|
||
=======
|
||
|
||
.. list-table:: Revisions
|
||
:header-rows: 1
|
||
|
||
* - Release Name
|
||
- Description
|
||
* - STX-8.0
|
||
- Introduced
|