Spec for kubernetes platform support

Submitting the following spec for review:
containerization-2002843-kubernetes-platform-support.rst

Change-Id: I86d28e38fe36572d5909eb2a36faa4aea18c6df8
Signed-off-by: Bart Wensley <barton.wensley@windriver.com>
This commit is contained in:
Bart Wensley 2018-10-11 13:35:23 -05:00
parent f7c04656d8
commit 65f6ef6324
1 changed files with 320 additions and 0 deletions

View File

@ -0,0 +1,320 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License. http://creativecommons.org/licenses/by/3.0/legalcode
===========================
Kubernetes Platform Support
===========================
Storyboard: https://storyboard.openstack.org/#!/story/2002843
This story will implement the initial items to integrate kubernetes on
StarlingX. The StarlingX kubernetes cluster will initially support:
* Docker runtime
* Calico CNI plugin
* Ceph as persistent storage backend
* Helm as the package manager
* Local Docker Image Registry
* 2 node HA solution
* Node labeling integration with inventory subsystem
Problem description
===================
In order to support containerized applications on the StarlingX platform,
we need to package and deploy kubernetes and associated components, such
as docker, calico and helm.
This is a first step in a larger effort. Please see the following for
details of the end goal:
https://wiki.openstack.org/wiki/Containerizing_StarlingX_Infrastructure
Use Cases
=========
Developers need the ability to install and configure a StarlingX system
with kubernetes master and worker nodes, in order to allow further
development efforts to support containerized services.
Developers/testers/users need the ability to install and configure a
StarlingX system without any kubernetes functionality, in order to avoid
any impacts to non-kubernetes related development and testing activities.
Proposed change
===============
The packages required to support a basic kubernetes deployment will be added
to the load. This includes packages related to:
* kubernetes
* etcd
* docker and docker registry
* helm client
A --kubernetes option will be added to config_controller. When this option is
not used, the system will be configured without any kubernetes functionality.
When the --kubernetes option is used, kubernetes will be configured. This
includes the following:
* The etcd will be started and will be managed by stx-ha Service Management
(SM) in an active/standby configuration.
* The bare metal OpenStack services will not be started and will not appear
in SM, with the exception of keystone, which will continue to run on
bare metal to provide authentication for platform services (e.g. System
Inventory (sysinv), patching, stx-nfv Virtual Infrastructure Manager (VIM)).
* The kubeadm tool will be used to automatically configure both controller
hosts as kubernetes master nodes. Kubernetes will initially use the existing
management network and floating management IP for the kube-apiserver.
* Kubeadm will download any required images from the usual external
image registries. An iptables NAT rule will be added on controller hosts to
allow compute hosts to access the external image registries through the
controller's OAM interface.
* The calico CNI plugin will be installed to provide pod networking.
* The k8s-keystone-auth will be configured as the authorization backend for
the kube-apiserver.
* Helm will be configured to manage kubernetes applications.
* When a compute host is installed, the kubeadm tool will be used to
automatically configure it as a kubernetes worker node.
* The VIM will provide activity management for kubernetes hosts. When hosts
are removed from service (by locking them or because they have failed),
the VIM will prevent dynamic pods from running on these hosts by applying
a NoExecute taint to the assocatied kubernetes node.
* New inventory CLIs (system host-label-assign/list/remove) will be provided
to allow labels to be added/removed to kubernetes nodes.
* When a host acting as a master/worker node is deleted, use the kubernetes
API to delete all resources associated with the node.
Alternatives
============
There are many alternatives to kubeadm that can be used to install a
kubernetes cluster (e.g. bootkube, kubespray, puppetlabs-kubernetes,
promenade). We chose kubeadm because:
* It has the most community support and is the most up-to-date with
kubernetes features.
* It should be easy to integrate with our existing installation/commissioning
architecture. Will get us up and running and allow us to understand any
limitations better.
* Promenade/bootkube/kubespray are too complex and do not fit well with our
existing architecture.
Data model impact
=================
To support applying kubernetes labels to hosts, a new label table will be
added to the sysinv DB, with the following columns:
* created_at, updated_at, deleted_at: standard columns
* id: primary key
* uuid: unique identifier
* host_id: reference to i_host table
* label: actual label
As usual, a new data migration will be added to create this table in the
sysinv DB.
REST API impact
===============
This impacts the sysinv REST API:
* The new resource /labels is added and the POST method will accept a
key/value pair as a label to be assigned to a host.
* URLS:
* /v1/labels/{host_uuid}
* Request Methods:
* POST
* JSON response example::
{"uuid": "cf415922-4242-4c58-94c7-785d69dc8971",
"host_uuid": "01f1d819-830d-4a84-a533-88fc5e7deba8",
"label_key": "prefix/name",
"label_value": "value"}
* The GET method would return all the labels assigned to a specified host.
* URLS:
* /v1/ihosts/{host_uuid}/labels
* Request Methods:
* GET
* JSON response example::
{"labels": [{"uuid": "692773cf-a2be-4b32-bc09-89a22521503d",
"host_uuid": "01f1d819-830d-4a84-a533-88fc5e7deba8",
"label_key": "prefix/name"
"label_value": "value"}]}
* The DELETE method would delete a label based on its uuid.
* URLS:
* /v1/labels/{uuid}
* Request Methods:
* DELETE
Security impact
===============
This change does not expose any new interfaces to the external world. The
kubernetes API will only be visible on the internal management network,
which is considered to be a secure network with no external access.
The kubeadm tool sets up the following security measures for the kubernetes
components:
* Certificates and keys are generated for the kube-apiserver. These
certificates are also used to provide secure access between other
kubernetes components (e.g. kubelet) and the kube-apiserver.
* Keystone tokens are used to authenticate users of the kube-apiserver.
* Bootstrap tokens are generated for each new worker node and are used to
ensure that only authorized worker nodes are allowed to join the cluster.
Other end user impact
=====================
None - this does not affect existing deployments
Performance Impact
==================
None - this does not affect existing deployments
Other deployer impact
=====================
A new --kubernetes option will be added to config_controller, which will
only be used for internal development purposes.
Developer impact
================
This change allows developers to install and configure a StarlingX system
with kubernetes master and worker nodes, in order to allow further
development efforts to support containerized services.
Upgrade impact
==============
None - this is the first StarlingX release so there are no previous releases
that we need to support upgrades from.
Implementation
==============
Assignee(s)
===========
Primary assignee:
* Bart Wensley (bartwensley)
Other contributors:
* Smith (kevin.smith.wrs)
* Chris Friesen (cbf123)
* Shoaib Nasir (snasir)
* Teresa Ho (teresaho)
Repos Impacted
==============
* stx-integ
* stx-config
* stx-nfv
Work Items
==========
Completed:
* Add packages to the load that are required to support a basic kubernetes
deployment, including kubernetes, etcd, docker, docker registry and helm
client.
* Add --kubernetes option to config_controller command.
* Integrate etcd as a new active/standby SM service, configured with puppet.
* Update SM puppet configuration to prevent configuration of the bare metal
OpenStack services, with the exception of keystone and horizon.
* Create new puppet manifest to use kubeadm tool to automatically configure
both controller hosts as kubernetes master nodes and install the calico
CNI plugin. The manifest will also use the kubeadm tool to automatically
configure compute hosts as kubernetes worker nodes.
* Update the VIM to provide activity management for kubernetes hosts.
* Create new puppet manifest to configure Helm.
* Add new inventory CLIs (system host-label-assign/list/remove) to allow
labels to be added/removed to kubernetes nodes.
* Support host-delete for master/worker nodes by using the kubernetes API to
delete all resources associated with the node.
To do:
* Allow the default NoSchedule taint to remain on controllers and use
tolerations on all pods that must run on controllers.
* Automatically add firewall rule to allow compute hosts to have external
network access through controller hosts.
* Integrate k8s-keystone-auth as authorization backend for the kube-apiserver.
Dependencies
============
This requires existing functionality from some projects that are not
currently used by StarlingX:
* docker
* etcd
* kubernetes
* helm
Testing
=======
One main thrust of the testing is to ensure the kubernetes functionality is
dormant when the --kubernetes option is not used. Existing unit tests will
cover some of this, but it is important to test full system installs (this
can be done in a virtual environment). Some of the key testcases are:
* Full install of 2 controller, 2 compute system
* Full install of 2 controller, 2 storage and 2 compute system
* Host maintenance operations (e.g. lock, unlock, swact, power cycle)
* Basic nova instance operations (e.g. stop, start, migrate, delete)
Testing for kubernetes enabled systems will also be done (can be done in a
virtual environment). Some of the key testcases are:
* Installation of 2 master, 2 worker system
* Host maintenance operations (e.g. lock, unlock, swact, power cycle)
* Deploying pods on all hosts and verifying pod networking
Documentation Impact
====================
None - this does not impact existing deployments
References
==========
None
History
=======
.. list-table:: Revisions
:header-rows: 1
* - Release Name
- Description
* - 2019.03
- Introduced