Spec for kubernetes platform support
Submitting the following spec for review: containerization-2002843-kubernetes-platform-support.rst Change-Id: I86d28e38fe36572d5909eb2a36faa4aea18c6df8 Signed-off-by: Bart Wensley <barton.wensley@windriver.com>
This commit is contained in:
parent
f7c04656d8
commit
65f6ef6324
|
@ -0,0 +1,320 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License. http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
===========================
|
||||
Kubernetes Platform Support
|
||||
===========================
|
||||
|
||||
Storyboard: https://storyboard.openstack.org/#!/story/2002843
|
||||
|
||||
This story will implement the initial items to integrate kubernetes on
|
||||
StarlingX. The StarlingX kubernetes cluster will initially support:
|
||||
|
||||
* Docker runtime
|
||||
* Calico CNI plugin
|
||||
* Ceph as persistent storage backend
|
||||
* Helm as the package manager
|
||||
* Local Docker Image Registry
|
||||
* 2 node HA solution
|
||||
* Node labeling integration with inventory subsystem
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
In order to support containerized applications on the StarlingX platform,
|
||||
we need to package and deploy kubernetes and associated components, such
|
||||
as docker, calico and helm.
|
||||
|
||||
This is a first step in a larger effort. Please see the following for
|
||||
details of the end goal:
|
||||
https://wiki.openstack.org/wiki/Containerizing_StarlingX_Infrastructure
|
||||
|
||||
Use Cases
|
||||
=========
|
||||
|
||||
Developers need the ability to install and configure a StarlingX system
|
||||
with kubernetes master and worker nodes, in order to allow further
|
||||
development efforts to support containerized services.
|
||||
|
||||
Developers/testers/users need the ability to install and configure a
|
||||
StarlingX system without any kubernetes functionality, in order to avoid
|
||||
any impacts to non-kubernetes related development and testing activities.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
The packages required to support a basic kubernetes deployment will be added
|
||||
to the load. This includes packages related to:
|
||||
|
||||
* kubernetes
|
||||
* etcd
|
||||
* docker and docker registry
|
||||
* helm client
|
||||
|
||||
A --kubernetes option will be added to config_controller. When this option is
|
||||
not used, the system will be configured without any kubernetes functionality.
|
||||
|
||||
When the --kubernetes option is used, kubernetes will be configured. This
|
||||
includes the following:
|
||||
|
||||
* The etcd will be started and will be managed by stx-ha Service Management
|
||||
(SM) in an active/standby configuration.
|
||||
* The bare metal OpenStack services will not be started and will not appear
|
||||
in SM, with the exception of keystone, which will continue to run on
|
||||
bare metal to provide authentication for platform services (e.g. System
|
||||
Inventory (sysinv), patching, stx-nfv Virtual Infrastructure Manager (VIM)).
|
||||
* The kubeadm tool will be used to automatically configure both controller
|
||||
hosts as kubernetes master nodes. Kubernetes will initially use the existing
|
||||
management network and floating management IP for the kube-apiserver.
|
||||
* Kubeadm will download any required images from the usual external
|
||||
image registries. An iptables NAT rule will be added on controller hosts to
|
||||
allow compute hosts to access the external image registries through the
|
||||
controller's OAM interface.
|
||||
* The calico CNI plugin will be installed to provide pod networking.
|
||||
* The k8s-keystone-auth will be configured as the authorization backend for
|
||||
the kube-apiserver.
|
||||
* Helm will be configured to manage kubernetes applications.
|
||||
* When a compute host is installed, the kubeadm tool will be used to
|
||||
automatically configure it as a kubernetes worker node.
|
||||
* The VIM will provide activity management for kubernetes hosts. When hosts
|
||||
are removed from service (by locking them or because they have failed),
|
||||
the VIM will prevent dynamic pods from running on these hosts by applying
|
||||
a NoExecute taint to the assocatied kubernetes node.
|
||||
* New inventory CLIs (system host-label-assign/list/remove) will be provided
|
||||
to allow labels to be added/removed to kubernetes nodes.
|
||||
* When a host acting as a master/worker node is deleted, use the kubernetes
|
||||
API to delete all resources associated with the node.
|
||||
|
||||
Alternatives
|
||||
============
|
||||
|
||||
There are many alternatives to kubeadm that can be used to install a
|
||||
kubernetes cluster (e.g. bootkube, kubespray, puppetlabs-kubernetes,
|
||||
promenade). We chose kubeadm because:
|
||||
|
||||
* It has the most community support and is the most up-to-date with
|
||||
kubernetes features.
|
||||
* It should be easy to integrate with our existing installation/commissioning
|
||||
architecture. Will get us up and running and allow us to understand any
|
||||
limitations better.
|
||||
* Promenade/bootkube/kubespray are too complex and do not fit well with our
|
||||
existing architecture.
|
||||
|
||||
Data model impact
|
||||
=================
|
||||
|
||||
To support applying kubernetes labels to hosts, a new label table will be
|
||||
added to the sysinv DB, with the following columns:
|
||||
|
||||
* created_at, updated_at, deleted_at: standard columns
|
||||
* id: primary key
|
||||
* uuid: unique identifier
|
||||
* host_id: reference to i_host table
|
||||
* label: actual label
|
||||
|
||||
As usual, a new data migration will be added to create this table in the
|
||||
sysinv DB.
|
||||
|
||||
REST API impact
|
||||
===============
|
||||
|
||||
This impacts the sysinv REST API:
|
||||
|
||||
* The new resource /labels is added and the POST method will accept a
|
||||
key/value pair as a label to be assigned to a host.
|
||||
|
||||
* URLS:
|
||||
|
||||
* /v1/labels/{host_uuid}
|
||||
|
||||
* Request Methods:
|
||||
|
||||
* POST
|
||||
|
||||
* JSON response example::
|
||||
|
||||
{"uuid": "cf415922-4242-4c58-94c7-785d69dc8971",
|
||||
"host_uuid": "01f1d819-830d-4a84-a533-88fc5e7deba8",
|
||||
"label_key": "prefix/name",
|
||||
"label_value": "value"}
|
||||
|
||||
* The GET method would return all the labels assigned to a specified host.
|
||||
|
||||
* URLS:
|
||||
|
||||
* /v1/ihosts/{host_uuid}/labels
|
||||
|
||||
* Request Methods:
|
||||
|
||||
* GET
|
||||
|
||||
* JSON response example::
|
||||
|
||||
{"labels": [{"uuid": "692773cf-a2be-4b32-bc09-89a22521503d",
|
||||
"host_uuid": "01f1d819-830d-4a84-a533-88fc5e7deba8",
|
||||
"label_key": "prefix/name"
|
||||
"label_value": "value"}]}
|
||||
|
||||
* The DELETE method would delete a label based on its uuid.
|
||||
|
||||
* URLS:
|
||||
|
||||
* /v1/labels/{uuid}
|
||||
|
||||
* Request Methods:
|
||||
|
||||
* DELETE
|
||||
|
||||
Security impact
|
||||
===============
|
||||
|
||||
This change does not expose any new interfaces to the external world. The
|
||||
kubernetes API will only be visible on the internal management network,
|
||||
which is considered to be a secure network with no external access.
|
||||
|
||||
The kubeadm tool sets up the following security measures for the kubernetes
|
||||
components:
|
||||
|
||||
* Certificates and keys are generated for the kube-apiserver. These
|
||||
certificates are also used to provide secure access between other
|
||||
kubernetes components (e.g. kubelet) and the kube-apiserver.
|
||||
* Keystone tokens are used to authenticate users of the kube-apiserver.
|
||||
* Bootstrap tokens are generated for each new worker node and are used to
|
||||
ensure that only authorized worker nodes are allowed to join the cluster.
|
||||
|
||||
Other end user impact
|
||||
=====================
|
||||
|
||||
None - this does not affect existing deployments
|
||||
|
||||
Performance Impact
|
||||
==================
|
||||
|
||||
None - this does not affect existing deployments
|
||||
|
||||
Other deployer impact
|
||||
=====================
|
||||
|
||||
A new --kubernetes option will be added to config_controller, which will
|
||||
only be used for internal development purposes.
|
||||
|
||||
Developer impact
|
||||
================
|
||||
|
||||
This change allows developers to install and configure a StarlingX system
|
||||
with kubernetes master and worker nodes, in order to allow further
|
||||
development efforts to support containerized services.
|
||||
|
||||
Upgrade impact
|
||||
==============
|
||||
|
||||
None - this is the first StarlingX release so there are no previous releases
|
||||
that we need to support upgrades from.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
===========
|
||||
|
||||
Primary assignee:
|
||||
|
||||
* Bart Wensley (bartwensley)
|
||||
|
||||
Other contributors:
|
||||
|
||||
* Smith (kevin.smith.wrs)
|
||||
* Chris Friesen (cbf123)
|
||||
* Shoaib Nasir (snasir)
|
||||
* Teresa Ho (teresaho)
|
||||
|
||||
Repos Impacted
|
||||
==============
|
||||
|
||||
* stx-integ
|
||||
* stx-config
|
||||
* stx-nfv
|
||||
|
||||
Work Items
|
||||
==========
|
||||
|
||||
Completed:
|
||||
|
||||
* Add packages to the load that are required to support a basic kubernetes
|
||||
deployment, including kubernetes, etcd, docker, docker registry and helm
|
||||
client.
|
||||
* Add --kubernetes option to config_controller command.
|
||||
* Integrate etcd as a new active/standby SM service, configured with puppet.
|
||||
* Update SM puppet configuration to prevent configuration of the bare metal
|
||||
OpenStack services, with the exception of keystone and horizon.
|
||||
* Create new puppet manifest to use kubeadm tool to automatically configure
|
||||
both controller hosts as kubernetes master nodes and install the calico
|
||||
CNI plugin. The manifest will also use the kubeadm tool to automatically
|
||||
configure compute hosts as kubernetes worker nodes.
|
||||
* Update the VIM to provide activity management for kubernetes hosts.
|
||||
* Create new puppet manifest to configure Helm.
|
||||
* Add new inventory CLIs (system host-label-assign/list/remove) to allow
|
||||
labels to be added/removed to kubernetes nodes.
|
||||
* Support host-delete for master/worker nodes by using the kubernetes API to
|
||||
delete all resources associated with the node.
|
||||
|
||||
To do:
|
||||
|
||||
* Allow the default NoSchedule taint to remain on controllers and use
|
||||
tolerations on all pods that must run on controllers.
|
||||
* Automatically add firewall rule to allow compute hosts to have external
|
||||
network access through controller hosts.
|
||||
* Integrate k8s-keystone-auth as authorization backend for the kube-apiserver.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
This requires existing functionality from some projects that are not
|
||||
currently used by StarlingX:
|
||||
|
||||
* docker
|
||||
* etcd
|
||||
* kubernetes
|
||||
* helm
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
One main thrust of the testing is to ensure the kubernetes functionality is
|
||||
dormant when the --kubernetes option is not used. Existing unit tests will
|
||||
cover some of this, but it is important to test full system installs (this
|
||||
can be done in a virtual environment). Some of the key testcases are:
|
||||
|
||||
* Full install of 2 controller, 2 compute system
|
||||
* Full install of 2 controller, 2 storage and 2 compute system
|
||||
* Host maintenance operations (e.g. lock, unlock, swact, power cycle)
|
||||
* Basic nova instance operations (e.g. stop, start, migrate, delete)
|
||||
|
||||
Testing for kubernetes enabled systems will also be done (can be done in a
|
||||
virtual environment). Some of the key testcases are:
|
||||
|
||||
* Installation of 2 master, 2 worker system
|
||||
* Host maintenance operations (e.g. lock, unlock, swact, power cycle)
|
||||
* Deploying pods on all hosts and verifying pod networking
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
None - this does not impact existing deployments
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
None
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - 2019.03
|
||||
- Introduced
|
Loading…
Reference in New Issue