35 KiB
Install Kubernetes Platform on All-in-one Simplex
partner
Overview
Minimum hardware requirements
starlingx
Bare Metal
Virtual
The following sections describe system requirements and host setup for a workstation hosting virtual machine(s) where StarlingX will be deployed; i.e., a for each StarlingX node (controller, AIO-controller, worker or storage node).
Hardware requirements
The host system should have at least:
- Processor: x86_64 only supported architecture with BIOS enabled hardware virtualization extensions
- Cores: 8
- Memory: 32GB RAM
- Hard Disk: 500GB HDD
- Network: One network adapter with active Internet connection
Software requirements
The host system should have at least:
- A workstation computer with Ubuntu 16.04 LTS 64-bit
All other required packages will be installed by scripts in the StarlingX tools repository.
Host setup
Set up the host with the following steps:
Update OS:
apt-get update
Clone the StarlingX tools repository:
apt-get install -y git cd $HOME git clone https://opendev.org/starlingx/virtual-deployment.git
Install required packages:
cd $HOME/virtual-deployment/libvirt bash install_packages.sh apt install -y apparmor-profiles apt-get install -y ufw ufw disable ufw status
Note
On Ubuntu 16.04, if apparmor-profile modules were installed as shown in the example above, you must reboot the server to fully install the apparmor-profile modules.
partner
Installation Prerequisites
starlingx
Bare Metal
Virtual
Several pre-requisites must be completed prior to starting the installation.
Before attempting to install , ensure that you have the the host installer ISO image file.
Get the latest ISO from the StarlingX mirror. Alternately, you can get an older release ISO from here.
partner
Prepare Servers for Installation
starlingx
Bare Metal
Virtual
Note
The following commands for host, virtual environment setup, and host
power-on use KVM / virsh for virtual machine and VM management
technology. For an alternative virtualization environment, see: Install StarlingX in VirtualBox
<install_virtualbox>
.
Prepare virtual environment.
Set up the virtual platform networks for virtual deployment:
bash setup_network.sh
Prepare virtual servers.
Create the XML definitions for the virtual servers required by this configuration option. This will create the XML virtual server definition for:
- simplex-controller-0
The following command will start/virtually power on:
- The 'simplex-controller-0' virtual server
- The X-based graphical virt-manager application
bash setup_configuration.sh -c simplex -i ./bootimage.iso
If there is no X-server present errors will occur and the X-based GUI for the virt-manager application will not start. The virt-manager GUI is not absolutely required and you can safely ignore errors and continue.
partner
Install Software on Controller-0
starlingx
Bare Metal
Virtual
In the last step of aio_simplex_environ
, the controller-0 virtual server
'simplex-controller-0' was started by the setup_configuration.sh
command.
On the host, attach to the console of virtual controller-0 and select the appropriate installer menu options to start the non-interactive install of StarlingX software on controller-0.
Note
When entering the console, it is very easy to miss the first installer menu selection. Use ESC to navigate to previous menus, to ensure you are at the first installer menu.
virsh console simplex-controller-0
Make the following menu selections in the installer:
- First menu: Select 'All-in-one Controller Configuration'
- Second menu: Select 'Serial Console'
Wait for the non-interactive install of software to complete and for the server to reboot. This can take 5-10 minutes, depending on the performance of the host machine.
partner
Bootstrap system on controller-0
Login using the username / password of "sysadmin" / "sysadmin". When logging in for the first time, you will be forced to change the password.
Login: sysadmin Password: Changing password for sysadmin. (current) UNIX Password: sysadmin New Password: (repeat) New Password:
Verify and/or configure IP connectivity.
starlingx
Bare Metal
Virtual
partner
Specify user configuration overrides for the Ansible bootstrap playbook.
Ansible is used to bootstrap StarlingX on controller-0. Key files for Ansible configuration are:
/etc/ansible/hosts
-
The default Ansible inventory file. Contains a single host: localhost.
/usr/share/ansible/stx-ansible/playbooks/bootstrap.yml
-
The Ansible bootstrap playbook.
/usr/share/ansible/stx-ansible/playbooks/host_vars/bootstrap/default.yml
-
The default configuration values for the bootstrap playbook.
sysadmin home directory ($HOME)
-
The default location where Ansible looks for and imports user configuration override files for hosts. For example:
$HOME/<hostname>.yml
.
starlingx
Specify the user configuration override file for the Ansible bootstrap playbook using one of the following methods:
Note
This Ansible Overrides file for the Bootstrap Playbook ($HOME/localhost.yml) contains security sensitive information, use the
ansible-vault create $HOME/localhost.yml
command to create it. You will be prompted for a password to protect/encrypt the file. Use theansible-vault edit $HOME/localhost.yml
command if the file needs to be edited after it is created.Use a copy of the default.yml file listed above to provide your overrides.
The default.yml file lists all available parameters for bootstrap configuration with a brief description for each parameter in the file comments.
To use this method, run the
ansible-vault create $HOME/localhost.yml
command and copy the contents of thedefault.yml
file into the ansible-vault editor, and edit the configurable values as required.Create a minimal user configuration override file.
To use this method, create your override file with the
ansible-vault create $HOME/localhost.yml
command and provide the minimum required parameters for the deployment configuration as shown in the example below. Use the OAM IP SUBNET and IP ADDRESSing applicable to your deployment environment.starlingx
In either of the above options, the bootstrap playbook's default values will pull all container images required for the from Docker hub
If you have setup a private Docker registry to use for bootstrapping then you will need to add the following lines in $HOME/localhost.yml:
partner
docker_registries: quay.io: url: myprivateregistry.abc.com:9001/quay.io docker.elastic.co: url: myprivateregistry.abc.com:9001/docker.elastic.co gcr.io: url: myprivateregistry.abc.com:9001/gcr.io ghcr.io: url: myprivateregistry.abc.com:9001/ghcr.io k8s.gcr.io: url: myprivateregistry.abc.com:9001/k8s.gcr.io docker.io: url: myprivateregistry.abc.com:9001/docker.io registry.k8s.io: url: myprivateregistry.abc.com:9001/registry.k8s.io icr.io: url: myprivateregistry.abc.com:9001/icr.io defaults: type: docker username: <your_myprivateregistry.abc.com_username> password: <your_myprivateregistry.abc.com_password> # Add the CA Certificate that signed myprivateregistry.abc.com’s # certificate as a Trusted CA ssl_ca_cert: /home/sysadmin/myprivateregistry.abc.com-ca-cert.pem
See
Use a Private Docker Registry <use-private-docker-registry-r7>
for more information.starlingx
If a firewall is blocking access to Docker hub or your private registry from your StarlingX deployment, you will need to add the following lines in $HOME/localhost.yml (see
Docker Proxy Configuration <docker_proxy_config>
for more details about Docker proxy settings):partner
# Add these lines to configure Docker to use a proxy server docker_http_proxy: http://my.proxy.com:1080 docker_https_proxy: https://my.proxy.com:1443 docker_no_proxy: - 1.2.3.4
Refer to
Ansible Bootstrap Configurations <ansible_bootstrap_configs_r7>
for information on additional Ansible bootstrap configurations for advanced Ansible bootstrap scenarios.
Run the Ansible bootstrap playbook:
ansible-playbook --ask-vault-pass /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml
Wait for Ansible bootstrap playbook to complete. This can take 5-10 minutes, depending on the performance of the host machine.
Configure controller-0
The newly installed controller needs to be configured.
Acquire admin credentials:
source /etc/platform/openrc
Configure the interface of controller-0 and specify the attached network as "oam".
starlingx
Bare Metal
Virtual
OAM_IF=enp7s1 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam
partner
Configure servers for network time synchronization:
starlingx
Bare Metal
~(keystone_admin)$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
To configure instead of , see
PTP Server Configuration <ptp-server-config-index>
.Virtual
Note
In a virtual environment, this can sometimes cause Ceph clock skew alarms. Also, the virtual instances clock is synchronized with the host clock, so it is not absolutely required to configure in this step.
~(keystone_admin)$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
partner
openstack
OpenStack-specific host configuration
Important
These steps are required only if the StarlingX OpenStack application (-openstack) will be installed.
For OpenStack only: Assign OpenStack host labels to controller-0 in support of installing the -openstack manifest and helm-charts later.
starlingx
~(keystone_admin)$ system host-label-assign controller-0 openstack-control-plane=enabled ~(keystone_admin)$ system host-label-assign controller-0 openstack-compute-node=enabled ~(keystone_admin)$ system host-label-assign controller-0
Note
If you have a that supports , then you can enable it by using the following:
~(keystone_admin)$ system host-label-assign controller-0 sriov=enabled
partner
For OpenStack only: Due to the additional OpenStack services running on the controller platform cores, additional platform cores may be required.
starlingx
Bare Metal
Virtual
The being used for hosts only have 4 cores; 2 for platform and 2 for . There are no additional cores available for platform in this scenario.
partner
Due to the additional OpenStack services' containers running on the controller host, the size of the Docker filesystem needs to be increased from the default size of 30G to 60G.
# check existing size of docker fs system host-fs-list controller-0 # check available space (Avail Size (GiB)) in cgts-vg LVG where docker fs is located system host-lvg-list controller-0 # if existing docker fs size + cgts-vg available space is less than # 60G, you will need to add a new disk to cgts-vg. # Get device path of BOOT DISK system host-show controller-0 | fgrep rootfs # Get UUID of ROOT DISK by listing disks system host-disk-list controller-0 # Add new disk to 'cgts-vg' local volume group system host-pv-add controller-0 cgts-vg <DISK_UUID> sleep 10 # wait for disk to be added # Confirm the available space and increased number of physical # volumes added to the cgts-vg colume group system host-lvg-list controller-0 # Increase docker filesystem to 60G system host-fs-modify controller-0 docker=60
For OpenStack only: Configure the system setting for the vSwitch.
starlingx
Bare Metal
StarlingX has (kernel-based) vSwitch configured as default:
- Runs in a container; defined within the helm charts of -openstack manifest.
- Shares the core(s) assigned to the platform.
If you require better performance, ( with the Data Plane Development Kit, which is supported only on bare metal hardware) should be used:
- Runs directly on the host (it is not containerized). Requires that at least 1 core be assigned/dedicated to the vSwitch function.
To deploy the default containerized :
~(keystone_admin)$ system modify --vswitch_type none
This does not run any vSwitch directly on the host, instead, it uses the containerized defined in the helm charts of -openstack manifest.
Virtual
The default vSwitch is the containerized that is packaged with the
stx-openstack
manifest/helm-charts. provides the option to use OVS-DPDK on the host, however, in the virtual environment OVS-DPDK is not supported, only is supported. Therefore, simply use the default vSwitch here.partner
For OpenStack only: Add an instances filesystem or set up a disk based nova-local volume group, which is needed for -openstack nova ephemeral disks.
starlingx
Bare Metal
Virtual
Set up an "instances" filesystem, which is needed for stx-openstack nova ephemeral disks.
~(keystone_admin)$ export NODE=controller-0 ~(keystone_admin)$ system host-fs-add ${NODE} instances=34
partner
For OpenStack only: Configure data interfaces for controller-0. Data class interfaces are vSwitch interfaces used by vSwitch to provide virtio vNIC connectivity to OpenStack Neutron Tenant Networks on the underlying assigned Data Network.
Important
A compute-labeled -controller host MUST have at least one Data class interface.
Configure the data interfaces for controller-0.
starlingx
Bare Metal
Virtual
~(keystone_admin)$ DATA0IF=eth1000 ~(keystone_admin)$ DATA1IF=eth1001 ~(keystone_admin)$ export NODE=controller-0 ~(keystone_admin)$ PHYSNET0='physnet0' ~(keystone_admin)$ PHYSNET1='physnet1' ~(keystone_admin)$ SPL=/tmp/tmp-system-port-list ~(keystone_admin)$ SPIL=/tmp/tmp-system-host-if-list ~(keystone_admin)$ system host-port-list ${NODE} --nowrap > ${SPL} ~(keystone_admin)$ system host-if-list -a ${NODE} --nowrap > ${SPIL} ~(keystone_admin)$ DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}') ~(keystone_admin)$ DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}') ~(keystone_admin)$ DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}') ~(keystone_admin)$ DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}') ~(keystone_admin)$ DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}') ~(keystone_admin)$ DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}') ~(keystone_admin)$ DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}') ~(keystone_admin)$ DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}') ~(keystone_admin)$ system datanetwork-add ${PHYSNET0} vlan ~(keystone_admin)$ system datanetwork-add ${PHYSNET1} vlan ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID} ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1
partner
Optionally Configure PCI-SRIOV Interfaces
Optionally, configure -SRIOV interfaces for controller-0.
This step is optional for Kubernetes. Do this step if using network attachments in hosted application containers.
openstack
This step is optional for OpenStack. Do this step if using vNICs in hosted application VMs. Note that -SRIOV interfaces can have the same Data Networks assigned to them as vswitch data interfaces.
Configure the -SRIOV interfaces for controller-0.
~(keystone_admin)$ export NODE=controller-0 # List inventoried host’s ports and identify ports to be used as ‘pci-sriov’ interfaces, # based on displayed linux port name, pci address and device type. ~(keystone_admin)$ system host-port-list ${NODE} # List host’s auto-configured ‘ethernet’ interfaces, # find the interfaces corresponding to the ports identified in previous step, and # take note of their UUID ~(keystone_admin)$ system host-if-list -a ${NODE} # Modify configuration for these interfaces # Configuring them as ‘pci-sriov’ class interfaces, MTU of 1500 and named sriov# ~(keystone_admin)$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov ${NODE} <sriov0-if-uuid> -N <num_vfs> ~(keystone_admin)$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov ${NODE} <sriov1-if-uuid> -N <num_vfs> # If not already created, create Data Networks that the 'pci-sriov' interfaces will # be connected to ~(keystone_admin)$ DATANET0='datanet0' ~(keystone_admin)$ DATANET1='datanet1' ~(keystone_admin)$ system datanetwork-add ${DATANET0} vlan ~(keystone_admin)$ system datanetwork-add ${DATANET1} vlan # Assign Data Networks to PCI-SRIOV Interfaces ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov0-if-uuid> ${DATANET0} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov1-if-uuid> ${DATANET1}
For Kubernetes Only: To enable using network attachments for the above interfaces in Kubernetes hosted application containers:
starlingx
Bare Metal
Virtual
Configure the Kubernetes device plugin.
~(keystone_admin)$ system host-label-assign controller-0 sriovdp=enabled
partner
If required, initialize a Ceph-based Persistent Storage Backend
A persistent storage backend is required if your application requires .
openstack
Important
The StarlingX OpenStack application requires .
starlingx
There are two options for persistent storage backend: the host-based Ceph solution and the Rook container-based Ceph solution.
For host-based Ceph:
Add host-based Ceph backend:
~(keystone_admin)$ system storage-backend-add ceph --confirmed
Add an on controller-0 for host-based Ceph:
# List host’s disks and identify disks you want to use for CEPH OSDs, taking note of their UUID # By default, /dev/sda is being used as system disk and can not be used for OSD. ~(keystone_admin)$ system host-disk-list controller-0 # Add disk as an OSD storage ~(keystone_admin)$ system host-stor-add controller-0 osd <disk-uuid> # List OSD storage devices ~(keystone_admin)$ system host-stor-list controller-0
starlingx
For Rook container-based Ceph:
Add Rook container-based backend:
~(keystone_admin)$ system storage-backend-add ceph-rook --confirmed
Assign Rook host labels to controller-0 in support of installing the rook-ceph-apps manifest/helm-charts later:
~(keystone_admin)$ system host-label-assign controller-0 ceph-mon-placement=enabled ~(keystone_admin)$ system host-label-assign controller-0 ceph-mgr-placement=enabled
Unlock controller-0
Unlock controller-0 to bring it into service:
~(keystone_admin)$ system host-unlock controller-0
Controller-0 will reboot in order to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine.
starlingx
If using Rook container-based Ceph, finish configuring the ceph-rook Persistent Storage Backend
On controller-0:
Wait for application rook-ceph-apps to be uploaded
$ source /etc/platform/openrc ~(keystone_admin)$ system application-list +---------------------+---------+-------------------------------+---------------+----------+-----------+ | application | version | manifest name | manifest file | status | progress | +---------------------+---------+-------------------------------+---------------+----------+-----------+ | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed | | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed | | rook-ceph-apps | 1.0-1 | rook-ceph-manifest | manifest.yaml | uploaded | completed | +---------------------+---------+-------------------------------+---------------+----------+-----------+
Configure rook to use /dev/sdb disk on controller-0 as a ceph .
~(keystone_admin)$ system host-disk-wipe -s --confirm controller-0 /dev/sdb
values.yaml
for rook-ceph-apps.cluster: storage: nodes: - name: controller-0 devices: - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0
~(keystone_admin)$ system helm-override-update rook-ceph-apps rook-ceph kube-system --values values.yaml
Apply the rook-ceph-apps application.
~(keystone_admin)$ system application-apply rook-ceph-apps
Wait for pod to be ready.
~(keystone_admin)$ kubectl get pods -n kube-system rook--ceph-crashcollector-controller-0-764c7f9c8-bh5c7 1/1 Running 0 62m rook--ceph-mgr-a-69df96f57-9l28p 1/1 Running 0 63m rook--ceph-mon-a-55fff49dcf-ljfnx 1/1 Running 0 63m rook--ceph-operator-77b64588c5-nlsf2 1/1 Running 0 66m rook--ceph-osd-0-7d5785889f-4rgmb 1/1 Running 0 62m rook--ceph-osd-prepare-controller-0-cmwt5 0/1 Completed 0 2m14s rook--ceph-tools-5778d7f6c-22tms 1/1 Running 0 64m rook--discover-kmv6c 1/1 Running 0 65m
starlingx
Next steps
partner
Complete system configuration by reviewing procedures in:
index-security-kub-81153c1254c3
index-sysconf-kub-78f0e1e9ca5a
index-admintasks-kub-ebc55fefc368