From 2462f3015a5601a9398400beef2fadc56cbaa89f Mon Sep 17 00:00:00 2001 From: Tao Liu Date: Sat, 12 Jan 2019 23:21:53 -0500 Subject: [PATCH 1/4] Clean up the stale files The keystone-api pod gets stuck in CrashLoopBackOff on AIO-SX lock/unlock attempt. When Kubernetes decides to kill the keystone-api pod due to readiness probe failure or other reasons, it calls the preStop hook immediately before the container is terminated. This hook starts a graceful shutdown process which includes removing pid, shared memory segment and wsgi sock files. If the container is not terminated within the grace period, a SIGKILL is sent, and the container is forced to shut down. When the container was forced to terminate without clean up, the stale files were left behind. On the restart, the application detected the file existed, and treated it as configuration failure, hence the exit. As a result, the pod went into a crash loop. This update removes any stale files when the pod starts. Story: 2004520 Task: 28392 Change-Id: I613a0db674de9578b3f9d1fa781a1612d9caf214 Signed-off-by: Tao Liu --- openstack/openstack-helm/centos/build_srpm.data | 2 +- .../Remove-stale-Apache2-service-pids-when-a-POD-starts.patch | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/openstack/openstack-helm/centos/build_srpm.data b/openstack/openstack-helm/centos/build_srpm.data index da17aaf7..2bcbdedb 100644 --- a/openstack/openstack-helm/centos/build_srpm.data +++ b/openstack/openstack-helm/centos/build_srpm.data @@ -5,4 +5,4 @@ TAR="$TAR_NAME-$SHA.tar.gz" COPY_LIST="${CGCS_BASE}/downloads/$TAR $PKG_BASE/files/* " -TIS_PATCH_VER=6 +TIS_PATCH_VER=7 diff --git a/openstack/openstack-helm/files/Remove-stale-Apache2-service-pids-when-a-POD-starts.patch b/openstack/openstack-helm/files/Remove-stale-Apache2-service-pids-when-a-POD-starts.patch index 78814baa..29b4c913 100644 --- a/openstack/openstack-helm/files/Remove-stale-Apache2-service-pids-when-a-POD-starts.patch +++ b/openstack/openstack-helm/files/Remove-stale-Apache2-service-pids-when-a-POD-starts.patch @@ -56,8 +56,8 @@ index 217d942..a5950a4 100644 source /etc/apache2/envvars fi -+ # Get rid of stale pid file if present. -+ rm -f /var/run/apache2/*.pid ++ # Get rid of stale pid, shared memory segment and wsgi sock files if present. ++ rm -f /var/run/apache2/* + # Start Apache2 exec apache2 -DFOREGROUND From 9e649fabd82b062fe6fdea3e5b0e93385d48e361 Mon Sep 17 00:00:00 2001 From: Don Penney Date: Fri, 18 Jan 2019 16:03:45 -0500 Subject: [PATCH 2/4] Add e2fsprogs to stx-nova package list This update adds e2fsprogs to the pike and master docker image build directives files for stx-nova, to provide the mkfs utilities required for ephemeral storage support. Change-Id: If9b901696169d7d157a37d6b96f7b8c4db0a24a5 Closes-Bug: 1812432 Signed-off-by: Don Penney --- openstack/python-nova/centos/stx-nova.master_docker_image | 2 +- openstack/python-nova/centos/stx-nova.pike_docker_image | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/openstack/python-nova/centos/stx-nova.master_docker_image b/openstack/python-nova/centos/stx-nova.master_docker_image index 090ba46d..d73fcc0e 100644 --- a/openstack/python-nova/centos/stx-nova.master_docker_image +++ b/openstack/python-nova/centos/stx-nova.master_docker_image @@ -3,7 +3,7 @@ LABEL=stx-nova PROJECT=nova PROJECT_REPO=https://github.com/openstack/nova.git PIP_PACKAGES="pycrypto httplib2 pylint" -DIST_PACKAGES="openssh-clients openssh-server libvirt " +DIST_PACKAGES="openssh-clients openssh-server libvirt e2fsprogs" PROFILES="fluent nova ceph linuxbridge openvswitch configdrive qemu apache" CUSTOMIZATION="yum install -y openssh-clients" diff --git a/openstack/python-nova/centos/stx-nova.pike_docker_image b/openstack/python-nova/centos/stx-nova.pike_docker_image index 265c32de..cb549b98 100644 --- a/openstack/python-nova/centos/stx-nova.pike_docker_image +++ b/openstack/python-nova/centos/stx-nova.pike_docker_image @@ -3,7 +3,7 @@ LABEL=stx-nova PROJECT=nova PROJECT_REPO=https://github.com/starlingx-staging/stx-nova.git PIP_PACKAGES="pycrypto tsconfig cgtsclient httplib2 pylint" -DIST_PACKAGES="openssh-clients openssh-server libvirt pam-config" +DIST_PACKAGES="openssh-clients openssh-server libvirt pam-config e2fsprogs" PROFILES="fluent nova ceph linuxbridge openvswitch configdrive qemu apache" From 6e74844f720eee4bd6d7b4067d3b6db3b40caa49 Mon Sep 17 00:00:00 2001 From: Gerry Kopec Date: Wed, 9 Jan 2019 20:52:05 -0500 Subject: [PATCH 3/4] Update helm charts to support cold migration To enable cold migration, need to update nova charts in openstack-helm and helm-toolkit chart in openstack-helm-infra. These changes build on existing upstream components which attempt to add a second container to the nova-compute pod which creates a sshd process listening on port 8022. Nova chart changes include: - Fix bug in ssh-config mapping so config file is generated properly in /root/.ssh/config in nova-compute container. - Move private key from sshd container to nova-compute container. - Map private and public ssh keys to new configmap-ssh which will default to acceptable file permissions (400) for ssh. Keys will be provided in overrides. - Add additional config to /etc/ssh/sshd_config to allow passwordless root logins over appropriate subnet passed in from overrides. This is the same as what is done in nova puppet currently. - Remove chmods from sshd bash script as they are failing. Function is replaced by configmap-ssh. To enable cold migration in nova helm chart, we need to allow multiple containers within the same daemonset pod. This requires a patch to the helm-toolkit _daemonset_overrides template to remove upstream restriction. This issue is tracked upstream by storyboard 2003876. These changes should be upstreamed but may require further refinement. Story: 2003909 Task: 28927 Change-Id: Id789ba051cec019e8b7564c713cf1b5296ecf9f6 Signed-off-by: Gerry Kopec --- .../centos/build_srpm.data | 2 +- .../centos/openstack-helm-infra.spec | 2 + ...ultiple-containers-per-daemonset-pod.patch | 35 ++++ .../openstack-helm/centos/build_srpm.data | 2 +- .../openstack-helm/centos/openstack-helm.spec | 2 + ...le-cold-migration-in-nova-helm-chart.patch | 174 ++++++++++++++++++ 6 files changed, 215 insertions(+), 2 deletions(-) create mode 100644 openstack/openstack-helm-infra/files/0004-Allow-multiple-containers-per-daemonset-pod.patch create mode 100644 openstack/openstack-helm/files/0006-Enable-cold-migration-in-nova-helm-chart.patch diff --git a/openstack/openstack-helm-infra/centos/build_srpm.data b/openstack/openstack-helm-infra/centos/build_srpm.data index ad6f9579..0fbc4ee3 100644 --- a/openstack/openstack-helm-infra/centos/build_srpm.data +++ b/openstack/openstack-helm-infra/centos/build_srpm.data @@ -5,4 +5,4 @@ TAR="$TAR_NAME-$SHA.tar.gz" COPY_LIST="${CGCS_BASE}/downloads/$TAR $PKG_BASE/files/*" -TIS_PATCH_VER=5 +TIS_PATCH_VER=6 diff --git a/openstack/openstack-helm-infra/centos/openstack-helm-infra.spec b/openstack/openstack-helm-infra/centos/openstack-helm-infra.spec index 70fe4584..96ca1c66 100644 --- a/openstack/openstack-helm-infra/centos/openstack-helm-infra.spec +++ b/openstack/openstack-helm-infra/centos/openstack-helm-infra.spec @@ -18,6 +18,7 @@ BuildArch: noarch Patch01: 0001-gnocchi-chart-updates.patch Patch02: Mariadb-Support-adoption-of-running-single-node-mari.patch Patch03: Mariadb-Share-container-PID-namespaces-under-docker.patch +Patch04: 0004-Allow-multiple-containers-per-daemonset-pod.patch BuildRequires: helm @@ -29,6 +30,7 @@ Openstack Helm Infra charts %patch01 -p1 %patch02 -p1 %patch03 -p1 +%patch04 -p1 %build # initialize helm and build the toolkit diff --git a/openstack/openstack-helm-infra/files/0004-Allow-multiple-containers-per-daemonset-pod.patch b/openstack/openstack-helm-infra/files/0004-Allow-multiple-containers-per-daemonset-pod.patch new file mode 100644 index 00000000..2dac2b8f --- /dev/null +++ b/openstack/openstack-helm-infra/files/0004-Allow-multiple-containers-per-daemonset-pod.patch @@ -0,0 +1,35 @@ +From 26844aac43f76afc65ed907fc94ab83ca93c86ae Mon Sep 17 00:00:00 2001 +From: Gerry Kopec +Date: Wed, 9 Jan 2019 20:11:33 -0500 +Subject: [PATCH] Allow multiple containers per daemonset pod + +Remove code that restricted daemonset pods to single containers. +Container names will default to name from helm chart template without +hostname and sha though the pod will still have them. + +May require further refinement before this can be upstreamed. +--- + helm-toolkit/templates/utils/_daemonset_overrides.tpl | 7 ------- + 1 file changed, 7 deletions(-) + +diff --git a/helm-toolkit/templates/utils/_daemonset_overrides.tpl b/helm-toolkit/templates/utils/_daemonset_overrides.tpl +index 8ba2241..b960a84 100644 +--- a/helm-toolkit/templates/utils/_daemonset_overrides.tpl ++++ b/helm-toolkit/templates/utils/_daemonset_overrides.tpl +@@ -217,13 +217,6 @@ limitations under the License. + {{- if not $context.Values.__daemonset_yaml.metadata.name }}{{- $_ := set $context.Values.__daemonset_yaml.metadata "name" dict }}{{- end }} + {{- $_ := set $context.Values.__daemonset_yaml.metadata "name" $current_dict.dns_1123_name }} + +- {{/* set container name +- assume not more than one container is defined */}} +- {{- $container := first $context.Values.__daemonset_yaml.spec.template.spec.containers }} +- {{- $_ := set $container "name" $current_dict.dns_1123_name }} +- {{- $cont_list := list $container }} +- {{- $_ := set $context.Values.__daemonset_yaml.spec.template.spec "containers" $cont_list }} +- + {{/* cross-reference configmap name to container volume definitions */}} + {{- $_ := set $context.Values "__volume_list" list }} + {{- range $current_volume := $context.Values.__daemonset_yaml.spec.template.spec.volumes }} +-- +1.8.3.1 + diff --git a/openstack/openstack-helm/centos/build_srpm.data b/openstack/openstack-helm/centos/build_srpm.data index 2bcbdedb..02fffd4f 100644 --- a/openstack/openstack-helm/centos/build_srpm.data +++ b/openstack/openstack-helm/centos/build_srpm.data @@ -5,4 +5,4 @@ TAR="$TAR_NAME-$SHA.tar.gz" COPY_LIST="${CGCS_BASE}/downloads/$TAR $PKG_BASE/files/* " -TIS_PATCH_VER=7 +TIS_PATCH_VER=8 diff --git a/openstack/openstack-helm/centos/openstack-helm.spec b/openstack/openstack-helm/centos/openstack-helm.spec index 95b6d0c5..30ec6ae1 100644 --- a/openstack/openstack-helm/centos/openstack-helm.spec +++ b/openstack/openstack-helm/centos/openstack-helm.spec @@ -23,6 +23,7 @@ Patch02: 0002-Add-Aodh-Chart.patch Patch03: 0003-Add-Panko-Chart.patch Patch04: Remove-stale-Apache2-service-pids-when-a-POD-starts.patch Patch05: 0005-Add-heat-purge-deleted-cron-job.patch +Patch06: 0006-Enable-cold-migration-in-nova-helm-chart.patch BuildRequires: helm BuildRequires: openstack-helm-infra @@ -38,6 +39,7 @@ Openstack Helm charts %patch03 -p1 %patch04 -p1 %patch05 -p1 +%patch06 -p1 %build # initialize helm and build the toolkit diff --git a/openstack/openstack-helm/files/0006-Enable-cold-migration-in-nova-helm-chart.patch b/openstack/openstack-helm/files/0006-Enable-cold-migration-in-nova-helm-chart.patch new file mode 100644 index 00000000..29227af0 --- /dev/null +++ b/openstack/openstack-helm/files/0006-Enable-cold-migration-in-nova-helm-chart.patch @@ -0,0 +1,174 @@ +From 7760815c98231ffd431f053f8fac35902f420118 Mon Sep 17 00:00:00 2001 +From: Gerry Kopec +Date: Thu, 10 Jan 2019 00:12:21 -0500 +Subject: [PATCH] Enable cold migration in nova helm chart + +- Move private key from sshd container to nova-compute container. +- Map private and public keys to configmap-ssh which will default to + correct file permissions. +- Add additional config to /etc/ssh/sshd_config to allow passwordless + root logins over appropriate subnet passed in from overrides. +- Remove chmods from sshd bash script as they are failing. + +Depends on helm-toolkit supporting multiple containers per pod. +--- + nova/templates/bin/_ssh-start.sh.tpl | 19 ++++++++++++++++--- + nova/templates/configmap-etc.yaml | 4 ++-- + nova/templates/configmap-ssh.yaml | 35 +++++++++++++++++++++++++++++++++++ + nova/templates/daemonset-compute.yaml | 14 +++++++++----- + nova/values.yaml | 5 +++++ + 5 files changed, 67 insertions(+), 10 deletions(-) + create mode 100755 nova/templates/configmap-ssh.yaml + +diff --git a/nova/templates/bin/_ssh-start.sh.tpl b/nova/templates/bin/_ssh-start.sh.tpl +index 1c10cb0..158090b 100644 +--- a/nova/templates/bin/_ssh-start.sh.tpl ++++ b/nova/templates/bin/_ssh-start.sh.tpl +@@ -33,8 +33,21 @@ if [[ $(stat -c %U:%G ~nova/.ssh) != "nova:nova" ]]; then + chown nova: ~nova/.ssh + fi + +-chmod 0600 ~root/.ssh/authorized_keys +-chmod 0600 ~root/.ssh/id_rsa +-chmod 0600 ~root/.ssh/id_rsa.pub ++{{- if .Values.network.sshd.enabled }} ++subnet_address="{{- .Values.network.sshd.from_subnet -}}" ++cat > /tmp/sshd_config_extend <> /etc/ssh/sshd_config ++rm /tmp/sshd_config_extend ++{{- end }} + + exec /usr/sbin/sshd -D -e -o Port=$SSH_PORT +diff --git a/nova/templates/configmap-etc.yaml b/nova/templates/configmap-etc.yaml +index 55aa311..0d1e7a5 100644 +--- a/nova/templates/configmap-etc.yaml ++++ b/nova/templates/configmap-etc.yaml +@@ -232,8 +232,8 @@ data: + logging.conf: {{ include "helm-toolkit.utils.to_oslo_conf" .Values.conf.logging | b64enc }} + nova-ironic.conf: {{ include "helm-toolkit.utils.to_oslo_conf" .Values.conf.nova_ironic | b64enc }} + {{- include "helm-toolkit.snippets.values_template_renderer" (dict "envAll" $envAll "template" .Values.conf.wsgi_placement "key" "wsgi-nova-placement.conf" "format" "Secret" ) | indent 2 }} +-# FIXME(portdirect): why is this file suffixed .sh? +-{{- include "helm-toolkit.snippets.values_template_renderer" (dict "envAll" $envAll "template" .Values.conf.ssh "key" "ssh-config.sh" "format" "Secret" ) | indent 2 }} ++{{- include "helm-toolkit.snippets.values_template_renderer" (dict "envAll" $envAll "template" .Values.conf.ssh "key" "ssh-config" "format" "Secret" ) | indent 2 }} ++ + {{- end }} + {{- end }} + {{- if .Values.manifests.configmap_etc }} +diff --git a/nova/templates/configmap-ssh.yaml b/nova/templates/configmap-ssh.yaml +new file mode 100755 +index 0000000..bab8e33 +--- /dev/null ++++ b/nova/templates/configmap-ssh.yaml +@@ -0,0 +1,35 @@ ++{{/* ++Copyright 2019 The Openstack-Helm Authors. ++ ++Licensed under the Apache License, Version 2.0 (the "License"); ++you may not use this file except in compliance with the License. ++You may obtain a copy of the License at ++ ++ http://www.apache.org/licenses/LICENSE-2.0 ++ ++Unless required by applicable law or agreed to in writing, software ++distributed under the License is distributed on an "AS IS" BASIS, ++WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ++See the License for the specific language governing permissions and ++limitations under the License. ++*/}} ++ ++{{- define "nova.configmap.ssh" }} ++{{- $envAll := index . 1 }} ++{{- with $envAll }} ++--- ++apiVersion: v1 ++kind: Secret ++metadata: ++ name: nova-ssh ++type: Opaque ++data: ++ ssh-key-private: {{ .Values.conf.ssh_private | b64enc }} ++{{- include "helm-toolkit.snippets.values_template_renderer" (dict "envAll" $envAll "template" .Values.conf.ssh_public "key" "ssh-key-public" "format" "Secret" ) | indent 2 }} ++ ++{{- end }} ++{{- end }} ++ ++{{- if .Values.manifests.configmap_etc }} ++{{- list "nova-ssh" . | include "nova.configmap.ssh" }} ++{{- end }} +diff --git a/nova/templates/daemonset-compute.yaml b/nova/templates/daemonset-compute.yaml +index 850f0b0..82f185f 100644 +--- a/nova/templates/daemonset-compute.yaml ++++ b/nova/templates/daemonset-compute.yaml +@@ -217,6 +217,9 @@ spec: + mountPath: /root/.ssh/config + subPath: ssh-config + readOnly: true ++ - name: nova-ssh ++ mountPath: /root/.ssh/id_rsa ++ subPath: ssh-key-private + {{- if .Values.conf.ceph.enabled }} + - name: etcceph + mountPath: /etc/ceph +@@ -273,13 +276,10 @@ spec: + mountPath: /var/lib/nova + - name: varliblibvirt + mountPath: /var/lib/libvirt +- - name: nova-etc +- mountPath: /root/.ssh/id_rsa +- subPath: ssh-key-private +- - name: nova-etc ++ - name: nova-ssh + mountPath: /root/.ssh/id_rsa.pub + subPath: ssh-key-public +- - name: nova-etc ++ - name: nova-ssh + mountPath: /root/.ssh/authorized_keys + subPath: ssh-key-public + - name: nova-bin +@@ -295,6 +295,10 @@ spec: + secret: + secretName: {{ $configMapName }} + defaultMode: 0444 ++ - name: nova-ssh ++ secret: ++ secretName: nova-ssh ++ defaultMode: 0400 + {{- if .Values.conf.ceph.enabled }} + - name: etcceph + emptyDir: {} +diff --git a/nova/values.yaml b/nova/values.yaml +index 4edf5c6..9646ded 100644 +--- a/nova/values.yaml ++++ b/nova/values.yaml +@@ -209,6 +209,9 @@ network: + ssh: + name: "nova-ssh" + port: 8022 ++ sshd: ++ enabled: false ++ from_subnet: 0.0.0.0/24 + + dependencies: + dynamic: +@@ -460,6 +463,8 @@ conf: + StrictHostKeyChecking no + UserKnownHostsFile /dev/null + Port {{ .Values.network.ssh.port }} ++ ssh_private: 'null' ++ ssh_public: 'null' + rally_tests: + run_tempest: false + tests: +-- +1.8.3.1 + From cf672440526fa0b3d4d27c9868523dedd294a0cb Mon Sep 17 00:00:00 2001 From: Don Penney Date: Wed, 23 Jan 2019 11:47:26 -0500 Subject: [PATCH 4/4] Update master stx-ceilometer to use panko-5.0.0 This commit updates the master docker image file for ceilometer to specify install of panko-5.0.0 explicitly. Depends-On: I854b75577b6dbc3f0a8171190f5a1aa839412dc8 Change-Id: I4db80c4bcaaa09046285e9ee0af34db1be54a606 Story: 2004520 Task: 29047 Signed-off-by: Don Penney --- .../python-ceilometer/centos/stx-ceilometer.master_docker_image | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openstack/python-ceilometer/centos/stx-ceilometer.master_docker_image b/openstack/python-ceilometer/centos/stx-ceilometer.master_docker_image index 2c24a276..180eb82d 100644 --- a/openstack/python-ceilometer/centos/stx-ceilometer.master_docker_image +++ b/openstack/python-ceilometer/centos/stx-ceilometer.master_docker_image @@ -2,6 +2,6 @@ BUILDER=loci LABEL=stx-ceilometer PROJECT=ceilometer PROJECT_REPO=https://github.com/openstack/ceilometer.git -PIP_PACKAGES="pylint libvirt-python panko gnocchiclient" +PIP_PACKAGES="pylint libvirt-python panko==5.0.0 gnocchiclient" DIST_PACKAGES="libvirt ipmitool"