From bf8218355fa7604f73bbab450773bb2c4d8bf980 Mon Sep 17 00:00:00 2001 From: Andy Ning Date: Tue, 7 Sep 2021 09:40:31 -0400 Subject: [PATCH] Update spec of kubernetes root CA certificate update Some minor adjustments to kubernetes root CA certficate update spec based on implementation for accuracy. Story: 2008675 Signed-off-by: Andy Ning Change-Id: I0eecee32e2ca17c64194151d8e96076c4754f7b6 --- ...urity-2008675-kubernetes-rootca-update.rst | 296 ++++++++++++------ 1 file changed, 194 insertions(+), 102 deletions(-) diff --git a/doc/source/specs/stx-6.0/approved/security-2008675-kubernetes-rootca-update.rst b/doc/source/specs/stx-6.0/approved/security-2008675-kubernetes-rootca-update.rst index b9f85b9..98c519e 100644 --- a/doc/source/specs/stx-6.0/approved/security-2008675-kubernetes-rootca-update.rst +++ b/doc/source/specs/stx-6.0/approved/security-2008675-kubernetes-rootca-update.rst @@ -14,6 +14,9 @@ This feature introduces CLI/REST APIs and execution orchestration for updating Kubernetes root CA certficate and certificates issued by the root CA in a rolling fashion so that the impact on the system is minimized. +This is the updated version of the approved spec security-2008675-kubernetes-rootca-update.rst. +This version reflects the adjustments from implementation. + Problem description =================== @@ -84,11 +87,12 @@ Sysinv operations for root CA certificate update A new set of sysinv CLI commands will be introduced to simplify the update procedure. It will be a procedure similar to software upgrade, with a start, -execute and complete cycle. There won't be support for "abort", but user can -retry the command if it fails. And user can choose to restart the update -procedure by uploading or re-generating a new root CA certficate. This also -provides a mechanism to resume to the original CA certificate if user chooses -to upload the original CA certificate. +execute and complete cycle. User can retry a step if it fails. There will also +be support for "abort", where user can choose to exit an on-going update. But +the user is supposed to restart the update procedure with either uploading or +re-generating a root CA certficate and run the update to full complete. This +also provides a mechanism to restore the original CA certificate if user +chooses to upload the original CA certificate. The following is a summary of the CLI commands and the steps to perform kubernetes root CA certificate update. @@ -112,28 +116,28 @@ kubernetes root CA certificate update. certificate and private key from a file instead of generating one * Change progress state to update-new-rootca-cert-uploaded -3. system kube-rootca-host-update --phase=trustBothCAs -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +3. system kube-rootca-host-update --phase=trust-both-cas +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Update apiserver's trusted CAs to include the new CA cert * Update scheduler's trusted CAs to include the new CA cert * Update controller-manager's trusted CAs to include the new CA cert * Update kubelet's trusted CAs to include the new CA cert * Update admin.conf's trusted CAs to include the new CA cert -* Change progress state to updated-host-trustBothCAs on success -* Change progress state to updating-host-trustBothCAs-failed on failure +* Change progress state to updated-host-trust-both-cas on success +* Change progress state to updating-host-trust-both-cas-failed on failure -4. system kube-rootca-pods-update --phase=trustBothCAs -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +4. system kube-rootca-pods-update --phase=trust-both-cas +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Annotate Daemonsets and Deployments to trigger pod replacement in a safer rolling fashion, to ensure pods to pick up the new root CA cert as its trusted CA along with the old root CA certificate -* Change progess state to updated-pods-trustBothCAs on success -* Change progess state to updating-pods-trustBothCAs-failed on failure +* Change progess state to updated-pods-trust-both-cas on success +* Change progess state to updating-pods-trust-both-cas-failed on failure -5. system kube-rootca-host-update --phase=updateCerts -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +5. system kube-rootca-host-update --phase=update-certs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Update admin.conf's client cert/key data with new ones signed by the new root CA @@ -143,27 +147,27 @@ kubernetes root CA certificate update. * Update controller-manager's client cert/key with new one signed by the new root CA * Update kubelet's client cert/key with new one signed by the new root CA -* Change progress state to updated-host-updateCerts on success -* Chante progress state to updating-host-updateCerts-failed on failure +* Change progress state to updated-host-update-certs on success +* Chante progress state to updating-host-update-certs-failed on failure -6. system kube-rootca-host-update --phase=trustNewCA -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +6. system kube-rootca-host-update --phase=trust-new-ca +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Update admin.conf's trusted CAs to remove the old root CA * Update apiserver's trusted CAs to remove the old root CA * Update controller-manager's trusted CAs to remove the old root CA * Update scheduler's trusted CAs to remove the old root CA * Update kubelet's trusted CAs to remove the old root CA -* Change progress state to updated-host-trustNewCA on success -* Change progress state to updating-host-trustNewCA-failed on failure +* Change progress state to updated-host-trust-new-ca on success +* Change progress state to updating-host-trust-new-ca-failed on failure -7. system kube-rootca-pods-update --phase=trustNewCA -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +7. system kube-rootca-pods-update --phase=trust-new-ca +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Annotate Daemonsets and Deployments to trigger pod replacement in a safer rolling fashion, to remove the old root CA from pods trusted CA list -* Change progress state to updated-pods-trustNewCA on success -* Change progress state to updating-pods-trustNewCA-failed on failure +* Change progress state to updated-pods-trust-new-ca on success +* Change progress state to updating-pods-trust-new-ca-failed on failure 8. system kube-rootca-host-update complete ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -171,17 +175,22 @@ kubernetes root CA certificate update. * Post-check to verify the update * Change the progress state to update-complete -system kube-rootca-update-list -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +9. system kube-rootca-host-update-list +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Run this command anytime to show the update status of all hosts in the cluster -system kube-rootca-update-show -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +10. system kube-rootca-update-show +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Run this command anytime to show the overall update status +11. system kube-rootca-update-abort +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* Run this command to abort the update at any step + VIM Orchestration Operations ---------------------------- @@ -272,96 +281,121 @@ each host. The following is the list of REST resources and APIs to be added: -The new resource /kube_update_ca is added -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The new resource /kube_rootca_update is added +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * URLS: - * /v1/kube_update_ca + * /v1/kube_rootca_update * Request Methods: - * POST /v1/kube_update_ca + * POST /v1/kube_rootca_update * Creates (starts) a new root CA cert update * Response body example:: - {"from_rootca_cert": "kubenetes-5118144266510589551", + {"uuid": "47dff2b6-17ba-45a2-b3d3-8b2a85a5dba9", + "to_rootca_cert": null, + "created_at": "2021-08-25T14:57:13.006034+00:00", + "from_rootca_cert": "d70efa2daaee06f8-91764", + "updated_at": null, "state": "update-started", - "uuid": "223ba65e-45d1-4383-baa7-f03bb4c46773", - "created_at": "2021-03-25T12:04:10.372399+00:00", - "updated_at": "2021-03-25T12:04:10.372399+00:00"} + "id": 1} - * GET /v1/kube_update_ca + * GET /v1/kube_rootca_update - * Return the current kube_update_ca + * Return the current root CA update * Response body example:: - {"from_rootca_cert": "kubenetes-5118144266510589551", - "to_rootca_cert": "kubenetes-6118144266510589551", + {"uuid": "47dff2b6-17ba-45a2-b3d3-8b2a85a5dba9", + "to_rootca_cert": null, + "created_at": "2021-08-25T14:57:13.006034+00:00", + "from_rootca_cert": "d70efa2daaee06f8-91764", + "updated_at": null, "state": "update-started", - "uuid": "223ba65e-45d1-4383-baa7-f03bb4c46773", - "created_at": "2021-03-25T12:04:10.372399+00:00", - "updated_at": "2021-03-25T14:45:43.252964+00:00"} + "id": 1} - * PATCH /v1/kube_update_ca + * PATCH /v1/kube_rootca_update * Modifies the current rootca_update. Used to update the state of the - update (e.g. to update_complete). + update (e.g. to update_complete, or update_aborted). + + * Request body example:: + + [{"path": "/state", + "value": "update-completed", + "op": "replace"}] + + [{"path": "/state", + "value": "update-aborted", + "op": "replace"}] * Response body example:: - {"from_rootca_cert": "kubenetes-5118144266510589551", - "to_rootca_cert": "kubenetes-6118144266510589551", - "state": "update-complete", - "uuid": "223ba65e-45d1-4383-baa7-f03bb4c46773", - "created_at": "2021-03-25T12:04:10.372399+00:00", - "updated_at": "2021-03-25T14:45:43.252964+00:00"} + {"uuid": "fb882423-ea26-42bf-b645-fd9de4248fd4", + "to_rootca_cert": "d70efa2daaee06f8-176046114160516196064588947858918572907", + "created_at": "2021-08-24T13:40:13.318822+00:00", + "from_rootca_cert": "d70efa2daaee06f8-199590289735612744821302170157251522966", + "updated_at": "2021-08-24T13:52:21.547899+00:00", + "state": "update-completed", + "id": 20} - * DELETE /v1/kube_update_ca + {"uuid": "7d07e384-f06d-4213-8e61-5e300aeb9d1c", + "to_rootca_cert": null, + "created_at": "2021-08-24T13:38:55.376395+00:00", + "from_rootca_cert": "d70efa2daaee06f8-199590289735612744821302170157251522966", + "updated_at": "2021-08-24T13:39:47.108582+00:00", + "state": "update-aborted", + "id": 19} - * Deletes the current rootca_update (after it is completed) - -The new resource /kube_rootca_certificate/upload is added +The new resource /kube_rootca_update/upload_cert is added ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * URLS: - * /v1/kube_rootca_certificate/upload + * /v1/kube_rootca_update/upload_cert * Request Methods: - * POST /v1/kube_rootca_certificate/upload + * POST /v1/kube_rootca_update/upload_cert * Upload a root CA cert and key from a file - * Request body example:: + * Request body example: + (The contents of the body is from a file containing both private key and certificate):: - {"ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyRENDQWNDZ0..." - "ca.key": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcGdJQk..."} + {"-----BEGIN PRIVATE KEY----- ...... -----END PRIVATE KEY----- ...... -----BEGIN CERTIFICATE----- ...... -----END CERTIFICATE-----} * Return body example:: - {"cert_id": "kubenetes-5118144266510589551"} + {"success": "8503e172a63b23e6-12808492498813125379", + "error": ""} -The new resource /v1/kube_rootca_certificate/generate is added +The new resource /v1/kube_rootca_update/generate_cert is added ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * URLS: - * /v1/kube_rootca_certificate/generate + * /v1/kube_rootca_update/generate_cert * Request Methods: - * POST /v1/kube_rootca_certificate/generate + * POST /v1/kube_rootca_update/generate_cert * Tell sysinv to generate a new root CA cert and key pair + * Request body example:: + + {"expiry_date": "2022-08-25", + "subject": "C=CA O=Company CN=kubernetes"} + * Return body example:: - {"cert_id": "kubenetes-5118144266510589551"} + {"success": "a8942428863f292b-253592702972967198587817983178843995169", + "error": ""} The existing resource /ihosts is modified to add new actions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -378,51 +412,97 @@ The existing resource /ihosts is modified to add new actions * Request body example:: - {"phase", "trustBothCAs"} + {"phase", "trust-both-cas"} * Response body example:: - {"id": "4", + {"target_rootca_cert": "8503e172a63b23e6-12808492498813125379", + "created_at": "2021-08-25T17:13:22.571151+00:00", "hostname": "controller-1", + "updated_at": "2021-08-25T17:58:59.809264+00:00", + "state": "updating-host-trust-both-cas", "personality": "controller", - "target_rootca_cert": "kubenetes-6118144266510589551", - "effective_rootca_cert": "kubenetes-5118144266510589551", - "state": "updating-host-trustBothCAs"} + "id": 8, + "effective_rootca_cert": "d70efa2daaee06f8-91764", + "uuid": "a597c090-731f-48f8-9f3f-344997c41317"} -The new resource /kube_hosts_update_ca -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The new resource /kube_rootca_update/hosts is added +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * URLs: - * /v1/kube_hosts_update_ca + * /v1/kube_rootca_update/hosts * Request Methods: - * GET /v1/kube_hosts_update_ca + * GET /v1/kube_rootca_update/hosts * Returns the update details of all hosts * Response body example:: { - "hosts": [ - {"id": "2", - "hostname": "controller-1", - "personality": "controller", - "target_rootca_cert": "kubenetes-6118144266510589551", - "effective_rootca_cert": "kubenetes-5118144266510589551", - "state": "updating-host-trustBothCAs" - }, - {"id": "4", - "hostname": "compute-0", - "personality": "compute", - "target_rootca_cert": "kubenetes-6118144266510589551", - "effective_rootca_cert": "kubenetes-5118144266510589551", - "state": "updating-host-updateCerts" - } - ] + "kube_host_updates": [ + {"target_rootca_cert": null, + "created_at": "2021-08-25T17:13:22.558411+00:00", + "hostname": "controller-0", + "updated_at": null, + "state": null, + "personality": "controller", + "id": 7, + "effective_rootca_cert": "d70efa2daaee06f8-91764", + "uuid": "7d7d05dd-900f-4004-951d-d92536faac8e" + }, + {"target_rootca_cert": "8503e172a63b23e6-12808492498813125379", + "created_at": "2021-08-25T17:13:22.571151+00:00", + "hostname": "controller-1", + "updated_at": "2021-08-25T17:59:16.097029+00:00", + "state": "updated-host-trust-both-cas", + "personality": "controller", + "id": 8, + "effective_rootca_cert": "d70efa2daaee06f8-91764", + "uuid": "a597c090-731f-48f8-9f3f-344997c41317" + }, + {"target_rootca_cert": null, + "created_at": "2021-08-25T17:13:22.584500+00:00", + "hostname": "worker-0", + "updated_at": null, + "state": null, + "personality": "worker", + "id": 9, + "effective_rootca_cert": "d70efa2daaee06f8-91764", + "uuid": "a4ca4eed-9b2f-4b4c-8ee7-45bbc573a55f" + } + ] } +The new resource /kube_rootca_update/pods is added +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* URLs: + + * /v1/kube_rootca_update/pods + +* Request Methods: + + * POST /v1/kube_rootca_update/pods + + * Update root CA cert for pods + + * Request body example:: + + {"phase", "trust-both-cas"} + + * Response body example:: + + {"uuid": "6cf4157b-75ff-4e86-bc96-8b08e4c9836d", + "to_rootca_cert": "8503e172a63b23e6-12808492498813125379", + "created_at": "2021-08-25T17:13:22.535798+00:00", + "from_rootca_cert": "d70efa2daaee06f8-91764", + "updated_at": "2021-08-25T18:37:02.574836+00:00", + "state": "updating-pods-trust-both-cas", + "id": 3} + Security impact --------------- @@ -496,8 +576,10 @@ Repos Impacted -------------- Impacted repo from this spec: + * config * stx-puppet +* fault Work Items ---------- @@ -526,7 +608,7 @@ Sysinv * root CA certficate and issuer creation in cert-manager * calculate the ID of the new root certificate -* kube-rootca-host-update --phase=trustBothCAs CLI/API +* kube-rootca-host-update --phase=trust-both-cas CLI/API * basic infrastructure * semantic checks @@ -535,14 +617,14 @@ Sysinv * agent RPC/implementation (apply puppet manifest, report back config status, etc...) -* kube-rootca-pods-update --phase=trustBothCAs CLI/API +* kube-rootca-pods-update --phase=trust-both-cas CLI/API * basic infrastructure * semantic checks * conductor implementation (generate hieradata, trigger puppet manifests apply, handle apply result, update progress state etc...) -* kube-rootca-host-update --phase=updateCerts CLI/API +* kube-rootca-host-update --phase=update-certs CLI/API * basic infrastructure * semantic checks @@ -552,7 +634,7 @@ Sysinv * agent RPC/implementation (apply puppet manifest, report back config status, etc...) -* kube-rootca-host-update --phase=trustNewCA CLI/API +* kube-rootca-host-update --phase=trust-new-ca CLI/API * basic infrastructure * semantic checks @@ -561,7 +643,7 @@ Sysinv * agent RPC/implementation (apply puppet manifest, report back config status, etc...) -* kube-rootca-pods-update --phase=trustNewCA CLI/API +* kube-rootca-pods-update --phase=trust-new-ca CLI/API * basic infrastructure * semantic checks @@ -578,19 +660,29 @@ Sysinv * kube-rootca-update-show CLI/API * basic infrastructure - * condutor database query + * conductor database query -* kube-rootca-update-list CLI/API +* kube-rootca-host-update-list CLI/API * basic infrastructure - * condutor database query + * conductor database query + +* kube-rootca-update-abort CLI/API + + * basic infrastructure + * semantic checks + * system health checks for update abort + * clear 'kube root CA update in progress' alarm + * raise 'kube root CA update aborted' alarm Puppet ^^^^^^ -* runtime manifest for host update trustBothCAs phase -* runtime manifest for host update updateCerts phase -* runtime manifest for host update trustNewCA phase +* runtime manifest for host update trust-both-cas phase +* runtime manifest for host update update-certs phase +* runtime manifest for host update trust-new-ca phase +* runtime manifest for pods update trust-both-cas phase +* runtime manifest for pods update trust-new-ca phase System Upgrade ^^^^^^^^^^^^^^