After the "deploy host" command is executed to deploy a
software release, an alarm is created by [1] to indicate
success/failure and follow-up actions needed.
This commit clears the success alarm when the host
reboots running with the new software release.
[1] https://review.opendev.org/c/starlingx/update/+/916688
Test Case
PASS: run "deploy host", verify the alarm is created, unlock
the host and verify the alarm is cleared after host
reports inventory
PASS: lock/unlock host
PASS: install/bootstrap/unlock AIO-DX
Story: 2010676
Task: 49933
Depends-on: https://review.opendev.org/c/starlingx/update/+/916688
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Change-Id: Ia4494142b9f5487a0ddf472833d6c80719382f8a
This commit creates the sw_version field on i_host table, changes the
pxe file creation to use this field and also creates a migrate script
that will be executed on 22.12 to 24.09 scenario.
There are other places where the version is read/write from loads
table, these should be addressed by another commit.
Test Plan:
The tests below for deploy host was done from 24.03 to 24.09 iso.
PASS: AIO-DX install/bootstrap/unlock
PASS: AIO-SX install/bootstrap/unlock
PASS: System host-show <hostname> returning respective value at cli
PASS: Deploy host of controller-1
Story: 2010676
Task: 49865
Change-Id: I7be0c4e48a10d4296d1cda50c49d6d1992e89139
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
Add the folowing enhancements to application updates during
Kubernetes upgrades:
1) Move the pre application update logic from kube-upgrade-start step
to a specific separated step called via a new command line option
named kube-pre-application-update, which can be triggered after the
download images step and before upgrading networking.
2) Move the post application update logic from kube-upgrade-complete
step to a specific separated step called via a new command line
option named kube-post-application-update, which can be triggered
after the kube-upgrade-complete stage and before the upgrade is
deleted.
3) Introduce validation logic to kube-upgrade-start step to check if
all applied apps have available versions compatible with intermediate
and target Kubernetes versions. Upgrades are blocked if apps marked
to be pre updated are incompatible with current and target Kubernetes
versions. Upgrades are also blocked if apps marked to be post updated
are incompatible with the target Kubernetes version.
4) Delete uploaded applications incompatible with the target Kubernetes
version and upload one that is compatible if available.
5) Restore kube-upgrade-start and kube-upgrade-complete to their
original logic before application updates during Kubernetes upgrades
was implemented on task 49416. The kube-upgrade-start step is
synchronous as it used to be before that change.
6) Update sysinv and cgts-client unit tests to account for the new
Kubernetes upgrade steps.
7) Create a helper function called "patch_kube_upgrade" to improve code
reuse when creating patch requests for new shell commands related to
Kubernetes upgrades.
Test Plan:
AIO-SX Test Cases:
PASS: Fresh install.
PASS: Successful Kubernetes single version upgrade with no apps that
need to be updated.
PASS: Successful Kubernetes multi-version upgrade with no apps that need
to be updated.
PASS: Successful Kubernetes upgrade with apps that need to be updated
before and after the new version is deployed.
PASS: Check if the upgrade is blocked if an app is incompatible with a
Kubernetes intermediate version during a multi-version
upgrade.
PASS: Check if the upgrade is blocked if an app marked to be pre updated
is incompatible with the Kubernetes target version.
PASS: Check if the upgrade is blocked if an app marked to be post
updated is incompatible with the Kubernetes target version.
PASS: Check if uploaded apps have been replaced by compatible versions.
PASS: Check if uploaded apps that do not have compatible versions were
removed.
PASS: Failure to run kube-pre-application-update and successful
retry.
PASS: Failure to run kube-post-application-update and successful
retry.
PASS: Abort during kube-pre-application-update and start over.
PASS: Reject aborting Kubernetes upgrade after post-updated-apps state.
AIO-DX Test Cases:
PASS: Fresh install.
PASS Successful Kubernetes upgrade with no apps that need to be
updated.
PASS: Successful Kubernetes upgrade with apps that need to be updated
before and after the new version is deployed.
PASS: Check if the upgrade is blocked if an app marked to be pre updated
is incompatible with the Kubernetes target version.
PASS: Check if the upgrade is blocked if an app marked to be post
updated is incompatible with the Kubernetes target version.
Story: 2010929
Task: 49595
Change-Id: I9b48567c39c9a12b7563d56ab90fbfe9dd7082aa
Signed-off-by: Igor Soares <Igor.PiresSoares@windriver.com>
check for /etc/swanctl/x509/system-ipsec-certificate-<hostname>.crt
exist and show in the output of "system certificate-list" also
show certificate details with "system certificate-show IPsec"
Test Cases:
PASS: Enable IPsec on controller-0, verify that IPsec certificate
list in the output of "system certificate-list" and
"system certificate-show IPsec" shows details of IPsec
certificate
PASS: Enable IPsec on controller-1, verify that IPsec certificate
list in the output of "system certificate-list" and
"system certificate-show IPsec" shows details of IPsec
certificate
PASS: verify that IPsec certificate not shown in the output of
"system certificate-list" if /etc/swanctl/x509/system-ipsec-
certificate-<hostname>.crt doesn't exit
Story: 2010940
Task: 49891
Change-Id: I95be304d99feff83e69750b90de289c1dde18b0c
Signed-off-by: amantri <ayyappa.mantri@windriver.com>
Add an API /v1/certificate/get_all_certs to retrieve all the
platform certs(oidc, wra, adminep, etcd,
service account certs, system-restapi-gui-certificate,
open-ldap, openstack, system-registry-local-certificate,
k8s certs) in JSON response and use this response to format
the "system certificate-list" output as "show-certs.sh" output.
Add an API /v1/certificate/get_all_k8s_certs to retrieve all the
tls,opaque certs in JSON response and use this response to
format the "system k8s-certificate-list" output as
"show-certs.sh -k" output
Implement "system certificate-show <cert name>",
"system k8s-certificate-show <cert name>" to show the full
details of the certificate.
Implement filters in api and cli to show the expired and expiry
certificates
Testcases:
PASS: Verify all the cert values(Residual Time,Issue Date, Expiry Date
,Issuer,Subject,filename,Renewal) are showing fine for all the
following cert paths when "system certificate-list" is executed
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/pki/ca-trust/source/anchors/dc-adminep-root-ca.crt
/etc/ssl/private/admin-ep-cert.pem
/etc/etcd/etcd-client.crt
/etc/etcd/etcd-server.crt
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-client.crt
/var/lib/kubelet/pki/kubelet-client-current.pem
/etc/kubernetes/pki/ca.crt
/etc/ldap/certs/openldap-cert.crt
/etc/ssl/private/registry-cert.crt
/etc/ssl/private/server-cert.pem
PASS: Verify all the cert values(Residual Time,Issue Date, Expiry Date
,Issuer,Subject,filename,Renewal) are showing fine for all the
service accts when "system certificate-list" is executed
/etc/kubernetes/scheduler.conf
/etc/kubernetes/admin.conf
/etc/kubernetes/controller-manager.conf
PASS: Verify the system-local-ca secret is shown in the output of
"system certificate-list"
PASS: List ns,secret name in the output of ssl,docker certs if the
system-restapi-gui-certificate, system-registry-local-certificate
exist on the system when "system certificate-list" executed
PASS: Apply oidc app verify that in "system certificate-list" output
"oidc-auth-apps-certificate", oidc ca issuer and wad cert are
shown with all proper values
PASS: Deploy WRA app verify that "mon-elastic-services-ca-crt",
"mon-elastic-services-extca-crt" secrets are showing in the
"system certificate-list" output and also kibana,
elastic-services cert from mon-elastic-services-secrets secret
PASS: Verify all the cert values(Residual Time,Issue Date, Expiry Date
,Issuer,Subject,filename,Renewal) are showing fine for all the
Opaque,tls type secrets when "system k8s-certificate-list" is
executed
PASS: Execute "system certificate-show <cert name>" for each
cert in the "system ceritificate-list" output and
check all details of it
PASS: Execute "system certificate-list --expired" shows the
certificates which are expired
PASS: Execute "system certificate-list --soon_to_expiry <N>"
shows the expiring certificates with in the specified
N days
PASS: Execute "system k8s-certificate-list --expired" shows the
certificates which are expired
PASS: Execute "system k8s-certificate-list --soon_to_expiry <N>"
shows the expiring certificates with in the specified
N days
PASS: On DC system verify that admin endpoint certificates are
shown with all values when "system certificate-list" is
executed
PASS: Verify the following apis
/v1/certificate/get_all_certs
/v1/certificate/get_all_k8s_certs
/v1/certificate/get_all_certs?soon_to_expiry=<no of days>
/v1/certificate/get_all_k8s_certs?soon_to_expiry=<no of days>
/v1/certificate/get_all_certs?expired=True
/v1/certificate/get_all_k8s_certs?expired=True
Story: 2010848
Task: 48730
Task: 48785
Task: 48786
Change-Id: Ia281fe1610348596ccc1e3fad7816fe577c836d1
Signed-off-by: amantri <ayyappa.mantri@windriver.com>
After the management reconfiguration, it was not possible to apply a reboot-required
patch because the sysinv was sending the old mgmt IP adress to the mtce.
Consequently, mtce wasn't creating the required file (/var/run/.node_locked) during
the host-lock command.
This file is essential for the sw-patch tool to proceed with the installation.
Additionally, the management network reconfiguration runtime manifest can be executed
prematurely if the MGMT_NETWORK_RECONFIGURATION_ONGOING flag is used.
However, users might introduce other changes that could unintentionally trigger the
runtime manifests before the host-unlock command.
This could lead to unexpected keystone changes, potentially causing CLI blockage or
system reboots.
The MGMT_NETWORK_RECONFIGURATION_ONGOING flag is created when initiating management
network reconfiguration commands and it is intended to avoid update on the dnsmasq
files until system reboot.
Changed to MGMT_NETWORK_RECONFIGURATION_UNLOCK because this flag is intended to
guarantee keystone changes only occur during the unlock command.
Tests dome:
IPv4 AIO-SX fresh install
IPv4 AIO-DX with mgmt in vlan fresh install
IPv4 DC with subcloud AIO-SX
IPv4 AIO-SX mgmt reconfig and apply a reboot-required patch
IPv4 subcloud AIO-SX mgmt reconfig and apply a reboot-required patch
Partial-Bug: #2060066
Story: 2010722
Task: 49810
Change-Id: I138d8e31edd60a41a4595cfb8bd2dc478bc01013
A call to _generate_dnsmasq_hosts_file is added to sysinv
conductor handle_dhcp_lease.
There are currently a number of calls to _generate_dnsmasq_hosts_file
as shown in this list.
_create_or_update_address
_allocate_addresses_for_host
_unallocate_addresses_for_host
_remove_addresses_for_host
mgmt_ip_set_by_ihost
reserve_ip_for_third_monitor_node
reserve_ip_for_cinder
_init_controller_for_upgrade
Most of these are for specific system configuration changes.
However, the introduction of the pxeboot network repurposed
the use and content of the dnsmasq.hosts file.
Given that the dnsmasq hosts file no longer maintains management
network addresses and hostname associations, the question of whether
the call to _generate_dnsmasq_hosts_file is needed for them at all
anymore. That could be discussed or commented on in this review.
Testing system responsiveness to dhcp lease changes revealed
that the dnsmasq.hosts file would only get updated when one of
the aforementioned procedures are called. In many cases it took
a long time after the leases file was updated before the change
was realized by the system through a dnamasq.hosts update.
Adding this call to every dhcp lease update makes sense in that
it makes the responsiveness of dhcp lease changes immediate.
Also, there is the possibility that a lease event is missed if it
occurs while the conductor is not running. This may occur over a
swact, patch or simply process restart by puppet.
This update addrersses this gap by adding a one time call to
_generate_dnsmasq_hosts_file in the _controller_config_active_apply
audit so that the dnsmasq hosts and addn_hosts files get update
over a conductor process restart.
Test Plan:
PASS: Build, test and install AIO DX Plus system.
PASS: Verify the dnsmasq.hosts and dnsmasq.addn_hosts files get
updated immediately following a dnsmasq dhcp lease event
and conductor process restart.
PASS: Verify the dnsmasq.hosts and dnsmasq.addn_hosts files get
update over a swact.
Story: 2010940
Task: 49788
Change-Id: I8d55b865f682bd5e0e210481a9dff318baab436b
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
Introduce automatic downgrade of StarlingX applications to the
multiple application version feature.
Auto downgrades are triggered by default in scenarios which the applied
application bundle is not available anymore under the applications
folder but an older version of the same app is. For instance, when
platform patches are removed and a previously available ostree is
deployed, thus restoring the old set of available apps under the
/usr/local/share/applications/helm/ directory.
A new section called 'downgrades' can be added to the metadata.yaml file
to disable the default behavior. For example:
downgrades:
auto_downgrade: false
When auto downgrades are disabled the current applied version remains
unchanged.
Test plan:
PASS: build-pkgs -a && build-image
PASS: AIO-SX fresh install.
PASS: Apply platform-integ-apps.
Update platform-integ-apps using a tarball that is not available
under /usr/local/share/applications/helm/ and that does not
contain the downgrade section.
Confirm that platform-integ-apps is downgraded.
PASS: Apply platform-integ-apps.
Update platform-integ-apps using a tarball that is not available
under /usr/local/share/applications/helm/ and that has the
auto_downgrade metadata option set to 'true'.
Confirm that platform-integ-apps is downgraded.
PASS: Apply platform-integ-apps.
Update platform-integ-apps using a tarball that is not available
under /usr/local/share/applications/helm/ and that has the
auto_downgrade metadata option set to 'false'.
Confirm that the originally applied platform-integ-apps version
remains unchanged.
PASS: Run a kubernetes upgrade with apps to be pre and post updated.
Confirm that apps are successfully updated and not downgraded
after the Kubernetes upgrade has finished.
Story: 2010929
Task: 49847
Change-Id: I33f0e0a5b8db128aef76fb93ba322364881097cf
Signed-off-by: Igor Soares <Igor.PiresSoares@windriver.com>
In change [1], a restore in progress check was added, however
the flag used for this is removed at the end of the restore
playbook that is executed on controller-0, causing possible
problems if agents on other nodes send a report with incomplete
information due to restore.
To resolve this, the _verify_restore_in_progress() function
was used, which queries the restore status in the database,
and is only modified when executing the "system restore-complete"
command. This way we will know that from then on the agent's
reports can be considered.
Additionally, the “system host-reinstall” command has also
been observed to cause similar issues if run on a restored system.
To prevent this from happening, another condition was added,
which checks if inv_state is "reinstalling".
[1]: https://review.opendev.org/c/starlingx/config/+/899510
Test-Plan:
PASS: AIO-SX fresh install
PASS: Standard fresh install
PASS: create/modify/delete a partition in the
controller-0/controller-1/compute-0 followed by a reboot and check the status
with 'system host-disk-partition-list'.
PASS: Restart of sysinv-conductor and/or sysinv-agent services
during puppet manifest applying.
PASS: AIO-SX Backup and Restore
PASS: Standard Backup and Restore
Closes-Bug: 2061170
Change-Id: I6c142439c9f13dcdeb493892a5a9283f6a1e2d00
Signed-off-by: Erickson Silva de Oliveira <Erickson.SilvadeOliveira@windriver.com>
Worker and storage nodes currently support pxeboot-N hostname
nslookup resolution because they dhcp for their pxeboot network
lease address provided by dnsmasq persists.
However, this is not true for the controllers. Although they may
initially dhcp for a pxeboot address, that address is overridden
by their statically assigned pxeboot network address(es).
Adding the controller pxeboot network hostnames and addresses to
the dnsmasq.addn_hosts file yields proper pxeboot hostname resolution
for controllers.
From Controllers:
[sysadmin@controller-0 ~$ nslookup pxeboot-2
Server: fdff:10:80:27::2
Address: fdff:10:80:27::2#53
Name: pxeboot-2
Address: 192.168.202.3
[sysadmin@controller-0 ~$ nslookup pxeboot-1
Server: fdff:10:80:27::2
Address: fdff:10:80:27::2#53
Name: pxeboot-1
Address: 192.168.202.2
sysadmin@controller-1:~$ nslookup pxeboot-1
Server: fdff:10:80:27::2
Address: fdff:10:80:27::2#53
Name: pxeboot-1
Address: 192.168.202.2
sysadmin@controller-1:~$ nslookup pxeboot-2
Server: fdff:10:80:27::2
Address: fdff:10:80:27::2#53
Name: pxeboot-2
Address: 192.168.202.3
From Worker:
sysadmin@worker-0:~$ nslookup pxeboot-1
Server: 192.168.204.1
Address: 192.168.204.1#53
Name: pxeboot-1
Address: 169.254.202.2
Now all hosts in the system support pxeboot hostname nslookup
Also, this update adds an explicit call to _generate_dnsmasq_hosts_file
to the conductor process restart. This handles the case where dnsmasq
publishes a new lease to handle while the sysinv conductor is not
running or being restarted.
Test Plan:
PASS: Verify build and install AIO DX Plus system.
PASS: Verify format of new additions to dhsmasq.addn_hosts file.
PASS: Verify nslookup using controller pxeboot hostnames from either
controller or even a worker node.
PASS: Verify no new pep8 warnings or errors are added to the conductor
manager.py.
Story: 2010940
Task: 49829
Change-Id: Ibacdaadd24cf8c73fec98167d4a79fece341b1e6
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
Recent commit https://opendev.org/starlingx/config/commit/634d4916
introducing changes for dual-stack networking made a change
to the DB api's address_get_by_name to return a list of IPv4 and
IPv6 addresses rather than a singular address. As such, the list
can be empty if there are no addresses associated with a
particular name, rather than throwing an AddressNotFoundByName
exception.
Currently, the interface_network code depends on the
AddressNotFoundByName exception to determine whether a new
address needs to be allocated for a dynamic network.
This can cause an issue for worker, storage nodes when one of
their interfaces is associated with certain networks (such as
the storage network).
The symptom of this may be an interface which is 'DOWN' after
unlock, as it's interface configuration file is marked for a
'static' address, with no address present (because it wasn't
allocated).
This commit fixes the issue by simply checking that the list
returned by address_get_by_name is empty.
Test Plan:
- Fresh install of a Standard system.
- Ensure named addresses are present in the DB for all
nodes (mgmt, cluster-host, oam for controllers)
- Create a new address pool and storage network and
assign it to a worker node interface.
- Unlock the worker node and ensure the address is
present on the interface and it is in 'UP' state.
Story: 2011027
Task: 49627
Change-Id: I9763f7c71797d9b321e7bf9e1b6db759378af632
Signed-off-by: Steven Webster <steven.webster@windriver.com>
This change fixes a type mismatch bug introduced in [1]. A python list
is expected but a python str is provided instead.
[1] https://review.opendev.org/c/starlingx/config/+/893566
This type mismatch will result in the 'deadlock' prevention logic to
never be invoked. In [2] below, the 'if classes' branch is never entered:
[2] 85a548ffcc/sysinv/sysinv/sysinv/sysinv/conductor/manager.py (L13481)
Test plan:
PASS: Run 'sudo chage -M 999 sysadmin; sudo chage -M 888 sysadmin;
sudo chage -M 777 sysadmin'. Notice 'out of config alarm' in
'fm alarm-list'. Verify that it clears up after about 5 min.
PASS: Verify in i_user db table and /etc/shadow that it correctly
contains the last password age, 777 in this case.
Note: In a managed subcloud, the value in /etc/shadow file will
be changed again in about 20 min to sync with the sysadmin password
and age in the system controller.
Closes-Bug: 2034446
Signed-off-by: Rei Oliveira <Reinildes.JoseMateusOliveira@windriver.com>
Change-Id: I24d9807e9eb2d94e026be7b8f3448a6cd42fcdd6
This removes the skip_update_config parameter from the
_config_apply_runtime_manifest() call when upgrading Kubernetes
control-plane. This parameter was unintentially set to True,
so this configuration step did not persist. This caused
generation of 250.001 config-out-of-date alarms during kube
upgrade.
The review that introduced the bug:
https://review.opendev.org/c/starlingx/config/+/911100
TEST PLAN:
- watch /var/log/nfv-vim.log for each orchestrated upgrade
PASS: orchestrated k8s upgrade (no faults)
- AIO-SX, AIO-DX, Standard
PASS: orchestrated k8s upgrade, with fault insertion during
control-plane upgrade first attempt
- AIO-SX
- AIO-DX (both controller-0, controller-1)
- Standard (both controller-0, controller-1)
PASS: orchestrated k8s upgrade, with fault insertion during
control-plane upgrade first and second attempt, trigger abort
- AIO-SX
- AIO-DX (first controller)
Closes-Bug: 2056326
Change-Id: I629c8133312faa5c95d06960b15d3e516e48e4cb
Signed-off-by: Jim Gauld <James.Gauld@windriver.com>
User unable to take subcloud backup when there
is a stale backup in progress alarm
Example:
When user tries to take subcloud backup in
Distributed cloud env if there is stale
210.001 alarm present in subcloud then user
can not trigger the subsequent subcloud backup
This Fix helps to identify the 210.001 alarms
and clear them if they are pending more than 1
hour
TEST PLAN:
PASS: DC-libvirt setup with 2 controllers
and 2 subclouds
PASS: verified stale 210.001 getting removed
Closes-Bug: 2058516
Change-Id: Iedcc5e41cd4245c538d331d9aa8c2b6cc445acce
Signed-off-by: rummadis <ramu.ummadishetty@windriver.com>
Max_cpu_mhz_personality is the attribute of the host which
can be configured in host where turbo freq is enabled.In case
of host whose role is both controller and worker, the personality
for the attribute was not taken care to include such scenario.
Made the changes in the sysinv conductor to update the host
personalities based on the function that node operates, which
handles the scenario when the host acts as both controller and
worker node.
TEST PLAN:
PASS: Build and deploy ISO on Simplex
PASS: Check whether the max cpu freq set on a simplex
Below are the commands:
system host-show <host_id> | grep is_max_cpu_configurable
system service-parameter-list --name cpu_max_freq_min_percentage
system service-parameter-modify platform config cpu_max_freq_min_percentage=<>
system host-update <host_id> max_cpu_mhz_configured=<value in mhz>
After above commands check whether cpu is set using below command:
sudo turbostat
Closes-Bug: 2058476
Change-Id: I08a5d1400834afca6a0eeaaa8813ac8d71a9db15
Signed-off-by: Poornima Y N <Poornima.Y.N@windriver.com>
In the case of a rare intermittent failure behaviour during the
upgrading control plane step where puppet hits timeout first before
the upgrade is completed or kubeadm hits its own Upgrade Manifest
timeout (at 5m).
This change will retry running the process by
reporting failure to conductor when puppet manifest apply fails.
Since it is using RPC to send messages with options, we don't get
the return code directly and hence, cannot use a retry decorator.
So we use the sysinv report callback feature to handle the
success/failure path.
TEST PLAN:
PASS: Perform simplex and duplex k8s upgrade successfully.
PASS: Install iso successfully.
PASS: Manually send STOP signal to pause the process so that
puppet manifest timeout and check whether retry code works
and in retry attempts the upgrade completes.
PASS: Manually decrease the puppet timeout to very low number
and verify that code retries 2 times and updates failure
state
PASS: Perform orchestrated k8s upgrade, Manually send STOP
signal to pause the kubeadm process during step
upgrading-first-master and perform system kube-upgrade-abort.
Verify that upgrade-aborted successfully and also verify
that code does not try the retry mechanism for
k8s upgrade control-plane as it is not in desired
KUBE_UPGRADING_FIRST_MASTER or KUBE_UPGRADING_SECOND_MASTER
state
PASS: Perform manual k8s upgrade, for k8s upgrade control-plane
failure perform manual upgrade-abort successfully.
Perform Orchestrated k8s upgrade, for k8s upgrade control-plane
failure after retries nfv aborts automatically.
Closes-Bug: 2056326
Depends-on: https://review.opendev.org/c/starlingx/nfv/+/912806https://review.opendev.org/c/starlingx/stx-puppet/+/911945https://review.opendev.org/c/starlingx/integ/+/913422
Change-Id: I5dc3b87530be89d623b40da650b7ff04c69f1cc5
Signed-off-by: Saba Touheed Mujawar <sabatouheed.mujawar@windriver.com>
This change allows the management and admin
network reconfig at same time in an AIO-DX
subcloud.
Currently, it is necessary to lock and unlock
the controller in order to reconfigure the
management network from AIO-SX.
If the customer changes the management network
fist, the new mgmt network will be in the database
but the changes will jsut be applied during the
unlock / reboot of the system.
But the admin network changes are applied in runtime,
if the admin network is changed after the management
network reconfig, the admin will apply the changes on
the system and some of them will apply the new mgmt
network values before the system is updated with the
new mgmt ip range, it will cause a puppet error and
the system will not be correctly configured.
Tests done:
IPv4 AIO-SX subcloud mgmt network reconfig
IPv4 AIO-SX subcloud admin network reconfig
IPv4 AIO-SX subcloud admin and mgmt network reconfig
IPv4 AIO-SX subcloud mgmt and admin network reconfig
Story: 2010722
Task: 49724
Change-Id: I113eab2618f34b305cb7c4ee9bb129597f3898bb
Signed-off-by: Fabiano Correa Mercer <fabiano.correamercer@windriver.com>
This commit fixes an LDAP authentication issue seen on worker nodes
of a subcloud after a rehoming procedure was performed.
There are two main parts:
1. Since every host of a subcloud authenticates with the system
controller, we need to reconfigure the LDAP URI across all nodes
of the system when the system controller network changes (upon
rehome). Currently, it is only being reconfigured on controller
nodes.
2. Currently, the system uses an SNAT rule to allow worker/storage
nodes to authenticate with the system controller when the admin
network is in use. This is because the admin network only exists
between controller nodes of a distributed cloud. The SNAT rule
is needed to allow traffic from the (private) management network
of the subcloud over the admin network to the system controller
and back again. If the admin network is _not_ being used,
worker/storage nodes of the subcloud can authenticate with the
system controller, but routes must be installed on the
worker/storage nodes to facilitate this. It becomes tricky to
manage in certain circumstances of rehoming/network config.
This traffic really should be treated in the same way as that
of the admin network.
This commit addresses the above by:
1. Reconfiguring the ldap_server config across all nodes upon
system controller network changes.
2. Generalizing the current admin network nat implementation to
handle the management network as well.
Test Plan:
IPv4, IPv6 distributed clouds
1. Rehome a subcloud to another system controller and back again
(mgmt network)
2. Update the subcloud to use the admin network (mgmt -> admin)
3. Rehome the subcloud to another system controller and back again
(admin network)
4. Update the subcloud to use the mgmt network (admin -> mgmt)
After each of the numbered steps, the following were performed:
a. Ensure the system controller could become managed, online, in-sync
b. Ensure the iptables SNAT rules were installed or updated
appropriately on the subcloud controller nodes.
c. Log into a worker node of the subcloud and ensure sudo commands
could be issued without LDAP timeout.
d. Log into worder node with LDAP USER X via console and verify
login succeed
In general, tcpdump was also used to ensure the SNAT translation was
actually happening.
Partial-Bug: #2056560
Change-Id: Ia675a4ff3a2cba93e4ef62b27dba91802811e097
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Details:
This change extracts the addresses from both the primary and secondary
address pools and makes them available for use in Puppet manifests.
To accommodate the dual stack configuration, the address allocation
for non-controller nodes was updated for both management and
cluster-host networks.
Since the task for upgrade data-migration is not ready yet, a logic
was added to access directly the network's field pool_uuid and get
the addresses with it, if the network_addresspools is empty (as it
would be the case after an upgrade)
As the data migration functionality for the upgrade is still under
development, a temporary solution was implemented. Logic was added
to directly access the network's "pool_uuid" field and retrieve
addresses through it whenever the "network_addresspools" list is
empty, which is expected to occur immediately following an upgrade.
This allows for uninterrupted network operation during the upgrade
process.
Variable Naming:
The following naming convention will be used for the variables:
$platform::network::[network_type]::[ipv4/ipv6]::params::{var_name}
Variable Usage:
Primary Pool: Existing variables will be maintained and populated with
addresses from the primary pool. This ensures compatibility with
applications that currently rely on them. They have the format
$platform::network::[network_type]::params::{var_name}
The variable platform::network::[network_type]::params::subnet_version
indicates the primary pool protocol.
Secondary Pool: New variables with the above naming convention will
be introduced, allowing applications to utilize addresses from the
secondary pool if needed.
Benefits:
Improved modularity and reusability of network configurations.
Clear separation of concerns between primary and secondary pools.
Easier implementation of applications requiring addresses from either pool.
Notes:
Replace [network_type] can be oam. mgmt, cluster_host, ...
Replace [ipv4/ipv6] with either "ipv4" or "ipv6" depending on
the address family.
Replace [variable_name] with a descriptive name for the specific
variable (e.g., "subnet_version", "interface_address").
Test Plan:
[PASS] unit tests implemented
[PASS] AIO-SX, Standard instalation (IPv4 and IPv6)
- using the dependency change the secondary pool was introduced
- system was lock/unlocked and no puppet manifests were
detected
- inspection of system.yaml and controller-0.yaml to verify
variables content
- no alarms or disabled services were found
- in standard added hosts with dual-stack config and verified
that addresses were allocated for mgmt and cluster-host and
after unlock the interface id was assigned to the respective
entries.
[PASS] For standard systems during upgrade, simulate node unlock by:
- Clearing the "network_addresspools" table after Ansible
execution and before DM configuration.
- Installing remaining nodes with the table empty. This mimics
the post-upgrade scenario.
Story: 2011027
Task: 49679
Depends-On: https://review.opendev.org/c/starlingx/config/+/908915
Change-Id: If252fa051b2ba5b5eb3033ff269683af741091d2
Signed-off-by: Andre Kantek <andrefernandozanella.kantek@windriver.com>
Add a configuration option to sysinv.conf that sets the default behavior
for automatically updating apps if not specified in the application
metadata.
The new option is named 'missing_auto_update' and it is available under
the 'app_framework' section in sysinv.conf. It is set to 'False' in de
code base to facilitate porting this change to previous releases, which
had the auto_update option disabled by default. A separate stx-puppet
change was done to set it to 'True' by default for the next releases:
https://review.opendev.org/c/starlingx/stx-puppet/+/911384
Test Plan:
PASS: AIO-SX fresh install.
PASS: Build platform-integ-apps without the 'upgrades' metadata
section.
Confirm that the default value was correctly inserted into the
'auto_update' field of the 'kube_app_bundle' database table after
a fresh install.
PASS: Change the default 'missing_auto_update' value in sysinv.conf.
Add a new version of platform-integ-apps without the 'upgrades'
metadata section to /usr/local/share/applications/helm/.
Restart sysinv-conductor.
Confirm that the 'auto_update' column for the new version matches
the new 'missing_auto_update' value.
Apply platform-integ-apps and confirm that the framework complies
with the new 'missing_auto_update' value.
Story: 2010929
Task: 49670
Change-Id: I068310bed70a5e1cf7e8668b2e673751276ed3a8
Signed-off-by: Igor Soares <Igor.PiresSoares@windriver.com>
This change introduces a new API that leverages the network-addrpool
table to establish relationships between network and address pool
resources. This functionality enhances network management by enabling
efficient address pool allocation and association with specific
networks.
The new dual-stack support introduces a prioritization mechanism for
address pools associated with a network. The primary pool will be the
default choice for address selection unless a preferred pool is
explicitly specified. For optimal flexibility, each network supports
a maximum of two pools – one per address family (IPv4 and IPv6).
Networks configured for PXE boot will currently only offer IPv4 address pools due to limitations in PXE over IPv6 support.
To enhance consistency during address pool assignments, the network
table now includes the primary_pool_family field. Networks initially
assigned to IPv4 pools cannot have IPv6 pools designated as primary
later. A separate task will make the network field pool_uuid optional.
To streamline access to primary pool addresses, a wrapper function
named get_primary_address_by_name() was implemented. This function
ensures retrieval of the correct address by name, avoiding potential
confusion caused by duplicate entries due to the presence of both
IPv4 and IPv6 pools.
Upgrade will be handled in a separate task within this story.
Test Plan
[PASS] new set of unit tests
[PASS] Install AIO-SX and AIO-DX and verify:
- no alarms or failed services
- direct access to the database tables to verify correct values
- Lock/Unlock and swact
[PASS] Install DC (1 subcloud AIO-SX)
Story: 2011027
Task: 49627
Change-Id: I52d66804560b6f10bcad62b20485ac8d17b6b85f
Signed-off-by: Andre Kantek <andrefernandozanella.kantek@windriver.com>
The release stx.9 with FQDN support for MGMT network
uses the hieradata with the new pattern:
<hostname>.yaml
But the release stx.8 is still using the old name:
<mgmt_ip>.yaml
During an upgrade controller-0 want to update
the <mgmt_ip>.yaml and controller-1 wants to use
the <hostname>.yaml, so it is necessary to change
the code to use/update the right hieradata.
Additionally, during an upgrade the active
controller running the old release can't resolve
the FQDN (i.e: controller.internal ), for this
reason during the controller-1 upgrade, the FQDN
can not be used.
Test Plan:
IPv6 AIO-SX fresh install
IPv6 AIO-DX fresh install
IPv4 AIO-SX upgrade from previous release
without story 2010722 to new release
that has the story 2010722 (not master)
IPv4 AIO-DX upgrade from previous release
without story 2010722 to new release
that has the story 2010722 (not master)
IPv4 STANDARD upgrade from previous release
without story 2010722 to new release
that has the story 2010722 (not master)
IPv6 AIO-DX upgrade from previous release
without story 2010722 to new release
that has the story 2010722 (not master)
IPv6 DC lab upgrade from previous release
without story 2010722 to new release
that has the story 2010722 (not master)
Story: 2010722
Task: 48609
Signed-off-by: Fabiano Correa Mercer <fabiano.correamercer@windriver.com>
Change-Id: I555185bea7fadb772a4023b6ecb4379e01e0f16c
Include health check for 'system-local-ca' overall sanity. Verify:
- Secret fields are in place and valid;
- ClusterIssuer is in place;
- RCA certificate is trusted by platform;
- ICA certificate chain can be verified;
- Expected certificates (Docker Registry, REST API / GUI and Local
OpenLDAP) are managed by cert-manager, are in place and their chain
can be verified.
In case of failures, a warning informing the user that the
certificates are expected to be managed by cert-manager before
upgrading will also be displayed after the specific error description.
Test plan:
(System deployed with create_platform_certificates enabled)
PASS: Perform 'system health-query-upgrade' command.
Verify message "Platform Issuer and expected certificates are
healthy: [OK]" is displayed.
PASS: Replace 'ca.crt' field from system-local-ca secret to an empty
string.
Perform 'system health-query-upgrade' command.
Verify message "Platform Issuer and expected certificates are
healthy: [OK]" is displayed.
PASS: Delete 'system-registry-local-certificate'.
Perform 'system health-query-upgrade' command.
Verify message "Platform Issuer and expected certificates are
healthy: [Fail]" is displayed, followed by the error description
and a warning for the user to perform the CA update procedure
(cert-manager migration).
PASS: Remove ClusterIssuer system-local-ca.
Perform 'system health-query-upgrade' command.
Verify message "Platform Issuer and expected certificates are
healthy: [Fail]" is displayed, followed by the error description
and a warning for the user to perform the CA update procedure
(cert-manager migration).
PASS: Uninstall system-local-ca's RCA ssl_ca certificate (system
certificate-uninstall -m ssl_ca).
Perform 'system health-query-upgrade' command.
Verify message "Platform Issuer and expected certificates are
healthy: [Fail]" is displayed, followed by the error description
and a warning for the user to perform the CA update procedure
(cert-manager migration).
PASS: Remove any field from system-local-ca secret.
Perform 'system health-query-upgrade' command.
Verify message "Platform Issuer and expected certificates are
healthy: [Fail]" is displayed, followed by the error description
and a warning for the user to perform the CA update procedure
(cert-manager migration).
PASS: Remove secret system-local-ca.
Perform 'system health-query-upgrade' command.
Verify message "Platform Issuer and expected certificates are
healthy: [Fail]" is displayed, followed by the error description
and a warning for the user to perform the CA update procedure
(cert-manager migration).
Story: 2009811
Task: 49606
Change-Id: I0190d6b9092351a61a0b92bab1ea107bb05f8633
Signed-off-by: Marcelo Loebens <Marcelo.DeCastroLoebens@windriver.com>
To speed up bootstrap, the puppet classes being applied during
initial inventory (openstack::keystone::endpoint::runtime and
openstack::barbican::runtime) were translated to python,
leveraging the possibility of using the same keystone authentication
for all requests, greatly reducing the necessary time to create
users/services/endpoints. The time was reduced from around 10 minutes
to 8 seconds.
Test plan:
Note: The "deploy" steps involve install, bootstrap and config post
bootstrap until the system was unlock/enabled/available.
Additionally, the users/services/endpoints list was compared
to a system deployed by previous method and verified the
output is the same.
- PASS: Deploy a SX system
- PASS: Deploy a DX system
- PASS: Deploy a DC system with 2 system controllers and 1 subcloud
- PASS: Repeat the previous tests re-executing the bootstrap before
initial unlock and changing oam_foating_ip on localhost.yml
to trigger a network reconfuration. Verify the bootstrap
finished without errors, the system was unlocked
successfully and the public endpoints were updated with the
new oam ip.
- PASS: On the subcloud, verify the docker registries were correctly
stored in barbican secrets and can be retrieved by
"openstack secret list".
Story: 2011035
Task: 49599
Change-Id: I0c6714516222d60a9700b68e01492d607d7e8f15
Signed-off-by: Victor Romano <victor.gluzromano@windriver.com>
This change provides methods to add a host as a subject to the
clusterrolebinding resource named system:node in kubernetes.
For static CPU allocation, a new mechanism
(see https://review.opendev.org/c/starlingx/integ/+/907637 for
details) to classify a pod as a platform will optionally make use of
namespace labels.
Before allocation, kubelet will request namespace information from the
Kubernetes API. For authenticating itself to the kube-apiserver, kubelet
uses /etc/kubernetes/kubelet.conf as a KUBECONFIG. The file contains
auth info of user system:node:<hostname> for the corresponding kubelet.
The user is not authorized to access the namespace information by
default. To provide the access, the user is added to the subjects list
in the system:node clusterrolebinding during host-add operation.
The user is removed from the subjects list during host-delete
operation.
Test Plan:
AIO-DX, Standard and Storage lab:
PASS: Host is added to the system:node clusterrolebinding object
after the host-add operation.
PASS: From all hosts, namespaces can be listed using 'kubectl'
and /etc/kubernetes/kubelet.conf set as a KUBECONFIG file.
PASS: Host is removed from the clusterrolebinding object after the
host-delete operation.
PASS: Host is added to the system:node clusterrolebinding object
even if kubernetes networking start is delayed after controller-0
unlock (retry mechanism).
Story: 2010612
Task: 47513
Change-Id: I7f08f1b0bacbfb4f3f0646d564982d0be6b2bce4
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
A periodic audit for K8S cluster that checks the health of the
endpoints: APISERVER, SCHEDULER, CONTROLLER and KUBELET.
The audit will set/clear K8S cluster health alarm every
3 minutes.
Test Plan:
PASS: Trigger K8S cluster down alarm by manually modifying
/etc/kubernetes/manifests/kube-apiserver.yaml configuration
file to break the K8s cluster.
Verify that alarm is raised within 3 minutes.
PASS: Restore the manually modified configuration. This will
restore K8S service.
Expect to see the alarm cleared within 3 minutes.
PASS: Fresh install on AIO-SX, and checked alarm audit log.
PASS: With K8S cluster down (for several minutes), initiate platform
upgrade.
Verify that k8s health check blocks the upgrade due to 850.002
alarm.
Story: 2011037
Task: 49534
Depends-On: https://review.opendev.org/c/starlingx/fault/+/907054
Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/907345
Change-Id: I958dbb46f151df602030bd2d7576b3b3705b8ca2
Signed-off-by: rakshith mr <rakshith.mr@windriver.com>
Application status was misleading after a failed override update with
illegal values. Application should be in failed (apply-failed) state,
and alarm should be raised accordingly. Instead, we're led to believe
that the update was completed successfully.
The solution consists of adding a default delay to the system of 60
seconds before changing the helmrelease status. This way we ensure
that reconciliation has already been called.
This also ensures that any application can override this default
value via metadata. Just create a variable with the same name with
the amount of time that is needed.
Test Plan:
PASS: Build-pkgs && build-image
PASS: Upload, apply, delete and update nginx-ingress-controller
PASS: Upload, apply, delete and update platform-integ-apps
PASS: Upload, apply, delete and update metrics-server
PASS: Update user overrides (system user-override-update) with illegal
values. When reapplying the app it should fail.
PASS: Update user overrides (system user-override-update) with correct
values. When reapplying the app it should complete successfully.
PASS: If the app has the fluxcd_hr_reconcile_check_delay key in its
metadata, the system's default delay value must be overwritten.
Closes-Bug: 2053276
Change-Id: I5e75745009be235e2646a79764cb4ff619a93d59
Signed-off-by: David Bastos <david.barbosabastos@windriver.com>
Adding coredump process_size_max, external_size_max and
keep_free default service parameters so coredump service is configured
with default values from the start, keeping it explicit for the user
what is configured on a fresh install.
Test plan
PASS: AIO-SX install, bootstrap and initial unlock
PASS: Verify if coredump service parameters are added after initial
unlock.
PASS: Verify if coredump config file is changed after initial unlock
Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/897856
Closes-bug: 2039064
Change-Id: I13b1c1e0d6c34b34cf6ed3f1cb86c8511ac24b44
Signed-off-by: Heron Vieira <heron.vieira@windriver.com>
Configuration of NTP servers post deployment requires all hosts to be
locked/unlocked to clear the configuration out of date alarm. However,
external NTP server configuration is only applicable to controller hosts
This fix scoped to controller hosts only
TEST PLAN:
PASS: Standard Configuration with controller storage
configuration of NTP servers need only controller hosts to lock/unlock to clear the configuration out of date alarm.
Closes-bug: 2052994
Change-Id: I468717aa609458251137ff43cd6613764c71bcf4
Signed-off-by: rummadis <ramu.ummadishetty@windriver.com>
Zuul