Upload process should not require the release id to be part of patch
filename. This change allows us to directly get release id from the
metadata inside the patch file.
Test Plan:
PASS: Successfully upload patch with different filename
from release id using software upload command.
PASS: Successfully upload patch with filename same as release id
using software upload command.
PASS: Successfully upload multiple patches with different filename
from release id using software upload-dir command.
PASS: Successfully upload multiple patches with filename same as
release id using software upload-dir command.
Story: 2010676
Task: 49868
Change-Id: Ibd5bcf9b8797b5de0eef3e46313055cc141da0b2
Signed-off-by: Dostoievski Batista <dostoievski.albinobatista@windriver.com>
This commit validates pre-conditions when software
deploy-host <hostname> is issued.
The pre-conditions are:
The host state is in pending state.
The specified host target is locked and online.
Nodes deployed to major release in the order below in DX system:
Controller-1 -> Controller-0 -> Storage nodes -> Compute nodes.
Nodes deployed to patch release in the order below in DX system:
Controllers -> Storages nodes -> Compute nodes.
Test Plan:
PASS: Deploy host to controller-1 validation.
PASS: Validated target locked and online checker.
PASS: Validated start-done deploy state checker.
Story: 2010676
Task: 49795
Change-Id: I8fe8faa85c594472bb6c8c021416205bf4a61fbb
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
This change introduced state machines for release state, deploy state
and deploy host state.
This change removed the direct reference to the software metadata from
software-controller and other modules. Replaced with encapuslated
release_data module.
Also include changes:
1. removed required parameter for software deploy activate and software
deploy complete RestAPI.
2. ensure reload metadata for each request
3. added feed_repo and commit-id to the deploy entity, to be
subsequently passed to deploy operations.
4. fix issues
TCs:
passed: software upload major and patching releases
passed: software deploy start major and patching releases
passed: software deploy host (mock) major and patching release
passed: software activate major and patching release
passed: software complete major release and patching release
passed: redeploy host after host deploy failed both major and
patching release
Story: 2010676
Task: 49849
Change-Id: I4b1894560eccb8ef4f613633a73bf3887b2b93fb
Signed-off-by: Bin Qian <bin.qian@windriver.com>
During the management network reconfiguration, the system is restarted
to controller_config script runs the puppet code and update
all services to use the new mgmt IP address.
But the sw-patch services start before the controller_config.
When they start they get the mgmt_ip using the python socket lib that
uses the IP address from the /etc/hosts.
But /etc/hosts at that time is not updated yet, so it get the old
management network IP.
To fix this issue, the sw-patch services will wait for the puppet
code to be applied to make sure the /etc/hosts and new management
network IPs were installed in the system.
Tests done:
IPv4 AIO-SX fresh install
IPv4 AIO-DX fresh install
IPv4 DC with subcloud AIO-SX fresh install
IPv4 AIO-SX mgmt reconfig and apply a non-reboot-required patch
IPv4 AIO-SX mgmt reconfig and apply a reboot-required patch
IPv4 subcloud AIO-SX mgmt reconfig and apply a non-reboot-required patch
IPv4 subcloud AIO-SX mgmt reconfig and apply a reboot-required patch
For this test the sw-patch was in failed state after the reboot,
It happens even without the mgmt reconfig and this fix
Partial-Bug: #2060066
Story: 2010722
Task: 49827
Depends-On: https://review.opendev.org/c/starlingx/config/+/914710
Change-Id: Ie544425513ef4fede73b4b55770ad6857cdf7eed
Signed-off-by: Fabiano Correa Mercer <fabiano.correamercer@windriver.com>
This commit adds the capability to run deploy host
for a major release deployment (release upgrade).
To achieve this, this commit essentially changes
some code that is already used by patching to allow:
1. Create a new remote pointing to the to_release feed ostree
2. Pull the to_release ostree commit to sysroot ostree
3. Deploy the to_release ostree commit
This commit also includes some additional steps for
sysinv/puppet integration with USM, and fixes minor
flake8 issues on the files that are being changed.
Test Plan
PASS: run "deploy host" for major release deployment
successfully on AIO-SX
PASS: run "deploy host" for major release deployment
successfully on AIO-DX
PASS: (regression) run "deploy host" successfully for a
patch release
Story: 2010676
Task: 49787
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Change-Id: Ib8b08d1cd85dcad7d6fc858e2fae623b5900cffc
This commit is to raise the deploy state out of sync alarm
when the deploy state in the software.json files in both controllers
are different.
The deploy state is checked every 30 seconds during the
deploying stage. If they are insync, the alarm will be cleared.
Depends-on: https://review.opendev.org/c/starlingx/fault/+/913581
Test Plan:
PASS: the alarm is raised when the state is out of sync
in both DX and SX
PASS: the alarm is cleared when the state is in sync in
both DX and SX
Task: 49737
Story: 2010676
Change-Id: Ic31c7166135d03591fa4696445783895254dfc95
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
The commit id changes after apt-ostree command is run
in software deploy start. We are updating that commit
id to metadata dictionary, which is accessed by software
show rest api. Updating the new commit id for remove case
also to metadata dictionary.
Test Plan:
PASS: Check ostree commit id after apply case
of software deploy start in software show
PASS: Check ostree commit id after remove case
of software deploy start in software show
PASS: Check commit id for a prepatched iso
Story: 2010676
Task: 49818
Change-Id: I247c35f2e95bf013e0af0c684e17c4c38d0c2802
Signed-off-by: sshathee <shunmugam.shatheesh@windriver.com>
This commit is to replace legacy sysinv endpoints for platform
upgrade with new USM endpoints.
New endpoints:
get_software_upgrade: get from/to versions and deploy state
for upgrade.
get_software_host_upgrade: get current/target versions and
deploy state for the given host.
Test Plan:
PASS: call the endpoints using curl before deploy start
PASS: call the endpoints using curl after deploy start.
Task: 49766
Story: 2010676
Change-Id: I574da8f60e1cf9fc046f5c6a727f7e17fe8c55f7
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
Fix software deploy show and deploy host-list not displaying information
issue.
Also default the deploy host state to pending when a deploy host entity
is created.
Story: 2010676
Task: 49645
TCs:
passed: software deploy show and software deploy host-list show
deploy data after deploy start command accepted.
passed: display "No deploy in progress" for software deploy show
when there is no deploy.
Change-Id: I9dc50804c66d5cb07df7717fd6623c23d0fca522
Signed-off-by: Bin Qian <bin.qian@windriver.com>
This commit fix the deploy precheck <release> command output which
is returning one line with an blank "Error:" when the output is correct.
Test Plan:
PASS: Deploy precheck <release> returning without error.
PASS: Deploy precheck <release> returning expected error.
PASS: Deploy start failed with expected error.
PASS: Deploy start
Note: Exit code 3 was chosen based on exit codes with special meaning
which the 1 is for catchall general errors and 2 for misuse of shell
builtins. The number 3 is not allocated with special meaning so it
was chosen for unhealthy precheck.
Closes-Bug: 2056106
Change-Id: Ifed48157f7810eec2881d8a9e011eae8941f3427
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
When using Keystone auth for software cli, only user with 'admin' role
is allowed to run any commands. When using software cli without 'sudo',
all software commands require user with 'admin' role.
This review also update the exception handling and error reporting.
Test Plan:
PASS: A Keystone user in the 'admin' project with 'admin' role should
be able to run ALL 'software' commands WITHOUT SUDO
PASS: A Keystone user in the 'admin' project with only 'member' and/or
'reader' role should NOT be able to run ANY 'software' commands
WITHOUT SUDO
Story: 2010676
Task: 49754
Change-Id: I46653021b1a82bccded5eb870dc0907cd5c2351b
Signed-off-by: Joseph Vazhappilly <joseph.vazhappillypaily@windriver.com>
There are additional steps needed to enable "deploy host"
operation for major release upgrades. This commit adds these
additional steps, which are:
1. Create TO release platform config directory, done during
upgrade-start on legacy upgrade procedure
2. Create TO release rabbitmq directory, done by upgrade
manifest after controller-1 reinstall on legacy upgrade
The commit also fixes some issues:
1. shell-utils logging functions logging nowhere after
being sourced by other scripts
2. sync-controllers-feed script was only syncing the
ostree_repo directory instead of the full feed content
3. major release deployment scripts ran from different
places, now all scripts are executed from the TO release
feed directory, or from checked out TO release ostree
repo in case of chroot
4. change umount command from chroot_mounts to lazy umount
Note: Since the "deploy host" endpoint for major release
deployment is not yet implemented, the test plan will have
test cases that simulate the "deploy host" operation.
Test Plan
PASS: simulate "deploy host" successfully for AIO-SX
PASS: simulate "deploy host" successfully for AIO-DX
Depends-on: https://review.opendev.org/c/starlingx/config/+/913715
Story: 2010676
Task: 49703
Change-Id: Ib6ae49b3590a1e50acb305ac7482e28bcc4de403
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Insert ostree commit id into metadata file after apt-ostree
runs. "previous_commit" tag is for commit id before apt-ostree
is executed and "commit" tag is for latest commit id after it
is executed.
Test Plan:
PASS: Run software deploy start for patch and check commit id
is inserted in metadata file
Story: 2010676
Task: 49753
Change-Id: I8fc3c33430188449d852770824e3ecd765583dd7
Signed-off-by: Charles Short <charles.short@windriver.com>
Signed-off-by: sshathee <shunmugam.shatheesh@windriver.com>
This commit is to fix 'software upload-dir' not having
respond that contains uploaded release info.
Test Plan:
PASS: upload files using 'software upload-dir'
Task: 49634
Story: 2010676
Change-Id: I635554fdbdb80fe31a38d1170202405fe6f32d3a
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
This change adds checks before deleting software releases:
1. software release is available or unavailable
2. When it is on a system controller, the release is not being used by a
subcloud
This change also update the following:
1. removed the exception handling in controller level, moved to
exception hook
2. CLI code to display HTTP error, only handles 500 status code, by
displaying message from API, all other 4xx, 5xx status code display
HTTP error directly.
3. ensure CLI return 1 for unsuccessful requets (status code 500)
4. fixed some minor issues
Story: 2010676
Task: 49657
TCs:
passed: observe delection rejected because of release not found,
release is not in available or unavailable state.
passed: delete an available release
passed: on system controller, successfully delete scenarios
passed: (simulated) on system controller with subcloud, delete
release used by subcloud is rejected
Change-Id: I306b1d8604113b92d907384844e8e8107835a463
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Currently upload-dir logic was uploading only the files from
the last directory passed as argument to the command, and it's
help text and output was divergent from the similar "software
upload" command.
This commit fixes the logic, allowing uploading from multiple
directories correctly, and fixes the help text and output to
align with "software upload".
Test Plan
PASS: show help text for "software upload-dir"
PASS: upload one directory containing patches
PASS: upload one directory containing iso + patches
PASS: upload multiple directories containing patches
PASS: upload multiple directories containing iso + patches
Story: 2010676
Task: 49693
Change-Id: I5886e8c5c55355e24ec471c4ae47e91ec3c84dfd
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
When doing load-import operation, the /var/www/pages/feed/ files must
be replicated in both controllers. The files were being created in
controller-0, but they were not copied to controller-1.
This fix added a way to get all the folders from one controller to
another and sync them with ostree the explanation is in the following
commit: I42c274079631a3c197015e636e03de1bc96de28b
Test-plan:
PASS: After a load-import, the inactive controller should have the
feed repo created.
Closes-bug: 2045321
Change-Id: I260951461d2c19550e9f57ad7ab9ec66a25de5bb
Signed-off-by: Lindley Werner <Lindley.Vieira@windriver.com>
'software deploy host <in-service-patch-release>' should NOT fail if
there is no restart-script (post-deploy script).
If the rsync does not find the scripts, does not raise an exception.
Test-plan:
PASS: software deploy host <in-service-patch-release> successful
Closes-bug: 2058393
Change-Id: I1b8cac9e0401c3f64c7334139c62bf272e9aeb56
Signed-off-by: Lindley Werner <Lindley.Vieira@windriver.com>
This commit is to include a versioned deploy precheck script into
/opt/software/scripts/rel-<sw_version> to be able to run the correct
precheck code for a specific release.
Along with this commit, the precheck api is changed to use the new
location of the script, and the precheck script is changed to add
support to patch-only prechecks, and as a consequence, minor wording
changes were done to return more accurate messages to the user.
1. For the iso upload scenario:
The upload process will copy all scripts under
<iso_root>/upgrades/software-deploy to /opt/software/rel-<ver>/scripts
2. For the patch upload scenario:
The upload process will check if patch contains the deploy-precheck
script. If it does, then the script is copied to
/opt/software/rel-<ver>/scripts, if not then a symlink will be created
to the patch 'required patch' versioned precheck script.
Notes:
- iso (prepatched or not) will always come with deploy-precheck script
- <ver> assumes the format MM.mm.pp
Test Plan:
PASS: Upload multiples patches, both with and without precheck
scripts, and verify the versioned directories are created
and the precheck script is created as expected
PASS: Run deploy precheck for an iso release and verify the upgrade
precheck output is returned as expected
PASS: Run deploy precheck for a patch release and verify the patch
precheck output is returned as expected
Depends-on: https://review.opendev.org/c/starlingx/metal/+/911595
Story: 2010676
Task: 49263
Change-Id: I04ff89d43579fd71592f7ec534db57a1ead79483
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
Co-signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
This change add support for https with SSL protocol and certificate.
The USM client can work with either insecure (disable SSL/TLS
certificate verification) or with SSL certificate. The client is
also modified to support sessions and versions. These changes are
adapted from cgtsclient.
This adds three authorization modes, [token, keystone & local-root].
In token mode, a keystone token and software-url is used for auth.
Eg: $ software \
--software-url "http://192.168.204.1:5497" \
--os-auth-token "${TOKEN}" list
In keystone mode, sourced keystone configs in env is used for auth.
Eg: $ source /etc/platform/openrc; software list
In local-root mode, authorization is by privileged user (root/sudo)
of the controller where software application is running.
Eg: $ sudo software list
Optional arguments specific to https:
-k, --insecure
--cert-file CERT_FILE
--key-file KEY_FILE
--ca-file CA_FILE
Example usage for insecure connection:
software -k list
Story: 2010676
Task: 49666
Test Plan:
PASS: Verify software cli output for http endpoints
PASS: Verify software cli output for https endpoints
Change-Id: I2e2ff115b8d03cddb02e026da84f389918238dab
Signed-off-by: Joseph Vazhappilly <joseph.vazhappillypaily@windriver.com>
This commit is to allow active controller periodically sending deploy
state message to the software agent on its peer controller. The
interval is set to 30 seconds.
Test Plan:
PASS: build and deploy the iso
PASS: start new deployment, file is synced in both controllers
Task: 49655
Story: 2010676
Change-Id: Ie95c5a7d45b3d88331569ca52d64d40a4f39d6c3
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
This commit fixes an issue that was preventing load import
script from copying the TO release pxe files correctly from
the load, and also gives pxe-update-*.sh script execution
permission to avoid permission denied errors during host-unlock.
Test Plan
PASS: upload TO release load, verify the TO release pxe.cfg.linux
files are copied to /var and that /etc/pxe-update script is
updated to 755
Story: 2010676
Task: 49698
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Change-Id: I222b484d3a28c603ed8d7c42d0405481086735f0
This commit add some changes on deploy host-list.
Adds a function to query the hostnames from sysinv
to deploy host-list entities during deploy start.
Changes endpoint to GET verb, the endpoint return in case
of no deployment in progress it will an empty list
and at CLI will print "No deploy in progress." In case
there is a deployment in progress the CLI will behave
the same but the endpoint will return the data below:
[{'hostname': '<hostname>',
'software_release': '<sw_version>',
'target_release': '<sw_version>',
'reboot_required': 'str<true/false>',
'host_state': '<host_deploy_state>'}]
This commit also changes the wait_for_install_complete function
to follow the new state logic.
Note: Software deploy host is affected by this change related
to states and will need a future commit regarding state changes
during deploy start and deploy host itself.
Test Plan:
PASS: Software deploy host-list with/without deployment in progress.
PASS: Deploy_host creation/update/get/delete.
PASS: Collect hostnames to deploy host entities during deploy start.
Story: 2010676
Task: 49586
Change-Id: I7b03df30fd8e326637a3ffc031e0fdf543cb6356
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
This commit add some changes to deploy show endpoint, the name was
changed to just deploy with GET verb and also changes the deploy to
be saved as a list of dict to attend the API requirements. Now the
api accepts from_release and to_release as optional params, in case
it is provided the endpoint will return a dict otherwise will return
a list of dict.
Test Plan:
PASS: Create deploy
PASS: Update deploy
PASS: Software deploy start
PASS: Software deploy show
Story: 2010676
Task: 49645
Change-Id: I68d243c05da88c7eecf2d866c7202c3c7be51a2b
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
This is to remove the unnecessary routing based on request
method in which the respond can't be decoded properly by
the sm-api
Test Plan:
PASS: run system host-swact
Task: 49661
Story: 2010676
Change-Id: Ife8862b7d5666de3a3dafff582dbb4c27e1adafa
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
The constraints file used for tox.ini was removed. We need to
update the file to use the StarlingX Debian constraints file.
Test Plan:
PASS - Run tox command
Closes-bug: 2055734
Change-Id: I306be11f6edc4538cbb3f7a164bac9e1ad08501f
Signed-off-by: Hugo Brito <hugo.brito@windriver.com>
This change is to create 2nd thread to provide concurrent service. In a
different commit [1], the haproxy is to be configured to distribute the
slow requests to the 2nd thread, and the fast requests to the primiary
thread.
TCs:
passed: concurrent keystone requests of "software upload/
deploy precheck/deploy start" and "software list/deploy show/
deploy host-list"
passed: keystone authenticated "software deploy precheck"
request completed.
Story: 2010676
Task: 49647
[1] https://review.opendev.org/c/starlingx/stx-puppet/+/910644
Change-Id: I0e8e8ac1b5177f1bbf40e047335c075b0a471fc1
Signed-off-by: Bin Qian <bin.qian@windriver.com>
The usm_load_import script is currently being copied to two
different places needlessly during the upload process for a
major release: (/etc/software and /opt/software/rel-<ver>/bin).
This commit keeps only the copy to the versioned directory under
/opt/software/rel-<ver> and also changes the code to use the
script from this location accordingly.
Test Plan
PASS: run 'software upload' successfully
Story: 2010676
Task: 49624
Relates-to: https://review.opendev.org/c/starlingx/update/+/910027
Change-Id: I0178007721505ff8e2c2244964692b79e8aafea9
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
This commit improves logging during deploy start by:
1. Creating a common module to be sourced by shell scripts to
load general-use functions, thus reducing code duplication
between the scripts
2. Replacing plain "echo" commands on the scripts by logging
functions present on the common module
3. Adding timestamps to the log messages
4. Centralizing all scripts logs into software.log, favouring
the troubleshooting, now that log lines contain timestamps
and the process/script that generated them
This commit also deletes the ostree_mounts.yaml file since
it would be used by apt-ostree integration, which was dropped.
Test Plan
PASS: run deploy start successfully and verify that deploy start
log messages are logged with the expected format
Story: 2010676
Task: 49607
Change-Id: I0bdebde8147faa5b29a642e35bfaf26e9862ed0a
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
This commit improves logging during "deploy start" by:
1. Creating a function on upgrade_utils to configure logging
2. Replacing basic log configuration with the function from
previous item
3. Configuring software/utilities/migrate.py module to
effectively log in a file
4. Suppressing meaningless SQL commands output generated
during databases migration
Test Plan
PASS: run deploy start/upload, verify the logs from the
scripts are created as expected
Story: 2010676
Task: 49620
Depends-on: https://review.opendev.org/c/starlingx/update/+/910262
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Change-Id: Ib78bdcd1dd98b0f221985a429b6e9eeddeba837b
This commit create a generic method to be used when a deploy state
is changed. For every modification on the file it will call the message
class to send the deploy state dataset to peer controller.
Test Plan:
PASS: Deploy state change sent to peer controller on deploy start
Depends-on: https://review.opendev.org/c/starlingx/update/+/904367
Story: 2010676
Task: 49335
Change-Id: Idc06074f059c2f0d683b5f291e594f6a1555d6d5
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
The deploy-precheck script is now stored in a versioned
directory under /opt/software/rel-<ver>, so there is no
need to check if the feed directory exists.
This commit removes deprecated variables and checks that are
not needed as result of the versioning of the release scripts
directory.
Test Plan
PASS: run "software deploy precheck" successfully
Story: 2010676
Task: 49633
Change-Id: I42b00143234f4aa5a3c871bbd6900f3bfce43c8d
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Zuul