Commit Graph

75 Commits

Author SHA1 Message Date
Scott Little 7c4d01df61 Remove CentOS/OpenSUSE build support
StarlingX stopped supporting CentOS builds in the after release 7.0.
This update will strip CentOS from our code base.  It will also remove
references to the failed OpenSUSE feature as well.

Story: 2011110
Task: 49944

Change-Id: I8cd4e23ab83f2fe064fa1f88553eb32a69a67265
Signed-off-by: Scott Little <scott.little@windriver.com>
2024-04-26 13:45:07 -04:00
Don Penney 320cc40de8 Add auto-versioning to starlingx/config packages
This update makes use of the PKG_GITREVCOUNT variable to auto-version
the packages in this repo.

Change-Id: I3a2c8caeb4b4647608978b1f2ccfcf0661508803
Depends-On: https://review.opendev.org/727837
Story: 2006166
Task: 39766
Signed-off-by: Don Penney <don.penney@windriver.com>
2020-05-20 14:11:21 +00:00
Teresa Ho d141e954fa Sysinv extensions for FPGA support
This update adds cli and restapi to support FPGA device
programming.

CLI commands:
system device-image-apply
system device-image-create
system device-image-delete
system device-image-list
system device-image-remove
system device-image-show
system device-image-state-list
system device-label-list
system host-device-image-update
system host-device-image-update-abort
system host-device-label-assign
system host-device-label-list
system host-device-label-remove

Story: 2006740
Task: 39498

Change-Id: I556c2e7a51b3931b5a66ab27b67f51e3a8aebd9f
Signed-off-by: Teresa Ho <teresa.ho@windriver.com>
2020-05-13 16:20:37 -04:00
Andy Ning c1c18871d7 Support multiple CA certificates installation
This update enhanced sysinv certificate install API to be able to
install multiple CA certs from a file. The returns from the API call
indicates the certs actually installed in the call (ie, excluding these
that are already in the system). This is neccessary especially for DC to
support multiple CA certs synchronization.

This update also added sysinv certficate uninstall API. The API is to
be used to remove a particular CA certficate from the system, identified
by its uuid. The API returns a json body with information about the
certificate that has been removed. This is required by DC sysinv api
proxy for certificate deletion synchronization, since DC tracks subcloud
certificates resource by signature while the uninstall API request
contains only uuid.

The uninstall API only supports ssl_ca certificate.

cgtsclient and system CLI are also updated to align with the updated
and new APIs. User can use "system certificate-install ..." to install
one or multiple CA certificates, and "system certificate-uninstall ..."
to remove a particular CA certificate from the system.

When multiple CA certificates are installed in the system,
"system certificate-list" will display each of the individual
certificates.

THe sysinv certificate configuration API reference is updated with the
new uninstall API. Unit tests are added for CA certificate install and
delete APIs.

Change-Id: I7dba11e56792b7d198403c436c37f71d7b7193c9
Depends-On: https://review.opendev.org/#/c/711633/
Closes-Bug: 1861438
Closes-Bug: 1860995
Signed-off-by: Andy Ning <andy.ning@windriver.com>
2020-03-20 10:32:43 -04:00
Jim Somerville de23dcfd05 Security: Allow disabling of spectre v1 swapgs mitigation
Most of the v1 mitigation is baked into the kernel and not
optional.  The swapgs barriers are, however, optional.
They have a negative performance impact so we disable them
by using the nospectre_v1 kernel bootarg.

Change-Id: Ia5938249ad0f0a53435251e505dac843b923ad62
Closes-Bug: 1860193
Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>
2020-01-28 18:13:32 -05:00
Al Bailey 89e2975eb8 Add missing requirements for sysinv
The sysinv code has imports for python components but does
not have all of those specified as rpm dependencies.

Adding:
 python-jsonpatch (used by several sysinv/api/controllers/v1/*)
 python-keystoneauth1 (used by sysinv/common/fm)
 python-keystonemiddleware (used by sysinv/api/acl)
 python-oslo-serialization (used by sysinv/conductor/manager)
 python-oslo-service (used by sysinv/common/wsgi_service)
 python-paste (used by api config file)
 python-psutil (used by sysinv/api/controllers/v1/host)
 python-requests (used by sysinv/common/ceph)
 python-retrying (used by sysinv/agent/manager)
 python-stevedore (used by sysinv/puppet)
 python-webob (used by sysinv/api/hooks)

This is a protective commit, since this only works because other
packages are pulling in the dependencies, and those packages can be
updated or removed in the future, thus breaking sysinv.

Change-Id: I9457e3817ae8d219ef0c582b0b0177510a8ec4b0
Story: 2004515
Task: 37985
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2020-01-09 12:33:29 -06:00
Kristine Bujold 1a220c2e7d Increase size of service-parameter field value
This submit increases the size of the “value” field for the
service-parameter table from 255 to 4096 characters. This is
to accommodate registries that can be saved as a list in
this field.

This change was tested in lab with a size of 4096 and 4097
for no_proxy's value.

Closes-Bug: 1856236
Change-Id: I5067f7ee1e5f8b532045d0736402080bad39be72
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-12-17 12:53:21 -05:00
Jim Gauld 539c29d717 AIO System Controller CPU assignment changes
This changes AIO running DC system controller CPU assignment so that
all logical cpus spanning all numa nodes are configured as Platform
function.

Change-Id: I6be3c8f63661786b193b0ed7a72781a7c48808cb
Closes-Bug: 1855920
Depends-On: https://review.opendev.org/#/c/698321/
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
2019-12-11 11:42:04 -05:00
Bin Qian fde5811001 Add per-host board management protocol setting in sysinv
Use bm_type field to store board management protocol setting,
available bm protocols:
  redfish
  ipmi
  dynamic
  none (bm is not provisioned)

The old service parameter bmc_access_method is removed.

Partial-Bug: 1852328
Change-Id: I5097e53f6fc1bfbe23d2a1b765b5bc0e25423c22
Signed-off-by: Bin Qian <bin.qian@windriver.com>
2019-12-05 09:00:25 -05:00
Kristine Bujold 65dfc46b9b Create sysinv-utils for generating host-overrides
This commits creates a new utility command /usr/bin/sysinv-utils
that can be used by ansible playbooks to generate the host overrides
file that will be used for upgrades and backup/restore.

This commits also enhanced the following CLI commands to include a
--column and --format options for better CLI outputs data manipulation.
	host-show
	host-list
	service-parameter-list
        application-show

These two new options are optional. The default behavior is not changed.

Story: 2006590
Task: 36892
Change-Id: I36b8ea411a4eb74d6e1a2194982025905e9299e5
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-11-20 10:54:39 -05:00
Jim Gauld c4bdce8ca2 Set uniform PM-QoS value for all cpus on lowlatency nodes
We observed poor cyclictest results on lowlatency nodes. This was
due to the fact that we were trying to save power by only marking
in-use CPUs as low-latency, but there was an underlying issue where
any high-latency CPUs result in a latency hit for all other CPUs.

As a workaround to provide consistent low-latency performance, set
all CPUs to only sleep as deep as C1 on lowlatency nodes.

Change-Id: Idd627eee6e7ff95939e428b180d07d8ab1cadad2
Partial-Bug: 1845735
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
2019-10-25 21:01:01 +00:00
Kristine Bujold 3194f2b559 Refactor upgrade re-tries on failure path
If image upgrade or downgrade fails for tiller or kubernetes
networking, the sysinv conductor will re-try in an hour until
success.

Tested a full install in an AIO-SX lab.

Tested an upgrade of the calico (kube networking) and tiller image.

Tested an upgrade failure of the calico image (by using an invalid
version number), verified there are retries every hour until success.

Tested an upgrade failure of the tiller image (by using an invalid
version number), verified there are retries.

Ensure that after the conductor is restarted or is in an upgrade kube
components failure path, that it is not locked by the greenthread.
Tested this by adding/removing a label to controller-0 (system
host-label-assign, system host-label-remove).

Tested upgrade after AIO-SX lab controller lock/unlock.

Tested upgrade after Standard lab controller lock/unlock.

Tested  controller swact while in upgrade failure: verified that
sysinv-conductor properly takes activity on new controller, upgrade
continued to fail as expected. Tested a successful upgrade.

Story: 2006590
Task: 36942
Change-Id: I9212f167ff4f8975c4f2df504d6850e03d4e9a6b
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-10-22 16:14:37 -04:00
Al Bailey 60bc70a668 Revert "Refactor upgrade re-tries on failure path"
This reverts commit 50f2d8c5aa.

In cases where this retry is triggered, the sysinv-conductor does not complete its startup, and causes ansible bootstrap to fail with an RPC error.

Change-Id: I062d1e85e71abb66146b66f4bcb627f814cbbc42
2019-10-18 22:45:51 +00:00
Kristine Bujold 50f2d8c5aa Refactor upgrade re-tries on failure path
If image upgrade or downgrade fails for tiller or kubernetes
networking, the sysinv conductor will re-try in an hour until
success.

Story: 2006590
Task: 36942
Change-Id: Ia3265ecb5a26db4be7778408fbd0a0d07c75e84a
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-10-17 16:21:07 -04:00
David Sullivan f17cc885c2 Allow CPU profiles with isolated cores
Update CPU profile rules to work with the constraints for isolated
cores. Use the same validation for CPU profiles and general CPU
modification.

Update the host_cpus_modify functionality for clarity. Validate and
apply all requested CPU changes in one shot.

Change-Id: I83b4771809fad9323ff8ea0aed1bc02e78ce356b
Story: 2006565
Task: 36899
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
2019-10-09 18:10:21 -04:00
Kristine Bujold 83ae827e1c Add new sysinv tables k8s upgrades
The commit creates two new sysinv tables called kube_upgrade and
kube_host_upgrade in preparation for kubernetes upgrades.

Tested an install to ensure the new tables are defined in the
database.

Story: 2006590
Task: 36941
Change-Id: I8eed72b4a6ca34a1ac96dacb789669f91829af8e
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-10-09 14:11:44 -04:00
Kristine Bujold 210cbe029a Kubernetes networking upgrade prep
This commit adds code that calls ansible-playbook
upgrade-k8s-networking.yml when the conductor starts up.

Story: 2006590
Task: 36726
Change-Id: I59e62654bc2a9b2a0980f2e2b60cc93a74a3deb6
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-10-07 10:58:02 -04:00
Kristine Bujold 18d22a950f Tiller upgrade prep
This commit spawns a new greenthread inside the _start() routing of
the conductor. This thread calls the new _upgrade_downgrade_tiller()
method which checks if the tiller image version has changed and if so
upgrade/downgrade the image. If there is an issue with downloading or
updating the new image, the code sleeps for 5 minutes and try again.

Removes references of the armada image name in sysinv.conf. Image
versions will be stored in sysinv/common/image_versions.py.

Story: 2006590
Task: 36725
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
Change-Id: I3b3037fef25d2b37c98c4dd7e82b405b41a45a23
2019-09-30 16:31:52 -04:00
Kristine Bujold 2d17e9849f Support for NTP/PTP coexistence
The NTP/PTP selection will now be done per host. NTP will the default
selection.

This commit;

-Removes the enabled flag NTP and PTP API. Updates the CLI commands
and the database.

-Adds the parameter clock_synchronization to the host API. Valid
values are ‘ntp’ and ‘ptp’. Updates the host CLI commands and the
database.

-Updates puppet to set NTP/PTP per host .

-Updates the RestAPI documentation.

Story: 2006499
Task: 36464

Change-Id: I37bbb30a014301f8786cb02e35f0a1bd39d2f4aa
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-09-17 14:45:18 -04:00
Robert Church 38abbef079 Rebase Armada to latest master
Rebasing Armada to use the latest docker image tag
8a1638098f88d92bf799ef4934abe569789b885e-ubuntu_bionic.

Change-Id: Ic48a2e053d0de7dacfd6a07d817947e11dc8d596
Story: 2006347
Task: 36105
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-08-15 16:54:51 -04:00
John Kung 168442b2e1 Create host state for determining initial inventory complete
Add host inv_state attribute to allow determination of when the
initial inventory collection has been completed.

Update references which were using disks/pvs as proxy for inventory
completion to reference the host inv_state attribute.

Description of issue (from Bug 1837097):
The system inventory agent needs to explicitly indicate that inventory
collection has finished for each host.  The current method for
determining whether a host has been inventoried successfully is to
wait for the disk/pv list to be non-empty.

That worked well until recently when the host file system feature
was merged.  The system inventory agent now collects/creates host file
systems after the disk list is populated so a provisioning system
waiting on the disk list will move ahead to unlock the node
prematurely before the host file systems have been created and reported
to system inventory.  This can lead to undefined behavior either on
the system being provisioned or the provisioning system that is
configuring the target system.

If we do not fix this properly with an explicit/deterministic flag then
we will trip over this issue each time someone adds a new inventory
collection step to the end of the system inventory agent's
initial process loop.

Change-Id: Ifdb8871a892414ee4c433cf7a6ec7e79390c6420
Closes-bug: 1837097
Signed-off-by: John Kung <john.kung@windriver.com>
2019-07-30 11:13:56 -04:00
Don Penney e5a9f0ed8e Add 1.0-17 to supported stx-openstack app version list
The formal CENGN build uses the --label option when building
the application tarballs, resulting in application versions of:
- 1.0-17-centos-stable-versioned
- 1.0-17-centos-stable-latest

These versions were added to the new application version check
in the following update:
https://review.opendev.org/670754

When the --label is not specified, however, the version does not
include a label. So most developer application build will just
have a version set to "1.0-17" (as a current example).

This update adds this version to the supported version list to
allow for developers using custom builds.

Change-Id: Ifb26e2f391a0195fc53d65b8adf8f87cb87f68e5
Closes-Bug: 1837105
Signed-off-by: Don Penney <don.penney@windriver.com>
2019-07-19 10:28:58 -04:00
Kristine Bujold e74ef5f7c4 Add new kubelet filesystem to host_fs
Add a new filesystem called "kubelet" to all hosts with a default
size of 10G. This new fs will be managed by the host_fs API.

Also made the scratch filesystem resizable on all hosts.

Tested with install of hardware Standard and AIO-DX labs. Also
tested install of a vbox AIO-SX lab.

Partial-Bug: 1830142
Depends-On: https://review.opendev.org/671120

Change-Id: I968f84b8ba7a069ec3d7027d4eb4a7355a06d9d3
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-07-17 16:18:36 -04:00
Kristine Bujold aa8b5637f3 Create new host_fs CLI and API
Create the new host_fs CLI commands and the APIs.
  system host-fs-list
  system host-fs-modify
  system host-fs-show

Remove the backup, scratch and the docker filesystems from the
controllerfs CLI as these are being managed by host-fs.

Changed the host’s boot_device and rootfs_device default values
to the full path name “/dev/sda”. Having “sda” only was causing
get_disk_capacity_mib() to fail to locate the disk.

The documentation for the API changes will be in another submit.

These changes were tested in Standard VBOX, Hardware AIO-DX,
Hardware with worker and also storage nodes. Installs and
configration of the labs were done.

Partial-Bug: 1830142

Change-Id: I2ca6adf9c5e9debaf0f4a23e67fadf47f2eaf670
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-07-09 11:49:59 -04:00
Kristine Bujold 10a595f22d Create new host_fs postgres table
Create a new host-fs postgres table to store information about a
host’s filesystem. Replicated filesystem will continue to be stored
in the controller_fs table.

This commit creates the table. The sysinv agent creates the docker,
backup and scratch filesystem for each supported host. If the
filesystems have changed the audit triggers an update.

These changes were testing with AIO-SX and Standard lab installs.

Change-Id: I8167dfe74688c05fd8b747caac3d418d2749a740
Partial-Bug: 1830142
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-07-02 13:01:14 -04:00
Tee Ngo 04a07648b4 Beef up sysinv URL validator
The current sysinv URL validator does not handle URLs
containing IPv6 address correctly. This commit fixes that.

Closes-Bug: 1833710
Change-Id: Ic5a450ede0390529e795ca0120200a0f7bbf52ce
Signed-off-by: Tee Ngo <Tee.Ngo@windriver.com>
2019-06-25 13:07:19 -04:00
Jerry Sun 4809c9f489 Upversion armada image
Upversion armada image from existing
af8a9ffd0873c2fbc915794e235dbd357f2adab1
to
dd2e56c473549fd16f94212b553ed58c48d1f51b-ubuntu_bionic

The specific image was chosen because it contained upstream
armada commit df68a90e057c2e1e3427d6b8497b437c8a4c3b7e, which
is a fix for keystone kubernetes auth. The ubuntu bionic image
was chosen because the old image was an ubuntu bionic based image.

Testing done by applying stx-openstack on standard, simplex,
and duplex systems.

Story: 2005860
Task: 33693

Change-Id: Ifd8a66d46e2dfd47ca7c5ab9807076ef43e67027
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
2019-06-21 09:47:40 -04:00
Robert Church c69942d0d7 Enable Cinder volume backups
To properly enable Cinder volume backup, the following configuration
changes are required:
- For Cinder, enable 'CephBackupDriver' as the Cinder backup_driver and
  'cinder' as the rbd_user for each Cinder backend
- For libvirt, enable Ceph and use 'cinder-volume-rbd-keyring' for the
  Ceph client user secret. This will create a libvirt secret that will
  be used with the 'cinder' user.
- For nova, enable the rbd_secret_uuid shared with libvirt and set the
  'rbd_user' to cinder.
- Update the chart group initialization sequence, so that
  'openstack-cinder' is initialized prior to 'openstack-compute-kit'.
  This is done because 'cinder-volume-rbd-keyring' is created by Cinder
  and is required by libvirt to successfully initialize.

With these configuration changes:
- Cinder volumes were created
- Cinder volumes were backed up
- Instances were booted by volume (from Cinder)
- Instances were booted by image (from Ceph ephemeral disks)

Change-Id: I29c7d3ed118f4a6726f2ea887a165f256bc32fd5
Depends-On: https://review.opendev.org/#/c/664619/
Story: 2004520
Task: 28266
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-06-13 16:18:08 -04:00
Kristine Bujold 92b4517165 Fix controllerfs-modify related bugs
A bug in the get_alarms_degrade was causing the API to return a list of
degrade affecting alarms when there was none. This could be observed by
raising NTP alarms by configuring the NTP servers with invalid
addresses. This bug would cause the controllerfs-modify to fail
increasing the size a filesystem.

Also fix a bug with the return value of the controllerfs-modify API. On
success "None" would be returned on the CLI.

Closes-Bug: 1828097
Change-Id: Id09652a8f88915ea4e5ec90c96f195fe0350a550
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-06-10 11:34:56 -04:00
Kristine Bujold 8dedd8fb59 Map armada logs to /var/log/armada
This commit directs application related logs in Armada service
container to /var/log/armadai/ on the host so they are retained
over Armada service restarts or controller swact.

Story: 2003908
Task: 28267

Change-Id: Ifcb72f7352b18e216a7b8a70ad1ef4ec028c060c
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-06-06 10:19:48 -04:00
Robert Church 12ff7c16f8 Update rbd-provisioner replicas based on installed controllers
Currently the number of rbd-provisioner replicas is driven by the
stx-openstack application's 'openstack-control-plane' labels.

On systems where this label has not been applied to the controllers,
this will result in zero provisioners being installed.

Break the dependency on the stx-openstack app and set the number of
replicas based on the number of installed controllers as the
rbd-provisioner node selector will install in k8s masters (i.e.
controllers).

Also update the provisioner's storage-init pod to align with the same
node selection criteria as the rbd-provisioner pod.

Change-Id: Ida180fd12a4923c8cdd5bccf25a1a1e2af4f8a90
Closes-Bug: #1830290
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-05-29 16:30:30 -04:00
Angie Wang b978111f19 Application dirs and overrides updates
This commit includes the following changes to support updating
versioned application:
  - helm application overrides are tied to a specific application
  - each application files are uploaded to its own versioned directory

    e.g. stx-openstack-1.0-13 app files will be located at,
         /opt/platform/armada/19.01/stx-openstack/1.0-13/...
         /opt/platform/helm/19.01/stx-openstack/1.0-13/...
         /scratch/apps/stx-openstack/1.0-13/...

Story: 2005350
Task: 33439
Change-Id: I75555deda57181d069f24d458dda5bf08e7e17cc
Signed-off-by: Angie Wang <angie.wang@windriver.com>
2019-05-23 12:02:42 -04:00
Jerry Sun 92f1658c29 Add Commands For Docker Registry Image Cleanup
Add commands for
registry-image-list, to list the existing images in the repo
registry-image-tags, to list the tags of a specified image
registry-image-delete, to delete a tag of a specified image
registry-garbage-collect, to delete space on the filesystem from
images that do not have any tags referencing them

It is currently impossible to delete images from the local Docker
registry on the controllers. This can be an issue if images are
continuously pushed. The only solution was to grow the file system.
This commit provides commands for a user to find unwanted images and
to delete them from the controller local Docker registry in order to
free up space on the file system.

There is a possibility of the registry getting stuck in read-only mode
due to garbage collect if puppet dies or there is a swact while
garbage collect is running. In this situation, the user should run
garbage collect again. Getting stuck in read-only mode does not
prevent image pulls or kubernetes deployments from existing images
from starting or completing.

Story: 2002840
Task: 28621

Change-Id: I4bb9301a1165db8b860418c413aa3238169bab03
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
2019-05-16 10:28:51 -04:00
Robert Church a8b23796fe Remove the Ceph related charts from the stx-openstack application
This will remove the rbd-provisioner and ceph-pools-audit charts from
the stx-openstack application and enable it to use the default platform
storage provisioner.

Changes include:
 - Update the rbd-provsioner and ceph-pools-audit helm plugin to provide
   overrides for the namespace defined by
   HELM_NS_STORAGE_PROVISIONER (currently: kube-system).
 - Update the cinder, glance, gnocchi, and nova helm plugins use the
   existing ceph-pool-kube-rbd secret for Ceph client access. This
   allows removing the pvc-ceph-client-key generation from the
   rbd-provisioner chart.
 - Add functions to kube_app.py to create/delete the required Ceph user
   secret for all namespaces of a supported application. This provides
   support for PVCs within the application's namespace(s). In the case
   of stx-openstack, this covers any claims made from the 'openstack'
   namespace.
 - Add functions to kube_app.py to support creating and deleting app
   specific resources that are not handled by the application charts.
   Using this enables copying the 'ceph-etc' configmap from the
   provisioner namespace to the openstack namespace for application use.
 - Add support through the kubernetes API to copy a secret from one
   namespace to another.
 - Add support through the kubernetes API to get, create, delete, and
   copy configmaps.
 - Remove the rbd-provisioner and ceph-pools-audit stevedore plugins
   from the stx-openstack application. Also, re-number the plugins.
 - Update the RBD provisioner to support creating namespaces and Ceph
   user secrets for additional namespaces other than that which the
   provisioner is installed. Also, enable PVCs for default
   namespaces (default and kube-public) against the 'general'
   storageclass.

Change-Id: I387e315545d2c99a1b6baa90d30bdb2a4e08f315
Depends-On: I67dba3f1a3a6e7c8169719ee622ddd533c69be31
Story: 2005424
Task: 30679
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-05-10 17:27:28 -04:00
Robert Church 0fe4655be5 Enable refreshing of helm repository information
This will automate execution of 'helm repo update' to ensure that the
helm repository information is up-to-date.

When the helm repos are updated via helm-upload and are re-indexed, the
local repository information also requires updating so that 'helm
search' commands report accurate information.

This will provide a utility function to perform this operation and the
utility function will be called after an application upload and when the
conductor is restarted.

This handles scenarios where an upload occurs on the active controller
followed by a swact. The newly actvated controller needs to make sure
that the local repository cache reflects any changes.

Change-Id: I67dba3f1a3a6e7c8169719ee622ddd533c69be31
Depends-On: Id2df178083961d46d069f3dc1590cb72a2cecd1b
Story: 2005424
Task: 30648
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-05-10 17:27:25 -04:00
Robert Church 2300e213bf Enable platform managed applications
Add support for uploading and launching applications without user
interaction.

This commit will:
 - provide a conductor periodic audit task that ensures that platform
   managed applications are uploaded and applied.
 - register the HELM_APP_PLATFORM application as a platform managed
   application.
 - define a well-known location for RPM installed applications that are
   only managed by the audit task.

Actions by the audit task are only performed by the unlocked/available
active controller.

Additional work will later enable automatic upgrade of platform managed
applications as a result of the patching

The user can interact with the platform managed applications using
existing CLI commands: 'system helm-override-xxx' and 'system
application-xxx'

Removed @memoized from the get_active_controller() utility check. During
early initial configuration, this was caching active controller
information that had not reached unlocked/enabled/available states. Once
that state was reached, this function was not reporting these states
accurately and causing checks against these states to fail when they
should pass.

Change-Id: Id2df178083961d46d069f3dc1590cb72a2cecd1b
Depends-On: I34ad8789768bfd081ab2dcd45d110d9cd8349875
Story: 2005424
Task: 30647
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-05-10 17:27:13 -04:00
Robert Church e42a0162fb Enable 'platform-integ-apps' application support
This commit enables a new application tarball by:
 - adding existence through the stevedore plugin framework
 - register it as a system app to that system overrides can be generated
 - provide initial overrides for the rbd-provisioner and
   ceph_pools_audit charts
 - updates the rbd-provisioner to support installation of multiple
   provisioners in the same cluster.

Change-Id: I34ad8789768bfd081ab2dcd45d110d9cd8349875
Depends-On: I0caaa878a6c6781d038b48b8caa2aa507ee9568a
Story: 2005424
Task: 30646
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-05-10 17:27:08 -04:00
Robert Church df65d20e07 Enable platform helm repo for RPM installed charts
Add a new helm repository, 'stx-platform', designed to hold charts that
need to be delivered as part of the basic platform. These charts will be
installed via RPMs as part of install and patching.

Update the existing stx-openstack armada application manifests to
reference the new location of the existing 'starlingx' repo. The
'starlingx' repo will be renamed with a future commit to 'stx-apps'.

Enable multiple repository support when generating helm overrides for
the chart location.

This updates both the puppet manifests and ansible playbook for initial
and subsequent configuration scenarios.

Change-Id: I0caaa878a6c6781d038b48b8caa2aa507ee9568a
Depends-On: I4b1a3615a6bd5d0bdd834a1cdf27c05d5a1057a0
Depends-On: I096d5ac126efc97f9a0a0f54f1e02323d936281c
Story: 2005424
Task: 30644
Signed-off-by: Robert Church <robert.church@windriver.com>
2019-05-10 17:27:04 -04:00
Angie Wang bc28061090 Support deploying application with versioned app tarball
This commit updates the upload workflow to support uploading versioned
application tarball with metadata file. The app name and version are
extracted from the metadata file. If the tarball doesn't contain the
app name and version, they need to be specified when uploading.

If the application has patch dependencies, the required patches
need to be fully applied before uploading. After application
upload/delete is done, sysinv reports the patch dependencies to
patch controller.

Tests conducted:
- upload/apply/remove/delete(stx-openstack and custom app)
- upload via url
- failure tests:
  - metadata file has empty name/version
  - no app name/version provided from CLI or metadata file
  - required patch(es) not applies
  - upload without specify tarball
  ...

Story: 2005350
Task: 30609
Depends-On: https://review.opendev.org/655494
Change-Id: I096d5ac126efc97f9a0a0f54f1e02323d936281c
Signed-off-by: Angie Wang <angie.wang@windriver.com>
2019-04-29 18:09:11 -04:00
Al Bailey b899cf351e Upversion Armada SHA to be a newer image
Using SHA: af8a9ffd0873c2fbc915794e235dbd357f2adab1
which was built and tagged on April 9, 2019.

The previous Armada SHA was from Sept 2018.

The manifest.xml is updated to not generate armada warnings
for libvirt, openvswitch, nova and neutron.
The warning was:
  "label_selector" not specified,
  waiting with no labels may cause unintended consequences.

Story: 2005198
Task: 30436
Change-Id: I97b633d9e6e1e4574e25dc8b69500faae4b4a809
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2019-04-11 15:13:41 -05:00
Kristine Bujold a1e2d1e183 Remove wrs-configutilities SDK Module
Remove configutilities and move what is being used in other components
to controllerconfig.

Tested with a clean install on AIO-DX and running config_controller.

With the StarlingX move to supporting pure upstream OpenStack, the
majority of the SDK Modules are related to functionality no longer
supported. The remaining SDK Modules will be moved to StarlingX
documentation.

Story: 2005275
Task: 30262

Change-Id: Ie496548dfc6efee677a501c98c227c586df0a7d6
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-04-02 11:50:23 -04:00
Al Bailey 066dfcd8f7 Remove nova-api requirement from sysinv OCF scripts
SysInv API and Conductor had a check in their OCF
script that required nova-api binary was present
in order for them to launch.

This commit eliminates that check, and therefore allows the
nova-api components to be removable from bare metal controller.

Story: 2004764
Task: 30064
Change-Id: I372f9e7e6fcb30ad81289c9bd6b33a6ad48670a9
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2019-03-18 15:29:35 -05:00
Al Bailey 0d2114a479 Add missing rpm dependencies to sysinv
The following dependencies are required by sysinv
but were being implicitly installed by magnum.
 * python-docker
 * python-kubernetes

When the unused magnum openstack component was removed
from the installation, sysinv conductor would fail to
launch.

This corrects that problem and allows those unused
bare metal opentack components to be safely removed.

Fixed up test-requirements and requirements since
docker is needed for runtime and not just testing.

Change-Id: Ibe06b4b401446ff09a83e2051f8495ee53244b1a
Story: 2004764
Task: 30058
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2019-03-18 18:52:16 +00:00
David Sullivan 5348bca741 Add replica and anti-affinity to rbd-provisioner
The rbd-provisioner does not follow the same template as the other
openstack-helm charts so we need to use specific methods to set the
affinity and replica parameters.

Story: 2004520
Task: 29845
Change-Id: I05b97b8254a541753031ba051707092eae301d6a
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
2019-03-07 14:59:05 -05:00
Angie Wang cb4b30bf56 Solve the stx-openstack reapply issue on controller-1
After stx-openstack applied, the stx-openstack reapply shouldn't
trigger the charts reinstallation if there has no overrides changed
for charts. However, the reinstallation happens after swacting active
controller to controller-1 due to the generated images overrides on
controller-1 are different from before. The images overrides generation
requires walking through the stx-openstack charts stored under
/scratch, but charts do not exist on controller-1's /scratch as it's
an unreplicated filesystem. This causes the images overrides to differ
between controller-1 and controller-0.

This commit updates to walk through charts and get the images for
charts during application-upload, then save the images list for each
chart into the existing images file under aramda directory
/opt/platform/armada. The images file would be used for retrieving
the images for charts to generate images overrides.

Closes-Bug: 1816173
Change-Id: I4f00c3031decb063f8f126d0c837acd4dde56fc3
Signed-off-by: Angie Wang <angie.wang@windriver.com>
2019-02-22 15:07:42 -05:00
Kristine Bujold c611917a0d Move horizon static configs to Armada manifest
Move all horizon static configurations from the overrides to the
Armada manifest.

This is being done so we have a consistent way of managing
containerized openstack configurations. Static configurations will
be located in the Armada manifest and dynamic configuration will be
located in the overrides files.

Story: 2003909
Task: 29635

Change-Id: I4abbc0eb158304774134e2d60f2b666c0d90bbd8
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-02-21 14:08:02 -05:00
David Sullivan 1c467789c4 Add replica settings for mariadb ingress pod
There was no mariadb replica override for the ingress pod. On AIO-SX
this caused two pods to be scheduled. When anti-affinity was added to
mariadb this broke application-apply on AIO-SX.

The mariadb ingress pod replication will be set to the number of
controllers.

Change-Id: Icf3f1979720629904ca9ddcabf59e8ecfab709e5
Story: 2004520
Task: 29570
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
2019-02-20 00:56:09 -05:00
David Sullivan 0dd4b86526 Add replica and anti-affinity settings
Add anti-affinity settings to openstack pods. Add replication to
novncproxy, aodh, panko and rbd_provisioner services.

Change-Id: I8091a54cab98ff295eba6e7dd6fa76827d149b5f
Story: 2004520
Task: 29418
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
2019-02-16 17:48:47 -05:00
Angie Wang 5b94294002 Support stx-openstack app install with the authed local registry
The functionality of local docker registry authentication will be
enabled in commit https://review.openstack.org/#/c/626355/.
However, local docker registry is currently used to pull/push images
during application apply without authentication and no credentials
passed to the kubernetes when pulling images on other nodes except
for active controller.

In order to install stx-openstack app with local docker registry that
has authentication turned on, this commit updates the following:
 1. Pass the user credentials when pulling/pushing images from local
    registry during application apply.
 2. Create a well-known registry secret "default-registry-key" which
    holds the authorization token during stx-openstack app apply and
    delete the secret during removal. The helm-toolkit is updated to
    refer to this secret in k8s openstack service account template for
    pulling images from local by kubelet. This secret is also added to
    rbd-provisioner service account as well since it is not using
    helm-toolkit to create service account.

Note: #2 is short-term solution. The long-term solution is to implement
the BP https://blueprints.launchpad.net/openstack-helm/+spec/support
-docker-registry-with-authentication-turned-on.

Story: 2002840
Task: 28945
Depends-On: https://review.openstack.org/636181
Change-Id: I015dccd12c5c7fa7a4bea74eef8d172f03b5d60e
Signed-off-by: Angie Wang <angie.wang@windriver.com>
2019-02-15 15:25:33 -05:00
Kristine Bujold d5db10f6b7 Move neutron static configs to Armada manifest
Move all neutron static configurations from the overrides to the
Armada manifest.

This is being done so we have a consistent way of managing
containerized openstack configurations. Static configurations will
be located in the Armada manifest and dynamic configuration will be
located in the overrides files.

Story: 2003909
Task: 29433

Change-Id: I5baf0bbc15912e0303955456151e69856bba0385
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-02-15 11:42:47 -05:00