StarlingX stopped supporting CentOS builds in the after release 7.0.
This update will strip CentOS from our code base. It will also remove
references to the failed OpenSUSE feature as well.
Story: 2011110
Task: 49965
Change-Id: I17943f1b22a57b0ed02f638cb6320855446c0be3
Signed-off-by: Scott Little <scott.little@windriver.com>
$MY_REPO/stx/downloads contains links to tarballs and other files
relevant to the build. It is populated by populate_downloads.sh.
On subsequent runs, new links might be added, but old links are
not cleaned up.
Adopt the convention used by generate-centos-repo.sh of moving
the old content under a timestamp backup directory, leaving
an empty directory to re-populate.
Closes-bug: 1908297
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Ib6c8b782db6c369cb00e8fe8c64be099d7a5d238
Commit https://review.opendev.org/c/starlingx/root/+/762700
was intended to include the code to select the release specific
mock config prototype.
e.g. mock.cfg.centos7.proto
As delivered, it would always select the default.
e.g. mock.cfg.proto
Other improvements:
- remove some trailing whitespace
- improved cleanup of temorary directories
Closes-Bug: 1906547
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I3e988fb0e861efcf9cd16b8a46b74398dbb1db17
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
Mellanox driver has been upgraded to 4.5.1 version in tarball_dl.lst
Need update the name in populate_downloads.sh accordingly.
Story: 2004521
Task: 29193
Change-Id: If4cb097f9ee2027f2e7eb6436057cdd607890f81
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
An update to stx-integ (https://review.openstack.org/#/c/591936/)
upreved the version of libibverbs that the build was expecting.
This change uprevs the version which is placed in downloads/ by
populate_downloads.sh to match the expected version.
Change-Id: I5ab727f373fe4c73b73135e6860beb99792537d6
Signed-off-by: jmckenna <jason.mckenna@windriver.com>
Story: 2003508
Task: 24786
This change improves support for organizational or personal local
mirrors. Download tools work in a wider variety of environmnets,
and the population of build environments is now driven by .lst
files.
The download_mirror.sh (and related files) now have options to
better support running in non-containerized environments. This
includes options to run using local yum.conf files (rather than
system files) and not executing sudo or chgrp commands. These
options are enabled by flags. Default behaviour remains
unchanged.
The generate-cgcs-centos-repo.sh tool now populates build
environment as driven by lst files. This allows a mirror to be
maintained (and grown) at an organizational level, while allowing
a developer to produce exactly the environment they need as per
the lst files they have checked out.
generate-cgcs-centos-repo.sh now also populates the downloads
directory by invoking the populate_downloads.sh script (also
driven by lst files)
Change-Id: I8aae32c776df7618a5ef545a0ce43903b41112d0
Story: 2003422
Task: 24586
Signed-off-by: jmckenna <jason.mckenna@windriver.com>
Scott Little