Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
python-gunicorn-19.7.1-1 is available in CentOS 7.6 repo now.
And there is only 1 meta patch to disable python3 compilation. So it
is ok to replace the srpm with rpm package directly.
Story: 2005484
Task: 30578
Change-Id: I1a56c098741c6ddda466caa740a530d6e8468403
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
These packages could be found in centos repo also, so move them to
centos lst.
Story: 2005484
Task: 30577
Change-Id: I23abc56b962c2a5d0be2b170a59e67632c12083f
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Issue:
'rate_limit' error when executing openstack network commands.
Failure was tracked to a failure to satisfy the lower-constraints.txt
of openstacksdk, keystoneauth1==3.13.0. We currently only provide
python2-keystoneauth1-3.10.0-1.el7.noarch.rpm.
Solution:
Investigation showed that there is currently no centos rpm
providing python2-keystoneauth1-3.13.0 or better. Solution
is to build python2-keystoneauth1-3.13.1 from openstack's git.
We can revert to a downloaded rpm when one becomes available.
Closes-Bug: 1819020
Change-Id: I30332820a8f4ae877b0719148022bade1943eadb
Depends-On: Ie2300f352dffc2fc87fbc84a9fc98d2a4e9484d4
Signed-off-by: Scott Little <scott.little@windriver.com>
Pulling in the needed RPMs in preparation for Ansible Playbook
inclusion in the image.
Tests conducted:
- Test of download mirror scripts to pull the required packages
- Successful installation
Story: 2004695
Task: 29376
Change-Id: Iaecd1dac75b34183cbaf7390d58ba057fdfabe7c
Signed-off by: Tee Ngo <tee.ngo@windriver.com>
Stein/Master clients required new RPMs or updated ones
The old versions have been kept (for now) and will be removed
by a later commit
Story: 2004751
Task: 28867
Change-Id: Iea58cb9484c75cf757397d53d7d1576a2f436d81
Depends-On: I4e4b4768b4c0e64d03960bd42e9a1d342884e9a0
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
These packages could be found in centos paas openshift-origin311
repo.
go-srpm-macros package is changed to the latest version in repo.
Test:
These packages are for kubernetes build only, not included in ISO
image.
Pass build and deploy test.
Story: 2004593
Task: 28471
Change-Id: I8904938b245d2fdf683bc2a58d6dffdec62c8063
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
1.8.2 contains the fix for the wait commands
Later versions are incompatable malformed iptables entries which
will cause ceph setup to fail.
Story: 2003910
Task: 28192
Change-Id: Id4c20a47c55615088b7ffbd66dd98c457544b7ae
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
Puppet was occasionally failing due to iptables
being busy while attempting to update.
This was fixed in puppet-firewall 1.8.2
Upversioning this component to the latest version that
is still compatable with pike
Tested in std config, containers config and aio containers
Story: 2003910
Task: 28192
Change-Id: If127d10255d837904e41b81867b988585370f8b7
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
This version changes the exec_run method signature
to return a tuple with exit-code and message.
Story: 2003908
Task: 28013
Change-Id: I7ea41d8c2010ade9cfad7fc120df853771d46b49
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
The 6.0 src rpm and binary rpms come from koji.
There are no changes to runtime requirements.
Story: 2002843
Task: 26751
Change-Id: Id9ee51d548adc1d4c9b4c4b83e8803accf7bfc2d
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
golang 1.10.2 is required to build kubernetes 1.12.1
The higher version rpms do not exist on centos mirror
but do exist on koji.
1.10.2 adds additional requirements:
mercurial
go-srpms-macros
subversion
Transitive dependencies for those are:
subversion-libs
neon
pakchois
Story: 2002843
Task: 26751
Change-Id: Ie43ff00da3518b04db8f32feafebd936ee9b6f8a
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
This update adds the following RPMs to rpms_3rdparties.lst:
python2-pip
python2-wheel
python3-pip
python3-wheel
These packages are required to allow building wheels for python modules
from the RPM spec file.
While an older python2-pip RPM is currently available from the configured
Starlingx-epel-7 repo, the other 3 are not, so all 4 are being explicitly
downloaded from Koji to have consistent versions.
Change-Id: I987666acbb4594facfa38bd9e8ea978d8ec0a329
Story: 2002876
Task: 26482
Signed-off-by: Don Penney <don.penney@windriver.com>
Story: 2003340
Task: 24378
No patch is needed for this package, so replace it with rpm package
Change-Id: I21eecb9fb898cf0c7484eb03376fc4703c81204e
Signed-off-by: slin14 <shuicheng.lin@intel.com>
-By standardizing the list files name it's possible to eliminate
one parameter in the download function in dl_rpms and reduce complexity.
-Now the download function does not receive it as a parameter anymore,
and instead it gets it from the RPM name.
-Also the download function now decides if build a wget command or
a yumdownloader commnad based on the content of the list (by identifying
the # character) instead of using the name of the list.
I adapted download_mirror.sh.
Change-Id: I041fc9c704156215f06149e5b4c16cd92990e17c
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
Scott Little