After compile-layer build, there should be a new container for distro-layer.
So do flock-layer.
These three layer's environemnt are different. However, the container is hard to change environment after launched.
So, Three layers, three environments and three containers.
The process is
cp localrc-compiler localrc
./tb.sh env
./tb.sh run
./tb.sh exec
cp localrc-distro localrc
./tb.sh env
./tb.sh run
./tb.sh exec
docker ps | awk 'NR>1{print $12}'
stx-distro-centos-builder
stx-compiler-centos-builder
Signed-off-by: ChantYuCN <chengde.yu@intel.com>
Change-Id: Ibded97dbc9f7541b0a7f0f8c71fb90816510a8b6
tb.sh create might fail to create the builder docker image.
Yum install of the mock package failed, but yum did not report
the failure because other packages in the instalation set succeeded.
A subsequent command in the dockerfile fails when it tries to
remove/relocate /var/lib/mock, but failes because it is not present.
The yum error reporting was corrected in a recent update.
But this does not address cached copies of old and broken
yum install steps that pre-date the fix.
The mock package is paricularly sensitive as it has cengn as
the only source, where as other packages have multiple sources.
One option is to force docker to not use the cache at all, which
is slow.
The second option is to change the docker file, placing the
yum command to install mock under a seperate docker RUN command.
The altered build instructions ensure that the docker cache
with the broken install can't be used. While we are at it,
move the user/project customization steps as far down as possible
to improve cache usage.
This change implements both.
Closes-Bug: 1917901
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I28041bb44af53384c00a750b7162c6c6808c4e2d
Systemd is once preventing a clean exit of the build
container.
Mapping /run and /tmp to tmpfs helps on some systems.
Partial-bug: 1907119
Change-Id: I3fc54792d18b632fbd5cab678ce4fa348bc96873
Signed-off-by: Scott Little <scott.little@windriver.com>
Opendev recently upgraded to a newer version of gerrit.
That gerrit demands that we run a newer version of git than
is available from Centos 7. This will prevent users from
posting code for review from within this build container.
This update compiles a newer version (2.29.2) of git from
downloaded source code, and adds it to the build image.
A newer version of git-review is downloaded from pypi,
and added to the build image.
A mechanism is provided to feed in a valid email address
to the idocker image's ~/.gitconfig by defining
MY_EMAIL=you@your-email-host in your 'localrc' prior to 'tb.sh create'
Closes-bug: 1905789
Change-Id: I28ed185ea477ba3a8e47e1dd11fd384816f72717
Signed-off-by: slittle1 <scott.little@windriver.com>
Since the base centos images now is 7.8.2003, updated the
TC_CONTAINER_TAG variable in the tb.sh script to reflect the same.
Change-Id: I5455867660d787dc7eba3f07db11ce6c56fb0bca
Signed-off-by: Poornima <poornima.y.n@intel.com>
1) Add config for the container layer.
2) Add support for the new config file 'required_layer_wheel_inc.cfg',
Which will specify the default urls for lower layer wheels.inc files.
These files must be downloaded by download_mirror.sh, and linked by
generate-cgcs-centos-repo.sh.
3) Add options to download_mirror.sh and generate-cgcs-centos-repo.sh
allowing overrides or additions to the set of url's for lower layer
wheels.inc
4) Changes to allow build scripts running within a docker container
to launch a sister (not daughter) container. This will allow
build-wheels to run within a container.
Story: 2006166
Task: 38980
Depends-On: https://review.opendev.org/711773
Depends-On: https://review.opendev.org/717755
Change-Id: I95c71a0327dc9d919ed835d3661a6f1ec7ce30c7
Signed-off-by: Scott Little <scott.little@windriver.com>
This changes a brand specific to be generic for configuration
Story: 2006387
Task: 36914
Change-Id: I826498cfc58b4e02fcdf4590a5cbfc25d38f7fc9
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
A makefile was used only for creation and clean up of the docker
image for building. In this process, all the environment variables
were sourced and included into the Makefile. However this can fail
on systems that has functions and cause a failing makefile.
Trying to fixing this issue I realize that this Makefile ain't needed,
as all the logic for docker management is under `tb.sh`, therefore makes
sense to remove this Makefile and use this script for the entire process.
Story: 2002560
Change-Id: I2795593521a5d16dc2033861946fd09c2270424e
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
Update the tb.sh after unify the Dockerfile
Also the Docker image and tag change to 7.4
Depends-On: https://review.openstack.org/#/c/601306/
Change-Id: I6b9e32bc0ffbd9870f83c37437afd2e67195fcc3
Signed-off-by: Felipe de Jesus Ruiz Garcia <felipe.de.jesus.ruiz.garcia@intel.com>
With this change, the docker image will be created using the
MYUNAME variable set in the localrc file.
Closes-Bug: 1796360
Change-Id: Ia82a3ac463d77b6f59b5ba7ab59ae7bbf3eba49c
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
This in order to get the shell back after executing run and don't need
another terminal session.
Story: 2002753
Change-Id: I2f65fe9d6cc4b2284bdf766b0a2ff3a6643f23d1
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
This includes Docker containers to perform the StarlingX build
and a set of scripts to maintain a local mirror of binary CentOS
and other packages required to populate the final ISO file.
Change-Id: I8140fd8fa2d00e7aa98c895a8e4962ab3748669d
ChantYuCN