Build layering, script and lst update

Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.

Current .lst and yum files under directory centos-mirror-tools
are relocated.  Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .

Lst entries for compilable content have been relocated to other
git repos by prior updates.  i.e.  those that list tarballs or
srpms to be compiled within that repo.

The original .lst files are deleted to make it easier
to identify new content during development.

Layer 'all' builds all layers in a single workspace.  The
lst files are identical to current content, minus the src.rpm and
tarball entries.

Other layers get only a subset of packages download. The minimum
required to build the layer.  The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.

An upper layer does not need to list rpms known to be provided by
a lower layer.  Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.

A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds.  These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.

Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso.  The transitive list of required rpms need not be listed.

Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.

To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.

The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git.  Substitute urls can use the file:///
syntax.  Just be sure to remove these changes before submitting.

Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.

The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.

e.g. For simplicity I'll only list the 'extra' arguements

download_mirror.sh -C <my-config-dir> \
                   -l <layer> \
                   ...

generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
                             --layer=<layer> \
                             ...

populate_downloads.sh --config-dir=<my-config-dir> \
                      --layer=<layer> \
                      ...

These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.

e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>

The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file.  Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.

e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso.  For simplicity I'll only list the 'extra' arguments

./download_mirror.sh -l flock \
    -L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    -I distro,std,file:///<my-distro-workspace>/rt/image.inc \
    ...

generate-cgcs-centos-repo.sh --layer=flock \
    --layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    --layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
    ...

NOTE: The triplet syntax for a package list url is
    <lower-layer>,<build-type>,<url-to-rpm.lst>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'rt', 'installer'

    Also if 'file:///' syntax is used, a matching change is made to
    the yum *.repo file.  This assumes that the rpm.lst is co-resident with
    repodata directory, as is the norm for our build outputs.

NOTE: The triplet syntax for a image inc url is
    <lower-layer>,<include-type>,<url-to-image.inc>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'dev'

A typical user is likely only working in the flock layer on the master
branch.  He should be content to use the compiler and distro layer
outputs from cengn.

His workflow looks like ...

1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.

1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Only a cross-layer developer should setup two or three copies of the
building environment, one per layer.  We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,

Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...

$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
 ... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...

$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh

Story: 2006166
Task: 37103

Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>

Dockerfile

@@ -108,6 +108,7 @@ COPY toCOPY/finishSetup.sh /usr/local/bin
 COPY toCOPY/populate_downloads.sh /usr/local/bin
 COPY toCOPY/generate-cgcs-tis-repo /usr/local/bin
 COPY toCOPY/generate-cgcs-centos-repo.sh /usr/local/bin
+COPY toCOPY/lst_utils.sh /usr/local/bin
 COPY toCOPY/.inputrc /home/$MYUNAME/
 COPY toCOPY/builder-constraints.txt /home/$MYUNAME/
 
@@ -158,7 +159,7 @@ RUN echo "# Load stx-builder configuration" >> /etc/profile.d/TC.sh && \
     echo "if [[ -r \${HOME}/buildrc ]]; then" >> /etc/profile.d/TC.sh && \
     echo "    source \${HOME}/buildrc" >> /etc/profile.d/TC.sh && \
     echo "    export PROJECT SRC_BUILD_ENVIRONMENT MYPROJECTNAME MYUNAME" >> /etc/profile.d/TC.sh && \
-    echo "    export MY_BUILD_CFG MY_BUILD_CFG_RT MY_BUILD_CFG_STD MY_BUILD_DIR MY_BUILD_ENVIRONMENT MY_BUILD_ENVIRONMENT_FILE MY_BUILD_ENVIRONMENT_FILE_RT MY_BUILD_ENVIRONMENT_FILE_STD MY_DEBUG_BUILD_CFG_RT MY_DEBUG_BUILD_CFG_STD MY_LOCAL_DISK MY_MOCK_ROOT MY_REPO MY_REPO_ROOT_DIR MY_SRC_RPM_BUILD_DIR MY_TC_RELEASE MY_WORKSPACE" >> /etc/profile.d/TC.sh && \
+    echo "    export MY_BUILD_CFG MY_BUILD_CFG_RT MY_BUILD_CFG_STD MY_BUILD_DIR MY_BUILD_ENVIRONMENT MY_BUILD_ENVIRONMENT_FILE MY_BUILD_ENVIRONMENT_FILE_RT MY_BUILD_ENVIRONMENT_FILE_STD MY_DEBUG_BUILD_CFG_RT MY_DEBUG_BUILD_CFG_STD MY_LOCAL_DISK MY_MOCK_ROOT MY_REPO MY_REPO_ROOT_DIR MY_SRC_RPM_BUILD_DIR MY_TC_RELEASE MY_WORKSPACE LAYER" >> /etc/profile.d/TC.sh && \
     echo "fi" >> /etc/profile.d/TC.sh && \
     echo "export FORMAL_BUILD=0" >> /etc/profile.d/TC.sh && \
     echo "export PATH=\$MY_REPO/build-tools:\$PATH" >> /etc/profile.d/TC.sh

buildrc

@@ -82,3 +82,5 @@ export MY_BUILD_CFG_RT=${MY_WORKSPACE}/rt/${MY_BUILD_ENVIRONMENT_FILE_RT}
 export MY_MOCK_ROOT=$MY_WORKSPACE/mock/root
 export MY_DEBUG_BUILD_CFG_STD=${MY_WORKSPACE}/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-std.b0.cfg
 export MY_DEBUG_BUILD_CFG_RT=${MY_WORKSPACE}/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-rt.b0.cfg
+export LAYER=$LAYER
+export STX_CONFIG_DIR="$STX_CONFIG_DIR"

centos-mirror-tools/config/centos/compiler/rpms_centos.lst

@@ -0,0 +1,287 @@
+apr-1.4.8-3.el7_4.1.x86_64.rpm
+apr-util-1.5.2-6.el7.x86_64.rpm
+atk-devel-2.28.1-1.el7.x86_64.rpm
+at-spi2-atk-devel-2.26.2-1.el7.x86_64.rpm
+autoconf-2.69-11.el7.noarch.rpm
+automake-1.13.4-3.el7.noarch.rpm
+# basesystem-10.0-7.el7.centos.noarch.rpm provided by mock
+bash-4.2.46-31.el7.x86_64.rpm
+# binutils-2.27-34.base.el7.x86_64.rpm provided by mock
+binutils-devel-2.27-34.base.el7.x86_64.rpm
+bison-3.0.4-2.el7.x86_64.rpm
+bluez-libs-5.44-4.el7_4.x86_64.rpm
+bluez-libs-devel-5.44-4.el7_4.x86_64.rpm
+# bzip2-1.0.6-13.el7.x86_64.rpm provided by mock
+bzip2-devel-1.0.6-13.el7.x86_64.rpm
+# bzip2-libs-1.0.6-13.el7.x86_64.rpm provided by mock
+# ca-certificates-2018.2.22-70.0.el7_5.noarch.rpm provided by mock
+# centos-release-7-6.1810.2.el7.centos.x86_64.rpm provided by mock
+# chkconfig-1.7.4-1.el7.x86_64.rpm provided by mock
+# coreutils-8.22-23.el7.x86_64.rpm provided by mock
+# cpio-2.11-27.el7.x86_64.rpm provided by mock
+# cpp-4.8.5-36.el7.x86_64.rpm provided by mock
+# cracklib-2.9.0-11.el7.x86_64.rpm provided by mock
+# cracklib-dicts-2.9.0-11.el7.x86_64.rpm provided by mock
+# cryptsetup-libs-2.0.3-3.el7.x86_64.rpm provided by mock
+# curl-7.29.0-51.el7_6.3.x86_64.rpm provided by mock
+# cyrus-sasl-lib-2.1.26-23.el7.x86_64.rpm provided by mock
+# dbus-1.10.24-12.el7.x86_64.rpm provided by mock
+dbus-devel-1.10.24-12.el7.x86_64.rpm
+# dbus-libs-1.10.24-12.el7.x86_64.rpm provided by mock
+dejavu-fonts-common-2.33-6.el7.noarch.rpm
+dejavu-sans-fonts-2.33-6.el7.noarch.rpm
+# device-mapper-1.02.146-4.el7.x86_64.rpm provided by mock
+device-mapper-devel-1.02.146-4.el7.x86_64.rpm
+device-mapper-event-1.02.146-4.el7.x86_64.rpm
+device-mapper-event-libs-1.02.146-4.el7.x86_64.rpm
+# device-mapper-libs-1.02.146-4.el7.x86_64.rpm provided by mock
+device-mapper-multipath-0.4.9-119.el7.x86_64.rpm
+device-mapper-multipath-libs-0.4.9-119.el7.x86_64.rpm
+device-mapper-persistent-data-0.7.3-3.el7.x86_64.rpm
+# diffutils-3.3-4.el7.x86_64.rpm provided by mock
+# dracut-033-554.el7.x86_64.rpm provided by mock
+# expat-2.1.0-10.el7_3.x86_64.rpm provided by mock
+expat-devel-2.1.0-10.el7_3.x86_64.rpm
+file-devel-5.11-35.el7.x86_64.rpm
+# filesystem-3.2-25.el7.x86_64.rpm provided by mock
+# findutils-4.5.11-6.el7.x86_64.rpm provided by mock
+fipscheck-1.4.1-6.el7.x86_64.rpm
+fipscheck-lib-1.4.1-6.el7.x86_64.rpm
+fontconfig-2.13.0-4.3.el7.x86_64.rpm
+fontconfig-devel-2.13.0-4.3.el7.x86_64.rpm
+fontpackages-filesystem-1.44-8.el7.noarch.rpm
+freetype-2.8-12.el7.x86_64.rpm
+freetype-devel-2.8-12.el7.x86_64.rpm
+# gawk-4.0.2-4.el7_3.1.x86_64.rpm provided by mock
+# gcc-4.8.5-36.el7.x86_64.rpm provided by mock
+# gcc-c++-4.8.5-36.el7.x86_64.rpm provided by mock
+# gdbm-1.10-8.el7.x86_64.rpm provided by mock
+gdbm-devel-1.10-8.el7.x86_64.rpm
+gettext-0.19.8.1-2.el7.x86_64.rpm
+gettext-common-devel-0.19.8.1-2.el7.noarch.rpm
+gettext-devel-0.19.8.1-2.el7.x86_64.rpm
+gettext-libs-0.19.8.1-2.el7.x86_64.rpm
+git-1.8.3.1-20.el7.x86_64.rpm
+# glibc-2.17-260.el7.x86_64.rpm provided by mock
+# glibc-common-2.17-260.el7.x86_64.rpm provided by mock
+# glibc-devel-2.17-260.el7.x86_64.rpm provided by mock
+# glibc-headers-2.17-260.el7.x86_64.rpm provided by mock
+glibc-static-2.17-260.el7.x86_64.rpm
+gl-manpages-1.1-7.20130122.el7.noarch.rpm
+# gmp-6.0.0-15.el7.x86_64.rpm provided by mock
+gmp-devel-6.0.0-15.el7.x86_64.rpm
+gnutls-3.3.29-8.el7.x86_64.rpm
+golang-1.11.2-3.el7.x86_64.rpm
+golang-bin-1.11.2-3.el7.x86_64.rpm
+golang-github-cpuguy83-go-md2man-1.0.4-4.el7.x86_64.rpm
+golang-src-1.11.2-3.el7.noarch.rpm
+# grep-2.20-3.el7.x86_64.rpm provided by mock
+# groff-base-1.22.2-8.el7.x86_64.rpm provided by mock
+# gzip-1.5-10.el7.x86_64.rpm provided by mock
+# hardlink-1.0-19.el7.x86_64.rpm provided by mock
+hwdata-0.252-9.1.el7.x86_64.rpm
+# info-5.1-5.el7.x86_64.rpm provided by mock
+# json-c-0.11-4.el7_0.x86_64.rpm provided by mock
+json-c-devel-0.11-4.el7_0.x86_64.rpm
+kernel-headers-3.10.0-1062.1.2.el7.x86_64.rpm
+# keyutils-libs-1.5.8-3.el7.x86_64.rpm provided by mock
+keyutils-libs-devel-1.5.8-3.el7.x86_64.rpm
+# kmod-20-23.el7.x86_64.rpm provided by mock
+# kmod-libs-20-23.el7.x86_64.rpm provided by mock
+# kpartx-0.4.9-119.el7.x86_64.rpm provided by mock
+krb5-devel-1.15.1-34.el7.x86_64.rpm
+# krb5-libs-1.15.1-34.el7.x86_64.rpm provided by mock
+less-458-9.el7.x86_64.rpm
+# libacl-2.2.51-14.el7.x86_64.rpm provided by mock
+libacl-devel-2.2.51-14.el7.x86_64.rpm
+libarchive-devel-3.1.2-10.el7_2.x86_64.rpm
+# libattr-2.4.46-13.el7.x86_64.rpm provided by mock
+libattr-devel-2.4.46-13.el7.x86_64.rpm
+# libcap-2.22-9.el7.x86_64.rpm provided by mock
+libcap-devel-2.22-9.el7.x86_64.rpm
+# libcap-ng-0.7.5-4.el7.x86_64.rpm provided by mock
+libcap-ng-devel-0.7.5-4.el7.x86_64.rpm
+libcroco-0.6.12-4.el7.x86_64.rpm
+# libcurl-7.29.0-51.el7_6.3.x86_64.rpm provided by mock
+# libcurl-devel-7.29.0-51.el7_6.3.x86_64.rpm provided by mock
+# libdb-5.3.21-24.el7.x86_64.rpm provided by mock
+libdb-devel-5.3.21-24.el7.x86_64.rpm
+# libdb-utils-5.3.21-24.el7.x86_64.rpm provided by mock
+libedit-3.0-12.20121213cvs.el7.x86_64.rpm
+# libffi-3.0.13-18.el7.x86_64.rpm provided by mock
+libffi-devel-3.0.13-18.el7.x86_64.rpm
+# libgcc-4.8.5-36.el7.x86_64.rpm provided by mock
+# libgcrypt-1.5.3-14.el7.x86_64.rpm provided by mock
+libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
+libglvnd-core-devel-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
+libglvnd-devel-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
+libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
+libglvnd-gles-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
+libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
+libglvnd-opengl-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
+# libgomp-4.8.5-36.el7.x86_64.rpm provided by mock
+# libgpg-error-1.12-3.el7.x86_64.rpm provided by mock
+libgpg-error-devel-1.12-3.el7.x86_64.rpm
+# libidn-1.28-4.el7.x86_64.rpm provided by mock
+libkadm5-1.15.1-34.el7.x86_64.rpm
+libmodman-2.0.1-8.el7.x86_64.rpm
+# libmount-2.23.2-59.el7.x86_64.rpm provided by mock
+# libmpc-1.0.1-3.el7.x86_64.rpm provided by mock
+libpciaccess-0.14-1.el7.x86_64.rpm
+libpng-1.5.13-7.el7_2.x86_64.rpm
+libpng-devel-1.5.13-7.el7_2.x86_64.rpm
+libproxy-0.4.11-11.el7.x86_64.rpm
+# libpwquality-1.2.3-5.el7.x86_64.rpm provided by mock
+# libselinux-2.5-14.1.el7.x86_64.rpm provided by mock
+libselinux-devel-2.5-14.1.el7.x86_64.rpm
+# libsepol-2.5-10.el7.x86_64.rpm provided by mock
+libsepol-devel-2.5-10.el7.x86_64.rpm
+# libsmartcols-2.23.2-59.el7.x86_64.rpm provided by mock
+# libssh2-1.4.3-12.el7.x86_64.rpm provided by mock
+# libstdc++-4.8.5-36.el7.x86_64.rpm provided by mock
+# libstdc++-devel-4.8.5-36.el7.x86_64.rpm provided by mock
+# libtasn1-4.10-1.el7.x86_64.rpm provided by mock
+libtool-2.4.2-22.el7_3.x86_64.rpm
+libunistring-0.9.3-9.el7.x86_64.rpm
+# libuser-0.60-9.el7.x86_64.rpm provided by mock
+# libutempter-1.1.6-4.el7.x86_64.rpm provided by mock
+# libuuid-2.23.2-59.el7.x86_64.rpm provided by mock
+libuuid-devel-2.23.2-59.el7.x86_64.rpm
+# libverto-0.2.5-4.el7.x86_64.rpm provided by mock
+libverto-devel-0.2.5-4.el7.x86_64.rpm
+libwayland-client-1.15.0-1.el7.x86_64.rpm
+libwayland-server-1.15.0-1.el7.x86_64.rpm
+libX11-1.6.7-2.el7.x86_64.rpm
+libX11-common-1.6.7-2.el7.noarch.rpm
+libX11-devel-1.6.7-2.el7.x86_64.rpm
+libXau-1.0.8-2.1.el7.x86_64.rpm
+libXau-devel-1.0.8-2.1.el7.x86_64.rpm
+libxcb-1.13-1.el7.x86_64.rpm
+libxcb-devel-1.13-1.el7.x86_64.rpm
+libXdamage-1.1.4-4.1.el7.x86_64.rpm
+libXdamage-devel-1.1.4-4.1.el7.x86_64.rpm
+libXext-1.3.3-3.el7.x86_64.rpm
+libXext-devel-1.3.3-3.el7.x86_64.rpm
+libXfixes-5.0.3-1.el7.x86_64.rpm
+libXfixes-devel-5.0.3-1.el7.x86_64.rpm
+libXft-2.3.2-2.el7.x86_64.rpm
+libXft-devel-2.3.2-2.el7.x86_64.rpm
+# libxml2-2.9.1-6.el7_2.3.x86_64.rpm provided by mock
+libXrender-0.9.10-1.el7.x86_64.rpm
+libXrender-devel-0.9.10-1.el7.x86_64.rpm
+libxshmfence-1.2-1.el7.x86_64.rpm
+libXxf86vm-1.1.4-1.el7.x86_64.rpm
+libXxf86vm-devel-1.1.4-1.el7.x86_64.rpm
+# lua-5.1.4-15.el7.x86_64.rpm provided by mock
+lua-devel-5.1.4-15.el7.x86_64.rpm
+# lz4-1.7.5-2.el7.x86_64.rpm provided by mock
+m4-1.4.16-10.el7.x86_64.rpm
+mercurial-2.6.2-8.el7_4.x86_64.rpm
+mesa-libGLU-9.0.0-4.el7.x86_64.rpm
+mesa-libGLU-devel-9.0.0-4.el7.x86_64.rpm
+# mpfr-3.1.1-4.el7.x86_64.rpm provided by mock
+# ncurses-5.9-14.20130511.el7_4.x86_64.rpm provided by mock
+# ncurses-base-5.9-14.20130511.el7_4.noarch.rpm provided by mock
+ncurses-devel-5.9-14.20130511.el7_4.x86_64.rpm
+# ncurses-libs-5.9-14.20130511.el7_4.x86_64.rpm provided by mock
+neon-0.30.0-3.el7.x86_64.rpm
+nettle-2.7.1-8.el7.x86_64.rpm
+# nspr-4.19.0-1.el7_5.x86_64.rpm provided by mock
+nspr-devel-4.19.0-1.el7_5.x86_64.rpm
+# nss-3.36.0-7.el7_5.x86_64.rpm provided by mock
+nss-devel-3.36.0-7.el7_5.x86_64.rpm
+# nss-pem-1.0.3-5.el7.x86_64.rpm provided by mock
+# nss-sysinit-3.36.0-7.el7_5.x86_64.rpm provided by mock
+# nss-tools-3.36.0-7.el7_5.x86_64.rpm provided by mock
+# nss-util-3.36.0-1.el7_5.x86_64.rpm provided by mock
+nss-util-devel-3.36.0-1.el7_5.x86_64.rpm
+openssh-7.4p1-16.el7.x86_64.rpm
+openssh-clients-7.4p1-16.el7.x86_64.rpm
+openssl-devel-1.0.2k-16.el7.x86_64.rpm
+# openssl-libs-1.0.2k-16.el7.x86_64.rpm provided by mock
+# p11-kit-0.23.5-3.el7.x86_64.rpm provided by mock
+p11-kit-devel-0.23.5-3.el7.x86_64.rpm
+# p11-kit-trust-0.23.5-3.el7.x86_64.rpm provided by mock
+pakchois-0.4-10.el7.x86_64.rpm
+# pam-1.1.8-22.el7.x86_64.rpm provided by mock
+# pcre-8.32-17.el7.x86_64.rpm provided by mock
+pcre-devel-8.32-17.el7.x86_64.rpm
+# perl-5.16.3-294.el7_6.x86_64.rpm provided by mock
+# perl-Carp-1.26-244.el7.noarch.rpm provided by mock
+# perl-constant-1.27-2.el7.noarch.rpm provided by mock
+perl-Data-Dumper-2.145-3.el7.x86_64.rpm
+# perl-Encode-2.51-7.el7.x86_64.rpm provided by mock
+perl-Encode-Locale-1.03-5.el7.noarch.rpm
+perl-Error-0.17020-2.el7.noarch.rpm
+# perl-Exporter-5.68-3.el7.noarch.rpm provided by mock
+# perl-File-Path-2.09-2.el7.noarch.rpm provided by mock
+# perl-File-Temp-0.23.01-3.el7.noarch.rpm provided by mock
+# perl-Filter-1.49-3.el7.x86_64.rpm provided by mock
+# perl-Getopt-Long-2.40-3.el7.noarch.rpm provided by mock
+perl-Git-1.8.3.1-20.el7.noarch.rpm
+# perl-HTTP-Tiny-0.033-3.el7.noarch.rpm provided by mock
+perl-libintl-1.20-12.el7.x86_64.rpm
+# perl-libs-5.16.3-294.el7_6.x86_64.rpm provided by mock
+# perl-macros-5.16.3-294.el7_6.x86_64.rpm provided by mock
+# perl-parent-0.225-244.el7.noarch.rpm provided by mock
+# perl-PathTools-3.40-5.el7.x86_64.rpm provided by mock
+# perl-Pod-Escapes-1.04-294.el7_6.noarch.rpm provided by mock
+# perl-podlators-2.5.1-3.el7.noarch.rpm provided by mock
+# perl-Pod-Perldoc-3.20-4.el7.noarch.rpm provided by mock
+# perl-Pod-Simple-3.28-4.el7.noarch.rpm provided by mock
+# perl-Pod-Usage-1.63-3.el7.noarch.rpm provided by mock
+# perl-Scalar-List-Utils-1.27-248.el7.x86_64.rpm provided by mock
+# perl-Socket-2.010-4.el7.x86_64.rpm provided by mock
+# perl-Storable-2.45-3.el7.x86_64.rpm provided by mock
+perl-TermReadKey-2.30-20.el7.x86_64.rpm
+perl-Test-Harness-3.28-3.el7.noarch.rpm
+# perl-Text-ParseWords-3.29-4.el7.noarch.rpm provided by mock
+perl-Text-Unidecode-0.04-20.el7.noarch.rpm
+# perl-threads-1.87-4.el7.x86_64.rpm provided by mock
+# perl-threads-shared-1.43-6.el7.x86_64.rpm provided by mock
+# perl-Time-HiRes-1.9725-3.el7.x86_64.rpm provided by mock
+# perl-Time-Local-1.2300-2.el7.noarch.rpm provided by mock
+# pkgconfig-0.27.1-4.el7.x86_64.rpm provided by mock
+# popt-1.13-16.el7.x86_64.rpm provided by mock
+popt-devel-1.13-16.el7.x86_64.rpm
+# procps-ng-3.3.10-23.el7.x86_64.rpm provided by mock
+pyparsing-2.1.10-3.el7.noarch.rpm
+python2-pyparsing-2.1.10-3.el7.noarch.rpm
+python-2.7.5-76.el7.x86_64.rpm
+python-devel-2.7.5-76.el7.x86_64.rpm
+python-libs-2.7.5-76.el7.x86_64.rpm
+# qrencode-libs-3.4.1-3.el7.x86_64.rpm provided by mock
+# readline-6.2-10.el7.x86_64.rpm provided by mock
+readline-devel-6.2-10.el7.x86_64.rpm
+rpm-4.11.3-35.el7.x86_64.rpm
+rpm-build-4.11.3-35.el7.x86_64.rpm
+rpm-build-libs-4.11.3-35.el7.x86_64.rpm
+rpm-libs-4.11.3-35.el7.x86_64.rpm
+rpm-python-4.11.3-35.el7.x86_64.rpm
+rsync-3.1.2-4.el7.x86_64.rpm
+# sed-4.2.2-5.el7.x86_64.rpm provided by mock
+# setup-2.8.71-10.el7.noarch.rpm provided by mock
+# shadow-utils-4.1.5.1-25.el7.x86_64.rpm provided by mock
+# shared-mime-info-1.8-4.el7.x86_64.rpm provided by mock
+# sqlite-3.7.17-8.el7.x86_64.rpm provided by mock
+sqlite-devel-3.7.17-8.el7.x86_64.rpm
+subversion-1.7.14-14.el7.x86_64.rpm
+subversion-libs-1.7.14-14.el7.x86_64.rpm
+# tar-1.26-35.el7.x86_64.rpm provided by mock
+tcl-8.5.13-8.el7.x86_64.rpm
+tcl-devel-8.5.13-8.el7.x86_64.rpm
+texinfo-5.1-5.el7.x86_64.rpm
+tix-8.4.3-12.el7.x86_64.rpm
+tix-devel-8.4.3-12.el7.x86_64.rpm
+tk-8.5.13-6.el7.x86_64.rpm
+tk-devel-8.5.13-6.el7.x86_64.rpm
+trousers-0.3.14-2.el7.x86_64.rpm
+# tzdata-2018g-1.el7.noarch.rpm provided by mock
+# ustr-1.0.4-16.el7.x86_64.rpm provided by mock
+ustr-devel-1.0.4-16.el7.x86_64.rpm
+xorg-x11-proto-devel-2018.4-1.el7.noarch.rpm
+# xz-5.2.2-1.el7.x86_64.rpm provided by mock
+xz-devel-5.2.2-1.el7.x86_64.rpm
+# xz-libs-5.2.2-1.el7.x86_64.rpm provided by mock
+# zlib-1.2.7-18.el7.x86_64.rpm provided by mock
+zlib-devel-1.2.7-18.el7.x86_64.rpm

centos-mirror-tools/config/centos/compiler/rpms_centos3rdparties.lst

@@ -0,0 +1,38 @@
+# audit-libs-2.8.4-4.el7.x86_64.rpm provided by mock
+audit-libs-devel-2.8.4-4.el7.x86_64.rpm
+audit-libs-python-2.8.4-4.el7.x86_64.rpm
+devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64.rpm
+devtoolset-7-libstdc++-devel-7.3.1-5.10.el7.x86_64.rpm
+# elfutils-default-yama-scope-0.176-2.el7.noarch.rpm provided by mock
+# elfutils-devel-0.176-2.el7.x86_64.rpm provided by mock
+# elfutils-libelf-0.176-2.el7.x86_64.rpm provided by mock
+# elfutils-libelf-devel-0.176-2.el7.x86_64.rpm provided by mock
+# elfutils-libs-0.176-2.el7.x86_64.rpm provided by mock
+# glib2-2.56.1-2.el7.x86_64.rpm provided by mock
+ima-evm-utils-devel-1.1-2.el7.x86_64.rpm
+# libblkid-2.23.2-59.el7.x86_64.rpm provided by mock
+# libcom_err-1.42.9-13.el7.x86_64.rpm provided by mock
+libcom_err-devel-1.42.9-13.el7.x86_64.rpm
+libdrm-2.4.91-3.el7.x86_64.rpm
+libdrm-devel-2.4.91-3.el7.x86_64.rpm
+# libsemanage-2.5-14.el7.x86_64.rpm provided by mock
+libsemanage-devel-2.5-14.el7.x86_64.rpm
+mesa-libEGL-18.0.5-3.el7.x86_64.rpm
+mesa-libEGL-devel-18.0.5-3.el7.x86_64.rpm
+mesa-libgbm-18.0.5-3.el7.x86_64.rpm
+mesa-libGL-18.0.5-3.el7.x86_64.rpm
+mesa-libglapi-18.0.5-3.el7.x86_64.rpm
+mesa-libGL-devel-18.0.5-3.el7.x86_64.rpm
+# nss-softokn-3.36.0-5.el7_5.x86_64.rpm provided by mock
+nss-softokn-devel-3.36.0-5.el7_5.x86_64.rpm
+# nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm provided by mock
+nss-softokn-freebl-devel-3.36.0-5.el7_5.x86_64.rpm
+# openldap-2.4.44-20.el7.x86_64.rpm provided by mock
+# systemd-219-67.el7.x86_64.rpm provided by mock
+# systemd-devel-219-67.el7.x86_64.rpm provided by mock
+# systemd-libs-219-67.el7.x86_64.rpm provided by mock
+# systemd-sysv-219-67.el7.x86_64.rpm provided by mock
+systemtap-sdt-devel-3.3-3.el7.x86_64.rpm
+# util-linux-2.23.2-59.el7.x86_64.rpm provided by mock
+valgrind-3.13.0-13.el7.x86_64.rpm
+valgrind-devel-3.13.0-13.el7.x86_64.rpm

centos-mirror-tools/config/centos/distro/required_layer_iso_inc.cfg

@@ -0,0 +1,2 @@
+compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc
+compiler,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc

centos-mirror-tools/config/centos/distro/required_layer_pkgs.cfg

@@ -0,0 +1 @@
+compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/rpm.lst

centos-mirror-tools/config/centos/distro/rpms_3rdparties.lst

@@ -0,0 +1,7 @@
+python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-openstackdocstheme/1.23.2/1.el7/noarch/python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm
+python2-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python2-pip-9.0.3-6.fc29.noarch.rpm
+python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-httpbin/0.2.3/6.el7/noarch/python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm
+python2-pytest-mock-1.6.0-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-mock/1.6.0/2.el7/noarch/python2-pytest-mock-1.6.0-2.el7.noarch.rpm
+python2-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python2-wheel-0.31.1-3.fc30.noarch.rpm
+python3-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python3-pip-9.0.3-6.fc29.noarch.rpm
+python3-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python3-wheel-0.31.1-3.fc30.noarch.rpm

centos-mirror-tools/config/centos/distro/rpms_centos3rdparties.lst

@@ -0,0 +1,90 @@
+adwaita-cursor-theme-3.28.0-1.el7.noarch.rpm
+adwaita-icon-theme-3.28.0-1.el7.noarch.rpm
+alsa-lib-1.1.6-2.el7.x86_64.rpm
+alsa-lib-devel-1.1.6-2.el7.x86_64.rpm
+# audit-libs-2.8.4-4.el7.x86_64.rpm provided by mock
+audit-libs-devel-2.8.4-4.el7.x86_64.rpm
+audit-libs-python-2.8.4-4.el7.x86_64.rpm
+copy-jdk-configs-3.3-10.el7_5.noarch.rpm
+CUnit-2.1.3-8.el7.x86_64.rpm
+CUnit-devel-2.1.3-8.el7.x86_64.rpm
+devtoolset-7-binutils-2.28-11.el7.x86_64.rpm
+devtoolset-7-gcc-7.3.1-5.10.el7.x86_64.rpm
+devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64.rpm
+devtoolset-7-libstdc++-devel-7.3.1-5.10.el7.x86_64.rpm
+devtoolset-7-runtime-7.1-4.el7.x86_64.rpm
+# elfutils-0.176-2.el7.x86_64.rpm provided by mock
+# elfutils-default-yama-scope-0.176-2.el7.noarch.rpm provided by mock
+# elfutils-devel-0.176-2.el7.x86_64.rpm provided by mock
+# elfutils-libelf-0.176-2.el7.x86_64.rpm provided by mock
+# elfutils-libelf-devel-0.176-2.el7.x86_64.rpm provided by mock
+# elfutils-libs-0.176-2.el7.x86_64.rpm provided by mock
+# glib2-2.56.1-2.el7.x86_64.rpm provided by mock
+glib2-devel-2.56.1-2.el7.x86_64.rpm
+glib2-doc-2.56.1-2.el7.noarch.rpm
+gsettings-desktop-schemas-3.28.0-2.el7.x86_64.rpm
+gssproxy-0.7.0-21.el7.x86_64.rpm
+gtk3-3.22.30-3.el7.x86_64.rpm
+gtk3-devel-3.22.30-3.el7.x86_64.rpm
+gtk3-devel-docs-3.22.30-3.el7.x86_64.rpm
+gtk-update-icon-cache-3.22.30-3.el7.x86_64.rpm
+java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64.rpm
+java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64.rpm
+java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64.rpm
+# libblkid-2.23.2-59.el7.x86_64.rpm provided by mock
+# libcom_err-1.42.9-13.el7.x86_64.rpm provided by mock
+libcom_err-devel-1.42.9-13.el7.x86_64.rpm
+libdrm-2.4.91-3.el7.x86_64.rpm
+libdrm-devel-2.4.91-3.el7.x86_64.rpm
+liboath-2.4.1-9.el7.x86_64.rpm
+liboath-devel-2.4.1-9.el7.x86_64.rpm
+# libsemanage-2.5-14.el7.x86_64.rpm provided by mock
+libsemanage-python-2.5-14.el7.x86_64.rpm
+libss-1.42.9-13.el7.x86_64.rpm
+libtalloc-2.1.13-1.el7.x86_64.rpm
+libtevent-0.9.36-1.el7.x86_64.rpm
+lvm2-2.02.177-4.el7.x86_64.rpm
+lvm2-libs-2.02.177-4.el7.x86_64.rpm
+mesa-libEGL-18.0.5-3.el7.x86_64.rpm
+mesa-libEGL-devel-18.0.5-3.el7.x86_64.rpm
+mesa-libgbm-18.0.5-3.el7.x86_64.rpm
+mesa-libGL-18.0.5-3.el7.x86_64.rpm
+mesa-libglapi-18.0.5-3.el7.x86_64.rpm
+mesa-libGL-devel-18.0.5-3.el7.x86_64.rpm
+NetworkManager-glib-1.12.0-8.el7_6.x86_64.rpm
+NetworkManager-glib-devel-1.12.0-8.el7_6.x86_64.rpm
+# nss-softokn-3.36.0-5.el7_5.x86_64.rpm provided by mock
+nss-softokn-devel-3.36.0-5.el7_5.x86_64.rpm
+# nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm provided by mock
+nss-softokn-freebl-devel-3.36.0-5.el7_5.x86_64.rpm
+# openldap-2.4.44-20.el7.x86_64.rpm provided by mock
+policycoreutils-2.5-29.el7.x86_64.rpm
+policycoreutils-devel-2.5-29.el7.x86_64.rpm
+policycoreutils-python-2.5-29.el7.x86_64.rpm
+pulseaudio-libs-10.0-5.el7.x86_64.rpm
+pulseaudio-libs-devel-10.0-5.el7.x86_64.rpm
+pulseaudio-libs-glib2-10.0-5.el7.x86_64.rpm
+pykickstart-1.99.66.19-2.el7.noarch.rpm
+python2-msgpack-0.6.1-2.el7.x86_64.rpm
+python2-pysocks-1.6.8-6.el7.noarch.rpm
+python2-scapy-2.4.0-3.el7.noarch.rpm
+python2-urllib3-1.21.1-1.el7.noarch.rpm
+python-requests-mock-1.3.0-1.el7.noarch.rpm
+rest-0.8.1-2.el7.x86_64.rpm
+selinux-policy-3.13.1-229.el7_6.6.noarch.rpm
+selinux-policy-devel-3.13.1-229.el7_6.6.noarch.rpm
+spice-protocol-0.12.14-1.el7.noarch.rpm
+spice-server-0.14.0-6.el7.x86_64.rpm
+spice-server-devel-0.14.0-6.el7.x86_64.rpm
+# systemd-219-67.el7.x86_64.rpm provided by mock
+# systemd-devel-219-67.el7.x86_64.rpm provided by mock
+# systemd-libs-219-67.el7.x86_64.rpm provided by mock
+# systemd-sysv-219-67.el7.x86_64.rpm provided by mock
+systemtap-3.3-3.el7.x86_64.rpm
+systemtap-client-3.3-3.el7.x86_64.rpm
+systemtap-devel-3.3-3.el7.x86_64.rpm
+systemtap-runtime-3.3-3.el7.x86_64.rpm
+systemtap-sdt-devel-3.3-3.el7.x86_64.rpm
+# util-linux-2.23.2-59.el7.x86_64.rpm provided by mock
+valgrind-3.13.0-13.el7.x86_64.rpm
+valgrind-devel-3.13.0-13.el7.x86_64.rpm

centos-mirror-tools/config/centos/distro/yum.repos.d/StarlingX_cengn_compiler_layer.repo

@@ -0,0 +1,12 @@
+#####
+# StarlingX compiler layer at CENGN
+
+[Starlingx-cengn_compiler_std_layer]
+name=Starlingx-cengn_compiler_std_layer
+baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/
+enabled=1
+
+[Starlingx-cengn_compiler_installer_layer]
+name=Starlingx-cengn_compiler_installer_layer
+baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/installer/
+enabled=1

centos-mirror-tools/config/centos/flock/other_downloads.lst

@@ -0,0 +1,29 @@
+folder:EFI
+folder:EFI/BOOT
+#file:EFI/BOOT/mmx64.efi
+#file:EFI/BOOT/grubia32.efi
+file:EFI/BOOT/grub.cfg
+file:EFI/BOOT/BOOTX64.EFI
+file:EFI/BOOT/grubx64.efi
+#file:EFI/BOOT/BOOTIA32.EFI
+#file:EFI/BOOT/mmia32.efi
+folder:EFI/BOOT/fonts
+file:EFI/BOOT/fonts/unicode.pf2
+folder:LiveOS
+file:LiveOS/squashfs.img
+folder:images
+folder:images/pxeboot
+file:images/pxeboot/initrd.img
+file:images/pxeboot/vmlinuz
+file:images/efiboot.img
+#file:images/boot.iso
+folder:isolinux
+file:isolinux/memtest
+file:isolinux/grub.conf
+file:isolinux/initrd.img
+file:isolinux/boot.msg
+file:isolinux/vmlinuz
+file:isolinux/isolinux.bin
+file:isolinux/splash.png
+file:isolinux/isolinux.cfg
+file:isolinux/vesamenu.c32

centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg

@@ -0,0 +1,4 @@
+compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc
+compiler,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc
+distro,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/image.inc
+distro,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/image-dev.inc

centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg

@@ -0,0 +1,4 @@
+compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/rpm.lst
+distro,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/rpm.lst
+distro,rt,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/rt/rpm.lst
+distro,installer,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/installer/rpm.lst

centos-mirror-tools/config/centos/flock/rpms_3rdparties.lst

@@ -1,15 +1,10 @@
 ansible-2.7.5-1.el7.ans.noarch.rpm#https://releases.ansible.com/ansible/rpm/release/epel-7-x86_64/ansible-2.7.5-1.el7.ans.noarch.rpm
-libvirt-python-4.7.0-1.fc28.src.rpm#https://libvirt.org/sources/python/libvirt-python-4.7.0-1.fc28.src.rpm
+influxdb-0.9.5.1-1.x86_64.rpm#https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm
 novnc-0.6.2-1.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/novnc/0.6.2/1.el7/noarch/novnc-0.6.2-1.el7.noarch.rpm
+libvirt-python-4.7.0-1.fc28.src.rpm#https://libvirt.org/sources/python/libvirt-python-4.7.0-1.fc28.src.rpm
 python2-httpbin-0.5.0-6.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-httpbin/0.5.0/6.el7/noarch/python2-httpbin-0.5.0-6.el7.noarch.rpm
 python2-kubernetes-8.0.0-8.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-kubernetes/8.0.0/8.el7/noarch/python2-kubernetes-8.0.0-8.el7.noarch.rpm
-python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-httpbin/0.2.3/6.el7/noarch/python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm
-python2-pytest-mock-1.6.0-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-mock/1.6.0/2.el7/noarch/python2-pytest-mock-1.6.0-2.el7.noarch.rpm
-python2-storops-0.4.7-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-storops/0.4.7/2.el7/noarch/python2-storops-0.4.7-2.el7.noarch.rpm
-influxdb-0.9.5.1-1.x86_64.rpm#https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm
-python2-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python2-wheel-0.31.1-3.fc30.noarch.rpm
-python3-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python3-wheel-0.31.1-3.fc30.noarch.rpm
 python2-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python2-pip-9.0.3-6.fc29.noarch.rpm
-python3-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python3-pip-9.0.3-6.fc29.noarch.rpm
+python2-storops-0.4.7-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-storops/0.4.7/2.el7/noarch/python2-storops-0.4.7-2.el7.noarch.rpm
-python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-openstackdocstheme/1.23.2/1.el7/noarch/python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm
+python2-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python2-wheel-0.31.1-3.fc30.noarch.rpm
 python2-sphinxcontrib-apidoc-0.2.1-6.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-sphinxcontrib-apidoc/0.2.1/6.el7/noarch/python2-sphinxcontrib-apidoc-0.2.1-6.el7.noarch.rpm

centos-mirror-tools/config/centos/flock/rpms_centos3rdparties.lst

@@ -1,46 +1,25 @@
-adwaita-cursor-theme-3.28.0-1.el7.noarch.rpm
-adwaita-icon-theme-3.28.0-1.el7.noarch.rpm
 alsa-lib-1.1.6-2.el7.x86_64.rpm
-alsa-lib-devel-1.1.6-2.el7.x86_64.rpm
 audit-2.8.4-4.el7.x86_64.rpm
-audit-libs-2.8.4-4.el7.x86_64.rpm
+# audit-libs-2.8.4-4.el7.x86_64.rpm provided by mock
 audit-libs-devel-2.8.4-4.el7.x86_64.rpm
 audit-libs-python-2.8.4-4.el7.x86_64.rpm
 collectd-5.8.1-4.el7.x86_64.rpm
 collectd-python-5.8.1-4.el7.x86_64.rpm
 containernetworking-plugins-0.8.1-1.el7.centos.x86_64.rpm
-copy-jdk-configs-3.3-10.el7_5.noarch.rpm
 cppcheck-1.88-3.el7.x86_64.rpm
-CUnit-2.1.3-8.el7.x86_64.rpm
-CUnit-devel-2.1.3-8.el7.x86_64.rpm
-devtoolset-7-binutils-2.28-11.el7.x86_64.rpm
-devtoolset-7-gcc-7.3.1-5.10.el7.x86_64.rpm
-devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64.rpm
-devtoolset-7-libstdc++-devel-7.3.1-5.10.el7.x86_64.rpm
-devtoolset-7-runtime-7.1-4.el7.x86_64.rpm
 docker-ce-18.09.6-3.el7.x86_64.rpm
 docker-ce-cli-18.09.6-3.el7.x86_64.rpm
-elfutils-0.176-2.el7.x86_64.rpm
+# elfutils-default-yama-scope-0.176-2.el7.noarch.rpm provided by mock
-elfutils-default-yama-scope-0.176-2.el7.noarch.rpm
+# elfutils-devel-0.176-2.el7.x86_64.rpm provided by mock
-elfutils-devel-0.176-2.el7.x86_64.rpm
+# elfutils-libelf-0.176-2.el7.x86_64.rpm provided by mock
-elfutils-libelf-0.176-2.el7.x86_64.rpm
+# elfutils-libelf-devel-0.176-2.el7.x86_64.rpm provided by mock
-elfutils-libelf-devel-0.176-2.el7.x86_64.rpm
+# elfutils-libs-0.176-2.el7.x86_64.rpm provided by mock
-elfutils-libs-0.176-2.el7.x86_64.rpm
+# glib2-2.56.1-2.el7.x86_64.rpm provided by mock
-glib2-2.56.1-2.el7.x86_64.rpm
 glib2-devel-2.56.1-2.el7.x86_64.rpm
-glib2-doc-2.56.1-2.el7.noarch.rpm
 gsettings-desktop-schemas-3.28.0-2.el7.x86_64.rpm
 gssproxy-0.7.0-21.el7.x86_64.rpm
-gtk3-3.22.30-3.el7.x86_64.rpm
+# ima-evm-utils-1.1-2.el7.x86_64.rpm provided by mock
-gtk3-devel-3.22.30-3.el7.x86_64.rpm
-gtk3-devel-docs-3.22.30-3.el7.x86_64.rpm
-gtk-update-icon-cache-3.22.30-3.el7.x86_64.rpm
-ima-evm-utils-1.1-2.el7.x86_64.rpm
-ima-evm-utils-devel-1.1-2.el7.x86_64.rpm
 iprutils-2.4.16.1-1.el7.x86_64.rpm
-java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64.rpm
-java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64.rpm
-java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64.rpm
 kata-containers-image-1.10.0-4.1.x86_64.rpm
 kata-ksm-throttler-1.10.0-4.1.x86_64.rpm
 kata-linux-container-4.19.86.60-4.1.x86_64.rpm
@@ -50,18 +29,14 @@ kata-proxy-bin-1.10.0-4.1.x86_64.rpm
 kata-runtime-1.10.0-4.1.x86_64.rpm
 kata-shim-1.10.0-4.1.x86_64.rpm
 kata-shim-bin-1.10.0-4.1.x86_64.rpm
-kernel-rt-3.10.0-1062.1.2.rt56.1025.el7.src.rpm
 kexec-tools-2.0.15-21.el7.x86_64.rpm
-libblkid-2.23.2-59.el7.x86_64.rpm
+# libblkid-2.23.2-59.el7.x86_64.rpm provided by mock
-libcom_err-1.42.9-13.el7.x86_64.rpm
+# libcom_err-1.42.9-13.el7.x86_64.rpm provided by mock
 libcom_err-devel-1.42.9-13.el7.x86_64.rpm
 libdrm-2.4.91-3.el7.x86_64.rpm
-libdrm-devel-2.4.91-3.el7.x86_64.rpm
 libldb-1.3.4-1.el7.x86_64.rpm
 liboath-2.4.1-9.el7.x86_64.rpm
-liboath-devel-2.4.1-9.el7.x86_64.rpm
+# libsemanage-2.5-14.el7.x86_64.rpm provided by mock
-libsemanage-2.5-14.el7.x86_64.rpm
-libsemanage-devel-2.5-14.el7.x86_64.rpm
 libsemanage-python-2.5-14.el7.x86_64.rpm
 libss-1.42.9-13.el7.x86_64.rpm
 libss-devel-1.42.9-13.el7.x86_64.rpm
@@ -71,36 +46,22 @@ libtevent-0.9.36-1.el7.x86_64.rpm
 libwbclient-4.8.3-4.el7.x86_64.rpm
 lvm2-2.02.177-4.el7.x86_64.rpm
 lvm2-libs-2.02.177-4.el7.x86_64.rpm
-mesa-libEGL-18.0.5-3.el7.x86_64.rpm
+# nss-softokn-3.36.0-5.el7_5.x86_64.rpm provided by mock
-mesa-libEGL-devel-18.0.5-3.el7.x86_64.rpm
-mesa-libgbm-18.0.5-3.el7.x86_64.rpm
-mesa-libGL-18.0.5-3.el7.x86_64.rpm
-mesa-libglapi-18.0.5-3.el7.x86_64.rpm
-mesa-libGL-devel-18.0.5-3.el7.x86_64.rpm
-NetworkManager-glib-1.12.0-8.el7_6.x86_64.rpm
-NetworkManager-glib-devel-1.12.0-8.el7_6.x86_64.rpm
-nss-softokn-3.36.0-5.el7_5.x86_64.rpm
 nss-softokn-devel-3.36.0-5.el7_5.x86_64.rpm
-nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm
+# nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm provided by mock
 nss-softokn-freebl-devel-3.36.0-5.el7_5.x86_64.rpm
 ntfs-3g-2017.3.23-11.el7.x86_64.rpm
 ntfs-3g-devel-2017.3.23-11.el7.x86_64.rpm
 ntfsprogs-2017.3.23-11.el7.x86_64.rpm
-openldap-2.4.44-20.el7.x86_64.rpm
 openscap-1.2.17-2.el7.x86_64.rpm
 openscap-scanner-1.2.17-2.el7.x86_64.rpm
 openstack-tempest-17.2.0-4.el7.noarch.rpm
 osinfo-db-20180531-1.el7.noarch.rpm
 OVMF-20180508-6.gitee3198e672e2.el7.noarch.rpm
 policycoreutils-2.5-29.el7.x86_64.rpm
-policycoreutils-devel-2.5-29.el7.x86_64.rpm
 policycoreutils-newrole-2.5-29.el7.x86_64.rpm
 policycoreutils-python-2.5-29.el7.x86_64.rpm
 pulseaudio-libs-10.0-5.el7.x86_64.rpm
-pulseaudio-libs-devel-10.0-5.el7.x86_64.rpm
-pulseaudio-libs-glib2-10.0-5.el7.x86_64.rpm
-pykickstart-1.99.66.19-2.el7.noarch.rpm
-python-2.7.5-76.el7.x86_64.rpm
 python2-google-api-client-1.6.3-1.el7.noarch.rpm
 python2-manilaclient-1.17.3-1.el7.noarch.rpm
 python2-mistralclient-3.1.4-1.el7.noarch.rpm
@@ -111,43 +72,25 @@ python2-scapy-2.4.0-3.el7.noarch.rpm
 python2-sushy-1.1.1-1.el7.noarch.rpm
 python2-tempestconf-1.1.3-1.el7.noarch.rpm
 python2-urllib3-1.21.1-1.el7.noarch.rpm
-python-devel-2.7.5-76.el7.x86_64.rpm
 python-ironic-inspector-client-2.1.0-1.el7.noarch.rpm
 python-ironic-lib-2.10.1-1.el7.noarch.rpm
 python-libguestfs-1.38.2-12.el7_6.1.x86_64.rpm
-python-libs-2.7.5-76.el7.x86_64.rpm
-python-requests-mock-1.3.0-1.el7.noarch.rpm
-python-setuptools-22.0.5-1.el7.src.rpm
 python-tempest-17.2.0-4.el7.noarch.rpm
 python-virtualenv-15.1.0-2.el7.noarch.rpm
 qemu-vanilla-4.1.0+git.9e06029aea-4.1.x86_64.rpm
 qemu-vanilla-bin-4.1.0+git.9e06029aea-4.1.x86_64.rpm
 qemu-vanilla-data-4.1.0+git.9e06029aea-4.1.x86_64.rpm
-rest-0.8.1-2.el7.x86_64.rpm
 samba-client-libs-4.8.3-4.el7.x86_64.rpm
 samba-common-4.8.3-4.el7.noarch.rpm
 samba-common-libs-4.8.3-4.el7.x86_64.rpm
 selinux-policy-3.13.1-229.el7_6.6.noarch.rpm
-selinux-policy-devel-3.13.1-229.el7_6.6.noarch.rpm
 selinux-policy-minimum-3.13.1-229.el7_6.6.noarch.rpm
 selinux-policy-mls-3.13.1-229.el7_6.6.noarch.rpm
 selinux-policy-targeted-3.13.1-229.el7_6.6.noarch.rpm
-spice-protocol-0.12.14-1.el7.noarch.rpm
 spice-server-0.14.0-6.el7.x86_64.rpm
-spice-server-devel-0.14.0-6.el7.x86_64.rpm
-systemd-219-67.el7.x86_64.rpm
-systemd-devel-219-67.el7.x86_64.rpm
-systemd-libs-219-67.el7.x86_64.rpm
-systemd-sysv-219-67.el7.x86_64.rpm
-systemtap-3.3-3.el7.x86_64.rpm
-systemtap-client-3.3-3.el7.x86_64.rpm
-systemtap-devel-3.3-3.el7.x86_64.rpm
-systemtap-runtime-3.3-3.el7.x86_64.rpm
 systemtap-sdt-devel-3.3-3.el7.x86_64.rpm
 tcpdump-4.9.2-3.el7.x86_64.rpm
 tkinter-2.7.5-76.el7.x86_64.rpm
 tuned-2.8.0-5.el7.noarch.rpm
 tuned-profiles-realtime-2.8.0-5.el7.noarch.rpm
-util-linux-2.23.2-59.el7.x86_64.rpm
+# util-linux-2.23.2-59.el7.x86_64.rpm provided by mock
-valgrind-3.13.0-13.el7.x86_64.rpm
-valgrind-devel-3.13.0-13.el7.x86_64.rpm

centos-mirror-tools/config/centos/flock/yum.repos.d/StarlingX_cengn_compiler_layer.repo

@@ -0,0 +1,12 @@
+#####
+# StarlingX compiler layer at CENGN
+
+[Starlingx-cengn_compiler_std_layer]
+name=Starlingx-cengn_compiler_std_layer
+baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/
+enabled=1
+
+[Starlingx-cengn_compiler_installer_layer]
+name=Starlingx-cengn_compiler_installer_layer
+baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/installer/
+enabled=1

centos-mirror-tools/config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo

@@ -0,0 +1,18 @@
+#####
+# StarlingX distro layer at CENGN
+
+[Starlingx-cengn_distro_std_layer]
+name=Starlingx-cengn_distro_std_layer
+baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/
+enabled=1
+
+[Starlingx-cengn_distro_rt_layer]
+name=Starlingx-cengn_distro_rt_layer
+baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/rt/
+enabled=1
+
+[Starlingx-cengn_distro_installer_layer]
+name=Starlingx-cengn_distro_installer_layer
+baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/installer/
+enabled=1
+

centos-mirror-tools/config/centos/mock/rpms_centos.lst

@@ -0,0 +1,180 @@
+acl-2.2.51-14.el7.x86_64.rpm
+basesystem-10.0-7.el7.centos.noarch.rpm
+# bash-4.2.46-31.el7.x86_64.rpm
+binutils-2.27-34.base.el7.x86_64.rpm
+bzip2-1.0.6-13.el7.x86_64.rpm
+bzip2-libs-1.0.6-13.el7.x86_64.rpm
+ca-certificates-2018.2.22-70.0.el7_5.noarch.rpm
+centos-release-7-6.1810.2.el7.centos.x86_64.rpm
+chkconfig-1.7.4-1.el7.x86_64.rpm
+coreutils-8.22-23.el7.x86_64.rpm
+cpio-2.11-27.el7.x86_64.rpm
+cpp-4.8.5-36.el7.x86_64.rpm
+cracklib-2.9.0-11.el7.x86_64.rpm
+cracklib-dicts-2.9.0-11.el7.x86_64.rpm
+cryptsetup-libs-2.0.3-3.el7.x86_64.rpm
+curl-7.29.0-51.el7_6.3.x86_64.rpm
+cyrus-sasl-lib-2.1.26-23.el7.x86_64.rpm
+dbus-1.10.24-12.el7.x86_64.rpm
+dbus-libs-1.10.24-12.el7.x86_64.rpm
+device-mapper-1.02.146-4.el7.x86_64.rpm
+device-mapper-libs-1.02.146-4.el7.x86_64.rpm
+diffutils-3.3-4.el7.x86_64.rpm
+dracut-033-554.el7.x86_64.rpm
+dwz-0.11-3.el7.x86_64.rpm
+epel-release-7-11.noarch.rpm
+epel-rpm-macros-7-21.noarch.rpm
+expat-2.1.0-10.el7_3.x86_64.rpm
+file-5.11-35.el7.x86_64.rpm
+file-libs-5.11-35.el7.x86_64.rpm
+filesystem-3.2-25.el7.x86_64.rpm
+findutils-4.5.11-6.el7.x86_64.rpm
+gawk-4.0.2-4.el7_3.1.x86_64.rpm
+gcc-4.8.5-36.el7.x86_64.rpm
+gcc-c++-4.8.5-36.el7.x86_64.rpm
+gdb-7.6.1-114.el7.x86_64.rpm
+gdbm-1.10-8.el7.x86_64.rpm
+glibc-2.17-260.el7.x86_64.rpm
+glibc-common-2.17-260.el7.x86_64.rpm
+glibc-devel-2.17-260.el7.x86_64.rpm
+glibc-headers-2.17-260.el7.x86_64.rpm
+gmp-6.0.0-15.el7.x86_64.rpm
+gnupg2-2.0.22-5.el7_5.x86_64.rpm
+gpgme-1.3.2-5.el7.x86_64.rpm
+grep-2.20-3.el7.x86_64.rpm
+groff-base-1.22.2-8.el7.x86_64.rpm
+gzip-1.5-10.el7.x86_64.rpm
+hardlink-1.0-19.el7.x86_64.rpm
+hostname-3.13-3.el7.x86_64.rpm
+info-5.1-5.el7.x86_64.rpm
+json-c-0.11-4.el7_0.x86_64.rpm
+keyutils-libs-1.5.8-3.el7.x86_64.rpm
+kmod-20-23.el7.x86_64.rpm
+kmod-libs-20-23.el7.x86_64.rpm
+kpartx-0.4.9-119.el7.x86_64.rpm
+krb5-libs-1.15.1-34.el7.x86_64.rpm
+lbzip2-2.5-1.el7.x86_64.rpm
+libacl-2.2.51-14.el7.x86_64.rpm
+libarchive-3.1.2-10.el7_2.x86_64.rpm
+libassuan-2.1.0-3.el7.x86_64.rpm
+libattr-2.4.46-13.el7.x86_64.rpm
+libcap-2.22-9.el7.x86_64.rpm
+libcap-ng-0.7.5-4.el7.x86_64.rpm
+libcurl-7.29.0-51.el7_6.3.x86_64.rpm
+libcurl-devel-7.29.0-51.el7_6.3.x86_64.rpm
+libdb-5.3.21-24.el7.x86_64.rpm
+libdb-utils-5.3.21-24.el7.x86_64.rpm
+libffi-3.0.13-18.el7.x86_64.rpm
+libgcc-4.8.5-36.el7.x86_64.rpm
+libgcrypt-1.5.3-14.el7.x86_64.rpm
+libgomp-4.8.5-36.el7.x86_64.rpm
+libgpg-error-1.12-3.el7.x86_64.rpm
+libidn-1.28-4.el7.x86_64.rpm
+libmount-2.23.2-59.el7.x86_64.rpm
+libmpc-1.0.1-3.el7.x86_64.rpm
+libpwquality-1.2.3-5.el7.x86_64.rpm
+libselinux-2.5-14.1.el7.x86_64.rpm
+libsepol-2.5-10.el7.x86_64.rpm
+libsmartcols-2.23.2-59.el7.x86_64.rpm
+libssh2-1.4.3-12.el7.x86_64.rpm
+libstdc++-4.8.5-36.el7.x86_64.rpm
+libstdc++-devel-4.8.5-36.el7.x86_64.rpm
+libtasn1-4.10-1.el7.x86_64.rpm
+libuser-0.60-9.el7.x86_64.rpm
+libutempter-1.1.6-4.el7.x86_64.rpm
+libuuid-2.23.2-59.el7.x86_64.rpm
+libverto-0.2.5-4.el7.x86_64.rpm
+libxml2-2.9.1-6.el7_2.3.x86_64.rpm
+lua-5.1.4-15.el7.x86_64.rpm
+lz4-1.7.5-2.el7.x86_64.rpm
+lzo-2.06-8.el7.x86_64.rpm
+make-3.82-23.el7.x86_64.rpm
+mpfr-3.1.1-4.el7.x86_64.rpm
+ncurses-5.9-14.20130511.el7_4.x86_64.rpm
+ncurses-base-5.9-14.20130511.el7_4.noarch.rpm
+ncurses-libs-5.9-14.20130511.el7_4.x86_64.rpm
+nspr-4.19.0-1.el7_5.x86_64.rpm
+nss-3.36.0-7.el7_5.x86_64.rpm
+nss-pem-1.0.3-5.el7.x86_64.rpm
+nss-sysinit-3.36.0-7.el7_5.x86_64.rpm
+nss-tools-3.36.0-7.el7_5.x86_64.rpm
+nss-util-3.36.0-1.el7_5.x86_64.rpm
+openldap-2.4.44-20.el7.x86_64.rpm
+openssl-libs-1.0.2k-16.el7.x86_64.rpm
+p11-kit-0.23.5-3.el7.x86_64.rpm
+p11-kit-trust-0.23.5-3.el7.x86_64.rpm
+pam-1.1.8-22.el7.x86_64.rpm
+patch-2.7.1-10.el7_5.x86_64.rpm
+pcre-8.32-17.el7.x86_64.rpm
+perl-5.16.3-294.el7_6.x86_64.rpm
+perl-Carp-1.26-244.el7.noarch.rpm
+perl-constant-1.27-2.el7.noarch.rpm
+perl-Encode-2.51-7.el7.x86_64.rpm
+perl-Exporter-5.68-3.el7.noarch.rpm
+perl-File-Path-2.09-2.el7.noarch.rpm
+perl-File-Temp-0.23.01-3.el7.noarch.rpm
+perl-Filter-1.49-3.el7.x86_64.rpm
+perl-Getopt-Long-2.40-3.el7.noarch.rpm
+perl-HTTP-Tiny-0.033-3.el7.noarch.rpm
+perl-libs-5.16.3-294.el7_6.x86_64.rpm
+perl-macros-5.16.3-294.el7_6.x86_64.rpm
+perl-parent-0.225-244.el7.noarch.rpm
+perl-PathTools-3.40-5.el7.x86_64.rpm
+perl-Pod-Escapes-1.04-294.el7_6.noarch.rpm
+perl-podlators-2.5.1-3.el7.noarch.rpm
+perl-Pod-Perldoc-3.20-4.el7.noarch.rpm
+perl-Pod-Simple-3.28-4.el7.noarch.rpm
+perl-Pod-Usage-1.63-3.el7.noarch.rpm
+perl-Scalar-List-Utils-1.27-248.el7.x86_64.rpm
+perl-Socket-2.010-4.el7.x86_64.rpm
+perl-srpm-macros-1-8.el7.noarch.rpm
+perl-Storable-2.45-3.el7.x86_64.rpm
+perl-Text-ParseWords-3.29-4.el7.noarch.rpm
+perl-Thread-Queue-3.02-2.el7.noarch.rpm
+perl-threads-1.87-4.el7.x86_64.rpm
+perl-threads-shared-1.43-6.el7.x86_64.rpm
+perl-Time-HiRes-1.9725-3.el7.x86_64.rpm
+perl-Time-Local-1.2300-2.el7.noarch.rpm
+pigz-2.3.4-1.el7.x86_64.rpm
+pinentry-0.8.1-17.el7.x86_64.rpm
+pkgconfig-0.27.1-4.el7.x86_64.rpm
+popt-1.13-16.el7.x86_64.rpm
+procps-ng-3.3.10-23.el7.x86_64.rpm
+pth-2.0.7-23.el7.x86_64.rpm
+pygpgme-0.3-9.el7.x86_64.rpm
+pyliblzma-0.5.3-11.el7.x86_64.rpm
+# python-2.7.5-76.el7.x86_64.rpm
+# python-devel-2.7.5-76.el7.x86_64.rpm
+python2-rpm-macros-3-25.el7.noarch.rpm
+python-iniparse-0.4-9.el7.noarch.rpm
+# python-libs-2.7.5-76.el7.x86_64.rpm
+python-pycurl-7.19.0-19.el7.x86_64.rpm
+python-rpm-macros-3-25.el7.noarch.rpm
+python-srpm-macros-3-25.el7.noarch.rpm
+python-urlgrabber-3.10-9.el7.noarch.rpm
+pyxattr-0.5.1-5.el7.x86_64.rpm
+qrencode-libs-3.4.1-3.el7.x86_64.rpm
+readline-6.2-10.el7.x86_64.rpm
+redhat-rpm-config-9.1.0-87.el7.centos.noarch.rpm
+# rpm-4.11.3-35.el7.x86_64.rpm
+# rpm-build-4.11.3-35.el7.x86_64.rpm
+# rpm-build-libs-4.11.3-35.el7.x86_64.rpm
+# rpm-libs-4.11.3-35.el7.x86_64.rpm
+# rpm-python-4.11.3-35.el7.x86_64.rpm
+sed-4.2.2-5.el7.x86_64.rpm
+setup-2.8.71-10.el7.noarch.rpm
+shadow-utils-4.1.5.1-25.el7.x86_64.rpm
+shared-mime-info-1.8-4.el7.x86_64.rpm
+sqlite-3.7.17-8.el7.x86_64.rpm
+tar-1.26-35.el7.x86_64.rpm
+tzdata-2018g-1.el7.noarch.rpm
+unzip-6.0-19.el7.x86_64.rpm
+ustr-1.0.4-16.el7.x86_64.rpm
+which-2.20-7.el7.x86_64.rpm
+xz-5.2.2-1.el7.x86_64.rpm
+xz-libs-5.2.2-1.el7.x86_64.rpm
+yum-3.4.3-161.el7.centos.noarch.rpm
+yum-metadata-parser-1.1.4-10.el7.x86_64.rpm
+yum-plugin-fastestmirror-1.1.31-50.el7.noarch.rpm
+zip-3.0-11.el7.x86_64.rpm
+zlib-1.2.7-18.el7.x86_64.rpm

centos-mirror-tools/config/centos/mock/rpms_centos3rdparties.lst

@@ -0,0 +1,19 @@
+audit-libs-2.8.4-4.el7.x86_64.rpm
+elfutils-0.176-2.el7.x86_64.rpm
+elfutils-default-yama-scope-0.176-2.el7.noarch.rpm
+elfutils-devel-0.176-2.el7.x86_64.rpm
+elfutils-libelf-0.176-2.el7.x86_64.rpm
+elfutils-libelf-devel-0.176-2.el7.x86_64.rpm
+elfutils-libs-0.176-2.el7.x86_64.rpm
+glib2-2.56.1-2.el7.x86_64.rpm
+ima-evm-utils-1.1-2.el7.x86_64.rpm
+libblkid-2.23.2-59.el7.x86_64.rpm
+libcom_err-1.42.9-13.el7.x86_64.rpm
+libsemanage-2.5-14.el7.x86_64.rpm
+nss-softokn-3.36.0-5.el7_5.x86_64.rpm
+nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm
+systemd-219-67.el7.x86_64.rpm
+systemd-devel-219-67.el7.x86_64.rpm
+systemd-libs-219-67.el7.x86_64.rpm
+systemd-sysv-219-67.el7.x86_64.rpm
+util-linux-2.23.2-59.el7.x86_64.rpm

centos-mirror-tools/dl_other_from_centos_repo.sh

@@ -151,6 +151,12 @@ for ff in $all; do
                     let i+=1
                     echo "$file_name is downloaded successfully"
 
+                    mkdir -p $save_path/$sub_path
+                    if [ $? -ne 0 ]; then
+                        echo "Error: mkdir -p '$save_path/$sub_path'"
+                        error_count=$((error_count + 1))
+                    fi
+
                     \mv -f ./$file_name $save_path/$_name
                     if [ $? -ne 0 ]; then
                         echo "Error: mv -f './$file_name' '$save_path/$_name'"

centos-mirror-tools/dl_rpms.sh

@@ -153,13 +153,15 @@ fi
 timestamp=$(date +%F_%H%M)
 echo $timestamp
 
-DESTDIR="output"
+export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
-MDIR_SRC=$DESTDIR/stx-r1/CentOS/pike/Source
+export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
-mkdir -p $MDIR_SRC
-MDIR_BIN=$DESTDIR/stx-r1/CentOS/pike/Binary
-mkdir -p $MDIR_BIN
 
-LOGSDIR="logs"
+MDIR_SRC="${DL_MIRROR_OUTPUT_DIR}/Source"
+mkdir -p "$MDIR_SRC"
+MDIR_BIN="${DL_MIRROR_OUTPUT_DIR}/Binary"
+mkdir -p "$MDIR_BIN"
+
+LOGSDIR="${DL_MIRROR_LOG_DIR}"
 from=$(get_from $rpms_list)
 LOG="$LOGSDIR/${match_level}_failmoved_url_${from}.log"
 MISSING_SRPMS="$LOGSDIR/${match_level}_srpms_missing_${from}.log"

centos-mirror-tools/dl_rpms_from_url.sh

@@ -0,0 +1,179 @@
+#!/bin/bash -e
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# download RPMs/SRPMs from a base url.
+# this script was originated by Scott Little
+
+set -o errexit
+set -o nounset
+
+# By default, we use "sudo" and we don't use a local yum.conf. These can
+# be overridden via flags.
+
+DL_RPMS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
+
+source $DL_RPMS_DIR/utils.sh
+
+BASE_URL=""
+
+usage() {
+    echo "$0 -u <base url> <rpms_list> "
+    echo ""
+    echo "Options:"
+    echo "  -u: Use provided base url"
+    echo ""
+    echo "Returns: 0 = All files downloaded successfully"
+    echo "         1 = Some files could not be downloaded"
+    echo "         2 = Bad arguements or other error"
+    echo ""
+}
+
+
+CLEAN_LOGS_ONLY=0
+dl_rc=0
+
+
+distro="centos"
+
+# Parse option flags
+while getopts "u:h" o; do
+    case "${o}" in
+        u)
+            # Use an alternate yum.conf
+            BASE_URL="$OPTARG"
+            ;;
+        h)
+            # Help
+            usage
+            exit 0
+            ;;
+        *)
+            usage
+            exit 2
+            ;;
+    esac
+done
+shift $((OPTIND-1))
+
+if [ $# -lt 1 ]; then
+    usage
+    exit 2
+fi
+
+if [ "$1" == "" ]; then
+    echo "Need to supply the rpm file list"
+    exit 2;
+else
+    rpms_list=$1
+    echo "using $rpms_list as the download name lists"
+fi
+
+if [ ! -f "${rpms_list}" ]; then
+    echo "Error: File not found: ${rpms_list}"
+    usage
+    exit 2
+fi
+
+timestamp=$(date +%F_%H%M)
+echo $timestamp
+
+export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
+export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
+
+MDIR_SRC="${DL_MIRROR_OUTPUT_DIR}/Source"
+mkdir -p "$MDIR_SRC"
+MDIR_BIN="${DL_MIRROR_OUTPUT_DIR}/Binary"
+mkdir -p "$MDIR_BIN"
+
+LOGSDIR="${DL_MIRROR_LOG_DIR}"
+from=$(get_from $rpms_list)
+LOG="$LOGSDIR/L1_failmoved_url_${from}.log"
+MISSING_SRPMS="$LOGSDIR/srpms_missing_${from}.log"
+MISSING_RPMS="$LOGSDIR/rpms_missing_${from}.log"
+FOUND_SRPMS="$LOGSDIR/srpms_found_${from}.log"
+FOUND_RPMS="$LOGSDIR/rpms_found_${from}.log"
+cat /dev/null > $LOG
+cat /dev/null > $MISSING_SRPMS
+cat /dev/null > $MISSING_RPMS
+cat /dev/null > $FOUND_SRPMS
+cat /dev/null > $FOUND_RPMS
+
+
+if [ $CLEAN_LOGS_ONLY -eq 1 ];then
+    exit 0
+fi
+
+if [ "$BASE_URL" == "" ]; then
+    BASE_URL=file://$(readlink -f $(dirname ${rpms_list}))
+fi
+
+# Function to download different types of RPMs in different ways
+download () {
+    local _file=$1
+    local _url=$2
+    local _list=""
+    local _from=""
+
+    local _arch=""
+
+    local rc=0
+    local download_cmd=""
+    local download_url=""
+    local rpm_name=""
+    local SFILE=""
+    local lvl
+    local dl_result
+
+    _list=$(cat $_file)
+    _from=$(get_from $_file)
+
+    echo "now the rpm will come from: $_from"
+    for ff in $_list; do
+        _arch=$(get_arch_from_rpm $ff)
+        rpm_name="$(get_rpm_name $ff)"
+        dest_dir="$(get_dest_directory $_arch)"
+
+        if [ ! -e $dest_dir/$rpm_name ]; then
+            dl_result=1
+
+            download_url="$_url/$rpm_name"
+            download_cmd="curl --silent --output $rpm_name ${download_url}"
+
+            echo "Looking for $rpm_name"
+            echo "--> run: $download_cmd"
+            if $download_cmd ; then
+                SFILE="$(get_rpm_level_name $rpm_name L1)"
+                process_result "$_arch" "$dest_dir" "$download_url" "$SFILE"
+                dl_result=0
+            else
+                echo "Warning: $rpm_name not found"
+            fi
+
+            if [ $dl_result -eq 1 ]; then
+                echo "Error: $rpm_name not found"
+                echo "missing_srpm:$rpm_name" >> $LOG
+                echo $rpm_name >> $MISSING_SRPMS
+                rc=1
+            fi
+        else
+            echo "Already have $dest_dir/$rpm_name"
+        fi
+        echo
+    done
+
+    return $rc
+}
+
+# Download files
+if [ -s "$rpms_list" ];then
+    echo "--> start searching $rpms_list"
+    download $rpms_list $BASE_URL
+    if [ $? -ne 0 ]; then
+        dl_rc=1
+    fi
+fi
+
+echo "Done!"
+
+exit $dl_rc

centos-mirror-tools/dl_tarball.sh

@@ -16,15 +16,16 @@
 # input files:
 # The file tarball-dl.lst contains the list of packages and artifacts for
 # building this sub-mirror.
-script_path="$(dirname $(readlink -f $0))"
+tarball_file=""
-tarball_file="$script_path/tarball-dl.lst"
 
+set -x
 DL_TARBALL_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
 
 source $DL_TARBALL_DIR/url_utils.sh
+source $DL_TARBALL_DIR/utils.sh
 
 usage () {
-    echo "$0 [-D <distro>] [-s|-S|-u|-U] [-h] <other_download_list.ini> <save_path> [<force_update>]"
+    echo "$0 [-D <distro>] [-s|-S|-u|-U] [-h] <path_to_tarball_dl.lst>"
 }
 
 # Permitted values of dl_source
@@ -96,6 +97,8 @@ while getopts "D:hsSuU" o; do
     esac
 done
 shift $((OPTIND-1))
+tarball_file="${1}"
+shift
 
 
 if [ ! -e $tarball_file ]; then
@@ -108,10 +111,12 @@ fi
 # - Puppet hosted under "downloads/puppet" output directory.
 # to be populated under $MY_REPO/addons/wr-cgcs/layers/cgcs/downloads/puppet
 
-logs_dir="$script_path/logs"
+export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
-output_main="$script_path/output"
+export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
+
+logs_dir="${DL_MIRROR_LOG_DIR}"
 output_log="$logs_dir/log_download_tarball_missing.txt"
-output_path=$output_main/stx-r1/CentOS/pike
+output_path="${DL_MIRROR_OUTPUT_DIR}"
 output_tarball=$output_path/downloads
 output_puppet=$output_tarball/puppet
 
@@ -122,10 +127,12 @@ if [ ! -d "$logs_dir" ]; then
 fi
 
 is_tarball() {
-    tarball_name="$1"
+    local tarball_name="$1"
+    local mime_type
+    local types=("gzip" "x-bzip2" "x-rpm" "x-xz" "x-gzip" "x-tar")
+    local FOUND=1
+
     mime_type=$(file --mime-type -b $tarball_name | cut -d "/" -f 2)
-    types=("gzip" "x-bzip2" "x-rpm" "x-xz" "x-gzip" "x-tar")
-    FOUND=1
     for t in "${types[@]}"; do
         if [ "$mime_type" == "$t" ]; then
             FOUND=0

centos-mirror-tools/download_mirror.sh

@@ -3,20 +3,47 @@
 # SPDX-License-Identifier: Apache-2.0
 #
 
+DOWNLOAD_MIRROR_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
+
+source $DOWNLOAD_MIRROR_DIR/../toCOPY/lst_utils.sh
+
+export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
+export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
+
 usage() {
-    echo "$0 [-n] [-c <yum.conf>] [-g] [-s|-S|-u|-U]"
+    echo "$0 [options]"
-    echo ""
+    echo
-    echo "Options:"
+    echo "Common Options:"
-    echo "  -n: Do not use sudo when performing operations (option passed on to"
+    echo "  -c <yum.conf>: Use an alternate yum.conf rather than the system file"
-    echo "      subscripts when appropriate)"
+    echo "                 Suggested valur is 'yum.conf.sample' in this directory."
-    echo "  -c: Use an alternate yum.conf rather than the system file (option passed"
+    echo "                 (option passed on to subscripts when appropriate)"
-    echo "      on to subscripts when appropriate)"
+    echo "  -d <distro>:   Download package to build designated distro. Default 'centos'"
-    echo "  -g: do not change group IDs of downloaded artifacts"
+    echo "  -g:            Do not change group IDs of downloaded artifacts"
+    echo "  -l <layer>:    Download only packages required to build a given layer."
+    echo "                 Default: use the LAYER environmnet variable, or 'all'."
+    echo "  -n:            Do not use sudo when performing operations."
+    echo "                 (option passed on to subscripts when appropriate)"
+    echo
+    echo "Download Source Options:  Only select one of these."
     echo "  -s: Download from StarlingX mirror only"
     echo "  -S: Download from StarlingX mirror, upstream as backup (default)"
     echo "  -u: Download from original upstream sources only"
     echo "  -U: Download from original upstream sources, StarlingX mirror as backup"
-    echo ""
+    echo
+    echo "Layered Build Options:   For use when building multiple layers locally."
+    echo "  -C <config_dir>: Use an alternate config directory rather than the system"
+    echo "                   defined one"
+    echo "  -I <lower_layer>,<build_type>,<url>:"
+    echo "                   Override the url for the image include file of a lower"
+    echo "                   layer's build type.  Normally the url(s) is read from"
+    echo "                   <config_dir>/<distro>/<layer>/required_layer_iso_inc.cfg"
+    echo "                   This option can be used more than once."
+    echo "  -L <lower_layer>,<build_type>,<url>:"
+    echo "                   Override the url for the package list of a lower"
+    echo "                   layer's build type.  Normally the url(s) is read from"
+    echo "                   <config_dir>/<distro>/<layer>/required_layer_pkgs.cfg."
+    echo "                   This option can be used more than once."
+    echo
 }
 
 generate_log_name() {
@@ -35,6 +62,15 @@ need_file(){
     done
 }
 
+make_if_needed_file(){
+    for f in $*; do
+        if [ ! -f $f ]; then
+            echo "Creating empty file '$f'"
+            touch $f
+        fi
+    done
+}
+
 need_dir(){
     for d in $*; do
         if [ ! -d $d ]; then
@@ -45,10 +81,11 @@ need_dir(){
 }
 
 # Downloader scripts
-rpm_downloader="./dl_rpms.sh"
+rpm_downloader="${DOWNLOAD_MIRROR_DIR}/dl_rpms.sh"
-tarball_downloader="./dl_tarball.sh"
+rpm_from_url_downloader="${DOWNLOAD_MIRROR_DIR}/dl_rpms_from_url.sh"
-other_downloader="./dl_other_from_centos_repo.sh"
+tarball_downloader="${DOWNLOAD_MIRROR_DIR}/dl_tarball.sh"
-make_stx_mirror_yum_conf="./make_stx_mirror_yum_conf.sh"
+other_downloader="${DOWNLOAD_MIRROR_DIR}/dl_other_from_centos_repo.sh"
+make_stx_mirror_yum_conf="${DOWNLOAD_MIRROR_DIR}/make_stx_mirror_yum_conf.sh"
 
 # track optional arguments
 change_group_ids=1
@@ -57,13 +94,18 @@ alternate_yum_conf=""
 alternate_repo_dir=""
 rpm_downloader_extra_args=""
 tarball_downloader_extra_args=""
-distro="centos"
+make_stx_mirror_yum_conf_extra_args=""
+
 
 # lst files to use as input
-rpms_from_3rd_parties="./rpms_3rdparties.lst"
+rpms_from_3rd_parties_template="rpms_3rdparties.lst"
-rpms_from_centos_repo="./rpms_centos.lst"
+rpms_from_centos_repo_template="rpms_centos.lst"
-rpms_from_centos_3rd_parties="./rpms_centos3rdparties.lst"
+rpms_from_centos_3rd_parties_template="rpms_centos3rdparties.lst"
-other_downloads="./other_downloads.lst"
+rpms_from_layer_build_dir=${DL_MIRROR_OUTPUT_DIR}/layer_pkg_lists
+image_inc_from_layer_build_dir=${DL_MIRROR_OUTPUT_DIR}/layer_image_inc
+build_info_from_layer_build_dir=${DL_MIRROR_OUTPUT_DIR}/layer_build_info
+tarball_downloads_template="tarball-dl.lst"
+other_downloads_template="other_downloads.lst"
 
 # Overall success
 success=1
@@ -103,22 +145,41 @@ multiple_dl_flag_check () {
     fi
 }
 
+
 # Parse out optional arguments
-while getopts "c:nghsSuU" o; do
+while getopts "c:Cd:ghI:sl:L:nSuU" o; do
     case "${o}" in
-        n)
-            # Pass -n ("no-sudo") to rpm downloader
-            rpm_downloader_extra_args="${rpm_downloader_extra_args} -n"
-            ;;
         c)
             # Pass -c ("use alternate yum.conf") to rpm downloader
             use_system_yum_conf=0
             alternate_yum_conf="${OPTARG}"
             ;;
+        C)
+            # Alternate config directory
+            set_and_validate_config_dir "${OPTARG}"
+            ;;
+        d)
+            # Alternate distro
+            set_and_validate_distro "${OPTARG}"
+            ;;
         g)
             # Do not attempt to change group IDs on downloaded packages
             change_group_ids=0
             ;;
+        I)
+            set_layer_image_inc_urls "${OPTARG}"
+            ;;
+        l)
+            # layer
+            set_and_validate_layer "${OPTARG}"
+            ;;
+        L)
+            set_layer_pkg_urls "${OPTARG}"
+            ;;
+        n)
+            # Pass -n ("no-sudo") to rpm downloader
+            rpm_downloader_extra_args="${rpm_downloader_extra_args} -n"
+            ;;
         s)
             # Download from StarlingX mirror only. Do not use upstream sources.
             multiple_dl_flag_check
@@ -156,6 +217,21 @@ while getopts "c:nghsSuU" o; do
 done
 shift $((OPTIND-1))
 
+
+TMP_LST_DIR=$(mktemp -d /tmp/tmp_lst_dir_XXXXXX)
+mkdir -p $TMP_LST_DIR
+rpms_from_3rd_parties="$TMP_LST_DIR/${rpms_from_3rd_parties_template}"
+rpms_from_centos_repo="$TMP_LST_DIR/${rpms_from_centos_repo_template}"
+rpms_from_centos_3rd_parties="$TMP_LST_DIR/${rpms_from_centos_3rd_parties_template}"
+tarball_downloads="$TMP_LST_DIR/${tarball_downloads_template}"
+other_downloads="$TMP_LST_DIR/${other_downloads_template}"
+
+merge_lst ${config_dir} ${distro} ${rpms_from_3rd_parties_template} > ${rpms_from_3rd_parties}
+merge_lst ${config_dir} ${distro} ${rpms_from_centos_repo_template} > ${rpms_from_centos_repo}
+merge_lst ${config_dir} ${distro} ${rpms_from_centos_3rd_parties_template} > ${rpms_from_centos_3rd_parties}
+merge_lst ${config_dir} ${distro} ${tarball_downloads_template} > ${tarball_downloads}
+merge_lst ${config_dir} ${distro} ${other_downloads_template} > ${other_downloads}
+
 echo "--------------------------------------------------------------"
 
 echo "WARNING: this script HAS TO access internet (http/https/ftp),"
@@ -168,14 +244,92 @@ mkdir -p $LOGSDIR
 
 # Check extistence of prerequisites files
 need_file ${rpm_downloader} ${other_downloader} ${tarball_downloader}
-need_file ${rpms_from_3rd_parties}
+make_if_needed_file ${rpms_from_3rd_parties}
-need_file ${rpms_from_centos_3rd_parties}
+make_if_needed_file ${rpms_from_centos_3rd_parties}
-need_file ${rpms_from_centos_repo}
+make_if_needed_file ${rpms_from_centos_repo}
-need_file ${other_downloads}
+make_if_needed_file ${other_downloads}
-need_file tarball-dl.lst
+make_if_needed_file ${tarball_downloads}
 
-#download RPMs/SRPMs from 3rd_party websites (not CentOS repos) by "wget"
+#
-echo "step #1: start downloading RPMs/SRPMs from 3rd-party websites..."
+# Dowlnoad package lst files, image inc files and build info files for lower layers.
+#
+# Also it may set up extra arguements for make_stx_mirror_yum_conf that
+# will exploy yum repos co-resident with the lst file.
+#
+\rm -rf ${rpms_from_layer_build_dir}
+mkdir -p ${rpms_from_layer_build_dir}
+
+for key in "${!layer_pkg_urls[@]}"; do
+    lower_layer="${key%,*}"
+    build_type="${key#*,}"
+    url="${layer_pkg_urls[${key}]}"
+    name_from_url=$(url_to_file_name $url)
+    list="${rpms_from_layer_build_dir}/${name_from_url}"
+    curl --silent --fail ${url} > ${list} ||
+    if [ $? -ne 0 ]; then
+        echo "ERROR: Failed to download from url: ${url}"
+        exit 1
+    fi
+
+    #
+    # If the lst file is co-resident with a yum repodata directory,
+    # then add arguements for our call to make_stx_mirror_yum_conf
+    # so that we'll use that repo.
+    #
+    url_type=${url%%:*}
+    if [ "${url_type}" == "file" ]; then
+        base_url=$(dirname $url)
+        repomod_url=${base_url}/repodata/repomd.xml
+        curl --silent --fail --output /dev/null ${repomod_url} ||
+        if [ $? -ne 0 ]; then
+            echo "ERROR: Failed to download from url: ${url}"
+            exit 1
+        fi
+        make_stx_mirror_yum_conf_extra_args+=" -u ${lower_layer},${build_type},${base_url}"
+    fi
+done
+
+\rm -rf ${image_inc_from_layer_build_dir}
+mkdir -p ${image_inc_from_layer_build_dir}
+
+for key in "${!layer_image_inc_urls[@]}"; do
+    lower_layer="${key%,*}"
+    inc_type="${key#*,}"
+    url="${layer_image_inc_urls[${key}]}"
+    name_from_url=$(url_to_file_name $url)
+    list="${image_inc_from_layer_build_dir}/${name_from_url}"
+    curl --silent --fail ${url} > ${list} ||
+    if [ $? -ne 0 ]; then
+        echo "ERROR: Failed to download from url: ${url}"
+        exit 1
+    fi
+done
+
+\rm -rf ${build_info_from_layer_build_dir}
+mkdir -p ${build_info_from_layer_build_dir}
+
+# Borrow std image.inc url as a proxy for the BUILD_INFO with a simple substitution
+for key in "${!layer_image_inc_urls[@]}"; do
+    lower_layer="${key%,*}"
+    inc_type="${key#*,}"
+    if [ "${inc_type}" != "std" ]; then
+        continue
+    fi
+    if [ "$(basename ${layer_image_inc_urls[${key}]})" != "image.inc" ]; then
+        continue
+    fi
+    url=$( echo ${layer_image_inc_urls[${key}]} | sed 's#image.inc$#BUILD_INFO#' )
+    name_from_url=$(url_to_file_name $url)
+    dest="${build_info_from_layer_build_dir}/${name_from_url}"
+    curl --silent --fail ${url} > ${dest} ||
+    if [ $? -ne 0 ]; then
+        echo "ERROR: Failed to download from url: ${url}"
+        exit 1
+    fi
+done
+
+
+echo "step #0: Configuring yum repos ..."
 
 if [ ${use_system_yum_conf} -ne 0 ]; then
     # Restore StarlingX_3rd repos from backup
@@ -184,6 +338,12 @@ if [ ${use_system_yum_conf} -ne 0 ]; then
     if [ -d $REPO_SOURCE_DIR ] && [ -d $REPO_DIR ]; then
         \cp -f $REPO_SOURCE_DIR/*.repo $REPO_DIR/
     fi
+
+    if [ $layer != "all" ]; then
+        if [ -d ${config_dir}/${distro}/${layer}/yum.repos.d ]; then
+            \cp -f ${config_dir}/${distro}/${layer}/yum.repos.d/*.repo $REPO_DIR
+        fi
+    fi
 fi
 
 if [ $use_system_yum_conf -eq 0 ]; then
@@ -193,6 +353,10 @@ if [ $use_system_yum_conf -eq 0 ]; then
         if [ "$alternate_repo_dir" == "" ]; then
             alternate_repo_dir="$(dirname "${alternate_yum_conf}"/yum.repos.d)"
         fi
+        if [[ $alternate_repo_dir != /* ]]; then
+            # Path is relative, so prefix with directory where yum.conf lives
+            alternate_repo_dir=$(dirname ${alternate_yum_conf})/${alternate_repo_dir}
+        fi
         need_dir "${alternate_repo_dir}"
     fi
 fi
@@ -224,10 +388,10 @@ else
         # point to the repos that need to be modified as well.
         if dl_from_upstream; then
             # add
-            ${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro
+            ${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
         else
             # substitute
-            ${make_stx_mirror_yum_conf} -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro
+            ${make_stx_mirror_yum_conf} -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
         fi
     else
         # Modify system yum.conf and yum.repos.d.  Remember that we expect to run this
@@ -235,16 +399,58 @@ else
         # in these scripts.
         if dl_from_upstream; then
             # add
-            ${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro
+            ${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
         else
             # substitute
-            ${make_stx_mirror_yum_conf} -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro
+            ${make_stx_mirror_yum_conf} -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
         fi
     fi
 
     rpm_downloader_extra_args="${rpm_downloader_extra_args} -c $TEMP_CONF"
 fi
 
+#download RPMs/SRPMs from lower layer builds
+echo "step #1: start downloading RPMs/SRPMs from lower layer builds..."
+retcode=0
+for key in "${!layer_pkg_urls[@]}"; do
+    lower_layer="${key%,*}"
+    build_type="${key#*,}"
+    url="${layer_pkg_urls[${key}]}"
+    name_from_url=$(url_to_file_name $url)
+    list="${rpms_from_layer_build_dir}/${name_from_url}"
+
+    url_type=${url%%:*}
+    if [ "${url_type}" == "file" ]; then
+        level=L1
+        logfile=$(generate_log_name $list level)
+        $rpm_from_url_downloader -u $(dirname $url) $list |& tee $logfile
+        local_retcode=${PIPESTATUS[0]}
+    else
+        #download RPMs/SRPMs from CentOS repos by "yumdownloader"
+        level=L1
+        logfile=$(generate_log_name $list $level)
+        $rpm_downloader ${rpm_downloader_extra_args} $list $level |& tee $logfile
+        local_retcode=${PIPESTATUS[0]}
+    fi
+
+    if [ $local_retcode -ne 0 ]; then
+        echo "ERROR: Something wrong with downloading files listed in $list."
+        echo "   Please check the log at $(pwd)/$logfile !"
+        echo ""
+        success=0
+        retcode=$local_retcode
+    fi
+done
+
+if [ $retcode -eq 0 ];then
+    echo "step #1: done successfully"
+else
+    echo "step #1: finished with errors"
+fi
+
+
+#download RPMs/SRPMs from 3rd_party websites (not CentOS repos) by "wget"
+echo "step #2: start downloading RPMs/SRPMs from 3rd-party websites..."
 list=${rpms_from_3rd_parties}
 level=L1
 logfile=$(generate_log_name $list $level)
@@ -263,23 +469,28 @@ level=L1
 logfile=$(generate_log_name $list $level)
 $rpm_downloader ${rpm_downloader_extra_args} $list $level |& tee $logfile
 retcode=${PIPESTATUS[0]}
-if [ $retcode -ne 0 ];then
+if [ $retcode -eq 0 ];then
+    echo "step #2: done successfully"
+else
+    echo "step #2: finished with errors"
     echo "ERROR: Something wrong with downloading files listed in $list."
     echo "   Please check the log at $(pwd)/$logfile !"
     echo ""
+    echo "step #2: finished with errors"
     success=0
 fi
 
 if [ ${use_system_yum_conf} -eq 1 ]; then
     # deleting the StarlingX_3rd to avoid pull centos packages from the 3rd Repo.
     \rm -f $REPO_DIR/StarlingX_3rd*.repo
+    \rm -f $REPO_DIR/StarlingX_cengn*.repo
     if [ "$TEMP_DIR" != "" ]; then
         \rm -f $TEMP_DIR/yum.repos.d/StarlingX_3rd*.repo
+        \rm -f $TEMP_DIR/yum.repos.d/StarlingX_cengn*.repo
     fi
 fi
 
-
+echo "step #3: start 1st round of downloading RPMs and SRPMs with L1 match criteria..."
-echo "step #2: start 1st round of downloading RPMs and SRPMs with L1 match criteria..."
 #download RPMs/SRPMs from CentOS repos by "yumdownloader"
 list=${rpms_from_centos_repo}
 level=L1
@@ -314,7 +525,7 @@ elif [ $retcode -eq 1 ]; then
 
         # Remove files found by K1 download from L1_rpms_missing_centos.txt to prevent
         # false reporting of missing files.
-        grep -v -x -F -f $LOGSDIR/K1_rpms_found_centos.log $LOGSDIR/L1_rpms_missing_centos.log  > $LOGSDIR/L1_rpms_missing_centos.tmp
+        grep -v -x -F -f $LOGSDIR/K1_rpms_found_centos.log $LOGSDIR/L1_rpms_missing_centos.log  > $LOGSDIR/L1_rpms_missing_centos.tmp || true
         mv -f $LOGSDIR/L1_rpms_missing_centos.tmp $LOGSDIR/L1_rpms_missing_centos.log
 
 
@@ -334,27 +545,34 @@ elif [ $retcode -eq 1 ]; then
     fi
 fi
 
-if [ $retcode -ne 0 ]; then
+if [ $retcode -eq 0 ];then
+    echo "step #3: done successfully"
+else
     echo "ERROR: Something wrong with downloading files listed in ${rpms_from_centos_repo}."
     echo "   Please check the logs at $(pwd)/$logfile"
     echo "   and $(pwd)/logs/$K1_logfile !"
     echo ""
+    echo "step #3: finished with errors"
     success=0
 fi
 
 ## verify all RPMs SRPMs we download for the GPG keys
-find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > $LOGSDIR/rpm-gpg-key-missing.txt
+find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > $LOGSDIR/rpm-gpg-key-missing.txt || true
 
 # remove all i686.rpms to avoid pollute the chroot dep chain
 find ./output -name "*.i686.rpm" | tee $LOGSDIR/all_i686.txt
 find ./output -name "*.i686.rpm" | xargs rm -f
 
-line1=`wc -l ${rpms_from_3rd_parties} | cut -d " " -f1-1`
+# Count unique rpms.  Strip extra fields from 'rpms_from_3rd_partiesIgnore',
-line2=`wc -l ${rpms_from_centos_repo} | cut -d " " -f1-1`
+# commented out entries, and blank lines.
-line3=`wc -l ${rpms_from_centos_3rd_parties} | cut -d " " -f1-1`
+total_line=$(sed 's/#.*//'  ${rpms_from_3rd_parties} \
-let total_line=$line1+$line2+$line3
+                            ${rpms_from_centos_repo} \
+                            ${rpms_from_centos_3rd_parties} \
+                | grep -v '^$' \
+                | sort --unique \
+                | wc -l)
 echo "We expected to download $total_line RPMs."
-num_of_downloaded_rpms=`find ./output -type f -name "*.rpm" | wc -l | cut -d" " -f1-1`
+num_of_downloaded_rpms=$(find ./output -type f -name "*.rpm" | wc -l | cut -d" " -f1-1)
 echo "There are $num_of_downloaded_rpms RPMs in output directory."
 if [ "$total_line" != "$num_of_downloaded_rpms" ]; then
     echo "WARNING: Not the same number of RPMs in output as RPMs expected to be downloaded, need to check outputs and logs."
@@ -365,16 +583,15 @@ if [ $change_group_ids -eq 1 ]; then
     chown  751:751 -R ./output
 fi
 
-
+echo "step #4: start downloading other files ..."
-echo "step #3: start downloading other files ..."
 
 logfile=$LOGSDIR"/otherfiles_centos_download.log"
-${other_downloader} ${dl_flag} -D "$distro" ${other_downloads} ./output/stx-r1/CentOS/pike/Binary/ |& tee $logfile
+${other_downloader} ${dl_flag} -D "$distro" ${other_downloads} ${DL_MIRROR_OUTPUT_DIR}/Binary/ |& tee $logfile
 retcode=${PIPESTATUS[0]}
 if [ $retcode -eq 0 ];then
-    echo "step #3: done successfully"
+    echo "step #4: done successfully"
 else
-    echo "step #3: finished with errors"
+    echo "step #4: finished with errors"
     echo "ERROR: Something wrong with downloading from ${other_downloads}."
     echo "   Please check the log at $(pwd)/$logfile!"
     echo ""
@@ -384,33 +601,35 @@ fi
 
 # StarlingX requires a group of source code pakages, in this section
 # they will be downloaded.
-echo "step #4: start downloading tarball compressed files"
+echo "step #5: start downloading tarball compressed files"
 logfile=$LOGSDIR"/tarballs_download.log"
-${tarball_downloader} ${dl_flag} -D "$distro" ${tarball_downloader_extra_args}  |& tee $logfile
+${tarball_downloader} ${dl_flag} -D "$distro" ${tarball_downloader_extra_args} ${tarball_downloads} |& tee $logfile
 retcode=${PIPESTATUS[0]}
 if [ $retcode -eq 0 ];then
-    echo "step #4: done successfully"
+    echo "step #5: done successfully"
 else
-    echo "step #4: finished with errors"
+    echo "step #5: finished with errors"
     echo "ERROR: Something wrong with downloading tarballs."
     echo "   Please check the log at $(pwd)/$logfile !"
     echo ""
     success=0
 fi
 
+
 #
 # Clean up the mktemp directory, if required.
 #
 if [ "$TEMP_DIR" != "" ]; then
+    echo "rm -rf $TEMP_DIR"
     \rm -rf "$TEMP_DIR"
 fi
 
 echo "IMPORTANT: The following 3 files are just bootstrap versions. Based"
 echo "on them, the workable images for StarlingX could be generated by"
 echo "running \"update-pxe-network-installer\" command after \"build-iso\""
-echo "    - out/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img"
+echo "    - ${DL_MIRROR_OUTPUT_DIR}/Binary/LiveOS/squashfs.img"
-echo "    - out/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img"
+echo "    - ${DL_MIRROR_OUTPUT_DIR}/Binary/images/pxeboot/initrd.img"
-echo "    - out/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz"
+echo "    - ${DL_MIRROR_OUTPUT_DIR}/Binary/images/pxeboot/vmlinuz"
 
 echo ""
 if [ $success -ne 1 ]; then

centos-mirror-tools/make_stx_mirror_yum_conf.sh

@@ -24,7 +24,7 @@ RETAIN_REPODIR=0
 
 usage () {
     echo ""
-    echo "$0 -d <dest_dir> [-D <distro>] [-y <src_yum_conf>] [-r <src_repos_dir>] [-R]"
+    echo "$0 -d <dest_dir> [-D <distro>] [-y <src_yum_conf>] [-r <src_repos_dir>] [-R] [-l <layer>] [-u <lower-layer>,<repo_url>]"
     echo ""
     echo "Replicate a yum.conf and yum.repo.d under a new directory and"
     echo "then modify the files to point to equivalent repos in the StarlingX"
@@ -36,13 +36,32 @@ usage () {
     echo "                'yum.conf.sample' in same directory as this script"
     echo "-r <repos_dir> = Path to yum.repos.d that we will modify.  Default is"
     echo "                 'yum.repos.d' in same directory as this script"
+    echo "-l <layer> = Download only packages required to build a given layer"
+    echo "-u <lower-layer>,<build-type>,<repo_url> = Add/change the repo baseurl for a lower layer"
+}
+
+declare -A layer_urls
+
+set_layer_urls () {
+    local option="${1}"
+    local layer_and_build_type="${option%,*}"
+    local layer="${layer_and_build_type%,*}"
+    local build_type="${layer_and_build_type#*,}"
+    local layer_url="${option##*,}"
+
+    # Enforce trailing '/'
+    if [ "${layer_url:${#layer_url}-1:1}" != "/" ]; then
+        layer_url+="/"
+    fi
+
+    layer_urls["${layer_and_build_type}"]="${layer_url}"
 }
 
 
 #
 # option processing
 #
-while getopts "D:d:Rr:y:" o; do
+while getopts "D:d:l:Rr:u:y:" o; do
     case "${o}" in
         D)
             DISTRO="${OPTARG}"
@@ -50,12 +69,18 @@ while getopts "D:d:Rr:y:" o; do
         d)
             TEMP_DIR="${OPTARG}"
             ;;
+        l)
+            LAYER="${OPTARG}"
+            ;;
         r)
             SRC_REPO_DIR="${OPTARG}"
             ;;
         R)
             RETAIN_REPODIR=1
             ;;
+        u)
+            set_layer_urls "${OPTARG}"
+            ;;
         y)
             SRC_YUM_CONF="${OPTARG}"
             ;;
@@ -136,11 +161,18 @@ ARCH=$(get_arch)
 #
 # Copy as yet unmodified yum.conf and yum.repos.d from source to dest.
 #
-echo "\cp -r '$SRC_REPO_DIR' '$CENGN_REPOS_DIR'"
+mkdir -p "$CENGN_REPOS_DIR"
-\cp -r "$SRC_REPO_DIR" "$CENGN_REPOS_DIR"
+echo "\cp -r '$SRC_REPO_DIR/*' '$CENGN_REPOS_DIR/'"
+\cp -r "$SRC_REPO_DIR"/* "$CENGN_REPOS_DIR/"
 echo "\cp '$SRC_YUM_CONF' '$CENGN_YUM_CONF'"
 \cp "$SRC_YUM_CONF" "$CENGN_YUM_CONF"
 
+if [ "$LAYER" != "all" ]; then
+    if [ -d ${MAKE_STX_MIRROR_YUM_CONF_DIR}/config/${DISTRO}/${LAYER}/yum.repos.d ]; then
+        \cp -f ${MAKE_STX_MIRROR_YUM_CONF_DIR}/config/${DISTRO}/${LAYER}/yum.repos.d/*.repo $CENGN_REPOS_DIR
+    fi
+fi
+
 #
 # Add or modify reposdir= value in our new yum.conf
 #
@@ -229,4 +261,21 @@ for REPO in $(find "$CENGN_REPOS_DIR" -type f -name '*repo'); do
     sed "s#^\[\([^]]*\)\]#[CENGN_\1]#" -i "$REPO"
 done
 
+for key in "${!layer_urls[@]}"; do
+    lower_layer="${key%,*}"
+    build_type="${key#*,}"
+    REPO="$CENGN_REPOS_DIR/StarlingX_cengn_${lower_layer}_layer.repo"
+    if [ -f "$REPO" ]; then
+        sed "s#^baseurl=.*/${lower_layer}/.*/${build_type}/\$#baseurl=${layer_urls[${key}]}#" -i "$REPO"
+    else
+        REPO="$CENGN_REPOS_DIR/StarlingX_local_${lower_layer}_${build_type}_layer.repo"
+        (
+        echo "[Starlingx-local_${lower_layer}_${build_type}_layer]"
+        echo "name=Starlingx-cengn_${lower_layer}_${build_type}_layer"
+        echo "baseurl=${layer_urls[${key}]}"
+        echo "enabled=1"
+        ) > "$REPO"
+    fi
+done
+
 echo $TEMP_DIR

centos-mirror-tools/tarball-dl.lst

@@ -1,82 +0,0 @@
-blkin-f24ceec055ea236a093988237a9821d145f5f7c8.tar.gz#blkin#https://api.github.com/repos/ceph/blkin/tarball/f24ceec055ea236a093988237a9821d145f5f7c8#https##
-boost_1_67_0.tar.bz2#boost_1_67_0#https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.bz2#https##
-ceph-erasure-code-corpus-2d7d78b9cc52e8a9529d8cc2d2954c7d375d5dd7.tar.gz#ceph-erasure-code-corpus#https://api.github.com/repos/ceph/ceph-erasure-code-corpus/tarball/2d7d78b9cc52e8a9529d8cc2d2954c7d375d5dd7#https##
-ceph-object-corpus-e32bf8ca3dc6151ebe7f205ba187815bc18e1cef.tar.gz#ceph-object-corpus#https://api.github.com/repos/ceph/ceph-object-corpus/tarball/e32bf8ca3dc6151ebe7f205ba187815bc18e1cef#https##
-civetweb-ff2881e2cd5869a71ca91083bad5d12cccd22136.tar.gz#civetweb#https://api.github.com/repos/ceph/civetweb/tarball/ff2881e2cd5869a71ca91083bad5d12cccd22136#https##
-containerd-v1.3.0.tar.gz#containerd#https://github.com/containerd/containerd/archive/v1.3.0.tar.gz#https##
-crictl-v1.16.0-linux-amd64.tar.gz#crictl#https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.16.0/crictl-v1.16.0-linux-amd64.tar.gz#https##
-dep-v0.5.0.tar.gz#dep-v0.5.0#https://github.com/golang/dep/archive/v0.5.0.tar.gz#http##
-docker-distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89.tar.gz#docker-distribution#github.com/docker/distribution/archive/48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89.tar.gz#http##
-docker-libtrust-fa567046d9b14f6aa788882a950d69651d230b21.tar.gz#docker-libtrust#https://github.com/docker/libtrust/archive/fa567046d9b14f6aa788882a950d69651d230b21.tar.gz#http##
-dpdk-18.11.tar.xz#dpdk-18.11#http://fast.dpdk.org/rel/dpdk-18.11.tar.xz#http##
-dpdk-6ece49ad5a26f5e2f5c4af6c06c30376c0ddc387.tar.gz#dpdk#https://api.github.com/repos/spdk/dpdk/tarball/6ece49ad5a26f5e2f5c4af6c06c30376c0ddc387#https##
-dpkg_1.18.24.tar.xz#dpkg-1.18.24#http://http.debian.net/debian/pool/main/d/dpkg/dpkg_1.18.24.tar.xz#http##
-drbd-8.4.11-1.tar.gz#drbd-8.4.11-1#http://www.linbit.com/downloads/drbd/8.4/drbd-8.4.11-1.tar.gz#http##
-drbd-8.4.3.tar.gz#drbd-8.4.3#http://www.linbit.com/downloads/drbd/8.4/archive/drbd-8.4.3.tar.gz#http##
-dtc-1.4.4.tar.gz#dtc-1.4.4#https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.4.tar.gz#http##
-e1000e-3.4.2.1.tar.gz#e1000e-3.4.2.1#https://sourceforge.net/projects/e1000/files/e1000e%20stable/3.4.2.1/e1000e-3.4.2.1.tar.gz#http##
-etcd-v3.3.15.tar.gz#etcd-v3.3.15#https://github.com/etcd-io/etcd/archive/v3.3.15.tar.gz#http##
-gf-complete-7e61b44404f0ed410c83cfd3947a52e88ae044e1.tar.gz#gf-complete#https://api.github.com/repos/ceph/gf-complete/tarball/7e61b44404f0ed410c83cfd3947a52e88ae044e1#https##
-gnocchi-4.2.5.tar.gz#gnocchi-4.2.5#https://pypi.io/packages/source/g/gnocchi/gnocchi-4.2.5.tar.gz#http##
-gnocchiclient-7.0.1.tar.gz#gnocchiclient-7.0.1#https://pypi.io/packages/source/g/gnocchiclient/gnocchiclient-7.0.1.tar.gz#http##
-gnulib-ffc927e.tar.gz#gnulib-ffc927e#http://git.savannah.gnu.org/cgit/gnulib.git/snapshot/gnulib-ffc927eef29016a5219cd969daad8928af6a1f4d.tar.gz#http##
-go1.12.10.src.tar.gz#go#https://dl.google.com/go/go1.12.10.src.tar.gz##https##
-googletest-0a439623f75c029912728d80cb7f1b8b48739ca4.tar.gz#googletest#https://api.github.com/repos/google/googletest/tarball/0a439623f75c029912728d80cb7f1b8b48739ca4#https##
-googletest-fdb850479284e2aae047b87df6beae84236d0135.tar.gz#googletest#https://api.github.com/repos/ceph/googletest/tarball/fdb850479284e2aae047b87df6beae84236d0135#https##
-gophercloud-gophercloud-aa00757ee3ab58e53520b6cb910ca0543116400a.tar.gz#gophercloud-gophercloud#github.com/gophercloud/gophercloud/archive/aa00757ee3ab58e53520b6cb910ca0543116400a.tar.gz#http##
-gorilla-context-08b5f424b9271eedf6f9f0ce86cb9396ed337a42.tar.gz#gorilla-context#https://github.com/gorilla/context/archive/08b5f424b9271eedf6f9f0ce86cb9396ed337a42.tar.gz#http##
-gorilla-mux-456bcfa82d672db7cae587c9b541463f65bc2718.tar.gz#gorilla-mux#https://github.com/gorilla/mux/archive/456bcfa82d672db7cae587c9b541463f65bc2718.tar.gz#http##
-helm-charts-92b6289ae93816717a8453cfe62bad51cbdb8ad0.tar.gz#helm-charts#https://github.com/helm/charts/archive/92b6289ae93816717a8453cfe62bad51cbdb8ad0.tar.gz#http##
-helm-charts-elastic-2bd7616ceddbdf2eee88965e2028ee37d304c79c.tar.gz#helm-charts-elastic#https://github.com/elastic/helm-charts/archive/2bd7616ceddbdf2eee88965e2028ee37d304c79c.tar.gz#http##
-helm-v2.13.1-linux-amd64.tar.gz#linux-amd64#https://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz#http##
-i40e-2.10.19.30.tar.gz#i40e-2.10.19.30#https://sourceforge.net/projects/e1000/files/i40e%20stable/2.10.19.30/i40e-2.10.19.30.tar.gz/download#http##
-iavf-3.7.61.20.tar.gz#iavf-3.7.61.20#https://sourceforge.net/projects/e1000/files/iavf%20stable/3.7.61.20/iavf-3.7.61.20.tar.gz/download#http##
-!integrity-kmod-e6aef069.tar.gz#integrity#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz#http_script##post-dl-script/integrity-kmod.sh
-isa-l-7e1a337433a340bc0974ed0f04301bdaca374af6.tar.gz#isa-l#https://api.github.com/repos/ceph/isa-l/tarball/7e1a337433a340bc0974ed0f04301bdaca374af6#https##
-isa-l_crypto-603529a4e06ac8a1662c13d6b31f122e21830352.tar.gz#isa-l_crypto#https://api.github.com/repos/01org/isa-l_crypto/tarball/603529a4e06ac8a1662c13d6b31f122e21830352#https##
-ixgbe-5.5.5.tar.gz#ixgbe-5.5.5#https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.5.5/ixgbe-5.5.5.tar.gz/download#http##
-ixgbevf-4.5.3.tar.gz#ixgbevf-4.5.3#https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/4.5.3/ixgbevf-4.5.3.tar.gz/download#http##
-jerasure-96c76b89d661c163f65a014b8042c9354ccf7f31.tar.gz#jerasure#https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8042c9354ccf7f31#https##
-keycodemapdb-16e5b07.tar.gz#keycodemapdb#https://github.com/CendioOssman/keycodemapdb/tarball/16e5b0787687d8904dad2c026107409eb9bfcb95#http##
-kubernetes-contrib-v1.16.2.tar.gz#kubernetes-contrib-1.16.2#https://github.com/kubernetes-retired/contrib/tarball/89f6948e24578fed2a90a87871b2263729f90ac3#http##
-kubernetes-v1.16.2.tar.gz#kubernetes-1.16.2#https://github.com/kubernetes/kubernetes/archive/v1.16.2.tar.gz#http##
-kvm-unit-tests.git-4ea7633.tar.bz2#kvm-unit-tests#https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git/snapshot/kvm-unit-tests-20171020.tar.gz#http##
-ldapscripts-2.0.8.tgz#ldapscripts-2.0.8#https://sourceforge.net/projects/ldapscripts/files/ldapscripts/ldapscripts-2.0.8/ldapscripts-2.0.8.tgz/download#http##
-libtpms-0.6.0-4f0d59d.tar.gz#libtpms-0.6.0#https://github.com/stefanberger/libtpms/tarball/c421ca0f4d00c0caceeda8d62c1efb2b7e47db04#http##
-lldpd-0.9.0.tar.gz#lldpd-0.9.0#https://media.luffy.cx/files/lldpd/lldpd-0.9.0.tar.gz#http##
-lua-1fce39c6397056db645718b8f5821571d97869a4.tar.gz#lua#https://api.github.com/repos/ceph/lua/tarball/1fce39c6397056db645718b8f5821571d97869a4#https##
-!mariadb-10.1.28.tar.gz#mariadb-10.1.28#https://github.com/MariaDB/server/archive/mariadb-10.1.28.tar.gz#http_script##post-dl-script/mariadb.sh
-!MLNX_OFED_SRC-4.5-1.0.1.0.tgz#MLNX_OFED_SRC-4.5-1.0.1.0#http://content.mellanox.com/ofed/MLNX_OFED-4.5-1.0.1.0/MLNX_OFED_SRC-4.5-1.0.1.0.tgz#http_script##post-dl-script/MLNX_OFEDqat1.7.sh
-opae-intel-fpga-driver-1.3.0-2.tar.gz#opae-intel-fpga-driver-1.3.0-2#https://github.com/OPAE/opae-sdk/releases/download/1.3.0-2/opae-intel-fpga-driver-1.3.0-2.tar.gz#http##
-openstack-helm-82c72367c85ca94270f702661c7b984899c1ae38.tar.gz#openstack-helm#https://github.com/openstack/openstack-helm/archive/82c72367c85ca94270f702661c7b984899c1ae38.tar.gz#http##
-openstack-helm-infra-c9d6676bf9a5aceb311dc31dadd07cba6a3d6392.tar.gz#openstack-helm-infra#https://github.com/openstack/openstack-helm-infra/archive/c9d6676bf9a5aceb311dc31dadd07cba6a3d6392.tar.gz#http##
-openvswitch-2.11.0.tar.gz#openvswitch-2.11.0#https://github.com/openvswitch/ovs/archive/v2.11.0.tar.gz#http##
-puppet-boolean-22b726dd78b0a60a224cc7054aebbf28e9306f62.tar.gz#puppet-boolean#https://github.com/voxpupuli/puppet-boolean/tarball/22b726dd78b0a60a224cc7054aebbf28e9306f62#http##
-puppet-dnsmasq-cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packstack/puppet/modules/dnsmasq/#https://github.com/procore/puppet-dnsmasq/tarball/cff07e90890662972c97684a2baee964f68ff3ed#http##
-puppet-filemapper-9b53310278e76827bbe12a36cc6470d77071abb2.tar.gz#packstack/puppet/modules/filemapper#https://github.com/voxpupuli/puppet-filemapper/tarball/9b53310278e76827bbe12a36cc6470d77071abb2#http##
-puppetlabs-create_resources-4639819a7f3a4fa9310d2ba583c63e467df7e2c3.tar.gz#packstack/puppet/modules/create_resources#https://github.com/puppetlabs/puppetlabs-create_resources/tarball/4639819a7f3a4fa9310d2ba583c63e467df7e2c3#http##
-puppetlabs-drbd-496b3ba9cd74a2d12636f9e90a718739a5451169.tar.gz#puppetlabs-drbd#https://github.com/voxpupuli/puppet-drbd/tarball/496b3ba9cd74a2d12636f9e90a718739a5451169#http##
-puppetlabs-lvm-d0283da637ae24550fb4ba109a48ef8d5d8c8b84.tar.gz#packstack/puppet/modules/lvm#https://github.com/puppetlabs/puppetlabs-lvm/tarball/d0283da637ae24550fb4ba109a48ef8d5d8c8b84#http##
-puppetlabs-postgresql-d022a56b28b2174456fc0f6adc51a4b54493afad.tar.gz#puppetlabs-postgresql#https://github.com/puppetlabs/puppetlabs-postgresql/tarball/d022a56b28b2174456fc0f6adc51a4b54493afad#http##
-puppet-ldap-480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1.tar.gz#puppet-ldap-master#https://github.com/torian/puppet-ldap/tarball/480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1#http##
-puppet-network-7deacd5fdc22c0543455878a8d1872f2f5417c1d.tar.gz#packstack/puppet/modules/network#https://github.com/voxpupuli/puppet-network/tarball/7deacd5fdc22c0543455878a8d1872f2f5417c1d#http##
-puppet-nslcd-b8c19b1ada89865f2e50758e054583798ad8011a.tar.gz#packstack/puppet/modules/nslcd#https://github.com/jlyheden/puppet-nslcd/tarball/b8c19b1ada89865f2e50758e054583798ad8011a#http##
-puppi-c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81.tar.gz#puppi-master#https://github.com/example42/puppi/tarball/c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81#http##
-python-cephclient-v0.1.0.5.tar.gz#python-cephclient-0.1.0.5#https://github.com/dmsimard/python-cephclient/archive/v0.1.0.5.tar.gz#http##
-python-setuptools-v38.5.1.tar.gz#setuptools-38.5.1#https://github.com/pypa/setuptools/archive/v38.5.1.tar.gz#http##
-!qat1.7.l.4.5.0-00034.tar.gz#quickassist#https://01.org/sites/default/files/downloads/qat1.7.l.4.5.0-00034.tar.gz#http_script##post-dl-script/qat1.7.sh
-rapidjson-f54b0e47a08782a6131cc3d60f94d038fa6e0a51.tar.gz#rapidjson#https://api.github.com/repos/ceph/rapidjson/tarball/f54b0e47a08782a6131cc3d60f94d038fa6e0a51#https##
-Redfishtool-1.1.0.tar.gz#Redfishtool-1.1.0#https://github.com/DMTF/Redfishtool/archive/1.1.0.tar.gz#http##
-requests-toolbelt-0.9.1.tar.gz#requests-toolbelt-0.9.1#https://github.com/requests/toolbelt/archive/0.9.1.tar.gz#http##
-rocksdb-f4a857da0b720691effc524469f6db895ad00d8e.tar.gz#rocksdb#https://api.github.com/repos/ceph/rocksdb/tarball/f4a857da0b720691effc524469f6db895ad00d8e#https##
-rook-1.1.1.tar.gz#rook-1.1.1#https://github.com/rook/rook/archive/v1.1.1.tar.gz##
-rpm-4.14.0.tar.bz2#rpm-4.14.0#https://ftp.osuosl.org/pub/rpm/releases/rpm-4.14.x/rpm-4.14.0.tar.bz2#http##
-runc-1.0.0-rc8.tar.gz#runc#https://github.com/opencontainers/runc/archive/v1.0.0-rc8.tar.gz#https##
-Sirupsen-logrus-55eb11d21d2a31a3cc93838241d04800f52e823d.tar.gz#Sirupsen-logrus#github.com/Sirupsen/logrus/archive/55eb11d21d2a31a3cc93838241d04800f52e823d.tar.gz#http##
-spdk-f474ce6930f0a44360e1cc4ecd606d2348481c4c.tar.gz#spdk#https://api.github.com/repos/ceph/spdk/tarball/f474ce6930f0a44360e1cc4ecd606d2348481c4c#https##
-spectre-meltdown-checker-0.37+-5cc77741.tar.gz#spectre-meltdown-checker#https://github.com/speed47/spectre-meltdown-checker/tarball/5cc77741af1d2f52140aa9f89339f56b6c4b6783#http##
-swtpm-0.1.0-253eac5.tar.gz#swtpm-0.1.0#https://github.com/stefanberger/swtpm/tarball/1303be7d03294fb02204cb8242f366cbf0da076d#http##
-!tpm-kmod-e6aef069.tar.gz#tpm#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz#http_script#post-dl-script/tpm-kmod.sh
-!tss2-930.tar.gz#tss2-930#https://git.code.sf.net/p/ibmtpm20tss/tss#git#v930#
-xxHash-1f40c6511fa8dd9d2e337ca8c9bc18b3e87663c9.tar.gz#xxHash#https://api.github.com/repos/ceph/xxHash/tarball/1f40c6511fa8dd9d2e337ca8c9bc18b3e87663c9#https##
-zstd-f4340f46b2387bc8de7d5320c0b83bb1499933ad.tar.gz#zstd#https://api.github.com/repos/facebook/zstd/tarball/f4340f46b2387bc8de7d5320c0b83bb1499933ad#https##

centos-mirror-tools/utils.sh

@@ -20,7 +20,7 @@ get_yum_command() {
     if [ "$arch" == "src" ]; then
         yumdownloader_extra_opts="--source"
     else
-        yumdownloader_extra_opts="--archlist=noarch,x86_64"
+        yumdownloader_extra_opts="--exclude='*.i686' --archlist=noarch,x86_64"
     fi
 
     echo "yumdownloader -q -C ${YUMCONFOPT} ${RELEASEVER} $yumdownloader_extra_opts $rpm_name"

tb.sh

@@ -89,6 +89,7 @@ case $CMD in
         echo "HOST_MIRROR_DIR=${HOST_MIRROR_DIR}"
         echo "MY_TC_RELEASE=${MY_TC_RELEASE}"
         echo "MY_REPO_ROOT_DIR=${MY_REPO_ROOT_DIR}"
+        echo "LAYER=${LAYER}"
         ;;
     create)
         create_container

toCOPY/generate-cgcs-centos-repo.sh

@@ -5,14 +5,64 @@
 # Copyright (C) 2019 Intel Corporation
 #
 
+GENERATE_CGCS_CENTOS_REPO_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
+
+source $GENERATE_CGCS_CENTOS_REPO_DIR/lst_utils.sh
+
+mirror_dir=""
+layer_dirs=""
+
+CREATEREPO=$(which createrepo_c)
+if [ $? -ne 0 ]; then
+    CREATEREPO="createrepo"
+fi
+
 usage () {
-    echo "$0 <mirror-path>"
+    echo
+    echo "Create a virtual rpm repo containing only rpms listed in various lst files."
+    echo "The virtual repo contains only symlinks to to previously downloaded or built rpms."
+    echo
+    echo "Usage"
+    echo
+    echo "$0 [Options] [ --mirror-dir=<mirror-path> | <mirror-path> ]"
+    echo
+    echo "Commin Options:"
+    echo "  --distro=<distro>:  Create repo for the designated distro."
+    echo "                      Default 'centos'"
+    echo "  --layer=<layer>:    Create a smaller repo, sufficient to build"
+    echo "                      only the given layer."
+    echo "                      Default: use the LAYER environmnet valiable, or 'all'."
+    echo "  --mirror-dir=<dir>: Set the mirror directory.  This is where"
+    echo "                      the previously downloaded rpms are located."
+    echo
+    echo "Override options: For use when working on a multi-layer change"
+    echo "  --config-dir=<dir>: Use an alternate config directory rather than the"
+    echo "                      system defined one"
+    echo "  --layer-inc-url=<lower_layer>,<build_type>,<url>:"
+    echo "                      Override the url for the image include file of a lower"
+    echo "                      layer's build type.  Normally the url(s) is read from"
+    echo "                      <config_dir>/<distro>/<layer>/required_layer_iso_inc.cfg"
+    echo "                      This option can be used more than once."
+    echo "  --layer-pkg-url=<lower_layer>,<build_type>,<url>:"
+    echo "                      Override the url for the package list of a lower"
+    echo "                      layer's build type.  Normally the url(s) is read from"
+    echo "                      <config_dir>/<distro>/<layer>/required_layer_pkgs.cfg."
+    echo "                      This option can be used more than once."
+    echo "  --layer-dir=<dir>:  Look in provided dir for packages to link to."
+    echo "                      This option can be used more than once."
+    echo
 }
 
-if [ $# -ne 1 ]; then
+cleanup () {
-    usage
+    if [ -e "${mirror_content}" ]; then
-    exit -1
+        \rm -f ${mirror_content}
-fi
+    fi
+    if [ -e "${TMP_LST_DIR}" ]; then
+        \rm -rf ${TMP_LST_DIR}
+    fi
+}
+
+trap "cleanup ; exit 1" INT
 
 if [ -z "$MY_REPO" ]; then
     echo "\$MY_REPO is not set. Ensure you are running this script"
@@ -21,7 +71,50 @@ if [ -z "$MY_REPO" ]; then
     exit -1
 fi
 
-mirror_dir=$1
+
+TEMP=$(getopt -o h --long help,config-dir:,distro:,layer:,layer-dir:,layer-inc-url:,layer-pkg-url:,mirror-dir: -n 'generate-cgcs-centos-repo' -- "$@")
+if [ $? -ne 0 ]; then
+    echo "getopt error"
+    usage
+    exit 1
+fi
+eval set -- "$TEMP"
+
+
+while true ; do
+    case "$1" in
+        --mirror-dir)     mirror_dir=$2 ; shift 2 ;;
+        --layer-dir)      layer_dirs+=" ${2/,/ }" ; shift 2 ;;
+        --layer-inc-url)  set_layer_image_inc_urls "${2}" ; shift 2 ;;
+        --layer-pkg-url)  set_layer_pkg_urls "${2}" ; shift 2 ;;
+        --config-dir)     set_and_validate_config_dir "${2}"; shift 2 ;;
+        --distro)         set_and_validate_distro "${2}"; shift 2 ;;
+        --layer)          set_and_validate_layer "${2}"; shift 2 ;;
+        -h|--help)        echo "help"; usage; exit 0 ;;
+        --)               shift ; break ;;
+        *)                usage; exit 1 ;;
+    esac
+done
+
+if [ "$mirror_dir" == "" ]; then
+    if [ $# -ne 1 ]; then
+        usage
+        exit -1
+    fi
+
+    mirror_dir=$1
+fi
+
+echo "mirror_dir=${mirror_dir}"
+echo "config_dir=${config_dir}"
+echo "distro=${distro}"
+echo "layer=${layer}"
+echo
+echo "layer_pkg_urls=${layer_pkg_urls[@]}"
+echo
+echo "layer_image_inc_urls=${layer_image_inc_urls[@]}"
+echo
+
 dest_dir=$MY_REPO/cgcs-centos-repo
 timestamp="$(date +%F_%H%M)"
 mock_cfg_file=$MY_REPO/build-tools/repo_files/mock.cfg.proto
@@ -29,12 +122,24 @@ comps_xml_file=$MY_REPO/build-tools/repo_files/comps.xml
 mock_cfg_dest_file=$MY_REPO/cgcs-centos-repo/mock.cfg.proto
 comps_xml_dest_file=$MY_REPO/cgcs-centos-repo/Binary/comps.xml
 
-lst_file_dir="$MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools"
+TMP_LST_DIR=$(mktemp -d /tmp/tmp_lst_dir_XXXXXX)
+mkdir -p $TMP_LST_DIR
+lst_file_dir="$TMP_LST_DIR"
+inc_file_dir="${dest_dir}/layer_image_inc"
+build_info_file_dir="${dest_dir}/layer_build_info"
+
 rpm_lst_files="rpms_3rdparties.lst rpms_centos3rdparties.lst rpms_centos.lst"
+rpm_lst_files_rt=""
 other_lst_file="other_downloads.lst"
+
+for template in $rpm_lst_files $other_lst_file; do
+    lst="$lst_file_dir/${template}"
+    merge_lst ${config_dir} ${distro} ${template} > ${lst}
+done
+
 missing_rpms_file=missing.txt
 
-rm -f ${missing_rpms_file}
+\rm -f ${missing_rpms_file}
 
 # Strip trailing / from mirror_dir if it was specified...
 mirror_dir=$(echo ${mirror_dir} | sed "s%/$%%")
@@ -45,6 +150,13 @@ if [[ ( ! -d ${mirror_dir}/Binary ) || ( ! -d ${mirror_dir}/Source ) ]]; then
     exit -1
 fi
 
+for layer_dir in ${layer_dirs}; do
+    if [ ! -d ${layer_dir} ]; then
+        echo "The layer-dir ${layer_dir} doesn't exist"
+        exit -1
+    fi
+done
+
 if [ ! -d "${dest_dir}" ]; then
     mkdir -p "${dest_dir}"
 fi
@@ -59,15 +171,149 @@ for t in "Binary" "Source" ; do
     fi
 done
 
-mirror_content=$(mktemp -t centos-repo-XXXXXX)
+#
-find -L ${mirror_dir} -type f > ${mirror_content}
+# Dowload image inc files from layer_image_inc_urls
+#
+\rm -rf ${inc_file_dir}
+mkdir -p ${inc_file_dir}
+for key in "${!layer_image_inc_urls[@]}"; do
+    lower_layer="${key%,*}"
+    inc_type="${key#*,}"
+    url="${layer_image_inc_urls[${key}]}"
+    name_from_url=$(url_to_file_name "${url}")
+
+    if [ "${inc_type}" == "std" ]; then
+        ideal_name="${lower_layer}_${image_inc_from_layer_build_template}"
+    elif [ "${inc_type}" == "dev" ]; then
+        ideal_name="${lower_layer}_${dev_image_inc_from_layer_build_template}"
+    else
+        ideal_name="${lower_layer}_${inc_type}_${image_inc_from_layer_build_template}"
+    fi
+
+    list="${ideal_name}"
+
+    for f in $(find -L ${layer_dirs} ${mirror_dir} -type f -name "${name_from_url}"); do
+        cp $f ${inc_file_dir}/${list}
+        if [ $? -ne 0 ]; then
+            echo "WARNING: Failed to copy from cached file '$f' to satisfy url '${url}'"
+        fi
+    done
+
+    if [ ! -f ${inc_file_dir}/${list} ]; then
+        curl --silent --fail ${url} > ${inc_file_dir}/${list}
+        if [ $? -ne 0 ]; then
+            echo "ERROR: Failed to download from url '${url}'"
+            exit 1
+        fi
+    fi
+done
+
+#
+# Dowload build info files
+#
+build_info_from_layer_build_template="BUILD_INFO"
+\rm -rf ${build_info_file_dir}
+mkdir -p ${build_info_file_dir}
+for key in "${!layer_image_inc_urls[@]}"; do
+    lower_layer="${key%,*}"
+    inc_type="${key#*,}"
+
+    if [ "${inc_type}" != "std" ]; then
+        continue
+    fi
+
+    if [ "$(basename ${layer_image_inc_urls[${key}]})" != "image.inc" ]; then
+        continue
+    fi
+
+    url=$( echo ${layer_image_inc_urls[${key}]} | sed 's#image.inc$#BUILD_INFO#' )
+    name_from_url=$(url_to_file_name "${url}")
+    ideal_name="${lower_layer}_${build_info_from_layer_build_template}"
+
+    list="${ideal_name}"
+
+    for f in $(find -L ${layer_dirs} ${mirror_dir} -type f -name "${name_from_url}"); do
+        cp $f ${build_info_file_dir}/${list}
+        if [ $? -ne 0 ]; then
+            echo "WARNING: Failed to copy from cached file '$f' to satisfy url '${url}'"
+        fi
+    done
+
+    if [ ! -f ${build_info_file_dir}/${list} ]; then
+        curl --silent --fail ${url} > ${build_info_file_dir}/${list}
+        if [ $? -ne 0 ]; then
+            echo "WARNING: Failed to download from url '${url}'"
+        fi
+    fi
+done
+
+
+#
+# Dowload lst files from layer_pkg_urls
+#
+for key in "${!layer_pkg_urls[@]}"; do
+    lower_layer="${key%,*}"
+    build_type="${key#*,}"
+    url="${layer_pkg_urls[${key}]}"
+    name_from_url=$(url_to_file_name "${url}")
+    ideal_name="${lower_layer}_${build_type}_${rpms_from_layer_build_template}"
+    list="${ideal_name}"
+
+    for f in $(find -L ${layer_dirs} ${mirror_dir} -type f -name "${name_from_url}"); do
+        cp $f ${lst_file_dir}/${list}
+        if [ $? -ne 0 ]; then
+            echo "WARNING: Failed to copy from cached file '$f' to satisfy url '${url}'"
+        fi
+    done
+
+    if [ ! -f ${lst_file_dir}/${list} ]; then
+        curl --silent --fail ${url} > ${lst_file_dir}/${list}
+        if [ $? -ne 0 ]; then
+            echo "ERROR: Failed to download from url '${url}'"
+            exit 1
+        fi
+    fi
+
+    if [ "${build_type}" == "rt" ]; then
+        rpm_lst_files_rt+=" ${list}"
+    else
+        rpm_lst_files+=" ${list}"
+    fi
+
+    url_type=${url%%:*}
+    if [ "${url_type}" == "file" ]; then
+        url_dir=$(dirname ${url#file://})
+        if [ ! -d ${url_dir} ]; then
+            echo "ERROR: No such directory '${url_dir}' derived from url: ${url}"
+            exit 1
+        fi
+        layer_dirs+=" ${url_dir}"
+    fi
+done
+
+echo "rpm_lst_files=${rpm_lst_files}"
+echo "rpm_lst_files_rt=${rpm_lst_files_rt}"
+echo "layer_dirs=${layer_dirs}"
+
+mirror_content=$(mktemp -t centos-repo-XXXXXX)
+find -L ${layer_dirs} ${mirror_dir} -type f -name '*.rpm' > ${mirror_content}
+
+sed_expression=""
+for d in ${mirror_dir} ${layer_dirs}; do
+    sed_expression+=" -e s%^${d}/%%"
+done
+
+
+process_lst_file () {
+    local lst_file="${1}"
+    local dest_dir="${2}"
 
-for lst_file in ${rpm_lst_files} ; do
     grep -v "^#" ${lst_file_dir}/${lst_file} | while IFS="#" read rpmname extrafields; do
         if [ -z "${rpmname}" ]; then
             continue
         fi
-        mirror_file=$(grep "/${rpmname}$" ${mirror_content})
+
+        mirror_file=$(grep "/${rpmname}$" ${mirror_content} | head -n 1)
         if [ -z "${mirror_file}" ]; then
             echo "Error -- could not find requested ${rpmname} in ${mirror_dir}"
             echo ${rpmname} >> ${missing_rpms_file}
@@ -75,9 +321,14 @@ for lst_file in ${rpm_lst_files} ; do
         fi
 
         # Great, we found the file!  Let's strip the mirror_dir prefix from it...
-        ff=$(echo ${mirror_file} | sed "s%^${mirror_dir}/%%")
+        ff=$(echo ${mirror_file} | sed ${sed_expression})
         f_name=$(basename "$ff")
-        sub_dir=$(dirname "$ff")
+        arch=$(echo ${f_name} | rev | cut -d '.' -f 2 | rev)
+        if [ "${arch}" == "src" ]; then
+            sub_dir="Source"
+        else
+            sub_dir="Binary/${arch}"
+        fi
 
         # Make sure we have a subdir (so we don't symlink the first file as
         # the subdir name)
@@ -85,36 +336,68 @@ for lst_file in ${rpm_lst_files} ; do
 
         # Link it!
         echo "Creating symlink for ${dest_dir}/${sub_dir}/${f_name}"
-        ln -sf "${mirror_dir}/$ff" "${dest_dir}/${sub_dir}"
+        ln -sf "${mirror_file}" "${dest_dir}/${sub_dir}/${f_name}"
         if [ $? -ne 0 ]; then
-            echo "Failed ${mirror_file}: ln -sf \"${mirror_dir}/$ff\" \"${dest_dir}/${sub_dir}\""
+            echo "Failed ${mirror_file}: ln -sf \"${mirror_file}\" \"${dest_dir}/${sub_dir}\""
         fi
     done
+}
+
+for lst_file in ${rpm_lst_files} ; do
+    process_lst_file "${lst_file}" "${dest_dir}" || exit 1
 done
 
-rm -f ${mirror_content}
+for lst_file in ${rpm_lst_files_rt} ; do
+    process_lst_file "${lst_file}" "${dest_dir}/rt" || exit 1
+done
 
-if [ ! -f "$mock_cfg_file" ]; then
+
-    echo "Cannot find mock.cfg.proto file!"
+echo "Copying comps.xml file."
-    exit 1
-fi
 
 if [ ! -f "$comps_xml_file" ]; then
     echo "Cannot find comps.xml file!"
     exit 1
 fi
 
-echo "Copying mock.cfg.proto and comps.xml files."
+if [ -f "$comps_xml_dest_file" ]; then
+    \cp -f "$comps_xml_dest_file" "$comps_xml_dest_file-backup-$timestamp"
+fi
+cp "$comps_xml_file" "$comps_xml_dest_file"
+
+
+echo "Createing yum repodata."
+
+for subdir in Source Binary; do
+    repo_dir="${dest_dir}/${subdir}"
+    mkdir -p "${repo_dir}"
+    if [ -f "${repo_dir}/comps.xml" ]; then
+        ${CREATEREPO} -g "${repo_dir}/comps.xml" -d "${repo_dir}"
+    else
+        ${CREATEREPO} -d "${repo_dir}"
+    fi
+
+    repo_dir="${dest_dir}/rt/${subdir}"
+    mkdir -p "${repo_dir}"
+    if [ -f "${repo_dir}/comps.xml" ]; then
+        ${CREATEREPO} -g "${repo_dir}/comps.xml" -d "${repo_dir}"
+    else
+        ${CREATEREPO} -d "${repo_dir}"
+    fi
+done
+
+
+echo "Copying mock.cfg.proto file."
+
+if [ ! -f "$mock_cfg_file" ]; then
+    echo "Cannot find mock.cfg.proto file!"
+    exit 1
+fi
 
 if [ -f "$mock_cfg_dest_file" ]; then
     \cp -f "$mock_cfg_dest_file" "$mock_cfg_dest_file-backup-$timestamp"
 fi
 cp "$mock_cfg_file" "$mock_cfg_dest_file"
 
-if [ -f "$comps_xml_dest_file" ]; then
-    \cp -f "$comps_xml_dest_file" "$comps_xml_dest_file-backup-$timestamp"
-fi
-cp "$comps_xml_file" "$comps_xml_dest_file"
 
 # Populate the contents from other list files
 cat ${lst_file_dir}/${other_lst_file} | grep -v "#" | while IFS=":" read targettype item extrafields; do

toCOPY/lst_utils.sh

@@ -0,0 +1,207 @@
+#
+# Copyright (c) 2020 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+rpms_from_layer_build_template="rpm.lst"
+image_inc_from_layer_build_template="image.inc"
+dev_image_inc_from_layer_build_template="image-dev.inc"
+
+config_dir=${MY_REPO}/../stx-tools/centos-mirror-tools/config
+distro="centos"
+layer="all"
+
+# Store urls for package list files of the various layers in an associative array
+declare -A layer_pkg_urls
+
+# Store urls for image include files of the various layers in an associative array
+declare -A layer_image_inc_urls
+
+url_to_file_name () {
+    echo "${1}" | sed 's#[:/ ]#-#g'
+}
+
+merge_lst () {
+    local cfg_dir=$1
+    local distro=$2
+    local template=$3
+
+    local cfg_name="${distro}_build_layer.cfg"
+    local layer_cfgs
+    local layers
+    local layer
+
+    if [ "$cfg_dir" == "" ] || [ "$distro" == "" ] || [ "$template" == "" ]; then
+        echo "ERROR: merge_lst: missing argument" >&2
+        return 1
+    fi
+
+    if [ "$MY_REPO" == "" ]; then
+        echo "ERROR: merge_lst: environment variable MY_REPO must be set" >&2
+        return 1
+    fi
+
+    layer_cfgs=$(find ${MY_REPO} -maxdepth 3 -name ${cfg_name})
+    if [ "$layer_cfgs" == "" ]; then
+        echo "ERROR: merge_lst: Could not find any '${cfg_name}' files" >&2
+        return 1
+    fi
+
+    layers=$(cat ${layer_cfgs} | sort --unique)
+    layers+=" mock"
+
+    (
+    for layer in ${layers}; do
+        for f in $(find ${cfg_dir}/${distro}/${layer} -name ${template} ); do
+            grep -v '^#' $f || true
+        done
+    done
+
+    for f in $(find ${MY_REPO} -maxdepth 3 -name ${distro}_${template};  \
+                find ${MY_REPO} -maxdepth 3 -name ${distro}_s${template}; \
+                ); do
+        grep -v '^#' $f || true
+    done
+    ) | sort --unique
+}
+
+set_layer_image_inc_urls () {
+    local option="${1}"
+
+    if [ "${option}" == "" ]; then
+        return
+    fi
+
+    local layer_and_inc_type="${option%,*}"
+    local layer="${layer_and_build_type%,*}"
+    local inc_type="${layer_and_inc_type#*,}"
+    local layer_image_inc_url="${option##*,}"
+
+    layer_image_inc_urls["${layer_and_inc_type}"]="${layer_image_inc_url}"
+}
+
+set_layer_pkg_urls () {
+    local option="${1}"
+
+    if [ "${option}" == "" ]; then
+        return
+    fi
+
+    local layer_and_build_type="${option%,*}"
+    local layer="${layer_and_build_type%,*}"
+    local build_type="${layer_and_build_type#*,}"
+    local layer_pkg_url="${option##*,}"
+
+    layer_pkg_urls["${layer_and_build_type}"]="${layer_pkg_url}"
+}
+
+read_layer_image_inc_urls () {
+    local layer="${1}"
+    local cfg="${config_dir}/${distro}/${layer}/required_layer_iso_inc.cfg"
+    local line=""
+    local key
+
+    if [ ! -f "${cfg}" ]; then
+        return 0;
+    fi
+
+    # Clear all pre-existing entries
+    for key in "${!layer_image_inc_urls[@]}"; do
+        unset layer_image_inc_urls[${key}]
+    done
+
+    while read line; do
+        line=$(echo "${line}" | sed 's/^[ \t]*//;s/[ \t]*$//' | grep '^[^#]')
+        if [ "${line}" == "" ]; then
+            continue
+        fi
+        set_layer_image_inc_urls "${line}"
+    done < "${cfg}"
+}
+
+read_layer_pkg_urls () {
+    local layer="${1}"
+    local cfg="${config_dir}/${distro}/${layer}/required_layer_pkgs.cfg"
+    local line=""
+    local key
+
+    if [ ! -f "${cfg}" ]; then
+        return 0;
+    fi
+
+    # Clear all pre-existing entries
+    for key in "${!layer_pkg_urls[@]}"; do
+        unset layer_pkg_urls[${key}]
+    done
+
+    while read line; do
+        line=$(echo "${line}" | sed 's/^[ \t]*//;s/[ \t]*$//' | grep '^[^#]')
+        if [ "${line}" == "" ]; then
+            continue
+        fi
+        set_layer_pkg_urls "${line}"
+    done < "${cfg}"
+}
+
+set_and_validate_config_dir () {
+    # Note: Setting the global 'config_dir' here.  Not local!
+    config_dir=${1}
+
+    if [ ! -d ${config_dir} ]; then
+        echo "Error: Invalid config_dir '$config_dir'"
+        echo "    Please select one of: $(find ${config_dir} -maxdepth 1 ! -path ${config_dir} -type d -exec basename {} \;)"
+        echo
+        usage
+        exit 1
+    fi
+}
+
+set_and_validate_distro () {
+    # Note: Setting the global 'distro' here.  Not local!
+    distro=${1}
+
+    if [ ! -d ${config_dir}/${distro} ]; then
+        echo "Error: Invalid distro '$distro'"
+        echo "    Please select one of: $(find ${config_dir} -maxdepth 1 ! -path ${config_dir} -type d -exec basename {} \;)"
+        echo
+        usage
+        exit 1
+    fi
+
+    if [ -d ${config_dir}/${distro}/${layer} ]; then
+        read_layer_pkg_urls ${layer}
+        read_layer_image_inc_urls ${layer}
+    else
+        echo "Warning: layer ${layer} not defined for distro '${distro}', please provide a valid layer via '-l <layer>'"
+    fi
+}
+
+set_and_validate_layer () {
+    # Note: Setting the global 'layer' here.  Not local!
+    layer=${1}
+
+    if [ ${layer} != "all" ] && [ ! -d ${config_dir}/${distro}/${layer} ]; then
+        echo "Error: Invalid layer '$layer'"
+        echo "    Please select one of: all $(find ${config_dir}/${distro} -maxdepth 1 ! -path ${config_dir}/${distro} -type d -exec basename {} \;)"
+        echo
+        usage
+        exit 1
+    fi
+
+    read_layer_pkg_urls ${layer}
+    read_layer_image_inc_urls ${layer}
+}
+
+# Pick up value of the config_dir from environment if set
+if [ "$STX_CONFIG_DIR" != "" ]; then
+    set_and_validate_config_dir "$STX_CONFIG_DIR"
+fi
+
+# Pick up value of layer from environment if set
+if [ "$LAYER" != "" ]; then
+    set_and_validate_layer "$LAYER"
+fi
+
+read_layer_pkg_urls ${layer}
+read_layer_image_inc_urls ${layer}

toCOPY/populate_downloads.sh

@@ -3,14 +3,24 @@
 # SPDX-License-Identifier: Apache-2.0
 #
 
+POPULATE_DOWNLOADS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
+
+source $POPULATE_DOWNLOADS_DIR/lst_utils.sh
+
 usage () {
-    echo "$0 <mirror-path>"
+    echo
+    echo "Create a virtual downloads directory containing only files (likely tarballs) listed in various lst files."
+    echo "The virtual directory contains only symlinks to to previously downloaded tarballs/files."
+    echo
+    echo "$0 [--config-dir=<dir>] [--distro=<distro>] [--layer=<layer>] [ --mirror-dir=<mirror-path> | <mirror-path> ]"
+    echo
+    echo "  --config-dir=<dir>: Use an alternate config directory rather than the system defined one"
+    echo "  --distro=<distro>: Set distro we intend to build.  Default 'centos'"
+    echo "  --layer=<layer>: Set layer we intend to build.  Default: use the LAYER environmnet valiable, or 'all'."
+    echo "  --mirror-dir=<dir>: Set the mirror directory.  This is where the previously download tarballs are located."
 }
 
-if [ $# -ne 1 ]; then
+mirror_dir=""
-    usage
-    exit -1
-fi
 
 if [ -z "$MY_REPO" ]; then
     echo "\$MY_REPO is not set. Ensure you are running this script"
@@ -19,10 +29,57 @@ if [ -z "$MY_REPO" ]; then
     exit -1
 fi
 
-mirror_dir=$1
+TEMP=$(getopt -o h --long help,config-dir:,distro:,layer:,mirror-dir: -n 'populate_downloads' -- "$@")
-tarball_lst=${MY_REPO_ROOT_DIR}/stx-tools/centos-mirror-tools/tarball-dl.lst
+if [ $? -ne 0 ]; then
+    echo "getopt error"
+    usage
+    exit 1
+fi
+eval set -- "$TEMP"
+
+while true ; do
+    case "$1" in
+        --mirror-dir)     mirror_dir=$2 ; shift 2 ;;
+        --config-dir)     config_dir="${2}"; shift 2 ;;
+        --distro)         set_and_validate_distro "${2}"; shift 2 ;;
+        --layer)          set_and_validate_layer "${2}"; shift 2 ;;
+        -h|--help)        echo "help"; usage; exit 0 ;;
+        --)               shift ; break ;;
+        *)                usage; exit 1 ;;
+    esac
+done
+
+if [ "$mirror_dir" == "" ]; then
+    if [ $# -ne 1 ]; then
+        usage
+        exit -1
+    fi
+
+    mirror_dir=$1
+fi
+
+echo "mirror_dir=${mirror_dir}"
+echo "config_dir=${config_dir}"
+echo "distro=${distro}"
+echo "layer=${layer}"
+
+tarball_downloads_template="tarball-dl.lst"
+extra_downloads_template="extra_downloads.lst"
+
+TMP_LST_DIR=$(mktemp -d /tmp/tmp_lst_dir_XXXXXX)
+mkdir -p $TMP_LST_DIR
+tarball_lst="$TMP_LST_DIR/${tarball_downloads_template}"
+extra_downloads_lst="$TMP_LST_DIR/${extra_downloads_template}"
+merge_lst ${config_dir} ${distro} ${tarball_downloads_template} > ${tarball_lst}
+merge_lst ${config_dir} ${distro} ${extra_downloads_template} > ${extra_downloads_lst}
+
 downloads_dir=${MY_REPO}/stx/downloads
-extra_downloads="mlnx-ofa_kernel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.src.rpm libibverbs-41mlnx1-OFED.4.5.0.1.0.45101.src.rpm rdma-core-45mlnx1-1.45101.src.rpm"
+
+extra_downloads=""
+if [ -f  ${extra_downloads_lst} ]; then
+    extra_downloads="$(grep -v '^#' ${extra_downloads_lst})"
+fi
+
 
 mkdir -p ${MY_REPO}/stx/downloads
 
@@ -43,7 +100,10 @@ grep -v "^#" ${tarball_lst} | while read x; do
         rel_dir_name=$(dirname ${rel_path})
         if [ ! -e ${downloads_dir}/${rel_dir_name}/${tarball_file} ]; then
             mkdir -p ${downloads_dir}/${rel_dir_name}
+            echo "Creating symlink for $(basename ${source_file})"
             ln -sf ${source_file} ${downloads_dir}/${rel_dir_name}/
+        else
+            echo "Already have symlink for $(basename ${source_file})"
         fi
     fi
 done
@@ -51,3 +111,5 @@ done
 for x in ${extra_downloads}; do
     ln -sf ${mirror_dir}/downloads/$x ${downloads_dir}
 done
+
+\rm -rf ${TMP_LST_DIR}