Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
The patch is based on 431885231a
And is updated with kata 1.10 repo and 1.10.0-4.1 rpms.
1. add kata container 1.10 stable repo with kata 1.10.0-4.1 rpms.
kata-runtime is the main rpm, and other rpms including qemu
are the rpms required by kata-runtime.
2. upgrade containerd to 1.3.0
To support kata container, kubernetes need switch runtime from
dockershim to containerd. And need use containered with 1.3.0
in order to support secure private registry.
3. add crictl as the CLI for containerd.
Story: 2006145
Task: 36744
Task: 36745
Task: 36746
Change-Id: I932e0dde0a0b48257e4acd17d6550f9ec5029555
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
The cgcs-users package provides a restricted shell ibsh (Iron Bars SHell)
that is no longer being referenced in StarlingX, so we can remove this
tarball from the download list.
Story: 2007102
Task: 38148
Depends-On: https://review.opendev.org/702939
Change-Id: I6e1a25e7318eb3e26150e916335d58c2b60a8f67
Signed-off-by: Saul Wold <sgw@linux.intel.com>
As python-smartpm is no longer required, it can now be dropped from
the tarball-dl.lst file.
Depends-On: https://review.opendev.org/702791
Change-Id: Ifc0a413688151ebb68a71b83a23adb888d1ece08
Story: 2006227
Task: 38138
Signed-off-by: Don Penney <don.penney@windriver.com>
1. add kata container 1.9 stable repo with kata 1.9.2-7.1 rpms.
kata-runtime is the main rpm, and other rpms including qemu
are the rpms required by kata-runtime.
2. upgrade containerd to 1.3.0
To support kata container, kubernetes need switch runtime from
dockershim to containerd. And need use containered with 1.3.0
in order to support secure private registry.
3. add crictl as the CLI for containerd.
Story: 2006145
Task: 36744
Task: 36745
Task: 36746
Change-Id: I04076681decfa24335cf8dd2a64fc5233452dfbe
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Uprev i40e to version 2.10.19.30
i40evf gets replaced by iavf version 3.7.61.20
The iavf driver supports both fortville and columbiaville,
so they decided to rename from i40evf to something more generic.
The Intel FPGA Programmable Acceleration Card N3000 contains
dual Intel XL710 NICs and an FPGA for acceleration purposes.
This driver upgrade is required to support those NICs.
Story: 2006740
Task: 37542
Change-Id: I53c731c1b519b1412acda5f5e2ee7dc33729d40d
Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>
Kubernetes 1.16.2 was released Oct 15, 2019
It has a number of fixes since 1.16.0 including fixes
for openstack, stability and security.
Changelog info can be found here:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md
The security fixes are primarily in golang.
Story: 2005860
Task: 37156
Change-Id: Id68754fb800006f7ea1a8011ec64c97472305fc1
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
Add the new elasticsearch helm charts to the tarball list.
Story: 2005733
Task: 36958
Change-Id: I6fc439f1e792b7cba6b4c1a3792b318464840829
Signed-off-by: Kevin Smith <kevin.smith@windriver.com>
Kubernetes 1.16.0 was just released.
It requires etcd 3.3.15.
Change-Id: I32903d6b841111deba241d905f3f060928db3625
Story: 2005860
Task: 36700
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
Include link for opae-intel-fpga-driver into stx download list.
Change-Id: Ia03f5cddfdcdf2afc201a06e540e9645188e3e95
Story: 2006495
Task: 36605
Signed-off-by: Alex Kozyrev <alex.kozyrev@windriver.com>
From https://pypi.org/project/requests-toolbelt/0.9.1/;
version 0.9.1 will support python3
Story: 2006158
Task: 35794
Change-Id: Ie35aca7266ca8b17d6b2f9b37dd4dcd851ca9df6
Signed-off-by: Sun Austin <austin.sun@intel.com>
Kubernetes 1.15.3 requires go version 1.12.9 to build.
Kubernetes and Golang do not have rpms or src rpms
on centos for those versions, so source tarballs
are required to build them.
Change-Id: I0369605a361fff4d24af26c0ecd2d01a0bf655da
Story: 2005860
Task: 35839
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
Some patches we upstreamed for our redfish feature have already
been merged to 1.1.0 version.
Deployment test on simplex and sensor list test pass!
Story: 2005861
Task: 36276
Change-Id: I6343b4acb555e3cb87e9496dae49dfd436212d43
Signed-off-by: zhipengl <zhipengs.liu@intel.com>
Cloud-Provider-Openstack is required for Keystone authentication with
Kubernetes. Since we are no longer doing that, we can remove
Cloud-Provider-Openstack from the build.
Story: 2002843
Task: 26958
Depends-On: https://review.opendev.org/670057
Change-Id: I6e8234791b0cc9f9e811e9c930bcd8026a20df7f
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
This commit upgrades the ixgbe and ixgbevf drivers to the
latest versions.
ixgbe is upversioned to 5.5.5 from 5.5.3
ixgbevf is upversioned to 4.5.3 from 4.5.1
For ixgbe, Intel has noted that RHEL 7.6 support is introduced
in version 5.5.5, whereas only RHEL 7.5 support is present in version
5.5.3. The 5.5.5 version will also pick up a bug fix needed
for SR-IOV operations.
For ixgbevf, Intel has verified RHEL 7.6 support in version 4.5.3,
while they have verified only RHEL 7.5 in version 4.5.1.
Change-Id: I6141409b4b97d4bc888c0ac071bef4be418eb0f7
Closes-Bug: #1830636
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Add github.com/helm/charts to to tarball-dl.lst.
This includes the stable elasticsearch, filebeat, and kibana
Helm charts required to enable Infrastructure and Cluster monitoring.
Change-Id: Id6e1180f2c4d6b736aaedd73430d0696ae9d0744
Story: 2005733
Task: 33415
Signed-off-by: John Kung <john.kung@windriver.com>
Cloud-Provider-Openstack is required for Keystone authentication with
Kubernetes. This commit brings in Cloud-Provider-Openstack as a
pre-built binary to shove into an RPM as part of the ISO. The source
is also downloaded, but not used at the moment.
Story: 2002843
Task: 26958
Change-Id: Ida08e78e0889eac68b0420b126d4ce70f875c99e
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
Keep lists sorted to easily identify duplicate
packages and libraries.
Existing files were piped through:
sed 's/!\(.*\)/\1!/' \
| LC_COLLATE=en_US.UTF-8 sort --numeric-sort \
| sed 's/\(.*\)!$/!\1/'
Please use the same command pipe when adding new lines
to ensure proper ordering.
Story: 2003605
Task: 28856
Depends-on: If4533ff264af33b7dea78914fe43eb86c8c0b7c3
Change-Id: Ibc8b2f92b220a817e8189413d766da6dba0c4991
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
Archive download can keep file mirrors instead of git checkout when
build each time, which is more maintainable and stable way.
This changes add the source of archive download from githb of Ceph
submodules.
Story: 2003605
Task: 28856
Depends-On: I6220a7cd27ebbde54d1d353b52cba80293d891df
Change-Id: If4533ff264af33b7dea78914fe43eb86c8c0b7c3
Signed-off-by: Tingjie Chen <tingjie.chen@intel.com>
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
This updates the Openstack Helm (OSH) chart baselines for openstack-helm
and openstack-helm-infra repositories.
Change-Id: Ic788a2c86edfbceca1f1ff18dd0344472546c81b
Story: 2004520
Task: 29966
Signed-off-by: Robert Church <robert.church@windriver.com>
There is no prebuilt version of etcd on centos for 3.2.24
so the src tarball for that version needs to be downloaded
and built using a spec file.
3.2.24 is required for kubernetes 1.13.5
Story: 2005198
Task: 30392
Change-Id: I8f7061a9577941c257046721dbf5e957375691ab
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
Switch to tarball since there is no srpm package in centos repo yet.
This upgrade is in order to support Mellanox NICs again.
Story: 2004743
Task: 29195
Change-Id: I61572318c5017a51a5eb196df15ee93cb3b05eb5
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
This reverts commit 40a9622d77.
The URL for tss2 is not always working and this causes an
incomplete mirror in a clean environment.
Closes-bug:1816813
Change-Id: I9c8fa64e1f0f9154dd45a67088fdc27f4b8f8313
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
tss2 is the only tarball that is generated using git clone, this
commit makes the downloader get all the tarballs using wget.
Change-Id: Ib6009341f6b7c890f43962f2c24fc73e95ec81b7
Story: 2004413
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
The registry-token-server package relies on a set of dependencies,
downloaded at mirror-download time. This dependencies are snap-
shots of various git repos as of specific git commits. This
change downloads those snapshots as tarballs and makes them
available to the build-srpms phase of the build
Story: 2002840
Task: 22783
Change-Id: I4f7709516df81bb920ced124dfb0d1f5ffd87c81
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
Signed-off-by: Jason McKenna <jason.mckenna@windriver.com>
This commit adds helm-v2.12.1-linux-amd64.tar.gz to the list of
downloads, in order to upversion helm to v2.12.1.
This is necessary as support for kubernetes v1.12 support is only
introduced in helm v2.12.`
Story: 2004520
Task: 28774
Change-Id: I2501cd1174f85810892e51ac8a44746302eeb120
Signed-off-by: Joseph Richard <joseph.richard@windriver.com>
Scott Little