Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
The patch is based on 431885231a
And is updated with kata 1.10 repo and 1.10.0-4.1 rpms.
1. add kata container 1.10 stable repo with kata 1.10.0-4.1 rpms.
kata-runtime is the main rpm, and other rpms including qemu
are the rpms required by kata-runtime.
2. upgrade containerd to 1.3.0
To support kata container, kubernetes need switch runtime from
dockershim to containerd. And need use containered with 1.3.0
in order to support secure private registry.
3. add crictl as the CLI for containerd.
Story: 2006145
Task: 36744
Task: 36745
Task: 36746
Change-Id: I932e0dde0a0b48257e4acd17d6550f9ec5029555
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
1. add kata container 1.9 stable repo with kata 1.9.2-7.1 rpms.
kata-runtime is the main rpm, and other rpms including qemu
are the rpms required by kata-runtime.
2. upgrade containerd to 1.3.0
To support kata container, kubernetes need switch runtime from
dockershim to containerd. And need use containered with 1.3.0
in order to support secure private registry.
3. add crictl as the CLI for containerd.
Story: 2006145
Task: 36744
Task: 36745
Task: 36746
Change-Id: I04076681decfa24335cf8dd2a64fc5233452dfbe
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
To fix below kernel CVE, std/rt kernel will be upgraded to a
higher version than current version.
So we will upgrade kernel srpm to below version, which will
cover this issue.
std kernel: kernel-3.10.0-1062.1.2.el7.src.rpm
https://lists.centos.org/pipermail/centos-announce/2019-October/023457.html
rt kernel: kernel-rt-3.10.0-1062.1.2.rt56.1025.el7.src.rpm
https://access.redhat.com/errata/RHSA-2019:2830
linux-firmware is brought forward due to a kernel spec file
build dependency.
CVE bug: CVE-2019-11810:kernel: a NULL pointer dereference in
drivers/scsi/megaraid/megaraid_sas_base.c leading to DoS
CVE bug: CVE-2019-11811: kernel: use-after-free in IPMI Edit
CVE bug: CVE-2019-14835: kernel: vhost-net: guest to host kernel
escape during migration
Closes-Bug: 1849206
Closes-Bug: 1849209
Closes-Bug: 1847817
Change-Id: Ic8c107e4850d0679470a4c8214c85c6d9a800beb
Signed-off-by: Robin Lu <bin1.lu@intel.com>
This solves:
elfutils: Double-free due to double decompression of sections in
crafted ELF causes crash (CVE-2018-16402)
along with quite a few other issues.
See the announcement link:
https://lists.centos.org/pipermail/centos-cr-announce/2019-August/005856.html
for more details.
Change-Id: Ia328b6043c1815a023ab45ea6f8142dcef91864b
Closes-Bug: 1849201
Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>
This solves:
systemd: line splitting via fgets() allows for state injection
during daemon-reexec (CVE-2018-15686)
along with some other less critical issues. See the security
announcement link:
https://lists.centos.org/pipermail/centos-cr-announce/2019-August/006149.html
for more details.
Change-Id: Ia0fcc7184efea5b31408d7514921b58377beb329
Partial-Bug: 1849200
Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>
This commit adds the containernetworking-plugins package, which
brings in a number of CNI plugins including static and dhcp.
The containernetworking-cni package is removed, as it used to
contain the actual plugins before a new separate package /
project was created.
Change-Id: Ia6db0103fff8edadef0ec1c881b766bf7e5f661a
Closes-Bug: #1840391
Signed-off-by: Steven Webster <steven.webster@windriver.com>
These packages has been updated in upstream, this will cause
that whenever ./download_mirror.sh -u is used it will generate
a broken mirror.
This update has been tested generating a complete build and running
a provisioning in a duplex configuration.
Closes-Bug: 1817351
Change-Id: I80defd8f305377fd3660b32f18fa6459c5d4da20
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
Security Fix(es):
(CVE-2019-11477)-
An integer overflow flaw was found in the way
the Linux kernel's networking subsystem processed
TCP Selective Acknowledgment (SACK) segments.
While processing SACK segments,
the Linux kernel's socket buffer (SKB) data structure
becomes fragmented. Each fragment is about TCP
maximum segment size (MSS) bytes.
To efficiently process SACK blocks, the Linux kernel merges
multiple fragmented SKBs into one, potentially overflowing
the variable holding the number of segments.
A remote attacker could use this flaw to crash the Linux kernel
by sending a crafted sequence of SACK segments on a TCP
connection with small value of TCP MSS,
resulting in a denial of service (DoS).
(CVE-2019-11478)-
Kernel: tcp: excessive resource consumption while processing
SACK blocks allows remote denial of service.
(CVE-2019-11479)-
Kernel: tcp: excessive resource consumption for TCP connections
with low MSS allows remote denial of service.
Details:
https://access.redhat.com/errata/RHSA-2019:1481https://access.redhat.com/errata/RHSA-2019:1486https://nvd.nist.gov/vuln/detail/
Closes-Bug: 1836685
Change-Id: If42765222e641218c2e2282bf7264f3a7f7b863c
Signed-off-by: zhao.shuai <zhaos@neusoft.com>
kmem accounting feature was backported by RedHat and seems to be
incomplete and prone to memory leaks (and possible deadlocks).
This issue is triggered by Docker container runtime that activates
the kmem accounting feature for any cgroup by default. As a result
kmem accounting in runc is disabled on RHEL/CentOS starting 18.19.1
Updating Docker version to 18.09.6 in StarlingX to fix the issue.
Note that the client and container runtime are in separate packages
in Docker Engine 18.09. So, adding containerd.io and docker-cli to
fulfill the depenndencies for Docker Engine.
Change-Id: Ib229eb7ac4db45dbdf1260c6505242147e06838f
Closes-bug: 1831485
Signed-off-by: Alex Kozyrev <alex.kozyrev@windriver.com>
New set of CVEs was reported against Intel CPUs: CVE-2018-12126,
CVE-2018-12127, CVE-2018-12130 and CVE-2019-11091.
For these CVEs there are RH and CentOS updates available.
CVE-2018-12126:
Microarchitectural Store Buffer Data Sampling (MSBDS):
Store buffers on some microprocessors utilizing speculative
execution may allow an authenticated user to potentially
enable information disclosure via a side channel with local access.
A list of impacted products can be found here:
https://www.intel.com/content/dam/www/public/us/en/documents/
corporate-information/SA00233-microcode-update-guidance_05132019.pdf
CVE-2018-12127:
Microarchitectural Load Port Data Sampling (MLPDS):
Load ports on some microprocessors utilizing speculative execution
may allow an authenticated user to potentially enable information
disclosure via a side channel with local access. A list of impacted
products can be found here:
https://www.intel.com/content/dam/www/public/us/en/documents/
corporate-information/SA00233-microcode-update-guidance_05132019.pdf
CVE-2018-12130:
Microarchitectural Fill Buffer Data Sampling (MFBDS):
Fill buffers on some microprocessors utilizing speculative execution
may allow an authenticated user to potentially enable information
disclosure via a side channel with local access. A list of impacted
products can be found here:
https://www.intel.com/content/dam/www/public/us/en/documents/
corporate-information/SA00233-microcode-update-guidance_05132019.pdf
CVE-2019-11091:
Microarchitectural Data Sampling Uncacheable Memory(MDSUM):
Uncacheable memory on some microprocessors utilizing speculative
execution may allow an authenticated user to potentially enable
information disclosure via a side channel with local access.
A list of impacted products can be found here:
https://www.intel.com/content/dam/www/public/us/en/documents/
corporate-information/SA00233-microcode-update-guidance_05132019.pdf
These are from the http://cve.mitre.org website.
These are the MDS security CVEs.
Closes-Bug: 1830487
Change-Id: I9c69ca78dc046128521d2a46b520f9c242fe6e56
Signed-off-by: zhiguo.zhang <zhiguox.zhang@intel.com>
Cengn builds periodically fail when an upstream rpm repo in anavailable.
By far the top offender is epel.blizoo.mk that seems to go offline
roughly monthly.
The only rpm that we currently use from that repo is
python-pyngus-2.2.1-1.el7.noarch.rpm, referenced from
rpms_centos3rdparties.lst.
However that rpm is eclipsed by the newer
python2-pyngus-2.2.4-1.el7.noarch.rpm which we reference from
rpms_centos.lst.
It is python2-pyngus-2.2.4-1.el7.noarch.rpm that is used in our ISO
build. Pyngus is not used in our docker images.
This update will remove python-pyngus-2.2.1-1.el7.noarch.rpm
from rpms_centos3rdparties.lst, and remove the repo ...
'yum.repos.d/StarlingX_3rd_epel_blizoo_mk.repo' as it will
no longer be supplying rpms that are both unique and needed.
Testing:
docker build --tag stx-builder --file Dockerfile .
build-pkgs
build-iso
build-stx-base
build-stx-images
Change-Id: I817dae97282474a0c9b56941e4b227b6ff82d9aa
Closes-Bug: 1831113
Signed-off-by: Scott Little <scott.little@windriver.com>
Keep lists sorted to easily identify duplicate
packages and libraries.
Existing files were piped through:
sed 's/!\(.*\)/\1!/' \
| LC_COLLATE=en_US.UTF-8 sort --numeric-sort \
| sed 's/\(.*\)!$/!\1/'
Please use the same command pipe when adding new lines
to ensure proper ordering.
Story: 2003605
Task: 28856
Depends-on: If4533ff264af33b7dea78914fe43eb86c8c0b7c3
Change-Id: Ibc8b2f92b220a817e8189413d766da6dba0c4991
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
For openstack pike release, nove request to acces
/usr/share/OVMF/OVMF_CODE.fd in nova/nova/virt/libvirt/driver.py,
which is removed in upgraded OVMF-20180508-3.gitee3198e672e2.el7.noarch.rpm
Rollback to previous OVMF package
Closes-Bug: 1814335
Change-Id: I2376bc7e0bbc21c61be3ef8964c527ddc7fcf250
Signed-off-by: Martin, Chen <haochuan.z.chen@intel.com>
new lvm2 rpm causes AIO duplex deploy failure, so lvm2 is kept with
old version currently. device-mapper-multipath should be kept with
old version also to avoid dependency failure.
Move device-mapper packages to rpms_centos.lst since all packages
could be found in centos repo.
Story: 2004522
Task: 29099
Change-Id: I5cd4d434a629201934a48a551d4fb354f8d57318
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
A few more packages are added in order to pass build
Story: 2004522
Task: 28444
Change-Id: I04a4eff125fffab3e422ab7516699c36eccd763e
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
A few packages are upgraded also to pass ISO build
Depends-On: https://review.openstack.org/629483
Story: 2004521
Task: 28558
Change-Id: I3408015fe9a818db7f0ea6f1c0f46b5a116cf5f1
Signed-off-by: Martin, Chen <haochuan.z.chen@intel.com>
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Package audit-config is created to config customized audit
config file. Since there is no other change for audit, we
could replace srpm with rpm directly.
Test:
Pass build and multi node deploy test.
Story: 2003768
Task: 27602
Change-Id: I96e9ca5c901c8caf9d226ad99a4e05369477d37e
Signed-off-by: slin14 <shuicheng.lin@intel.com>
Mirror for kernel-rt source rpm was not working so it was updated with a
working one, also updated rpms_centos3rdparties.lst with new version
of python2-pysocks.
Closes-Bug:1799729
Change-Id: Ifecf95dbc998f2f87e97ea5f3294e7c5b52ae318
Signed-off-by: Hayde Martinez <hayde.martinez.landa@intel.com>
Right now lshell is looked for in StarlingX_3rd_epel_blizoo_mk.repo,
and it should be in centos3rdparties instead of centos list.
As it is now, there's a bug because right before centos list is
downloaded, the downwload_mirror.sh script removes yum 3rd parties config
files, and later the yumdownloader doesn't find the lshell package.
Closes-Bug: 1793615
Change-Id: I1eab20226fb92b4b2e05f50958112092bb643bc1
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
Story: 2003596
Task: 24917
kernel version is upgraded from 862.6.3 to 862.11.6
Change-Id: If2efecc9510617cae5645ca07732620a40b2ebed
Signed-off-by: slin14 <shuicheng.lin@intel.com>
Story: 2003389
Task: 24556
Some rpms are not upgraded for different reason:
1. libguestfs/python-libguestfs is not upgraded due to 7.5 version needs libvirt-daemon-kvm >= 3.9.0-1
2. puppet modules rpm is not upgraded due to openstack puppet has dependency check.
3. packages has higher version than CentOS7.5 is not changed.
Change-Id: I9e9dee9a51cdbed9d486dd802725b0956bfe4a3f
Signed-off-by: slin14 <shuicheng.lin@intel.com>
-By standardizing the list files name it's possible to eliminate
one parameter in the download function in dl_rpms and reduce complexity.
-Now the download function does not receive it as a parameter anymore,
and instead it gets it from the RPM name.
-Also the download function now decides if build a wget command or
a yumdownloader commnad based on the content of the list (by identifying
the # character) instead of using the name of the list.
I adapted download_mirror.sh.
Change-Id: I041fc9c704156215f06149e5b4c16cd92990e17c
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
Scott Little