download_mirror.sh fails when run without '-n'.
It seems that some functions were relocated to utils.sh that
require the 'SUDO' variable to be defined. Those common utilities
are called from both download_mirror.sh and dl_rpms.sh.
download_mirror.sh sets it correctly, but dl_rpms.sh does not.
Instead dl_rpms.sh is setting 'SUDOCMD', which the utils.sh ignores.
The result is that some yum commands are runs under sudo, while others
are not. Eventually yumdownloader failes with ...
Permission denied: '/var/lib/rpm/.dbenv.lock'
The fix is to convert dl_rpms.sh to use SUDO rather than SUDOCMD.
Closes-Bug: 1980684
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Ib4afdba0260e67f06997c360272f40f28fb11834
Mirror scripts sometimes leave corrupted/partial files behind.
Problems
========
1) wget is called with the -O flag, and the server returns an HTTP
error for the requested URL (404 etc). Wget leaves a zero-length file
behind. This doesn't seem to happen without the -O flag.
2) wget starts the download which stalls & times out half-way; wget
gives up and requests the same file with a byte offset of the form
"Range: bytes=1234-", and the web server doesn't support open-ended
ranges. In this case wget prints out a warning, leaves a partial file
behind and returns success.
3) Sites like GitHub generate repo tarballs on the fly, eg:
https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.19.3.tar.gz
Since tags can move, downloading such a file twice may result in a
different file. Therefore HTTP "resume download" may corrupt files in
this case.
4) Git "keyword expansion" feature may result in differences in source
files being downloaded. For example, this file:
https://github.com/kubernetes/kubernetes/blob/v1.19.3/staging/src/k8s.io/component-base/version/base.go
contains lines similar to:
gitVersion = "v0.0.0-master+$Format:%h$"
where %h is replaced with a short SHA when the tar file is
exported/downloaded. How short the SHA is depends on git history and
sometimes results in shortened SHAs of different lengths. So
downloading that file may result in different files.
Therefore HTTP "Range" header may corrupt files in this case as
well.
5) Curl is invoked with the "--retry" option and starts the download;
connection stalls; curl gives up, connects again, skips the 1st N
bytes and appends to the partial file. If the file changes while we
are doing this, it will end up corrupting the file. This is very
unlikely to happen and I haven't been able to reproduce this case.
Problems with HTTP Range header
===============================
Curl/wget "resume/continue download" feature has no way of verifying
whether the partial file on disk, and the one being re-requested, are in
fact the same file. If the file changes on the server between
downloads, "resume download" will corrupt it.
Some web servers don't support this at all, which triggers case (2)
with wget.
Some web servers support the Range header, but require that the end
byte position is present. This is not compatible with wget & curl.
For example curl & wget add headers similar to: "Range: bytes=1234-"
means give me the file starting at offset 1234 and till EOF. This also
triggers case (2).
This patch
==========
* Always download the file to a temporary name, then rename into place
* Use curl instead wget (better error handling). The only exception is
"recursive downloads", which curl doesn't support.
Bug: https://bugs.launchpad.net/starlingx/+bug/1950017
Change-Id: Iaa89009ce23efe5b73ecb8163556ce6db932028b
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Download_mirror.sh takes 15 hours to download all the rpms and
tarballs required to build StarlingX into a fresh workspace.
It should be much faster than that.
Replace the current serial download algorithm with one that is
parallel. I'll cap it at 8 parallel downloads for now. I'm
a little worried about overwelming CENGN. This is sufficient
to drop download times from 15 to 3 hours for a fresh workspace,
and 30 min to 5 min to refresh an existing workspace.
Closes-Bug: 1918477
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I469b4fee3cb304fe2984aa697ce2dc6cec52e79e
StarlingX has moved beyond the OpenStack pike release and will
be following master and the latest release, so don't include
OpenStack release names.
This also removes the r1 reference from the download directory
path.
Story: 2006835
Task: 37410
Change-Id: I0c548e9c97f2fb162394d0c381f02a40072e7400
Signed-off-by: Saul Wold <sgw@linux.intel.com>
The default yumdownloader approach to downloading files
does so one at a time, and is very slow. Too slow for layered
build which is built around the assumption that files can
be downloaded faster than they can be built.
This update will switch from yumdownloader to reposync
for the download of lower layer rpms. It exploits the
fact that each layer and build type publishes it's own lst file
of rpms to download. The lst file is transformed into an'
includepkgs directive in a custom yum.conf which is passed
to reposync, so we only sync the desited rpms.
Reposync won't redownload rpms that it already has, even if the
upstream repodata indicates that the files checksum has changed.
Forceing the redownload of these rpms requires that we manually
download the upstream repodata and that we use verifytree to
identify and delete the new obsolete rpms.
Also including two small bug fixes found while investivgating an
alternative solution to launchpad 1864221.
- incorrect userid in a chown
- Bug in get_url, --url pass being passed in the wrong place
Story: 2006166
Task: 39307
Closes-Bug: 1864221
Change-Id: If12b98ff4f5f24d9318250356920f397419f0f80
Signed-off-by: Scott Little <scott.little@windriver.com>
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
The 'yum makecache' step is failing. The issue is seen when the fast
mirror plugin for yum is in use (default).
The metadata for a yum repo as scattered across several files. The
master file is repomd.xml, and it may list several supporting files,
e.g.
filelists.xml.gz
primary.xml.gz
filelists.sqlite.bz2
other.sqlite.bz2
other.xml.gz
primary.sqlite.bz2
When fast mirror plugin is in use, each file might download from a
separate source. This opens the door for a race condition when a repo
update is rolling out across the mirrors.
The error suggests we have already downloaded a repomd.xml, but it is
obsolete and references a supporting file that no longer exists.
A second possible source of inconsistent data is caching proxies.
Solution:
1) Disabling the yum fast cache plugin is undesirable, and there
is no option forcing metadata iof a given repo to be single sourced.
So this update adds a limited number of retries for the
'yum makecache' command.
So far we've never seen 'yum makecache' fail twice in a row...
largely because the 'fastest mirror' plugin rarely downloaded from the
same source twice.
2) Add 'http_caching=packages' to yum.conf, which asks upstream proxies
to not cache repodata.
Change-Id: I3fa3f61e38d868fb14f4175b87c0d323abb57def
Closes-bug: 1818911
Signed-off-by: Scott Little <scott.little@windriver.com>
Problem:
The wiki states that one of the criteria for verifying
a successful download_mirror.sh is that logs/*_failmove_*.log
be empty.
In reality the logs are not empty due to content improperly
routed to these logs.
e.g.
logs/L1_failmoved_url_centos.log:url_srpm:http://mirror.starlingx.cengn.ca:80/mirror/centos/centos/mirror.centos.org/centos/7.5.1804/updates/x86_64/Packages/iptables-1.4.21-24.1.el7_5.i686.rpm
logs/L1_failmoved_url_centos.log:http://mirror.starlingx.cengn.ca:80/mirror/centos/centos/mirror.centos.org/centos/7.5.1804/updates/x86_64/Packages/iptables-1.4.21-24.1.el7_5.x86_64.rpm
logs/L1_failmoved_url_centos.log:found_srpm:iptables-1.4.21-24.1.el7_5.x86_64.rpm
...
Solution:
1) 'url_srpm:' and 'found_srpm:' should not be routed to the
_failmove_ log. We'll send those to stdout instead.
2) The http: log was traces to the get_url function returning two
entries. It was discovered that appending --url to the yum download
command causes it to ignore the --archlist arguement. As a result
the output must be manually filtered to find the arch we are looking
for.
Story: 2003906
Task: 27974
Change-Id: Iabf0859ff3317acc1f298e9c791bd47b95366477
Signed-off-by: Scott Little <scott.little@windriver.com>
StarlingX needs to download a variety of rpms and tarballs
from various upstream sources. Unfortunately the upstream sources
are not always dependable. Either servers go down, are unreachable,
or drop older content that we still depend on.
Our proposed solution is to run our own mirror to capture an
independent copy of the content needed by StarlingX.
For this purpose, a server has been set up at
http://mirror.starlingx.cengn.ca/mirror/centos
The mirror will use deterministic paths derived from the upstream
urls. Scripts will be used to convert an upstream url to
the mirror's equivalent url (see function url_to_stx_mirror_url in
url_utils.sh)
The mirror will be refreshed daily. New .lst entries will be
processed at that time. Processing of changes under yum.repos.d
is not automated by this update. Expect a follow-up update to
address this issue soon. These scripts are found under the
'stx_mirror_scripts' subdirectory.
Changes are made to the download_mirror.sh script, and it's
supporting scripts. New arguments have been added to each
script to select the download source.
-s StarlingX mirror only
-S StarlingX mirror, with upstream source as backup
-u Upstream source only
-U Upstream source, with StarlingX mirror as backup
You do not need to provide any of these flags. Continue to
us download_mirror.sh as you always have. The default
behavior is currently set to '-S', i.e. first try the
StarlingX mirror, with upstream source as backup.
If this proves to place to heavy a load on the existing
server, we might switch the default to '-U', i.e. first
try the upstream source, with StarlingX mirror as backup.
If poor download performance is seen, you might want to try
explicitly adding -U as an argument.
The remaining two options are not recommended for regular use.
Upstream only, i.e. '-u', restores original behaviour, but
you may once again encounter rpms that have aged out, and
been removed from their original repos. StarlingX only,
i.e. '-s', is vulnerable if a .lst file has been updated,
but the mirror has not yet processed it.
Change-Id: I7e0f3d9fb99253662f9f4bf12457d39250408c0b
Story: 2003906
Task: 26785
Signed-off-by: Scott Little <scott.little@windriver.com>
A refactor was made in the dl_rpms.sh script to simplify the
maintenance and development. A new file utils.sh was created to
store all the common functions that could be used by another scripts.
Also, some unit testing was added to the util.sh file.
Story: 2002736
Task: 26194
Change-Id: Iaac2e74a84abbfc9f40ef51b1da819702d040655
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
The dl_rpms.sh script may be passed an optional yum.conf file. If
this file is used, then the releasever variable is read from the
rather than use the default releasever=7. If the releasever is
not defined in the file, the grep which searches for the variable
will fail, and the dl_rpms.sh script exits with an error.
This change checks for the presence of releasever before attempting
to read it from the file. Change made to dl_rpms.sh and
mirror-check.sh
Tested using yum.conf files both with and without releasever
present.
Change-Id: Ie13241e5662188d75b5d147537915c0a35ea7397
Closes-Bug: 1791113
Signed-off-by: Jason McKenna <jason.mckenna@windriver.com>
This solves a problem when the default releasever is not set for
distros that dont have a YUM as a standard tool.
Read from the yum.conf if set on the command line since that is mostly
likely going to be when we need to know this, in the default case (no -c
yum.conf) it will get the correct value from the system.
Change-Id: I21b0e8f881d054d8cdf6cdd831836a7f3e3020b2
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Eight log files per RPM list is complicated for error detection and
managing. This commmit is changing that to three logs.
One for general information about errors, other for missing RPMs and
another for missing SRPMs. These could be used by download_mirror.sh
script to look for packages with K1.
All logs are now kept in logs directory.
And now the download_mirror.sh has a function for generating logs'
names.
Change-Id: I901657bc56209d54a3fe5be492d76716249f2755
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
This adds two sets of deployments scripts to aid in setting up virtual
environments for testing StarligX, for libvirt/qemu and VirtualBox.
This is the first exposure of an internal project and is not fully
error-proofed, it should be examined carefully to understand what
the scripts will do to your system before running them.
The first set of fixes for this to make it less instrusive and work
in a shared server follow in https://review.openstack.org/#/c/597643.
There are also a couple of bashate errors fixed since that is now a voting job...
[NOTE(dtroyer): I converted the original commit message into a README
because that's information that needs to be with the scripts.]
Needed-by: https://review.openstack.org/#/c/597643
Change-Id: I0a6a148720b7a239380fd48f7ffdab272472e664
Co-authored-by: Lianhao Lu <lianhao.lu@intel.com>
Co-authored-by: Brian Avery <brian.avery@intel.com>
Co-authored-by: Nakul Dahiwade <nakul.dahiwade@intel.com>
Co-authored-by: Yan Chen <yan.chen@intel.com>
Co-authored-by: Ruijing Guo<ruijing.guo@intel.com>
Co-authored-by: Shuicheng Lin <shuicheng.lin@intel.com>
Co-authored-by: Felipe de Jesus Ruiz Garcia <felipe.de.jesus.ruiz.garcia@intel.com>
Co-authored-by: Jose Perez Carranza <jose.perez.carranza@intel.com>
Co-authored-by: Eddie Ramirez <eddie.ramirez@intel.com>
Co-authored-by: Kailun Qin <kailun.qin@intel.com>
Co-authored-by: Forrest Zhao <forrest.zhao@intel.com>
Co-authored-by: Kailun Qin <kailun.qin@intel.com>
Signed-off-by: Abraham Arce <abraham.arce.moreno@intel.com>
Signed-off-by: Dean Troyer <dtroyer@gmail.com>
EPEL7 rpms builds are visible at kojipkgs.fedoraproject.org
and often remain available there long after they age out
of the official repo.
Use kojipkgs.fedoraproject.org as a backup source for
packages when not found in the repo. Log seperately
so that these dropouts can be identified and corrected
in the lst files.
Also improved the tests for already downloaded packages
in several places to accelerate the test cycle.
Change-Id: Iaaa9fe05ac605a3604acf0048c571c6e88303692
Story: 2003157
Task: 23292
Signed-off-by: Scott Little <scott.little@windriver.com>
-By standardizing the list files name it's possible to eliminate
one parameter in the download function in dl_rpms and reduce complexity.
-Now the download function does not receive it as a parameter anymore,
and instead it gets it from the RPM name.
-Also the download function now decides if build a wget command or
a yumdownloader commnad based on the content of the list (by identifying
the # character) instead of using the name of the list.
I adapted download_mirror.sh.
Change-Id: I041fc9c704156215f06149e5b4c16cd92990e17c
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
This change improves support for organizational or personal local
mirrors. Download tools work in a wider variety of environmnets,
and the population of build environments is now driven by .lst
files.
The download_mirror.sh (and related files) now have options to
better support running in non-containerized environments. This
includes options to run using local yum.conf files (rather than
system files) and not executing sudo or chgrp commands. These
options are enabled by flags. Default behaviour remains
unchanged.
The generate-cgcs-centos-repo.sh tool now populates build
environment as driven by lst files. This allows a mirror to be
maintained (and grown) at an organizational level, while allowing
a developer to produce exactly the environment they need as per
the lst files they have checked out.
generate-cgcs-centos-repo.sh now also populates the downloads
directory by invoking the populate_downloads.sh script (also
driven by lst files)
Change-Id: I8aae32c776df7618a5ef545a0ce43903b41112d0
Story: 2003422
Task: 24586
Signed-off-by: jmckenna <jason.mckenna@windriver.com>
Benefit:
- Confirm repos are working as expected.
- Discover origin of a package for security/risk assesment purposes.
- Determine if a repo is no longer providing packages and can be retired.
URL's will be logged in:
output/3rd-centos_rpms_urls_L1.txt
output/3rd-centos_srpms_urls_L1.txt
output/3rd_rpms_urls_L1.txt
output/3rd_srpms_urls_L1.txt
output/centos_rpms_urls_L1.txt
output/centos_srpms_urls_L1.txt
URL's for packages that are already available and do not require
re-download are not logged.
Story: 2003155
Task: 23290
Change-Id: I069f688241296b71e32054b00d5d1d34af22f202
Signed-off-by: Scott Little <scott.little@windriver.com>
For the mirror creation:
3rd packages in rpms_from_3rd_parties.lst file are downloaded
through urls with wget tool. Mostly times, some 3rd packages are missed
because the url download changes or the package is remove on the server.
Some servers that provide 3rd packages also works as Yum repository;
Taking advantage of this, Yumdownloader can handle the 3rd downloads
by adding these serves from rpms_from_3rd_parties.lst as Yum repos.
This have many advantages, for example:
if package A is download from server A is not available,
yumdownloader automatly can search the Package A the other servers.
In this way, the url is not required to download the 3rd package.
The file rpms_from_centos_3rd_parties.lst contains the name packages
that can be downloaded from Yum repos.
Change-Id: If1581b718e64b13ff6fd1484dd6af7b3889abb61
Signed-off-by: Felipe de Jesus Ruiz Garcia <felipe.de.jesus.ruiz.garcia@intel.com>
This includes Docker containers to perform the StarlingX build
and a set of scripts to maintain a local mirror of binary CentOS
and other packages required to populate the final ISO file.
Change-Id: I8140fd8fa2d00e7aa98c895a8e4962ab3748669d
Scott Little