Helm plugin mapkubeapis needs to be installed to help application user
deal with deprecated kubernetes resources apis.
The plugin tarball must be added to the exceptions of dl_tarball as
the script by default untars with `--strip-components 1`, thus removing
the plugin binary before recompressing
Story: 2009138
Task: 46022
Signed-off-by: Lucas Cavalcante <lucasmedeiros.cavalcante@windriver.com>
Change-Id: I4aac5b865bd1c4db519717135fbaa81ea2bda6cb
After download_mirror.sh, system download kernel-rt and centos-git-common.
However, one important script, "get_sources.sh", is not executable.
It will cause built error.
this patch will change the file mode to fix this problem.
Closes-Bug: #1979341
Signed-off-by: ChantYuCN <chengde.yu@intel.com>
Change-Id: I136b92ebc943b2a9a8c6ad9cb85fe802acce3c09
Helm plugin 2to3 needs to be installed to upgrade apps to FluxCD.
The plugin tarball must be added to the exceptions of dl_tarball as
the script by default untars with `--strip-components 1`, thus removing
the plugin binary before recompressing
Story: 2009138
Task: 45584
Signed-off-by: Lucas Cavalcante <lucasmedeiros.cavalcante@windriver.com>
Change-Id: I19b541500fd0872660c98e2f20baccf3f52c77da
ICE driver update to support the Intel Logan Beach NIC (E810-CQDA2T).
and that dictates the upgrade of ddp firmware to ice_comms-1.3.35.0,
which is included in 27_1.zip.
https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html
Remove the extra extraction step because it is zip file that we
want, rather than zips of zips.
Verification:
Success Path
- run download_mirror.sh, verify that the ice_comms zip file is
downloaded in downloads folder.
Story: 2009952
Task: 44895
Signed-off-by: Jiping Ma <jiping.ma2@windriver.com>
Change-Id: I7561543ba0bce5d2fd82a0315f81ba3dc3f1ba0b
This commit adds support for downloading Broadcom's NetXtreme-E
driver/library tar archive file. The tar file consists of multiple
nested archives and source RPM files. We are only interested in the
libbnxt_re Infiniband verbs library SRPM and the bnxt_en/bnxt_re kernel
driver source code archive:
- libbnxt_re-220.0.5.0-rhel7u9.src.rpm
- netxtreme-bnxt_en-1.10.2-220.0.13.0.tar.gz
If the archive has already been downloaded, the sha256sum of the archive
is verified. (The checksum is stored in the utility/"util" field in the
listing file.) If not already downloaded, then the archive is downloaded
and the sha256sum of the archive is checked. Finally, the desired files
are extracted from the main tar archive.
(Checksum verification is added as a package-specific behaviour, because
this feature does not exist in the build system, and we would like to be
aware in case the software package is modified.)
Testing:
- dl_tarball.sh correctly removes a pre-existing tar archive with an
incorrect sha256sum and re-downloads the tar archive.
- If the tar archive does not already exist, then the archive is
correctly downloaded and the sha256sum is correctly checked.
- If the download_file function or the check_sha256sum function fails,
then the shell script correctly reports an error message and
continues. (This was verified with shell script instrumentation.)
- If the tar archive extraction fails or if the expected/desired files
cannot be found, an error is correctly reported by the shell script,
and the tar archive is removed to allow follow-up attempts to try
again. (Also verified with instrumentation.)
Story: 2009915
Task: 44761
Change-Id: Id021a33e7f26643139d6ef0dda5c7146cfb7f172
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
dl_tarball.sh assumes an el8 kernel, and lacks of error
handling. This may result in an attempted download of a
non-el8 reporting success, but producing a src.rpm containing
an el8 kernel instead.
Testing:
- Download current el8 rt kernel
- Download previous el7 rt kernel
- DL using invalid URL - error caught
- DL using invalid sha - error caught
Closes-Bug: 1964156
Change-Id: I5ca5130a7f1c4e967c876060d95f9d6a2c2e8cf5
Signed-off-by: Scott Little <scott.little@windriver.com>
Copy the following packages to downloads folder, the ones are included
in MLNX_OFED_SRC-5.5-1.0.3.2.tgz.
mlnx-ofa_kernel-5.5-OFED.5.5.1.0.3.1.src.rpm
rdma-core-55mlnx37-1.55103.src.rpm
mlnx-tools-5.2.0-0.55103.src.rpm
mstflint-4.16.0-1.55103.src.rpm
Removed the original mlnx-ofa_kernel-5.3-OFED.5.3.1.0.0.1.src.rpm from
rpms_3rdparties.lst, that added in commit
e8d164e801e3fec796fa24b2eaf90f50726c38a(Prep: Upgrade kernel
5.10 related packages).
Story: 2009878
Task: 44610
CentOs Test:
- PASS: Run download_mirror.sh that can download all mlnx related
packages.
- PASS: Build image
- PASS: Boot image
Signed-off-by: Jiping Ma <jiping.ma2@windriver.com>
Change-Id: I67e058f9c9bdab31e87d46daf0dec3b353dc68a6
This is a firmware package for Intel Ice NIC hardware,
addressing dynamic device personalization for the
communications market.
Intel likes their zips of zips, causing us to have to
perform an extra extraction step to get the zip file
that we want.
Verification:
Success Path
- create a test .lst file with just the entry for the
intel ice comms ddp firmware in it
- run the dl_tarball.sh script against the test .lst file
- verify that the ice_comms zip file is successfully
produced in the proper directory
- verify that the wrapper zip file is also left in the
proper directory
Failure Path
- modify test .lst so that the url to the wrapper zip
is no longer valid
- verify that the script reports the download error
properly and handles it properly
Story: 2009823
Task: 44410
Change-Id: I2f8229d9a67ca986c4ad97049e8abf141cd07895
Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>
Mirror scripts sometimes leave corrupted/partial files behind.
Problems
========
1) wget is called with the -O flag, and the server returns an HTTP
error for the requested URL (404 etc). Wget leaves a zero-length file
behind. This doesn't seem to happen without the -O flag.
2) wget starts the download which stalls & times out half-way; wget
gives up and requests the same file with a byte offset of the form
"Range: bytes=1234-", and the web server doesn't support open-ended
ranges. In this case wget prints out a warning, leaves a partial file
behind and returns success.
3) Sites like GitHub generate repo tarballs on the fly, eg:
https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.19.3.tar.gz
Since tags can move, downloading such a file twice may result in a
different file. Therefore HTTP "resume download" may corrupt files in
this case.
4) Git "keyword expansion" feature may result in differences in source
files being downloaded. For example, this file:
https://github.com/kubernetes/kubernetes/blob/v1.19.3/staging/src/k8s.io/component-base/version/base.go
contains lines similar to:
gitVersion = "v0.0.0-master+$Format:%h$"
where %h is replaced with a short SHA when the tar file is
exported/downloaded. How short the SHA is depends on git history and
sometimes results in shortened SHAs of different lengths. So
downloading that file may result in different files.
Therefore HTTP "Range" header may corrupt files in this case as
well.
5) Curl is invoked with the "--retry" option and starts the download;
connection stalls; curl gives up, connects again, skips the 1st N
bytes and appends to the partial file. If the file changes while we
are doing this, it will end up corrupting the file. This is very
unlikely to happen and I haven't been able to reproduce this case.
Problems with HTTP Range header
===============================
Curl/wget "resume/continue download" feature has no way of verifying
whether the partial file on disk, and the one being re-requested, are in
fact the same file. If the file changes on the server between
downloads, "resume download" will corrupt it.
Some web servers don't support this at all, which triggers case (2)
with wget.
Some web servers support the Range header, but require that the end
byte position is present. This is not compatible with wget & curl.
For example curl & wget add headers similar to: "Range: bytes=1234-"
means give me the file starting at offset 1234 and till EOF. This also
triggers case (2).
This patch
==========
* Always download the file to a temporary name, then rename into place
* Use curl instead wget (better error handling). The only exception is
"recursive downloads", which curl doesn't support.
Bug: https://bugs.launchpad.net/starlingx/+bug/1950017
Change-Id: Iaa89009ce23efe5b73ecb8163556ce6db932028b
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
In preparation of updating kexec-tools for the v5.10 kernel, this
commit:
- Adds the build dependencies of kexec-tools to the distro layer,
- Removes the kexec source RPM file reference from the flock layer,
given that kexec-tools will now be built in the distro layer, and
- Modifies the dl_tarball.sh script so that the kexec-tools source RPM
is downloaded and the "kdump-anaconda-addon-003-29-g4c517c5.tar.gz"
source code archive is extracted from the source RPM file.
This is done because this tarball does not appear to be publicly
available otherwise. The only reference to the commit ID 4c517c5 is on
a Red Hat employee's personal Github account at:
https://github.com/ryncsn/kdump-anaconda-addon/commits/rhel-7
Verification: kexec-tools was successfully built with this patch in a
monolithic StarlingX build environment. A potential failure of wget was
found to be gracefully handled by inserting a "false" command after the
line where wget is called, and, in a separate experiment, by replacing
the wget command line with "false". (Thanks to Jim Somerville for
pointing out the buggy failure path in an earlier revision of this
patch.)
Story: 2008921
Task: 43040
Signed-off-by: Jiping Ma <jiping.ma2@windriver.com>
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
Change-Id: Id8df3435d8660070cce31b9f994b386b020ed6e6
Commit 0e8d164e80,
with subject "Prep: Upgrade kernel 5.10 related packages",
omitted a needed quote.
Partial-Story: 2008921
Task: 42590
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I0358c44f941f6bf5acdaf01ac80a8b7ab2913f6f
out of tree kernel modules need to be upgraded with kernel 5.10.
mlnx-ofa_kernel-5.3-OFED.5.3.1.0.0.1.src.rpm,
QAT1.7.L.4.14.0-00031.tar.gz,
dpdk-kmods-2a9f0f72a2d926382634cf8f1de10e1acf57542b.tar.gz,
OPAE_1.3.7-5_el7.zip that included
opae-intel-fpga-driver-2.0.1-10.src.rpm.
need to be downloaded, so modify dl_tarball.sh to support it.
This commit makes it possible to download newer versions in preparation
for switching over to using them later.
Previously, mlnx-ofa_kernel has been done as a tarball with the .lst
file in the kernel repo. We now take the source as an rpm, and feel
that it is better done via the rpms_3rdparties.lst file, so we move
it there.
Partial-Story: 2008921
Task: 42590
Signed-off-by: Jiping Ma <jiping.ma2@windriver.com>
Change-Id: Ie81ed225c6912bbdc046c90b86abddcec69c33da
The records of prior download failures are never being cleared
from log_download_tarball_missing.txt.
log_download_tarball_missing.txt needs to be cleared at the start of
each new download attempt.
Closes-bug: 1928131
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Id38b49f8d7d8ab7c31e54d70fad097dd806f2dd5
StarlingX has moved beyond the OpenStack pike release and will
be following master and the latest release, so don't include
OpenStack release names.
This also removes the r1 reference from the download directory
path.
Story: 2006835
Task: 37410
Change-Id: I0c548e9c97f2fb162394d0c381f02a40072e7400
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Subversion 2.1.8.1
Includes the associated components, namely
libibverbs and rdma-core.
Passed platform sanity tests on a Wind River lab equipped with
Mellanox cards.
Change-Id: Ie0dccef3a220246d547cc945c64f9aad357afbe0
Partial-Bug: 1887556
Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>
This enables chartmuseum binary v0.12.0 as a download target
instead of expecting a tarball. This is looking for the specific
tarball-dl.lst entry !chartmuseum-v0.12.0-amd64 .
The chartmuseum binary is required to build helm charts.
Change-Id: Iae7f3543313894310c85f6c55bb6d8f58cfb71ae
Story: 2007000
Task: 38893
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Now use the kernel built-in tpmdd
and integrity is no longer supported.
Change-Id: I64dd3db829978f2a7adea60b5049604d866f213b
Story: 2007308
Task: 39742
Depends-On: https://review.opendev.org/#/c/726999/
Signed-off-by: Dongqi Chen <chen.dq@neusoft.com>
It is to align the package name in srpm_path
Story: 2007308
Task: 39617
Change-Id: I106024a15c2916dbd25c86ecac73cd05d4c15dfc
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
The CentOS 4.18.0-147.3.1 kernel is not the same as upstream 4.18
kernel, and cause tpm/integrity from upstream cannot pass build.
There are many patches need be cherry-picked or reverted in order
to pass build with CentOS 8.1 kernel.
So switch to get tpm/integrity code from CentOS 8.1 kernel tarball.
With it, we just need focus on STX's patch.
Story: 2007308
Task: 38950
Depends-On: https://review.opendev.org/720325
Change-Id: I8090bc1aff7d5fe194567520093738099464b572
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
tarball download script is updated with the new srpm name.
Story: 2007308
Task: 38926
Depends-On: https://review.opendev.org/720325
Change-Id: Ie1caef3083b9c7ab6aca2873ceef2fd4a86ef4b0
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
This is for downloading the rt-setup tarball from CentOS
and then use it to create the SRPM via the CentOS common tooling.
This is needed as CentOS does not currently create a rt-setup.src.rpm
package.
We clone the CentOS rpm/rt-setup which contains the rt-setup.spec
and use the into_srpm.sh tool which then downloads the correct linux
tarball from CentOS and generates the .src.rpm. Since we use the rpm
name in the .lst file it will only be re-created if the rpm is not in
the downloads directory.
Signed-off-by: Yong Fu <fuyong@neusoft.com>
Story: 2007308
Task: 39558
Change-Id: Ie43ee5c6b345bf338ea63b9c2a8bfc2cb1430387
This is for downloading the linux-rt tarball from CentOS
and then use it to create the SRPM via the CentOS common tooling.
This is needed as CentOS does not currently create a kernel-rt.src.rpm
package.
We clone the CentOS rpm/kernel-rt which contains the kernel-rt.spec
and use the into_srpm.sh tool which then downloads the correct linux
tarball from CentOS and generates the .src.rpm. Since we use the rpm
name in the .lst file it will only be re-created if the rpm is not in
the downloads directory.
Story: 2007308
Task: 39570
Change-Id: Ieb0443fae4431b78173ef3fa42933b3d795a0816
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
The tarball downloader uses wget to retrieve content from the urls.
In some cases, this url doesn't point directly to a file in the server.
For cases like sourceforge, the url points to an API that will deliver
the requested file. This is problematic for wget because still gets a
200 HTTP reponse even if the is not there, and then downloads a text
file with the content of the requested page.
This fix proposes to check the mime type of the downloaded file and if
the file is not one of the used types the it will delete it.
Closes-Bug: 1795739
Change-Id: Ia9286ca094389d8d5196fce8ccda8a38cc754075
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
This reverts commit 40a9622d77.
The URL for tss2 is not always working and this causes an
incomplete mirror in a clean environment.
Closes-bug:1816813
Change-Id: I9c8fa64e1f0f9154dd45a67088fdc27f4b8f8313
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
tss2 is the only tarball that is generated using git clone, this
commit makes the downloader get all the tarballs using wget.
Change-Id: Ib6009341f6b7c890f43962f2c24fc73e95ec81b7
Story: 2004413
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
Previous StarlingX API Reference documentation was hosted under
restapi-doc package and managed using maven and wadl files.
StarlingX has enabled the OpenStack API guideline to manage REST API
services allowing all previous restapi-doc references to be removed.
Maven is part of the restapi-doc requirements.
Depends-On: https://review.openstack.org/#/c/609714/
Change-Id: I007f8c7b8f736ab5b6c0b712f350d350df14744f
Signed-off-by: Abraham Arce <abraham.arce.moreno@intel.com>
StarlingX needs to download a variety of rpms and tarballs
from various upstream sources. Unfortunately the upstream sources
are not always dependable. Either servers go down, are unreachable,
or drop older content that we still depend on.
Our proposed solution is to run our own mirror to capture an
independent copy of the content needed by StarlingX.
For this purpose, a server has been set up at
http://mirror.starlingx.cengn.ca/mirror/centos
The mirror will use deterministic paths derived from the upstream
urls. Scripts will be used to convert an upstream url to
the mirror's equivalent url (see function url_to_stx_mirror_url in
url_utils.sh)
The mirror will be refreshed daily. New .lst entries will be
processed at that time. Processing of changes under yum.repos.d
is not automated by this update. Expect a follow-up update to
address this issue soon. These scripts are found under the
'stx_mirror_scripts' subdirectory.
Changes are made to the download_mirror.sh script, and it's
supporting scripts. New arguments have been added to each
script to select the download source.
-s StarlingX mirror only
-S StarlingX mirror, with upstream source as backup
-u Upstream source only
-U Upstream source, with StarlingX mirror as backup
You do not need to provide any of these flags. Continue to
us download_mirror.sh as you always have. The default
behavior is currently set to '-S', i.e. first try the
StarlingX mirror, with upstream source as backup.
If this proves to place to heavy a load on the existing
server, we might switch the default to '-U', i.e. first
try the upstream source, with StarlingX mirror as backup.
If poor download performance is seen, you might want to try
explicitly adding -U as an argument.
The remaining two options are not recommended for regular use.
Upstream only, i.e. '-u', restores original behaviour, but
you may once again encounter rpms that have aged out, and
been removed from their original repos. StarlingX only,
i.e. '-s', is vulnerable if a .lst file has been updated,
but the mirror has not yet processed it.
Change-Id: I7e0f3d9fb99253662f9f4bf12457d39250408c0b
Story: 2003906
Task: 26785
Signed-off-by: Scott Little <scott.little@windriver.com>
EPEL7 rpms builds are visible at kojipkgs.fedoraproject.org
and often remain available there long after they age out
of the official repo.
Use kojipkgs.fedoraproject.org as a backup source for
packages when not found in the repo. Log seperately
so that these dropouts can be identified and corrected
in the lst files.
Also improved the tests for already downloaded packages
in several places to accelerate the test cycle.
Change-Id: Iaaa9fe05ac605a3604acf0048c571c6e88303692
Story: 2003157
Task: 23292
Signed-off-by: Scott Little <scott.little@windriver.com>
This change improves support for organizational or personal local
mirrors. Download tools work in a wider variety of environmnets,
and the population of build environments is now driven by .lst
files.
The download_mirror.sh (and related files) now have options to
better support running in non-containerized environments. This
includes options to run using local yum.conf files (rather than
system files) and not executing sudo or chgrp commands. These
options are enabled by flags. Default behaviour remains
unchanged.
The generate-cgcs-centos-repo.sh tool now populates build
environment as driven by lst files. This allows a mirror to be
maintained (and grown) at an organizational level, while allowing
a developer to produce exactly the environment they need as per
the lst files they have checked out.
generate-cgcs-centos-repo.sh now also populates the downloads
directory by invoking the populate_downloads.sh script (also
driven by lst files)
Change-Id: I8aae32c776df7618a5ef545a0ce43903b41112d0
Story: 2003422
Task: 24586
Signed-off-by: jmckenna <jason.mckenna@windriver.com>
This in order to keep consistency between downloader script names
(dl_rpms.sh, dl_other_from_centos_repo.sh and now dl_tarball.sh).
Change-Id: I50d029bced2db6c97039d82f08e1e120476e47d9
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
Lucas Cavalcante