Commit Graph

42 Commits

Author SHA1 Message Date
Scott Little 6f3d1e5dd2 cengn reference removal - centos
mirror.starlingx.cengn.ca no longer exists. CENGN is kindly forwarding
requests to the new location mirror.starlingx.windriver.com for now, but
that will only last a few months. We need to replace all the references
with the new URL.

I will also remove as many 'cengn' references as possible, replacing
them with 'stx_mirror'

Partial-Bug: 2033555
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I09e3f564edef2049786c965a86dbcaacac359801
2023-11-07 11:23:12 -05:00
ChantYuCN 34247fdb36 fix: Add the variable no_proxy into dockerfile
If set a http_proxy in container, it should add a no_proxy. Otherwise, system can't curl 127.0.0.1:443

Closes-Bug: #1979174

Signed-off-by: ChantYuCN <chengde.yu@intel.com>
Change-Id: I0189171df6d955ffccd62d358ce38e556394f6ce
2022-06-20 06:50:37 +00:00
Davlet Panech eec9163e0e Dockerfile: install tox in its own virtualenv
Fix multiple problems with python modules that cause build errors.

* Avoid replacing RPM-owned python modules with pip:
  - Remove python-tox: installed via pip later in Dockerfile
  - Remove python-testrepository: installed via pip later in Dockerfile
    (required by pbr)
  - Add python-virtualenv: was pulled in by one of the removed packages
    above

* builder-constraints.txt: used only for global packages:
  - Move all version constraints of Dockerfile into constraints file
  - Remove filelock and platformdirs packages (required by tox -- see
    below).
  - git-review: downgrade to 1.28.0, latest official version compatible
    with python 2.7

* Install a sane python 2.7 virtual environment that doesn't conflict
  with RPM modules, that includes tox. Create a symlink to tox in
  /usr/bin/. Uses a separate contraints file.

* builder-opt-py27-constraints.txt: new file for the virtualenv in /opt:
  - tox==3.23.0
  - Remove "filelock" and "platformdirs" packages formerly in the
    original constraints file. They resolve correctly by the tox
    requirement. Note that this downgrades the packages slightly compared
    to the explicit requirements, back to the latest official versions
    compatible with python 2.7.

    See revisions starlingx/tools revisions:
        0d67f81bdf
        7bde482bcb

Closes-Bug: 1960675
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I7e3e7e4f4afa52a614cb9f58b2d6172441ea7bc1
2022-02-11 17:00:28 -05:00
Scott Little 42ed522880 Update ca-certificates to pull from Centos 7.9
https download fails with errors such as...

00:21:13.479 ERROR: cannot verify opendev.org's certificate, issued by
‘/C=US/O=Let's Encrypt/CN=R3’:
00:21:13.479   Issued certificate has expired.

The upstream root certificate expire on Oct 1, 2021.

ca-certificates needs to be updated to version 2021.2.50
but that version is only available in Centos 7.9 or newer.
This build container is locked down to Centos 7.8 without
updates.

This update adds repo definitions for 7.9 and 7.8 updates,
but sets them to disabled by default.

When updating ca-certificates I temporarily enable the
Centos 7.9 updates repo.

I also disable the yum module that causes the
"system is not registered with an entitlement server"
messages, but those messages turned out to NOT be the root
cause of ca-certificates failure to update.

Closes-bug: 1946151
Closes-bug: 1946122
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I7ac47582e731ec5003f76ddfa24331ca87e1bf6c
2021-10-06 12:01:27 -04:00
Scott Little 874f70f91f Update ca-certificates in various docker build environments
Partial-bug: 1946122
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I82f4cdb8cf65aeae7e2b0e9f382c03cd53a40da4
2021-10-05 12:08:43 -04:00
Davlet Panech a85cf1cbac Dockerfile: pin testtools==2.4.0
Latest python package testtools, 2.5.0 is not compatible with Python 2.
Use the previous version in Dockerfile

Change-Id: Ie4143dd575b34f42d72f4962f7c0e40437a344fc
Partial-Bug: 1936832
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2021-07-19 11:09:57 -04:00
Scott Little ac05493480 fix git-review within docker build environment
'tb create' fails to create a build environment since
upstream git-review was updated of Apr 26.

Fix is to install/update pbr ahead of git-review.

Also, to reduce the likelyhood of this recurring, lock
down specific versions of the pypi supplied tools we
know to work.

Closes-bug: 1927137
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Ib9fe6fd33de4d637f254ac421cc0427ee6131b65
2021-05-04 12:50:55 -04:00
Scott Little f1010717c7 fix for tb.sh dies on rmdir /var/lib/mock
tb.sh create might fail to create the builder docker image.
Yum install of the mock package failed, but yum did not report
the failure because other packages in the instalation set succeeded.
A subsequent command in the dockerfile fails when it tries to
remove/relocate /var/lib/mock, but failes because it is not present.

The yum error reporting was corrected in a recent update.
But this does not address cached copies of old and broken
yum install steps that pre-date the fix.

The mock package is paricularly sensitive as it has cengn as
the only source, where as other packages have multiple sources.

One option is to force docker to not use the cache at all, which
is slow.

The second option is to change the docker file, placing the
yum command to install mock under a seperate docker RUN command.
The altered build instructions ensure that the docker cache
with the broken install can't be used.  While we are at it,
move the user/project customization steps as far down as possible
to improve cache usage.

This change implements both.

Closes-Bug: 1917901
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I28041bb44af53384c00a750b7162c6c6808c4e2d
2021-03-10 15:06:53 -05:00
Davlet Panech 511859ef18 Dockerfile: fail in "yum install" on missing packages
By default "yum install" ignores packages that can't be downloaded and
return 0 to the shell.

In a Dockerfile such falsely successful commands are cached by
"docker build", so that the next time we run "docker build" the
"yum install" is not even attempted:

  # this "succeeds" and gets cached during the first "docker build",
  # even if the URL returns an error (eg DNS error or HTTP 404).
  # During a second "docker build" this bringis back the FS layer
  # from the cache and doesn't attempt to re-download the package
  # from that URL
  RUN yum install python3 http://some/url.rpm
  ...

This patch avoids the problem in case CENGN mirror is down.

Change-Id: I65f2185ce8a1832f3fc1be4135b0a49653654b5b
Closes-Bug: 1912682
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2021-01-28 10:49:59 -05:00
Davlet Panech 1f3f3e08bc Update CentOS vault URL to HTTPS
CentOS vault HTTP url redirects to HTTPS, but yum displays misleading
error messages when the secondary HTTPS request fails.
Use HTTPS directly.

Change-Id: I687c1de2378de11abb5ad981bc73b66d8c40ba2a
Closes-Bug: 1908088
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2020-12-14 11:32:12 -05:00
slittle1 d31f56dd75 Update git and git-review in Dockerfile
Opendev recently upgraded to a newer version of gerrit.
That gerrit demands that we run a newer version of git than
is available from Centos 7.  This will prevent users from
posting code for review from within this build container.

This update compiles a newer version (2.29.2) of git from
downloaded source code, and adds it to the build image.

A newer version of git-review is downloaded from pypi,
and added to the build image.

A mechanism is provided to feed in a valid email address
to the idocker image's ~/.gitconfig by defining
MY_EMAIL=you@your-email-host in your 'localrc' prior to 'tb.sh create'

Closes-bug: 1905789
Change-Id: I28ed185ea477ba3a8e47e1dd11fd384816f72717
Signed-off-by: slittle1 <scott.little@windriver.com>
2020-11-27 13:15:08 -05:00
Scott Little ec588cd408 Dockerfile: Centos 7.8 has moved to vault
Centos 7.8 has relocated
from mirror.centos.org/centos/7.8.2003/
to vault.centos.org/centos/7.8.2003/ .

This will cause 'tb.sh create' to fail.

Pre-existing docker build environments should
continue to work.

This update adjusts the url within the dockerfile
to reflect the new location.

Closes-bug: 1904785
Change-Id: I0f44ddfc0d126c738e5f570ea36e463720ccf6bf
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-11-18 15:22:31 -05:00
Scott Little 420a989cf5 Add backward compatability wrapper for generate-cgcs-centos-repo.sh
generate-cgcs-centos-repo.sh was renamed generate-centos-repo.sh
in a prior update.  This commit restores the old command
as a wrapper for the renamed command.  This will provide
backward compatability for a time.

Story: 2006387
Task: 36912
Depends-On: https://review.opendev.org/#/c/687403
Change-Id: I69714d46ab20b52828f31625cc4e7955fbd10387
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-18 22:32:42 -04:00
Scott Little b827624e16 rename cgcs-centos-repo to centos-repo
Story: 2006387
Task: 36912
Depends-On: https://review.opendev.org/749974
Depends-On: https://review.opendev.org/750041
Change-Id: I745a822e88aca68c207a92f14fb5370d6e3d0050
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-18 22:32:41 -04:00
Scott Little 9a188af410 Add backward compatability wrapper for generate-cgcs-tis-repo
generate-cgcs-tis-repo was renamed generate-local-repo.sh
in a prior update.  This commit restores the old command
as a wrapper for the renamed command.  This will provide
backward compatability for a time.

Story: 2006387
Task: 36910
Depends-On: https://review.opendev.org/#/c/687401
Change-Id: Ibe7c70b1b3f1081e3c7c8a1ca775927322653a5b
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-18 22:32:41 -04:00
Saul Wold 4e73613a36 Rename cgcs-tis-repo to local-repo
Both the repo and scripts need to be updated for this rename
There is a corresponding change needed to the starlingx/root
repo.

This is tested with a clean build and rebuild to ensure RPMs
are found correctly.

Story: 2006387
Task: 36910
Depends-On: https://review.opendev.org/749974
Depends-On: https://review.opendev.org/750042
Change-Id: I997728f3283077365555eeb9bdbe028ccac255f1
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-18 22:32:41 -04:00
Scott Little 922ebeb15a Add python2-ruamel-yaml to docker file in support of helm chart build
Partial-Bug: 1893261
Change-Id: I690af011a01fd409d7b4fdcc9f439635dffd31ce
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-09 12:06:04 -04:00
Davlet Panech aa23b041b1 Dockerfile: add mirror repos to global yum config
Copy mirror repo files to /etc/yum.repos.d because tb.sh needs them.
This change fixes a problem introduced by

  Commit: 78263e8784
  Dockerfile: pin centos & epel repo versions

that caused centos-mirror-tools/download_mirror.sh to fail when executed
from a docker container created by tb.sh -- "rpm not found" for most
packages being downloaded.

This is because download_mirror.sh without additional arguments expects
toCOPY/yum.repos.d/ files to have been copied to /etc/yum.repos.d/ in the
docker container by default.

Change-Id: If214aea6560632837d0f196c21801fc2d06c8259
Closes-Bug: 1885624
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2020-06-29 16:23:02 -04:00
Davlet Panech 78263e8784 Dockerfile: pin centos & epel repo versions
Avoid different outcomes every time we run "docker build":

- Use centos 7.8 base image
- Use fixed versions of CentOS & EPEL repos
- Pin mock & its dependencies to 1.4.16
- Don't install unnecessary packages
- Don't install unnecessary/conflicting repo files

Change-Id: Ic9a52f373809ed66a0e61675f4d6276fd62e6fa0
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2020-06-19 15:19:14 -04:00
Scott Little f6755e179a Enable build of docker images, charts and wheels from a layered build
1) Add config for the container layer.

2) Add support for the new config file 'required_layer_wheel_inc.cfg',
Which will specify the default urls for lower layer wheels.inc files.
These files must be downloaded by download_mirror.sh, and linked by
generate-cgcs-centos-repo.sh.

3) Add options to download_mirror.sh and generate-cgcs-centos-repo.sh
allowing overrides or additions to the set of url's for lower layer
wheels.inc

4) Changes to allow build scripts running within a docker container
to launch a sister (not daughter) container.  This will allow
build-wheels to run within a container.

Story: 2006166
Task: 38980
Depends-On: https://review.opendev.org/711773
Depends-On: https://review.opendev.org/717755
Change-Id: I95c71a0327dc9d919ed835d3661a6f1ec7ce30c7
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-04-14 22:00:52 -04:00
Zuul 6279b3f773 Merge "Remove TC reference from RELEASE Variable and config scripts" 2020-03-09 21:02:28 +00:00
Saul Wold 6226422aee Remove TC reference from RELEASE Variable and config scripts
This changes a brand specific to be generic for configuration

Story: 2006387
Task: 36914
Change-Id: I826498cfc58b4e02fcdf4590a5cbfc25d38f7fc9
Signed-off-by: Saul Wold <sgw@linux.intel.com>
2020-02-28 10:44:47 -08:00
Dongqi Chen daf360c213 Update Dockerfile to resolve "--edit" for kernel-4.18
Install bison, flex rpms to resolve the
"bison, flex command cannot found" build error
when executing "build-pkgs --edit kernel"

Story: 2007308
Task: 38886

Change-Id: I5ca69730bc0d0c680438704b42ba3bbb413de420
Signed-off-by: Dongqi Chen <chen.dq@neusoft.com>
2020-02-28 18:06:22 +08:00
SidneyAn aee9a5c175 support build srpms with python3
install rpm python3-devel to support build srpm
with python3

Change-Id: I153dd699311a7c1e66d8dd896c2c2ffa4fdec7f1
Story: 2007106
Task: 38873
Signed-off-by: SidneyAn <ran1.an@intel.com>
2020-02-26 15:26:58 +08:00
Scott Little d51f8050cd Build layering, script and lst update
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.

Current .lst and yum files under directory centos-mirror-tools
are relocated.  Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .

Lst entries for compilable content have been relocated to other
git repos by prior updates.  i.e.  those that list tarballs or
srpms to be compiled within that repo.

The original .lst files are deleted to make it easier
to identify new content during development.

Layer 'all' builds all layers in a single workspace.  The
lst files are identical to current content, minus the src.rpm and
tarball entries.

Other layers get only a subset of packages download. The minimum
required to build the layer.  The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.

An upper layer does not need to list rpms known to be provided by
a lower layer.  Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.

A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds.  These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.

Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso.  The transitive list of required rpms need not be listed.

Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.

To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.

The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git.  Substitute urls can use the file:///
syntax.  Just be sure to remove these changes before submitting.

Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.

The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.

e.g. For simplicity I'll only list the 'extra' arguements

download_mirror.sh -C <my-config-dir> \
                   -l <layer> \
                   ...

generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
                             --layer=<layer> \
                             ...

populate_downloads.sh --config-dir=<my-config-dir> \
                      --layer=<layer> \
                      ...

These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.

e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>

The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file.  Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.

e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso.  For simplicity I'll only list the 'extra' arguments

./download_mirror.sh -l flock \
    -L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    -I distro,std,file:///<my-distro-workspace>/rt/image.inc \
    ...

generate-cgcs-centos-repo.sh --layer=flock \
    --layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    --layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
    ...

NOTE: The triplet syntax for a package list url is
    <lower-layer>,<build-type>,<url-to-rpm.lst>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'rt', 'installer'

    Also if 'file:///' syntax is used, a matching change is made to
    the yum *.repo file.  This assumes that the rpm.lst is co-resident with
    repodata directory, as is the norm for our build outputs.

NOTE: The triplet syntax for a image inc url is
    <lower-layer>,<include-type>,<url-to-image.inc>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'dev'

A typical user is likely only working in the flock layer on the master
branch.  He should be content to use the compiler and distro layer
outputs from cengn.

His workflow looks like ...

1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.

1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Only a cross-layer developer should setup two or three copies of the
building environment, one per layer.  We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,

Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...

$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
 ... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...

$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh

Story: 2006166
Task: 37103

Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-02-10 10:45:40 -05:00
Don Penney 12c3dc4772 Constrain more-itertools for build failure
A recent update to the more-itertools python module causes a failure
in the STX_BUILD_container_setup CENGN build job. This module dropped
python-2.7 support after its 5.0.0 release. The newest update causes a
failure due to code that does not work with 2.7.

This commit adds a builder-constraints.txt file that the Dockerfile
passes to the pip install command to constrain module versions during
setup of the build container, allowing us to constrain the
more-itertools version to 5.0.0, the last version to officially
support python 2.7.

Change-Id: I3432c204ecd7c4ddedd8a7dea14216d4ec31e0aa
Closes-Bug: 1859642
Signed-off-by: Don Penney <don.penney@windriver.com>
2020-01-14 11:20:50 -05:00
Saul Wold 73c1d66677 Rename cgcs_overlay to mock_overlay
This helps to debrand (remove cgcs) from the tools area

Story: 2006387
Task: 36208
Change-Id: I357837a710207f8cec6409d1de99f443f8c50c84
Signed-off-by: Saul Wold <sgw@linux.intel.com>
2019-10-08 12:32:50 -07:00
Saul Wold 2a4eec0305 add additional packages needed by new tool
The new tool requires additional host tools for the script to run
correctly, so update the install list.

Story: 2006595
Task: 36831
Change-Id: Id49be9f28862dae33a5659cdfe2d1aac39c74540
Signed-off-by: Saul Wold <sgw@linux.intel.com>
2019-09-26 11:24:43 -07:00
Scott Little de2ee28404 Docker build environment does not exit cleanly.
Our docker builder runs services under systemd.
Systemd requires SIGRTMIN+3 to trigger a clean exit.
The 'docker stop' command will send SIGTERM by default,
which systemd ignores.

Failure to exit cleanly results in mounts being left in
place, and ultimately the container will get stuck in
'Removal in progress' state, and a docker image that
can't be removed.

Change-Id: I28edb021a4f9c8941a7be57bfbb2ffa34b15ccd7
Closes-Bug: 1833467
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-06-20 11:10:09 -04:00
Scott Little 67e2d3e4c0 Dockerfile: relocate "yum install golang"
Dockerfile command "RUN yum install -y golang" failed with a
404 error on ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64.

In the Dockerfile, we are trying to run....
   RUN yum install -y golang
after we install StarlingX repos into /etc/yum.repos.d/ with
   COPY centos-mirror-tools/yum.repos.d/* /etc/yum.repos.d/

All 'yum install' commands must preceed any modifications to
/etc/yum.repos.d in support of StarlingX package downloading!

I'll also add 'skip_if_unavailable' to all StarlingX repos
in case other build scripts introduce a 'yum install'.

NOTE: Package downloading itself will try CENGN's mirror in
addition to the primary site.  So unavailability of a single
site isn't fatal in that case.

Change-Id: If0220d3d5742810b0927f8e55e0093f0f0172c59
Closes-Bug: 1833444
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-06-19 16:52:50 -04:00
Scott Little d351bd4b3d build-helm-charts.sh now requires python-yaml
A recent change, https://review.opendev.org/#/c/662087/ ,
added 'import yaml' to build-helm-charts.sh.  However
the docker build environmnet does not include python-yaml
in it's Dockerfile.

This update adds the missing python-yaml to the Dockerfile

Change-Id: Ia446bc587efb4b1997008661a4b584183891bb4f
Closes-Bug: 1832017
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-06-07 12:09:01 -04:00
Scott Little d9eec60061 Ensure that python36-psutil is present in docker build environment
mockchain-parallel imports psutil.  However the version of python
to be used for mockchain-parallel is set by mock, and could be
2.7 or 3.6.  The mock package itself doesn't require psutil so
we must import it ourself, and make sure both are present.

Change-Id: I7c5d9b52eed49cdce02224603b9bfd17e4426558
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-06-06 10:52:55 -04:00
Mariana Gonzalez 44ce0d24e9 Remove unnecessary devel packages from Dockerfile
Story:2005075
Task:29650

Change-Id: I8f2886f9b45a2ef2378f1195d60246ade1c97dd4
Signed-off-by: Mariana Gonzalez <mariana.gonzalez.velarde@intel.com>
2019-04-26 10:42:41 -05:00
Zuul 4883bee901 Merge "fix build-pkgs fail to edit tpm2-tools srpm" 2019-04-18 14:45:52 +00:00
Shuicheng Lin 5040ba46da fix build-pkgs fail to edit tpm2-tools srpm
The error message is "configure.ac:77: error: possibly undefined macro:
AC_MSG_ERROR". Add package autoconf-archive to fix it.

Closes-Bug: 1825287

Change-Id: Ia7497f5ea79f314ece8e39df8eaaafa902337e22
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
2019-04-18 13:19:12 +08:00
Shuicheng Lin 2878fbb578 fix build-pkgs fail to edit python-eventlet srpm
The error message is “/bin/sh: sphinx-build: command not found”.
Add python-sphinx package to fix it.

Closes-Bug: 1824932

Change-Id: Ia07421a44d9b890148f70625b4b06eea84b4a7b9
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
2019-04-18 10:35:31 +08:00
Erich Cordoba e75bb2e7c3 Add notices on Intel authored files.
Story: 2005265
Task: 30095

Change-Id: I9eecbe7c0a15eec2ffe1e43964cd4253fc04b91c
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
2019-03-20 10:21:32 -06:00
Scott Little cf435e6c2a download_mirror.sh fails with 404 errors
The 'yum makecache' step is failing. The issue is seen when the fast
mirror plugin for yum is in use (default).

The metadata for a yum repo as scattered across several files. The
master file is repomd.xml, and it may list several supporting files,
e.g.
   filelists.xml.gz
   primary.xml.gz
   filelists.sqlite.bz2
   other.sqlite.bz2
   other.xml.gz
   primary.sqlite.bz2

When fast mirror plugin is in use, each file might download from a
separate source. This opens the door for a race condition when a repo
update is rolling out across the mirrors.

The error suggests we have already downloaded a repomd.xml, but it is
obsolete and references a supporting file that no longer exists.

A second possible source of inconsistent data is caching proxies.

Solution:
1) Disabling the yum fast cache plugin is undesirable, and there
is no option forcing metadata iof a given repo to be single sourced.
So this update adds a limited number of retries for the
'yum makecache' command.

So far we've never seen 'yum makecache' fail twice in a row... 
largely because the 'fastest mirror' plugin rarely downloaded from the
same source twice.

2) Add 'http_caching=packages' to yum.conf, which asks upstream proxies
to not cache repodata.

Change-Id: I3fa3f61e38d868fb14f4175b87c0d323abb57def
Closes-bug: 1818911
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-03-13 14:32:03 +00:00
Felipe de Jesus Ruiz Garcia 2a6a96302f Add proxy env comments to Dockerfile
Behind a VPN network proxy must be required for
yum daemon.

Change-Id: I963f76c8f7977bfffd2a389f773c86cd52214ec9
Signed-off-by: Felipe de Jesus Ruiz Garcia <felipe.de.jesus.ruiz.garcia@intel.com>
2018-11-21 16:50:18 -06:00
Jerry Sun e0aeb488a4 Add dep for golang in the build container
We would like to have dep for golang as part of the build to pull in
dependencies. This allows us to avoid checking in dependency package
in our code base and to pull in the dependencies when building.

Story:2002840

Change-Id: Ifff8dab38aebdc5638909679b7bef8128c6a84a7
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
2018-11-15 14:20:25 -05:00
slin14 e7994bec73 fix build-srpms edit failure in openssh/openldap
It is due to these packages need autoreconf support.
Add required package automake/autogen/libtool to solve it.

Closes-Bug: 1799160

Change-Id: I5c6c03a6b74e168d73f0fb054bd5ccab367e3a94
Signed-off-by: slin14 <shuicheng.lin@intel.com>
2018-11-13 23:59:33 +08:00
Mario Alfredo Carrillo Arevalo 4b4fc873b5 Add unified dockerfile
The objective of this commit is to unify the customization of
the follow docker images in one file.
- Dockerfile.centos73
- Dockerfile.centos73.TC-builder
- centos-mirror-tools/Dockerfile

This is order to create a generic StarlingX docker image, this image
should be able to handle all building process.

Furthermore this unification will reduce the instructions of the
installation guide for our end users.

Story: 2003712
Task: 226370

Change-Id: Id9b405743763be2e0d7aa37c8070c6788bc099c3
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
2018-09-13 10:56:11 -05:00