From 1f196aa31057577ef13a496866532a28ffd68092 Mon Sep 17 00:00:00 2001 From: Luan Nunes Utimura Date: Fri, 22 Sep 2023 13:18:37 -0300 Subject: [PATCH] Add sources.list for non-default layers After reestructuring `deb-local-binary` into multiple binary repositories, one for each layer, the image build now has access to three repositories by default: * deb-local-binary; * deb-local-binary-containers; * deb-local-build. Although these repositories are sufficient for the vast majority of images, there are some, such as `stx-openstackclients`, that require dependencies that are found in another repository: `deb-local-binary-openstack`. This image, as the name suggests, contains all the OpenStack clients currently supported by the StarlingX OpenStack application. As one can see in [1], although most clients are installed in the form of pip packages, i.e., through wheel files collected from the upstream indexes, there are two clients that are installed in the form of distribution packages: * python3-cinderclient; * python3-openstackclient. The reason why they are installed as distribution packages is because they have patches that are specific to StarlingX OpenStack -- [2] and [3] -- and also because the build procedure can easily identify them in `deb-local-build` and use the patched versions as expected by us (of course, it they have been built previously). The problem, however, is that although this image is currently being built, there is a detail in it that can cause problems at run time. Despite the build locating (and using) our patched packages, when it searches for their dependencies, it always ends up using binaries from repositories `deb-local-binary`, `deb-local-binary-containers` or `deb-local-build`. Which means that, if we have a dependency on `deb-local-binary-openstack` meant for OpenStack Antelope, and the same dependency on `deb-local-binary` meant for OpenStack Victoria, the image build will use the latter, for the simple fact that it do not consider the `openstack` layer (and, therefore, `deb-local-binary-openstack`). To get around this problem, this change seeks to add a `sources.list` for each build layer. By default, they are all disabled. However, if necessary, they can be enabled and used in image builds based on Docker or LOCI. Note: This change is the prerequisite of [4]. The test plan with the `stx-openstackclients` image will be done there. [1] https://opendev.org/starlingx/openstack-armada-app/src/branch/f/antelope/upstream/openstack/python-openstackclient/debian/stx-openstackclient.stable_docker_image#L26-L27 [2] https://opendev.org/starlingx/openstack-armada-app/src/branch/f/antelope/upstream/openstack/python-cinderclient/debian/patches [3] https://opendev.org/starlingx/openstack-armada-app/src/branch/f/antelope/upstream/openstack/python-openstackclient/debian/patches [4] https://review.opendev.org/c/starlingx/openstack-armada-app/+/896462 Test Plan: PASS - Build `stx-debian` base image PASS - Build all container images with --clean and confirm that: 1. All container images were built successfully; 2. All container images were removed successfully. Partial-Bug: 2037339 Change-Id: I36d1af990afad9fd69b723bf3a13b88673dd08ad Signed-off-by: Luan Nunes Utimura (cherry picked from commit 5fa60a542120d7dd510aabae237c18839127705e) --- .../build-docker-images/build-stx-base.sh | 20 ++++++++++++++++++- .../stx-debian/Dockerfile.stable | 9 +++++++++ .../stx-debian/apt/layer.sources.list.in | 1 + 3 files changed, 29 insertions(+), 1 deletion(-) create mode 100644 build-tools/build-docker-images/stx-debian/apt/layer.sources.list.in diff --git a/build-tools/build-docker-images/build-stx-base.sh b/build-tools/build-docker-images/build-stx-base.sh index ebfddb8c..8d447484 100755 --- a/build-tools/build-docker-images/build-stx-base.sh +++ b/build-tools/build-docker-images/build-stx-base.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright (c) 2018-2019 Wind River Systems, Inc. +# Copyright (c) 2018-2023 Wind River Systems, Inc. # # SPDX-License-Identifier: Apache-2.0 # @@ -10,6 +10,7 @@ MY_SCRIPT_DIR=$(dirname $(readlink -f $0)) source ${MY_SCRIPT_DIR}/../utils.sh +source ${MY_SCRIPT_DIR}/../git-utils.sh # Required env vars if [ -z "${MY_WORKSPACE}" -o -z "${MY_REPO}" ]; then @@ -356,6 +357,7 @@ else -e "s!@DEBIAN_DISTRIBUTION@!${DEBIAN_DISTRIBUTION}!g" \ -e "s!@REPOMGR_DEPLOY_URL@!${REPOMGR_DEPLOY_URL}!g" \ -e "s!@REPOMGR_ORIGIN@!${REPOMGR_ORIGIN}!g" \ + -e "s!@LAYER@!${LAYER}!g" \ "$@" } @@ -365,6 +367,22 @@ else # debian.sources.list replace_vars "${SRC_DOCKER_DIR}/apt/debian.sources.list.in" >"${BUILDDIR}/apt/debian.sources.list" + # .sources.list + # These can be optionally used if it is necessary to build an image that + # requires dependencies that are in repositories not listed in + # `stx.sources.list`. + layer_cfg_name="${OS}_build_layer.cfg" + layer_cfgs=($(find ${GIT_LIST} -maxdepth 1 -name ${layer_cfg_name})) + LAYERS=($( + for layer_cfg in "${layer_cfgs[@]}"; do + echo $(cat "${layer_cfg}") + done | sort --unique + )) + + for LAYER in "${LAYERS[@]}"; do + replace_vars "${SRC_DOCKER_DIR}/apt/layer.sources.list.in" >"${BUILDDIR}/apt/${LAYER}.layer.sources.list" + done + # stx.sources: if user provided any --repo's use them instead of the template if [[ "${#REPO_LIST[@]}" -gt 0 ]] ; then rm -f "${BUILDDIR}/apt/stx.sources.list" diff --git a/build-tools/build-docker-images/stx-debian/Dockerfile.stable b/build-tools/build-docker-images/stx-debian/Dockerfile.stable index 8c11a0b1..f9814e94 100644 --- a/build-tools/build-docker-images/stx-debian/Dockerfile.stable +++ b/build-tools/build-docker-images/stx-debian/Dockerfile.stable @@ -18,6 +18,15 @@ COPY apt/debian.sources.list /etc/apt/sources.list.d/debian.list.disabled COPY apt/stx.sources.list /etc/apt/sources.list.d/stx.list.disabled COPY apt/stx.preferences /etc/apt/preferences.d/stx +# Install layer-specific binary repositories. +# Note: They are all supposed to be disabled by default, but can be optionally +# enabled if it is necessary to build an image that requires +# dependencies that are in repositories not listed in `stx.sources.list`. +COPY apt/*.layer.sources.list /etc/apt/sources.list.d/ +RUN for layer in /etc/apt/sources.list.d/*.layer.sources.list; do \ + mv "${layer}" "$(echo "${layer}" | sed s/.layer.sources.list/.list.disabled/)"; \ + done + # repo templates: # /etc/apt/sources.list.d/ # debian.list.disabled - vanilla debian repos diff --git a/build-tools/build-docker-images/stx-debian/apt/layer.sources.list.in b/build-tools/build-docker-images/stx-debian/apt/layer.sources.list.in new file mode 100644 index 00000000..23adb872 --- /dev/null +++ b/build-tools/build-docker-images/stx-debian/apt/layer.sources.list.in @@ -0,0 +1 @@ +deb [trusted=yes] @REPOMGR_DEPLOY_URL@/deb-local-binary-@LAYER@ @DEBIAN_DISTRIBUTION@ main