summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorScott Little <scott.little@windriver.com>2019-03-06 16:47:22 -0500
committerScott Little <scott.little@windriver.com>2019-03-13 14:32:03 +0000
commitcf435e6c2a8f8b0be479fbe16a1cc3fa1f752d69 (patch)
tree8fbee4b60783d67745451d0e8c638bab4e962c30
parenteecfe6ed5ee5c9709edee95541a98d1ca30973b2 (diff)
download_mirror.sh fails with 404 errors
The 'yum makecache' step is failing. The issue is seen when the fast mirror plugin for yum is in use (default). The metadata for a yum repo as scattered across several files. The master file is repomd.xml, and it may list several supporting files, e.g. filelists.xml.gz primary.xml.gz filelists.sqlite.bz2 other.sqlite.bz2 other.xml.gz primary.sqlite.bz2 When fast mirror plugin is in use, each file might download from a separate source. This opens the door for a race condition when a repo update is rolling out across the mirrors. The error suggests we have already downloaded a repomd.xml, but it is obsolete and references a supporting file that no longer exists. A second possible source of inconsistent data is caching proxies. Solution: 1) Disabling the yum fast cache plugin is undesirable, and there is no option forcing metadata iof a given repo to be single sourced. So this update adds a limited number of retries for the 'yum makecache' command. So far we've never seen 'yum makecache' fail twice in a row... largely because the 'fastest mirror' plugin rarely downloaded from the same source twice. 2) Add 'http_caching=packages' to yum.conf, which asks upstream proxies to not cache repodata. Change-Id: I3fa3f61e38d868fb14f4175b87c0d323abb57def Closes-bug: 1818911 Signed-off-by: Scott Little <scott.little@windriver.com>
Notes
Notes (review): Code-Review+2: Don Penney <don.penney@windriver.com> Code-Review+1: Bruce Jones <bruce.e.jones@intel.com> Code-Review+2: Erich Cordoba <erich.cordoba.malibran@intel.com> Workflow+1: Erich Cordoba <erich.cordoba.malibran@intel.com> Verified+2: Zuul Submitted-by: Zuul Submitted-at: Wed, 13 Mar 2019 16:49:44 +0000 Reviewed-on: https://review.openstack.org/641495 Project: openstack/stx-tools Branch: refs/heads/master
-rw-r--r--Dockerfile2
-rwxr-xr-xcentos-mirror-tools/dl_rpms.sh19
-rw-r--r--centos-mirror-tools/yum.conf.sample1
3 files changed, 21 insertions, 1 deletions
diff --git a/Dockerfile b/Dockerfile
index 6462ca7..74ed724 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -21,6 +21,8 @@ FROM centos:7.4.1708
21# echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\ 21# echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
22#export ftp_proxy=$ftp_proxy" >> /root/.bashrc 22#export ftp_proxy=$ftp_proxy" >> /root/.bashrc
23 23
24RUN echo "http_caching=packages" >> /etc/yum.conf
25
24# username you will docker exec into the container as. 26# username you will docker exec into the container as.
25# It should NOT be your host username so you can easily tell 27# It should NOT be your host username so you can easily tell
26# if you are in our out of the container. 28# if you are in our out of the container.
diff --git a/centos-mirror-tools/dl_rpms.sh b/centos-mirror-tools/dl_rpms.sh
index 54ff628..15f1f71 100755
--- a/centos-mirror-tools/dl_rpms.sh
+++ b/centos-mirror-tools/dl_rpms.sh
@@ -251,7 +251,24 @@ download () {
251 251
252 252
253# Prime the cache 253# Prime the cache
254${SUDOCMD} yum ${YUMCONFOPT} ${RELEASEVER} makecache 254loop_count=0
255max_loop_count=5
256echo "${SUDOCMD} yum ${YUMCONFOPT} ${RELEASEVER} makecache"
257while ! ${SUDOCMD} yum ${YUMCONFOPT} ${RELEASEVER} makecache ; do
258 # To protect against intermittent 404 errors, we'll retry
259 # a few times. The suspected issue is pulling repodata
260 # from multiple source that are temporarily inconsistent.
261 loop_count=$((loop_count + 1))
262 if [ $loop_count -gt $max_loop_count ]; then
263 break
264 fi
265 echo "makecache retry: $loop_count"
266
267 # Wipe the inconsistent data from the last try
268 echo "yum ${YUMCONFOPT} ${RELEASEVER} clean all"
269 yum ${YUMCONFOPT} ${RELEASEVER} clean all
270done
271
255 272
256# Download files 273# Download files
257if [ -s "$rpms_list" ];then 274if [ -s "$rpms_list" ];then
diff --git a/centos-mirror-tools/yum.conf.sample b/centos-mirror-tools/yum.conf.sample
index 11a6ce7..b968a8a 100644
--- a/centos-mirror-tools/yum.conf.sample
+++ b/centos-mirror-tools/yum.conf.sample
@@ -13,6 +13,7 @@ bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://b
13distroverpkg=centos-release 13distroverpkg=centos-release
14override_install_langs=en_US.utf8 14override_install_langs=en_US.utf8
15tsflags=nodocs 15tsflags=nodocs
16http_caching=packages
16reposdir=yum.repos.d 17reposdir=yum.repos.d
17 18
18 19