starlingx
/
tools
Code Issues Proposed changes

Enable starlingx mirror 

StarlingX needs to download a variety of rpms and tarballs
from various upstream sources.  Unfortunately the upstream sources
are not always dependable. Either servers go down, are unreachable,
or drop older content that we still depend on.

Our proposed solution is to run our own mirror to capture an
independent copy of the content needed by StarlingX.
For this purpose, a server has been set up at
http://mirror.starlingx.cengn.ca/mirror/centos

The mirror will use deterministic paths derived from the upstream
urls.  Scripts will be used to convert an upstream url to
the mirror's equivalent url (see function url_to_stx_mirror_url in
url_utils.sh)

The mirror will be refreshed daily.  New .lst entries will be
processed at that time.  Processing of changes under yum.repos.d
is not automated by this update.  Expect a follow-up update to
address this issue soon.  These scripts are found under the
'stx_mirror_scripts' subdirectory.

Changes are made to the download_mirror.sh script, and it's
supporting scripts.  New arguments have been added to each
script to select the download source.
   -s  StarlingX mirror only
   -S  StarlingX mirror, with upstream source as backup
   -u  Upstream source only
   -U  Upstream source, with StarlingX mirror as backup

You do not need to provide any of these flags.  Continue to
us download_mirror.sh as you always have.  The default
behavior is currently set to '-S', i.e. first try the
StarlingX mirror, with upstream source as backup.
If this proves to place to heavy a load on the existing
server, we might switch the default to '-U', i.e. first
try the upstream source, with StarlingX mirror as backup.
If poor download performance is seen, you might want to try
explicitly adding -U as an argument.

The remaining two options are not recommended for regular use.
Upstream only, i.e. '-u', restores original behaviour, but
you may once again encounter rpms that have aged out, and
been removed from their original repos.  StarlingX only,
i.e. '-s', is vulnerable if a .lst file has been updated,
but the mirror has not yet processed it.

Change-Id: I7e0f3d9fb99253662f9f4bf12457d39250408c0b
Story: 2003906
Task: 26785
Signed-off-by: Scott Little <scott.little@windriver.com>
This commit is contained in:
Scott Little 2018-10-31 11:18:17 -04:00
parent c5f3e6de55
commit ae68691bf0
14 changed files with 1756 additions and 111 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

160
centos-mirror-tools/dl_other_from_centos_repo.sh
View File

@ -1,8 +1,93 @@
#!/bin/bash -e
# download non-RPM files from http://vault.centos.org/7.4.1708/os/x86_64/
#
# SPDX-License-Identifier: Apache-2.0
#
#
# Download non-RPM files from http://vault.centos.org/7.4.1708/os/x86_64/
#
DL_OTHER_FROM_CENTOS_REPO_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $DL_OTHER_FROM_CENTOS_REPO_DIR/url_utils.sh
usage () {
echo "$0 [-D <distro>] [-s|-S|-u|-U] [-h] <other_download_list.ini> <save_path> [<force_update>]"
}
# Permitted values of dl_source
dl_from_stx_mirror="stx_mirror"
dl_from_upstream="upstream"
dl_from_stx_then_upstream="$dl_from_stx_mirror $dl_from_upstream"
dl_from_upstream_then_stx="$dl_from_upstream $dl_from_stx_mirror"
# Download from what source?
# dl_from_stx_mirror = StarlingX mirror only
# dl_from_upstream = Original upstream source only
# dl_from_stx_then_upstream = Either source, STX prefered (default)"
# dl_from_upstream_then_stx = Either source, UPSTREAM prefered"
dl_source="$dl_from_stx_then_upstream"
dl_flag=""
distro="centos"
MULTIPLE_DL_FLAG_ERROR_MSG="Error: Please use only one of: -s,-S,-u,-U"
multiple_dl_flag_check () {
if [ "$dl_flag" != "" ]; then
echo "$MULTIPLE_DL_FLAG_ERROR_MSG"
usage
exit 1
fi
}
# Parse out optional arguments
while getopts "D:hsSuU" o; do
case "${o}" in
D)
distro="${OPTARG}"
;;
s)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_mirror"
dl_flag="-s"
;;
S)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_then_upstream"
dl_flag="-S"
;;
u)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream"
dl_flag="-u"
;;
U)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream_then_stx"
dl_flag="-U"
;;
h)
# Help
usage
exit 0
;;
*)
usage
exit 1
;;
esac
done
shift $((OPTIND-1))
if [ $# -lt 2 ]; then
echo "$0 <other_download_list.ini> <save_path> [<force_update>]"
usage
exit -1
fi
@ -13,12 +98,15 @@ if [ ! -e $download_list ];then
fi
save_path=$2
url_prefix="http://vault.centos.org/7.4.1708/os/x86_64/"
echo "NOTE: please assure Internet access to $url_prefix !!"
upstream_url_prefix="http://vault.centos.org/7.4.1708/os/x86_64/"
stx_mirror_url_prefix="$(url_to_stx_mirror_url "$upstream_url_prefix" "$distro")"
echo "NOTE: please assure Internet access to $upstream_url_prefix !!"
force_update=$3
i=0
error_count=0
all=`cat $download_list`
for ff in $all; do
## skip commented_out item which starts with '#'
@ -30,28 +118,68 @@ for ff in $all; do
_name=`echo $ff | cut -d":" -f2-2`
if [ "$_type" == "folder" ];then
mkdir -p $save_path/$_name
if [ $? -ne 0 ]; then
echo "Error: mkdir -p '$save_path/$_name'"
error_count=$((error_count + 1))
fi
else
if [ -e "$save_path/$_name" ]; then
echo "Already have $save_path/$_name"
continue
fi
echo "remote path: $url_prefix/$_name"
echo "local path: $save_path/$_name"
if wget $url_prefix/$_name; then
file_name=`basename $_name`
sub_path=`dirname $_name`
if [ -e "./$file_name" ]; then
let i+=1
echo "$file_name is downloaded successfully"
mv -f ./$file_name $save_path/$_name
ls -l $save_path/$_name
for dl_src in $dl_source; do
case $dl_src in
$dl_from_stx_mirror)
url_prefix="$stx_mirror_url_prefix"
;;
$dl_from_upstream)
url_prefix="$upstream_url_prefix"
;;
*)
echo "Error: Unknown dl_source '$dl_src'"
continue
;;
esac
echo "remote path: $url_prefix/$_name"
echo "local path: $save_path/$_name"
if wget $url_prefix/$_name; then
file_name=`basename $_name`
sub_path=`dirname $_name`
if [ -e "./$file_name" ]; then
let i+=1
echo "$file_name is downloaded successfully"
\mv -f ./$file_name $save_path/$_name
if [ $? -ne 0 ]; then
echo "Error: mv -f './$file_name' '$save_path/$_name'"
error_count=$((error_count + 1))
fi
ls -l $save_path/$_name
fi
break
else
echo "Warning: failed to download $url_prefix/$_name"
fi
else
echo "ERROR: failed to download $url_prefix/$_name"
done
if [ ! -e "$save_path/$_name" ]; then
echo "Error: failed to download '$url_prefix/$_name'"
error_count=$((error_count + 1))
continue
fi
fi
done
echo ""
echo "totally $i files are downloaded!"
if [ $error_count -ne 0 ]; then
echo ""
echo "Encountered $error_count errors"
exit 1
fi
exit 0

111
centos-mirror-tools/dl_rpms.sh
View File

@ -15,7 +15,9 @@ SUDOCMD="sudo -E"
RELEASEVER="--releasever=7"
YUMCONFOPT=""
source utils.sh
DL_RPMS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $DL_RPMS_DIR/utils.sh
usage() {
echo "$0 [-n] [-c <yum.conf>] <rpms_list> <match_level> "
@ -45,8 +47,34 @@ usage() {
CLEAN_LOGS_ONLY=0
dl_rc=0
# Permitted values of dl_source
dl_from_stx_mirror="stx_mirror"
dl_from_upstream="upstream"
dl_from_stx_then_upstream="$dl_from_stx_mirror $dl_from_upstream"
dl_from_upstream_then_stx="$dl_from_upstream $dl_from_stx_mirror"
# Download from what source?
# dl_from_stx_mirror = StarlingX mirror only
# dl_from_upstream = Original upstream source only
# dl_from_stx_then_upstream = Either source, STX prefered (default)"
# dl_from_upstream_then_stx = Either source, UPSTREAM prefered"
dl_source="$dl_from_stx_then_upstream"
dl_flag=""
distro="centos"
MULTIPLE_DL_FLAG_ERROR_MSG="Error: Please use only one of: -s,-S,-u,-U"
multiple_dl_flag_check () {
if [ "$dl_flag" != "" ]; then
echo "$MULTIPLE_DL_FLAG_ERROR_MSG"
usage
exit 1
fi
}
# Parse option flags
while getopts "c:nxh" o; do
while getopts "c:nxD:sSuUh" o; do
case "${o}" in
n)
# No-sudo
@ -61,6 +89,35 @@ while getopts "c:nxh" o; do
YUMCONFOPT="-c $OPTARG"
grep -q "releasever=" $OPTARG && RELEASEVER="--$(grep releasever= ${OPTARG})"
;;
D)
distro="${OPTARG}"
;;
s)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_mirror"
dl_flag="-s"
;;
S)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_then_upstream"
dl_flag="-S"
;;
u)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream"
dl_flag="-u"
;;
U)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream_then_stx"
dl_flag="-U"
;;
h)
# Help
usage
@ -124,8 +181,8 @@ fi
download () {
local _file=$1
local _level=$2
local _list=$(cat $_file)
local _from=$(get_from $_file)
local _list=""
local _from=""
local _arch=""
@ -134,23 +191,51 @@ download () {
local download_url=""
local rpm_name=""
local SFILE=""
local lvl
local dl_result
_list=$(cat $_file)
_from=$(get_from $_file)
echo "now the rpm will come from: $_from"
for ff in $_list; do
_arch=$(get_arch_from_rpm $ff)
rpm_name="$(get_rpm_name $ff)"
download_cmd="$(get_download_cmd $ff $_level)"
dest_dir="$(get_dest_directory $_arch)"
if [ ! -e $dest_dir/$rpm_name ]; then
echo "Looking for $rpm_name"
echo "--> run: $download_cmd"
if $download_cmd ; then
download_url="$(get_url $ff $_level)"
SFILE="$(get_rpm_level_name $rpm_name $_level)"
process_result "$_arch" "$dest_dir" "$download_url" "$SFILE"
else
echo "Warning: $rpm_name not found"
dl_result=1
for dl_src in $dl_source; do
case $dl_src in
$dl_from_stx_mirror)
lvl=$dl_from_stx_mirror
;;
$dl_from_upstream)
lvl=$_level
;;
*)
echo "Error: Unknown dl_source '$dl_src'"
continue
;;
esac
download_cmd="$(get_download_cmd $ff $lvl)"
echo "Looking for $rpm_name"
echo "--> run: $download_cmd"
if $download_cmd ; then
download_url="$(get_url $ff $lvl)"
SFILE="$(get_rpm_level_name $rpm_name $lvl)"
process_result "$_arch" "$dest_dir" "$download_url" "$SFILE"
dl_result=0
break
else
echo "Warning: $rpm_name not found"
fi
done
if [ $dl_result -eq 1 ]; then
echo "Error: $rpm_name not found"
echo "missing_srpm:$rpm_name" >> $LOG
echo $rpm_name >> $MISSING_SRPMS
rc=1

308
centos-mirror-tools/dl_tarball.sh
View File

@ -1,5 +1,9 @@
#!/usr/bin/env bash
#
# SPDX-License-Identifier: Apache-2.0
#
# The build of StarlingX relies, besides RPM Binaries and Sources, in this
# repository which is a collection of packages in the form of Tar Compressed
# files and 3 RPMs obtained from a Tar Compressed file. This script and a text
@ -14,6 +18,85 @@ script_path="$(dirname $(readlink -f $0))"
tarball_file="$script_path/tarball-dl.lst"
mvn_artf_file="$script_path/mvn-artifacts.lst"
DL_TARBALL_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $DL_TARBALL_DIR/url_utils.sh
usage () {
echo "$0 [-D <distro>] [-s|-S|-u|-U] [-h] <other_download_list.ini> <save_path> [<force_update>]"
}
# Permitted values of dl_source
dl_from_stx_mirror="stx_mirror"
dl_from_upstream="upstream"
dl_from_stx_then_upstream="$dl_from_stx_mirror $dl_from_upstream"
dl_from_upstream_then_stx="$dl_from_upstream $dl_from_stx_mirror"
# Download from what source?
# dl_from_stx_mirror = StarlingX mirror only
# dl_from_upstream = Original upstream source only
# dl_from_stx_then_upstream = Either source, STX prefered (default)"
# dl_from_upstream_then_stx = Either source, UPSTREAM prefered"
dl_source="$dl_from_stx_then_upstream"
dl_flag=""
distro="centos"
MULTIPLE_DL_FLAG_ERROR_MSG="Error: Please use only one of: -s,-S,-u,-U"
multiple_dl_flag_check () {
if [ "$dl_flag" != "" ]; then
echo "$MULTIPLE_DL_FLAG_ERROR_MSG"
usage
exit 1
fi
}
# Parse out optional arguments
while getopts "D:hsSuU" o; do
case "${o}" in
D)
distro="${OPTARG}"
;;
s)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_mirror"
dl_flag="-s"
;;
S)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_then_upstream"
dl_flag="-S"
;;
u)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream"
dl_flag="-u"
;;
U)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream_then_stx"
dl_flag="-U"
;;
h)
# Help
usage
exit 0
;;
*)
usage
exit 1
;;
esac
done
shift $((OPTIND-1))
if [ ! -e $tarball_file -o ! -e $mvn_artf_file ];then
echo "$download_list does not exist, please have a check!"
exit -1
@ -40,20 +123,55 @@ fi
# Download function using wget command
download_package() {
wget --spider $1
if [ $? != 0 ]; then
echo "$1 is broken"
else
wget -t 5 --wait=15 $1
local upstream_url="$1"
local stx_url=""
local url=""
local rc=1
stx_url="$(url_to_stx_mirror_url "$upstream_url" "$distro")"
for dl_src in $dl_source; do
case $dl_src in
$dl_from_stx_mirror)
url="$stx_url"
;;
$dl_from_upstream)
url="$upstream_url"
;;
*)
echo "Error: Unknown dl_source '$dl_src'"
continue
;;
esac
wget --spider "$url"
if [ $? != 0 ]; then
echo "$1" > "$output_log"
echo "Warning: '$url' is broken"
else
wget -t 5 --wait=15 "$url"
if [ $? -eq 0 ]; then
rc=0
break
else
echo "Warning: failed to download '$url'"
continue
fi
fi
done
if [ $rc != 0 ]; then
echo "Error: failed to download '$upstream_url'"
echo "$upstream_url" > "$output_log"
fi
return $rc
}
# This script will iterate over the tarball.lst text file and execute specific
# tasks based on the name of the package:
error_count=0;
for line in $(cat $tarball_file); do
# A line from the text file starting with "#" character is ignored
@ -69,11 +187,31 @@ for line in $(cat $tarball_file); do
# denotes special handling is required tarball_name=`echo $line | cut -d"#" -f1-1`
# - Column 2, name of the directory path after it is decompressed as it is
# referenced in the build system recipe.
# - Column 3, the URL for the package download
# - Column 3, the URL for the file or git to download
# - Column 4, download method, one of
# http - download a simple file
# http_filelist - download multiple files by appending a list of subpaths
# to the base url. Tar up the lot.
# http_script - download a simple file, run script whos output is a tarball
# git - download a git, checkout branch and tar it up
# git_script - download a git, checkout branch, run script whos output is a tarball
#
# - Column 5, utility field
# If method is git or git_script, this is a branch,tag,sha we need to checkout
# If method is http_filelist, this is the path to a file containing subpaths.
# Subpaths are appended to the urls and downloaded.
# Otherwise unused
# - Column 6, Path to script.
# Not yet supported.
# Intent is to run this script to produce the final tarball, replacing
# all the special case code currently embedded in this script.
tarball_name=$(echo $line | cut -d"#" -f1-1)
directory_name=$(echo $line | cut -d"#" -f2-2)
tarball_url=$(echo $line | cut -d"#" -f3-3)
method=$(echo $line | cut -d"#" -f4-4)
util=$(echo $line | cut -d"#" -f5-5)
script=$(echo $line | cut -d"#" -f6-6)
# Remove leading '!' if present
tarball_name="${tarball_name//!/}"
@ -104,6 +242,12 @@ for line in $(cat $tarball_file); do
pushd $output_tarball
if [ "$tarball_name" = "integrity-kmod-e6aef069.tar.gz" ]; then
download_package $tarball_url
if [ $? -ne 0 ]; then
error_count=$((error_count + 1))
popd # pushd $output_tarball
continue
fi
tar xf e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz
mv linux-tpmdd-e6aef06/security/integrity/ $directory_name
tar czvf $tarball_name $directory_name
@ -111,6 +255,12 @@ for line in $(cat $tarball_file); do
rm e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz
elif [ "$tarball_name" = "mariadb-10.1.28.tar.gz" ]; then
download_package $tarball_url
if [ $? -ne 0 ]; then
error_count=$((error_count + 1))
popd # pushd $output_tarball
continue
fi
mkdir $directory_name
tar xf $tarball_name --strip-components 1 -C $directory_name
rm $tarball_name
@ -122,6 +272,7 @@ for line in $(cat $tarball_file); do
popd
tar czvf $tarball_name $directory_name
rm -rf $directory_name
popd # pushd $directory_name
# The mvn.repo.tgz tarball will be created downloading a serie of
# of maven artifacts described in mvn-artifacts file.
elif [ "$tarball_name" = "mvn.repo.tgz" ]; then
@ -130,11 +281,49 @@ for line in $(cat $tarball_file); do
echo "$mvn_artf_file no found" 1>&2
exit 1
fi
success_all=0
while read -r artf; do
echo "download: $(basename $artf)"
wget "$tarball_url/$artf" -P "$directory_name/$(dirname $artf)"
success=1
for dl_src in $dl_source; do
case $dl_src in
$dl_from_stx_mirror)
url="$(url_to_stx_mirror_url "$tarball_url/$artf" "$distro")"
;;
$dl_from_upstream)
url="$tarball_url/$artf"
;;
*)
echo "Error: Unknown dl_source '$dl_src'"
continue
;;
esac
wget "$url" -P "$directory_name/$(dirname $artf)"
if [ $? -eq 0 ]; then
success=0
break
else
echo "Warning: Failed to download from $url"
continue;
fi
done
if [ $success -ne 0 ]; then
success_all=1
echo "Error: Failed to download from '$tarball_url/$artf'"
fi
done < "$mvn_artf_file"
if [ $success_all -ne 0 ]; then
echo "Error: Failed to download one or more components from '$tarball_url'"
echo "$tarball_url" > "$output_log"
error_count=$((error_count + 1))
popd # pushd $output_tarball
continue
fi
# Create tarball
tar -zcvf "$tarball_name" -C "$directory_name"/ .
rm -rf "$directory_name"
@ -142,7 +331,14 @@ for line in $(cat $tarball_file); do
elif [[ "$tarball_name" =~ ^'MLNX_OFED_LINUX' ]]; then
pkg_version=$(echo "$tarball_name" | cut -d "-" -f2-3)
srpm_path="MLNX_OFED_SRC-${pkg_version}/SRPMS/"
download_package "$tarball_url"
if [ $? -ne 0 ]; then
error_count=$((error_count + 1))
popd # pushd $output_tarball
continue
fi
tar -xf "$tarball_name"
tar -xf "$directory_name/src/MLNX_OFED_SRC-${pkg_version}.tgz"
# This section of code gets specific SRPMs versions according
@ -168,8 +364,20 @@ for line in $(cat $tarball_file); do
rm -rf "$directory_name"
elif [ "$tarball_name" = "qat1.7.upstream.l.1.0.3-42.tar.gz" ]; then
download_package $tarball_url
if [ $? -ne 0 ]; then
error_count=$((error_count + 1))
popd # pushd $output_tarball
continue
fi
elif [ "$tarball_name" = "tpm-kmod-e6aef069.tar.gz" ]; then
download_package $tarball_url
if [ $? -ne 0 ]; then
error_count=$((error_count + 1))
popd # pushd $output_tarball
continue
fi
tar xf e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz
mv linux-tpmdd-e6aef06/drivers/char/tpm $directory_name
tar czvf $tarball_name $directory_name
@ -177,22 +385,74 @@ for line in $(cat $tarball_file); do
rm -rf $directory_name
rm e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz
elif [ "$tarball_name" = "tss2-930.tar.gz" ]; then
git clone https://git.code.sf.net/p/ibmtpm20tss/tss ibmtpm20tss-tss
pushd ibmtpm20tss-tss
git checkout v930
dest_dir=ibmtpm20tss-tss
for dl_src in $dl_source; do
case $dl_src in
$dl_from_stx_mirror)
url="$(url_to_stx_mirror_url "$tarball_url" "$distro")"
;;
$dl_from_upstream)
url="$tarball_url"
;;
*)
echo "Error: Unknown dl_source '$dl_src'"
continue
;;
esac
git clone $url $dest_dir
if [ $? -eq 0 ]; then
# Success
break
else
echo "Warning: Failed to git clone from '$url'"
continue
fi
done
if [ ! -d $dest_dir ]; then
echo "Error: Failed to git clone from '$tarball_url'"
echo "$tarball_url" > "$output_log"
error_count=$((error_count + 1))
popd # pushd $output_tarball
continue
fi
pushd $dest_dir
branch=$util
git checkout $branch
rm -rf .git
popd
mv ibmtpm20tss-tss $directory_name
tar czvf $tarball_name $directory_name
rm -rf $directory_name
popd # pushd $dest_dir
fi
popd
popd # pushd $output_tarball
continue
fi
download_cmd="wget -t 5 --wait=15 $tarball_url -O $download_path"
if [ -e $download_path ]; then
echo "Already have $download_path"
continue
fi
for dl_src in $dl_source; do
case $dl_src in
$dl_from_stx_mirror)
url="$(url_to_stx_mirror_url "$tarball_url" "$distro")"
;;
$dl_from_upstream)
url="$tarball_url"
;;
*)
echo "Error: Unknown dl_source '$dl_src'"
continue
;;
esac
download_cmd="wget -t 5 --wait=15 $url -O $download_path"
if [ ! -e $download_path ]; then
if $download_cmd ; then
echo "Ok: $download_path"
pushd $download_directory
@ -204,16 +464,26 @@ for line in $(cat $tarball_file); do
rm -r $directory_name
fi
popd
break
else
echo "Error: Failed to download $tarball_url" 1>&2
echo "$tarball_url" > "$output_log"
echo "Warning: Failed to download $url" 1>&2
continue
fi
done
else
echo "Already have $download_path"
if [ ! -e $download_path ]; then
echo "Error: Failed to download $tarball_url" 1>&2
echo "$tarball_url" > "$output_log"
error_count=$((error_count + 1))
fi
done
# End of file
if [ $error_count -ne 0 ]; then
echo ""
echo "Encountered $error_count errors"
exit 1
fi
exit 0

148
centos-mirror-tools/download_mirror.sh
View File

@ -4,7 +4,7 @@
#
usage() {
echo "$0 [-n] [-c <yum.conf>] [-g]"
echo "$0 [-n] [-c <yum.conf>] [-g] [-s|-S|-u|-U]"
echo ""
echo "Options:"
echo " -n: Do not use sudo when performing operations (option passed on to"
@ -12,6 +12,10 @@ usage() {
echo " -c: Use an alternate yum.conf rather than the system file (option passed"
echo " on to subscripts when appropriate)"
echo " -g: do not change group IDs of downloaded artifacts"
echo " -s: Download from StarlingX mirror only"
echo " -S: Download from StarlingX mirror, upstream as backup (default)"
echo " -u: Download from original upstream sources only"
echo " -U: Download from original upstream sources, StarlingX mirror as backup"
echo ""
}
@ -24,8 +28,17 @@ generate_log_name() {
need_file(){
for f in $*; do
if [ ! -e $f ]; then
echo "ERROR: $f does not exist."
if [ ! -f $f ]; then
echo "ERROR: File $f does not exist."
exit 1
fi
done
}
need_dir(){
for d in $*; do
if [ ! -d $d ]; then
echo "ERROR: Directory $d does not exist."
exit 1
fi
done
@ -35,12 +48,16 @@ need_file(){
rpm_downloader="./dl_rpms.sh"
tarball_downloader="./dl_tarball.sh"
other_downloader="./dl_other_from_centos_repo.sh"
make_stx_mirror_yum_conf="./make_stx_mirror_yum_conf.sh"
# track optional arguments
change_group_ids=1
use_system_yum_conf=1
alternate_yum_conf=""
alternate_repo_dir=""
rpm_downloader_extra_args=""
tarball_downloader_extra_args=""
distro="centos"
# lst files to use as input
rpms_from_3rd_parties="./rpms_3rdparties.lst"
@ -51,8 +68,43 @@ other_downloads="./other_downloads.lst"
# Overall success
success=1
# Parse out optional -c or -n arguments
while getopts "c:ngh" o; do
# Permitted values of dl_source
dl_from_stx_mirror="stx_mirror"
dl_from_upstream="upstream"
dl_from_stx_then_upstream="$dl_from_stx_mirror $dl_from_upstream"
dl_from_upstream_then_stx="$dl_from_upstream $dl_from_stx_mirror"
# Download from what source?
# dl_from_stx_mirror = StarlingX mirror only
# dl_from_upstream = Original upstream source only
# dl_from_stx_then_upstream = Either source, STX prefered (default)"
# dl_from_upstream_then_stx = Either source, UPSTREAM prefered"
dl_source="$dl_from_stx_then_upstream"
dl_flag=""
dl_from_stx () {
local re="\\b$dl_from_stx_mirror\\b"
[[ "$dl_source" =~ $re ]]
}
dl_from_upstream () {
local re="\\b$dl_from_upstream\\b"
[[ "$dl_source" =~ $re ]]
}
MULTIPLE_DL_FLAG_ERROR_MSG="Error: Please use only one of: -s,-S,-u,-U"
multiple_dl_flag_check () {
if [ "$dl_flag" != "" ]; then
echo "$MULTIPLE_DL_FLAG_ERROR_MSG"
usage
exit 1
fi
}
# Parse out optional arguments
while getopts "c:nghsSuU" o; do
case "${o}" in
n)
# Pass -n ("no-sudo") to rpm downloader
@ -61,12 +113,36 @@ while getopts "c:ngh" o; do
c)
# Pass -c ("use alternate yum.conf") to rpm downloader
use_system_yum_conf=0
rpm_downloader_extra_args="${rpm_downloader_extra_args} -c ${OPTARG}"
alternate_yum_conf="${OPTARG}"
;;
g)
# Do not attempt to change group IDs on downloaded packages
change_group_ids=0
;;
s)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_mirror"
dl_flag="-s"
;;
S)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
dl_source="$dl_from_stx_then_upstream"
dl_flag="-S"
;;
u)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream"
dl_flag="-u"
;;
U)
# Download from upstream only. Do not use StarlingX mirror.
multiple_dl_flag_check
dl_source="$dl_from_upstream_then_stx"
dl_flag="-U"
;;
h)
# Help
usage
@ -98,7 +174,6 @@ need_file ${rpms_from_centos_repo}
need_file ${other_downloads}
need_file tarball-dl.lst mvn-artifacts.lst
#download RPMs/SRPMs from 3rd_party websites (not CentOS repos) by "wget"
echo "step #1: start downloading RPMs/SRPMs from 3rd-party websites..."
@ -111,6 +186,51 @@ if [ ${use_system_yum_conf} -ne 0 ]; then
fi
fi
if [ $use_system_yum_conf -eq 0 ]; then
need_file "${alternate_yum_conf}"
if [ "$alternate_repo_dir" == "" ]; then
alternate_repo_dir=$(grep '^repodir=' "${alternate_yum_conf}" | cut -d '=' -f 2)
need_dir "${alternate_repo_dir}"
fi
fi
TEMP_DIR=""
rpm_downloader_extra_args="${rpm_downloader_extra_args} -D $distro"
if [ "$dl_flag" == "" ]; then
if [ $use_system_yum_conf -eq 0 ]; then
rpm_downloader_extra_args="${rpm_downloader_extra_args} -c ${alternate_yum_conf}"
fi
else
rpm_downloader_extra_args="${rpm_downloader_extra_args} $dl_flag"
if ! dl_from_stx; then
if [ $use_system_yum_conf -eq 0 ]; then
rpm_downloader_extra_args="${rpm_downloader_extra_args} -c ${alternate_yum_conf}"
fi
else
TEMP_DIR=$(mktemp -d /tmp/stx_mirror_XXXXXX)
TEMP_CONF="$TEMP_DIR/yum.conf"
need_file ${make_stx_mirror_yum_conf}
need_dir ${TEMP_DIR}
if [ $use_system_yum_conf -eq 0 ]; then
if dl_from_upstream; then
${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro
else
${make_stx_mirror_yum_conf} -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro
fi
rpm_downloader_extra_args="${rpm_downloader_extra_args} -c $TEMP_CONF"
else
if dl_from_upstream; then
${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro
else
${make_stx_mirror_yum_conf} -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro
fi
rpm_downloader_extra_args="${rpm_downloader_extra_args} -c $TEMP_CONF"
fi
fi
fi
list=${rpms_from_3rd_parties}
level=L1
logfile=$(generate_log_name $list $level)
@ -139,6 +259,9 @@ fi
if [ ${use_system_yum_conf} -eq 1 ]; then
# deleting the StarlingX_3rd to avoid pull centos packages from the 3rd Repo.
\rm -f $REPO_DIR/StarlingX_3rd*.repo
if [ "$TEMP_DIR" != "" ]; then
\rm -f $TEMP_DIR/yum.repos.d/StarlingX_3rd*.repo
fi
fi
@ -232,7 +355,7 @@ fi
echo "step #3: start downloading other files ..."
logfile=$LOGSDIR"/otherfiles_centos_download.log"
${other_downloader} ${other_downloads} ./output/stx-r1/CentOS/pike/Binary/ |& tee $logfile
${other_downloader} ${dl_flag} -D "$distro" ${other_downloads} ./output/stx-r1/CentOS/pike/Binary/ |& tee $logfile
retcode=${PIPESTATUS[0]}
if [ $retcode -eq 0 ];then
echo "step #3: done successfully"
@ -249,7 +372,7 @@ fi
# they will be downloaded.
echo "step #4: start downloading tarball compressed files"
logfile=$LOGSDIR"/tarballs_download.log"
${tarball_downloader} ${tarball_downloader_extra_args} |& tee $logfile
${tarball_downloader} ${dl_flag} -D "$distro" ${tarball_downloader_extra_args} |& tee $logfile
retcode=${PIPESTATUS[0]}
if [ $retcode -eq 0 ];then
echo "step #4: done successfully"
@ -261,6 +384,13 @@ else
success=0
fi
#
# Clean up the mktemp directory, if required.
#
if [ "$TEMP_DIR" != "" ]; then
\rm -rf "$TEMP_DIR"
fi
echo "IMPORTANT: The following 3 files are just bootstrap versions. Based"
echo "on them, the workable images for StarlingX could be generated by"
echo "running \"update-pxe-network-installer\" command after \"build-iso\""

226
centos-mirror-tools/make_stx_mirror_yum_conf.sh Executable file
View File

@ -0,0 +1,226 @@
#!/bin/bash
#
# SPDX-License-Identifier: Apache-2.0
#
#
# Replicate a yum.conf and yum.repo.d under a temporary directory and
# then modify the files to point to equivalent repos in the StarlingX mirror.
# This script was originated by Scott Little
#
MAKE_STX_MIRROR_YUM_CONF_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source "$MAKE_STX_MIRROR_YUM_CONF_DIR/url_utils.sh"
DISTRO="centos"
TEMP_DIR=""
SRC_REPO_DIR="$MAKE_STX_MIRROR_YUM_CONF_DIR/yum.repos.d"
SRC_YUM_CONF="$MAKE_STX_MIRROR_YUM_CONF_DIR/yum.conf.sample"
RETAIN_REPODIR=0
usage () {
echo ""
echo "$0 -d <dest_dir> [-D <distro>] [-y <src_yum_conf>] [-r <src_repos_dir>] [-R]"
echo ""
echo "Replicate a yum.conf and yum.repo.d under a new directory and"
echo "then modify the files to point to equivalent repos in the StarlingX"
echo "mirror."
echo ""
echo "-d <dest_dir> = Place modified yum.conf and yum.repo.d into this directory"
echo "-D <distro> = Target distro on StarlingX mirror. Default is 'centos'"
echo "-y <yum_conf> = Path to yum.conf file that we will modify. Default is"
echo " 'yum.conf.sample' in same directory as this script"
echo "-r <repos_dir> = Path to yum.repos.d that we will modify. Default is"
echo " 'yum.repos.d' in same directory as this script"
}
#
# option processing
#
while getopts "D:d:Rr:y:" o; do
case "${o}" in
D)
DISTRO="${OPTARG}"
;;
d)
TEMP_DIR="${OPTARG}"
;;
r)
SRC_REPO_DIR="${OPTARG}"
;;
R)
RETAIN_REPODIR=1
;;
y)
SRC_YUM_CONF="${OPTARG}"
;;
*)
usage
exit 1
;;
esac
done
#
# option validation
#
if [ ! -f $SRC_YUM_CONF ]; then
echo "Error: yum.conf not found at '$SRC_YUM_CONF'"
exit 1
fi
if [ ! -d $SRC_REPO_DIR ]; then
echo "Error: repo dir not found at '$SRC_REPO_DIR'"
exit 1
fi
if [ "$TEMP_DIR" == "" ]; then
echo "Error: working dir not provided"
usage
exit 1
fi
if [ ! -d $TEMP_DIR ]; then
echo "Error: working dir not found at '$TEMP_DIR'"
exit 1
fi
#
# Get the value of the $releasever variable.
#
# If the source yum.conf has a releasever= setting, we will honor
# that, even though yum will not.
#
# Otherwise use yum to query the host environment (Docker).
# This assumes the host environmnet has the same releasever
# as that which will be used inside StarlingX.
#
# NOTE: In other scripts we will read releasever= out of yum.conf
# and push it back into yum via --releasever=<#>.
#
get_releasever () {
if [ -f $SRC_YUM_CONF ] && grep -q '^releasever=' $SRC_YUM_CONF; then
grep '^releasever=' $SRC_YUM_CONF | cut -d '=' -f 2
else
yum version nogroups | grep Installed | cut -d ' ' -f 2 | cut -d '/' -f 1
fi
}
#
# Get the value of the $basearch variable.
#
# Just use yum to query the host environment (Docker) as we don't support
# cross compiling.
#
get_arch () {
yum version nogroups | grep Installed | cut -d ' ' -f 2 | cut -d '/' -f 2
}
#
# Global variables we will use later.
#
CENGN_REPOS_DIR="$TEMP_DIR/yum.repos.d"
CENGN_YUM_CONF="$TEMP_DIR/yum.conf"
CENGN_YUM_LOG="$TEMP_DIR/yum.log"
CENGN_YUM_CACHDIR="$TEMP_DIR/cache/yum/\$basearch/\$releasever"
RELEASEVER=$(get_releasever)
ARCH=$(get_arch)
#
# Copy as yet unmodified yum.conf and yum.repos.d from source to dest.
#
echo "\cp -r '$SRC_REPO_DIR' '$CENGN_REPOS_DIR'"
\cp -r "$SRC_REPO_DIR" "$CENGN_REPOS_DIR"
echo "\cp '$SRC_YUM_CONF' '$CENGN_YUM_CONF'"
\cp "$SRC_YUM_CONF" "$CENGN_YUM_CONF"
#
# Add or modify reposdir= value in our new yum.conf
#
if grep -q '^reposdir=' $CENGN_YUM_CONF; then
# reposdir= already exists, modify it
if [ $RETAIN_REPODIR -eq 1 ]; then
# Append CENGN_REPOS_DIR
sed "s#^reposdir=.*\$#reposdir=\1 $CENGN_REPOS_DIR#" -i $CENGN_YUM_CONF
else
# replace with CENGN_REPOS_DIR
sed "s#^reposdir=.*\$#reposdir=$CENGN_REPOS_DIR#" -i $CENGN_YUM_CONF
fi
else
# reposdir= doeas not yet exist, add it
if [ $RETAIN_REPODIR -eq 1 ]; then
# Add both SRC_REPO_DIR and CENGN_REPOS_DIR
echo "reposdir=$SRC_REPO_DIR $CENGN_REPOS_DIR" >> $CENGN_YUM_CONF
else
# Add CENGN_REPOS_DIR only
echo "reposdir=$CENGN_REPOS_DIR" >> $CENGN_YUM_CONF
fi
fi
#
# modify or add logfile= value in our new yum.conf
#
if grep -q '^logfile=' $CENGN_YUM_CONF; then
sed "s#^logfile=.*\$#logfile=$CENGN_YUM_LOG#" -i $CENGN_YUM_CONF
else
echo "logfile=$CENGN_YUM_LOG" >> $CENGN_YUM_CONF
fi
#
# modify or add cachedir= value in our new yum.conf
#
if grep -q '^cachedir=' $CENGN_YUM_CONF; then
sed "s#^cachedir=.*\$#cachedir=$CENGN_YUM_CACHDIR#" -i $CENGN_YUM_CONF
else
echo "cachedir=$CENGN_YUM_CACHDIR" >> $CENGN_YUM_CONF
fi
#
# Modify all the repo files in our new yum.repos.d
#
for REPO in $(find "$CENGN_REPOS_DIR" -type f -name '*repo'); do
#
# Replace mirrorlist with baseurl if required
#
if grep -q '^mirrorlist=' "$REPO" ; then
sed '/^mirrorlist=/d' -i "$REPO"
sed 's%^#baseurl%baseurl%' -i "$REPO"
fi
#
# Substitute any $releasever or $basearch variables
#
sed "s#/[$]releasever/#/$RELEASEVER/#g" -i "$REPO"
sed "s#/[$]basearch/#/$ARCH/#g" -i "$REPO"
#
# Turn off gpgcheck for now.
# Must revisit this at a later date!
#
sed 's#^gpgcheck=1#gpgcheck=0#' -i "$REPO"
sed '/^gpgkey=/d' -i "$REPO"
#
# Convert baseurl(s) to cengn equivalent
#
for URL in $(grep '^baseurl=' "$REPO" | sed 's#^baseurl=##'); do
CENGN_URL="$(url_to_stx_mirror_url "$URL" "$DISTRO")"
sed "s#^baseurl=$URL\$#baseurl=$CENGN_URL#" -i "$REPO"
done
#
# Prefix repoid and name with CENGN
#
sed "s#^name=\(.*\)#name=CENGN_\1#" -i "$REPO"
sed "s#^\[\([^]]*\)\]#[CENGN_\1]#" -i "$REPO"
done
echo $TEMP_DIR

383
centos-mirror-tools/stx_mirror_scripts/daily_dl_sync.sh Executable file
View File

@ -0,0 +1,383 @@
#!/bin/bash
#
# SPDX-License-Identifier: Apache-2.0
#
# Daily update script for mirror.starlingx.cengn.ca covering
# tarballs and other files not downloaded from a yum repository.
# This script was originated by Scott Little.
#
# IMPORTANT: This script is only to be run on the StarlingX mirror.
# It is not for use by the general StarlinX developer.
#
# This script was originated by Scott Little.
#
DAILY_DL_SYNC_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
if [ -f "$DAILY_DL_SYNC_DIR/url_utils.sh" ]; then
source "$DAILY_DL_SYNC_DIR/url_utils.sh"
elif [ -f "$DAILY_DL_SYNC_DIR/../url_utils.sh" ]; then
source "$DAILY_DL_SYNC_DIR/../url_utils.sh"
else
echo "Error: Can't find 'url_utils.sh'"
exit 1
fi
LOGFILE=/export/log/daily_dl_sync.log
DOWNLOAD_PATH_ROOT=/export/mirror/centos
STX_TOOLS_BRANCH="master"
STX_TOOLS_BRANCH_ROOT_DIR="$HOME"
STX_TOOLS_GIT_URL="https://git.starlingx.io/stx-tools.git"
STX_TOOLS_OS_SUBDIR="centos-mirror-tools"
usage () {
echo "$0 [-b <branch>] [-d <dir>]"
echo ""
echo "Options:"
echo " -b: Use an alternate branch of stx-tools. Default is 'master'."
echo " -d: Directory where we will clone stx-tools. Default is \$HOME."
echo ""
}
while getopts "b:d:h" opt; do
case "${opt}" in
b)
# branch
STX_TOOLS_BRANCH="${OPTARG}"
if [ $"STX_TOOLS_BRANCH" == "" ]; then
usage
exit 1
fi
;;
d)
# download directory for stx-tools
STX_TOOLS_BRANCH_ROOT_DIR="${OPTARG}"
if [ "$STX_TOOLS_BRANCH_ROOT_DIR" == "" ]; then
usage
exit 1
fi
;;
h)
# Help
usage
exit 0
;;
*)
usage
exit 1
;;
esac
done
STX_TOOLS_DL_ROOT_DIR="$STX_TOOLS_BRANCH_ROOT_DIR/$STX_TOOLS_BRANCH"
STX_TOOLS_DL_DIR="$STX_TOOLS_DL_ROOT_DIR/stx-tools"
LST_FILE_DIR="$STX_TOOLS_DL_DIR/$STX_TOOLS_OS_SUBDIR"
dl_git_from_url () {
local GIT_URL="$1"
local BRANCH="$2"
local DL_DIR="$3"
local DL_ROOT_DIR=""
local SAVE_DIR
local CMD=""
SAVE_DIR="$(pwd)"
if [ "$DL_DIR" == "" ]; then
DL_DIR="$DOWNLOAD_PATH_ROOT/$(repo_url_to_sub_path "$GIT_URL" | sed 's#[.]git$##')"
fi
echo "dl_git_from_url GIT_URL='$GIT_URL' BRANCH='$BRANCH' DL_DIR='$DL_DIR'"
DL_ROOT_DIR=$(dirname "$DL_DIR")
if [ ! -d "$DL_DIR" ]; then
if [ ! -d "$DL_ROOT_DIR" ]; then
CMD="mkdir -p '$DL_ROOT_DIR'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
fi
CMD="cd '$DL_ROOT_DIR'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
CMD="git clone --bare '$GIT_URL' '$DL_DIR'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
CMD="cd '$DL_DIR'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
CMD="git --bare update-server-info"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
if [ -f hooks/post-update.sample ]; then
CMD="mv -f hooks/post-update.sample hooks/post-update"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
fi
fi
CMD="cd '$DL_DIR'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
CMD="git fetch"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
CMD="git checkout '$BRANCH'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
cd "$SAVE_DIR"
return 1
fi
cd "$SAVE_DIR"
return 0
}
dl_file_from_url () {
local URL="$1"
local DOWNLOAD_PATH=""
local DOWNLOAD_DIR=""
local PROTOCOL=""
local CMD=""
DOWNLOAD_PATH="$DOWNLOAD_PATH_ROOT/$(repo_url_to_sub_path "$URL")"
DOWNLOAD_DIR="$(dirname "$DOWNLOAD_PATH")"
PROTOCOL=$(url_protocol $URL)
echo "$PROTOCOL $URL $DOWNLOAD_PATH"
if [ -f "$DOWNLOAD_PATH" ]; then
echo "Already have '$DOWNLOAD_PATH'"
return 0
fi
case "$PROTOCOL" in
https|http)
if [ ! -d "$DOWNLOAD_DIR" ]; then
CMD="mkdir -p '$DOWNLOAD_DIR'"
echo "$CMD"
eval "$CMD"
if [ $? -ne 0 ]; then
echo "Error: $CMD"
return 1
fi
fi
CMD="wget '$URL' --tries=5 --wait=15 --output-document='$DOWNLOAD_PATH'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
return 1
fi
;;
*)
echo "Error: Unknown protocol '$PROTOCOL' for url '$URL'"
;;
esac
return 0
}
raw_dl_from_rpm_lst () {
local FILE="$1"
local RPM=""
local URL=""
local ERROR_COUNT=0
# Expected format <rpm>#<url>
grep -v '^#' $FILE | while IFS='#' read -r RPM URL; do
echo "Processing: RPM=$RPM URL=$URL"
dl_file_from_url "$URL"
ERR_COUNT=$((ERR_COUNT+$?))
done
return $ERR_COUNT
}
raw_dl_from_non_rpm_lst () {
local FILE="$1"
local TAR=""
local URL=""
local METHOD=""
local UTIL=""
local SCRIPT=""
local BRANCH=""
local SUBDIRS_FILE=""
local TARBALL_NAME=""
local ERROR_COUNT=0
# Expected format <tar-file>#<tar-dir>#<url>
# or !<tar-file>#<tar-dir>#<url>#<method>#[<util>]#[<script>]
grep -v '^#' $FILE | while IFS='#' read -r TAR DIR URL METHOD UTIL SCRIPT; do
if [ "$URL" == "" ]; then
continue
fi
echo "Processing: TAR=$TAR DIR=$DIR URL=$URL METHOD=$METHOD UTIL=$UTIL SCRIPT=$SCRIPT"
TARBALL_NAME="${TAR//!/}"
if [[ "$TAR" =~ ^'!' ]]; then
case $METHOD in
http|http_script)
dl_file_from_url "$URL"
if [ $? -ne 0 ]; then
echo "Error: Failed to download '$URL' while processing '$TARBALL_NAME'"
ERR_COUNT=$((ERR_COUNT+1))
fi
;;
http_filelist|http_filelist_script)
SUBDIRS_FILE="$LST_FILE_DIR/$UTIL"
if [ ! -f "$SUBDIRS_FILE" ]; then
echo "$SUBDIRS_FILE no found" 1>&2
ERR_COUNT=$((ERR_COUNT+1))
fi
grep -v '^#' "$SUBDIRS_FILE" | while read -r ARTF; do
if [ "$ARTF" == "" ]; then
continue
fi
dl_file_from_url "$URL/$ARTF"
if [ $? -ne 0 ]; then
echo "Error: Failed to download artifact '$ARTF' from list '$SUBDIRS_FILE' while processing '$TARBALL_NAME'"
ERR_COUNT=$((ERR_COUNT+1))
break
fi
done
;;
git|git_script)
BRANCH="$UTIL"
dl_git_from_url "$URL" "$BRANCH" ""
if [ $? -ne 0 ]; then
echo "Error: Failed to download '$URL' while processing '$TARBALL_NAME'"
ERR_COUNT=$((ERR_COUNT+1))
fi
;;
*)
echo "Error: Unknown method '$METHOD' while processing '$TARBALL_NAME'"
ERR_COUNT=$((ERR_COUNT+1))
;;
esac
else
dl_file_from_url "$URL"
if [ $? -ne 0 ]; then
echo "Error: Failed to download '$URL' while processing '$TARBALL_NAME'"
ERR_COUNT=$((ERR_COUNT+1))
fi
fi
done
return $ERR_COUNT
}
stx_tool_clone_or_update () {
local CMD
CMD="mkdir -p '$STX_TOOLS_DL_DIR'"
echo "$CMD"
eval "$CMD"
if [ $? -ne 0 ]; then
echo "Error: $CMD"
return 1
fi
dl_git_from_url "$STX_TOOLS_GIT_URL" "$STX_TOOLS_BRANCH" "$STX_TOOLS_DL_DIR"
if [ $? -ne 0 ]; then
echo "Error: Failed to download '$STX_TOOLS_GIT_URL'"
return 1;
fi
return 0
}
if [ -f $LOGFILE ]; then
rm -f $LOGFILE
fi
(
ERR_COUNT=0
stx_tool_clone_or_update
if [ $? -ne 0 ]; then
echo "Error: Failed to update stx_tools. Can't continue."
exit 1
fi
# At time of writing, only expect rpms_3rdparties.lst
RPM_LST_FILES=$(grep -l '://' $LST_FILE_DIR/rpms*.lst)
# At time of writing, only expect tarball-dl.lst
NON_RPM_FILES=$(grep -l '://' $LST_FILE_DIR/*lst | grep -v '[/]rpms[^/]*$')
for RPM_LST_FILE in $RPM_LST_FILES; do
raw_dl_from_rpm_lst "$RPM_LST_FILE"
ERR_COUNT=$((ERR_COUNT+$?))
done
for NON_RPM_FILE in $NON_RPM_FILES; do
raw_dl_from_non_rpm_lst "$NON_RPM_FILE"
ERR_COUNT=$((ERR_COUNT+$?))
done
if [ $ERR_COUNT -ne 0 ]; then
echo "Error: Failed to download $ERR_COUNT files"
exit 1
fi
exit 0
) | tee $LOGFILE
exit ${PIPESTATUS[0]}

129
centos-mirror-tools/stx_mirror_scripts/daily_repo_sync.sh Executable file
View File

@ -0,0 +1,129 @@
#!/bin/bash
#
# SPDX-License-Identifier: Apache-2.0
#
# Daily update script for mirror.starlingx.cengn.ca covering
# rpms and src.rpms downloaded from a yum repository.
#
# IMPORTANT: This script is only to be run on the StarlingX mirror.
# It is not for use by the general StarlinX developer.
#
# Configuration files for repositories to be downloaded are currently
# stored at mirror.starlingx.cengn.ca:/export/config/yum.repos.d.
# Those repos were derived from stx-tools/centos-mirror-tools/yum.repos.d
# with some modifications that will need to be automated in a
# future update.
#
# This script was originated by Scott Little.
#
LOGFILE="/export/log/daily_repo_sync.log"
YUM_CONF_DIR="/export/config"
YUM_REPOS_DIR="$YUM_CONF_DIR/yum.repos.d"
DOWNLOAD_PATH_ROOT="/export/mirror/centos"
URL_UTILS="url_utils.sh"
DAILY_REPO_SYNC_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
if [ -f "$DAILY_REPO_SYNC_DIR/$URL_UTILS" ]; then
source "$DAILY_REPO_SYNC_DIR/$URL_UTILS"
elif [ -f "$DAILY_REPO_SYNC_DIR/../$URL_UTILS" ]; then
source "$DAILY_REPO_SYNC_DIR/../$URL_UTILS"
else
echo "Error: Can't find '$URL_UTILS'"
exit 1
fi
CREATEREPO=$(which createrepo_c)
if [ $? -ne 0 ]; then
CREATEREPO="createrepo"
fi
number_of_cpus () {
/usr/bin/nproc
}
if [ -f $LOGFILE ]; then
rm -f $LOGFILE
fi
ERR_COUNT=0
YUM_CONF="$YUM_CONF_DIR/yum.conf"
if [ ! -f "$YUM_CONF" ]; then
echo "Error: Missing yum.conf file at '$YUM_CONF'"
exit 1
fi
for REPO in $(find $YUM_REPOS_DIR -name '*.repo'); do
for REPO_ID in $(grep '^[[]' $REPO | sed 's#[][]##g'); do
REPO_URL=$(yum repoinfo --config="$YUM_CONF" --disablerepo="*" --enablerepo="$REPO_ID" | grep Repo-baseurl | cut -d ' ' -f 3)
DOWNLOAD_PATH="$DOWNLOAD_PATH_ROOT/$(repo_url_to_sub_path "$REPO_URL")"
echo "Processing: REPO=$REPO REPO_ID=$REPO_ID REPO_URL=$REPO_URL DOWNLOAD_PATH=$DOWNLOAD_PATH"
# Assume it's a repo of binary rpms unless repoid ends in
# some variation of 'source'.
SOURCE_FLAG=""
echo "$REPO_ID" | grep -q '[-_][Ss]ource$' && SOURCE_FLAG="--source"
echo "$REPO_ID" | grep -q '[-_][Ss]ources$' && SOURCE_FLAG="--source"
echo "$REPO_ID" | grep -q '[-_][Ss]ource[-_]' && SOURCE_FLAG="--source"
echo "$REPO_ID" | grep -q '[-_][Ss]ources[-_]' && SOURCE_FLAG="--source"
if [ ! -d "$DOWNLOAD_PATH" ]; then
CMD="mkdir -p '$DOWNLOAD_PATH'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
ERR_COUNT=$((ERR_COUNT+1))
continue
fi
fi
CMD="reposync --norepopath $SOURCE_FLAG -l --config=$YUM_CONF --repoid=$REPO_ID --download_path='$DOWNLOAD_PATH'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
ERR_COUNT=$((ERR_COUNT+1))
continue
fi
CMD="pushd '$DOWNLOAD_PATH'"
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
ERR_COUNT=$((ERR_COUNT+1))
continue
fi
OPTIONS="--workers $(number_of_cpus)"
if [ -f comps.xml ]; then
OPTIONS="$OPTIONS -g comps.xml"
fi
if [ -d repodata ]; then
OPTIONS="$OPTIONS --update"
fi
CMD="$CREATEREPO $OPTIONS ."
echo "$CMD"
eval $CMD
if [ $? -ne 0 ]; then
echo "Error: $CMD"
ERR_COUNT=$((ERR_COUNT+1))
popd
continue
fi
popd
done
done | tee $LOGFILE
if [ $ERR_COUNT -ne 0 ]; then
exit 1
fi
exit 0

100
centos-mirror-tools/tarball-dl.lst
View File

@ -1,50 +1,50 @@
dpkg_1.18.24.tar.xz#dpkg-1.18.24#http://http.debian.net/debian/pool/main/d/dpkg/dpkg_1.18.24.tar.xz
drbd-8.4.3.tar.gz#drbd-8.4.3#http://www.linbit.com/downloads/drbd/8.4/archive/drbd-8.4.3.tar.gz
drbd-8.4.7-1.tar.gz#drbd-8.4.7-1#http://www.linbit.com/downloads/drbd/8.4/drbd-8.4.7-1.tar.gz
dtc-1.4.4.tar.gz#dtc-1.4.4#https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.4.tar.gz
e1000e-3.4.1.1.tar.gz#e1000e-3.4.1.1#https://sourceforge.net/projects/e1000/files/e1000e%20stable/3.4.1.1/e1000e-3.4.1.1.tar.gz
gnulib-ffc927e.tar.gz#gnulib-ffc927e#http://git.savannah.gnu.org/cgit/gnulib.git/snapshot/gnulib-ffc927eef29016a5219cd969daad8928af6a1f4d.tar.gz
i40e-2.4.10.tar.gz#i40e-2.4.10#https://sourceforge.net/projects/e1000/files/i40e%20stable/2.4.10/i40e-2.4.10.tar.gz/download
i40evf-3.5.13.tar.gz#i40evf-3.5.13#https://sourceforge.net/projects/e1000/files/i40evf%20stable/3.5.13/i40evf-3.5.13.tar.gz/download
ibsh-0.3e.tar.gz#cgcs-users-1.0#https://sourceforge.net/projects/ibsh/files/ibsh/ibsh-0.3e/ibsh-0.3e.tar.gz/download
!integrity-kmod-e6aef069.tar.gz#integrity#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz
ixgbe-5.3.7.tar.gz#ixgbe-5.3.7#https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.3.7/ixgbe-5.3.7.tar.gz/download
ixgbevf-4.3.5.tar.gz#ixgbevf-4.3.5#https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/4.3.5/ixgbevf-4.3.5.tar.gz/download
keycodemapdb-16e5b07.tar.gz#keycodemapdb#https://github.com/CendioOssman/keycodemapdb/tarball/16e5b0787687d8904dad2c026107409eb9bfcb95
kvm-unit-tests.git-4ea7633.tar.bz2#kvm-unit-tests#https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git/snapshot/kvm-unit-tests-20171020.tar.gz
ldapscripts-2.0.8.tgz#ldapscripts-2.0.8#https://sourceforge.net/projects/ldapscripts/files/ldapscripts/ldapscripts-2.0.8/ldapscripts-2.0.8.tgz/download
libtpms-0.6.0-4f0d59d.tar.gz#libtpms-0.6.0#https://github.com/stefanberger/libtpms/tarball/c421ca0f4d00c0caceeda8d62c1efb2b7e47db04
lldpd-0.9.0.tar.gz#lldpd-0.9.0#https://media.luffy.cx/files/lldpd/lldpd-0.9.0.tar.gz
!mariadb-10.1.28.tar.gz#mariadb-10.1.28#https://github.com/MariaDB/server/archive/mariadb-10.1.28.tar.gz
!MLNX_OFED_LINUX-4.2-1.2.0.0-rhel7.4-x86_64.tgz#MLNX_OFED_LINUX-4.2-1.2.0.0-rhel7.4-x86_64#http://www.mellanox.com/downloads/ofed/MLNX_OFED-4.2-1.2.0.0/MLNX_OFED_LINUX-4.2-1.2.0.0-rhel7.4-x86_64.tgz
!MLNX_OFED_LINUX-4.3-1.0.1.0-rhel7.4-x86_64.tgz#MLNX_OFED_LINUX-4.3-1.0.1.0-rhel7.4-x86_64#http://www.mellanox.com/downloads/ofed/MLNX_OFED-4.3-1.0.1.0/MLNX_OFED_LINUX-4.3-1.0.1.0-rhel7.4-x86_64.tgz
!MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64.tgz#MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64#http://www.mellanox.com/downloads/ofed/MLNX_OFED-4.3-3.0.2.1/MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64.tgz
!mvn.repo.tgz#mvn#https://repo.maven.apache.org/maven2
python-cephclient-v0.1.0.5.tar.gz#python-cephclient-0.1.0.5#https://github.com/dmsimard/python-cephclient/archive/v0.1.0.5.tar.gz
python-setuptools-v38.5.1.tar.gz#setuptools-38.5.1#https://github.com/pypa/setuptools/archive/v38.5.1.tar.gz
python-smartpm-1.4.1.tar.gz#smart-1.4.1#http://launchpad.net/smart/trunk/1.4.1/+download/smart-1.4.1.tar.bz2
!qat1.7.upstream.l.1.0.3-42.tar.gz#quickassist#https://01.org/sites/default/files/downloads/intelr-quickassist-technology/qat1.7.upstream.l.1.0.3-42.tar.gz
requests-toolbelt-0.5.1.tar.gz#requests-toolbelt-0.5.1#https://github.com/requests/toolbelt/archive/0.5.1.tar.gz
rpm-4.14.0.tar.bz2#rpm-4.14.0#https://ftp.osuosl.org/pub/rpm/releases/rpm-4.14.x/rpm-4.14.0.tar.bz2
swtpm-0.1.0-253eac5.tar.gz#swtpm-0.1.0#https://github.com/stefanberger/swtpm/tarball/1303be7d03294fb02204cb8242f366cbf0da076d
!tpm-kmod-e6aef069.tar.gz#tpm#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz
!tss2-930.tar.gz#tss2-930#https://sourceforge.net/code-snapshots/git/i/ib/ibmtpm20tss/tss.git/ibmtpm20tss-tss-52539cb81c811c973b26ed23fafd28a700b7cc78.zip
spectre-meltdown-checker-0.37+-5cc77741.tar.gz#spectre-meltdown-checker#https://github.com/speed47/spectre-meltdown-checker/tarball/5cc77741af1d2f52140aa9f89339f56b6c4b6783
puppet-boolean-22b726dd78b0a60a224cc7054aebbf28e9306f62.tar.gz#puppet-boolean#https://github.com/voxpupuli/puppet-boolean/tarball/22b726dd78b0a60a224cc7054aebbf28e9306f62
puppet-dnsmasq-cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packstack/puppet/modules/dnsmasq/#https://github.com/procore/puppet-dnsmasq/tarball/cff07e90890662972c97684a2baee964f68ff3ed
puppet-filemapper-9b53310278e76827bbe12a36cc6470d77071abb2.tar.gz#packstack/puppet/modules/filemapper#https://github.com/voxpupuli/puppet-filemapper/tarball/9b53310278e76827bbe12a36cc6470d77071abb2
puppetlabs-create_resources-4639819a7f3a4fa9310d2ba583c63e467df7e2c3.tar.gz#packstack/puppet/modules/create_resources#https://github.com/puppetlabs/puppetlabs-create_resources/tarball/4639819a7f3a4fa9310d2ba583c63e467df7e2c3
puppetlabs-drbd-496b3ba9cd74a2d12636f9e90a718739a5451169.tar.gz#puppetlabs-drbd#https://github.com/voxpupuli/puppet-drbd/tarball/496b3ba9cd74a2d12636f9e90a718739a5451169
puppetlabs-lvm-d0283da637ae24550fb4ba109a48ef8d5d8c8b84.tar.gz#packstack/puppet/modules/lvm#https://github.com/puppetlabs/puppetlabs-lvm/tarball/d0283da637ae24550fb4ba109a48ef8d5d8c8b84
puppetlabs-postgresql-d022a56b28b2174456fc0f6adc51a4b54493afad.tar.gz#puppetlabs-postgresql#https://github.com/puppetlabs/puppetlabs-postgresql/tarball/d022a56b28b2174456fc0f6adc51a4b54493afad
puppet-ldap-480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1.tar.gz#puppet-ldap-master#https://github.com/torian/puppet-ldap/tarball/480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1
puppet-network-7deacd5fdc22c0543455878a8d1872f2f5417c1d.tar.gz#packstack/puppet/modules/network#https://github.com/voxpupuli/puppet-network/tarball/7deacd5fdc22c0543455878a8d1872f2f5417c1d
puppet-nslcd-b8c19b1ada89865f2e50758e054583798ad8011a.tar.gz#packstack/puppet/modules/nslcd#https://github.com/jlyheden/puppet-nslcd/tarball/b8c19b1ada89865f2e50758e054583798ad8011a
puppi-c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81.tar.gz#puppi-master#https://github.com/example42/puppi/tarball/c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81
helm-v2.11.0-linux-amd64.tar.gz#linux-amd64#https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz
openstack-helm-b9fab949aaf9627be59487c35156428ca12f7442.tar.gz#openstack-helm#https://github.com/openstack/openstack-helm/archive/b9fab949aaf9627be59487c35156428ca12f7442.tar.gz
openstack-helm-infra-d9457c8860e3e20ff84b53b5de71b5889043cf04.tar.gz#openstack-helm-infra#https://github.com/openstack/openstack-helm-infra/archive/d9457c8860e3e20ff84b53b5de71b5889043cf04.tar.gz
gnocchi-4.2.5.tar.gz#gnocchi-4.2.5#https://pypi.io/packages/source/g/gnocchi/gnocchi-4.2.5.tar.gz
gnocchiclient-7.0.1.tar.gz#gnocchiclient-7.0.1#https://pypi.io/packages/source/g/gnocchiclient/gnocchiclient-7.0.1.tar.gz
kubernetes-v1.12.1.tar.gz#kubernetes-1.12.1#https://github.com/kubernetes/kubernetes/archive/v1.12.1.tar.gz
kubernetes-contrib-v1.12.1.tar.gz#kubernetes-contrib-1.12.1#https://github.com/kubernetes/contrib/tarball/4b2a8512a95611acb174bd306e501719ce70b1cf
dpkg_1.18.24.tar.xz#dpkg-1.18.24#http://http.debian.net/debian/pool/main/d/dpkg/dpkg_1.18.24.tar.xz#http##
drbd-8.4.3.tar.gz#drbd-8.4.3#http://www.linbit.com/downloads/drbd/8.4/archive/drbd-8.4.3.tar.gz#http##
drbd-8.4.7-1.tar.gz#drbd-8.4.7-1#http://www.linbit.com/downloads/drbd/8.4/drbd-8.4.7-1.tar.gz#http##
dtc-1.4.4.tar.gz#dtc-1.4.4#https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.4.tar.gz#http##
e1000e-3.4.1.1.tar.gz#e1000e-3.4.1.1#https://sourceforge.net/projects/e1000/files/e1000e%20stable/3.4.1.1/e1000e-3.4.1.1.tar.gz#http##
gnulib-ffc927e.tar.gz#gnulib-ffc927e#http://git.savannah.gnu.org/cgit/gnulib.git/snapshot/gnulib-ffc927eef29016a5219cd969daad8928af6a1f4d.tar.gz#http##
i40e-2.4.10.tar.gz#i40e-2.4.10#https://sourceforge.net/projects/e1000/files/i40e%20stable/2.4.10/i40e-2.4.10.tar.gz/download#http##
i40evf-3.5.13.tar.gz#i40evf-3.5.13#https://sourceforge.net/projects/e1000/files/i40evf%20stable/3.5.13/i40evf-3.5.13.tar.gz/download#http##
ibsh-0.3e.tar.gz#cgcs-users-1.0#https://sourceforge.net/projects/ibsh/files/ibsh/ibsh-0.3e/ibsh-0.3e.tar.gz/download#http##
!integrity-kmod-e6aef069.tar.gz#integrity#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz#http_script##post-dl-script/integrity-kmod.sh
ixgbe-5.3.7.tar.gz#ixgbe-5.3.7#https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.3.7/ixgbe-5.3.7.tar.gz/download#http##
ixgbevf-4.3.5.tar.gz#ixgbevf-4.3.5#https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/4.3.5/ixgbevf-4.3.5.tar.gz/download#http##
keycodemapdb-16e5b07.tar.gz#keycodemapdb#https://github.com/CendioOssman/keycodemapdb/tarball/16e5b0787687d8904dad2c026107409eb9bfcb95#http##
kvm-unit-tests.git-4ea7633.tar.bz2#kvm-unit-tests#https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git/snapshot/kvm-unit-tests-20171020.tar.gz#http##
ldapscripts-2.0.8.tgz#ldapscripts-2.0.8#https://sourceforge.net/projects/ldapscripts/files/ldapscripts/ldapscripts-2.0.8/ldapscripts-2.0.8.tgz/download#http##
libtpms-0.6.0-4f0d59d.tar.gz#libtpms-0.6.0#https://github.com/stefanberger/libtpms/tarball/c421ca0f4d00c0caceeda8d62c1efb2b7e47db04#http##
lldpd-0.9.0.tar.gz#lldpd-0.9.0#https://media.luffy.cx/files/lldpd/lldpd-0.9.0.tar.gz#http##
!mariadb-10.1.28.tar.gz#mariadb-10.1.28#https://github.com/MariaDB/server/archive/mariadb-10.1.28.tar.gz#http_script##post-dl-script/mariadb.sh
!MLNX_OFED_LINUX-4.2-1.2.0.0-rhel7.4-x86_64.tgz#MLNX_OFED_LINUX-4.2-1.2.0.0-rhel7.4-x86_64#http://www.mellanox.com/downloads/ofed/MLNX_OFED-4.2-1.2.0.0/MLNX_OFED_LINUX-4.2-1.2.0.0-rhel7.4-x86_64.tgz#http_script##post-dl-script/MLNX_OFED.sh
!MLNX_OFED_LINUX-4.3-1.0.1.0-rhel7.4-x86_64.tgz#MLNX_OFED_LINUX-4.3-1.0.1.0-rhel7.4-x86_64#http://www.mellanox.com/downloads/ofed/MLNX_OFED-4.3-1.0.1.0/MLNX_OFED_LINUX-4.3-1.0.1.0-rhel7.4-x86_64.tgz#http_script##post-dl-script/MLNX_OFEDqat1.7.sh
!MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64.tgz#MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64#http://www.mellanox.com/downloads/ofed/MLNX_OFED-4.3-3.0.2.1/MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64.tgz#http_script##post-dl-script/MLNX_OFEDqat1.7.sh
!mvn.repo.tgz#mvn#https://repo.maven.apache.org/maven2#http_filelist#mvn-artifacts.lst#
python-cephclient-v0.1.0.5.tar.gz#python-cephclient-0.1.0.5#https://github.com/dmsimard/python-cephclient/archive/v0.1.0.5.tar.gz#http##
python-setuptools-v38.5.1.tar.gz#setuptools-38.5.1#https://github.com/pypa/setuptools/archive/v38.5.1.tar.gz#http##
python-smartpm-1.4.1.tar.gz#smart-1.4.1#http://launchpad.net/smart/trunk/1.4.1/+download/smart-1.4.1.tar.bz2#http##
!qat1.7.upstream.l.1.0.3-42.tar.gz#quickassist#https://01.org/sites/default/files/downloads/intelr-quickassist-technology/qat1.7.upstream.l.1.0.3-42.tar.gz#http_script##post-dl-script/qat1.7.sh
requests-toolbelt-0.5.1.tar.gz#requests-toolbelt-0.5.1#https://github.com/requests/toolbelt/archive/0.5.1.tar.gz#http##
rpm-4.14.0.tar.bz2#rpm-4.14.0#https://ftp.osuosl.org/pub/rpm/releases/rpm-4.14.x/rpm-4.14.0.tar.bz2#http##
swtpm-0.1.0-253eac5.tar.gz#swtpm-0.1.0#https://github.com/stefanberger/swtpm/tarball/1303be7d03294fb02204cb8242f366cbf0da076d#http##
!tpm-kmod-e6aef069.tar.gz#tpm#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz#http_script#post-dl-script/tpm-kmod.sh
!tss2-930.tar.gz#tss2-930#https://git.code.sf.net/p/ibmtpm20tss/tss#git#v930#
spectre-meltdown-checker-0.37+-5cc77741.tar.gz#spectre-meltdown-checker#https://github.com/speed47/spectre-meltdown-checker/tarball/5cc77741af1d2f52140aa9f89339f56b6c4b6783#http##
puppet-boolean-22b726dd78b0a60a224cc7054aebbf28e9306f62.tar.gz#puppet-boolean#https://github.com/voxpupuli/puppet-boolean/tarball/22b726dd78b0a60a224cc7054aebbf28e9306f62#http##
puppet-dnsmasq-cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packstack/puppet/modules/dnsmasq/#https://github.com/procore/puppet-dnsmasq/tarball/cff07e90890662972c97684a2baee964f68ff3ed#http##
puppet-filemapper-9b53310278e76827bbe12a36cc6470d77071abb2.tar.gz#packstack/puppet/modules/filemapper#https://github.com/voxpupuli/puppet-filemapper/tarball/9b53310278e76827bbe12a36cc6470d77071abb2#http##
puppetlabs-create_resources-4639819a7f3a4fa9310d2ba583c63e467df7e2c3.tar.gz#packstack/puppet/modules/create_resources#https://github.com/puppetlabs/puppetlabs-create_resources/tarball/4639819a7f3a4fa9310d2ba583c63e467df7e2c3#http##
puppetlabs-drbd-496b3ba9cd74a2d12636f9e90a718739a5451169.tar.gz#puppetlabs-drbd#https://github.com/voxpupuli/puppet-drbd/tarball/496b3ba9cd74a2d12636f9e90a718739a5451169#http##
puppetlabs-lvm-d0283da637ae24550fb4ba109a48ef8d5d8c8b84.tar.gz#packstack/puppet/modules/lvm#https://github.com/puppetlabs/puppetlabs-lvm/tarball/d0283da637ae24550fb4ba109a48ef8d5d8c8b84#http##
puppetlabs-postgresql-d022a56b28b2174456fc0f6adc51a4b54493afad.tar.gz#puppetlabs-postgresql#https://github.com/puppetlabs/puppetlabs-postgresql/tarball/d022a56b28b2174456fc0f6adc51a4b54493afad#http##
puppet-ldap-480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1.tar.gz#puppet-ldap-master#https://github.com/torian/puppet-ldap/tarball/480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1#http##
puppet-network-7deacd5fdc22c0543455878a8d1872f2f5417c1d.tar.gz#packstack/puppet/modules/network#https://github.com/voxpupuli/puppet-network/tarball/7deacd5fdc22c0543455878a8d1872f2f5417c1d#http##
puppet-nslcd-b8c19b1ada89865f2e50758e054583798ad8011a.tar.gz#packstack/puppet/modules/nslcd#https://github.com/jlyheden/puppet-nslcd/tarball/b8c19b1ada89865f2e50758e054583798ad8011a#http##
puppi-c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81.tar.gz#puppi-master#https://github.com/example42/puppi/tarball/c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81#http##
helm-v2.11.0-linux-amd64.tar.gz#linux-amd64#https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz#http##
openstack-helm-b9fab949aaf9627be59487c35156428ca12f7442.tar.gz#openstack-helm#https://github.com/openstack/openstack-helm/archive/b9fab949aaf9627be59487c35156428ca12f7442.tar.gz#http##
openstack-helm-infra-d9457c8860e3e20ff84b53b5de71b5889043cf04.tar.gz#openstack-helm-infra#https://github.com/openstack/openstack-helm-infra/archive/d9457c8860e3e20ff84b53b5de71b5889043cf04.tar.gz#http##
gnocchi-4.2.5.tar.gz#gnocchi-4.2.5#https://pypi.io/packages/source/g/gnocchi/gnocchi-4.2.5.tar.gz#http##
gnocchiclient-7.0.1.tar.gz#gnocchiclient-7.0.1#https://pypi.io/packages/source/g/gnocchiclient/gnocchiclient-7.0.1.tar.gz#http##
kubernetes-v1.12.1.tar.gz#kubernetes-1.12.1#https://github.com/kubernetes/kubernetes/archive/v1.12.1.tar.gz#http##
kubernetes-contrib-v1.12.1.tar.gz#kubernetes-contrib-1.12.1#https://github.com/kubernetes/contrib/tarball/4b2a8512a95611acb174bd306e501719ce70b1cf#http##

266
centos-mirror-tools/url_utils.sh Executable file
View File

@ -0,0 +1,266 @@
#!/bin/bash
#
# SPDX-License-Identifier: Apache-2.0
#
#
# A set of bash utility functions to parse a URL.
# This script was originated by Scott Little
#
url_protocol () {
local URL="$1"
if [ "$URL" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
echo "$URL" | sed 's#^\(.*\)://.*$#\1#'
return 0
}
url_login () {
local URL="$1"
if [ "$URL" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
echo "$URL" | sed 's#^.*://\([^/]*\)/.*$#\1#'
return 0
}
url_user () {
local URL="$1"
local LOGIN
if [ "$URL" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
url_login "$URL" | sed -e '/@/! s#.*## ; s#\([^@]*\)@.*#\1#'
if [ ${PIPESTATUS[0]} -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): url_login failed"
return 1
fi
return 0
}
url_port () {
local URL="$1"
local LOGIN
if [ "$URL" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
url_login "$URL" | sed -e '/:/! s#.*## ; s#[^:]*:\([^:]*\)#\1#'
if [ ${PIPESTATUS[0]} -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): url_login failed"
return 1
fi
return 0
}
url_server () {
local URL="$1"
local LOGIN
if [ "$URL" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
url_login "$URL" | sed 's#^.*@## ; s#:.*$##'
if [ ${PIPESTATUS[0]} -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): url_login failed"
return 1
fi
return 0
}
url_path () {
local URL="$1"
if [ "$URL" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
echo "$URL" | sed 's#^.*://[^/]*/\(.*\)$#\1#'
return 0
}
#
# url_path_to_fs_path:
#
# Convert url format path to file system format.
# e.g. replace %20 with ' '.
#
# Note: Does NOT test the output path to ensure there are
# no illegal file system characters.
#
url_path_to_fs_path () {
local INPUT_PATH="$1"
local TEMP
if [ "$INPUT_PATH" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
# Deviate from URI spec by not substituding '+' with ' '.
# It would alias '%20' and we need unique mappings.
# TEMP="${INPUT_PATH//+/ }"
TEMP="$INPUT_PATH"
printf '%b' "${TEMP//%/\\x}"
return 0
}
#
# fs_path_to_url_path:
#
# Convert file system format path to url format.
# e.g. replace ' ' with %20.
#
fs_path_to_url_path () {
local INPUT_PATH="$1"
local LENGTH
local POS
local CHAR
if [ "$INPUT_PATH" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
LENGTH="${#INPUT_PATH}"
for (( POS = 0; POS < LENGTH; POS++ )); do
CHAR="${1:POS:1}"
case $CHAR in
[/a-zA-Z0-9.~_-])
# Reference https://metacpan.org/pod/URI::Escape
printf "$CHAR"
;;
*)
printf '%%%02X' "'$CHAR"
;;
esac
done
return 0
}
#
# normalize_path:
#
# 1) replace // with /
# 2) replace /./ with /
# 3) Remove trailing /
# 4) Remove leading ./
#
normalize_path () {
local INPUT_PATH="$1"
if [ "$INPUT_PATH" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
echo "$INPUT_PATH" | sed 's#[/]\+#/#g ; s#[/][.][/]#/#g ; s#/$## ; s#^[.]/##'
return 0
}
#
# repo_url_to_sub_path:
#
repo_url_to_sub_path () {
local URL="$1"
local FAMILY=""
local SERVER=""
local URL_PATH=""
local FS_PATH=""
if [ "$URL" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
# set FAMILY from URL
echo $URL | grep -q 'centos[.]org' && FAMILY=centos
echo $URL | grep -q 'fedoraproject[.]org[/]pub[/]epel' && FAMILY=epel
SERVER=$(url_server "$URL")
if [ $? -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): url_server '$URL'"
return 1
fi
URL_PATH="$(url_path "$URL")"
if [ $? -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): url_path '$URL'"
return 1
fi
FS_PATH="$(url_path_to_fs_path "$URL_PATH")"
if [ $? -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): url_path_to_fs_path '$URL_PATH'"
return 1
fi
FS_PATH="$(normalize_path "$FS_PATH")"
if [ $? -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): normalize_path '$FS_PATH'"
return 1
fi
normalize_path "./$FAMILY/$SERVER/$FS_PATH"
if [ $? -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): normalize_path './$FAMILY/$SERVER/$FS_PATH'"
return 1
fi
return 0
}
CENGN_PROTOCOL="http"
CENGN_HOST="mirror.starlingx.cengn.ca"
CENGN_PORT="80"
CENGN_URL_ROOT="mirror"
url_to_stx_mirror_url () {
local URL="$1"
local DISTRO="$2"
local URL_PATH=""
local FS_PATH=""
if [ "$URL" == "" ] || [ "$DISTRO" == "" ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): empty argument"
return 1
fi
FS_PATH="$(repo_url_to_sub_path "$URL")"
if [ $? -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): repo_url_to_sub_path '$URL'"
return 1
fi
URL_PATH=$(fs_path_to_url_path "$FS_PATH")
if [ $? -ne 0 ]; then
>&2 echo "Error: $FUNCNAME (${LINENO}): fs_path_to_url_path '$FS_PATH'"
return 1
fi
echo "$CENGN_PROTOCOL://$CENGN_HOST:$CENGN_PORT/$CENGN_URL_ROOT/$DISTRO/$URL_PATH"
return 0
}

11
centos-mirror-tools/utils.sh
View File

@ -2,6 +2,10 @@
# SPDX-License-Identifier: Apache-2.0
#
UTILS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $UTILS_DIR/url_utils.sh
get_yum_command() {
local _file=$1
local _level=$2
@ -38,6 +42,8 @@ get_rpm_level_name() {
SFILE=`echo $_rpm_name | rev | cut -d'.' -f3- | rev`
elif [ $_level == "L2" ];then
SFILE=`echo $_rpm_name | rev | cut -d'-' -f2- | rev`
elif [ $_level == "cengn" ];then
SFILE=`echo $_rpm_name | rev | cut -d'-' -f2- | rev`
else
SFILE=`echo $_rpm_name | rev | cut -d'-' -f3- | rev`
fi
@ -53,6 +59,9 @@ get_url() {
_ret="$(koji_url $_name)"
elif [[ "$_name" == *"#"* ]]; then
_ret="$(echo $_name | cut -d'#' -f2-2)"
if [ $_level == "stx_mirror" ]; then
_ret="$(url_to_stx_mirror_url $_ret $distro)"
fi
else
_url_cmd="$(get_yum_command $_name $_level)"
_ret="$($_url_cmd --url)"
@ -155,7 +164,7 @@ get_download_cmd() {
fi
else
# Build wget command
rpm_url=$(echo $ff | cut -d"#" -f2-2)
rpm_url=$(get_url "$ff" "$_level")
download_cmd="$(get_wget_command $rpm_url)"
fi

7
centos-mirror-tools/yum.repos.d/StarlingX-Centos-7.2.repo
View File

@ -77,3 +77,10 @@ baseurl=http://vault.centos.org/centos/7.2.1511/cr/x86_64
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1
[Starlingx-C7.2.1511-cr-source]
name=Starlingx-CentOS-7.2.1511 - cr-source
baseurl=http://vault.centos.org/centos/7.2.1511/cr/Source
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1

6
centos-mirror-tools/yum.repos.d/StarlingX-Centos-7.3.repo
View File

@ -123,3 +123,9 @@ gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1
[Starlingx-C7.3.1611-cr-source]
name=Starlingx-CentOS-7.3.1611 - cr-source
baseurl=http://vault.centos.org/centos/7.3.1611/cr/Source
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1

6
centos-mirror-tools/yum.repos.d/StarlingX-Centos-7.4.repo
View File

@ -129,3 +129,9 @@ gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1
[Starlingx-C7.4.1708-cr-source]
name=Starlingx-CentOS-7.4.1708 - cr-source
baseurl=http://vault.centos.org/centos/7.4.1708/cr/Source
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1

6
centos-mirror-tools/yum.repos.d/StarlingX-EPEL-7.repo
View File

@ -25,21 +25,21 @@ enabled=1
[Starlingx-epel-7-testing]
name=Starlingx-Epel-7
name=Starlingx-Epel-7-testing
baseurl=https://dl.fedoraproject.org/pub/epel/testing/7/x86_64/
gpgcheck=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1
[Starlingx-epel-7-testing-debug]
name=Starlingx-Epel-7
name=Starlingx-Epel-7-testing-debug
baseurl=https://dl.fedoraproject.org/pub/epel/testing/7/x86_64/debug/
gpgcheck=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=1
[Starlingx-epel-7-testing-source]
name=Starlingx-Epel-7-source
name=Starlingx-Epel-7-testing-source
baseurl=https://dl.fedoraproject.org/pub/epel/testing/7/SRPMS/
gpgcheck=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7