Mirror scripts sometimes leave corrupted/partial files behind.
Problems
========
1) wget is called with the -O flag, and the server returns an HTTP
error for the requested URL (404 etc). Wget leaves a zero-length file
behind. This doesn't seem to happen without the -O flag.
2) wget starts the download which stalls & times out half-way; wget
gives up and requests the same file with a byte offset of the form
"Range: bytes=1234-", and the web server doesn't support open-ended
ranges. In this case wget prints out a warning, leaves a partial file
behind and returns success.
3) Sites like GitHub generate repo tarballs on the fly, eg:
https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.19.3.tar.gz
Since tags can move, downloading such a file twice may result in a
different file. Therefore HTTP "resume download" may corrupt files in
this case.
4) Git "keyword expansion" feature may result in differences in source
files being downloaded. For example, this file:
https://github.com/kubernetes/kubernetes/blob/v1.19.3/staging/src/k8s.io/component-base/version/base.go
contains lines similar to:
gitVersion = "v0.0.0-master+$Format:%h$"
where %h is replaced with a short SHA when the tar file is
exported/downloaded. How short the SHA is depends on git history and
sometimes results in shortened SHAs of different lengths. So
downloading that file may result in different files.
Therefore HTTP "Range" header may corrupt files in this case as
well.
5) Curl is invoked with the "--retry" option and starts the download;
connection stalls; curl gives up, connects again, skips the 1st N
bytes and appends to the partial file. If the file changes while we
are doing this, it will end up corrupting the file. This is very
unlikely to happen and I haven't been able to reproduce this case.
Problems with HTTP Range header
===============================
Curl/wget "resume/continue download" feature has no way of verifying
whether the partial file on disk, and the one being re-requested, are in
fact the same file. If the file changes on the server between
downloads, "resume download" will corrupt it.
Some web servers don't support this at all, which triggers case (2)
with wget.
Some web servers support the Range header, but require that the end
byte position is present. This is not compatible with wget & curl.
For example curl & wget add headers similar to: "Range: bytes=1234-"
means give me the file starting at offset 1234 and till EOF. This also
triggers case (2).
This patch
==========
* Always download the file to a temporary name, then rename into place
* Use curl instead wget (better error handling). The only exception is
"recursive downloads", which curl doesn't support.
Bug: https://bugs.launchpad.net/starlingx/+bug/1950017
Change-Id: Iaa89009ce23efe5b73ecb8163556ce6db932028b
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
A refactor was made in the dl_rpms.sh script to simplify the
maintenance and development. A new file utils.sh was created to
store all the common functions that could be used by another scripts.
Also, some unit testing was added to the util.sh file.
Story: 2002736
Task: 26194
Change-Id: Iaac2e74a84abbfc9f40ef51b1da819702d040655
Signed-off-by: Marcela Rosales <marcela.a.rosales.jimenez@intel.com>
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
Davlet Panech