starlingx
/
docs
Code Issues Proposed changes

Merge "Create Docker Registry with Pull Through Cache"

This commit is contained in:
Zuul 2024-01-12 15:04:01 +00:00 committed by Gerrit Code Review
parent 98cf1cb951 97593aaf5a
commit b8d3ae4f68
1 changed files with 331 additions and 161 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

492
doc/source/developer_resources/local_registry.rst
View File

@ -13,174 +13,314 @@ This will also speed up the bootstrap process as images will be downloaded only
once. This guide assumes that you are installing this local external docker
registry on a Linux system with Docker installed and configured.
Optionally, you can also create multiple registries with pull-through cache,
skipping the need to pre-populate the registry and keeping the images and tags
up-to-date.
.. rubric:: |proc|
#. Create folders to store your local registry images (``storage``) and to
place setup files that will be used later on (``images``):
.. code-block:: shell
mkdir -p $HOME/docker-registry/storage
mkdir -p $HOME/docker-registry/images
#. Create a configuration file that will be used by Docker's official
Registry image later on:
.. code-block:: shell
cat > $HOME/docker-registry/config.yml << EOF
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
EOF
#. Run the docker container registry:
.. code-block:: shell
export LOCAL_REG=$HOME/docker-registry
docker run -d \
--restart=always \
--name registry \
-v "$LOCAL_REG"/storage:/var/lib/registry \
-v "$LOCAL_REG"/config.yml:/etc/docker/registry/config.yml \
-e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
-p 5000:5000 \
registry:2
.. tabs::
.. group-tab:: Private Registry
.. note::
*Optional*: the ``-p`` parameter configures a mapping between the host
port and the container port. If you want to listen on another port on
your host, say 9000, change from ``-p 5000:5000 \`` to
``-p 9000:5000 \``.
In this section, you will pre-populate a local registry with multiple
images. Currently, the generated images list will not contain all
images, making it necessary to manually pull some images used by the
kubeadm tool. Alternatively, you can follow the "pull-through cache"
section, allowing docker to get the image from remote, if it doesn't
exists locally.
#. Create the list of images that will populate the registry.
Obtain the Kubernetes version your |prod| uses. This can be found in the
``fresh_install_k8s_version`` value of the `Kubernetes versions`_ file.
Use the branches and tags to find the value for your version.
With the Kubernetes version, you can find the corresponding folder in
`system images`_ and locate the ``system-images.yml`` file. This file
contains the list of images to be loaded into your registry.
To make the list of images for |prod| 8.0, take the raw address of the
corresponding ``system-images.yml`` file and set a variable with it:
.. code-block:: shell
export IMAGES_YAML_RAW_FILE="https://opendev.org/starlingx/ansible-playbooks/raw/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars/k8s-v1.24.4/system-images.yml"
Use the command to generate a ``list.lst`` file:
.. code-block:: shell
curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst
.. note:: *Optional*: If you have a running |prod| setup, you can run
the following to create an Ansible Playbook to get the exact images
you will need instead:
#. Create folders to store your local registry images (``storage``) and to
place setup files that will be used later on (``images``):
.. code-block:: shell
cat > list-images.yml << EOF
---
- hosts: localhost
gather_facts: true
tasks:
- name: Load image info
include_role:
name: /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information
mkdir -p $HOME/docker-registry/storage
mkdir -p $HOME/docker-registry/images
- name: Print image list
debug:
msg: "{{ (kubernetes_images + networking_images + static_images + storage_images + security_images) }}"
#. Create a configuration file that will be used by Docker's official
Registry image later on:
.. code-block:: shell
cat > $HOME/docker-registry/config.yml << EOF
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
EOF
Then, run the following to execute the Ansible Playbook:
#. Run the docker container registry:
.. code-block:: shell
K8S_VERSION=<version>
ansible-playbook list-images.yml -e "kubernetes_version=${K8S_VERSION}"
export LOCAL_REG=$HOME/docker-registry
docker run -d \
--restart=always \
--name registry \
-v "$LOCAL_REG"/storage:/var/lib/registry \
-v "$LOCAL_REG"/config.yml:/etc/docker/registry/config.yml \
-e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
-p 5000:5000 \
registry:2
You will find the Kubernetes version to assign to the `K8S_VERSION`
variable on the aforementioned `Kubernetes versions`_ file.
.. note::
*Optional*: the ``-p`` parameter configures a mapping between the
host port and the container port. If you want to listen on another
port on your host, say 9000, change from ``-p 5000:5000 \`` to
``-p 9000:5000 \``.
#. Create and run a script that will populate the registry based on the list
of images:
#. Create the list of images that will populate the registry.
.. code-block:: shell
Obtain the Kubernetes version your |prod| uses. This can be found in
the ``fresh_install_k8s_version`` value of the `Kubernetes versions`_
file. Use the branches and tags to find the value for your version.
export REG_SCRIPT=$HOME/docker-registry/images/populate_registry.sh
cat > $REG_SCRIPT <<'EOF'
#!/bin/bash
With the Kubernetes version, you can find the corresponding folder in
`system images`_ and locate the ``system-images.yml`` file. This file
contains the list of images to be loaded into your registry.
if [[ -z $1 ]]; then
echo "Please provide a file with a list of Docker images."
exit 1
fi
To make the list of images for |prod| 8.0, take the raw address of the
corresponding ``system-images.yml`` file and set a variable with it:
TAGS_FILE=$1
LOCAL_REGISTRY=localhost:5000
.. code-block:: shell
while read DOCKER_IMAGE;
do
echo ""
echo -n "--- ${DOCKER_IMAGE}: ";
export IMAGES_YAML_RAW_FILE="https://opendev.org/starlingx/ansible-playbooks/raw/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars/k8s-v1.24.4/system-images.yml"
IMAGE_ARRAY=($(echo $DOCKER_IMAGE | tr ":" " "))
REPO=${IMAGE_ARRAY[0]}
TAG=${IMAGE_ARRAY[1]}
REPO_TAGS_URL="http://${LOCAL_REGISTRY}/v2/${REPO}/tags/list"
if curl -s -X GET --insecure ${REPO_TAGS_URL} | jq | grep ${TAG} &>/dev/null; then
echo -n "Skipping..."
continue
Use the command to generate a ``list.lst`` file:
.. code-block:: shell
curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst
The expected image list will be presented in this format:
.. code-block:: text
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>...
.. important::
Due to ``kubeadm`` dynamically pulling the necessary images for
creating the k8s cluster, based on the ``Kubernetes version`` used,
this list doesn't contain all necessary images.
You'll need to manually add the extra images to the ``list.lst``
file, using `kubeadm`_. You can check the official k8s docs on how
to get them.
.. note::
*Optional*: If you have a running |prod| setup, you can run the
following to create an Ansible Playbook to get the exact images you
will need instead:
.. code-block:: shell
cat > list-images.yml << EOF
---
- hosts: localhost
gather_facts: true
tasks:
- name: Load image info
include_role:
name: /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information
- name: Print image list
debug:
msg: "{{ (kubernetes_images + networking_images + static_images + storage_images + security_images) }}"
EOF
Then, run the following to execute the Ansible Playbook:
.. code-block:: shell
K8S_VERSION=<version>
ansible-playbook list-images.yml -e "kubernetes_version=${K8S_VERSION}"
You will find the Kubernetes version to assign to the `K8S_VERSION`
variable on the aforementioned `Kubernetes versions`_ file.
#. Create and run a script that will populate the registry based on the
list of images:
.. code-block:: shell
export REG_SCRIPT=$HOME/docker-registry/images/populate_registry.sh
cat > $REG_SCRIPT <<'EOF'
#!/bin/bash
if [[ -z $1 ]]; then
echo "Please provide a file with a list of Docker images."
exit 1
fi
echo "Pulling..."
TAGS_FILE=$1
LOCAL_REGISTRY=localhost:5000
set -x
docker pull ${DOCKER_IMAGE};
REGISTRY_IMAGE=${LOCAL_REGISTRY}/${DOCKER_IMAGE}
docker tag ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
docker push ${REGISTRY_IMAGE};
docker rmi ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
set +x
while read DOCKER_IMAGE;
do
echo ""
echo -n "--- ${DOCKER_IMAGE}: ";
done < $TAGS_FILE
EOF
chmod +x $REG_SCRIPT
$REG_SCRIPT $HOME/docker-registry/images/list.lst
IMAGE_ARRAY=($(echo $DOCKER_IMAGE | tr ":" " "))
REPO=${IMAGE_ARRAY[0]}
TAG=${IMAGE_ARRAY[1]}
REPO_TAGS_URL="http://${LOCAL_REGISTRY}/v2/${REPO}/tags/list"
if curl -s -X GET --insecure ${REPO_TAGS_URL} | jq | grep ${TAG} &>/dev/null; then
echo -n "Skipping..."
continue
fi
.. note::
The ``populate_registry.sh`` script checks if each image in the list is
already present, which means you can update the list and re-run the script
to get new images whenever necessary.
echo "Pulling..."
.. note::
The Docker CLI exclusively permits insecure (HTTP) registries when on
the local host. When executing the provided script remotely, in
addition to modifying the ``LOCAL_REGISTRY`` variable to match the IP
address of the registry's location, it is necessary to insert an entry
in the ``insecure-registries:`` section within the
``etc/docker/daemon.json`` file. Following this adjustment, you must
restart the Docker service.
set -x
docker pull ${DOCKER_IMAGE};
REGISTRY_IMAGE=${LOCAL_REGISTRY}/${DOCKER_IMAGE}
docker tag ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
docker push ${REGISTRY_IMAGE};
docker rmi ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
set +x
done < $TAGS_FILE
EOF
chmod +x $REG_SCRIPT
$REG_SCRIPT $HOME/docker-registry/images/list.lst
.. note::
The ``populate_registry.sh`` script checks if each image in the
list is already present, which means you can update the list and
re-run the script to get new images whenever necessary.
.. group-tab:: With Pull-through cache
#. Create folders to store your local registry images (``storage``) and to place setup files that will be used later on (``images``):
.. code-block:: shell
mkdir -p $HOME/docker-registry/storage
mkdir -p $HOME/docker-registry/images
mkdir -p $HOME/docker-registry/config
#. Create a configuration file that will be used by Docker's official
registry image later on. The script will replace the PORT and
REGISTRY values:
.. code-block:: shell
cat > $HOME/docker-registry/config.yml << EOF
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: <PORT>
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
proxy:
remoteurl: https://<REGISTRY>
EOF
#. Create and run a script that will create several private docker
registries:
.. code-block:: shell
export REG_CACHE_SCRIPT=$HOME/docker-registry/images/create_registries.sh
cat > $REG_CACHE_SCRIPT <<'EOF'
#!/bin/bash
if [ -z "$1" ]; then
echo "No argument supplied"
exit 1
fi
export LOCAL_REG=$HOME/docker-registry
INSECURE_REGISTRY_LIST=()
LOCALHOST_IP=$1
[ -z "$2" ] && BASE_PORT=5000 || BASE_PORT=$2
REGISTRY_LIST=("quay.io" "gcr.io" "k8s.gcr.io" "registry-1.docker.io"
"docker.elastic.co" "ghcr.io" "registry.k8s.io" "icr.io")
CONFIG_BASE=$(cat "$LOCAL_REG"/config.yml)
for cur_registry in "${REGISTRY_LIST[@]}"; do
CONTAINER_NAME="registry_$cur_registry"
NEW_CONFIG_FILE_NAME="config_$cur_registry.yml"
NEW_CONFIG_FILE="$LOCAL_REG/configs/$NEW_CONFIG_FILE_NAME"
NEW_CONFIG=$(echo "$CONFIG_BASE" | sed "s/<PORT>/$BASE_PORT/" | sed "s/<REGISTRY>/$cur_registry/")
echo "$NEW_CONFIG" >"$NEW_CONFIG_FILE"
docker run -d \
--restart=always \
--name "$CONTAINER_NAME" \
-v "$LOCAL_REG"/storage:/var/lib/registry \
-v "$NEW_CONFIG_FILE":/etc/docker/registry/config.yml \
-e REGISTRY_HTTP_ADDR=0.0.0.0:"$BASE_PORT" \
-p "$BASE_PORT":"$BASE_PORT" \
registry:2.6.2
INSECURE_REGISTRY_LIST+=("$cur_registry - $LOCALHOST_IP:$BASE_PORT")
((BASE_PORT++))
done
for reg in "${INSECURE_REGISTRY_LIST[@]}"; do
echo "$reg"
done
EOF
chmod +x $REG_CACHE_SCRIPT
$REG_CACHE_SCRIPT <your ip address> <starting port (default 5000)>
.. important::
The ``create_registries.sh`` script create 8 registry
containers. You'll need 8 free ports, in sequence, starting from
the specified starting port.
Ex.: Passing port ``6000`` as the second argument, will create containers on ``6000-6007`` port range.
.. note::
On it's current version, Docker CLI throws a warning when
pulling images that're still using the v1 manifest format.
The official Docker image registry, on it's latest version, does
not allow images that still uses them.
A few images used by the StarlingX OS are outdated, so, as of
now, it'll be necessary to use a older version of the registry
image (v2.6.2).
Check the official `registry as a pull through cache`_ docker
guide for more info.
.. note::
The Docker CLI exclusively permits insecure (HTTP) registries when on
the local host. When executing the provided script remotely, in
addition to modifying the ``LOCAL_REGISTRY`` variable to match the IP
address of the registry's location, it is necessary to insert an entry
in the ``insecure-registries:`` section within the
``etc/docker/daemon.json`` file. Following this adjustment, you must
restart the Docker service.
.. rubric:: |result|
@ -188,28 +328,56 @@ Your registry is ready! On your next |prod| installation, update your
``/home/sysadmin/localhost.yml`` bootstrap overrides file with the
following lines to use it:
.. code-block:: yaml
.. tabs::
.. group-tab:: Private Registry
docker_registries:
quay.io:
url: <your IP address>:5000/quay.io
gcr.io:
url: <your IP address>:5000/gcr.io
k8s.gcr.io:
url: <your IP address>:5000/k8s.gcr.io
docker.io:
url: <your IP address>:5000/docker.io
docker.elastic.co:
url: <your IP address>:5000/docker.elastic.co
ghcr.io:
url: <your IP address>:5000/ghcr.io
registry.k8s.io:
url: <your IP address>:5000/registry.k8s.io
icr.io:
url: <your IP address>:5000/icr.io
defaults:
type: docker
secure: false
.. code-block:: yaml
docker_registries:
quay.io:
url: <your IP address>:5000/quay.io
gcr.io:
url: <your IP address>:5000/gcr.io
k8s.gcr.io:
url: <your IP address>:5000/k8s.gcr.io
docker.io:
url: <your IP address>:5000/docker.io
docker.elastic.co:
url: <your IP address>:5000/docker.elastic.co
ghcr.io:
url: <your IP address>:5000/ghcr.io
registry.k8s.io:
url: <your IP address>:5000/registry.k8s.io
icr.io:
url: <your IP address>:5000/icr.io
defaults:
type: docker
secure: false
.. group-tab:: With Pull-through cache
.. code-block:: yaml
docker_registries:
quay.io:
url: <your IP address>:<port_1>
gcr.io:
url: <your IP address>:<port_2>
k8s.gcr.io:
url: <your IP address>:<port_3>
docker.io:
url: <your IP address>:<port_4>
docker.elastic.co:
url: <your IP address>:<port_5>
ghcr.io:
url: <your IP address>:<port_6>
registry.k8s.io:
url: <your IP address>:<port_7>
icr.io:
url: <your IP address>:<port_8>
defaults:
type: docker
secure: false
.. note::
This procedure configured |prod| to use an insecure registry via the
@ -219,3 +387,5 @@ following lines to use it:
.. _Kubernetes versions: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/bootstrap/validate-config/vars/main.yml
.. _system images: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars
.. _registry as a pull through cache: https://docs.docker.com/docker-hub/mirror/
.. _kubeadm: https://kubernetes.io/pt-br/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-images-list