From 97593aaf5a746d8e7786ee822151b4d9c881ad8a Mon Sep 17 00:00:00 2001 From: Felipe Freire Date: Thu, 16 Nov 2023 16:03:02 -0300 Subject: [PATCH] Create Docker Registry with Pull Through Cache Create a separate section in the docker registry page, showing how to use docker registries as pull through caches. Add some important notes on the current local docker registry guide. Story: 2005051 Task: 49030 Change-Id: Id1c56498f77eb02b481da11cb2daccc8c86edfdd Signed-off-by: Felipe Freire --- .../developer_resources/local_registry.rst | 492 ++++++++++++------ 1 file changed, 331 insertions(+), 161 deletions(-) diff --git a/doc/source/developer_resources/local_registry.rst b/doc/source/developer_resources/local_registry.rst index 10226fa66..85814d481 100644 --- a/doc/source/developer_resources/local_registry.rst +++ b/doc/source/developer_resources/local_registry.rst @@ -13,174 +13,314 @@ This will also speed up the bootstrap process as images will be downloaded only once. This guide assumes that you are installing this local external docker registry on a Linux system with Docker installed and configured. +Optionally, you can also create multiple registries with pull-through cache, +skipping the need to pre-populate the registry and keeping the images and tags +up-to-date. + .. rubric:: |proc| -#. Create folders to store your local registry images (``storage``) and to - place setup files that will be used later on (``images``): - - .. code-block:: shell - - mkdir -p $HOME/docker-registry/storage - mkdir -p $HOME/docker-registry/images - -#. Create a configuration file that will be used by Docker's official - Registry image later on: - - .. code-block:: shell - - cat > $HOME/docker-registry/config.yml << EOF - version: 0.1 - log: - fields: - service: registry - storage: - cache: - blobdescriptor: inmemory - filesystem: - rootdirectory: /var/lib/registry - http: - addr: :5000 - headers: - X-Content-Type-Options: [nosniff] - health: - storagedriver: - enabled: true - interval: 10s - threshold: 3 - EOF - -#. Run the docker container registry: - - .. code-block:: shell - - export LOCAL_REG=$HOME/docker-registry - docker run -d \ - --restart=always \ - --name registry \ - -v "$LOCAL_REG"/storage:/var/lib/registry \ - -v "$LOCAL_REG"/config.yml:/etc/docker/registry/config.yml \ - -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \ - -p 5000:5000 \ - registry:2 +.. tabs:: + .. group-tab:: Private Registry .. note:: - *Optional*: the ``-p`` parameter configures a mapping between the host - port and the container port. If you want to listen on another port on - your host, say 9000, change from ``-p 5000:5000 \`` to - ``-p 9000:5000 \``. + In this section, you will pre-populate a local registry with multiple + images. Currently, the generated images list will not contain all + images, making it necessary to manually pull some images used by the + kubeadm tool. Alternatively, you can follow the "pull-through cache" + section, allowing docker to get the image from remote, if it doesn't + exists locally. -#. Create the list of images that will populate the registry. - - Obtain the Kubernetes version your |prod| uses. This can be found in the - ``fresh_install_k8s_version`` value of the `Kubernetes versions`_ file. - Use the branches and tags to find the value for your version. - - With the Kubernetes version, you can find the corresponding folder in - `system images`_ and locate the ``system-images.yml`` file. This file - contains the list of images to be loaded into your registry. - - To make the list of images for |prod| 8.0, take the raw address of the - corresponding ``system-images.yml`` file and set a variable with it: - - .. code-block:: shell - - export IMAGES_YAML_RAW_FILE="https://opendev.org/starlingx/ansible-playbooks/raw/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars/k8s-v1.24.4/system-images.yml" - - Use the command to generate a ``list.lst`` file: - - .. code-block:: shell - - curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst - - .. note:: *Optional*: If you have a running |prod| setup, you can run - the following to create an Ansible Playbook to get the exact images - you will need instead: + #. Create folders to store your local registry images (``storage``) and to + place setup files that will be used later on (``images``): .. code-block:: shell - cat > list-images.yml << EOF - --- - - hosts: localhost - gather_facts: true - tasks: - - name: Load image info - include_role: - name: /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information + mkdir -p $HOME/docker-registry/storage + mkdir -p $HOME/docker-registry/images - - name: Print image list - debug: - msg: "{{ (kubernetes_images + networking_images + static_images + storage_images + security_images) }}" + #. Create a configuration file that will be used by Docker's official + Registry image later on: + + .. code-block:: shell + + cat > $HOME/docker-registry/config.yml << EOF + version: 0.1 + log: + fields: + service: registry + storage: + cache: + blobdescriptor: inmemory + filesystem: + rootdirectory: /var/lib/registry + http: + addr: :5000 + headers: + X-Content-Type-Options: [nosniff] + health: + storagedriver: + enabled: true + interval: 10s + threshold: 3 EOF - Then, run the following to execute the Ansible Playbook: + #. Run the docker container registry: .. code-block:: shell - K8S_VERSION= - ansible-playbook list-images.yml -e "kubernetes_version=${K8S_VERSION}" + export LOCAL_REG=$HOME/docker-registry + docker run -d \ + --restart=always \ + --name registry \ + -v "$LOCAL_REG"/storage:/var/lib/registry \ + -v "$LOCAL_REG"/config.yml:/etc/docker/registry/config.yml \ + -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \ + -p 5000:5000 \ + registry:2 - You will find the Kubernetes version to assign to the `K8S_VERSION` - variable on the aforementioned `Kubernetes versions`_ file. + .. note:: + *Optional*: the ``-p`` parameter configures a mapping between the + host port and the container port. If you want to listen on another + port on your host, say 9000, change from ``-p 5000:5000 \`` to + ``-p 9000:5000 \``. -#. Create and run a script that will populate the registry based on the list - of images: + #. Create the list of images that will populate the registry. - .. code-block:: shell + Obtain the Kubernetes version your |prod| uses. This can be found in + the ``fresh_install_k8s_version`` value of the `Kubernetes versions`_ + file. Use the branches and tags to find the value for your version. - export REG_SCRIPT=$HOME/docker-registry/images/populate_registry.sh - cat > $REG_SCRIPT <<'EOF' - #!/bin/bash + With the Kubernetes version, you can find the corresponding folder in + `system images`_ and locate the ``system-images.yml`` file. This file + contains the list of images to be loaded into your registry. - if [[ -z $1 ]]; then - echo "Please provide a file with a list of Docker images." - exit 1 - fi + To make the list of images for |prod| 8.0, take the raw address of the + corresponding ``system-images.yml`` file and set a variable with it: - TAGS_FILE=$1 - LOCAL_REGISTRY=localhost:5000 + .. code-block:: shell - while read DOCKER_IMAGE; - do - echo "" - echo -n "--- ${DOCKER_IMAGE}: "; + export IMAGES_YAML_RAW_FILE="https://opendev.org/starlingx/ansible-playbooks/raw/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars/k8s-v1.24.4/system-images.yml" - IMAGE_ARRAY=($(echo $DOCKER_IMAGE | tr ":" " ")) - REPO=${IMAGE_ARRAY[0]} - TAG=${IMAGE_ARRAY[1]} - REPO_TAGS_URL="http://${LOCAL_REGISTRY}/v2/${REPO}/tags/list" - if curl -s -X GET --insecure ${REPO_TAGS_URL} | jq | grep ${TAG} &>/dev/null; then - echo -n "Skipping..." - continue + Use the command to generate a ``list.lst`` file: + + .. code-block:: shell + + curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst + + The expected image list will be presented in this format: + + .. code-block:: text + + : + : + :... + + .. important:: + Due to ``kubeadm`` dynamically pulling the necessary images for + creating the k8s cluster, based on the ``Kubernetes version`` used, + this list doesn't contain all necessary images. + You'll need to manually add the extra images to the ``list.lst`` + file, using `kubeadm`_. You can check the official k8s docs on how + to get them. + + .. note:: + *Optional*: If you have a running |prod| setup, you can run the + following to create an Ansible Playbook to get the exact images you + will need instead: + + .. code-block:: shell + + cat > list-images.yml << EOF + --- + - hosts: localhost + gather_facts: true + tasks: + - name: Load image info + include_role: + name: /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information + + - name: Print image list + debug: + msg: "{{ (kubernetes_images + networking_images + static_images + storage_images + security_images) }}" + EOF + + Then, run the following to execute the Ansible Playbook: + + .. code-block:: shell + + K8S_VERSION= + ansible-playbook list-images.yml -e "kubernetes_version=${K8S_VERSION}" + + You will find the Kubernetes version to assign to the `K8S_VERSION` + variable on the aforementioned `Kubernetes versions`_ file. + + #. Create and run a script that will populate the registry based on the + list of images: + + .. code-block:: shell + + export REG_SCRIPT=$HOME/docker-registry/images/populate_registry.sh + cat > $REG_SCRIPT <<'EOF' + #!/bin/bash + + if [[ -z $1 ]]; then + echo "Please provide a file with a list of Docker images." + exit 1 fi - echo "Pulling..." + TAGS_FILE=$1 + LOCAL_REGISTRY=localhost:5000 - set -x - docker pull ${DOCKER_IMAGE}; - REGISTRY_IMAGE=${LOCAL_REGISTRY}/${DOCKER_IMAGE} - docker tag ${DOCKER_IMAGE} ${REGISTRY_IMAGE}; - docker push ${REGISTRY_IMAGE}; - docker rmi ${DOCKER_IMAGE} ${REGISTRY_IMAGE}; - set +x + while read DOCKER_IMAGE; + do + echo "" + echo -n "--- ${DOCKER_IMAGE}: "; - done < $TAGS_FILE - EOF - chmod +x $REG_SCRIPT - $REG_SCRIPT $HOME/docker-registry/images/list.lst + IMAGE_ARRAY=($(echo $DOCKER_IMAGE | tr ":" " ")) + REPO=${IMAGE_ARRAY[0]} + TAG=${IMAGE_ARRAY[1]} + REPO_TAGS_URL="http://${LOCAL_REGISTRY}/v2/${REPO}/tags/list" + if curl -s -X GET --insecure ${REPO_TAGS_URL} | jq | grep ${TAG} &>/dev/null; then + echo -n "Skipping..." + continue + fi - .. note:: - The ``populate_registry.sh`` script checks if each image in the list is - already present, which means you can update the list and re-run the script - to get new images whenever necessary. + echo "Pulling..." - .. note:: - The Docker CLI exclusively permits insecure (HTTP) registries when on - the local host. When executing the provided script remotely, in - addition to modifying the ``LOCAL_REGISTRY`` variable to match the IP - address of the registry's location, it is necessary to insert an entry - in the ``insecure-registries:`` section within the - ``etc/docker/daemon.json`` file. Following this adjustment, you must - restart the Docker service. + set -x + docker pull ${DOCKER_IMAGE}; + REGISTRY_IMAGE=${LOCAL_REGISTRY}/${DOCKER_IMAGE} + docker tag ${DOCKER_IMAGE} ${REGISTRY_IMAGE}; + docker push ${REGISTRY_IMAGE}; + docker rmi ${DOCKER_IMAGE} ${REGISTRY_IMAGE}; + set +x + + done < $TAGS_FILE + EOF + chmod +x $REG_SCRIPT + $REG_SCRIPT $HOME/docker-registry/images/list.lst + + .. note:: + The ``populate_registry.sh`` script checks if each image in the + list is already present, which means you can update the list and + re-run the script to get new images whenever necessary. + + .. group-tab:: With Pull-through cache + + #. Create folders to store your local registry images (``storage``) and to place setup files that will be used later on (``images``): + + .. code-block:: shell + + mkdir -p $HOME/docker-registry/storage + mkdir -p $HOME/docker-registry/images + mkdir -p $HOME/docker-registry/config + + #. Create a configuration file that will be used by Docker's official + registry image later on. The script will replace the PORT and + REGISTRY values: + + .. code-block:: shell + + cat > $HOME/docker-registry/config.yml << EOF + version: 0.1 + log: + fields: + service: registry + storage: + cache: + blobdescriptor: inmemory + filesystem: + rootdirectory: /var/lib/registry + http: + addr: + headers: + X-Content-Type-Options: [nosniff] + health: + storagedriver: + enabled: true + interval: 10s + threshold: 3 + proxy: + remoteurl: https:// + EOF + + #. Create and run a script that will create several private docker + registries: + + .. code-block:: shell + + export REG_CACHE_SCRIPT=$HOME/docker-registry/images/create_registries.sh + cat > $REG_CACHE_SCRIPT <<'EOF' + #!/bin/bash + + if [ -z "$1" ]; then + echo "No argument supplied" + exit 1 + fi + + export LOCAL_REG=$HOME/docker-registry + INSECURE_REGISTRY_LIST=() + LOCALHOST_IP=$1 + [ -z "$2" ] && BASE_PORT=5000 || BASE_PORT=$2 + REGISTRY_LIST=("quay.io" "gcr.io" "k8s.gcr.io" "registry-1.docker.io" + "docker.elastic.co" "ghcr.io" "registry.k8s.io" "icr.io") + CONFIG_BASE=$(cat "$LOCAL_REG"/config.yml) + + for cur_registry in "${REGISTRY_LIST[@]}"; do + CONTAINER_NAME="registry_$cur_registry" + NEW_CONFIG_FILE_NAME="config_$cur_registry.yml" + NEW_CONFIG_FILE="$LOCAL_REG/configs/$NEW_CONFIG_FILE_NAME" + NEW_CONFIG=$(echo "$CONFIG_BASE" | sed "s//$BASE_PORT/" | sed "s//$cur_registry/") + echo "$NEW_CONFIG" >"$NEW_CONFIG_FILE" + + docker run -d \ + --restart=always \ + --name "$CONTAINER_NAME" \ + -v "$LOCAL_REG"/storage:/var/lib/registry \ + -v "$NEW_CONFIG_FILE":/etc/docker/registry/config.yml \ + -e REGISTRY_HTTP_ADDR=0.0.0.0:"$BASE_PORT" \ + -p "$BASE_PORT":"$BASE_PORT" \ + registry:2.6.2 + + INSECURE_REGISTRY_LIST+=("$cur_registry - $LOCALHOST_IP:$BASE_PORT") + ((BASE_PORT++)) + + done + + for reg in "${INSECURE_REGISTRY_LIST[@]}"; do + echo "$reg" + done + EOF + + chmod +x $REG_CACHE_SCRIPT + $REG_CACHE_SCRIPT + + .. important:: + The ``create_registries.sh`` script create 8 registry + containers. You'll need 8 free ports, in sequence, starting from + the specified starting port. + Ex.: Passing port ``6000`` as the second argument, will create containers on ``6000-6007`` port range. + + .. note:: + On it's current version, Docker CLI throws a warning when + pulling images that're still using the v1 manifest format. + The official Docker image registry, on it's latest version, does + not allow images that still uses them. + A few images used by the StarlingX OS are outdated, so, as of + now, it'll be necessary to use a older version of the registry + image (v2.6.2). + Check the official `registry as a pull through cache`_ docker + guide for more info. + + +.. note:: + The Docker CLI exclusively permits insecure (HTTP) registries when on + the local host. When executing the provided script remotely, in + addition to modifying the ``LOCAL_REGISTRY`` variable to match the IP + address of the registry's location, it is necessary to insert an entry + in the ``insecure-registries:`` section within the + ``etc/docker/daemon.json`` file. Following this adjustment, you must + restart the Docker service. .. rubric:: |result| @@ -188,28 +328,56 @@ Your registry is ready! On your next |prod| installation, update your ``/home/sysadmin/localhost.yml`` bootstrap overrides file with the following lines to use it: -.. code-block:: yaml +.. tabs:: + .. group-tab:: Private Registry - docker_registries: - quay.io: - url: :5000/quay.io - gcr.io: - url: :5000/gcr.io - k8s.gcr.io: - url: :5000/k8s.gcr.io - docker.io: - url: :5000/docker.io - docker.elastic.co: - url: :5000/docker.elastic.co - ghcr.io: - url: :5000/ghcr.io - registry.k8s.io: - url: :5000/registry.k8s.io - icr.io: - url: :5000/icr.io - defaults: - type: docker - secure: false + .. code-block:: yaml + + docker_registries: + quay.io: + url: :5000/quay.io + gcr.io: + url: :5000/gcr.io + k8s.gcr.io: + url: :5000/k8s.gcr.io + docker.io: + url: :5000/docker.io + docker.elastic.co: + url: :5000/docker.elastic.co + ghcr.io: + url: :5000/ghcr.io + registry.k8s.io: + url: :5000/registry.k8s.io + icr.io: + url: :5000/icr.io + defaults: + type: docker + secure: false + + .. group-tab:: With Pull-through cache + + .. code-block:: yaml + + docker_registries: + quay.io: + url: : + gcr.io: + url: : + k8s.gcr.io: + url: : + docker.io: + url: : + docker.elastic.co: + url: : + ghcr.io: + url: : + registry.k8s.io: + url: : + icr.io: + url: : + defaults: + type: docker + secure: false .. note:: This procedure configured |prod| to use an insecure registry via the @@ -219,3 +387,5 @@ following lines to use it: .. _Kubernetes versions: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/bootstrap/validate-config/vars/main.yml .. _system images: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars +.. _registry as a pull through cache: https://docs.docker.com/docker-hub/mirror/ +.. _kubeadm: https://kubernetes.io/pt-br/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-images-list