docs/doc/source/developer_guide/index.rst

.. _developer-guide:

===============
Developer Guide
===============

This section contains the steps for building a StarlingX ISO from
the "r/2018.10" branch.

------------
Requirements
------------

The recommended minimum requirements include:

*********************
Hardware Requirements
*********************

A workstation computer with:

-  Processor: x86_64 is the only supported architecture
-  Memory: At least 32GB RAM
-  Hard Disk: 500GB HDD
-  Network: Network adapter with active Internet connection

*********************
Software Requirements
*********************

A workstation computer with:

-  Operating System: Ubuntu 16.04 LTS 64-bit
-  Docker
-  Android Repo Tool
-  Proxy Settings Configured (If Required)

   -  See
      http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000136.html
      for more details

-  Public SSH Key

-----------------------------
Development Environment Setup
-----------------------------

This section describes how to set up a StarlingX development system on a
workstation computer. After completing these steps, you can
build a StarlingX ISO image on the following Linux distribution:

-  Ubuntu 16.04 LTS 64-bit

****************************
Update Your Operating System
****************************

Before proceeding with the build, ensure your Ubuntu distribution is up to date.
You first need to update the local database list of available packages:

.. code:: sh

   $ sudo apt-get update

******************************************
Installation Requirements and Dependencies
******************************************

^^^
Git
^^^

1. Install the required packages on the Ubuntu host system:

   .. code:: sh

      $ sudo apt-get install make git curl

2. Make sure to setup your identity with the following two commands.
   Be sure to provide your actual name and email address:

   .. code:: sh

      $ git config --global user.name "Name LastName"
      $ git config --global user.email "Email Address"

^^^^^^^^^
Docker CE
^^^^^^^^^

3. Install the required Docker CE packages in the Ubuntu host system. See
   `Get Docker CE for
   Ubuntu <https://docs.docker.com/install/linux/docker-ce/ubuntu/#os-requirements>`__
   for more information.

^^^^^^^^^^^^^^^^^
Android Repo Tool
^^^^^^^^^^^^^^^^^

4. Install the required Android Repo Tool in the Ubuntu host system. Follow
   the steps in the `Installing
   Repo <https://source.android.com/setup/build/downloading#installing-repo>`__
   section.

**********************
Install Public SSH Key
**********************

#. Follow these instructions on GitHub to `Generate a Public SSH
   Key <https://help.github.com/articles/connecting-to-github-with-ssh>`__.
   Then upload your public key to your GitHub and Gerrit account
   profiles:

   -  `Upload to
      Github <https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account>`__
   -  `Upload to
      Gerrit <https://review.openstack.org/#/settings/ssh-keys>`__

*************************
Install stx-tools Project
*************************

#. While in your $HOME directory, create a local copy of the stx-tools project
   based on the "r/2018.10" branch:

   .. code:: sh

      $ cd $HOME
      $ git clone -b r/2018.10 https://git.starlingx.io/stx-tools

****************************
Create a Workspace Directory
****************************

#. Create a *starlingx* workspace directory on your system.
   Best practices dictate creating the workspace directory
   in your $HOME directory:

   .. code:: sh

      $ mkdir -p $HOME/starlingx/

----------------------------------
Build the CentOS Mirror Repository
----------------------------------

This section describes how to build the CentOS Mirror Repository.

*********************************
Setup Repository Docker Container
*********************************

| Run the following commands under a terminal identified as "**One**".

#. Navigate to the *$HOME/stx-tools/centos-mirror-tool* project
   directory:

   .. code:: sh

      $ cd $HOME/stx-tools/centos-mirror-tools/

#. If necessary, set the http/https proxy in your
   Dockerfile before building the docker image.

   .. code:: sh

      ENV http_proxy " http://your.actual_http_proxy.com:your_port "
      ENV https_proxy " https://your.actual_https_proxy.com:your_port "
      ENV ftp_proxy " http://your.actual_ftp_proxy.com:your_port "
      RUN echo " proxy=http://your-proxy.com:port " >> /etc/yum.conf

#. Build your *<user>:<tag>* base container image. For example, use
   *user:centos-mirror-repository*:

   .. code:: sh

      $ docker build --tag $USER:centos-mirror-repository --file Dockerfile .

#. Launch a *<user>* docker container using the previously created Docker
   base container image *<user>:<tag>* (e.g.
   *-centos-mirror-repository*). As /localdisk is defined as the workdir
   of the container, the same folder name should be used to define the
   volume. The container will start to run and populate the output folders
   *logs* and *output* in this directory. The container is run from the
   same directory where the other scripts are stored:

   .. code:: sh

      $ docker run -itd --name $USER-centos-mirror-repository --volume $(pwd):/localdisk $USER:centos-mirror-repository

   **NOTE**: The above command creates the container in the background.
   With the container running in the background, you must attach to it
   manually. An advantage of attaching to the container manually is that
   you can enter and exit from the container as many times as you want.

*****************
Download Packages
*****************

#. Attach to the docker repository previously created:

   ::

      $ docker exec -it <CONTAINER ID> /bin/bash

#. Inside Repository Docker container, enter the following command to
   download the required packages to populate the CentOS Mirror
   Repository:

   ::

      # bash download_mirror.sh

#. Monitor the download of packages until it is complete. When the download
   is complete, the following message appears:

   ::

      totally 17 files are downloaded!
      step #3: done successfully
      IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images
      for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso"
          - out/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img
          - out/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img
          - out/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz

***************
Verify Packages
***************

#. Verify no missing or failed packages exist:

   ::

      # cat logs/*_missing_*.log
      # cat logs/*_failmove_*.log

#. In case missing or failed packages do exist, which is usually caused by
   network instability (or timeout), you need to download the packages
   manually.
   Doing so assures you get all RPMs listed in
   *rpms_3rdparties.lst*/*rpms_centos.lst*/*rpms_centos3rdparties.lst*.

******************
Packages Structure
******************

The following is a general overview of the packages structure resulting
from downloading the packages:

::

   /home/<user>/stx-tools/centos-mirror-tools/output
   └── stx-r1
       └── CentOS
           └── pike
               ├── Binary
               │   ├── EFI
               │   ├── images
               │   ├── isolinux
               │   ├── LiveOS
               │   ├── noarch
               │   └── x86_64
               ├── downloads
               │   ├── integrity
               │   └── puppet
               └── Source

*******************************
Create CentOS Mirror Repository
*******************************

Outside your Repository Docker container, in another terminal identified
as "**Two**", run the following commands:

#. From terminal identified as "**Two**", create a *mirror/CentOS*
   directory under your *starlingx* workspace directory:

   .. code:: sh

      $ mkdir -p $HOME/starlingx/mirror/CentOS/

#. Copy the built CentOS Mirror Repository built under
   *$HOME/stx-tools/centos-mirror-tool* to the *$HOME/starlingx/mirror/*
   workspace directory:

   .. code:: sh

      $ cp -r $HOME/stx-tools/centos-mirror-tools/output/stx-r1/ $HOME/starlingx/mirror/CentOS/


-------------------------
Create StarlingX Packages
-------------------------

*******************************
Setup Building Docker Container
*******************************

#. From the terminal identified as "**Two**", create the workspace folder:

   .. code:: sh

      $ mkdir -p $HOME/starlingx/workspace

#. Navigate to the *$HOME/stx-tools* project directory:

   .. code:: sh

      $ cd $HOME/stx-tools

#. Copy your Git options to *toCopy* folder:

   .. code:: sh

      $ cp ~/.gitconfig toCOPY

#. Create a *localrc* file:

   .. code:: sh

      $ cat <<- EOF > localrc
      # tbuilder localrc
      MYUNAME=$USER
      PROJECT=starlingx
      HOST_PREFIX=$HOME/starlingx/workspace
      HOST_MIRROR_DIR=$HOME/starlingx/mirror
      EOF

#. If necessary, you might have to set the http/https proxy in your
   *Dockerfile.centos73* file before building the docker image:

   .. code:: sh

      ENV http_proxy  "http://your.actual_http_proxy.com:your_port"
      ENV https_proxy "https://your.actual_https_proxy.com:your_port"
      ENV ftp_proxy "http://your.actual_ftp_proxy.com:your_port"
      RUN echo "proxy=$http_proxy" >> /etc/yum.conf && \
      echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
      export ftp_proxy=$ftp_proxy" >> /root/.bashrc

#. Set up the base container:

   If you are using a Fedora distribution, you will
   see the following error:

   .. code:: sh

     .makeenv:88: \**\* missing separator. Stop.

   To continue, do the following:

   -  delete the functions define in the .makeenv ( module () { ... } )
   -  delete the line 19 in the Makefile and ( NULL := $(shell bash -c
      "source buildrc ... ).

   .. code:: sh

      $ make base-build

#. Set up the build container:

   .. code:: sh

      $ make build

#. Verify environment variables:

   .. code:: sh

      $ bash tb.sh env

#. Run the build container:

   .. code:: sh

      $ bash tb.sh run

#. Execute the built container:

   .. code:: sh

      $ bash tb.sh exec

*********************************
Download Source Code Repositories
*********************************

#. From the terminal identified as "**Two**", which is now inside the
   Building Docker container, start the internal environment:

   .. code:: sh

      $ eval $(ssh-agent)
      $ ssh-add

#. Use the repo tool to create a local clone of the stx-manifest
   Git repository based on the "r/2018.10" branch:

   .. code:: sh

      $ cd $MY_REPO_ROOT_DIR
      $ repo init -u https://git.starlingx.io/stx-manifest -m default.xml -b r/2018.10

   **NOTE:** To use the "repo" command to clone the stx-manifest repository and
   check out the "master" branch, omit the "-b r/2018.10" option.
   Following is an example:

   .. code:: sh

      $ repo init -u https://git.starlingx.io/stx-manifest -m default.xml

#. Synchronize the repository:

   .. code:: sh

      $ repo sync -j`nproc`

#. Create a tarballs repository:

   .. code:: sh

      $ ln -s /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/ $MY_REPO/stx/

   Alternatively, you can run the "populate_downloads.sh" script to copy
   the tarballs instead of using a symlink:

   .. code:: sh

      $ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/

   Outside the container

#. From another terminal identified as "**Three**", create mirror binaries:

   .. code:: sh

      $ mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer
      $ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img
      $ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz
      $ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img

**************
Build Packages
**************

#. Go back to the terminal identified as "**Two**", which is the Building Docker container.

#. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing /
   corrupted rpm and tarball packages generated during
   `Build the CentOS Mirror Repository`_, which will cause the next step
   to fail. If that step does fail, manually download those missing /
   corrupted packages.

#. Update the symbolic links:

   .. code:: sh

      $ generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/

#. Build the packages:

   .. code:: sh

      $ build-pkgs

#. **Optional!** Generate-Cgcs-Tis-Repo:

   While this step is optional, it improves performance on subsequent
   builds. The cgcs-tis-repo has the dependency information that
   sequences the build order. To generate or update the information, you
   need to execute the following command after building modified or new
   packages.

   .. code:: sh

      $ generate-cgcs-tis-repo

-------------------
Build StarlingX ISO
-------------------

#. Build the image:

   .. code:: sh

      $ build-iso

---------------
Build Installer
---------------

To get your StarlingX ISO ready to use, you must create the initialization
files used to boot the ISO, additional controllers, and compute nodes.

**NOTE:** You only need this procedure during your first build and
every time you upgrade the kernel.

After running "build-iso", run:

.. code:: sh

   $ build-pkgs --installer

This builds *rpm* and *anaconda* packages. Then run:

.. code:: sh

   $ update-pxe-network-installer

The *update-pxe-network-installer* covers the steps detailed in
*$MY_REPO/stx/stx-metal/installer/initrd/README*. This script
creates three files on
*/localdisk/loadbuild///pxe-network-installer/output*.

::

   new-initrd.img
   new-squashfs.img
   new-vmlinuz

Rename the files as follows:

::

   initrd.img-stx-0.2
   squashfs.img-stx-0.2
   vmlinuz-stx-0.2

Two ways exist for using these files:

#. Store the files in the */import/mirror/CentOS/stx-installer/* folder
   for future use.
#. Store the files in an arbitrary location and modify the
   *$MY_REPO/stx/stx-metal/installer/pxe-network-installer/centos/build_srpm.data*
   file to point to these files.

Recreate the *pxe-network-installer* package and rebuild the image:

.. code:: sh

   $ build-pkgs --clean pxe-network-installer
   $ build-pkgs pxe-network-installer
   $ build-iso

Your ISO image should be able to boot.

****************
Additional Notes
****************

-  In order to get the first boot working, this complete procedure needs
   to be done. However, once the init files are created, these can be
   stored in a shared location where different developers can make use
   of them. Updating these files is not a frequent task and should be
   done whenever the kernel is upgraded.
-  StarlingX is in active development.  Consequently, it is possible that in the
   future the **0.2** version will change to a more generic solution.

---------------
Build Avoidance
---------------

*******
Purpose
*******

Greatly reduce build times after using "repo" to syncronized a local
repository with an upstream source (i.e. "repo sync").
Build avoidance works well for designers working
within a regional office. Starting from a new workspace, "build-pkgs"
typically requires three or more hours to complete. Build avoidance
reduces this step to approximately 20 minutes.

***********
Limitations
***********

-  Little or no benefit for designers who refresh a pre-existing
   workspace at least daily (e.g. download_mirror.sh, repo sync,
   generate-cgcs-centos-repo.sh, build-pkgs, build-iso). In these cases,
   an incremental build (i.e. reuse of same workspace without a "build-pkgs
   --clean") is often just as efficient.
-  Not likely to be useful to solo designers, or teleworkers that wish
   to compile on using their home computers. Build avoidance downloads build
   artifacts from a reference build, and WAN speeds are generally too
   slow.

*****************
Method (in brief)
*****************

#. Reference Builds

   -  A server in the regional office performs regular (e.g. daily)
      automated builds using existing methods. These builds are called
      "reference builds".
   -  The builds are timestamped and preserved for some time (i.e. a
      number of weeks).
   -  A build CONTEXT, which is a file produced by "build-pkgs"
      at location *$MY_WORKSPACE/CONTEXT*, is captured. It is a bash script that can
      cd to each and every Git and checkout the SHA that contributed to
      the build.
   -  For each package built, a file captures the md5sums of all the
      source code inputs required to build that package. These files are
      also produced by "build-pkgs" at location
      *$MY_WORKSPACE//rpmbuild/SOURCES//srpm_reference.md5*.
   -  All these build products are accessible locally (e.g. a regional
      office) using "rsync".

      **NOTE:** Other protocols can be added later.

#. Designers

   - Request a build avoidance build. Recommended after you have
     done synchronized the repository (i.e. "repo sync").

     ::

        repo sync
        generate-cgcs-centos-repo.sh
        populate_downloads.sh
        build-pkgs --build-avoidance

   - Use combinations of additional arguments, environment variables, and a
     configuration file unique to the regional office to specify an URL
     to the reference builds.

      - Using a configuration file to specify the location of your reference build:

        ::

           mkdir -p $MY_REPO/local-build-data

           cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source
           # Optional, these are already the default values.
           BUILD_AVOIDANCE_DATE_FORMAT="%Y%m%d"
           BUILD_AVOIDANCE_TIME_FORMAT="%H%M%S"
           BUILD_AVOIDANCE_DATE_TIME_DELIM="T"
           BUILD_AVOIDANCE_DATE_TIME_POSTFIX="Z"
           BUILD_AVOIDANCE_DATE_UTC=1
           BUILD_AVOIDANCE_FILE_TRANSFER="rsync"

           # Required, unique values for each regional office
           BUILD_AVOIDANCE_USR="jenkins"
           BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
           BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
           EOF

      - Using command-line arguments to specify the location of your reference
        build:

        ::

           build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins

   -  Prior to your build attempt, you need to accept the host key.
      Doing so prevents "rsync" failures on a "yes/no" prompt.
      You only have to do this once.

      ::

         grep -q $BUILD_AVOIDANCE_HOST $HOME/.ssh/known_hosts
         if [ $? != 0 ]; then
         ssh-keyscan $BUILD_AVOIDANCE_HOST >> $HOME/.ssh/known_hosts
         fi


   -  "build-pkgs" does the following:

      -  From newest to oldest, scans the CONTEXTs of the various
         reference builds. Selects the first (i.e. most recent) context that
         satisfies the following requirement: every Git the SHA
         specifies in the CONTEXT is present.
      -  The selected context might be slightly out of date, but not by
         more than a day. This assumes daily reference builds are run.
      -  If the context has not been previously downloaded, then
         download it now. This means you need to download select portions of the
         reference build workspace into the designer's workspace. This
         includes all the SRPMS, RPMS, MD5SUMS, and miscellaneous supporting
         files. Downloading these files usually takes about 10 minutes
         over an office LAN.
      -  The designer could have additional commits or uncommitted changes
         not present in the reference builds. Affected packages are
         identified by the differing md5sum's.  In these cases, the packages
         are re-built.  Re-builds usually take five or more minutes,
         depending on the packages that have changed.

   -  What if no valid reference build is found? Then build-pkgs will fall
      back to a regular build.

****************
Reference Builds
****************

-  The regional office implements an automated build that pulls the
   latest StarlingX software and builds it on a regular basis (e.g.
   daily builds).  Jenkins, cron, or similar tools can trigger these builds.
-  Each build is saved to a unique directory, and preserved for a time
   that is reflective of how long a designer might be expected to work
   on a private branch without syncronizing with the master branch.
   This takes about two weeks.

- The *MY_WORKSPACE* directory for the build shall have a common root
  directory, and a leaf directory that is a sortable time stamp. The
  suggested format is *YYYYMMDDThhmmss*.

  .. code:: sh

     $ sudo apt-get update
     BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
     BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
     MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}

-  Designers can access all build products over the internal network of
   the regional office. The current prototype employs "rsync". Other
   protocols that can efficiently share, copy, or transfer large directories
   of content can be added as needed.

^^^^^^^^^^^^^^
Advanced Usage
^^^^^^^^^^^^^^

Can the reference build itself use build avoidance? Yes it can.
Can it reference itself? Yes it can.
In both these cases, caution is advised. To protect against any possible
'divergence from reality', you should limit how many steps you remove
a build avoidance build from a full build.

Suppose we want to implement a self-referencing daily build in an
environment where a full build already occurs every Saturday.
To protect ourselves from a
build failure on Saturday we also want a limit of seven days since
the last full build. Your build script might look like this ...

::

   ...
   BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
   BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
   FULL_BUILD_DAY="Saturday"
   MAX_AGE_DAYS=7

   LAST_FULL_BUILD_LINK="$BUILD_AVOIDANCE_DIR/latest_full_build"
   LAST_FULL_BUILD_DAY=""
   NOW_DAY=$(date -u "+%A")
   BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
   MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}

   # update software
   repo init -u ${BUILD_REPO_URL} -b ${BUILD_BRANCH}
   repo sync --force-sync
   $MY_REPO_ROOT_DIR/stx-tools/toCOPY/generate-cgcs-centos-repo.sh
   $MY_REPO_ROOT_DIR/stx-tools/toCOPY/populate_downloads.sh

   # User can optionally define BUILD_METHOD equal to one of 'FULL', 'AVOIDANCE', or 'AUTO'
   # Sanitize BUILD_METHOD
   if [ "$BUILD_METHOD" != "FULL" ] && [ "$BUILD_METHOD" != "AVOIDANCE" ]; then
       BUILD_METHOD="AUTO"
   fi

   # First build test
   if [ "$BUILD_METHOD" != "FULL" ] && [ ! -L $LAST_FULL_BUILD_LINK ]; then
       echo "latest_full_build symlink missing, forcing full build"
       BUILD_METHOD="FULL"
   fi

   # Build day test
   if [ "$BUILD_METHOD" == "AUTO" ] && [ "$NOW_DAY" == "$FULL_BUILD_DAY" ]; then
       echo "Today is $FULL_BUILD_DAY, forcing full build"
       BUILD_METHOD="FULL"
   fi

   # Build age test
   if [ "$BUILD_METHOD" != "FULL" ]; then
       LAST_FULL_BUILD_DATE=$(basename $(readlink $LAST_FULL_BUILD_LINK) | cut -d '_' -f 1)
       LAST_FULL_BUILD_DAY=$(date -d $LAST_FULL_BUILD_DATE "+%A")
       AGE_SECS=$(( $(date "+%s") - $(date -d $LAST_FULL_BUILD_DATE "+%s") ))
       AGE_DAYS=$(( $AGE_SECS/60/60/24 ))
       if [ $AGE_DAYS -ge $MAX_AGE_DAYS ]; then
           echo "Haven't had a full build in $AGE_DAYS days, forcing full build"
           BUILD_METHOD="FULL"
       fi
       BUILD_METHOD="AVOIDANCE"
   fi

   #Build it
   if [ "$BUILD_METHOD" == "FULL" ]; then
       build-pkgs --no-build-avoidance
   else
       build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER
   fi
   if [ $? -ne 0 ]; then
       echo "Build failed in build-pkgs"
       exit 1
   fi

   build-iso
   if [ $? -ne 0 ]; then
       echo "Build failed in build-iso"
       exit 1
   fi

   if [ "$BUILD_METHOD" == "FULL" ]; then
       # A successful full build.  Set last full build symlink.
       if [ -L $LAST_FULL_BUILD_LINK ]; then
           rm -rf $LAST_FULL_BUILD_LINK
       fi
       ln -sf $MY_WORKSPACE $LAST_FULL_BUILD_LINK
   fi
   ...

A final note....

To use the full build day as your avoidance build reference point,
modify the "build-pkgs" commands above to use "--build-avoidance-day ",
as shown in the following two examples:

::

   build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $FULL_BUILD_DAY

   # Here is another example with a bit more shuffling of the above script.

   build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $LAST_FULL_BUILD_DAY

The advantage is that our build is never more than one step removed
from a full build. This assumes the full build was successful.

The disadvantage is that by the end of the week, the reference build is getting
rather old. During active weeks, build times could approach build times for
full builds.