From 1bd070a63d6963d19d05d0d755db582b651a4cbc Mon Sep 17 00:00:00 2001 From: Jason McKenna Date: Fri, 16 Nov 2018 14:26:20 -0500 Subject: [PATCH] Add spec for publishing build artifacts on server This change adds the spec file which described how official build artifacts will be published on the world-facing server. Change-Id: I0704b3ecbd1f34138edc7c3a5efe5cd888821fe5 Signed-off-by: Jason McKenna Authored-By: Jason McKenna Co-Authored-By: Scott Little Story: 2003906 Task: 27974 --- .../mirror_2003906_published_builds.rst | 307 ++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100644 specs/2019.03/approved/mirror_2003906_published_builds.rst diff --git a/specs/2019.03/approved/mirror_2003906_published_builds.rst b/specs/2019.03/approved/mirror_2003906_published_builds.rst new file mode 100644 index 0000000..94d7908 --- /dev/null +++ b/specs/2019.03/approved/mirror_2003906_published_builds.rst @@ -0,0 +1,307 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. http://creativecommons.org/licenses/by/3.0/legalcode + +.. + +=================================================== +StarlingX: Host Build Artifacts on StarlingX Mirror +=================================================== + +https://storyboard.openstack.org/#!/story/2003906 + +An external mirror hosting build artifacts allows for quicker ramp-up +of community members, and provides for reference artifacts. + +This story will implement the building of artifacts on a server and +the population of those artifacts to the existing community-accessible +mirror. + + +Problem description +=================== + +A new community member who wishes to evaluate StarlingX currently needs +to download the source code, set up a build environment, and build all +the StarlingX components. They will then generate their own ISO. This +process can take a significant amount of time, and lead to a poor initial +impression of StarlingX. + +The initial implementation of hosted builds would have a server: +* Build all RPMs used for the build +* Build an installer image based on the produced RPMs +* Build a reference ISO based on the produced RPMs +* Publish RPMs, installer, and reference ISO to the existing mirror + +The initial implementation would provide artifacts for the r/2018.10 +and the master branch of StarlingX. Future work would build and host +artifacts for evolving branches at an interval to be determined. + + +Use Cases +========= + +Evaluators of the StarlingX project want to use a prebuilt ISO image to +install and test StarlingX. + +Developers want to compare their local changes to a package with a +reference image containing an unmodified version of the package + +Producers of an ISO want to use the prebuilt installer image rather than +one they produce themselves. + + +Proposed change +=============== + +Provision a server such that it is able to build the StarlingX +project. Allow for this build to produce artifacts based on either +the master branch, release branches (e.g. r/2018.10), or feature branches +(e.g. f/somefeature). Building and hosting of other builds (branches or +tags) should be accomodated. Provide a mechanism for the build to publish its +artifacts (RPMs, ISO, installer) to the existing mirror. Initial +implementation will use a build server at CENGN. + +Periodic building of the master branch, and publishing of the resulting +artifacts via the StarlingX mirror, will be performed. Other branches +(e.g. r/2018.10) will also be built and published, but only as required. + +The publication path for each build shall be unique and will include branch +and UTC time stamp. An alias for the latest build of the branch shall be +provided. + +Initial rate of builds is 1 build of the master branch per day. This +cadence may be adjusted in the future. Non-master branch builds shall +be performed on an on-demand basis as determined by the build team. +Non-master branch builds include release builds, and milestone builds. +Milestone builds are to be built periodically, at a cadence to be determined. + +The retention period of master branch build shall initially be 14 days. +The retention period may be revisited and adjusted at a later time. +The retention time for release branch builds will be much greater then +master branch builds, possibly indefinitely. Milestone builds shall +be retained for at least one release period past their introduction. + +Hosted artifacts proposed directory structure: + +Build artifacts hosted by the mirror shall be available in following +directory structure: + +http://mirror.starlingx.cengn.ca/mirror/starlingx/ + + = ///outputs/ + ///inputs/ + + = 'master' + 'feature/' + 'release/' + 'milestone/' + ... + 'r2018.10' (legacy symlink to release/2018.10) + + = 'centos' + + = 'centos76' + ... + + = '2018.10' + ... + + = $(date +%Y%m%dT%H%M%SZ) # time stamp of start of 'repo sync' + # a symlink to the last successful build date + + = RPMS/ + SRPMS/ + downloads/ + + = RPMS// + SRPMS// + iso/ + installer/ + wheels/ + helm-charts/ + CONTEXT + + logs/ + + = std + rt + installer + container + ... + + = 'success' | 'fail' + +Non-hosted artifacts: + +Other artifacts produced by the build may be hosted outside of the +mirror. In particular, container images shall be hosted on DockerHub. + +The DockerHub container image path shall be + +=/: + +=starlingx + +=stx- + +=--[-] + +=centos | ubuntu | clear-linux + +=pike | queens | rocky ... + +=aodh | ceilometer| cinder | glance | gnocchi | heat | horizon | + ironic | keystone | libvirt | magnum | murano | neutron | + nova-api-proxy | nova | panko ... + += | latest | stable + +=dev | r-2018.10.0 | m-2019.05b1 | ... + (This naming convention is for reference only, and has not been finalized) + + + +Alternatives +============ + +The existing mechanism of requiring all persons to perform their own +builds is an alternative. + +Providing a prebuilt version of StarlingX via a container (rather than +as build artifacts) has not been considered, as it would still depend +on an official build of StarlingX artifacts to install in said +container. + + +Data model impact +================= + +None + + +REST API impact +=============== + +None + + +Security impact +=============== + +Server administration: +The build server is to be administered by a small group of core +developers from the existing StarlingX Build team. The server +should be kept up to date with security related OS updates. The +address and access mechanism of the server are not to be published +to those outside the StarlingX Build team or StarlingX Security +team. + +Build initiation: +There shall be a mechanism for triggering builds. This mechanism +shall support self-initiated builds (periodic, or based on +observed external git commits, etc). The mechanism shall also +support manually-initiated builds. The mechanism for manually +initiated builds shall initially be through the administration +interface (discussed above). Providing other mechanisms to +initialize builds (for example, providing a web interface through +Jenkins, or a similar mechanism) is beyond the scope of this +document, however security concerns must be taken into account +if providing an additional interface. + + +Other end user impact +===================== + +End user will only see a change in StarlingX deployment if they +choose to make use of the pre-published artifacts. The automated +use of pre-published artifacts (example: having build scripts +use the pre-built installer by default) are potential future +improvements outside the scope of this document. + + +Performance Impact +================== + +No impact to running systems. Time to obtain and evaluation +copy of StarlingX will be drastically reduced. + + +Other Deployer Impact +===================== + +None + + +Developer impact +================= + +Reference RPMs, Installer image, and ISO can be used to compare code +in development. + + +Upgrade impact +=============== + +None + + +Implementation +============== + + +Assignee(s) +=========== + +Primary assignee: + Scott Little + +Other contributors: + Jason McKenna + Don Penney + Al Bailey + + +Repos Impacted +============== + +stx-tools - scripts to assist in publication of build outputs + + +Work Items +=========== + +* Setup base metal (CENGN) Server to perform builds +* Create mechanism for server to perform periodic build of master branch +* Create mechanism for server to perform build of designated release branch +* Create script or mechanism to publish builds to existing mirror + + +Dependencies +============ + +None + + +Testing +======= + +Basic installation and load sanity testing of produced iso. + + +Documentation Impact +==================== + +Documentation for the artifacts will be posted on the StarlingX wiki + + +References +========== + +https://www.cengn.ca/ +http://mirror.starlingx.cengn.ca/mirror/ + + +History +======= +