starlingx
/
manifest
Code Issues Proposed changes

Restore repo sync performance by limiting kernel history. 

The yocto kernels, added to the manifest last week, ar large
(~2.7GB each) and take a long time to download (~30 min each)
via repo sync.  This has increased overall repo sync times
from 10 min to over an hour for a new project.

Proposed solution is to only download a very limited history
for the kernel.

- depth=100 means only download 100 commits of history from
any reference (excluding sha).
- sync-c="true" means only download the specified upstream branch
and revision.
- sync-tags="false" means do not doownload all tags, as we would get
100 commits of history for each one.
- upstream=... downloads the branch that includes the tag/sha we want.

NOTE: The distance (in commits) between the 'upstream' branch head
and the desired 'revision' must not exceed the 'depth'.

NOTE: When a merge occures in the branch history. The 'depth' is
retrieve for both parent branches.  The high number of merge commits
in the prempt-rt branch results in an near exponential growth in
the download size with increases of 'depth'.

Note: Upstream will add commits to the branch head without warning.
We will eventually fail to repo sync if 'depth' restriction noted above
is exceeded.  An increase in depth may be require, however the upstream
server will fail to serve the git if the requested 'depth' is too
great (e.g. 500 does not work for rt).

Other options considered:
- Create a mirror an a larger git server, like github, and hope that
significantly improves the download speed.

- Download a tarball of the yocto kernel, rather than pulling in it's
git tree.  This will make kernel changes (e.g. CVE's) harder to add.

Closes-Bug: 1933357
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Id3f0b8ae8b2358243f58a55ab02f4aada7a86b66
This commit is contained in:
Scott Little 2021-06-23 10:57:35 -04:00
parent 8967bfcdcd
commit 4430dd4633
2 changed files with 10 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
default.xml
View File

@ -77,8 +77,10 @@
<project remote="gnocchi" revision="master" name="python-gnocchiclient.git" path="cgcs-root/stx/git/python-gnocchiclient"/>
<!-- linux-yocto Upstream -->
<project remote="linux-yocto" revision="1e798745fa8ef91ffe4fd38d443f9d44b59e3cb3" name="linux-yocto" path="cgcs-root/stx/git/linux-yocto-std"/>
<project remote="linux-yocto" revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
<project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="500" upstream="v5.10/standard/intel-x86" revision="refs/tags/v5.10.30" name="linux-yocto" path="cgcs-root/stx/git/linux-yocto-std"/>
<!-- this sha merges 5.10.37 into preempt-rt -->
<project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
<!-- Pure Upstream -->
<project remote="openstack" revision="stable/train" name="horizon.git" path="cgcs-root/stx/git/horizon"/>

6
distro.xml
View File

@ -6,6 +6,7 @@
<remote name="openstack" fetch="https://opendev.org/openstack" />
<remote name="starlingx" fetch="https://opendev.org/starlingx" />
<remote name="stx-staging" fetch="https://github.com/starlingx-staging"/>
<remote name="linux-yocto" fetch="https://git.yoctoproject.org/git/" />
<!-- note, you can also provide personal overrides for your own github -->
<!-- for tests/experiments -->
@ -37,6 +38,11 @@
<project remote="openstack" revision="stable/train" name="python-pankoclient.git" path="cgcs-root/stx/git/python-pankoclient"/>
<project remote="gnocchi" revision="master" name="python-gnocchiclient.git" path="cgcs-root/stx/git/python-gnocchiclient"/>
<!-- linux-yocto Upstream -->
<project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="500" upstream="v5.10/standard/intel-x86" revision="refs/tags/v5.10.30" name="linux-yocto" path="cgcs-root/stx/git/linux-yocto-std"/>
<!-- this sha merges 5.10.37 into preempt-rt -->
<project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
<!-- Pure Upstream -->
<project remote="openstack" revision="stable/train" name="horizon.git" path="cgcs-root/stx/git/horizon"/>