Buildah Vendor Treadmill

This document describes the Buildah Vendor Treadmill, a two-part system used for easing the burden of vendoring buildah into podman. The two parts are:

1. sync - run periodically by a human (as of this writing, Ed). In theory this is very simple:
   1. Preserve current set of treadmill patches (i.e. results of last sync + fixes)
   2. git pull to bring in latest podman @ main;
   3. Rebase the treadmill PR onto podman @ main;
   4. Bump go.mod, pull buildah @ main, and make vendor. git commit this;
   5. Apply rolling set of treadmill fixes (if any) and git commit;
   6. run a few tests; fix any problems;
   7. (It's a little trickier than that, and for the most part Ed worries about it so you don't have to)
2. pick - run ad hoc by a developer needing to vendor in a new buildah into podman. This is the part that you, dear reader, probably want to know about.

We will cover those in reverse order, because chances are you're here for vendoring buildah.

---

For Vendoring Buildah (Start Here)

Target Audience: a developer vendoring buildah into podman

Most of the time the vendoring will go smoothly. If you're reading this it's because things haven't gone smoothly. Most likely the bud tests are failing: either the patching step, or the bud tests themselves. You should now run, from your buildah-vendor branch:

$ sudo dnf install perl-open perl-JSON perl-libwww-perl perl-Term-ANSIColor
$ hack/buildah-vendor-treadmill --pick

This will identify the treadmill PR (currently #13808) and cherry-pick its patches onto your branch. If Ed hasn't been slacking, this will resolve your vendoring problems and you can git commit --amend (to update the commit message) then git push --force and make it past CI.

As of this writing, the pick process only works when vendoring into main. If you need to vendor a new buildah into a maintenance branch, I'm sorry, you're on your own. (But chances are good that someone has already vendored that buildah into main, and you can cherry-pick the required changes).

As we gain experience with this process, please update this document to reflect best practices.

---

For Periodic Integration (What Ed Does)

Target audience: someone willing to run the sync step daily/weekly and fix problems. This is what Ed does periodically. Steps involved:

• keep a vendor_buildah_latest branch active
• run the sync step
• fix problems if they arise

In the years since the development of this process, it has been super easy: it works cleanly:

$ hack/buildah-vendor-treadmill --update
-> buildah old = v1.25.2-0.20220412203738-d41a4fd27c19
|
+---> HEAD is buildah vendor (as expected); dropping it...
|
+---> Pulling podman main...
remote: Enumerating objects: 15, done.
[...]
 * branch                main       -> FETCH_HEAD
   d6f47e692..712c3bb22  main       -> upstream/main
|
+---> Rebasing on podman main...
Successfully rebased and updated refs/heads/vendor_buildah_latest.
|
+---> Vendoring in buildah...
[...]
all modules verified
-> buildah new = v1.25.2-0.20220418231153-74cd96acf358
|
+---> Running 'make' to confirm that podman builds cleanly...
[...]
|
+---> Cross-checking man pages...
|
+---> Confirming that buildah-bud-tests patches still apply...
+ git clone -q https://github.com/containers/buildah test-buildah-v1.25.2-0.20220418231153-74cd96acf358
+ git checkout -q 74cd96acf358
+ git tag buildah-bud-in-podman
+ make bin/buildah
[...]
+ git am --reject
Applying: tweaks for running buildah tests under podman
Checking patch tests/helpers.bash...
Applied patch tests/helpers.bash cleanly.
+ /home/esm/.../podman/test/buildah-bud/apply-podman-deltas
|
+---> All OK. It's now up to you to 'git push --force'
|
+--->  --- Reminder: New buildah, new podman. Good candidate for pushing.

On several occasions it hasn't worked cleanly, the errors tend to fall into a small number of categories:

• man page checker - a new option got added to buildah bud. Solution: look at the corresponding buildah PR, find the new docs, copy/paste them into podman-build man page. However, when this happens, you also need to check for:
• new options - these are parsed by podman, but you probably need to muck with buildahDefine.BuildOptions in build.go. Possibly also do something so podman-remote will work.
  • if the new option makes no sense in podman-remote, you will want to add a skip_if_remote to apply-podman-deltas
• other test failures - maybe the new buildah bud tests don't pass under podman build and you just don't want to deal with fixing them now. You can add skips to apply-podman-deltas
• conflicts - when applying the buildah-bud-under-podman patches. Usually happens when someone edits buildah's helpers.bash. Solution: see the buildah bud tests README and take steps as needed. The failure message from the buildah-vendor-treadmill script will give step-by-step instructions, but some of those are handwavy.
• Cirrus changes - changes to podman's .cirrus.yml that change task configuration. These may manifest overtly as syntax/parsing errors when the job runs, in which case they should be easy to fix, or silently in ways that can only be detected by watching CI jobs: perhaps the rootless jobs don't run, or perhaps jobs don't run in the expected order. Not much I can say here, just, be aware.

---

There's also a slight corner case: after a new buildah is vendored into podman, the sync step will probably fail (because of conflicts, because the changes needed to bring in the vendor are probably slightly different than the ones Ed has been tweaking in the treadmill script). When this happens, the best solution is to start from scratch:

• Check out your vendor_buildah branch
• Drop all commits (i.e. make sure your vendor_buildah branch is a no-commit one. No buildah-vendor commit, no treadmill commit)
• Run hack/buildah-vendor-treadmill --reset. This will create two dummy empty commits fitting the magic formula. Once those commits exist, --sync will start working again.