fix(release): land the version bump before the build, not after it

The first real automatic release did everything right — chose 0.25.0, bumped, passed the gate, built all six binaries, smoke-tested the linux one — and then threw it away, because main had moved during the eleven minutes it took and the bump would not fast-forward.

Holding the bump until the end was meant to keep a failed release from leaving anything behind. What it actually did was make every release race every other push to main, over a window as long as the whole job, and lose the entire build when it lost.

Push the bump seconds after checkout instead, rebasing onto main and retrying up to three times. Before anything is built, a rebase costs nothing and stays honest: the gate and the builds then run on exactly the tree that gets tagged. Rebasing at the END would have been worse than losing the race — the binaries come from this tree, and the tag would have named a commit whose tree was never built.

The tag still goes out last, after everything has proven itself. It is what publishes, so it is what is worth withholding.

A release that fails now leaves a bump commit with no tag. That is self-healing, not wreckage: the next run picks the same version, finds the files already at it, and prepare-release.sh takes its tag-only path — verified against a clone.

Author: piotrlaczkowski

.github/workflows/release.yml

@@ -141,6 +141,46 @@ jobs:
           version="$(./scripts/next-version.sh --bump "$BUMP" --explain)"
           echo "selected ${version} for release"
           ./scripts/prepare-release.sh "$version" --no-gate
+
+          # Push the bump NOW, seconds after checkout, rather than at the end.
+          #
+          # It used to go out after the builds, so that a failed release left the
+          # branch untouched. That made every release lose a race it could not
+          # win: the job takes ~40 minutes, and any push in that window — a
+          # dependabot merge, an unrelated fix — made the bump un-fast-forwardable
+          # and threw the whole build away. It happened on the first real run.
+          #
+          # Rebasing at the END instead would be worse than losing the race: the
+          # binaries are built from this tree, and a rebased tag would name a
+          # commit whose tree is not the one that was built. Here, before
+          # anything is built, a rebase is free and honest — the gate and the
+          # builds then run on exactly what was pushed.
+          #
+          # What this trades: a release that fails later leaves the bump on the
+          # branch with no tag and no release. That state is self-healing, not
+          # wreckage — the next run computes the same version, finds the files
+          # already at it, and prepare-release.sh takes its tag-only path.
+          pushed=0
+          for attempt in 1 2 3; do
+            if git push origin "HEAD:${GITHUB_REF_NAME}"; then
+              pushed=1
+              break
+            fi
+            echo "push rejected — ${GITHUB_REF_NAME} moved (attempt ${attempt}/3); rebasing the bump onto it"
+            git fetch origin "${GITHUB_REF_NAME}"
+            git rebase "origin/${GITHUB_REF_NAME}"
+          done
+          if [[ "$pushed" != 1 ]]; then
+            echo "::error::could not land the version bump on ${GITHUB_REF_NAME} after 3 attempts — it is being pushed to faster than this job can rebase. Nothing has been published and no tag exists; try again when it is quieter."
+            exit 1
+          fi
+
+          # A rebase moves the commit, and prepare-release.sh's tag still points
+          # at where it used to be. Re-point it at what was actually pushed, so
+          # the tag published later names the tree that gets built below.
+          git tag -f "v${version}" HEAD
+          echo "bump landed on ${GITHUB_REF_NAME} as $(git rev-parse --short HEAD)"
+
           echo "version=${version}" >> "$GITHUB_OUTPUT"
           echo "tag=v${version}" >> "$GITHUB_OUTPUT"
 
@@ -345,32 +385,33 @@ jobs:
           print("binary reports version=%s commit=%s built=%s" % (info["version"], info["commit"], info["built"]))
           PY
 
-      # The point of no return for an automatic release, and deliberately the
-      # last thing before publishing: the gate has passed, the Studio UI is
-      # verified embedded, all six binaries are built and the linux/amd64 one
-      # has been run and asked what version it thinks it is. Only now do the
-      # bump and its tag leave this runner.
+      # The point of no return, and deliberately the last thing before
+      # publishing: the gate has passed, the Studio UI is verified embedded, all
+      # six binaries are built and the linux/amd64 one has been run and asked
+      # what version it thinks it is. Only now does the tag exist.
+      #
+      # The bump commit itself went out at the start (see "Choose the version and
+      # bump it"), because holding it until here meant losing a 40-minute race
+      # with every other push to the branch. The tag is what actually publishes —
+      # it is what the GitHub Release is cut from and what users download — so it
+      # is the thing worth withholding until everything has proven itself. A bump
+      # without a tag is a no-op that the next run absorbs.
       #
-      # A rejected push means the branch moved during the ~40 minutes this took.
-      # That is NOT something to force past: the binaries above were built from
-      # THIS tree, and forcing would publish them under a branch that no longer
-      # contains the commit they came from. Fail and let a person look.
+      # Unlike the branch, a tag cannot be rejected for moving: nothing else
+      # creates it. The only way this fails is if the tag already exists, which
+      # prepare-release.sh refuses up front.
       #
-      # Pushing the tag here does not start a second Release run: events raised
-      # by GITHUB_TOKEN do not trigger workflows. That restriction is usually a
+      # Pushing it here does not start a second Release run: events raised by
+      # GITHUB_TOKEN do not trigger workflows. That restriction is usually a
       # nuisance; here it is what keeps one button from producing two releases.
-      - name: Publish the version bump (automatic releases)
+      - name: Publish the release tag (automatic releases)
         if: steps.auto.outputs.tag != ''
         env:
           TAG: ${{ steps.auto.outputs.tag }}
         run: |
           set -euo pipefail
-          if ! git push origin "HEAD:${GITHUB_REF_NAME}"; then
-            echo "::error::${GITHUB_REF_NAME} moved while this release was building, so the bump could not be fast-forwarded. Nothing has been published and no tag exists. Re-run the release — the new run reads the newer commits and picks its version again."
-            exit 1
-          fi
           git push origin "refs/tags/${TAG}"
-          echo "pushed the bump and ${TAG} to ${GITHUB_REF_NAME}"
+          echo "published ${TAG} at $(git rev-parse --short HEAD)"
 
       - name: Create GitHub Release
         uses: softprops/action-gh-release@v3

RELEASE.md

@@ -124,11 +124,24 @@ reversible with `git tag -d` and `git reset`.
 beforehand: no local checkout, no bump, no tag. The job reads the version off the
 conventional commits since the last tag (`scripts/next-version.sh`), runs
 `prepare-release.sh` to bump `version.go`, the Makefile, the Formula, `docs/install.md`
-and the changelog, and commits it — **in the runner's tree only**. The bump and its tag are
-pushed at the very end, after the gate has passed, the Studio UI is verified embedded, all
-six binaries are built and the linux/amd64 one has been asked what version it reports. A
-release that fails before then leaves `main` untouched: no bump, no tag, nothing to roll
-back.
+and the changelog, and commits it.
+
+Two things leave the runner, at two different moments:
+
+- **The bump commit goes out immediately**, seconds after checkout. If `main` moved in
+  between, the job rebases onto it and retries (three times) — so the gate and the six
+  builds run on exactly the tree that will be tagged. Holding the bump until the end
+  instead meant losing a 40-minute race with every other push to `main`, and one lost
+  race threw away the whole build.
+- **The tag goes out at the very end**, after the gate has passed, the Studio UI is
+  verified embedded, all six binaries are built and the linux/amd64 one has been asked
+  what version it reports. The tag is what publishes — the GitHub Release is cut from it —
+  so it is the thing worth withholding.
+
+A release that fails partway therefore leaves a `chore: release vX.Y.Z` commit on `main`
+with no tag and no release. That is a no-op, not wreckage, and it heals itself: the next
+run picks the same version, finds the files already at it, and `prepare-release.sh` takes
+its tag-only path. Nothing to revert.
 
 The `bump` dropdown forces the level when you disagree with the commits: