Stamp image tags from STABLE_ status keys - #312
Closed
matx-sjacob wants to merge 1 commit into
Closed
Conversation
matx-sjacob
marked this pull request as draft
August 5, 2026 18:13
matx-sjacob
force-pushed
the
sjacob/stable-stamp-keys
branch
5 times, most recently
from
August 6, 2026 09:17
db8da3d to
d9b098f
Compare
matx-sjacob
marked this pull request as ready for review
August 6, 2026 09:48
matx-sjacob
force-pushed
the
sjacob/stable-stamp-keys
branch
from
August 6, 2026 14:07
d9b098f to
61d53fd
Compare
BUILD_SCM_REVISION and BUILD_SCM_TIMESTAMP have no STABLE_ prefix, so Bazel files them in volatile-status.txt and reports an unchanging digest for that file whatever it contains. The expand_template in bb-storage that turns them into the image tag is therefore never invalidated, and _stamped.tags.txt is generated once and reused indefinitely. Ephemeral CI never sees this, since each job starts with an empty cache. On a build machine whose cache persists, every later build pushes its image to the tag produced by the first build on that machine, silently overwriting what was published there before. Bump bb-storage to 33530b6 so the template consuming these keys expects the prefixed names. Renaming the keys without the bump would have no effect, since the pinned template still reads the volatile ones. Move go.mod to the matching commit as well, so a plain "go build" resolves the same version the Bazel build does.
matx-sjacob
force-pushed
the
sjacob/stable-stamp-keys
branch
from
August 6, 2026 21:47
61d53fd to
a00df6d
Compare
Author
|
@EdSchouten please could I get a review for this one? It should be identical to buildbarn/bb-remote-execution#244 |
Collaborator
|
Looks good! I took your changes and updated the module lock file with the latest commits on main in #336. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related buildbarn/bb-storage#358
BUILD_SCM_REVISIONandBUILD_SCM_TIMESTAMPhave noSTABLE_prefix, so Bazel files them involatile-status.txtand reports an unchanging digest for that file whatever it contains. Theexpand_templatein bb-storage that turns them into the image tag is therefore never invalidated, and_stamped.tags.txtis generated once and reused indefinitely — on a build machine whose cache persists, every later build pushes its image to the tag produced by the first build on that machine.This is where we hit it, on a self-hosted runner. In a single job the checkout was at
0250440andvolatile-status.txtcorrectly saidBUILD_SCM_REVISION 0250440, while_stamped.tags.txtstill held20260407T093634Z-220cdb7with an April mtime — four months and two version bumps stale.The bb-storage bump is part of the fix, not incidental. That template is what expands these keys, and the pin here (
10acc76a) predates33530b6, which switched it to the prefixed names. Renaming the keys alone would leave the pinned template still reading the volatile ones, so nothing would change.