bounded retrying multi-repo push and publish (knit) - #178
Merged
marc-merino merged 1 commit intoAug 29, 2026
Conversation
A 50-100 repo bundle used to open one thread per repo: 100 simultaneous git pushes and 100 simultaneous forge writes, each able to block forever on one stalled connection, with no distinction between a host that was busy and a host that said no. Streaming the output (alpha.12) fixed the silence; it did not fix the storm. Three things now hold the fan-out together: - A small std worker pool (src/parallel.rs) runs at most KNIT_GIT_JOBS (8) git pushes and KNIT_FORGE_JOBS (4) forge writes at a time. Both knit push and both knit publish create paths use it, and the per-repo streaming shape is unchanged. An invalid limit is an error, not a silently ignored setting. - Every git push is bounded: git_output_with_timeout spawns the child, drains its pipes, and kills it after KNIT_GIT_PUSH_TIMEOUT (300s), naming the repo, the branch and the limit. The two copies of run_push are now one shared function. - Failures that are weather are retried; failures that are answers are not (src/retry.rs). git push retries a reset connection, a hung-up remote or a timeout up to 3 times and never retries a rejected, stale-lease or unauthenticated push. Forge calls retry 5xx, 429, a secondary-rate-limit 403 and transport errors up to 4 times with 1s/2s/4s backoff, honoring Retry-After (capped at 60s), and never retry 401/404/422 or any other 403. Both GitHub transports are covered: the gh CLI through providers::cli_output and the native client through a typed HttpFailure that carries status and Retry-After. Retries announce themselves on the same channel as the repo's result, so the main thread stays the only writer. Because a retried PR create may follow a POST that in fact succeeded, a create that comes back "already exists" now adopts the existing review instead of failing a run that worked. With that, a re-run only does the work that is missing: pushed branches are up to date and published repos are found, so both commands are safe to run again, and their failure messages say so. Docs and the generated agent guides now teach commit then publish create as the review path -- publish pushes the branches itself -- and describe knit push as the branch-only path. Test scaffolding: hooks are installed in the git *common* dir, because git never ran the per-worktree hooks the parallel-push fixture was writing; with that fixed the existing gate hooks actually gate. Knit-Group: kg_20260829_12fda0 Knit-Bundle: bounded-retrying-multi-repo-push-and-publish
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Knit Bundle
This PR is part of Knit bundle
bounded-retrying-multi-repo-push-and-publish.See the other review objects in this bundle:
knit: bounded retrying multi-repo push and publish (knit) #178 (this PR)Bundle id:
bounded-retrying-multi-repo-push-and-publishBundle title: bounded retrying multi-repo push and publish