Skip to content

ci: bound full Python shards across concurrent runs - #6274

Draft
loopx-agent wants to merge 2 commits into
mainfrom
codex/ci-shard-admission-20261012
Draft

loopx-agent wants to merge 2 commits into
mainfrom
codex/ci-shard-admission-20261012

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Concurrent full Python matrices can occupy the shared GitHub-hosted pool and delay release work. This candidate adds four repository-wide job concurrency groups, one for each existing matrix.shard, with queue: max and no cross-candidate cancellation. A current candidate waits behind the same shard of another run rather than replacing it. Workflow-level cancellation still retires an obsolete head of the same PR.

Scope: S10 reliability/capacity and S12 developer/release operations in the overall roadmap. Base: main. GitHub's concurrency contract permits one running job and up to 100 pending jobs per group; excess jobs are cancelled.

Author Declaration

Written by: model_agent — OpenAI Codex, GPT-6 family.

Implemented against: this accepted capacity mitigation request and GitHub's job-concurrency contract; existing CI gate at base fb5273002fae9d652a0c21c579bb65ee07755f18.

Criterion Disposition Evidence
Four shared long-test lanes; retain valid candidates implemented test-shard.concurrency; exact group/context and nonreplacement fixtures
Preserve complete collection, four partitions, two workers, deadlines, reports, coverage floor and required gates implemented Whole parsed workflow differs only in job concurrency; executable gate regressions
Cancellation/overflow status or missing coverage cannot pass the required gate implemented locally Actual gate Bash rejects all non-success combinations and every missing/empty coverage partition
GitHub workflow admission and cancelled-candidate terminal readback implemented PR/manual runs accepted at this head; owned manual terminates cancelled with merge-gate/pytest/test-shard cancelled
Same-shard serialization, four-running bound and added lane-wait cost deferred to live qualification The current PR has all four shards on real runners; the new owned manual reached verified terminal inside the original 20-minute bound, before shard admission. Cross-run scheduling and added wait still need real evidence
Hard global release reservation or highest priority out_of_scope Requires separately qualified independent capacity; other workflows/repos and old versions remain outside this cap

Scope And Continuation

This is a reversible mitigation for long Python shards, independent of closed-PR cleanup #6269 and the desktop download resolver #6271. All three changed files serve this outcome: workflow admission, regression checks, and bilingual operating/rollback guidance. Existing GitHub Actions owns admission; no LoopX scheduler, capability, permission, or release publisher is introduced.

Valid contributors can wait longer when full runs overlap. Only runs carrying this workflow participate, and other jobs still use the organization's shared pool. Rollback removes the job-level concurrency block while preserving workflow-level cancellation and the mandatory gate. The same draft owns live qualification; it is not ready for merge until that evidence and exact-head review exist. No merge authorization is assumed.

Future-facing pass: retained the existing four-value matrix and gate file, using small test-local extraction helpers to execute the real Bash guards. No scheduler simulation or rehearsal-only workflow was added.

Validation

Tested revision: 4d3536ca0259965b1a6374bb8ee9c8e95481f538 (unchanged committed source after local validation).
Run state: running; live cross-run qualification and three failed non-shard checks remain open.
Input classes: synthetic, public_fixture.

Check kind Result Evidence / limitation
unit / real_entrypoint passed uv run --python 3.11 --extra test python -m unittest discover -s scripts/ci -p 'test_*.py': 12 contracts; actual workflow Bash exercised over 25 result combinations and all missing/empty coverage partitions
regression_parity passed New admission fixture fails on unchanged base; existing gate and coverage fixtures pass before and after. Complete parsed YAML comparison confirms the sole workflow delta is test-shard concurrency
static passed Changed Python Ruff, diff hygiene, signed commits; all three public paths scanned without warnings/errors
static passed Diff semantic advisory then full examples/semantic-vocabulary-drift-smoke.py; advisory does not analyze YAML. Initial missing npm dependencies recovered with npm ci --ignore-scripts
integration passed Risk-based premerge: 4 direct checks, 9 catalog canaries, 8 risk-profile smokes and explicit public-boundary scan; exact-scope quality receipt valid
real_backend partial / hold Current PR run has all four shards on assigned runners. Windows SQLite CLI, packaged browser and dashboard acceptance checks failed. Their command/test owners and non-shard workflow definitions are unchanged from the exact base, and the same two Windows note-corruption failures now occur in the exact base run. The two base browser checks remain queued, so their empirical attribution is still unproven. The first owned manual has all 18 jobs terminal/cancelled or skipped, but missed the frozen terminal bound by 647s. One changed-window owned overlap trial now has all 18 jobs terminal (17 cancelled, one skipped); provider terminal time is 16m16s and observer readback is 16m24s, within the original 20-minute bound. It never received a runner before the first prerequisite job was cancelled; no cross-run shard overlap was exercised

Coverage: no collection, timeout, worker, artifact, permission, or gate changes. Fork test jobs receive no new secret or publishing authority. Queue overflow maps to cancellation under the provider contract; local tests reject cancellation without claiming a 101-job overflow drill. Source equality is not a waiver for the failed checks. The dashboard refresh owner has an existing candidate #6273, which is unmerged and not adopted by this PR.

Same-index cross-run serialization, the four-running bound and added lane-wait cost remain open. The new manual's first changes job waited for the shared runner pool until cancellation, before per-shard concurrency admission; that prerequisite wait cannot be charged to the new lane. The deadline supervisor requested ordinary cancel 4.65s after the 16-minute target, then freshly requalified the same owned run before force-cancel. Acceptance was followed by a complete terminal job readback; merge-gate, pytest and test-shard are cancelled and cannot be treated as green. This trial does not rewrite the earlier 647-second failure. Another same-head overlap trial requires evidence that its prerequisites can obtain runners within the unchanged terminal window; additional PR shard starts alone are insufficient.

Frontend / Visual Evidence

UI impact: none. Developer-facing results remain in existing GitHub Actions/Checks; no LoopX frontend/Lark/CLI capability or first screen changed. Shared-authority RFC fixture impact: N/A.

Type / Boundary

Build/CI behavior, public documentation, durable regression tests. Default change: full Python jobs now wait across runs by shard. Neither this diff nor its evidence publishes private state, local paths, credentials, raw traces, or internal links. Both commits carry DCO sign-offs. No benchmark jobs or release publication is launched by this change.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant