Skip to content

Add per-role effort configuration to Quest - #181

Open
KjellKod wants to merge 8 commits into
mainfrom
feat/quest-effort-dispatch
Open

KjellKod wants to merge 8 commits into
mainfrom
feat/quest-effort-dispatch

Conversation

@KjellKod

@KjellKod KjellKod commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

⭐ Why this matters

Configure reasoning effort separately for each Quest role, so planners, builders and reviewers can use the settings you choose for their work. Quest applies those settings during dispatch and preserves them in journals and celebrations for later review.


Summary

Add per-role effort configuration: set defaults in .ai/allowlist.json, save the selected settings in each quest's orchestration.json, and apply them whenever a role runs.

The initial default is medium for Claude and Codex roles. Existing model assignments and review policy remain unchanged; medium is a starting policy, not a measured cost/quality optimum.

Warning

Existing allowlists need the new effort map. Incompatible saved quests require explicit reconfiguration or a new quest. The installer refuses incompatible upgrades before writes, including locally modified framework files that also changed upstream. See the upgrade guide.

Changes

  • Configuration: Add effort.<role> defaults and validation. Reviewer A/B settings remain independent. Resume uses the saved quest settings. Gemini effort pins are unsupported and reported explicitly; standalone /gpt keeps its existing scalar setting.
  • Dispatch: Apply saved model and effort when launching roles. Claude uses the existing background runner and persona under either orchestrator; Codex retains its native or CLI route. Effort is not duplicated in agent Markdown.
  • Reviewable history: Show configured effort beside agents in journals, saved celebrations and credits. Embed it in the existing journal payload so replay does not depend on current defaults. The label describes the last saved configuration, not verified effective effort for every invocation.
  • Installation: Check upgrade prerequisites before changing files in the current installer pass, while preserving the existing manifest backup/replacement behavior. An older installer may update itself before restarting into this check.
  • Background cleanup: Fix a defect reproduced during live validation by using claude stop and verifying session removal. Intentional human-input parking remains unchanged. Requires a Claude CLI supporting stop, tested with 2.1.286.

Validation

Pending end-to-end acceptance

These two checks are still outstanding. They exercise the installed workflow, beyond the completed role-level tests below. Agents should drive them where runtime access permits; human involvement is limited to required approval gates or access. Use a separate disposable Git repository for each run, with a Python test command configured in its allowlist.

Install this PR branch from inside each disposable repository:

curl -fsSL https://raw.githubusercontent.com/KjellKod/quest/feat/quest-effort-dispatch/scripts/quest_installer.sh -o /tmp/quest-effort-installer.sh
bash /tmp/quest-effort-installer.sh --branch feat/quest-effort-dispatch

Record the installed revision and the orchestrator model/version. Use the normal supported orchestrator model, with authenticated Claude background execution and Codex access available. Set role effort defaults to planner medium, builder high, and code reviewers low; leave other valid role defaults in place. The differing values make accidental default reuse visible.

  • Claude-orchestrated installed Quest: Start Claude Code in the first repository and run /quest "Add a greeting function that returns Hello, <name>! and a meaningful automated test. Keep the change minimal." Complete planning, required approvals, build, review and completion/archive. Confirm the saved role efforts and actual dispatch arguments match the configured values, the greeting test passes, canonical artifacts exist, and owned background sessions are cleaned up. Verify archived orchestration.json, journal and celebration show the configured effort. Replay the generated journal with /celebrate <journal-path> and confirm the settings persist. Record evidence and any unavailable effective-effort metadata; do not substitute agent self-report for runtime evidence.
  • Codex-orchestrated installed Quest: Start Codex in the second repository and run the same Quest using $quest. Complete the same lifecycle and checks. Confirm Codex roles use native dispatch and Claude roles use the background runner, with explicit saved effort controls. After completion, change the repository's effort defaults and replay the generated journal with $celebrate <journal-path>; expected: historical configured effort remains unchanged. Confirm no missing/invalid pin silently falls back by testing the existing configuration validator against a disposable copy with one active role's effort removed; expected: a clear rejection before dispatch. Record the installed revision, dispatch evidence, test result, archive/replay result and cleanup result.

A tiny greeting task is sufficient for this integration check. It does not establish an effort level's quality/cost benefit. Mark these boxes complete only after the full runs and evidence have been recorded.

Completed validation

  • Final commit 49efd16 CI: 1,374 passed, 7 skipped. Configuration, security, formatting and Codex review checks passed.

  • Optional-cache follow-up: 28/28 preflight cases passed, including successful probes with unwritable cache paths in both automatic and explicit background mode. Regression failed first; self-review completed before push.

  • October 5 review fixes: 1,381 tests passed before the final optional-cache follow-up, plus Black, generated-default, manifest and configuration checks. Every fix was self-reviewed before push; final independent cross-check found no actionable defects. Scoped checks cover saved transport recovery, invalid configuration/state, preflight cleanup, retained sessions, stale framework symlinks and missing-model effort display.

Completed by agents, no human repetition needed:

  • Journal follow-up: 167 focused tests passed, including seven new cases covering archive preservation, distinct reviewer settings, journal replay and missing/malformed effort. Black and whitespace checks passed; independent review found no actionable issues.

  • Unchanged-main baseline: 1,316 tests. Replacement suite before the journal follow-up: 1,356 passed.

  • Shell checks: orchestration 36/36, state 72/72, runtime 72/72, preflight 26/26, handoff contracts 3/3.

  • Black, generated-model defaults, configuration, manifest and whitespace checks.

  • Full installer acceptance: fresh install, incompatible-upgrade refusal without mutations, explicit recovery, custom configuration preservation and manifest backup.

  • Independent configuration, dispatch, installer and scope reviews; real Claude source review. Findings addressed.

  • Live Claude persona/tool/model/effort evidence, local and outside-in; Claude-led Claude builder at high effort; Claude-led Codex builder; controlled permission denial; canonical artifacts.

  • Fresh cleanup retest: session absent immediately and 44.1 seconds later without manual cleanup.

Boundaries: effective Codex model/effort was not independently reported. Native Codex selection is covered deterministically; the earlier native live smoke belongs to #179. No API bridge inference was used. Installer recovery did not launch a provider from the recovered installation.

Human attention: review the medium default and deliberate upgrade/resume policy. Full evidence and limitations: validation ledger. Design decisions and lessons: replacement plan.

     ▐▛███▜▌
    ▝▜█████▛▘
      ▘▘ ▝▝
Quest/Co-Authored by
Co-Authored-By: Codex 
Co-Authored-By: Claude Opus 5.5 
in collaboration with KjellKod

KjellKod and others added 2 commits September 30, 2026 23:48
Keep per-role effort in the saved orchestration snapshot and dispatch
Claude roles through the existing background runner. Preserve model defaults.

Refuse incompatible upgrades before writes. Verify background cleanup with
claude stop and session roster removal after live acceptance exposed survivors.
Record independent review and automated/live validation in the history ledger.

Quest/Co-Authored by
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
in collaboration with KjellKod <kjell.hedstrom@gmail.com>
Preserve each role's last saved effort in the existing celebration payload
so archived journals replay without consulting current defaults. Omit absent
or malformed values and label settings as configured rather than effective.

Quest/Co-Authored by
Co-Authored-By: Codex <noreply@openai.com>
in collaboration with KjellKod <kjell.hedstrom@gmail.com>
@KjellKod
KjellKod marked this pull request as ready for review October 1, 2026 06:14
@KjellKod
KjellKod deployed to codex-ci-review October 1, 2026 06:15 — with GitHub Actions Active
@KjellKod KjellKod changed the title Honor saved effort in Quest role dispatch Add per-role effort configuration to Quest Oct 1, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 42 files

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic

Comment thread scripts/quest_installer.sh Outdated
Comment thread scripts/quest_preflight.sh
Comment thread scripts/quest_preflight.sh
Comment thread .ai/schemas/allowlist.schema.json
Comment thread scripts/quest_runtime/orchestration.py
Comment thread docs/implementation/history/quest-effort-validation.md Outdated
Comment thread scripts/quest_installer.sh
Comment thread scripts/quest_validate-quest-state.sh Outdated
Comment thread scripts/quest_complete.py Outdated
Comment thread scripts/quest_claude_bg_run.py
Comment thread scripts/quest_installer.sh
@KjellKod
KjellKod deployed to codex-ci-review October 5, 2026 14:50 — with GitHub Actions Active

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 4 files (changes from recent commits).

Requires human review: Auto-approval blocked by 17 unresolved issues from previous reviews.

Re-trigger cubic

@KjellKod
KjellKod deployed to codex-ci-review October 5, 2026 14:56 — with GitHub Actions Active

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 6 files (changes from recent commits).

Requires human review: Auto-approval blocked by 9 unresolved issues from previous reviews.

Re-trigger cubic

@KjellKod
KjellKod deployed to codex-ci-review October 5, 2026 15:02 — with GitHub Actions Active
@KjellKod
KjellKod deployed to codex-ci-review October 5, 2026 15:04 — with GitHub Actions Active
@KjellKod
KjellKod deployed to codex-ci-review October 5, 2026 15:09 — with GitHub Actions Active

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread scripts/quest_preflight.sh
@KjellKod
KjellKod deployed to codex-ci-review October 5, 2026 15:12 — with GitHub Actions Active

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 6 files (changes from recent commits).

Requires human review: Adds per-role effort configuration with saved orchestration compatibility checks and Claude background dispatch under both orchestrators; end-to-end acceptance is pending and the upgrade/dispatch policies need human sign-off.

Re-trigger cubic

This branch was successfully deployed

1 active deployment
codex-ci-review — 49efd161 Deployed Oct 5, 2026 by KjellKod via codex-review #750
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant