Skip to content

Commit 76f2627

Browse files
committed
Document --config's OAuth-compose behavior, add wiring-level effort rejection test
--config is a general CLI flag, not eval-only: it now composes with home-level codex/xai OAuth credentials instead of excluding them. Document that in docs/PRODUCT.md and docs/IMPLEMENTATION.md (prose and flag table) so an operator relying on --config to pin a single API-key provider on a shared/CI machine knows OAuth sessions in ~/.corbits/codex-auth.json / xai-auth.json now apply too when their config names a codex/* or xai/* provider. Export validateVariantEfforts and add a wiring-level test exercising the full parseArgs -> parseMatrix -> validateVariantEfforts path for an unsupported model/effort matrix cell (grok-composer-2.5-fast:xhigh), plus the matching accept case.
1 parent a49c1d1 commit 76f2627

4 files changed

Lines changed: 42 additions & 4 deletions

File tree

‎docs/IMPLEMENTATION.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -283,6 +283,8 @@ OpenAI-compatible `baseURL` values are normalized during provider resolution. A
283283

284284
`--config <path>` replaces the global settings file as the provider source (useful for CI to inject a provider per run). The per-repo `.corbits/settings.json` selection still applies on top of a `--config` source (definitions come from `--config`, selection from the local file; CLI `--provider`/`--model` override both). A provider must be defined in one of these settings files; there is no environment-variable fallback.
285285

286+
`--config` composes with, rather than replaces, the home-level OAuth profile catalog: codex/xai credentials live in `~/.corbits/codex-auth.json` and `xai-auth.json`, entirely separate from settings.json, and are merged into the resolved provider catalog on every run regardless of `--config` (CL-6973). A `--config` file that names a `codex/*` or `xai/*` provider by ID does not by itself grant that provider's credentials — those come from the OAuth store whenever a matching profile exists there, independent of which settings file supplied the provider definitions. The only way to fully exclude the home OAuth catalog is the programmatic `globalSettingsPath` option to `loadConfig`, used by tests for full isolation; it is not exposed as a CLI flag.
287+
286288
### Profiles (`src/config/profiles.ts`)
287289

288290
Profiles supply per-project or named-profile overrides for `model`, `maxTurns`, and `systemPromptExtensions` (the only allowed keys; any other key is rejected on load).
@@ -323,7 +325,7 @@ Printed by `corbits --help` / `-h` from `CLI_HELP_TEXT` in `src/config/index.ts`
323325
| `resume <session-id>` | — | Reopen a specific session |
324326
| `resume --pick` / `--list` | — | Interactive session picker |
325327
| `--cwd <dir>` | `process.cwd()` | Working directory |
326-
| `--config <path>` | `~/.corbits/settings.json` | Settings file to use |
328+
| `--config <path>` | `~/.corbits/settings.json` | Settings file to use for provider definitions; composes with (does not exclude) home-level codex/xai OAuth credentials |
327329
| `--provider <name>` | from settings | Select a configured provider |
328330
| `--model <id>` | provider default | Select a model for the active provider |
329331
| `--profile <name>` | — | Settings profile |

‎docs/PRODUCT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ The exact turn thresholds are model-family-dependent (tighter for models with ob
132132

133133
## Configuration
134134

135-
Providers and models are configured in `~/.corbits/settings.json` (holds providers + credentials), with a selection-only per-repo `.corbits/settings.json` override. Select at launch with `--provider` / `--model`, or point at an alternate file with `--config <path>`. Credentials are read only from these settings files — there is no environment-variable override and `.env` files are not loaded, so a stale or exported key can't shadow the configured provider. The agent is denied read access to both settings files.
135+
Providers and models are configured in `~/.corbits/settings.json` (holds providers + credentials), with a selection-only per-repo `.corbits/settings.json` override. Select at launch with `--provider` / `--model`, or point at an alternate file with `--config <path>`. `--config` only overrides where provider _definitions_ come from; it composes with, rather than replaces, credentials for codex/xai OAuth-profile providers, which live in separate home-level auth stores (`~/.corbits/codex-auth.json`, `xai-auth.json`) and are merged into the catalog regardless of `--config`. Credentials are read only from these settings files and the OAuth auth stores — there is no environment-variable override and `.env` files are not loaded, so a stale or exported key can't shadow the configured provider. The agent is denied read access to both settings files.
136136

137137
## Optional Capabilities (plugins)
138138

‎scripts/eval-capability.test.ts‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,14 @@ import { join } from "node:path";
55
import { execFile } from "node:child_process";
66
import { promisify } from "node:util";
77

8-
import { initEvalGitRepo, mapPool, parseArgs, buildEvalDiagnostics } from "./eval-capability.js";
8+
import {
9+
initEvalGitRepo,
10+
mapPool,
11+
parseArgs,
12+
buildEvalDiagnostics,
13+
validateVariantEfforts,
14+
} from "./eval-capability.js";
15+
import { parseMatrix } from "../evals/capability/lib.js";
916
import type { Config } from "../src/config/index.js";
1017

1118
const execFileAsync = promisify(execFile);
@@ -174,6 +181,35 @@ describe("parseArgs", () => {
174181
});
175182
});
176183

184+
describe("validateVariantEfforts", () => {
185+
// Wiring-level regression: parseArgs -> parseMatrix -> validateVariantEfforts,
186+
// the same path main() runs before any inference. A matrix cell pairing an
187+
// effort the model does not accept must fail fast, naming the model and its
188+
// accepted levels, rather than silently falling back to the provider default
189+
// and poisoning the matrix.
190+
test("rejects an unsupported model/effort matrix cell before any inference runs", async () => {
191+
const opts = parseArgs(["--matrix", "xai/thegreataxios:grok-composer-2.5-fast:xhigh"]);
192+
const variants = parseMatrix(opts.matrix, {
193+
...(opts.provider !== undefined ? { provider: opts.provider } : {}),
194+
...(opts.model !== undefined ? { model: opts.model } : {}),
195+
...(opts.effort !== undefined ? { effort: opts.effort } : {}),
196+
});
197+
await expect(validateVariantEfforts(variants, opts)).rejects.toThrow(
198+
/grok-composer-2\.5-fast.*does not support reasoning effort "xhigh".*supported: low, medium, high/s,
199+
);
200+
});
201+
202+
test("accepts a supported model/effort matrix cell", async () => {
203+
const opts = parseArgs(["--matrix", "xai/thegreataxios:grok-4.6:xhigh"]);
204+
const variants = parseMatrix(opts.matrix, {
205+
...(opts.provider !== undefined ? { provider: opts.provider } : {}),
206+
...(opts.model !== undefined ? { model: opts.model } : {}),
207+
...(opts.effort !== undefined ? { effort: opts.effort } : {}),
208+
});
209+
await expect(validateVariantEfforts(variants, opts)).resolves.toBeUndefined();
210+
});
211+
});
212+
177213
describe("mapPool", () => {
178214
test("N overlapping jobs with concurrency N finish in ~one job duration", async () => {
179215
const jobMs = 80;

‎scripts/eval-capability.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -583,7 +583,7 @@ async function resolveVariantLabels(
583583
* gpt-5.6 family also takes max/ultra) — silently running at the provider's
584584
* default instead would poison a matrix without anyone noticing.
585585
*/
586-
async function validateVariantEfforts(
586+
export async function validateVariantEfforts(
587587
variants: readonly EvalVariant[],
588588
opts: CliOptions,
589589
): Promise<void> {

0 commit comments

Comments
 (0)