feat(eval): replace internal harness with black-box eval - #104
Conversation
feat(admin): add OpenPlait observability dashboard
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4f60b921fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| github.ref == 'refs/heads/main' && (github.event_name == 'push' || needs.changes.outputs.release == 'true') && | ||
| needs.changes.outputs.publish == 'true' && | ||
| (needs.checks.result == 'success' || needs.checks.result == 'skipped') && | ||
| (needs.integration.result == 'success' || needs.integration.result == 'skipped') |
There was a problem hiding this comment.
Checked .github/workflows/ci.yml: publishing on main depends only on the scoped checks and integration jobs, and a repo-wide search finds no caller of the new standalone black-box-eval.yml workflow. Consequently, a production release can publish images and proceed through manifest update and rollout without running the Candidate/Judge evaluation or validating a reviewed baseline, so the advertised release gate cannot block a regression; add the reusable evaluation job to this dependency chain and require its success.
Useful? React with 👍 / 👎.
| </style> | ||
| </head> | ||
| <body> | ||
| <header><h1>Archify visual-check</h1><p>agent-os-deep-dive-canvas.html · automated containment fail · visual review pending</p></header> |
There was a problem hiding this comment.
This commit checks in browser-generated visual-check and Playwright artifacts, including screenshot sets and reports explicitly marked as automated with visual review still pending; these add roughly 18 MB of generated test output and establish the browser-automation artifact surface that the repository rules prohibit. Remove the generated reports, screenshots, and Playwright review harness from the commit rather than retaining them as repository assets.
AGENTS.md reference: AGENTS.md:L38-L40
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
4f60b92 to
ad11b1a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad11b1a18e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (error instanceof ModelError) { | ||
| if (phase === 'candidate') sample.candidate = error.usage | ||
| else sample.judge = addUsage(sample.judge, error.usage) |
There was a problem hiding this comment.
When a Candidate or Judge request reaches the provider but fails before a valid usage block is parsed—for example on a timeout, non-2xx response, malformed JSON, or missing usage—ModelError.usage is still the initial zero-valued object, and these assignments record that as known usage. The report consequently omits candidate_usage_unknown and reports zero or partial cost for a potentially billable call (Judge usage is likewise always initialized to zero), making the persisted cost evidence unreliable; preserve an explicit unknown/incomplete-usage state on these error paths rather than treating accumulated zero usage as complete.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b14138f5d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| `SELECT id,email,display_name,email_verified_at | ||
| FROM users | ||
| WHERE id=$1 AND deleted_at IS NULL AND suspended_at IS NULL`, |
There was a problem hiding this comment.
For any authenticated non-admin user, the generic proxy in workers/control-plane/src/index.ts:600-610 forwards /api/admin/* with a valid gateway assertion, while this lookup accepts every active product user without verifying the Better Auth admin role. Such a user can therefore access cross-tenant resources and invoke the suspend/delete endpoints; either exclude /api/admin/* from the generic proxy or carry and verify an authenticated administrator claim at the origin.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
| - if: needs.changes.outputs.deploy_contract == 'true' | ||
| run: node --test scripts/production-rag-contract.test.mjs |
There was a problem hiding this comment.
When a change touches only one of the configured Open Notebook/RAG paths, computeScope marks the image for publication, but the only Open-Notebook-conditioned verification here is this JavaScript deployment-contract test. The workflow never installs the Python environment or runs the owning pytest, Ruff, or mypy checks, so broken RAG imports, migrations, or backend behavior can pass checks and be published and rolled out automatically; add the scoped vendored backend gates to this job.
AGENTS.md reference: third_party/open-notebook/AGENTS.md:L22-L22
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 92c58f80dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| void run(async () => { | ||
| const result = await authApi.signIn(String(data.get('email')), String(data.get('password')), captchaToken) | ||
| if (result.error) throw new Error(result.error.message) | ||
| }, () => location.assign(parameters.get('returnTo') ?? '/'), true) |
There was a problem hiding this comment.
When a user signs in through an attacker-crafted URL, returnTo is read directly from the query string and passed to location.assign; values such as an external HTTPS URL create a post-authentication open redirect, while a javascript: value can execute in the authenticated app origin where no CSP blocks it. Parse this value and permit only same-origin application paths before navigating.
Useful? React with 👍 / 👎.
| resetPassword: (newPassword: string, token: string) => authClient.resetPassword({ newPassword, token }), | ||
| sendVerification: (email: string) => authClient.emailOtp.sendVerificationOtp({ email, type: 'email-verification' }), | ||
| verifyEmail: (email: string, otp: string) => authClient.emailOtp.verifyEmail({ email, otp }), | ||
| deleteAccount: () => http<DeleteAccountResponse>('/me/account', { method: 'DELETE' }), |
There was a problem hiding this comment.
Every self-service deletion attempt still calls DELETE /api/me/account, but this change deletes identityServiceRoutes, removes it from the API router, and a repo-wide search of the reviewed commit finds no replacement server handler. The request therefore reaches the origin and returns 404, leaving both the PostgreSQL product account and Better Auth/D1 identity intact; restore an authenticated deletion flow that cleans up both stores and revokes the user's sessions.
Useful? React with 👍 / 👎.
1 participant
Replaces the product-coupled Eval stack with an independent, replayable black-box package. Candidate and Autoevals Judge configuration, API calls, usage, cost and traces are isolated; versioned datasets/suites, SQLite jobs, baselines, regression gates, reports and an OpenTelemetry-compatible telemetry contract are included.\n\nRemoves the old server Eval modules, Admin dashboard/API, bootstrap tables, scripts, tests and CI paths. AgentOS remains a type-only target boundary for a future adapter.\n\nValidation:
pm run eval:check; classifier tests; version/brand/AgentOS/LLM guards. Root type/lint checks could not complete because the local root dependency install remained incomplete after cleanup; CI will run the full matrix.