feat: expand capability taxonomy and projections - #22
Merged
Conversation
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdds a layered capability taxonomy (v0.8.0): canonical CapabilityIds, richer NamespaceClassification with facets/evidence/confidence/runner-up data, facet/capability override config, an adapter projection layer mapping classifications to adapter-specific buckets, plus docs, tests, schema, and status output integration. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Classifier as CapabilityClassifier
participant FacetRunner as FacetInference
participant Config as Config/Overrides
participant Projector as AdapterProjector
participant Status as StatusRunner
Client->>Classifier: classifyNamespace(namespace, tools, options)
Classifier->>Classifier: score patterns, build evidence
Classifier->>FacetRunner: inferNamespaceFacets(namespace, tools, facetOverrides)
FacetRunner-->>Classifier: facets + evidence
Classifier->>Config: apply capability/facet overrides
Config-->>Classifier: adjusted classification
Classifier-->>Client: NamespaceClassification (canonical, facets, confidence, runnerUp)
Client->>Projector: projectNamespaceClassification(adapterId, classification, adapterConfig)
Projector->>Projector: score profiles, normalize scores
alt adapter namespace override
Projector-->>Client: AdapterProjectionResult (adapter_override)
else matched profile
Projector-->>Client: AdapterProjectionResult (matched_profile)
else fallback
Projector-->>Client: AdapterProjectionResult (fallback / canonical)
end
Status->>Classifier: classifyNamespaces(inventories, options)
Classifier-->>Status: NamespaceClassification[]
Status->>Projector: projectNamespaceClassifications(adapterId, classifications, config)
Projector-->>Status: AdapterProjectionResult[]
Status-->>Client: StatusResult (routers, classifications, adapterProjection)
Possibly related PRs
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #22 +/- ##
==========================================
+ Coverage 68.76% 69.84% +1.07%
==========================================
Files 111 112 +1
Lines 11844 12520 +676
==========================================
+ Hits 8145 8744 +599
- Misses 3699 3776 +77 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75ce97a640
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
README.md (1)
220-237:⚠️ Potential issue | 🟡 MinorKeep the public capability list exhaustive.
This section says MCP² exposes one router per non-empty capability, but
ai_media_generationis still part of the canonical taxonomy elsewhere in the PR. Leaving it out here makes the list look complete when it isn't.📝 Suggested doc fix
- `design_workspace` +- `ai_media_generation` - `hosting_deploy`🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@README.md` around lines 220 - 237, The public capability list under "Tool API (Capability Routers)" is missing the ai_media_generation capability; update the bullet list (the set that currently includes `code_search`, `docs`, ... `general`) to include `ai_media_generation` so the README matches the canonical taxonomy used elsewhere in the PR and remains exhaustive.src/capabilities/semantic-classifier.ts (1)
178-205:⚠️ Potential issue | 🟠 MajorDon't return boosted ranking scores as
confidence.Line 183 adds the prior directly to cosine similarity, and Lines 199-205 then expose that adjusted score as
confidence. That can push confidence above 1 and makesclassifyBatch()thresholding depend on the heuristic boost rather than the underlying embedding similarity. Keep a separate ranking score here, or at least clamp the exported confidence before returning it.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/capabilities/semantic-classifier.ts` around lines 178 - 205, The code currently adds prior boosts to the cosine similarity and returns that boosted value as confidence; instead compute raw similarity = EmbeddingGenerator.cosineSimilarity(signalResult.embedding, refEmb) separately and use boostedScore = rawSimilarity + (priorBoosts[capId] ?? 0) for ranking only (keep scores array entries with both rawSimilarity and boostedScore), sort by boostedScore, but set the returned confidence to a clamped rawSimilarity (e.g., clamp(rawSimilarity, 0, 1) or map from [-1,1] to [0,1]) for both best and runnerUp so exported confidence reflects the underlying embedding similarity not the heuristic boost; update references around computePriorBoosts, referenceEmbeddings, and the returned object (capability, confidence, runnerUp) accordingly.
🧹 Nitpick comments (1)
tests/capability-inference.test.ts (1)
98-191: Move these cases out of the misclassification-regression block.These assertions now encode the desired canonical outputs, but the enclosing suite/comment still says it tracks the current wrong heuristic results. Keeping them here will make future failures harder to interpret.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/capability-inference.test.ts` around lines 98 - 191, The tests that assert correct canonical outputs (the tests named "Sentry: classified as observability", "Slack: classified as messaging", "Stripe: classified as payments", "Prisma: classified as database", "shadcn: correctly classified as docs (fixed — was design)", and "Supabase: classified as database") should be removed from the misclassification-regression suite and placed into a normal passing-spec suite (or top-level tests) that validates inferNamespaceCapability directly; update or remove the surrounding comment that labels them as tracking the current wrong heuristic. Locate calls to inferNamespaceCapability(...) and the corresponding expect(...).toBe(...) assertions and move those test blocks into a new or existing describe/test group that reflects they are expected-correct canonical behaviors. Ensure the misclassification-regression block only contains tests that intentionally assert the known-broken outputs.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/capabilities/inference.ts`:
- Around line 153-155: The current regex patterns in the observability
capability (the pattern property for capability: "observability") and the other
similar capability entries use raw substring matches that falsely match short
tokens inside unrelated namespaces; update these patterns to use separator-aware
matching (e.g., require word boundaries or common separators like ^, $, ., /, -,
_, or whitespace around short tokens) so tokens like "log", "dm", "ui" only
match as standalone segments rather than inside words; locate the pattern
properties in the observability capability and the other two similar entries and
replace the simple substring alternation with separator-aware variants for each
short token.
- Around line 80-85: The capabilityOverrideSources property in
NamespaceClassificationOptions currently allows any
CapabilityClassificationSource (including "heuristic" and "semantic") but needs
to be restricted to only the override variants so values are assignable to
ClassificationEvidenceSource where they are later reused (see usage around the
evidence array and source propagation). Update the type of
capabilityOverrideSources to only permit override-only variants (e.g., constrain
to Extract<CapabilityClassificationSource, "user_override" |
"computed_override"> or directly to ClassificationEvidenceSource), so the values
stored in capabilityOverrideSources are type-compatible with the evidence
construction code (identify the NamespaceClassificationOptions interface and the
usage that reads capabilityOverrideSources into evidence/source).
In `@src/capabilities/projection.ts`:
- Around line 91-103: The filter currently only excludes null scores, allowing
score === 0 (or negative) to be treated as a match; update the predicate used
when building ranked (and the similar block at 105-125) to treat non-positive
scores as "no match" by filtering entries where score is a number and > 0 (e.g.,
change the type guard from entry.score !== null to entry.score != null &&
entry.score > 0), so that profiles with score <= 0 fall through to
fallbackBucket and matched_profile is not returned with confidence: 0; ensure
you use the existing scoreProfile function and keep the ranking/sort logic
unchanged.
In `@src/status/runner.ts`:
- Around line 58-61: collectStatus() currently always includes the
adapterProjection property (possibly set to undefined), which violates
exactOptionalPropertyTypes for StatusResult.adapterProjection; change
collectStatus() to only include adapterProjection in the returned object when a
real value exists (e.g., use a conditional spread or add the property inside an
if block) so that adapterProjection is omitted entirely when undefined; update
any related return expressions in collectStatus() and ensure the type remains
StatusResult.
In `@tests/capability-classification.test.ts`:
- Around line 112-123: The "general" AdapterCapabilityProfile literal is missing
the required prefersFacets and rejectsFacets arrays; update the object with both
properties (e.g., prefersFacets: [] and rejectsFacets: []) so it conforms to
AdapterCapabilityProfile. Locate the object with id "general" in
tests/capability-classification.test.ts and add appropriate empty or specific
facet arrays for prefersFacets and rejectsFacets to satisfy TypeScript type
checking.
---
Outside diff comments:
In `@README.md`:
- Around line 220-237: The public capability list under "Tool API (Capability
Routers)" is missing the ai_media_generation capability; update the bullet list
(the set that currently includes `code_search`, `docs`, ... `general`) to
include `ai_media_generation` so the README matches the canonical taxonomy used
elsewhere in the PR and remains exhaustive.
In `@src/capabilities/semantic-classifier.ts`:
- Around line 178-205: The code currently adds prior boosts to the cosine
similarity and returns that boosted value as confidence; instead compute raw
similarity = EmbeddingGenerator.cosineSimilarity(signalResult.embedding, refEmb)
separately and use boostedScore = rawSimilarity + (priorBoosts[capId] ?? 0) for
ranking only (keep scores array entries with both rawSimilarity and
boostedScore), sort by boostedScore, but set the returned confidence to a
clamped rawSimilarity (e.g., clamp(rawSimilarity, 0, 1) or map from [-1,1] to
[0,1]) for both best and runnerUp so exported confidence reflects the underlying
embedding similarity not the heuristic boost; update references around
computePriorBoosts, referenceEmbeddings, and the returned object (capability,
confidence, runnerUp) accordingly.
---
Nitpick comments:
In `@tests/capability-inference.test.ts`:
- Around line 98-191: The tests that assert correct canonical outputs (the tests
named "Sentry: classified as observability", "Slack: classified as messaging",
"Stripe: classified as payments", "Prisma: classified as database", "shadcn:
correctly classified as docs (fixed — was design)", and "Supabase: classified as
database") should be removed from the misclassification-regression suite and
placed into a normal passing-spec suite (or top-level tests) that validates
inferNamespaceCapability directly; update or remove the surrounding comment that
labels them as tracking the current wrong heuristic. Locate calls to
inferNamespaceCapability(...) and the corresponding expect(...).toBe(...)
assertions and move those test blocks into a new or existing describe/test group
that reflects they are expected-correct canonical behaviors. Ensure the
misclassification-regression block only contains tests that intentionally assert
the known-broken outputs.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7ebd5d73-2c0e-4dd3-864e-5e76f64d8538
📒 Files selected for processing (21)
CHANGELOG.mdREADME.mddocs/ARCHITECTURE.mddocs/capability-taxonomy-design.mdpackage.jsonsrc/capabilities/inference.tssrc/capabilities/projection.tssrc/capabilities/semantic-classifier.tssrc/config/index.tssrc/config/schema.tssrc/security/policy.tssrc/status/runner.tssrc/utils/capability-meta.tstests/capability-classification.test.tstests/capability-inference.test.tstests/capability-meta.test.tstests/config.test.tstests/figma-classification.test.tstests/fixtures/figma-mcp-tools.jsontests/security-policy.test.tstests/status-runner.test.ts
Co-authored-by: Codex <codex@openai.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/capability-classification.test.ts`:
- Around line 91-128: The adapter config object literal used in the test (the
object with keys mode, fallbackBucket, capabilities) is missing the required
namespaceBucketOverrides property; add a namespaceBucketOverrides field to that
same config literal (matching the shape used in the second test) so the
TypeScript compiler is satisfied—e.g., include an object mapping any relevant
namespaces to bucket names under the key namespaceBucketOverrides alongside
mode, fallbackBucket, and capabilities.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: abbb12c0-0d3d-458e-aee9-84553cfd163b
📒 Files selected for processing (5)
src/capabilities/inference.tssrc/capabilities/projection.tssrc/status/runner.tstests/capability-classification.test.tstests/capability-inference.test.ts
Co-authored-by: Codex <codex@openai.com>
There was a problem hiding this comment.
🧹 Nitpick comments (2)
tests/capability-classification.test.ts (2)
188-204: Decouple the semantic mock from reference-text copy.
embedBatch()is keying off literal snippets like"structured design workspace files"and"visual design artifacts". That makes this test fail on copy-only edits to the reference texts, even if the classifier behavior is still correct. Prefer mapping from stable capability identifiers/constants instead of hard-coded substrings.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/capability-classification.test.ts` around lines 188 - 204, The mock embedBatch in fakeGenerator currently matches on hard-coded substrings ("structured design workspace files", "visual design artifacts") making tests fragile; change it to map inputs to vectors using stable capability identifiers/constants (e.g., CAP_DESIGN_WORKSPACE, CAP_VISUAL_DESIGN) or an explicit text->vector map keyed by exact capability tokens rather than substrings, update the test reference texts to use those constants/tokens, and have embedBatch return designWorkspaceVector, designVector or otherVector based on that identifier lookup so copy-only edits to descriptive text won't break the test.
84-166: Add a direct test fornamespaceBucketOverridesprecedence.Both projection cases pass
namespaceBucketOverrides: {}, so the highest-precedence branch insrc/capabilities/projection.tsnever runs here. That leaves the new override behavior unprotected even though it is part of this PR.🧪 Suggested coverage
describe("adapter projection", () => { + test("prefers namespace bucket overrides over profile scoring", () => { + const classification = classifyNamespace("pencil", PENCIL_TOOLS); + + const projection = projectNamespaceClassification("gateway", classification, { + mode: "projected", + fallbackBucket: "general", + capabilities: [ + { + id: "design", + title: "Design Analysis", + summary: "Analyze screenshots, diagrams, and visual diffs.", + acceptsCanonical: ["design_workspace"], + prefersFacets: [], + rejectsFacets: [], + }, + ], + namespaceBucketOverrides: { pencil: "general" }, + }); + + expect(projection.bucket).toBe("general"); + expect(projection.source).toBe("adapter_override"); + }); + test("can project a canonical design_workspace classification away from screenshot-analysis design buckets", () => {As per coding guidelines, "For new features: Write a failing test first, then implement until it passes."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/capability-classification.test.ts` around lines 84 - 166, The new behavior for namespaceBucketOverrides isn't covered by tests; add a test that calls projectNamespaceClassification with a non-empty namespaceBucketOverrides mapping (e.g., {"pencil": "design"}) and a capabilities list, then assert the projection selects the override bucket (expect(projection.bucket).toBe("design")) and that the projection.source indicates the override branch (e.g., "namespace_override" or whatever the code sets) to ensure the highest-precedence override path in projectNamespaceClassification is exercised.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@tests/capability-classification.test.ts`:
- Around line 188-204: The mock embedBatch in fakeGenerator currently matches on
hard-coded substrings ("structured design workspace files", "visual design
artifacts") making tests fragile; change it to map inputs to vectors using
stable capability identifiers/constants (e.g., CAP_DESIGN_WORKSPACE,
CAP_VISUAL_DESIGN) or an explicit text->vector map keyed by exact capability
tokens rather than substrings, update the test reference texts to use those
constants/tokens, and have embedBatch return designWorkspaceVector, designVector
or otherVector based on that identifier lookup so copy-only edits to descriptive
text won't break the test.
- Around line 84-166: The new behavior for namespaceBucketOverrides isn't
covered by tests; add a test that calls projectNamespaceClassification with a
non-empty namespaceBucketOverrides mapping (e.g., {"pencil": "design"}) and a
capabilities list, then assert the projection selects the override bucket
(expect(projection.bucket).toBe("design")) and that the projection.source
indicates the override branch (e.g., "namespace_override" or whatever the code
sets) to ensure the highest-precedence override path in
projectNamespaceClassification is exercised.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a663bb5c-66da-497a-8696-3b5685c9f720
📒 Files selected for processing (1)
tests/capability-classification.test.ts
Co-authored-by: Codex <codex@openai.com>
Owner
Author
|
Addressed the latest CodeRabbit nitpicks in fab9590.
Validated locally with bun test, bun run build, bun run lint, bun test --coverage, bun run coverage:check, and bun run eval:routing --strict. |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
Co-authored-by: Codex codex@openai.com
Summary by CodeRabbit
New Features
Documentation
Tests