fix(provider): stop treating unknown modalities as unsupported for BYOK models - #2056
fix(provider): stop treating unknown modalities as unsupported for BYOK models#2056wqymi wants to merge 4 commits into
Conversation
…OK models
A model declared under `provider.<id>.models.<id>` without `modalities`
had its non-text input capabilities resolved by a two-step `??` chain that
ended in a hardcoded `false`. The second step consulted models.dev, but
only within the SAME provider id, so a user-invented provider never
matched anything: a model whose catalog entry plainly lists image input
still came out `image: false`, and `provider/transform.ts` then replaced
the user's image with an error sentence addressed to the model. The model
correctly reported it had received no image while the image sat in the
session record, which reads exactly like a regression in the client.
Resolution is now four ordered tiers (provider/modality-inference.ts):
1. declared — `modalities` in config, always authoritative
2. provider-entry — the same-named provider's existing catalog entry
3. directory — NEW: bare model id looked up across the whole
models.dev directory, taking the first-party entry
4. assumed — nothing is known, and that is recorded as unknown
Tier 3 derives first-partyness from the directory's own shape rather than
a hand-maintained vendor table: aggregators list resold models under the
vendor namespace (`xiaomi/mimo-v2.5`), the vendor lists it bare, so the
namespaces that providers agree on, intersected with the providers holding
the bare id, name the vendor entry. Exactly one survivor is required — no
union of aggregator claims and no majority vote, so a contradictory or
absent directory falls through to tier 4 instead of guessing.
Tier 4 resolves permissively. The two ways of being wrong do not cost the
same: guessing "supported" wrongly produces an explicit upstream 4xx that
names the rejected media, while guessing "unsupported" wrongly drops the
content before the request exists and leaves nothing anywhere saying the
engine chose to. Its scope is only config-declared models no catalog entry
covers; catalog-sourced models still answer strictly from metadata.
Inferred verdicts are marked (`capabilities.inferred`) with the tier and
the exact models.dev ref they came from, so a wrong verdict is
attributable to the inheritance instead of being anonymous. Autonomous
capability-based selection (`getVisionModel`) additionally refuses
`assumed` evidence via `hasEvidencedImageInput` — the permissive default
exists to honour content the user explicitly attached, not to let the
engine pick a text-only model to look at a screenshot.
`ProviderTransform.unsupportedParts` replaces a file part the model is recorded as unable to read with "ERROR: Cannot read … Inform the user." That sentence is addressed to the MODEL and is written from inside an AI SDK middleware: nothing about the substitution reaches the transcript. The observable result is a model insisting it received no image while the image is right there in the session record — which reads like the consuming client mangled the attachment, not like the engine deciding to withhold it. A component that decides to do less has to be able to say so. The decision is now announced on the user's own channel, using the mechanism this repo already has for exactly this (`MEMORY_WRITE_OFF_FALLBACK_NOTICE`): a synthetic text part with `ignored: true`, so it stays out of the model's context, and `time.end` set, so the CLI emits it. Single-language English, matching that precedent — the engine cannot know the reader's locale and the consuming client already carries its own translations. The notice names the file, the modality, and the model, and when the verdict was INFERRED rather than declared it says so and names the entry it was inherited from. That last part is the difference between "your config says this model is text-only" and "we guessed, and the guess may be wrong". The verdict itself is factored into `ProviderTransform.withheldModality` and `unsupportedParts` now calls it, so the announcement and the substitution cannot drift apart.
mimo-auto is a free-tier routing alias: no catalog publishes it, and it dispatches to a vision-capable model. Its image support was applied by assigning `capabilities.input.image = true` onto the finished model, after modality resolution had already recorded the verdict's provenance as `assumed` — nothing was known about this id. Both halves were then true at once: the model claimed image input, and the claim was labelled a guess. `hasEvidencedImageInput` exists precisely to refuse guesses when the ENGINE picks a model on the user's behalf, so the alias was filtered out of `getVisionModel`. On a free-tier configuration mimo-auto is the only vision channel there is, so a user on a text-only model who read an image got told, by the read tool, that no vision-capable model was configured — the same class of wrong advice this branch set out to remove. Fixed at the source rather than by widening the predicate: resolution grows a BUILTIN tier, ranked directly below the user's own `modalities` and above every catalog lookup, holding the modalities this repo itself vouches for its own aliases. A stated fact now enters the pipeline as part of the verdict, and post-hoc mutation of a constructed model — the actual cause — is gone. The entry is the complete input map (`text`, `image`), which is the shape mimo-auto has always had; the permissive `assumed` tier is for ids nobody looked up, which is the opposite of an alias this repo defines. `hasEvidencedImageInput` needed no change: it tests against `assumed` specifically rather than allow-listing good provenances, so a new source of knowledge counts as evidence by default. Callers that report a verdict as inferred or possibly wrong now ask `ModalityInference.isStated` instead of comparing against `declared`, so a stated fact is not described to the model as a guess. Also names the permissive counterpart `acceptsImageInput`, so the two image questions — may an image be handed to the model the user chose, and may the engine pick a model to look at one — stop being told apart by whichever literal a call site happened to write. Tests pin the cause as an absence: mimo-auto's verdict must never be recorded as `assumed`, since that is what removes it from selection however the selection is later written. Also pinned: an assumed verdict is honoured for the user's own model yet stays unselectable, and a user's own declaration still overrides the builtin entry.
Three call sites answered the question three ways. `getVisionModel` required evidence; the vision list in the system prompt and `actor models --vision` both read `capabilities.input.image` raw. So a model whose image support is merely assumed — a BYOK id no catalog covers — was advertised to the model as a `--model` target it could dispatch a subagent to, while the engine's own selection would never have chosen it. Advertising and picking are the same decision: a ref the engine hands over is one the model will dispatch to. Both lists now filter on `hasEvidencedImageInput`. That disagreement was also masking the mimo-auto exclusion fixed in the previous commit. The system prompt fell back to `visionModels[0]` from its own permissive list, which still contained the alias that selection had dropped, so the block kept naming a usable model; only the read tool, which has no such fallback, showed the failure. Both halves are needed — one predicate, and a correct verdict feeding it. The remaining sites ask the opposite question: whether an image may be handed to the model the user already chose. Those keep the permissive answer and now say so by name, `acceptsImageInput`. In session/system.ts the two questions sit one line apart, which is where the distinction most needed to stop being invisible. Per-modality mime tables that read `input.image` alongside `input.pdf` and `input.audio` are left alone: they are uniform lookups, not this policy choice. The fake provider's `getVisionModel` answered from the raw capability too, so a test could have passed on a model production would skip. It now uses the real predicate. The new test pins the negative: a model whose image support is only assumed must never appear in the refs offered to the model, and the free-tier alias must not be missing from them.
|
Both findings fixed — head Finding 1 — fixed, but not at the site you suggestedYou proposed marking the verdict as evidence inside the So const BUILTIN_INPUT: Record<string, readonly InputModality[]> = {
"mimo/mimo-auto": ["text", "image"],
}New provenance tier Not reusing
Two follow-on sites that would otherwise have mis-reported a builtin fact as inferred (
Finding 2 — agreed it matters, and it was worse than reported: three copies, not two
I did both unify and split, because there really are two questions — but the boundary is not where the review put it:
The reasoning on Left alone deliberately: Also fixed: Your masking observation, reproduced liveThe E2E now drives the real read tool (fake upstream answers request 1 with a real Read tool output as the engine put it on the wire — fixed: Same harness, same config, src reverted to Sentinels, scoped to the tool message rather than the whole request body: That middle line is your Finding-2 point observed directly: pre-fix Tests, pinned as absences
Discriminating power checked rather than assumed — the pinned catalog has 182 providers, no
Still not verified
|
1 participant
Problem
A model declared under
provider.<id>.models.<id>withoutmodalitieshad its non-text input capabilities resolved by a two-step??chain ending in a hardcodedfalse:The middle step consults models.dev, but only within the same provider id. A user-invented provider id matches nothing there, so a model whose catalog entry plainly lists
["text","image","audio","video"]still resolved toimage: false.provider/transform.tsthen replaced the user's image part withERROR: Cannot read "<file>" (this model does not support image input).— text addressed to the model, written from inside an AI SDK middleware.The observable result: the model truthfully reports it received no image, the image is sitting in the session record the whole time (a single
filepart's data URL measured 456,955 chars in the wild), and nothing anywhere says the engine chose to drop it. It reads like the consuming client corrupted the attachment.?? falseturned "we never looked this up" into "we know it cannot do this".Resolution order
modalitiesin config. Always authoritative.false.Deriving first-partyness without a vendor table
models.dev publishes no "first party" flag, so tier 3 reads the directory's own shape: aggregators list a resold model under the vendor namespace (
xiaomi/mimo-v2.5,openai/gpt-5), while the vendor lists it under the bare id. Intersecting the namespaces providers agree on with the providers holding the bare id names the vendor's entry.Exactly one survivor is required — no union of aggregator claims, no majority vote. A namespace no provider serves and an aggregator namespacing under its own id both drop out for free. Against the real directory:
mimo-v2.5xiaomitext,image,audio,videomimo-v2.5-proxiaomitextgpt-5openaitext,imageclaude-sonnet-4-5anthropictext,image,pdfgemini-2.5-progoogletext,image,audio,video,pdfdeepseek-v3mimo-1000Note
mimo-v2.5is multimodal whilemimo-v2.5-prois text-only — which is exactly why there is no family-name fallback here.Unknown resolves permissively
The two ways of being wrong do not cost the same:
modalities.Scope is narrow: config-declared models that no catalog entry covers. Catalog-sourced models (
fromModelsDevModel) still answer strictly from metadata. Output modalities get tiers 1–3 but not tier 4's permissiveness — nothing decides whether to send based on them, so there is no silent-drop risk to correct for.Inference is marked and attributable
capabilities.inferred(new, optional) records the tier and the exact models.dev ref a verdict was inherited from, so a wrong verdict is attributable to the inheritance instead of anonymous. It is absent when the user declared the modalities.Autonomous selection does not trust
assumedgetVisionModel()ranks by cheapest, and a BYOK model defaults tocost.input = 0— so a blanket permissive default would have made a possibly-text-only BYOK model the systematic first pick for "the vision model". That would trade this bug's silent drop for a new silent failure.getVisionModelnow filters throughhasEvidencedImageInput, which rejectsassumed.The asymmetry: honouring what the user explicitly attached is permissive; the engine picking a model on the user's behalf requires evidence.
Second commit: the withhold decision is now reportable
unsupportedPartsstill substitutes when a modality genuinely is unsupported — but the decision is no longer invisible. It is announced on the user's own channel using the mechanism already in the repo for this (MEMORY_WRITE_OFF_FALLBACK_NOTICE): a synthetic text part withignored: trueso it stays out of the model's context, andtime.endset so the CLI emits it. Single-language English, matching that precedent — the engine cannot know the reader's locale and the consuming client already carries its own translations.The notice names the file, the modality, the model, and — when the verdict was inferred — the tier and the entry it came from:
The verdict is factored into
ProviderTransform.withheldModality, whichunsupportedPartsnow calls, so the announcement and the substitution cannot drift apart.Verification
End-to-end against a fake OpenAI-compatible upstream that records request bodies verbatim, with a custom provider id absent from models.dev, a pinned
models.json, and a real 64×64 PNG. Baseline is the same tree atHEAD~1, materialised withgit archive+ symlinkednode_modules.acme-byok/mimo-v2.5, nomodalitiesimage_url=0; "no vision support"image_url=1, 12,570-char data URLmimo-v2.5-pro(catalog: text-only)image_url=0image_url=0— correctly withheldacme-private-9000(absent)image_url=0image_url=1mimo-v2.5+modalities.input:["text"]image_url=0image_url=0— config wins over catalogPOST /session/:id/messageignored: truenoticeThe baseline run's own advice is worth quoting, because it is the bug in one line — it suggested dispatching a subagent with
--model xiaomi/mimo-v2.5, i.e. the directory knew perfectly well that this exact model has vision, while the instance under a custom provider id was marked blind.Every inference emits a log line carrying its provenance:
bun test test/provider474 pass / 0 fail.bun test test/session910 pass / 0 fail, identical counts to the baseline tree. Repo-wideturbo typecheckclean.capabilities.inferredis optional, so the change is schema-compatible.Not verified
hasEvidencedImageInputinsidegetVisionModelis not covered end-to-end — it typechecks and the predicate is trivial, but no run exercised anassumedmodel being excluded from vision auto-selection.imagewas exercised. The code path is modality-generic, butaudio/video/pdftier-4 permissiveness was not observed.fileparts on the user message. Theread-tool path pre-checks and emits its own user-visible text, so it never reaches the transform. Reachability was verified through the server route; other client entry points were not audited.inferred.outputis written but nothing reads it — recorded for attribution only.prettier --writewas never run (known hazard on this repo).provider.ts/prompt.ts/transform.tsremain prettier-dirty in exactly the hunks that were already dirty at HEAD;modality-inference.tsis clean.test/sessionflakiness is asserted, not root-caused. 9 failures appeared on one run and did not reproduce on rerun or on the baseline tree.