feat(agents): multimodal-RAG — attach page-image hits as inline image_urls + VISION_MODEL_ALIASES - #12789
Open
gloeckle-direct-ki wants to merge 4 commits into
Open
Conversation
Upstream v0.8.2 only routes file uploads through rag-api when the user
explicitly chose an Agent with the file_search tool. Normal chat-input
paperclip uploads of PDFs, DOCX, etc. never get indexed — even when
RAG_API_URL is configured — so chat-time retrieval (createContextHandlers)
has nothing to query against.
This patch adds a minimal auto-embed pass in processAgentFileUpload's
"standard single storage" branch: for chat-input paperclip uploads
(message_file === true, no tool_resource) with a text-bearing MIME type,
fire-and-forget uploadVectors to rag-api after the primary storage write.
On success, the File doc is flagged embedded:true so the existing
createContextHandlers pipeline picks it up at chat time. On any rag-api
failure, the upload still succeeds with embedded:false (graceful
degradation — user sees the inline attachment).
- New: api/server/services/Files/VectorDB/autoEmbed.js
- isTextBearingMimeType(mimetype)
- tryEmbedChatAttachment({req, file, file_id, entity_id}) — never throws
- New: api/server/services/Files/VectorDB/autoEmbed.test.js
- MIME classification, success + error + unknown-type paths
- Modified: api/server/services/Files/process.js
- Standard-storage branch now invokes tryEmbedChatAttachment for
messageAttachment && !tool_resource && text-bearing && RAG_API_URL
Images (→ Vision) and audio (→ STT) are intentionally skipped.
No schema change: fileSchema.embedded already exists upstream.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dels
Extends createContextHandlers so that when the rag-api fork returns
visual matches (multimodal-ingest branch), we:
1. Pass `include_visual: true` on the /query body, controlled by
RAG_INCLUDE_VISUAL (default on).
2. Accept both the legacy flat-list /query response and the new
`{chunks, visual_matches}` shape, so mixed-version deploys don't
break.
3. For vision-capable models (validateVisionModel), base64-encode each
matching page PNG and surface it as an image_url the caller can
attach to the latest user message.
4. For text-only models, append a short German hint to the RAG
context so the LLM doesn't claim to have seen the layout.
The AgentClient's non-agent RAG branch merges getVisualImageURLs() onto
`latestMessage.image_urls` right after createContext(). This keeps the
attachment flow provider-agnostic — Claude / OpenAI adapters handle the
final shape transformation downstream, same as for paperclip-uploaded
images.
Defense-in-depth: rejects any image_path that isn't absolute so a
compromised sidecar cannot nudge us into reading arbitrary files.
Tests: 7 new jest cases (no RAG_API_URL guard, include_visual
propagation, vision vs text-only branching, base64 encoding, path
traversal rejection, legacy response compat, feature flag off).
The earlier patch attached page PNGs to latestMessage.image_urls, but formattedMessages — built via orderedMessages.map(formatMessage) earlier in the same buildMessages call — holds independent objects, not refs. The provider payload is constructed from formattedMessages downstream, so the mutation on latestMessage never reached the LLM. Vision-capable models received text context only and (correctly) reported they had no images to analyse. Mirror the image_urls assignment onto the matching formattedMessages entry. End-to-end verified on prod with a Claude Opus chat: page PNGs now arrive at the Anthropic API and the model can analyse them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gloeckle-direct-ki
marked this pull request as ready for review
April 23, 2026 07:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
rag_apican return visual matches alongside text chunks (see danny-avila/rag_api#283) but LibreChat has no client-side integration to actually attach them to the LLM call. Vision-capable models therefore receive text context only and (correctly) report they cannot see images, even when the user uploaded a design-heavy PDF and asked about layout / colors / photos.Solution
Three coordinated additions — none of them touch existing code paths unless the new feature is configured:
1.
createContextHandlersexposesgetVisualImageURLs()When
RAG_INCLUDE_VISUAL=true(default on ifRAG_API_URLis set), the existingquery()request gainsinclude_visual: true. Visual matches in the response are collected. A new exposed method loads each page PNG from disk, base64-encodes it, returnsimage_urlparts in the shape downstream provider adapters expect.For text-only models, a German-language hint is injected into the system prompt (
\"[Hinweis: visuelle Treffer auf Seiten X verfügbar, aber das aktuell gewählte Modell kann keine Bilder auswerten]\") so the LLM knows visual context exists but is currently inaccessible.2.
agents/client.jsconsumesgetVisualImageURLs()After
createContext(), ifgetVisualImageURLsis defined: load the URLs and attach tolatestMessage.image_urlsAND to the correspondingformattedMessages[-1].image_urls. The latter is essential —formattedMessagesis built earlier viaorderedMessages.map(formatMessage)and holds independent objects, so mutatinglatestMessagealone does not propagate to the provider payload. (We hit exactly this in production:image_urlswere loaded into memory but never reached the LLM.)3.
VISION_MODEL_ALIASESenv varvalidateVisionModel({ model })upstream uses substring matches against a static list of canonical names (claude-opus-4,gpt-4o, etc.). Any deployment with custom display names — common with proxies like LiteLLM where users see "Claude Opus - Premium" instead ofclaude-opus-4— falls back to text-only treatment. We allow operators to extend the recognized list via env (CSV of substrings to match in addition to the upstream defaults).Backward compatibility
RAG_API_URLandRAG_INCLUDE_VISUAL: identical to today./queryresponse shape (flat list of chunks) is detected and used as-is — only{chunks, visual_matches}shapes trigger the new code path.getVisualImageURLsis a typeof-check away from being a no-op if missing.Soft-fail policy
image_path.Tests
7 jest cases covering: vision-model recognition with aliases, image_url shape correctness, soft-fail on missing files, path-traversal rejection, no-attachment-for-text-only-models, hint injection.
Stacking
rag_apito begin with.Why upstream
This is the missing client integration that turns
rag_api's visual capabilities into actual answers from vision-capable models. TheformattedMessagesmirror fix is a real bug we hit in production — vision models silently received no images for two days while we debugged. Worth landing standalone even if reviewers prefer to defer theVISION_MODEL_ALIASESenv knob.Happy to split into smaller PRs if that helps the review.
Co-authored-by: Claude (Anthropic)