Skip to content

fix: honor base64 encoding for embedding responses - #741

Draft
emecii wants to merge 1 commit into
Lightning-AI:mainfrom
emecii:fix/embedding-base64-response
Draft

fix: honor base64 encoding for embedding responses#741
emecii wants to merge 1 commit into
Lightning-AI:mainfrom
emecii:fix/embedding-base64-response

Conversation

@emecii

@emecii emecii commented Sep 6, 2026

Copy link
Copy Markdown

What does this PR do?

Addresses #740. Draft for scope review; maintainer agreement is pending.

When a client sends encoding_format="base64" to /v1/embeddings, LitServe currently returns a JSON float array. This returns base64-encoded float32 vectors as requested, while retaining float arrays for omitted or explicit float encoding. Response indices, model and usage metadata are preserved.

The request option was introduced in #388; the response encoder was missing. Conversion runs after existing vector normalization and validation, using the Python standard library without adding runtime dependencies.

Regression tests run real CPU inference workers with multiprocessing and ZeroMQ transports. They check single and multiple text inputs, encoded byte length and decoded values, default/float compatibility, response metadata, and concurrent mixed-encoding requests on a batching-enabled server.

Validation:

  • Before: 6 base64 cases failed; 8 default/float controls passed.
  • After: 25 focused tests passed; broader related suite: 98 passed.
  • Every added executable line covered using --cov=src/litserve.
  • Applicable pre-commit hooks, wheel/sdist build and strict metadata checks passed. Whole-config pre-commit was blocked installing its non-applicable Prettier hook; applicable hooks were run individually.
  • Full cross-platform/GPU CI has not been run locally.

Before submitting

No new configuration or documentation API is introduced; this makes an already accepted request field take effect.

AI assistance was used to investigate, implement and test this change. Human review is pending. No competing PR code was copied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant