feat: add embedded adapters (granite switch) to openai backend

[jakelorocco]

Misc PR

Type of PR

• Bug Fix
• New Feature
• Documentation
• Other

Description

• Link to Issue: Fixes N/A

Adds support for granite switch models for OpenAIBackends.

Changes:

• OpenAIBackends can utilize EmbeddedIntrinsicAdapters with any of the regular intrinsic functionality.
• Adapters are loaded by default when load_embedded_adapters=True at init.
• Added a placeholder IBM_GRANITE_SWITCH_4_1_8B that points to the temp repo for now.
• Added documentation, examples, tests.
• When generating from an intrinsic, we now only grab temperature and seed from model options. This is also correctly noted.
• call_intrinsic function now works with both OpenAIBackends and LocalHFBackends

Please see most recent comment below for additional context: #881 (comment)

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code as added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

• AI coding assistants used

[github-actions]

The PR description has been updated. Please fill out the template for your PR to be reviewed.

[jakelorocco]

I'm still in the process of testing some of these changes from an e2e perspective. I also still need to add documentation for how to use embedded models along with examples.

These changes do seem to work though and allow utilizing intrinsics with an embedded adapter / granite switch model.

[lastras]

Bug: model_options temperature is silently dropped for intrinsic calls

The seed parameter is correctly forwarded to api_params at line 639, but temperature (and any other model_options entries) are not. They are written into request_json at line 617, which gets absorbed into the ChatCompletion object by rewriter.transform(), but never extracted back out when building api_params. The result is that callers passing model_options={ModelOption.TEMPERATURE: 0.0} get the model's default temperature instead — causing non-deterministic outputs even when greedy decoding is explicitly requested.

Fix: mirror the seed pattern immediately after line 640:

if ModelOption.TEMPERATURE in model_options:
    api_params["temperature"] = model_options[ModelOption.TEMPERATURE]

Verified: with this fix, 5 back-to-back pipeline runs (guardian → rewrite → answerability → QC → base model, no sleep between calls) produce bit-for-bit identical outputs including long-form base model answers. Without the fix, outputs vary run-to-run despite temperature=0.0 being passed by the caller. The warning at line 557 ("some model options may be overwritten / ignored") is technically accurate but understated — temperature is always silently dropped, not just sometimes.

[nrfulton]

@lastras Your comment states that all model options get dropped but that the fix is to add (another) special case for temperature in model_options.

It sounds from your comment like there's an underlying bug here that should be fixed. Can you clarify?

[jakelorocco]

@lastras @nrfulton, I am going to move this convo to slack for faster resolution.

[jakelorocco]

I fixed a few random issues and also the issues we talked about (call_intrinsic and seed/temperature setting). I'm working on adding some documentation and minor examples.

The functional parts of the code should be stable now. I have additional things that I'd like to clean up, but I will delay them until after we make sure everything is working as intended and post-release, so I don't impact any functionality.

[jakelorocco]

Updated the PR description. Fixed a few additional errors / bugs. Added documentation and examples. We should be good to start reviewing. I need to run one more round of tests with the latest from the granite-switch team.

I have plans to open a few additional issues to clean things up after this PR is merged.

Additional changes that should be made before the official release:

• Update IBM_GRANITE_SWITCH_4_1_8B to point to the correct model
• Update our documentation to point to the official granite switch documentation when published

[ajbozarth]

Good overall direction — the EmbeddedIntrinsicAdapter class is well-structured and the design of delegating I/O config loading to the existing io.yaml / IntrinsicsRewriter machinery makes sense. A few issues need addressing before this merges, ranging from a runtime bug to a dependency footprint concern.

[ajbozarth]

huggingface-hub is now a hard dependency for all Mellea users, including those who only use the Ollama or non-switch OpenAI backends. It's also duplicated — it already appears in the hf extra at line 51.

AGENTS.md §5 calls for optional backend imports to be wrapped in try/except ImportError with a helpful message. The HF download only happens when using a Granite Switch model, so this should either be gated behind an optional extra (e.g., openai or a new switch extra) with a try/except ImportError wrapping calls in EmbeddedIntrinsicAdapter.from_hub, or at minimum the duplicate in the hf extra should be removed.

[ajbozarth]

IBM_GRANITE_SWITCH_4_1_8B points to a personal repo (GrizleeBer/gs-test-2). Any user who imports this constant today — or who passes it as a model ID with load_embedded_adapters=True — will silently hit that personal repo.

Please block merge on updating this to the real repo ID, or make it raise explicitly when accessed so it can't accidentally ship in a release.

[ajbozarth]

_uses_embedded_adapters is unconditionally True for every OpenAIBackend, regardless of whether the served model is actually a Switch model. This means calling any intrinsic on a non-Switch OpenAI backend will attempt a HuggingFace download and fail with a confusing FileNotFoundError (no adapter_index.json) rather than a clear "this model does not support embedded intrinsics" error.

Suggest tying this flag to whether load_embedded_adapters=True was passed (or whether register_embedded_adapter_model has been called), and raising a clear error in _generate_from_intrinsic when no adapters are registered.

[ajbozarth]

This guard raises NotImplementedError whenever ModelOption.STREAM appears in model_options at all — including STREAM=False. Compare the analogous check at line 877 which correctly uses:

if model_opts.get(ModelOption.STREAM, False):

Fix:

if model_options.get(ModelOption.STREAM, False):
    raise NotImplementedError("Intrinsics do not support streaming.")

[ajbozarth]

The log message says "only temperature and seed are kept" but SYSTEM_PROMPT is also extracted and used a few lines below:

system_prompt = model_options.get(ModelOption.SYSTEM_PROMPT, "")

Either update the message to accurately list all supported options, or narrow the log condition to only fire when unrecognized options are present.

[ajbozarth]

assert is disabled when Python runs with -O (optimized mode), making this a silent no-op in production environments. Replace with an explicit guard:

if not isinstance(adapter, EmbeddedIntrinsicAdapter):
    raise TypeError(
        f"OpenAIBackend only supports EmbeddedIntrinsicAdapter, got: {type(adapter).__name__}"
    )

[ajbozarth]

Accessing private attributes (_uses_embedded_adapters, _model_id) of a concrete class via getattr from outside that class couples _util.py to OpenAIBackend internals without going through the AdapterMixin interface.

Suggest adding a method to AdapterMixin (e.g., uses_embedded_adapters() -> bool) so this is an explicit part of the interface contract rather than a duck-typed private attribute check.

[ajbozarth]

self.intrinsic_name is set here, but super().__init__(intrinsic_name, adapter_type) already sets self.name = intrinsic_name (line 49 of the base class). The test even asserts adapter.intrinsic_name == adapter.name.

Pick one: use self.name (the inherited attribute) or rename self.name to self.intrinsic_name in the base class. Having both on EmbeddedIntrinsicAdapter creates ambiguity about which is canonical.

[ajbozarth]

raise ... from None suppresses the original exception chain, which includes the local model path that was searched — useful information for debugging. Use raise ... from e (or just raise) to preserve the context.

[ajbozarth]

The entire E2E test file is unconditionally skipped. The unit tests in test_embedded_adapter.py cover adapter loading well, but the ~200-line _generate_from_intrinsic code path in OpenAIBackend has no test coverage at all.

At minimum, add unit tests that mock the OpenAI client and verify the generate path end-to-end (chat_template_kwargs is set, result_processor is applied, temperature/seed are forwarded). These can live alongside the existing unit tests rather than requiring a real vLLM server.

[ajbozarth]

This is pretty cool, I had Claude explain it to me along with its review. Afaik the review comments look important to address, but I didn't block on them just incase some are intentional. I also ran uv run pytest locally and it failed on the same test as CI

[ajbozarth]

TestFromHub.test_missing_huggingface_hub_raises fails (both in CI and locally) because of this top-level import.

The test patches sys.modules["huggingface_hub"] to None to simulate the library being absent, but since the name is already bound here at module load time, the patch has no effect — snapshot_download runs for real, hits HuggingFace with some/repo, gets a 401, and raises RepositoryNotFoundError instead of the expected ImportError.

Two consistent fixes:

Option A — keep huggingface_hub optional (recommended): Remove this top-level import and move it inside from_hub:

@staticmethod
def from_hub(repo_id, ...):
    try:
        import huggingface_hub
    except ImportError:
        raise ImportError(
            "huggingface_hub is required to load embedded adapters from the Hub. "
            "Install it with: pip install huggingface-hub"
        ) from None
    local_root = huggingface_hub.snapshot_download(...)

The test then passes as-is, and huggingface_hub can be removed from base dependencies.

Option B — accept it as a hard dependency: Remove test_missing_huggingface_hub_raises (the scenario it tests can never happen with a top-level import) and update the from_hub docstring to drop the ImportError entry from Raises:.