Conversation
…lmbr#438) The self-hosted LLM sections told operators to configure endpoints with a top-level `providers` block, and the air-gapped section added a `default_provider` key. Nothing reads either one. An admin following the guide wrote a config.json that was silently ignored and NBI ran on defaults, so the symptom (no models, or requests going to the wrong upstream) surfaced a long way from the cause. Verified before rewriting: `default_provider`, `default_chat_model` and `default_inline_completion_model` appear only in this file, with no references in notebook_intelligence/ or src/; no code reads a key named `providers`; and NBIConfig's accessors cover chat_model, inline_completion_model, claude_settings and friends with nothing of that shape. The four examples (Azure, vLLM/TGI, LiteLLM, Ollama) now use the real surface. Endpoint settings are per-provider LLMProviderProperty values persisted inside the model selection as {id, value} pairs, which ai_service_manager applies to the selected model at startup. The old text was wrong three ways at once: the outer key, the inner keys, and the `model` field, which for these two providers is a fixed placeholder id because the upstream model name belongs in the `model_id` property. Ollama needed different treatment rather than a corrected JSON block. That provider declares no properties at all, so there is no `base_url` to set and writing one has no effect; a remote host comes from OLLAMA_HOST in the server's environment, which the client reads directly. Its autocomplete ids also come from a fixed list in the provider, where the old example's `codellama:7b` should have been `codellama:7b-code`. Review remediation, from a technical-accuracy pass over every claim and an admin-experience pass over the instructions: - The examples contradicted the API-key handling section two above them, which promises env-var fallback. That promise does not hold for openai-compatible: the provider forwards an empty string and the OpenAI SDK only consults the environment when no key was supplied, so the fallback is skipped. The list now says so and cross-references that section; the code fix that would make the promise true is plmbr#439. - Nothing told operators how to tell whether the file took effect, which is the same silent-failure class this change exists to fix. Added a pointer to the readiness endpoint and the Status card, which already report exactly that. - Called out that the dropdown shows the placeholder model id, so an admin sanity-checking their work does not read it as a config that was ignored. - Called out `context_window`: when unset, NBI skips chat-history pruning entirely rather than assuming a default, so a small self-hosted window overflows. - Promoted the OLLAMA_HOST note to a blockquote callout, matching how the guide flags other easy-to-miss requirements, and made the placeholder secrets obviously fake rather than plausible key prefixes.
|
The This PR only edits markdown, so it cannot affect dependency resolution. The failure comes from the release build: The same job on #444 failed in the same window with A durable fix would be moving the devDependency from |
The previous check_release run failed on a transient network timeout during the frontend build, unrelated to this change.
Summary
The admin guide's self-hosted LLM sections told operators to configure endpoints with a top-level
providersblock, and the air-gapped section added adefault_providerkey. Nothing reads either one. An admin who followed the guide wrote aconfig.jsonthat was silently ignored, NBI ran on defaults, and the symptom (no models, or requests going to the wrong upstream) surfaced a long way from the cause.Verified before rewriting anything:
default_provider,default_chat_modelanddefault_inline_completion_modelappear only indocs/admin-guide.md, with zero references innotebook_intelligence/orsrc/.providers. The 42 substring matches are all other identifiers (llm_providers,disabled_providers,allowed_context_providers,allow_enabling_providers_with_env, and so on).NBIConfig's accessors coverchat_model,inline_completion_model,claude_settings,acp_settings,mcp_server_settingsand friends. Nothing of the documented shape exists.Solution
The four examples (Azure, vLLM/TGI, LiteLLM proxy, Ollama) now use the real surface. Endpoint settings are per-provider
LLMProviderPropertyvalues persisted inside the model selection as{id, value}pairs, whichai_service_manager.pyapplies to the selected model at startup.The old text was wrong three ways simultaneously: the outer key, the inner keys, and the
modelfield, which for these two providers is a fixed placeholder id (openai-compatible-chat-model) because the real upstream model name belongs in themodel_idproperty.Ollama needed different treatment rather than a corrected JSON block. That provider declares no properties at all, so there is no
base_urlto set and writing one has no effect; a remote host comes fromOLLAMA_HOSTin the server's environment, which the client reads directly (_parse_host(host or os.getenv('OLLAMA_HOST'))). Its autocomplete ids also come from a fixed list in the provider, where the old example'scodellama:7bshould have beencodellama:7b-code.Testing
prettierclean on all tracked files including this one,tsc --noEmitclean, jest 423 passed, pytest 1753 passed. Documentation-only; no code changed.Both intra-document anchors were checked against the real headings (
## API-key handlingat :167,## Configuration readinessat :792) rather than inferred from section names.Review
Two reviewers plus my own pass. A technical-accuracy reviewer verified every factual claim in the new text against the code: the
{id, value}shape, all four placeholder model ids,model_idas a real declared property, and all four Ollama claims including the:7b-codesuffix. All were reported verified. An admin-experience reviewer blocked on two items, both now fixed:openai-compatible: the provider forwards an empty string and the OpenAI SDK only consults the environment when the key isNone, so the fallback is skipped and the request fails on a missing credential. Note the asymmetry:base_urlgets an empty-to-Nonecoercion two lines earlier andapi_keydoes not. The list now states this and cross-references that section. The code fix that would make the original promise true is fix(providers): openai-compatible sends an empty api_key, so the documented env-var fallback never happens #439.Also taken from the review: the dropdown shows the placeholder model id (so an admin sanity-checking their work does not read it as an ignored config);
context_windowdeserves a callout because when unset NBI skips chat-history pruning entirely rather than assuming a default, so a small self-hosted window overflows; theOLLAMA_HOSTrequirement is now a blockquote callout matching how the guide flags other easy-to-miss requirements; and the placeholder secrets are obviously fake rather than plausible key prefixes.Risks and follow-ups
api_keyguidance is accurate but unsatisfying. It tells operators the key must live in the file and that keeping it out means templating or mounting. If fix(providers): openai-compatible sends an empty api_key, so the documented env-var fallback never happens #439 lands, this paragraph should be revisited to describe the working fallback instead.litellm-compatiblewas not traced end to end for the same empty-key behaviour. Its upstream usesapi_key or get_secret(...)patterns where a falsy value does chain to a lookup, so it may already behave as the earlier section promises. Noted in fix(providers): openai-compatible sends an empty api_key, so the documented env-var fallback never happens #439.Closes #438