Fail closed on a missing provider key instead of sending an unauthenticated request

[steve8708]

Follow-up measurement of the 2026-07-31 reliability wave (#2556, deployed 17:41Z) found the fixes largely worked, and surfaced two things that did not.

What moved (matched 7.75h windows either side of the deploy, 13 prod app DBs)

slice	metric	pre	post
interactive chats	dead turns	15 / 95 turns	4 / 33 turns
interactive chats	unknown (the 182-run baseline bucket)	7	0
scheduled jobs	no_progress	63	0
scheduled jobs	dispatch_payload_missing	35	0
scheduled jobs	run_timeout	19	0
scheduled jobs	unknown	19	0
plan (dominant job)	fail rate	32%	21%

What did not move — fixed here

Clips video-view-notifier: still 18/18 failing. The "skip gateway-rejected Builder credentials" fix could not help, because this never went through Builder credential selection. The app runs ai-sdk:openrouter and its production environment has no OPENROUTER_API_KEY. AISDKEngine resolved no key, omitted config.apiKey entirely, and the SDK sent a request with no Authorization header — hence the literal upstream Missing Authentication header. classifyProviderError coded it http_401, a transport error naming the wrong cause, and the job retried it every 30 minutes forever. The deploy only changed the classification (unknown → http_401); the outage was identical.

builder-engine and anthropic-engine both already fail closed with missing_credentials in this situation. ai-sdk-engine was the only engine that did not. It now does, and names the env var it wants. A configured baseUrl stays exempt, since a self-hosted gateway may legitimately need no key.

This makes the failure diagnosable and stops the retry loop. It does not by itself make that job succeed — the clips app still needs a key or a different engine, which is app configuration, not code.

New stale_run code, 73 runs. Not a regression so much as a relocation: runBackgroundAutomation now self-claims its run into background-processing, which subjects it to the 45s post-claim stale window. That tight window exists to reach a durable successor sooner — but a self-claimed automation carries no dispatch_payload, so attemptStaleRunRecovery can never build a successor for it. Reaping early was therefore pure loss: it killed still-working scheduled jobs (they died at 45–185s) that nothing could recover. Those rows now get the 90s background window; genuine HTTP workers keep the tight one, where an early reap does buy something.

The recovery path also reported this as payload_missing, which reads as "we are losing payloads" for the one case where nothing was ever lost — every HTTP-dispatched run writes its payload in the same INSERT that creates the row, so a payload-less row was simply never HTTP-dispatched. Renamed not_redispatchable.

Verification

Four tests, each confirmed to fail against the pre-fix code and pass with it (guard temporarily reverted, tests re-run, then restored).

• pnpm guards — all pass
• tsc --noEmit -p packages/core — clean
• vitest run src/agent/run-store src/agent/engine — 427 passed / 21 files

Not addressed

dispatch and design scheduled jobs had 0 runs in either window, so the 100%-failure baseline for them could not be re-measured here; their last runs were 07-28/07-29 and showed the old no_progress / background_worker_never_started codes.

[github-actions]

Here's a visual recap of what changed:

Open the full interactive recap

[builder-io-integration]

Builder reviewed your changes — looks good ✅

Review Details

Code Review Summary

PR #2566 addresses two reliability failures in the core agent runtime. First, AISDKEngine now fails closed with missing_credentials before constructing a provider or sending a request when a required provider key is absent, while preserving keyless custom baseUrl gateways and providers such as Ollama. Second, stale-run recovery distinguishes in-process background automations from HTTP-redispatchable runs: payload-less runs receive the wider background stale window and are reported as not_redispatchable rather than implying payload loss.

This is a high-risk change because it affects credential handling and background-run recovery, but the implementation is consistent with the existing engine error contract and run-store recovery flow. The regression tests cover missing keys, explicit keys, custom gateways, and both stale-window variants. Independent reviews found no confirmed bugs or compatibility issues.

✅ Good patterns: fail-closed behavior avoids unauthenticated provider requests; the baseUrl exemption is explicit; stale recovery semantics are documented and tested; diagnostic naming now reflects the actual condition.

🧪 Browser testing: Skipped — PR only modifies backend/config/docs/tests, no UI impact