Add Inferrail Hosted: shared per-run budgets with self-serve credits (experimental)

[domondi1]

Founder review required: this PR adds product behavior, payments and public pricing. Do not merge without review. Merging does not deploy anything.

1. What a customer can now do

An agent or developer can:

1. find the offer (GET /pricing, GET /llms.txt, docs/hosted/);
2. create a workspace with one call (POST /v1/workspaces, no account);
3. point an OpenAI or Anthropic SDK at it, with the provider key in X-Provider-Api-Key (sent per request, never stored);
4. name each run and give it a dollar ceiling (X-Inferrail-Attribute-Work-Id, X-Inferrail-Budget-Usd);
5. see enforcement and receipts (GET /v1/work/{id});
6. get 402 allowance_exhausted, listing how to buy, when the free allowance runs out;
7. buy credits, as an agent with USDC over x402 (no account) or as a person through Stripe Checkout;
8. keep going.

test_complete_customer_journey walks this whole path.

2. What the paid hosted offering adds

The open-source gateway already enforces per-run budgets on one machine, and this PR doesn't touch src/inferrail (0 lines changed) or restrict it.

The hosted service adds:

• operation: nothing to run;
• a durable workspace;
• one central budget ledger that every agent and machine calling the workspace shares;
• self-service billing in which agents and people buy from the same credit ledger.

The underlying property is the one in the public per-run budget benchmark: a budget declared on the fly in a header holds under concurrent fan-out.

3. Implemented and tested

The service is hosted/budget_gateway/, about 1,430 lines. It reuses the gateway's engines, BudgetEnforcer, ReceiptsStore, attribution, pricing and error codes, and the existing fake facilitator. It is not a fork of the trial service: the trial's in-memory tenants and key vault are the wrong lifecycle for paid workspaces.

File	What
service.py	API, provider proxying, the x402 and Stripe rails, the instance lock, reconciliation loop, key scrubbing
workspaces.py	Workspaces, governed-run metering, the credit ledger and its financial states
stripe_checkout.py	Checkout session creation; webhook signature verification (no SDK)
chain_check.py	USDC authorizationState read for on-chain reconciliation

39 tests, in tests/unit/hosted/test_budget_gateway.py, plus a CI job that fails if any of them is skipped.

• Metering: billing only when a call is answered; held, released and billed runs; monthly reset.
• Concurrency: 25 threads competing for one credit; 12 parallel calls on one run budget.
• Restarts: budget spend survives a restart; orphaned holds are released; a second instance on the same data directory refuses to start.
• Isolation: workspaces are isolated.
• Payments:
  • a payment grants at most once;
  • webhook signature, staleness, replay and amount checks;
  • an interrupted webhook is credited exactly once;
  • a checkout redirect alone grants nothing;
  • refunds are proportional, monotonic and can block a negative balance;
  • x402 settlement success and failure;
  • a crash after settlement is recovered by on-chain reconciliation;
  • an invalid workspace is never charged.
• Keys: provider keys are never stored, never returned, and scrubbed from upstream error text; workspace keys are hash-only.
• Guards: oversized bodies and creation throttling.
• Rails and terms: both rails are off unless configured; mainnet needs explicit approval plus an RPC; the published terms match.

Regression, on this branch's code:

Suite	Result
tests/unit/hosted	710 passed / 17 skipped / 8 xfailed (all skips and xfails pre-existing)
Core (--ignore=tests/unit/hosted)	867 passed / 2 skipped
Site pages	passed
ruff check .	clean
Concurrency and crash subset, run 15 times	stable
Diff under src/	0 lines

Security review (self-review; findings fixed in this PR):

• Shared-connection reads. Ledger reads on the shared SQLite connection weren't serialized, so a concurrent request could hit it mid-transaction. Fixed with one reentrant lock.
• Provider key in upstream errors. Upstream errors could echo the provider key. Fixed by scrubbing the request's key from all error text; unhandled errors log the exception type only.
• Creation throttle behind a proxy. The per-IP throttle saw only the proxy's address. Fixed with BG_FORWARDED_ALLOW_IPS.
• Mainnet x402 without reconciliation. That configuration could strand a settled payment. Startup now refuses.
• Already sound:
  • the webhook trusts only signed events, with a 5-minute tolerance and the amount checked against the published price;
  • credits come from our own session metadata, never from client input;
  • provider base URLs are fixed (no SSRF);
  • declared run ceilings are capped by BG_PER_WORK_MAX_USD.
• Accepted for this experiment (listed in the README):
  • no per-workspace request rate limit;
  • chunked bodies aren't size-checked;
  • x402 is a bearer payment: whoever submits a signed authorization chooses the workspace it credits.

4. What remains experimental

• Pricing is a first offer, not validated demand: 2,000 governed runs a month free, then $0.001 per run, prepaid at $1 (x402) or $10 (card). The service labels it "status": "experimental pricing".
• Not deployed. There are no customers and no revenue.

5. Exact guarantees and deployment constraints

• Single instance.
  • Guarantees hold for exactly one instance per data directory (one worker) on a local persistent disk.
  • Many client machines may call that instance; that's the fleet-wide guarantee.
  • Multiple gateway replicas sharing state are not supported and not claimed.
  • An exclusive flock on the data directory enforces this at startup.
• Billing: a governed run (a distinct work id in a UTC month) is billed only if at least one of its calls is answered by the provider. Failed or budget-refused runs cost nothing. There are no subscriptions, auto top-ups or overage bills.
• Credits are granted only from:
  • a signature-verified, paid Stripe Checkout session at the published amount, once per session; or
  • a settled x402 payment, once per payer and nonce, confirmed either by the settle hook or by USDC authorizationState on chain.
• Refunds: Stripe refunds reverse credits automatically. x402 refunds are an operator action.
• The full mechanism and test table is in hosted/budget_gateway/README.md.

6. Still needed before accepting real money

1. Founder review and merge of this PR.
2. Host: a single instance with a persistent local disk (Render with a disk, or equivalent), TLS, BG_FORWARDED_ALLOW_IPS, and DNS for the public URL.
3. At least one payment rail, configured in the host environment:
   • Card: a Stripe account and restricted secret key, plus a webhook for checkout.session.completed and charge.refunded.
   • USDC: a founder-held BG_X402_PAY_TO, BG_X402_MAINNET_APPROVED=1, BG_BASE_RPC_URL, and the CDP API key for the facilitator.
4. A live smoke test: a real provider key, a Stripe test-mode purchase, and one small mainnet x402 purchase from a non-customer wallet, recorded and excluded from revenue.
5. Known limits (see the README) to accept or fix:
   • no per-workspace request rate limit;
   • chunked bodies aren't size-checked;
   • x402 refunds and Stripe disputes are manual;
   • no schema migrations yet.

7. How a legitimate customer completes the first paid run

1. POST /v1/workspaces returns a key.
2. Governed calls run on the free allowance.
3. Once the allowance is used, a call returns 402 allowance_exhausted, listing POST /v1/credits/checkout (card) and POST /v1/credits/x402 (USDC).
4. Card:
   • the customer opens the returned Stripe URL and pays $10;
   • Stripe's signed webhook grants 10,000 runs;
   • the next call proceeds.
5. Agent:
   • the agent's x402 client pays the 402 on /v1/credits/x402 ($1 USDC);
   • settlement grants 1,000 runs;
   • the next call proceeds.

8. Commercial evidence still missing

• No customer has used or paid for this. Willingness to pay is unproven, and so are price, activation and distribution.
• The benchmark shows a technical difference, not demand.

The plan is to publish the price, run it for 30 days, and read the result:

Outcome	Reading
Fewer than 10 workspaces	Distribution
Fewer than 3 activated	Value or onboarding
Activated workspaces hit the limit but nobody buys	Price or value
One or more independent purchases	First evidence of willingness to pay