feat: add aggregate /.well-known/x402 and external buyer-tool compat

CLAUDE.md

@@ -62,6 +62,7 @@ Integration tests use `//go:build integration`; skip when prerequisites missing.
 | Public (tunnel) | none | `/.well-known/agent-registration.json` | ERC-8004 httpd |
 | Public (tunnel) | none | `/skill.md` | service catalog |
 | Public (tunnel) | none | `/api/services.json` | service catalog JSON feed (`displayName`, `tagline`, `logoUrl`, `theme`, `themeVars`, `faviconUrl`, `ogImageUrl`, `description(+Html)`, `customCss`, `services[]`) |
+| Public (tunnel) | none | `/.well-known/x402` | aggregate x402 discovery fallback (AgentCash/x402scan-style crawlers that don't parse `/openapi.json`'s `x-payment-info`) |
 | Public (tunnel) | tunnel hostname only | `/` | storefront landing page (Next.js) |
 
 **NEVER remove hostname restrictions from frontend or eRPC HTTPRoutes** — exposing the frontend/RPC to the public internet is a critical security flaw.
@@ -457,7 +458,7 @@ A registry digest pin instead of `:latest` on the verifier means your dev rewrit
 20. **402 page silently falls back to JSON when the template errors** — `sendPaymentRequiredHTML` swallows template-exec errors and re-sends the JSON body, so referencing a field in `payment_required.html` that isn't in the render's data struct doesn't crash anything: browsers just start getting JSON. Symptom: `Content-Type: application/json` on an `Accept: text/html` request. Any template-field addition needs the struct field added in `paymentrequired.go` AND a test asserting the HTML branch still renders (the existing branding tests check Content-Type/markup).
 21. **`html/template` rejects `data:` URIs in URL contexts (`#ZgotmplZ`)** — inline logos/favicons from `sell info set --logo-file` are `data:image/...;base64` URIs; interpolating them into `src=`/`href=` via a plain string yields the literal `#ZgotmplZ` (broken image). Branding asset URLs must go through `storefront.SafeAssetURL` (validates http(s)/`data:image` then returns `template.URL`); regression test `TestPaymentRequiredHTML_InlineDataURILogo`.
 22. **Poisoned LiteLLM model group — intermittent 404 `{'detail': 'Not Found'}` on ~50% of requests** — two `model_list` deployments share a `model_name` but disagree on `api_base` (one with `/v1`, one without); LiteLLM shuffles between them and does not retry a 404. Historic cause: auto-discovery registering the same host endpoint that `obol model setup custom` later added correctly (#745). Diagnosis: compare the router's live view (`GET /model/info`) against the `litellm-config` CM — the drift checker compares model NAMES only and cannot see divergent `api_base` (#746). The vLLM access log is decisive: alternating `POST /chat/completions 404` / `POST /v1/chat/completions 200`.
-23. **Buyer disconnect ≠ no charge (zombie settlement)** — a client abort routinely does NOT propagate past cloudflared; the upstream agent finishes the turn, the handler returns 2xx, and settlement fires → buyer debited for a response nobody received (proven on-chain 2026-07-14, the Bankr incident). The verifier skips settlement when the request context is already canceled (#743), but that only covers propagated cancels — the real protection is keeping paid agent runs SHORT (`Agent.spec.maxTurns`; runs must finish inside the ~100s tunnel window and typical client timeouts). Concurrency: Hermes caps simultaneous runs in-process (`gateway.api_server.max_concurrent_runs`, internal default 10, 429 "Too many concurrent runs"; exposed as `Agent.spec.maxConcurrentRuns`, 0 = disabled) — keep it coherent with the edge `spec.limits.maxInFlight` Traefik gate; 4xx responses are never settled, so both gates are financially safe.
+23. **Buyer disconnect ≠ no charge (zombie settlement)** — a client abort routinely does NOT propagate past cloudflared; the upstream agent can finish 2xx and older seller builds still called `/settle` → buyer debited while the UI showed failure (Bankr ~30s retries ×3 on-chain, 2026-08-05). This deferred-to-`finalize()` protection applies ONLY to `text/event-stream` (SSE) responses — `settlementInterceptor.WriteHeader` still settles eagerly, before committing the status, for every non-streaming response (plain `sell http`, non-streaming `sell agent`), exactly as before this fix. For SSE, `finalize()` skips `/settle` on canceled context, **client Write errors** (broken pipe), or zero body bytes — see `docs/observability.md` ("External buyers (Bankr)"). Separately, Bankr auto-pay often fails verify with `facilitator_error` because it signs EIP-3009 `validAfter=now` (use past buffer + ≥180s timeout). Still keep paid agent runs SHORT (`Agent.spec.maxTurns`) inside the tunnel window. Concurrency: Hermes `max_concurrent_runs` / `Agent.spec.maxConcurrentRuns` must stay coherent with edge `spec.limits.maxInFlight`; 4xx is never settled.
 
 For a fuller debug catalog with symptom->fix mapping, see `.agents/skills/obol-stack-dev/references/release-smoke-debugging.md`.
 
@@ -478,6 +479,7 @@ The Cloudflare tunnel exposes the cluster to the public internet. Only x402-gate
 - `/.well-known/agent-registration.json` — ERC-8004 discovery
 - `/skill.md` — machine-readable service catalog
 - `/api/services.json` — service catalog envelope (`displayName`, `tagline`, `logoUrl`, theme/branding fields, `services[]`)
+- `/.well-known/x402` — aggregate x402 discovery fallback (AgentCash/x402scan-style discovery convention; no secret material)
 - `/` on tunnel hostname — public storefront landing page (Next.js)
 
 ## Dependencies

docs/observability.md

@@ -252,6 +252,58 @@ A `Transfer` to the expected recipient that exists while the buyer reports
 
 ---
 
+## External buyers (Bankr): two failure modes, one canonical ledger
+
+Bankr chat / `bankr x402 call` against Obol agent offers confused operators
+because the UI error and the on-chain outcome often disagreed.
+
+**Root cause (2026-08-05 investigation): Bankr chat/CLI auto-pay is not
+built to pay arbitrary third-party x402 endpoints at all.** Per Bankr's own
+docs, auto-pay is scoped to endpoints deployed through `bankr x402 deploy`
+and then approved into Bankr's own discovery index (criteria undocumented) —
+[docs.bankr.bot/x402-cloud/quick-start](https://docs.bankr.bot/x402-cloud/quick-start/).
+Their Apps SDK (`bankr.x402.fetch`) separately requires a pre-declared
+`allowedHosts` allowlist —
+[docs.bankr.bot/apps/overview](https://docs.bankr.bot/apps/overview/). Every
+documented example targets `"network":"base"` (mainnet); no Base Sepolia
+support was found anywhere in their docs or the `BankrBot/skills` repo. An
+arbitrary seller like ours is out of scope for chat auto-pay independent of
+the two wire-level failure modes below — `bankr wallet sign`
+([docs.bankr.bot/cli](https://docs.bankr.bot/cli/)) is Bankr's own
+officially documented manual-signing primitive, not a workaround hack, and
+is the only proven path today.
+
+When Bankr auto-pay *does* attempt a call anyway, there are **two separate
+wire-level failure modes**:
+
+| Mode | What the buyer sees | What happened | Charge? |
+|---|---|---|---|
+| **A — Voucher** | JSON `503` with `reason:facilitator_error`, `detail:unexpected_error` | Bankr auto-pay signed EIP-3009 with `validAfter=wall-clock now`. Base USDC requires `block.timestamp > validAfter`, so facilitator `/verify` rejects. | Usually **no** (verify never succeeded). |
+| **B — Timeout / zombie** | Timeout, 504, or a generic “payment failed” after ~30s | Verify **succeeded**, agent was still running (often 30–120s to first SSE byte). Bankr’s short client timeout aborted the UI. Cloudflare often does **not** cancel the seller’s request context, so older seller builds still called `/settle` after upstream finished → BaseScan shows 0.001 USDC Transfers in a triple-retry burst. | **Yes** on older builds. |
+
+**Seller hardenings** (HandleProxy settlement interceptor):
+
+- Settle SSE responses only in `finalize()` after the stream completes —
+  never on the first `WriteHeader(200)`.
+- Skip `/settle` when `r.Context()` is canceled, when a body `Write` to the
+  client fails (broken pipe — the reliable signal when cancel does not
+  propagate), or when zero body bytes were written.
+- Upstream proxy errors after verify return structured JSON with
+  `paymentVerified:true`, `paymentSettled:false`, `retriable:false` so buyers
+  do not auto-retry storms.
+
+**Buyer guidance** (storefront Bankr prompts are type-specific):
+- **http** — prefer Bankr chat auto-pay (fast enough for the ~30s window).
+  Do not ask chat to run `bankr wallet sign` (it cannot).
+- **agent / inference** — do not use Bankr chat/Apps auto-pay (`rpc timeout`);
+  use `bankr wallet sign` with a past `validAfter` buffer and HTTP timeout ≥180s.
+  After any timeout, check BaseScan before retrying.
+
+**Still true:** chain Transfers are canonical. Seller `paymentSettled:false`
+and Bankr UI copy are best-effort signals.
+
+---
+
 ## Recording rule conventions
 
 Naming follows the standard Prometheus pattern:

flows/clients/x402-generic-buyer.go

@@ -0,0 +1,141 @@
+//go:build ignore
+
+// Generic x402 HTTP buyer client for flow-22-external-buyer-compat.
+//
+// Built entirely from the public x402-foundation/x402/go/v2 SDK's
+// documented client pattern (see CLIENT.md "Basic HTTP Client" in the SDK
+// module): wrap a plain *http.Client with x402 payment handling and call it
+// like any other HTTP client. No Obol CLI, no buy.py, no PurchaseRequest CR
+// — this is what a third-party buyer tool's own wallet/agent does under the
+// hood (AgentCash, Bankr, or any other x402-compliant client), so a
+// successful "paid" run here is evidence the seller works with any
+// standards-compliant buyer, not just Obol's own tooling.
+//
+// Run from the repo root (module context):
+//
+//	go run flows/clients/x402-generic-buyer.go \
+//	  -mode <unpaid|paid> -url <resource-url> \
+//	  [-method GET|POST] [-body '<json>']
+//
+// Modes:
+//
+//	unpaid  call the endpoint with no signer registered — expect a 402
+//	paid    call the endpoint with a signer from X402_CLIENT_KEY — the
+//	        wrapped client detects the 402, signs, retries, and returns
+//	        whatever the seller sends back (expect 200 on success)
+//
+// -method/-body cover both the plain HTTP demo (GET, empty body) and the
+// OpenAI-compatible chat-completions shape (POST + JSON body) used by
+// inference/agent offers.
+//
+// X402_CLIENT_KEY carries the buyer's 0x-prefixed private key via env so it
+// never appears on argv. Output is ONE JSON object on stdout; the flow
+// asserts on its fields.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	x402 "github.com/x402-foundation/x402/go/v2"
+	x402http "github.com/x402-foundation/x402/go/v2/http"
+	exactclient "github.com/x402-foundation/x402/go/v2/mechanisms/evm/exact/client"
+	evmsigner "github.com/x402-foundation/x402/go/v2/signers/evm"
+)
+
+func main() {
+	url := flag.String("url", "", "paid resource URL")
+	mode := flag.String("mode", "paid", "unpaid|paid")
+	network := flag.String("network", "eip155:84532", "CAIP-2 network to register the signer for")
+	method := flag.String("method", "GET", "HTTP method")
+	body := flag.String("body", "", "optional request body (e.g. chat-completions JSON)")
+	timeout := flag.Duration("timeout", 120*time.Second, "per-request timeout")
+	flag.Parse()
+
+	out := map[string]any{"mode": *mode, "method": strings.ToUpper(*method)}
+	if *url == "" {
+		out["error"] = "-url is required"
+		emit(out)
+		os.Exit(1)
+	}
+	if err := run(*url, *mode, *network, strings.ToUpper(*method), *body, *timeout, out); err != nil {
+		out["error"] = err.Error()
+		emit(out)
+		os.Exit(1)
+	}
+	emit(out)
+}
+
+func emit(m map[string]any) {
+	b, _ := json.Marshal(m)
+	fmt.Println(string(b))
+}
+
+func run(url, mode, network, method, body string, timeout time.Duration, out map[string]any) error {
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+
+	// mode=unpaid deliberately skips the x402 wrapper entirely — a real
+	// unpaid buyer is just a plain HTTP client with no x402 awareness, and
+	// the wrapper's RoundTrip returns an error (not a 402 response) when no
+	// scheme is registered for the seller's network, since it can't build a
+	// payment payload. Bypassing it here is what lets this mode observe the
+	// seller's raw 402 challenge, exactly like a naive caller would.
+	httpClient := http.DefaultClient
+	if mode == "paid" {
+		key := os.Getenv("X402_CLIENT_KEY")
+		if key == "" {
+			return fmt.Errorf("X402_CLIENT_KEY is required for -mode paid")
+		}
+		signer, err := evmsigner.NewClientSignerFromPrivateKey(key)
+		if err != nil {
+			return fmt.Errorf("signer: %w", err)
+		}
+		out["buyerAddress"] = signer.Address()
+
+		// Following CLIENT.md's quick-start verbatim: create the core
+		// client, register the scheme, wrap a plain http.Client.
+		client := x402.Newx402Client().
+			Register(x402.Network(network), exactclient.NewExactEvmScheme(signer, nil))
+		httpClient = x402http.WrapHTTPClientWithPayment(http.DefaultClient, x402http.Newx402HTTPClient(client))
+	}
+
+	var bodyReader io.Reader
+	if body != "" {
+		bodyReader = strings.NewReader(body)
+	}
+	req, err := http.NewRequestWithContext(ctx, method, url, bodyReader)
+	if err != nil {
+		return fmt.Errorf("build request: %w", err)
+	}
+	if body != "" {
+		req.Header.Set("Content-Type", "application/json")
+	}
+	resp, err := httpClient.Do(req)
+	if err != nil {
+		return fmt.Errorf("do: %w", err)
+	}
+	defer resp.Body.Close()
+
+	respBody, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return fmt.Errorf("read body: %w", err)
+	}
+
+	out["status"] = resp.StatusCode
+	out["body"] = string(respBody)
+	if v := resp.Header.Get("X-PAYMENT-RESPONSE"); v != "" {
+		out["paymentResponseHeader"] = v
+	}
+	if v := resp.Header.Get("PAYMENT-RESPONSE"); v != "" {
+		out["paymentResponseHeaderV2"] = v
+	}
+	return nil
+}