test(gateway): document private setup and verify HTTP workflow

README.md

@@ -24,6 +24,9 @@ The operator configures `FAVA_TRAILS_AGENT_ID` on a dedicated process; caller
 `agent_id` must match it. A shared endpoint is one identity boundary. Configure
 `FAVA_TRAILS_OPERATOR=1` only on a separate operator-controlled endpoint.
 
+For a long-lived private ChatGPT connection, follow the deployment-neutral
+[Secure MCP Tunnel operator guide](docs/secure-mcp-tunnel.md).
+
 ## Why
 
 - **Supersession tracking** — a proposed correction leaves the original current; approved replacements make predecessors historical. No contradictory memories.

docs/secure-mcp-tunnel.md

@@ -0,0 +1,182 @@
+# Connect a private FAVA repository through Secure MCP Tunnel
+
+Run one long-lived gateway for one FAVA data repository. The gateway owns a
+loopback MCP server and an OpenAI `tunnel-client` process in the operator's
+environment. ChatGPT reaches that server through the authenticated OpenAI tunnel;
+FAVA does not need a public listening address. Existing stdio clients can continue
+using `fava-trails-server` independently.
+
+The gateway belongs to the data repository. Its lifetime and configuration must
+not depend on an agent session or the current product working directory. A
+service manager can supervise `fava-trails-tunnel run`; the operator supplies the
+repository, credentials, executable paths, and restart policy.
+
+## Prepare the repository and credentials
+
+Install FAVA and JJ, then use `fava-trails clone` or `fava-trails bootstrap` as
+explained in the [setup instructions](../AGENTS_SETUP_INSTRUCTIONS.md). Confirm
+that the chosen directory contains the intended `config.yaml`, `trails/`, and JJ
+repository, and that its remote and bookmark tracking are configured for sync.
+Use a dedicated clone for this long-lived gateway when other clients have their
+own working copies.
+
+Set an absolute repository path and an ordinary server identity in the service
+environment:
+
+```bash
+export FAVA_TRAILS_DATA_REPO=/absolute/path/to/fava-trails-data
+export FAVA_TRAILS_AGENT_ID=private-repository-gateway
+export FAVA_TRAILS_SCOPE_HINT=example/engineering  # optional discovery hint
+```
+
+Keep `FAVA_TRAILS_OPERATOR` unset on the ordinary shared endpoint. All clients of
+that endpoint share its configured authoring identity; a caller-supplied
+`agent_id` cannot impersonate another identity. The optional scope hint helps
+clients discover context and grants no access. See [governed recall](governed-recall.md).
+
+The repository's `config.yaml` must declare a supported Trust Gate policy.
+Configure the provider, exact model, and credential source for this host; machine
+settings in `~/.config/fava-trails/config.yaml` can override repository settings.
+For example, with a configured OpenAI-compatible provider:
+
+```yaml
+trust_gate: llm-oneshot
+trust_gate_provider: openai
+trust_gate_model: YOUR_CONFIGURED_MODEL_ID
+trust_gate_api_base: https://YOUR_PROVIDER_ENDPOINT/v1
+trust_gate_api_key_file: /absolute/private/path/to/trust-gate-key
+```
+
+Use an owner-readable credential file supplied by the operator's secret mechanism,
+not a key committed to the data repository. `trust_gate_api_key_env` is also
+supported when a service injects the key through its environment. Keep the review
+prompt at `trails/trust-gate-prompt.md`, or in the appropriate scope hierarchy.
+The gateway checks configuration and credentials before exposing the tunnel;
+actual proposal acceptance also requires the configured reviewer to respond.
+
+## Configure and validate the tunnel
+
+Follow OpenAI's [Secure MCP Tunnel guide](https://developers.openai.com/api/docs/guides/secure-mcp-tunnels)
+to create a tunnel in the intended Platform organization and install the official
+[`tunnel-client` release](https://github.com/openai/tunnel-client/releases/latest)
+for the host. The creating identity needs tunnel Read and Manage permissions;
+running or selecting a tunnel needs Read and Use. ChatGPT developer-mode access
+and the correct workspace/Platform association are separate requirements.
+
+Inject the control-plane API key as `CONTROL_PLANE_API_KEY` through the service's
+secret mechanism. It is separate from the Trust Gate provider key. Initialize a
+named profile, substituting the assigned tunnel ID:
+
+```bash
+tunnel-client init \
+  --profile private-fava \
+  --tunnel-id tunnel_YOUR_ASSIGNED_ID \
+  --mcp-server-url http://127.0.0.1:8765/mcp/ \
+  --control-plane-api-key-ref env:CONTROL_PLANE_API_KEY \
+  --health-listen-addr 127.0.0.1:0
+```
+
+Run this while no other gateway owns the selected loopback port:
+
+```bash
+fava-trails-tunnel preflight \
+  --data-repo "$FAVA_TRAILS_DATA_REPO" \
+  --profile private-fava --host 127.0.0.1 --port 8765 --mcp-path /mcp/ \
+  --ready-timeout 45
+```
+
+Preflight validates the repository, starts the local HTTP child, verifies MCP and
+data readiness, then stops that child. It does not start an external tunnel or
+prove that the remote reviewer is reachable. Resolve any reported failure before
+starting the service.
+
+Start the foreground gateway using the same repository, port, and profile:
+
+```bash
+fava-trails-tunnel run \
+  --data-repo "$FAVA_TRAILS_DATA_REPO" \
+  --profile private-fava --host 127.0.0.1 --port 8765 --mcp-path /mcp/ \
+  --ready-timeout 45 --sync-on-start --sync-interval-seconds 300
+```
+
+The supervisor starts the HTTP child, checks readiness, and then runs this
+`tunnel-client` command itself:
+
+```bash
+tunnel-client run --profile private-fava
+```
+
+Do not start a second copy of that command alongside the supervisor. For a manual
+tunnel-client diagnostic, stop the supervised gateway, start its HTTP child in
+one terminal, and run the tunnel client in a second terminal with the same service
+environment:
+
+```bash
+# Terminal 1: diagnostic child; stop it when diagnosis is complete.
+fava-trails-tunnel _serve-http \
+  --data-repo "$FAVA_TRAILS_DATA_REPO" \
+  --profile private-fava --host 127.0.0.1 --port 8765 --mcp-path /mcp/
+
+# Terminal 2: validate the profile and live local MCP server, then connect.
+tunnel-client doctor --profile private-fava --explain
+tunnel-client run --profile private-fava
+```
+
+`_serve-http` is the supervisor's internal child command, shown only for bounded
+diagnosis. Stop both diagnostic processes before restarting the managed gateway.
+When the managed gateway is already running, `tunnel-client doctor --profile
+private-fava --explain` can inspect its profile and local target without starting
+another tunnel. Use `curl --fail http://127.0.0.1:8765/healthz` for a local readiness
+check. Health includes repository readiness; process existence alone is
+insufficient. The startup output identifies the repository, trails directory,
+loopback URL, and tunnel profile.
+
+`--sync-on-start` requires successful repository sync before exposure. Periodic
+sync updates the long-lived clone; use `--sync-interval-seconds 0` only if another
+owned mechanism handles it. Dirty state, case collisions, or merge conflicts
+require operator attention. Do not grant operator access to the shared MCP
+endpoint merely to enable ordinary `sync`.
+
+## Connect ChatGPT and verify the workflow
+
+In ChatGPT's developer app setup, choose a Tunnel connection and select or enter
+the assigned tunnel ID. If it is unavailable, check the workspace/organization
+association and tunnel permissions in the OpenAI guide. Keep approval settings
+appropriate to the connected repository's data and write access.
+
+Through the actual connected app, perform this bounded acceptance sequence:
+
+1. Call `get_usage_guide`, then `list_scopes(prefix="example/engineering")`. Use an
+   exact returned `path` as `trail_name`; a prefix is a path prefix, not partial
+   text matching. Read-only lookup does not create a missing scope.
+2. Call `recall` with that explicit `trail_name`, then `get_thought` for an exact
+   returned ID. Default reads should contain only current approved records.
+3. With an operator-approved synthetic example, call `save_thought` in an intended
+   scope. Default reads must hide the new draft; `mode="authoring"` can retrieve
+   it through the same configured identity.
+4. Call `propose_truth`. Confirm a real configured Trust Gate result and recorded
+   approval provenance. A failed or rejected review is not successful approval.
+5. Call ordinary `sync`, then retrieve the approved ID through default
+   `get_thought`. Restart the gateway service and repeat that read to check
+   persistence and reconnection.
+
+Ordinary clients cannot use operator history, `forget`, or `rollback`, or claim a
+different author identity. Keep any separate administrative endpoint under
+operator control. Server-side failures are returned as structured tool results;
+SDK input validation failures use MCP's error result. Automated HTTP coverage in
+`tests/test_gateway_workflow.py` exercises the workflow against temporary JJ/Git
+repositories with only the external LLM result substituted. Live acceptance must
+use the real provider and connected client to establish those additional facts.
+
+## Deployment boundaries
+
+The loopback server has no standalone public authentication boundary. Do not
+expose private FAVA data on a public endpoint without separately designed
+authentication, authorization, and hosting controls.
+
+This v1 path covers one repository-owned gateway through OpenAI Secure MCP
+Tunnel. Per-agent aggregation, per-machine registries, Tailscale Aperture
+integration, Cloudflare Tunnel as the mainline path, WiseMachine host assumptions,
+and public HTTPS hosting are outside this path. Host-specific container and
+service runbooks belong to the deploying project. This guide does not authorize
+public publication or data migration.

tests/test_gateway_workflow.py

@@ -0,0 +1,161 @@
+"""Exercise ordinary gateway governance over real HTTP, storage and local Git sync."""
+
+from __future__ import annotations
+
+import json
+import subprocess
+from unittest.mock import AsyncMock
+
+import pytest
+from starlette.testclient import TestClient
+
+from fava_trails import server
+from fava_trails.http_runtime import create_streamable_http_app
+from fava_trails.tools import navigation
+from fava_trails.trust_gate import TrustResult
+
+
+@pytest.fixture
+def gateway_repo(tmp_fava_home, tmp_path, monkeypatch):
+    """Keep all storage and sync real; only the external LLM response is synthetic."""
+    monkeypatch.setenv("FAVA_TRAILS_DIR", str(tmp_fava_home / "trails"))
+    monkeypatch.setenv("FAVA_TRAILS_AGENT_ID", "synthetic-gateway")
+    monkeypatch.delenv("FAVA_TRAILS_OPERATOR", raising=False)
+    monkeypatch.setenv("FAVA_TRAILS_LOG_DIR", str(tmp_path / "logs"))
+    monkeypatch.setenv("SYNTHETIC_TRUST_GATE_KEY", "test-only-key")
+    (tmp_fava_home / "config.yaml").write_text(
+        "trails_dir: trails\ntrust_gate: llm-oneshot\npush_strategy: manual\n"
+        "trust_gate_api_key_env: SYNTHETIC_TRUST_GATE_KEY\n"
+    )
+    (tmp_fava_home / "trails" / "trust-gate-prompt.md").write_text("Synthetic review policy.\n")
+    (tmp_fava_home / ".gitignore").write_text(".jj/\n")
+
+    def git(*args):
+        return subprocess.run(
+            ["git", *args], cwd=tmp_fava_home, check=True, capture_output=True, text=True,
+        )
+
+    remote = tmp_path / "remote.git"
+    git("init", "--bare", str(remote))
+    git("add", ".")
+    git("-c", "user.name=Synthetic Gateway", "-c", "user.email=synthetic@example.invalid",
+        "commit", "-m", "Synthetic gateway fixture")
+    git("remote", "add", "origin", str(remote))
+    git("push", "origin", "HEAD:refs/heads/main")
+    review = AsyncMock(return_value=TrustResult(
+        verdict="approve", reasoning="Synthetic evaluator result", reviewer="synthetic-reviewer",
+    ))
+    monkeypatch.setattr(navigation, "review_thought", review)
+    monkeypatch.setattr(server, "_trail_managers", {})
+    monkeypatch.setattr(server, "_trail_init_lock", None)
+    return tmp_fava_home, review
+
+
+def http_rpc(client, version):
+    headers = {"Accept": "application/json, text/event-stream"}
+    next_id = 0
+
+    def rpc(method, params=None, *, notification=False):
+        nonlocal next_id
+        next_id += 1
+        request = {"jsonrpc": "2.0", "method": method}
+        if params is not None:
+            request["params"] = params
+        if not notification:
+            request["id"] = next_id
+        response = client.post("/mcp/", json=request, headers=headers)
+        if "mcp-session-id" in response.headers:
+            headers["Mcp-Session-Id"] = response.headers["mcp-session-id"]
+        if notification:
+            assert response.status_code == 202, response.text
+            return None
+        assert response.status_code == 200, response.text
+        if "text/event-stream" in response.headers["content-type"]:
+            payload = json.loads(next(
+                line[6:] for line in response.text.splitlines() if line.startswith("data: ")
+            ))
+        else:
+            payload = response.json()
+        assert payload["id"] == next_id
+        assert "result" in payload, payload
+        return payload["result"]
+
+    initialized = rpc("initialize", {
+        "protocolVersion": version, "capabilities": {},
+        "clientInfo": {"name": "synthetic-http-gateway", "version": "1"},
+    })
+    assert initialized["protocolVersion"] == version
+    headers["MCP-Protocol-Version"] = version
+    rpc("notifications/initialized", notification=True)
+    return rpc
+
+
+@pytest.mark.parametrize("version", ["2024-11-05", "2025-03-26", "2025-11-25"])
+def test_http_discover_draft_review_sync_and_governed_readback(gateway_repo, version):
+    repo, review = gateway_repo
+    scope = "synthetic/gateway"
+    with TestClient(create_streamable_http_app()) as client:
+        rpc = http_rpc(client, version)
+        listed = rpc("tools/list")
+        assert {"list_scopes", "save_thought", "propose_truth", "sync", "get_thought", "recall"} <= {
+            tool["name"] for tool in listed["tools"]
+        }
+
+        def call(name, arguments):
+            result = rpc("tools/call", {"name": name, "arguments": arguments})
+            assert not result.get("isError", False), result
+            assert json.loads(result["content"][0]["text"]) == result["structuredContent"]
+            return result["structuredContent"]
+
+        invalid = rpc("tools/call", {"name": "save_thought", "arguments": {"content": "missing scope"}})
+        assert invalid["isError"] is True
+        assert not (repo / "trails" / scope).exists()
+        assert call("list_scopes", {})["scopes"] == []
+        assert call("recall", {"trail_name": scope})["status"] == "error"
+        assert not (repo / "trails" / scope).exists()
+        saved = call("save_thought", {"trail_name": scope, "content": "Synthetic gateway acceptance."})
+        thought_id = saved["thought"]["thought_id"]
+        assert saved["thought"]["agent_id"] == "synthetic-gateway"
+        assert call("list_scopes", {"prefix": "synthetic"})["scopes"] == [{"path": scope}]
+        assert call("recall", {"trail_name": scope})["count"] == 0
+        assert call("get_thought", {"trail_name": scope, "thought_id": thought_id})["status"] == "error"
+        owned = call("get_thought", {"trail_name": scope, "thought_id": thought_id, "mode": "authoring"})
+        assert owned["thought"]["validation_status"] == "draft"
+        for name, arguments in (
+            ("recall", {"mode": "history"}),
+            ("save_thought", {"content": "spoof", "agent_id": "another-agent"}),
+            ("rollback", {"op_id": "must-not-run"}),
+            ("forget", {}),
+            ("propose_truth", {"thought_id": thought_id, "approval": "human"}),
+        ):
+            denied = call(name, {"trail_name": scope, **arguments})
+            assert denied["status"] == "error", denied
+        review.assert_not_awaited()
+        proposed = call("propose_truth", {"trail_name": scope, "thought_id": thought_id})
+        assert proposed["status"] == "ok", proposed
+        assert proposed["thought"]["validation_status"] == "approved"
+        review.assert_awaited_once()
+        assert call("sync", {"trail_name": scope}) == {"status": "ok", "message": "Sync complete."}
+        recalled = call("recall", {"trail_name": scope})
+        assert [thought["thought_id"] for thought in recalled["thoughts"]] == [thought_id]
+        assert call("get_thought", {"trail_name": scope, "thought_id": thought_id})["thought"]["validation_status"] == "approved"
+
+
+def test_http_sync_refuses_dirty_repository_without_disclosing_paths(gateway_repo):
+    repo, _ = gateway_repo
+    with TestClient(create_streamable_http_app()) as client:
+        rpc = http_rpc(client, "2025-03-26")
+        saved = rpc("tools/call", {"name": "save_thought", "arguments": {
+            "trail_name": "synthetic/gateway", "content": "Synthetic sync fixture",
+        }})
+        assert saved["structuredContent"]["status"] == "ok"
+        private_path = repo / "operator-private-fixture.txt"
+        private_path.write_text("Synthetic uncommitted bytes")
+        blocked = rpc("tools/call", {"name": "sync", "arguments": {"trail_name": "synthetic/gateway"}})
+        assert not blocked.get("isError", False)
+        assert blocked["structuredContent"] == {
+            "status": "blocked",
+            "message": "Sync blocked by uncommitted repository changes. Operator attention is required.",
+        }
+        assert "operator-private-fixture" not in json.dumps(blocked)
+        assert private_path.read_text() == "Synthetic uncommitted bytes"