diff --git a/README.md b/README.md index 2f6414e..58ceee2 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,9 @@ The operator configures `FAVA_TRAILS_AGENT_ID` on a dedicated process; caller `agent_id` must match it. A shared endpoint is one identity boundary. Configure `FAVA_TRAILS_OPERATOR=1` only on a separate operator-controlled endpoint. +For a long-lived private ChatGPT connection, follow the deployment-neutral +[Secure MCP Tunnel operator guide](docs/secure-mcp-tunnel.md). + ## Why - **Supersession tracking** — a proposed correction leaves the original current; approved replacements make predecessors historical. No contradictory memories. diff --git a/docs/secure-mcp-tunnel.md b/docs/secure-mcp-tunnel.md new file mode 100644 index 0000000..46b2486 --- /dev/null +++ b/docs/secure-mcp-tunnel.md @@ -0,0 +1,182 @@ +# Connect a private FAVA repository through Secure MCP Tunnel + +Run one long-lived gateway for one FAVA data repository. The gateway owns a +loopback MCP server and an OpenAI `tunnel-client` process in the operator's +environment. ChatGPT reaches that server through the authenticated OpenAI tunnel; +FAVA does not need a public listening address. Existing stdio clients can continue +using `fava-trails-server` independently. + +The gateway belongs to the data repository. Its lifetime and configuration must +not depend on an agent session or the current product working directory. A +service manager can supervise `fava-trails-tunnel run`; the operator supplies the +repository, credentials, executable paths, and restart policy. + +## Prepare the repository and credentials + +Install FAVA and JJ, then use `fava-trails clone` or `fava-trails bootstrap` as +explained in the [setup instructions](../AGENTS_SETUP_INSTRUCTIONS.md). Confirm +that the chosen directory contains the intended `config.yaml`, `trails/`, and JJ +repository, and that its remote and bookmark tracking are configured for sync. +Use a dedicated clone for this long-lived gateway when other clients have their +own working copies. + +Set an absolute repository path and an ordinary server identity in the service +environment: + +```bash +export FAVA_TRAILS_DATA_REPO=/absolute/path/to/fava-trails-data +export FAVA_TRAILS_AGENT_ID=private-repository-gateway +export FAVA_TRAILS_SCOPE_HINT=example/engineering # optional discovery hint +``` + +Keep `FAVA_TRAILS_OPERATOR` unset on the ordinary shared endpoint. All clients of +that endpoint share its configured authoring identity; a caller-supplied +`agent_id` cannot impersonate another identity. The optional scope hint helps +clients discover context and grants no access. See [governed recall](governed-recall.md). + +The repository's `config.yaml` must declare a supported Trust Gate policy. +Configure the provider, exact model, and credential source for this host; machine +settings in `~/.config/fava-trails/config.yaml` can override repository settings. +For example, with a configured OpenAI-compatible provider: + +```yaml +trust_gate: llm-oneshot +trust_gate_provider: openai +trust_gate_model: YOUR_CONFIGURED_MODEL_ID +trust_gate_api_base: https://YOUR_PROVIDER_ENDPOINT/v1 +trust_gate_api_key_file: /absolute/private/path/to/trust-gate-key +``` + +Use an owner-readable credential file supplied by the operator's secret mechanism, +not a key committed to the data repository. `trust_gate_api_key_env` is also +supported when a service injects the key through its environment. Keep the review +prompt at `trails/trust-gate-prompt.md`, or in the appropriate scope hierarchy. +The gateway checks configuration and credentials before exposing the tunnel; +actual proposal acceptance also requires the configured reviewer to respond. + +## Configure and validate the tunnel + +Follow OpenAI's [Secure MCP Tunnel guide](https://developers.openai.com/api/docs/guides/secure-mcp-tunnels) +to create a tunnel in the intended Platform organization and install the official +[`tunnel-client` release](https://github.com/openai/tunnel-client/releases/latest) +for the host. The creating identity needs tunnel Read and Manage permissions; +running or selecting a tunnel needs Read and Use. ChatGPT developer-mode access +and the correct workspace/Platform association are separate requirements. + +Inject the control-plane API key as `CONTROL_PLANE_API_KEY` through the service's +secret mechanism. It is separate from the Trust Gate provider key. Initialize a +named profile, substituting the assigned tunnel ID: + +```bash +tunnel-client init \ + --profile private-fava \ + --tunnel-id tunnel_YOUR_ASSIGNED_ID \ + --mcp-server-url http://127.0.0.1:8765/mcp/ \ + --control-plane-api-key-ref env:CONTROL_PLANE_API_KEY \ + --health-listen-addr 127.0.0.1:0 +``` + +Run this while no other gateway owns the selected loopback port: + +```bash +fava-trails-tunnel preflight \ + --data-repo "$FAVA_TRAILS_DATA_REPO" \ + --profile private-fava --host 127.0.0.1 --port 8765 --mcp-path /mcp/ \ + --ready-timeout 45 +``` + +Preflight validates the repository, starts the local HTTP child, verifies MCP and +data readiness, then stops that child. It does not start an external tunnel or +prove that the remote reviewer is reachable. Resolve any reported failure before +starting the service. + +Start the foreground gateway using the same repository, port, and profile: + +```bash +fava-trails-tunnel run \ + --data-repo "$FAVA_TRAILS_DATA_REPO" \ + --profile private-fava --host 127.0.0.1 --port 8765 --mcp-path /mcp/ \ + --ready-timeout 45 --sync-on-start --sync-interval-seconds 300 +``` + +The supervisor starts the HTTP child, checks readiness, and then runs this +`tunnel-client` command itself: + +```bash +tunnel-client run --profile private-fava +``` + +Do not start a second copy of that command alongside the supervisor. For a manual +tunnel-client diagnostic, stop the supervised gateway, start its HTTP child in +one terminal, and run the tunnel client in a second terminal with the same service +environment: + +```bash +# Terminal 1: diagnostic child; stop it when diagnosis is complete. +fava-trails-tunnel _serve-http \ + --data-repo "$FAVA_TRAILS_DATA_REPO" \ + --profile private-fava --host 127.0.0.1 --port 8765 --mcp-path /mcp/ + +# Terminal 2: validate the profile and live local MCP server, then connect. +tunnel-client doctor --profile private-fava --explain +tunnel-client run --profile private-fava +``` + +`_serve-http` is the supervisor's internal child command, shown only for bounded +diagnosis. Stop both diagnostic processes before restarting the managed gateway. +When the managed gateway is already running, `tunnel-client doctor --profile +private-fava --explain` can inspect its profile and local target without starting +another tunnel. Use `curl --fail http://127.0.0.1:8765/healthz` for a local readiness +check. Health includes repository readiness; process existence alone is +insufficient. The startup output identifies the repository, trails directory, +loopback URL, and tunnel profile. + +`--sync-on-start` requires successful repository sync before exposure. Periodic +sync updates the long-lived clone; use `--sync-interval-seconds 0` only if another +owned mechanism handles it. Dirty state, case collisions, or merge conflicts +require operator attention. Do not grant operator access to the shared MCP +endpoint merely to enable ordinary `sync`. + +## Connect ChatGPT and verify the workflow + +In ChatGPT's developer app setup, choose a Tunnel connection and select or enter +the assigned tunnel ID. If it is unavailable, check the workspace/organization +association and tunnel permissions in the OpenAI guide. Keep approval settings +appropriate to the connected repository's data and write access. + +Through the actual connected app, perform this bounded acceptance sequence: + +1. Call `get_usage_guide`, then `list_scopes(prefix="example/engineering")`. Use an + exact returned `path` as `trail_name`; a prefix is a path prefix, not partial + text matching. Read-only lookup does not create a missing scope. +2. Call `recall` with that explicit `trail_name`, then `get_thought` for an exact + returned ID. Default reads should contain only current approved records. +3. With an operator-approved synthetic example, call `save_thought` in an intended + scope. Default reads must hide the new draft; `mode="authoring"` can retrieve + it through the same configured identity. +4. Call `propose_truth`. Confirm a real configured Trust Gate result and recorded + approval provenance. A failed or rejected review is not successful approval. +5. Call ordinary `sync`, then retrieve the approved ID through default + `get_thought`. Restart the gateway service and repeat that read to check + persistence and reconnection. + +Ordinary clients cannot use operator history, `forget`, or `rollback`, or claim a +different author identity. Keep any separate administrative endpoint under +operator control. Server-side failures are returned as structured tool results; +SDK input validation failures use MCP's error result. Automated HTTP coverage in +`tests/test_gateway_workflow.py` exercises the workflow against temporary JJ/Git +repositories with only the external LLM result substituted. Live acceptance must +use the real provider and connected client to establish those additional facts. + +## Deployment boundaries + +The loopback server has no standalone public authentication boundary. Do not +expose private FAVA data on a public endpoint without separately designed +authentication, authorization, and hosting controls. + +This v1 path covers one repository-owned gateway through OpenAI Secure MCP +Tunnel. Per-agent aggregation, per-machine registries, Tailscale Aperture +integration, Cloudflare Tunnel as the mainline path, WiseMachine host assumptions, +and public HTTPS hosting are outside this path. Host-specific container and +service runbooks belong to the deploying project. This guide does not authorize +public publication or data migration. diff --git a/tests/test_gateway_workflow.py b/tests/test_gateway_workflow.py new file mode 100644 index 0000000..539bf33 --- /dev/null +++ b/tests/test_gateway_workflow.py @@ -0,0 +1,161 @@ +"""Exercise ordinary gateway governance over real HTTP, storage and local Git sync.""" + +from __future__ import annotations + +import json +import subprocess +from unittest.mock import AsyncMock + +import pytest +from starlette.testclient import TestClient + +from fava_trails import server +from fava_trails.http_runtime import create_streamable_http_app +from fava_trails.tools import navigation +from fava_trails.trust_gate import TrustResult + + +@pytest.fixture +def gateway_repo(tmp_fava_home, tmp_path, monkeypatch): + """Keep all storage and sync real; only the external LLM response is synthetic.""" + monkeypatch.setenv("FAVA_TRAILS_DIR", str(tmp_fava_home / "trails")) + monkeypatch.setenv("FAVA_TRAILS_AGENT_ID", "synthetic-gateway") + monkeypatch.delenv("FAVA_TRAILS_OPERATOR", raising=False) + monkeypatch.setenv("FAVA_TRAILS_LOG_DIR", str(tmp_path / "logs")) + monkeypatch.setenv("SYNTHETIC_TRUST_GATE_KEY", "test-only-key") + (tmp_fava_home / "config.yaml").write_text( + "trails_dir: trails\ntrust_gate: llm-oneshot\npush_strategy: manual\n" + "trust_gate_api_key_env: SYNTHETIC_TRUST_GATE_KEY\n" + ) + (tmp_fava_home / "trails" / "trust-gate-prompt.md").write_text("Synthetic review policy.\n") + (tmp_fava_home / ".gitignore").write_text(".jj/\n") + + def git(*args): + return subprocess.run( + ["git", *args], cwd=tmp_fava_home, check=True, capture_output=True, text=True, + ) + + remote = tmp_path / "remote.git" + git("init", "--bare", str(remote)) + git("add", ".") + git("-c", "user.name=Synthetic Gateway", "-c", "user.email=synthetic@example.invalid", + "commit", "-m", "Synthetic gateway fixture") + git("remote", "add", "origin", str(remote)) + git("push", "origin", "HEAD:refs/heads/main") + review = AsyncMock(return_value=TrustResult( + verdict="approve", reasoning="Synthetic evaluator result", reviewer="synthetic-reviewer", + )) + monkeypatch.setattr(navigation, "review_thought", review) + monkeypatch.setattr(server, "_trail_managers", {}) + monkeypatch.setattr(server, "_trail_init_lock", None) + return tmp_fava_home, review + + +def http_rpc(client, version): + headers = {"Accept": "application/json, text/event-stream"} + next_id = 0 + + def rpc(method, params=None, *, notification=False): + nonlocal next_id + next_id += 1 + request = {"jsonrpc": "2.0", "method": method} + if params is not None: + request["params"] = params + if not notification: + request["id"] = next_id + response = client.post("/mcp/", json=request, headers=headers) + if "mcp-session-id" in response.headers: + headers["Mcp-Session-Id"] = response.headers["mcp-session-id"] + if notification: + assert response.status_code == 202, response.text + return None + assert response.status_code == 200, response.text + if "text/event-stream" in response.headers["content-type"]: + payload = json.loads(next( + line[6:] for line in response.text.splitlines() if line.startswith("data: ") + )) + else: + payload = response.json() + assert payload["id"] == next_id + assert "result" in payload, payload + return payload["result"] + + initialized = rpc("initialize", { + "protocolVersion": version, "capabilities": {}, + "clientInfo": {"name": "synthetic-http-gateway", "version": "1"}, + }) + assert initialized["protocolVersion"] == version + headers["MCP-Protocol-Version"] = version + rpc("notifications/initialized", notification=True) + return rpc + + +@pytest.mark.parametrize("version", ["2024-11-05", "2025-03-26", "2025-11-25"]) +def test_http_discover_draft_review_sync_and_governed_readback(gateway_repo, version): + repo, review = gateway_repo + scope = "synthetic/gateway" + with TestClient(create_streamable_http_app()) as client: + rpc = http_rpc(client, version) + listed = rpc("tools/list") + assert {"list_scopes", "save_thought", "propose_truth", "sync", "get_thought", "recall"} <= { + tool["name"] for tool in listed["tools"] + } + + def call(name, arguments): + result = rpc("tools/call", {"name": name, "arguments": arguments}) + assert not result.get("isError", False), result + assert json.loads(result["content"][0]["text"]) == result["structuredContent"] + return result["structuredContent"] + + invalid = rpc("tools/call", {"name": "save_thought", "arguments": {"content": "missing scope"}}) + assert invalid["isError"] is True + assert not (repo / "trails" / scope).exists() + assert call("list_scopes", {})["scopes"] == [] + assert call("recall", {"trail_name": scope})["status"] == "error" + assert not (repo / "trails" / scope).exists() + saved = call("save_thought", {"trail_name": scope, "content": "Synthetic gateway acceptance."}) + thought_id = saved["thought"]["thought_id"] + assert saved["thought"]["agent_id"] == "synthetic-gateway" + assert call("list_scopes", {"prefix": "synthetic"})["scopes"] == [{"path": scope}] + assert call("recall", {"trail_name": scope})["count"] == 0 + assert call("get_thought", {"trail_name": scope, "thought_id": thought_id})["status"] == "error" + owned = call("get_thought", {"trail_name": scope, "thought_id": thought_id, "mode": "authoring"}) + assert owned["thought"]["validation_status"] == "draft" + for name, arguments in ( + ("recall", {"mode": "history"}), + ("save_thought", {"content": "spoof", "agent_id": "another-agent"}), + ("rollback", {"op_id": "must-not-run"}), + ("forget", {}), + ("propose_truth", {"thought_id": thought_id, "approval": "human"}), + ): + denied = call(name, {"trail_name": scope, **arguments}) + assert denied["status"] == "error", denied + review.assert_not_awaited() + proposed = call("propose_truth", {"trail_name": scope, "thought_id": thought_id}) + assert proposed["status"] == "ok", proposed + assert proposed["thought"]["validation_status"] == "approved" + review.assert_awaited_once() + assert call("sync", {"trail_name": scope}) == {"status": "ok", "message": "Sync complete."} + recalled = call("recall", {"trail_name": scope}) + assert [thought["thought_id"] for thought in recalled["thoughts"]] == [thought_id] + assert call("get_thought", {"trail_name": scope, "thought_id": thought_id})["thought"]["validation_status"] == "approved" + + +def test_http_sync_refuses_dirty_repository_without_disclosing_paths(gateway_repo): + repo, _ = gateway_repo + with TestClient(create_streamable_http_app()) as client: + rpc = http_rpc(client, "2025-03-26") + saved = rpc("tools/call", {"name": "save_thought", "arguments": { + "trail_name": "synthetic/gateway", "content": "Synthetic sync fixture", + }}) + assert saved["structuredContent"]["status"] == "ok" + private_path = repo / "operator-private-fixture.txt" + private_path.write_text("Synthetic uncommitted bytes") + blocked = rpc("tools/call", {"name": "sync", "arguments": {"trail_name": "synthetic/gateway"}}) + assert not blocked.get("isError", False) + assert blocked["structuredContent"] == { + "status": "blocked", + "message": "Sync blocked by uncommitted repository changes. Operator attention is required.", + } + assert "operator-private-fixture" not in json.dumps(blocked) + assert private_path.read_text() == "Synthetic uncommitted bytes"