diff --git a/docs/tee/architecture.md b/docs/tee/architecture.md index b650ebd4..c5e798db 100644 --- a/docs/tee/architecture.md +++ b/docs/tee/architecture.md @@ -63,7 +63,7 @@ process holds it, and says what anchors it. Five long-lived processes, and the oneshots that prepare them, all measured into the image: -![the guest's processes and what reaches each one: clients through nginx, +![the guest's processes and what reaches each one: clients through Caddy, the operator on tdx-init and on the attestation service's harvest and status endpoints, peers on summit, reth, and the attestation service; the custodian holds root_key behind a unix socket; the setup units hand state to the @@ -78,7 +78,7 @@ survives a reboot](diagrams/node-processes.svg) | `summit-keygen`, `summit-persist` | Summit's consensus keypairs, as two oneshots and no process: one runs `summit keys generate` into tmpfs at boot, before any configuration exists, and the other copies the keys into summit's keystore once `/persistent` is mounted ([network founding](network-founding.md#summits-keys-before-luks)). | | `reth` | Execution: the EVM, Ethereum JSON-RPC, chain state, and the tx-gossip plane. | | `summit` | Consensus: BFT voting, checkpoints, and reth's forkchoice over the Engine API. | -| `nginx` | TLS termination and path routing to loopback backends. Its certificate key is generated in the guest and renewed by certbot. | +| `caddy` | The public HTTPS proxy: TLS termination and path routing to loopback backends, as an unprivileged user. It takes and renews its certificate itself, from Let's Encrypt or ZeroSSL, with a key generated in the guest. | Unit ordering, users, groups, and flag-by-flag rationale live with the image that ships them, in the @@ -161,7 +161,7 @@ sequenceDiagram participant SP as summit-persist
(oneshot) participant SU as summit participant RE as reth - participant NG as nginx + participant CA as caddy end Note over O: provision: pulumi up —
the box boots the measured image @@ -183,7 +183,7 @@ sequenceDiagram end O->>TI: node configure — POST manifest + reth genesis + summit genesis
(founding: the pinned box first) - TI-)NG: domain.env + TI-)CA: domain.env destroy TI TI-)AS: /run/seismic/conf, then its done marker Note over AS: :7879 now serves only /v1/keys,
/v1/quote answers 410 @@ -213,9 +213,11 @@ sequenceDiagram SP-)SU: keystore in
/persistent/summit/keys Note over SU: systemd starts summit,
which only reads the keystore SU->>RE: Engine API — drives forkchoice - Note over NG: systemd starts nginx-ssl-setup
once /persistent is mounted:
first boot: certbot takes the certificate
every boot: starts nginx - NG->>RE: proxies /rpc and /ws - Note over NG: nothing waits on it: a failed
certificate costs public HTTPS only + Note over CA: systemd starts caddy
once /persistent is mounted:
first boot: takes the certificate
then renews it in-process + CA->>RE: proxies /rpc, /ws, /metrics/reth + CA->>SU: proxies /summit, /metrics/summit + CA->>AS: proxies /attestation + Note over CA: nothing waits on it: a failed
certificate costs public HTTPS only opt founding only O->>AS: launch checks — GET :7879/v1/keys @@ -325,7 +327,7 @@ node's cloud firewall rules | Port | Source | What answers | | --- | --- | --- | -| `:443`, `:80` | anyone | nginx, proxying `/rpc` and `/ws` to reth, `/summit` to summit's JSON-RPC (health, checkpoints, staking and deposit queries), `/attestation` to the attestation service, and `/metrics/reth` + `/metrics/summit` to the two Prometheus endpoints | +| `:443` | anyone | Caddy, proxying `/rpc` and `/ws` to reth, `/summit` to summit's JSON-RPC (health, checkpoints, staking and deposit queries), `/attestation` to the attestation service, and `/metrics/reth` + `/metrics/summit` to the two Prometheus endpoints | | `:7878` | anyone | the attestation service directly: the root-key handshake, tx-io evidence, health, first-boot disk-provisioning progress, and admission-chain status | | `:18551` | anyone | summit's consensus plane | | `:30303` TCP+UDP | anyone | reth's devp2p plane | @@ -412,7 +414,7 @@ unrepresentable, and a node that must fetch `root_key` but is POSTed with no usable peer fails the POST rather than booting with no way to obtain it ([`peers.rs`](https://github.com/SeismicSystems/enclave/blob/seismic/bin/tdx-init/src/peers.rs)). -Browsers reach the attestation service through nginx's `/attestation` route +Browsers reach the attestation service through Caddy's `/attestation` route instead, since mixed-content rules bar a plain-HTTP fetch from an HTTPS origin. The TLS hop is hygiene, not a boundary: the evidence a browser fetches is verifiable on its own. @@ -578,7 +580,7 @@ per-VM keys persist inside LUKS](diagrams/key-families.svg) | summit BLS12-381 | per-VM random | born at boot in tmpfs by `summit-keygen`, persisted under `/persistent/summit/keys` | the harvest quote at founding; the validator set the manifest pins, or the deposit path afterwards | | summit Ed25519 | per-VM random | same | same, and it is the commonware-p2p peer id | | reth devp2p secp256k1 | per-VM random | generated by reth at `/discovery-secret`, inside LUKS | nothing — transport identity only; its public form is served by `seismic_nodeInfo` | -| nginx TLS certificate key | per-VM random | generated in the guest, stored with certbot's state inside LUKS | Let's Encrypt, renewed on a timer | +| TLS certificate key | per-VM random | generated in the guest by Caddy, stored in `/persistent/caddy` inside LUKS | Let's Encrypt, or ZeroSSL when that fails; renewed by Caddy | The per-VM keys are stable for the life of the node because they sit on the LUKS volume — the enode a peer dialed yesterday still answers today. @@ -684,7 +686,7 @@ with the responder still attesting the joiner. ## `/persistent`: the LUKS volume One LUKS2 volume holds everything a node keeps across boots: reth's datadir, -summit's keystore and database, and certbot's certificates. Its unlock key is +summit's keystore and database, and Caddy's certificate. Its unlock key is derived from `root_key`, so the volume is readable only by a guest that the network has admitted. No human and no cloud operator ever holds the unlock material, and there is no TPM seal to migrate or scrub. @@ -797,6 +799,18 @@ under its own sandbox, and systemd already expresses the ordering. A long-running process kept only to carry state between phases costs a lifecycle, an IPC surface and a crash mode for something a tmpfs file does. +**An unprivileged Caddy rather than nginx and certbot** +([what the outside can reach](#what-the-outside-can-reach)). In-guest root +defeats every access control on the node: root can read `root_key` out of the +custodian's memory, whatever the socket's ACL or the units' sandboxing. So the +lever left is running as little as possible as root, and the public proxy is +the most exposed process on the node while nothing in its job needs +privilege: binding `:443` takes `CAP_NET_BIND_SERVICE` alone. nginx runs its +master as root, and certbot is a root Python process that talks to the network +and rewrites the proxy's config. Caddy takes and renews its certificate in the +same unprivileged process that serves, from a static config in the measured +image, so every node runs the config its measurement names. + **`root_key` in RAM only, with no on-disk backup** ([key custody](#key-custody-one-process-holds-root_key)). Sealing it to the platform would buy restart independence at the cost of a second sealing policy diff --git a/docs/tee/diagrams/key-families.svg b/docs/tee/diagrams/key-families.svg index 51fdf53d..6c7a98f2 100644 --- a/docs/tee/diagrams/key-families.svg +++ b/docs/tee/diagrams/key-families.svg @@ -103,14 +103,14 @@ - nginx TLS certificate key + caddy TLS certificate key /persistent — LUKS2 - reth datadir · summit keys and db · certbot state + reth datadir · summit keys and db · caddy certificate @@ -167,7 +167,7 @@ - nginx TLS cert key + caddy TLS cert key same kinds of keys — its own randomness diff --git a/docs/tee/diagrams/node-processes.svg b/docs/tee/diagrams/node-processes.svg index 4f581e1c..5f93ac5f 100644 --- a/docs/tee/diagrams/node-processes.svg +++ b/docs/tee/diagrams/node-processes.svg @@ -66,7 +66,7 @@ - nginx :443 + caddy :443 the only TLS terminator @@ -146,7 +146,7 @@ /persistent — LUKS2 - reth datadir · summit keystore and db · certbot state + reth datadir · summit keystore and db · caddy certificate diff --git a/docs/tee/network-founding.md b/docs/tee/network-founding.md index 18f22cd8..b9c9eeb1 100644 --- a/docs/tee/network-founding.md +++ b/docs/tee/network-founding.md @@ -166,7 +166,7 @@ while a restart of `summit.service` alone does not. because nothing deletes the tmpfs keys before the next reboot empties tmpfs. - **The attestation service serves the harvest on `:7879`.** It starts at boot, and until the POST arrives this is all it serves. The port is plain HTTP: - nginx and TLS certificates exist only after the POST, and deploy tooling + Caddy and its TLS certificate exist only after the POST, and deploy tooling already polls raw ports during first boot. `GET /v1/keys` returns both pubkeys from the public-keys file. `GET /v1/quote?nonce=…` adds the custodian's candidate `tx_io_pk@0`, from the candidate file the custodian diff --git a/docs/tee/trust-model.md b/docs/tee/trust-model.md index 07cc9873..0693a555 100644 --- a/docs/tee/trust-model.md +++ b/docs/tee/trust-model.md @@ -317,7 +317,7 @@ detects the rewind. The instances: - **LUKS is tamper-evident, not rollback-protected.** The header MAC and dm-integrity refuse an *edited* volume, but a snapshot of the whole volume is internally consistent, MAC and all, and restores cleanly. Everything - under `/persistent` — reth's datadir, summit's database, certbot state — + under `/persistent` — reth's datadir, summit's database, Caddy's certificate — can be rewound together. - **A chain view can be held at block 0.** That lands the responder's admission gate on the founding policy, where no timestamp check bites and "still at diff --git a/tee/cli/common/src/http.rs b/tee/cli/common/src/http.rs index 0be63024..e53934c0 100644 --- a/tee/cli/common/src/http.rs +++ b/tee/cli/common/src/http.rs @@ -3,7 +3,8 @@ //! A deployed node is a sealed VM: there is no SSH, so every interaction is an //! HTTP or JSON-RPC call to one of the ports below. Two of them are //! operator-only and reachable only from the CIDR the node's NSG pins -//! (`operator_ip_cidr`); the RPC is public, behind nginx and a real cert. +//! (`operator_ip_cidr`); the RPC is public, behind the node's HTTPS proxy and a +//! real cert. use std::time::Duration; @@ -11,7 +12,7 @@ use crate::descriptor::NodeDescriptor; use crate::error::Result; /// tdx-init's one-shot config receiver, up only until a boot's POST is -/// consumed. Plain HTTP: it runs before certbot has issued anything. +/// consumed. Plain HTTP: it runs before the node has a certificate. pub const TDX_INIT_PORT: u16 = 8080; /// The attestation service's JSON-RPC: LUKS provisioning status, deploy /// verification evidence, the root-key handshake. @@ -79,7 +80,7 @@ impl NodeDescriptor { format!("http://{}:{HARVEST_PORT}", self.public_ip) } - /// The node's public Ethereum JSON-RPC: nginx proxies `/rpc` to reth, + /// The node's public Ethereum JSON-RPC: its HTTPS proxy sends `/rpc` to reth, /// which is why this one goes to the FQDN over TLS rather than to the IP. pub fn eth_rpc_url(&self) -> String { format!("https://{}/rpc", self.fqdn) diff --git a/tee/cli/node/src/bootnodes.rs b/tee/cli/node/src/bootnodes.rs index 35e18779..0ef6dd8d 100644 --- a/tee/cli/node/src/bootnodes.rs +++ b/tee/cli/node/src/bootnodes.rs @@ -8,7 +8,7 @@ //! beside the descriptors as `nodes/bootnodes.json`. //! //! `seismic_nodeInfo` is read over the node's public JSON-RPC -//! (`https:///rpc`, nginx in front of reth) with the same client the +//! (`https:///rpc`, the HTTPS proxy in front of reth) with the same client the //! attestation service is read with ([`rpc::Client`]); the `seismic` namespace //! deliberately keeps nodeInfo public (reth's `admin` namespace is disabled on //! these nodes). diff --git a/tee/cli/node/src/configure.rs b/tee/cli/node/src/configure.rs index 0f2e0099..08410517 100644 --- a/tee/cli/node/src/configure.rs +++ b/tee/cli/node/src/configure.rs @@ -465,7 +465,7 @@ pub fn print_summary(fqdn: &str, public_ip: &str, record: &Path) { println!("\nNode: {fqdn}"); println!("IP Address: {public_ip}"); println!("Config: {} (as POSTed)", record.display()); - println!("\nNginx + SSL set up automatically after initialization."); + println!("\nThe node takes its TLS certificate itself after initialization."); println!("Endpoints (once the node settles):"); println!(" https://{fqdn}/rpc"); println!(" https://{fqdn}/ws"); @@ -553,7 +553,7 @@ pub struct ConfigureArgs { )] pub bootnode: Vec, - /// Contact email for the node's Let's Encrypt registration (certbot); goes + /// Contact email for the node's Let's Encrypt registration; goes /// into [node.domain].email of the POSTed config. Same across a cohort. #[arg(long, default_value = DEFAULT_EMAIL)] pub email: String,