Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 25 additions & 11 deletions docs/tee/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ process holds it, and says what anchors it.
Five long-lived processes, and the oneshots that prepare them, all measured
into the image:

![the guest's processes and what reaches each one: clients through nginx,
![the guest's processes and what reaches each one: clients through Caddy,
the operator on tdx-init and on the attestation service's harvest and status
endpoints, peers on summit, reth, and the attestation service; the custodian
holds root_key behind a unix socket; the setup units hand state to the
Expand All @@ -78,7 +78,7 @@ survives a reboot](diagrams/node-processes.svg)
| `summit-keygen`, `summit-persist` | Summit's consensus keypairs, as two oneshots and no process: one runs `summit keys generate` into tmpfs at boot, before any configuration exists, and the other copies the keys into summit's keystore once `/persistent` is mounted ([network founding](network-founding.md#summits-keys-before-luks)). |
| `reth` | Execution: the EVM, Ethereum JSON-RPC, chain state, and the tx-gossip plane. |
| `summit` | Consensus: BFT voting, checkpoints, and reth's forkchoice over the Engine API. |
| `nginx` | TLS termination and path routing to loopback backends. Its certificate key is generated in the guest and renewed by certbot. |
| `caddy` | The public HTTPS proxy: TLS termination and path routing to loopback backends, as an unprivileged user. It takes and renews its certificate itself, from Let's Encrypt or ZeroSSL, with a key generated in the guest. |

Unit ordering, users, groups, and flag-by-flag rationale live with the image
that ships them, in the
Expand Down Expand Up @@ -161,7 +161,7 @@ sequenceDiagram
participant SP as summit-persist<br/>(oneshot)
participant SU as summit
participant RE as reth
participant NG as nginx
participant CA as caddy
end

Note over O: provision: pulumi up —<br/>the box boots the measured image
Expand All @@ -183,7 +183,7 @@ sequenceDiagram
end

O->>TI: node configure — POST manifest + reth genesis + summit genesis<br/>(founding: the pinned box first)
TI-)NG: domain.env
TI-)CA: domain.env
destroy TI
TI-)AS: /run/seismic/conf, then its done marker
Note over AS: :7879 now serves only /v1/keys,<br/>/v1/quote answers 410
Expand Down Expand Up @@ -213,9 +213,11 @@ sequenceDiagram
SP-)SU: keystore in<br/>/persistent/summit/keys
Note over SU: systemd starts summit,<br/>which only reads the keystore
SU->>RE: Engine API — drives forkchoice
Note over NG: systemd starts nginx-ssl-setup<br/>once /persistent is mounted:<br/>first boot: certbot takes the certificate<br/>every boot: starts nginx
NG->>RE: proxies /rpc and /ws
Note over NG: nothing waits on it: a failed<br/>certificate costs public HTTPS only
Note over CA: systemd starts caddy<br/>once /persistent is mounted:<br/>first boot: takes the certificate<br/>then renews it in-process
CA->>RE: proxies /rpc, /ws, /metrics/reth
CA->>SU: proxies /summit, /metrics/summit
CA->>AS: proxies /attestation
Note over CA: nothing waits on it: a failed<br/>certificate costs public HTTPS only

opt founding only
O->>AS: launch checks — GET :7879/v1/keys
Expand Down Expand Up @@ -325,7 +327,7 @@ node's cloud firewall rules

| Port | Source | What answers |
| --- | --- | --- |
| `:443`, `:80` | anyone | nginx, proxying `/rpc` and `/ws` to reth, `/summit` to summit's JSON-RPC (health, checkpoints, staking and deposit queries), `/attestation` to the attestation service, and `/metrics/reth` + `/metrics/summit` to the two Prometheus endpoints |
| `:443` | anyone | Caddy, proxying `/rpc` and `/ws` to reth, `/summit` to summit's JSON-RPC (health, checkpoints, staking and deposit queries), `/attestation` to the attestation service, and `/metrics/reth` + `/metrics/summit` to the two Prometheus endpoints |
| `:7878` | anyone | the attestation service directly: the root-key handshake, tx-io evidence, health, first-boot disk-provisioning progress, and admission-chain status |
| `:18551` | anyone | summit's consensus plane |
| `:30303` TCP+UDP | anyone | reth's devp2p plane |
Expand Down Expand Up @@ -412,7 +414,7 @@ unrepresentable, and a node that must fetch `root_key` but is POSTed with no
usable peer fails the POST rather than booting with no way to obtain it
([`peers.rs`](https://github.com/SeismicSystems/enclave/blob/seismic/bin/tdx-init/src/peers.rs)).

Browsers reach the attestation service through nginx's `/attestation` route
Browsers reach the attestation service through Caddy's `/attestation` route
instead, since mixed-content rules bar a plain-HTTP fetch from an HTTPS origin.
The TLS hop is hygiene, not a boundary: the evidence a browser fetches is
verifiable on its own.
Expand Down Expand Up @@ -578,7 +580,7 @@ per-VM keys persist inside LUKS](diagrams/key-families.svg)
| summit BLS12-381 | per-VM random | born at boot in tmpfs by `summit-keygen`, persisted under `/persistent/summit/keys` | the harvest quote at founding; the validator set the manifest pins, or the deposit path afterwards |
| summit Ed25519 | per-VM random | same | same, and it is the commonware-p2p peer id |
| reth devp2p secp256k1 | per-VM random | generated by reth at `<datadir>/discovery-secret`, inside LUKS | nothing — transport identity only; its public form is served by `seismic_nodeInfo` |
| nginx TLS certificate key | per-VM random | generated in the guest, stored with certbot's state inside LUKS | Let's Encrypt, renewed on a timer |
| TLS certificate key | per-VM random | generated in the guest by Caddy, stored in `/persistent/caddy` inside LUKS | Let's Encrypt, or ZeroSSL when that fails; renewed by Caddy |

The per-VM keys are stable for the life of the node because they sit on the LUKS
volume — the enode a peer dialed yesterday still answers today.
Expand Down Expand Up @@ -684,7 +686,7 @@ with the responder still attesting the joiner.
## `/persistent`: the LUKS volume

One LUKS2 volume holds everything a node keeps across boots: reth's datadir,
summit's keystore and database, and certbot's certificates. Its unlock key is
summit's keystore and database, and Caddy's certificate. Its unlock key is
derived from `root_key`, so the volume is readable only by a guest that the
network has admitted. No human and no cloud operator ever holds the unlock
material, and there is no TPM seal to migrate or scrub.
Expand Down Expand Up @@ -797,6 +799,18 @@ under its own sandbox, and systemd already expresses the ordering. A
long-running process kept only to carry state between phases costs a
lifecycle, an IPC surface and a crash mode for something a tmpfs file does.

**An unprivileged Caddy rather than nginx and certbot**
([what the outside can reach](#what-the-outside-can-reach)). In-guest root
defeats every access control on the node: root can read `root_key` out of the
custodian's memory, whatever the socket's ACL or the units' sandboxing. So the
lever left is running as little as possible as root, and the public proxy is
the most exposed process on the node while nothing in its job needs
privilege: binding `:443` takes `CAP_NET_BIND_SERVICE` alone. nginx runs its
master as root, and certbot is a root Python process that talks to the network
and rewrites the proxy's config. Caddy takes and renews its certificate in the
same unprivileged process that serves, from a static config in the measured
image, so every node runs the config its measurement names.

**`root_key` in RAM only, with no on-disk backup**
([key custody](#key-custody-one-process-holds-root_key)). Sealing it to the
platform would buy restart independence at the cost of a second sealing policy
Expand Down
6 changes: 3 additions & 3 deletions docs/tee/diagrams/key-families.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/tee/diagrams/node-processes.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/tee/network-founding.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ while a restart of `summit.service` alone does not.
because nothing deletes the tmpfs keys before the next reboot empties tmpfs.
- **The attestation service serves the harvest on `:7879`.** It starts at boot,
and until the POST arrives this is all it serves. The port is plain HTTP:
nginx and TLS certificates exist only after the POST, and deploy tooling
Caddy and its TLS certificate exist only after the POST, and deploy tooling
already polls raw ports during first boot. `GET /v1/keys` returns both
pubkeys from the public-keys file. `GET /v1/quote?nonce=…` adds the
custodian's candidate `tx_io_pk@0`, from the candidate file the custodian
Expand Down
2 changes: 1 addition & 1 deletion docs/tee/trust-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,7 @@ detects the rewind. The instances:
- **LUKS is tamper-evident, not rollback-protected.** The header MAC and
dm-integrity refuse an *edited* volume, but a snapshot of the whole volume
is internally consistent, MAC and all, and restores cleanly. Everything
under `/persistent` — reth's datadir, summit's database, certbot state —
under `/persistent` — reth's datadir, summit's database, Caddy's certificate —
can be rewound together.
- **A chain view can be held at block 0.** That lands the responder's admission
gate on the founding policy, where no timestamp check bites and "still at
Expand Down
7 changes: 4 additions & 3 deletions tee/cli/common/src/http.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,16 @@
//! A deployed node is a sealed VM: there is no SSH, so every interaction is an
//! HTTP or JSON-RPC call to one of the ports below. Two of them are
//! operator-only and reachable only from the CIDR the node's NSG pins
//! (`operator_ip_cidr`); the RPC is public, behind nginx and a real cert.
//! (`operator_ip_cidr`); the RPC is public, behind the node's HTTPS proxy and a
//! real cert.

use std::time::Duration;

use crate::descriptor::NodeDescriptor;
use crate::error::Result;

/// tdx-init's one-shot config receiver, up only until a boot's POST is
/// consumed. Plain HTTP: it runs before certbot has issued anything.
/// consumed. Plain HTTP: it runs before the node has a certificate.
pub const TDX_INIT_PORT: u16 = 8080;
/// The attestation service's JSON-RPC: LUKS provisioning status, deploy
/// verification evidence, the root-key handshake.
Expand Down Expand Up @@ -79,7 +80,7 @@ impl NodeDescriptor {
format!("http://{}:{HARVEST_PORT}", self.public_ip)
}

/// The node's public Ethereum JSON-RPC: nginx proxies `/rpc` to reth,
/// The node's public Ethereum JSON-RPC: its HTTPS proxy sends `/rpc` to reth,
/// which is why this one goes to the FQDN over TLS rather than to the IP.
pub fn eth_rpc_url(&self) -> String {
format!("https://{}/rpc", self.fqdn)
Expand Down
2 changes: 1 addition & 1 deletion tee/cli/node/src/bootnodes.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
//! beside the descriptors as `nodes/bootnodes.json`.
//!
//! `seismic_nodeInfo` is read over the node's public JSON-RPC
//! (`https://<fqdn>/rpc`, nginx in front of reth) with the same client the
//! (`https://<fqdn>/rpc`, the HTTPS proxy in front of reth) with the same client the
//! attestation service is read with ([`rpc::Client`]); the `seismic` namespace
//! deliberately keeps nodeInfo public (reth's `admin` namespace is disabled on
//! these nodes).
Expand Down
4 changes: 2 additions & 2 deletions tee/cli/node/src/configure.rs
Original file line number Diff line number Diff line change
Expand Up @@ -465,7 +465,7 @@ pub fn print_summary(fqdn: &str, public_ip: &str, record: &Path) {
println!("\nNode: {fqdn}");
println!("IP Address: {public_ip}");
println!("Config: {} (as POSTed)", record.display());
println!("\nNginx + SSL set up automatically after initialization.");
println!("\nThe node takes its TLS certificate itself after initialization.");
println!("Endpoints (once the node settles):");
println!(" https://{fqdn}/rpc");
println!(" https://{fqdn}/ws");
Expand Down Expand Up @@ -553,7 +553,7 @@ pub struct ConfigureArgs {
)]
pub bootnode: Vec<String>,

/// Contact email for the node's Let's Encrypt registration (certbot); goes
/// Contact email for the node's Let's Encrypt registration; goes
/// into [node.domain].email of the POSTed config. Same across a cohort.
#[arg(long, default_value = DEFAULT_EMAIL)]
pub email: String,
Expand Down
Loading