| layout | default |
|---|---|
| title | Security |
This is the primary security overview for Lemonade-Nexus. It explains the roles, how authority is established and renewed, what the protocol protects against, and what it does not yet protect.
- Identity and Cryptography
- Server Roles
- Epoch Authority
- Genesis and Trust Continuity
- Compiled Rules
- Attestation and Key Boundaries
- Threat Assumptions
- Transport and Message Security
- Permission Tree Administration
- Application Authority Gaps
- Current Limitations
- Retired Mechanisms
| Primitive | Algorithm | Purpose |
|---|---|---|
| Identity keys | Ed25519 | Server and client identity, signing |
| Mesh keys | X25519 (Curve25519) | Tunnel encryption (derived from the Ed25519 identity) |
| Consensus votes | Ed25519 (per-epoch vote keys) | Chained HotStuff voting |
| Authority signatures | FROST(Ed25519, SHA-512) | Threshold signing by the epoch committee |
| Symmetric AEAD | XChaCha20-Poly1305-IETF | Encrypted objects at rest and in transfer |
| Key derivation | HKDF-SHA256 | Derive encryption keys from shared secrets |
| Hashing | SHA-256 / SHA-512 | Digests, canonical encoding, challenge binding |
| TLS | OpenSSL | HTTPS for the public and private APIs |
Every encrypted object under Nexus control uses one construction:
EncryptedBlob {
version // 1
nonce // exactly 24 bytes, random per encryption
ciphertext // includes the 16-byte Poly1305 tag
}
Version 1 is always XChaCha20-Poly1305-IETF. The reader checks the version and
the exact nonce length before any bytes reach the AEAD; unknown versions fail
closed. Objects carrying this format include the wrapped node identity
(identity/keypair.enc), ACL values, epoch vote keys, DDNS credentials, and
the issued-certificate bundle.
Authentication for clients uses Ed25519 challenge-response (primary) and WebAuthn passkeys (backup). Password authentication is a deprecated server stub that always fails.
| Role | What establishes it | Capabilities |
|---|---|---|
| Tier 2 | A valid network-bound server certificate and authenticated mesh participation | A pure proxy of end-to-end encrypted data and connections: relay, STUN, discovery DNS, and the security transport. Holds no permission tree, serves no tree route or client session, and forwards no packet between dataplane peers. No Tier 1 authority share. |
| Pending member | Selection in a finalized next-epoch plan | State synchronization and readiness preparation. No active Tier 1 authority. |
| Tier 1 | Verified platform evidence and mesh observations establish eligibility; the protocol's finalized eligibility, plan, readiness, and handoff establish the role | HotStuff consensus, epoch transitions, dealerless DKG participation, threshold authority signing. Holds and serves the permission tree and client sessions. |
Tier 1 is an epoch-scoped role. A transport connection or a valid certificate never grants it, and a pending candidate holds preparation rights only — proving participation grants no authority before the finalized handoff.
Clients use the mesh under their application permissions. They do not become members of the server authority committee.
The permission tree follows finalized epoch membership, never configuration
or a peer's word (Tree/TreeRole.hpp):
- A current Tier 1 member loads the tree and serves the tree routes,
client sessions (
/api/auth*,/api/join), and endpoint forwarding. - Any other server refuses every tree route with
421and the body{"error":"this server does not serve the tree"}: no redirect, no other server named, and no tree data (Api/TreeRouteGate.hpp). It runs no tree replication, no record transfer pool, and no tree log, takes no replicated tree entry, and keeps no tree file. - A server that leaves Tier 1 closes the gate, stops those services, and
deletes its tree copy in the tick in which finalized state shows that its
last epoch ended. Each file under
data/tree/is overwritten with zeros, flushed, and removed. This is best effort: a journaling or copy-on-write file system, snapshots, or SSD remapping can keep older blocks. - A server whose membership is not known (before Epoch 1 activates, or before its persisted state verifies) serves no tree and deletes nothing. Unknown is not evidence that it left Tier 1.
A server that enters Tier 1 gets the tree through Tier 1 tree log replication. Tree operations are the only tree write path; no server changes its tree outside the committed log.
Servers that wrote their trees before tree operations hold different trees. The Tree Merge Ceremony commits one merged tree as tree log height 1, once. Its security rules:
- Authority is the root management key's signature. Every server
verifies the merge commit against its own configured
root_pubkeyand the network id from its own Genesis key, whoever delivers it (the"tree"stream, orPOST /api/tree/merge/commit). The deliverer is not trusted. - One-shot. The commit is valid only as height 1 on the empty base. A server whose log holds height 1 refuses another merge; a merge entry at any other height does not verify. No command replaces the tree after that.
- Nothing takes height 1 first. A server that holds a pre-merge tree opens
its log on the empty base, refuses tree writes (
503, retryable), proposes nothing, withholds its vote from every tree block, requests no root, and refuses every consensus entry. It applies only the merge commit. - Signed, complete inputs. A Tier 1 member gives its export only for a
request that the root management key signed for that member, issued within
10 minutes, with a nonce it has not served. It signs the export with its
gossip identity under
lemonade-nexus/tree-export-signature:v1, bound to the network. The root server merges only when the signers are exactly the current Tier 1 members by finalized membership, each once, plus at most its own backup export, which only it may mark as the root server's. Without a backup the merged root is the root management key's application root. - Empty start. A Tier 1 server with no tree log and an empty tree awaits
no merge. Tier 1 commits the
EstablishRootoperation for the configuredroot_pubkeyas height 1; replicas accept it only for their own configured key, only while no root exists, and only through consensus. - Deterministic merge. The merge rule (
tree::merge_tree_exports, TreeMerge.hpp) depends only on the exports. It always decides, and it reports every tie-break and every dropped node. - Narrow signing. On the root server's control socket (root peers only) the root management key signs two document types: an export request for a current member, and the preimage of a merge that the server built itself from verified exports, and only when the rebuilt tree digest equals the reviewed dry run.
- Keys stay in the process. The data key reads the root server's backup
(
StorageVault::copy_view, read-only, same relative paths, current data key only). The script holds no key.
Limits:
- The root holder reviews the dry run. The merge rule decides by evidence on disk (IPAM records, the delta log) and then by the tie-break; it cannot know which of two differing copies a user meant.
- Dropped nodes (shared identifiers, nodes that do not reach the root) are gone after the merge; those devices join again.
- The script reaches Tier 1 members over their public HTTPS API. For an address URL it checks the certificate chain but not the name. The exports stay authenticated by their signatures; their confidentiality against an active attacker on that path then rests on the chain check only.
- A backup whose files are under an earlier data key (before a monthly rotation) does not open.
Clients send every tree read and write directly to a Tier 1 server. The SDK verifies the membership from its pinned Genesis key and accepts a server only after it signs an endpoint proof for the host and port the client reached (see Client SDK). A server never vouches for a session it did not issue.
A client with no stored Genesis key can discover it on first connect from
GET /api/onboard/info. The SDK pins the key only when at least 3 hosts in the
network DNS domain report the same key and network id over verified HTTPS.
This is trust on first use: it rests on domain ownership and host agreement,
not on a signature. After the key is pinned, discovery never replaces it.
A Tier 1 server issues a relay ticket (POST /api/relay/ticket) only for the
caller's own session. It signs the ticket with its node identity key, the key
the finalized membership names, under lemonade-nexus/relay-ticket:v2. The
ticket names the network, the issuer's epoch, the session, the relay, a fresh
nonce, and an expiry at most 120 seconds after issue. Any relay accepts it
only when the issuer is a member of the Tier 1 membership the relay verified
from the authority chain, the network and epoch match, the ticket names this
relay by its base64 node identity key, the signature verifies, and it has not
expired (Relay/RelayTicketAuth.hpp). A relay with no relay id refuses every
ticket. Each issuer and nonce allocates one session; the relay keeps that
record until the ticket expires. The relay needs no tree and no session
state, so a Tier 2 relay carries ciphertext for any session a Tier 1 server
authorized. A ticket from the previous epoch is refused after an epoch change
and must be requested again.
Two peers that cannot reach each other directly share one relay session
(Relay/RelayService.hpp, wire format in Relay/RelayWire.hpp). Each peer
sends the relay an Allocate request with its own ticket and its rendezvous
material. A peer derives that material from the X25519 secret it shares with
the other peer, the relay id, both mesh public keys, and a 300-second window.
Only the two peers can derive it. The material is a rendezvous id, the
sender's proof, and a commitment to the other side's proof.
- The first request creates a waiting session bound to the sender's endpoint.
- The relay joins a second request to it only when each request opens the commitment that the other made, the two tickets name different holders, and the two requests come from different endpoints. A copy of the first request with another ticket is refused.
- A side may return with a fresh ticket of its own holder, from a new endpoint (a device that restarted or resumed from suspension). The relay moves that side of the waiting or paired session to the new endpoint. A Tier 1 server issues a ticket only for the caller's own identity, and each ticket allocates once, so a copy of a side's earlier request cannot move it.
- The relay forwards data only between the two bound endpoints. It drops data from any other sender, and data that starts with the relay magic, so a peer cannot forge relay control messages for the other peer.
- Only a bound endpoint gets heartbeat answers or may end the session.
- A ticket allocates once. A retransmission from the same endpoint learns the same session; the same ticket from anywhere else is refused.
RelayLimits bounds every table: at most 4096 sessions, 16 sessions for each
ticket holder, 16384 used-ticket records, and 6000 Allocate checks per
minute across all senders. The rate check runs before the ticket signature
check. A session that waits 30 seconds for its second side, is idle for 90
seconds, or reaches one hour is dropped. The peers then allocate again with
new tickets, so their authorization is checked again.
The relay sees only BoringTun ciphertext; the BoringTun session between the peers authenticates every packet. An attacker on the network path of the second request who holds a valid ticket can race that request to the relay. The attacker then receives ciphertext it cannot read, and the BoringTun handshake fails. This is a denial of service on that relay session, which an on-path attacker can cause anyway.
The SDK punches direct paths between mesh peers (HolePunch.hpp). No new
cryptography is involved, and a punch grants no authority:
- Probe targets come from the authenticated tree. A peer's endpoint comes
only from the verified Tier 1 server's peer list (
GET /api/mesh/peers), which lists only peers the caller may connect to. The SDK never takes a probe target from a received packet. It ignores a listed endpoint that is not a dotted IPv4 unicast address, and it accepts a loopback endpoint only when the session server is on loopback. - A probe changes no state. A probe is a normal BoringTun handshake
initiation sent to the listed endpoint (
UserspaceDataplane::probe_endpoint); the peer's stored endpoint stays as it is. Only a valid handshake answer from the probed address moves the peer there (the dataplane's existing authenticated roaming). The handshake authenticates the peer by its mesh key, so an attacker who answers in its place cannot complete it. - STUN is a hint. STUN answers are not authenticated. The dataplane hands
STUN messages to the reflexive-address query and never to a Noise session.
The query accepts an answer only from the queried server, with its own
random transaction id, and with a valid IPv4
XOR-MAPPED-ADDRESS(Network/StunWire.hpp, shared with the server'sStunService). The SDK publishes the address only when the STUN server could have observed it (reflexive_publishable). A forged answer can at worst publish a wrong endpoint: peers then probe a dead address and fall back to a relay. - The server stores what the node's owner reports. The heartbeat writes
the endpoint only for the caller's own node, as a
SetEndpointtree operation when tree operations are on. It refuses IPv4 addresses that are not unicast. An empty endpoint keeps the stored one. - LAN endpoints go only to authorized peers. The heartbeat may carry
local_endpoints: up to four of the device's private IPv4 addresses (10/8, 172.16/12, 192.168/16, and 100.64/10 only when STUN saw that very address) with its dataplane port. The server accepts them only from the node's own management key on the private API, refuses public, loopback (unless the server itself is on loopback), link-local, and malformed entries, and keeps them in memory for 120 seconds, not in the tree. They travel with the device's presence to the other Tier 1 servers (PresenceSync, backbone only, Tier 1 to Tier 1); a receiving server applies the same rules again without the loopback exception and refuses the whole message on a failing entry, since the sender is a server and not the device. Every server returns them only in theGET /api/mesh/peersentries of callers that already list the node, that is, peers that may connect to it. The public API never returns them, and an unauthenticated caller never sees them. What this discloses: the private addresses of a device's LAN interfaces to the peers that may already open a BoringTun session with it and that already see its public endpoint. A private address is not routable from the internet, and it reveals little more than the public endpoint does. The SDK skips VPN, tunnel, container, and cellular interfaces, so their addresses are not disclosed. - A LAN endpoint only changes where packets go. The SDK probes a peer's
LAN endpoints only when they are private addresses and the peer is on its
own network (the same public address, or a LAN endpoint inside one of its
subnets;
lan_probe_targets). A probe is the same BoringTun handshake initiation, sent to the LAN endpoints and the listed endpoint at once (UserspaceDataplane::probe_endpoints). The peer moves to a LAN address only when an authenticated handshake answers from it, so a device that claims a LAN address cannot impersonate the peer: it does not hold the peer's static key. Handshakes from addresses that are not listed are handled as before (authenticated roaming). A false LAN endpoint can at worst aim a few handshake initiations at a private address on the victim's network, at the rate limits below, and delay the relay by the 1.5-second LAN window. - A change notice adds no peer. The heartbeat answer's
peers_revisionis an opaque hash, present only when the caller holds Read on its node. A changed value only makes the SDK fetch the verified peer list sooner; it names no peer and no address. - A peer list long-poll is bounded.
wait_revisionneeds Read on the node, like the list itself, and the answer is built after the wait under the same checks. A server holds at most 256 waiting requests in all and 2 per device key, and refuses more with429andRetry-After. A wait lasts at most 30 seconds, and a revision longer than 64 characters is refused. The private API has enough worker threads for every held wait plus the normal requests. - An event socket is bounded and carries no authority.
GET /api/eventsgets the checks of the peer list (a valid session, Read on the named node) before the WebSocket upgrade, and a refused request does not switch protocols. The server checks the session and the grant again on every pass and closes the socket with 1008 when either ends. A socket counts against the same caps as a long-poll (256 in all, 2 per device key) and closes with 1013 beyond them. A message carries only two opaque revisions: it names no device and grants nothing, and the device reads its lists through the routes under their own checks. - Presence is a hint, not authority.
is_onlineandlast_seenin the peer list come from device presence (gossip::DevicePresence). Only the node's own session makes a device present: a heartbeat that passes the ownership check, or an event socket whose session subject is the named node. A socket that only holds Read on another device's node watches it and does not hold it online. Tier 1 servers share presence withPresenceSync, accepted only on the encrypted backbone, only from a certified current Tier 1 member, and only by a Tier 1 server. A message holds at most 256 entries and 16 KiB, and a malformed one is refused whole. A time more than 5 seconds ahead of the receiver's clock is ignored, so a sender cannot hold a device online with a future time. Each server sends only what it heard itself, so presence is never relayed. No authorization decision reads presence. - Public counts reveal network size, nothing else.
GET /api/statsneeds no session. It reports how many devices, groups and relays the tree holds and how many devices are online, with no id, key, name, or address. Anyone can therefore watch the network's size and, in a small network, when a device comes online or goes offline. - Probes are rate limited. At most 16 peers are probed per pass, each probe goes to at most five addresses (four LAN endpoints and the listed endpoint), and passes are at least 250 ms apart. A peer gets a burst of five probes, then two probes per 15-second slot, or per 60-second slot while it is relayed. A peer with a fresh direct handshake gets none. A new burst starts only when the peer's listed endpoint or LAN endpoints change, its LAN path goes silent, the peer comes back online at the server, the application requests a refresh, the peer list changes at the server, or the device resumes from a suspension.
Endpoints, the devices that join a group, come in two roles. Neither holds any server or authority role:
| Role | What establishes it | Capabilities |
|---|---|---|
| Tier 3 | An endpoint that joins with ln_set_inference_node(client, 1) (the Lemonade server, for example). The server records is_inference when it creates the endpoint, and the routing identifier starts with infer-. |
Serves inference to the group members its permissions allow. |
| Tier 4 | Any other endpoint (the default). Its routing identifier starts with client-. |
Uses the group's inference nodes and other services under its permissions. |
Which members may reach a Tier 3 node is decided by the permission tree, as for any endpoint. The role is set when the endpoint is created; a device that already joined keeps its role.
Two device permissions control reachability. read lets a member see a
device in the tree. connect_private or connect_shared lets a member
connect to it. The mesh peer list (GET /api/mesh/peers/{node_id}) and the
routing layer use the same rule: a device in the member's group is listed or
resolved only when the member holds a connect permission on that device
itself. admin includes both connect permissions. A grant on the group does
not count, because permissions are not inherited. Relays in the group are
listed with read.
Each peer entry in the list carries mgmt_pubkey, the peer node's management
public key as the tree stores it, and peer_permissions: the sorted,
de-duplicated names that this key holds on the requesting device and on its
ancestor groups, plus admin when the peer administers the device
(tree::rules::peer_permissions). The two fields let the device attribute a
connecting peer to its key and tell the device what that peer may do on it,
including service permissions (see "Service permissions" below), without
reading group entries it cannot see. The fields report; they add no peer to
the list and grant nothing on the server.
A client's server peer covers the whole tunnel plane, so the server's
dataplane sees traffic that one endpoint sends to another. The server forwards
it only when the source endpoint's key may connect to the destination
endpoint: the destination is in the source's group, and the source key holds
connect_private, connect_shared, or admin on the destination device
itself. Only a Tier 1 server forwards between endpoints: forwarding needs
the tree, so a Tier 2 server installs a policy that forwards no packet between
dataplane peers. Endpoints that cannot reach each other directly use the relay
service, which carries only ciphertext. This is the rule the routing layer
applies. Replies return without a
grant in the other direction: a TCP segment that is not a bare SYN, and ICMP
echo replies and errors. A device that may only be reached cannot open a TCP
connection back, and a UDP reply needs a grant in its own direction. Relays
and addresses outside the tunnel plane, such as backbone servers, are not
restricted by this rule. An address in the plane that is not an endpoint or
relay in the tree is dropped. The server rebuilds the rule off the packet path
when the tree changes; a revoked grant stops forwarding within about one
second.
After Genesis, the current committee finalizes four transitions in order at each epoch boundary, each a pure function of finalized inputs:
eligibility state -> NextEpochPlan -> CandidateReadiness -> EpochHandoff
- Eligibility is computed from verified platform evidence and mesh observations signed by current members. Observations are epoch-scoped and affect the next epoch's eligibility only.
- The next-epoch plan selects the target membership.
- Candidate readiness requires the selected candidates (including new members) to sync to the checkpoint, create their vote keys, and attest freshly under a plan-bound challenge.
- The epoch handoff activates the new membership. A fresh dealerless FROST DKG ceremony follows finalized readiness and establishes the new epoch's authority key. The finalized handoff, verified against the predecessor's anchor, is what makes the new members authoritative.
HotStuff (chained, lock on the two-chain, commit on the three-chain) uses individual Ed25519 vote signatures. FROST supplies threshold authority signatures; it is not the consensus voting mechanism. Old vote keys and old FROST shares are discarded on epoch replacement, and the full FROST private key is not routinely reconstructed.
Membership and rule changes apply only at the epoch boundary. An epoch can legitimately outlive its target duration; the target is a pacing goal, not a timer that expires authority.
A current member can miss a boundary stage, for example a lost message or a restart inside the boundary. Its own commit of the handoff then does not activate it, because it did not prepare the same handoff. It leaves the finished epoch when it verifies the finalized handoff proof: the broadcast proof, or the authority chain it requests after an announcement of a newer epoch. It enters the new epoch as a member only when the handoff names it with its registered vote key and it holds the DKG outcome the handoff names. Otherwise it continues as Tier 2, as a restart would make it. A proof that does not verify changes nothing.
A pending candidate can lose its boundary the same way: the attempt that named it is replaced by one that leaves it out, or the handoff of its plan never reaches it. Its preparation ends when it verifies the replacement plan, the finalized handoff under another plan, or the authority chain it requests once a hint of a newer epoch outlives the sync window. It then continues as Tier 2 and adopts the next plan that names it. Nothing it prepared grants a role in the epoch it missed.
An idle Tier 2 node also asks for the next chain page on a slow cadence without any hint, because a node that was away during a boundary heard none, and its storage unlock names its stale epoch until it catches up. A server with nothing newer stays silent.
Genesis is a temporary bootstrap authority, not a permanent root:
- The founding path requires five qualifying founders, agreement on the founding facts (signed transcript attestations from every founder), and a fresh Epoch 1 DKG. When more than five candidates qualify, the five lowest node identities found Epoch 1.
- Genesis signs the bootstrap certificate. Its unilateral security-protocol authority ends when Epoch 1 activates. There is no restart, recovery, or administrative path that restores it.
- Later authority verifies from the pinned Genesis anchor through the finalized epoch handoff chain. Certificates, peer announcements, local configuration, and attestation results cannot independently create Tier 1 authority.
Persisted anchors and epochs are candidate data, not authority. At startup, a node reauthenticates the Epoch 1 base from the pinned Genesis key (the bootstrap certificate plus its exact founder/vote-key listing) and then verifies every sequential handoff proof before it restores authority or membership. A recomputed record digest provides corruption detection, not authentication.
Any surviving post-Genesis durable state consumes the one-shot Genesis signing role before startup selects a phase, so a former Genesis node cannot reuse its bootstrap role. A node that finds only an authentic historical chain resumes as Tier 2 and walks the chain. Corrupt or contradictory state fails closed.
This does not protect against deleting or rolling back the whole store to a valid pre-Epoch-1 snapshot, and an authenticated historical chain does not by itself prove it is the newest chain. See Current Limitations.
The deterministic rules live in
SecurityConstants.hpp.
They are compiled into the binary, not operator settings:
| Rule | Value |
|---|---|
| Maximum Byzantine faults | f = floor((N - 1) / 3) |
| Consensus quorum | N - f |
| Authority-signing threshold | max(5, N - f) |
| Genesis founding threshold | 5 qualifying founders |
| Active Tier 1 bounds | minimum 5, preferred minimum 7, maximum 31, plus a reserve of 2 |
| Target epoch duration | 3600 s (pacing goal, not an authority expiry) |
| Re-attestation interval | 900 s |
| Final attestation max age | 300 s |
The target Tier 1 population table (for example 5 admitted servers target 5 active members; 100 admitted target 13) is a selection goal. It is not available address capacity, not a quorum, and not a measured scalability claim.
See Attestation for the provider matrix and the evidence path. The boundary that matters here:
- The Tier 1 profile is
snp-svsm-vtpm: AMD SEV-SNP with a COCONUT-SVSM vTPM. The provider verifies the SNP and SVSM evidence chain, the enrolled attestation key, PCR 8 and 9, and IMA runtime measurements, bound to a fresh challenge, the node identity, the epoch, and the network. The release values come from the signed release approval, never from the host. nexus-attestdruns under a separate service identity with scoped TPM/IMA access and filesystem restrictions. It collects evidence without holding the server's Nexus identity key, consensus vote key, or FROST share. It seals and unseals a PCR-bound key for the server's boot set but never receives the key material that key protects. The server itself has no TPM access.- The threshold storage key is a separate key role. Each server encrypts
its covered data files under its own data key and wraps that key to the
storage public key. Unlock needs partials from
t = ceil(0.65 × n)holders, and the first key needs 5 eligible, attested holders. See Design: Tree Consensus and Threshold Storage Key. - Confidential computing protects against a hostile host or hypervisor. It is not a claim that privileged root inside the guest cannot inspect guest processes.
What the protocol defends:
- Forging or replaying consensus votes or authority signatures (quorum plus FROST threshold, canonical digests, domain separation, epoch and network binding).
- A compromised or partitioned minority of the committee (
fByzantine faults). - A hostile hypervisor substituting the guest image or forging vTPM quotes outside the confidential boundary (SNP launch measurement plus the SVSM-bound vTPM, IMA runtime measurements).
- Substituting the running binary (IMA measurements checked against the binary hashes in the signed release approval, whose release key the pinned Genesis key delegates).
What it does not defend:
- Privileged root inside the guest.
- Rolling back or deleting the whole durable security store to an authentic older snapshot (see Current Limitations).
- A network that never reaches Tier 1. Without a signed release approval for the network and a qualifying platform, no node qualifies, and the deployment has no authority committee at all.
- Public API: HTTPS only. Listeners are withheld until usable certificates exist; there is no plaintext fallback.
- Private API: HTTPS on virtual mesh addresses, forwarded through the userspace netstack to a loopback listener. Every private route is JWT-gated.
- Mesh: the BoringTun dataplane over UDP
51940. - Security protocol: signed gossip envelopes on UDP
9102. The envelope is authenticated and size-bounded before parsing, never relayed, and carries attestation, consensus, DKG, and epoch-announcement messages. Sensitive pairwise DKG payloads are additionally sealed to the recipient identity. Security messages are not all carried inside the BoringTun dataplane.
Rate limiting is a request budget (Api/RequestBudget.hpp). A request that
carries a valid session token spends the budget of the node the session
names (600 requests/minute, burst 60: a join or a resume is a burst of
twenty-odd requests, and many devices share one address behind a carrier
NAT). A request without a valid session spends the per-address budget
(default 120 requests/minute, burst 20): probes, discovery, and the
challenge round-trip before a session exists. A token that does not verify
falls back to the address budget, so a forged token buys nothing, and a
valid one moves the request to another bucket and nothing more: every route
still authenticates and authorizes on its own.
GET /api/tier1/authority has its own budget per client address (600
requests/minute, burst 120): a client walks the whole authority chain from
Genesis before its first tree request, and under the general burst a chain
longer than about 80 epochs could not be walked at 4 links per page. The
pages carry only proofs the client verifies, and a page holds at most
kMaxHttpChainLinks (32) links. A 429 answer carries Retry-After: 1.
Security messages have per-peer budgets and deduplication windows compiled
into the router.
Each tree node carries assignments: a management key and the permissions it holds on that node. Permissions are not inherited, with one exception for administration (below). A signed tree delta must pass these rules on the server that applies it:
- The operation permission.
create_nodeneedsadd_childon the parent.update_nodeandupdate_assignmentneededit_nodeon the node.delete_nodeneedsdelete_nodeon the node. - An update keeps the node's identity. The
id,parent_id,type, management key, and endpoint identifier cannot change. A record that renames itself onto another node, or moves into another group, is refused. A node changes groups only through the move route (see "Moves" below). - Only an admin changes assignments. The signer must hold
adminon the node or on one of its ancestors. That nearest admin node is the signer's scope.edit_nodealone never changes who holds what. - Only joined identities are granted. Each added or changed assignment must name a key that is the management key of a node inside the signer's scope. A key that never joined, or joined a different group, is refused.
- A create names no one else unless the signer is an admin. The new node may name its creator and its own management key. Any other key needs an admin of the parent and the joined rule above.
- The root is fixed. No delta creates a second root or deletes the root.
The creator of a group holds admin on it, so the owner can promote a member
device by giving its key admin on the group.
An admin of a group also administers the group's devices without an entry on each one: it reads a device's full assignment list and removes the device. No caller removes the root node through the API.
A device also removes its own node through the API (it leaves the network),
even when it joined through an invite and holds no delete_node there. The
rule covers only the device node that the caller's key joined as: an
Endpoint whose id is the key's join node id and whose management key is
the caller, with no node beneath it. It never covers another device, a
group, or a signed delta, and it only removes the caller's own presence.
When a node is deleted, every server that applies the delete runs the
removed-nodes cascade (TreeApiHandler::cascade_node_cleanup). The cascade
revokes the node's management key when no remaining node owns it, so the key
can no longer authenticate. A device's own leave is the exception: its key
stays valid, and the device can join again later with a new invite and the
same identity. The exception follows from the committed op alone
(tree::is_own_device_removal: a DeleteNode session op whose target is the
signer's join node id, on a node whose management key is the signer), so
every replica decides the same. An admin's removal of a device and an account
deletion still revoke the key.
Clients change assignments through POST /api/tree/node/assignments/{id}
with {"pubkey", "permissions"}. The server sets exactly that list for the
key, replacing its entry, and an empty list removes it. The same admin and
joined rules apply. An admin cannot change its own entry on the node that
grants its admin, so no admin locks itself out; another admin must do it. Groups created before this
rule gain admin for their owner on the owner's next join.
On load, a node file whose record names a different node is refused.
A service permission is a name of the form svc:<family>. The family has 1
to 63 bytes: the first is a lowercase letter or a digit, and the rest are
lowercase letters, digits, _, ., or -. Examples: svc:inference,
svc:files.read. The assignment route and invites accept a well-formed
service permission and refuse every other unknown name
(tree::rules::permission_name_known).
The server is opaque to a service permission. It stores and replicates the
name like any other assignment name and attaches no permission bit to it. A
key that holds only svc: names cannot read, edit, delete, or connect to
anything, and it administers nothing. The device that a peer connects to
enforces the service permission in its SDK.
A service permission on a group applies to every device beneath that group,
in the same shape as membership reach. admin on a device or on an ancestor
implies every service permission. The mesh peer list reports this effective
set to each device in peer_permissions (tree::rules::peer_permissions;
see "Direct paths" above). An invite grants a service permission only when
the inviter's own entry on the target group names it, or the inviter
administers the target.
An invite is a single-use link token for a target group
(POST /api/link/token, optional target_node_id and permissions), issued
together with an 8-digit invite code that names the same token (see "Invite
codes" below). Without a target, the target is the caller's own group. The
rules (tree::rules::invite_refusal) are:
- The target is a group. An invite never targets a device or the root.
- The inviter has authority on the target. The inviter holds
add_childon the target or administers it (adminon the target or an ancestor). - An invite never grants more than the inviter holds. Each permission is
one of
read,write,add_child,connect_private, andconnect_shared, or a service permission, and the inviter holds it on the target or administers the target.admin,edit_node, anddelete_nodeare never granted by an invite, so neither side gains control over the other's group. Withoutpermissions, the invite carriesreadandconnect_private, reduced to what the inviter holds.
A new device uses the token in POST /api/join and is placed under the
target. Its key's entry on the target is the invite's permissions with
read (TreeOpBody::grant_membership), so a device invited with a role
holds its service permissions as soon as it is listed; every replica checks
at the block that the inviter still holds them on the target
(tree::rules::invite_refusal) and refuses the join otherwise. A join from a
release before this rule, or with a token minted without permissions, places
the device with read alone. An identity that already joined uses the token in
POST /api/link/accept (or POST /api/join from its existing device). It
becomes a member: the server sets exactly the invite's permissions as the
identity's entry on the target (AcceptInvite). Nothing the member owns
moves, and no other node changes. At acceptance the server checks the rules
again against the current tree (tree::rules::invite_acceptance_refusal):
an inviter that lost its authority voids the invite, and an identity that
already joined inside the target, or administers it, is refused. A refused
acceptance does not spend the token. Tokens stay single use and expire after
at most one hour; the code of a token stops working after at most ten
minutes. Any Tier 1 server redeems a token or its code, whichever Tier 1
server minted it (see "Invite replication" below).
Linked devices see each other. When a new device joins with a token, each
device already in the target group (same parent group, Endpoint nodes only)
and the new device get read and connect_private on each other
(tree::rules::grant_link_peer_access). The group owner's key already holds
an entry on the new device (it keeps control of the device), so the grant
adds the two permissions to that entry; earlier releases skipped it, and the
owner's first device did not list devices linked later. No other group and
no other permission is touched.
Membership reach. A literal connect_private or connect_shared entry
for key K on a group G is a membership. It lets each device whose management
key is K connect to every device beneath G, and lets every device beneath G
connect to those devices (tree::rules::endpoint_may_connect). It covers no
other device: the member's other devices (other keys) and the rest of its
network stay invisible to G, and the member sees nothing of the inviter's
other groups. An admin entry is not a membership and gives no connect
reach. The mesh peer list, the routing chokepoint (resolve_authorized), and
the dataplane forwarding policy (ForwardAuthorizer) apply this one rule.
Listing. GET /api/tree/devices/{group} lists a member's device with the
group's devices (member: true), because the device stays in the member's own
network and GET /api/tree/children never returns it. The route needs a read
right on the group and walks only the subgroups the caller may read. It lists a
member's device to callers that administer or edit the group (the readers that
see every assignment), to callers whose device may connect to it (the peer-list
rule above), and to the member itself. Two members of one group do not reach
each other, so neither sees the other in the list. The member's other devices
are never listed: their keys hold no entry on the group.
Leaving. An admin of the group removes a member through
POST /api/tree/node/assignments/{id} with an empty list. A member removes
its own entry the same way without admin, but only on a group it has not
joined and only when that entry holds no admin
(tree::rules::assignment_change_refusal). Removing the entry removes all
membership reach at once.
Limit: an admin can remove a member but cannot change a member's permissions in place, because the joined-identity rule above refuses a key that did not join inside the group. Send a new invite instead.
POST /api/tree/node/move/{id} with {"parent_id"} moves a device or a
subgroup under another group (MoveNode, tree::rules::move_refusal):
- The node is an Endpoint or a Customer group. The root and every top-level group (a group directly under the root) never move, so no one moves another user's network out from under its owner.
- The new parent is a group, differs from the current parent, and is not the node or one of its descendants.
- For a device, the signer holds
delete_nodeon it or administers it. For a subgroup, the signer administers it:delete_nodealone cannot take over a subtree. - The signer holds
add_childon the new parent or administers it.
A move keeps the node's id, tunnel address, endpoint identifier, and own assignments. It clears the node's parent signature, which no longer covers the record. It grants nothing: access in the new group comes from that group's rules (its admins administer the node; its memberships cover it). Connect grants that depended on the old group's scope stop applying. When a moved device's key owns no other node, the entries that its old group and old sibling devices held for that key (the link-join grants) are removed, as a delete removes them.
The mesh port names a device by its hostname folded to one DNS label
(tree::device_label, Tree/DeviceLabel.hpp: lower case, every run of
characters other than ASCII letters and digits becomes one hyphen, no leading
or trailing hyphen, at most 63 characters). "Work PC" and "work-pc" are the
same name there. The tree therefore keeps labels unique within a group
(tree::rules::device_name_refusal), in the apply step that every Tier 1
server runs on the committed operation:
- A
Jointhat creates a device, aCreateNodeof a device, and anUpdateFieldsthat changes a device's label are refused when another device of the same group folds to the same label, and when a nonempty name folds to no label at all. The HTTP answer is409with a message that names the colliding device and asks for another name. A device with no name has no label and is never refused; the mesh port lists it under its node id. - A rename that keeps the label, and an edit of other fields, pass. A collision stored before this rule stays as it is; the mesh port lists such devices under their node ids and refuses the ambiguous name.
- Another group may reuse the label. A move into a group is not checked.
POST /api/tree/device-name/check (public API, session required) lets a
client check a name first. With group_node_id the caller needs read on
the group; with link_token the invite's target group is checked and the
token is not spent. The answer is advisory; the operation rule above decides.
A token is minted on one Tier 1 server and can be redeemed on any Tier 1
server (Auth/LinkTokenLedger.hpp, TokenLinkAuthProvider).
- The token never leaves the minting server. Every record names it by
link_token_hash_hex(token), SHA-256 under the domainlemonade-nexus/link-token-hash:v1. The token is returned once, to the inviter. - Events, not mutable records. A token's life is a set of events:
Mint(target group, permissions, inviter, network id, issue and expiry times),Spend(a server consumed generation g), andRefund(the server whose spend counts restored the token as generation g + 1). Each event is signed by the gossip identity of the server that made it, under the domainlemonade-nexus/link-token-event:v1, and binds the network id. Replicas that hold the same events derive the same state in any arrival order. When two servers spend the same generation, the spend with the smallest origin key counts on every server, and a refund by any other server never restores the token. ARevokefrom any server ends the token for good: an account deletion revokes the tokens the account minted and the tokens that target a removed group (TokenLinkAuthProvider::revoke_where), and the revocation replicates like every other event. - Transport. Events use the change-numbered catch-up replication
(
sync::OriginReplica, streamlink, gossip typeLinkTokenSync) that DNS records and account envelopes use. They travel only on the encrypted backbone, only from a sender that is a current Tier 1 member, and only to Tier 1: a Tier 2 server neither receives, requests, nor serves them, and a server that leaves Tier 1 deletes them with the tree files. A receiver checks the network id, the origin signature, and that the origin holds a root-signed certificate, the same origin rule as account envelopes. - Bounds. At most 64 live tokens per inviter, 16384 tokens per server, 64 events per token, and 8 refunds per token. A token's events are kept for 10 minutes after its expiry and then pruned on every server; an event outside that window, or with an expiry or timestamp too far ahead of the receiver's clock, is refused.
- Storage. The provider's ledger (
link_tokens/ledger.json) and the replica file (link_replica/replica.json) are storage-vault files, encrypted under the data key once the vault runs encrypted. On restart the replica entries are verified again before use. A token file from an earlier release (link_tokens/<sha256>.json) is still redeemed, once, on the server that minted it, until it expires. It is a storage-vault file too: an encrypted server encrypts it in place, and a record that does not open refuses redemption.
Single use. A Join with a token and an
AcceptInvite carry the signed Mint in a link extension of the op
(TreeOpBody::invite_grant). Every replica checks, against the block: the
grant is a well-formed mint signed by a trusted submitter of the block's
epoch, for this network, unexpired at the block time, and bound to the op's
group, inviter, and permissions. The op records the invite's redemption id in
the committed id set until the invite expires, so a second op that redeems
the same token is refused (Replayed) on every replica, whichever server
submitted it. A refused op does not commit the redemption, and its submitter
restores the token. An invite minted by a server that is not a trusted
submitter of the block's epoch is refused.
The replicated Spend and Refund events keep every server's ledger in
step; they decide no redemption. A redemption fails while the token's Mint
has not reached the redeeming server yet (normally under a second after
minting).
Every mint also issues an invite code: 8 decimal digits (about 26.6 bits),
returned once to the inviter as code next to link_token. A string of
exactly 8 digits in a token position (link_token in POST /api/join,
POST /api/link/accept, and POST /api/tree/device-name/check) is a code;
anything else is a token. The code is an alias of the token, not a second
invite:
-
Hash only. The
Mintcarriescode_hash, SHA-256 of the code under the domainlemonade-nexus/link-code-hash:v1; the code itself is never stored or logged. The origin signs the binding (token_hash,code_hash, network id, expiry, origin, sequence) on its own underlemonade-nexus/link-code-binding:v1ascode_signature, so the event signature, the ledger file, and the tree-op encoding a release without codes computes stay as they are. A receiver verifies both signatures and refuses a mint whose binding does not verify. -
One generation. A redemption by code resolves the code to its token and spends that token's current generation; the token is refused until a refund opens the next one, and the other way round. The tree operation carries the signed
Mintwithout its code binding, so every replica checks and records the redemption of the token, whichever form the holder presented. -
Lifetime. A code works until
min(expires_at, created_at + 600 s)(code_expires_atin the mint response); the token works untilexpires_at. -
Attempt limits. Each Tier 1 server counts wrong code attempts: a code that resolves to no live token on that server (unknown, expired, spent, or shared). Two limits apply, both over a 10-minute window:
- Per source address: after 5 wrong attempts from one address within the window, that address's code attempts are refused for 10 minutes.
- Per server: after 100 wrong attempts from any addresses within the window, the server refuses codes until the window passes.
A refused attempt counts toward neither limit, so an address that is locked out cannot close codes for everyone, and tokens are never refused by either limit. An attacker with N addresses gets at most min(5N, 100) guesses per 10 minutes per server against a space of 10^8 codes that each live at most 10 minutes. The hash and the lookup run before the limits are consulted, so a refused or wrong code costs the same work as a wrong token (neither reveals how close the guess was). The counters are per server and in memory: a restart or another server starts them again. A server tracks at most 4096 addresses; beyond that a new address's code attempts are refused until an entry expires.
-
Uniqueness and collisions. The minting server draws a code that no unexpired code in its own ledger uses (a bounded number of draws; a mint fails when none is free). Two servers can still mint the same code in the same window, before either sees the other's mint. A receiver that holds two unexpired mints with one
code_hashrefuses that code (it never picks one); both tokens stay redeemable. With at most 16384 tokens per server and 10^8 codes, the chance of a collision per mint is below 2 x 10^-4. -
Older replicas. A server from a release without codes ignores the two fields, redeems the token only, and passes the mint on without them. A server with codes keeps the copy that carries the binding, whichever copy arrives first.
An account deletes itself through POST /api/account/delete (SDK
delete_account, C ln_account_delete). There is no form of the route that
names another account, and no operator or admin path deletes an account.
Fresh proof. A session token alone does not delete an account. The client
gets a challenge from the session's server
(POST /api/account/delete/challenge, the Ed25519 challenge store of
/api/auth/challenge) and signs auth::account_delete_proof_message: the
canonical encoding, under the domain lemonade-nexus/account-delete:v1, of
the challenge and the account's public key. The server checks that the
challenge was issued to the session's key, is less than 60 seconds old, and
was not used, and then verifies the signature. A login signature never
verifies as a deletion proof, and a deletion proof never opens a session.
Scope (tree::rules::account_scope). The deletion removes:
- every Customer or Endpoint node whose management key is the account key
(its
customer-<user_id>group, its device, and any node it created elsewhere), and every node beneath one of them. This includes devices and subgroups that other identities placed in the account's groups. Those identities keep their own accounts, but they lose access to this network; a device key that owns no other node is revoked. - every assignment, on every remaining node, that names the account key or the key of a removed node that owns no remaining node: the account's memberships in other users' groups, and the connect and link-join grants that named its keys.
The root node is never removed or changed, whoever calls. When the root management key also owns a Customer group, deleting that account removes the group and leaves the root, the root's owner assignment, and the key itself.
Cleanup on every server that applies the deletion. The removed-nodes
cascade (TreeApiHandler::cascade_node_cleanup) revokes device keys that own
no node, releases IPAM allocations, removes mesh peers, removes the private
DNS name, and removes the device and group certificates. The account cleanup
(AccountDataApiHandler::cleanup_deleted_account) deletes the account's chat
blobs and group-key envelopes (on disk and in the replicated envelope
stream), revokes the link tokens it minted or that target a removed group
(a replicated Revoke event, see "Invite replication"), and deletes its
passkey credentials. When the account key owns no node any longer, the
server revokes it. A revoked key cannot authenticate, and its sessions end,
including a session that names only the user id (passkey).
Ordering. The route submits one DeleteAccount operation (kind 10).
Every Tier 1 server validates and applies it at the same block, computes the
same scope, and runs both cleanups. The committed op id keeps a replay
refused after a restart.
Idempotence. A deletion with nothing left changes nothing and succeeds. After a deletion the revoked key cannot request another one.
The answer leaves first. The request arrives over the device's own
tunnel, and the removal cascade closes that tunnel by removing the device's
dataplane peer. The server that handles the request keeps the peers of the
account's own devices until the response has been written or the connection
failed (PeerRemovalHold), then removes them. Other Tier 1 servers remove
their copies of the peers at once; they do not carry this tunnel. The same
hold applies to a device that deletes its own node (a leave). A client whose
answer is lost anyway confirms the deletion through the revoked key (see
Client SDK).
Limits:
- An envelope is dropped from the replicated stream without a tombstone. A peer that has not yet applied the deletion can offer it again; a server that applied the deletion refuses it, because its tree no longer authorizes the envelope.
- Key revocations, passkey credentials, and chat blobs are stored per server.
Each Tier 1 server that applies the deletion removes its own copies; a
server outside Tier 1 at that time keeps its copies. Link token revocations
replicate to every Tier 1 server; a token whose
Minthas not reached the deleting server yet (normally under a second after minting) is not revoked there. - Local files (the identity key, the account image cache, and certificate directories) stay on the client. The client deletes them (see Client SDK).
Admission and several replicated application stores are not derived from finalized security state. This is the boundary between what the security protocol authorizes and what the application layer still gates on identity or root:
- Server admission (certificate issuance, enrollment tokens, supersede) is root-gated or token-gated. It grants transport membership, not Tier 1.
- Replicated application stores (ACL, DNS records, backbone IPAM, NS slots, account group-key envelopes, link token events) sync through gossip under root-signed peer certificates. A Tier 2 server holds no tree, so it accepts no account envelope and no link token event. Without tree operations, link token single use is not consensus-authorized (see "Invite replication"). Their mutation paths are audited against permission rules but are not yet consensus-authorized.
- Author-signed record receipt is not authorization. Receiving a signed tree record transfers it; it never by itself authorizes or applies a remote tree or ACL mutation.
- Public authentication does not initialize an application-root owner. Fresh-network administrative approval flows remain incomplete.
No one of these paths can touch epochs, eligibility, or authority keys.
These are implementation limits, not settings an operator can disable:
- Tier 1 needs a signed release for its network. Without a verified
release approval, the attestation profile has no approved binary digests
and no PCR 8/9 values. Every attestation then fails with
ProfileIncomplete. An approval binds to one network ID, and a locally built package carries only an unsigned example. Only the VM shapes whose launch measurements an approval lists can qualify. See Attestation. - Provider support is narrow.
snp-svsm-vtpmis the only profile under which the binary issues Tier 1 challenges. Thesnp-hcl-vtpmverifier exists but is not the Tier 1 profile. The direct-boot provider refuses withProviderUnsupported. SGX, TDX, and Secure Enclave are not implemented Tier 1 providers here. - Application authority is not fully integrated. Admission and several replicated application stores still use root- or identity-gated paths. Receiving an author-signed record does not by itself authorize remote tree/ACL mutation. Public authentication does not initialize an application-root owner.
- Recovery and freshness have limits. The quorum-authorized incarnation-advance lifecycle is unfinished. Deleting or rolling back the whole store to a valid pre-Epoch-1 snapshot is not addressed, and a valid historical authority chain cannot in isolation prove it is the newest chain.
- Lost storage state stops the server. A server whose files are encrypted refuses to start when the storage state that opens them is missing, and the log names the files to restore.
- No tree without Tier 1. Only Tier 1 members hold the tree. Before Epoch 1 activates, and while no server qualifies for Tier 1 (item 1), no server serves a tree route or a client session, and no client can join.
- Relay fallback and hole punching are tested only in software. Relay
allocation, pairing, forwarding, expiry, the SDK fallback, reflexive-address
learning, and hole punching have unit tests and loopback end-to-end tests
(
test_relay_service,test_sdk_relay_fallback,test_sdk_hole_punch). They are not tested across real NAT devices or under load. Each device publishes one reflexive endpoint from one STUN server, and the SDK does not detect endpoint-dependent (symmetric) NAT mappings. LAN endpoints are tested on loopback only (test_sdk_lan_paths,test_sdk_hole_punch,test_sdk_transport). A Tier 1 server keeps the LAN endpoints of the devices whose heartbeats reach it, in memory: they are not replicated, so two devices whose sessions are on different Tier 1 servers do not learn each other's LAN endpoints, and two such devices behind one NAT connect directly only when the NAT supports hairpinning. Pair slots assume that the two device clocks agree to within the NAT mapping lifetime. A peer without a listed direct endpoint always uses a relay. Both peers pick the relay from their own candidate lists; when the lists differ, the peers can pick different relays until a failed relay is skipped. The client dataplane reaches relays over IPv4 only. - Tree-derived DNS names are Tier 1 answers. A nameserver that is not a Tier 1 member has no tree and does not answer endpoint host names.
The security overhaul removed the old network-authority system. These terms
appear in older documents and in the C ABI compatibility stubs, which return
LN_ERR_UNSUPPORTED:
- Democratic parameter governance and governance votes
- Root key rotation on a fixed interval and Shamir secret sharing of the root
- Enrollment ballots and operator quorum knobs
- The
AttestationTokenrolling-trust layer - The
/api/trust/*,/api/enrollment/status, and/api/governance/*endpoints, and the old gossip wire types in the range0x07–0x10(reserved forever)
They are described here only so that historical material can be read correctly. They are not current behavior, and they are not coming back.