feat(egress): fleet DoH-443 blocking, telemetry, per-sandbox netns OUTPUT and connection refresh (OSEP-0022) - #1608
Open
Pangjiping wants to merge 5 commits into
Conversation
…TPUT and per-subject connection refresh (OSEP-0022)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5644f1a801
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…pensandbox-group#1608) - sandboxnft: create DoH blocklist sets only on fresh installs (a policy swap re-adding an existing set failed with "File exists", blocking activation under a blocklist configuration) - sandboxnft: scope the DNS exception to slot.Gateway (a denying sandbox could otherwise reach any host-local resolver via the INPUT path the sandbox layer is meant to cover) - sandboxnft: ApplySlotUpdate reinstalls the table on unchanged-fencing netns/gateway moves, keeping the defense layer aligned with dispatch - fleet: startup recovery now wipes live sandbox tables too (slot store + netns mount dir), not just the Pod table - fleetnft: ApplyDispatchUpdate stores the updated slot so the connection refresh keeps bucketing by the moved source IP
Pangjiping
commented
Aug 25, 2026
Pangjiping
commented
Aug 25, 2026
Pangjiping
commented
Aug 25, 2026
Pangjiping
commented
Aug 25, 2026
Pangjiping
commented
Aug 25, 2026
Pangjiping
commented
Aug 25, 2026
Pangjiping
commented
Aug 25, 2026
opensandbox-group#1608) - refreshTick: batch all subjects' refreshes into ONE nft transaction and build the src->subject index once per tick, so the applier lock is never held across per-subject exec chains (a slow tick cannot stall policy ops) - refresh: mark IPs pending redelivery when the sandbox mirror fails; the next tick re-adds them unconditionally, so a transient mirror miss can never self-lock a subject until its lease expires and the client re-resolves - fleetnft: egress.nftables.rules.count gauge now sums across all installed subjects (deny-first installs, dispatch updates, rebuilds no longer drift) - sandboxnft: startup Reset counts successful deletions as nft updates and downgrades expected not-found netns to debug level - docs: TCP-only refresh limitation and strict DoH (all-443, policy cannot override) called out; opentelemetry.md notes fleet gauge semantics and uncounted expected sandbox-layer cases - smoke: Test 9's stale-rule assertions now check the DNS-learned dyn lease (1.1.1.1) which is the only stale rule that can survive a restart; new Test 10 gives strict DoH mode real-kernel coverage
${VAR:-default} treats a set-but-empty EGRESS_DOH_BLOCKLIST as unset and
fell back to the blocklist, so Test 10 ran blocklist mode: the bare-443
assertion passed on the daddr-rule substring and the no-blocklist-set
assertion failed. Use ${VAR-default} so an explicitly empty value stays
empty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enhancements to the fleet profile (multi-sandbox egress control plane, OSEP-0022), on top of #1595.
Four problems solved
1. Encrypted DNS over 443 (DoH) was not blocked
The master chain only blocked DoT 853; encrypted DNS on port 443 had no interception mechanism. The fleet profile now supports
OPENSANDBOX_EGRESS_BLOCK_DOH_443+OPENSANDBOX_EGRESS_DOH_BLOCKLISTwith the same semantics as the sidecar profile: drop TCP 443 to a configurable IP/CIDR list, or all TCP 443 in strict mode when the list is empty. The rules are global in the master dispatch chain and survive table resets/rebuilds.2. The fleet profile exposed no observability metrics
Only the sidecar profile exported OpenTelemetry metrics; the fleet assembly had no telemetry at all. The fleet profile now wires the same OTLP export as the sidecar: DNS metrics (
egress.dns.*) come from the shared proxy, and every nft operation (deny-first install, policy apply, DNS-learned dynamic adds, dispatch updates, reset, remove) recordsegress.nftables.updates.*success/failure counters plus the rules gauge.3. Per-sandbox netns had only the Pod forward hook as a single layer of defense
OSEP-0022 requires per-sandbox OUTPUT enforcement installed from the host as defense in depth; previously only the Pod-netns forward hook was enforced. Each sandbox netns now gets a mirrored policy chain (table
opensandbox-fleet-ns) installed viansenter --net=<slot.HostNetnsPath>: deny-first full block, static allow/deny sets, DNS-learned dynamic leases, DoT/DoH blocking, and an explicit default verdict. This layer catches traffic the forward hook never sees (sandbox to Pod-host-local destinations, which take the INPUT path). Install failures fail closed (registration retries, subject stays denying); removal is best effort since the rules die with the netns.4. Active TCP connections relied on DNS lease TTLs only
There was no connection refresh mechanism, so active connections could be dropped after their lease expired. A per-subject bucketed refresh loop now runs every 30s: one read of the Pod-netns conntrack table (
/proc/net/nf_conntrack— the forward hook carries every sandbox flow, so a single read bucketed by source IP serves all subjects), renewing active leases, pruning expired idle entries, and issuing one final refresh when activity ends as the bounded reconnect grace period (mirroring the sidecar tracker). The sandbox-netns layer receives the same refreshed leases through a mirror callback, keeping both layers identical.Wiring
The two layers (Pod-netns forward hook, authoritative; sandbox-netns OUTPUT, defense in depth) are composed behind the existing
fleetNftAppliersurface viafleetEnforcer—fleet_server.gois unchanged. A failure in either layer fails the whole operation (fail closed).Tests
smoke-fleet.shextended with real-kernel per-sandbox netns assertions (rule install, policy mirror, rebind/restart wipe, unload removal); the existing data-path tests now traverse both layers.docs/components/egress.md,opentelemetry.md,policy-traffic-vault-flow.md).