Skip to content

feat(egress): fleet DoH-443 blocking, telemetry, per-sandbox netns OUTPUT and connection refresh (OSEP-0022) - #1608

Open
Pangjiping wants to merge 5 commits into
opensandbox-group:mainfrom
Pangjiping:feat/egress-fleet-a2a5-enforcement
Open

feat(egress): fleet DoH-443 blocking, telemetry, per-sandbox netns OUTPUT and connection refresh (OSEP-0022)#1608
Pangjiping wants to merge 5 commits into
opensandbox-group:mainfrom
Pangjiping:feat/egress-fleet-a2a5-enforcement

Conversation

@Pangjiping

@Pangjiping Pangjiping commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Enhancements to the fleet profile (multi-sandbox egress control plane, OSEP-0022), on top of #1595.

Four problems solved

1. Encrypted DNS over 443 (DoH) was not blocked

The master chain only blocked DoT 853; encrypted DNS on port 443 had no interception mechanism. The fleet profile now supports OPENSANDBOX_EGRESS_BLOCK_DOH_443 + OPENSANDBOX_EGRESS_DOH_BLOCKLIST with the same semantics as the sidecar profile: drop TCP 443 to a configurable IP/CIDR list, or all TCP 443 in strict mode when the list is empty. The rules are global in the master dispatch chain and survive table resets/rebuilds.

2. The fleet profile exposed no observability metrics

Only the sidecar profile exported OpenTelemetry metrics; the fleet assembly had no telemetry at all. The fleet profile now wires the same OTLP export as the sidecar: DNS metrics (egress.dns.*) come from the shared proxy, and every nft operation (deny-first install, policy apply, DNS-learned dynamic adds, dispatch updates, reset, remove) records egress.nftables.updates.* success/failure counters plus the rules gauge.

3. Per-sandbox netns had only the Pod forward hook as a single layer of defense

OSEP-0022 requires per-sandbox OUTPUT enforcement installed from the host as defense in depth; previously only the Pod-netns forward hook was enforced. Each sandbox netns now gets a mirrored policy chain (table opensandbox-fleet-ns) installed via nsenter --net=<slot.HostNetnsPath>: deny-first full block, static allow/deny sets, DNS-learned dynamic leases, DoT/DoH blocking, and an explicit default verdict. This layer catches traffic the forward hook never sees (sandbox to Pod-host-local destinations, which take the INPUT path). Install failures fail closed (registration retries, subject stays denying); removal is best effort since the rules die with the netns.

4. Active TCP connections relied on DNS lease TTLs only

There was no connection refresh mechanism, so active connections could be dropped after their lease expired. A per-subject bucketed refresh loop now runs every 30s: one read of the Pod-netns conntrack table (/proc/net/nf_conntrack — the forward hook carries every sandbox flow, so a single read bucketed by source IP serves all subjects), renewing active leases, pruning expired idle entries, and issuing one final refresh when activity ends as the bounded reconnect grace period (mirroring the sidecar tracker). The sandbox-netns layer receives the same refreshed leases through a mirror callback, keeping both layers identical.

Wiring

The two layers (Pod-netns forward hook, authoritative; sandbox-netns OUTPUT, defense in depth) are composed behind the existing fleetNftApplier surface via fleetEnforcerfleet_server.go is unchanged. A failure in either layer fails the whole operation (fail closed).

Tests

  • 23 new unit tests (sandboxnft script shape / policy swap / rebind reset / DoH mirror; conntrack parsing / lease renewal and pruning / bucketed refresh).
  • smoke-fleet.sh extended with real-kernel per-sandbox netns assertions (rule install, policy mirror, rebind/restart wipe, unload removal); the existing data-path tests now traverse both layers.
  • Docs updated (docs/components/egress.md, opentelemetry.md, policy-traffic-vault-flow.md).

…TPUT and per-subject connection refresh (OSEP-0022)
@github-actions github-actions Bot added component/egress documentation Improvements or additions to documentation size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 24, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5644f1a801

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread components/egress/pkg/sandboxnft/sandboxnft.go
Comment thread components/egress/pkg/sandboxnft/sandboxnft.go Outdated
Comment thread components/egress/fleet.go Outdated
Comment thread components/egress/fleet.go Outdated
Comment thread components/egress/pkg/fleetnft/refresh.go Outdated
…pensandbox-group#1608)

- sandboxnft: create DoH blocklist sets only on fresh installs (a policy
  swap re-adding an existing set failed with "File exists", blocking
  activation under a blocklist configuration)
- sandboxnft: scope the DNS exception to slot.Gateway (a denying sandbox
  could otherwise reach any host-local resolver via the INPUT path the
  sandbox layer is meant to cover)
- sandboxnft: ApplySlotUpdate reinstalls the table on unchanged-fencing
  netns/gateway moves, keeping the defense layer aligned with dispatch
- fleet: startup recovery now wipes live sandbox tables too (slot store +
  netns mount dir), not just the Pod table
- fleetnft: ApplyDispatchUpdate stores the updated slot so the connection
  refresh keeps bucketing by the moved source IP
Comment thread components/egress/pkg/fleetnft/refresh.go Outdated
Comment thread components/egress/pkg/fleetnft/refresh.go
Comment thread components/egress/pkg/fleetnft/fleetnft.go
Comment thread components/egress/pkg/fleetnft/fleetnft.go Outdated
Comment thread components/egress/fleet.go
Comment thread components/egress/pkg/sandboxnft/sandboxnft.go
Comment thread components/egress/tests/smoke-fleet.sh Outdated
opensandbox-group#1608)

- refreshTick: batch all subjects' refreshes into ONE nft transaction and
  build the src->subject index once per tick, so the applier lock is never
  held across per-subject exec chains (a slow tick cannot stall policy ops)
- refresh: mark IPs pending redelivery when the sandbox mirror fails; the
  next tick re-adds them unconditionally, so a transient mirror miss can
  never self-lock a subject until its lease expires and the client
  re-resolves
- fleetnft: egress.nftables.rules.count gauge now sums across all installed
  subjects (deny-first installs, dispatch updates, rebuilds no longer drift)
- sandboxnft: startup Reset counts successful deletions as nft updates and
  downgrades expected not-found netns to debug level
- docs: TCP-only refresh limitation and strict DoH (all-443, policy cannot
  override) called out; opentelemetry.md notes fleet gauge semantics and
  uncounted expected sandbox-layer cases
- smoke: Test 9's stale-rule assertions now check the DNS-learned dyn lease
  (1.1.1.1) which is the only stale rule that can survive a restart; new
  Test 10 gives strict DoH mode real-kernel coverage
${VAR:-default} treats a set-but-empty EGRESS_DOH_BLOCKLIST as unset and
fell back to the blocklist, so Test 10 ran blocklist mode: the bare-443
assertion passed on the daddr-rule substring and the no-blocklist-set
assertion failed. Use ${VAR-default} so an explicitly empty value stays
empty.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/egress documentation Improvements or additions to documentation size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants