feat(kyber): add bounded Incus Crabbox pilot

named-hosts/kyber/README.md

@@ -2,6 +2,9 @@
 
 Ubuntu Linux server managed via home-manager with Tailscale VPN.
 
+The [Incus Crabbox pilot](incus/README.md) provisions bounded local containers
+on Kyber without changing other hosts or the Crabbox coordinator.
+
 ## Initial Setup
 
 On the Kyber server, run:

named-hosts/kyber/default.nix

@@ -34,6 +34,7 @@ home-manager.lib.homeManagerConfiguration {
   modules = [
     agenix.homeManagerModules.default
     ../../home-manager/default.nix
+    ./incus
     (
       { config, lib, ... }:
       {

named-hosts/kyber/incus/README.md

@@ -0,0 +1,107 @@
+# Incus pilot on Kyber
+
+Kyber's Home Manager profile installs `~/.config/crabbox/config.yaml` with
+`provider: incus` and a `kyber-incus-setup` command. Other hosts are unchanged.
+Incus is a direct CLI provider, not a backend for the Crabbox Node coordinator
+or its browser portal. This does not change any repository's `ci:ship` routing.
+Repo configuration, environment variables, and explicit flags override these
+user defaults; pass `--provider incus` when the repo selects another provider.
+
+## Install and activate
+
+On Kyber, from the reviewed dotfiles revision:
+
+```sh
+cd ~/dotfiles
+HOST=kyber make build
+HOST=kyber make nix-switch
+kyber-incus-setup
+```
+
+The setup command requires non-interactive sudo and refuses other hosts or
+users. It installs Ubuntu's native `incus`, `incus-client`, and `btrfs-progs`
+packages without adding a package repository. Ubuntu APT owns their security
+updates. It enables Incus and applies the checked-in `preseed.yaml`; subsequent
+runs reapply the same named resources, refusing unmarked name collisions.
+It does not delete existing instances or modify the default Incus profile.
+
+Setup adds `ubuntu` to `incus-admin`, which grants **root-equivalent host
+authority**. Only the host operator gets this access; guests do not receive the
+Incus socket. Reconnect with `ssh kyber` after setup to acquire the new group.
+Existing long-lived Herdr sessions must be restarted separately before they
+inherit group membership; setup does not interrupt those sessions.
+
+## Pilot boundaries
+
+| Setting | Value |
+| --- | --- |
+| Project / profile | `crabbox` / `crabbox` |
+| Maximum existing instances | 2, including stopped instances |
+| Per container | 4 CPUs, 8 GiB RAM, 1,024 processes |
+| Project CPU / memory budget | 8 CPUs / 16 GiB RAM |
+| Storage | 64 GiB Btrfs loop-backed pool; 24 GiB root quota per container |
+| Per-container write limit | 10 MiB/s |
+| Network | `incus-crabbox`, `10.203.0.1/24`, IPv4 NAT |
+| Management | Local Unix socket only; no HTTPS listener configured |
+| Guest image | Ubuntu 24.04 cloud image |
+
+The pool is a file under Incus storage, **not a physical disk to format**. The
+64 GiB pool bounds its guest/image storage, not all Incus logs, backups, or host
+usage. Btrfs per-container quotas are not a hostile-tenant security boundary.
+Containers share Kyber's kernel and can reach host services: use trusted work
+only. Privileged containers, nesting, host bind mounts, and VMs are not enabled;
+Docker-in-Docker validation is outside this first pilot.
+
+The bridge is separate from K3s and Docker networks. A systemd oneshot permits
+only bridge-to-uplink forwarding and established return traffic through
+`DOCKER-USER`, including after Docker/Incus restarts. It does not open public
+ports or change Docker's global forwarding policy. Incus handles bridge NAT,
+DHCP, and DNS. Verify connectivity before treating the pilot as ready.
+
+## Verify and try two concurrent boxes
+
+In the new SSH session, from the repository you intend to sync:
+
+```sh
+id -nG
+systemctl is-active incus kyber-incus-network
+incus project show crabbox
+incus profile show crabbox
+crabbox config show --provider incus
+crabbox doctor --provider incus
+```
+
+After doctor passes, create two separate leases. Each command prints a lease
+ID; retain both IDs for explicit cleanup, including if bootstrap fails:
+
+```sh
+crabbox warmup --provider incus --slug kyber-smoke-1
+crabbox warmup --provider incus --slug kyber-smoke-2
+incus list --project crabbox
+```
+
+Use each printed ID in a separate terminal to verify sync and execution:
+
+```sh
+crabbox run --provider incus --id <lease-id> -- sh -c 'hostname; id; getent hosts github.com; test -d .git'
+```
+
+The containers use separate bridge IPs with port 22; there is no shared host
+proxy port to collide. They are not automatically named Tailscale machines.
+Release only the two IDs created by this smoke test:
+
+```sh
+crabbox stop --provider incus <first-lease-id>
+crabbox stop --provider incus <second-lease-id>
+incus list --project crabbox
+```
+
+Default release deletes the instance and frees its slot. There is no daily
+start quota configured here, but the two-instance cap still applies. Raise it
+in `preseed.yaml` only after measuring CPU, memory, disk, and I/O contention.
+Removing the Home Manager import alone does not uninstall Incus or destroy its
+data; retirement requires an explicit inventory and cleanup operation.
+
+References: [Crabbox Incus provider](https://crabbox.sh/providers/incus.html),
+[Incus initialization](https://linuxcontainers.org/incus/docs/main/howto/initialize/),
+[Docker coexistence](https://linuxcontainers.org/incus/docs/main/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-incus-and-docker).

named-hosts/kyber/incus/crabbox.yaml

@@ -0,0 +1,16 @@
+# User defaults on Kyber only; explicit repo configuration and flags still win.
+provider: incus
+target: linux
+os: ubuntu:24.04
+incus:
+  socket: /var/lib/incus/unix.socket
+  project: crabbox
+  instanceType: container
+  image: images:ubuntu/24.04/cloud
+  profile: crabbox
+  user: crabbox
+  workRoot: /work/crabbox
+  deleteOnRelease: true
+  startTimeout: 10m
+  launchPort: "22"
+# Reach each guest directly on its bridge IP; a shared proxy port would collide.

named-hosts/kyber/incus/default.nix

@@ -0,0 +1,23 @@
+{ pkgs, ... }:
+let
+  setup = pkgs.replaceVars ./setup.sh {
+    preseed = "${./preseed.yaml}";
+    networkScript = "${./network.sh}";
+    networkService = "${./kyber-incus-network.service}";
+  };
+in
+{
+  xdg.configFile."crabbox/config.yaml".source = ./crabbox.yaml;
+
+  # Ubuntu owns the privileged daemon and its security updates. A normal
+  # Home Manager switch installs the setup command without changing the host.
+  home.packages = [
+    (pkgs.writeShellApplication {
+      name = "kyber-incus-setup";
+      runtimeInputs = [ pkgs.jq ];
+      text = ''
+        exec ${pkgs.bash}/bin/bash ${setup}
+      '';
+    })
+  ];
+}

named-hosts/kyber/incus/kyber-incus-network.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Kyber Incus bridge egress through Docker firewall
+Requires=incus.service docker.service
+After=incus.service docker.service network-online.target
+Wants=network-online.target
+PartOf=docker.service incus.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/kyber-incus-network
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target

named-hosts/kyber/incus/network.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Docker's FORWARD policy otherwise drops Incus egress. Permit only traffic
+# from this bridge to the default uplink and its established return traffic.
+uplink="$(ip -4 route show default | awk '/default/ { for (i=1; i<=NF; i++) if ($i == "dev") { print $(i+1); exit } }')"
+if [ -z "$uplink" ] || [ "$uplink" = incus-crabbox ]; then
+  echo "Cannot determine the Incus egress uplink." >&2
+  exit 1
+fi
+iptables -w -S DOCKER-USER >/dev/null
+if ! iptables -w -C DOCKER-USER -i incus-crabbox -o "$uplink" -j ACCEPT 2>/dev/null; then
+  iptables -w -I DOCKER-USER -i incus-crabbox -o "$uplink" -j ACCEPT
+fi
+if ! iptables -w -C DOCKER-USER -i "$uplink" -o incus-crabbox -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null; then
+  iptables -w -I DOCKER-USER -i "$uplink" -o incus-crabbox -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+fi

named-hosts/kyber/incus/preseed.yaml

@@ -0,0 +1,49 @@
+# Owned pilot resources only. Do not modify the default profile or expose HTTPS.
+storage_pools:
+  - name: crabbox
+    driver: btrfs
+    config:
+      user.dotfiles: kyber-crabbox
+      size: 64GiB
+networks:
+  - name: incus-crabbox
+    type: bridge
+    config:
+      user.dotfiles: kyber-crabbox
+      ipv4.address: 10.203.0.1/24
+      ipv4.nat: "true"
+      ipv6.address: none
+projects:
+  - name: crabbox
+    config:
+      user.dotfiles: kyber-crabbox
+      features.images: "true"
+      features.profiles: "false"
+      features.networks: "false"
+      limits.instances: "2"
+      limits.virtual-machines: "0"
+      limits.cpu: "8"
+      limits.memory: 16GiB
+      restricted: "true"
+      restricted.networks.access: incus-crabbox
+profiles:
+  - name: crabbox
+    config:
+      user.dotfiles: kyber-crabbox
+      limits.cpu: "4"
+      limits.memory: 8GiB
+      limits.memory.swap: "false"
+      limits.processes: "1024"
+      security.privileged: "false"
+      security.nesting: "false"
+    devices:
+      eth0:
+        name: eth0
+        network: incus-crabbox
+        type: nic
+      root:
+        path: /
+        pool: crabbox
+        size: 24GiB
+        limits.write: 10MiB
+        type: disk

named-hosts/kyber/incus/setup.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [ "$(uname -s)" != Linux ] || [ "$(hostname -s)" != kyber ] || [ "$(id -un)" != ubuntu ]; then
+  echo "Run kyber-incus-setup as ubuntu on Kyber only." >&2
+  exit 1
+fi
+
+sudo -n true
+if ! command -v incus >/dev/null 2>&1 || ! command -v mkfs.btrfs >/dev/null 2>&1; then
+  sudo -n apt-get update
+  sudo -n env DEBIAN_FRONTEND=noninteractive apt-get install -y incus incus-client btrfs-progs
+fi
+sudo -n systemctl enable --now incus.service
+
+# Always address the local daemon, regardless of the invoking user's remotes.
+incus_local() {
+  sudo -n env INCUS_SOCKET=/var/lib/incus/unix.socket incus --force-local --project default "$@"
+}
+
+# Preseed can update resources. Refuse to adopt an unrelated resource with a
+# matching name, or hide its API error by treating it as a missing resource.
+for kind in storage network project profile; do
+  name=crabbox
+  [ "$kind" != network ] || name=incus-crabbox
+  inventory="$(incus_local "$kind" list --format=json)"
+  if ! jq -e --arg name "$name" '
+    all(.[]; .name != $name or .config["user.dotfiles"] == "kyber-crabbox")
+  ' <<<"$inventory" >/dev/null; then
+    echo "Refusing unmanaged Incus $kind $name." >&2
+    exit 1
+  fi
+done
+
+# Refuse to take over an existing route on another interface. The native
+# preseed owns the address, and reapplying it to our own bridge is allowed.
+routes="$(ip -4 route show 10.203.0.0/24)"
+if printf '%s\n' "$routes" | grep -v ' dev incus-crabbox ' | grep -q .; then
+  echo "10.203.0.0/24 is already routed outside incus-crabbox." >&2
+  exit 1
+fi
+
+incus_local admin init --preseed <'@preseed@'
+
+sudo -n install -m 0755 '@networkScript@' /usr/local/sbin/kyber-incus-network
+sudo -n install -m 0644 '@networkService@' /etc/systemd/system/kyber-incus-network.service
+sudo -n systemctl daemon-reload
+sudo -n systemctl enable kyber-incus-network.service
+sudo -n systemctl restart kyber-incus-network.service
+
+# This is root-equivalent Incus administration, for the existing host operator
+# only. New SSH sessions acquire the group; no daemon or user-manager restart.
+if ! id -nG ubuntu | tr ' ' '\n' | grep -qx incus-admin; then
+  sudo -n usermod -aG incus-admin ubuntu
+fi
+echo "Incus pilot configured. Open a new SSH session, then run crabbox doctor --provider incus."

spec/coverage_spec.sh

@@ -396,6 +396,10 @@ It 'has behavior tests for named-hosts/kamino/activate.sh'
 The path "tests/test_kamino_activation.py" should be exist
 End
 
+It 'has behavior tests for named-hosts/kyber/incus/setup.sh and network.sh'
+The path "tests/test_kyber_incus.py" should be exist
+End
+
 It 'has spec file for named-hosts/kyber/activate-ip-forwarding.sh'
 The path "spec/activate_kyber_spec.sh" should be exist
 End
@@ -640,6 +644,8 @@ named-hosts/kyber/activate-fish-ssh-compat.sh
 named-hosts/kyber/activate-ip-forwarding.sh
 named-hosts/kyber/activate-sshd.sh
 named-hosts/kyber/activate-user-service-priority.sh
+named-hosts/kyber/incus/network.sh
+named-hosts/kyber/incus/setup.sh
 named-hosts/kyber/prepare-containerd-disk.sh
 named-hosts/kyber/rekey-galactica.sh
 named-hosts/kyber/setup.sh