Skip to content

remote.md

Latest commit

 

History

History
390 lines (337 loc) · 21.3 KB

remote.md

File metadata and controls

390 lines (337 loc) · 21.3 KB
Raw

Remote builds

justrust can run check, clippy, and test on a bigger machine and stream the compact output back, choosing local or remote per build from measured walls. Phase 4 of toolchain.md. Why remote, the shared cache, and licensing are in remote-strategy.md.

Hosted builds (the Jcode build fleet) are the remote path going forward. The aws backend (a machine in your own AWS account) is legacy and is removed once hosted matches it. The ssh backend (any machine you have) was removed: a backend.json naming it reads as no backend.

justrust login                      sign in: selects hosted builds
justrust remote up                  start the machine (hosted, or create the AWS one)
justrust remote use hosted|aws|off  choose the backend, or turn remote off
justrust remote env                 JR_REMOTE_KIND/DEST/SSH for scripts (starts the machine)
justrust remote status              backend, machine health, sync daemon, routing
                    --json, --waybar for a Waybar module, --watch N to refresh
justrust remote ssh [cmd]           shell on the current backend, or run one command
justrust remote down                stop the machine (disk and caches are kept)
justrust remote destroy --yes       terminate the AWS machine and delete its disk
justrust remote check|test|build|clippy <cargo args>
                                    run there explicitly (starts the machine if needed)

With no backend configured nothing remote ever happens. Every build runs locally and routing costs one failed stat.

Measured (2026-10-09, c7i.8xlarge spot in us-west-2, 29 ms round trip)

case local (16 threads) remote
no-op check -p jcode-base, warm 0.3 s 0.6 s (sync 32 ms)
same, before the sync daemon (rsync per call) 0.3 s 1.1 s (sync 0.4 s)
one-line edit in jcode-base, check -p jcode-base 3.6 to 4.1 s 3.5 to 3.6 s
test -p jcode-desktop-model, warm 4.4 s 1.6 s
cold check of Jcode Desktop, 883 units ~180 s 44 s
one-line edit in justrust, check 2.5 s 2.5 s

The sync daemon took sync off the critical path: a warm no-op went from 1.1 s (0.4 s of rsync) to 0.6 s (32 ms of sync, the rest is ssh and remote cargo startup). Remote wins clearly on cold builds, big recompiles, and test suites, ties on small edits, and still loses on no-ops. That split is exactly what routing measures per command.

Benchmarking remote builds

Two questions, two tools:

  • Does the remote route hold up from the agent's seat? bench/versus.py run ... --remote (or --synth SEED --remote) adds a remote stack: JUSTRUST_REMOTE=force, run from the laptop, so its wall includes sync, ssh and output streaming, exactly what an agent waits for. A sample that fell back to a local build is an error, not a number (the justrust remote: ok on ... line must be there). Instruction counts are dropped for this stack: the build tree runs on the machine.
  • How fast is the machine itself? bench/on-remote.sh start -- bench/versus.py run --synth 1 runs any benchmark on the machine: it copies the justrust binary and the repo, ships the pinned snapshot trees as git archive tarballs (JUSTRUST_BENCH_SRC), enables perf counters (aws), keeps the idle stop from firing, and runs in tmux. status, attach, and fetch (copies results back as *-remote-<type>.json, the type read from the machine's instance metadata). It reaches the machine through justrust remote env, so it works on hosted and aws alike.

The first --remote run found two bugs that made every remote build of the Desktop snapshot slower than local, now fixed: a workspace outside git was mirrored crate by crate, so a build script's rerun-if-changed root Cargo.toml was missing and 8 crates recompiled on every call (no-op test 11.4 s remote vs 0.5 s local; 0.5 s remote after the fix), and the machine lacked libraries Desktop links (libxkbcommon-x11, libX11, dbus, EGL).

Backends

~/.justrust/remote/backend.json holds the choice (src/remote_backend.rs). Everything above the backend (daemon, protocol, routing) only needs an ssh-reachable machine with a shell.

backend what set up with status
aws one machine in your own AWS account, created and controlled through the aws CLI justrust remote up built
hosted justrust's own build fleet, billed per build (card required: first month of $20 free, then from $20/mo for 5,000, 2,500 per $10) (src/remote_hosted.rs, src/account.rs) justrust login client built against HOSTED_BUILDS.md
none every build is local justrust remote use off

A machine created before backend.json existed counts as aws.

Hosted

Funnel: justrust login (device flow, client_name: justrust-cli, opens the browser and prints the URL; without a TTY it prints "Ask your user to open"), which writes JCODE_API_KEY, JCODE_ACCOUNT_ID, JCODE_ACCOUNT_EMAIL, JCODE_TIER to ~/.config/jcode/jcode-subscription.env (0600, shared with Jcode, other lines kept), selects hosted when no backend is set, and prints builds left. justrust login --no-wait saves the flow in hosted/pending-login.json (0600) and exits. Every later justrust command sends at most one token poll per interval and completes the login once approved (src/account.rs). justrust logout removes only the key lines. justrust upgrade [--plan 10] [--no-wait] opens Stripe Checkout and waits (up to 10 min) for builds.can_build.

Account calls go to JCODE_API_BASE (default https://api.jcode.sh/v1), fleet calls (build/...) to JUSTRUST_BUILD_BASE (default https://build.jcode.sh/v1). HTTP goes through curl with the key on stdin (--config -), never in argv or a URL. Only https, or http on loopback for tests.

api   POST /auth/device {client_name}  POST /auth/token {device_code} (428 pending, 429 slow_down)
api   GET  /me                         builds {trial_remaining, included, included_used, can_build, next_step}
api   POST /billing/subscribe {plan_usd} -> {url}
build POST /build/runs {run_id}        200 charged | 402 {error:{next_step: subscribe|raise_limit}}
build POST /build/host/connect  {"public_key"}: 202 while starting, 200 {address, port, user, host_keys, builds}
build GET  /build/host          {state, builds}
build POST /build/host/stop     {state}
  • Every hosted remote run first POSTs build/runs with the run's id (the same id it is recorded under, so retries are free). Any non-200 means no remote run: a 402 prints one line for the agent to relay (remote builds need a card on file: run justrust upgrade ... or monthly usage limit reached: ... https://jcode.sh/account), the build runs locally, and auto routing stays local for the 10 minute backoff.
  • Routing may start a stopped hosted machine when signed in and the local build is predicted over 20 s, or there is no local sample and the workspace has no target/debug. Otherwise hosted counts as up only while the ssh master is alive or within 60 s of a ready.
  • Each connect generates a fresh ed25519 key, polls connect every 2 s (up to 6 min for a first creation), pins exactly the returned host keys (StrictHostKeyChecking=yes), and opens the ssh ControlMaster at once: the key is accepted for only 60 s, the master then carries every ssh and rsync for 10 minutes after last use.
  • remote status shows account and builds left (cached 30 s). --json adds signed_in, pending_login_url, builds, next_step.
  • remote down asks the server to stop the machine and closes the master.
  • The user there is ubuntu, with no general sudo. Mirrored paths under /home and /Users are plain mkdir -p. Other absolute paths go through sudo -n /usr/local/sbin/jcode-mkdir <path>.

up, down, ssh, env, and status work for aws and hosted. destroy manages the AWS machine only. remote status on hosted shows the machine type, region, and spot from the machine's own instance metadata (IMDSv2, read by the probe), the same fields --json reports for aws (instance_type, region, spot, public_ip).

The AWS machine

  • Region us-west-2. Measured round trip from the dev laptop: 29 to 42 ms. us-west-1 is closer (21 ms) but the account's vCPU quota there is 5. Pass --region.
  • c7i.8xlarge spot (32 vCPUs, 64 GB): about $0.50 to $0.55/h spot, $1.43/h on demand (--on-demand). A persistent spot request with stop on interruption, so the disk survives.
  • 200 GB gp3 at 6000 IOPS and 500 MB/s, kept across stops, so warm target dirs and caches survive.
  • Idle stop after 30 minutes (--idle-minutes). A systemd timer records activity (load above 1.0 or an interactive login) and powers off after the idle window. Stop, not terminate: the next remote up boots in about 30 s with everything still on disk.
  • ssh only from this machine's public IP. The security group justrust-remote is updated with the current IP on every up, and again when a status probe times out (new network). The key is ~/.justrust/remote/id_ed25519 and is used only for this machine.
  • First boot. cloud-init installs build-essential, clang, lld, mold, cmake, protobuf, poppler, xvfb, and the -sys libraries the Jcode projects link against (wayland, xkbcommon, alsa, fontconfig, freetype, xcb, vulkan), then rustup stable, and writes /var/lib/justrust/ready. status shows setup until then.

The sync daemon

justrust __daemon (src/remote_daemon.rs), one per user, started by the first remote build and kept while builds keep coming. It exits after an hour without clients or when the backend changes.

  1. Source roots. cargo metadata lists the workspace and every path dependency. Each is widened to its git root and nested roots are dropped, so Jcode Desktop mirrors ~/jcode-desktop and ~/jcode. Cached per lockfile mtime.
  2. Same absolute paths. Roots land at the same paths on the machine, so relative path dependencies resolve and diagnostics point at local files. A machine where those paths cannot be created gets a prefix instead.
  3. First mirror by rsync, then only changes travel. A scan (git ls-files plus one stat per file, about 8 ms for the 2,100 files of Jcode) is compared with the manifest of what the machine has, and each changed file goes out as one Write frame. Above 300 changed files it uses rsync again. Skipped: .git, gitignored files, target/, and files over 8 MB.
  4. Push on save. inotify on the roots triggers that scan as files are saved, so by the time a build asks there is usually nothing left to send. The scan, not inotify, decides what is pushed, so a missed or overflowed watch can delay a push but never lose one.
  5. One ssh session to justrust __agent on the machine (src/remote_agent.rs) carries every push and every run. Several runs can be in flight at once (parallel agents).
  6. Remote justrust is rebuilt on the machine from this checkout's source whenever the local binary changes, so remote output, slots, and depcache match.

justrust remote status shows the daemon from its socket without touching the network: connected or idle, roots mirrored, files pushed. The log is ~/.justrust/remote/daemon.log.

Protocol

src/remote_proto.rs. Every frame is [u32 header len][header JSON][u32 payload len][payload], big endian. The same framing runs over the daemon's unix socket (client and daemon) and over the ssh channel (daemon and agent), so a hosted backend only has to provide a byte stream.

  • Hello {version} first in both directions. The agent refuses other versions, and a client restarts a daemon built from a different binary.
  • Write {path, mode} (payload is the content, written to a temp file and renamed), Delete, Mkdir, applied in order.
  • Barrier {id} and BarrierAck {id}: an ack means every earlier frame is applied, so the tree matches.
  • Run {id, cwd, args, env, roots, start, run_id}. The daemon rescans the roots, pushes what is left, waits for the barrier, answers Flushed {files, ms, machine, prefix}, then relays Out {stream} frames and Exit {code} (payload: the remote run's summary.json, so the run is recorded locally too, marked remote, under the same run id).
  • Cancel {id} sends SIGINT to the run's process group.
  • Error {id, msg} is an infrastructure failure, never a build failure.
  • Ping and Pong {roots, machine, pushed, connected, build} for status.
  • Shutdown stops the daemon.
  • Warm (client to daemon): connect in the background if the machine is known to be up.
  • v3, peer cache, in both directions between daemon and agent and from shims to their relay: CacheQuery {id, base} / CacheLines {lines, outs, busy, bw}, CacheFetch {id, out} / CacheData {last, ok} (bundle in 256 KB chunks so build output keeps flowing), CacheKeys / CacheKeySet (payload: every base key), CacheList {since} / CacheJournal {items, next}, CacheOffer {items} / CacheWant {outs}, CachePut {base, line, out, last}. The agent still serves v1 and v2 daemons, without the cache frames.

Peer dependency cache

src/peercache.rs. The laptop and the build machine share their dependency caches (~/.justrust/cache/depcache) over the sync daemon's ssh session, with no server in between. It complements the HTTP tier (src/depcache_remote.rs, build.jcode.sh), which serves account-wide and shared entries.

 laptop                                       build machine
 rustc shim --unix--> daemon ==ssh frames==> agent <--unix-- rustc shim
              peer.sock   (Relay)     (Relay)   peer.sock

Both ends run a relay: it answers the other side's requests from its own cache, and forwards its own rustc shims' requests across the link.

when what happens
a dependency unit misses locally (and in the HTTP tier) the shim asks its relay. The relay answers at once from a copy of the peer's key set when the peer lacks the key, else asks the peer for its manifest lines. The shim picks the newest line whose inputs hash the same here, then fetches only if bundle bytes / 3 / measured throughput (plus queueing behind bundles in flight) is under 70% of the entry's recorded compile time and saves at least 0.15 s
a build stored new entries on either side the daemon's background sync (every 5 s, only while no remote run is relayed) offers the laptop's new entries to the machine and pulls the machine's, most compile seconds per byte first, and only for compilers the receiving side has used (peer-toolchains)
a hit came from the HTTP tier it is journaled too, so the other machine gets it on the next sync

Order of tiers in the shim: local, HTTP, peer. Run summaries count peer hits separately (depcache 66 hit (66 peer)/310 miss). JUSTRUST_PEER_EXPLAIN=1 prints why each lookup did not fetch (absent, inputs differ, not worth it with the numbers, no relay). JUSTRUST_PEER_CACHE=0 turns it off per build.

What makes entries portable between machines (current depcache scheme 8):

  • the cargo home and target dir are placeholders in keys and manifest lines. Remote runs send JUSTRUST_CLIENT_CARGO_HOME, and the agent builds with CARGO_HOME at the laptop's path (a directory whose registry and git link to the machine's own), because registry source paths are compiled into crate metadata and every dependent's SVH;
  • metadata-only .rmeta inputs are keyed by their Rust metadata, but code-bearing --extern artifacts (.rlib, .so, .dylib, .dll) are keyed by complete content. Bundled native objects or executed proc-macro helpers can change behavior without changing lib.rmeta or .rustc; a real metadata-only cache collision returned stale native code. Memo namespace user.justrust.m3 invalidates the old metadata-only artifact identities. Full artifact identity may cost more or reduce cross-host hits, so do not claim metadata-only portability as a sound optimization;
  • dependency units compile with --remap-path-prefix=<target>=/justrust-target, so crates that include! generated OUT_DIR files (serde, toml, ring, aws-lc) produce the same metadata in every target dir;
  • CARGO_HOME and CARGO_TARGET_DIR are not keyed (cargo passes them to rustc only when set);
  • proc macros and build scripts are restored only where the host glibc is at least the newest GLIBC_x.y they need (read from .dynstr at store);
  • gitignored Cargo.lock files are mirrored, so the machine resolves the same dependency versions.

Safety is the same as a local hit: an entry is restored only if every input file hashes the same here, unpack verifies every binary output's content hash, and a peer can only write o/<32 hex> and m/<32 hex>.jsonl.

Routing uses the caches too: a command with no history in a workspace with no build output goes remote (that is where the dependency cache is warm and the cores are), and a local build that will compile dependencies (cold, or slow history) asks the daemon to connect in the background (Warm, never starts a stopped machine) so its shims can fetch from the machine.

Routing

src/route.rs. Every agent-mode check, clippy, and test asks the router first. run and debug build always stay local because their artifacts are needed here and a debug binary (100 MB, mostly debug info) costs more to bring back than to link. An optimized build (--release or a custom profile, host target) routes on history like check: tried remotely once the local build takes over 5 s, kept remote while it wins, and its binaries come back through src/remote_artifacts.rs (justrust __artifacts on the machine): a zstd --patch-from delta against the copy fetched last time, or a full zstd copy after a local build. The fetch is part of the client wall the router compares. FINDINGS 45.

JUSTRUST_REMOTE controls it:

  • unset (default): signed in to hosted builds (JCODE_API_KEY from an account API key or justrust login, hosted backend) means remote for check, clippy, and test, starting the hosted machine if needed. The 10 minute failure backoff still applies. Any other backend behaves as auto.
  • auto or router: remote only when predicted faster. Never starts a machine (except a hosted start the router predicts pays off), never starts anything when no backend is configured, and never touches the network to decide.
  • 0 or off: always local, nothing is read.
  • 1 or force: remote for check, clippy, and test, starting the AWS machine if needed.

The prediction uses the run index: for this cwd and these exact args, the median of the last 5 local walls against the median of the last 5 remote client walls (what the caller waited, sync and ssh included). Remote must win by 20% and by at least 0.5 s. A command with no recent remote sample and a local median over 3 s tries remote once to get one. Remote samples expire after 2 days, so a command that lost is re-explored now and then. For the AWS backend, auto mode also requires evidence the machine is alive (a recent probe, run, or IP within the idle window), since it stops itself without telling us.

JUSTRUST_REMOTE_EXPLAIN=1 prints why a build stayed local.

Failure and fallback

Local is always the safe answer.

  • Any infrastructure problem (cannot reach the machine, daemon or agent error, lost connection) becomes an Error frame. The client prints one line, builds locally, and auto routing stays local for 10 minutes (~/.justrust/remote/route-fail).
  • If remote output was already shown when the failure hit, the client exits with an error instead of rebuilding, so output is never duplicated.
  • The daemon never assumes the machine is in sync: after a reconnect every root is rescanned against a fresh rsync.
  • A stopped AWS machine is never started by auto routing. A stale cached IP is not trusted past the idle window.
  • remote use off stops the daemon and removes backend.json.

Local state

~/.justrust/remote/
  backend.json    hosted | aws
  state.json      AWS: instance id, region, type, spot, idle minutes
  id_ed25519      AWS: ssh key for the machine
  known_hosts     AWS: its host key
  ip              AWS: last known IP of the running machine (cleared on down)
  probe.json      AWS: last ssh probe, reused by status for 30 s
  probe-hosted.json  hosted: last probe of the hosted machine
  hosted/         hosted: id_ed25519 (fresh per connect), known_hosts (pinned),
                  state.json {address, user, port, ready_at}, status.json (30 s)
  price.json      AWS: hourly price, cached for an hour
  daemon.sock     sync daemon socket (daemon.log, daemon.lock beside it)
  roots.json      source roots per lockfile
  route-fail      last infrastructure failure (10 minute local backoff)
  peer-offsets.json  how far each side's store journal was synced
  events.log      remote runs and machine events

Waybar

justrust remote status --waybar prints one line of custom-module JSON: round trip, temperature when the machine exposes one, cpu, ram, disk, uptime, probe age, and the sync daemon badge (sync 3r 17↑ means connected, 3 roots mirrored, 17 files pushed, sync idle means the daemon holds no session). Hosted shows hosted and the builds left (4990b). Nothing is shown when remote builds are not set up. The probe is cached for 30 s and the daemon part reads only the local socket, so a 15 s interval is cheap. Classes: unconfigured, pending, setup, running, busy, stopped, error.