[WIP] miles dashboard by yueming-yuan · Pull Request #1654 · radixark/miles

yueming-yuan · 2026-07-13T22:07:47Z

Observability dashboard for miles RL post-training: an efficiency view (per-GPU wall-clock timeline: phase bands, GPU util, sglang engine overlay, rank carpet), a training-dynamics view (step metrics → per-sample table → token-level entropy/logprob drill-down), and a trajectory lifecycle view (per-sample generate/tool/queue swimlanes with segment-level weight-version staleness — the fully-async batch anatomy).

Design principles

Offline-first, file-mediated. The collector is the only writer; everything lands under {dump-details}/dashboard/ as append-only JSONL (high-rate streams hourly-partitioned). The server is a stateless reader — a finished (or crashed) run reopens identically from files alone.
Observability must never kill training. All hooks are fire-and-forget with rate-limited warnings; sinks batch (64 events / 2 s); no sink attached = no-op.
Minimal core footprint. Outside miles/dashboard/ + tests the whole feature is ~200 lines across 15 files (registry entry, CLI args, Timer event sinks, engine topology accessor, transport-free dump-helper extractions, and the trajectory lifecycle probes in generate_and_rm / multi_turn / openai record assembly — gated behind a stdlib-only seam in miles/utils/lifecycle.py, same layer as Timer.event_sinks).
Scales by construction: columnar in-memory store (polars, ~16 B/row), lazy partition reads (open parses nothing; queries O(window)), hard 4 h viewport with lazy pan, rank carpet + manual lane selection (rank:/node:/gpu:/engine:/role: grammar, URL-shareable), 32-lane render budget, binary heatmap endpoint.

Usage

# training: add to train args (train.py and train_async.py both wired)
--dump-details /path/to/dump --use-miles-dashboard

# view (live: --follow; post-hoc: without)
python -m miles.dashboard.serve --dump-details /path/to/dump --follow --port 7788

Validation

~140 dashboard unit/contract tests; full suite incl. tests/fast/rollout/ green (530 passed on an H200 devbox against real dump data).
Validated live against real runs: qwen3-30B-A3B colocate (8×H200, wait-ratio bubble discovery) and 4 train + 4 rollout fully async (role-split lanes, engine overlay, staleness).

Review guide

23 stacked single-purpose commits, each self-contained with its tests — commit-by-commit is the intended review path. Bottom-up: store → dump reader → server/SPA → timeline → scraper/sampler/collector → core wiring → carpet/selection → scaling (columnar/partitions/viewport) → trajectory lifecycle (probes → sink → columns → views).

WIP

Agentic L2 message-structure projection pending dump-format alignment.
SessionRecord.request_timestamp (+3 lines) only if sglang chat meta_info lacks e2e_latency — to verify on a live engine.
Based on 01a6d7bb7; will rebase onto current main (the in-flight session-server refactors were designed for — lifecycle metadata rides the Sample through assembly, pinned by test_lifecycle_metadata.py).

🤖 Generated with Claude Code

gemini-code-assist

Code Review

This pull request introduces a comprehensive training-dynamics and efficiency dashboard for miles runs, featuring a live telemetry collector, GPU utilization samplers, sglang engine metric scrapers, and a standalone FastAPI server with a static single-page application. The feedback is highly constructive, highlighting a critical concurrency race condition in the collector's router setup that could leak scraper threads. Additionally, the reviewer correctly points out multiple instances where assert statements are used for argument and CLI validation, recommending they be replaced with ValueError or parser.error() to prevent them from being optimized away under Python's -O flag.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-07-13T22:09:49Z

+        with self._lock:
+            previous = self._scraper
+            if previous is not None and previous.router_addr == router_addr and previous.mode == mode:
+                return
+            self._scraper = None
+        if previous is not None:
+            previous.stop()
+        kwargs = dict(
+            mode=mode,
+            router_addr=router_addr,
+            engine_addrs=self._current_engine_addrs,
+            interval=self.config.scrape_interval_seconds,
+            whitelist=self.config.metric_whitelist,
+        )
+        if self._scraper_http_get is not None:
+            kwargs["http_get"] = self._scraper_http_get
+        scraper = SglangScraper(self._append, **kwargs)
+        with self._lock:
+            self._scraper = scraper
+        scraper.start()


There is a concurrency race condition in set_router. If multiple threads call set_router concurrently, a previously created scraper can be overwritten in self._scraper without being stopped, leading to leaked running scraper threads. Creating the new scraper inside the lock and assigning it to self._scraper before releasing the lock ensures that concurrent calls correctly identify existing scrapers and avoid duplicate/leaked threads.

Suggested change

with self._lock:

previous = self._scraper

if previous is not None and previous.router_addr == router_addr and previous.mode == mode:

return

self._scraper = None

if previous is not None:

previous.stop()

kwargs = dict(

mode=mode,

router_addr=router_addr,

engine_addrs=self._current_engine_addrs,

interval=self.config.scrape_interval_seconds,

whitelist=self.config.metric_whitelist,

)

if self._scraper_http_get is not None:

kwargs["http_get"] = self._scraper_http_get

scraper = SglangScraper(self._append, **kwargs)

with self._lock:

self._scraper = scraper

scraper.start()

with self._lock:

previous = self._scraper

if previous is not None and previous.router_addr == router_addr and previous.mode == mode:

return

kwargs = dict(

mode=mode,

router_addr=router_addr,

engine_addrs=self._current_engine_addrs,

interval=self.config.scrape_interval_seconds,

whitelist=self.config.metric_whitelist,

)

if self._scraper_http_get is not None:

kwargs["http_get"] = self._scraper_http_get

new_scraper = SglangScraper(self._append, **kwargs)

self._scraper = new_scraper

if previous is not None:

previous.stop()

new_scraper.start()

gemini-code-assist · 2026-07-13T22:09:50Z

+    assert args.dump_details is not None, (
+        "--use-miles-dashboard writes telemetry under {dump-details}/dashboard/ and the "
+        "trajectory views read the rollout/train dumps, so --dump-details is required"
+    )


Using assert for validating command-line arguments or function arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

Suggested change

assert args.dump_details is not None, (

"--use-miles-dashboard writes telemetry under {dump-details}/dashboard/ and the "

"trajectory views read the rollout/train dumps, so --dump-details is required"

)

if args.dump_details is None:

raise ValueError(

"--use-miles-dashboard writes telemetry under {dump-details}/dashboard/ and the "

"trajectory views read the rollout/train dumps, so --dump-details is required"

)

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+def collector_config_from_args(args, *, start_ts: float) -> CollectorConfig:
+    if args.dashboard_sglang_metrics is not None:
+        whitelist = tuple(m for m in args.dashboard_sglang_metrics.split(",") if m)
+        assert whitelist, f"empty --dashboard-sglang-metrics: {args.dashboard_sglang_metrics!r}"


Using assert for validating function arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

Suggested change

assert whitelist, f"empty --dashboard-sglang-metrics: {args.dashboard_sglang_metrics!r}"

if not whitelist:

raise ValueError(f"empty --dashboard-sglang-metrics: {args.dashboard_sglang_metrics!r}")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+        interval: float = 1.0,
+        nvml=None,
+    ):
+        assert interval > 0, f"{interval=}"


Using assert for validating constructor arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

Suggested change

assert interval > 0, f"{interval=}"

if interval <= 0:

raise ValueError(f"interval must be positive, got {interval}")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+    if handle is None:
+        return
+    # a None ip here is a wiring-order bug, not runtime flakiness: fail loud
+    assert args.sglang_router_ip is not None, "register_router must run after start_rollout_servers"


Using assert for validating function arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

if args.sglang_router_ip is None: raise ValueError("register_router must run after start_rollout_servers")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+    if args.demo:
+        import tempfile
+
+        dump_dir = make_demo_dir(Path(tempfile.mkdtemp(prefix="miles_dashboard_demo_")))
+    else:
+        assert args.dump_details is not None, "--dump-details is required (or use --demo)"
+        dump_dir = Path(args.dump_details)
+    assert dump_dir.is_dir(), f"--dump-details directory not found: {dump_dir}"


Using assert for validating CLI arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Using parser.error() is standard and safer for CLI argument validation.

Suggested change

if args.demo:

import tempfile

dump_dir = make_demo_dir(Path(tempfile.mkdtemp(prefix="miles_dashboard_demo_")))

else:

assert args.dump_details is not None, "--dump-details is required (or use --demo)"

dump_dir = Path(args.dump_details)

assert dump_dir.is_dir(), f"--dump-details directory not found: {dump_dir}"

if args.demo:

import tempfile

dump_dir = make_demo_dir(Path(tempfile.mkdtemp(prefix="miles_dashboard_demo_")))

else:

if args.dump_details is None:

parser.error("--dump-details is required (or use --demo)")

dump_dir = Path(args.dump_details)

if not dump_dir.is_dir():

parser.error(f"--dump-details directory not found: {dump_dir}")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+        if mode == ScrapeMode.ROUTER:
+            assert router_addr is not None, "router mode needs router_addr"
+        else:
+            assert engine_addrs is not None, "direct mode needs an engine_addrs provider"


Using assert for validating constructor arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

Suggested change

if mode == ScrapeMode.ROUTER:

assert router_addr is not None, "router mode needs router_addr"

else:

assert engine_addrs is not None, "direct mode needs an engine_addrs provider"

if mode == ScrapeMode.ROUTER and router_addr is None:

raise ValueError("router mode needs router_addr")

if mode != ScrapeMode.ROUTER and engine_addrs is None:

raise ValueError("direct mode needs an engine_addrs provider")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+            else:
+                raise ValueError(f"unknown lane selector kind {kind!r} in {token!r}")
+        return selected
+


Using assert for validating function arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

Suggested change

if x_buckets < 2:

raise ValueError(f"x_buckets must be at least 2, got {x_buckets}")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+
+def stride_downsample(xs, ys, max_points: int) -> tuple[np.ndarray, np.ndarray]:
+    xs, ys = np.asarray(xs), np.asarray(ys)
+    assert max_points >= 2, f"{max_points=}"


Using assert for validating function arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

Suggested change

assert max_points >= 2, f"{max_points=}"

if max_points < 2:

raise ValueError(f"max_points must be at least 2, got {max_points}")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

gemini-code-assist · 2026-07-13T22:09:50Z

+    """Bucketed downsampling keeping each bucket's min and max, so spikes and
+    dips survive. Output length is at most ``max_points``."""
+    xs, ys = np.asarray(xs), np.asarray(ys)
+    assert max_points >= 2, f"{max_points=}"


Using assert for validating function arguments is discouraged because assertions can be optimized away when Python is run with the -O flag. Raising a ValueError is preferred for argument validation.

Suggested change

assert max_points >= 2, f"{max_points=}"

if max_points < 2:

raise ValueError(f"max_points must be at least 2, got {max_points}")

References

Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

split_train_data_by_dp_local, process_rollout_data_shard, and save_debug_train_data_for_rank carry the pure logic; the original functions keep their signatures and wrap ray/torch.distributed transport around them. No behavior change.

…eline tests/fast/dashboard/dummy_dump.py pushes fake samples through the actual save_debug_rollout_data / convert_samples_to_train_data / split_train_data_by_dp_local / process_rollout_data_shard / save_debug_train_data_for_rank chain, so dumping-logic changes break the reader tests instead of drifting silently. Building it this way surfaced a real reader bug, fixed here: raw_reward is stored batch-global (unlike every other per-shard column) and must be indexed by rollout position, not shard row.

…caches summary() computes the per-sample L1 table (masked per-token stats, spec accept rate, prefix-cache hit rate, batch-global raw_reward) and caches it as parquet keyed on source mtimes + SUMMARY_VERSION. groups() adds GRPO-group aggregates with zero_std degeneracy flags; step_aggregates() is the dump-derived L0 fallback. tokens() serves the per-token L2 payload (imp_ratio, lp_diff, entropy, advantages) over a clamped token window, decoding text via the dumped tokenizer; joined() bounds tensor memory with a per-rollout LRU.

yueming-yuan · 2026-07-14T03:07:40Z

Pushed a batch of fixes from live dogfooding on real runs (colocate 8×H200 3-step, 4+4 fully-async, and a 744B disaggregated agentic run):

Live phase bands (open intervals): Timer.start now emits an OPEN marker; in-progress phases render as growing bands instead of appearing only when they end — the first agentic rollout (40 min+) is visible from second one.
External-engine overlay: pure sglang engines (no miles actor) are synthesized into the topology from scraper worker addrs; the frontend attaches their series by node when GPU placement is unknown.
Global lane numbering: g{index} labels everywhere + g:0-31 selection grammar; per-node relative ids demoted to secondary text.
Spaced default selection: opening the timeline seeds 8 evenly-spaced lanes as a real, editable selection (chips/URL) instead of first-N.
Data freshness badge: live · data → HH:MM:SS from the newest server-side timestamp (a stalled collector reads as stale).
Batch anatomy: zoom/pan; per-sample lifecycle lane embedded on the token (L2) page; single-turn/aborted gen spans close at attempt end (no more phantom uniform end times); session-server agent gaps emitted as tool-wait spans, bounded exactly by a new optional SessionRecord.request_timestamp (+3 lines, wire-compatible).
Robustness: dumps without train log_probs degrade to null columns instead of HTTP 404; Cache-Control: no-cache on the SPA so upgrades never serve a stale frontend (existing browsers need one hard refresh); trainer log now prints the serve command.

Same 23-commit stack shape; all commits amended in place. 149 dashboard tests green.

make_app(store, reader) exposes /api/meta, /api/metrics (store keys plus dump/<column> aggregates so L0 works without metrics.jsonl), rollout summary/groups tables, and the per-token payload. Error mapping: DumpStillWriting -> 503, missing rollout/sample -> 404, bad input -> 400; NaN/inf sanitize to null. serve.py runs it standalone over any --dump-details directory, with --follow tailing telemetry streams.

yueming-yuan · 2026-07-14T21:55:26Z

Batch: metrics L0 redesign (wandb/grafana parity) — force-pushed d34d87b04.

Category page replaces the pinned-chart L0 (PR-6): left sidebar with one button per wandb category (rollout / perf / train / eval — names verbatim from the log() fan-out), single-select, every key of the active category renders as a line chart. Filter box narrows within the category.
sglang category (PR-8): appears when scraped engine metrics exist (/api/meta.engine_metric_keys, derived from the data, not a hardcoded whitelist). Each metric renders as a multi-line chart, one thin line per engine, hover carries engine identity. Verified against the qwen30b-async3s2 run: 8 metrics incl. disagg queues.
No perpetual repaint (PR-6/8): follow mode still polls every 5s, but each chart keeps a data signature and repaints only when its series actually changed — a quiet run leaves the page visually still.
Chart click-through removed; steps is top-level nav (PR-6): charts are hover-only; the per-step data view (batch anatomy, samples, groups) is reached via a steps link next to metrics · timeline, landing on the newest train step.
dump/* gated server-side (PR-5): dump-derived keys are advertised only for dump-only dirs (no telemetry stream); with telemetry the L0 never reads dumps. mean_abs_lp_diff's telemetry counterpart is train_inference_abs_diff; composition stats moved to the step page.
mixed-version frac stat box (PR-22) added to the step-page statgrid next to zero-std groups.
Fix: render crash from an interim bad merge (crumbs referenced meta before fetch).

Verification: local tip 151 passed; devbox tests/fast/dashboard 152+3 passed (one failure was a stale local Ray cluster on the box — cleared), tests/fast/rollout 388 passed. Devbox serve on :7788 updated to this tip.

yueming-yuan · 2026-07-14T22:08:29Z

sglang page polish (follow-up, force-pushed d32fd8c3e): per-engine line colors from a fixed 10-color categorical palette assigned by sorted-addr index (stable across refreshes, cycles past 10) — PR-8; metrics grid now two charts per row and the category sidebar narrowed to 160px — PR-6. Verified on the qwen30b-async3s2 run.

yueming-yuan · 2026-07-14T22:14:30Z

Metrics interaction follow-up (force-pushed 5f11f2747): all L0 charts now support drag-to-zoom on the x-range (translucent selection band, double-click resets, zoom survives live data refreshes since it lives on the canvas, per-chart independent) — implemented once as installDragZoom in charts.js and shared by drawChart (PR-6) and drawMultiLine (PR-8). Sub-4px presses still count as clicks, so the step-page scatter's open-sample behavior is unchanged. Multi-line colors are assigned by full-list index so an engine keeps its color when zoom drops other series; x labels stay run-relative under zoom. Layout: sidebar 200px, chart columns capped at 500px.

yueming-yuan · 2026-07-14T22:18:36Z

Nav + step-jump follow-up (force-pushed 874548a1b): top bar is now dark with bold 15px nav blocks at equal spacing — Metrics · Compute Utilization · Data / Trajectories — whole-block gray hover, accent underline marks the active section (replaces the › efficiency breadcrumb). Step view header gained a number input: type a step and press Enter to jump straight to it (PR-6/8). Also fixed at its origin: PR-6's crumbs() was called before the meta fetch (the TDZ crash previously patched at PR-8 actually originated here).

yueming-yuan · 2026-07-14T22:23:31Z

Drag-zoom fix (force-pushed 49aa28397, PR-6): real drags route mousemove/mouseup to whatever element is under the pointer, so the previous canvas-local handlers lost any drag that strayed outside the 220px chart band (or crossed into the next chart) and the zoom never completed — synthetic-event tests had masked this. installDragZoom now anchors on mousedown, listens for move/up on window until release, and preventDefault()s the native text selection. Verified with genuine mouse input against the live run on :7788.

yueming-yuan · 2026-07-14T22:26:31Z

sglang legend (force-pushed 8839f1b18, PR-8): the sglang page gains an ENGINES panel on the right — one checkbox per scraped engine (color swatch matches its line), all checked by default. Unchecking hides that engine's line in every chart on the page and drops it from the y-scale. Colors are pinned to the page-wide sorted engine order (opts.colorIndex in drawMultiLine), so a hidden or zoom-clipped engine never shifts the others' colors; toggles repaint from each chart's cached series without refetching. Verified on the live run (2 engines).

yueming-yuan · 2026-07-14T22:32:06Z

Zoom rework + slow-load notice (force-pushed c7e322a77):

Why most drags "did nothing": the old x-zoom silently reset whenever the selected window held <2 data points — on 3-point step charts (rollout/perf/train of a 3-step run) nearly every reasonable drag hit that fallback; only wide sweeps (e.g. out to step 0) survived. The fallback is gone: zoom domains are now explicit, and a window with few or zero points just renders that window (PR-6/8).
Both axes: drag now selects a box — canvas._zoom = {x, y}. An axis participates only when dragged ≥10px, so a horizontal-ish sweep zooms x and lets y re-autoscale to the window instead of pinning y to hand wobble; the live overlay shows a full-height band until the drag gains y-extent. Marks are canvas-clipped to the plot area so out-of-domain data can't paint over margins. Double-click still resets.
Detokenize notice (PR-6): the sample page now says "loading sample… the first open of a step detokenizes its whole dump and can take several minutes" instead of a bare "loading…", so a minutes-long first open reads as work, not a hang.

Verified with real mouse input on the live run: box drag on gen_throughput zooms x to +6:40–+23:20 and y to the selected band; narrow drags on 3-point charts now zoom reliably.

yueming-yuan · 2026-07-14T22:36:11Z

Weight-version hue ring (force-pushed ae2b9ee36, PR-23): the batch-anatomy view now colors each weight version from a cyclic spectral ring — hsl(55 + (v·137.508) % 320, 70%, 46%): golden-angle steps keep consecutive versions far apart on the wheel for any version count, and the ring starts past the tool-wait amber band so version colors never read as span colors (gen green / tool amber / queue gray). Detailed rows color the v# label (mixed rows: each endpoint in its own color with a stale dash between); compact rows get a colored left tick per lane, two ticks when versions are mixed. Legend text updated. Verified on the live run: v1 lanes tick in the ring's 192.5° cyan.

yueming-yuan · 2026-07-14T22:50:27Z

Weight-version coloring, corrected semantics (force-pushed 222a80f41, PR-23): generation spans themselves are now colored by the weight version that produced them, using a sequential cyclic ring — hsl(55 + (105 + 25v) % 320, 69%, 37%). v0 is the familiar gen green, each +1 steps 25° (green → teal → blue → … → back around every ~13 versions), so neighbouring versions read as neighbouring colors and every version bump is a visible new shade. The ring skips the tool-wait amber band; tool/queue/consume and the Compute-Utilization view are untouched. Version ticks/labels reuse the same ring. Also fixed at the read side: coarse gen spans open before any version exists, so trajectory_lanes now backfills the span's version from the closing attempt_end event — old dumps get correct per-span versions without re-running. Verified on the live run: gen bars render versionColor(1) teal, matching the dump's weight_version=1 for all three steps.

yueming-yuan · 2026-07-14T23:12:08Z

Follow-up (force-pushed 7547be767, PR-23): removed the separate weight-version label column / left ticks from batch anatomy — the generation spans themselves carry the version color now, so the extra column was redundant. Detailed-mode left margin shrinks 150→96px; legend cleaned up. Note on the uniform color in the current run: the dump records weight_version=1 for all samples of all three steps (core-side accounting), so a single hue is the faithful rendering.

yueming-yuan · 2026-07-14T23:19:46Z

avg staleness stat (force-pushed 53b2f65ec, PR-22): the step page statgrid gains avg staleness = mean over samples of (trainer version at consume − sample's oldest generation version), with trainer-at-step-N = v(N+1) (one startup push + one per train step). Train view only; renders — when version data is absent. Verified on the live run: step 0 → 0, step 2 → 2 (trainer at v3 consuming v1 data).

Zero-dependency ES-module frontend served from /static: hand-rolled canvas charts (no vendored libs, no build step). L0 metric explorer with store + dump/ catalogs and click-through to steps; L1 rollout view with headline stats, reward-vs-length scatter, sortable sample table, GRPO groups tab, eval toggle; L2 token strip colored by imp_ratio / entropy / advantages with loss-mask dimming, prompt/response split, hover stats, and a per-token chart.

dummy_telemetry.py fabricates collector output through the real MetricStore write path (colocate cadence, long-tail drain, mid-rollout engine restart, util spike) and is the output contract the live collector must satisfy. Store gains lanes/topology-window/phase-lane queries — rollout_manager events expand onto rollout-engine lanes with clipping at topology boundaries — plus downsampled gpu/engine series and the per-step bubble strip. Server exposes them and flips the has_timeline/has_engine_series capabilities.

Canvas lane renderer per the design mockup: phase band (train_wait hatched as explicit idle), GPU util area, engine-metric overlay mapped onto each engine's lanes via topology windows, per-step bubble strip with click-to-zoom, wheel zoom / drag pan, hover inspection of phase, util, overlay and owning engine. serve --demo generates dump + telemetry demo data via the dummy generators for cluster-free development.

Router mode does one GET /engine_metrics per tick (sgl router fans out and labels samples with worker_addr); direct mode fetches each engine's /metrics concurrently for --use-miles-router runs, skipping failed engines. Names are normalized (sglang:x == sglang_x), whitelisted, and label-reduced; failures leave gaps with warnings rate-limited to one per 5 minutes. The sink is a plain callable, so no collector or Ray dependency.

Plain-Python NVML sampler (the collector will spawn it as a per-node Ray actor): daemon thread samples every physical device at 1 Hz, buffers locally, and pushes one batch per 5 s flush through the injected push callable. NVML being unavailable disables it with a single warning; a device failing mid-run (GPU reset) is skipped per tick with rate-limited warnings while the rest keep reporting.

…heus switch Ray-free by design: backend glue will wrap it in the named driver-node actor and spawn per-node samplers, so every behavior here is plain unit-testable. The contract test replays dummy_telemetry records through the RPC surface and asserts timeline queries match the fixture's direct writes exactly. Topology snapshots dedup on change; set_router resolves auto mode (miles router -> direct) and re-points the scraper on router restarts without holding the ingest lock during stop (deadlock). Disk failures stay loud on every flush while buffers drop oldest past a cap; the optional prometheus forwarding (default off) pushes sanitized latest-value gauges from caches that survive flushes.

Timer.end() now reports (name, start_ts, end_ts) to registered event_sinks before accumulating, giving observers wall-clock phase intervals instead of durations only. No sinks registered (the default) leaves behavior byte-identical; miles.dashboard registers one when enabled.

backend.init_dashboard implements the design's call matrix: the driver creates the named DashboardCollector actor (driver-node pinned) and spawns one GpuSampler actor per GPU node; secondaries resolve the named actor with a timeout and warn-once-then-noop; the rollout manager additionally registers the router (starting the scraper) and a rollout-role phase sink. hooks.py carries the Timer phase sink (batched, identity stamped lazily until torch.distributed knows the rank, exceptions swallowed with rate-limited warnings) plus register_train_actor / register_engines — the engine fingerprint costs one tuple compare per rollout step and re-registers topology only when actor handles change. args.py adds the --dashboard-* CLI group, validation, and the CollectorConfig builder.

…pology The complete core-file budget from the design doc, ~90 lines: MilesDashboardBackend adapter + registry entry in tracking_utils; --dashboard-* argument group and validation hookup; rollout_manager registers engine topology and wraps rollout/eval in timers (interval events only — perf metrics are unchanged); the train actor attaches the per-rank phase sink outside the main-rank gate; SGLangEngine gains the additive get_topology_info() (node-physical GPU ids matching NVML order, best-effort UUIDs). The e2e test drives the real tracking registry on local Ray: init -> log fan-out -> Timer sink -> finish, then loads and serves the produced directory.

yueming-yuan · 2026-07-15T00:59:56Z

Per-token metrics chart (force-pushed edf549dd→tip, PR-6): the L2 "imp_ratio per token position" chart is now a general metric vs token position panel — a button row selects the metric (imp_ratio / lp_diff / advantages / returns / train / rollout / ref log-probs, whatever the dump carries), the chart spans the whole response (fetched once, independent of the token-strip window), and drag-zoom selects a token range with double-click reset. drawChart gained generic auto-bucketing: above 700 in-domain points, adjacent tokens collapse into mean buckets drawn as a polyline (hover shows pos a–b · mean (n pts)); zooming in re-buckets until raw per-token points with dots reappear. Switching metric keeps the x-range and re-autoscales y. Verified on the live run against an 8337-token sample: full-view bucketed line, drag → tokens 510–1804, chip switch preserves range.

…indow cap

yueming-yuan · 2026-07-15T01:47:33Z

PR-24 + PR-25 end-to-end verified on a real multi-turn run (tip fde52b9f0).

Run recipe (simplest trajectory trigger, zero new code): the qwen30b async recipe with --custom-generate-function-path miles.rollout.generate_hub.multi_turn.generate + retool_v2's code_interpreter sandbox tools, on 8 tool-forcing micro-prompts. Result: 64/64 samples recorded conversations with real tool calls; the sample page renders USER → ASSISTANT (<think> + <tool_call>{"code": "print(2 ** 20)"}) → TOOL·CODE_INTERPRETER (Output: 1048576) → ASSISTANT (Answer: \boxed{1048576}), coexisting with the token strip and per-token metrics.

Field notes from getting there, each worth knowing:

--generate-* args only register under MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1 (the arg-hook is gated; the test harness sets it, real launches must too);
tools reach the chat template only when sample.prompt stays a message list — --apply-chat-template pre-renders to a string and silently drops tool advertisement for the engine multi-turn path (retool_v2 sidesteps this via its SFT model);
base Qwen3-30B on dapo prompts never volunteers tool calls even with tools advertised — instruction-level forcing was needed;
examples/fully_async/fully_async_rollout.py:288,324 crash on list prompts (prompt + response debug prints) — two-line upstream fix worth submitting separately.

Also in this push (PR-18): time_range() no longer leaks the open-interval t1=-1.0 sentinel through partition edge_stamps — the epoch-scale +29734410:00 axis-label bug; line_stamps now reuses each record's own timestamps().

yueming-yuan · 2026-07-15T02:03:56Z

L2 split into tabs (force-pushed, PR-25): the sample page now opens on a conversation | tokens tab bar when a trajectory sidecar exists — conversation (sidecar read, instant) is the default; the token machinery (probe → full dump load → detokenize, the several-minutes first-open cost) fires only when the tokens tab is first clicked (verified: zero /tokens requests before the click). Runs without a sidecar keep today's behavior: no tab bar, tokens load directly.

yueming-yuan · 2026-07-15T02:06:49Z

Raw-assistant display parsing (force-pushed, PR-25): the engine multi_turn path records the model's verbatim emission, so the conversation renderer now does a display-level split when the structured fields are absent — <think>…</think> becomes the italic thinking block, <tool_call>…</tool_call> becomes a name chip with pretty-printed JSON arguments, and trailing special tokens (<|im_end|> etc.) are stripped; unknown markup stays visible verbatim. Session-path rows with structured reasoning_content/tool_calls render as before; the dump itself stays raw (display-only transform). Verified on the live retool run: think block + code_interpreter chip + clean final answer.

gemini-code-assist Bot reviewed Jul 13, 2026

View reviewed changes

yueming-yuan added 4 commits July 13, 2026 17:33

dashboard: metric store — JSONL schemas, write/load/follow

fd19f5b

yueming-yuan force-pushed the miles-dashboard branch from 49cecf5 to f755958 Compare July 14, 2026 03:07

yueming-yuan force-pushed the miles-dashboard branch from f755958 to d34d87b Compare July 14, 2026 21:40

yueming-yuan force-pushed the miles-dashboard branch from d34d87b to d32fd8c Compare July 14, 2026 22:07

yueming-yuan force-pushed the miles-dashboard branch from d32fd8c to 5f11f27 Compare July 14, 2026 22:14

yueming-yuan force-pushed the miles-dashboard branch from 5f11f27 to 874548a Compare July 14, 2026 22:18

yueming-yuan force-pushed the miles-dashboard branch from 874548a to 49aa283 Compare July 14, 2026 22:22

yueming-yuan force-pushed the miles-dashboard branch from 49aa283 to 8839f1b Compare July 14, 2026 22:25

yueming-yuan force-pushed the miles-dashboard branch from 8839f1b to c7e322a Compare July 14, 2026 22:30

yueming-yuan force-pushed the miles-dashboard branch from c7e322a to ae2b9ee Compare July 14, 2026 22:35

yueming-yuan force-pushed the miles-dashboard branch from ae2b9ee to 222a80f Compare July 14, 2026 22:44

yueming-yuan force-pushed the miles-dashboard branch from 222a80f to 7547be7 Compare July 14, 2026 23:11

yueming-yuan force-pushed the miles-dashboard branch from 7547be7 to 53b2f65 Compare July 14, 2026 23:19

yueming-yuan force-pushed the miles-dashboard branch 2 times, most recently from 0c4d838 to edf549d Compare July 15, 2026 00:54

yueming-yuan added 12 commits July 14, 2026 17:57

dashboard: lane selection grammar, binary heatmap, outlier ranking

669ef9b

dashboard: carpet view + lane selection UI

7adf8d6

dashboard: columnar in-memory store for high-rate streams

0db1f49

yueming-yuan force-pushed the miles-dashboard branch from edf549d to d8d7fc9 Compare July 15, 2026 00:58

yueming-yuan added 7 commits July 14, 2026 18:28

dashboard: hourly partitions, lazy windowed reads, lane catalog, 4h w…

0e33f81

…indow cap

dashboard: 4h viewport clamp, windowed lazy fetches, default lane budget

381cdc7

rollout: trajectory lifecycle probes

16562cd

dashboard: trajectory event sink + stream

d8d669e

dashboard: staleness, turns and tool columns from the dump

6d38006

dashboard: trajectory views — batch anatomy + run-wide carpet

4312a25

rollout: dump raw per-sample trajectories next to rollout data

16a9fff

yueming-yuan force-pushed the miles-dashboard branch from 9153bf2 to fde52b9 Compare July 15, 2026 01:30

yueming-yuan force-pushed the miles-dashboard branch from fde52b9 to 44d7f39 Compare July 15, 2026 02:03

dashboard: conversation view from the trajectory dump

d918901

yueming-yuan force-pushed the miles-dashboard branch from 44d7f39 to d918901 Compare July 15, 2026 02:05

	assert whitelist, f"empty --dashboard-sglang-metrics: {args.dashboard_sglang_metrics!r}"
	if not whitelist:
	raise ValueError(f"empty --dashboard-sglang-metrics: {args.dashboard_sglang_metrics!r}")

	assert interval > 0, f"{interval=}"
	if interval <= 0:
	raise ValueError(f"interval must be positive, got {interval}")

-        if mode == ScrapeMode.ROUTER:
-            assert router_addr is not None, "router mode needs router_addr"
-        else:
-            assert engine_addrs is not None, "direct mode needs an engine_addrs provider"
+        if mode == ScrapeMode.ROUTER and router_addr is None:
+            raise ValueError("router mode needs router_addr")
+        if mode != ScrapeMode.ROUTER and engine_addrs is None:
+            raise ValueError("direct mode needs an engine_addrs provider")


	if x_buckets < 2:
	raise ValueError(f"x_buckets must be at least 2, got {x_buckets}")

	assert max_points >= 2, f"{max_points=}"
	if max_points < 2:
	raise ValueError(f"max_points must be at least 2, got {max_points}")

Uh oh!

Conversation

yueming-yuan commented Jul 13, 2026

Design principles

Usage

Validation

Review guide

WIP

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 14, 2026

Uh oh!

yueming-yuan commented Jul 15, 2026

Uh oh!

yueming-yuan commented Jul 15, 2026

Uh oh!

yueming-yuan commented Jul 15, 2026

Uh oh!

yueming-yuan commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone