eval: checkpoint-interfaced evaluation for fully-async training (dedicated fleet / pause-the-world / external service)

[Zhichenzzz]

Motivation

Fully-async rollout (#1717) has no evaluation story: the producer never stops, so there is no quiet window to evaluate in, and running eval on the training engines is structurally broken — eval requests get aborted by every weight update, and no well-defined weight version is measured. Heavy eval sets also steal training inference capacity even in sync mode.

Design

The only interface between training and eval is a (rollout_id, HF snapshot dir) pair. Eval engines never join training weight updates; weights reach them only through update_weights_from_disk(hf_dir, weight_version=str(rollout_id)), which loads the snapshot and stamps the version atomically.

Pick a posture by two questions — is this a test run or a real run (are checkpoints persisted anyway), and does eval get standalone GPUs:

	Test run	Real run (ckpts saved anyway)
No standalone eval	Pause-the-world: eval shares the rollout engines; the producer pauses new submissions for the duration of the eval	External service watches --save-hf output outside the job; pause-the-world if points must be strictly on-time
Standalone eval fleet	Fleet + tmpfs snapshots: --eval-num-gpus N --eval-hf-dir /dev/shm/... — export is seconds, never disk-bound	Fleet + checkpoint reuse: --eval-num-gpus N with no --eval-hf-dir — the snapshot IS the periodic --save-hf checkpoint, zero extra export

All four cells are flag composition over the same controller (pin / verify / skip / GC).

Semantics

• Fire-and-forget: the trainer exports a snapshot and keeps training; the point lands at the dispatch-time step even when it completes later (eval/lag_steps reports how late).
• Version pinning: the engine-stamped weight_version is verified after every load; eval/{ds}/weight_version/mean == eval/step and mixed_version_ratio == 0 prove each point measured exactly the intended weights.
• Where the weights come from: pause-the-world needs no weight movement — training's own per-step update_weights broadcast already put the step's weights on the shared engines, and pinning is by ordering (the driver awaits the eval, so the next update cannot interleave; this is also why shared-engine eval must stay blocking). The fleet and the external service sit outside the broadcast group, so they load the actor's exact step-R weights from the snapshot/checkpoint — which is what makes them safely asynchronous. Eval engines never join the training broadcast: collectives cannot skip members, so an in-group fleet would be rewritten by every update.
• Pause is a gate, not a retract: shared-engine eval only stops new producer submissions; in-flight rollout requests finish and buffer (no aborts, no wasted tokens). The generation pause around each weight broadcast (--pause-generation-mode) is training's per-step business — eval adds no pause, retract, or weight sync of its own, and works under both in_place and retract pause modes (retract additionally requires the flush-after-retract fix from [sglang-miles] Fix flush_cache() no-op after pause_generation in retract sgl-project/sglang#31962 + [Fix] retract-mode flush_cache no-op crash #1750).
• Training is never stalled: every failure mode — unhealthy fleet, incomplete snapshot, pin violation, crashed eval, overflow — degrades to a skipped point logged at that step (eval/skipped_{unhealthy,ckpt_missing,pin_violation,crashed,busy}). Overflow behavior is explicit (--eval-overflow-policy backpressure|skip), and the final eval point is never skipped (training is already over, so waiting is free).
• Self-healing without FT: before each eval the controller probes every fleet engine, marks unreachable ones stopped, recovers them, and re-probes the router end-to-end (one-token generate) before dispatching. A zombie engine costs one skipped point, bounded by a per-attempt weight-load timeout — never a stalled trainer.

What's in the PR

• miles/rollout/checkpoint_eval.py — retargeted-args helpers + EvalFleetSession (lazy eval-fleet state shared by both rollout fns)
• FullyAsyncRolloutFn eval support — dedicated fleet when configured, pause-the-world otherwise
• miles/ray/rollout/eval_dispatch.py — driver-side EvalDispatcher (bounded in-flight, overflow policy, drain-before-dispose)
• RolloutManager._eval_on_dedicated_fleet — the pin/verify/skip/GC controller
• export_hf_model_direct — HF export through miles' direct megatron→HF converters, the same machinery as weight updates, so export coverage always matches weight-sync coverage (the AutoBridge silently exports zero weights for specs it has no mapping for, e.g. qwen3.5); every HF export now writes a .complete marker
• tools/checkpoint_eval_service.py — standalone watcher with its own sglang server: marker/quiescence-gated snapshot discovery, idempotent ledger for restart/backfill, --catchup {all,latest}, --once
• New args + validation matrix, docs (docs/user-guide/fully-async.md), example script, e2e CI test, ~25 unit tests

Verification (8×H200, Qwen3.5-4B, 4 actor TP2 + rollout engines ± 1 eval engine)

All four matrix cells ran end-to-end (dapo-math training, gsm8k eval):

• Fleet + tmpfs: all 5 eval points landed pinned, gsm8k 0.640 → 0.882 over 20 steps, lag_steps ≤ 11, training uninterrupted
• Fleet + reuse: zero-extra-export points 0.641 → 0.847; overflow under skip policy observed and attributed (eval/skipped_busy at the affected step)
• Pause-the-world: producer pause/resume verified, points pinned with mixed_version_ratio == 0
• External service: backfilled every checkpoint after the run; on the same checkpoint the service score matched the in-job score exactly

Kill tests: an eval engine killed mid-run is detected by the pre-eval probe, marked stopped, revived by recover(), the router probe rides out the post-revival 503 window, and the next eval lands pinned — training untouched throughout.

Production-scale stress (512 in-flight requests, global-batch 128 → 4 optimizer steps + weight broadcasts per rollout, real --save/--save-hf persistence, pause-the-world gsm8k eval): completed under both pause modes with statistically identical curves (in_place 0.651→0.936, retract 0.639→0.935), mixed_version_ratio == 0 throughout, and equivalent step time (~0.8 s per broadcast). At this concurrency the router must be sized for the eval burst (--router-queue-size/--router-queue-timeout-secs, circuit breaker off) — with default limits the burst 503s past the client retry budget.

tests/fast green; e2e tests/e2e/short/test_qwen2.5_0.5B_fully_async_eval.py green on 8×H200.

Stacked on #1717.

[gemini-code-assist]

Code Review

This pull request introduces a dedicated evaluation fleet for fully-async training, allowing evaluations to run on separate engines pinned to Hugging Face checkpoint snapshots without stalling training. It adds new CLI configuration options, an EvalDispatcher to manage async evaluation tasks, a standalone checkpoint evaluation service tool, and corresponding tests. The review feedback focuses on improving robustness and error handling, specifically suggesting to verify if the checkpoint path is a local directory before exporting, replacing assert statements with explicit exceptions (like ValueError and RuntimeError) to prevent them from being stripped under Python optimization, and narrowing exception handling in health check probes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If args.hf_checkpoint is a Hugging Face Hub model ID (e.g., Qwen/Qwen2.5-7B-Instruct) rather than a local directory, Path(args.hf_checkpoint).iterdir() will raise a FileNotFoundError and crash the direct HF export process. Check if the path is a directory before attempting to copy metadata files.

Suggested change

	if is_writer:
	assert weight_map, f"HF export to {path} produced no weights"
	for meta_file in Path(args.hf_checkpoint).iterdir():
	# Copy tokenizer/config metadata only — never base weight files or the
	# base checkpoint's safetensors index (ours is written below).
	if meta_file.is_file() and not any(s in meta_file.name for s in HF_METADATA_SKIP_SUFFIXES):
	shutil.copy2(meta_file, path / meta_file.name)
	if is_writer:
	assert weight_map, f"HF export to {path} produced no weights"
	checkpoint_path = Path(args.hf_checkpoint)
	if not checkpoint_path.is_dir():
	raise FileNotFoundError(
	f"hf_checkpoint '{args.hf_checkpoint}' is not a local directory. "
	f"Direct HF export requires a local directory containing the model metadata files."
	)
	for meta_file in checkpoint_path.iterdir():
	# Copy tokenizer/config metadata only — never base weight files or the
	# base checkpoint's safetensors index (ours is written below).
	if meta_file.is_file() and not any(s in meta_file.name for s in HF_METADATA_SKIP_SUFFIXES):
	shutil.copy2(meta_file, path / meta_file.name)

[gemini-code-assist]

Use ValueError instead of assert for validating command-line arguments. If Python is run with optimization (-O), assert statements are compiled away, which would silently bypass all validation checks.

    if args.eval_num_gpus > 0:
        if not enable_experimental_rollout_refactor():
            raise ValueError("--eval-num-gpus requires the class-based rollout API (MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1).")
        if args.eval_interval is None:
            raise ValueError("--eval-num-gpus requires --eval-interval.")
        if args.eval_hf_dir is None and args.save_hf is None:
            raise ValueError(
                "--eval-num-gpus requires a snapshot source: set --eval-hf-dir (staging exports) "
                "or --save-hf (reuse periodic HF checkpoints)."
            )
        if args.colocate:
            raise ValueError(
                "--eval-num-gpus is not supported with --colocate; "
                "use tools/checkpoint_eval_service.py against --save-hf checkpoints instead."
            )
        if args.debug_train_only or args.debug_rollout_only:
            raise ValueError("--eval-num-gpus is not supported with debug_train_only/debug_rollout_only.")
        if args.eval_num_gpus % args.eval_num_gpus_per_engine != 0:
            raise ValueError(
                f"eval_num_gpus ({args.eval_num_gpus}) must be divisible by "
                f"eval_num_gpus_per_engine ({args.eval_num_gpus_per_engine})."
            )
        if args.eval_keep_snapshots < args.eval_max_in_flight:
            raise ValueError(
                f"--eval-keep-snapshots ({args.eval_keep_snapshots}) must be >= --eval-max-in-flight "
                f"({args.eval_max_in_flight}), otherwise a pending eval's snapshot could be GC'd."
            )
        if args.eval_hf_dir is None:
            # Reuse mode: every eval-due step must coincide with a save-due step.
            if args.save_interval is None or args.eval_interval % args.save_interval != 0:
                raise ValueError(
                    "Reusing --save-hf checkpoints for eval requires eval_interval to be a "
                    f"multiple of save_interval (got eval_interval={args.eval_interval}, "
                    f"save_interval={args.save_interval}). Set --eval-hf-dir for independent snapshots."
                )

References

1. Use ValueError instead of assert for validating function or constructor arguments.

[gemini-code-assist]

Avoid catching broad exceptions like Exception in health check probes. Only catch expected network, timeout, or unavailability exceptions (such as httpx.HTTPError and asyncio.TimeoutError) to prevent swallowing internal programming errors or bugs.

Suggested change

	try:
	await get(f"http://{ip}:{port}/health_generate", max_retries=1)
	return
	except Exception:
	await asyncio.sleep(5)
	import httpx
	try:
	await get(f"http://{ip}:{port}/health_generate", max_retries=1)
	return
	except (httpx.HTTPError, asyncio.TimeoutError):
	await asyncio.sleep(5)

References

1. In health check endpoints or probes, avoid catching broad exceptions (like Exception). Only catch expected network, timeout, or unavailability exceptions.

[gemini-code-assist]

Do not use assert for runtime checks that must always execute, as they are stripped when Python is run with optimization (-O). Use RuntimeError instead.

Suggested change

	assert (
	str(info.get("weight_version")) == weight_version
	), f"weight_version pin failed: engine reports {info.get('weight_version')}, expected {weight_version}"
	if str(info.get("weight_version")) != weight_version:
	raise RuntimeError(
	f"weight_version pin failed: engine reports {info.get('weight_version')}, expected {weight_version}"
	)

[gemini-code-assist]

Use ValueError instead of assert for validating command-line arguments or configurations to prevent checks from being stripped under Python optimization (-O).

Suggested change

	if args.wandb_mode == "shared":
	assert args.wandb_run_id, "--wandb-mode shared requires --wandb-run-id of the training run"
	if args.wandb_mode == "shared" and not args.wandb_run_id:
	raise ValueError("--wandb-mode shared requires --wandb-run-id of the training run")

References

1. Use ValueError instead of assert for validating function or constructor arguments.

[gemini-code-assist]

Use FileNotFoundError instead of assert for validating file or directory existence to prevent checks from being stripped under Python optimization (-O).

Suggested change

	watch_dir = Path(service_args.watch_dir)
	assert watch_dir.is_dir(), f"--watch-dir {watch_dir} does not exist"
	watch_dir = Path(service_args.watch_dir)
	if not watch_dir.is_dir():
	raise FileNotFoundError(f"--watch-dir {watch_dir} does not exist")

References

1. Use ValueError instead of assert for validating function or constructor arguments.