Add Qwen3 / Qwen3-Coder support to swe-agent-v2 example (ROCm-safe)

examples/experimental/swe-agent-v2-amd/README.md

@@ -0,0 +1,47 @@
+# Agent V2 — Qwen3 / Qwen3-Coder on ROCm/AMD
+
+ROCm/AMD-oriented variant of [`../swe-agent-v2`](../swe-agent-v2). It is **self-contained** —
+it carries its own copies of `run.py`, `generate.py`, and `swe_agent_function.py` so it runs
+without modifying the shared example. See [`../swe-agent-v2/README.md`](../swe-agent-v2/README.md)
+for the full pipeline/architecture; this doc covers only the Qwen3 launcher and the ROCm notes.
+
+## Qwen3 / Qwen3-Coder
+
+`run-qwen3-swe.py` runs the same pipeline with Qwen3 instead of GLM-4.7-Flash. It reuses
+`run.py` and only swaps the model + SGLang/TITO parsers (`qwen25` / `qwen3` / `tito-model qwen3`).
+
+```bash
+# General model
+python run-qwen3-swe.py --prompt-data /root/swe_train.jsonl
+
+# Coding-specialised base (recommended for SWE-bench — see "Reward signal" below)
+python run-qwen3-swe.py --coder --prompt-data /root/swe_train.jsonl
+```
+
+`--coder` uses **Qwen3-Coder-30B-A3B-Instruct**, which is arch-identical to Qwen3-30B-A3B
+(same converter and `scripts/models/qwen3-30B-A3B.sh`) but uses `rope_theta=1e7`. The launcher
+sets `MODEL_ARGS_ROTARY_BASE=10000000` for the Megatron side automatically; SGLang reads the
+value from the HF config.
+
+### Reward signal (why the coder)
+
+GRPO learns only from **within-group reward variance**. A cold general model that solves ~0%
+of SWE-bench Verified produces all-zero groups → zero advantage → no learning. Use a base that
+lands in the "some pass / some fail" band (the coder), an easier task set (e.g. SWE-Gym), or a
+warm-started checkpoint. With `rollout_batch_size=2`, also consider
+`--dynamic-sampling-filter-path miles.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std`
+plus over-sampling so every step contains variance-bearing groups.
+
+### ROCm / AMD prerequisites
+
+- **SGLang build with the `return_meta_info` chat-completion patch.** The TITO session server
+  requires `choice.meta_info.output_token_logprobs` from `/v1/chat/completions`; a build
+  without it returns HTTP 502 on every agent call. (This is a miles-SGLang-version
+  requirement, not AMD-specific.)
+- **GPU visibility:** handled automatically — `run.py` sets
+  `RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES=1` only on ROCm.
+- **agent-env container:** needs the Docker Compose v2 plugin (`docker compose`) — the Debian
+  `docker.io` package does not bundle it — and, for `mini-swe-agent`, a litellm install that
+  includes its proxy deps (recent litellm eagerly imports them): install with
+  `--with 'litellm[proxy]'` or pin litellm, ideally baked into the task base image so it is not
+  reinstalled per trial.

examples/experimental/swe-agent-v2-amd/generate.py

@@ -0,0 +1,111 @@
+"""
+Agent V2: reward, metrics, and rollout class.
+
+The generate function is provided by:
+    miles.rollout.generate_hub.agentic_tool_call.generate
+with --custom-agent-function-path pointing to swe_agent_function.run
+
+Task-type agnostic — reward is pre-computed by the Harbor environment
+and stored in sample.metadata["reward"] regardless of task type.
+
+Dynamic filter uses the general-purpose ``check_no_aborted`` from
+``miles.rollout.filter_hub.dynamic_sampling_filters``.
+
+Components:
+  - reward_func: reads pre-computed reward from sample metadata
+  - aggregate_agent_metrics: aggregates agent timing/count metrics
+  - RolloutFn: InferenceRolloutFn subclass that logs agent metrics
+"""
+
+import logging
+
+from miles.rollout.base_types import RolloutFnTrainInput, RolloutFnTrainOutput
+from miles.rollout.inference_rollout.inference_rollout_common import InferenceRolloutFn
+from miles.utils.types import Sample
+
+logger = logging.getLogger(__name__)
+
+
+# -- Reward --
+
+
+async def reward_func(args, samples: Sample | list[Sample], **kwargs) -> float | list[float]:
+    """Reward is pre-computed by the agent environment during generate().
+
+    Handles both single-sample calls (from ``async_rm``) and batched calls
+    (from ``batched_async_rm`` when ``--custom-rm-path`` is set).
+    """
+    if isinstance(samples, list):
+        return [s.metadata.get("reward", 0.0) for s in samples]
+    return samples.metadata.get("reward", 0.0)
+
+
+# -- Agent Metrics Aggregation --
+
+
+def _collect_values(all_metrics: list[dict], key: str) -> list[float]:
+    return [m.get(key, 0) for m in all_metrics]
+
+
+def _agg_mean(metrics: dict, all_metrics: list[dict], keys: list[str], prefix: str = "agent/", suffix: str = "_mean"):
+    for key in keys:
+        values = _collect_values(all_metrics, key)
+        if values:
+            metrics[f"{prefix}{key}{suffix}"] = sum(values) / len(values)
+
+
+def aggregate_agent_metrics(samples: list[Sample]) -> dict:
+    """Aggregate agent metrics across samples for logging."""
+    all_metrics = [
+        s.metadata.get("agent_metrics", {})
+        for s in samples
+        if hasattr(s, "metadata") and s.metadata and s.metadata.get("agent_metrics")
+    ]
+    if not all_metrics:
+        return {}
+
+    metrics = {}
+
+    for key in ["turns", "tool_calls"]:
+        values = _collect_values(all_metrics, key)
+        if values:
+            metrics[f"agent/{key}_mean"] = sum(values) / len(values)
+            metrics[f"agent/{key}_sum"] = sum(values)
+
+    _agg_mean(metrics, all_metrics, ["model_query_time_sum", "env_execution_time_sum", "eval_time", "agent_run_time"])
+    _agg_mean(metrics, all_metrics, ["time_per_turn", "model_query_time_avg", "env_execution_time_avg"], suffix="")
+    _agg_mean(metrics, all_metrics, ["model_time_ratio", "env_time_ratio", "eval_time_ratio"], suffix="")
+
+    values = _collect_values(all_metrics, "total_time")
+    if values:
+        metrics["agent/total_time_mean"] = sum(values) / len(values)
+        metrics["agent/total_time_max"] = max(values)
+        metrics["agent/total_time_min"] = min(values)
+
+    return metrics
+
+
+# -- Rollout Function --
+
+
+class RolloutFn(InferenceRolloutFn):
+    """Rollout function with agent metrics aggregation."""
+
+    async def _call_train(self, input: RolloutFnTrainInput) -> RolloutFnTrainOutput:
+        output = await super()._call_train(input)
+
+        all_samples = []
+        for group in output.samples:
+            if isinstance(group, list):
+                all_samples.extend(group)
+            else:
+                all_samples.append(group)
+
+        agent_metrics = aggregate_agent_metrics(all_samples)
+        if agent_metrics:
+            metrics = output.metrics or {}
+            metrics.update(agent_metrics)
+            output.metrics = metrics
+            logger.info(f"Agent metrics for rollout {input.rollout_id}: {agent_metrics}")
+
+        return output

examples/experimental/swe-agent-v2-amd/run-qwen3-swe.py

@@ -0,0 +1,74 @@
+"""Agent V2 launcher (Qwen3 / Qwen3-Coder): Miles <-> Harbor agent orchestration.
+
+Thin wrapper over ``run.py`` — reuses its ``cleanup()`` / ``prepare()`` /
+``execute()`` and only overrides the model identity and the SGLang/TITO
+parsers, which are the sole differences from the GLM-4.7-Flash default.
+
+Usage:
+    # Qwen3-30B-A3B (general model)
+    python run-qwen3-swe.py --prompt-data /root/swe_train.jsonl
+
+    # Qwen3-Coder-30B-A3B-Instruct (coding-specialised; recommended base for
+    # SWE-bench so rollouts produce non-zero reward variance)
+    python run-qwen3-swe.py --coder --prompt-data /root/swe_train.jsonl
+
+    # quick pipeline check (rollout only, no weight updates)
+    python run-qwen3-swe.py --coder --mode debug_rollout_only
+"""
+
+import os
+from dataclasses import dataclass
+
+import typer
+
+# run.py lives in the same directory (added to PYTHONPATH by execute()).
+from run import ScriptArgs as BaseScriptArgs
+from run import cleanup, execute, prepare
+
+import miles.utils.external_utils.command_utils as U
+
+
+@dataclass
+class ScriptArgs(BaseScriptArgs):
+    # Model identity (Qwen3-30B-A3B). ``--coder`` swaps these to the coder below.
+    megatron_model_type: str = "qwen3-30B-A3B"
+    model_name: str = "Qwen3-30B-A3B"
+    hf_checkpoint: str = "Qwen/Qwen3-30B-A3B"
+    ref_load: str = "/root/Qwen3-30B-A3B_torch_dist"
+    save_dir: str = "/root/Qwen3-30B-A3B_agent_v2/"
+
+    # SGLang / TITO parsers for Qwen3 (GLM defaults are glm47 / glm45 / glm47).
+    sglang_tool_call_parser: str = "qwen25"
+    sglang_reasoning_parser: str = "qwen3"
+    tito_model: str = "qwen3"
+
+    wandb_project: str = os.environ.get("WANDB_PROJECT", "qwen3-swe-agentic")
+    wandb_run_name: str = "qwen3-swe-tito"
+    prometheus_run_name: str = "qwen3-swe-tito"
+
+    # Use Qwen3-Coder-30B-A3B-Instruct instead of the general model. The coder
+    # is arch-identical (same converter/model script) but uses rope_theta=1e7
+    # (vs 1e6), so we override MODEL_ARGS_ROTARY_BASE for the Megatron side;
+    # SGLang picks up the correct value from the HF config automatically.
+    coder: bool = False
+
+
+@U.dataclass_cli
+def main(args: ScriptArgs):
+    if args.coder:
+        args.model_name = "Qwen3-Coder-30B-A3B-Instruct"
+        args.hf_checkpoint = "Qwen/Qwen3-Coder-30B-A3B-Instruct"
+        args.ref_load = "/root/Qwen3-Coder-30B-A3B-Instruct_torch_dist"
+        args.save_dir = "/root/Qwen3-Coder-30B-A3B-Instruct_agent_v2/"
+        # Read by scripts/models/qwen3-30B-A3B.sh (${MODEL_ARGS_ROTARY_BASE:-1000000})
+        # when the model script is sourced for both conversion and training.
+        os.environ["MODEL_ARGS_ROTARY_BASE"] = "10000000"
+
+    cleanup()
+    if not args.skip_prepare:
+        prepare(args)
+    execute(args)
+
+
+if __name__ == "__main__":
+    typer.run(main)