diff --git a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md index 386542b05..082d243c3 100644 --- a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md +++ b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md @@ -174,11 +174,14 @@ Three things to know before running this on Blackwell: `setup/env_vars` derives `IFACE` from the default route, so the interface name is not something you need to know per instance family. -**Serving on Blackwell is not validated here.** The DeepEP-EFA kernels are — the same -`567632d` + EFA patch, same NVSHMEM 3.7.0, is measured out to 256 ranks on `p6-b300` in -[`ep-backend-comparison`](../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md). -The serving-side gap matters, because **on B300 the published SGLang comparison goes the other -way**: see [Blackwell: expect DeepEP to lose at 2 nodes](./benchmarks/README.md#blackwell-expect-deepep-to-lose-at-2-nodes). +**Serving on Blackwell is not validated here.** The older B300 backend comparison previously cited +here has been retired because its backend-native timing and byte accounting did not support a +cross-backend ranking. The replacement +[`ep-backend-comparison`](../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) +defines common boundaries for synthetic Decode-like and Prefill-like communication workloads at +EP16 and EP32 on B200. Its replacement result matrix is pending and does not establish B300 serving +behavior. See +[Blackwell serving needs a matched measurement](./benchmarks/README.md#blackwell-serving-needs-a-matched-measurement). ## Smoke-test the EFA transport before loading the model diff --git a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md index da3b5b90d..31dcec060 100644 --- a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md +++ b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md @@ -813,60 +813,35 @@ it was added for was real on some host during bring-up, just not one that has be since. Treat it as a knob to try if NVSHMEM init fails on a `normal` role, not as a requirement — and do not conclude from a passing `normal` run that your host disagrees with this table. -# Blackwell: expect DeepEP to lose at 2 nodes - -The serving tables in this document are H200. The question comes up — *"we benchmarked DeepEP vs the -NCCL all-to-all on 2× B300 and DeepEP was slower in every configuration; is that expected?"* — and -the answer from the data that **is** here is **yes at 2 nodes, and it is not an EFA problem.** -Reported shape of such a result: output throughput −7% to −26%, median TTFT +17% to +82%, P99 ITL -1.2–1.9 s vs 0.8–0.9 s, `normal` (HT) slowest where it ran, at TP16/EP16 across two nodes, 8K input -/ 1K output, concurrency 128. - -Read this section as a **mechanism argument extrapolated from H200 serving data plus B300 kernel -data** — not as a Blackwell serving measurement. No B300 serving sweep exists here yet. - -1. **16 ranks is DeepEP's worst case on the hardware measured here, and the mechanism is not - GPU-specific.** Every table in this document says so: the colocated decode sweep has DeepEP at - 0.23–0.75× the baseline's throughput and 1.4–4.3× its TPOT; the 2P2D sweep 0.55–0.71× at 1.4–1.8× - TPOT. The reason — 256 experts over 16 ranks means 16 experts per GPU, so the fan-out is small and - mostly intra-node NVLink, and DeepEP's per-layer dispatch/combine cost is not amortised — is a - function of EP width and expert count, not of the GPU generation. That makes a −7% to −26% - *aggregate* regression on a mixed 8K/1K workload **milder than what we measure on Hopper**, though - confirming the H200 magnitude carries to B300 requires the B300 sweep. - **One Blackwell configuration note either way:** on B200 every rank logs `Only use 20 SMs for DeepEP - communication ... Consider using --deepep-config`, so any Blackwell number taken with the default - config is a **floor**, not DeepEP's best (thanks @KeitaW for the observation). -2. **The published B300 kernel numbers are healthy, which localises the gap above the transport.** - At 2 nodes / 16 ranks on `p6-b300`, DeepEP-over-EFA dispatch/combine is **126.6 / 106.4 GB/s** — - best of the three backends there, *above* the NCCL all-to-all's 104.9 GB/s at matched payload - ([`ep-backend-comparison`](../../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md)). - So fabric and kernels are fine at that scale; the serving regression is per-layer - launch/scheduling overhead and MoE-runner choice, not bytes on the wire. -3. **`normal`/HT being slowest at decode-heavy concurrency is by design.** On a 1K-output workload - the run is TPOT-dominated, where `low_latency` is the intended mode. - -Before concluding anything from such a run, eliminate these: - -- **The four settings in [How to measure this correctly](#how-to-measure-this-correctly)** — the - harness defaults and the mode pinning in particular. `--random-range-ratio` alone moved a nominal - 256/512/conc-64 point from 374 to 1127 tok/s. -- **The MoE runner is not held constant** in the usual formulation: DeepEP rows run - `--moe-runner-backend deep_gemm` while the no-DeepEP rows resolve `auto` to `flashinfer_trtllm` on - Blackwell. That is two variables, and TRT-LLM's Blackwell MoE kernels are heavily tuned. Re-run - DeepEP against `flashinfer_trtllm` (or the baseline against `deep_gemm`) before attributing the - delta to the all-to-all. -- **DeepGEMM JIT warmup.** It inflates early TTFT and P99 ITL specifically — the two metrics that - move most in reports like this. Pre-warm on **both** nodes (`recipe/serve-pd.sh precompile`). -- **An HT-path hang that does not reproduce.** At ≥128 ranks the NVSHMEM-libfabric host proxy - exhausts libfabric retries (`EAGAIN` in `nvshmemi_process_multisend_rma`) and kills a different - pair of ranks each run — a documented statistical fan-out limit, not a bad node. At 16 ranks it - should not fire, but a non-reproducing hang on the HT path has the same signature. - -**The load-bearing point for a large fleet: 2 nodes measures the wrong thing.** DeepEP is built for -EP domains where experts are spread thin enough that every token crosses the fabric. Kernel scaling -to 256 ranks on `p6-b300` is already characterised: the useful envelope is ~64–160 ranks, with hard -implementation caps past that (HT: 160 PEs at `deep_ep.cpp:158`; low-latency: between 64 and 128 -PEs). **A production-EP-width run — EP32 or EP64, not EP16 — is the measurement that decides this.** +# Blackwell serving needs a matched measurement + +The serving tables in this document are H200. No matched B200 or B300 serving sweep is reported +here, so the H200 direction must not be assumed to carry across GPU generations. + +The current +[`ep-backend-comparison`](../../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) +defines common CUDA timing boundaries and common logical payload accounting for synthetic +Decode-like and Prefill-like dispatch-plus-combine workloads. The replacement result matrix is +pending. It excludes expert compute, scheduling, communication/computation overlap, and +request-level serving behavior, and it does not provide a B300 serving result. + +Before attributing a Blackwell serving delta to the all-to-all backend, control these variables: + +- Apply the 4 settings in [How to measure this correctly](#how-to-measure-this-correctly), including + the harness defaults and explicit mode pinning. `--random-range-ratio` alone moved a nominal + 256-token/512-token/concurrency-64 point from 374 tokens/s to 1,127 tokens/s in the H200 campaign. +- Hold the MoE runner constant. DeepEP rows commonly select `deep_gemm`, while no-DeepEP rows can + resolve `auto` to `flashinfer_trtllm` on Blackwell. Changing both the MoE kernel and communication + backend does not isolate either effect. +- Pre-warm DeepGEMM JIT on every node with `recipe/serve-pd.sh precompile` before collecting TTFT or + P99 ITL. +- Match model, EP and TP widths, input and output lengths, concurrency, request seeds, image digests, + runtime versions, and named nodes. Rotate backend order across independent process starts. +- Compare request-level throughput and latency directly. Treat each backend's native GB/s fields as + diagnostics unless they share the same timing boundary and byte numerator. +- Test the production EP width. An EP16 result cannot decide an EP32 or EP64 deployment. + +The Blackwell serving conclusion remains unmeasured until that matched sweep is run. # Reproduce diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index f7e50bde4..a5e0a27e0 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -1,175 +1,200 @@ -# Expert-Parallelism Backend Comparison (NCCL vs UCCL vs NVSHMEM) on EKS +# Expert-Parallelism Backend Comparison on EFA -Head-to-head MoE dispatch/combine micro-benchmark across three communication backends, run -at the **same EP world size** on the same GPU nodes (designed for 8× `p6-b300.48xlarge`, -64 ranks; also exercised at **32 nodes / 256 ranks** — see -[Scaling beyond 8 nodes](#scaling-beyond-8-nodes-256-rank-findings) for the hard backend -limits that appear there). This directory is the orchestration layer; the benchmarks -themselves live in the sibling directories. +This directory compares 3 expert-parallel dispatch/combine backends through common Decode-like and Prefill-like communication workloads: -| Config | What it is | Source benchmark | -|---|---|---| -| **NCCL** (baseline) | Raw all-to-all over EFA. The transport-level **reference ceiling** — moves bytes, but does *not* do token routing or combine-reduction. | [`nccl-alltoall.yaml`](nccl-alltoall.yaml) (built from [`../../nccl-tests`](../../nccl-tests)) | -| **UCCL** | DeepEP-style dispatch/combine over the UCCL all-to-all backend. | [`../uccl-ep-benchmark/kubernetes`](../uccl-ep-benchmark/kubernetes) | -| **NVSHMEM** | DeepEP dispatch/combine over NVSHMEM (libfabric/EFA). | [`../deepep-benchmark/kubernetes`](../deepep-benchmark/kubernetes) | +| Backend | Implementation and transport | +|---|---| +| UCCL | DeepEP-compatible dispatch/combine over UCCL all-to-all and EFA | +| DeepEP V1 NVSHMEM | DeepEP V1 `Buffer` over NVSHMEM, libfabric, and EFA | +| DeepEP V2 NCCL GIN | DeepEP V2 `ElasticBuffer` over NCCL GIN EFA-GDA | + +Each workload uses one external CUDA timing boundary and one logical payload definition. Backend-native latency and bandwidth fields remain diagnostics because they do not share one timing boundary or byte numerator. + +The B200 report and backend box plots are in [RESULTS.md](RESULTS.md). EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes. + +## Workload profiles + +| Profile | Tokens | UCCL and DeepEP V1 API | DeepEP V2 API | Primary metric | +|---|---:|---|---|---| +| Decode-like | 128 tokens/rank | `low_latency_dispatch` and `low_latency_combine` | `ElasticBuffer.dispatch` and `ElasticBuffer.combine` | Slowest-rank latency, in ms | +| Prefill-like | 4,096 tokens/rank | Normal `Buffer.dispatch` and `Buffer.combine` | `ElasticBuffer.dispatch` and `ElasticBuffer.combine` | Effective logical throughput, in GB/s/rank | + +The Prefill-like timing boundary starts with a BF16 input and exact route ready. It includes the dispatch layout required by the normal UCCL and DeepEP V1 APIs, dispatch, and combine completion. The Decode-like boundary starts with the BF16 input ready and includes dispatch and combine completion. Required FP8 conversion is inside both boundaries. -> **Why no "DeepEP without a backend"?** DeepEP at the pinned commit (`567632d`, pre-EPv2) has -> no internode dispatch/combine path without a transport backend, so a literal "no-backend -> DeepEP on 8 nodes" does not exist. The NCCL all-to-all stands in as the neutral baseline and -> is labelled as a transport ceiling, not as an equal dispatch/combine number. +Both profiles hold the following controls constant across backends: -## Matched configuration (what makes the numbers comparable) +| Control | Rule | +|---|---| +| Input | One deterministic BF16 tensor per profile and EP size | +| Routing | One exact top-k route and one set of weights, verified by SHA-256 across all arms and starts | +| Model shape | Hidden size 7,168, 256 experts, top-k 8 experts/token | +| Operations | FP8 or BF16 dispatch followed by BF16 combine | +| Rank reduction | Maximum elapsed time across all ranks for each measured iteration | +| Warmup | 20 warmup iterations per dtype and process start | +| Measurement | 100 measured iterations per dtype and process start | +| Replication | 3 independent process starts per arm and workload cell | +| Order | Backend, dtype, and workload-profile order rotate across starts | +| Hardware | The same named nodes serve every arm at a given EP size | +| Runtime | Every result reports the same GPU, PyTorch, CUDA, and NCCL versions | +| Correctness | Every rank passes the common identity-expert result before timing | + +Each process start contributes its median of 100 slowest-rank iteration measurements. The report then takes the median across 3 process starts. Iterations within one process are not treated as independent replicates. + +## Common logical throughput + +Each valid expert assignment contributes the dispatch tensor, FP8 scales when FP8 dispatch is selected, and the BF16 combine tensor. Backend metadata is excluded. Scale-out logical bytes include only assignments whose destination expert is on another node. + +```text +logical GB/s/rank = average logical bytes/rank / median slowest-rank latency +scale-out logical GB/s/rank = average remote logical bytes/rank / median slowest-rank latency +``` + +These are logical efficiency metrics, not observed wire bandwidth. -All runs use the **same EP problem size** — otherwise the table is meaningless: +## Files -| Parameter | Value | +| File | Purpose | |---|---| -| World size | 8 nodes × 8 GPU = **64 ranks** | -| `num-tokens` | 4096 (internode) / 128 (low-latency) | -| `hidden` | 7168 | -| `num-topk` | 8 | -| `num-experts` | 256 (divides evenly across 64 ranks) | -| dtype | bf16 | +| [`ep_benchmark.py`](ep_benchmark.py) | Workload profiles, backend adapters, correctness checks, CUDA timing, and logical-byte accounting | +| [`run_ep_rank.sh`](run_ep_rank.sh) | Per-node `torchrun` entry point and backend-specific transport environment | +| [`run_ep_comparison.sh`](run_ep_comparison.sh) | EKS admission, shared-Lease coordination, rotated matrix, durable harvest, and verified teardown | +| [`result_io.py`](result_io.py) | Robust result-marker parsing from interleaved native output | +| [`extract_results.py`](extract_results.py) | Canonical JSONL extraction from a rank-zero log | +| [`summarize_results.py`](summarize_results.py) | Matrix validation, per-start aggregation, paired deltas, and bootstrap intervals | +| [`plot_results.py`](plot_results.py) | Box plots of the independent-start primary values for every backend arm | +| [`RESULTS.md`](RESULTS.md) | Human-readable result, provenance, and scope limits | + +## Requirements + +The scored B200 matrix requires: + +- 4 named, Ready `p6-b200.48xlarge` nodes in one EKS cluster; +- 8 allocatable GPUs and 8 allocatable EFA devices on every selected node; +- no active GPU requests on the selected nodes before each arm; +- the NVIDIA and EFA Kubernetes device plugins; +- `uvm_disable_hmm=Y` or `uvm_disable_hmm=1` on every selected host; +- `/dev/gdrdrv` as a character device on every selected host; +- `aws`, `kubectl`, `jq`, `rg`, Python 3, and Bash on the launch host; and +- access to the 3 digest-pinned backend images. -The UCCL manifests bake these args into the `python3 bench/test_*.py` invocation; the DeepEP -test hard-codes its config in-image. **Before running, confirm the DeepEP image's config is the -anchor** and align UCCL to it: +DeepEP V2 receives an INFO-level EP16 Decode-like admission run before the scored matrix. All 3 backends then receive an EP16 Prefill-like admission run. A missing HMM mitigation, GDRCopy device, GDAKI proof, or profile correctness result stops the campaign before scoring. + +## Run on an exclusive node set + +Use a unique namespace and durable artifact directory. `KUBECTL_CONTEXT` is required explicitly so a concurrent process changing the default context cannot redirect the campaign. ```bash -# Read the DeepEP test config from the NVSHMEM image and match UCCL's CLI args to it. -docker run --rm ${NVSHMEM_IMAGE_URI} sed -n '1,60p' /DeepEP/tests/test_internode.py +campaign_id=ep-b200-$(date -u +%Y%m%d%H%M%S) +CAMPAIGN_ID="${campaign_id}" \ +EP_BENCHMARK_NODES=node-a,node-b,node-c,node-d \ +PROTECTED_NODES_CSV="" \ +ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ +KUBECTL_CONTEXT=aps1-shared \ +LOCK_MODE=exclusive \ +./run_ep_comparison.sh ``` -If the DeepEP values differ from 4096/7168/8/256, edit the bench args in -`../uccl-ep-benchmark/kubernetes/test-*.yaml` to match. - -## Prerequisites +`LOCK_MODE=exclusive` claims the configured shared Lease only when its holder is empty. The runner releases only a Lease that it still owns. -- EKS cluster with EFA + GPU nodes; NVIDIA device plugin + AWS EFA device plugin; Kubeflow MPI - Operator (`kubectl get crd mpijobs.kubeflow.org`). See each benchmark's `kubernetes/README.md`. -- The container images in ECR: - - NVSHMEM: `../deepep-benchmark/deepep.Dockerfile` (CUDA 13, `sm_90`+`sm_100`) - - UCCL: `../uccl-ep-benchmark/uccl-ep.Dockerfile` (CUDA 13; pinned UCCL commit; Hopper + Blackwell via PTX) - - NCCL: **reuse the NVSHMEM/DeepEP image** — it already builds `/opt/nccl-tests/build/alltoall_perf` - with `sm_100` gencode, so no separate `nccl-tests` build is needed for the baseline. +## Run beside a coordinated campaign -## Account / cluster safety (run first) +Observe mode is allowed only when the other campaign has a known Lease holder and a disjoint named node set: ```bash -aws sts get-caller-identity # confirm the target account -kubectl config current-context # confirm the target cluster -kubectl get nodes -l node.kubernetes.io/instance-type=p6-b300.48xlarge # confirm $NUM_NODES schedulable -kubectl get crd mpijobs.kubeflow.org # confirm MPI Operator +campaign_id=ep-b200-$(date -u +%Y%m%d%H%M%S) +CAMPAIGN_ID="${campaign_id}" \ +EP_BENCHMARK_NODES=node-a,node-b,node-c,node-d \ +PROTECTED_NODES_CSV=foreign-node-a,foreign-node-b \ +ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ +KUBECTL_CONTEXT=aps1-shared \ +LOCK_MODE=observe \ +EXPECTED_LOCK_HOLDER=foreign-campaign-id \ +./run_ep_comparison.sh ``` -## Run order (serial — each config needs all 8 nodes) +Observe mode never mutates the shared Lease. It verifies the exact holder before every arm and again before aggregation. Any selected/protected node overlap or Lease-holder change stops the run. + +## Execution matrix + +The scored order uses 3 independent starts: + +| Start index | Backend order | Profile order | Dtype order | +|---:|---|---|---| +| 1 | UCCL, DeepEP V1, DeepEP V2 | Decode-like, Prefill-like | FP8, BF16 | +| 2 | DeepEP V2, UCCL, DeepEP V1 | Prefill-like, Decode-like | BF16, FP8 | +| 3 | DeepEP V1, DeepEP V2, UCCL | Decode-like, Prefill-like | FP8, BF16 | + +The runner executes the rotation first at 16 ranks on 2 nodes and then at 32 ranks on 4 nodes. Arms run serially, and every StatefulSet and its GPU pods must be gone before the next arm is admitted. The full matrix contains 36 distributed process starts and 72 scored dtype results. + +## Durable artifacts and teardown + +The campaign writes the following layout under `ARTIFACT_ROOT`: + +```text +control/ + aws-caller-identity.json + fleet-nodes-before.json + fleet-nodes-after.json + fleet-pods-before.json + fleet-pods-after.json + provenance.json + selected-nodes.txt +runs/ + decode/ep16/{admission,measurement}-repeat-*// + decode/ep32/measurement-repeat-*// + prefill/ep16/{admission,measurement}-repeat-*// + prefill/ep32/measurement-repeat-*// +summary/ + summary.json + summary.md +teardown/ + namespace-delete.log + remaining-resources.json + shared-lease-after.json +CAMPAIGN_COMPLETE +SHA256SUMS +STATUS +``` + +Every rank log, rendered Pod manifest, Pod description, canonical rank-zero JSONL result, case status, input/route hash, and immutable image reference is retained. `CAMPAIGN_COMPLETE` is written only after the full scored matrix succeeds and teardown verifies that the owned namespace and labeled resources are absent. `SHA256SUMS` is generated after the final status markers. + +## Re-aggregate preserved logs -**Smoke first.** Before any 8-node job, run the single-node `test-intranode.yaml` for each EP -image. It validates the image, that `sm_100` actually runs on B300, and the launch path in -minutes instead of failing eight nodes deep. Intranode is NVLink-only (same for every backend), -so it is a smoke test, not a comparison row. +```bash +python3 summarize_results.py /path/to/artifacts/runs \ + --starts=3 \ + --provenance=/path/to/artifacts/control/provenance.json \ + --json=/path/to/artifacts/summary/summary.json \ + --markdown=/path/to/artifacts/summary/summary.md +``` + +If a native library appends a diagnostic to the JSON marker's physical line, use the repository parser: ```bash -cp env_vars.example env_vars # then edit image URIs / topology -source env_vars - -# 1) NVSHMEM (DeepEP) -( cd ../deepep-benchmark/kubernetes - IMAGE_URI=$NVSHMEM_IMAGE_URI NUM_NODES=$NUM_NODES \ - envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' < test-internode.yaml | kubectl apply -f - - # ...wait, save logs, delete. Then low-latency -- see the override note below. ) - -# 2) UCCL (UCCL-EP) — one MPI rank per GPU (NP = NUM_NODES * GPU_PER_NODE) -( cd ../uccl-ep-benchmark/kubernetes - IMAGE_URI=$UCCL_IMAGE_URI NUM_NODES=$NUM_NODES NP=$NP \ - envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < test-internode.yaml | kubectl apply -f - - # ...then test-low-latency.yaml (already pinned to --num-experts=256) ) - -# 3) NCCL baseline (reuses the DeepEP image's alltoall_perf) -IMAGE_URI=$NCCL_IMAGE_URI \ -envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < nccl-alltoall.yaml | kubectl apply -f - +python3 extract_results.py rank-zero.log results.jsonl ``` -> **DeepEP low-latency at 8 nodes — required override.** The merged DeepEP low-latency manifest -> runs `python3 /DeepEP/tests/test_low_latency.py` with no args, so it uses the upstream default -> `--num-experts=288`. The test asserts `num_experts % num_ranks == 0`; at 8 nodes (64 ranks), -> `288 % 64 ≠ 0` and it aborts. Match the comparison's 256 by patching the rendered manifest: -> ```bash -> cd ../deepep-benchmark/kubernetes -> IMAGE_URI=$NVSHMEM_IMAGE_URI NUM_NODES=$NUM_NODES \ -> envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' < test-low-latency.yaml \ -> | sed 's#test_low_latency.py#test_low_latency.py --num-experts 256#' | kubectl apply -f - -> ``` -> (DeepEP internode defaults are already 4096/7168/8/**256**, so internode needs no override.) - -Save each launcher log (`kubectl logs <…-launcher> > .log`) and **delete the job before -the next run** (MPIJob names are fixed; re-applying collides, and each job needs all 8 nodes): +The summarizer rejects an incomplete matrix, correctness failure, mutable image tag, route/input mismatch, runtime-stack mismatch, or disagreement in common logical payload accounting. + +Regenerate the box plots from the committed machine-readable summary: ```bash -kubectl delete mpijob deepep-internode uccl-ep-internode nccl-alltoall # etc. +python3 plot_results.py results/b200-ap-south-1-2026-08-25.json \ + --output=results/b200-ap-south-1-2026-08-25-boxplots.png ``` -## Collate +Plot generation requires Matplotlib. Each box uses the 3 independent process-start medians for one backend and workload cell. The plot also shows every underlying point. + +## Local validation ```bash -python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log \ - --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log \ - --uccl-lowlat uccl_lowlat.log \ - --nccl nccl_alltoall.log +python3 -m pytest -q test_ep_benchmark.py test_summarize_results.py +python3 -m py_compile \ + ep_benchmark.py result_io.py extract_results.py summarize_results.py plot_results.py +bash -n run_ep_comparison.sh run_ep_rank.sh +shellcheck run_ep_comparison.sh run_ep_rank.sh ``` -The parser reports, for internode, the **RDMA** leg of the "Best dispatch/combine" line (the -cross-node bottleneck — *not* the intra-node NVL number printed on the same line), and for the -NCCL baseline the busbw **at the EP per-rank payload size** (`num_tokens*hidden*2`, ~56 MiB -target; the power-of-two sweep reports the nearest sampled row, 64 MiB) as -well as the asymptotic peak. Record the table in [`RESULTS.md`](RESULTS.md) with image tags, -date, and any config deltas. Eyeball one real launcher log against the parser before trusting it. - -Results are recorded per platform: [`RESULTS.md`](RESULTS.md) (B300) and -[`RESULTS-p5.md`](RESULTS-p5.md) (P5/H100). For other instance types set `INSTANCE_TYPE` and -`EFA_PER_NODE` accordingly (e.g. `p5.48xlarge` exposes **32** EFA NICs vs **16** on `p6-b300`). - -## Scaling beyond 8 nodes (256-rank findings) - -The full matrix was pushed to 16 and 32 nodes (128 / 256 ranks) on a 32× `p6-b300` Capacity -Block on 2026-07-14. **Every DeepEP-class kernel hits a hard implementation limit between -65 and 256 ranks; only the NCCL reference runs at 256.** Details and the per-limit source -citations are in [`RESULTS.md`](RESULTS.md) ("32 / 16 nodes" section). Operational notes for -anyone re-running at scale: - -- **HT internode**: DeepEP asserts at >160 ranks (`NUM_MAX_NVL_PEERS 8 × NUM_MAX_RDMA_PEERS 20`, - `kernels/configs.cuh`) and its stock combine tuning tables already abort at 16 nodes; UCCL - overflows an `int32` buffer bound above 64 ranks. Treat the HT comparison as an - **8-nodes-per-EP-domain benchmark** — which matches how training deploys these kernels - (EP32/EP64 groups inside a larger world). -- **Low-latency**: both implementations cap between 64 and 128 ranks (UCCL: compile-time - signaling-buffer arena; NVSHMEM/DeepEP: libfabric host-proxy retry exhaustion with moving - victims per run). -- **GDRCopy at scale (NVSHMEM)**: past ~1 GiB of LL buffer, NVSHMEM grows its symmetric heap - dynamically and must register each chunk over libfabric via **GDRCopy inside the container**. - The manifests set `NVIDIA_GDRCOPY=enabled`, but some clusters' nvidia container toolkit - ignores it — if every rank dies at `mem_heap.cpp:1361 register_mem_handle failed` after a - `GDRCopy support not enabled` warning, hostPath-mount `/dev/gdrdrv` into the worker - (requires `privileged: true`) and ensure the host loads `gdrdrv` (gdrcopy-loader DaemonSet - or DLAMI). -- **NCCL at 32 nodes** works unmodified (`NUM_NODES=32`, `NP=256`); expect matched-size busbw - to drop vs 8 nodes (fan-out cost). - -## Caveats - -- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure transport throughput; the - EP dispatch/combine numbers carry routing + reduction overhead, so they should sit *below* the - NCCL ceiling. Compare against the **matched-size** busbw, not the asymptotic peak. -- **Internode = RDMA leg.** DeepEP/UCCL print both an RDMA (cross-node) and an NVL (intra-node) - bandwidth on the same line; only the RDMA number reflects the inter-node transport being - compared. -- **`num-experts` must divide the world size.** Both tests assert `num_experts % num_ranks == 0`. - At 8 nodes (64 ranks) the comparison uses 256 (= 4/rank). The DeepEP low-latency default (288) - is not divisible by 64 and must be overridden (see the run-order note). -- **Toolchain.** All three images are CUDA 13 (NVSHMEM/NCCL share the DeepEP image; UCCL is - CUDA 13 per `uccl-ep.Dockerfile`), so there is no CUDA skew across backends. -- **UCCL bench scripts** are pulled from upstream `uccl/ep/bench` at image-build time and pinned - via `UCCL_COMMIT`. If upstream renames CLI flags, adjust the bench args in the UCCL - manifests. +## Scope limits + +These profiles measure synthetic dispatch-plus-combine communication. Prefill-like does not measure TTFT, and Decode-like does not measure TPOT. Neither profile measures expert compute, communication/computation overlap, end-to-end training, serving throughput, or end-to-end latency. Results apply only to the reported profile, EP size, routing distribution, hardware, and runtime stack. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md deleted file mode 100644 index d9184ead4..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md +++ /dev/null @@ -1,76 +0,0 @@ -# EP-Backend Comparison Results — P5 (H100) - -Same harness and matched config as [`RESULTS.md`](RESULTS.md) (B300), run on `p5.48xlarge` -(H100). Use this to compare the three dispatchers **across GPU generations**. - -## Environment - -| Field | Value | -|---|---| -| Date | 2026-06-22 | -| Hardware | `p5.48xlarge` on EKS (H100, 8 GPU + **32 EFA** / node) | -| EP config | num-tokens=4096 (LL: 128), hidden=7168, num-topk=8, num-experts=256, bf16 (identical to B300) | -| NVSHMEM image | `deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13` (sm_90) | -| UCCL image | UCCL `0dc87eb`, CUDA 13 (Hopper sm_90 path of the committed `uccl-ep.Dockerfile`) | -| NCCL image | DeepEP image (`/opt/nccl-tests/build/alltoall_perf`) | - -> **Node bring-up note.** One of the 8 freshly-deployed P5 nodes initially had its NVSwitch -> **Fabric Manager stuck in `In Progress`** (every CUDA op on it failed with `error 802: -> system not yet initialized`, including a trivial 1-GPU test). The 4-node run was done on the -> 7 healthy nodes meanwhile; after that node was recycled the 8-node run completed on all 8. - -## 8 nodes — 64 ranks - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 39.6 | 39.2 | -| NVSHMEM (DeepEP) | low-latency | 6.5 | 15.6 | -| UCCL (UCCL-EP) | internode (RDMA) | 29.9 | 26.7 | -| UCCL (UCCL-EP) | low-latency | 3.1¹ | 3.7¹ | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 40.4 | -| busbw peak (asymptotic) | peak | 51.1 | - -¹ Same as B300, the standard FP8 low-latency path (`round_scale=False`) **passes** correctness at -64 ranks (max diff 1.07e-4 vs the 9e-4 FP8 tolerance — 8× margin) and gives these numbers. The -unpatched test aborts *first* on the coarser `round_scale=True` FP8 sub-case, which upstream DeepEP -exempts via `if not round_scale`; matching that gating recovers the bandwidth. The per-sub-case -errors are **identical to B300** (the reference is generated from fixed seeds), confirming this is a -quantization-recipe property, not GPU-arch. See [`RESULTS.md`](RESULTS.md) for the sub-case table. - -## 4 nodes — 32 ranks - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 43.7 | 46.1 | -| NVSHMEM (DeepEP) | low-latency | 9.6 | 20.7 | -| UCCL (UCCL-EP) | internode (RDMA) | 38.2 | 30.2 | -| UCCL (UCCL-EP) | low-latency | 5.0 | 5.0 | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 45.1 | -| busbw peak (asymptotic) | peak | 53.9 | - -## Observations (and the B300 contrast) - -- **The winner flips by GPU generation.** On **B300** (see `RESULTS.md`) UCCL matches/beats - NVSHMEM at 4 nodes and clearly wins at 8. On **P5/H100 the order reverses at both scales**: - NVSHMEM leads UCCL on internode (4n 44/46 vs 38/30; 8n **40/39 vs 30/27** GB/s RDMA) and on - low-latency (8n LL 6.5/15.6 vs 3.1/3.7). UCCL-EP's kernels lean on SM90+ features tuned for - Blackwell; on H100 they trail NVSHMEM here. **Pick the dispatcher per target GPU, not globally.** -- **Absolute bandwidth is ~half of B300.** P5 internode tops out ~40–46 GB/s (RDMA) and the NCCL - reference ~40–45 (matched) / ~51–54 (peak), versus ~73–96 / ~103–117 on B300 — a combination of - EFA throughput and NVLink-generation differences. -- **UCCL low-latency's 64-rank abort is a test-gate divergence, not a kernel fault.** The standard - `round_scale=False` FP8 path passes at 64 ranks (max diff 1.07e-4, identical on P5 and B300); the - default test aborts only on the coarser `round_scale=True` sub-case that DeepEP exempts via - `if not round_scale`. Matching that gating recovers the LL bandwidth (P5 3.1/3.7, B300 28.4/24.8). - See [`RESULTS.md`](RESULTS.md) for the sub-case breakdown. - -## Reproduce - -Identical to [`README.md`](README.md), with `INSTANCE_TYPE=p5.48xlarge` and `EFA_PER_NODE=32` -(p5.48xlarge exposes 32 EFA NICs vs 16 on p6-b300). diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index 198f52473..3e9ae3efc 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -1,183 +1,109 @@ -# EP-Backend Comparison Results — B300 +# EP Backend Comparison Results on B200 -Generated by running the three backends at a matched EP config on `p6-b300.48xlarge` -nodes and collating with [`collect_results.py`](collect_results.py). For the **P5 (H100)** run -of the same matrix — and the cross-generation contrast (the winner flips by GPU) — see -[`RESULTS-p5.md`](RESULTS-p5.md). +Status: `PASS` -## Environment +The previous Decode-only result has been discarded. It must not be combined with this two-profile matrix because Prefill-like uses the normal high-throughput API and a different timing boundary. -| Field | Value | +This is a synthetic expert-parallel communication microbenchmark. + +## Campaign configuration + +| Control | Value | |---|---| -| Date | 2026-06-21 | -| Hardware | `p6-b300.48xlarge` on EKS (Blackwell B300, 8 GPU + 16 EFA / node) | -| EP config | num-tokens=4096 (LL: 128), hidden=7168, num-topk=8, num-experts=256, bf16 | -| NVSHMEM image | `deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13` (CUDA 13) | -| UCCL image | UCCL `0dc87eb`, CUDA 13, Hopper+Blackwell — see image note below | -| NCCL image | DeepEP image (reuses `/opt/nccl-tests/build/alltoall_perf`, sm_100) | - -`num-experts=256` divides evenly at both 32 and 64 ranks, so the config is identical across the -4-node and 8-node runs. - -> **UCCL image note.** The UCCL dispatch/combine numbers below were measured with a prebuilt -> image of the **same UCCL commit (`0dc87eb`) and the same CUDA-13 `setup.py`+PTX build method** -> as the committed [`uccl-ep.Dockerfile`](../uccl-ep-benchmark/uccl-ep.Dockerfile). That committed -> Dockerfile is independently validated to **build** and to **run the benchmark on B300** — a -> 2-node internode run from it gives dispatch ~92 / combine ~60 GB/s (RDMA), consistent with the -> table — so it reproduces these results. - -## 32 / 16 nodes — 256 / 128 ranks (2026-07-14 scale run) - -A 32× `p6-b300.48xlarge` Capacity Block (**us-east-1-atl-2a local zone**, EKS, same images -as below) was used to sweep the same matched config from 2 to 32 nodes. Headline: -**every DeepEP-class kernel hits a hard implementation limit between 65 and 256 ranks — -at 256 ranks only the NCCL all-to-all reference runs.** - -**HT internode scaling (like-for-like, this cluster, 2026-07-14).** The matched config -(`num-experts=256`) is only *runnable* at power-of-2 node counts: -`tests/test_internode.py:30` asserts `num_experts % num_ranks == 0`, and 256 divides -16/32/64/128/256 ranks but not the 24/48/96/144/160 of DeepEP's other instantiated -shapes ({3, 6, 12, 18, 20} nodes — a menu cut for 288-expert models, where those counts -do divide). Dispatch/combine are the RDMA leg in GB/s: - -| nodes | ranks | NVSHMEM (DeepEP) disp / comb | UCCL disp / comb | NCCL matched / peak | -|---|---|---|---|---| -| 2 | 16 | **126.6 / 106.4** | 91.9 / 59.6 | 104.9 / 179.6 | -| 4 | 32 | 97.1 / 95.4 | **102.1 / 95.3** | 94.0 / 116.9 | -| 8 | 64 | 84.2 / 73.1 | **93.9 / 90.5** | 74.0 / 103.2 | -| 16 | 128 | 74.7 / tuning abort¹ | constructor abort³ | 73.8 / 84.0 | -| 32 | 256 | constructor abort² | constructor abort³ | 54.7 / 74.4 | - -Two reads: **(a)** the 8-node row reproduces the June `us-west-2` primary table below -within ~1% on a different cluster (NVSHMEM 84.2/73.1 vs 83.7/72.7; UCCL 93.9/90.5 vs -93.4/90.7; NCCL 74.0 vs 72.5) — strong cross-cluster reproducibility for these -benchmarks. **(b)** the winner flips with scale: NVSHMEM leads at 2 nodes, UCCL from -4 nodes up — per-rank bandwidth decays smoothly for all three as fan-out grows. - -**Low-latency kernels** cap between 64 and 128 ranks on both implementations -(⁴ and ⁵ below); at 256 ranks nothing but NCCL runs: - -| Backend / kernel | 128 ranks (16n) | 256 ranks (32n) | Limit (source) | -|---|---|---|---| -| NVSHMEM (DeepEP) low-latency | init traffic abort⁴ | init traffic abort⁴ | 64 < cap ≤ 128 PEs (host-proxy) | -| UCCL (UCCL-EP) low-latency | buffer-config abort⁵ | buffer-config abort⁵ | 64 < cap ≤ 128 ranks | - -¹ `internode.cu:2363` — `num_max_nvl_chunked_recv_tokens / num_rdma_ranks > max(send chunk)`: -the shipped chunk configs were tuned for ≤8-node EP domains; at 16 RDMA peers the combine -sweep violates the constraint. Dispatch (74.7 GB/s RDMA, ~143 GB/s NVL) is a valid datapoint. -² `deep_ep.cpp:158` — `num_ranks <= NUM_MAX_NVL_PEERS(8) × NUM_MAX_RDMA_PEERS(20) = 160 or -low_latency_mode`. DeepEP v1's HT kernels decompose every rank into `(rdma_rank, nvl_rank)` -against fixed 8×20 compile-time tables (`kernels/configs.cuh`). The same assert block also -bounds `num_rdma_bytes <= INT_MAX` for HT — waiting behind the peer cap even if it were raised. -³ `uccl_ep.cc:431` — HT `num_rdma_bytes` (∝ ranks at matched config) exceeds `INT_MAX` -somewhere between 64 ranks (8n, June: passes at 93 GB/s) and 128 ranks. -⁴ NVSHMEM 3.7 libfabric **host-proxy** retry exhaustion: `Max amount of libfabric retries -reached, -11 (EAGAIN)` in `nvshmemi_process_multisend_rma`, killing ~2 nodes per run with -**different victims each run** (4 runs: ranks {12,29}, {9,12}, {12}, {6,11}) — a statistical -fan-out limit of the single proxy thread at ≥128 PEs on EFA, not a bad node and not geometry. -Registration itself is solvable: the ≥1 GiB LL buffer forces dynamic (CUDA-VMM) heap growth -whose remote-chunk registration needs **GDRCopy in-container** — on clusters whose toolkit -ignores `NVIDIA_GDRCOPY=enabled`, hostPath-mount `/dev/gdrdrv` (privileged); a >2 GiB static -heap is no workaround (exceeds EFA's single-MR registration limit). -⁵ `ep_config.hpp:279` — LL per-peer signaling buffer (∝ ranks) exceeds the compile-time -`kAtomicBufferSize` arena at ≥128 ranks. - -**Reading.** DeepEP-class dispatchers are engineered for EP domains of ~64–160 ranks — -matching how training actually deploys them (EP32/EP64 groups inside a larger world; the -[`kimi-k2`](../../../3.test_cases/megatron/megatron-bridge/kimi-k2/benchmarks/RESULTS.md) -NVSHMEM arm ran clean on 256 GPUs precisely because its `deep_ep` domains are 32-rank EP -groups). A *flat* EP domain >64 ranks is already off the map for both low-latency paths on -EFA, and >160 for HT; NCCL all-to-all is the only working option there. NCCL's per-rank -busbw at the matched ~56 MiB payload decays smoothly with fan-out across the whole sweep: -104.9 → 94.0 → 74.0 → 73.8 → 54.7 GB/s over 16 → 256 ranks (like-for-like, this cluster). - -## 8 nodes — 64 ranks (primary) - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 83.7 | 72.7 | -| NVSHMEM (DeepEP) | low-latency | 9.2 | 18.6 | -| UCCL (UCCL-EP) | internode (RDMA) | 93.4 | 90.7 | -| UCCL (UCCL-EP) | low-latency | 28.4¹ | 24.8¹ | - -¹ The standard FP8 low-latency path (`round_scale=False`) **passes** correctness at 64 ranks and -gives these numbers. The unpatched test still aborts *first* on the coarser `round_scale=True` FP8 -sub-case — which upstream DeepEP exempts from the same check — so reproducing with the committed -manifest shows the abort; matching DeepEP's gating recovers the bandwidth. See the note below. - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 72.5 | -| busbw peak (asymptotic) | peak | 103.1 | - -> **UCCL low-latency at 64 ranks: the abort is a test-gate divergence, not a kernel precision regression.** -> `bench/test_low_latency.py` validates a dispatch→(identity GEMM)→combine round-trip against the -> closed-form reference `current_x · Σ(top-k gate weights)`, using the global similarity error -> `calc_diff(x,y) = 1 − 2·⟨x,y⟩ / (‖x‖²+‖y‖²)`. UCCL asserts `diff < 9e-4` for **every** FP8 -> sub-case: -> -> ```python -> assert diff < (9e-4 if dispatch_use_fp8_case else 1e-5) # uccl ep/bench/test_low_latency.py:373 -> ``` -> -> whereas upstream DeepEP gates the *same* assert with `if not round_scale` -> (`deepseek-ai/DeepEP tests/test_low_latency.py:178`) — it deliberately exempts the coarse, -> power-of-2-scale (`round_scale=True`) FP8 path. Instrumenting the run to log every sub-case's error -> at 64 ranks (256 samples per sub-case across the 64 ranks) shows the split exactly: -> -> | dispatch_use_fp8 | round_scale | max diff | tolerance | result | -> |---|---|---:|---:|---| -> | False (bf16) | — | 1.7e-6 | 1e-5 | pass | -> | True | **False** | **1.07e-4** | 9e-4 | **pass (8× margin)** | -> | True | **True** | **1.80e-3** | 9e-4 | fail | -> -> So the **standard `round_scale=False` FP8 path is numerically correct at 64 ranks** (and at 32); -> the default test aborts only because UCCL applies the tight FP8 tolerance to the `round_scale=True` -> sub-case that DeepEP does not gate. `round_scale=True` is a coarser (power-of-2 / UE8M0-family) -> scaling recipe whose larger quantization error is expected — hence DeepEP's exemption. The -> dispatch/combine numbers in the table are the `round_scale=False` FP8 timed phase, recovered by -> matching DeepEP's gating (skipping the `round_scale=True` assert) — **not** by loosening the -> tolerance on the path the table measures. - -## 4 nodes — 32 ranks (scaling reference) - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 97.2 | 95.6 | -| NVSHMEM (DeepEP) | low-latency | 11.2 | 25.2 | -| UCCL (UCCL-EP) | internode (RDMA) | 101.3 | 94.4 | -| UCCL (UCCL-EP) | low-latency | 44.5 | 45.3 | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 93.9 | -| busbw peak (asymptotic) | peak | 116.9 | - -## Observations - -- **Internode high-throughput: a tie at 4 nodes, a UCCL win at 8.** At 32 ranks NVSHMEM and UCCL - are within a few percent (97/96 vs 101/94). At 64 ranks UCCL pulls ahead on both legs — - dispatch 93 vs 84 and **combine 91 vs 73 GB/s (RDMA)** — i.e. UCCL's combine degrades far less - as the RDMA peer count doubles. Bandwidth drops with scale for both backends (more RDMA peers, - more contention). -- **Low-latency (decode path): UCCL wins.** At 32 ranks UCCL delivers ~4× dispatch and ~1.8× - combine over NVSHMEM (44/45 vs 11/25). At 64 ranks UCCL leads on both legs too — **28.4/24.8 vs - 9.2/18.6** (~3× dispatch, ~1.3× combine) — once the test's `round_scale=True` gate (which DeepEP - exempts) is matched; the standard `round_scale=False` FP8 path passes at this scale (see note). -- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure data movement (no token - routing / combine-reduction) and is measured differently from the EP kernels' RDMA-only, - NVL-overlapped bandwidth — so the EP numbers can sit *above* the NCCL matched-size busbw (e.g. - UCCL 93 vs 72.5 at 8 nodes) while staying below the asymptotic peak (103). Treat it as a - transport context line, not a hard ceiling. - -## Reproduce - -See [`README.md`](README.md). Logs were collected with `kubectl logs > .log` -and collated with: - -```bash -python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log --uccl-lowlat uccl_lowlat.log \ - --nccl nccl_alltoall.log -``` +| Hardware | 4 `p6-b200.48xlarge` nodes, each with 8 NVIDIA B200 GPUs and 8 EFA devices | +| EP sizes | 16 GPU ranks on 2 nodes and 32 GPU ranks on 4 nodes | +| Model shape | Hidden size 7,168 dimensions, 256 experts, top-k 8 experts/token | +| Decode-like profile | 128 tokens/rank, low-latency dispatch and combine | +| Prefill-like profile | 4,096 tokens/rank, normal dispatch and combine with required layout | +| Data types | FP8 or BF16 dispatch, BF16 combine | +| Sampling | 20 warmup iterations and 100 measured iterations per dtype/start | +| Replication | 3 independent process starts per backend/profile/EP-size cell | +| Runtime | PyTorch 2.13.0+cu130, CUDA 13.0, NCCL 2.29.7 | +| Region and cluster | `ap-south-1c`, EKS `ml-clusters-shared-ap-south-1` | +| Campaign | `ep-b200-profiles-20260825t023933z`, commit `97c93ee33535ffb41b7635befc4a97617ab8acef` | + +Each backend receives the same deterministic BF16 input, exact top-k route, and weights for a given profile and EP size. SHA-256 input and route hashes must agree across every backend and process start. One external CUDA event boundary measures from input readiness through required conversion/layout, dispatch, and combine completion. Each iteration uses the maximum elapsed time across ranks. + +Decode-like reports slowest-rank latency. Prefill-like reports logical GB/s/rank using one payload numerator: useful dispatch data, required FP8 scales, and BF16 combine data per valid expert assignment. Backend metadata is excluded. These values are logical efficiency metrics, not observed wire bandwidth. + +## Results + +Each cell has 3 independent process starts. Values are medians across independent starts. Decode-like results use slowest-rank latency as the primary metric; prefill-like results use common logical throughput. These are synthetic communication workloads, not end-to-end training or serving results. + +![Box plots comparing the three backend arms for every Decode-like and Prefill-like workload cell](results/b200-ap-south-1-2026-08-25-boxplots.png) + +Each box contains the 3 independent process-start medians for one backend arm and workload cell. The box spans Q1 to Q3, the center line is the median, the whiskers are the minimum and maximum, and the markers show all 3 values. The individual markers are the primary evidence because each box contains only 3 independent starts. + +## Decode-like latency, 128 tokens/rank + +| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) | +|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| +| 16 ranks | FP8 | UCCL | 0.5215 ms | [0.5124, 0.5321] ms | 1.89% | 3,926,862.25 tokens/s | 42.66 GB/s/rank | 21.33 GB/s/rank | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 1.0319 ms | [1.0148, 1.0429] ms | 1.38% | 1,984,619.22 tokens/s | 21.56 GB/s/rank | 10.78 GB/s/rank | +| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 0.6151 ms | [0.6112, 0.6488] ms | 3.31% | 3,329,691.42 tokens/s | 36.17 GB/s/rank | 18.09 GB/s/rank | +| 16 ranks | BF16 | UCCL | 0.5895 ms | [0.5859, 0.5971] ms | 0.97% | 3,474,201.34 tokens/s | 49.81 GB/s/rank | 24.90 GB/s/rank | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 1.0438 ms | [1.0214, 1.0520] ms | 1.52% | 1,962,016.67 tokens/s | 28.13 GB/s/rank | 14.06 GB/s/rank | +| 16 ranks | BF16 | DeepEP V2 NCCL GIN | 0.4921 ms | [0.4858, 0.5009] ms | 1.54% | 4,161,789.51 tokens/s | 59.66 GB/s/rank | 29.83 GB/s/rank | +| 32 ranks | FP8 | UCCL | 0.7633 ms | [0.7614, 0.7741] ms | 0.89% | 5,366,426.29 tokens/s | 29.15 GB/s/rank | 21.86 GB/s/rank | +| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 1.5522 ms | [1.5435, 1.5657] ms | 0.72% | 2,638,821.59 tokens/s | 14.33 GB/s/rank | 10.75 GB/s/rank | +| 32 ranks | FP8 | DeepEP V2 NCCL GIN | 0.9642 ms | [0.9454, 0.9716] ms | 1.41% | 4,248,046.17 tokens/s | 23.08 GB/s/rank | 17.31 GB/s/rank | +| 32 ranks | BF16 | UCCL | 0.8653 ms | [0.8620, 0.8686] ms | 0.38% | 4,733,640.25 tokens/s | 33.93 GB/s/rank | 25.45 GB/s/rank | +| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 1.5551 ms | [1.5467, 1.5694] ms | 0.74% | 2,633,988.78 tokens/s | 18.88 GB/s/rank | 14.16 GB/s/rank | +| 32 ranks | BF16 | DeepEP V2 NCCL GIN | 0.9477 ms | [0.9448, 0.9491] ms | 0.23% | 4,322,061.08 tokens/s | 30.98 GB/s/rank | 23.24 GB/s/rank | + +## Prefill-like throughput, 4,096 tokens/rank + +| EP size | Dispatch dtype | Backend | Logical throughput (GB/s/rank) | 95% bootstrap CI (GB/s/rank) | Run-to-run CV (%) | Latency (ms) | Input throughput (tokens/s) | Scale-out logical throughput (GB/s/rank) | +|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| +| 16 ranks | FP8 | UCCL | 128.77 GB/s/rank | [128.60, 129.14] GB/s/rank | 0.22% | 5.5289 ms | 11,853,360.76 tokens/s | 64.39 GB/s/rank | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 213.89 GB/s/rank | [213.23, 214.41] GB/s/rank | 0.28% | 3.3287 ms | 19,687,953.23 tokens/s | 106.94 GB/s/rank | +| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 252.84 GB/s/rank | [252.42, 254.65] GB/s/rank | 0.47% | 2.8160 ms | 23,272,859.42 tokens/s | 126.42 GB/s/rank | +| 16 ranks | BF16 | UCCL | 170.98 GB/s/rank | [170.42, 171.27] GB/s/rank | 0.26% | 5.4950 ms | 11,926,565.80 tokens/s | 85.49 GB/s/rank | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 263.71 GB/s/rank | [262.95, 265.06] GB/s/rank | 0.40% | 3.5627 ms | 18,395,265.04 tokens/s | 131.86 GB/s/rank | +| 16 ranks | BF16 | DeepEP V2 NCCL GIN | 277.37 GB/s/rank | [276.15, 277.75] GB/s/rank | 0.30% | 3.3873 ms | 19,347,586.02 tokens/s | 138.68 GB/s/rank | +| 32 ranks | FP8 | UCCL | 85.46 GB/s/rank | [85.41, 85.49] GB/s/rank | 0.05% | 8.3311 ms | 15,732,818.61 tokens/s | 64.10 GB/s/rank | +| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 54.07 GB/s/rank | [53.40, 54.41] GB/s/rank | 0.95% | 13.1683 ms | 9,953,621.84 tokens/s | 40.55 GB/s/rank | +| 32 ranks | FP8 | DeepEP V2 NCCL GIN | 105.21 GB/s/rank | [105.11, 105.36] GB/s/rank | 0.12% | 6.7672 ms | 19,368,582.57 tokens/s | 78.91 GB/s/rank | +| 32 ranks | BF16 | UCCL | 95.31 GB/s/rank | [95.26, 95.38] GB/s/rank | 0.06% | 9.8578 ms | 13,296,240.83 tokens/s | 71.48 GB/s/rank | +| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 70.95 GB/s/rank | [70.48, 71.07] GB/s/rank | 0.44% | 13.2417 ms | 9,898,441.78 tokens/s | 53.21 GB/s/rank | +| 32 ranks | BF16 | DeepEP V2 NCCL GIN | 108.20 GB/s/rank | [108.14, 108.36] GB/s/rank | 0.11% | 8.6835 ms | 15,094,422.79 tokens/s | 81.15 GB/s/rank | + +## Paired DeepEP V2 improvements + +Positive values mean DeepEP V2 had lower latency for Decode-like cells or higher logical throughput for Prefill-like cells. A direction is supported only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV. + +| Profile | EP size | Dispatch dtype | Baseline | Primary metric | Median improvement (%) | 95% bootstrap CI (%) | Direction supported | +|:---|---:|:---:|:---|:---|---:|:---:|:---:| +| decode | 16 ranks | FP8 | UCCL | slowest-rank latency in milliseconds | -20.05% | [-21.94, -17.19]% | yes | +| decode | 16 ranks | FP8 | DeepEP V1 NVSHMEM | slowest-rank latency in milliseconds | 40.40% | [36.06, 41.40]% | yes | +| decode | 16 ranks | BF16 | UCCL | slowest-rank latency in milliseconds | 16.52% | [16.12, 17.08]% | yes | +| decode | 16 ranks | BF16 | DeepEP V1 NVSHMEM | slowest-rank latency in milliseconds | 52.39% | [51.82, 53.46]% | yes | +| decode | 32 ranks | FP8 | UCCL | slowest-rank latency in milliseconds | -25.51% | [-26.33, -24.16]% | yes | +| decode | 32 ranks | FP8 | DeepEP V1 NVSHMEM | slowest-rank latency in milliseconds | 37.88% | [37.06, 39.62]% | yes | +| decode | 32 ranks | BF16 | UCCL | slowest-rank latency in milliseconds | -9.18% | [-10.10, -9.10]% | yes | +| decode | 32 ranks | BF16 | DeepEP V1 NVSHMEM | slowest-rank latency in milliseconds | 38.97% | [38.92, 39.61]% | yes | +| prefill | 16 ranks | FP8 | UCCL | effective logical gigabytes per second per rank | 96.02% | [95.78, 98.02]% | yes | +| prefill | 16 ranks | FP8 | DeepEP V1 NVSHMEM | effective logical gigabytes per second per rank | 18.57% | [17.73, 19.06]% | yes | +| prefill | 16 ranks | BF16 | UCCL | effective logical gigabytes per second per rank | 62.45% | [61.23, 62.76]% | yes | +| prefill | 16 ranks | BF16 | DeepEP V1 NVSHMEM | effective logical gigabytes per second per rank | 4.79% | [4.72, 5.48]% | yes | +| prefill | 32 ranks | FP8 | UCCL | effective logical gigabytes per second per rank | 23.19% | [22.99, 23.23]% | yes | +| prefill | 32 ranks | FP8 | DeepEP V1 NVSHMEM | effective logical gigabytes per second per rank | 94.40% | [93.36, 97.30]% | yes | +| prefill | 32 ranks | BF16 | UCCL | effective logical gigabytes per second per rank | 13.58% | [13.47, 13.61]% | yes | +| prefill | 32 ranks | BF16 | DeepEP V1 NVSHMEM | effective logical gigabytes per second per rank | 52.73% | [52.23, 53.43]% | yes | + +## Qualification and custody + +- All 4 admission cases and all 36 scored distributed starts passed. +- All 8 admission records and all 72 scored records passed the common correctness check. +- The report uses the median of 100 iterations within each start, followed by the median across 3 independent starts. The displayed 95 percent intervals are paired bootstrap intervals across starts. +- Backend, profile, and dtype order rotated across starts. Every arm at a given EP size used the same named nodes. +- All 3 container images were pinned by SHA-256 digest. Runtime, input hash, route hash, payload accounting, and derived metrics were validated during aggregation. +- The durable checksum manifest contains 573 entries, and all 573 entries passed independent verification. +- The owned namespace is absent, the shared Lease holder is empty, remaining owned resources are 0 resources, and active GPU Pods on the 4 selected nodes are 0 Pods. +- The selected node set was disjoint from 12 protected ap-south-1 nodes. No protected node appears in any rendered Pod record. +- With the pinned UCCL image, normal-mode explicit proxy destruction invalidated the CUDA context after result emission. Prefill-like UCCL workers therefore synchronize, flush their results, and exit the worker process. This cleanup happens after all timed iterations and does not change the timing boundary. +- The [machine-readable summary](results/b200-ap-south-1-2026-08-25.json) has SHA-256 `16f12a0bfbbc0c9c9f93eeb19d410cc8843ed6cbc836115df851cd97996a9603`. + +## Interpretation limits + +Prefill-like does not measure time to first token, and Decode-like does not measure time per output token. Neither profile measures expert computation, communication/computation overlap, end-to-end training, end-to-end serving throughput, or end-to-end latency. Conclusions apply only to the reported payload, routing distribution, EP size, B200 hardware, and runtime stack. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py deleted file mode 100644 index 86406ad7c..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -"""Collate EP-backend comparison logs into a single markdown table. - -Parses the launcher logs from the three benchmarks run at the same world size: - - * NVSHMEM (DeepEP) -- dispatch/combine bandwidth - * UCCL (UCCL-EP) -- dispatch/combine bandwidth - * NCCL (nccl-tests alltoall_perf) -- busbw (transport reference) - -Output formats parsed (DeepEP and UCCL print identically -- UCCL's bench is -derived from DeepEP's tests): - - internode (test_internode.py): - [tuning] Best dispatch (BF16): ... BW: GB/s (RDMA), GB/s (NVL) - [tuning] Best combine: ... BW: GB/s (RDMA), GB/s (NVL) - -> we report the RDMA leg (the cross-node bottleneck). Reporting the NVL leg - here would be wrong: it is the intra-node ~hundreds-GB/s number. - - low-latency (test_low_latency.py): - [rank N] Dispatch bandwidth: GB/s, avg_t=... | Combine bandwidth: GB/s, avg_t=... - -> single bandwidth per dispatch/combine (no RDMA/NVL split). - - NCCL alltoall_perf: the standard size-sweep table; busbw is column 7 - (out-of-place) / 11 (in-place). We report busbw at the sampled row whose size - is closest to the EP per-rank dispatch payload (num_tokens * hidden * 2 bytes, - ~56 MiB target; the power-of-two sweep lands on the 64 MiB row) AND the - asymptotic peak, because the peak overstates the transport ceiling relative to - EP's smaller messages. - -Both EP formats vary slightly across versions; if a value comes back N/A, print -the raw log and adjust the regexes. - -Usage: - python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log \ - --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log \ - --uccl-lowlat uccl_lowlat.log \ - --nccl nccl_alltoall.log \ - --nccl-target-bytes 58720256 -""" -import argparse -import re -import sys - -_BW = r"([0-9]+(?:\.[0-9]+)?)" -# internode: pull the RDMA leg from the "Best dispatch/combine" summary lines. -BEST_DISPATCH_RDMA = re.compile(r"Best dispatch[^\n]*?" + _BW + r"\s*GB/s\s*\(RDMA\)", re.I) -BEST_COMBINE_RDMA = re.compile(r"Best combine[^\n]*?" + _BW + r"\s*GB/s\s*\(RDMA\)", re.I) -# low-latency: "Dispatch bandwidth: X GB/s" / "Combine bandwidth: Y GB/s". -# Case-SENSITIVE on purpose: the aggregate line "Dispatch + combine bandwidth: Z" -# uses a lowercase "combine" and must NOT match the per-leg "Combine bandwidth:". -LL_DISPATCH = re.compile(r"Dispatch bandwidth:\s*" + _BW + r"\s*GB/s") -LL_COMBINE = re.compile(r"Combine bandwidth:\s*" + _BW + r"\s*GB/s") -# nccl-tests data row: size count type redop root time algbw busbw #wrong ... -NCCL_ROW_RE = re.compile(r"^\s*\d+\s+\d+\s+\w+") -NCCL_BUSBW_COLS = (7, 11) -# mpirun --tag-output prefixes every line with e.g. "[1,0]:". -TAG_PREFIX_RE = re.compile(r"^\[\d+,\d+\]:") - - -def _last(text, regex): - m = regex.findall(text) - return float(m[-1]) if m else None - - -def _max(text, regex): - m = regex.findall(text) - return max(float(x) for x in m) if m else None - - -def parse_internode(path): - """(dispatch_rdma, combine_rdma) GB/s from the Best-config summary lines.""" - if not path: - return None, None - with open(path) as f: - text = f.read() - # The "Best" line is printed once per config after tuning; take the last. - return _last(text, BEST_DISPATCH_RDMA), _last(text, BEST_COMBINE_RDMA) - - -def parse_lowlat(path): - """(dispatch, combine) GB/s. Per-rank lines; take the max across ranks.""" - if not path: - return None, None - with open(path) as f: - text = f.read() - return _max(text, LL_DISPATCH), _max(text, LL_COMBINE) - - -def parse_nccl(path, target_bytes): - """Return (busbw_at_target, size_at_target, busbw_peak) GB/s.""" - if not path: - return None, None, None - peak = None - best_at = None # (abs_size_delta, size, busbw) - with open(path) as f: - for line in f: - line = TAG_PREFIX_RE.sub("", line) - if line.lstrip().startswith("#") or not NCCL_ROW_RE.match(line): - continue - cols = line.split() - try: - size = int(cols[0]) - except ValueError: - continue - for idx in NCCL_BUSBW_COLS: - if idx >= len(cols): - continue - try: - bw = float(cols[idx]) - except ValueError: - continue - peak = bw if peak is None else max(peak, bw) - delta = abs(size - target_bytes) - if best_at is None or delta < best_at[0]: - best_at = (delta, size, bw) - if best_at is None: - return None, None, peak - return best_at[2], best_at[1], peak - - -def fmt(v): - return f"{v:.1f}" if isinstance(v, float) else "N/A" - - -def main(): - p = argparse.ArgumentParser() - p.add_argument("--nvshmem-internode") - p.add_argument("--nvshmem-lowlat") - p.add_argument("--uccl-internode") - p.add_argument("--uccl-lowlat") - p.add_argument("--nccl", help="NCCL alltoall_perf log (transport reference)") - p.add_argument("--nccl-target-bytes", type=int, default=4096 * 7168 * 2, - help="EP per-rank dispatch payload to read busbw at (default num_tokens*hidden*2)") - args = p.parse_args() - - nv_i_d, nv_i_c = parse_internode(args.nvshmem_internode) - nv_l_d, nv_l_c = parse_lowlat(args.nvshmem_lowlat) - uc_i_d, uc_i_c = parse_internode(args.uccl_internode) - uc_l_d, uc_l_c = parse_lowlat(args.uccl_lowlat) - nccl_at, nccl_size, nccl_peak = parse_nccl(args.nccl, args.nccl_target_bytes) - - out = sys.stdout - out.write("| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) |\n") - out.write("|---|---|---:|---:|\n") - out.write(f"| NVSHMEM (DeepEP) | internode (RDMA) | {fmt(nv_i_d)} | {fmt(nv_i_c)} |\n") - out.write(f"| NVSHMEM (DeepEP) | low-latency | {fmt(nv_l_d)} | {fmt(nv_l_c)} |\n") - out.write(f"| UCCL (UCCL-EP) | internode (RDMA) | {fmt(uc_i_d)} | {fmt(uc_i_c)} |\n") - out.write(f"| UCCL (UCCL-EP) | low-latency | {fmt(uc_l_d)} | {fmt(uc_l_c)} |\n") - out.write("\n") - sz_mib = f"{nccl_size / 2**20:.0f} MiB" if isinstance(nccl_size, int) else "N/A" - out.write("| Reference (NCCL all-to-all, transport ceiling) | Metric | GB/s |\n") - out.write("|---|---|---:|\n") - out.write(f"| busbw at EP payload (~{sz_mib}) | matched-size | {fmt(nccl_at)} |\n") - out.write(f"| busbw peak (asymptotic, overstates ceiling) | peak | {fmt(nccl_peak)} |\n") - - -if __name__ == "__main__": - main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example b/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example deleted file mode 100644 index 1a653a5f4..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 - -# Shared topology for the EP-backend comparison. Copy to env_vars (gitignored), -# edit, then source it before applying the per-backend manifests. -# cp env_vars.example env_vars && source env_vars - -# GPU node instance type and per-node device counts (p6-b300.48xlarge => 16 EFA). -export INSTANCE_TYPE=p6-b300.48xlarge -export GPU_PER_NODE=8 -export EFA_PER_NODE=16 -# 8 = primary 3-way comparison. 32 (256 ranks) works for NCCL and the low-latency kernels -# only — both HT internode paths hit hard caps past 64-160 ranks (see README "Scaling -# beyond 8 nodes"). -export NUM_NODES=8 -export NP=$((NUM_NODES * GPU_PER_NODE)) # ranks; used by the NCCL baseline (64 at 8 nodes) - -# Per-backend image URIs. -# NVSHMEM: built from ../deepep-benchmark/deepep.Dockerfile (CUDA 13, sm_90+sm_100). -# UCCL: built from ../uccl-ep-benchmark/uccl-ep.Dockerfile (CUDA 13; Hopper + Blackwell via PTX). -# NCCL: the DeepEP image already ships /opt/nccl-tests/build/alltoall_perf built with -# sm_100 gencode, so reuse it for the baseline -- no separate nccl-tests build. -export NVSHMEM_IMAGE_URI=.dkr.ecr..amazonaws.com/deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13 -export UCCL_IMAGE_URI=.dkr.ecr..amazonaws.com/uccl-ep:efa1.48.0-uccl0dc87eb-cu13 -export NCCL_IMAGE_URI=$NVSHMEM_IMAGE_URI diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py new file mode 100755 index 000000000..315251c23 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py @@ -0,0 +1,786 @@ +#!/usr/bin/env python3 +"""Common-boundary DeepEP-compatible dispatch/combine benchmark. + +The benchmark deliberately avoids each backend's native timing and byte +accounting. Every arm receives the same deterministic BF16 input, exact route +indices, and top-k weights. A CUDA event pair surrounds the complete dispatch +followed by combine operation, and the slowest rank is the iteration latency. + +The decode profile uses each backend's low-latency path with 128 tokens/rank. +The prefill profile uses the normal high-throughput path with 4,096 tokens/rank +and includes any required dispatch-layout work. FP8 conversion starts from a +BF16 input and remains inside both profiles' timed region. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import statistics +import sys +import traceback +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import torch +import torch.distributed as dist + + +RESULT_PREFIX = "ADAI_EP_RESULT " +SCHEMA_VERSION = 2 +NORMAL_NUM_NVL_BYTES = 2_000_000_000 +NORMAL_NUM_RDMA_BYTES = 1_000_000_000 +NORMAL_NUM_QPS_PER_RANK = 24 + + +def deepep_v2_build_lib(root: Path = Path("/opt/amazon/deepep-v2")) -> Path: + """Locate the single built DeepEP V2 package containing its C extension.""" + + candidates = [ + path + for path in sorted((root / "build").glob("lib.*")) + if any((path / "deep_ep").glob("_C*.so")) + ] + if len(candidates) != 1: + rendered = ", ".join(str(path) for path in candidates) or "none" + raise RuntimeError( + f"expected exactly one built DeepEP V2 package, found: {rendered}" + ) + return candidates[0] + + +def preload_backend(arm: str) -> None: + """Load V2 before NCCL initializes its OFI plugin and tuner libraries.""" + + if arm == "deepep-v2-gin-gda": + sys.path.insert(0, str(deepep_v2_build_lib())) + __import__("deep_ep") + + +def percentile(values: list[float], quantile: float) -> float: + """Return a linearly interpolated percentile without a NumPy dependency.""" + + if not values: + raise ValueError("percentile requires at least one value") + ordered = sorted(values) + position = (len(ordered) - 1) * quantile + lower = math.floor(position) + upper = math.ceil(position) + if lower == upper: + return ordered[lower] + fraction = position - lower + return ordered[lower] * (1 - fraction) + ordered[upper] * fraction + + +def tensor_sha256(tensor: torch.Tensor) -> str: + raw = tensor.detach().contiguous().view(torch.uint8).cpu().numpy().tobytes() + return hashlib.sha256(raw).hexdigest() + + +def global_digest(local_digest: str, device: torch.device) -> str: + digest_tensor = torch.tensor( + list(bytes.fromhex(local_digest)), dtype=torch.uint8, device=device + ) + gathered = [torch.empty_like(digest_tensor) for _ in range(dist.get_world_size())] + dist.all_gather(gathered, digest_tensor) + payload = b"".join(bytes(item.cpu().tolist()) for item in gathered) + return hashlib.sha256(payload).hexdigest() + + +def make_route( + rank: int, + num_tokens: int, + num_experts: int, + top_k: int, + seed: int, + device: torch.device, +) -> torch.Tensor: + """Create a balanced, backend-independent route with unique experts/token.""" + + stride = 31 + if math.gcd(stride, num_experts) != 1: + raise ValueError("route stride must be coprime with num_experts") + global_token = rank * num_tokens + torch.arange( + num_tokens, dtype=torch.int64, device=device + ) + slots = torch.arange(top_k, dtype=torch.int64, device=device) + route = (global_token[:, None] * 17 + slots[None, :] * stride + seed) % num_experts + if top_k > num_experts: + raise ValueError("top_k cannot exceed num_experts") + return route.contiguous() + + +def make_input( + rank: int, num_tokens: int, hidden: int, device: torch.device +) -> torch.Tensor: + """Create deterministic, bounded BF16 data without backend RNG state.""" + + row = rank * num_tokens + torch.arange(num_tokens, dtype=torch.int64, device=device) + column = torch.arange(hidden, dtype=torch.int64, device=device) + values = (row[:, None] * 17 + column[None, :] * 13 + 19) % 257 + return ((values - 128).to(torch.float32) / 128.0).to(torch.bfloat16) + + +def normalized_diff(actual: torch.Tensor, expected: torch.Tensor) -> float: + actual64 = actual.double() + 1 + expected64 = expected.double() + 1 + denominator = (actual64.square() + expected64.square()).sum() + similarity = 2 * (actual64 * expected64).sum() / denominator + return float((1 - similarity).item()) + + +def logical_payload_bytes_per_rank( + route: torch.Tensor, + rank: int, + world_size: int, + local_world_size: int, + hidden: int, + dispatch_dtype: str, + num_experts: int, +) -> tuple[int, int, int]: + """Return all, scale-out, and valid route selections' logical payload bytes. + + Each valid expert assignment contributes its useful dispatch tensor, FP8 + scales when applicable, and one BF16 combine tensor. Backend metadata is + excluded. Scale-out bytes count assignments owned by a different node. + """ + + if num_experts % world_size: + raise ValueError("num_experts must divide world_size") + dispatch_bytes = hidden * 2 + if dispatch_dtype == "fp8": + dispatch_bytes = hidden + math.ceil(hidden / 128) * 4 + elif dispatch_dtype != "bf16": + raise ValueError(f"unsupported dispatch dtype: {dispatch_dtype}") + bytes_per_selection = dispatch_bytes + hidden * 2 + valid = route >= 0 + valid_selections = int(valid.sum().item()) + expert_owner = route // (num_experts // world_size) + source_node = rank // local_world_size + destination_node = expert_owner // local_world_size + remote_selections = int(((destination_node != source_node) & valid).sum().item()) + return ( + valid_selections * bytes_per_selection, + remote_selections * bytes_per_selection, + valid_selections, + ) + + +@dataclass(frozen=True) +class WorkloadProfile: + name: str + tokens_per_rank: int + api_mode: str + primary_metric: str + timing_boundary: str + + +WORKLOAD_PROFILES = { + "decode": WorkloadProfile( + name="decode", + tokens_per_rank=128, + api_mode="low-latency", + primary_metric="slowest-rank latency in milliseconds", + timing_boundary=( + "BF16 input ready through dispatch and combine completion; " + "slowest rank CUDA elapsed time" + ), + ), + "prefill": WorkloadProfile( + name="prefill", + tokens_per_rank=4_096, + api_mode="normal", + primary_metric="effective logical gigabytes per second per rank", + timing_boundary=( + "BF16 input and route ready through required layout, dispatch, and " + "combine completion; slowest rank CUDA elapsed time" + ), + ), +} + + +@dataclass +class DispatchState: + recv_x: Any + recv_topk_idx: torch.Tensor | None + recv_topk_weights: torch.Tensor | None + handle: Any + + +class BackendAdapter: + def __init__( + self, + arm: str, + group: dist.ProcessGroup, + profile: WorkloadProfile, + num_tokens: int, + hidden: int, + num_experts: int, + top_k: int, + ) -> None: + self.arm = arm + self.group = group + self.profile = profile + self.num_tokens = num_tokens + self.hidden = hidden + self.num_experts = num_experts + self.world_size = dist.get_world_size(group) + self.buffer: Any + self._cast_back: Any = None + self._cast_to_fp8: Any = None + + if arm == "uccl": + sys.path.insert(0, "/opt/uccl/ep/bench") + from buffer import Buffer # type: ignore[import-not-found] + from utils import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) + + self._cast_back = per_token_cast_back + self._cast_to_fp8 = per_token_cast_to_fp8 + if profile.api_mode == "low-latency": + rdma_bytes = Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + ) + else: + self.buffer = Buffer( + group, + num_nvl_bytes=NORMAL_NUM_NVL_BYTES, + num_rdma_bytes=NORMAL_NUM_RDMA_BYTES, + low_latency_mode=False, + num_qps_per_rank=NORMAL_NUM_QPS_PER_RANK, + explicitly_destroy=True, + ) + elif arm == "deepep-v1-nvshmem": + sys.path.insert(0, "/opt/amazon/deepep/tests") + sys.path.insert(0, "/opt/amazon/deepep") + import deep_ep # type: ignore[import-not-found] + from utils import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) + + self._cast_back = per_token_cast_back + self._cast_to_fp8 = per_token_cast_to_fp8 + if profile.api_mode == "low-latency": + rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = deep_ep.Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + allow_mnnvl=False, + ) + else: + self.buffer = deep_ep.Buffer( + group, + num_nvl_bytes=NORMAL_NUM_NVL_BYTES, + num_rdma_bytes=NORMAL_NUM_RDMA_BYTES, + low_latency_mode=False, + num_qps_per_rank=NORMAL_NUM_QPS_PER_RANK, + explicitly_destroy=True, + allow_mnnvl=False, + ) + elif arm == "deepep-v2-gin-gda": + import deep_ep # type: ignore[import-not-found] + from deep_ep.utils.math import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) + + self._cast_back = per_token_cast_back + self._cast_to_fp8 = per_token_cast_to_fp8 + self.buffer = deep_ep.ElasticBuffer( + group, + num_max_tokens_per_rank=num_tokens, + hidden=hidden, + num_topk=top_k, + deterministic=False, + allow_hybrid_mode=True, + allow_multiple_reduction=True, + prefer_overlap_with_compute=False, + sl_idx=0, + num_allocated_qps=0, + explicitly_destroy=True, + num_gpu_timeout_secs=180, + num_cpu_timeout_secs=180, + ) + else: + raise ValueError(f"unsupported arm: {arm}") + + @property + def is_elastic(self) -> bool: + return self.arm == "deepep-v2-gin-gda" + + def prepare_dispatch_input(self, x: torch.Tensor, dispatch_dtype: str) -> Any: + if dispatch_dtype == "bf16": + return x + if dispatch_dtype != "fp8": + raise ValueError(f"unsupported dispatch dtype: {dispatch_dtype}") + if self.is_elastic or self.profile.api_mode == "normal": + return self._cast_to_fp8(x) + return x + + def dispatch( + self, + prepared_x: Any, + topk_idx: torch.Tensor, + topk_weights: torch.Tensor, + dispatch_dtype: str, + ) -> DispatchState: + if self.is_elastic: + recv_x, recv_idx, recv_weights, handle, event = self.buffer.dispatch( + x=prepared_x, + topk_idx=topk_idx, + topk_weights=topk_weights, + num_experts=self.num_experts, + num_max_tokens_per_rank=self.num_tokens, + expert_alignment=1, + async_with_compute_stream=True, + allocate_on_comm_stream=False, + do_handle_copy=True, + do_cpu_sync=True, + ) + event.current_stream_wait() + return DispatchState(recv_x, recv_idx, recv_weights, handle) + + if self.profile.api_mode == "normal": + ( + num_tokens_per_rank, + num_tokens_per_rdma_rank, + num_tokens_per_expert, + is_token_in_rank, + _, + ) = self.buffer.get_dispatch_layout(topk_idx, self.num_experts) + ( + recv_x, + recv_idx, + recv_weights, + _, + handle, + event, + ) = self.buffer.dispatch( + x=prepared_x, + num_tokens_per_rank=num_tokens_per_rank, + num_tokens_per_rdma_rank=num_tokens_per_rdma_rank, + is_token_in_rank=is_token_in_rank, + num_tokens_per_expert=num_tokens_per_expert, + topk_idx=topk_idx, + topk_weights=topk_weights, + expert_alignment=1, + async_finish=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return DispatchState(recv_x, recv_idx, recv_weights, handle) + + recv_x, _, handle, event, _ = self.buffer.low_latency_dispatch( + prepared_x, + topk_idx, + self.num_tokens, + self.num_experts, + use_fp8=dispatch_dtype == "fp8", + async_finish=True, + return_recv_hook=False, + ) + event.current_stream_wait() + return DispatchState(recv_x, None, None, handle) + + def received_as_bf16(self, recv_x: Any, dispatch_dtype: str) -> torch.Tensor: + if dispatch_dtype == "bf16": + return recv_x + if self.is_elastic: + return self._cast_back(recv_x[0], recv_x[1]) + fp8, scales = recv_x + return self._cast_back( + fp8.reshape(-1, self.hidden), + scales.reshape(-1, self.hidden // 128), + ).reshape(fp8.shape) + + def identity_expert_output( + self, state: DispatchState, dispatch_dtype: str + ) -> torch.Tensor: + output = self.received_as_bf16(state.recv_x, dispatch_dtype) + if state.recv_topk_idx is None or state.recv_topk_weights is None: + return output + local_weights = state.recv_topk_weights.masked_fill( + state.recv_topk_idx < 0, 0 + ).sum(dim=1, keepdim=True) + return output * local_weights.to(output.dtype) + + def combine( + self, + combine_input: torch.Tensor, + state: DispatchState, + topk_idx: torch.Tensor, + topk_weights: torch.Tensor, + ) -> torch.Tensor: + if self.is_elastic: + combined, _, event = self.buffer.combine( + x=combine_input, + handle=state.handle, + topk_weights=state.recv_topk_weights, + async_with_compute_stream=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return combined + + if self.profile.api_mode == "normal": + combined, _, event = self.buffer.combine( + x=combine_input, + handle=state.handle, + topk_weights=state.recv_topk_weights, + async_finish=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return combined + + combined, event, _ = self.buffer.low_latency_combine( + combine_input, + topk_idx, + topk_weights, + state.handle, + use_logfmt=False, + async_finish=True, + return_recv_hook=False, + ) + event.current_stream_wait() + return combined + + def destroy(self) -> None: + self.buffer.destroy() + + +def initialize_distributed() -> tuple[int, int, int, torch.device, dist.ProcessGroup]: + local_rank = int(os.environ["LOCAL_RANK"]) + local_world_size = int(os.environ.get("LOCAL_WORLD_SIZE", "1")) + torch.cuda.set_device(local_rank) + device = torch.device(f"cuda:{local_rank}") + dist.init_process_group("nccl", device_id=device) + world_size = dist.get_world_size() + group = dist.new_group(list(range(world_size))) + torch.set_default_dtype(torch.bfloat16) + return dist.get_rank(), world_size, local_world_size, device, group + + +def run_dtype( + adapter: BackendAdapter, + x: torch.Tensor, + route: torch.Tensor, + topk_weights: torch.Tensor, + dispatch_dtype: str, + warmups: int, + iterations: int, + rank: int, + world_size: int, + local_world_size: int, + args: argparse.Namespace, + route_hash: str, + input_hash: str, +) -> dict[str, Any]: + prepared = adapter.prepare_dispatch_input(x, dispatch_dtype) + correctness_state = adapter.dispatch(prepared, route, topk_weights, dispatch_dtype) + # Normal and ElasticBuffer dispatch send one token per destination rank. + # Apply the local identity experts' gated reduction before combine. The + # low-latency API performs its corresponding reduction internally. + correctness_input = adapter.identity_expert_output( + correctness_state, dispatch_dtype + ) + correctness_output = adapter.combine( + correctness_input, correctness_state, route, topk_weights + ) + torch.cuda.synchronize() + expected = x * topk_weights.sum(dim=1, keepdim=True).to(torch.bfloat16) + diff = normalized_diff(correctness_output, expected) + max_abs_error = float( + (correctness_output.float() - expected.float()).abs().max().item() + ) + tolerance = 9e-4 if dispatch_dtype == "fp8" else 1e-5 + correctness_pass = diff <= tolerance and bool( + torch.isfinite(correctness_output).all().item() + ) + correctness_tensor = torch.tensor( + [1 if correctness_pass else 0], dtype=torch.int32, device=x.device + ) + dist.all_reduce(correctness_tensor, op=dist.ReduceOp.MIN) + if int(correctness_tensor.item()) != 1: + raise RuntimeError( + f"correctness failed for {dispatch_dtype}: diff={diff}, " + f"tolerance={tolerance}, max_abs_error={max_abs_error}" + ) + + # Timed combine uses a stable, preallocated expert-output tensor. This + # keeps expert computation and FP8 dequantization outside the communication + # boundary while retaining the handle created by each timed dispatch. + combine_input = torch.zeros_like(correctness_input, dtype=torch.bfloat16) + + def iteration() -> None: + current_x = adapter.prepare_dispatch_input(x, dispatch_dtype) + state = adapter.dispatch(current_x, route, topk_weights, dispatch_dtype) + adapter.combine(combine_input, state, route, topk_weights) + + for _ in range(warmups): + dist.barrier(group=adapter.group) + iteration() + torch.cuda.synchronize() + + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + max_rank_latency_ms: list[float] = [] + for _ in range(iterations): + dist.barrier(group=adapter.group) + start.record() + iteration() + end.record() + end.synchronize() + local_latency = torch.tensor( + [start.elapsed_time(end)], dtype=torch.float32, device=x.device + ) + dist.all_reduce(local_latency, op=dist.ReduceOp.MAX, group=adapter.group) + max_rank_latency_ms.append(float(local_latency.item())) + + logical_bytes, scaleout_bytes, valid_selections = logical_payload_bytes_per_rank( + route, + rank, + world_size, + local_world_size, + args.hidden, + dispatch_dtype, + args.experts, + ) + counters = torch.tensor( + [logical_bytes, scaleout_bytes, valid_selections], + dtype=torch.int64, + device=x.device, + ) + dist.all_reduce(counters, op=dist.ReduceOp.SUM, group=adapter.group) + avg_logical_bytes = int(counters[0].item()) / world_size + avg_scaleout_bytes = int(counters[1].item()) / world_size + global_valid_selections = int(counters[2].item()) + + median_ms = statistics.median(max_rank_latency_ms) + mean_ms = statistics.fmean(max_rank_latency_ms) + stdev_ms = statistics.stdev(max_rank_latency_ms) if iterations > 1 else 0.0 + result = { + "schema_version_dimensionless": SCHEMA_VERSION, + "benchmark": "common-boundary-dispatch-combine", + "workload_profile": args.profile, + "backend_api_mode": ( + "elastic" if adapter.is_elastic else adapter.profile.api_mode + ), + "primary_metric": adapter.profile.primary_metric, + "layout_in_timed_region": args.profile == "prefill", + "arm": args.arm, + "run_index_dimensionless": args.run_index, + "dispatch_dtype": dispatch_dtype, + "world_size_ranks": world_size, + "nodes": world_size // local_world_size, + "gpus_per_node": local_world_size, + "tokens_per_rank": args.tokens, + "global_input_tokens": args.tokens * world_size, + "hidden_dimensions": args.hidden, + "experts": args.experts, + "top_k_dimensionless": args.top_k, + "route_seed_dimensionless": args.seed, + "route_hash_sha256": route_hash, + "input_hash_sha256": input_hash, + "global_valid_expert_selections": global_valid_selections, + "warmup_iterations": warmups, + "measured_iterations": iterations, + "timing_boundary": adapter.profile.timing_boundary, + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "avg_logical_payload_bytes_per_rank": avg_logical_bytes, + "avg_scaleout_logical_payload_bytes_per_rank": avg_scaleout_bytes, + "latency_ms": { + "median": median_ms, + "mean": mean_ms, + "p95": percentile(max_rank_latency_ms, 0.95), + "minimum": min(max_rank_latency_ms), + "maximum": max(max_rank_latency_ms), + "stdev": stdev_ms, + "cv_percent": stdev_ms / mean_ms * 100 if mean_ms else 0.0, + }, + "aggregate_input_tokens_per_second": ( + args.tokens * world_size / (median_ms / 1e3) + ), + "effective_logical_gigabytes_per_second_per_rank": ( + avg_logical_bytes / (median_ms / 1e3) / 1e9 + ), + "effective_scaleout_logical_gigabytes_per_second_per_rank": ( + avg_scaleout_bytes / (median_ms / 1e3) / 1e9 + ), + "correctness": { + "status": "PASS", + "normalized_diff_dimensionless": diff, + "tolerance_dimensionless": tolerance, + "max_abs_error_bf16_value": max_abs_error, + }, + "runtime": { + "image_reference": os.environ.get("ADAI_IMAGE_REFERENCE", "unknown"), + "torch_version": torch.__version__, + "cuda_version": torch.version.cuda, + "nccl_version": list(torch.cuda.nccl.version()), + "gpu": torch.cuda.get_device_name(), + }, + } + return result + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument( + "--arm", + required=True, + choices=("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda"), + ) + parser.add_argument("--profile", required=True, choices=tuple(WORKLOAD_PROFILES)) + parser.add_argument("--hidden", type=int, default=7168) + parser.add_argument("--top-k", type=int, default=8) + parser.add_argument("--experts", type=int, default=256) + parser.add_argument("--seed", type=int, default=20260824) + parser.add_argument("--warmups", type=int, default=20) + parser.add_argument("--iterations", type=int, default=100) + parser.add_argument("--run-index", type=int, required=True) + parser.add_argument( + "--dispatch-dtypes", + default="fp8,bf16", + help="Comma-separated dtype order; allowed values are fp8 and bf16", + ) + args = parser.parse_args() + args.tokens = WORKLOAD_PROFILES[args.profile].tokens_per_rank + args.dispatch_dtypes = args.dispatch_dtypes.split(",") + if sorted(args.dispatch_dtypes) != ["bf16", "fp8"]: + parser.error("--dispatch-dtypes must contain fp8 and bf16 exactly once") + if args.warmups < 1 or args.iterations < 2: + parser.error("at least 1 warmup and 2 measured iterations are required") + return args + + +def main() -> None: + args = parse_args() + preload_backend(args.arm) + rank, world_size, local_world_size, device, group = initialize_distributed() + if args.experts % world_size: + raise SystemExit("experts must divide the distributed world size") + route = make_route(rank, args.tokens, args.experts, args.top_k, args.seed, device) + x = make_input(rank, args.tokens, args.hidden, device) + topk_weights = torch.full( + (args.tokens, args.top_k), + 1.0 / args.top_k, + dtype=torch.float32, + device=device, + ) + route_hash = global_digest(tensor_sha256(route), device) + input_hash = global_digest(tensor_sha256(x), device) + route_histogram = torch.bincount(route.flatten(), minlength=args.experts).to( + torch.int64 + ) + dist.all_reduce(route_histogram, op=dist.ReduceOp.SUM, group=group) + + if rank == 0: + print( + "ADAI_EP_CONFIG " + + json.dumps( + { + "arm": args.arm, + "workload_profile": args.profile, + "world_size_ranks": world_size, + "tokens_per_rank": args.tokens, + "hidden_dimensions": args.hidden, + "experts": args.experts, + "top_k_dimensionless": args.top_k, + "route_hash_sha256": route_hash, + "input_hash_sha256": input_hash, + "route_histogram_min_selections": int(route_histogram.min().item()), + "route_histogram_max_selections": int(route_histogram.max().item()), + }, + sort_keys=True, + ), + flush=True, + ) + + adapter = BackendAdapter( + args.arm, + group, + WORKLOAD_PROFILES[args.profile], + args.tokens, + args.hidden, + args.experts, + args.top_k, + ) + completed = False + try: + for dispatch_dtype in args.dispatch_dtypes: + result = run_dtype( + adapter, + x, + route, + topk_weights, + dispatch_dtype, + args.warmups, + args.iterations, + rank, + world_size, + local_world_size, + args, + route_hash, + input_hash, + ) + if rank == 0: + print(RESULT_PREFIX + json.dumps(result, sort_keys=True), flush=True) + completed = True + except BaseException: + if args.arm == "uccl" and args.profile == "prefill": + # The pinned UCCL high-throughput proxy teardown can strand the + # interpreter after an error. Preserve the traceback and let + # torchrun observe an unambiguous worker failure instead. + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(1) + raise + finally: + if args.arm == "uccl" and args.profile == "prefill": + if completed: + # UCCL's pinned normal-mode proxy cleanup destroys the CUDA + # context while PyTorch still owns CUDA tensors, which hangs + # worker shutdown. All communication has completed here and + # this barrier keeps every rank alive through result output. + # Process exit then releases the CUDA context, QPs, and file + # descriptors outside the measured benchmark boundary. + if rank == 0: + print( + "ADAI_EP_PROCESS_LIFETIME_CLEANUP arm=uccl profile=prefill", + flush=True, + ) + dist.barrier(group=group) + sys.stdout.flush() + sys.stderr.flush() + os._exit(0) + else: + adapter.destroy() + dist.barrier(group=group) + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py new file mode 100644 index 000000000..15fc55440 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Extract canonical JSONL result records from a mixed benchmark log.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from result_io import PREFIX, load_result_log + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("input_log", type=Path) + parser.add_argument("output_jsonl", type=Path) + args = parser.parse_args() + + results = load_result_log(args.input_log) + if not results: + raise SystemExit(f"no benchmark results found in {args.input_log}") + args.output_jsonl.write_text( + "".join( + PREFIX + json.dumps(result, sort_keys=True) + "\n" for result in results + ) + ) + print(f"extracted {len(results)} benchmark results from {args.input_log}") + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml b/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml deleted file mode 100644 index 7f14fd384..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml +++ /dev/null @@ -1,106 +0,0 @@ -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -# -# Baseline for the EP-backend comparison: a raw NCCL all-to-all over the same -# world size as the EP benchmarks (NUM_NODES x GPU_PER_NODE ranks). This is the -# transport-level reference ceiling -- it moves bytes but does NOT do MoE token -# routing or combine-reduction, so its busbw is an upper bound, not a -# like-for-like dispatch/combine number. -# -# Built from micro-benchmarks/nccl-tests (CUDA 13.0.2 image ships alltoall_perf -# and sm_100/sm_103). Set NP = NUM_NODES * GPU_PER_NODE in env_vars. -apiVersion: kubeflow.org/v2beta1 -kind: MPIJob -metadata: - name: nccl-alltoall -spec: - runPolicy: - cleanPodPolicy: Running - backoffLimit: 3 - slotsPerWorker: ${GPU_PER_NODE} - mpiReplicaSpecs: - Launcher: - replicas: 1 - template: - spec: - restartPolicy: OnFailure - nodeSelector: - node.kubernetes.io/instance-type: ${INSTANCE_TYPE} - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - - key: workload - value: bench - operator: Equal - effect: NoSchedule - - key: capacity-reservation - operator: Exists - effect: NoSchedule - containers: - - image: ${IMAGE_URI} - imagePullPolicy: IfNotPresent - name: nccl-alltoall-launcher - # PATH / LD_LIBRARY_PATH come from the image ENV and are forwarded to - # workers via `mpirun -x PATH -x LD_LIBRARY_PATH` below (Kubernetes does - # not expand a bare $PATH in an env value, so we do not set one here). - command: - - /bin/bash - - -c - - | - # Wait for each worker's sshd before launching: mpi-operator starts - # the launcher as soon as workers are Running, but their sshd / DNS - # may not be ready yet, and the launcher would lose the SSH race. - for host in $(awk '{print $1}' /etc/mpi/hostfile); do - until ssh -o ConnectTimeout=3 -o BatchMode=yes -o StrictHostKeyChecking=no "$host" true 2>/dev/null; do - echo "waiting for sshd on $host"; sleep 3 - done - done - /opt/amazon/openmpi/bin/mpirun \ - --allow-run-as-root --tag-output \ - -np ${NP} -N ${GPU_PER_NODE} --bind-to none \ - -x PATH -x LD_LIBRARY_PATH \ - -x FI_PROVIDER=efa \ - -x FI_EFA_USE_DEVICE_RDMA=1 \ - -x FI_EFA_FORK_SAFE=1 \ - -x NCCL_DEBUG=INFO \ - /opt/nccl-tests/build/alltoall_perf -b 8 -e 8G -f 2 -g 1 -c 1 -n 100 - Worker: - replicas: ${NUM_NODES} - template: - spec: - nodeSelector: - node.kubernetes.io/instance-type: ${INSTANCE_TYPE} - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - - key: workload - value: bench - operator: Equal - effect: NoSchedule - - key: capacity-reservation - operator: Exists - effect: NoSchedule - containers: - - image: ${IMAGE_URI} - imagePullPolicy: IfNotPresent - name: nccl-alltoall-worker - volumeMounts: - - name: shmem - mountPath: /dev/shm - resources: - limits: - nvidia.com/gpu: ${GPU_PER_NODE} - hugepages-2Mi: 5120Mi - vpc.amazonaws.com/efa: ${EFA_PER_NODE} - memory: 32000Mi - requests: - nvidia.com/gpu: ${GPU_PER_NODE} - hugepages-2Mi: 5120Mi - vpc.amazonaws.com/efa: ${EFA_PER_NODE} - memory: 32000Mi - volumes: - - name: shmem - hostPath: - path: /dev/shm diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py new file mode 100644 index 000000000..1ff34b352 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""Render backend box plots from an EP comparison summary.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path +from typing import Any + +import matplotlib + +matplotlib.use("Agg") + +from matplotlib import pyplot as plt +from matplotlib.patches import Patch + + +ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") +ARM_STYLES = { + "uccl": { + "color": "#0072B2", + "hatch": "///", + "label": "UCCL", + "marker": "o", + }, + "deepep-v1-nvshmem": { + "color": "#E69F00", + "hatch": "\\\\", + "label": "DeepEP V1 NVSHMEM", + "marker": "s", + }, + "deepep-v2-gin-gda": { + "color": "#009E73", + "hatch": "...", + "label": "DeepEP V2 NCCL GIN", + "marker": "^", + }, +} +CELL_ORDER = ((16, "fp8"), (16, "bf16"), (32, "fp8"), (32, "bf16")) +PROFILE_CONFIG = { + "decode": { + "title": "Decode-like: slowest-rank latency (lower is better)", + "ylabel": "Latency (ms)", + }, + "prefill": { + "title": "Prefill-like: common logical throughput (higher is better)", + "ylabel": "Logical throughput (GB/s/rank)", + }, +} + + +def load_plot_data( + path: Path, +) -> tuple[int, dict[tuple[str, int, str], dict[str, Any]]]: + summary = json.loads(path.read_text()) + if summary.get("status") != "PASS": + raise ValueError("the input summary does not have PASS status") + + starts = summary.get("independent_starts_per_cell") + if not isinstance(starts, int) or starts < 1: + raise ValueError("independent_starts_per_cell must be a positive integer") + + cells: dict[tuple[str, int, str], dict[str, Any]] = {} + for cell in summary.get("cells", []): + key = ( + cell.get("workload_profile"), + cell.get("world_size_ranks"), + cell.get("dispatch_dtype"), + ) + if key in cells: + raise ValueError(f"duplicate workload cell: {key}") + cells[key] = cell + + expected_keys = { + (profile, world_size, dtype) + for profile in PROFILE_CONFIG + for world_size, dtype in CELL_ORDER + } + if cells.keys() != expected_keys: + missing = sorted(expected_keys - cells.keys()) + extra = sorted(cells.keys() - expected_keys) + raise ValueError(f"workload cell mismatch; missing={missing}, extra={extra}") + + for key, cell in cells.items(): + arms = cell.get("arms", {}) + if arms.keys() != set(ARMS): + missing = sorted(set(ARMS) - arms.keys()) + extra = sorted(arms.keys() - set(ARMS)) + raise ValueError( + f"backend arm mismatch for {key}; missing={missing}, extra={extra}" + ) + for arm in ARMS: + values = arms[arm].get("per_start_primary_values") + if not isinstance(values, list) or len(values) != starts: + raise ValueError( + f"{key}/{arm} must contain {starts} per-start primary values" + ) + if not all( + isinstance(value, (int, float)) and math.isfinite(value) and value > 0 + for value in values + ): + raise ValueError(f"{key}/{arm} contains an invalid primary value") + return starts, cells + + +def render_box_plots(summary_path: Path, output_path: Path) -> None: + starts, cells = load_plot_data(summary_path) + matplotlib.rcParams.update( + { + "axes.edgecolor": "#333333", + "axes.labelcolor": "#222222", + "font.size": 10, + "savefig.facecolor": "white", + "text.color": "#222222", + "xtick.color": "#333333", + "ytick.color": "#333333", + } + ) + + figure, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True) + group_positions = list(range(1, len(CELL_ORDER) + 1)) + arm_offsets = (-0.24, 0.0, 0.24) + point_offsets = tuple((index - (starts - 1) / 2) * 0.025 for index in range(starts)) + + for axis, profile in zip(axes, PROFILE_CONFIG, strict=True): + all_values: list[float] = [] + for arm, arm_offset in zip(ARMS, arm_offsets, strict=True): + values_by_cell = [ + cells[(profile, world_size, dtype)]["arms"][arm][ + "per_start_primary_values" + ] + for world_size, dtype in CELL_ORDER + ] + all_values.extend(value for values in values_by_cell for value in values) + positions = [position + arm_offset for position in group_positions] + style = ARM_STYLES[arm] + box_plot = axis.boxplot( + values_by_cell, + positions=positions, + widths=0.19, + whis=(0, 100), + showfliers=False, + patch_artist=True, + manage_ticks=False, + boxprops={ + "facecolor": style["color"], + "edgecolor": "#222222", + "hatch": style["hatch"], + "linewidth": 1.0, + "alpha": 0.55, + }, + whiskerprops={"color": "#333333", "linewidth": 1.0}, + capprops={"color": "#333333", "linewidth": 1.0}, + medianprops={"color": "#111111", "linewidth": 1.8}, + ) + for median in box_plot["medians"]: + median.set_zorder(4) + for position, values in zip(positions, values_by_cell, strict=True): + axis.scatter( + [position + offset for offset in point_offsets], + values, + color=style["color"], + edgecolor="#111111", + linewidth=0.6, + marker=style["marker"], + s=29, + zorder=5, + ) + + config = PROFILE_CONFIG[profile] + axis.set_title(config["title"], loc="left", fontweight="bold", pad=10) + axis.set_ylabel(config["ylabel"]) + axis.set_ylim(0, max(all_values) * 1.12) + axis.set_xlim(0.5, len(CELL_ORDER) + 0.5) + axis.grid(axis="y", color="#D9D9D9", linewidth=0.8) + axis.set_axisbelow(True) + axis.spines["top"].set_visible(False) + axis.spines["right"].set_visible(False) + + axes[-1].set_xticks( + group_positions, + [f"{world_size} ranks\n{dtype.upper()}" for world_size, dtype in CELL_ORDER], + ) + axes[-1].set_xlabel("Expert-parallel size and dispatch dtype", labelpad=9) + + legend_handles = [ + Patch( + facecolor=ARM_STYLES[arm]["color"], + edgecolor="#222222", + hatch=ARM_STYLES[arm]["hatch"], + alpha=0.55, + label=ARM_STYLES[arm]["label"], + ) + for arm in ARMS + ] + figure.legend( + handles=legend_handles, + loc="upper center", + bbox_to_anchor=(0.5, 0.945), + frameon=False, + ncol=len(ARMS), + ) + figure.suptitle( + "B200 expert-parallel backend comparison", + fontsize=15, + fontweight="bold", + y=0.99, + ) + figure.text( + 0.5, + 0.012, + ( + f"Each arm has {starts} independent process starts. " + "Box: Q1 to Q3; line: median; whiskers: minimum to maximum; " + "markers: per-start medians." + ), + ha="center", + fontsize=9, + color="#444444", + ) + figure.subplots_adjust(left=0.1, right=0.98, top=0.88, bottom=0.12, hspace=0.34) + + output_path.parent.mkdir(parents=True, exist_ok=True) + figure.savefig( + output_path, + dpi=160, + metadata={"Software": "matplotlib via plot_results.py"}, + ) + plt.close(figure) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("summary", type=Path, help="machine-readable summary JSON") + parser.add_argument("--output", type=Path, required=True, help="output image path") + args = parser.parse_args() + render_box_plots(args.summary, args.output) + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py new file mode 100644 index 000000000..1d53cee33 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py @@ -0,0 +1,45 @@ +"""Parse benchmark result markers despite interleaved native stdout.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Iterator + + +PREFIX = "ADAI_EP_RESULT " + + +def iter_result_objects(text: str, source: str = "") -> Iterator[dict[str, Any]]: + """Yield JSON objects following result markers in mixed process output. + + Native libraries can flush a diagnostic after Python has emitted a JSON + object but before its newline reaches the combined log. JSONDecoder's + raw_decode identifies the exact end of the object without treating that + trailing diagnostic as part of the result. + """ + + decoder = json.JSONDecoder() + for line_number, line in enumerate(text.splitlines(), 1): + search_from = 0 + while True: + marker = line.find(PREFIX, search_from) + if marker < 0: + break + payload_start = marker + len(PREFIX) + try: + result, consumed = decoder.raw_decode(line[payload_start:]) + except json.JSONDecodeError as error: + raise ValueError( + f"invalid benchmark result in {source}:{line_number}: {error}" + ) from error + if not isinstance(result, dict): + raise ValueError( + f"benchmark result in {source}:{line_number} is not a JSON object" + ) + yield result + search_from = payload_start + consumed + + +def load_result_log(path: Path) -> list[dict[str, Any]]: + return list(iter_result_objects(path.read_text(errors="replace"), str(path))) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png new file mode 100644 index 000000000..91fd47ad9 Binary files /dev/null and b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png differ diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25.json b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25.json new file mode 100644 index 000000000..76a06e7fe --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25.json @@ -0,0 +1,1114 @@ +{ + "bootstrap_samples_dimensionless": 20000, + "campaign_provenance": { + "campaign_id": "ep-b200-profiles-20260825t023933z", + "cluster": "ml-clusters-shared-ap-south-1", + "comparison": { + "experts": 256, + "hidden_dimensions": 7168, + "independent_starts": 3, + "measured_iterations": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "effective logical gigabytes per second per rank", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations": 20 + }, + "created_at_utc": "2026-08-25T02:41:59Z", + "git_commit": "97c93ee33535ffb41b7635befc4a97617ab8acef", + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "region": "ap-south-1" + }, + "cells": [ + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.0147680044174194, + 1.0428639650344849 + ], + "median_aggregate_input_tokens_per_second": 1984619.2206983268, + "median_effective_logical_gigabytes_per_second_per_rank": 21.560903213666624, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.780451606833312, + "median_latency_ms": 1.03193598985672, + "median_primary_value": 1.03193598985672, + "per_start_median_latency_ms": [ + 1.03193598985672, + 1.0428639650344849, + 1.0147680044174194 + ], + "per_start_primary_values": [ + 1.03193598985672, + 1.0428639650344849, + 1.0147680044174194 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.3752408024975253, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.6111679971218109, + 0.6488159894943237 + ], + "median_aggregate_input_tokens_per_second": 3329691.4185950146, + "median_effective_logical_gigabytes_per_second_per_rank": 36.17376757161624, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 18.08688378580812, + "median_latency_ms": 0.6150720119476318, + "median_primary_value": 0.6150720119476318, + "per_start_median_latency_ms": [ + 0.6150720119476318, + 0.6111679971218109, + 0.6488159894943237 + ], + "per_start_primary_values": [ + 0.6150720119476318, + 0.6111679971218109, + 0.6488159894943237 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 3.3121127855081234, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5123519897460938, + 0.5320799946784973 + ], + "median_aggregate_input_tokens_per_second": 3926862.247229175, + "median_effective_logical_gigabytes_per_second_per_rank": 42.661431453897755, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.330715726948878, + "median_latency_ms": 0.5215359926223755, + "median_primary_value": 0.5215359926223755, + "per_start_median_latency_ms": [ + 0.5123519897460938, + 0.5215359926223755, + 0.5320799946784973 + ], + "per_start_primary_values": [ + 0.5123519897460938, + 0.5215359926223755, + 0.5320799946784973 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.8911904515714288, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 36.06262843625914, + 41.39523297253817 + ], + "direction_supported": true, + "median_paired_improvement_percent": 40.39630190308296, + "paired_improvement_percent_per_start": [ + 40.39630190308296, + 41.39523297253817, + 36.06262843625914 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -21.939557206311182, + -17.186158916616627 + ], + "direction_supported": true, + "median_paired_improvement_percent": -20.04872124190306, + "paired_improvement_percent_per_start": [ + -20.04872124190306, + -17.186158916616627, + -21.939557206311182 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.0214080214500427, + 1.052016019821167 + ], + "median_aggregate_input_tokens_per_second": 1962016.6651629754, + "median_effective_logical_gigabytes_per_second_per_rank": 28.127470911776417, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.063735455888208, + "median_latency_ms": 1.0438239574432373, + "median_primary_value": 1.0438239574432373, + "per_start_median_latency_ms": [ + 1.0438239574432373, + 1.052016019821167, + 1.0214080214500427 + ], + "per_start_primary_values": [ + 1.0438239574432373, + 1.052016019821167, + 1.0214080214500427 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.5249277826940286, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.48580799996852875, + 0.5008960068225861 + ], + "median_aggregate_input_tokens_per_second": 4161789.511411154, + "median_effective_logical_gigabytes_per_second_per_rank": 59.663414435590305, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 29.831707217795152, + "median_latency_ms": 0.4920960068702698, + "median_primary_value": 0.4920960068702698, + "per_start_median_latency_ms": [ + 0.48580799996852875, + 0.5008960068225861, + 0.4920960068702698 + ], + "per_start_primary_values": [ + 0.48580799996852875, + 0.5008960068225861, + 0.4920960068702698 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.5374847658084227, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5859040021896362, + 0.5971360206604004 + ], + "median_aggregate_input_tokens_per_second": 3474201.342724331, + "median_effective_logical_gigabytes_per_second_per_rank": 49.80615044929601, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 24.903075224648006, + "median_latency_ms": 0.5894879996776581, + "median_primary_value": 0.5894879996776581, + "per_start_median_latency_ms": [ + 0.5859040021896362, + 0.5971360206604004, + 0.5894879996776581 + ], + "per_start_primary_values": [ + 0.5859040021896362, + 0.5971360206604004, + 0.5894879996776581 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.9710265933603435, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 51.8217992676751, + 53.45881874962169 + ], + "direction_supported": true, + "median_paired_improvement_percent": 52.38703618717386, + "paired_improvement_percent_per_start": [ + 53.45881874962169, + 52.38703618717386, + 51.8217992676751 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 16.116933246026264, + 17.08402773270526 + ], + "direction_supported": true, + "median_paired_improvement_percent": 16.521454696388034, + "paired_improvement_percent_per_start": [ + 17.08402773270526, + 16.116933246026264, + 16.521454696388034 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.543503999710083, + 1.5656799674034119 + ], + "median_aggregate_input_tokens_per_second": 2638821.590379041, + "median_effective_logical_gigabytes_per_second_per_rank": 14.334078878938953, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.750559159204213, + "median_latency_ms": 1.5522080063819885, + "median_primary_value": 1.5522080063819885, + "per_start_median_latency_ms": [ + 1.5522080063819885, + 1.5656799674034119, + 1.543503999710083 + ], + "per_start_primary_values": [ + 1.5522080063819885, + 1.5656799674034119, + 1.543503999710083 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.7190825412863633, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.9453760087490082, + 0.9715520143508911 + ], + "median_aggregate_input_tokens_per_second": 4248046.16583251, + "median_effective_logical_gigabytes_per_second_per_rank": 23.075386772802197, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 17.306540079601646, + "median_latency_ms": 0.9642079770565033, + "median_primary_value": 0.9642079770565033, + "per_start_median_latency_ms": [ + 0.9642079770565033, + 0.9453760087490082, + 0.9715520143508911 + ], + "per_start_primary_values": [ + 0.9642079770565033, + 0.9453760087490082, + 0.9715520143508911 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.405863396881843, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.7613919973373413, + 0.7740640044212341 + ], + "median_aggregate_input_tokens_per_second": 5366426.29256539, + "median_effective_logical_gigabytes_per_second_per_rank": 29.150427621215194, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.862820715911397, + "median_latency_ms": 0.763264000415802, + "median_primary_value": 0.763264000415802, + "per_start_median_latency_ms": [ + 0.763264000415802, + 0.7613919973373413, + 0.7740640044212341 + ], + "per_start_primary_values": [ + 0.763264000415802, + 0.7613919973373413, + 0.7740640044212341 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.8926876291866787, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 37.05542619044861, + 39.61882195396173 + ], + "direction_supported": true, + "median_paired_improvement_percent": 37.88152276678711, + "paired_improvement_percent_per_start": [ + 37.88152276678711, + 39.61882195396173, + 37.05542619044861 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -26.326929677180296, + -24.164164064644243 + ], + "direction_supported": true, + "median_paired_improvement_percent": -25.513137001806243, + "paired_improvement_percent_per_start": [ + -26.326929677180296, + -24.164164064644243, + -25.513137001806243 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.5467199683189392, + 1.5694079995155334 + ], + "median_aggregate_input_tokens_per_second": 2633988.784611133, + "median_effective_logical_gigabytes_per_second_per_rank": 18.8804316080926, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.160323706069452, + "median_latency_ms": 1.5550559759140015, + "median_primary_value": 1.5550559759140015, + "per_start_median_latency_ms": [ + 1.5550559759140015, + 1.5694079995155334, + 1.5467199683189392 + ], + "per_start_primary_values": [ + 1.5550559759140015, + 1.5694079995155334, + 1.5467199683189392 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.7370410374246165, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.9447679817676544, + 0.9491199851036072 + ], + "median_aggregate_input_tokens_per_second": 4322061.081339236, + "median_effective_logical_gigabytes_per_second_per_rank": 30.980533831039647, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 23.235400373279735, + "median_latency_ms": 0.9476960003376007, + "median_primary_value": 0.9476960003376007, + "per_start_median_latency_ms": [ + 0.9491199851036072, + 0.9476960003376007, + 0.9447679817676544 + ], + "per_start_primary_values": [ + 0.9491199851036072, + 0.9476960003376007, + 0.9447679817676544 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.2342596155941813, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.8620480000972748, + 0.8686240017414093 + ], + "median_aggregate_input_tokens_per_second": 4733640.246388126, + "median_effective_logical_gigabytes_per_second_per_rank": 33.93073328611009, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 25.44804996458257, + "median_latency_ms": 0.8652960062026978, + "median_primary_value": 0.8652960062026978, + "per_start_median_latency_ms": [ + 0.8620480000972748, + 0.8686240017414093, + 0.8652960062026978 + ], + "per_start_primary_values": [ + 0.8620480000972748, + 0.8686240017414093, + 0.8652960062026978 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.3799833309701129, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 38.91796827356664, + 39.61442782054452 + ], + "direction_supported": true, + "median_paired_improvement_percent": 38.965542089521804, + "paired_improvement_percent_per_start": [ + 38.965542089521804, + 39.61442782054452, + 38.91796827356664 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -10.100595906087255, + -9.103133051546882 + ], + "direction_supported": true, + "median_paired_improvement_percent": -9.18436870103156, + "paired_improvement_percent_per_start": [ + -10.100595906087255, + -9.103133051546882, + -9.18436870103156 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 213.2319192391771, + 214.41244335810896 + ], + "median_aggregate_input_tokens_per_second": 19687953.230441127, + "median_effective_logical_gigabytes_per_second_per_rank": 213.8899238955124, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 106.9449619477562, + "median_latency_ms": 3.3287360668182373, + "median_primary_value": 213.8899238955124, + "per_start_median_latency_ms": [ + 3.3287360668182373, + 3.320623993873596, + 3.339008092880249 + ], + "per_start_primary_values": [ + 213.8899238955124, + 214.41244335810896, + 213.2319192391771 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.276628898113071, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 252.42471250242997, + 254.64928630242528 + ], + "median_aggregate_input_tokens_per_second": 23272859.416482452, + "median_effective_logical_gigabytes_per_second_per_rank": 252.83634470066536, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 126.41817235033268, + "median_latency_ms": 2.815984010696411, + "median_primary_value": 252.83634470066536, + "per_start_median_latency_ms": [ + 2.7959359884262085, + 2.8205760717391968, + 2.815984010696411 + ], + "per_start_primary_values": [ + 254.64928630242528, + 252.42471250242997, + 252.83634470066536 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.4672510049725491, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 128.59999990696588, + 129.14415760507953 + ], + "median_aggregate_input_tokens_per_second": 11853360.76013532, + "median_effective_logical_gigabytes_per_second_per_rank": 128.7749112981101, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 64.38745564905506, + "median_latency_ms": 5.52889609336853, + "median_primary_value": 128.7749112981101, + "per_start_median_latency_ms": [ + 5.536416053771973, + 5.52889609336853, + 5.51308798789978 + ], + "per_start_primary_values": [ + 128.59999990696588, + 128.7749112981101, + 129.14415760507953 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.21561850838250798, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 17.72857421377984, + 19.056233068194594 + ], + "direction_supported": true, + "median_paired_improvement_percent": 18.573403833159205, + "paired_improvement_percent_per_start": [ + 19.056233068194594, + 17.72857421377984, + 18.573403833159205 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 95.7783839311061, + 98.01655247795354 + ], + "direction_supported": true, + "median_paired_improvement_percent": 96.02010201977482, + "paired_improvement_percent_per_start": [ + 98.01655247795354, + 96.02010201977482, + 95.7783839311061 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 262.95281518810515, + 265.06324509499836 + ], + "median_aggregate_input_tokens_per_second": 18395265.03985649, + "median_effective_logical_gigabytes_per_second_per_rank": 263.7145196113826, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 131.8572598056913, + "median_latency_ms": 3.5626559257507324, + "median_primary_value": 263.7145196113826, + "per_start_median_latency_ms": [ + 3.544528007507324, + 3.572975993156433, + 3.5626559257507324 + ], + "per_start_primary_values": [ + 265.06324509499836, + 262.95281518810515, + 263.7145196113826 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.40496166504210196, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 276.15387615995263, + 277.7540250741766 + ], + "median_aggregate_input_tokens_per_second": 19347586.023543168, + "median_effective_logical_gigabytes_per_second_per_rank": 277.36699323351485, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 138.68349661675742, + "median_latency_ms": 3.387295961380005, + "median_primary_value": 277.36699323351485, + "per_start_median_latency_ms": [ + 3.3825759887695312, + 3.387295961380005, + 3.402176022529602 + ], + "per_start_primary_values": [ + 277.7540250741766, + 277.36699323351485, + 276.15387615995263 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.30129296612441553, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 170.41603779313468, + 171.27497910590617 + ], + "median_aggregate_input_tokens_per_second": 11926565.800798107, + "median_effective_logical_gigabytes_per_second_per_rank": 170.97924732024165, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 85.48962366012083, + "median_latency_ms": 5.494959831237793, + "median_primary_value": 170.97924732024165, + "per_start_median_latency_ms": [ + 5.494959831237793, + 5.513120174407959, + 5.485471963882446 + ], + "per_start_primary_values": [ + 170.97924732024165, + 170.41603779313468, + 171.27497910590617 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.25534341659717424, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 4.716978256222297, + 5.481659527051807 + ], + "direction_supported": true, + "median_paired_improvement_percent": 4.787830909800359, + "paired_improvement_percent_per_start": [ + 4.787830909800359, + 5.481659527051807, + 4.716978256222297 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 61.23421973340055, + 62.758738452895045 + ], + "direction_supported": true, + "median_paired_improvement_percent": 62.44896935003305, + "paired_improvement_percent_per_start": [ + 62.44896935003305, + 62.758738452895045, + 61.23421973340055 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 53.399459764271676, + 54.41046183143514 + ], + "median_aggregate_input_tokens_per_second": 9953621.843209853, + "median_effective_logical_gigabytes_per_second_per_rank": 54.068073852315926, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 40.55105538923694, + "median_latency_ms": 13.168272018432617, + "median_primary_value": 54.068073852315926, + "per_start_median_latency_ms": [ + 13.333151817321777, + 13.168272018432617, + 13.085408210754395 + ], + "per_start_primary_values": [ + 53.399459764271676, + 54.068073852315926, + 54.41046183143514 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.9529367915600195, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 105.10601540037308, + 105.3551119621716 + ], + "median_aggregate_input_tokens_per_second": 19368582.567262173, + "median_effective_logical_gigabytes_per_second_per_rank": 105.21014050536813, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 78.9076053790261, + "median_latency_ms": 6.767247915267944, + "median_primary_value": 105.21014050536813, + "per_start_median_latency_ms": [ + 6.757936000823975, + 6.773952007293701, + 6.767247915267944 + ], + "per_start_primary_values": [ + 105.3551119621716, + 105.1060154003731, + 105.21014050536813 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.11889443898406217, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 85.4062152365377, + 85.49400388769168 + ], + "median_aggregate_input_tokens_per_second": 15732818.609831655, + "median_effective_logical_gigabytes_per_second_per_rank": 85.46067068860556, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 64.09550301645416, + "median_latency_ms": 8.331120014190674, + "median_primary_value": 85.46067068860556, + "per_start_median_latency_ms": [ + 8.327871799468994, + 8.331120014190674, + 8.336431980133057 + ], + "per_start_primary_values": [ + 85.49400388769168, + 85.46067068860556, + 85.4062152365377 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.05185948130512096, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 93.36380718713903, + 97.29621315881221 + ], + "direction_supported": true, + "median_paired_improvement_percent": 94.39570880121344, + "paired_improvement_percent_per_start": [ + 97.29621315881221, + 94.39570880121344, + 93.36380718713903 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 22.987585462966464, + 23.23099535795548 + ], + "direction_supported": true, + "median_paired_improvement_percent": 23.187920473916623, + "paired_improvement_percent_per_start": [ + 23.23099535795548, + 22.987585462966464, + 23.187920473916623 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "f49abec913e14c19c444e1599a536dde42c0ecf5399444a229f780ebbb8b2181", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "9b0358f379e45666ab47399443462b55fd2be657fe4a0b0de5bc0581acfd8930", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 70.48387203078285, + 71.07303480105705 + ], + "median_aggregate_input_tokens_per_second": 9898441.780961027, + "median_effective_logical_gigabytes_per_second_per_rank": 70.95203068592863, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 53.21402301444648, + "median_latency_ms": 13.241680145263672, + "median_primary_value": 70.95203068592863, + "per_start_median_latency_ms": [ + 13.241680145263672, + 13.32963228225708, + 13.219135761260986 + ], + "per_start_primary_values": [ + 70.95203068592863, + 70.48387203078285, + 71.07303480105705 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.43926775013658587, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 108.14282406101631, + 108.36234822283106 + ], + "median_aggregate_input_tokens_per_second": 15094422.788183186, + "median_effective_logical_gigabytes_per_second_per_rank": 108.19682254569707, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 81.14761690927281, + "median_latency_ms": 8.683472156524658, + "median_primary_value": 108.19682254569707, + "per_start_median_latency_ms": [ + 8.670207977294922, + 8.6878080368042, + 8.683472156524658 + ], + "per_start_primary_values": [ + 108.36234822283106, + 108.14282406101631, + 108.19682254569707 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.10568432043857662, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 95.25844126098058, + 95.3781989559693 + ], + "median_aggregate_input_tokens_per_second": 13296240.83483305, + "median_effective_logical_gigabytes_per_second_per_rank": 95.3074543040833, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 71.48059072806247, + "median_latency_ms": 9.857823848724365, + "median_primary_value": 95.3074543040833, + "per_start_median_latency_ms": [ + 9.850512027740479, + 9.857823848724365, + 9.862895965576172 + ], + "per_start_primary_values": [ + 95.3781989559693, + 95.3074543040833, + 95.25844126098058 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.06316609677985278, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 52.23329473485195, + 53.429175987645 + ], + "direction_supported": true, + "median_paired_improvement_percent": 52.726211181326654, + "paired_improvement_percent_per_start": [ + 52.726211181326654, + 53.429175987645, + 52.23329473485195 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 13.467330389479418, + 13.613330309220672 + ], + "direction_supported": true, + "median_paired_improvement_percent": 13.582398697107667, + "paired_improvement_percent_per_start": [ + 13.613330309220672, + 13.467330389479418, + 13.582398697107667 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "f49abec913e14c19c444e1599a536dde42c0ecf5399444a229f780ebbb8b2181", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "9b0358f379e45666ab47399443462b55fd2be657fe4a0b0de5bc0581acfd8930", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 32 + } + ], + "comparison_scope": "synthetic decode-like and prefill-like dispatch-plus-combine communication workloads; not end-to-end training or serving", + "configuration": { + "dispatch_dtypes": [ + "fp8", + "bf16" + ], + "experts_dimensionless": 256, + "hidden_dimensions": 7168, + "measured_iterations_dimensionless": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "effective logical gigabytes per second per rank", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": 20, + "world_sizes_ranks": [ + 16, + 32 + ] + }, + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "independent_starts_per_cell": 3, + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "maximum_run_to_run_cv_percent_for_direction_support": 5.0, + "runtime": { + "cuda_version": "13.0", + "gpu": "NVIDIA B200", + "nccl_version": [ + 2, + 29, + 7 + ], + "torch_version": "2.13.0+cu130" + }, + "schema_version_dimensionless": 2, + "scored_result_records_dimensionless": 72, + "status": "PASS", + "timing_boundaries": { + "decode": "BF16 input ready through dispatch and combine completion; slowest rank CUDA elapsed time", + "prefill": "BF16 input and route ready through required layout, dispatch, and combine completion; slowest rank CUDA elapsed time" + } +} diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh new file mode 100755 index 000000000..64c5568d9 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh @@ -0,0 +1,606 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${CAMPAIGN_ID:?Set a unique CAMPAIGN_ID}" +: "${EP_BENCHMARK_NODES:?Set 4 comma-separated B200 node names}" +: "${PROTECTED_NODES_CSV:=}" +: "${ARTIFACT_ROOT:?Set the durable artifact directory}" +: "${KUBECTL_CONTEXT:?Set the target kubectl context explicitly}" +: "${CAMPAIGN_NAMESPACE:=${CAMPAIGN_ID}}" +: "${SHARED_LOCK_NAME:=adai-ap-south-1-gpu-campaign-lock}" +: "${SHARED_LOCK_NAMESPACE:=default}" +: "${LOCK_MODE:=exclusive}" +: "${EXPECTED_LOCK_HOLDER:=}" +: "${LOCK_DURATION_SECONDS:=28800}" +: "${INDEPENDENT_STARTS:=3}" +: "${WARMUP_ITERATIONS:=20}" +: "${MEASURED_ITERATIONS:=100}" +: "${CASE_TIMEOUT_SECONDS:=1800}" +: "${EFA_PER_NODE:=8}" +: "${UCCL_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a}" +: "${DEEPEP_V1_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac}" +: "${DEEPEP_V2_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb}" + +[[ "${CAMPAIGN_ID}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] +[[ "${CAMPAIGN_NAMESPACE}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] +[[ "${LOCK_MODE}" == exclusive || "${LOCK_MODE}" == observe ]] +for value in "${LOCK_DURATION_SECONDS}" "${INDEPENDENT_STARTS}" \ + "${WARMUP_ITERATIONS}" "${MEASURED_ITERATIONS}" \ + "${CASE_TIMEOUT_SECONDS}" "${EFA_PER_NODE}"; do + [[ "${value}" =~ ^[1-9][0-9]*$ ]] || { + printf 'Iteration, timeout, Lease, and EFA values must be positive integers\n' >&2 + exit 2 + } +done +if [[ "${LOCK_MODE}" == observe && -z "${EXPECTED_LOCK_HOLDER}" ]]; then + printf 'LOCK_MODE=observe requires EXPECTED_LOCK_HOLDER\n' >&2 + exit 2 +fi +if [[ "${LOCK_MODE}" == observe && -z "${PROTECTED_NODES_CSV}" ]]; then + printf 'LOCK_MODE=observe requires PROTECTED_NODES_CSV\n' >&2 + exit 2 +fi +[[ "${INDEPENDENT_STARTS}" -eq 3 ]] || { + printf 'This scored matrix requires exactly 3 independent starts\n' >&2 + exit 2 +} +[[ "${WARMUP_ITERATIONS}" -eq 20 && "${MEASURED_ITERATIONS}" -eq 100 ]] || { + printf 'This scored matrix requires 20 warmup and 100 measured iterations\n' >&2 + exit 2 +} + +case_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [[ -e "${ARTIFACT_ROOT}" && ! -d "${ARTIFACT_ROOT}" ]]; then + printf 'ARTIFACT_ROOT exists and is not a directory: %s\n' "${ARTIFACT_ROOT}" >&2 + exit 2 +fi +if [[ -d "${ARTIFACT_ROOT}" && -n "$(find "${ARTIFACT_ROOT}" -mindepth 1 -print -quit)" ]]; then + printf 'Refusing to reuse nonempty ARTIFACT_ROOT: %s\n' "${ARTIFACT_ROOT}" >&2 + exit 2 +fi +mkdir -p "${ARTIFACT_ROOT}/control" "${ARTIFACT_ROOT}/runs" \ + "${ARTIFACT_ROOT}/summary" "${ARTIFACT_ROOT}/teardown" +K=(kubectl --context "${KUBECTL_CONTEXT}") + +IFS=, read -r -a selected_nodes <<<"${EP_BENCHMARK_NODES}" +protected_nodes=() +if [[ -n "${PROTECTED_NODES_CSV}" ]]; then + IFS=, read -r -a protected_nodes <<<"${PROTECTED_NODES_CSV}" +fi +((${#selected_nodes[@]} == 4)) || { + printf 'EP_BENCHMARK_NODES must contain exactly 4 nodes\n' >&2 + exit 2 +} +[[ "$(printf '%s\n' "${selected_nodes[@]}" | sort -u | wc -l)" -eq 4 ]] + +declare -A protected=() +for node in "${protected_nodes[@]}"; do + protected["${node}"]=1 +done +for node in "${selected_nodes[@]}"; do + [[ -z "${protected[${node}]:-}" ]] || { + printf 'Selected node is protected by the concurrent campaign: %s\n' "${node}" >&2 + exit 1 + } +done + +declare -A images=( + [uccl]="${UCCL_IMAGE}" + [deepep-v1-nvshmem]="${DEEPEP_V1_IMAGE}" + [deepep-v2-gin-gda]="${DEEPEP_V2_IMAGE}" +) +for arm in "${!images[@]}"; do + [[ "${images[${arm}]}" =~ @sha256:[0-9a-f]{64}$ ]] || { + printf 'Image for %s is not pinned by digest: %s\n' \ + "${arm}" "${images[${arm}]}" >&2 + exit 2 + } +done + +current_case="" +namespace_created=0 +lock_claimed=0 + +check_shared_lock() { + local holder + holder="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o jsonpath='{.spec.holderIdentity}' 2>/dev/null || true)" + if [[ "${LOCK_MODE}" == exclusive ]]; then + [[ "${holder}" == "${CAMPAIGN_ID}" ]] || { + printf 'Exclusive shared Lease is no longer held by %s: holder=%s\n' \ + "${CAMPAIGN_ID}" "${holder}" >&2 + return 1 + } + elif [[ "${holder}" != "${EXPECTED_LOCK_HOLDER}" ]]; then + printf 'Shared Lease holder changed from protected campaign %s to %s\n' \ + "${EXPECTED_LOCK_HOLDER}" "${holder}" >&2 + return 1 + fi +} + +claim_shared_lock() { + local attempt current holder now candidate + [[ "${LOCK_MODE}" == exclusive ]] || return 0 + for attempt in 1 2 3; do + current="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o json)" + holder="$(jq -r '.spec.holderIdentity // ""' <<<"${current}")" + if [[ "${holder}" == "${CAMPAIGN_ID}" ]]; then + lock_claimed=1 + printf '%s\n' "${current}" >"${ARTIFACT_ROOT}/control/shared-lease-claimed.json" + return 0 + fi + [[ -z "${holder}" ]] || { + printf 'Shared Lease is held by another campaign: %s\n' "${holder}" >&2 + return 1 + } + now="$(date -u +%FT%T.000000Z)" + candidate="$(jq \ + --arg holder "${CAMPAIGN_ID}" --arg now "${now}" \ + --argjson duration "${LOCK_DURATION_SECONDS}" ' + .spec.holderIdentity=$holder | + .spec.acquireTime=$now | + .spec.renewTime=$now | + .spec.leaseDurationSeconds=$duration | + .metadata.labels["adai.aws/campaign"]=$holder | + .metadata.labels["adai.aws/owner"]="ep-backend-comparison"' \ + <<<"${current}")" + if printf '%s\n' "${candidate}" | "${K[@]}" replace -f - \ + >"${ARTIFACT_ROOT}/control/shared-lease-claim-attempt-${attempt}.json" 2>&1; then + lock_claimed=1 + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/control/shared-lease-claimed.json" + return 0 + fi + done + printf 'Failed to claim shared Lease after 3 optimistic attempts\n' >&2 + return 1 +} + +release_shared_lock() { + local current holder now + ((lock_claimed == 1)) || return 0 + current="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o json 2>/dev/null || true)" + holder="$(jq -r '.spec.holderIdentity // ""' <<<"${current}")" + [[ "${holder}" == "${CAMPAIGN_ID}" ]] || { + printf 'Refusing to release shared Lease held by %s\n' "${holder}" >&2 + return 1 + } + now="$(date -u +%FT%T.000000Z)" + jq --arg now "${now}" ' + .spec.holderIdentity="" | + .spec.renewTime=$now | + .spec.leaseDurationSeconds=1 | + del(.metadata.labels["adai.aws/campaign"], .metadata.labels["adai.aws/owner"])' \ + <<<"${current}" | "${K[@]}" replace -f - \ + >"${ARTIFACT_ROOT}/teardown/shared-lease-release.json" + lock_claimed=0 +} + +gpu_requests_on_node() { + local node="$1" + "${K[@]}" get pods -A --field-selector "spec.nodeName=${node}" -o json | jq ' + [.items[] + | select(.status.phase != "Succeeded" and .status.phase != "Failed") + | ([.spec.containers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] + | add // 0) as $app + | ([.spec.initContainers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] + | max // 0) as $init + | [$app, $init] | max] + | add // 0' +} + +verify_node_free() { + local node="$1" ready instance_type gpu efa requests + ready="$("${K[@]}" get node "${node}" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" + instance_type="$("${K[@]}" get node "${node}" -o jsonpath='{.metadata.labels.node\.kubernetes\.io/instance-type}')" + gpu="$("${K[@]}" get node "${node}" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}')" + efa="$("${K[@]}" get node "${node}" -o jsonpath='{.status.allocatable.vpc\.amazonaws\.com/efa}')" + requests="$(gpu_requests_on_node "${node}")" + [[ "${ready}" == True && "${instance_type}" == p6-b200.48xlarge && \ + "${gpu}" == 8 && "${efa}" == 8 && "${requests}" -eq 0 ]] || { + printf 'Node admission failed: node=%s ready=%s type=%s gpu=%s efa=%s requested_gpu=%s\n' \ + "${node}" "${ready}" "${instance_type}" "${gpu}" "${efa}" "${requests}" >&2 + return 1 + } +} + +cleanup_case() { + local case_to_delete cleanup_status=0 + [[ -n "${current_case}" ]] || return 0 + case_to_delete="${current_case}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete statefulset "${case_to_delete}" \ + --ignore-not-found --wait=true --timeout=5m >/dev/null 2>&1 || cleanup_status=1 + # StatefulSet deletion can return before its cascading Pod deletions finish. + # Wait for the GPU requests to disappear before admitting the next arm. + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" wait --for=delete pod \ + -l "app=${case_to_delete}" --timeout=5m >/dev/null 2>&1 || cleanup_status=1 + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete service "${case_to_delete}" \ + --ignore-not-found >/dev/null 2>&1 || cleanup_status=1 + current_case="" + return "${cleanup_status}" +} + +finish() { + local command_status=$? teardown_status=0 owned="" remaining=0 namespace_remaining=0 + trap - EXIT INT TERM + set +e + cleanup_case || teardown_status=1 + if ((namespace_created == 1)); then + owned="$("${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" \ + -o jsonpath='{.metadata.labels.adai\.aws/campaign}' 2>/dev/null || true)" + if [[ "${owned}" == "${CAMPAIGN_ID}" ]]; then + "${K[@]}" delete namespace "${CAMPAIGN_NAMESPACE}" \ + --wait=true --timeout=10m >"${ARTIFACT_ROOT}/teardown/namespace-delete.log" 2>&1 || \ + teardown_status=1 + else + printf 'Refusing to delete namespace without owned campaign label: %s\n' \ + "${CAMPAIGN_NAMESPACE}" >"${ARTIFACT_ROOT}/teardown/refused.txt" + teardown_status=1 + fi + fi + if "${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" >/dev/null 2>&1; then + namespace_remaining=1 + teardown_status=1 + fi + if ((namespace_created == 1 || lock_claimed == 1)); then + check_shared_lock || teardown_status=1 + fi + release_shared_lock || teardown_status=1 + "${K[@]}" get all -A -l "adai.aws/campaign=${CAMPAIGN_ID}" -o json \ + >"${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>&1 || teardown_status=1 + remaining="$(jq '.items | length' "${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>/dev/null || printf '1')" + [[ "${remaining}" -eq 0 ]] || teardown_status=1 + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/teardown/shared-lease-after.json" 2>&1 || true + "${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-after.json" 2>&1 || \ + teardown_status=1 + "${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-after.json" 2>&1 || \ + teardown_status=1 + if ((command_status == 0 && teardown_status == 0)); then + printf 'PASS teardown_verified=1_dimensionless remaining_resources=0_resources\n' \ + >"${ARTIFACT_ROOT}/STATUS" + printf 'PASS completed_at=%s\n' "$(date -u +%FT%TZ)" \ + >"${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" + else + printf 'FAIL command_status=%s_dimensionless teardown_status=%s_dimensionless remaining_resources=%s_resources namespace_remaining=%s_namespaces\n' \ + "${command_status}" "${teardown_status}" "${remaining}" \ + "${namespace_remaining}" >"${ARTIFACT_ROOT}/STATUS" + rm -f "${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" + fi + find "${ARTIFACT_ROOT}" -type f ! -name SHA256SUMS -print0 | sort -z | \ + xargs -0 sha256sum >"${ARTIFACT_ROOT}/SHA256SUMS" + if ((command_status == 0 && teardown_status != 0)); then + command_status=1 + fi + exit "${command_status}" +} +trap finish EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +"${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-before.json" +"${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-before.json" +"${K[@]}" get namespaces -o json >"${ARTIFACT_ROOT}/control/namespaces-before.json" +"${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/control/shared-lease-before.json" +if "${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" >/dev/null 2>&1; then + printf 'Refusing to reuse existing namespace: %s\n' "${CAMPAIGN_NAMESPACE}" >&2 + exit 1 +fi +claim_shared_lock +check_shared_lock +aws sts get-caller-identity --output json >"${ARTIFACT_ROOT}/control/aws-caller-identity.json" +printf '%s\n' "${selected_nodes[@]}" >"${ARTIFACT_ROOT}/control/selected-nodes.txt" +: >"${ARTIFACT_ROOT}/control/protected-nodes.txt" +if ((${#protected_nodes[@]} > 0)); then + printf '%s\n' "${protected_nodes[@]}" | sort -u \ + >"${ARTIFACT_ROOT}/control/protected-nodes.txt" +fi + +for node in "${selected_nodes[@]}"; do + verify_node_free "${node}" +done + +"${K[@]}" create namespace "${CAMPAIGN_NAMESPACE}" --dry-run=client -o json | \ + jq --arg campaign "${CAMPAIGN_ID}" ' + .metadata.labels["adai.aws/campaign"]=$campaign | + .metadata.labels["adai.aws/owner"]="ep-backend-comparison"' | \ + "${K[@]}" create -f - >/dev/null +namespace_created=1 + +# Read the live host mitigation before invoking DeepEP V2. The EFA 3.3.0g +# revalidation exposed a UVM HMM kernel panic, so a non-mitigated node is a hard +# admission failure rather than a benchmark attempt. +for index in 0 1 2 3; do + node="${selected_nodes[${index}]}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" apply -f - >/dev/null <&2 + exit 1 + } + rg -q '^GDRDRV=character-device$' "${ARTIFACT_ROOT}/control/host-audit-${index}.log" || { + printf 'Benchmark admission blocked by missing /dev/gdrdrv on node %s\n' \ + "${selected_nodes[${index}]}" >&2 + exit 1 + } +done +"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete pod -l \ + "adai.aws/campaign=${CAMPAIGN_ID}" --wait=true --timeout=5m >/dev/null + +"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" create configmap ep-benchmark-scripts \ + --from-file=ep_benchmark.py="${case_dir}/ep_benchmark.py" \ + --from-file=run_ep_rank.sh="${case_dir}/run_ep_rank.sh" \ + --dry-run=client -o yaml | "${K[@]}" apply -f - >/dev/null + +jq -n \ + --arg campaign_id "${CAMPAIGN_ID}" \ + --arg created_at_utc "$(date -u +%FT%TZ)" \ + --arg region ap-south-1 \ + --arg cluster ml-clusters-shared-ap-south-1 \ + --arg git_commit "$(git -C "${case_dir}" rev-parse HEAD)" \ + --arg uccl "${UCCL_IMAGE}" \ + --arg v1 "${DEEPEP_V1_IMAGE}" \ + --arg v2 "${DEEPEP_V2_IMAGE}" \ + --argjson warmups "${WARMUP_ITERATIONS}" \ + --argjson iterations "${MEASURED_ITERATIONS}" \ + --argjson starts "${INDEPENDENT_STARTS}" \ + '{campaign_id:$campaign_id,created_at_utc:$created_at_utc,region:$region, + cluster:$cluster,git_commit:$git_commit, + images:{uccl:$uccl,"deepep-v1-nvshmem":$v1,"deepep-v2-gin-gda":$v2}, + comparison:{profiles:{decode:{tokens_per_rank:128,api_mode:"low-latency", + primary_metric:"slowest-rank latency in milliseconds"}, + prefill:{tokens_per_rank:4096,api_mode:"normal", + primary_metric:"effective logical gigabytes per second per rank"}}, + hidden_dimensions:7168,experts:256,top_k_dimensionless:8, + warmup_iterations:$warmups,measured_iterations:$iterations, + independent_starts:$starts}}' \ + >"${ARTIFACT_ROOT}/control/provenance.json" + +run_case() { + local profile="$1" arm="$2" world_size="$3" run_index="$4" dtype_order="$5" + local warmups="${6:-${WARMUP_ITERATIONS}}" iterations="${7:-${MEASURED_ITERATIONS}}" + local nccl_debug="${8:-WARN}" label="${9:-measurement}" + local nodes=$((world_size / 8)) safe_arm="${arm//-}" node_values="" out="" + current_case="ep-${profile:0:1}${world_size}-r${run_index}-${safe_arm:0:20}-${label}" + current_case="${current_case:0:63}" + out="${ARTIFACT_ROOT}/runs/${profile}/ep${world_size}/${label}-repeat-${run_index}/${arm}" + mkdir -p "${out}" + + check_shared_lock + for ((index = 0; index < nodes; index++)); do + verify_node_free "${selected_nodes[${index}]}" + node_values+=$'\n - '"${selected_nodes[${index}]}" + done + "${K[@]}" get pods -A -o json >"${out}/pods-before.json" + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${out}/shared-lease-before.json" + + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" apply -f - >/dev/null <"${out}/statefulset.yaml" + + local deadline=$((SECONDS + CASE_TIMEOUT_SECONDS)) complete=0 pod="" + while ((SECONDS < deadline)); do + complete=0 + for ((index = 0; index < nodes; index++)); do + pod="${current_case}-${index}" + if "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=30 2>/dev/null | \ + rg -q '^ADAI_EP_COMPLETE$'; then + complete=$((complete + 1)) + elif "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=80 2>/dev/null | \ + rg -q '^ADAI_EP_FAILED$'; then + complete=-1 + break + fi + done + ((complete == nodes || complete == -1)) && break + sleep 10 + done + + for ((index = 0; index < nodes; index++)); do + pod="${current_case}-${index}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" >"${out}/${pod}.log" 2>&1 || true + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" get pod "${pod}" -o yaml \ + >"${out}/${pod}.yaml" 2>&1 || true + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" describe pod "${pod}" \ + >"${out}/${pod}-describe.txt" 2>&1 || true + done + ((complete == nodes)) || { + printf 'Case failed or timed out: profile=%s arm=%s EP%s repeat=%s label=%s complete=%s/%s\n' \ + "${profile}" "${arm}" "${world_size}" "${run_index}" "${label}" "${complete}" "${nodes}" >&2 + return 1 + } + local rank_zero_log="${out}/${current_case}-0.log" result_count + python3 "${case_dir}/extract_results.py" \ + "${rank_zero_log}" "${out}/results.jsonl" >/dev/null + result_count="$(wc -l <"${out}/results.jsonl")" + [[ "${result_count}" -eq 2 ]] || { + printf 'Expected 2 benchmark results in %s, found %s\n' "${rank_zero_log}" "${result_count}" >&2 + return 1 + } + printf 'PASS profile=%s arm=%s EP%s repeat=%s label=%s at %s UTC\n' \ + "${profile}" "${arm}" "${world_size}" "${run_index}" "${label}" "$(date -u +%FT%TZ)" \ + >"${out}/STATUS" + cleanup_case +} + +# A short V2-only admission run proves the HMM mitigation, GIN/GDAKI path, and +# common-harness correctness before any scored matrix work. +run_case decode deepep-v2-gin-gda 16 0 bf16,fp8 2 5 INFO admission +v2_admission_dir="${ARTIFACT_ROOT}/runs/decode/ep16/admission-repeat-0/deepep-v2-gin-gda" +rg -q 'GDAKI.*createContext|gin GDAKI: createContext done' \ + "${v2_admission_dir}"/*.log || { + printf 'DeepEP V2 admission completed without a GDAKI context proof\n' >&2 + exit 1 +} +printf 'PASS\n' >"${v2_admission_dir}/GIN_ADMISSION_STATUS" + +# Exercise every backend's normal high-throughput API before entering the +# doubled scored matrix. These short runs are admission evidence only. +for arm in uccl deepep-v1-nvshmem deepep-v2-gin-gda; do + run_case prefill "${arm}" 16 0 fp8,bf16 2 5 WARN admission +done + +for world_size in 16 32; do + for run_index in 1 2 3; do + case "${run_index}" in + 1) + order=(uccl deepep-v1-nvshmem deepep-v2-gin-gda) + profiles=(decode prefill) + dtypes=fp8,bf16 + ;; + 2) + order=(deepep-v2-gin-gda uccl deepep-v1-nvshmem) + profiles=(prefill decode) + dtypes=bf16,fp8 + ;; + 3) + order=(deepep-v1-nvshmem deepep-v2-gin-gda uccl) + profiles=(decode prefill) + dtypes=fp8,bf16 + ;; + esac + for profile in "${profiles[@]}"; do + for arm in "${order[@]}"; do + run_case "${profile}" "${arm}" "${world_size}" "${run_index}" "${dtypes}" + done + done + done +done + +check_shared_lock +for node in "${selected_nodes[@]}"; do + verify_node_free "${node}" +done +python3 "${case_dir}/summarize_results.py" "${ARTIFACT_ROOT}/runs" \ + --starts="${INDEPENDENT_STARTS}" \ + --provenance="${ARTIFACT_ROOT}/control/provenance.json" \ + --json="${ARTIFACT_ROOT}/summary/summary.json" \ + --markdown="${ARTIFACT_ROOT}/summary/summary.md" +printf 'PASS EP backend comparison completed at %s UTC\n' "$(date -u +%FT%TZ)" diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh new file mode 100755 index 000000000..0cb2a6939 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${EP_ARM:?Set EP_ARM}" +: "${EP_NODES:?Set EP_NODES}" +: "${EP_SERVICE:?Set EP_SERVICE}" +: "${EP_MASTER_ADDR:?Set EP_MASTER_ADDR}" +: "${POD_NAME:?Set POD_NAME}" +: "${EP_RUN_INDEX:?Set EP_RUN_INDEX}" +: "${EP_WORKLOAD_PROFILE:?Set EP_WORKLOAD_PROFILE}" +: "${EP_DISPATCH_DTYPES:?Set EP_DISPATCH_DTYPES}" +: "${EP_WARMUPS:=20}" +: "${EP_ITERATIONS:=100}" +: "${EP_SEED:=20260824}" +: "${EP_NCCL_DEBUG:=WARN}" + +ordinal="${POD_NAME##*-}" +export FI_PROVIDER=efa +export FI_EFA_USE_DEVICE_RDMA=1 +export NCCL_SOCKET_IFNAME='^lo,docker,veth' +export NCCL_NET_PLUGIN=ofi +export NCCL_DEBUG="${EP_NCCL_DEBUG}" +export CUDA_DEVICE_MAX_CONNECTIONS=1 + +case "${EP_ARM}" in + uccl) + export PER_EXPERT_BATCHING=1 + export UCCL_SOCKET_IFNAME='^lo,docker,veth' + ;; + deepep-v1-nvshmem) + export NVSHMEM_REMOTE_TRANSPORT=libfabric + export NVSHMEM_LIBFABRIC_PROVIDER=efa + export NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME='^lo,docker,veth' + export NVSHMEM_NETDEVS_POLICY=EXTERNAL_SHARING_PCIE_SWITCH_NIC_EXCLUSIVE + ;; + deepep-v2-gin-gda) + export FI_EFA_USE_HW_CNTR=1 + export NCCL_GIN_TYPE=5 + export NCCL_SYM_GIN_KERNELS_ENABLE=0 + export EP_BUFFER_DEBUG=1 + if [[ "${NCCL_DEBUG}" == INFO ]]; then + export NCCL_DEBUG_SUBSYS=INIT,ENV,NET + export FI_LOG_LEVEL=info + export FI_LOG_PROV=efa + export FI_LOG_SUBSYS=cntr + fi + ;; + *) + printf 'Unsupported EP_ARM=%s\n' "${EP_ARM}" >&2 + exit 2 + ;; +esac + +printf 'ADAI_EP_LAUNCH profile=%s arm=%s nodes=%s ranks=%s run_index=%s_dimensionless dtype_order=%s warmups=%s_iterations measured=%s_iterations\n' \ + "${EP_WORKLOAD_PROFILE}" "${EP_ARM}" "${EP_NODES}" "$((EP_NODES * 8))" "${EP_RUN_INDEX}" \ + "${EP_DISPATCH_DTYPES}" "${EP_WARMUPS}" "${EP_ITERATIONS}" + +set +e +torchrun \ + --nnodes="${EP_NODES}" \ + --nproc-per-node=8 \ + --node-rank="${ordinal}" \ + --master-addr="${EP_MASTER_ADDR}" \ + --master-port=29400 \ + /opt/benchmark/ep_benchmark.py \ + --arm="${EP_ARM}" \ + --profile="${EP_WORKLOAD_PROFILE}" \ + --hidden=7168 \ + --top-k=8 \ + --experts=256 \ + --seed="${EP_SEED}" \ + --warmups="${EP_WARMUPS}" \ + --iterations="${EP_ITERATIONS}" \ + --run-index="${EP_RUN_INDEX}" \ + --dispatch-dtypes="${EP_DISPATCH_DTYPES}" +status=$? +set -e + +printf 'ADAI_EP_EXIT_STATUS=%s_dimensionless\n' "${status}" +if ((status == 0)); then + printf 'ADAI_EP_COMPLETE\n' +else + printf 'ADAI_EP_FAILED\n' +fi +sleep infinity diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py new file mode 100755 index 000000000..b725b3858 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py @@ -0,0 +1,661 @@ +#!/usr/bin/env python3 +"""Validate and summarize common-boundary EP benchmark logs.""" + +from __future__ import annotations + +import argparse +import json +import math +import random +import re +import statistics +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable + +from result_io import load_result_log + + +ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") +WORLD_SIZES = (16, 32) +DTYPES = ("fp8", "bf16") +PROFILES = ("decode", "prefill") +PROFILE_CONFIG = { + "decode": { + "tokens_per_rank": 128, + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "timing_boundary": ( + "BF16 input ready through dispatch and combine completion; " + "slowest rank CUDA elapsed time" + ), + }, + "prefill": { + "tokens_per_rank": 4_096, + "api_mode": "normal", + "primary_metric": ("effective logical gigabytes per second per rank"), + "timing_boundary": ( + "BF16 input and route ready through required layout, dispatch, and " + "combine completion; slowest rank CUDA elapsed time" + ), + }, +} +ARM_LABELS = { + "uccl": "UCCL", + "deepep-v1-nvshmem": "DeepEP V1 NVSHMEM", + "deepep-v2-gin-gda": "DeepEP V2 NCCL GIN", +} +BOOTSTRAP_SAMPLES = 20_000 +MAX_RUN_TO_RUN_CV_PERCENT = 5.0 +EXPECTED_STARTS = 3 +EXPECTED_WARMUPS = 20 +EXPECTED_ITERATIONS = 100 +LOGICAL_PAYLOAD_DEFINITION = ( + "per valid expert assignment: dispatch tensor plus FP8 scales when selected " + "plus BF16 combine tensor; backend metadata excluded" +) +SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") + + +def percentile(values: list[float], quantile: float) -> float: + ordered = sorted(values) + if not ordered: + raise ValueError("percentile requires at least one value") + position = (len(ordered) - 1) * quantile + lower = int(position) + upper = min(lower + 1, len(ordered) - 1) + fraction = position - lower + return ordered[lower] * (1 - fraction) + ordered[upper] * fraction + + +def bootstrap_median_ci( + values: list[float], seed: int, samples: int = BOOTSTRAP_SAMPLES +) -> tuple[float, float]: + rng = random.Random(seed) + medians = [ + statistics.median(rng.choices(values, k=len(values))) for _ in range(samples) + ] + return percentile(medians, 0.025), percentile(medians, 0.975) + + +def load_results(root: Path) -> list[dict[str, Any]]: + results: dict[tuple[str, str, int, int, str], dict[str, Any]] = {} + sources: dict[tuple[str, str, int, int, str], Path] = {} + for path in sorted(root.rglob("*.log")): + for result in load_result_log(path): + key = ( + result["workload_profile"], + result["arm"], + result["world_size_ranks"], + result["run_index_dimensionless"], + result["dispatch_dtype"], + ) + if key in results and results[key] != result: + raise ValueError( + f"conflicting result for {key}: {sources[key]} and {path}" + ) + results[key] = result + sources[key] = path + return list(results.values()) + + +def validate(results: list[dict[str, Any]], starts: int) -> None: + if starts != EXPECTED_STARTS: + raise ValueError(f"scored matrix requires exactly {EXPECTED_STARTS} starts") + expected = { + (profile, arm, world, run, dtype) + for profile in PROFILES + for arm in ARMS + for world in WORLD_SIZES + for run in range(1, starts + 1) + for dtype in DTYPES + } + observed = { + ( + result["workload_profile"], + result["arm"], + result["world_size_ranks"], + result["run_index_dimensionless"], + result["dispatch_dtype"], + ) + for result in results + if result["run_index_dimensionless"] > 0 + } + missing = expected - observed + extra = observed - expected + if missing or extra: + raise ValueError( + f"result matrix mismatch; missing={sorted(missing)}, extra={sorted(extra)}" + ) + + measured = [result for result in results if result["run_index_dimensionless"] > 0] + if len(measured) != len(expected): + raise ValueError( + f"result matrix has {len(measured)} scored records; expected {len(expected)}" + ) + common_shape = { + "benchmark": "common-boundary-dispatch-combine", + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "gpus_per_node": 8, + "warmup_iterations": EXPECTED_WARMUPS, + "measured_iterations": EXPECTED_ITERATIONS, + "logical_payload_definition": LOGICAL_PAYLOAD_DEFINITION, + } + for result in measured: + profile = result.get("workload_profile") + if profile not in PROFILE_CONFIG: + raise ValueError(f"unexpected workload profile: {profile}") + profile_config = PROFILE_CONFIG[profile] + expected_shape = { + **common_shape, + "tokens_per_rank": profile_config["tokens_per_rank"], + "timing_boundary": profile_config["timing_boundary"], + "primary_metric": profile_config["primary_metric"], + "layout_in_timed_region": profile == "prefill", + "backend_api_mode": ( + "elastic" + if result.get("arm") == "deepep-v2-gin-gda" + else profile_config["api_mode"] + ), + } + for field, expected_value in expected_shape.items(): + if result.get(field) != expected_value: + raise ValueError( + f"unexpected {field}: expected {expected_value}, got {result.get(field)}" + ) + world_size = result["world_size_ranks"] + if result.get("nodes") != world_size // 8: + raise ValueError(f"invalid node count for EP{world_size}: {result}") + tokens_per_rank = profile_config["tokens_per_rank"] + if result.get("global_input_tokens") != tokens_per_rank * world_size: + raise ValueError(f"invalid global input token count: {result}") + if result["correctness"]["status"] != "PASS": + raise ValueError(f"correctness did not pass: {result}") + tolerance = 9e-4 if result["dispatch_dtype"] == "fp8" else 1e-5 + correctness = result["correctness"] + if correctness.get("tolerance_dimensionless") != tolerance: + raise ValueError(f"unexpected correctness tolerance: {result}") + if not math.isfinite( + correctness.get("normalized_diff_dimensionless", math.inf) + ): + raise ValueError(f"non-finite correctness result: {result}") + if correctness["normalized_diff_dimensionless"] > tolerance: + raise ValueError(f"correctness exceeds tolerance: {result}") + image_reference = result["runtime"]["image_reference"] + digest = image_reference.rsplit("@sha256:", 1)[-1] + if "@sha256:" not in image_reference or not SHA256_PATTERN.fullmatch(digest): + raise ValueError( + f"image is not digest pinned: {result['runtime']['image_reference']}" + ) + for field in ("gpu", "torch_version", "cuda_version", "nccl_version"): + if not result["runtime"].get(field): + raise ValueError(f"runtime is missing {field}: {result}") + for field in ("route_hash_sha256", "input_hash_sha256"): + if not SHA256_PATTERN.fullmatch(result.get(field, "")): + raise ValueError(f"invalid {field}: {result}") + expected_selections = result["global_input_tokens"] * 8 + if result.get("global_valid_expert_selections") != expected_selections: + raise ValueError(f"invalid valid-expert selection count: {result}") + dispatch_bytes = 7_168 * 2 + if result["dispatch_dtype"] == "fp8": + dispatch_bytes = 7_168 + math.ceil(7_168 / 128) * 4 + expected_logical_bytes = tokens_per_rank * 8 * (dispatch_bytes + 7_168 * 2) + logical_bytes = result.get("avg_logical_payload_bytes_per_rank") + scaleout_bytes = result.get("avg_scaleout_logical_payload_bytes_per_rank") + if logical_bytes != expected_logical_bytes: + raise ValueError(f"invalid logical payload: {result}") + if not isinstance(scaleout_bytes, (int, float)) or not ( + 0 < scaleout_bytes <= logical_bytes + ): + raise ValueError(f"invalid scale-out logical payload: {result}") + latency = result.get("latency_ms", {}).get("median", 0) + if ( + not isinstance(latency, (int, float)) + or not math.isfinite(latency) + or latency <= 0 + ): + raise ValueError(f"invalid median latency: {result}") + elapsed_seconds = latency / 1e3 + expected_metrics = { + "aggregate_input_tokens_per_second": result["global_input_tokens"] + / elapsed_seconds, + "effective_logical_gigabytes_per_second_per_rank": logical_bytes + / elapsed_seconds + / 1e9, + "effective_scaleout_logical_gigabytes_per_second_per_rank": scaleout_bytes + / elapsed_seconds + / 1e9, + } + for field in ( + "aggregate_input_tokens_per_second", + "effective_logical_gigabytes_per_second_per_rank", + "effective_scaleout_logical_gigabytes_per_second_per_rank", + ): + value = result.get(field, 0) + if ( + not isinstance(value, (int, float)) + or not math.isfinite(value) + or value <= 0 + ): + raise ValueError(f"invalid positive metric {field}: {result}") + if not math.isclose( + value, expected_metrics[field], rel_tol=1e-12, abs_tol=1e-9 + ): + raise ValueError( + f"metric {field} does not match common accounting: {result}" + ) + + common_fields = ( + "warmup_iterations", + "measured_iterations", + "logical_payload_definition", + ) + for field in common_fields: + values = {json.dumps(result[field], sort_keys=True) for result in measured} + if len(values) != 1: + raise ValueError(f"scored results disagree on {field}: {sorted(values)}") + + runtime_signatures = { + json.dumps( + { + key: result["runtime"].get(key) + for key in ("gpu", "torch_version", "cuda_version", "nccl_version") + }, + sort_keys=True, + ) + for result in measured + } + if len(runtime_signatures) != 1: + raise ValueError(f"runtime stack mismatch: {sorted(runtime_signatures)}") + + for arm in ARMS: + image_references = { + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + } + if len(image_references) != 1: + raise ValueError( + f"{arm} did not use one immutable image: {image_references}" + ) + + for profile in PROFILES: + for world in WORLD_SIZES: + same_world = [ + result + for result in measured + if result["workload_profile"] == profile + and result["world_size_ranks"] == world + ] + route_hashes = {result["route_hash_sha256"] for result in same_world} + input_hashes = {result["input_hash_sha256"] for result in same_world} + if len(route_hashes) != 1 or len(input_hashes) != 1: + raise ValueError( + f"{profile} EP{world} did not replay one route/input: " + f"routes={route_hashes}, inputs={input_hashes}" + ) + for dtype in DTYPES: + same_cell = [ + result for result in same_world if result["dispatch_dtype"] == dtype + ] + for field in ( + "avg_logical_payload_bytes_per_rank", + "avg_scaleout_logical_payload_bytes_per_rank", + "global_valid_expert_selections", + ): + values = {result[field] for result in same_cell} + if len(values) != 1: + raise ValueError( + f"{profile} EP{world} {dtype} disagrees on {field}: " + f"{sorted(values)}" + ) + + +def validate_provenance( + provenance: dict[str, Any], results: list[dict[str, Any]], starts: int +) -> None: + measured = [result for result in results if result["run_index_dimensionless"] > 0] + expected_images = { + arm: next( + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + ) + for arm in ARMS + } + if provenance.get("images") != expected_images: + raise ValueError( + f"provenance images do not match scored results: {provenance.get('images')}" + ) + expected_comparison = { + "profiles": { + profile: { + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "api_mode": PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + } + for profile in PROFILES + }, + "hidden_dimensions": 7_168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": EXPECTED_WARMUPS, + "measured_iterations": EXPECTED_ITERATIONS, + "independent_starts": starts, + } + if provenance.get("comparison") != expected_comparison: + raise ValueError( + "provenance comparison does not match the scored matrix: " + f"{provenance.get('comparison')}" + ) + for field in ("campaign_id", "created_at_utc", "region", "cluster", "git_commit"): + if not provenance.get(field): + raise ValueError(f"provenance is missing {field}") + + +def primary_metric_value(result: dict[str, Any]) -> float: + if result["workload_profile"] == "decode": + return result["latency_ms"]["median"] + return result["effective_logical_gigabytes_per_second_per_rank"] + + +def arm_summary( + results: Iterable[dict[str, Any]], seed: int, profile: str +) -> dict[str, Any]: + ordered = sorted(results, key=lambda item: item["run_index_dimensionless"]) + latencies = [item["latency_ms"]["median"] for item in ordered] + token_rates = [item["aggregate_input_tokens_per_second"] for item in ordered] + logical_rates = [ + item["effective_logical_gigabytes_per_second_per_rank"] for item in ordered + ] + scaleout_rates = [ + item["effective_scaleout_logical_gigabytes_per_second_per_rank"] + for item in ordered + ] + primary_values = [primary_metric_value(item) for item in ordered] + primary_mean = statistics.fmean(primary_values) + primary_stdev = statistics.stdev(primary_values) if len(primary_values) > 1 else 0.0 + ci_low, ci_high = bootstrap_median_ci(primary_values, seed) + return { + "starts": len(ordered), + "run_indices_dimensionless": [ + item["run_index_dimensionless"] for item in ordered + ], + "per_start_median_latency_ms": latencies, + "median_latency_ms": statistics.median(latencies), + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "per_start_primary_values": primary_values, + "median_primary_value": statistics.median(primary_values), + "bootstrap_95_percent_ci_primary_value": [ci_low, ci_high], + "run_to_run_cv_percent": ( + primary_stdev / primary_mean * 100 if primary_mean else 0.0 + ), + "median_aggregate_input_tokens_per_second": statistics.median(token_rates), + "median_effective_logical_gigabytes_per_second_per_rank": statistics.median( + logical_rates + ), + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": statistics.median( + scaleout_rates + ), + } + + +def summarize( + results: list[dict[str, Any]], + starts: int, + provenance: dict[str, Any] | None = None, +) -> dict[str, Any]: + measured = [result for result in results if result["run_index_dimensionless"] > 0] + by_cell_arm: dict[tuple[str, int, str, str], list[dict[str, Any]]] = defaultdict( + list + ) + for result in measured: + by_cell_arm[ + ( + result["workload_profile"], + result["world_size_ranks"], + result["dispatch_dtype"], + result["arm"], + ) + ].append(result) + + cells = [] + for profile_index, profile in enumerate(PROFILES): + for world in WORLD_SIZES: + for dtype in DTYPES: + arms = { + arm: arm_summary( + by_cell_arm[(profile, world, dtype, arm)], + seed=( + 20260824 + + profile_index * 1_000 + + world + + len(dtype) + + index + ), + profile=profile, + ) + for index, arm in enumerate(ARMS) + } + comparisons = {} + v2_by_run = { + result["run_index_dimensionless"]: primary_metric_value(result) + for result in by_cell_arm[ + (profile, world, dtype, "deepep-v2-gin-gda") + ] + } + for index, baseline in enumerate(("uccl", "deepep-v1-nvshmem")): + baseline_by_run = { + result["run_index_dimensionless"]: primary_metric_value(result) + for result in by_cell_arm[(profile, world, dtype, baseline)] + } + if profile == "decode": + paired = [ + (baseline_by_run[run] - v2_by_run[run]) + / baseline_by_run[run] + * 100 + for run in range(1, starts + 1) + ] + else: + paired = [ + (v2_by_run[run] - baseline_by_run[run]) + / baseline_by_run[run] + * 100 + for run in range(1, starts + 1) + ] + ci_low, ci_high = bootstrap_median_ci( + paired, + 20260900 + profile_index * 1_000 + world + index, + ) + stable = ( + arms[baseline]["run_to_run_cv_percent"] + <= MAX_RUN_TO_RUN_CV_PERCENT + and arms["deepep-v2-gin-gda"]["run_to_run_cv_percent"] + <= MAX_RUN_TO_RUN_CV_PERCENT + ) + comparisons[f"deepep-v2-gin-gda_vs_{baseline}"] = { + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "paired_improvement_percent_per_start": paired, + "median_paired_improvement_percent": statistics.median(paired), + "bootstrap_95_percent_ci_improvement_percent": [ + ci_low, + ci_high, + ], + "direction_supported": stable and (ci_low > 0 or ci_high < 0), + } + same_cell = [ + result + for result in measured + if result["workload_profile"] == profile + and result["world_size_ranks"] == world + and result["dispatch_dtype"] == dtype + ] + cells.append( + { + "workload_profile": profile, + "world_size_ranks": world, + "dispatch_dtype": dtype, + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "route_hash_sha256": same_cell[0]["route_hash_sha256"], + "input_hash_sha256": same_cell[0]["input_hash_sha256"], + "arms": arms, + "comparisons": comparisons, + } + ) + images = { + arm: next( + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + ) + for arm in ARMS + } + runtime = { + field: measured[0]["runtime"][field] + for field in ("gpu", "torch_version", "cuda_version", "nccl_version") + } + summary = { + "schema_version_dimensionless": 2, + "status": "PASS", + "scored_result_records_dimensionless": len(measured), + "independent_starts_per_cell": starts, + "bootstrap_samples_dimensionless": BOOTSTRAP_SAMPLES, + "maximum_run_to_run_cv_percent_for_direction_support": MAX_RUN_TO_RUN_CV_PERCENT, + "timing_boundaries": { + profile: PROFILE_CONFIG[profile]["timing_boundary"] for profile in PROFILES + }, + "logical_payload_definition": measured[0]["logical_payload_definition"], + "comparison_scope": ( + "synthetic decode-like and prefill-like dispatch-plus-combine " + "communication workloads; not end-to-end training or serving" + ), + "configuration": { + "world_sizes_ranks": list(WORLD_SIZES), + "dispatch_dtypes": list(DTYPES), + "profiles": { + profile: { + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "api_mode": PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + } + for profile in PROFILES + }, + "hidden_dimensions": 7_168, + "experts_dimensionless": 256, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": EXPECTED_WARMUPS, + "measured_iterations_dimensionless": EXPECTED_ITERATIONS, + }, + "runtime": runtime, + "images": images, + "cells": cells, + } + if provenance is not None: + summary["campaign_provenance"] = provenance + return summary + + +def markdown(summary: dict[str, Any]) -> str: + lines = [ + "# Common-Boundary EP Results", + "", + f"Each cell has {summary['independent_starts_per_cell']} independent process starts. Values are medians across independent starts. Decode-like results use slowest-rank latency as the primary metric; prefill-like results use common logical throughput. These are synthetic communication workloads, not end-to-end training or serving results.", + ] + for profile in PROFILES: + title = ( + "Decode-like latency" if profile == "decode" else "Prefill-like throughput" + ) + tokens = PROFILE_CONFIG[profile]["tokens_per_rank"] + lines.extend(["", f"## {title}, {tokens:,} tokens/rank", ""]) + if profile == "decode": + lines.extend( + [ + "| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) |", + "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", + ] + ) + else: + lines.extend( + [ + "| EP size | Dispatch dtype | Backend | Logical throughput (GB/s/rank) | 95% bootstrap CI (GB/s/rank) | Run-to-run CV (%) | Latency (ms) | Input throughput (tokens/s) | Scale-out logical throughput (GB/s/rank) |", + "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", + ] + ) + for cell in summary["cells"]: + if cell["workload_profile"] != profile: + continue + for arm in ARMS: + value = cell["arms"][arm] + ci = value["bootstrap_95_percent_ci_primary_value"] + if profile == "decode": + row = ( + f"{value['median_latency_ms']:.4f} ms | " + f"[{ci[0]:.4f}, {ci[1]:.4f}] ms" + ) + else: + row = ( + f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " + f"[{ci[0]:.2f}, {ci[1]:.2f}] GB/s/rank" + ) + lines.append( + f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[arm]} | " + f"{row} | {value['run_to_run_cv_percent']:.2f}% | " + + ( + f"{value['median_aggregate_input_tokens_per_second']:,.2f} tokens/s | " + f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " + if profile == "decode" + else f"{value['median_latency_ms']:.4f} ms | {value['median_aggregate_input_tokens_per_second']:,.2f} tokens/s | " + ) + + f"{value['median_effective_scaleout_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank |" + ) + lines.extend( + [ + "", + "## Paired DeepEP V2 improvements", + "", + "Positive values mean DeepEP V2 had lower latency for Decode-like cells or higher logical throughput for Prefill-like cells. A direction is supported only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV.", + "", + "| Profile | EP size | Dispatch dtype | Baseline | Primary metric | Median improvement (%) | 95% bootstrap CI (%) | Direction supported |", + "|:---|---:|:---:|:---|:---|---:|:---:|:---:|", + ] + ) + for cell in summary["cells"]: + for baseline in ("uccl", "deepep-v1-nvshmem"): + comparison = cell["comparisons"][f"deepep-v2-gin-gda_vs_{baseline}"] + ci = comparison["bootstrap_95_percent_ci_improvement_percent"] + lines.append( + f"| {cell['workload_profile']} | {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[baseline]} | {comparison['primary_metric']} | " + f"{comparison['median_paired_improvement_percent']:.2f}% | " + f"[{ci[0]:.2f}, {ci[1]:.2f}]% | " + f"{'yes' if comparison['direction_supported'] else 'no'} |" + ) + lines.append("") + return "\n".join(lines) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("root", type=Path) + parser.add_argument("--starts", type=int, default=3) + parser.add_argument("--provenance", type=Path, required=True) + parser.add_argument("--json", type=Path, required=True) + parser.add_argument("--markdown", type=Path, required=True) + args = parser.parse_args() + results = load_results(args.root) + validate(results, args.starts) + provenance = json.loads(args.provenance.read_text()) + validate_provenance(provenance, results, args.starts) + summary = summarize(results, args.starts, provenance) + args.json.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n") + args.markdown.write_text(markdown(summary)) + print( + f"PASS EP matrix: {len(summary['cells'])} cells, " + f"{args.starts} independent starts per arm/cell" + ) + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py new file mode 100644 index 000000000..b1fc9c522 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py @@ -0,0 +1,132 @@ +import importlib.util +from pathlib import Path +import sys +import tempfile +import unittest + +try: + import torch +except ModuleNotFoundError: + torch = None + + +MODULE = None +if torch is not None: + module_path = Path(__file__).with_name("ep_benchmark.py") + spec = importlib.util.spec_from_file_location("ep_benchmark", module_path) + assert spec and spec.loader + MODULE = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = MODULE + spec.loader.exec_module(MODULE) + + +@unittest.skipIf(torch is None, "PyTorch is required for benchmark helper tests") +class EpBenchmarkTest(unittest.TestCase): + def test_route_is_balanced_and_unique(self): + routes = [ + MODULE.make_route(rank, 128, 256, 8, 20260824, torch.device("cpu")) + for rank in range(32) + ] + route = torch.cat(routes) + histogram = torch.bincount(route.flatten(), minlength=256) + self.assertEqual(histogram.min().item(), 128) + self.assertEqual(histogram.max().item(), 128) + for row in route: + self.assertEqual(torch.unique(row).numel(), 8) + + def test_common_payload_formula(self): + route = MODULE.make_route(0, 128, 256, 8, 20260824, torch.device("cpu")) + bf16_all, bf16_remote, selections = MODULE.logical_payload_bytes_per_rank( + route, 0, 16, 8, 7168, "bf16", 256 + ) + fp8_all, fp8_remote, _ = MODULE.logical_payload_bytes_per_rank( + route, 0, 16, 8, 7168, "fp8", 256 + ) + self.assertEqual(selections, 1024) + self.assertEqual(bf16_all, 1024 * (7168 * 2 + 7168 * 2)) + self.assertEqual(fp8_all, 1024 * (7168 + 56 * 4 + 7168 * 2)) + self.assertGreater(bf16_remote, 0) + self.assertGreater(fp8_remote, 0) + self.assertLess(bf16_remote, bf16_all) + self.assertLess(fp8_remote, fp8_all) + + def test_profile_shapes_are_fixed(self): + self.assertEqual(MODULE.WORKLOAD_PROFILES["decode"].tokens_per_rank, 128) + self.assertEqual(MODULE.WORKLOAD_PROFILES["decode"].api_mode, "low-latency") + self.assertEqual(MODULE.WORKLOAD_PROFILES["prefill"].tokens_per_rank, 4096) + self.assertEqual(MODULE.WORKLOAD_PROFILES["prefill"].api_mode, "normal") + + def test_prefill_payload_scales_with_tokens(self): + decode_route = MODULE.make_route(0, 128, 256, 8, 20260824, torch.device("cpu")) + prefill_route = MODULE.make_route( + 0, 4096, 256, 8, 20260824, torch.device("cpu") + ) + decode_bytes, _, _ = MODULE.logical_payload_bytes_per_rank( + decode_route, 0, 16, 8, 7168, "bf16", 256 + ) + prefill_bytes, _, _ = MODULE.logical_payload_bytes_per_rank( + prefill_route, 0, 16, 8, 7168, "bf16", 256 + ) + self.assertEqual(prefill_bytes, decode_bytes * 32) + + def test_percentile_interpolates(self): + self.assertEqual(MODULE.percentile([1.0, 2.0, 3.0], 0.5), 2.0) + self.assertAlmostEqual(MODULE.percentile([1.0, 2.0], 0.95), 1.95) + + def test_deepep_v2_build_lib_requires_one_extension_package(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + package = root / "build" / "lib.linux-x86_64-cpython-312" / "deep_ep" + package.mkdir(parents=True) + (package / "_C.cpython-312-x86_64-linux-gnu.so").touch() + self.assertEqual(MODULE.deepep_v2_build_lib(root), package.parent) + + second = root / "build" / "lib.second" / "deep_ep" + second.mkdir(parents=True) + (second / "_C.so").touch() + with self.assertRaisesRegex(RuntimeError, "exactly one"): + MODULE.deepep_v2_build_lib(root) + + def test_received_fp8_accepts_noncontiguous_scales(self): + adapter = object.__new__(MODULE.BackendAdapter) + adapter.arm = "uccl" + adapter.profile = MODULE.WORKLOAD_PROFILES["decode"] + adapter.hidden = 256 + observed = {} + + def cast_back(fp8, scales): + observed["fp8_shape"] = tuple(fp8.shape) + observed["scales_shape"] = tuple(scales.shape) + return torch.zeros((fp8.shape[0], 256), dtype=torch.bfloat16) + + adapter._cast_back = cast_back + fp8 = torch.zeros((2, 3, 256), dtype=torch.float8_e4m3fn) + scales = torch.arange(12, dtype=torch.float32).reshape(2, 6).t() + self.assertFalse(scales.is_contiguous()) + + received = adapter.received_as_bf16((fp8, scales), "fp8") + + self.assertEqual(observed["fp8_shape"], (6, 256)) + self.assertEqual(observed["scales_shape"], (6, 2)) + self.assertEqual(tuple(received.shape), (2, 3, 256)) + + def test_identity_expert_output_applies_local_gates(self): + adapter = object.__new__(MODULE.BackendAdapter) + adapter.arm = "uccl" + adapter.hidden = 2 + adapter.profile = MODULE.WORKLOAD_PROFILES["prefill"] + state = MODULE.DispatchState( + recv_x=torch.tensor([[2.0, 4.0]], dtype=torch.bfloat16), + recv_topk_idx=torch.tensor([[0, -1, 1]]), + recv_topk_weights=torch.tensor([[0.25, 0.5, 0.125]]), + handle=None, + ) + output = adapter.identity_expert_output(state, "bf16") + torch.testing.assert_close( + output, + torch.tensor([[0.75, 1.5]], dtype=torch.bfloat16), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py new file mode 100644 index 000000000..dad3924c5 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py @@ -0,0 +1,204 @@ +import importlib.util +from pathlib import Path +import sys +import tempfile +import unittest + + +module_path = Path(__file__).with_name("summarize_results.py") +sys.path.insert(0, str(module_path.parent)) +spec = importlib.util.spec_from_file_location("summarize_results", module_path) +assert spec and spec.loader +summary_module = importlib.util.module_from_spec(spec) +sys.modules[spec.name] = summary_module +spec.loader.exec_module(summary_module) + + +def fake_result(profile, arm, world_size, run_index, dtype, latency_ms): + digest = { + "uccl": "a", + "deepep-v1-nvshmem": "b", + "deepep-v2-gin-gda": "c", + }[arm] + tokens_per_rank = summary_module.PROFILE_CONFIG[profile]["tokens_per_rank"] + decode_logical_bytes = 22_249_472 if dtype == "fp8" else 29_360_128 + logical_bytes = decode_logical_bytes * tokens_per_rank // 128 + scaleout_bytes = logical_bytes * (world_size - 8) // world_size + tolerance = 9e-4 if dtype == "fp8" else 1e-5 + global_input_tokens = tokens_per_rank * world_size + result = { + "benchmark": "common-boundary-dispatch-combine", + "workload_profile": profile, + "backend_api_mode": ( + "elastic" + if arm == "deepep-v2-gin-gda" + else summary_module.PROFILE_CONFIG[profile]["api_mode"] + ), + "primary_metric": summary_module.PROFILE_CONFIG[profile]["primary_metric"], + "layout_in_timed_region": profile == "prefill", + "arm": arm, + "world_size_ranks": world_size, + "nodes": world_size // 8, + "gpus_per_node": 8, + "run_index_dimensionless": run_index, + "dispatch_dtype": dtype, + "tokens_per_rank": tokens_per_rank, + "global_input_tokens": global_input_tokens, + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": 20, + "measured_iterations": 100, + "route_hash_sha256": ("d" if world_size == 16 else "e") * 64, + "input_hash_sha256": ("f" if world_size == 16 else "0") * 64, + "global_valid_expert_selections": global_input_tokens * 8, + "avg_logical_payload_bytes_per_rank": logical_bytes, + "avg_scaleout_logical_payload_bytes_per_rank": scaleout_bytes, + "correctness": { + "status": "PASS", + "tolerance_dimensionless": tolerance, + "normalized_diff_dimensionless": 0.0, + }, + "runtime": { + "image_reference": f"example.invalid/{arm}@sha256:{digest * 64}", + "gpu": "NVIDIA B200", + "torch_version": "2.13.0+cu130", + "cuda_version": "13.0", + "nccl_version": [2, 29, 7], + }, + "latency_ms": {"median": latency_ms}, + "aggregate_input_tokens_per_second": global_input_tokens / (latency_ms / 1e3), + "effective_logical_gigabytes_per_second_per_rank": logical_bytes + / (latency_ms / 1e3) + / 1e9, + "effective_scaleout_logical_gigabytes_per_second_per_rank": scaleout_bytes + / (latency_ms / 1e3) + / 1e9, + "timing_boundary": summary_module.PROFILE_CONFIG[profile]["timing_boundary"], + "logical_payload_definition": summary_module.LOGICAL_PAYLOAD_DEFINITION, + } + return result + + +class SummarizeResultsTest(unittest.TestCase): + def setUp(self): + arm_latency = { + "uccl": 1.0, + "deepep-v1-nvshmem": 1.2, + "deepep-v2-gin-gda": 0.8, + } + self.results = [ + fake_result( + profile, + arm, + world, + run, + dtype, + arm_latency[arm] * (1 + (run - 2) * 0.01), + ) + for profile in summary_module.PROFILES + for arm in summary_module.ARMS + for world in summary_module.WORLD_SIZES + for run in range(1, 4) + for dtype in summary_module.DTYPES + ] + + def test_valid_matrix_and_paired_delta(self): + summary_module.validate(self.results, 3) + summary = summary_module.summarize(self.results, 3) + self.assertEqual(summary["status"], "PASS") + self.assertEqual(len(summary["cells"]), 8) + comparison = summary["cells"][0]["comparisons"]["deepep-v2-gin-gda_vs_uccl"] + self.assertAlmostEqual(comparison["median_paired_improvement_percent"], 20.0) + self.assertTrue(comparison["direction_supported"]) + + prefill = next( + cell + for cell in summary["cells"] + if cell["workload_profile"] == "prefill" + and cell["world_size_ranks"] == 16 + and cell["dispatch_dtype"] == "fp8" + ) + prefill_comparison = prefill["comparisons"]["deepep-v2-gin-gda_vs_uccl"] + self.assertAlmostEqual( + prefill_comparison["median_paired_improvement_percent"], 25.0 + ) + + def test_missing_start_is_rejected(self): + with self.assertRaises(ValueError): + summary_module.validate(self.results[:-1], 3) + + def test_route_mismatch_is_rejected(self): + self.results[0]["route_hash_sha256"] = "different" + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_runtime_mismatch_is_rejected(self): + self.results[0]["runtime"]["nccl_version"] = [9, 9, 9] + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_logical_payload_mismatch_is_rejected(self): + self.results[0]["avg_logical_payload_bytes_per_rank"] += 1 + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_derived_metric_mismatch_is_rejected(self): + self.results[0]["aggregate_input_tokens_per_second"] += 1 + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_provenance_image_mismatch_is_rejected(self): + provenance = { + "campaign_id": "test-campaign", + "created_at_utc": "2026-08-24T00:00:00Z", + "region": "ap-south-1", + "cluster": "test-cluster", + "git_commit": "a" * 40, + "images": { + arm: next( + result["runtime"]["image_reference"] + for result in self.results + if result["arm"] == arm + ) + for arm in summary_module.ARMS + }, + "comparison": { + "profiles": { + profile: { + "tokens_per_rank": summary_module.PROFILE_CONFIG[profile][ + "tokens_per_rank" + ], + "api_mode": summary_module.PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": summary_module.PROFILE_CONFIG[profile][ + "primary_metric" + ], + } + for profile in summary_module.PROFILES + }, + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": 20, + "measured_iterations": 100, + "independent_starts": 3, + }, + } + provenance["images"]["uccl"] = "example.invalid/changed@sha256:" + "9" * 64 + with self.assertRaises(ValueError): + summary_module.validate_provenance(provenance, self.results, 3) + + def test_load_results_accepts_native_diagnostic_after_json(self): + result = fake_result("decode", "deepep-v2-gin-gda", 32, 1, "fp8", 0.9) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "rank-zero.log" + path.write_text( + "ADAI_EP_RESULT " + + summary_module.json.dumps(result) + + "Elastic buffer uses 3 channels per SM\n" + ) + self.assertEqual(summary_module.load_results(Path(directory)), [result]) + + +if __name__ == "__main__": + unittest.main()