Replace EP backend comparison with Decode-like and Prefill-like B200 benchmarks

[KeitaW]

Purpose

Relates to #1234.

Replace the previous backend-native EP result tables with a common-boundary B200 microbenchmark. The retired B300, B200, H100, and Decode-only result files are removed because their latency and bandwidth fields did not share one timing boundary or byte numerator.

This microbenchmark covers synthetic Decode-like and Prefill-like expert-parallel communication.

What changed

• Add separate Decode-like and Prefill-like profiles. Decode-like uses 128 tokens/rank and the low-latency API. Prefill-like uses 4,096 tokens/rank and the normal high-throughput API, including required layout work.
• Run UCCL, DeepEP V1 NVSHMEM, and DeepEP V2 NCCL GIN with the same deterministic BF16 input, exact top-k route and weights, model shape, named nodes, and runtime stack.
• Measure one external CUDA event boundary from input readiness through required FP8 conversion or layout, dispatch, and combine completion. Each iteration reports the slowest rank.
• Derive cross-backend logical throughput from one useful-payload definition. Backend-native bandwidth fields are not used for comparison.
• Rotate backend, profile, and dtype order across 3 independent process starts per cell. Each start contains 20 warmup iterations and 100 measured iterations per dtype.
• Reject incomplete matrices, correctness failures, mutable image tags, route or input mismatches, runtime mismatches, payload mismatches, and derived metrics that disagree with latency and logical-byte accounting.
• Coordinate through the shared Lease, reject selected/protected node overlap, retain every log and rendered object, and require verified teardown before writing CAMPAIGN_COMPLETE.
• Replace the retired result set with a human-readable report and a machine-readable 2026-08-25 summary. Update the SGLang documentation that referenced the retired tables.

B200 results

Decode-like primary metric, slowest-rank dispatch-plus-combine latency:

EP size	Dispatch dtype	UCCL	DeepEP V1 NVSHMEM	DeepEP V2 NCCL GIN
16 ranks	FP8	0.5215 ms	1.0319 ms	0.6151 ms
16 ranks	BF16	0.5895 ms	1.0438 ms	0.4921 ms
32 ranks	FP8	0.7633 ms	1.5522 ms	0.9642 ms
32 ranks	BF16	0.8653 ms	1.5551 ms	0.9477 ms

Prefill-like primary metric, common logical throughput:

EP size	Dispatch dtype	UCCL	DeepEP V1 NVSHMEM	DeepEP V2 NCCL GIN
16 ranks	FP8	128.77 GB/s/rank	213.89 GB/s/rank	252.84 GB/s/rank
16 ranks	BF16	170.98 GB/s/rank	263.71 GB/s/rank	277.37 GB/s/rank
32 ranks	FP8	85.46 GB/s/rank	54.07 GB/s/rank	105.21 GB/s/rank
32 ranks	BF16	95.31 GB/s/rank	70.95 GB/s/rank	108.20 GB/s/rank

Each box contains the 3 independent process-start medians for one backend arm and workload cell. The markers show all 3 underlying values.

UCCL has the lowest Decode-like latency in 3 cells of 4 cells, while DeepEP V2 has the lowest latency for EP16 BF16. DeepEP V2 has the highest Prefill-like logical throughput in all 4 cells. The maximum run-to-run CV is 3.31 percent. All 72 scored records passed correctness and matrix validation.

The full per-start values, 95 percent bootstrap intervals, paired improvements, provenance, and scope limits are in RESULTS.md. The machine-readable summary is b200-ap-south-1-2026-08-25.json, SHA-256 16f12a0bfbbc0c9c9f93eeb19d410cc8843ed6cbc836115df851cd97996a9603.

Qualification and custody

• EKS cluster ml-clusters-shared-ap-south-1 in ap-south-1c.
• 4 named p6-b200.48xlarge nodes, each with 8 NVIDIA B200 GPUs and 8 allocatable EFA devices.
• EP16 used 2 nodes and 16 GPU ranks. EP32 used 4 nodes and 32 GPU ranks.
• PyTorch 2.13.0+cu130, CUDA 13.0, and NCCL 2.29.7 across all scored records.
• 3 backend images pinned by SHA-256 digest.
• 4 admission cases, 36 scored distributed starts, 8 admission records, 72 scored records, and 0 correctness failures.
• Exclusive shared-Lease ownership for the run. The selected node set remained disjoint from 12 protected ap-south-1 nodes, and no protected node appears in any Pod record.
• Durable SHA-256 manifest covering 573 entries. Independent verification passed all 573 entries.
• Verified teardown: owned namespace absent, 0 owned resources, empty shared-Lease holder, 4 selected nodes Ready, and 0 active GPU Pods on the selected nodes.
• The pinned UCCL normal-mode worker uses synchronized process-lifetime cleanup after result output because explicit proxy destruction invalidates the CUDA context. This happens after all timed iterations and does not change the timing boundary.

Validation

• Real B200 EKS campaign: all 4 admission cases and all 36 scored distributed starts passed.
• Preserved-log reaggregation: all 8 workload cells and 3 independent starts per arm/cell passed, with byte-for-byte reproduction of the committed machine summary.
• Local unit suite: 8 tests passed and 8 CUDA-dependent tests skipped because local PyTorch is unavailable.
• ruff check, ruff format --check, py_compile, bash -n, shellcheck, and git diff --check: passed.
• Durable teardown and checksum verification: passed.

Final directory

micro-benchmarks/expert-parallelism/ep-backend-comparison/
├── README.md
├── RESULTS.md
├── ep_benchmark.py
├── extract_results.py
├── result_io.py
├── results/
│   └── b200-ap-south-1-2026-08-25.json
├── run_ep_comparison.sh
├── run_ep_rank.sh
├── summarize_results.py
├── test_ep_benchmark.py
└── test_summarize_results.py

[Xuan-1998]

What does latency here mean? why the latency here is lower but the bandwidth below is also lower? In DeepEPv2's benchmark bandwidth = (number of bytes) / latency

[KeitaW]

Update: I removed the previous backend-native tables and reran the comparison with separate Decode-like and Prefill-like profiles.

Latency now means the slowest-rank CUDA elapsed time for one dispatch followed by one combine. The boundary starts with the BF16 input and route ready, includes required FP8 conversion, and ends after combine completion. Prefill-like also includes the dispatch layout required by the normal UCCL and DeepEP V1 APIs.

Cross-backend throughput is now calculated as common logical payload bytes divided by that same elapsed time. For a fixed profile, EP size, and dtype, every backend has the same byte numerator, so lower latency necessarily corresponds to higher logical GB/s/rank. FP8 and BF16 have different useful-byte numerators and remain separate cells. Backend-native SO, SU, and aggregate bandwidth values are no longer presented as interchangeable metrics.

The replacement B200 run contains 3 independent process starts per cell, 20 warmup iterations and 100 measured iterations per dtype/start, and 72 scored records. All records passed correctness and matrix validation; the maximum run-to-run CV is 3.31 percent. The updated method, Decode-like and Prefill-like tables, provenance, and scope limits are in the replacement report.

[whn09]

Independent run on 2× p6-b300, plus a methodology review

Thanks for putting a single common timing boundary and a SHA-256-verified input/route on this — that part is a genuine improvement over what we do internally, and we're adopting the input+route hash check for our own comparisons.

We rebuilt this harness on 2× p6-b300.48xlarge (B300, sm_103, 16 EFA devices/node, EFA installer 1.50.0) and ran your ep_benchmark.py with one addition: a --num-sms / --tokens-per-rank flag pair and a num_sms_dimensionless record field. Arms: deepep-v2-gin-gda (DeepEP at 8e7b42e, deployed with the recipe merged in #1234) and uccl. deepep-v1-nvshmem was not built — DeepEP V1 cannot go cross-node on EFA. Your 16 unit tests pass against the edited file inside our V2 image.

2 nodes is exactly EP16, so we cannot check your EP32 column or any 16→32 scaling conclusion. Everything below is 3 starts × (20 warmup + 100 measured), median of per-start medians, dtype order rotated between starts, a fresh process group and port per run.

Reviewed at head b7a5e107.

---

1. The FP8-slower-than-BF16 decode result is the harness, not DeepEP

Your EP16 decode row has V2 at 0.6151 ms FP8 vs 0.4921 ms BF16 (1.250×) while FP8 puts fewer bytes on the wire. It reproduces on our hardware at every SM count and node count:

cell (24 SM)	FP8	BF16	FP8/BF16
V2 decode, 2N / EP16	667.1 µs	517.5 µs	1.289
V2 decode, 1N / EP8	333.9 µs	171.4 µs	1.949
UCCL decode, 2N / EP16	433.8 µs	465.5 µs	0.932

(logical payload 22.25 MB/rank FP8 vs 29.36 MB/rank BF16; UCCL is the control — it shows the expected FP8 win.)

We then re-ran the same cells through DeepEP's own tests/elastic/test_ep.py — same nodes, same image, same env, same SM count, same shapes, with the dispatch mode pinned to the one ep_benchmark.py uses (expert_alignment=1, async_with_compute_stream=1, do_handle_copy=1, allocate_on_comm_stream=0, no bias). That reports per-op kernel time from the profiler instead of one wall-clock event pair. Comparable pair: dispatch_impl + dispatch_copy_epilogue_impl, and combine_impl + combine_reduce_epilogue_impl.

At kernel level the sign flips in every cell:

cell (24 SM, dispatch+combine, slowest rank)	kernel µs	your wall µs	wall/kernel	overhead
decode 1N bf16	102.4	171.4	1.67×	+69.0
decode 1N fp8	94.8	333.9	3.52×	+239.1
decode 2N bf16	466.1	517.5	1.11×	+51.5
decode 2N fp8	453.4	667.1	1.47×	+213.7
prefill 1N bf16	1206.5	1677.2	1.39×	+470.7
prefill 1N fp8	921.6	1451.6	1.57×	+529.9
prefill 2N bf16	2141.9	2451.0	1.14×	+309.1
prefill 2N fp8	1766.8	2218.7	1.26×	+451.8

Kernel FP8/BF16 is 0.926 at 1 node and 0.973 at 2 nodes. FP8 dispatch is the faster kernel (272.8 vs 286.1 µs at 2N) and combine is identical to 0.1% (182.94 vs 182.78 µs) because combine is BF16 either way. There is no FP8 penalty in the kernels.

Where it comes from. iteration() (ep_benchmark.py:536-539) times prepare_dispatch_input + dispatch + combine under one event pair. prepare_dispatch_input (:332-339) returns x unchanged for BF16, but returns self._cast_to_fp8(x) when self.is_elastic or self.profile.api_mode == "normal". For the V2 arm that resolves to deep_ep.utils.math.per_token_cast_to_fp8, which is @torch.compile(dynamic=True) (deep_ep/utils/math.py:30 in the image). Timed alone on b300 at the decode shape it costs 89.6 µs device / 77.1 µs CPU (empty-event-pair floor 7.3 µs) — CPU ≈ device means it is launch/guard-bound with the GPU idle on Python, not arithmetic. In situ it costs about twice that: the FP8-only excess is 239.1 − 69.0 ≈ 170 µs at 1 node.

Two asymmetries, both against V2:

• Decode: only the V2 arm casts. The non-elastic low-latency path returns x unquantized and lets low_latency_dispatch(use_fp8=True) quantize inside the kernel, so UCCL and V1 never pay this at all.
• Prefill: all arms cast (api_mode == "normal"), but not with the same function. V2 uses the torch.compiled deep_ep one; UCCL uses /opt/uccl/ep/bench/utils.py:720, which has no @torch.compile decorator. We verified both in the running images.

A supporting hint in your own data: the FP8 penalty is +123.0 µs at EP16 but only +16.5 µs at EP32 (0.9642 vs 0.9477 ms). A property of the FP8 dispatch kernels would not shrink 7× when you add two nodes.

Fix (one line). x is fixed across the timed loop, so the cast can be hoisted out of iteration() entirely and done once before timing. If you'd rather keep conversion inside the boundary — a defensible choice, since it's real work in a real serving path — then use the same implementation for every arm and say so, because right now the cell measures three different quantizers.

Consequence for the headline: V2's FP8 dispatch+combine kernels are 453.4 µs against UCCL's 433.8 µs wall-clock on the same box, so the 1.54× FP8 decode gap is mostly harness. We can't close this properly — test_ep.py is DeepEP's bench and yields no kernel-only UCCL number — but the current figure is not a backend result.

---

2. The "95% bootstrap CI" is identically [min, max] of the 3 starts

bootstrap_median_ci (summarize_results.py:71-78) resamples 3 values with replacement and returns percentile(medians, 0.025), percentile(medians, 0.975). For n=3 the bootstrap median can only take the 3 sample values, with

P(median = min) = P(median = max) = 7/27 = 0.259

Since 0.259 > 0.025, the 2.5th percentile is always the sample min and the 97.5th always the sample max — for any data, any seed, and any number of BOOTSTRAP_SAMPLES. The interval is [min, max] by construction and its width carries no confidence information.

The published tables show exactly this signature: in all 40 rows (24 primary + 16 paired) the point estimate is strictly inside its interval and never equal to a bound.

Two knock-ons:

• direction_supported = stable and (ci_low > 0 or ci_high < 0) (:486) reduces to "all 3 paired deltas had the same sign." Under a null of independent per-start sign that is p = 0.25, not 0.05. All 16 directions are currently marked supported.
• paired (:455-468) matches arm A's start k with arm B's start k. But starts are independent process launches, and backend/dtype order rotates across starts, so "start 1" is not the same condition for both arms. The pairing is arbitrary and buys no variance reduction.

Fix: with n=3, report min / median / max explicitly and drop the CI and "direction supported" language; or raise the start count to where a percentile bootstrap means something (≥20); or use a test appropriate for n=3 and state the exact p.

---

3. The per-selection logical numerator over-counts DeepEP by a node-count-dependent factor — it inverts the 16→32 scaling sign

README.md:45: "Scale-out logical bytes include only assignments whose destination expert is on another node," and each assignment contributes a full tensor. But DeepEP's internode path sends one copy per destination node and NVLink-forwards to the local experts. So the numerator over-counts by (remote assignments) / (distinct remote nodes) — and that factor changes with node count.

Your own table pins it exactly:

	scale-out / total	remote assignments	remote nodes	over-count
EP16 (2 nodes)	126.42 / 252.84 = 0.500	4 of 8	1	4×
EP32 (4 nodes)	78.91 / 105.21 = 0.750	6 of 8	3	2×

Correcting each cell by its own factor:

	reported scale-out	corrected
EP16 FP8 prefill	126.42 GB/s/rank	31.6 GB/s/rank
EP32 FP8 prefill	78.91 GB/s/rank	39.5 GB/s/rank	+25%

So real per-rank scale-out bandwidth rises 25% from EP16 to EP32, while the reported metric falls 58% (252.84 → 105.21 GB/s/rank). The conclusion flips sign. Because the distortion is node-count dependent, this metric is not comparable along the EP-size axis at all — which is the only axis the prefill table has.

Your own Input throughput (tokens/s) column already tells a much more honest story across the same two cells: 23,272,859 → 19,368,582 tokens/s, a 16.8% drop, not 58%.

Fix: make latency (or the aggregate tokens/s you already compute) the prefill primary metric, or use a denominator that accounts for per-node deduplication. The "these are logical efficiency metrics, not observed wire bandwidth" disclaimer covers the absolute level but not the cross-EP-size comparison, which is where it actually misleads.

---

4. num_sms is uncontrolled, unrecorded, and derived from one NIC's bandwidth

You never pass num_sms, so ElasticBuffer auto-sizes. On b300 that does work at 2 nodes on EFA (at pin 8e7b42e, get_rdma_gbs reads sysfs before the ibstat fallback — which matters, because ibstat -l on p6-b300 lists only the two DOWN ConnectX-7 devices and never the 16 EFA ones). It prints:

rdma_gbs=50.0 … num_sms=64
[WARN] DeepEP GIN signal budget reduced the number of channels per SM from 4 to 3

50 GB/s is one EFA device's 400 Gb/s. A b300 GPU has two NICs, so the real per-GPU figure is 100 GB/s; the heuristic is fed half the bandwidth. Separately, prefer_overlap_with_compute=False floors the result at 64, so on this hardware the wrong input happens not to change the outcome — but the value is neither recorded in the results nor listed in the controls table, and it is the single largest knob we found.

Measured SM sensitivity on b300, 2N/EP16:

profile	dtype	12 SM	24 SM	48 SM	64 SM
decode	bf16	569.3	517.5	499.8	502.4
decode	fp8	710.9	667.1	659.6	642.5
prefill	bf16	2843.9	2451.0	2196.7	2134.2
prefill	fp8	2569.1	2218.7	1910.0	1894.9

(µs, medians of 3 starts.) Decode is essentially SM-flat above 24; prefill is not — 24 SM is 14.9% (bf16) / 17.1% (fp8) off the best point. The knee is shape-dependent, so there is no single value that could be hard-coded instead. Suggest recording num_sms in every result and adding it to the controls table.

---

5. EP_BUFFER_DEBUG=1 is set on the V2 arm only, and it is not free

run_ep_rank.sh:40 enables it for V2 and nowhere else; FI_EFA_USE_HW_CNTR=1 (:37) likewise. EP_BUFFER_DEBUG makes csrc/elastic/buffer.hpp printf ~24 numbers per dispatch inside the CPU-wait path, and you run do_cpu_sync=True.

A/B on b300, 2N/EP16/24 SM:

dtype	debug on	debug off	cost
fp8	667.1 µs	648.2 µs	+2.9%
bf16	517.5 µs	504.3 µs	+2.6%

Small in absolute terms, but ~5× the run-to-run CV of those cells (0.36% / 0.56%) and charged to one arm only. Likely worse on your side: ours goes to a file, yours to a Kubernetes pod log pipe. (FI_EFA_USE_HW_CNTR=1 we measured as neutral.)

---

6. runtime.nccl_version reports torch's constant, not the loaded library

ep_benchmark.py:641 records list(torch.cuda.nccl.version()), which is torch's compile-time constant. In our running V2 container /proc/self/maps shows /opt/nccl/build/lib/libnccl.so.2.31.2 actually loaded, and DeepEP itself prints DeepEP initialized with NCCL version: 2.31.2 (loaded library) while the record says (2, 28, 9). The field is misleading precisely in the source-built-NCCL image that GIN requires. Reading it from the loaded object (as DeepEP does) fixes it.

---

7. summarize_results.py cannot summarize anything smaller than a 4-node campaign

WORLD_SIZES = (16, 32) (:20) and validate() (:102-110) hard-require the full 3 arms × 2 world sizes × 2 dtypes × 2 profiles × 3 starts matrix, so a 2-node campaign is rejected outright rather than partially summarized. Anyone with 2 nodes — which is most people who will want to re-run this — has to write their own aggregator. We did, and we reused your result_io.load_result_log, which is genuinely necessary: a plain json.loads on a result line with native diagnostics appended raises Extra data. Nice catch, worth keeping.

---

8. Deleted material the "口径 doesn't match" rationale doesn't cover

RESULTS-p5.md and nccl-alltoall.yaml are deleted outright. The new harness maxes at 4 nodes, so several results in there can never be re-derived from it:

• the 32-node / 256-rank runnability result, and the source-level rank caps behind it (deep_ep.cpp:158, the 64 < cap ≤ 128 PE low-latency limit, the nvshmemi_process_multisend_rma -11 (EAGAIN) fan-out limit at ≥128 PEs on EFA);
• the NCCL all-to-all reference baseline — the only backend-independent absolute calibration in the whole comparison, whose loss compounds finding 3;
• your own note that DeepEP's default config uses only 20 SMs, i.e. that any default-config Blackwell number is a floor — which is finding 4.

benchmarks/README.md still keeps "An EP16 result cannot decide an EP32 or EP64 deployment" while now pointing at a matrix that is only EP16/EP32. (No dangling links, though — we checked.) Suggestion: keep the old file as a clearly-dated historical document rather than deleting it.

---

9. Smaller notes

• The route is perfectly balanced. make_route (:94) is (t*17 + k*31 + seed) % 256 with a stride coprime to num_experts, so every token hits exactly 8 distinct experts and every expert carries identical load. V2's own get_theoretical_num_sms docstring warns it assumes a balanced gate and must not be used for V3's group-limited gate. test_ep.py has --unbalanced-ratio / --masked-ratio; rankings may not survive skew. We haven't tested this axis yet either.
• API asymmetry in the decode cell. The PR body says decode uses "the low-latency API" for all arms; the README's API table is the correct one. V2 has only ElasticBuffer on EFA, so that cell pits UCCL/V1 purpose-built LL kernels against V2's general kernels. It penalises V2 and deserves a stated caveat.
• Every iteration is a cold start. The per-iteration end.synchronize() + all-reduce + .item() (:554-559) measures single-shot latency, not pipelined steady state. Defensible, but it should be labelled — it is a large part of the wall-vs-kernel gap in finding 1, and it means these absolutes cannot be compared against kernel-time figures from any other benchmark.
• No backend source commits in provenance. Only image digests, torch/cuda/nccl, and your repo's git_commit. We have measured ~1.8× swings in 2-node decode dispatch between two DeepEP V2 commits a few weeks apart on the same nodes, so a digest alone doesn't let a reader know which DeepEP produced a number. Recording the DeepEP/UCCL SHAs and the EFA installer version would be cheap.
• combine_input is all zeros in the timed loop (:534). Harmless as configured (BF16 combine, no logfmt), but it would matter the moment a compressed combine path is enabled.
• CUDA_DEVICE_MAX_CONNECTIONS=1 (run_ep_rank.sh:23) is set for all three arms and may serialise comm/compute given async_with_compute_stream=True.
• Controls table (README.md:23-38) omits SM/channel count, QP count, route distribution character, and the per-arm env vars from finding 5.
• Replicate outliers. Three of our single-node decode cells contain one replicate ~1.6× the others (277.5 vs ~170 µs; 526.6 vs ~331 µs). With n=3 that single replicate drives the run-to-run CV to 23-26%, so your 5% gate would reject the cell rather than report it. Medians across starts are unaffected, which is why we report those — but it's worth knowing the gate can silently drop cells rather than flagging a single bad start.

---

What reproduced

Your direction holds on different hardware with your own harness: UCCL wins decode, DeepEP V2 wins prefill. b300, 2N/EP16/24 SM, V2/UCCL ratio (>1 = V2 slower):

profile	BF16	FP8
decode	1.112	1.538
prefill	0.431	0.388

So "UCCL wins decode" survives — the BF16 figure is a real kernel result. The FP8 decode margin is mostly finding 1, and the prefill margin is larger than you report (+96.02% FP8 / +62.45% BF16 in your EP16 table vs 2.58×/2.32× here). One more note on prefill: FP8's advantage is larger at kernel level than a wall-clock boundary shows. Our wall-clock FP8/BF16 ratios are 0.87-0.91, while the kernels give 0.76 (1N) / 0.83 (2N) — the cast sits on the FP8 side in both arms here, so it compresses the measured gap.

Happy to share the full kit (launchers, table generator, raw logs) if that's useful, or to re-run any cell you'd like on this hardware.

[whn09]

Follow-up to my earlier review, on one point only: the decode column depends on which DeepEP tree the V2 arm is built from, and with two small kernel-side patches the ranking reverses. Same 2 × p6-b300, same harness, same nodes.

Disclosure: the two DeepEP PRs below are mine, so please weight this accordingly — everything here is reproducible from the recipe at the bottom, including the negative control.

The pin

This is not a stale-pin report. Our V2 arm is built at 8e7b42e, which was tip of amazon-contributing/DeepEP main when we built it and is still only 1 commit behind today's main (9c1f251, an unrelated QP-range clamp) — so "track main" does not fix decode; the fix is in two still-open PRs. Which commit your images contain we cannot tell, since the PR records no backend SHA (section 9 of the earlier review) — 8e7b42e is what the #1234 recipe resolved to for us, and the mechanism below holds for every mainline commit to date.

On that tree kNumParts is chosen from the GIN indexed-signal budget alone and is never lowered because a channel holds too few tokens — so at 128 tokens/rank a decode channel gets described as more parts than it has tokens for, and those tokens leave as separate single-token puts. That budget is loosest exactly at decode shapes, so decode lands on the worst end of the axis with no way to opt out.

amazon-contributing/DeepEP #2 adds the missing clamp; #1 is the JIT plumbing it stacks on. compare/8e7b42e...5a594a5 is ahead 2, behind 0 and touches two files, so the two images below differ by exactly those PRs and nothing else.

Effect on this PR's own numbers

Your ep_benchmark.py, 2 nodes = EP16, 24 SM, EP_BUFFER_DEBUG off on both arms, 3 starts × (20 + 100), medians. The UCCL denominator was re-measured under these same conditions rather than reused:

V2 arm	fp8 µs	bf16 µs	V2/UCCL fp8	V2/UCCL bf16
DeepEP 8e7b42e (mainline, unpatched)	647.9	504.4	1.489	1.064
DeepEP 5a594a5 (#1 + #2)	554.2	392.9	1.274	0.829
5a594a5 + EP_MIN_TOKENS_PER_PART=1 (clamp off, same image)	649.0	504.9	1.492	1.065
UCCL	435.1	474.2	—	—

So BF16 decode goes from V2 losing by 6.4% to V2 winning by 17%. The clamp-off control lands on the old image to within 0.1%, in the same binary, so this is the clamp rather than a build or environment difference.

Through DeepEP's own test_ep.py on the same cell, the effect is confined to dispatch: 283.35 → 176.84 µs (−37.6%), with combine flat to +0.7%. Prefill is unchanged (within 0.23%, run-to-run CV ≤ 0.9%), so nothing is being traded away. The clamp also flattens the SM curve — patched, 12 SM equals 24 SM; unpatched, dropping to 12 SM costs 11%.

And the FP8 term from section 1, now measured exactly

In this same cell the wall-clock-minus-kernel term is 33.0 µs for BF16 against 201.8 µs for FP8 (slowest rank, dispatch + combine, same code, same image). That 168.8 µs is prepare_dispatch_input's @torch.compiled per_token_cast_to_fp8, which only the elastic arm pays inside the timed region. 554.2 − 168.8 = 385.4, right next to BF16's 392.9 — the two dtypes agree once it is removed, which is what you would expect since FP8 moves fewer bytes.

With the clamp in and the cast hoisted out of the timed region, V2 wins decode in both dtypes on this hardware.

Suggestion

No change is needed to your harness for the first part — only the DeepEP tree the V2 arm is built from. #1 and #2 are both still open, so the low-friction order is probably:

1. Independently of the PRs, hoist the FP8 cast out of the timed region (or quantize once outside the loop). That is orthogonal and fixes the fp8 column regardless of any pin.
2. Once Fix numpy version in ddp test case requirements #2 lands, bump the V2 arm's DeepEP ref and re-run decode. If you want to check it before then, you do not need a second image: build once at 5a594a5 and A/B EP_MIN_TOKENS_PER_PART=1 against unset, which is the exact pre-patch geometry in the same binary.

One reproduction trap, if you do build two images. DeepEP's JIT cache key is name$$compiler$$flags$$code where code is the generated wrapper — the impl headers are only #included, so their content is not in the key, and with the env knobs unset flags is byte-identical between the two trees. Sharing one EP_JIT_CACHE_DIR (or one host mount for ~/.deep_ep) between the images makes the patched one load the old cubin, and the clamp measures as a clean no-op. One cache directory per image.

Caveats

• 2 nodes = EP16 only. Do not extrapolate to your EP32 column: on H200 with NCCL_GIN_TYPE=5 we measured the same clamp at 4 nodes and the decode-dispatch win drops to about −8%, because the indexed-signal budget tightens with rank count.
• On sm_103 the win is entirely Fix numpy version in ddp test case requirements #2. Bump numpy from 1.21.5 to 1.22.0 in /3.test_cases/4.DDP #1's EP_NUM_SUB_PARTS=1 is neutral to slightly negative here (397.0 vs 392.9 µs), unlike on H200 where it stacks further.
• Arms were run in blocks rather than interleaved within a rep; the clamp-off control ran at the end of each block and still matched the base image, which bounds drift over a block.