From 46793a8d47446418333a501540c7cbb1f437566d Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 17:28:59 +0000 Subject: [PATCH 01/19] feat(ep): add DeepEP V2 B200 comparison --- .../ep-backend-comparison/README.md | 88 ++++++------ .../ep-backend-comparison/RESULTS-b200.md | 135 ++++++++++++++++++ .../ep-backend-comparison/RESULTS-p5.md | 4 +- .../ep-backend-comparison/RESULTS.md | 4 +- .../ep-backend-comparison/collect_results.py | 96 +++++++++++-- .../deepep_v2_selected_cases.py | 96 +++++++++++++ .../ep-backend-comparison/env_vars.example | 10 +- .../test_collect_results.py | 55 +++++++ 8 files changed, 425 insertions(+), 63 deletions(-) create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index f7e50bde4..2e5f89d86 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -1,39 +1,33 @@ -# Expert-Parallelism Backend Comparison (NCCL vs UCCL vs NVSHMEM) on EKS +# Expert-Parallelism Backend Comparison on EFA -Head-to-head MoE dispatch/combine micro-benchmark across three communication backends, run -at the **same EP world size** on the same GPU nodes (designed for 8× `p6-b300.48xlarge`, -64 ranks; also exercised at **32 nodes / 256 ranks** — see -[Scaling beyond 8 nodes](#scaling-beyond-8-nodes-256-rank-findings) for the hard backend -limits that appear there). This directory is the orchestration layer; the benchmarks -themselves live in the sibling directories. +Head-to-head MoE dispatch/combine micro-benchmark across 3 EP backends, with raw NCCL all-to-all retained as a separately labeled transport reference. Runs within a result set use the same EP world size, problem shape, common dependency stack, and GPU nodes. The latest campaign adds DeepEP V2 over NCCL GIN EFA-GDA on B200 at EP16 and EP32. The historical B300 campaign used 64 ranks and also exercised a 256-rank transport reference. + +This directory provides comparison orchestration and log collation. Backend build and launch harnesses live in sibling directories or the linked DeepEP V2 contribution. | Config | What it is | Source benchmark | |---|---|---| -| **NCCL** (baseline) | Raw all-to-all over EFA. The transport-level **reference ceiling** — moves bytes, but does *not* do token routing or combine-reduction. | [`nccl-alltoall.yaml`](nccl-alltoall.yaml) (built from [`../../nccl-tests`](../../nccl-tests)) | +| **NCCL** (reference) | Raw all-to-all over EFA. It moves bytes but does *not* do token routing or combine-reduction, so it is not an EP-backend row. | [`nccl-alltoall.yaml`](nccl-alltoall.yaml) (built from [`../../nccl-tests`](../../nccl-tests)) | | **UCCL** | DeepEP-style dispatch/combine over the UCCL all-to-all backend. | [`../uccl-ep-benchmark/kubernetes`](../uccl-ep-benchmark/kubernetes) | -| **NVSHMEM** | DeepEP dispatch/combine over NVSHMEM (libfabric/EFA). | [`../deepep-benchmark/kubernetes`](../deepep-benchmark/kubernetes) | +| **DeepEP V1 NVSHMEM** | DeepEP V1 dispatch/combine over NVSHMEM libfabric/EFA. | [`../deepep-benchmark/kubernetes`](../deepep-benchmark/kubernetes) | +| **DeepEP V2 NCCL GIN** | DeepEP V2 `ElasticBuffer` dispatch/combine over NCCL GIN EFA-GDA. | [DeepEP V2 benchmark PR 1234](https://github.com/awslabs/awsome-distributed-ai/pull/1234) and [`deepep_v2_selected_cases.py`](deepep_v2_selected_cases.py) | -> **Why no "DeepEP without a backend"?** DeepEP at the pinned commit (`567632d`, pre-EPv2) has -> no internode dispatch/combine path without a transport backend, so a literal "no-backend -> DeepEP on 8 nodes" does not exist. The NCCL all-to-all stands in as the neutral baseline and -> is labelled as a transport ceiling, not as an equal dispatch/combine number. +> **Backend naming.** DeepEP V1 at commit `567632d` has no internode path without an external transport, so this comparison names its NVSHMEM transport explicitly. DeepEP V2 is a distinct NCCL GIN backend and does not replace the raw NCCL transport reference. Raw NCCL remains context only and is never treated as an equal dispatch/combine measurement. ## Matched configuration (what makes the numbers comparable) -All runs use the **same EP problem size** — otherwise the table is meaningless: +Compare backends only within the same platform campaign. B200, B300, and H100 measurements are kept on separate result pages. The latest B200 campaign used the following matched shapes: | Parameter | Value | |---|---| -| World size | 8 nodes × 8 GPU = **64 ranks** | +| World size | EP16: 2 nodes × 8 GPUs = 16 ranks; EP32: 4 nodes × 8 GPUs = 32 ranks | | `num-tokens` | 4096 (internode) / 128 (low-latency) | | `hidden` | 7168 | | `num-topk` | 8 | -| `num-experts` | 256 (divides evenly across 64 ranks) | -| dtype | bf16 | +| `num-experts` | 256 | +| dispatch dtype | FP8 and BF16 | +| combine dtype | BF16 headline; DeepEP V2 also prints FP8 diagnostic data | -The UCCL manifests bake these args into the `python3 bench/test_*.py` invocation; the DeepEP -test hard-codes its config in-image. **Before running, confirm the DeepEP image's config is the -anchor** and align UCCL to it: +The historical B300 and H100 result pages use 4-node and 8-node topologies with the same token, hidden-size, top-k, and expert values. The UCCL manifests bake these arguments into the `python3 bench/test_*.py` invocation; the DeepEP V1 test hard-codes its config in-image. Before running V1, confirm the image config and align UCCL to it: ```bash # Read the DeepEP test config from the NVSHMEM image and match UCCL's CLI args to it. @@ -48,10 +42,12 @@ If the DeepEP values differ from 4096/7168/8/256, edit the bench args in - EKS cluster with EFA + GPU nodes; NVIDIA device plugin + AWS EFA device plugin; Kubeflow MPI Operator (`kubectl get crd mpijobs.kubeflow.org`). See each benchmark's `kubernetes/README.md`. - The container images in ECR: - - NVSHMEM: `../deepep-benchmark/deepep.Dockerfile` (CUDA 13, `sm_90`+`sm_100`) + - DeepEP V1 NVSHMEM: `../deepep-benchmark/deepep.Dockerfile` (CUDA 13, `sm_90`+`sm_100`) - UCCL: `../uccl-ep-benchmark/uccl-ep.Dockerfile` (CUDA 13; pinned UCCL commit; Hopper + Blackwell via PTX) + - DeepEP V2 NCCL GIN: the standalone build and launch workflow from [PR 1234](https://github.com/awslabs/awsome-distributed-ai/pull/1234), with all revisions pinned to the campaign being reproduced - NCCL: **reuse the NVSHMEM/DeepEP image** — it already builds `/opt/nccl-tests/build/alltoall_perf` with `sm_100` gencode, so no separate `nccl-tests` build is needed for the baseline. +- For EFA-GDA, verify the host EFA driver and GDRCopy requirements documented by the DeepEP V2 harness before scheduling a multi-node run. ## Account / cluster safety (run first) @@ -62,18 +58,17 @@ kubectl get nodes -l node.kubernetes.io/instance-type=p6-b300.48xlarge # confi kubectl get crd mpijobs.kubeflow.org # confirm MPI Operator ``` -## Run order (serial — each config needs all 8 nodes) +## Run order + +Run backends serially on the same named node set. Smoke each image on 1 node first. An intranode smoke is NVLink-only and is not a scored EFA comparison row. The commands below cover the existing DeepEP V1, UCCL, and raw NCCL EKS manifests. Run DeepEP V2 with the linked harness and the matched wrapper in this directory, using 4,096 tokens/rank for prefill and 128 tokens/rank for decode. -**Smoke first.** Before any 8-node job, run the single-node `test-intranode.yaml` for each EP -image. It validates the image, that `sm_100` actually runs on B300, and the launch path in -minutes instead of failing eight nodes deep. Intranode is NVLink-only (same for every backend), -so it is a smoke test, not a comparison row. +`deepep_v2_selected_cases.py` matches the synthetic DeepEP V2 revision pinned in [`RESULTS-b200.md`](RESULTS-b200.md). Review its upstream `Namespace` fields before using it with a different DeepEP V2 revision. ```bash cp env_vars.example env_vars # then edit image URIs / topology source env_vars -# 1) NVSHMEM (DeepEP) +# 1) DeepEP V1 NVSHMEM ( cd ../deepep-benchmark/kubernetes IMAGE_URI=$NVSHMEM_IMAGE_URI NUM_NODES=$NUM_NODES \ envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' < test-internode.yaml | kubectl apply -f - @@ -85,7 +80,17 @@ source env_vars envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < test-internode.yaml | kubectl apply -f - # ...then test-low-latency.yaml (already pinned to --num-experts=256) ) -# 3) NCCL baseline (reuses the DeepEP image's alltoall_perf) +# 3) DeepEP V2 NCCL GIN +# Run an INFO-level admission first and require NCCL_GIN_TYPE=5, a successful +# Libfabric_GDAKI context, a nonzero GIN layout, and bidirectional EFA deltas. +# Use deepep_v2_selected_cases.py inside the pinned V2 image, then repeat with +# NCCL_DEBUG=WARN for the scored run. +# The multi-node launcher must provide WORLD_SIZE, RANK, MASTER_ADDR, and +# MASTER_PORT to each node. Inside each node's container, run one of: +# python3 deepep_v2_selected_cases.py --num-processes=8 --num-tokens=4096 --hidden=7168 --num-topk=8 --num-experts=256 +# python3 deepep_v2_selected_cases.py --num-processes=8 --num-tokens=128 --hidden=7168 --num-topk=8 --num-experts=256 + +# 4) NCCL reference (reuses the DeepEP V1 image's alltoall_perf) IMAGE_URI=$NCCL_IMAGE_URI \ envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < nccl-alltoall.yaml | kubectl apply -f - ``` @@ -117,27 +122,18 @@ python3 collect_results.py \ --nvshmem-lowlat nvshmem_lowlat.log \ --uccl-internode uccl_internode.log \ --uccl-lowlat uccl_lowlat.log \ + --deepep-v2-prefill deepep_v2_prefill.log \ + --deepep-v2-decode deepep_v2_decode.log \ --nccl nccl_alltoall.log ``` -The parser reports, for internode, the **RDMA** leg of the "Best dispatch/combine" line (the -cross-node bottleneck — *not* the intra-node NVL number printed on the same line), and for the -NCCL baseline the busbw **at the EP per-rank payload size** (`num_tokens*hidden*2`, ~56 MiB -target; the power-of-two sweep reports the nearest sampled row, 64 MiB) as -well as the asymptotic peak. Record the table in [`RESULTS.md`](RESULTS.md) with image tags, -date, and any config deltas. Eyeball one real launcher log against the parser before trusting it. +For DeepEP V1 and UCCL internode logs, the parser reports the RDMA leg of the `Best dispatch/combine` line, not the intra-node NVL value printed beside it. For DeepEP V2, it reports rank-zero SO bandwidth, SU bandwidth, and latency for each dispatch dtype and operation. For raw NCCL, it reports bus bandwidth at the EP per-rank payload size, approximately 56 MiB, plus the asymptotic peak. Eyeball one real launcher log against the parser before trusting it. -Results are recorded per platform: [`RESULTS.md`](RESULTS.md) (B300) and -[`RESULTS-p5.md`](RESULTS-p5.md) (P5/H100). For other instance types set `INSTANCE_TYPE` and -`EFA_PER_NODE` accordingly (e.g. `p5.48xlarge` exposes **32** EFA NICs vs **16** on `p6-b300`). +Results are recorded per platform: [`RESULTS-b200.md`](RESULTS-b200.md) (B200 with DeepEP V2), [`RESULTS.md`](RESULTS.md) (B300 historical), and [`RESULTS-p5.md`](RESULTS-p5.md) (P5/H100 historical). For other instance types set `INSTANCE_TYPE` and `EFA_PER_NODE` to the devices actually exposed by the target nodes. ## Scaling beyond 8 nodes (256-rank findings) -The full matrix was pushed to 16 and 32 nodes (128 / 256 ranks) on a 32× `p6-b300` Capacity -Block on 2026-07-14. **Every DeepEP-class kernel hits a hard implementation limit between -65 and 256 ranks; only the NCCL reference runs at 256.** Details and the per-limit source -citations are in [`RESULTS.md`](RESULTS.md) ("32 / 16 nodes" section). Operational notes for -anyone re-running at scale: +The historical UCCL and DeepEP V1 matrix was pushed to 16 and 32 nodes, or 128 and 256 ranks, on a 32-node `p6-b300.48xlarge` Capacity Block on 14 July 2026. Those V1-era kernels hit implementation limits between 65 and 256 ranks; only the raw NCCL reference ran at 256 ranks. These findings do not establish a DeepEP V2 limit. Details and per-limit source citations are in [`RESULTS.md`](RESULTS.md). Operational notes for rerunning the V1-era matrix at scale: - **HT internode**: DeepEP asserts at >160 ranks (`NUM_MAX_NVL_PEERS 8 × NUM_MAX_RDMA_PEERS 20`, `kernels/configs.cuh`) and its stock combine tuning tables already abort at 16 nodes; UCCL @@ -159,17 +155,15 @@ anyone re-running at scale: ## Caveats -- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure transport throughput; the - EP dispatch/combine numbers carry routing + reduction overhead, so they should sit *below* the - NCCL ceiling. Compare against the **matched-size** busbw, not the asymptotic peak. +- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure transport throughput, while EP dispatch/combine includes routing and reduction and uses backend-specific bandwidth accounting. Treat matched-size NCCL bus bandwidth as transport context, not as a hard ceiling or an EP-backend row. - **Internode = RDMA leg.** DeepEP/UCCL print both an RDMA (cross-node) and an NVL (intra-node) bandwidth on the same line; only the RDMA number reflects the inter-node transport being compared. +- **DeepEP V2 accounting differs.** V2 prints SO and SU bandwidth plus per-operation latency. SO is the cross-node leg, but it is not numerically interchangeable with the V1/UCCL RDMA accounting. Use latency as the primary cross-backend metric and retain backend-native bandwidth as directional evidence. - **`num-experts` must divide the world size.** Both tests assert `num_experts % num_ranks == 0`. At 8 nodes (64 ranks) the comparison uses 256 (= 4/rank). The DeepEP low-latency default (288) is not divisible by 64 and must be overridden (see the run-order note). -- **Toolchain.** All three images are CUDA 13 (NVSHMEM/NCCL share the DeepEP image; UCCL is - CUDA 13 per `uccl-ep.Dockerfile`), so there is no CUDA skew across backends. +- **Toolchain.** Verify the exact toolchain per result page. The B200 comparison used CUDA 13.0.3 and the same vLLM wheel in all 3 EP-backend images. The historical raw NCCL reference shares the DeepEP V1 image. - **UCCL bench scripts** are pulled from upstream `uccl/ep/bench` at image-build time and pinned via `UCCL_COMMIT`. If upstream renames CLI flags, adjust the bench args in the UCCL manifests. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md new file mode 100644 index 000000000..f7ab70cc9 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md @@ -0,0 +1,135 @@ +# EP-Backend Comparison Results: B200 with DeepEP V2 + +This page records the matched direct expert-parallelism campaign completed on B200 in `ap-south-1`. It adds DeepEP V2 over NCCL GIN EFA-GDA to the UCCL and DeepEP V1 comparison. These results are separate from the historical [B300 results](RESULTS.md) and [P5/H100 results](RESULTS-p5.md). Results from different GPU generations or campaigns are not combined. + +## Result summary + +DeepEP V2 had the lowest measured decode dispatch-plus-combine latency in all 4 B200 cells. Relative to UCCL, its latency was 19.04 percent lower for EP16 FP8, 44.70 percent lower for EP16 BF16, 6.82 percent lower for EP32 FP8, and 18.88 percent lower for EP32 BF16. Relative to DeepEP V1, the reductions ranged from 52.22 percent to 56.34 percent. + +This does not establish an overall backend winner. Each cell has 1 independent benchmark start, so run-to-run variance and coefficient of variation are unavailable. This scored B200 comparison contains direct EP tests only; it produced no DeepEP V2 serving throughput, TTFT, TPOT, E2E latency, or NIXL result. DeepEP V2 also used substantially more HBM for prefill than the other 2 backends. + +The synthetic DeepEP source contains PR 5, but EP16 and EP32 do not validate its greater-than-20-NVLink-domain scale path. The independent 256-rank PR 5 gate was not completed and no PR 5 scale-success claim is inferred from these performance cells. + +## Environment and provenance + +| Field | Value | +|---|---| +| Measurement period | 23 August 2026 through 24 August 2026 | +| Region and Availability Zone | `ap-south-1`, `ap-south-1c` | +| EKS cluster | `ml-clusters-shared-ap-south-1` | +| Hardware | `p6-b200.48xlarge`, 8 B200 GPUs and 8 EFA devices per node | +| Fleet census | 36 Ready B200 nodes | +| EP16 topology | 2 nodes, 16 ranks | +| EP32 topology | 4 nodes, 32 ranks | +| Prefill shape | 4,096 tokens/rank, hidden size 7,168, top-k 8, experts 256 | +| Decode shape | 128 tokens/rank, hidden size 7,168, top-k 8, experts 256 | +| Selected operations | FP8 dispatch, BF16 dispatch, BF16 combine | +| Independent starts | 1 start per cell | +| Correctness | 12 of 12 backend cells passed | + +All 3 scored images used the same vLLM wheel with SHA-256 `60800409bd2ede00aed65ac7b463541cbdc382716715e21c5bbef87d6f2423c4`. The common stack was vLLM commit `185cada36bb25aa55f762d004d54c5ca1e3fc753`, CUDA 13.0.3, PyTorch `2.13.0+cu130`, NCCL tag `v2.31.2-1` with runtime version 23,102 dimensionless, aws-ofi-nccl 1.21.1, libfabric API 2.6.0, and NIXL 1.3.2. + +| Backend | Backend revision | Scored image digest | +|---|---|---| +| UCCL | `0dc87eb3b40c372a16b70ef320f37daaa5299ca7` | `sha256:d2e3e500524e168a3d577f059325df34872dde850bf77505c080daabfb9f8288` | +| DeepEP V1 NVSHMEM | DeepEP `567632dd59810d77b3cc05553df953cc0f779799`, NVSHMEM 3.7.0 | `sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac` | +| DeepEP V2 NCCL GIN | Synthetic commit `b56ebf8bb4ece24cd78aa8c12550b24e35ac255b` | `sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb` | + +The DeepEP V2 source was synthesized reproducibly from base `02efc268a37802fc00812ede8f5ad7f535ceea0e`, PR 3 head `dd0f87261a80cf0ce8aa66e4ab2041843851d810`, and PR 5 head `2542d9641f2ec280213e875feb04be7862dda57c`. The final Git tree was `4da3a118b3b316de79b5af2b4770646035dc5802`. The image contained B200-specific `sm_100` and `sm_100a` code. + +Before measurement, an INFO-level EP16 admission ran on both nodes with `NCCL_GIN_TYPE=5` and `NCCL_SYM_GIN_KERNELS_ENABLE=0`. Logs showed the aws-ofi-nccl `Libfabric_GDAKI` version 14 dimensionless plugin, GPU Direct RDMA enabled on EFA HCAs, nonzero GIN layouts, and balanced cluster EFA TX and RX deltas of 235,008 bytes each. Every scored cell also recorded nonzero EFA RDMA writes, so the cross-node path was measured rather than inferred from topology. + +## Decode latency comparison + +Lower latency is better. UCCL and DeepEP V1 print an aggregate dispatch-plus-combine latency. DeepEP V2 prints dispatch and combine separately, so the V2 value below is their sum for the same dispatch dtype. + +| EP ranks | Dispatch dtype | UCCL latency | DeepEP V1 latency | DeepEP V2 latency | V2 vs UCCL | V2 vs V1 | +|---:|---|---:|---:|---:|---:|---:| +| 16 ranks | FP8 | 475.540 us | 881.700 us | **384.994 us** | 19.04 percent lower | 56.34 percent lower | +| 16 ranks | BF16 | 710.180 us | 886.920 us | **392.738 us** | 44.70 percent lower | 55.72 percent lower | +| 32 ranks | FP8 | 693.840 us | 1,403.720 us | **646.522 us** | 6.82 percent lower | 53.94 percent lower | +| 32 ranks | BF16 | 844.520 us | 1,433.780 us | **685.084 us** | 18.88 percent lower | 52.22 percent lower | + +UCCL had the smallest latency increase when scaling from EP16 to EP32, while DeepEP V2 remained fastest in absolute latency. For FP8, the increases were 45.91 percent for UCCL, 59.21 percent for DeepEP V1, and 67.93 percent for DeepEP V2. This is a single-start scale observation, not a scaling-efficiency claim. + +The native decode bandwidth reports are included for completeness. UCCL and DeepEP V1 report one dispatch-plus-combine bandwidth; DeepEP V2 reports separate SO legs. The V2 dispatch/combine pair is not an aggregate and is not directly comparable to the first 2 columns. + +| EP ranks | Dispatch dtype | UCCL aggregate | DeepEP V1 aggregate | DeepEP V2 dispatch / combine SO | +|---:|---|---:|---:|---:| +| 16 ranks | FP8 | 46.37 GB/s | 25.01 GB/s | 5 GB/s / 9 GB/s | +| 16 ranks | BF16 | 40.94 GB/s | 32.78 GB/s | 9 GB/s / 9 GB/s | +| 32 ranks | FP8 | 31.78 GB/s | 15.71 GB/s | 10 GB/s / 14 GB/s | +| 32 ranks | BF16 | 34.43 GB/s | 20.28 GB/s | 16 GB/s / 14 GB/s | + +## Prefill comparison + +The table reports each backend's native latency and cross-node bandwidth fields. For UCCL, latency is `transmit + notify` and bandwidth is the RDMA leg. For DeepEP V1, latency is the sum of the 2 printed components and bandwidth is the RDMA leg. For DeepEP V2, latency is the operation latency and bandwidth is the scale-out, or SO, leg. These measurement boundaries and bandwidth accounting conventions differ, so the table is directional. Do not rank the backends by comparing the GB/s columns alone. + +| EP ranks | Operation | UCCL latency / RDMA bandwidth | DeepEP V1 latency / RDMA bandwidth | DeepEP V2 latency / SO bandwidth | +|---:|---|---:|---:|---:| +| 16 ranks | FP8 dispatch | 1,161.120 us / 55.59 GB/s | 1,014.000 us / 65.95 GB/s | **864.688 us** / 35 GB/s | +| 16 ranks | BF16 dispatch | 1,637.970 us / 74.42 GB/s | 1,639.000 us / 78.14 GB/s | **1,501.000 us** / 39 GB/s | +| 16 ranks | BF16 combine | 2,096.580 us / 58.81 GB/s | 1,633.220 us / 78.62 GB/s | **1,615.000 us** / 36 GB/s | +| 32 ranks | FP8 dispatch | 2,229.950 us / 51.38 GB/s | 2,369.000 us / 50.95 GB/s | **2,041.000 us** / 41 GB/s | +| 32 ranks | BF16 dispatch | 3,852.970 us / 56.69 GB/s | 4,125.000 us / 56.94 GB/s | **3,753.000 us** / 43 GB/s | +| 32 ranks | BF16 combine | 3,950.790 us / 55.52 GB/s | 4,106.520 us / 55.80 GB/s | **3,745.000 us** / 43 GB/s | + +## DeepEP V2 operation results + +SO is scale-out bandwidth over the internode path and SU is scale-up bandwidth over the intranode path. FP8 combine is included because the selected V2 test prints it, but the matched headline operation set uses BF16 combine. + +| Topology and workload | Dispatch dtype | Operation | SO bandwidth | SU bandwidth | Latency | Payload | +|---|---|---|---:|---:|---:|---:| +| EP16 decode | FP8 | dispatch | 5 GB/s | 27 GB/s | 185.637 us | 4,972,032 bytes | +| EP16 decode | FP8 | combine | 9 GB/s | 48 GB/s | 199.357 us | 9,540,352 bytes | +| EP16 decode | BF16 | dispatch | 9 GB/s | 49 GB/s | 195.257 us | 9,582,848 bytes | +| EP16 decode | BF16 | combine | 9 GB/s | 48 GB/s | 197.481 us | 9,540,352 bytes | +| EP32 decode | FP8 | dispatch | 10 GB/s | 19 GB/s | 275.659 us | 5,106,816 bytes | +| EP32 decode | FP8 | combine | 14 GB/s | 26 GB/s | 370.863 us | 9,798,976 bytes | +| EP32 decode | BF16 | dispatch | 16 GB/s | 31 GB/s | 314.156 us | 9,842,624 bytes | +| EP32 decode | BF16 | combine | 14 GB/s | 26 GB/s | 370.928 us | 9,798,976 bytes | +| EP16 prefill | FP8 | dispatch | 35 GB/s | 203 GB/s | 864.688 us | 175,106,880 bytes | +| EP16 prefill | FP8 | combine | 36 GB/s | 208 GB/s | 1,614.000 us | 335,995,680 bytes | +| EP16 prefill | BF16 | dispatch | 39 GB/s | 225 GB/s | 1,501.000 us | 337,492,320 bytes | +| EP16 prefill | BF16 | combine | 36 GB/s | 208 GB/s | 1,615.000 us | 335,995,680 bytes | +| EP32 prefill | FP8 | dispatch | 41 GB/s | 95 GB/s | 2,041.000 us | 194,860,224 bytes | +| EP32 prefill | FP8 | combine | 43 GB/s | 100 GB/s | 3,749.000 us | 373,898,464 bytes | +| EP32 prefill | BF16 | dispatch | 43 GB/s | 100 GB/s | 3,753.000 us | 375,563,936 bytes | +| EP32 prefill | BF16 | combine | 43 GB/s | 100 GB/s | 3,745.000 us | 373,898,464 bytes | + +The DeepEP V2 EFA RDMA write deltas were 19,708,762,240 bytes for EP16 decode, 111,413,870,476 bytes for EP32 decode, 507,290,310,848 bytes for EP16 prefill, and 2,742,751,387,856 bytes for EP32 prefill. + +## Memory tradeoff + +All 12 cells reached 100 percent sampled peak GPU utilization. Peak HBM use was materially higher for DeepEP V2 prefill. + +| Backend | EP16 decode | EP32 decode | EP16 prefill | EP32 prefill | +|---|---:|---:|---:|---:| +| UCCL | 9,627 MiB/GPU | 9,501 MiB/GPU | **13,697 MiB/GPU** | **20,735 MiB/GPU** | +| DeepEP V1 NVSHMEM | 10,597 MiB/GPU | 11,239 MiB/GPU | 17,047 MiB/GPU | 23,793 MiB/GPU | +| DeepEP V2 NCCL GIN | **7,041 MiB/GPU** | **7,271 MiB/GPU** | 39,657 MiB/GPU | 44,315 MiB/GPU | + +## Subsequent EFA 3.3.0g revalidation + +On 24 August 2026, a separate revalidation updated the current 32-node B200 fleet to the EFA 3.3.0g kernel module and rdma-core 64 on kernel `6.12.100-125.179.amzn2023.x86_64`. The post-update audit passed on 32 of 32 nodes. The first targeted DeepEP V2 admission then triggered a kernel panic in the Linux device-memory mapping path called by NVIDIA UVM, with the first preserved frame at `__init_zone_device_page`. The remaining runs were stopped to avoid risking more nodes. + +That trace does not establish EFA 3.3.0g as the root cause. The completed results above are from the earlier matched B200 campaign and are not presented as post-update measurements. The revalidation's benchmark resources were harvested and removed; all 32 nodes returned Ready. + +## Reproduce and collate + +The standalone DeepEP V2 build and launch harness is being added in [PR 1234](https://github.com/awslabs/awsome-distributed-ai/pull/1234). Pin the image and source revisions recorded above, use `NCCL_GIN_TYPE=5` and `NCCL_SYM_GIN_KERNELS_ENABLE=0`, and first run an INFO-level admission that proves the `Libfabric_GDAKI` context and nonzero GIN layout. Return `NCCL_DEBUG` to `WARN` for measurement. + +The collector accepts rank-zero DeepEP V2 logs from `tests/elastic/test_ep.py`: + +```bash +python3 collect_results.py \ + --nvshmem-internode nvshmem_prefill.log \ + --nvshmem-lowlat nvshmem_decode.log \ + --uccl-internode uccl_prefill.log \ + --uccl-lowlat uccl_decode.log \ + --deepep-v2-prefill deepep_v2_prefill.log \ + --deepep-v2-decode deepep_v2_decode.log \ + --nccl nccl_alltoall.log +``` + +Preserve every rank's log, rendered launch manifest, immutable image reference, command line, correctness result, and before-and-after EFA counters. `collect_results.py` summarizes rank zero, but correctness and path validation must cover every rank and every node. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md index d9184ead4..7ae324719 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md @@ -1,7 +1,9 @@ # EP-Backend Comparison Results — P5 (H100) Same harness and matched config as [`RESULTS.md`](RESULTS.md) (B300), run on `p5.48xlarge` -(H100). Use this to compare the three dispatchers **across GPU generations**. +(H100). Use this to compare the three historical dispatchers **across GPU generations**. The +newer B200 direct comparison with DeepEP V2 is in [`RESULTS-b200.md`](RESULTS-b200.md) and is +not mixed with these H100 measurements. ## Environment diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index 198f52473..b49023181 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -3,7 +3,9 @@ Generated by running the three backends at a matched EP config on `p6-b300.48xlarge` nodes and collating with [`collect_results.py`](collect_results.py). For the **P5 (H100)** run of the same matrix — and the cross-generation contrast (the winner flips by GPU) — see -[`RESULTS-p5.md`](RESULTS-p5.md). +[`RESULTS-p5.md`](RESULTS-p5.md). For the newer **B200** direct comparison that adds DeepEP V2 +over NCCL GIN EFA-GDA, see [`RESULTS-b200.md`](RESULTS-b200.md). Hardware generations and +campaigns are reported separately. ## Environment diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py index 86406ad7c..041db3af4 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py @@ -1,16 +1,17 @@ #!/usr/bin/env python3 # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # SPDX-License-Identifier: MIT-0 -"""Collate EP-backend comparison logs into a single markdown table. +"""Collate EP-backend comparison logs into markdown tables. -Parses the launcher logs from the three benchmarks run at the same world size: +Parses launcher logs from the EP backends run at the same world size: - * NVSHMEM (DeepEP) -- dispatch/combine bandwidth + * NVSHMEM (DeepEP V1) -- dispatch/combine bandwidth * UCCL (UCCL-EP) -- dispatch/combine bandwidth + * DeepEP V2 (NCCL GIN) -- scale-out/scale-up bandwidth and latency * NCCL (nccl-tests alltoall_perf) -- busbw (transport reference) -Output formats parsed (DeepEP and UCCL print identically -- UCCL's bench is -derived from DeepEP's tests): +Output formats parsed (DeepEP V1 and UCCL print identically -- UCCL's bench is +derived from the V1 tests): internode (test_internode.py): [tuning] Best dispatch (BF16): ... BW: GB/s (RDMA), GB/s (NVL) @@ -22,11 +23,18 @@ [rank N] Dispatch bandwidth: GB/s, avg_t=... | Combine bandwidth: GB/s, avg_t=... -> single bandwidth per dispatch/combine (no RDMA/NVL split). + DeepEP V2 (tests/elastic/test_ep.py): + > Testing with ... use_fp8_dispatch=<0|1> ... + * EP: 0/ | dispatch: GB/s (SO), GB/s (SU), us, ... + @ EP: 0/ | combine: GB/s (SO), GB/s (SU), us, ... + -> report rank zero for each dispatch dtype and operation. SO is the + scale-out leg and SU is the scale-up leg. + NCCL alltoall_perf: the standard size-sweep table; busbw is column 7 (out-of-place) / 11 (in-place). We report busbw at the sampled row whose size is closest to the EP per-rank dispatch payload (num_tokens * hidden * 2 bytes, ~56 MiB target; the power-of-two sweep lands on the 64 MiB row) AND the - asymptotic peak, because the peak overstates the transport ceiling relative to + asymptotic peak, because the peak overstates the transport context relative to EP's smaller messages. Both EP formats vary slightly across versions; if a value comes back N/A, print @@ -38,6 +46,8 @@ --nvshmem-lowlat nvshmem_lowlat.log \ --uccl-internode uccl_internode.log \ --uccl-lowlat uccl_lowlat.log \ + --deepep-v2-prefill deepep_v2_prefill.log \ + --deepep-v2-decode deepep_v2_decode.log \ --nccl nccl_alltoall.log \ --nccl-target-bytes 58720256 """ @@ -59,6 +69,16 @@ NCCL_BUSBW_COLS = (7, 11) # mpirun --tag-output prefixes every line with e.g. "[1,0]:". TAG_PREFIX_RE = re.compile(r"^\[\d+,\d+\]:") +# DeepEP V2 prints the dtype before each selected test case, then one result per +# EP rank. Restrict results to rank zero and to the base dispatch/combine rows; +# expanded/cached dispatch rows have different semantics. +V2_DTYPE_RE = re.compile(r"use_fp8_dispatch=(0|1)") +V2_RANK_ZERO_RE = re.compile( + r"[*!#@+\-]\s*EP:\s*0/\d+\s*\|\s*" + r"(dispatch|combine):\s*" + _BW + r"\s*GB/s\s*\(SO\),\s*" + + _BW + r"\s*GB/s\s*\(SU\),\s*" + _BW + r"\s*us", + re.I, +) def _last(text, regex): @@ -122,16 +142,50 @@ def parse_nccl(path, target_bytes): return best_at[2], best_at[1], peak +def parse_deepep_v2(path): + """Return rank-zero DeepEP V2 metrics keyed by (dtype, operation). + + Each value is ``(scale_out_gbps, scale_up_gbps, latency_us)``. A log that + omits the dtype marker is retained under ``unspecified`` instead of being + silently discarded. + """ + if not path: + return {} + dtype = "unspecified" + results = {} + with open(path) as f: + for line in f: + dtype_match = V2_DTYPE_RE.search(line) + if dtype_match: + dtype = "FP8" if dtype_match.group(1) == "1" else "BF16" + result_match = V2_RANK_ZERO_RE.search(line) + if not result_match: + continue + operation, scale_out, scale_up, latency = result_match.groups() + results[(dtype, operation.lower())] = ( + float(scale_out), + float(scale_up), + float(latency), + ) + return results + + def fmt(v): return f"{v:.1f}" if isinstance(v, float) else "N/A" +def fmt_latency(v): + return f"{v:.3f}" if isinstance(v, float) else "N/A" + + def main(): p = argparse.ArgumentParser() p.add_argument("--nvshmem-internode") p.add_argument("--nvshmem-lowlat") p.add_argument("--uccl-internode") p.add_argument("--uccl-lowlat") + p.add_argument("--deepep-v2-prefill") + p.add_argument("--deepep-v2-decode") p.add_argument("--nccl", help="NCCL alltoall_perf log (transport reference)") p.add_argument("--nccl-target-bytes", type=int, default=4096 * 7168 * 2, help="EP per-rank dispatch payload to read busbw at (default num_tokens*hidden*2)") @@ -141,21 +195,43 @@ def main(): nv_l_d, nv_l_c = parse_lowlat(args.nvshmem_lowlat) uc_i_d, uc_i_c = parse_internode(args.uccl_internode) uc_l_d, uc_l_c = parse_lowlat(args.uccl_lowlat) + v2_prefill = parse_deepep_v2(args.deepep_v2_prefill) + v2_decode = parse_deepep_v2(args.deepep_v2_decode) nccl_at, nccl_size, nccl_peak = parse_nccl(args.nccl, args.nccl_target_bytes) out = sys.stdout out.write("| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) |\n") out.write("|---|---|---:|---:|\n") - out.write(f"| NVSHMEM (DeepEP) | internode (RDMA) | {fmt(nv_i_d)} | {fmt(nv_i_c)} |\n") - out.write(f"| NVSHMEM (DeepEP) | low-latency | {fmt(nv_l_d)} | {fmt(nv_l_c)} |\n") + out.write(f"| DeepEP V1 (NVSHMEM) | internode (RDMA) | {fmt(nv_i_d)} | {fmt(nv_i_c)} |\n") + out.write(f"| DeepEP V1 (NVSHMEM) | low-latency | {fmt(nv_l_d)} | {fmt(nv_l_c)} |\n") out.write(f"| UCCL (UCCL-EP) | internode (RDMA) | {fmt(uc_i_d)} | {fmt(uc_i_c)} |\n") out.write(f"| UCCL (UCCL-EP) | low-latency | {fmt(uc_l_d)} | {fmt(uc_l_c)} |\n") out.write("\n") sz_mib = f"{nccl_size / 2**20:.0f} MiB" if isinstance(nccl_size, int) else "N/A" - out.write("| Reference (NCCL all-to-all, transport ceiling) | Metric | GB/s |\n") + out.write("| Reference (NCCL all-to-all, transport context) | Metric | GB/s |\n") out.write("|---|---|---:|\n") out.write(f"| busbw at EP payload (~{sz_mib}) | matched-size | {fmt(nccl_at)} |\n") - out.write(f"| busbw peak (asymptotic, overstates ceiling) | peak | {fmt(nccl_peak)} |\n") + out.write(f"| busbw peak (asymptotic context) | peak | {fmt(nccl_peak)} |\n") + + if v2_prefill or v2_decode: + out.write("\n") + out.write( + "| Backend | Workload | Dispatch dtype | Operation | " + "Scale-out (GB/s) | Scale-up (GB/s) | Latency (us) |\n" + ) + out.write("|---|---|---|---|---:|---:|---:|\n") + for workload, results in (("prefill", v2_prefill), ("decode", v2_decode)): + for dtype in ("FP8", "BF16", "unspecified"): + for operation in ("dispatch", "combine"): + values = results.get((dtype, operation)) + if not values: + continue + scale_out, scale_up, latency = values + out.write( + f"| DeepEP V2 (NCCL GIN) | {workload} | {dtype} | " + f"{operation} | {fmt(scale_out)} | {fmt(scale_up)} | " + f"{fmt_latency(latency)} |\n" + ) if __name__ == "__main__": diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py new file mode 100644 index 000000000..2c1e09be9 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +"""Run the matched FP8/BF16 DeepEP V2 direct-EP cases. + +This wrapper targets the pinned synthetic DeepEP V2 revision documented in +RESULTS-b200.md. It limits the upstream elastic benchmark to the 2 dispatch +dtypes used by the comparison while retaining correctness checks. +""" + +import argparse +import sys +from argparse import Namespace +from pathlib import Path + +import torch + + +def _load_upstream(): + candidates = ( + Path("/opt/amazon/deepep-v2/tests/elastic"), + Path("/opt/amazon/deepep/tests/elastic"), + ) + for candidate in candidates: + if (candidate / "test_ep.py").is_file(): + sys.path.insert(0, str(candidate)) + import test_ep # pylint: disable=import-outside-toplevel + + return test_ep + locations = ", ".join(str(path) for path in candidates) + raise RuntimeError(f"DeepEP V2 test_ep.py not found under: {locations}") + + +UPSTREAM = _load_upstream() + + +def selected_modes(): + """Yield FP8 and BF16 dispatch with the same remaining mode controls.""" + # handle copy, expert alignment, FP8 dispatch, bias count, previous event, + # async compute stream, allocate on communication stream + yield (1, 128, 1, 0, 0, 0, 0) + yield (1, 128, 0, 0, 0, 0, 0) + + +UPSTREAM.enumerate_ep_modes = selected_modes + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--num-processes", type=int, default=8) + parser.add_argument("--num-tokens", type=int, required=True) + parser.add_argument("--hidden", type=int, default=7168) + parser.add_argument("--num-topk", type=int, default=8) + parser.add_argument("--num-experts", type=int, default=256) + parser.add_argument("--seed", type=int, default=0) + args = parser.parse_args() + + selected = Namespace( + num_processes=args.num_processes, + num_sms=0, + num_qps=0, + num_allocated_qps=0, + num_gpu_timeout_secs=180, + num_cpu_timeout_secs=180, + sl_idx=0, + num_tokens=args.num_tokens, + hidden=args.hidden, + num_topk=args.num_topk, + num_experts=args.num_experts, + do_cpu_sync=1, + allow_hybrid_mode=1, + allow_multiple_reduction=1, + prefer_overlap_with_compute=0, + deterministic=False, + seed=args.seed, + skip_check=False, + skip_perf_test=False, + do_pressure_test=False, + pressure_iterations=0, + reuse_elastic_buffer=False, + test_first_only=False, + unbalanced_ratio=1.0, + precise_unbalanced_ratio=False, + masked_ratio=0.0, + dump_profile_traces="", + ignore_local_traffic=True, + ) + torch.multiprocessing.spawn( + UPSTREAM.test_loop, + args=(args.num_processes, selected), + nprocs=args.num_processes, + ) + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example b/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example index 1a653a5f4..1259a2446 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example @@ -5,13 +5,14 @@ # edit, then source it before applying the per-backend manifests. # cp env_vars.example env_vars && source env_vars -# GPU node instance type and per-node device counts (p6-b300.48xlarge => 16 EFA). +# GPU node instance type and per-node device counts. The defaults retain the +# historical B300 run. For B200 use p6-b200.48xlarge and 8 EFA devices per node. export INSTANCE_TYPE=p6-b300.48xlarge export GPU_PER_NODE=8 export EFA_PER_NODE=16 -# 8 = primary 3-way comparison. 32 (256 ranks) works for NCCL and the low-latency kernels -# only — both HT internode paths hit hard caps past 64-160 ranks (see README "Scaling -# beyond 8 nodes"). +# B200 direct EP16 uses 2 nodes and EP32 uses 4 nodes. The historical B300/H100 +# primary comparison uses 8 nodes. The 32-node findings apply only to the +# historical V1-era matrix; see README "Scaling beyond 8 nodes". export NUM_NODES=8 export NP=$((NUM_NODES * GPU_PER_NODE)) # ranks; used by the NCCL baseline (64 at 8 nodes) @@ -22,4 +23,5 @@ export NP=$((NUM_NODES * GPU_PER_NODE)) # ranks; used by the NCCL baseline (64 # sm_100 gencode, so reuse it for the baseline -- no separate nccl-tests build. export NVSHMEM_IMAGE_URI=.dkr.ecr..amazonaws.com/deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13 export UCCL_IMAGE_URI=.dkr.ecr..amazonaws.com/uccl-ep:efa1.48.0-uccl0dc87eb-cu13 +export DEEPEP_V2_IMAGE_URI=.dkr.ecr..amazonaws.com/deepep-v2@sha256: export NCCL_IMAGE_URI=$NVSHMEM_IMAGE_URI diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py new file mode 100644 index 000000000..6cccef07a --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +"""Unit tests for the EP comparison log collector.""" + +import importlib.util +import tempfile +import unittest +from pathlib import Path + + +MODULE_PATH = Path(__file__).with_name("collect_results.py") +SPEC = importlib.util.spec_from_file_location("collect_results", MODULE_PATH) +COLLECT_RESULTS = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(COLLECT_RESULTS) + + +class ParseDeepEPV2Test(unittest.TestCase): + def test_parses_rank_zero_selected_cases(self): + log = """ +> Testing with do_handle_copy=1, use_fp8_dispatch=1, num_bias=0 ... + * EP: 1/16 | dispatch: 34 GB/s (SO), 201 GB/s (SU), 866.000 us, 1 bytes + * EP: 0/16 | dispatch: 35 GB/s (SO), 203 GB/s (SU), 864.688 us, 2 bytes + @ EP: 0/16 | combine: 36 GB/s (SO), 208 GB/s (SU), 1614.000 us, 3 bytes + - EP: 0/16 | expanded dispatch: 99 GB/s (SO), 999 GB/s (SU), 1.000 us, 4 bytes +> Testing with do_handle_copy=1, use_fp8_dispatch=0, num_bias=0 ... + * EP: 0/16 | dispatch: 39 GB/s (SO), 225 GB/s (SU), 1501.000 us, 5 bytes + @ EP: 0/16 | combine: 36 GB/s (SO), 208 GB/s (SU), 1615.000 us, 6 bytes +""" + with tempfile.NamedTemporaryFile("w", encoding="utf-8") as handle: + handle.write(log) + handle.flush() + result = COLLECT_RESULTS.parse_deepep_v2(handle.name) + + self.assertEqual(result[("FP8", "dispatch")], (35.0, 203.0, 864.688)) + self.assertEqual(result[("FP8", "combine")], (36.0, 208.0, 1614.0)) + self.assertEqual(result[("BF16", "dispatch")], (39.0, 225.0, 1501.0)) + self.assertEqual(result[("BF16", "combine")], (36.0, 208.0, 1615.0)) + self.assertEqual(len(result), 4) + + def test_retains_result_without_dtype_marker(self): + log = "* EP: 0/32 | dispatch: 10 GB/s (SO), 19 GB/s (SU), 275.659 us, 1 bytes\n" + with tempfile.NamedTemporaryFile("w", encoding="utf-8") as handle: + handle.write(log) + handle.flush() + result = COLLECT_RESULTS.parse_deepep_v2(handle.name) + + self.assertEqual( + result[("unspecified", "dispatch")], + (10.0, 19.0, 275.659), + ) + + +if __name__ == "__main__": + unittest.main() From 137f849dcb240a8381507b2609a92d32aaf8a1aa Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 19:46:36 +0000 Subject: [PATCH 02/19] feat(ep): add fair common-boundary benchmark --- .../ep-backend-comparison/README.md | 22 +- .../fair_ep_benchmark.py | 603 ++++++++++++++++++ .../run_fair_ep_comparison.sh | 425 ++++++++++++ .../ep-backend-comparison/run_fair_ep_rank.sh | 84 +++ .../summarize_fair_results.py | 276 ++++++++ .../test_fair_ep_benchmark.py | 58 ++ .../test_summarize_fair_results.py | 84 +++ 7 files changed, 1551 insertions(+), 1 deletion(-) create mode 100755 micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py create mode 100755 micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh create mode 100755 micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_rank.sh create mode 100755 micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index 2e5f89d86..6ace254a3 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -37,6 +37,26 @@ docker run --rm ${NVSHMEM_IMAGE_URI} sed -n '1,60p' /DeepEP/tests/test_internode If the DeepEP values differ from 4096/7168/8/256, edit the bench args in `../uccl-ep-benchmark/kubernetes/test-*.yaml` to match. +### Fair common-boundary decode comparison + +The backend-native reports above do not share one timing boundary or byte numerator. Use [`fair_ep_benchmark.py`](fair_ep_benchmark.py) when making a direct cross-backend latency or effective-bandwidth claim. It supplies all 3 backends with the same deterministic BF16 input, exact top-k route, and weights, then times BF16-input readiness through dispatch and combine completion with one CUDA Event boundary. Each iteration uses the slowest rank's elapsed time. + +The common logical-byte numerator counts the useful dispatch tensor, FP8 scales when selected, and BF16 combine tensor for each valid expert assignment. It excludes backend metadata. Backend-native UCCL, NVSHMEM RDMA, and DeepEP V2 SO/SU measurements remain useful diagnostics, but they are reported separately. + +[`run_fair_ep_comparison.sh`](run_fair_ep_comparison.sh) runs the B200 decode matrix at 128 tokens/rank for EP16 and EP32. It uses the same named nodes for every arm, performs 3 independent process starts per cell in rotated arm order, validates route and input hashes, and tears down only its labeled namespace. The concurrent campaign's nodes and shared Lease are read and protected, never modified. + +```bash +CAMPAIGN_ID=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) \ +FAIR_EP_NODES=node-a,node-b,node-c,node-d \ +PROTECTED_NODES_CSV=foreign-node-a,foreign-node-b \ +EXPECTED_LOCK_HOLDER=foreign-campaign-id \ +ARTIFACT_ROOT=/shared/artifacts/${CAMPAIGN_ID} \ +KUBECTL_CONTEXT=target-context \ +./run_fair_ep_comparison.sh +``` + +DeepEP V2 admission is conditional on `uvm_disable_hmm=Y` on every selected host and requires a logged GDAKI context. The runner aborts before the scored matrix if either condition is absent. + ## Prerequisites - EKS cluster with EFA + GPU nodes; NVIDIA device plugin + AWS EFA device plugin; Kubeflow MPI @@ -159,7 +179,7 @@ The historical UCCL and DeepEP V1 matrix was pushed to 16 and 32 nodes, or 128 a - **Internode = RDMA leg.** DeepEP/UCCL print both an RDMA (cross-node) and an NVL (intra-node) bandwidth on the same line; only the RDMA number reflects the inter-node transport being compared. -- **DeepEP V2 accounting differs.** V2 prints SO and SU bandwidth plus per-operation latency. SO is the cross-node leg, but it is not numerically interchangeable with the V1/UCCL RDMA accounting. Use latency as the primary cross-backend metric and retain backend-native bandwidth as directional evidence. +- **Backend-native accounting differs.** V2 prints SO and SU bandwidth plus per-operation latency, while V1 and UCCL use their own aggregate timing and byte accounting. Neither the native GB/s values nor the native latency values establish an apples-to-apples ranking. Use the common-boundary harness for cross-backend claims and retain native values as diagnostics. - **`num-experts` must divide the world size.** Both tests assert `num_experts % num_ranks == 0`. At 8 nodes (64 ranks) the comparison uses 256 (= 4/rank). The DeepEP low-latency default (288) is not divisible by 64 and must be overridden (see the run-order note). diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py new file mode 100755 index 000000000..9fa21c52e --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py @@ -0,0 +1,603 @@ +#!/usr/bin/env python3 +"""Common-boundary DeepEP-compatible dispatch/combine benchmark. + +The benchmark deliberately avoids each backend's native timing and byte +accounting. Every arm receives the same deterministic BF16 input, exact route +indices, and top-k weights. A CUDA event pair surrounds the complete dispatch +followed by combine operation, and the slowest rank is the iteration latency. + +For FP8 dispatch, the timed boundary starts with a BF16 input. DeepEP V2's +explicit BF16-to-FP8 preparation is therefore inside the timed region, matching +the conversion already performed inside the V1 and UCCL low-latency APIs. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import statistics +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import torch +import torch.distributed as dist + + +RESULT_PREFIX = "ADAI_FAIR_RESULT " +SCHEMA_VERSION = 1 + + +def percentile(values: list[float], quantile: float) -> float: + """Return a linearly interpolated percentile without a NumPy dependency.""" + + if not values: + raise ValueError("percentile requires at least one value") + ordered = sorted(values) + position = (len(ordered) - 1) * quantile + lower = math.floor(position) + upper = math.ceil(position) + if lower == upper: + return ordered[lower] + fraction = position - lower + return ordered[lower] * (1 - fraction) + ordered[upper] * fraction + + +def tensor_sha256(tensor: torch.Tensor) -> str: + raw = tensor.detach().contiguous().view(torch.uint8).cpu().numpy().tobytes() + return hashlib.sha256(raw).hexdigest() + + +def global_digest(local_digest: str, device: torch.device) -> str: + digest_tensor = torch.tensor( + list(bytes.fromhex(local_digest)), dtype=torch.uint8, device=device + ) + gathered = [torch.empty_like(digest_tensor) for _ in range(dist.get_world_size())] + dist.all_gather(gathered, digest_tensor) + payload = b"".join(bytes(item.cpu().tolist()) for item in gathered) + return hashlib.sha256(payload).hexdigest() + + +def make_route( + rank: int, + num_tokens: int, + num_experts: int, + top_k: int, + seed: int, + device: torch.device, +) -> torch.Tensor: + """Create a balanced, backend-independent route with unique experts/token.""" + + stride = 31 + if math.gcd(stride, num_experts) != 1: + raise ValueError("route stride must be coprime with num_experts") + global_token = rank * num_tokens + torch.arange( + num_tokens, dtype=torch.int64, device=device + ) + slots = torch.arange(top_k, dtype=torch.int64, device=device) + route = ( + global_token[:, None] * 17 + slots[None, :] * stride + seed + ) % num_experts + if top_k > num_experts: + raise ValueError("top_k cannot exceed num_experts") + return route.contiguous() + + +def make_input( + rank: int, num_tokens: int, hidden: int, device: torch.device +) -> torch.Tensor: + """Create deterministic, bounded BF16 data without backend RNG state.""" + + row = rank * num_tokens + torch.arange( + num_tokens, dtype=torch.int64, device=device + ) + column = torch.arange(hidden, dtype=torch.int64, device=device) + values = (row[:, None] * 17 + column[None, :] * 13 + 19) % 257 + return ((values - 128).to(torch.float32) / 128.0).to(torch.bfloat16) + + +def normalized_diff(actual: torch.Tensor, expected: torch.Tensor) -> float: + actual64 = actual.double() + 1 + expected64 = expected.double() + 1 + denominator = (actual64.square() + expected64.square()).sum() + similarity = 2 * (actual64 * expected64).sum() / denominator + return float((1 - similarity).item()) + + +def logical_payload_bytes_per_rank( + route: torch.Tensor, + rank: int, + world_size: int, + local_world_size: int, + hidden: int, + dispatch_dtype: str, + num_experts: int, +) -> tuple[int, int, int]: + """Return all, scale-out, and valid route selections' logical payload bytes. + + Each valid expert assignment contributes its useful dispatch tensor, FP8 + scales when applicable, and one BF16 combine tensor. Backend metadata is + excluded. Scale-out bytes count assignments owned by a different node. + """ + + if num_experts % world_size: + raise ValueError("num_experts must divide world_size") + dispatch_bytes = hidden * 2 + if dispatch_dtype == "fp8": + dispatch_bytes = hidden + math.ceil(hidden / 128) * 4 + elif dispatch_dtype != "bf16": + raise ValueError(f"unsupported dispatch dtype: {dispatch_dtype}") + bytes_per_selection = dispatch_bytes + hidden * 2 + valid = route >= 0 + valid_selections = int(valid.sum().item()) + expert_owner = route // (num_experts // world_size) + source_node = rank // local_world_size + destination_node = expert_owner // local_world_size + remote_selections = int(((destination_node != source_node) & valid).sum().item()) + return ( + valid_selections * bytes_per_selection, + remote_selections * bytes_per_selection, + valid_selections, + ) + + +@dataclass +class DispatchState: + recv_x: Any + recv_topk_weights: torch.Tensor | None + handle: Any + + +class BackendAdapter: + def __init__( + self, + arm: str, + group: dist.ProcessGroup, + num_tokens: int, + hidden: int, + num_experts: int, + ) -> None: + self.arm = arm + self.group = group + self.num_tokens = num_tokens + self.hidden = hidden + self.num_experts = num_experts + self.world_size = dist.get_world_size(group) + self.buffer: Any + self._cast_back: Any = None + self._cast_to_fp8: Any = None + + if arm == "uccl": + sys.path.insert(0, "/opt/uccl/ep/bench") + from buffer import Buffer # type: ignore[import-not-found] + from utils import per_token_cast_back # type: ignore[import-not-found] + + self._cast_back = per_token_cast_back + rdma_bytes = Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + ) + elif arm == "deepep-v1-nvshmem": + sys.path.insert(0, "/opt/amazon/deepep/tests") + sys.path.insert(0, "/opt/amazon/deepep") + import deep_ep # type: ignore[import-not-found] + from utils import per_token_cast_back # type: ignore[import-not-found] + + self._cast_back = per_token_cast_back + rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = deep_ep.Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + allow_mnnvl=False, + ) + elif arm == "deepep-v2-gin-gda": + sys.path.insert(0, "/opt/amazon/deepep-v2") + import deep_ep # type: ignore[import-not-found] + from deep_ep.utils.math import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) + + self._cast_back = per_token_cast_back + self._cast_to_fp8 = per_token_cast_to_fp8 + self.buffer = deep_ep.ElasticBuffer( + group, + num_max_tokens_per_rank=num_tokens, + hidden=hidden, + deterministic=False, + allow_hybrid_mode=True, + allow_multiple_reduction=True, + prefer_overlap_with_compute=False, + sl_idx=0, + num_allocated_qps=0, + explicitly_destroy=True, + num_gpu_timeout_secs=180, + num_cpu_timeout_secs=180, + ) + else: + raise ValueError(f"unsupported arm: {arm}") + + @property + def is_elastic(self) -> bool: + return self.arm == "deepep-v2-gin-gda" + + def prepare_dispatch_input(self, x: torch.Tensor, dispatch_dtype: str) -> Any: + if dispatch_dtype == "bf16": + return x + if dispatch_dtype != "fp8": + raise ValueError(f"unsupported dispatch dtype: {dispatch_dtype}") + if self.is_elastic: + return self._cast_to_fp8(x) + return x + + def dispatch( + self, + prepared_x: Any, + topk_idx: torch.Tensor, + topk_weights: torch.Tensor, + dispatch_dtype: str, + ) -> DispatchState: + if self.is_elastic: + recv_x, _, recv_weights, handle, event = self.buffer.dispatch( + x=prepared_x, + topk_idx=topk_idx, + topk_weights=topk_weights, + num_experts=self.num_experts, + num_max_tokens_per_rank=self.num_tokens, + expert_alignment=1, + async_with_compute_stream=True, + allocate_on_comm_stream=False, + do_handle_copy=True, + do_cpu_sync=True, + ) + event.current_stream_wait() + return DispatchState(recv_x, recv_weights, handle) + + recv_x, _, handle, event, _ = self.buffer.low_latency_dispatch( + prepared_x, + topk_idx, + self.num_tokens, + self.num_experts, + use_fp8=dispatch_dtype == "fp8", + async_finish=True, + return_recv_hook=False, + ) + event.current_stream_wait() + return DispatchState(recv_x, None, handle) + + def received_as_bf16(self, recv_x: Any, dispatch_dtype: str) -> torch.Tensor: + if dispatch_dtype == "bf16": + return recv_x + if self.is_elastic: + return self._cast_back(recv_x[0], recv_x[1]) + fp8, scales = recv_x + return self._cast_back( + fp8.view(-1, self.hidden), + scales.view(-1, self.hidden // 128), + ).view(fp8.shape) + + def combine( + self, + combine_input: torch.Tensor, + state: DispatchState, + topk_idx: torch.Tensor, + topk_weights: torch.Tensor, + ) -> torch.Tensor: + if self.is_elastic: + combined, _, event = self.buffer.combine( + x=combine_input, + handle=state.handle, + topk_weights=state.recv_topk_weights, + async_with_compute_stream=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return combined + + combined, event, _ = self.buffer.low_latency_combine( + combine_input, + topk_idx, + topk_weights, + state.handle, + use_logfmt=False, + async_finish=True, + return_recv_hook=False, + ) + event.current_stream_wait() + return combined + + def destroy(self) -> None: + self.buffer.destroy() + + +def initialize_distributed() -> tuple[int, int, int, torch.device, dist.ProcessGroup]: + local_rank = int(os.environ["LOCAL_RANK"]) + local_world_size = int(os.environ.get("LOCAL_WORLD_SIZE", "1")) + torch.cuda.set_device(local_rank) + device = torch.device(f"cuda:{local_rank}") + dist.init_process_group("nccl", device_id=device) + world_size = dist.get_world_size() + group = dist.new_group(list(range(world_size))) + torch.set_default_dtype(torch.bfloat16) + return dist.get_rank(), world_size, local_world_size, device, group + + +def run_dtype( + adapter: BackendAdapter, + x: torch.Tensor, + route: torch.Tensor, + topk_weights: torch.Tensor, + dispatch_dtype: str, + warmups: int, + iterations: int, + rank: int, + world_size: int, + local_world_size: int, + args: argparse.Namespace, + route_hash: str, + input_hash: str, +) -> dict[str, Any]: + prepared = adapter.prepare_dispatch_input(x, dispatch_dtype) + correctness_state = adapter.dispatch(prepared, route, topk_weights, dispatch_dtype) + correctness_input = adapter.received_as_bf16( + correctness_state.recv_x, dispatch_dtype + ) + correctness_output = adapter.combine( + correctness_input, correctness_state, route, topk_weights + ) + torch.cuda.synchronize() + expected = x * topk_weights.sum(dim=1, keepdim=True).to(torch.bfloat16) + diff = normalized_diff(correctness_output, expected) + max_abs_error = float( + (correctness_output.float() - expected.float()).abs().max().item() + ) + tolerance = 9e-4 if dispatch_dtype == "fp8" else 1e-5 + correctness_pass = diff <= tolerance and bool( + torch.isfinite(correctness_output).all().item() + ) + correctness_tensor = torch.tensor( + [1 if correctness_pass else 0], dtype=torch.int32, device=x.device + ) + dist.all_reduce(correctness_tensor, op=dist.ReduceOp.MIN) + if int(correctness_tensor.item()) != 1: + raise RuntimeError( + f"correctness failed for {dispatch_dtype}: diff={diff}, " + f"tolerance={tolerance}, max_abs_error={max_abs_error}" + ) + + # Timed combine uses a stable, preallocated expert-output tensor. This + # keeps expert computation and FP8 dequantization outside the communication + # boundary while retaining the handle created by each timed dispatch. + combine_input = torch.zeros_like(correctness_input, dtype=torch.bfloat16) + + def iteration() -> None: + current_x = adapter.prepare_dispatch_input(x, dispatch_dtype) + state = adapter.dispatch(current_x, route, topk_weights, dispatch_dtype) + adapter.combine(combine_input, state, route, topk_weights) + + for _ in range(warmups): + dist.barrier(group=adapter.group) + iteration() + torch.cuda.synchronize() + + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + max_rank_latency_ms: list[float] = [] + for _ in range(iterations): + dist.barrier(group=adapter.group) + start.record() + iteration() + end.record() + end.synchronize() + local_latency = torch.tensor( + [start.elapsed_time(end)], dtype=torch.float32, device=x.device + ) + dist.all_reduce(local_latency, op=dist.ReduceOp.MAX, group=adapter.group) + max_rank_latency_ms.append(float(local_latency.item())) + + logical_bytes, scaleout_bytes, valid_selections = ( + logical_payload_bytes_per_rank( + route, + rank, + world_size, + local_world_size, + args.hidden, + dispatch_dtype, + args.experts, + ) + ) + counters = torch.tensor( + [logical_bytes, scaleout_bytes, valid_selections], + dtype=torch.int64, + device=x.device, + ) + dist.all_reduce(counters, op=dist.ReduceOp.SUM, group=adapter.group) + avg_logical_bytes = int(counters[0].item()) / world_size + avg_scaleout_bytes = int(counters[1].item()) / world_size + global_valid_selections = int(counters[2].item()) + + median_ms = statistics.median(max_rank_latency_ms) + mean_ms = statistics.fmean(max_rank_latency_ms) + stdev_ms = statistics.stdev(max_rank_latency_ms) if iterations > 1 else 0.0 + result = { + "schema_version_dimensionless": SCHEMA_VERSION, + "benchmark": "common-boundary-dispatch-combine", + "arm": args.arm, + "run_index_dimensionless": args.run_index, + "dispatch_dtype": dispatch_dtype, + "world_size_ranks": world_size, + "nodes": world_size // local_world_size, + "gpus_per_node": local_world_size, + "tokens_per_rank": args.tokens, + "global_input_tokens": args.tokens * world_size, + "hidden_dimensions": args.hidden, + "experts": args.experts, + "top_k_dimensionless": args.top_k, + "route_seed_dimensionless": args.seed, + "route_hash_sha256": route_hash, + "input_hash_sha256": input_hash, + "global_valid_expert_selections": global_valid_selections, + "warmup_iterations": warmups, + "measured_iterations": iterations, + "timing_boundary": "BF16 input ready through dispatch and combine completion; slowest rank CUDA elapsed time", + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "avg_logical_payload_bytes_per_rank": avg_logical_bytes, + "avg_scaleout_logical_payload_bytes_per_rank": avg_scaleout_bytes, + "latency_ms": { + "median": median_ms, + "mean": mean_ms, + "p95": percentile(max_rank_latency_ms, 0.95), + "minimum": min(max_rank_latency_ms), + "maximum": max(max_rank_latency_ms), + "stdev": stdev_ms, + "cv_percent": stdev_ms / mean_ms * 100 if mean_ms else 0.0, + }, + "aggregate_input_tokens_per_second": ( + args.tokens * world_size / (median_ms / 1e3) + ), + "effective_logical_gigabytes_per_second_per_rank": ( + avg_logical_bytes / (median_ms / 1e3) / 1e9 + ), + "effective_scaleout_logical_gigabytes_per_second_per_rank": ( + avg_scaleout_bytes / (median_ms / 1e3) / 1e9 + ), + "correctness": { + "status": "PASS", + "normalized_diff_dimensionless": diff, + "tolerance_dimensionless": tolerance, + "max_abs_error_bf16_value": max_abs_error, + }, + "runtime": { + "image_reference": os.environ.get("ADAI_IMAGE_REFERENCE", "unknown"), + "torch_version": torch.__version__, + "cuda_version": torch.version.cuda, + "nccl_version": list(torch.cuda.nccl.version()), + "gpu": torch.cuda.get_device_name(), + }, + } + return result + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument( + "--arm", + required=True, + choices=("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda"), + ) + parser.add_argument("--tokens", type=int, default=128) + parser.add_argument("--hidden", type=int, default=7168) + parser.add_argument("--top-k", type=int, default=8) + parser.add_argument("--experts", type=int, default=256) + parser.add_argument("--seed", type=int, default=20260824) + parser.add_argument("--warmups", type=int, default=20) + parser.add_argument("--iterations", type=int, default=100) + parser.add_argument("--run-index", type=int, required=True) + parser.add_argument( + "--dispatch-dtypes", + default="fp8,bf16", + help="Comma-separated dtype order; allowed values are fp8 and bf16", + ) + args = parser.parse_args() + args.dispatch_dtypes = args.dispatch_dtypes.split(",") + if sorted(args.dispatch_dtypes) != ["bf16", "fp8"]: + parser.error("--dispatch-dtypes must contain fp8 and bf16 exactly once") + if args.warmups < 1 or args.iterations < 2: + parser.error("at least 1 warmup and 2 measured iterations are required") + return args + + +def main() -> None: + args = parse_args() + rank, world_size, local_world_size, device, group = initialize_distributed() + if args.experts % world_size: + raise SystemExit("experts must divide the distributed world size") + route = make_route( + rank, args.tokens, args.experts, args.top_k, args.seed, device + ) + x = make_input(rank, args.tokens, args.hidden, device) + topk_weights = torch.full( + (args.tokens, args.top_k), + 1.0 / args.top_k, + dtype=torch.float32, + device=device, + ) + route_hash = global_digest(tensor_sha256(route), device) + input_hash = global_digest(tensor_sha256(x), device) + route_histogram = torch.bincount( + route.flatten(), minlength=args.experts + ).to(torch.int64) + dist.all_reduce(route_histogram, op=dist.ReduceOp.SUM, group=group) + + if rank == 0: + print( + "ADAI_FAIR_CONFIG " + + json.dumps( + { + "arm": args.arm, + "world_size_ranks": world_size, + "tokens_per_rank": args.tokens, + "hidden_dimensions": args.hidden, + "experts": args.experts, + "top_k_dimensionless": args.top_k, + "route_hash_sha256": route_hash, + "input_hash_sha256": input_hash, + "route_histogram_min_selections": int( + route_histogram.min().item() + ), + "route_histogram_max_selections": int( + route_histogram.max().item() + ), + }, + sort_keys=True, + ), + flush=True, + ) + + adapter = BackendAdapter( + args.arm, group, args.tokens, args.hidden, args.experts + ) + try: + for dispatch_dtype in args.dispatch_dtypes: + result = run_dtype( + adapter, + x, + route, + topk_weights, + dispatch_dtype, + args.warmups, + args.iterations, + rank, + world_size, + local_world_size, + args, + route_hash, + input_hash, + ) + if rank == 0: + print(RESULT_PREFIX + json.dumps(result, sort_keys=True), flush=True) + finally: + adapter.destroy() + dist.barrier(group=group) + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh new file mode 100755 index 000000000..307bdc37f --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh @@ -0,0 +1,425 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${CAMPAIGN_ID:?Set a unique CAMPAIGN_ID}" +: "${FAIR_EP_NODES:?Set 4 comma-separated B200 node names}" +: "${PROTECTED_NODES_CSV:?Set the concurrent campaign protected node names}" +: "${EXPECTED_LOCK_HOLDER:?Set the observed shared Lease holder}" +: "${ARTIFACT_ROOT:?Set the durable artifact directory}" +: "${KUBECTL_CONTEXT:=aps1}" +: "${CAMPAIGN_NAMESPACE:=${CAMPAIGN_ID}}" +: "${SHARED_LOCK_NAME:=adai-ap-south-1-gpu-campaign-lock}" +: "${SHARED_LOCK_NAMESPACE:=default}" +: "${INDEPENDENT_STARTS:=3}" +: "${WARMUP_ITERATIONS:=20}" +: "${MEASURED_ITERATIONS:=100}" +: "${CASE_TIMEOUT_SECONDS:=1800}" +: "${EFA_PER_NODE:=8}" +: "${UCCL_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a}" +: "${DEEPEP_V1_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac}" +: "${DEEPEP_V2_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb}" + +[[ "${CAMPAIGN_ID}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] +[[ "${CAMPAIGN_NAMESPACE}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] +[[ "${INDEPENDENT_STARTS}" -eq 3 ]] || { + printf 'This scored matrix requires exactly 3 independent starts\n' >&2 + exit 2 +} + +case_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +mkdir -p "${ARTIFACT_ROOT}/control" "${ARTIFACT_ROOT}/runs" \ + "${ARTIFACT_ROOT}/summary" "${ARTIFACT_ROOT}/teardown" +K=(kubectl --context "${KUBECTL_CONTEXT}") + +IFS=, read -r -a selected_nodes <<<"${FAIR_EP_NODES}" +IFS=, read -r -a protected_nodes <<<"${PROTECTED_NODES_CSV}" +((${#selected_nodes[@]} == 4)) || { + printf 'FAIR_EP_NODES must contain exactly 4 nodes\n' >&2 + exit 2 +} +[[ "$(printf '%s\n' "${selected_nodes[@]}" | sort -u | wc -l)" -eq 4 ]] + +declare -A protected=() +for node in "${protected_nodes[@]}"; do + protected["${node}"]=1 +done +for node in "${selected_nodes[@]}"; do + [[ -z "${protected[${node}]:-}" ]] || { + printf 'Selected node is protected by the concurrent campaign: %s\n' "${node}" >&2 + exit 1 + } +done + +declare -A images=( + [uccl]="${UCCL_IMAGE}" + [deepep-v1-nvshmem]="${DEEPEP_V1_IMAGE}" + [deepep-v2-gin-gda]="${DEEPEP_V2_IMAGE}" +) + +current_case="" +namespace_created=0 + +check_shared_lock() { + local holder + holder="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o jsonpath='{.spec.holderIdentity}' 2>/dev/null || true)" + if [[ -n "${holder}" && "${holder}" != "${EXPECTED_LOCK_HOLDER}" ]]; then + printf 'Shared Lease holder changed from protected campaign %s to %s\n' \ + "${EXPECTED_LOCK_HOLDER}" "${holder}" >&2 + return 1 + fi +} + +gpu_requests_on_node() { + local node="$1" + "${K[@]}" get pods -A --field-selector "spec.nodeName=${node}" -o json | jq ' + [.items[] + | select(.status.phase != "Succeeded" and .status.phase != "Failed") + | [.spec.containers[]?.resources.requests["nvidia.com/gpu"] // 0] + | add // 0] + | add // 0' +} + +verify_node_free() { + local node="$1" ready instance_type gpu efa requests + ready="$("${K[@]}" get node "${node}" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" + instance_type="$("${K[@]}" get node "${node}" -o jsonpath='{.metadata.labels.node\.kubernetes\.io/instance-type}')" + gpu="$("${K[@]}" get node "${node}" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}')" + efa="$("${K[@]}" get node "${node}" -o jsonpath='{.status.allocatable.vpc\.amazonaws\.com/efa}')" + requests="$(gpu_requests_on_node "${node}")" + [[ "${ready}" == True && "${instance_type}" == p6-b200.48xlarge && \ + "${gpu}" == 8 && "${efa}" == 8 && "${requests}" -eq 0 ]] || { + printf 'Node admission failed: node=%s ready=%s type=%s gpu=%s efa=%s requested_gpu=%s\n' \ + "${node}" "${ready}" "${instance_type}" "${gpu}" "${efa}" "${requests}" >&2 + return 1 + } +} + +cleanup_case() { + [[ -n "${current_case}" ]] || return 0 + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete statefulset "${current_case}" \ + --ignore-not-found --wait=true --timeout=5m >/dev/null 2>&1 || true + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete service "${current_case}" \ + --ignore-not-found >/dev/null 2>&1 || true + current_case="" +} + +finish() { + local command_status=$? teardown_status=0 owned="" remaining=0 + trap - EXIT INT TERM + set +e + cleanup_case + if ((namespace_created == 1)); then + owned="$("${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" \ + -o jsonpath='{.metadata.labels.adai\.aws/campaign}' 2>/dev/null || true)" + if [[ "${owned}" == "${CAMPAIGN_ID}" ]]; then + "${K[@]}" delete namespace "${CAMPAIGN_NAMESPACE}" \ + --wait=true --timeout=10m >"${ARTIFACT_ROOT}/teardown/namespace-delete.log" 2>&1 || \ + teardown_status=1 + else + printf 'Refusing to delete namespace without owned campaign label: %s\n' \ + "${CAMPAIGN_NAMESPACE}" >"${ARTIFACT_ROOT}/teardown/refused.txt" + teardown_status=1 + fi + fi + "${K[@]}" get all -A -l "adai.aws/campaign=${CAMPAIGN_ID}" -o json \ + >"${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>&1 || teardown_status=1 + remaining="$(jq '.items | length' "${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>/dev/null || printf '1')" + [[ "${remaining}" -eq 0 ]] || teardown_status=1 + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/teardown/shared-lease-untouched.json" 2>&1 || true + find "${ARTIFACT_ROOT}" -type f ! -name SHA256SUMS -print0 | sort -z | \ + xargs -0 sha256sum >"${ARTIFACT_ROOT}/SHA256SUMS" + if ((command_status == 0 && teardown_status == 0)); then + printf 'PASS\n' >"${ARTIFACT_ROOT}/STATUS" + else + printf 'FAIL command_status=%s_dimensionless teardown_status=%s_dimensionless remaining_resources=%s_resources\n' \ + "${command_status}" "${teardown_status}" "${remaining}" >"${ARTIFACT_ROOT}/STATUS" + fi + if ((command_status == 0 && teardown_status != 0)); then + command_status=1 + fi + exit "${command_status}" +} +trap finish EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +check_shared_lock +"${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-before.json" +"${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-before.json" +"${K[@]}" get namespaces -o json >"${ARTIFACT_ROOT}/control/namespaces-before.json" +"${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/control/shared-lease-before.json" +aws sts get-caller-identity --output json >"${ARTIFACT_ROOT}/control/aws-caller-identity.json" +printf '%s\n' "${selected_nodes[@]}" >"${ARTIFACT_ROOT}/control/selected-nodes.txt" +printf '%s\n' "${protected_nodes[@]}" | sort -u >"${ARTIFACT_ROOT}/control/protected-nodes.txt" + +for node in "${selected_nodes[@]}"; do + verify_node_free "${node}" +done + +"${K[@]}" create namespace "${CAMPAIGN_NAMESPACE}" --dry-run=client -o json | \ + jq --arg campaign "${CAMPAIGN_ID}" ' + .metadata.labels["adai.aws/campaign"]=$campaign | + .metadata.labels["adai.aws/owner"]="fair-ep-comparison"' | \ + "${K[@]}" apply -f - >/dev/null +namespace_created=1 + +# Read the live host mitigation before invoking DeepEP V2. The EFA 3.3.0g +# revalidation exposed a UVM HMM kernel panic, so a non-mitigated node is a hard +# admission failure rather than a benchmark attempt. +for index in 0 1 2 3; do + node="${selected_nodes[${index}]}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" apply -f - >/dev/null <&2 + exit 1 + } +done +"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete pod -l \ + "adai.aws/campaign=${CAMPAIGN_ID}" --wait=true --timeout=5m >/dev/null + +"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" create configmap fair-ep-scripts \ + --from-file=fair_ep_benchmark.py="${case_dir}/fair_ep_benchmark.py" \ + --from-file=run_fair_ep_rank.sh="${case_dir}/run_fair_ep_rank.sh" \ + --dry-run=client -o yaml | "${K[@]}" apply -f - >/dev/null + +jq -n \ + --arg campaign_id "${CAMPAIGN_ID}" \ + --arg created_at_utc "$(date -u +%FT%TZ)" \ + --arg region ap-south-1 \ + --arg cluster ml-clusters-shared-ap-south-1 \ + --arg git_commit "$(git -C "${case_dir}" rev-parse HEAD)" \ + --arg uccl "${UCCL_IMAGE}" \ + --arg v1 "${DEEPEP_V1_IMAGE}" \ + --arg v2 "${DEEPEP_V2_IMAGE}" \ + --argjson warmups "${WARMUP_ITERATIONS}" \ + --argjson iterations "${MEASURED_ITERATIONS}" \ + --argjson starts "${INDEPENDENT_STARTS}" \ + '{campaign_id:$campaign_id,created_at_utc:$created_at_utc,region:$region, + cluster:$cluster,git_commit:$git_commit, + images:{uccl:$uccl,"deepep-v1-nvshmem":$v1,"deepep-v2-gin-gda":$v2}, + comparison:{tokens_per_rank:128,hidden_dimensions:7168,experts:256, + top_k_dimensionless:8,warmup_iterations:$warmups, + measured_iterations:$iterations,independent_starts:$starts}}' \ + >"${ARTIFACT_ROOT}/control/provenance.json" + +run_case() { + local arm="$1" world_size="$2" run_index="$3" dtype_order="$4" + local warmups="${5:-${WARMUP_ITERATIONS}}" iterations="${6:-${MEASURED_ITERATIONS}}" + local nccl_debug="${7:-WARN}" label="${8:-measurement}" + local nodes=$((world_size / 8)) safe_arm="${arm//-}" node_values="" out="" + current_case="fair-ep${world_size}-r${run_index}-${safe_arm:0:20}-${label}" + current_case="${current_case:0:63}" + out="${ARTIFACT_ROOT}/runs/ep${world_size}/${label}-repeat-${run_index}/${arm}" + mkdir -p "${out}" + + check_shared_lock + for ((index = 0; index < nodes; index++)); do + verify_node_free "${selected_nodes[${index}]}" + node_values+=$'\n - '"${selected_nodes[${index}]}" + done + "${K[@]}" get pods -A -o json >"${out}/pods-before.json" + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${out}/shared-lease-before.json" + + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" apply -f - >/dev/null <"${out}/statefulset.yaml" + + local deadline=$((SECONDS + CASE_TIMEOUT_SECONDS)) complete=0 pod="" + while ((SECONDS < deadline)); do + complete=0 + for ((index = 0; index < nodes; index++)); do + pod="${current_case}-${index}" + if "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=30 2>/dev/null | \ + rg -q '^ADAI_FAIR_COMPLETE$'; then + complete=$((complete + 1)) + elif "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=80 2>/dev/null | \ + rg -q '^ADAI_FAIR_FAILED$'; then + complete=-1 + break + fi + done + ((complete == nodes || complete == -1)) && break + sleep 10 + done + + for ((index = 0; index < nodes; index++)); do + pod="${current_case}-${index}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" >"${out}/${pod}.log" 2>&1 || true + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" get pod "${pod}" -o yaml \ + >"${out}/${pod}.yaml" 2>&1 || true + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" describe pod "${pod}" \ + >"${out}/${pod}-describe.txt" 2>&1 || true + done + ((complete == nodes)) || { + printf 'Case failed or timed out: arm=%s EP%s repeat=%s label=%s complete=%s/%s\n' \ + "${arm}" "${world_size}" "${run_index}" "${label}" "${complete}" "${nodes}" >&2 + return 1 + } + local rank_zero_log="${out}/${current_case}-0.log" result_count + result_count="$(rg -c '^ADAI_FAIR_RESULT ' "${rank_zero_log}" || true)" + [[ "${result_count}" -eq 2 ]] || { + printf 'Expected 2 fair results in %s, found %s\n' "${rank_zero_log}" "${result_count}" >&2 + return 1 + } + rg '^ADAI_FAIR_RESULT ' "${rank_zero_log}" >"${out}/results.jsonl" + printf 'PASS arm=%s EP%s repeat=%s label=%s at %s UTC\n' \ + "${arm}" "${world_size}" "${run_index}" "${label}" "$(date -u +%FT%TZ)" \ + >"${out}/STATUS" + cleanup_case +} + +# A short V2-only admission run proves the HMM mitigation, GIN/GDAKI path, and +# common-harness correctness before any scored matrix work. +run_case deepep-v2-gin-gda 16 0 bf16,fp8 2 5 INFO admission +v2_admission_dir="${ARTIFACT_ROOT}/runs/ep16/admission-repeat-0/deepep-v2-gin-gda" +rg -q 'GDAKI.*createContext|gin GDAKI: createContext done' \ + "${v2_admission_dir}"/*.log || { + printf 'DeepEP V2 admission completed without a GDAKI context proof\n' >&2 + exit 1 +} +printf 'PASS\n' >"${v2_admission_dir}/GIN_ADMISSION_STATUS" + +for world_size in 16 32; do + for run_index in 1 2 3; do + case "${run_index}" in + 1) order=(uccl deepep-v1-nvshmem deepep-v2-gin-gda); dtypes=fp8,bf16 ;; + 2) order=(deepep-v2-gin-gda uccl deepep-v1-nvshmem); dtypes=bf16,fp8 ;; + 3) order=(deepep-v1-nvshmem deepep-v2-gin-gda uccl); dtypes=fp8,bf16 ;; + esac + for arm in "${order[@]}"; do + run_case "${arm}" "${world_size}" "${run_index}" "${dtypes}" + done + done +done + +python3 "${case_dir}/summarize_fair_results.py" "${ARTIFACT_ROOT}/runs" \ + --starts="${INDEPENDENT_STARTS}" \ + --json="${ARTIFACT_ROOT}/summary/summary.json" \ + --markdown="${ARTIFACT_ROOT}/summary/summary.md" +"${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-after.json" +"${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-after.json" +touch "${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" +printf 'PASS fair EP comparison completed at %s UTC\n' "$(date -u +%FT%TZ)" diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_rank.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_rank.sh new file mode 100755 index 000000000..1f5fea229 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_rank.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${EP_ARM:?Set EP_ARM}" +: "${EP_NODES:?Set EP_NODES}" +: "${EP_SERVICE:?Set EP_SERVICE}" +: "${EP_MASTER_ADDR:?Set EP_MASTER_ADDR}" +: "${POD_NAME:?Set POD_NAME}" +: "${EP_RUN_INDEX:?Set EP_RUN_INDEX}" +: "${EP_DISPATCH_DTYPES:?Set EP_DISPATCH_DTYPES}" +: "${EP_WARMUPS:=20}" +: "${EP_ITERATIONS:=100}" +: "${EP_SEED:=20260824}" +: "${EP_NCCL_DEBUG:=WARN}" + +ordinal="${POD_NAME##*-}" +export FI_PROVIDER=efa +export FI_EFA_USE_DEVICE_RDMA=1 +export NCCL_SOCKET_IFNAME='^lo,docker,veth' +export NCCL_NET_PLUGIN=ofi +export NCCL_DEBUG="${EP_NCCL_DEBUG}" +export CUDA_DEVICE_MAX_CONNECTIONS=1 + +case "${EP_ARM}" in + uccl) + export PER_EXPERT_BATCHING=1 + export UCCL_SOCKET_IFNAME='^lo,docker,veth' + ;; + deepep-v1-nvshmem) + export NVSHMEM_REMOTE_TRANSPORT=libfabric + export NVSHMEM_LIBFABRIC_PROVIDER=efa + export NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME='^lo,docker,veth' + export NVSHMEM_NETDEVS_POLICY=EXTERNAL_SHARING_PCIE_SWITCH_NIC_EXCLUSIVE + ;; + deepep-v2-gin-gda) + export FI_EFA_USE_HW_CNTR=1 + export NCCL_GIN_TYPE=5 + export NCCL_SYM_GIN_KERNELS_ENABLE=0 + export EP_BUFFER_DEBUG=1 + if [[ "${NCCL_DEBUG}" == INFO ]]; then + export NCCL_DEBUG_SUBSYS=INIT,ENV,NET + export FI_LOG_LEVEL=info + export FI_LOG_PROV=efa + export FI_LOG_SUBSYS=cntr + fi + ;; + *) + printf 'Unsupported EP_ARM=%s\n' "${EP_ARM}" >&2 + exit 2 + ;; +esac + +printf 'ADAI_FAIR_LAUNCH arm=%s nodes=%s ranks=%s run_index=%s_dimensionless dtype_order=%s warmups=%s_iterations measured=%s_iterations\n' \ + "${EP_ARM}" "${EP_NODES}" "$((EP_NODES * 8))" "${EP_RUN_INDEX}" \ + "${EP_DISPATCH_DTYPES}" "${EP_WARMUPS}" "${EP_ITERATIONS}" + +set +e +torchrun \ + --nnodes="${EP_NODES}" \ + --nproc-per-node=8 \ + --node-rank="${ordinal}" \ + --master-addr="${EP_MASTER_ADDR}" \ + --master-port=29400 \ + /opt/benchmark/fair_ep_benchmark.py \ + --arm="${EP_ARM}" \ + --tokens=128 \ + --hidden=7168 \ + --top-k=8 \ + --experts=256 \ + --seed="${EP_SEED}" \ + --warmups="${EP_WARMUPS}" \ + --iterations="${EP_ITERATIONS}" \ + --run-index="${EP_RUN_INDEX}" \ + --dispatch-dtypes="${EP_DISPATCH_DTYPES}" +status=$? +set -e + +printf 'ADAI_FAIR_EXIT_STATUS=%s_dimensionless\n' "${status}" +if ((status == 0)); then + printf 'ADAI_FAIR_COMPLETE\n' +else + printf 'ADAI_FAIR_FAILED\n' +fi +sleep infinity diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py new file mode 100755 index 000000000..23ecc24b4 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python3 +"""Validate and summarize common-boundary EP benchmark logs.""" + +from __future__ import annotations + +import argparse +import json +import random +import statistics +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable + + +PREFIX = "ADAI_FAIR_RESULT " +ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") +WORLD_SIZES = (16, 32) +DTYPES = ("fp8", "bf16") + + +def percentile(values: list[float], quantile: float) -> float: + ordered = sorted(values) + if not ordered: + raise ValueError("percentile requires at least one value") + position = (len(ordered) - 1) * quantile + lower = int(position) + upper = min(lower + 1, len(ordered) - 1) + fraction = position - lower + return ordered[lower] * (1 - fraction) + ordered[upper] * fraction + + +def bootstrap_median_ci( + values: list[float], seed: int, samples: int = 20_000 +) -> tuple[float, float]: + rng = random.Random(seed) + medians = [ + statistics.median(rng.choices(values, k=len(values))) for _ in range(samples) + ] + return percentile(medians, 0.025), percentile(medians, 0.975) + + +def load_results(root: Path) -> list[dict[str, Any]]: + results: dict[tuple[str, int, int, str], dict[str, Any]] = {} + sources: dict[tuple[str, int, int, str], Path] = {} + for path in sorted(root.rglob("*.log")): + for line in path.read_text(errors="replace").splitlines(): + if not line.startswith(PREFIX): + continue + result = json.loads(line[len(PREFIX) :]) + key = ( + result["arm"], + result["world_size_ranks"], + result["run_index_dimensionless"], + result["dispatch_dtype"], + ) + if key in results and results[key] != result: + raise ValueError( + f"conflicting result for {key}: {sources[key]} and {path}" + ) + results[key] = result + sources[key] = path + return list(results.values()) + + +def validate(results: list[dict[str, Any]], starts: int) -> None: + expected = { + (arm, world, run, dtype) + for arm in ARMS + for world in WORLD_SIZES + for run in range(1, starts + 1) + for dtype in DTYPES + } + observed = { + ( + result["arm"], + result["world_size_ranks"], + result["run_index_dimensionless"], + result["dispatch_dtype"], + ) + for result in results + if result["run_index_dimensionless"] > 0 + } + missing = expected - observed + extra = observed - expected + if missing or extra: + raise ValueError(f"result matrix mismatch; missing={sorted(missing)}, extra={sorted(extra)}") + + measured = [result for result in results if result["run_index_dimensionless"] > 0] + for result in measured: + if result["correctness"]["status"] != "PASS": + raise ValueError(f"correctness did not pass: {result}") + if "@sha256:" not in result["runtime"]["image_reference"]: + raise ValueError(f"image is not digest pinned: {result['runtime']['image_reference']}") + + for world in WORLD_SIZES: + same_world = [result for result in measured if result["world_size_ranks"] == world] + route_hashes = {result["route_hash_sha256"] for result in same_world} + input_hashes = {result["input_hash_sha256"] for result in same_world} + if len(route_hashes) != 1 or len(input_hashes) != 1: + raise ValueError( + f"EP{world} did not replay one route/input: " + f"routes={route_hashes}, inputs={input_hashes}" + ) + + +def arm_summary(results: Iterable[dict[str, Any]], seed: int) -> dict[str, Any]: + ordered = sorted(results, key=lambda item: item["run_index_dimensionless"]) + latencies = [item["latency_ms"]["median"] for item in ordered] + token_rates = [item["aggregate_input_tokens_per_second"] for item in ordered] + logical_rates = [ + item["effective_logical_gigabytes_per_second_per_rank"] for item in ordered + ] + scaleout_rates = [ + item["effective_scaleout_logical_gigabytes_per_second_per_rank"] + for item in ordered + ] + mean_latency = statistics.fmean(latencies) + stdev_latency = statistics.stdev(latencies) if len(latencies) > 1 else 0.0 + ci_low, ci_high = bootstrap_median_ci(latencies, seed) + return { + "starts": len(ordered), + "run_indices_dimensionless": [ + item["run_index_dimensionless"] for item in ordered + ], + "per_start_median_latency_ms": latencies, + "median_latency_ms": statistics.median(latencies), + "bootstrap_95_percent_ci_latency_ms": [ci_low, ci_high], + "run_to_run_cv_percent": ( + stdev_latency / mean_latency * 100 if mean_latency else 0.0 + ), + "median_aggregate_input_tokens_per_second": statistics.median(token_rates), + "median_effective_logical_gigabytes_per_second_per_rank": statistics.median( + logical_rates + ), + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": statistics.median( + scaleout_rates + ), + } + + +def summarize(results: list[dict[str, Any]], starts: int) -> dict[str, Any]: + measured = [result for result in results if result["run_index_dimensionless"] > 0] + by_cell_arm: dict[tuple[int, str, str], list[dict[str, Any]]] = defaultdict(list) + for result in measured: + by_cell_arm[ + (result["world_size_ranks"], result["dispatch_dtype"], result["arm"]) + ].append(result) + + cells = [] + for world in WORLD_SIZES: + for dtype in DTYPES: + arms = { + arm: arm_summary( + by_cell_arm[(world, dtype, arm)], + seed=20260824 + world + len(dtype) + index, + ) + for index, arm in enumerate(ARMS) + } + comparisons = {} + v2_by_run = { + result["run_index_dimensionless"]: result["latency_ms"]["median"] + for result in by_cell_arm[(world, dtype, "deepep-v2-gin-gda")] + } + for index, baseline in enumerate(("uccl", "deepep-v1-nvshmem")): + baseline_by_run = { + result["run_index_dimensionless"]: result["latency_ms"]["median"] + for result in by_cell_arm[(world, dtype, baseline)] + } + paired = [ + (baseline_by_run[run] - v2_by_run[run]) + / baseline_by_run[run] + * 100 + for run in range(1, starts + 1) + ] + ci_low, ci_high = bootstrap_median_ci( + paired, 20260900 + world + index + ) + stable = ( + arms[baseline]["run_to_run_cv_percent"] <= 5 + and arms["deepep-v2-gin-gda"]["run_to_run_cv_percent"] <= 5 + ) + comparisons[f"deepep-v2-gin-gda_vs_{baseline}"] = { + "paired_latency_reduction_percent_per_start": paired, + "median_paired_latency_reduction_percent": statistics.median( + paired + ), + "bootstrap_95_percent_ci_reduction_percent": [ci_low, ci_high], + "winner_supported": stable and (ci_low > 0 or ci_high < 0), + } + same_world = [result for result in measured if result["world_size_ranks"] == world] + cells.append( + { + "world_size_ranks": world, + "dispatch_dtype": dtype, + "route_hash_sha256": same_world[0]["route_hash_sha256"], + "input_hash_sha256": same_world[0]["input_hash_sha256"], + "arms": arms, + "comparisons": comparisons, + } + ) + return { + "schema_version_dimensionless": 1, + "status": "PASS", + "independent_starts_per_cell": starts, + "timing_boundary": measured[0]["timing_boundary"], + "logical_payload_definition": measured[0]["logical_payload_definition"], + "cells": cells, + } + + +def markdown(summary: dict[str, Any]) -> str: + lines = [ + "# Fair Common-Boundary EP Results", + "", + f"Each cell has {summary['independent_starts_per_cell']} independent process starts. Latency is the slowest-rank CUDA elapsed time from BF16 input readiness through dispatch and combine completion. Values are medians across independent starts.", + "", + "| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) |", + "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", + ] + for cell in summary["cells"]: + for arm in ARMS: + value = cell["arms"][arm] + ci = value["bootstrap_95_percent_ci_latency_ms"] + lines.append( + f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {arm} | " + f"{value['median_latency_ms']:.4f} ms | [{ci[0]:.4f}, {ci[1]:.4f}] ms | " + f"{value['run_to_run_cv_percent']:.2f}% | " + f"{value['median_aggregate_input_tokens_per_second']:.2f} tokens/s | " + f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " + f"{value['median_effective_scaleout_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank |" + ) + lines.extend( + [ + "", + "## Paired DeepEP V2 latency deltas", + "", + "Positive values mean DeepEP V2 had lower latency. A winner is supported only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV.", + "", + "| EP size | Dispatch dtype | Baseline | Median reduction (%) | 95% bootstrap CI (%) | Winner supported |", + "|---:|:---:|:---|---:|:---:|:---:|", + ] + ) + for cell in summary["cells"]: + for baseline in ("uccl", "deepep-v1-nvshmem"): + comparison = cell["comparisons"][f"deepep-v2-gin-gda_vs_{baseline}"] + ci = comparison["bootstrap_95_percent_ci_reduction_percent"] + lines.append( + f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {baseline} | " + f"{comparison['median_paired_latency_reduction_percent']:.2f}% | " + f"[{ci[0]:.2f}, {ci[1]:.2f}]% | " + f"{'yes' if comparison['winner_supported'] else 'no'} |" + ) + lines.append("") + return "\n".join(lines) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("root", type=Path) + parser.add_argument("--starts", type=int, default=3) + parser.add_argument("--json", type=Path, required=True) + parser.add_argument("--markdown", type=Path, required=True) + args = parser.parse_args() + results = load_results(args.root) + validate(results, args.starts) + summary = summarize(results, args.starts) + args.json.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n") + args.markdown.write_text(markdown(summary)) + print( + f"PASS fair EP matrix: {len(summary['cells'])} cells, " + f"{args.starts} independent starts per arm/cell" + ) + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py new file mode 100644 index 000000000..1765a99aa --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py @@ -0,0 +1,58 @@ +import importlib.util +from pathlib import Path +import sys +import unittest + +try: + import torch +except ModuleNotFoundError: + torch = None + + +MODULE = None +if torch is not None: + module_path = Path(__file__).with_name("fair_ep_benchmark.py") + spec = importlib.util.spec_from_file_location("fair_ep_benchmark", module_path) + assert spec and spec.loader + MODULE = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = MODULE + spec.loader.exec_module(MODULE) + + +@unittest.skipIf(torch is None, "PyTorch is required for benchmark helper tests") +class FairEpBenchmarkTest(unittest.TestCase): + def test_route_is_balanced_and_unique(self): + routes = [ + MODULE.make_route(rank, 128, 256, 8, 20260824, torch.device("cpu")) + for rank in range(32) + ] + route = torch.cat(routes) + histogram = torch.bincount(route.flatten(), minlength=256) + self.assertEqual(histogram.min().item(), 128) + self.assertEqual(histogram.max().item(), 128) + for row in route: + self.assertEqual(torch.unique(row).numel(), 8) + + def test_common_payload_formula(self): + route = MODULE.make_route(0, 128, 256, 8, 20260824, torch.device("cpu")) + bf16_all, bf16_remote, selections = MODULE.logical_payload_bytes_per_rank( + route, 0, 16, 8, 7168, "bf16", 256 + ) + fp8_all, fp8_remote, _ = MODULE.logical_payload_bytes_per_rank( + route, 0, 16, 8, 7168, "fp8", 256 + ) + self.assertEqual(selections, 1024) + self.assertEqual(bf16_all, 1024 * (7168 * 2 + 7168 * 2)) + self.assertEqual(fp8_all, 1024 * (7168 + 56 * 4 + 7168 * 2)) + self.assertGreater(bf16_remote, 0) + self.assertGreater(fp8_remote, 0) + self.assertLess(bf16_remote, bf16_all) + self.assertLess(fp8_remote, fp8_all) + + def test_percentile_interpolates(self): + self.assertEqual(MODULE.percentile([1.0, 2.0, 3.0], 0.5), 2.0) + self.assertAlmostEqual(MODULE.percentile([1.0, 2.0], 0.95), 1.95) + + +if __name__ == "__main__": + unittest.main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py new file mode 100644 index 000000000..008dfd809 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py @@ -0,0 +1,84 @@ +import importlib.util +from pathlib import Path +import sys +import unittest + + +module_path = Path(__file__).with_name("summarize_fair_results.py") +spec = importlib.util.spec_from_file_location("summarize_fair_results", module_path) +assert spec and spec.loader +summary_module = importlib.util.module_from_spec(spec) +sys.modules[spec.name] = summary_module +spec.loader.exec_module(summary_module) + + +def fake_result(arm, world_size, run_index, dtype, latency_ms): + digest = { + "uccl": "a", + "deepep-v1-nvshmem": "b", + "deepep-v2-gin-gda": "c", + }[arm] + return { + "arm": arm, + "world_size_ranks": world_size, + "run_index_dimensionless": run_index, + "dispatch_dtype": dtype, + "route_hash_sha256": f"route-{world_size}", + "input_hash_sha256": f"input-{world_size}", + "correctness": {"status": "PASS"}, + "runtime": {"image_reference": f"example.invalid/{arm}@sha256:{digest * 64}"}, + "latency_ms": {"median": latency_ms}, + "aggregate_input_tokens_per_second": 1000 / latency_ms, + "effective_logical_gigabytes_per_second_per_rank": 10 / latency_ms, + "effective_scaleout_logical_gigabytes_per_second_per_rank": 5 / latency_ms, + "timing_boundary": "common boundary", + "logical_payload_definition": "common payload", + } + + +class SummarizeFairResultsTest(unittest.TestCase): + def setUp(self): + arm_latency = { + "uccl": 1.0, + "deepep-v1-nvshmem": 1.2, + "deepep-v2-gin-gda": 0.8, + } + self.results = [ + fake_result( + arm, + world, + run, + dtype, + arm_latency[arm] * (1 + (run - 2) * 0.01), + ) + for arm in summary_module.ARMS + for world in summary_module.WORLD_SIZES + for run in range(1, 4) + for dtype in summary_module.DTYPES + ] + + def test_valid_matrix_and_paired_delta(self): + summary_module.validate(self.results, 3) + summary = summary_module.summarize(self.results, 3) + self.assertEqual(summary["status"], "PASS") + self.assertEqual(len(summary["cells"]), 4) + comparison = summary["cells"][0]["comparisons"][ + "deepep-v2-gin-gda_vs_uccl" + ] + self.assertAlmostEqual( + comparison["median_paired_latency_reduction_percent"], 20.0 + ) + self.assertTrue(comparison["winner_supported"]) + + def test_missing_start_is_rejected(self): + with self.assertRaises(ValueError): + summary_module.validate(self.results[:-1], 3) + + def test_route_mismatch_is_rejected(self): + self.results[0]["route_hash_sha256"] = "different" + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + +if __name__ == "__main__": + unittest.main() From 0acfea91cad8763291d02acbf4f8c64bfd6060b6 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 19:51:16 +0000 Subject: [PATCH 03/19] fix(ep): coordinate fair runs through shared lease --- .../ep-backend-comparison/README.md | 3 +- .../run_fair_ep_comparison.sh | 82 ++++++++++++++++++- 2 files changed, 81 insertions(+), 4 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index 6ace254a3..08801e998 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -49,12 +49,13 @@ The common logical-byte numerator counts the useful dispatch tensor, FP8 scales CAMPAIGN_ID=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) \ FAIR_EP_NODES=node-a,node-b,node-c,node-d \ PROTECTED_NODES_CSV=foreign-node-a,foreign-node-b \ -EXPECTED_LOCK_HOLDER=foreign-campaign-id \ ARTIFACT_ROOT=/shared/artifacts/${CAMPAIGN_ID} \ KUBECTL_CONTEXT=target-context \ ./run_fair_ep_comparison.sh ``` +The default `LOCK_MODE=exclusive` claims the shared Lease only when it is empty and releases it during verified teardown. If an explicitly coordinated campaign is still active on a disjoint node set, use `LOCK_MODE=observe EXPECTED_LOCK_HOLDER=foreign-campaign-id`; this mode checks the holder throughout the run and never mutates the Lease. + DeepEP V2 admission is conditional on `uvm_disable_hmm=Y` on every selected host and requires a logged GDAKI context. The runner aborts before the scored matrix if either condition is absent. ## Prerequisites diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh index 307bdc37f..c2d2b6f55 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh @@ -4,12 +4,14 @@ set -euo pipefail : "${CAMPAIGN_ID:?Set a unique CAMPAIGN_ID}" : "${FAIR_EP_NODES:?Set 4 comma-separated B200 node names}" : "${PROTECTED_NODES_CSV:?Set the concurrent campaign protected node names}" -: "${EXPECTED_LOCK_HOLDER:?Set the observed shared Lease holder}" : "${ARTIFACT_ROOT:?Set the durable artifact directory}" : "${KUBECTL_CONTEXT:=aps1}" : "${CAMPAIGN_NAMESPACE:=${CAMPAIGN_ID}}" : "${SHARED_LOCK_NAME:=adai-ap-south-1-gpu-campaign-lock}" : "${SHARED_LOCK_NAMESPACE:=default}" +: "${LOCK_MODE:=exclusive}" +: "${EXPECTED_LOCK_HOLDER:=}" +: "${LOCK_DURATION_SECONDS:=28800}" : "${INDEPENDENT_STARTS:=3}" : "${WARMUP_ITERATIONS:=20}" : "${MEASURED_ITERATIONS:=100}" @@ -21,6 +23,11 @@ set -euo pipefail [[ "${CAMPAIGN_ID}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] [[ "${CAMPAIGN_NAMESPACE}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] +[[ "${LOCK_MODE}" == exclusive || "${LOCK_MODE}" == observe ]] +if [[ "${LOCK_MODE}" == observe && -z "${EXPECTED_LOCK_HOLDER}" ]]; then + printf 'LOCK_MODE=observe requires EXPECTED_LOCK_HOLDER\n' >&2 + exit 2 +fi [[ "${INDEPENDENT_STARTS}" -eq 3 ]] || { printf 'This scored matrix requires exactly 3 independent starts\n' >&2 exit 2 @@ -58,18 +65,85 @@ declare -A images=( current_case="" namespace_created=0 +lock_claimed=0 check_shared_lock() { local holder holder="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ "${SHARED_LOCK_NAME}" -o jsonpath='{.spec.holderIdentity}' 2>/dev/null || true)" - if [[ -n "${holder}" && "${holder}" != "${EXPECTED_LOCK_HOLDER}" ]]; then + if [[ "${LOCK_MODE}" == exclusive ]]; then + [[ "${holder}" == "${CAMPAIGN_ID}" ]] || { + printf 'Exclusive shared Lease is no longer held by %s: holder=%s\n' \ + "${CAMPAIGN_ID}" "${holder}" >&2 + return 1 + } + elif [[ -n "${holder}" && "${holder}" != "${EXPECTED_LOCK_HOLDER}" ]]; then printf 'Shared Lease holder changed from protected campaign %s to %s\n' \ "${EXPECTED_LOCK_HOLDER}" "${holder}" >&2 return 1 fi } +claim_shared_lock() { + local attempt current holder now candidate + [[ "${LOCK_MODE}" == exclusive ]] || return 0 + for attempt in 1 2 3; do + current="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o json)" + holder="$(jq -r '.spec.holderIdentity // ""' <<<"${current}")" + if [[ "${holder}" == "${CAMPAIGN_ID}" ]]; then + lock_claimed=1 + printf '%s\n' "${current}" >"${ARTIFACT_ROOT}/control/shared-lease-claimed.json" + return 0 + fi + [[ -z "${holder}" ]] || { + printf 'Shared Lease is held by another campaign: %s\n' "${holder}" >&2 + return 1 + } + now="$(date -u +%FT%T.000000Z)" + candidate="$(jq \ + --arg holder "${CAMPAIGN_ID}" --arg now "${now}" \ + --argjson duration "${LOCK_DURATION_SECONDS}" ' + .spec.holderIdentity=$holder | + .spec.acquireTime=$now | + .spec.renewTime=$now | + .spec.leaseDurationSeconds=$duration | + .metadata.labels["adai.aws/campaign"]=$holder | + .metadata.labels["adai.aws/owner"]="fair-ep-comparison"' \ + <<<"${current}")" + if printf '%s\n' "${candidate}" | "${K[@]}" replace -f - \ + >"${ARTIFACT_ROOT}/control/shared-lease-claim-attempt-${attempt}.json" 2>&1; then + lock_claimed=1 + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/control/shared-lease-claimed.json" + return 0 + fi + done + printf 'Failed to claim shared Lease after 3 optimistic attempts\n' >&2 + return 1 +} + +release_shared_lock() { + local current holder now + ((lock_claimed == 1)) || return 0 + current="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o json 2>/dev/null || true)" + holder="$(jq -r '.spec.holderIdentity // ""' <<<"${current}")" + [[ "${holder}" == "${CAMPAIGN_ID}" ]] || { + printf 'Refusing to release shared Lease held by %s\n' "${holder}" >&2 + return 1 + } + now="$(date -u +%FT%T.000000Z)" + jq --arg now "${now}" ' + .spec.holderIdentity="" | + .spec.renewTime=$now | + .spec.leaseDurationSeconds=1 | + del(.metadata.labels["adai.aws/campaign"], .metadata.labels["adai.aws/owner"])' \ + <<<"${current}" | "${K[@]}" replace -f - \ + >"${ARTIFACT_ROOT}/teardown/shared-lease-release.json" + lock_claimed=0 +} + gpu_requests_on_node() { local node="$1" "${K[@]}" get pods -A --field-selector "spec.nodeName=${node}" -o json | jq ' @@ -122,6 +196,7 @@ finish() { teardown_status=1 fi fi + release_shared_lock || teardown_status=1 "${K[@]}" get all -A -l "adai.aws/campaign=${CAMPAIGN_ID}" -o json \ >"${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>&1 || teardown_status=1 remaining="$(jq '.items | length' "${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>/dev/null || printf '1')" @@ -145,12 +220,13 @@ trap finish EXIT trap 'exit 130' INT trap 'exit 143' TERM -check_shared_lock "${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-before.json" "${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-before.json" "${K[@]}" get namespaces -o json >"${ARTIFACT_ROOT}/control/namespaces-before.json" "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ >"${ARTIFACT_ROOT}/control/shared-lease-before.json" +claim_shared_lock +check_shared_lock aws sts get-caller-identity --output json >"${ARTIFACT_ROOT}/control/aws-caller-identity.json" printf '%s\n' "${selected_nodes[@]}" >"${ARTIFACT_ROOT}/control/selected-nodes.txt" printf '%s\n' "${protected_nodes[@]}" | sort -u >"${ARTIFACT_ROOT}/control/protected-nodes.txt" From 4d8a55b01c8c1de3d6b5b0a430d7fa0fedb8bb08 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 20:06:52 +0000 Subject: [PATCH 04/19] fix(ep): gate fair runs on GDRCopy device --- .../expert-parallelism/ep-backend-comparison/README.md | 2 +- .../ep-backend-comparison/run_fair_ep_comparison.sh | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index 08801e998..09ef59d53 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -56,7 +56,7 @@ KUBECTL_CONTEXT=target-context \ The default `LOCK_MODE=exclusive` claims the shared Lease only when it is empty and releases it during verified teardown. If an explicitly coordinated campaign is still active on a disjoint node set, use `LOCK_MODE=observe EXPECTED_LOCK_HOLDER=foreign-campaign-id`; this mode checks the holder throughout the run and never mutates the Lease. -DeepEP V2 admission is conditional on `uvm_disable_hmm=Y` on every selected host and requires a logged GDAKI context. The runner aborts before the scored matrix if either condition is absent. +DeepEP V2 admission is conditional on `uvm_disable_hmm=Y` and a character-device `/dev/gdrdrv` on every selected host, and requires a logged GDAKI context. The runner aborts before the scored matrix if any condition is absent. ## Prerequisites diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh index c2d2b6f55..90925443e 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh @@ -270,15 +270,19 @@ spec: printf 'NODE=%s\n' "\${NODE_NAME}" printf 'UVM_DISABLE_HMM='; cat /host-sys/module/nvidia_uvm/parameters/uvm_disable_hmm printf 'EFA_KMOD='; cat /host-sys/module/efa/version + printf 'GDRDRV='; if test -c /host-dev/gdrdrv; then echo character-device; else echo missing; fi sleep 300 env: - name: NODE_NAME valueFrom: {fieldRef: {fieldPath: spec.nodeName}} volumeMounts: - {name: host-sys, mountPath: /host-sys, readOnly: true} + - {name: host-dev, mountPath: /host-dev, readOnly: true} volumes: - name: host-sys hostPath: {path: /sys, type: Directory} + - name: host-dev + hostPath: {path: /dev, type: Directory} YAML done "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" wait --for=condition=Ready pod \ @@ -291,6 +295,11 @@ for index in 0 1 2 3; do "${selected_nodes[${index}]}" >&2 exit 1 } + rg -q '^GDRDRV=character-device$' "${ARTIFACT_ROOT}/control/host-audit-${index}.log" || { + printf 'Benchmark admission blocked by missing /dev/gdrdrv on node %s\n' \ + "${selected_nodes[${index}]}" >&2 + exit 1 + } done "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete pod -l \ "adai.aws/campaign=${CAMPAIGN_ID}" --wait=true --timeout=5m >/dev/null From 1bb50cf16ff6d9bf1389c7e203eef1a2a337af61 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 20:17:31 +0000 Subject: [PATCH 05/19] fix(ep): load built DeepEP V2 extension --- .../ep-backend-comparison/fair_ep_benchmark.py | 18 +++++++++++++++++- .../test_fair_ep_benchmark.py | 15 +++++++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py index 9fa21c52e..a48e3713a 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py @@ -32,6 +32,22 @@ SCHEMA_VERSION = 1 +def deepep_v2_build_lib(root: Path = Path("/opt/amazon/deepep-v2")) -> Path: + """Locate the single built DeepEP V2 package containing its C extension.""" + + candidates = [ + path + for path in sorted((root / "build").glob("lib.*")) + if any((path / "deep_ep").glob("_C*.so")) + ] + if len(candidates) != 1: + rendered = ", ".join(str(path) for path in candidates) or "none" + raise RuntimeError( + f"expected exactly one built DeepEP V2 package, found: {rendered}" + ) + return candidates[0] + + def percentile(values: list[float], quantile: float) -> float: """Return a linearly interpolated percentile without a NumPy dependency.""" @@ -208,7 +224,7 @@ def __init__( allow_mnnvl=False, ) elif arm == "deepep-v2-gin-gda": - sys.path.insert(0, "/opt/amazon/deepep-v2") + sys.path.insert(0, str(deepep_v2_build_lib())) import deep_ep # type: ignore[import-not-found] from deep_ep.utils.math import ( # type: ignore[import-not-found] per_token_cast_back, diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py index 1765a99aa..f34c9a481 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py @@ -1,6 +1,7 @@ import importlib.util from pathlib import Path import sys +import tempfile import unittest try: @@ -53,6 +54,20 @@ def test_percentile_interpolates(self): self.assertEqual(MODULE.percentile([1.0, 2.0, 3.0], 0.5), 2.0) self.assertAlmostEqual(MODULE.percentile([1.0, 2.0], 0.95), 1.95) + def test_deepep_v2_build_lib_requires_one_extension_package(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + package = root / "build" / "lib.linux-x86_64-cpython-312" / "deep_ep" + package.mkdir(parents=True) + (package / "_C.cpython-312-x86_64-linux-gnu.so").touch() + self.assertEqual(MODULE.deepep_v2_build_lib(root), package.parent) + + second = root / "build" / "lib.second" / "deep_ep" + second.mkdir(parents=True) + (second / "_C.so").touch() + with self.assertRaisesRegex(RuntimeError, "exactly one"): + MODULE.deepep_v2_build_lib(root) + if __name__ == "__main__": unittest.main() From f6f7287bf17d381ed9cdbe50ac1fa74353cb4c6f Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 20:22:39 +0000 Subject: [PATCH 06/19] fix(ep): import DeepEP V2 before NCCL setup --- .../ep-backend-comparison/fair_ep_benchmark.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py index a48e3713a..c3475834b 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py @@ -48,6 +48,14 @@ def deepep_v2_build_lib(root: Path = Path("/opt/amazon/deepep-v2")) -> Path: return candidates[0] +def preload_backend(arm: str) -> None: + """Load V2 before NCCL initializes its OFI plugin and tuner libraries.""" + + if arm == "deepep-v2-gin-gda": + sys.path.insert(0, str(deepep_v2_build_lib())) + __import__("deep_ep") + + def percentile(values: list[float], quantile: float) -> float: """Return a linearly interpolated percentile without a NumPy dependency.""" @@ -224,7 +232,6 @@ def __init__( allow_mnnvl=False, ) elif arm == "deepep-v2-gin-gda": - sys.path.insert(0, str(deepep_v2_build_lib())) import deep_ep # type: ignore[import-not-found] from deep_ep.utils.math import ( # type: ignore[import-not-found] per_token_cast_back, @@ -542,6 +549,7 @@ def parse_args() -> argparse.Namespace: def main() -> None: args = parse_args() + preload_backend(args.arm) rank, world_size, local_world_size, device, group = initialize_distributed() if args.experts % world_size: raise SystemExit("experts must divide the distributed world size") From 797d4cefb910d43a4117103a4c45f5600ff9dd5b Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 20:30:09 +0000 Subject: [PATCH 07/19] fix(ep): normalize V2 identity combine semantics --- .../fair_ep_benchmark.py | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py index c3475834b..fc89891e2 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py @@ -172,6 +172,7 @@ def logical_payload_bytes_per_rank( @dataclass class DispatchState: recv_x: Any + recv_topk_idx: torch.Tensor | None recv_topk_weights: torch.Tensor | None handle: Any @@ -278,7 +279,7 @@ def dispatch( dispatch_dtype: str, ) -> DispatchState: if self.is_elastic: - recv_x, _, recv_weights, handle, event = self.buffer.dispatch( + recv_x, recv_idx, recv_weights, handle, event = self.buffer.dispatch( x=prepared_x, topk_idx=topk_idx, topk_weights=topk_weights, @@ -291,7 +292,7 @@ def dispatch( do_cpu_sync=True, ) event.current_stream_wait() - return DispatchState(recv_x, recv_weights, handle) + return DispatchState(recv_x, recv_idx, recv_weights, handle) recv_x, _, handle, event, _ = self.buffer.low_latency_dispatch( prepared_x, @@ -303,7 +304,7 @@ def dispatch( return_recv_hook=False, ) event.current_stream_wait() - return DispatchState(recv_x, None, handle) + return DispatchState(recv_x, None, None, handle) def received_as_bf16(self, recv_x: Any, dispatch_dtype: str) -> torch.Tensor: if dispatch_dtype == "bf16": @@ -382,6 +383,18 @@ def run_dtype( correctness_input = adapter.received_as_bf16( correctness_state.recv_x, dispatch_dtype ) + if adapter.is_elastic: + assert correctness_state.recv_topk_idx is not None + assert correctness_state.recv_topk_weights is not None + # V2's non-expanded dispatch sends each token once per destination rank. + # Model the identity experts' local gated reduction before the network + # combine so that every backend returns the same weighted token. + local_weights = correctness_state.recv_topk_weights.masked_fill( + correctness_state.recv_topk_idx < 0, 0 + ).sum(dim=1, keepdim=True) + correctness_input = correctness_input * local_weights.to( + correctness_input.dtype + ) correctness_output = adapter.combine( correctness_input, correctness_state, route, topk_weights ) From 40623ef8e8987f26206231bfd1c5ff6b33ad55d3 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 20:53:16 +0000 Subject: [PATCH 08/19] fix(ep): parse Kubernetes GPU quantities --- .../ep-backend-comparison/run_fair_ep_comparison.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh index 90925443e..e3d9b99da 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh @@ -149,7 +149,7 @@ gpu_requests_on_node() { "${K[@]}" get pods -A --field-selector "spec.nodeName=${node}" -o json | jq ' [.items[] | select(.status.phase != "Succeeded" and .status.phase != "Failed") - | [.spec.containers[]?.resources.requests["nvidia.com/gpu"] // 0] + | [.spec.containers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] | add // 0] | add // 0' } From 48af02d5b216f61240c79c5f7c3cbeef200b51bd Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 21:00:26 +0000 Subject: [PATCH 09/19] fix(ep): wait for benchmark pods to exit --- .../run_fair_ep_comparison.sh | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh index e3d9b99da..1c65d7ee9 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh @@ -170,19 +170,26 @@ verify_node_free() { } cleanup_case() { + local case_to_delete cleanup_status=0 [[ -n "${current_case}" ]] || return 0 - "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete statefulset "${current_case}" \ - --ignore-not-found --wait=true --timeout=5m >/dev/null 2>&1 || true - "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete service "${current_case}" \ - --ignore-not-found >/dev/null 2>&1 || true + case_to_delete="${current_case}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete statefulset "${case_to_delete}" \ + --ignore-not-found --wait=true --timeout=5m >/dev/null 2>&1 || cleanup_status=1 + # StatefulSet deletion can return before its cascading Pod deletions finish. + # Wait for the GPU requests to disappear before admitting the next arm. + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" wait --for=delete pod \ + -l "app=${case_to_delete}" --timeout=5m >/dev/null 2>&1 || cleanup_status=1 + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete service "${case_to_delete}" \ + --ignore-not-found >/dev/null 2>&1 || cleanup_status=1 current_case="" + return "${cleanup_status}" } finish() { local command_status=$? teardown_status=0 owned="" remaining=0 trap - EXIT INT TERM set +e - cleanup_case + cleanup_case || teardown_status=1 if ((namespace_created == 1)); then owned="$("${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" \ -o jsonpath='{.metadata.labels.adai\.aws/campaign}' 2>/dev/null || true)" From 249f9e53113cf7706813b19520ddba2c2983e7d6 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 21:09:30 +0000 Subject: [PATCH 10/19] fix(ep): reshape noncontiguous FP8 scales --- .../fair_ep_benchmark.py | 6 ++--- .../test_fair_ep_benchmark.py | 22 +++++++++++++++++++ 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py index fc89891e2..f516444f0 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py @@ -313,9 +313,9 @@ def received_as_bf16(self, recv_x: Any, dispatch_dtype: str) -> torch.Tensor: return self._cast_back(recv_x[0], recv_x[1]) fp8, scales = recv_x return self._cast_back( - fp8.view(-1, self.hidden), - scales.view(-1, self.hidden // 128), - ).view(fp8.shape) + fp8.reshape(-1, self.hidden), + scales.reshape(-1, self.hidden // 128), + ).reshape(fp8.shape) def combine( self, diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py index f34c9a481..74c0fdea2 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py @@ -68,6 +68,28 @@ def test_deepep_v2_build_lib_requires_one_extension_package(self): with self.assertRaisesRegex(RuntimeError, "exactly one"): MODULE.deepep_v2_build_lib(root) + def test_received_fp8_accepts_noncontiguous_scales(self): + adapter = object.__new__(MODULE.BackendAdapter) + adapter.arm = "uccl" + adapter.hidden = 256 + observed = {} + + def cast_back(fp8, scales): + observed["fp8_shape"] = tuple(fp8.shape) + observed["scales_shape"] = tuple(scales.shape) + return torch.zeros((fp8.shape[0], 256), dtype=torch.bfloat16) + + adapter._cast_back = cast_back + fp8 = torch.zeros((2, 3, 256), dtype=torch.float8_e4m3fn) + scales = torch.arange(12, dtype=torch.float32).reshape(2, 6).t() + self.assertFalse(scales.is_contiguous()) + + received = adapter.received_as_bf16((fp8, scales), "fp8") + + self.assertEqual(observed["fp8_shape"], (6, 256)) + self.assertEqual(observed["scales_shape"], (6, 2)) + self.assertEqual(tuple(received.shape), (2, 3, 256)) + if __name__ == "__main__": unittest.main() From bb58561f5518d684ab9b992bbf6d467d0f82c406 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 22:29:37 +0000 Subject: [PATCH 11/19] fix(ep): parse results from interleaved logs --- .../extract_fair_results.py | 29 ++++++++++++ .../ep-backend-comparison/fair_result_io.py | 45 +++++++++++++++++++ .../run_fair_ep_comparison.sh | 5 ++- .../summarize_fair_results.py | 8 ++-- .../test_summarize_fair_results.py | 12 +++++ 5 files changed, 92 insertions(+), 7 deletions(-) create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_fair_results.py create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_result_io.py diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_fair_results.py new file mode 100644 index 000000000..2e71741b5 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_fair_results.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +"""Extract canonical JSONL result records from a mixed benchmark log.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from fair_result_io import PREFIX, load_result_log + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("input_log", type=Path) + parser.add_argument("output_jsonl", type=Path) + args = parser.parse_args() + + results = load_result_log(args.input_log) + if not results: + raise SystemExit(f"no fair results found in {args.input_log}") + args.output_jsonl.write_text( + "".join(PREFIX + json.dumps(result, sort_keys=True) + "\n" for result in results) + ) + print(f"extracted {len(results)} fair results from {args.input_log}") + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_result_io.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_result_io.py new file mode 100644 index 000000000..94f32d10e --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_result_io.py @@ -0,0 +1,45 @@ +"""Parse fair benchmark result markers despite interleaved native stdout.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Iterator + + +PREFIX = "ADAI_FAIR_RESULT " + + +def iter_result_objects(text: str, source: str = "") -> Iterator[dict[str, Any]]: + """Yield JSON objects following result markers in mixed process output. + + Native libraries can flush a diagnostic after Python has emitted a JSON + object but before its newline reaches the combined log. JSONDecoder's + raw_decode identifies the exact end of the object without treating that + trailing diagnostic as part of the result. + """ + + decoder = json.JSONDecoder() + for line_number, line in enumerate(text.splitlines(), 1): + search_from = 0 + while True: + marker = line.find(PREFIX, search_from) + if marker < 0: + break + payload_start = marker + len(PREFIX) + try: + result, consumed = decoder.raw_decode(line[payload_start:]) + except json.JSONDecodeError as error: + raise ValueError( + f"invalid fair result in {source}:{line_number}: {error}" + ) from error + if not isinstance(result, dict): + raise ValueError( + f"fair result in {source}:{line_number} is not a JSON object" + ) + yield result + search_from = payload_start + consumed + + +def load_result_log(path: Path) -> list[dict[str, Any]]: + return list(iter_result_objects(path.read_text(errors="replace"), str(path))) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh index 1c65d7ee9..53d021475 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh @@ -471,12 +471,13 @@ YAML return 1 } local rank_zero_log="${out}/${current_case}-0.log" result_count - result_count="$(rg -c '^ADAI_FAIR_RESULT ' "${rank_zero_log}" || true)" + python3 "${case_dir}/extract_fair_results.py" \ + "${rank_zero_log}" "${out}/results.jsonl" >/dev/null + result_count="$(wc -l <"${out}/results.jsonl")" [[ "${result_count}" -eq 2 ]] || { printf 'Expected 2 fair results in %s, found %s\n' "${rank_zero_log}" "${result_count}" >&2 return 1 } - rg '^ADAI_FAIR_RESULT ' "${rank_zero_log}" >"${out}/results.jsonl" printf 'PASS arm=%s EP%s repeat=%s label=%s at %s UTC\n' \ "${arm}" "${world_size}" "${run_index}" "${label}" "$(date -u +%FT%TZ)" \ >"${out}/STATUS" diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py index 23ecc24b4..891cfbb0d 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py @@ -11,8 +11,9 @@ from pathlib import Path from typing import Any, Iterable +from fair_result_io import load_result_log + -PREFIX = "ADAI_FAIR_RESULT " ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") WORLD_SIZES = (16, 32) DTYPES = ("fp8", "bf16") @@ -43,10 +44,7 @@ def load_results(root: Path) -> list[dict[str, Any]]: results: dict[tuple[str, int, int, str], dict[str, Any]] = {} sources: dict[tuple[str, int, int, str], Path] = {} for path in sorted(root.rglob("*.log")): - for line in path.read_text(errors="replace").splitlines(): - if not line.startswith(PREFIX): - continue - result = json.loads(line[len(PREFIX) :]) + for result in load_result_log(path): key = ( result["arm"], result["world_size_ranks"], diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py index 008dfd809..9c9b88f3c 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py @@ -1,6 +1,7 @@ import importlib.util from pathlib import Path import sys +import tempfile import unittest @@ -79,6 +80,17 @@ def test_route_mismatch_is_rejected(self): with self.assertRaises(ValueError): summary_module.validate(self.results, 3) + def test_load_results_accepts_native_diagnostic_after_json(self): + result = fake_result("deepep-v2-gin-gda", 32, 1, "fp8", 0.9) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "rank-zero.log" + path.write_text( + "ADAI_FAIR_RESULT " + + summary_module.json.dumps(result) + + "Elastic buffer uses 3 channels per SM\n" + ) + self.assertEqual(summary_module.load_results(Path(directory)), [result]) + if __name__ == "__main__": unittest.main() From f834b300466c5dc8e48949bd224197b2cf573500 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 22:39:22 +0000 Subject: [PATCH 12/19] docs(ep): report fair B200 rerun --- .../ep-backend-comparison/README.md | 4 +- .../ep-backend-comparison/RESULTS-b200.md | 131 ++++++++++++++---- 2 files changed, 110 insertions(+), 25 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index 09ef59d53..fd95aee59 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -39,12 +39,14 @@ If the DeepEP values differ from 4096/7168/8/256, edit the bench args in ### Fair common-boundary decode comparison -The backend-native reports above do not share one timing boundary or byte numerator. Use [`fair_ep_benchmark.py`](fair_ep_benchmark.py) when making a direct cross-backend latency or effective-bandwidth claim. It supplies all 3 backends with the same deterministic BF16 input, exact top-k route, and weights, then times BF16-input readiness through dispatch and combine completion with one CUDA Event boundary. Each iteration uses the slowest rank's elapsed time. +The backend-native benchmark outputs do not share one timing boundary or byte numerator. Use [`fair_ep_benchmark.py`](fair_ep_benchmark.py) when making a direct cross-backend latency or effective-bandwidth claim. It supplies all 3 backends with the same deterministic BF16 input, exact top-k route, and weights, then times BF16-input readiness through dispatch and combine completion with one CUDA Event boundary. Each iteration uses the slowest rank's elapsed time. The common logical-byte numerator counts the useful dispatch tensor, FP8 scales when selected, and BF16 combine tensor for each valid expert assignment. It excludes backend metadata. Backend-native UCCL, NVSHMEM RDMA, and DeepEP V2 SO/SU measurements remain useful diagnostics, but they are reported separately. [`run_fair_ep_comparison.sh`](run_fair_ep_comparison.sh) runs the B200 decode matrix at 128 tokens/rank for EP16 and EP32. It uses the same named nodes for every arm, performs 3 independent process starts per cell in rotated arm order, validates route and input hashes, and tears down only its labeled namespace. The concurrent campaign's nodes and shared Lease are read and protected, never modified. +The completed 24 August 2026 rerun, including common logical throughput, paired latency deltas, 95% bootstrap intervals, correctness, and scope limits, is reported in [`RESULTS-b200.md`](RESULTS-b200.md#fair-common-boundary-decode-rerun). Keep its primary table separate from the historical backend-native tables on the same page. + ```bash CAMPAIGN_ID=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) \ FAIR_EP_NODES=node-a,node-b,node-c,node-d \ diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md index f7ab70cc9..ba96ae734 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md @@ -1,16 +1,99 @@ # EP-Backend Comparison Results: B200 with DeepEP V2 -This page records the matched direct expert-parallelism campaign completed on B200 in `ap-south-1`. It adds DeepEP V2 over NCCL GIN EFA-GDA to the UCCL and DeepEP V1 comparison. These results are separate from the historical [B300 results](RESULTS.md) and [P5/H100 results](RESULTS-p5.md). Results from different GPU generations or campaigns are not combined. +This page records 2 separate B200 campaigns in `ap-south-1`. The fair common-boundary decode rerun completed on 24 August 2026 is the primary cross-backend comparison. The earlier backend-native campaign is retained below as historical diagnostic evidence only. These results are separate from the historical [B300 results](RESULTS.md) and [P5/H100 results](RESULTS-p5.md). Results from different campaigns, timing boundaries, byte-accounting conventions, or GPU generations are not combined. -## Result summary +## Fair common-boundary decode rerun -DeepEP V2 had the lowest measured decode dispatch-plus-combine latency in all 4 B200 cells. Relative to UCCL, its latency was 19.04 percent lower for EP16 FP8, 44.70 percent lower for EP16 BF16, 6.82 percent lower for EP32 FP8, and 18.88 percent lower for EP32 BF16. Relative to DeepEP V1, the reductions ranged from 52.22 percent to 56.34 percent. +The rerun passed all 36 scored dtype results from 18 distributed jobs. UCCL had the lowest median latency in 3 of 4 workload cells: EP16 FP8, EP32 FP8, and EP32 BF16. DeepEP V2 had the lowest median latency in EP16 BF16. DeepEP V2 was faster than DeepEP V1 in all 4 workload cells. -This does not establish an overall backend winner. Each cell has 1 independent benchmark start, so run-to-run variance and coefficient of variation are unavailable. This scored B200 comparison contains direct EP tests only; it produced no DeepEP V2 serving throughput, TTFT, TPOT, E2E latency, or NIXL result. DeepEP V2 also used substantially more HBM for prefill than the other 2 backends. +These conclusions apply only to the exact synthetic dispatch-plus-combine workload and timing boundary below. They do not establish end-to-end training or serving throughput, prefill performance, or a universal backend winner. Each arm and workload cell has 3 independent process starts. The 95% bootstrap intervals therefore summarize these 3 starts and should not be treated as a high-power population estimate. -The synthetic DeepEP source contains PR 5, but EP16 and EP32 do not validate its greater-than-20-NVLink-domain scale path. The independent 256-rank PR 5 gate was not completed and no PR 5 scale-success claim is inferred from these performance cells. +### Common workload and method -## Environment and provenance +| Field | Value | +|---|---| +| Measurement period | 24 August 2026 | +| Region and Availability Zone | `ap-south-1`, `ap-south-1c` | +| EKS cluster | `ml-clusters-shared-ap-south-1` | +| Hardware | `p6-b200.48xlarge`, 8 B200 GPUs and 8 EFA devices per node | +| EP16 topology | 2 nodes, 16 ranks | +| EP32 topology | 4 nodes, 32 ranks | +| Workload | 128 tokens/rank, hidden size 7,168, top-k 8, experts 256 | +| Dispatch dtype | FP8 and BF16 | +| Combine dtype | BF16 | +| Timing boundary | BF16 input ready through dispatch and combine completion, measured with CUDA Events and reduced to the slowest rank | +| Logical payload | Useful dispatch tensor, FP8 scales when selected, and BF16 combine tensor for every valid expert assignment; backend metadata excluded | +| Iterations | 20 warmup iterations and 100 measured iterations per dtype and start | +| Replication | 3 independent process starts per arm and workload cell | +| Order control | Backend order rotated across starts; dtype order also rotated | +| Input control | Deterministic BF16 input, exact top-k route, and weights; matching SHA-256 input and route hashes required within each EP size | +| Correctness | 36 of 36 scored dtype results passed; FP8 tolerance `9e-4` dimensionless and BF16 tolerance `1e-5` dimensionless | + +All 3 images used PyTorch `2.13.0+cu130`, CUDA 13.0, NCCL 2.29.7, and digest-pinned images: + +| Backend | Scored image digest | +|---|---| +| UCCL | `sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a` | +| DeepEP V1 NVSHMEM | `sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac` | +| DeepEP V2 NCCL GIN | `sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb` | + +The 3 backends ran serially on the same named nodes for each EP size. A concurrent vLLM campaign used 12 disjoint nodes, with 0 selected-node overlap. This campaign observed the shared Lease and did not mutate the vLLM campaign's Lease, nodes, namespace, images, or artifacts. Because the campaigns shared the cluster fabric, unobserved background-fabric effects cannot be excluded completely; order rotation and paired starts reduce but do not eliminate that risk. + +### Common-boundary results + +Values are medians across 3 independent starts. Logical throughput uses the common logical-byte numerator defined above, not a backend-native bandwidth field. Lower latency is better. + +| EP size | Dispatch dtype | Backend | Median latency | 95% bootstrap CI | Run-to-run CV | Input throughput | Logical throughput | Scale-out logical throughput | +|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| +| 16 ranks | FP8 | UCCL | **0.5213 ms** | [0.5190, 0.5277] ms | 0.86% | 3,928,790.51 tokens/s | 42.68 GB/s/rank | 21.34 GB/s/rank | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 1.0179 ms | [1.0167, 1.0203] ms | 0.18% | 2,011,977.59 tokens/s | 21.86 GB/s/rank | 10.93 GB/s/rank | +| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 0.6219 ms | [0.6180, 0.6375] ms | 1.65% | 3,293,197.58 tokens/s | 35.78 GB/s/rank | 17.89 GB/s/rank | +| 16 ranks | BF16 | UCCL | 0.5920 ms | [0.5918, 0.5978] ms | 0.58% | 3,459,365.90 tokens/s | 49.59 GB/s/rank | 24.80 GB/s/rank | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 1.0297 ms | [1.0251, 1.0304] ms | 0.28% | 1,988,936.56 tokens/s | 28.51 GB/s/rank | 14.26 GB/s/rank | +| 16 ranks | BF16 | DeepEP V2 NCCL GIN | **0.4850 ms** | [0.4826, 0.5043] ms | 2.42% | 4,222,332.23 tokens/s | 60.53 GB/s/rank | 30.27 GB/s/rank | +| 32 ranks | FP8 | UCCL | **0.7655 ms** | [0.7640, 0.7714] ms | 0.51% | 5,350,835.03 tokens/s | 29.07 GB/s/rank | 21.80 GB/s/rank | +| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 1.5542 ms | [1.5513, 1.5628] ms | 0.38% | 2,635,371.65 tokens/s | 14.32 GB/s/rank | 10.74 GB/s/rank | +| 32 ranks | FP8 | DeepEP V2 NCCL GIN | 0.9537 ms | [0.9217, 0.9675] ms | 2.48% | 4,294,725.45 tokens/s | 23.33 GB/s/rank | 17.50 GB/s/rank | +| 32 ranks | BF16 | UCCL | **0.8686 ms** | [0.8683, 0.8696] ms | 0.08% | 4,715,590.93 tokens/s | 33.80 GB/s/rank | 25.35 GB/s/rank | +| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 1.5589 ms | [1.5526, 1.5714] ms | 0.61% | 2,627,446.53 tokens/s | 18.83 GB/s/rank | 14.13 GB/s/rank | +| 32 ranks | BF16 | DeepEP V2 NCCL GIN | 0.9509 ms | [0.9482, 0.9606] ms | 0.68% | 4,307,661.17 tokens/s | 30.88 GB/s/rank | 23.16 GB/s/rank | + +### Paired DeepEP V2 latency deltas + +Positive values mean DeepEP V2 had lower latency than the named baseline. A directional winner is supported for this workload only when the paired 95% bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV. + +| EP size | Dispatch dtype | Baseline | Median V2 latency reduction | 95% bootstrap CI | Direction supported | +|---:|:---:|:---|---:|:---:|:---:| +| 16 ranks | FP8 | UCCL | -19.30% | [-22.84, -17.11]% | UCCL lower | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 39.05% | [37.37, 39.22]% | V2 lower | +| 16 ranks | BF16 | UCCL | 18.44% | [14.82, 18.87]% | V2 lower | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 52.68% | [51.06, 53.13]% | V2 lower | +| 32 ranks | FP8 | UCCL | -24.83% | [-25.41, -20.40]% | UCCL lower | +| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 38.64% | [37.64, 41.02]% | V2 lower | +| 32 ranks | BF16 | UCCL | -9.35% | [-10.63, -9.16]% | UCCL lower | +| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 38.87% | [38.76, 39.18]% | V2 lower | + +The common logical throughput columns are the inverse-latency view of the same common payload and timing boundary, so they produce the same ranking as latency. Backend-native UCCL aggregate bandwidth, DeepEP V1 bandwidth, and DeepEP V2 SO/SU bandwidth use different numerators or aggregation boundaries and remain diagnostic only. + +### Fair-rerun provenance and custody + +The durable campaign ID is `fair-ep-b200-20260824t195145z`. Its machine-readable summary validates the complete 2-EP-size by 2-dtype by 3-backend by 3-start matrix, matching input and route hashes, digest-pinned images, and per-result correctness. The DeepEP V2 admission additionally required HMM disabled, `/dev/gdrdrv` present as a character device, and a logged GDAKI context before scoring. + +The 4 selected nodes were Ready after the run, each with 8 allocatable GPUs and 8 allocatable EFA devices. The benchmark namespace and all campaign-labeled pods, Jobs, and MPIJobs were absent after teardown. The shared Lease still belonged to the concurrent vLLM campaign. + +## Earlier backend-native campaign + +The remaining sections preserve the earlier campaign's backend-native measurements. Their timing boundaries and byte numerators differ by backend, so the latency and GB/s columns must not be used for a direct cross-backend ranking. The numerically lower V2 values reported in that campaign remain observations of the native benchmark outputs, not evidence that V2 won a fair common-boundary comparison. + +### Native campaign summary + +Using each backend's native fields, the DeepEP V2 dispatch-plus-combine sum was numerically lower in all 4 B200 cells. Relative to the UCCL native aggregate, the differences were 19.04% for EP16 FP8, 44.70% for EP16 BF16, 6.82% for EP32 FP8, and 18.88% for EP32 BF16. Relative to the DeepEP V1 native aggregate, the differences ranged from 52.22% to 56.34%. Because the measurement boundaries differ, these percentages are retained as historical diagnostics and are not fair cross-backend speedups. + +Each historical cell has 1 independent benchmark start, so run-to-run variance and coefficient of variation are unavailable. This campaign contains direct EP tests only; it produced no DeepEP V2 serving throughput, TTFT, TPOT, E2E latency, or NIXL result. DeepEP V2 also used substantially more HBM for prefill than the other 2 backends. + +The synthetic DeepEP source contains PR 5, but EP16 and EP32 do not validate its greater-than-20-NVLink-domain scale path. The independent 256-rank PR 5 gate was not completed, and no PR 5 scale-success claim is inferred from these performance cells. + +### Native campaign environment and provenance | Field | Value | |---|---| @@ -39,18 +122,18 @@ The DeepEP V2 source was synthesized reproducibly from base `02efc268a37802fc008 Before measurement, an INFO-level EP16 admission ran on both nodes with `NCCL_GIN_TYPE=5` and `NCCL_SYM_GIN_KERNELS_ENABLE=0`. Logs showed the aws-ofi-nccl `Libfabric_GDAKI` version 14 dimensionless plugin, GPU Direct RDMA enabled on EFA HCAs, nonzero GIN layouts, and balanced cluster EFA TX and RX deltas of 235,008 bytes each. Every scored cell also recorded nonzero EFA RDMA writes, so the cross-node path was measured rather than inferred from topology. -## Decode latency comparison +## Historical native decode diagnostics -Lower latency is better. UCCL and DeepEP V1 print an aggregate dispatch-plus-combine latency. DeepEP V2 prints dispatch and combine separately, so the V2 value below is their sum for the same dispatch dtype. +These values come from backend-native reports, not the common-boundary harness. UCCL and DeepEP V1 print an aggregate dispatch-plus-combine latency. DeepEP V2 prints dispatch and combine separately, so the V2 value below is their sum for the same dispatch dtype. The boundaries differ, and the table does not support a direct latency ranking. | EP ranks | Dispatch dtype | UCCL latency | DeepEP V1 latency | DeepEP V2 latency | V2 vs UCCL | V2 vs V1 | |---:|---|---:|---:|---:|---:|---:| -| 16 ranks | FP8 | 475.540 us | 881.700 us | **384.994 us** | 19.04 percent lower | 56.34 percent lower | -| 16 ranks | BF16 | 710.180 us | 886.920 us | **392.738 us** | 44.70 percent lower | 55.72 percent lower | -| 32 ranks | FP8 | 693.840 us | 1,403.720 us | **646.522 us** | 6.82 percent lower | 53.94 percent lower | -| 32 ranks | BF16 | 844.520 us | 1,433.780 us | **685.084 us** | 18.88 percent lower | 52.22 percent lower | +| 16 ranks | FP8 | 475.540 us | 881.700 us | 384.994 us | 19.04 percent lower | 56.34 percent lower | +| 16 ranks | BF16 | 710.180 us | 886.920 us | 392.738 us | 44.70 percent lower | 55.72 percent lower | +| 32 ranks | FP8 | 693.840 us | 1,403.720 us | 646.522 us | 6.82 percent lower | 53.94 percent lower | +| 32 ranks | BF16 | 844.520 us | 1,433.780 us | 685.084 us | 18.88 percent lower | 52.22 percent lower | -UCCL had the smallest latency increase when scaling from EP16 to EP32, while DeepEP V2 remained fastest in absolute latency. For FP8, the increases were 45.91 percent for UCCL, 59.21 percent for DeepEP V1, and 67.93 percent for DeepEP V2. This is a single-start scale observation, not a scaling-efficiency claim. +In the native outputs, UCCL had the smallest numerical latency increase when scaling from EP16 to EP32, while the DeepEP V2 sum remained numerically smallest. For FP8, the increases were 45.91% for UCCL, 59.21% for DeepEP V1, and 67.93% for DeepEP V2. This is a single-start diagnostic observation across different timing boundaries, not a scaling-efficiency claim. The native decode bandwidth reports are included for completeness. UCCL and DeepEP V1 report one dispatch-plus-combine bandwidth; DeepEP V2 reports separate SO legs. The V2 dispatch/combine pair is not an aggregate and is not directly comparable to the first 2 columns. @@ -61,20 +144,20 @@ The native decode bandwidth reports are included for completeness. UCCL and Deep | 32 ranks | FP8 | 31.78 GB/s | 15.71 GB/s | 10 GB/s / 14 GB/s | | 32 ranks | BF16 | 34.43 GB/s | 20.28 GB/s | 16 GB/s / 14 GB/s | -## Prefill comparison +## Historical native prefill diagnostics The table reports each backend's native latency and cross-node bandwidth fields. For UCCL, latency is `transmit + notify` and bandwidth is the RDMA leg. For DeepEP V1, latency is the sum of the 2 printed components and bandwidth is the RDMA leg. For DeepEP V2, latency is the operation latency and bandwidth is the scale-out, or SO, leg. These measurement boundaries and bandwidth accounting conventions differ, so the table is directional. Do not rank the backends by comparing the GB/s columns alone. | EP ranks | Operation | UCCL latency / RDMA bandwidth | DeepEP V1 latency / RDMA bandwidth | DeepEP V2 latency / SO bandwidth | |---:|---|---:|---:|---:| -| 16 ranks | FP8 dispatch | 1,161.120 us / 55.59 GB/s | 1,014.000 us / 65.95 GB/s | **864.688 us** / 35 GB/s | -| 16 ranks | BF16 dispatch | 1,637.970 us / 74.42 GB/s | 1,639.000 us / 78.14 GB/s | **1,501.000 us** / 39 GB/s | -| 16 ranks | BF16 combine | 2,096.580 us / 58.81 GB/s | 1,633.220 us / 78.62 GB/s | **1,615.000 us** / 36 GB/s | -| 32 ranks | FP8 dispatch | 2,229.950 us / 51.38 GB/s | 2,369.000 us / 50.95 GB/s | **2,041.000 us** / 41 GB/s | -| 32 ranks | BF16 dispatch | 3,852.970 us / 56.69 GB/s | 4,125.000 us / 56.94 GB/s | **3,753.000 us** / 43 GB/s | -| 32 ranks | BF16 combine | 3,950.790 us / 55.52 GB/s | 4,106.520 us / 55.80 GB/s | **3,745.000 us** / 43 GB/s | +| 16 ranks | FP8 dispatch | 1,161.120 us / 55.59 GB/s | 1,014.000 us / 65.95 GB/s | 864.688 us / 35 GB/s | +| 16 ranks | BF16 dispatch | 1,637.970 us / 74.42 GB/s | 1,639.000 us / 78.14 GB/s | 1,501.000 us / 39 GB/s | +| 16 ranks | BF16 combine | 2,096.580 us / 58.81 GB/s | 1,633.220 us / 78.62 GB/s | 1,615.000 us / 36 GB/s | +| 32 ranks | FP8 dispatch | 2,229.950 us / 51.38 GB/s | 2,369.000 us / 50.95 GB/s | 2,041.000 us / 41 GB/s | +| 32 ranks | BF16 dispatch | 3,852.970 us / 56.69 GB/s | 4,125.000 us / 56.94 GB/s | 3,753.000 us / 43 GB/s | +| 32 ranks | BF16 combine | 3,950.790 us / 55.52 GB/s | 4,106.520 us / 55.80 GB/s | 3,745.000 us / 43 GB/s | -## DeepEP V2 operation results +## Historical DeepEP V2 operation diagnostics SO is scale-out bandwidth over the internode path and SU is scale-up bandwidth over the intranode path. FP8 combine is included because the selected V2 test prints it, but the matched headline operation set uses BF16 combine. @@ -99,7 +182,7 @@ SO is scale-out bandwidth over the internode path and SU is scale-up bandwidth o The DeepEP V2 EFA RDMA write deltas were 19,708,762,240 bytes for EP16 decode, 111,413,870,476 bytes for EP32 decode, 507,290,310,848 bytes for EP16 prefill, and 2,742,751,387,856 bytes for EP32 prefill. -## Memory tradeoff +## Historical native memory tradeoff All 12 cells reached 100 percent sampled peak GPU utilization. Peak HBM use was materially higher for DeepEP V2 prefill. @@ -109,11 +192,11 @@ All 12 cells reached 100 percent sampled peak GPU utilization. Peak HBM use was | DeepEP V1 NVSHMEM | 10,597 MiB/GPU | 11,239 MiB/GPU | 17,047 MiB/GPU | 23,793 MiB/GPU | | DeepEP V2 NCCL GIN | **7,041 MiB/GPU** | **7,271 MiB/GPU** | 39,657 MiB/GPU | 44,315 MiB/GPU | -## Subsequent EFA 3.3.0g revalidation +## Separate EFA 3.3.0g revalidation On 24 August 2026, a separate revalidation updated the current 32-node B200 fleet to the EFA 3.3.0g kernel module and rdma-core 64 on kernel `6.12.100-125.179.amzn2023.x86_64`. The post-update audit passed on 32 of 32 nodes. The first targeted DeepEP V2 admission then triggered a kernel panic in the Linux device-memory mapping path called by NVIDIA UVM, with the first preserved frame at `__init_zone_device_page`. The remaining runs were stopped to avoid risking more nodes. -That trace does not establish EFA 3.3.0g as the root cause. The completed results above are from the earlier matched B200 campaign and are not presented as post-update measurements. The revalidation's benchmark resources were harvested and removed; all 32 nodes returned Ready. +That trace does not establish EFA 3.3.0g as the root cause. The historical native tables are from the earlier B200 campaign and are not presented as post-update measurements. The fair common-boundary rerun used the post-update hosts only after the required HMM and GDRCopy-device admission checks passed. The failed revalidation's benchmark resources were harvested and removed; all 32 nodes returned Ready. ## Reproduce and collate From 9f0adf4cd3fb170b9ecf80087886bfb5f32472ce Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 23:39:37 +0000 Subject: [PATCH 13/19] refactor(ep): replace native tables with fair B200 results --- .../pytorch/sglang/dsr1-deepep-efa/README.md | 12 +- .../dsr1-deepep-efa/benchmarks/README.md | 84 ++-- .../ep-backend-comparison/README.md | 300 +++++------ .../ep-backend-comparison/RESULTS-b200.md | 218 -------- .../ep-backend-comparison/RESULTS-p5.md | 78 --- .../ep-backend-comparison/RESULTS.md | 311 +++++------- .../ep-backend-comparison/collect_results.py | 238 --------- .../deepep_v2_selected_cases.py | 96 ---- .../ep-backend-comparison/env_vars.example | 27 - .../ep-backend-comparison/nccl-alltoall.yaml | 106 ---- .../results/b200-ap-south-1-2026-08-24.json | 473 ++++++++++++++++++ .../run_fair_ep_comparison.sh | 99 +++- .../summarize_fair_results.py | 297 ++++++++++- .../test_collect_results.py | 55 -- .../test_summarize_fair_results.py | 102 +++- 15 files changed, 1232 insertions(+), 1264 deletions(-) delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py diff --git a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md index 386542b05..f54d430e0 100644 --- a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md +++ b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md @@ -174,11 +174,13 @@ Three things to know before running this on Blackwell: `setup/env_vars` derives `IFACE` from the default route, so the interface name is not something you need to know per instance family. -**Serving on Blackwell is not validated here.** The DeepEP-EFA kernels are — the same -`567632d` + EFA patch, same NVSHMEM 3.7.0, is measured out to 256 ranks on `p6-b300` in -[`ep-backend-comparison`](../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md). -The serving-side gap matters, because **on B300 the published SGLang comparison goes the other -way**: see [Blackwell: expect DeepEP to lose at 2 nodes](./benchmarks/README.md#blackwell-expect-deepep-to-lose-at-2-nodes). +**Serving on Blackwell is not validated here.** The older B300 backend comparison previously cited +here has been retired because its backend-native timing and byte accounting did not support a +cross-backend ranking. The replacement +[`ep-backend-comparison`](../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) +uses a common boundary for a synthetic decode communication workload at EP16 and EP32 on B200. It +does not establish B300 serving behavior. See +[Blackwell serving needs a matched measurement](./benchmarks/README.md#blackwell-serving-needs-a-matched-measurement). ## Smoke-test the EFA transport before loading the model diff --git a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md index da3b5b90d..497b6fd0e 100644 --- a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md +++ b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md @@ -813,60 +813,36 @@ it was added for was real on some host during bring-up, just not one that has be since. Treat it as a knob to try if NVSHMEM init fails on a `normal` role, not as a requirement — and do not conclude from a passing `normal` run that your host disagrees with this table. -# Blackwell: expect DeepEP to lose at 2 nodes - -The serving tables in this document are H200. The question comes up — *"we benchmarked DeepEP vs the -NCCL all-to-all on 2× B300 and DeepEP was slower in every configuration; is that expected?"* — and -the answer from the data that **is** here is **yes at 2 nodes, and it is not an EFA problem.** -Reported shape of such a result: output throughput −7% to −26%, median TTFT +17% to +82%, P99 ITL -1.2–1.9 s vs 0.8–0.9 s, `normal` (HT) slowest where it ran, at TP16/EP16 across two nodes, 8K input -/ 1K output, concurrency 128. - -Read this section as a **mechanism argument extrapolated from H200 serving data plus B300 kernel -data** — not as a Blackwell serving measurement. No B300 serving sweep exists here yet. - -1. **16 ranks is DeepEP's worst case on the hardware measured here, and the mechanism is not - GPU-specific.** Every table in this document says so: the colocated decode sweep has DeepEP at - 0.23–0.75× the baseline's throughput and 1.4–4.3× its TPOT; the 2P2D sweep 0.55–0.71× at 1.4–1.8× - TPOT. The reason — 256 experts over 16 ranks means 16 experts per GPU, so the fan-out is small and - mostly intra-node NVLink, and DeepEP's per-layer dispatch/combine cost is not amortised — is a - function of EP width and expert count, not of the GPU generation. That makes a −7% to −26% - *aggregate* regression on a mixed 8K/1K workload **milder than what we measure on Hopper**, though - confirming the H200 magnitude carries to B300 requires the B300 sweep. - **One Blackwell configuration note either way:** on B200 every rank logs `Only use 20 SMs for DeepEP - communication ... Consider using --deepep-config`, so any Blackwell number taken with the default - config is a **floor**, not DeepEP's best (thanks @KeitaW for the observation). -2. **The published B300 kernel numbers are healthy, which localises the gap above the transport.** - At 2 nodes / 16 ranks on `p6-b300`, DeepEP-over-EFA dispatch/combine is **126.6 / 106.4 GB/s** — - best of the three backends there, *above* the NCCL all-to-all's 104.9 GB/s at matched payload - ([`ep-backend-comparison`](../../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md)). - So fabric and kernels are fine at that scale; the serving regression is per-layer - launch/scheduling overhead and MoE-runner choice, not bytes on the wire. -3. **`normal`/HT being slowest at decode-heavy concurrency is by design.** On a 1K-output workload - the run is TPOT-dominated, where `low_latency` is the intended mode. - -Before concluding anything from such a run, eliminate these: - -- **The four settings in [How to measure this correctly](#how-to-measure-this-correctly)** — the - harness defaults and the mode pinning in particular. `--random-range-ratio` alone moved a nominal - 256/512/conc-64 point from 374 to 1127 tok/s. -- **The MoE runner is not held constant** in the usual formulation: DeepEP rows run - `--moe-runner-backend deep_gemm` while the no-DeepEP rows resolve `auto` to `flashinfer_trtllm` on - Blackwell. That is two variables, and TRT-LLM's Blackwell MoE kernels are heavily tuned. Re-run - DeepEP against `flashinfer_trtllm` (or the baseline against `deep_gemm`) before attributing the - delta to the all-to-all. -- **DeepGEMM JIT warmup.** It inflates early TTFT and P99 ITL specifically — the two metrics that - move most in reports like this. Pre-warm on **both** nodes (`recipe/serve-pd.sh precompile`). -- **An HT-path hang that does not reproduce.** At ≥128 ranks the NVSHMEM-libfabric host proxy - exhausts libfabric retries (`EAGAIN` in `nvshmemi_process_multisend_rma`) and kills a different - pair of ranks each run — a documented statistical fan-out limit, not a bad node. At 16 ranks it - should not fire, but a non-reproducing hang on the HT path has the same signature. - -**The load-bearing point for a large fleet: 2 nodes measures the wrong thing.** DeepEP is built for -EP domains where experts are spread thin enough that every token crosses the fabric. Kernel scaling -to 256 ranks on `p6-b300` is already characterised: the useful envelope is ~64–160 ranks, with hard -implementation caps past that (HT: 160 PEs at `deep_ep.cpp:158`; low-latency: between 64 and 128 -PEs). **A production-EP-width run — EP32 or EP64, not EP16 — is the measurement that decides this.** +# Blackwell serving needs a matched measurement + +The serving tables in this document are H200. No matched B200 or B300 serving sweep is reported +here, so the H200 direction must not be assumed to carry across GPU generations. + +The current +[`ep-backend-comparison`](../../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) +is useful transport evidence with a common CUDA timing boundary and a common logical payload. It is +still a synthetic decode dispatch-plus-combine workload. It excludes expert compute, scheduling, +communication/computation overlap, and request-level serving behavior. Its EP16 B200 result is also +mixed by dtype: UCCL has lower FP8 latency, while DeepEP V2 has lower BF16 latency. At EP32, UCCL has +lower latency for both measured dtypes. None of those cells is a B300 serving result. + +Before attributing a Blackwell serving delta to the all-to-all backend, control these variables: + +- Apply the 4 settings in [How to measure this correctly](#how-to-measure-this-correctly), including + the harness defaults and explicit mode pinning. `--random-range-ratio` alone moved a nominal + 256-token/512-token/concurrency-64 point from 374 tokens/s to 1,127 tokens/s in the H200 campaign. +- Hold the MoE runner constant. DeepEP rows commonly select `deep_gemm`, while no-DeepEP rows can + resolve `auto` to `flashinfer_trtllm` on Blackwell. Changing both the MoE kernel and communication + backend does not isolate either effect. +- Pre-warm DeepGEMM JIT on every node with `recipe/serve-pd.sh precompile` before collecting TTFT or + P99 ITL. +- Match model, EP and TP widths, input and output lengths, concurrency, request seeds, image digests, + runtime versions, and named nodes. Rotate backend order across independent process starts. +- Compare request-level throughput and latency directly. Treat each backend's native GB/s fields as + diagnostics unless they share the same timing boundary and byte numerator. +- Test the production EP width. An EP16 result cannot decide an EP32 or EP64 deployment. + +The fair conclusion for Blackwell serving remains unmeasured until that matched sweep is run. # Reproduce diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index fd95aee59..09bc1eec3 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -1,192 +1,192 @@ -# Expert-Parallelism Backend Comparison on EFA +# Fair Expert-Parallelism Backend Comparison on EFA -Head-to-head MoE dispatch/combine micro-benchmark across 3 EP backends, with raw NCCL all-to-all retained as a separately labeled transport reference. Runs within a result set use the same EP world size, problem shape, common dependency stack, and GPU nodes. The latest campaign adds DeepEP V2 over NCCL GIN EFA-GDA on B200 at EP16 and EP32. The historical B300 campaign used 64 ranks and also exercised a 256-rank transport reference. +This directory compares 3 expert-parallel dispatch/combine backends through one common semantic workload and one external timing boundary: -This directory provides comparison orchestration and log collation. Backend build and launch harnesses live in sibling directories or the linked DeepEP V2 contribution. +| Backend | Implementation and transport | +|---|---| +| UCCL | DeepEP-compatible dispatch/combine over UCCL all-to-all and EFA | +| DeepEP V1 NVSHMEM | DeepEP V1 `Buffer` over NVSHMEM, libfabric, and EFA | +| DeepEP V2 NCCL GIN | DeepEP V2 `ElasticBuffer` over NCCL GIN EFA-GDA | -| Config | What it is | Source benchmark | -|---|---|---| -| **NCCL** (reference) | Raw all-to-all over EFA. It moves bytes but does *not* do token routing or combine-reduction, so it is not an EP-backend row. | [`nccl-alltoall.yaml`](nccl-alltoall.yaml) (built from [`../../nccl-tests`](../../nccl-tests)) | -| **UCCL** | DeepEP-style dispatch/combine over the UCCL all-to-all backend. | [`../uccl-ep-benchmark/kubernetes`](../uccl-ep-benchmark/kubernetes) | -| **DeepEP V1 NVSHMEM** | DeepEP V1 dispatch/combine over NVSHMEM libfabric/EFA. | [`../deepep-benchmark/kubernetes`](../deepep-benchmark/kubernetes) | -| **DeepEP V2 NCCL GIN** | DeepEP V2 `ElasticBuffer` dispatch/combine over NCCL GIN EFA-GDA. | [DeepEP V2 benchmark PR 1234](https://github.com/awslabs/awsome-distributed-ai/pull/1234) and [`deepep_v2_selected_cases.py`](deepep_v2_selected_cases.py) | +The primary metric is slowest-rank CUDA latency from BF16 input readiness through dispatch and combine completion. Backend-native latency and bandwidth fields are not used for cross-backend rankings because they do not share one timing boundary or byte numerator. -> **Backend naming.** DeepEP V1 at commit `567632d` has no internode path without an external transport, so this comparison names its NVSHMEM transport explicitly. DeepEP V2 is a distinct NCCL GIN backend and does not replace the raw NCCL transport reference. Raw NCCL remains context only and is never treated as an equal dispatch/combine measurement. +The validated B200 result is in [RESULTS.md](RESULTS.md). EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes. -## Matched configuration (what makes the numbers comparable) +## What makes the comparison fair -Compare backends only within the same platform campaign. B200, B300, and H100 measurements are kept on separate result pages. The latest B200 campaign used the following matched shapes: +The backend implementation is the intended independent variable. The harness holds these inputs and measurement rules constant: -| Parameter | Value | +| Control | Rule | |---|---| -| World size | EP16: 2 nodes × 8 GPUs = 16 ranks; EP32: 4 nodes × 8 GPUs = 32 ranks | -| `num-tokens` | 4096 (internode) / 128 (low-latency) | -| `hidden` | 7168 | -| `num-topk` | 8 | -| `num-experts` | 256 | -| dispatch dtype | FP8 and BF16 | -| combine dtype | BF16 headline; DeepEP V2 also prints FP8 diagnostic data | +| Input | One deterministic BF16 tensor per EP size | +| Routing | One exact top-k route and one set of weights, verified by SHA-256 across all arms and starts | +| Shape | 128 tokens/rank, hidden size 7,168, 256 experts, top-k 8 experts/token | +| Operations | FP8 or BF16 dispatch followed by BF16 combine | +| Timing | One CUDA Event boundary around input preparation, dispatch, and combine | +| Rank reduction | Maximum elapsed time across all ranks for each measured iteration | +| Warmup | 20 warmup iterations per dtype and process start | +| Measurement | 100 measured iterations per dtype and process start | +| Replication | 3 independent process starts per arm and workload cell | +| Order | Backend order rotates across starts; dtype order also rotates | +| Hardware | The same named nodes serve every arm at a given EP size | +| Runtime | Every result must report the same GPU, PyTorch, CUDA, and NCCL versions | +| Correctness | Every rank must pass the common identity-expert result before timing | + +Each process start contributes its median of 100 slowest-rank iteration latencies. The report then takes the median across 3 process starts. Iterations within one process are not treated as independent replicates. + +### Common logical throughput + +The harness derives effective logical throughput from a common useful-payload numerator. Each valid expert assignment contributes: + +- the dispatch tensor; +- FP8 scales when FP8 dispatch is selected; and +- the BF16 combine tensor. + +Backend metadata is excluded. Scale-out logical bytes include only assignments whose destination expert is on another node. + +```text +logical GB/s/rank = average logical bytes/rank / median slowest-rank latency +scale-out logical GB/s/rank = average remote logical bytes/rank / median slowest-rank latency +``` -The historical B300 and H100 result pages use 4-node and 8-node topologies with the same token, hidden-size, top-k, and expert values. The UCCL manifests bake these arguments into the `python3 bench/test_*.py` invocation; the DeepEP V1 test hard-codes its config in-image. Before running V1, confirm the image config and align UCCL to it: +These are logical efficiency metrics, not observed wire bandwidth. DeepEP V2 SO/SU bandwidth, DeepEP V1 native bandwidth, and UCCL native bandwidth remain useful backend diagnostics, but their numerators and aggregation boundaries differ and must not be compared directly. -```bash -# Read the DeepEP test config from the NVSHMEM image and match UCCL's CLI args to it. -docker run --rm ${NVSHMEM_IMAGE_URI} sed -n '1,60p' /DeepEP/tests/test_internode.py -``` +## Files -If the DeepEP values differ from 4096/7168/8/256, edit the bench args in -`../uccl-ep-benchmark/kubernetes/test-*.yaml` to match. +| File | Purpose | +|---|---| +| [`fair_ep_benchmark.py`](fair_ep_benchmark.py) | Common workload, backend adapters, correctness check, CUDA timing, and logical-byte accounting | +| [`run_fair_ep_rank.sh`](run_fair_ep_rank.sh) | Per-node `torchrun` entry point and backend-specific transport environment | +| [`run_fair_ep_comparison.sh`](run_fair_ep_comparison.sh) | EKS admission, shared-Lease coordination, rotated matrix, durable harvest, and verified teardown | +| [`fair_result_io.py`](fair_result_io.py) | Robust result-marker parsing from interleaved native output | +| [`extract_fair_results.py`](extract_fair_results.py) | Canonical JSONL extraction from a rank-zero log | +| [`summarize_fair_results.py`](summarize_fair_results.py) | Matrix validation, per-start aggregation, paired deltas, and bootstrap intervals | +| [`results/b200-ap-south-1-2026-08-24.json`](results/b200-ap-south-1-2026-08-24.json) | Machine-readable validated result summary | +| [`RESULTS.md`](RESULTS.md) | Human-readable result, provenance, and scope limits | + +## Requirements -### Fair common-boundary decode comparison +The scored B200 matrix requires: -The backend-native benchmark outputs do not share one timing boundary or byte numerator. Use [`fair_ep_benchmark.py`](fair_ep_benchmark.py) when making a direct cross-backend latency or effective-bandwidth claim. It supplies all 3 backends with the same deterministic BF16 input, exact top-k route, and weights, then times BF16-input readiness through dispatch and combine completion with one CUDA Event boundary. Each iteration uses the slowest rank's elapsed time. +- 4 named, Ready `p6-b200.48xlarge` nodes in one EKS cluster; +- 8 allocatable GPUs and 8 allocatable EFA devices on every selected node; +- no active GPU requests on the selected nodes before each arm; +- the NVIDIA and EFA Kubernetes device plugins; +- `uvm_disable_hmm=Y` or `uvm_disable_hmm=1` on every selected host; +- `/dev/gdrdrv` as a character device on every selected host; +- `aws`, `kubectl`, `jq`, `rg`, Python 3, and Bash on the launch host; and +- access to the 3 digest-pinned backend images. -The common logical-byte numerator counts the useful dispatch tensor, FP8 scales when selected, and BF16 combine tensor for each valid expert assignment. It excludes backend metadata. Backend-native UCCL, NVSHMEM RDMA, and DeepEP V2 SO/SU measurements remain useful diagnostics, but they are reported separately. +DeepEP V2 receives an INFO-level EP16 admission run before the scored matrix. The admission must log a GDAKI context. A missing HMM mitigation, GDRCopy device, or GDAKI proof stops the campaign before scoring. -[`run_fair_ep_comparison.sh`](run_fair_ep_comparison.sh) runs the B200 decode matrix at 128 tokens/rank for EP16 and EP32. It uses the same named nodes for every arm, performs 3 independent process starts per cell in rotated arm order, validates route and input hashes, and tears down only its labeled namespace. The concurrent campaign's nodes and shared Lease are read and protected, never modified. +## Run on an exclusive node set -The completed 24 August 2026 rerun, including common logical throughput, paired latency deltas, 95% bootstrap intervals, correctness, and scope limits, is reported in [`RESULTS-b200.md`](RESULTS-b200.md#fair-common-boundary-decode-rerun). Keep its primary table separate from the historical backend-native tables on the same page. +Use a unique namespace and durable artifact directory. `KUBECTL_CONTEXT` is required explicitly so a concurrent process changing the default context cannot redirect the campaign. ```bash -CAMPAIGN_ID=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) \ +campaign_id=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) +CAMPAIGN_ID="${campaign_id}" \ FAIR_EP_NODES=node-a,node-b,node-c,node-d \ -PROTECTED_NODES_CSV=foreign-node-a,foreign-node-b \ -ARTIFACT_ROOT=/shared/artifacts/${CAMPAIGN_ID} \ -KUBECTL_CONTEXT=target-context \ +PROTECTED_NODES_CSV="" \ +ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ +KUBECTL_CONTEXT=aps1-shared \ +LOCK_MODE=exclusive \ ./run_fair_ep_comparison.sh ``` -The default `LOCK_MODE=exclusive` claims the shared Lease only when it is empty and releases it during verified teardown. If an explicitly coordinated campaign is still active on a disjoint node set, use `LOCK_MODE=observe EXPECTED_LOCK_HOLDER=foreign-campaign-id`; this mode checks the holder throughout the run and never mutates the Lease. - -DeepEP V2 admission is conditional on `uvm_disable_hmm=Y` and a character-device `/dev/gdrdrv` on every selected host, and requires a logged GDAKI context. The runner aborts before the scored matrix if any condition is absent. +`LOCK_MODE=exclusive` claims the configured shared Lease only when its holder is empty. The runner releases only a Lease that it still owns. -## Prerequisites +## Run beside a coordinated campaign -- EKS cluster with EFA + GPU nodes; NVIDIA device plugin + AWS EFA device plugin; Kubeflow MPI - Operator (`kubectl get crd mpijobs.kubeflow.org`). See each benchmark's `kubernetes/README.md`. -- The container images in ECR: - - DeepEP V1 NVSHMEM: `../deepep-benchmark/deepep.Dockerfile` (CUDA 13, `sm_90`+`sm_100`) - - UCCL: `../uccl-ep-benchmark/uccl-ep.Dockerfile` (CUDA 13; pinned UCCL commit; Hopper + Blackwell via PTX) - - DeepEP V2 NCCL GIN: the standalone build and launch workflow from [PR 1234](https://github.com/awslabs/awsome-distributed-ai/pull/1234), with all revisions pinned to the campaign being reproduced - - NCCL: **reuse the NVSHMEM/DeepEP image** — it already builds `/opt/nccl-tests/build/alltoall_perf` - with `sm_100` gencode, so no separate `nccl-tests` build is needed for the baseline. -- For EFA-GDA, verify the host EFA driver and GDRCopy requirements documented by the DeepEP V2 harness before scheduling a multi-node run. - -## Account / cluster safety (run first) +Observe mode is allowed only when the other campaign has a known Lease holder and a disjoint named node set: ```bash -aws sts get-caller-identity # confirm the target account -kubectl config current-context # confirm the target cluster -kubectl get nodes -l node.kubernetes.io/instance-type=p6-b300.48xlarge # confirm $NUM_NODES schedulable -kubectl get crd mpijobs.kubeflow.org # confirm MPI Operator +campaign_id=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) +CAMPAIGN_ID="${campaign_id}" \ +FAIR_EP_NODES=node-a,node-b,node-c,node-d \ +PROTECTED_NODES_CSV=foreign-node-a,foreign-node-b \ +ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ +KUBECTL_CONTEXT=aps1-shared \ +LOCK_MODE=observe \ +EXPECTED_LOCK_HOLDER=foreign-campaign-id \ +./run_fair_ep_comparison.sh ``` -## Run order +Observe mode never mutates the shared Lease. It verifies the exact holder before every arm and again before aggregation. Any selected/protected node overlap or Lease-holder change stops the run. + +## Execution matrix + +The scored order is a 3-start rotation: + +| Start index | Backend order | Dtype order | +|---:|---|---| +| 1 | UCCL, DeepEP V1, DeepEP V2 | FP8, BF16 | +| 2 | DeepEP V2, UCCL, DeepEP V1 | BF16, FP8 | +| 3 | DeepEP V1, DeepEP V2, UCCL | FP8, BF16 | + +The runner executes this rotation first at 16 ranks on 2 nodes and then at 32 ranks on 4 nodes. Arms run serially, and every StatefulSet and its GPU pods must be gone before the next arm is admitted. + +## Durable artifacts and teardown + +The campaign writes the following layout under `ARTIFACT_ROOT`: + +```text +control/ + aws-caller-identity.json + fleet-nodes-before.json + fleet-nodes-after.json + fleet-pods-before.json + fleet-pods-after.json + provenance.json + selected-nodes.txt +runs/ + ep16/{admission,measurement}-repeat-*// + ep32/measurement-repeat-*// +summary/ + summary.json + summary.md +teardown/ + namespace-delete.log + remaining-resources.json + shared-lease-after.json +CAMPAIGN_COMPLETE +SHA256SUMS +STATUS +``` + +Every rank log, rendered Pod manifest, Pod description, canonical rank-zero JSONL result, case status, input/route hash, and immutable image reference is retained. `CAMPAIGN_COMPLETE` is written only after the full scored matrix succeeds and teardown verifies that the owned namespace and labeled resources are absent. `SHA256SUMS` is generated after the final status markers. -Run backends serially on the same named node set. Smoke each image on 1 node first. An intranode smoke is NVLink-only and is not a scored EFA comparison row. The commands below cover the existing DeepEP V1, UCCL, and raw NCCL EKS manifests. Run DeepEP V2 with the linked harness and the matched wrapper in this directory, using 4,096 tokens/rank for prefill and 128 tokens/rank for decode. +## Re-aggregate preserved logs -`deepep_v2_selected_cases.py` matches the synthetic DeepEP V2 revision pinned in [`RESULTS-b200.md`](RESULTS-b200.md). Review its upstream `Namespace` fields before using it with a different DeepEP V2 revision. +The normal campaign aggregates automatically. To validate a preserved artifact tree again: ```bash -cp env_vars.example env_vars # then edit image URIs / topology -source env_vars - -# 1) DeepEP V1 NVSHMEM -( cd ../deepep-benchmark/kubernetes - IMAGE_URI=$NVSHMEM_IMAGE_URI NUM_NODES=$NUM_NODES \ - envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' < test-internode.yaml | kubectl apply -f - - # ...wait, save logs, delete. Then low-latency -- see the override note below. ) - -# 2) UCCL (UCCL-EP) — one MPI rank per GPU (NP = NUM_NODES * GPU_PER_NODE) -( cd ../uccl-ep-benchmark/kubernetes - IMAGE_URI=$UCCL_IMAGE_URI NUM_NODES=$NUM_NODES NP=$NP \ - envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < test-internode.yaml | kubectl apply -f - - # ...then test-low-latency.yaml (already pinned to --num-experts=256) ) - -# 3) DeepEP V2 NCCL GIN -# Run an INFO-level admission first and require NCCL_GIN_TYPE=5, a successful -# Libfabric_GDAKI context, a nonzero GIN layout, and bidirectional EFA deltas. -# Use deepep_v2_selected_cases.py inside the pinned V2 image, then repeat with -# NCCL_DEBUG=WARN for the scored run. -# The multi-node launcher must provide WORLD_SIZE, RANK, MASTER_ADDR, and -# MASTER_PORT to each node. Inside each node's container, run one of: -# python3 deepep_v2_selected_cases.py --num-processes=8 --num-tokens=4096 --hidden=7168 --num-topk=8 --num-experts=256 -# python3 deepep_v2_selected_cases.py --num-processes=8 --num-tokens=128 --hidden=7168 --num-topk=8 --num-experts=256 - -# 4) NCCL reference (reuses the DeepEP V1 image's alltoall_perf) -IMAGE_URI=$NCCL_IMAGE_URI \ -envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < nccl-alltoall.yaml | kubectl apply -f - +python3 summarize_fair_results.py /path/to/artifacts/runs \ + --starts=3 \ + --provenance=/path/to/artifacts/control/provenance.json \ + --json=/path/to/artifacts/summary/summary.json \ + --markdown=/path/to/artifacts/summary/summary.md ``` -> **DeepEP low-latency at 8 nodes — required override.** The merged DeepEP low-latency manifest -> runs `python3 /DeepEP/tests/test_low_latency.py` with no args, so it uses the upstream default -> `--num-experts=288`. The test asserts `num_experts % num_ranks == 0`; at 8 nodes (64 ranks), -> `288 % 64 ≠ 0` and it aborts. Match the comparison's 256 by patching the rendered manifest: -> ```bash -> cd ../deepep-benchmark/kubernetes -> IMAGE_URI=$NVSHMEM_IMAGE_URI NUM_NODES=$NUM_NODES \ -> envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' < test-low-latency.yaml \ -> | sed 's#test_low_latency.py#test_low_latency.py --num-experts 256#' | kubectl apply -f - -> ``` -> (DeepEP internode defaults are already 4096/7168/8/**256**, so internode needs no override.) - -Save each launcher log (`kubectl logs <…-launcher> > .log`) and **delete the job before -the next run** (MPIJob names are fixed; re-applying collides, and each job needs all 8 nodes): +If a native library appends a diagnostic to the JSON marker's physical line, extract the JSON object with the repository parser rather than `grep` or line splitting: ```bash -kubectl delete mpijob deepep-internode uccl-ep-internode nccl-alltoall # etc. +python3 extract_fair_results.py rank-zero.log results.jsonl ``` -## Collate +The summarizer rejects an incomplete matrix, correctness failure, mutable image tag, route/input mismatch, runtime-stack mismatch, or disagreement in the common logical payload. + +## Local validation ```bash -python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log \ - --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log \ - --uccl-lowlat uccl_lowlat.log \ - --deepep-v2-prefill deepep_v2_prefill.log \ - --deepep-v2-decode deepep_v2_decode.log \ - --nccl nccl_alltoall.log +python3 -m pytest -q test_fair_ep_benchmark.py test_summarize_fair_results.py +python3 -m py_compile \ + fair_ep_benchmark.py fair_result_io.py extract_fair_results.py \ + summarize_fair_results.py +bash -n run_fair_ep_comparison.sh run_fair_ep_rank.sh +shellcheck run_fair_ep_comparison.sh run_fair_ep_rank.sh ``` -For DeepEP V1 and UCCL internode logs, the parser reports the RDMA leg of the `Best dispatch/combine` line, not the intra-node NVL value printed beside it. For DeepEP V2, it reports rank-zero SO bandwidth, SU bandwidth, and latency for each dispatch dtype and operation. For raw NCCL, it reports bus bandwidth at the EP per-rank payload size, approximately 56 MiB, plus the asymptotic peak. Eyeball one real launcher log against the parser before trusting it. - -Results are recorded per platform: [`RESULTS-b200.md`](RESULTS-b200.md) (B200 with DeepEP V2), [`RESULTS.md`](RESULTS.md) (B300 historical), and [`RESULTS-p5.md`](RESULTS-p5.md) (P5/H100 historical). For other instance types set `INSTANCE_TYPE` and `EFA_PER_NODE` to the devices actually exposed by the target nodes. - -## Scaling beyond 8 nodes (256-rank findings) - -The historical UCCL and DeepEP V1 matrix was pushed to 16 and 32 nodes, or 128 and 256 ranks, on a 32-node `p6-b300.48xlarge` Capacity Block on 14 July 2026. Those V1-era kernels hit implementation limits between 65 and 256 ranks; only the raw NCCL reference ran at 256 ranks. These findings do not establish a DeepEP V2 limit. Details and per-limit source citations are in [`RESULTS.md`](RESULTS.md). Operational notes for rerunning the V1-era matrix at scale: - -- **HT internode**: DeepEP asserts at >160 ranks (`NUM_MAX_NVL_PEERS 8 × NUM_MAX_RDMA_PEERS 20`, - `kernels/configs.cuh`) and its stock combine tuning tables already abort at 16 nodes; UCCL - overflows an `int32` buffer bound above 64 ranks. Treat the HT comparison as an - **8-nodes-per-EP-domain benchmark** — which matches how training deploys these kernels - (EP32/EP64 groups inside a larger world). -- **Low-latency**: both implementations cap between 64 and 128 ranks (UCCL: compile-time - signaling-buffer arena; NVSHMEM/DeepEP: libfabric host-proxy retry exhaustion with moving - victims per run). -- **GDRCopy at scale (NVSHMEM)**: past ~1 GiB of LL buffer, NVSHMEM grows its symmetric heap - dynamically and must register each chunk over libfabric via **GDRCopy inside the container**. - The manifests set `NVIDIA_GDRCOPY=enabled`, but some clusters' nvidia container toolkit - ignores it — if every rank dies at `mem_heap.cpp:1361 register_mem_handle failed` after a - `GDRCopy support not enabled` warning, hostPath-mount `/dev/gdrdrv` into the worker - (requires `privileged: true`) and ensure the host loads `gdrdrv` (gdrcopy-loader DaemonSet - or DLAMI). -- **NCCL at 32 nodes** works unmodified (`NUM_NODES=32`, `NP=256`); expect matched-size busbw - to drop vs 8 nodes (fan-out cost). - -## Caveats - -- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure transport throughput, while EP dispatch/combine includes routing and reduction and uses backend-specific bandwidth accounting. Treat matched-size NCCL bus bandwidth as transport context, not as a hard ceiling or an EP-backend row. -- **Internode = RDMA leg.** DeepEP/UCCL print both an RDMA (cross-node) and an NVL (intra-node) - bandwidth on the same line; only the RDMA number reflects the inter-node transport being - compared. -- **Backend-native accounting differs.** V2 prints SO and SU bandwidth plus per-operation latency, while V1 and UCCL use their own aggregate timing and byte accounting. Neither the native GB/s values nor the native latency values establish an apples-to-apples ranking. Use the common-boundary harness for cross-backend claims and retain native values as diagnostics. -- **`num-experts` must divide the world size.** Both tests assert `num_experts % num_ranks == 0`. - At 8 nodes (64 ranks) the comparison uses 256 (= 4/rank). The DeepEP low-latency default (288) - is not divisible by 64 and must be overridden (see the run-order note). -- **Toolchain.** Verify the exact toolchain per result page. The B200 comparison used CUDA 13.0.3 and the same vLLM wheel in all 3 EP-backend images. The historical raw NCCL reference shares the DeepEP V1 image. -- **UCCL bench scripts** are pulled from upstream `uccl/ep/bench` at image-build time and pinned - via `UCCL_COMMIT`. If upstream renames CLI flags, adjust the bench args in the UCCL - manifests. +## Scope limits + +This harness measures a synthetic decode dispatch-plus-combine communication workload. It does not measure prefill, expert compute, communication/computation overlap, end-to-end training, serving throughput, TTFT, TPOT, or E2E latency. A result from this harness does not establish a universal backend winner or a limit at an unmeasured EP size. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md deleted file mode 100644 index ba96ae734..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-b200.md +++ /dev/null @@ -1,218 +0,0 @@ -# EP-Backend Comparison Results: B200 with DeepEP V2 - -This page records 2 separate B200 campaigns in `ap-south-1`. The fair common-boundary decode rerun completed on 24 August 2026 is the primary cross-backend comparison. The earlier backend-native campaign is retained below as historical diagnostic evidence only. These results are separate from the historical [B300 results](RESULTS.md) and [P5/H100 results](RESULTS-p5.md). Results from different campaigns, timing boundaries, byte-accounting conventions, or GPU generations are not combined. - -## Fair common-boundary decode rerun - -The rerun passed all 36 scored dtype results from 18 distributed jobs. UCCL had the lowest median latency in 3 of 4 workload cells: EP16 FP8, EP32 FP8, and EP32 BF16. DeepEP V2 had the lowest median latency in EP16 BF16. DeepEP V2 was faster than DeepEP V1 in all 4 workload cells. - -These conclusions apply only to the exact synthetic dispatch-plus-combine workload and timing boundary below. They do not establish end-to-end training or serving throughput, prefill performance, or a universal backend winner. Each arm and workload cell has 3 independent process starts. The 95% bootstrap intervals therefore summarize these 3 starts and should not be treated as a high-power population estimate. - -### Common workload and method - -| Field | Value | -|---|---| -| Measurement period | 24 August 2026 | -| Region and Availability Zone | `ap-south-1`, `ap-south-1c` | -| EKS cluster | `ml-clusters-shared-ap-south-1` | -| Hardware | `p6-b200.48xlarge`, 8 B200 GPUs and 8 EFA devices per node | -| EP16 topology | 2 nodes, 16 ranks | -| EP32 topology | 4 nodes, 32 ranks | -| Workload | 128 tokens/rank, hidden size 7,168, top-k 8, experts 256 | -| Dispatch dtype | FP8 and BF16 | -| Combine dtype | BF16 | -| Timing boundary | BF16 input ready through dispatch and combine completion, measured with CUDA Events and reduced to the slowest rank | -| Logical payload | Useful dispatch tensor, FP8 scales when selected, and BF16 combine tensor for every valid expert assignment; backend metadata excluded | -| Iterations | 20 warmup iterations and 100 measured iterations per dtype and start | -| Replication | 3 independent process starts per arm and workload cell | -| Order control | Backend order rotated across starts; dtype order also rotated | -| Input control | Deterministic BF16 input, exact top-k route, and weights; matching SHA-256 input and route hashes required within each EP size | -| Correctness | 36 of 36 scored dtype results passed; FP8 tolerance `9e-4` dimensionless and BF16 tolerance `1e-5` dimensionless | - -All 3 images used PyTorch `2.13.0+cu130`, CUDA 13.0, NCCL 2.29.7, and digest-pinned images: - -| Backend | Scored image digest | -|---|---| -| UCCL | `sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a` | -| DeepEP V1 NVSHMEM | `sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac` | -| DeepEP V2 NCCL GIN | `sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb` | - -The 3 backends ran serially on the same named nodes for each EP size. A concurrent vLLM campaign used 12 disjoint nodes, with 0 selected-node overlap. This campaign observed the shared Lease and did not mutate the vLLM campaign's Lease, nodes, namespace, images, or artifacts. Because the campaigns shared the cluster fabric, unobserved background-fabric effects cannot be excluded completely; order rotation and paired starts reduce but do not eliminate that risk. - -### Common-boundary results - -Values are medians across 3 independent starts. Logical throughput uses the common logical-byte numerator defined above, not a backend-native bandwidth field. Lower latency is better. - -| EP size | Dispatch dtype | Backend | Median latency | 95% bootstrap CI | Run-to-run CV | Input throughput | Logical throughput | Scale-out logical throughput | -|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| -| 16 ranks | FP8 | UCCL | **0.5213 ms** | [0.5190, 0.5277] ms | 0.86% | 3,928,790.51 tokens/s | 42.68 GB/s/rank | 21.34 GB/s/rank | -| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 1.0179 ms | [1.0167, 1.0203] ms | 0.18% | 2,011,977.59 tokens/s | 21.86 GB/s/rank | 10.93 GB/s/rank | -| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 0.6219 ms | [0.6180, 0.6375] ms | 1.65% | 3,293,197.58 tokens/s | 35.78 GB/s/rank | 17.89 GB/s/rank | -| 16 ranks | BF16 | UCCL | 0.5920 ms | [0.5918, 0.5978] ms | 0.58% | 3,459,365.90 tokens/s | 49.59 GB/s/rank | 24.80 GB/s/rank | -| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 1.0297 ms | [1.0251, 1.0304] ms | 0.28% | 1,988,936.56 tokens/s | 28.51 GB/s/rank | 14.26 GB/s/rank | -| 16 ranks | BF16 | DeepEP V2 NCCL GIN | **0.4850 ms** | [0.4826, 0.5043] ms | 2.42% | 4,222,332.23 tokens/s | 60.53 GB/s/rank | 30.27 GB/s/rank | -| 32 ranks | FP8 | UCCL | **0.7655 ms** | [0.7640, 0.7714] ms | 0.51% | 5,350,835.03 tokens/s | 29.07 GB/s/rank | 21.80 GB/s/rank | -| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 1.5542 ms | [1.5513, 1.5628] ms | 0.38% | 2,635,371.65 tokens/s | 14.32 GB/s/rank | 10.74 GB/s/rank | -| 32 ranks | FP8 | DeepEP V2 NCCL GIN | 0.9537 ms | [0.9217, 0.9675] ms | 2.48% | 4,294,725.45 tokens/s | 23.33 GB/s/rank | 17.50 GB/s/rank | -| 32 ranks | BF16 | UCCL | **0.8686 ms** | [0.8683, 0.8696] ms | 0.08% | 4,715,590.93 tokens/s | 33.80 GB/s/rank | 25.35 GB/s/rank | -| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 1.5589 ms | [1.5526, 1.5714] ms | 0.61% | 2,627,446.53 tokens/s | 18.83 GB/s/rank | 14.13 GB/s/rank | -| 32 ranks | BF16 | DeepEP V2 NCCL GIN | 0.9509 ms | [0.9482, 0.9606] ms | 0.68% | 4,307,661.17 tokens/s | 30.88 GB/s/rank | 23.16 GB/s/rank | - -### Paired DeepEP V2 latency deltas - -Positive values mean DeepEP V2 had lower latency than the named baseline. A directional winner is supported for this workload only when the paired 95% bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV. - -| EP size | Dispatch dtype | Baseline | Median V2 latency reduction | 95% bootstrap CI | Direction supported | -|---:|:---:|:---|---:|:---:|:---:| -| 16 ranks | FP8 | UCCL | -19.30% | [-22.84, -17.11]% | UCCL lower | -| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 39.05% | [37.37, 39.22]% | V2 lower | -| 16 ranks | BF16 | UCCL | 18.44% | [14.82, 18.87]% | V2 lower | -| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 52.68% | [51.06, 53.13]% | V2 lower | -| 32 ranks | FP8 | UCCL | -24.83% | [-25.41, -20.40]% | UCCL lower | -| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 38.64% | [37.64, 41.02]% | V2 lower | -| 32 ranks | BF16 | UCCL | -9.35% | [-10.63, -9.16]% | UCCL lower | -| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 38.87% | [38.76, 39.18]% | V2 lower | - -The common logical throughput columns are the inverse-latency view of the same common payload and timing boundary, so they produce the same ranking as latency. Backend-native UCCL aggregate bandwidth, DeepEP V1 bandwidth, and DeepEP V2 SO/SU bandwidth use different numerators or aggregation boundaries and remain diagnostic only. - -### Fair-rerun provenance and custody - -The durable campaign ID is `fair-ep-b200-20260824t195145z`. Its machine-readable summary validates the complete 2-EP-size by 2-dtype by 3-backend by 3-start matrix, matching input and route hashes, digest-pinned images, and per-result correctness. The DeepEP V2 admission additionally required HMM disabled, `/dev/gdrdrv` present as a character device, and a logged GDAKI context before scoring. - -The 4 selected nodes were Ready after the run, each with 8 allocatable GPUs and 8 allocatable EFA devices. The benchmark namespace and all campaign-labeled pods, Jobs, and MPIJobs were absent after teardown. The shared Lease still belonged to the concurrent vLLM campaign. - -## Earlier backend-native campaign - -The remaining sections preserve the earlier campaign's backend-native measurements. Their timing boundaries and byte numerators differ by backend, so the latency and GB/s columns must not be used for a direct cross-backend ranking. The numerically lower V2 values reported in that campaign remain observations of the native benchmark outputs, not evidence that V2 won a fair common-boundary comparison. - -### Native campaign summary - -Using each backend's native fields, the DeepEP V2 dispatch-plus-combine sum was numerically lower in all 4 B200 cells. Relative to the UCCL native aggregate, the differences were 19.04% for EP16 FP8, 44.70% for EP16 BF16, 6.82% for EP32 FP8, and 18.88% for EP32 BF16. Relative to the DeepEP V1 native aggregate, the differences ranged from 52.22% to 56.34%. Because the measurement boundaries differ, these percentages are retained as historical diagnostics and are not fair cross-backend speedups. - -Each historical cell has 1 independent benchmark start, so run-to-run variance and coefficient of variation are unavailable. This campaign contains direct EP tests only; it produced no DeepEP V2 serving throughput, TTFT, TPOT, E2E latency, or NIXL result. DeepEP V2 also used substantially more HBM for prefill than the other 2 backends. - -The synthetic DeepEP source contains PR 5, but EP16 and EP32 do not validate its greater-than-20-NVLink-domain scale path. The independent 256-rank PR 5 gate was not completed, and no PR 5 scale-success claim is inferred from these performance cells. - -### Native campaign environment and provenance - -| Field | Value | -|---|---| -| Measurement period | 23 August 2026 through 24 August 2026 | -| Region and Availability Zone | `ap-south-1`, `ap-south-1c` | -| EKS cluster | `ml-clusters-shared-ap-south-1` | -| Hardware | `p6-b200.48xlarge`, 8 B200 GPUs and 8 EFA devices per node | -| Fleet census | 36 Ready B200 nodes | -| EP16 topology | 2 nodes, 16 ranks | -| EP32 topology | 4 nodes, 32 ranks | -| Prefill shape | 4,096 tokens/rank, hidden size 7,168, top-k 8, experts 256 | -| Decode shape | 128 tokens/rank, hidden size 7,168, top-k 8, experts 256 | -| Selected operations | FP8 dispatch, BF16 dispatch, BF16 combine | -| Independent starts | 1 start per cell | -| Correctness | 12 of 12 backend cells passed | - -All 3 scored images used the same vLLM wheel with SHA-256 `60800409bd2ede00aed65ac7b463541cbdc382716715e21c5bbef87d6f2423c4`. The common stack was vLLM commit `185cada36bb25aa55f762d004d54c5ca1e3fc753`, CUDA 13.0.3, PyTorch `2.13.0+cu130`, NCCL tag `v2.31.2-1` with runtime version 23,102 dimensionless, aws-ofi-nccl 1.21.1, libfabric API 2.6.0, and NIXL 1.3.2. - -| Backend | Backend revision | Scored image digest | -|---|---|---| -| UCCL | `0dc87eb3b40c372a16b70ef320f37daaa5299ca7` | `sha256:d2e3e500524e168a3d577f059325df34872dde850bf77505c080daabfb9f8288` | -| DeepEP V1 NVSHMEM | DeepEP `567632dd59810d77b3cc05553df953cc0f779799`, NVSHMEM 3.7.0 | `sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac` | -| DeepEP V2 NCCL GIN | Synthetic commit `b56ebf8bb4ece24cd78aa8c12550b24e35ac255b` | `sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb` | - -The DeepEP V2 source was synthesized reproducibly from base `02efc268a37802fc00812ede8f5ad7f535ceea0e`, PR 3 head `dd0f87261a80cf0ce8aa66e4ab2041843851d810`, and PR 5 head `2542d9641f2ec280213e875feb04be7862dda57c`. The final Git tree was `4da3a118b3b316de79b5af2b4770646035dc5802`. The image contained B200-specific `sm_100` and `sm_100a` code. - -Before measurement, an INFO-level EP16 admission ran on both nodes with `NCCL_GIN_TYPE=5` and `NCCL_SYM_GIN_KERNELS_ENABLE=0`. Logs showed the aws-ofi-nccl `Libfabric_GDAKI` version 14 dimensionless plugin, GPU Direct RDMA enabled on EFA HCAs, nonzero GIN layouts, and balanced cluster EFA TX and RX deltas of 235,008 bytes each. Every scored cell also recorded nonzero EFA RDMA writes, so the cross-node path was measured rather than inferred from topology. - -## Historical native decode diagnostics - -These values come from backend-native reports, not the common-boundary harness. UCCL and DeepEP V1 print an aggregate dispatch-plus-combine latency. DeepEP V2 prints dispatch and combine separately, so the V2 value below is their sum for the same dispatch dtype. The boundaries differ, and the table does not support a direct latency ranking. - -| EP ranks | Dispatch dtype | UCCL latency | DeepEP V1 latency | DeepEP V2 latency | V2 vs UCCL | V2 vs V1 | -|---:|---|---:|---:|---:|---:|---:| -| 16 ranks | FP8 | 475.540 us | 881.700 us | 384.994 us | 19.04 percent lower | 56.34 percent lower | -| 16 ranks | BF16 | 710.180 us | 886.920 us | 392.738 us | 44.70 percent lower | 55.72 percent lower | -| 32 ranks | FP8 | 693.840 us | 1,403.720 us | 646.522 us | 6.82 percent lower | 53.94 percent lower | -| 32 ranks | BF16 | 844.520 us | 1,433.780 us | 685.084 us | 18.88 percent lower | 52.22 percent lower | - -In the native outputs, UCCL had the smallest numerical latency increase when scaling from EP16 to EP32, while the DeepEP V2 sum remained numerically smallest. For FP8, the increases were 45.91% for UCCL, 59.21% for DeepEP V1, and 67.93% for DeepEP V2. This is a single-start diagnostic observation across different timing boundaries, not a scaling-efficiency claim. - -The native decode bandwidth reports are included for completeness. UCCL and DeepEP V1 report one dispatch-plus-combine bandwidth; DeepEP V2 reports separate SO legs. The V2 dispatch/combine pair is not an aggregate and is not directly comparable to the first 2 columns. - -| EP ranks | Dispatch dtype | UCCL aggregate | DeepEP V1 aggregate | DeepEP V2 dispatch / combine SO | -|---:|---|---:|---:|---:| -| 16 ranks | FP8 | 46.37 GB/s | 25.01 GB/s | 5 GB/s / 9 GB/s | -| 16 ranks | BF16 | 40.94 GB/s | 32.78 GB/s | 9 GB/s / 9 GB/s | -| 32 ranks | FP8 | 31.78 GB/s | 15.71 GB/s | 10 GB/s / 14 GB/s | -| 32 ranks | BF16 | 34.43 GB/s | 20.28 GB/s | 16 GB/s / 14 GB/s | - -## Historical native prefill diagnostics - -The table reports each backend's native latency and cross-node bandwidth fields. For UCCL, latency is `transmit + notify` and bandwidth is the RDMA leg. For DeepEP V1, latency is the sum of the 2 printed components and bandwidth is the RDMA leg. For DeepEP V2, latency is the operation latency and bandwidth is the scale-out, or SO, leg. These measurement boundaries and bandwidth accounting conventions differ, so the table is directional. Do not rank the backends by comparing the GB/s columns alone. - -| EP ranks | Operation | UCCL latency / RDMA bandwidth | DeepEP V1 latency / RDMA bandwidth | DeepEP V2 latency / SO bandwidth | -|---:|---|---:|---:|---:| -| 16 ranks | FP8 dispatch | 1,161.120 us / 55.59 GB/s | 1,014.000 us / 65.95 GB/s | 864.688 us / 35 GB/s | -| 16 ranks | BF16 dispatch | 1,637.970 us / 74.42 GB/s | 1,639.000 us / 78.14 GB/s | 1,501.000 us / 39 GB/s | -| 16 ranks | BF16 combine | 2,096.580 us / 58.81 GB/s | 1,633.220 us / 78.62 GB/s | 1,615.000 us / 36 GB/s | -| 32 ranks | FP8 dispatch | 2,229.950 us / 51.38 GB/s | 2,369.000 us / 50.95 GB/s | 2,041.000 us / 41 GB/s | -| 32 ranks | BF16 dispatch | 3,852.970 us / 56.69 GB/s | 4,125.000 us / 56.94 GB/s | 3,753.000 us / 43 GB/s | -| 32 ranks | BF16 combine | 3,950.790 us / 55.52 GB/s | 4,106.520 us / 55.80 GB/s | 3,745.000 us / 43 GB/s | - -## Historical DeepEP V2 operation diagnostics - -SO is scale-out bandwidth over the internode path and SU is scale-up bandwidth over the intranode path. FP8 combine is included because the selected V2 test prints it, but the matched headline operation set uses BF16 combine. - -| Topology and workload | Dispatch dtype | Operation | SO bandwidth | SU bandwidth | Latency | Payload | -|---|---|---|---:|---:|---:|---:| -| EP16 decode | FP8 | dispatch | 5 GB/s | 27 GB/s | 185.637 us | 4,972,032 bytes | -| EP16 decode | FP8 | combine | 9 GB/s | 48 GB/s | 199.357 us | 9,540,352 bytes | -| EP16 decode | BF16 | dispatch | 9 GB/s | 49 GB/s | 195.257 us | 9,582,848 bytes | -| EP16 decode | BF16 | combine | 9 GB/s | 48 GB/s | 197.481 us | 9,540,352 bytes | -| EP32 decode | FP8 | dispatch | 10 GB/s | 19 GB/s | 275.659 us | 5,106,816 bytes | -| EP32 decode | FP8 | combine | 14 GB/s | 26 GB/s | 370.863 us | 9,798,976 bytes | -| EP32 decode | BF16 | dispatch | 16 GB/s | 31 GB/s | 314.156 us | 9,842,624 bytes | -| EP32 decode | BF16 | combine | 14 GB/s | 26 GB/s | 370.928 us | 9,798,976 bytes | -| EP16 prefill | FP8 | dispatch | 35 GB/s | 203 GB/s | 864.688 us | 175,106,880 bytes | -| EP16 prefill | FP8 | combine | 36 GB/s | 208 GB/s | 1,614.000 us | 335,995,680 bytes | -| EP16 prefill | BF16 | dispatch | 39 GB/s | 225 GB/s | 1,501.000 us | 337,492,320 bytes | -| EP16 prefill | BF16 | combine | 36 GB/s | 208 GB/s | 1,615.000 us | 335,995,680 bytes | -| EP32 prefill | FP8 | dispatch | 41 GB/s | 95 GB/s | 2,041.000 us | 194,860,224 bytes | -| EP32 prefill | FP8 | combine | 43 GB/s | 100 GB/s | 3,749.000 us | 373,898,464 bytes | -| EP32 prefill | BF16 | dispatch | 43 GB/s | 100 GB/s | 3,753.000 us | 375,563,936 bytes | -| EP32 prefill | BF16 | combine | 43 GB/s | 100 GB/s | 3,745.000 us | 373,898,464 bytes | - -The DeepEP V2 EFA RDMA write deltas were 19,708,762,240 bytes for EP16 decode, 111,413,870,476 bytes for EP32 decode, 507,290,310,848 bytes for EP16 prefill, and 2,742,751,387,856 bytes for EP32 prefill. - -## Historical native memory tradeoff - -All 12 cells reached 100 percent sampled peak GPU utilization. Peak HBM use was materially higher for DeepEP V2 prefill. - -| Backend | EP16 decode | EP32 decode | EP16 prefill | EP32 prefill | -|---|---:|---:|---:|---:| -| UCCL | 9,627 MiB/GPU | 9,501 MiB/GPU | **13,697 MiB/GPU** | **20,735 MiB/GPU** | -| DeepEP V1 NVSHMEM | 10,597 MiB/GPU | 11,239 MiB/GPU | 17,047 MiB/GPU | 23,793 MiB/GPU | -| DeepEP V2 NCCL GIN | **7,041 MiB/GPU** | **7,271 MiB/GPU** | 39,657 MiB/GPU | 44,315 MiB/GPU | - -## Separate EFA 3.3.0g revalidation - -On 24 August 2026, a separate revalidation updated the current 32-node B200 fleet to the EFA 3.3.0g kernel module and rdma-core 64 on kernel `6.12.100-125.179.amzn2023.x86_64`. The post-update audit passed on 32 of 32 nodes. The first targeted DeepEP V2 admission then triggered a kernel panic in the Linux device-memory mapping path called by NVIDIA UVM, with the first preserved frame at `__init_zone_device_page`. The remaining runs were stopped to avoid risking more nodes. - -That trace does not establish EFA 3.3.0g as the root cause. The historical native tables are from the earlier B200 campaign and are not presented as post-update measurements. The fair common-boundary rerun used the post-update hosts only after the required HMM and GDRCopy-device admission checks passed. The failed revalidation's benchmark resources were harvested and removed; all 32 nodes returned Ready. - -## Reproduce and collate - -The standalone DeepEP V2 build and launch harness is being added in [PR 1234](https://github.com/awslabs/awsome-distributed-ai/pull/1234). Pin the image and source revisions recorded above, use `NCCL_GIN_TYPE=5` and `NCCL_SYM_GIN_KERNELS_ENABLE=0`, and first run an INFO-level admission that proves the `Libfabric_GDAKI` context and nonzero GIN layout. Return `NCCL_DEBUG` to `WARN` for measurement. - -The collector accepts rank-zero DeepEP V2 logs from `tests/elastic/test_ep.py`: - -```bash -python3 collect_results.py \ - --nvshmem-internode nvshmem_prefill.log \ - --nvshmem-lowlat nvshmem_decode.log \ - --uccl-internode uccl_prefill.log \ - --uccl-lowlat uccl_decode.log \ - --deepep-v2-prefill deepep_v2_prefill.log \ - --deepep-v2-decode deepep_v2_decode.log \ - --nccl nccl_alltoall.log -``` - -Preserve every rank's log, rendered launch manifest, immutable image reference, command line, correctness result, and before-and-after EFA counters. `collect_results.py` summarizes rank zero, but correctness and path validation must cover every rank and every node. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md deleted file mode 100644 index 7ae324719..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md +++ /dev/null @@ -1,78 +0,0 @@ -# EP-Backend Comparison Results — P5 (H100) - -Same harness and matched config as [`RESULTS.md`](RESULTS.md) (B300), run on `p5.48xlarge` -(H100). Use this to compare the three historical dispatchers **across GPU generations**. The -newer B200 direct comparison with DeepEP V2 is in [`RESULTS-b200.md`](RESULTS-b200.md) and is -not mixed with these H100 measurements. - -## Environment - -| Field | Value | -|---|---| -| Date | 2026-06-22 | -| Hardware | `p5.48xlarge` on EKS (H100, 8 GPU + **32 EFA** / node) | -| EP config | num-tokens=4096 (LL: 128), hidden=7168, num-topk=8, num-experts=256, bf16 (identical to B300) | -| NVSHMEM image | `deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13` (sm_90) | -| UCCL image | UCCL `0dc87eb`, CUDA 13 (Hopper sm_90 path of the committed `uccl-ep.Dockerfile`) | -| NCCL image | DeepEP image (`/opt/nccl-tests/build/alltoall_perf`) | - -> **Node bring-up note.** One of the 8 freshly-deployed P5 nodes initially had its NVSwitch -> **Fabric Manager stuck in `In Progress`** (every CUDA op on it failed with `error 802: -> system not yet initialized`, including a trivial 1-GPU test). The 4-node run was done on the -> 7 healthy nodes meanwhile; after that node was recycled the 8-node run completed on all 8. - -## 8 nodes — 64 ranks - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 39.6 | 39.2 | -| NVSHMEM (DeepEP) | low-latency | 6.5 | 15.6 | -| UCCL (UCCL-EP) | internode (RDMA) | 29.9 | 26.7 | -| UCCL (UCCL-EP) | low-latency | 3.1¹ | 3.7¹ | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 40.4 | -| busbw peak (asymptotic) | peak | 51.1 | - -¹ Same as B300, the standard FP8 low-latency path (`round_scale=False`) **passes** correctness at -64 ranks (max diff 1.07e-4 vs the 9e-4 FP8 tolerance — 8× margin) and gives these numbers. The -unpatched test aborts *first* on the coarser `round_scale=True` FP8 sub-case, which upstream DeepEP -exempts via `if not round_scale`; matching that gating recovers the bandwidth. The per-sub-case -errors are **identical to B300** (the reference is generated from fixed seeds), confirming this is a -quantization-recipe property, not GPU-arch. See [`RESULTS.md`](RESULTS.md) for the sub-case table. - -## 4 nodes — 32 ranks - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 43.7 | 46.1 | -| NVSHMEM (DeepEP) | low-latency | 9.6 | 20.7 | -| UCCL (UCCL-EP) | internode (RDMA) | 38.2 | 30.2 | -| UCCL (UCCL-EP) | low-latency | 5.0 | 5.0 | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 45.1 | -| busbw peak (asymptotic) | peak | 53.9 | - -## Observations (and the B300 contrast) - -- **The winner flips by GPU generation.** On **B300** (see `RESULTS.md`) UCCL matches/beats - NVSHMEM at 4 nodes and clearly wins at 8. On **P5/H100 the order reverses at both scales**: - NVSHMEM leads UCCL on internode (4n 44/46 vs 38/30; 8n **40/39 vs 30/27** GB/s RDMA) and on - low-latency (8n LL 6.5/15.6 vs 3.1/3.7). UCCL-EP's kernels lean on SM90+ features tuned for - Blackwell; on H100 they trail NVSHMEM here. **Pick the dispatcher per target GPU, not globally.** -- **Absolute bandwidth is ~half of B300.** P5 internode tops out ~40–46 GB/s (RDMA) and the NCCL - reference ~40–45 (matched) / ~51–54 (peak), versus ~73–96 / ~103–117 on B300 — a combination of - EFA throughput and NVLink-generation differences. -- **UCCL low-latency's 64-rank abort is a test-gate divergence, not a kernel fault.** The standard - `round_scale=False` FP8 path passes at 64 ranks (max diff 1.07e-4, identical on P5 and B300); the - default test aborts only on the coarser `round_scale=True` sub-case that DeepEP exempts via - `if not round_scale`. Matching that gating recovers the LL bandwidth (P5 3.1/3.7, B300 28.4/24.8). - See [`RESULTS.md`](RESULTS.md) for the sub-case breakdown. - -## Reproduce - -Identical to [`README.md`](README.md), with `INSTANCE_TYPE=p5.48xlarge` and `EFA_PER_NODE=32` -(p5.48xlarge exposes 32 EFA NICs vs 16 on p6-b300). diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index b49023181..53b835b51 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -1,185 +1,138 @@ -# EP-Backend Comparison Results — B300 +# Fair EP Backend Comparison Results on B200 -Generated by running the three backends at a matched EP config on `p6-b300.48xlarge` -nodes and collating with [`collect_results.py`](collect_results.py). For the **P5 (H100)** run -of the same matrix — and the cross-generation contrast (the winner flips by GPU) — see -[`RESULTS-p5.md`](RESULTS-p5.md). For the newer **B200** direct comparison that adds DeepEP V2 -over NCCL GIN EFA-GDA, see [`RESULTS-b200.md`](RESULTS-b200.md). Hardware generations and -campaigns are reported separately. +This report replaces the retired backend-native B300, B200, and H100 tables. Those tables mixed timing boundaries and byte numerators, so their latency and bandwidth columns were not suitable for cross-backend ranking. -## Environment +The replacement measures one synthetic decode dispatch-plus-combine workload with one external CUDA timing boundary and one logical payload definition. It is not an end-to-end training or serving benchmark. EP32 means 32 GPU ranks on 4 `p6-b200.48xlarge` nodes, not 32 instances. + +## Result + +UCCL has the lowest latency in 3 of the 4 measured workload cells. DeepEP V2 has the lowest latency at EP16 with BF16 dispatch. DeepEP V2 is faster than DeepEP V1 in all 4 cells. These observations apply only to the configuration and scales in this report. + +Each value is the median across 3 independent process starts. Each process start contributes the median of 100 slowest-rank measured iterations after 20 warmup iterations. + +| EP size | Dispatch dtype | Backend | Latency | 95% bootstrap CI | Run-to-run CV | Input throughput | Logical throughput | Scale-out logical throughput | +|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| +| 16 ranks | FP8 | UCCL | 0.5213 ms | [0.5190, 0.5277] ms | 0.86% | 3,928,790.51 tokens/s | 42.68 GB/s/rank | 21.34 GB/s/rank | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 1.0179 ms | [1.0167, 1.0203] ms | 0.18% | 2,011,977.59 tokens/s | 21.86 GB/s/rank | 10.93 GB/s/rank | +| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 0.6219 ms | [0.6180, 0.6375] ms | 1.65% | 3,293,197.58 tokens/s | 35.78 GB/s/rank | 17.89 GB/s/rank | +| 16 ranks | BF16 | UCCL | 0.5920 ms | [0.5918, 0.5978] ms | 0.58% | 3,459,365.90 tokens/s | 49.59 GB/s/rank | 24.80 GB/s/rank | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 1.0297 ms | [1.0251, 1.0304] ms | 0.28% | 1,988,936.56 tokens/s | 28.51 GB/s/rank | 14.26 GB/s/rank | +| 16 ranks | BF16 | DeepEP V2 NCCL GIN | 0.4850 ms | [0.4826, 0.5043] ms | 2.42% | 4,222,332.23 tokens/s | 60.53 GB/s/rank | 30.27 GB/s/rank | +| 32 ranks | FP8 | UCCL | 0.7655 ms | [0.7640, 0.7714] ms | 0.51% | 5,350,835.03 tokens/s | 29.07 GB/s/rank | 21.80 GB/s/rank | +| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 1.5542 ms | [1.5513, 1.5628] ms | 0.38% | 2,635,371.65 tokens/s | 14.32 GB/s/rank | 10.74 GB/s/rank | +| 32 ranks | FP8 | DeepEP V2 NCCL GIN | 0.9537 ms | [0.9217, 0.9675] ms | 2.48% | 4,294,725.45 tokens/s | 23.33 GB/s/rank | 17.50 GB/s/rank | +| 32 ranks | BF16 | UCCL | 0.8686 ms | [0.8683, 0.8696] ms | 0.08% | 4,715,590.93 tokens/s | 33.80 GB/s/rank | 25.35 GB/s/rank | +| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 1.5589 ms | [1.5526, 1.5714] ms | 0.61% | 2,627,446.53 tokens/s | 18.83 GB/s/rank | 14.13 GB/s/rank | +| 32 ranks | BF16 | DeepEP V2 NCCL GIN | 0.9509 ms | [0.9482, 0.9606] ms | 0.68% | 4,307,661.17 tokens/s | 30.88 GB/s/rank | 23.16 GB/s/rank | + +The maximum run-to-run CV is 2.48%. The 95% intervals use 20,000 bootstrap resamples of the 3 independent process-start medians. With only 3 independent starts, the intervals describe this campaign but should not be read as precise estimates of production variability. + +## Paired DeepEP V2 latency deltas + +Starts are paired by EP size, dtype, input, route, named nodes, and rotation index. A positive value means DeepEP V2 had lower latency than the baseline. A direction is marked supported only when the paired bootstrap interval excludes 0% and both arms have no more than 5% run-to-run CV. + +| EP size | Dispatch dtype | Baseline | Median V2 latency reduction | 95% bootstrap CI | Direction supported | +|---:|:---:|:---|---:|:---:|:---:| +| 16 ranks | FP8 | UCCL | -19.30% | [-22.84%, -17.11%] | Yes | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 39.05% | [37.37%, 39.22%] | Yes | +| 16 ranks | BF16 | UCCL | 18.44% | [14.82%, 18.87%] | Yes | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 52.68% | [51.06%, 53.13%] | Yes | +| 32 ranks | FP8 | UCCL | -24.83% | [-25.41%, -20.40%] | Yes | +| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 38.64% | [37.64%, 41.02%] | Yes | +| 32 ranks | BF16 | UCCL | -9.35% | [-10.63%, -9.16%] | Yes | +| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 38.87% | [38.76%, 39.18%] | Yes | + +"Direction supported" is a campaign-level reproducibility rule, not a universal performance claim or a formal significance test. + +## Common comparison boundary + +The harness holds the semantic workload and measurement rules constant across the 3 backends: + +| Control | Value | +|---|---| +| Input | Deterministic BF16 tensor, 128 tokens/rank | +| Model shape | Hidden size 7,168, 256 experts, top-k 8 experts/token | +| Operation | FP8 or BF16 dispatch followed by BF16 combine | +| Expert work | Identity-expert semantics; expert compute is outside the timed boundary | +| Timing | CUDA Events from input preparation through dispatch and combine completion | +| Rank reduction | Maximum elapsed time across all ranks for every measured iteration | +| Warmup and measurement | 20 warmup iterations and 100 measured iterations per dtype and process start | +| Replication | 3 independent process starts per backend and workload cell | +| Order control | Backend order and dtype order rotate across process starts | +| Hardware control | The same named nodes are used for every backend at each EP size | +| Input control | Input and routing SHA-256 hashes must match across all backends and starts in each EP size | +| Runtime control | GPU model, PyTorch, CUDA, and NCCL versions must match across every scored result | +| Image control | Every backend image must be digest-pinned and invariant across the matrix | +| Correctness | Every rank must pass the common identity-expert output check before timing | + +DeepEP V2 uses non-expanded dispatch, which sends a token once per destination rank. The correctness path applies the local gated identity-expert reduction before combine so V2 and the expanded backends return the same weighted token. This normalizes semantic output, not the internal algorithm. + +## Logical throughput definition + +The cross-backend throughput columns use the same useful-payload numerator. Each valid expert assignment contributes its dispatch tensor, FP8 scales when FP8 is selected, and its BF16 combine tensor. Backend-specific metadata is excluded. Scale-out logical bytes count only assignments whose destination expert is on a different node. + +```text +logical GB/s/rank = average logical bytes/rank / median slowest-rank latency +scale-out logical GB/s/rank = average remote logical bytes/rank / median slowest-rank latency +``` + +These values are effective logical throughput, not observed wire bandwidth. They intentionally replace the retired side-by-side native bandwidth columns. In DeepEP V2, for example, SO bandwidth uses `num_scaleout_bytes / t`, SU bandwidth uses `num_scaleup_bytes / t`, and the trailing `bytes` field is the SU numerator. DeepEP V1 and UCCL use their own byte accounting and aggregation boundaries. Those native metrics remain useful for diagnosing a backend, but comparing them as one GB/s metric is not valid. + +## Correctness and admission + +All 36 scored dtype results passed on every rank. The normalized-difference tolerance was `9e-4` dimensionless for FP8 dispatch and `1e-5` dimensionless for BF16 dispatch. The matrix contains 18 distributed process starts: 2 EP sizes multiplied by 3 backends and 3 independent starts, with 2 dtype results from each start. + +Before scoring, the DeepEP V2 EP16 admission run also verified: + +- `uvm_disable_hmm=1` on all 4 selected hosts; +- `/dev/gdrdrv` present as a character device on all 4 selected hosts; and +- an INFO-level GDAKI context in the DeepEP V2 log. + +## Environment and provenance | Field | Value | |---|---| -| Date | 2026-06-21 | -| Hardware | `p6-b300.48xlarge` on EKS (Blackwell B300, 8 GPU + 16 EFA / node) | -| EP config | num-tokens=4096 (LL: 128), hidden=7168, num-topk=8, num-experts=256, bf16 | -| NVSHMEM image | `deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13` (CUDA 13) | -| UCCL image | UCCL `0dc87eb`, CUDA 13, Hopper+Blackwell — see image note below | -| NCCL image | DeepEP image (reuses `/opt/nccl-tests/build/alltoall_perf`, sm_100) | - -`num-experts=256` divides evenly at both 32 and 64 ranks, so the config is identical across the -4-node and 8-node runs. - -> **UCCL image note.** The UCCL dispatch/combine numbers below were measured with a prebuilt -> image of the **same UCCL commit (`0dc87eb`) and the same CUDA-13 `setup.py`+PTX build method** -> as the committed [`uccl-ep.Dockerfile`](../uccl-ep-benchmark/uccl-ep.Dockerfile). That committed -> Dockerfile is independently validated to **build** and to **run the benchmark on B300** — a -> 2-node internode run from it gives dispatch ~92 / combine ~60 GB/s (RDMA), consistent with the -> table — so it reproduces these results. - -## 32 / 16 nodes — 256 / 128 ranks (2026-07-14 scale run) - -A 32× `p6-b300.48xlarge` Capacity Block (**us-east-1-atl-2a local zone**, EKS, same images -as below) was used to sweep the same matched config from 2 to 32 nodes. Headline: -**every DeepEP-class kernel hits a hard implementation limit between 65 and 256 ranks — -at 256 ranks only the NCCL all-to-all reference runs.** - -**HT internode scaling (like-for-like, this cluster, 2026-07-14).** The matched config -(`num-experts=256`) is only *runnable* at power-of-2 node counts: -`tests/test_internode.py:30` asserts `num_experts % num_ranks == 0`, and 256 divides -16/32/64/128/256 ranks but not the 24/48/96/144/160 of DeepEP's other instantiated -shapes ({3, 6, 12, 18, 20} nodes — a menu cut for 288-expert models, where those counts -do divide). Dispatch/combine are the RDMA leg in GB/s: - -| nodes | ranks | NVSHMEM (DeepEP) disp / comb | UCCL disp / comb | NCCL matched / peak | -|---|---|---|---|---| -| 2 | 16 | **126.6 / 106.4** | 91.9 / 59.6 | 104.9 / 179.6 | -| 4 | 32 | 97.1 / 95.4 | **102.1 / 95.3** | 94.0 / 116.9 | -| 8 | 64 | 84.2 / 73.1 | **93.9 / 90.5** | 74.0 / 103.2 | -| 16 | 128 | 74.7 / tuning abort¹ | constructor abort³ | 73.8 / 84.0 | -| 32 | 256 | constructor abort² | constructor abort³ | 54.7 / 74.4 | - -Two reads: **(a)** the 8-node row reproduces the June `us-west-2` primary table below -within ~1% on a different cluster (NVSHMEM 84.2/73.1 vs 83.7/72.7; UCCL 93.9/90.5 vs -93.4/90.7; NCCL 74.0 vs 72.5) — strong cross-cluster reproducibility for these -benchmarks. **(b)** the winner flips with scale: NVSHMEM leads at 2 nodes, UCCL from -4 nodes up — per-rank bandwidth decays smoothly for all three as fan-out grows. - -**Low-latency kernels** cap between 64 and 128 ranks on both implementations -(⁴ and ⁵ below); at 256 ranks nothing but NCCL runs: - -| Backend / kernel | 128 ranks (16n) | 256 ranks (32n) | Limit (source) | -|---|---|---|---| -| NVSHMEM (DeepEP) low-latency | init traffic abort⁴ | init traffic abort⁴ | 64 < cap ≤ 128 PEs (host-proxy) | -| UCCL (UCCL-EP) low-latency | buffer-config abort⁵ | buffer-config abort⁵ | 64 < cap ≤ 128 ranks | - -¹ `internode.cu:2363` — `num_max_nvl_chunked_recv_tokens / num_rdma_ranks > max(send chunk)`: -the shipped chunk configs were tuned for ≤8-node EP domains; at 16 RDMA peers the combine -sweep violates the constraint. Dispatch (74.7 GB/s RDMA, ~143 GB/s NVL) is a valid datapoint. -² `deep_ep.cpp:158` — `num_ranks <= NUM_MAX_NVL_PEERS(8) × NUM_MAX_RDMA_PEERS(20) = 160 or -low_latency_mode`. DeepEP v1's HT kernels decompose every rank into `(rdma_rank, nvl_rank)` -against fixed 8×20 compile-time tables (`kernels/configs.cuh`). The same assert block also -bounds `num_rdma_bytes <= INT_MAX` for HT — waiting behind the peer cap even if it were raised. -³ `uccl_ep.cc:431` — HT `num_rdma_bytes` (∝ ranks at matched config) exceeds `INT_MAX` -somewhere between 64 ranks (8n, June: passes at 93 GB/s) and 128 ranks. -⁴ NVSHMEM 3.7 libfabric **host-proxy** retry exhaustion: `Max amount of libfabric retries -reached, -11 (EAGAIN)` in `nvshmemi_process_multisend_rma`, killing ~2 nodes per run with -**different victims each run** (4 runs: ranks {12,29}, {9,12}, {12}, {6,11}) — a statistical -fan-out limit of the single proxy thread at ≥128 PEs on EFA, not a bad node and not geometry. -Registration itself is solvable: the ≥1 GiB LL buffer forces dynamic (CUDA-VMM) heap growth -whose remote-chunk registration needs **GDRCopy in-container** — on clusters whose toolkit -ignores `NVIDIA_GDRCOPY=enabled`, hostPath-mount `/dev/gdrdrv` (privileged); a >2 GiB static -heap is no workaround (exceeds EFA's single-MR registration limit). -⁵ `ep_config.hpp:279` — LL per-peer signaling buffer (∝ ranks) exceeds the compile-time -`kAtomicBufferSize` arena at ≥128 ranks. - -**Reading.** DeepEP-class dispatchers are engineered for EP domains of ~64–160 ranks — -matching how training actually deploys them (EP32/EP64 groups inside a larger world; the -[`kimi-k2`](../../../3.test_cases/megatron/megatron-bridge/kimi-k2/benchmarks/RESULTS.md) -NVSHMEM arm ran clean on 256 GPUs precisely because its `deep_ep` domains are 32-rank EP -groups). A *flat* EP domain >64 ranks is already off the map for both low-latency paths on -EFA, and >160 for HT; NCCL all-to-all is the only working option there. NCCL's per-rank -busbw at the matched ~56 MiB payload decays smoothly with fan-out across the whole sweep: -104.9 → 94.0 → 74.0 → 73.8 → 54.7 GB/s over 16 → 256 ranks (like-for-like, this cluster). - -## 8 nodes — 64 ranks (primary) - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 83.7 | 72.7 | -| NVSHMEM (DeepEP) | low-latency | 9.2 | 18.6 | -| UCCL (UCCL-EP) | internode (RDMA) | 93.4 | 90.7 | -| UCCL (UCCL-EP) | low-latency | 28.4¹ | 24.8¹ | - -¹ The standard FP8 low-latency path (`round_scale=False`) **passes** correctness at 64 ranks and -gives these numbers. The unpatched test still aborts *first* on the coarser `round_scale=True` FP8 -sub-case — which upstream DeepEP exempts from the same check — so reproducing with the committed -manifest shows the abort; matching DeepEP's gating recovers the bandwidth. See the note below. - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 72.5 | -| busbw peak (asymptotic) | peak | 103.1 | - -> **UCCL low-latency at 64 ranks: the abort is a test-gate divergence, not a kernel precision regression.** -> `bench/test_low_latency.py` validates a dispatch→(identity GEMM)→combine round-trip against the -> closed-form reference `current_x · Σ(top-k gate weights)`, using the global similarity error -> `calc_diff(x,y) = 1 − 2·⟨x,y⟩ / (‖x‖²+‖y‖²)`. UCCL asserts `diff < 9e-4` for **every** FP8 -> sub-case: -> -> ```python -> assert diff < (9e-4 if dispatch_use_fp8_case else 1e-5) # uccl ep/bench/test_low_latency.py:373 -> ``` -> -> whereas upstream DeepEP gates the *same* assert with `if not round_scale` -> (`deepseek-ai/DeepEP tests/test_low_latency.py:178`) — it deliberately exempts the coarse, -> power-of-2-scale (`round_scale=True`) FP8 path. Instrumenting the run to log every sub-case's error -> at 64 ranks (256 samples per sub-case across the 64 ranks) shows the split exactly: -> -> | dispatch_use_fp8 | round_scale | max diff | tolerance | result | -> |---|---|---:|---:|---| -> | False (bf16) | — | 1.7e-6 | 1e-5 | pass | -> | True | **False** | **1.07e-4** | 9e-4 | **pass (8× margin)** | -> | True | **True** | **1.80e-3** | 9e-4 | fail | -> -> So the **standard `round_scale=False` FP8 path is numerically correct at 64 ranks** (and at 32); -> the default test aborts only because UCCL applies the tight FP8 tolerance to the `round_scale=True` -> sub-case that DeepEP does not gate. `round_scale=True` is a coarser (power-of-2 / UE8M0-family) -> scaling recipe whose larger quantization error is expected — hence DeepEP's exemption. The -> dispatch/combine numbers in the table are the `round_scale=False` FP8 timed phase, recovered by -> matching DeepEP's gating (skipping the `round_scale=True` assert) — **not** by loosening the -> tolerance on the path the table measures. - -## 4 nodes — 32 ranks (scaling reference) - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 97.2 | 95.6 | -| NVSHMEM (DeepEP) | low-latency | 11.2 | 25.2 | -| UCCL (UCCL-EP) | internode (RDMA) | 101.3 | 94.4 | -| UCCL (UCCL-EP) | low-latency | 44.5 | 45.3 | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 93.9 | -| busbw peak (asymptotic) | peak | 116.9 | - -## Observations - -- **Internode high-throughput: a tie at 4 nodes, a UCCL win at 8.** At 32 ranks NVSHMEM and UCCL - are within a few percent (97/96 vs 101/94). At 64 ranks UCCL pulls ahead on both legs — - dispatch 93 vs 84 and **combine 91 vs 73 GB/s (RDMA)** — i.e. UCCL's combine degrades far less - as the RDMA peer count doubles. Bandwidth drops with scale for both backends (more RDMA peers, - more contention). -- **Low-latency (decode path): UCCL wins.** At 32 ranks UCCL delivers ~4× dispatch and ~1.8× - combine over NVSHMEM (44/45 vs 11/25). At 64 ranks UCCL leads on both legs too — **28.4/24.8 vs - 9.2/18.6** (~3× dispatch, ~1.3× combine) — once the test's `round_scale=True` gate (which DeepEP - exempts) is matched; the standard `round_scale=False` FP8 path passes at this scale (see note). -- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure data movement (no token - routing / combine-reduction) and is measured differently from the EP kernels' RDMA-only, - NVL-overlapped bandwidth — so the EP numbers can sit *above* the NCCL matched-size busbw (e.g. - UCCL 93 vs 72.5 at 8 nodes) while staying below the asymptotic peak (103). Treat it as a - transport context line, not a hard ceiling. - -## Reproduce - -See [`README.md`](README.md). Logs were collected with `kubectl logs > .log` -and collated with: - -```bash -python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log --uccl-lowlat uccl_lowlat.log \ - --nccl nccl_alltoall.log +| Campaign | `fair-ep-b200-20260824t195145z` | +| Date | 2026-08-24 UTC | +| AWS Region and Availability Zone | `ap-south-1`, `ap-south-1c` | +| EKS cluster | `ml-clusters-shared-ap-south-1` | +| Instance type | `p6-b200.48xlarge` | +| Node topology | 4 nodes, 8 NVIDIA B200 GPUs/node, 8 allocatable EFA devices/node | +| Runtime | PyTorch `2.13.0+cu130`, CUDA `13.0`, NCCL `2.29.7` | + +Digest-pinned images: + +| Backend | Image | +|---|---| +| UCCL | `159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a` | +| DeepEP V1 NVSHMEM | `159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac` | +| DeepEP V2 NCCL GIN | `159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb` | + +Selected nodes: + +```text +ip-10-6-124-201.ap-south-1.compute.internal +ip-10-6-126-97.ap-south-1.compute.internal +ip-10-6-127-248.ap-south-1.compute.internal +ip-10-6-71-166.ap-south-1.compute.internal ``` + +## Concurrent campaign isolation + +The campaign ran in coordinated observe mode beside the vLLM campaign identified by Lease holder `dsv3-b200-parallel-r7-20260824t2122z`. The vLLM campaign had 12 protected nodes. The selected-node overlap was 0 nodes. The runner did not mutate the shared Lease and rechecked its exact holder before each arm and before aggregation. + +Node disjointness prevents direct GPU and host contention. It does not prove absence of shared network-fabric effects, so these results remain specific to the observed cluster conditions. + +## Artifact custody and teardown + +The durable artifact contains 19 canonical result files and 38 result records, including 36 scored records and 2 admission records. It retains the rendered manifests, all-rank logs, Pod descriptions, image references, input and route hashes, admission evidence, cluster snapshots, teardown evidence, and a `SHA256SUMS` manifest covering 523 files. + +The detached runner returned a nonzero status after all 18 scored jobs passed because one backend diagnostic was appended to a JSON result line. The parser was corrected, canonical records were re-extracted from the preserved rank-zero logs, and the complete matrix was revalidated. No benchmark value was reconstructed or rerun during recovery. + +Final teardown verification passed: the owned namespace was absent, 0 owned resources remained, all 4 selected nodes were Ready, and 0 selected-node GPU Pods remained. The concurrent vLLM Lease holder was unchanged. + +The committed machine-readable summary is [`results/b200-ap-south-1-2026-08-24.json`](results/b200-ap-south-1-2026-08-24.json), SHA-256 `129877f60412e4a87fe1b8dd29074bcf803457d9c8b5c816276616fd09700f66`. + +## Limits + +This benchmark does not measure prefill, expert compute, communication/computation overlap, memory footprint, end-to-end training, serving throughput, TTFT, TPOT, or end-to-end latency. It covers only EP16 and EP32 on B200 in one Availability Zone, with 3 independent starts per cell. It does not establish behavior on B300, H100, larger EP domains, different routing distributions, or production Kimi K2 training replicas. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py deleted file mode 100644 index 041db3af4..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py +++ /dev/null @@ -1,238 +0,0 @@ -#!/usr/bin/env python3 -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -"""Collate EP-backend comparison logs into markdown tables. - -Parses launcher logs from the EP backends run at the same world size: - - * NVSHMEM (DeepEP V1) -- dispatch/combine bandwidth - * UCCL (UCCL-EP) -- dispatch/combine bandwidth - * DeepEP V2 (NCCL GIN) -- scale-out/scale-up bandwidth and latency - * NCCL (nccl-tests alltoall_perf) -- busbw (transport reference) - -Output formats parsed (DeepEP V1 and UCCL print identically -- UCCL's bench is -derived from the V1 tests): - - internode (test_internode.py): - [tuning] Best dispatch (BF16): ... BW: GB/s (RDMA), GB/s (NVL) - [tuning] Best combine: ... BW: GB/s (RDMA), GB/s (NVL) - -> we report the RDMA leg (the cross-node bottleneck). Reporting the NVL leg - here would be wrong: it is the intra-node ~hundreds-GB/s number. - - low-latency (test_low_latency.py): - [rank N] Dispatch bandwidth: GB/s, avg_t=... | Combine bandwidth: GB/s, avg_t=... - -> single bandwidth per dispatch/combine (no RDMA/NVL split). - - DeepEP V2 (tests/elastic/test_ep.py): - > Testing with ... use_fp8_dispatch=<0|1> ... - * EP: 0/ | dispatch: GB/s (SO), GB/s (SU), us, ... - @ EP: 0/ | combine: GB/s (SO), GB/s (SU), us, ... - -> report rank zero for each dispatch dtype and operation. SO is the - scale-out leg and SU is the scale-up leg. - - NCCL alltoall_perf: the standard size-sweep table; busbw is column 7 - (out-of-place) / 11 (in-place). We report busbw at the sampled row whose size - is closest to the EP per-rank dispatch payload (num_tokens * hidden * 2 bytes, - ~56 MiB target; the power-of-two sweep lands on the 64 MiB row) AND the - asymptotic peak, because the peak overstates the transport context relative to - EP's smaller messages. - -Both EP formats vary slightly across versions; if a value comes back N/A, print -the raw log and adjust the regexes. - -Usage: - python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log \ - --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log \ - --uccl-lowlat uccl_lowlat.log \ - --deepep-v2-prefill deepep_v2_prefill.log \ - --deepep-v2-decode deepep_v2_decode.log \ - --nccl nccl_alltoall.log \ - --nccl-target-bytes 58720256 -""" -import argparse -import re -import sys - -_BW = r"([0-9]+(?:\.[0-9]+)?)" -# internode: pull the RDMA leg from the "Best dispatch/combine" summary lines. -BEST_DISPATCH_RDMA = re.compile(r"Best dispatch[^\n]*?" + _BW + r"\s*GB/s\s*\(RDMA\)", re.I) -BEST_COMBINE_RDMA = re.compile(r"Best combine[^\n]*?" + _BW + r"\s*GB/s\s*\(RDMA\)", re.I) -# low-latency: "Dispatch bandwidth: X GB/s" / "Combine bandwidth: Y GB/s". -# Case-SENSITIVE on purpose: the aggregate line "Dispatch + combine bandwidth: Z" -# uses a lowercase "combine" and must NOT match the per-leg "Combine bandwidth:". -LL_DISPATCH = re.compile(r"Dispatch bandwidth:\s*" + _BW + r"\s*GB/s") -LL_COMBINE = re.compile(r"Combine bandwidth:\s*" + _BW + r"\s*GB/s") -# nccl-tests data row: size count type redop root time algbw busbw #wrong ... -NCCL_ROW_RE = re.compile(r"^\s*\d+\s+\d+\s+\w+") -NCCL_BUSBW_COLS = (7, 11) -# mpirun --tag-output prefixes every line with e.g. "[1,0]:". -TAG_PREFIX_RE = re.compile(r"^\[\d+,\d+\]:") -# DeepEP V2 prints the dtype before each selected test case, then one result per -# EP rank. Restrict results to rank zero and to the base dispatch/combine rows; -# expanded/cached dispatch rows have different semantics. -V2_DTYPE_RE = re.compile(r"use_fp8_dispatch=(0|1)") -V2_RANK_ZERO_RE = re.compile( - r"[*!#@+\-]\s*EP:\s*0/\d+\s*\|\s*" - r"(dispatch|combine):\s*" + _BW + r"\s*GB/s\s*\(SO\),\s*" - + _BW + r"\s*GB/s\s*\(SU\),\s*" + _BW + r"\s*us", - re.I, -) - - -def _last(text, regex): - m = regex.findall(text) - return float(m[-1]) if m else None - - -def _max(text, regex): - m = regex.findall(text) - return max(float(x) for x in m) if m else None - - -def parse_internode(path): - """(dispatch_rdma, combine_rdma) GB/s from the Best-config summary lines.""" - if not path: - return None, None - with open(path) as f: - text = f.read() - # The "Best" line is printed once per config after tuning; take the last. - return _last(text, BEST_DISPATCH_RDMA), _last(text, BEST_COMBINE_RDMA) - - -def parse_lowlat(path): - """(dispatch, combine) GB/s. Per-rank lines; take the max across ranks.""" - if not path: - return None, None - with open(path) as f: - text = f.read() - return _max(text, LL_DISPATCH), _max(text, LL_COMBINE) - - -def parse_nccl(path, target_bytes): - """Return (busbw_at_target, size_at_target, busbw_peak) GB/s.""" - if not path: - return None, None, None - peak = None - best_at = None # (abs_size_delta, size, busbw) - with open(path) as f: - for line in f: - line = TAG_PREFIX_RE.sub("", line) - if line.lstrip().startswith("#") or not NCCL_ROW_RE.match(line): - continue - cols = line.split() - try: - size = int(cols[0]) - except ValueError: - continue - for idx in NCCL_BUSBW_COLS: - if idx >= len(cols): - continue - try: - bw = float(cols[idx]) - except ValueError: - continue - peak = bw if peak is None else max(peak, bw) - delta = abs(size - target_bytes) - if best_at is None or delta < best_at[0]: - best_at = (delta, size, bw) - if best_at is None: - return None, None, peak - return best_at[2], best_at[1], peak - - -def parse_deepep_v2(path): - """Return rank-zero DeepEP V2 metrics keyed by (dtype, operation). - - Each value is ``(scale_out_gbps, scale_up_gbps, latency_us)``. A log that - omits the dtype marker is retained under ``unspecified`` instead of being - silently discarded. - """ - if not path: - return {} - dtype = "unspecified" - results = {} - with open(path) as f: - for line in f: - dtype_match = V2_DTYPE_RE.search(line) - if dtype_match: - dtype = "FP8" if dtype_match.group(1) == "1" else "BF16" - result_match = V2_RANK_ZERO_RE.search(line) - if not result_match: - continue - operation, scale_out, scale_up, latency = result_match.groups() - results[(dtype, operation.lower())] = ( - float(scale_out), - float(scale_up), - float(latency), - ) - return results - - -def fmt(v): - return f"{v:.1f}" if isinstance(v, float) else "N/A" - - -def fmt_latency(v): - return f"{v:.3f}" if isinstance(v, float) else "N/A" - - -def main(): - p = argparse.ArgumentParser() - p.add_argument("--nvshmem-internode") - p.add_argument("--nvshmem-lowlat") - p.add_argument("--uccl-internode") - p.add_argument("--uccl-lowlat") - p.add_argument("--deepep-v2-prefill") - p.add_argument("--deepep-v2-decode") - p.add_argument("--nccl", help="NCCL alltoall_perf log (transport reference)") - p.add_argument("--nccl-target-bytes", type=int, default=4096 * 7168 * 2, - help="EP per-rank dispatch payload to read busbw at (default num_tokens*hidden*2)") - args = p.parse_args() - - nv_i_d, nv_i_c = parse_internode(args.nvshmem_internode) - nv_l_d, nv_l_c = parse_lowlat(args.nvshmem_lowlat) - uc_i_d, uc_i_c = parse_internode(args.uccl_internode) - uc_l_d, uc_l_c = parse_lowlat(args.uccl_lowlat) - v2_prefill = parse_deepep_v2(args.deepep_v2_prefill) - v2_decode = parse_deepep_v2(args.deepep_v2_decode) - nccl_at, nccl_size, nccl_peak = parse_nccl(args.nccl, args.nccl_target_bytes) - - out = sys.stdout - out.write("| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) |\n") - out.write("|---|---|---:|---:|\n") - out.write(f"| DeepEP V1 (NVSHMEM) | internode (RDMA) | {fmt(nv_i_d)} | {fmt(nv_i_c)} |\n") - out.write(f"| DeepEP V1 (NVSHMEM) | low-latency | {fmt(nv_l_d)} | {fmt(nv_l_c)} |\n") - out.write(f"| UCCL (UCCL-EP) | internode (RDMA) | {fmt(uc_i_d)} | {fmt(uc_i_c)} |\n") - out.write(f"| UCCL (UCCL-EP) | low-latency | {fmt(uc_l_d)} | {fmt(uc_l_c)} |\n") - out.write("\n") - sz_mib = f"{nccl_size / 2**20:.0f} MiB" if isinstance(nccl_size, int) else "N/A" - out.write("| Reference (NCCL all-to-all, transport context) | Metric | GB/s |\n") - out.write("|---|---|---:|\n") - out.write(f"| busbw at EP payload (~{sz_mib}) | matched-size | {fmt(nccl_at)} |\n") - out.write(f"| busbw peak (asymptotic context) | peak | {fmt(nccl_peak)} |\n") - - if v2_prefill or v2_decode: - out.write("\n") - out.write( - "| Backend | Workload | Dispatch dtype | Operation | " - "Scale-out (GB/s) | Scale-up (GB/s) | Latency (us) |\n" - ) - out.write("|---|---|---|---|---:|---:|---:|\n") - for workload, results in (("prefill", v2_prefill), ("decode", v2_decode)): - for dtype in ("FP8", "BF16", "unspecified"): - for operation in ("dispatch", "combine"): - values = results.get((dtype, operation)) - if not values: - continue - scale_out, scale_up, latency = values - out.write( - f"| DeepEP V2 (NCCL GIN) | {workload} | {dtype} | " - f"{operation} | {fmt(scale_out)} | {fmt(scale_up)} | " - f"{fmt_latency(latency)} |\n" - ) - - -if __name__ == "__main__": - main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py deleted file mode 100644 index 2c1e09be9..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/deepep_v2_selected_cases.py +++ /dev/null @@ -1,96 +0,0 @@ -#!/usr/bin/env python3 -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -"""Run the matched FP8/BF16 DeepEP V2 direct-EP cases. - -This wrapper targets the pinned synthetic DeepEP V2 revision documented in -RESULTS-b200.md. It limits the upstream elastic benchmark to the 2 dispatch -dtypes used by the comparison while retaining correctness checks. -""" - -import argparse -import sys -from argparse import Namespace -from pathlib import Path - -import torch - - -def _load_upstream(): - candidates = ( - Path("/opt/amazon/deepep-v2/tests/elastic"), - Path("/opt/amazon/deepep/tests/elastic"), - ) - for candidate in candidates: - if (candidate / "test_ep.py").is_file(): - sys.path.insert(0, str(candidate)) - import test_ep # pylint: disable=import-outside-toplevel - - return test_ep - locations = ", ".join(str(path) for path in candidates) - raise RuntimeError(f"DeepEP V2 test_ep.py not found under: {locations}") - - -UPSTREAM = _load_upstream() - - -def selected_modes(): - """Yield FP8 and BF16 dispatch with the same remaining mode controls.""" - # handle copy, expert alignment, FP8 dispatch, bias count, previous event, - # async compute stream, allocate on communication stream - yield (1, 128, 1, 0, 0, 0, 0) - yield (1, 128, 0, 0, 0, 0, 0) - - -UPSTREAM.enumerate_ep_modes = selected_modes - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--num-processes", type=int, default=8) - parser.add_argument("--num-tokens", type=int, required=True) - parser.add_argument("--hidden", type=int, default=7168) - parser.add_argument("--num-topk", type=int, default=8) - parser.add_argument("--num-experts", type=int, default=256) - parser.add_argument("--seed", type=int, default=0) - args = parser.parse_args() - - selected = Namespace( - num_processes=args.num_processes, - num_sms=0, - num_qps=0, - num_allocated_qps=0, - num_gpu_timeout_secs=180, - num_cpu_timeout_secs=180, - sl_idx=0, - num_tokens=args.num_tokens, - hidden=args.hidden, - num_topk=args.num_topk, - num_experts=args.num_experts, - do_cpu_sync=1, - allow_hybrid_mode=1, - allow_multiple_reduction=1, - prefer_overlap_with_compute=0, - deterministic=False, - seed=args.seed, - skip_check=False, - skip_perf_test=False, - do_pressure_test=False, - pressure_iterations=0, - reuse_elastic_buffer=False, - test_first_only=False, - unbalanced_ratio=1.0, - precise_unbalanced_ratio=False, - masked_ratio=0.0, - dump_profile_traces="", - ignore_local_traffic=True, - ) - torch.multiprocessing.spawn( - UPSTREAM.test_loop, - args=(args.num_processes, selected), - nprocs=args.num_processes, - ) - - -if __name__ == "__main__": - main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example b/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example deleted file mode 100644 index 1259a2446..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example +++ /dev/null @@ -1,27 +0,0 @@ -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 - -# Shared topology for the EP-backend comparison. Copy to env_vars (gitignored), -# edit, then source it before applying the per-backend manifests. -# cp env_vars.example env_vars && source env_vars - -# GPU node instance type and per-node device counts. The defaults retain the -# historical B300 run. For B200 use p6-b200.48xlarge and 8 EFA devices per node. -export INSTANCE_TYPE=p6-b300.48xlarge -export GPU_PER_NODE=8 -export EFA_PER_NODE=16 -# B200 direct EP16 uses 2 nodes and EP32 uses 4 nodes. The historical B300/H100 -# primary comparison uses 8 nodes. The 32-node findings apply only to the -# historical V1-era matrix; see README "Scaling beyond 8 nodes". -export NUM_NODES=8 -export NP=$((NUM_NODES * GPU_PER_NODE)) # ranks; used by the NCCL baseline (64 at 8 nodes) - -# Per-backend image URIs. -# NVSHMEM: built from ../deepep-benchmark/deepep.Dockerfile (CUDA 13, sm_90+sm_100). -# UCCL: built from ../uccl-ep-benchmark/uccl-ep.Dockerfile (CUDA 13; Hopper + Blackwell via PTX). -# NCCL: the DeepEP image already ships /opt/nccl-tests/build/alltoall_perf built with -# sm_100 gencode, so reuse it for the baseline -- no separate nccl-tests build. -export NVSHMEM_IMAGE_URI=.dkr.ecr..amazonaws.com/deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13 -export UCCL_IMAGE_URI=.dkr.ecr..amazonaws.com/uccl-ep:efa1.48.0-uccl0dc87eb-cu13 -export DEEPEP_V2_IMAGE_URI=.dkr.ecr..amazonaws.com/deepep-v2@sha256: -export NCCL_IMAGE_URI=$NVSHMEM_IMAGE_URI diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml b/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml deleted file mode 100644 index 7f14fd384..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml +++ /dev/null @@ -1,106 +0,0 @@ -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -# -# Baseline for the EP-backend comparison: a raw NCCL all-to-all over the same -# world size as the EP benchmarks (NUM_NODES x GPU_PER_NODE ranks). This is the -# transport-level reference ceiling -- it moves bytes but does NOT do MoE token -# routing or combine-reduction, so its busbw is an upper bound, not a -# like-for-like dispatch/combine number. -# -# Built from micro-benchmarks/nccl-tests (CUDA 13.0.2 image ships alltoall_perf -# and sm_100/sm_103). Set NP = NUM_NODES * GPU_PER_NODE in env_vars. -apiVersion: kubeflow.org/v2beta1 -kind: MPIJob -metadata: - name: nccl-alltoall -spec: - runPolicy: - cleanPodPolicy: Running - backoffLimit: 3 - slotsPerWorker: ${GPU_PER_NODE} - mpiReplicaSpecs: - Launcher: - replicas: 1 - template: - spec: - restartPolicy: OnFailure - nodeSelector: - node.kubernetes.io/instance-type: ${INSTANCE_TYPE} - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - - key: workload - value: bench - operator: Equal - effect: NoSchedule - - key: capacity-reservation - operator: Exists - effect: NoSchedule - containers: - - image: ${IMAGE_URI} - imagePullPolicy: IfNotPresent - name: nccl-alltoall-launcher - # PATH / LD_LIBRARY_PATH come from the image ENV and are forwarded to - # workers via `mpirun -x PATH -x LD_LIBRARY_PATH` below (Kubernetes does - # not expand a bare $PATH in an env value, so we do not set one here). - command: - - /bin/bash - - -c - - | - # Wait for each worker's sshd before launching: mpi-operator starts - # the launcher as soon as workers are Running, but their sshd / DNS - # may not be ready yet, and the launcher would lose the SSH race. - for host in $(awk '{print $1}' /etc/mpi/hostfile); do - until ssh -o ConnectTimeout=3 -o BatchMode=yes -o StrictHostKeyChecking=no "$host" true 2>/dev/null; do - echo "waiting for sshd on $host"; sleep 3 - done - done - /opt/amazon/openmpi/bin/mpirun \ - --allow-run-as-root --tag-output \ - -np ${NP} -N ${GPU_PER_NODE} --bind-to none \ - -x PATH -x LD_LIBRARY_PATH \ - -x FI_PROVIDER=efa \ - -x FI_EFA_USE_DEVICE_RDMA=1 \ - -x FI_EFA_FORK_SAFE=1 \ - -x NCCL_DEBUG=INFO \ - /opt/nccl-tests/build/alltoall_perf -b 8 -e 8G -f 2 -g 1 -c 1 -n 100 - Worker: - replicas: ${NUM_NODES} - template: - spec: - nodeSelector: - node.kubernetes.io/instance-type: ${INSTANCE_TYPE} - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - - key: workload - value: bench - operator: Equal - effect: NoSchedule - - key: capacity-reservation - operator: Exists - effect: NoSchedule - containers: - - image: ${IMAGE_URI} - imagePullPolicy: IfNotPresent - name: nccl-alltoall-worker - volumeMounts: - - name: shmem - mountPath: /dev/shm - resources: - limits: - nvidia.com/gpu: ${GPU_PER_NODE} - hugepages-2Mi: 5120Mi - vpc.amazonaws.com/efa: ${EFA_PER_NODE} - memory: 32000Mi - requests: - nvidia.com/gpu: ${GPU_PER_NODE} - hugepages-2Mi: 5120Mi - vpc.amazonaws.com/efa: ${EFA_PER_NODE} - memory: 32000Mi - volumes: - - name: shmem - hostPath: - path: /dev/shm diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json new file mode 100644 index 000000000..74319eb18 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json @@ -0,0 +1,473 @@ +{ + "bootstrap_samples_dimensionless": 20000, + "campaign_provenance": { + "campaign_id": "fair-ep-b200-20260824t195145z", + "cluster": "ml-clusters-shared-ap-south-1", + "comparison": { + "experts": 256, + "hidden_dimensions": 7168, + "independent_starts": 3, + "measured_iterations": 100, + "tokens_per_rank": 128, + "top_k_dimensionless": 8, + "warmup_iterations": 20 + }, + "created_at_utc": "2026-08-24T21:33:19Z", + "git_commit": "249f9e53113cf7706813b19520ddba2c2983e7d6", + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "region": "ap-south-1" + }, + "cells": [ + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_latency_ms": [ + 1.0167359709739685, + 1.0202720165252686 + ], + "median_aggregate_input_tokens_per_second": 2011977.5863053314, + "median_effective_logical_gigabytes_per_second_per_rank": 21.85812449762112, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.92906224881056, + "median_latency_ms": 1.017903983592987, + "per_start_median_latency_ms": [ + 1.017903983592987, + 1.0167359709739685, + 1.0202720165252686 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.17692560833015608, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.6179679930210114, + 0.6375360190868378 + ], + "median_aggregate_input_tokens_per_second": 3293197.584829476, + "median_effective_logical_gigabytes_per_second_per_rank": 35.777298561587436, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 17.888649280793718, + "median_latency_ms": 0.6218879818916321, + "per_start_median_latency_ms": [ + 0.6375360190868378, + 0.6179679930210114, + 0.6218879818916321 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.6544046759560789, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.5190079808235168, + 0.5276960134506226 + ], + "median_aggregate_input_tokens_per_second": 3928790.5148029686, + "median_effective_logical_gigabytes_per_second_per_rank": 42.68238015281945, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.341190076409724, + "median_latency_ms": 0.5212800204753876, + "per_start_median_latency_ms": [ + 0.5190079808235168, + 0.5276960134506226, + 0.5212800204753876 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.8620724389950091, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_reduction_percent": [ + 37.36776460619894, + 39.22040621528938 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": 39.04684517276182, + "paired_latency_reduction_percent_per_start": [ + 37.36776460619894, + 39.22040621528938, + 39.04684517276182 + ] + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_reduction_percent": [ + -22.837421126983617, + -17.10681477013578 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": -19.300176002236547, + "paired_latency_reduction_percent_per_start": [ + -22.837421126983617, + -17.10681477013578, + -19.300176002236547 + ] + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_latency_ms": [ + 1.0250880122184753, + 1.0304319858551025 + ], + "median_aggregate_input_tokens_per_second": 1988936.5642491602, + "median_effective_logical_gigabytes_per_second_per_rank": 28.51339458507596, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.25669729253798, + "median_latency_ms": 1.029695987701416, + "per_start_median_latency_ms": [ + 1.0304319858551025, + 1.0250880122184753, + 1.029695987701416 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.2816354149488261, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.48263999819755554, + 0.504287987947464 + ], + "median_aggregate_input_tokens_per_second": 4222332.229970289, + "median_effective_logical_gigabytes_per_second_per_rank": 60.53135484885407, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 30.265677424427036, + "median_latency_ms": 0.48503999412059784, + "per_start_median_latency_ms": [ + 0.504287987947464, + 0.48503999412059784, + 0.48263999819755554 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 2.418493715218177, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.5917760133743286, + 0.5978399813175201 + ], + "median_aggregate_input_tokens_per_second": 3459365.896020597, + "median_effective_logical_gigabytes_per_second_per_rank": 49.59346948535127, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 24.796734742675635, + "median_latency_ms": 0.5920160114765167, + "per_start_median_latency_ms": [ + 0.5920160114765167, + 0.5978399813175201, + 0.5917760133743286 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.5782085314350778, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_reduction_percent": [ + 51.06052656847785, + 53.12791309647133 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": 52.6830878579017, + "paired_latency_reduction_percent_per_start": [ + 51.06052656847785, + 52.6830878579017, + 53.12791309647133 + ] + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_reduction_percent": [ + 14.818522105551635, + 18.86792297636796 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": 18.442115379850478, + "paired_latency_reduction_percent_per_start": [ + 14.818522105551635, + 18.86792297636796, + 18.442115379850478 + ] + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_latency_ms": [ + 1.55131196975708, + 1.5627520084381104 + ], + "median_aggregate_input_tokens_per_second": 2635371.6483700224, + "median_effective_logical_gigabytes_per_second_per_rank": 14.315338793945964, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.736504095459473, + "median_latency_ms": 1.5542399883270264, + "per_start_median_latency_ms": [ + 1.5542399883270264, + 1.5627520084381104, + 1.55131196975708 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.3819040831100996, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.9216639995574951, + 0.9674720168113708 + ], + "median_aggregate_input_tokens_per_second": 4294725.449273138, + "median_effective_logical_gigabytes_per_second_per_rank": 23.32894864045168, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 17.496711480338764, + "median_latency_ms": 0.9537280201911926, + "per_start_median_latency_ms": [ + 0.9537280201911926, + 0.9216639995574951, + 0.9674720168113708 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 2.4805943961903028, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.7640160024166107, + 0.7714239954948425 + ], + "median_aggregate_input_tokens_per_second": 5350835.031806838, + "median_effective_logical_gigabytes_per_second_per_rank": 29.065735892774743, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.799301919581055, + "median_latency_ms": 0.7654879987239838, + "per_start_median_latency_ms": [ + 0.7640160024166107, + 0.7654879987239838, + 0.7714239954948425 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.5113275578246953, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_reduction_percent": [ + 37.635238064793164, + 41.02301615477362 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": 38.637016975880336, + "paired_latency_reduction_percent_per_start": [ + 38.637016975880336, + 41.02301615477362, + 37.635238064793164 + ] + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_reduction_percent": [ + -25.413783141496673, + -20.402148837584143 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": -24.830895841777636, + "paired_latency_reduction_percent_per_start": [ + -24.830895841777636, + -20.402148837584143, + -25.413783141496673 + ] + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_latency_ms": [ + 1.552623987197876, + 1.5713760256767273 + ], + "median_aggregate_input_tokens_per_second": 2627446.5313618258, + "median_effective_logical_gigabytes_per_second_per_rank": 18.833536736801566, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.125152552601175, + "median_latency_ms": 1.5589280128479004, + "per_start_median_latency_ms": [ + 1.5589280128479004, + 1.5713760256767273, + 1.552623987197876 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.6113033112150486, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.9481759965419769, + 0.9605599939823151 + ], + "median_aggregate_input_tokens_per_second": 4307661.165769053, + "median_effective_logical_gigabytes_per_second_per_rank": 30.87731523623257, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 23.15798642717443, + "median_latency_ms": 0.9508640170097351, + "per_start_median_latency_ms": [ + 0.9481759965419769, + 0.9605599939823151, + 0.9508640170097351 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.683392543166502, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_latency_ms": [ + 0.868256002664566, + 0.8695679903030396 + ], + "median_aggregate_input_tokens_per_second": 4715590.93391919, + "median_effective_logical_gigabytes_per_second_per_rank": 33.80135581433276, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 25.351016860749567, + "median_latency_ms": 0.8686079978942871, + "per_start_median_latency_ms": [ + 0.8686079978942871, + 0.868256002664566, + 0.8695679903030396 + ], + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.07816060809911046, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_reduction_percent": [ + 38.75761131799704, + 39.177692059698245 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": 38.87141089806043, + "paired_latency_reduction_percent_per_start": [ + 39.177692059698245, + 38.87141089806043, + 38.75761131799704 + ] + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_reduction_percent": [ + -10.630964949793603, + -9.16040363899269 + ], + "direction_supported": true, + "median_paired_latency_reduction_percent": -9.349013258683126, + "paired_latency_reduction_percent_per_start": [ + -9.16040363899269, + -10.630964949793603, + -9.349013258683126 + ] + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "world_size_ranks": 32 + } + ], + "comparison_scope": "synthetic decode dispatch-plus-combine communication workload; not end-to-end training or serving", + "configuration": { + "dispatch_dtypes": [ + "fp8", + "bf16" + ], + "experts_dimensionless": 256, + "hidden_dimensions": 7168, + "measured_iterations_dimensionless": 100, + "tokens_per_rank": 128, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": 20, + "world_sizes_ranks": [ + 16, + 32 + ] + }, + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "independent_starts_per_cell": 3, + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "maximum_run_to_run_cv_percent_for_direction_support": 5.0, + "runtime": { + "cuda_version": "13.0", + "gpu": "NVIDIA B200", + "nccl_version": [ + 2, + 29, + 7 + ], + "torch_version": "2.13.0+cu130" + }, + "schema_version_dimensionless": 1, + "scored_result_records_dimensionless": 36, + "status": "PASS", + "timing_boundary": "BF16 input ready through dispatch and combine completion; slowest rank CUDA elapsed time" +} diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh index 53d021475..3d5c0c740 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh @@ -3,9 +3,9 @@ set -euo pipefail : "${CAMPAIGN_ID:?Set a unique CAMPAIGN_ID}" : "${FAIR_EP_NODES:?Set 4 comma-separated B200 node names}" -: "${PROTECTED_NODES_CSV:?Set the concurrent campaign protected node names}" +: "${PROTECTED_NODES_CSV:=}" : "${ARTIFACT_ROOT:?Set the durable artifact directory}" -: "${KUBECTL_CONTEXT:=aps1}" +: "${KUBECTL_CONTEXT:?Set the target kubectl context explicitly}" : "${CAMPAIGN_NAMESPACE:=${CAMPAIGN_ID}}" : "${SHARED_LOCK_NAME:=adai-ap-south-1-gpu-campaign-lock}" : "${SHARED_LOCK_NAMESPACE:=default}" @@ -24,22 +24,49 @@ set -euo pipefail [[ "${CAMPAIGN_ID}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] [[ "${CAMPAIGN_NAMESPACE}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] [[ "${LOCK_MODE}" == exclusive || "${LOCK_MODE}" == observe ]] +for value in "${LOCK_DURATION_SECONDS}" "${INDEPENDENT_STARTS}" \ + "${WARMUP_ITERATIONS}" "${MEASURED_ITERATIONS}" \ + "${CASE_TIMEOUT_SECONDS}" "${EFA_PER_NODE}"; do + [[ "${value}" =~ ^[1-9][0-9]*$ ]] || { + printf 'Iteration, timeout, Lease, and EFA values must be positive integers\n' >&2 + exit 2 + } +done if [[ "${LOCK_MODE}" == observe && -z "${EXPECTED_LOCK_HOLDER}" ]]; then printf 'LOCK_MODE=observe requires EXPECTED_LOCK_HOLDER\n' >&2 exit 2 fi +if [[ "${LOCK_MODE}" == observe && -z "${PROTECTED_NODES_CSV}" ]]; then + printf 'LOCK_MODE=observe requires PROTECTED_NODES_CSV\n' >&2 + exit 2 +fi [[ "${INDEPENDENT_STARTS}" -eq 3 ]] || { printf 'This scored matrix requires exactly 3 independent starts\n' >&2 exit 2 } +[[ "${WARMUP_ITERATIONS}" -eq 20 && "${MEASURED_ITERATIONS}" -eq 100 ]] || { + printf 'This scored matrix requires 20 warmup and 100 measured iterations\n' >&2 + exit 2 +} case_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [[ -e "${ARTIFACT_ROOT}" && ! -d "${ARTIFACT_ROOT}" ]]; then + printf 'ARTIFACT_ROOT exists and is not a directory: %s\n' "${ARTIFACT_ROOT}" >&2 + exit 2 +fi +if [[ -d "${ARTIFACT_ROOT}" && -n "$(find "${ARTIFACT_ROOT}" -mindepth 1 -print -quit)" ]]; then + printf 'Refusing to reuse nonempty ARTIFACT_ROOT: %s\n' "${ARTIFACT_ROOT}" >&2 + exit 2 +fi mkdir -p "${ARTIFACT_ROOT}/control" "${ARTIFACT_ROOT}/runs" \ "${ARTIFACT_ROOT}/summary" "${ARTIFACT_ROOT}/teardown" K=(kubectl --context "${KUBECTL_CONTEXT}") IFS=, read -r -a selected_nodes <<<"${FAIR_EP_NODES}" -IFS=, read -r -a protected_nodes <<<"${PROTECTED_NODES_CSV}" +protected_nodes=() +if [[ -n "${PROTECTED_NODES_CSV}" ]]; then + IFS=, read -r -a protected_nodes <<<"${PROTECTED_NODES_CSV}" +fi ((${#selected_nodes[@]} == 4)) || { printf 'FAIR_EP_NODES must contain exactly 4 nodes\n' >&2 exit 2 @@ -62,6 +89,13 @@ declare -A images=( [deepep-v1-nvshmem]="${DEEPEP_V1_IMAGE}" [deepep-v2-gin-gda]="${DEEPEP_V2_IMAGE}" ) +for arm in "${!images[@]}"; do + [[ "${images[${arm}]}" =~ @sha256:[0-9a-f]{64}$ ]] || { + printf 'Image for %s is not pinned by digest: %s\n' \ + "${arm}" "${images[${arm}]}" >&2 + exit 2 + } +done current_case="" namespace_created=0 @@ -77,7 +111,7 @@ check_shared_lock() { "${CAMPAIGN_ID}" "${holder}" >&2 return 1 } - elif [[ -n "${holder}" && "${holder}" != "${EXPECTED_LOCK_HOLDER}" ]]; then + elif [[ "${holder}" != "${EXPECTED_LOCK_HOLDER}" ]]; then printf 'Shared Lease holder changed from protected campaign %s to %s\n' \ "${EXPECTED_LOCK_HOLDER}" "${holder}" >&2 return 1 @@ -149,8 +183,11 @@ gpu_requests_on_node() { "${K[@]}" get pods -A --field-selector "spec.nodeName=${node}" -o json | jq ' [.items[] | select(.status.phase != "Succeeded" and .status.phase != "Failed") - | [.spec.containers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] - | add // 0] + | ([.spec.containers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] + | add // 0) as $app + | ([.spec.initContainers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] + | max // 0) as $init + | [$app, $init] | max] | add // 0' } @@ -186,7 +223,7 @@ cleanup_case() { } finish() { - local command_status=$? teardown_status=0 owned="" remaining=0 + local command_status=$? teardown_status=0 owned="" remaining=0 namespace_remaining=0 trap - EXIT INT TERM set +e cleanup_case || teardown_status=1 @@ -203,21 +240,37 @@ finish() { teardown_status=1 fi fi + if "${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" >/dev/null 2>&1; then + namespace_remaining=1 + teardown_status=1 + fi + if ((namespace_created == 1 || lock_claimed == 1)); then + check_shared_lock || teardown_status=1 + fi release_shared_lock || teardown_status=1 "${K[@]}" get all -A -l "adai.aws/campaign=${CAMPAIGN_ID}" -o json \ >"${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>&1 || teardown_status=1 remaining="$(jq '.items | length' "${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>/dev/null || printf '1')" [[ "${remaining}" -eq 0 ]] || teardown_status=1 "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ - >"${ARTIFACT_ROOT}/teardown/shared-lease-untouched.json" 2>&1 || true - find "${ARTIFACT_ROOT}" -type f ! -name SHA256SUMS -print0 | sort -z | \ - xargs -0 sha256sum >"${ARTIFACT_ROOT}/SHA256SUMS" + >"${ARTIFACT_ROOT}/teardown/shared-lease-after.json" 2>&1 || true + "${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-after.json" 2>&1 || \ + teardown_status=1 + "${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-after.json" 2>&1 || \ + teardown_status=1 if ((command_status == 0 && teardown_status == 0)); then - printf 'PASS\n' >"${ARTIFACT_ROOT}/STATUS" + printf 'PASS teardown_verified=1_dimensionless remaining_resources=0_resources\n' \ + >"${ARTIFACT_ROOT}/STATUS" + printf 'PASS completed_at=%s\n' "$(date -u +%FT%TZ)" \ + >"${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" else - printf 'FAIL command_status=%s_dimensionless teardown_status=%s_dimensionless remaining_resources=%s_resources\n' \ - "${command_status}" "${teardown_status}" "${remaining}" >"${ARTIFACT_ROOT}/STATUS" + printf 'FAIL command_status=%s_dimensionless teardown_status=%s_dimensionless remaining_resources=%s_resources namespace_remaining=%s_namespaces\n' \ + "${command_status}" "${teardown_status}" "${remaining}" \ + "${namespace_remaining}" >"${ARTIFACT_ROOT}/STATUS" + rm -f "${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" fi + find "${ARTIFACT_ROOT}" -type f ! -name SHA256SUMS -print0 | sort -z | \ + xargs -0 sha256sum >"${ARTIFACT_ROOT}/SHA256SUMS" if ((command_status == 0 && teardown_status != 0)); then command_status=1 fi @@ -232,11 +285,19 @@ trap 'exit 143' TERM "${K[@]}" get namespaces -o json >"${ARTIFACT_ROOT}/control/namespaces-before.json" "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ >"${ARTIFACT_ROOT}/control/shared-lease-before.json" +if "${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" >/dev/null 2>&1; then + printf 'Refusing to reuse existing namespace: %s\n' "${CAMPAIGN_NAMESPACE}" >&2 + exit 1 +fi claim_shared_lock check_shared_lock aws sts get-caller-identity --output json >"${ARTIFACT_ROOT}/control/aws-caller-identity.json" printf '%s\n' "${selected_nodes[@]}" >"${ARTIFACT_ROOT}/control/selected-nodes.txt" -printf '%s\n' "${protected_nodes[@]}" | sort -u >"${ARTIFACT_ROOT}/control/protected-nodes.txt" +: >"${ARTIFACT_ROOT}/control/protected-nodes.txt" +if ((${#protected_nodes[@]} > 0)); then + printf '%s\n' "${protected_nodes[@]}" | sort -u \ + >"${ARTIFACT_ROOT}/control/protected-nodes.txt" +fi for node in "${selected_nodes[@]}"; do verify_node_free "${node}" @@ -246,7 +307,7 @@ done jq --arg campaign "${CAMPAIGN_ID}" ' .metadata.labels["adai.aws/campaign"]=$campaign | .metadata.labels["adai.aws/owner"]="fair-ep-comparison"' | \ - "${K[@]}" apply -f - >/dev/null + "${K[@]}" create -f - >/dev/null namespace_created=1 # Read the live host mitigation before invoking DeepEP V2. The EFA 3.3.0g @@ -508,11 +569,13 @@ for world_size in 16 32; do done done +check_shared_lock +for node in "${selected_nodes[@]}"; do + verify_node_free "${node}" +done python3 "${case_dir}/summarize_fair_results.py" "${ARTIFACT_ROOT}/runs" \ --starts="${INDEPENDENT_STARTS}" \ + --provenance="${ARTIFACT_ROOT}/control/provenance.json" \ --json="${ARTIFACT_ROOT}/summary/summary.json" \ --markdown="${ARTIFACT_ROOT}/summary/summary.md" -"${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-after.json" -"${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-after.json" -touch "${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" printf 'PASS fair EP comparison completed at %s UTC\n' "$(date -u +%FT%TZ)" diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py index 891cfbb0d..9648d6cbf 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py @@ -5,7 +5,9 @@ import argparse import json +import math import random +import re import statistics from collections import defaultdict from pathlib import Path @@ -17,6 +19,25 @@ ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") WORLD_SIZES = (16, 32) DTYPES = ("fp8", "bf16") +ARM_LABELS = { + "uccl": "UCCL", + "deepep-v1-nvshmem": "DeepEP V1 NVSHMEM", + "deepep-v2-gin-gda": "DeepEP V2 NCCL GIN", +} +BOOTSTRAP_SAMPLES = 20_000 +MAX_RUN_TO_RUN_CV_PERCENT = 5.0 +EXPECTED_STARTS = 3 +EXPECTED_WARMUPS = 20 +EXPECTED_ITERATIONS = 100 +TIMING_BOUNDARY = ( + "BF16 input ready through dispatch and combine completion; " + "slowest rank CUDA elapsed time" +) +LOGICAL_PAYLOAD_DEFINITION = ( + "per valid expert assignment: dispatch tensor plus FP8 scales when selected " + "plus BF16 combine tensor; backend metadata excluded" +) +SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") def percentile(values: list[float], quantile: float) -> float: @@ -31,7 +52,7 @@ def percentile(values: list[float], quantile: float) -> float: def bootstrap_median_ci( - values: list[float], seed: int, samples: int = 20_000 + values: list[float], seed: int, samples: int = BOOTSTRAP_SAMPLES ) -> tuple[float, float]: rng = random.Random(seed) medians = [ @@ -61,6 +82,8 @@ def load_results(root: Path) -> list[dict[str, Any]]: def validate(results: list[dict[str, Any]], starts: int) -> None: + if starts != EXPECTED_STARTS: + raise ValueError(f"scored matrix requires exactly {EXPECTED_STARTS} starts") expected = { (arm, world, run, dtype) for arm in ARMS @@ -81,17 +104,153 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: missing = expected - observed extra = observed - expected if missing or extra: - raise ValueError(f"result matrix mismatch; missing={sorted(missing)}, extra={sorted(extra)}") + raise ValueError( + f"result matrix mismatch; missing={sorted(missing)}, extra={sorted(extra)}" + ) measured = [result for result in results if result["run_index_dimensionless"] > 0] + if len(measured) != len(expected): + raise ValueError( + f"result matrix has {len(measured)} scored records; expected {len(expected)}" + ) + expected_shape = { + "benchmark": "common-boundary-dispatch-combine", + "tokens_per_rank": 128, + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "gpus_per_node": 8, + "warmup_iterations": EXPECTED_WARMUPS, + "measured_iterations": EXPECTED_ITERATIONS, + "timing_boundary": TIMING_BOUNDARY, + "logical_payload_definition": LOGICAL_PAYLOAD_DEFINITION, + } for result in measured: + for field, expected_value in expected_shape.items(): + if result.get(field) != expected_value: + raise ValueError( + f"unexpected {field}: expected {expected_value}, got {result.get(field)}" + ) + world_size = result["world_size_ranks"] + if result.get("nodes") != world_size // 8: + raise ValueError(f"invalid node count for EP{world_size}: {result}") + if result.get("global_input_tokens") != 128 * world_size: + raise ValueError(f"invalid global input token count: {result}") if result["correctness"]["status"] != "PASS": raise ValueError(f"correctness did not pass: {result}") - if "@sha256:" not in result["runtime"]["image_reference"]: - raise ValueError(f"image is not digest pinned: {result['runtime']['image_reference']}") + tolerance = 9e-4 if result["dispatch_dtype"] == "fp8" else 1e-5 + correctness = result["correctness"] + if correctness.get("tolerance_dimensionless") != tolerance: + raise ValueError(f"unexpected correctness tolerance: {result}") + if not math.isfinite( + correctness.get("normalized_diff_dimensionless", math.inf) + ): + raise ValueError(f"non-finite correctness result: {result}") + if correctness["normalized_diff_dimensionless"] > tolerance: + raise ValueError(f"correctness exceeds tolerance: {result}") + image_reference = result["runtime"]["image_reference"] + digest = image_reference.rsplit("@sha256:", 1)[-1] + if "@sha256:" not in image_reference or not SHA256_PATTERN.fullmatch(digest): + raise ValueError( + f"image is not digest pinned: {result['runtime']['image_reference']}" + ) + for field in ("gpu", "torch_version", "cuda_version", "nccl_version"): + if not result["runtime"].get(field): + raise ValueError(f"runtime is missing {field}: {result}") + for field in ("route_hash_sha256", "input_hash_sha256"): + if not SHA256_PATTERN.fullmatch(result.get(field, "")): + raise ValueError(f"invalid {field}: {result}") + expected_selections = result["global_input_tokens"] * 8 + if result.get("global_valid_expert_selections") != expected_selections: + raise ValueError(f"invalid valid-expert selection count: {result}") + dispatch_bytes = 7_168 * 2 + if result["dispatch_dtype"] == "fp8": + dispatch_bytes = 7_168 + math.ceil(7_168 / 128) * 4 + expected_logical_bytes = 128 * 8 * (dispatch_bytes + 7_168 * 2) + logical_bytes = result.get("avg_logical_payload_bytes_per_rank") + scaleout_bytes = result.get("avg_scaleout_logical_payload_bytes_per_rank") + if logical_bytes != expected_logical_bytes: + raise ValueError(f"invalid logical payload: {result}") + if not isinstance(scaleout_bytes, (int, float)) or not ( + 0 < scaleout_bytes <= logical_bytes + ): + raise ValueError(f"invalid scale-out logical payload: {result}") + latency = result.get("latency_ms", {}).get("median", 0) + if ( + not isinstance(latency, (int, float)) + or not math.isfinite(latency) + or latency <= 0 + ): + raise ValueError(f"invalid median latency: {result}") + elapsed_seconds = latency / 1e3 + expected_metrics = { + "aggregate_input_tokens_per_second": result["global_input_tokens"] + / elapsed_seconds, + "effective_logical_gigabytes_per_second_per_rank": logical_bytes + / elapsed_seconds + / 1e9, + "effective_scaleout_logical_gigabytes_per_second_per_rank": scaleout_bytes + / elapsed_seconds + / 1e9, + } + for field in ( + "aggregate_input_tokens_per_second", + "effective_logical_gigabytes_per_second_per_rank", + "effective_scaleout_logical_gigabytes_per_second_per_rank", + ): + value = result.get(field, 0) + if ( + not isinstance(value, (int, float)) + or not math.isfinite(value) + or value <= 0 + ): + raise ValueError(f"invalid positive metric {field}: {result}") + if not math.isclose( + value, expected_metrics[field], rel_tol=1e-12, abs_tol=1e-9 + ): + raise ValueError( + f"metric {field} does not match common accounting: {result}" + ) + + common_fields = ( + "warmup_iterations", + "measured_iterations", + "timing_boundary", + "logical_payload_definition", + ) + for field in common_fields: + values = {json.dumps(result[field], sort_keys=True) for result in measured} + if len(values) != 1: + raise ValueError(f"scored results disagree on {field}: {sorted(values)}") + + runtime_signatures = { + json.dumps( + { + key: result["runtime"].get(key) + for key in ("gpu", "torch_version", "cuda_version", "nccl_version") + }, + sort_keys=True, + ) + for result in measured + } + if len(runtime_signatures) != 1: + raise ValueError(f"runtime stack mismatch: {sorted(runtime_signatures)}") + + for arm in ARMS: + image_references = { + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + } + if len(image_references) != 1: + raise ValueError( + f"{arm} did not use one immutable image: {image_references}" + ) for world in WORLD_SIZES: - same_world = [result for result in measured if result["world_size_ranks"] == world] + same_world = [ + result for result in measured if result["world_size_ranks"] == world + ] route_hashes = {result["route_hash_sha256"] for result in same_world} input_hashes = {result["input_hash_sha256"] for result in same_world} if len(route_hashes) != 1 or len(input_hashes) != 1: @@ -99,6 +258,55 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: f"EP{world} did not replay one route/input: " f"routes={route_hashes}, inputs={input_hashes}" ) + for dtype in DTYPES: + same_cell = [ + result for result in same_world if result["dispatch_dtype"] == dtype + ] + for field in ( + "avg_logical_payload_bytes_per_rank", + "avg_scaleout_logical_payload_bytes_per_rank", + "global_valid_expert_selections", + ): + values = {result[field] for result in same_cell} + if len(values) != 1: + raise ValueError( + f"EP{world} {dtype} disagrees on {field}: {sorted(values)}" + ) + + +def validate_provenance( + provenance: dict[str, Any], results: list[dict[str, Any]], starts: int +) -> None: + measured = [result for result in results if result["run_index_dimensionless"] > 0] + expected_images = { + arm: next( + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + ) + for arm in ARMS + } + if provenance.get("images") != expected_images: + raise ValueError( + f"provenance images do not match scored results: {provenance.get('images')}" + ) + expected_comparison = { + "tokens_per_rank": 128, + "hidden_dimensions": 7_168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": EXPECTED_WARMUPS, + "measured_iterations": EXPECTED_ITERATIONS, + "independent_starts": starts, + } + if provenance.get("comparison") != expected_comparison: + raise ValueError( + "provenance comparison does not match the scored matrix: " + f"{provenance.get('comparison')}" + ) + for field in ("campaign_id", "created_at_utc", "region", "cluster", "git_commit"): + if not provenance.get(field): + raise ValueError(f"provenance is missing {field}") def arm_summary(results: Iterable[dict[str, Any]], seed: int) -> dict[str, Any]: @@ -136,7 +344,11 @@ def arm_summary(results: Iterable[dict[str, Any]], seed: int) -> dict[str, Any]: } -def summarize(results: list[dict[str, Any]], starts: int) -> dict[str, Any]: +def summarize( + results: list[dict[str, Any]], + starts: int, + provenance: dict[str, Any] | None = None, +) -> dict[str, Any]: measured = [result for result in results if result["run_index_dimensionless"] > 0] by_cell_arm: dict[tuple[int, str, str], list[dict[str, Any]]] = defaultdict(list) for result in measured: @@ -165,17 +377,14 @@ def summarize(results: list[dict[str, Any]], starts: int) -> dict[str, Any]: for result in by_cell_arm[(world, dtype, baseline)] } paired = [ - (baseline_by_run[run] - v2_by_run[run]) - / baseline_by_run[run] - * 100 + (baseline_by_run[run] - v2_by_run[run]) / baseline_by_run[run] * 100 for run in range(1, starts + 1) ] - ci_low, ci_high = bootstrap_median_ci( - paired, 20260900 + world + index - ) + ci_low, ci_high = bootstrap_median_ci(paired, 20260900 + world + index) stable = ( - arms[baseline]["run_to_run_cv_percent"] <= 5 - and arms["deepep-v2-gin-gda"]["run_to_run_cv_percent"] <= 5 + arms[baseline]["run_to_run_cv_percent"] <= MAX_RUN_TO_RUN_CV_PERCENT + and arms["deepep-v2-gin-gda"]["run_to_run_cv_percent"] + <= MAX_RUN_TO_RUN_CV_PERCENT ) comparisons[f"deepep-v2-gin-gda_vs_{baseline}"] = { "paired_latency_reduction_percent_per_start": paired, @@ -183,9 +392,11 @@ def summarize(results: list[dict[str, Any]], starts: int) -> dict[str, Any]: paired ), "bootstrap_95_percent_ci_reduction_percent": [ci_low, ci_high], - "winner_supported": stable and (ci_low > 0 or ci_high < 0), + "direction_supported": stable and (ci_low > 0 or ci_high < 0), } - same_world = [result for result in measured if result["world_size_ranks"] == world] + same_world = [ + result for result in measured if result["world_size_ranks"] == world + ] cells.append( { "world_size_ranks": world, @@ -196,21 +407,52 @@ def summarize(results: list[dict[str, Any]], starts: int) -> dict[str, Any]: "comparisons": comparisons, } ) - return { + images = { + arm: next( + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + ) + for arm in ARMS + } + runtime = { + field: measured[0]["runtime"][field] + for field in ("gpu", "torch_version", "cuda_version", "nccl_version") + } + summary = { "schema_version_dimensionless": 1, "status": "PASS", + "scored_result_records_dimensionless": len(measured), "independent_starts_per_cell": starts, + "bootstrap_samples_dimensionless": BOOTSTRAP_SAMPLES, + "maximum_run_to_run_cv_percent_for_direction_support": MAX_RUN_TO_RUN_CV_PERCENT, "timing_boundary": measured[0]["timing_boundary"], "logical_payload_definition": measured[0]["logical_payload_definition"], + "comparison_scope": "synthetic decode dispatch-plus-combine communication workload; not end-to-end training or serving", + "configuration": { + "world_sizes_ranks": list(WORLD_SIZES), + "dispatch_dtypes": list(DTYPES), + "tokens_per_rank": 128, + "hidden_dimensions": 7_168, + "experts_dimensionless": 256, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": EXPECTED_WARMUPS, + "measured_iterations_dimensionless": EXPECTED_ITERATIONS, + }, + "runtime": runtime, + "images": images, "cells": cells, } + if provenance is not None: + summary["campaign_provenance"] = provenance + return summary def markdown(summary: dict[str, Any]) -> str: lines = [ - "# Fair Common-Boundary EP Results", + "# Common-Boundary EP Results", "", - f"Each cell has {summary['independent_starts_per_cell']} independent process starts. Latency is the slowest-rank CUDA elapsed time from BF16 input readiness through dispatch and combine completion. Values are medians across independent starts.", + f"Each cell has {summary['independent_starts_per_cell']} independent process starts. Latency is the slowest-rank CUDA elapsed time from BF16 input readiness through dispatch and combine completion. Values are medians across independent starts. This is a synthetic communication workload, not an end-to-end training or serving result.", "", "| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) |", "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", @@ -220,10 +462,10 @@ def markdown(summary: dict[str, Any]) -> str: value = cell["arms"][arm] ci = value["bootstrap_95_percent_ci_latency_ms"] lines.append( - f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {arm} | " + f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[arm]} | " f"{value['median_latency_ms']:.4f} ms | [{ci[0]:.4f}, {ci[1]:.4f}] ms | " f"{value['run_to_run_cv_percent']:.2f}% | " - f"{value['median_aggregate_input_tokens_per_second']:.2f} tokens/s | " + f"{value['median_aggregate_input_tokens_per_second']:,.2f} tokens/s | " f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " f"{value['median_effective_scaleout_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank |" ) @@ -232,9 +474,9 @@ def markdown(summary: dict[str, Any]) -> str: "", "## Paired DeepEP V2 latency deltas", "", - "Positive values mean DeepEP V2 had lower latency. A winner is supported only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV.", + "Positive values mean DeepEP V2 had lower latency. A direction is supported for this workload only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV.", "", - "| EP size | Dispatch dtype | Baseline | Median reduction (%) | 95% bootstrap CI (%) | Winner supported |", + "| EP size | Dispatch dtype | Baseline | Median reduction (%) | 95% bootstrap CI (%) | Direction supported |", "|---:|:---:|:---|---:|:---:|:---:|", ] ) @@ -243,10 +485,10 @@ def markdown(summary: dict[str, Any]) -> str: comparison = cell["comparisons"][f"deepep-v2-gin-gda_vs_{baseline}"] ci = comparison["bootstrap_95_percent_ci_reduction_percent"] lines.append( - f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {baseline} | " + f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[baseline]} | " f"{comparison['median_paired_latency_reduction_percent']:.2f}% | " f"[{ci[0]:.2f}, {ci[1]:.2f}]% | " - f"{'yes' if comparison['winner_supported'] else 'no'} |" + f"{'yes' if comparison['direction_supported'] else 'no'} |" ) lines.append("") return "\n".join(lines) @@ -256,12 +498,15 @@ def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("root", type=Path) parser.add_argument("--starts", type=int, default=3) + parser.add_argument("--provenance", type=Path, required=True) parser.add_argument("--json", type=Path, required=True) parser.add_argument("--markdown", type=Path, required=True) args = parser.parse_args() results = load_results(args.root) validate(results, args.starts) - summary = summarize(results, args.starts) + provenance = json.loads(args.provenance.read_text()) + validate_provenance(provenance, results, args.starts) + summary = summarize(results, args.starts, provenance) args.json.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n") args.markdown.write_text(markdown(summary)) print( diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py deleted file mode 100644 index 6cccef07a..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_collect_results.py +++ /dev/null @@ -1,55 +0,0 @@ -#!/usr/bin/env python3 -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -"""Unit tests for the EP comparison log collector.""" - -import importlib.util -import tempfile -import unittest -from pathlib import Path - - -MODULE_PATH = Path(__file__).with_name("collect_results.py") -SPEC = importlib.util.spec_from_file_location("collect_results", MODULE_PATH) -COLLECT_RESULTS = importlib.util.module_from_spec(SPEC) -SPEC.loader.exec_module(COLLECT_RESULTS) - - -class ParseDeepEPV2Test(unittest.TestCase): - def test_parses_rank_zero_selected_cases(self): - log = """ -> Testing with do_handle_copy=1, use_fp8_dispatch=1, num_bias=0 ... - * EP: 1/16 | dispatch: 34 GB/s (SO), 201 GB/s (SU), 866.000 us, 1 bytes - * EP: 0/16 | dispatch: 35 GB/s (SO), 203 GB/s (SU), 864.688 us, 2 bytes - @ EP: 0/16 | combine: 36 GB/s (SO), 208 GB/s (SU), 1614.000 us, 3 bytes - - EP: 0/16 | expanded dispatch: 99 GB/s (SO), 999 GB/s (SU), 1.000 us, 4 bytes -> Testing with do_handle_copy=1, use_fp8_dispatch=0, num_bias=0 ... - * EP: 0/16 | dispatch: 39 GB/s (SO), 225 GB/s (SU), 1501.000 us, 5 bytes - @ EP: 0/16 | combine: 36 GB/s (SO), 208 GB/s (SU), 1615.000 us, 6 bytes -""" - with tempfile.NamedTemporaryFile("w", encoding="utf-8") as handle: - handle.write(log) - handle.flush() - result = COLLECT_RESULTS.parse_deepep_v2(handle.name) - - self.assertEqual(result[("FP8", "dispatch")], (35.0, 203.0, 864.688)) - self.assertEqual(result[("FP8", "combine")], (36.0, 208.0, 1614.0)) - self.assertEqual(result[("BF16", "dispatch")], (39.0, 225.0, 1501.0)) - self.assertEqual(result[("BF16", "combine")], (36.0, 208.0, 1615.0)) - self.assertEqual(len(result), 4) - - def test_retains_result_without_dtype_marker(self): - log = "* EP: 0/32 | dispatch: 10 GB/s (SO), 19 GB/s (SU), 275.659 us, 1 bytes\n" - with tempfile.NamedTemporaryFile("w", encoding="utf-8") as handle: - handle.write(log) - handle.flush() - result = COLLECT_RESULTS.parse_deepep_v2(handle.name) - - self.assertEqual( - result[("unspecified", "dispatch")], - (10.0, 19.0, 275.659), - ) - - -if __name__ == "__main__": - unittest.main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py index 9c9b88f3c..93d5bbb48 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py @@ -19,22 +19,54 @@ def fake_result(arm, world_size, run_index, dtype, latency_ms): "deepep-v1-nvshmem": "b", "deepep-v2-gin-gda": "c", }[arm] - return { + logical_bytes = 22_249_472 if dtype == "fp8" else 29_360_128 + scaleout_bytes = logical_bytes * (world_size - 8) // world_size + tolerance = 9e-4 if dtype == "fp8" else 1e-5 + global_input_tokens = 128 * world_size + result = { + "benchmark": "common-boundary-dispatch-combine", "arm": arm, "world_size_ranks": world_size, + "nodes": world_size // 8, + "gpus_per_node": 8, "run_index_dimensionless": run_index, "dispatch_dtype": dtype, - "route_hash_sha256": f"route-{world_size}", - "input_hash_sha256": f"input-{world_size}", - "correctness": {"status": "PASS"}, - "runtime": {"image_reference": f"example.invalid/{arm}@sha256:{digest * 64}"}, + "tokens_per_rank": 128, + "global_input_tokens": global_input_tokens, + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": 20, + "measured_iterations": 100, + "route_hash_sha256": ("d" if world_size == 16 else "e") * 64, + "input_hash_sha256": ("f" if world_size == 16 else "0") * 64, + "global_valid_expert_selections": global_input_tokens * 8, + "avg_logical_payload_bytes_per_rank": logical_bytes, + "avg_scaleout_logical_payload_bytes_per_rank": scaleout_bytes, + "correctness": { + "status": "PASS", + "tolerance_dimensionless": tolerance, + "normalized_diff_dimensionless": 0.0, + }, + "runtime": { + "image_reference": f"example.invalid/{arm}@sha256:{digest * 64}", + "gpu": "NVIDIA B200", + "torch_version": "2.13.0+cu130", + "cuda_version": "13.0", + "nccl_version": [2, 29, 7], + }, "latency_ms": {"median": latency_ms}, - "aggregate_input_tokens_per_second": 1000 / latency_ms, - "effective_logical_gigabytes_per_second_per_rank": 10 / latency_ms, - "effective_scaleout_logical_gigabytes_per_second_per_rank": 5 / latency_ms, - "timing_boundary": "common boundary", - "logical_payload_definition": "common payload", + "aggregate_input_tokens_per_second": global_input_tokens / (latency_ms / 1e3), + "effective_logical_gigabytes_per_second_per_rank": logical_bytes + / (latency_ms / 1e3) + / 1e9, + "effective_scaleout_logical_gigabytes_per_second_per_rank": scaleout_bytes + / (latency_ms / 1e3) + / 1e9, + "timing_boundary": summary_module.TIMING_BOUNDARY, + "logical_payload_definition": summary_module.LOGICAL_PAYLOAD_DEFINITION, } + return result class SummarizeFairResultsTest(unittest.TestCase): @@ -63,13 +95,11 @@ def test_valid_matrix_and_paired_delta(self): summary = summary_module.summarize(self.results, 3) self.assertEqual(summary["status"], "PASS") self.assertEqual(len(summary["cells"]), 4) - comparison = summary["cells"][0]["comparisons"][ - "deepep-v2-gin-gda_vs_uccl" - ] + comparison = summary["cells"][0]["comparisons"]["deepep-v2-gin-gda_vs_uccl"] self.assertAlmostEqual( comparison["median_paired_latency_reduction_percent"], 20.0 ) - self.assertTrue(comparison["winner_supported"]) + self.assertTrue(comparison["direction_supported"]) def test_missing_start_is_rejected(self): with self.assertRaises(ValueError): @@ -80,6 +110,50 @@ def test_route_mismatch_is_rejected(self): with self.assertRaises(ValueError): summary_module.validate(self.results, 3) + def test_runtime_mismatch_is_rejected(self): + self.results[0]["runtime"]["nccl_version"] = [9, 9, 9] + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_logical_payload_mismatch_is_rejected(self): + self.results[0]["avg_logical_payload_bytes_per_rank"] += 1 + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_derived_metric_mismatch_is_rejected(self): + self.results[0]["aggregate_input_tokens_per_second"] += 1 + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_provenance_image_mismatch_is_rejected(self): + provenance = { + "campaign_id": "test-campaign", + "created_at_utc": "2026-08-24T00:00:00Z", + "region": "ap-south-1", + "cluster": "test-cluster", + "git_commit": "a" * 40, + "images": { + arm: next( + result["runtime"]["image_reference"] + for result in self.results + if result["arm"] == arm + ) + for arm in summary_module.ARMS + }, + "comparison": { + "tokens_per_rank": 128, + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": 20, + "measured_iterations": 100, + "independent_starts": 3, + }, + } + provenance["images"]["uccl"] = "example.invalid/changed@sha256:" + "9" * 64 + with self.assertRaises(ValueError): + summary_module.validate_provenance(provenance, self.results, 3) + def test_load_results_accepts_native_diagnostic_after_json(self): result = fake_result("deepep-v2-gin-gda", 32, 1, "fp8", 0.9) with tempfile.TemporaryDirectory() as directory: From ed1b2debf09e40713ee1947965accc039b72c52f Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 17:23:00 -0700 Subject: [PATCH 14/19] Update RESULTS.md to clarify backend comparison Removed outdated explanation regarding retired backend-native tables for clarity. --- .../expert-parallelism/ep-backend-comparison/RESULTS.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index 53b835b51..06e72ae0a 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -1,7 +1,5 @@ # Fair EP Backend Comparison Results on B200 -This report replaces the retired backend-native B300, B200, and H100 tables. Those tables mixed timing boundaries and byte numerators, so their latency and bandwidth columns were not suitable for cross-backend ranking. - The replacement measures one synthetic decode dispatch-plus-combine workload with one external CUDA timing boundary and one logical payload definition. It is not an end-to-end training or serving benchmark. EP32 means 32 GPU ranks on 4 `p6-b200.48xlarge` nodes, not 32 instances. ## Result From 02328418c69ea7fe656e32f8c122de55c114b292 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Mon, 24 Aug 2026 17:23:18 -0700 Subject: [PATCH 15/19] Revise title in RESULTS.md for EP backend comparison Updated the title of the results section for clarity. --- .../expert-parallelism/ep-backend-comparison/RESULTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index 06e72ae0a..640e7653e 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -1,4 +1,4 @@ -# Fair EP Backend Comparison Results on B200 +# EP Backend Comparison Results on B200 The replacement measures one synthetic decode dispatch-plus-combine workload with one external CUDA timing boundary and one logical payload definition. It is not an end-to-end training or serving benchmark. EP32 means 32 GPU ranks on 4 `p6-b200.48xlarge` nodes, not 32 instances. From 1199194b2fb54b6c3305883ec6eedf7346d79922 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Tue, 25 Aug 2026 02:13:26 +0000 Subject: [PATCH 16/19] feat(ep): add decode and prefill benchmark profiles --- .../pytorch/sglang/dsr1-deepep-efa/README.md | 5 +- .../dsr1-deepep-efa/benchmarks/README.md | 11 +- .../ep-backend-comparison/README.md | 112 ++--- .../ep-backend-comparison/RESULTS.md | 142 +----- .../{fair_ep_benchmark.py => ep_benchmark.py} | 269 +++++++--- ...act_fair_results.py => extract_results.py} | 10 +- .../{fair_result_io.py => result_io.py} | 8 +- .../results/b200-ap-south-1-2026-08-24.json | 473 ------------------ ..._ep_comparison.sh => run_ep_comparison.sh} | 99 ++-- .../{run_fair_ep_rank.sh => run_ep_rank.sh} | 15 +- ...e_fair_results.py => summarize_results.py} | 384 +++++++++----- ...r_ep_benchmark.py => test_ep_benchmark.py} | 43 +- ...r_results.py => test_summarize_results.py} | 62 ++- 13 files changed, 696 insertions(+), 937 deletions(-) rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{fair_ep_benchmark.py => ep_benchmark.py} (72%) rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{extract_fair_results.py => extract_results.py} (62%) rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{fair_result_io.py => result_io.py} (83%) delete mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{run_fair_ep_comparison.sh => run_ep_comparison.sh} (85%) rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{run_fair_ep_rank.sh => run_ep_rank.sh} (80%) rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{summarize_fair_results.py => summarize_results.py} (53%) rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{test_fair_ep_benchmark.py => test_ep_benchmark.py} (64%) rename micro-benchmarks/expert-parallelism/ep-backend-comparison/{test_summarize_fair_results.py => test_summarize_results.py} (70%) diff --git a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md index f54d430e0..082d243c3 100644 --- a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md +++ b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/README.md @@ -178,8 +178,9 @@ Three things to know before running this on Blackwell: here has been retired because its backend-native timing and byte accounting did not support a cross-backend ranking. The replacement [`ep-backend-comparison`](../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) -uses a common boundary for a synthetic decode communication workload at EP16 and EP32 on B200. It -does not establish B300 serving behavior. See +defines common boundaries for synthetic Decode-like and Prefill-like communication workloads at +EP16 and EP32 on B200. Its replacement result matrix is pending and does not establish B300 serving +behavior. See [Blackwell serving needs a matched measurement](./benchmarks/README.md#blackwell-serving-needs-a-matched-measurement). ## Smoke-test the EFA transport before loading the model diff --git a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md index 497b6fd0e..31dcec060 100644 --- a/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md +++ b/3.test_cases/pytorch/sglang/dsr1-deepep-efa/benchmarks/README.md @@ -820,11 +820,10 @@ here, so the H200 direction must not be assumed to carry across GPU generations. The current [`ep-backend-comparison`](../../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) -is useful transport evidence with a common CUDA timing boundary and a common logical payload. It is -still a synthetic decode dispatch-plus-combine workload. It excludes expert compute, scheduling, -communication/computation overlap, and request-level serving behavior. Its EP16 B200 result is also -mixed by dtype: UCCL has lower FP8 latency, while DeepEP V2 has lower BF16 latency. At EP32, UCCL has -lower latency for both measured dtypes. None of those cells is a B300 serving result. +defines common CUDA timing boundaries and common logical payload accounting for synthetic +Decode-like and Prefill-like dispatch-plus-combine workloads. The replacement result matrix is +pending. It excludes expert compute, scheduling, communication/computation overlap, and +request-level serving behavior, and it does not provide a B300 serving result. Before attributing a Blackwell serving delta to the all-to-all backend, control these variables: @@ -842,7 +841,7 @@ Before attributing a Blackwell serving delta to the all-to-all backend, control diagnostics unless they share the same timing boundary and byte numerator. - Test the production EP width. An EP16 result cannot decide an EP32 or EP64 deployment. -The fair conclusion for Blackwell serving remains unmeasured until that matched sweep is run. +The Blackwell serving conclusion remains unmeasured until that matched sweep is run. # Reproduce diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index 09bc1eec3..93af4ed9c 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -1,6 +1,6 @@ -# Fair Expert-Parallelism Backend Comparison on EFA +# Expert-Parallelism Backend Comparison on EFA -This directory compares 3 expert-parallel dispatch/combine backends through one common semantic workload and one external timing boundary: +This directory compares 3 expert-parallel dispatch/combine backends through common Decode-like and Prefill-like communication workloads: | Backend | Implementation and transport | |---|---| @@ -8,60 +8,59 @@ This directory compares 3 expert-parallel dispatch/combine backends through one | DeepEP V1 NVSHMEM | DeepEP V1 `Buffer` over NVSHMEM, libfabric, and EFA | | DeepEP V2 NCCL GIN | DeepEP V2 `ElasticBuffer` over NCCL GIN EFA-GDA | -The primary metric is slowest-rank CUDA latency from BF16 input readiness through dispatch and combine completion. Backend-native latency and bandwidth fields are not used for cross-backend rankings because they do not share one timing boundary or byte numerator. +Each workload uses one external CUDA timing boundary and one logical payload definition. Backend-native latency and bandwidth fields remain diagnostics because they do not share one timing boundary or byte numerator. -The validated B200 result is in [RESULTS.md](RESULTS.md). EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes. +The B200 report is in [RESULTS.md](RESULTS.md). EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes. -## What makes the comparison fair +## Workload profiles -The backend implementation is the intended independent variable. The harness holds these inputs and measurement rules constant: +| Profile | Tokens | UCCL and DeepEP V1 API | DeepEP V2 API | Primary metric | +|---|---:|---|---|---| +| Decode-like | 128 tokens/rank | `low_latency_dispatch` and `low_latency_combine` | `ElasticBuffer.dispatch` and `ElasticBuffer.combine` | Slowest-rank latency, in ms | +| Prefill-like | 4,096 tokens/rank | Normal `Buffer.dispatch` and `Buffer.combine` | `ElasticBuffer.dispatch` and `ElasticBuffer.combine` | Effective logical throughput, in GB/s/rank | + +The Prefill-like timing boundary starts with a BF16 input and exact route ready. It includes the dispatch layout required by the normal UCCL and DeepEP V1 APIs, dispatch, and combine completion. The Decode-like boundary starts with the BF16 input ready and includes dispatch and combine completion. Required FP8 conversion is inside both boundaries. + +Both profiles hold the following controls constant across backends: | Control | Rule | |---|---| -| Input | One deterministic BF16 tensor per EP size | +| Input | One deterministic BF16 tensor per profile and EP size | | Routing | One exact top-k route and one set of weights, verified by SHA-256 across all arms and starts | -| Shape | 128 tokens/rank, hidden size 7,168, 256 experts, top-k 8 experts/token | +| Model shape | Hidden size 7,168, 256 experts, top-k 8 experts/token | | Operations | FP8 or BF16 dispatch followed by BF16 combine | -| Timing | One CUDA Event boundary around input preparation, dispatch, and combine | | Rank reduction | Maximum elapsed time across all ranks for each measured iteration | | Warmup | 20 warmup iterations per dtype and process start | | Measurement | 100 measured iterations per dtype and process start | | Replication | 3 independent process starts per arm and workload cell | -| Order | Backend order rotates across starts; dtype order also rotates | +| Order | Backend, dtype, and workload-profile order rotate across starts | | Hardware | The same named nodes serve every arm at a given EP size | -| Runtime | Every result must report the same GPU, PyTorch, CUDA, and NCCL versions | -| Correctness | Every rank must pass the common identity-expert result before timing | - -Each process start contributes its median of 100 slowest-rank iteration latencies. The report then takes the median across 3 process starts. Iterations within one process are not treated as independent replicates. +| Runtime | Every result reports the same GPU, PyTorch, CUDA, and NCCL versions | +| Correctness | Every rank passes the common identity-expert result before timing | -### Common logical throughput +Each process start contributes its median of 100 slowest-rank iteration measurements. The report then takes the median across 3 process starts. Iterations within one process are not treated as independent replicates. -The harness derives effective logical throughput from a common useful-payload numerator. Each valid expert assignment contributes: +## Common logical throughput -- the dispatch tensor; -- FP8 scales when FP8 dispatch is selected; and -- the BF16 combine tensor. - -Backend metadata is excluded. Scale-out logical bytes include only assignments whose destination expert is on another node. +Each valid expert assignment contributes the dispatch tensor, FP8 scales when FP8 dispatch is selected, and the BF16 combine tensor. Backend metadata is excluded. Scale-out logical bytes include only assignments whose destination expert is on another node. ```text logical GB/s/rank = average logical bytes/rank / median slowest-rank latency scale-out logical GB/s/rank = average remote logical bytes/rank / median slowest-rank latency ``` -These are logical efficiency metrics, not observed wire bandwidth. DeepEP V2 SO/SU bandwidth, DeepEP V1 native bandwidth, and UCCL native bandwidth remain useful backend diagnostics, but their numerators and aggregation boundaries differ and must not be compared directly. +These are logical efficiency metrics, not observed wire bandwidth. ## Files | File | Purpose | |---|---| -| [`fair_ep_benchmark.py`](fair_ep_benchmark.py) | Common workload, backend adapters, correctness check, CUDA timing, and logical-byte accounting | -| [`run_fair_ep_rank.sh`](run_fair_ep_rank.sh) | Per-node `torchrun` entry point and backend-specific transport environment | -| [`run_fair_ep_comparison.sh`](run_fair_ep_comparison.sh) | EKS admission, shared-Lease coordination, rotated matrix, durable harvest, and verified teardown | -| [`fair_result_io.py`](fair_result_io.py) | Robust result-marker parsing from interleaved native output | -| [`extract_fair_results.py`](extract_fair_results.py) | Canonical JSONL extraction from a rank-zero log | -| [`summarize_fair_results.py`](summarize_fair_results.py) | Matrix validation, per-start aggregation, paired deltas, and bootstrap intervals | -| [`results/b200-ap-south-1-2026-08-24.json`](results/b200-ap-south-1-2026-08-24.json) | Machine-readable validated result summary | +| [`ep_benchmark.py`](ep_benchmark.py) | Workload profiles, backend adapters, correctness checks, CUDA timing, and logical-byte accounting | +| [`run_ep_rank.sh`](run_ep_rank.sh) | Per-node `torchrun` entry point and backend-specific transport environment | +| [`run_ep_comparison.sh`](run_ep_comparison.sh) | EKS admission, shared-Lease coordination, rotated matrix, durable harvest, and verified teardown | +| [`result_io.py`](result_io.py) | Robust result-marker parsing from interleaved native output | +| [`extract_results.py`](extract_results.py) | Canonical JSONL extraction from a rank-zero log | +| [`summarize_results.py`](summarize_results.py) | Matrix validation, per-start aggregation, paired deltas, and bootstrap intervals | | [`RESULTS.md`](RESULTS.md) | Human-readable result, provenance, and scope limits | ## Requirements @@ -77,21 +76,21 @@ The scored B200 matrix requires: - `aws`, `kubectl`, `jq`, `rg`, Python 3, and Bash on the launch host; and - access to the 3 digest-pinned backend images. -DeepEP V2 receives an INFO-level EP16 admission run before the scored matrix. The admission must log a GDAKI context. A missing HMM mitigation, GDRCopy device, or GDAKI proof stops the campaign before scoring. +DeepEP V2 receives an INFO-level EP16 Decode-like admission run before the scored matrix. All 3 backends then receive an EP16 Prefill-like admission run. A missing HMM mitigation, GDRCopy device, GDAKI proof, or profile correctness result stops the campaign before scoring. ## Run on an exclusive node set Use a unique namespace and durable artifact directory. `KUBECTL_CONTEXT` is required explicitly so a concurrent process changing the default context cannot redirect the campaign. ```bash -campaign_id=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) +campaign_id=ep-b200-$(date -u +%Y%m%d%H%M%S) CAMPAIGN_ID="${campaign_id}" \ -FAIR_EP_NODES=node-a,node-b,node-c,node-d \ +EP_BENCHMARK_NODES=node-a,node-b,node-c,node-d \ PROTECTED_NODES_CSV="" \ ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ KUBECTL_CONTEXT=aps1-shared \ LOCK_MODE=exclusive \ -./run_fair_ep_comparison.sh +./run_ep_comparison.sh ``` `LOCK_MODE=exclusive` claims the configured shared Lease only when its holder is empty. The runner releases only a Lease that it still owns. @@ -101,30 +100,30 @@ LOCK_MODE=exclusive \ Observe mode is allowed only when the other campaign has a known Lease holder and a disjoint named node set: ```bash -campaign_id=fair-ep-b200-$(date -u +%Y%m%d%H%M%S) +campaign_id=ep-b200-$(date -u +%Y%m%d%H%M%S) CAMPAIGN_ID="${campaign_id}" \ -FAIR_EP_NODES=node-a,node-b,node-c,node-d \ +EP_BENCHMARK_NODES=node-a,node-b,node-c,node-d \ PROTECTED_NODES_CSV=foreign-node-a,foreign-node-b \ ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ KUBECTL_CONTEXT=aps1-shared \ LOCK_MODE=observe \ EXPECTED_LOCK_HOLDER=foreign-campaign-id \ -./run_fair_ep_comparison.sh +./run_ep_comparison.sh ``` Observe mode never mutates the shared Lease. It verifies the exact holder before every arm and again before aggregation. Any selected/protected node overlap or Lease-holder change stops the run. ## Execution matrix -The scored order is a 3-start rotation: +The scored order uses 3 independent starts: -| Start index | Backend order | Dtype order | -|---:|---|---| -| 1 | UCCL, DeepEP V1, DeepEP V2 | FP8, BF16 | -| 2 | DeepEP V2, UCCL, DeepEP V1 | BF16, FP8 | -| 3 | DeepEP V1, DeepEP V2, UCCL | FP8, BF16 | +| Start index | Backend order | Profile order | Dtype order | +|---:|---|---|---| +| 1 | UCCL, DeepEP V1, DeepEP V2 | Decode-like, Prefill-like | FP8, BF16 | +| 2 | DeepEP V2, UCCL, DeepEP V1 | Prefill-like, Decode-like | BF16, FP8 | +| 3 | DeepEP V1, DeepEP V2, UCCL | Decode-like, Prefill-like | FP8, BF16 | -The runner executes this rotation first at 16 ranks on 2 nodes and then at 32 ranks on 4 nodes. Arms run serially, and every StatefulSet and its GPU pods must be gone before the next arm is admitted. +The runner executes the rotation first at 16 ranks on 2 nodes and then at 32 ranks on 4 nodes. Arms run serially, and every StatefulSet and its GPU pods must be gone before the next arm is admitted. The full matrix contains 36 distributed process starts and 72 scored dtype results. ## Durable artifacts and teardown @@ -140,8 +139,10 @@ control/ provenance.json selected-nodes.txt runs/ - ep16/{admission,measurement}-repeat-*// - ep32/measurement-repeat-*// + decode/ep16/{admission,measurement}-repeat-*// + decode/ep32/measurement-repeat-*// + prefill/ep16/{admission,measurement}-repeat-*// + prefill/ep32/measurement-repeat-*// summary/ summary.json summary.md @@ -158,35 +159,32 @@ Every rank log, rendered Pod manifest, Pod description, canonical rank-zero JSON ## Re-aggregate preserved logs -The normal campaign aggregates automatically. To validate a preserved artifact tree again: - ```bash -python3 summarize_fair_results.py /path/to/artifacts/runs \ +python3 summarize_results.py /path/to/artifacts/runs \ --starts=3 \ --provenance=/path/to/artifacts/control/provenance.json \ --json=/path/to/artifacts/summary/summary.json \ --markdown=/path/to/artifacts/summary/summary.md ``` -If a native library appends a diagnostic to the JSON marker's physical line, extract the JSON object with the repository parser rather than `grep` or line splitting: +If a native library appends a diagnostic to the JSON marker's physical line, use the repository parser: ```bash -python3 extract_fair_results.py rank-zero.log results.jsonl +python3 extract_results.py rank-zero.log results.jsonl ``` -The summarizer rejects an incomplete matrix, correctness failure, mutable image tag, route/input mismatch, runtime-stack mismatch, or disagreement in the common logical payload. +The summarizer rejects an incomplete matrix, correctness failure, mutable image tag, route/input mismatch, runtime-stack mismatch, or disagreement in common logical payload accounting. ## Local validation ```bash -python3 -m pytest -q test_fair_ep_benchmark.py test_summarize_fair_results.py +python3 -m pytest -q test_ep_benchmark.py test_summarize_results.py python3 -m py_compile \ - fair_ep_benchmark.py fair_result_io.py extract_fair_results.py \ - summarize_fair_results.py -bash -n run_fair_ep_comparison.sh run_fair_ep_rank.sh -shellcheck run_fair_ep_comparison.sh run_fair_ep_rank.sh + ep_benchmark.py result_io.py extract_results.py summarize_results.py +bash -n run_ep_comparison.sh run_ep_rank.sh +shellcheck run_ep_comparison.sh run_ep_rank.sh ``` ## Scope limits -This harness measures a synthetic decode dispatch-plus-combine communication workload. It does not measure prefill, expert compute, communication/computation overlap, end-to-end training, serving throughput, TTFT, TPOT, or E2E latency. A result from this harness does not establish a universal backend winner or a limit at an unmeasured EP size. +These profiles measure synthetic dispatch-plus-combine communication. Prefill-like does not measure TTFT, and Decode-like does not measure TPOT. Neither profile measures expert compute, communication/computation overlap, end-to-end training, serving throughput, or end-to-end latency. Results apply only to the reported profile, EP size, routing distribution, hardware, and runtime stack. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index 53b835b51..b0b5ca48a 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -1,138 +1,16 @@ -# Fair EP Backend Comparison Results on B200 +# EP Backend Comparison Results on B200 -This report replaces the retired backend-native B300, B200, and H100 tables. Those tables mixed timing boundaries and byte numerators, so their latency and bandwidth columns were not suitable for cross-backend ranking. +Status: `NOT_RUN_PROFILE_MATRIX_PENDING` -The replacement measures one synthetic decode dispatch-plus-combine workload with one external CUDA timing boundary and one logical payload definition. It is not an end-to-end training or serving benchmark. EP32 means 32 GPU ranks on 4 `p6-b200.48xlarge` nodes, not 32 instances. +The previous Decode-only result has been retired. It must not be combined with the new two-profile matrix because Prefill-like uses the normal high-throughput API and a different timing boundary. -## Result +The replacement campaign will score these workload profiles separately: -UCCL has the lowest latency in 3 of the 4 measured workload cells. DeepEP V2 has the lowest latency at EP16 with BF16 dispatch. DeepEP V2 is faster than DeepEP V1 in all 4 cells. These observations apply only to the configuration and scales in this report. +| Profile | Tokens | API class | Primary metric | +|---|---:|---|---| +| Decode-like | 128 tokens/rank | Low-latency dispatch and combine | Slowest-rank latency, in ms | +| Prefill-like | 4,096 tokens/rank | Normal high-throughput dispatch and combine, including required layout | Effective logical throughput, in GB/s/rank | -Each value is the median across 3 independent process starts. Each process start contributes the median of 100 slowest-rank measured iterations after 20 warmup iterations. +Both profiles use hidden size 7,168, 256 experts, top-k 8 experts/token, FP8 and BF16 dispatch, BF16 combine, EP16 and EP32, 20 warmup iterations, 100 measured iterations, and 3 independent process starts per cell. -| EP size | Dispatch dtype | Backend | Latency | 95% bootstrap CI | Run-to-run CV | Input throughput | Logical throughput | Scale-out logical throughput | -|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| -| 16 ranks | FP8 | UCCL | 0.5213 ms | [0.5190, 0.5277] ms | 0.86% | 3,928,790.51 tokens/s | 42.68 GB/s/rank | 21.34 GB/s/rank | -| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 1.0179 ms | [1.0167, 1.0203] ms | 0.18% | 2,011,977.59 tokens/s | 21.86 GB/s/rank | 10.93 GB/s/rank | -| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 0.6219 ms | [0.6180, 0.6375] ms | 1.65% | 3,293,197.58 tokens/s | 35.78 GB/s/rank | 17.89 GB/s/rank | -| 16 ranks | BF16 | UCCL | 0.5920 ms | [0.5918, 0.5978] ms | 0.58% | 3,459,365.90 tokens/s | 49.59 GB/s/rank | 24.80 GB/s/rank | -| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 1.0297 ms | [1.0251, 1.0304] ms | 0.28% | 1,988,936.56 tokens/s | 28.51 GB/s/rank | 14.26 GB/s/rank | -| 16 ranks | BF16 | DeepEP V2 NCCL GIN | 0.4850 ms | [0.4826, 0.5043] ms | 2.42% | 4,222,332.23 tokens/s | 60.53 GB/s/rank | 30.27 GB/s/rank | -| 32 ranks | FP8 | UCCL | 0.7655 ms | [0.7640, 0.7714] ms | 0.51% | 5,350,835.03 tokens/s | 29.07 GB/s/rank | 21.80 GB/s/rank | -| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 1.5542 ms | [1.5513, 1.5628] ms | 0.38% | 2,635,371.65 tokens/s | 14.32 GB/s/rank | 10.74 GB/s/rank | -| 32 ranks | FP8 | DeepEP V2 NCCL GIN | 0.9537 ms | [0.9217, 0.9675] ms | 2.48% | 4,294,725.45 tokens/s | 23.33 GB/s/rank | 17.50 GB/s/rank | -| 32 ranks | BF16 | UCCL | 0.8686 ms | [0.8683, 0.8696] ms | 0.08% | 4,715,590.93 tokens/s | 33.80 GB/s/rank | 25.35 GB/s/rank | -| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 1.5589 ms | [1.5526, 1.5714] ms | 0.61% | 2,627,446.53 tokens/s | 18.83 GB/s/rank | 14.13 GB/s/rank | -| 32 ranks | BF16 | DeepEP V2 NCCL GIN | 0.9509 ms | [0.9482, 0.9606] ms | 0.68% | 4,307,661.17 tokens/s | 30.88 GB/s/rank | 23.16 GB/s/rank | - -The maximum run-to-run CV is 2.48%. The 95% intervals use 20,000 bootstrap resamples of the 3 independent process-start medians. With only 3 independent starts, the intervals describe this campaign but should not be read as precise estimates of production variability. - -## Paired DeepEP V2 latency deltas - -Starts are paired by EP size, dtype, input, route, named nodes, and rotation index. A positive value means DeepEP V2 had lower latency than the baseline. A direction is marked supported only when the paired bootstrap interval excludes 0% and both arms have no more than 5% run-to-run CV. - -| EP size | Dispatch dtype | Baseline | Median V2 latency reduction | 95% bootstrap CI | Direction supported | -|---:|:---:|:---|---:|:---:|:---:| -| 16 ranks | FP8 | UCCL | -19.30% | [-22.84%, -17.11%] | Yes | -| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 39.05% | [37.37%, 39.22%] | Yes | -| 16 ranks | BF16 | UCCL | 18.44% | [14.82%, 18.87%] | Yes | -| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 52.68% | [51.06%, 53.13%] | Yes | -| 32 ranks | FP8 | UCCL | -24.83% | [-25.41%, -20.40%] | Yes | -| 32 ranks | FP8 | DeepEP V1 NVSHMEM | 38.64% | [37.64%, 41.02%] | Yes | -| 32 ranks | BF16 | UCCL | -9.35% | [-10.63%, -9.16%] | Yes | -| 32 ranks | BF16 | DeepEP V1 NVSHMEM | 38.87% | [38.76%, 39.18%] | Yes | - -"Direction supported" is a campaign-level reproducibility rule, not a universal performance claim or a formal significance test. - -## Common comparison boundary - -The harness holds the semantic workload and measurement rules constant across the 3 backends: - -| Control | Value | -|---|---| -| Input | Deterministic BF16 tensor, 128 tokens/rank | -| Model shape | Hidden size 7,168, 256 experts, top-k 8 experts/token | -| Operation | FP8 or BF16 dispatch followed by BF16 combine | -| Expert work | Identity-expert semantics; expert compute is outside the timed boundary | -| Timing | CUDA Events from input preparation through dispatch and combine completion | -| Rank reduction | Maximum elapsed time across all ranks for every measured iteration | -| Warmup and measurement | 20 warmup iterations and 100 measured iterations per dtype and process start | -| Replication | 3 independent process starts per backend and workload cell | -| Order control | Backend order and dtype order rotate across process starts | -| Hardware control | The same named nodes are used for every backend at each EP size | -| Input control | Input and routing SHA-256 hashes must match across all backends and starts in each EP size | -| Runtime control | GPU model, PyTorch, CUDA, and NCCL versions must match across every scored result | -| Image control | Every backend image must be digest-pinned and invariant across the matrix | -| Correctness | Every rank must pass the common identity-expert output check before timing | - -DeepEP V2 uses non-expanded dispatch, which sends a token once per destination rank. The correctness path applies the local gated identity-expert reduction before combine so V2 and the expanded backends return the same weighted token. This normalizes semantic output, not the internal algorithm. - -## Logical throughput definition - -The cross-backend throughput columns use the same useful-payload numerator. Each valid expert assignment contributes its dispatch tensor, FP8 scales when FP8 is selected, and its BF16 combine tensor. Backend-specific metadata is excluded. Scale-out logical bytes count only assignments whose destination expert is on a different node. - -```text -logical GB/s/rank = average logical bytes/rank / median slowest-rank latency -scale-out logical GB/s/rank = average remote logical bytes/rank / median slowest-rank latency -``` - -These values are effective logical throughput, not observed wire bandwidth. They intentionally replace the retired side-by-side native bandwidth columns. In DeepEP V2, for example, SO bandwidth uses `num_scaleout_bytes / t`, SU bandwidth uses `num_scaleup_bytes / t`, and the trailing `bytes` field is the SU numerator. DeepEP V1 and UCCL use their own byte accounting and aggregation boundaries. Those native metrics remain useful for diagnosing a backend, but comparing them as one GB/s metric is not valid. - -## Correctness and admission - -All 36 scored dtype results passed on every rank. The normalized-difference tolerance was `9e-4` dimensionless for FP8 dispatch and `1e-5` dimensionless for BF16 dispatch. The matrix contains 18 distributed process starts: 2 EP sizes multiplied by 3 backends and 3 independent starts, with 2 dtype results from each start. - -Before scoring, the DeepEP V2 EP16 admission run also verified: - -- `uvm_disable_hmm=1` on all 4 selected hosts; -- `/dev/gdrdrv` present as a character device on all 4 selected hosts; and -- an INFO-level GDAKI context in the DeepEP V2 log. - -## Environment and provenance - -| Field | Value | -|---|---| -| Campaign | `fair-ep-b200-20260824t195145z` | -| Date | 2026-08-24 UTC | -| AWS Region and Availability Zone | `ap-south-1`, `ap-south-1c` | -| EKS cluster | `ml-clusters-shared-ap-south-1` | -| Instance type | `p6-b200.48xlarge` | -| Node topology | 4 nodes, 8 NVIDIA B200 GPUs/node, 8 allocatable EFA devices/node | -| Runtime | PyTorch `2.13.0+cu130`, CUDA `13.0`, NCCL `2.29.7` | - -Digest-pinned images: - -| Backend | Image | -|---|---| -| UCCL | `159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a` | -| DeepEP V1 NVSHMEM | `159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac` | -| DeepEP V2 NCCL GIN | `159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb` | - -Selected nodes: - -```text -ip-10-6-124-201.ap-south-1.compute.internal -ip-10-6-126-97.ap-south-1.compute.internal -ip-10-6-127-248.ap-south-1.compute.internal -ip-10-6-71-166.ap-south-1.compute.internal -``` - -## Concurrent campaign isolation - -The campaign ran in coordinated observe mode beside the vLLM campaign identified by Lease holder `dsv3-b200-parallel-r7-20260824t2122z`. The vLLM campaign had 12 protected nodes. The selected-node overlap was 0 nodes. The runner did not mutate the shared Lease and rechecked its exact holder before each arm and before aggregation. - -Node disjointness prevents direct GPU and host contention. It does not prove absence of shared network-fabric effects, so these results remain specific to the observed cluster conditions. - -## Artifact custody and teardown - -The durable artifact contains 19 canonical result files and 38 result records, including 36 scored records and 2 admission records. It retains the rendered manifests, all-rank logs, Pod descriptions, image references, input and route hashes, admission evidence, cluster snapshots, teardown evidence, and a `SHA256SUMS` manifest covering 523 files. - -The detached runner returned a nonzero status after all 18 scored jobs passed because one backend diagnostic was appended to a JSON result line. The parser was corrected, canonical records were re-extracted from the preserved rank-zero logs, and the complete matrix was revalidated. No benchmark value was reconstructed or rerun during recovery. - -Final teardown verification passed: the owned namespace was absent, 0 owned resources remained, all 4 selected nodes were Ready, and 0 selected-node GPU Pods remained. The concurrent vLLM Lease holder was unchanged. - -The committed machine-readable summary is [`results/b200-ap-south-1-2026-08-24.json`](results/b200-ap-south-1-2026-08-24.json), SHA-256 `129877f60412e4a87fe1b8dd29074bcf803457d9c8b5c816276616fd09700f66`. - -## Limits - -This benchmark does not measure prefill, expert compute, communication/computation overlap, memory footprint, end-to-end training, serving throughput, TTFT, TPOT, or end-to-end latency. It covers only EP16 and EP32 on B200 in one Availability Zone, with 3 independent starts per cell. It does not establish behavior on B300, H100, larger EP domains, different routing distributions, or production Kimi K2 training replicas. +No backend performance conclusion is reported until all 72 scored dtype results pass validation and the owned EKS resources pass teardown verification. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py similarity index 72% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py index f516444f0..4b96c3a27 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py @@ -6,9 +6,10 @@ indices, and top-k weights. A CUDA event pair surrounds the complete dispatch followed by combine operation, and the slowest rank is the iteration latency. -For FP8 dispatch, the timed boundary starts with a BF16 input. DeepEP V2's -explicit BF16-to-FP8 preparation is therefore inside the timed region, matching -the conversion already performed inside the V1 and UCCL low-latency APIs. +The decode profile uses each backend's low-latency path with 128 tokens/rank. +The prefill profile uses the normal high-throughput path with 4,096 tokens/rank +and includes any required dispatch-layout work. FP8 conversion starts from a +BF16 input and remains inside both profiles' timed region. """ from __future__ import annotations @@ -28,8 +29,11 @@ import torch.distributed as dist -RESULT_PREFIX = "ADAI_FAIR_RESULT " -SCHEMA_VERSION = 1 +RESULT_PREFIX = "ADAI_EP_RESULT " +SCHEMA_VERSION = 2 +NORMAL_NUM_NVL_BYTES = 2_000_000_000 +NORMAL_NUM_RDMA_BYTES = 1_000_000_000 +NORMAL_NUM_QPS_PER_RANK = 24 def deepep_v2_build_lib(root: Path = Path("/opt/amazon/deepep-v2")) -> Path: @@ -103,9 +107,7 @@ def make_route( num_tokens, dtype=torch.int64, device=device ) slots = torch.arange(top_k, dtype=torch.int64, device=device) - route = ( - global_token[:, None] * 17 + slots[None, :] * stride + seed - ) % num_experts + route = (global_token[:, None] * 17 + slots[None, :] * stride + seed) % num_experts if top_k > num_experts: raise ValueError("top_k cannot exceed num_experts") return route.contiguous() @@ -116,9 +118,7 @@ def make_input( ) -> torch.Tensor: """Create deterministic, bounded BF16 data without backend RNG state.""" - row = rank * num_tokens + torch.arange( - num_tokens, dtype=torch.int64, device=device - ) + row = rank * num_tokens + torch.arange(num_tokens, dtype=torch.int64, device=device) column = torch.arange(hidden, dtype=torch.int64, device=device) values = (row[:, None] * 17 + column[None, :] * 13 + 19) % 257 return ((values - 128).to(torch.float32) / 128.0).to(torch.bfloat16) @@ -169,6 +169,39 @@ def logical_payload_bytes_per_rank( ) +@dataclass(frozen=True) +class WorkloadProfile: + name: str + tokens_per_rank: int + api_mode: str + primary_metric: str + timing_boundary: str + + +WORKLOAD_PROFILES = { + "decode": WorkloadProfile( + name="decode", + tokens_per_rank=128, + api_mode="low-latency", + primary_metric="slowest-rank latency in milliseconds", + timing_boundary=( + "BF16 input ready through dispatch and combine completion; " + "slowest rank CUDA elapsed time" + ), + ), + "prefill": WorkloadProfile( + name="prefill", + tokens_per_rank=4_096, + api_mode="normal", + primary_metric="effective logical gigabytes per second per rank", + timing_boundary=( + "BF16 input and route ready through required layout, dispatch, and " + "combine completion; slowest rank CUDA elapsed time" + ), + ), +} + + @dataclass class DispatchState: recv_x: Any @@ -182,12 +215,15 @@ def __init__( self, arm: str, group: dist.ProcessGroup, + profile: WorkloadProfile, num_tokens: int, hidden: int, num_experts: int, + top_k: int, ) -> None: self.arm = arm self.group = group + self.profile = profile self.num_tokens = num_tokens self.hidden = hidden self.num_experts = num_experts @@ -199,39 +235,68 @@ def __init__( if arm == "uccl": sys.path.insert(0, "/opt/uccl/ep/bench") from buffer import Buffer # type: ignore[import-not-found] - from utils import per_token_cast_back # type: ignore[import-not-found] + from utils import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) self._cast_back = per_token_cast_back - rdma_bytes = Buffer.get_low_latency_rdma_size_hint( - num_tokens, hidden, self.world_size, num_experts - ) - self.buffer = Buffer( - group, - num_rdma_bytes=rdma_bytes, - low_latency_mode=True, - num_qps_per_rank=num_experts // self.world_size, - allow_nvlink_for_low_latency_mode=True, - explicitly_destroy=True, - ) + self._cast_to_fp8 = per_token_cast_to_fp8 + if profile.api_mode == "low-latency": + rdma_bytes = Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + ) + else: + self.buffer = Buffer( + group, + num_nvl_bytes=NORMAL_NUM_NVL_BYTES, + num_rdma_bytes=NORMAL_NUM_RDMA_BYTES, + low_latency_mode=False, + num_qps_per_rank=NORMAL_NUM_QPS_PER_RANK, + explicitly_destroy=True, + ) elif arm == "deepep-v1-nvshmem": sys.path.insert(0, "/opt/amazon/deepep/tests") sys.path.insert(0, "/opt/amazon/deepep") import deep_ep # type: ignore[import-not-found] - from utils import per_token_cast_back # type: ignore[import-not-found] + from utils import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) self._cast_back = per_token_cast_back - rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint( - num_tokens, hidden, self.world_size, num_experts - ) - self.buffer = deep_ep.Buffer( - group, - num_rdma_bytes=rdma_bytes, - low_latency_mode=True, - num_qps_per_rank=num_experts // self.world_size, - allow_nvlink_for_low_latency_mode=True, - explicitly_destroy=True, - allow_mnnvl=False, - ) + self._cast_to_fp8 = per_token_cast_to_fp8 + if profile.api_mode == "low-latency": + rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = deep_ep.Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + allow_mnnvl=False, + ) + else: + self.buffer = deep_ep.Buffer( + group, + num_nvl_bytes=NORMAL_NUM_NVL_BYTES, + num_rdma_bytes=NORMAL_NUM_RDMA_BYTES, + low_latency_mode=False, + num_qps_per_rank=NORMAL_NUM_QPS_PER_RANK, + explicitly_destroy=True, + allow_mnnvl=False, + ) elif arm == "deepep-v2-gin-gda": import deep_ep # type: ignore[import-not-found] from deep_ep.utils.math import ( # type: ignore[import-not-found] @@ -245,6 +310,7 @@ def __init__( group, num_max_tokens_per_rank=num_tokens, hidden=hidden, + num_topk=top_k, deterministic=False, allow_hybrid_mode=True, allow_multiple_reduction=True, @@ -267,7 +333,7 @@ def prepare_dispatch_input(self, x: torch.Tensor, dispatch_dtype: str) -> Any: return x if dispatch_dtype != "fp8": raise ValueError(f"unsupported dispatch dtype: {dispatch_dtype}") - if self.is_elastic: + if self.is_elastic or self.profile.api_mode == "normal": return self._cast_to_fp8(x) return x @@ -294,6 +360,36 @@ def dispatch( event.current_stream_wait() return DispatchState(recv_x, recv_idx, recv_weights, handle) + if self.profile.api_mode == "normal": + ( + num_tokens_per_rank, + num_tokens_per_rdma_rank, + num_tokens_per_expert, + is_token_in_rank, + _, + ) = self.buffer.get_dispatch_layout(topk_idx, self.num_experts) + ( + recv_x, + recv_idx, + recv_weights, + _, + handle, + event, + ) = self.buffer.dispatch( + x=prepared_x, + num_tokens_per_rank=num_tokens_per_rank, + num_tokens_per_rdma_rank=num_tokens_per_rdma_rank, + is_token_in_rank=is_token_in_rank, + num_tokens_per_expert=num_tokens_per_expert, + topk_idx=topk_idx, + topk_weights=topk_weights, + expert_alignment=1, + async_finish=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return DispatchState(recv_x, recv_idx, recv_weights, handle) + recv_x, _, handle, event, _ = self.buffer.low_latency_dispatch( prepared_x, topk_idx, @@ -317,6 +413,17 @@ def received_as_bf16(self, recv_x: Any, dispatch_dtype: str) -> torch.Tensor: scales.reshape(-1, self.hidden // 128), ).reshape(fp8.shape) + def identity_expert_output( + self, state: DispatchState, dispatch_dtype: str + ) -> torch.Tensor: + output = self.received_as_bf16(state.recv_x, dispatch_dtype) + if state.recv_topk_idx is None or state.recv_topk_weights is None: + return output + local_weights = state.recv_topk_weights.masked_fill( + state.recv_topk_idx < 0, 0 + ).sum(dim=1, keepdim=True) + return output * local_weights.to(output.dtype) + def combine( self, combine_input: torch.Tensor, @@ -335,6 +442,17 @@ def combine( event.current_stream_wait() return combined + if self.profile.api_mode == "normal": + combined, _, event = self.buffer.combine( + x=combine_input, + handle=state.handle, + topk_weights=state.recv_topk_weights, + async_finish=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return combined + combined, event, _ = self.buffer.low_latency_combine( combine_input, topk_idx, @@ -380,21 +498,12 @@ def run_dtype( ) -> dict[str, Any]: prepared = adapter.prepare_dispatch_input(x, dispatch_dtype) correctness_state = adapter.dispatch(prepared, route, topk_weights, dispatch_dtype) - correctness_input = adapter.received_as_bf16( - correctness_state.recv_x, dispatch_dtype + # Normal and ElasticBuffer dispatch send one token per destination rank. + # Apply the local identity experts' gated reduction before combine. The + # low-latency API performs its corresponding reduction internally. + correctness_input = adapter.identity_expert_output( + correctness_state, dispatch_dtype ) - if adapter.is_elastic: - assert correctness_state.recv_topk_idx is not None - assert correctness_state.recv_topk_weights is not None - # V2's non-expanded dispatch sends each token once per destination rank. - # Model the identity experts' local gated reduction before the network - # combine so that every backend returns the same weighted token. - local_weights = correctness_state.recv_topk_weights.masked_fill( - correctness_state.recv_topk_idx < 0, 0 - ).sum(dim=1, keepdim=True) - correctness_input = correctness_input * local_weights.to( - correctness_input.dtype - ) correctness_output = adapter.combine( correctness_input, correctness_state, route, topk_weights ) @@ -448,16 +557,14 @@ def iteration() -> None: dist.all_reduce(local_latency, op=dist.ReduceOp.MAX, group=adapter.group) max_rank_latency_ms.append(float(local_latency.item())) - logical_bytes, scaleout_bytes, valid_selections = ( - logical_payload_bytes_per_rank( - route, - rank, - world_size, - local_world_size, - args.hidden, - dispatch_dtype, - args.experts, - ) + logical_bytes, scaleout_bytes, valid_selections = logical_payload_bytes_per_rank( + route, + rank, + world_size, + local_world_size, + args.hidden, + dispatch_dtype, + args.experts, ) counters = torch.tensor( [logical_bytes, scaleout_bytes, valid_selections], @@ -475,6 +582,12 @@ def iteration() -> None: result = { "schema_version_dimensionless": SCHEMA_VERSION, "benchmark": "common-boundary-dispatch-combine", + "workload_profile": args.profile, + "backend_api_mode": ( + "elastic" if adapter.is_elastic else adapter.profile.api_mode + ), + "primary_metric": adapter.profile.primary_metric, + "layout_in_timed_region": args.profile == "prefill", "arm": args.arm, "run_index_dimensionless": args.run_index, "dispatch_dtype": dispatch_dtype, @@ -492,7 +605,7 @@ def iteration() -> None: "global_valid_expert_selections": global_valid_selections, "warmup_iterations": warmups, "measured_iterations": iterations, - "timing_boundary": "BF16 input ready through dispatch and combine completion; slowest rank CUDA elapsed time", + "timing_boundary": adapter.profile.timing_boundary, "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", "avg_logical_payload_bytes_per_rank": avg_logical_bytes, "avg_scaleout_logical_payload_bytes_per_rank": avg_scaleout_bytes, @@ -538,7 +651,7 @@ def parse_args() -> argparse.Namespace: required=True, choices=("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda"), ) - parser.add_argument("--tokens", type=int, default=128) + parser.add_argument("--profile", required=True, choices=tuple(WORKLOAD_PROFILES)) parser.add_argument("--hidden", type=int, default=7168) parser.add_argument("--top-k", type=int, default=8) parser.add_argument("--experts", type=int, default=256) @@ -552,6 +665,7 @@ def parse_args() -> argparse.Namespace: help="Comma-separated dtype order; allowed values are fp8 and bf16", ) args = parser.parse_args() + args.tokens = WORKLOAD_PROFILES[args.profile].tokens_per_rank args.dispatch_dtypes = args.dispatch_dtypes.split(",") if sorted(args.dispatch_dtypes) != ["bf16", "fp8"]: parser.error("--dispatch-dtypes must contain fp8 and bf16 exactly once") @@ -566,9 +680,7 @@ def main() -> None: rank, world_size, local_world_size, device, group = initialize_distributed() if args.experts % world_size: raise SystemExit("experts must divide the distributed world size") - route = make_route( - rank, args.tokens, args.experts, args.top_k, args.seed, device - ) + route = make_route(rank, args.tokens, args.experts, args.top_k, args.seed, device) x = make_input(rank, args.tokens, args.hidden, device) topk_weights = torch.full( (args.tokens, args.top_k), @@ -578,17 +690,18 @@ def main() -> None: ) route_hash = global_digest(tensor_sha256(route), device) input_hash = global_digest(tensor_sha256(x), device) - route_histogram = torch.bincount( - route.flatten(), minlength=args.experts - ).to(torch.int64) + route_histogram = torch.bincount(route.flatten(), minlength=args.experts).to( + torch.int64 + ) dist.all_reduce(route_histogram, op=dist.ReduceOp.SUM, group=group) if rank == 0: print( - "ADAI_FAIR_CONFIG " + "ADAI_EP_CONFIG " + json.dumps( { "arm": args.arm, + "workload_profile": args.profile, "world_size_ranks": world_size, "tokens_per_rank": args.tokens, "hidden_dimensions": args.hidden, @@ -596,12 +709,8 @@ def main() -> None: "top_k_dimensionless": args.top_k, "route_hash_sha256": route_hash, "input_hash_sha256": input_hash, - "route_histogram_min_selections": int( - route_histogram.min().item() - ), - "route_histogram_max_selections": int( - route_histogram.max().item() - ), + "route_histogram_min_selections": int(route_histogram.min().item()), + "route_histogram_max_selections": int(route_histogram.max().item()), }, sort_keys=True, ), @@ -609,7 +718,13 @@ def main() -> None: ) adapter = BackendAdapter( - args.arm, group, args.tokens, args.hidden, args.experts + args.arm, + group, + WORKLOAD_PROFILES[args.profile], + args.tokens, + args.hidden, + args.experts, + args.top_k, ) try: for dispatch_dtype in args.dispatch_dtypes: diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py similarity index 62% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_fair_results.py rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py index 2e71741b5..15fc55440 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_fair_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py @@ -7,7 +7,7 @@ import json from pathlib import Path -from fair_result_io import PREFIX, load_result_log +from result_io import PREFIX, load_result_log def main() -> None: @@ -18,11 +18,13 @@ def main() -> None: results = load_result_log(args.input_log) if not results: - raise SystemExit(f"no fair results found in {args.input_log}") + raise SystemExit(f"no benchmark results found in {args.input_log}") args.output_jsonl.write_text( - "".join(PREFIX + json.dumps(result, sort_keys=True) + "\n" for result in results) + "".join( + PREFIX + json.dumps(result, sort_keys=True) + "\n" for result in results + ) ) - print(f"extracted {len(results)} fair results from {args.input_log}") + print(f"extracted {len(results)} benchmark results from {args.input_log}") if __name__ == "__main__": diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_result_io.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py similarity index 83% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_result_io.py rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py index 94f32d10e..1d53cee33 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/fair_result_io.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py @@ -1,4 +1,4 @@ -"""Parse fair benchmark result markers despite interleaved native stdout.""" +"""Parse benchmark result markers despite interleaved native stdout.""" from __future__ import annotations @@ -7,7 +7,7 @@ from typing import Any, Iterator -PREFIX = "ADAI_FAIR_RESULT " +PREFIX = "ADAI_EP_RESULT " def iter_result_objects(text: str, source: str = "") -> Iterator[dict[str, Any]]: @@ -31,11 +31,11 @@ def iter_result_objects(text: str, source: str = "") -> Iterator[dict[str, result, consumed = decoder.raw_decode(line[payload_start:]) except json.JSONDecodeError as error: raise ValueError( - f"invalid fair result in {source}:{line_number}: {error}" + f"invalid benchmark result in {source}:{line_number}: {error}" ) from error if not isinstance(result, dict): raise ValueError( - f"fair result in {source}:{line_number} is not a JSON object" + f"benchmark result in {source}:{line_number} is not a JSON object" ) yield result search_from = payload_start + consumed diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json deleted file mode 100644 index 74319eb18..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-24.json +++ /dev/null @@ -1,473 +0,0 @@ -{ - "bootstrap_samples_dimensionless": 20000, - "campaign_provenance": { - "campaign_id": "fair-ep-b200-20260824t195145z", - "cluster": "ml-clusters-shared-ap-south-1", - "comparison": { - "experts": 256, - "hidden_dimensions": 7168, - "independent_starts": 3, - "measured_iterations": 100, - "tokens_per_rank": 128, - "top_k_dimensionless": 8, - "warmup_iterations": 20 - }, - "created_at_utc": "2026-08-24T21:33:19Z", - "git_commit": "249f9e53113cf7706813b19520ddba2c2983e7d6", - "images": { - "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", - "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", - "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" - }, - "region": "ap-south-1" - }, - "cells": [ - { - "arms": { - "deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_latency_ms": [ - 1.0167359709739685, - 1.0202720165252686 - ], - "median_aggregate_input_tokens_per_second": 2011977.5863053314, - "median_effective_logical_gigabytes_per_second_per_rank": 21.85812449762112, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.92906224881056, - "median_latency_ms": 1.017903983592987, - "per_start_median_latency_ms": [ - 1.017903983592987, - 1.0167359709739685, - 1.0202720165252686 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.17692560833015608, - "starts": 3 - }, - "deepep-v2-gin-gda": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.6179679930210114, - 0.6375360190868378 - ], - "median_aggregate_input_tokens_per_second": 3293197.584829476, - "median_effective_logical_gigabytes_per_second_per_rank": 35.777298561587436, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 17.888649280793718, - "median_latency_ms": 0.6218879818916321, - "per_start_median_latency_ms": [ - 0.6375360190868378, - 0.6179679930210114, - 0.6218879818916321 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 1.6544046759560789, - "starts": 3 - }, - "uccl": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.5190079808235168, - 0.5276960134506226 - ], - "median_aggregate_input_tokens_per_second": 3928790.5148029686, - "median_effective_logical_gigabytes_per_second_per_rank": 42.68238015281945, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.341190076409724, - "median_latency_ms": 0.5212800204753876, - "per_start_median_latency_ms": [ - 0.5190079808235168, - 0.5276960134506226, - 0.5212800204753876 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.8620724389950091, - "starts": 3 - } - }, - "comparisons": { - "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_reduction_percent": [ - 37.36776460619894, - 39.22040621528938 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": 39.04684517276182, - "paired_latency_reduction_percent_per_start": [ - 37.36776460619894, - 39.22040621528938, - 39.04684517276182 - ] - }, - "deepep-v2-gin-gda_vs_uccl": { - "bootstrap_95_percent_ci_reduction_percent": [ - -22.837421126983617, - -17.10681477013578 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": -19.300176002236547, - "paired_latency_reduction_percent_per_start": [ - -22.837421126983617, - -17.10681477013578, - -19.300176002236547 - ] - } - }, - "dispatch_dtype": "fp8", - "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", - "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", - "world_size_ranks": 16 - }, - { - "arms": { - "deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_latency_ms": [ - 1.0250880122184753, - 1.0304319858551025 - ], - "median_aggregate_input_tokens_per_second": 1988936.5642491602, - "median_effective_logical_gigabytes_per_second_per_rank": 28.51339458507596, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.25669729253798, - "median_latency_ms": 1.029695987701416, - "per_start_median_latency_ms": [ - 1.0304319858551025, - 1.0250880122184753, - 1.029695987701416 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.2816354149488261, - "starts": 3 - }, - "deepep-v2-gin-gda": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.48263999819755554, - 0.504287987947464 - ], - "median_aggregate_input_tokens_per_second": 4222332.229970289, - "median_effective_logical_gigabytes_per_second_per_rank": 60.53135484885407, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 30.265677424427036, - "median_latency_ms": 0.48503999412059784, - "per_start_median_latency_ms": [ - 0.504287987947464, - 0.48503999412059784, - 0.48263999819755554 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 2.418493715218177, - "starts": 3 - }, - "uccl": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.5917760133743286, - 0.5978399813175201 - ], - "median_aggregate_input_tokens_per_second": 3459365.896020597, - "median_effective_logical_gigabytes_per_second_per_rank": 49.59346948535127, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 24.796734742675635, - "median_latency_ms": 0.5920160114765167, - "per_start_median_latency_ms": [ - 0.5920160114765167, - 0.5978399813175201, - 0.5917760133743286 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.5782085314350778, - "starts": 3 - } - }, - "comparisons": { - "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_reduction_percent": [ - 51.06052656847785, - 53.12791309647133 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": 52.6830878579017, - "paired_latency_reduction_percent_per_start": [ - 51.06052656847785, - 52.6830878579017, - 53.12791309647133 - ] - }, - "deepep-v2-gin-gda_vs_uccl": { - "bootstrap_95_percent_ci_reduction_percent": [ - 14.818522105551635, - 18.86792297636796 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": 18.442115379850478, - "paired_latency_reduction_percent_per_start": [ - 14.818522105551635, - 18.86792297636796, - 18.442115379850478 - ] - } - }, - "dispatch_dtype": "bf16", - "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", - "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", - "world_size_ranks": 16 - }, - { - "arms": { - "deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_latency_ms": [ - 1.55131196975708, - 1.5627520084381104 - ], - "median_aggregate_input_tokens_per_second": 2635371.6483700224, - "median_effective_logical_gigabytes_per_second_per_rank": 14.315338793945964, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.736504095459473, - "median_latency_ms": 1.5542399883270264, - "per_start_median_latency_ms": [ - 1.5542399883270264, - 1.5627520084381104, - 1.55131196975708 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.3819040831100996, - "starts": 3 - }, - "deepep-v2-gin-gda": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.9216639995574951, - 0.9674720168113708 - ], - "median_aggregate_input_tokens_per_second": 4294725.449273138, - "median_effective_logical_gigabytes_per_second_per_rank": 23.32894864045168, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 17.496711480338764, - "median_latency_ms": 0.9537280201911926, - "per_start_median_latency_ms": [ - 0.9537280201911926, - 0.9216639995574951, - 0.9674720168113708 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 2.4805943961903028, - "starts": 3 - }, - "uccl": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.7640160024166107, - 0.7714239954948425 - ], - "median_aggregate_input_tokens_per_second": 5350835.031806838, - "median_effective_logical_gigabytes_per_second_per_rank": 29.065735892774743, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.799301919581055, - "median_latency_ms": 0.7654879987239838, - "per_start_median_latency_ms": [ - 0.7640160024166107, - 0.7654879987239838, - 0.7714239954948425 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.5113275578246953, - "starts": 3 - } - }, - "comparisons": { - "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_reduction_percent": [ - 37.635238064793164, - 41.02301615477362 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": 38.637016975880336, - "paired_latency_reduction_percent_per_start": [ - 38.637016975880336, - 41.02301615477362, - 37.635238064793164 - ] - }, - "deepep-v2-gin-gda_vs_uccl": { - "bootstrap_95_percent_ci_reduction_percent": [ - -25.413783141496673, - -20.402148837584143 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": -24.830895841777636, - "paired_latency_reduction_percent_per_start": [ - -24.830895841777636, - -20.402148837584143, - -25.413783141496673 - ] - } - }, - "dispatch_dtype": "fp8", - "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", - "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", - "world_size_ranks": 32 - }, - { - "arms": { - "deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_latency_ms": [ - 1.552623987197876, - 1.5713760256767273 - ], - "median_aggregate_input_tokens_per_second": 2627446.5313618258, - "median_effective_logical_gigabytes_per_second_per_rank": 18.833536736801566, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.125152552601175, - "median_latency_ms": 1.5589280128479004, - "per_start_median_latency_ms": [ - 1.5589280128479004, - 1.5713760256767273, - 1.552623987197876 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.6113033112150486, - "starts": 3 - }, - "deepep-v2-gin-gda": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.9481759965419769, - 0.9605599939823151 - ], - "median_aggregate_input_tokens_per_second": 4307661.165769053, - "median_effective_logical_gigabytes_per_second_per_rank": 30.87731523623257, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 23.15798642717443, - "median_latency_ms": 0.9508640170097351, - "per_start_median_latency_ms": [ - 0.9481759965419769, - 0.9605599939823151, - 0.9508640170097351 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.683392543166502, - "starts": 3 - }, - "uccl": { - "bootstrap_95_percent_ci_latency_ms": [ - 0.868256002664566, - 0.8695679903030396 - ], - "median_aggregate_input_tokens_per_second": 4715590.93391919, - "median_effective_logical_gigabytes_per_second_per_rank": 33.80135581433276, - "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 25.351016860749567, - "median_latency_ms": 0.8686079978942871, - "per_start_median_latency_ms": [ - 0.8686079978942871, - 0.868256002664566, - 0.8695679903030396 - ], - "run_indices_dimensionless": [ - 1, - 2, - 3 - ], - "run_to_run_cv_percent": 0.07816060809911046, - "starts": 3 - } - }, - "comparisons": { - "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { - "bootstrap_95_percent_ci_reduction_percent": [ - 38.75761131799704, - 39.177692059698245 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": 38.87141089806043, - "paired_latency_reduction_percent_per_start": [ - 39.177692059698245, - 38.87141089806043, - 38.75761131799704 - ] - }, - "deepep-v2-gin-gda_vs_uccl": { - "bootstrap_95_percent_ci_reduction_percent": [ - -10.630964949793603, - -9.16040363899269 - ], - "direction_supported": true, - "median_paired_latency_reduction_percent": -9.349013258683126, - "paired_latency_reduction_percent_per_start": [ - -9.16040363899269, - -10.630964949793603, - -9.349013258683126 - ] - } - }, - "dispatch_dtype": "bf16", - "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", - "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", - "world_size_ranks": 32 - } - ], - "comparison_scope": "synthetic decode dispatch-plus-combine communication workload; not end-to-end training or serving", - "configuration": { - "dispatch_dtypes": [ - "fp8", - "bf16" - ], - "experts_dimensionless": 256, - "hidden_dimensions": 7168, - "measured_iterations_dimensionless": 100, - "tokens_per_rank": 128, - "top_k_dimensionless": 8, - "warmup_iterations_dimensionless": 20, - "world_sizes_ranks": [ - 16, - 32 - ] - }, - "images": { - "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", - "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", - "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" - }, - "independent_starts_per_cell": 3, - "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", - "maximum_run_to_run_cv_percent_for_direction_support": 5.0, - "runtime": { - "cuda_version": "13.0", - "gpu": "NVIDIA B200", - "nccl_version": [ - 2, - 29, - 7 - ], - "torch_version": "2.13.0+cu130" - }, - "schema_version_dimensionless": 1, - "scored_result_records_dimensionless": 36, - "status": "PASS", - "timing_boundary": "BF16 input ready through dispatch and combine completion; slowest rank CUDA elapsed time" -} diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh similarity index 85% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh index 3d5c0c740..64c5568d9 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_comparison.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh @@ -2,7 +2,7 @@ set -euo pipefail : "${CAMPAIGN_ID:?Set a unique CAMPAIGN_ID}" -: "${FAIR_EP_NODES:?Set 4 comma-separated B200 node names}" +: "${EP_BENCHMARK_NODES:?Set 4 comma-separated B200 node names}" : "${PROTECTED_NODES_CSV:=}" : "${ARTIFACT_ROOT:?Set the durable artifact directory}" : "${KUBECTL_CONTEXT:?Set the target kubectl context explicitly}" @@ -62,13 +62,13 @@ mkdir -p "${ARTIFACT_ROOT}/control" "${ARTIFACT_ROOT}/runs" \ "${ARTIFACT_ROOT}/summary" "${ARTIFACT_ROOT}/teardown" K=(kubectl --context "${KUBECTL_CONTEXT}") -IFS=, read -r -a selected_nodes <<<"${FAIR_EP_NODES}" +IFS=, read -r -a selected_nodes <<<"${EP_BENCHMARK_NODES}" protected_nodes=() if [[ -n "${PROTECTED_NODES_CSV}" ]]; then IFS=, read -r -a protected_nodes <<<"${PROTECTED_NODES_CSV}" fi ((${#selected_nodes[@]} == 4)) || { - printf 'FAIR_EP_NODES must contain exactly 4 nodes\n' >&2 + printf 'EP_BENCHMARK_NODES must contain exactly 4 nodes\n' >&2 exit 2 } [[ "$(printf '%s\n' "${selected_nodes[@]}" | sort -u | wc -l)" -eq 4 ]] @@ -143,7 +143,7 @@ claim_shared_lock() { .spec.renewTime=$now | .spec.leaseDurationSeconds=$duration | .metadata.labels["adai.aws/campaign"]=$holder | - .metadata.labels["adai.aws/owner"]="fair-ep-comparison"' \ + .metadata.labels["adai.aws/owner"]="ep-backend-comparison"' \ <<<"${current}")" if printf '%s\n' "${candidate}" | "${K[@]}" replace -f - \ >"${ARTIFACT_ROOT}/control/shared-lease-claim-attempt-${attempt}.json" 2>&1; then @@ -306,7 +306,7 @@ done "${K[@]}" create namespace "${CAMPAIGN_NAMESPACE}" --dry-run=client -o json | \ jq --arg campaign "${CAMPAIGN_ID}" ' .metadata.labels["adai.aws/campaign"]=$campaign | - .metadata.labels["adai.aws/owner"]="fair-ep-comparison"' | \ + .metadata.labels["adai.aws/owner"]="ep-backend-comparison"' | \ "${K[@]}" create -f - >/dev/null namespace_created=1 @@ -372,9 +372,9 @@ done "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete pod -l \ "adai.aws/campaign=${CAMPAIGN_ID}" --wait=true --timeout=5m >/dev/null -"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" create configmap fair-ep-scripts \ - --from-file=fair_ep_benchmark.py="${case_dir}/fair_ep_benchmark.py" \ - --from-file=run_fair_ep_rank.sh="${case_dir}/run_fair_ep_rank.sh" \ +"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" create configmap ep-benchmark-scripts \ + --from-file=ep_benchmark.py="${case_dir}/ep_benchmark.py" \ + --from-file=run_ep_rank.sh="${case_dir}/run_ep_rank.sh" \ --dry-run=client -o yaml | "${K[@]}" apply -f - >/dev/null jq -n \ @@ -392,19 +392,23 @@ jq -n \ '{campaign_id:$campaign_id,created_at_utc:$created_at_utc,region:$region, cluster:$cluster,git_commit:$git_commit, images:{uccl:$uccl,"deepep-v1-nvshmem":$v1,"deepep-v2-gin-gda":$v2}, - comparison:{tokens_per_rank:128,hidden_dimensions:7168,experts:256, - top_k_dimensionless:8,warmup_iterations:$warmups, - measured_iterations:$iterations,independent_starts:$starts}}' \ + comparison:{profiles:{decode:{tokens_per_rank:128,api_mode:"low-latency", + primary_metric:"slowest-rank latency in milliseconds"}, + prefill:{tokens_per_rank:4096,api_mode:"normal", + primary_metric:"effective logical gigabytes per second per rank"}}, + hidden_dimensions:7168,experts:256,top_k_dimensionless:8, + warmup_iterations:$warmups,measured_iterations:$iterations, + independent_starts:$starts}}' \ >"${ARTIFACT_ROOT}/control/provenance.json" run_case() { - local arm="$1" world_size="$2" run_index="$3" dtype_order="$4" - local warmups="${5:-${WARMUP_ITERATIONS}}" iterations="${6:-${MEASURED_ITERATIONS}}" - local nccl_debug="${7:-WARN}" label="${8:-measurement}" + local profile="$1" arm="$2" world_size="$3" run_index="$4" dtype_order="$5" + local warmups="${6:-${WARMUP_ITERATIONS}}" iterations="${7:-${MEASURED_ITERATIONS}}" + local nccl_debug="${8:-WARN}" label="${9:-measurement}" local nodes=$((world_size / 8)) safe_arm="${arm//-}" node_values="" out="" - current_case="fair-ep${world_size}-r${run_index}-${safe_arm:0:20}-${label}" + current_case="ep-${profile:0:1}${world_size}-r${run_index}-${safe_arm:0:20}-${label}" current_case="${current_case:0:63}" - out="${ARTIFACT_ROOT}/runs/ep${world_size}/${label}-repeat-${run_index}/${arm}" + out="${ARTIFACT_ROOT}/runs/${profile}/ep${world_size}/${label}-repeat-${run_index}/${arm}" mkdir -p "${out}" check_shared_lock @@ -466,7 +470,7 @@ spec: - name: benchmark image: ${images[${arm}]} imagePullPolicy: IfNotPresent - command: [/bin/bash, /opt/benchmark/run_fair_ep_rank.sh] + command: [/bin/bash, /opt/benchmark/run_ep_rank.sh] securityContext: {privileged: true} env: - {name: EP_ARM, value: "${arm}"} @@ -476,6 +480,7 @@ spec: - name: POD_NAME valueFrom: {fieldRef: {fieldPath: metadata.name}} - {name: EP_RUN_INDEX, value: "${run_index}"} + - {name: EP_WORKLOAD_PROFILE, value: "${profile}"} - {name: EP_DISPATCH_DTYPES, value: "${dtype_order}"} - {name: EP_WARMUPS, value: "${warmups}"} - {name: EP_ITERATIONS, value: "${iterations}"} @@ -485,13 +490,13 @@ spec: requests: {nvidia.com/gpu: 8, vpc.amazonaws.com/efa: ${EFA_PER_NODE}} limits: {nvidia.com/gpu: 8, vpc.amazonaws.com/efa: ${EFA_PER_NODE}} volumeMounts: - - {name: scripts, mountPath: /opt/benchmark/fair_ep_benchmark.py, subPath: fair_ep_benchmark.py} - - {name: scripts, mountPath: /opt/benchmark/run_fair_ep_rank.sh, subPath: run_fair_ep_rank.sh} + - {name: scripts, mountPath: /opt/benchmark/ep_benchmark.py, subPath: ep_benchmark.py} + - {name: scripts, mountPath: /opt/benchmark/run_ep_rank.sh, subPath: run_ep_rank.sh} - {name: dshm, mountPath: /dev/shm} - {name: gdrdrv, mountPath: /dev/gdrdrv} volumes: - name: scripts - configMap: {name: fair-ep-scripts, defaultMode: 0755} + configMap: {name: ep-benchmark-scripts, defaultMode: 0755} - name: dshm emptyDir: {medium: Memory, sizeLimit: 64Gi} - name: gdrdrv @@ -506,10 +511,10 @@ YAML for ((index = 0; index < nodes; index++)); do pod="${current_case}-${index}" if "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=30 2>/dev/null | \ - rg -q '^ADAI_FAIR_COMPLETE$'; then + rg -q '^ADAI_EP_COMPLETE$'; then complete=$((complete + 1)) elif "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=80 2>/dev/null | \ - rg -q '^ADAI_FAIR_FAILED$'; then + rg -q '^ADAI_EP_FAILED$'; then complete=-1 break fi @@ -527,28 +532,28 @@ YAML >"${out}/${pod}-describe.txt" 2>&1 || true done ((complete == nodes)) || { - printf 'Case failed or timed out: arm=%s EP%s repeat=%s label=%s complete=%s/%s\n' \ - "${arm}" "${world_size}" "${run_index}" "${label}" "${complete}" "${nodes}" >&2 + printf 'Case failed or timed out: profile=%s arm=%s EP%s repeat=%s label=%s complete=%s/%s\n' \ + "${profile}" "${arm}" "${world_size}" "${run_index}" "${label}" "${complete}" "${nodes}" >&2 return 1 } local rank_zero_log="${out}/${current_case}-0.log" result_count - python3 "${case_dir}/extract_fair_results.py" \ + python3 "${case_dir}/extract_results.py" \ "${rank_zero_log}" "${out}/results.jsonl" >/dev/null result_count="$(wc -l <"${out}/results.jsonl")" [[ "${result_count}" -eq 2 ]] || { - printf 'Expected 2 fair results in %s, found %s\n' "${rank_zero_log}" "${result_count}" >&2 + printf 'Expected 2 benchmark results in %s, found %s\n' "${rank_zero_log}" "${result_count}" >&2 return 1 } - printf 'PASS arm=%s EP%s repeat=%s label=%s at %s UTC\n' \ - "${arm}" "${world_size}" "${run_index}" "${label}" "$(date -u +%FT%TZ)" \ + printf 'PASS profile=%s arm=%s EP%s repeat=%s label=%s at %s UTC\n' \ + "${profile}" "${arm}" "${world_size}" "${run_index}" "${label}" "$(date -u +%FT%TZ)" \ >"${out}/STATUS" cleanup_case } # A short V2-only admission run proves the HMM mitigation, GIN/GDAKI path, and # common-harness correctness before any scored matrix work. -run_case deepep-v2-gin-gda 16 0 bf16,fp8 2 5 INFO admission -v2_admission_dir="${ARTIFACT_ROOT}/runs/ep16/admission-repeat-0/deepep-v2-gin-gda" +run_case decode deepep-v2-gin-gda 16 0 bf16,fp8 2 5 INFO admission +v2_admission_dir="${ARTIFACT_ROOT}/runs/decode/ep16/admission-repeat-0/deepep-v2-gin-gda" rg -q 'GDAKI.*createContext|gin GDAKI: createContext done' \ "${v2_admission_dir}"/*.log || { printf 'DeepEP V2 admission completed without a GDAKI context proof\n' >&2 @@ -556,15 +561,35 @@ rg -q 'GDAKI.*createContext|gin GDAKI: createContext done' \ } printf 'PASS\n' >"${v2_admission_dir}/GIN_ADMISSION_STATUS" +# Exercise every backend's normal high-throughput API before entering the +# doubled scored matrix. These short runs are admission evidence only. +for arm in uccl deepep-v1-nvshmem deepep-v2-gin-gda; do + run_case prefill "${arm}" 16 0 fp8,bf16 2 5 WARN admission +done + for world_size in 16 32; do for run_index in 1 2 3; do case "${run_index}" in - 1) order=(uccl deepep-v1-nvshmem deepep-v2-gin-gda); dtypes=fp8,bf16 ;; - 2) order=(deepep-v2-gin-gda uccl deepep-v1-nvshmem); dtypes=bf16,fp8 ;; - 3) order=(deepep-v1-nvshmem deepep-v2-gin-gda uccl); dtypes=fp8,bf16 ;; + 1) + order=(uccl deepep-v1-nvshmem deepep-v2-gin-gda) + profiles=(decode prefill) + dtypes=fp8,bf16 + ;; + 2) + order=(deepep-v2-gin-gda uccl deepep-v1-nvshmem) + profiles=(prefill decode) + dtypes=bf16,fp8 + ;; + 3) + order=(deepep-v1-nvshmem deepep-v2-gin-gda uccl) + profiles=(decode prefill) + dtypes=fp8,bf16 + ;; esac - for arm in "${order[@]}"; do - run_case "${arm}" "${world_size}" "${run_index}" "${dtypes}" + for profile in "${profiles[@]}"; do + for arm in "${order[@]}"; do + run_case "${profile}" "${arm}" "${world_size}" "${run_index}" "${dtypes}" + done done done done @@ -573,9 +598,9 @@ check_shared_lock for node in "${selected_nodes[@]}"; do verify_node_free "${node}" done -python3 "${case_dir}/summarize_fair_results.py" "${ARTIFACT_ROOT}/runs" \ +python3 "${case_dir}/summarize_results.py" "${ARTIFACT_ROOT}/runs" \ --starts="${INDEPENDENT_STARTS}" \ --provenance="${ARTIFACT_ROOT}/control/provenance.json" \ --json="${ARTIFACT_ROOT}/summary/summary.json" \ --markdown="${ARTIFACT_ROOT}/summary/summary.md" -printf 'PASS fair EP comparison completed at %s UTC\n' "$(date -u +%FT%TZ)" +printf 'PASS EP backend comparison completed at %s UTC\n' "$(date -u +%FT%TZ)" diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_rank.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh similarity index 80% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_rank.sh rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh index 1f5fea229..0cb2a6939 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_fair_ep_rank.sh +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh @@ -7,6 +7,7 @@ set -euo pipefail : "${EP_MASTER_ADDR:?Set EP_MASTER_ADDR}" : "${POD_NAME:?Set POD_NAME}" : "${EP_RUN_INDEX:?Set EP_RUN_INDEX}" +: "${EP_WORKLOAD_PROFILE:?Set EP_WORKLOAD_PROFILE}" : "${EP_DISPATCH_DTYPES:?Set EP_DISPATCH_DTYPES}" : "${EP_WARMUPS:=20}" : "${EP_ITERATIONS:=100}" @@ -50,8 +51,8 @@ case "${EP_ARM}" in ;; esac -printf 'ADAI_FAIR_LAUNCH arm=%s nodes=%s ranks=%s run_index=%s_dimensionless dtype_order=%s warmups=%s_iterations measured=%s_iterations\n' \ - "${EP_ARM}" "${EP_NODES}" "$((EP_NODES * 8))" "${EP_RUN_INDEX}" \ +printf 'ADAI_EP_LAUNCH profile=%s arm=%s nodes=%s ranks=%s run_index=%s_dimensionless dtype_order=%s warmups=%s_iterations measured=%s_iterations\n' \ + "${EP_WORKLOAD_PROFILE}" "${EP_ARM}" "${EP_NODES}" "$((EP_NODES * 8))" "${EP_RUN_INDEX}" \ "${EP_DISPATCH_DTYPES}" "${EP_WARMUPS}" "${EP_ITERATIONS}" set +e @@ -61,9 +62,9 @@ torchrun \ --node-rank="${ordinal}" \ --master-addr="${EP_MASTER_ADDR}" \ --master-port=29400 \ - /opt/benchmark/fair_ep_benchmark.py \ + /opt/benchmark/ep_benchmark.py \ --arm="${EP_ARM}" \ - --tokens=128 \ + --profile="${EP_WORKLOAD_PROFILE}" \ --hidden=7168 \ --top-k=8 \ --experts=256 \ @@ -75,10 +76,10 @@ torchrun \ status=$? set -e -printf 'ADAI_FAIR_EXIT_STATUS=%s_dimensionless\n' "${status}" +printf 'ADAI_EP_EXIT_STATUS=%s_dimensionless\n' "${status}" if ((status == 0)); then - printf 'ADAI_FAIR_COMPLETE\n' + printf 'ADAI_EP_COMPLETE\n' else - printf 'ADAI_FAIR_FAILED\n' + printf 'ADAI_EP_FAILED\n' fi sleep infinity diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py similarity index 53% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py index 9648d6cbf..b725b3858 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_fair_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py @@ -13,12 +13,33 @@ from pathlib import Path from typing import Any, Iterable -from fair_result_io import load_result_log +from result_io import load_result_log ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") WORLD_SIZES = (16, 32) DTYPES = ("fp8", "bf16") +PROFILES = ("decode", "prefill") +PROFILE_CONFIG = { + "decode": { + "tokens_per_rank": 128, + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "timing_boundary": ( + "BF16 input ready through dispatch and combine completion; " + "slowest rank CUDA elapsed time" + ), + }, + "prefill": { + "tokens_per_rank": 4_096, + "api_mode": "normal", + "primary_metric": ("effective logical gigabytes per second per rank"), + "timing_boundary": ( + "BF16 input and route ready through required layout, dispatch, and " + "combine completion; slowest rank CUDA elapsed time" + ), + }, +} ARM_LABELS = { "uccl": "UCCL", "deepep-v1-nvshmem": "DeepEP V1 NVSHMEM", @@ -29,10 +50,6 @@ EXPECTED_STARTS = 3 EXPECTED_WARMUPS = 20 EXPECTED_ITERATIONS = 100 -TIMING_BOUNDARY = ( - "BF16 input ready through dispatch and combine completion; " - "slowest rank CUDA elapsed time" -) LOGICAL_PAYLOAD_DEFINITION = ( "per valid expert assignment: dispatch tensor plus FP8 scales when selected " "plus BF16 combine tensor; backend metadata excluded" @@ -62,11 +79,12 @@ def bootstrap_median_ci( def load_results(root: Path) -> list[dict[str, Any]]: - results: dict[tuple[str, int, int, str], dict[str, Any]] = {} - sources: dict[tuple[str, int, int, str], Path] = {} + results: dict[tuple[str, str, int, int, str], dict[str, Any]] = {} + sources: dict[tuple[str, str, int, int, str], Path] = {} for path in sorted(root.rglob("*.log")): for result in load_result_log(path): key = ( + result["workload_profile"], result["arm"], result["world_size_ranks"], result["run_index_dimensionless"], @@ -85,7 +103,8 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: if starts != EXPECTED_STARTS: raise ValueError(f"scored matrix requires exactly {EXPECTED_STARTS} starts") expected = { - (arm, world, run, dtype) + (profile, arm, world, run, dtype) + for profile in PROFILES for arm in ARMS for world in WORLD_SIZES for run in range(1, starts + 1) @@ -93,6 +112,7 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: } observed = { ( + result["workload_profile"], result["arm"], result["world_size_ranks"], result["run_index_dimensionless"], @@ -113,19 +133,33 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: raise ValueError( f"result matrix has {len(measured)} scored records; expected {len(expected)}" ) - expected_shape = { + common_shape = { "benchmark": "common-boundary-dispatch-combine", - "tokens_per_rank": 128, "hidden_dimensions": 7168, "experts": 256, "top_k_dimensionless": 8, "gpus_per_node": 8, "warmup_iterations": EXPECTED_WARMUPS, "measured_iterations": EXPECTED_ITERATIONS, - "timing_boundary": TIMING_BOUNDARY, "logical_payload_definition": LOGICAL_PAYLOAD_DEFINITION, } for result in measured: + profile = result.get("workload_profile") + if profile not in PROFILE_CONFIG: + raise ValueError(f"unexpected workload profile: {profile}") + profile_config = PROFILE_CONFIG[profile] + expected_shape = { + **common_shape, + "tokens_per_rank": profile_config["tokens_per_rank"], + "timing_boundary": profile_config["timing_boundary"], + "primary_metric": profile_config["primary_metric"], + "layout_in_timed_region": profile == "prefill", + "backend_api_mode": ( + "elastic" + if result.get("arm") == "deepep-v2-gin-gda" + else profile_config["api_mode"] + ), + } for field, expected_value in expected_shape.items(): if result.get(field) != expected_value: raise ValueError( @@ -134,7 +168,8 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: world_size = result["world_size_ranks"] if result.get("nodes") != world_size // 8: raise ValueError(f"invalid node count for EP{world_size}: {result}") - if result.get("global_input_tokens") != 128 * world_size: + tokens_per_rank = profile_config["tokens_per_rank"] + if result.get("global_input_tokens") != tokens_per_rank * world_size: raise ValueError(f"invalid global input token count: {result}") if result["correctness"]["status"] != "PASS": raise ValueError(f"correctness did not pass: {result}") @@ -166,7 +201,7 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: dispatch_bytes = 7_168 * 2 if result["dispatch_dtype"] == "fp8": dispatch_bytes = 7_168 + math.ceil(7_168 / 128) * 4 - expected_logical_bytes = 128 * 8 * (dispatch_bytes + 7_168 * 2) + expected_logical_bytes = tokens_per_rank * 8 * (dispatch_bytes + 7_168 * 2) logical_bytes = result.get("avg_logical_payload_bytes_per_rank") scaleout_bytes = result.get("avg_scaleout_logical_payload_bytes_per_rank") if logical_bytes != expected_logical_bytes: @@ -215,7 +250,6 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: common_fields = ( "warmup_iterations", "measured_iterations", - "timing_boundary", "logical_payload_definition", ) for field in common_fields: @@ -247,31 +281,36 @@ def validate(results: list[dict[str, Any]], starts: int) -> None: f"{arm} did not use one immutable image: {image_references}" ) - for world in WORLD_SIZES: - same_world = [ - result for result in measured if result["world_size_ranks"] == world - ] - route_hashes = {result["route_hash_sha256"] for result in same_world} - input_hashes = {result["input_hash_sha256"] for result in same_world} - if len(route_hashes) != 1 or len(input_hashes) != 1: - raise ValueError( - f"EP{world} did not replay one route/input: " - f"routes={route_hashes}, inputs={input_hashes}" - ) - for dtype in DTYPES: - same_cell = [ - result for result in same_world if result["dispatch_dtype"] == dtype + for profile in PROFILES: + for world in WORLD_SIZES: + same_world = [ + result + for result in measured + if result["workload_profile"] == profile + and result["world_size_ranks"] == world ] - for field in ( - "avg_logical_payload_bytes_per_rank", - "avg_scaleout_logical_payload_bytes_per_rank", - "global_valid_expert_selections", - ): - values = {result[field] for result in same_cell} - if len(values) != 1: - raise ValueError( - f"EP{world} {dtype} disagrees on {field}: {sorted(values)}" - ) + route_hashes = {result["route_hash_sha256"] for result in same_world} + input_hashes = {result["input_hash_sha256"] for result in same_world} + if len(route_hashes) != 1 or len(input_hashes) != 1: + raise ValueError( + f"{profile} EP{world} did not replay one route/input: " + f"routes={route_hashes}, inputs={input_hashes}" + ) + for dtype in DTYPES: + same_cell = [ + result for result in same_world if result["dispatch_dtype"] == dtype + ] + for field in ( + "avg_logical_payload_bytes_per_rank", + "avg_scaleout_logical_payload_bytes_per_rank", + "global_valid_expert_selections", + ): + values = {result[field] for result in same_cell} + if len(values) != 1: + raise ValueError( + f"{profile} EP{world} {dtype} disagrees on {field}: " + f"{sorted(values)}" + ) def validate_provenance( @@ -291,7 +330,14 @@ def validate_provenance( f"provenance images do not match scored results: {provenance.get('images')}" ) expected_comparison = { - "tokens_per_rank": 128, + "profiles": { + profile: { + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "api_mode": PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + } + for profile in PROFILES + }, "hidden_dimensions": 7_168, "experts": 256, "top_k_dimensionless": 8, @@ -309,7 +355,15 @@ def validate_provenance( raise ValueError(f"provenance is missing {field}") -def arm_summary(results: Iterable[dict[str, Any]], seed: int) -> dict[str, Any]: +def primary_metric_value(result: dict[str, Any]) -> float: + if result["workload_profile"] == "decode": + return result["latency_ms"]["median"] + return result["effective_logical_gigabytes_per_second_per_rank"] + + +def arm_summary( + results: Iterable[dict[str, Any]], seed: int, profile: str +) -> dict[str, Any]: ordered = sorted(results, key=lambda item: item["run_index_dimensionless"]) latencies = [item["latency_ms"]["median"] for item in ordered] token_rates = [item["aggregate_input_tokens_per_second"] for item in ordered] @@ -320,9 +374,10 @@ def arm_summary(results: Iterable[dict[str, Any]], seed: int) -> dict[str, Any]: item["effective_scaleout_logical_gigabytes_per_second_per_rank"] for item in ordered ] - mean_latency = statistics.fmean(latencies) - stdev_latency = statistics.stdev(latencies) if len(latencies) > 1 else 0.0 - ci_low, ci_high = bootstrap_median_ci(latencies, seed) + primary_values = [primary_metric_value(item) for item in ordered] + primary_mean = statistics.fmean(primary_values) + primary_stdev = statistics.stdev(primary_values) if len(primary_values) > 1 else 0.0 + ci_low, ci_high = bootstrap_median_ci(primary_values, seed) return { "starts": len(ordered), "run_indices_dimensionless": [ @@ -330,9 +385,12 @@ def arm_summary(results: Iterable[dict[str, Any]], seed: int) -> dict[str, Any]: ], "per_start_median_latency_ms": latencies, "median_latency_ms": statistics.median(latencies), - "bootstrap_95_percent_ci_latency_ms": [ci_low, ci_high], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "per_start_primary_values": primary_values, + "median_primary_value": statistics.median(primary_values), + "bootstrap_95_percent_ci_primary_value": [ci_low, ci_high], "run_to_run_cv_percent": ( - stdev_latency / mean_latency * 100 if mean_latency else 0.0 + primary_stdev / primary_mean * 100 if primary_mean else 0.0 ), "median_aggregate_input_tokens_per_second": statistics.median(token_rates), "median_effective_logical_gigabytes_per_second_per_rank": statistics.median( @@ -350,63 +408,103 @@ def summarize( provenance: dict[str, Any] | None = None, ) -> dict[str, Any]: measured = [result for result in results if result["run_index_dimensionless"] > 0] - by_cell_arm: dict[tuple[int, str, str], list[dict[str, Any]]] = defaultdict(list) + by_cell_arm: dict[tuple[str, int, str, str], list[dict[str, Any]]] = defaultdict( + list + ) for result in measured: by_cell_arm[ - (result["world_size_ranks"], result["dispatch_dtype"], result["arm"]) + ( + result["workload_profile"], + result["world_size_ranks"], + result["dispatch_dtype"], + result["arm"], + ) ].append(result) cells = [] - for world in WORLD_SIZES: - for dtype in DTYPES: - arms = { - arm: arm_summary( - by_cell_arm[(world, dtype, arm)], - seed=20260824 + world + len(dtype) + index, - ) - for index, arm in enumerate(ARMS) - } - comparisons = {} - v2_by_run = { - result["run_index_dimensionless"]: result["latency_ms"]["median"] - for result in by_cell_arm[(world, dtype, "deepep-v2-gin-gda")] - } - for index, baseline in enumerate(("uccl", "deepep-v1-nvshmem")): - baseline_by_run = { - result["run_index_dimensionless"]: result["latency_ms"]["median"] - for result in by_cell_arm[(world, dtype, baseline)] + for profile_index, profile in enumerate(PROFILES): + for world in WORLD_SIZES: + for dtype in DTYPES: + arms = { + arm: arm_summary( + by_cell_arm[(profile, world, dtype, arm)], + seed=( + 20260824 + + profile_index * 1_000 + + world + + len(dtype) + + index + ), + profile=profile, + ) + for index, arm in enumerate(ARMS) } - paired = [ - (baseline_by_run[run] - v2_by_run[run]) / baseline_by_run[run] * 100 - for run in range(1, starts + 1) + comparisons = {} + v2_by_run = { + result["run_index_dimensionless"]: primary_metric_value(result) + for result in by_cell_arm[ + (profile, world, dtype, "deepep-v2-gin-gda") + ] + } + for index, baseline in enumerate(("uccl", "deepep-v1-nvshmem")): + baseline_by_run = { + result["run_index_dimensionless"]: primary_metric_value(result) + for result in by_cell_arm[(profile, world, dtype, baseline)] + } + if profile == "decode": + paired = [ + (baseline_by_run[run] - v2_by_run[run]) + / baseline_by_run[run] + * 100 + for run in range(1, starts + 1) + ] + else: + paired = [ + (v2_by_run[run] - baseline_by_run[run]) + / baseline_by_run[run] + * 100 + for run in range(1, starts + 1) + ] + ci_low, ci_high = bootstrap_median_ci( + paired, + 20260900 + profile_index * 1_000 + world + index, + ) + stable = ( + arms[baseline]["run_to_run_cv_percent"] + <= MAX_RUN_TO_RUN_CV_PERCENT + and arms["deepep-v2-gin-gda"]["run_to_run_cv_percent"] + <= MAX_RUN_TO_RUN_CV_PERCENT + ) + comparisons[f"deepep-v2-gin-gda_vs_{baseline}"] = { + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "paired_improvement_percent_per_start": paired, + "median_paired_improvement_percent": statistics.median(paired), + "bootstrap_95_percent_ci_improvement_percent": [ + ci_low, + ci_high, + ], + "direction_supported": stable and (ci_low > 0 or ci_high < 0), + } + same_cell = [ + result + for result in measured + if result["workload_profile"] == profile + and result["world_size_ranks"] == world + and result["dispatch_dtype"] == dtype ] - ci_low, ci_high = bootstrap_median_ci(paired, 20260900 + world + index) - stable = ( - arms[baseline]["run_to_run_cv_percent"] <= MAX_RUN_TO_RUN_CV_PERCENT - and arms["deepep-v2-gin-gda"]["run_to_run_cv_percent"] - <= MAX_RUN_TO_RUN_CV_PERCENT + cells.append( + { + "workload_profile": profile, + "world_size_ranks": world, + "dispatch_dtype": dtype, + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "route_hash_sha256": same_cell[0]["route_hash_sha256"], + "input_hash_sha256": same_cell[0]["input_hash_sha256"], + "arms": arms, + "comparisons": comparisons, + } ) - comparisons[f"deepep-v2-gin-gda_vs_{baseline}"] = { - "paired_latency_reduction_percent_per_start": paired, - "median_paired_latency_reduction_percent": statistics.median( - paired - ), - "bootstrap_95_percent_ci_reduction_percent": [ci_low, ci_high], - "direction_supported": stable and (ci_low > 0 or ci_high < 0), - } - same_world = [ - result for result in measured if result["world_size_ranks"] == world - ] - cells.append( - { - "world_size_ranks": world, - "dispatch_dtype": dtype, - "route_hash_sha256": same_world[0]["route_hash_sha256"], - "input_hash_sha256": same_world[0]["input_hash_sha256"], - "arms": arms, - "comparisons": comparisons, - } - ) images = { arm: next( result["runtime"]["image_reference"] @@ -420,19 +518,31 @@ def summarize( for field in ("gpu", "torch_version", "cuda_version", "nccl_version") } summary = { - "schema_version_dimensionless": 1, + "schema_version_dimensionless": 2, "status": "PASS", "scored_result_records_dimensionless": len(measured), "independent_starts_per_cell": starts, "bootstrap_samples_dimensionless": BOOTSTRAP_SAMPLES, "maximum_run_to_run_cv_percent_for_direction_support": MAX_RUN_TO_RUN_CV_PERCENT, - "timing_boundary": measured[0]["timing_boundary"], + "timing_boundaries": { + profile: PROFILE_CONFIG[profile]["timing_boundary"] for profile in PROFILES + }, "logical_payload_definition": measured[0]["logical_payload_definition"], - "comparison_scope": "synthetic decode dispatch-plus-combine communication workload; not end-to-end training or serving", + "comparison_scope": ( + "synthetic decode-like and prefill-like dispatch-plus-combine " + "communication workloads; not end-to-end training or serving" + ), "configuration": { "world_sizes_ranks": list(WORLD_SIZES), "dispatch_dtypes": list(DTYPES), - "tokens_per_rank": 128, + "profiles": { + profile: { + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "api_mode": PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + } + for profile in PROFILES + }, "hidden_dimensions": 7_168, "experts_dimensionless": 256, "top_k_dimensionless": 8, @@ -452,41 +562,73 @@ def markdown(summary: dict[str, Any]) -> str: lines = [ "# Common-Boundary EP Results", "", - f"Each cell has {summary['independent_starts_per_cell']} independent process starts. Latency is the slowest-rank CUDA elapsed time from BF16 input readiness through dispatch and combine completion. Values are medians across independent starts. This is a synthetic communication workload, not an end-to-end training or serving result.", - "", - "| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) |", - "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", + f"Each cell has {summary['independent_starts_per_cell']} independent process starts. Values are medians across independent starts. Decode-like results use slowest-rank latency as the primary metric; prefill-like results use common logical throughput. These are synthetic communication workloads, not end-to-end training or serving results.", ] - for cell in summary["cells"]: - for arm in ARMS: - value = cell["arms"][arm] - ci = value["bootstrap_95_percent_ci_latency_ms"] - lines.append( - f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[arm]} | " - f"{value['median_latency_ms']:.4f} ms | [{ci[0]:.4f}, {ci[1]:.4f}] ms | " - f"{value['run_to_run_cv_percent']:.2f}% | " - f"{value['median_aggregate_input_tokens_per_second']:,.2f} tokens/s | " - f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " - f"{value['median_effective_scaleout_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank |" + for profile in PROFILES: + title = ( + "Decode-like latency" if profile == "decode" else "Prefill-like throughput" + ) + tokens = PROFILE_CONFIG[profile]["tokens_per_rank"] + lines.extend(["", f"## {title}, {tokens:,} tokens/rank", ""]) + if profile == "decode": + lines.extend( + [ + "| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) |", + "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", + ] ) + else: + lines.extend( + [ + "| EP size | Dispatch dtype | Backend | Logical throughput (GB/s/rank) | 95% bootstrap CI (GB/s/rank) | Run-to-run CV (%) | Latency (ms) | Input throughput (tokens/s) | Scale-out logical throughput (GB/s/rank) |", + "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", + ] + ) + for cell in summary["cells"]: + if cell["workload_profile"] != profile: + continue + for arm in ARMS: + value = cell["arms"][arm] + ci = value["bootstrap_95_percent_ci_primary_value"] + if profile == "decode": + row = ( + f"{value['median_latency_ms']:.4f} ms | " + f"[{ci[0]:.4f}, {ci[1]:.4f}] ms" + ) + else: + row = ( + f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " + f"[{ci[0]:.2f}, {ci[1]:.2f}] GB/s/rank" + ) + lines.append( + f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[arm]} | " + f"{row} | {value['run_to_run_cv_percent']:.2f}% | " + + ( + f"{value['median_aggregate_input_tokens_per_second']:,.2f} tokens/s | " + f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " + if profile == "decode" + else f"{value['median_latency_ms']:.4f} ms | {value['median_aggregate_input_tokens_per_second']:,.2f} tokens/s | " + ) + + f"{value['median_effective_scaleout_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank |" + ) lines.extend( [ "", - "## Paired DeepEP V2 latency deltas", + "## Paired DeepEP V2 improvements", "", - "Positive values mean DeepEP V2 had lower latency. A direction is supported for this workload only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV.", + "Positive values mean DeepEP V2 had lower latency for Decode-like cells or higher logical throughput for Prefill-like cells. A direction is supported only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV.", "", - "| EP size | Dispatch dtype | Baseline | Median reduction (%) | 95% bootstrap CI (%) | Direction supported |", - "|---:|:---:|:---|---:|:---:|:---:|", + "| Profile | EP size | Dispatch dtype | Baseline | Primary metric | Median improvement (%) | 95% bootstrap CI (%) | Direction supported |", + "|:---|---:|:---:|:---|:---|---:|:---:|:---:|", ] ) for cell in summary["cells"]: for baseline in ("uccl", "deepep-v1-nvshmem"): comparison = cell["comparisons"][f"deepep-v2-gin-gda_vs_{baseline}"] - ci = comparison["bootstrap_95_percent_ci_reduction_percent"] + ci = comparison["bootstrap_95_percent_ci_improvement_percent"] lines.append( - f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[baseline]} | " - f"{comparison['median_paired_latency_reduction_percent']:.2f}% | " + f"| {cell['workload_profile']} | {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[baseline]} | {comparison['primary_metric']} | " + f"{comparison['median_paired_improvement_percent']:.2f}% | " f"[{ci[0]:.2f}, {ci[1]:.2f}]% | " f"{'yes' if comparison['direction_supported'] else 'no'} |" ) @@ -510,7 +652,7 @@ def main() -> None: args.json.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n") args.markdown.write_text(markdown(summary)) print( - f"PASS fair EP matrix: {len(summary['cells'])} cells, " + f"PASS EP matrix: {len(summary['cells'])} cells, " f"{args.starts} independent starts per arm/cell" ) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py similarity index 64% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py index 74c0fdea2..b1fc9c522 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_fair_ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py @@ -12,8 +12,8 @@ MODULE = None if torch is not None: - module_path = Path(__file__).with_name("fair_ep_benchmark.py") - spec = importlib.util.spec_from_file_location("fair_ep_benchmark", module_path) + module_path = Path(__file__).with_name("ep_benchmark.py") + spec = importlib.util.spec_from_file_location("ep_benchmark", module_path) assert spec and spec.loader MODULE = importlib.util.module_from_spec(spec) sys.modules[spec.name] = MODULE @@ -21,7 +21,7 @@ @unittest.skipIf(torch is None, "PyTorch is required for benchmark helper tests") -class FairEpBenchmarkTest(unittest.TestCase): +class EpBenchmarkTest(unittest.TestCase): def test_route_is_balanced_and_unique(self): routes = [ MODULE.make_route(rank, 128, 256, 8, 20260824, torch.device("cpu")) @@ -50,6 +50,25 @@ def test_common_payload_formula(self): self.assertLess(bf16_remote, bf16_all) self.assertLess(fp8_remote, fp8_all) + def test_profile_shapes_are_fixed(self): + self.assertEqual(MODULE.WORKLOAD_PROFILES["decode"].tokens_per_rank, 128) + self.assertEqual(MODULE.WORKLOAD_PROFILES["decode"].api_mode, "low-latency") + self.assertEqual(MODULE.WORKLOAD_PROFILES["prefill"].tokens_per_rank, 4096) + self.assertEqual(MODULE.WORKLOAD_PROFILES["prefill"].api_mode, "normal") + + def test_prefill_payload_scales_with_tokens(self): + decode_route = MODULE.make_route(0, 128, 256, 8, 20260824, torch.device("cpu")) + prefill_route = MODULE.make_route( + 0, 4096, 256, 8, 20260824, torch.device("cpu") + ) + decode_bytes, _, _ = MODULE.logical_payload_bytes_per_rank( + decode_route, 0, 16, 8, 7168, "bf16", 256 + ) + prefill_bytes, _, _ = MODULE.logical_payload_bytes_per_rank( + prefill_route, 0, 16, 8, 7168, "bf16", 256 + ) + self.assertEqual(prefill_bytes, decode_bytes * 32) + def test_percentile_interpolates(self): self.assertEqual(MODULE.percentile([1.0, 2.0, 3.0], 0.5), 2.0) self.assertAlmostEqual(MODULE.percentile([1.0, 2.0], 0.95), 1.95) @@ -71,6 +90,7 @@ def test_deepep_v2_build_lib_requires_one_extension_package(self): def test_received_fp8_accepts_noncontiguous_scales(self): adapter = object.__new__(MODULE.BackendAdapter) adapter.arm = "uccl" + adapter.profile = MODULE.WORKLOAD_PROFILES["decode"] adapter.hidden = 256 observed = {} @@ -90,6 +110,23 @@ def cast_back(fp8, scales): self.assertEqual(observed["scales_shape"], (6, 2)) self.assertEqual(tuple(received.shape), (2, 3, 256)) + def test_identity_expert_output_applies_local_gates(self): + adapter = object.__new__(MODULE.BackendAdapter) + adapter.arm = "uccl" + adapter.hidden = 2 + adapter.profile = MODULE.WORKLOAD_PROFILES["prefill"] + state = MODULE.DispatchState( + recv_x=torch.tensor([[2.0, 4.0]], dtype=torch.bfloat16), + recv_topk_idx=torch.tensor([[0, -1, 1]]), + recv_topk_weights=torch.tensor([[0.25, 0.5, 0.125]]), + handle=None, + ) + output = adapter.identity_expert_output(state, "bf16") + torch.testing.assert_close( + output, + torch.tensor([[0.75, 1.5]], dtype=torch.bfloat16), + ) + if __name__ == "__main__": unittest.main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py similarity index 70% rename from micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py rename to micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py index 93d5bbb48..dad3924c5 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_fair_results.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py @@ -5,33 +5,44 @@ import unittest -module_path = Path(__file__).with_name("summarize_fair_results.py") -spec = importlib.util.spec_from_file_location("summarize_fair_results", module_path) +module_path = Path(__file__).with_name("summarize_results.py") +sys.path.insert(0, str(module_path.parent)) +spec = importlib.util.spec_from_file_location("summarize_results", module_path) assert spec and spec.loader summary_module = importlib.util.module_from_spec(spec) sys.modules[spec.name] = summary_module spec.loader.exec_module(summary_module) -def fake_result(arm, world_size, run_index, dtype, latency_ms): +def fake_result(profile, arm, world_size, run_index, dtype, latency_ms): digest = { "uccl": "a", "deepep-v1-nvshmem": "b", "deepep-v2-gin-gda": "c", }[arm] - logical_bytes = 22_249_472 if dtype == "fp8" else 29_360_128 + tokens_per_rank = summary_module.PROFILE_CONFIG[profile]["tokens_per_rank"] + decode_logical_bytes = 22_249_472 if dtype == "fp8" else 29_360_128 + logical_bytes = decode_logical_bytes * tokens_per_rank // 128 scaleout_bytes = logical_bytes * (world_size - 8) // world_size tolerance = 9e-4 if dtype == "fp8" else 1e-5 - global_input_tokens = 128 * world_size + global_input_tokens = tokens_per_rank * world_size result = { "benchmark": "common-boundary-dispatch-combine", + "workload_profile": profile, + "backend_api_mode": ( + "elastic" + if arm == "deepep-v2-gin-gda" + else summary_module.PROFILE_CONFIG[profile]["api_mode"] + ), + "primary_metric": summary_module.PROFILE_CONFIG[profile]["primary_metric"], + "layout_in_timed_region": profile == "prefill", "arm": arm, "world_size_ranks": world_size, "nodes": world_size // 8, "gpus_per_node": 8, "run_index_dimensionless": run_index, "dispatch_dtype": dtype, - "tokens_per_rank": 128, + "tokens_per_rank": tokens_per_rank, "global_input_tokens": global_input_tokens, "hidden_dimensions": 7168, "experts": 256, @@ -63,13 +74,13 @@ def fake_result(arm, world_size, run_index, dtype, latency_ms): "effective_scaleout_logical_gigabytes_per_second_per_rank": scaleout_bytes / (latency_ms / 1e3) / 1e9, - "timing_boundary": summary_module.TIMING_BOUNDARY, + "timing_boundary": summary_module.PROFILE_CONFIG[profile]["timing_boundary"], "logical_payload_definition": summary_module.LOGICAL_PAYLOAD_DEFINITION, } return result -class SummarizeFairResultsTest(unittest.TestCase): +class SummarizeResultsTest(unittest.TestCase): def setUp(self): arm_latency = { "uccl": 1.0, @@ -78,12 +89,14 @@ def setUp(self): } self.results = [ fake_result( + profile, arm, world, run, dtype, arm_latency[arm] * (1 + (run - 2) * 0.01), ) + for profile in summary_module.PROFILES for arm in summary_module.ARMS for world in summary_module.WORLD_SIZES for run in range(1, 4) @@ -94,12 +107,22 @@ def test_valid_matrix_and_paired_delta(self): summary_module.validate(self.results, 3) summary = summary_module.summarize(self.results, 3) self.assertEqual(summary["status"], "PASS") - self.assertEqual(len(summary["cells"]), 4) + self.assertEqual(len(summary["cells"]), 8) comparison = summary["cells"][0]["comparisons"]["deepep-v2-gin-gda_vs_uccl"] + self.assertAlmostEqual(comparison["median_paired_improvement_percent"], 20.0) + self.assertTrue(comparison["direction_supported"]) + + prefill = next( + cell + for cell in summary["cells"] + if cell["workload_profile"] == "prefill" + and cell["world_size_ranks"] == 16 + and cell["dispatch_dtype"] == "fp8" + ) + prefill_comparison = prefill["comparisons"]["deepep-v2-gin-gda_vs_uccl"] self.assertAlmostEqual( - comparison["median_paired_latency_reduction_percent"], 20.0 + prefill_comparison["median_paired_improvement_percent"], 25.0 ) - self.assertTrue(comparison["direction_supported"]) def test_missing_start_is_rejected(self): with self.assertRaises(ValueError): @@ -141,7 +164,18 @@ def test_provenance_image_mismatch_is_rejected(self): for arm in summary_module.ARMS }, "comparison": { - "tokens_per_rank": 128, + "profiles": { + profile: { + "tokens_per_rank": summary_module.PROFILE_CONFIG[profile][ + "tokens_per_rank" + ], + "api_mode": summary_module.PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": summary_module.PROFILE_CONFIG[profile][ + "primary_metric" + ], + } + for profile in summary_module.PROFILES + }, "hidden_dimensions": 7168, "experts": 256, "top_k_dimensionless": 8, @@ -155,11 +189,11 @@ def test_provenance_image_mismatch_is_rejected(self): summary_module.validate_provenance(provenance, self.results, 3) def test_load_results_accepts_native_diagnostic_after_json(self): - result = fake_result("deepep-v2-gin-gda", 32, 1, "fp8", 0.9) + result = fake_result("decode", "deepep-v2-gin-gda", 32, 1, "fp8", 0.9) with tempfile.TemporaryDirectory() as directory: path = Path(directory) / "rank-zero.log" path.write_text( - "ADAI_FAIR_RESULT " + "ADAI_EP_RESULT " + summary_module.json.dumps(result) + "Elastic buffer uses 3 channels per SM\n" ) From 97c93ee33535ffb41b7635befc4a97617ab8acef Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Tue, 25 Aug 2026 02:38:40 +0000 Subject: [PATCH 17/19] fix(ep): make UCCL prefill workers exit cleanly --- .../ep-backend-comparison/ep_benchmark.py | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py index 4b96c3a27..315251c23 100755 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py @@ -21,6 +21,7 @@ import os import statistics import sys +import traceback from dataclasses import dataclass from pathlib import Path from typing import Any @@ -726,6 +727,7 @@ def main() -> None: args.experts, args.top_k, ) + completed = False try: for dispatch_dtype in args.dispatch_dtypes: result = run_dtype( @@ -745,10 +747,39 @@ def main() -> None: ) if rank == 0: print(RESULT_PREFIX + json.dumps(result, sort_keys=True), flush=True) + completed = True + except BaseException: + if args.arm == "uccl" and args.profile == "prefill": + # The pinned UCCL high-throughput proxy teardown can strand the + # interpreter after an error. Preserve the traceback and let + # torchrun observe an unambiguous worker failure instead. + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(1) + raise finally: - adapter.destroy() - dist.barrier(group=group) - dist.destroy_process_group() + if args.arm == "uccl" and args.profile == "prefill": + if completed: + # UCCL's pinned normal-mode proxy cleanup destroys the CUDA + # context while PyTorch still owns CUDA tensors, which hangs + # worker shutdown. All communication has completed here and + # this barrier keeps every rank alive through result output. + # Process exit then releases the CUDA context, QPs, and file + # descriptors outside the measured benchmark boundary. + if rank == 0: + print( + "ADAI_EP_PROCESS_LIFETIME_CLEANUP arm=uccl profile=prefill", + flush=True, + ) + dist.barrier(group=group) + sys.stdout.flush() + sys.stderr.flush() + os._exit(0) + else: + adapter.destroy() + dist.barrier(group=group) + dist.destroy_process_group() if __name__ == "__main__": From e3e58311c568c8c0e81af7c64d7830e96c8afe42 Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Tue, 25 Aug 2026 06:21:12 +0000 Subject: [PATCH 18/19] docs(ep): remove unrelated model disclaimer --- .../expert-parallelism/ep-backend-comparison/RESULTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index 0f3f85646..dd92ff123 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -4,7 +4,7 @@ Status: `PASS` The previous Decode-only result has been discarded. It must not be combined with this two-profile matrix because Prefill-like uses the normal high-throughput API and a different timing boundary. -This is a synthetic expert-parallel communication microbenchmark. It does not target Kimi K2 and does not model a Kimi K2 training replica. +This is a synthetic expert-parallel communication microbenchmark. ## Campaign configuration From b7a5e107eca156bba46c0e350f38a0c65827c42c Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Tue, 25 Aug 2026 06:38:16 +0000 Subject: [PATCH 19/19] docs(ep): add backend box plots --- .../ep-backend-comparison/README.md | 14 +- .../ep-backend-comparison/RESULTS.md | 4 + .../ep-backend-comparison/plot_results.py | 243 ++++++++++++++++++ .../b200-ap-south-1-2026-08-25-boxplots.png | Bin 0 -> 131382 bytes 4 files changed, 259 insertions(+), 2 deletions(-) create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py create mode 100644 micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index 93af4ed9c..a5e0a27e0 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -10,7 +10,7 @@ This directory compares 3 expert-parallel dispatch/combine backends through comm Each workload uses one external CUDA timing boundary and one logical payload definition. Backend-native latency and bandwidth fields remain diagnostics because they do not share one timing boundary or byte numerator. -The B200 report is in [RESULTS.md](RESULTS.md). EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes. +The B200 report and backend box plots are in [RESULTS.md](RESULTS.md). EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes. ## Workload profiles @@ -61,6 +61,7 @@ These are logical efficiency metrics, not observed wire bandwidth. | [`result_io.py`](result_io.py) | Robust result-marker parsing from interleaved native output | | [`extract_results.py`](extract_results.py) | Canonical JSONL extraction from a rank-zero log | | [`summarize_results.py`](summarize_results.py) | Matrix validation, per-start aggregation, paired deltas, and bootstrap intervals | +| [`plot_results.py`](plot_results.py) | Box plots of the independent-start primary values for every backend arm | | [`RESULTS.md`](RESULTS.md) | Human-readable result, provenance, and scope limits | ## Requirements @@ -175,12 +176,21 @@ python3 extract_results.py rank-zero.log results.jsonl The summarizer rejects an incomplete matrix, correctness failure, mutable image tag, route/input mismatch, runtime-stack mismatch, or disagreement in common logical payload accounting. +Regenerate the box plots from the committed machine-readable summary: + +```bash +python3 plot_results.py results/b200-ap-south-1-2026-08-25.json \ + --output=results/b200-ap-south-1-2026-08-25-boxplots.png +``` + +Plot generation requires Matplotlib. Each box uses the 3 independent process-start medians for one backend and workload cell. The plot also shows every underlying point. + ## Local validation ```bash python3 -m pytest -q test_ep_benchmark.py test_summarize_results.py python3 -m py_compile \ - ep_benchmark.py result_io.py extract_results.py summarize_results.py + ep_benchmark.py result_io.py extract_results.py summarize_results.py plot_results.py bash -n run_ep_comparison.sh run_ep_rank.sh shellcheck run_ep_comparison.sh run_ep_rank.sh ``` diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index dd92ff123..3e9ae3efc 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -30,6 +30,10 @@ Decode-like reports slowest-rank latency. Prefill-like reports logical GB/s/rank Each cell has 3 independent process starts. Values are medians across independent starts. Decode-like results use slowest-rank latency as the primary metric; prefill-like results use common logical throughput. These are synthetic communication workloads, not end-to-end training or serving results. +![Box plots comparing the three backend arms for every Decode-like and Prefill-like workload cell](results/b200-ap-south-1-2026-08-25-boxplots.png) + +Each box contains the 3 independent process-start medians for one backend arm and workload cell. The box spans Q1 to Q3, the center line is the median, the whiskers are the minimum and maximum, and the markers show all 3 values. The individual markers are the primary evidence because each box contains only 3 independent starts. + ## Decode-like latency, 128 tokens/rank | EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) | diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py new file mode 100644 index 000000000..1ff34b352 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""Render backend box plots from an EP comparison summary.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path +from typing import Any + +import matplotlib + +matplotlib.use("Agg") + +from matplotlib import pyplot as plt +from matplotlib.patches import Patch + + +ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") +ARM_STYLES = { + "uccl": { + "color": "#0072B2", + "hatch": "///", + "label": "UCCL", + "marker": "o", + }, + "deepep-v1-nvshmem": { + "color": "#E69F00", + "hatch": "\\\\", + "label": "DeepEP V1 NVSHMEM", + "marker": "s", + }, + "deepep-v2-gin-gda": { + "color": "#009E73", + "hatch": "...", + "label": "DeepEP V2 NCCL GIN", + "marker": "^", + }, +} +CELL_ORDER = ((16, "fp8"), (16, "bf16"), (32, "fp8"), (32, "bf16")) +PROFILE_CONFIG = { + "decode": { + "title": "Decode-like: slowest-rank latency (lower is better)", + "ylabel": "Latency (ms)", + }, + "prefill": { + "title": "Prefill-like: common logical throughput (higher is better)", + "ylabel": "Logical throughput (GB/s/rank)", + }, +} + + +def load_plot_data( + path: Path, +) -> tuple[int, dict[tuple[str, int, str], dict[str, Any]]]: + summary = json.loads(path.read_text()) + if summary.get("status") != "PASS": + raise ValueError("the input summary does not have PASS status") + + starts = summary.get("independent_starts_per_cell") + if not isinstance(starts, int) or starts < 1: + raise ValueError("independent_starts_per_cell must be a positive integer") + + cells: dict[tuple[str, int, str], dict[str, Any]] = {} + for cell in summary.get("cells", []): + key = ( + cell.get("workload_profile"), + cell.get("world_size_ranks"), + cell.get("dispatch_dtype"), + ) + if key in cells: + raise ValueError(f"duplicate workload cell: {key}") + cells[key] = cell + + expected_keys = { + (profile, world_size, dtype) + for profile in PROFILE_CONFIG + for world_size, dtype in CELL_ORDER + } + if cells.keys() != expected_keys: + missing = sorted(expected_keys - cells.keys()) + extra = sorted(cells.keys() - expected_keys) + raise ValueError(f"workload cell mismatch; missing={missing}, extra={extra}") + + for key, cell in cells.items(): + arms = cell.get("arms", {}) + if arms.keys() != set(ARMS): + missing = sorted(set(ARMS) - arms.keys()) + extra = sorted(arms.keys() - set(ARMS)) + raise ValueError( + f"backend arm mismatch for {key}; missing={missing}, extra={extra}" + ) + for arm in ARMS: + values = arms[arm].get("per_start_primary_values") + if not isinstance(values, list) or len(values) != starts: + raise ValueError( + f"{key}/{arm} must contain {starts} per-start primary values" + ) + if not all( + isinstance(value, (int, float)) and math.isfinite(value) and value > 0 + for value in values + ): + raise ValueError(f"{key}/{arm} contains an invalid primary value") + return starts, cells + + +def render_box_plots(summary_path: Path, output_path: Path) -> None: + starts, cells = load_plot_data(summary_path) + matplotlib.rcParams.update( + { + "axes.edgecolor": "#333333", + "axes.labelcolor": "#222222", + "font.size": 10, + "savefig.facecolor": "white", + "text.color": "#222222", + "xtick.color": "#333333", + "ytick.color": "#333333", + } + ) + + figure, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True) + group_positions = list(range(1, len(CELL_ORDER) + 1)) + arm_offsets = (-0.24, 0.0, 0.24) + point_offsets = tuple((index - (starts - 1) / 2) * 0.025 for index in range(starts)) + + for axis, profile in zip(axes, PROFILE_CONFIG, strict=True): + all_values: list[float] = [] + for arm, arm_offset in zip(ARMS, arm_offsets, strict=True): + values_by_cell = [ + cells[(profile, world_size, dtype)]["arms"][arm][ + "per_start_primary_values" + ] + for world_size, dtype in CELL_ORDER + ] + all_values.extend(value for values in values_by_cell for value in values) + positions = [position + arm_offset for position in group_positions] + style = ARM_STYLES[arm] + box_plot = axis.boxplot( + values_by_cell, + positions=positions, + widths=0.19, + whis=(0, 100), + showfliers=False, + patch_artist=True, + manage_ticks=False, + boxprops={ + "facecolor": style["color"], + "edgecolor": "#222222", + "hatch": style["hatch"], + "linewidth": 1.0, + "alpha": 0.55, + }, + whiskerprops={"color": "#333333", "linewidth": 1.0}, + capprops={"color": "#333333", "linewidth": 1.0}, + medianprops={"color": "#111111", "linewidth": 1.8}, + ) + for median in box_plot["medians"]: + median.set_zorder(4) + for position, values in zip(positions, values_by_cell, strict=True): + axis.scatter( + [position + offset for offset in point_offsets], + values, + color=style["color"], + edgecolor="#111111", + linewidth=0.6, + marker=style["marker"], + s=29, + zorder=5, + ) + + config = PROFILE_CONFIG[profile] + axis.set_title(config["title"], loc="left", fontweight="bold", pad=10) + axis.set_ylabel(config["ylabel"]) + axis.set_ylim(0, max(all_values) * 1.12) + axis.set_xlim(0.5, len(CELL_ORDER) + 0.5) + axis.grid(axis="y", color="#D9D9D9", linewidth=0.8) + axis.set_axisbelow(True) + axis.spines["top"].set_visible(False) + axis.spines["right"].set_visible(False) + + axes[-1].set_xticks( + group_positions, + [f"{world_size} ranks\n{dtype.upper()}" for world_size, dtype in CELL_ORDER], + ) + axes[-1].set_xlabel("Expert-parallel size and dispatch dtype", labelpad=9) + + legend_handles = [ + Patch( + facecolor=ARM_STYLES[arm]["color"], + edgecolor="#222222", + hatch=ARM_STYLES[arm]["hatch"], + alpha=0.55, + label=ARM_STYLES[arm]["label"], + ) + for arm in ARMS + ] + figure.legend( + handles=legend_handles, + loc="upper center", + bbox_to_anchor=(0.5, 0.945), + frameon=False, + ncol=len(ARMS), + ) + figure.suptitle( + "B200 expert-parallel backend comparison", + fontsize=15, + fontweight="bold", + y=0.99, + ) + figure.text( + 0.5, + 0.012, + ( + f"Each arm has {starts} independent process starts. " + "Box: Q1 to Q3; line: median; whiskers: minimum to maximum; " + "markers: per-start medians." + ), + ha="center", + fontsize=9, + color="#444444", + ) + figure.subplots_adjust(left=0.1, right=0.98, top=0.88, bottom=0.12, hspace=0.34) + + output_path.parent.mkdir(parents=True, exist_ok=True) + figure.savefig( + output_path, + dpi=160, + metadata={"Software": "matplotlib via plot_results.py"}, + ) + plt.close(figure) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("summary", type=Path, help="machine-readable summary JSON") + parser.add_argument("--output", type=Path, required=True, help="output image path") + args = parser.parse_args() + render_box_plots(args.summary, args.output) + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png new file mode 100644 index 0000000000000000000000000000000000000000..91fd47ad96bf52a74d3faceaba0da65d2c6c76ed GIT binary patch literal 131382 zcmeFZbyU`A+b)cwjx#DU76=L$gn)pO0s+uzNNy_bX+bJk0D5Wl6RHC5R zc8`K$E5#qb<8R`;6a?`%g`)^G#>WW)pE-PMCymD@K^S^(vAC}(jA9eJ}_Tw-3E(!m8v_$id5*aFb zC8;B#W#ZE{u2~&DdV(GM-p<^4Drq+vUSK^`-&!vE>d z6f1`*cK+A@i5uOr>H2^F>Gfm|U7T{~ z>)3=;-B)j0ENz;2W_wV;>|?fZvusnJ!w|jgSQ{sm$v@i~-5FgI=tbM68sjgvdLpHs6X z>D#q3SM7L($hKf!L!0$ghn`Xwts_T{JbCtc zSi8us?s#33%ZET0Yiaq`BjRCNu4hH`v{h96rMhC{HWx6TKK;o|T=w$i4-Y!=P!6a6 z=Ptf~znw)@$&+b4BroNgoSeKoKarf*=c>iPz(Bp=_VqEV&6l@ZldF4kOoImN!#Tvo z#l7hGr5n!7c&aY>`uXKJFIl|IJ53#~9Q*Y`V{TGj$L}~QDk_Y>ynQ&@o*o$$9nCoz)F>ch z(Nh@fb4+ryoJw59%F61|3JWu{@$7IT9j}3B1%LQ{8lPeE3fU{B>0({?p1iv4@{Mq?itudgl%J50xPIP1DD#V*f|+{zulcaLIi z@tZfxh=qj(otSgJW}ZcAe*WoH-7=k-p$0)(xib#adO2ntc!mMht(jPH&#YPG!x9>! zB|~FyswS*w7RzD)x_Yscw=d4X{1iDkFP_ViRmH@*R?CHNvd+W zWYs7{iaD+?4XGC2z2i4Qt})M|XLhn^%4BY|m8I*!{rfiKpH8)=YPIWi(datAjEQ0L z!*+>J#yv<>9>12KQMx?J5xsWw=utk)-ZNeY1-aB~@OaPG|CpPr8)L_VI?Q;4JmzPA z=bCp_S5-+pe*BoF>;8iWb-&+JPt{~L9P-?#wX(5k`Sj{mj_aBuGZWKuvohDULMe*H z>4BQ%p-4B``P3^f?7Ib&c#)K+@&sr-*KVGeqeAgkVD_F9an|x=b{&L zzM5y*r*`*lepq<;z_)MFA^fIESPIx50Tqp(Oxsc#@tGFBy@j@>*jZI(5sa>jZ^n&= z8AKh87QcUa)bQYU$|ii8*ZdtTtETyhF6P)RGgv*zJw@11sam*Lm5xk(vX4;_$OU6#&(oh8|>SM4}5 zrc+v4Iz2b1ir3m2b!ms_@^Cb^PPK{kW#5}OZ_c)9S&p+ZjgGQjk zZfRP7;?t`O+o%{?=EmChGca&TN=iC7IVEf6n3%Ms1mosd)z#Z~dT)GjXEdpNO3yOF zp`NDlcn|YFtaI`8#S3ohOYvzsB}vbpAK1^WbB*$#fJ&s8i%Og}#goyNL_d6P(Ab*5>C;!c@+{lQjr{m}si(wAePOc4&)eJk2@Q9e zbO3uId99x^UaFQ=4AqA>*N2PPE`Gn!lYi&JmoHy%drGfvR`gFzrTeaT{Z8{ zR}2*}Z^Ek$c=qfCc3+}#bKGciy!Vs+90so#g6F*d{r3;DOUXmWW1`g0!zLw#(An9U zN3^}jergnjLfCaRzdlSTI#ke-osUoD)TvWew{KtZIeKa5TA}OOvPoBN>h}HY{x^by zTSbl^zl@dE93kptvoddf>Egu%y!en)w;ri3)d|^-`eQ4_HODKAjkYE~dGbW*=FO<$ z;$oEoD}7v~!bDeIp2NOZqARKvu6?Ky+xPtJskFE4>~arK7HF_+(Fv)#P(i^Wx5lVTszr9@^rvsXJ}!FVgzTnGnJ z@O(>NvPxnUx-M-j{;>=x%F;5FDJfT9o%dsrQ_|CmX`IKMnd0_CM4fWu!(dfdVp#k9rDfS#TT-D}SFW7(V69Jj zdU~(Rf;JXBhs<1kLWy;uj!<3ingDnIkZZtbQ>;vgux(PVMUTmR-sszhJA7lva?LWR zQ^NMoM)OJ@y(KRz>xs1!Q{uD`VwS9)9ucPyX|ZRu#Ca(*PBu7-JhgD+^Sw3$NmqyQ?-lKu#N<`^jT?Mx$-PdK0FEg=J~3a6x-V2)-WM!l_V9m ztoGxr9a)C_=m`_OC0$>|@RrDCA{_xOOA}44{47X&J2SJC^XJdkS|qEb zhDZSyRn|^wl{ngZRAyvPEdKabpk}9bLc34}FR}jQZT~jZ*V)nJ^bpF-)2B}xz4xRw zkPP-bC@7Epf6Do(mscwRBxw#W&p(`PRo4xsZ#1aK@|f#&nhHKMEI&3_7wR~X)4ntM zEaxjjEjn9vLE?CTW zg;|NuCZlg8$^>yOV zcfr(yf)+;oAAFX-ms6RSPK~29uPx8<{`q_Cej1v1vc|A0?H}nAAT~2tSCgSl#c(D8 z*lQGpLZG!-ZDoE!8C9A*?kqRI=YEa0I%TdUg}C4w&$v<-7w@L(lxQl5uVp`F5H(Sa zn5{f`>gL5qeX&?*rjy+T!j7{E5pL@aexEHhQDGg|=Gtvhm_tWj88@?(tS^tTyy=_O z_damyy!ggyR)q^XijtrvTD_$DXV3nMc7GM=x|~=aA!_RD)}4QcrCC|i_xbY#Z}Ig{NiN^Me&w_3zl?fS zGd>??v_odk-o5;&M4?ap7(Zo|WPJlr%pKWeXE0%&ey=EXF4MhxJWpuAby!_1*R1h_ z&$0dV^yUlSLj34f`2{Pc_9~otIaRjN+s_=lwz7bo*IRw^yCJZl%HlKmDu!iO+ zFc6G-STSEyZCQbuzLR(Jr|NE_n{1ZL0VSnV*mzs_E1$*NxZGJ_9gp^}zj%n2Hj$8o zl|VLSiCdLV>)L+Eqsn9Xv@rEx_kJCImaY6YEK4_ zWo2IV;r9Le@Ily}CcVWDEN{I3+{==H%F3=&oUiEE949wdwy~z5>oOOhxcrn}B-^Zm zXK7`{6g`Ti(q(xh-dv|lKuAafc?VXMlkGcr!PV4bpz?HG6$XZ=pB>)NkukI7<#P`G5^k&Hx@we&}eBb?K-Kxd-rZSQOE3#>w#HWS&p;AFIgjFrxZbuS`qkSEpL*|`{HNHl9xh$N$n3Kow#aQuEo#&l2zmz9Y;mUH## zW$T)trrFC|!$21SsZ54Hz9f+G>({T@dR2R}xc46tbyUHcQmS~e->Ye=r^xV*=1}(9 z5q5TItgp6(J~zv`glk&NEG(lX3%ywf`lmjEDZPn}XE4F`MKeA@T?wQFIdOK#mk{hSl{Ldm2l<}n?g(F^RU>Y5rE9UUDjv}ptdvBm6u;Ae?+N_ zjn+RhlKA1nWpqC|f{FtXx{*x#B*lzC0P6A0{QHU^`aR7Oh zz$1JX6cltM)o!9QNi!#jbbG%w^>m$7{9K=r_DGn;4jF^G5dGQfg;oRAgdL%*QEp<8 zp}H?BDmpMVWjXh1yn$miSGkObr~d7|msb-b=!EFnY%@WQK#P$#w3|F0KW-%K9GJUCPYX;B zeV;K{d#??iu))-RfQ)v*KIdEePNpfxKgA|a11~$^by!wjzHPiC3x)7R^62RJcnYK2 zsv`O*e+;Ptg?1Am9d>8%EmCKV$Oq|CaA+51OSvWBLr~5Pa>YND10NdLsci>^aRKE8 z1oTjJL@S(qw^w190HThd-Jo}dc7>w=`Nmia(?5QkiV>atc)_GQ|CQt1sQ#W$U77kG zX!*Bnua{nZbyKs@Ch?19JidTVhx$}JA;sLHElH{}gvh;lu+3}U!^5K)5I`kfo<4aL zJ@X~_eudGW-bB~IifcxLx8%9BG}%*R3$lg!^5%GnWnU@X$?InyRpN64@#GRfp3FP5 ztBkq;l}Im{%iQD3XHa{fWO+3CQ3HCQ8tP0`|{GE^RtwV zO?K5E9O~-oVCpiEjRYbq4_X%A*vE94U>a}@1y+R!KIt-`fL-*$G1Q!z2b!!3IcrBy zn=2HhMkE*XTR*>fAXncP<}J2-8O=CfRlhU3BzD6o@eA$CWD!h%y#{4~b5h*w(@DSV_g)7>x7=f%Y5(X;qjUGo@x{=j z8|9B2SASH>b&8 z-KFCBysPd!D8oo0oecJ9uz72|d^k^kO=|Atbi;;-;7LDiAoW)!*(ID~AqIIR$`DG7 zn_{G(mT3p->it1Qr4IQarSli$)>qY;>k9MLT?vS_vjg2u*D2xQ&#jnXN^9T%wsZ@88H{!cWIV77bte3-_4?E zC+n0rCS83M1yxTK?^88hr)M~?{Mxg1aY z&5r{Db~M_}&wsk*t#Qcfd0@e7SExFHEBECFu{UI3Nkk*31tKDdQjx3m~?Q94*u$lB8Z)w0aW1$qjK7l*Uf zTXoUuf2-~3(OzF)f3(4_o|b~DWV*WegSy?~TBe><9M$)u$Byye{(Ls;jV>c2BSeCC zNuSg8md*tgyf!!JY6=SQ)7RpNZhWud)1moJ(4spde`0xV3~02j=??v&Lp))=jZL#p zj5nVt=%G$5E-9IC?C!qqJs+sl_O;I9S-o0Yw!KaZNAh%1-{3#1-RUQ`&P?#jIT;W| zLhkv=Bh71>nF5K-ZS!00?d`KhaEpk6`iGzHj$V{jo`pgCTgn|fnt@t42a6nLh~OCw;1q0@e61i;E$vz&TB?RN zl?>Gg&vve>oq?_sC1jb+GZX|iUiOBXVgd|7ofmn=K`GLCS}M@n)R@mx&w)v+bA;16 zi&IHRNT`F4-^}%f_dML2HV993lkNF;Zjp9FErZpqs%rdWK`|NYRO$FA&xy z)O6HV94$Nr^71^>{9?_4pLv(fMnts1KG7!Rpe&zSmPcU|t(lVx*vowY`a*)!!lY)A zT`KxfJb~swe(DVoqOC9mvXpHqD=WvMy~VKggFKrUXI-~KCf7Z+l@Izk3nX4*_}Ho3 zkLB-{4;PMY*Kv}ATF2MXhFkoJ_Nn&sHWxcs6XStt?e*WA4Hmq;y!eX(L(ul<1k9eB z>TA`YO%qTHjNsXXKADc4-El1Fn`V#H$F%73718S8omIH0NpWtsdHm_OF6m) zg@sAZOVitT?&On(Oxup~kP@N#mhS1(r_1Q6tz~X*5SSW_x&XcVzkZEc=yhr^>t$C> zdS1Kl%j9GlzO8O>w_^eAz9cLWsV?azfGAFREz?ACJ;HTtp@#a+;$a(qd~r@7$Z#^` zGMmd{5V=CRCMrg;6oB{8m~=?OV0tVg_zFSVR3AV6 zt|(kOi}Bxo|1IpcUfj5^Cve@`-o72;jP3lmMwaPoLR~*)++b-ISy^06IR9`v%L>}i ziiT#o)tmcU4J1>-g>9u#WqGCSZ{I%oXV{R9mY#cT&cV95$FtIbCyZ6sa5-kSE}31u zBb&91$4s~`=V1q|p?j#cb0<%mmhkvSuB4_;TbB{Nxnnj_Io{y(C>QLp9n>c>Mh_%n z-6Q}By&qq2h2XOc0jh40x19#~G#;%^vy}!yb2bpv8a`8?hbnXIRV4#&XzA@Q=d+ss z(i)|jzdj=D23Mx=?vJPnXDKPE5MINlQjRF7wUZk#?4FC}K{MdgE$udo0D00prO!S zOz!pXuDYhGOW+OWs6tBME}8Z1$ONoO!c#H@u!JAL6XX8q(I}{KD-p_wbmS7IlB${0 zM0_v1{z=fNIU;SwER43+jL$Y=wYLBB=U$u97PjcmSKvJp4FY>k4+|y6>Xr8KFX5%lX7O{t}SPVDoJd8!Y0Wz!Mrb`SwbIXA4+#C!}c?{giZ;2y+s+ z;_&_RIcR^f?diILG2y7`+-0ko9f~9TCM{P$U$X3!9EM@|5edul{t#5k=6jnq*RJ=d zLCYluppZ-f$>5)T< z%QJcg1`B}_5N06D@cGFeQ-6j72NIy|OZ4{kKJoU>J8%!ZFG!BTU{Io?ElDZ%36$&W z)1QR_=&}qO2w5E%A3uS%4-b0;Z-+M5UXutLcv5mrkxr9BqD%dLzA?m8$Nlqmx(_a+1i!c_s5GDnMhHF&PgYAy`OyUb&DIe0;i0MXm z=59vE>oF9ux%n>38rZbBNoiSG6<94tzFQdAE!4sYR>UvIjkN_i35od9v%Gh|U%wCn z%S__wp+pe04@Q0O-dzOQR{$Z8+-WN-E!~7`H+St|VO?BWN<_yoi09nEU1i_-_Fi`7 zLCHQ2HCgx+0xK(fsj1_DA+HcT3YaqhV)60gmGXtEZVzENm&6Vy(njV4Dp@$Yt~Nqkf~iRJ@EnH_}?Yni&eS6Cg(k(XDyeEHwT1-AIHo8I%J&UE-B`|hKmG29o!p(3ku2qWPM8gW%Ng>|7Uv0&CDt!AMqJW13QR3Fk!31FSeAA96^B#@b*@SIMi2JDd|DOO%fidoqesX zswnO)L>as@(bj#uOOv1vlyN*>CK9}fD`u7nY(aI7g&N-djQZfgWH`+1!orrPYl@S&eCi*RJuoW-(HWD2zczvl!l_(~0xIDvr@95}gJk5wN+3E8ZJDf-qK{ZJw z@eo^=8Vr3R&??8v*JU%RC#%W9TD+C!vN7|Y{>wS(?n;ciy62~ksi&`?&JhDXxJN+0 zdLM~^NEnmvg9;W60khL43>$O=1zUBhw!`BKAISJvoAt*|yqdb)loHthb zHnOk3`3>rLS8VZ*ZkD|3a}_kYgkx!zIOf;^G{HfolL$X8B9cr5M`V>87ba7o zgA96Pnm6WI>Wa?P@{(eU%KZ`#;6R&>EO8T2FA4P}KS0NCqL(Q#K$_KHEv?@)dXb4; zwl+}_@aRg=uO&vXS{3MA0TLS7)(97ME_RS)Ssf+)Q;Rv~q21h>X&eNfA(q=9Bx2pm33#gzZSvCDMN@ z-1~%>%i1`2mtOUCesvvDC*D7ZbWESyhBlUQE9#I}Tel7L2LuvSq9rK#EqjUDQ`2;( zi2(J<-jZaZU7LlTz7vI=)q;m{K@&IEf~$k_Cq;yLe26XFifCsrH6?5cA}QvWcd3Ex z)HI7tAm~D(M}+39rCfc9vP$Q=O$K)bY0U^PoLxE2lk}@Y=?Va2+2&mW<~@a~p$Il$ z@e}XYrVpA{0wJ+bg%hEy(gwCCt8+v4l_HrLFc@1jDAWtO31#@tt?9bD3ZirV5%Bk& zXL@R%0?R89iH@42lH;QBmAl(jjqHEo7F6dP#Tc=0A|IW1(V=!bH1e^225<%qRc;`@u$h+qP|lo812LmO6eps~;S@BciPZwp85i zeM;3TArKs(EcgZmsmD=`Ov`lS{_E|njtCJXaV)_btvLlmV1BI^%%jhpp$>$KMB z3_a^xPURdJ%ySL!`RAW4byJsm=R9$uc6{~+LX;;R-n?mZ4cD=o zVP#>VtFM5p5)lz$_h-F9p6)=!FU{pw##7HHXOq;YZ4;CfA3QmK!u}U8e#HgRsL8MY zjfm;nBLG@L=^R)-6y4qyeWfIV7tM$D6Tqn@;BBIk(b!9IuXf9}ZDV+hEF%Eufqs6< z3@v})2R^v`bgIcE;Ma?L*MBAjfBory=c4|<%Dw&{h}!<&ul)Zh!~DNI*rDPyth0MU zK>A>S)ng`l8i=j@^VNamxY4vL|9@5?2ZoI+Mkvn;`sibr?pfLO-dU%!A+%F6PqtQEdK9 zB!S|SBJp(6JdN%dlw*h5Ptg3d0eS-k=?|J2zyYp z))FUskAYRFMg+GoRiS1f2$6GDO@e1st0T31?cPcnNkXD8emY(UJdbj$NH755w0LNs zHXUMWI=uF2fu6Ly5(r)9KIfjxtLp31g;1|T!cy$^)&6Wxo<3DWY;yL)F>k%X4SZSx z7-kZL?$M7gA26{2pc1}}Mq~?)Qt|#haHtZ=XW~C&RnaDCCs*W1e?|_@?b6veF z^?t%n2eqC^#pJ$72SmSyw$}Yf7dy1B5>sGx-uaMyv|pCy$&iMg3mm- zvk*!O-2$sieyj>@kdOTFS(zmlAMYV33P~c8f+NxYV4JDDE|<<6f_mS?M?cD>EIzGDyLfNytK1sZg#c>Uk!FE zVG9+wIfxIz_+$VVqzKj@WfSWK@VM~ym-~&S!9B`6zVoXPMa792HttHoqrg~{6Sur) z9(#JWAeEPc56W`5rmfxj^1?Qv@|uMs+~9kI;w~iQWduxAqAvdFM{&DzrWY0??7yR| zJIVz6To%IZCS?oM?>ah$Y|)))%*)hrU#Y4OrQzvgV>H&TcIAHa4yk%Xm@%Qy-yegp z9SLMP4!?9Cw;S&^s4Ivgpz8=lkcJp`DM`zO0GEb^^SbV&-m@nPQaA@R4qk)W54Zc! z?}_$}@Ymzfd+5N=sytLu!H`>!_S1wFeIIF%? z6cH*gICSMQH@1v@J%i%S^TpjymOs>#J*jTVwXo6AL_EE4b;@9Cr1%Zx((c={!-^%- z)g0R?DFZ zG=CTOn2zlI?j1tOPF>rl+0>3pj@pcy{cA^`e?*F1IrCr5R_@q=P%AE@CPYy9p~U+l z?vf0g6fR^f0gXk|nCLvq5Y;8VNKJ4^MbzK)# zBn16EJO*4_;A91y&`^S0(S}e1iVdsB%I{mZ(%yH0jw=uoDPk{=L4aqGk&z(+=1HK> z1YOfcq;rxqHc7T_k*RxCxwMm(=OVCbAh^6{u2~`qGc3rPg~&~lD2@G;4x;YjE3AQ= z&<3DB%8*Vm))vQol{m-v1Bq1M&4mH=k`NXyI>%|jf`mIp@%yK(NAhAs)vXbA@ii_*Dx3_#5Z9LY zPf$^0r%D!d9#tX^m#2n|3$X|wuuyv+lY(HVjvOyc1`-~UmUh@QiG7xYTnK|h|Ly=T zZK4m5j0DWvC|K!6P-o%0It02MEW_HIo9yWtH`^BY?%g|*X>CNZ@t70|)R4yk@O~UF zs(RYBMq)OR-g)}=S(16zp6*m$c2q$I0Icc?_SyjK4j_8+|C8Vl{CdzwW-Q^B!Ebu8 zE@;ft$4E~5aGy1s%2lq(_f(%Jd-GGqBsYAc{j$1Co+K+Z@6y=5-Ye#EIJj_)!Ip-(E*N)SkT=pR{Z{bLmng)9`|*dh5z#>FK#lx4)t_w8i&x^{1F`=(N{~ zG5%$`K*iREv!P0W2h8YEK3=G7Oyd+WKHw>tD?rfJ(g1`b;-vgo zNbbE5J14F(gTD{6MRRYqF&;m9^gOIH%~v-clAt|YDicWL(4%WRs|?NoevzD~RMVGo zcitXx{>2PbVGrde(i=4YUS>>5z~(%%Xle)xhpi_Bgvo$WsAguE?s=?T8Z+(iJ?7`TH9K5?RIj4v~{lkOYuFR~hVBDVN>Gf74Hl4+N^a1nL9|0c45_p(}9 z=I%o;h68j_soHEmA~DrA8!h)NMqNIHVFQk#oItE z_2O27*j1~!9B!Q8|07Iz&tiKWjkOum_h^1+W+}JvlBlBHckHcRUFhWioCg z>NtC{vlL-fRPj2dd*88lO}>AALqJJ~RhF+Ms74{;1PpGgcd!xKn7%Ip7Y}17^9mHc zEUP2|4%H-oG{hUKA7cULlXS~UcOeH4&^C;ul3>RWuoZa`-S9^+suGrvsR3<|TFc+LX;{$gj|D2n~|}%^A_I2olysFjWOjMUd`|6Dm=-sFNE0CL2=94rh8R znc&9CTBD{xLNo>y16hf&;_jG1Hs}6ODwv1r_@4*<*8KM(0xBpFVz}$?Q3%NpG2* zW#_AQz|f+ z8$DI}ax)h(E;iQJ$cF2$UN~)jwk>4E0??A*dQg^3u;GQPL2Q(rms&fRi0~5i=F&kZ zKx2~;IgR7QwnH@|({3&^b%L;$uEjaHuFa`SDPcEx-ER$yS##nNahPD^Kd^y$h0v4< zQz^)2!2W8TG(^h3xdl<#e9Qi!&z8Z$as$dQAIa(WX9J0_A3v_IISO(9O#AfP;%t^#H*9_cSM>1?ykFZSH>(lwKUL>Io-aKGzP$XjmRX8cx8Vms^m_v9bEt=Cn2 zmlIlO{WYxW)Xle(qm!+n+lB03hp)X@26>qVxFuzntuu@M;;qcVfnZpHZIy0&%O%LK5;BBFVj$1Re4ji*GN(UalX z`I@jMs8H})fBm7acQrlz;-AIqzN1ty!^Sd)mbGi+r6hI=Mt7-wq>7KdCeB3 z>X1hC(*P#9aN$;?iHlN>lLbS>Z*qaBO}gxM8m_QZXJa7|5n_i+j9mcQQzwZ_=-jh# zz>~nxS(ZRmj845iB1OXMPQ|mJ{v@AXZqrJVLVVs#SP~kf{K5$#viF7S+S0*IB%Q@?7;+Hm6)Y zzhI<1=w&o0?5P;9DU--Vphj2cRd%aG%NzvIY9Mgl&==x1>WurU|L@#gX!CKl+y?Q~j7S39nE%7Hqq7v!*RPbwv#Cqng;7A()$MQkR*wq{Yx*K;#Uv=$)yW~^ z;Pa|`=Wju?xo*L^mnpCH6fL-o8LtEEkzpkiQR6-D0%GumWB99fG6zAjC2&hRs)NX` z^1NTBRPE@+UAh?My7*{UBSzr}Q+{Rd-j+ZmVtE3~I=YFa#L_CL$#)CyP&k!ZPPDUBVka zFV>3L$Ryl8K%$9Vu-e@Z&yJ7<2&@y0q0~hDo_fLSiL}GH#t@o-@!<9dp5#hAyjgUk z=P}-xp&X>h(9>G^4Jv~=5&O@UQq$O~tM8fwOXGc`doVU_EeOs!nYLo+)o11A2?{;; zxfpY*N2DaiuEagf*oj0Wre2t1oh6my&4^>un#|Vs>hvS(4mso2Vu@{zZAZ`@3C@>otZRwbPqsifY=z|r4fXx_b z08{fx;0q+m>%gg_GRoYBIhGw+yA~cqt;edql)IG`ZN;4UOkGjcEa-=S;U($CB&C%> z+DYZ?YsP-IZp|}kN;N6wgUvwmo|B=88}^VJ(Y#?j?HqfFM#PRI zBm%fCJJq07B99OOf#GK?ZqsM@${Jx!K!m>34Sm->f5J!)Zd)0%BPRxB8esUNTC@^X z6twcdZ?C~G$N>{f!7j8_VPxVDq(&WDDjBNm7_dx&cb*J@8iM&^1QB?Nu0*{6@lFME zNU&l}XYOSBwMRUm=1juFIk0T+NcKr1Kgkee*`Cq*g;Hn(u zlXvg*iq*_U=g&V}=f3xR{^b}>5l}62bxHOP4KL29fAH6qb!1nVVpHOuv(J5*Kxsp$ z=GD%bc=|VitMGteA5f)BM}Z9 ziTh)#$DnAAq8^cft3!tlO`rL!jyx)9U*zl(l!#X3LbXV$-Cys&|pnQnqrAN zzY7sdcSeB?dt(fp!9|WiXr;^9bG$!?i1#D#`h~3rJv%0h4O{ZDbaOYq_MMy9LBh2^ z&o)`o%Iw#vUHE(Fy0kA+s!X&J0-C;sq7&t9&~r%Y5_B|^sL!GI^E;|`jP36~{2e^O zW_9t76x*9pl|-sRR2fUVkil#QOf8drix)7xupk)96`K-itk6L@8B4r# zFzW{8eGFK*5r^QAa7oq*(I|<41@=xxkV4}w*30WSk%2qHv7nAABYF_%7o7NcqOS87 z*862Q)fGG|V&^kiIQzR>?8Xi~G6es`)6*XkL6sP|J_+J8C1QO@bU{suCj=R(EoIOV zRjg-^N@*w{Sjm)?wemVw(+~ena6A8Yv`Ml+aDL7!w$mZlPY5`) z9zPH+psiHO*wYMM+2uzC4LSNDO}EU#_VCd>BF_O>qy%EZ7B5D>&`mx}0d6?cfabF( zK(!>Dl0u@;7ZenbaqR-D?p3vI{Us!VhzLUf6C|!yqjv;m=D4ibk>eD|Q*G5Z0IDV@ zdI(v)>$mj7xFFH6eEq-lQ6quaTVjh>kcyLokeN#$x-C|m{-UT1ZV-hk9Ys9}5}0PD z-eUknpJOoe(<2EQhpZ?G>4T>TK{^*cY*}J+k)$D_OaV87A(~#n(FK{|?GYqs56KPD z9aEx^^h_U>T>95LVRXc_#6@^~R)sZJuX5ZC)Q7k#oGBNaGg4q)mWo8LU$)CwJ z3~(VQ?HTvlF9zceNmxFZOh7?fo5k(ZwoS^6LC13Z@{nd0iOf&T+-81sTez-$==f{P zu3a^EJEvAhfzL3?b`R3(Q9zRVY-K-t#0ZgC=!C&TPE)X3ZovFCIXwZ+CD0`i>1;l~ z-W2`QF2R%aRg!o*V2HSdNk?X^udnYc*y=D6^Xypk9do=h)wqNKr1wZ(6|j|d(sXR; zkcgcus$WJv^#B%RG;FHNzC?*D+574Z8hHv-rxB7N)wJgq9B>!zk6Ro*&{JY{;f{|Y z^}=Hh&!!#@!F`P8OFDFWsb3bUTO2QXwu&{<2z}3>1}E2eqGIa>IUAG14z};u!RsuK zTws5d-+rjG$a*E=%7PReLX3Q_~KF0fUPedJoqZ_$L~ctR^Suc{xgF zyxhsgyy!IdS^iRY0hbiJcCyctK=$-Sr$9!5E+qxi{g?qI>rR-pO;isGz_YW0j8JaJJCvNKTmiJcc94K_%qm7mNz3T)+MzEG%rc zmbc*(vt<%Mq09F-+o&7T@zfllX-u?VFZgU{4VA<0nsEQL6r<76NSo@{kfsFdL!>et>G! zj?)$5VYm_o2W2aUM7H3y3`1f;7PuW{n^jV>(riE`87PI&P~xhfosgyasBs2@k{W`l zyl^l`!XEgntNW2gBZi=e;02iB&qVBi@*GYaxx@$krF^J`ViEt$i{Y#HFK4*kZjFT0 zh?a`uO&TGq^Yruu&X7e#hQg6rA?S5_W=0t*7#RY@xZMb-q_0xkTSh*kE4d97nqUhb zrvDXmnoAvT`Qp0Ujhp$|K^odr53PB2uA|cRlMhv1X)71398lNsEjIIGaMKil6w_4nV&PB-Z*Eot0wK%*|G8nbm!z;R~H z1QFCxt=SGAJ_}REhY38PzJH3^ddQ?68$Ja)r=#tj#ue~h(I4;jc4pjo@I@?O-JUZQ zHXB`=Y#OFc8rK9loz~Zyk#%rE*7=ds3!w{-b}Hf^70g1VKo2m*-Xr5BsyM_5JD(k^ z2G)Zr4&lM`*Uu|XK2b^1BsU6&@Ix(ks>6AuaG4Kun>eP9kPlbN035| z*OZ=HY4%rJ_D>`8;x8j}$UZgCv2Xa9){|WY$E2({MwwX+X~GL=jhcTzve9PcjGt5J)E%%yS7I-$x)duUryC#R?>Wyt+YdM9N4?kYtVV zR0~0(q-gQ1+HlU0tVEqYYdTX;fwd$WaMoQ5!0`obibeS3b=MrvGnZH!JQ;lc>0)@n zp|pSgVWK~?KZS1plf-9Q_35^T$zZ#q7{6f|Ax{9nAyHk&;SyAMp3ppyWHN+9Vz}gx zF1U&Uos&3*BR;~hTFjIFt&CM5R_lqZ7|r7Xw#t${CKuE)3)Bj#P=O&127RT zD!u>v52@``RDvt?I2A1h=S~Q)JFSkB_cCx3$QTdhqu>sAPjHs!968_b_6YUZW zy={8gyyDWOn$!gM$*m8Qo_Pl~itTQhBzujupe2~}6@*|94iIsOOFjxRxJ?4$Dk z?z#0%I~ zaEEh-csf|CgrNY0#{hfKn!CZDCnHNm1Aa_yZf<_%?Tq>du#ZwmfPy>RFD?T^URw&1{@ zx(HD%R{5}#vgP4PnBF9F0aZLZI;uk1qsK-jCd^eSKH<0vOpK^waqvTFC4Mb3A>04> z<7DR)yp%?AMjJRu=b066q5mxA{%2+u4C06dhI0*Gio=;F%~vfxJd z7z9U=@h+mrk==-6$zZ0C=97fa&~ix|#5Y$M&E7UC#c5Cp;9tRmd96?d5Su-Y*58FiEyTS0mDU>N2GQaaHk-#&G!0+VykmxORT}Rwf%J%1)q3xYxBZGr+c!CCnx{fd> z$ng+gwjRZ-4D#4oU?74;WwEZldw-HP#-E3PE-67Cii$d+kQdgspH1oA_`d9Xx1I)< z>2p)wBb#4SOj32*PbrY!;g=O7nee&i43)ld@|e-z<1Z9`J3INc6daI(LIWX@vrn() z-+oCvB7Xsp2qe*l6up#c;klua@H^Y#^^h&)0xVTY^7$k$R#Z~N`_y%>^A zz}As4hFnh~$2cH@=wcxBiFn*3SU{rXp>4TNf02V}Sj>ejy7SLaQ2<+N0Y&mU3((}} zw2M*wl`6Y7zh%8)qM9O zwz)FOw1DjHV*MIwl;ewtp*dXR{dJMlo35X2LRXe!=+u9Q6#?^04X;uK5dKgKEyZdi z4r;<6h?9gds&xxS7p4;=KFD3$DCWQQe&+?S3DU9g!Uu17UH5=+!zL`e{C9}70LOyo z%7oZAY-J1`5VP(X6M=Pvzu*tgEBQAwn2|}<7%9K2))=s+_;6v%y}wWV0lBQ$VI~`q zJyP*N2x`XIH{GRRVrC95PT5087h=g+TV9&K?l1%>Mkw{pNc7qyd`Um9AQ=#9$acS# zw%R1^!Z(*eA}HcCFVN}ZZ2C#_MDoMl#?60zLyo{CnoD{z2}z#$2-ZrCvoWLzbVF}{ z`}Qpn0O0k=Agc7p1s@zwoP3Z~$8TPO(;2xL@#I*$t#kDB^fMU(IEYgKp?79h)`Md9 za1a`T_|86oBU97W)pgvO&?;ROufTu7DXX-;8`%!mtBvN`Z9vCZLPkpi z@fot^e^v>-=;j|3GEJxn{LYKF3-a^(Z>0YGrVk=eG7sK$a4-gOl!hXSAE6vUT_*4x zv5Q+j&M8A0Bf%hyWgN%hgb-8x0OrZZ16(A7f2*|~VPlgbDMSDsa^?UIE&PDx1KCXp zf)hA=-Od>V$=QL-5U)_XeLDk9T_xK{F#xtQ%8nY;kCe1ZsFQOjEyr=}AfAsSL@8Cw zOC(Q+CI^XjRx*(62$l?;7|Gj1&dVtXBgrs~><|p2=t$6w{3VC!eyOcnwhRMSA*b~W z5(CamXeH*b&TH+{~3oAO#pbr4hi6&;X2F6CO`8r?_-dfQwRN@8ZRi z&%?XM=lwtIy$4W~Y1b{<7{(d%I3_SFq6ASvFcQ=eZGvQwj7p9oK?DhtoN>g|BoPIa zoRJ(PIE*3)B3Y840!l`rFRD%6V7Y?V2p`D2~kkOu5ySqn2H?)F7C__BX3a1bf?*goOyxkHC4%FXw$g2|| zK_}{JaMDz9APlv5%5x201sx3l83j>_X3*(DouG)+4n|8NXw{y8A46gJLxfaPeyls> zsDOwC6=7&4$ewHu6kAzy^smTn)uS9mKft7O`w!Mn%ZJ3I-V%#kKtN5%#TcL6Z%jl@6sHv zfno~*@?oTh$D!>bxm^u$N1Qy2JKXU-#=mZ)WiHENi=)ig-XL48z)rMw6I3OBL1PoAo zti4D%hj-D1x+fd3n?Rd&WyCG`2nHiZIo0qU1s~ZddaOSrQ;s8PsKK_>lq?|z!i-CY zANt}d&EXJpxm4o^#x`zlBK-(y3M;XQ&0yd4s;tG46nLN(3iwcNLRO2o2clUn2moYw zqgaPFcH(jdITa#jBa>o2Uf#mDt|+IYbh1)i$J)11d~ADng1l?Vrw&xYr*Fm4i6n}m z({Pt*;toE(B7>i&@Vg|W%zuxD$m;HQfPDA9c039vkju$PtyXs{1GS|?r^T^0TTR2V z9u>FefGnjgo|Y(NN|YF%&_WRAjYh&wgh+Dt8C-`qiwS^V12Gs$#|98w_zYQB`Mr^ZoH(4MO%SzHw17$8c&IEC8#zmU%A3g3Eel8AzY2CC?a6x zy?XmlfO#-Vfd$xB@#twSawxiJlTF{dLNl7D&HFrg#Ui=9R1x$KmKzqm{0*hp_Y zT5C;jm44E&9EfQUE{>h_*9d6HEIn0m9Fz}L5V?b$%W!J_04fY#>rV+-{9texJC8`a zo`YDBoG#aKx>H(=j@+q*R$*)cI!+E5G%9%Ne*zyHzG>jR9>{Nw_v+tu2$GoXJJ9o- z{oO(`ABSF2x*U{x_DTW~-YBgJTO826$PZItbe@o1k+GwP%CK4B1P^-ugcC9QaCW?) z0;24MkRfBEqeCiN7_NtL)Z23k1CWy{Oc%5Xu~#0$Wz0QHEory}U^Yy^F-(AsyJt!njk z6*xeL`k(LgN0;mPmAP~0wgcqy3s6Q5H#IRJk1x+(F|H6r2ns0t!aoRxQI`S!J2KHl z(&lL0!k_qtM>&x;(F?6@Y2|c&Rw$Y%b+XYo(bfekA_M{6U^M`+Fufc!LbwYd6mS_E z++F7i@bI35+iPw;B>QaHHXJ{VgU91jkQ{;OXKu^cF80@mgso6k?kgGHMrU}qKftN? zx~s_w*3@Hw$%}Qhc-^L4Zr+-BdAvOL(I;$nO?%K4@S49FlqU@pv}ZgO=fj6__t+s& z)AwOD&1+{3W z353~1j^rt5LO%dCtpM-VJTLL@D{!-dONZ0dYE7voyfHyF;fG!w&&s}T?&IHFY|WYGdq7i5gGM;(q{fWaVN6GO+t39!z#D*%C`7Oi|4o}7CzK+o zBPA<=T9=c%QZFLief|3NXutSbnGIA)dV~o}fC-2PYil?>GjE}P&8$4hmB_k^xOZR# zKwmHMfMCiOT{eu8#tfz2<#}&#d5HibLc#9f941i)X>(E6oM62Ol!yngzS^541UPZj zu%ObJ>%a%{2&+E?P!jmk>125@zGy)LwmHF~2?B_bCU5_}a@hsVK5}#3p`HHnro-5v zspp^LNVv)=^)$@x`)9$O&n-aD%uvJl;X-+OC}x%0J%>8+{KL^YIe`M|ax$a@Y$A{Z z435xKDK}Zyiaij8D(>>1Pap*z1IQs47MJ2X_zQy^m~2SKK*7@^b@(n&3Tf&MoxUc82APOe!zn0vMY=p4{w>9wU%2FMh4jI< zmRNZ_OIQ`(Np{)CCRU`9lnFNu<+pynJ{pBe)OBET4li4=B9xNCxdFOvllJ+09?{wX z03{G-xF3lY`Hf5Md=>-qT|~XpVaj_EEjxi5R1pBqBsf{ub<{AJw=NhYc{q)U!4k_a z;5oF@dmOq>1$QFKSmI79!06Jitu(4DRZ%V8?PQa2;&~U`w?R-<{4c$yShhs`Q(nvX zQYfJJwAvyL6XAMwwm}l=RRHuw5fY(prc|nfGwFKy{n?Xas{*(=U3(MX9?qWDVAW1O zS%3cBg%TDtkwg}Vh{X=INL)OCaM(jmeRfFmIs5Ef z&MnKa$lz8+nUU#+WGiH~22qbO#^FwjGf;oIFimZ=fED-S43?_bb9^%w99<@vz~W09 zc}Yy>H-ZUU)v`&0icz+h6c36^Zd*2ZyPogRLv0F)LFsYluCf#2Aa*E9DaDo3<8YZ4 ziuqFv-%c8ABN1t5RMjFlvpwgPkdukuvVICcPN(=nJ&b^b{-Ich^(7bAj>?wocY~+F zu{IdkAENi?E*y7=e$<7$CgsQEt#~Mk6Qzw;l^lOa+a5|O@AgldLg$naXyQd+7ph0D z(9jLBw8W342AjoHJGfGC#+LTI$@KU44?v0(oQy@PaaVL-EZSKOOSQd}>UO!Tk>ORYFg8ML zC8bx{g>cy4fC#U69SzsJmcbsc#>G8#)#6HV&lf5m2q(u9kh9H2H@@%a-~%=koQ&Q= z;Rjn7N+>Sf13IaJ!<871DQ^$GE@|(3Q~g1&rv(bTrp_$6Xwxf44vwCw zO5onX!^2aNNIdev48Zwhw^XHky#DCPWJReimq6xD{+faY>pD0A87>QXeggLoQE3Uy zyt$kVFVbVA;paAm8-i3WCxA$v$13K2{)_h=tSA(Z7?F^P$<8%6uG`?O>^HuKDSdIY z1sx?RQy|2Y!km9p(!E1s=ttWoIp>q@i|C2xP5#cS z;@-afSy!$yYMB>N=LM?r%bm>I$}6tBqKp-(g~3d5admmJ`gn^vq~I?>BGk9Z${1Mk z{r3tWk*c=Nn>)Al*=$`fgh-FW{o!T@a0eoR6ETHkEo8ca0VSuEmzFeSXm#xL8)EVr zhTKI8r8tJn0%-_8o=e_MbU3z%SiE|<%f1VLtO^wwtcc1Z9u0D=aMp=#FsZ+*0K342 zX}}E35vMd^k6vgomvi^XGC8NVu)4i_E$!5#9+5X{Ms@1`S}D%L_HkLYIC%%R3MJ}U z5g~YC;0Ue?ZTn~fP%9+%f-*sVh@-1n9!8ZX?17#L#OF_a+fNsJQ!3?)bp|Rq)%!b} z=J~XQi^C0B$8(Qu(`oiLT0e)5&NAdImMkBKkUx`btDEA}(b2fv`f8cviG!?dNfWr6 z>m{-bHTY`OAg#+KJOn4c_3-Hsuj~O|0|%Ys&VG>bU0%p}rs^mG<^C~~zMQs`+b8Xs zBZWFWm5Rpv+Qa1=k#`a2Azx+z+Vq4+mYv8VpK=(?@f4iczkKB_WT;VzGXace=Vn+<<&=-BX#F3mB*!A zr^fT{(h-v#@u9QR-1Z*(?ri@aY+{4QU@Oa3uks2()w^b@8$w=eBTmae_?(w?Ci)=H z3Au|V?w&Xk2M!T?4=?^z! zu9qiz1LVQ(*I%u(HB|Qfry~CN4>);MfCPCA%h$(6>LOqfKP&28sGe470D9WgUFn3a zY9(q_)b+cGZiNJphCRvtx&e!$2DF}7!KS*LZJ~qlgT+Jn;0JT2Od`5at1ACJ0D78% zu${3CT4yBZ$h6+QJ6oPQ{77woV`@8DKvhUe87=7*p8tWYJg93U&94;l7m25V4|0vU zVCxVgpWpnRFwnmTV3Mp6KQ1Lja#V6I)(O8Pw{vIxc8S-<#V*sOa5c8w4RmO8UYWRW%_Ew^K#0D;t*dPnNEA&VG5zuOVk7EYZ#qc6U}R zpRo~X*NN9g+!8~#2AkxlIbs|_cq$_(2#mQ`B<$HD<*|#isthDYAjrji88CyYgL811 zIAYx17)6s<*i@MS$dfV!^-B>Mcx=%$96kQU6ag>Z~p~SJJqU!Si`Y6DYqYaBh40J=cHXu%wiL zhQG#vVX_1d>%U9TU|5132#P2uu1Wn4l936hF;bmHmob({NJt3{H#Ki_tL*0_7^G@Mc$f1-<2AKT zleQ0fjnv0_qr8MBr-A%h4yeQu^%!&^liO#;a8F1%S0)bGdr`GZ0)*0J5?)pp3ZNDw z`g@5g>ql}Qc`N6d`_R}FfD6LXXh;ape<{Pe^Ojiuvn{Ny`nJ^9!ilVH^hA>m0F0K8 zi;K(9UBcda{jJmhl+L3GP8IP@;c@c;aL?sI zM8QIsub4D}8DdWQcKBAZ4I{~-drx8rQf;0kxo&{6K>2<4;T54qpo*5c0zLCZ4)n$r zUqG(<0!&t~C~81_OcOCm0?uj@1oAHkp-ql5>UMB>tEHH!2J_}`CF7_n z6mA9|51V75fYoG}m=nsLK0*DIv7O@LuiI?@0-i;j53t!ZMJ7$m?(E?3K^hPVjGrC< z2s*r0PP}1uiicwHwFQHCnLNR7E3OB~Sehe~2=?7j9)H{{&ca!T-#IxJ0xYKFThO&N z?i{IEUwo}tI#;TlN_NmJ8={mhLy-~#o6nN%D9BFXVJPeC>tA165nyT1Ig~cy@^n3p z0fYW>1!QRWQ40ryZX7OX61n;9h{h+AC81&+wLry3;34cUBD(0Dp>apwyrC&yUv}P3 zXjJw-qSX_v1zSXF9NXhXUt_AVzj5o{cdj|5Ye8E3kd)C36H>gE%t!vVvI|WQU{z@S zDZvkpVDA0|73m+QaJbY^vsTdwloJs!xvB2GNg8p&BZ#sAOoD{3pm_c9`|lr64PCZ> zNvNvWL|;CO8`K`~yrswjj&B-WAhO0IzAP9Vq*EoOJlZboW?FtTTByXKr=G+nYt;B8 z1E8`DQHti0F#CeRydj*OiJ(2X$8GA;QBCN6i3Tp-1?L zGSFTkTT_u$_E!$Uvj89nID2z?0AlYDYHgm=rEUl!37J3<-i`02j9UTt*A8gt(U%`k z)T?7n+J;P#6#hpR4hYgJKs(ujh5&jL#63W3PN;~WjHFIYBLmFApbrJ4zt9w$5w;7R zDp;aA2umNH%;IgmKZuVvuqk>=F47(*o=Rc$D7wPl*dm83z8)j9BY&;4HotR9H;xE{ z!3Y21cqf@R#`aY7{rVJ4j^dLm%&prujLEYqS_i~8nmU`<2z-hUekTDCpjNCb zIl>*E^tQ?D%rkgi*td5kCY>f7uMQ|ID+zJ|~)gp+EK*n1t{> z*1-Kg*OOAx&}LB`5u}h|RRGho+B5&3JBbxeFjb_vd*~zlYqH!FyXVQg3{$qQf60;> zB(#f^4Ut8IP~>7XvJ>bv^ekr`CL1oizIYtZw1SW+Q_QrGTGCtqm>*!>-h8M4a)%48 zi5w5_Q)(SZ1t*Y#oPJ5~(#Jp6u3I;RWkrT!1iKT5EdMpot-U$ z^4CtG7yECRy%H5YG>V>VXG}#@ouh(KT#6&MG{7sT~>nY5D&;(^V-=L#~92)4UtEO6Lcv{lBB$IQxk#D~z zDt-3Hr)wy+fAV80lx1W}Ge}@{OmrK-Tj&!yh%M-yOtm?cBIE^ zW_?5>zY5539QBaV0A2Z1Nt7DYDPkfAyPK{dWEu ziQGnG?npdtw332R4?Ms(^lL0Y*o5zcDl)GfKJVc2MULko2QeUwoKIwQ$W1g{enW6F zvMs~t;(6T^R0b}TX#0Ubl0>M4XNokfWK91uH{#hvb~Z z`4cCv9bqgS^Z3>D#&%JfOnG24`^<|*g~3*ieh>>%4hWL(LC1z z#ziMJe|_2%*B+Yv=^8WU5PwD}00uk46wPG%3t zrpW6DXD-)^)7bb@=0D2tc20#YaStK5@fqIYnpE(<0|)MSEK8~pS1S68(>B-@aLQ@X z4>pizLxsb{TagDGrmPUD2p5gRtoZFkQYxY9-zEJ#CR_L?Oy@z{4AGCS=HbJKFmF^x z3a5&b{%XxT9Hzn=TavI#Ij^6P!`H##5)Pb-yqS=SgBDZ)jjKh){usU7*9Mkd6bc?K ze~2Eg=SDqbb+TdSOHZi9dwUI~%JDWC#?TjOt_cHQnUY7+~|{K-fOLochmbp!m1% zuRkxqJ8tQv<`CzmHWnuqe`7y&R?%v+L)A)TA0R8RAY!Rb?nB;-gkB}(&AbeM)+ zjJ1c&2kpgk3$ce`Y*NuOkuhzUyQH?r=X|$KZ;6>00rUuoq*bD44eLkCiYCw6(UMz< z4yDiHv80GMk7JWde0*$-22YN*9pjzJNN2Gf;TqD=$Rf+A;C3%K(Gb_DZfGBcS>$bt z0^xgAbx9;;e_s$us9iBNE&n~RQAz5O<#VwMmYXBDXK~>zM`BTzteNX}bp=0W3>~PQ z+()%-S@IOtx-y*p8gI=XXbS^=d{607Z*-d$sD(tzwFC9OX6$10rDC; zQ@Kgc4CI9d!(t`3e<7hTZc6-@5zK7IPy8yq{x1z({x61Q(?J;eSWU^ zSIBY3YK@i9NsxyCsx#^=hK!Go8y{p`LYIiN8W@rFccCaia%A8Yf4b<`H_V&- zY!NjGSO4HMZpzGO$QqKGzpn>m1rU@UHxq_3T$L}T*6rKQ!JYCKq448B}mny zw`fjnm{J-M-U|Pb!ndh#M1lal9id$#o(`m$*ZLC#6L<36KE_x*Dh2QWo^{+zfbmKI zerNzCz^`McPoExxtjZp4f<$zO5vLkj)BWjBzr0I}Jfg4yD#faC9M7ll11}zPaXj0B z`(wTD;KAqc4#)AC={yqWI`Q$iD#qy~OhPbQd2`dz<$&A}&NNpr00ss>v5XAzSLniYWEx%W;V&yn(RcgJu%_!4N|EA~V z7b6Mn$9Nc{tLQC}E#QaKsI8n-uUvs4Kg&04*omR<4KeH4U*27V{Wiz(`%z|*A&=s| zbJqqNYE&9%7MlkTLQvOmr5zfgzvi-kV`G8F!Po(OW$uqb>lil&Vn!{cCv|4^>2GFu zXkUQsF!91!*a#aU749FoY)M@MQvHM9L!<6H>eEPk${jkPAKFHGxE}Sj?DsvSka!># z%+g`lFb08t9t;V49J#dbcPPex5}mW{5^m9X2cDKWEvqiAxwV4^qQlCcmw9;u;RfUd zmmrUVt}5N;Z2o(w2F#G5vO?JHRQK#hkJz3SAXtqdlO_i)Rk-fJ_m?!zDDI^D5-GBH(sASTk`R$ zwQFO5l&lE_vd1i%NaKypl_6J4vtcr5v~ej z!Eu$DN7yiOe&BGH*s<_`y<@pF1em`ZKFM{U=_nG1%X^$9Ox|xg$S}~uzNA6$S>C?L za!AmzeFsz;R2Vc~-()y>fo-Vy@#8ISat7(i-2(2vcX;?v@NJHX4YTr|`2hQ_y&DdU zhAPFdafq4Pp(QAkPGPU2yxALv4v)hjvpl$jnR`z`*fCGg|rFWG-i|Cp^=@u`x!}9%lg(L^rcrEdYrit)%?-C?BJ+ zI@DTAuFdNv92230huJ?gXGzWu$B5VV7OBbwEOayd~&!qwM(;wsI) zq}NRj^Duj#klUZUZmHlTPyKWggl^}A;k3fKNyc)4uO`XDhYl%)*{0fSzOTH*V7ywB zE60fWn)2)l1`CE9{xCu-_QgBk0XTOxQg#NXF&L>ug-ty8$X|c^@{#(#q&(Kv+U2x> zadX}9l;ztCKWumW@+=t{+ry8qoXyZ*K)(W7_2;X6?r5@@@l5`c&aC+>DF%Lf<%xg& zUHqXW$JdLZADRFC{|j>0Z_)PuPU?4eAamCC)#SVgCo38=5A_Y9$U4uFcVvO0(}KJK zVBf10i?25aA)DbLaP{xM%RyKl1Oh_C?coUI#SH+jUWD-oJ~DE5_bIYkIy+@Z#rFA? z;LaO#%aABJz)y&ML#k>Z>JAc55zNTh?s-j3P0tKS!2X@Ni;!E9jqxDXb|*3>nkTgf z^AWQ)ob7KZM4I;ER^m)SgP(W*^)h_`;9!i=*1ztZi>@RIPl+c7Li72qeP6Bw+~CLG z>$k{YnpP!hwfksT(44M|IwU)oc{S%qB3=@1%FV&&Bid-|2hAlA9s)ap0TCr6Ve|N(}MQvZ~)ZJvQ(I7NwZM+qPpf` zDk0jd0$6Q`AmPI#ga** zs65gh`dw^gRJGhS9p9>}26_Ch7CN`#D1VvdKy|kgh$|dZ!OtYD0w?X80q_VHRqWe~ zpb5i>j8kX7eygBnq$w7D`)$KOmeBWFxpu9^_kA;JdiY(nI#hl6$>|%A%2-DgP4Amx zYG$U{o%K#!zVx^QlK8!n_aAJJ{ww(1!lDONil?=WR8)RGuDJG`fx>e2Im?&-vU!%3 zWbKc<5epT9BsDS>?(`%V_AUt<{qat!slRA<_xR8uaTY|tBhE)zKI4V^S1}ZNmokE- zsr!KY>{Gi9jO#ZxfSxCHbD_dL*S-%q3Kr_dq(8piYj>gB7NY(o6~`C>OMm)_EeaVp zxAn`lGu)kTi|Ks*FHkMSzK2%(%PVi=|Nmbz_u~JFJMl&oP2QryLPcHO0Ih=LS_`W+ z-wL~3Le}!D+%un>rLc!`ecpGIK%AjmRtGnI5OA}qSuyw!)QKWhGR<2Z9JKQ*F~aJX zM#)g~>fu=^6d z5-GzU74#}q90}2v1iq{pMxM#{?uw>HNg5b1odIP)Bw#YWy?fP_!(=1NZ-dBR3Ew=L zvJnYkHc4rooGByfDlu+AUl@BYF&&9wG_(ENTQc5lWJrbToG;u;6z;AYAT2~63QN-8 zi+m8B|C>M^i7fG?pnxAA1W|&j_)O747p2c_B)^K09)$_WtdIu}HX9llIi(W_H}eV_ z-o%i)cZeEBR!#Bs1?#dLmEE@ns(H$*dgxXv7t!pf0{ zRwDJ3_x%NbamW;;w$YNF$Fq7ZaZf%*5rIPvrA90$iPv%6Jhs#tN{(7Oly zJB&m1XTm0dhtNEsb}X>?_GMq!i~E5qj92b4YSBVMLS%}H(ug1MEY2Y$m-gcip&vC! z(cH=eTc5@4UlMlr?pkM5h#M|CPfyFR2XBduG{6xQ;d=#6*rs7}!i8xArae|dNfHMD z;h=|VF~s;r!u+?tK_sg-GerLap$=U@Z5m{QGQ->3yEUQFLbMZv237PlXhaOy;srZ6 zI`A36ShIhol76+$JU&Ij6(TUQarAvU4ssPYv$nQ26I^APZp80HXhy{TGXy)X&o3z{ zhIRogcoh*Bb4Uv_F00*ssd>$^Wuz`8vrg1%gd4C&Jwh(1hW%#;>(37KlNHf#553X* z{{1#Lw`rkq?S12E>8WpBWO+REMU7>S9L(>+#MjTSAa2hRqoj+O->bmsdVG)=K~Rjc zxJl%K@{l}#s-S`b(f8RVZA{B%_lk&chY>JkKx>13{dE>VgbtESqgN|ECq-)2|LLK> zlu@Hd2C3*`g+WhgijM7A{)Q=q`;)@L!Z;oHNR3IBn4dK?*>|F1@BK$C(BCs;UKIX| z$J+f$Q1=9bv8x?93HIY2r$Liz_(M3pDxj6^&;M;XD^y_dIbG$<`|@QOySS3VXWDZ! zf0RCtBjK?{Q;rU48z5utvc#$(_YRt=g!py>ceP*tkFUSah*~hh5WPvyG%l-02~f@; zok*Ac$mWl_45PNjAoFB-3`^Bo?IrF|tv};#dv0{ZO`u{@yrPEbyZEZB1 zOPO^2Ad>=)I>{m&z%fzfKYr7l@k$Z`ij7Pf9hGf~l~RP~fNa&i*3h?|JoV7-0J0tm z9;&?ew`3%UYbj&RGt@QQ)Z!UVMdMC}6_?q;$7G=gD45Z*j9HEf+C2Tpe|#ABmL$?l zZf1LX`*12aA$(}z0HYcfjjrNjCe~VrXl?xZNH*m5P~J6wabO*cQ%#SZ>?5Z99%`&rRe> z^P=Z|{hVg2V%&9{L$Wydm5C09PKTj^feAP_i-q06M-yabw)$utQvH@9x|^LzwApeh zzp$i46_UX#w-w)K6#UqH>3z8lABjh@I}7&*AHX#JYM$lyB5@-h;RoQ ztJ$+FzkLr$xlIhwU00txn3%ZaEn0XG^j0jCh&NH+-auv5EdAfeAr(Th`y^Tdjekc{ zFP+->{2B$rZ$BEfh9lt@@U)IxXt4QHNm+R-696l?q&re_F%xc)>bK7g;-RMr4YAjq zJ5gYbf*7n(-1F@ob#TtopFN6JEI0GUq5kjR?8Jw3`~gl1A3y)|)+wmdF;FGy>EAP- zedsElGm=) zXt$rS@GlKDhI@wtiXaF*Q;v)mPucotO0D#+-^p7g^RRcG>IdT)uu)RyC>9Dh1>xeO zEk9)syf~QIom#(q*|N9oEXL{+hJw#ziW6rsRAg=TbxKG0dzq(X)H)&PShv$I1T^|6``)w&#(>J81sL;&kE2p5uDA<*NUwhd_!*KV z8!Kn-Q^Z<4nEeotaFbl`&VE}XKbPDY$zC>3(JY0ls&+;@SG8~iO(kl^7QJ}k(0LnR zl(f8jnuIu{{aOVV+K|5>%*6J&QoQN!Kd-e8psdhnIJy~Z`(XgA0KXNY zTPGwWD2-#BX|=Wu!)%ATZ=Hay(N7EoH$!7%b8vr&-WDzs$pUNAIdcxQiW+FK&R^VS z%+G#Y=XTPwg+f+EK_LiBs1s-9*^9=Et;<)g?#An3lJNb@M&GNT*rNf153>Jt}_d){46D zvq@%rGn+c^7_K1^f-;?pt=?n3Zqw0VKI+uyT)2mX4GtO;@uM>_#yj_jJ!BV+FnGjL zykKGtRc2^`*NdiLG7(dOsQ>6Vh);GHSlI!lu2*hur|@pv)8_S9nxG`wvsdxrT>16T zOA`w4$I}ziLwv^0tvQnWpftdYnChv0aI(_XHQW%rIo*CHLPjk9e0|v3&!-&mPs8C{ z5Kf|(Gu|WR5);#bD@A5URCLSx_v&x1+OJBgwHT{!nxe^XSSu%BM;Qo6hWw7{z^B15 zVCxS7-d1bhC#geohSNOGrf7Qm`o4eZ6A%yw*gn&6A}SN719>Si!5TRGRDw|{WRz6f zJ)$gO-{D2pBW~IX=Uhiy`j6>?i7w}rNftU;zkWC>_hp5(5GmEO{2%S4k zE@{whe>gvlb5p(D5plMfq&`c`A)tz0%b3QjO7a&$)IrB>9kx7^MLaPwjwD1Lt*^mw ziOIs#qTlJO^aPlCvsiD>-O?GGUeFzPzV+8jhQ^wtZg;)!0{$TBbwZ!oT*lF;u=lpC z_X2y+j5Irn_9DpB?u^o2!@e|09Z9;h2Akv8)|b}QsH1yewQc}Qz?&P{{Hy$?GwRwl ze^4<4Sjh}3jyZV|s~W^A__l2eFCYJI!g`j9eMw^GDa8JIV?PdtSs)+>l8)W)MoF_` z%x?vnKQe+DKbMbg4FxHB3lq&lPmnx@dz<3xt+HbONg?+}vY$E{J9D#A)*%=727Ud+ z0f~1tY-3~W0^FI1!bC>?Ifq->pB|+13p+Qh3yeG(o-H!@oRJM043}g3VMNRDAopj zM;ZddA|-#g3zUyrnczd3$a&9Q{ddyUt|u~*>UKufI1{9cD|dpFo~s%qE|~l!Jy*c( zv4^ctLFI#E3(`CcAD%Kus-^hi^O2AL;6W`>e>s+PY-qX|f*~YqXU<4B`XG+N)CKWO zEkP;I(FZcgH=&}X@J{I%e~)9{JL(<&X{D!$@Z&1TEFc`@tmXq|FiJpt3L{s15D^KzuYz&`erP*93spK? zIXO9rY@UN&B>BD~(!5kv*HP^diW6!7M=xZczaO<-{xTQL!rID2bt-s`bJZ&Kj?}lc zk5h)~IVE*`0t0tOzFk$Pm2C3Zc^RYG0f+rfm5=r7v~dciyd5#Xg=C_1f?q_sxOY(e*vaSlfFN!UV=UR*op+Fy24#>*(xv6r#`J*6TnH{{WyL)nAu$3GHCF^_xXpY?qDrKy}- ztDzAGe1i`B!g!@mgsBR-y%#e*gvhQ8t_;51G+}n}h#}_zy9n`#zN0t>tjX(P4=l~d zM;)KF^HJZg+u7-lDr z^~m-1jK2zY-NKOaJ_OYmQGB_1g@u(_UM$vwdFvhXUK<1A&FsnS?79JWt7I69FyFgf z$yLCiPh;gikTVXrnPj0H0WK2Z$q3=#w+>S0va~clQ-0CRX}7}b-o02LwC3=<5OM8_ znf@b1W3PS1i~Z42cH_eSwIL6xuc^L@^W%<8uvA`=Y87MiZo|z&Gh6jSLn3bjo(6oN zy`^9_BV8RAu6HYE&XaR<^)?*!?gDcTQnyMV1SBYDD!IDmWLx6=mu4~}U)~1qxg48= zSY4r^p}cU!TzT~v7z<^&Q$ssp-II3U1dG_~=uQ|8n}QB?rC@rr zY1tv11t&*1K^C*hXkAIBXMTQuFrXv&u0eT*EYDa7n;uce7kuH9mPVekYW|bcFH%>f zSQsqvy1HWCb6qCyoW~4zA=Q@GgDcvdPb(C=v@5QhrLRJ^IsA|05s+kL`fkheU}AaZ zGSwN}CP+L&_LO;IofkI9L`~QE14dUQYjf^*>2osN&s6AL)EH~ZZl7FouX<2jS5PFu zKE?XjuV>u9v|Ze_h8{VeO4*Qqo=-J5el)TnVK~u0PW9Zo6U=h!rl}AQ$(^l@4LXY! z-&HV)Q*~SYvNQ*ta5el}^0YTA2&%nGjhtV#8Mu&FZE>Sv=d4-!Gb@K-Wv^3V@-=y} ztM`?#P*k+55X!H6AS!1+T=QGzL!E7~Ofs);S81K<-R#^G#NQ%j>@8TB7==aoJ}ppo ze{S}klf(RN@81_3vtdO1k(iVO8@ev8gRh@ND|@Bz9XmEwrykMw^6%wnq^iNT2s|$H z!E~x%zFfqQt5-+7dUY!8nlJVSAQf+uQs4qmb$cnFJmoJOk#(|IBNWPaCF2)H5eagh z%U0Fqjyj~=*=rfr;<-%)^8hV4#QUXB^4p*C&%NE2y2a-Vrd7WB3izIB*XJ=J zTvuf5&5m~AU$%VqGLFwpN5nch9X0ubY>B0-BRGZNkG)QXpZkb z!dijcB#F@oB;t_H$=Lr1Q92ivkf2~{dY!}C!66maP%_mIHrLD(@;AI{73Fc|x3%?_ zqJO;5h>hMHJUC_^&r22XCODF=HJXrupHsd6s3;Ea7XdaDXjaz1>4i9`ImVH ze*0dvcmMEc$z6rMyQ(yA#4xE=yJvj3VcNv_=@li{%PztE`GRE!r$=v8Xy$`{+uEv{ z@ss_p&WxIM6H;e3qH?Jwy#?|2DQZ(xxqoz?yzl2dt@1Y4hw-5$&s#qM16pt+R@S7noU%p&sxnz3i$4H$4p<;XyALb77*r zn4$kYoQ?2oPDBbjU0qSy<=vvx6qM0)!txH^StZ_)NAHbSOFv$yKRW|3vwe3;O$V}3 z@*MIJ(0G78qXv;FKrs!YVh4c0wxnNYP6&ma{{l;Q&4)#P$z47pp+cdAXwerMp$MhB zf1A%xI8CR78CP3*cDrTHA_Mc)3RZfynqfGe1oqnVPv%W5Jj#2U!$KdybPQs7auVKO zJU{2{-~0rNR*!2uE@sRWUb|8^r8P!pfvZyB@H)T4a~Tm!f$YGq zX+?&kwUa-$C;wzIkH(f?yOM|>!ecx)OaAOl4*0^FKDr9`fd2!`jyoA5o8|Z>FJja=QwTstU7nnDWgs!TaBWkA{MziAIAGs zf$lb^w|8uC<7JnS-9EwutH%IBnP8 z{YM%B@R5O&ZPUH*-}M}7>knS&4*NKAqXJGrt(~1(Z)`XiaqrgWMWm(A`*JFJl~yV2 zMH^;|)HO9EkMlK$6m$ng5=Z}h=H;w5b>*YkImfI67up}}T)9&T;+fF=GYrYv@7)&} z;q)O;cZIt4CSdl<>d`cg_V8E;#jPUpXAwR99Y+`oXErFq&{m{Ur?0Rev$IFSO#ZFn zz4qz!nENF%9)>SYsJ+Upsp35#9q2l7>*>_T^}hH7_fC8QH=M$VvWGaI3}!?4b4hUa z$DNRb5Bla#93$9zUR2DtR&0K^qOV*8RwUNF(ZBE?fJT4*18zr;Rl`W>I<@RAj&9g0Suv>I}^KS;U0PAn!VEmx%{ z{fFmM_o0(EL*+4SIfKMSHFNq1?%I`5Y2Lw5ScR1P8T3wDnK*_`BV7i^RH`Hq_1B2KJC$w zQnBe@L7(q9Q?-Hs3@jm(fa#h~=P~4)`FDJGez;46Nk)OsyVM_v?bs1pXa1$9Qz^&e zuR7X2=F=3vQJrNlu!J|)S~u*GkbLHzIPTW8PJYjl4kbNC-bJj=Wt^P6O!PwZBVACm zdwY3tw@(PoAgQ=$uczJ$HEr*9GKE@=eCpFSpZ+-NYAEkGi{bS%5Cs?&hjO>0t5AU( zbaz+-=F^FZiFSoseIwBB6w&z7rOcB9T(@G)8m;aEo2`{8qklwf4~e|A#A17{_1u5n z4otqttc+(TxvN*7iY;Kc-+3zLzQ_PYAj;mUP(nCCD^y#_KhZ$g zxv;lc%J=Dzyob84RpU_pg-$-D1B^;@!vDddB<&Q6$ud&3uCYo#46Cd%IGj*Xfrgtc z`p`8fC+*>W=ai}a`T0uj5S^N`>EHiIuFa!GXxvYj)#~kZYR7{I%Ts%Xl*avqSu0X| z>{Betj^J9d_E`OvUoV1R@D@kva|1i44P!`wDqJPi5RrUA7$)TnPB>UGD${&Lr}6GI zL(u{CFRk+JjW{BzGGtcy9%+ng%rvbkTG6q5UFb{0X@$NU*3ZX32wn_b&l)MynLqW$ z@(;oBKM$l?MylHI>#R?csfmNk?v#B14`cQ9LKH@abmm0;iFP%5;a-7(@c_VTeSdp; z5qcmN^o9GDsDD{sDaz=jNO^F@{*D$$oc)qkDZMTJZb$vjmljRyKjKW>tWF^W{Qfpz^n{K;E9Te zc=npywh``3Q3d6uqN>??wGDa#je%axF!&frG$2~Rk>0I}* zzfcsDZ5MSR&KsT?LDw>BEdvOd@`6=@UWEqb08&CRl9)(WWcvv7L!hy-fabFf>1t}; zd3s|*JdqpGe#pc`FuI*+b^$Jsqn0R1RH0q-nfmfteugqP!I3}S~=B#b7|C+?^E2lois%VnT83HIPlH9UFuO^ z4xyDy6OU=`jLQ$k-#+v`oq#}vxtVE=4*o_j?rL-wj05Gs@oZlrQnT;(T$UZ~SSMzk zDT0VwJu}cXeNP*t*jt$Zz#YDK+uHs&f80w{)UnXy^6%bVCbsQ!UVzV8{a05pB+rsq*VoWIzB-n=f0JUY<*u0E}W zi^iMtfulx~MV6_4K>`_!r%%zpI}c5_IVixS8nsG)bJx54WMEh3#z$K028^baey0r}(@_KhIrSg&q7QC)LmZ0)MPC=~!TAZtk?1b?#u| za}lG#(VeG#hpoXr#mA$k{fmI^qf^lKDIy6W2$Aj`(E8!=Hr=kzjcY#pm<7{^9Ra(B zyZl|x-+$vcGG9=hrNM2$r`)902F!XxLqbdd0+WL+xe^gCB=q&RPe;(d4YlGASf{O( zdyl;++_OT$!p#1SC<9)sa5#E>(VD*77E_yqS_7)$uY7cR^Tc%Qd~MCxnU9;t|JJ4J z?BGErhvBMM(7HBsdxU=qjW;>IyXL*veKwRH$ezKRA{Ih^i{HfCO$GepHZ>49gkc{h zY!IKmIwPMV70|FfBaIiOKN6S-GAyqhEu! zlh+4rw?;PPi08QSnb;dfzOmkIZen}Syz!`Yw%BoEV&td7V?xP#H~Wq`yxDs}h5QQW zQmOR~q*EP9eOEo9!w^i6@x)f6t>NL}boAWf#yqxvTyN-u)IwY459okp5!VM6ZY11Fx&%JI21+1h$+#3nMV;e+{LeH{&ry@{C8)&03e#CI`_?tvK~4L)>OtUzK6A{z{VhorWp zrUrPN#<|md$SjoKZukps%a!Q$ZDMAk=h4lPz)hYH!bZg}0q9ecOUJo8h0KhlrT+$n zD7>;##aV)Vm(BjJGE0fqY!!U*^m$Q{(qY#efiwxNw(})E{?j}IzVgqF=$hM_`rM|M z;CkOqsYO?9Vqjc_&yzDYrVG{QeD~dfOYPJ(avui=;3akdiS=pv6&Y9oT%@iQl?p(J zs}QG%ypjSDLQw$?6;(NPitY0Yt*Cc)qk#}HZ@45}Fa7Z}3lR_$rqIuF&M`EY*QU3CCqzC^8Q;|kI7qkb-y;!`YcWveim zg1xN)-rVcNoGR~6{-c^3Wd?#XT9B$Jzv>B&XY@@|xdWsS#I-Y5oWDoac!ITf-GLAq zu))jAOC3bD)^|8}*|Tn;oJD(56a+dNwS)Deirsau=co1StpXo+cIoEAnWch^7r+!I zG!qFLQR;926M~LugnTvZlUmZ!AW_4+Db9LyvR}yXMVeQAP1~vm|2z}z7fV1VFO;Br4pcus2Yz7nKVR0y0*i#}kqwJ7s@^&6~faKwUZ(@*0kn~`VOh*Jsm;CCTN?Q zuz!ojqU&87Cuj?G?6XQevk|F?3OXj_5$QJBsYiO{{7HNE+%?R-RYTOhqVW%Y8uZx= z;B=@%s-Hd7)`-j4EOT(x&w4w|&z+s?HXx5=_ulYsV{ZOW{%&e9FQxG)4DE>-62!F| zRXAKQH!=*+vro^UlfwQb^m$HBPATi(XRfiTeDSZNp?8vY zQpwJr)pGSm*S}JjDu1E7$Y6naT%Ol-k$J6z-G7sOwm%i7*soAQRVi9?tfI+3kv+cW zX7YGOeyNw@*5ELFB0lj~vCf}9S=veaC1k{z2faFr%3mCJE919@77T+Qda?$Rox5en zhXorm%_q7v$f4qri%X-EINceu^^{lq@duNId{Wme@SlH-A?o`r;`s$i3@nAoH$?j%kuBiRX|56P0R(N{8 zy%lp zH`TvN-s^k%nv9|!k%Cr5za6aZ(#YH#BKg?dn9pETWp4bvNPe!BAJF3%lxvU%Nlz_; z9=-e6WAsKX0t9FE3=k#=q6(ssZ=nGTz$EgMa0uEo1XeQ3f$0n(!aAlam0rDaylIE7yJW%CGu}q z`>bYhq~Xty!A%)YY?{gQT1@)i2 zuXEdGH9Lg`1xdnvz?Ylc_42LCl7rK#F55Yc zl0nozNDit@KDX5AQ=?XKoLiT6=E-~_Qrx!$QO>XjbmCyDQ;dXv%44q$_T_Ea1Vjm< zif}Kl<;-&D-fZ%IYv1WMdX)??AH?GU0mOnchdknTm0jeQ?}X3b#Aob9f;JwAw+3Xr z$L@myokqWXZotrw_RS$05&EF-sQ>)s<+v0aNU#!7ZSHSu>n_ogU`P@L9L?D9=U;v( zo#Tg~$6ywEoMXn(p`8;{_5^io_kyPeuQ%lU#Uc<`XJFb0{AIw*SabxGIaTYNU=W^_k3SCMM2) zzB7{dsM=rwqG6IQI%HaJRwmV`?{BzG1k(%=7Zf5ChKVIJ@W(91+RvOV%Xhfv?Ih>9X)1*AuDSI7oU{C!S9s06d%=qJ)xlY(L zuUWcuBU@L;ja}=!A*nV7sTP?4k?$YMpP5e$naP_X%g*uy9=iXJsGQ1wwl;ACg4*Q# zy)&C2nnPpK^A6unpTD5OVF3=yjlSCFuW%Fe!3r@dd7+mlFDsiog2y?mVFbpki1sE% zkmxfoEp<=Eou8-karzpM=lOYh7jwk+*4RGza>Y@^fyG>bRpe7m11!1QJEG&OapNO_ zKdKM~65VRDe!*#1gxLys&chau84TuoUsYY$JIJ`$+S)=LsAm+2^r{gXgcU-B3y=aA z*H1h4W=k*LQ)@iElHa0UB{j9AYO|2^2A96Wkttt_O*10WaRQ&#z4Y-Iyff6DCO#IB zvTzb8MKv%MJJiN7_xnfu;3w4J$vampVw}4Nb~+NE`U3k_;fttoxtvED#|Ay+D{=fN zjjx#ZuUfP!+Nz7sTGu8XtA~3~^}mm|UfAR3Wv{Gm>7OJ`AOevy9$d{bo`>|R$inFG zm;HBc)o1#3dMV1p7!WRKd;kV)(40_kB`%O@32F+kLWvheKF80XqK_?jftzh)6b4_h zh64`Z&J*^+w%o|-J-FZtI+S+phrCZ{BO!O|zIAq|_M;O8tm{$7p8I6lRz@g|x3;u= z04u*7Ekip%Skb^R7N6z57+6s|@9f=Pi=@am_0g4yFT;z^rpwH9AfqS0sLvGi0ORXp zKe1U34yk`n5^_VQMj)=%~eEr3rv8Un((~*M{q(;Wy zbp?bGi-76_2FkM?DYGRFCmn?zPMowd*zX;6mU)~w71v{HLsw*_@uk~@ROB2Rohx0s zZwq*OA|E^L1zxu7l1fdg&3z)vW7yQ7Q_vQ}H?X?roKD3Dvo$kRDG;=b;3oe8pYcNOd-xVa&{$7KUfHr zAyeXJzJC3hMyOy}0I~MEc~g?HFi6eh;3rLyzLDr#XJcK57p+LK3?oLywvQp@@^_U= zf4H!}+ga#zV86KD{f1u}LqxKcJ7}yC$#|5lCNX|FM(&8=g1%eo=QSe>5=Y0M)o~{m zpH+F1(Qs&;_K;_N!Q4ZeP~?Ir058q0J!zjg434un&WX{PhR`B|$Prz>56mDmi|#;y zMV3h5ck^S=4cboRBb8#`sYW;{CKB5G;bY|r&IbW1@OxmNdFw7T+($CWET^m+)=Yd= zix@o#-JYR!!-eHsVn#3MTnf4CpV>KhS9Q$s)tvB23nS0JHTSB$E@`aNcT1!%f`*(0 ztC)2}SFf%7tP+lTNl8e0mhL`4bZ*~Iii6$ ztU{S%g6$j8abwnywoMno$vbiUSJwbNNftdM_FaeJry z@+yiJn6E5m$u~>fUQ%{}asZtl^aK)To)a*`h^@kb0_nKi%#F4h6EVG>^II50XdjaHHb;&|7*4gCG}21OZK7 z0|$NPw?Tmg0;*IX-939Ip>rJ|r45HkP4O zDI5x2X%;X^9IjrsZrz8Qzbv8tDZ1=GFSIK%>T#oMu9Au(HOQ&UouCj~FU-UW(~`BE{k(f}FRUADoxTEA+aP(jbd@#8t~+?leU!lVTIdxj&E)Ga%aPdStn-S4xgeoqC+4T_)w5j z0b~}Ik$E!tgoXr+X71#z9UZR$_agxBfvV^e7tPATT-3>@H6$7OU3o*_}9p42x2*ywh5DbD~j5STVPa^u~QsO4v z^B+>B%0e%h#JXzfgcR+v*_``Vdi^zN?eHgOZmv=$0?xLKi3X$|?9bDxC6 zH{6jJ!&p0kUMCI+4rS^o*O@I}2H zepC{fzevJ=*<6b^xLS&y7sB3G1xa9Nyi@|9ZW ze86yofwi4ciO}DrJc--HC>F=oNM=N97bzt<#p1Ms(|<>GgASLQppj+vN>s<5=XXRb z=CG#9y$?>Vbd8o6do;>b{v|EC)EESJqBEtV7_s zM!&C!_AA7jlDd_84^;Jhn1hukwAZEU;YRE`yO@Z$9K{tC%+E(IfMgnsco9Q1#8QYu zz8n~+5w7_g6ZRnI_pCe+e|IsXcp61)g1)HT0_u>i*R%IW@I5Z12QSDy9*em(7UK@j zE*b-SHf~mu!()n}=jk=!B7dbFI-8;aCnV?W4IgJd{Ebkg`>jo(TNUo*3uX$3-K}`W z{@}kPO5Un1AW-j&`AH1H^aCf#dNnQ$kNrk~wTnSijsj1~2rlu|n|)>MBQ*feC>gur z%72Kiqf|C@W6lDz%qK~Hg!vsY5o7!}#sP5z%K;iB8-Zl*XWa5$O`QFxd@1T)bf}0+KjxHgY39 z2OjnZplExIPKh`)n82Xq-$k1{G9W@Db#n9+-$1{DGKr0KSa^A9&^JhmJK_1?0ceeA z#4!^d(^6WAm?#gPdGiP0pVF5}<9$%Z7s#dIpsuk7Lz+{}ds#6FV^xM-(;t=F)r{ zrbHq4TfVlR3}fy|>s*XscpMxT7iTdP`0$~yu&@$F)wmX7F=o|M>Pb!OuUBcGVbB-o z>%8x(25yCyS!K#oBO=#|4~rZ}abp?bh_Jh+k3hz($`Ot7+sTu1-^9rZH9IRO-K(5v zwofNRS_|@_(g~4+LN9aT6NbJFgdINhH@t=gm|4nq)%+@y7p;nQ6j7=9U|r$NGrynx z`JbUgwFBlo;)7=Gd(aT=hD~T+A2vtgck#&X7{8wY;vI5jEOGhTnPv)`h10NXv-P2= z0|P>aCEGwWsXWm;$X9;v&lS>ErR>1B84R8eO*s}rv4Qy=@dJ{>bz|GKLqihNON^~& zxL~zJ_ajsbT1P5 zAwM8Z8J}+ZP0LuZ_PpO}@$&eK5|FFS%|0rudNw8P*foRhSt^H*E*;>j{#bb0X|Czt zdH?R>$5XJk5}H#09S_m7IVOW~kDuAs7yhK+{^(9_Lw<{g(BnAE(PJaHJ+hiQ1ineW>?+Ec)k;el|L|O-!Y4 z&Rowb$y4on2A0*Ws0ha(F7PvU_`LM{dy-BxIO?B2bFJ;~>rqkO;zn!5#G>X8iDCQ; zkGjzEnVu|V{d>8`(+p6AT{Xc3lQ41`&Y|&7EJcZS@YHoAx1s|S1|B{xXwTE)N zoOEo)<}59Fs8_cEdJFn#dIHN1m&)g+62fSr&FW#1V*(trFAFMR+9Q-=9R!=TyRUs7 zGK-+S6gl7dyFEL*eoFl`tVZo6Osh*ynw+Ri7`tz+iD2@6!R&Q{ypo z({ZQsE&_|Gw;jhP6pu^;dF436G1FSII@ouS{4pKLnCi5r@=pTq9Y_z!s8T*LDBrQ^ zCkc#3Olw+OsvifYUDjFyOJ$}iC>~HV@xc8ZcuImlz()l$m71``%{P5~!pM%-R$p=; zsv;1Klvj0i>D$vmroL4ou-)^L`Ib8U*fn1wJh-IHRJw!m6g$WC_uLt|5_4sIB^B&q z88O6-hezrcqJP|AyZlf`cf+HaQ4l8*+;EMYu;CHEgKt~a+3+cz^(qf zEU2y5rl!vt5%Aa|o%j!-AH*Gx{EJhZyR7pXlaA!oRbsW~s&YZ2!5YGk=0$V}A&kuHL7^-)I zG!1j_C7$Fo@o61tF+AO2Edc2O_tt1xfwc0mKkDCZ30vkP)}g03K4sl6aU9R;X+VJ< zekDs0Sp{%s-Y7OayWK`x+g;zCG>Ub0(~(I@_-DeZMxl8RTxtf~y&30!7;mf#!wlx$ z=Wov@s7KJj_?o9UKeKDk-F%`5#53@Z%|Ta znsB^{Zlmd8sDb9n4elo?ciS4yGra%gyzywyZ1E8fubjEY4}6T3pYgpqe&A(mjyA94 z0rs&yorTi3_dJ^27V2)P4#n%^Q?2`R?@$4L7lh}vx&a61UK1g-<|m4Z>9 zH(ZlG`~IhP8Jm|470-yTO>4gTOvLZO6J_g@RTB85f#0c$xLqUm7KvT z1@EaCp*Sa`1K>#n z46K^{$A&5F<{n9Ya43~XKZ3#NfY9=Ktn;d8&OAZTMqAL22Y{k35H7Gg5mbi>FLBeT zU*bSV;5PuDS&Nwj!B&Gv<`5PiMOU*49gGf8b6)7ie8Rp?WpG57VBmv+ixjgEF~Ff1 zfn73#i^G0Q$px9qBU>3)6@~pO97|XlTWWNxb?^L= zLB5%IbOUIXgF$-4nH%F*1pic!{R6+h=4*=u8Q{JloHJE6vIJwe+?^1YP0BXhwF&0w zBYi?8-+gF`SCRXK^#LrLBLx8E0JKb<>hZDl3fa1{dl2B8$}Sm6zkU0*Y66B2LU47! ziDAiociw67f-FGU>+QmgE-Gul&=@^&^VNsA>%?eV3?h1xOa}=IF|i;~qe1hDZ%2La zPq>gM&YUqLdTfiBSb>rM-Me?^JLRyEUPAsu@KSh#No$yH0OdufCh!z+M@H1;e|tGY z=nBs6d)QF`{R9|FLzsOhTFIz^T?~1iSEhEnu?J+6bh8kB=&;u5mv?X6I9%6<;YujU z7`IHI#dyz*{p6H{d$x?m4R8br!(%J>7zxh1*(jl!3hhWK3$dN5F~r-y6QzO;-l^Wx zE8m|L1E})?p;OZ!DFYIC%b~PiWpK z^qtaZp=#*s13XAG<7Vdj>E6Xf`LwKC|NPXz$vR-rTBBHLqo(LnIz&_{?DeqK(2}`P5722cy z^DyVM!flI34OaQtBE6aa{=1tE!@}wfeBVC*tBXZVdiIU?y&kV%U{g zA=H4fb-;C!GSb|q1qnm;i$Bm-#4h1)e|$#;=MHb^dIe6 zg^~s=rZH9Q2<@DZbV{v(j|#ZOPJ!rsk~TGh_CT>eM7y{z3k5Pz*HWw&NGVzY;t9*} zc+AF$v1X!>JEv&noSEW9uv&9THbL$KI{@Rv+TVsf0rLx*^Te%Qn=(OC+lr>N){3h6WSRI_LJci|2AYqa9A2?u`u8a8SiS$ZMaaPaYUMC;*FlFXi2 zNRI+=!O8?czD%Ue!l>;Y7!SQLf2nGY*#)9Dno3p?+&sSiSTC5gF7zmDuIT9TL14Kc zZS3{Ea~ysIp@9a|;`WHJum5_WYMLFVXa3-c%BEhavH5p~GKO&fK?S2D&5+GS zb@lz{E#G)0KK@_y=l%Cj|66B*s@}}g$nB&u1fi9ri;0KV!q=ajI~Z4mhREOstGF=2 zwQ5V#oE&9GztqFCX;;KAUs*t)B}#d^?DSX*LFCw%EcsigUO`*ZIXlomzKIZhkP^ih zWq$mTJzw9PaZip+G)Y1LQ3E?bezfa4o_r6v!H+APV30?Xajc}9%qln0nA+-x7pBn* zQ6=p`WkkagKIDnpJNoOXRM*w%Q7ws{3PYekM7lkF@Wc1%orU4wV$_lOU$^54CP_tQ)tB!btn4n_|HNUNO6hCuI!~U8Al{2DOomp0^?B9T8Oo2 z6&ZMDfqi(^|1iVjz_9C8v`nPr%CH|coy;g+IjlYL&z9+IF|};h`$m8visV506C5n< zxs7RsXz!H$_AfTJ+oywdH|wkf{<(lmYB>Mwozjo9M;OD=4NNv3cHh1==5x8YK9_&r zkQAJ#momaif|XW?Nj@1xa#)gX2%s3SH~ryK$PuFfF&9jB+_9WR533LZ{NZ)Mf&Y2Z z+kV zX?|(#<;;_z@5NS1JW&`ACZn}0iPVw4G{TLba`?_LIkPB0B^vhM50?Ro!{nQ<0#I16 zFjOnVHb|Lh(x#oDP>+uGr#qhc{>%5$CMtD>8c#4#;Tb|&mxTwGj)u7B?>zu8pKwV!=HHB&v{(C0=dnW$-Jbb+e{%<-H zR6;29GWYIf0Y)T4Z#W!f1C@t^JgDR^R+Sp3|eiYnfmpEORQ_pPP#+bdETG)8( z4XOz~dsbpz9fBMyvbX~P9|^iTZ+P_kTD$r=r2VD*&7yX2o)TdIEqm!PEcQKSxQwR| zopf$nyWKiac}HUF?T*Mbh~^pKa|!{jSX<+7CFx61#F zD)u;b0A@zAedPw+PSpe2Wj%FTP7c1so|X|J*2R(t%q+z3k#Q`OQeFc=Yg~hI+H((1 zy)O!p#K&RD3b8a7wM8fuYLKHs+EdblyL+(mVatrf^PxIFQTTdJ0&gqo$>%HEv_Re? z4#8a3g~5MT>N+ACJ`UtZlHk8WGKyLKl=!X3YL%?pW7A;~#Jm2;1!5R*rd*BrzG0Zh zIwo$Q+cLv-hRZocaJWmLO9!w|ySUWYYzCt~xTCBVZ50{8SMZaP3|vf`oN@&-)jR#FI>T+waM&R| zAMi0|A=z{db)+bx59W)UO~3O#0=OfZeFix%8O&C`YG`4HzPkF0U`^r%IQY9^^9rdtzaZ z!({6s5?27=9I8kAr#55}bLOHNzaPx--uwYym0~oBg1%$@Sx8h}2p-V8Y&tr|3!EWU zxYIHF>zvcKRe>r`(YVraWPyJL^Iv~CC*wy?R1dE8oNtCsaP=F=pcU`x=^(#}6c=g_ zU?>@^v_F3Htu)Jc08Ly3}HC(PLFqlM_B71mGI zzc-n@dWZ@~?sv;7B~8oU&SN-(*UKGyiV~#2pV}?O|2h)t2vGb8jirW@JXj9PfMQ3b z6zsee@g##VPA1i0*M*{ghS9qi1=Lp&8TisrE6)%>QgP6XEDFeqg;(0fG-<^6{3E~m zE74GJNS^eA{%l!gicNc{mU&X(_AyLF4RIfW#t6wGHMNkdaJoy(poO|MVf&K=*U8S; zePD?r9VOwJwO$*p7gdBfH$)m5`0|C^?5WhpbE(}iN(rFQ*|as0jpanLi7}4x&JaR1 zRY>N#t%|IbK&|6hw8$9zXA%^E$zqLJ0%jHaxSfCzK>R_2B2X#AcDOAf+c*+`9Xc&z zE2lwfec-5fOGTOFO$z0c2doWn*Pt;)+M%_JjPMB34LN zE661H8?%>&*JJ3N%5X`{owaK%8}=Nln@%P~IEV3o-7c1hxd|0KZJ*NeoYnJi}!&n%sSE?57(_-yo8h3=BI z!N^E{XpmGt5Sm}68v`G9b-+b^S(t_(_w1x==IX}Crph~&JyJ=U$DI35WjQAQQ?I)g z)(sdDu;4Nay*!y}LTpANr(L_ka6^6Wvl)|T3J;!5*4*LWo&CfM-9*dde*aavE1(c6 zMU==Y@(rgE7m@9o`z7ga&Pc_R#6>{=WyTlGGq|sRU3RsX?Gr`UN7;JdMzmMa*lHd_ z3|t-@ug&FScijMhHWPab*Ad1+1}E#FhPpg-iAVa>TNk*b@9#(gx*wMRQ%0a{7SG|> z3iACkfyDxMyM~UnN70?ysPxff3G?cYR%;=Z0Utva#>#=BfvHG6aDpQOx7l3Tp8Xz^ z--0LI3Y4`N2;wr&vM2j7?IZ0c*5FaW$L-}Y+fF{|x+rn7R=Z_Als3UI?~6)?J+80$ z7&3sAp7dcXA_YsweZZwOz_NV)7@w7Qd;K;Ty;8SDcncXXay!9FJ1p}W&g%tXA!=aM!Dst^inEumGn}3?rVK;&x-?V?5y5Nwi^pkhK z+!XWV=AikLqEmF;0#7$twn4priO*p-o)zozMu;A<)|WS*xGC)y)neDKV+w~H?(4LE zkXGGrHa7Xi&KHX_%a}2tqa7+9{Z8oFi-6$oav1XY9N6zu=taPRQj#=2(h(>`jn(&$K0 zTc#69VvPmUVqqJ5HV_Z&{|WrHt3mR4kHMufxo5~usP&km6R9HnO)e-z}nBmQis zSV;RgMEtjYDc?;(3v1kSWI-ufX7m2vnb4F6ZC@hnJGb*8Ld~(_z>}iTh?7bHFSQ;l zGNo7QSxuH;g+`*a^_($1pxb{Ixl3R-Z~biN@h-o6h)kGqE~G}nt05T8ytce`-O}jL z(9d$4!2rgsuX+57Fq6o9afUZ`URxFt9=VAH; zY`L@zMFNoUNRUbU_{8v#$|ml5niBsz(uYe$_Ah11)%tXlpws8xHzSKRBw0PrZK3ce z{I_r?d+Fxka9_adA;_g7ITm?rgQ3M#1)fWy@CePAKLDSWsFSU<__6F;^B0GUAyuKb zvrZ2iUm?z#HI%6j7uTQt@^1QGU#a+i?Q1M@M5rD`Y-T8AQ)}eqF?WG(l$K~vdS zzPxWCaW4idv}%DgeAQ_Pm4*j`Cy}HB_O}*RszPjyak#ZFohCd&x+C%VgRHQQrWV)5 zx|D4LK|HxKaZ-5T%d^cNMti#mQK@;P{pCfq{aF+tBB!I2u>c|~HW=TJHcyNlH=W2| zmQbu)(!VtZ^O!I^zJ14-*hi_vg-lB7$u)A1rm=PNLw_5kkr$W)1)!!aY;Ll^1lH9%JEQ%1gNIt-m zL}}sp$P%pI%b$5tk~D5L>%^K18+n}4sz)kQG%@DB%u>a2t9gHYX!piPkJI*rcH3V6 zTwB!fHwg9EA4|`5Us?fWFF-_?0gX?mI!1Wd5^lWRak9C}gt+)j;7?QR#-W|D# zl4tAIa|B93i}?DmKQ_9L2)wb?%s!3hntSnYu$YNJ^An7JV+FoR=L{9otr;$`>U#*r zY2Ez@=mu)1W#BkUa5mq+wt+H>glgRV_hS!5&@}w{3vsaaonXV#H#AwX5Leg=t5cQO zRWi*cic!#>JE|hRRi+ZDpPtPU{38mD56-Nv#f<2{?x@&VVj0rXVoZ86mU}K*w#6a< z?_PiEE*u>`B4bCy_D&{dKx8)zRSG-bdF$co#M9hPkPDEII2HR^Uw1OLE2^d({bA7a z5z~$24{RkFiznW8k)TNRB1W;&YtI=}_*Ur3wp{oL&a&hMgP~3Kvao+}*Gc$9{Ch(A zbUHc&bZ;OKAbwTN!z8L@NP!LVI4Q%BBT-^>>UfMM#kyQ77{L{7zP!p0{@5BG1@|>^ zj}-#~eCVG8@(1drocjDS@=2gL2zbC(*Y?4~LTJsJke#*s9tQ`XtkiYuICD(|OdV0} zUdGa z6n9S8wh=_2ko$F!0q?in=XywIZDQ$bYoYP>U)zdEYN}{LVNx)JZ9TD^e z_Lt|?8mHdmMJk#b(&{>;sUZG}GF>r09bsNI(huEXaDAxinSfEg#pQ3G2E;#tg_1%DzwOI0#7)vkcG9za1$GV=}=?YM`-Pv~re<>XDSJ z7ugQVDLF6nxL)Pv7Nga&f+NcO*xg{2iR))D5+Y8i-044HE#K6k?xVXRzqX*kX&3pDR63G>39tkrG}+E6n@2&W8F@E7C|v^mwVqph<6MHL=-hs%9n@?Xk& zk&~}$7t@AHDDr4SLm3a6$xS^Ljqe*)1$N(e-&m$?=?=knqd0WGGg>smu6%lEtEFaNF`Sv}|&+bi;KKz01_et!SA6({pI= zoG$kt*}>{{I&~j71<6XmgQAwv+&pqbY0F~0qh9%9CXNFQUK&q6^3qC=38`vJc%iqQ zv)b`#r}jQsR5P$l+A$w#Q)I@5rDAhjCk(N%B%_4OmeDJDRUsP-GhU%hJFT6lXt z+@GjAfWQApMGEntr4|{o5zh#Fx9=Z2z zBX{rIX)fgO6kS?&FlMX$0f%j(k8-Bzzsy=(Ad~WU?8$5~&FGB4(?~lRy`DP6ef`m% zD*3bd0s*T6{+RL8ynkHce%%|eo9FUhTr-ldSpPih^uc3`b3%t&LWA!`y%)YWSvYde zxv96JC;$1^rmZtljaIL@ut}9!Fg*I!>)?@R5i`>18{oNUOvi(91J^E^94Od^wlvJKq327=u(?ZhfSi$Cs( z4+%YeLhB&$2aM3{JycIPXC54N8Y?XyJkd2bwjp(^2gLQhy8l zn>hYAU^KO$9(??zvRGY5PWjs4h%Recv+R-P^h@zN4RdJ)G+gpih62xEer@hl> zHEHoCG%k8j`o!@7>X9Uh;JSlfQixPW!W+UcyPp9xDoR}9vh2DYA2I*}62{y+N8ca-c%ECpXG`2m8s}$e3 zF;?AjQ*qhu6OR6xVL_d;rtxLwd+B)xFNd1;$WJ+%=^EfIOMXXs34CNbOiOx2>u#2MuI=p| za@uU-c$Ve3>JV>~vGv{q#l?EUT5{=c-@aAMx_R@aNIt8?&aAlK5`dMw8kciTbv1=< zLA`d{nQ;~hOSiD{$b}^%{bB3n_)%A@>pjpmNMGC<7TRPGk^tn?xG6NJwdC)$HI-#5 z9op}MJ8xbbt$h&Fce$^phg0m)UWA^j2)hQ);>8bvLw*Fr}TWs1Hcx`y}EpQraei)JFZkL3_M;0$lw>KGN z&UOoPKN|%ikVhsxgw*wWWICfT56eFp>J6QFPSfIFemU6u1*Nr+I~X)Aq>R&fum~qD z8a=^qG!|&$uWGX5t#pA%a45ppm3NDqT{Ns{VLGfxwG0;rG7*z+jV&4V?NW5dDddae0 zrz5SMvTYG4fKfxJf_P;QRfjT>577bT8ej)TKOEN2)sTF~&vd}MtRYsd$`VhU1q8HQ z*GWa#+BvSKn)Cc@|Lvdjshm_FJp8I|v@b#N@Vf?pZt+;04@KBL59Gfsb|@~m&@XqX zq&d{{vaEO0%~2+B+PY#VfQuA=6M9!3GjKQqmSU?BS`O5<6Hw_tTw|JtB3L&uhz~(+ z%ae%<%L&SPuSWsgvf>BoM@M>vC+;kklDg2(|CtcHT7T| z5Zt4rkF1ucIQD6rIi9}ac)BcYj-CDK;ax)+FLl3Oir;Yh$vzEmq6lAqUteV0(c?aaVgr+3UzD#mi>hs9ljhb}STJh$^eI^wcZV#7tgtihpWo)jHl=uKvBl|r(Ft<-0lY|W=+4ks0NC%>;Y()c0mZET&(C51j=FO!`muLy< zZto-q6XNqUYVmk_dQ!76m_9Znj{=$n#sk1#Rn)~~=LBCW6lY8dK{9{Rl{w9kTP(GQ z4JcF;-fOqBu)PML2{c6D4A;z=V8wY;*F;iEyd{siG_?~nhPdM9mS;e?oI1K0Vh*7$ zfbhyPkY7#?u%kwG_O+nVLlvI~2;%||By61o=;Gv|Nw^y|kN~8xbE!)tiI_RbP4`qE zqqpY7aPsno!g$OC%Li=tc0A~~Ph%yGFXsk9$FhbWYACwGM(KrMEkG_|AzJ)m$zFzRzX*XX0*4011ERDB*<0BRY<0zul& zlSv&lEN>N)t+K9SHb#TDlD?Ai&KHArsPCXOILUJKc}KpZ_^rrw$8*Y*j1&87(VG*x zJ9^R(b?0N#3d*-AG4{LoHn?-au94aYqFBOAhNu2SV@3=eC6aV?SL2Tq8nqCROiec!73{__r|aoZEiRKT1CB+bEcg)Pa8N+YqWcs zV%J>fFXypC{=Rsz*UnUPr}EDu-LPUHLA1x{zUt~~f;(A>2X^kIXlkU{rBT#x3moXtCHRyeGxRmjF%B{6wcYHMyJ|GK4Yz|89AYJ$ z3|FgSd;TzJF1W&$U)^aCG37pxNYc_b75Ho5x`Ukzv!^J)zK9vPpUtxwWUxV)$qQda z-C_vpB~Kw+I6z8P{FBTOf{1$0srU&w7bYk`(ZhsZ98C_$*yCs8E@;t!U z5MbX*%vhmgr0ig{yiuR+DM(AovCq^s2K!sV%7mGsqN1f>!BIJbnKy4p5!X{m83KZWf{j^#fFRhY7by@?YM9LL(TA;Wn#_dUV z@W={-pxaxXpNh^njb_vZR&cH=4(m@o%MC}OI2w9^@dx)700C-%O$3N@)--^TP=sEtj#WK4TUnSeY&Oef0`I06zst5J5Fh|`=z zJhVK#yvC;^GabYZb@MudaO^GXXyfV)w>|IYp%jjbT84yA81kcKV}y+^J)BdPQ`xU; zMv^DQH8lQO^8EQW`H`NR$RHIeuIL_V+ueDyF08emZ%$Wy$m8zg<3bm-@*1$|J*r?k8r-zy>_t9A#blD{oGG2zu zP7-j!!BqvoRAUSvd~CpO9@`An`s~C?7cwM|5lq~RQo@@bWg^GXq`mxx^%w7{YS_wh zxf~sFz(wR3)?N;|DwdS~#Hsjz=sTdvQ_RVt>lEin+vcAOnIOy0-^yirt^Xf`%h#?U zZq0w73yOY^n>X=bi@tN3T)YnCh7Luu{FV(8@dPv;1sPS%o1 zRWM#c6G<5v8PLuHu#qC&JmC)~f?PqKl(u#Yx{A-4iBxYYTG4ag&Vr9m``H}L+Eohe zNSYu5W@8pDv8U%Ap^s6h2e8RtNO%EPzHs4!u?gk6Ibl7rnd|a8u1nWy8F?O;SLt-i zO$sl1uxl<{-JjQ*(MuV|aJynKOphb!5JY;ira!vj&e~E4)^<97dCuRHF=g({nwr!b z(*Jt=+5da^(;<$Snz3oev;!>pX1qc0160feTmqj-KK1U;>Y5rJ*mBT#9N0;p`CN^8 zyBx&$HlI?!Hw;~8gUxC6+O^>-Lkh>P^~iCs7)CIx+~{D&e++-W+Gqat5BRq}LO#C# z@%8oz^&jUyDJMf?@vF_r5T%k{@;IL{S{0T7Gf6lV&a5z4qKw$xN{oAk%=@~ z+X-g|{CAhVs4>5PQ|}r7c~*W1&}Q7b{G+JRW#ymWeE9KtdSS65Xr{18H+J$vae`WV zY5UiZ4B9_G;Pc;>BT=vyJtW7`#NxwWKz@&HxKoTmkX8tXPfw?VS+3GBfN5bVpfy^E zuzuzg6&-+wM&SZw8PesJ5C8bS$Jc>D>nJ%Xi(=#v4_`H0j>Q4M2ded(77U|1jwHT|)vUsCXhKR-2%nq$v<;1h#;Nh6dK^77S%QpX%?&Y6kq|(7Px2 zUaO%SMf+EZYC054EJqfCF`h7rnh|W>#MIxL|6S^$z_{u#10_0YP=OGX0MBCZVnsFv zHLI{1aAhISgT!&nLiA{+7T@(_3|!gg*xo&E-1tpWQWEV&%(EB_Aa=81%4gJ7jXH%B z1a9`@YhIPq&dJzv%fo}n&usjgkUYb8&s^C7VPGpvHZ1ej<}U{1$dp3+ zl3Eb9PfbnDBMV^>q_03qMOY(;p~bl%+1SgU){1C??Gy%Y_7-f=A?x7sqm_!TpT>Pc zSM{FQy%|PmrkGnd#$Xzu5v}&$uchd3jEgL7ZEd>cn6&fa67e8?5hzRumPrruNjf(l zJp%9fslMUs?juKzgvzL=NtLTOOMgF;!N|4H6a>&tNV-5sT$~zlJWy=?14>pi1{tMP z8X@I^Ii?dR3qeD!?l|u*blz!~1$lN(#IZ zG}#OJ`A?p{29FY^DzNERr|;LrMF|9SF)-q8_SES=)<)Yaz$EtDi*5|@S8(h^|HtL$ zz15jy0&34?j&U<2MiHlwrEr)X0J#{fOv0YUpo=65GdlptQq_m=PZ`mopTfH}Y77oX zC4e}3FsfbyM?rEmM=?Ze_DA5(kY?Ti%gB?fiQLWU=7eL-!sW{&s8+yGgJVYwh7}%? zU<6!3^B?!>hl?X{9w9#!DKj@mBHdDG%a)|`;5 zS|6I5D|YM95@TqJ1xb8V{tvTTo}NPR63ecqfpVY!<26k&>)B85-Bpo|Hyf3r(LDww zp=x8#>h##nW0zj=oo?!VrX<$BIc0n*!=%G#M7)apE{?O7AN=O(!kZLv1@Xtd~C^lvH?%8=>vFhULRSs=q z{OexzkHlMS@e$f~r(rBL*WUE1;_ruE^_E$F_Dzp$OnO+ST7UO3Auk|k5!Fs5lCwQ2 zY&$$PdJ2tgdS9*b2g`FWW+a6~=ma4D>iP2rnU-+u3c{xJkzjpP$@j^3p1Li#*ZJ)H zxD1D-aNyYLjcc6ubmmN(oy}iVbtGDg@1y7TIJSU?)od1XXc22#N@G&kopFtg+I2^* zEzww#o8b;#MsD0wXDu?!y}e%D73Q2tmM1uhL+n-L9`Ooa}>bKbW;D)RV-p^&_%c?3Z zjJ~g2yY~3i8F;*BBPC;#T<;sBN+0`3RUAivLGYgCS|PRyCbEMsSCAQ@@oYDFD4Edm zg=Hs+`FG>i5!#D7b~QTiWLcG2XkX1F#@DjGKNxLCQ7Qp(JIXS%uy};7vAm!D&hXjQ zoeuyum`OcMPj`0B!nm+3O*VaXf6RK8|Iu};`-673`=;;f>AqpS&CcM<*w=`z`v6#F?$)TUf!%Qpkq-(ZIHBx#x~S{4 zmmeBddIj_djjgv&W}cIivU+>ZVrOr-!yo6jZh7EfTvh7dlWSRUh-Xo{#8B@v2OD~0 z#wL+)hRp`NA4lQ|qY6wxG{cinyb~&qJ^mN>qr>!zEuvryrr5OIQdDY?w6%8cUHM?^ zvER-%q8U%dbIq&1sOf4;h*@_TttR1<$5)(bKjEl}MOg=!7Y*@VuyBk5otzNy%SkDM zFqDv&67(U`)?E0Mduu*HJ&Y#`)R6-QX5h0z=lt6?oBkraO;_to9x!@mdm70s~M6h!$3RcF0Vfp|N73jx=c6hmUg! zsvjQcj^}bifJeAr)r%K90EkJ{O^o-0F7RaR`&e5`=WF2mLFGu=23iHmjO)VKvX zm~sO{rX8QjrWqV6`y33_@_n~VH=hJ=h_`kCw&5yYaL7LajMYE%aftYdnOuG!bCA&F zGqv!aBQ`jwn&f3kR=bgrk>FXZFN>Z}+h%^inTF0Mmn=#o65}!lV~s?w=(ZiGzj9ce zhf{R92v2RkH^m3`fEU&Som5bd(M^b|H`%S%7-ncnM=(5h(ivbOh zl;`H+I^xEIH^y6k`qIo<^TVhThI>blWe`%1;A7aBg?c64@4@4zk@0-a!(Cjq9}Ye? zy@QtCR3begLDII3a|IX@82sUlGKA2r&<%%sz@Sr|cL(+US)f?QR8*s8Q$t`0Edmm& zI3Me>ozYf|0XmFCaSQ#5-6m$`>ea!uLo>eDu&223%z%inm6{{$6(~Hl%-TOURdHfJjrsEFNIC04i%dy=n&wTJ-Q6Ip9KJs{Nvqa zS*R?DdE&!Aed^R#B_3n&zo(yT?*6))?*L%fog8<>g(I!b&&RMEPsAp9(Yam)K}p4S z9Tl4l&i4xHaGeqB96y?jU*<8Se}P{t$3N3on$ECk0q#xq!qoDtd|%FcGd8c9tiP%V zRszJ)MXqM7tuTWjooLrZYV@|6X^c&`L4+ol5s036Y37R;M&Jyp7ntRRzOO7EziOM2 zR!L1QrsXiDDaatajFFuaUn17Av4&{DMy?)9RKFK z6tK%s$~=Y{VXaUET~ONK2X7i`kL||&?c!%i9t+^;k#kHCCH^Q(Pb-nTQu|eFnMG&M5hK()cR>0EGd(G_{7VfDe{JNuhx#)R=5V zsf(egZ1(#0{q^h*EmkNrI%U{3ZpjO&I!X+Oqiq9L5?h&Ev($!l&^C!Uq5=W|o(0jy zwY9a?5#!GfZu`F&OG+F-tF>;-{pA<;og_>ES1vm@SB--4%&LNb^$~NLl7n7!LmfgD zms?iop^+49-ei-+KB_z52>CJqZAD^(0eJ`m+qI%`=41sCz^x>^mGJGxnsD+ifFSFHNu|!zwlD)ir_Y}I`+T(^b(b`K3$)kJw zXI%sM39flwS}9FU7fHlYE(ocjcaqW>7&`lJjRy(?cDvEYwKy4G`<*! zh!F07I-S3`H$-g%YHFW6&Srq8E%5@Qd?p#840hwjg7-MD2@o&cD8?7e_@X!0B6fpj zhbSj0RRW35K%dg3^d44Y0!sZ6piV4m@V+#-_njv?N)(&Zad@(1Lr(^o2hs%oppu8Z>F~3kh)qfn-Nh-kC);aNwQvJ&;6FKoc$NZd z;489BW~>&RB4U72ie552&B$vv@9vfZFoX(GGud!wyq1Yf?K|Yw?Qz|!Y?6XOHv~3T zR#p$R!?H`#8~?m=-;8yxtMmL3lUCkC`0uXmUoRb_H%sK6Z!QL30!(x#@pdw>yE&OQ z5JW+Vz?>?k6*+mcyveD~-hjMW5udhG)+2W>>YKdO= z9zXL9@v~oxkBV4*{?f&Ja`6boF`rEBRd=`Mcj~zTn#Qr`C)5sNF#)2#w1;=%a=+e> z5DWdzMQeXYS&o4Q`)FEktecx#z?B4ja}dliZw#c#|EpK8fD^^r^HZ6C?@Up-q;e=c zjd~FiS|x}vX<})A2JnglNS!d&55^EuH^)=tW>{-r!eKl|N%<%j-&@Pd$_j>LYB8s{ zY=n71(P-+>(bd8?>#9rCeJwYL9|*b@5t^zK7h)isrfN5ddbfQge!$|eC)S_YLl!w4+Ss#z6*b?Rgz4Y#g{Cc~=75pXHJ zkL9=pT}&gR9E~QXyI^Dh1Z#{QvLI&(wZp9Sn2%6L;K1Mii}J`pt~tS7HgC*_&K@w@ zZ5d`R9PvX}i@j3aX*npkg5%bZIGR=4P#n#kb{74b zF;yR`{G+Hj)U>tN;$cim9_^1Jf*Scq=0)`tc)|;I;dJmHkaVQd>be2whh|x<633X(CJ1L%uVgwxUaH#g`>J zA@mp+S8n_3>l4agXzEW)V_K2Dtb%J_G|^H_ORvRyc)$zLpoTaIm>P7~L*w;wSh)cHgBt_-iWYHa$YwUEh0|~u3Xvj%ykxYl%+%(C5tbpt62GD)Jy46Fb5-T z8|JGI06tP^2)srAeq$KkkDK=Ek2lrzMF@t$ZNC>cxB1^%a$}Hxr)o4=I^roMN8$0+$bV=#^W9+Yt>SZ4#Z?avHiSozltWLjz$?Q| z$k=8PvCKEca_vtU-#*09Si8N3VHyTDG5ziF+}rmt+@S--!@3`RNUGNNCjE7W=A`Ni z0Wd&<+|9z;nrfkj=4PW#&t<%5aK66j>*DCi<*%qc#Oz5Oi9W;-gHf0w8>oIub`9JQ{Aw6+nx=9_oK`yulPvS+d6JCP}fuBO#$dw86u# zuVXNHm*RGw?3>bNS4^KSI${@CeBmtD&W{usxehDLnAqMLt~?mxOjY*&p3cjpQyi>Hhy_|_`+|vlWRbpOIQ#hlNx!u zHDO6E$i)8dUiItugG>1hRSgKze+!@_t(HMQBSE=nAmlX0eD+9(W&eNv`f1K2VySuJ zR$P`6h(!W0W&Q|gjrvNWELkE!Ee!<}7D_Yj@ib1(r)_e=ku_eE?x3^FY_-49jn=sk;} z`$F-4yz}w*x7_?I)=$b{GpcxWj6C(g{S;tgtXKrfBV?Pa5noo3zg$X89VD2EAG+yn z<^$8IOg#NS!nFJz(9bZ)me>pV_()f3jCFwqH<+`wRflx{z#C9FfPM4=jlVIvGe)UA zh+7umm>?6myRUJ3(0OE{P>|3cpwSKgbXzpH!UzkCi_E2M@YfeHFseJXz!Il`^N3+J;RMy7O9u%{6;D-rwXea%u!~-!aDZqM`dV21 zSom9U5WfWl1PpPYm9S<+QQuzkwS)$&J1vt-Upq!~((Gbgn45sw5E0qm)um!*XE(*> z+wCiw#r1aO5$wBUyFkqw^jee#+-E-1WkJl!^1b`^m0p;+z*x3_0+lgXewfA>oS)uZ zjJBVE>=2Ag+cVxRt)P`V-_)UQ*5I^HHJ~F2~luy zF-Uoca{Cs_d(_P)FwvlTV7c`4x3jY~MoSmA`R&~(C}`y|8FL2YVM5kryiHmNzb6`JYwDb1tYzOq@pIhFWdu8a61b&V`5bddA%4x-PlsF zhP&5FGOAVsUBW(**NFef+PQP5YYY}*pFBN1q<~?r<|Tj`a5lU{k6SuAI_7|{Z7TQm zsx13?QuKN9d<;->9Mp|ObATp^g}Dr3wuF|UD5bm03g8I5>o5%6NBny%R7I$o+iDaa z+?KJoIG-#1eag1A@oPz*p%T3?`za^Y`o+8iKr(EQ_0koyhDSyOnb;}kl+h$@;^z{^ zLT^!n2^@T479yUoK}4Sa>t8VxiU>L8&K;ji^_f3!rimWVknbD%>$l}o%9*A$$t>!N z*LmK!p;Et|k)LO{33mDkTjh)3PFw=s%u>_Q%JRo%J_vm~(kb!v6HXn%()Yf`wl1RR z+uh1wXuxQ5dKo+bQ45z2oIzQ()Mg5!uwv6+ zu!AIg5OXqErc@|qA-F4qWH%t|C)k;z^dKW6C(L+3j}j*WN-Nxu%78-|6m@^SBn%Gz zc3uk|3vRrf*ZItPw@{gY;*wlT5eJXbr*vSD_EEIiB|v@9FOYi*!5(B(4A7Hs1B{DG zu^S2b4xv4#YWNn|P4LX49fT;L*a{p#-6!A|dtp`EcP`3XMXZT$k@*fBA1A7$3Crzj zWFx?`-3R^QL$v&51A}nk_?ujt2sK_(fX}}A%SN)6A`vCrIrN-ISOifBkdrW31LkFs z$0{)x37Vyf1mxbbq3<8dP_6O0_>YQq;;F6dwHJMzhwJ%!3zfvykVIrXr~UY|axf8e z0wLG*hLPb0G3OfsXgZJ|o%*Z`gK-o>dnBt*ZF8Nc!KQ``2L>{NG78-Ppuo&x!pR3Q<3Kf^QEsqcwPXJT$A#zrh5$SoU%gL)u~fW3 zGoXrIIga#>bKx@t+a|TWB}bAG3c7_BW3x%ZvvDJ|CtJHmlaM}5x8DbNEEmaP+b1TV85U@m z$&}%{=LuVgHMO^k06QR*0sZWv{@o&%rY`UW#pZ#WR#?=tfXy(1|@_FMGGNYl9g<_?nhDHx9{(J{dZlr>yN8H zJ|Ay7&+~P@UeD+8JdVfXaXgObvu9;H$?!Z(a{@?FqYcM?Js$&J9{l+)v6g&9LsIGz z2wOZir7sKUg*t?}!b3$Hk7F#dO+#Av=dLQi7(se_1QVb!(VwT7{5a`hvh+9z3MO3| zXh))My{BeGTlo3SJg@Swqky{G>*AvDXg$b^@UUP2{Ne z*JuFdz_$n#@->@^SW<2I+@|-_{ynVnJ0?LT(*#B*ZWnSph1i?#Y9Z0qP3n3 zq?Cm#tP%C$^Mxpp2yb?GPt9raBIrMXeTyP{&Go(gOo1T`P+*^X4qBMz*?zQyCbsG^ zlNS$AqpCx7AAj@tQMj7y6~<@Wf&vl~Y`Hx^rE>eZ^}*qz_+J61)* zWp6zE*hjLs+1xU!lM}C0xNR7(NxMuMwkI<*n<<|i{o-Ox>#_|{Qp+Fq_b*wL^ht@3 zxK!$3>iLu_fXS1J5a#YSWuExL_QSvTFS%W6Ix8-UUUFcEYgevZc@V18l5FJ9Pfu+k z;!{pN-fU)hQlwDmC=Q+bGd5a&IlJorBl7UGXh+krTOvkmYD|z{w(PT0R(YhG-IowL zy+kFp{D}vA>4ved{#-g@oR*d2$L_gc9r2eCT%lZpLJ9b*onFqb=4HQo_TPwV5sj`A z%gGPae2Z7o)yC2?4eIiH?!c?`f11~>3TzoL!TC?#M8!Ucr=ChkQO%D}b8E8dbQ)_3_k|4ny`CS16Bboz&nUTp`B{{Hy1#CN! zrD4&1*Qn~dHUA~81-o6N0e9r)IQk>O%hTgB+d)F|_n`aAbmpW~Y{`~U5&EleC zzwF;O4u1YRv+d-Eq3%=L|Mupe`>*0`4eU7f28`by_RqwCmpIbLoV&NML?S!bb0E!1 z!;9yv)4FnQTKB*%v&Ib4h+FWhtLvG#6CS+_eifJZ)A4P| z6z*$L^*uj7|D5qUrJL5SB-{Hj+EA9t)2-ym@-`jVvx6^I zr-p;ALd;Zf9}>c4SdU#vM$NhhCieYo^Dy}A%7?ZWr|4-pez|I`9%(yHPknQymY9qWWu{h{UMc!@o*hY}fltP~DQx zRzy_Ivu@;tKwf4qk>bga2{ZvS-tOezRra+V6!Yy#WOawJ^Y@j!aF&_z?^q@~Iq&uv z@3Yu#koEM@QNuWcsN!#4`a!-Y_sq&!>X3JLtcK6(d97;Itoiu6h^zQXuQAAYkro2W z=lA+q;PZ4)hLgczMu0r|K9pUF{(XuemeWV$Obu#=wQm@%76^go~IeGb^@)geE3O(pgD4@Z7& zIe*)=lTo$5ntk{&FUD{0okR2Q#{?trzj9+=aG&&oY?EU1ZyVqEE}9a#_|wZf6Q`^$ zIzp5vco$N-Qd1i@P(0a$iHJhc)s(f0#K-p0$kzFTUk@x?zV)h=U$3IA%7a$LLTB05 z-qfw0L#XdNpi-B{4pY(b@T~52fD;vBj zEsNvRO3xJNPH_s90Y!*vNzQztmlmi)FmZg!&_=yncF+o#K54wjGvPq zm707BGu(~@K?*X~5DtJQcBL&4w64;cEly36+VDhSSWZnmQ~EmjU*$&E{JrkLeW~Tq zO&Z+zmp+Xcsp4pc#b+lP4uo}iTDSb9%dW+TA^6*=X30z(<4jOhX|`4>AKR)Z1!;qV z!#vf%V~i|({QT&D>&@h+=k?3J(yYn`WQZYPZ#KH z@@cUJz!wc)ro$v{Szr5iH8 z$QAA?beIc-(lBi48xpxmwOtBd0L6gZF;@H{G7A9wfBUG|rQE2$s(enOeZ zq&q4O6Vzpqk-u-*s}-SC87*o2gTbY_WJg?FTxOW1UC~!DPevIt_kA{o5_SuJTy8og z9Xc54|1p6J;kEm&`f>LB&rcRI)D8y(zwS08EcgA6(R|I+!*XB$Pcq0U`!nJ^8`FO% zp$ptYcpCa_?UWSXzkiqMMrWdbNfC=w{VAc7+HR_O z^b$@{=AZVhe1-2p8tl~vqf@ni`u2^!aB+$SiQR3Wp)jyw3g%!KQRV*2?Y)C{Vz1Uw zk3UCTej=p-MTJs}7WWLgl%es8aqe8+t}z|-4A<9f`8sE#gN&vUAU0ta(2<-2q6gzx;x3fZtvdM`Ur(MaBj=>LFyCb-V&d;VP~;f<>R2<0ar2 zrIsnm2k@KlsDU6SsO&wBp4EKx;XU2e!A>}k{RdYlm{|TfYJJ}PJC>~L|18OL=g!MO ztWKJm5eZ*}e3n62GT%yIX}SWExDZch_PJq;9U^~Z&ZWtzq=Rm1nbHILwpoRUN+8MF z4?jdrrs66>*_rPYX;&##{7QR0 zp?N~@)5hl$vOvN6&+Ms;u)P(buFi=dKCJAI&?V=@ zSrUCGh#3_mf}S*3x>7|M{y1@Y?D+7{h16vwYTR_^^1Bn?z`oY(^q2Hln*#5;9Vb@7 zuH5}`mc}zMNh!CH$oHircU#UBD?FBufWo~TV^$d#Yw)PGS7PCRNi9|;nop^Qx35hG0Q-Zb@; zA)uEo=~{wpfi(FP(lQ1m9E4P;#6>G z4`mN=$RCS9;FR(GlYi&%<;s= zUJZUnrY4F=Nl)x-XwLEW^9#Ck=~Cj{@@jc-JE<&7&{W#jj4hg?M+P2UoP96EtcU2j zq!TtHVr+S2il{|cdO}Zvo;-GMZIZp`{@J92{r4gI)Du6ON=vJk?+q(*lrh|~_IlD# z@wM=UMoZ++f|QG+C@rVlt8tz?uuFwqxyDpvVY}}B6*>|h!EYM$6Q!3J5Oa{?HLK6G zJQG3?t&WHI8=?Hb)pn|(nYn<%xu%&ZXacbmU@y05QdqM|k-O z(Fe?Sr@*I$*>JOh2_lY(sQ9J$|D4kBwTuV|0s;y%0w+QiJORo@#BXA?34iiwe%hn| z@WGD|!NAZBhrq}pyC|vJ{@ki+QxP`&`49ddfJ#$wRBS6SDn%P%60<+DT>PpSOW^YI z{oP>_HicFUO6V3Lt>c^6*0cjvj+}P6W|to?ZCd8EwrlnFK@Y>*0ly|l{>oXGQKq3) zj1|S-Kk$ismyDtgUdJlLRCsmcZbUD5$3`8;X0m;a)5Esb95JxH*U|ly8F!nWu}mBP zORx2+*io&!?g}5-Y9&o9&yvqi8(uc|ZuW`V9U3NdkQO323&pY*^_1B7z{`G`pFFi( zL37RfsA1r}M^zV3%5Q$g^7^>96F#>Z@bPQhz>L;!_8uIz@O`xA<2l}6*BkpK@A6eE z_E`MX$~;ijbgONC#q;}v=eJBKx*y%lq~f0&=h9*0|6168!WShl&E}wOZHG#{HsZOJ z{uq73=VwhjXL>JwSnSdNbEwg$f^ZXyS^rB_&qJ|SqiRkU)Rl7yHevn7OsJ zwd;)qTFpXh+Wj>7Iregalg5JVMj1C=e82wC_<18ko4sm|3sz0Je&dI#>#>l`l@=QG zYF>C=MluI(NuxBLx^~^VARs^%6Yx)9dTgke~ZUi7EfCI5n&ok}nl|L0so(;H9kq9d+k1|c5U1Kd0x?r*8S)lT#m@?*&Yl4QP7{P zyJY{^5VXa+4hI5lkx)>wTxaDYX@R12Wvj~_qQEX1ikH7FwUukt;?*~G%M;R<69 zhiCntoDLqHHQzkyM}EyG)P*a4!yG^wx7!ui|KHT{fnpg+xz=;ge8`XJ&zsX9oGnO- zR1NWt@yWUIdcvxihLO%yn?@FNxw%qfM#zC1z(zBVexCg;_OtrOvapb91a$hp6!Iax z=e=sJ^z-Td>8~9vJf?<@zB?zfMm?YR-=kz@x))K>JMPcvysTTdsC0v-?|x2pczb6e zfd84iGWJ$s-?-|;uRs4?qcH}wUkwZI#2b4jo0X$lSt^-J=LWCt@>BQon5gm9>Zuqk z{5~-0t-b4*eb;APz3KU4!99SWRz=kBwO*|DFWqwCao-F@-L?FAS2{_r(D~t%i z*-c=};P=Vlr@cMBr(%VOpT7}1ll74T07fl3QUU;8deh_P?RF=80>=J4M4L}H2;<=V zvN!cpC*X^+>>MsZJnT_!qNEmse0kEG>U*OX)0*|kI4hzT>@Np3fDmTX2Hu4hW%_tU z>JD!40dbyy!;1d8kV^weC4HIxp!HegTHkQ*6r3=;SIbrvBA&DfDSJ2(bp-1w*n{9m z1`wz*yK$jl^-QPd;zSpC5%xe5MxG-PX0Rl^QeNc=7 zytN3{F1u5Tr6Uly@>tvJQIBP0ihSOaezw%y0p;bteQki|0MSVt7Ac9Dc=RA0tyxt5 zHhu%S<2~RZ^lm|zInd0Ye}zn55qSd=G4E}9*p&0Dt>*WZ|I|L zK3)h*eD24GWD$lcNA-Bo@NZc5_e)*oA?2$f#)jM&aZ8k;c_1%*sYxWiukM^YDy_;$ zRGrV17sdYDr8pJL0_}fZ@$d|&|EL6S}d9d@D@~<3!K{hRk zo}S^%nE@+Xl)o<5UT_g8yEb|v3o7lKE4kdk#$ByqXBXbRo3c@(>F@IsIyHne%MP^Q z)BN8^CBGbZ5_N#sZjoj53J$%#=j@5!_bA>;tP#cFPddgy2r`5ngmpCMsW-{*h&4Bh zNPCgbjkf=jcp5xWYiUlXcns80=|AdWh?R|3$<^#GiKm?bp5OgBm8t$4I;oi08#_Uj?l$axpX%N&5JwVN4)^CmUqe#E>zZy`iMblCt`(@iuS!EGnc zbchVCq8RM?Kl|B|lwvY7GiQ!-u^9Yh(`8)aN^X|l7CaPeu~U;k5k|L)Q&z_5CW_o_ z+znb3IrZ7J4$U5}1!$}%SVMM|HB=Qz!oa_L*T ze6q`;by{AuVx6BG@BgFwxfz{;R}8bVD|yZFY7w`ElVvwYq?tVrK^ZMMOjd-V(T?=O0$vDETLfZs5SlD9H|#;hS&pBQCoeLR%NuB@=n-`E^z|vGGwNk#W}ME<%(iQQNcKQn7|ke_?7vV(+vgNd zIi6439*B+xJ*~zCVl#jQz)bsvONB~vix20Xl9kH9ntIj6YqUA zXk~+yFT#T-mOS`Q4|@;9m%_NI<0>m zC@hKe9*fR#e97~qB|;RMqT8{5l%RYQ|CifhxQo7zR;2+SuC`+utA`k_P&Nx8v=fJc(Hm< z7vuOq7tYkml`D_(qykoc=EMLD$!RI4+FO3>(!_2q#VPQD4+o*>aR`L^o@C+~(hvuF z{Eb6}2wp?5-ZFv8mP6e?g-0$8zkK)g{@DflHouOnuINs4KRt*>TfWd%9)oF^h)zFNA>N?KjKipY8DLUedkp zW1sbBpHTz`$eAUGm6*_0j&d0A8*MOcV)xlRJ~QR1s15arvQo$1iprgm&~@vkOZKcnB3 zPXvxAllX+GWWI@^JY8t@9#E# z{P^I6j(R55q~&LA+}Ua3sm~U@-E_*z&_-AA!3)n@5&CA|ila!~y5Jj7m6$(?yl@Te zEx6^RZ$yax^(IY^w&*;L_>u7I|8HWTGH*f7kmq}emYUh@f$_K zk#-C36@AbsPK`4SPx~-qik_isdev&x?iIFID2}3R+^tbuaD{sWFQd`3=P499R&mtC zWt%`@?MxEtr!(gSyspOL91ZotdcpSnJN7qB+^5&~f zpM6wSdg$pf3>ATWbC2SmmaZJFwn&gY4@)EQpE#p9+Ylvguk4%-t`ja7d^2i(Yo|tG zy@|fNx9Xca9Cw^I^<**>T+fqa&oXQ^;|H%n zXo+DJN-AR`BQ-q{MAhmR>yK?Hw3@!iG0Vk4_aKUmnoTI2rZQ~E>CJ9^dWCx*rb6Uz zLxQR(CMO{34kRg~q0{p)lD919^WZ5aclz< zQ)Lp`cj#d{0eO0+TzEt8fsk4HkpKar;D_684Dy8X)oDsGko-hq?U8&Hnv;HH-o#^n3fFL!5DA+KU!M19vA)f`KN1ty`!DEE2n?S=kGLjP`DIp0sDKM z+da6b)GQ#|d@l%AfPm&ayx`A8tG^fYmR>lE+pvFP72)`Uu!5+9*?!icpAHA+J7xmc zm%0@!`BR*=c2I+TfzH5Me*~a>4gbaTn`oIrj2F2(mA3 zp6SUR&gq9{jbHgQ!^bndP2}n5d+kfwxE5YkExiaG*Hfo@mY!{A5?i}iSfCWCpuqn! zoiR|6?+$HsXtQ*`W9-Q3-Ny3W(bGMM9eU<~Z3RVUJ4K?Bdui)LZXk8n|Au)5zPq^C z&X(L#p)kfy(iC|ZagP-@-N&3!uhm`3uSK;mv{$`WMiXKNs=t)_dzH`pjpe}9sTFrj zJ;~$N*Lt46ePrvCw&P4H3-kmn)XcPVMQcejT@L-jrN8Vxp>N}+;kDZBGx1$L>4>t; zOiuxgp9Cf6O`#aCdE(XE*`rFt3sWHG%#av@xnDg@Dx7rmQr zR-d1BG|B(;WZ%`hl5$oU-3l7*5jlU>>f=r~5>993z0XS%!**0Ak)kr0J6~S>XNlN1 zt2)uQX1l5xlTEjEv6(u~BXZrpbE5+Go+wq6sJR|X2{M-#Y}xp`izR-|uKqu5L%KJ9 zW?^hM@BZrW`@bQlE$$YB`$aI~LgvX>E$;s@b(rn_=tz&grBz*>>)ZBQ+IU>MemltA zF8`Nv!EeM#rz?l{C;TJ{$)~LrZ*JLd=kHHyG3Vt^y9KfdpPbg{9Xl)1p+mhv-i}|A zerD}AG8`Lv?c(_M7fdrIncnCzKgLP-)2({D&xbp_m8(^ud$E!;TLTXltMERw>6M!6 z(8i0^=dTHZaVG9Nb1Y?0NK#*(-e$L`Mg6|Jf9T;A6xzl1p2fcJCP|Zq?d{$}57090 zl;his{1S3GqT6r!CcYKgOEo=2woBiXwAQ$cE=PPs$5g5Zsj^~_sb#orJ0{YrAGm8? zf^eDysy5xK{NnY4(P=^5iKA#iD2rTURU~4f~ibA{{b@gtugZ6lqWE=(!eGYV?|>k zEtF`e3o<(Vc?l*uG<|46MR}{ymo)L#>ILs7(|R8kRs8EaO^wIk`+`vn{I~HBmF4uL zN>fB%FG^`OX-S=I0~JYQfzXMROL*YeJSpoMSL_-3KU=Z<2TBpd92&KMFqKSr0zx)N zj4!1<4+)K)ArGr%bXmt%oeLoLIAKr`!ld}nARQdKTXX8r(C^uCs|U5!%=MSI<7=OS z6%Zd{&n4In0D)B^G!uCawM0RQ z?_J5Aj&S>s8W|Ar>9;LaO^0N=dnMjjDFHt#Mu_yV5mE)|wL%=I-np}UK^cko3dmXp z!qVW?331F=1_tm;A?NOuA&0$Y&YLtku-0T>zPtgmikZQJQG|S9F)2jO!ST9-Gi`75 zANoL1Wc(YjwuxIkcRXF17`(#@q_Y2FkG$miqxz3%kiy}-xfkrB=snogl4?#hbo(%FG1mUUYl-8xu$bTnd)hb3( zwM3(v^tlM<1;vNBXO`7h@kx9Nq;FoVXx+P=Po=he_R9QgxP7UeiZQ!D+O<=uD@uQ6chYu? zn00%zNd`&l&LLukb<||QmoMpItp-04J`?yoi<(H(Au>!am3IxKk6^SXGd-ki{rRMI z@iw&q@8T;heckxayJj0YzON^DGCo*5M1r=GkeXkwYdQk{;E!LQe@?0iv{(Qf6wc3x+3grl(!cTdAg@VH%3`J;-G;rF4yq!QaK>VMM z+ClivGvz2|d|lT59z;dx*b8`qWk1!d%TkPzzblo)qf5q2UrkSXkz`oi)D-1sf2r_i zf0N>s{5%-LExlWWF`<}i+GKVbRIXo;KEBuSGRPNg=VG_btV_lj%HH=>0ysXBvH+HYM1$TSiPLr{h~Qg2gJ(eukj> zh_HG;R1NG)Pl$}GW}yjri^pcqq036&aqLLPT=U%qXXe=-QyrQd;uqH|l7DcC+mywKL&reM%jcHZ7*zpmfG@b0CIfFblbc4nKDIQ*lQ5ZxKnO|OEcC3jh8KVP1R zdh@|N9nq?w^)g2oQ>uXP6(3&IN92J+ndgv7kBSCvs!}YXzDk1pleQo=nNvaQ&uO(e zM62gN?UaWRv827vy?Ar8?gtH5Qm^h?`N@p&dcpX!d_%br>5y+Cx(uF}D_KfWLalB+ z&5KK)x;)|@5=e;KJJgEcLX!L-FvJ@LYL?0OK58%n(sKkn%LAOd!zvjyvH6cH)|y5^ z9G#&e|3cev4vCH!dx>g~M=lH+p8R6{O%QqYuFyO-T zE7EA#Vf?|=`?0LBO$BRKS*Skxm3h4?S z2cBuOjpz#lral!L2PJCv2-bUw# z-R2tUmU&;bpfvjH3_O%JyKtr5&!0c9uMW#=`mTiq>Yelh!>T!-k=0tQ#H|X&Ra-s! zJ$)Z6e^RG#Q|=<^)ned+nx2@TAhS3;`hY!zwMcz&IUAARoAaC%B5Muvj8nKJiRYV` zx9;%uH9K(H-#`9`#p!25KEL(ey1(DDup!^eqv4(gBFB-YESk^kP_Rf7nrMCjsA#1b z$rnc@>bWGZ_fVGbj6z4T=s;(BaAYgT!xO&sRQ6z!-7XsXn*0_4g0Yyh3M4OO`ef1CFZ~&rLMZxA)p>Q_j7XG9-}q6)Dzh=?60tM$tZW88M;XJM8&{)f2GPQsaN923=N)fXuMK^{$eJNhk2byJC;3 z7j!&q5Bvg$id}}Ba$#e*1Zj|j3I81n4iIxwE5bhykG&hbB<2uuQW14YA}q7HWUz`5EhYRXEyZNy zGQci9ZuM&C{#{CLt^7~?fikuf$~{$P&C3``0c)3NJViVsZhi)fHF(ru;iei$$NqM`-V>p}-FJ)$O%$4755o6B1DCdM#)*z$#G3e8q ze&-Vli(7xua2|iBq1q@_wa7EK4sNR3xJ$v&VRI%pSBYFeYlC4T++XxWV|FEQWv?c!qU=`c`|Jnz!~jcU6FluVLOobI%p`+r4AcAw$8eB>u3X1!c=@6vdyGA zzbXzU-bE0g7&Y9YLx&CqrlyR^iM!>H^VmD_p2m5@z7N znEUkUS_Yt^%AyyW=(+VhwpxT+4QQsUtlGVM1J&)-6}hip*QSVW*1UO!QFps@6cZH` z=sqnkwr`83rR*n2Wp0HZO%TSa;bT*e(o99MZQs5vILmL)Rc+Ry#S@aKOOQ~_f3z=q z4S#UDGCOlgg}@aI1lh4gSyD>3X|Xu>Skj zs~+t8@BjZdWq0KNzZ+$@f_OI3SwR3DNl4YisP1fZbf$gYe|DB_$J2@c^A}KP%?7RNFE41>vSs$2J3}ljTOv^U zf@R1;>J6uso1OVhd!{QkNJRvHz^F{QgZ6;HjVFJ9PCBFUt# zyn_w@esxCQ+KP#xi_oLdZJm&o)`0VT-TCta74a!4wW&JSZ{Oa4uu~PM9ObV6{Qb%a z1sCOWk$6>7tiUKKHGM~`gLhL?>odKhqN1wHe}d>!qKK_GQzjKI%t_7p3Q%mA{P(*x zNGne3?A+kstM{)mG^Wp*RY^hr#nZ0seD`^J9yIbpZaz_`Tl@A^Y;A47yganksPliW zM6aNtEHk4miEHEQng*|TR~U7|f!H%M>v=t>F> z>2!NN?!MKNNG{a2krhW-V6R6G8(-d!zuGiI`PHl9taee8I`533zpkxI)*6c$dq{^K zS7~?9P|y5s;pY*$x{+1F#}YGlzl$c0MAir&t^2m>mI`Ac*YU94y@mOVN`8tEqVwqg zgqiT}3V!AE>0VrN1qHF(D`w2xxt$g94<4Mqn?zb({`2v)+%_dJL&abUiJY&Vd7jVy zvt(WtYw~XTyh{ZI^lRg)hkUBns&x@kSfP+9DxXgL=O$+?m8X3RcVC@8ecJZx*MbBg zk=cOtTeaGPPJJ41MJILjp_m0^SNu=ZA656dUsuHhJ!NI(Ns+cnI8(Y}N1p=aYP$;R zS#$NTAIj=2|Ng8A1pytJxL?+`tbR}?tz1Y>t_84m^~Q~LQBi%sV%EV!53#U_32+}- z{)M@l8r+`0|3cSjqG# zDg#$ELqUHEPJB@CboUgp4!8P4WkXCGDFAv$UyRp;o(}ytg6?mw>~sR*Vw3LULfPPC8$>NdB)nDGVrIE~?@}7(X@I^td+!Z=z>Nymh`Mf)wzdwBR z=pn|J1-K8@{Wy+|;u;>_t<}I48>mNk$s{wg13(5ZAI1$rDLp;2f&1mEJ*xes?ut6B!G@V8=@yAe>JLgGqd6CRSzzU zXHH3kQ<3e4%hQhKCB;te+PU*3j6m!elyEE$WzZGs%=e5BnF@~VZEmHi8g!2f(*&Ca zH%?B8S(4Wfb*~34V#%>{TNAbCKZyP3B$D3xR2^%mt!(JhB`Xa)aiUR!3HLvOgtmkI znCUclu!8=rtt^bk)Z1QuxM;Ocf!ujh^cpTXJ8E%9%DHvCfvH*4Msi}3u)v+y*gale zjqaYF=Z=Y@HT8{tsLzfaYw_J{*P}-xh_5f$lcyrCQVbq7sygh&c1&Tthj-+(>cnnO zOiY}8KU$lq!1ds{zThGg`G!e=_Ef_>x?Bb-jKs}<>a=P9^1$~DTHQ)@d*G(g<8^f% zu5%Pjoi%GcJ|!0~U#_xr>C&q=Z;qTjyAv&ut$9Y%=zn_m;lsa_5a~{|jlRqbZNU<) z4kt!1znGQPl+%b9reNUr7Bs9`)&(0jyqw1{y0x4x2Z7ik&YW3);6PhI-Q0o#4?x0c z2p;Sm9EKySTI*nSenpKQJ$fw3&#c4I5pnwTItm1^_6QV(hXVrS<8>B2t=_qFXWNLS zx*tzZE4g3HHE+Gy!=pm%{60rQFrSf|gz5bkz9?T(HHG2M?bp?7eWPiCuI}Lz;Bz$x z{rYk!waZL6Aj5og9FO7I3?@#z$RkO^oqMVketD@xPZDlpJ>QKR0^cxku%B2w`@Zq( zx8$V@EYCDz!iJ3-LC)$Ob{d4WVWrx2>dd&6e>YM?x#w&btW0xJZ0rK0%#m2$xDPl; z%DYNn0536tk;uxIe++zRSRq&L&FbP`TEia0R@Id4iVM02C(E7z&Q0i|92=SLy5Ut5 zIc#m4>%B8o0!=%+v_M%QYs2uulp5OF+Am+e?6h|#E0WUMO_mKKHFTUmb!H0DHe$-G zS+me4R*TO+?#RrS%}|+YYHHP=EWnbw0jI9lwUg7M;q)%k54DgHSyXcImo8NXLP!Kv zuF$1Rmm|r3yLY?uOxyM9)r9f%7uaELO@{(M>MVWT46uKjmlv(+uP!`z&6_p zH(WQov^0L;lS^lJHN^_RO~VThJ(tk2roR}JMxrE3UqT*Tc7)9~9XnQ|J$L<)Bkf6W zfR1zX^W6cd+YT7e5+U#G%#h}2ixtk!-wkh%9X z6V?^M_+ol`W1xxkk&)UiS;PAN`22A7yU(9T>g(4$6&WdOoO_guAGu+k`B>eLUDXTj zrk$EI5pVEXFN!_&WA@ci?pYm!B3(*HP>9sSdX0`GEkd+H9csfYMsO*^HOgu-kIf!0E)j@+=o0ymoJQKuG{6o=J?yuT{ z*N%$R)YN!F{8rXf(qx~HMMX*hv+V5LnTGa=JaYq6OQKas9h>!A+L?Ot_?+7vN1s?wb6z^!la+&7==s4?0bCu!!f-(b*c_jdy(YP z|Hu(HR@ziJ;6)EdsvAz8yqRp|F{GAU9WIS)3wG_@TZb=7NjHA^zLO_UUX%+2)4E#u z`o>WMWih`}=)F5bA&o`*yd#HwSwL71a{NqXlKIe9@x}MEXU?1{J7ZDI&hy0Jx~=+e z0Q(+&Uq}B|B*Ee6(T>RHJ4Nn`iyOq2L`>SNlfy8~2{yq?sP^wK6Lc@Z9J)2_L{frh z>D+wh+2h7CJ>_U`on`>wEjKr53O8#iy(2uhbof1N)U(l7!tneo+$BP`=8$nifnvFO@9SoybRYjwCg2*zjM@JBgC)c<-9O81t*(+G7lgJsn@?L^Wb6UhhGB z+N%aBu4TsutukmAH6Q3mq{y?0#~0QPZk9n}nf+UzsJhha=kO8G{o0nC0}>Ws+P zCB@(JyUoQ=`&IP97dKk67OKi#e<_sG;eYzUNcDrRNU0Cpe z$Bs#o2Od0toX7r2Vx8CbW17k*ah@dq2pXWOS`ipmchMp(z%&mmBK5CNVYCmOMQMleD zES8RV|3imdu~>}1f4?QjVM49%_-t|3T^GKvzSDL&t)z!4VwSrnbCWi~(huw0#0J9@w(xq*ZUjXP zzFj7DP9*qk;>=7S#6tuw|Mu}8RQ5M!bx=LHh%U24BB(u~dA6dx{sW1a?H{? zQdD9MXmq0=dkQA)5;_K#fh)gJl3BePb_l@wn75Uu(nFACWDa> z?uuigiyW)huGNs(E=dC3XqPTuzL1sWbMan-V=jPD2m7{Z*RGPn+SYax?ER>fZGVv* zKU%j~9@~^3|B$s;XN<@N?&XZInxb$+s5aI-^Vb)9Rm6Zfo2yD#$$YLAZ49& zenphgwlm~iys2mV33{k zRm*il;HK#_XI4~%goH#acsNk@we2ss(f9ZrTJvn!GUt-=*eNQg>$iQMdYioZ>Cy|I zFr^;K+p*xU(1-E#^cbWr1L#a|6Z=xml-$RU|3z@#mI4OVLCu&}*RNf>z$Mv#|1>Wa zITIkqv6K^@;jc+3H%MF7u3reH4orz#72CBzgJW&Fb*snHRlJLSQA2OEwDC0~sOukX z)TGHePKaR>CpMx#`yW#?wjUFwx1b=pmV&cLOSr)fg$>U^f-!8jow${dW0}lXOd2+< zlEgigmMz_6Ak$Do!-hnc^>RCu4Rls~(d4wNf|p^4WlR%+H41UYonT;~09Q|;kJ#3V zGyXj?`UAAf9-B86?!M~FqiTZbckVnUNGU??KGd>_`qg_UW{jpx zn)W6qifqvQScP0HD@U1QO?A_~@hu6Xd&~4c(Nyl44j$VVtRCm>?f#gcLxw&m3!(3fma}*2678d5lSD-zFbQ&c<$Vgge4DGR7fLx0}5)R zt*wML-69KfSn{bTW8_fX)^?cYyt_T4o_S5K#Wkm4vf+E6?s7|2RYNQ0k*BW@S6I)P zQ-dAsyEsY}@?lrcISp}N&&|uLK{vx9?<4G$MXqu2`HP*2lrREP)u>YaJ%1K?FkhTI z5C0q25c%{a*p_YBV?>N0GAEF)ST9(xm6XMwV?Zle06^9tYP?b7#x-%ey$ulYMfOKs zUG-7=`kv9fjs*u_OiP;(@Mx{-wByZ?YhPd!K6>-U>PS3 zc^KDCNwG7xN2Bc@nkhP{231m==+wsrD9m-pW9b_Cu z%`1S?;ml&?>>D>gi1@p#qw-a)B#-Bs7a0cTEj$yPf@&m7}q*J|Ai+EpFllwbdVMt zr!p~g9t304Tylc+bfYnTJ!`OSx-K)*kLfyK<+sMvwqE^q9#|Wc)&AQ0iJQVkGaDwJ zJ1%?~VM;?KEBorz>T)%FWl6K8{^K+pOpcX?KGWNIE7VYyXh)^wZ_{uic~V7sqW1J# zRu?K2Vy$^elUII|rcG_GdA8Sa{Gf)SD}vCvX!Y7d;ol?c@uS=K+=bsV!g zEMB}ACgs`~pUJZtTl%nH)(EgbKxM0UcIfwmM?-Os^piqk=;o##`&p#bl&XQf%XqLCs_&qQWZ*_V83+fI+nL) z*wqlTll5D)&^T4S>#?WV?LJ(!sJ8sRsu|Q|H>=oqK$g*UBp>~*v;18f4)!f5U6wmK z9*Bsj53l!hnlijp?Kdt zz6{*OS{n`uKDtY#1`W;ygoSn0zOev}?mK~v(gP<5M24q;3#nm}Wp8!$njA61ZEe*% zjN23VVg0BD>Rk+?34UY%{z$JVKQsk6fz<+%kpi|DTj<7N!GULL)fk>73@Vs%n}Gvc z(V=w2%7TF}8(AWP7gVoVbEvg-8@?Pv)ilvojoG7fJSD#!S2fedrk&Dq%xF6@#9#~S z*#?W`nw9oTCJX>DlKfhsJ1}vrzzU1X z-}%{z?5+~l9sOC#3 ztBsVyC0cjF64#>#4pd-r8SI4Vt7hZ4sZ|uLfU2zXZNB>vyU%gd6POaqnakbce*8fJ<|kVl?3!=06WGitfSSxzE-1lSYGr>iQ?TTZ-yFs17uX~$m|30JOE6C|92S?ToYTNDP&Q*zsh8_0l*{Br~XETHa2 z?9p5wprU7ogM-~^kWmoA4Zp0J0Jtgf11Di?#s(?X{k8KsmA}MG5RLZ~XV)rkYX3yV z#h0NN)Fr-Eh;4OyAc2r7fJ&U7;g zxAK5o7mSt@*RYWL-<=8PlS$bbq))6aPvy$ig#?xAc# zAf1FsE!#`{0y0pvS@t^N1D!_|+QjvHLF@vjJ#*`W`Y{ANM2bPyGiJDP8dw7&>4zQ- z4fPOI06%9C2_Z95zF2_vFN}_kPWbYgQq4PN^K_>fRz7OoG#;odYo$Pawvdra`pLm+ z)IFW2oL9lUf4JOg8jNv=_(w8{O6R{mZvj|Q4O#11o|5;D?EoL|U*GAWDDJH?M6X`! zfl}t9g;|4js?nTxO~1tD8**ShJ>77im;ycf$=NrJ%^)}lYfqriBbhyV_SB`M*Tj23 z5k4pkvhRb}WFHco>HqHJlXmr*r`#2+426d^56)QmM-~0mGZ`qENQ@6sn=*0YM6zK4 zD&M_-e}SE;w6wXU=el)O$+Ws-qXhLw8qgd^2V=t#2cy@Or1$ZOiB-9UV( zR4XH+#;bl7nJQewH>^eM2hZ435Al<*zYVh@U>r^OA-6tkKgtnXa>oI$NF}C`d>dN& z51>kuY=GkgJDE;nx{+4`HyaokRUSL`x>k&FJHD3KF)_V%X+Z*zlUp7c_oSGmUFh#R z5D>7A*{XrJKJ1wM1R&Y1X%|KPrcKwgagu(H3OYe>vW8J?rfIF%H*n=9uAnXRVr+eR z;<+4~2_wxKQcj;;`MsB-x@CBS2HRJVFiXPEQFS=rD3C38Zs>w~3+j3-e*0hm)(zb# zk`sBB-B)~hr`|0Xcn&;HYELk`T2F4{_he0mkxk#3_w$EGMmLp8_U_%g)HJOmch@d{ zaLzgV)SO0;0&XWNgkM>?qw-NlvNiSlvW4-e_C5Z9E%ONEkOH#O@&QIeu5DdI!lubW zkSY)ame)@_X4%`c$+aH0sUr{X8_lig&Q|r!x0^xwnLva{V{i)?oef)E%cf2@i_^7f zC)qhC0{*7c$2}7s9N)h2f%fZ)e&F?d&eojplT<35mU!GkH+aCpW+Qh{732jOSTh~_ zHjsc57&4dR0<$Lhy0>{INAy(mabEx?-#=VkeBsuuhG^3zxVIDfl9b%ATTf(J8z4mzz*8ukG<)`3OP)>gzSgG%98dq^p(MjHKxr$H^D-Q@ zn>Bdij~^@8lS2p+lFyQWYIKR~O=c=rB|r;kcz31`$v~7Z*6KPsv>)WeBQfObROX?; zJ=e^#C6)dN;GUGk3DIyz|^9ZE$e6|uu#HdlD216auUa*!j#Yo?36Fz}W zB=P1zIgXN&d-()(pt6GMb{gk;cBEa?Q>RW5&Z^X@Q%9|1$6Bqwn2hw_1AVa*?u9(v zniPS%>`Gw8O%_p4*Z!ZG!jM2qtnE3emJlb!=NGID5mNuhr=p_EgldHX ztU*&-`!cAuo3#U9CHK`U^BK?D1&uDr(Pz-d*`QUct~~}|^6}+N>>Di%#^15Jq}T#H z@_hHN)#%Z+VD>KKAfX;v)J-8w+*RssHWoQfEkOYI4V+3Z5zV?|k*|M!d@Hyefcw@0 z9g~dJ-9AbX4S$ZNT|H*i?K=6a3+0Y}RB6_dZrT+9=v!=qg&u_Of zD&m0o`EMOnop$4+S%p!}tpy}z!u%7?ZH3k6OSx~}Y+=bBV&tP%zMfMRW+MAITY7Dq z9IT#!(u)*u)bYjG&`uOJGisM^W=FA{6z zflTh5k)EDPZ^nx61!rDzG<*Ai+t15{5MUuZW_xVdvIX2QMDwN`4$hww@#(C_>2j<{ z|C5R7qP@MDk7;b{h|J+Q!)QXxBWtoH1-NI^pjdPx)Fj-y7o%S(K0d$RVL;7FRDzSw zIVq8XUIvV|o;!ClCLfQekn!JHm+tT&DXAvVc*uuGEjGMrsJ-yoFD*!}M67+;d56+P z(Puqt)=_&v-GNN3lC#9Z;@%<$4>b0!po^&D4g%XB>1sg_cx-1ljE0FlN$rP#Znl$4OkC0IO%r>|p$@Ji`qBAPvVxDupXvZiEG?eRNfUJchb%NsFrWIK2f zaz)SRfSr;1rk{_^Uut5~LckH)HAnOnl+a9kaY8N|ra1%*Sg@ zAreA+?2KNDAxIl){+9zb4rmWPHBPNzTh4y6(xKgyHj&GBJEae#kbGQ5WRO6GUzDDRboi&k=m^h+SoK{{^%sy|v&_G30Rv1loP^^*?8V~HH zHNg@!(BC?Y1`}EyutJFRw{}VOVA6&E@mv3I{>=fAMyVe3^V+#{9edr-r@NNcP_iF= z%aH(d0RaJ=JDq#n1bdQGMXlihz8w)(W6GnA8(xNbQTSt>b}7ZsFd_&5oxv+DR0W1NhknRR%ghv23fX#c z%0wOz=3t{NaNJaB4|9HI0&8a9pw(S_SP}yXwMPcJcQQYtCfcmKch&cFb9~EaTQ?{< zjVbmD&^-D}sz5~hhlYmUiqzU!Nu}-J+jiKxQtJM8#MEJe(97;3K#e{6w=+NU8??(EOweVN4NP~B%;meNJ{t$QXEecPj~FM$i@Os zb$$2kSu|B?`YFNRICGF8p$!?iaA9{vE{RKOdgt#8UxL_YpPW?>2(x3zJuW~nMkzX{ z%&p%bVgAPK-&lV>XH8fnYTRNb{}S3Ic7n^@1#)k-DH zf&U_?Sd++CuTkRyk<)Dk+vhzt47H{lZ=tN5P)e0BhTt-1g(khcE3k(4Ht&X_V=pue zh+DTGds$a70+$jKr6SV&}R9p7-w_4Flz?0|tG8QrT?= zb$)jSueW_?H*;oH*m_y0G`LFQOb6cvDQ6d57ik(*sas@Zx zAM_*X(e2OO`PQ4eN!FI|FsOfzjk>Dx3xsNg)u63r=H{vpw^9%t9PZ9t3>s2f(WpTK zXmb}(5!2awiLe4;ldOtrP|ynz`zTzfAKF2SuzMg4`k+0jr4Xt^R5C@cjOS)YQ~>8O z4Qeq5Kq{uWvhri#7Q@@A$?bydx&vHH0V1>@t=xD{kDlG3mtg8X)x*6ohdpv)?_v)=17p{8U0>^&2#p5c&*yP1=*jj@Hw=z?r+S z;I1~#NHySRqjI_)corJWp<7q5vr`KmwgR&nzGO*H&dPxFXh6<2`}emNbcK>Za-;-< zYUBdy1I{#<@N8l?=LS@YC+K6E$_%)KYbT+HV*&#_L z8#%*#CJS%?0uhobBUmBKIOUFA)&E50^lQ&(23AXz~i8e1Q0 zVn!#9A$>uM<83&Co8KFaf{hf|>`KmOBxV(Ii<%X2C#on?_6@M8{;OhmOzs|%GX+%Q1=K3=+M7^e?@GoLerri5<~>AmH#p???O(_WZOP+aS)X? zEWP>Pj7fbYIRu1m)~uN|?O56-b-4;>9EsQCey{ivH!HyN$%Et;&;6e+ZoJ=j!-fa* zSbRf+ANsABNp&K`DF(PD)oEhV5Zz7h!?tq;#?%1{veePLI@G91l{|XpR}_k*>%$%8JJyJib#3KLv2Cq=Q}=5N~^+R1UIR?nV| z!<-?pc9>ad!9;Bw_PlaTeZ_Y0o%s7iS>oM<3&FaTAJ$oYCGz!qJJH}qCp$s>}Q7G`TW zp-CYGp2&wN_T1hCiExQK<>mY{Eh}q1E#9~CM2s=}tHlSKqrf*Jvk(NCtj;Yz7p+eO zB#Y>X26PZK@h3&|;jn%pJw$6ttgfSIrZs&-!spow!Yn-DN{ntcde+`?AP&@SY~f$9 z{vzpC;f$Xk%pm<|r^l$dR&S?AWnFUvZJ= zlJYCYlOTv0eoY@CjbTUgd(X{QAjklTZVS-O=`C_gdgR|z9|eCrz?+>r2k_Qq8O>++ zCCiQ~~2vxn-%S8QAnq zkrOCfczYWZk)sHqY{dxLob<1GA*1fis*1|;y=7jkUQ>DZu3gq3c&U*70aOrDC7*Tv z+!Ywjdj9;nisiq4^m+5(oV%Q)P;Tmxy`bsO_<)~-iMUCYz_lZmG1=#wMM*dDQU8sj9R5D%f=`XR z=6{@EC2KL(aq$ZDi?h=Mx2K=L)7x$9)_MdnO+ZzV*(!vB6-ZQWCbj!oXkgRj7seqx z@!=Ec8&d7@lRQcgC@jxK3kRe+G|xdJW`$=l=|e9;tYRw5vBW>I955yp214?vp33)Z*4WBF$dCMgU@Xo1f9jd-+nTcyu2SE{TAh z{w%VRmIMZVK0}RpFj&{&=r<@6Bqh-+G?=x$p1v2$IbBGj`m#Rz?e$O0>)#hL=N0H1E*C zpC)=H(3jQB0rrz8Pv`%km5|R*zZm1zb@!%C#j0R1AB%GvQ6L{6lI4;W?Li91P8~aT zd`|*Dz1^7lKSCs4K;Rw#YyC+>HEJmyPV33pugM~*t$^a{{njcOludDA&nOLz7OQ-( zK~9QWF|sjDszSkOYgJk;^UF&j-%7k&gYzG z)-z{vJE)bWBgzL5POZyXdAYgejT-AzOi*OW2_&0RVP|^*&#?-kLPUIQ2V3KFzwbHi z`QxT3CG$$4aS(2Ft+QSp7`(LBg_8d7H2Y3NC7NMT(UI$Ux6q!3RdI;Y$OhYoa5PRu zzozJV=1fF!#U0ZwDhdHI0Lrdq4Ttbk$X2({G%^0Uaq0;4$H9Wd?EZb>7Eq_@+;do? z7Dg+;LEVkck$Nh>-sk5zj~*Hk$;Tm|IJMHBAeNGjk9l?p_ai7ow0JN!*5O$S2c2L1 zEw1of{{|{~l(sfVO>{!r=(TtzEdBo3`-obiK~zgAOB7j>Y6D|2de z1V}$ZjhIFMoGj^8tY}2Mv!e=e zxNAYeOquW2hePEhd8`=W#1ZQBmsQN(B_Y5H5 zG|3~x=0s)udVDPaC)?RgF9g5ZzI`J@EFF0{NZ}!fJgs1)l?4+56INK9UKO*&U6OZ% zI+r1VIl?b4T);^bJ&LR_|DOBt|G)D8bTa+lbeaC=ITvrmcl;DO)?~?&zpU}uT0opW0@-+ z`j2<+=mcq(kPd|BoN({2cw1EDN6cm84!<|1BZ`0XT+OmcY{{$j$xg!s+l$!p?*}ao z5{lt=_{fH@?CfF_i|NkvMdc2{+-O}l_Y2617{Fs!oQ|5e@Z;%sJCnU894)ErE-w;O z|3ANb>GI{`RGOdM)>u88D#}$-(HL+wmmH|sB6lA9O6$(gsyS2HX+T?RfzZhAXv~lk zT}6C=ka3FXSjzr7b*hd+8YwQm6$pEy2nCQM%Hhs=2rGIAR*z}0SJof^TgGPL&Os2+ zS8e7%(^@cPvfgl0Q`0Y1Ro!67fqY-zkMAP2i@9E*5cN0gVxy9#ZTl&h*p;rUZ6ckp zW{blJy(;XL=oN4}Krgk#FQAn>4=!vjSpiAeeflatn|W}X_;cP7z6})xs9t)0{?yrB zKqcw`1-Bi#{OBZAln;(6D-&`)25j9b#BkV~;Hha_-j*?UyucoEcG;_@AQW|ZmjVm` zODcCwL*KSf@CH)H^q`%^g_uu4Mz4&-2}@uaN)oms^o}@XZF$%B+w}utk5O zj;E$^^LkMG_Z%{|9>YV%Jp6ue!S@i|saxP0XArZWp7d=zkNhId^BrRIg#sT@dEiOS z0*a|6N*sPZ-VChMfKj!vsA)S7v)^S$rw@?T4QvqN|Idb9zkK~F+6VBEzo9kCU*G8- zy_{~*9}6cMP}YUt3RFyH^#QfjqKL9cX{?VV4T97ZnE{dkRU=YXh$Zelv(R#wEY zuyvtoU;4^H@Zh|s0Ucjdh7p{5RHkFQc^Qv?^Kt!_IMhq-d)M065};KS6k#uFuk`d9 zLc=srKWbVRWV+8KduATbJy@Q*ND*j!SQpPZ;dhvsB0eT~YbE+FAQGK%@ueSV5or>Y z|8Bv8DZF_JSde1}6?X)06QNWSy1lB}wQ7mPTka8^>Wwms^yW(fkFh)fyDw5dfPvfg zpHW9tImk4etH=`Zy;{)|Q?7x=9o+hY`s|7|c}c{$X@vR&{AAxzqYb|)N0&OFzw zg<;It+0&Gd<*!YAzq!ngqJQz|!@bw$R@ z;1*l)RXHn7fIG&bzY*7n@KX2;Sms*?hF@jfm9krU4SiDsl4f=`b!mdeok^Dhg|AB!g{_mI_<< z;kGY-OgimFuca@g0=V5dYgd2B@=)_vz~S;e0<+(t5e{=xey-XlMgFMO=9Gz0jAHq`L6X!Ive~k?rmXV4l?w@=8vGnw>p*bgXMC?d(!` z+`mDMpCgpcym5>CsA?47NtKukQyyHuZnoy@$9Yj_rnux-=(P4neten6)a|Y#IYjbb zQ>3bt*z8|^Wc;3^5c?yf%ng1i zAC>KI9rIS*%(}qH(ml=g)#78vmW{$5a+bAP^E*MHE>rB>GO3Qeti6<G#M9;%GvHnyfJ7=lF`^x(a?fqL7aJ z4Lu_I{e+81x@!?@ZIZ1VZt#u*h$&~2kk|CB?xweC$!rwXz80H2ZZxq zz?~L{_dJVsp6nwYe zm)75;$IQq00XnOys%jmN{rAC%S^&|soRO3ja6VN_t{ z8U27_?D5awUy3c;O`lelA2MI^*NMAQePyi$dOEpvba072(Z#}A{o7fgLWM>Y#QGUs zDtr?DoJ-VpMZX0P`uUaE{##V_#HB4ib*9q`OA^nx@#A-3S?4{zpabbg>iXFUESKoJ ziKkA%wKY?W@`V>(xpRL%Y_^OA?QjI z3ZC1-UjKYF52e#LlTm@EBVHsN^T-XJ?eS^;!Y=`4H@W!RtjIo1&*t)vm3L{BMdv9d zw6yAqMwlhHkvlnX{=LQsLH1`<-MjLuL%6f2hQFket#AUK2>C?@p?Rt&(s5w${&be& zq-v;1V@K?y}+Tbp}^#)HXAME5b4|$wFPckhOKIsX1gnF%~8}1z(qE%@= zVS?)WWUs}lhZPsNqcGa=iH-72h)wRd$bqbUR?kK89tqHfOb~Uu?f`|@?__l$9)TL( zp00aiZA|nu_qS~tTwmInR04KmnC(rP_c2t$fFn}^Pt^2KF8f;cxO$b-TBPShHDznt z)uohY#2hddkbrAN=+~do^2u}d$#(czE3u>+^c1hPRwZc?QgE7EjwjVjbVpRs8sHVb z5DsBqZ0^DmQDAyI+GS_NM{*3#q#i4LBwu$|DP{ns95jawq@b=^l{{`0u7Y&zi6Ts1Fpe-de)+9N+wwN4#Uug1IXIix?D zZ*h9UJFF#uYn;9CB4~ROxOat%yTok7iAC}9A;Ujw^4}kPx8v`s61)b~P3`K(tr_H3j{m6jQ`t|$>ubto`xqL9CE z@1AXxkG;zgsqdZA`n%k0zzbc))T7p_)p}agaOA{NOR;l}8a-M(BjOd?y<7R=>rstw zql1?@QDvR6lGc!f;=Wpu_cqY=?yvT6tg?nxVEyjnsxhUjB8DP0O7VrqIoPXuFi3F% zlEdgCk#CS*m9=M3Go%$#oMzQhrO6oVUffC?L3AF@tEe@C=|ixM?Nf&F00o3Iv#Izx zB~C*B3E~CzXYaRf-`d2jpm08W_pS-e??Hbn#gY{(?1m_gnWw@^*C@^)*NCRGda;9e zi7q@wauJan*fs6#l;yqB?yGch7-k2*3*js*v+vH zvK18Z>(rwHS~^&n6l=EFtfr>s!=GYJa&**Y3^#?-#fAG@V!kqZXUv;|vrB6ufELV! zMLhZC$3UBhzhy)o=|xiW65!T!f|A%+U=Ve4a-&S=28RBC=yUPix>)Q7Kr~v8X(wzx z{W#A9M4d0rWtz_GS+}T`Y}EJo!P$f16f{wu(xkFy%h9rnLM&?N(cQD>eRF(R)7dm| zxg~vr=wc+qWB&X`eA0v8=55|mdNn&}MHg*{G--tSyi1lXyF>w14>^>4t4!*&v2yJ{ zAdiN6YP+sSwToR-H@P~n%73#Ae$n<<6VETOd8IS?F9o9uT$1<;=~Q(Gg#4PO|69qHJA_T* zJGMtH1-Qs`GFtT=Wua(96kqHQQzbt(T*~;*7grlig#OgM=GJy4CN>W52ic|qH`Ga( z+!*%^I97w=VwdgfuXxscM0b^c(2|;elK*ixet5v>e+kKIe#hZI52EJR@{9kUi-sFj z6D3r37}WzGA}D!lSXetYgms^y22Gln(Cu;Ol>!x~>-CerC#|W{!_I!2etNgAc^Ckh z8oL@vhO8+B0C2Xpy#ruoV89M>L@8}D?&JO2j?79txLTp*3qR}L-#3<(S$jok-j3Y} zG*pqHqChzWv8HcxVhiCwxG0TDy#Rx82FQaK95H(Ip)B`C4I8Rq+JVYq_aHTInmg(F zme{fTTDE8*?}D+o!F{;Yv4N%-e;(mDY(Qin=5nAm%*#f=x-D4sJg|I17c?x;lMouYu80STcoT5B( zf7hS#8GMb?r#7)E(f8Jvd4OxMPuGI--~ylYs$wT29Y>*dtD7r;V?o-;nsi*~5>p?^ z0=6?}6cI3Up}XfGdmU(Ju*wT0K*eZwb;z$u zy`GoXk-4GP`=Q~x7&_=9_0dIyiP zYs`XN&ut!dH|H5nY~DOLfT|p0NpF0agnNg(m&4CoKnQaORUMk-Mt2cNNi5UbM~RJC z=*pprYDqg#_TRi|7H0^b*V@&7_UxX5^Kpbp-VCE@+`;VLSD9aE-F%UfaQ&1dAkgHE z-{oDDp-!4fd&wYV-IVjKS{$bhS-qUGoZSGTu6mxb;2O8KSj+wu1weN7Ul59i)z#IX zVI|xE)rgnCHcI6wCC7Hkqrcz9HePfnW)MbF`>@uj-&Y!{SuVW#p;dFshfGKjXD0BH zC{`E^8{kSEY+x|j)bvP>J6|&X&_;am;tS-f?(1H~p!G{n-VDgE@BKD@86h8TlyB7Z zC@C%$+|}yE2lFcvvWU()bf`Oxncs2>l0e;Pk}jc50as8aWnCx%dN-|r;{t8i<<))b_jRh<}Y5^*# z_JWbqb{M+O&(G8fot2VH_kzm&8Pj08A?FyKOpyKIiUqF*Ww!Yx3>dg5Z*ol3JenOI zQ#cj{xCk)o5c?3E4bVHafZhH&d2%$a;#S-AcO$)KWy8u}1ouK9*Dy4|-+%Os85-!2 z#L@%!b8V{a!=hHb7edUIbw>%OASX~a@jyYxwBCaT$-q5!{CGdy4=^}I`v>q_-+w8e zi4r)St%{$XLwN*U86Z#4&G-nCydk{>fS@-dM9ATpn@cAH)|BX_?N7~B!H^scvgjfE z3ZRRSlh1DVHTgw9t3NbTOh=POK-QToJvzFLB6hlUV7q9AxiaI=hHD5y*qENC95O95 zh?F1RZ*2txQsgIqknZgVEz(I`!-~QhCW?AQN<6(@sZF#F-Q=T%PBEnfPg_%77pIUk zR#6#NP8Cx<{X?q%v|M3qQ0{@k%c1ww^aurY6RYfxq|BOU?zBGDAZ z|Ikv+`@S4g=w2?|xs#Bc(V=C_b*$0XB_)TL5uysFh5UYgA$XZAZ;S-Lx{{>SW;V;c zds$dc(R|S%?cp#lo|_0b4GhF~em(NYf*}@<$Do6Qp=w^blem+k%R{4t;)}r51LHBNfXUtdDot-t4kRf=Fxf~hi4U_WD$Wl zXS7bAKF^AawGxjNwiiQKN1ykitqO+yDsql3s&C(j}+EF`8N z%Yf&@5bXzOiFil1dbWCW*4OLSj}k}aE*odEy)N=j-d(SK)8@Ugy;pwxt@{*dco>#^ z0mxYNmIOijWrpahE-^VXXWspB;l-sbk)pPwqx^V})qjDS$GA!7sWWa6^jD1mY(!h; z2})Pya{V{hj6l9J^H!cmeAlS8j(FUovOjk6I(}`;u6(8DU66IXKAKk!&vKx26gtmo-5+Fowaa7wl$k0Z|XD z_b|Ix``VLF9KUhnMsPt>YKjDc#yKWwjJpdA5oxy5yw4>d4-sibU)U}u9Uv;|>M5?P z$7V#u$8U#E8C!Q^UVeUma11J_(!6tGVrfQDY~jjSNrq^@x)SY&_5g+B{&zZ4t}-aW zHmP?$tvR&`W5#p>6$$o<&RIpncM0OiC7b;W;M76FnWEoNrFZpn_Prg@PeFl~SfjHZ z9}ef7KpHC&JlQDY{E3!gmJ~65w9CQ;3*K{wNrSYyE}_=OVU?+=sn5#FnkmE?cRN*A zA;&0Riz)R3fVe3uo9gX+o(d#6=n}@0It%N4I`2TFLzoadB4$xb_JG1#7FMM^RZ!H; zqg}0QT`EQd&zuA3wuDv=v?j`HIFjzaLDtLDetw_my4uVs=J&a`M?{xzT+EUO|C8tp zx7r6=ASvCjXHN$j#iejeQL9}h*^=M&{@esJvmIXpAVnR|Hsw8uiXEHx&f?N1SVke= zY+_1Y#~fo$B$?@c=Ah{0#pc}?SLX)4h;b!pg-g+_*1nd;m+>#iBTNfVF|lJE$^4Sw zmc!a#$H&-j4=k$X11jAe1+xOZ;5O+p(QJ#60;$pl1_I}Num7sdXTgI zCc?b_-apnDl4c@zVC1Blp|-R_c1$m(?iC6rGJ=D7NmYf57#j6v zdh_nvgDbW+I_tsVgdEs;`tnq?|@|KBKe@MY}SCQk-tOs0BJ=7IP_r+?$cuH&jMNUNHM%-_{7*0C!;P9BY?l`VLTJChqOhJWjUlnC zzVQS)Y_YRI)R-NgiLIO}Wf28%GQJupQX*BcDaia!MWD`y!_sl-XgXTp^?GJvL%<$v zH8W{ik{dKt-G=M_SgaNHnPnVVT0~sEDC$0vY&ttNUQ}t+o2V2xFVO(l)lQ zq^XW>ZHN`7eK$h`_Q570fh{3Jluv2bgrU2iid(P5#nbq>PRLe zI!u2U?WcTZJ(Z{6p4~{>An62~;b`QMh#nKT(F{FHd{(?gruBr)Fg#uR_m`ZSbpZhZ zHYEiD34{2dkCQAZvP(D1>%niJ8#NLN8e^OIUs%Dy9|~Q-9=zt+Ht;t*I^=k*ovb4y zp%?;4>VFQQBEB$$<9!mVksj>;aEB?Sxf|7^#WNrqIcvo20@l}{&m59H{{r$APa^4Y zXzE90ZkN}u^WA*e=hCTSF`{fUNJ$_XGRaMlt-TVxTo%_eQ`o=<&7IN;^cUB>dFruV z^BC;JFb2IR2|OZ{rzN9RMnG8yxyHCNn}Kg6MTDMh{XH)Da$d>s&05>Yqccm*(p~(k zyl1j3#X-#6jAy1>rSFRyeM8hO-3=`6g`p8qgwu$Nn~#BsV_Ga)0^>e-APJtMR?#QQ zva8t{6Ohe_1Ba<>B!XtTt}%EAMZwIZNYOQ|uXt5hC=DYbJ?i!(jA`+=7n{<5f-f~|51`HkyJ1Px2d%^L-)ugy^=JFV|v@OyszM)w=C@M0NXuB zs3~s}mJ9{C+}{B7C`=d2r>)VNv*Ug+ahZ^6%pAC$4q61)^wusf?e44!Euz+^Cr;35 zE!9-e1&YSgmoIx(S~d1DzdC_0T!NZA4qaZK*vm_BA_@iYnVb-at=Dp*+iQ7Vx5&ji zprvjYm||$zbIBucLCMf#g_tkwpu}|^fh_i{jQUNQL^;g}uC6b)U)HqLe1XNq4e4os zha~X|fvW`DmLs+d-fib{ZwL|&I!p-~U|8Yi`nr-KMzPn+j7eUcq0O|yUs*>DB0zXM z@UY>^_X-J_q&#VMG;JYVBX;!g=;+P#kCGqH!Voxvvckv}m3%>W@i5V^lXfF2P5=yq z*Q2m>3p$5}RB8*;uXrgWfKK)da`e|Z(K~a>Y;r4p(8q}HaLG;43$qhNj1ufH_$R=$ zu;lVBGea=ZbOU%~jR|c+J@>xmXlj$+C1%kR7^7zl+^EYAE;;W(CK!jrK?Z~mZjKcz zu5aI#VLid*@hix@bWb1dIU(>tf~}=lqId0^7@UV_UifFUJK-QJ=_%xOizfmymqUI1 zzI~mTog_;gMu{-zMxu={`^Jnunm&|<2N;ZKutBUtXHXryJgw~)`x|Q*uKv`RQiaj% z)cYBOC^n+7!bX#=5*g?LSAP(Hn>1t4erg>%Zp8)C63Z)l9+hD|Bsk>=LjW?InFV&~ zxopYy5N-;-8g5`C3mmdzJz$$yBj7S@qFtQagU%#>po*G8aV*8hg6`-bC|~*jv+JRs z<-nnrQH&VelQ3q<3uE9qguRYqbadjqB6c>Xzidn$-yQbHr_YXP%tb%eowlf~ik9o4XBs&pA z5Rn>>t<`>KVNu-U*E9DL7*TN7pFkUODiv$AkkCrk@C&I@}{D ziHr6o_j2DBeQc&wQqo4)$RQbn&nr`PyC4oEy;Map&^4z%c(WjN;>IWl=zS>VvGv3G zhqXR8FgDizh1F12LR}X+1V+}~x4v+G%Aj+{iNJ3yMUu8?Y+c(y3m?@XK!KIqg@)Xh zueV=c3`50>H-iP>buxKfs_fB1peoWaFMzp3CjVt)oa<&T83as&e}2>GJ77RdfrKFZ z{NlUZA`p~Hi>P6t1gG4KUg$%MLA94CXH&^jm-A$Pu;I+eoYi)AzGo zsk!dd0nbnE;63}~YEA_56FoPSn0{#mCMWLg?Oj=U+?WPK^3O#H%Kq5d{w@-#+Cq$B zzY-Fla_uM}DqsQWn;P~#gtI^vlvLYWx0=zh9bNLz)BxP{{^Lhux;qJ_zv8Iep!GPZ za465{3THu-Qw#BbGF|F@0HEkM|DFz#lovcCi3ahxf3X}r05fJ=dEdn`M!LEaV)P%< zaEPvr4!1QFTIee0SFlNBzX{+v`RtNzaHJps5;Tk4)Xei*;NHDk~aP4z~5 zE4~{TevV7cGnXUT5%OKb4_V~)!4g2}^P}fwoAtc=s3+4epBzQ|k(`g&6|r+ugUh9l ziT|l7TEB7o4X3>HE1}6&4*f&&9-kcwXR}J z%e2t7E?BhR;sv)8(Q1GB^5sHywpHn3#|o7o?)RX>)^hX*ivJ5!nCOY5ddnPtBUY~@ zW&VBL=+><`xQI86%WE?_rgTq*`S|gdB^Q+-rTU-MMaoNH3K2~%Te*|>izmH|5V0KA zJfMR)?!!4@#GK41D5u&%O;eAO9bis;4vWgh&&lHE8xZCW1X+Bq<+dNgpVx!Y8_OYqoz+*2V67(QjW4eyv0~U2ybe!36WCc!bb*wS2=p# z)8X`j&7yvRI#0J)c|A2}&H>L20>D#=K)`A%@ZvP<*|VqKr7lP9CwTPuyq?+n&qqyZ z1Ka32PCDy_ut!{aL{(5Vjf%qD>+Ir;1SoCP&%pi)0`!c&M%6_ogh?2^O?<*=Jm<7bNM-6Qx z(`)vy>aq4e&2A22-HIxT!~#Jqlk!#W&+>>h@CKD#xoXvO+C3AjeGHSo-Sy?_>Mqpx zIttN5iXXGz#B6_}657aH@#`|e)?_Ml&^nRJEnnBSfh|Hp$Ko#wDGn^d4d7tnc)?q2 zD95UxS+tBcNsqIWml)=JEVIk??A5I-rSh=McTjc=9hca+{^HlM`Tz&C8I`j*JCOwZ z`L%5!%3nfigdOgBx8)evCwXq}KW<*Tre<_Or)sJ8`>zH1Mn+Cr0US?+woWW^KFx8{ z53>B(lw&xL___ws7p$jVS-?YwRa*DpZoRDeetNIkhF5I&J5ix+VX}R(GCmW?P$c^b z(ejOo_W-~L{=14XPpReYapuU0`{P_b4Qp<>Py#M;djX32&76O}O8@Y=a|Y;f>m7=S zAfdkd@+LFM7y{&Qen>?y^r~(vUJ@vmS)Fwa4GjxlR^1r%WNfghgF^?N)m1r*xjEQo zpH1&MW?UR-`9)(QS}cafkm9e0InCwM(+$W~KhYPgX=$c21OENDaa2a*lv1lzd=*v~ zxAWv9=#oJwdeZYo`W~Yr+(7RShRegC391Kll&8a$IgKtlU0-);&&U;6VAg!RV`?7S zd}V&rHYE4JoN7Rfw7xRQu#kH#d)wWG^4lZ-BeTZj;>`IBKaxO64~^z*r}Zu9P{_#v+pb^(oNPRm=i{tvqu+o8pmkFFAJA0^6VykE?FD zoBvR~JOhLBSrw)_e!Ge5{?ycAV&eiez4YYCwEEA@lC$^g&kxBO4P~F26M7558+lMB zXo3aHh%SA}3(!fg4}Ht&iu`LW2YuGn)}S?`x;gq8mFva zGW3o-w^?4@37rUR@FOHjFZ1{&{+x|PGUukG6XLu8akv6%?l&{e@ai`3C^RJN=mKA` z6XiBL%*^k(iaNuyT|U!iu)Tih+YQHc0$qBP*bKU!Bt`+NwN1p^NtO@HnncMm0@BJw z_uH3oY#H$8o$!&hXSujATePQ97nBKWm(GAcVq(4>P7(gHR2v&qpA|tcOW?)DnF{f?r-fq4lly zBm^Mnta##9U2Zd=3$@n8^JDWL-#<@dc6*FhdD;?AvmyGI4K3ujJo?@|I=F z0IB16R*h_}FSwswHhiZUG~qO-Uy!8JXKnxdGp{?7!HAv;u}5b*E>jO~?vrJ@mmaC; z>(j<7JB`!v*!BnQGBcD=iy1CNfs2xZa~&GQ+;eX&1ugE$oU1+IQi1#jVPKIwyx>|F zaxDuU?fS{y!z2;FiE2@oP=#Bg2@~@nn4wm!TKLP;Tx0zsH*R4QLko3fR;4><$*Er{ zU32&6kI%zpYQZ`&Qd&tv4_~G!^SU>Klx?tA1>aYQn6<|HH4O2O(=@T685Q2DvUFG9-cf3Q%jA-`{e=CO2V{llm}eFNcF$ zQT*rLkK3119z(PZ<6nCk{FWPTT8^oyYFQb1pHXQ1FI-@3&IxkQxsj-e=zH8h^s=mo zop61;@hPO%Gq^H)9gxVZ69##n%Ga&Kq9I>d+yw&hY#CH`1Q2<+Ar`< zwD{2nG_g3R_kz5$%>^FA#bYy}Y<$dudey!__o`*-hqWv-5mE-ytH&Ei7QT+(mZCtoCZX?gteK%W07m5fLiht5s}}fn5~k3 z192A>6_rp7K zA=2?m2%gpL{kOwZj<9!5_XgimPF-XIcRtIhAK03-D#DoK^zOFYleg5JtA`|EYX&xD z$x1`|72NsUd7Z$LTYF|YUhh8T<&FNAVlC#q>WO0-w&*Fpg7CRs`W$BNKS zid2cdyPhDPta|Gx-9g$GRL7!%F4qm>&2Q#YGIaHC1QPYju)4E%Xrak*M%%pdQhEUB z6vR*F_C2o~%m&gT!K%;ay!cIQh0_ZjYV19tUgQv{_3e-$Yg4L^uy`O3P>!-^!FBE+ zm_DevoTk>V>r&>a*vP#?gUJm?z%P=E2c^2v-``&?(#sl9gUTt66u~qXX`5U(O2eOq z${@C1*`Tt%imK7ZVqTX3Kt4(L#%XdfCMapr=k7(7v%7ZiS+X^A&u>Gn;``9*G$MBE zoDQ3ddo5m=HLYUp=ECkz#KC^y)~$(AAB4kpj#>wK(6PA&-uEK<`_W#&)s}~ zgIMZz#fcFy+x#kGUY8#)c27F>ZH3U$iwhj16ia?>%;MyUT{U7FBWSqeAkhW_G|_Sk zKEVopW@1U;A77@Qi)J5v^V5?iZAfV|G4Niq3g0C4K6A{X0{S00>OYGYB?+vzGVVlB zZj)u#zv3vF!E5!x1O!1X*FjM85dc30oyW(a3c})7#GXhXbw_l!SO)|KqITBh_!pf- zanV^VFuM0B^}ZPW_%f3xS<5cYaU?Pc z!8F-O=nb5?@*q94sY__=TqBoW^6EoOz2ywPyL265B3@T`oarBs5>?l&e zq=8gNjJon(KZw*I$P0k9|nf<88dh z({PZklS4`_qT<`4SO1gn36PxVZ`~UxKbd;q^KK}TK`c_DS`Q8V?c=!Hn1=EYpJOO~ z_9ZP?If1fXjm}Xnw2;IO@j;9Jfftl6n-pU!R}W^{wjDZj=V%t~l)RWl1R|8{ikk9?ox<%Ry)-CFpSy_C31g>Ei!uTcK|GXql;t;ZB>`LxohRy7q0M z&72=wdab?{Yd%p${kT6>P4Fr_Df~hiaVff^N-~igY31#g>oF>egKd(F%HujW3l8=+ z`Q$w1^Eq)U?N0(U(Ee~R47uI=p>yvOQ0VvO+Ui^3y~;Q&BLJi1)6pr`=c^Hn7HEWl z_IS|heJ8xup@~*#o$&&Dxdy7gyJu2}&Ar43t?mEWFotEVTSZ^{Wb-M{a;hLC!6L0& zr7hqrupND<^c7LbzID6a?Gzem5t7mNoTYk9$cKgQG>n-1XV8)9O$&HJoWg70lRe;x zadle1-mgz-lk8+>cW!Yy;%0YS)2ws-R(0>kZYi99TeMk={#<*%S^o;({(&=3q8z{} z7oP1QF}dKBdVNpQ|3_w6?J%^l+~;nmG;x|8ePW-*Oae-2{(ycp7*bvy)JIpYH4KoP z4o!nHL9?m%YTKZ@_pq7sz$Ns)^XSL)Q-GrS=ecbchJ>!JKEu@Cq>0I8%H2A#7seX7 zyb~AQNE{+O?wxO*Nm_^bitBw3X7#rOEZ$gnmS%l4`92`IN~1q|K>sB$m!m5fVQN|r zP_i#}fZkg_Pick;H>|1pqYZLq$As9q>4^CE)YmqfjkTH_5e+SUqy$AnuyRQqY*if$ zGW%>@cAk12T*rsT!LM^&u_XZAVWE1?!$&-JEXUSCbgbeMx=lnETH)=i;{kG+6z_km zyXkz|4RCF_wLzdk8hQyFGmH8yDQzz3Nz%Dx!AAhI(uWLWs+EAw(~I&OB3}@4FT27c zlq+2LKAO>->5c5I^qL<${$a@$!!2={$~Qn!$WlzO&j}P5y*dbV;NsP*rlqU+t|T8! zv?Z7mBpwNn;5Xia^;DBkD~IhHizgL$9^ES7BJF`xPZ{8VETiMXN^)cC(aGU`l=s|l zbuVirI-2#<4=}A^CtuPx72LhyutO?8S2Q~^H@R@okx_WAu1Bl!p+~g59Tq?LtkoPL$}6pj#x|G9Cs#=+T~&JH!Zlc)sN^5DomQNg z$J~f?>^WV6-kT!8Z8V%$GOJDp=6txJyHV0!ii$K(uA$&%*@dvhSMs=2g7VX~w6!gq z2MvRdL4v31V_p)coD$gP2>nFs^O2!7cV&F<^JYT{k|p6}_r#^#3t)Bz)7-He+;bCC zu0?Q^hR4x}p(SxzY%@|OLrWTYD^D+*M{y(~RU%(e(O-`|BY4qA{-uPOwo+G@$EWw0 z@cPLQW^;6Ni&q~$nw(5Yl{y$6eIteEe^?-wpeqCSSYHn{yN~Ax z19dxhuPG^f5QnWoM((JA#-4>}bi zY9}f-0n1xy3o<=>?*V`UPn64^c;m;jpYA9eq`SSxHFt>bu+|*}yP-#ij&xboext*ipGyzT=78rc$=ebDO7 zpE|VjJbtqP( z6&1Ib#`699_alhQX-cSJ^F#HnTD*e_8V;%v^b@0eZeP4O^6XknEkj^k#Djy+0-YlAVD~ETt!BC-=UTS zG9iVeJ~Az6a2d9lEJ+Y}@E~aSG(a2y5dnQg6MfV!zOz6z&6}$rA>r6itE1w~ z(XouCmI&1ZEFssH0Uk?~JeM3cVs;37dK8|AU$Y55N%*)gjFZy3OA#D0q#fgcFs>+& zyXoKD4?QPWW8lybhMC!44r`EjZjRDA6%jpFP;=aoX-0C1kyxvx#m5;)f>KcN6}#Yf zjv;g@Er3oXsRtr(efT>d&T=fk0QoPVTiS&DMBpq22oANbJQfNyr+!@=2y(5F+^WIA zMOGn^H4=+3VZWj9-{CM!D7+--hd+tLW;^hnV0OaY39c>a$g~Y&%7G6OxjF5Rq~Ss} zzA0*D>Ed~PaV5Y#?4Me4oidyiB4Nij%LPc&Dau5fYEy%F!fSbBV-u%OvYl5IZMBFh zB{FNibqRjXt{U$B!?D;tl*D4#e-aRCDS%XZ=MEj2J97>kzQ9Eso$KTL>H0VEUC#u3 zCJr&m<}0iHGQLs4dj=aAXe&43Uy`N;SebdGl@SElz=e;b7|8`fNQmDvf1uxcCb0oi zhNBT+vQTubQzX*F)0rE)cpbIT4q!4B#TxjsF@X6&auV72(yyUix&+%LvHOBoaVMoO z3QA7v9)*=fq0_X2Cz=#@1i<4xN-M!YEq5X@S(GEgBqdz#zL1>saVx$QJshMwPLn44 zlkOa~?rO=xz#t|evnb`8LoCAlJ;jTdK2(27GiByCMpTJq5Wwrowod-dhF7Z*+jn+F zS!XIsTz2Z7!XI;?G3AC!Y#d!X&Sg0RBq9sx0WmOK&jMPw9q&+|1sZ(*ds+F z0@PYZAt#tz7(|EdOi$=`^+BAuzNcYMVc8kOZ{UyCAXiw( zbs}Q#W}p~}226yB!VF>!mK%G-@hSPHSt$?+5;qI1tqQwCJ~y74bRMVp<=NgA0sgCC z?5IxlKY!2A;>tWk7q`>Az0sAwpv0zsTTf{HsN!}9^-7Lx(z?GT`(}UTQk|fXY3`FM zlNT&nv<++aGhiVhEztcnBh#LSQtlJw{0jCs6MpUI{4%F?5n4Y7`*??DsxIApx%Gp~ znq%wMGVfX^X}$lP>=}P2tg+0Pm2$mX@*RJ-Sv}g-UPt&Xh~FIx4)c z*4?Lju33X>XhGtv@)_BGKKBZl8iQvw{K%1dBr)&%;^PgJ+_bJw)V&XDD8i&*`;a0S zaq?u#ef#!3AHvy3gX>^8*MS=>&CSgxSz4~eUE`pm{~Gv+Gqk*n0V60RBqAlHm#xxkE4HZF-9Y`s1fh&mqZL zPqVYOUWfT25N)CxTIN0!rm9N1<*Q#M%v1FgC`pf0~$H>CfuC5L75Cqn~0693y z#KiB$GK38eQdgh3+`sq2SF9@zh>_TTO(soNtFwH?ibJPQx0Vb|i@vEn_Rn9mXv2gf zmzLcc_G-!dYux2zx~=0yKmy4GlbiC^yjIOhOoGwE0e|a|t zVAN^J`Q&O}q*Y@KPg8t^N_I=?DEtz*e7~VXUG)!nvsIh+9qHZI#kN8E(>BkKus0mInyVyz2~ zX>HoIYmn+x7PJQ2aRYkF(_i}hMNVq}$$|7;9BSbwPu_KPk49PX?T7bR4j!$6FUMl8P{g%rBuD#!?TuHfCTR3ddod*1qB6&<42mAH4udzOl|C(R?zB`Vso2Dy?FI% zBh5tH#;UPlyAeLyI^63>=p0U?+i#W!*1AB!PN(D_dt?H;n{niCE&HhZ_azPttk6Ng|KRp7c9q7|BqZfW{oP2C!FB4iIsoIdc zqYmBa$^_^5si^{|Kz{>+`Af5OP-yvf-E?wjYYqJaAEKIM-Pm;E^98M6CcFd>-62CB z&fYh6yr>*|?!2zyO$B2m(k9BCc8zylUr)mWQKho0k)h$I2Q#0S&}#;us_o2l$LCsm zeO5YVGf_B!SJ(Z}hE7V(L?)`&Wo1{td>IT8%Gu=$x3cugofluT-sP0L`p;(tOiiOW1n*LGx4<& z$#Cb-zXbwX{W_E|po*ymeCfe_LQWX@`>hKxFP5M{h6+=<~m*ao$-$ETK|Kt!GgZS z{6;-na{k8ZFoWdKT}eOB?xEocGS{RizbmNSX`TTxJv4IZ#ME$o-r~x3&_6?&Wu(E* zL*SreL54=1k$-mFW!e4Kycg5lkqlvk16-{wzT&}qf8W$U`u|D(&QE0-^m$}+WDcQ;eV z!BeMNjXBg@b)5>eY30+k^+dV$+y0g5u08=dzn`9T{*-+m;X6dVA~D!&={+;FDx@I4x~j_1m`tYdG?8FH@PQe$nVB+%U317}#*zsLj$O#>4oT%J} z8W=1sxz`7pm!(XRxiEL2{m1bX02s6p6DDlSU11fQJA>EceCPD>J*)G>`{1x$Wl>hD z8w^U?>jr&V^NKrA5@uX(2rzl@{Q1rd_=+E2(+)*N)uri?i^$Xn4M9FEpMA#DBUM#5 zp45xn9i8hcS10D@=X1oh?b78cQgr!KkKU#w+V>gPb7!IYfzDoEoNbg!Nxz1S+~BBd z3dnKx#_jx&m^GiqzR_--d{xKYVRElwt3ytBTD%LDFAU6rCH$Owe|*RF9L>;;I+$j& zE)Ub|z=LxW`?z>MDjpE!v|81B^R=*9KOM4EJz&O{KLO9`jzts)&E}IQpJz|LW$bS~ zb!wATr)B|T)^Yb)eGZD7H?vPT#Ob9~-g9W%x$_mI)|$r$kNDYV!EeUKpC=r-`wR^f z+b^+$agSGbODd1X#f`DF+%fp+UX%(QUp(k%^2;vXffXe4J!?E7CMM(I!=%0;hqBmm zJTS6*I;>kYtIgvE&F^G?0aTEbao+2uP1juhjmzdO<)07B zGcY#(*^!MP_iKZ%ZUKiv>c45z>uG-f+W!A>yKA($s2TptX3qz!`Ejhn0r%@Cb>#C7 z3WZ{ynej-QfBu#3zv+