Skip to content

Latest commit

 

History

History
238 lines (174 loc) · 9.8 KB

README.md

File metadata and controls

238 lines (174 loc) · 9.8 KB

Weaver

Weaver is a multi-layer collection and diagnosis prototype for GPU training workloads. It puts Python/operator events, CUDA kernel events, and warp/block auxiliary information onto one timeline, then uses the timeline to diagnose abnormal kernel ordering, dependency blocking, CPU enqueue delay, and GPU execution slowdown.

The current project supports:

  • Low-intrusion collection: a daemon receives events, and target training processes report events asynchronously through a Unix datagram socket.
  • Python/operator layer: explicit operator scopes and adaptive Python stack sampling are supported. Full Python profiling is kept for debugging, but it is not the recommended low-overhead default.
  • CUDA/NCCL kernel layer: an LD_PRELOAD hook intercepts CUDA driver/runtime APIs and NCCL APIs to collect kernel launch metadata, CPU enqueue time, GPU start/end time, streams, grid/block configuration, and related information.
  • Warp/block auxiliary layer: launch configuration is used to derive blocks_total, threads_per_block, warps_per_block, and total_warps. Weaver can also run a sidecar that performs Neutrino-inspired static disassembly summaries on captured kernel binaries.
  • Sketch-based diagnosis: Weaver first builds an expected execution sketch from a clean trace, then compares an anomalous trace against the sketch and reports a root-cause evidence chain.

Note: the current warp/block layer does not fully reproduce Neutrino runtime programmable probing. Weaver does not yet insert probes into PTX, reassemble probed kernels, replace original kernel launches, or collect true per-warp/per-block runtime traces. The current implementation is better described as low-overhead warp-scale metadata plus static structure features for diagnosis.

Collection Design

1. Daemon

The daemon receives online collection events and writes them to an NDJSON file:

python -m weaver.daemon.server \
  --sock /tmp/weaver.sock \
  --http-port 18731 \
  --out ./weaver_events.ndjson

Each event is a JSON object. Common fields include:

  • ts_ns: event timestamp.
  • pid / tid / rank / local_rank.
  • layer: event layer, such as python, operator, cuda, hook, or neutrino.
  • kind: event type, such as operator, kernel_launch, stream_wait_event, or kernel_disassembly.
  • payload: layer-specific details.

2. Python / Operator Layer

Weaver currently supports two operator sources:

  • The diagnosis example uses explicit operator_scope calls. The workload reports operator start/end timestamps around key operators.
  • The generic collection path supports adaptive Python stack sampling. When the CUDA hook reports an extra kernel, the daemon sends a signal to the target process, and the target process samples the current Python stack once to emit an operator clue.

The older full CPython profiling path is still present in the codebase, but it has higher overhead and is not the default low-overhead path.

3. CUDA / NCCL Kernel Layer

The kernel layer is implemented by hooks/libweaver_hook.so. It intercepts APIs through LD_PRELOAD, including:

  • cuLaunchKernel / cuLaunchKernelEx
  • cudaLaunchKernel / cudaLaunchKernelExC
  • cuGetProcAddress
  • cuModuleLoad* / cuModuleGetFunction
  • NCCL collective APIs such as ncclAllReduce

Collected fields include:

  • Kernel name.
  • CPU enqueue start/end time.
  • GPU start/end time and GPU duration, measured with CUDA Events.
  • Stream, stream index, and rank.
  • Grid/block/shared-memory configuration.
  • threads_per_block, blocks_total, warps_per_block, and total_warps.
  • Collection-mode markers such as selective, full, name-only, and drop.

The recommended default is selective mode: important GEMM/NCCL/sync kernels are timed with GPU start/end; some layout/reduction kernels are recorded as name-only metadata; many low-value small kernels can be dropped to reduce overhead.

4. Warp / Block Auxiliary Layer

The current warp/block layer has two sources:

  • Launch-derived metadata:
    • blocks_total = gridDimX * gridDimY * gridDimZ
    • threads_per_block = blockDimX * blockDimY * blockDimZ
    • warps_per_block = ceil(threads_per_block / 32)
    • total_warps = blocks_total * warps_per_block
  • Neutrino-inspired static disassembly sidecar:
    • Captures loaded kernel binaries.
    • Invokes nvdisasm or cuobjdump.
    • Summarizes static structure features such as instruction count, global load/store count, shared-memory operations, tensor operations, barriers, branches, and returns.

To enable disassembly:

export WEAVER_ENABLE_DISASM=1
export WEAVER_EMIT_CODE_EVENTS=1

If nvdisasm or cuobjdump is unavailable, kernel-layer collection still works, but disassembly summaries will be missing or degraded.

Environment Setup

Recommended environment:

  • Linux server.
  • NVIDIA GPU with a working CUDA driver.
  • Python 3.9+.
  • GPU-enabled PyTorch.
  • gcc and make.
  • Optional: CUDA Toolkit binaries such as nvdisasm and cuobjdump for disassembly summaries.

Install and build:

cd ~/tmp/weaver

python3 -m venv .venv
source .venv/bin/activate

pip install -U pip setuptools wheel
pip install -e .

# Install the PyTorch build that matches your server's CUDA version.
# Use the official PyTorch install command for torch/torchvision/torchaudio.

make -C hooks
make -C weaver/collector PYTHON=$(which python)

If you move the project between machines or copy a build artifact from macOS to Linux and see invalid ELF header, rebuild the hook:

make -C hooks clean
make -C hooks

Recommended runtime environment variables:

export PYTHONPATH=$PWD
export PYTHONPYCACHEPREFIX=/tmp/weaver_pycache
mkdir -p /tmp/weaver_pycache

Basic one-shot collection for a training script:

PYTHONPATH=. \
PYTHONPYCACHEPREFIX=/tmp/weaver_pycache \
python -m weaver.collector.launch \
  --out ./weaver_events.ndjson \
  --collection-mode selective \
  --python-mode adaptive \
  --warp-mode launch \
  -- python your_training.py

To enable Neutrino-inspired disassembly summaries, replace --warp-mode launch with:

--warp-mode neutrino

Diagnosis Example

The diagnosis example is a controlled single-GPU CUDA workload. In the clean run, each iteration executes GEMM_A -> GEMM_B -> Reduction/Layout. In the anomalous run, Weaver inserts an extra transpose().contiguous() layout kernel between GEMM_A and the target GEMM_B. The system first builds an expected sketch from the clean trace, then diagnoses the anomalous trace against that sketch. The expected root cause is that compute.extra_transpose blocks compute.gemm_B.

Run the single-GPU diagnosis example:

cd ~/tmp/weaver

rm -rf ./sequence_diag_1gpu_demo
mkdir -p /tmp/weaver_pycache

CUDA_VISIBLE_DEVICES=0 \
PYTHONPATH=. \
PYTHONPYCACHEPREFIX=/tmp/weaver_pycache \
python examples/run_sequence_diagnosis_single_gpu.py \
  --output-dir ./sequence_diag_1gpu_demo \
  --iters 4 \
  --strict-validation

The script runs three stages:

  1. Clean full collection: run the normal workload and collect full kernel information.
  2. Sketch construction: build an expected execution sketch from the clean trace.
  3. Anomaly diagnosis: inject the extra_transpose anomaly, collect an anomalous trace, and compare it against the expected sketch.

Main output files:

  • sequence_diag_1gpu_demo/clean_full/weaver_events.ndjson: clean trace.
  • sequence_diag_1gpu_demo/expected_sequence_sketch.json: expected execution sketch.
  • sequence_diag_1gpu_demo/anomaly_extra_transpose/weaver_events.ndjson: anomalous trace.
  • sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.txt: diagnosis report.
  • sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.json: structured diagnosis report.
  • sequence_diag_1gpu_demo/summary.json: experiment summary and validation result.

Inspect the result:

cat ./sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.txt
python -m json.tool ./sequence_diag_1gpu_demo/summary.json | less

In a successful run, extra_transpose.validation.passed in summary.json should be true, and the diagnosis report should include:

  • Abnormal type: dependency_blocked.
  • Target operator: compute.gemm_B.
  • Root-cause operator: compute.extra_transpose.
  • Meaning: GEMM_B was expected to immediately follow GEMM_A, but an extra layout kernel was inserted before it, delaying its GPU start.

If you want the script to generate reports without returning a non-zero code when validation fails, remove --strict-validation.

Reading Common Outputs

  • weaver_events.ndjson: raw event stream, one JSON object per line.
  • expected_sequence_sketch.json: expected execution order and dependency rules generated from the clean trace.
  • diagnosis_report.txt: diagnosis summary. Start with root cause, abnormal type, confidence, and evidence.
  • summary.json: machine-readable experiment summary, useful for tables and scripts.

Useful checks:

grep -c '"kind":"kernel_launch"' ./sequence_diag_1gpu_demo/anomaly_extra_transpose/weaver_events.ndjson
grep -c '"layer":"operator"' ./sequence_diag_1gpu_demo/anomaly_extra_transpose/weaver_events.ndjson
grep -n 'extra_transpose\|dependency_blocked\|gemm_B' ./sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.txt

If you find it useful in your research, please consider citing:

  • BibTeX:
@inproceedings{zhu2026weaver,
  author = {Zhu, Jingyuan and Zhang, Menghao and Chen, Yuxuan},
  title = {Weaver: Diagnosing Extra Kernel and Synchronization Interference in GPU Workloads},
  year = {2026},
  isbn = {9798400724671},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3789240.3830303},
  doi = {10.1145/3789240.3830303},
  booktitle = {Proceedings of the ACM SIGCOMM 2026 Posters and Demos},
  numpages = {3},
  location = {Denver, CO, USA},
  series = {SIGCOMM Posters and Demos '26}
}