Weaver is a multi-layer collection and diagnosis prototype for GPU training workloads. It puts Python/operator events, CUDA kernel events, and warp/block auxiliary information onto one timeline, then uses the timeline to diagnose abnormal kernel ordering, dependency blocking, CPU enqueue delay, and GPU execution slowdown.
The current project supports:
- Low-intrusion collection: a daemon receives events, and target training processes report events asynchronously through a Unix datagram socket.
- Python/operator layer: explicit operator scopes and adaptive Python stack sampling are supported. Full Python profiling is kept for debugging, but it is not the recommended low-overhead default.
- CUDA/NCCL kernel layer: an
LD_PRELOADhook intercepts CUDA driver/runtime APIs and NCCL APIs to collect kernel launch metadata, CPU enqueue time, GPU start/end time, streams, grid/block configuration, and related information. - Warp/block auxiliary layer: launch configuration is used to derive
blocks_total,threads_per_block,warps_per_block, andtotal_warps. Weaver can also run a sidecar that performs Neutrino-inspired static disassembly summaries on captured kernel binaries. - Sketch-based diagnosis: Weaver first builds an expected execution sketch from a clean trace, then compares an anomalous trace against the sketch and reports a root-cause evidence chain.
Note: the current warp/block layer does not fully reproduce Neutrino runtime programmable probing. Weaver does not yet insert probes into PTX, reassemble probed kernels, replace original kernel launches, or collect true per-warp/per-block runtime traces. The current implementation is better described as low-overhead warp-scale metadata plus static structure features for diagnosis.
The daemon receives online collection events and writes them to an NDJSON file:
python -m weaver.daemon.server \
--sock /tmp/weaver.sock \
--http-port 18731 \
--out ./weaver_events.ndjsonEach event is a JSON object. Common fields include:
ts_ns: event timestamp.pid/tid/rank/local_rank.layer: event layer, such aspython,operator,cuda,hook, orneutrino.kind: event type, such asoperator,kernel_launch,stream_wait_event, orkernel_disassembly.payload: layer-specific details.
Weaver currently supports two operator sources:
- The diagnosis example uses explicit
operator_scopecalls. The workload reports operator start/end timestamps around key operators. - The generic collection path supports adaptive Python stack sampling. When the CUDA hook reports an extra kernel, the daemon sends a signal to the target process, and the target process samples the current Python stack once to emit an operator clue.
The older full CPython profiling path is still present in the codebase, but it has higher overhead and is not the default low-overhead path.
The kernel layer is implemented by hooks/libweaver_hook.so. It intercepts APIs through LD_PRELOAD, including:
cuLaunchKernel/cuLaunchKernelExcudaLaunchKernel/cudaLaunchKernelExCcuGetProcAddresscuModuleLoad*/cuModuleGetFunction- NCCL collective APIs such as
ncclAllReduce
Collected fields include:
- Kernel name.
- CPU enqueue start/end time.
- GPU start/end time and GPU duration, measured with CUDA Events.
- Stream, stream index, and rank.
- Grid/block/shared-memory configuration.
threads_per_block,blocks_total,warps_per_block, andtotal_warps.- Collection-mode markers such as selective, full, name-only, and drop.
The recommended default is selective mode: important GEMM/NCCL/sync kernels are timed with GPU start/end; some layout/reduction kernels are recorded as name-only metadata; many low-value small kernels can be dropped to reduce overhead.
The current warp/block layer has two sources:
- Launch-derived metadata:
blocks_total = gridDimX * gridDimY * gridDimZthreads_per_block = blockDimX * blockDimY * blockDimZwarps_per_block = ceil(threads_per_block / 32)total_warps = blocks_total * warps_per_block
- Neutrino-inspired static disassembly sidecar:
- Captures loaded kernel binaries.
- Invokes
nvdisasmorcuobjdump. - Summarizes static structure features such as instruction count, global load/store count, shared-memory operations, tensor operations, barriers, branches, and returns.
To enable disassembly:
export WEAVER_ENABLE_DISASM=1
export WEAVER_EMIT_CODE_EVENTS=1If nvdisasm or cuobjdump is unavailable, kernel-layer collection still works, but disassembly summaries will be missing or degraded.
Recommended environment:
- Linux server.
- NVIDIA GPU with a working CUDA driver.
- Python 3.9+.
- GPU-enabled PyTorch.
gccandmake.- Optional: CUDA Toolkit binaries such as
nvdisasmandcuobjdumpfor disassembly summaries.
Install and build:
cd ~/tmp/weaver
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip setuptools wheel
pip install -e .
# Install the PyTorch build that matches your server's CUDA version.
# Use the official PyTorch install command for torch/torchvision/torchaudio.
make -C hooks
make -C weaver/collector PYTHON=$(which python)If you move the project between machines or copy a build artifact from macOS to Linux and see invalid ELF header, rebuild the hook:
make -C hooks clean
make -C hooksRecommended runtime environment variables:
export PYTHONPATH=$PWD
export PYTHONPYCACHEPREFIX=/tmp/weaver_pycache
mkdir -p /tmp/weaver_pycacheBasic one-shot collection for a training script:
PYTHONPATH=. \
PYTHONPYCACHEPREFIX=/tmp/weaver_pycache \
python -m weaver.collector.launch \
--out ./weaver_events.ndjson \
--collection-mode selective \
--python-mode adaptive \
--warp-mode launch \
-- python your_training.pyTo enable Neutrino-inspired disassembly summaries, replace --warp-mode launch with:
--warp-mode neutrinoThe diagnosis example is a controlled single-GPU CUDA workload. In the clean run, each iteration executes GEMM_A -> GEMM_B -> Reduction/Layout. In the anomalous run, Weaver inserts an extra transpose().contiguous() layout kernel between GEMM_A and the target GEMM_B. The system first builds an expected sketch from the clean trace, then diagnoses the anomalous trace against that sketch. The expected root cause is that compute.extra_transpose blocks compute.gemm_B.
Run the single-GPU diagnosis example:
cd ~/tmp/weaver
rm -rf ./sequence_diag_1gpu_demo
mkdir -p /tmp/weaver_pycache
CUDA_VISIBLE_DEVICES=0 \
PYTHONPATH=. \
PYTHONPYCACHEPREFIX=/tmp/weaver_pycache \
python examples/run_sequence_diagnosis_single_gpu.py \
--output-dir ./sequence_diag_1gpu_demo \
--iters 4 \
--strict-validationThe script runs three stages:
- Clean full collection: run the normal workload and collect full kernel information.
- Sketch construction: build an expected execution sketch from the clean trace.
- Anomaly diagnosis: inject the
extra_transposeanomaly, collect an anomalous trace, and compare it against the expected sketch.
Main output files:
sequence_diag_1gpu_demo/clean_full/weaver_events.ndjson: clean trace.sequence_diag_1gpu_demo/expected_sequence_sketch.json: expected execution sketch.sequence_diag_1gpu_demo/anomaly_extra_transpose/weaver_events.ndjson: anomalous trace.sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.txt: diagnosis report.sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.json: structured diagnosis report.sequence_diag_1gpu_demo/summary.json: experiment summary and validation result.
Inspect the result:
cat ./sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.txt
python -m json.tool ./sequence_diag_1gpu_demo/summary.json | lessIn a successful run, extra_transpose.validation.passed in summary.json should be true, and the diagnosis report should include:
- Abnormal type:
dependency_blocked. - Target operator:
compute.gemm_B. - Root-cause operator:
compute.extra_transpose. - Meaning:
GEMM_Bwas expected to immediately followGEMM_A, but an extra layout kernel was inserted before it, delaying its GPU start.
If you want the script to generate reports without returning a non-zero code when validation fails, remove --strict-validation.
weaver_events.ndjson: raw event stream, one JSON object per line.expected_sequence_sketch.json: expected execution order and dependency rules generated from the clean trace.diagnosis_report.txt: diagnosis summary. Start with root cause, abnormal type, confidence, and evidence.summary.json: machine-readable experiment summary, useful for tables and scripts.
Useful checks:
grep -c '"kind":"kernel_launch"' ./sequence_diag_1gpu_demo/anomaly_extra_transpose/weaver_events.ndjson
grep -c '"layer":"operator"' ./sequence_diag_1gpu_demo/anomaly_extra_transpose/weaver_events.ndjson
grep -n 'extra_transpose\|dependency_blocked\|gemm_B' ./sequence_diag_1gpu_demo/anomaly_extra_transpose/diagnosis_report.txtIf you find it useful in your research, please consider citing:
- BibTeX:
@inproceedings{zhu2026weaver,
author = {Zhu, Jingyuan and Zhang, Menghao and Chen, Yuxuan},
title = {Weaver: Diagnosing Extra Kernel and Synchronization Interference in GPU Workloads},
year = {2026},
isbn = {9798400724671},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3789240.3830303},
doi = {10.1145/3789240.3830303},
booktitle = {Proceedings of the ACM SIGCOMM 2026 Posters and Demos},
numpages = {3},
location = {Denver, CO, USA},
series = {SIGCOMM Posters and Demos '26}
}