docs: TraceLens-free trace analysis guide and raw-trace parser

[amd-bartgips]

Why this exists alongside the existing guide

TRACE_ANALYSIS_GUIDE.md tells analysts to run TraceLens to produce an xlsx
report and then read its sheets for per-op facts. That works well when TraceLens
knows the ops. But TraceLens silently drops operations it does not recognize.
For any trace that exercises new models or new kernels — the exact situation where
we most need a complete picture — the ranking is incomplete and analysts cannot
hand off accurate ticket information.

This PR adds a parallel workflow (nothing in the existing guide is changed or
removed). It reads the raw Kineto/roctracer JSON directly, so every operation
appears in the output regardless of whether TraceLens has seen it before.

Raised in #tiger-aiter-perf-eng / #tiger-aiter-kernel-support: Sara flagged
the TraceLens blind-spot; Sami proposed relying on raw trace data for shape
extraction; this is the concrete tooling outcome of that thread.

---

What's added

docs/examples/analyze_trace.py

Standalone raw-trace parser. No TraceLens, no external packages (stdlib gzip +
json only).

python docs/examples/analyze_trace.py <eager_trace.json.gz> [--threshold 2.0]
                                       [--op NAME] [--json out.json]

Reads the Kineto cpu_op / kernel / cuda_runtime event stream, attributes
GPU kernel time to its launching op via the correlation chain, and reports:

• op ranking by % of total GPU kernel time
• per-op: full un-truncated GPU kernel names with per-kernel timing breakdown
• per-op: shape variants (Input Dims / Input type / Strides / Concrete Inputs
  exactly as Kineto recorded them on the cpu_op event)
• --json for downstream tools (e.g. /silotiger-ticket)

docs/TRACE_ANALYSIS_GUIDE_RAW.md

New guide mirroring the structure of TRACE_ANALYSIS_GUIDE.md but replacing the
TraceLens extraction sections with raw-trace equivalents driven by the script.
Carries over Sections 3–6 (ticket brief template, reproducer, MoE untuned CSV,
gotchas/checklist) unchanged — those sections were always tool-agnostic. Adds an
explicit note from the Slack thread: report concrete extracted facts; do not
fabricate roofline/NV-comparison targets.

---

Example output — MI355 Qwen3-VL w4a4 eager trace

Validated against mi355_w4a4_vllm23_offline_8x_eager.trace.json.gz
(17.2 M events, 8× MI355, offline batch).

Op ranking

Op                                                                %       total    calls
------------------------------------------------------------------------------------------
aiter::fused_moe_                                             32.1%  1456.30 ms     4136
vllm::unified_attention_with_output                           23.7%  1076.18 ms     1880
aiter::mha_varlen_fwd                                         13.3%   605.38 ms      108
aiter::_gemm_a4w4_asm                                          7.3%   330.46 ms     3760
aiter::fused_qk_norm_mrope_3d_cache_pts_quant_shuffle          6.8%   306.51 ms     1880
aten::addmm                                                    4.6%   209.86 ms      468
aiter::dynamic_per_group_scaled_quant                          3.1%   139.66 ms     6016
_C::fused_add_rms_norm                                         2.5%   114.09 ms     3760

  Total attributed GPU kernel time: 4538.82 ms

37,342 / 37,522 kernels attributed (remainder are memcpy / setup events outside
profiled ops). Numbers match the TraceLens baseline to within 0.2%.

--op aiter::fused_moe_ — what works well

This is an op that TraceLens drops from its detail sheet (it treats it as an
aggregation parent and shows 0 rows in unified_perf_summary). The raw approach
gets the full picture:

Op: aiter::fused_moe_
    32.1% of total GPU kernel time  |  1456.30 ms total  |  4136 calls

  GPU kernels dispatched:
    [  376×] mean   1.45 ms  total  546.24 ms  min  502.4 µs  max   1.91 ms
             mfma_moe1_silu_mul_afp4_wfp4_bf16_t64x128x256_pm1_async_v32
    [  376×] mean   1.28 ms  total  479.80 ms  min  366.0 µs  max   1.71 ms
             mfma_moe2_afp4_wfp4_bf16_cshuffle_t64x256x256_vscale_fix3_fp4opt_v1_persist_cu256_acc0
    [ 1504×] mean  129.4 µs  total  194.65 ms  min  110.4 µs  max  154.4 µs
             mfma_moe1_silu_mul_afp4_wfp4_bf16_t64x64x256_pm1_async_v32
    [  376×] mean  328.4 µs  total  123.47 ms  min   93.7 µs  max  466.6 µs
             moe_reduction_kernel_plain_bf16_topk8_md4096
    [ 1504×] mean   74.6 µs  total  112.14 ms  min   63.7 µs  max   86.8 µs
             mfma_moe2_afp4_wfp4_bf16_cshuffle_t64x256x256_vscale_fix3_fp4opt_v1_persist_cu256

  Shape variants (19 distinct call groups, top variant shown):

  Variant 1  (564 calls,  mean 1.31 ms)
    Input Dims:      [[32768, 4096], [128, 3072, 2048], [128, 4096, 768], [32768, 8], [32768, 8], [], ..., [128, 3072, 128], [128, 4096, 48], ...]
    Input type:      ['c10::BFloat16', 'c10::Float4_e2m1fn_x2', 'c10::Float4_e2m1fn_x2', 'float', 'int', ...]
    Input Strides:   [[4096, 1], [6291456, 2048, 1], [3145728, 768, 1], [8, 1], [8, 1], ...]
    Concrete Inputs: ['', '', '', '', '', '', '0', '3', 'False', ..., '0', '15', '0', '0', ..., '0.', '']

All five dispatched GPU kernels are named and timed individually. Input shapes
show M=32768 activations (BFloat16), FP4 gate+up weights [128, 3072, 2048]
and down-projection weights [128, 4096, 768] — physical K, so logical K = ×2.
Scale tensors [128, 3072, 128] and [128, 4096, 48] are also visible
(one E8M0 value per 32 weight elements).

--op vllm::unified_attention_with_output — known limitation

The Q tensor shape is visible; KV-cache tensors and scalar parameters are empty —
this is the decode attention problem (gotcha 4/5 in both guides: the kernel reads
KV cache via internal vLLM state, not explicit tensor arguments):

Op: vllm::unified_attention_with_output
    23.7% of total GPU kernel time  |  1076.18 ms total  |  1880 calls

  GPU kernels dispatched:
    [ 1880×] mean  572.4 µs  total 1076.18 ms
             kernel_unified_attention_2d

  Shape variants (top variant):
    Input Dims:      [[32768, 64, 128], [], [], [32768, 64, 128], [], [], [], []]
    Input type:      ['c10::Float8_e4m3fn', '', '', 'c10::BFloat16', '', '', '', '']
    Concrete Inputs: ['', '', '', '', '', '', '', '']

We can read nheads=64, head_dim=128 (padded; logical=72 via softmax_scale), Q dtype
FP8 e4m3fn, output dtype BF16. Block_size, page_size, and KV cache layout remain
invisible — must be read from the serving config or from the KV-cache-writing op
(aiter::fused_qk_norm_mrope_3d_cache_pts_quant_shuffle, whose Concrete Inputs
carry block_size=64 and page_size=16 at positions 24–25). This limitation is the
same whether you use TraceLens or the raw trace; it is a property of how vLLM
passes KV cache state, not of the analysis tool.

---

What's not duplicated

This PR does not overlap with PR #1. PR #1 added TRACE_ANALYSIS_GUIDE.md (the
TraceLens-based guide). This PR adds two new files alongside it; the existing
guide and any ongoing TraceLens-based workflows are unaffected.

AI assistance

This PR was developed with Claude Code assistance. The commit message, guide
content, script, and PR description were reviewed and are accurate to the best of
my knowledge. The example output was produced by running the script against the
real MI355 trace and is not generated or hallucinated.