Skip to content

perf: route low-token qwen3 decode through optimized nz quantmatmul. - #2248

Draft
maojunx99 wants to merge 3 commits into
xLLM-AI:mainfrom
maojunx99:perf/qwen3vl-decode-nofusion
Draft

perf: route low-token qwen3 decode through optimized nz quantmatmul.#2248
maojunx99 wants to merge 3 commits into
xLLM-AI:mainfrom
maojunx99:perf/qwen3vl-decode-nofusion

Conversation

@maojunx99

Copy link
Copy Markdown
Collaborator

Description

  • Add a dedicated graph variant for optimized W8A8 BF16 decode.
  • Route eligible QKV, gate/up, and down projections with up to 16 tokens.
  • Preserve the original graph and eager paths as fallbacks for unsupported shapes.
  • Update xllm-ops and xllm-atb-layers

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: python setup.py build test has passed on an NPU machine.
  • MLU: python setup.py build test has passed on an MLU machine.

@maojunx99
maojunx99 force-pushed the perf/qwen3vl-decode-nofusion branch from a682c10 to 2bce490 Compare August 17, 2026 03:06
- route eligible low-token decode shapes through the non-fused kernel\n- pin xllm-ops and catlass dependencies containing the optimized tiling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant