Skip to content

ggml-cpu: extend RVV quantization vec dot to higher VLENs#22754

Open
rehan-10xengineer wants to merge 4 commits intoggml-org:masterfrom
riseproject-dev:10x/riscv-quant-vec-dot-vlens
Open

ggml-cpu: extend RVV quantization vec dot to higher VLENs#22754
rehan-10xengineer wants to merge 4 commits intoggml-org:masterfrom
riseproject-dev:10x/riscv-quant-vec-dot-vlens

Conversation

@rehan-10xengineer
Copy link
Copy Markdown
Contributor

Summary

This PR adds RVV implementations for quantized vector dot kernels (for VLENs 512-bit and 1024-bit).

Key Changes

  • Added the following RVV kernels:
Kernel VLEN
ggml_vec_dot_q3_K_q8_K 512, 1024
ggml_vec_dot_q6_K_q8_K 512, 1024
ggml_vec_dot_iq1_s_q8_K 512, 1024
ggml_vec_dot_iq1_m_q8_K 512, 1024
ggml_vec_dot_iq2_s_q8_K 512, 1024 and above
ggml_vec_dot_iq2_xs_q8_K 512 and above
ggml_vec_dot_iq2_xxs_q8_K 512 and above
ggml_vec_dot_iq3_s_q8_K 512 and above
ggml_vec_dot_iq3_xxs_q8_K 512, 1024 and above
ggml_vec_dot_iq4_xs_q8_K 512, 1024
ggml_vec_dot_tq1_0_q8_K 512 and above
  • Each kernel that is VLEN-dependent has its own separate function call now, for example ggml_vec_dot_tq1_0_q8_K_vl512.

Testing

Kernels were functionally tested through test-quantize-fns for VLENs 512-bit and 1024-bit with QEMU, using test-quantize-fns.

@rehan-10xengineer rehan-10xengineer changed the title 10x/riscv quant vec dot vlens ggml-cpu: extend RVV quantization vec dot to higher VLENs May 6, 2026
@github-actions github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants