Skip to content

Use mixed-radix decompositions for CurveFFT - #166

Open
ValarDragon wants to merge 1 commit into
optimize-one-inversion-curve-fft-cleanfrom
optimize-mixed-radix-curve-fft-pr
Open

Use mixed-radix decompositions for CurveFFT#166
ValarDragon wants to merge 1 commit into
optimize-one-inversion-curve-fft-cleanfrom
optimize-mixed-radix-curve-fft-pr

Conversation

@ValarDragon

@ValarDragon ValarDragon commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Stacked on #164. The Apple AArch64 specialization is inherited from that
base; this PR adds no assembly changes.

What changed

  • Factor every supported CurveFFT size at k >= 6 into DFT8 and DFT16
    tiers. The Orchard schedule is 16 x 16 x 8; k = 13 uses
    16 x 8 x 8 x 8.
  • Apply the required Cooley-Tukey diagonal twiddles between tiers while
    transposing into contiguous, locally bit-reversed codelets.
  • Run each codelet substage across all codelets before advancing, retaining
    one large affine inversion batch per substage.
  • Decompose each distinct twiddle once, group equal scalar schedules, and use
    free negation for exponents in the upper half of the root-of-unity cycle.
  • Reuse the affine butterfly scratch allocation across every tier and
    alternate one N-point affine scratch vector with the output.
  • Keep the existing identity-free fast path, exact exceptional fallback,
    affine GLV ladders, and first-value accumulator seeding.
  • Retain the existing path below k = 6: k = 3 and 4 are already one
    DFT8/DFT16 codelet, while a 2 x 16 split at k = 5 saves no scalar
    multiplications and would add transpose traffic.

Operation counts

The tests instrument the executed path rather than only checking the
factorization on paper.

For a plain radix-2 Cooley-Tukey FFT with no codelets, each layer has N / 2
butterflies and one multiplication by the trivial twiddle 1 per block. The
number of nontrivial point-scalar multiplications is therefore
kN / 2 - (N - 1).

k factors, inner to outer plain radix-2 previous codelet schedule mixed mixed vs radix-2 mixed vs previous
10 8 x 8 x 16 4,097 3,905 3,649 -10.94% -6.56%
11 8 x 16 x 16 9,217 8,833 8,193 -11.11% -7.25%
12 16 x 16 x 16 20,481 19,713 18,177 -11.25% -7.79%
13 8 x 8 x 8 x 16 45,057 43,521 40,449 -10.23% -7.06%
k curve additions twiddle decompositions layout passes
10 11,136 364 4N
11 24,576 740 4N
12 53,760 1,470 4N
13 115,712 2,938 6N

The codelet tiers use 14, 16, 18, and 18 sequential affine-addition
inversion batches respectively at k = 10..13. The tests pin every batch
width. The extra k = 13 copy required by the even number of tiers is
included in its six measured layout passes.

CurveFFT benchmark

Four rotated Criterion rounds per revision, with compilation outside the
measurement window. Each row is the median of the four reported centers.
Linux compares the portable parent 1dc0e71 to algorithm commit 3d5c478;
the two Mac hosts use the same comparison with the existing field ASM enabled.

host k11 single k11 multi k13 single k13 multi
macOS 1 129.95 -> 120.44 ms (-7.32%) 31.70 -> 28.33 ms (-10.65%) 632.50 -> 580.99 ms (-8.14%) 138.98 -> 126.36 ms (-9.08%)
macOS 2 130.32 -> 120.61 ms (-7.45%) 31.68 -> 28.33 ms (-10.57%) 633.14 -> 581.79 ms (-8.11%) 138.94 -> 126.13 ms (-9.22%)
Linux 1 296.77 -> 274.04 ms (-7.66%) 55.11 -> 47.23 ms (-14.29%) 1,486.20 -> 1,354.50 ms (-8.86%) 242.77 -> 216.42 ms (-10.85%)
Linux 2 311.64 -> 286.27 ms (-8.14%) 56.88 -> 50.11 ms (-11.91%) 1,567.35 -> 1,411.10 ms (-9.97%) 259.70 -> 225.56 ms (-13.15%)

Both Macs reported no thermal or performance warning. Pre-run process checks
found no competing benchmark workloads. The two Linux hosts were idle before
the run; post-run load reflected only the benchmark itself.

Correctness

  • Matches the native projective radix-2 FFT on both Pasta curves through
    k = 13.
  • Covers regular inputs plus identities, equal/opposite points, and affine
    exceptional fallbacks through k = 12; k = 13 gets a direct comparison
    on both curves.
  • Explicitly proves every generated mixed-radix input, inter-tier, and output
    permutation is bijective for k = 6..16.
  • Passes the exact operation-count and inversion-width assertions with and
    without the multicore feature.

The final PR tree is byte-for-byte identical to tested stacked revision
110008b (tree e6c0dc4); only the commit order was changed so this PR is a
clean descendant of #164.

@ValarDragon

Copy link
Copy Markdown
Contributor Author

Follow-up CurveFFT kernel experiments (2026-08-23)

I evaluated the remaining kernel ideas on top of this PR. None of items 1-7
survived balanced benchmark-server validation, so the PR remains unchanged.

  1. Fuse fixed-scalar groups into one heterogeneous affine ladder. A
    generic pair fusion was 1-5% slower locally. Dedicated three- and
    four-group scalar-run kernels, which decoded each scalar only once, were
    still 1-3% slower. The existing tight same-scalar loops beat the saved
    divstep inversions.
  2. Two-chain the L+R/L-R batch inversion. Correct, but isolated Mac runs
    were 0.5-1% slower. Linux direction changed with host drift, and the
    combined candidate was statistically flat. Rejected.
  3. Stage the complete L+R/L-R finishing formula. Correct, including the
    Apple paired field-multiplication path, but flat to slightly slower because
    of the extra structure-of-arrays memory passes. Rejected.
  4. Retain reusable DFT8/DFT16 intermediate scratch. About a 0.5% local
    trend at k=11 and flat at k=13. In combination with item 2 it measured
    approximately 0-0.5% on the Macs and mixed within 1% on Linux. Rejected as
    noise-sized.
  5. Replace diagonal Vec grouping with a flat counting layout. Both a
    preinitialized point buffer and a flat entry/order variant were 1-2%
    slower. The current small, one-pass groups are more cache-friendly.
  6. Fuse routing with codelet I/O. Fusing the initial gather was about 2.5%
    slower at k=11; strided final writes were 3-4% slower. The explicit
    transpose is doing useful cache organization.
  7. Change factor order. The three k=11 orders and the extreme k=13
    placements were indistinguishable. The canonical order stays.

Item 8 was kept on a separate experimental branch as requested:
optimize-dft32-curvefft (unpublished as a PR).

  • The split DFT32 is correct on both Pasta curves and costs 42 point-scalar
    multiplications versus 43 for DFT16 plus radix-2.
  • At k=13, factors 32 x 16 x 16 execute 40,193 point-scalar
    multiplications, down 256 (0.63%) from this PR's 40,449, down 3,328 from
    the previous codelet schedule, and down 4,864 from plain radix-2
    Cooley-Tukey (45,057).
  • It adds 512 curve additions (116,224 versus 115,712), keeps 2,938 distinct
    twiddle decompositions, and reduces layout traffic from 6N to 4N.
  • The simple composition has 24 sequential affine-addition inversion batches
    for the full k=13 schedule versus 18 here. A fused prototype reduced that
    to 20, but did not improve server performance and was reverted.
  • Balanced single-core centers versus this PR were approximately -0.7%,
    -0.9%, -0.9%, and +0.2% on macOS-1, macOS-2, Linux-1, and Linux-2.
  • Multicore was architecture-dependent: about 1.5-1.6% faster on both Macs,
    but repeatable reverse-order runs were 2.5-3.0% slower on both Linux hosts.

Because DFT32 regresses portable multicore performance, I did not open the
separate stacked PR. The experimental branch preserves both the correct
prototype and the rejected fusion history for future work.

@ValarDragon
ValarDragon marked this pull request as ready for review August 24, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant