Skip to content

CurveFFT: Use one-inversion affine GLV ladders - #164

Draft
ValarDragon wants to merge 4 commits into
mainfrom
optimize-one-inversion-curve-fft-clean
Draft

CurveFFT: Use one-inversion affine GLV ladders#164
ValarDragon wants to merge 4 commits into
mainfrom
optimize-one-inversion-curve-fft-clean

Conversation

@ValarDragon

Copy link
Copy Markdown
Contributor

Summary

This draft collects the remaining CurveFFT affine-ladder optimizations above
#157. It is intentionally split into a portable algorithm commit and an Apple
AArch64 assembly commit so they can be reviewed and benchmarked independently.

The portable change replaces the two dependent inversion batches used for an
active affine 2P + Q ladder column with a direct formula whose denominator is

(u - x)^2 (2x + u) - (v - y)^2

This leaves one batch inversion for the column. It also:

  • reuses five field-element workspaces instead of retaining the intermediate
    slope and x-coordinate vectors;
  • shares a single denominator batch across the doubling and active-addition
    work in the different-scalar ladder;
  • narrows the exceptional schedules to Q = P and Q = -2P; Q = -P is
    handled directly by the new formula;
  • retains the even/odd two-chain Montgomery batch inversion; and
  • seeds those chains from their first two values, avoiding both prefix
    multiplications by one. A one-element batch keeps its dedicated path.

The Apple-specific commit then:

  • adds a two-lane CIOS Montgomery multiplication kernel for adjacent,
    independent field products;
  • supports exact out == lhs aliasing by loading each pair before storing it;
  • canonicalizes both outputs;
  • performs the even/odd batch-inversion back-substitution in paired assembly
    lanes while keeping the carried accumulators in their permitted lazy range;
    and
  • routes the independent direct-formula stages through the paired kernel.

Non-Apple targets retain the generic portable field path.

Commit split

  1. 1dc0e71 — portable one-inversion formula, workspace reuse, seeded
    two-chain inversion, exceptional-case changes, and tests.
  2. 2b1766b — Apple AArch64 paired Montgomery multiplication and paired
    inversion back-substitution.

CurveFFT benchmark

Lower is better. Each cell is the arithmetic mean of six Criterion center
estimates. Portable -> final is the ASM step on macOS and a non-ASM control
build on Linux.

Host Mode Base Portable Final Base -> portable Portable -> final Total
macOS M4 #1 single-core 125.542 ms 124.567 ms 117.652 ms 0.78% 5.55% 6.29%
macOS M4 #1 multicore 32.359 ms 31.733 ms 31.392 ms 1.94% 1.07% 2.99%
macOS M4 #2 single-core 126.165 ms 125.307 ms 118.540 ms 0.68% 5.40% 6.04%
macOS M4 #2 multicore 32.355 ms 31.789 ms 31.255 ms 1.75% 1.68% 3.40%
Linux EPYC #1 single-core 303.667 ms 296.123 ms 295.682 ms 2.48% 0.15% control 2.63%
Linux EPYC #1 multicore 56.453 ms 55.569 ms 55.344 ms 1.57% 0.40% control 1.96%
Linux EPYC #2 single-core 320.787 ms 311.815 ms 311.685 ms 2.80% 0.04% control 2.84%
Linux EPYC #2 multicore 58.048 ms 57.357 ms 57.021 ms 1.19% 0.59% control 1.77%

The Linux portable -> final residual is not assembly: the Apple overrides are
not compiled there. Its 0.04–0.59% range is the observed control floor from
separate optimized builds and the two benchmark VMs. The replicated 5.40–5.55%
single-core step on the two M4 hosts is therefore well separated from that
floor. The smaller multicore step is expected because the per-worker field
kernel is a smaller fraction of wall time.

Method

  • Benchmark: curve-fft/affine-eisenstein-k11.
  • Single-core binaries: --no-default-features --features batch.
  • Multicore binaries: default features.
  • Criterion: 1 s warmup, 4 s measurement, 20 single-core or 50 multicore
    samples per estimate.
  • Three symmetric six-run rotations per mode:
    base/portable/final/final/portable/base and two rotated orders.
  • Immutable binaries and process/load snapshots before and after every run.
  • Both Macs reported no thermal, performance, or CPU-power warnings throughout;
    no competing Cargo, rustc, Criterion, or CurveFFT process appeared in the
    snapshots.

The benchmarked revisions were cfa5a2b, 0914b72, and 8372e98. The target
then advanced by ec3c137, which only registers and adds a separate GLV-table
microbenchmark. The two patches were mechanically rebased to the commit IDs
listed above; no library or CurveFFT benchmark code changed in that target
advance.

Correctness and validation

  • Direct-formula results are checked against native group arithmetic, including
    Q = -P and the exceptional Q = P and Q = -2P cases.
  • Batch inversion and exact-alias paired multiplication cover tiny, odd, even,
    boundary, and TWIDDLE_MAJOR_MAX_CHUNK-sized batches.
  • The crafted schedule test verifies that Q = -P now takes the optimized
    direct formula.
  • Apple AArch64 assembly is exercised by the field and GLV tests on arm64.
  • cargo test --manifest-path pasta_curves/Cargo.toml --all-features
  • cargo test --manifest-path pasta_curves/Cargo.toml --no-default-features
  • cargo check --release --manifest-path pasta_curves/Cargo.toml --all-features
  • cargo fmt --all -- --check
  • git diff --check

API surface

There is no new downstream-public API and no existing public signature or
visibility change.

Review-relevant internal additions are:

  • private glv::private::Sealed hooks for paired base-field multiplication and,
    on Apple AArch64, batch-inversion back-substitution;
  • Apple-only pub(crate) Fp/Fq runtime helpers
    mul_assign_pairs_runtime and batch_invert_backsub_runtime; and
  • pub(super) AArch64 assembly wrappers mul_pairs and
    batch_invert_backsub.

The changelog records both the portable formula/inversion change and the Apple
assembly backend change.

Base automatically changed from derive-squared-glv-rotation to main August 23, 2026 10:28
@ValarDragon ValarDragon changed the title Use one-inversion affine GLV ladders CurveFFT: Use one-inversion affine GLV ladders Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant