Repository navigation
Benchmark AMICATorchNG runtime vs Fortran binary #59
Description
Activity
- added a commit that references this issue
on Jul 7, 2026 - linked a pull request that will close this issuebench: NG vs Fortran runtime harness (#59) #62
on Jul 7, 2026 Benchmark results
Harness added in #62 (
benchmarks/benchmark_runtime.py). Real sample EEG (32 ch x 30504 frames), matched config (num_mix=3, do_newton, lrate=0.05, pdftype=0), 3 repeats. Final LL agrees between backends at every cell, confirming equivalent work.Host: Apple M4 Pro (arm64), torch 2.12.1, NG device
cpu.
Caveat:amica15macis x86_64, so Fortran runs under Rosetta 2 emulation here. Native x86_64/Linux Fortran would be faster, i.e. these ratios are a lower bound on the gap.threads iters NG (s) Fortran (s) NG/Fortran NG ms/it Fortran ms/it NG LL Fort LL 1 50 10.967 3.426 3.20x 219.3 68.5 -3.4486 -3.4412 1 100 21.944 6.836 3.21x 219.4 68.4 -3.4168 -3.4107 8 50 10.135 1.251 8.10x 202.7 25.0 -3.4486 -3.4412 8 100 20.349 2.654 7.67x 203.5 26.5 -3.4168 -3.4105 Marginal per-iteration (slope, fixed overhead removed): threads=1 -> 3.22x; threads=8 -> 7.28x.
Verdict on the "within 2-3x" criterion
Missed on the CPU float64 parity path. ~3.2x single-thread (just over the bar, with Fortran emulated) and worse multi-threaded, for one dominant reason: NG barely parallelizes on CPU (1->8 threads moves per-iter only 219->204 ms, ~7%) while Fortran OpenMP scales ~2.4x (68->28 ms).
Profile (cProfile, 20-iter fit, single thread)
Top internal-time functions:
Tensor.pow-- 2.545s / 5.15s (~49%) <- generalized-Gaussian|z|^rho_get_block_updates(core.py:717) 0.463 self / 5.078 cumtorch.exp0.321,torch.logsumexp0.270,_log_pdf_and_deriv(core.py:94) 0.239 self / 1.965 cum,torch.softmax0.236,torch.log0.197
The
|z|^rhofractional pow (evaluated separately for the density and its derivative) is the single biggest lever.Follow-ups
- Optimization work filed as Optimize AMICATorchNG per-iteration cost (pow hot spot + CPU threading) #63 (reuse
|z|^rho, special-case rho in {1,2}, CPU-threading / block vectorization, and quantify the CUDA path -- NG's intended fast path, since MPS cannot do float64). - Recommendation: make the runtime criterion device-aware in the docs -- NG targets GPU; on CPU it trades speed for float64 Fortran parity. Re-run this harness on a CUDA host to characterize the fast path before v0.1.0 sign-off.
- added a commit that references this issue
on Jul 7, 2026
The
.context/plan.mdsuccess criterion "runtime within 2-3x of Fortran" has never been measured. Establish a repeatable benchmark.Goal
Time
AMICATorchNGvs the Fortran binary (pyAMICA/sample_data/amica15mac) on the real sample EEG and record the numbers.Tasks
max_iter, warm/cold separated, report total + per-iteration wall-clock.amica15macon the same data/iteration budget.Notes
Real sample data only (NO-MOCK). Fortran is OpenMP-parallel; document thread counts used on both sides for a fair comparison.