Skip to content

Re-measure parity and sweep docs for epic #324 #351

Description

@neuromechanist

Final measurement phase of epic #324: re-measure every user-facing parity figure under the finished code, and update the documents that quote them, including the paper.

Epic #324 changed default trajectories on every backend:

Every figure measured before these changes is now a claim about code that no longer ships.

Scope

  1. Per-backend harness rows. Re-measure the rows in docs/guides/validation.md with validate_implementations.py --backend all, along with the bundled amicaout fixture comparisons.
  2. The paper's Table 1, both tiers of benchmarks/reproduce_table1.py, same protocol as before.
    • The external tier (ds002718 sub-002, 5 seeds at 2000 iterations) runs on the CUDA workstation.
    • The bundled tier gives the Amari, score-function and multi-model rows.
    • Update paper.md's table and the prose around it, README.md, AGENTS.md and validation.md wherever they quote these figures.
  3. The Multi-model NG partition matching vs Fortran (>0.95 cross-corr) #27 multi-model distributional-equivalence study. Re-run it (N=20 ensembles per implementation) and regenerate docs/assets/figures/multimodel-ensemble.png from the new fits.
    • The ensemble log-likelihood row is the one metric on which the old ensembles differed significantly. Re-test the explanation that it is convergence speed ("reaches Fortran's mean by 200 iterations, passes it by 300"), or replace it with what the new fits show.
  4. The paper's other measured claims:
    • the Newton-enabled independent-seed statement (one seed of three with ten of seventy components in a different basin; a matched initialization restores ~0.997);
    • "all backends converge to the same single-model log-likelihood (maximum pairwise difference ~0.003)";
    • "component-level float32 agreement is not yet characterized at a matched iteration budget", which the per-backend harness rows now characterize.
  5. ADR 0003's keep_best figures (the log-likelihood spread relative to Fortran's, and how often a restore fires). Re-measure them with a seeded multi-model ensemble against the pinned binary. Phase 7 (Phase 7: normalize component rows in doscaling #340) found that the old overshoots came mostly from the column-rule doscaling bug.
  6. The share_comps merge counts quoted in docs/guides/amica-differences.md, which Phase 11's order change made stale.

Every re-measured figure is recorded with its run record (command, commit, host, raw output) under .context/issue-351/.

The documentation audit and the dead-code cleanup that were first listed here moved to #352.

Part of epic #324.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions