Skip to content

Avoid repacking compatible problems outside autodiff - #585

Merged
ChrisRackauckas merged 3 commits into
SciML:masterfrom
ChrisRackauckas:perf/primal-problem-adaptation
Nov 5, 2026
Merged

ChrisRackauckas merged 3 commits into
SciML:masterfrom
ChrisRackauckas:perf/primal-problem-adaptation

Conversation

@ChrisRackauckas

@ChrisRackauckas ChrisRackauckas commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Filed by an AI agent running as @ChrisRackauckas; Chris has not seen this.
Harness: Codex CLI 0.164.0 · Model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

Ignore this draft until reviewed by @ChrisRackauckas. Depends on #586 (which depends on #584). Restacked head: 492cbe6877cc18e1a8dec7f884ce353346a7e18c. Measurements below isolate the primal optimization against keyword-fix parent 692fe2898d1d56098d303541f868f200a9f411b8.

Primal ensemble kernels consume the adapted compatible problem directly. Enzyme differentiation retains the numeric-first problem record and its noinline construction boundary. Adaptation still runs exactly once per trajectory, and returned solutions retain each original host problem. The change extends the existing CPU behavior through the core adapter; it adds no public API, return-type or output-layout change.

Measurements

GPU 1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, sm_120; AMD Ryzen 9 9950X3D; four Julia threads; Julia 1.13.2; CUDA/CUDACore 6.4.1. Same Lorenz setup as the benchmark: tspan=(0,1), dt=0.001, save_everystep=false, dense=false, CPU offload=0, adaptive abstol=reltol=1e-8. The confirmation uses four fresh processes in parent/change/change/parent order. Each table cell contains four repetitions (two per process), each the minimum of five synchronized samples after warmup. Device-only calls use a pre-uploaded problem array and include device output allocation/initialization.

Precision Adaptive N High-level before (ms) High-level after (ms) Device-only before (ms) Device-only after (ms)
Float32 false 8192 0.764 / 0.861 / 0.910 / 0.871 0.821 / 0.686 / 0.781 / 0.377 0.090 / 0.087 / 0.088 / 0.087 0.088 / 0.090 / 0.088 / 0.087
Float32 false 131072 11.248 / 5.999 / 12.754 / 6.287 10.977 / 5.214 / 5.223 / 5.280 0.464 / 0.439 / 0.444 / 0.442 0.439 / 0.440 / 0.440 / 0.437
Float32 false 2097152 242.086 / 232.284 / 229.689 / 233.109 232.404 / 224.608 / 208.763 / 217.191 6.499 / 6.533 / 6.530 / 6.532 6.525 / 6.519 / 6.603 / 6.510
Float32 true 8192 1.043 / 0.942 / 0.773 / 0.886 0.828 / 0.832 / 0.611 / 0.856 0.168 / 0.164 / 0.167 / 0.165 0.173 / 0.165 / 0.166 / 0.165
Float32 true 131072 10.660 / 6.572 / 11.462 / 5.848 9.939 / 5.253 / 5.558 / 11.913 0.320 / 0.316 / 0.315 / 0.315 0.316 / 0.318 / 0.316 / 0.337
Float32 true 2097152 244.764 / 244.267 / 217.688 / 235.618 229.324 / 205.372 / 212.994 / 224.761 2.692 / 2.727 / 2.793 / 2.713 2.694 / 2.686 / 2.712 / 2.689
Float64 false 8192 2.448 / 3.041 / 2.949 / 2.782 2.645 / 2.751 / 2.939 / 2.948 1.978 / 1.980 / 1.980 / 1.976 1.974 / 1.981 / 2.000 / 1.978
Float64 false 131072 37.962 / 28.573 / 35.980 / 29.100 35.205 / 27.390 / 32.240 / 27.874 21.289 / 21.280 / 21.261 / 21.291 21.256 / 21.253 / 21.387 / 21.366
Float64 false 2097152 644.387 / 639.124 / 661.298 / 632.578 639.045 / 614.002 / 603.268 / 624.109 347.833 / 347.387 / 346.891 / 346.708 347.283 / 347.066 / 347.106 / 346.846
Float64 true 8192 1.222 / 1.694 / 1.265 / 1.729 1.235 / 1.763 / 1.185 / 1.609 0.747 / 0.744 / 0.743 / 0.765 0.743 / 0.743 / 0.745 / 0.743
Float64 true 131072 23.475 / 14.207 / 22.651 / 13.566 22.987 / 13.044 / 18.267 / 13.036 6.839 / 6.827 / 6.811 / 6.830 6.819 / 6.821 / 6.819 / 6.822
Float64 true 2097152 366.781 / 391.631 / 356.279 / 387.488 333.782 / 364.199 / 337.428 / 377.371 71.285 / 71.294 / 71.134 / 71.142 71.143 / 71.221 / 71.180 / 71.185

The preparation-only measurement uses the same problem construction at each size. Each cell below gives the median of ten samples in each of two fresh processes; a full garbage collection runs before each sample, outside the timed region. Allocated bytes are identical in every corresponding sample.

Precision N Preparation before (ms) Preparation after (ms) Allocated bytes before = after
Float32 8192 0.118 / 0.116 0.066 / 0.071 458896
Float32 131072 1.550 / 1.535 0.779 / 0.780 7340176
Float32 2097152 44.384 / 43.811 32.964 / 31.465 117448688
Float64 8192 0.140 / 0.138 0.082 / 0.086 917648
Float64 131072 4.592 / 4.495 3.617 / 3.718 14680208
Float64 2097152 76.441 / 74.274 58.057 / 59.195 234889200

At 2^21, preparation medians decrease by 26.9% for Float32 and 22.2% for Float64. High-level median reductions in the balanced confirmation range from 3.5% to 8.8%, while the four-repeat high-level spreads range from 4.5% to 12.4% before and 5.9% to 13.1% after. Device-only spreads are 0.23–3.74% before and 0.11–1.44% after. These observations support faster preparation; they do not establish a universal end-to-end speedup or a device-kernel gain.

The initial validation pair also completed all endpoint checks but produced mixed total-solve timings, including slower Float32 results at 2^21. That variation prompted the balanced confirmation; the initial numbers remain disclosed below. Phase timings include allocation/GC and are not additive estimates of a minimum full solve.

Initial validation matrix (two repeats per checkout)
Precision Adaptive N High-level before (ms) High-level after (ms) Device-only before (ms) Device-only after (ms)
Float32 false 8192 0.803/0.526 0.790/0.770 0.089/0.089 0.090/0.090
Float32 false 131072 10.830/12.471 10.311/11.634 0.439/0.437 0.437/0.435
Float32 false 2097152 228.254/238.411 248.423/247.392 6.589/6.524 6.576/6.510
Float32 true 8192 0.622/0.649 0.628/0.712 0.171/0.165 0.167/0.168
Float32 true 131072 11.059/6.059 9.828/5.588 0.322/0.316 0.318/0.315
Float32 true 2097152 220.023/231.829 240.802/245.774 2.702/2.702 2.690/2.702
Float64 false 8192 3.050/2.472 2.668/2.388 1.974/2.000 1.995/1.978
Float64 false 131072 33.444/28.234 34.024/27.598 21.299/21.300 21.283/21.280
Float64 false 2097152 644.606/741.183 619.413/740.357 347.000/346.768 347.123/346.910
Float64 true 8192 1.397/1.754 1.218/1.639 0.742/0.745 0.749/0.744
Float64 true 131072 20.469/14.903 19.812/12.237 6.818/6.856 6.841/6.827
Float64 true 2097152 360.025/408.413 403.792/405.117 71.301/71.187 71.272/71.197

The parent and candidate matrix environments use CUDA 6.4.1 and match apart from checkout paths. For context only, the supplied V100 rendering reports JAX 5541 ms fixed / 1145 ms adaptive at 2^21 and PyTorch 964 ms fixed at 2^19. The different GPU and host prevent a controlled cross-library speedup claim.

Validation

Existing adaptation fixtures are shared across CPU, CUDA and JLArrays. Tests cover stateful adaptation counts, precision-changing adaptation, captured RHS values, original host parameter identity, and the primal compatible-problem representation. Existing assertions and tolerances are preserved. Both precision arms and first/middle/last high-level versus device-resident endpoints/times match exactly in the timing runs.

The identical shared adaptation test file gives 37 passes / 8 failures on keyword-fix parent 692fe2898d1d56098d303541f868f200a9f411b8, then 45/45 passes on this change. The eight failures are the two primal-representation assertions in both precisions and both safety-copy modes; all numerical/host-retention assertions remain active.

Check Observed result
Runic, typos, git diff --check Pass on final edited files
GROUP=QA, Julia 1.13.2 Quality Assurance 27/27; JET 75/75 in the final rerun
Full GROUP=CPU, Julia 1.13.2 Pass
Full GROUP=CPU, Julia 1.10.12 DAE 82 passed / 2 failed, matching the clean-master initialization defect; later groups not reached
Full GROUP=JLArrays, Julia 1.13.2 Pass, including all 45 shared adaptation assertions
Full GROUP=Enzyme, Julia 1.13.2 and 1.10.12 Both groups pass all 58 assertions: 8 imports, 40 CPU/JLArrays transfer/adaptation checks and 10 ensemble-gradient checks
Full GROUP=CUDA, Julia 1.13.2 838 non-Enzyme assertions, 8 imports and 6 active-field assertions pass, then the known allocation-record error; group exits 1
Focused CUDA Enzyme checks 35 passes / 3 adaptive-gradient errors, matching the unchanged clean-master signatures

The LTS initialization defect is tracked in SciMLBase. The full CUDA group fails at zero(::Type{EnzymeAllocationRecord}), matching the unchanged clean-master run and the known CUDACore issue. The 838 successful non-Enzyme CUDA assertions comprise the same 793 baseline assertions plus the 45 shared adaptation assertions. The focused Enzyme files execute separately because the full suite stops before reaching them. No tests or tolerances are skipped, disabled or weakened. CUDA.functional and a 256-element add-one GPU kernel pass before GPU validation.

The adaptive CUDA errors contain cannot handle unknown binary operator at the bit mask in FastPower.fastlog2, consistent with the parent PR and the existing Enzyme deferred-codegen rule issue. No adaptive CUDA Enzyme success is claimed. Fixed Float32/Float64, fixed Lorenz, and initial-state/time/step-size gradient tests pass.

The parent keyword fix supplies a dedicated Enzyme regression that checks gradients through the initial state, both time endpoints, parameters, a captured RHS value, and a numeric keyword, with two accumulation calls in each precision. It also checks that differentiation retains the internal numeric-first record. No custom differentiation rule is added.

Commands use REPO for the checkout and TEST_ENV for an isolated copy of its declared test environment developed against that checkout. The CUDA suite retains its two distributed workers and Enzyme child.

julia --project=@runic -m Runic --check src/solve.jl test/runtests.jl test/gpu_kernel_de/host_fastpath.jl test/gpu_kernel_de/problem_adaptation.jl test/enzyme/problem_adaptation.jl test/enzyme_environment.jl
GROUP=QA julia +release --startup-file=no --compiled-modules=existing --project="$REPO" -e 'using Pkg; Pkg.test(;julia_args=["--check-bounds=auto","--compiled-modules=existing"])'
GROUP=CPU julia +release --startup-file=no --compiled-modules=existing --project="$TEST_ENV" -e 'cd(joinpath(ARGS[1],"test")); include("runtests.jl")' "$REPO"
GROUP=CPU julia +lts --startup-file=no --compiled-modules=no --project="$TEST_ENV" -e 'cd(joinpath(ARGS[1],"test")); include("runtests.jl")' "$REPO"
GROUP=JLArrays julia +release --startup-file=no --compiled-modules=existing --project="$TEST_ENV" -e 'cd(joinpath(ARGS[1],"test")); include("runtests.jl")' "$REPO"
GROUP=Enzyme julia +release --startup-file=no --compiled-modules=existing --project="$TEST_ENV" -e 'cd(joinpath(ARGS[1],"test")); include("runtests.jl")' "$REPO"
GROUP=Enzyme julia +lts --startup-file=no --compiled-modules=no --pkgimages=no --project="$TEST_ENV" -e 'cd(joinpath(ARGS[1],"test")); include("runtests.jl")' "$REPO"
GROUP=CUDA julia +release --startup-file=no --compiled-modules=existing --project="$TEST_ENV" -e 'cd(joinpath(ARGS[1],"test")); include("runtests.jl")' "$REPO"
GROUP=CUDA julia +release --startup-file=no --compiled-modules=existing --project=test/enzyme -e 'using DiffEqGPU; include("test/enzyme/imports.jl"); include("test/enzyme/transfers.jl")'
GROUP=CUDA julia +release --startup-file=no --compiled-modules=existing --project=test/enzyme -e 'using DiffEqGPU,Test; @testset "CUDA Enzyme all gradient testsets" begin include("test/enzyme/imports.jl"); include("test/enzyme/gradients.jl"); end'

No docstring, docs or public API change is included; a local docs build is not required. Other physical GPU backends are not available for local execution and no result is claimed for them.

Published-commit CI

Exact-head CI is complete. The test matrix passes both Enzyme versions (58 each), both release CPU architectures (979 each), JLArrays release (613, including the explicit 45 adaptation checks), OpenCL release (559), and QA/JET (27/75). Backend CI passes Metal and oneAPI release (560 each). Documentation, Runic and spelling pass. Green non-RC prerelease jobs skip execution and are not runtime coverage.

CUDA CI passes all 838 non-Enzyme assertions, including the explicit 45 adaptation checks, then 8 Enzyme imports and all 6 new active-field checks before the known zero(::Type{EnzymeAllocationRecord}) error. Later transfer/gradient files remain unexecuted in that full CI job; their local continuation is listed above.

Terminal failures match the inspected baselines: LTS initialization assertions (82 passes / 2 failures), AMDGPU LTS initialization (80 passes / 2 failures / 1 InvalidIRError), and AMDGPU release's missing lufact! method (36 passes / 2 errors). The minimum-dependency job reports the same GPUVern9 saved-state norm 0.0010757452f0 versus 0.001 as version-only source baseline 582. A current clean-master control passes all 50 assertions with the same 295 printed package versions, Julia 1.10.13 and runner image. CPU models are unknown; no deterministic cause or tolerance change is claimed.

The auxiliary Dependabot workflow fails before creating jobs because the reusable workflow requests permissions its caller does not grant. The caller is byte-identical on master and both stacked PR heads, and the parent run has the same annotation. This is separate from solver execution and is already tracked in SciML/.github issue 141.

Verification scope

The keyword-gradient fix is supplied by the parent PR. Its dedicated test file runs immediately after Enzyme imports, before the unchanged CUDA allocation-record test. This branch shares the primal representation fixtures across CPU, JLArrays and CUDA without duplicating the parent's active-field test. Published-commit CI has completed with the results above. A second fresh independent review accepts the exact head as MERGE-WITH-ISSUES and considers G2 satisfied for the changed paths. Results from the earlier head are not presented as validation of this commit.

Risk assessment

  • Risk: medium-low (ML3: measured host-preparation optimization with unchanged numerical paths and public results).
  • Blast radius: primal EnsembleGPUKernel problem adaptation across backends; differentiated paths retain record construction. Original host problem storage and output assembly are unchanged.
  • Evidence: focused CPU/CUDA adaptation checks, repeated fixed/adaptive Float32/Float64 timing matrix, exact sampled endpoint/time equality; full validation results listed above.
  • Independent review: Codex CLI 0.164.0 / gpt-6.1-sol independently rates medium-low (ML3; parent also ML1), high confidence in classification and scope. MERGE-WITH-ISSUES, no actionable source defects. Completed exact-head CUDA/backend and Enzyme CI resolves the prior G2 blocker for the changed paths. Final review summary.
  • Merge: needs human review; this remains a draft for Chris with an accepted independent review and disclosed baseline failures. No approval or merge has been performed.

🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-benchgap (no session URL)
@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Fresh independent review of 119bb9d7e155aa5a33897e60db57c9ef2344b09c by Codex CLI 0.164.0 / gpt-6.1-sol, session 01a19748-84e6-7d73-9646-15889a7102f8 (no session URL): CHANGES. The reviewer found no source defect in the primal optimization and rated it medium-low (ML3), medium confidence, before reading the author's rating.

The review identified the new Float32 keyword-gradient failure on Julia 1.10 and incomplete CUDA verification. Subsequent author checks establish the following:

  • The identical active-field test gives 44 passes / 4 failures on both clean master 9cef7c8 and this head. A focused private-record normalization fix gives 48/48 and passes the full Enzyme groups on Julia 1.10 and 1.13. That fix is still undergoing GPU validation and is not part of this PR. The regression assertions remain intact.
  • Completed CUDA CI on this head passes all 838 non-Enzyme assertions, including all 45 new adaptation assertions, and all 8 Enzyme import checks. It then hits the same zero(::Type{EnzymeAllocationRecord}) error observed on clean master. Later Enzyme tests are not reached.

The reviewer did not run GPU assertions: its sandbox could not access the NVIDIA driver. The later CI classification above is the author's follow-up evidence, not a new accepted review verdict. The PR remains CHANGES until the keyword-gradient finding is addressed and the final head is reviewed again.

Risk assessment

  • Risk: medium-low (ML3: measured host preparation improvement with public solver outputs preserved); author and reviewer agree.
  • Blast radius: primal GPU problem preparation across backends; AD retains private records. Public output layouts, solver algorithms and controllers remain intact.
  • Evidence: before/after timing matrices, representation fail-before/pass-after, successful release backend checks and matched CUDA baseline error. The newly exposed LTS gradient failure still needs its separately validated fix.
  • Independent review: Codex CLI 0.164.0 / gpt-6.1-sol, medium-low ML3, medium confidence, CHANGES.
  • Merge: not eligible with an unresolved review finding; remains a draft for Chris.

🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

ChrisRackauckas and others added 2 commits November 4, 2026 06:05
Normalize private record keywords to a NamedTuple while preserving the
noinline constructor and numeric-field order. Cover every active problem
field and accumulated gradients on CPU, JLArrays and CUDA.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-benchgap (no session URL)
Keep the adapted immutable problem on the primal path for every backend.
Preserve the private noinline record conversion during differentiation and
exercise host retention, adaptation and representation across backends.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-benchgap (no session URL)
@ChrisRackauckas
ChrisRackauckas force-pushed the perf/primal-problem-adaptation branch from 119bb9d to 492cbe6 Compare November 4, 2026 08:06
@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Fresh independent review of restacked head 492cbe6877cc18e1a8dec7f884ce353346a7e18c: CHANGES — verification required, with no actionable source defect. Reviewer: Codex CLI 0.164.0 / gpt-6.1-sol, local session 01a1a599-3bac-7f83-aabb-62b14656874b (no session URL).

The parent keyword fix addresses the earlier LTS gradient finding. The reviewer inspected the failing-before/passing-after controls, full local validation and all six timing matrices. It independently recomputed 26.9%/22.2% lower preparation medians at 2^21, unchanged allocations and the larger variation in total solve times. No universal end-to-end gain is claimed.

The remaining finding is G2: at the review snapshot, exact-head test CI, CUDA CI and backend CI were incomplete. The reviewer personally ran read-only Git/Python/live-CI checks; it did not execute Julia or GPU tests. Author logs and green prerelease jobs that skip execution do not substitute for this gate.

Independent risk, recorded before reading the author rating: medium-low (ML3; parent also ML1), high confidence in scope and moderate confidence in cross-backend correctness. The author agrees. This remains a draft for Chris; a fresh re-review will assess completed exact-head CI and any failures.


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Fresh independent re-review of unchanged restacked head 492cbe6877cc18e1a8dec7f884ce353346a7e18c: MERGE-WITH-ISSUES, with no actionable source defects. Reviewer: Codex CLI 0.164.0 / gpt-6.1-sol, local session 01a1a5c3-380c-7f40-bf12-aae7c4c94d51 (no session URL).

Independent risk, recorded before reading the author's body or previous reviews: medium-low (ML3; stacked keyword fix also ML1), high confidence. The author agrees. The reviewer inspected the complete stack and focused parent diff, retaining exactly-once adaptation, original host problems, stateful/precision-changing adaptation and the noinline record during differentiation. Public outputs and numerical controllers remain unchanged.

The previous G2 blocker is resolved by completed exact-head execution:

  • Test matrix: CPU release x64/x86 each 979 passes; JLArrays 613 including 45 adaptation checks; OpenCL 559; both Enzyme versions 58; QA/JET 27/75.
  • CUDA: 838 non-Enzyme assertions, including all 45 adaptation checks, then 8 imports and all 6 active-field checks pass before the known allocation-record error.
  • Backend CI: Metal and oneAPI release each pass 560 assertions. Documentation and formatting pass.

CI is not wholly passing. LTS initialization failures, AMDGPU failures, the allocation-record error and the minimum-dependency GPUVern9 failure match inspected controls. The auxiliary workflow startup failure is separately tracked in the existing owner issue. Prerelease skips are not runtime coverage, and CUDA transfer/gradient files after the allocation error remain unexecuted in that full CI job. The local continuation's 35 passes / 3 known adaptive errors is disclosed separately.

The reviewer personally performed read-only source/Git, CSV/TOML/hash checks and live CI/log inspection; it launched no Julia/GPU processes. It regenerated both balanced tables byte-for-byte: preparation medians improve 26.9%/22.2% at 2^21 with unchanged allocations. Total-solve gains overlap some repeat spreads, and the initial mixed timing matrix remains disclosed.

This remains a draft for Chris, subject to normal review gates. No tests or tolerances were weakened, and no approval or merge has been performed.


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Second independent review: MERGE-WITH-ISSUES at exact head 492cbe6877cc18e1a8dec7f884ce353346a7e18c.

Together with the first counted review (#585 (comment)), this gives the two counted medium-low ratings from different sessions. The PR stays a draft for @ChrisRackauckas; nothing has been approved or merged.


🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5
Conversation: https://claude.ai/code/session_0193DoGeYoy9sJDG2UUDBG7h

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review November 5, 2026 03:34
@ChrisRackauckas
ChrisRackauckas merged commit fa07cd4 into SciML:master Nov 5, 2026
21 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant