Skip to content

Specialize kernel problem preparation on RNG callbacks - #584

Merged
ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas:perf/compatible-problem-construction
Nov 5, 2026
Merged

ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas:perf/compatible-problem-construction

Conversation

@ChrisRackauckas

@ChrisRackauckas ChrisRackauckas commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Filed by an AI agent running as @ChrisRackauckas; Chris has not seen this.
Harness: Codex CLI 0.164.0 · Model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

Ignore this draft until reviewed by @ChrisRackauckas.

The host preparation helpers forward an RNG callback for every trajectory. Specializing those two helpers on the callback type removes temporary allocation and dynamic dispatch during EnsembleGPUKernel problem preparation. At 2^21 Float32 Lorenz trajectories, the high-level solve drops from about 475 ms to 210–220 ms; construction drops from 337–360 ms to about 42 ms. Callback invocation, Enzyme's noinline construction boundary, numerical methods, public return types, and output layouts are preserved.

Measurements

GPU 1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (sm_120); AMD Ryzen 9 9950X3D; four Julia threads; Julia 1.13.2; CUDA.jl 6.2.2. Source baseline: 9cef7c8360782dd472d46bcf03bd99ac7797f65e. The Lorenz setup matches lorenz_ensemble.jmd: tspan=(0,1), dt=0.001, save_everystep=false, dense=false, CPU offload=0, adaptive abstol=reltol=1e-8. The Float64 arm uses the same explicitly typed setup.

Milliseconds, repeat 1 / repeat 2. Each repeat is the minimum of five synchronized samples after a warmup. Device timings use device-resident problems through vectorized_solve/vectorized_asolve, including output allocation and initialization but excluding transfers and host materialization.

Precision Adaptive N High before High after Device before Device after
Float32 false 8192 1.453 / 0.995 0.665 / 0.461 0.102 / 0.100 0.100 / 0.099
Float32 false 32768 4.856 / 4.805 1.388 / 1.594 0.168 / 0.168 0.167 / 0.195
Float32 false 131072 22.788 / 20.550 10.894 / 8.216 0.474 / 0.466 0.465 / 0.462
Float32 false 524288 99.625 / 104.852 46.830 / 44.778 1.866 / 1.913 1.874 / 1.866
Float32 false 2097152 476.995 / 475.035 209.887 / 217.346 6.905 / 6.806 6.883 / 6.762
Float32 true 8192 1.162 / 1.432 0.638 / 0.592 0.184 / 0.187 0.185 / 0.183
Float32 true 32768 3.976 / 3.683 1.423 / 1.816 0.206 / 0.206 0.206 / 0.205
Float32 true 131072 21.058 / 19.229 10.264 / 5.867 0.324 / 0.325 0.320 / 0.328
Float32 true 524288 97.150 / 110.136 40.984 / 39.377 0.898 / 0.875 0.873 / 0.880
Float32 true 2097152 470.191 / 490.632 211.758 / 219.608 3.075 / 3.013 3.073 / 3.013
Float64 false 8192 4.800 / 5.745 4.344 / 4.760 3.747 / 3.780 3.737 / 3.757
Float64 false 32768 10.644 / 10.827 8.101 / 10.081 6.543 / 6.489 6.491 / 6.497
Float64 false 131072 46.709 / 46.363 36.928 / 32.040 23.746 / 23.753 23.812 / 23.776
Float64 false 524288 278.417 / 275.923 161.340 / 164.713 103.529 / 103.447 103.526 / 103.430
Float64 false 2097152 1048.785 / 1035.768 651.293 / 687.368 387.537 / 387.133 387.614 / 387.203
Float64 true 8192 2.260 / 2.234 1.707 / 1.623 1.207 / 1.206 1.204 / 1.203
Float64 true 32768 6.023 / 6.125 3.487 / 5.344 1.922 / 1.917 1.923 / 1.919
Float64 true 131072 30.706 / 30.679 19.366 / 20.111 7.409 / 7.412 7.400 / 7.388
Float64 true 524288 199.699 / 161.935 81.151 / 91.947 21.476 / 21.445 21.515 / 21.488
Float64 true 2097152 675.445 / 658.093 426.379 / 430.819 77.231 / 77.056 77.219 / 77.040

At N=2^21, repeat-to-repeat minimum spread was 0.4–4.4% before and 1.0–5.5% after for high-level solves; device-only spread was at most 2.1%. Smaller host timings are noisier (up to 75% between minima), so the large-N results are the strongest performance evidence. Dependency resolution for the test environment overlapped part of the Float64 after-run; observed system load stayed below 10. Solver dependency versions are identical between runs; the development manifest normalizes standard-library entries to the running Julia version.

For each precision/mode/N, sampled trajectory initial/final states and times from the high-level and unchanged device-only paths matched exactly, and sampled return codes succeeded. This is sampled equality, not an exhaustive comparison of all trajectories.

The V100 values in the rendered benchmark come from different hardware. At N=2^21, those rendered JAX values are 5541 ms fixed and 1145 ms adaptive. They are context, not a controlled cross-library speedup measurement against this GPU 1 run.

Validation

The new CPU allocation regression uses the retained-buffer size as its reference, allowing allocator overhead while detecting per-trajectory temporaries.

Before the source change:

Kernel preparation temporary allocations: Test Failed
Expression: bytes < 4storage
Evaluated: 1253632 < 917888

The final fixture constructs problems inside a ::Type{T} function barrier, so its callback has a statically known precision on both supported Julia versions. With this identical fixture, clean master fails on Julia 1.13.2 (1253632 < 917888) and Julia 1.10.12 (1253584 < 917888). After the two production signature changes, the assertion passes for Float32 and Float64 on both versions. All 35 existing assertions in that host-storage file also pass. The allocation limit is unchanged.

Validation uses isolated copies of the package's declared test environment developed against the checkout. GPU checks use physical GPU 1, NVIDIA RTX PRO 6000 Blackwell Max-Q. No tests or tolerances are skipped, disabled, or loosened.

Check Observed result
Runic, typos, git diff --check Pass on final edited files
GROUP=QA, Julia 1.13.2 Quality Assurance 27/27; JET 75/75
Full GROUP=CPU, Julia 1.13.2 Pass
Host-storage file, Julia 1.10.12 and 1.13.2 All 35 existing assertions and both new allocation assertions pass
Full GROUP=CPU, Julia 1.10.12 DAE 82/84; identical clean-master failures tracked in SciMLBase; later groups not reached
Standalone GROUP=Enzyme, Julia 1.13.2 All 46 assertions pass: 8 imports, 28 CPU/JLArrays transfers and 10 ensemble-gradient checks
Full GROUP=CUDA, Julia 1.13.2 All pre-Enzyme groups pass (793 assertions), including Distributed Multi-GPU and callbacks; Enzyme imports pass 8/8, then known upstream allocation error
Clean-master full CUDA comparison Same pass counts and the identical nonnumeric Enzyme allocation error on unmodified 9cef7c8360782dd472d46bcf03bd99ac7797f65e

The CUDA Enzyme error is MethodError: no method matching zero(::Type{EnzymeAllocationRecord}) in the unchanged cuda_records.jl test, matching the known CUDACore issue. The resolved test environment uses CUDA/CUDACore 6.4.1 and Enzyme 0.13.216. The full suite stops before transfer/gradient assertions; the focused runs below execute those unchanged files separately. The separate CUDA.functional and 256-element add-one kernel probes pass. No package docstring, public API or docs change is included, so no docs build was run.

Group commands use REPO for the checkout and TEST_ENV for its isolated test environment:

GROUP=QA julia --project="$REPO" -e 'using Pkg; Pkg.test(; julia_args=["--check-bounds=auto"])'
GROUP=CPU julia +release --startup-file=no --project="$TEST_ENV" "$REPO/test/runtests.jl"
GROUP=CPU julia +lts --startup-file=no --project="$TEST_ENV" "$REPO/test/runtests.jl"
GROUP=Enzyme julia --startup-file=no --compiled-modules=existing --project="$TEST_ENV" "$REPO/test/runtests.jl"
GROUP=CUDA julia --startup-file=no --compiled-modules=existing --project="$TEST_ENV" -e 'cd(joinpath(ARGS[1], "test")); include("runtests.jl")' "$REPO"

Julia uses four threads and one precompile task. The full CUDA group retains its two distributed workers and launches the Enzyme child unchanged. The final focused LTS regression runs use --compiled-modules=no to avoid nested cold-cache compiler processes.

Focused CUDA Enzyme verification

The independent reviewer requested execution of the files after the allocation failure. Both this head and clean master pass all 14 CUDA transfer assertions plus 8 extension-import assertions. The unchanged gradient file, included inside an outer testset so later testsets also execute, gives the same 15 passed / 3 errors on both revisions (the pass count includes 8 import and 7 gradient assertions):

CUDA gradient coverage PR head Clean master
Fixed Float32 and Float64 exponential ensembles Pass Pass
Fixed Lorenz parameter gradients Pass Pass
Initial time, final time, step size, initial state Pass Pass
Adaptive Float32, Float64 and Lorenz KernelException Same KernelException

The adaptive errors contain Enzyme's cannot handle unknown binary operator at a bit mask in FastPower.fastlog2, reached through _controller_pow. This is distinct from the record-allocation failure in the full CUDA suite. No adaptive CUDA Enzyme success is claimed. A standalone reproducer importing only CUDA, Enzyme, KernelAbstractions and standard libraries passes its CPU custom-rule control (4/4), then reproduces the bit-mask exception in both direct deferred CUDA and KA kernels (2 primal checks pass, 1 differentiation error each). This is consistent with the existing Enzyme deferred-codegen custom-rule issue. Enzyme main and a runtime version bisect were not tested. This PR does not change FastPower or the controller.

Focused commands, run in each checkout's root with the same resolved dependencies, are shown below. Local launchers additionally assert the loaded checkout and run the CUDA functional/small-kernel probe before these includes.

GROUP=CUDA julia --startup-file=no --compiled-modules=existing --project=test/enzyme -e 'using DiffEqGPU; include("test/enzyme/imports.jl"); include("test/enzyme/transfers.jl")'
GROUP=CUDA julia --startup-file=no --compiled-modules=existing --project=test/enzyme -e 'using DiffEqGPU, Test; @testset "CUDA Enzyme all gradient testsets" begin include("test/enzyme/imports.jl"); include("test/enzyme/gradients.jl"); end'

Current-head release CPU, Enzyme, JLArrays, OpenCL, Metal, oneAPI, QA, formatting, spelling and documentation checks pass. CPU, JLArrays, OpenCL, Metal and oneAPI on Julia 1.10 report the same two initialization assertions described above, with 82 passed / 2 failed. Completed CUDA CI fails at the same zero(::Type{EnzymeAllocationRecord}) error as the local clean-master control; later Enzyme checks are not reached.

AMDGPU release CI reports 36 passed / 2 errored in implicit array solves, with missing lufact!(::ROCBackend, ::ROCArray{Float32,3,...}) for Rosenbrock23 and Rodas5P. The clean-base release job at 9cef7c8 has the identical failures. Both use AMDGPU 2.8.0, GPUCompiler 2.6.0 and SciMLBase 3.58.0; Julia patch and ModelingToolkitBase versions differ. This is an observed base failure, not a fully matched local AMDGPU rerun. AMDGPU LTS CI completes with 80 passes, the two known initialization assertion failures, and an additional InvalidIRError involving ijl_field_index / ijl_has_no_field_error through NamedTuple/Pairs indexing in initialization. The version-only source baseline at 1123971edcb05bcbe593498f67e649f0b1a727cd reproduces the same 80/2/1 result and both unsupported calls without this PR. Its source/tests match base 9cef7c8; only the package version changes. Julia 1.10.12, ModelingToolkitBase 1.79.0, NonlinearSolveBase 2.55.1 and the relevant GPU/compiler versions match; IRTools differs (0.4.24 versus 0.4.25). This establishes the signature as pre-existing relative to this PR, without claiming a byte-identical environment or upstream cause. Detailed baseline comparison. No local AMDGPU execution was performed. These results come from inspected job logs, not inferred badges.

Risk assessment

  • Risk: medium-low (ML3: measured performance change in DiffEqGPU's host preparation, with numerical paths and public results preserved).
  • Blast radius: EnsembleGPUKernel host problem preparation, including calls differentiated with Enzyme. No kernel/controller or public signature changes.
  • Evidence: failing-before/passing-after allocation regression; fixed/adaptive Float32/Float64 timing matrix and sampled endpoint/time equality; QA, release CPU and standalone Enzyme passes; full CUDA reaches the known upstream Enzyme failure; LTS initialization failure reproduced on clean master.
  • Independent review: Codex CLI 0.164.0 / gpt-6.1-sol rated medium-low (ML3; L2 also applies), high confidence in source scope and moderate confidence in full runtime coverage; MERGE-WITH-ISSUES, no actionable source correctness findings. The verification request is addressed by the matched focused CUDA runs (full independent review).
  • Merge: needs human review; independent review is complete, but this remains a draft for Chris with the classified baseline failures visible. No approval or merge has been performed.

🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-benchgap (no session URL)
@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Independent review from a fresh Codex CLI 0.164.0 / gpt-6.1-sol session, reproduced below. The author posts this review artifact; the reviewer did not post or approve anything.

VERDICT: MERGE-WITH-ISSUES

Head: 1ba0a7674b10e4cb81801cd7e3d57c70b52a277b
Base: 9cef7c8360782dd472d46bcf03bd99ac7797f65e
Reviewer: Codex CLI 0.164.0 / gpt-6.1-sol
Session: 01a196fe-e8fd-7b91-bc36-2affb5bbbec8 (no session URL)

No actionable source correctness findings. The prior verification request is addressed, with the adaptive CUDA failures remaining explicit.

The two signature changes specialize an unconstrained callable type. They preserve callback execution, RNG handling, function bodies, and the @noinline construction boundary.

I inspected the focused launchers, unchanged test files, matching dependency manifests, and candidate/baseline logs:

Inspected CUDA evidence Candidate Exact baseline
Transfers 14 passes + 8 import passes Same
Gradient run, including imports 15 passes / 3 errors Same
Full CUDA group 793 pre-Enzyme passes; record-allocation error Same

Fixed Float32/Float64 exponential gradients, fixed Lorenz gradients, initial/final time, step-size, and initial-state gradients pass. Transfer assertions exercise parameters, times, and captured function values.

All three adaptive cases fail with matching Enzyme cannot handle unknown binary operator signatures in FastPower.fastlog2, through _controller_pow. Successful adaptive CUDA Enzyme differentiation remains unverified. These matched baseline failures provide no evidence against this patch.

The scripts include the original files without adding AD rules, weakening assertions, or swallowing failures. Gradient runs correctly exit unsuccessfully; the outer testset allows later testsets to report results.

The separate full-suite failure remains zero(::Type{EnzymeAllocationRecord}) in cuda_records.jl, consistent with the existing allocation issue.

Risk assessment

  • Initial independent rating: medium-low (ML3; L2 also applies), stated before reading the author’s rating or prior review. High confidence in the narrow source scope; moderate confidence in complete runtime coverage. Final rating unchanged.
  • Blast radius: EnsembleGPUKernel host preparation across backends, including differentiated solves. No public API, controller, numerical-method, or device-stepper changes.
  • Evidence: Inspected allocation failures before specialization on Julia 1.13.2 and 1.10.12, followed by both precision assertions passing. Release CPU, QA, and standalone CPU/JLArrays Enzyme logs pass. Recomputed large-N timing minima support approximately 2.19–2.27× Float32 and 1.51–1.61× Float64 high-level improvements; device timings remain effectively unchanged.
  • Limits: Timing evidence uses five-sample minima and sampled output equality. Adaptive CUDA gradients remain broken on both revisions. Full LTS CPU execution stops at initialization failures.
  • Baseline CI failures: CPU, JLArrays, OpenCL, and Metal LTS logs show the same two host-side initialization assertions, reproduced in the clean-baseline DAE file and tracked in SciMLBase issue 1659. Separate full baseline backend jobs were not run. Historical green master CI resolved different dependencies.
  • Author comparison: Agree with medium-low / ML3.
  • Merge eligibility: This review removes my verification objection to the patch. It does not establish immediate merge readiness: CUDA remains queued; x86 CPU, release JLArrays, and LTS oneAPI remain pending. Review/CI gates still apply, and the PR remains a draft for Chris.

I personally ran only read-only Git/GitHub queries and static evidence analysis. I launched no Julia, GPU work, tests, package operations, or benchmarks; edited no repository files; and posted or approved nothing.

Links


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

AMDGPU Julia 1.10 error on this PR: the same error occurs without this PR's change.

The extra error in build 1850 is in the "Computed initialization maps" test (test/gpu_kernel_de/stiff_ode/gpu_ode_modelingtoolkit_dae.jl:383): InvalidIRError compiling DiffEqGPU.gpu_ode_solve_kernel, with unsupported calls to ijl_field_index and ijl_has_no_field_error. The stack runs from getindex on a NamedTuple/Base.Pairs through SimpleNonlinearSolve solve and solve_initialization_problem inside the kernel. The job reports 80 pass / 2 fail / 1 error.

Build Commit Contains this PR ModelingToolkitBase :383 InvalidIRError
1847 (#582) 1123971 no 1.79.0 yes, 80/2/1
1849 (#583) de46ac2 no 1.79.0 yes, 80/2/1
1850 (this PR) 1ba0a76 yes 1.79.0 yes, 80/2/1

SciMLBase 3.58.0, AMDGPU 2.8.0, GPUCompiler 2.6.0 and KernelAbstractions 0.9.45 resolve identically in all three. The master run at 9cef7c8 (2026-10-21) resolved ModelingToolkitBase 1.77.5, and nobody has checked whether it had this error. This PR's diff only changes the host-side _make_kernel_problem / _prepare_kernel_problems signatures in src/solve.jl. It does not touch the in-kernel initialization path in the stack.

I compared the saved logs by grep only. Nothing was run on AMD hardware.


🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5 (log comparison by a claude-sonnet-5-5 subagent)
Conversation: https://claude.ai/code/session_0193DoGeYoy9sJDG2UUDBG7h

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

The previously unclassified AMDGPU Julia 1.10 initialization InvalidIRError also occurs without this PR. I inspected the completed version-only PR 582 CI run at commit 1123971edcb05bcbe593498f67e649f0b1a727cd. Its only difference from this PR's base is the package version, 3.21.6 → 3.21.7; source and tests match the base, and the RNG specialization change is absent.

Both that run and this PR's AMDGPU LTS run report 80 passed, 2 failed, 1 error in the DAE test group. The additional error has the same unsupported ijl_field_index and ijl_has_no_field_error calls through NamedTuple/Pairs indexing, SimpleNonlinearSolve, and solve_initialization_problem in the computed-initialization test.

The runs both use Julia 1.10.12, ModelingToolkitBase 1.79.0, NonlinearSolveBase 2.55.1, SimpleNonlinearSolve 2.14.6, SciMLBase 3.58.0, AMDGPU 2.8.0, GPUCompiler 2.6.0, KernelAbstractions 0.9.45, and Adapt 4.7.4. The printed dependency versions differ in IRTools, 0.4.24 versus 0.4.25, as well as the intentional DiffEqGPU version change. I am not claiming identical environments. A second branch without this change corroborates the same signature with ModelingToolkitBase 1.79.0.

This establishes that the extra AMDGPU LTS compilation error is pre-existing relative to this PR. It does not establish the upstream cause or a fix. This conclusion is based on inspected public CI logs; no local AMDGPU execution was performed, and no tests or tolerances were changed.

Links


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review November 5, 2026 03:33
@ChrisRackauckas
ChrisRackauckas merged commit 347e193 into SciML:master Nov 5, 2026
22 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant