Repository navigation
Specialize kernel problem preparation on RNG callbacks - #584
ChrisRackauckas merged 1 commit into
Conversation
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Codex <noreply@openai.com> Agent-Harness: Codex CLI 0.164.0 Agent-Model: gpt-6-astra Agent-Session: local job diffeqgpu-benchgap (no session URL)
Independent review from a fresh Codex CLI 0.164.0 / gpt-6.1-sol session, reproduced below. The author posts this review artifact; the reviewer did not post or approve anything. VERDICT: MERGE-WITH-ISSUES Head: No actionable source correctness findings. The prior verification request is addressed, with the adaptive CUDA failures remaining explicit. The two signature changes specialize an unconstrained callable type. They preserve callback execution, RNG handling, function bodies, and the I inspected the focused launchers, unchanged test files, matching dependency manifests, and candidate/baseline logs:
Fixed Float32/Float64 exponential gradients, fixed Lorenz gradients, initial/final time, step-size, and initial-state gradients pass. Transfer assertions exercise parameters, times, and captured function values. All three adaptive cases fail with matching Enzyme The scripts include the original files without adding AD rules, weakening assertions, or swallowing failures. Gradient runs correctly exit unsuccessfully; the outer testset allows later testsets to report results. The separate full-suite failure remains Risk assessment
I personally ran only read-only Git/GitHub queries and static evidence analysis. I launched no Julia, GPU work, tests, package operations, or benchmarks; edited no repository files; and posted or approved nothing. Links🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra |
AMDGPU Julia 1.10 error on this PR: the same error occurs without this PR's change. The extra error in build 1850 is in the "Computed initialization maps" test (
SciMLBase 3.58.0, AMDGPU 2.8.0, GPUCompiler 2.6.0 and KernelAbstractions 0.9.45 resolve identically in all three. The master run at 9cef7c8 (2026-10-21) resolved ModelingToolkitBase 1.77.5, and nobody has checked whether it had this error. This PR's diff only changes the host-side I compared the saved logs by 🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5 (log comparison by a claude-sonnet-5-5 subagent) |
The previously unclassified AMDGPU Julia 1.10 initialization Both that run and this PR's AMDGPU LTS run report 80 passed, 2 failed, 1 error in the DAE test group. The additional error has the same unsupported The runs both use Julia 1.10.12, ModelingToolkitBase 1.79.0, NonlinearSolveBase 2.55.1, SimpleNonlinearSolve 2.14.6, SciMLBase 3.58.0, AMDGPU 2.8.0, GPUCompiler 2.6.0, KernelAbstractions 0.9.45, and Adapt 4.7.4. The printed dependency versions differ in IRTools, 0.4.24 versus 0.4.25, as well as the intentional DiffEqGPU version change. I am not claiming identical environments. A second branch without this change corroborates the same signature with ModelingToolkitBase 1.79.0. This establishes that the extra AMDGPU LTS compilation error is pre-existing relative to this PR. It does not establish the upstream cause or a fix. This conclusion is based on inspected public CI logs; no local AMDGPU execution was performed, and no tests or tolerances were changed. Links
🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra |
Ignore this draft until reviewed by @ChrisRackauckas.
The host preparation helpers forward an RNG callback for every trajectory. Specializing those two helpers on the callback type removes temporary allocation and dynamic dispatch during
EnsembleGPUKernelproblem preparation. At 2^21 Float32 Lorenz trajectories, the high-level solve drops from about 475 ms to 210–220 ms; construction drops from 337–360 ms to about 42 ms. Callback invocation, Enzyme's noinline construction boundary, numerical methods, public return types, and output layouts are preserved.Measurements
GPU 1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (sm_120); AMD Ryzen 9 9950X3D; four Julia threads; Julia 1.13.2; CUDA.jl 6.2.2. Source baseline:
9cef7c8360782dd472d46bcf03bd99ac7797f65e. The Lorenz setup matches lorenz_ensemble.jmd: tspan=(0,1), dt=0.001, save_everystep=false, dense=false, CPU offload=0, adaptive abstol=reltol=1e-8. The Float64 arm uses the same explicitly typed setup.Milliseconds, repeat 1 / repeat 2. Each repeat is the minimum of five synchronized samples after a warmup. Device timings use device-resident problems through
vectorized_solve/vectorized_asolve, including output allocation and initialization but excluding transfers and host materialization.At N=2^21, repeat-to-repeat minimum spread was 0.4–4.4% before and 1.0–5.5% after for high-level solves; device-only spread was at most 2.1%. Smaller host timings are noisier (up to 75% between minima), so the large-N results are the strongest performance evidence. Dependency resolution for the test environment overlapped part of the Float64 after-run; observed system load stayed below 10. Solver dependency versions are identical between runs; the development manifest normalizes standard-library entries to the running Julia version.
For each precision/mode/N, sampled trajectory initial/final states and times from the high-level and unchanged device-only paths matched exactly, and sampled return codes succeeded. This is sampled equality, not an exhaustive comparison of all trajectories.
The V100 values in the rendered benchmark come from different hardware. At N=2^21, those rendered JAX values are 5541 ms fixed and 1145 ms adaptive. They are context, not a controlled cross-library speedup measurement against this GPU 1 run.
Validation
The new CPU allocation regression uses the retained-buffer size as its reference, allowing allocator overhead while detecting per-trajectory temporaries.
Before the source change:
The final fixture constructs problems inside a
::Type{T}function barrier, so its callback has a statically known precision on both supported Julia versions. With this identical fixture, clean master fails on Julia 1.13.2 (1253632 < 917888) and Julia 1.10.12 (1253584 < 917888). After the two production signature changes, the assertion passes for Float32 and Float64 on both versions. All 35 existing assertions in that host-storage file also pass. The allocation limit is unchanged.Validation uses isolated copies of the package's declared test environment developed against the checkout. GPU checks use physical GPU 1, NVIDIA RTX PRO 6000 Blackwell Max-Q. No tests or tolerances are skipped, disabled, or loosened.
9cef7c8360782dd472d46bcf03bd99ac7797f65eThe CUDA Enzyme error is
MethodError: no method matching zero(::Type{EnzymeAllocationRecord})in the unchangedcuda_records.jltest, matching the known CUDACore issue. The resolved test environment uses CUDA/CUDACore 6.4.1 and Enzyme 0.13.216. The full suite stops before transfer/gradient assertions; the focused runs below execute those unchanged files separately. The separate CUDA.functional and 256-element add-one kernel probes pass. No package docstring, public API or docs change is included, so no docs build was run.Group commands use
REPOfor the checkout andTEST_ENVfor its isolated test environment:Julia uses four threads and one precompile task. The full CUDA group retains its two distributed workers and launches the Enzyme child unchanged. The final focused LTS regression runs use
--compiled-modules=noto avoid nested cold-cache compiler processes.Focused CUDA Enzyme verification
The independent reviewer requested execution of the files after the allocation failure. Both this head and clean master pass all 14 CUDA transfer assertions plus 8 extension-import assertions. The unchanged gradient file, included inside an outer testset so later testsets also execute, gives the same 15 passed / 3 errors on both revisions (the pass count includes 8 import and 7 gradient assertions):
The adaptive errors contain Enzyme's
cannot handle unknown binary operatorat a bit mask inFastPower.fastlog2, reached through_controller_pow. This is distinct from the record-allocation failure in the full CUDA suite. No adaptive CUDA Enzyme success is claimed. A standalone reproducer importing only CUDA, Enzyme, KernelAbstractions and standard libraries passes its CPU custom-rule control (4/4), then reproduces the bit-mask exception in both direct deferred CUDA and KA kernels (2 primal checks pass, 1 differentiation error each). This is consistent with the existing Enzyme deferred-codegen custom-rule issue. Enzyme main and a runtime version bisect were not tested. This PR does not change FastPower or the controller.Focused commands, run in each checkout's root with the same resolved dependencies, are shown below. Local launchers additionally assert the loaded checkout and run the CUDA functional/small-kernel probe before these includes.
Current-head release CPU, Enzyme, JLArrays, OpenCL, Metal, oneAPI, QA, formatting, spelling and documentation checks pass. CPU, JLArrays, OpenCL, Metal and oneAPI on Julia 1.10 report the same two initialization assertions described above, with 82 passed / 2 failed. Completed CUDA CI fails at the same
zero(::Type{EnzymeAllocationRecord})error as the local clean-master control; later Enzyme checks are not reached.AMDGPU release CI reports 36 passed / 2 errored in implicit array solves, with missing
lufact!(::ROCBackend, ::ROCArray{Float32,3,...})for Rosenbrock23 and Rodas5P. The clean-base release job at9cef7c8has the identical failures. Both use AMDGPU 2.8.0, GPUCompiler 2.6.0 and SciMLBase 3.58.0; Julia patch and ModelingToolkitBase versions differ. This is an observed base failure, not a fully matched local AMDGPU rerun. AMDGPU LTS CI completes with 80 passes, the two known initialization assertion failures, and an additionalInvalidIRErrorinvolvingijl_field_index/ijl_has_no_field_errorthrough NamedTuple/Pairs indexing in initialization. The version-only source baseline at1123971edcb05bcbe593498f67e649f0b1a727cdreproduces the same 80/2/1 result and both unsupported calls without this PR. Its source/tests match base9cef7c8; only the package version changes. Julia 1.10.12, ModelingToolkitBase 1.79.0, NonlinearSolveBase 2.55.1 and the relevant GPU/compiler versions match; IRTools differs (0.4.24 versus 0.4.25). This establishes the signature as pre-existing relative to this PR, without claiming a byte-identical environment or upstream cause. Detailed baseline comparison. No local AMDGPU execution was performed. These results come from inspected job logs, not inferred badges.Risk assessment
🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)