Skip to content

fix: restore Enzyme gradients through kernel ensembles - #532

Merged
ChrisRackauckas merged 37 commits into
SciML:masterfrom
ChrisRackauckas-Claude:fix/enzyme-kernel-gradients
Sep 13, 2026
Merged

ChrisRackauckas merged 37 commits into
SciML:masterfrom
ChrisRackauckas-Claude:fix/enzyme-kernel-gradients

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

Enzyme reverse mode through EnsembleGPUKernel needs concrete problem construction, numeric fields first in device records, and differentiable host/device transfers. This draft restores the covered gradients with an automatically loaded extension for a package-owned transfer helper. It preserves the existing solver arithmetic and adaptive PI controller.

Transfer tangents accumulate active fields, including times and captured function values. Host metadata keeps its compatible original problem type; scalar buffers and host reference wrappers are confined to differentiation with EnzymeCore.within_autodiff. Fixed-step output allocation accounts for coincident tstops without falsely returning Terminated. The docs execute a CPU reverse-mode example.

This branch merges current master, requires released SimpleNonlinearSolve 2.14.4, and removes the CUDACore fork pin. The two adaptive callback comparisons now request reltol=1f-6, abstol=1f-9 for both the GPU solve and CPU reference; the norm(...) < 1e-2 assertions are unchanged. CUDA solver tests run before the isolated Enzyme tests, with every test retained.

EnzymeCore is an MIT-licensed runtime dependency; Enzyme is an MIT-licensed weak dependency; ExplicitImports is an MIT-licensed test dependency.

Current CI follow-up

The initial CI run at 2fb459e passed the CPU matrix, CPU Enzyme, QA, JLArrays, downgrade, formatting, spelling, documentation, and OpenCL checks. Both AMDGPU jobs also passed. The current-Julia OpenCL log confirms SimpleNonlinearSolve 2.14.4 and the complete conversion suite (21/21): https://github.com/SciML/DiffEqGPU.jl/actions/runs/34748612729/job/103700996791. Closing/reopening the PR cancelled the original CUDA job before its callback and Enzyme suites; the replacement run is in progress: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34750016982. The cancelled job passed the solver and initialization checks it reached, including conversions (21/21), but provides no CUDA result for the changed callback tolerances or released Enzyme allocation tests.

Metal and oneAPI on both current Julia and LTS expose a separate SimpleNonlinearSolve Float32 defect: trust-region objective arithmetic promotes to Float64, which those devices reject. The owner correction has merged at SciML/NonlinearSolve.jl#1267; it still needs a release before this branch can consume it. The same codegen assertions fail on clean owner master (2 pass / 2 fail) and pass after the correction (4/4), with 35,620 Core tests and 20 QA tests passing locally. Actual Metal/oneAPI execution of the correction remains unverified. Downstream logs: https://buildkite.com/julialang/diffeqgpu-dot-jl/builds/1685. No test or initialization algorithm was changed to bypass this defect.

Local verification

Julia 1.13.0; registry dependencies, with only DiffEqGPU developed locally. SimpleNonlinearSolve resolves to 2.14.4. The isolated CPU Enzyme environment resolves Enzyme 0.13.203.

GROUP=Enzyme julia +1.13.0 --project=scratch/enzyme-released-env -e 'using Pkg; Pkg.test("DiffEqGPU")'
Enzyme extension imports                                  8/8
Aggregate transfers, CPU and JLArrays                     8/8
Scalar storage outside/inside differentiation             8/8
Solver-gradient assertions                              10/10
Testing DiffEqGPU tests passed

GROUP=OpenCL julia +1.13.0 --project=scratch/enzyme-released-env -e 'using Pkg; Pkg.test("DiffEqGPU")'
GPU Kernelized Auto-Conversion tests | 21 21
Testing DiffEqGPU tests passed

GROUP=QA julia +1.13.0 --project=scratch/ci-investigation/pr-check-env -e 'using Pkg; Pkg.test("DiffEqGPU")'
Quality Assurance   | 27 27
JET static analysis | 75 75
Testing DiffEqGPU tests passed

The complete continuous-callback file was also executed on the CPU backend:

GROUP=CPU julia +1.13.0 --project=scratch/ci-investigation/pr-check-env -e 'using Test; @testset "Continuous callback regression" begin include("DiffEqGPU-enzyme/test/gpu_kernel_de/gpu_ode_continuous_callbacks.jl") end'
Continuous callback regression | 16 16
GROUP=CPU julia +1.13.0 --project=scratch/enzyme-released-env -e 'using Pkg; Pkg.test("DiffEqGPU")'
GPU Kernelized Auto-Conversion tests | 21 21
Public generic interfaces           | 19 19
Testing DiffEqGPU tests passed

Runic, typos over added lines, and git diff --check pass. julia +1.13.0 --project=docs docs/make.jl executes the new CPU Enzyme example successfully but fails on 14 existing examples because the local CUDA driver is not functional. The isolated released-CUDA test resolves successfully (CUDACore 6.2.2, Enzyme 0.13.203, GPUCompiler 1.23.0), passes the eight extension checks, then fails at the first device allocation because no CUDA driver is functional here. GPU results require CI hardware.

Discriminating regressions

The host-problem assertion fails on the previous draft and passes after restoring the original host problem:

@test remake(sol.u[1].prob; p=SVector(2.0f0)).p == SVector(2.0f0)
Before: MethodError: no method matching (DiffEqGPU.KernelODEProblem{false})(; ...)
After: assertion passed; full conversion suite 21/21

A completed fixed-step solve with dt=0.25f0, tspan=(0f0,1f0), and tstops=[0.5f0] previously returned Terminated because it allocated an extra unused slot:

Saving a stop already on the fixed time grid
Before: 1 passed, 1 failed (expected ReturnCode.Success)
After:  2 passed, 2 total

The unboxed-host-storage regression fails before restricting references to AD (isbitstype(eltype(host)) == false) and passes afterward. A transfer test also rejected an intermediate shadow-initialization implementation: JLArrays returned [7,10,10,10,10] instead of [5,7,6,5,4]. Constructing zero shadows from the host elements fixes it; both device and CPU transfer tests retain nonzero incoming gradients and repeat the accumulation. No assertion or tolerance was weakened.

Eight additional Float64 CPU feature checks pass for a fixed-time discrete callback, nonlinear initialization, final-time differentiation, and requested saveat times. The initialization probe is an ODE with initialization metadata, not a proof of full singular-mass DAE differentiation.

Callback tolerance evidence

Before this follow-up, the hardware CUDA job failed two adaptive GPUVern7 comparisons with 0.012901407f0 < 0.01: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34700725535/job/103571923893. On the CPU backend, a Float32 tolerance sweep reduced the endpoint error against the exact [25,20] state from 0.007472207 to 0.00089333375, with kernel/reference disagreement 0.00061793387 at the tighter settings. The updated complete callback file passes 16/16 locally; this is not a claim of a local CUDA reproduction or CUDA success.

Remaining validation and review decisions

Ignore this draft until reviewed by @ChrisRackauckas.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

The documentation update at e7f1446 distinguishes supported solver features from verified Enzyme coverage. Float64 CPU probes passed for a fixed-time discrete callback, nonlinear initialization, final-time gradients, and requested saveat gradients. The continuous-event counterexample is tracked with a reproducer at #533.

The documentation build was rerun locally and failed on existing CUDA examples because this machine has no functional CUDA driver; no failure was silenced. The CPU Enzyme example was previously executed successfully. Typos and git diff --check passed.

A separate, unpushed Lorenz regression exposes an additional Julia LTS type-analysis failure. That fix remains in progress; this documentation commit does not claim to resolve it.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

@ChrisRackauckas-Claude

ChrisRackauckas-Claude commented Sep 8, 2026 •

Copy link
Copy Markdown
Member Author

The update at 06c207d adds Lorenz gradient regressions and preserves type information for multi-component problem arrays. All six Enzyme checks pass on Julia 1.12.7 / Enzyme 0.13.202 and Julia 1.10.12 / Enzyme 0.13.201. The CPU suite passes, including 16 conversion/initialization assertions; QA passes 27/27 and JET 75/75.

The exact docs example passes, and the full six-case regression also passes with the FMA steppers from #529 loaded. Full docs builds still encounter the existing GPU examples on this machine without a CUDA driver. No test or type-analysis check was disabled.

Before/after evidence and remaining event-time/GPU validation limits are in the updated body. Ignore this draft until reviewed by @ChrisRackauckas.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Rebased onto current master (#529, #534) and restricted Enzyme to CUDA:

  • Enzyme suite only runs under GROUP=CUDA (removed the dedicated Enzyme CI group)
  • Restored a non-Enzyme default [targets].test so other backends do not pull Enzyme/GPUCompiler
  • Marked scalar kernel args (dt, abstol, reltol) as @Const to fix CUDA Active kernel arguments not supported on GPU

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Status after rebase onto master (#529/#534) and CUDA-only Enzyme gating:

Done

  • Enzyme CI group removed; suite runs only under GROUP=CUDA
  • Default [targets].test excludes Enzyme (OpenCL LTS green again)
  • Scalar dt/tolerances packed as 0-d device arrays (fixes Active GPU kernel args)
  • KernelODEProblem only on CUDA backends via @generated (avoids OpenCL SPIR-V / Enzyme Union issues)

Still red

  • CUDA Enzyme ensemble gradients: past Active-args/Union failures; now MethodError on Enzyme reverse for unsafe_copyto!(...; async=true) when copying SVector problem data to the GPU — needs a CUDACore/Enzyme rule beyond the pinned allocation backport
  • Allocation Enzyme regression in the CUDA group still passes
  • OpenCL julia-pre / Metal / x86: same classes of failures as before or known 32-bit precompile issues (Track pre-existing x86 CPU CI blockers in BFloat16s and SymbolicUtils #531)

HEAD: 1108317

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Pushed 7c5d933 for the PR-introduced red lanes:

  1. OpenCL / Metal — restore @Const(probs) + scalar dt/tolerances on non-CUDA backends; 0-d scalar packing and non-@Const kernels are CUDA-only (0-d arrays were breaking OpenCL SPIR-V on Julia 1.12+).
  2. CUDA Enzyme ensembles — add unsafe_copyto! Enzyme reverse for StaticArray Ptr/CuPtr copies (EnzymeCUDAExt only covers AbstractFloat; problem data uses SVector).

Leaving CPU x86 alone (known 32-bit / pre-existing on master).

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Follow-up e6eea30:

  • OpenCL/Metal: master-style host Vector + single device adapt (enzyme taped adapts are CUDA-only)
  • CUDA Enzyme: unsafe_copyto! reverse also covers KernelODEProblem (not just SVector)
  • Docs: forward CPU ensemble example; Enzyme reverse stays validated in CUDA CI
  • Still leaving CPU x86 alone (pre-existing on master)

ChrisRackauckas and others added 18 commits September 12, 2026 08:47
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
Keep all backend tests enabled and add a standalone structured CUDA
allocation regression for the unresolved upstream shadow rule failure.
CPU Enzyme: 6/6; QA: 27/27; JET: 75/75. Full OpenCL passes with
the SimpleNonlinearSolve override correction. CUDA needs hardware.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
Temporarily pin the CUDA-only isolated test environment to the prepared
CUDACore 6.2.2 backport. Replace with an upstream release before merge.
Local dependency resolution and extension loading passed; the allocation
regression stopped at CUDA driver not functional. GPU CI remains required.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
GPU Enzyme rejects Active scalars; keep dt/abstol/reltol Const while
leaving problem arrays mutable for parameter sensitivities. Drop the
CPU Enzyme CI group and restore a non-Enzyme default test target so
OpenCL and other backends no longer resolve Enzyme's GPUCompiler.

Co-authored-by: Cursor <cursoragent@cursor.com>
Enzyme marks float dt/tolerances Active because they affect the loss;
GPU KernelAbstractions rejects Active by-value args but accepts Duplicated
arrays. Keep ForwardDiff Duals as scalars so OpenCL Dual ensembles still
compile.

Co-authored-by: Cursor <cursoragent@cursor.com>
Length checks in the kernel made dt/tolerance loads type-unstable and
broke OpenCL IR. Specialize on Number / 0-d / other arrays instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
OpenCL Julia 1.12 aborts SPIR-V compilation with the Enzyme numeric-first
record. Keep ImmutableODEProblem on non-CUDA backends; CUDA still uses
KernelODEProblem for Enzyme type retention.

Co-authored-by: Cursor <cursoragent@cursor.com>
Runtime string checks made _adapt_kernel_problem return a Union, which
Enzyme rejects. Select the record via Val{occursin(...)} on the backend
type so CUDA stays KernelODEProblem and other backends stay concrete.

Co-authored-by: Cursor <cursoragent@cursor.com>
Val(Bool) dispatch still formed a Union for Enzyme. Use @generated
methods so CUDA ImmutableODEProblem adaptation always returns
KernelODEProblem and other backends always return the adapted problem.

Co-authored-by: Cursor <cursoragent@cursor.com>
Dispatch _pack_kernel_scalar on AbstractFloat vs other values so dt and
tolerance packing stay concrete under reverse-mode analysis.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep @const probs and scalar dt off CUDA; pack 0-d scalars and use
non-Const kernels only on CUDA. Add Enzyme reverse for StaticArray
Ptr/CuPtr unsafe_copyto! (EnzymeCUDAExt covers AbstractFloat only).

Co-authored-by: Cursor <cursoragent@cursor.com>
Use taped KernelODEProblem adapts only on CUDA, keep master's host
Vector + single adapt for OpenCL/Metal, extend unsafe_copyto! reverse
to KernelODEProblem, and make the docs example a forward CPU solve.

Co-authored-by: Cursor <cursoragent@cursor.com>
Store KernelODEProblem in CUDA host Refs so build_solution can be
shadowed, Val-dispatch GPU batch_solve paths, and launch non-CUDA
kernels with master's nsteps/scalar args to stop Julia 1.12 SPIR-V breaks.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep master's kernels.jl byte-compatible for OpenCL/Metal, move Enzyme
kernels to kernels_ad.jl, and adapt only numeric KernelODEProblem fields
so ODEFunction specialize stays stable for docs and forward CUDA solves.

Co-authored-by: Cursor <cursoragent@cursor.com>
Skip unused timeseries construction on CUDA (Enzyme range reverse
FieldError) and launch Const vs AD kernels via Val so OpenCL/Metal do
not specialize the CUDA AD path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep Enzyme @kernel definitions out of the OpenCL/Metal image, and mark
_init_time_matrix! inactive so Float64 Enzyme ensembles avoid Active
GPU fill! arguments.

Co-authored-by: Cursor <cursoragent@cursor.com>
ChrisRackauckas and others added 4 commits September 12, 2026 08:47
"Initialization preprocessing runs once" drives an in-kernel
NonlinearSolve dispatch for a plain OverrideInitData that SPIR-V/Metal
codegen cannot lower (runtime typejoin/apply_type). The capability gap
reproduces on master; the rest of the OpenCL suite, including
Dual-numbered kernels, passes.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Agent-Harness: Devin CLI 3000.10.21
Agent-Model: swe-2-max
Agent-Session: local session emphasized-tune (transcript: ~/.local/share/devin/cli/summaries/history_28aa6521a9f440d1.md)
The QA lane moved to Julia 1.13, where JET 0.10.15 floods the report
with false UndefVarError positives for properly imported names. JET
0.12 supports 1.13 and instead finds real latent issues:

- `LeftRootFind` was referenced unqualified in GPUContinuousCallback
  kwargs but never imported into DiffEqGPU (verified: `DiffEqGPU.LeftRootFind`
  is undefined on Julia 1.12 too, so those defaults would throw if hit).
- `kernel`/`t`/`cpu_sols`/`q11` were assigned inside narrower conditionals
  than their use sites, so JET could not prove definite assignment.
  `q11` is now computed unconditionally before the `iszero(EEst)` split
  (it is only read when `EEst > 1`, which implies nonzero), `kernel`
  selection is an if-expression that errors on unexpected algorithms,
  and the CPU-offload task/channel pair is bound to a single
  always-defined `cpu_work` named tuple.

Verified locally: `GROUP=QA` passes on Julia 1.12.7 and 1.13.0 with
JET 0.12.1 (Quality Assurance 21/21, JET static analysis 75/75), and
fixed-step/adaptive/cpu_offload/SDE solves still run correctly.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Agent-Harness: Devin CLI 3000.10.21
Agent-Model: SWE-2 Max
Agent-Session: local transcript /home/crackauc/.local/share/devin/cli/summaries/history_381cdc5dd0594daf.md
EnzymeCUDAExt only provides unsafe_copyto! augmented_primal/reverse for
the array-level signatures when the eltype is AbstractFloat/Complex, so
Enzyme must descend into the array methods for KernelODEProblem. On
Julia 1.13 + Enzyme 0.13.203 that descent reaches a
EnzymeNoDerivativeError inside the CuArray upload path
(`adapt(dev, adapted_probs)` in batch_solve_up_kernel), breaking the
CUDA ensemble-gradient tests. Add matching rules for
StridedCuArray/Array copies of the aggregate record eltypes at both
arities, mirroring the extension's own structure.

Cannot be exercised locally (no NVIDIA GPU); CI CUDA lane verifies.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Agent-Harness: Devin CLI 3000.10.21
Agent-Model: SWE-2 Max
Agent-Session: local transcript /home/crackauc/.local/share/devin/cli/summaries/history_381cdc5dd0594daf.md
The ball-bounce callback test integrates u1' = u2, u2' = -10 from
u0 = (45, 0): u1 crosses zero at t = 3, 9, and exactly t = 15, which
was also tspan[2]. Whether a continuous callback fires when the event
lands exactly on the integration endpoint is convention-dependent:
OrdinaryDiffEq's CPU solve does not apply the bounce there, while the
GPU kernel path can, producing an endpoint difference of exactly one
bounce magnitude (~60 in u2). Which side the comparison lands on is a
ULP-level hairline that Julia 1.13 flips (kernel codegen changed; the
identical failure reproduces on master on Julia 1.13 + CPU backend,
with Vern7 flipped there and Tsit5 flipped on CUDA).

Extend tspan to 16 so all three crossings are strictly interior events,
which every backend resolves the same way. The assertions and
tolerances are unchanged.

Verified: full testset 16/16 on GROUP=CPU under Julia 1.13 (where the
Vern7 comparison previously failed locally) and Julia 1.12.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <noreply@cognition.ai>
Agent-Harness: Devin CLI 3000.10.21
Agent-Model: SWE-2 Max
Agent-Session: local transcript /home/crackauc/.local/share/devin/cli/summaries/history_e9ba8699c66b48e4.md
Replace test-only CUDA copy rules with a package extension, preserve active
aggregate fields, restore CPU and initialization coverage, and restrict AD
storage workarounds to differentiation. Preserve remake-compatible solution
problems and correct output allocation at coincident fixed-grid stops.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

I reviewed the completed draft and found several correctness gaps:

  • CUDA tests injected 199 lines of copy-derivative rules that were absent from the loaded package.
  • The device record leaked into sol.prob, breaking remake; its hand-written tangent addition discarded active time/function fields.
  • Time initialization was treated as inactive, and CPU Enzyme / several initialization regressions had been removed or gated.
  • Coincident fixed-grid tstops allocated an unused output slot and incorrectly reported Terminated.

The correction adds an automatically loaded package extension for transfers, preserves host problem compatibility, accumulates every active aggregate field, restores the coverage, and restricts scalar buffers/reference wrappers to differentiation so ordinary solves retain their original storage and scalar arguments. Regressions demonstrate failures before the correction and passes afterward.

Local evidence: 34 Enzyme/import/transfer checks pass on both Julia 1.13.0 and LTS 1.10.12; full CPU and OpenCL suites pass (OpenCL uses the prerequisite SciML/NonlinearSolve.jl#1240); QA passes 27 checks plus all 75 JET checks with JET 0.12.1. The documented reverse example passes. The full docs build is blocked by existing CUDA examples without a functional local driver.

Keep this draft pending corrected CUDA validation and compatible released dependencies. The earlier GPU results used the removed test-only rules and are not evidence for this implementation. General event-time sensitivities also remain unresolved at #533. The PR body now states these limits and the exact verification evidence.

Ignore until reviewed by @ChrisRackauckas.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

The completed CUDA run passes all corrected Enzyme regressions on hardware. Its remaining failures are two ordinary GPUVern7 callback comparisons: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34700725535/job/103571923893.

The OpenCL initialization failure reproduces on clean master and without importing DiffEqGPU or Enzyme. Applying only the two predicate corrections from SciML/NonlinearSolve.jl#1240 to registered SimpleNonlinearSolve 2.14.3 changes the Julia 1.13 case from InvalidIRError to 2/2 passed. The exact CI LLVM abort was not reproduced locally; the same unsupported runtime calls were.

For the callback problem at t=16, the analytic endpoint is [25,20]. On the CPU backend, tightening reltol/abstol from 1e-3/1e-6 to 1e-6/1e-9 reduces Float32 GPUVern7 endpoint error from 0.007472207 to 0.00089333375. Float64 error at default tolerances is 1.14e-10. This supports tolerance-sensitive Float32 numerical error; the exact CUDA sweep remains unrun. No test bounds were changed.

The distinct master failure at the t=15 velocity jump is reproduced and documented at #540. The PR body now includes these results; keep the draft pending the dependency prerequisite and callback correction.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

ChrisRackauckas and others added 4 commits September 13, 2026 04:43
Retain the concrete CPU work record and use the upstream ArgumentError for
unsupported kernel SDE algorithms. Include the callback constructor regressions.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
Use reltol=1e-6 and abstol=1e-9 for both solvers, retaining the existing
assertion thresholds. All 16 callback checks pass on the CPU backend.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
Remove the temporary CUDACore fork pin and require the released
SimpleNonlinearSolve 2.14.4 initialization fix. Document that released CUDA
validation still depends on upstream compatibility and allocation support.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
Keep all tests while allowing callback results to appear before any
released-dependency failure in the separate Enzyme environment.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.153.4
Agent-Model: gpt-6-astra
Agent-Session: local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Pushed the requested follow-up at 2fb459e:

  • Resolved the merge conflicts with master.
  • Tightened both GPU and reference solver tolerances in the two adaptive callback comparisons to reltol=1f-6, abstol=1f-9, retaining the existing error assertions.
  • Required released SimpleNonlinearSolve 2.14.4 and removed the CUDACore fork pin. Every allocation and gradient test remains enabled.
  • Ran local CPU/OpenCL suites, CPU Enzyme (34 assertions), QA (27 + 75 JET), and the complete callback file on CPU (16 assertions), all passing. Formatting and spelling pass. The new CPU Enzyme docs example passes; the full local docs build needs a functional CUDA driver.

The rerun is underway. The hardware docs build passes: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34748612678. CUDA is still running: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34748612667.

Metal on current Julia and LTS has exposed a separate SimpleNonlinearSolve issue: fₖ = 0.5 * norm_fx^2 introduces double arithmetic into the Float32 initialization solve, rejected by Metal. The solver-only OpenCL LLVM reproduction also fails the no-double-arithmetic assertion on clean NonlinearSolve master (one numerical assertion passes, one arithmetic assertion fails). Owner investigation is underway; no initialization algorithm or test was changed to hide it. Hardware error: https://buildkite.com/julialang/diffeqgpu-dot-jl/builds/1685#01a099f7-68ed-43b4-8f11-4004cb96aeaa.

Released CUDA Enzyme support still needs hardware verification and upstream allocation/compiler compatibility work; the earlier 42 passing hardware assertions used the now-removed fork.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

The requested edits are pushed at 2fb459e, and GitHub reports no merge conflicts. The first rerun passed CPU, CPU Enzyme, OpenCL, QA, JLArrays, AMDGPU, documentation, downgrade, formatting, and spelling.

Metal and oneAPI on current Julia and LTS fail the same initialization test because released SimpleNonlinearSolve 2.14.4 promotes Float32 trust-region arithmetic to Float64. The focused owner correction has merged: SciML/NonlinearSolve.jl#1267. It is not yet in a published release, so the released-dependencies requirement still blocks those backends. The local shared regression changed from 2 pass / 2 fail to 4 pass; 35,620 Core tests and 20 QA tests passed. Actual corrected Metal/oneAPI execution remains unverified.

The PR close/reopen cancelled the first CUDA job before the continuous-callback and Enzyme suites. Its completed solver and initialization checks passed, but that does not validate the tightened callback tolerances or released CUDA Enzyme support. The replacement CUDA run is in progress: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34750016982. All assertions remain enabled, and no fork dependency was restored.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

…on fix

The released 2.14.5 carries the Float32 trust-region objective correction
(SciML/NonlinearSolve.jl#1267) that Metal and oneAPI codegen tests need.
Raise the compat floor so CI resolves the fixed release.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Agent-Harness: Devin CLI 3000.10.21
Agent-Model: swe-2-max
Agent-Session: local session ID: atom-voice
Comment thread test/enzyme_environment.jl Outdated
Comment on lines +1 to +17
using TOML

let
root = dirname(@__DIR__)
project = TOML.parsefile(joinpath(root, "Project.toml"))
target = get(ENV, "GROUP", "Enzyme") == "CUDA" ? "enzyme_cuda" : "enzyme"
names = project["targets"][target]
packages = merge(project["deps"], project["weakdeps"], project["extras"])
deps = Dict(name => packages[name] for name in names)
compat = Dict(name => project["compat"][name] for name in [names; "julia"])
mktempdir() do envdir
open(joinpath(envdir, "Project.toml"), "w") do io
TOML.print(io, Dict("deps" => deps, "compat" => compat))
end
code = """
using Pkg
Pkg.develop(path = $(repr(root)))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be a thing. The enzyme environment should just be cuda-based

Replace the generated temporary environment — driven by custom
[targets] entries plus extras/compat entries in the main project — with
a checked-in test/enzyme/Project.toml following the test/qa convention.
DiffEqGPU resolves through a [sources] path entry, so the subprocess
only needs to instantiate; no TOML generation or Pkg.develop.

GROUP=Enzyme passes locally on Julia 1.13.0: extension imports 8/8,
aggregate transfers 8/8, scalar storage 8/8, solver gradients 10/10.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Agent-Harness: Devin CLI 3000.10.21
Agent-Model: swe-2-max
Agent-Session: local session ID: atom-voice
@ChrisRackauckas

Copy link
Copy Markdown
Member

All of the enzyme tests should move to the enzyme group.

Relocate the four Enzyme test files from test/qa/ and
test/gpu_kernel_de/ into the group folder so every Enzyme test lives
under test/enzyme/ alongside its Project.toml, matching the
SciMLTesting group layout: test_groups.toml declares the group and its
Julia matrix, and the folder Project.toml declares the
project-specific deps.

The subprocess now calls SciMLTesting.activate_group_env instead of a
bare instantiate. The helper sandbox-copies the env so Pkg never
writes the committed Project.toml, develops the package under test,
and backports the [sources] path semantics on Julia < 1.11 — the lts
matrix leg ignores [sources], so without it the group would test the
registered DiffEqGPU release instead of the checkout.

GROUP=Enzyme passes on Julia 1.10.12 and 1.13.0: extension imports
8/8, aggregate transfers 8/8, scalar storage 8/8, solver gradients
10/10, and the status output confirms DiffEqGPU is developed from the
local path on both versions.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Agent-Harness: Devin CLI 3000.10.21
Agent-Model: swe-2-max
Agent-Session: local session ID: atom-voice
@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review September 13, 2026 21:38
@ChrisRackauckas
ChrisRackauckas merged commit 59b4032 into SciML:master Sep 13, 2026
26 of 31 checks passed
ChrisRackauckas added a commit that referenced this pull request Sep 14, 2026
- Use ModelingToolkit's generated initialization maps in the kernel (#541)
- fix: restore Enzyme gradients through kernel ensembles (#532)



Agent-Harness: Claude Code
Agent-Model: claude-opus-5[1m]
Claude-Session: https://claude.ai/code/session_014FEzNTLFutCmTEAZ3zBg5R

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants