Skip to content

Fix EnsembleGPUArray CPU time leak with per-trajectory tspan - #553

Merged
ChrisRackauckas merged 4 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/551-egpuarray-time-leak
Sep 28, 2026
Merged

ChrisRackauckas merged 4 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/551-egpuarray-time-leak

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Summary

EnsembleGPUArray ParamWrapper kernels reassigned the kernel argument t while reparameterizing from normalized time [0,1] to each trajectory's physical tspan. On the KernelAbstractions CPU backend, that mutated value leaked across lanes in a workgroup, so trajectories sharing a batch but not a tspan got wrong right-hand-side times (and wrong solutions). Physical time is now kept in a separate local t_phys in all ParamWrapper gpu_kernel/jac_kernel/W_kernel/Wt_kernel variants (IIP and OOP). The CPU analytic regression covers in-place and out-of-place RHS and equal vs unequal span lengths.

Verification

Before (master archive), parameterized probe (IIP/OOP × equal/unequal spans)

julia --project=scratch/master-env -e 'using Pkg; Pkg.develop(path=".../scratch/base"); include("scratch/probe_cases.jl")'
passes=4 fails=12
# e.g. iip=true equal_span=false traj=2 got=3.8984375 exact=3.96875
#      iip=false equal_span=true traj=2 got=3.21875 exact=3.25

After fix, same probe on this branch

passes=16 fails=0
# all abserr=0.0

CPU test suite

GROUP=CPU julia --project=test -e 'using SafeTestsets; @time @safetestset "Public generic interfaces" begin include("test/public_interface.jl") end'
Test Summary:             | Pass  Total     Time
Public generic interfaces |   35     35  1m27.3s
cpu_exit=0

Runic

julia --project=~/.julia/environments/runic -e 'using Runic; for path in ("src/ensemblegpuarray/kernels.jl", "test/public_interface.jl"); s=read(path,String); @assert s==Runic.format_string(s); end'
round-trip ok: src/ensemblegpuarray/kernels.jl
round-trip ok: test/public_interface.jl

Not verified

  • No GPU device on this host; CUDA/AMDGPU EnsembleGPUArray paths with heterogeneous tspan were not run.
  • jac_kernel and jac_kernel_oop could not be validated on CPU because of a pre-existing __index_Global_Linear MethodError (reproduced on master too). RHS / W / Wt ParamWrapper kernels were checked analytically; the Jacobian helpers only received the same t_phys edit by construction.
  • Full GROUP=CPU runtests.jl was not completed end-to-end here.

Risk assessment

Low. The change is a local rename of the reparameterized time binding so CPU lanes cannot overwrite a shared kernel argument; numerical behavior with identical tspans is unchanged. CPU evidence covers IIP and OOP RHS with equal and unequal span lengths (16/16 analytic passes on head vs 4/16 on master). Residual risk is limited to GPU backends (not exercised here) and to the two CPU Jacobian helpers that already throw on master before any time transform runs.

Reviewer pushback

  • Prefer also placing this regression under GROUP in SUPPORTS_LUFACT for CUDA CI?
  • Want a follow-up to fix the pre-existing jac_kernel CPU MethodError so those helpers can be probed?
  • Confirm GPU backends with workgroup size > 1 and mixed tspans.

Closes #551

Please ignore this draft until reviewed by @ChrisRackauckas.

Harness: Cursor Agent CLI 2026.09.26-dd393fe; model: auto; transcript: /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu

Risk assessment

  • Risk: low
  • Blast radius: Only EnsembleGPUArray users, through the ParamWrapper kernels in src/ensemblegpuarray/kernels.jl (gpu_kernel, jac_kernel, W_kernel and Wt_kernel, both IIP and OOP). The change renames a local variable: the reparameterized time now goes into t_phys instead of overwriting the kernel argument t. On the KernelAbstractions CPU backend this fixes wrong results when trajectories in one batch have different tspans. On GPU backends each thread already has its own copy of t, so behaviour there should be the same as before. No public API changes, so no SemVer impact. No non-public dependency internals are used (remake is exported by SciMLBase). No tests were weakened; the PR only adds a 16-assertion analytic testset.
  • Evidence: On head 906d240 these pass: CPU (lts, 1, pre, x86), JLArrays, OpenCL, Enzyme, QA, Downgrade CPU, Runic and typos. The CPU (Julia 1) log shows "Public generic interfaces 35/35", which includes the new testset. Two checks did not pass, and neither is a regression:
    • CUDA Tests (Julia 1) failed. The error is in test/enzyme/cuda_records.jl:9 (InvalidIRError, jl_f_throw_methoderror in gpu_fill_kernel!). The same error is on master 81186889fb, and the same check also failed on b3d6a17db0 and 4b59cb4147. Pre-existing.
    • Documentation was cancelled. The job hung in ExpandTemplates from 12:03 until it was killed at the 6 h limit, which looks like the runner hanging. On the merge commit 46c4a76165 the doc examples ran all the way through, then failed on :missing_docs ("2 docstrings not included"). Master 5e8faee36c (Fix EnsembleGPUArray scalar parameter packing for batch_size=1 #555) fails the same way, and this PR adds no docstrings. Pre-existing.
    • After the merge, CUDA on 46c4a76165 fails at test/ensemblegpuarray_scalar_batch.jl:95/102 with host-side scalar indexing of a CuArray. That test comes from Fix EnsembleGPUArray scalar parameter packing for batch_size=1 #555, whose own CUDA run was cancelled. It is not caused by this PR.
  • Independent review: Devin CLI (Mac) / fusion-claude-opus-5-5-high-sidekick-swe-2-medium rated it low, high confidence: the fix is a mechanical rename to a local variable, it has a correct analytic regression test that runs green in CPU CI, and every red check on head or on the merge commit also fails on master.
  • Merge: auto-merge candidate. The PR is already merged: @ChrisRackauckas merged it on 2026-09-28 as 46c4a76165. Things to follow up:

🤖 Risk assessment posted by an AI agent (fleet master) — harness: Devin CLI (Mac) / fusion-claude-opus-5-5-high-sidekick-swe-2-medium; dispatched by Claude Code head, model claude-opus-5-5[1m]
Conversation: local Claude Code session 3cd6500a-1f81-46b5-ac0b-c466e15b6a53 on Chris's Mac (session ID, no URL)

ChrisRackauckas and others added 3 commits September 26, 2026 16:07
Reassigning the kernel argument `t` during tspan reparameterization let one
CPU lane's physical time carry into the next; keep physical time in `t_phys`.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
RK4/Tsit5 are exact for u'=p+t; assert solutions against the closed form
so CPU workgroup time leaks cannot silently regress (SciML#551).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
Parameterize the CPU analytic check over IIP/OOP and unequal span lengths,
keep the t_phys rationale in one place, and Runic-format the new assertions.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review September 28, 2026 11:54
@ChrisRackauckas
ChrisRackauckas merged commit 46c4a76 into SciML:master Sep 28, 2026
26 of 31 checks passed
ChrisRackauckas referenced this pull request Sep 29, 2026
Agent-Harness: Devin CLI 3000.11.3 (Mac)
Agent-Model: Fusion (claude-opus-5-5 medium + swe-2 medium)
Agent-Session: local transcript ~/.local/share/devin/cli/transcripts/helpful-exception.json

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EnsembleGPUArray CPU backend: time leaks between lanes with per-trajectory tspan (wrong results)

2 participants