Repository navigation
Fix EnsembleGPUArray CPU time leak with per-trajectory tspan - #553
Merged
ChrisRackauckas merged 4 commits intoSep 28, 2026
Merged
ChrisRackauckas merged 4 commits into
ChrisRackauckas merged 4 commits into
Conversation
Reassigning the kernel argument `t` during tspan reparameterization let one CPU lane's physical time carry into the next; keep physical time in `t_phys`. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Cursor Agent <noreply@cursor.com> Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe Agent-Model: auto Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
RK4/Tsit5 are exact for u'=p+t; assert solutions against the closed form so CPU workgroup time leaks cannot silently regress (SciML#551). Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Cursor Agent <noreply@cursor.com> Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe Agent-Model: auto Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Cursor Agent <noreply@cursor.com> Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe Agent-Model: auto Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
Parameterize the CPU analytic check over IIP/OOP and unequal span lengths, keep the t_phys rationale in one place, and Runic-format the new assertions. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Cursor Agent <noreply@cursor.com> Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe Agent-Model: auto Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
ChrisRackauckas
marked this pull request as ready for review
September 28, 2026 11:54
This was referenced Sep 28, 2026
Merged
ChrisRackauckas
referenced
this pull request
Sep 29, 2026
Agent-Harness: Devin CLI 3000.11.3 (Mac) Agent-Model: Fusion (claude-opus-5-5 medium + swe-2 medium) Agent-Session: local transcript ~/.local/share/devin/cli/transcripts/helpful-exception.json Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EnsembleGPUArrayParamWrapper kernels reassigned the kernel argumenttwhile reparameterizing from normalized time[0,1]to each trajectory's physicaltspan. On the KernelAbstractions CPU backend, that mutated value leaked across lanes in a workgroup, so trajectories sharing a batch but not atspangot wrong right-hand-side times (and wrong solutions). Physical time is now kept in a separate localt_physin all ParamWrappergpu_kernel/jac_kernel/W_kernel/Wt_kernelvariants (IIP and OOP). The CPU analytic regression covers in-place and out-of-place RHS and equal vs unequal span lengths.Verification
Before (master archive), parameterized probe (IIP/OOP × equal/unequal spans)
After fix, same probe on this branch
CPU test suite
Runic
Not verified
tspanwere not run.jac_kernelandjac_kernel_oopcould not be validated on CPU because of a pre-existing__index_Global_LinearMethodError (reproduced on master too). RHS / W / Wt ParamWrapper kernels were checked analytically; the Jacobian helpers only received the samet_physedit by construction.GROUP=CPUruntests.jlwas not completed end-to-end here.Risk assessment
Low. The change is a local rename of the reparameterized time binding so CPU lanes cannot overwrite a shared kernel argument; numerical behavior with identical
tspans is unchanged. CPU evidence covers IIP and OOP RHS with equal and unequal span lengths (16/16 analytic passes on head vs 4/16 on master). Residual risk is limited to GPU backends (not exercised here) and to the two CPU Jacobian helpers that already throw on master before any time transform runs.Reviewer pushback
GROUP in SUPPORTS_LUFACTfor CUDA CI?jac_kernelCPU MethodError so those helpers can be probed?tspans.Closes #551
Please ignore this draft until reviewed by @ChrisRackauckas.
Harness: Cursor Agent CLI 2026.09.26-dd393fe; model: auto; transcript: /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/551-cursor/log.txt on amdci2.julia.csail.mit.edu
Risk assessment
EnsembleGPUArrayusers, through the ParamWrapper kernels insrc/ensemblegpuarray/kernels.jl(gpu_kernel,jac_kernel,W_kernelandWt_kernel, both IIP and OOP). The change renames a local variable: the reparameterized time now goes intot_physinstead of overwriting the kernel argumentt. On the KernelAbstractions CPU backend this fixes wrong results when trajectories in one batch have differenttspans. On GPU backends each thread already has its own copy oft, so behaviour there should be the same as before. No public API changes, so no SemVer impact. No non-public dependency internals are used (remakeis exported by SciMLBase). No tests were weakened; the PR only adds a 16-assertion analytic testset.906d240these pass: CPU (lts, 1, pre, x86), JLArrays, OpenCL, Enzyme, QA, Downgrade CPU, Runic and typos. The CPU (Julia 1) log shows "Public generic interfaces 35/35", which includes the new testset. Two checks did not pass, and neither is a regression:CUDA Tests (Julia 1)failed. The error is intest/enzyme/cuda_records.jl:9(InvalidIRError,jl_f_throw_methoderroringpu_fill_kernel!). The same error is on master81186889fb, and the same check also failed onb3d6a17db0and4b59cb4147. Pre-existing.Documentationwas cancelled. The job hung in ExpandTemplates from 12:03 until it was killed at the 6 h limit, which looks like the runner hanging. On the merge commit46c4a76165the doc examples ran all the way through, then failed on:missing_docs("2 docstrings not included"). Master5e8faee36c(Fix EnsembleGPUArray scalar parameter packing for batch_size=1 #555) fails the same way, and this PR adds no docstrings. Pre-existing.46c4a76165fails attest/ensemblegpuarray_scalar_batch.jl:95/102with host-side scalar indexing of aCuArray. That test comes from Fix EnsembleGPUArray scalar parameter packing for batch_size=1 #555, whose own CUDA run was cancelled. It is not caused by this PR.46c4a76165. Things to follow up:🤖 Risk assessment posted by an AI agent (fleet master) — harness: Devin CLI (Mac) / fusion-claude-opus-5-5-high-sidekick-swe-2-medium; dispatched by Claude Code head, model claude-opus-5-5[1m]
Conversation: local Claude Code session 3cd6500a-1f81-46b5-ac0b-c466e15b6a53 on Chris's Mac (session ID, no URL)