Repository navigation
Keep adaptive Tsit5 dense-output state out of registers; skip qold^0 pow in I controller - #566
Merged
ChrisRackauckas merged 1 commit intoOct 2, 2026
Conversation
Endpoint-only adaptive solves (`saveat === nothing`, `save_everystep = false`) write `us[2] = integ.u` after the step loop. The overshoot interpolation `us[end] = integ(tspan[2])` before it targets the same element (the output has two rows), so its result was always overwritten, but it kept k1-k6, uprev and tprev live across every step. Only run it when its result is kept. The I controller has beta2 = 0, so `qold^beta2` is exactly one. Skip that `pow` call instead of evaluating it on every step attempt. Both changes are output-neutral. For the Lorenz ensemble compiled for sm_70 the GPUTsit5IController Float64 kernel goes from 186 to 99 registers and from two `__internal_accurate_pow` calls per attempt to one. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Agent-Harness: Claude Code 2.1.280 Agent-Model: claude-opus-5-5[1m] Agent-Session: https://claude.ai/code/session_01SJfy3ez7u4CqvAerKrT5np
ChrisRackauckas
marked this pull request as ready for review
October 2, 2026 18:59
ChrisRackauckas-Claude
pushed a commit
to ChrisRackauckas-Claude/DiffEqGPU.jl
that referenced
this pull request
Oct 2, 2026
…l-host-fastpath Picks up SciML#566 (dense-output state out of registers) and SciML#568, whose Annotation-typed backend parameter in the _kernel_transfer Enzyme rule supersedes this branch's unannotated-parameter variant; the file is identical to master after resolution. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Devin <noreply@cognition.ai> Agent-Harness: Devin CLI 3000.11.3 Agent-Model: swe-2-high Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/560-devin/log.txt on amdci2.julia.csail.mit.edu
ChrisRackauckas
referenced
this pull request
Oct 14, 2026
Agent-Harness: Devin CLI 3000.11.3 Agent-Model: Fusion (Claude Opus 5.5 Medium + SWE-2 Medium) Agent-Session: local transcript ~/.local/share/devin/cli/transcripts/flash-hour.json Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ignore this draft until reviewed by @ChrisRackauckas.
What changed and why
SciMLBenchmarks' Lorenz work-precision benchmark measures
GPUTsit5IControllerat 1.4–2x the per-trajectory kernel time of GRADSOLVE'scuda_tsit5, even though both take the same step sequence. I compiled both kernels for sm_70 (V100) and compared them. Two causes are contained in DiffEqGPU:ode_asolve_kernelinterpolatesus[end] = integ(tspan[2])ifinteg.t > tspan[2], and then unconditionally writesus[2] = integ.u. Withsaveat === nothingandsave_everystep = falsethe output has two rows, sous[end]isus[2]and the interpolated value is always overwritten. The compiler can't prove that, sok1–k6,uprev,tprevanddtstay live across every step. The interpolation now runs only on the path where its result is kept. This takes the Float64 kernel from 186 to 102 registers.qold^0on every step attempt.beta2 = 0forGPUTsit5IController, butq11 / qold^beta2still emits a second__internal_accurate_powcall (about 139 SASS instructions, 82 of them FP64)._qold_factorreturnsone(qold)for the I controller.pow(x, 0) == 1for everyx, NaN included, so the result is unchanged.Both changes leave the output unchanged. Endpoint and
saveatoutputs ofvectorized_asolveon the KA CPU backend are bitwise identical between master and this branch in all 24 cases I compared: Lorenz, 64 trajectories, both algorithms, Float32 and Float64, rtol 1e-6 and 1e-9, tf of 1 and 7.PTX / ptxas evidence (sm_70, no GPU)
The kernel is
ode_asolve_kernelasvectorized_asolvelaunches it onCUDABackend(prefer_blocks = true), running the benchmark's@muladdLorenz RHS. I compiled it through GPUCompiler withPTXCompilerTarget(cap = v"7.0", ptx = v"7.8"), linked libdevice from CUDA 12.9, and ranptxas -v -arch=sm_70(CUDA 12.9). The "loop" figures count SASS instructions in the main step loop's body only. Called subroutines are excluded, sopowis listed separately.powcalls per attemptFor the F64 I controller, the loop body plus its
powbodies goes from 925 (647 + 2 × 139) to 683 (548 + 135) SASS instructions per attempt (GRADSOLVE: 378 + 128 = 506). FP64-pipe operations go from 427 to 334 (GRADSOLVE: 290). Every variant has a 32-byte stack frame that only the exception-reporting path uses. There is nold.localanywhere, the loop has nost.local, and the tableau is constant-folded intocmem[2], so the integrator is not in local memory, either before or after this change.Register-limited occupancy ceilings, calculated with NVIDIA's
cuda_occupancy.hfor sm_70 (which accounts for the four register-file partitions), scanning block sizes 32–1024: F64 I goes from 8 to 16 resident warps per SM, F64 PI from 8 to 16, F32 I from 20 to 36, and F32 PI from 20 to 36. The figures are the same at 128 threads per block. GRADSOLVE is at 24 (F64) and 48 (F32). These are calculated ceilings, not measured achieved occupancy.GPU speed is unverified
This machine has no NVIDIA GPU. The numbers above are static compiler output, not timings. The expected effect is a large part of the gap for the F64 I controller and a smaller effect for the PI controller, which still needs two
powcalls per step. Someone needs to rerunbenchmarks/DiffEqGPU/lorenz_workprecision.jmdon a GPU to confirm it.Verification
Runic --checkon the changed files: clean.typos: clean._qold_factorfor the I controller is===toqold^zero(T)for finite,floatmax,InfandNaNinputs, which is the property that makes skippingpowexact. This is a performance change, not a bug fix, so there is no failing-before test. The register counts are the discriminating evidence, and CI cannot check them without a GPU.Each run ended with
Testing DiffEqGPU tests passed.Not run: the CUDA group, GPU timing, and the
preJulia channel.What remains of the gap (not in this PR)
These are listed roughly by estimated size. They all change rounding or behaviour, so they belong in separate PRs:
EEst^beta1andqold^beta2with two accuratepowcalls per attempt. Sharing alogbetween them, or computing the factor in Float32 (GRADSOLVE's-DCHEAP_CTRL), would remove most of the rest of the PI gap, but both change the step sizes at the roundoff level.q11 / gamma,q / gamma,dt / q), and Julia's NaN-propagatingmax/minaddDSETP.NAN+FSELpairs.dt < 1e-14 && error("dt<dtmin")costs one compare per attempt plus the exception path (vprintf, trap). It is minor.ImmutableODEProblemarray, and results are written to(2, N)matrices ofSVectors, where GRADSOLVE uses coalesced structure-of-arrays I/O. This happens only once per trajectory, and there are no global loads or stores inside the step loop, so it is a few percent at most.Risk assessment
ode_asolve_kerneland the I-controller history factor. The other saving branches keep their interpolation, and PI controller arithmetic is unchanged.saveatcases above. An independent review (Codex, gpt-6-astra) also found 256 more base/head comparisons that matched exactly, covering callbacks, termination, multipletstops, prefilled buffers and a callback-forced overshoot through the edited branch. The same review reproduced the CPU/OpenCL/Enzyme/QA results and the register and loop counts.🤖 Generated with Claude Code (model: claude-opus-5-5[1m])
https://claude.ai/code/session_01SJfy3ez7u4CqvAerKrT5np