Skip to content

Build EnsembleGPUArray batch matrices in one allocation - #550

Closed
ChrisRackauckas-Claude wants to merge 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:perf/ensemblegpuarray-hcat
Closed

ChrisRackauckas-Claude wants to merge 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:perf/ensemblegpuarray-hcat

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

batch_solve in src/solve.jl built each EnsembleGPUArray batch with reduce(hcat, generator). On a generator that is a left fold, so every added trajectory copies the matrix built so far and a batch of n trajectories allocates Θ(n²). Base.reduce(hcat, ::AbstractVector{<:AbstractVecOrMat}) allocates once, so the columns are collected into a Vector first (src/solve.jl:324 for u0, and the same helper for p). A one-column batch still returns that column. Parameters that are not arrays (scalars and ParamWrapper) are placed with reshape(cols, 1, :): reduce(hcat, ::Vector{<:Number}) is only the pairwise reducer and still allocated about 2.7 GiB at n = 2^20.

CPU only, Julia 1.11.9, KernelAbstractions.CPU(), EnsembleGPUArray(backend, 0.0), Float32 Lorenz, RK4, dt = 0.001, save_everystep = false. One solve.

Before, DiffEqGPU v3.21.2:

n prob_func u0 hcat p hcat ODE total
2^18 1.57 s 104 s 17.4 s 12.6 s 135 s
2^20 6.48 s 1236 s 273 s 54.1 s 1570 s

u0 allocation scaled exactly ×16 when n ×4 (384 GiB at 2^18, 6144 GiB at 2^20). Log-log exponent of the u0 concatenation was 2.08.

After this change:

n prob_func u0 hcat p hcat full solve
2^18 1.55 s 0.024 s 0.021 s 15.3 s
2^20 6.50 s 0.116 s 0.116 s 65.2 s

The full solve is linear (exponent 1.04 across 2^14…2^20). The remaining time is the ODE.

Same script, one solve at n = 2^18, separate environments. The fold is unchanged across these releases. It was introduced in 16997f1a (2020-08-22, “asynchronous mixing of CPU with GPU”). v3.10 calls prob_func(prob, i, repeat); v3.14+ passes an EnsembleContext. Allocations match.

DiffEqGPU seconds allocated
3.10.0 138.6 512.6 GiB
3.15.0 128.6 512.6 GiB
3.18.0 127.0 512.6 GiB
3.20.0 128.5 512.6 GiB
3.21.2 128.7 512.6 GiB

test/ensemblegpuarray_hcat.jl checks that allocations grow by less than 8× when the trajectory count grows 4× (n = 1024 vs 4096).

On master (git show HEAD:src/solve.jl restored, test file present):

EnsembleGPUArray concatenation allocation: Test Failed at test/ensemblegpuarray_hcat.jl:59
  Expression: b12 / b10 < 8
   Evaluated: 13.345480114788382 < 8
Test Summary:                             | Pass  Fail  Total   Time
EnsembleGPUArray concatenation allocation |    5     1      6  15.1s
ERROR: LoadError: Some tests did not pass: 5 passed, 1 failed, 0 errored, 0 broken.

With this commit:

Test Summary:                             | Pass  Total   Time
EnsembleGPUArray concatenation allocation |    6      6  14.0s

GROUP=CPU ~/.juliaup/bin/julia +1.11 --project=. -e 'using Pkg; Pkg.test()' on this commit: Testing DiffEqGPU tests passed. Public generic interfaces 19/19, EnsembleGPUArray host concatenation 6/6.

Not verified: CUDA or any GPU run.

Found from SciMLBenchmarks benchmarks/DiffEqGPU/lorenz_ensemble.jmd on a V100, where EnsembleGPUArray at 2^21 trajectories took about 7.2 h: https://github.com/SciML/SciMLBenchmarks.jl/actions/runs/35832799318

Ignore this draft until reviewed by @ChrisRackauckas.

Risk assessment

  • Risk: medium
  • Blast radius: EnsembleGPUArray batch construction for every backend. The column matrices are the same shapes as before, including a one-trajectory batch. No public API change. Downstream packages that call solve on EnsembleGPUArray are the callers.
  • Evidence: allocation ratio 13.35 before and the test passing (6/6) after; CPU phase timings above; GROUP=CPU passed. No GPU run.
  • Independent review: none yet
  • Merge: needs human review — GPU execution was not run

🤖 Generated with Grok Build (model: grok-4.7)

Session: /home/crackauc/.grok/sessions/%2Fhome%2Fcrackauc%2Fsandbox%2Ftmp_20260923_091211_54765%2Fgrok-ensemblegpuarray/01a0d6f0-b9fb-79e2-9044-9bcfa0e598c9

Risk assessment

  • Risk: low

  • Blast radius: Every EnsembleGPUArray solve, on all backends, runs through batch_solve in src/solve.jl. The change covers how the u0 matrix is built, plus the p matrix in both branches (the ParamWrapper branch for differing tspans and the plain-p branch). The new _hcat_batch returns the same results as the old reduce(hcat, generator) fold: the same matrix shapes, a one-trajectory batch still returns its bare column or scalar, and non-array p becomes a 1×n row. There are two small differences, both on inputs that are already broken. A batch whose p mixes arrays and non-arrays is now routed by cols[1] isa AbstractArray. An empty batch still throws, now as a BoundsError. Public API does not change and no SemVer bump is needed. The only calls are Base.reduce(hcat, ::AbstractVector) and reshape, which are public Base, so no dependency internals are used. No existing test is weakened. One test file is added, test/ensemblegpuarray_hcat.jl, wired into GROUP=CPU. It checks that the GPU-array and serial results agree, that a batch_size = 1 batch works, and that allocation grows by less than 8× when trajectories grow 4×. The expected ratio is about 4 and master measures 13.3, so the margin against flakiness is reasonable.

  • Evidence: On head 3d7256a, 3 jobs fail (plus the Buildkite rollup), and each fails the same way on the master commits b3d6a17 and 4b59cb4, so all are pre-existing:

    • CUDA Tests (Julia 1) fails with InvalidIRError in test/enzyme/cuda_records.jl:9 ("Allocate records"), so the "Enzyme ensemble gradients" testset errors. The master run shows the same error, and the non-Enzyme CUDA testsets passed on head.
    • buildkite/.../julia-oneapi-run-tests-on-julia-v1 and -v1-dot-10 both error at "Initialization preprocessing runs once" in test/gpu_kernel_de/conversions.jl:82 (15 passed, 1 errored). Master build 1714 has the same error.
    • buildkite/diffeqgpu-dot-jl is only the rollup of those two oneAPI jobs.

    Everything else passes: CPU (1, lts, pre, x86), Downgrade CPU, JLArrays, OpenCL, Enzyme, QA, Runic, typos, Documentation, benchmark, and AMDGPU and Metal on both 1 and 1.10. In the CPU Julia 1 log the new testset ran and passed: EnsembleGPUArray host concatenation | 6 6. No regressions.

  • Independent review: Devin CLI (Mac) / fusion-claude-opus-5-5-high-sidekick-swe-2-medium rated it low, high confidence: the change is 13 lines, a like-for-like swap of the Θ(n²) reduce(hcat, generator) fold for a single-allocation reduce(hcat, ::Vector) / reshape that gives the same results, and every CI failure matches master.

  • Merge: auto-merge candidate. There are no test regressions and no code findings. Three things for whoever merges:


🤖 Risk assessment posted by an AI agent (fleet master) — harness: Devin CLI (Mac) / fusion-claude-opus-5-5-high-sidekick-swe-2-medium; dispatched by Claude Code head, model claude-opus-5-5[1m]
Conversation: local Claude Code session 3cd6500a-1f81-46b5-ac0b-c466e15b6a53 on Chris's Mac (session ID, no URL)

Collect state and parameter columns into a vector before reduce(hcat, ...).
Base allocates that reduction once; a generator left-folds and copies Θ(n²).
A one-column batch still returns that column. Non-array parameters become one row.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Grok <noreply@x.ai>
Agent-Harness: Grok Build 1.0.41
Agent-Model: grok-4.7
Agent-Session: /home/crackauc/.grok/sessions/%2Fhome%2Fcrackauc%2Fsandbox%2Ftmp_20260923_091211_54765%2Fgrok-ensemblegpuarray/01a0d6f0-b9fb-79e2-9044-9bcfa0e598c9
ChrisRackauckas-Claude pushed a commit to ChrisRackauckas-Claude/DiffEqGPU.jl that referenced this pull request Sep 27, 2026
Stack on SciML#550's `_hcat_batch`: keep ordinary scalars as a 1-D vector
(including n = 1) instead of a 1×n matrix or bare Number, and select
`p[i]` vs `p[:, i]` via `ensemble_param` so length-1 arrays stay arrays.
Apply the same packing in the lower-level array path.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/552-cursor/log.txt on amdci2.julia.csail.mit.edu
ChrisRackauckas added a commit that referenced this pull request Sep 27, 2026
* Build EnsembleGPUArray batch matrices in one allocation

Collect state and parameter columns into a vector before reduce(hcat, ...).
Base allocates that reduction once; a generator left-folds and copies Θ(n²).
A one-column batch still returns that column. Non-array parameters become one row.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Grok <noreply@x.ai>
Agent-Harness: Grok Build 1.0.41
Agent-Model: grok-4.7
Agent-Session: /home/crackauc/.grok/sessions/%2Fhome%2Fcrackauc%2Fsandbox%2Ftmp_20260923_091211_54765%2Fgrok-ensemblegpuarray/01a0d6f0-b9fb-79e2-9044-9bcfa0e598c9

* Preserve scalar EnsembleGPUArray parameter contract through kernels.

Stack on #550's `_hcat_batch`: keep ordinary scalars as a 1-D vector
(including n = 1) instead of a 1×n matrix or bare Number, and select
`p[i]` vs `p[:, i]` via `ensemble_param` so length-1 arrays stay arrays.
Apply the same packing in the lower-level array path.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/552-cursor/log.txt on amdci2.julia.csail.mit.edu

* Test EnsembleGPUArray scalar RHS for batch_size 1, 2, and 4.

Use the issue's unchanged `p * u[1]` scalar RHS plus vector and inplace
cases with analytic endpoints; import Tsit5 from OrdinaryDiffEq.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/552-cursor/log.txt on amdci2.julia.csail.mit.edu

* Keep singleton array parameters as n×1 matrices for EnsembleGPUArray.

`_hcat_batch` returned a bare vector for n=1, which `ensemble_param` treated
as a scalar batch and truncated. Pack ordinary array `p` via `hcat`/`reduce`
so columns stay matrices, and restore lower-level `u0` to `hcat` for n×1 shape.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/552-cursor/log.txt on amdci2.julia.csail.mit.edu

* Regress EnsembleGPUArray vector p with typed RHS and singleton remainder.

Cover sum(p) endpoints for batch_size 1–4, length-1 AbstractVector RHS, and
lower-level singleton state shape (1,1), while keeping the scalar issue RHS.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/552-cursor/log.txt on amdci2.julia.csail.mit.edu

* Support scalar vector packs in batch_solve_up reverse-mode seeding.

`seed_duals` now accepts the 1-D scalar layout from `pack_ordinary_parameters`,
keeping Dual parameters as scalars at the RHS, and the pullback returns a
vector cotangent instead of forcing a matrix.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/552-cursor/log.txt on amdci2.julia.csail.mit.edu

* Test batch_solve_up rrule for scalar and array parameter packs.

Analytic ∂u/∂p = exp(p) for scalar batches (including singleton) and
exp(sum(p)) components for matrix packs, via the package ChainRules rule.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/DiffEqGPU.jl/jobs/552-cursor/log.txt on amdci2.julia.csail.mit.edu

---------

Co-authored-by: ChrisRackauckas-Claude <accounts@chrisrackauckas.com>
Co-authored-by: Grok <noreply@x.ai>
Co-authored-by: Cursor Agent <noreply@cursor.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Landed on master through #555, which was stacked on this branch (commit 5e8faee contains _hcat_batch in src/solve.jl). Closing because nothing is left to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants