Skip to content

Support DynamicalODEFunction and SecondOrderODEProblem in EnsembleGPUArray - #592

Open
sasmith-14 wants to merge 2 commits into
SciML:masterfrom
sasmith-14:dynamical-ode-function
Open

sasmith-14 wants to merge 2 commits into
SciML:masterfrom
sasmith-14:dynamical-ode-function

Conversation

@sasmith-14

Copy link
Copy Markdown

Checklist

  • Appropriate tests were added
  • Any code changes were done in a way that does not break public API
  • All documentation related to code changes were updated
  • The new code follows the
    contributor guidelines, in particular the SciML Style Guide and
    COLPRAC.
  • Any new documentation only uses public API

Additional context

fixes #109

DynamicalODEProblem and SecondOrderODEProblem fail with EnsembleGPUArray and EnsembleCPUArray, because generate_problem reads prob.f.f and a DynamicalODEFunction only has f1 and f2:

using DiffEqGPU, OrdinaryDiffEq
prob = DynamicalODEProblem{false}((v, x, p, t) -> -x, (v, x, p, t) -> v, [1f0], [0f0], (0f0, 1f0))
solve(EnsembleProblem(prob), Tsit5(), EnsembleCPUArray(); trajectories = 4)
# master: FieldError: type SciMLBase.DynamicalODEFunction has no field `f`
# this PR: sol.u[1].u[end] == Float32[0.54030234, 0.841471]  (cos(1), sin(1))

each column of the batched state is one trajectory's ArrayPartition(v, u) flattened to [v; u], so this wraps f1/f2 in a small callable that splits the column and calls both (in-place and out-of-place). other problems still go through prob.f.f.

tests: a new CPU testset with in-place and out-of-place DynamicalODEProblems and a SecondOrderODEProblem, checked against the exact solution and EnsembleSerial. it errors on master and passes with the change. the full GROUP=CPU suite and the QA group pass locally. i also ran the in-place case with EnsembleGPUArray(MetalBackend()) on an M-series mac, and it matches EnsembleSerial.

claude helped me write and test this; I reviewed the changes.

notes:

  • results come back as flat [v; u] vectors like other array-ensemble results, not as ArrayPartitions. happy to change that if you prefer.
  • SplitODEFunction isn't covered here.
  • unrelated thing i hit: broadcasting (du .= ...) inside an in-place f fails on the CPU backend on master (no method matching elsize(::Type{Base.Experimental.Const{Float32, 2}}) from the aliasing check), so the test uses loops.

`generate_problem` read `prob.f.f`, which a `DynamicalODEFunction` doesn't
have, so `DynamicalODEProblem` and `SecondOrderODEProblem` failed with the
array ensembles. Split each batched `[v; u]` column and call `f1` and `f2`.

Fixes SciML#109
@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Independent review (devin / fusion-gpt-6-1-sol-high-sidekick-swe-2-medium), reviewer session unknown, reviewed head 6aa6606

  • Verdict: CHANGES
  • Risk assessment: low (L6: fixes a missing-field error for dynamical problems; ordinary ODE dispatch and numerical algorithms remain unchanged). The CUDA defect below is a merge blocker independently of blast-radius classification.
  • Round / slot: 1 / 1

This is a blind code-only review by a session other than the PR author's, copied below from the reviewer's report with only the grading metadata removed. The posting agent did not re-run the reviewer's commands. A MERGE verdict is advice; merging is left to the maintainers.

CI: green
Session: unknown

Risk

low (L6: fixes a missing-field error for dynamical problems; ordinary ODE dispatch and numerical algorithms remain unchanged). The CUDA defect below is a merge blocker independently of blast-radius classification.

Blocking findings

  • [P2] src/ensemblegpuarray/problem_generation.jl:45-48: The adapter's generic vcat prevents a GPU-compatible out-of-place RHS from compiling when a component returns a state view. Confirmed by running, not merely inferred: an allocation-free acceleration returning SVector(-p[1]*x[1], -p[1]*x[2]) and dx(v,x,p,t) = v passes on EnsembleCPUArray, but fails on EnsembleGPUArray(CUDA.CUDABackend(), 0.0) with InvalidIRError. The compiler trace identifies _typed_vcat! / typed_vcat / vcat at this added line, including unsupported runtime calls and allocations. SecondOrderODEProblem{false} fails identically because its built-in second component returns v. Both use static initial states and allocation-free user functions; this is not a user-RHS allocation violation. Controls using the same acceleration with both components returning SVector, and the in-place version, each pass all 9 assertions on the same CUDA device. Evaluate and copy the two derivative components directly into their output slices, or otherwise concatenate without a generic allocating array operation. Add a CUDA regression for out-of-place SecondOrderODEProblem and a component returning its input. The added regression runs only under GROUP=CPU (test/runtests.jl:135-147) and explicitly selects EnsembleCPUArray (test/ensemblegpuarray_dynamical.jl:43), so green CUDA CI does not exercise this case. Evidence: scratch/logs/probe_secondorder_cuda.log, scratch/logs/probe_oop_view_cuda.log, and passing control logs.

  • Verdict: CHANGES

  • Round: 1

  • Blocking priorities: P2

  • Deferred P2 issues: None supplied.

Non-blocking findings

None. The new analytic oscillator oracle is independent of the implementation; no assertions were deleted, skipped, or loosened in the diff. No dependency, licensing, public-export, controller, or stepper-inlining changes appear.

What I ran

All paths below are relative to the reviewed checkout unless stated otherwise. Exported TMPDIR=(a local scratch path) before commands; preserved the inherited layered JULIA_DEPOT_PATH. Scratch files and resolved environments remain outside the checkout.

  • git diff origin/master...HEAD, surrounding source/tests/instructions, git diff --check origin/master...HEAD, and git status --porcelain: diff check passes; checkout stays clean. Base and merge base are d896a5295b7e3cef94bfa0672444451dd31fc65c.
  • Julia reports julia version 1.13.2. There is no committed Julia Manifest or interpreter lock. Created ../scratch/env from the existing test/Project.toml, developed the checkout there, and instantiated successfully. Captured resolution: scratch/logs/Manifest.head.toml (SciMLBase 3.58.0, OrdinaryDiffEq 7.8.1, CUDA 6.4.1, KernelAbstractions 0.9.46).
  • julia --project=../scratch/env --check-bounds=auto test/ensemblegpuarray_dynamical.jl: exit 0; 24/24 assertions pass across the three new cases. scratch/logs/test_dynamical_head.log.
  • Archived origin/master into ../scratch/base, repointed a separate scratch environment to it, and ran a copy of HEAD's new test: exit 1, FieldError: type SciMLBase.DynamicalODEFunction has no field f in base generate_problem:32. This establishes fail-before/pass-after for the CPU regression. scratch/logs/test_dynamical_base.log.
  • GROUP=CPU julia --project=../scratch/env --check-bounds=auto test/runtests.jl: exit 0, including the new 24 assertions and existing scalar-parameter, concatenation, implicit-method, and kernel tests. Complete output: scratch/logs/group_cpu.log.
  • julia --project=../scratch/env --check-bounds=auto ../scratch/dynamical_gpu_probe.jl <mode> <device> for each mode inplace, oop_static, oop_view, secondorder, and each device cpu, cuda: all four CPU cases pass 9/9; CUDA inplace and oop_static pass 9/9; CUDA oop_view and secondorder exit 1 with the compiler error quoted above. CUDA device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition. Outputs: scratch/logs/probe_<mode>_<device>.log.
  • julia --project=../scratch/env --check-bounds=auto ../scratch/dynamical_partition_probe.jl <device> for cpu and cuda: each passes 42/42. Independent polynomial integrals exercise unequal partition lengths, scalar per-trajectory parameters, nonzero initial time, in-place/out-of-place functions, and batch sizes 1, 2, 3.
  • uv run --locked python --version: exit 0, prints Python 3.15.0 and warning: --locked has no effect when used outside of a project. This Julia checkout has neither uv.lock nor pyproject.toml; the result is uv interpreter discovery, not a locked project Python. scratch/logs/uv.log.

What I did not verify

  • CI status is the supplied green metadata, not an independently inspected CI run. No PR body, comments, reviews, author risk rating, or sibling review artifacts were read.
  • No locked project Python interpreter exists here. Julia dependencies were resolved in scratch and retained, not reproduced from an author-provided lock.
  • Did not run the full CUDA group, other GPU backends, Julia 1.10/1.12, QA, docs, or differentiation suites. The CPU runner's ForwardDiff testset reports zero assertions; its successful exit is not gradient-validation evidence.
  • Did not test matrix-valued partitions, dynamical Jacobian/time-gradient metadata, mass matrices, or callbacks. No additional defects are claimed for these unverified surfaces.
  • No edits inside the checkout, fixes, commits, pushes, GitHub comments, issues, or merge actions. All review-started commands have finished.

🤖 Posted by an AI agent — harness: Claude Code 2.1.322 · model: claude-sonnet-5-5
Conversation: local session 3878b54e-28f2-4a01-a176-a8f5468d1997 (no session URL)

Concatenating the two halves with vcat is generic array code, which doesn't compile for GPUs when a half returns a view of the state (e.g. dx(v, x, p, t) = v, and every out-of-place SecondOrderODEProblem). Write each half into its slice of du instead, through the in-place kernel, and run the DynamicalODEFunction tests on the CUDA, AMDGPU and Metal groups too, with allocation-free out-of-place cases.
@sasmith-14

Copy link
Copy Markdown
Author

thanks for the review! fixed in b72a36d: the out-of-place halves are now written into their slices of du (through the in-place kernel) instead of being vcated, so a half that returns a view of the state compiles on GPUs.

the dynamical tests now also run in the CUDA, AMDGPU and Metal groups, with out-of-place DynamicalODEProblem and SecondOrderODEProblem on static states where dx returns v (plus the in-place case). the broadcasting versions allocate, so they stay CPU-only.

on Metal (M5), those out-of-place cases fail with the same InvalidIRError before the change and pass after (24/24). GROUP=CPU passes (4206 tests), and QA/JET pass. i don't have an NVIDIA or AMD GPU, so the CUDA and AMDGPU groups are untested locally.

@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

PR: #592

Reviewed head: b72a36d
Raw SHA-256: 3b52b49d0e0cea9e383e0906f3a91cda8b01b7d1ce645d4d141621a161c3acfb
Source SHA-256: b3c6d51d6d634f52144d653b52ca6080b5591d6a371e945aeddf2c2f08e1de87
Source: standard-review-v1 (harness-log)
Reviewer session: 01a1d233-6297-7bb3-8570-cc0337ac22d3
Slot: 1
Confidence: unknown
Own risk fixed before author rating/discussion: yes

Risk assessment

  • Risk: low (L6)
  • Verdict: MERGE
  • Reviewed head: b72a36d

Review report

VERDICT: MERGE
Round: 2
Head: b72a36d
Slot: 1
CI: CI is not green
Session: 01a1d233-6297-7bb3-8570-cc0337ac22d3

Risk

low (L6: fixes errors on dynamical-problem paths, with failing-before/passing-after evidence; ordinary ODE dispatch and numerical algorithms remain unchanged).

Blocking findings

None. The round-one blocker is resolved: both allocation-free out-of-place reproducers now compile on CUDA and match independent analytic solutions. The delta adds GPU regression coverage without deleting, skipping, or loosening assertions.

  • Verdict: MERGE
  • Round: 2
  • Blocking priorities: none
  • Deferred P2 issues: None supplied; no remaining blockers.

Non-blocking findings

None found in the delta or unresolved prior findings.

What I ran

AI reviewer: Codex CLI 0.164.0, model gpt-6.1-sol. Reviewed the supplied previous-head review, delta, whole PR diff, surrounding source, tests, and root instructions. No author risk rating was read.

Exported the requested TMPDIR, preserved the inherited layered JULIA_DEPOT_PATH, and kept scripts/environments/logs in the designated scratch directory. Julia reports julia version 1.13.2. Created scratch/env from test/Project.toml, developed the checkout, and instantiated: SciMLBase 3.58.0, OrdinaryDiffEq 7.8.1, CUDA 6.4.1, KernelAbstractions 0.9.46. All Julia tests used --check-bounds=auto and the scratch environment, with four threads and memory scopes.

  • git diff --check origin/master...HEAD and typos on the three changed files: pass. Supplied delta matches git diff 6aa660617c5f00ca5c4528f3315309ee30b1381a HEAD; checkout remains clean.
  • GROUP=CPU julia --project=../scratch/env --check-bounds=auto test/runtests.jl: exit 0, including all 40 dynamical assertions. Standalone test/ensemblegpuarray_dynamical.jl also passes 40/40. Log: group_cpu.log.
  • GROUP=CUDA julia --project=../scratch/env --check-bounds=auto ../scratch/cuda_array_tests.jl "$PWD": CUDA functional check and minimal device arithmetic pass; new dynamical suite passes 24/24. The subsequent existing array suite stops in ROBER at test/ensemblegpuarray.jl:220 with GPU memory exhaustion during cublasCreate_v2, after five passing assertions. Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q; the inference server occupies most device memory. Log: cuda_array.log.
  • view_probe.jl dynamical and view_probe.jl secondorder, on CUDA: each exits 1 with InvalidIRError through vcat on archived previous head 6aa660617c5f00ca5c4528f3315309ee30b1381a; each exits 0 with 8/8 analytic assertions on this head. Identical dependency manifests differ only in the checkout path. Logs: view_{before,head}_{dynamical,secondorder}.log.
  • partition_probe.jl cpu and partition_probe.jl cuda: each passes 186/186 assertions. Exercises unequal partitions, both input-view derivatives, in-place/out-of-place functions, Float32/Float64, scalar parameters, actual singleton batches, reverse integration, and differing trajectory time spans against analytic integrals. Reviewer draft probes first exposed invalid immutable in-place state and an undefined loop variable; both setup errors were corrected before these successful runs.
  • GROUP=CUDA ... test/ensemblegpuarray_scalar_batch.jl: exit 0, 55/55 assertions. Log: scalar_cuda.log.
  • uv run --locked python --version: exit 0, Python 3.15.0, with warning that --locked has no effect outside a project.

What I did not verify

  • CI is supplied as not green; no GitHub CI results, PR body, comments, author rating, or same-head review artifacts were inspected.
  • No committed Julia Manifest, uv.lock, or Python project exists. Dependencies were resolved in scratch; a locked project interpreter/environment was unavailable.
  • Full CUDA suite and remaining legacy array tests after the memory error; AMDGPU, Metal, other backends, older Julia versions, QA, docs, or dedicated differentiation suites. The CPU ForwardDiff testset reports zero assertions and is not gradient-validation evidence.
  • No checkout edits, fixes, commits, pushes, or GitHub writes. All review-started commands have finished.

AUTHOR_GRADE: {"task": "bugfix", "score": 5, "difficulty": 3, "rounds": 2, "author_harness": "unknown", "author_model": "unknown", "note": "The delta fixes both independently reproduced CUDA compilation failures and adds backend regression coverage. The full CPU group, 24 CUDA regression assertions, 186 targeted assertions on each of CPU and CUDA, and 55 CUDA ordinary-ODE controls pass. Broader CUDA verification is limited by device memory. The newest commit has no Agent-Harness or Agent-Model trailers."}

Head link: b72a36d


🤖 Posted by an AI agent — harness: codex-cli 0.164.0 · model: gpt-6.1-sol
Conversation: local session 01a1d233-6297-7bb3-8570-cc0337ac22d3 (no session URL)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support DynamicalODEFunction

2 participants