Skip to content

fix: per-trajectory error norm for EnsembleGPUArray - #571

Open
SebastianM-C wants to merge 1 commit into
SciML:masterfrom
SebastianM-C:smc/gpuarray-traj-norm
Open

SebastianM-C wants to merge 1 commit into
SciML:masterfrom
SebastianM-C:smc/gpuarray-traj-norm

Conversation

@SebastianM-C

Copy link
Copy Markdown
Member

EnsembleGPUArray solves all trajectories as one batched problem, and used diffeqgpunorm as its internalnorm: an RMS over all N·ntraj components. A trajectory that needs small steps is outvoted by the easy ones and silently misses its tolerance, by more the larger the batch. The norm was also passed after kwargs..., so a user-supplied internalnorm couldn't override it.

Example: u' = ω cos(ω t) on [0, 10], Tsit5, abstol = reltol = 1e-8, trajectory 1 at ω = 50 and the rest at ω = 1. Trajectory 1's error at t = 10 is 1.6e-9 solved alone and 5.0e-7 inside a batch of 1000, with no warning.

This PR adds TrajectoryNorm(len), which takes the RMS of each trajectory's own components and returns the largest. It handles N×ntraj state arrays and their vectorized form, host and device arrays, and Dual-valued arrays. With it the error above is 1.6e-9 at any batch size. The default is now set before kwargs..., so internalnorm passed to solve replaces it. The EnsembleGPUArray docstring gains a Step-size control section that describes this and the override.

Behaviour change: the shared step is now the one the hardest trajectory needs, so heterogeneous batches take more steps. Lorenz, the EnsembleGPUArray docstring example (Float32, random p, saveat = 1, CUDA):

  • Tsit5, 10_000 trajectories on [0, 100]: naccept 1076 → 2092, nreject 0 → 6
  • Rosenbrock23, 1_000 trajectories on [0, 10]: naccept 265 → 797, nreject 15 → 128

Those extra steps are the cost of every trajectory actually meeting its tolerance. To get the old behaviour back, pass internalnorm = DiffEqGPU.diffeqgpunorm.

Tests: test/ensemblegpuarray_trajectory_norm.jl: the norm on host, device and Dual arrays, and the ω = 50 trajectory meeting its tolerance in batches of 2 and 1000. On an RTX 4080 SUPER (CUDA group): 12/12, and 12/12 in the CPU group. The existing EnsembleGPUArray test files still pass with the new default: ensemblegpuarray.jl 6/6, ensemblegpuarray_scalar_batch.jl 55/55, ensemblegpuarray_sde.jl 1/1, reduction.jl 3/3, and ensemblegpuarray_oop.jl, ensemblegpuarray_inputtypes.jl, lower_level_api.jl run without errors.


🤖 Generated with Claude Code · model: claude-opus-5-5[1m]
Review: unreviewed

The batched solve used `diffeqgpunorm`, an RMS over all N·ntraj components, as
`internalnorm`, and passed it after `kwargs...` so it could not be overridden. A
trajectory that needs small steps is outvoted by the easy ones and silently misses its
tolerance, by more the larger the batch. For u' = ω cos(ω t) on [0, 10] with Tsit5 at
abstol = reltol = 1e-8, trajectory 1 at ω = 50 and the rest at ω = 1, its error at
t = 10 is 1.6e-9 solved alone and 5.0e-7 inside a batch of 1000.

`TrajectoryNorm(len)` takes the RMS of each trajectory's own components and returns the
largest, for N×ntraj state arrays and their vectorized form, on host and device arrays,
and for Dual-valued arrays. With it the error above is 1.6e-9 for any batch size. The
default is set before `kwargs...`, so `internalnorm` passed to `solve` replaces it.

The shared step is now the one the hardest trajectory needs, so batches take more steps.
Lorenz, the EnsembleGPUArray docstring example (Float32, random p, saveat = 1, CUDA):
- Tsit5, 10_000 trajectories on [0, 100]: naccept 1076 -> 2092, nreject 0 -> 6
- Rosenbrock23, 1_000 trajectories on [0, 10]: naccept 265 -> 797, nreject 15 -> 128

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

PR: #571

Reviewed head: 0257163
Raw SHA-256: abd432218e6d7e0494b6c35cb04fa9268db7b3bdb41a8bfedca08aec6f114a21
Source SHA-256: ae4473470048c720dc7e32854ce2555407113afe4618752eeb01f7764606048b
Source: standard-review-v1 (harness-log)
Reviewer session: 01a1d253-f99e-7252-b26e-596585539a0e
Slot: 1
Confidence: unknown
Own risk fixed before author rating/discussion: yes

Risk assessment

  • Risk: medium (M3)
  • Verdict: CHANGES
  • Reviewed head: 0257163

Review report

VERDICT: CHANGES
Round: 1
Head: 0257163
Slot: 1
CI: CI is not green (regress:buildkite/diffeqgpu-dot-jl; buildkite/diffeqgpu-dot-jl/julia-oneapi-run-tests-on-julia-v1; buildkite/diffeqgpu-dot-jl/julia-oneapi-run-tests-on-julia-v1-dot-10)
Session: 01a1d253-f99e-7252-b26e-596585539a0e

Risk

medium (M3: a hot-path error norm changes for both array ensemble implementations without benchmark evidence in the permitted material; ML10 also applies to the changed default numerics).

Blocking findings

  • Verdict: CHANGES
  • Round: 1
  • Blocking priorities: P2, P2
  1. [P2] Real-GPU execution remains unverified (G2). src/utils.jl:30, src/solve.jl:468, src/solve.jl:571, and src/ensemblegpuarray/lowerlevel_solve.jl:103 introduce a dimensional reduction followed by a maximum in the ordinary, lower-level, and pullback solves. The CPU results below pass, but they do not establish device reduction, stiff-solver, or adaptive dual-number compatibility on a GPU. The supplied CI summary does not establish a successful GPU run. Supply current-head CUDA/AMDGPU test output exercising these paths, or run the prepared probes on available hardware. This is a verification blocker, not a confirmed numerical defect.

  2. [P2] The reported oneAPI regressions remain unresolved. .buildkite/runtests.yml:29 defines the two failing oneAPI jobs identified in the supplied CI status. I could not reproduce or classify those failures using this checkout and the available NVIDIA hardware. Establish that the reviewed head has no regression against the default branch, by fixing the failures or providing a controlled baseline comparison. I am not attributing these failures to the norm implementation without evidence. This verification gap is not eligible for P2 deferral.

Non-blocking findings

None confirmed. The added tests retain existing assertions, use an analytic sine reference, and pass on CPU. I found no confirmed code-level correctness defect from inspection.

What I ran

Review produced by Codex CLI 0.164.0, model gpt-6.1-sol; local session identified above (no session URL available). No PR body, comments, reviews, or author risk rating were read.

  • git diff origin/master...HEAD, root instructions, affected source/tests, and the risk criteria; git diff --check origin/master...HEAD passed. The final checkout remained clean at the requested SHA. The newest commit has neither requested agent trailer, so author attribution below is unknown.
  • julia --version: 1.13.2. There is no checked-in Manifest or interpreter lock. I copied the declared test project into scratch/testenv, developed this checkout there, and instantiated a scratch manifest while preserving JULIA_DEPOT_PATH. Resolved versions included OrdinaryDiffEq 7.8.1, SciMLBase 3.58.0, ForwardDiff 1.4.7, and CUDA 6.4.1; listing CUDA is not execution evidence.
  • With the requested TMPDIR, JULIA_NUM_THREADS=4, and JULIA_NUM_PRECOMPILE_TASKS=4: GROUP=CPU timeout 10800 julia --project=../scratch/testenv test/runtests.jl, inside a 14 GiB systemd memory scope. Exit 0; 779 assertions passed. This includes the new per-trajectory norm suite (12/12), scalar batch/pullback suite (55/55), and host concatenation suite (6/6). Full output is in scratch/cpu.log.
  • uv run --locked python --version: Python 3.15.0. uv warned that --locked has no effect outside a Python project; this repository has no uv.lock or Python project environment.
  • nvidia-smi identified two NVIDIA RTX PRO 6000 Blackwell Max-Q cards, both heavily occupied. No CUDA.jl functional test completed.
  • Wrote adversarial probes in scratch/probe_body.jl with a sequential CPU/CUDA driver in scratch/targeted.jl: complex/vectorized norms, NaN/Inf propagation, original-norm negative controls, analytic two-component and stiff solves, custom norm overrides, and an adaptive pullback. These probes did not execute. Their launch gate never simultaneously met the prescribed memory/load/worker limits during the remaining review time. I interrupted the idle waiter before Julia launched (exit 130); no test was interrupted, disabled, or skipped in the checkout. All commands started by this review have finished.

What I did not verify

  • Actual CUDA/AMDGPU/oneAPI execution, CPU Enzyme, or the complete GPU groups.
  • The prepared negative controls, stiff/adaptive-gradient probes, or a failing-before/passing-after execution against the original norm.
  • Performance, allocation overhead, work–precision behavior, QA, documentation build, or Julia 1.10 compatibility.
  • CI logs or the cause of the supplied oneAPI failures. Those statuses are supplied evidence, not locally reproduced failures.

AUTHOR_GRADE: {"task":"bugfix","score":4,"difficulty":3,"rounds":1,"author_harness":"unknown","author_model":"unknown","note":"The full CPU group passes 779 assertions, including all 12 new norm checks. Inspection found no confirmed correctness defect; real-GPU execution, negative controls, and the supplied oneAPI regressions remain unverified."}

Head link: 0257163


🤖 Posted by an AI agent — harness: codex-cli 0.164.0 · model: gpt-6.1-sol
Conversation: local session 01a1d253-f99e-7252-b26e-596585539a0e (no session URL)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants