Skip to content

Master CUDA CI: GPUVern7 callback test compares opposite sides of the final bounce #540

Open

Description

@ChrisRackauckas-Claude

The CUDA workflow on unmodified master fails the two adaptive GPUVern7 bouncing-ball endpoint comparisons in test/gpu_kernel_de/gpu_ode_continuous_callbacks.jl. Both errors are 59.99492f0 < 0.01, with 43 other assertions passing in the testset. This also occurs before the Enzyme changes in #532.

Observed on current master 9008ff6:
https://github.com/SciML/DiffEqGPU.jl/actions/runs/34728066137/job/103645686341

GPU Kernelized Non Stiff ODE ContinuousCallback: Test Failed ...:117
Expression: norm(bench_sol.u[end] - (sol.u[1]).u[end]) < 0.01
Evaluated: 59.99492f0 < 0.01
GPU Kernelized Non Stiff ODE ContinuousCallback: Test Failed ...:135
Expression: norm(bench_sol.u[end] - (sol.u[1]).u[end]) < 0.01
Evaluated: 59.99492f0 < 0.01
Some tests did not pass: 43 passed, 2 failed, 0 errored, 0 broken.

The test ends at t=15, exactly the third impact of the analytic elastic bouncing-ball solution (initial height45, gravity10; impact times3,9,15). Velocity jumps from -30 to +30 there. Comparing the two pointwise endpoint velocities therefore magnifies small event-time differences to an error around60. That observation explains the error scale but does not by itself establish which computed event times are sufficiently accurate. Ending away from the discontinuity still needs an accuracy check against the analytic solution rather than simply raising the assertion threshold.

Regression boundary

The last green master CUDA run used 510232f (Julia1.12.7):
https://github.com/SciML/DiffEqGPU.jl/actions/runs/34287164055/job/102265362561

The next completed master run at c040dd1 fails the same two assertions (Julia1.13.0):
https://github.com/SciML/DiffEqGPU.jl/actions/runs/34685669384

The only repository changes between these revisions add .github/workflows/benchmark.yml and benchmark/benchmarks.jl. Solver and callback-test sources are identical. This bounds the recurrence to environment changes, not an intervening solver commit. It is not a completed dependency/Julia bisect: SciMLBase changed3.53.1→3.53.3, DiffEqBase7.20.1→7.21.1, and OrdinaryDiffEqCore4.17.0→4.17.1 at the same time. CUDA6.3.1, GPUCompiler2.6.0, LLVM9.13.1, and OrdinaryDiffEqVerner2.4.1 stayed unchanged.

This is related to the previously closed report #412. The recurrence should be tracked separately from the Enzyme gradient repair.

Local reproduction on clean master

A detached clean checkout of master reproduces the same qualitative failure with EnsembleGPUKernel(CPU()), so this is not restricted to CUDA or differentiation. No local CUDA driver is available; the CUDA values above are from the actual master CI log.

using DiffEqGPU, OrdinaryDiffEq, StaticArrays, KernelAbstractions, LinearAlgebra

f(u, p, t) = SVector(u[2], -p[1])
condition(u, t, integrator) = u[1]
affect!(integrator) = (integrator.u += SVector(0.0f0, -2.0f0) .* integrator.u)
prob = ODEProblem{false}(
    f, SVector(45.0f0, 0.0f0), (0.0f0, 15.0f0), SVector(10.0f0))
cb = ContinuousCallback(condition, affect!; save_positions = (false, false))
ensemble = EnsembleProblem(prob, safetycopy = false)
sol = solve(ensemble, GPUVern7(), EnsembleGPUKernel(CPU());
    trajectories = 2, adaptive = true, dt = 0.1f0,
    callback = cb, merge_callbacks = true)
ref = solve(prob, Vern7(); adaptive = true, save_everystep = false,
    dt = 0.1f0, callback = cb, merge_callbacks = true)
@show sol.u[1].u[end] ref.u[end] norm(sol.u[1].u[end] - ref.u[end])

Executed with Julia1.13.0, clean master9008ff6, DiffEqBase7.21.1, SciMLBase3.53.3, OrdinaryDiffEq7.8.1, OrdinaryDiffEqCore4.17.2, and OrdinaryDiffEqVerner2.4.1:

GPUVern7 endpoint = Float32[0.006505424, 29.997293]
Vern7 endpoint = Float32[0.00038910535, -29.999973]
error = 59.99727

Local version information:

Julia Version 1.13.0
Commit d1c37793dd2 (2026-09-09 19:00 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 7502 32-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-20.1.8 (ORCJIT, znver2)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 128 virtual cores)
Environment:
  JULIA_LOAD_PATH = @:@stdlib

No assertion thresholds were changed. A solver/dependency bisect beyond the source/environment boundary above remains incomplete; this report does not claim a Julia compiler defect.

Controlled Julia-version comparison of the CPU reproduction

I ran the same script against the same clean master checkout and the same manifest, without resolving or changing dependencies between invocations:

julia +1.12 --project=<same-local-environment> cuda_baseline_callbacks.jl
GPUVern7 endpoint = Float32[0.006505424, 29.997293]
Vern7 endpoint = Float32[0.0017566098, 29.999956]
error = 0.005444354

julia +1.13.0 --project=<same-local-environment> cuda_baseline_callbacks.jl
GPUVern7 endpoint = Float32[0.006505424, 29.997293]
Vern7 endpoint = Float32[0.00038910535, -29.999973]
error = 59.99727

The DiffEqGPU CPU-backend result is identical in these two runs; the OrdinaryDiffEq reference falls on opposite sides of the final impact. This isolates Julia-version sensitivity for the CPU reproduction, but it does not establish a compiler bug or uniquely identify why the separate CUDA CI runs changed. The endpoint discontinuity makes a continuous norm comparison unsuitable for measuring small timing errors there.

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session ID: 01a07fcc-1c4f-7ee3-9a1e-51eaab7df293).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions