Skip to content

Require OrdinaryDiffEqCore 4.18.1 for adaptive reference tests - #591

Draft
ChrisRackauckas wants to merge 1 commit into
masterfrom
fix/downgrade-vern9
Draft

ChrisRackauckas wants to merge 1 commit into
masterfrom
fix/downgrade-vern9

Conversation

@ChrisRackauckas

@ChrisRackauckas ChrisRackauckas commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Filed by an AI agent running as @ChrisRackauckas; Chris has not seen this.
Harness: Codex CLI 0.164.0 · Model: gpt-6-astra
Conversation: local job diffeqgpu-downgrade-vern9 (no session URL)

Require OrdinaryDiffEqCore 4.18.1 in the test environment so the adaptive Float32 reference solves use the upstream clock-rounding fix. The existing GPUVern9 regression fails with the older reference solver because its state advances by a step length that differs from the rounded floating-point time increment. This changes only the test dependency floor; OrdinaryDiffEqCore is an existing transitive test dependency licensed under MIT.

Please leave this draft pending review by @ChrisRackauckas.

Reproduction and version boundary

The failure in https://github.com/SciML/DiffEqGPU.jl/actions/runs/41962545816 reproduces exactly on Julia 1.10.13 with -C znver3, the CI package versions, and the unchanged assertion in test/gpu_kernel_de/gpu_ode_regression.jl:46:

Expression: norm(bench_asol.u - (asol.u[1]).u) < 0.001
Evaluated: 0.0010757452f0 < 0.001
GPU Kernelized Non Stiff ODE Regression | 49 passed, 1 failed

The workflow's matrix label says alldeps, but it passes no mode to the reusable workflow, which uses deps. Reproduced the exact action at julia-actions/julia-downgrade-compat@30b83cf with merged root/test resolution, stdlib exclusions, removal of the secondary test manifest, and Pkg.test(allow_reresolve=false) environment construction. Every registry package version matched the failing CI log.

OrdinaryDiffEqCore DiffEqBase Existing regression GPUVern9 discrepancy
4.17.4 (original downgrade) 7.19.0 49 passed, 1 failed 0.0010757452
4.18.0 7.21.2 49 passed, 1 failed 0.0010757452
4.18.1 7.21.2 50 passed 0.0007073194
4.18.0 + isolated upstream clock fix 7.21.2 50 passed 0.0007073194

The adjacent-version pair differs only in Core. DiffEqBase 7.21.2 is required by both versions. The clock-only control isolates SciML/OrdinaryDiffEq.jl@498c765. GPUVern9's endpoint is bitwise unchanged across the pair. Against a Float64 Vern9 reference at abstol=reltol=1e-13, the CPU reference error falls from 4.36616e-4 to 7.05399e-5, while GPUVern9's error stays 6.44729e-4. The test tolerance and assertions remain unchanged.

The original environment passes on native Zen 5 and generic targets, so -C znver3 is necessary to reproduce this exact failure locally.

Validation

All numerical tests use the repository's existing assertions. CPU groups ran through the official test/runtests.jl in Pkg.test-captured environments, with --startup-file=no --compiled-modules=no and GROUP=CPU:

  • Julia 1.10.13, fresh downgrade resolution, -C znver3: full CPU group passed; affected testset 50/50.
  • Julia LTS 1.10.12, current dependencies, -C znver3: full CPU group passed; affected testset 50/50.
  • Julia release 1.13.2, current dependencies: full CPU group passed; affected testset 50/50. Recapturing the environment with the final test constraint produced an identical dependency graph.
  • julia +release --project=<Runic environment> -m Runic --check src ext test: passed.
  • typos --config .typos.toml <git diff> and git diff --check: passed.
  • JULIA_NUM_THREADS=2 JULIA_NUM_PRECOMPILE_TASKS=2 GROUP=QA julia +lts --startup-file=no --project -e 'using Pkg; Pkg.test(; julia_args=["--check-bounds=auto"])': passed, Quality Assurance 25/25 and JET static analysis 75/75 (exit 0).

QA required completing cold dependency precompilation first. An initial four-task attempt was OOM-killed. Two-task attempts reported 24 passes and one Aqua persistent-task timeout while ModelingToolkitBase was still compiling. Pkg.precompile(["ModelingToolkitBase", "DiffEqGPU"]; strict=true) completed the three pending dependencies in 89 seconds (ModelingToolkitBase alone: 83.5 seconds). The unchanged full QA rerun then passed with normal compiled modules; Aqua's 60-second limit and all assertions remain unchanged.

The CPU-only scope does not exercise GPU hardware. Documentation building is not applicable to this test-environment metadata change. The numerical test processes used --compiled-modules=no; QA also runs its explicit persistent-task precompilation check. Assertions were neither skipped nor weakened.

Risk assessment

  • Risk: medium-low (ML4: raising a compat lower bound, the higher independent rating). The author assessed low under L3 because the change affects only test-environment compatibility.
  • Blast radius: test dependency resolution; the package's runtime dependencies, public API and solver code are unchanged. The new joint downgrade floor resolves Core 4.18.1 and DiffEqBase 7.21.2; the workflow's test sandbox selects Core 4.19.1. The explicit adjacent-version runs above also test the first supported Core release itself.
  • Evidence: exact CI failure reproduced; adjacent 4.18.0/4.18.1 fail/pass; clock-only causal control; final validation above.
  • Independent review: Codex CLI 0.164.0 / gpt-6.1-sol rated it medium-low (ML4), high confidence, MERGE-WITH-ISSUES on head 90a1cbc61d1e36e6636e1ee440c40a20fa61820d: no blocking correctness findings; CI and cold-cache QA remain validation limits. The reviewer recorded its rating before reading the author's. Review summary. No commits follow the review.
  • Merge: needs human review; this remains a draft awaiting @ChrisRackauckas, and exact-head CI is pending. This worker does not merge.

🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-downgrade-vern9 (no session URL)

The Float32 Vern9 reference needs the adaptive clock-rounding fix first
released in OrdinaryDiffEqCore 4.18.1. On a Zen 3 target, Core 4.18.0
fails the existing GPUVern9 comparison at 0.0010757452; 4.18.1 passes
at 0.0007073194 with an identical GPUVern9 result. An isolated clock-only
control reproduces the pass. Keep the existing assertions and tolerances.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-downgrade-vern9 (no session URL)
@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Fresh read-only review completed by Codex CLI 0.164.0 / gpt-6.1-sol, session 01a1aa8e-95d0-7da2-b348-ede9e127c515 (no session URL), on head 90a1cbc.

VERDICT: MERGE-WITH-ISSUES

No blocking correctness findings. The reviewer verified that the adjacent 4.18.0/4.18.1 environments differ only in OrdinaryDiffEqCore, the original CI discrepancy is reproduced exactly, and the isolated upstream clock fix produces the same passing result as 4.18.1. The two-line change constrains only test dependency resolution; solver code, assertions and tolerances are unchanged. No commits follow this review.

The reviewer inspected the saved full CPU and QA results; it did not run new Julia tests. Final local QA passed 25/25 quality checks and 75/75 JET checks after dependency precompilation, with Aqua's timeout unchanged. Cold-cache QA success and local GPU execution are not established. Exact-head CI was still running at review time; this verdict is not a GitHub approval or a merge waiver.

Risk assessment

  • Independent risk: medium-low (ML4: compat lower-bound increase; L3: test-environment scope), recorded before reading the author's assessment.
  • Author risk: low (L3). The higher medium-low rating governs.
  • Blast radius: test dependency resolution and reference solves; no runtime or public API change.
  • Confidence: high in the correctness and scope of the dependency floor.
  • Merge: needs human review; the PR remains a draft and CI is pending.

Links:


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-downgrade-vern9 (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

A second independent review was done in a fresh gpt-6.1-sol session (01a1aaa5-1546-7233-adb6-244faf9a3b30), against head 90a1cbc.

  • Verdict: MERGE-WITH-ISSUES, rated medium-low (ML4: raising a compat lower bound). The reviewer first rated it low (L3) without seeing the PR body, then raised it to ML4 under the policy's highest-matching-criterion rule.
  • Requested changes: none. The reviewer confirmed the diagnosis against upstream SciML/OrdinaryDiffEq.jl@498c765. The floor applies only to the test environment, and no assertion or tolerance changed.
  • P3 finding: the Downgrade CI sandbox resolves OrdinaryDiffEqCore 4.19.1, not the exact 4.18.1 floor. The 4.18.0 (fail) → 4.18.1 (pass) evidence comes from local runs.
  • Pending CI: the Julia 1 CPU/JLArrays/OpenCL, CUDA and Buildkite jobs were still pending when it was reviewed. The CUDA Enzyme failure tracked in CUDA CI: test/enzyme/cuda_records.jl fails on released CUDACore (Enzyme undef-shadow fill! for non-numeric eltypes) #546 already happens on master.

With this, the PR has two counted medium-low ratings from separate sessions (the first was in 01a1aa8e…).


🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5
Conversation: https://claude.ai/code/session_0193DoGeYoy9sJDG2UUDBG7h

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Buildkite 1869 has two red jobs on this head (90a1cbc). Neither is caused by this PR:

  • oneAPI, Julia v1: artifact install failed with HTTP/2 504 downloading CUDA_Driver_jll before any test ran. This is an infrastructure failure.
  • AMDGPU, Julia v1: Lower level API errored with GPU Kernel Exception: Unknown exception inside AMDGPU's partial_mapreduce_device (the sum(abs2, ...) in diffeqgpunorm during _ode_init). Master d896a52 passed the same job in build 1865 with the same resolved versions (AMDGPU v2.8.0, OrdinaryDiffEqCore v4.19.1, DiffEqGPU v3.21.6). That rules out the new compat lower bound, which does not change what resolves on that runner. This looks like an intermittent AMDGPU runtime failure. A rerun of the job should settle it.

All other checks are green except CUDA, which has the existing failure tracked in #546.


🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5
Conversation: https://claude.ai/code/session_0193DoGeYoy9sJDG2UUDBG7h

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant