Skip to content

Resize the member caches of a CompositeCache in a DDEIntegrator - #4958

Open
singhharsh1708 wants to merge 7 commits into
SciML:masterfrom
singhharsh1708:fix-4952
Open

singhharsh1708 wants to merge 7 commits into
SciML:masterfrom
singhharsh1708:fix-4952

Conversation

@singhharsh1708

@singhharsh1708 singhharsh1708 commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

DelayDiffEq only resized the Jacobian config of a RosenbrockMutableCache and fell through to a no-op for anything else, so a Rosenbrock member of a CompositeCache kept its old config. A DDEIntegrator method for CompositeCache now forwards to each member, as OrdinaryDiffEqCore does for ODEIntegrator.

Resizing a composite whose members use different AD backends then threw Cannot convert ... JacobianPrep, in ODE as well as DDE and already on master: resize_jac_config!, resize_grad_config! and resize_J_W! took the backend from alg_autodiff(integrator.alg), which for a composite is the last member's. They now take an alg argument (defaulting to integrator.alg) and the Rosenbrock and DDE resize methods pass each member's cache.alg. Fixing it in alg_autodiff(::CompositeAlgorithm) instead would change other consumers such as DAE initialization.

The repro in #4952 (MethodOfSteps(AutoTsit5(Rosenbrock23())), resized while on the stiff member) threw a DimensionMismatch and now returns Success at [0.5, 0.5]. New ODE and DDE testsets cover a FiniteDiff and a ForwardDiff Rosenbrock23, dense and with KrylovJL_GMRES, in both orders; they error without the src change and pass with it.

Not covered here: growing a composite with SDIRK members fails on master in resize_nlsolver!, and its Newton path still resizes with integrator.alg.

Fixes #4952

Risk assessment

  • Risk: medium-low (ML2: resize! paths only; the shared helpers gain an alg argument whose default keeps every existing caller's behaviour).
  • Blast radius: Resize paths only. In OrdinaryDiffEqDifferentiation, resize_J_W!, resize_jac_config! and resize_grad_config! take an optional alg argument that defaults to integrator.alg, so existing callers are unchanged. OrdinaryDiffEqRosenbrock's resize_non_user_cache! passes cache.alg. In a composite ODE integrator, each Rosenbrock member now rebuilds its J, W and AD configs with its own backend, where before every member used integrator.alg. In DelayDiffEq, resize!, addat! and deleteat! on a DDE integrator with a CompositeCache now resize every member cache. Rosenbrock members resize J and W through OrdinaryDiffEqCore.resize_J_W! and their AD configs with cache.alg, as the ODE path does. This means matrix-free (Krylov) members work. That case used to throw a MethodError even for a single MethodOfSteps(Rosenbrock23(linsolve = KrylovJL_GMRES())). Dense Rosenbrock members are resized as before. For non-composite integrators cache.alg is integrator.alg, so behaviour there is unchanged. DelayDiffEq and OrdinaryDiffEqRosenbrock now require OrdinaryDiffEqDifferentiation 3.13.1.
  • Evidence: the new ODE and DDE testsets (FiniteDiff and ForwardDiff Rosenbrock23, dense and KrylovJL_GMRES, both orders) error without the src change and pass with it; DDE resize.jl and the ODE resize_tests.jl pass on this head. The red checks at a8fca87f (AD, GPU, Core, ProbNumDiffEq, SciMLSensitivity Core2) also fail on master.

AI Disclosure: Claude assisted with this change.

@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

PR: #4958

Reviewed head: d436d9c
Raw SHA-256: 73724c22dc6a78522de8c1ff56cbd22801ea276f44b60c04119db99932277541
Source SHA-256: 681965a75349d469ba5ce5146b7de0322b07ddc6f8b9c0fdbe25720fbe51c0d9
Source: standard-review-v1 (harness-log)
Reviewer session: 01a1d257-7f93-74e1-a6be-c01a65be15a9
Slot: 1
Confidence: unknown
Own risk fixed before author rating/discussion: yes

Risk assessment

  • Risk: medium (M1)
  • Verdict: CHANGES
  • Reviewed head: d436d9c

Review report

VERDICT: CHANGES
Round: 1
Head: d436d9c
Slot: 1
CI: CI is not green (regress:SciMLSensitivity.jl/Core4/1 / Downstream Tests - Core4)
Session: 01a1d257-7f93-74e1-a6be-c01a65be15a9

Risk

medium (M1: a new specialization of DiffEqBase.resize_non_user_cache! changes dispatch for existing composite DDE integrators).

Blocking findings

  • Verdict: CHANGES
  • Round: 1
  • Blocking priorities: P2
  • Deferred P2 issues: None.

[P2] Preserve each member's AD backend when resizing its cache — lib/DelayDiffEq/src/integrators/interface.jl:290. Confirmed by running: a composite containing finite-difference and ForwardDiff Rosenbrock23 members throws on even a same-size resize!(integ, 1), in either member order. The exception is MethodError: Cannot convert between ForwardDiffTwoArgJacobianPrep and FiniteDiffTwoArgJacobianPrep. With only the added method deleted at runtime, both otherwise identical cases finish with Success and state [0.49999999954711405].

The new loop passes the whole composite integrator to each cache's resize helper. Reading lib/OrdinaryDiffEqDifferentiation/src/derivative_wrappers.jl:337 and lib/OrdinaryDiffEqDifferentiation/src/alg_utils.jl:7 explains the observed failure: the helper selects the final algorithm's backend for every member. Rebuild configurations with the corresponding member's backend, and add a regression covering heterogeneous backends and same-size resizing.

Runnable reproduction in the resolved test environment:

using DelayDiffEq, OrdinaryDiffEqRosenbrock
using OrdinaryDiffEqCore: CompositeAlgorithm
using ADTypes: AutoFiniteDiff
f!(du, u, h, p, t) = (du .= -1000 .* (u .- 0.5); nothing)
prob = DDEProblem(f!, [1.0], (p, t) -> [0.5], (0.0, 0.1);
    constant_lags = [1.0])
alg = CompositeAlgorithm((Rosenbrock23(autodiff = AutoFiniteDiff()),
    Rosenbrock23()), _ -> 1)
integ = init(prob, MethodOfSteps(alg))
step!(integ)
resize!(integ, 1) # MethodError on this head

Non-blocking findings

None established.

What I ran

  • AI reviewer: codex-cli 0.164.0, model gpt-6.1-sol; session identified above. Blind code review; no PR body, comments, reviews, or author risk rating read.
  • git diff origin/master...HEAD, git diff --check origin/master...HEAD, repository instruction discovery, public cache API and resize-helper inspection. Diff check passed; no root AGENTS.md or CLAUDE.md was present. The newest commit has neither Agent-Harness nor Agent-Model trailers.
  • julia --version: 1.13.2. Scratch test environment uses the package's declared dependencies, extras, compat bounds and local monorepo sources; original layered JULIA_DEPOT_PATH preserved. Resolved DelayDiffEq 6.4.7, OrdinaryDiffEqCore 4.20.0, OrdinaryDiffEqDifferentiation 3.13.0, DiffEqBase 7.21.5, SciMLBase 3.58.0, LinearSolve 5.18.3, ForwardDiff 1.4.7.
  • julia --startup-file=no setup.jl: dependency resolution passed. Targeted Pkg.precompile(...): 27 dependencies successfully precompiled; 231 already precompiled.
  • Fresh-process julia --startup-file=no --check-bounds=yes --project=testenv mixed_regression.jl: two same-size head cases failed as described; removing only the new method made both controls succeed. No source file was changed for this control.
  • julia --startup-file=no --check-bounds=yes --project=testenv run_review.jl: the official GROUP=Integrators entrypoint passed 383 assertions, with 36 existing broken assertions (SDIRK); no failures or errors in that group. The added regression passed 2/2. Deleting only the added method made that same regression error with DimensionMismatch: ... OneTo(1) and ... OneTo(2).
  • Initial adversarial batch: 66 passed, 2 failed, 6 errored. Two errors reproduce the blocking mixed-backend bug. Four errors concern ForwardDiff chunk size 3 after shrinking to 2; their baseline status was not verified. The two failed assertions assumed AutoSwitch would remain on member 2; that probe assumption was invalid, and both analytic-solution checks passed. The optional deterministic-member follow-up was not run.
  • Isolated main-case probe passed: the stiff member was reached after 21 steps at t=0.05768237953913356; growth to 2 followed by solve! returned Success, t=3.0, state [0.5000000000000001, 0.5].
  • Final git status --short and git diff --exit-code: checkout unchanged. No commits, pushes, or GitHub posts.
  • uv run --locked python --version: Python 3.15.0; uv warned that --locked has no effect outside a project. This Julia checkout has no supplied Python project/uv lock environment. No tracked Julia interpreter/Manifest lock was present either; the scratch environment was resolved from the package constraints.

What I did not verify

  • Whether the four shrinking/deletion errors with ForwardDiff chunk size 3 are pre-existing; direct-solver/baseline controls were not run. The optional deterministic-member delayed-solution follow-up also did not start: its idle resource queue was interrupted after the required suite and probes completed. These are not additional confirmed PR regressions.
  • Supplied SciMLSensitivity/Core4 CI failure, other Julia versions, full monorepo/downstream suites, QA and documentation builds.

AUTHOR_GRADE: {"task": "bugfix", "score": 3, "difficulty": 2, "rounds": 1, "author_harness": "unknown", "author_model": "unknown", "note": "The original bug has failing-before/passing-after evidence and the affected group passes. A targeted head/control comparison confirms a newly introduced mixed-AD-backend regression even for same-size resizing."}

Head link: d436d9c


🤖 Posted by an AI agent — harness: codex-cli 0.164.0 · model: gpt-6.1-sol
Conversation: local session 01a1d257-7f93-74e1-a6be-c01a65be15a9 (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

@singhharsh1708 The PR body has no risk assessment, and the only review on record (2026-11-13) is for an older head. At a8fca87f the checks have settled; every red check I compared against the base branch is also red there or fails for a reason unrelated to this diff (AD, GPU, Core, ProbNumDiffEq downstream, SciMLSensitivity Core2).

Please add this block to the PR description, filled in for this change, so a fresh review can rate it:

## Risk assessment
- **Risk:** low | medium-low | medium | medium-high | high (<criterion ID>: <why>)
- **Blast radius:** code paths touched, public API or not, downstream packages
- **Evidence:** failing-before/passing-after test output, CI runs

The change resizes the member caches of a CompositeCache in a DDEIntegrator and touches three libraries (DelayDiffEq, OrdinaryDiffEqDifferentiation, OrdinaryDiffEqRosenbrock), so please state which previously working resize! calls still behave the same and which now error. The criteria are in the risk policy.


🤖 Posted by an AI agent — harness: Claude Code 2.1.325 · model: claude-sonnet-5-5
Conversation: local session 3878b54e-28f2-4a01-a176-a8f5468d1997 (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

PR: #4958

Reviewed head: a8fca87
Raw SHA-256: d88bb12fb33ecbb38b7b0414ef833b6dc5154dbbb76893a665a7e876206907b5
Source SHA-256: 0fb590fd30ca2784d7d08c1863c0e42b877532d131e88cc45ed8493811e24429
Source: standard-review-v1 (harness-log)
Reviewer session: a27ceb70-56f0-453b-a103-a196959dc8d9
Slot: 1
Confidence: unknown
Own risk fixed before author rating/discussion: yes
Redaction: ran (substitutions=0)

Risk assessment

  • Risk: medium-low (ML4)
  • Verdict: CHANGES
  • Reviewed head: a8fca87

Review report

VERDICT: CHANGES
Round: 4
Head: a8fca87
Slot: 1
CI: CI unreadable; review only; merge gates unchanged
Session: a27ceb70-56f0-453b-a103-a196959dc8d9
Reviewer: Antigravity CLI 1.5.10; model: gemini-4-carbon-high

Risk

medium-low (ML4: raises OrdinaryDiffEqDifferentiation compat lower bounds to 3.13.1 in DelayDiffEq and OrdinaryDiffEqRosenbrock; reaches peripheral composite-integrator resize paths and adds backward-compatible optional alg arguments to public developer resize helpers without changing default ODE or DDE stepping)

Blocking findings

  • Blocking priorities: P2

[P2] Resize J and W via resize_J_W! for RosenbrockMutableCache in DDEIntegrator — lib/DelayDiffEq/src/integrators/interface.jl:299-300. Confirmed by running: commit 231d4e74f updated OrdinaryDiffEqDifferentiation.resize_J_W! (lib/OrdinaryDiffEqDifferentiation/src/derivative_utils.jl:1522) and OrdinaryDiffEqRosenbrock.resize_non_user_cache! (lib/OrdinaryDiffEqRosenbrock/src/integrator_interface.jl:5) to forward cache.alg so composite ODE integrators with matrix-free Rosenbrock members (linsolve = KrylovJL_GMRES(), where cache.J is a JVPCache and cache.W is a WOperator) rebuild each member's operator with its own AD backend, and added (fdk, adk) coverage to test/Integrators_II/resize_tests.jl:263-265. However, DiffEqBase.resize_non_user_cache!(integrator::DDEIntegrator, cache::RosenbrockMutableCache, i) at lib/DelayDiffEq/src/integrators/interface.jl:299-300 still runs cache.J = similar(cache.J, i, i) and cache.W = similar(cache.W, i, i). Because this PR also adds DiffEqBase.resize_non_user_cache!(integrator::DDEIntegrator, cache::OrdinaryDiffEqCore.CompositeCache, i) (lib/DelayDiffEq/src/integrators/interface.jl:288-293), calling resize!, deleteat!, or addat! on any composite DDEIntegrator containing a matrix-free Rosenbrock member now dispatches into similar(cache.J, i, i) on JVPCache and throws MethodError: no method matching similar(::OrdinaryDiffEqDifferentiation.JVPCache{...}, ::Int64, ::Int64).

This regresses composite DDE solves that succeeded before this PR:

  • Calling resize!(integ, 2) on MethodOfSteps(AutoTsit5(Rosenbrock23(linsolve = KrylovJL_GMRES()))) on a non-stiff DDE (while Tsit5 is active and Rosenbrock23 has never stepped) throws MethodError on this head; deleting only the added CompositeCache method at runtime (origin/master baseline control) finishes with ReturnCode.Success and state [0.8032653477358567, 0.8401888679978896].
  • Calling same-size resize!(integ, 1) on MethodOfSteps(CompositeAlgorithm((fdk, adk), _ -> 1)) or (adk, fdk) (with fdk = Rosenbrock23(autodiff = AutoFiniteDiff(), linsolve = KrylovJL_GMRES()), adk = Rosenbrock23(linsolve = KrylovJL_GMRES())) throws the same MethodError on this head; the origin/master baseline control finishes with ReturnCode.Success and state [0.49999999954711405].
  • Growing resize!(integ, 2) on MethodOfSteps(CompositeAlgorithm((fdk, adk), _ -> 1)) and (adk, fdk) (the DDE counterpart of test/Integrators_II/resize_tests.jl:263-265) likewise throws MethodError.

Runnable reproduction in the DelayDiffEq test environment:

using DelayDiffEq, OrdinaryDiffEqTsit5, OrdinaryDiffEqRosenbrock
using LinearSolve: KrylovJL_GMRES

f_nonstiff!(du, u, h, p, t) = (du .= -1.0 .* (u .- 0.5); nothing)
prob = DDEProblem(f_nonstiff!, [1.0], (p, t) -> [0.5], (0.0, 0.5); constant_lags = [1.0])
integ = init(prob, MethodOfSteps(AutoTsit5(Rosenbrock23(linsolve = KrylovJL_GMRES()))))
step!(integ)       # active member is Tsit5 (integ.cache.current == 1)
resize!(integ, 2)  # throws MethodError: no method matching similar(::JVPCache, ::Int64, ::Int64)

Suggested fix (verified at runtime in a scratch harness without modifying the checkout): replace cache.J = similar(cache.J, i, i) and cache.W = similar(cache.W, i, i) at lib/DelayDiffEq/src/integrators/interface.jl:299-300 with OrdinaryDiffEqCore.resize_J_W!(cache, integrator, i, cache.alg) (matching lib/OrdinaryDiffEqRosenbrock/src/integrator_interface.jl:5; DelayDiffEq already calls OrdinaryDiffEqCore.resize_J_W! at line 269 and sets OrdinaryDiffEqDifferentiation = "3.13.1"), and extend lib/DelayDiffEq/test/integrators/resize.jl:135 to cover (fdk, adk) and (adk, fdk).

Non-blocking findings

  • lib/OrdinaryDiffEqDifferentiation/src/derivative_utils.jl:1522 (and lib/OrdinaryDiffEqCore/src/OrdinaryDiffEqCore.jl:237) — Document the optional alg argument on resize_J_W!. Confirmed from reading the diff and docs/src/devtools/internals/public_api.md:256: OrdinaryDiffEqCore.resize_J_W! is part of the documented public developer API (lib/OrdinaryDiffEqCore/src/OrdinaryDiffEqCore.jl:450). Commit ff4dfc393 updated the docstrings of resize_jac_config! and resize_grad_config! (lib/OrdinaryDiffEqDifferentiation/src/derivative_wrappers.jl:325,361) to document alg = integrator.alg, whereas commit 231d4e74f added alg = integrator.alg to resize_J_W!(cache, integrator, i, alg = integrator.alg) without updating lib/OrdinaryDiffEqCore/src/OrdinaryDiffEqCore.jl:237-241 or adding a method docstring at lib/OrdinaryDiffEqDifferentiation/src/derivative_utils.jl:1522. Suggested fix: document the optional alg = integrator.alg parameter on resize_J_W!.
  • lib/DelayDiffEq/test/integrators/resize.jl:136 and test/Integrators_II/resize_tests.jl:266 — Mixed-backend composite tests never step the second member after resize!. Confirmed by reading and running: both @testset "resize! with composite members on different AD backends" blocks pass _ -> 1 as the composite choice_function, so solve!(integ) after resize!(integ, 2) continues stepping member 1 only and never exercises member 2's resized cache during stepping (targeted probes with integ -> length(integ.u) == 1 ? 1 : 2 passed on ODEIntegrator for all four pairs and on DDEIntegrator for dense (fd, ad) and (ad, fd)). Suggested fix: use a switching choice function such as integ -> length(integ.u) == 1 ? 1 : 2 so the post-resize solve steps the second member.

What I ran

  • Blind code review on build host; no PR body, comments, reviews, sibling review directories, or author risk rating read. Checked git diff origin/master...HEAD, commit history since d436d9c75e7a0b4a2948fda113cd06f7b8dbbd69 (two author commits ff4dfc393 and 231d4e74f plus two clean merges of upstream/master), and git diff --check origin/master...HEAD (clean). Neither AGENTS.md nor CLAUDE.md is present in the repository; checked CONTRIBUTING.md.
  • Style and spell checks on all PR-modified files: typos passed (exit 0) and julia --project=@runic -m Runic --check passed (exit 0).
  • uv run --locked python --version: Python 3.15.0 (uv noted --locked has no effect outside a Python project; this Julia repository has no uv.lock).
  • julia --version: 1.13.2. Resolved two isolated scratch environments on build host preserving the layered JULIA_DEPOT_PATH and developing all local lib/* monorepo sublibraries (DelayDiffEq 6.4.7, OrdinaryDiffEq 7.8.2, OrdinaryDiffEqCore 4.20.1, OrdinaryDiffEqDifferentiation 3.13.1, OrdinaryDiffEqRosenbrock 2.7.7, OrdinaryDiffEqNonlinearSolve 2.9.10, DiffEqBase 7.21.7, SciMLBase 3.58.0, LinearSolve 5.18.3, ForwardDiff 1.5.0, FiniteDiff 2.33.0, DifferentiationInterface 0.7.22).
  • Affected test suites:
    • lib/DelayDiffEq/test/integrators/resize.jl (--check-bounds=yes): 60/60 passed.
    • lib/DelayDiffEq/test/qa/qa_tests.jl (Aqua + ExplicitImports): 20/20 passed.
    • test/Integrators_II/resize_tests.jl: 98/98 passed.
    • test/Integrators_II/deleteat_addat_tests.jl: 48/48 passed.
    • lib/OrdinaryDiffEqRosenbrock/test/rosenbrock_ad_tests.jl: 65/65 passed.
    • ExplicitImports (check_no_implicit_imports, check_no_stale_explicit_imports) on OrdinaryDiffEqDifferentiation and OrdinaryDiffEqRosenbrock: 4/4 passed.
  • Targeted adversarial probes and baseline controls:
    • Verified the prior Round 1 blocker is resolved for dense Rosenbrock members: mixed (AutoFiniteDiff, AutoForwardDiff) and (AutoForwardDiff, AutoFiniteDiff) Rosenbrock23 pairs in DDEIntegrator passed same-size resize!(integ, 1), growing resize!(integ, 2) with post-resize member switching checked against the exact piecewise analytic delayed solution (rtol = 1e-6), and explicit-last (Rosenbrock23(), Tsit5()) (16/16 passed).
    • Verified ODEIntegrator post-resize member switching (integ -> length(integ.u) == 1 ? 1 : 2) against the exact analytic ODE solution (rtol = 1e-5) across (fd, ad), (ad, fd), (fdk, adk), (adk, fdk), (fd, adk), (adk, fd), (ad, Tsit5()), (fdk, Tsit5()), and (Tsit5(), adk) (36/36 passed).
    • Reproduced the blocking DDEIntegrator matrix-free KrylovJL_GMRES() MethodError on HEAD across non-stiff AutoTsit5(adk) resize!(integ, 2), same-size (fdk, adk) / (adk, fdk) resize!(integ, 1), and growing (fdk, adk) / (adk, fdk) resize!(integ, 2); verified in a fresh Julia process with Base.delete_method on the added CompositeCache specialization that the first three succeeded before this PR while #4952 failed with DimensionMismatch.
    • Verified in an isolated runtime override (without touching checkout files) that replacing similar(cache.J, i, i) / similar(cache.W, i, i) with OrdinaryDiffEqCore.resize_J_W!(cache, integrator, i, cache.alg) in DelayDiffEq passes all 17 matrix-free and dense composite DDE resize assertions (17/17 passed).
    • Checked the unverified Round 1 note on shrinking u from length 3 to 2 with Rosenbrock23(): confirmed pre-existing on standalone MethodOfSteps(Rosenbrock23()) (ArgumentError: chunk size cannot be greater than ForwardDiff.structural_length(x) (3 > 2)), not introduced by this PR.

What I did not verify

  • Remote GitHub CI status (reported unreadable), full monorepo/downstream test suites (such as SciMLSensitivity), other Julia versions (e.g., 1.10 LTS / 1.11 downgrade jobs), GPU tests, and full Documenter builds.

AUTHOR_GRADE: {"task": "bugfix", "score": 3, "difficulty": 2, "rounds": 4, "author_harness": "unknown", "author_model": "unknown", "note": "The delta cleanly resolves the Round 1 mixed-AD-backend bug across OrdinaryDiffEqDifferentiation, OrdinaryDiffEqRosenbrock, and dense DelayDiffEq composite caches, and wires resize_J_W! for matrix-free composite ODE members, but leaves DelayDiffEq's RosenbrockMutableCache resize path calling similar(cache.J, i, i) and similar(cache.W, i, i), causing composite DDE integrators with matrix-free Rosenbrock members to regress with MethodError on resize!."}

Head link: a8fca87


🤖 Posted by an AI agent — harness: Antigravity CLI 1.5.10 · model: gemini-4-carbon-high
Conversation: local session a27ceb70-56f0-453b-a103-a196959dc8d9 (no session URL)

@singhharsh1708

Copy link
Copy Markdown
Member Author

Fixed in 7c7ea2d: the DDE Rosenbrock resize now goes through resize_J_W!(cache, integrator, i, cache.alg), so matrix-free members resize (single and composite, which threw before); the Krylov composites and the AutoTsit5 reproducer are in resize.jl.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DelayDiffEq resize! does not resize the Rosenbrock Jacobian config inside a CompositeAlgorithm

2 participants