Skip to content

Preserve keyword sensitivities in private problem records - #586

Merged
ChrisRackauckas merged 2 commits into
SciML:masterfrom
ChrisRackauckas:fix/enzyme-keyword-records
Nov 5, 2026
Merged

ChrisRackauckas merged 2 commits into
SciML:masterfrom
ChrisRackauckas:fix/enzyme-keyword-records

Conversation

@ChrisRackauckas

@ChrisRackauckas ChrisRackauckas commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Filed by an AI agent running as @ChrisRackauckas; Chris has not seen this.
Harness: Codex CLI 0.164.0 · Model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

Ignore this draft until reviewed by @ChrisRackauckas. Depends on #584.

Enzyme on Julia 1.10 can silently discard a Float32 keyword-field sensitivity across the private problem-record constructor. The constructor stores keyword values as a NamedTuple while retaining its noinline boundary and numeric-field ordering. Public problems and solution return types are unchanged. The regression exercises initial state, both times, parameters, a captured function value and a keyword value with nonzero gradient accumulation, for Float32 and Float64 on CPU/JLArrays and CUDA.

Regression

The same focused transfer test on clean master 9cef7c8360782dd472d46bcf03bd99ac7797f65e and the primal-adaptation branch 119bb9d7e155aa5a33897e60db57c9ef2344b09c gives 44 passes / 4 failures under Julia 1.10.12, Enzyme 0.13.216, EnzymeCore 0.8.26 and Enzyme_jll 0.0.303+1. In both CPU and JLArrays Float32 cases, the last derivative stays 1 instead of accumulating to 15 and then 29. The analytical gradient is [4, 1.25, 0.25, 1, 3, 14]; all expected values are exactly representable.

With keyword normalization, the identical assertions give 48 passes / 0 failures. No custom AD rules, skipped tests, or changed tolerances are used. A clean environment importing only Enzyme and standard libraries reproduces the failure on Enzyme main eb6a6df5da57ecb8843698919cb81a2a7a0ae338 (32 passes / 4 failures for the full six-field reduction). A smaller two-field reorder and an independent padded-Float32 control also fail, while identity, NamedTuple normalization and Float64 controls pass. This resembles the aggregate-padding defects in Enzyme issue 3699 and issue 3849; no patched-core causal test or introducing-commit bisect is claimed.

The new regression has its own test file, included immediately after Enzyme import checks. CUDA CI therefore exercises it before the existing allocation-record regression. Existing record, transfer and gradient tests remain enabled and unchanged.

Validation

Completed local checks use Julia 1.13.2 and Julia 1.10.12 with isolated project environments:

Check Command Observed result
Formatting julia +release --project=@runic -m Runic --check src/solve.jl test/enzyme_environment.jl test/enzyme/problem_adaptation.jl Pass
Spelling and whitespace typos on the three changed files; git diff --check Pass
QA GROUP=QA julia +release --project -e 'using Pkg; Pkg.test(;julia_args=["--check-bounds=auto","--compiled-modules=existing"])' 27 QA assertions and 75 JET checks pass
CPU release GROUP=CPU julia +release --project=<isolated-test-env> test/runtests.jl Full group passes
CPU LTS Same direct test entrypoint with julia +lts --compiled-modules=no DAE initialization stops with 82 passes / 2 failures, matching clean master and SciMLBase issue 1659; later groups are not reached
Enzyme release and LTS GROUP=Enzyme through test/runtests.jl, with the corresponding isolated release/LTS environments Both full groups pass all 58 assertions

The new regression has zero added comment lines. No public API, docstring, dependency or documentation changes are made, so a docs build is not applicable.

Full GROUP=CUDA uses the unchanged official test/runtests.jl, a flushed logging wrapper, four Julia processes including both distributed workers and the Enzyme child, and a 30-minute no-output watchdog. On GPU 1 (NVIDIA RTX PRO 6000 Blackwell Max-Q), CUDA.functional() and a 256-element kernel pass. 793 non-Enzyme assertions, 8 Enzyme import checks and all 6 new CUDA adaptation checks pass, then the existing allocation-record test errors with zero(::Type{EnzymeAllocationRecord}), matching clean master and issue 546. No watchdog fired. This is a completed group invocation with a known error, not a green full-group result.

The explicit follow-up runs all official import, problem-adaptation, transfer and gradient test files inside an outer testset: 35 passes / 3 errors. All 6 new adaptation and 14 existing transfer assertions pass. The three adaptive-gradient errors (Float32, Float64 and Lorenz) and unsupported and i32 diagnostics match the clean-master run; the seven other gradient assertions pass. A separate no-SciML CUDA reproducer supports the existing deferred custom-rule limitation, without a runtime version bisect. No expected values or tolerances are changed.

Published-commit CI

The Enzyme LTS job and release job each pass all 58 assertions on Julia 1.10.13 and 1.13.3, with Enzyme 0.13.217 / EnzymeCore 0.8.27 / Enzyme_jll 0.0.304+0. These are separate from the local matched regression on the versions listed above.

Exact-head CUDA CI passes all 793 non-Enzyme assertions, 8 import checks and all 6 new CUDA active-field checks, then fails at the same zero(::Type{EnzymeAllocationRecord}) as clean master. The full group is not green. CPU release (both architectures), JLArrays release, OpenCL release, Metal release, oneAPI release, QA, docs, Runic and spelling pass. LTS initialization failures and AMDGPU release's missing lufact! method match baseline evidence. The AMDGPU LTS comparison establishes the same initialization InvalidIRError without the stacked changes. Green prerelease jobs skip non-RC Julia tests and are not runtime coverage.

The downgrade CPU job reports a GPUVern9 saved-state-vector norm of 0.0010757452f0, exceeding 0.001 (49 passes / 1 failure). A version-only source-baseline job without this PR reports the exact same norm and failure. Its source/tests match base 9cef7c8; only the package version differs. A current-dependency clean-master rerun passes all 50 assertions with Julia 1.10.13 and all 295 printed package versions matching this PR. Its runner image differs and neither job records the CPU model, so this is not an identical-hardware control. No tolerance is changed and no cause or fix for that existing numerical-check failure is claimed.

The auxiliary Dependabot workflow fails validation before creating jobs because of a shared caller/reusable permissions mismatch. It is separate from the runtime checks and is already tracked in SciML/.github issue 141.

Before/after timings

GPU 1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (sm_120), Julia 1.13.2, CUDA/CUDACore 6.4.1, four Julia threads. Before is parent 1ba0a7674b10e4cb81801cd7e3d57c70b52a277b; after adds only this fix. The resolved matrix environments match apart from the source path. Lorenz uses tspan=(0,1), dt=0.001, save_everystep=false, dense=false, CPU offload=0 and adaptive abstol=reltol=1e-8. Each cell gives two repeats, each the minimum of five synchronized samples after warmup. Device-only calls use resident problem arrays and include output allocation/initialization. Both runs complete all 432 timing rows and exact high-level/device endpoint checks.

Precision Adaptive N High-level before (ms) High-level after (ms) Device-only before (ms) Device-only after (ms)
Float32 false 8192 0.877/0.641 0.803/0.526 0.087/0.087 0.089/0.089
Float32 false 131072 12.104/13.282 10.830/12.471 0.441/0.454 0.439/0.437
Float32 false 2097152 226.410/245.370 228.254/238.411 6.540/6.512 6.589/6.524
Float32 true 8192 0.728/0.531 0.622/0.649 0.165/0.170 0.171/0.165
Float32 true 131072 9.466/6.295 11.059/6.059 0.316/0.336 0.322/0.316
Float32 true 2097152 203.867/239.630 220.023/231.829 2.741/2.690 2.702/2.702
Float64 false 8192 2.763/2.508 3.050/2.472 1.972/1.975 1.974/2.000
Float64 false 131072 33.824/28.397 33.444/28.234 21.271/21.279 21.299/21.300
Float64 false 2097152 626.233/713.555 644.606/741.183 347.113/347.031 347.000/346.768
Float64 true 8192 1.193/1.756 1.397/1.754 0.743/0.745 0.742/0.745
Float64 true 131072 19.155/13.963 20.469/14.903 6.878/6.831 6.818/6.856
Float64 true 2097152 348.842/368.618 360.025/408.413 71.135/71.206 71.301/71.187

No general speedup is claimed for this correctness fix. At N=2^21 the high-level repeat spread is 5.67–17.54% before and 4.45–14.98% after; device-only spread is 0.02–1.90% before and 0.003–0.99% after. Phase timings include host allocation/GC and are not additive estimates of the high-level minimum. For context, the supplied V100 rendering reports JAX 5541 ms fixed / 1145 ms adaptive at N=2^21 and PyTorch 964 ms fixed at N=2^19; different GPU/host hardware prevents a controlled cross-library speedup claim.

The earlier CUDA attempt timed out during a CPU TRBDF2 solve inside the CUDA group; that incomplete execution is not a pass. A rerun completed that same solve in 11.47 seconds; the earlier unseeded-input timeout remains unexplained and is not claimed fixed by this patch.

Risk assessment

  • Risk: medium-low (ML1: fixes an incorrect active-field gradient with an analytical failing-before/passing-after regression).
  • Blast radius: private compatible-problem records used by GPU primal preparation and Enzyme differentiation; the host problem, public solution types, numerical solvers and controller remain intact.
  • Evidence: identical LTS transfer test changes from 44/48 to 48/48; full CPU/release and Enzyme/release/LTS pass; CUDA exercises the new regression before the matched allocation-record error, and focused CUDA checks reproduce only matched adaptive errors.
  • Independent review: Codex CLI 0.164.0 / gpt-6.1-sol independently rates medium-low (ML1), high confidence; MERGE-WITH-ISSUES, no actionable source defects. Completed exact-head CUDA adaptation and both Enzyme CI groups satisfy G2 for the changed paths. Baseline failures remain disclosed. Fresh review summary.
  • Merge: needs human review; this remains a draft for Chris with an accepted independent review and matched baseline failures. No approval or merge has been performed.

🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

ChrisRackauckas and others added 2 commits November 1, 2026 10:30
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-benchgap (no session URL)
Normalize private record keywords to a NamedTuple while preserving the
noinline constructor and numeric-field order. Cover every active problem
field and accumulated gradients on CPU, JLArrays and CUDA.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-benchgap (no session URL)
@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Fresh independent review of head 692fe2898d1d56098d303541f868f200a9f411b8: CHANGES — verification required; no source changes requested. Reviewer: Codex CLI 0.164.0 / gpt-6.1-sol, local session 01a1a561-5dd8-7b43-a089-bd6fb77ad815 (no session URL). The reviewer assessed the complete diff before reading the author's risk assessment: medium-low, ML1, high confidence; the stacked specialization change fits L7/ML3 and does not raise that rating.

No actionable source defects were found. The review confirms preservation of the private noinline boundary, numeric-field order, adapted keyword values, host problems and public output layout. It inspected the identical 44-pass/4-fail versus 48/48 regression, both 58-assertion Enzyme groups, 793 passing non-Enzyme CUDA assertions and the new 6/6 CUDA adaptation checks, followed by the matched allocation-record error. The focused continuation's three adaptive-gradient errors match clean master. Both 432-row timing matrices were checked; no performance gain is established.

The reviewer personally ran read-only Git/source, Python log/CSV/TOML, whitespace and live CI checks, with no Julia/GPU execution. At the review snapshot, the published commit's CUDA job and Enzyme jobs were incomplete. G2 remains unresolved until CI exercises those paths or the reviewer runs them. Author logs and pending checks are not counted as that evidence. A further review will inspect the completed results and their failure identities.

Risk assessment

  • Author: medium-low (ML1: analytical failing-before/passing-after coverage for an incorrect active-field gradient).
  • Independent reviewer: medium-low (ML1; L7/ML3 for the stacked specialization), high confidence in source scope.
  • Blast radius: private problem preparation and Enzyme differentiation; public return types and numerical controllers remain intact.
  • Evidence: inspected local regression, quality, backend and timing logs; published-commit GPU/AD CI incomplete.
  • Merge: needs human review; draft for Chris, with G2 still unmet.

Links


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Fresh independent re-review of unchanged head 692fe2898d1d56098d303541f868f200a9f411b8: MERGE-WITH-ISSUES, with no actionable source defects. Reviewer: Codex CLI 0.164.0 / gpt-6.1-sol; fresh launch session 01a1a59d-e63e-7070-b524-fe3c502e89d7, delegated reviewer session 01a1a59e-6017-79f2-a705-9b833097bff4 (no session URLs).

The reviewer inspected the full diff before the author's risk assessment and independently rated medium-low (ML1), high confidence. The inherited specialization change also fits ML3/L7. It agrees with the author's rating.

Completed exact-head CI resolves the earlier G2 finding: both Enzyme groups pass 58 assertions, and CUDA CI passes 793 non-Enzyme assertions, 8 imports and all 6 new active-field checks before the known allocation-record error. The reviewer personally inspected live job logs, source and matched before/after controls; it launched no Julia/GPU processes.

The review confirms preservation of adapted keyword values, symbol lookup, the private noinline construction boundary, host problem retention and public return/output layouts. The analytical regression checks every active field and repeated accumulation in both precisions; 44 passes / 4 failures before → 48/48 after. No test-defined AD rule or weakened assertion is introduced.

Existing failures remain disclosed: allocation-record and adaptive CUDA AD errors, LTS initialization failures, AMDGPU's existing errors and the minimum-dependency GPUVern9 check. The version-only baseline has the exact GPUVern9 failure; a matching-version clean-master control passes on a different runner image. No numerical cause, hang fix or performance gain is claimed.

The reviewer considers G1/G2 satisfied for the changed paths and finds no new terminal failure signature. This is still a draft for Chris; no approval or merge has been performed.


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Second independent review: MERGE-WITH-ISSUES at exact head 692fe2898d1d56098d303541f868f200a9f411b8.

Together with the first counted review (#586 (comment)), this gives the two counted medium-low ratings from different sessions. The PR stays a draft for @ChrisRackauckas; nothing has been approved or merged.


🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5
Conversation: https://claude.ai/code/session_0193DoGeYoy9sJDG2UUDBG7h

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review November 5, 2026 03:33
@ChrisRackauckas
ChrisRackauckas merged commit 9d54845 into SciML:master Nov 5, 2026
21 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant