Repository navigation
Preserve keyword sensitivities in private problem records - #586
Conversation
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Codex <noreply@openai.com> Agent-Harness: Codex CLI 0.164.0 Agent-Model: gpt-6-astra Agent-Session: local job diffeqgpu-benchgap (no session URL)
Normalize private record keywords to a NamedTuple while preserving the noinline constructor and numeric-field order. Cover every active problem field and accumulated gradients on CPU, JLArrays and CUDA. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Codex <noreply@openai.com> Agent-Harness: Codex CLI 0.164.0 Agent-Model: gpt-6-astra Agent-Session: local job diffeqgpu-benchgap (no session URL)
Fresh independent review of head No actionable source defects were found. The review confirms preservation of the private noinline boundary, numeric-field order, adapted keyword values, host problems and public output layout. It inspected the identical 44-pass/4-fail versus 48/48 regression, both 58-assertion Enzyme groups, 793 passing non-Enzyme CUDA assertions and the new 6/6 CUDA adaptation checks, followed by the matched allocation-record error. The focused continuation's three adaptive-gradient errors match clean master. Both 432-row timing matrices were checked; no performance gain is established. The reviewer personally ran read-only Git/source, Python log/CSV/TOML, whitespace and live CI checks, with no Julia/GPU execution. At the review snapshot, the published commit's CUDA job and Enzyme jobs were incomplete. G2 remains unresolved until CI exercises those paths or the reviewer runs them. Author logs and pending checks are not counted as that evidence. A further review will inspect the completed results and their failure identities. Risk assessment
Links
🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra |
Fresh independent re-review of unchanged head The reviewer inspected the full diff before the author's risk assessment and independently rated medium-low (ML1), high confidence. The inherited specialization change also fits ML3/L7. It agrees with the author's rating. Completed exact-head CI resolves the earlier G2 finding: both Enzyme groups pass 58 assertions, and CUDA CI passes 793 non-Enzyme assertions, 8 imports and all 6 new active-field checks before the known allocation-record error. The reviewer personally inspected live job logs, source and matched before/after controls; it launched no Julia/GPU processes. The review confirms preservation of adapted keyword values, symbol lookup, the private noinline construction boundary, host problem retention and public return/output layouts. The analytical regression checks every active field and repeated accumulation in both precisions; 44 passes / 4 failures before → 48/48 after. No test-defined AD rule or weakened assertion is introduced. Existing failures remain disclosed: allocation-record and adaptive CUDA AD errors, LTS initialization failures, AMDGPU's existing errors and the minimum-dependency GPUVern9 check. The version-only baseline has the exact GPUVern9 failure; a matching-version clean-master control passes on a different runner image. No numerical cause, hang fix or performance gain is claimed. The reviewer considers G1/G2 satisfied for the changed paths and finds no new terminal failure signature. This is still a draft for Chris; no approval or merge has been performed. 🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra |
Second independent review: MERGE-WITH-ISSUES at exact head
Together with the first counted review (#586 (comment)), this gives the two counted medium-low ratings from different sessions. The PR stays a draft for @ChrisRackauckas; nothing has been approved or merged. 🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5 |
Ignore this draft until reviewed by @ChrisRackauckas. Depends on #584.
Enzyme on Julia 1.10 can silently discard a Float32 keyword-field sensitivity across the private problem-record constructor. The constructor stores keyword values as a NamedTuple while retaining its noinline boundary and numeric-field ordering. Public problems and solution return types are unchanged. The regression exercises initial state, both times, parameters, a captured function value and a keyword value with nonzero gradient accumulation, for Float32 and Float64 on CPU/JLArrays and CUDA.
Regression
The same focused transfer test on clean master
9cef7c8360782dd472d46bcf03bd99ac7797f65eand the primal-adaptation branch119bb9d7e155aa5a33897e60db57c9ef2344b09cgives 44 passes / 4 failures under Julia 1.10.12, Enzyme 0.13.216, EnzymeCore 0.8.26 and Enzyme_jll 0.0.303+1. In both CPU and JLArrays Float32 cases, the last derivative stays 1 instead of accumulating to 15 and then 29. The analytical gradient is[4, 1.25, 0.25, 1, 3, 14]; all expected values are exactly representable.With keyword normalization, the identical assertions give 48 passes / 0 failures. No custom AD rules, skipped tests, or changed tolerances are used. A clean environment importing only Enzyme and standard libraries reproduces the failure on Enzyme main
eb6a6df5da57ecb8843698919cb81a2a7a0ae338(32 passes / 4 failures for the full six-field reduction). A smaller two-field reorder and an independent padded-Float32 control also fail, while identity, NamedTuple normalization and Float64 controls pass. This resembles the aggregate-padding defects in Enzyme issue 3699 and issue 3849; no patched-core causal test or introducing-commit bisect is claimed.The new regression has its own test file, included immediately after Enzyme import checks. CUDA CI therefore exercises it before the existing allocation-record regression. Existing record, transfer and gradient tests remain enabled and unchanged.
Validation
Completed local checks use Julia 1.13.2 and Julia 1.10.12 with isolated project environments:
julia +release --project=@runic -m Runic --check src/solve.jl test/enzyme_environment.jl test/enzyme/problem_adaptation.jltyposon the three changed files;git diff --checkGROUP=QA julia +release --project -e 'using Pkg; Pkg.test(;julia_args=["--check-bounds=auto","--compiled-modules=existing"])'GROUP=CPU julia +release --project=<isolated-test-env> test/runtests.jljulia +lts --compiled-modules=noGROUP=Enzymethroughtest/runtests.jl, with the corresponding isolated release/LTS environmentsThe new regression has zero added comment lines. No public API, docstring, dependency or documentation changes are made, so a docs build is not applicable.
Full
GROUP=CUDAuses the unchanged officialtest/runtests.jl, a flushed logging wrapper, four Julia processes including both distributed workers and the Enzyme child, and a 30-minute no-output watchdog. On GPU 1 (NVIDIA RTX PRO 6000 Blackwell Max-Q),CUDA.functional()and a 256-element kernel pass. 793 non-Enzyme assertions, 8 Enzyme import checks and all 6 new CUDA adaptation checks pass, then the existing allocation-record test errors withzero(::Type{EnzymeAllocationRecord}), matching clean master and issue 546. No watchdog fired. This is a completed group invocation with a known error, not a green full-group result.The explicit follow-up runs all official import, problem-adaptation, transfer and gradient test files inside an outer testset: 35 passes / 3 errors. All 6 new adaptation and 14 existing transfer assertions pass. The three adaptive-gradient errors (Float32, Float64 and Lorenz) and unsupported
and i32diagnostics match the clean-master run; the seven other gradient assertions pass. A separate no-SciML CUDA reproducer supports the existing deferred custom-rule limitation, without a runtime version bisect. No expected values or tolerances are changed.Published-commit CI
The Enzyme LTS job and release job each pass all 58 assertions on Julia 1.10.13 and 1.13.3, with Enzyme 0.13.217 / EnzymeCore 0.8.27 / Enzyme_jll 0.0.304+0. These are separate from the local matched regression on the versions listed above.
Exact-head CUDA CI passes all 793 non-Enzyme assertions, 8 import checks and all 6 new CUDA active-field checks, then fails at the same
zero(::Type{EnzymeAllocationRecord})as clean master. The full group is not green. CPU release (both architectures), JLArrays release, OpenCL release, Metal release, oneAPI release, QA, docs, Runic and spelling pass. LTS initialization failures and AMDGPU release's missinglufact!method match baseline evidence. The AMDGPU LTS comparison establishes the same initialization InvalidIRError without the stacked changes. Green prerelease jobs skip non-RC Julia tests and are not runtime coverage.The downgrade CPU job reports a GPUVern9 saved-state-vector norm of
0.0010757452f0, exceeding0.001(49 passes / 1 failure). A version-only source-baseline job without this PR reports the exact same norm and failure. Its source/tests match base9cef7c8; only the package version differs. A current-dependency clean-master rerun passes all 50 assertions with Julia 1.10.13 and all 295 printed package versions matching this PR. Its runner image differs and neither job records the CPU model, so this is not an identical-hardware control. No tolerance is changed and no cause or fix for that existing numerical-check failure is claimed.The auxiliary Dependabot workflow fails validation before creating jobs because of a shared caller/reusable permissions mismatch. It is separate from the runtime checks and is already tracked in SciML/.github issue 141.
Before/after timings
GPU 1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (sm_120), Julia 1.13.2, CUDA/CUDACore 6.4.1, four Julia threads. Before is parent
1ba0a7674b10e4cb81801cd7e3d57c70b52a277b; after adds only this fix. The resolved matrix environments match apart from the source path. Lorenz uses tspan=(0,1), dt=0.001, save_everystep=false, dense=false, CPU offload=0 and adaptive abstol=reltol=1e-8. Each cell gives two repeats, each the minimum of five synchronized samples after warmup. Device-only calls use resident problem arrays and include output allocation/initialization. Both runs complete all 432 timing rows and exact high-level/device endpoint checks.No general speedup is claimed for this correctness fix. At N=2^21 the high-level repeat spread is 5.67–17.54% before and 4.45–14.98% after; device-only spread is 0.02–1.90% before and 0.003–0.99% after. Phase timings include host allocation/GC and are not additive estimates of the high-level minimum. For context, the supplied V100 rendering reports JAX 5541 ms fixed / 1145 ms adaptive at N=2^21 and PyTorch 964 ms fixed at N=2^19; different GPU/host hardware prevents a controlled cross-library speedup claim.
The earlier CUDA attempt timed out during a CPU TRBDF2 solve inside the CUDA group; that incomplete execution is not a pass. A rerun completed that same solve in 11.47 seconds; the earlier unseeded-input timeout remains unexplained and is not claimed fixed by this patch.
Risk assessment
🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-benchgap (no session URL)