Skip to content

Omit host verbosity from device initialization problems - #588

Merged
ChrisRackauckas merged 1 commit into
masterfrom
fix/device-initialization-keywords
Nov 5, 2026
Merged

ChrisRackauckas merged 1 commit into
masterfrom
fix/device-initialization-keywords

Conversation

@ChrisRackauckas

@ChrisRackauckas ChrisRackauckas commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Filed by an AI agent running as @ChrisRackauckas; Chris has not seen this.
Harness: Codex CLI 0.164.0 · Model: gpt-6-astra
Conversation: local job diffeqgpu-amdgpu-dae-fix (no session URL)

Please ignore this draft until reviewed by @ChrisRackauckas.

Fixes #587 (#587).

ModelingToolkitBase 1.79 supplies verbose = Minimal() on nonlinear initialization problems. Forwarding this host logging keyword into SciMLBase's immutable problem constructors produces nonempty Base.Pairs, which is non-isbits and keeps a runtime field lookup in the device nonlinear solve. DiffEqGPU now omits verbose when preparing square and least-squares initialization problems, while forwarding every other solver keyword. The existing problem type families, solution layout, and numerical initialization algorithms are preserved.

The tests cover mutable and immutable nonlinear problems and least-squares problems, empty keywords, host verbosity values, preservation of other solver keywords, and input immutability. The existing ModelingToolkit computed-initialization test exercises the actual Minimal() case and compares GPU initialization with an independent CPU solve. No dependency is added.

Regression evidence

Julia 1.10.12, ModelingToolkitBase 1.79.0, SciMLBase 3.58.0, SimpleNonlinearSolve 2.14.6, CUDA 6.3.1, GPUCompiler 2.6.0. CUDA execution used GPU 1 (RTX PRO 6000 Blackwell Max-Q); the functional scalar-write/readback test passed first.

Before the fix, running the DAE test file on master with the new assertions produced:

Device initialization verbosity | 24 Pass 15 Fail 39 Total
Computed initialization maps    |  1 Pass  1 Fail  1 Error  3 Total
Reason: unsupported call to an unknown function (call to ijl_field_index)
Reason: unsupported call to an unknown function (call to ijl_has_no_field_error)

After the fix, with the identical environment and GPU:

Device initialization verbosity               | 39 Pass 39 Total
Computed initialization maps                   |  4 Pass  4 Total
Initialization maps must be kernel-compatible  |  4 Pass  4 Total

The full DAE file changes from 101 pass / 17 fail / 2 errors to 120 pass / 1 error. The remaining bounded-initialization CUDA error also reproduces on a clean master checkout: 3 pass / 1 error, with dynamic NonlinearFunction invocation. Its kwargs are empty; it is separate from this bug and from #546. No assertion is skipped or weakened.

Validation

All Julia runs used JULIA_NUM_THREADS=4, JULIA_NUM_PRECOMPILE_TASKS=1, task-local environments and writable caches, and the wrapper systemd-run --user --scope -p MemoryMax=18G -p MemoryHigh=infinity timeout 14400. CUDA runs used CUDA_VISIBLE_DEVICES=1. Both test environments contain ModelingToolkitBase 1.79.0; the release environment uses CUDA 6.4.1 / GPUCompiler 2.10.0.

Check Observed result
julia +release --project=@runic -m Runic --check src/utils.jl test/gpu_kernel_de/stiff_ode/gpu_ode_modelingtoolkit_dae.jl exit 0
typos on all changed files; git diff --check exit 0
CPU group, Julia 1.10.12 984/984 pass
CPU group, Julia 1.13.2 984/984 pass
GROUP=QA julia +release --project -e 'using Pkg; Pkg.test(; julia_args=["--check-bounds=auto"])' 27/27 QA, 75/75 JET; Pkg reports tests passed
Full CUDA group, Julia 1.13.2, GPU 1 832 pass / 1 error; all 123 DAE tests pass
GKSwstype=100 julia +release --compiled-modules=existing --project=docs docs/make.jl exit 0; doctests, checks, links and HTML rendering complete

The CPU/CUDA group driver runs the official runner inside an outer testset so independent failures are collected:

using Test
@testset "DiffEqGPU $(ENV["GROUP"])" begin
    include("test/runtests.jl")
end

It was invoked with GROUP=CPU on julia +lts and julia +release, and GROUP=CUDA on julia +release, using --startup-file=no --project=<test-environment> (environment path omitted). The DAE before/after driver wraps include("test/gpu_kernel_de/stiff_ode/gpu_ode_modelingtoolkit_dae.jl") in the same way. Paths in these commands are relative to the repository root.

The full release CUDA run's sole error is the known #546 failure: MethodError: no method matching zero(::Type{EnzymeAllocationRecord}) in CUDACore's Enzyme allocation rule. It prevents later Enzyme transfer/gradient checks from running. Bounded initialization passes in this release environment; its separate failure above occurs with the older reconstructed CUDA/compiler versions. No new GPU test failure is observed.

The downgrade CPU CI job reports 49 passes / 1 failure in the adaptive GPUVern9 saved-state comparison: 0.0010757452f0 < 0.001. I fetched and read the version-only source-baseline log, which reports the exact same norm and failure. Its commit differs from the tested master base only in the package version; source, tests, and compat bounds match. This establishes a pre-existing occurrence, not a diagnosis of the numerical discrepancy or an identical-hardware control. No tolerance is changed.

The exact-head Metal CI logs report package tests passed and 123/123 DAE tests with ModelingToolkitBase 1.79.0 on both release and LTS. I read both complete logs. oneAPI release also reports package tests passed and 123/123 DAE tests with ModelingToolkitBase 1.79.0 in its complete log. oneAPI LTS also reports package tests passed and 123/123 DAE tests with ModelingToolkitBase 1.79.0. CPU release x64/x86, JLArrays, OpenCL, Enzyme, QA, docs, formatting and spelling CI checks pass.

Exact-head CUDA CI passes 123/123 DAE tests with ModelingToolkitBase 1.79.0, then fails at zero(::Type{EnzymeAllocationRecord}). I read the complete current log and the clean-master baseline log, which has the identical error. The baseline's head is the same master base 9cef7c8. This matches the local full-CUDA result and #546; the full CUDA group is not green.

AMDGPU release and LTS jobs remain queued, so neither has an execution result.

Scope and overlap

Based on master at 9cef7c8. #584, #585, and #586 change outer kernel preparation in src/solve.jl. This patch changes nested initialization conversion in src/utils.jl; it does not touch their implementation or test hunks. The outer keyword conversion in PR 586 does not convert nested initialization kwargs.

AMD hardware execution is not verified. Other nonempty solver kwargs retain their current representation; this patch specifically removes host verbosity and does not claim general support for arbitrary non-isbits keyword values. The CPU backend of EnsembleGPUKernel uses the same preparation and therefore also omits initialization verbosity.

Risk assessment

  • Risk: medium-low (ML1: initialization verbosity is intentionally omitted by kernel problem preparation, including the CPU backend; numerical algorithms and solver options other than verbosity are unchanged).
  • Blast radius: square, least-squares, and SCC-lowered initialization preparation for EnsembleGPUKernel; no public signature or solution-layout change.
  • Evidence: failing-before/passing-after isbits assertions and actual CUDA computed-initialization execution with ModelingToolkitBase 1.79.0; full validation above.
  • Independent review: Codex CLI 0.164.0 / gpt-6.1-sol rated this head low (L6), high confidence, verdict MERGE, with no findings (review summary). The reviewer selected their rating without reading the author assessment. The higher author rating, medium-low (ML1), stands; no commits follow the review.
  • Merge: needs human review; draft awaiting @ChrisRackauckas, with baseline CUDA and downgrade failures disclosed.

🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-amdgpu-dae-fix (no session URL)

Drop the host logging keyword when converting nonlinear initialization
problems while preserving numerical solver keywords and problem types.
Cover square and least-squares conversion and retain computed-map GPU tests.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-amdgpu-dae-fix (no session URL)
@ChrisRackauckas

ChrisRackauckas commented Nov 4, 2026 •

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

A fresh read-only review by Codex CLI 0.164.0 / gpt-6.1-sol of head 71cb062 returned MERGE, with no findings. The reviewer independently rated the patch low risk (L6), high confidence, before reading any author assessment. The higher author rating, medium-low (ML1), remains in effect. No commits have landed after that review.

The reviewer confirmed that the helper consumes only verbose, preserves all other solver keywords and input problems, and keeps the existing problem families, initialization algorithms, and solution layout. They considered the upstream representation issue and accepted this focused host-preparation fix. They inspected the recorded test evidence; they did not personally execute Julia tests.

Validation recorded on this head:

Exact-head hardware CI passed the full package tests and all 123 DAE tests with ModelingToolkitBase 1.79.0 on Metal release, Metal LTS, and oneAPI release. I read each complete job log.

The downgrade CPU job reports 49 pass / 1 fail: 0.0010757452f0 < 0.001 in the adaptive GPUVern9 saved-state comparison. The version-only source baseline reports the exact same norm and failure. I read both logs and verified that baseline source, tests, and compat bounds match master. This establishes a pre-existing occurrence; the numerical cause remains undiagnosed. No tolerance changed.

oneAPI LTS also completed successfully, including 123/123 DAE tests. Both release CPU architectures, JLArrays, OpenCL, Enzyme, QA, docs, formatting and spelling CI checks pass.

Exact-head CUDA CI passes 123/123 DAE tests with ModelingToolkitBase 1.79.0 and then reports the same zero(::Type{EnzymeAllocationRecord}) error as the clean-master job. I read both complete logs and verified the baseline head is the PR's master base 9cef7c8. CUDA and downgrade remain red from the disclosed baseline failures.

AMDGPU release and LTS jobs remain queued, with no execution result. No local AMD hardware validation is claimed. No source changes followed the independent review. The draft awaits @ChrisRackauckas's review.

Reviewer session: local session 01a1a6ca-4c78-7513-b1ca-f6a432dfa4b7 (no session URL).


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-amdgpu-dae-fix (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Second independent review: MERGE at exact head 71cb0624a41fa3a651bc95c1c55df6e27d53bade.

  • Reviewer: Codex CLI 0.164.0 / gpt-6.1-sol, fresh session 01a1a73d-faaf-7bf1-a652-6f5a9044f147, holding none of the author's or the first reviewer's context. It recorded its own rating before reading the PR body, the comments or the earlier review.
  • Independent risk: low (L6), with L1 for documentation and L2 for strengthened tests, high confidence. The author rated it medium-low (ML1), and the higher rating governs.
  • Findings: none.
  • Issue 587 is fixed on CI. Both AMDGPU jobs in https://buildkite.com/julialang/diffeqgpu-dot-jl/builds/1854 pass the DAE suite 123/123, and their logs contain neither ijl_field_index nor ijl_has_no_field_error.
  • The remaining red jobs are pre-existing:

Together with the first review, this gives the two counted ratings that medium-low requires, from different sessions. The PR stays a draft for @ChrisRackauckas; nothing has been approved or merged.


🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5
Conversation: https://claude.ai/code/session_0193DoGeYoy9sJDG2UUDBG7h

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review November 5, 2026 03:34
@ChrisRackauckas
ChrisRackauckas merged commit 1c55eb0 into master Nov 5, 2026
27 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AMDGPU LTS DAE initialization emits invalid IR with ModelingToolkitBase 1.79

1 participant