Skip to content

Add portable batched LU for AMDGPU ensemble solves - #590

Merged
ChrisRackauckas merged 2 commits into
masterfrom
fix/amdgpu-batched-lu
Nov 5, 2026
Merged

ChrisRackauckas merged 2 commits into
masterfrom
fix/amdgpu-batched-lu

Conversation

@ChrisRackauckas

@ChrisRackauckas ChrisRackauckas commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Filed by an AI agent running as @ChrisRackauckas; Chris has not seen this.
Harness: Codex CLI 0.164.0 · Model: gpt-6-astra
Conversation: local job diffeqgpu-amdgpu-lufact (no session URL)

AMDGPU EnsembleGPUArray solves with an analytic Jacobian throw a missing-method error when constructing their batched LU factors. Add a KernelAbstractions fallback that applies the existing no-pivot LU helper to one matrix per work item, preserving the CPU loop and CUDA cuBLAS specialization. Document the factorization contract and test reconstruction and solve residuals on CPU, JLArrays, CUDA, and AMDGPU.

Fixes #589.

Draft: please ignore until reviewed by @ChrisRackauckas. No dependency or compatibility changes.

Regression evidence

The new test runs native dispatch and explicitly invokes the generic backend method, so CUDA exercises both its cuBLAS specialization and the portable kernel. Inputs include nonsymmetric matrices, Float32/Float64, sizes 1/3/5, batches 1/257, and an empty fallback batch. Reconstruction and solve residuals use the original matrices and right-hand sides as independent references. All buffers and input transfers are prepared before factorization; factorization and solve enqueue consecutively, and host copies follow both. The unchanged analytic-Jacobian testset runs before the DAE testset for its existing CPU/CUDA/AMDGPU groups, so independent DAE failures cannot mask this regression coverage.

On the original source, the first case reports:

CPU:      5 pass, 1 error: no method matching invoke lufact!(::Backend, ::Any)
JLArrays: 4 pass, 2 errors: missing native JLBackend and generic Backend methods
CUDA:     5 pass, 1 error: no method matching invoke lufact!(::Backend, ::Any)

After the patch, the complete regression file passes 3,146 assertions on each backend. CUDA execution used physical GPU 1 (CUDA_VISIBLE_DEVICES=1); a separate device fill/broadcast/host-comparison functional check also passed. No AMD hardware is available locally; AMD execution requires this PR's Buildkite jobs.

Validation

Local test environments copy the tracked test project and develop this checkout; source-path assertions confirm the patched package is loaded. Julia runs use four threads, isolated depots, and an 18 GiB memory cap. Commands below are relative to the validation workspace.

  • julia +release --project=@runic -m Runic --check on the four changed Julia files: passed. typos on all five changed files and git diff --check: passed.
  • GROUP=CPU julia +release --compiled-modules=existing --project=test-env -e 'include("repo/test/runtests.jl")': 4,091 passed.
  • Same official runner with GROUP=JLArrays: 3,714 passed on release at 7d6be38; the strengthened final LU file separately passes 3,146 / 3,146 on release and LTS.
  • GROUP=QA julia +release --compiled-modules=existing --project=test-env -e 'push!(LOAD_PATH, abspath("test-env")); include("repo/test/runtests.jl")': 27 QA + 75 JET passed. This invokes the official QA runner directly; early Pkg.test() setup attempts did not complete.
  • CPU and JLArrays on Julia 1.10.12 (+lts --compiled-modules=no): each passes 3,146 LU assertions, then stops at the existing ModelingToolkit DAE failure, 82 passed / 2 failed. A clean-master comparison reproduces this with ModelingToolkitBase 1.79.0; changing only that package to 1.78.1 passes 84 / 84. See AMDGPU LTS DAE initialization emits invalid IR with ModelingToolkitBase 1.79 #587. The final reordered LTS CPU runner passes 54 / 54 implicit-method assertions before reaching the same DAE failure.
  • GROUP=CUDA CUDA_VISIBLE_DEVICES=1 on Julia 1.13.2: the LU regression passes 3,146 / 3,146, and the existing implicit-method file passes 54 / 54. All 24 non-distributed/non-subprocess CUDA files pass when run in their original SafeTestset groupings: 3,934 assertions in total. The standalone launcher initially isolated the two Tsit5 files incorrectly; the continuation restores their shared setup and passes. Repository tests are unchanged.
  • The original CUDA Enzyme subprocess body passes its eight import assertions, then fails at zero(::Type{EnzymeAllocationRecord}), CUDACore 6.4.1 / Enzyme 0.13.217, matching CUDA CI: test/enzyme/cuda_records.jl fails on released CUDACore (Enzyme undef-shadow fill! for non-numeric eltypes) #546. Transfers/gradients are not reached. The complete CUDA runner requires addprocs(2), exceeding the explicit two-Julia-process limit; distributed execution is unverified locally. The final CUDA CI job passes 3,146 LU, 54 implicit-method, and five distributed assertions before failing with the same Enzyme error.
  • GKSwstype=100 julia +release --compiled-modules=existing --project=docs-check/docs -e 'cd("docs-check"); include("docs/make.jl")': both attempts execute examples/document checks, then abort on HTTP 429 for the existing ColPrac link. The documentation CI job on final candidate bbe1a1f built and deployed successfully. No check is disabled and no tolerance is changed.

No AMD hardware is available locally. AMDGPU execution and the analytic-Jacobian integration cases must be checked in this draft's Buildkite jobs. Three fresh Sol reviews found no implementation defect and held G2 open pending AMD execution. The final-candidate review accepts the uninterrupted device test and earlier analytic-Jacobian coverage; its CHANGES verdict is solely for missing AMD hardware verification. Re-review follows the AMD evidence.

Risk assessment


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-amdgpu-lufact (no session URL)

ChrisRackauckas and others added 2 commits November 4, 2026 16:56
Provide a KernelAbstractions fallback for analytic-Jacobian ensemble solves
on AMDGPU. Preserve the CPU loop and CUDA cuBLAS methods, document the
no-pivot contract, and check LU reconstruction and solve residuals across
backend dispatch paths.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-amdgpu-lufact (no session URL)
Allocate and populate all device buffers before factorization, then enqueue
factorization and solve before collecting host results. Run the existing
implicit-method testset before DAE tests so unrelated initialization failures
do not prevent analytic-Jacobian regression coverage.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Codex <noreply@openai.com>
Agent-Harness: Codex CLI 0.164.0
Agent-Model: gpt-6-astra
Agent-Session: local job diffeqgpu-amdgpu-lufact (no session URL)
@ChrisRackauckas

ChrisRackauckas commented Nov 4, 2026 •

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Validation and independent review for commit bbe1a1f215bb5981d893fc46c6541146cafd1c4a:

  • The new regression fails on the original source for missing generic dispatch, then passes 3,146 assertions per backend on CPU, JLArrays, and CUDA. CPU/JLArrays LTS also pass the new regression.
  • Release CPU: 4,091 passed. Release JLArrays: 3,714 passed at the first implementation commit; the strengthened final LU regression was rerun successfully. QA/JET: 27 + 75 passed. CUDA single-process test files: 3,934 passed, including all 54 implicit-method assertions. Runic and typos pass locally.
  • The LTS DAE failure reproduces on clean master with ModelingToolkitBase 1.79.0 and passes with only that dependency changed to 1.78.1; tracked in AMDGPU LTS DAE initialization emits invalid IR with ModelingToolkitBase 1.79 #587. CUDA Enzyme reproduces CUDA CI: test/enzyme/cuda_records.jl fails on released CUDACore (Enzyme undef-shadow fill! for non-numeric eltypes) #546. The final CUDA CI job passes LU 3,146/3,146, implicit methods 54/54, and distributed tests 5/5 before the known Enzyme failure. Distributed execution could not run under the two-process local limit. Headless local docs reach an existing link's HTTP 429 failure; final documentation CI renders and deploys successfully. No test was disabled or tolerance weakened.

As of 2026-11-04 21:11 UTC, Buildkite 1858 has not started either AMDGPU job. There is no candidate AMD execution log. The 3,146 LU and 54 implicit-method assertions, including both analytic-Jacobian cases, must pass on both Julia channels before the AMD hardware gate is satisfied. After about an hour since the test matrix was uploaded, both AMD jobs remain scheduled without runners. AMD hardware verification and a positive final review are pending; no AMD pass is claimed.

Fresh read-only review of this exact head: Codex CLI 0.164.0 / gpt-6.1-sol, independently rated medium-low (ML7) before reading the author assessment, with high source-review confidence. VERDICT: CHANGES — missing AMD hardware verification (G2). No demonstrated code defect was found; the reviewer accepted the uninterrupted factorization/solve test, independent numerical references, inactive-lane guard, preserved dispatch, and unchanged assertions. A positive verdict remains pending the AMD logs.

The PR remains a draft for Chris's review.


🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-amdgpu-lufact (no session URL)

@ChrisRackauckas

Copy link
Copy Markdown
Member Author

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

AMD hardware results for head bbe1a1f (Buildkite build 1858):

Job Batched LU Implicit methods (incl. Rosenbrock23/Rodas5P with jac) Job result
AMDGPU Julia 1 3146/3146 54/54 pass
AMDGPU Julia 1.10 3146/3146 54/54 fail, only in the DAE testset (80 pass, 2 fail, 1 InvalidIRError from verbose::SciMLLogging.Minimal in the init kwargs, #587)

The oneAPI and Metal Julia 1.10 jobs fail the same two DAE assertions (82 pass, 2 fail). #588 fixes that. CUDA fails on the existing #546.

Two fresh gpt-6.1-sol review sessions, each rating risk before reading evidence:

  • session 01a1a8c4-1620-7313-b339-b48fc30d3e9b: MERGE-WITH-ISSUES, medium-low (ML7), no code changes requested.
  • session 01a1a8cb-0396-7092-98c5-eb64666f4b4d: MERGE-WITH-ISSUES, medium-low (ML7), no code changes requested. It also checked that inactive lanes are masked by KernelAbstractions bounds checking and that the CPU/CUDA lufact! methods still dispatch ahead of the fallback.

The "issues" in both reviews are the pre-existing #587 and #546 failures, not defects in this patch. The PR body's risk section is updated. Suggested merge order is after #588.


🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5
Conversation: https://claude.ai/code/session_0193DoGeYoy9sJDG2UUDBG7h

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review November 5, 2026 03:35
@ChrisRackauckas
ChrisRackauckas merged commit d896a52 into master Nov 5, 2026
24 of 32 checks passed
@ChrisRackauckas
ChrisRackauckas deleted the fix/amdgpu-batched-lu branch November 5, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AMDGPU analytic-Jacobian ensemble solves have no batched LU method

1 participant