Repository navigation
Add portable batched LU for AMDGPU ensemble solves - #590
Conversation
Provide a KernelAbstractions fallback for analytic-Jacobian ensemble solves on AMDGPU. Preserve the CPU loop and CUDA cuBLAS methods, document the no-pivot contract, and check LU reconstruction and solve residuals across backend dispatch paths. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Codex <noreply@openai.com> Agent-Harness: Codex CLI 0.164.0 Agent-Model: gpt-6-astra Agent-Session: local job diffeqgpu-amdgpu-lufact (no session URL)
Allocate and populate all device buffers before factorization, then enqueue factorization and solve before collecting host results. Run the existing implicit-method testset before DAE tests so unrelated initialization failures do not prevent analytic-Jacobian regression coverage. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Codex <noreply@openai.com> Agent-Harness: Codex CLI 0.164.0 Agent-Model: gpt-6-astra Agent-Session: local job diffeqgpu-amdgpu-lufact (no session URL)
Validation and independent review for commit
As of 2026-11-04 21:11 UTC, Buildkite 1858 has not started either AMDGPU job. There is no candidate AMD execution log. The 3,146 LU and 54 implicit-method assertions, including both analytic-Jacobian cases, must pass on both Julia channels before the AMD hardware gate is satisfied. After about an hour since the test matrix was uploaded, both AMD jobs remain scheduled without runners. AMD hardware verification and a positive final review are pending; no AMD pass is claimed. Fresh read-only review of this exact head: Codex CLI 0.164.0 / gpt-6.1-sol, independently rated medium-low (ML7) before reading the author assessment, with high source-review confidence. VERDICT: CHANGES — missing AMD hardware verification (G2). No demonstrated code defect was found; the reviewer accepted the uninterrupted factorization/solve test, independent numerical references, inactive-lane guard, preserved dispatch, and unchanged assertions. A positive verdict remains pending the AMD logs. The PR remains a draft for Chris's review. 🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra |
AMD hardware results for head
The oneAPI and Metal Julia 1.10 jobs fail the same two DAE assertions (82 pass, 2 fail). #588 fixes that. CUDA fails on the existing #546. Two fresh gpt-6.1-sol review sessions, each rating risk before reading evidence:
The "issues" in both reviews are the pre-existing #587 and #546 failures, not defects in this patch. The PR body's risk section is updated. Suggested merge order is after #588. 🤖 Posted by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5 |
AMDGPU
EnsembleGPUArraysolves with an analytic Jacobian throw a missing-method error when constructing their batched LU factors. Add a KernelAbstractions fallback that applies the existing no-pivot LU helper to one matrix per work item, preserving the CPU loop and CUDA cuBLAS specialization. Document the factorization contract and test reconstruction and solve residuals on CPU, JLArrays, CUDA, and AMDGPU.Fixes #589.
Draft: please ignore until reviewed by @ChrisRackauckas. No dependency or compatibility changes.
Regression evidence
The new test runs native dispatch and explicitly invokes the generic backend method, so CUDA exercises both its cuBLAS specialization and the portable kernel. Inputs include nonsymmetric matrices, Float32/Float64, sizes 1/3/5, batches 1/257, and an empty fallback batch. Reconstruction and solve residuals use the original matrices and right-hand sides as independent references. All buffers and input transfers are prepared before factorization; factorization and solve enqueue consecutively, and host copies follow both. The unchanged analytic-Jacobian testset runs before the DAE testset for its existing CPU/CUDA/AMDGPU groups, so independent DAE failures cannot mask this regression coverage.
On the original source, the first case reports:
After the patch, the complete regression file passes 3,146 assertions on each backend. CUDA execution used physical GPU 1 (
CUDA_VISIBLE_DEVICES=1); a separate device fill/broadcast/host-comparison functional check also passed. No AMD hardware is available locally; AMD execution requires this PR's Buildkite jobs.Validation
Local test environments copy the tracked test project and develop this checkout; source-path assertions confirm the patched package is loaded. Julia runs use four threads, isolated depots, and an 18 GiB memory cap. Commands below are relative to the validation workspace.
julia +release --project=@runic -m Runic --checkon the four changed Julia files: passed.typoson all five changed files andgit diff --check: passed.GROUP=CPU julia +release --compiled-modules=existing --project=test-env -e 'include("repo/test/runtests.jl")': 4,091 passed.GROUP=JLArrays: 3,714 passed on release at7d6be38; the strengthened final LU file separately passes 3,146 / 3,146 on release and LTS.GROUP=QA julia +release --compiled-modules=existing --project=test-env -e 'push!(LOAD_PATH, abspath("test-env")); include("repo/test/runtests.jl")': 27 QA + 75 JET passed. This invokes the official QA runner directly; earlyPkg.test()setup attempts did not complete.+lts --compiled-modules=no): each passes 3,146 LU assertions, then stops at the existing ModelingToolkit DAE failure, 82 passed / 2 failed. A clean-master comparison reproduces this with ModelingToolkitBase 1.79.0; changing only that package to 1.78.1 passes 84 / 84. See AMDGPU LTS DAE initialization emits invalid IR with ModelingToolkitBase 1.79 #587. The final reordered LTS CPU runner passes 54 / 54 implicit-method assertions before reaching the same DAE failure.GROUP=CUDA CUDA_VISIBLE_DEVICES=1on Julia 1.13.2: the LU regression passes 3,146 / 3,146, and the existing implicit-method file passes 54 / 54. All 24 non-distributed/non-subprocess CUDA files pass when run in their original SafeTestset groupings: 3,934 assertions in total. The standalone launcher initially isolated the two Tsit5 files incorrectly; the continuation restores their shared setup and passes. Repository tests are unchanged.zero(::Type{EnzymeAllocationRecord}), CUDACore 6.4.1 / Enzyme 0.13.217, matching CUDA CI: test/enzyme/cuda_records.jl fails on released CUDACore (Enzyme undef-shadow fill! for non-numeric eltypes) #546. Transfers/gradients are not reached. The complete CUDA runner requiresaddprocs(2), exceeding the explicit two-Julia-process limit; distributed execution is unverified locally. The final CUDA CI job passes 3,146 LU, 54 implicit-method, and five distributed assertions before failing with the same Enzyme error.GKSwstype=100 julia +release --compiled-modules=existing --project=docs-check/docs -e 'cd("docs-check"); include("docs/make.jl")': both attempts execute examples/document checks, then abort on HTTP 429 for the existing ColPrac link. The documentation CI job on final candidatebbe1a1fbuilt and deployed successfully. No check is disabled and no tolerance is changed.No AMD hardware is available locally. AMDGPU execution and the analytic-Jacobian integration cases must be checked in this draft's Buildkite jobs. Three fresh Sol reviews found no implementation defect and held G2 open pending AMD execution. The final-candidate review accepts the uninterrupted device test and earlier analytic-Jacobian coverage; its CHANGES verdict is solely for missing AMD hardware verification. Re-review follows the AMD evidence.
Risk assessment
lufact!developer interface, with existing specialized dispatch unchanged).bbe1a1fafter AMD hardware results: batched LU 3146 and implicit 54 pass on AMDGPU Julia 1 and 1.10 (Buildkite 1858). Remaining red jobs are the existing DAE initialization failure (AMDGPU LTS DAE initialization emits invalid IR with ModelingToolkitBase 1.79 #587, fixed by Omit host verbosity from device initialization problems #588) and CUDA Enzyme (CUDA CI: test/enzyme/cuda_records.jl fails on released CUDACore (Enzyme undef-shadow fill! for non-numeric eltypes) #546).🤖 Posted by an AI agent — harness: Codex CLI 0.164.0 · model: gpt-6-astra
Conversation: local job diffeqgpu-amdgpu-lufact (no session URL)