Repository navigation
Description
Summary
CUDA Tests (Julia 1) has failed on every master run since 59b4032 (#532), and therefore on every PR since (#541 through #545). The failing test is test/enzyme/cuda_records.jl:9. It exercises no DiffEqGPU code: it differentiates CuArray{EnzymeAllocationRecord}(undef, dims) with Enzyme, where EnzymeAllocationRecord is a non-numeric isbits struct.
Because test/enzyme_environment.jl includes cuda_records.jl before transfers.jl and gradients.jl in the same process, the error aborts that process and the CUDA Enzyme transfer/gradient tests for DiffEqGPU do not run on CI.
Cause (upstream, CUDACore)
The forward rule for CuArray{T}(undef, ...) in CUDACore's Enzyme extension zero-initializes the shadow with fill!(shadow, 0) (released v6.3.1, CUDACore/ext/EnzymeCoreExt.jl:135). For a struct eltype, setindex! in the GPUArrays fill kernel has to convert(EnzymeAllocationRecord, 0), which is a MethodError, and the kernel fails GPU compilation:
Allocate records (): Error During Test at test/enzyme/cuda_records.jl:9
InvalidIRError: compiling MethodInstance for (::GPUArrays.var"#gpu_fill_kernel!#gpu_fill_kernel!##0")(..., ::CuDeviceArray{EnzymeAllocationRecord, 0, 1}, ::Int64) resulted in invalid LLVM IR
Reason: unsupported call to an unknown function (call to jl_f_throw_methoderror)
...
[24] fill!(A::CuArray{EnzymeAllocationRecord, 0, CUDACore.DeviceMemory}, x::Int64)
@ GPUArrays ~/.julia/packages/GPUArrays/bXuVc/src/host/construction.jl:32
[25] forward(config::EnzymeCore.EnzymeRules.FwdConfigWidth{1, true, true, false, false}, ofn::Const{Type{CuArray{EnzymeAllocationRecord}}}, ...)
@ EnzymeCoreExt ~/.julia/packages/CUDACore/GtzaO/ext/EnzymeCoreExt.jl:135
Resolved versions in the failing job: CUDA/CUDACore 6.3.1, GPUArrays 11.5.14, GPUCompiler 2.6.0, KernelAbstractions 0.9.42, Julia 1.13.0.
CUDA.jl master (ce09a7a09, 2026-09-23) changed this to fill!(shadow, zero(eltype(shadow))), which still fails for this test: zero(EnzymeAllocationRecord) has no method. The byte-zeroing that the test relies on (zero_undef_shadow!, which zeroes the storage as UInt8 for non-numeric eltypes) exists only in the fork commit ChrisRackauckas-Claude/CUDA.jl@081de78 that #532 pinned during development. The pin was removed before merge, and I found no upstream CUDA.jl PR or issue carrying the change. #532's own CUDA check failed with this error: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34765382400/job/103745342091
Before #532 the CUDA lane was red for a different reason (the GPUVern7 callback comparison, #540). That failure does not appear in the current runs.
Reproduction
Needs an NVIDIA GPU:
using Enzyme, CUDACore
struct EnzymeAllocationRecord
value::Float64
flag::Bool
end
recordalloc(dims) = CuArray{EnzymeAllocationRecord}(undef, dims)
Enzyme.autodiff(ForwardWithPrimal, recordalloc, Duplicated, Const(()))Or GROUP=CUDA julia --project -e 'using Pkg; Pkg.test()'.
Logs
- Latest (Fuse adaptive GPUTsit5 stage arithmetic #545): https://github.com/SciML/DiffEqGPU.jl/actions/runs/35852671408/job/107153778413
- Master at the fix: restore Enzyme gradients through kernel ensembles #532 merge: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34784432463
Resolution options
- Upstream the byte-zero shadow initialization to JuliaGPU/CUDA.jl (for example
make_zero, or zeroing through aUInt8reinterpretation for non-numeric isbits eltypes). Then raise theCUDACorefloor intest/enzyme/Project.tomlto the release that includes it. - Until then, the test can't pass against any released CUDACore. This is a maintainer decision. No skip or
@test_brokenhas been added.
I did not run this locally because the investigating machine has no GPU. The diagnosis comes from the CI logs and the CUDACore source at v6.3.1 and at master.
🤖 Generated with Claude Code (model: claude-opus-5-5[1m])
Activity