Skip to content

CUDA CI: test/enzyme/cuda_records.jl fails on released CUDACore (Enzyme undef-shadow fill! for non-numeric eltypes) #546

Open

Description

@ChrisRackauckas-Claude

Summary

CUDA Tests (Julia 1) has failed on every master run since 59b4032 (#532), and therefore on every PR since (#541 through #545). The failing test is test/enzyme/cuda_records.jl:9. It exercises no DiffEqGPU code: it differentiates CuArray{EnzymeAllocationRecord}(undef, dims) with Enzyme, where EnzymeAllocationRecord is a non-numeric isbits struct.

Because test/enzyme_environment.jl includes cuda_records.jl before transfers.jl and gradients.jl in the same process, the error aborts that process and the CUDA Enzyme transfer/gradient tests for DiffEqGPU do not run on CI.

Cause (upstream, CUDACore)

The forward rule for CuArray{T}(undef, ...) in CUDACore's Enzyme extension zero-initializes the shadow with fill!(shadow, 0) (released v6.3.1, CUDACore/ext/EnzymeCoreExt.jl:135). For a struct eltype, setindex! in the GPUArrays fill kernel has to convert(EnzymeAllocationRecord, 0), which is a MethodError, and the kernel fails GPU compilation:

Allocate records (): Error During Test at test/enzyme/cuda_records.jl:9
  InvalidIRError: compiling MethodInstance for (::GPUArrays.var"#gpu_fill_kernel!#gpu_fill_kernel!##0")(..., ::CuDeviceArray{EnzymeAllocationRecord, 0, 1}, ::Int64) resulted in invalid LLVM IR
  Reason: unsupported call to an unknown function (call to jl_f_throw_methoderror)
  ...
   [24] fill!(A::CuArray{EnzymeAllocationRecord, 0, CUDACore.DeviceMemory}, x::Int64)
      @ GPUArrays ~/.julia/packages/GPUArrays/bXuVc/src/host/construction.jl:32
   [25] forward(config::EnzymeCore.EnzymeRules.FwdConfigWidth{1, true, true, false, false}, ofn::Const{Type{CuArray{EnzymeAllocationRecord}}}, ...)
      @ EnzymeCoreExt ~/.julia/packages/CUDACore/GtzaO/ext/EnzymeCoreExt.jl:135

Resolved versions in the failing job: CUDA/CUDACore 6.3.1, GPUArrays 11.5.14, GPUCompiler 2.6.0, KernelAbstractions 0.9.42, Julia 1.13.0.

CUDA.jl master (ce09a7a09, 2026-09-23) changed this to fill!(shadow, zero(eltype(shadow))), which still fails for this test: zero(EnzymeAllocationRecord) has no method. The byte-zeroing that the test relies on (zero_undef_shadow!, which zeroes the storage as UInt8 for non-numeric eltypes) exists only in the fork commit ChrisRackauckas-Claude/CUDA.jl@081de78 that #532 pinned during development. The pin was removed before merge, and I found no upstream CUDA.jl PR or issue carrying the change. #532's own CUDA check failed with this error: https://github.com/SciML/DiffEqGPU.jl/actions/runs/34765382400/job/103745342091

Before #532 the CUDA lane was red for a different reason (the GPUVern7 callback comparison, #540). That failure does not appear in the current runs.

Reproduction

Needs an NVIDIA GPU:

using Enzyme, CUDACore
struct EnzymeAllocationRecord
    value::Float64
    flag::Bool
end
recordalloc(dims) = CuArray{EnzymeAllocationRecord}(undef, dims)
Enzyme.autodiff(ForwardWithPrimal, recordalloc, Duplicated, Const(()))

Or GROUP=CUDA julia --project -e 'using Pkg; Pkg.test()'.

Logs

Resolution options

  1. Upstream the byte-zero shadow initialization to JuliaGPU/CUDA.jl (for example make_zero, or zeroing through a UInt8 reinterpretation for non-numeric isbits eltypes). Then raise the CUDACore floor in test/enzyme/Project.toml to the release that includes it.
  2. Until then, the test can't pass against any released CUDACore. This is a maintainer decision. No skip or @test_broken has been added.

I did not run this locally because the investigating machine has no GPU. The diagnosis comes from the CI logs and the CUDACore source at v6.3.1 and at master.

🤖 Generated with Claude Code (model: claude-opus-5-5[1m])

https://claude.ai/code/session_01SJfy3ez7u4CqvAerKrT5np

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions