Skip to content

[WIP] ci(gcp-nvkind): run on Kubernetes 1.37 with kind v0.33.0 and nvkind main - #1524

Closed
harche wants to merge 1 commit into
kubernetes-sigs:mainfrom
harche:gcp-nvkind-k8s-1.37
Closed

harche wants to merge 1 commit into
kubernetes-sigs:mainfrom
harche:gcp-nvkind-k8s-1.37

Conversation

@harche

@harche harche commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Moves the GCP nvkind e2e harness (hack/ci/gcp-nvkind) from Kubernetes 1.34 to 1.37 so it can exercise newer kubelet features, such as DRA device health status (KEP-4680, ResourceHealthStatus, beta and on by default since 1.36), without extra feature gates.

Changing only K8S_VERSION isn't enough; two tooling issues block newer node images:

  • kubeadm config format. kind v0.31.0, which nvkind vendors, generates a kubeadm.k8s.io/v1beta3 config. kubeadm 1.37 rejects it ("Please use kubeadm v1.36 instead"). kind v0.32.0+ generates v1beta4 for 1.36+. This PR installs kind v0.33.0 and rebuilds nvkind against it, since nvkind still vendors v0.31.0 (Bump sigs.k8s.io/kind from 0.31.0 to 0.32.0 NVIDIA/nvkind#75 is open to bump it).
  • containerd runtime config. With 1.36+ images, nvkind's nvidia-ctk runtime configure --config-source=command writes a drop-in that containerd fails to restart with. Update nvidia-ctk --config-source flag from 'command' to 'file' NVIDIA/nvkind#76 switched to --config-source=file. This PR pins nvkind to c5705049 (current main), which includes it.

It also sets COPYFILE_DISABLE=1 and excludes ._* when packing the source. Without that, macOS tar adds AppleDouble files, and on a local run from a Mac, Helm fails parsing crds/._*.yaml as CRDs. Prow runs on Linux and is unaffected.

Which issue(s) this PR is related to:

N/A. Found while testing the KEP-4680 device health reporting rework (follow-up to #1243) on a real GPU.

Special notes for your reviewer:

  • Prow uses the shared copy of lib/ in kubernetes/test-infra (experiment/gcp-nvkind/lib/) and sets K8S_VERSION in the job config. For the Prow job to pick this up, setup-nvkind-node.sh and K8S_VERSION need the matching change there. I'll send that once this direction is agreed.
  • Validated with a local run on a GCE T4 (us-central1-b): nvkind cluster on kindest/node:v1.37.0, GPU Operator v26.3.1, driver built from source, and the Ginkgo suite passed (8/8).
  • Each problem above reproduced on its own: on 1.37 with the current harness, kubeadm init fails; on 1.36.4 with the current nvkind, containerd fails to restart.
  • AI disclosure: an AI coding assistant helped investigate, reproduce the runs, and draft this description.

Does this PR introduce a user-facing change?

NONE

Additional documentation (design docs, usage docs, etc.):


Kubernetes 1.37 dropped the kubeadm v1beta3 config that kind v0.31.0
generates, and newer kindest/node images ship a containerd that fails to
restart with the drop-in nvidia-ctk writes from --config-source=command.
Use kind v0.33.0 and rebuild nvkind at main, which includes NVIDIA/nvkind#76,
against it.

Also keep macOS tar from adding AppleDouble files to the source tarball;
Helm fails to parse them as CRDs on local runs from a Mac.
@kubernetes-prow

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@kubernetes-prow kubernetes-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Oct 5, 2026
@netlify

netlify Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for dra-driver-nvidia-gpu canceled.

Name Link
🔨 Latest commit b99ca1b
🔍 Latest deploy log https://app.netlify.com/projects/dra-driver-nvidia-gpu/deploys/6ac3d4c2f731d7000896b35f

@kubernetes-prow
kubernetes-prow Bot requested review from dims and shengnuo October 5, 2026 16:48
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: harche
Once this PR has been reviewed and has the lgtm label, please assign varunrsekar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 5, 2026
@harche

harche commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #1525.

@harche harche closed this Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant