Skip to content

[V2] Upgrade Kubernetes version to 1.34 in demo and dev environments#7190

Draft
pingsutw wants to merge 3 commits intov2from
upgrade-k8s-1.34
Draft

[V2] Upgrade Kubernetes version to 1.34 in demo and dev environments#7190
pingsutw wants to merge 3 commits intov2from
upgrade-k8s-1.34

Conversation

@pingsutw
Copy link
Copy Markdown
Member

@pingsutw pingsutw commented Apr 10, 2026

Why are the changes needed?

The demo/sandbox environment was using Kubernetes 1.29 (k3s v1.29.0-k3s1), which is outdated. Upgrading to 1.34 brings security patches, performance improvements, and access to newer Kubernetes features.

What changes were proposed in this pull request?

  • Bump k3s base image from v1.29.0-k3s1 to v1.34.6-k3s1 in docker/demo-bundled/Dockerfile.
  • Bump kind node image from v1.26.0 to v1.34.0 in executor/DEVELOPMENT.md and dataproxy/DEVELOPMENT.md.
  • Fix docker/demo-bundled/bin/k3d-entrypoint-cgroupv2.sh by replacing busybox xargs with the standalone xargs.

Why the cgroupv2 script change is needed

After bumping to rancher/k3s:v1.34.6-k3s1, make sandbox-run hung at Waiting for kubeconfig... because the container exited with code 1 before k3s ever started. Root cause:

  1. The k3d entrypoint runs /bin/k3d-entrypoint-cgroupv2.sh before launching k3s. That script (copy-pasted from moby/moby's dind helper) calls busybox xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || : to evacuate processes from the root cgroup into /init so that writing to cgroup.subtree_control does not fail with EBUSY.
  2. The busybox shipped inside rancher/k3s:v1.34.6-k3s1 was built without the xargs applet, so busybox xargs fails with xargs: applet not found. The || : swallows the error, so the script appears to continue.
  3. But because the processes were never actually evacuated, the next line — sed ... > /sys/fs/cgroup/cgroup.subtree_control — fails with sed: write error (EBUSY). set -o errexit is in effect, so the entrypoint exits 1. k3s never starts, the kubeconfig is never written, and make sandbox-run waits forever.
  4. The standalone /bin/xargs binary does exist in the k3s v1.34 image, so simply dropping the busybox prefix is enough — the cgroup evacuation then works and k3s boots normally.

Entrypoint log captured from the failing container before the fix:

[2026-04-10T06:55:06+00:00] Running k3d entrypoints...
[2026-04-10T06:55:06+00:00] Running /bin/k3d-entrypoint-cgroupv2.sh
[2026-04-10T06:55:06+00:00] [CgroupV2 Fix] Evacuating Root Cgroup ...
xargs: applet not found
sed: write error

How was this patch tested?

Rebuilt the demo image locally and ran make sandbox-run:

$ kubectl --context flyte-demo get nodes
NAME           STATUS   ROLES           AGE   VERSION
3494856c3ac2   Ready    control-plane   23s   v1.34.6+k3s1

$ kubectl --context flyte-demo get pods -n flyte
NAME                               READY   STATUS              RESTARTS   AGE
docker-registry-5b9f54d5d5-5wwpz   0/1     ContainerCreating   0          16s
flyte-binary-7b57dd88c6-dxcqg      0/1     Init:0/1            0          16s
flyte-console-69b86957f6-dpcd6     0/1     Running             0          16s
minio-6cb99cfc49-rbv5p             0/1     Running             0          16s

Cluster comes up on v1.34.6+k3s1, kubeconfig is written and merged into ~/.kube/config, and all demo workloads schedule normally.

Labels

  • changed: Kubernetes version upgrade in demo and dev environments.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.
  • main
    • Flyte 2 WIP #6583
      • [V2] Upgrade Kubernetes version to 1.34 in demo and dev environments 👈

Signed-off-by: Kevin Su <pingsutw@apache.org>
@github-actions github-actions bot mentioned this pull request Apr 9, 2026
3 tasks
pingsutw added 2 commits April 9, 2026 23:58
The k3s v1.34 base image ships a busybox build without the xargs applet,
which caused the cgroupv2 fix script to fail silently and then the sed
write to cgroup.subtree_control to fail, exiting the container before
k3s could start and write the kubeconfig.

Use the standalone /bin/xargs (present in the image) instead of
busybox xargs.

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw pingsutw self-assigned this Apr 10, 2026
@pingsutw pingsutw added this to the V2 GA milestone Apr 10, 2026
@pingsutw pingsutw marked this pull request as draft April 10, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant