capz: add MAX_PODS knob to override kubelet max pods on Windows nodes#567
Conversation
|
|
k8s-ci-robot
added
the
tide/merge-method-squash
k8s-ci-robot
added
cncf-cla: no
|
Hi @rzlink. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/M
rzlink
force-pushed
the
fix/hyperv-maxpods-20
branch
from
ca890e3 to
e478e00
Compare
k8s-ci-robot
added
cncf-cla: yes
|
/ok-to-test |
1 similar comment
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
|
@zylxjtu Do you think that instead of detecting whether Hyper-V is enabled and setting |
+1 to this, let's try and then we can merge this if needed later |
rzlink
force-pushed
the
fix/hyperv-maxpods-20
branch
from
e478e00 to
a3699b8
Compare
|
Updated per @marosset's review: replaced the Hyper-V auto-default with a generic |
rzlink
changed the title
Replaces the Hyper-V auto-detected HYPERV_MAX_PODS with a generic MAX_PODS env var. Default is empty (no override); CI jobs that need a lower cap (e.g. the hyperv-serial-slow Prow jobs, where each UVM consumes host kernel pool kubelet eviction stats do not see) can set MAX_PODS in the job env. Derivation and per-VM-SKU recommendations: https://github.com/kubernetes-sigs/windows-testing-benchmarks/blob/benchmarks/hyperv-resource-comparison/benchmarks/hyperv-resource-comparison/docs/customer-guidance.md
rzlink
force-pushed
the
fix/hyperv-maxpods-20
branch
from
a3699b8 to
b027224
Compare
Each Hyper-V UVM consumes host non-paged kernel pool that kubelet eviction stats do not see; the default maxPods=110 causes HNS exhaustion and MemoryPressure cascades on 16 GiB workers, which shows up as flakes on capz-windows-master-hyperv and capz-windows-master-hyperv-serial-slow. Companion to kubernetes-sigs/windows-testing#567 which adds the generic MAX_PODS knob to run-capz-e2e.sh.
I would still prefer the explicitly setting of hyperv, "MAX_PODS" does not seem to necessarily be related with hyper-v |
|
/approve |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rzlink, zylxjtu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
…ption specs Removes 3 entries from the GINKGO_SKIP regex now that MAX_PODS=20 (kubernetes-sigs/windows-testing#567 + test-infra#37141) prevents the Hyper-V vmmem-overhead-induced MemoryPressure that caused them to be skipped. Validated 30/30 PASS on fresh CAPZ Hyper-V (WS2025, 2x D4s_v3, MAX_PODS=20) — 10 back-to-back rounds, 0 MemoryPressure events.
4 participants
What this PR does / why we need it
On Windows-on-Hyper-V worker nodes, every pod that lands on the
runhcs-wcow-hypervisorRuntimeClass spins up a Hyper-V UVM that consumes ~500 MiB of host non-paged kernel pool and substantial HNS state. This overhead is invisible to kubelet's pod-level stats (it lives invmmem.exe/vmwp.exe), so the default--max-pods=110allows enough pods to exhaust the kernel pool. Once that happens, new pods fail to start withHCN_E_ADDR_INVALID_OR_RESERVED(0x803b002f) and HNS returns0xe, cascading into MemoryPressure evictions,SchedulerPreemptionfailures, and Calico HNS endpoint cleanup issues. This is what makes thecapz-windows-master-hyperv*testgrid dashboards flake.This PR caps
maxPodsat 20 on Windows workers wheneverHYPERV=trueby templatingHYPERV_MAX_PODSinto the kubelet config for theKubeadmConfigTemplateof every Windows machine pool, defaulted inrun-capz-e2e.sh. SetHYPERV_MAX_PODS=""to opt out.Validation
Three back-to-back full provision → e2e → cleanup iterations against the upstream
capz-windows-master-hypervshape (uksouth, AKS mgmt + CAPZ workload, 1× CP + 2× WS2025 Hyper-VStandard_D4s_v3workers, ginkgo--nodes=4, focus/skip matching the Prow job verbatim):0x803b002fcascadesFor comparison, recent un-patched runs of the same Prow job have hundreds of
0x803b002fmarkers and 1–8 spec failures per run.Special notes for your reviewer
HYPERV_MAX_PODS=""disables injection).windows-base,windows-ci,windows-pr,gmsa-ci,gmsa-pr,shared-image-gallery-ci).HYPERV=false) are unaffected: the envsubst block expands to empty.Release note