feat: cache versioned kubelet kubectl package binaries

[awesomenix]

What this does

This change avoids unnecessary kubelet/kubectl package installation work during CSE when the corresponding binaries are already
available on the VHD.

Today, Ubuntu and Mariner/Azure Linux cache the kubelet/kubectl PMC packages on the VHD, but CSE still installs them via the package
manager/runtime package flow. That has a few downsides:

• it re-runs package installation logic during provisioning
• on Ubuntu, dpkg -i triggers package postinst behavior for kubelet.service
• we pay extra provisioning latency even though AKS already owns kubelet service configuration and startup

This PR changes the flow so VHD build materializes versioned kubelet/kubectl binaries from the cached package artifacts into:

• /opt/bin/kubelet-<k8sVersion>
• /opt/bin/kubectl-<k8sVersion>

Then, during CSE, if the requested version is already present in /opt/bin, we reuse the existing cache-first path and do the final
rename into place instead of reinstalling from the package.

Changes

VHD build

• extend vhdbuilder/packer/install-dependencies.sh so cached kubelet/kubectl package artifacts also produce versioned binaries
  under /opt/bin
• support both:
  • Ubuntu: extract /usr/bin/<tool> from cached .deb
  • Mariner/Azure Linux: extract /usr/bin/<tool> from cached .rpm

CSE

• add a shared helper to detect whether versioned kubelet/kubectl binaries already exist in /opt/bin
• update Ubuntu and Mariner package-based install paths to:
  • use the cached versioned binaries when available
  • fall back to existing package install behavior when the cache is not present
  • continue respecting SHOULD_ENFORCE_KUBE_PMC_INSTALL=true

Tests

• add/extend coverage for the cache-first kubelet/kubectl flow
• extend Linux VHD content validation to verify package-backed kubelet/kubectl versions also materialize versioned binaries in
  /opt/bin

Why

This keeps kubelet/kubectl aligned with the existing kubernetes-binaries flow:

• cache versioned binaries on the image
• do a cheap final move during provisioning

Expected benefits:

• lower CSE latency
• avoid unnecessary package-manager work during provisioning
• avoid Ubuntu kubelet package postinst side effects when the VHD already has the requested binary

Notes

• this does not remove the existing package fallback path
• if the versioned binary is missing, CSE still falls back to the current package installation behavior
• SHOULD_ENFORCE_KUBE_PMC_INSTALL=true still forces the package path for validation / test scenarios

[Copilot]

Pull request overview

This PR optimizes Linux node provisioning by reusing kubelet/kubectl binaries already cached on the VHD (materialized as versioned files under /opt/bin) instead of reinstalling via the package manager during CSE.

Changes:

• VHD build: extract kubelet/kubectl binaries from cached .deb/.rpm artifacts into /opt/bin/<tool>-<k8sVersion>.
• CSE: add shared helpers to detect/move cached versioned kube binaries; update Ubuntu and Mariner package-based kubelet/kubectl install paths to prefer cache unless SHOULD_ENFORCE_KUBE_PMC_INSTALL=true.
• Tests: extend VHD content validation and add/extend ShellSpec coverage for the cache-first behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
vhdbuilder/packer/install-dependencies.sh	Adds VHD-build extraction of kubelet/kubectl binaries from cached package artifacts into versioned /opt/bin paths.
vhdbuilder/packer/test/linux-vhd-content-test.sh	Extends VHD content tests to validate versioned package-backed kubelet/kubectl binaries exist and match expected versions.
parts/linux/cloud-init/artifacts/cse_install.sh	Adds shared helpers to detect and move cached versioned kubelet/kubectl binaries; reuses them in the URL-based install flow.
parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh	Updates Ubuntu package-based kubelet/kubectl install path to prefer cached versioned binaries.
parts/linux/cloud-init/artifacts/mariner/cse_install_mariner.sh	Updates Mariner/AzureLinux package-based kubelet/kubectl install path to prefer cached versioned binaries.
spec/parts/linux/cloud-init/artifacts/cse_install_ubuntu_spec.sh	Adds ShellSpec coverage for Ubuntu cache-first kubelet/kubectl install behavior.
spec/parts/linux/cloud-init/artifacts/cse_install_mariner_spec.sh	Adds ShellSpec coverage for Mariner cache-first kubelet/kubectl install behavior.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

[Copilot]

This version parsing is much simpler than testKubeBinariesPresent and may diverge for version strings that include additional qualifiers (e.g., akslts, hotfix, underscore-delimited patches). If the VHD naming logic is intended to match testKubeBinariesPresent semantics, consider reusing the same normalization/parsing approach here (or extracting a shared helper) so the test doesn’t produce false negatives when package versions include those qualifiers.

[Copilot]

The PR description focuses on kubelet/kubectl caching and avoiding package installs, but this change also alters credential provider installation semantics (extracting a binary to a new target path rather than installing the package and symlinking /usr/bin). If this is intentional, it should be called out in the PR description (and any assumptions about runtime deps / postinst behavior should be validated). If not intentional, consider scoping this change back to kubelet/kubectl-only.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

[Copilot]

extractRPMBinaryFromFile assigns the extracted binary to binaryPath, but later tries to move ${sourceBinary}, which is undefined. This will fail at runtime and break kubelet/kubectl (and credential-provider) extraction. Replace ${sourceBinary} with the selected ${binaryPath} (and consider using install -m0755 to avoid cross-filesystem mv edge cases).

Suggested change

	mv "${sourceBinary}" "${targetPath}"
	chown root:root "${targetPath}"
	chmod 0755 "${targetPath}"
	install -m 0755 "${binaryPath}" "${targetPath}"

[Copilot]

The cache file naming on the VHD is /opt/bin/<tool>-<k8sVersion> (version stripped to X.Y.Z), but this helper looks for /opt/bin/<tool>-<packageVersion> verbatim. For RPMs, packageVersion may include a release suffix (e.g., 1.34.0-5.azl3), causing the cache-first path to be skipped even when /opt/bin/kubelet-1.34.0 exists. Normalize packageVersion to the intended <k8sVersion> (strip epoch and anything after the first -) and/or check both possible filenames.

[Copilot]

This switches azure-acr-credential-provider from an RPM install (which would ensure runtime dependencies are present) to “extract the binary from the RPM.” If the binary isn’t fully static or relies on RPM-triggered setup, this can break at runtime. If the intent is only to optimize kubelet/kubectl, keep credential-provider on the RPM install path; otherwise, add validation (e.g., dependency checks) and tests to guarantee the extracted binary works without installing the RPM.

[Copilot]

Using [[ ... =~ ${k8sVersion} ]] treats k8sVersion as a regex. Since Kubernetes versions contain dots, . will match any character, making this check more permissive than intended and potentially hiding mismatches. Prefer a fixed-string check (e.g., grep -F) or escape regex metacharacters before using =~.

Suggested change

	# shellcheck disable=SC3010
	if [[ ! ${versionOutput} =~ ${k8sVersion} ]]; then
	if ! printf '%s\n' "${versionOutput}" | grep -Fq -- "${k8sVersion}"; then

[Copilot]

The specs assert that installRPMPackageFromFile would call extractRPMBinaryFromFile, but logs_to_events is stubbed to only echo the command string and does not execute it. This means runtime issues inside extractRPMBinaryFromFile (e.g., the current undefined sourceBinary bug) won’t be caught by tests. Update the stub to execute the provided command (e.g., eval "$2") and assert on resulting filesystem effects, or directly unit-test extractRPMBinaryFromFile with mocked rpm2cpio/cpio.

Suggested change

	return 0
	eval "$2"

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

[Copilot]

Unquoted ${packageName} and ${desiredVersion} can cause word-splitting/globbing and also make grep interpret version characters as regex metacharacters. Quote variables and consider grep -F -- \"${desiredVersion}-\" (fixed string) to avoid regex surprises and reduce injection risk from unexpected input.

[Copilot]

The VHD build path extracts versioned kubelet/kubectl binaries for Mariner and Azure Linux (isMarinerOrAzureLinux), but this validation only runs for Ubuntu/Mariner. Expanding the condition to include Azure Linux (and relevant variants, if applicable) would ensure the new cache behavior is covered on all intended distros.