Skip to content

feat: add CSE timing regression tests for all Linux VHDs (Ubuntu 22.04/24.04, Azure Linux V3) #8284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
djsly wants to merge 10 commits into
base: main
Choose a base branch
from
+714 −0

fix: address review comments on CSE timing

9af8911
Select commit
Loading
Failed to load commit list.
Open

feat: add CSE timing regression tests for all Linux VHDs (Ubuntu 22.04/24.04, Azure Linux V3) #8284

fix: address review comments on CSE timing
9af8911
Select commit
Loading
Failed to load commit list.
Azure Pipelines / Agentbaker E2E failed Apr 16, 2026 in 4m 5s

Build #20260416.9 had test failures

1 errors / 0 warnings

Details

Tests

  • Failed: 331 (90.19%)
  • Passed: 36 (9.81%)
  • Other: 0 (0.00%)
  • Total: 367

Annotations

Check failure on line 2004 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Build log #L2004 

Script failed with exit code: 1

Check failure on line 1 in Test_AzureLinuxV3_CSE_FullInstallPerformance/default

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_AzureLinuxV3_CSE_FullInstallPerformance/default 

Failed
Raw output 
=== RUN   Test_AzureLinuxV3_CSE_FullInstallPerformance/default
=== PAUSE Test_AzureLinuxV3_CSE_FullInstallPerformance/default
=== CONT  Test_AzureLinuxV3_CSE_FullInstallPerformance/default
    test_helpers.go:386: [11.441s] TAGS {Name:Test_AzureLinuxV3_CSE_FullInstallPerformance/default ImageName:AzureLinuxV3gen2 OS:azurelinux Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [15.943s] → running scenario...
    test_helpers.go:258: [30.114s] → preparing AKS node...
    vmss.go:452: [32.123s] → creating VMSS 6o2e-2026-04-16-azurelinuxv3csefullinstallperformancedefa...
    panic.go:262: [32.138s] ✓ creating VMSS 6o2e-2026-04-16-azurelinuxv3csefullinstallperformancedefa done (0.0s)
    panic.go:262: [32.147s] ✓ preparing AKS node done (2.0s)
    panic.go:262: [32.147s] ✓ running scenario done (16.2s)
--- FAIL: Test_AzureLinuxV3_CSE_FullInstallPerformance/default (32.15s)

Check failure on line 1 in Test_AzureLinuxV3_CSE_FullInstallPerformance

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_AzureLinuxV3_CSE_FullInstallPerformance 

Failed
Raw output 
=== RUN   Test_AzureLinuxV3_CSE_FullInstallPerformance
=== PAUSE Test_AzureLinuxV3_CSE_FullInstallPerformance
=== CONT  Test_AzureLinuxV3_CSE_FullInstallPerformance
--- FAIL: Test_AzureLinuxV3_CSE_FullInstallPerformance (32.15s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb2fcad]

goroutine 231 [running]:
testing.tRunner.func1.2({0x20282c0, 0x3adb0d0})
	/home/cloudtest/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.12.linux-amd64/src/testing/testing.go:1734 +0x21c
testing.tRunner.func1()
	/home/cloudtest/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.12.linux-amd64/src/testing/testing.go:1737 +0x35e
panic({0x20282c0?, 0x3adb0d0?})
	/home/cloudtest/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.12.linux-amd64/src/runtime/panic.go:792 +0x132
github.com/Azure/agentbaker/pkg/agent.(*agentBakerImpl).GetNodeBootstrapping(0xc0031e3850, {0xc000798e10?, 0x0?}, 0x0)
	/mnt/vss/_work/1/s/pkg/agent/bakerapi.go:40 +0x2d
github.com/Azure/agentbaker/e2e.createVMSSModel({0x27eddd8, 0xc00021a9a0}, 0xc0006f90e0)
	/mnt/vss/_work/1/s/e2e/vmss.go:330 +0x1e9
github.com/Azure/agentbaker/e2e.CreateVMSS({0x27eddd8, 0xc00021a9a0}, 0xc0006f90e0, {0xc000211b30, 0x2f})
	/mnt/vss/_work/1/s/e2e/vmss.go:458 +0x113
github.com/Azure/agentbaker/e2e.CreateVMSSWithRetry({0x27eddd8, 0xc00021a9a0}, 0xc0006f90e0)
	/mnt/vss/_work/1/s/e2e/vmss.go:428 +0x7d
github.com/Azure/agentbaker/e2e.ConfigureAndCreateVMSS({0x27eddd8?, 0xc00021a9a0?}, 0xc0006f90e0)
	/mnt/vss/_work/1/s/e2e/vmss.go:78 +0x25
github.com/Azure/agentbaker/e2e.prepareAKSNode({0x27eddd8, 0xc00021a9a0}, 0xc0006f90e0)
	/mnt/vss/_work/1/s/e2e/test_helpers.go:322 +0x8bb
github.com/Azure/agentbaker/e2e.runScenario({0x280e7c0, 0xc0008ccfc0}, 0xc0006f90e0)
	/mnt/vss/_work/1/s/e2e/test_helpers.go:242 +0x626
github.com/Azure/agentbaker/e2e.RunScenario.func2(0xc0008ccfc0)
	/mnt/vss/_work/1/s/e2e/test_helpers.go:88 +0xd3
testing.tRunner(0xc0008ccfc0, 0xc0003d06c0)
	/home/cloudtest/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.12.linux-amd64/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 46
	/home/cloudtest/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.12.linux-amd64/src/testing/testing.go:1851 +0x413
exit status 2

Check failure on line 1 in Test_Flatcar_CustomCATrust/default

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_Flatcar_CustomCATrust/default 

Failed
Raw output 
=== RUN   Test_Flatcar_CustomCATrust/default
=== PAUSE Test_Flatcar_CustomCATrust/default
=== CONT  Test_Flatcar_CustomCATrust/default
    azure.go:478: [0.000s] Looking up images in https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-flatcar-gen2/overview
    azure.go:567: [12.306s] Image version /subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/flatcargen2/versions/1.1776326662.11752 is already in region westus3
    vhd.go:324: [12.306s] got version by tag branch=refs/heads/main: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-flatcar-gen2/versions/1.1776326662.11752/overview
    test_helpers.go:386: [12.307s] TAGS {Name:Test_Flatcar_CustomCATrust/default ImageName:flatcargen2 OS:flatcar Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [15.941s] → running scenario...
    test_helpers.go:258: [30.114s] → preparing AKS node...
    vmss.go:452: [32.123s] → creating VMSS mkk3-2026-04-16-flatcarcustomcatrustdefault...

Check failure on line 1 in Test_Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled/default

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled/default 

Failed
Raw output 
=== RUN   Test_Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled/default
=== PAUSE Test_Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled/default
=== CONT  Test_Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled/default
    test_helpers.go:386: [10.441s] TAGS {Name:Test_Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled/default ImageName:2204gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [15.941s] → running scenario...
    aks_model.go:36: [16.781s] Available Kubernetes versions for location westus3:
        - 1.35
         - - 1.35.0
         - - first latest found, updating to: 1.35.0
         - - 1.35.1
         - - new latest found, updating to: 1.35.1
        - 1.34
         - - 1.34.0
         - - 1.34.1
         - - 1.34.2
         - - 1.34.3
         - - 1.34.4
        - 1.33
         - - 1.33.0
         - - 1.33.2
         - - 1.33.6
         - - 1.33.1
         - - 1.33.3
         - - 1.33.4
         - - 1.33.5
         - - 1.33.7
         - - 1.33.8
        - 1.32
         - - 1.32.0
         - - 1.32.2
         - - 1.32.3
         - - 1.32.6
         - - 1.32.1
         - - 1.32.10
         - - 1.32.11
         - - 1.32.4
         - - 1.32.5
         - - 1.32.7
         - - 1.32.8
         - - 1.32.9
        - 1.31
         - - 1.31.100
         - - 1.31.13
        - 1.30
         - - 1.30.100
         - - 1.30.101
        - 1.29
         - - 1.29.101
         - - 1.29.100
        Latest GA Kubernetes version for location westus3: 1.35.1
    cluster.go:68: [16.781s] → preparing cluster...
    cluster.go:238: [16.781s] → get or create cluster abe2e-latest-kubernetes-version-8a3e6...
    cluster.go:246: [17.092s] ✓ get or create cluster abe2e-latest-kubernetes-version-8a3e6 done (0.3s)
    aks_model.go:305: [18.317s] → adding firewall rules...
    aks_model.go:344: [19.603s] Creating subnet AzureFirewallSubnet in VNet aks-vnet-35791171
    aks_model.go:363: [22.861s] Created firewall subnet with ID: /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-latest-kubernetes-version-8a3e6_westus3/providers/Microsoft.Network/virtualNetworks/aks-vnet-35791171/subnets/AzureFirewallSubnet
    aks_model.go:377: [22.861s] Creating public IP abe2e-fw-pip
    aks_model.go:395: [23.392s] Created public IP with ID: /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-latest-kubernetes-version-8a3e6_westus3/providers/Microsoft.Network/publicIPAddresses/abe2e-fw-pip
    aks_model.go:289: [23.392s] Firewall rules configured successfully
    aks_model.go:413: [23.949s] Firewall private IP: 10.225.0.4
    aks_model.go:444: [23.949s] Adding route "vnet-local" to AKS route table "abe2e-fw-rt"
    aks_model.go:444: [24.319s] Adding route "default-route-to-firewall" to AKS route table "abe2e-fw-rt"
    aks_model.go:455: [24.625s] Successfully added firewall routes to AKS route table "abe2e-fw-rt"
    aks_model.go:456: [24.625s] ✓ adding firewall rules done (6.3s)
    kube.go:365: [24.625s] Creating daemonset debug-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    cluster.go:704: [24.625s] → collecting garbage VMSS...
    kube.go:365: [24.721s] Creating daemonset debugnonhost-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    cluster.go:754: [24.910s] → collecting garbage K8s nodes...
    cluster.go:782: [24.961s] deleted stale K8s node "2cq1-2026-04-16-randomvhdwithlatestkubernetesversionscrip000000" (VMSS "2cq1-2026-04-16-randomvhdwithlatestkubernetesversionscrip" not found)
    cluster.go:782: [24.983s] deleted stale K8s node "2uyq-2026-04-16-ubuntu2204gen2imagepullidentitybindingdis000000" (VMSS "2uyq-2026-04-16-ubuntu2204gen2imagepullidentitybindingdis" not found)
    cluster.go:782: [25.006s] deleted
View more details on Azure Pipelines