Skip to content

feat: add CSE timing regression tests for all Linux VHDs (Ubuntu 22.04/24.04, Azure Linux V3) #8284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
djsly wants to merge 10 commits into
base: main
Choose a base branch
from
+714 −0

fix: address review comments on CSE timing

17f1e2d
Select commit
Loading
Failed to load commit list.
Open

feat: add CSE timing regression tests for all Linux VHDs (Ubuntu 22.04/24.04, Azure Linux V3) #8284

fix: address review comments on CSE timing
17f1e2d
Select commit
Loading
Failed to load commit list.
Azure Pipelines / Agentbaker GPU E2E failed Apr 16, 2026 in 31m 45s

Build #20260416.2 had test failures

1 errors / 0 warnings

Details

Tests

  • Failed: 17 (8.06%)
  • Passed: 194 (91.94%)
  • Other: 0 (0.00%)
  • Total: 211

Annotations

Check failure on line 1597 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker GPU E2E

Build log #L1597 

Script failed with exit code: 1

Check failure on line 1 in Test_AzureLinuxV3_GPUAzureCNI/default

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker GPU E2E

Test_AzureLinuxV3_GPUAzureCNI/default 

Failed
Raw output 
=== RUN   Test_AzureLinuxV3_GPUAzureCNI/default
=== PAUSE Test_AzureLinuxV3_GPUAzureCNI/default
=== CONT  Test_AzureLinuxV3_GPUAzureCNI/default
    test_helpers.go:386: [0.000s] TAGS {Name:Test_AzureLinuxV3_GPUAzureCNI/default ImageName:AzureLinuxV3gen2 OS:azurelinux Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:true WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [0.000s] → running scenario...
    cluster.go:68: [0.000s] → preparing cluster...
    cluster.go:238: [0.001s] → get or create cluster abe2e-azure-network-v3-f69fa...
    cluster.go:246: [0.411s] ✓ get or create cluster abe2e-azure-network-v3-f69fa done (0.4s)
    aks_model.go:305: [0.483s] → adding firewall rules...
    aks_model.go:344: [1.179s] Creating subnet AzureFirewallSubnet in VNet aks-vnet-30633331
    aks_model.go:363: [1.756s] Created firewall subnet with ID: /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.Network/virtualNetworks/aks-vnet-30633331/subnets/AzureFirewallSubnet
    aks_model.go:377: [1.756s] Creating public IP abe2e-fw-pip
    aks_model.go:395: [2.046s] Created public IP with ID: /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.Network/publicIPAddresses/abe2e-fw-pip
    aks_model.go:289: [2.046s] Firewall rules configured successfully
    aks_model.go:413: [3.323s] Firewall private IP: 10.225.0.4
    aks_model.go:444: [3.323s] Adding route "vnet-local" to AKS route table "abe2e-fw-rt"
    aks_model.go:444: [3.787s] Adding route "default-route-to-firewall" to AKS route table "abe2e-fw-rt"
    aks_model.go:455: [4.127s] Successfully added firewall routes to AKS route table "abe2e-fw-rt"
    aks_model.go:456: [4.127s] ✓ adding firewall rules done (3.6s)
    kube.go:365: [4.127s] Creating daemonset debug-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    cluster.go:704: [4.127s] → collecting garbage VMSS...
    kube.go:365: [4.215s] Creating daemonset debugnonhost-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    cluster.go:754: [4.389s] → collecting garbage K8s nodes...
    cluster.go:782: [4.464s] deleted stale K8s node "win4sl4000000" (VMSS "win4sl4" not found)
    cluster.go:788: [4.464s] ✓ collecting garbage K8s nodes done (0.1s)
    cluster.go:746: [4.464s] ✓ collecting garbage VMSS done (0.3s)
    cluster.go:115: [4.464s] ✓ preparing cluster done (4.5s)
    test_helpers.go:258: [4.464s] → preparing AKS node...
    vmss.go:452: [4.464s] → creating VMSS 3wex-2026-04-16-azurelinuxv3gpuazurecnidefault...
    vmss.go:360: [5.314s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/3wex-2026-04-16-azurelinuxv3gpuazurecnidefault/overview
    vmss.go:366: [5.321s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-azure-network-v3-f69fa/overview
    vmss.go:485: [10.004s] VM will be automatically deleted after the test finishes, to preserve it for debugging purposes set KEEP_VMSS=true or pause the test with a breakpoint before the test finishes or failed
    vmss.go:489: [10.004s] SSH Instructions: (may take a few minutes for the VM to be ready for SSH)
        ========================
        az network bastion ssh --target-resource-id "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westu

Check failure on line 1 in Test_AzureLinuxV3_GPUAzureCNI_Scriptless/default

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker GPU E2E

Test_AzureLinuxV3_GPUAzureCNI_Scriptless/default 

Failed
Raw output 
=== RUN   Test_AzureLinuxV3_GPUAzureCNI_Scriptless/default
=== PAUSE Test_AzureLinuxV3_GPUAzureCNI_Scriptless/default
=== CONT  Test_AzureLinuxV3_GPUAzureCNI_Scriptless/default
    test_helpers.go:386: [0.000s] TAGS {Name:Test_AzureLinuxV3_GPUAzureCNI_Scriptless/default ImageName:AzureLinuxV3gen2 OS:azurelinux Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:true WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:true VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [0.000s] → running scenario...
    test_helpers.go:258: [4.396s] → preparing AKS node...
    vmss.go:452: [4.396s] → creating VMSS aitn-2026-04-16-azurelinuxv3gpuazurecniscriptlessdefault...
    vmss.go:360: [4.448s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/aitn-2026-04-16-azurelinuxv3gpuazurecniscriptlessdefault/overview
    vmss.go:366: [4.449s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-azure-network-v3-f69fa/overview
    vmss.go:485: [10.437s] VM will be automatically deleted after the test finishes, to preserve it for debugging purposes set KEEP_VMSS=true or pause the test with a breakpoint before the test finishes or failed
    vmss.go:489: [10.437s] SSH Instructions: (may take a few minutes for the VM to be ready for SSH)
        ========================
        az network bastion ssh --target-resource-id "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/aitn-2026-04-16-azurelinuxv3gpuazurecniscriptlessdefault/virtualMachines/0" --name "abe2e-azure-network-v3-f69fa-bastion" --resource-group MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3 --auth-type ssh-key --username azureuser --ssh-key /tmp/private-key-1143053464
        
    bastionssh.go:304: [748.497s] Attempt 1/5 establishing SSH over bastion to 10.224.1.120
    bastionssh.go:311: [749.072s] Attempt 1/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [759.073s] Attempt 2/5 establishing SSH over bastion to 10.224.1.120
    bastionssh.go:311: [759.640s] Attempt 2/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [769.641s] Attempt 3/5 establishing SSH over bastion to 10.224.1.120
    bastionssh.go:311: [770.211s] Attempt 3/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [780.212s] Attempt 4/5 establishing SSH over bastion to 10.224.1.120
    bastionssh.go:311: [780.776s] Attempt 4/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [790.777s] Attempt 5/5 establishing SSH over bastion to 10.224.1.120
    bastionssh.go:311: [791.354s] Attempt 5/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp:

Check failure on line 1 in Test_AzureLinuxV3_GPUAzureCNI_Scriptless

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker GPU E2E

Test_AzureLinuxV3_GPUAzureCNI_Scriptless 

Failed
Raw output 
=== RUN   Test_AzureLinuxV3_GPUAzureCNI_Scriptless
=== PAUSE Test_AzureLinuxV3_GPUAzureCNI_Scriptless
=== CONT  Test_AzureLinuxV3_GPUAzureCNI_Scriptless
--- FAIL: Test_AzureLinuxV3_GPUAzureCNI_Scriptless (0.03s)

Check failure on line 1 in Test_AzureLinuxV3_GPUAzureCNI/scriptless_nbc

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker GPU E2E

Test_AzureLinuxV3_GPUAzureCNI/scriptless_nbc 

Failed
Raw output 
=== RUN   Test_AzureLinuxV3_GPUAzureCNI/scriptless_nbc
=== PAUSE Test_AzureLinuxV3_GPUAzureCNI/scriptless_nbc
=== CONT  Test_AzureLinuxV3_GPUAzureCNI/scriptless_nbc
    test_helpers.go:386: [0.000s] TAGS {Name:Test_AzureLinuxV3_GPUAzureCNI/scriptless_nbc ImageName:AzureLinuxV3gen2 OS:azurelinux Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:true WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [0.003s] → running scenario...
    test_helpers.go:258: [4.454s] → preparing AKS node...
    vmss.go:452: [4.454s] → creating VMSS cm2v-2026-04-16-azurelinuxv3gpuazurecniscriptlessnbc...
    vmss.go:360: [5.220s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/cm2v-2026-04-16-azurelinuxv3gpuazurecniscriptlessnbc/overview
    vmss.go:366: [5.220s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-azure-network-v3-f69fa/overview
    vmss.go:485: [10.107s] VM will be automatically deleted after the test finishes, to preserve it for debugging purposes set KEEP_VMSS=true or pause the test with a breakpoint before the test finishes or failed
    vmss.go:489: [10.107s] SSH Instructions: (may take a few minutes for the VM to be ready for SSH)
        ========================
        az network bastion ssh --target-resource-id "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/cm2v-2026-04-16-azurelinuxv3gpuazurecniscriptlessnbc/virtualMachines/0" --name "abe2e-azure-network-v3-f69fa-bastion" --resource-group MC_abe2e-westus3_abe2e-azure-network-v3-f69fa_westus3 --auth-type ssh-key --username azureuser --ssh-key /tmp/private-key-1143053464
        
    bastionssh.go:304: [810.460s] Attempt 1/5 establishing SSH over bastion to 10.224.0.131
    bastionssh.go:311: [811.035s] Attempt 1/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [821.036s] Attempt 2/5 establishing SSH over bastion to 10.224.0.131
    bastionssh.go:311: [821.618s] Attempt 2/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [831.619s] Attempt 3/5 establishing SSH over bastion to 10.224.0.131
    bastionssh.go:311: [832.230s] Attempt 3/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [842.231s] Attempt 4/5 establishing SSH over bastion to 10.224.0.131
    bastionssh.go:311: [842.831s] Attempt 4/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com on 127.0.0.53:53: no such host
    bastionssh.go:304: [852.832s] Attempt 5/5 establishing SSH over bastion to 10.224.0.131
    bastionssh.go:311: [853.416s] Attempt 5/5 failed to create bastion tunnel: Post "https://bst-8f7cce2a-36c8-4089-b1b8-cdb112f69e9a.bastion.azure.com/api/tokens": dial tcp: lookup bst-8f7cce2a-36c8-40
View more details on Azure Pipelines