Periodic CI job periodic-knmstate-e2e-handler-k8s-latest failed in run 2074312552950534144.
Analysis
All critical facts verified against source. The test at line 108-113 waits up to 5 minutes for lldpNeighbors(lldpdPod.Spec.NodeName, primaryNic) to be non-empty, while the lldpd pod readiness gate (line 72-77) only checks PodRunning phase — not that dnf install finished or lldpd is actually exchanging frames. Emitting the final report.
Summary
The Prow job periodic-knmstate-e2e-handler-k8s-latest (run 2074312552950534144) failed on a single e2e spec out of 101 (100 Passed, 1 Failed, 2 Pending):
LLDP configuration with nmpolicy [It] should enable LLDP on all ethernet interfaces that are up and show neighbors
test/e2e/handler/lldp_with_nmpolicy_test.go:113
The test passed its first assertion — LLDP was successfully enabled on eth0 across all nodes (lldp.enabled == "true"). It then failed the second assertion, which polls the node's reported NodeNetworkState for up to 5 minutes waiting for eth0 to report at least one LLDP neighbor, and timed out:
Interface eth0 should have lldp neighbors
Timed out after 300.00s
No handler crash, no nmstatectl error, no pod restart accompanied the failure. The lldpd helper pod started cleanly on node02 and ran until cleanup deleted it ~5 min later.
Root Cause
The failure is a timing/environmental race in the test's neighbor-discovery step, not a fault in kubernetes-nmstate.
The test's lldpd helper pod (test/e2e/handler/lldp_with_nmpolicy_test.go:53-77) is gated only on Kubernetes reporting PodRunning:
Command: []string{"/bin/bash"},
Args: []string{"-c", "dnf install -y lldpd && lldpd -d"},
...
}).WithTimeout(time.Minute).Should(Equal(corev1.PodRunning))PodRunning means the container process started — it does not mean dnf install -y lldpd finished, nor that the lldpd daemon is up and actively emitting LLDP frames on the wire. The test then immediately expects node02's own eth0 to receive and report a neighbor:
return lldpNeighbors(lldpdPod.Spec.NodeName, primaryNic) // same node running lldpd
For a neighbor to appear, the full chain must complete inside 5 minutes: Fedora mirror reachable → dnf install lldpd succeeds → lldpd -d starts → an LLDP frame is transmitted and reflected/received on eth0 → NetworkManager parses it → the handler surfaces it into NodeNetworkState.Status.CurrentState. Any slowness in the external dnf install (Fedora mirror latency) or LLDP's inherently slow/periodic advertisement timing can push discovery past the deadline. The absence of any restart event (default restartPolicy: Always) confirms dnf/lldpd did not fail-and-exit — the install was simply slow or the LLDP exchange hadn't occurred yet when the timer expired.
The handler behaved correctly throughout: it applied the LLDP-enable policy (reapply fell back to a clean deactivate/reactivate at 02:20:19), reported lldp.enabled == "true", and the neighbor value is a straight passthrough of nmstatectl show. A genuine code bug here would fail deterministically, not 2-of-5 times.
Classification
FLAKY_TEST
Evidence:
- Intermittent by record: the flakefinder 672h report (2026-06-29) lists this exact spec with a per-job result of
2 / 3 / 2 (failed / passed / skipped) — it passes the majority of runs on the same code.
- Isolated: 1 of 101 specs failed; the other 100 passed, including everything else in the same suite/node.
- External + timing dependency: relies on a live
dnf install -y lldpd from Fedora mirrors and on LLDP's slow periodic frame exchange, gated only on PodRunning.
- No code-side fault signal: no handler crash, no nmstatectl error,
lldp.enabled assertion passed; the node02 handler's later restarts (~03:54, API dial timeouts during node-reboot tests) are ~90 min after this 02:20 failure and unrelated.
Not INFRASTRUCTURE: the cluster, handler, and job were otherwise healthy (100 specs passed); nothing points to a systemic CI/infra outage — only this one externally-dependent, timing-sensitive spec tipped over its deadline.
Suggested Fix
Owned by the kubernetes-nmstate test suite (test/e2e/handler/lldp_with_nmpolicy_test.go). Options, most impactful first:
-
Remove the external package-install dependency. Bake lldpd into a preinstalled image instead of dnf install -y lldpd at runtime, so a slow/unavailable Fedora mirror can never eat into the discovery budget. Use a pinned image with lldpd already present and run lldpd -d directly.
-
Gate readiness on lldpd actually running, not just PodRunning. Before starting the neighbor poll, wait for confirmation the daemon is live inside the pod (e.g. kubectl exec … pgrep lldpd or lldpctl returning cleanly), so the 5-minute neighbor timer starts only once lldpd is genuinely transmitting.
-
If a runtime install is kept, separate its budget. Give dnf install its own explicit readiness wait so install latency is not silently charged against the 300s neighbor-discovery Eventually, and surface lldpd pod stdout on failure (currently the default-namespace pod logs aren't collected and DeferCleanup deletes the pod, which blocks post-mortem of whether install/daemon actually succeeded).
No changes to production handler/operator code are warranted — the reported network state was correct.
Note: one link remains uncertain but does not change the classification — I could not obtain the lldpd pod's stdout (the default-namespace pod logs weren't collected as artifacts, and DeferCleanup deletes the pod), so I can't definitively distinguish "dnf still installing at timeout" from "lldpd running but frame not yet received." Both branches are environmental/timing flakes, so the FLAKY_TEST classification holds either way.
🤖 Generated by oompa. Use /oompa to talk with me.
Periodic CI job periodic-knmstate-e2e-handler-k8s-latest failed in run 2074312552950534144.
Analysis
All critical facts verified against source. The test at line 108-113 waits up to 5 minutes for
lldpNeighbors(lldpdPod.Spec.NodeName, primaryNic)to be non-empty, while the lldpd pod readiness gate (line 72-77) only checksPodRunningphase — not thatdnf installfinished or lldpd is actually exchanging frames. Emitting the final report.Summary
The Prow job
periodic-knmstate-e2e-handler-k8s-latest(run2074312552950534144) failed on a single e2e spec out of 101 (100 Passed, 1 Failed, 2 Pending):The test passed its first assertion — LLDP was successfully enabled on
eth0across all nodes (lldp.enabled == "true"). It then failed the second assertion, which polls the node's reportedNodeNetworkStatefor up to 5 minutes waiting foreth0to report at least one LLDP neighbor, and timed out:No handler crash, no nmstatectl error, no pod restart accompanied the failure. The
lldpdhelper pod started cleanly onnode02and ran until cleanup deleted it ~5 min later.Root Cause
The failure is a timing/environmental race in the test's neighbor-discovery step, not a fault in kubernetes-nmstate.
The test's
lldpdhelper pod (test/e2e/handler/lldp_with_nmpolicy_test.go:53-77) is gated only on Kubernetes reportingPodRunning:PodRunningmeans the container process started — it does not meandnf install -y lldpdfinished, nor that thelldpddaemon is up and actively emitting LLDP frames on the wire. The test then immediately expectsnode02's owneth0to receive and report a neighbor:For a neighbor to appear, the full chain must complete inside 5 minutes: Fedora mirror reachable →
dnf install lldpdsucceeds →lldpd -dstarts → an LLDP frame is transmitted and reflected/received oneth0→ NetworkManager parses it → the handler surfaces it intoNodeNetworkState.Status.CurrentState. Any slowness in the externaldnf install(Fedora mirror latency) or LLDP's inherently slow/periodic advertisement timing can push discovery past the deadline. The absence of any restart event (defaultrestartPolicy: Always) confirmsdnf/lldpddid not fail-and-exit — the install was simply slow or the LLDP exchange hadn't occurred yet when the timer expired.The handler behaved correctly throughout: it applied the LLDP-enable policy (reapply fell back to a clean deactivate/reactivate at 02:20:19), reported
lldp.enabled == "true", and the neighbor value is a straight passthrough ofnmstatectl show. A genuine code bug here would fail deterministically, not 2-of-5 times.Classification
FLAKY_TEST
Evidence:
2 / 3 / 2(failed / passed / skipped) — it passes the majority of runs on the same code.dnf install -y lldpdfrom Fedora mirrors and on LLDP's slow periodic frame exchange, gated only onPodRunning.lldp.enabledassertion passed; thenode02handler's later restarts (~03:54, API dial timeouts during node-reboot tests) are ~90 min after this 02:20 failure and unrelated.Not INFRASTRUCTURE: the cluster, handler, and job were otherwise healthy (100 specs passed); nothing points to a systemic CI/infra outage — only this one externally-dependent, timing-sensitive spec tipped over its deadline.
Suggested Fix
Owned by the kubernetes-nmstate test suite (
test/e2e/handler/lldp_with_nmpolicy_test.go). Options, most impactful first:Remove the external package-install dependency. Bake
lldpdinto a preinstalled image instead ofdnf install -y lldpdat runtime, so a slow/unavailable Fedora mirror can never eat into the discovery budget. Use a pinned image withlldpdalready present and runlldpd -ddirectly.Gate readiness on lldpd actually running, not just
PodRunning. Before starting the neighbor poll, wait for confirmation the daemon is live inside the pod (e.g.kubectl exec … pgrep lldpdorlldpctlreturning cleanly), so the 5-minute neighbor timer starts only oncelldpdis genuinely transmitting.If a runtime install is kept, separate its budget. Give
dnf installits own explicit readiness wait so install latency is not silently charged against the 300s neighbor-discoveryEventually, and surfacelldpdpod stdout on failure (currently thedefault-namespace pod logs aren't collected andDeferCleanupdeletes the pod, which blocks post-mortem of whether install/daemon actually succeeded).No changes to production handler/operator code are warranted — the reported network state was correct.
Note: one link remains uncertain but does not change the classification — I could not obtain the
lldpdpod's stdout (thedefault-namespace pod logs weren't collected as artifacts, andDeferCleanupdeletes the pod), so I can't definitively distinguish "dnf still installing at timeout" from "lldpd running but frame not yet received." Both branches are environmental/timing flakes, so the FLAKY_TEST classification holds either way.🤖 Generated by oompa. Use
/oompato talk with me.