operator: add startupProbe to nmstate-operator deployment - #1569
operator: add startupProbe to nmstate-operator deployment#1569mkowalski wants to merge 1 commit into
Conversation
The nmstate-operator container only gets ~40s to bind its health probe endpoint (initialDelaySeconds=10, periodSeconds=10, failureThreshold=3) before the liveness probe kills it. On clusters where startup is slow (e.g. restrictive network policies combined with a cluster-wide proxy delaying API/DNS access during initialization), the operator takes 50-75s to start listening on :8081 and enters a permanent CrashLoopBackOff. Since the deployment is managed by OLM (CSV), users cannot persistently relax the probes themselves. Apply the same pattern already used for the webhook and metrics operands: a startupProbe with failureThreshold=18 gives the container up to 3 minutes to initialize, after which the regular readiness/liveness probes (no initialDelaySeconds needed) take over. Assisted-By: Claude Fable 5 Signed-off-by: Mateusz Kowalski <mko@redhat.com>
kubevirt-prow
Bot
added
dco-signoff: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
Adds startup protection for slow nmstate-operator initialization across Helm and OLM deployments.
Changes:
- Adds a 3-minute startup probe.
- Removes initial delays from readiness and liveness probes.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
charts/kubernetes-nmstate/templates/operator.yaml |
Updates Helm probe configuration. |
bundle/manifests/kubernetes-nmstate-operator.clusterserviceversion.yaml |
Synchronizes OLM deployment probes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@mkowalski: The following tests failed, say
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/release-note-none |
kubevirt-prow
Bot
added
release-note-none
What this PR does
Adds a
startupProbeto thenmstate-operatorcontainer (helm chart template + OLM bundle CSV) and removesinitialDelaySecondsfrom its readiness/liveness probes, mirroring the pattern already applied to the webhook and metrics operands.Why
The operator container currently gets only ~40s to bind
:8081(initialDelaySeconds=10,periodSeconds=10,failureThreshold=3) before the liveness probe kills it. In environments where startup is slow — observed in the field at 50–75s on clusters with restrictive network policies and a cluster-wide proxy delaying API/DNS access during initialization — the operator never survives long enough to become ready and enters a permanent CrashLoopBackOff. Because the deployment is OLM-managed, any manual probe adjustment is reverted by the CSV reconciliation, so there is no persistent user-side workaround.The startupProbe (
failureThreshold=18,periodSeconds=10) gives the container up to 3 minutes to initialize; once it passes, the regular probes take over without initial delay.Downstream reference: https://issues.redhat.com/browse/OCPBUGS-94072 (the previous fix covered only the webhook/metrics operands, not the operator itself).
This PR was prepared with AI assistance. Please verify before acting on it.