OCPBUGS-94072: operator: add startupProbe to nmstate-operator deployment - #774
OCPBUGS-94072: operator: add startupProbe to nmstate-operator deployment#774mkowalski wants to merge 1 commit into
Conversation
|
@mkowalski: This pull request references Jira Issue OCPBUGS-94072, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mkowalski The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/label jira/valid-bug |
The previous fix for OCPBUGS-94072 added startupProbes only to the nmstate-webhook and kube-rbac-proxy (metrics) containers. The nmstate-operator container itself still only gets ~40s to bind its health probe endpoint (initialDelaySeconds=10, periodSeconds=10, failureThreshold=3) before the liveness probe kills it. In the affected environment the operator takes 50-75s to start listening on :8081, so it enters a permanent CrashLoopBackOff and OLM marks the CSV Failed/NeedsReinstall. Because the deployment is CSV-managed, manual probe edits are reverted by OLM. Apply the same pattern as the operands: startupProbe with failureThreshold=18 (up to 3 minutes to initialize), then regular readiness/liveness probes without initialDelaySeconds. Assisted-By: Claude Fable 5 Signed-off-by: Mateusz Kowalski <mko@redhat.com>
81405ed to
09ed744
Compare
|
/override-sticky ci/prow/security |
|
@mkowalski: Overrode contexts on behalf of mkowalski: ci/prow/security These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-handler-ovn-ipv4 |
|
@mkowalski: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
What this PR does
Adds a
startupProbeto thenmstate-operatorcontainer (deploy template + bundle CSV + ART manifests) and removesinitialDelaySecondsfrom its readiness/liveness probes — the same pattern PR #759 applied to the webhook and kube-rbac-proxy operand containers.Why
The OCPBUGS-94072 fix shipped in 4.20.29 is incomplete: the customer is still down on 4.20.30/4.20.32, now with the operator pod in CrashLoopBackOff (900+ liveness kills observed in the must-gather host journals). The operator container gets only ~40s to bind
:8081(initialDelay 10s, period 10s, failureThreshold 3), while measured startup in the affected environment is 50–75s. The webhook/metrics pods — which did get startupProbes — recover fine in the same environment, proving the pattern works. Because the operator deployment is CSV-managed, OLM reverts any manual probe edits and marks the CSV Failed/NeedsReinstall.Evidence and full analysis in OCPBUGS-94072.
Upstream: nmstate#1569
This PR was prepared with AI assistance. Please verify before acting on it.