OCPBUGS-94072: operator: add startupProbe to nmstate-operator deployment - #775
Conversation
|
@mkowalski: This pull request references Jira Issue OCPBUGS-94072, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mkowalski The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/label jira/valid-bug |
The previous fix for OCPBUGS-94072 added startupProbes only to the nmstate-webhook and kube-rbac-proxy (metrics) containers. The nmstate-operator container itself still only gets ~40s to bind its health probe endpoint (initialDelaySeconds=10, periodSeconds=10, failureThreshold=3) before the liveness probe kills it. In the affected environment the operator takes 50-75s to start listening on :8081, so it enters a permanent CrashLoopBackOff and OLM marks the CSV Failed/NeedsReinstall. Because the deployment is CSV-managed, manual probe edits are reverted by OLM. Apply the same pattern as the operands: startupProbe with failureThreshold=18 (up to 3 minutes to initialize), then regular readiness/liveness probes without initialDelaySeconds. Assisted-By: Claude Fable 5 Signed-off-by: Mateusz Kowalski <mko@redhat.com>
9268487 to
f9ef5a5
Compare
|
/lgtm |
|
@cybertron: Overrode contexts on behalf of cybertron: ci/prow/security DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-handler-ovn-ipv4 |
|
/jira refresh |
|
@rbbratta: This pull request references Jira Issue OCPBUGS-94072, which is invalid:
Comment Retaining the jira/valid-bug label as it was manually added. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/override ci/prow/security |
|
/override-sticky ci/prow/security |
|
@mkowalski: Overrode contexts on behalf of mkowalski: ci/prow/security These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@mkowalski: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3007dbe
into
openshift:release-4.20
|
@mkowalski: Jira Issue OCPBUGS-94072: Some pull requests linked via external trackers have merged: The following pull request, linked via external tracker, has not merged:
All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-94072 has not been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
What this PR does
Adds a
startupProbeto thenmstate-operatorcontainer (deploy template + bundle CSV + ART manifests) and removesinitialDelaySecondsfrom its readiness/liveness probes — the same pattern PR #759 applied to the webhook and kube-rbac-proxy operand containers.Why
The OCPBUGS-94072 fix shipped in 4.20.29 is incomplete: the customer is still down on 4.20.30/4.20.32, now with the operator pod in CrashLoopBackOff (900+ liveness kills observed in the must-gather host journals). The operator container gets only ~40s to bind
:8081(initialDelay 10s, period 10s, failureThreshold 3), while measured startup in the affected environment is 50–75s. The webhook/metrics pods — which did get startupProbes — recover fine in the same environment, proving the pattern works. Because the operator deployment is CSV-managed, OLM reverts any manual probe edits and marks the CSV Failed/NeedsReinstall.Evidence and full analysis in OCPBUGS-94072.
Upstream: nmstate#1569
This PR was prepared with AI assistance. Please verify before acting on it.