Skip to content

OCPBUGS-94072: operator: add startupProbe to nmstate-operator deployment - #775

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:release-4.20from
mkowalski:startup-probe-operator-4.20
Aug 13, 2026
Merged

OCPBUGS-94072: operator: add startupProbe to nmstate-operator deployment#775
openshift-merge-bot[bot] merged 1 commit into
openshift:release-4.20from
mkowalski:startup-probe-operator-4.20

Conversation

@mkowalski

Copy link
Copy Markdown

What this PR does

Adds a startupProbe to the nmstate-operator container (deploy template + bundle CSV + ART manifests) and removes initialDelaySeconds from its readiness/liveness probes — the same pattern PR #759 applied to the webhook and kube-rbac-proxy operand containers.

Why

The OCPBUGS-94072 fix shipped in 4.20.29 is incomplete: the customer is still down on 4.20.30/4.20.32, now with the operator pod in CrashLoopBackOff (900+ liveness kills observed in the must-gather host journals). The operator container gets only ~40s to bind :8081 (initialDelay 10s, period 10s, failureThreshold 3), while measured startup in the affected environment is 50–75s. The webhook/metrics pods — which did get startupProbes — recover fine in the same environment, proving the pattern works. Because the operator deployment is CSV-managed, OLM reverts any manual probe edits and marks the CSV Failed/NeedsReinstall.

Evidence and full analysis in OCPBUGS-94072.

Upstream: nmstate#1569


This PR was prepared with AI assistance. Please verify before acting on it.

@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 12, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@mkowalski: This pull request references Jira Issue OCPBUGS-94072, which is invalid:

  • expected the bug to be open, but it isn't
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Closed (Done) instead
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-97950 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does

Adds a startupProbe to the nmstate-operator container (deploy template + bundle CSV + ART manifests) and removes initialDelaySeconds from its readiness/liveness probes — the same pattern PR #759 applied to the webhook and kube-rbac-proxy operand containers.

Why

The OCPBUGS-94072 fix shipped in 4.20.29 is incomplete: the customer is still down on 4.20.30/4.20.32, now with the operator pod in CrashLoopBackOff (900+ liveness kills observed in the must-gather host journals). The operator container gets only ~40s to bind :8081 (initialDelay 10s, period 10s, failureThreshold 3), while measured startup in the affected environment is 50–75s. The webhook/metrics pods — which did get startupProbes — recover fine in the same environment, proving the pattern works. Because the operator deployment is CSV-managed, OLM reverts any manual probe edits and marks the CSV Failed/NeedsReinstall.

Evidence and full analysis in OCPBUGS-94072.

Upstream: nmstate#1569


This PR was prepared with AI assistance. Please verify before acting on it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 10605a02-25ae-40f7-b396-bd6f12542e7f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from emy and phoracek August 12, 2026 15:10
@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mkowalski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mkowalski

Copy link
Copy Markdown
Author

/label jira/valid-bug

@openshift-ci openshift-ci Bot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Aug 12, 2026
The previous fix for OCPBUGS-94072 added startupProbes only to the
nmstate-webhook and kube-rbac-proxy (metrics) containers. The
nmstate-operator container itself still only gets ~40s to bind its
health probe endpoint (initialDelaySeconds=10, periodSeconds=10,
failureThreshold=3) before the liveness probe kills it. In the
affected environment the operator takes 50-75s to start listening
on :8081, so it enters a permanent CrashLoopBackOff and OLM marks
the CSV Failed/NeedsReinstall. Because the deployment is
CSV-managed, manual probe edits are reverted by OLM.

Apply the same pattern as the operands: startupProbe with
failureThreshold=18 (up to 3 minutes to initialize), then regular
readiness/liveness probes without initialDelaySeconds.

Assisted-By: Claude Fable 5
Signed-off-by: Mateusz Kowalski <mko@redhat.com>
@mkowalski
mkowalski force-pushed the startup-probe-operator-4.20 branch from 9268487 to f9ef5a5 Compare August 12, 2026 15:35
@cybertron

Copy link
Copy Markdown
Member

/lgtm
/override ci/prow/security

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 12, 2026
@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown

@cybertron: Overrode contexts on behalf of cybertron: ci/prow/security

Details

In response to this:

/lgtm
/override ci/prow/security

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@mkowalski

Copy link
Copy Markdown
Author

/test e2e-handler-ovn-ipv4

@rbbratta

Copy link
Copy Markdown

/jira refresh

@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Aug 12, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@rbbratta: This pull request references Jira Issue OCPBUGS-94072, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-97950 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Retaining the jira/valid-bug label as it was manually added.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot

Copy link
Copy Markdown

/retest-required

Remaining retests: 0 against base HEAD 239a36a and 2 for PR HEAD f9ef5a5 in total

@rbbratta

Copy link
Copy Markdown

/override ci/prow/security

@mkowalski

Copy link
Copy Markdown
Author

/override-sticky ci/prow/security

@openshift-ci

openshift-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

@mkowalski: Overrode contexts on behalf of mkowalski: ci/prow/security

These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use /override-cancel to remove them.

Details

In response to this:

/override-sticky ci/prow/security

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

@mkowalski: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/security f9ef5a5 link true /test security

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 3007dbe into openshift:release-4.20 Aug 13, 2026
9 checks passed
@openshift-ci-robot

Copy link
Copy Markdown

@mkowalski: Jira Issue OCPBUGS-94072: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-94072 has not been moved to the MODIFIED state.

Details

In response to this:

What this PR does

Adds a startupProbe to the nmstate-operator container (deploy template + bundle CSV + ART manifests) and removes initialDelaySeconds from its readiness/liveness probes — the same pattern PR #759 applied to the webhook and kube-rbac-proxy operand containers.

Why

The OCPBUGS-94072 fix shipped in 4.20.29 is incomplete: the customer is still down on 4.20.30/4.20.32, now with the operator pod in CrashLoopBackOff (900+ liveness kills observed in the must-gather host journals). The operator container gets only ~40s to bind :8081 (initialDelay 10s, period 10s, failureThreshold 3), while measured startup in the affected environment is 50–75s. The webhook/metrics pods — which did get startupProbes — recover fine in the same environment, proving the pattern works. Because the operator deployment is CSV-managed, OLM reverts any manual probe edits and marks the CSV Failed/NeedsReinstall.

Evidence and full analysis in OCPBUGS-94072.

Upstream: nmstate#1569


This PR was prepared with AI assistance. Please verify before acting on it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mkowalski
mkowalski deleted the startup-probe-operator-4.20 branch August 13, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants