Skip to content

OCPBUGS-94072: operator: add startupProbe to nmstate-operator deployment - #774

Open
mkowalski wants to merge 1 commit into
openshift:release-4.21from
mkowalski:startup-probe-operator-4.21
Open

OCPBUGS-94072: operator: add startupProbe to nmstate-operator deployment#774
mkowalski wants to merge 1 commit into
openshift:release-4.21from
mkowalski:startup-probe-operator-4.21

Conversation

@mkowalski

Copy link
Copy Markdown

What this PR does

Adds a startupProbe to the nmstate-operator container (deploy template + bundle CSV + ART manifests) and removes initialDelaySeconds from its readiness/liveness probes — the same pattern PR #759 applied to the webhook and kube-rbac-proxy operand containers.

Why

The OCPBUGS-94072 fix shipped in 4.20.29 is incomplete: the customer is still down on 4.20.30/4.20.32, now with the operator pod in CrashLoopBackOff (900+ liveness kills observed in the must-gather host journals). The operator container gets only ~40s to bind :8081 (initialDelay 10s, period 10s, failureThreshold 3), while measured startup in the affected environment is 50–75s. The webhook/metrics pods — which did get startupProbes — recover fine in the same environment, proving the pattern works. Because the operator deployment is CSV-managed, OLM reverts any manual probe edits and marks the CSV Failed/NeedsReinstall.

Evidence and full analysis in OCPBUGS-94072.

Upstream: nmstate#1569


This PR was prepared with AI assistance. Please verify before acting on it.

@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 12, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@mkowalski: This pull request references Jira Issue OCPBUGS-94072, which is invalid:

  • expected the bug to be open, but it isn't
  • expected the bug to target either version "4.21.z." or "openshift-4.21.z.", but it targets "4.20.z" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Closed (Done) instead
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-97950 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is New instead
  • expected dependent Jira Issue OCPBUGS-97950 to target a version in 4.22.0, but it targets "4.21.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does

Adds a startupProbe to the nmstate-operator container (deploy template + bundle CSV + ART manifests) and removes initialDelaySeconds from its readiness/liveness probes — the same pattern PR #759 applied to the webhook and kube-rbac-proxy operand containers.

Why

The OCPBUGS-94072 fix shipped in 4.20.29 is incomplete: the customer is still down on 4.20.30/4.20.32, now with the operator pod in CrashLoopBackOff (900+ liveness kills observed in the must-gather host journals). The operator container gets only ~40s to bind :8081 (initialDelay 10s, period 10s, failureThreshold 3), while measured startup in the affected environment is 50–75s. The webhook/metrics pods — which did get startupProbes — recover fine in the same environment, proving the pattern works. Because the operator deployment is CSV-managed, OLM reverts any manual probe edits and marks the CSV Failed/NeedsReinstall.

Evidence and full analysis in OCPBUGS-94072.

Upstream: nmstate#1569


This PR was prepared with AI assistance. Please verify before acting on it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fa29f56c-a457-4ce7-8890-1a0b92b88537

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from emy and phoracek August 12, 2026 15:10
@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mkowalski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mkowalski

Copy link
Copy Markdown
Author

/label jira/valid-bug

@openshift-ci openshift-ci Bot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Aug 12, 2026
The previous fix for OCPBUGS-94072 added startupProbes only to the
nmstate-webhook and kube-rbac-proxy (metrics) containers. The
nmstate-operator container itself still only gets ~40s to bind its
health probe endpoint (initialDelaySeconds=10, periodSeconds=10,
failureThreshold=3) before the liveness probe kills it. In the
affected environment the operator takes 50-75s to start listening
on :8081, so it enters a permanent CrashLoopBackOff and OLM marks
the CSV Failed/NeedsReinstall. Because the deployment is
CSV-managed, manual probe edits are reverted by OLM.

Apply the same pattern as the operands: startupProbe with
failureThreshold=18 (up to 3 minutes to initialize), then regular
readiness/liveness probes without initialDelaySeconds.

Assisted-By: Claude Fable 5
Signed-off-by: Mateusz Kowalski <mko@redhat.com>
@mkowalski
mkowalski force-pushed the startup-probe-operator-4.21 branch from 81405ed to 09ed744 Compare August 12, 2026 15:35
@mkowalski

Copy link
Copy Markdown
Author

/override-sticky ci/prow/security

@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown

@mkowalski: Overrode contexts on behalf of mkowalski: ci/prow/security

These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use /override-cancel to remove them.

Details

In response to this:

/override-sticky ci/prow/security

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@mkowalski

Copy link
Copy Markdown
Author

/test e2e-handler-ovn-ipv4

@openshift-ci

openshift-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

@mkowalski: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/security 09ed744 link true /test security

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants