OCPBUGS-94072: netpol: DNS egress policies with openshift-dns 5353 (includes missing #759 forward-port) - #777
Conversation
Add startupProbe to nmstate-webhook and kube-rbac-proxy containers to allow sufficient time for initialization before liveness/readiness probes begin checking. Without a startupProbe, pods that take longer than ~30s to start are killed by the liveness probe and enter CrashLoopBackOff. The startupProbe uses TCP socket checks with initialDelaySeconds=10, periodSeconds=10, and failureThreshold=18, giving containers up to 3 minutes to start. Once the startupProbe succeeds, regular liveness/readiness probes take over (no initialDelaySeconds needed since startup was already verified). Additionally, add DNS egress network policies (UDP/TCP 53) for webhook, metrics, and operator pods. The existing default-deny policies block all egress except TCP 6443 (API server), which prevents DNS resolution. While in-cluster API server access uses the KUBERNETES_SERVICE_HOST IP directly, various libraries and sidecars (kube-rbac-proxy) may attempt DNS lookups during initialization, causing startup delays that compound with aggressive probe timing. Signed-off-by: Mateusz Kowalski <mko@redhat.com> Generated-by: AI Signed-off-by: Mateusz Kowalski <mko@redhat.com> (cherry picked from commit 0fd6985)
The allow-{webhook,metrics,operator}-egress-dns NetworkPolicies only
allow egress on port 53. On OpenShift the dns-default service maps
port 53 to CoreDNS pods listening on port 5353, and OVN-Kubernetes
evaluates egress NetworkPolicy rules after service DNAT, so the
port-53 rule never matches the actual DNS traffic and lookups from
the selected pods are silently dropped. This is the documented
OpenShift NetworkPolicy pattern for DNS: allow UDP/TCP 5353 towards
the openshift-dns namespace.
Blocked DNS is the main contributor to the 50-75s startup stall of
the nmstate pods observed in OCPBUGS-94072 on a cluster with a
cluster-wide proxy, which together with tight probe timing results
in CrashLoopBackOff.
Keep the port-53 rule for vanilla Kubernetes deployments and gate
the 5353 rule on IsOpenShift, which is already provided as render
data to this template.
Assisted-By: Claude Fable 5
Signed-off-by: Mateusz Kowalski <mko@redhat.com>
openshift-ci-robot
added
jira/severity-critical
|
@mkowalski: This pull request references Jira Issue OCPBUGS-94072, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mkowalski The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/label jira/valid-bug |
openshift-ci
Bot
added
the
jira/valid-bug
|
/override-sticky ci/prow/security |
|
@mkowalski: Overrode contexts on behalf of mkowalski: ci/prow/security These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-operator-ovn-ipv4 |
|
/jira refresh |
|
/lgtm |
|
@rbbratta: This pull request references Jira Issue OCPBUGS-94072, which is invalid:
Comment Retaining the jira/valid-bug label as it was manually added. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
removed
the
jira/invalid-bug
|
@mkowalski: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
c0398f9
into
openshift:release-4.21
|
@mkowalski: Jira Issue OCPBUGS-94072: Some pull requests linked via external trackers have merged: The following pull request, linked via external tracker, has not merged:
All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-94072 has not been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
3 participants
What this PR does
Two commits:
openshift-dns) to the threeallow-*-egress-dnsNetworkPolicies.Why
On OpenShift the
dns-defaultservice maps port 53 to CoreDNS pods listening on 5353, and OVN-Kubernetes evaluates egress NetworkPolicy after service DNAT. The port-53allow-*-egress-dnsrules therefore never match real DNS traffic — DNS from the webhook, metrics and operator pods is silently dropped despite the allow policy. Blocked DNS is the main contributor to the 50–75s startup stall analysed in OCPBUGS-94072 (cluster-wide proxy environment), which combined with tight probe timing produces permanent CrashLoopBackOff.This adds the documented OpenShift DNS NetworkPolicy pattern: UDP/TCP 5353 towards the
openshift-dnsnamespace, gated on the existingIsOpenShiftrender flag. The port-53 rule is kept for vanilla Kubernetes.Upstream: nmstate#1570
Related probe fix: #773 #774 #775
This PR was prepared with AI assistance. Please verify before acting on it.