[oshoval]

What this PR does / why we need it:

Summary

Mount /run/multus as a directory in the dynamic-networks controller instead
of mounting multus.sock directly. This prevents the controller from staying
attached to a stale socket when Multus removes and recreates its socket during
a restart.

Problem

The controller's readiness and liveness probes connect to Multus through a
Unix socket. A file bind mount pins the socket inode that existed when the
pod was created. If Multus recreates the socket, the controller's mount still
points to the old inode, and restarting the controller container does not
refresh the pod's volume mount. The probes then fail and the controller can
crashloop.

This behavior was observed while investigating CNAO PR
#2989,
which upgrades Multus to v4.3.1. In the failing workflow, CNAO updates the
Multus configuration and deploys the controller in the same reconciliation.

Two Multus behaviors create the stale-mount window:

1. FilesystemPreRequirements
   removes and recreates the socket directory at daemon startup, producing a
   new socket inode.
2. Multus v4.3.1 added a 10-second graceful-shutdown delay
   (#1525,
   SigTermCancelAfter). The old daemon can keep its socket open while the
   controller pod is created and mounts it.

The stale-mount failure happens in this order:

1. The test redeploys Multus and the dynamic-networks controller together.
   Multus' old pod receives SIGTERM, but in v4.3.1 graceful shutdown keeps its
   daemon alive for about 10 seconds.
2. While the old socket still exists, kubelet mounts the socket file into the
   new controller pod. That mount is pinned to the old socket inode.
3. The new Multus daemon starts and
   FilesystemPreRequirements removes the entire /run/multus directory
   with os.RemoveAll, recreates it, and then creates a new socket there.
4. The controller's file mount still points to the old, removed socket inode
   instead of the new socket. Its probes fail, and restarting its container
   does not refresh the pod's mount.

Change

The controller now mounts the socket's parent directory:

- mountPath: /host/run/multus/multus.sock
+ mountPath: /host/run/multus
...
- path: /run/multus/multus.sock
- type: Socket
+ path: /run/multus
+ type: Directory

The directory mount stays valid when Multus recreates the socket, so the
controller resolves the current socket at its path.

Validation

On a local k8s-1.36 cluster, a rolling Multus restart reproduced the stale
socket mount on the unmodified controller: probes failed, and restarting the
container did not recover it. With the directory mount, the controller
recovered after Multus recreated the socket and remained ready.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer (optional):

Note:
Seems this repo still use old kind / k8s version via multus-cni, best to bump it on multus-cni,
but not relevant to this PR

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[oshoval]

Hi @maiqueb
beside the 4th commit which i thought required to fix what CNAO workflow e2e tests uncovered,
the rest are bumping pkgs and so

are you interested that i will drop the 4th commit and keep the rest ?
it will update the pkgs to 0.36.x, go to 1.26, and so on

so far it seems the multus fix is enough (in parallel, i am checking if i can break this assumption locally)

[maiqueb]

Hi @maiqueb beside the 4th commit which i thought required to fix what CNAO workflow e2e tests uncovered, the rest are bumping pkgs and so

are you interested that i will drop the 4th commit and keep the rest ? it will update the pkgs to 0.36.x, go to 1.26, and so on

so far it seems the multus fix is enough (in parallel, i am checking if i can break this assumption locally)

Yes, please drop the topmost commit.

[oshoval]

@maiqueb

created #319
(it doesnt have the commit we said to drop, it just focus on maintenance)

so this PR will keep the PR desc and the commit if i manage to simulate the bug

[oshoval]

rebased and reopen, seems i might managed to simulate the bug, drafting now
will update once i am sure