fix: missing globalmount dir and missing stage secrets by howels · Pull Request #1221 · kubernetes-csi/csi-driver-smb

howels · 2026-09-17T21:44:37Z

What type of PR is this?

What this PR does / why we need it:

When using this driver I wanted a robust SMB mount experience but when Pods died I was seeing issues with the globalmount directory missing and also with some failed mounts not able to recover if network conditions interrupted the SMB connection or if the local driver Pod experienced an error.
This PR solves these issues in my local testing.

Missing globalmount: kubelet can call NodePublishVolume while the staged CIFS mount is gone. The driver used to bind an empty directory into the pod. It now remounts CIFS at the staging path, or returns FailedPrecondition instead of publishing emptiness.

Plugin restart with no Stage secrets: kubelet does not pass nodeStageSecretRef on Publish. The process cache is lost when plugin dies. Publish now loads the optional secretname / secretnamespace volume attributes and fetches that Secret so restage still works after a crash. Existing PVs without those fields are unchanged.

Which issue(s) this PR fixes:

Fixes #

#737

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

Special notes for your reviewer:

Release note:

none

linux-foundation-easycla · 2026-09-17T21:44:44Z

The committers listed above are authorized under a signed CLA.

✅ login: howels / name: Stephen Howell (0473af6, 321785d, a339bd6, c3dd164, ce6f3a0, d5e3efc, e27bb58)

kubernetes-prow · 2026-09-17T21:44:44Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: howels
Once this PR has been reviewed and has the lgtm label, please assign msau42 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubernetes-prow · 2026-09-17T21:44:47Z

Welcome @howels!

It looks like this is your first PR to kubernetes-csi/csi-driver-smb 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-csi/csi-driver-smb has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

kubernetes-prow · 2026-09-17T21:44:47Z

Hi @howels. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

howels · 2026-09-21T10:54:04Z

Hello, please let me know if anything else is needed here.

andyzhangx · 2026-09-29T03:02:51Z

/ok-to-test

howels · 2026-09-29T22:04:56Z

Rebased to latest master.

howels · 2026-09-30T09:43:36Z

Looks like the Windows units tests are not maintained - using 1.16 and there is some issue with the tests themselves. Trivy scan fails due to usage limits on Trivy. Otherwise this appears to pass tests.
@andyzhangx please review.

howels · 2026-10-16T16:30:13Z

Please can I get a review here? @ZeroMagic @andyzhangx

My own testing shows that the additional credentials and checks make the csi-smb driver much more robust. This would be a quality-of-life improvement and reduce the likelihood of stuck mounts.

Copilot

🟡 Changes recommended

Recovery correctness, synchronization, Kerberos credential handling, and Windows behavior need correction before approval.

5 open issues

What changed in this PR

Adds SMB staging recovery during volume publication to address missing globalmount failures reported in #737.

Changes:

Detects missing staging mounts and attempts recovery before publishing.
Caches Stage credentials and adds optional recovery Secret lookup.
Adds recovery tests and configuration guidance.

File	Description
pkg/smb/stage_secrets.go	Adds credential cache helpers.
pkg/smb/stage_secrets_test.go	Tests cache isolation and deletion.
pkg/smb/smb.go	Initializes the credential cache.
pkg/smb/nodeserver.go	Adds staging recovery and stale-bind handling.
pkg/smb/nodeserver_test.go	Tests recovery and staging detection.
known-issues.md	Documents missing staging mounts and workarounds.
docs/driver-parameters.md	Documents recovery Secret options.

_{Review effort: Balanced}

Static PVs set nodeStageSecretRef only. NodePublishVolume has no credentials of its own. Store a process-local copy keyed by volume ID and staging path. Never log it. Drop it on NodeUnstageVolume.

mkdir of the staging path is not a fix. Point nodePublishSecretRef at the same Secret as nodeStageSecretRef so NodePublishVolume can restage after a node-plugin restart.

Use optional secretname and secretnamespace only when Publish must rebuild a missing staging mount after other credential sources are lost.

Releasing the lock after restage lets NodeUnstageVolume remove the share before the pod bind. Keep it through the stale-bind check and the bind.

A missing mount-table entry is a lookup failure. Treating it as a stale bind unmounts a healthy pod mount.

The previous note said a stale bind is unmounted only after restage. Publish checks the mounted target on every call.

Unrelated Secret data must not enter the process cache. Keep username, password, domain, and krb5cc_<uid>.

howels · 2026-10-19T10:23:23Z

/retest

kubernetes-prow · 2026-10-19T11:53:14Z

@howels: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-csi-driver-smb-external-e2e	`ce6f3a0`	link	true	`/test pull-csi-driver-smb-external-e2e`
pull-csi-driver-smb-e2e-windows-2022-hostprocess	`ce6f3a0`	link	true	`/test pull-csi-driver-smb-e2e-windows-2022-hostprocess`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

kubernetes-prow Bot added the release-note-none Denotes a PR that doesn't merit a release note. label Sep 17, 2026

kubernetes-prow Bot requested review from andyzhangx and jingxu97 September 17, 2026 21:44

kubernetes-prow Bot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Sep 17, 2026

kubernetes-prow Bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 17, 2026

howels mentioned this pull request Sep 17, 2026

Periodic failure to mount #302

Closed

howels changed the title ~~Fix globalmount missing~~ fix: missing globalmount dir and missing stage secrets Sep 17, 2026

kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Sep 18, 2026

howels force-pushed the fix-globalmount-missing branch from 9005e01 to 81c1682 Compare September 29, 2026 22:04

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 29, 2026

howels mentioned this pull request Oct 12, 2026

Possible old bug - getting reports of globalmount does not exist sporadically. #737

Closed

andyzhangx requested a balanced review from Copilot October 18, 2026 01:54

Copilot started reviewing on behalf of andyzhangx October 18, 2026 01:55

View session

Copilot AI reviewed Oct 18, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver_test.go

Comment thread pkg/smb/nodeserver.go Outdated

kubernetes-prow Bot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 18, 2026

howels added 7 commits October 19, 2026 11:06

fix: cache NodeStageVolume credentials for Publish restage

c3dd164

Static PVs set nodeStageSecretRef only. NodePublishVolume has no credentials of its own. Store a process-local copy keyed by volume ID and staging path. Never log it. Drop it on NodeUnstageVolume.

docs: document globalmount recovery and nodePublishSecretRef

0473af6

mkdir of the staging path is not a fix. Point nodePublishSecretRef at the same Secret as nodeStageSecretRef so NodePublishVolume can restage after a node-plugin restart.

fix: recover Stage credentials from volume attributes

e27bb58

Use optional secretname and secretnamespace only when Publish must rebuild a missing staging mount after other credential sources are lost.

fix: hold the staging lock until the pod bind finishes

321785d

Releasing the lock after restage lets NodeUnstageVolume remove the share before the pod bind. Keep it through the stale-bind check and the bind.

fix: keep a mounted target when the staging mount is not listed

d5e3efc

A missing mount-table entry is a lookup failure. Treating it as a stale bind unmounts a healthy pod mount.

docs: state that every publish compares the target with the staged share

a339bd6

The previous note said a stale bind is unmounted only after restage. Publish checks the mounted target on every call.

fix: cache only credential keys from the recovery Secret

ce6f3a0

Unrelated Secret data must not enter the process cache. Keep username, password, domain, and krb5cc_<uid>.

howels force-pushed the fix-globalmount-missing branch from a636c61 to ce6f3a0 Compare October 19, 2026 10:22

Conversation

howels commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kubernetes-prow Bot commented Sep 17, 2026

Uh oh!

kubernetes-prow Bot commented Sep 17, 2026

Uh oh!

kubernetes-prow Bot commented Sep 17, 2026

Uh oh!

howels commented Sep 21, 2026

Uh oh!

andyzhangx commented Sep 29, 2026

Uh oh!

howels commented Sep 29, 2026

Uh oh!

howels commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

howels commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

howels commented Oct 19, 2026

Uh oh!

kubernetes-prow Bot commented Oct 19, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

howels commented Sep 17, 2026 •

edited

Loading

linux-foundation-easycla Bot commented Sep 17, 2026 •

edited

Loading

howels commented Sep 30, 2026 •

edited

Loading

howels commented Oct 16, 2026 •

edited

Loading