Skip to content

Updated credentials rotation topic #1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
n-boshnakov wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Updated credentials rotation topic #1004

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
99c0e23
Updated credentials rotation topic
n-boshnakov Jun 17, 2026
36ac829
Resolved comments
n-boshnakov Jun 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 34 additions & 44 deletions hugo/content/docs/getting-started/features/credential-rotation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,71 +13,61 @@ next: false
local: true
---

# Credential Rotation
## Overview

## Keys
Gardener deals with two distinct classes of credentials for Shoot clusters. They differ in scope, ownership, and how they are rotated:

There are plenty of keys in Gardener.
The ETCD needs one to store resources like secrets encrypted at rest.
Gardener generates certificate authorities (CAs) to ensure secured communication between the various components and actors and service account tokens are signed with a dedicated key.
There is also an SSH key pair to allow debugging of nodes and the observability stack has its own passwords too.
| Class | Examples | Scope | Who rotates |
|---|---|---|---|
| **Infrastructure credentials** | Cloud provider keys (AWS, Azure, GCP, OpenStack) | Project-scoped — shared across Shoots via `CredentialsBinding` | You |
| **Shoot credentials** | CAs, SSH key pair, ETCD encryption key, ServiceAccount signing key, observability passwords | Per-Shoot — generated and managed by Gardener | You, via `kubectl annotate` operations on the Shoot |

![keys](/docs/getting-started/features/images/keys.webp)
Infrastructure credentials are **not** part of the Shoot itself — they are `Secret`s in the garden cluster's project namespace, referenced by Shoots via a `CredentialsBinding`. Shoot credentials are generated per Shoot by Gardener and rotate through well-defined phases.
Comment on lines +20 to +25

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Clarify user versus Gardener responsibilities.

“Who rotates: You” and “You, via kubectl annotate” can imply that users perform the rotation mechanics. State explicitly that users choose the timing, update dependent clients, and trigger completion, while Gardener creates, distributes, and invalidates Gardener-managed credentials. Add a short decision guide for choosing between the two workflows.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 20 - 25, Update the credential responsibility table and surrounding
explanation to distinguish user actions from Gardener-managed rotation
mechanics: users choose timing, update dependent clients, and trigger
completion, while Gardener creates, distributes, and invalidates Shoot
credentials. Add a concise decision guide explaining when to use the
infrastructure-credentials workflow versus the Shoot-credentials workflow,
anchored to the existing “Infrastructure credentials” and “Shoot credentials”
sections.


All of these keys share a common property: they are managed by Gardener.
Rotating them, however, is potentially very disruptive.
Hence, Gardener does not do it automatically, but offers you means to perform these tasks easily.
For a single cluster, you may conveniently use the dashboard.
For step-by-step instructions, go directly to the relevant section in the [Credentials Rotation for Shoot Clusters](https://github.com/gardener/gardener/blob/master/docs/usage/shoot-operations/shoot_credentials_rotation.md) documentation:
- [Infrastructure credentials](https://github.com/gardener/gardener/blob/master/docs/usage/shoot-operations/shoot_credentials_rotation.md#infrastructure-credentials-project-scoped) (cloud provider keys)
- [Shoot credentials](https://github.com/gardener/gardener/blob/master/docs/usage/shoot-operations/shoot_credentials_rotation.md#shoot-credentials-gardener-managed) (CAs, SSH, ETCD, etc.)

![Prepare the rotation of all credentials from the Gardener dashboard](/docs/getting-started/features/images/prepare-rotation-of-all-credentials.webp)
## Two-Phase Rotation Model

Where possible, the rotation happens in two phases - Preparing and Completing.
For Gardener-managed credentials, rotation happens in two phases where possible.

### Prepare Rotation of All Credentials
![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)

The Preparing phase introduces new keys while the old ones are still valid.
Users can safely exchange keys / CA bundles wherever they are used.
It is possible to start the preparation by annotating the shoot resource accordingly:
In the **Preparing phase**, new credentials are created alongside the old ones — both sets are valid simultaneously.
This gives you time to update any API clients, kubeconfigs, or tooling that depend on the old credentials before they are invalidated.

```bash
kubectl -n <shoot-namespace> annotate shoot <shoot-name> gardener.cloud/operation=rotate-credentials-start
```
In the **Completing phase**, the old credentials are invalidated and only the new set remains.
You should only trigger this phase after all clients have been updated to use the new credentials.

### Complete Rotation of All Credentials
The shoot's status always reflects the current rotation phase, readable at `.status.credentials.rotation`.
Comment on lines +31 to +43

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add readiness checks and failure consequences to the two-phase workflow.

The page tells users to update clients before completing rotation, but does not identify how to verify readiness, what happens if a client is missed, or how to recover from a failed completion. This is especially risky for CA and kubeconfig consumers, where premature completion can break access.

Suggested addition 
 In the **Completing phase**, the old credentials are invalidated and only the new set remains.
 You should only trigger this phase after all clients have been updated to use the new credentials.
+Before completing, verify every affected client and confirm the rotation status is ready.
+If any client still uses the old credentials, it may lose access after completion; document the
+recovery procedure and whether another rotation is required.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## Two-Phase Rotation Model
Where possible, the rotation happens in two phases - Preparing and Completing.
For Gardener-managed credentials, rotation happens in two phases where possible.
### Prepare Rotation of All Credentials
![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)
The Preparing phase introduces new keys while the old ones are still valid.
Users can safely exchange keys / CA bundles wherever they are used.
It is possible to start the preparation by annotating the shoot resource accordingly:
In the **Preparing phase**, new credentials are created alongside the old ones — both sets are valid simultaneously.
This gives you time to update any API clients, kubeconfigs, or tooling that depend on the old credentials before they are invalidated.
```bash
kubectl -n <shoot-namespace> annotate shoot <shoot-name> gardener.cloud/operation=rotate-credentials-start
```
In the **Completing phase**, the old credentials are invalidated and only the new set remains.
You should only trigger this phase after all clients have been updated to use the new credentials.
### Complete Rotation of All Credentials
The shoot's status always reflects the current rotation phase, readable at `.status.credentials.rotation`.
## Two-Phase Rotation Model
For Gardener-managed credentials, rotation happens in two phases where possible.
![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)
In the **Preparing phase**, new credentials are created alongside the old ones — both sets are valid simultaneously.
This gives you time to update any API clients, kubeconfigs, or tooling that depend on the old credentials before they are invalidated.
In the **Completing phase**, the old credentials are invalidated and only the new set remains.
You should only trigger this phase after all clients have been updated to use the new credentials.
Before completing, verify every affected client and confirm the rotation status is ready.
If any client still uses the old credentials, it may lose access after completion; document the recovery procedure and whether another rotation is required.
The shoot's status always reflects the current rotation phase, readable at `.status.credentials.rotation`.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 31 - 43, Expand the “Two-Phase Rotation Model” section with concrete
readiness checks before completing rotation, including how to verify all CA and
kubeconfig consumers use the new credentials. Describe the access failures
caused by completing with a missed client and document the recovery path for a
failed completion, using the existing rotation status reference where
applicable.


Afterward, the Completing phase will invalidate the old keys / CA bundles.
Annotate the shoot resource accordingly:
You can also conveniently trigger rotation from the Gardener dashboard:

```bash
kubectl -n <shoot-namespace> annotate shoot <shoot-name> gardener.cloud/operation=rotate-credentials-complete
```
![Prepare the rotation of all credentials from the Gardener dashboard](./images/prepare-rotation-of-all-credentials.webp)

## Rotation Phases
## Automatic Rotation

![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)
Some Gardener-managed credential types support automatic rotation during the maintenance window via `.spec.maintenance.autoRotation.credentials`:

At the beginning, only the old set of credentials exists.
By triggering the rotation, new credentials are created in the Preparing phase and both sets are valid.
Now, all clients have to update and start using the new credentials.
Only afterward it is safe to trigger the Completing phase, which invalidates the old credentials.
- SSH key pair
- ETCD encryption key (enabled by default on new shoots)
- Observability passwords

The shoot's status will always show the current status / phase of the rotation.
Certificate authorities and the ServiceAccount signing key require user action between phases and therefore cannot be rotated automatically.
Comment on lines +49 to +57

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Document credential types that cannot be rotated.

The automatic-rotation list omits the OpenVPN TLS auth key, which the detailed guide identifies as not rotatable manually or automatically. Explicitly distinguish manual-only credentials from credentials that cannot be rotated at all, so users do not search for or attempt an unavailable procedure.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 49 - 57, Update the “Automatic Rotation” section to explicitly list the
OpenVPN TLS auth key as a credential that cannot be rotated manually or
automatically. Distinguish it from certificate authorities and the
ServiceAccount signing key, which require manual rotation between phases, and
preserve the existing automatic-rotation list.


For more information, see [Credentials Rotation for Shoot Clusters](/docs/gardener/shoot-operations/shoot_credentials_rotation/).
For configuration details, see [Automatic Credentials Rotation](https://github.com/gardener/gardener/blob/master/docs/usage/shoot/shoot_maintenance.md#automatic-credentials-rotation).

## User-Provided Credentials
## Infrastructure Credentials

![user-provided-keys](/docs/getting-started/features/images/user-provided-keys.webp)

You grant Gardener permissions to create resources by handing over cloud provider keys.
These keys are stored in a secret and referenced to a shoot via a SecretBinding.
Gardener uses the keys to create the network for the cluster resources, routes, VMs, disks, and IP addresses.
Infrastructure credentials are cloud provider keys you supply to Gardener so it can manage your cluster's infrastructure (networks, VMs, disks, load balancers).
These keys are stored in a `Secret` in the garden cluster's project namespace and referenced by your Shoot via a `CredentialsBinding`. A single `Secret` can be shared across multiple Shoots.

When you rotate credentials, the new keys have to be stored in the same secret and the shoot needs to reconcile successfully to ensure the replication to every controller.
Afterward, the old keys can be deleted safely from Gardener's perspective.

While the reconciliation can be triggered manually, there is no need for it (if you're not in a hurry).
Each shoot reconciles once within 24h and the new keys will be picked up during the next maintenance window.
When you rotate these credentials, you update the `Secret` with new keys, wait for all Shoots referencing that `Secret` to reconcile successfully, and only then deactivate the old keys in your cloud provider account.
Comment on lines +61 to +68

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add verification and failure handling for shared infrastructure credentials.

The instruction to “wait for all Shoots” is not actionable enough. Provide the expected reconciliation/status checks, timing expectations, and a procedure for handling one Shoot that fails to reconcile. Because the Secret may be shared, deactivating old cloud keys prematurely can disrupt multiple Shoots.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 61 - 68, Expand the credential-rotation guidance around “When you rotate
these credentials” with actionable verification: specify how to confirm every
referencing Shoot has reconciled successfully, the expected timing or polling
interval, and the status conditions to check. Add failure handling for an
individual Shoot, including investigating or remediating it and keeping the old
provider keys active until all shared-Shoot reconciliations succeed.


> [!NOTE]
> It is not possible to move a shoot to a different infrastructure account (at all!).
> It is not possible to move a Shoot to a different infrastructure account.

For the full rotation procedure, see [Infrastructure Credentials (Project-Scoped)](https://github.com/gardener/gardener/blob/master/docs/usage/shoot-operations/shoot_credentials_rotation.md#infrastructure-credentials-project-scoped) in the Credentials Rotation guide.
Loading