PMM-14979 Detect and reject a mismatched encryption key

documentation/docs/admin/security/data_encryption.md

@@ -12,17 +12,33 @@ For enhanced security control, PMM supports custom encryption keys.
 
 **Key format requirements:**
 
-- The key must be a 32-byte (256-bit) random value, suitable for AES-256-GCM encryption.
-- The file must contain exactly 32 raw bytes (not a hex-encoded or base64-encoded string).
+The key file must contain a base64-encoded Tink keyset created from the `AES256GCMKeyTemplate`. This is not a raw 32-byte value, so a key produced with a general-purpose tool such as `openssl rand` cannot be used: PMM fails to start if it cannot parse the keyset.
 
+Generate a key in the correct format with the Encryption Rotation Tool, which prints a new key to stdout without touching the database:
 
-PMM uses this key with the TINK `AES256GCMKeyTemplate` output prefix type.
+```bash
+pmm-encryption-rotation --generate-key
+```
 
-To set up a custom key, configure the `PMM_ENCRYPTION_KEY_PATH` environment variable to point to your custom key file.
+To set up a custom key, write that value to a file and point the `PMM_ENCRYPTION_KEY_PATH` environment variable at it.
 
 !!! hint alert alert-success "Important"
     Configure this **before** any data encryption occurs: either before upgrading to PMM 3 or before initially starting a new PMM 3.x instance.
 
+### High availability deployments
+
+All PMM Server nodes in a [highly available deployment](../../install-pmm/install-HA-clustered.md) share one PostgreSQL database, but each node reads its encryption key from its own local file. Every node must therefore use the **same** encryption key.
+
+A node holding a different key cannot decrypt credentials written by the other nodes. PMM detects this and refuses to start the affected node, because otherwise it would hand unusable credentials to PMM Clients and monitoring would stop for the affected services.
+
+Generate the key once, place it on every node before starting them, and back it up with the rest of your cluster configuration:
+
+```bash
+pmm-encryption-rotation --generate-key > pmm-encryption.key
+```
+
+To rotate the key in an HA cluster, run the [rotation procedure](#rotating-the-encryption-key) on a single node, then copy the resulting key file to all the other nodes and restart them.
+
 ### Key management requirements
 
 Once configured, PMM will use the custom key to encrypt and decrypt all sensitive data stored within the system.

documentation/docs/install-pmm/install-HA-clustered.md

@@ -617,6 +617,35 @@ certs:
       -----END DH PARAMETERS-----
 ```
 
+### Manage the encryption key
+
+PMM encrypts the credentials it stores for monitored services. All replicas share one PostgreSQL database, so they must all use the **same** encryption key: a replica holding a different key cannot decrypt credentials written by the others, and the affected services stop being monitored.
+
+The chart handles this for you. On installation it generates one key, stores it in a Kubernetes secret named `pg-encryption-key`, and mounts it into every replica at the path given by `PMM_ENCRYPTION_KEY_PATH`. Upgrades and rescaling reuse the existing key, so you do not need to configure anything.
+
+Two things follow from this:
+
+- **Back up the secret.** It is the only copy of the key. Without it, the credentials in a restored database cannot be decrypted:
+
+    ```sh
+    kubectl get secret pg-encryption-key -n pmm -o yaml > pg-encryption-key-backup.yaml
+    ```
+
+- **Keep the secret when reinstalling against existing data.** The secret is not owned by the Helm release and survives `helm uninstall`. If you delete it but keep the PostgreSQL data, a fresh installation generates a new key that cannot read the existing rows. Restore the backup before reinstalling:
+
+    ```sh
+    kubectl apply -f pg-encryption-key-backup.yaml
+    ```
+
+To supply your own key instead, create the secret before installing the chart:
+
+```sh
+kubectl create secret generic pg-encryption-key -n pmm \
+  --from-literal=key="$(pmm-encryption-rotation --generate-key)"
+```
+
+See [PMM data encryption](../admin/security/data_encryption.md) for the key format and rotation.
+
 ### Configure storage
 
 PMM HA stores data in distributed databases, not on the PMM server pods themselves. To increase storage capacity, configure the ClickHouse and VictoriaMetrics clusters.

documentation/docs/install-pmm/install-pmm-server/deployment-options/docker/preview_env_var.md

@@ -14,6 +14,13 @@
 | `PMM_HA_GRAFANA_GOSSIP_PORT`      | HA Grafana gossip port.
 | `PMM_HA_PEERS`                    | HA Peers.
 
+!!! caution alert alert-warning "All HA nodes must share one encryption key"
+    HA nodes share one PostgreSQL database, but each node reads its encryption key from its own `/srv/pmm-encryption.key`. A node that generated its own key cannot decrypt the credentials stored by the other nodes, and the services those credentials belong to stop being monitored.
+
+    Generate the key once with `pmm-encryption-rotation --generate-key`, place it at `/srv/pmm-encryption.key` on every node, and only then start them. With `PMM_HA_ENABLE` set, a node without a key file refuses to start rather than generating one of its own.
+
+    See [PMM data encryption](../../../../admin/security/data_encryption.md) for details.
+
 ## Available preview variables
 
 | Variable                          | Description

managed/cmd/pmm-managed-init/main.go

@@ -16,6 +16,7 @@
 package main
 
 import (
+	"fmt"
 	"os"
 	"strconv"
 
@@ -24,6 +25,7 @@ import (
 	"github.com/percona/pmm/managed/models"
 	"github.com/percona/pmm/managed/services/clickhouse"
 	"github.com/percona/pmm/managed/services/supervisord"
+	"github.com/percona/pmm/managed/utils/encryption"
 	"github.com/percona/pmm/managed/utils/env"
 	"github.com/percona/pmm/managed/utils/envvars"
 	"github.com/percona/pmm/utils/logger"
@@ -67,6 +69,12 @@ func main() {
 	isHAEnabled, _ := strconv.ParseBool(os.Getenv("PMM_HA_ENABLE"))
 	if isHAEnabled {
 		pmmConfigParams["AgentConfigFilePath"] = "/srv/pmm-agent/config/pmm-agent.yaml"
+
+		err = checkHAEncryptionKey()
+		if err != nil {
+			logrus.Errorf("Configuration error: %s", err)
+			os.Exit(1)
+		}
 	}
 
 	err = supervisord.SavePMMConfig(pmmConfigParams)
@@ -75,3 +83,26 @@ func main() {
 		os.Exit(1)
 	}
 }
+
+// checkHAEncryptionKey refuses to start an HA node that has no encryption key yet.
+//
+// All nodes of an HA cluster share one PostgreSQL database, but the encryption key is a local
+// file. Letting a node generate its own key leaves it unable to decrypt rows written by the
+// other nodes, and the credentials it hands to pmm-agent are then unusable. The key has to be
+// generated once and copied to every node before they start.
+func checkHAEncryptionKey() error {
+	path := encryption.KeyPath()
+
+	_, err := os.Stat(path)
+	switch {
+	case err == nil:
+		return nil
+	case os.IsNotExist(err):
+		return fmt.Errorf("encryption key %s not found. In HA mode all PMM Server nodes must share "+
+			"one encryption key, so it is never generated automatically. Generate it once with "+
+			"`pmm-encryption-rotation --generate-key`, place the output at %s on every node, then start them",
+			path, path)
+	default:
+		return fmt.Errorf("cannot read encryption key %s: %w", path, err)
+	}
+}

managed/cmd/pmm-managed-init/main_test.go

@@ -0,0 +1,47 @@
+// Copyright (C) 2023 Percona LLC
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/percona/pmm/managed/utils/encryption"
+)
+
+func TestCheckHAEncryptionKey(t *testing.T) {
+	t.Run("missing key is rejected", func(t *testing.T) {
+		path := filepath.Join(t.TempDir(), "pmm-encryption.key")
+		t.Setenv(encryption.CustomEncryptionKeyPathEnvVar, path)
+
+		err := checkHAEncryptionKey()
+		require.Error(t, err)
+		assert.Contains(t, err.Error(), path)
+		assert.Contains(t, err.Error(), "--generate-key")
+	})
+
+	t.Run("existing key is accepted", func(t *testing.T) {
+		path := filepath.Join(t.TempDir(), "pmm-encryption.key")
+		require.NoError(t, os.WriteFile(path, []byte("key"), 0o600))
+		t.Setenv(encryption.CustomEncryptionKeyPathEnvVar, path)
+
+		assert.NoError(t, checkHAEncryptionKey())
+	})
+}

managed/cmd/pmm-managed/main.go

@@ -107,6 +107,7 @@ import (
 	"github.com/percona/pmm/managed/services/vmalert"
 	"github.com/percona/pmm/managed/utils/clean"
 	"github.com/percona/pmm/managed/utils/distribution"
+	"github.com/percona/pmm/managed/utils/encryption"
 	"github.com/percona/pmm/managed/utils/envvars"
 	"github.com/percona/pmm/managed/utils/interceptors"
 	platformClient "github.com/percona/pmm/managed/utils/platform"
@@ -146,6 +147,15 @@ const (
 
 var pprofSemaphore = semaphore.NewWeighted(1)
 
+// mEncryptionKeyMismatch is set when this node's encryption key is not the one the database was
+// encrypted with. Standalone PMM keeps running in that state, so the condition needs to be
+// visible to monitoring rather than only present in the log.
+var mEncryptionKeyMismatch = prom.NewGauge(prom.GaugeOpts{
+	Namespace: "pmm_managed",
+	Name:      "encryption_key_mismatch",
+	Help:      "1 if the local encryption key does not match the key the database was encrypted with, 0 otherwise.",
+})
+
 func addLogsHandler(mux *http.ServeMux, logs *server.Logs) {
 	l := logrus.WithField("component", "logs.zip")
 
@@ -652,6 +662,32 @@ func migrateDB(ctx context.Context, sqlDB *sql.DB, params models.SetupDBParams)
 	}
 }
 
+// verifyEncryptionKey checks that this node holds the encryption key the database was encrypted
+// with.
+//
+// In HA the nodes share one database but each keeps its own key file, so a node holding a
+// different key cannot read the stored credentials and must not start: it would keep handing
+// undecryptable credentials to pmm-agent and write rows the other nodes cannot read. A
+// standalone node only logs the problem and exposes a metric, so that an upgrade cannot turn an
+// installation whose key went missing into one that no longer boots.
+func verifyEncryptionKey(l *logrus.Entry, db *reform.DB, haEnabled bool) {
+	err := models.VerifyEncryptionKey(db)
+	switch {
+	case err == nil:
+		mEncryptionKeyMismatch.Set(0)
+	case errors.Is(err, models.ErrEncryptionKeyMismatch):
+		mEncryptionKeyMismatch.Set(1)
+		if haEnabled {
+			l.Fatalf("%s. Every PMM Server node in an HA cluster must use the same encryption key: "+
+				"copy %s from a node that works and restart this one.", err, encryption.KeyPath())
+		}
+		l.Errorf("%s. Stored credentials cannot be decrypted, so monitoring will not work until the "+
+			"matching key is restored to %s.", err, encryption.KeyPath())
+	default:
+		l.Panicf("Failed to verify encryption key: %+v", err)
+	}
+}
+
 // newClickhouseDB return a new Clickhouse db.
 func newClickhouseDB(dsn string, maxIdleConns, maxOpenConns int) (*sql.DB, error) {
 	db, err := sql.Open("clickhouse", dsn)
@@ -900,6 +936,9 @@ func main() { //nolint:gocognit,maintidx,cyclop
 	prom.MustRegister(reformL)
 	db := reform.NewDB(sqlDB, postgresql.Dialect, reformL)
 
+	prom.MustRegister(mEncryptionKeyMismatch)
+	verifyEncryptionKey(l, db, *haEnabled)
+
 	// Generate unique PMM Server ID if it's not already.
 	err = models.SetPMMServerID(db)
 	if err != nil {