PMM-14979 Detect and reject a mismatched encryption key

[ademidoff]

Ticket number: PMM-14979

Feature build: Percona-Lab/pmm-submodules#4510

Important

Based on #5587 (PMM-15188) to avoid conflicts: both change managed/utils/encryption/encryption.go and managed/models/database.go. Retarget to main once #5587 merges. Review the diff against that base.

Problem

Reported on the forum against a 3-node Docker HA cluster: after pmm-agents reconnected, QAN and postgres_exporter data vanished for several PostgreSQL services, and agents logged

pq: password authentication failed for user "AQ+rKT/93psPSlwWLR8Qb0zsQqLDdhfXYNB9EBYk+507mw1Y"

That "username" is the encrypted value from the agents table. Followers only logged decryption: aead_factory: decryption failed at warning level.

In HA all nodes share one PostgreSQL database, but each reads its encryption key from a local file, and New() in managed/utils/encryption/encryption.go generates one whenever the file is absent. migrateDB encrypts on whichever node migrates first and records the columns in settings.EncryptedItems, so the other nodes skip encryption and keep an unrelated key.

The failure was silent because Decrypt returned the input ciphertext next to the error, and agentEncryption in managed/models/encryption_helpers.go logged a warning and assigned that value anyway. The ciphertext then flowed through Agent.DSN (managed/models/agent_model.go:543) into SetState.

A worse variant was reachable: if a follower won the migration race on a fresh database, the leader's later writes used a different key, leaving one table encrypted under two keys, which pmm-encryption-rotation cannot repair.

Changes

Fail closed rather than degrade

• Decrypt and Encrypt return an empty string on failure, so a caller that ignores the error cannot pass ciphertext off as plaintext or persist a secret unencrypted. The ciphertext is no longer interpolated into the base64 error message, which was putting secret material in logs.
• EncryptAgent/DecryptAgent return an error; all 20 call sites in managed/models/agent_helpers.go, managed/services/agents/service_info_broker.go and managed/services/realtimeanalytics/service.go propagate it. Errors name the agent and column.

Detect a mismatched key

• Settings.EncryptionKeyFingerprint plus encryption.Fingerprint(). Recorded in dbEncryption in the same transaction that marks the columns encrypted, so a concurrently starting node cannot see encrypted data with no fingerprint and adopt its own key. Cleared on the decrypt path, which keeps key rotation (decrypt -> new key -> encrypt) correct.
• models.VerifyEncryptionKey compares the local key at startup. Under HA a mismatch is fatal: the remedy is copying one file, and continuing means handing out unusable credentials. Standalone PMM logs it and sets pmm_managed_encryption_key_mismatch, so an installation whose key went missing does not stop booting on upgrade. Databases with no fingerprint adopt the local key only if it decrypts what is already stored.

Prevent it at the source

• pmm-managed-init refuses to start an HA node with no key file instead of letting one be generated per node. build/docker/server/entrypoint.sh runs under set -o errexit, so the non-zero exit does abort startup.

Documentation

• documentation/docs/admin/security/data_encryption.md described the key as 32 raw bytes; the code writes a base64-encoded Tink keyset, so anyone following it to pre-provision a shared key would have hit a startup failure. Corrected, with an HA section.
• Added key handling to documentation/docs/install-pmm/install-HA-clustered.md (which had no mention of encryption) and a warning to the Docker PMM_HA_* table in documentation/docs/install-pmm/install-pmm-server/deployment-options/docker/preview_env_var.md.

Related

• PMM-14979 Never replace an existing PMM encryption key percona-helm-charts#921 — chart side of the same ticket

Testing

Added: encrypt/decrypt round trip and a regression test that a value this node cannot decrypt surfaces as an error instead of being returned (managed/models/encryption_helpers_test.go); five fingerprint cases via sqlmock, including the reported follower scenario (managed/models/encryption_key_test.go); the HA preflight (managed/cmd/pmm-managed-init/main_test.go).

go build ./... and go vet ./... clean, golangci-lint reports 0 new issues against the base.

Not verified locally: testdb-backed suites need /srv/.postgres_password, so they fail with pq: password authentication failed here. Failure counts are identical on this branch and the base, but TestDefaultAgentEncryptionColumnsRoundTrip and the realtimeanalytics suite are unexercised against these changes and need CI.