diff --git a/documentation/docs/admin/security/data_encryption.md b/documentation/docs/admin/security/data_encryption.md
index 5a4701368df..a4ffbe7486f 100644
--- a/documentation/docs/admin/security/data_encryption.md
+++ b/documentation/docs/admin/security/data_encryption.md
@@ -12,17 +12,33 @@ For enhanced security control, PMM supports custom encryption keys.
**Key format requirements:**
-- The key must be a 32-byte (256-bit) random value, suitable for AES-256-GCM encryption.
-- The file must contain exactly 32 raw bytes (not a hex-encoded or base64-encoded string).
+The key file must contain a base64-encoded Tink keyset created from the `AES256GCMKeyTemplate`. This is not a raw 32-byte value, so a key produced with a general-purpose tool such as `openssl rand` cannot be used: PMM fails to start if it cannot parse the keyset.
+Generate a key in the correct format with the Encryption Rotation Tool, which prints a new key to stdout without touching the database:
-PMM uses this key with the TINK `AES256GCMKeyTemplate` output prefix type.
+```bash
+pmm-encryption-rotation --generate-key
+```
-To set up a custom key, configure the `PMM_ENCRYPTION_KEY_PATH` environment variable to point to your custom key file.
+To set up a custom key, write that value to a file and point the `PMM_ENCRYPTION_KEY_PATH` environment variable at it.
!!! hint alert alert-success "Important"
Configure this **before** any data encryption occurs: either before upgrading to PMM 3 or before initially starting a new PMM 3.x instance.
+### High availability deployments
+
+All PMM Server nodes in a [highly available deployment](../../install-pmm/install-HA-clustered.md) share one PostgreSQL database, but each node reads its encryption key from its own local file. Every node must therefore use the **same** encryption key.
+
+A node holding a different key cannot decrypt credentials written by the other nodes. PMM detects this and refuses to start the affected node, because otherwise it would hand unusable credentials to PMM Clients and monitoring would stop for the affected services.
+
+Generate the key once, place it on every node before starting them, and back it up with the rest of your cluster configuration:
+
+```bash
+pmm-encryption-rotation --generate-key > pmm-encryption.key
+```
+
+To rotate the key in an HA cluster, run the [rotation procedure](#rotating-the-encryption-key) on a single node, then copy the resulting key file to all the other nodes and restart them.
+
### Key management requirements
Once configured, PMM will use the custom key to encrypt and decrypt all sensitive data stored within the system.
diff --git a/documentation/docs/install-pmm/install-HA-clustered.md b/documentation/docs/install-pmm/install-HA-clustered.md
index 704fe512ed2..bcd7748267d 100644
--- a/documentation/docs/install-pmm/install-HA-clustered.md
+++ b/documentation/docs/install-pmm/install-HA-clustered.md
@@ -617,6 +617,35 @@ certs:
-----END DH PARAMETERS-----
```
+### Manage the encryption key
+
+PMM encrypts the credentials it stores for monitored services. All replicas share one PostgreSQL database, so they must all use the **same** encryption key: a replica holding a different key cannot decrypt credentials written by the others, and the affected services stop being monitored.
+
+The chart handles this for you. On installation it generates one key, stores it in a Kubernetes secret named `pg-encryption-key`, and mounts it into every replica at the path given by `PMM_ENCRYPTION_KEY_PATH`. Upgrades and rescaling reuse the existing key, so you do not need to configure anything.
+
+Two things follow from this:
+
+- **Back up the secret.** It is the only copy of the key. Without it, the credentials in a restored database cannot be decrypted:
+
+ ```sh
+ kubectl get secret pg-encryption-key -n pmm -o yaml > pg-encryption-key-backup.yaml
+ ```
+
+- **Keep the secret when reinstalling against existing data.** The secret is not owned by the Helm release and survives `helm uninstall`. If you delete it but keep the PostgreSQL data, a fresh installation generates a new key that cannot read the existing rows. Restore the backup before reinstalling:
+
+ ```sh
+ kubectl apply -f pg-encryption-key-backup.yaml
+ ```
+
+To supply your own key instead, create the secret before installing the chart:
+
+```sh
+kubectl create secret generic pg-encryption-key -n pmm \
+ --from-literal=key="$(pmm-encryption-rotation --generate-key)"
+```
+
+See [PMM data encryption](../admin/security/data_encryption.md) for the key format and rotation.
+
### Configure storage
PMM HA stores data in distributed databases, not on the PMM server pods themselves. To increase storage capacity, configure the ClickHouse and VictoriaMetrics clusters.
diff --git a/documentation/docs/install-pmm/install-pmm-server/deployment-options/docker/preview_env_var.md b/documentation/docs/install-pmm/install-pmm-server/deployment-options/docker/preview_env_var.md
index a776bbcd417..69bb96abd7f 100644
--- a/documentation/docs/install-pmm/install-pmm-server/deployment-options/docker/preview_env_var.md
+++ b/documentation/docs/install-pmm/install-pmm-server/deployment-options/docker/preview_env_var.md
@@ -14,6 +14,13 @@
| `PMM_HA_GRAFANA_GOSSIP_PORT` | HA Grafana gossip port.
| `PMM_HA_PEERS` | HA Peers.
+!!! caution alert alert-warning "All HA nodes must share one encryption key"
+ HA nodes share one PostgreSQL database, but each node reads its encryption key from its own `/srv/pmm-encryption.key`. A node that generated its own key cannot decrypt the credentials stored by the other nodes, and the services those credentials belong to stop being monitored.
+
+ Generate the key once with `pmm-encryption-rotation --generate-key`, place it at `/srv/pmm-encryption.key` on every node, and only then start them. With `PMM_HA_ENABLE` set, a node without a key file refuses to start rather than generating one of its own.
+
+ See [PMM data encryption](../../../../admin/security/data_encryption.md) for details.
+
## Available preview variables
| Variable | Description
diff --git a/managed/cmd/pmm-managed-init/main.go b/managed/cmd/pmm-managed-init/main.go
index 8cff42b19d5..35484196403 100644
--- a/managed/cmd/pmm-managed-init/main.go
+++ b/managed/cmd/pmm-managed-init/main.go
@@ -16,6 +16,7 @@
package main
import (
+ "fmt"
"os"
"strconv"
@@ -24,6 +25,7 @@ import (
"github.com/percona/pmm/managed/models"
"github.com/percona/pmm/managed/services/clickhouse"
"github.com/percona/pmm/managed/services/supervisord"
+ "github.com/percona/pmm/managed/utils/encryption"
"github.com/percona/pmm/managed/utils/env"
"github.com/percona/pmm/managed/utils/envvars"
"github.com/percona/pmm/utils/logger"
@@ -67,6 +69,12 @@ func main() {
isHAEnabled, _ := strconv.ParseBool(os.Getenv("PMM_HA_ENABLE"))
if isHAEnabled {
pmmConfigParams["AgentConfigFilePath"] = "/srv/pmm-agent/config/pmm-agent.yaml"
+
+ err = checkHAEncryptionKey()
+ if err != nil {
+ logrus.Errorf("Configuration error: %s", err)
+ os.Exit(1)
+ }
}
err = supervisord.SavePMMConfig(pmmConfigParams)
@@ -75,3 +83,26 @@ func main() {
os.Exit(1)
}
}
+
+// checkHAEncryptionKey refuses to start an HA node that has no encryption key yet.
+//
+// All nodes of an HA cluster share one PostgreSQL database, but the encryption key is a local
+// file. Letting a node generate its own key leaves it unable to decrypt rows written by the
+// other nodes, and the credentials it hands to pmm-agent are then unusable. The key has to be
+// generated once and copied to every node before they start.
+func checkHAEncryptionKey() error {
+ path := encryption.KeyPath()
+
+ _, err := os.Stat(path)
+ switch {
+ case err == nil:
+ return nil
+ case os.IsNotExist(err):
+ return fmt.Errorf("encryption key %s not found. In HA mode all PMM Server nodes must share "+
+ "one encryption key, so it is never generated automatically. Generate it once with "+
+ "`pmm-encryption-rotation --generate-key`, place the output at %s on every node, then start them",
+ path, path)
+ default:
+ return fmt.Errorf("cannot read encryption key %s: %w", path, err)
+ }
+}
diff --git a/managed/cmd/pmm-managed-init/main_test.go b/managed/cmd/pmm-managed-init/main_test.go
new file mode 100644
index 00000000000..fc1db3e54ee
--- /dev/null
+++ b/managed/cmd/pmm-managed-init/main_test.go
@@ -0,0 +1,47 @@
+// Copyright (C) 2023 Percona LLC
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program. If not, see .
+
+package main
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+
+ "github.com/percona/pmm/managed/utils/encryption"
+)
+
+func TestCheckHAEncryptionKey(t *testing.T) {
+ t.Run("missing key is rejected", func(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "pmm-encryption.key")
+ t.Setenv(encryption.CustomEncryptionKeyPathEnvVar, path)
+
+ err := checkHAEncryptionKey()
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), path)
+ assert.Contains(t, err.Error(), "--generate-key")
+ })
+
+ t.Run("existing key is accepted", func(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "pmm-encryption.key")
+ require.NoError(t, os.WriteFile(path, []byte("key"), 0o600))
+ t.Setenv(encryption.CustomEncryptionKeyPathEnvVar, path)
+
+ assert.NoError(t, checkHAEncryptionKey())
+ })
+}
diff --git a/managed/cmd/pmm-managed/main.go b/managed/cmd/pmm-managed/main.go
index 2468e9778a0..749c1c2f98f 100644
--- a/managed/cmd/pmm-managed/main.go
+++ b/managed/cmd/pmm-managed/main.go
@@ -107,6 +107,7 @@ import (
"github.com/percona/pmm/managed/services/vmalert"
"github.com/percona/pmm/managed/utils/clean"
"github.com/percona/pmm/managed/utils/distribution"
+ "github.com/percona/pmm/managed/utils/encryption"
"github.com/percona/pmm/managed/utils/envvars"
"github.com/percona/pmm/managed/utils/interceptors"
platformClient "github.com/percona/pmm/managed/utils/platform"
@@ -146,6 +147,15 @@ const (
var pprofSemaphore = semaphore.NewWeighted(1)
+// mEncryptionKeyMismatch is set when this node's encryption key is not the one the database was
+// encrypted with. Standalone PMM keeps running in that state, so the condition needs to be
+// visible to monitoring rather than only present in the log.
+var mEncryptionKeyMismatch = prom.NewGauge(prom.GaugeOpts{
+ Namespace: "pmm_managed",
+ Name: "encryption_key_mismatch",
+ Help: "1 if the local encryption key does not match the key the database was encrypted with, 0 otherwise.",
+})
+
func addLogsHandler(mux *http.ServeMux, logs *server.Logs) {
l := logrus.WithField("component", "logs.zip")
@@ -652,6 +662,32 @@ func migrateDB(ctx context.Context, sqlDB *sql.DB, params models.SetupDBParams)
}
}
+// verifyEncryptionKey checks that this node holds the encryption key the database was encrypted
+// with.
+//
+// In HA the nodes share one database but each keeps its own key file, so a node holding a
+// different key cannot read the stored credentials and must not start: it would keep handing
+// undecryptable credentials to pmm-agent and write rows the other nodes cannot read. A
+// standalone node only logs the problem and exposes a metric, so that an upgrade cannot turn an
+// installation whose key went missing into one that no longer boots.
+func verifyEncryptionKey(l *logrus.Entry, db *reform.DB, haEnabled bool) {
+ err := models.VerifyEncryptionKey(db)
+ switch {
+ case err == nil:
+ mEncryptionKeyMismatch.Set(0)
+ case errors.Is(err, models.ErrEncryptionKeyMismatch):
+ mEncryptionKeyMismatch.Set(1)
+ if haEnabled {
+ l.Fatalf("%s. Every PMM Server node in an HA cluster must use the same encryption key: "+
+ "copy %s from a node that works and restart this one.", err, encryption.KeyPath())
+ }
+ l.Errorf("%s. Stored credentials cannot be decrypted, so monitoring will not work until the "+
+ "matching key is restored to %s.", err, encryption.KeyPath())
+ default:
+ l.Panicf("Failed to verify encryption key: %+v", err)
+ }
+}
+
// newClickhouseDB return a new Clickhouse db.
func newClickhouseDB(dsn string, maxIdleConns, maxOpenConns int) (*sql.DB, error) {
db, err := sql.Open("clickhouse", dsn)
@@ -900,6 +936,9 @@ func main() { //nolint:gocognit,maintidx,cyclop
prom.MustRegister(reformL)
db := reform.NewDB(sqlDB, postgresql.Dialect, reformL)
+ prom.MustRegister(mEncryptionKeyMismatch)
+ verifyEncryptionKey(l, db, *haEnabled)
+
// Generate unique PMM Server ID if it's not already.
err = models.SetPMMServerID(db)
if err != nil {
diff --git a/managed/models/agent_helpers.go b/managed/models/agent_helpers.go
index 20da2a24104..6df9932a8e8 100644
--- a/managed/models/agent_helpers.go
+++ b/managed/models/agent_helpers.go
@@ -296,7 +296,10 @@ func FindAgents(q *reform.Querier, filters AgentFilters) ([]*Agent, error) {
agents := make([]*Agent, len(structs))
for i, s := range structs {
- decryptedAgent := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ decryptedAgent, err := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ if err != nil {
+ return nil, err
+ }
agents[i] = &decryptedAgent
}
@@ -317,7 +320,13 @@ func FindAgentByID(q *reform.Querier, id string) (*Agent, error) {
}
return nil, err
}
- return new(DecryptAgent(*agent)), nil
+
+ decryptedAgent, err := DecryptAgent(*agent)
+ if err != nil {
+ return nil, err
+ }
+
+ return new(decryptedAgent), nil
}
// FindAgentsByIDs finds Agents by IDs.
@@ -339,7 +348,10 @@ func FindAgentsByIDs(q *reform.Querier, ids []string) ([]*Agent, error) {
res := make([]*Agent, len(structs))
for i, s := range structs {
- decryptedAgent := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ decryptedAgent, err := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ if err != nil {
+ return nil, err
+ }
res[i] = &decryptedAgent
}
return res, nil
@@ -392,7 +404,10 @@ func FindDBConfigForService(q *reform.Querier, serviceID string) (*DBConfig, err
res := make([]*Agent, len(structs))
for i, s := range structs {
- decryptedAgent := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ decryptedAgent, err := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ if err != nil {
+ return nil, err
+ }
res[i] = &decryptedAgent
}
@@ -420,7 +435,10 @@ func FindPMMAgentsRunningOnNode(q *reform.Querier, nodeID string) ([]*Agent, err
res := make([]*Agent, 0, len(structs))
for _, str := range structs {
- decryptedAgent := DecryptAgent(*str.(*Agent)) //nolint:forcetypeassert
+ decryptedAgent, err := DecryptAgent(*str.(*Agent)) //nolint:forcetypeassert
+ if err != nil {
+ return nil, err
+ }
res = append(res, &decryptedAgent)
}
@@ -465,7 +483,10 @@ func FindPMMAgentsForService(q *reform.Querier, serviceID string) ([]*Agent, err
}
res := make([]*Agent, 0, len(pmmAgentRecords))
for _, str := range pmmAgentRecords {
- decryptedAgent := DecryptAgent(*str.(*Agent)) //nolint:forcetypeassert
+ decryptedAgent, err := DecryptAgent(*str.(*Agent)) //nolint:forcetypeassert
+ if err != nil {
+ return nil, err
+ }
res = append(res, &decryptedAgent)
}
@@ -547,7 +568,10 @@ func FindAgentsForScrapeConfig(q *reform.Querier, pmmAgentID *string, pushMetric
res := make([]*Agent, len(allAgents))
for i, s := range allAgents {
- decryptedAgent := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ decryptedAgent, err := DecryptAgent(*s.(*Agent)) //nolint:forcetypeassert
+ if err != nil {
+ return nil, err
+ }
res[i] = &decryptedAgent
}
return res, nil
@@ -592,7 +616,12 @@ func FindPmmAgentIDToRunActionOrJob(pmmAgentID string, agents []*Agent) (string,
// UpdateAgent updates the Agent in the database.
func UpdateAgent(q *reform.Querier, agent *Agent) error {
- err := q.Update(new(EncryptAgent(*agent)))
+ encryptedAgent, err := EncryptAgent(*agent)
+ if err != nil {
+ return err
+ }
+
+ err = q.Update(new(encryptedAgent))
if err != nil {
return fmt.Errorf("failed to update Agent: %w", err)
}
@@ -706,12 +735,22 @@ func CreateNodeExporter(q *reform.Querier,
return nil, err
}
- encryptedAgent := EncryptAgent(*row)
+ encryptedAgent, err := EncryptAgent(*row)
+ if err != nil {
+ return nil, err
+ }
+
err = q.Insert(&encryptedAgent)
if err != nil {
return nil, err
}
- return new(DecryptAgent(encryptedAgent)), nil
+
+ decryptedAgent, err := DecryptAgent(encryptedAgent)
+ if err != nil {
+ return nil, err
+ }
+
+ return new(decryptedAgent), nil
}
// CreateExternalExporterParams params for add external exporter.
@@ -796,12 +835,22 @@ func CreateExternalExporter(q *reform.Querier, params *CreateExternalExporterPar
return nil, err
}
- encryptedAgent := EncryptAgent(*row)
+ encryptedAgent, err := EncryptAgent(*row)
+ if err != nil {
+ return nil, err
+ }
+
err = q.Insert(&encryptedAgent)
if err != nil {
return nil, err
}
- return new(DecryptAgent(encryptedAgent)), nil
+
+ decryptedAgent, err := DecryptAgent(encryptedAgent)
+ if err != nil {
+ return nil, err
+ }
+
+ return new(decryptedAgent), nil
}
// CreateAgentParams params for add common exporter.
@@ -998,12 +1047,22 @@ func CreateAgent(q *reform.Querier, agentType AgentType, params *CreateAgentPara
// do nothing
}
- encryptedAgent := EncryptAgent(trimUnicodeNilsInCertFiles(*row))
+ encryptedAgent, err := EncryptAgent(trimUnicodeNilsInCertFiles(*row))
+ if err != nil {
+ return nil, err
+ }
+
err = q.Insert(&encryptedAgent)
if err != nil {
return nil, err
}
- return new(DecryptAgent(encryptedAgent)), nil
+
+ decryptedAgent, err := DecryptAgent(encryptedAgent)
+ if err != nil {
+ return nil, err
+ }
+
+ return new(decryptedAgent), nil
}
func trimUnicodeNilsInCertFiles(agent Agent) Agent {
@@ -1431,13 +1490,23 @@ func ChangeAgent(q *reform.Querier, agentID string, params *ChangeAgentParams) (
row.RTAOptions.Merge(params.RTAOptions)
// need to encrypt Agent's sensitive data before update
- row = new(EncryptAgent(*row))
+ encryptedAgent, err := EncryptAgent(*row)
+ if err != nil {
+ return nil, err
+ }
+
+ row = new(encryptedAgent)
err = q.Update(row)
if err != nil {
return nil, err
}
- return new(DecryptAgent(*row)), nil
+ decryptedAgent, err := DecryptAgent(*row)
+ if err != nil {
+ return nil, err
+ }
+
+ return new(decryptedAgent), nil
}
// RemoveAgent removes Agent by ID.
diff --git a/managed/models/agent_helpers_test.go b/managed/models/agent_helpers_test.go
index 462b0420790..c86858bd94e 100644
--- a/managed/models/agent_helpers_test.go
+++ b/managed/models/agent_helpers_test.go
@@ -205,7 +205,9 @@ func TestAgentHelpers(t *testing.T) {
},
} {
if v, ok := str.(*models.Agent); ok {
- str = new(models.EncryptAgent(*v))
+ encrypted, err := models.EncryptAgent(*v)
+ require.NoError(t, err)
+ str = new(encrypted)
}
require.NoError(t, q.Insert(str))
}
diff --git a/managed/models/agent_model_test.go b/managed/models/agent_model_test.go
index c5b012975ba..eb1a2f7d819 100644
--- a/managed/models/agent_model_test.go
+++ b/managed/models/agent_model_test.go
@@ -588,7 +588,9 @@ func TestExporterURL(t *testing.T) {
},
} {
if v, ok := str.(*models.Agent); ok {
- str = new(models.EncryptAgent(*v))
+ encrypted, err := models.EncryptAgent(*v)
+ require.NoError(t, err)
+ str = new(encrypted)
}
require.NoError(t, q.Insert(str), "failed to INSERT %+v", str)
}
diff --git a/managed/models/database.go b/managed/models/database.go
index 7a3895d5aab..b390a42f365 100644
--- a/managed/models/database.go
+++ b/managed/models/database.go
@@ -1356,13 +1356,23 @@ func dbEncryption(tx *reform.TX, database string, items []encryption.Table,
return err
}
+ // The fingerprint is recorded in the same transaction as the encrypted column list, so a
+ // concurrently starting node cannot observe encrypted data with no fingerprint to check its
+ // own key against. Decrypting clears it, which is what lets key rotation record the new key.
encryptedItems := []string{}
+ fingerprint := ""
if expectedState {
encryptedItems = prepared
+
+ fingerprint, err = encryption.Fingerprint()
+ if err != nil {
+ return err
+ }
}
_, err = UpdateSettings(tx, &ChangeSettingsParams{
- EncryptedItems: encryptedItems,
+ EncryptedItems: encryptedItems,
+ EncryptionKeyFingerprint: &fingerprint,
})
if err != nil {
return err
diff --git a/managed/models/dsn_helpers_test.go b/managed/models/dsn_helpers_test.go
index 7fed57d57d7..71f4d72e21e 100644
--- a/managed/models/dsn_helpers_test.go
+++ b/managed/models/dsn_helpers_test.go
@@ -163,7 +163,9 @@ func TestFindDSNByServiceID(t *testing.T) {
},
} {
if v, ok := str.(*models.Agent); ok {
- str = new(models.EncryptAgent(*v))
+ encrypted, err := models.EncryptAgent(*v)
+ require.NoError(t, err)
+ str = new(encrypted)
}
require.NoError(t, q.Insert(str))
}
diff --git a/managed/models/encryption_helpers.go b/managed/models/encryption_helpers.go
index ba9aea666a0..aa979fd2648 100644
--- a/managed/models/encryption_helpers.go
+++ b/managed/models/encryption_helpers.go
@@ -18,113 +18,201 @@ package models
import (
"database/sql"
"encoding/json"
+ "errors"
+ "fmt"
- "github.com/sirupsen/logrus"
+ "gopkg.in/reform.v1"
"github.com/percona/pmm/managed/utils/encryption"
)
// EncryptAgent encrypt agent.
-func EncryptAgent(agent Agent) Agent {
+func EncryptAgent(agent Agent) (Agent, error) {
return agentEncryption(agent, encryption.Encrypt)
}
// DecryptAgent decrypt agent.
-func DecryptAgent(agent Agent) Agent {
+// An error means a stored value could not be decrypted, for example because this node's
+// encryption key differs from the one the data was encrypted with. The returned Agent is only
+// partially decrypted in that case and must not be used: passing it on is how ciphertext used
+// to reach pmm-agent in place of credentials.
+func DecryptAgent(agent Agent) (Agent, error) {
return agentEncryption(agent, encryption.Decrypt)
}
-func agentEncryption(agent Agent, handler func(string) (string, error)) Agent { //nolint:gocognit
- if agent.Username != nil {
- username, err := handler(*agent.Username)
+// ErrEncryptionKeyMismatch is returned when this node's encryption key is not the key the data
+// in the database was encrypted with.
+var ErrEncryptionKeyMismatch = errors.New("encryption key does not match the database")
+
+// VerifyEncryptionKey reports whether this node holds the encryption key the database was
+// encrypted with, and records the key's fingerprint when none is stored yet.
+//
+// Every node of an HA cluster shares one database but keeps its own key file. A node that
+// generated its own key cannot decrypt credentials written by the others, which previously
+// surfaced only as a decryption warning while unusable credentials were handed to pmm-agent.
+func VerifyEncryptionKey(q reform.DBTX) error {
+ fingerprint, err := encryption.Fingerprint()
+ if err != nil {
+ return err
+ }
+
+ settings, err := GetSettings(q)
+ if err != nil {
+ return err
+ }
+
+ if settings.EncryptionKeyFingerprint == "" {
+ // Either a fresh install or an upgrade from a version that did not record the
+ // fingerprint. This node's key is adopted only if it can read what is already stored.
+ err = checkStoredSecretsReadable(q, settings)
if err != nil {
- logrus.Warning(err)
+ return err
}
- agent.Username = &username
+
+ settings.EncryptionKeyFingerprint = fingerprint
+
+ return SaveSettings(q, settings)
+ }
+
+ if settings.EncryptionKeyFingerprint != fingerprint {
+ return fmt.Errorf("%w: this node's key fingerprint is %s, the database was encrypted with %s",
+ ErrEncryptionKeyMismatch, fingerprint, settings.EncryptionKeyFingerprint)
+ }
+
+ return nil
+}
+
+// checkStoredSecretsReadable decrypts one stored agent username to tell a matching key from a
+// foreign one on databases that carry no fingerprint yet.
+func checkStoredSecretsReadable(q reform.DBTX, settings *Settings) error {
+ if len(settings.EncryptedItems) == 0 {
+ // Nothing has been encrypted yet, so there is nothing that could contradict this key.
+ return nil
+ }
+
+ var username string
+ err := q.QueryRow("SELECT username FROM agents WHERE username IS NOT NULL AND username != '' LIMIT 1").Scan(&username)
+ switch {
+ case errors.Is(err, sql.ErrNoRows):
+ return nil
+ case err != nil:
+ return fmt.Errorf("failed to read stored agent credentials: %w", err)
}
- if agent.Password != nil {
- password, err := handler(*agent.Password)
+ _, err = encryption.Decrypt(username)
+ if err != nil {
+ return fmt.Errorf("%w: stored agent credentials cannot be decrypted with this node's key: %w",
+ ErrEncryptionKeyMismatch, err)
+ }
+
+ return nil
+}
+
+func agentEncryption(agent Agent, handler func(string) (string, error)) (Agent, error) { //nolint:gocognit
+ // The *string fields are shared with the caller's Agent, so a new pointer is assigned
+ // instead of writing through the existing one.
+ ptrField := func(name string, val *string) (*string, error) {
+ if val == nil {
+ return nil, nil //nolint:nilnil
+ }
+ res, err := handler(*val)
if err != nil {
- logrus.Warning(err)
+ return nil, fmt.Errorf("agent %s: %s: %w", agent.AgentID, name, err)
}
- agent.Password = &password
+ return &res, nil
}
- if agent.AgentPassword != nil {
- agentPassword, err := handler(*agent.AgentPassword)
+ strField := func(name string, val string) (string, error) {
+ res, err := handler(val)
if err != nil {
- logrus.Warning(err)
+ return "", fmt.Errorf("agent %s: %s: %w", agent.AgentID, name, err)
}
- agent.AgentPassword = &agentPassword
+ return res, nil
}
var err error
+
+ agent.Username, err = ptrField("username", agent.Username)
+ if err != nil {
+ return agent, err
+ }
+
+ agent.Password, err = ptrField("password", agent.Password)
+ if err != nil {
+ return agent, err
+ }
+
+ agent.AgentPassword, err = ptrField("agent_password", agent.AgentPassword)
+ if err != nil {
+ return agent, err
+ }
+
if !agent.AWSOptions.IsEmpty() {
- agent.AWSOptions.AWSAccessKey, err = handler(agent.AWSOptions.AWSAccessKey)
+ agent.AWSOptions.AWSAccessKey, err = strField("aws_options.access_key", agent.AWSOptions.AWSAccessKey)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
- agent.AWSOptions.AWSSecretKey, err = handler(agent.AWSOptions.AWSSecretKey)
+ agent.AWSOptions.AWSSecretKey, err = strField("aws_options.secret_key", agent.AWSOptions.AWSSecretKey)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
}
if !agent.AzureOptions.IsEmpty() {
- agent.AzureOptions.ClientID, err = handler(agent.AzureOptions.ClientID)
+ agent.AzureOptions.ClientID, err = strField("azure_options.client_id", agent.AzureOptions.ClientID)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
- agent.AzureOptions.ClientSecret, err = handler(agent.AzureOptions.ClientSecret)
+ agent.AzureOptions.ClientSecret, err = strField("azure_options.client_secret", agent.AzureOptions.ClientSecret)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
- agent.AzureOptions.SubscriptionID, err = handler(agent.AzureOptions.SubscriptionID)
+ agent.AzureOptions.SubscriptionID, err = strField("azure_options.subscription_id", agent.AzureOptions.SubscriptionID)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
- agent.AzureOptions.TenantID, err = handler(agent.AzureOptions.TenantID)
+ agent.AzureOptions.TenantID, err = strField("azure_options.tenant_id", agent.AzureOptions.TenantID)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
}
if !agent.MongoDBOptions.IsEmpty() {
- agent.MongoDBOptions.TLSCertificateKey, err = handler(agent.MongoDBOptions.TLSCertificateKey)
+ agent.MongoDBOptions.TLSCertificateKey, err = strField("mongo_options.tls_certificate_key", agent.MongoDBOptions.TLSCertificateKey)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
- agent.MongoDBOptions.TLSCertificateKeyFilePassword, err = handler(agent.MongoDBOptions.TLSCertificateKeyFilePassword)
+ agent.MongoDBOptions.TLSCertificateKeyFilePassword, err = strField(
+ "mongo_options.tls_certificate_key_file_password", agent.MongoDBOptions.TLSCertificateKeyFilePassword)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
}
if !agent.MySQLOptions.IsEmpty() {
- agent.MySQLOptions.TLSCert, err = handler(agent.MySQLOptions.TLSCert)
+ agent.MySQLOptions.TLSCert, err = strField("mysql_options.tls_cert", agent.MySQLOptions.TLSCert)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
- agent.MySQLOptions.TLSKey, err = handler(agent.MySQLOptions.TLSKey)
+ agent.MySQLOptions.TLSKey, err = strField("mysql_options.tls_key", agent.MySQLOptions.TLSKey)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
}
if !agent.PostgreSQLOptions.IsEmpty() {
- agent.PostgreSQLOptions.SSLCert, err = handler(agent.PostgreSQLOptions.SSLCert)
+ agent.PostgreSQLOptions.SSLCert, err = strField("postgresql_options.ssl_cert", agent.PostgreSQLOptions.SSLCert)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
- agent.PostgreSQLOptions.SSLKey, err = handler(agent.PostgreSQLOptions.SSLKey)
+ agent.PostgreSQLOptions.SSLKey, err = strField("postgresql_options.ssl_key", agent.PostgreSQLOptions.SSLKey)
if err != nil {
- logrus.Warning(err)
+ return agent, err
}
}
- return agent
+ return agent, nil
}
// EncryptAWSOptionsHandler returns encrypted AWS Options.
diff --git a/managed/models/encryption_helpers_test.go b/managed/models/encryption_helpers_test.go
index eac032f1fa3..1a99256be8b 100644
--- a/managed/models/encryption_helpers_test.go
+++ b/managed/models/encryption_helpers_test.go
@@ -18,6 +18,7 @@ package models_test
import (
"context"
"database/sql"
+ "encoding/base64"
"encoding/json"
"path/filepath"
"testing"
@@ -90,6 +91,63 @@ func TestDefaultAgentEncryptionColumnsRoundTrip(t *testing.T) {
assert.Equal(t, original, readAgentSecrets(ctx, t, sqlDB))
}
+// TestEncryptDecryptAgentRoundTrip covers the happy path and the invariant that the caller's
+// Agent is left untouched: the *string fields are shared with the caller, so the handlers must
+// replace the pointers instead of writing through them.
+func TestEncryptDecryptAgentRoundTrip(t *testing.T) {
+ agent := models.Agent{
+ AgentID: "/agent_id/1",
+ Username: new("username"),
+ Password: new("password"),
+ MySQLOptions: models.MySQLOptions{
+ TLSCert: "mysql-tls-cert",
+ TLSKey: "mysql-tls-key",
+ },
+ }
+
+ encrypted, err := models.EncryptAgent(agent)
+ require.NoError(t, err)
+ require.NotNil(t, encrypted.Username)
+ assert.NotEqual(t, "username", *encrypted.Username)
+ assert.NotEqual(t, "mysql-tls-cert", encrypted.MySQLOptions.TLSCert)
+
+ require.NotNil(t, agent.Username)
+ assert.Equal(t, "username", *agent.Username, "input agent must not be mutated")
+
+ decrypted, err := models.DecryptAgent(encrypted)
+ require.NoError(t, err)
+ require.NotNil(t, decrypted.Username)
+ require.NotNil(t, decrypted.Password)
+ assert.Equal(t, "username", *decrypted.Username)
+ assert.Equal(t, "password", *decrypted.Password)
+ assert.Equal(t, "mysql-tls-cert", decrypted.MySQLOptions.TLSCert)
+ assert.Equal(t, "mysql-tls-key", decrypted.MySQLOptions.TLSKey)
+}
+
+// TestDecryptAgentDoesNotReturnCiphertext guards the fix for
+// https://perconadev.atlassian.net/browse/PMM-14979: a value that this node's key cannot
+// decrypt must surface as an error, and the undecrypted value must not be handed back to the
+// caller. It previously came back as the field value with only a warning logged, so the
+// ciphertext reached pmm-agent as a username and produced
+// `password authentication failed for user "AQ+rKT/..."`.
+func TestDecryptAgentDoesNotReturnCiphertext(t *testing.T) {
+ // Valid base64 but not ciphertext produced by this node's key, which is what an HA
+ // follower reads when the row was encrypted with another node's key.
+ foreignCiphertext := base64.StdEncoding.EncodeToString([]byte("encrypted-with-another-key"))
+
+ agent := models.Agent{
+ AgentID: "/agent_id/1",
+ Username: new(foreignCiphertext),
+ Password: new(foreignCiphertext),
+ }
+
+ decrypted, err := models.DecryptAgent(agent)
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "/agent_id/1", "error should identify the agent")
+ assert.Contains(t, err.Error(), "username", "error should identify the field")
+ assert.Nil(t, decrypted.Username, "ciphertext must not be returned as the decrypted value")
+}
+
//nolint:dupword
func insertAgentWithSecrets(ctx context.Context, t *testing.T, db *sql.DB) {
t.Helper()
diff --git a/managed/models/encryption_key_test.go b/managed/models/encryption_key_test.go
new file mode 100644
index 00000000000..fa1a6258bf4
--- /dev/null
+++ b/managed/models/encryption_key_test.go
@@ -0,0 +1,122 @@
+// Copyright (C) 2023 Percona LLC
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program. If not, see .
+
+package models
+
+import (
+ "encoding/base64"
+ "encoding/json"
+ "testing"
+
+ sqlmock "github.com/DATA-DOG/go-sqlmock"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+ "gopkg.in/reform.v1"
+ "gopkg.in/reform.v1/dialects/postgresql"
+
+ "github.com/percona/pmm/managed/utils/encryption"
+)
+
+// TestVerifyEncryptionKey covers detection of the HA misconfiguration behind
+// https://perconadev.atlassian.net/browse/PMM-14979, where each node generated its own
+// encryption key while sharing one database.
+func TestVerifyEncryptionKey(t *testing.T) {
+ localFingerprint, err := encryption.Fingerprint()
+ require.NoError(t, err)
+ require.NotEmpty(t, localFingerprint)
+
+ foreignCiphertext := base64.StdEncoding.EncodeToString([]byte("encrypted-with-another-key"))
+
+ readableCiphertext, err := encryption.Encrypt("pmm-managed")
+ require.NoError(t, err)
+
+ settingsJSON := func(t *testing.T, s Settings) []byte {
+ t.Helper()
+ b, err := json.Marshal(s) //nolint:musttag
+ require.NoError(t, err)
+
+ return b
+ }
+
+ newMock := func(t *testing.T) (*reform.DB, sqlmock.Sqlmock) {
+ t.Helper()
+
+ sqlDB, mock, err := sqlmock.New()
+ require.NoError(t, err)
+ t.Cleanup(func() {
+ assert.NoError(t, mock.ExpectationsWereMet())
+ _ = mock.ExpectClose()
+ assert.NoError(t, sqlDB.Close())
+ })
+
+ return reform.NewDB(sqlDB, postgresql.Dialect, nil), mock
+ }
+
+ expectSettings := func(t *testing.T, mock sqlmock.Sqlmock, s Settings) {
+ t.Helper()
+ mock.ExpectQuery("SELECT settings FROM settings").
+ WillReturnRows(sqlmock.NewRows([]string{"settings"}).AddRow(settingsJSON(t, s)))
+ }
+
+ t.Run("matching fingerprint is accepted and not rewritten", func(t *testing.T) {
+ db, mock := newMock(t)
+ expectSettings(t, mock, Settings{EncryptionKeyFingerprint: localFingerprint})
+
+ assert.NoError(t, VerifyEncryptionKey(db))
+ })
+
+ t.Run("foreign fingerprint is reported as a mismatch", func(t *testing.T) {
+ db, mock := newMock(t)
+ expectSettings(t, mock, Settings{EncryptionKeyFingerprint: "0123456789abcdef"})
+
+ err := VerifyEncryptionKey(db)
+ require.ErrorIs(t, err, ErrEncryptionKeyMismatch)
+ assert.Contains(t, err.Error(), localFingerprint)
+ assert.Contains(t, err.Error(), "0123456789abcdef")
+ })
+
+ t.Run("fingerprint is recorded when nothing is encrypted yet", func(t *testing.T) {
+ db, mock := newMock(t)
+ expectSettings(t, mock, Settings{})
+ mock.ExpectExec("UPDATE settings SET settings").
+ WithArgs(sqlmock.AnyArg()).
+ WillReturnResult(sqlmock.NewResult(0, 1))
+
+ assert.NoError(t, VerifyEncryptionKey(db))
+ })
+
+ t.Run("fingerprint is adopted when stored data decrypts", func(t *testing.T) {
+ db, mock := newMock(t)
+ expectSettings(t, mock, Settings{EncryptedItems: []string{"pmm-managed.agents.username"}})
+ mock.ExpectQuery("SELECT username FROM agents").
+ WillReturnRows(sqlmock.NewRows([]string{"username"}).AddRow(readableCiphertext))
+ mock.ExpectExec("UPDATE settings SET settings").
+ WithArgs(sqlmock.AnyArg()).
+ WillReturnResult(sqlmock.NewResult(0, 1))
+
+ assert.NoError(t, VerifyEncryptionKey(db))
+ })
+
+ // The upgrade path for the reported cluster: a follower with its own key, against a database
+ // whose rows were encrypted by another node and which carries no fingerprint yet.
+ t.Run("fingerprint is not adopted when stored data cannot be decrypted", func(t *testing.T) {
+ db, mock := newMock(t)
+ expectSettings(t, mock, Settings{EncryptedItems: []string{"pmm-managed.agents.username"}})
+ mock.ExpectQuery("SELECT username FROM agents").
+ WillReturnRows(sqlmock.NewRows([]string{"username"}).AddRow(foreignCiphertext))
+
+ require.ErrorIs(t, VerifyEncryptionKey(db), ErrEncryptionKeyMismatch)
+ })
+}
diff --git a/managed/models/settings.go b/managed/models/settings.go
index d76fc4552cf..6836010f022 100644
--- a/managed/models/settings.go
+++ b/managed/models/settings.go
@@ -118,6 +118,11 @@ type Settings struct {
// Contains all encrypted tables in format 'db.table.column'.
EncryptedItems []string `json:"encrypted_items"`
+
+ // EncryptionKeyFingerprint identifies the encryption key the data in this database was
+ // encrypted with. In HA all nodes share the database but keep their own key file, so a node
+ // compares this against its own key to detect that it cannot read the stored credentials.
+ EncryptionKeyFingerprint string `json:"encryption_key_fingerprint"`
}
// IsAlertingEnabled returns true if alerting is enabled.
diff --git a/managed/models/settings_helpers.go b/managed/models/settings_helpers.go
index a48c5d74510..d15e704a645 100644
--- a/managed/models/settings_helpers.go
+++ b/managed/models/settings_helpers.go
@@ -102,6 +102,9 @@ type ChangeSettingsParams struct {
// List of items in format 'db.table.column' to be encrypted.
EncryptedItems []string
+
+ // EncryptionKeyFingerprint identifies the key the data was encrypted with. Empty clears it.
+ EncryptionKeyFingerprint *string
}
// SetPMMServerID should be run on start up to generate unique PMM Server ID.
@@ -243,6 +246,10 @@ func UpdateSettings(q reform.DBTX, params *ChangeSettingsParams) (*Settings, err
settings.EncryptedItems = params.EncryptedItems
}
+ if params.EncryptionKeyFingerprint != nil {
+ settings.EncryptionKeyFingerprint = *params.EncryptionKeyFingerprint
+ }
+
err = SaveSettings(q, settings)
if err != nil {
return nil, err
diff --git a/managed/services/agents/service_info_broker.go b/managed/services/agents/service_info_broker.go
index d0165990110..cd270d2ce40 100644
--- a/managed/services/agents/service_info_broker.go
+++ b/managed/services/agents/service_info_broker.go
@@ -194,7 +194,12 @@ func (c *ServiceInfoBroker) GetInfoFromService(ctx context.Context, q *reform.Qu
case models.MySQLServiceType:
agent.MySQLOptions.TableCount = &sInfo.TableCount
l.Debugf("Updating table count: %d.", sInfo.TableCount)
- err = q.Update(new(models.EncryptAgent(*agent)))
+ encryptedAgent, err := models.EncryptAgent(*agent)
+ if err != nil {
+ return err
+ }
+
+ err = q.Update(new(encryptedAgent))
if err != nil {
return fmt.Errorf("failed to update table count: %w", err)
}
@@ -216,7 +221,12 @@ func (c *ServiceInfoBroker) GetInfoFromService(ctx context.Context, q *reform.Qu
agent.PostgreSQLOptions.DatabaseCount = int32(databaseCount - excludedDatabaseCount)
l.Debugf("Updating PostgreSQL options, database count: %d.", agent.PostgreSQLOptions.DatabaseCount)
- err = q.Update(new(models.EncryptAgent(*agent)))
+ encryptedAgent, err := models.EncryptAgent(*agent)
+ if err != nil {
+ return err
+ }
+
+ err = q.Update(new(encryptedAgent))
if err != nil {
return fmt.Errorf("failed to update database count: %w", err)
}
diff --git a/managed/services/realtimeanalytics/service.go b/managed/services/realtimeanalytics/service.go
index 0414f9b6558..f3691c977a5 100644
--- a/managed/services/realtimeanalytics/service.go
+++ b/managed/services/realtimeanalytics/service.go
@@ -269,7 +269,12 @@ func (s *Service) StartSession(ctx context.Context, req *rtav1.StartSessionReque
// Need to update CreatedAt to reflect the new session start time.
rtaAgent.CreatedAt = time.Now()
// Encrypt agent's sensitive data before updating it in the database.
- rtaAgent = new(models.EncryptAgent(*rtaAgent))
+ encryptedAgent, err := models.EncryptAgent(*rtaAgent)
+ if err != nil {
+ return err
+ }
+
+ rtaAgent = new(encryptedAgent)
err = tx.Update(rtaAgent)
if err != nil {
@@ -434,7 +439,12 @@ func (s *Service) StopSession(ctx context.Context, req *rtav1.StopSessionRequest
rtaAgent := existingRTAAgents[0]
rtaAgent.Disabled = true
// Encrypt agent's sensitive data before updating it in the database.
- rtaAgent = new(models.EncryptAgent(*rtaAgent))
+ encryptedAgent, err := models.EncryptAgent(*rtaAgent)
+ if err != nil {
+ return err
+ }
+
+ rtaAgent = new(encryptedAgent)
err = tx.Update(rtaAgent)
if err != nil {
diff --git a/managed/services/victoriametrics/victoriametrics_test.go b/managed/services/victoriametrics/victoriametrics_test.go
index 77f67e2452e..3e0c7f245f4 100644
--- a/managed/services/victoriametrics/victoriametrics_test.go
+++ b/managed/services/victoriametrics/victoriametrics_test.go
@@ -287,7 +287,9 @@ func TestVictoriaMetrics(t *testing.T) {
},
} {
if str, ok := str.(*models.Agent); ok {
- *str = models.EncryptAgent(*str)
+ encrypted, err := models.EncryptAgent(*str)
+ check.NoError(err)
+ *str = encrypted
}
err := db.Insert(str)
diff --git a/managed/utils/encryption/encryption.go b/managed/utils/encryption/encryption.go
index 9fd9fc090bb..3d8402d35fc 100644
--- a/managed/utils/encryption/encryption.go
+++ b/managed/utils/encryption/encryption.go
@@ -18,7 +18,9 @@ package encryption
import (
"bytes"
+ "crypto/sha256"
"encoding/base64"
+ "encoding/hex"
"errors"
"fmt"
"os"
@@ -152,7 +154,7 @@ func RotateEncryptionKey() error {
// RestoreOldEncryptionKey is a wrapper around DefaultEncryption.RestoreOldEncryptionKey.
func RestoreOldEncryptionKey() error {
- err := os.Rename(strings.TrimSuffix(encryptionKeyPath(), ".key")+"_old.key", encryptionKeyPath())
+ err := os.Rename(strings.TrimSuffix(KeyPath(), ".key")+"_old.key", KeyPath())
if err != nil {
return fmt.Errorf("could not restore old encryption key: %w", err)
}
@@ -161,7 +163,7 @@ func RestoreOldEncryptionKey() error {
}
func backupOldEncryptionKey() error {
- err := os.Rename(encryptionKeyPath(), strings.TrimSuffix(encryptionKeyPath(), ".key")+"_old.key")
+ err := os.Rename(KeyPath(), strings.TrimSuffix(KeyPath(), ".key")+"_old.key")
if err != nil {
return fmt.Errorf("failed to backup old encryption key: %w", err)
}
@@ -185,6 +187,32 @@ func (e *Encryption) GenerateKey() (string, error) {
return base64.StdEncoding.EncodeToString(buff.Bytes()), nil
}
+// Fingerprint is a wrapper around DefaultEncryption.Fingerprint.
+func Fingerprint() (string, error) {
+ return getDefaultEncryption().Fingerprint()
+}
+
+// Fingerprint returns a stable identifier of the encryption key.
+//
+// It is recorded next to the encrypted data so a node can tell whether it holds the key the
+// data was encrypted with without decrypting anything, which also works when there is no
+// encrypted row to test against yet. It is a digest of the key, so it cannot be used to
+// reconstruct it.
+func (e *Encryption) Fingerprint() (string, error) {
+ if e == nil || e.Key == "" {
+ return "", ErrEncryptionNotInitialized
+ }
+
+ serializedKeyset, err := base64.StdEncoding.DecodeString(e.Key)
+ if err != nil {
+ return "", fmt.Errorf("failed to decode keyset: %w", err)
+ }
+
+ sum := sha256.Sum256(serializedKeyset)
+
+ return hex.EncodeToString(sum[:]), nil
+}
+
func (e *Encryption) generateAndPersistKey() error {
key, err := e.GenerateKey()
if err != nil {
@@ -204,16 +232,18 @@ func Encrypt(secret string) (string, error) {
}
// Encrypt returns input string encrypted.
+// On failure it returns an empty string rather than the plaintext, so that a caller which
+// ignores the error cannot persist an unencrypted secret.
func (e *Encryption) Encrypt(secret string) (string, error) {
if e == nil || e.Primitive == nil {
- return secret, ErrEncryptionNotInitialized
+ return "", ErrEncryptionNotInitialized
}
if secret == "" {
return secret, nil
}
cipherText, err := e.Primitive.Encrypt([]byte(secret), []byte(""))
if err != nil {
- return secret, fmt.Errorf("encryption: %w", err)
+ return "", fmt.Errorf("encryption: %w", err)
}
return base64.StdEncoding.EncodeToString(cipherText), nil
@@ -269,20 +299,24 @@ func Decrypt(cipherText string) (string, error) {
}
// Decrypt returns input string decrypted.
+// On failure it returns an empty string rather than the input ciphertext. Returning the
+// ciphertext let callers that only logged the error pass it on as if it were the decrypted
+// value, which is how encrypted credentials reached pmm-agent as usernames and passwords
+// when a node's encryption key did not match the data in a shared database.
func (e *Encryption) Decrypt(cipherText string) (string, error) {
if e == nil || e.Primitive == nil {
- return cipherText, ErrEncryptionNotInitialized
+ return "", ErrEncryptionNotInitialized
}
if cipherText == "" {
return cipherText, nil
}
decoded, err := base64.StdEncoding.DecodeString(cipherText)
if err != nil {
- return cipherText, fmt.Errorf("decryption: %w, %s", err, cipherText)
+ return "", fmt.Errorf("decryption: %w", err)
}
secret, err := e.Primitive.Decrypt(decoded, []byte(""))
if err != nil {
- return cipherText, fmt.Errorf("decryption: %w", err)
+ return "", fmt.Errorf("decryption: %w", err)
}
return string(secret), nil
diff --git a/managed/utils/encryption/helpers.go b/managed/utils/encryption/helpers.go
index 0cbbd2c3728..9fb6a5bf564 100644
--- a/managed/utils/encryption/helpers.go
+++ b/managed/utils/encryption/helpers.go
@@ -25,8 +25,9 @@ import (
"gopkg.in/reform.v1"
)
-func encryptionKeyPath() string {
- customKeyPath := os.Getenv("PMM_ENCRYPTION_KEY_PATH")
+// KeyPath returns the path PMM reads the encryption key from.
+func KeyPath() string {
+ customKeyPath := os.Getenv(CustomEncryptionKeyPathEnvVar)
if customKeyPath != "" {
return customKeyPath
}