feat(midaz): add unified ledger config (CRM/Fees/KMS/tracer) and tracer templates

charts/midaz/README.md

@@ -3,7 +3,7 @@
 ## Chart Contract
 
 - Chart type: `multi-component`
-- Required secrets: `ledger.secrets.RABBITMQ_DEFAULT_PASS` and `ledger.secrets.RABBITMQ_CONSUMER_PASS` (operator-provided — see "Known limitation" below) plus `crm.secrets.LCRYPTO_HASH_SECRET_KEY` and `crm.secrets.LCRYPTO_ENCRYPT_SECRET_KEY` (app crypto material). The database, replica, and cache passwords (`DB_ONBOARDING_PASSWORD`, `DB_ONBOARDING_REPLICA_PASSWORD`, `MONGO_ONBOARDING_PASSWORD`, `DB_TRANSACTION_PASSWORD`, `DB_TRANSACTION_REPLICA_PASSWORD`, `MONGO_TRANSACTION_PASSWORD`, `REDIS_PASSWORD`, `crm.secrets.MONGO_PASSWORD`) are single-sourced from the bundled Bitnami subcharts and are only required when the matching backend is external.
+- Required secrets: `ledger.secrets.RABBITMQ_DEFAULT_PASS` and `ledger.secrets.RABBITMQ_CONSUMER_PASS` (operator-provided — see "Known limitation" below) plus `crm.secrets.LCRYPTO_HASH_SECRET_KEY` and `crm.secrets.LCRYPTO_ENCRYPT_SECRET_KEY` (app crypto material). With a 4.x `ledger.image.tag` and `KMS_VENDOR=none`, `ledger.secrets.LCRYPTO_HASH_SECRET_KEY` and `ledger.secrets.LCRYPTO_ENCRYPT_SECRET_KEY` are required too: the unified binary serves CRM in-process and initializes its cipher from them at boot. The database, replica, and cache passwords (`DB_ONBOARDING_PASSWORD`, `DB_ONBOARDING_REPLICA_PASSWORD`, `MONGO_ONBOARDING_PASSWORD`, `DB_TRANSACTION_PASSWORD`, `DB_TRANSACTION_REPLICA_PASSWORD`, `MONGO_TRANSACTION_PASSWORD`, `REDIS_PASSWORD`, `crm.secrets.MONGO_PASSWORD`) are single-sourced from the bundled Bitnami subcharts and are only required when the matching backend is external.
 - Dependency notes: PostgreSQL, MongoDB, and Valkey passwords are single-sourced from the bundled Bitnami subchart Secrets (`<release>-postgresql` key `password`/`replication-password`, `<release>-mongodb` key `mongodb-root-password`, `<release>-valkey` key `valkey-password`) and injected into the ledger/crm workloads via `secretKeyRef`. RabbitMQ and optional OpenTelemetry are also bundled. When a backend is external (`<subchart>.enabled=false`/`.external=true`), supply its password through the component `secrets` block or point the subchart at an `auth.existingSecret`.
 - Production overrides: Provide RabbitMQ and CRM crypto credentials through the component `secrets` (or `useExistingSecret`); let the bundled Bitnami subcharts own the database/cache passwords (or set `<subchart>.auth.existingSecret`/`<subchart>.auth.password`). Override image tags, ingress, resources, namespace, and persistence as needed.
 - Source/license: Source is in `github.com/LerianStudio/helm`; license is Apache-2.0.
@@ -295,6 +295,28 @@ crm:
   #   MONGO_PASSWORD: "<your-mongo-password>"
 ```
 
+### Tracer
+
+The `tracer` service provides real-time transaction validation and fraud prevention. From midaz v4 it ships from the monorepo (`components/tracer`) and is published as `lerianstudio/midaz-tracer`; the standalone `lerianstudio/tracer` 1.x image predates this configuration contract. It is disabled by default (`tracer.enabled=false`) and existing releases render unchanged.
+
+- **Database.** Tracer uses the bundled PostgreSQL with the same `midaz` role that owns `onboarding` and `transaction`. Its `tracer` database is created by `files/midaz/init.sql` on a fresh internal cluster, and by the external bootstrap Job (`global.externalPostgresDefinitions.enabled=true`) otherwise. On a pre-existing internal cluster, initdb scripts no longer run: create the database once by hand (`CREATE DATABASE tracer;`) before enabling tracer.
+- **Migrations.** Neither tracer nor the v4 ledger migrates at startup anymore. `tracer.migrations` and `ledger.migrations` render the dedicated `midaz-tracer-migrations` / `midaz-ledger-migrations` runner Jobs, whose tag defaults to the matching application tag. The tracer Job is on whenever tracer is; the ledger Job is on for 4.x ledger tags only (`ledger.migrations.enabled` pins it either way, e.g. `false` when the schema is applied out-of-band). Both are plain Sync-phase resources because the bundled PostgreSQL Secret they read is only created during Sync; against a pre-provisioned database, set `argocd.argoproj.io/hook: PreSync` under `<component>.migrations.annotations` to order them ahead of the Deployment.
+- **Memory.** `tracer.configmap.GOMEMLIMIT` must track `tracer.resources.limits.memory` (~90%). The image ships `GOMEMLIMIT=1800MiB`, sized for a 2Gi container, so without the override the Go heap outgrows a 512Mi cgroup and the pod is OOM-killed.
+- **Fail-fast configuration.** The chart mirrors the tracer boot validators, so misconfiguration fails `helm template` instead of crash-looping: `API_KEY_ENABLED=true` requires an API key and rejects wildcard CORS; `MULTI_TENANT_ENABLED=true` requires `PLUGIN_AUTH_ENABLED=true`, a service API key, and rejects `API_KEY_ENABLED_ONLY_VALIDATION=true`; `tracer.useExistingSecret=true` requires `tracer.existingSecretName`.
+
+```yaml
+tracer:
+  enabled: true
+  configmap:
+    API_KEY_ENABLED: "true"
+    CORS_ALLOWED_ORIGINS: "https://app.example.com"
+  secrets:
+    API_KEY: "<your-api-key>"
+  # DB_PASSWORD is single-sourced from the bundled Bitnami postgresql subchart
+  # (Secret `midaz-postgresql`, key `password`) — only set it here when using an
+  # EXTERNAL PostgreSQL (postgresql.enabled=false / postgresql.external=true).
+```
+
 ## Observability
 
 Midaz uses [Grafana Docker OpenTelemetry LGTM](https://github.com/grafana/docker-otel-lgtm) for observability. This component collects, processes, and exports telemetry data such as traces and metrics.

charts/midaz/files/midaz/init.sql

@@ -8,4 +8,11 @@ SELECT pg_create_physical_replication_slot('replication_slot');
 SELECT * FROM pg_create_logical_replication_slot('logical_slot', 'pgoutput');
 
 CREATE DATABASE onboarding;
-CREATE DATABASE transaction;
+CREATE DATABASE transaction;
+
+-- Tracer shares this cluster and the `midaz` role that owns the databases above
+-- (its Deployment reads the same subchart Secret). Created unconditionally, and
+-- not gated on tracer.enabled, because initdb scripts run exactly once: gating
+-- would leave the database missing for anyone enabling tracer after install.
+-- An unused empty database costs nothing.
+CREATE DATABASE tracer;

charts/midaz/templates/_helpers.tpl

@@ -166,6 +166,51 @@ app.kubernetes.io/name: {{ include "midaz.name" .context }}-{{ .name }}
 app.kubernetes.io/instance: {{ .context.Release.Name }}
 {{- end }}
 
+{{/*
+Create a default fully qualified app name for Tracer.
+*/}}
+{{- define "midaz-tracer.fullname" -}}
+{{- if .Values.tracer.fullnameOverride }}
+{{- .Values.tracer.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" (include "midaz.name" .) (default .Values.tracer.name .Values.tracer.nameOverride) | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create Tracer app version
+*/}}
+{{- define "tracer.defaultTag" -}}
+{{- default .Chart.AppVersion .Values.tracer.image.tag }}
+{{- end -}}
+
+{{/*
+Return valid Tracer version label
+*/}}
+{{- define "tracer.versionLabelValue" -}}
+{{ regexReplaceAll "[^-A-Za-z0-9_.]" (include "tracer.defaultTag" .) "-" | trunc 63 | trimAll "-" | trimAll "_" | trimAll "." | quote }}
+{{- end -}}
+
+{{/*
+Tracer Common labels
+*/}}
+{{- define "midaz-tracer.labels" -}}
+helm.sh/chart: {{ include "midaz.chart" .context }}
+{{ include "midaz-tracer.selectorLabels" (dict "context" .context "name" .name) }}
+app.kubernetes.io/version: {{ include "tracer.versionLabelValue" .context }}
+app.kubernetes.io/managed-by: {{ .context.Release.Service }}
+{{- end }}
+
+{{/*
+Tracer Selector labels
+*/}}
+{{- define "midaz-tracer.selectorLabels" -}}
+{{- if .name -}}
+app.kubernetes.io/name: {{ include "midaz.name" .context }}-{{ .name }}
+{{- end }}
+app.kubernetes.io/instance: {{ .context.Release.Name }}
+{{- end }}
+
 {{/*
 Enable internal dependencies
 */}}
@@ -215,3 +260,98 @@ Secret/Service names render even when all bundled subcharts are disabled
 {{- end -}}
 {{- end -}}
 {{- end -}}
+
+{{/*
+midaz.tagIsV4 — reports "true" when the passed image tag is a semver >= 4.0.0
+(pre-releases included), "" otherwise. Non-semver tags ("latest", digests,
+branch builds) resolve to "" so they never trip a version-gated requirement.
+
+Midaz v4 changed two boot contracts that 3.x does not have: the unified ledger
+binary serves CRM in-process (so it initializes the CRM cipher at startup) and
+neither ledger nor tracer migrates its schema anymore. Both are gated on this
+helper so 3.x releases keep rendering exactly as before.
+*/}}
+{{- define "midaz.tagIsV4" -}}
+{{- $tag := . | toString | trimPrefix "v" -}}
+{{- if regexMatch "^[0-9]+\\.[0-9]+\\.[0-9]+" $tag -}}
+{{- if semverCompare ">=4.0.0-0" $tag -}}true{{- end -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+midaz.ledgerMigrationsEnabled — resolves the tri-state ledger.migrations.enabled.
+Unset (null) means "auto": on for 4.x ledger tags, off for 3.x, which still runs
+its migrations in-process. An explicit true/false always wins, so operators who
+apply the schema out-of-band (managed-Postgres S3 pipeline) can pin it off.
+*/}}
+{{- define "midaz.ledgerMigrationsEnabled" -}}
+{{- $enabled := dig "migrations" "enabled" nil .Values.ledger -}}
+{{- if kindIs "invalid" $enabled -}}
+{{- include "midaz.tagIsV4" (.Values.ledger.image.tag | default .Chart.AppVersion) -}}
+{{- else if eq (toString $enabled) "true" -}}
+true
+{{- end -}}
+{{- end -}}
+
+{{/*
+midaz-tracer.migrationsFullname — one Job name per migration image tag. A Job
+spec is immutable, so a stable name would silently skip the run on upgrade; the
+tag suffix makes every version bump create a new Job. Re-running is safe:
+golang-migrate tracks progress in the schema_migrations table.
+*/}}
+{{- define "midaz-tracer.migrationsFullname" -}}
+{{- $tag := include "midaz-tracer.migrationsTag" . -}}
+{{- printf "%s-migrations-%s" (include "midaz-tracer.fullname" .) (regexReplaceAll "[^a-z0-9.]+" (lower $tag) "-") | trunc 63 | trimSuffix "-" | trimSuffix "." -}}
+{{- end -}}
+
+{{- define "midaz-ledger.migrationsFullname" -}}
+{{- $tag := include "midaz-ledger.migrationsTag" . -}}
+{{- printf "%s-migrations-%s" (include "midaz-ledger.fullname" .) (regexReplaceAll "[^a-z0-9.]+" (lower $tag) "-") | trunc 63 | trimSuffix "-" | trimSuffix "." -}}
+{{- end -}}
+
+{{/*
+Migration-runner image tags default to the matching application image tag so the
+schema and the binary reading it can never drift when only one is bumped.
+*/}}
+{{- define "midaz-tracer.migrationsTag" -}}
+{{- dig "migrations" "image" "tag" "" .Values.tracer | default .Values.tracer.image.tag | default .Chart.AppVersion -}}
+{{- end -}}
+
+{{- define "midaz-ledger.migrationsTag" -}}
+{{- dig "migrations" "image" "tag" "" .Values.ledger | default .Values.ledger.image.tag | default .Chart.AppVersion -}}
+{{- end -}}
+
+{{/*
+midaz-tracer.validate — render-time mirror of the tracer bootstrap validators
+(components/tracer/internal/bootstrap). Every rule below is a configuration the
+v4 process rejects at boot, so failing the render turns a CrashLoopBackOff into
+a helm error the operator can read.
+*/}}
+{{- define "midaz-tracer.validate" -}}
+{{- $tracer := .Values.tracer -}}
+{{- $cm := $tracer.configmap | default dict -}}
+{{- $secrets := $tracer.secrets | default dict -}}
+{{- $extra := $tracer.extraEnvVars | default dict -}}
+{{- if and $tracer.useExistingSecret (not $tracer.existingSecretName) -}}
+{{- fail "tracer.useExistingSecret=true requires tracer.existingSecretName (an empty secretRef.name is rejected by the API server)" -}}
+{{- end -}}
+{{- if eq ($cm.API_KEY_ENABLED | default "false" | toString) "true" -}}
+{{- if and (not $secrets.API_KEY) (not $tracer.useExistingSecret) -}}
+{{- fail "tracer.secrets.API_KEY is required when API_KEY_ENABLED=true (ValidateAuthConfig rejects an empty key at boot); or set tracer.useExistingSecret" -}}
+{{- end -}}
+{{- if eq ($cm.CORS_ALLOWED_ORIGINS | default "*" | toString) "*" -}}
+{{- fail "tracer.configmap.CORS_ALLOWED_ORIGINS=\"*\" is rejected at boot when API_KEY_ENABLED=true: any site could drive authenticated calls once the key leaks. Set a concrete origin allow-list" -}}
+{{- end -}}
+{{- end -}}
+{{- if eq ($cm.MULTI_TENANT_ENABLED | default "false" | toString) "true" -}}
+{{- if ne ($cm.PLUGIN_AUTH_ENABLED | default "false" | toString) "true" -}}
+{{- fail "tracer.configmap.PLUGIN_AUTH_ENABLED must be \"true\" when MULTI_TENANT_ENABLED=true: API-key-only auth cannot verify tenant JWT signatures, so any caller could forge a tenantId" -}}
+{{- end -}}
+{{- if or (eq ($cm.API_KEY_ENABLED_ONLY_VALIDATION | default "false" | toString) "true") (eq (dig "API_KEY_ENABLED_ONLY_VALIDATION" "false" $extra | toString) "true") -}}
+{{- fail "API_KEY_ENABLED_ONLY_VALIDATION=true is incompatible with MULTI_TENANT_ENABLED=true: it lets /v1/validations bypass plugin auth, reopening cross-tenant forgery" -}}
+{{- end -}}
+{{- if and (not $secrets.MULTI_TENANT_SERVICE_API_KEY) (not $tracer.useExistingSecret) -}}
+{{- fail "tracer.secrets.MULTI_TENANT_SERVICE_API_KEY is required when MULTI_TENANT_ENABLED=true" -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}

charts/midaz/templates/bootstrap-postgres.yaml

@@ -83,6 +83,9 @@ spec:
           ONB_EXISTS=0
           TRX_EXISTS=0
           ROLE_EXISTS=0
+          {{- if .Values.tracer.enabled }}
+          TRACER_EXISTS=0
+          {{- end }}
 
           if PGPASSWORD="$DB_ADMIN_PASSWORD" psql -At -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d "$DB_DATABASE" -c "SELECT 1 FROM pg_database WHERE datname='onboarding'" | grep -q 1; then
             ONB_EXISTS=1
@@ -93,9 +96,14 @@ spec:
           if PGPASSWORD="$DB_ADMIN_PASSWORD" psql -At -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d "$DB_DATABASE" -c "SELECT 1 FROM pg_roles WHERE rolname='midaz'" | grep -q 1; then
             ROLE_EXISTS=1
           fi
+          {{- if .Values.tracer.enabled }}
+          if PGPASSWORD="$DB_ADMIN_PASSWORD" psql -At -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d "$DB_DATABASE" -c "SELECT 1 FROM pg_database WHERE datname='{{ .Values.tracer.configmap.DB_NAME | default "tracer" }}'" | grep -q 1; then
+            TRACER_EXISTS=1
+          fi
+          {{- end }}
 
-          if [ "$ONB_EXISTS" = "1" ] && [ "$TRX_EXISTS" = "1" ] && [ "$ROLE_EXISTS" = "1" ]; then
-            echo "Postgres bootstrap config already exists (databases 'onboarding' and 'transaction' and role 'midaz'). Skipping."
+          if [ "$ONB_EXISTS" = "1" ] && [ "$TRX_EXISTS" = "1" ] && [ "$ROLE_EXISTS" = "1" ]{{ if .Values.tracer.enabled }} && [ "$TRACER_EXISTS" = "1" ]{{ end }}; then
+            echo "Postgres bootstrap config already exists (databases 'onboarding' and 'transaction'{{ if .Values.tracer.enabled }} and '{{ .Values.tracer.configmap.DB_NAME | default "tracer" }}'{{ end }} and role 'midaz'). Skipping."
             exit 0
           fi
 
@@ -113,6 +121,16 @@ spec:
             echo "Creating database 'transaction'..."
             PGPASSWORD="$DB_ADMIN_PASSWORD" psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d "$DB_DATABASE" -c "CREATE DATABASE transaction"
           fi
+          {{- if .Values.tracer.enabled }}
+
+          # Tracer runs as the same 'midaz' role, so only its database is created here.
+          if [ "$TRACER_EXISTS" = "1" ]; then
+            echo "Database '{{ .Values.tracer.configmap.DB_NAME | default "tracer" }}' already exists. Skipping creation."
+          else
+            echo "Creating database '{{ .Values.tracer.configmap.DB_NAME | default "tracer" }}'..."
+            PGPASSWORD="$DB_ADMIN_PASSWORD" psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d "$DB_DATABASE" -c "CREATE DATABASE {{ .Values.tracer.configmap.DB_NAME | default "tracer" }}"
+          fi
+          {{- end }}
 
           # Role
           if [ "$ROLE_EXISTS" = "1" ]; then
@@ -128,4 +146,8 @@ spec:
           PGPASSWORD="$DB_ADMIN_PASSWORD" psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d "$DB_DATABASE" -c "GRANT ALL PRIVILEGES ON DATABASE transaction TO midaz"
           PGPASSWORD="$DB_ADMIN_PASSWORD" psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d onboarding -c "GRANT ALL ON SCHEMA public TO midaz"
           PGPASSWORD="$DB_ADMIN_PASSWORD" psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d transaction -c "GRANT ALL ON SCHEMA public TO midaz"
+          {{- if .Values.tracer.enabled }}
+          PGPASSWORD="$DB_ADMIN_PASSWORD" psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d "$DB_DATABASE" -c "GRANT ALL PRIVILEGES ON DATABASE {{ .Values.tracer.configmap.DB_NAME | default "tracer" }} TO midaz"
+          PGPASSWORD="$DB_ADMIN_PASSWORD" psql -v ON_ERROR_STOP=1 -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER_ADMIN" -d {{ .Values.tracer.configmap.DB_NAME | default "tracer" }} -c "GRANT ALL ON SCHEMA public TO midaz"
+          {{- end }}
 {{- end }}