diff --git a/.github/styles/base/Dictionary.txt b/.github/styles/base/Dictionary.txt index 8d1266def5..eb98af7445 100644 --- a/.github/styles/base/Dictionary.txt +++ b/.github/styles/base/Dictionary.txt @@ -158,6 +158,7 @@ cors cosmiconfig cosocket CPUs +vCPUs crd crds created_at @@ -677,6 +678,7 @@ perf performant petstore PDFs +pg_max_concurrent_queries pgdump pgvector pipeline diff --git a/app/_support/pg_max_concurrent_queries-per-worker-process.md b/app/_support/pg_max_concurrent_queries-per-worker-process.md new file mode 100644 index 0000000000..c0fecff750 --- /dev/null +++ b/app/_support/pg_max_concurrent_queries-per-worker-process.md @@ -0,0 +1,110 @@ +--- +title: "pg_max_concurrent_queries is per worker process, not per node" +content_type: support +description: "How to limit the total number of PostgreSQL database connections when running {{site.base_gateway}} on multiple nodes in Kubernetes." +products: + - gateway + +works_on: + - on-prem + - konnect + +related_resources: + - text: "{{site.base_gateway}} configuration reference - `pg_max_concurrent_queries`" + url: /gateway/configuration/#pg-max-concurrent-queries + - text: "{{site.base_gateway}} configuration reference - `nginx_worker_processes`" + url: /gateway/configuration/#nginx-worker-processes + +tldr: + q: Why does my PostgreSQL connection count exceed the limit I set with pg_max_concurrent_queries? + a: | + `pg_max_concurrent_queries` is applied **per nginx worker process**, so the actual maximum + number of concurrent database connections is: + + ``` + pg_max_concurrent_queries × nginx_worker_processes × number of Kong nodes + ``` + + To bring the total under your database connection limit, explicitly set + `nginx_worker_processes` to a fixed integer value rather than relying on the + default `auto` setting, which bases the count on the host's vCPUs and can + produce more workers than expected in Kubernetes. + +--- + +## Problem + +The PostgreSQL database used by {{site.base_gateway}} has a maximum number of permitted +connections. After setting `pg_max_concurrent_queries` to a value below that maximum, +the connection limit is still exceeded. {{site.base_gateway}} is running on multiple +nodes deployed in Kubernetes. + +## Cause + +The `pg_max_concurrent_queries` setting is scoped **per nginx worker process**, not +per Kong node. The actual maximum number of concurrent database connections is therefore: + +``` +pg_max_concurrent_queries × nginx_worker_processes × number of Kong nodes +``` + +The `nginx_worker_processes` parameter defaults to `auto`, which sets the number of +worker processes equal to the number of available vCPUs. In a Kubernetes environment, +the calculation is based on the **host** vCPUs (not the container's CPU limit), which +can result in significantly more workers — and therefore more database connections — +than expected. + +## Solution + +1. Determine the current number of nginx worker processes on a running Kong node: + + ```bash + ps aux | grep "[n]ginx: worker process" | wc -l + ``` + +2. Calculate the maximum connection count you need to stay within your database limit. + For example, if your database allows 200 connections, you have 4 Kong nodes, and + you want `pg_max_concurrent_queries` set to 10: + + ``` + 200 connections ÷ (4 nodes × 10 queries) = 5 worker processes per node + ``` + +3. Explicitly set `nginx_worker_processes` to a fixed integer in `kong.conf`: + + ``` + nginx_worker_processes = 5 + ``` + + Or via an environment variable: + + ```bash + KONG_NGINX_WORKER_PROCESSES=5 + ``` + +4. Reload or restart {{site.base_gateway}} for the change to take effect: + + ```bash + kong reload + ``` + +5. Verify the worker count reflects the new value: + + ```bash + ps aux | grep "[n]ginx: worker process" | wc -l + ``` + +## Validation + +After restarting, confirm the total connection count stays within bounds by monitoring +your database connection metrics. For PostgreSQL, you can check active connections with: + +```sql +SELECT count(*) FROM pg_stat_activity WHERE datname = ''; +``` + +The count should remain at or below: + +``` +pg_max_concurrent_queries × nginx_worker_processes × number of Kong nodes +```