diff --git a/.github/DOCUMENTATION_SYNC.md b/.github/DOCUMENTATION_SYNC.md new file mode 100644 index 00000000..2213fa98 --- /dev/null +++ b/.github/DOCUMENTATION_SYNC.md @@ -0,0 +1,98 @@ +# Documentation sync: PR checklist, labels, drift CI + +Public docs here track Convoy server behavior for **`convoy.json`**, env, CLI, and metrics. Use this when reviewing or writing PRs. + +## PR template checklist + +The [pull request template](pull_request_template.md) asks authors to update the relevant **config / env / CLI / metrics / Grafana** pages or mark the line **N/A** with a short reason (e.g. copy-only change, dashboard screenshot swap, internal nav). N/A is allowed so trivial PRs are not blocked; anything config-adjacent still needs an explicit call. + +Suggested entry points: + +- **Config & env:** `docs/deployment/configuration.mdx` (and deployment pages that repeat env names). +- **CLI:** `docs/cli-file/convoy.mdx` (and CLI-adjacent guides). +- **Metrics / Grafana:** `docs/product-manual/metrics.mdx`, `docs/resources/telemetry.mdx` where scrape paths and env overlap. + +## Labels + +Labels are not defined in-repo; they live in org or repo settings on GitHub. For docs-sync work: + +- **`documentation`:** docs-only or mostly docs. +- **`config`**, **`metrics`**, **`cli`**, etc.: optional; helps when many PRs land at once. + +Authors and reviewers set labels manually. + +## Automation modes + +Three independent ways docs can stay aligned with the server. Use one or more. + +### Mode A: Docs PR drift checks + +**What it does:** On PRs that change docs or drift tooling, CI runs static checks in this repo only. It does not clone the Convoy server or regenerate JSON. + +**Implemented here:** Yes. [docs-drift.yml](workflows/docs-drift.yml) runs when the diff touches `docs/**`, `.github/workflows/docs-drift.yml`, `.github/scripts/docs-drift-check.sh`, `.github/docs-drift/**`, `.github/pull_request_template.md`, or `.github/DOCUMENTATION_SYNC.md`. It fails if: + +- Any **forbidden substring** from [`.github/docs-drift/forbidden-patterns.txt`](docs-drift/forbidden-patterns.txt) appears under `docs/` (wrong config filenames, deprecated env naming, etc.). +- **Consistency checks** fail (e.g. `configuration.mdx` must still link to `config/config.go`, state CLI > env > `convoy.json` precedence exactly once, and key metrics env names must appear in both the configuration reference and `metrics.mdx`). + +Prose under `docs/` stays human-authored. + +### Mode B: Generated reference sync (receive) + +**What it does:** A workflow checks out the Convoy repo, runs `make docs-generated`, copies `docs/generated/*` into **`docs/generated/convoy/`**, then opens or updates one long-lived PR limited to those paths. Maintainers merge it like any other PR. + +**Implemented here:** Yes. [convoy-docs-sync-receive.yml](workflows/convoy-docs-sync-receive.yml): + +- **Triggers:** `repository_dispatch` with type **`convoy_docs_sync`**; **`workflow_dispatch`** with optional input `convoy_ref` (default `main`). +- **PR branch:** `chore/sync-convoy-generated-docs` (reused; not deleted after merge). +- **PR title:** `chore(docs): sync Convoy generated reference artifacts` +- **Committed paths:** `docs/generated/**` only (see workflow `add-paths`). + +**Secrets (convoy-website)** + +| Secret | Required? | Purpose | +|--------|-----------|---------| +| `CONVOY_CHECKOUT_TOKEN` | **Optional** | If set, used as `token` for the second `actions/checkout` of `{owner}/convoy`. If unset, the job uses `GITHUB_TOKEN`, which is enough when Convoy is **public** (or readable by that token). Use a PAT when Convoy is **private** or the default token cannot read it. | + +No secret is required on convoy-website **to receive** a dispatch; GitHub authenticates the incoming event. + +**Secrets / token (Convoy repo, caller)** + +To call **`POST /repos/{owner}/{repo}/dispatches`** on **this** repo from Convoy CI or a script, use a PAT or GitHub App token allowed for that endpoint. Typical setup: + +- Fine-grained or classic PAT with access to **convoy-website** (at minimum **Contents** / repository dispatch on that repo; classic `repo` on the website repo also works). +- Store it on the **Convoy** repo, e.g. **`CONVOY_WEBSITE_DISPATCH_TOKEN`**. +- Dispatch from Convoy: + +```bash +curl -sS -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${CONVOY_WEBSITE_DISPATCH_TOKEN}" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "https://api.github.com/repos/frain-dev/convoy-website/dispatches" \ + -d '{"event_type":"convoy_docs_sync","client_payload":{"convoy_ref":"main"}}' +``` + +The URL assumes **frain-dev** as owner; change the path for a fork. Optional `client_payload.convoy_ref` overrides the ref (same as the workflow `workflow_dispatch` input). + +**Not in this repo:** A Convoy-side job that runs the `curl` (or `peter-evans/repository-dispatch`) after `main` updates. Add that in **convoy** to auto-trigger sync from server merges. + +### Mode C: Hands-off pipeline (caller + optional auto-merge) + +**Operationally:** (1) Convoy CI (or another process) dispatches on every relevant server change so Mode B runs without **Run workflow**. (2) Optionally, generated-docs PRs auto-merge when checks pass. + +**Implemented here:** **Partially.** Only the receive side (Mode B) lives here. Auto-dispatch after server merges belongs in **convoy**. **Auto-merge** for the generated PR is not implemented here unless added on purpose. + +Extending drift checks to assert on `docs/generated/convoy/` is optional follow-up. + +When you rename or remove an env var or config file on the server, update docs **and** add the old string to `forbidden-patterns.txt` so CI catches regressions. + +## Related files + +| Item | Path | +|------|------| +| PR template | [`.github/pull_request_template.md`](pull_request_template.md) | +| Mode A workflow | `.github/workflows/docs-drift.yml` | +| Mode B workflow | `.github/workflows/convoy-docs-sync-receive.yml` | +| Mirrored generated artifacts | `docs/generated/convoy/` (see `docs/generated/README.md`) | +| Check script | `.github/scripts/docs-drift-check.sh` | +| Forbidden literals | `.github/docs-drift/forbidden-patterns.txt` | diff --git a/.github/docs-drift/forbidden-patterns.txt b/.github/docs-drift/forbidden-patterns.txt new file mode 100644 index 00000000..28087586 --- /dev/null +++ b/.github/docs-drift/forbidden-patterns.txt @@ -0,0 +1,19 @@ +# Substrings that must not appear under docs/ (stale or incorrect product docs). +# Lines starting with # and blank lines are ignored. +# Extend this list when env vars or filenames are renamed or removed in the Convoy server. + +# Primary server config file is convoy.json, not YAML. +convoy.yaml +convoy.yml + +# Wrong env naming (metrics use CONVOY_METRICS_* in config/config.go). +CONVOY_PROMETHEUS_ENABLED +CONVOY_PROMETHEUS_BACKEND + +# Wrong DB pool env names (see CONVOY_DB_MAX_OPEN_CONN / CONVOY_DB_MAX_IDLE_CONN). +CONVOY_DB_MAX_CONNECTIONS +CONVOY_DB_MAX_CONNECTION + +# Inverted or outdated precedence wording (canonical line uses cli > env > file). +environment variables` > `cli flags +environment variables > cli flags diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..d155c19e --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,12 @@ +## Summary + + + +## Checklist + +- [ ] **Config / env / CLI / metrics / Grafana docs** — Updated [`docs/deployment/configuration.mdx`](docs/deployment/configuration.mdx), [`docs/cli-file/convoy.mdx`](docs/cli-file/convoy.mdx), [`docs/product-manual/metrics.mdx`](docs/product-manual/metrics.mdx), and related pages **or** explicitly **N/A** (brief reason in a PR comment). See [Documentation sync (maintainer guide)](.github/DOCUMENTATION_SYNC.md). +- [ ] **Tests / preview** — Docs PRs: Mintlify preview (if applicable) looks correct. + +## Labels (optional) + +For docs-only or config-doc alignment work, consider **`documentation`** plus topic labels your team uses (**`config`**, **`metrics`**, **`cli`**, etc.) so these PRs are easy to filter. diff --git a/.github/scripts/docs-drift-check.sh b/.github/scripts/docs-drift-check.sh new file mode 100644 index 00000000..f00dcec3 --- /dev/null +++ b/.github/scripts/docs-drift-check.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Docs drift guard: forbidden substrings + lightweight cross-file consistency. +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +DOCS="$ROOT/docs" +PATTERNS_FILE="$ROOT/.github/docs-drift/forbidden-patterns.txt" +CONFIG_MDX="$DOCS/deployment/configuration.mdx" +METRICS_MDX="$DOCS/product-manual/metrics.mdx" +FEATURE_FLAGS_MDX="$DOCS/resources/feature-flags.mdx" + +fail() { + echo "docs-drift-check: ERROR: $*" >&2 + exit 1 +} + +if [[ ! -d "$DOCS" ]]; then + fail "docs directory missing: $DOCS" +fi + +echo "== Forbidden patterns ($PATTERNS_FILE) ==" +if [[ ! -f "$PATTERNS_FILE" ]]; then + fail "forbidden patterns file missing" +fi +while IFS= read -r line || [[ -n "$line" ]]; do + [[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue + pattern="$line" + if grep -rIlF --include='*.mdx' --include='*.md' "$pattern" "$DOCS" 2>/dev/null | grep -q .; then + echo "Matched forbidden pattern (fixed-string): $pattern" >&2 + grep -rInF --include='*.mdx' --include='*.md' "$pattern" "$DOCS" >&2 || true + fail "remove or update stale pattern above" + fi +done <"$PATTERNS_FILE" + +echo "== Required anchors (configuration.mdx) ==" +[[ -f "$CONFIG_MDX" ]] || fail "missing $CONFIG_MDX" + +require_in_file() { + local file="$1" + local needle="$2" + local msg="$3" + grep -qF "$needle" "$file" || fail "$msg (expected in $file): $needle" +} + +require_in_file "$CONFIG_MDX" "frain-dev/convoy/blob/main/config/config.go" \ + "configuration doc must link to server config source" +require_in_file "$CONFIG_MDX" 'cli flags` > `environment variables` > `convoy.json' \ + "configuration doc must document CLI > env > convoy.json precedence" + +count="$(grep -cF 'cli flags` > `environment variables` > `convoy.json' "$CONFIG_MDX" || true)" +[[ "$count" -eq 1 ]] || fail "precedence line must appear exactly once in configuration.mdx (count=$count)" + +echo "== Metrics / feature-flag naming consistency ==" +[[ -f "$METRICS_MDX" ]] || fail "missing $METRICS_MDX" +[[ -f "$FEATURE_FLAGS_MDX" ]] || fail "missing $FEATURE_FLAGS_MDX" + +# Keep metrics onboarding aligned with canonical env names in configuration.mdx. +for sym in CONVOY_METRICS_ENABLED CONVOY_METRICS_BACKEND CONVOY_METRICS_QUERY_TIMEOUT; do + require_in_file "$CONFIG_MDX" "$sym" "configuration reference must mention $sym" + require_in_file "$METRICS_MDX" "$sym" "metrics doc must mention $sym" +done +require_in_file "$CONFIG_MDX" "CONVOY_METRICS_SAMPLE_TIME" "configuration reference must mention CONVOY_METRICS_SAMPLE_TIME" + +require_in_file "$METRICS_MDX" "CONVOY_ENABLE_FEATURE_FLAG" "metrics doc must document feature flag env for prometheus" +require_in_file "$FEATURE_FLAGS_MDX" "CONVOY_ENABLE_FEATURE_FLAG" "feature flags doc must document CONVOY_ENABLE_FEATURE_FLAG" + +echo "docs-drift-check: OK" diff --git a/.github/workflows/convoy-docs-sync-receive.yml b/.github/workflows/convoy-docs-sync-receive.yml new file mode 100644 index 00000000..46cf3006 --- /dev/null +++ b/.github/workflows/convoy-docs-sync-receive.yml @@ -0,0 +1,90 @@ +# Receives repository_dispatch convoy_docs_sync from the Convoy server repo (or manual +# workflow_dispatch), regenerates docs on convoy main (or requested ref), copies +# docs/generated/* into docs/generated/convoy/, then opens or updates PR branch +# chore/sync-convoy-generated-docs via peter-evans/create-pull-request. +name: Receive Convoy docs sync + +on: + repository_dispatch: + types: + - convoy_docs_sync + workflow_dispatch: + inputs: + convoy_ref: + description: "Convoy repo ref (branch, tag, or SHA)" + required: false + default: main + +permissions: + contents: write + pull-requests: write + +concurrency: + group: convoy-docs-sync-receive-${{ github.repository }} + cancel-in-progress: true + +jobs: + sync: + runs-on: ubuntu-latest + env: + CONVOY_REF: ${{ github.event.inputs.convoy_ref || github.event.client_payload.convoy_ref || 'main' }} + steps: + - name: Check out convoy-website + uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 + with: + ref: main + + - name: Check out Convoy server + uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 + with: + repository: ${{ github.repository_owner }}/convoy + ref: ${{ env.CONVOY_REF }} + path: convoy-src + # Prefer PAT when Convoy is private or GITHUB_TOKEN cannot read it; else default token (public convoy). + token: ${{ secrets.CONVOY_CHECKOUT_TOKEN || github.token }} + + - name: Set up Go + uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6 + with: + go-version: "1.25.7" + + - name: Cache Go modules (Convoy) + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4 + with: + path: ~/go/pkg/mod + key: ${{ runner.os }}-convoy-docs-sync-receive-${{ hashFiles('convoy-src/go.sum') }} + + - name: Download modules + working-directory: convoy-src + run: go mod download + + - name: Regenerate reference artifacts + working-directory: convoy-src + run: make docs-generated + + - name: Copy generated artifacts into website tree + run: | + rm -rf docs/generated/convoy + mkdir -p docs/generated/convoy + cp -R convoy-src/docs/generated/. docs/generated/convoy/ + + - name: Open or update PR when drift exists + uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6 + with: + commit-message: "chore(docs): sync Convoy generated reference artifacts" + title: "chore(docs): sync Convoy generated reference artifacts" + body: | + This PR was opened or updated automatically after a **`convoy_docs_sync`** `repository_dispatch` (or a manual run of this workflow). + + **Source:** `${{ github.repository_owner }}/convoy` at ref **`${{ env.CONVOY_REF }}`**, `make docs-generated`, mirrored under `docs/generated/convoy/`. + + **What to do:** Review the diff; if it matches the config/CLI state you expect on Convoy `main`, merge. + + **Branch:** `chore/sync-convoy-generated-docs` is reused and updated when new drift appears. + + Maintainer reference: **Automation modes** → **Mode B — Generated reference sync (receive)** in `.github/DOCUMENTATION_SYNC.md` (dispatch token on the Convoy repo, optional `CONVOY_CHECKOUT_TOKEN` here). + branch: chore/sync-convoy-generated-docs + base: main + delete-branch: false + add-paths: | + docs/generated/** diff --git a/.github/workflows/docs-drift.yml b/.github/workflows/docs-drift.yml new file mode 100644 index 00000000..985c6f2c --- /dev/null +++ b/.github/workflows/docs-drift.yml @@ -0,0 +1,23 @@ +name: Docs drift + +on: + pull_request: + paths: + - 'docs/**' + - '.github/workflows/docs-drift.yml' + - '.github/scripts/docs-drift-check.sh' + - '.github/docs-drift/**' + - '.github/pull_request_template.md' + - '.github/DOCUMENTATION_SYNC.md' + +permissions: + contents: read + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Run docs drift checks + run: bash .github/scripts/docs-drift-check.sh diff --git a/docs/cli-file/convoy.mdx b/docs/cli-file/convoy.mdx index 576e226f..a60fc1bb 100644 --- a/docs/cli-file/convoy.mdx +++ b/docs/cli-file/convoy.mdx @@ -20,10 +20,10 @@ The Convoy CLI can be installed directly from your package manager or by buildin The installation procedures for Linux is split into two: - - Ubuntu and Debain users + - Ubuntu and Debian users - CentOS and RHEL users - **Ubuntu and Debain OS** + **Ubuntu and Debian** Install the Convoy CLI from apt repository: @@ -99,10 +99,21 @@ Available Commands: Use "Convoy [command] --help" for more information about a command. ``` -Global flags (database, Redis, tracing, metrics, billing, and so on) are defined on the root command; run `convoy --help` on your installed binary for the full list for that version. +Global flags (database, Redis, tracing, metrics, billing, feature flags, rate limits, and so on) are defined on the root command; run `convoy --help` on your installed binary for the authoritative list for that version. To get help for a subcommand, pass `-h` (for example `convoy agent -h`). +### Core commands + +| Command | Purpose | +| :------ | :------ | +| **`convoy server`** | Control plane: dashboard and management HTTP API on **`server.http.port`** (flag **`--port`**). | +| **`convoy agent`** | Data plane: queue consumers, broker ingest, SMTP-backed notifications, and data-plane HTTP (including **`/metrics`**) on **`server.http.agent_port`** (flag **`--port`**). | +| **`convoy migrate up`** | Apply all pending database migrations against the DB from your config / flags. Run before starting server and agent in a new environment or after upgrading. | +| **`convoy config`** | Print the resolved configuration as JSON (same merge rules as other commands: **`cli flags`** > **`environment variables`** > **`convoy.json`**—see [Configuration](/deployment/configuration)). Use the same **`--config`** and environment as the process you are inspecting. | + +Typical production layout: one or more **`server`** replicas behind a load balancer and multiple **`agent`** replicas; both use the same `convoy.json` (or equivalent env). See [Architecture](/deployment/architecture) and [Configuration](/deployment/configuration). + ### Historical CLI commands Older docs referred to **`convoy worker`**, **`convoy ingest`**, **`convoy stream`**, and **`convoy scheduler`**. Current Convoy exposes **`convoy server`** for the control-plane HTTP API and **`convoy agent`** for the data plane (queue consumers, broker ingest, and data-plane HTTP including `/metrics` on **`server.http.agent_port`**). A common local layout is **`convoy server`** on the API port and **`convoy agent`** on the agent port—for example **5005** / **5008** in [docker-compose.dev.yml](https://github.com/frain-dev/convoy/blob/main/docker-compose.dev.yml). See the [core gateway changelog](/changelog/core-gateway) for the unified agent change. @@ -157,7 +168,7 @@ The bootstrap creates a new user account. ### Migrate -Command: `convoy migrate` +Command: `convoy migrate` with subcommands **`up`**, **`down`**, and **`create`**. #### Synopsis @@ -175,36 +186,19 @@ Available Commands: Flags: -h, --help help for migrate - -Global Flags: - --config string Configuration file for convoy (default "./convoy.json") - --db-database string Database Database - --db-host string Database Host - --db-options string Database Options - --db-password string Database Password - --db-port int Database Port - --db-scheme string Database Scheme - --db-type string Database provider - --db-username string Database Username - --redis-database string Redis database - --redis-host string Redis Host - --redis-password string Redis Password - --redis-port int Redis Port - --redis-scheme string Redis Scheme - --redis-username string Redis Username - - ``` +Run **`convoy migrate up`** (alias **`migrate-up`**) to apply pending schema changes. It uses the same database settings as the rest of the CLI (`--config`, **`CONVOY_DB_*`**, or **`--db-*`** flags). For the full global flag list, run **`convoy migrate up --help`** on your binary. #### Description -The migrate command is responsible for running pending migrations and rolling back migrations. +Use **`migrate up`** in deploy scripts and release runbooks before starting **`server`** and **`agent`**. **`migrate down`** rolls back (optional **`--max`** on down—see **`convoy migrate down --help`**). **`migrate create`** is for Convoy developers adding SQL migrations. #### Command Flags -- `down`: Rollback migrations. -- `up`: Run all pending migrations +- **`up`**: Apply all pending migrations. +- **`down`**: Roll back migrations (see subcommand help for options). +- **`create`**: Generate a new migration file (development workflow). ### Config @@ -221,34 +215,17 @@ Usage: Flags: -h, --help help for config - -Global Flags: - --config string Configuration file for convoy (default "./convoy.json") - --db-database string Database Database - --db-host string Database Host - --db-options string Database Options - --db-password string Database Password - --db-port int Database Port - --db-scheme string Database Scheme - --db-type string Database provider - --db-username string Database Username - --redis-database string Redis database - --redis-host string Redis Host - --redis-password string Redis Password - --redis-port int Redis Port - --redis-scheme string Redis Scheme - --redis-username string Redis Username - - ``` +Global flags are the same as the root command; run **`convoy config --help`** for the full list. + ##### Description -The config command outputs the configuration for your active instances. +**`convoy config`** loads configuration the same way **`server`** and **`agent`** do, then prints the merged struct as indented JSON ([implementation](https://github.com/frain-dev/convoy/blob/main/cmd/config/config.go)). Use it to verify effective values; precedence is **`cli flags`** > **`environment variables`** > **`convoy.json`** (see [Configuration](/deployment/configuration)). It does not start HTTP services or run migrations. #### Command Flags -- `--help`: Get help on the `config` command. +- **`--help`**: Help for the `config` command. ### Server @@ -300,7 +277,7 @@ The **`server`** command runs Convoy’s **control-plane** HTTP API (dashboard, - **`--env`**, **`--logger`**, **`--host`**, **`--root-path`**: Process environment, logging, hostname, and path prefix behind a reverse proxy. - **`--max-response-size`**: Cap on stored subscriber response payload size (see [configuration](/deployment/configuration)). - **`--native`**: Enable native-realm authentication. -- **`--promaddr`**: Appears on `convoy server --help` as “Prometheus dsn”; it is **not** passed through in `buildServerCliConfiguration` in current source—use root metrics flags / `convoy.json` for Prometheus settings instead. +- **`--promaddr`**: Listed on `convoy server --help` as “Prometheus dsn”; prefer **`metrics`** in `convoy.json`, **`CONVOY_METRICS_*`**, and root **`--metrics-*`** flags for Prometheus export (see [Metrics](/product-manual/metrics)). - **`--proxy`**: Outbound HTTP proxy for the server process. - **`--ssl`**, **`--ssl-cert-file`**, **`--ssl-key-file`**: TLS for the control HTTP server. @@ -388,7 +365,7 @@ Global Flags: #### Description -At core, convoy is an asynchronous messaging service. It relies on message brokers to carry out its duty. This command is used to filter event deliveries that match the filters set; it will purge the queues for all matched event deliveries and re-enqueue them all. +**`convoy retry`** selects event deliveries matching **`--status`** and **`--time`** and re-queues them for delivery. Use it for operational recovery after incidents or misconfiguration; it does not replace normal retry policies on each endpoint. #### Command Flags diff --git a/docs/deployment/architecture.mdx b/docs/deployment/architecture.mdx index fcf6962a..af9b6ae6 100644 --- a/docs/deployment/architecture.mdx +++ b/docs/deployment/architecture.mdx @@ -41,10 +41,10 @@ These instances are responsible for the following: Processing all asynchronous tasks in the queue. These can range from important functions like processing webhook events to ancillary tasks like sending emails, purging stale events ([retention policies](./configuration#configuration-reference)) etc. - These workers are designed to be stateless and can be scaled horizontally to increase the throughput of the entire system. + Agent replicas are stateless and can be scaled horizontally to increase the throughput of the entire system. 2. **Message Brokers Integration**: - These stateless workers are used to consume webhook events from [message brokers](/product-manual/message-brokers). + Agent replicas consume webhook events from [message brokers](/product-manual/message-brokers). They poll source configuration from the database and consume events from the broker and write it to the redis queue. @@ -77,6 +77,10 @@ We highly recommend and use [Smokescreen](https://github.com/stripe/smokescreen) **Sizing guidance:** Treat each egress instance like a stateless proxy: a practical starting point is **1 vCPU (1000m)** and **2GB of RAM (2000Mi)** per instance, then raise resources if you see high concurrent outbound connections or large response bodies. Keep **at least three** instances behind a load balancer for availability, as described above. +## Observability + +Prometheus scrapes **`/metrics`** on the processes you run (typically **`convoy server`** and **`convoy agent`** on **`server.http.port`** and **`server.http.agent_port`**). See [Metrics](/product-manual/metrics) for scrape examples, metric names, and [Grafana-oriented guidance](/product-manual/metrics#grafana-and-dashboards). Optional product telemetry (Mixpanel) is described in [Telemetry](/resources/telemetry). + ## Third-Party Dependencies ### PostgreSQL 15+ diff --git a/docs/deployment/configuration.mdx b/docs/deployment/configuration.mdx index 1ca36fd6..d204746d 100644 --- a/docs/deployment/configuration.mdx +++ b/docs/deployment/configuration.mdx @@ -22,6 +22,8 @@ values set in both of them will override the same values set in your `convoy.jso --- +The JSON keys and environment variable names below follow the Convoy server’s `Configuration` struct in [`config/config.go`](https://github.com/frain-dev/convoy/blob/main/config/config.go) (`DefaultConfiguration` holds built-in defaults). This block uses JSON-with-comments style for documentation only; copy fields into valid `convoy.json` as needed. + Here's the full configuration reference ```json Config @@ -31,13 +33,13 @@ Here's the full configuration reference // for the REST API. This config specifies which version your // instance should use. The environment variable equivalent is // CONVOY_API_VERSION. The default value is the latest at the time of release. - "api_version": "2024-01-01", + "api_version": "2025-11-24", // [optional] // This is used to specify the host the server is running on. // It is used in emails with links to access the instance. // E.g. user invitation. The environment variable equivalent is - // CONVOY_HOST. The default value is localhost. + // CONVOY_HOST. The default in code is localhost:5005. "host": "hooks.domain.com", // [required] @@ -70,7 +72,7 @@ Here's the full configuration reference // Database port. The environment equivalent is // CONVOY_DB_PORT. The default value is 5432. - "port": 5432 + "port": 5432, // This is used to tune the connection to the database // for maximum performance. It specifies the maximum @@ -86,9 +88,9 @@ Here's the full configuration reference // This is used to tune the connection to the database // for maximum performance. It specifies how long any - // connection is kept in the pool before being killed. + // connection is kept in the pool before being killed (seconds). // The environment variable equivalent is CONVOY_DB_CONN_MAX_LIFETIME - "conn_max_lifetime": 10 + "conn_max_lifetime": 3600, // [optional] // Read replicas configuration. These are additional database @@ -158,13 +160,12 @@ Here's the full configuration reference // This is used to configure the port the ingest server runs on. This // config has no effect on the other components. The environment - // variable equivalent is INGEST_PORT. - "ingest_port": 5007, + // variable equivalent is INGEST_PORT. The default in code is 5009. + "ingest_port": 5009, - // This is used to configure the port the socket server runs on. This - // config has no effect on the other components. The environment - // variable equivalent is SOCKET_PORT. - "socket_port": 5008, + // Port for the socket server when used. The environment variable + // equivalent is SOCKET_PORT (no non-zero default in DefaultConfiguration). + "socket_port": 0, // This is used to configure the forward proxy URL. This must be set // on all workers if you want to configure a forward proxy. The @@ -192,28 +193,29 @@ Here's the full configuration reference // Redis username. The environment variable equivalent is // CONVOY_REDIS_USERNAME. - "username": "redis", + "username": "", // Redis password. The environment variable equivalent is // CONVOY_REDIS_PASSWORD. - "password": "redis", + "password": "", - // Redis database. The environment variable equivalent is - // CONVOY_REDIS_DATABASE. The default: 0 - "database": 0, + // Redis logical database index (string in config). The environment variable equivalent is + // CONVOY_REDIS_DATABASE. + "database": "0", // [optional] - // Enable TLS encryption for Redis connections. The environment - // variable equivalent is CONVOY_REDIS_TLS_ENABLED. The default - // value is false. - "tls_enabled": false, + // TLS: skip verify, cert paths, or CA for Redis. Env vars: + // CONVOY_REDIS_TLS_SKIP_VERIFY, CONVOY_REDIS_TLS_CERT_FILE, CONVOY_REDIS_TLS_KEY_FILE, + // CONVOY_REDIS_TLS_CA_CERT_FILE. Prefer scheme "rediss" when the server expects TLS. + "tls_skip_verify": false, + "tls_cert_file": "", + "tls_key_file": "", + "tls_ca_cert_file": "", // [optional] - // Redis cluster addresses. The environment variable equivalent is - // CONVOY_REDIS_ADDRESSES. The value takes precedence over the - // other redis configuration in this object. The urls are fully - // qualified domain names. - "addresses": "{url}, {url}, ...", + // Redis cluster / sentinel address list (comma-separated). The environment variable equivalent is + // CONVOY_REDIS_CLUSTER_ADDRESSES. CLI flag: --redis-addresses. Takes precedence over host/port when set. + "addresses": "", // [optional] // The master name for Redis Sentinel setups. Required when scheme is redis-sentinel. @@ -231,6 +233,11 @@ Here's the full configuration reference "sentinel_password": "" }, + // [optional] + // Separate from `metrics`: legacy Prometheus push/address field used by some tooling. Env: CONVOY_PROM_DSN. + "prometheus": { + "dsn": "" + }, // [optional] // your convoy license key, you will to specify this in order to access paid features. @@ -284,7 +291,7 @@ Here's the full configuration reference // Specify the smtp from email. The environment variable equivalent is // CONVOY_SMTP_FROM "from": "[email protected]" - } + }, // [optional] // Specify the retention policy configurations for your convoy instance @@ -296,7 +303,7 @@ Here's the full configuration reference // Specify whether the retention policy job should run. The environment variable equivalent is CONVOY_RETENTION_POLICY_ENABLED "enabled": false, - } + }, // [optional] // This is used to configure the logger for any Convoy component. @@ -306,7 +313,7 @@ Here's the full configuration reference // are: fatal, error, warn, info, debug. The environment variable // equivalent is CONVOY_LOGGER_LEVEL. The default value is error. "level": "debug" - } + }, // [optional] // This is used to configure how we generate and push traces to. While @@ -328,7 +335,7 @@ Here's the full configuration reference "sample_rate": 1.0, // OTel backend URL. Only grpc ingestion urls are supported. - // The environment variable equivalent is CONVOY_OTEL_SAMPLE_RATE + // The environment variable equivalent is CONVOY_OTEL_COLLECTOR_URL "collector_url": "", // Specifies if server certs should be verified or not. @@ -359,6 +366,12 @@ Here's the full configuration reference } }, + // [optional] + // Product telemetry (Mixpanel), separate from Prometheus metrics. The environment variable equivalent is CONVOY_ANALYTICS_ENABLED. + "analytics": { + "enabled": true + }, + // [optional] // We use this to configure deleting events away from PostgreSQL. This config // is optional, but if you're configuring retention policies on your project, @@ -366,7 +379,8 @@ Here's the full configuration reference "storage_policy": { // The type can be either s3 or on_prem. The environment variable equivalent is CONVOY_STORAGE_POLICY_TYPE. - "type": "s3", + // The default in code is on_prem. + "type": "on_prem", // If type above is set to s3, then this config needs to be supplied. "s3": { @@ -406,7 +420,7 @@ Here's the full configuration reference // The environment variable equivalent is CONVOY_STORAGE_PREM_PATH. "path": "" } - } + }, // [optional] // This is used to configure authentication on your instance. The default values @@ -475,11 +489,12 @@ Here's the full configuration reference // The environment variable equivalent is CONVOY_SSO_REDIRECT_URL. "redirect_url": "" } - } + }, // [optional] - // This is used to configure the server's rate limit. It defaults to 25 reqs/sec. - "api_rate_limit": 25, + // HTTP API rate limit (requests per second) for control-plane and ingest HTTP routes that use this limiter. + // The environment variable equivalent is CONVOY_API_RATE_LIMIT. The default in code is 1000. + "api_rate_limit": 1000, // [optional] // This is used to configure the root path for Convoy deployments. This is useful @@ -499,15 +514,14 @@ Here's the full configuration reference "max_response_size": 50, // [optional] - // This is used to configure the ingestion rate limit on the data plane. Internally, - // it is a distributed rate limiter implemented on Redis to ensure consistent limits - // across the cluster. - "instance_ingest_rate": 50 + // Ingestion rate limit (events per second) for ingest paths that use InstanceIngestRate (API event creation, brokers, etc.). + // The environment variable equivalent is CONVOY_INSTANCE_INGEST_RATE. The default in code is 1000. + "instance_ingest_rate": 1000, // [optional] - // This is used to control the agent's execution mode. The values can be either - // default, retry and events. - "execution_mode": "retry" + // Agent execution mode: default, retry, or events. Maps to CLI `convoy agent --mode`. + // The environment variable equivalent is CONVOY_WORKER_EXECUTION_MODE. + "worker_execution_mode": "default", // [optional] // This is used to configure how we circuit break client endpoints. @@ -516,38 +530,38 @@ Here's the full configuration reference // This is the frequency of polling the database in seconds to calculate // an endpoint's error rate. // The environment variable equivalent is CONVOY_CIRCUIT_BREAKER_SAMPLE_RATE. - "sample_rate": 10, + "sample_rate": 30, // This is the total amount of time in seconds wait in the half-open state // to enable the endpoint to recover. // The environment variable equivalent is CONVOY_CIRCUIT_BREAKER_ERROR_TIMEOUT. - "error_timeout": 60, + "error_timeout": 30, // This is the % number of failures in the observability window, before // we tripe the circuit. // The environment variable equivalent is CONVOY_CIRCUIT_BREAKER_FAILURE_THRESHOLD. - "failure_threshold": 10, + "failure_threshold": 70, // This is the total number of successes in the half-open state that'll // trigger the circuit back to a closed state. // The environment variable equivalent is CONVOY_CIRCUIT_BREAKER_SUCCESS_THRESHOLD. - "success_threshold": 50, + "success_threshold": 5, // This is the total number of requests an endpoint must have before // we begin calculating error rate. // The environment variable equivalent is CONVOY_CIRCUIT_BREAKER_MINIMUM_REQUEST_COUNT. - "minimum_request_count": 15, + "minimum_request_count": 10, // This is the total time window to use in calculating the endpoint error rate. // It is configured in minutes. // The environment variable equivalent is CONVOY_CIRCUIT_BREAKER_OBSERVABILITY_WINDOW. - "observability_window": 30, + "observability_window": 5, // This is the total number of consecutive transitions to the open state // before we finally disable the endpoint. // The environment variable equivalent is CONVOY_CIRCUIT_BREAKER_CONSECUTIVE_FAILURE_THRESHOLD. "consecutive_failure_threshold": 10 - } + }, // [optional] // This configures a set of IP addresses that we can connect to. The goal @@ -563,7 +577,7 @@ Here's the full configuration reference // This is used to configure the set of IP addresses that are blocked. // The equivalent environment variable is CONVOY_DISPATCHER_BLOCK_LIST. - "block_list": [ "127.0.0.0/8", "169.254.169.254", "::1/128" ], + "block_list": [ "127.0.0.0/8", "::1/128" ], // [optional] Custom CA configuration for the dispatcher. // The environment variable equivalent is CONVOY_DISPATCHER_CACERT_PATH @@ -580,14 +594,14 @@ Here's the full configuration reference // Whether to skip ping validation entirely. // The environment variable equivalent is CONVOY_DISPATCHER_SKIP_PING_VALIDATION "skip_ping_validation": false - } + }, // [optional] // This is the total time in seconds used to cap the exponential backoff // retry algorithm. For example, if set to 7200, it means we will never // exceed 2 hours in retry delay. - // The equivalent environment variable is CONVOY_MAX_RETRY_SECONDS. - "max_retry_seconds": 2000, + // The equivalent environment variable is CONVOY_MAX_RETRY_SECONDS. The root CLI flag --max-retry-seconds defaults to 7200. + "max_retry_seconds": 7200, // [optional] // Configure Pyroscope profiling capabilities. @@ -696,3 +710,9 @@ Here's the full configuration reference } ``` + +## Related documentation + +- [Metrics](/product-manual/metrics) — Prometheus endpoints, metric semantics, and Grafana operational guidance. +- [Rate limits](/product-manual/rate-limits) — how **`api_rate_limit`** and **`instance_ingest_rate`** differ. +- [Telemetry](/resources/telemetry) — optional product analytics (Mixpanel), separate from application metrics. diff --git a/docs/generated/README.md b/docs/generated/README.md new file mode 100644 index 00000000..b498b39e --- /dev/null +++ b/docs/generated/README.md @@ -0,0 +1,17 @@ +# Generated reference artifacts (Convoy mirror) + +**Mirrored outputs** from the Convoy server repo so the docs site and automation use the same JSON the server generates. + +| Path | Meaning | +|------|---------| +| `convoy/` | Copy of the server’s `docs/generated/` after `make docs-generated` (e.g. `config-reference.json`, `cli-reference.json`, upstream README). | + +## How it is updated + +- **Mode B (implemented):** [.github/workflows/convoy-docs-sync-receive.yml](../../.github/workflows/convoy-docs-sync-receive.yml) runs on `repository_dispatch` **`convoy_docs_sync`** or manual **`workflow_dispatch`**. It checks out this repo on `main`, checks out `{owner}/convoy` (same GitHub `owner` as convoy-website) at `main` or the ref from the payload / workflow input, runs `make docs-generated`, copies into `docs/generated/convoy/`, then opens or updates PR branch **`chore/sync-convoy-generated-docs`** against **`main`**. Only `docs/generated/**` is committed. + +- **Manual:** Run `make docs-generated` in a local Convoy clone, copy `convoy/docs/generated/*` into `docs/generated/convoy/`, open a PR. + +**Related:** Mode A: [.github/workflows/docs-drift.yml](../../.github/workflows/docs-drift.yml) runs drift and forbidden-pattern checks on PRs touching `docs/**` (and related paths); it does not refresh this directory. + +Triggers and secrets: **Automation modes** in [.github/DOCUMENTATION_SYNC.md](../../.github/DOCUMENTATION_SYNC.md). diff --git a/docs/product-manual/message-brokers/kafka.mdx b/docs/product-manual/message-brokers/kafka.mdx index babd0d02..e3e0ee54 100644 --- a/docs/product-manual/message-brokers/kafka.mdx +++ b/docs/product-manual/message-brokers/kafka.mdx @@ -9,7 +9,7 @@ This is currently only supported in [outgoing projects](../../product-manual/org To connect your Kafka topic, you need to supply: - The kafka **cluster hostname and port** (e.g., localhost:9092) - The **topic name** (e.g., audit-log-events) -- The **number of workers** -- the **number of partitions** for the topic should match the number of workers. +- The number of **Convoy agent replicas** consuming the topic — the **number of partitions** for the topic should match how you scale agents for that source. - The **consumer group id** --it is recommended that you set one-- which by default this is set to an empty string. - Supply authentication configured on the cluster @@ -55,6 +55,6 @@ There are two ways to ingest events into Convoy from your Kafka topic: ``` ## Things to note -- Messages are read of the topic one-by-one by each worker reading off a partition. +- Messages are read off the topic one-by-one by each agent consumer reading a partition. - Acknowledgements are done per message. -- Ingestion is rate-limited (50 per second), and this can be configured by setting the `CONVOY_INGEST_RATE` environment variable. +- Ingestion is subject to the **instance ingest rate** (`CONVOY_INSTANCE_INGEST_RATE` / `instance_ingest_rate`, default **1000**/s in code). See [Rate limits](/product-manual/rate-limits). diff --git a/docs/product-manual/message-brokers/sqs.mdx b/docs/product-manual/message-brokers/sqs.mdx index f0307a2b..97a4f0cf 100644 --- a/docs/product-manual/message-brokers/sqs.mdx +++ b/docs/product-manual/message-brokers/sqs.mdx @@ -70,4 +70,4 @@ For a full list of reference payloads, see [our guide on ingesting events](../.. ## Things to note - Messages are read of the topic one-by-one by each worker reading off the queue. - Acknowledgements are done per message. -- Ingestion is rate-limited (50 per second), and this can be configured by setting the `CONVOY_INGEST_RATE` environment variable. +- Ingestion is subject to the **instance ingest rate** (`CONVOY_INSTANCE_INGEST_RATE` / `instance_ingest_rate`, default **1000**/s in code). See [Rate limits](/product-manual/rate-limits). diff --git a/docs/product-manual/metrics.mdx b/docs/product-manual/metrics.mdx index 22cf485a..2747c546 100644 --- a/docs/product-manual/metrics.mdx +++ b/docs/product-manual/metrics.mdx @@ -10,11 +10,12 @@ Convoy exports metrics about the state of events received and sent via Prometheu ## Enabling Metrics -Metrics are currently in beta, and aren't enabled by default. To enable them, you need to +Metrics are not enabled by default. To turn them on you need all of the following: -- Enable the `prometheus` feature flag using `CONVOY_ENABLE_FEATURE_FLAG=prometheus` -- Set the metrics backend env var `CONVOY_METRICS_BACKEND` -- Ensure your **license** allows Prometheus export (the server checks license capabilities before registering metrics) +- Enable the **`prometheus` feature flag** (`CONVOY_ENABLE_FEATURE_FLAG=prometheus` or `--enable-feature-flag=prometheus`). +- Set the metrics backend to Prometheus: JSON **`metrics.metrics_backend`**: `"prometheus"`, env **`CONVOY_METRICS_BACKEND=prometheus`**, or CLI **`--metrics-backend=prometheus`**. +- Set **`metrics.enabled`** / **`CONVOY_METRICS_ENABLED=true`** together with **`metrics.metrics_backend`**: `"prometheus"` / **`CONVOY_METRICS_BACKEND=prometheus`** when using `convoy.json` or environment. Pure CLI use can instead pass **`--metrics-backend=prometheus`**, which (with the prometheus feature flag) enables the merged metrics configuration loaded for **`server`** and **`agent`**. +- Ensure your **license** allows Prometheus export where the handler enforces entitlements. Either one of the two code blocks below will work. @@ -23,8 +24,10 @@ convoy agent --metrics-backend=prometheus --enable-feature-flag=prometheus ``` ```shell enabling convoy metrics using env vars +export CONVOY_METRICS_ENABLED=true export CONVOY_METRICS_BACKEND=prometheus -convoy agent --enable-feature-flag=prometheus +export CONVOY_ENABLE_FEATURE_FLAG=prometheus +convoy agent ``` Scrape **`GET /metrics`** on each process you run. Typical split deployments use **`convoy server`** (control plane API, default HTTP port **5005**) and **`convoy agent`** (data plane: ingest, queue consumers, and data-plane HTTP including `/metrics`, default **`agent_port` 5008**). For example, [docker-compose.dev.yml](https://github.com/frain-dev/convoy/blob/main/docker-compose.dev.yml) maps **`web`** → `5005` and **`agent`** → `5008`. Both can register the shared Prometheus registry when Redis + Postgres are available. Export still requires the **license** to allow Prometheus metrics where the handler enforces it. @@ -69,6 +72,31 @@ max(convoy_event_queue_backlog_seconds) Histogram series use Prometheus’ usual `_bucket` / `_sum` / `_count` suffixes for `convoy_end_to_end_latency`. +## Grafana and dashboards + +Use Grafana (or any Prometheus-backed UI) to track whether observability keeps pace with the product and stays trustworthy. + +**Dashboard and query hygiene** + +- **Version and review**: Treat dashboard JSON like code—store it in git when possible, review changes in PRs, and note which Convoy version each dashboard targets. When upgrading Convoy, re-check panels that use concrete metric or label names (see tables below and [`internal/pkg/metrics`](https://github.com/frain-dev/convoy/tree/main/internal/pkg/metrics) in the Convoy repo). +- **Metric contract**: Prefer recording rules or documented PromQL snippets next to dashboards so renames or label changes surface during review, not only in production. +- **Environment parity**: If staging and production differ (extra labels, fewer scrapes, or different scrape intervals), document that on the dashboard or in your internal runbook so comparisons stay honest. + +**Data quality checks** + +- **Scrape health**: Alert on Prometheus **`up`** for each `job` that scrapes Convoy, on **`scrape_samples_scraped`** collapsing to zero for critical jobs, and on rule evaluation errors if you use recording rules. +- **Series sanity**: Compare **`convoy_ingest_total`** rates to traffic you expect; sudden flatlines or orders-of-magnitude drift often indicate scrape, network, or process issues—not just low traffic. +- **Cardinality**: High-cardinality labels (many unique `endpoint`, `source`, or `project` values) increase cost; use Grafana’s **Explore** or Prometheus **TSDB status** to spot exploding label sets after config or feature changes. +- **Postgres-backed gauges**: Queue depth metrics refresh on **`metrics.prometheus_metrics.sample_time`**; stale values can reflect sampling interval, DB load, or query timeouts (`CONVOY_METRICS_QUERY_TIMEOUT` / `query_timeout`), not only empty queues. + +**Alerts and runbooks** + +- **Noise vs signal**: Pair alerts on error rates or backlog with minimum traffic thresholds (for example, a `for:` duration, or a separate guard on ingest rate) to avoid flapping on idle environments. +- **Runbook links**: Add a **runbook URL** or on-call note to each alert annotation (Grafana Alerting, Alertmanager `annotations.runbook_url`, etc.) describing first steps: check Convoy process health, Redis, Postgres, recent deploys, and the relevant section of this metrics doc. +- **License-gated metrics**: If `/metrics` returns minimal or empty output after a license change, verify entitlements before chasing infrastructure issues. + +For product usage analytics (Mixpanel), see [Telemetry](/resources/telemetry)—that pipeline is separate from Prometheus. + ## Ingest counters and end-to-end latency These are registered from **`internal/pkg/metrics/data_plane.go`** when Prometheus is enabled and the license allows export. Labels: `project` and `source` on ingest counters; `project` and `endpoint` on the histogram. @@ -86,7 +114,7 @@ The code also defines a **`convoy_ingest_latency`** histogram (per `project`); y These come from **custom collectors**, not from `data_plane.go`. When metrics are enabled, `RegisterQueueMetrics` attaches the **Redis queue** and **Postgres** implementations to the same registry, so they appear alongside the series above on `/metrics` for that process. In **server + agent** deployments, queue and ingest series are normally observed on the **agent** scrape target (data plane); the control server exposes its own `/metrics` for whatever it registers. -Postgres-backed values are refreshed on a **sample interval** (`metrics.prometheus.sample_time`). Depending on version and schema, queries may use **materialized views** or live SQL—see the server release notes if you upgrade. +Postgres-backed values are refreshed on a **sample interval** (`metrics.prometheus_metrics.sample_time`). Depending on version and schema, queries may use **materialized views** or live SQL—see the server release notes if you upgrade. ### Redis (Asynq) queues diff --git a/docs/product-manual/rate-limits.mdx b/docs/product-manual/rate-limits.mdx index 373a54ed..d3e8edd8 100644 --- a/docs/product-manual/rate-limits.mdx +++ b/docs/product-manual/rate-limits.mdx @@ -8,8 +8,10 @@ title: Rate Limits ## Ingress Rate limits -This controls how many events Convoy can ingest per second -This is configured by setting the `CONVOY_INGEST_RATE` env var. +This controls how many events Convoy can ingest per second across the instance for routes that use the instance ingest limiter (message brokers, event-creation APIs, and related ingest paths). + +Configure it with **`instance_ingest_rate`** in `convoy.json`, the environment variable **`CONVOY_INSTANCE_INGEST_RATE`**, or the root CLI flag **`--instance-ingest-rate`**. The default in [`config/config.go`](https://github.com/frain-dev/convoy/blob/main/config/config.go) is **1000** events per second unless you override it. + This value limits the number of: - Messages read off message brokers @@ -25,13 +27,16 @@ There is no instance wide configuration, instead egress rates are configured per ## HTTP API Rate Limits -This controls how requests per second Convoy can process. +This controls how many HTTP requests per second Convoy allows on routes protected by the HTTP API rate limiter. + +Configure it with **`api_rate_limit`** in `convoy.json`, **`CONVOY_API_RATE_LIMIT`**, or **`--api-rate-limit`**. The default in code is **1000** requests per second. + This includes: - Headless control plane APIs (used for managing project resources). -- Ingress APIs +- Some ingest HTTP routes that share this limiter (see server routing in the Convoy repo for details). -Reaching this limit will cause requests to receive a `429 (Rate Limit Exceeded)` error response from all APIs. +Reaching this limit will cause affected requests to receive a `429 (Rate Limit Exceeded)` error response. > Requests made to the Convoy instance via the UI don't count towards rate limits. diff --git a/docs/resources/feature-flags.mdx b/docs/resources/feature-flags.mdx index e68313bc..9757f70b 100644 --- a/docs/resources/feature-flags.mdx +++ b/docs/resources/feature-flags.mdx @@ -28,7 +28,7 @@ These features can be enabled per organization: ### Prometheus Metrics `--enable-feature-flag=prometheus`
-This is Convoy native metrics that enable you observe the internal state of the data plane from the ingestion pipeline to the delivery pipeline. See the [docs](/product-manual/metrics). +This is Convoy native metrics that enable you observe the internal state of the data plane from the ingestion pipeline to the delivery pipeline. See [Metrics](/product-manual/metrics) and [Grafana-oriented operations](/product-manual/metrics#grafana-and-dashboards). ### Read Replicas diff --git a/docs/resources/telemetry.mdx b/docs/resources/telemetry.mdx index 2c09d0ed..1dbdbea1 100644 --- a/docs/resources/telemetry.mdx +++ b/docs/resources/telemetry.mdx @@ -8,7 +8,7 @@ title: Telemetry Convoy tracks usage of its products which helps the team to improve and deliver better software. This page explains what we track, how we track it and how to opt out of telemetry at any time. -This is **product telemetry** (Mixpanel), not application metrics or distributed traces. For **Prometheus** metrics exposed by your instance, see [Metrics](/product-manual/metrics). For **OpenTelemetry / Sentry / Datadog** tracing configuration, see the [Tracing](/product-manual/metrics#tracing) section on that same page. +This is **product telemetry** (Mixpanel), not application metrics or distributed traces. For **Prometheus** metrics exposed by your instance, see [Metrics](/product-manual/metrics) (including [Grafana and dashboards](/product-manual/metrics#grafana-and-dashboards)). For **OpenTelemetry / Sentry / Datadog** tracing configuration, see the [Tracing](/product-manual/metrics#tracing) section on that same page. ### What data is collected