From cdfcf61d1e982ae0eba131652f15656615eeeaac Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Fri, 13 Nov 2026 23:02:30 +0100 Subject: [PATCH 1/9] Harden defaults: require API auth, guard callbacks, restrict job networks - HadesAPI: AUTH_KEY required (ALLOW_UNAUTHENTICATED=true as dev escape hatch), Swagger behind auth, /build body capped at 1 MiB, duplicate step ids rejected - Dashboard: atomic login attempt counter, login body cap, broader secret redaction - HadesLogManager: callback and status webhook targets on internal addresses refused (CALLBACK_ALLOWED_HOSTS exempts), no redirects, proxy env ignored - Docker executor: step network allow-list (DOCKER_ALLOWED_NETWORKS), optional default/max job timeouts, redacted debug logs - Operator and Helm: hardened job pods, reduced operator role, dedicated service accounts, optional NetworkPolicies, DeleteOnComplete default true - Compose, deploy overlay, Ansible: loopback port binding, required credentials, 0600 secret files - CI: least-privilege permissions, SHA-pinned actions, no secrets inherit - Docs and generated OpenAPI/Helm docs updated Co-Authored-By: Claude Mythos 5.2 --- .env.example | 2 +- .github/workflows/benchmark.yml | 10 +- .github/workflows/ci.yml | 50 ++++++---- .github/workflows/deploy-docs.yml | 4 +- .github/workflows/deploy-k8s-test.yml | 2 + .github/workflows/deploy-k8s.yml | 7 +- .github/workflows/e2e.yml | 8 +- .github/workflows/release-chart.yml | 12 ++- .github/workflows/verify-crd.yml | 5 +- .github/workflows/verify-openapi.yml | 4 +- AGENTS.md | 8 +- HadesAPI/Readme.md | 8 +- HadesAPI/dashboard/auth.go | 35 +++---- HadesAPI/dashboard/dashboard_test.go | 62 +++++++++++- HadesAPI/main.go | 22 ++++- HadesAPI/main_test.go | 13 +++ HadesAPI/router.go | 30 ++++-- HadesAPI/router_test.go | 48 ++++++++++ HadesAPI/web/e2e/serve.sh | 1 + HadesLogManager/Readme.md | 13 +++ HadesLogManager/http_client.go | 94 ++++++++++++++++++- HadesLogManager/http_client_test.go | 82 ++++++++++++++++ HadesLogManager/main.go | 13 ++- HadesLogManager/processor.go | 3 +- HadesLogManager/processor_test.go | 36 ++++++- HadesLogManager/status_dispatcher.go | 5 + HadesLogManager/status_dispatcher_test.go | 2 +- HadesLogManager/status_webhook.go | 6 +- HadesLogManager/status_webhook_test.go | 30 +++++- .../controller/buildjob_build_test.go | 27 ++++++ .../controller/buildjob_controller.go | 23 ++++- HadesScheduler/docker/env.go | 11 +++ HadesScheduler/docker/job.go | 8 +- HadesScheduler/docker/job_test.go | 15 ++- HadesScheduler/docker/limits_test.go | 71 ++++++++++++++ HadesScheduler/docker/options.go | 28 ++++++ HadesScheduler/docker/scheduler.go | 29 ++++++ HadesScheduler/docker/scheduler_test.go | 1 + HadesScheduler/main.go | 5 +- Makefile | 8 +- Readme.md | 26 ++++- ansible/hades/README.md | 8 +- ansible/hades/defaults/main.yml | 4 +- ansible/hades/tasks/check_host_config.yml | 7 ++ ansible/hades/tasks/install_api.yml | 6 +- ansible/hades/tasks/install_scheduler.yml | 6 +- .../hades/templates/docker-compose-api.yml.j2 | 8 +- bruno/HadesLogManager/Get Job Build Logs.bru | 2 +- bruno/HadesLogManager/Get Job Status.bru | 2 +- bruno/HadesLogManager/Get Jobs.bru | 2 +- bruno/HadesLogManager/health check.bru | 2 +- .../Create Build Job (Image Pull Failure).bru | 2 +- .../Create Build Job (Resource Limits).bru | 2 +- ...Job (Test Fail) with Result Parse Step.bru | 2 +- bruno/api/Create Build Job (Test Fail).bru | 2 +- ... Job (Test Succeed - Result Container).bru | 2 +- ... (Test Succeed) with Result Parse Step.bru | 2 +- bruno/api/Create Build Job (Test Succeed).bru | 2 +- .../Create Build Job (long-running-debug).bru | 2 +- bruno/api/Health Check (Ping).bru | 2 +- bruno/environments/Hades Host.bru | 1 + bruno/environments/Kubernetes.bru | 1 + bruno/environments/Local-Development.bru | 1 + bruno/environments/docker-compose.bru | 1 + compose.yml | 24 +++-- docker-compose.deploy.yml | 17 ++-- docker-compose.test.yml | 2 +- docs/api.md | 9 +- docs/configuration.md | 13 ++- helm/hades/Readme.md | 12 ++- .../hades/templates/hades-api-deployment.yaml | 7 ++ .../templates/hades-api-serviceaccount.yaml | 6 ++ .../hades-log-manager-deployment.yaml | 4 + .../hades-log-manager-serviceaccount.yaml | 6 ++ helm/hades/templates/hades-operator-role.yaml | 4 +- helm/hades/templates/networkpolicy.yaml | 70 ++++++++++++++ helm/hades/values.yaml | 46 +++++++-- renovate.json | 3 +- shared/redact/redact.go | 55 ++++++++++- website/docs/deployment/traefik.md | 23 +++-- website/docs/installation/docker-mode.md | 11 ++- website/docs/operation-modes/k8s-operator.md | 8 +- website/docs/reference/api.md | 2 +- website/docs/reference/configuration.md | 11 ++- website/docs/usage/submitting-jobs.md | 2 +- 85 files changed, 1088 insertions(+), 183 deletions(-) create mode 100644 HadesAPI/main_test.go create mode 100644 HadesLogManager/http_client_test.go create mode 100644 HadesScheduler/docker/limits_test.go create mode 100644 helm/hades/templates/hades-api-serviceaccount.yaml create mode 100644 helm/hades/templates/hades-log-manager-serviceaccount.yaml create mode 100644 helm/hades/templates/networkpolicy.yaml diff --git a/.env.example b/.env.example index 5c4531be..be56d2dc 100644 --- a/.env.example +++ b/.env.example @@ -9,7 +9,7 @@ NATS_TLS_ENABLED=false # Enable TLS for NATS connection (default: false) ### HadesAPI Options ### API_PORT=8080 # Port for the API to listen on (default: 8080) -AUTH_KEY="" # Key used for Basic Auth with the API (default: "" for no auth) +AUTH_KEY="" # Key used for Basic Auth with the API. REQUIRED: the API refuses to start without it unless ALLOW_UNAUTHENTICATED=true, and compose.yml refuses to start without it. Generate one with `openssl rand -hex 32`. ### Dashboard (operator web UI, served by HadesAPI) ### # The dashboard is disabled unless USERNAME, PASSWORD_HASH and SESSION_SECRET are diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index acce3544..631a308f 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -45,6 +45,9 @@ jobs: - name: Determine benchmark mode and host id: mode_resolve + env: + JENKINS_USER_SECRET: ${{ secrets.JENKINS_USER }} + JENKINS_TOKEN_SECRET: ${{ secrets.JENKINS_TOKEN }} run: | set -euo pipefail echo "Selected mode: ${{ inputs.mode }}" @@ -61,8 +64,9 @@ jobs: jenkins) echo "MODE_PATH=jenkins" >> $GITHUB_ENV echo "BENCHMARK_HOST=${{ vars.JENKINS_HOST }}" >> $GITHUB_ENV - echo "JENKINS_USER=${{ secrets.JENKINS_USER }}" >> $GITHUB_ENV - echo "JENKINS_TOKEN=${{ secrets.JENKINS_TOKEN }}" >> $GITHUB_ENV + echo "JENKINS_USER=${JENKINS_USER_SECRET}" >> $GITHUB_ENV + echo "::add-mask::${JENKINS_TOKEN_SECRET}" + echo "JENKINS_TOKEN=${JENKINS_TOKEN_SECRET}" >> $GITHUB_ENV echo "JENKINS_JOB_PATH=${{ vars.JENKINS_JOB_PATH }}" >> $GITHUB_ENV ;; *) @@ -178,7 +182,7 @@ jobs: QUERY="host=${ENCODED_HOST}&count=${COUNT}&commit_hash=${COMMIT_HASH}" fi - echo "POST ${URL}?${QUERY}" + echo "POST ${URL}?${QUERY//${JENKINS_TOKEN:-__none__}/***}" curl -sS --fail-with-body --retry 3 --retry-delay 2 --max-time 120 \ -X POST "${URL}?${QUERY}" \ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d7a53aa7..394fffd9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -25,16 +25,16 @@ concurrency: permissions: contents: read - packages: write - actions: write jobs: lint: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false - name: Setup Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 with: go-version-file: go.work cache-dependency-path: "**/*.sum" @@ -53,9 +53,11 @@ jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false - name: Setup Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 with: go-version-file: go.work cache-dependency-path: "**/*.sum" @@ -79,9 +81,11 @@ jobs: - name: HadesOperator path: HadesScheduler/HadesOperator steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false - name: Setup Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 with: go-version-file: go.work cache-dependency-path: "**/*.sum" @@ -91,7 +95,7 @@ jobs: run: cd ${{ matrix.path }} && go test ./... -json > "${{ github.workspace }}/TestResults-${{ matrix.name }}.json" - name: Upload test results if: always() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 with: name: Go-results-${{ matrix.name }} path: TestResults-${{ matrix.name }}.json @@ -99,11 +103,11 @@ jobs: ui: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false - name: Setup Node - uses: actions/setup-node@v4 + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: 22 cache: npm @@ -120,7 +124,10 @@ jobs: build-api: needs: [lint, build, test, ui] if: github.actor != 'dependabot[bot]' - uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0 + uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0 + permissions: + contents: read + packages: write with: image-name: hades-scheduler/hades/hades-api docker-file: ./HadesAPI/Dockerfile @@ -128,12 +135,14 @@ jobs: type=semver,pattern={{version}} type=semver,pattern={{major}}.{{minor}} type=raw,value=latest,enable={{is_default_branch}} - secrets: inherit build-scheduler: needs: [lint, build, test] if: github.actor != 'dependabot[bot]' - uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0 + uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0 + permissions: + contents: read + packages: write with: image-name: hades-scheduler/hades/hades-scheduler docker-file: ./HadesScheduler/Dockerfile @@ -141,12 +150,14 @@ jobs: type=semver,pattern={{version}} type=semver,pattern={{major}}.{{minor}} type=raw,value=latest,enable={{is_default_branch}} - secrets: inherit build-operator: needs: [lint, build, test] if: github.actor != 'dependabot[bot]' - uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0 + uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0 + permissions: + contents: read + packages: write with: image-name: hades-scheduler/hades/hades-operator docker-file: ./HadesScheduler/HadesOperator/Dockerfile @@ -154,12 +165,14 @@ jobs: type=semver,pattern={{version}} type=semver,pattern={{major}}.{{minor}} type=raw,value=latest,enable={{is_default_branch}} - secrets: inherit build-log-manager: needs: [lint, build, test] if: github.actor != 'dependabot[bot]' - uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0 + uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0 + permissions: + contents: read + packages: write with: image-name: hades-scheduler/hades/hades-log-manager docker-file: ./HadesLogManager/Dockerfile @@ -167,4 +180,3 @@ jobs: type=semver,pattern={{version}} type=semver,pattern={{major}}.{{minor}} type=raw,value=latest,enable={{is_default_branch}} - secrets: inherit diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml index c37a5233..5fd82c74 100644 --- a/.github/workflows/deploy-docs.yml +++ b/.github/workflows/deploy-docs.yml @@ -18,12 +18,12 @@ jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 with: fetch-depth: 0 persist-credentials: false - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: "22" cache: yarn diff --git a/.github/workflows/deploy-k8s-test.yml b/.github/workflows/deploy-k8s-test.yml index 4dd54759..df50d6cb 100644 --- a/.github/workflows/deploy-k8s-test.yml +++ b/.github/workflows/deploy-k8s-test.yml @@ -16,6 +16,8 @@ name: Deploy to Kubernetes (test) on: workflow_dispatch: inputs: + # For pr-N the chart and CRDs of that PR (fork PRs included) are applied with the + # test cluster's kubeconfig. Review the PR's helm/ and CRD changes before dispatching. version: description: "Image tag to deploy (e.g. latest, 1.0.0, pr-123, or a branch/sha tag)." required: true diff --git a/.github/workflows/deploy-k8s.yml b/.github/workflows/deploy-k8s.yml index b5da0382..94201aa6 100644 --- a/.github/workflows/deploy-k8s.yml +++ b/.github/workflows/deploy-k8s.yml @@ -56,15 +56,16 @@ jobs: environment: ${{ inputs.environment }} steps: - name: Checkout chart source - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: + persist-credentials: false ref: ${{ inputs.chart-ref }} - name: Install Helm - uses: azure/setup-helm@v4 + uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4 - name: Install kubectl - uses: azure/setup-kubectl@v4 + uses: azure/setup-kubectl@776406bce94f63e41d621b960d78ee25c8b76ede # v4 - name: Configure kubeconfig env: diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 05d84e04..419c9a90 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -24,16 +24,16 @@ jobs: e2e: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false - name: Setup Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 with: go-version-file: go.work cache-dependency-path: "**/*.sum" - name: Setup Node - uses: actions/setup-node@v4 + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: 22 cache: npm @@ -48,7 +48,7 @@ jobs: run: cd HadesAPI/web && npm run test:e2e - name: Upload report if: ${{ !cancelled() }} - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 with: name: playwright-report path: HadesAPI/web/playwright-report/ diff --git a/.github/workflows/release-chart.yml b/.github/workflows/release-chart.yml index c2ef69c4..6eca6213 100644 --- a/.github/workflows/release-chart.yml +++ b/.github/workflows/release-chart.yml @@ -31,10 +31,12 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + with: + persist-credentials: false - name: Install Helm - uses: azure/setup-helm@v4 + uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4 - name: Lint and template run: | @@ -52,10 +54,12 @@ jobs: packages: write # push the chart to GHCR steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + with: + persist-credentials: false - name: Install Helm - uses: azure/setup-helm@v4 + uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4 - name: Log in to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | helm registry login ghcr.io -u "${{ github.actor }}" --password-stdin diff --git a/.github/workflows/verify-crd.yml b/.github/workflows/verify-crd.yml index dcdfb61c..d2e956c6 100644 --- a/.github/workflows/verify-crd.yml +++ b/.github/workflows/verify-crd.yml @@ -26,12 +26,13 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: + persist-credentials: false fetch-depth: 0 - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 with: go-version-file: go.work cache: true diff --git a/.github/workflows/verify-openapi.yml b/.github/workflows/verify-openapi.yml index 302fdc28..5921cb5f 100644 --- a/.github/workflows/verify-openapi.yml +++ b/.github/workflows/verify-openapi.yml @@ -34,13 +34,13 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: fetch-depth: 0 persist-credentials: false - name: Set up Go - uses: actions/setup-go@v7 + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 with: go-version-file: go.work cache: true diff --git a/AGENTS.md b/AGENTS.md index 125b82ed..4e273591 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ Go workspace (`go.work`, Go 1.27) with five modules: | Module | Binary / role | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `HadesAPI/` | Gin HTTP server. `POST /build` validates a payload, assigns a UUID, publishes to NATS by priority (and now also publishes `hades.jobstatus.Queued`). `GET /ping` health check. Optional Basic Auth via `AUTH_KEY`. Also hosts the optional **dashboard** (`HadesAPI/dashboard/` + embedded SPA in `HadesAPI/web/`). | +| `HadesAPI/` | Gin HTTP server. `POST /build` validates a payload, assigns a UUID, publishes to NATS by priority (and now also publishes `hades.jobstatus.Queued`). `GET /ping` health check. Basic Auth via `AUTH_KEY` is required (exits at start when empty unless `ALLOW_UNAUTHENTICATED=true`, dev only); `/build` body capped at 1 MiB, duplicate step ids rejected; Swagger (`DEBUG=true`) sits behind the same auth. Also hosts the optional **dashboard** (`HadesAPI/dashboard/` + embedded SPA in `HadesAPI/web/`). | | `HadesScheduler/` | NATS consumer. Reads `HADES_EXECUTOR` and dispatches to either the `docker/` or `k8s/` package. | | `HadesScheduler/docker/` | Runs each step as a Docker container; shares state between steps via a per-job named volume `shared-`. | | `HadesScheduler/k8s/` | Creates a `BuildJob` CR via the dynamic client for the operator to reconcile. | @@ -26,10 +26,11 @@ Go workspace (`go.work`, Go 1.27) with five modules: - **Status subjects:** `hades.jobstatus.{Queued,Running,Succeeded,Failed,Stopped}` (payload = job ID string, optional `X-Hades-Reason` header). `HadesAPI` publishes `Queued` on enqueue; the scheduler/operator publish `Running`/`Succeeded`/`Failed`. Nothing publishes `Stopped` today. `HadesLogManager` and the dashboard subscribe to the whole `hades.jobstatus.*` lifecycle over **core NATS**. - **Status stream:** `HADES_JOB_STATUS` is a JetStream stream over `hades.jobstatus.*` that durably captures those same core-NATS publishes. `HadesLogManager` consumes it through the durable `HADES_STATUS_WEBHOOK` consumer to deliver the **job-status webhook** (`status_callback_url`). Publishers are unchanged and core subscribers still receive every event. - **Log subjects:** see `shared/buildlogs/buildlog_stream.go`. +- **Callbacks:** `callback_url` / `status_callback_url` targets on loopback, private, CGNAT, link-local, unspecified and multicast addresses are refused at connect time (SSRF guard in `HadesLogManager/http_client.go`); `CALLBACK_ALLOWED_HOSTS` exempts internal receivers. No redirects on the log callback, proxy env ignored, a blocked status webhook is not retried. - **Job leases:** the scheduler acks a job only after the executor returns, and the Docker executor blocks for the whole job. `shared/nats/consumer.go` therefore heartbeats `msg.InProgress()` on a ticker while a job runs so the `NATS_ACK_WAIT` (default 1m) timer keeps resetting - job duration must never determine AckWait. `NATS_MAX_DELIVER` (default 3) caps redeliveries; the last delivery is not executed but publishes a terminal `Failed` status and terminates the message. - **Job DTO:** `shared/payload/payload.go` (`QueuePayload` with ordered `Step`s). Job-level: `timeout_seconds` (whole-job timeout). Each step has `image`, optional `script`, `metadata` env vars, `cpu_limit` (whole cores), `memory_limit`, and the Docker-only limits `network`, `memory_swap`, `pids_limit`. - **Two outbound pushes, do not conflate them:** `callback_url` forwards the aggregated **log array** and only after `stopWatchingJobLogs` drains the JetStream log consumer; `status_callback_url` receives the **job-status webhook** (`{event, job_id, name, status, reason, queued_at, started_at, finished_at, duration_ms, attempt}`) on the terminal status itself, with at-least-once retry. Neither may be repurposed as the other. - - **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only**; the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field). + - **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only** (`network` must be in `DOCKER_ALLOWED_NETWORKS`, default `none,bridge,default`, else the job fails; `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` / `HADES_MAX_JOB_TIMEOUT_SECONDS` also Docker only); the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field). - **CRD:** `BuildJob` in `HadesScheduler/HadesOperator/api/v1/buildjob_types.go`. **Important:** if you change `BuildJobSpec`, run `make -C HadesScheduler/HadesOperator manifests generate` and commit `helm/hades/crds/build.hades.tum.de_buildjobs.yaml` and `zz_generated.deepcopy.go` - the `verify-crd.yml` GitHub workflow will fail otherwise. The `BuildJobSpec` is intentionally duplicated from `shared/payload`; keep them in sync manually. ## Build / test @@ -81,6 +82,9 @@ The top-level `Makefile` wraps the common workflows (`make help` lists every tar - Don't add fields to `BuildJobSpec` without regenerating the CRD (CI will block the PR). - Don't introduce package-level mutable globals for dependencies - the codebase has been moving away from them (see `setupRouter` taking `JobPublisher` as a parameter). - `HadesLogManager` is deployed (Helm + image) but ClusterIP-internal and single-replica (in-memory aggregation); don't put it on the ingress or scale it past one replica. The status-webhook dispatcher living there keeps no durable state in memory (its retry schedule is in JetStream), but it still assumes one replica. The same single-replica constraint applies to the API's in-memory dashboard read-model. (It just isn't in the `compose.yml` stack.) +- NATS itself still has no authentication; it must not be published beyond loopback (compose binds it to `127.0.0.1`, the deploy stack does not publish it). Never list the NATS network in `DOCKER_ALLOWED_NETWORKS`. +- Don't re-open callbacks to internal addresses or add a redirect-following/proxy-aware callback client; use `CALLBACK_ALLOWED_HOSTS`. +- Job pods must keep no service account token, `RuntimeDefault` seccomp and `allowPrivilegeEscalation: false`; the operator role gets no secrets/configmaps access. - The dashboard treats all job/step `Metadata` and step `script` bodies as potentially secret: redact via `shared/redact` before returning any payload; never add an endpoint that returns raw metadata or an unscrubbed script. Job logs are proxied verbatim (a known, documented gap). diff --git a/HadesAPI/Readme.md b/HadesAPI/Readme.md index 8c3d806c..e5890672 100644 --- a/HadesAPI/Readme.md +++ b/HadesAPI/Readme.md @@ -15,14 +15,14 @@ It does **not** run jobs itself and holds no state. | Method | Path | Auth | Description | | ------ | ---- | ---- | ----------- | | `GET` | `/ping` | none | Liveness probe (`{"status":"ok","timestamp":...}`). | -| `POST` | `/build` | Basic (when `AUTH_KEY` set) | Validate and enqueue a job. Returns the assigned `job_id`. | +| `POST` | `/build` | Basic | Validate and enqueue a job. Returns the assigned `job_id`. | The full request/response contract and the job payload schema are in [docs/api.md](../docs/api.md) and the [published API reference](https://hades-scheduler.github.io/hades/docs/reference/api). ### Example ```bash -# When AUTH_KEY is set, add: -u "hades:$AUTH_KEY" +# Add: -u "hades:$AUTH_KEY" curl -X POST -H "Content-Type: application/json" \ -d '{"name":"Example","steps":[{"id":1,"name":"hi","image":"alpine:latest","script":"echo hello"}]}' \ http://localhost:8080/build @@ -30,7 +30,7 @@ curl -X POST -H "Content-Type: application/json" \ ## Authentication -If `AUTH_KEY` is set, `/build` requires HTTP Basic Auth with username `hades` and the key as the password. If it is empty, the endpoint is open and a warning is logged at startup. +`/build` requires HTTP Basic Auth with username `hades` and `AUTH_KEY` as the password. `AUTH_KEY` is required: with an empty key the API exits at start unless `ALLOW_UNAUTHENTICATED=true` (local development only; `make run` sets it). Bodies are capped at 1 MiB (413 above), duplicate step ids get 400, and the `DEBUG=true` Swagger UI sits behind the same Basic Auth. ## Priority @@ -38,7 +38,7 @@ If `AUTH_KEY` is set, `/build` requires HTTP Basic Auth with username `hades` an ## Configuration -Common variables: `API_PORT` (default `8080`), `AUTH_KEY`, `NATS_URL`. See [docs/configuration.md](../docs/configuration.md) for the complete list. +Common variables: `API_PORT` (default `8080`), `AUTH_KEY`, `ALLOW_UNAUTHENTICATED`, `NATS_URL`. See [docs/configuration.md](../docs/configuration.md) for the complete list. ## Run locally diff --git a/HadesAPI/dashboard/auth.go b/HadesAPI/dashboard/auth.go index f3922e05..12b55abb 100644 --- a/HadesAPI/dashboard/auth.go +++ b/HadesAPI/dashboard/auth.go @@ -27,6 +27,8 @@ const ( // maxFailureEntries bounds the lockout map so a client rotating its source // key cannot grow it without limit (memory DoS). maxFailureEntries = 4096 + // maxLoginBodyBytes caps the login request body. + maxLoginBodyBytes = 4 << 10 ) // authenticator verifies credentials and mints/validates signed session cookies. @@ -81,15 +83,11 @@ func (a *authenticator) verifyCredentials(username, password string) bool { return userOK && passOK } -// locked reports whether key (client IP) is currently locked out. -func (a *authenticator) locked(key string) bool { - a.mu.Lock() - defer a.mu.Unlock() - fs := a.failures[key] - return fs != nil && fs.count >= maxFailedAttempts && time.Now().Before(fs.until) -} - -func (a *authenticator) recordFailure(key string) { +// reserve counts a login attempt for key before its password is verified, so +// parallel requests cannot evaluate more than maxFailedAttempts guesses per +// window. It returns false when key is locked out. A successful login clears +// the count via recordSuccess. +func (a *authenticator) reserve(key string) bool { now := time.Now() a.mu.Lock() defer a.mu.Unlock() @@ -103,18 +101,21 @@ func (a *authenticator) recordFailure(key string) { } fs := a.failures[key] - if fs == nil || now.After(fs.until) { + if fs == nil { if len(a.failures) >= maxFailureEntries { // Hard cap reached (all entries still within the window): refuse to // track new keys rather than grow unbounded. Existing locked-out // keys keep their state. - return + return true } fs = &failureState{} a.failures[key] = fs + } else if fs.count >= maxFailedAttempts { + return false } fs.count++ fs.until = now.Add(lockoutWindow) + return true } func (a *authenticator) recordSuccess(key string) { @@ -176,19 +177,19 @@ type loginRequest struct { func (s *Server) handleLogin(c *gin.Context) { key := c.ClientIP() - if s.auth.locked(key) { - c.JSON(http.StatusTooManyRequests, gin.H{"error": "too many failed attempts, try again later"}) - return - } - + c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxLoginBodyBytes) var req loginRequest if err := c.ShouldBindJSON(&req); err != nil { c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request"}) return } + if !s.auth.reserve(key) { + c.JSON(http.StatusTooManyRequests, gin.H{"error": "too many failed attempts, try again later"}) + return + } + if !s.auth.verifyCredentials(req.Username, req.Password) { - s.auth.recordFailure(key) // Deliberately do not log the submitted username (avoid recording // credential-guessing input / PII in logs). slog.Warn("Dashboard login failed", "client_ip", key) diff --git a/HadesAPI/dashboard/dashboard_test.go b/HadesAPI/dashboard/dashboard_test.go index a474e788..effe47c8 100644 --- a/HadesAPI/dashboard/dashboard_test.go +++ b/HadesAPI/dashboard/dashboard_test.go @@ -7,6 +7,7 @@ import ( "net/http/httptest" "strconv" "strings" + "sync" "testing" "time" @@ -212,6 +213,65 @@ func TestLoginLockout(t *testing.T) { } } +func TestLoginLockoutParallelBurst(t *testing.T) { + s := testServer(t, true) + r := newRouter(s) + const burst = 40 + var wg sync.WaitGroup + var mu sync.Mutex + codes := map[int]int{} + for i := 0; i < burst; i++ { + wg.Add(1) + go func() { + defer wg.Done() + w := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodPost, "/api/login", strings.NewReader(`{"username":"admin","password":"wrong"}`)) + req.Header.Set("Content-Type", "application/json") + r.ServeHTTP(w, req) + mu.Lock() + codes[w.Code]++ + mu.Unlock() + }() + } + wg.Wait() + if codes[http.StatusUnauthorized] != maxFailedAttempts || codes[http.StatusTooManyRequests] != burst-maxFailedAttempts { + t.Fatalf("want %d verified and %d locked, got %v", maxFailedAttempts, burst-maxFailedAttempts, codes) + } +} + +func TestLoginSuccessClearsReservedAttempts(t *testing.T) { + s := testServer(t, true) + r := newRouter(s) + wrong := func() int { + w := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodPost, "/api/login", strings.NewReader(`{"username":"admin","password":"wrong"}`)) + req.Header.Set("Content-Type", "application/json") + r.ServeHTTP(w, req) + return w.Code + } + for i := 0; i < maxFailedAttempts-1; i++ { + wrong() + } + login(t, r) + for i := 0; i < maxFailedAttempts; i++ { + if code := wrong(); code != http.StatusUnauthorized { + t.Fatalf("attempt %d: want 401 after a successful login reset, got %d", i, code) + } + } +} + +func TestLoginRejectsOversizedBody(t *testing.T) { + r := newRouter(testServer(t, true)) + body := `{"username":"admin","password":"` + strings.Repeat("a", 1<<20) + `"}` + w := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodPost, "/api/login", strings.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + r.ServeHTTP(w, req) + if w.Code != http.StatusBadRequest { + t.Fatalf("expected 400 for oversized body, got %d", w.Code) + } +} + func TestListJobsReflectsTracker(t *testing.T) { s := testServer(t, true) r := newRouter(s) @@ -371,7 +431,7 @@ func TestLockoutMapBounded(t *testing.T) { s := testServer(t, true) // Far more distinct keys than the cap; the map must not grow unbounded. for i := 0; i < maxFailureEntries+500; i++ { - s.auth.recordFailure("key-" + strconv.Itoa(i)) + s.auth.reserve("key-" + strconv.Itoa(i)) } s.auth.mu.Lock() n := len(s.auth.failures) diff --git a/HadesAPI/main.go b/HadesAPI/main.go index 4cc39c71..9b24e084 100644 --- a/HadesAPI/main.go +++ b/HadesAPI/main.go @@ -20,13 +20,16 @@ import ( "github.com/prometheus/client_golang/prometheus" ) -// HadesAPIConfig holds the API server configuration. An empty AuthKey disables -// Basic Auth on the /build endpoint. +// HadesAPIConfig holds the API server configuration. An empty AuthKey is +// refused at start unless AllowUnauthenticated is set, which leaves /build open. type HadesAPIConfig struct { APIPort uint `env:"API_PORT,notEmpty" envDefault:"8080"` MetricsPort uint `env:"METRICS_PORT,notEmpty" envDefault:"8082"` NatsConfig hadesnats.ConnectionConfig AuthKey string `env:"AUTH_KEY"` + // AllowUnauthenticated is the explicit opt-out for local development: it + // lets the API start without an AuthKey and serve /build to everyone. + AllowUnauthenticated bool `env:"ALLOW_UNAUTHENTICATED" envDefault:"false"` // TrustedProxies are the CIDRs/IPs of front proxies (e.g. the ingress) that // may set X-Forwarded-For. Empty (default) trusts none, so ClientIP() uses // the direct RemoteAddr and cannot be spoofed via a forged XFF header - which @@ -37,6 +40,13 @@ type HadesAPIConfig struct { var cfg HadesAPIConfig +func (c HadesAPIConfig) validate() error { + if c.AuthKey == "" && !c.AllowUnauthenticated { + return errors.New("AUTH_KEY is empty: set AUTH_KEY, or set ALLOW_UNAUTHENTICATED=true to serve /build without authentication (local development only)") + } + return nil +} + // @title HadesAPI // @version 1.0 // @description Job-submission API for Hades, a scalable scheduler for containerized jobs. Submit a multi-step job and it is validated, assigned a UUID, and queued on NATS by priority for the scheduler to execute. @@ -50,6 +60,11 @@ func main() { os.Exit(1) } + if err := cfg.validate(); err != nil { + slog.Error("Invalid configuration", "error", err) + os.Exit(1) + } + slog.Info("HadesAPI configuration", "api_port", cfg.APIPort, "metrics_port", cfg.MetricsPort, @@ -110,6 +125,9 @@ func main() { os.Exit(1) } slog.Info("Dashboard enabled") + if len(cfg.TrustedProxies) == 0 { + slog.Warn("DASHBOARD_TRUSTED_PROXIES is empty: behind a reverse proxy all clients share the proxy's address, so the dashboard login lockout is shared by everyone and one client can lock the others out. Set it to the proxy's address range") + } } else { slog.Warn("Dashboard disabled (set DASHBOARD_USERNAME, DASHBOARD_PASSWORD_HASH and DASHBOARD_SESSION_SECRET to enable)") } diff --git a/HadesAPI/main_test.go b/HadesAPI/main_test.go new file mode 100644 index 00000000..2357abf9 --- /dev/null +++ b/HadesAPI/main_test.go @@ -0,0 +1,13 @@ +package main + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestConfigValidateAuth(t *testing.T) { + assert.Error(t, HadesAPIConfig{}.validate()) + assert.NoError(t, HadesAPIConfig{AuthKey: "secret"}.validate()) + assert.NoError(t, HadesAPIConfig{AllowUnauthenticated: true}.validate()) +} diff --git a/HadesAPI/router.go b/HadesAPI/router.go index ba030794..d3b29b5c 100644 --- a/HadesAPI/router.go +++ b/HadesAPI/router.go @@ -58,6 +58,9 @@ func securityHeaders() gin.HandlerFunc { } } +// maxBuildBodyBytes caps the /build request body (1 MiB). +const maxBuildBodyBytes = 1 << 20 + func setupRouter(authKey string, producer hades.JobPublisher, statusPublisher buildstatus.StatusPublisher, dash *dashboard.Server) *gin.Engine { r := gin.New() r.Use(gin.ErrorLogger()) @@ -78,12 +81,6 @@ func setupRouter(authKey string, producer hades.JobPublisher, statusPublisher bu r.GET("/ping", ping) - // Swagger UI (and its doc.json) are only exposed when DEBUG is enabled, so - // the API contract is never served in production. - if os.Getenv("DEBUG") == "true" { - r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler)) - } - api := r.Group("/") if authKey == "" { slog.Warn("No auth key set") @@ -92,6 +89,12 @@ func setupRouter(authKey string, producer hades.JobPublisher, statusPublisher bu api.Use(gin.BasicAuth(gin.Accounts{"hades": authKey})) } + // Swagger UI (and its doc.json) are only exposed when DEBUG is enabled and + // sit inside the auth group, so they need credentials when a key is set. + if os.Getenv("DEBUG") == "true" { + api.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler)) + } + api.POST("/build", func(c *gin.Context) { addBuildToQueue(c, producer, statusPublisher, dash) }) @@ -161,7 +164,14 @@ func bindErrorMessage(err error) string { func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublisher buildstatus.StatusPublisher, dash *dashboard.Server) { var p payload.RESTPayload p.Priority = 3 + c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxBuildBodyBytes) if err := c.ShouldBind(&p); err != nil { + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusRequestEntityTooLarge, fmt.Sprintf("Request body must not exceed %d bytes", maxBuildBodyBytes)) + return + } msg := bindErrorMessage(err) slog.Error("Failed to bind request payload", "error", err) buildRequestsTotal.WithLabelValues("rejected").Inc() @@ -180,7 +190,15 @@ func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublishe return } + seenStepIDs := make(map[int]bool, len(p.QueuePayload.Steps)) for _, step := range p.QueuePayload.Steps { + if seenStepIDs[step.ID] { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, fmt.Sprintf("Duplicate step id %d: step ids must be unique", step.ID)) + return + } + seenStepIDs[step.ID] = true + var memLimitBytes int64 if step.MemoryLimit != "" { parsed, err := utils.ParseMemoryLimit(step.MemoryLimit) diff --git a/HadesAPI/router_test.go b/HadesAPI/router_test.go index 44b70a6f..d744fa80 100644 --- a/HadesAPI/router_test.go +++ b/HadesAPI/router_test.go @@ -358,6 +358,54 @@ func (suite *APISuite) TestInvalidStatusCallbackURL() { assert.Contains(suite.T(), w.Body.String(), "Invalid status_callback_url") } +func (suite *APISuite) TestDuplicateStepIDs() { + w := httptest.NewRecorder() + restPayload := payload.RESTPayload{ + Priority: 1, + QueuePayload: payload.QueuePayload{ + Name: "example", + Steps: []payload.Step{ + {ID: 1, Name: "a", Image: "image1", Script: "s"}, + {ID: 1, Name: "b", Image: "image1", Script: "s"}, + }, + }, + } + jsonValue, _ := json.Marshal(restPayload) + req, _ := http.NewRequest("POST", "/build", bytes.NewBuffer(jsonValue)) + req.Header.Set("Content-Type", "application/json") + suite.router.ServeHTTP(w, req) + + assert.Equal(suite.T(), 400, w.Code) + assert.Contains(suite.T(), w.Body.String(), "Duplicate step id 1") +} + +func (suite *APISuite) TestBuildBodyTooLarge() { + w := httptest.NewRecorder() + big := `{"name":"example","steps":[{"id":1,"name":"a","image":"i","script":"` + strings.Repeat("x", maxBuildBodyBytes) + `"}]}` + req, _ := http.NewRequest("POST", "/build", strings.NewReader(big)) + req.Header.Set("Content-Type", "application/json") + suite.router.ServeHTTP(w, req) + + assert.Equal(suite.T(), 413, w.Code) +} + +func (suite *APISuite) TestSwaggerBehindAuth() { + suite.T().Setenv("DEBUG", "true") + router := setupRouter("secret", suite.hadesProducer, suite.statusPublisher, nil) + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/swagger/doc.json", nil) + router.ServeHTTP(w, req) + assert.Equal(suite.T(), 401, w.Code) + + w = httptest.NewRecorder() + req, _ = http.NewRequest("GET", "/swagger/index.html", nil) + req.RequestURI = req.URL.Path + req.SetBasicAuth("hades", "secret") + router.ServeHTTP(w, req) + assert.Equal(suite.T(), 200, w.Code) +} + func (suite *APISuite) TestInvalidJSON() { w := httptest.NewRecorder() restPayload := struct { diff --git a/HadesAPI/web/e2e/serve.sh b/HadesAPI/web/e2e/serve.sh index 96380862..63d9a787 100755 --- a/HadesAPI/web/e2e/serve.sh +++ b/HadesAPI/web/e2e/serve.sh @@ -50,6 +50,7 @@ API_BIN="$(mktemp -t hades-e2e-api.XXXXXX)" echo "[e2e] starting HadesAPI on :$API_PORT" export NATS_URL="nats://localhost:${NATS_PORT}" export API_PORT +export ALLOW_UNAUTHENTICATED=true export DASHBOARD_USERNAME="admin" # bcrypt hash of "test-password" (see e2e/fixtures.ts). export DASHBOARD_PASSWORD_HASH='$2y$10$oKaVlKDJczMUQJyWZxMIGenHvvP.4mIOTPSOHLcIVewfZi6JioAii' diff --git a/HadesLogManager/Readme.md b/HadesLogManager/Readme.md index d6c0e328..fb7ed77c 100644 --- a/HadesLogManager/Readme.md +++ b/HadesLogManager/Readme.md @@ -229,6 +229,7 @@ live feed - continue to receive every event unchanged. | `LOG_RETENTION` | `1h` | How long completed job logs are kept in memory.| | `MAX_JOB_LOGS` | `1000` | Max log entries retained per container of a job (oldest drop first; container slots are never removed). | | `LOG_WATCHER_MAX_LIFETIME` | `6h` | Longest a job's live log watcher runs without a terminal status, for jobs without `timeout_seconds` (jobs with one use it plus 10m). Also used when the job's `HADES_JOBS` entry can't be read. Past it, live logs stop updating; the full log is replayed when the job finishes. | +| `CALLBACK_ALLOWED_HOSTS` | | Comma-separated host names, IPs or CIDRs exempt from the callback address block (see below). | | `DEBUG` | `false` | Enable verbose (debug-level) logging. | The forwarding destination is no longer a global env var. Each job sets its own @@ -237,6 +238,18 @@ The forwarding destination is no longer a global env var. Each job sets its own if it was not prefetched. Jobs without a `callback_url` are not forwarded. +### Callback address restrictions + +Callbacks (`callback_url`, `status_callback_url`) to loopback, private (RFC 1918, +`fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), +unspecified and multicast addresses are refused at connect time. Set +`CALLBACK_ALLOWED_HOSTS` (Helm: `hadesLogManager.callbackAllowedHosts`) to exempt +internal receivers such as Artemis in the same cluster or compose network. A host +name entry must equal the host in the callback URL and trusts whatever it resolves +to. The log callback does not follow redirects, and HTTP proxy environment +variables are ignored. A blocked status webhook is logged as an error and not +retried. + ### Job-status webhook | Env var | Default | Description | diff --git a/HadesLogManager/http_client.go b/HadesLogManager/http_client.go index 35fdfef9..4d9b3a57 100644 --- a/HadesLogManager/http_client.go +++ b/HadesLogManager/http_client.go @@ -1,11 +1,92 @@ package main import ( + "context" + "errors" + "fmt" "io" + "net" "net/http" + "net/netip" + "strings" + "syscall" "time" ) +// CallbackConfig configures which callback targets the log manager may reach. +// Loopback, private, link-local, unspecified and multicast addresses are +// refused unless listed here. +type CallbackConfig struct { + // AllowedHosts is a comma-separated list of host names, IPs or CIDRs + // (e.g. "artemis,10.0.0.0/8") that callbacks may reach despite the block. + AllowedHosts []string `env:"CALLBACK_ALLOWED_HOSTS" envSeparator:","` +} + +// errBlockedTarget marks a callback target the dialer refused. Retrying cannot +// fix it, so callers treat it as permanent. +var errBlockedTarget = errors.New("callback target is not allowed") + +// cgnat is the shared address space (RFC 6598). It is not public and some +// clouds serve instance metadata from it. +var cgnat = netip.MustParsePrefix("100.64.0.0/10") + +// callbackAllowlist holds the operator's exemptions from the target block. +type callbackAllowlist struct { + hosts map[string]bool + prefixes []netip.Prefix +} + +func parseCallbackAllowlist(entries []string) (*callbackAllowlist, error) { + a := &callbackAllowlist{hosts: map[string]bool{}} + for _, e := range entries { + e = strings.TrimSpace(e) + if e == "" { + continue + } + if p, err := netip.ParsePrefix(e); err == nil { + a.prefixes = append(a.prefixes, p.Masked()) + } else if ip, err := netip.ParseAddr(e); err == nil { + a.prefixes = append(a.prefixes, netip.PrefixFrom(ip.Unmap(), ip.Unmap().BitLen())) + } else if strings.Contains(e, "/") { + return nil, fmt.Errorf("invalid CALLBACK_ALLOWED_HOSTS entry %q", e) + } else { + a.hosts[strings.ToLower(e)] = true + } + } + return a, nil +} + +func (a *callbackAllowlist) allowsIP(ip netip.Addr) bool { + for _, p := range a.prefixes { + if p.Contains(ip) { + return true + } + } + return false +} + +// control runs after DNS resolution, once per connection attempt, so DNS +// rebinding and redirects cannot reach a blocked address. +func (a *callbackAllowlist) control(_, address string, _ syscall.RawConn) error { + ap, err := netip.ParseAddrPort(address) + if err != nil { + return fmt.Errorf("%w: %v", errBlockedTarget, err) + } + ip := ap.Addr().Unmap() + if a.allowsIP(ip) { + return nil + } + if ip.IsLoopback() || ip.IsPrivate() || cgnat.Contains(ip) || ip.IsLinkLocalUnicast() || ip.IsUnspecified() || ip.IsMulticast() { + return fmt.Errorf("%w: %s", errBlockedTarget, ip) + } + return nil +} + +// noRedirect returns the 3xx to the caller instead of following it. +func noRedirect(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse +} + // minIdleConnsPerHost is the keep-alive pool floor per callback host. Both // callbacks typically go to one adapter, and Go's default of two idle // connections per host would make most concurrent deliveries dial afresh. @@ -18,9 +99,20 @@ const maxDrainedResponseBytes = 64 << 10 // newCallbackTransport returns the transport shared by the log callback and the // status webhook, so both reuse one connection pool. maxConcurrent is the // status webhook concurrency; the per-host idle pool is sized to cover it. -func newCallbackTransport(maxConcurrent int) *http.Transport { +// Connections to blocked targets fail unless allow exempts them. Proxies are +// not used, since a proxy would do the dialing and bypass the check. +func newCallbackTransport(maxConcurrent int, allow *callbackAllowlist) *http.Transport { perHost := max(maxConcurrent, minIdleConnsPerHost) t := http.DefaultTransport.(*http.Transport).Clone() + t.Proxy = nil + guarded := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second, Control: allow.control} + open := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second} + t.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) { + if host, _, err := net.SplitHostPort(addr); err == nil && allow.hosts[strings.ToLower(host)] { + return open.DialContext(ctx, network, addr) + } + return guarded.DialContext(ctx, network, addr) + } t.MaxIdleConnsPerHost = perHost t.MaxIdleConns = max(t.MaxIdleConns, 4*perHost) t.IdleConnTimeout = 90 * time.Second diff --git a/HadesLogManager/http_client_test.go b/HadesLogManager/http_client_test.go new file mode 100644 index 00000000..2eccf09a --- /dev/null +++ b/HadesLogManager/http_client_test.go @@ -0,0 +1,82 @@ +package main + +import ( + "context" + "net/http" + "net/http/httptest" + "net/netip" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// loopbackTransport allows 127.0.0.0/8 so tests can post to httptest servers. +func loopbackTransport(t *testing.T) *http.Transport { + t.Helper() + allow, err := parseCallbackAllowlist([]string{"127.0.0.0/8"}) + require.NoError(t, err) + return newCallbackTransport(0, allow) +} + +func loopbackClient(t *testing.T) *http.Client { + t.Helper() + return &http.Client{Timeout: httpClientTimeout, Transport: loopbackTransport(t), CheckRedirect: noRedirect} +} + +func TestCallbackControlBlocksInternalAddresses(t *testing.T) { + allow, err := parseCallbackAllowlist(nil) + require.NoError(t, err) + + for _, addr := range []string{ + "127.0.0.1:80", "[::1]:80", "10.1.2.3:80", "172.16.0.1:80", "192.168.1.1:80", + "[fc00::1]:80", "169.254.169.254:80", "[fe80::1]:80", "0.0.0.0:80", + "224.0.0.1:80", "[::ffff:127.0.0.1]:80", "100.100.100.200:80", + } { + assert.ErrorIs(t, allow.control("tcp", addr, nil), errBlockedTarget, addr) + } + for _, addr := range []string{"93.184.216.34:443", "[2606:4700::1111]:443"} { + assert.NoError(t, allow.control("tcp", addr, nil), addr) + } +} + +func TestCallbackAllowlistExemptsListedTargets(t *testing.T) { + allow, err := parseCallbackAllowlist([]string{"10.0.0.0/8", " 192.168.1.5 ", "Artemis"}) + require.NoError(t, err) + + assert.NoError(t, allow.control("tcp", "10.9.9.9:80", nil)) + assert.NoError(t, allow.control("tcp", "192.168.1.5:80", nil)) + assert.ErrorIs(t, allow.control("tcp", "192.168.1.6:80", nil), errBlockedTarget) + assert.ErrorIs(t, allow.control("tcp", "169.254.169.254:80", nil), errBlockedTarget) + assert.True(t, allow.hosts["artemis"]) + assert.True(t, allow.allowsIP(netip.MustParseAddr("10.0.0.1"))) +} + +func TestParseCallbackAllowlistRejectsBadCIDR(t *testing.T) { + _, err := parseCallbackAllowlist([]string{"10.0.0.0/99"}) + assert.Error(t, err) +} + +func TestCallbackTransportBlocksLoopbackAndAllowsViaAllowlist(t *testing.T) { + var hits int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + atomic.AddInt32(&hits, 1) + })) + defer srv.Close() + + blocked, err := parseCallbackAllowlist(nil) + require.NoError(t, err) + client := &http.Client{Timeout: 5 * time.Second, Transport: newCallbackTransport(0, blocked)} + req, _ := http.NewRequestWithContext(context.Background(), http.MethodPost, srv.URL, nil) + _, err = client.Do(req) + require.ErrorIs(t, err, errBlockedTarget) + assert.Equal(t, int32(0), atomic.LoadInt32(&hits)) + + req, _ = http.NewRequestWithContext(context.Background(), http.MethodPost, srv.URL, nil) + resp, err := loopbackClient(t).Do(req) + require.NoError(t, err) + drainAndClose(resp) + assert.Equal(t, int32(1), atomic.LoadInt32(&hits)) +} diff --git a/HadesLogManager/main.go b/HadesLogManager/main.go index e21eada4..e683a71a 100644 --- a/HadesLogManager/main.go +++ b/HadesLogManager/main.go @@ -117,7 +117,16 @@ func run(cfg HadesLogManagerConfig) error { "concurrency", webhookCfg.Concurrency, "max_pending", webhookCfg.MaxPending, ) - transport := newCallbackTransport(webhookCfg.Concurrency) + var callbackCfg CallbackConfig + if err := utils.LoadConfig(&callbackCfg); err != nil { + return fmt.Errorf("loading callback configuration: %w", err) + } + allow, err := parseCallbackAllowlist(callbackCfg.AllowedHosts) + if err != nil { + return err + } + slog.Info("Callback configuration", "allowed_hosts", callbackCfg.AllowedHosts) + transport := newCallbackTransport(webhookCfg.Concurrency, allow) // Create log aggregator var aggregatorConfig AggregatorConfig @@ -129,7 +138,7 @@ func run(cfg HadesLogManagerConfig) error { "retention", aggregatorConfig.Retention, "max_job_logs", aggregatorConfig.MaxJobLogs, ) - logClient := &http.Client{Timeout: httpClientTimeout, Transport: transport} + logClient := &http.Client{Timeout: httpClientTimeout, Transport: transport, CheckRedirect: noRedirect} logAggregator := NewLogAggregator(ctx, consumer, resolver, logClient, aggregatorConfig) var watcherCfg LogWatcherConfig diff --git a/HadesLogManager/processor.go b/HadesLogManager/processor.go index 8bf12960..37f6e9c8 100644 --- a/HadesLogManager/processor.go +++ b/HadesLogManager/processor.go @@ -85,7 +85,8 @@ func NewLogAggregator(ctx context.Context, hlc *buildlogs.HadesLogConsumer, reso } if client == nil { - client = &http.Client{Timeout: httpClientTimeout} + allow, _ := parseCallbackAllowlist(nil) + client = &http.Client{Timeout: httpClientTimeout, Transport: newCallbackTransport(0, allow), CheckRedirect: noRedirect} } la := &NATSLogAggregator{ diff --git a/HadesLogManager/processor_test.go b/HadesLogManager/processor_test.go index 0908d263..bf9b2039 100644 --- a/HadesLogManager/processor_test.go +++ b/HadesLogManager/processor_test.go @@ -35,7 +35,7 @@ func newTestAggregator(t *testing.T, resolver jobCallbackResolver) buildlogs.Log t.Helper() ctx, cancel := context.WithCancel(context.Background()) t.Cleanup(cancel) - return NewLogAggregator(ctx, nil, resolver, nil, testConfig()) + return NewLogAggregator(ctx, nil, resolver, loopbackClient(t), testConfig()) } // newTestAggregatorMax builds an aggregator with a specific per-container entry @@ -303,3 +303,37 @@ func TestCleanupStaleJobs(t *testing.T) { _, err = agg.GetJobStatus("fresh") assert.NoError(t, err) } + +func TestSendJobLogs_DoesNotFollowRedirects(t *testing.T) { + var targetHits int32 + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + atomic.AddInt32(&targetHits, 1) + })) + defer target.Close() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, target.URL, http.StatusTemporaryRedirect) + })) + defer srv.Close() + + agg := newTestAggregator(t, fakeResolver{url: srv.URL}) + agg.AddLog(buildlogs.Log{JobID: "job-1", ContainerID: "c1"}) + + require.Error(t, agg.SendJobLogs(context.Background(), "job-1")) + assert.Equal(t, int32(0), atomic.LoadInt32(&targetHits), "redirect target must not be contacted") +} + +func TestSendJobLogs_BlocksLoopbackByDefault(t *testing.T) { + var hits int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + atomic.AddInt32(&hits, 1) + })) + defer srv.Close() + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + agg := NewLogAggregator(ctx, nil, fakeResolver{url: srv.URL}, nil, testConfig()) + agg.AddLog(buildlogs.Log{JobID: "job-1", ContainerID: "c1"}) + + require.ErrorIs(t, agg.SendJobLogs(context.Background(), "job-1"), errBlockedTarget) + assert.Equal(t, int32(0), atomic.LoadInt32(&hits)) +} diff --git a/HadesLogManager/status_dispatcher.go b/HadesLogManager/status_dispatcher.go index 150fda58..03630023 100644 --- a/HadesLogManager/status_dispatcher.go +++ b/HadesLogManager/status_dispatcher.go @@ -298,6 +298,11 @@ func (d *StatusWebhookDispatcher) deliver(ctx context.Context, msg statusMsg, jo event := buildEvent(jobID, info.Name, status, reason, queuedAt, startedAt, finishedAt, attempt) if err := d.sender.Send(sendCtx, info.StatusCallbackURL, event); err != nil { + if errors.Is(err, errBlockedTarget) { + // Retrying cannot make a blocked target allowed, so settle it. + slog.Error("Status webhook target blocked, not retrying", "job_id", jobID, "error", err) + return true + } if errors.Is(err, context.Canceled) { slog.Debug("Status webhook cancelled during shutdown", "job_id", jobID) } else { diff --git a/HadesLogManager/status_dispatcher_test.go b/HadesLogManager/status_dispatcher_test.go index bb888427..89da3ce3 100644 --- a/HadesLogManager/status_dispatcher_test.go +++ b/HadesLogManager/status_dispatcher_test.go @@ -106,7 +106,7 @@ func startDispatcher(t *testing.T, ctx context.Context, cfg StatusWebhookConfig) kv, err := js.CreateOrUpdateKeyValue(ctx, hadesnats.JobsKVConfig()) require.NoError(t, err) - dispatcher := NewStatusWebhookDispatcher(js, newKVCallbackResolver(kv), nil, cfg) + dispatcher := NewStatusWebhookDispatcher(js, newKVCallbackResolver(kv), newHTTPWebhookSender(cfg.Timeout, loopbackTransport(t)), cfg) done := make(chan struct{}) go func() { diff --git a/HadesLogManager/status_webhook.go b/HadesLogManager/status_webhook.go index 51fbd22c..8716e171 100644 --- a/HadesLogManager/status_webhook.go +++ b/HadesLogManager/status_webhook.go @@ -159,8 +159,12 @@ type httpWebhookSender struct { } // newHTTPWebhookSender builds a sender bounded by timeout. transport may be nil, -// in which case http.DefaultTransport is used. +// in which case a transport that blocks internal targets is used. func newHTTPWebhookSender(timeout time.Duration, transport http.RoundTripper) *httpWebhookSender { + if transport == nil { + allow, _ := parseCallbackAllowlist(nil) + transport = newCallbackTransport(0, allow) + } return &httpWebhookSender{client: &http.Client{ Timeout: timeout, Transport: transport, diff --git a/HadesLogManager/status_webhook_test.go b/HadesLogManager/status_webhook_test.go index 9386e0bc..0ba3cb50 100644 --- a/HadesLogManager/status_webhook_test.go +++ b/HadesLogManager/status_webhook_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "errors" + "fmt" "io" "net/http" "net/http/httptest" @@ -298,6 +299,29 @@ func TestHandleInvalidStatusCallbackURLIsNotRetried(t *testing.T) { assert.Equal(t, 1, msg.acked, "a malformed URL cannot be fixed by retrying") } +func TestHandleBlockedTargetIsNotRetried(t *testing.T) { + sender := &recordingSender{err: fmt.Errorf("sending status webhook: %w", errBlockedTarget)} + d := testDispatcher(t, webhookResolver("http://127.0.0.1:9/hook"), sender) + + msg := newFakeMsg(buildstatus.StatusSucceeded, "job-8", 1) + d.handle(context.Background(), msg) + + assert.Equal(t, 1, msg.acked, "a blocked target cannot be fixed by retrying") + assert.Empty(t, msg.naks()) +} + +func TestHTTPWebhookSenderBlocksLoopbackByDefault(t *testing.T) { + var hits int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + atomic.AddInt32(&hits, 1) + })) + defer srv.Close() + + err := newHTTPWebhookSender(5*time.Second, nil).Send(context.Background(), srv.URL, JobStatusEvent{JobID: "job-1"}) + require.ErrorIs(t, err, errBlockedTarget) + assert.Equal(t, int32(0), atomic.LoadInt32(&hits)) +} + func TestHandleResolverErrorIsRetried(t *testing.T) { sender := &recordingSender{} d := testDispatcher(t, fakeInfoResolver{err: errors.New("kv unavailable")}, sender) @@ -338,7 +362,7 @@ func TestHTTPWebhookSenderPostsJSONWithHeaders(t *testing.T) { })) defer srv.Close() - sender := newHTTPWebhookSender(5*time.Second, nil) + sender := newHTTPWebhookSender(5*time.Second, loopbackTransport(t)) event := buildEvent("job-1", "Example Job", buildstatus.StatusFailed, "boom", time.Date(2026, 8, 21, 12, 0, 0, 0, time.UTC), time.Date(2026, 8, 21, 12, 0, 1, 0, time.UTC), @@ -366,7 +390,7 @@ func TestHTTPWebhookSenderTreatsNon2xxAsFailure(t *testing.T) { })) defer srv.Close() - sender := newHTTPWebhookSender(5*time.Second, nil) + sender := newHTTPWebhookSender(5*time.Second, loopbackTransport(t)) err := sender.Send(context.Background(), srv.URL, JobStatusEvent{JobID: "job-1"}) require.Error(t, err) assert.Contains(t, err.Error(), "500") @@ -385,7 +409,7 @@ func TestHTTPWebhookSenderDoesNotFollowRedirects(t *testing.T) { })) defer srv.Close() - sender := newHTTPWebhookSender(5*time.Second, nil) + sender := newHTTPWebhookSender(5*time.Second, loopbackTransport(t)) err := sender.Send(context.Background(), srv.URL, JobStatusEvent{JobID: "job-1"}) // Following the redirect would downgrade the POST to a bodyless GET and diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go index 41a39140..36cd49a0 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go @@ -349,3 +349,30 @@ func TestBuildK8sJob_HonoursFinalizerImageOverride(t *testing.T) { } t.Fatalf("no %s container in the pod spec", FinalizerContainerName) } + +// Job pods run untrusted step code, so they must not mount a service account +// token and every container must forbid privilege escalation. +func TestBuildK8sJob_PodHardening(t *testing.T) { + bj := &buildv1.BuildJob{ + ObjectMeta: metav1.ObjectMeta{Name: "job-1"}, + Spec: buildv1.BuildJobSpec{ + Steps: []buildv1.BuildStep{{ID: 1, Image: "alpine", Script: "true"}}, + }, + } + + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + spec := job.Spec.Template.Spec + + if spec.AutomountServiceAccountToken == nil || *spec.AutomountServiceAccountToken { + t.Error("automountServiceAccountToken must be false") + } + if spec.SecurityContext == nil || spec.SecurityContext.SeccompProfile == nil || + spec.SecurityContext.SeccompProfile.Type != corev1.SeccompProfileTypeRuntimeDefault { + t.Error("pod seccomp profile must be RuntimeDefault") + } + for _, c := range append(spec.InitContainers, spec.Containers...) { + if c.SecurityContext == nil || c.SecurityContext.AllowPrivilegeEscalation == nil || *c.SecurityContext.AllowPrivilegeEscalation { + t.Errorf("container %q must set allowPrivilegeEscalation=false", c.Name) + } + } +} diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go index 902f5809..9669bb2b 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go @@ -116,6 +116,11 @@ const ( LabelPriority = hades.MetadataKeyPriority ) +func noPrivilegeEscalation() *corev1.SecurityContext { + f := false + return &corev1.SecurityContext{AllowPrivilegeEscalation: &f} +} + // ManagedByValue is the value of the LabelManagedBy label applied to Jobs the // operator owns. const ManagedByValue = "hades-operator" @@ -314,7 +319,7 @@ func positiveOrDefault(d, fallback time.Duration) time.Duration { // +kubebuilder:rbac:groups=build.hades.tum.de,resources=buildjobs;buildjobs/status;buildjobs/finalizers,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;update;patch;delete -// +kubebuilder:rbac:groups="",resources=pods;events;configmaps;secrets,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups="",resources=pods;events,verbs=get;list;watch;create;update;patch;delete // Reconcile ensures the cluster state matches the desired state of a BuildJob. // It creates/owns a batch Job, updates BuildJob status and each ContainerStatus of the BuildJob, and cleans up on completion. @@ -830,9 +835,10 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su // ran in / here and in /shared under Docker, and with an image // declaring WORKDIR /app it would run in /app. That silently breaks // the promise that the same job runs unmodified on either executor. - WorkingDir: sharedMount.MountPath, - Env: envFromMeta(env), - VolumeMounts: []corev1.VolumeMount{sharedMount}, + WorkingDir: sharedMount.MountPath, + Env: envFromMeta(env), + VolumeMounts: []corev1.VolumeMount{sharedMount}, + SecurityContext: noPrivilegeEscalation(), } if hasScript { @@ -908,10 +914,19 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su ImagePullPolicy: corev1.PullIfNotPresent, Command: []string{"sh", "-c", "echo build finished"}, VolumeMounts: []corev1.VolumeMount{sharedMount}, + SecurityContext: noPrivilegeEscalation(), } gracePeriod := int64(podTerminationGracePeriodSeconds) + noToken := false podSpec := corev1.PodSpec{ + // Steps run untrusted code and never talk to the Kubernetes API. + AutomountServiceAccountToken: &noToken, + // Root and default capabilities stay allowed on purpose: arbitrary step + // images often need them (package installs). + SecurityContext: &corev1.PodSecurityContext{ + SeccompProfile: &corev1.SeccompProfile{Type: corev1.SeccompProfileTypeRuntimeDefault}, + }, InitContainers: initCtrs, Containers: []corev1.Container{dummy}, Volumes: []corev1.Volume{{ diff --git a/HadesScheduler/docker/env.go b/HadesScheduler/docker/env.go index 66160b08..a3406e0a 100644 --- a/HadesScheduler/docker/env.go +++ b/HadesScheduler/docker/env.go @@ -19,4 +19,15 @@ type EnvConfig struct { // "30s". Zero keeps "always" meaning every job. Concurrent pulls of the // same image are shared regardless of this setting. ImagePullFreshness time.Duration `env:"DOCKER_IMAGE_PULL_FRESHNESS" envDefault:"0s"` + + // AllowedNetworks lists the network modes a step may request. A step with + // no network uses the Docker default and is always allowed. Anything else + // (e.g. "host" or the name of the control-plane network) fails the job. + AllowedNetworks []string `env:"DOCKER_ALLOWED_NETWORKS" envSeparator:"," envDefault:"none,bridge,default"` + + // DefaultJobTimeoutSeconds applies to jobs with timeout_seconds 0 and + // MaxJobTimeoutSeconds caps every job (also those without a timeout). + // Zero turns the respective setting off. + DefaultJobTimeoutSeconds int64 `env:"HADES_DEFAULT_JOB_TIMEOUT_SECONDS" envDefault:"0"` + MaxJobTimeoutSeconds int64 `env:"HADES_MAX_JOB_TIMEOUT_SECONDS" envDefault:"0"` } diff --git a/HadesScheduler/docker/job.go b/HadesScheduler/docker/job.go index 0fe81e57..e027b5af 100644 --- a/HadesScheduler/docker/job.go +++ b/HadesScheduler/docker/job.go @@ -10,6 +10,7 @@ import ( "github.com/hades-scheduler/hades/shared/buildlogs" "github.com/hades-scheduler/hades/shared/payload" + "github.com/hades-scheduler/hades/shared/redact" "github.com/hades-scheduler/hades/shared/timing" "github.com/moby/moby/client" ) @@ -38,6 +39,10 @@ func (d Job) execute(ctx context.Context) error { // executor, where such a step never fails the job at all. stepErr := error(nil) + if err := d.checkNetworks(d.Steps); err != nil { + return err + } + // Pull all images in parallel up front. Pulling per step would make each // later step pay for its pull only after the previous step finished. puller := imagePuller{ @@ -116,5 +121,6 @@ func (d Job) execute(ctx context.Context) error { // API keys, and the script can embed them too, so neither belongs in Info logs. func logStep(logger *slog.Logger, msg string, step payload.Step) { logger.Info(msg, slog.Int("step_id", step.ID), slog.String("step_name", step.Name), slog.String("image", step.Image)) - logger.Debug(msg, slog.Any("step", step)) + redacted := redact.Default().Payload(payload.QueuePayload{Steps: []payload.Step{step}}) + logger.Debug(msg, slog.Any("step", redacted.Steps[0])) } diff --git a/HadesScheduler/docker/job_test.go b/HadesScheduler/docker/job_test.go index 7367d577..19e0b690 100644 --- a/HadesScheduler/docker/job_test.go +++ b/HadesScheduler/docker/job_test.go @@ -10,7 +10,7 @@ import ( ) // The step metadata becomes the container environment and routinely carries -// tokens and API keys, so it must never reach the log at Info level. +// tokens and API keys, so the raw values must reach the log at no level. func TestLogStepKeepsEnvironmentOutOfInfo(t *testing.T) { step := payload.Step{ ID: 2, @@ -21,11 +21,10 @@ func TestLogStepKeepsEnvironmentOutOfInfo(t *testing.T) { } for _, tc := range []struct { - level slog.Level - wantSecret bool + level slog.Level }{ - {slog.LevelInfo, false}, - {slog.LevelDebug, true}, + {slog.LevelInfo}, + {slog.LevelDebug}, } { var buf bytes.Buffer logger := slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: tc.level})) @@ -38,10 +37,8 @@ func TestLogStepKeepsEnvironmentOutOfInfo(t *testing.T) { t.Errorf("level %v: log %q lacks %q", tc.level, out, want) } } - for _, secret := range []string{"secret-value-123", "echo $API_KEY"} { - if got := strings.Contains(out, secret); got != tc.wantSecret { - t.Errorf("level %v: contains %q = %v, want %v\n%s", tc.level, secret, got, tc.wantSecret, out) - } + if strings.Contains(out, "secret-value-123") { + t.Errorf("level %v: log leaks the metadata value\n%s", tc.level, out) } } } diff --git a/HadesScheduler/docker/limits_test.go b/HadesScheduler/docker/limits_test.go new file mode 100644 index 00000000..950754eb --- /dev/null +++ b/HadesScheduler/docker/limits_test.go @@ -0,0 +1,71 @@ +package docker + +import ( + "testing" + + "github.com/hades-scheduler/hades/shared/payload" + "github.com/stretchr/testify/require" +) + +func TestCheckNetworks(t *testing.T) { + opts := Options{allowedNetworks: []string{"none", "bridge", "default"}} + for _, tc := range []struct { + network string + ok bool + }{ + {"", true}, + {"none", true}, + {"bridge", true}, + {"default", true}, + {"host", false}, + {"hades_default", false}, + } { + err := opts.checkNetworks([]payload.Step{{ID: 1}, {ID: 2, Network: tc.network}}) + if tc.ok { + require.NoError(t, err, tc.network) + } else { + require.ErrorContains(t, err, "step 2") + require.ErrorContains(t, err, tc.network) + } + } +} + +func TestWithAllowedNetworksTrimsAndExtends(t *testing.T) { + s := &Scheduler{} + require.NoError(t, WithAllowedNetworks([]string{" none", "hades_jobs ", ""})(s)) + require.NoError(t, s.checkNetworks([]payload.Step{{Network: "hades_jobs"}})) + require.Error(t, s.checkNetworks([]payload.Step{{Network: "bridge"}})) +} + +func TestJobExecuteRejectsDisallowedNetworkBeforeAnyWork(t *testing.T) { + // A nil client would panic if execute got as far as pulling an image. + job := Job{ + Options: Options{allowedNetworks: []string{"none"}}, + QueuePayload: payload.QueuePayload{Steps: []payload.Step{{ID: 1, Image: "x", Network: "host"}}}, + } + require.ErrorContains(t, job.execute(t.Context()), `network "host" is not allowed`) +} + +func TestEffectiveTimeout(t *testing.T) { + for _, tc := range []struct { + name string + def, max, in, want int64 + }{ + {"off keeps payload", 0, 0, 30, 30}, + {"off keeps zero", 0, 0, 0, 0}, + {"default fills zero", 60, 0, 0, 60}, + {"default does not override", 60, 0, 30, 30}, + {"max caps larger", 0, 100, 500, 100}, + {"max keeps smaller", 0, 100, 50, 50}, + {"max applies to zero", 0, 100, 0, 100}, + {"default above max is capped", 200, 100, 0, 100}, + } { + got := Options{defaultJobTimeout: tc.def, maxJobTimeout: tc.max}.effectiveTimeout(tc.in) + require.Equal(t, tc.want, got, tc.name) + } +} + +func TestWithJobTimeoutsRejectsNegative(t *testing.T) { + require.Error(t, WithJobTimeouts(-1, 0)(&Scheduler{})) + require.NoError(t, WithJobTimeouts(10, 20)(&Scheduler{})) +} diff --git a/HadesScheduler/docker/options.go b/HadesScheduler/docker/options.go index 4f6d9f5f..ae8a31fe 100644 --- a/HadesScheduler/docker/options.go +++ b/HadesScheduler/docker/options.go @@ -2,6 +2,7 @@ package docker import ( "fmt" + "strings" "time" "github.com/hades-scheduler/hades/shared/buildlogs" @@ -109,3 +110,30 @@ func WithImagePullFreshness(window time.Duration) DockerOption { return nil } } + +// WithAllowedNetworks sets the network modes a step may request. A step without +// a network is always allowed. +func WithAllowedNetworks(networks []string) DockerOption { + return func(s *Scheduler) error { + s.allowedNetworks = nil + for _, n := range networks { + if n = strings.TrimSpace(n); n != "" { + s.allowedNetworks = append(s.allowedNetworks, n) + } + } + return nil + } +} + +// WithJobTimeouts sets the default timeout for jobs without one and the maximum +// timeout for all jobs, in seconds. Zero turns a setting off. +func WithJobTimeouts(defaultSeconds, maxSeconds int64) DockerOption { + return func(s *Scheduler) error { + if defaultSeconds < 0 || maxSeconds < 0 { + return fmt.Errorf("job timeouts must not be negative") + } + s.defaultJobTimeout = defaultSeconds + s.maxJobTimeout = maxSeconds + return nil + } +} diff --git a/HadesScheduler/docker/scheduler.go b/HadesScheduler/docker/scheduler.go index 4d223fd0..0745e51e 100644 --- a/HadesScheduler/docker/scheduler.go +++ b/HadesScheduler/docker/scheduler.go @@ -10,6 +10,7 @@ import ( "context" "fmt" "log/slog" + "slices" "sync" "time" @@ -35,6 +36,9 @@ type Options struct { containerLogsOptions container.LogConfig imagePullPolicy PullPolicy imagePullFreshness time.Duration + allowedNetworks []string + defaultJobTimeout int64 + maxJobTimeout int64 } // Scheduler runs jobs on the local Docker daemon and publishes their logs and @@ -86,6 +90,7 @@ func NewDefaultScheduler() (*Scheduler, error) { // built programmatically without options must not silently become less // safe than one built before this setting existed. imagePullPolicy: PullAlways, + allowedNetworks: []string{"none", "bridge", "default"}, } scheduler := &Scheduler{ @@ -99,6 +104,29 @@ func NewDefaultScheduler() (*Scheduler, error) { return scheduler, nil } +// effectiveTimeout applies the operator's default (for 0) and maximum (cap, +// also for 0 when no default applies) to a job's timeout_seconds. +func (o Options) effectiveTimeout(seconds int64) int64 { + if seconds <= 0 { + seconds = o.defaultJobTimeout + } + if o.maxJobTimeout > 0 && (seconds <= 0 || seconds > o.maxJobTimeout) { + seconds = o.maxJobTimeout + } + return seconds +} + +// checkNetworks rejects a step that asks for a network outside the allowlist. +// An empty network means the Docker default and is always allowed. +func (o Options) checkNetworks(steps []payload.Step) error { + for _, step := range steps { + if step.Network != "" && !slices.Contains(o.allowedNetworks, step.Network) { + return fmt.Errorf("step %d: network %q is not allowed (DOCKER_ALLOWED_NETWORKS)", step.ID, step.Network) + } + } + return nil +} + // ScheduleJob runs every step of job sequentially in its own container. It // creates the per-job shared volume, publishes Running/Succeeded/Failed status // transitions around execution, and removes the volume when done (even on @@ -185,6 +213,7 @@ func (d *Scheduler) ScheduleJob(ctx context.Context, job payload.QueuePayload) e // context is cancelled, which unblocks the running step's ContainerWait and // triggers its force-cleanup so no container is left running. execCtx := ctx + job.TimeoutSeconds = d.effectiveTimeout(job.TimeoutSeconds) if job.TimeoutSeconds > 0 { // Clamp to the overflow-safe bound (the API rejects larger values, but a // payload may reach the scheduler from other producers): seconds * diff --git a/HadesScheduler/docker/scheduler_test.go b/HadesScheduler/docker/scheduler_test.go index 0bfcf22f..3a499b7d 100644 --- a/HadesScheduler/docker/scheduler_test.go +++ b/HadesScheduler/docker/scheduler_test.go @@ -92,6 +92,7 @@ func newTestScheduler(t *testing.T, publisher buildlogs.LogPublisher) *Scheduler // alpine ships /bin/sh, not bash. scriptExecutor: "/bin/sh -c", containerAutoremove: false, + allowedNetworks: []string{"none", "bridge", "default"}, }, logPublisher: publisher, statusPublisher: log.NewNoopPublisher(), diff --git a/HadesScheduler/main.go b/HadesScheduler/main.go index 911046b4..69c80ad4 100644 --- a/HadesScheduler/main.go +++ b/HadesScheduler/main.go @@ -16,6 +16,7 @@ import ( "github.com/hades-scheduler/hades/shared/metrics" hadesnats "github.com/hades-scheduler/hades/shared/nats" "github.com/hades-scheduler/hades/shared/payload" + "github.com/hades-scheduler/hades/shared/redact" "github.com/hades-scheduler/hades/shared/timing" "github.com/hades-scheduler/hades/shared/utils" "github.com/prometheus/client_golang/prometheus" @@ -106,6 +107,8 @@ func main() { docker.WithMemoryLimit(dockerCfg.MemoryLimit), docker.WithImagePullPolicy(dockerCfg.ImagePullPolicy), docker.WithImagePullFreshness(dockerCfg.ImagePullFreshness), + docker.WithAllowedNetworks(dockerCfg.AllowedNetworks), + docker.WithJobTimeouts(dockerCfg.DefaultJobTimeoutSeconds, dockerCfg.MaxJobTimeoutSeconds), docker.WithLogPublisher(publisher), docker.WithStatusPublisher(publisher), ) @@ -158,7 +161,7 @@ func main() { consumer.DequeueJob(ctx, func(p payload.QueuePayload) { slog.Info("Received job", "id", p.ID.String()) - slog.Debug("Job payload", "payload", p) + slog.Debug("Job payload", "payload", redact.Default().Payload(p)) if err := scheduler.ScheduleJob(ctx, p); err != nil { slog.Error("Failed to schedule job", "error", err, "id", p.ID.String()) diff --git a/Makefile b/Makefile index 9c539f08..6ac418e2 100644 --- a/Makefile +++ b/Makefile @@ -24,14 +24,14 @@ run: docker-run-nats docker-run-jaeger ## Run api, scheduler, and logmanager loc @# OTEL endpoint points at the Jaeger container published on the host. @trap 'kill 0' INT TERM EXIT; \ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317; \ - (cd HadesAPI && METRICS_PORT=8082 go run .) & \ + (cd HadesAPI && ALLOW_UNAUTHENTICATED=true METRICS_PORT=8082 go run .) & \ (cd HadesScheduler && METRICS_PORT=8084 go run .) & \ (cd HadesLogManager && METRICS_PORT=8086 go run .) & \ wait .PHONY: run-api run-api: ## Run HadesAPI locally via go run. - cd HadesAPI && go run . + cd HadesAPI && ALLOW_UNAUTHENTICATED=true go run . .PHONY: run-scheduler run-scheduler: ## Run HadesScheduler locally via go run. @@ -61,11 +61,11 @@ docker-run-scheduler: ## Start only the HadesScheduler service via docker compos .PHONY: docker-run-nats docker-run-nats: ## Start only the NATS service via docker compose. - $(COMPOSE) -f $(COMPOSE_FILE) up -d nats + AUTH_KEY=$${AUTH_KEY:-unused} $(COMPOSE) -f $(COMPOSE_FILE) up -d nats .PHONY: docker-run-jaeger docker-run-jaeger: ## Start only the Jaeger tracing backend via docker compose (UI on :16686). - $(COMPOSE) -f $(COMPOSE_FILE) up -d jaeger + AUTH_KEY=$${AUTH_KEY:-unused} $(COMPOSE) -f $(COMPOSE_FILE) up -d jaeger .PHONY: docker-stop docker-stop: ## Stop the local docker compose stack. diff --git a/Readme.md b/Readme.md index 4240ff0d..5fd0e6e6 100644 --- a/Readme.md +++ b/Readme.md @@ -313,7 +313,10 @@ Login uses a signed, `HttpOnly; Secure; SameSite=Strict` session cookie; all `DASHBOARD_COOKIE_INSECURE=true` - never in production. - **Set `DASHBOARD_TRUSTED_PROXIES`** to your ingress' address range when behind a reverse proxy. Otherwise `X-Forwarded-For` is ignored (the login lockout keys on - the direct, un-spoofable address). + the direct, un-spoofable address), so all clients share the proxy's address and + one lockout; a warning is logged at start. +- Login allows 5 attempts per client address per 15 minutes, counted before the + password check. The login body is capped at 4 KiB. - Responses carry `Content-Security-Policy`, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, and HSTS. @@ -325,7 +328,9 @@ variables and routinely carries credentials. The dashboard redacts it sensitive (`token`, `password`, `secret`, ...) **or** the value itself looks like a secret (credentials embedded in a URL, a PEM block, a JWT, a high-entropy token). Step **scripts** are scanned with the same heuristics and have inline secrets -masked. Keys stay visible so operators can see which variables exist; +masked: Bearer/Basic authorization headers, token-only URL credentials, +`--password ` style flags, `docker login -p `, and every masked metadata +value that appears in the script. Put secrets in `metadata`, not in scripts. Keys stay visible so operators can see which variables exist; `SECRET_REDACT_MODE=all` masks every metadata value. **Residual exposure (by design):** job **logs** are shown *verbatim* - streamed live @@ -360,13 +365,28 @@ Hades is configured through environment variables (or a `.env` file for local ru | `HADES_EXECUTOR` | Execution platform: `docker` or `k8s` | `docker` | | `CONCURRENCY` | Number of jobs to process concurrently | `1` | | `API_PORT` | Port for the Hades API | `8080` | -| `AUTH_KEY` | HTTP Basic Auth key for the API (empty = no auth) | `` | +| `AUTH_KEY` | HTTP Basic Auth key for the API (required; the API exits at start when empty unless `ALLOW_UNAUTHENTICATED=true`) | `` | +| `ALLOW_UNAUTHENTICATED` | Serve `/build` without auth when `AUTH_KEY` is empty (local development only; `make run` sets it) | `false` | | `NATS_URL` | NATS server URL | `nats://localhost:4222` | | `DEBUG` | Verbose (debug-level) logging | `false` | | `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP endpoint for tracing (empty = tracing off) | `` | See **[docs/configuration.md](./docs/configuration.md)** for the complete, per-component reference (Docker/Kubernetes executor, operator, and Log Manager options). A ready-to-copy `.env.example` lives at the repository root. +### Upgrade notes + +#### Security hardening (unreleased) + +Breaking changes: + +- `AUTH_KEY` is required. The API exits at start with an empty key unless `ALLOW_UNAUTHENTICATED=true` (local development only). In Helm, an empty `hadesApi.authKeySecret` now needs `hadesApi.allowUnauthenticated: true`. +- Callbacks (`callback_url`, `status_callback_url`) to loopback, private, CGNAT, link-local, unspecified and multicast addresses are refused. List internal receivers (for example Artemis in the same cluster or compose network) in `CALLBACK_ALLOWED_HOSTS` (Helm: `hadesLogManager.callbackAllowedHosts`). The log callback no longer follows redirects and callbacks ignore HTTP proxy variables. +- Docker executor: steps may only request the network modes in `DOCKER_ALLOWED_NETWORKS` (default `none,bridge,default`). `host` and named networks fail the job unless listed. +- `/build` bodies are capped at 1 MiB (413 above) and duplicate step ids are rejected with 400. +- `compose.yml` requires `AUTH_KEY` in `.env` and publishes API, log manager, NATS and Jaeger on `127.0.0.1` only. `docker-compose.deploy.yml` publishes only Traefik on 80/443 (Compose 2.24 or later) and no longer routes the log manager publicly. +- Kubernetes: job pods no longer mount a service account token and run with the `RuntimeDefault` seccomp profile and `allowPrivilegeEscalation: false`. The operator role has no access to secrets or configmaps. `hadesOperator.DeleteOnComplete` now defaults to `true`. Optional `networkPolicy.enabled` (default `false`) is new. +- Dashboard login allows 5 attempts per client address per 15 minutes. Behind a reverse proxy, set `DASHBOARD_TRUSTED_PROXIES`. + ### Measuring Hades overhead Hades instruments how much overhead it adds around each job, per step and per phase, splitting the wall-clock into `overhead` (Hades/Kubernetes coordination) and `runtime` (the user's container executing). Every service exposes Prometheus histograms (`hades_phase_seconds`, `hades_job_overhead_seconds`, …) on its `/metrics` port and logs a per-job `job timing summary` with `overhead_pct`. Setting `OTEL_EXPORTER_OTLP_ENDPOINT` additionally emits an OpenTelemetry trace per job - a waterfall across API → scheduler → operator. `make run` and `make docker-run` start a Jaeger UI on . See **[Overhead timing & tracing](./docs/configuration.md#overhead-timing--tracing-all-components)** for the full phase taxonomy. diff --git a/ansible/hades/README.md b/ansible/hades/README.md index 5d611687..3d188f77 100644 --- a/ansible/hades/README.md +++ b/ansible/hades/README.md @@ -16,17 +16,20 @@ All variables and their defaults are defined in [`defaults/main.yml`](defaults/m | -------- | ------- | ----------- | | `hades_node_role` | `scheduler` | Which component to install on the host: `api` or `scheduler`. | | `hades_version` | `latest` | Image tag for both components (overridable per component via `hades_api_version` / `hades_scheduler_version`). | -| `hades_api_port` | `8080` | Port the API listens on. | +| `hades_api_password` | `""` | **Required on `api` nodes.** Basic Auth key for `POST /build` (passed to the API as `AUTH_KEY`). The role fails if it is empty. | +| `hades_api_port` | `8080` | Port the API container is published on, bound to `127.0.0.1` only (nginx proxies to `localhost:8080`). | | `hades_api_host` | `localhost` | Public hostname of the API node. | | `hades_api_certificate_fullchain_path` | `""` | Path to the TLS fullchain certificate on the API host (enables HTTPS via nginx). | | `hades_api_certificate_key_path` | `""` | Path to the TLS private key on the API host. | | `hades_nats_url` | `nats://localhost:4222` | NATS URL both components connect to. | -| `hades_nats_username` / `hades_nats_password` | `""` | NATS credentials (optional). | +| `hades_nats_username` / `hades_nats_password` | `""` | NATS credentials. **Required on `api` nodes**: the NATS server there enforces them, and scheduler nodes must use the same values. Port `4222` is published on the API node for the scheduler hosts (restrict it with a firewall to those hosts); the monitor port `8222` is bound to `127.0.0.1`. | | `hades_nats_tls_enabled` | `false` | Enable TLS for the NATS connection. | | `hades_scheduler_concurrency` | `1` | Number of jobs the scheduler runs concurrently. | | `hades_executor` | `docker` | Scheduler executor (`docker` or `k8s`). | | `hades_debug` | `false` | Verbose logging for both components (`hades_api_debug` / `hades_scheduler_debug` inherit this). | +The rendered `.env` and compose files contain secrets and are written with mode `0600`. + The corresponding runtime environment variables are documented in [docs/configuration.md](../../docs/configuration.md). ## Example playbook @@ -50,6 +53,7 @@ The corresponding runtime environment variables are documented in [docs/configur vars: hades_version: "latest" hades_node_role: "api" + hades_api_password: "{{ vault_hades_api_password }}" hades_api_certificate_fullchain_path: "/var/lib/cert/cert.fullchain.pem" hades_api_certificate_key_path: "/var/lib/cert/cert.privkey.pem" hades_nats_url: "nats://nats.hades.example:4222" diff --git a/ansible/hades/defaults/main.yml b/ansible/hades/defaults/main.yml index 711baa29..dffff1ba 100644 --- a/ansible/hades/defaults/main.yml +++ b/ansible/hades/defaults/main.yml @@ -6,7 +6,7 @@ hades_api_version: "{{ hades_version }}" hades_node_role: "scheduler" # possible options are "scheduler" and "api" hades_api_user: "hades" # Currently not used -hades_api_password: "" +hades_api_password: "" # Required on api nodes: Basic Auth key for POST /build (user "hades") hades_api_port: 8080 hades_api_host: "localhost" @@ -14,7 +14,7 @@ hades_api_certificate_fullchain_path: "" hades_api_certificate_key_path: "" hades_nats_url: "nats://localhost:4222" -hades_nats_username: "" +hades_nats_username: "" # Required on api nodes: the NATS server on the api node enforces these credentials hades_nats_password: "" hades_nats_tls_enabled: false diff --git a/ansible/hades/tasks/check_host_config.yml b/ansible/hades/tasks/check_host_config.yml index d0896b51..f9115bef 100644 --- a/ansible/hades/tasks/check_host_config.yml +++ b/ansible/hades/tasks/check_host_config.yml @@ -9,3 +9,10 @@ ansible.builtin.fail: msg: "Docker is not installed" when: docker_installed.stat.exists == False + +- name: Fail if the api node has no auth key or NATS credentials + ansible.builtin.fail: + msg: "Set hades_api_password (API Basic Auth key), hades_nats_username and hades_nats_password for api nodes" + when: + - hades_node_role == "api" + - hades_api_password == "" or hades_nats_username == "" or hades_nats_password == "" diff --git a/ansible/hades/tasks/install_api.yml b/ansible/hades/tasks/install_api.yml index 4ddd6810..6a0a3d1a 100644 --- a/ansible/hades/tasks/install_api.yml +++ b/ansible/hades/tasks/install_api.yml @@ -8,18 +8,20 @@ - name: Create hades config env file become: true + no_log: true ansible.builtin.template: src: hades.env.j2 dest: /opt/hades/api/.env - mode: 0644 + mode: "0600" notify: Restart hades api - name: Add hades api docker compose become: true + no_log: true ansible.builtin.template: src: docker-compose-api.yml.j2 dest: /opt/hades/api/docker-compose.yml - mode: 0644 + mode: "0600" notify: Restart hades api - name: Pull new api docker image diff --git a/ansible/hades/tasks/install_scheduler.yml b/ansible/hades/tasks/install_scheduler.yml index b31baa19..f0a1c13b 100644 --- a/ansible/hades/tasks/install_scheduler.yml +++ b/ansible/hades/tasks/install_scheduler.yml @@ -8,17 +8,19 @@ - name: Create hades config env file become: true + no_log: true ansible.builtin.template: src: hades.env.j2 dest: /opt/hades/scheduler/.env - mode: 0644 + mode: "0600" - name: Add hades scheduler docker compose become: true + no_log: true ansible.builtin.template: src: docker-compose-scheduler.yml.j2 dest: /opt/hades/scheduler/docker-compose.yml - mode: 0644 + mode: "0600" - name: Pull new scheduler docker image become: true diff --git a/ansible/hades/templates/docker-compose-api.yml.j2 b/ansible/hades/templates/docker-compose-api.yml.j2 index b294c0ad..1ff61ecc 100644 --- a/ansible/hades/templates/docker-compose-api.yml.j2 +++ b/ansible/hades/templates/docker-compose-api.yml.j2 @@ -3,7 +3,7 @@ services: hadesAPI: image: ghcr.io/hades-scheduler/hades/hades-api:{{ hades_api_version }} ports: - - "{{hades_api_port}}:8080" + - "127.0.0.1:{{ hades_api_port }}:8080" networks: - hades depends_on: @@ -15,15 +15,15 @@ services: - NATS_PASSWORD={{ hades_nats_password }} - NATS_TLS_ENABLED={{ hades_nats_tls_enabled }} - DEBUG={{ hades_api_debug }} - - AUTH_KEY={{ hades_api_auth_key | default('') }} + - AUTH_KEY={{ hades_api_password }} nats: image: nats:2.11.4 - command: ["-js", "-m", "8222"] + command: ["-js", "-m", "8222", "--user", {{ hades_nats_username | to_json }}, "--pass", {{ hades_nats_password | to_json }}] ports: - "4222:4222" - - "8222:8222" + - "127.0.0.1:8222:8222" networks: - hades healthcheck: diff --git a/bruno/HadesLogManager/Get Job Build Logs.bru b/bruno/HadesLogManager/Get Job Build Logs.bru index 79b27a9f..cb46c05e 100644 --- a/bruno/HadesLogManager/Get Job Build Logs.bru +++ b/bruno/HadesLogManager/Get Job Build Logs.bru @@ -5,7 +5,7 @@ meta { } get { - url: http://{{logmanager_url}}/jobs/631f94c6-316c-44bc-a974-cbb15d682c5f/logs + url: {{scheme}}://{{logmanager_url}}/jobs/631f94c6-316c-44bc-a974-cbb15d682c5f/logs body: none auth: inherit } diff --git a/bruno/HadesLogManager/Get Job Status.bru b/bruno/HadesLogManager/Get Job Status.bru index 09c2f5af..8487ce2a 100644 --- a/bruno/HadesLogManager/Get Job Status.bru +++ b/bruno/HadesLogManager/Get Job Status.bru @@ -5,7 +5,7 @@ meta { } get { - url: http://{{logmanager_url}}/jobs/631f94c6-316c-44bc-a974-cbb15d682c5f/status + url: {{scheme}}://{{logmanager_url}}/jobs/631f94c6-316c-44bc-a974-cbb15d682c5f/status body: none auth: inherit } diff --git a/bruno/HadesLogManager/Get Jobs.bru b/bruno/HadesLogManager/Get Jobs.bru index 456edd60..01053c20 100644 --- a/bruno/HadesLogManager/Get Jobs.bru +++ b/bruno/HadesLogManager/Get Jobs.bru @@ -5,7 +5,7 @@ meta { } get { - url: http://{{logmanager_url}}/jobs + url: {{scheme}}://{{logmanager_url}}/jobs body: none auth: inherit } diff --git a/bruno/HadesLogManager/health check.bru b/bruno/HadesLogManager/health check.bru index ef78a83f..6009a414 100644 --- a/bruno/HadesLogManager/health check.bru +++ b/bruno/HadesLogManager/health check.bru @@ -5,7 +5,7 @@ meta { } get { - url: http://{{logmanager_url}}/health + url: {{scheme}}://{{logmanager_url}}/health body: none auth: inherit } diff --git a/bruno/api/Create Build Job (Image Pull Failure).bru b/bruno/api/Create Build Job (Image Pull Failure).bru index bbd7f368..285f3cec 100644 --- a/bruno/api/Create Build Job (Image Pull Failure).bru +++ b/bruno/api/Create Build Job (Image Pull Failure).bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Create Build Job (Resource Limits).bru b/bruno/api/Create Build Job (Resource Limits).bru index d8cb3dc5..ce0f27f9 100644 --- a/bruno/api/Create Build Job (Resource Limits).bru +++ b/bruno/api/Create Build Job (Resource Limits).bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Create Build Job (Test Fail) with Result Parse Step.bru b/bruno/api/Create Build Job (Test Fail) with Result Parse Step.bru index 5c66e333..424fa52a 100644 --- a/bruno/api/Create Build Job (Test Fail) with Result Parse Step.bru +++ b/bruno/api/Create Build Job (Test Fail) with Result Parse Step.bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Create Build Job (Test Fail).bru b/bruno/api/Create Build Job (Test Fail).bru index 089e6451..a2199369 100644 --- a/bruno/api/Create Build Job (Test Fail).bru +++ b/bruno/api/Create Build Job (Test Fail).bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Create Build Job (Test Succeed - Result Container).bru b/bruno/api/Create Build Job (Test Succeed - Result Container).bru index ead3272f..5c97ae52 100644 --- a/bruno/api/Create Build Job (Test Succeed - Result Container).bru +++ b/bruno/api/Create Build Job (Test Succeed - Result Container).bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Create Build Job (Test Succeed) with Result Parse Step.bru b/bruno/api/Create Build Job (Test Succeed) with Result Parse Step.bru index e77b3a2a..4e1e0ede 100644 --- a/bruno/api/Create Build Job (Test Succeed) with Result Parse Step.bru +++ b/bruno/api/Create Build Job (Test Succeed) with Result Parse Step.bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Create Build Job (Test Succeed).bru b/bruno/api/Create Build Job (Test Succeed).bru index 3945add7..b242c01f 100644 --- a/bruno/api/Create Build Job (Test Succeed).bru +++ b/bruno/api/Create Build Job (Test Succeed).bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Create Build Job (long-running-debug).bru b/bruno/api/Create Build Job (long-running-debug).bru index c5b590ba..6e15e355 100644 --- a/bruno/api/Create Build Job (long-running-debug).bru +++ b/bruno/api/Create Build Job (long-running-debug).bru @@ -5,7 +5,7 @@ meta { } post { - url: http://{{hostname}}/build + url: {{scheme}}://{{hostname}}/build body: json auth: basic } diff --git a/bruno/api/Health Check (Ping).bru b/bruno/api/Health Check (Ping).bru index 6ec10bd6..95ba1a1e 100644 --- a/bruno/api/Health Check (Ping).bru +++ b/bruno/api/Health Check (Ping).bru @@ -5,7 +5,7 @@ meta { } get { - url: http://{{hostname}}/ping + url: {{scheme}}://{{hostname}}/ping body: none auth: none } diff --git a/bruno/environments/Hades Host.bru b/bruno/environments/Hades Host.bru index ef960073..3656dbc7 100644 --- a/bruno/environments/Hades Host.bru +++ b/bruno/environments/Hades Host.bru @@ -1,4 +1,5 @@ vars { + scheme: https hostname: hades-test.aet.cit.tum.de logmanager_url: hades-test.aet.cit.tum.de } diff --git a/bruno/environments/Kubernetes.bru b/bruno/environments/Kubernetes.bru index 91dfa209..ec41c9a9 100644 --- a/bruno/environments/Kubernetes.bru +++ b/bruno/environments/Kubernetes.bru @@ -1,4 +1,5 @@ vars { + scheme: https hostname: hades.student.k8s.aet.cit.tum.de logmanager_url: hades.student.k8s.aet.cit.tum.de } diff --git a/bruno/environments/Local-Development.bru b/bruno/environments/Local-Development.bru index 582fb306..bbb6948e 100644 --- a/bruno/environments/Local-Development.bru +++ b/bruno/environments/Local-Development.bru @@ -1,4 +1,5 @@ vars { + scheme: http hostname: localhost:8080 logmanager_url: localhost:8081 } diff --git a/bruno/environments/docker-compose.bru b/bruno/environments/docker-compose.bru index d84fbf82..b7bead0d 100644 --- a/bruno/environments/docker-compose.bru +++ b/bruno/environments/docker-compose.bru @@ -1,4 +1,5 @@ vars { + scheme: http hostname: localhost:8081 logmanager_url: localhost:8081 } diff --git a/compose.yml b/compose.yml index 73602398..ae53500f 100644 --- a/compose.yml +++ b/compose.yml @@ -8,7 +8,7 @@ services: dockerfile: ./HadesAPI/Dockerfile ports: # Set the Port to 8081 to avoid conflicts with the artemis server - - "8081:8080" + - "127.0.0.1:8081:8080" networks: - hades depends_on: @@ -17,6 +17,15 @@ services: environment: - NATS_URL=nats://nats:4222 - HADES_VERSION=${HADES_VERSION:-dev} + - AUTH_KEY=${AUTH_KEY:?set AUTH_KEY in .env} + - DASHBOARD_USERNAME=${DASHBOARD_USERNAME:-} + - DASHBOARD_PASSWORD_HASH=${DASHBOARD_PASSWORD_HASH:-} + - DASHBOARD_SESSION_SECRET=${DASHBOARD_SESSION_SECRET:-} + - DASHBOARD_SESSION_TTL=${DASHBOARD_SESSION_TTL:-12h} + - DASHBOARD_JOB_RETENTION=${DASHBOARD_JOB_RETENTION:-1h} + - DASHBOARD_TRUSTED_PROXIES=${DASHBOARD_TRUSTED_PROXIES:-} + - DASHBOARD_COOKIE_INSECURE=${DASHBOARD_COOKIE_INSECURE:-false} + - LOG_MANAGER_URL=http://hadesLogManager:8081 - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317 hadesScheduler: @@ -48,7 +57,7 @@ services: context: . dockerfile: ./HadesLogManager/Dockerfile ports: - - "8083:8081" # Map to 8083 to avoid conflicts + - "127.0.0.1:8083:8081" # Map to 8083 to avoid conflicts networks: - hades depends_on: @@ -59,6 +68,9 @@ services: environment: - HADESLOGMANAGER_API_PORT=8081 - NATS_URL=nats://nats:4222 + # Internal callback receivers (host names, IPs, CIDRs) the log manager may + # POST to; empty = public addresses only. + - CALLBACK_ALLOWED_HOSTS=${CALLBACK_ALLOWED_HOSTS:-} - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317 nats: @@ -67,8 +79,8 @@ services: networks: - hades ports: - - "4222:4222" - - "8222:8222" + - "127.0.0.1:4222:4222" + - "127.0.0.1:8222:8222" healthcheck: test: ["CMD", "nats-server", "--health"] interval: 10s @@ -85,8 +97,8 @@ services: networks: - hades ports: - - "16686:16686" # Jaeger UI - - "4317:4317" # OTLP gRPC receiver + - "127.0.0.1:16686:16686" # Jaeger UI + - "127.0.0.1:4317:4317" # OTLP gRPC receiver networks: hades: diff --git a/docker-compose.deploy.yml b/docker-compose.deploy.yml index b4b1c216..730546b2 100644 --- a/docker-compose.deploy.yml +++ b/docker-compose.deploy.yml @@ -17,7 +17,7 @@ services: - hades hadesAPI: - ports: [] + ports: !reset [] expose: - "8080" networks: @@ -34,23 +34,18 @@ services: - hades hadesLogManager: - ports: [] - expose: - - "8081" + ports: !reset [] networks: - hades - labels: - - "traefik.enable=true" - - "traefik.http.routers.hadeslogmanager.rule=Host(`${HADES_LOGMANAGER_HOST:-logs.${HADES_API_HOST}}`)" - - "traefik.http.routers.hadeslogmanager.entrypoints=web,websecure" - - "traefik.http.routers.hadeslogmanager.tls.certresolver=letsencrypt" - - "traefik.http.services.hadeslogmanager.loadbalancer.server.port=8081" nats: - ports: [] + ports: !reset [] networks: - hades + jaeger: + ports: !reset [] + networks: hades: external: false \ No newline at end of file diff --git a/docker-compose.test.yml b/docker-compose.test.yml index 567cb6a3..d5cf15cf 100644 --- a/docker-compose.test.yml +++ b/docker-compose.test.yml @@ -13,7 +13,7 @@ services: - NATS_URL=nats://nats:4222 # Deployed version shown in the dashboard header; matches the image tag. - HADES_VERSION=${IMAGE_TAG} - - AUTH_KEY=${AUTH_KEY:-} + - AUTH_KEY=${AUTH_KEY:?AUTH_KEY is required} - RETENTION_IN_MIN=${RETENTION_IN_MIN:-30} - MAX_RETRIES=${MAX_RETRIES:-3} - TIMEOUT_IN_MIN=${TIMEOUT_IN_MIN:-0} diff --git a/docs/api.md b/docs/api.md index 2916d3e2..bd4826ca 100644 --- a/docs/api.md +++ b/docs/api.md @@ -27,7 +27,7 @@ curl http://localhost:8080/ping Validate a job, assign it a UUID, and enqueue it on NATS by priority. -**Auth:** HTTP Basic Auth (`hades` / `AUTH_KEY`) when the server was started with a non-empty `AUTH_KEY`; otherwise open. +**Auth:** HTTP Basic Auth (`hades` / `AUTH_KEY`). `AUTH_KEY` is required: the API exits at start with an empty key unless `ALLOW_UNAUTHENTICATED=true` (local development only), in which case the endpoint is open. **Request body** - a `RESTPayload` (see [Job payload schema](#job-payload-schema)): @@ -58,7 +58,8 @@ curl -X POST -H "Content-Type: application/json" \ | Status | Body | Meaning | | ------ | ---- | ------- | | `200 OK` | `{ "message": "Successfully enqueued job", "job_id": "" }` | Job accepted and queued. | -| `400 Bad Request` | plain-text message naming the offending field | Invalid JSON or failed validation (e.g. missing `name`, unparseable `memory_limit`). | +| `400 Bad Request` | plain-text message naming the offending field | Invalid JSON or failed validation (e.g. missing `name`, unparseable `memory_limit`, duplicate step ids). | +| `413 Request Entity Too Large` | | The body exceeds 1 MiB. | | `500 Internal Server Error` | `Failed to enqueue job` | NATS publish failed. | --- @@ -117,7 +118,7 @@ Submitted to `POST /build`. Field semantics come from `shared/payload/payload.go | `metadata` | map[string]string | Step-specific environment variables. | | `cpu_limit` | uint | CPU limit in whole cores (e.g. `1` = 1 core, `2` = 2 cores). | | `memory_limit` | string | Memory limit, e.g. `512M`, `2G`. | -| `network` | string | Docker network mode: `none`, `bridge`, `host`, `default`, or a named network. Empty = executor default. **Docker executor only** - accepted but not enforced on Kubernetes. | +| `network` | string | Docker network mode: `none`, `bridge`, `host`, `default`, or a named network. Empty = executor default. The scheduler only accepts the modes in `DOCKER_ALLOWED_NETWORKS` (default `none,bridge,default`); `host` and named networks fail the job with a reason unless listed. **Docker executor only** - accepted but not enforced on Kubernetes. | | `memory_swap` | string | Total memory+swap limit (same format as `memory_limit`); must be set together with `memory_limit` and be `>=` it. **Docker executor only.** | | `pids_limit` | int64 | Max PIDs in the container; `0` = unlimited. **Docker executor only.** | @@ -133,7 +134,7 @@ Steps of a job share a per-job volume, so a file written by one step is visible ## Interactive OpenAPI spec -HadesLogManager serves an interactive Swagger UI **only when `DEBUG=true`** (the spec is never exposed in production): +HadesAPI and HadesLogManager serve an interactive Swagger UI **only when `DEBUG=true`** (the spec is never exposed in production). On HadesAPI it sits behind the same Basic Auth as `/build`: - HadesAPI: `http://localhost:8080/swagger/index.html` - HadesLogManager: `http://localhost:8081/swagger/index.html` diff --git a/docs/configuration.md b/docs/configuration.md index 723b4b67..7fa27273 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -74,7 +74,11 @@ The bundled Jaeger UI Service is cluster-internal (ClusterIP) by default, so you | -------- | ------- | ----------- | ------ | | `API_PORT` | `8080` | Port the HTTP API listens on. | `HadesAPI/main.go` (`HadesAPIConfig`) | | `METRICS_PORT` | `8082` | Port the Prometheus `/metrics` endpoint listens on (see [Metrics](#metrics-all-components)). | `HadesAPI/main.go` (`HadesAPIConfig`) | -| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Empty disables auth (a warning is logged). | `HadesAPI/main.go` (`HadesAPIConfig`) | +| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Required: the API exits at start when it is empty unless `ALLOW_UNAUTHENTICATED=true`. | `HadesAPI/main.go` (`HadesAPIConfig`) | +| `ALLOW_UNAUTHENTICATED` | `false` | Start with an empty `AUTH_KEY` and serve `/build` without auth. Local development only; `make run` sets it. | `HadesAPI/main.go` (`HadesAPIConfig`) | +| `DASHBOARD_TRUSTED_PROXIES` | | Comma-separated proxy addresses or CIDRs whose `X-Forwarded-For` is trusted for the dashboard login lockout (5 attempts per client address per 15 minutes). Set it behind a reverse proxy, otherwise all clients share one lockout (a warning is logged at start). | `HadesAPI/main.go` (`HadesAPIConfig`) | + +`POST /build` bodies are capped at 1 MiB (413 above) and duplicate step ids get 400. The dashboard login body is capped at 4 KiB. Plus the [NATS](#nats-connection-all-components) and [global](#global-all-components) variables. @@ -105,6 +109,9 @@ Plus the [NATS](#nats-connection-all-components) and [global](#global-all-compon | `DOCKER_SCRIPT_EXECUTOR` | `/bin/bash -c` | Shell used to run each step's `script`. | `HadesScheduler/docker/env.go` | | `DOCKER_CPU_LIMIT` | | Default CPU limit (whole CPUs, e.g. `6`) when a step sets none. | `HadesScheduler/docker/env.go` | | `DOCKER_MEMORY_LIMIT` | | Default memory limit (e.g. `4g`) when a step sets none. | `HadesScheduler/docker/env.go` | +| `DOCKER_ALLOWED_NETWORKS` | `none,bridge,default` | Comma-separated network modes a step may request. `host` and named networks fail the job with a reason unless listed. Never list the network that carries NATS. | `HadesScheduler/docker/env.go` | +| `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` | `0` | Job timeout applied when the payload's `timeout_seconds` is 0. `0` = off. Docker executor only. | `HadesScheduler/docker/env.go` | +| `HADES_MAX_JOB_TIMEOUT_SECONDS` | `0` | Upper bound on every job's timeout. `0` = off. Docker executor only. | `HadesScheduler/docker/env.go` | ### Kubernetes executor (`HADES_EXECUTOR=k8s`) @@ -154,7 +161,9 @@ Deployed by the Helm chart (`hades-log-manager`); also run locally via `make run | `STATUS_WEBHOOK_CONCURRENCY` | `16` | Deliveries in flight at once; keeps a dead receiver from delaying other jobs. | `HadesLogManager/status_webhook.go` | | `STATUS_WEBHOOK_MAX_PENDING` | `1000` | Status events awaiting acknowledgement (in flight or backing off). | `HadesLogManager/status_webhook.go` | -Both outbound pushes are configured per job, not globally: +| `CALLBACK_ALLOWED_HOSTS` | | Comma-separated host names, IPs or CIDRs exempt from the callback address block (internal receivers such as Artemis in the same cluster or compose network). A host name entry must equal the host in the callback URL and trusts whatever it resolves to. | `HadesLogManager/http_client.go` | + +Both outbound pushes are configured per job, not globally. Callbacks to loopback, private (RFC 1918, `fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), unspecified and multicast addresses are refused at connect time unless allowed by `CALLBACK_ALLOWED_HOSTS`. The log callback does not follow redirects, and HTTP proxy environment variables are ignored. A blocked status webhook is logged as an error and not retried. - `callback_url` - an absolute `http`/`https` URL with a host. The Log Manager forwards that job's aggregated **logs** there once the log stream has drained. If omitted, the job's logs are not forwarded. - `status_callback_url` - an absolute `http`/`https` URL with a host. Receives the **job-status webhook** on the terminal status, independently of log forwarding. See [`HadesLogManager/Readme.md`](../HadesLogManager/Readme.md#job-status-webhook). diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index 411419b4..49eea8cc 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -102,7 +102,8 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | Key | Type | Default | Description | |-----|------|---------|-------------| | debug | bool | `false` | Global debug switch. When true, every Hades container gets `DEBUG=true` (and the operator additionally gets `DEV_MODE=true`) for verbose logging. | -| hadesApi.authKeySecret | string | `""` | Name of an existing Secret holding key `AUTH_KEY` to protect the `/build` endpoint. Empty = no auth (endpoint is open). | +| hadesApi.allowUnauthenticated | bool | `false` | Serve `/build` without authentication when `authKeySecret` is empty. Only for throwaway test installs; the API refuses to start otherwise. | +| hadesApi.authKeySecret | string | `""` | Name of an existing Secret holding key `AUTH_KEY` to protect the `/build` endpoint. Required unless `allowUnauthenticated` is true. | | hadesApi.dashboard | object | `{"cookieInsecure":false,"jobRetention":"1h","logManagerURL":"http://hades-log-manager-service:8081","redactMode":"","secretKeyPatterns":"","secretName":"","sessionTTL":"12h","trustedProxies":[]}` | Operator dashboard (web UI) served by the API. | | hadesApi.dashboard.cookieInsecure | bool | `false` | Drop the Secure flag on the session cookie. Only for local HTTP dev; never enable in a real deployment (requires TLS otherwise login breaks). | | hadesApi.dashboard.jobRetention | string | `"1h"` | How long finished jobs stay in the dashboard's in-memory list (Go duration). | @@ -117,6 +118,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | hadesApi.resources | object | `{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"200m","memory":"256Mi"}}` | Resource requests and limits for the API pod. | | hadesApi.service.port | int | `8080` | Service port for the API. | | hadesApi.service.targetPort | int | `8080` | Container port the API listens on. | +| hadesLogManager.callbackAllowedHosts | string | `""` | Comma-separated host names, IPs or CIDRs that `callback_url` and `status_callback_url` may target although they are loopback, private or link-local, e.g. the in-cluster Artemis service (`artemis.artemis.svc.cluster.local`). Empty = public addresses only. | | hadesLogManager.image.pullPolicy | string | `"Always"` | Image pull policy for the log manager. | | hadesLogManager.replicaCount | int | `1` | Keep at 1: the log manager aggregates per-job logs in memory and uses plain (non-queue, non-durable) NATS subscriptions, so multiple replicas would independently forward duplicate/split logs to the Artemis adapter. | | hadesLogManager.resources | object | `{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"200m","memory":"256Mi"}}` | Resource requests and limits for the log manager pod. | @@ -128,7 +130,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | hadesLogManager.statusWebhook.maxBackoff | string | `"5m"` | Ceiling for the retry delay. | | hadesLogManager.statusWebhook.maxPending | int | `1000` | Status events awaiting acknowledgement (in flight or waiting out a retry backoff). | | hadesLogManager.statusWebhook.timeout | string | `"10s"` | Bound on a single delivery, including the callback-URL lookup. | -| hadesOperator.DeleteOnComplete | bool | `false` | Delete BuildJob CRs (and their Jobs) once they finish. Defaults to false so completed BuildJobs/pods linger (they self-clean via the Job TTL, ~1h) and the operator can capture every step's logs before the pod is removed; Kubernetes only serves container logs while the pod exists, so immediate deletion can lose logs for fast jobs. Set to true only once the operator captures all logs before deletion (the companion barrier fix). | +| hadesOperator.DeleteOnComplete | bool | `true` | Delete BuildJob CRs (and their Jobs) once they finish. Defaults to true so job scripts and metadata (which may hold secrets) do not stay in the cluster. The operator waits until every step's logs are published (bounded by `logDrainTimeout`) before it deletes, because Kubernetes only serves container logs while the pod exists. Set to false to keep finished BuildJobs and pods for debugging; they then self-clean via the Job TTL (~1h) but the BuildJob CR stays. | | hadesOperator.clusterWide | bool | `false` | Grant cluster-wide RBAC instead of namespace-scoped. | | hadesOperator.finalizerImage | string | `"busybox:1.37.0"` | Image for the main container of every BuildJob pod. Kubernetes requires at least one non-init container and the build steps run as init containers, so this one exists only to exit once they are done: it runs `sh -c "echo build finished"` and needs nothing but a shell. Override it for clusters that cannot reach Docker Hub, or to point at a mirror. Keep it pinned to an explicit tag - an untagged image resolves to ":latest", which Kubernetes special-cases to imagePullPolicy: Always, adding a registry round trip to every job for an image the node already has. Empty falls back to the operator's built-in default. | | hadesOperator.image.pullPolicy | string | `"Always"` | Image pull policy for the operator. | @@ -162,6 +164,12 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | nats.config.jetstream.memoryStore.maxSize | string | `"1Gi"` | Maximum in-memory JetStream store size. | | nats.host | string | `"hades-nats.hades.svc.cluster.local"` | In-cluster NATS service hostname. | | nats.port | int | `4222` | NATS client port. | +| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. | +| networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | +| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach, on top of every pod in the release namespace. Set this to the cluster pod CIDR on CNIs that match pod IPs against ipBlock rules, otherwise the "outside" rule can still reach pods. | +| networkPolicy.nats.podLabels | object | `{}` | Labels of the NATS pods. Defaults match the bundled nats chart (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). | +| rbac.hadesApi.serviceAccountName | string | `"hades-api"` | ServiceAccount name for the API (no Kubernetes API access, token not mounted). | +| rbac.hadesLogManager.serviceAccountName | string | `"hades-log-manager"` | ServiceAccount name for the log manager (no Kubernetes API access, token not mounted). | | rbac.hadesOperator.serviceAccountName | string | `"hades-operator"` | ServiceAccount name for the operator. | | rbac.hadesScheduler.serviceAccountName | string | `"hades-scheduler"` | ServiceAccount name for the scheduler. | | tracing | object | `{"deployJaeger":false,"enabled":false,"endpoint":"","jaeger":{"image":"jaegertracing/all-in-one:1.62.0","ui":{"auth":{"existingSecret":"","password":"","username":"admin"},"ingress":{"annotations":{},"className":"nginx","enabled":false,"host":"","tls":{"enabled":false,"secretName":""}}}}}` | OpenTelemetry tracing. When enabled, every Hades service exports spans to an OTLP endpoint so a job's overhead/runtime breakdown is visible as a per-job trace waterfall (API -> scheduler -> operator). Disabled by default: with no endpoint the services run a noop tracer, so the chart installs cleanly without a tracing backend. | diff --git a/helm/hades/templates/hades-api-deployment.yaml b/helm/hades/templates/hades-api-deployment.yaml index 7638b8ba..81d92466 100644 --- a/helm/hades/templates/hades-api-deployment.yaml +++ b/helm/hades/templates/hades-api-deployment.yaml @@ -15,6 +15,8 @@ spec: labels: app: hades-api-selector spec: + serviceAccountName: {{ .Values.rbac.hadesApi.serviceAccountName }} + automountServiceAccountToken: false containers: - name: "hades-api" image: "{{ .Values.hadesApi.image.repository }}:{{ .Values.hadesApi.image.tag | default .Chart.AppVersion }}" @@ -63,6 +65,11 @@ spec: secretKeyRef: name: {{ .Values.hadesApi.authKeySecret | quote }} key: AUTH_KEY + {{- else if .Values.hadesApi.allowUnauthenticated }} + - name: ALLOW_UNAUTHENTICATED + value: "true" + {{- else }} + {{- fail "hadesApi.authKeySecret is empty: set it to a Secret with key AUTH_KEY, or set hadesApi.allowUnauthenticated=true to serve /build without authentication" }} {{- end }} {{- if .Values.hadesApi.dashboard.secretName }} # Dashboard credentials. When this Secret is set the operator UI is diff --git a/helm/hades/templates/hades-api-serviceaccount.yaml b/helm/hades/templates/hades-api-serviceaccount.yaml new file mode 100644 index 00000000..40b24b97 --- /dev/null +++ b/helm/hades/templates/hades-api-serviceaccount.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.rbac.hadesApi.serviceAccountName }} + namespace: {{ .Release.Namespace }} +automountServiceAccountToken: false diff --git a/helm/hades/templates/hades-log-manager-deployment.yaml b/helm/hades/templates/hades-log-manager-deployment.yaml index 3912bedf..6d1adbb0 100644 --- a/helm/hades/templates/hades-log-manager-deployment.yaml +++ b/helm/hades/templates/hades-log-manager-deployment.yaml @@ -12,6 +12,8 @@ spec: labels: app: hades-log-manager-selector spec: + serviceAccountName: {{ .Values.rbac.hadesLogManager.serviceAccountName }} + automountServiceAccountToken: false containers: - name: "hades-log-manager" image: "{{ .Values.hadesLogManager.image.repository }}:{{ .Values.hadesLogManager.image.tag | default .Chart.AppVersion }}" @@ -52,6 +54,8 @@ spec: value: "nats://{{ .Values.nats.host }}:{{ .Values.nats.port }}" - name: HADESLOGMANAGER_API_PORT value: {{ .Values.hadesLogManager.service.targetPort | quote }} + - name: CALLBACK_ALLOWED_HOSTS + value: {{ .Values.hadesLogManager.callbackAllowedHosts | quote }} - name: STATUS_WEBHOOK_ENABLED value: {{ .Values.hadesLogManager.statusWebhook.enabled | quote }} - name: STATUS_WEBHOOK_MAX_ATTEMPTS diff --git a/helm/hades/templates/hades-log-manager-serviceaccount.yaml b/helm/hades/templates/hades-log-manager-serviceaccount.yaml new file mode 100644 index 00000000..15cc85e9 --- /dev/null +++ b/helm/hades/templates/hades-log-manager-serviceaccount.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.rbac.hadesLogManager.serviceAccountName }} + namespace: {{ .Release.Namespace }} +automountServiceAccountToken: false diff --git a/helm/hades/templates/hades-operator-role.yaml b/helm/hades/templates/hades-operator-role.yaml index 752d7816..da536d81 100644 --- a/helm/hades/templates/hades-operator-role.yaml +++ b/helm/hades/templates/hades-operator-role.yaml @@ -6,7 +6,7 @@ metadata: name: {{ .Values.rbac.hadesOperator.role.name }} rules: - apiGroups: [""] - resources: ["configmaps","pods","secrets"] + resources: ["pods"] verbs: ["get","list","watch","create","update","patch","delete"] - apiGroups: [""] @@ -37,7 +37,7 @@ metadata: name: {{ .Values.rbac.hadesOperator.role.name }} rules: - apiGroups: [""] - resources: ["configmaps","pods","secrets"] + resources: ["pods"] verbs: ["get","list","watch","create","update","patch","delete"] - apiGroups: [""] diff --git a/helm/hades/templates/networkpolicy.yaml b/helm/hades/templates/networkpolicy.yaml new file mode 100644 index 00000000..24295253 --- /dev/null +++ b/helm/hades/templates/networkpolicy.yaml @@ -0,0 +1,70 @@ +{{- if .Values.networkPolicy.enabled }} +# Job pods run untrusted step code: no connections into the release namespace. +# Other namespaces (DNS) and the outside stay reachable. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: hades-job-pods + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + hades.tum.de/managed-by: hades-operator + policyTypes: + - Egress + egress: + - to: + - namespaceSelector: + matchExpressions: + - key: kubernetes.io/metadata.name + operator: NotIn + values: [{{ .Release.Namespace | quote }}] + - to: + - ipBlock: + cidr: 0.0.0.0/0 + {{- with .Values.networkPolicy.jobPods.blockedCIDRs }} + except: + {{- toYaml . | nindent 14 }} + {{- end }} +--- +# Only the Hades components (and NATS itself, for clustering) reach NATS. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: hades-nats + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + {{- if .Values.networkPolicy.nats.podLabels }} + {{- toYaml .Values.networkPolicy.nats.podLabels | nindent 6 }} + {{- else }} + app.kubernetes.io/name: nats + app.kubernetes.io/instance: {{ .Release.Name }} + {{- end }} + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchExpressions: + - key: app + operator: In + values: + - hades-api-selector + - hades-scheduler-selector + - hades-operator-selector + - hades-log-manager-selector + ports: + - protocol: TCP + port: {{ .Values.nats.port }} + - from: + - podSelector: + matchLabels: + {{- if .Values.networkPolicy.nats.podLabels }} + {{- toYaml .Values.networkPolicy.nats.podLabels | nindent 14 }} + {{- else }} + app.kubernetes.io/name: nats + app.kubernetes.io/instance: {{ .Release.Name }} + {{- end }} +{{- end }} diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index ee18cfb2..d689435f 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -26,8 +26,11 @@ hadesApi: cpu: "200m" memory: "256Mi" # -- Name of an existing Secret holding key `AUTH_KEY` to protect the `/build` - # endpoint. Empty = no auth (endpoint is open). + # endpoint. Required unless `allowUnauthenticated` is true. authKeySecret: "" + # -- Serve `/build` without authentication when `authKeySecret` is empty. + # Only for throwaway test installs; the API refuses to start otherwise. + allowUnauthenticated: false # -- Operator dashboard (web UI) served by the API. dashboard: # -- Name of an existing Secret with keys `DASHBOARD_USERNAME`, @@ -118,13 +121,13 @@ hadesOperator: enabled: true # -- Namespace the operator watches for BuildJob CRs (empty = release namespace). targetNamespace: "" - # -- Delete BuildJob CRs (and their Jobs) once they finish. Defaults to false so - # completed BuildJobs/pods linger (they self-clean via the Job TTL, ~1h) and the - # operator can capture every step's logs before the pod is removed; Kubernetes - # only serves container logs while the pod exists, so immediate deletion can lose - # logs for fast jobs. Set to true only once the operator captures all logs before - # deletion (the companion barrier fix). - DeleteOnComplete: false + # -- Delete BuildJob CRs (and their Jobs) once they finish. Defaults to true so + # job scripts and metadata (which may hold secrets) do not stay in the cluster. + # The operator waits until every step's logs are published (bounded by + # `logDrainTimeout`) before it deletes, because Kubernetes only serves container + # logs while the pod exists. Set to false to keep finished BuildJobs and pods for + # debugging; they then self-clean via the Job TTL (~1h) but the BuildJob CR stays. + DeleteOnComplete: true # -- Maximum number of Jobs the operator admits concurrently; excess are suspended. maxParallelism: "100" # -- Safety resync for a BuildJob whose Job is still running (Go duration). The @@ -186,6 +189,11 @@ hadesLogManager: requests: cpu: "200m" memory: "256Mi" + # -- Comma-separated host names, IPs or CIDRs that `callback_url` and + # `status_callback_url` may target although they are loopback, private or + # link-local, e.g. the in-cluster Artemis service + # (`artemis.artemis.svc.cluster.local`). Empty = public addresses only. + callbackAllowedHosts: "" # Where aggregated job logs are forwarded is configured per job via the # "callback_url" field on the build request, not globally on the log manager. statusWebhook: @@ -207,6 +215,12 @@ hadesLogManager: maxPending: 1000 rbac: + hadesApi: + # -- ServiceAccount name for the API (no Kubernetes API access, token not mounted). + serviceAccountName: hades-api + hadesLogManager: + # -- ServiceAccount name for the log manager (no Kubernetes API access, token not mounted). + serviceAccountName: hades-log-manager hadesScheduler: # -- ServiceAccount name for the scheduler. serviceAccountName: hades-scheduler @@ -316,6 +330,22 @@ tracing: # `existingSecret` is given; the chart hashes it into an htpasswd Secret. password: "" +# -- NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by +# default because the NATS policy assumes the labels of the bundled NATS chart. +networkPolicy: + # -- Render the NetworkPolicies below. Verify the NATS pod labels first + # (`kubectl get pods -n --show-labels`), then enable. + enabled: false + jobPods: + # -- CIDRs job pods may not reach, on top of every pod in the release + # namespace. Set this to the cluster pod CIDR on CNIs that match pod IPs + # against ipBlock rules, otherwise the "outside" rule can still reach pods. + blockedCIDRs: [] + nats: + # -- Labels of the NATS pods. Defaults match the bundled nats chart + # (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). + podLabels: {} + nats: config: jetstream: diff --git a/renovate.json b/renovate.json index 5db72dd6..3129bd2c 100644 --- a/renovate.json +++ b/renovate.json @@ -1,6 +1,7 @@ { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ - "config:recommended" + "config:recommended", + "helpers:pinGitHubActionDigests" ] } diff --git a/shared/redact/redact.go b/shared/redact/redact.go index 5ae5793f..aaad8377 100644 --- a/shared/redact/redact.go +++ b/shared/redact/redact.go @@ -22,6 +22,7 @@ import ( "fmt" "math" "regexp" + "sort" "strings" "github.com/hades-scheduler/hades/shared/payload" @@ -127,18 +128,59 @@ func (r *Redactor) Metadata(m map[string]string) map[string]string { // redacted, and secrets scrubbed from each step's Script. The input payload and // its slices/maps are never modified. func (r *Redactor) Payload(job payload.QueuePayload) payload.QueuePayload { + secrets := r.maskedValues(job.Metadata, nil) + for _, st := range job.Steps { + secrets = r.maskedValues(st.Metadata, secrets) + } + // Longest first so a secret that contains another one is replaced whole. + sort.Slice(secrets, func(i, j int) bool { return len(secrets[i]) > len(secrets[j]) }) + pairs := make([]string, 0, 2*len(secrets)) + for _, v := range secrets { + pairs = append(pairs, v, Mask) + } + scrub := strings.NewReplacer(pairs...) + job.Metadata = r.Metadata(job.Metadata) steps := make([]payload.Step, len(job.Steps)) copy(steps, job.Steps) for i := range steps { steps[i].Metadata = r.Metadata(steps[i].Metadata) - steps[i].Script = r.Text(steps[i].Script) + steps[i].Script = r.Text(scrub.Replace(steps[i].Script)) } job.Steps = steps return job } +// minScriptSecretLen is the shortest masked metadata value that is also +// scrubbed from scripts; shorter values ("1", "on") would mangle ordinary text. +const minScriptSecretLen = 4 + +// maskedValues appends the values of m that Value masks (and that are long +// enough to search for) to dst. +func (r *Redactor) maskedValues(m map[string]string, dst []string) []string { + for k, v := range m { + if _, masked := r.Value(k, v); masked && len(v) >= minScriptSecretLen { + dst = append(dst, v) + } + } + return dst +} + var ( + // authHeader matches an Authorization header or assignment with a Bearer, + // Basic or Token scheme; group 2 is the credential. + authHeader = regexp.MustCompile(`(?i)(authorization\s*[:=]\s*["']?(?:bearer|basic|token)\s+)([^\s"';]+)`) + // urlUserToken matches a token-only URL credential (scheme://token@host). + urlUserToken = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9+.\-]*://)([A-Za-z0-9_.\-]{12,})(@)`) + // spacedSecretFlag matches --password/--token style flags whose value is the + // next word; "--password-stdin" does not match because the flag must end + // with the keyword. + spacedSecretFlag = regexp.MustCompile(`(?i)(\s--[\w\-]*?(?:token|secret|passwd|password|pwd|api[_-]?key|credential)\s+)(["']?)([^\s"';\-][^\s"';]*)`) + // loginPassword matches -p only on a registry login command line, + // where -p is a password; elsewhere it is a path or port flag. + loginPassword = regexp.MustCompile(`(\b(?:docker|podman|buildah|skopeo|crane|oras)\b[^\n;&|]*?\blogin\b[^\n;&|]*?\s-p\s+)(["']?)([^\s"';]+)`) + // mysqlPassword matches the attached -p form of the mysql tools. + mysqlPassword = regexp.MustCompile(`(\bmysql(?:dump|admin)?\b[^\n;&|]*?\s-p)(["']?)([^\s"';]+)`) // urlCredentials captures the password segment of a scheme://user:pass@host URL. urlCredentials = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9+.\-]*://[^/\s:@]+:)([^/\s@]+)(@)`) // sensitiveAssignment matches KEY=VALUE / KEY: VALUE / export KEY=VALUE where @@ -162,6 +204,17 @@ func (r *Redactor) Text(s string) string { return s } s = urlCredentials.ReplaceAllString(s, "${1}"+Mask+"${3}") + s = urlUserToken.ReplaceAllStringFunc(s, func(m string) string { + sub := urlUserToken.FindStringSubmatch(m) + if !strings.ContainsAny(sub[2], "0123456789_-") { + return m + } + return sub[1] + Mask + sub[3] + }) + s = authHeader.ReplaceAllString(s, "${1}"+Mask) + s = spacedSecretFlag.ReplaceAllString(s, "${1}${2}"+Mask) + s = loginPassword.ReplaceAllString(s, "${1}${2}"+Mask) + s = mysqlPassword.ReplaceAllString(s, "${1}${2}"+Mask) s = sensitiveAssignment.ReplaceAllString(s, "${1}${2}"+Mask) s = pemBlock.ReplaceAllString(s, Mask) s = jwtInline.ReplaceAllString(s, Mask) diff --git a/website/docs/deployment/traefik.md b/website/docs/deployment/traefik.md index 203f9393..4dc3b797 100644 --- a/website/docs/deployment/traefik.md +++ b/website/docs/deployment/traefik.md @@ -13,7 +13,7 @@ The Traefik setup uses Docker Compose and consists of two compose files that are | File | Purpose | |---|---| | `compose.yml` | Core Hades services (API, Scheduler, NATS) | -| `docker-compose.deploy.yml` | Adds Traefik reverse proxy with TLS and removes host port bindings | +| `docker-compose.deploy.yml` | Adds Traefik reverse proxy with TLS and removes the host port bindings of every service except Traefik | Traefik handles: - HTTP → HTTPS redirect (port 80 → 443) @@ -44,11 +44,14 @@ cp .env.example .env Edit `.env` and set the following values: ```env -# Your email address — Let's Encrypt will send certificate expiry notices here +# Your email address - Let's Encrypt will send certificate expiry notices here LETSENCRYPT_EMAIL=you@example.com # The public domain name for the Hades API HADES_API_HOST=hades.example.com + +# Required: Basic Auth key for POST /build (user "hades"); compose refuses to start without it +AUTH_KEY= ``` ### 3. Create the Traefik ACME Storage File @@ -73,10 +76,13 @@ docker compose -f compose.yml -f docker-compose.deploy.yml up -d ``` This starts: -- **Traefik** on ports 80 and 443 — handles TLS termination and routing -- **hadesAPI** — exposed only internally (no host port binding) -- **hadesScheduler** — connected to the Docker socket -- **nats** — internal only +- **Traefik** on ports 80 and 443 - handles TLS termination and routing +- **hadesAPI** - reachable only through Traefik (no host port binding) +- **hadesScheduler** - connected to the Docker socket +- **nats** - internal only (no host port binding; it has no authentication, so never publish it) +- **hadesLogManager** and **jaeger** - internal only, not routed through Traefik + +The log manager has no authentication of its own and is not exposed. The dashboard reads logs through its authenticated proxy. ## How It Works @@ -114,7 +120,7 @@ certificatesResolvers: ``` Key points: -- `exposedByDefault: false` — only services with the `traefik.enable=true` label are exposed. +- `exposedByDefault: false` - only services with the `traefik.enable=true` label are exposed. - The `web` entrypoint automatically redirects all HTTP traffic to HTTPS. - The `letsencrypt` resolver uses the **HTTP-01 challenge** over port 80 to verify domain ownership. @@ -147,7 +153,7 @@ labels: docker compose -f compose.yml -f docker-compose.deploy.yml ps ``` -All services should show `running`. Traefik may take 30–60 seconds on first start to issue a certificate. +All services should show `running`. Traefik may take 30-60 seconds on first start to issue a certificate. ### Test HTTPS @@ -159,6 +165,7 @@ Or submit a test job: ```bash curl -X POST https://hades.example.com/build \ + -u "hades:$AUTH_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Traefik Test", diff --git a/website/docs/installation/docker-mode.md b/website/docs/installation/docker-mode.md index 483995f0..50354f70 100644 --- a/website/docs/installation/docker-mode.md +++ b/website/docs/installation/docker-mode.md @@ -28,14 +28,16 @@ Copy the example environment file and review the defaults: cp .env.example .env ``` -The default configuration uses `docker` as the executor, so no further changes are required for local testing. The most common variables: +The default configuration uses `docker` as the executor. You must set `AUTH_KEY` in `.env` (for example `openssl rand -hex 32`): `compose.yml` refuses to start without it and passes it to the API. + +The most common variables: | Variable | Default | Description | |---|---|---| | `HADES_EXECUTOR` | `docker` | Execution backend. Use `docker` for local mode. | | `CONCURRENCY` | `1` | Number of jobs processed concurrently. The bundled `compose.yml` sets this to `4` unless overridden. | | `API_PORT` | `8080` | Port the Hades API listens on. | -| `AUTH_KEY` | *(empty)* | HTTP Basic Auth key for `/build` (empty = no auth). | +| `AUTH_KEY` | *(required)* | HTTP Basic Auth key for `/build` (user `hades`). The API refuses to start without it unless `ALLOW_UNAUTHENTICATED=true`. | See the [Configuration Reference](../reference/configuration) for the full list. @@ -52,9 +54,9 @@ make docker-run # equivalent to: docker compose -f compose.yml up -d ``` With `make docker-run` this starts: -- **hadesAPI** on port `8081` (mapped from internal `8080`) +- **hadesAPI** on `127.0.0.1:8081` (mapped from internal `8080`, reachable from the host only) - **hadesScheduler** connected to the Docker socket -- **nats** on ports `4222` (client) and `8222` (monitoring) +- **nats** on `127.0.0.1` ports `4222` (client) and `8222` (monitoring) :::note `make run` also starts the **Log Manager** (port `8081`) for inspecting job status and logs. The Log Manager is not part of the Docker Compose stack. @@ -73,6 +75,7 @@ You can also open the NATS monitoring dashboard at [http://localhost:8222](http: ```bash curl -X POST http://localhost:8081/build \ + -u "hades:" \ -H "Content-Type: application/json" \ -d '{ "name": "Hello World", diff --git a/website/docs/operation-modes/k8s-operator.md b/website/docs/operation-modes/k8s-operator.md index 3e718adc..23c045fe 100644 --- a/website/docs/operation-modes/k8s-operator.md +++ b/website/docs/operation-modes/k8s-operator.md @@ -26,7 +26,7 @@ The Operator introduces a `BuildJob` Custom Resource Definition (CRD). When the reason recorded in the status and surfaced in the dashboard, and its `Job` is deleted so it stops retrying. 3. Publishes status transitions and logs back to NATS. -4. Cleans up completed resources (configurable via `DeleteOnComplete`). +4. Cleans up completed resources (`DeleteOnComplete`, default `true`). ``` NATS ──▶ Scheduler ──▶ BuildJob (CRD) ──▶ Operator Controller ──▶ Job / Pods @@ -47,6 +47,12 @@ hadesScheduler: The scheduler uses a dynamic Kubernetes client to create `BuildJob` custom resources; the Operator does the rest. Authentication uses the pod's in-cluster `ServiceAccount` - no `kubeconfig` file is needed. +## Job pod hardening + +Job pods do not mount a service account token, use the `RuntimeDefault` seccomp profile and set `allowPrivilegeEscalation: false`. They still run as the image's user. The API and the log manager have their own service accounts, and the operator role has no access to secrets or configmaps. + +Set `networkPolicy.enabled: true` (default `false`) to add two NetworkPolicies: job pods cannot reach the release namespace, and only Hades components reach NATS on 4222. Before enabling, check the NATS pod labels. On CNIs that match pod IPs against `ipBlock`, set `networkPolicy.jobPods.blockedCIDRs` to the pod CIDR. + ## RBAC The Helm chart automatically creates the `ServiceAccount`, `Role`, and `RoleBinding` that grant the scheduler and operator the minimum permissions required to manage `BuildJob` resources and Jobs within the release namespace. diff --git a/website/docs/reference/api.md b/website/docs/reference/api.md index 140d15ff..be8b7e9b 100644 --- a/website/docs/reference/api.md +++ b/website/docs/reference/api.md @@ -19,7 +19,7 @@ The raw specs are also available as JSON: [`/openapi/hades-api.json`](/openapi/h | Method | Path | Auth | Description | |---|---|---|---| | `GET` | `/ping` | none | Liveness probe. | -| `POST` | `/build` | Basic (when `AUTH_KEY` set) | Validate and enqueue a job. Returns `{ "message", "job_id" }`. | +| `POST` | `/build` | Basic | Validate and enqueue a job. Returns `{ "message", "job_id" }`. | See [Submitting Jobs](../usage/submitting-jobs) for the payload schema and examples, and the [interactive reference](/api/hades) for the full contract. diff --git a/website/docs/reference/configuration.md b/website/docs/reference/configuration.md index 32ca9309..91cff673 100644 --- a/website/docs/reference/configuration.md +++ b/website/docs/reference/configuration.md @@ -51,7 +51,9 @@ Hades measures how much overhead it adds around a job, per step and per phase, c | -------- | ------- | ----------- | | `API_PORT` | `8080` | Port the HTTP API listens on. | | `METRICS_PORT` | `8082` | Port the Prometheus `/metrics` endpoint listens on. | -| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Empty disables auth. | +| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Required: the API exits at start when it is empty unless `ALLOW_UNAUTHENTICATED=true`. | +| `ALLOW_UNAUTHENTICATED` | `false` | Serve `/build` without auth when `AUTH_KEY` is empty. Local development only. | +| `DASHBOARD_TRUSTED_PROXIES` | | Proxy addresses or CIDRs whose `X-Forwarded-For` is trusted for the dashboard login lockout (5 attempts per client address per 15 minutes). Set it behind a reverse proxy. | :::note Reserved (not yet implemented) `PROMETHEUS_ADDRESS`, `RETENTION_IN_MIN`, `MAX_RETRIES`, and `TIMEOUT_IN_MIN` appear in `.env.example` but are **not read by any component today**. They are placeholders for planned features. Prometheus metrics are served on `METRICS_PORT`, not `PROMETHEUS_ADDRESS`. @@ -80,6 +82,9 @@ Depending on `HADES_EXECUTOR`, the Docker or Kubernetes variables below also app | `DOCKER_SCRIPT_EXECUTOR` | `/bin/bash -c` | Shell used to run each step's `script`. | | `DOCKER_CPU_LIMIT` | | Default CPU limit (whole CPUs) when a step sets none. | | `DOCKER_MEMORY_LIMIT` | | Default memory limit (e.g. `4g`) when a step sets none. | +| `DOCKER_ALLOWED_NETWORKS` | `none,bridge,default` | Network modes a step may request. `host` and named networks fail the job unless listed. Never list the network that carries NATS. | +| `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` | `0` | Job timeout when `timeout_seconds` is 0. `0` = off. Docker executor only. | +| `HADES_MAX_JOB_TIMEOUT_SECONDS` | `0` | Upper bound on every job's timeout. `0` = off. Docker executor only. | ### Kubernetes executor (`HADES_EXECUTOR=k8s`) @@ -129,6 +134,10 @@ Deployed by the Helm chart (`hades-log-manager`); also run locally via `make run | `STATUS_WEBHOOK_CONCURRENCY` | `16` | Deliveries in flight at once; keeps a dead receiver from delaying other jobs. | | `STATUS_WEBHOOK_MAX_PENDING` | `1000` | Status events awaiting acknowledgement (in flight or backing off). | +| `CALLBACK_ALLOWED_HOSTS` | | Host names, IPs or CIDRs exempt from the callback address block (internal receivers such as Artemis in the same cluster or compose network). A host name entry must equal the host in the callback URL and trusts whatever it resolves to. | + +Callbacks to loopback, private (RFC 1918, `fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), unspecified and multicast addresses are refused at connect time unless allowed by `CALLBACK_ALLOWED_HOSTS`. The log callback does not follow redirects, HTTP proxy environment variables are ignored, and a blocked status webhook is logged as an error and not retried. + Both outbound pushes are configured per job, not globally: - `callback_url` - an absolute `http`/`https` URL with a host. The Log Manager forwards that job's aggregated **logs** there once the log stream has drained. If omitted, the job's logs are not forwarded. diff --git a/website/docs/usage/submitting-jobs.md b/website/docs/usage/submitting-jobs.md index 30c665c7..954fa19b 100644 --- a/website/docs/usage/submitting-jobs.md +++ b/website/docs/usage/submitting-jobs.md @@ -18,7 +18,7 @@ http://localhost:8081 ## Authentication -If the API is started with a non-empty `AUTH_KEY`, `POST /build` requires HTTP Basic Auth with username `hades` and the key as the password. If `AUTH_KEY` is empty, the endpoint is open. +`POST /build` requires HTTP Basic Auth with username `hades` and `AUTH_KEY` as the password. `AUTH_KEY` is required: with an empty key the API exits at start unless `ALLOW_UNAUTHENTICATED=true` (local development only). Request bodies are capped at 1 MiB (413 above) and duplicate step ids get 400. ## Job Payload Structure From 0de366d92b77ba8036bca987557d5725e001fc8d Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Fri, 13 Nov 2026 23:51:49 +0100 Subject: [PATCH 2/9] Address review feedback and fix chart validation - Helm: pass hadesApi.authKeySecret in the chart validation job and in every documented install, upgrade and template command - Dashboard: refuse login attempts that cannot be counted when the lockout map is full - Redaction: mask quoted flag values up to the closing quote - Scheduler: log a step after the job metadata is merged so its values are scrubbed from the script - Ansible: keep NATS credentials in a mounted config file, not in the process arguments; note that host firewalls do not filter published Docker ports - Docs: authenticated curl examples, callback allowlist hint in Bruno, NetworkPolicy scope with clusterWide Co-Authored-By: Claude Mythos 5.2 --- .github/workflows/release-chart.yml | 2 +- AGENTS.md | 4 ++-- HadesAPI/dashboard/auth.go | 8 ++++---- HadesAPI/dashboard/dashboard_test.go | 3 +++ HadesScheduler/HadesOperator/Readme.md | 5 ++++- HadesScheduler/docker/job.go | 3 +-- Readme.md | 12 +++++++++--- ansible/hades/README.md | 2 +- ansible/hades/tasks/install_api.yml | 9 +++++++++ .../hades/templates/docker-compose-api.yml.j2 | 4 +++- ansible/hades/templates/nats-auth.conf.j2 | 6 ++++++ bruno/api/Create Build Job (Test Fail).bru | 2 +- bruno/api/Create Build Job (Test Succeed).bru | 2 +- helm/hades/Readme.md | 19 +++++++++++++------ helm/hades/Readme.md.gotmpl | 17 ++++++++++++----- helm/hades/values.yaml | 2 ++ shared/redact/redact.go | 14 +++++++------- shared/redact/redact_test.go | 4 ++++ website/docs/deployment/helm-values.md | 17 ++++++++++++----- website/docs/deployment/helm.md | 6 ++++-- website/docs/deployment/traefik.md | 1 + website/docs/installation/helm-chart.md | 16 +++++++++++++--- website/docs/usage/submitting-jobs.md | 3 +++ 23 files changed, 116 insertions(+), 45 deletions(-) create mode 100644 ansible/hades/templates/nats-auth.conf.j2 diff --git a/.github/workflows/release-chart.yml b/.github/workflows/release-chart.yml index 6eca6213..8551324b 100644 --- a/.github/workflows/release-chart.yml +++ b/.github/workflows/release-chart.yml @@ -43,7 +43,7 @@ jobs: helm repo add nats https://nats-io.github.io/k8s/helm/charts/ helm dependency build helm/hades helm lint helm/hades - helm template hades helm/hades > /dev/null + helm template hades helm/hades --set hadesApi.authKeySecret=hades-auth > /dev/null # Publishes only on pushes to main (and manual dispatch), never on PRs. release: diff --git a/AGENTS.md b/AGENTS.md index 4e273591..b4d33248 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ make -C HadesScheduler/HadesOperator manifests generate fmt vet test # helm chart helm dependency build ./helm/hades helm lint ./helm/hades -helm template hades ./helm/hades -n hades +helm template hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth # regenerate OpenAPI specs after changing a handler annotation or a request/response DTO make docs-api @@ -67,7 +67,7 @@ The top-level `Makefile` wraps the common workflows (`make help` lists every tar - **CLI mode:** `make run` runs `HadesAPI`, `HadesScheduler`, and `HadesLogManager` via `go run` and auto-starts NATS in Docker. - **Docker mode:** `make docker-run` brings up `hadesAPI` (8081→8080), `hadesScheduler` (docker executor), and `nats` via `compose.yml`. API requests: see `bruno/api/*.bru`. -- **K8s mode:** `helm upgrade --install hades ./helm/hades -n hades --create-namespace`. The chart deploys API, scheduler, operator, log manager, and embedded NATS JetStream. +- **K8s mode:** `helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth` (create that Secret with key `AUTH_KEY` first; `--set hadesApi.allowUnauthenticated=true` is for throwaway installs only). The chart deploys API, scheduler, operator, log manager, and embedded NATS JetStream. ## Conventions diff --git a/HadesAPI/dashboard/auth.go b/HadesAPI/dashboard/auth.go index 12b55abb..0006d8e0 100644 --- a/HadesAPI/dashboard/auth.go +++ b/HadesAPI/dashboard/auth.go @@ -103,10 +103,10 @@ func (a *authenticator) reserve(key string) bool { fs := a.failures[key] if fs == nil { if len(a.failures) >= maxFailureEntries { - // Hard cap reached (all entries still within the window): refuse to - // track new keys rather than grow unbounded. Existing locked-out - // keys keep their state. - return true + // Hard cap reached (all entries still within the window): an attempt + // that cannot be counted is refused, otherwise new keys would get + // unlimited guesses. + return false } fs = &failureState{} a.failures[key] = fs diff --git a/HadesAPI/dashboard/dashboard_test.go b/HadesAPI/dashboard/dashboard_test.go index effe47c8..d2fcef52 100644 --- a/HadesAPI/dashboard/dashboard_test.go +++ b/HadesAPI/dashboard/dashboard_test.go @@ -439,6 +439,9 @@ func TestLockoutMapBounded(t *testing.T) { if n > maxFailureEntries { t.Fatalf("failures map grew to %d, cap is %d", n, maxFailureEntries) } + if s.auth.reserve("one-key-too-many") { + t.Fatal("a full map must refuse attempts it cannot count") + } } func TestJobLogsRejectsNonUUID(t *testing.T) { diff --git a/HadesScheduler/HadesOperator/Readme.md b/HadesScheduler/HadesOperator/Readme.md index 0efa1e92..36c2c4f5 100644 --- a/HadesScheduler/HadesOperator/Readme.md +++ b/HadesScheduler/HadesOperator/Readme.md @@ -33,8 +33,11 @@ The Hades Operator is designed to: - (Optional) If you want to have the operator to have the Cluster wide access, modify `hadesOperator.clusterWide` top be `true` ```sh - helm upgrade --install hades ./helm/hades -n hades --create-namespace + kubectl create namespace hades + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth ``` +> **NOTE**: `/build` then needs Basic Auth with user `hades` and that key. `--set hadesApi.allowUnauthenticated=true` is for throwaway test installs only. > **NOTE**: These commands will install the project within the hades namespace. **Create instances of your solution** diff --git a/HadesScheduler/docker/job.go b/HadesScheduler/docker/job.go index e027b5af..b9e1ecde 100644 --- a/HadesScheduler/docker/job.go +++ b/HadesScheduler/docker/job.go @@ -70,8 +70,6 @@ func (d Job) execute(ctx context.Context) error { }() for _, step := range d.Steps { - logStep(d.logger, "Executing step", step) - // Copy the global envs and add the step specific ones var envs = make(map[string]string) maps.Copy(envs, d.Metadata) @@ -79,6 +77,7 @@ func (d Job) execute(ctx context.Context) error { envs["UUID"] = d.ID.String() envs["JOB_NAME"] = d.Name step.Metadata = envs + logStep(d.logger, "Executing step", step) dockerStep := Step{ cli: d.cli, diff --git a/Readme.md b/Readme.md index 5fd0e6e6..6e1d862c 100644 --- a/Readme.md +++ b/Readme.md @@ -108,10 +108,16 @@ For production deployments, Hades is designed to run natively within a Kubernete Replace `` in the commands below with the chart version you want to install (for example `1.0.0`). + Create the API auth Secret first. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. + ```fish + kubectl create namespace hades + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + # Install the published chart from GHCR (recommended) helm upgrade --install hades oci://ghcr.io/hades-scheduler/charts/hades \ - --version -n hades --create-namespace + --version -n hades --create-namespace \ + --set hadesApi.authKeySecret=hades-auth ``` Or install from a local checkout of this repository: @@ -119,7 +125,7 @@ For production deployments, Hades is designed to run natively within a Kubernete ```fish helm repo add nats https://nats-io.github.io/k8s/helm/charts helm dependency build ./helm/hades/ - helm upgrade --install hades ./helm/hades -n hades --create-namespace + helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth ``` > **Note:** Helm does not upgrade CRDs after the first install. When a release @@ -508,7 +514,7 @@ Checklist when cutting a chart release: the updated `Chart.lock`. CI vendors the subchart at package time; the `charts/` directory itself is not committed. 5. **Validate before merging**: `helm lint helm/hades`, - `helm template helm/hades`, and ideally a throwaway + `helm template helm/hades --set hadesApi.authKeySecret=hades-auth`, and ideally a throwaway `helm install` in a scratch namespace. 6. **Package visibility**: the GHCR chart package must be **public** for anonymous `helm install` (same as the container images). Set once in the org diff --git a/ansible/hades/README.md b/ansible/hades/README.md index 3d188f77..c0f3e78a 100644 --- a/ansible/hades/README.md +++ b/ansible/hades/README.md @@ -22,7 +22,7 @@ All variables and their defaults are defined in [`defaults/main.yml`](defaults/m | `hades_api_certificate_fullchain_path` | `""` | Path to the TLS fullchain certificate on the API host (enables HTTPS via nginx). | | `hades_api_certificate_key_path` | `""` | Path to the TLS private key on the API host. | | `hades_nats_url` | `nats://localhost:4222` | NATS URL both components connect to. | -| `hades_nats_username` / `hades_nats_password` | `""` | NATS credentials. **Required on `api` nodes**: the NATS server there enforces them, and scheduler nodes must use the same values. Port `4222` is published on the API node for the scheduler hosts (restrict it with a firewall to those hosts); the monitor port `8222` is bound to `127.0.0.1`. | +| `hades_nats_username` / `hades_nats_password` | `""` | NATS credentials. **Required on `api` nodes**: the NATS server there enforces them, and scheduler nodes must use the same values. Port `4222` is published on the API node for the scheduler hosts (restrict it to those hosts with an upstream firewall or `DOCKER-USER` iptables rules; plain host firewall rules such as UFW do not filter ports that Docker publishes); the monitor port `8222` is bound to `127.0.0.1`. | | `hades_nats_tls_enabled` | `false` | Enable TLS for the NATS connection. | | `hades_scheduler_concurrency` | `1` | Number of jobs the scheduler runs concurrently. | | `hades_executor` | `docker` | Scheduler executor (`docker` or `k8s`). | diff --git a/ansible/hades/tasks/install_api.yml b/ansible/hades/tasks/install_api.yml index 6a0a3d1a..f08d03d8 100644 --- a/ansible/hades/tasks/install_api.yml +++ b/ansible/hades/tasks/install_api.yml @@ -15,6 +15,15 @@ mode: "0600" notify: Restart hades api +- name: Add nats auth config + become: true + no_log: true + ansible.builtin.template: + src: nats-auth.conf.j2 + dest: /opt/hades/api/nats-auth.conf + mode: "0600" + notify: Restart hades api + - name: Add hades api docker compose become: true no_log: true diff --git a/ansible/hades/templates/docker-compose-api.yml.j2 b/ansible/hades/templates/docker-compose-api.yml.j2 index 1ff61ecc..d089695f 100644 --- a/ansible/hades/templates/docker-compose-api.yml.j2 +++ b/ansible/hades/templates/docker-compose-api.yml.j2 @@ -20,7 +20,9 @@ services: nats: image: nats:2.11.4 - command: ["-js", "-m", "8222", "--user", {{ hades_nats_username | to_json }}, "--pass", {{ hades_nats_password | to_json }}] + command: ["-js", "-m", "8222", "-c", "/etc/nats/auth.conf"] + volumes: + - ./nats-auth.conf:/etc/nats/auth.conf:ro ports: - "4222:4222" - "127.0.0.1:8222:8222" diff --git a/ansible/hades/templates/nats-auth.conf.j2 b/ansible/hades/templates/nats-auth.conf.j2 new file mode 100644 index 00000000..aa5fad46 --- /dev/null +++ b/ansible/hades/templates/nats-auth.conf.j2 @@ -0,0 +1,6 @@ +# Process arguments are readable by every user on the host, so the NATS +# credentials live here and not on the command line. +authorization { + user: {{ hades_nats_username | to_json }} + password: {{ hades_nats_password | to_json }} +} diff --git a/bruno/api/Create Build Job (Test Fail).bru b/bruno/api/Create Build Job (Test Fail).bru index a2199369..6466aa66 100644 --- a/bruno/api/Create Build Job (Test Fail).bru +++ b/bruno/api/Create Build Job (Test Fail).bru @@ -29,7 +29,7 @@ body:json { }, "timestamp": "2021-01-01T00:00:00.000Z", "priority": 3, // optional, default 3 - "callback_url": "http://localhost:8082/adapter/logs", // optional; where the Log Manager forwards aggregated logs. Omit to skip forwarding. + "callback_url": "http://localhost:8082/adapter/logs", // optional; where the Log Manager forwards aggregated logs. Omit to skip forwarding. Local targets need CALLBACK_ALLOWED_HOSTS=localhost on the Log Manager. "steps": [ { "id": 1, // mandatory to declare the order of execution diff --git a/bruno/api/Create Build Job (Test Succeed).bru b/bruno/api/Create Build Job (Test Succeed).bru index b242c01f..0494dea9 100644 --- a/bruno/api/Create Build Job (Test Succeed).bru +++ b/bruno/api/Create Build Job (Test Succeed).bru @@ -29,7 +29,7 @@ body:json { }, "timestamp": "2021-01-01T00:00:00.000Z", "priority": 1, // optional, default 3 - "callback_url": "http://localhost:8082/adapter/logs", // optional; where the Log Manager forwards aggregated logs. Omit to skip forwarding. + "callback_url": "http://localhost:8082/adapter/logs", // optional; where the Log Manager forwards aggregated logs. Omit to skip forwarding. Local targets need CALLBACK_ALLOWED_HOSTS=localhost on the Log Manager. "status_callback_url": "http://localhost:8082/adapter/job-status", // optional; receives the job-status webhook on the terminal status. Independent of callback_url. "steps": [ { diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index 49eea8cc..30d54f6d 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -57,14 +57,21 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, cat ./helm/hades/values.yaml ``` -3. Install the chart (default: operator mode) +3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. ```bash - helm upgrade --install hades ./helm/hades -n hades --create-namespace + kubectl create namespace hades + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + ``` + +4. Install the chart (default: operator mode) + ```bash + helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth ``` or if you prefer to use the `--set` flag to override values directly in the command line, you can do so like this: ```bash helm upgrade --install hades ./helm/hades -n hades --create-namespace \ + --set hadesApi.authKeySecret=hades-auth \ --set ingress.host=hades.example.com \ --set ingress.tls.secretName=my-secret ``` @@ -75,7 +82,7 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, > >The second "hades" after -n is the Kubernetes namespace where the resources will be deployed. This namespace will be created automatically if it does not exist using --create-namespace -4. Tail the Scheduler logs to verify connectivity +5. Tail the Scheduler logs to verify connectivity ```bash kubectl -n hades logs deploy/hades-scheduler -f ``` @@ -164,7 +171,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | nats.config.jetstream.memoryStore.maxSize | string | `"1Gi"` | Maximum in-memory JetStream store size. | | nats.host | string | `"hades-nats.hades.svc.cluster.local"` | In-cluster NATS service hostname. | | nats.port | int | `4222` | NATS client port. | -| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. | +| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | | networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | | networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach, on top of every pod in the release namespace. Set this to the cluster pod CIDR on CNIs that match pod IPs against ipBlock rules, otherwise the "outside" rule can still reach pods. | | networkPolicy.nats.podLabels | object | `{}` | Labels of the NATS pods. Defaults match the bundled nats chart (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). | @@ -194,7 +201,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc ```bash # Upgrade in place -helm upgrade hades ./helm/hades -n hades +helm upgrade hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth ``` > **CRDs are not upgraded by Helm.** When a release changes the `BuildJob` CRD, @@ -228,5 +235,5 @@ kubectl delete namespace hades helm lint ./helm/hades # Render templates without deploying -helm template hades ./helm/hades -n hades +helm template hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth ``` diff --git a/helm/hades/Readme.md.gotmpl b/helm/hades/Readme.md.gotmpl index 2505d1ce..c91b5f04 100644 --- a/helm/hades/Readme.md.gotmpl +++ b/helm/hades/Readme.md.gotmpl @@ -57,14 +57,21 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, cat ./helm/hades/values.yaml ``` -3. Install the chart (default: operator mode) +3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. ```bash - helm upgrade --install hades ./helm/hades -n hades --create-namespace + kubectl create namespace hades + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + ``` + +4. Install the chart (default: operator mode) + ```bash + helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth ``` or if you prefer to use the `--set` flag to override values directly in the command line, you can do so like this: ```bash helm upgrade --install hades ./helm/hades -n hades --create-namespace \ + --set hadesApi.authKeySecret=hades-auth \ --set ingress.host=hades.example.com \ --set ingress.tls.secretName=my-secret ``` @@ -75,7 +82,7 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, > >The second "hades" after -n is the Kubernetes namespace where the resources will be deployed. This namespace will be created automatically if it does not exist using --create-namespace -4. Tail the Scheduler logs to verify connectivity +5. Tail the Scheduler logs to verify connectivity ```bash kubectl -n hades logs deploy/hades-scheduler -f ``` @@ -107,7 +114,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc ```bash # Upgrade in place -helm upgrade hades ./helm/hades -n hades +helm upgrade hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth ``` > **CRDs are not upgraded by Helm.** When a release changes the `BuildJob` CRD, @@ -141,5 +148,5 @@ kubectl delete namespace hades helm lint ./helm/hades # Render templates without deploying -helm template hades ./helm/hades -n hades +helm template hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth ``` diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index d689435f..be3ec04a 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -332,6 +332,8 @@ tracing: # -- NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by # default because the NATS policy assumes the labels of the bundled NATS chart. +# The job-pod policy covers the release namespace only: with +# `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. networkPolicy: # -- Render the NetworkPolicies below. Verify the NATS pod labels first # (`kubectl get pods -n --show-labels`), then enable. diff --git a/shared/redact/redact.go b/shared/redact/redact.go index aaad8377..9d02a3dd 100644 --- a/shared/redact/redact.go +++ b/shared/redact/redact.go @@ -173,14 +173,14 @@ var ( // urlUserToken matches a token-only URL credential (scheme://token@host). urlUserToken = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9+.\-]*://)([A-Za-z0-9_.\-]{12,})(@)`) // spacedSecretFlag matches --password/--token style flags whose value is the - // next word; "--password-stdin" does not match because the flag must end + // next word or quoted string; "--password-stdin" does not match because the flag must end // with the keyword. - spacedSecretFlag = regexp.MustCompile(`(?i)(\s--[\w\-]*?(?:token|secret|passwd|password|pwd|api[_-]?key|credential)\s+)(["']?)([^\s"';\-][^\s"';]*)`) + spacedSecretFlag = regexp.MustCompile(`(?i)(\s--[\w\-]*?(?:token|secret|passwd|password|pwd|api[_-]?key|credential)\s+)((?:"[^"]*"|'[^']*'|[^\s"';\-][^\s"';]*))`) // loginPassword matches -p only on a registry login command line, // where -p is a password; elsewhere it is a path or port flag. - loginPassword = regexp.MustCompile(`(\b(?:docker|podman|buildah|skopeo|crane|oras)\b[^\n;&|]*?\blogin\b[^\n;&|]*?\s-p\s+)(["']?)([^\s"';]+)`) + loginPassword = regexp.MustCompile(`(\b(?:docker|podman|buildah|skopeo|crane|oras)\b[^\n;&|]*?\blogin\b[^\n;&|]*?\s-p\s+)((?:"[^"]*"|'[^']*'|[^\s"';]+))`) // mysqlPassword matches the attached -p form of the mysql tools. - mysqlPassword = regexp.MustCompile(`(\bmysql(?:dump|admin)?\b[^\n;&|]*?\s-p)(["']?)([^\s"';]+)`) + mysqlPassword = regexp.MustCompile(`(\bmysql(?:dump|admin)?\b[^\n;&|]*?\s-p)((?:"[^"]*"|'[^']*'|[^\s"';]+))`) // urlCredentials captures the password segment of a scheme://user:pass@host URL. urlCredentials = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9+.\-]*://[^/\s:@]+:)([^/\s@]+)(@)`) // sensitiveAssignment matches KEY=VALUE / KEY: VALUE / export KEY=VALUE where @@ -212,9 +212,9 @@ func (r *Redactor) Text(s string) string { return sub[1] + Mask + sub[3] }) s = authHeader.ReplaceAllString(s, "${1}"+Mask) - s = spacedSecretFlag.ReplaceAllString(s, "${1}${2}"+Mask) - s = loginPassword.ReplaceAllString(s, "${1}${2}"+Mask) - s = mysqlPassword.ReplaceAllString(s, "${1}${2}"+Mask) + s = spacedSecretFlag.ReplaceAllString(s, "${1}"+Mask) + s = loginPassword.ReplaceAllString(s, "${1}"+Mask) + s = mysqlPassword.ReplaceAllString(s, "${1}"+Mask) s = sensitiveAssignment.ReplaceAllString(s, "${1}${2}"+Mask) s = pemBlock.ReplaceAllString(s, Mask) s = jwtInline.ReplaceAllString(s, Mask) diff --git a/shared/redact/redact_test.go b/shared/redact/redact_test.go index 07242a9d..db016129 100644 --- a/shared/redact/redact_test.go +++ b/shared/redact/redact_test.go @@ -163,6 +163,10 @@ func TestText_VariousSecretForms(t *testing.T) { {"curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.dBjftJeZ4CVPmB92K27uhbUJU1p1r'", "dBjftJeZ4CVPmB92K27uhbUJU1p1r"}, {"echo ZmFrZS1oaWdoLWVudHJvcHktdG9rZW4tMTIzNDU2Nzg5MA > /tmp/x", "ZmFrZS1oaWdoLWVudHJvcHktdG9rZW4tMTIzNDU2Nzg5MA"}, {"psql postgres://admin:s3cr3t@db:5432/app", "s3cr3t"}, + // a quoted value is masked up to its closing quote + {"docker login -u ci -p 'alpha beta' registry.example", "beta"}, + {"mytool --password \"alpha beta\" run", "beta"}, + {"mysql -p'alpha beta' -e 'select 1'", "beta"}, } for _, c := range cases { out := r.Text(c.in) diff --git a/website/docs/deployment/helm-values.md b/website/docs/deployment/helm-values.md index b5e3aa82..33f27711 100644 --- a/website/docs/deployment/helm-values.md +++ b/website/docs/deployment/helm-values.md @@ -63,14 +63,21 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, cat ./helm/hades/values.yaml ``` -3. Install the chart (default: operator mode) +3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. ```bash - helm upgrade --install hades ./helm/hades -n hades --create-namespace + kubectl create namespace hades + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + ``` + +4. Install the chart (default: operator mode) + ```bash + helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth ``` or if you prefer to use the `--set` flag to override values directly in the command line, you can do so like this: ```bash helm upgrade --install hades ./helm/hades -n hades --create-namespace \ + --set hadesApi.authKeySecret=hades-auth \ --set ingress.host=hades.example.com \ --set ingress.tls.secretName=my-secret ``` @@ -81,7 +88,7 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, > >The second "hades" after -n is the Kubernetes namespace where the resources will be deployed. This namespace will be created automatically if it does not exist using --create-namespace -4. Tail the Scheduler logs to verify connectivity +5. Tail the Scheduler logs to verify connectivity ```bash kubectl -n hades logs deploy/hades-scheduler -f ``` @@ -192,7 +199,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc ```bash # Upgrade in place -helm upgrade hades ./helm/hades -n hades +helm upgrade hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth ``` > **CRDs are not upgraded by Helm.** When a release changes the `BuildJob` CRD, @@ -226,5 +233,5 @@ kubectl delete namespace hades helm lint ./helm/hades # Render templates without deploying -helm template hades ./helm/hades -n hades +helm template hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth ``` diff --git a/website/docs/deployment/helm.md b/website/docs/deployment/helm.md index e62a4480..2e9cad53 100644 --- a/website/docs/deployment/helm.md +++ b/website/docs/deployment/helm.md @@ -54,6 +54,7 @@ By default the Operator is scoped to the release namespace. To schedule jobs acr ```bash helm upgrade hades oci://ghcr.io/hades-scheduler/charts/hades --version -n hades \ + --set hadesApi.authKeySecret=hades-auth \ --set hadesOperator.clusterWide=true ``` @@ -62,7 +63,8 @@ This switches the `Role`/`RoleBinding` to a `ClusterRole`/`ClusterRoleBinding`. ## Upgrade & Rollback ```bash -helm upgrade hades oci://ghcr.io/hades-scheduler/charts/hades --version -n hades +helm upgrade hades oci://ghcr.io/hades-scheduler/charts/hades --version -n hades \ + --set hadesApi.authKeySecret=hades-auth helm history hades -n hades helm rollback hades -n hades @@ -79,7 +81,7 @@ kubectl delete namespace hades ```bash helm lint ./helm/hades -helm template hades ./helm/hades -n hades +helm template hades ./helm/hades -n hades --set hadesApi.authKeySecret=hades-auth make docs-helm # regenerate the values table ``` diff --git a/website/docs/deployment/traefik.md b/website/docs/deployment/traefik.md index 4dc3b797..081d7b15 100644 --- a/website/docs/deployment/traefik.md +++ b/website/docs/deployment/traefik.md @@ -164,6 +164,7 @@ curl https://hades.example.com/ping Or submit a test job: ```bash +AUTH_KEY= curl -X POST https://hades.example.com/build \ -u "hades:$AUTH_KEY" \ -H "Content-Type: application/json" \ diff --git a/website/docs/installation/helm-chart.md b/website/docs/installation/helm-chart.md index 501192c8..061c124b 100644 --- a/website/docs/installation/helm-chart.md +++ b/website/docs/installation/helm-chart.md @@ -20,9 +20,17 @@ The chart is published to GHCR as an OCI artifact. In every command below, repla `` with the chart version you want to install (for example `1.0.0`; see the [available versions](https://github.com/Hades-Scheduler/hades/pkgs/container/charts%2Fhades)): +Create the API auth Secret first. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. + +```bash +kubectl create namespace hades +kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= +``` + ```bash helm upgrade --install hades oci://ghcr.io/hades-scheduler/charts/hades \ - --version -n hades --create-namespace + --version -n hades --create-namespace \ + --set hadesApi.authKeySecret=hades-auth ``` ### Option B - From a local checkout @@ -33,7 +41,7 @@ helm repo add nats https://nats-io.github.io/k8s/helm/charts helm dependency build ./helm/hades/ # 2. Install (default: operator mode) -helm upgrade --install hades ./helm/hades -n hades --create-namespace +helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth ``` Override values inline as needed - at minimum set your ingress host: @@ -41,6 +49,7 @@ Override values inline as needed - at minimum set your ingress host: ```bash helm upgrade --install hades oci://ghcr.io/hades-scheduler/charts/hades --version \ -n hades --create-namespace \ + --set hadesApi.authKeySecret=hades-auth \ --set ingress.host=hades.example.com \ --set ingress.tls.secretName=my-tls-secret ``` @@ -76,7 +85,8 @@ helm show crds oci://ghcr.io/hades-scheduler/charts/hades --version | ## Upgrade & Rollback ```bash -helm upgrade hades oci://ghcr.io/hades-scheduler/charts/hades --version -n hades +helm upgrade hades oci://ghcr.io/hades-scheduler/charts/hades --version -n hades \ + --set hadesApi.authKeySecret=hades-auth # Roll back to a previous revision helm history hades -n hades diff --git a/website/docs/usage/submitting-jobs.md b/website/docs/usage/submitting-jobs.md index 954fa19b..7235c6bf 100644 --- a/website/docs/usage/submitting-jobs.md +++ b/website/docs/usage/submitting-jobs.md @@ -57,6 +57,7 @@ http://localhost:8081 ```bash curl -X POST http://localhost:8081/build \ + -u "hades:$AUTH_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Hello World", @@ -78,6 +79,7 @@ Steps run sequentially. Each step runs in its own container but shares a common ```bash curl -X POST http://localhost:8081/build \ + -u "hades:$AUTH_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Multi-Step Example", @@ -97,6 +99,7 @@ react the moment a job ends - the Artemis integration, a code-review bot - set ```bash curl -X POST http://localhost:8081/build \ + -u "hades:$AUTH_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Graded Exercise", From ea352821aa5d67b03572c06d4bf7c1bf45e40fcf Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Sat, 14 Nov 2026 01:09:51 +0100 Subject: [PATCH 3/9] Validate step ids and steps, add job pod defaults on Kubernetes Findings from testing the PR on the test environment: - API: reject jobs without steps, with an empty image, or with a step id outside 1..2147483647. A step id above int32 was stored as a BuildJob the operator could not decode, which stopped it for every job. - CRD: maximum for the step id. - Scheduler (Kubernetes): a job the API server rejects as invalid now gets a terminal Failed status with a reason instead of staying Queued. - Operator: default job timeout (1h), default CPU and memory limits per step (2 / 4Gi), size limit for the shared volume (10Gi), all configurable; job containers drop NET_RAW and MKNOD; service links off. - Test values: enable the NetworkPolicies so job pods cannot reach NATS or the log manager. Co-Authored-By: Claude Mythos 5.2 --- AGENTS.md | 4 +- HadesAPI/router.go | 19 +++ HadesAPI/router_test.go | 32 +++++ .../HadesOperator/api/v1/buildjob_types.go | 1 + HadesScheduler/HadesOperator/cmd/main.go | 44 +++++++ HadesScheduler/HadesOperator/cmd/main_test.go | 14 +++ .../controller/buildjob_build_test.go | 109 ++++++++++++++++-- .../controller/buildjob_controller.go | 94 ++++++++++++--- .../controller/buildjob_events_test.go | 10 +- .../controller/buildjob_statuswrite_test.go | 4 +- HadesScheduler/k8s/k8s.go | 38 +++++- HadesScheduler/k8s/k8s_test.go | 92 +++++++++++++++ Readme.md | 1 + docs/api.md | 4 +- docs/configuration.md | 5 + helm/hades/Readme.md | 5 + .../crds/build.hades.tum.de_buildjobs.yaml | 1 + .../templates/hades-operator-deployment.yaml | 10 ++ helm/hades/values-test.yaml | 9 ++ helm/hades/values.yaml | 18 +++ website/docs/operation-modes/k8s-operator.md | 2 +- website/docs/reference/configuration.md | 5 + website/docs/usage/submitting-jobs.md | 2 +- 23 files changed, 482 insertions(+), 41 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b4d33248..7335794f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ Go workspace (`go.work`, Go 1.27) with five modules: - **Job leases:** the scheduler acks a job only after the executor returns, and the Docker executor blocks for the whole job. `shared/nats/consumer.go` therefore heartbeats `msg.InProgress()` on a ticker while a job runs so the `NATS_ACK_WAIT` (default 1m) timer keeps resetting - job duration must never determine AckWait. `NATS_MAX_DELIVER` (default 3) caps redeliveries; the last delivery is not executed but publishes a terminal `Failed` status and terminates the message. - **Job DTO:** `shared/payload/payload.go` (`QueuePayload` with ordered `Step`s). Job-level: `timeout_seconds` (whole-job timeout). Each step has `image`, optional `script`, `metadata` env vars, `cpu_limit` (whole cores), `memory_limit`, and the Docker-only limits `network`, `memory_swap`, `pids_limit`. - **Two outbound pushes, do not conflate them:** `callback_url` forwards the aggregated **log array** and only after `stopWatchingJobLogs` drains the JetStream log consumer; `status_callback_url` receives the **job-status webhook** (`{event, job_id, name, status, reason, queued_at, started_at, finished_at, duration_ms, attempt}`) on the terminal status itself, with at-least-once retry. Neither may be repurposed as the other. - - **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only** (`network` must be in `DOCKER_ALLOWED_NETWORKS`, default `none,bridge,default`, else the job fails; `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` / `HADES_MAX_JOB_TIMEOUT_SECONDS` also Docker only); the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field). + - **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only** (`network` must be in `DOCKER_ALLOWED_NETWORKS`, default `none,bridge,default`, else the job fails; `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` / `HADES_MAX_JOB_TIMEOUT_SECONDS` also Docker only); the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field). The operator has its own defaults for what a job leaves unset (`JOB_DEFAULT_TIMEOUT_SECONDS` 3600, `JOB_MAX_TIMEOUT_SECONDS` 0, `JOB_DEFAULT_CPU_LIMIT` 2, `JOB_DEFAULT_MEMORY_LIMIT` 4Gi, `JOB_SHARED_VOLUME_SIZE_LIMIT` 10Gi; empty or 0 disables each). - **CRD:** `BuildJob` in `HadesScheduler/HadesOperator/api/v1/buildjob_types.go`. **Important:** if you change `BuildJobSpec`, run `make -C HadesScheduler/HadesOperator manifests generate` and commit `helm/hades/crds/build.hades.tum.de_buildjobs.yaml` and `zz_generated.deepcopy.go` - the `verify-crd.yml` GitHub workflow will fail otherwise. The `BuildJobSpec` is intentionally duplicated from `shared/payload`; keep them in sync manually. ## Build / test @@ -84,7 +84,7 @@ The top-level `Makefile` wraps the common workflows (`make help` lists every tar - `HadesLogManager` is deployed (Helm + image) but ClusterIP-internal and single-replica (in-memory aggregation); don't put it on the ingress or scale it past one replica. The status-webhook dispatcher living there keeps no durable state in memory (its retry schedule is in JetStream), but it still assumes one replica. The same single-replica constraint applies to the API's in-memory dashboard read-model. (It just isn't in the `compose.yml` stack.) - NATS itself still has no authentication; it must not be published beyond loopback (compose binds it to `127.0.0.1`, the deploy stack does not publish it). Never list the NATS network in `DOCKER_ALLOWED_NETWORKS`. - Don't re-open callbacks to internal addresses or add a redirect-following/proxy-aware callback client; use `CALLBACK_ALLOWED_HOSTS`. -- Job pods must keep no service account token, `RuntimeDefault` seccomp and `allowPrivilegeEscalation: false`; the operator role gets no secrets/configmaps access. +- Job pods must keep no service account token, `RuntimeDefault` seccomp, `allowPrivilegeEscalation: false`, `enableServiceLinks: false` and dropped `NET_RAW`/`MKNOD` capabilities (not `ALL`: grading images install packages as root); the operator role gets no secrets/configmaps access. - The dashboard treats all job/step `Metadata` and step `script` bodies as potentially secret: redact via `shared/redact` before returning any payload; never add an endpoint that returns raw metadata or an unscrubbed script. Job logs are proxied verbatim (a known, documented gap). diff --git a/HadesAPI/router.go b/HadesAPI/router.go index d3b29b5c..3c83142a 100644 --- a/HadesAPI/router.go +++ b/HadesAPI/router.go @@ -4,6 +4,7 @@ import ( "errors" "fmt" "log/slog" + "math" "net/http" "os" "reflect" @@ -190,8 +191,26 @@ func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublishe return } + // The BuildJob CRD rejects these, which would leave an accepted job Queued + // forever, so refuse them here. + if len(p.QueuePayload.Steps) == 0 { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, "At least one step is required") + return + } + seenStepIDs := make(map[int]bool, len(p.QueuePayload.Steps)) for _, step := range p.QueuePayload.Steps { + if step.ID < 1 || step.ID > math.MaxInt32 { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, fmt.Sprintf("Invalid step id %d: step ids must be between 1 and %d", step.ID, math.MaxInt32)) + return + } + if strings.TrimSpace(step.Image) == "" { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, fmt.Sprintf("Step %d has an empty image: every step needs an image", step.ID)) + return + } if seenStepIDs[step.ID] { buildRequestsTotal.WithLabelValues("rejected").Inc() c.String(http.StatusBadRequest, fmt.Sprintf("Duplicate step id %d: step ids must be unique", step.ID)) diff --git a/HadesAPI/router_test.go b/HadesAPI/router_test.go index d744fa80..a5716989 100644 --- a/HadesAPI/router_test.go +++ b/HadesAPI/router_test.go @@ -379,6 +379,38 @@ func (suite *APISuite) TestDuplicateStepIDs() { assert.Contains(suite.T(), w.Body.String(), "Duplicate step id 1") } +func (suite *APISuite) TestInvalidStepsRejected() { + cases := []struct { + name string + body string + want string + }{ + {"empty steps", `{"name":"x","steps":[]}`, "At least one step"}, + {"no steps field", `{"name":"x"}`, "At least one step"}, + {"empty image", `{"name":"x","steps":[{"id":1,"image":""}]}`, "empty image"}, + {"blank image", `{"name":"x","steps":[{"id":1,"image":" "}]}`, "empty image"}, + {"id zero", `{"name":"x","steps":[{"id":0,"image":"i"}]}`, "Invalid step id 0"}, + {"id negative", `{"name":"x","steps":[{"id":-1,"image":"i"}]}`, "Invalid step id -1"}, + {"id too large", `{"name":"x","steps":[{"id":9223372036854775807,"image":"i"}]}`, "Invalid step id 9223372036854775807"}, + {"id just above int32", `{"name":"x","steps":[{"id":2147483648,"image":"i"}]}`, "Invalid step id 2147483648"}, + } + for _, tc := range cases { + suite.Run(tc.name, func() { + w := httptest.NewRecorder() + req, _ := http.NewRequest("POST", "/build", strings.NewReader(tc.body)) + req.Header.Set("Content-Type", "application/json") + suite.router.ServeHTTP(w, req) + assert.Equal(suite.T(), 400, w.Code) + assert.Contains(suite.T(), w.Body.String(), tc.want) + }) + } +} + +func (suite *APISuite) TestMaxInt32StepIDAccepted() { + w := suite.postStep(payload.Step{ID: 2147483647, Name: "s", Image: "image1"}, 0) + assert.Equal(suite.T(), 200, w.Code) +} + func (suite *APISuite) TestBuildBodyTooLarge() { w := httptest.NewRecorder() big := `{"name":"example","steps":[{"id":1,"name":"a","image":"i","script":"` + strings.Repeat("x", maxBuildBodyBytes) + `"}]}` diff --git a/HadesScheduler/HadesOperator/api/v1/buildjob_types.go b/HadesScheduler/HadesOperator/api/v1/buildjob_types.go index 8c5e35a0..c8c75c5b 100644 --- a/HadesScheduler/HadesOperator/api/v1/buildjob_types.go +++ b/HadesScheduler/HadesOperator/api/v1/buildjob_types.go @@ -70,6 +70,7 @@ type BuildStep struct { // Steps execute in ascending ID order: step with id=1 runs first, then id=2, etc. // This is mandatory and must be unique within the BuildJob. // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=2147483647 ID int32 `json:"id"` // Descriptive name for this step, shown in logs and UI. diff --git a/HadesScheduler/HadesOperator/cmd/main.go b/HadesScheduler/HadesOperator/cmd/main.go index 2c8b8910..25516c35 100644 --- a/HadesScheduler/HadesOperator/cmd/main.go +++ b/HadesScheduler/HadesOperator/cmd/main.go @@ -19,6 +19,7 @@ package main import ( "context" "flag" + "fmt" "os" "strings" "time" @@ -32,6 +33,7 @@ import ( hadesnats "github.com/hades-scheduler/hades/shared/nats" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" utilruntime "k8s.io/apimachinery/pkg/util/runtime" @@ -79,6 +81,37 @@ type OperatorConfig struct { // if-not-present or never. If empty, Kubernetes pulls :latest and untagged // images on every step. ImagePullPolicy string `env:"K8S_IMAGE_PULL_POLICY"` + // Defaults for what a job leaves unset. Empty or 0 disables each one. + JobDefaultTimeoutSeconds int64 `env:"JOB_DEFAULT_TIMEOUT_SECONDS" envDefault:"3600"` + JobMaxTimeoutSeconds int64 `env:"JOB_MAX_TIMEOUT_SECONDS" envDefault:"0"` + JobDefaultCPULimit string `env:"JOB_DEFAULT_CPU_LIMIT" envDefault:"2"` + JobDefaultMemoryLimit string `env:"JOB_DEFAULT_MEMORY_LIMIT" envDefault:"4Gi"` + JobSharedVolumeSizeLimit string `env:"JOB_SHARED_VOLUME_SIZE_LIMIT" envDefault:"10Gi"` +} + +// parseQuantity parses an optional Kubernetes quantity; empty means unset. +func parseQuantity(name, v string) (*resource.Quantity, error) { + if strings.TrimSpace(v) == "" { + return nil, nil + } + q, err := resource.ParseQuantity(strings.TrimSpace(v)) + if err != nil { + return nil, fmt.Errorf("invalid %s %q: %w", name, v, err) + } + return &q, nil +} + +func (c *OperatorConfig) jobDefaults() (controller.JobDefaults, error) { + d := controller.JobDefaults{TimeoutSeconds: c.JobDefaultTimeoutSeconds, MaxTimeoutSeconds: c.JobMaxTimeoutSeconds} + var err error + if d.CPULimit, err = parseQuantity("JOB_DEFAULT_CPU_LIMIT", c.JobDefaultCPULimit); err != nil { + return d, err + } + if d.MemoryLimit, err = parseQuantity("JOB_DEFAULT_MEMORY_LIMIT", c.JobDefaultMemoryLimit); err != nil { + return d, err + } + d.SharedVolumeSizeLimit, err = parseQuantity("JOB_SHARED_VOLUME_SIZE_LIMIT", c.JobSharedVolumeSizeLimit) + return d, err } // normalize replaces non-positive tuning values with their defaults. A @@ -181,6 +214,11 @@ func main() { setupLog.Error(err, "invalid K8S_IMAGE_PULL_POLICY") os.Exit(1) } + jobDefaults, err := operatorConfig.jobDefaults() + if err != nil { + setupLog.Error(err, "invalid job default") + os.Exit(1) + } if nsConfig.WatchNamespace != "" { setupLog.Info("scoping cache to a single namespace", "namespace", nsConfig.WatchNamespace) @@ -247,6 +285,11 @@ func main() { "log_drain_timeout", operatorConfig.LogDrainTimeout, "finalizer_image", operatorConfig.FinalizerImage, "image_pull_policy", operatorConfig.ImagePullPolicy, + "job_default_timeout_seconds", operatorConfig.JobDefaultTimeoutSeconds, + "job_max_timeout_seconds", operatorConfig.JobMaxTimeoutSeconds, + "job_default_cpu_limit", operatorConfig.JobDefaultCPULimit, + "job_default_memory_limit", operatorConfig.JobDefaultMemoryLimit, + "job_shared_volume_size_limit", operatorConfig.JobSharedVolumeSizeLimit, "metrics_addr", metricsAddr, "dev_mode", enableDevMode, "nats_url", natsConfig.URL, @@ -281,6 +324,7 @@ func main() { RequeueDelay: operatorConfig.RequeueDelay, LogDrainTimeout: operatorConfig.LogDrainTimeout, FinalizerImage: operatorConfig.FinalizerImage, + JobDefaults: jobDefaults, ImagePullPolicy: pullPolicy, Publisher: publisher, LogStreams: controller.NewLogStreamRegistry(), diff --git a/HadesScheduler/HadesOperator/cmd/main_test.go b/HadesScheduler/HadesOperator/cmd/main_test.go index a993ed4c..9f5f3a71 100644 --- a/HadesScheduler/HadesOperator/cmd/main_test.go +++ b/HadesScheduler/HadesOperator/cmd/main_test.go @@ -175,3 +175,17 @@ func TestCacheOptions(t *testing.T) { } } } + +func TestJobDefaults(t *testing.T) { + d, err := (&OperatorConfig{JobDefaultCPULimit: "2", JobDefaultMemoryLimit: "4Gi", JobSharedVolumeSizeLimit: "10Gi"}).jobDefaults() + if err != nil || d.CPULimit == nil || d.MemoryLimit == nil || d.SharedVolumeSizeLimit == nil { + t.Fatalf("jobDefaults() = %+v, %v", d, err) + } + d, err = (&OperatorConfig{}).jobDefaults() + if err != nil || d.CPULimit != nil || d.MemoryLimit != nil || d.SharedVolumeSizeLimit != nil { + t.Errorf("empty config must disable the defaults, got %+v, %v", d, err) + } + if _, err := (&OperatorConfig{JobDefaultMemoryLimit: "lots"}).jobDefaults(); err == nil { + t.Error("an invalid quantity must fail") + } +} diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go index 36cd49a0..17996e11 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go @@ -2,12 +2,14 @@ package controller import ( "fmt" + "slices" "strings" "testing" buildv1 "github.com/hades-scheduler/hades/HadesScheduler/HadesOperator/api/v1" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) @@ -57,7 +59,7 @@ func TestBuildK8sJob_ContinueOnError(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) // Normal step: run the script directly so a failure fails the container. normal := initContainer(t, job, fmt.Sprintf(BuildStepPrefix, 1)) @@ -107,7 +109,7 @@ func TestBuildK8sJob_JobMetadataInjected(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) c := initContainer(t, job, fmt.Sprintf(BuildStepPrefix, 1)) wants := map[string]string{ @@ -147,7 +149,7 @@ func TestBuildK8sJob_NoDuplicateStepScriptEnv(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) c := initContainer(t, job, fmt.Sprintf(BuildStepPrefix, 1)) if n := envCount(c, "HADES_STEP_SCRIPT"); n != 1 { @@ -173,7 +175,7 @@ func TestBuildK8sJob_ActiveDeadlineSeconds(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) if job.Spec.ActiveDeadlineSeconds == nil { t.Fatalf("ActiveDeadlineSeconds not set, want %d", timeout) } @@ -193,7 +195,7 @@ func TestBuildK8sJob_NoTimeoutLeavesDeadlineUnset(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) if job.Spec.ActiveDeadlineSeconds != nil { t.Errorf("ActiveDeadlineSeconds = %v, want nil", *job.Spec.ActiveDeadlineSeconds) } @@ -226,7 +228,7 @@ func TestBuildK8sJob_StepsRunInSharedVolume(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) for _, id := range []int{1, 2} { c := initContainer(t, job, fmt.Sprintf(BuildStepPrefix, id)) @@ -268,7 +270,7 @@ func TestBuildK8sJob_FinalizerImageIsPinnedAndNotAlwaysPulled(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) var found bool for _, c := range job.Spec.Template.Spec.Containers { @@ -331,7 +333,7 @@ func TestBuildK8sJob_HonoursFinalizerImageOverride(t *testing.T) { } const custom = "registry.internal/busybox:1.36" - job := buildK8sJob(bj, "job-1", true, false, custom, "") + job := buildK8sJob(bj, "job-1", true, false, custom, "", JobDefaults{}) for _, c := range job.Spec.Template.Spec.Containers { if c.Name == FinalizerContainerName { @@ -360,7 +362,7 @@ func TestBuildK8sJob_PodHardening(t *testing.T) { }, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) spec := job.Spec.Template.Spec if spec.AutomountServiceAccountToken == nil || *spec.AutomountServiceAccountToken { @@ -376,3 +378,92 @@ func TestBuildK8sJob_PodHardening(t *testing.T) { } } } + +func defaultsTestJob(timeout *int64, s buildv1.BuildStep) *buildv1.BuildJob { + s.ID, s.Image = 1, "alpine" + return &buildv1.BuildJob{ + ObjectMeta: metav1.ObjectMeta{Name: "job-1"}, + Spec: buildv1.BuildJobSpec{TimeoutSeconds: timeout, Steps: []buildv1.BuildStep{s}}, + } +} + +func TestBuildK8sJob_TimeoutDefaults(t *testing.T) { + i := func(v int64) *int64 { return &v } + cases := []struct { + name string + job *int64 + defaults JobDefaults + want *int64 + }{ + {"default applied", nil, JobDefaults{TimeoutSeconds: 3600}, i(3600)}, + {"job timeout wins over default", i(60), JobDefaults{TimeoutSeconds: 3600}, i(60)}, + {"max clamps larger job timeout", i(9000), JobDefaults{TimeoutSeconds: 3600, MaxTimeoutSeconds: 600}, i(600)}, + {"max clamps missing job timeout", nil, JobDefaults{MaxTimeoutSeconds: 600}, i(600)}, + {"smaller job timeout kept under max", i(60), JobDefaults{MaxTimeoutSeconds: 600}, i(60)}, + {"disabled", nil, JobDefaults{}, nil}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := buildK8sJob(defaultsTestJob(tc.job, buildv1.BuildStep{}), "job-1", true, false, DefaultFinalizerImage, "", tc.defaults).Spec.ActiveDeadlineSeconds + if (got == nil) != (tc.want == nil) || (got != nil && *got != *tc.want) { + t.Errorf("ActiveDeadlineSeconds = %v, want %v", got, tc.want) + } + }) + } +} + +func TestBuildK8sJob_ResourceDefaults(t *testing.T) { + cpu, mem := resource.MustParse("2"), resource.MustParse("4Gi") + defaults := JobDefaults{CPULimit: &cpu, MemoryLimit: &mem} + + // Nothing set by the step: both defaults apply and requests follow limits. + c := buildK8sJob(defaultsTestJob(nil, buildv1.BuildStep{}), "job-1", true, false, DefaultFinalizerImage, "", defaults).Spec.Template.Spec.InitContainers[0] + if !c.Resources.Limits.Cpu().Equal(cpu) || !c.Resources.Limits.Memory().Equal(mem) { + t.Errorf("limits = %v, want cpu=2 memory=4Gi", c.Resources.Limits) + } + if !c.Resources.Requests.Cpu().Equal(cpu) || !c.Resources.Requests.Memory().Equal(mem) { + t.Errorf("requests = %v, want to follow limits", c.Resources.Requests) + } + + // The step sets only memory: its value wins, CPU falls back to the default. + stepMem := resource.MustParse("512Mi") + c = buildK8sJob(defaultsTestJob(nil, buildv1.BuildStep{MemoryLimit: &stepMem}), "job-1", true, false, DefaultFinalizerImage, "", defaults).Spec.Template.Spec.InitContainers[0] + if !c.Resources.Limits.Memory().Equal(stepMem) || !c.Resources.Limits.Cpu().Equal(cpu) { + t.Errorf("limits = %v, want memory=512Mi cpu=2", c.Resources.Limits) + } + + // Disabled defaults leave the step unlimited. + c = buildK8sJob(defaultsTestJob(nil, buildv1.BuildStep{}), "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}).Spec.Template.Spec.InitContainers[0] + if len(c.Resources.Limits) != 0 || len(c.Resources.Requests) != 0 { + t.Errorf("resources = %v, want none", c.Resources) + } +} + +func TestBuildK8sJob_SharedVolumeSizeLimit(t *testing.T) { + size := resource.MustParse("10Gi") + vol := func(d JobDefaults) *corev1.EmptyDirVolumeSource { + return buildK8sJob(defaultsTestJob(nil, buildv1.BuildStep{}), "job-1", true, false, DefaultFinalizerImage, "", d).Spec.Template.Spec.Volumes[0].EmptyDir + } + if got := vol(JobDefaults{SharedVolumeSizeLimit: &size}).SizeLimit; got == nil || !got.Equal(size) { + t.Errorf("sizeLimit = %v, want 10Gi", got) + } + if got := vol(JobDefaults{}).SizeLimit; got != nil { + t.Errorf("sizeLimit = %v, want nil when disabled", got) + } +} + +func TestBuildK8sJob_DropsCapabilitiesAndServiceLinks(t *testing.T) { + spec := buildK8sJob(defaultsTestJob(nil, buildv1.BuildStep{}), "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}).Spec.Template.Spec + if spec.EnableServiceLinks == nil || *spec.EnableServiceLinks { + t.Error("enableServiceLinks must be false") + } + for _, c := range append(spec.InitContainers, spec.Containers...) { + caps := c.SecurityContext.Capabilities + if caps == nil || !slices.Contains(caps.Drop, "NET_RAW") || !slices.Contains(caps.Drop, "MKNOD") { + t.Errorf("container %q must drop NET_RAW and MKNOD, got %+v", c.Name, caps) + } + if slices.Contains(caps.Drop, "ALL") { + t.Errorf("container %q must not drop ALL", c.Name) + } + } +} diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go index 9669bb2b..783be007 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go @@ -30,6 +30,7 @@ import ( "time" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/kubernetes" @@ -116,9 +117,53 @@ const ( LabelPriority = hades.MetadataKeyPriority ) -func noPrivilegeEscalation() *corev1.SecurityContext { +// hardenedContainerContext is the security context of every job container. +// NET_RAW (packet spoofing) and MKNOD (device nodes) go away; the other default +// capabilities stay because step images often install packages as root. +func hardenedContainerContext() *corev1.SecurityContext { f := false - return &corev1.SecurityContext{AllowPrivilegeEscalation: &f} + return &corev1.SecurityContext{ + AllowPrivilegeEscalation: &f, + Capabilities: &corev1.Capabilities{Drop: []corev1.Capability{"NET_RAW", "MKNOD"}}, + } +} + +// JobDefaults are operator-wide fallbacks for what a job leaves unset, so a job +// cannot run unbounded. The zero value disables every default. +type JobDefaults struct { + // TimeoutSeconds is the Job's activeDeadlineSeconds when the BuildJob has none. + TimeoutSeconds int64 + // MaxTimeoutSeconds caps the job timeout (also when the job has none). 0 = no cap. + MaxTimeoutSeconds int64 + // CPULimit and MemoryLimit apply per step container for a resource the step + // did not set. Nil leaves it unlimited. + CPULimit *resource.Quantity + MemoryLimit *resource.Quantity + // SharedVolumeSizeLimit is the sizeLimit of the shared emptyDir. Nil = none. + SharedVolumeSizeLimit *resource.Quantity +} + +// helperResources keeps the finalizer container (it only echoes a line) small. +func helperResources() corev1.ResourceRequirements { + l := corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("100m"), + corev1.ResourceMemory: resource.MustParse("64Mi"), + } + return corev1.ResourceRequirements{Limits: l, Requests: l.DeepCopy()} +} + +// effectiveTimeout applies the default and the cap to the job's own timeout. +func (d JobDefaults) effectiveTimeout(job *int64) *int64 { + t := job + if t == nil && d.TimeoutSeconds > 0 { + v := d.TimeoutSeconds + t = &v + } + if d.MaxTimeoutSeconds > 0 && (t == nil || *t > d.MaxTimeoutSeconds) { + v := d.MaxTimeoutSeconds + t = &v + } + return t } // ManagedByValue is the value of the LabelManagedBy label applied to Jobs the @@ -160,6 +205,9 @@ type BuildJobReconciler struct { // to DefaultFinalizerImage. FinalizerImage string + // JobDefaults are the timeout, resource and volume fallbacks for jobs. + JobDefaults JobDefaults + // ImagePullPolicy is applied to every step container. Empty leaves it unset, // which keeps Kubernetes' default (Always for :latest or untagged images). ImagePullPolicy corev1.PullPolicy @@ -602,7 +650,7 @@ func (r *BuildJobReconciler) createJob(ctx context.Context, bj *buildv1.BuildJob active = r.admission.activeCount(bj.Namespace, jobs, time.Now()) suspended = active >= r.MaxParallelism - k8sJob := buildK8sJob(bj, jobName, r.DeleteOnComplete, suspended, r.finalizerImage(), r.ImagePullPolicy) + k8sJob := buildK8sJob(bj, jobName, r.DeleteOnComplete, suspended, r.finalizerImage(), r.ImagePullPolicy, r.JobDefaults) if err := controllerutil.SetControllerReference(bj, k8sJob, r.Scheme); err != nil { return 0, false, err } @@ -797,7 +845,7 @@ func (r *BuildJobReconciler) setStatusPending(ctx context.Context, nn types.Name } // buildK8sJob creates a one-time Job with InitContainers that execute each Step in order -func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, suspend bool, finalizerImage string, pullPolicy corev1.PullPolicy) *batchv1.Job { +func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, suspend bool, finalizerImage string, pullPolicy corev1.PullPolicy, defaults JobDefaults) *batchv1.Job { sharedVolName := "shared" sharedMount := corev1.VolumeMount{Name: sharedVolName, MountPath: "/shared"} @@ -838,7 +886,7 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su WorkingDir: sharedMount.MountPath, Env: envFromMeta(env), VolumeMounts: []corev1.VolumeMount{sharedMount}, - SecurityContext: noPrivilegeEscalation(), + SecurityContext: hardenedContainerContext(), } if hasScript { @@ -859,14 +907,21 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su } } - // Set resource limits if specified - if s.CPULimit != nil || s.MemoryLimit != nil { + // Set resource limits: the step's own, else the operator default + cpuLimit, memLimit := s.CPULimit, s.MemoryLimit + if cpuLimit == nil { + cpuLimit = defaults.CPULimit + } + if memLimit == nil { + memLimit = defaults.MemoryLimit + } + if cpuLimit != nil || memLimit != nil { c.Resources.Limits = corev1.ResourceList{} - if s.CPULimit != nil { - c.Resources.Limits[corev1.ResourceCPU] = *s.CPULimit + if cpuLimit != nil { + c.Resources.Limits[corev1.ResourceCPU] = *cpuLimit } - if s.MemoryLimit != nil { - c.Resources.Limits[corev1.ResourceMemory] = *s.MemoryLimit + if memLimit != nil { + c.Resources.Limits[corev1.ResourceMemory] = *memLimit } } @@ -914,16 +969,20 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su ImagePullPolicy: corev1.PullIfNotPresent, Command: []string{"sh", "-c", "echo build finished"}, VolumeMounts: []corev1.VolumeMount{sharedMount}, - SecurityContext: noPrivilegeEscalation(), + SecurityContext: hardenedContainerContext(), + Resources: helperResources(), } gracePeriod := int64(podTerminationGracePeriodSeconds) noToken := false + noLinks := false podSpec := corev1.PodSpec{ // Steps run untrusted code and never talk to the Kubernetes API. AutomountServiceAccountToken: &noToken, - // Root and default capabilities stay allowed on purpose: arbitrary step - // images often need them (package installs). + // Skip the env vars Kubernetes injects for every service in the namespace. + EnableServiceLinks: &noLinks, + // Root and most default capabilities stay allowed on purpose: arbitrary + // step images often need them (package installs). SecurityContext: &corev1.PodSecurityContext{ SeccompProfile: &corev1.SeccompProfile{Type: corev1.SeccompProfileTypeRuntimeDefault}, }, @@ -932,7 +991,7 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su Volumes: []corev1.Volume{{ Name: sharedVolName, VolumeSource: corev1.VolumeSource{ - EmptyDir: &corev1.EmptyDirVolumeSource{}, + EmptyDir: &corev1.EmptyDirVolumeSource{SizeLimit: defaults.SharedVolumeSizeLimit}, }, }}, RestartPolicy: corev1.RestartPolicyNever, @@ -971,8 +1030,9 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su BackoffLimit: &backoff, // Whole-job timeout: Kubernetes fails the Job (reason DeadlineExceeded) // and deletes its pod once this elapses, covering time spent in init - // containers (the build steps). nil when no timeout is configured. - ActiveDeadlineSeconds: bj.Spec.TimeoutSeconds, + // containers (the build steps). The operator default and cap apply, so + // an unschedulable pod cannot hold a parallelism slot forever. + ActiveDeadlineSeconds: defaults.effectiveTimeout(bj.Spec.TimeoutSeconds), Template: corev1.PodTemplateSpec{ // The operator's pod informer is filtered on LabelManagedBy and its // pod watch maps LabelBuildJob back to the BuildJob, so the pod diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go index 8c25b0f8..702fcad5 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go @@ -23,7 +23,7 @@ func TestBuildK8sJob_PodTemplateLabelsAndGracePeriod(t *testing.T) { ObjectMeta: metav1.ObjectMeta{Name: "job-1", Labels: map[string]string{LabelPriority: "3"}}, Spec: buildv1.BuildJobSpec{Steps: []buildv1.BuildStep{{ID: 1, Image: "alpine"}}}, } - job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) podLabels := job.Spec.Template.Labels if podLabels[LabelManagedBy] != ManagedByValue || podLabels[LabelBuildJob] != "job-1" || podLabels[LabelPriority] != "3" { @@ -254,7 +254,7 @@ func TestResolvePod(t *testing.T) { Labels: map[string]string{LabelManagedBy: ManagedByValue, LabelBuildJob: "job-2"}, }} r := newAdmissionTestReconciler(t, 1, pod, other) - job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) got, err := r.resolvePod(ctx, bj, job) if err != nil || got.Name != pod.Name { @@ -264,7 +264,7 @@ func TestResolvePod(t *testing.T) { t.Run("no pod yet is ErrPodNotFound", func(t *testing.T) { r := newAdmissionTestReconciler(t, 1) - job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) if _, err := r.resolvePod(ctx, bj, job); !errors.Is(err, k8s.ErrPodNotFound) { t.Fatalf("resolvePod() error = %v, want ErrPodNotFound", err) } @@ -331,7 +331,7 @@ func TestUpdateContainerStatuses_StopsOnLiveFinalizedBuildJob(t *testing.T) { Labels: map[string]string{LabelManagedBy: ManagedByValue, LabelBuildJob: "job-1"}, }} r := newAdmissionTestReconciler(t, 1, bj, pod) - job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "") + job := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) stale := bj.DeepCopy() stale.Status.Phase = "Running" @@ -493,7 +493,7 @@ func TestUpdateContainerStateMap_LivePodVetoesRestartAfterGrace(t *testing.T) { // not the one this pass computed. func TestCreateJob_AlreadyExistsReportsExistingSuspend(t *testing.T) { bj := &buildv1.BuildJob{ObjectMeta: metav1.ObjectMeta{Name: "job-1", Namespace: "ns", UID: "uid-1"}} - existing := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "") + existing := buildK8sJob(bj, "buildjob-job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) // MaxParallelism 0 makes this pass want to suspend; the existing Job runs. r := newAdmissionTestReconciler(t, 0, bj, existing) diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_statuswrite_test.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_statuswrite_test.go index d8116828..b923ded3 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_statuswrite_test.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_statuswrite_test.go @@ -147,14 +147,14 @@ func TestBuildK8sJob_AppliesImagePullPolicyToSteps(t *testing.T) { }}, } - job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, corev1.PullIfNotPresent) + job := buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, corev1.PullIfNotPresent, JobDefaults{}) for _, c := range job.Spec.Template.Spec.InitContainers { if c.ImagePullPolicy != corev1.PullIfNotPresent { t.Errorf("%s ImagePullPolicy = %q, want IfNotPresent", c.Name, c.ImagePullPolicy) } } - job = buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "") + job = buildK8sJob(bj, "job-1", true, false, DefaultFinalizerImage, "", JobDefaults{}) for _, c := range job.Spec.Template.Spec.InitContainers { if c.ImagePullPolicy != "" { t.Errorf("%s ImagePullPolicy = %q, want unset", c.Name, c.ImagePullPolicy) diff --git a/HadesScheduler/k8s/k8s.go b/HadesScheduler/k8s/k8s.go index 687b0243..615ebe8e 100644 --- a/HadesScheduler/k8s/k8s.go +++ b/HadesScheduler/k8s/k8s.go @@ -6,12 +6,16 @@ package k8s import ( "context" + "errors" "fmt" "log/slog" + "math" + "strings" "time" "github.com/hades-scheduler/hades/hadesScheduler/log" hades "github.com/hades-scheduler/hades/shared" + status "github.com/hades-scheduler/hades/shared/buildstatus" "github.com/hades-scheduler/hades/shared/payload" "github.com/hades-scheduler/hades/shared/utils" "github.com/nats-io/nats.go" @@ -31,7 +35,7 @@ type Scheduler struct { namespace string config K8sConfig gvr schema.GroupVersionResource - publisher log.NATSPublisher + publisher status.StatusPublisher } // K8sConfig holds the base Kubernetes executor configuration. @@ -104,7 +108,7 @@ func NewK8sScheduler(nc *nats.Conn) (*Scheduler, error) { if err != nil { return nil, fmt.Errorf("creating NATS publisher: %w", err) } - scheduler.publisher = *publisher + scheduler.publisher = publisher } else { slog.Warn("NATS connection is nil, publisher not created, logs will not be published") } @@ -163,6 +167,14 @@ func (k Scheduler) createBuildJobCR(ctx context.Context, job payload.QueuePayloa return fmt.Errorf("dynamic client is nil: operator mode not initialized") } + // Payloads can reach NATS without passing the API, so check what the CRD + // would reject anyway. Without a terminal status the job would stay Queued. + for _, s := range job.Steps { + if s.ID < 1 || s.ID > math.MaxInt32 { + return k.failInvalid(ctx, job, fmt.Sprintf("step id %d must be between 1 and %d", s.ID, math.MaxInt32)) + } + } + obj := buildBuildJobObject(job, k.namespace) _, err := k.dynClient.Resource(k.gvr).Namespace(k.namespace).Create(ctx, obj, metav1.CreateOptions{}) @@ -171,6 +183,10 @@ func (k Scheduler) createBuildJobCR(ctx context.Context, job payload.QueuePayloa slog.Info("BuildJob already exists (idempotent)", "name", job.ID.String()) return nil } + // The API server will never accept this job, so retrying is pointless. + if apierrors.IsInvalid(err) || apierrors.IsBadRequest(err) { + return k.failInvalid(ctx, job, err.Error()) + } return err } @@ -178,6 +194,24 @@ func (k Scheduler) createBuildJobCR(ctx context.Context, job payload.QueuePayloa return nil } +// maxReasonLen keeps the X-Hades-Reason header and webhook reason short. +const maxReasonLen = 300 + +// failInvalid publishes a terminal Failed status for a job that can never be +// created, and returns the error for the caller to log. +func (k Scheduler) failInvalid(ctx context.Context, job payload.QueuePayload, msg string) error { + reason := "invalid job: " + msg + if len(reason) > maxReasonLen { + reason = strings.ToValidUTF8(reason[:maxReasonLen], "") + } + if k.publisher == nil { + slog.Warn("No status publisher, cannot report invalid job as Failed", "id", job.ID.String()) + } else if err := k.publisher.PublishJobStatus(ctx, status.StatusFailed, job.ID.String(), reason); err != nil { + slog.Error("Failed to publish Failed status for invalid job", "error", err, "id", job.ID.String()) + } + return errors.New(reason) +} + // buildBuildJobObject maps a queue payload to the unstructured BuildJob custom // resource the operator reconciles. func buildBuildJobObject(job payload.QueuePayload, namespace string) *unstructured.Unstructured { diff --git a/HadesScheduler/k8s/k8s_test.go b/HadesScheduler/k8s/k8s_test.go index ee1c2ee5..dce85acf 100644 --- a/HadesScheduler/k8s/k8s_test.go +++ b/HadesScheduler/k8s/k8s_test.go @@ -1,13 +1,24 @@ package k8s import ( + "context" + "errors" + "math" + "strings" "testing" "time" "github.com/google/uuid" + status "github.com/hades-scheduler/hades/shared/buildstatus" "github.com/hades-scheduler/hades/shared/payload" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/validation/field" + "k8s.io/client-go/dynamic" ) // TestBuildBuildJobObject verifies the payload-to-BuildJob mapping used by the @@ -81,3 +92,84 @@ func TestNewK8sSchedulerRejectsNonPositiveRateLimits(t *testing.T) { require.ErrorContains(t, err, "K8S_CLIENT_QPS") } } + +type recordingPublisher struct { + statuses []status.JobStatus + reasons []string +} + +func (r *recordingPublisher) PublishJobStatus(_ context.Context, s status.JobStatus, _ string, reason ...string) error { + r.statuses = append(r.statuses, s) + r.reasons = append(r.reasons, status.FirstReason(reason...)) + return nil +} + +// stubDynamic counts Create calls and returns a canned error. The client-go +// fake cannot be used: it deep-copies the object and panics on the plain Go +// types (int, []map[string]interface{}) the scheduler builds. +type stubDynamic struct { + dynamic.Interface + dynamic.NamespaceableResourceInterface + createErr error + creates int +} + +func (d *stubDynamic) Resource(schema.GroupVersionResource) dynamic.NamespaceableResourceInterface { + return d +} + +func (d *stubDynamic) Namespace(string) dynamic.ResourceInterface { return d } + +func (d *stubDynamic) Create(context.Context, *unstructured.Unstructured, metav1.CreateOptions, ...string) (*unstructured.Unstructured, error) { + d.creates++ + return nil, d.createErr +} + +func newTestScheduler(createErr error) (Scheduler, *recordingPublisher, *stubDynamic) { + dyn := &stubDynamic{createErr: createErr} + pub := &recordingPublisher{} + return Scheduler{dynClient: dyn, namespace: "ns", publisher: pub}, pub, dyn +} + +func TestCreateBuildJobCRFailureHandling(t *testing.T) { + gr := schema.GroupResource{Group: "build.hades.tum.de", Resource: "buildjobs"} + tests := []struct { + name string + stepID int + createErr error + wantErr bool + wantFailed bool + wantCreate int + }{ + {"success", 1, nil, false, false, 1}, + {"invalid from API server", 1, apierrors.NewInvalid(schema.GroupKind{Group: gr.Group, Kind: "BuildJob"}, "x", field.ErrorList{field.Invalid(field.NewPath("spec", "steps"), "[]", "bad")}), true, true, 1}, + {"bad request", 1, apierrors.NewBadRequest("nope"), true, true, 1}, + {"transient conflict", 1, apierrors.NewConflict(gr, "x", errors.New("boom")), true, false, 1}, + {"already exists is idempotent", 1, apierrors.NewAlreadyExists(gr, "x"), false, false, 1}, + {"step id too large", math.MaxInt32 + 1, nil, true, true, 0}, + {"step id zero", 0, nil, true, true, 0}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s, pub, dyn := newTestScheduler(tt.createErr) + job := payload.QueuePayload{ID: uuid.New(), Steps: []payload.Step{{ID: tt.stepID, Image: "alpine"}}} + err := s.createBuildJobCR(context.Background(), job) + assert.Equal(t, tt.wantErr, err != nil) + assert.Equal(t, tt.wantCreate, dyn.creates) + if tt.wantFailed { + require.Equal(t, []status.JobStatus{status.StatusFailed}, pub.statuses) + assert.True(t, strings.HasPrefix(pub.reasons[0], "invalid job: ")) + } else { + assert.Empty(t, pub.statuses) + } + }) + } +} + +func TestFailInvalidTruncatesReason(t *testing.T) { + s, pub, _ := newTestScheduler(nil) + err := s.failInvalid(context.Background(), payload.QueuePayload{ID: uuid.New()}, strings.Repeat("e", 1000)) + require.Error(t, err) + require.Len(t, pub.reasons, 1) + assert.LessOrEqual(t, len(pub.reasons[0]), maxReasonLen) +} diff --git a/Readme.md b/Readme.md index 6e1d862c..5adaf101 100644 --- a/Readme.md +++ b/Readme.md @@ -391,6 +391,7 @@ Breaking changes: - `/build` bodies are capped at 1 MiB (413 above) and duplicate step ids are rejected with 400. - `compose.yml` requires `AUTH_KEY` in `.env` and publishes API, log manager, NATS and Jaeger on `127.0.0.1` only. `docker-compose.deploy.yml` publishes only Traefik on 80/443 (Compose 2.24 or later) and no longer routes the log manager publicly. - Kubernetes: job pods no longer mount a service account token and run with the `RuntimeDefault` seccomp profile and `allowPrivilegeEscalation: false`. The operator role has no access to secrets or configmaps. `hadesOperator.DeleteOnComplete` now defaults to `true`. Optional `networkPolicy.enabled` (default `false`) is new. +- Kubernetes: jobs now get a default 1 h timeout, 2 CPU and 4Gi memory limits per step, a 10Gi shared volume and no `NET_RAW`/`MKNOD` capabilities or service-link variables. Change or disable the defaults with `hadesOperator.jobDefaultTimeoutSeconds`, `jobMaxTimeoutSeconds`, `jobDefaultCPULimit`, `jobDefaultMemoryLimit` and `jobSharedVolumeSizeLimit` (empty or `0` disables each). - Dashboard login allows 5 attempts per client address per 15 minutes. Behind a reverse proxy, set `DASHBOARD_TRUSTED_PROXIES`. ### Measuring Hades overhead diff --git a/docs/api.md b/docs/api.md index bd4826ca..2ed6cb74 100644 --- a/docs/api.md +++ b/docs/api.md @@ -58,7 +58,7 @@ curl -X POST -H "Content-Type: application/json" \ | Status | Body | Meaning | | ------ | ---- | ------- | | `200 OK` | `{ "message": "Successfully enqueued job", "job_id": "" }` | Job accepted and queued. | -| `400 Bad Request` | plain-text message naming the offending field | Invalid JSON or failed validation (e.g. missing `name`, unparseable `memory_limit`, duplicate step ids). | +| `400 Bad Request` | plain-text message naming the offending field | Invalid JSON or failed validation (e.g. missing `name`, no steps, a step id outside 1..2147483647, a step without `image`, unparseable `memory_limit`, duplicate step ids). | | `413 Request Entity Too Large` | | The body exceeds 1 MiB. | | `500 Internal Server Error` | `Failed to enqueue job` | NATS publish failed. | @@ -101,7 +101,7 @@ Submitted to `POST /build`. Field semantics come from `shared/payload/payload.go | `name` | string | **yes** | Human-readable job name. | | `timestamp` | RFC3339 | no | Job creation time. | | `metadata` | map[string]string | no | Job-level metadata (also injected as environment variables). | -| `steps` | `[]Step` | no | Ordered steps to execute. | +| `steps` | `[]Step` | yes | Ordered steps to execute (at least one). | | `callback_url` | string | no | Absolute http/https URL to forward aggregated logs/results to. | | `status_callback_url` | string | no | Absolute http/https URL that receives the job-status webhook when the job reaches a terminal status. Independent of `callback_url`; see [`HadesLogManager/Readme.md`](../HadesLogManager/Readme.md#job-status-webhook). | | `timeout_seconds` | int64 | no | Whole-job timeout in seconds. The job is killed and marked failed once exceeded. `0` (default) = no timeout. Enforced on both executors (Docker: context deadline; Kubernetes: Job `activeDeadlineSeconds`). | diff --git a/docs/configuration.md b/docs/configuration.md index 7fa27273..d738fbb3 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -137,6 +137,11 @@ Only deployed when the scheduler runs in `operator` mode. | `LOG_DRAIN_TIMEOUT` | `45s` | How long a completed `BuildJob` is kept while its container logs drain, before it is deleted anyway (Go duration). Only relevant with `DELETE_ON_COMPLETE=true`. A zero or negative duration falls back to the default; a value that is not a valid Go duration fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | | `FINALIZER_IMAGE` | `busybox:1.37.0` | Image for the main container of every `BuildJob` pod. Kubernetes requires at least one non-init container and the build steps run as init containers, so this one exists only to exit once they are done: it runs `sh -c "echo build finished"` and needs nothing but a shell. Override it for clusters that cannot reach Docker Hub, or to point at a mirror. Keep it pinned to an explicit tag: an untagged image resolves to `:latest`, which Kubernetes special-cases to `imagePullPolicy: Always`, adding a registry round trip to every job for an image the node already has. Empty falls back to the default. | `HadesScheduler/HadesOperator/cmd/main.go` | | `K8S_IMAGE_PULL_POLICY` | | `imagePullPolicy` for build step containers: `always`, `if-not-present` or `never` (case-insensitive). Empty leaves it unset, so Kubernetes pulls `:latest` and untagged images on every step. The finalizer container is not affected. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | +| `JOB_DEFAULT_TIMEOUT_SECONDS` | `3600` | Timeout applied to jobs that set no `timeout_seconds` (Job `activeDeadlineSeconds`), so a hung or unschedulable job cannot hold a parallelism slot forever. Empty or `0` disables it. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | +| `JOB_MAX_TIMEOUT_SECONDS` | `0` | Upper bound for the job timeout; a larger or missing job timeout is clamped to it. `0` means no cap. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | +| `JOB_DEFAULT_CPU_LIMIT` | `2` | CPU limit for each step that sets no `cpu_limit` (Kubernetes quantity; requests default to limits). Empty disables it. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | +| `JOB_DEFAULT_MEMORY_LIMIT` | `4Gi` | Memory limit for each step that sets no `memory_limit` (Kubernetes quantity). Empty disables it. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | +| `JOB_SHARED_VOLUME_SIZE_LIMIT` | `10Gi` | `sizeLimit` of the `emptyDir` shared between the steps of a job (`/shared`). Empty disables it. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | | `DEV_MODE` | `false` | Enable the controller-runtime development logger. | `HadesScheduler/HadesOperator/cmd/main.go` | Plus the [NATS](#nats-connection-all-components) variables (the operator publishes status/log events). The operator also accepts standard controller-runtime **flags**: `--health-probe-bind-address` (default `:8083`), `--metrics-bind-address` (default `:8082`, set `0` to disable), `--leader-elect`, and the log flags bound via `opts.BindFlags`. diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index 30d54f6d..6e5409d4 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -142,6 +142,11 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | hadesOperator.finalizerImage | string | `"busybox:1.37.0"` | Image for the main container of every BuildJob pod. Kubernetes requires at least one non-init container and the build steps run as init containers, so this one exists only to exit once they are done: it runs `sh -c "echo build finished"` and needs nothing but a shell. Override it for clusters that cannot reach Docker Hub, or to point at a mirror. Keep it pinned to an explicit tag - an untagged image resolves to ":latest", which Kubernetes special-cases to imagePullPolicy: Always, adding a registry round trip to every job for an image the node already has. Empty falls back to the operator's built-in default. | | hadesOperator.image.pullPolicy | string | `"Always"` | Image pull policy for the operator. | | hadesOperator.imagePullPolicy | string | `""` | imagePullPolicy for build step containers: `always`, `if-not-present` or `never` (case-insensitive). If empty, Kubernetes pulls `:latest` and untagged images on every step. `if-not-present` skips the pull when the node already has the image (so a moved tag isn't picked up), `never` fails steps whose image is missing. The finalizer container is not affected. Invalid values fail operator startup. | +| hadesOperator.jobDefaultCPULimit | string | `"2"` | CPU limit (Kubernetes quantity) for each step that sets no `cpu_limit`. Empty disables the default. An invalid value fails operator startup. | +| hadesOperator.jobDefaultMemoryLimit | string | `"4Gi"` | Memory limit (Kubernetes quantity) for each step that sets no `memory_limit`. Empty disables the default. An invalid value fails operator startup. | +| hadesOperator.jobDefaultTimeoutSeconds | string | `"3600"` | Timeout (seconds) applied to jobs that set no `timeout_seconds`, so a hung or unschedulable job cannot hold a parallelism slot forever. Empty or 0 disables the default. | +| hadesOperator.jobMaxTimeoutSeconds | string | `"0"` | Upper bound (seconds) for the job timeout. A larger or missing job timeout is clamped to it. Empty or 0 means no cap. | +| hadesOperator.jobSharedVolumeSizeLimit | string | `"10Gi"` | Size limit of the `emptyDir` shared between the steps of a job (mounted at `/shared`). Without it a job can fill the node's disk. Empty disables the limit. An invalid value fails operator startup. | | hadesOperator.leaderElection.enabled | bool | `true` | Enable leader election for the operator (recommended with >1 replica). | | hadesOperator.logDrainTimeout | string | `"45s"` | How long a completed BuildJob is kept (undeleted) while its container logs are still being published, before it is deleted anyway (Go duration). Kubernetes only serves container logs while the pod exists, so this is the safety valve on the log-drain gate. Only relevant when DeleteOnComplete is true. A zero or negative duration falls back to the default; a value that is not a valid Go duration fails operator startup. | | hadesOperator.maxConcurrentReconciles | string | `"8"` | Number of BuildJobs the operator reconciles in parallel. A zero or negative value falls back to the default. | diff --git a/helm/hades/crds/build.hades.tum.de_buildjobs.yaml b/helm/hades/crds/build.hades.tum.de_buildjobs.yaml index 1c1ed0e3..ab6fe271 100644 --- a/helm/hades/crds/build.hades.tum.de_buildjobs.yaml +++ b/helm/hades/crds/build.hades.tum.de_buildjobs.yaml @@ -104,6 +104,7 @@ spec: Steps execute in ascending ID order: step with id=1 runs first, then id=2, etc. This is mandatory and must be unique within the BuildJob. format: int32 + maximum: 2147483647 minimum: 1 type: integer image: diff --git a/helm/hades/templates/hades-operator-deployment.yaml b/helm/hades/templates/hades-operator-deployment.yaml index 64017896..b6b5608e 100644 --- a/helm/hades/templates/hades-operator-deployment.yaml +++ b/helm/hades/templates/hades-operator-deployment.yaml @@ -56,6 +56,16 @@ spec: value: {{ .Values.hadesOperator.finalizerImage | quote }} - name: K8S_IMAGE_PULL_POLICY value: {{ .Values.hadesOperator.imagePullPolicy | quote }} + - name: JOB_DEFAULT_TIMEOUT_SECONDS + value: {{ .Values.hadesOperator.jobDefaultTimeoutSeconds | quote }} + - name: JOB_MAX_TIMEOUT_SECONDS + value: {{ .Values.hadesOperator.jobMaxTimeoutSeconds | quote }} + - name: JOB_DEFAULT_CPU_LIMIT + value: {{ .Values.hadesOperator.jobDefaultCPULimit | quote }} + - name: JOB_DEFAULT_MEMORY_LIMIT + value: {{ .Values.hadesOperator.jobDefaultMemoryLimit | quote }} + - name: JOB_SHARED_VOLUME_SIZE_LIMIT + value: {{ .Values.hadesOperator.jobSharedVolumeSizeLimit | quote }} - name: MAX_CONCURRENT_RECONCILES value: {{ .Values.hadesOperator.maxConcurrentReconciles | quote }} - name: DELETE_ON_COMPLETE diff --git a/helm/hades/values-test.yaml b/helm/hades/values-test.yaml index 3f20b617..9903a95d 100644 --- a/helm/hades/values-test.yaml +++ b/helm/hades/values-test.yaml @@ -33,3 +33,12 @@ nats: # values.yaml hardcodes the "hades" namespace in the NATS service FQDN, so the # test namespace must point at its own NATS release (hades-nats.). host: hades-nats.hades-test.svc.cluster.local + +# Job pods run untrusted code: keep them away from NATS, the log manager and the +# other Hades pods. The cluster CNI (Calico) matches ipBlock rules against pod +# IPs, so the pod network has to be excluded from the "outside" rule explicitly. +networkPolicy: + enabled: true + jobPods: + blockedCIDRs: + - 198.18.0.0/15 diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index be3ec04a..8e1f3f54 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -164,6 +164,24 @@ hadesOperator: # image is missing. The finalizer container is not affected. Invalid values # fail operator startup. imagePullPolicy: "" + # -- Timeout (seconds) applied to jobs that set no `timeout_seconds`, so a hung + # or unschedulable job cannot hold a parallelism slot forever. Empty or 0 + # disables the default. + jobDefaultTimeoutSeconds: "3600" + # -- Upper bound (seconds) for the job timeout. A larger or missing job timeout + # is clamped to it. Empty or 0 means no cap. + jobMaxTimeoutSeconds: "0" + # -- CPU limit (Kubernetes quantity) for each step that sets no `cpu_limit`. + # Empty disables the default. An invalid value fails operator startup. + jobDefaultCPULimit: "2" + # -- Memory limit (Kubernetes quantity) for each step that sets no + # `memory_limit`. Empty disables the default. An invalid value fails operator + # startup. + jobDefaultMemoryLimit: "4Gi" + # -- Size limit of the `emptyDir` shared between the steps of a job (mounted at + # `/shared`). Without it a job can fill the node's disk. Empty disables the + # limit. An invalid value fails operator startup. + jobSharedVolumeSizeLimit: "10Gi" hadesLogManager: image: diff --git a/website/docs/operation-modes/k8s-operator.md b/website/docs/operation-modes/k8s-operator.md index 23c045fe..68b7e1b9 100644 --- a/website/docs/operation-modes/k8s-operator.md +++ b/website/docs/operation-modes/k8s-operator.md @@ -49,7 +49,7 @@ The scheduler uses a dynamic Kubernetes client to create `BuildJob` custom resou ## Job pod hardening -Job pods do not mount a service account token, use the `RuntimeDefault` seccomp profile and set `allowPrivilegeEscalation: false`. They still run as the image's user. The API and the log manager have their own service accounts, and the operator role has no access to secrets or configmaps. +Job pods do not mount a service account token, use the `RuntimeDefault` seccomp profile and set `allowPrivilegeEscalation: false`. They still run as the image's user, but every container drops the `NET_RAW` and `MKNOD` capabilities and the pod gets no service-link environment variables (`enableServiceLinks: false`). The operator also applies defaults to what a job leaves unset: a 1 h timeout (`JOB_DEFAULT_TIMEOUT_SECONDS`, cap with `JOB_MAX_TIMEOUT_SECONDS`), a 2 CPU and 4Gi memory limit per step (`JOB_DEFAULT_CPU_LIMIT`, `JOB_DEFAULT_MEMORY_LIMIT`) and a 10Gi size limit on the shared volume (`JOB_SHARED_VOLUME_SIZE_LIMIT`). Set a value to empty or `0` to disable it (Helm: `hadesOperator.jobDefaultTimeoutSeconds` and friends). The API and the log manager have their own service accounts, and the operator role has no access to secrets or configmaps. Set `networkPolicy.enabled: true` (default `false`) to add two NetworkPolicies: job pods cannot reach the release namespace, and only Hades components reach NATS on 4222. Before enabling, check the NATS pod labels. On CNIs that match pod IPs against `ipBlock`, set `networkPolicy.jobPods.blockedCIDRs` to the pod CIDR. diff --git a/website/docs/reference/configuration.md b/website/docs/reference/configuration.md index 91cff673..8a086e34 100644 --- a/website/docs/reference/configuration.md +++ b/website/docs/reference/configuration.md @@ -110,6 +110,11 @@ Only deployed when the scheduler runs in `operator` mode. | `LOG_DRAIN_TIMEOUT` | `45s` | How long a completed `BuildJob` is kept while its container logs drain, before it is deleted anyway (Go duration). A zero or negative duration falls back to the default; a value that is not a valid Go duration fails operator startup. | | `FINALIZER_IMAGE` | `busybox:1.37.0` | Image for the main container of every `BuildJob` pod; it only runs `sh -c "echo build finished"`. Override for clusters without Docker Hub access. Keep it pinned - an untagged image means `:latest`, which Kubernetes forces to `imagePullPolicy: Always`. | | `K8S_IMAGE_PULL_POLICY` | | `imagePullPolicy` for build step containers: `always`, `if-not-present` or `never`. Empty leaves the Kubernetes default. Invalid values fail operator startup. | +| `JOB_DEFAULT_TIMEOUT_SECONDS` | `3600` | Timeout applied to jobs that set no `timeout_seconds` (Job `activeDeadlineSeconds`), so a hung or unschedulable job cannot hold a parallelism slot forever. Empty or `0` disables it. | +| `JOB_MAX_TIMEOUT_SECONDS` | `0` | Upper bound for the job timeout; a larger or missing job timeout is clamped to it. `0` means no cap. | +| `JOB_DEFAULT_CPU_LIMIT` | `2` | CPU limit for each step that sets no `cpu_limit` (Kubernetes quantity; requests default to limits). Empty disables it. An invalid value fails operator startup. | +| `JOB_DEFAULT_MEMORY_LIMIT` | `4Gi` | Memory limit for each step that sets no `memory_limit` (Kubernetes quantity). Empty disables it. An invalid value fails operator startup. | +| `JOB_SHARED_VOLUME_SIZE_LIMIT` | `10Gi` | `sizeLimit` of the `emptyDir` shared between the steps of a job (`/shared`). Empty disables it. An invalid value fails operator startup. | | `DEV_MODE` | `false` | Enable the controller-runtime development logger. | The operator also accepts standard controller-runtime flags: `--health-probe-bind-address` (default `:8083`), `--metrics-bind-address` (default `:8082`, set `0` to disable), `--leader-elect`, and the log flags. diff --git a/website/docs/usage/submitting-jobs.md b/website/docs/usage/submitting-jobs.md index 7235c6bf..07e8d935 100644 --- a/website/docs/usage/submitting-jobs.md +++ b/website/docs/usage/submitting-jobs.md @@ -18,7 +18,7 @@ http://localhost:8081 ## Authentication -`POST /build` requires HTTP Basic Auth with username `hades` and `AUTH_KEY` as the password. `AUTH_KEY` is required: with an empty key the API exits at start unless `ALLOW_UNAUTHENTICATED=true` (local development only). Request bodies are capped at 1 MiB (413 above) and duplicate step ids get 400. +`POST /build` requires HTTP Basic Auth with username `hades` and `AUTH_KEY` as the password. `AUTH_KEY` is required: with an empty key the API exits at start unless `ALLOW_UNAUTHENTICATED=true` (local development only). Request bodies are capped at 1 MiB (413 above) and an empty `steps` list, a step id outside 1..2147483647, a step without `image` or duplicate step ids get 400. ## Job Payload Structure From 04e823a722dbe042dd29fb5a6ca06dd81a525a34 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Sat, 14 Nov 2026 01:47:04 +0100 Subject: [PATCH 4/9] Cap steps and limits, fail jobs the operator cannot create Findings from the second test round on the test environment: - API: at most 100 steps per job, cpu_limit at most 1024, memory_limit and memory_swap at most 1T; memory parsing no longer overflows. - CRD: maxItems for the steps list. - Operator: JOB_MAX_CPU_LIMIT (16) and JOB_MAX_MEMORY_LIMIT (32Gi) clamp what a step asks for; a Job the API server rejects as invalid or too large marks the BuildJob Failed instead of being retried forever. Co-Authored-By: Claude Mythos 5.2 --- AGENTS.md | 2 +- HadesAPI/router.go | 32 ++++++++ HadesAPI/router_test.go | 11 +++ .../HadesOperator/api/v1/buildjob_types.go | 1 + HadesScheduler/HadesOperator/cmd/main.go | 8 ++ HadesScheduler/HadesOperator/cmd/main_test.go | 7 ++ .../controller/buildjob_build_test.go | 44 +++++++++++ .../controller/buildjob_controller.go | 58 +++++++++++++- .../controller/buildjob_events_test.go | 76 +++++++++++++++++++ Readme.md | 2 +- docs/api.md | 2 +- docs/configuration.md | 2 + helm/hades/Readme.md | 2 + .../crds/build.hades.tum.de_buildjobs.yaml | 1 + .../templates/hades-operator-deployment.yaml | 4 + helm/hades/values.yaml | 8 ++ shared/utils/config.go | 11 ++- shared/utils/config_test.go | 2 + website/docs/operation-modes/k8s-operator.md | 2 +- website/docs/reference/configuration.md | 2 + website/docs/usage/submitting-jobs.md | 2 +- 21 files changed, 269 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7335794f..cb7f0593 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ Go workspace (`go.work`, Go 1.27) with five modules: - **Job leases:** the scheduler acks a job only after the executor returns, and the Docker executor blocks for the whole job. `shared/nats/consumer.go` therefore heartbeats `msg.InProgress()` on a ticker while a job runs so the `NATS_ACK_WAIT` (default 1m) timer keeps resetting - job duration must never determine AckWait. `NATS_MAX_DELIVER` (default 3) caps redeliveries; the last delivery is not executed but publishes a terminal `Failed` status and terminates the message. - **Job DTO:** `shared/payload/payload.go` (`QueuePayload` with ordered `Step`s). Job-level: `timeout_seconds` (whole-job timeout). Each step has `image`, optional `script`, `metadata` env vars, `cpu_limit` (whole cores), `memory_limit`, and the Docker-only limits `network`, `memory_swap`, `pids_limit`. - **Two outbound pushes, do not conflate them:** `callback_url` forwards the aggregated **log array** and only after `stopWatchingJobLogs` drains the JetStream log consumer; `status_callback_url` receives the **job-status webhook** (`{event, job_id, name, status, reason, queued_at, started_at, finished_at, duration_ms, attempt}`) on the terminal status itself, with at-least-once retry. Neither may be repurposed as the other. - - **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only** (`network` must be in `DOCKER_ALLOWED_NETWORKS`, default `none,bridge,default`, else the job fails; `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` / `HADES_MAX_JOB_TIMEOUT_SECONDS` also Docker only); the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field). The operator has its own defaults for what a job leaves unset (`JOB_DEFAULT_TIMEOUT_SECONDS` 3600, `JOB_MAX_TIMEOUT_SECONDS` 0, `JOB_DEFAULT_CPU_LIMIT` 2, `JOB_DEFAULT_MEMORY_LIMIT` 4Gi, `JOB_SHARED_VOLUME_SIZE_LIMIT` 10Gi; empty or 0 disables each). + - **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only** (`network` must be in `DOCKER_ALLOWED_NETWORKS`, default `none,bridge,default`, else the job fails; `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` / `HADES_MAX_JOB_TIMEOUT_SECONDS` also Docker only); the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field). The operator has its own defaults for what a job leaves unset (`JOB_DEFAULT_TIMEOUT_SECONDS` 3600, `JOB_MAX_TIMEOUT_SECONDS` 0, `JOB_DEFAULT_CPU_LIMIT` 2, `JOB_DEFAULT_MEMORY_LIMIT` 4Gi, `JOB_MAX_CPU_LIMIT` 16, `JOB_MAX_MEMORY_LIMIT` 32Gi (clamp a step above them), `JOB_SHARED_VOLUME_SIZE_LIMIT` 10Gi; empty or 0 disables each). - **CRD:** `BuildJob` in `HadesScheduler/HadesOperator/api/v1/buildjob_types.go`. **Important:** if you change `BuildJobSpec`, run `make -C HadesScheduler/HadesOperator manifests generate` and commit `helm/hades/crds/build.hades.tum.de_buildjobs.yaml` and `zz_generated.deepcopy.go` - the `verify-crd.yml` GitHub workflow will fail otherwise. The `BuildJobSpec` is intentionally duplicated from `shared/payload`; keep them in sync manually. ## Build / test diff --git a/HadesAPI/router.go b/HadesAPI/router.go index 3c83142a..5598f4d9 100644 --- a/HadesAPI/router.go +++ b/HadesAPI/router.go @@ -62,6 +62,16 @@ func securityHeaders() gin.HandlerFunc { // maxBuildBodyBytes caps the /build request body (1 MiB). const maxBuildBodyBytes = 1 << 20 +// Sanity bounds on what one job may ask for. The step count keeps the +// Kubernetes Job within what the API server accepts, the resource bounds keep +// absurd values out of both executors (the Docker one multiplies cores into +// nano CPUs). +const ( + maxSteps = 100 + maxCPULimit = 1024 + maxMemoryBytes = 1 << 40 +) + func setupRouter(authKey string, producer hades.JobPublisher, statusPublisher buildstatus.StatusPublisher, dash *dashboard.Server) *gin.Engine { r := gin.New() r.Use(gin.ErrorLogger()) @@ -199,6 +209,12 @@ func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublishe return } + if len(p.QueuePayload.Steps) > maxSteps { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, fmt.Sprintf("Too many steps: a job can have at most %d steps", maxSteps)) + return + } + seenStepIDs := make(map[int]bool, len(p.QueuePayload.Steps)) for _, step := range p.QueuePayload.Steps { if step.ID < 1 || step.ID > math.MaxInt32 { @@ -218,6 +234,12 @@ func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublishe } seenStepIDs[step.ID] = true + if step.CPULimit > maxCPULimit { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, fmt.Sprintf("cpu_limit must not exceed %d cores", maxCPULimit)) + return + } + var memLimitBytes int64 if step.MemoryLimit != "" { parsed, err := utils.ParseMemoryLimit(step.MemoryLimit) @@ -227,6 +249,11 @@ func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublishe c.String(http.StatusBadRequest, "Failed to parse RAM limit") return } + if parsed > maxMemoryBytes { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, "memory_limit must not exceed 1T (1024G)") + return + } memLimitBytes = parsed } @@ -237,6 +264,11 @@ func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublishe c.String(http.StatusBadRequest, "Failed to parse memory_swap limit") return } + if swapBytes > maxMemoryBytes { + buildRequestsTotal.WithLabelValues("rejected").Inc() + c.String(http.StatusBadRequest, "memory_swap must not exceed 1T (1024G)") + return + } // Docker requires memory_swap (total memory+swap) to be set together // with a memory limit and to be at least as large as it. if step.MemoryLimit == "" { diff --git a/HadesAPI/router_test.go b/HadesAPI/router_test.go index a5716989..f28f63ae 100644 --- a/HadesAPI/router_test.go +++ b/HadesAPI/router_test.go @@ -392,6 +392,12 @@ func (suite *APISuite) TestInvalidStepsRejected() { {"id zero", `{"name":"x","steps":[{"id":0,"image":"i"}]}`, "Invalid step id 0"}, {"id negative", `{"name":"x","steps":[{"id":-1,"image":"i"}]}`, "Invalid step id -1"}, {"id too large", `{"name":"x","steps":[{"id":9223372036854775807,"image":"i"}]}`, "Invalid step id 9223372036854775807"}, + {"too many steps", `{"name":"x","steps":[` + strings.TrimSuffix(strings.Repeat(`{"id":1,"image":"i"},`, maxSteps+1), ",") + `]}`, "at most 100 steps"}, + {"cpu above max", `{"name":"x","steps":[{"id":1,"image":"i","cpu_limit":1025}]}`, "cpu_limit must not exceed 1024"}, + {"cpu uint max", `{"name":"x","steps":[{"id":1,"image":"i","cpu_limit":18446744073709551615}]}`, "cpu_limit must not exceed 1024"}, + {"memory above max", `{"name":"x","steps":[{"id":1,"image":"i","memory_limit":"1025G"}]}`, "memory_limit must not exceed 1T"}, + {"memory overflow", `{"name":"x","steps":[{"id":1,"image":"i","memory_limit":"999999999999999G"}]}`, "Failed to parse RAM limit"}, + {"swap above max", `{"name":"x","steps":[{"id":1,"image":"i","memory_limit":"1G","memory_swap":"1025G"}]}`, "memory_swap must not exceed 1T"}, {"id just above int32", `{"name":"x","steps":[{"id":2147483648,"image":"i"}]}`, "Invalid step id 2147483648"}, } for _, tc := range cases { @@ -406,6 +412,11 @@ func (suite *APISuite) TestInvalidStepsRejected() { } } +func (suite *APISuite) TestLimitsAtTheMaximumAccepted() { + w := suite.postStep(payload.Step{ID: 1, Name: "s", Image: "image1", CPULimit: 1024, MemoryLimit: "1024G", MemorySwap: "1024G"}, 0) + assert.Equal(suite.T(), 200, w.Code) +} + func (suite *APISuite) TestMaxInt32StepIDAccepted() { w := suite.postStep(payload.Step{ID: 2147483647, Name: "s", Image: "image1"}, 0) assert.Equal(suite.T(), 200, w.Code) diff --git a/HadesScheduler/HadesOperator/api/v1/buildjob_types.go b/HadesScheduler/HadesOperator/api/v1/buildjob_types.go index c8c75c5b..9903aab6 100644 --- a/HadesScheduler/HadesOperator/api/v1/buildjob_types.go +++ b/HadesScheduler/HadesOperator/api/v1/buildjob_types.go @@ -49,6 +49,7 @@ type BuildJobSpec struct { // Each step runs in its own container and can share data with other steps via mounted volumes. // Typical pipeline: Step 1 clones code, Step 2 builds it, Step 3 runs tests. // +kubebuilder:validation:MinItems=1 + // +kubebuilder:validation:MaxItems=100 Steps []BuildStep `json:"steps"` // Maximum time (in seconds) allowed for the entire BuildJob to complete. diff --git a/HadesScheduler/HadesOperator/cmd/main.go b/HadesScheduler/HadesOperator/cmd/main.go index 25516c35..2594046b 100644 --- a/HadesScheduler/HadesOperator/cmd/main.go +++ b/HadesScheduler/HadesOperator/cmd/main.go @@ -86,6 +86,8 @@ type OperatorConfig struct { JobMaxTimeoutSeconds int64 `env:"JOB_MAX_TIMEOUT_SECONDS" envDefault:"0"` JobDefaultCPULimit string `env:"JOB_DEFAULT_CPU_LIMIT" envDefault:"2"` JobDefaultMemoryLimit string `env:"JOB_DEFAULT_MEMORY_LIMIT" envDefault:"4Gi"` + JobMaxCPULimit string `env:"JOB_MAX_CPU_LIMIT" envDefault:"16"` + JobMaxMemoryLimit string `env:"JOB_MAX_MEMORY_LIMIT" envDefault:"32Gi"` JobSharedVolumeSizeLimit string `env:"JOB_SHARED_VOLUME_SIZE_LIMIT" envDefault:"10Gi"` } @@ -110,6 +112,12 @@ func (c *OperatorConfig) jobDefaults() (controller.JobDefaults, error) { if d.MemoryLimit, err = parseQuantity("JOB_DEFAULT_MEMORY_LIMIT", c.JobDefaultMemoryLimit); err != nil { return d, err } + if d.MaxCPULimit, err = parseQuantity("JOB_MAX_CPU_LIMIT", c.JobMaxCPULimit); err != nil { + return d, err + } + if d.MaxMemoryLimit, err = parseQuantity("JOB_MAX_MEMORY_LIMIT", c.JobMaxMemoryLimit); err != nil { + return d, err + } d.SharedVolumeSizeLimit, err = parseQuantity("JOB_SHARED_VOLUME_SIZE_LIMIT", c.JobSharedVolumeSizeLimit) return d, err } diff --git a/HadesScheduler/HadesOperator/cmd/main_test.go b/HadesScheduler/HadesOperator/cmd/main_test.go index 9f5f3a71..4990cb81 100644 --- a/HadesScheduler/HadesOperator/cmd/main_test.go +++ b/HadesScheduler/HadesOperator/cmd/main_test.go @@ -185,6 +185,13 @@ func TestJobDefaults(t *testing.T) { if err != nil || d.CPULimit != nil || d.MemoryLimit != nil || d.SharedVolumeSizeLimit != nil { t.Errorf("empty config must disable the defaults, got %+v, %v", d, err) } + d, err = (&OperatorConfig{JobMaxCPULimit: "16", JobMaxMemoryLimit: "32Gi"}).jobDefaults() + if err != nil || d.MaxCPULimit == nil || d.MaxMemoryLimit == nil { + t.Fatalf("jobDefaults() max = %+v, %v", d, err) + } + if _, err := (&OperatorConfig{JobMaxMemoryLimit: "lots"}).jobDefaults(); err == nil { + t.Error("an invalid max quantity must fail") + } if _, err := (&OperatorConfig{JobDefaultMemoryLimit: "lots"}).jobDefaults(); err == nil { t.Error("an invalid quantity must fail") } diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go index 17996e11..7561b1ab 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go @@ -467,3 +467,47 @@ func TestBuildK8sJob_DropsCapabilitiesAndServiceLinks(t *testing.T) { } } } + +func TestBuildK8sJob_MaxLimitsClamp(t *testing.T) { + q := resource.MustParse + maxCPU, maxMem := q("16"), q("32Gi") + big, bigMem := q("100"), q("1Ti") + first := func(step buildv1.BuildStep, d JobDefaults) corev1.Container { + return buildK8sJob(defaultsTestJob(nil, step), "job-1", true, false, DefaultFinalizerImage, "", d).Spec.Template.Spec.InitContainers[0] + } + + t.Run("step limits above the max are clamped, requests follow", func(t *testing.T) { + c := first(buildv1.BuildStep{CPULimit: &big, MemoryLimit: &bigMem}, JobDefaults{MaxCPULimit: &maxCPU, MaxMemoryLimit: &maxMem}) + if !c.Resources.Limits.Cpu().Equal(maxCPU) || !c.Resources.Limits.Memory().Equal(maxMem) { + t.Errorf("limits = %v, want 16 and 32Gi", c.Resources.Limits) + } + if !c.Resources.Requests.Cpu().Equal(maxCPU) || !c.Resources.Requests.Memory().Equal(maxMem) { + t.Errorf("requests = %v, want 16 and 32Gi", c.Resources.Requests) + } + }) + t.Run("explicit requests above the max are clamped", func(t *testing.T) { + c := first(buildv1.BuildStep{CPURequest: &big, MemoryRequest: &bigMem}, JobDefaults{MaxCPULimit: &maxCPU, MaxMemoryLimit: &maxMem}) + if !c.Resources.Requests.Cpu().Equal(maxCPU) || !c.Resources.Requests.Memory().Equal(maxMem) { + t.Errorf("requests = %v, want 16 and 32Gi", c.Resources.Requests) + } + }) + t.Run("values below the max are untouched", func(t *testing.T) { + small := q("2") + c := first(buildv1.BuildStep{CPULimit: &small}, JobDefaults{MaxCPULimit: &maxCPU}) + if !c.Resources.Limits.Cpu().Equal(small) { + t.Errorf("cpu limit = %v, want 2", c.Resources.Limits.Cpu()) + } + }) + t.Run("defaults above the max are clamped too", func(t *testing.T) { + c := first(buildv1.BuildStep{}, JobDefaults{CPULimit: &big, MemoryLimit: &bigMem, MaxCPULimit: &maxCPU, MaxMemoryLimit: &maxMem}) + if !c.Resources.Limits.Cpu().Equal(maxCPU) || !c.Resources.Limits.Memory().Equal(maxMem) { + t.Errorf("limits = %v, want 16 and 32Gi", c.Resources.Limits) + } + }) + t.Run("no max leaves the value alone", func(t *testing.T) { + c := first(buildv1.BuildStep{CPULimit: &big}, JobDefaults{}) + if !c.Resources.Limits.Cpu().Equal(big) { + t.Errorf("cpu limit = %v, want 100", c.Resources.Limits.Cpu()) + } + }) +} diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go index 783be007..7a7c30ba 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go @@ -139,10 +139,42 @@ type JobDefaults struct { // did not set. Nil leaves it unlimited. CPULimit *resource.Quantity MemoryLimit *resource.Quantity + // MaxCPULimit and MaxMemoryLimit cap what a step may ask for, limits and + // requests alike (also the defaults above). Nil = no cap. + MaxCPULimit *resource.Quantity + MaxMemoryLimit *resource.Quantity // SharedVolumeSizeLimit is the sizeLimit of the shared emptyDir. Nil = none. SharedVolumeSizeLimit *resource.Quantity } +// clampQuantity returns max when q is above it and says so in the log, so a +// job cannot ask for more than the cluster operator allows. Nil on either side +// returns q unchanged. +func clampQuantity(q, max *resource.Quantity, what, buildJob string, stepID int32) *resource.Quantity { + if q == nil || max == nil || q.Cmp(*max) <= 0 { + return q + } + slog.Info("Clamping step resource to the operator maximum", "buildJob", buildJob, "step", stepID, "resource", what, "requested", q.String(), "max", max.String()) + return max +} + +// maxCreateMessage bounds the API server message kept as the failure reason, +// which for a huge Job can list thousands of field errors. +const maxCreateMessage = 300 + +// jobNeverCreatable reports whether retrying the Job create cannot succeed +// because the API server rejects the object itself. +func jobNeverCreatable(err error) bool { + return apierrors.IsInvalid(err) || apierrors.IsBadRequest(err) || apierrors.IsRequestEntityTooLargeError(err) +} + +func truncateMessage(s string) string { + if len(s) <= maxCreateMessage { + return s + } + return strings.ToValidUTF8(s[:maxCreateMessage], "") + "..." +} + // helperResources keeps the finalizer container (it only echoes a line) small. func helperResources() corev1.ResourceRequirements { l := corev1.ResourceList{ @@ -601,7 +633,25 @@ func (r *BuildJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) (c // bj.Spec.Steps), suspended if the limit is reached. active, shouldSuspend, err := r.createJob(ctx, &bj, jobName) if err != nil { - return ctrl.Result{}, err + if !jobNeverCreatable(err) { + return ctrl.Result{}, err + } + // The API server rejects this Job itself (invalid, too many steps, too + // large), so a retry would fail the same way forever. End the BuildJob. + slog.Error("Job can never be created; failing BuildJob", "buildJob", bj.Name, "error", err) + if _, serr := r.setStatusCompleted(ctx, req.NamespacedName, bj.Name, false, "cannot create Job: "+truncateMessage(err.Error())); serr != nil { + if apierrors.IsConflict(serr) { + return ctrl.Result{RequeueAfter: conflictRequeueDelay}, nil + } + return ctrl.Result{}, serr + } + if r.DeleteOnComplete { + policy := metav1.DeletePropagationForeground + if derr := r.Delete(ctx, &bj, &client.DeleteOptions{PropagationPolicy: &policy}); derr != nil && !apierrors.IsNotFound(derr) { + return ctrl.Result{}, derr + } + } + return ctrl.Result{}, nil } // 3.3.1 Initialize container statuses @@ -915,6 +965,8 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su if memLimit == nil { memLimit = defaults.MemoryLimit } + cpuLimit = clampQuantity(cpuLimit, defaults.MaxCPULimit, "cpu limit", bj.Name, s.ID) + memLimit = clampQuantity(memLimit, defaults.MaxMemoryLimit, "memory limit", bj.Name, s.ID) if cpuLimit != nil || memLimit != nil { c.Resources.Limits = corev1.ResourceList{} if cpuLimit != nil { @@ -929,10 +981,10 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su if s.CPURequest != nil || s.MemoryRequest != nil { c.Resources.Requests = corev1.ResourceList{} if s.CPURequest != nil { - c.Resources.Requests[corev1.ResourceCPU] = *s.CPURequest + c.Resources.Requests[corev1.ResourceCPU] = *clampQuantity(s.CPURequest, defaults.MaxCPULimit, "cpu request", bj.Name, s.ID) } if s.MemoryRequest != nil { - c.Resources.Requests[corev1.ResourceMemory] = *s.MemoryRequest + c.Resources.Requests[corev1.ResourceMemory] = *clampQuantity(s.MemoryRequest, defaults.MaxMemoryLimit, "memory request", bj.Name, s.ID) } } diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go index 702fcad5..dd006398 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_events_test.go @@ -3,6 +3,7 @@ package controller import ( "context" "errors" + "strings" "testing" "time" @@ -11,11 +12,16 @@ import ( "github.com/hades-scheduler/hades/hadesScheduler/log" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/validation/field" k8sfake "k8s.io/client-go/kubernetes/fake" + ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" ) func TestBuildK8sJob_PodTemplateLabelsAndGracePeriod(t *testing.T) { @@ -505,3 +511,73 @@ func TestCreateJob_AlreadyExistsReportsExistingSuspend(t *testing.T) { t.Fatal("createJob reported suspended for an existing running Job") } } + +// A Job the API server will never accept ends the BuildJob as Failed instead of +// being retried with backoff forever; a transient error is still returned. +func TestReconcile_JobCreateErrors(t *testing.T) { + gr := schema.GroupResource{Group: "batch", Resource: "jobs"} + long := strings.Repeat("x", 5000) + cases := []struct { + name string + err error + deleteOnDone bool + wantFailed bool + }{ + {"invalid", apierrors.NewInvalid(schema.GroupKind{Group: "batch", Kind: "Job"}, "buildjob-job-1", field.ErrorList{field.TooMany(field.NewPath("spec"), 5000, 250)}), false, true}, + {"invalid with a huge message", apierrors.NewBadRequest(long), false, true}, + {"too large", apierrors.NewRequestEntityTooLargeError("too big"), false, true}, + {"failed CR is deleted on complete", apierrors.NewBadRequest("no"), true, true}, + {"timeout is retried", apierrors.NewTimeoutError("slow", 1), false, false}, + {"unavailable is retried", apierrors.NewServiceUnavailable("down"), false, false}, + {"conflict is retried", apierrors.NewConflict(gr, "buildjob-job-1", errors.New("c")), false, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + bj := &buildv1.BuildJob{ + ObjectMeta: metav1.ObjectMeta{Name: "job-1", Namespace: "ns"}, + Spec: buildv1.BuildJobSpec{Steps: []buildv1.BuildStep{{ID: 1, Image: "alpine"}}}, + } + r := newAdmissionTestReconciler(t, 1) + r.Client = interceptor.NewClient( + fake.NewClientBuilder().WithScheme(r.Scheme).WithObjects(bj).WithStatusSubresource(bj).Build(), + interceptor.Funcs{Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if _, ok := obj.(*batchv1.Job); ok { + return tc.err + } + return c.Create(ctx, obj, opts...) + }}) + // The nil NATS connection makes the publish fail, which is only logged. + r.Publisher = &log.NATSPublisher{} + r.DeleteOnComplete = tc.deleteOnDone + + _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(bj)}) + var got buildv1.BuildJob + getErr := r.Get(ctx, client.ObjectKeyFromObject(bj), &got) + if !tc.wantFailed { + if err == nil { + t.Fatal("Reconcile() = nil, want the transient error returned for a retry") + } + if getErr != nil || got.Status.Phase != "" { + t.Errorf("BuildJob phase = %q (err %v), want untouched", got.Status.Phase, getErr) + } + return + } + if err != nil { + t.Fatalf("Reconcile() = %v, want nil", err) + } + if tc.deleteOnDone { + if !apierrors.IsNotFound(getErr) { + t.Errorf("BuildJob still exists (err %v), want it deleted", getErr) + } + return + } + if getErr != nil || got.Status.Phase != "Failed" { + t.Fatalf("phase = %q (err %v), want Failed", got.Status.Phase, getErr) + } + if !strings.HasPrefix(got.Status.Message, "cannot create Job: ") || len(got.Status.Message) > maxCreateMessage+40 { + t.Errorf("message = %q (%d bytes), want a short cannot-create reason", got.Status.Message, len(got.Status.Message)) + } + }) + } +} diff --git a/Readme.md b/Readme.md index 5adaf101..0d976dd8 100644 --- a/Readme.md +++ b/Readme.md @@ -388,7 +388,7 @@ Breaking changes: - `AUTH_KEY` is required. The API exits at start with an empty key unless `ALLOW_UNAUTHENTICATED=true` (local development only). In Helm, an empty `hadesApi.authKeySecret` now needs `hadesApi.allowUnauthenticated: true`. - Callbacks (`callback_url`, `status_callback_url`) to loopback, private, CGNAT, link-local, unspecified and multicast addresses are refused. List internal receivers (for example Artemis in the same cluster or compose network) in `CALLBACK_ALLOWED_HOSTS` (Helm: `hadesLogManager.callbackAllowedHosts`). The log callback no longer follows redirects and callbacks ignore HTTP proxy variables. - Docker executor: steps may only request the network modes in `DOCKER_ALLOWED_NETWORKS` (default `none,bridge,default`). `host` and named networks fail the job unless listed. -- `/build` bodies are capped at 1 MiB (413 above) and duplicate step ids are rejected with 400. +- `/build` bodies are capped at 1 MiB (413 above) and duplicate step ids, more than 100 steps and cpu or memory limits above 1024 cores or 1T are rejected with 400. The operator also caps a step at `JOB_MAX_CPU_LIMIT` (16) and `JOB_MAX_MEMORY_LIMIT` (32Gi) and fails a job the API server cannot create instead of retrying it. - `compose.yml` requires `AUTH_KEY` in `.env` and publishes API, log manager, NATS and Jaeger on `127.0.0.1` only. `docker-compose.deploy.yml` publishes only Traefik on 80/443 (Compose 2.24 or later) and no longer routes the log manager publicly. - Kubernetes: job pods no longer mount a service account token and run with the `RuntimeDefault` seccomp profile and `allowPrivilegeEscalation: false`. The operator role has no access to secrets or configmaps. `hadesOperator.DeleteOnComplete` now defaults to `true`. Optional `networkPolicy.enabled` (default `false`) is new. - Kubernetes: jobs now get a default 1 h timeout, 2 CPU and 4Gi memory limits per step, a 10Gi shared volume and no `NET_RAW`/`MKNOD` capabilities or service-link variables. Change or disable the defaults with `hadesOperator.jobDefaultTimeoutSeconds`, `jobMaxTimeoutSeconds`, `jobDefaultCPULimit`, `jobDefaultMemoryLimit` and `jobSharedVolumeSizeLimit` (empty or `0` disables each). diff --git a/docs/api.md b/docs/api.md index 2ed6cb74..2ad0c241 100644 --- a/docs/api.md +++ b/docs/api.md @@ -58,7 +58,7 @@ curl -X POST -H "Content-Type: application/json" \ | Status | Body | Meaning | | ------ | ---- | ------- | | `200 OK` | `{ "message": "Successfully enqueued job", "job_id": "" }` | Job accepted and queued. | -| `400 Bad Request` | plain-text message naming the offending field | Invalid JSON or failed validation (e.g. missing `name`, no steps, a step id outside 1..2147483647, a step without `image`, unparseable `memory_limit`, duplicate step ids). | +| `400 Bad Request` | plain-text message naming the offending field | Invalid JSON or failed validation (e.g. missing `name`, no steps, a step id outside 1..2147483647, a step without `image`, unparseable `memory_limit`, duplicate step ids, more than 100 steps, `cpu_limit` above 1024, `memory_limit` or `memory_swap` above 1T). | | `413 Request Entity Too Large` | | The body exceeds 1 MiB. | | `500 Internal Server Error` | `Failed to enqueue job` | NATS publish failed. | diff --git a/docs/configuration.md b/docs/configuration.md index d738fbb3..1fb43ef7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -141,6 +141,8 @@ Only deployed when the scheduler runs in `operator` mode. | `JOB_MAX_TIMEOUT_SECONDS` | `0` | Upper bound for the job timeout; a larger or missing job timeout is clamped to it. `0` means no cap. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | | `JOB_DEFAULT_CPU_LIMIT` | `2` | CPU limit for each step that sets no `cpu_limit` (Kubernetes quantity; requests default to limits). Empty disables it. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | | `JOB_DEFAULT_MEMORY_LIMIT` | `4Gi` | Memory limit for each step that sets no `memory_limit` (Kubernetes quantity). Empty disables it. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | +| `JOB_MAX_CPU_LIMIT` | `16` | Largest CPU limit and request a step may have (Kubernetes quantity). A larger value, the default included, is clamped to it. Empty disables the cap. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | +| `JOB_MAX_MEMORY_LIMIT` | `32Gi` | Largest memory limit and request a step may have (Kubernetes quantity). A larger value, the default included, is clamped to it. Empty disables the cap. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | | `JOB_SHARED_VOLUME_SIZE_LIMIT` | `10Gi` | `sizeLimit` of the `emptyDir` shared between the steps of a job (`/shared`). Empty disables it. An invalid value fails operator startup. | `HadesScheduler/HadesOperator/cmd/main.go` (`OperatorConfig`) | | `DEV_MODE` | `false` | Enable the controller-runtime development logger. | `HadesScheduler/HadesOperator/cmd/main.go` | diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index 6e5409d4..2dfbb786 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -145,6 +145,8 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | hadesOperator.jobDefaultCPULimit | string | `"2"` | CPU limit (Kubernetes quantity) for each step that sets no `cpu_limit`. Empty disables the default. An invalid value fails operator startup. | | hadesOperator.jobDefaultMemoryLimit | string | `"4Gi"` | Memory limit (Kubernetes quantity) for each step that sets no `memory_limit`. Empty disables the default. An invalid value fails operator startup. | | hadesOperator.jobDefaultTimeoutSeconds | string | `"3600"` | Timeout (seconds) applied to jobs that set no `timeout_seconds`, so a hung or unschedulable job cannot hold a parallelism slot forever. Empty or 0 disables the default. | +| hadesOperator.jobMaxCPULimit | string | `"16"` | Largest CPU limit and request (Kubernetes quantity) a step may have. A larger value is clamped to it, the defaults included. Empty disables the cap. An invalid value fails operator startup. | +| hadesOperator.jobMaxMemoryLimit | string | `"32Gi"` | Largest memory limit and request (Kubernetes quantity) a step may have. A larger value is clamped to it, the defaults included. Empty disables the cap. An invalid value fails operator startup. | | hadesOperator.jobMaxTimeoutSeconds | string | `"0"` | Upper bound (seconds) for the job timeout. A larger or missing job timeout is clamped to it. Empty or 0 means no cap. | | hadesOperator.jobSharedVolumeSizeLimit | string | `"10Gi"` | Size limit of the `emptyDir` shared between the steps of a job (mounted at `/shared`). Without it a job can fill the node's disk. Empty disables the limit. An invalid value fails operator startup. | | hadesOperator.leaderElection.enabled | bool | `true` | Enable leader election for the operator (recommended with >1 replica). | diff --git a/helm/hades/crds/build.hades.tum.de_buildjobs.yaml b/helm/hades/crds/build.hades.tum.de_buildjobs.yaml index ab6fe271..36b31e50 100644 --- a/helm/hades/crds/build.hades.tum.de_buildjobs.yaml +++ b/helm/hades/crds/build.hades.tum.de_buildjobs.yaml @@ -185,6 +185,7 @@ spec: - id - image type: object + maxItems: 100 minItems: 1 type: array timeoutSeconds: diff --git a/helm/hades/templates/hades-operator-deployment.yaml b/helm/hades/templates/hades-operator-deployment.yaml index b6b5608e..b54e37bb 100644 --- a/helm/hades/templates/hades-operator-deployment.yaml +++ b/helm/hades/templates/hades-operator-deployment.yaml @@ -64,6 +64,10 @@ spec: value: {{ .Values.hadesOperator.jobDefaultCPULimit | quote }} - name: JOB_DEFAULT_MEMORY_LIMIT value: {{ .Values.hadesOperator.jobDefaultMemoryLimit | quote }} + - name: JOB_MAX_CPU_LIMIT + value: {{ .Values.hadesOperator.jobMaxCPULimit | quote }} + - name: JOB_MAX_MEMORY_LIMIT + value: {{ .Values.hadesOperator.jobMaxMemoryLimit | quote }} - name: JOB_SHARED_VOLUME_SIZE_LIMIT value: {{ .Values.hadesOperator.jobSharedVolumeSizeLimit | quote }} - name: MAX_CONCURRENT_RECONCILES diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index 8e1f3f54..97d3d888 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -178,6 +178,14 @@ hadesOperator: # `memory_limit`. Empty disables the default. An invalid value fails operator # startup. jobDefaultMemoryLimit: "4Gi" + # -- Largest CPU limit and request (Kubernetes quantity) a step may have. A + # larger value is clamped to it, the defaults included. Empty disables the + # cap. An invalid value fails operator startup. + jobMaxCPULimit: "16" + # -- Largest memory limit and request (Kubernetes quantity) a step may have. A + # larger value is clamped to it, the defaults included. Empty disables the + # cap. An invalid value fails operator startup. + jobMaxMemoryLimit: "32Gi" # -- Size limit of the `emptyDir` shared between the steps of a job (mounted at # `/shared`). Without it a job can fill the node's disk. Empty disables the # limit. An invalid value fails operator startup. diff --git a/shared/utils/config.go b/shared/utils/config.go index 76f7dda7..74d0fde7 100644 --- a/shared/utils/config.go +++ b/shared/utils/config.go @@ -3,6 +3,7 @@ package utils import ( "fmt" "log/slog" + "math" "net/url" "regexp" "strconv" @@ -66,14 +67,20 @@ func ParseMemoryLimit(limit string) (int64, error) { return 0, fmt.Errorf("memory limit cannot be negative: %d", value) } + var factor int64 switch strings.ToUpper(unit) { case "G": - return value * gigabyte, nil + factor = gigabyte case "M": - return value * megabyte, nil + factor = megabyte default: return 0, fmt.Errorf("unsupported memory unit %q: must be G (gigabytes) or M (megabytes)", unit) } + // A huge value would wrap around to a small or negative byte count. + if value > math.MaxInt64/factor { + return 0, fmt.Errorf("memory limit too large: %s", limit) + } + return value * factor, nil } // namedNetworkPattern matches a valid Docker network name: it must start with an diff --git a/shared/utils/config_test.go b/shared/utils/config_test.go index 2bb78d23..0c91d024 100644 --- a/shared/utils/config_test.go +++ b/shared/utils/config_test.go @@ -118,6 +118,8 @@ func TestParseMemoryLimit_InvalidInput(t *testing.T) { {"single character", "G", "invalid memory limit format"}, {"negative value", "-5G", "memory limit cannot be negative"}, {"no unit", "123", "unsupported memory unit"}, + {"overflowing gigabytes", "9223372036G", "memory limit too large"}, + {"overflowing megabytes", "9223372036854775807M", "memory limit too large"}, } for _, tt := range tests { diff --git a/website/docs/operation-modes/k8s-operator.md b/website/docs/operation-modes/k8s-operator.md index 68b7e1b9..1bac19f9 100644 --- a/website/docs/operation-modes/k8s-operator.md +++ b/website/docs/operation-modes/k8s-operator.md @@ -49,7 +49,7 @@ The scheduler uses a dynamic Kubernetes client to create `BuildJob` custom resou ## Job pod hardening -Job pods do not mount a service account token, use the `RuntimeDefault` seccomp profile and set `allowPrivilegeEscalation: false`. They still run as the image's user, but every container drops the `NET_RAW` and `MKNOD` capabilities and the pod gets no service-link environment variables (`enableServiceLinks: false`). The operator also applies defaults to what a job leaves unset: a 1 h timeout (`JOB_DEFAULT_TIMEOUT_SECONDS`, cap with `JOB_MAX_TIMEOUT_SECONDS`), a 2 CPU and 4Gi memory limit per step (`JOB_DEFAULT_CPU_LIMIT`, `JOB_DEFAULT_MEMORY_LIMIT`) and a 10Gi size limit on the shared volume (`JOB_SHARED_VOLUME_SIZE_LIMIT`). Set a value to empty or `0` to disable it (Helm: `hadesOperator.jobDefaultTimeoutSeconds` and friends). The API and the log manager have their own service accounts, and the operator role has no access to secrets or configmaps. +Job pods do not mount a service account token, use the `RuntimeDefault` seccomp profile and set `allowPrivilegeEscalation: false`. They still run as the image's user, but every container drops the `NET_RAW` and `MKNOD` capabilities and the pod gets no service-link environment variables (`enableServiceLinks: false`). The operator also applies defaults to what a job leaves unset: a 1 h timeout (`JOB_DEFAULT_TIMEOUT_SECONDS`, cap with `JOB_MAX_TIMEOUT_SECONDS`), a 2 CPU and 4Gi memory limit per step (`JOB_DEFAULT_CPU_LIMIT`, `JOB_DEFAULT_MEMORY_LIMIT`) and clamps any step CPU or memory limit and request to 16 CPU and 32Gi (`JOB_MAX_CPU_LIMIT`, `JOB_MAX_MEMORY_LIMIT`). A job the API server rejects as invalid or too large is marked `Failed` instead of being retried and a 10Gi size limit on the shared volume (`JOB_SHARED_VOLUME_SIZE_LIMIT`). Set a value to empty or `0` to disable it (Helm: `hadesOperator.jobDefaultTimeoutSeconds` and friends). The API and the log manager have their own service accounts, and the operator role has no access to secrets or configmaps. Set `networkPolicy.enabled: true` (default `false`) to add two NetworkPolicies: job pods cannot reach the release namespace, and only Hades components reach NATS on 4222. Before enabling, check the NATS pod labels. On CNIs that match pod IPs against `ipBlock`, set `networkPolicy.jobPods.blockedCIDRs` to the pod CIDR. diff --git a/website/docs/reference/configuration.md b/website/docs/reference/configuration.md index 8a086e34..5f1ca467 100644 --- a/website/docs/reference/configuration.md +++ b/website/docs/reference/configuration.md @@ -114,6 +114,8 @@ Only deployed when the scheduler runs in `operator` mode. | `JOB_MAX_TIMEOUT_SECONDS` | `0` | Upper bound for the job timeout; a larger or missing job timeout is clamped to it. `0` means no cap. | | `JOB_DEFAULT_CPU_LIMIT` | `2` | CPU limit for each step that sets no `cpu_limit` (Kubernetes quantity; requests default to limits). Empty disables it. An invalid value fails operator startup. | | `JOB_DEFAULT_MEMORY_LIMIT` | `4Gi` | Memory limit for each step that sets no `memory_limit` (Kubernetes quantity). Empty disables it. An invalid value fails operator startup. | +| `JOB_MAX_CPU_LIMIT` | `16` | Largest CPU limit and request a step may have (Kubernetes quantity). A larger value, the default included, is clamped to it. Empty disables the cap. An invalid value fails operator startup. | +| `JOB_MAX_MEMORY_LIMIT` | `32Gi` | Largest memory limit and request a step may have (Kubernetes quantity). A larger value, the default included, is clamped to it. Empty disables the cap. An invalid value fails operator startup. | | `JOB_SHARED_VOLUME_SIZE_LIMIT` | `10Gi` | `sizeLimit` of the `emptyDir` shared between the steps of a job (`/shared`). Empty disables it. An invalid value fails operator startup. | | `DEV_MODE` | `false` | Enable the controller-runtime development logger. | diff --git a/website/docs/usage/submitting-jobs.md b/website/docs/usage/submitting-jobs.md index 07e8d935..b219fcb6 100644 --- a/website/docs/usage/submitting-jobs.md +++ b/website/docs/usage/submitting-jobs.md @@ -18,7 +18,7 @@ http://localhost:8081 ## Authentication -`POST /build` requires HTTP Basic Auth with username `hades` and `AUTH_KEY` as the password. `AUTH_KEY` is required: with an empty key the API exits at start unless `ALLOW_UNAUTHENTICATED=true` (local development only). Request bodies are capped at 1 MiB (413 above) and an empty `steps` list, a step id outside 1..2147483647, a step without `image` or duplicate step ids get 400. +`POST /build` requires HTTP Basic Auth with username `hades` and `AUTH_KEY` as the password. `AUTH_KEY` is required: with an empty key the API exits at start unless `ALLOW_UNAUTHENTICATED=true` (local development only). Request bodies are capped at 1 MiB (413 above) and an empty `steps` list, a step id outside 1..2147483647, a step without `image` duplicate step ids, more than 100 steps, a `cpu_limit` above 1024 or a `memory_limit` or `memory_swap` above 1T get 400. ## Job Payload Structure From b70be45746f9debd4d7c2ed8ed463563326a435a Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Sat, 14 Nov 2026 02:15:48 +0100 Subject: [PATCH 5/9] Close callback, job network, body format and log memory gaps Findings from the third and fourth test round on the test environment: - Log manager: the callback guard also refuses the special-purpose ranges (198.18.0.0/15 and others) that clusters use for pods and services; CALLBACK_BLOCKED_CIDRS adds cluster-specific ranges. - NetworkPolicy: job pods reach only DNS inside the cluster, so other namespaces are no longer open to them. - API: /build decodes JSON only; form, XML, YAML and TOML bodies are rejected. - Log manager: LOG_MAX_TOTAL_BYTES bounds the logs kept in memory and evicts the oldest completed jobs first. Co-Authored-By: Claude Mythos 5.2 --- AGENTS.md | 4 +- HadesAPI/router.go | 2 +- HadesAPI/router_test.go | 19 +++ HadesLogManager/Readme.md | 11 +- HadesLogManager/http_client.go | 66 +++++++++-- HadesLogManager/http_client_test.go | 23 ++++ HadesLogManager/main.go | 6 +- HadesLogManager/processor.go | 83 +++++++++++++ HadesLogManager/processor_test.go | 109 ++++++++++++++++++ Readme.md | 2 +- compose.yml | 2 + docs/configuration.md | 4 +- helm/hades/Readme.md | 4 +- .../hades-log-manager-deployment.yaml | 4 + helm/hades/templates/networkpolicy.yaml | 16 +-- helm/hades/values.yaml | 17 ++- website/docs/reference/configuration.md | 4 +- 17 files changed, 348 insertions(+), 28 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index cb7f0593..505e6a01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ Go workspace (`go.work`, Go 1.27) with five modules: - **Status subjects:** `hades.jobstatus.{Queued,Running,Succeeded,Failed,Stopped}` (payload = job ID string, optional `X-Hades-Reason` header). `HadesAPI` publishes `Queued` on enqueue; the scheduler/operator publish `Running`/`Succeeded`/`Failed`. Nothing publishes `Stopped` today. `HadesLogManager` and the dashboard subscribe to the whole `hades.jobstatus.*` lifecycle over **core NATS**. - **Status stream:** `HADES_JOB_STATUS` is a JetStream stream over `hades.jobstatus.*` that durably captures those same core-NATS publishes. `HadesLogManager` consumes it through the durable `HADES_STATUS_WEBHOOK` consumer to deliver the **job-status webhook** (`status_callback_url`). Publishers are unchanged and core subscribers still receive every event. - **Log subjects:** see `shared/buildlogs/buildlog_stream.go`. -- **Callbacks:** `callback_url` / `status_callback_url` targets on loopback, private, CGNAT, link-local, unspecified and multicast addresses are refused at connect time (SSRF guard in `HadesLogManager/http_client.go`); `CALLBACK_ALLOWED_HOSTS` exempts internal receivers. No redirects on the log callback, proxy env ignored, a blocked status webhook is not retried. +- **Callbacks:** `callback_url` / `status_callback_url` targets on loopback, private, CGNAT, link-local, unspecified, multicast and other special-purpose (RFC 2544, documentation, NAT64, 6to4, Teredo, reserved) addresses are refused at connect time (SSRF guard in `HadesLogManager/http_client.go`); `CALLBACK_ALLOWED_HOSTS` exempts internal receivers and wins over the built-in list and over `CALLBACK_BLOCKED_CIDRS` (extra ranges to refuse for clusters with public-looking pod/service networks). No redirects on the log callback, proxy env ignored, a blocked status webhook is not retried. - **Job leases:** the scheduler acks a job only after the executor returns, and the Docker executor blocks for the whole job. `shared/nats/consumer.go` therefore heartbeats `msg.InProgress()` on a ticker while a job runs so the `NATS_ACK_WAIT` (default 1m) timer keeps resetting - job duration must never determine AckWait. `NATS_MAX_DELIVER` (default 3) caps redeliveries; the last delivery is not executed but publishes a terminal `Failed` status and terminates the message. - **Job DTO:** `shared/payload/payload.go` (`QueuePayload` with ordered `Step`s). Job-level: `timeout_seconds` (whole-job timeout). Each step has `image`, optional `script`, `metadata` env vars, `cpu_limit` (whole cores), `memory_limit`, and the Docker-only limits `network`, `memory_swap`, `pids_limit`. - **Two outbound pushes, do not conflate them:** `callback_url` forwards the aggregated **log array** and only after `stopWatchingJobLogs` drains the JetStream log consumer; `status_callback_url` receives the **job-status webhook** (`{event, job_id, name, status, reason, queued_at, started_at, finished_at, duration_ms, attempt}`) on the terminal status itself, with at-least-once retry. Neither may be repurposed as the other. @@ -81,7 +81,7 @@ The top-level `Makefile` wraps the common workflows (`make help` lists every tar - Don't bypass the operator path by writing a new direct-K8s scheduler; the operator is the strategic direction. - Don't add fields to `BuildJobSpec` without regenerating the CRD (CI will block the PR). - Don't introduce package-level mutable globals for dependencies - the codebase has been moving away from them (see `setupRouter` taking `JobPublisher` as a parameter). -- `HadesLogManager` is deployed (Helm + image) but ClusterIP-internal and single-replica (in-memory aggregation); don't put it on the ingress or scale it past one replica. The status-webhook dispatcher living there keeps no durable state in memory (its retry schedule is in JetStream), but it still assumes one replica. The same single-replica constraint applies to the API's in-memory dashboard read-model. (It just isn't in the `compose.yml` stack.) +- `HadesLogManager` is deployed (Helm + image) but ClusterIP-internal and single-replica (in-memory aggregation, bounded by `LOG_MAX_TOTAL_BYTES`: oldest completed jobs lose their logs first); don't put it on the ingress or scale it past one replica. The status-webhook dispatcher living there keeps no durable state in memory (its retry schedule is in JetStream), but it still assumes one replica. The same single-replica constraint applies to the API's in-memory dashboard read-model. (It just isn't in the `compose.yml` stack.) - NATS itself still has no authentication; it must not be published beyond loopback (compose binds it to `127.0.0.1`, the deploy stack does not publish it). Never list the NATS network in `DOCKER_ALLOWED_NETWORKS`. - Don't re-open callbacks to internal addresses or add a redirect-following/proxy-aware callback client; use `CALLBACK_ALLOWED_HOSTS`. - Job pods must keep no service account token, `RuntimeDefault` seccomp, `allowPrivilegeEscalation: false`, `enableServiceLinks: false` and dropped `NET_RAW`/`MKNOD` capabilities (not `ALL`: grading images install packages as root); the operator role gets no secrets/configmaps access. diff --git a/HadesAPI/router.go b/HadesAPI/router.go index 5598f4d9..17705b3d 100644 --- a/HadesAPI/router.go +++ b/HadesAPI/router.go @@ -176,7 +176,7 @@ func addBuildToQueue(c *gin.Context, producer hades.JobPublisher, statusPublishe var p payload.RESTPayload p.Priority = 3 c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxBuildBodyBytes) - if err := c.ShouldBind(&p); err != nil { + if err := c.ShouldBindJSON(&p); err != nil { var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { buildRequestsTotal.WithLabelValues("rejected").Inc() diff --git a/HadesAPI/router_test.go b/HadesAPI/router_test.go index f28f63ae..83aa11dc 100644 --- a/HadesAPI/router_test.go +++ b/HadesAPI/router_test.go @@ -379,6 +379,25 @@ func (suite *APISuite) TestDuplicateStepIDs() { assert.Contains(suite.T(), w.Body.String(), "Duplicate step id 1") } +// /build is a JSON API: a form, XML or TOML body must not be decoded into a job. +func (suite *APISuite) TestNonJSONBodyRejected() { + cases := map[string]string{ + "application/x-www-form-urlencoded": "name=x&Name=x", + "application/xml": "x", + "application/toml": `Name = "x"`, + } + for contentType, body := range cases { + suite.Run(contentType, func() { + w := httptest.NewRecorder() + req, _ := http.NewRequest("POST", "/build", strings.NewReader(body)) + req.Header.Set("Content-Type", contentType) + suite.router.ServeHTTP(w, req) + assert.Equal(suite.T(), 400, w.Code) + assert.Contains(suite.T(), w.Body.String(), "Invalid JSON body") + }) + } +} + func (suite *APISuite) TestInvalidStepsRejected() { cases := []struct { name string diff --git a/HadesLogManager/Readme.md b/HadesLogManager/Readme.md index fb7ed77c..97d482bb 100644 --- a/HadesLogManager/Readme.md +++ b/HadesLogManager/Readme.md @@ -227,9 +227,11 @@ live feed - continue to receive every event unchanged. | `HADESLOGMANAGER_API_PORT` | `8081` | HTTP API port. | | `LOG_BATCH_SIZE` | `100` | Log entries buffered before a flush. | | `LOG_RETENTION` | `1h` | How long completed job logs are kept in memory.| +| `LOG_MAX_TOTAL_BYTES` | `134217728` | Total log message bytes kept across all jobs (`0` = unbounded). Past it the logs of the oldest completed jobs are evicted (status stays; `GET /jobs/:id/logs` then returns a "logs evicted" entry); if only running jobs hold the budget, new lines are dropped. Heap use is a few times this value. | | `MAX_JOB_LOGS` | `1000` | Max log entries retained per container of a job (oldest drop first; container slots are never removed). | | `LOG_WATCHER_MAX_LIFETIME` | `6h` | Longest a job's live log watcher runs without a terminal status, for jobs without `timeout_seconds` (jobs with one use it plus 10m). Also used when the job's `HADES_JOBS` entry can't be read. Past it, live logs stop updating; the full log is replayed when the job finishes. | | `CALLBACK_ALLOWED_HOSTS` | | Comma-separated host names, IPs or CIDRs exempt from the callback address block (see below). | +| `CALLBACK_BLOCKED_CIDRS` | | Comma-separated CIDRs to refuse on top of the built-in list (see below). | | `DEBUG` | `false` | Enable verbose (debug-level) logging. | The forwarding destination is no longer a global env var. Each job sets its own @@ -242,11 +244,16 @@ are not forwarded. Callbacks (`callback_url`, `status_callback_url`) to loopback, private (RFC 1918, `fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), -unspecified and multicast addresses are refused at connect time. Set +unspecified, multicast and other special-purpose addresses (benchmarking +`198.18.0.0/15`, documentation, reserved, NAT64, 6to4, Teredo) are refused at +connect time. The built-in list cannot know a cluster's pod and service ranges +when those use public-looking addresses; list them in `CALLBACK_BLOCKED_CIDRS` +(Helm: `hadesLogManager.callbackBlockedCIDRs`). Set `CALLBACK_ALLOWED_HOSTS` (Helm: `hadesLogManager.callbackAllowedHosts`) to exempt internal receivers such as Artemis in the same cluster or compose network. A host name entry must equal the host in the callback URL and trusts whatever it resolves -to. The log callback does not follow redirects, and HTTP proxy environment +to. An allowlist entry wins over both the built-in list and +`CALLBACK_BLOCKED_CIDRS`. The log callback does not follow redirects, and HTTP proxy environment variables are ignored. A blocked status webhook is logged as an error and not retried. diff --git a/HadesLogManager/http_client.go b/HadesLogManager/http_client.go index 4d9b3a57..ab46310d 100644 --- a/HadesLogManager/http_client.go +++ b/HadesLogManager/http_client.go @@ -14,26 +14,75 @@ import ( ) // CallbackConfig configures which callback targets the log manager may reach. -// Loopback, private, link-local, unspecified and multicast addresses are -// refused unless listed here. +// Loopback, private, link-local, unspecified, multicast and special-purpose +// addresses are refused unless listed in AllowedHosts. type CallbackConfig struct { // AllowedHosts is a comma-separated list of host names, IPs or CIDRs // (e.g. "artemis,10.0.0.0/8") that callbacks may reach despite the block. AllowedHosts []string `env:"CALLBACK_ALLOWED_HOSTS" envSeparator:","` + // BlockedCIDRs adds ranges to refuse, for clusters whose pod or service + // network uses addresses that look public. An AllowedHosts entry still + // wins over it. + BlockedCIDRs []string `env:"CALLBACK_BLOCKED_CIDRS" envSeparator:","` } // errBlockedTarget marks a callback target the dialer refused. Retrying cannot // fix it, so callers treat it as permanent. var errBlockedTarget = errors.New("callback target is not allowed") -// cgnat is the shared address space (RFC 6598). It is not public and some -// clouds serve instance metadata from it. -var cgnat = netip.MustParsePrefix("100.64.0.0/10") +// specialPurpose lists the non-global ranges the net.IP helpers do not cover. +// Clusters often use them for pod and service networks. +var specialPurpose = []netip.Prefix{ + netip.MustParsePrefix("0.0.0.0/8"), // "this network" + netip.MustParsePrefix("100.64.0.0/10"), // CGNAT (RFC 6598), some clouds serve metadata here + netip.MustParsePrefix("192.0.0.0/24"), // IETF protocol assignments + netip.MustParsePrefix("192.0.2.0/24"), // documentation (TEST-NET-1) + netip.MustParsePrefix("192.88.99.0/24"), // deprecated 6to4 relay anycast + netip.MustParsePrefix("198.18.0.0/15"), // benchmarking, used for pod and service networks + netip.MustParsePrefix("198.51.100.0/24"), // documentation (TEST-NET-2) + netip.MustParsePrefix("203.0.113.0/24"), // documentation (TEST-NET-3) + netip.MustParsePrefix("240.0.0.0/4"), // reserved, includes broadcast + netip.MustParsePrefix("64:ff9b::/96"), // NAT64, maps to an IPv4 target + netip.MustParsePrefix("64:ff9b:1::/48"), // local-use NAT64, maps to an IPv4 target + netip.MustParsePrefix("100::/64"), // discard-only + netip.MustParsePrefix("2001::/32"), // Teredo + netip.MustParsePrefix("2001:db8::/32"), // documentation + netip.MustParsePrefix("2002::/16"), // 6to4 +} -// callbackAllowlist holds the operator's exemptions from the target block. +// callbackAllowlist holds the operator's exemptions from the target block and +// any extra ranges to refuse. type callbackAllowlist struct { hosts map[string]bool prefixes []netip.Prefix + blocked []netip.Prefix +} + +// blockCIDRs adds operator-configured CIDRs to the refused ranges. +func (a *callbackAllowlist) blockCIDRs(entries []string) error { + for _, e := range entries { + e = strings.TrimSpace(e) + if e == "" { + continue + } + p, err := netip.ParsePrefix(e) + if err != nil { + return fmt.Errorf("invalid CALLBACK_BLOCKED_CIDRS entry %q", e) + } + a.blocked = append(a.blocked, p.Masked()) + } + return nil +} + +func isSpecialPurpose(ip netip.Addr, extra []netip.Prefix) bool { + for _, list := range [][]netip.Prefix{specialPurpose, extra} { + for _, p := range list { + if p.Contains(ip) { + return true + } + } + } + return false } func parseCallbackAllowlist(entries []string) (*callbackAllowlist, error) { @@ -65,7 +114,8 @@ func (a *callbackAllowlist) allowsIP(ip netip.Addr) bool { return false } -// control runs after DNS resolution, once per connection attempt, so DNS +// control refuses blocked ranges, but an allowlist entry wins over both the +// built-in list and CALLBACK_BLOCKED_CIDRS. It runs after DNS resolution, once per connection attempt, so DNS // rebinding and redirects cannot reach a blocked address. func (a *callbackAllowlist) control(_, address string, _ syscall.RawConn) error { ap, err := netip.ParseAddrPort(address) @@ -76,7 +126,7 @@ func (a *callbackAllowlist) control(_, address string, _ syscall.RawConn) error if a.allowsIP(ip) { return nil } - if ip.IsLoopback() || ip.IsPrivate() || cgnat.Contains(ip) || ip.IsLinkLocalUnicast() || ip.IsUnspecified() || ip.IsMulticast() { + if ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsUnspecified() || ip.IsMulticast() || isSpecialPurpose(ip, a.blocked) { return fmt.Errorf("%w: %s", errBlockedTarget, ip) } return nil diff --git a/HadesLogManager/http_client_test.go b/HadesLogManager/http_client_test.go index 2eccf09a..584032d3 100644 --- a/HadesLogManager/http_client_test.go +++ b/HadesLogManager/http_client_test.go @@ -34,6 +34,11 @@ func TestCallbackControlBlocksInternalAddresses(t *testing.T) { "127.0.0.1:80", "[::1]:80", "10.1.2.3:80", "172.16.0.1:80", "192.168.1.1:80", "[fc00::1]:80", "169.254.169.254:80", "[fe80::1]:80", "0.0.0.0:80", "224.0.0.1:80", "[::ffff:127.0.0.1]:80", "100.100.100.200:80", + "0.1.2.3:80", "192.0.0.8:80", "192.0.2.1:80", "192.88.99.1:80", + "198.18.0.5:80", "198.19.238.201:8081", "[::ffff:198.19.0.1]:80", "198.51.100.1:80", + "203.0.113.1:80", "240.0.0.1:80", "255.255.255.255:80", + "[64:ff9b::a9fe:a9fe]:80", "[64:ff9b:1::1]:80", "[100::1]:80", + "[2001:db8::1]:80", "[2002:a9fe:a9fe::1]:80", "[2001::1]:80", } { assert.ErrorIs(t, allow.control("tcp", addr, nil), errBlockedTarget, addr) } @@ -54,6 +59,24 @@ func TestCallbackAllowlistExemptsListedTargets(t *testing.T) { assert.True(t, allow.allowsIP(netip.MustParseAddr("10.0.0.1"))) } +func TestCallbackBlockedCIDRs(t *testing.T) { + allow, err := parseCallbackAllowlist(nil) + require.NoError(t, err) + require.NoError(t, allow.blockCIDRs([]string{" 93.184.216.0/24 ", ""})) + assert.ErrorIs(t, allow.control("tcp", "93.184.216.34:443", nil), errBlockedTarget) + assert.NoError(t, allow.control("tcp", "8.8.8.8:443", nil)) + + // an allowlist entry wins over a blocked CIDR + allow, err = parseCallbackAllowlist([]string{"93.184.216.34"}) + require.NoError(t, err) + require.NoError(t, allow.blockCIDRs([]string{"93.184.216.0/24"})) + assert.NoError(t, allow.control("tcp", "93.184.216.34:443", nil)) + assert.ErrorIs(t, allow.control("tcp", "93.184.216.35:443", nil), errBlockedTarget) + + assert.Error(t, allow.blockCIDRs([]string{"not-a-cidr"})) + assert.Error(t, allow.blockCIDRs([]string{"10.0.0.1"})) +} + func TestParseCallbackAllowlistRejectsBadCIDR(t *testing.T) { _, err := parseCallbackAllowlist([]string{"10.0.0.0/99"}) assert.Error(t, err) diff --git a/HadesLogManager/main.go b/HadesLogManager/main.go index e683a71a..1d1e0404 100644 --- a/HadesLogManager/main.go +++ b/HadesLogManager/main.go @@ -125,7 +125,10 @@ func run(cfg HadesLogManagerConfig) error { if err != nil { return err } - slog.Info("Callback configuration", "allowed_hosts", callbackCfg.AllowedHosts) + if err := allow.blockCIDRs(callbackCfg.BlockedCIDRs); err != nil { + return err + } + slog.Info("Callback configuration", "allowed_hosts", callbackCfg.AllowedHosts, "blocked_cidrs", callbackCfg.BlockedCIDRs) transport := newCallbackTransport(webhookCfg.Concurrency, allow) // Create log aggregator @@ -137,6 +140,7 @@ func run(cfg HadesLogManagerConfig) error { "batch_size", aggregatorConfig.BatchSize, "retention", aggregatorConfig.Retention, "max_job_logs", aggregatorConfig.MaxJobLogs, + "max_total_bytes", aggregatorConfig.MaxTotalBytes, ) logClient := &http.Client{Timeout: httpClientTimeout, Transport: transport, CheckRedirect: noRedirect} logAggregator := NewLogAggregator(ctx, consumer, resolver, logClient, aggregatorConfig) diff --git a/HadesLogManager/processor.go b/HadesLogManager/processor.go index 37f6e9c8..cae1f71f 100644 --- a/HadesLogManager/processor.go +++ b/HadesLogManager/processor.go @@ -27,6 +27,11 @@ const ( // A job still running past it loses only its in-memory copy: its terminal // status replays the logs from JetStream. staleJobTTL = 24 * time.Hour + // evictedContainerID names the placeholder slot that replaces the logs of a + // job evicted for the memory budget, so a reader sees why the logs are gone + // instead of an empty job. + evictedContainerID = "hades-log-manager" + evictedMessage = "logs evicted: log manager memory budget reached" ) // NATSLogAggregator implements LogAggregator using in-memory storage for fast log retrieval. @@ -41,6 +46,13 @@ type NATSLogAggregator struct { targets sync.Map // jobID (string) -> logTarget, resolved when the watcher starts client *http.Client cfg AggregatorConfig + + // mu guards the byte accounting below and serializes AddLog, eviction, flush + // and reset so the total never drifts. Readers stay lock-free on the sync.Maps. + mu sync.Mutex + jobBytes map[string]int // jobID -> stored message bytes + totalBytes int + dropWarned map[string]bool // jobs whose new lines are being dropped (warned once) } // statusEntry records when a status was last set so jobs that never complete @@ -61,6 +73,8 @@ type AggregatorConfig struct { BatchSize int `env:"LOG_BATCH_SIZE" envDefault:"100"` Retention time.Duration `env:"LOG_RETENTION" envDefault:"1h"` MaxJobLogs int `env:"MAX_JOB_LOGS" envDefault:"1000"` + // MaxTotalBytes bounds the stored log message bytes across all jobs; 0 disables it. + MaxTotalBytes int `env:"LOG_MAX_TOTAL_BYTES" envDefault:"134217728"` } // NewLogAggregator creates a new NATS-based LogAggregator instance with the specified configuration. @@ -94,6 +108,9 @@ func NewLogAggregator(ctx context.Context, hlc *buildlogs.HadesLogConsumer, reso resolver: resolver, client: client, cfg: config, + + jobBytes: make(map[string]int), + dropWarned: make(map[string]bool), } // Start background cleanup goroutine @@ -153,6 +170,17 @@ func (la *NATSLogAggregator) AddLog(log buildlogs.Log) { slog.Debug("Adding log to aggregator", "job_id", log.JobID, "container_id", log.ContainerID, "entries", len(log.Logs)) + la.mu.Lock() + defer la.mu.Unlock() + if la.cfg.MaxTotalBytes > 0 && !la.makeRoom(log.JobID, entriesBytes(log.Logs)) { + // Keep the container slot (steps are positional) but drop the lines. + if !la.dropWarned[log.JobID] { + la.dropWarned[log.JobID] = true + slog.Warn("Log memory budget reached by running jobs; dropping new log lines", "job_id", log.JobID, "budget", la.cfg.MaxTotalBytes) + } + log.Logs = nil + } + // Use LoadOrStore and CompareAndSwap for thread-safe updates. Load first so // the empty placeholder is only allocated for a job's very first log. for { @@ -198,6 +226,12 @@ func (la *NATSLogAggregator) AddLog(log buildlogs.Log) { // Atomically swap if the value hasn't changed if la.logs.CompareAndSwap(log.JobID, old, newVal) { + b := 0 + for _, l := range newLogs { + b += entriesBytes(l.Logs) + } + la.totalBytes += b - la.jobBytes[log.JobID] + la.jobBytes[log.JobID] = b slog.Debug("Merged log into aggregator", "job_id", log.JobID, "container_id", log.ContainerID, "containers", len(newLogs)) break } @@ -205,6 +239,49 @@ func (la *NATSLogAggregator) AddLog(log buildlogs.Log) { } } +func entriesBytes(entries []buildlogs.LogEntry) int { + n := 0 + for _, e := range entries { + n += len(e.Message) + } + return n +} + +// makeRoom evicts the logs of the oldest completed jobs (never keep's own) until +// need more bytes fit in the budget. It reports false when nothing evictable is +// left and the budget would still be exceeded. Callers hold la.mu. +func (la *NATSLogAggregator) makeRoom(keep string, need int) bool { + for la.totalBytes+need > la.cfg.MaxTotalBytes { + var oldestID string + var oldest time.Time + la.completed.Range(func(key, value any) bool { + id, at := key.(string), value.(time.Time) + if id != keep && la.jobBytes[id] > 0 && (oldestID == "" || at.Before(oldest)) { + oldestID, oldest = id, at + } + return true + }) + if oldestID == "" { + return false + } + freed := la.jobBytes[oldestID] + marker := []buildlogs.Log{{JobID: oldestID, ContainerID: evictedContainerID, Logs: []buildlogs.LogEntry{ + {Timestamp: time.Now(), Message: evictedMessage, OutputStream: "stderr"}, + }}} + la.logs.Store(oldestID, logsVersion{ptr: &marker}) + la.release(oldestID) + slog.Warn("Evicted completed job logs for memory budget", "job_id", oldestID, "freed_bytes", freed) + } + return true +} + +// release forgets a job's byte accounting. Callers hold la.mu. +func (la *NATSLogAggregator) release(jobID string) { + la.totalBytes -= la.jobBytes[jobID] + delete(la.jobBytes, jobID) + delete(la.dropWarned, jobID) +} + // trimContainerEntries caps a single container's entries at MaxJobLogs, dropping // the oldest entries of that container. It never drops a whole container element, // which would shift step indices and break the Artemis logs[index] contract. @@ -226,6 +303,9 @@ func (la *NATSLogAggregator) trimContainerEntries(jobID, containerID string, ent // Returns: // - error: Always nil in current implementation func (la *NATSLogAggregator) FlushJob(jobID string) error { + la.mu.Lock() + la.release(jobID) + la.mu.Unlock() value, logsExists := la.logs.LoadAndDelete(jobID) if logsExists { v := value.(logsVersion) @@ -250,6 +330,9 @@ func (la *NATSLogAggregator) FlushJob(jobID string) error { // mark, since a job that is watched again is running and the retention sweep // would otherwise flush it mid-run. func (la *NATSLogAggregator) ResetJobLogs(jobID string) { + la.mu.Lock() + la.release(jobID) + la.mu.Unlock() la.logs.Delete(jobID) la.completed.Delete(jobID) } diff --git a/HadesLogManager/processor_test.go b/HadesLogManager/processor_test.go index bf9b2039..b70894dd 100644 --- a/HadesLogManager/processor_test.go +++ b/HadesLogManager/processor_test.go @@ -7,12 +7,14 @@ import ( "io" "net/http" "net/http/httptest" + "strings" "sync" "sync/atomic" "testing" "time" "github.com/hades-scheduler/hades/shared/buildlogs" + "github.com/hades-scheduler/hades/shared/buildstatus" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -337,3 +339,110 @@ func TestSendJobLogs_BlocksLoopbackByDefault(t *testing.T) { require.ErrorIs(t, agg.SendJobLogs(context.Background(), "job-1"), errBlockedTarget) assert.Equal(t, int32(0), atomic.LoadInt32(&hits)) } + +func newBudgetAggregator(t *testing.T, budget int) *NATSLogAggregator { + t.Helper() + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + cfg := AggregatorConfig{BatchSize: 100, Retention: time.Hour, MaxJobLogs: 1000, MaxTotalBytes: budget} + return NewLogAggregator(ctx, nil, nil, nil, cfg) +} + +func addBytes(agg *NATSLogAggregator, jobID string, n int) { + agg.AddLog(buildlogs.Log{JobID: jobID, ContainerID: "c", Logs: []buildlogs.LogEntry{logEntry(strings.Repeat("x", n))}}) +} + +func completeJob(agg *NATSLogAggregator, jobID string, at time.Time) { + agg.UpdateJobStatus(jobID, "Succeeded") + agg.completed.Store(jobID, at) +} + +func isEvicted(agg *NATSLogAggregator, jobID string) bool { + l := agg.GetJobLogs(jobID) + return len(l) == 1 && l[0].ContainerID == evictedContainerID && l[0].Logs[0].Message == evictedMessage +} + +func TestLogBudget_EvictsOldestCompletedFirst(t *testing.T) { + agg := newBudgetAggregator(t, 100) + now := time.Now() + addBytes(agg, "old", 40) + addBytes(agg, "new", 40) + completeJob(agg, "new", now) + completeJob(agg, "old", now.Add(-time.Minute)) + + addBytes(agg, "run", 40) // 120 > 100: only the oldest completed job must go + + assert.True(t, isEvicted(agg, "old")) + assert.False(t, isEvicted(agg, "new")) + assert.Equal(t, 80, agg.totalBytes) + assert.LessOrEqual(t, agg.totalBytes, 100) +} + +func TestLogBudget_KeepsRunningLogsAndStatusAfterEviction(t *testing.T) { + agg := newBudgetAggregator(t, 100) + addBytes(agg, "run", 50) + addBytes(agg, "done", 40) + completeJob(agg, "done", time.Now()) + + addBytes(agg, "run", 30) + + assert.True(t, isEvicted(agg, "done"), "the eviction marker is visible instead of an empty log") + s, err := agg.GetJobStatus("done") + assert.NoError(t, err) + assert.Equal(t, buildstatus.JobStatus("Succeeded"), s) + logs := agg.GetJobLogs("run") + assert.Len(t, logs, 1) + assert.Len(t, logs[0].Logs, 2, "running job keeps all its lines") +} + +func TestLogBudget_DisabledWithZero(t *testing.T) { + agg := newBudgetAggregator(t, 0) + addBytes(agg, "a", 1000) + completeJob(agg, "a", time.Now()) + addBytes(agg, "b", 1000) + assert.False(t, isEvicted(agg, "a")) + assert.Len(t, agg.GetJobLogs("b")[0].Logs, 1) +} + +func TestLogBudget_DropsNewLinesWhenOnlyRunningJobsHoldBudget(t *testing.T) { + agg := newBudgetAggregator(t, 100) + addBytes(agg, "r1", 80) + addBytes(agg, "r2", 40) // does not fit, nothing completed to evict + addBytes(agg, "r2", 40) + + assert.Equal(t, 80, agg.totalBytes) + logs := agg.GetJobLogs("r2") + assert.Len(t, logs, 1, "the container slot is still registered") + assert.Empty(t, logs[0].Logs) + assert.Len(t, agg.GetJobLogs("r1")[0].Logs, 1) +} + +func TestLogBudget_FlushAndResetReleaseBytes(t *testing.T) { + agg := newBudgetAggregator(t, 100) + addBytes(agg, "a", 30) + addBytes(agg, "b", 30) + assert.Equal(t, 60, agg.totalBytes) + agg.ResetJobLogs("a") + assert.NoError(t, agg.FlushJob("b")) + assert.Equal(t, 0, agg.totalBytes) +} + +func TestLogBudget_ConcurrentAddsNoRaceAndWithinBudget(t *testing.T) { + agg := newBudgetAggregator(t, 500) + var wg sync.WaitGroup + for i := 0; i < 8; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + id := fmt.Sprintf("job-%d", i) + for j := 0; j < 50; j++ { + addBytes(agg, id, 10) + } + completeJob(agg, id, time.Now()) + }(i) + } + wg.Wait() + agg.mu.Lock() + defer agg.mu.Unlock() + assert.LessOrEqual(t, agg.totalBytes, 500) +} diff --git a/Readme.md b/Readme.md index 0d976dd8..4619fe14 100644 --- a/Readme.md +++ b/Readme.md @@ -386,7 +386,7 @@ See **[docs/configuration.md](./docs/configuration.md)** for the complete, per-c Breaking changes: - `AUTH_KEY` is required. The API exits at start with an empty key unless `ALLOW_UNAUTHENTICATED=true` (local development only). In Helm, an empty `hadesApi.authKeySecret` now needs `hadesApi.allowUnauthenticated: true`. -- Callbacks (`callback_url`, `status_callback_url`) to loopback, private, CGNAT, link-local, unspecified and multicast addresses are refused. List internal receivers (for example Artemis in the same cluster or compose network) in `CALLBACK_ALLOWED_HOSTS` (Helm: `hadesLogManager.callbackAllowedHosts`). The log callback no longer follows redirects and callbacks ignore HTTP proxy variables. +- Callbacks (`callback_url`, `status_callback_url`) to loopback, private, CGNAT, link-local, unspecified, multicast and other special-purpose addresses (for example `198.18.0.0/15`) are refused. List internal receivers (for example Artemis in the same cluster or compose network) in `CALLBACK_ALLOWED_HOSTS` (Helm: `hadesLogManager.callbackAllowedHosts`). The log callback no longer follows redirects and callbacks ignore HTTP proxy variables. The built-in list cannot know a cluster's pod and service ranges when those use public-looking addresses; list them in `CALLBACK_BLOCKED_CIDRS` (comma-separated CIDRs, Helm: `hadesLogManager.callbackBlockedCIDRs`). An entry in `CALLBACK_ALLOWED_HOSTS` still wins over it. - Docker executor: steps may only request the network modes in `DOCKER_ALLOWED_NETWORKS` (default `none,bridge,default`). `host` and named networks fail the job unless listed. - `/build` bodies are capped at 1 MiB (413 above) and duplicate step ids, more than 100 steps and cpu or memory limits above 1024 cores or 1T are rejected with 400. The operator also caps a step at `JOB_MAX_CPU_LIMIT` (16) and `JOB_MAX_MEMORY_LIMIT` (32Gi) and fails a job the API server cannot create instead of retrying it. - `compose.yml` requires `AUTH_KEY` in `.env` and publishes API, log manager, NATS and Jaeger on `127.0.0.1` only. `docker-compose.deploy.yml` publishes only Traefik on 80/443 (Compose 2.24 or later) and no longer routes the log manager publicly. diff --git a/compose.yml b/compose.yml index ae53500f..f260e22f 100644 --- a/compose.yml +++ b/compose.yml @@ -71,6 +71,8 @@ services: # Internal callback receivers (host names, IPs, CIDRs) the log manager may # POST to; empty = public addresses only. - CALLBACK_ALLOWED_HOSTS=${CALLBACK_ALLOWED_HOSTS:-} + # Extra CIDRs to refuse, for networks that use public-looking addresses. + - CALLBACK_BLOCKED_CIDRS=${CALLBACK_BLOCKED_CIDRS:-} - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317 nats: diff --git a/docs/configuration.md b/docs/configuration.md index 1fb43ef7..08d15123 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -158,6 +158,7 @@ Deployed by the Helm chart (`hades-log-manager`); also run locally via `make run | `METRICS_PORT` | `8082` | Port the Prometheus `/metrics` endpoint listens on (see [Metrics](#metrics-all-components)). | `HadesLogManager/main.go` (`HadesLogManagerConfig`) | | `LOG_BATCH_SIZE` | `100` | Log entries buffered before a flush. | `HadesLogManager/processor.go` (`AggregatorConfig`) | | `LOG_RETENTION` | `1h` | How long completed-job logs are kept in memory (Go duration). | `HadesLogManager/processor.go` | +| `LOG_MAX_TOTAL_BYTES` | `134217728` | Total log message bytes kept in memory across all jobs; oldest completed jobs are evicted first, `0` disables. | `HadesLogManager/processor.go` | | `MAX_JOB_LOGS` | `1000` | Max log entries retained per job. | `HadesLogManager/processor.go` | | `LOG_WATCHER_MAX_LIFETIME` | `6h` | Longest a job's log watcher lives without seeing a terminal status, for jobs without `timeout_seconds` (jobs with one use it plus 10m). On expiry the watcher just stops, so a missed status event cannot leak it; the terminal status then replays the full log from JetStream before forwarding it. | `HadesLogManager/log_subscriber.go` | | `STATUS_WEBHOOK_ENABLED` | `true` | Deliver the job-status webhook. Inert for jobs without a `status_callback_url`. | `HadesLogManager/status_webhook.go` (`StatusWebhookConfig`) | @@ -169,8 +170,9 @@ Deployed by the Helm chart (`hades-log-manager`); also run locally via `make run | `STATUS_WEBHOOK_MAX_PENDING` | `1000` | Status events awaiting acknowledgement (in flight or backing off). | `HadesLogManager/status_webhook.go` | | `CALLBACK_ALLOWED_HOSTS` | | Comma-separated host names, IPs or CIDRs exempt from the callback address block (internal receivers such as Artemis in the same cluster or compose network). A host name entry must equal the host in the callback URL and trusts whatever it resolves to. | `HadesLogManager/http_client.go` | +| `CALLBACK_BLOCKED_CIDRS` | | Comma-separated CIDRs to refuse on top of the built-in list. Use it when the cluster's pod or service network uses public-looking addresses, which the built-in list cannot know. An entry in `CALLBACK_ALLOWED_HOSTS` still wins. | `HadesLogManager/http_client.go` | -Both outbound pushes are configured per job, not globally. Callbacks to loopback, private (RFC 1918, `fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), unspecified and multicast addresses are refused at connect time unless allowed by `CALLBACK_ALLOWED_HOSTS`. The log callback does not follow redirects, and HTTP proxy environment variables are ignored. A blocked status webhook is logged as an error and not retried. +Both outbound pushes are configured per job, not globally. Callbacks to loopback, private (RFC 1918, `fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), unspecified, multicast and other special-purpose addresses (such as `198.18.0.0/15`) are refused at connect time unless allowed by `CALLBACK_ALLOWED_HOSTS`. The log callback does not follow redirects, and HTTP proxy environment variables are ignored. A blocked status webhook is logged as an error and not retried. - `callback_url` - an absolute `http`/`https` URL with a host. The Log Manager forwards that job's aggregated **logs** there once the log stream has drained. If omitted, the job's logs are not forwarded. - `status_callback_url` - an absolute `http`/`https` URL with a host. Receives the **job-status webhook** on the terminal status, independently of log forwarding. See [`HadesLogManager/Readme.md`](../HadesLogManager/Readme.md#job-status-webhook). diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index 2dfbb786..56681061 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -126,7 +126,9 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | hadesApi.service.port | int | `8080` | Service port for the API. | | hadesApi.service.targetPort | int | `8080` | Container port the API listens on. | | hadesLogManager.callbackAllowedHosts | string | `""` | Comma-separated host names, IPs or CIDRs that `callback_url` and `status_callback_url` may target although they are loopback, private or link-local, e.g. the in-cluster Artemis service (`artemis.artemis.svc.cluster.local`). Empty = public addresses only. | +| hadesLogManager.callbackBlockedCIDRs | string | `""` | Comma-separated CIDRs that `callback_url` and `status_callback_url` must not target, on top of the built-in list. Use it when the cluster's pod or service network uses public-looking addresses. An entry in `callbackAllowedHosts` still wins. Empty = built-in list only. | | hadesLogManager.image.pullPolicy | string | `"Always"` | Image pull policy for the log manager. | +| hadesLogManager.logMaxTotalBytes | string | `"134217728"` | Total bytes of job log messages the log manager keeps in memory across all jobs. When it is reached, the logs of the oldest completed jobs are evicted (their status stays). Heap use is a few times this value, so it must stay well below the memory limit. "0" disables the bound. | | hadesLogManager.replicaCount | int | `1` | Keep at 1: the log manager aggregates per-job logs in memory and uses plain (non-queue, non-durable) NATS subscriptions, so multiple replicas would independently forward duplicate/split logs to the Artemis adapter. | | hadesLogManager.resources | object | `{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"200m","memory":"256Mi"}}` | Resource requests and limits for the log manager pod. | | hadesLogManager.service.port | int | `8081` | Service port for the log manager. | @@ -180,7 +182,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | nats.port | int | `4222` | NATS client port. | | networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | | networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | -| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach, on top of every pod in the release namespace. Set this to the cluster pod CIDR on CNIs that match pod IPs against ipBlock rules, otherwise the "outside" rule can still reach pods. | +| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. | | networkPolicy.nats.podLabels | object | `{}` | Labels of the NATS pods. Defaults match the bundled nats chart (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). | | rbac.hadesApi.serviceAccountName | string | `"hades-api"` | ServiceAccount name for the API (no Kubernetes API access, token not mounted). | | rbac.hadesLogManager.serviceAccountName | string | `"hades-log-manager"` | ServiceAccount name for the log manager (no Kubernetes API access, token not mounted). | diff --git a/helm/hades/templates/hades-log-manager-deployment.yaml b/helm/hades/templates/hades-log-manager-deployment.yaml index 6d1adbb0..6f9893fa 100644 --- a/helm/hades/templates/hades-log-manager-deployment.yaml +++ b/helm/hades/templates/hades-log-manager-deployment.yaml @@ -54,8 +54,12 @@ spec: value: "nats://{{ .Values.nats.host }}:{{ .Values.nats.port }}" - name: HADESLOGMANAGER_API_PORT value: {{ .Values.hadesLogManager.service.targetPort | quote }} + - name: LOG_MAX_TOTAL_BYTES + value: {{ .Values.hadesLogManager.logMaxTotalBytes | quote }} - name: CALLBACK_ALLOWED_HOSTS value: {{ .Values.hadesLogManager.callbackAllowedHosts | quote }} + - name: CALLBACK_BLOCKED_CIDRS + value: {{ .Values.hadesLogManager.callbackBlockedCIDRs | quote }} - name: STATUS_WEBHOOK_ENABLED value: {{ .Values.hadesLogManager.statusWebhook.enabled | quote }} - name: STATUS_WEBHOOK_MAX_ATTEMPTS diff --git a/helm/hades/templates/networkpolicy.yaml b/helm/hades/templates/networkpolicy.yaml index 24295253..0461ba93 100644 --- a/helm/hades/templates/networkpolicy.yaml +++ b/helm/hades/templates/networkpolicy.yaml @@ -1,6 +1,7 @@ {{- if .Values.networkPolicy.enabled }} -# Job pods run untrusted step code: no connections into the release namespace. -# Other namespaces (DNS) and the outside stay reachable. +# Job pods run untrusted step code: inside the cluster they reach DNS only, so +# neither this release nor other namespaces (another Hades install, for one) +# are exposed to them. The outside stays reachable. apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -14,11 +15,12 @@ spec: - Egress egress: - to: - - namespaceSelector: - matchExpressions: - - key: kubernetes.io/metadata.name - operator: NotIn - values: [{{ .Release.Namespace | quote }}] + - namespaceSelector: {} + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 - to: - ipBlock: cidr: 0.0.0.0/0 diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index 97d3d888..2d24068c 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -215,11 +215,21 @@ hadesLogManager: requests: cpu: "200m" memory: "256Mi" + # -- Total bytes of job log messages the log manager keeps in memory across all + # jobs. When it is reached, the logs of the oldest completed jobs are evicted + # (their status stays). Heap use is a few times this value, so it must stay well + # below the memory limit. "0" disables the bound. + logMaxTotalBytes: "134217728" # -- Comma-separated host names, IPs or CIDRs that `callback_url` and # `status_callback_url` may target although they are loopback, private or # link-local, e.g. the in-cluster Artemis service # (`artemis.artemis.svc.cluster.local`). Empty = public addresses only. callbackAllowedHosts: "" + # -- Comma-separated CIDRs that `callback_url` and `status_callback_url` must + # not target, on top of the built-in list. Use it when the cluster's pod or + # service network uses public-looking addresses. An entry in + # `callbackAllowedHosts` still wins. Empty = built-in list only. + callbackBlockedCIDRs: "" # Where aggregated job logs are forwarded is configured per job via the # "callback_url" field on the build request, not globally on the log manager. statusWebhook: @@ -365,9 +375,10 @@ networkPolicy: # (`kubectl get pods -n --show-labels`), then enable. enabled: false jobPods: - # -- CIDRs job pods may not reach, on top of every pod in the release - # namespace. Set this to the cluster pod CIDR on CNIs that match pod IPs - # against ipBlock rules, otherwise the "outside" rule can still reach pods. + # -- CIDRs job pods may not reach. Inside the cluster job pods are allowed + # DNS only; set this to the cluster pod and service CIDRs on CNIs that match + # pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can + # still reach pods. blockedCIDRs: [] nats: # -- Labels of the NATS pods. Defaults match the bundled nats chart diff --git a/website/docs/reference/configuration.md b/website/docs/reference/configuration.md index 5f1ca467..c3e19792 100644 --- a/website/docs/reference/configuration.md +++ b/website/docs/reference/configuration.md @@ -131,6 +131,7 @@ Deployed by the Helm chart (`hades-log-manager`); also run locally via `make run | `METRICS_PORT` | `8082` | Port the Prometheus `/metrics` endpoint listens on. | | `LOG_BATCH_SIZE` | `100` | Log entries buffered before a flush. | | `LOG_RETENTION` | `1h` | How long completed-job logs are kept in memory (Go duration). | +| `LOG_MAX_TOTAL_BYTES` | `134217728` | Total log message bytes kept in memory across all jobs; oldest completed jobs are evicted first, `0` disables. | | `MAX_JOB_LOGS` | `1000` | Max log entries retained per job. | | `LOG_WATCHER_MAX_LIFETIME` | `6h` | Longest a job's log watcher lives without a terminal status, for jobs without `timeout_seconds`. | | `STATUS_WEBHOOK_ENABLED` | `true` | Deliver the [job-status webhook](../usage/submitting-jobs#job-status-webhook). Inert for jobs without a `status_callback_url`. | @@ -142,8 +143,9 @@ Deployed by the Helm chart (`hades-log-manager`); also run locally via `make run | `STATUS_WEBHOOK_MAX_PENDING` | `1000` | Status events awaiting acknowledgement (in flight or backing off). | | `CALLBACK_ALLOWED_HOSTS` | | Host names, IPs or CIDRs exempt from the callback address block (internal receivers such as Artemis in the same cluster or compose network). A host name entry must equal the host in the callback URL and trusts whatever it resolves to. | +| `CALLBACK_BLOCKED_CIDRS` | | CIDRs to refuse on top of the built-in list. Use it when the cluster's pod or service network uses public-looking addresses, which the built-in list cannot know. An entry in `CALLBACK_ALLOWED_HOSTS` still wins. | -Callbacks to loopback, private (RFC 1918, `fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), unspecified and multicast addresses are refused at connect time unless allowed by `CALLBACK_ALLOWED_HOSTS`. The log callback does not follow redirects, HTTP proxy environment variables are ignored, and a blocked status webhook is logged as an error and not retried. +Callbacks to loopback, private (RFC 1918, `fc00::/7`), CGNAT (`100.64.0.0/10`), link-local (including `169.254.169.254`), unspecified, multicast and other special-purpose addresses (such as `198.18.0.0/15`) are refused at connect time unless allowed by `CALLBACK_ALLOWED_HOSTS`. The log callback does not follow redirects, HTTP proxy environment variables are ignored, and a blocked status webhook is logged as an error and not retried. Both outbound pushes are configured per job, not globally: From 2960315f9d428177b8b1621327e7b872f7a08718 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Sat, 14 Nov 2026 02:37:53 +0100 Subject: [PATCH 6/9] Keep job pods and callbacks away from the test cluster nodes Job pods could still reach the node addresses (API server, kubelet, etcd, SSH and node-exporter ports), because those are outside the pod and service networks. The test values now list the node addresses for the job pod policy and for the callback guard; the chart comment says that node addresses belong into blockedCIDRs. Co-Authored-By: Claude Mythos 5.2 --- helm/hades/Readme.md | 2 +- helm/hades/values-test.yaml | 20 ++++++++++++++++++++ helm/hades/values.yaml | 3 ++- 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index 56681061..d0deb685 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -182,7 +182,7 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | nats.port | int | `4222` | NATS client port. | | networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | | networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | -| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. | +| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. Add the node addresses as well: the API server, kubelet and other node ports are "outside" for a NetworkPolicy. | | networkPolicy.nats.podLabels | object | `{}` | Labels of the NATS pods. Defaults match the bundled nats chart (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). | | rbac.hadesApi.serviceAccountName | string | `"hades-api"` | ServiceAccount name for the API (no Kubernetes API access, token not mounted). | | rbac.hadesLogManager.serviceAccountName | string | `"hades-log-manager"` | ServiceAccount name for the log manager (no Kubernetes API access, token not mounted). | diff --git a/helm/hades/values-test.yaml b/helm/hades/values-test.yaml index 9903a95d..ed5723d0 100644 --- a/helm/hades/values-test.yaml +++ b/helm/hades/values-test.yaml @@ -3,6 +3,12 @@ # Only non-secret configuration lives here; credentials are injected as # Kubernetes Secrets (hades-auth, hades-dashboard) created by the deploy workflow. +hadesLogManager: + # Callbacks must not reach the cluster nodes either (same list as + # networkPolicy.jobPods.blockedCIDRs below; pods and services are covered by + # the built-in list). + callbackBlockedCIDRs: "131.159.88.18/31,131.159.88.26/31,131.159.88.28/31,131.159.88.43/32,131.159.88.52/30,131.159.88.56/29,131.159.88.67/32,131.159.88.68/30,131.159.88.75/32,131.159.88.76/30" + hadesApi: # Protect the /build endpoint with the AUTH_KEY from the hades-auth Secret. authKeySecret: hades-auth @@ -42,3 +48,17 @@ networkPolicy: jobPods: blockedCIDRs: - 198.18.0.0/15 + # Node addresses of the test cluster (API server, kubelet, etcd, SSH and + # node-exporter ports are otherwise reachable from job pods). The ingress + # load balancers share the subnet and must stay reachable, hence no /24. + # Keep in sync when nodes are added. + - 131.159.88.18/31 + - 131.159.88.26/31 + - 131.159.88.28/31 + - 131.159.88.43/32 + - 131.159.88.52/30 + - 131.159.88.56/29 + - 131.159.88.67/32 + - 131.159.88.68/30 + - 131.159.88.75/32 + - 131.159.88.76/30 diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index 2d24068c..5f9c6e9d 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -378,7 +378,8 @@ networkPolicy: # -- CIDRs job pods may not reach. Inside the cluster job pods are allowed # DNS only; set this to the cluster pod and service CIDRs on CNIs that match # pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can - # still reach pods. + # still reach pods. Add the node addresses as well: the API + # server, kubelet and other node ports are "outside" for a NetworkPolicy. blockedCIDRs: [] nats: # -- Labels of the NATS pods. Defaults match the bundled nats chart From 8792e4df5ec2f7dc216b19cc235ce5e64c3997ac Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Sat, 14 Nov 2026 03:04:57 +0100 Subject: [PATCH 7/9] Let job pods reach hosts behind the cluster's ingress controller Blocking the pod network for job pods also cut off every host that the same cluster publishes through its ingress controller. The job pod policy gets an optional exception for the ingress controller namespaces on ports 80 and 443 (networkPolicy.jobPods.ingressNamespaces), enabled for ingress-nginx in the test values. Co-Authored-By: Claude Mythos 5.2 --- helm/hades/Readme.md | 3 ++- helm/hades/templates/networkpolicy.yaml | 15 +++++++++++++++ helm/hades/values-test.yaml | 2 ++ helm/hades/values.yaml | 5 +++++ 4 files changed, 24 insertions(+), 1 deletion(-) diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index d0deb685..efb9587a 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -180,9 +180,10 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | nats.config.jetstream.memoryStore.maxSize | string | `"1Gi"` | Maximum in-memory JetStream store size. | | nats.host | string | `"hades-nats.hades.svc.cluster.local"` | In-cluster NATS service hostname. | | nats.port | int | `4222` | NATS client port. | -| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | +| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[],"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | | networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | | networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. Add the node addresses as well: the API server, kubelet and other node ports are "outside" for a NetworkPolicy. | +| networkPolicy.jobPods.ingressNamespaces | list | `[]` | Namespaces of the cluster's ingress controller(s). Job pods may reach their pods on ports 80 and 443, so that hosts served by this cluster (a git server the job clones from, for example) stay reachable although the pod network is blocked. | | networkPolicy.nats.podLabels | object | `{}` | Labels of the NATS pods. Defaults match the bundled nats chart (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). | | rbac.hadesApi.serviceAccountName | string | `"hades-api"` | ServiceAccount name for the API (no Kubernetes API access, token not mounted). | | rbac.hadesLogManager.serviceAccountName | string | `"hades-log-manager"` | ServiceAccount name for the log manager (no Kubernetes API access, token not mounted). | diff --git a/helm/hades/templates/networkpolicy.yaml b/helm/hades/templates/networkpolicy.yaml index 0461ba93..a46644b4 100644 --- a/helm/hades/templates/networkpolicy.yaml +++ b/helm/hades/templates/networkpolicy.yaml @@ -21,6 +21,21 @@ spec: port: 53 - protocol: TCP port: 53 + {{- with .Values.networkPolicy.jobPods.ingressNamespaces }} + # Sites published through the cluster's own ingress controller resolve to + # controller pods, which the pod network block below would cut off. + - to: + - namespaceSelector: + matchExpressions: + - key: kubernetes.io/metadata.name + operator: In + values: {{ toJson . }} + ports: + - protocol: TCP + port: 80 + - protocol: TCP + port: 443 + {{- end }} - to: - ipBlock: cidr: 0.0.0.0/0 diff --git a/helm/hades/values-test.yaml b/helm/hades/values-test.yaml index ed5723d0..7b2fd0db 100644 --- a/helm/hades/values-test.yaml +++ b/helm/hades/values-test.yaml @@ -46,6 +46,8 @@ nats: networkPolicy: enabled: true jobPods: + ingressNamespaces: + - ingress-nginx blockedCIDRs: - 198.18.0.0/15 # Node addresses of the test cluster (API server, kubelet, etcd, SSH and diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index 5f9c6e9d..2a4b3b90 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -381,6 +381,11 @@ networkPolicy: # still reach pods. Add the node addresses as well: the API # server, kubelet and other node ports are "outside" for a NetworkPolicy. blockedCIDRs: [] + # -- Namespaces of the cluster's ingress controller(s). Job pods may reach + # their pods on ports 80 and 443, so that hosts served by this cluster (a + # git server the job clones from, for example) stay reachable although the + # pod network is blocked. + ingressNamespaces: [] nats: # -- Labels of the NATS pods. Defaults match the bundled nats chart # (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). From b04e2da806f1a81791e84d45680df078dc6e6281 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Sat, 14 Nov 2026 04:06:32 +0100 Subject: [PATCH 8/9] Address review feedback on job defaults, log budget and DNS rule - Operator: an empty JOB_DEFAULT_* / JOB_MAX_* value now really disables the setting (the env library replaced empty values with the default); a defaulted limit is raised to a step's explicit request, and the maximum also applies to steps without a limit. - Log manager: the byte budget reserves only the net growth of a capped container; FlushJob and ResetJobLogs release and delete under one lock. - NetworkPolicy: the DNS rule selects the cluster DNS pods instead of port 53 in every namespace. - Docs: install steps for the unauthenticated test setup and the Secret the upgrade needs; regenerated values tables. Co-Authored-By: Claude Mythos 5.2 --- HadesLogManager/processor.go | 35 ++++++++++++++-- HadesLogManager/processor_test.go | 38 +++++++++++++++++ HadesScheduler/HadesOperator/cmd/main.go | 42 +++++++++++++++---- HadesScheduler/HadesOperator/cmd/main_test.go | 33 +++++++++++++++ .../controller/buildjob_build_test.go | 31 ++++++++++++++ .../controller/buildjob_controller.go | 28 +++++++++---- helm/hades/Readme.md | 6 ++- helm/hades/Readme.md.gotmpl | 2 +- helm/hades/templates/networkpolicy.yaml | 10 ++++- helm/hades/values.yaml | 9 ++++ website/docs/deployment/helm-values.md | 26 ++++++++++-- website/docs/installation/helm-chart.md | 6 ++- 12 files changed, 239 insertions(+), 27 deletions(-) diff --git a/HadesLogManager/processor.go b/HadesLogManager/processor.go index cae1f71f..5ee6d1f3 100644 --- a/HadesLogManager/processor.go +++ b/HadesLogManager/processor.go @@ -172,7 +172,7 @@ func (la *NATSLogAggregator) AddLog(log buildlogs.Log) { la.mu.Lock() defer la.mu.Unlock() - if la.cfg.MaxTotalBytes > 0 && !la.makeRoom(log.JobID, entriesBytes(log.Logs)) { + if need := la.netNewBytes(log); la.cfg.MaxTotalBytes > 0 && need > 0 && !la.makeRoom(log.JobID, need) { // Keep the container slot (steps are positional) but drop the lines. if !la.dropWarned[log.JobID] { la.dropWarned[log.JobID] = true @@ -239,6 +239,33 @@ func (la *NATSLogAggregator) AddLog(log buildlogs.Log) { } } +// netNewBytes is how much AddLog grows the job's memory use: the incoming bytes +// minus what trimming the container to MaxJobLogs drops from its oldest entries. +// Without this, a container at its cap could never replace old lines once the +// budget is full. Callers hold la.mu. +func (la *NATSLogAggregator) netNewBytes(log buildlogs.Log) int { + need := entriesBytes(log.Logs) + if la.cfg.MaxJobLogs <= 0 { + return need + } + value, ok := la.logs.Load(log.JobID) + if !ok { + return need + } + var prev []buildlogs.LogEntry + for _, l := range *value.(logsVersion).ptr { + if l.ContainerID == log.ContainerID { + prev = l.Logs + break + } + } + combined := append(append([]buildlogs.LogEntry(nil), prev...), log.Logs...) + if drop := len(combined) - la.cfg.MaxJobLogs; drop > 0 { + need -= entriesBytes(combined[:drop]) + } + return need +} + func entriesBytes(entries []buildlogs.LogEntry) int { n := 0 for _, e := range entries { @@ -303,10 +330,12 @@ func (la *NATSLogAggregator) trimContainerEntries(jobID, containerID string, ent // Returns: // - error: Always nil in current implementation func (la *NATSLogAggregator) FlushJob(jobID string) error { + // Release and delete under one lock, or a concurrent AddLog could re-charge + // logs that are about to be deleted and leak that budget. la.mu.Lock() la.release(jobID) - la.mu.Unlock() value, logsExists := la.logs.LoadAndDelete(jobID) + la.mu.Unlock() if logsExists { v := value.(logsVersion) logs := *v.ptr @@ -332,8 +361,8 @@ func (la *NATSLogAggregator) FlushJob(jobID string) error { func (la *NATSLogAggregator) ResetJobLogs(jobID string) { la.mu.Lock() la.release(jobID) - la.mu.Unlock() la.logs.Delete(jobID) + la.mu.Unlock() la.completed.Delete(jobID) } diff --git a/HadesLogManager/processor_test.go b/HadesLogManager/processor_test.go index b70894dd..316392b3 100644 --- a/HadesLogManager/processor_test.go +++ b/HadesLogManager/processor_test.go @@ -446,3 +446,41 @@ func TestLogBudget_ConcurrentAddsNoRaceAndWithinBudget(t *testing.T) { defer agg.mu.Unlock() assert.LessOrEqual(t, agg.totalBytes, 500) } + +// A container at its MaxJobLogs cap replaces its oldest line, so the new line +// must not count as extra budget even when the budget is full. +func TestLogBudget_CappedContainerReplacesOldestLine(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + agg := NewLogAggregator(ctx, nil, nil, nil, AggregatorConfig{BatchSize: 100, Retention: time.Hour, MaxJobLogs: 1, MaxTotalBytes: 100}) + + agg.AddLog(buildlogs.Log{JobID: "j", ContainerID: "c", Logs: []buildlogs.LogEntry{logEntry(strings.Repeat("a", 60))}}) + agg.AddLog(buildlogs.Log{JobID: "j", ContainerID: "c", Logs: []buildlogs.LogEntry{logEntry(strings.Repeat("b", 60))}}) + + logs := agg.GetJobLogs("j") + require.Len(t, logs[0].Logs, 1) + assert.Equal(t, strings.Repeat("b", 60), logs[0].Logs[0].Message) + assert.Equal(t, 60, agg.totalBytes) +} + +// Flush and reset race with AddLog: the byte accounting must end at zero once +// the logs are gone. +func TestLogBudget_FlushRaceLeavesNoAccounting(t *testing.T) { + agg := newBudgetAggregator(t, 1<<20) + done := make(chan struct{}) + go func() { + defer close(done) + for i := 0; i < 200; i++ { + addBytes(agg, "j", 10) + } + }() + for i := 0; i < 200; i++ { + _ = agg.FlushJob("j") + agg.ResetJobLogs("j") + } + <-done + _ = agg.FlushJob("j") + + assert.Equal(t, 0, agg.totalBytes) + assert.Empty(t, agg.jobBytes) +} diff --git a/HadesScheduler/HadesOperator/cmd/main.go b/HadesScheduler/HadesOperator/cmd/main.go index 2594046b..4cbab25d 100644 --- a/HadesScheduler/HadesOperator/cmd/main.go +++ b/HadesScheduler/HadesOperator/cmd/main.go @@ -81,14 +81,41 @@ type OperatorConfig struct { // if-not-present or never. If empty, Kubernetes pulls :latest and untagged // images on every step. ImagePullPolicy string `env:"K8S_IMAGE_PULL_POLICY"` - // Defaults for what a job leaves unset. Empty or 0 disables each one. - JobDefaultTimeoutSeconds int64 `env:"JOB_DEFAULT_TIMEOUT_SECONDS" envDefault:"3600"` + // Defaults for what a job leaves unset. Empty or 0 disables each one. These + // carry no envDefault tag: the env library swaps an explicitly empty value for + // the default, which would make disabling impossible. applyUnsetJobDefaults + // sets them only when the variable is absent. + JobDefaultTimeoutSeconds int64 `env:"JOB_DEFAULT_TIMEOUT_SECONDS"` JobMaxTimeoutSeconds int64 `env:"JOB_MAX_TIMEOUT_SECONDS" envDefault:"0"` - JobDefaultCPULimit string `env:"JOB_DEFAULT_CPU_LIMIT" envDefault:"2"` - JobDefaultMemoryLimit string `env:"JOB_DEFAULT_MEMORY_LIMIT" envDefault:"4Gi"` - JobMaxCPULimit string `env:"JOB_MAX_CPU_LIMIT" envDefault:"16"` - JobMaxMemoryLimit string `env:"JOB_MAX_MEMORY_LIMIT" envDefault:"32Gi"` - JobSharedVolumeSizeLimit string `env:"JOB_SHARED_VOLUME_SIZE_LIMIT" envDefault:"10Gi"` + JobDefaultCPULimit string `env:"JOB_DEFAULT_CPU_LIMIT"` + JobDefaultMemoryLimit string `env:"JOB_DEFAULT_MEMORY_LIMIT"` + JobMaxCPULimit string `env:"JOB_MAX_CPU_LIMIT"` + JobMaxMemoryLimit string `env:"JOB_MAX_MEMORY_LIMIT"` + JobSharedVolumeSizeLimit string `env:"JOB_SHARED_VOLUME_SIZE_LIMIT"` +} + +// applyUnsetJobDefaults fills the job defaults of variables that are not set at +// all. A variable set to the empty string stays empty and disables its default. +func (c *OperatorConfig) applyUnsetJobDefaults() { + unset := func(k string) bool { _, ok := os.LookupEnv(k); return !ok } + if unset("JOB_DEFAULT_TIMEOUT_SECONDS") { + c.JobDefaultTimeoutSeconds = 3600 + } + if unset("JOB_DEFAULT_CPU_LIMIT") { + c.JobDefaultCPULimit = "2" + } + if unset("JOB_DEFAULT_MEMORY_LIMIT") { + c.JobDefaultMemoryLimit = "4Gi" + } + if unset("JOB_MAX_CPU_LIMIT") { + c.JobMaxCPULimit = "16" + } + if unset("JOB_MAX_MEMORY_LIMIT") { + c.JobMaxMemoryLimit = "32Gi" + } + if unset("JOB_SHARED_VOLUME_SIZE_LIMIT") { + c.JobSharedVolumeSizeLimit = "10Gi" + } } // parseQuantity parses an optional Kubernetes quantity; empty means unset. @@ -216,6 +243,7 @@ func main() { os.Exit(1) } + operatorConfig.applyUnsetJobDefaults() operatorConfig.normalize() pullPolicy, err := controller.ParseImagePullPolicy(operatorConfig.ImagePullPolicy) if err != nil { diff --git a/HadesScheduler/HadesOperator/cmd/main_test.go b/HadesScheduler/HadesOperator/cmd/main_test.go index 4990cb81..dbf8f149 100644 --- a/HadesScheduler/HadesOperator/cmd/main_test.go +++ b/HadesScheduler/HadesOperator/cmd/main_test.go @@ -36,6 +36,39 @@ func TestOperatorConfigDefaults(t *testing.T) { } } +// An absent job default variable gets its default, but one set to the empty +// string stays empty: that is how an operator disables the default. +func TestOperatorConfigJobDefaultsEmptyDisables(t *testing.T) { + jobVars := []string{"JOB_DEFAULT_TIMEOUT_SECONDS", "JOB_DEFAULT_CPU_LIMIT", "JOB_DEFAULT_MEMORY_LIMIT", "JOB_MAX_CPU_LIMIT", "JOB_MAX_MEMORY_LIMIT", "JOB_SHARED_VOLUME_SIZE_LIMIT"} + for _, k := range jobVars { + if old, ok := os.LookupEnv(k); ok { + t.Cleanup(func() { _ = os.Setenv(k, old) }) + } + _ = os.Unsetenv(k) + } + + cfg := loadOperatorConfig(t) + cfg.applyUnsetJobDefaults() + if cfg.JobDefaultTimeoutSeconds != 3600 || cfg.JobDefaultCPULimit != "2" || cfg.JobDefaultMemoryLimit != "4Gi" || + cfg.JobMaxCPULimit != "16" || cfg.JobMaxMemoryLimit != "32Gi" || cfg.JobSharedVolumeSizeLimit != "10Gi" { + t.Errorf("unset variables did not get their defaults: %+v", cfg) + } + + for _, k := range jobVars { + t.Setenv(k, "") + } + cfg = loadOperatorConfig(t) + cfg.applyUnsetJobDefaults() + if cfg.JobDefaultTimeoutSeconds != 0 || cfg.JobDefaultCPULimit != "" || cfg.JobDefaultMemoryLimit != "" || + cfg.JobMaxCPULimit != "" || cfg.JobMaxMemoryLimit != "" || cfg.JobSharedVolumeSizeLimit != "" { + t.Errorf("empty variables must stay empty: %+v", cfg) + } + d, err := cfg.jobDefaults() + if err != nil || d.CPULimit != nil || d.MemoryLimit != nil || d.MaxCPULimit != nil || d.MaxMemoryLimit != nil || d.SharedVolumeSizeLimit != nil { + t.Errorf("jobDefaults() = %+v, %v, want everything disabled", d, err) + } +} + func TestOperatorConfigParsesDurations(t *testing.T) { clearOperatorEnv(t) t.Setenv("REQUEUE_DELAY", "250ms") diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go index 7561b1ab..1abe76fd 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_build_test.go @@ -439,6 +439,37 @@ func TestBuildK8sJob_ResourceDefaults(t *testing.T) { } } +// A request-only step must not get a default limit below its request, and the +// maximum must bound a step even when the defaults are disabled. +func TestBuildK8sJob_RequestAndMaxLimits(t *testing.T) { + q := resource.MustParse + cpu, mem := q("2"), q("4Gi") + maxCPU, maxMem := q("16"), q("32Gi") + initC := func(step buildv1.BuildStep, d JobDefaults) corev1.Container { + return buildK8sJob(defaultsTestJob(nil, step), "job-1", true, false, DefaultFinalizerImage, "", d).Spec.Template.Spec.InitContainers[0] + } + + reqCPU, reqMem := q("8"), q("8Gi") + c := initC(buildv1.BuildStep{CPURequest: &reqCPU, MemoryRequest: &reqMem}, JobDefaults{CPULimit: &cpu, MemoryLimit: &mem, MaxCPULimit: &maxCPU, MaxMemoryLimit: &maxMem}) + if !c.Resources.Limits.Cpu().Equal(reqCPU) || !c.Resources.Limits.Memory().Equal(reqMem) { + t.Errorf("limits = %v, want them raised to the requests", c.Resources.Limits) + } + + // A request above the maximum is clamped, and so is the raised limit. + bigMem := q("64Gi") + c = initC(buildv1.BuildStep{MemoryRequest: &bigMem}, JobDefaults{MemoryLimit: &mem, MaxMemoryLimit: &maxMem}) + if !c.Resources.Limits.Memory().Equal(maxMem) || !c.Resources.Requests.Memory().Equal(maxMem) { + t.Errorf("resources = %v, want limit and request at 32Gi", c.Resources) + } + + // Only maxima configured: a step without limits gets the maximum as limit, + // but its (unset) request does not become the maximum. + c = initC(buildv1.BuildStep{}, JobDefaults{MaxCPULimit: &maxCPU, MaxMemoryLimit: &maxMem}) + if !c.Resources.Limits.Cpu().Equal(maxCPU) || !c.Resources.Limits.Memory().Equal(maxMem) { + t.Errorf("limits = %v, want the maxima", c.Resources.Limits) + } +} + func TestBuildK8sJob_SharedVolumeSizeLimit(t *testing.T) { size := resource.MustParse("10Gi") vol := func(d JobDefaults) *corev1.EmptyDirVolumeSource { diff --git a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go index 7a7c30ba..e56c3291 100644 --- a/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go +++ b/HadesScheduler/HadesOperator/internal/controller/buildjob_controller.go @@ -158,6 +158,22 @@ func clampQuantity(q, max *resource.Quantity, what, buildJob string, stepID int3 return max } +// effectiveLimit picks the step's own limit, else the operator default raised to +// the step's request (Kubernetes rejects a request above its limit), else the +// operator maximum. The caller clamps the result to the maximum. +func effectiveLimit(limit, request, def, max *resource.Quantity) *resource.Quantity { + if limit != nil { + return limit + } + if def != nil && request != nil && request.Cmp(*def) > 0 { + return request + } + if def != nil { + return def + } + return max +} + // maxCreateMessage bounds the API server message kept as the failure reason, // which for a huge Job can list thousands of field errors. const maxCreateMessage = 300 @@ -957,14 +973,10 @@ func buildK8sJob(bj *buildv1.BuildJob, jobName string, deleteOnComplete bool, su } } - // Set resource limits: the step's own, else the operator default - cpuLimit, memLimit := s.CPULimit, s.MemoryLimit - if cpuLimit == nil { - cpuLimit = defaults.CPULimit - } - if memLimit == nil { - memLimit = defaults.MemoryLimit - } + // Set resource limits: the step's own, else the operator default, else + // the operator maximum, so the maximum holds even with defaults disabled. + cpuLimit := effectiveLimit(s.CPULimit, s.CPURequest, defaults.CPULimit, defaults.MaxCPULimit) + memLimit := effectiveLimit(s.MemoryLimit, s.MemoryRequest, defaults.MemoryLimit, defaults.MaxMemoryLimit) cpuLimit = clampQuantity(cpuLimit, defaults.MaxCPULimit, "cpu limit", bj.Name, s.ID) memLimit = clampQuantity(memLimit, defaults.MaxMemoryLimit, "memory limit", bj.Name, s.ID) if cpuLimit != nil || memLimit != nil { diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index efb9587a..df081ca9 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -57,7 +57,7 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, cat ./helm/hades/values.yaml ``` -3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. +3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, skip the Secret and pass `--set hadesApi.allowUnauthenticated=true` instead of `--set hadesApi.authKeySecret=hades-auth` (a set `authKeySecret` always wins). ```bash kubectl create namespace hades kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= @@ -180,9 +180,11 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | nats.config.jetstream.memoryStore.maxSize | string | `"1Gi"` | Maximum in-memory JetStream store size. | | nats.host | string | `"hades-nats.hades.svc.cluster.local"` | In-cluster NATS service hostname. | | nats.port | int | `4222` | NATS client port. | -| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[],"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | +| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[],"dns":{"namespaceLabels":{"kubernetes.io/metadata.name":"kube-system"},"podLabels":{"k8s-app":"kube-dns"}},"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | | networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | | networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. Add the node addresses as well: the API server, kubelet and other node ports are "outside" for a NetworkPolicy. | +| networkPolicy.jobPods.dns.namespaceLabels | object | `{"kubernetes.io/metadata.name":"kube-system"}` | Labels of the namespace running the cluster DNS. Job pods may reach port 53 only on pods in it that match `podLabels`. The defaults fit CoreDNS and kube-dns; adjust them for other DNS deployments. | +| networkPolicy.jobPods.dns.podLabels | object | `{"k8s-app":"kube-dns"}` | Labels of the cluster DNS pods. | | networkPolicy.jobPods.ingressNamespaces | list | `[]` | Namespaces of the cluster's ingress controller(s). Job pods may reach their pods on ports 80 and 443, so that hosts served by this cluster (a git server the job clones from, for example) stay reachable although the pod network is blocked. | | networkPolicy.nats.podLabels | object | `{}` | Labels of the NATS pods. Defaults match the bundled nats chart (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). | | rbac.hadesApi.serviceAccountName | string | `"hades-api"` | ServiceAccount name for the API (no Kubernetes API access, token not mounted). | diff --git a/helm/hades/Readme.md.gotmpl b/helm/hades/Readme.md.gotmpl index c91b5f04..23a71dad 100644 --- a/helm/hades/Readme.md.gotmpl +++ b/helm/hades/Readme.md.gotmpl @@ -57,7 +57,7 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, cat ./helm/hades/values.yaml ``` -3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. +3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, skip the Secret and pass `--set hadesApi.allowUnauthenticated=true` instead of `--set hadesApi.authKeySecret=hades-auth` (a set `authKeySecret` always wins). ```bash kubectl create namespace hades kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= diff --git a/helm/hades/templates/networkpolicy.yaml b/helm/hades/templates/networkpolicy.yaml index a46644b4..fb49037d 100644 --- a/helm/hades/templates/networkpolicy.yaml +++ b/helm/hades/templates/networkpolicy.yaml @@ -1,7 +1,8 @@ {{- if .Values.networkPolicy.enabled }} # Job pods run untrusted step code: inside the cluster they reach DNS only, so # neither this release nor other namespaces (another Hades install, for one) -# are exposed to them. The outside stays reachable. +# are exposed to them. The outside stays reachable. DNS goes to the cluster DNS +# pods only, so a stray listener on port 53 elsewhere is not exempted. apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -15,7 +16,12 @@ spec: - Egress egress: - to: - - namespaceSelector: {} + - namespaceSelector: + matchLabels: + {{- toYaml .Values.networkPolicy.jobPods.dns.namespaceLabels | nindent 14 }} + podSelector: + matchLabels: + {{- toYaml .Values.networkPolicy.jobPods.dns.podLabels | nindent 14 }} ports: - protocol: UDP port: 53 diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index 2a4b3b90..22fcff63 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -381,6 +381,15 @@ networkPolicy: # still reach pods. Add the node addresses as well: the API # server, kubelet and other node ports are "outside" for a NetworkPolicy. blockedCIDRs: [] + dns: + # -- Labels of the namespace running the cluster DNS. Job pods may reach + # port 53 only on pods in it that match `podLabels`. The defaults fit + # CoreDNS and kube-dns; adjust them for other DNS deployments. + namespaceLabels: + kubernetes.io/metadata.name: kube-system + # -- Labels of the cluster DNS pods. + podLabels: + k8s-app: kube-dns # -- Namespaces of the cluster's ingress controller(s). Job pods may reach # their pods on ports 80 and 443, so that hosts served by this cluster (a # git server the job clones from, for example) stay reachable although the diff --git a/website/docs/deployment/helm-values.md b/website/docs/deployment/helm-values.md index 33f27711..3668ef1b 100644 --- a/website/docs/deployment/helm-values.md +++ b/website/docs/deployment/helm-values.md @@ -63,7 +63,7 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, cat ./helm/hades/values.yaml ``` -3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. +3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, skip the Secret and pass `--set hadesApi.allowUnauthenticated=true` instead of `--set hadesApi.authKeySecret=hades-auth` (a set `authKeySecret` always wins). ```bash kubectl create namespace hades kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= @@ -115,7 +115,8 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | Key | Type | Default | Description | |-----|------|---------|-------------| | debug | bool | `false` | Global debug switch. When true, every Hades container gets `DEBUG=true` (and the operator additionally gets `DEV_MODE=true`) for verbose logging. | -| hadesApi.authKeySecret | string | `""` | Name of an existing Secret holding key `AUTH_KEY` to protect the `/build` endpoint. Empty = no auth (endpoint is open). | +| hadesApi.allowUnauthenticated | bool | `false` | Serve `/build` without authentication when `authKeySecret` is empty. Only for throwaway test installs; the API refuses to start otherwise. | +| hadesApi.authKeySecret | string | `""` | Name of an existing Secret holding key `AUTH_KEY` to protect the `/build` endpoint. Required unless `allowUnauthenticated` is true. | | hadesApi.dashboard | object | `{"cookieInsecure":false,"jobRetention":"1h","logManagerURL":"http://hades-log-manager-service:8081","redactMode":"","secretKeyPatterns":"","secretName":"","sessionTTL":"12h","trustedProxies":[]}` | Operator dashboard (web UI) served by the API. | | hadesApi.dashboard.cookieInsecure | bool | `false` | Drop the Secure flag on the session cookie. Only for local HTTP dev; never enable in a real deployment (requires TLS otherwise login breaks). | | hadesApi.dashboard.jobRetention | string | `"1h"` | How long finished jobs stay in the dashboard's in-memory list (Go duration). | @@ -130,7 +131,10 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | hadesApi.resources | object | `{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"200m","memory":"256Mi"}}` | Resource requests and limits for the API pod. | | hadesApi.service.port | int | `8080` | Service port for the API. | | hadesApi.service.targetPort | int | `8080` | Container port the API listens on. | +| hadesLogManager.callbackAllowedHosts | string | `""` | Comma-separated host names, IPs or CIDRs that `callback_url` and `status_callback_url` may target although they are loopback, private or link-local, e.g. the in-cluster Artemis service (`artemis.artemis.svc.cluster.local`). Empty = public addresses only. | +| hadesLogManager.callbackBlockedCIDRs | string | `""` | Comma-separated CIDRs that `callback_url` and `status_callback_url` must not target, on top of the built-in list. Use it when the cluster's pod or service network uses public-looking addresses. An entry in `callbackAllowedHosts` still wins. Empty = built-in list only. | | hadesLogManager.image.pullPolicy | string | `"Always"` | Image pull policy for the log manager. | +| hadesLogManager.logMaxTotalBytes | string | `"134217728"` | Total bytes of job log messages the log manager keeps in memory across all jobs. When it is reached, the logs of the oldest completed jobs are evicted (their status stays). Heap use is a few times this value, so it must stay well below the memory limit. "0" disables the bound. | | hadesLogManager.replicaCount | int | `1` | Keep at 1: the log manager aggregates per-job logs in memory and uses plain (non-queue, non-durable) NATS subscriptions, so multiple replicas would independently forward duplicate/split logs to the Artemis adapter. | | hadesLogManager.resources | object | `{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"200m","memory":"256Mi"}}` | Resource requests and limits for the log manager pod. | | hadesLogManager.service.port | int | `8081` | Service port for the log manager. | @@ -141,11 +145,18 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | hadesLogManager.statusWebhook.maxBackoff | string | `"5m"` | Ceiling for the retry delay. | | hadesLogManager.statusWebhook.maxPending | int | `1000` | Status events awaiting acknowledgement (in flight or waiting out a retry backoff). | | hadesLogManager.statusWebhook.timeout | string | `"10s"` | Bound on a single delivery, including the callback-URL lookup. | -| hadesOperator.DeleteOnComplete | bool | `false` | Delete BuildJob CRs (and their Jobs) once they finish. Defaults to false so completed BuildJobs/pods linger (they self-clean via the Job TTL, ~1h) and the operator can capture every step's logs before the pod is removed; Kubernetes only serves container logs while the pod exists, so immediate deletion can lose logs for fast jobs. Set to true only once the operator captures all logs before deletion (the companion barrier fix). | +| hadesOperator.DeleteOnComplete | bool | `true` | Delete BuildJob CRs (and their Jobs) once they finish. Defaults to true so job scripts and metadata (which may hold secrets) do not stay in the cluster. The operator waits until every step's logs are published (bounded by `logDrainTimeout`) before it deletes, because Kubernetes only serves container logs while the pod exists. Set to false to keep finished BuildJobs and pods for debugging; they then self-clean via the Job TTL (~1h) but the BuildJob CR stays. | | hadesOperator.clusterWide | bool | `false` | Grant cluster-wide RBAC instead of namespace-scoped. | | hadesOperator.finalizerImage | string | `"busybox:1.37.0"` | Image for the main container of every BuildJob pod. Kubernetes requires at least one non-init container and the build steps run as init containers, so this one exists only to exit once they are done: it runs `sh -c "echo build finished"` and needs nothing but a shell. Override it for clusters that cannot reach Docker Hub, or to point at a mirror. Keep it pinned to an explicit tag - an untagged image resolves to ":latest", which Kubernetes special-cases to imagePullPolicy: Always, adding a registry round trip to every job for an image the node already has. Empty falls back to the operator's built-in default. | | hadesOperator.image.pullPolicy | string | `"Always"` | Image pull policy for the operator. | | hadesOperator.imagePullPolicy | string | `""` | imagePullPolicy for build step containers: `always`, `if-not-present` or `never` (case-insensitive). If empty, Kubernetes pulls `:latest` and untagged images on every step. `if-not-present` skips the pull when the node already has the image (so a moved tag isn't picked up), `never` fails steps whose image is missing. The finalizer container is not affected. Invalid values fail operator startup. | +| hadesOperator.jobDefaultCPULimit | string | `"2"` | CPU limit (Kubernetes quantity) for each step that sets no `cpu_limit`. Empty disables the default. An invalid value fails operator startup. | +| hadesOperator.jobDefaultMemoryLimit | string | `"4Gi"` | Memory limit (Kubernetes quantity) for each step that sets no `memory_limit`. Empty disables the default. An invalid value fails operator startup. | +| hadesOperator.jobDefaultTimeoutSeconds | string | `"3600"` | Timeout (seconds) applied to jobs that set no `timeout_seconds`, so a hung or unschedulable job cannot hold a parallelism slot forever. Empty or 0 disables the default. | +| hadesOperator.jobMaxCPULimit | string | `"16"` | Largest CPU limit and request (Kubernetes quantity) a step may have. A larger value is clamped to it, the defaults included. Empty disables the cap. An invalid value fails operator startup. | +| hadesOperator.jobMaxMemoryLimit | string | `"32Gi"` | Largest memory limit and request (Kubernetes quantity) a step may have. A larger value is clamped to it, the defaults included. Empty disables the cap. An invalid value fails operator startup. | +| hadesOperator.jobMaxTimeoutSeconds | string | `"0"` | Upper bound (seconds) for the job timeout. A larger or missing job timeout is clamped to it. Empty or 0 means no cap. | +| hadesOperator.jobSharedVolumeSizeLimit | string | `"10Gi"` | Size limit of the `emptyDir` shared between the steps of a job (mounted at `/shared`). Without it a job can fill the node's disk. Empty disables the limit. An invalid value fails operator startup. | | hadesOperator.leaderElection.enabled | bool | `true` | Enable leader election for the operator (recommended with >1 replica). | | hadesOperator.logDrainTimeout | string | `"45s"` | How long a completed BuildJob is kept (undeleted) while its container logs are still being published, before it is deleted anyway (Go duration). Kubernetes only serves container logs while the pod exists, so this is the safety valve on the log-drain gate. Only relevant when DeleteOnComplete is true. A zero or negative duration falls back to the default; a value that is not a valid Go duration fails operator startup. | | hadesOperator.maxConcurrentReconciles | string | `"8"` | Number of BuildJobs the operator reconciles in parallel. A zero or negative value falls back to the default. | @@ -175,6 +186,15 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | nats.config.jetstream.memoryStore.maxSize | string | `"1Gi"` | Maximum in-memory JetStream store size. | | nats.host | string | `"hades-nats.hades.svc.cluster.local"` | In-cluster NATS service hostname. | | nats.port | int | `4222` | NATS client port. | +| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[],"dns":{"namespaceLabels":{"kubernetes.io/metadata.name":"kube-system"},"podLabels":{"k8s-app":"kube-dns"}},"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | +| networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | +| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. Add the node addresses as well: the API server, kubelet and other node ports are "outside" for a NetworkPolicy. | +| networkPolicy.jobPods.dns.namespaceLabels | object | `{"kubernetes.io/metadata.name":"kube-system"}` | Labels of the namespace running the cluster DNS. Job pods may reach port 53 only on pods in it that match `podLabels`. The defaults fit CoreDNS and kube-dns; adjust them for other DNS deployments. | +| networkPolicy.jobPods.dns.podLabels | object | `{"k8s-app":"kube-dns"}` | Labels of the cluster DNS pods. | +| networkPolicy.jobPods.ingressNamespaces | list | `[]` | Namespaces of the cluster's ingress controller(s). Job pods may reach their pods on ports 80 and 443, so that hosts served by this cluster (a git server the job clones from, for example) stay reachable although the pod network is blocked. | +| networkPolicy.nats.podLabels | object | `{}` | Labels of the NATS pods. Defaults match the bundled nats chart (`app.kubernetes.io/name: nats`, `app.kubernetes.io/instance: `). | +| rbac.hadesApi.serviceAccountName | string | `"hades-api"` | ServiceAccount name for the API (no Kubernetes API access, token not mounted). | +| rbac.hadesLogManager.serviceAccountName | string | `"hades-log-manager"` | ServiceAccount name for the log manager (no Kubernetes API access, token not mounted). | | rbac.hadesOperator.serviceAccountName | string | `"hades-operator"` | ServiceAccount name for the operator. | | rbac.hadesScheduler.serviceAccountName | string | `"hades-scheduler"` | ServiceAccount name for the scheduler. | | tracing | object | `{"deployJaeger":false,"enabled":false,"endpoint":"","jaeger":{"image":"jaegertracing/all-in-one:1.62.0","ui":{"auth":{"existingSecret":"","password":"","username":"admin"},"ingress":{"annotations":{},"className":"nginx","enabled":false,"host":"","tls":{"enabled":false,"secretName":""}}}}}` | OpenTelemetry tracing. When enabled, every Hades service exports spans to an OTLP endpoint so a job's overhead/runtime breakdown is visible as a per-job trace waterfall (API -> scheduler -> operator). Disabled by default: with no endpoint the services run a noop tracer, so the chart installs cleanly without a tracing backend. | diff --git a/website/docs/installation/helm-chart.md b/website/docs/installation/helm-chart.md index 061c124b..4cc3d447 100644 --- a/website/docs/installation/helm-chart.md +++ b/website/docs/installation/helm-chart.md @@ -20,7 +20,7 @@ The chart is published to GHCR as an OCI artifact. In every command below, repla `` with the chart version you want to install (for example `1.0.0`; see the [available versions](https://github.com/Hades-Scheduler/hades/pkgs/container/charts%2Fhades)): -Create the API auth Secret first. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, `--set hadesApi.allowUnauthenticated=true` skips the Secret. +Create the API auth Secret first. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, skip the Secret and pass `--set hadesApi.allowUnauthenticated=true` instead of `--set hadesApi.authKeySecret=hades-auth` (a set `authKeySecret` always wins). ```bash kubectl create namespace hades @@ -35,6 +35,8 @@ helm upgrade --install hades oci://ghcr.io/hades-scheduler/charts/hades \ ### Option B - From a local checkout +Create the `hades-auth` Secret first, as shown in Option A, or use the unauthenticated test setting described there. + ```bash # 1. Add the NATS sub-chart dependency helm repo add nats https://nats-io.github.io/k8s/helm/charts @@ -84,6 +86,8 @@ helm show crds oci://ghcr.io/hades-scheduler/charts/hades --version | ## Upgrade & Rollback +The command below needs the `hades-auth` Secret to exist. If your release already uses a different Secret name, pass that name instead. + ```bash helm upgrade hades oci://ghcr.io/hades-scheduler/charts/hades --version -n hades \ --set hadesApi.authKeySecret=hades-auth From 938123e73aca48ea57e07aba136465cc5b6ecee0 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Mon, 16 Nov 2026 10:59:19 +0100 Subject: [PATCH 9/9] Require NATS credentials and sane secrets, enable network policies by default - Helm: the bundled NATS server requires the credentials from the hades-nats-auth Secret and all components connect with them; rendering fails when the server side is removed or the names differ - Helm: NetworkPolicies are on by default and block the private, CGNAT, link-local and benchmarking ranges for job pods - values-prod.yaml gets the same job network and callback block lists as values-test.yaml (same cluster) - API refuses an AUTH_KEY under 16 characters; NATS_USERNAME and NATS_PASSWORD must be set together - Deploy workflow stops on missing or weak secrets before it touches the cluster and generates the NATS credentials once per namespace Co-Authored-By: Claude Mythos 5.2 --- .github/workflows/deploy-k8s.yml | 64 ++++++++++++++----- AGENTS.md | 4 +- HadesAPI/main.go | 6 ++ HadesAPI/main_test.go | 4 +- HadesScheduler/HadesOperator/Readme.md | 4 +- Readme.md | 12 +++- docs/configuration.md | 6 +- helm/hades/Readme.md | 11 ++-- helm/hades/Readme.md.gotmpl | 4 +- helm/hades/templates/_nats.tpl | 33 ++++++++++ .../hades/templates/hades-api-deployment.yaml | 3 +- .../hades-log-manager-deployment.yaml | 3 +- .../templates/hades-operator-deployment.yaml | 3 +- .../templates/hades-scheduler-deployment.yaml | 3 +- helm/hades/values-prod.yaml | 41 +++++++++++- helm/hades/values-test.yaml | 12 +++- helm/hades/values.yaml | 54 ++++++++++++---- shared/nats/connection.go | 8 ++- shared/nats/connection_test.go | 14 ++++ website/docs/deployment/helm-values.md | 11 ++-- .../deployment/kubernetes-github-actions.md | 6 +- website/docs/installation/helm-chart.md | 4 +- website/docs/operation-modes/k8s-operator.md | 2 +- website/docs/reference/configuration.md | 6 +- 24 files changed, 253 insertions(+), 65 deletions(-) create mode 100644 helm/hades/templates/_nats.tpl create mode 100644 shared/nats/connection_test.go diff --git a/.github/workflows/deploy-k8s.yml b/.github/workflows/deploy-k8s.yml index 94201aa6..2ba37dba 100644 --- a/.github/workflows/deploy-k8s.yml +++ b/.github/workflows/deploy-k8s.yml @@ -11,9 +11,13 @@ name: Deploy to Kubernetes # DASHBOARD_USERNAME - dashboard login user # DASHBOARD_PASSWORD_HASH - bcrypt hash of the dashboard password # DASHBOARD_SESSION_SECRET - dashboard session signing secret (>=32 chars) +# The deploy stops before it touches the cluster when one of them is missing or +# too weak (AUTH_KEY under 16 characters, session secret under 32, password hash +# that is not bcrypt). # The AUTH_KEY and dashboard values are materialised as in-cluster Secrets # (hades-auth, hades-dashboard) on every run; the first run creates them and later runs -# keep them in sync. When the dashboard credentials are present, a matching +# keep them in sync. The NATS credentials (hades-nats-auth) are internal to the +# release: the first run generates them in the cluster and later runs keep them. When the dashboard credentials are present, a matching # hades-jaeger-ui-auth Secret is also created so the Jaeger tracing UI (if enabled by # the values file) is protected by the same username/password as the dashboard. @@ -61,6 +65,36 @@ jobs: persist-credentials: false ref: ${{ inputs.chart-ref }} + - name: Check deployment secrets + env: + KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }} + AUTH_KEY: ${{ secrets.AUTH_KEY }} + DASHBOARD_USERNAME: ${{ secrets.DASHBOARD_USERNAME }} + DASHBOARD_PASSWORD_HASH: ${{ secrets.DASHBOARD_PASSWORD_HASH }} + DASHBOARD_SESSION_SECRET: ${{ secrets.DASHBOARD_SESSION_SECRET }} + # Inline (not a script file) so it also guards deploys of a chart ref + # that predates it. The values are never printed. + run: | + set -uo pipefail + + failed=0 + fail() { + echo "::error title=Deployment secret::$1" + failed=1 + } + + [ -n "${KUBE_CONFIG:-}" ] || fail "KUBE_CONFIG is not set" + [ "${#AUTH_KEY}" -ge 16 ] || fail "AUTH_KEY is missing or shorter than 16 characters" + [ -n "${DASHBOARD_USERNAME:-}" ] || fail "DASHBOARD_USERNAME is not set" + [[ "${DASHBOARD_PASSWORD_HASH:-}" =~ ^\$2[aby]\$[0-9]{2}\$.{53}$ ]] || fail "DASHBOARD_PASSWORD_HASH is not a bcrypt hash" + [ "${#DASHBOARD_SESSION_SECRET}" -ge 32 ] || fail "DASHBOARD_SESSION_SECRET is missing or shorter than 32 characters" + + case "${AUTH_KEY:-}" in + changeme | password | "" | unused) fail "AUTH_KEY is a placeholder" ;; + esac + + exit "$failed" + - name: Install Helm uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4 @@ -102,24 +136,24 @@ jobs: --from-literal=DASHBOARD_PASSWORD_HASH="$DASHBOARD_PASSWORD_HASH" \ --from-literal=DASHBOARD_SESSION_SECRET="$DASHBOARD_SESSION_SECRET" \ --dry-run=client -o yaml | kubectl apply -f - + # NATS credentials: nobody outside the release needs them, so they are + # generated once and kept. Letters and digits only, starting with a + # letter, because the NATS server parses the value from its environment. + if ! kubectl get secret hades-nats-auth --namespace "$NAMESPACE" >/dev/null 2>&1; then + kubectl create secret generic hades-nats-auth \ + --namespace "$NAMESPACE" \ + --from-literal=NATS_USERNAME=hades \ + --from-literal=NATS_PASSWORD="hades$(openssl rand -hex 32)" + fi # Basic-auth Secret for the Jaeger tracing UI, reusing the dashboard # credentials so the UI shares the dashboard's username/password. The # value is an htpasswd line ":"; the dashboard hash is - # bcrypt, which the ingress-nginx basic-auth handler accepts. Only created - # when the dashboard credentials exist; the chart consumes it via + # bcrypt, which the ingress-nginx basic-auth handler accepts. The chart consumes it via # tracing.jaeger.ui.auth.existingSecret (test values enable this). - if [ -n "$DASHBOARD_USERNAME" ] && [ -n "$DASHBOARD_PASSWORD_HASH" ]; then - kubectl create secret generic hades-jaeger-ui-auth \ - --namespace "$NAMESPACE" \ - --from-literal=auth="${DASHBOARD_USERNAME}:${DASHBOARD_PASSWORD_HASH}" \ - --dry-run=client -o yaml | kubectl apply -f - - else - # Fail closed: if dashboard credentials are removed, drop the stale - # basic-auth Secret so the Jaeger UI cannot keep accepting the old - # password. - kubectl delete secret hades-jaeger-ui-auth \ - --namespace "$NAMESPACE" --ignore-not-found - fi + kubectl create secret generic hades-jaeger-ui-auth \ + --namespace "$NAMESPACE" \ + --from-literal=auth="${DASHBOARD_USERNAME}:${DASHBOARD_PASSWORD_HASH}" \ + --dry-run=client -o yaml | kubectl apply -f - - name: Apply CRDs # Helm does not upgrade CRDs after the first install, so apply them explicitly. diff --git a/AGENTS.md b/AGENTS.md index 505e6a01..dd01d6e3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -67,7 +67,7 @@ The top-level `Makefile` wraps the common workflows (`make help` lists every tar - **CLI mode:** `make run` runs `HadesAPI`, `HadesScheduler`, and `HadesLogManager` via `go run` and auto-starts NATS in Docker. - **Docker mode:** `make docker-run` brings up `hadesAPI` (8081→8080), `hadesScheduler` (docker executor), and `nats` via `compose.yml`. API requests: see `bruno/api/*.bru`. -- **K8s mode:** `helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth` (create that Secret with key `AUTH_KEY` first; `--set hadesApi.allowUnauthenticated=true` is for throwaway installs only). The chart deploys API, scheduler, operator, log manager, and embedded NATS JetStream. +- **K8s mode:** `helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth` (create that Secret with key `AUTH_KEY`, at least 16 characters, and the `hades-nats-auth` Secret with keys `NATS_USERNAME`/`NATS_PASSWORD` first; `--set hadesApi.allowUnauthenticated=true` is for throwaway installs only). The chart deploys API, scheduler, operator, log manager, and embedded NATS JetStream. ## Conventions @@ -82,7 +82,7 @@ The top-level `Makefile` wraps the common workflows (`make help` lists every tar - Don't add fields to `BuildJobSpec` without regenerating the CRD (CI will block the PR). - Don't introduce package-level mutable globals for dependencies - the codebase has been moving away from them (see `setupRouter` taking `JobPublisher` as a parameter). - `HadesLogManager` is deployed (Helm + image) but ClusterIP-internal and single-replica (in-memory aggregation, bounded by `LOG_MAX_TOTAL_BYTES`: oldest completed jobs lose their logs first); don't put it on the ingress or scale it past one replica. The status-webhook dispatcher living there keeps no durable state in memory (its retry schedule is in JetStream), but it still assumes one replica. The same single-replica constraint applies to the API's in-memory dashboard read-model. (It just isn't in the `compose.yml` stack.) -- NATS itself still has no authentication; it must not be published beyond loopback (compose binds it to `127.0.0.1`, the deploy stack does not publish it). Never list the NATS network in `DOCKER_ALLOWED_NETWORKS`. +- NATS requires credentials in the Helm chart (Secret `nats.auth.secretName`, default `hades-nats-auth`, read by the NATS server and all components; rendering fails when the server side is removed) and in the Ansible role. The compose stacks still run it without authentication; it must not be published beyond loopback (compose binds it to `127.0.0.1`, the deploy stack does not publish it). Never list the NATS network in `DOCKER_ALLOWED_NETWORKS`. - Don't re-open callbacks to internal addresses or add a redirect-following/proxy-aware callback client; use `CALLBACK_ALLOWED_HOSTS`. - Job pods must keep no service account token, `RuntimeDefault` seccomp, `allowPrivilegeEscalation: false`, `enableServiceLinks: false` and dropped `NET_RAW`/`MKNOD` capabilities (not `ALL`: grading images install packages as root); the operator role gets no secrets/configmaps access. - The dashboard treats all job/step `Metadata` and step `script` bodies as potentially secret: redact via `shared/redact` before returning any payload; never add an endpoint that returns raw metadata or an unscrubbed script. Job logs are proxied verbatim (a known, documented gap). diff --git a/HadesAPI/main.go b/HadesAPI/main.go index 9b24e084..e7d0a24b 100644 --- a/HadesAPI/main.go +++ b/HadesAPI/main.go @@ -40,10 +40,16 @@ type HadesAPIConfig struct { var cfg HadesAPIConfig +const minAuthKeyLen = 16 + func (c HadesAPIConfig) validate() error { if c.AuthKey == "" && !c.AllowUnauthenticated { return errors.New("AUTH_KEY is empty: set AUTH_KEY, or set ALLOW_UNAUTHENTICATED=true to serve /build without authentication (local development only)") } + // A short key is guessable over the network; refuse it like an empty one. + if c.AuthKey != "" && len(c.AuthKey) < minAuthKeyLen { + return fmt.Errorf("AUTH_KEY must be at least %d characters", minAuthKeyLen) + } return nil } diff --git a/HadesAPI/main_test.go b/HadesAPI/main_test.go index 2357abf9..0496d3b1 100644 --- a/HadesAPI/main_test.go +++ b/HadesAPI/main_test.go @@ -8,6 +8,8 @@ import ( func TestConfigValidateAuth(t *testing.T) { assert.Error(t, HadesAPIConfig{}.validate()) - assert.NoError(t, HadesAPIConfig{AuthKey: "secret"}.validate()) + assert.Error(t, HadesAPIConfig{AuthKey: "secret"}.validate()) + assert.Error(t, HadesAPIConfig{AuthKey: "secret", AllowUnauthenticated: true}.validate()) + assert.NoError(t, HadesAPIConfig{AuthKey: "0123456789abcdef"}.validate()) assert.NoError(t, HadesAPIConfig{AllowUnauthenticated: true}.validate()) } diff --git a/HadesScheduler/HadesOperator/Readme.md b/HadesScheduler/HadesOperator/Readme.md index 36c2c4f5..0984e717 100644 --- a/HadesScheduler/HadesOperator/Readme.md +++ b/HadesScheduler/HadesOperator/Readme.md @@ -34,7 +34,9 @@ The Hades Operator is designed to: modify `hadesOperator.clusterWide` top be `true` ```sh kubectl create namespace hades - kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + # NATS credentials, shared by the NATS server and the Hades components (letters and digits only) + kubectl -n hades create secret generic hades-nats-auth --from-literal=NATS_USERNAME=hades --from-literal=NATS_PASSWORD="hades$(openssl rand -hex 32)" helm upgrade --install hades ./helm/hades -n hades --create-namespace --set hadesApi.authKeySecret=hades-auth ``` > **NOTE**: `/build` then needs Basic Auth with user `hades` and that key. `--set hadesApi.allowUnauthenticated=true` is for throwaway test installs only. diff --git a/Readme.md b/Readme.md index 4619fe14..6ef2df1d 100644 --- a/Readme.md +++ b/Readme.md @@ -112,7 +112,9 @@ For production deployments, Hades is designed to run natively within a Kubernete ```fish kubectl create namespace hades - kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + # NATS credentials, shared by the NATS server and the Hades components (letters and digits only) + kubectl -n hades create secret generic hades-nats-auth --from-literal=NATS_USERNAME=hades --from-literal=NATS_PASSWORD="hades$(openssl rand -hex 32)" # Install the published chart from GHCR (recommended) helm upgrade --install hades oci://ghcr.io/hades-scheduler/charts/hades \ @@ -385,12 +387,14 @@ See **[docs/configuration.md](./docs/configuration.md)** for the complete, per-c Breaking changes: +- NATS requires authentication on Kubernetes. The chart needs a Secret `hades-nats-auth` with keys `NATS_USERNAME` and `NATS_PASSWORD` (`nats.auth.secretName`); the NATS server and every component read it, so nothing starts without it. Setting only one of `NATS_USERNAME` and `NATS_PASSWORD` now fails at start. +- `AUTH_KEY` must have at least 16 characters; the API exits at start otherwise. - `AUTH_KEY` is required. The API exits at start with an empty key unless `ALLOW_UNAUTHENTICATED=true` (local development only). In Helm, an empty `hadesApi.authKeySecret` now needs `hadesApi.allowUnauthenticated: true`. - Callbacks (`callback_url`, `status_callback_url`) to loopback, private, CGNAT, link-local, unspecified, multicast and other special-purpose addresses (for example `198.18.0.0/15`) are refused. List internal receivers (for example Artemis in the same cluster or compose network) in `CALLBACK_ALLOWED_HOSTS` (Helm: `hadesLogManager.callbackAllowedHosts`). The log callback no longer follows redirects and callbacks ignore HTTP proxy variables. The built-in list cannot know a cluster's pod and service ranges when those use public-looking addresses; list them in `CALLBACK_BLOCKED_CIDRS` (comma-separated CIDRs, Helm: `hadesLogManager.callbackBlockedCIDRs`). An entry in `CALLBACK_ALLOWED_HOSTS` still wins over it. - Docker executor: steps may only request the network modes in `DOCKER_ALLOWED_NETWORKS` (default `none,bridge,default`). `host` and named networks fail the job unless listed. - `/build` bodies are capped at 1 MiB (413 above) and duplicate step ids, more than 100 steps and cpu or memory limits above 1024 cores or 1T are rejected with 400. The operator also caps a step at `JOB_MAX_CPU_LIMIT` (16) and `JOB_MAX_MEMORY_LIMIT` (32Gi) and fails a job the API server cannot create instead of retrying it. - `compose.yml` requires `AUTH_KEY` in `.env` and publishes API, log manager, NATS and Jaeger on `127.0.0.1` only. `docker-compose.deploy.yml` publishes only Traefik on 80/443 (Compose 2.24 or later) and no longer routes the log manager publicly. -- Kubernetes: job pods no longer mount a service account token and run with the `RuntimeDefault` seccomp profile and `allowPrivilegeEscalation: false`. The operator role has no access to secrets or configmaps. `hadesOperator.DeleteOnComplete` now defaults to `true`. Optional `networkPolicy.enabled` (default `false`) is new. +- Kubernetes: job pods no longer mount a service account token and run with the `RuntimeDefault` seccomp profile and `allowPrivilegeEscalation: false`. The operator role has no access to secrets or configmaps. `hadesOperator.DeleteOnComplete` now defaults to `true`. `networkPolicy.enabled` is new and on by default: job pods reach only DNS inside the cluster and no private, CGNAT or link-local address (`networkPolicy.jobPods.blockedCIDRs`), and only Hades components reach NATS. A job that needs a host on a private address, or a cluster DNS that is not `kube-dns` in `kube-system`, needs the matching `networkPolicy.jobPods` values. - Kubernetes: jobs now get a default 1 h timeout, 2 CPU and 4Gi memory limits per step, a 10Gi shared volume and no `NET_RAW`/`MKNOD` capabilities or service-link variables. Change or disable the defaults with `hadesOperator.jobDefaultTimeoutSeconds`, `jobMaxTimeoutSeconds`, `jobDefaultCPULimit`, `jobDefaultMemoryLimit` and `jobSharedVolumeSizeLimit` (empty or `0` disables each). - Dashboard login allows 5 attempts per client address per 15 minutes. Behind a reverse proxy, set `DASHBOARD_TRUSTED_PROXIES`. @@ -481,11 +485,13 @@ on the first deploy and kept in sync afterwards): | Secret | Purpose | |---|---| | `KUBE_CONFIG` | kubeconfig (plain YAML) for the target cluster | -| `AUTH_KEY` | protects the API `/build` endpoint (Secret `hades-auth`) | +| `AUTH_KEY` | protects the API `/build` endpoint (Secret `hades-auth`, >=16 chars) | | `DASHBOARD_USERNAME` | dashboard login user (Secret `hades-dashboard`) | | `DASHBOARD_PASSWORD_HASH` | bcrypt hash of the dashboard password | | `DASHBOARD_SESSION_SECRET` | dashboard session signing secret (>=32 chars) | +The deploy stops before it touches the cluster when one of them is missing or too weak (`AUTH_KEY` under 16 characters, a password hash that is not bcrypt, a session secret under 32 characters). The NATS credentials (Secret `hades-nats-auth`) are generated in the cluster on the first deploy and kept afterwards. + `k8s-prod` uses a required-reviewer protection rule, so a production deploy waits for manual approval. TLS is handled in-cluster by cert-manager (the `letsencrypt-prod` cluster-issuer), so no ACME secrets are needed in the workflow. diff --git a/docs/configuration.md b/docs/configuration.md index 08d15123..7d9499aa 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -19,8 +19,8 @@ Every component connects to NATS with the same `ConnectionConfig`. | Variable | Default | Description | Source | | -------- | ------- | ----------- | ------ | | `NATS_URL` | `nats://localhost:4222` | NATS server URL. | `shared/nats/connection.go` | -| `NATS_USERNAME` | | NATS username (optional). | `shared/nats/connection.go` | -| `NATS_PASSWORD` | | NATS password (optional). | `shared/nats/connection.go` | +| `NATS_USERNAME` | | NATS username. Set together with `NATS_PASSWORD` or not at all. | `shared/nats/connection.go` | +| `NATS_PASSWORD` | | NATS password. Set together with `NATS_USERNAME` or not at all. | `shared/nats/connection.go` | | `NATS_TLS_ENABLED` | `false` | Enable TLS for the NATS connection. | `shared/nats/connection.go` | ## Metrics (all components) @@ -74,7 +74,7 @@ The bundled Jaeger UI Service is cluster-internal (ClusterIP) by default, so you | -------- | ------- | ----------- | ------ | | `API_PORT` | `8080` | Port the HTTP API listens on. | `HadesAPI/main.go` (`HadesAPIConfig`) | | `METRICS_PORT` | `8082` | Port the Prometheus `/metrics` endpoint listens on (see [Metrics](#metrics-all-components)). | `HadesAPI/main.go` (`HadesAPIConfig`) | -| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Required: the API exits at start when it is empty unless `ALLOW_UNAUTHENTICATED=true`. | `HadesAPI/main.go` (`HadesAPIConfig`) | +| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Required, at least 16 characters: the API exits at start when it is empty (unless `ALLOW_UNAUTHENTICATED=true`) or shorter. | `HadesAPI/main.go` (`HadesAPIConfig`) | | `ALLOW_UNAUTHENTICATED` | `false` | Start with an empty `AUTH_KEY` and serve `/build` without auth. Local development only; `make run` sets it. | `HadesAPI/main.go` (`HadesAPIConfig`) | | `DASHBOARD_TRUSTED_PROXIES` | | Comma-separated proxy addresses or CIDRs whose `X-Forwarded-For` is trusted for the dashboard login lockout (5 attempts per client address per 15 minutes). Set it behind a reverse proxy, otherwise all clients share one lockout (a warning is logged at start). | `HadesAPI/main.go` (`HadesAPIConfig`) | diff --git a/helm/hades/Readme.md b/helm/hades/Readme.md index df081ca9..406aa772 100644 --- a/helm/hades/Readme.md +++ b/helm/hades/Readme.md @@ -60,7 +60,9 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, 3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, skip the Secret and pass `--set hadesApi.allowUnauthenticated=true` instead of `--set hadesApi.authKeySecret=hades-auth` (a set `authKeySecret` always wins). ```bash kubectl create namespace hades - kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + # NATS credentials, shared by the NATS server and the Hades components (letters and digits only) + kubectl -n hades create secret generic hades-nats-auth --from-literal=NATS_USERNAME=hades --from-literal=NATS_PASSWORD="hades$(openssl rand -hex 32)" ``` 4. Install the chart (default: operator mode) @@ -176,13 +178,14 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | monitoring.serviceMonitor.interval | string | `"30s"` | Prometheus scrape interval. | | monitoring.serviceMonitor.labels | object | `{}` | Extra labels on the ServiceMonitor so a Prometheus instance selects it (e.g. `{ release: kube-prometheus-stack }`). | | monitoring.serviceMonitor.scrapeTimeout | string | `"10s"` | Per-scrape timeout. | +| nats.auth.secretName | string | `"hades-nats-auth"` | Name of an existing Secret with keys `NATS_USERNAME` and `NATS_PASSWORD`. Required: the bundled NATS server only accepts these credentials and every Hades component connects with them, so nothing starts until the Secret exists. The password must be alphanumeric and start with a letter (the NATS server parses it from its environment). When you change the name, change the two `nats.container.env` references as well. | | nats.config.jetstream.enabled | bool | `true` | Enable JetStream (required by Hades). | | nats.config.jetstream.memoryStore.maxSize | string | `"1Gi"` | Maximum in-memory JetStream store size. | | nats.host | string | `"hades-nats.hades.svc.cluster.local"` | In-cluster NATS service hostname. | | nats.port | int | `4222` | NATS client port. | -| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[],"dns":{"namespaceLabels":{"kubernetes.io/metadata.name":"kube-system"},"podLabels":{"k8s-app":"kube-dns"}},"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | -| networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | -| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. Add the node addresses as well: the API server, kubelet and other node ports are "outside" for a NetworkPolicy. | +| networkPolicy | object | `{"enabled":true,"jobPods":{"blockedCIDRs":["10.0.0.0/8","172.16.0.0/12","192.168.0.0/16","100.64.0.0/10","169.254.0.0/16","198.18.0.0/15"],"dns":{"namespaceLabels":{"kubernetes.io/metadata.name":"kube-system"},"podLabels":{"k8s-app":"kube-dns"}},"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy (elsewhere they have no effect). The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | +| networkPolicy.enabled | bool | `true` | Render the NetworkPolicies below. Leave it on: job pods run untrusted code, and without the policies they reach NATS, the log manager and every other pod in the cluster. | +| networkPolicy.jobPods.blockedCIDRs | list | `["10.0.0.0/8","172.16.0.0/12","192.168.0.0/16","100.64.0.0/10","169.254.0.0/16","198.18.0.0/15"]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only. The default covers the private, CGNAT, link-local (cloud metadata) and benchmarking ranges, which is where pod, service and node networks usually live. On a cluster whose pods, services or nodes use other (public) addresses, add those ranges: for a NetworkPolicy the API server, kubelet and other node ports are "outside". Setting this replaces the default list. | | networkPolicy.jobPods.dns.namespaceLabels | object | `{"kubernetes.io/metadata.name":"kube-system"}` | Labels of the namespace running the cluster DNS. Job pods may reach port 53 only on pods in it that match `podLabels`. The defaults fit CoreDNS and kube-dns; adjust them for other DNS deployments. | | networkPolicy.jobPods.dns.podLabels | object | `{"k8s-app":"kube-dns"}` | Labels of the cluster DNS pods. | | networkPolicy.jobPods.ingressNamespaces | list | `[]` | Namespaces of the cluster's ingress controller(s). Job pods may reach their pods on ports 80 and 443, so that hosts served by this cluster (a git server the job clones from, for example) stay reachable although the pod network is blocked. | diff --git a/helm/hades/Readme.md.gotmpl b/helm/hades/Readme.md.gotmpl index 23a71dad..909bf232 100644 --- a/helm/hades/Readme.md.gotmpl +++ b/helm/hades/Readme.md.gotmpl @@ -60,7 +60,9 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, 3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, skip the Secret and pass `--set hadesApi.allowUnauthenticated=true` instead of `--set hadesApi.authKeySecret=hades-auth` (a set `authKeySecret` always wins). ```bash kubectl create namespace hades - kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + # NATS credentials, shared by the NATS server and the Hades components (letters and digits only) + kubectl -n hades create secret generic hades-nats-auth --from-literal=NATS_USERNAME=hades --from-literal=NATS_PASSWORD="hades$(openssl rand -hex 32)" ``` 4. Install the chart (default: operator mode) diff --git a/helm/hades/templates/_nats.tpl b/helm/hades/templates/_nats.tpl new file mode 100644 index 00000000..90ec895b --- /dev/null +++ b/helm/hades/templates/_nats.tpl @@ -0,0 +1,33 @@ +{{/* +NATS connection env entries: the URL plus the credentials from the +nats.auth.secretName Secret. The bundled NATS server reads the same Secret +(nats.container.env), so rendering fails when the two names differ. Include +with `nindent 12` inside a container's env list. +*/}} +{{- define "hades.natsEnv" -}} +{{- $secret := .Values.nats.auth.secretName -}} +{{- if not $secret -}} +{{- fail "nats.auth.secretName is empty: set it to a Secret with keys NATS_USERNAME and NATS_PASSWORD (NATS does not run without authentication)" -}} +{{- end -}} +{{- range $key := list "NATS_USERNAME" "NATS_PASSWORD" -}} +{{- $server := dig $key "valueFrom" "secretKeyRef" "name" "" ($.Values.nats.container.env | default dict) -}} +{{- if ne $server $secret -}} +{{- fail (printf "nats.container.env.%s must read Secret %q (nats.auth.secretName), got %q: the NATS server and the Hades components need the same credentials" $key $secret $server) -}} +{{- end -}} +{{- end -}} +{{- if ne (dig "authorization" "password" "" (.Values.nats.config.merge | default dict)) "<< $NATS_PASSWORD >>" -}} +{{- fail "nats.config.merge.authorization was removed or changed: NATS must require the credentials from nats.auth.secretName" -}} +{{- end }} +- name: NATS_URL + value: "nats://{{ .Values.nats.host }}:{{ .Values.nats.port }}" +- name: NATS_USERNAME + valueFrom: + secretKeyRef: + name: {{ $secret | quote }} + key: NATS_USERNAME +- name: NATS_PASSWORD + valueFrom: + secretKeyRef: + name: {{ $secret | quote }} + key: NATS_PASSWORD +{{- end -}} diff --git a/helm/hades/templates/hades-api-deployment.yaml b/helm/hades/templates/hades-api-deployment.yaml index 81d92466..36f8f405 100644 --- a/helm/hades/templates/hades-api-deployment.yaml +++ b/helm/hades/templates/hades-api-deployment.yaml @@ -53,8 +53,7 @@ spec: value: {{ .Values.debug | quote }} - name: METRICS_PORT value: {{ .Values.monitoring.port | quote }} - - name: NATS_URL - value: "nats://{{ .Values.nats.host }}:{{ .Values.nats.port }}" + {{- include "hades.natsEnv" . | nindent 12 }} # Deployed version shown in the dashboard header. Derived from the # image tag so it always matches the running build. - name: HADES_VERSION diff --git a/helm/hades/templates/hades-log-manager-deployment.yaml b/helm/hades/templates/hades-log-manager-deployment.yaml index 6f9893fa..4362b2ea 100644 --- a/helm/hades/templates/hades-log-manager-deployment.yaml +++ b/helm/hades/templates/hades-log-manager-deployment.yaml @@ -50,8 +50,7 @@ spec: value: {{ .Values.debug | quote }} - name: METRICS_PORT value: {{ .Values.monitoring.port | quote }} - - name: NATS_URL - value: "nats://{{ .Values.nats.host }}:{{ .Values.nats.port }}" + {{- include "hades.natsEnv" . | nindent 12 }} - name: HADESLOGMANAGER_API_PORT value: {{ .Values.hadesLogManager.service.targetPort | quote }} - name: LOG_MAX_TOTAL_BYTES diff --git a/helm/hades/templates/hades-operator-deployment.yaml b/helm/hades/templates/hades-operator-deployment.yaml index b54e37bb..a26f8754 100644 --- a/helm/hades/templates/hades-operator-deployment.yaml +++ b/helm/hades/templates/hades-operator-deployment.yaml @@ -44,8 +44,7 @@ spec: # DEV_MODE; wire it to the same toggle so debug mode is effective here. - name: DEV_MODE value: {{ .Values.debug | quote }} - - name: NATS_URL - value: "nats://{{ .Values.nats.host }}:{{ .Values.nats.port }}" + {{- include "hades.natsEnv" . | nindent 12 }} # Everything except WATCH_NAMESPACE is independent of the watch # scope, so it is injected in both namespaced and cluster-wide mode. - name: REQUEUE_DELAY diff --git a/helm/hades/templates/hades-scheduler-deployment.yaml b/helm/hades/templates/hades-scheduler-deployment.yaml index 526afceb..bc8f4b26 100644 --- a/helm/hades/templates/hades-scheduler-deployment.yaml +++ b/helm/hades/templates/hades-scheduler-deployment.yaml @@ -33,8 +33,7 @@ spec: value: {{ .Values.debug | quote }} - name: METRICS_PORT value: {{ .Values.monitoring.port | quote }} - - name: NATS_URL - value: "nats://{{ .Values.nats.host }}:{{ .Values.nats.port }}" + {{- include "hades.natsEnv" . | nindent 12 }} - name: HADES_EXECUTOR value: {{ .Values.hadesScheduler.executor }} - name: CONCURRENCY diff --git a/helm/hades/values-prod.yaml b/helm/hades/values-prod.yaml index 8fa7ba37..6b9ab669 100644 --- a/helm/hades/values-prod.yaml +++ b/helm/hades/values-prod.yaml @@ -1,7 +1,7 @@ # Production overrides, layered on top of values.yaml by the # "Deploy to Kubernetes (prod)" workflow (namespace: hades). # Only non-secret configuration lives here; credentials are injected as -# Kubernetes Secrets (hades-auth, hades-dashboard) created by the deploy workflow. +# Kubernetes Secrets (hades-auth, hades-dashboard, hades-nats-auth) created by the deploy workflow. hadesApi: # Protect the /build endpoint with the AUTH_KEY from the hades-auth Secret. @@ -12,3 +12,42 @@ hadesApi: ingress: host: hades.student.k8s.aet.cit.tum.de + +hadesLogManager: + # Callbacks must not reach the cluster nodes (same list as + # networkPolicy.jobPods.blockedCIDRs below; pods and services are covered by + # the built-in list). A receiver inside the cluster needs an entry in + # callbackAllowedHosts. + callbackBlockedCIDRs: "131.159.88.18/31,131.159.88.26/31,131.159.88.28/31,131.159.88.43/32,131.159.88.52/30,131.159.88.56/29,131.159.88.67/32,131.159.88.68/30,131.159.88.75/32,131.159.88.76/30" + +# Same cluster as the test install, hence the same lists as values-test.yaml. +# Job pods run untrusted code: keep them away from NATS, the log manager and the +# other Hades pods. The cluster CNI (Calico) matches ipBlock rules against pod +# IPs, so the pod network has to be excluded from the "outside" rule explicitly. +networkPolicy: + jobPods: + ingressNamespaces: + - ingress-nginx + blockedCIDRs: + # Chart defaults (this list replaces them). 198.18.0.0/15 holds the pod + # and service networks of this cluster. + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + - 100.64.0.0/10 + - 169.254.0.0/16 + - 198.18.0.0/15 + # Node addresses of the cluster (API server, kubelet, etcd, SSH and + # node-exporter ports are otherwise reachable from job pods). The ingress + # load balancers share the subnet and must stay reachable, hence no /24. + # Keep in sync when nodes are added. + - 131.159.88.18/31 + - 131.159.88.26/31 + - 131.159.88.28/31 + - 131.159.88.43/32 + - 131.159.88.52/30 + - 131.159.88.56/29 + - 131.159.88.67/32 + - 131.159.88.68/30 + - 131.159.88.75/32 + - 131.159.88.76/30 diff --git a/helm/hades/values-test.yaml b/helm/hades/values-test.yaml index 7b2fd0db..2ffaee7d 100644 --- a/helm/hades/values-test.yaml +++ b/helm/hades/values-test.yaml @@ -1,7 +1,7 @@ # Test overrides, layered on top of values.yaml by the # "Deploy to Kubernetes (test)" workflow (namespace: hades-test). # Only non-secret configuration lives here; credentials are injected as -# Kubernetes Secrets (hades-auth, hades-dashboard) created by the deploy workflow. +# Kubernetes Secrets (hades-auth, hades-dashboard, hades-nats-auth) created by the deploy workflow. hadesLogManager: # Callbacks must not reach the cluster nodes either (same list as @@ -44,13 +44,19 @@ nats: # other Hades pods. The cluster CNI (Calico) matches ipBlock rules against pod # IPs, so the pod network has to be excluded from the "outside" rule explicitly. networkPolicy: - enabled: true jobPods: ingressNamespaces: - ingress-nginx blockedCIDRs: + # Chart defaults (this list replaces them). 198.18.0.0/15 holds the pod + # and service networks of this cluster. + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + - 100.64.0.0/10 + - 169.254.0.0/16 - 198.18.0.0/15 - # Node addresses of the test cluster (API server, kubelet, etcd, SSH and + # Node addresses of the cluster (API server, kubelet, etcd, SSH and # node-exporter ports are otherwise reachable from job pods). The ingress # load balancers share the subnet and must stay reachable, hence no /24. # Keep in sync when nodes are added. diff --git a/helm/hades/values.yaml b/helm/hades/values.yaml index 22fcff63..15ed5ee2 100644 --- a/helm/hades/values.yaml +++ b/helm/hades/values.yaml @@ -366,21 +366,29 @@ tracing: # `existingSecret` is given; the chart hashes it into an htpasswd Secret. password: "" -# -- NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by -# default because the NATS policy assumes the labels of the bundled NATS chart. -# The job-pod policy covers the release namespace only: with +# -- NetworkPolicies. They need a CNI that enforces NetworkPolicy (elsewhere they +# have no effect). The job-pod policy covers the release namespace only: with # `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. networkPolicy: - # -- Render the NetworkPolicies below. Verify the NATS pod labels first - # (`kubectl get pods -n --show-labels`), then enable. - enabled: false + # -- Render the NetworkPolicies below. Leave it on: job pods run untrusted + # code, and without the policies they reach NATS, the log manager and every + # other pod in the cluster. + enabled: true jobPods: # -- CIDRs job pods may not reach. Inside the cluster job pods are allowed - # DNS only; set this to the cluster pod and service CIDRs on CNIs that match - # pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can - # still reach pods. Add the node addresses as well: the API - # server, kubelet and other node ports are "outside" for a NetworkPolicy. - blockedCIDRs: [] + # DNS only. The default covers the private, CGNAT, link-local (cloud + # metadata) and benchmarking ranges, which is where pod, service and node + # networks usually live. On a cluster whose pods, services or nodes use other + # (public) addresses, add those ranges: for a NetworkPolicy the API server, + # kubelet and other node ports are "outside". Setting this replaces the + # default list. + blockedCIDRs: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + - 100.64.0.0/10 + - 169.254.0.0/16 + - 198.18.0.0/15 dns: # -- Labels of the namespace running the cluster DNS. Job pods may reach # port 53 only on pods in it that match `podLabels`. The defaults fit @@ -401,7 +409,31 @@ networkPolicy: podLabels: {} nats: + auth: + # -- Name of an existing Secret with keys `NATS_USERNAME` and `NATS_PASSWORD`. + # Required: the bundled NATS server only accepts these credentials and every + # Hades component connects with them, so nothing starts until the Secret + # exists. The password must be alphanumeric and start with a letter (the NATS + # server parses it from its environment). When you change the name, change + # the two `nats.container.env` references as well. + secretName: hades-nats-auth + container: + env: + NATS_USERNAME: + valueFrom: + secretKeyRef: + name: hades-nats-auth + key: NATS_USERNAME + NATS_PASSWORD: + valueFrom: + secretKeyRef: + name: hades-nats-auth + key: NATS_PASSWORD config: + merge: + authorization: + user: << $NATS_USERNAME >> + password: << $NATS_PASSWORD >> jetstream: # -- Enable JetStream (required by Hades). enabled: true diff --git a/shared/nats/connection.go b/shared/nats/connection.go index f25a47b2..f8953307 100644 --- a/shared/nats/connection.go +++ b/shared/nats/connection.go @@ -2,6 +2,7 @@ package nats import ( "crypto/tls" + "errors" "log/slog" "time" @@ -39,8 +40,11 @@ func SetupNatsConnection(name string, timeout, reconnectWait time.Duration, maxR nats.MaxReconnects(maxReconnects), } - // Add credentials if provided - if config.Username != "" && config.Password != "" { + // Half a credential pair would silently connect without authentication. + if (config.Username == "") != (config.Password == "") { + return nil, errors.New("NATS_USERNAME and NATS_PASSWORD must be set together") + } + if config.Username != "" { opts = append(opts, nats.UserInfo(config.Username, config.Password)) } diff --git a/shared/nats/connection_test.go b/shared/nats/connection_test.go new file mode 100644 index 00000000..2d9c1eb7 --- /dev/null +++ b/shared/nats/connection_test.go @@ -0,0 +1,14 @@ +package nats + +import "testing" + +func TestSetupNatsConnectionRejectsHalfCredentials(t *testing.T) { + for _, c := range []ConnectionConfig{ + {URL: "nats://127.0.0.1:1", Username: "hades"}, + {URL: "nats://127.0.0.1:1", Password: "secret"}, + } { + if _, err := SetupDefaultNatsConnection(c); err == nil || err.Error() != "NATS_USERNAME and NATS_PASSWORD must be set together" { + t.Errorf("config %+v: got %v", c, err) + } + } +} diff --git a/website/docs/deployment/helm-values.md b/website/docs/deployment/helm-values.md index 3668ef1b..5f41971b 100644 --- a/website/docs/deployment/helm-values.md +++ b/website/docs/deployment/helm-values.md @@ -66,7 +66,9 @@ The Scheduler does not create Pods directly; instead it creates `BuildJob` CRs, 3. Create the API auth Secret. `/build` then needs Basic Auth with user `hades` and that key. For throwaway test installs only, skip the Secret and pass `--set hadesApi.allowUnauthenticated=true` instead of `--set hadesApi.authKeySecret=hades-auth` (a set `authKeySecret` always wins). ```bash kubectl create namespace hades - kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= + # NATS credentials, shared by the NATS server and the Hades components (letters and digits only) + kubectl -n hades create secret generic hades-nats-auth --from-literal=NATS_USERNAME=hades --from-literal=NATS_PASSWORD="hades$(openssl rand -hex 32)" ``` 4. Install the chart (default: operator mode) @@ -182,13 +184,14 @@ The table below is generated by [helm-docs](https://github.com/norwoodj/helm-doc | monitoring.serviceMonitor.interval | string | `"30s"` | Prometheus scrape interval. | | monitoring.serviceMonitor.labels | object | `{}` | Extra labels on the ServiceMonitor so a Prometheus instance selects it (e.g. `{ release: kube-prometheus-stack }`). | | monitoring.serviceMonitor.scrapeTimeout | string | `"10s"` | Per-scrape timeout. | +| nats.auth.secretName | string | `"hades-nats-auth"` | Name of an existing Secret with keys `NATS_USERNAME` and `NATS_PASSWORD`. Required: the bundled NATS server only accepts these credentials and every Hades component connects with them, so nothing starts until the Secret exists. The password must be alphanumeric and start with a letter (the NATS server parses it from its environment). When you change the name, change the two `nats.container.env` references as well. | | nats.config.jetstream.enabled | bool | `true` | Enable JetStream (required by Hades). | | nats.config.jetstream.memoryStore.maxSize | string | `"1Gi"` | Maximum in-memory JetStream store size. | | nats.host | string | `"hades-nats.hades.svc.cluster.local"` | In-cluster NATS service hostname. | | nats.port | int | `4222` | NATS client port. | -| networkPolicy | object | `{"enabled":false,"jobPods":{"blockedCIDRs":[],"dns":{"namespaceLabels":{"kubernetes.io/metadata.name":"kube-system"},"podLabels":{"k8s-app":"kube-dns"}},"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy and are off by default because the NATS policy assumes the labels of the bundled NATS chart. The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | -| networkPolicy.enabled | bool | `false` | Render the NetworkPolicies below. Verify the NATS pod labels first (`kubectl get pods -n --show-labels`), then enable. | -| networkPolicy.jobPods.blockedCIDRs | list | `[]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only; set this to the cluster pod and service CIDRs on CNIs that match pod IPs against ipBlock rules (Calico), otherwise the "outside" rule can still reach pods. Add the node addresses as well: the API server, kubelet and other node ports are "outside" for a NetworkPolicy. | +| networkPolicy | object | `{"enabled":true,"jobPods":{"blockedCIDRs":["10.0.0.0/8","172.16.0.0/12","192.168.0.0/16","100.64.0.0/10","169.254.0.0/16","198.18.0.0/15"],"dns":{"namespaceLabels":{"kubernetes.io/metadata.name":"kube-system"},"podLabels":{"k8s-app":"kube-dns"}},"ingressNamespaces":[]},"nats":{"podLabels":{}}}` | NetworkPolicies. They need a CNI that enforces NetworkPolicy (elsewhere they have no effect). The job-pod policy covers the release namespace only: with `hadesOperator.clusterWide`, BuildJobs created in other namespaces are not covered. | +| networkPolicy.enabled | bool | `true` | Render the NetworkPolicies below. Leave it on: job pods run untrusted code, and without the policies they reach NATS, the log manager and every other pod in the cluster. | +| networkPolicy.jobPods.blockedCIDRs | list | `["10.0.0.0/8","172.16.0.0/12","192.168.0.0/16","100.64.0.0/10","169.254.0.0/16","198.18.0.0/15"]` | CIDRs job pods may not reach. Inside the cluster job pods are allowed DNS only. The default covers the private, CGNAT, link-local (cloud metadata) and benchmarking ranges, which is where pod, service and node networks usually live. On a cluster whose pods, services or nodes use other (public) addresses, add those ranges: for a NetworkPolicy the API server, kubelet and other node ports are "outside". Setting this replaces the default list. | | networkPolicy.jobPods.dns.namespaceLabels | object | `{"kubernetes.io/metadata.name":"kube-system"}` | Labels of the namespace running the cluster DNS. Job pods may reach port 53 only on pods in it that match `podLabels`. The defaults fit CoreDNS and kube-dns; adjust them for other DNS deployments. | | networkPolicy.jobPods.dns.podLabels | object | `{"k8s-app":"kube-dns"}` | Labels of the cluster DNS pods. | | networkPolicy.jobPods.ingressNamespaces | list | `[]` | Namespaces of the cluster's ingress controller(s). Job pods may reach their pods on ports 80 and 443, so that hosts served by this cluster (a git server the job clones from, for example) stay reachable although the pod network is blocked. | diff --git a/website/docs/deployment/kubernetes-github-actions.md b/website/docs/deployment/kubernetes-github-actions.md index cf3952fd..13eb4518 100644 --- a/website/docs/deployment/kubernetes-github-actions.md +++ b/website/docs/deployment/kubernetes-github-actions.md @@ -30,6 +30,8 @@ Both are `workflow_dispatch` only - nothing deploys automatically. They call the 3. **Ensure the namespace** exists. 4. **Create/update the app Secrets** (`hades-auth`, `hades-dashboard`) from the environment secrets. The first deploy creates them; later deploys keep them in sync. + The run fails before this step when a secret is missing or too weak. The NATS + credentials (`hades-nats-auth`) are generated once in the cluster. 5. **Apply the CRDs** with a server-side apply. Helm never upgrades CRDs after the first install, and server-side apply avoids the client-side annotation size limit on the `BuildJob` CRD. @@ -103,11 +105,13 @@ Add these to **each** GitHub environment (`k8s-prod` and `k8s-test`): | Secret | Purpose | |---|---| | `KUBE_CONFIG` | kubeconfig (plain YAML) for the target cluster | -| `AUTH_KEY` | protects the API `/build` endpoint (Secret `hades-auth`) | +| `AUTH_KEY` | protects the API `/build` endpoint (Secret `hades-auth`, >=16 chars) | | `DASHBOARD_USERNAME` | dashboard login user (Secret `hades-dashboard`) | | `DASHBOARD_PASSWORD_HASH` | bcrypt hash of the dashboard password | | `DASHBOARD_SESSION_SECRET` | dashboard session signing secret (>=32 chars) | +The deploy stops before it touches the cluster when one of them is missing or too weak (`AUTH_KEY` under 16 characters, a password hash that is not bcrypt, a session secret under 32 characters). The NATS credentials (Secret `hades-nats-auth`) are generated in the cluster on the first deploy and kept afterwards. + Paste the kubeconfig file's contents directly as `KUBE_CONFIG` (plain YAML, no encoding). ## Production approval diff --git a/website/docs/installation/helm-chart.md b/website/docs/installation/helm-chart.md index 4cc3d447..97c8e360 100644 --- a/website/docs/installation/helm-chart.md +++ b/website/docs/installation/helm-chart.md @@ -24,7 +24,9 @@ Create the API auth Secret first. `/build` then needs Basic Auth with user `hade ```bash kubectl create namespace hades -kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= +kubectl -n hades create secret generic hades-auth --from-literal=AUTH_KEY= +# NATS credentials, shared by the NATS server and the Hades components (letters and digits only) +kubectl -n hades create secret generic hades-nats-auth --from-literal=NATS_USERNAME=hades --from-literal=NATS_PASSWORD="hades$(openssl rand -hex 32)" ``` ```bash diff --git a/website/docs/operation-modes/k8s-operator.md b/website/docs/operation-modes/k8s-operator.md index 1bac19f9..03c67dc8 100644 --- a/website/docs/operation-modes/k8s-operator.md +++ b/website/docs/operation-modes/k8s-operator.md @@ -51,7 +51,7 @@ The scheduler uses a dynamic Kubernetes client to create `BuildJob` custom resou Job pods do not mount a service account token, use the `RuntimeDefault` seccomp profile and set `allowPrivilegeEscalation: false`. They still run as the image's user, but every container drops the `NET_RAW` and `MKNOD` capabilities and the pod gets no service-link environment variables (`enableServiceLinks: false`). The operator also applies defaults to what a job leaves unset: a 1 h timeout (`JOB_DEFAULT_TIMEOUT_SECONDS`, cap with `JOB_MAX_TIMEOUT_SECONDS`), a 2 CPU and 4Gi memory limit per step (`JOB_DEFAULT_CPU_LIMIT`, `JOB_DEFAULT_MEMORY_LIMIT`) and clamps any step CPU or memory limit and request to 16 CPU and 32Gi (`JOB_MAX_CPU_LIMIT`, `JOB_MAX_MEMORY_LIMIT`). A job the API server rejects as invalid or too large is marked `Failed` instead of being retried and a 10Gi size limit on the shared volume (`JOB_SHARED_VOLUME_SIZE_LIMIT`). Set a value to empty or `0` to disable it (Helm: `hadesOperator.jobDefaultTimeoutSeconds` and friends). The API and the log manager have their own service accounts, and the operator role has no access to secrets or configmaps. -Set `networkPolicy.enabled: true` (default `false`) to add two NetworkPolicies: job pods cannot reach the release namespace, and only Hades components reach NATS on 4222. Before enabling, check the NATS pod labels. On CNIs that match pod IPs against `ipBlock`, set `networkPolicy.jobPods.blockedCIDRs` to the pod CIDR. +The chart adds two NetworkPolicies (`networkPolicy.enabled`, default `true`): inside the cluster job pods reach DNS only, and only Hades components reach NATS on 4222. `networkPolicy.jobPods.blockedCIDRs` defaults to the private, CGNAT, link-local and benchmarking ranges; add the pod, service and node ranges of a cluster that uses other addresses. NATS also requires the credentials from the `hades-nats-auth` Secret (`nats.auth.secretName`), so a pod that gets past the policy still cannot read or publish. ## RBAC diff --git a/website/docs/reference/configuration.md b/website/docs/reference/configuration.md index c3e19792..bbb1bef8 100644 --- a/website/docs/reference/configuration.md +++ b/website/docs/reference/configuration.md @@ -22,8 +22,8 @@ Hades is configured entirely through environment variables (loaded via `caarlos0 | Variable | Default | Description | | -------- | ------- | ----------- | | `NATS_URL` | `nats://localhost:4222` | NATS server URL. | -| `NATS_USERNAME` | | NATS username (optional). | -| `NATS_PASSWORD` | | NATS password (optional). | +| `NATS_USERNAME` | | NATS username. Set together with `NATS_PASSWORD` or not at all. | +| `NATS_PASSWORD` | | NATS password. Set together with `NATS_USERNAME` or not at all. | | `NATS_TLS_ENABLED` | `false` | Enable TLS for the NATS connection. | ## Metrics (all components) @@ -51,7 +51,7 @@ Hades measures how much overhead it adds around a job, per step and per phase, c | -------- | ------- | ----------- | | `API_PORT` | `8080` | Port the HTTP API listens on. | | `METRICS_PORT` | `8082` | Port the Prometheus `/metrics` endpoint listens on. | -| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Required: the API exits at start when it is empty unless `ALLOW_UNAUTHENTICATED=true`. | +| `AUTH_KEY` | | HTTP Basic Auth key for the `hades` user. Required, at least 16 characters: the API exits at start when it is empty (unless `ALLOW_UNAUTHENTICATED=true`) or shorter. | | `ALLOW_UNAUTHENTICATED` | `false` | Serve `/build` without auth when `AUTH_KEY` is empty. Local development only. | | `DASHBOARD_TRUSTED_PROXIES` | | Proxy addresses or CIDRs whose `X-Forwarded-For` is trusted for the dashboard login lockout (5 attempts per client address per 15 minutes). Set it behind a reverse proxy. |