Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ NATS_TLS_ENABLED=false # Enable TLS for NATS connection (default: false)

### HadesAPI Options ###
API_PORT=8080 # Port for the API to listen on (default: 8080)
AUTH_KEY="" # Key used for Basic Auth with the API (default: "" for no auth)
AUTH_KEY="" # Key used for Basic Auth with the API. REQUIRED: the API refuses to start without it unless ALLOW_UNAUTHENTICATED=true, and compose.yml refuses to start without it. Generate one with `openssl rand -hex 32`.

### Dashboard (operator web UI, served by HadesAPI) ###
# The dashboard is disabled unless USERNAME, PASSWORD_HASH and SESSION_SECRET are
Expand Down
10 changes: 7 additions & 3 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ jobs:

- name: Determine benchmark mode and host
id: mode_resolve
env:
JENKINS_USER_SECRET: ${{ secrets.JENKINS_USER }}
JENKINS_TOKEN_SECRET: ${{ secrets.JENKINS_TOKEN }}
run: |
set -euo pipefail
echo "Selected mode: ${{ inputs.mode }}"
Expand All @@ -61,8 +64,9 @@ jobs:
jenkins)
echo "MODE_PATH=jenkins" >> $GITHUB_ENV
echo "BENCHMARK_HOST=${{ vars.JENKINS_HOST }}" >> $GITHUB_ENV
echo "JENKINS_USER=${{ secrets.JENKINS_USER }}" >> $GITHUB_ENV
echo "JENKINS_TOKEN=${{ secrets.JENKINS_TOKEN }}" >> $GITHUB_ENV
echo "JENKINS_USER=${JENKINS_USER_SECRET}" >> $GITHUB_ENV
echo "::add-mask::${JENKINS_TOKEN_SECRET}"
echo "JENKINS_TOKEN=${JENKINS_TOKEN_SECRET}" >> $GITHUB_ENV
echo "JENKINS_JOB_PATH=${{ vars.JENKINS_JOB_PATH }}" >> $GITHUB_ENV
;;
*)
Expand Down Expand Up @@ -178,7 +182,7 @@ jobs:
QUERY="host=${ENCODED_HOST}&count=${COUNT}&commit_hash=${COMMIT_HASH}"
fi

echo "POST ${URL}?${QUERY}"
echo "POST ${URL}?${QUERY//${JENKINS_TOKEN:-__none__}/***}"

curl -sS --fail-with-body --retry 3 --retry-delay 2 --max-time 120 \
-X POST "${URL}?${QUERY}" \
Expand Down
50 changes: 31 additions & 19 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,16 @@ concurrency:

permissions:
contents: read
packages: write
actions: write

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: go.work
cache-dependency-path: "**/*.sum"
Expand All @@ -53,9 +53,11 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: go.work
cache-dependency-path: "**/*.sum"
Expand All @@ -79,9 +81,11 @@ jobs:
- name: HadesOperator
path: HadesScheduler/HadesOperator
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: go.work
cache-dependency-path: "**/*.sum"
Expand All @@ -91,19 +95,19 @@ jobs:
run: cd ${{ matrix.path }} && go test ./... -json > "${{ github.workspace }}/TestResults-${{ matrix.name }}.json"
- name: Upload test results
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: Go-results-${{ matrix.name }}
path: TestResults-${{ matrix.name }}.json

ui:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Setup Node
uses: actions/setup-node@v4
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 22
cache: npm
Expand All @@ -120,51 +124,59 @@ jobs:
build-api:
needs: [lint, build, test, ui]
if: github.actor != 'dependabot[bot]'
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0
permissions:
contents: read
packages: write
with:
image-name: hades-scheduler/hades/hades-api
docker-file: ./HadesAPI/Dockerfile
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable={{is_default_branch}}
secrets: inherit

build-scheduler:
needs: [lint, build, test]
if: github.actor != 'dependabot[bot]'
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0
permissions:
contents: read
packages: write
with:
image-name: hades-scheduler/hades/hades-scheduler
docker-file: ./HadesScheduler/Dockerfile
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable={{is_default_branch}}
secrets: inherit

build-operator:
needs: [lint, build, test]
if: github.actor != 'dependabot[bot]'
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0
permissions:
contents: read
packages: write
with:
image-name: hades-scheduler/hades/hades-operator
docker-file: ./HadesScheduler/HadesOperator/Dockerfile
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable={{is_default_branch}}
secrets: inherit

build-log-manager:
needs: [lint, build, test]
if: github.actor != 'dependabot[bot]'
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@v1.2.0
uses: ls1intum/.github/.github/workflows/build-and-push-docker-image.yml@ee1b52f07891ed489887681c09c52b4a83fa9fb2 # v1.2.0
permissions:
contents: read
packages: write
with:
image-name: hades-scheduler/hades/hades-log-manager
docker-file: ./HadesLogManager/Dockerfile
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable={{is_default_branch}}
secrets: inherit
4 changes: 2 additions & 2 deletions .github/workflows/deploy-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0
persist-credentials: false

- uses: actions/setup-node@v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: "22"
cache: yarn
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/deploy-k8s-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ name: Deploy to Kubernetes (test)
on:
workflow_dispatch:
inputs:
# For pr-N the chart and CRDs of that PR (fork PRs included) are applied with the
# test cluster's kubeconfig. Review the PR's helm/ and CRD changes before dispatching.
version:
description: "Image tag to deploy (e.g. latest, 1.0.0, pr-123, or a branch/sha tag)."
required: true
Expand Down
7 changes: 4 additions & 3 deletions .github/workflows/deploy-k8s.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,15 +56,16 @@ jobs:
environment: ${{ inputs.environment }}
steps:
- name: Checkout chart source
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
ref: ${{ inputs.chart-ref }}

- name: Install Helm
uses: azure/setup-helm@v4
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4

- name: Install kubectl
uses: azure/setup-kubectl@v4
uses: azure/setup-kubectl@776406bce94f63e41d621b960d78ee25c8b76ede # v4

- name: Configure kubeconfig
env:
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,16 @@ jobs:
e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: go.work
cache-dependency-path: "**/*.sum"
- name: Setup Node
uses: actions/setup-node@v4
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 22
cache: npm
Expand All @@ -48,7 +48,7 @@ jobs:
run: cd HadesAPI/web && npm run test:e2e
- name: Upload report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: playwright-report
path: HadesAPI/web/playwright-report/
Expand Down
12 changes: 8 additions & 4 deletions .github/workflows/release-chart.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false

- name: Install Helm
uses: azure/setup-helm@v4
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4

- name: Lint and template
run: |
Expand All @@ -52,10 +54,12 @@ jobs:
packages: write # push the chart to GHCR
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false

- name: Install Helm
uses: azure/setup-helm@v4
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4

- name: Log in to GHCR
run: echo "${{ secrets.GITHUB_TOKEN }}" | helm registry login ghcr.io -u "${{ github.actor }}" --password-stdin
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/verify-crd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,13 @@ jobs:

steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
fetch-depth: 0

- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: go.work
cache: true
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/verify-openapi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,13 @@ jobs:

steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
fetch-depth: 0
persist-credentials: false

- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: go.work
cache: true
Expand Down
8 changes: 6 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Go workspace (`go.work`, Go 1.27) with five modules:

| Module | Binary / role |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `HadesAPI/` | Gin HTTP server. `POST /build` validates a payload, assigns a UUID, publishes to NATS by priority (and now also publishes `hades.jobstatus.Queued`). `GET /ping` health check. Optional Basic Auth via `AUTH_KEY`. Also hosts the optional **dashboard** (`HadesAPI/dashboard/` + embedded SPA in `HadesAPI/web/`). |
| `HadesAPI/` | Gin HTTP server. `POST /build` validates a payload, assigns a UUID, publishes to NATS by priority (and now also publishes `hades.jobstatus.Queued`). `GET /ping` health check. Basic Auth via `AUTH_KEY` is required (exits at start when empty unless `ALLOW_UNAUTHENTICATED=true`, dev only); `/build` body capped at 1 MiB, duplicate step ids rejected; Swagger (`DEBUG=true`) sits behind the same auth. Also hosts the optional **dashboard** (`HadesAPI/dashboard/` + embedded SPA in `HadesAPI/web/`). |
| `HadesScheduler/` | NATS consumer. Reads `HADES_EXECUTOR` and dispatches to either the `docker/` or `k8s/` package. |
| `HadesScheduler/docker/` | Runs each step as a Docker container; shares state between steps via a per-job named volume `shared-<uuid>`. |
| `HadesScheduler/k8s/` | Creates a `BuildJob` CR via the dynamic client for the operator to reconcile. |
Expand All @@ -26,10 +26,11 @@ Go workspace (`go.work`, Go 1.27) with five modules:
- **Status subjects:** `hades.jobstatus.{Queued,Running,Succeeded,Failed,Stopped}` (payload = job ID string, optional `X-Hades-Reason` header). `HadesAPI` publishes `Queued` on enqueue; the scheduler/operator publish `Running`/`Succeeded`/`Failed`. Nothing publishes `Stopped` today. `HadesLogManager` and the dashboard subscribe to the whole `hades.jobstatus.*` lifecycle over **core NATS**.
- **Status stream:** `HADES_JOB_STATUS` is a JetStream stream over `hades.jobstatus.*` that durably captures those same core-NATS publishes. `HadesLogManager` consumes it through the durable `HADES_STATUS_WEBHOOK` consumer to deliver the **job-status webhook** (`status_callback_url`). Publishers are unchanged and core subscribers still receive every event.
- **Log subjects:** see `shared/buildlogs/buildlog_stream.go`.
- **Callbacks:** `callback_url` / `status_callback_url` targets on loopback, private, CGNAT, link-local, unspecified and multicast addresses are refused at connect time (SSRF guard in `HadesLogManager/http_client.go`); `CALLBACK_ALLOWED_HOSTS` exempts internal receivers. No redirects on the log callback, proxy env ignored, a blocked status webhook is not retried.
- **Job leases:** the scheduler acks a job only after the executor returns, and the Docker executor blocks for the whole job. `shared/nats/consumer.go` therefore heartbeats `msg.InProgress()` on a ticker while a job runs so the `NATS_ACK_WAIT` (default 1m) timer keeps resetting - job duration must never determine AckWait. `NATS_MAX_DELIVER` (default 3) caps redeliveries; the last delivery is not executed but publishes a terminal `Failed` status and terminates the message.
- **Job DTO:** `shared/payload/payload.go` (`QueuePayload` with ordered `Step`s). Job-level: `timeout_seconds` (whole-job timeout). Each step has `image`, optional `script`, `metadata` env vars, `cpu_limit` (whole cores), `memory_limit`, and the Docker-only limits `network`, `memory_swap`, `pids_limit`.
- **Two outbound pushes, do not conflate them:** `callback_url` forwards the aggregated **log array** and only after `stopWatchingJobLogs` drains the JetStream log consumer; `status_callback_url` receives the **job-status webhook** (`{event, job_id, name, status, reason, queued_at, started_at, finished_at, duration_ms, attempt}`) on the terminal status itself, with at-least-once retry. Neither may be repurposed as the other.
- **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only**; the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field).
- **Executor parity:** `timeout_seconds`, env (`metadata`), `cpu_limit`, `memory_limit` are enforced on both executors (Kubernetes: per-container resources + Job `activeDeadlineSeconds`). `network`, `memory_swap`, `pids_limit` are enforced on the **Docker executor only** (`network` must be in `DOCKER_ALLOWED_NETWORKS`, default `none,bridge,default`, else the job fails; `HADES_DEFAULT_JOB_TIMEOUT_SECONDS` / `HADES_MAX_JOB_TIMEOUT_SECONDS` also Docker only); the operator accepts them for schema parity but does not apply them (pod containers share a network namespace; no per-container swap/PID field).
- **CRD:** `BuildJob` in `HadesScheduler/HadesOperator/api/v1/buildjob_types.go`. **Important:** if you change `BuildJobSpec`, run `make -C HadesScheduler/HadesOperator manifests generate` and commit `helm/hades/crds/build.hades.tum.de_buildjobs.yaml` and `zz_generated.deepcopy.go` - the `verify-crd.yml` GitHub workflow will fail otherwise. The `BuildJobSpec` is intentionally duplicated from `shared/payload`; keep them in sync manually.

## Build / test
Expand Down Expand Up @@ -81,6 +82,9 @@ The top-level `Makefile` wraps the common workflows (`make help` lists every tar
- Don't add fields to `BuildJobSpec` without regenerating the CRD (CI will block the PR).
- Don't introduce package-level mutable globals for dependencies - the codebase has been moving away from them (see `setupRouter` taking `JobPublisher` as a parameter).
- `HadesLogManager` is deployed (Helm + image) but ClusterIP-internal and single-replica (in-memory aggregation); don't put it on the ingress or scale it past one replica. The status-webhook dispatcher living there keeps no durable state in memory (its retry schedule is in JetStream), but it still assumes one replica. The same single-replica constraint applies to the API's in-memory dashboard read-model. (It just isn't in the `compose.yml` stack.)
- NATS itself still has no authentication; it must not be published beyond loopback (compose binds it to `127.0.0.1`, the deploy stack does not publish it). Never list the NATS network in `DOCKER_ALLOWED_NETWORKS`.
- Don't re-open callbacks to internal addresses or add a redirect-following/proxy-aware callback client; use `CALLBACK_ALLOWED_HOSTS`.
- Job pods must keep no service account token, `RuntimeDefault` seccomp and `allowPrivilegeEscalation: false`; the operator role gets no secrets/configmaps access.
- The dashboard treats all job/step `Metadata` and step `script` bodies as potentially secret: redact via `shared/redact` before returning any payload; never add an endpoint that returns raw metadata or an unscrubbed script. Job logs are proxied verbatim (a known, documented gap).


Expand Down
Loading
Loading