Skip to content

deployment.md

Latest commit

 

History

History
1057 lines (819 loc) · 44.5 KB

deployment.md

File metadata and controls

1057 lines (819 loc) · 44.5 KB
Raw

Deployment Guide (Linux)

Mortred targets Linux only. This is the complete operations manual behind the Quick Start: architecture, decision guide, step-by-step walkthroughs of all three install tracks, the profile system, weights and engine management, security, upgrades/rollback, monitoring and troubleshooting.

Unsupported boundaries (no false expectations): unsupported-boundaries.md.

First-hour lane: pick one install entry below (§4–§7), then mortredctl next repeatedly (tokens → loopback listen → start → GPU pack prepare/calibrate if needed → doctor --strict → one infer). It prints exactly one next: command each time.

After reading this you can: bring Mortred up on a clean Ubuntu machine in 20 minutes and pass the acceptance gates (§16).

Contents

1. Architecture

A Mortred deployment is one control plane plus a set of model processes, all on the same machine (or inside the same container):

flowchart LR
    subgraph Clients["External clients"]
        C["SDK / curl / browser"]
    end

    subgraph ControlPlane["Control plane (the only two exposed ports)"]
        GW["mortred-gateway :8080<br/>inference entry · auth · rate limit · routing"]
        SUP["mortred-supervisor :8787<br/>process mgmt · web console · mgmt API"]
    end

    subgraph DataPlane["Data plane (loopback only, unreachable externally)"]
        M1["mobilenetv2_server :9002"]
        M2["yolov8_server :9056"]
        M3["...more model servers"]
    end

    C -->|"POST /v1/models/{id}/infer"| GW
    GW -->|"internal token"| M1
    GW --> M2
    GW --> M3
    SUP -.->|"spawn/restart/probe"| M1
    SUP -.-> M2
    SUP -.-> M3
    C -->|"Web console / mgmt API"| SUP
Loading

Key design decisions:

Decision Meaning
Only 2 external ports gateway 8080 (inference), supervisor 8787 (management + web UI)
Model servers bind loopback only Nothing bypasses the gateway; supervisor injects an internal token
The supervisor manages a pack Listed conf/packs/*.toml ids autostart; MORTRED_AUTOSTART=true does not boot the whole conf/server tree
Fail-closed Missing inference/management auth or scrape token refuses to start, including loopback. Wildcard bind needs MORTRED_EXPOSE=docker or unsafe. TLS is Nginx (mortredctl init-edge).

Ports at a glance:

Port Process Purpose Auth
8080 mortred-gateway recommended inference POST /v1/models/{id}/infer; legacy {server_uri} (e.g. /mortred_ai_server_v1/...) still accepted; /healthz, /metrics Bearer on infer/jobs; /healthz public; /metrics requires MORTRED_METRICS_TOKEN (including loopback)
8787 mortred-supervisor mgmt API /api/v1/*, web console Bearer token; /api/v1/health and /api/v1/version are public
9002+ model servers loopback only internal token (including GET /metrics)

2. Five-Minute Decision Guide

2.1 First, the profile: cpu or gpu?

flowchart TD
    A["NVIDIA GPU present?"] -->|"nvidia-smi -L succeeds"| GPU["gpu profile"]
    A -->|"no GPU / unsure"| CPU["cpu profile"]
    GPU --> G1["full model zoo<br/>MNN-CUDA / ORT-CUDA / TensorRT"]
    CPU --> C1["curated 2 models<br/>MNN-CPU, no TensorRT"]
Loading
gpu (default) cpu
Backends MNN-CUDA / ORT-CUDA / TensorRT MNN-CPU / ORT-CPU (TensorRT compiled out)
Hardware NVIDIA GPU + driver, CUDA 12 + TensorRT 10 any x64 machine
Models HTTP catalog (classification / detection / OCR (dbnet text detection only, no recognition) / seg / matting / enhancement / SuperPoint / depth / DINOv2 / SAM AMG / diffusion). Bench-only: CLIP, LightGlue, FastSAM, SAM prompt. No MOT. RT-DETR is unimplemented. curated: mobilenetv2, resnet
Weight size full manifest (tens of GB) curated subset (~1 GB)
Engine conversion pack engines on this GPU (§10.2); zoo-wide convert is opt-in not needed

Unsure? Pick cpu. The worst case of a wrong choice is redoing with the other profile; the data-plane configs are fully compatible.

2.2 Then, the track: Docker or tarball?

Docker track Tarball track
For Docker shops; fastest path to a running service bare-metal prod; no Docker; native systemd
Artifact dual images (ghcr.io/...:vX.Y.Z-cpu/-gpu) self-contained tarball + install.sh + systemd unit
Upgrade swap image tag mortredctl upgrade (in place, conf backed up)
Isolation container-level apt deps installed by the installer
Shared same verify_deployment.sh acceptance, same profile system, same mortredctl core

All three entries (bootstrap / compose / tarball) share one mortredctl core and converge on mortredctl doctor - there are no divergent paths.

3. Prerequisites

3.1 Hardware

Profile Minimum Recommended
cpu 2 cores / 4 GB / 10 GB disk 8 cores / 16 GB / SSD
gpu the above + a CUDA 12 GPU RTX 3060+ / 8 GB VRAM / 50 GB disk

3.2 OS and software

Item Requirement Check
OS Ubuntu 20.04 / 22.04 (x64) lsb_release -rs
curl any recent version curl --version
python3 ≥ 3.8 (weights only) python3 --version
docker + compose Docker track only docker compose version
sudo tarball installer only -
NVIDIA driver gpu profile only nvidia-smi

3.3 Network

  • Install time: access to GitHub Releases / Hugging Face (weights). Offline: §9.4.
  • Runtime: compose/docker run examples bind 8080/8787 to 127.0.0.1 on the host. LAN/WAN exposure is your reverse proxy's decision (TLS belongs there). Publishing those ports on 0.0.0.0 without a proxy sends Bearer tokens in the clear.

4. Entry 1: One-Line Bootstrap

The fastest path - hardware detection, track selection, straight through:

curl -fsSL https://raw.githubusercontent.com/MaybeShewill-CV/mortred_model_server/main/scripts/bootstrap.sh | bash

What it does:

  1. Probes nvidia-smi -L → recommends gpu or cpu;
  2. Docker present → prints the compose instructions (§5);
  3. No Docker → resolves the latest GitHub release tag, downloads mortred_model_server-<version>-<profile>-linux-x64.tar.gz (there is no ...-latest-... tarball filename), verifies the sibling .sha256 when published, then sudo ./install.sh (§6);
  4. No Release yet, or that profile asset is missing → prints [WARN] no versioned release tarball published yet for profile … and the source-build path (§7). This is a documented track, not a hidden failure.

Expected output (no GPU, Docker present):

== Mortred bootstrap ==
  detected profile: cpu
== docker track ==
next:
  1. git clone https://github.com/MaybeShewill-CV/mortred_model_server.git && cd mortred_model_server
  2. python3 scripts/fetch_weights.py --profile cpu
  3. ./scripts/mortredctl_init-trust.sh && set -a && . conf/local/trust.env && set +a
  4. docker compose --profile cpu up -d
  5. curl -fs http://localhost:8787/api/v1/health

Expected output (no GPU, no Docker, no GitHub Release yet):

== Mortred bootstrap ==
  detected profile: cpu
  [WARN] no versioned release tarball published yet for profile cpu
  [WARN] (GitHub latest tag missing or that profile asset is unpublished; there is no ...-latest-... tarball name)
== manual track ==
  1. git clone https://github.com/MaybeShewill-CV/mortred_model_server.git && cd mortred_model_server
  2. ./scripts/install_deps.sh --cpu --all
  3. cmake --preset full-cpu && cmake --build --preset full-cpu
  4. mortredctl init --profile cpu

The bootstrap stays deliberately thin: detection and delegation only. Upgrading mortredctl upgrades every entry point.

5. Entry 2: Docker Compose

5.1 Install (four steps)

# 1. get the code (compose file + weight scripts live in the repo)
git clone https://github.com/MaybeShewill-CV/mortred_model_server.git
cd mortred_model_server

# 2. fetch the weight subset for your profile (resumable + sha256 verified)
python3 scripts/fetch_weights.py --profile cpu     # gpu machines: gpu

# 3. three distinct tokens (compose interpolates all three; missing scrape fails closed)
./scripts/mortredctl_init-trust.sh
set -a && . conf/local/trust.env && set +a

# 4. start (builds locally; first build ~10-25 min)
docker compose --profile cpu up -d      # GPU machines: --profile gpu

Compose publishes 8080/8787 on 127.0.0.1 of the host. localhost clients keep working; other machines cannot reach those ports until you put a TLS proxy in front or override the port mappings.

The gpu track needs the NVIDIA Container Toolkit (docker run --gpus all works = installed).

5.2 Verify

curl -fs http://localhost:8787/api/v1/health        # supervisor health
curl -fs http://localhost:8787/api/v1/version       # supervisor version badge (public, used by the web console)
curl -fs http://localhost:8080/healthz               # gateway health (public)
curl -fs -H "Authorization: Bearer $MORTRED_METRICS_TOKEN" \
    http://localhost:8080/metrics | head -5          # gateway metrics

curl -fs -H "Authorization: Bearer $MORTRED_API_TOKEN" \
    http://localhost:8787/api/v1/catalog | python3 -m json.tool | head -20

Expected: /api/v1/health returns OK; the catalog lists models for the current profile and pack. With the shipped demo pack under cpu, expect supervisor + gateway only (no shipped *_cpu product entries; self-build a cpu pack per §10.7 — demo is a TRT seed).

5.3 Day-2 operations

Operation Command
Logs docker compose --profile cpu logs -f mortred-cpu
Restart docker compose --profile cpu restart
Stop docker compose --profile cpu down
Upgrade image docker compose --profile cpu pull && docker compose --profile cpu up -d
Shell into container docker exec -it mortred-cpu bash
GPU pack engines mortredctl prepare (§10.2); zoo-wide MORTRED_AUTO_BUILD_ENGINES stays opt-in

5.4 Prebuilt images (no local build)

docker pull ghcr.io/maybeshewill-cv/mortred_model_server:v0.1.0-cpu
docker run -d --name mortred \
  -p 127.0.0.1:8787:8787 -p 127.0.0.1:8080:8080 \
  -v "$PWD/weights:/opt/mortred/weights" \
  -e MORTRED_API_TOKEN=... -e MORTRED_GATEWAY_AUTH_TOKEN=... \
  -e MORTRED_METRICS_TOKEN=... \
  ghcr.io/maybeshewill-cv/mortred_model_server:v0.1.0-cpu

# GPU runtime image (needs NVIDIA Container Toolkit):
docker pull ghcr.io/maybeshewill-cv/mortred_model_server:v0.1.0-gpu
docker run -d --name mortred --gpus all \
  -p 127.0.0.1:8787:8787 -p 127.0.0.1:8080:8080 \
  -v "$PWD/weights:/opt/mortred/weights" \
  -e MORTRED_API_TOKEN=... -e MORTRED_GATEWAY_AUTH_TOKEN=... \
  -e MORTRED_METRICS_TOKEN=... \
  ghcr.io/maybeshewill-cv/mortred_model_server:v0.1.0-gpu

6. Entry 3: Tarball + systemd

For bare-metal production: no Docker dependency, native systemd, self-healing restarts.

6.1 Download and verify

From Releases (example: v0.1.0 / cpu):

VER=0.1.0
curl -fLO https://github.com/MaybeShewill-CV/mortred_model_server/releases/download/v$VER/mortred_model_server-$VER-cpu-linux-x64.tar.gz
curl -fLO https://github.com/MaybeShewill-CV/mortred_model_server/releases/download/v$VER/mortred_model_server-$VER-cpu-linux-x64.tar.gz.sha256
sha256sum -c mortred_model_server-$VER-cpu-linux-x64.tar.gz.sha256   # must print OK

Tarball contents: opt/mortred/ (installed tree) + deploy/mortred-supervisor.service

  • install.sh + a PROFILE marker. Weights are NOT bundled (tens of GB) - fetch them per §9 after installing.

6.2 Install (root)

mkdir unpack
tar -xzf mortred_model_server-$VER-cpu-linux-x64.tar.gz -C unpack
cd unpack   # flat archive: install.sh, opt/mortred/, deploy/ at the root
sudo ./install.sh

What install.sh does, step by step (idempotent, safe to re-run):

Step Content
1 apt runtime deps (glog / OpenCV / openssl; gpu adds TensorRT/cuDNN runtime)
2 install tree to /opt/mortred; create the mortred system user
3 install + enable the systemd unit (cpu profile injects MORTRED_PROFILE=cpu)
4 generate /etc/mortred/supervisor.env (mode 600) and print next steps

6.3 Tokens and weights

# generate three tokens (overwrites the comment-only placeholder from install.sh)
sudo /opt/mortred/bin/mortredctl.out init-trust --force --out /etc/mortred/supervisor.env
# or sudoedit the three MORTRED_*_TOKEN lines yourself

cd /opt/mortred
sudo -u mortred python3 scripts/fetch_weights.py --profile cpu

6.4 Start and verify

sudo systemctl start mortred-supervisor
sudo systemctl status mortred-supervisor --no-pager    # active (running)
curl -fs http://127.0.0.1:8787/api/v1/health

Unit highlights: Restart=always, TimeoutStopSec=120 (ordered shutdown - models first, gateway last), EnvironmentFile=/etc/mortred/supervisor.env (600).

7. Building from Source

For contributors and custom builds.

7.1 Dependencies (version matrix + sha256 pinned + idempotent stamps)

./scripts/install_deps.sh --all            # gpu line (CUDA 12 / TensorRT 10.3)
./scripts/install_deps.sh --cpu --all      # cpu line: MNN-CPU + ORT-CPU, no NVIDIA/TRT
sudo ./scripts/install_deps.sh --nvidia    # gpu line CUDA/TRT/cuDNN (root; nothing else needs it)
./scripts/install_deps.sh --check          # gpu: verify 3rd_party before cmake
./scripts/install_deps.sh --cpu --check    # cpu: same

Offline: --offline DIR uses a pre-downloaded package dir. ORT tarballs are sha256-verified fail-closed (a missing hash refuses the install).

CMake configure is fail-closed on the same footguns --check covers for a source build: missing libworkflow / libcrypto, ORT_API_VERSION ≠ pin for libonnxruntime.so.1.29.0, leftover libonnxruntime.so.1.18* (and on gpu, leftover TensorRT 8). The FATAL text prints the matching install_deps fix.

7.2 Build (presets carry the profile)

cmake --preset full && cmake --build --preset full            # gpu full
cmake --preset full-cpu && cmake --build --preset full-cpu    # cpu full
cmake --preset tests-only && cmake --build --preset tests-only
# buildPreset default target is `check` (builds EXCLUDE_FROM_ALL tests + runs ctest)
Preset Purpose
tests-only / tests-only-werror unit tests; build preset defaults to check
tests-only-tsan / tests-only-asan sanitizer gates (§16)
full / full-werror gpu full
full-cpu cpu full (no CUDA/TRT)

7.3 Pack a tarball yourself

./scripts/make_release_tarball.sh cpu 0.1.0 build    # -> dist/*.tar.gz + .sha256
# archive is flat (install.sh / opt/mortred / deploy at the root); unpack into an empty dir

8. The Profile System

One switch, four layers - profiles are not two products but two resource tiers of one product:

Layer Switch cpu effect gpu effect
Build MORTRED_BUILD_PROFILE TRT compiled out; factory errors clearly for type="tensorrt" full build
Deps install_deps.sh --cpu MNN built MNN_CUDA=OFF; cpu ORT tarball; NVIDIA deb skipped + CUDA/TRT/cuDNN
Catalog server TOML profile field + runtime MORTRED_PROFILE only profile="cpu"/"any" entries; absent field = gpu, so the cpu catalog is always explicitly curated everything
Weights fetch_weights.py --profile only files tagged profiles=["cpu","gpu"] full manifest

8.1 Runtime switching

export MORTRED_PROFILE=cpu     # read by both supervisor and gateway; default gpu

Filtering happens during catalog load, before the duplicate checks - cpu and gpu variants of one model may therefore reuse the same port (only one variant set is active at a time).

8.2 Extending the cpu curated set (a CHANGELOG-level change)

  1. point a conf/server/... file with profile="cpu" at the same model toml (do not add a second *_cpu_config.toml);
  2. that toml must use mnn/onnx — type=tensorrt with device=cpu is a configuration error;
  3. on a CPU box set device = "cpu" in that one file (git defaults are gpu);
  4. add the weight path to CPU_WEIGHTS in scripts/gen_weights_manifest.py;
  5. regenerate the manifest;
  6. add a cpu smoke verification for the model; record it in CHANGELOG.

For a TensorRT product there is no shared mnn toml to point at — serve it on cpu with a machine-pack model_config override instead (§10.7).

The curated set is deliberately frozen per release: extending it is a release decision (performance + acceptance ownership), not a config tweak.

9. Weights Management

9.1 Mechanism

Hugging Face is the ONNX interchange repo (MaybeShewill-CV/mortred_model_server). Product configs keep type=mnn or type=tensorrt except already-ONNX ids (diffusion, MSOCRNET, SAM decoders) and LIBFACE after the YuNet 2026may cutover. GPU engines are built on this machine (mortredctl prepare / convert_trt_engines.sh). CPU .mnn files stay the serving format until a local convert path exists; do not switch product tomls to ONNX for performance.

Per-id source, IO contract, and license notes: conf/onnx_sources.json. Bytes and sha256: conf/weights_manifest.json. Dual-file export rules and the remaining-gap list: onnx-interchange.md.

  • Manifest: conf/weights_manifest.json - per file path / size / sha256 / hf_path / profiles / kind;
  • Downloads: Hugging Face, resumable, skipped when present with matching sha256;
  • Product interchange is python3 scripts/fetch_weights.py --kind onnx (39 static_bs1 graphs). .engine files are skipped unless --kind engine; build them on this GPU with prepare.
  • Verification: --check verifies without downloading.

9.2 Commands

python3 scripts/fetch_weights.py --kind onnx       # 39 product static_bs1 ONNX graphs
python3 scripts/fetch_weights.py --profile cpu     # curated runtime subset (~1 GB)
python3 scripts/fetch_weights.py --profile gpu     # runtime weights; .engine files are skipped
python3 scripts/fetch_weights.py --kind engine     # explicit engine download; not the out-of-box path
python3 scripts/fetch_weights.py --only yolov8     # paths containing yolov8
python3 scripts/fetch_weights.py --check           # verify local integrity
python3 scripts/fetch_weights.py --dry-run         # print what would happen

9.3 Disk planning

Profile First download Reserve
cpu ~1 GB 5 GB
gpu tens of GB (model-dependent) 60 GB+

Partial gpu install? Pull in batches with --only <keyword> and confirm with verify_deployment.sh --full.

9.4 Offline environments

Fetch weights/ on a networked machine → copy to the target → run fetch_weights.py --check. Dependencies work the same way (--offline DIR).

10. Machine pack and TensorRT

Compose and the container entrypoint set MORTRED_AUTOSTART=true and MORTRED_PACK (default conf/packs/demo.toml). That combination starts the listed catalog ids, not every file under conf/server/. Identity is still one process per catalog id; conf/server worker_nums stays 1. Pack worker_nums / model_config override the child via env.

GPU pack chain (copy → prepare → calibrate --write-pack → doctor → start):

cp conf/packs/gpu_yolo_metric3d.toml /tmp/machine-pack.toml
mortredctl prepare --pack /tmp/machine-pack.toml
mortredctl calibrate --pack /tmp/machine-pack.toml --write-pack
export MORTRED_PACK=/tmp/machine-pack.toml
mortredctl doctor
# then start the supervisor / compose stack with that MORTRED_PACK

Git TensorRT packs ship seed occupancy (gpu_mem_source="seed"). Per-id files conf/packs/<id>.toml cover all 39 product ids; yolov8.toml, metric3d.toml, and gpu_yolo_metric3d.toml remain short multi-id examples. Seed MiB is a reusable starting sample only — it must be overwritten on each host by calibrate --write-pack. Do not commit calibrated worker_nums, non-seed gpu_mem_mib, or [pack] gpu_name / gpu_memory_total_mib. The demo pack is the seeded TensorRT MobilenetV2 product: recalibrate it on every host like any TRT pack, and on cpu-only machines it autostarts nothing (§10.7). --write-pack refuses to write conf/packs/ directly; always copy first.

Per-id product packs

conf/product_static_bs1.json lists one catalog id per static batch=1 TensorRT engine (39 ids). Each id has a generated seed pack at conf/packs/<id>.toml: exactly one [pack.<id>] section with worker_nums = 1, gpu_mem_mib = 1800, gpu_mem_at_workers = 1, gpu_mem_source = "seed", and that id's model_config. Regenerate after the product table changes:

python3 scripts/product_static_bs1.py --emit-packs conf/packs
python3 scripts/product_static_bs1.py --check

--check fails when an id has no pack, the file drifted from the generator, the section id does not match the filename, or a conf/packs/*.toml file is neither <id>.toml for a product id nor one of the legacy multi-id examples demo.toml, yolov8.toml, metric3d.toml, and gpu_yolo_metric3d.toml. Those four stay as short examples. Product coverage is the per-id set.

mortredctl prepare, scripts/prepare_pack.sh, scripts/pack_trt.py, and MORTRED_PACK each take one pack file. To co-start several ids, merge the per-id files into a machine-local pack, then use that path for the rest of the chain. scripts/compose_packs.py copies each [pack.<id>] in argument order, merges a [pack] table when the assignment text matches, and rejects a repeated id. It refuses to write conf/packs/ or conf/server/.

python3 scripts/compose_packs.py \
  conf/packs/yolov8s.toml conf/packs/libface.toml \
  -o /tmp/yolov8s-libface.toml
mortredctl prepare --pack /tmp/yolov8s-libface.toml
mortredctl calibrate --pack /tmp/yolov8s-libface.toml --write-pack
export MORTRED_PACK=/tmp/yolov8s-libface.toml
mortredctl doctor

A single id uses the same chain with cp conf/packs/<id>.toml /tmp/<id>.toml (or compose of that one file). Prepare converts only the ids listed in the pack. One id's pack does not build the other engines. Co-starting every id at once is a host-memory choice; compose the subset that should be resident.

10.1 Pack file

# conf/packs/demo.toml — shipped demo: seeded TRT MobilenetV2
# (keep worker_nums=1 and the seed stamp in git; recalibrate on every host)
[pack.mobilenetv2]
worker_nums = 1
model_config = "conf/model/classification/mobilenetv2/mobilenetv2.toml"
gpu_mem_mib = 1800
gpu_mem_at_workers = 1
gpu_mem_source = "seed"

# machine-local copy after calibrate --write-pack, e.g. /tmp/machine-pack.toml
[pack.yolov8s]
worker_nums = 4
gpu_mem_mib = 2100
gpu_mem_at_workers = 4
gpu_mem_source = "nvml_pid"
# model_config = "conf/model/object_detection/yolov8/yolov8s.toml"  # optional variant

Unknown ids fail supervisor start. Point MORTRED_PACK at the machine file (compose env, systemd supervisor.env, or the process environment).

10.2 Prepare pack TensorRT engines (GPU)

Engines are bound to this GPU + this TensorRT. Convert only what the pack uses (step 2 of the chain above):

mortredctl prepare --pack /tmp/machine-pack.toml       # or: scripts/prepare_pack.sh

The supervisor refuses to spawn a TensorRT id whose engine file is missing or empty (status failed, no crash-loop). /ready is real loadability, not just a nonempty file. After prepare, TensorRT ids still need a host occupancy stamp (§10.3) before spawn — seed in git is not enough.

Every TensorRT pack — including the demo's MobilenetV2 — needs prepare on the target GPU. MORTRED_AUTO_BUILD_ENGINES=true still converts the whole zoo and stays off by default.

10.3 Calibrate worker_nums and occupancy stamps

Stop supervisor / leftover mortred-model-server first. The script starts its own server on the catalog port (YOLOV8 = 9056); a busy port is start_failed, not OOM. Copy the git example pack to a machine-local file before --write-pack (step 1), then calibrate (step 3):

ss -ltnp | grep 9056 || true
python3 scripts/calibrate_pack.py --pack /tmp/machine-pack.toml \
    --output logs/calibrate-yolov8.json --write-pack

The default duration is 30s. A shorter --duration sets release_measurement=false and --write-pack does not persist. Omit --workers to use the per-id grid: Throughput ≥ 500 qps in docs/benchmark-h3/<id>.txt sweeps 1,2,4,8,16; every other id sweeps 1,2,4,8. [pack.<id>] worker_sweep overrides that grid; --workers overrides both. Each point runs at concurrency equal to that worker count. The sweep stops when a point fails to start, OOMs, has a non-429 error rate or a 429 rate of at least 1%, exceeds [pack.<id>] p99_ms_cap (or the built-in seed cap), cannot read queue and inference histograms, has queue p50 above inference p50, or gains under 10% RPS. Every id in conf/product_static_bs1.json has a seed in scripts/calibrate_pack.py P99_CAP_MS. An id with neither that seed nor [pack.<id>] p99_ms_cap aborts before the sweep and prints a missing-cap table; the latency gate is not skipped. An id that fails at worker_nums=1 stays eligible=false and is not written. With two or more eligible ids, the joint pass only measures memory. The budget is [pack] gpu_memory_budget_mib, or 85% of the card. Over budget or OOM demotes the largest id one grid step onto a worker count that already passed; a busy port does not demote. --write-pack stores that final_worker_nums.

JSON gpu_mem_mib_* is process occupancy (nvml_pid / nvml_name) or a pre-spawn device delta on WSL — not whole-card memory.used. --write-pack overwrites seed (or any prior stamp) with gpu_mem_mib, gpu_mem_source, gpu_mem_at_workers, and [pack] gpu_name / gpu_memory_total_mib on the machine copy only.

The supervisor refuses to spawn a TensorRT id in an active pack that has no gpu_mem_mib stamp, or whose worker_nums does not match gpu_mem_at_workers (status failed, no crash-loop). The error names the next command: mortredctl calibrate --pack <pack> --write-pack. Joint budget uses running siblings' stamps plus this stamp versus gpu_memory_total_mib × (1 − gpu_reserve_pct/100) when the total is present. occupancy_policy=off or MORTRED_OCCUPANCY_ENFORCE=0 skips the spawn gate (unsafe, same class as MORTRED_EXPOSE=unsafe); mortredctl doctor --strict still fails. Invalid occupancy_policy refuses pack apply.

10.3.1 Doctor then start

After calibrate, run doctor (step 4) then start with the same pack (step 5):

export MORTRED_PACK=/tmp/machine-pack.toml
mortredctl doctor          # or: mortredctl doctor --strict
# start supervisor / compose so MORTRED_PACK injects worker_nums

10.4 Zoo-wide convert (optional)

./scripts/convert_trt_engines.sh --list
./scripts/convert_trt_engines.sh            # every missing engine in the manifest
./scripts/convert_trt_engines.sh --force

Needs trtexec (sudo ./scripts/install_deps.sh --nvidia → 3rd_party/bin/).

10.5 First-start auto-conversion in containers (optional)

docker compose --profile gpu up -d -e MORTRED_AUTO_BUILD_ENGINES=true

Runs before supervisor autostart, minutes-long, off by default. Prefer §10.2 for a pack. mortredctl doctor warns on missing pack engines or occupancy stamps; --strict fails.

10.6 ONNX Runtime CUDA arena vs pack occupancy

ORT CUDA used to set gpu_mem_limit = 0 (arena grows without bound). The default is now 2048 MiB per session (gpu_mem_limit_mb on [MODEL.backend], or MORTRED_ORT_GPU_MEM_LIMIT_MB). worker_nums=4 means up to four arenas. 0 restores unlimited. MNN and TensorRT have no equivalent knob. Process occupancy is gated at supervisor spawn from the pack stamp (§10.3), not by wiring gpu_mem_limit_mb into TRT/MNN sessions.

10.7 Running CPU-only (no default cpu product)

The repo deliberately ships no default cpu product configs: there is no conf/model/**/*_cpu.toml and no conf/server/**/*_cpu_config.toml (the consistency gate fails if one reappears). CPU usage is assembled per machine from the building blocks above:

  1. Build / install the cpu tier — install_deps.sh --cpu, CMake preset full-cpu, or docker compose --profile cpu build.

  2. Write a cpu model toml for the model you want — type = "mnn" or type = "onnx" with device = "cpu". The canonical sample is conf/ci/mobilenetv2_hosted.toml (kept honest by the hosted golden gate). A type="tensorrt" toml on cpu fails init with a clear error by design.

  3. Write a machine pack that selects the catalog id and points model_config at your cpu toml (cpu backends need no occupancy stamp):

    # /tmp/cpu-pack.toml
[pack.mobilenetv2]
worker_nums = 1
model_config = "/tmp/mobilenetv2_cpu.toml"

  4. Fetch the cpu-tagged weights (profiles=["cpu","gpu"] rows in conf/weights_manifest.json): python3 scripts/fetch_weights.py --profile cpu.

  5. Start with the cpu profile — export MORTRED_PROFILE=cpu and export MORTRED_PACK=/tmp/cpu-pack.toml, then start the supervisor (compose with the pack mounted, or the tarball / systemd entry).

The shipped conf/packs/demo.toml is the seeded TensorRT product: its MobilenetV2 id is not in the MORTRED_PROFILE=cpu catalog, so a cpu compose boot with the default pack starts supervisor + gateway only. That is the intended "no default cpu product" behavior, not a failure.

10.8 HTTP RPS evidence (PF1 / TP1)

Qualified HTTP RPS is one id at a time: prepare → calibrate (--raw) → deploy that pack alone → scripts/server/http_infer_rps.py --raw at concurrency worker_nums×2 for at least 30s. The driver records 60s.

The nine coverage-set ids (the ops / TP2 subset) are docs/perf/coverage_set_http_rps.md. The full product catalog is the 39 ids in conf/product_static_bs1.json. Their checklist is docs/perf/product_static_http_rps.md. That file is a skeleton until a GPU host fills it. It does not invent RPS. --check stays non-zero while any id is blocked / evidence=missing. --check-skeleton only checks headings and that a missing row is explicitly not PASS.

python3 scripts/coverage_http_report.py --check
python3 scripts/coverage_http_report.py --catalog product_static --check-skeleton
python3 scripts/coverage_http_report.py --catalog product_static --check
python3 scripts/coverage_http_report.py --catalog product_static --id yolov8n

prepare is mortredctl prepare --id <id>. If conf/packs/<id>.toml exists and lists only that id, the driver copies that seed (including its model_config) to a temp pack, because calibrate --write-pack refuses conf/packs/. Otherwise it writes a one-id [pack.<id>] worker_nums = 1 pack under /tmp. Git packs are not modified, and the run never requires the other ids to be resident. A PASS row is recorded with pf5_retest.record_baseline (docs/perf/baselines/<id>.json, create-once on the same hardware).

11. Authentication & Security

11.1 Tokens

Token Protects Where
MORTRED_API_TOKEN supervisor mgmt API + web console /etc/mortred/supervisor.env (tarball) / container env
MORTRED_GATEWAY_AUTH_TOKEN gateway inference entry same
MORTRED_METRICS_TOKEN gateway GET /metrics scrape Bearer same (required on every listen; distinct from the two above) 
openssl rand -hex 24    # generate (one independent value per token)

Fail-closed semantics: a listener without its token refuses to start and prints why, including on loopback. The gateway also refuses if MORTRED_METRICS_TOKEN is empty or matches the inference/management token. Wildcard bind requires MORTRED_EXPOSE=docker (containers) or unsafe. That gate does not terminate TLS (mortredctl init-edge / Nginx) or reject a short token at process start. mortredctl doctor --strict fails on those warnings (plaintext listen, missing scrape token, short tokens, identical tokens). Default doctor still prints them without failing. Do not reuse the inference token as the scrape secret.

11.2 Multi-tenant API keys (gateway layer)

Beyond the single static token, the gateway supports per-key management (hashed at rest, scopes, rate limits). There is no hot reload: restart the gateway child after editing the file. Mint keys with ./scripts/mint_api_key.sh --name client-a --scope inference --append conf/api_keys.toml (32 random bytes; the plaintext is printed once). Never hash a memorable phrase: verification is unsalted SHA-256, so a leaked file makes a weak key reversible, and mortredctl doctor rejects the documented example secret.

# conf/api_keys.toml
[keys.client-a]
hash = "sha256(<sha256 of a 32-byte random key>)"   # use scripts/mint_api_key.sh
scope = "inference"
rate_limit_qps = 100
enabled = true
curl -X POST -H "Authorization: Bearer $MORTRED_API_TOKEN" \
     http://localhost:8787/api/v1/servers/__gateway/restart

See api-keys.md for the full guide incl. zero-downtime rotation.

11.2.1 Gateway rate-limit order (ST2)

Model routes are gated in this order:

OPTIONS → /healthz → /metrics → route → optional pre-auth abuse limit → authentication → authorized per-source GCRA → per-key queue / inflight → forward.

[gateway.rate_limit] is the authorized infer budget (response layer gateway-gcra). It runs only after Bearer authentication succeeds. A missing or invalid token is 401 and does not admit that bucket, so an unauthenticated burst does not reduce the remaining budget a later valid client from the same address would use. The gateway does not admit that bucket and then release it on 401.

[gateway.abuse_limit] is optional and off unless the section is present and enabled (or shadow = true). It is a separate limiter (layer gateway-abuse) for shedding abuse before authentication. It does not share the authorized infer bucket. MORTRED_RATE_LIMIT=off|shadow|enforce selects only [gateway.rate_limit].

Per-key rate_limit_qps (layer api-key-gcra) is charged only after that key is recognized and allowed to infer. An unknown, disabled, or non-infer key does not take a per-key slot.

# optional; omit the section to leave pre-auth shedding off
[gateway.abuse_limit]
enabled = true
rate_per_sec = 20
burst = 40
trusted_proxies = "127.0.0.1,::1"

11.3 Pre-launch security checklist

  • both tokens are ≥32-char random values, distinct from each other
  • /etc/mortred/supervisor.env is mode 600, owned by mortred
  • 8080/8787 published on 127.0.0.1 unless a TLS reverse proxy sits in front
  • keep 8787 off the public internet even behind TLS
  • conf/api_keys.toml (if used) mode 600, never committed or baked into images
  • TLS terminated at Nginx on the host network (Mortred itself is plain HTTP); see §11.4
  • no model-server ports in the firewall allowlist (they are loopback-only anyway)
  • Grafana/Prometheus ports stay on loopback; Grafana password is not the image default
  • Prometheus does not scrape model ports that have been published off-loopback
  • if 8080 is reachable off-loopback, MORTRED_METRICS_TOKEN is set and distinct from the inference token

11.4 TLS reverse proxy (Nginx)

Mortred does not terminate TLS. First-class edge is Nginx on the host network (mortredctl init-edge). Do not put TLS inside the gateway or supervisor process. Do not run Nginx in a Docker bridge network and expect it to reach 127.0.0.1:8080 on the host.

mortredctl init-trust
set -a && . conf/local/trust.env && set +a
mortredctl init-edge --mode lan --server-name localhost
# optional: sudo cp -a conf/local/edge /etc/mortred/edge
#           sudo cp deploy/nginx/mortred-edge.service /etc/systemd/system/
#           sudo systemctl enable --now mortred-edge
nginx -t -p conf/local/edge -c nginx.conf
# LAN: trust conf/local/edge/tls/ca.pem once in the browser
# Public DNS: mortredctl init-edge --mode acme --server-name infer.example.com
#   then: certbot certonly --webroot -w /var/www/mortred-acme -d infer.example.com
#   (do not use certbot --nginx; it rewrites the site file)

Keep 8080/8787 on 127.0.0.1. Compose profile: edge is Linux network_mode: host only.

mortredctl doctor warns when the effective listen is not loopback, when a token is shorter than 32 characters, when tokens are identical, or when MORTRED_METRICS_TOKEN is unset. Those lines fail doctor --strict. Doctor does not implement TLS.

12. Upgrades & Rollback

12.1 mortredctl upgrade (in place, recommended)

mortredctl upgrade              # latest release, keeps the running profile
mortredctl upgrade v0.2.0       # a specific version

Flow: download the profile's tarball → verify sha256 → back up conf/ to conf.backup-<timestamp> → install over /opt/mortred (weights untouched) → restart → run doctor automatically.

12.2 Rollback

cd /opt/mortred
sudo cp -a conf.backup-<timestamp> conf          # restore config
# reinstall the old tarball (or switch the docker tag back), then:
mortredctl doctor

12.3 Version policy

  • In-place upgrades are supported between adjacent minor versions;
  • Larger jumps: export configs → fresh-install the target → migrate manually (scripts/migrate_model_config.py helps);
  • Config compatibility breaks are recorded per version in CHANGELOG.md.

13. Monitoring

Out-of-the-box Prometheus endpoints:

Endpoint Content
GET :8080/metrics gateway: HTTP counts/latency, inference latency, queue wait, worker availability (MORTRED_METRICS_TOKEN required, including loopback)
GET :8787/api/v1/metrics supervisor: process states, restart counters (Bearer MORTRED_API_TOKEN)

A local monitoring stack ships in the repo (Prometheus + Grafana + alert rules). Ports bind loopback; set a Grafana password before up. Default Prometheus scrape is gateway /metrics only — see monitoring-guide.md.

Prometheus in that compose file runs as uid 65534. The scrape secret file must be mode 600 owned by that uid (or, on bare metal, by user prometheus). write_prometheus_credentials.sh does the chown. Inside the Prometheus container, scrape host.docker.internal:8080 (not localhost:8080 — that is the Prometheus process itself). Linux compose maps the name via extra_hosts.

set -a && . conf/local/trust.env && set +a   # MORTRED_METRICS_TOKEN
./scripts/write_prometheus_credentials.sh    # may prompt sudo to chown 65534:65534
export GRAFANA_ADMIN_PASSWORD="$(openssl rand -hex 16)"
docker compose -f deploy/docker-compose.monitoring.yml up -d
# Grafana: http://localhost:3000
# Alert rules: deploy/alert-rules.yml (includes overload-rejection alerting)
# Bare metal instead: sudo cp deploy/prometheus.yml /etc/prometheus/ &&
#   sudo ./scripts/write_prometheus_credentials.sh /etc/prometheus/mortred_metrics_token

14. Troubleshooting

Run this first - it localizes most problems directly:

mortredctl doctor          # or: verify_deployment.sh --live

14.1 Symptom quick-reference

Symptom Most likely cause Fix
refuses to start, log says so non-loopback listener without token set both tokens (§11.1)
401 with WWW-Authenticate wrong/missing token check Authorization: Bearer ...
empty catalog MORTRED_PROFILE / pack mismatch, or cpu + demo TRT seed check env var and pack; cpu needs a self-built pack (§10.7), not shipped *_cpu products
model server crash-loops missing weights / missing engine / bad config mortredctl status, mortredctl logs <id>; TRT: mortredctl prepare
calibrate Cannot start server / port busy supervisor or leftover model still listening stop systemd/compose/supervisor first, then ss -ltnp on the catalog port (§10.3)
weight download 404/timeout HF unreachable offline flow (§9.4) or a mirror
container has zero engines pack not prepared, auto-build off §10.2; zoo-wide convert is opt-in
sha256 mismatch corrupted download / stale manifest delete the file and refetch; regenerate the manifest
429 responses queue full or key rate-limited tune max_queue_depth / rate_limit_qps; check /metrics
gpu model init: "tensorrt backend is not compiled" cpu build given a trt config use the gpu build/image, or a cpu config for that model

14.2 Log locations

Track Command
Docker docker compose --profile <p> logs -f
systemd journalctl -u mortred-supervisor -f
one model server mortredctl logs <server-id> --limit 200

14.3 Deep dives

Why does the supervisor bind model servers to loopback?

Security-boundary design: external traffic can only reach models through the gateway (auth/rate-limit/audit as a single choke point); supervisor↔model uses an internal token as a second factor. Even a accidentally exposed 9xxx port cannot serve inference without it.

A model stopped working after an upgrade - now what?
  1. mortredctl logs <id> for the model-server error;
  2. diff the model's config against conf.backup-<timestamp>;
  3. check CHANGELOG for a config-incompatibility note on that version;
  4. still stuck → roll back with the config backup + the old tarball (§12.2).

15. FAQ

How do I verify long-task POST /jobs returns 202 immediately?

Follow the step-by-step customer procedure in async-jobs-customer-test.md. POST /v1/models/{id}/jobs must return in milliseconds while the same payload on /infer still takes the full sampling time. Immediate /result must be 409.

Will the cpu profile support more models over time?

Yes, version by version via the formal process in §8.2 - but it stays a curated set: every addition carries CPU-performance and acceptance ownership. See CHANGELOG for the extension history.

Can I run cpu and gpu side by side?

One runtime profile per machine (MORTRED_PROFILE). To serve both load classes, deploy two instances with distinct ports.

Any functional difference between the Docker and tarball tracks?

None. Same binaries, same configs, same acceptance script. Choose by ops habit.

Must weights live in /opt/mortred/weights?

Model configs resolve paths relative to the install tree. The tarball track defaults to /opt/mortred/weights; the Docker track mounts into that container path - the host-side location is up to you.

Where do I set worker_nums?

Git conf/server stays worker_nums=1. Put concurrency on the machine pack ([pack.<ID>] worker_nums) and inject via MORTRED_PACK. Calibrate with mortredctl calibrate; persist with --write-pack on a machine-local pack file, then restart the supervisor (§10.3).

16. Acceptance Gates

Gate Command Coverage
Static ./scripts/verify_deployment.sh --basic script syntax / manifests / compose YAML / dependency inventory / security_warn.sh --self-test
Full ./scripts/verify_deployment.sh --full + local weight sha256 + 3rd_party completeness
Live ./scripts/verify_deployment.sh --live + gateway probes (public healthz, authed inference)
One-shot mortredctl doctor live wrapper + security warnings (non-fatal unless --strict)

CI side: every change runs the cpu-profile full build + full unit suite on a GPU-less runner - the conditional-compilation path cannot silently rot; the sanitizers job keeps the TSAN/ASan gates running.

Found a discrepancy between this document and actual behavior? That is a bug - please open an issue.