Prefly

Rehearse your deploy on a cheap throwaway environment before you ship.

Prefly spins up an ephemeral test stand, hammers it with real load, runs security checks, tells you if it'll survive prod — and what it'll cost. Then it shuts down automatically.

build → provision → deploy → smoke → load + security → report (pass/fail + $$) → teardown → gate real deploy

Why

• No more "it worked in staging" surprises
• Know your p95/p99 latency and max RPS before users hit it
• Right-size your infra: find the cheapest instance that meets your SLO
• Works for APIs, web apps, ML model servers

Quick start

pip install prefly

# scaffold config
prefly init

# edit prefly.yaml, then:
prefly run

# view report
prefly report reports/latest-result.json

Exit code 0 = pass, 1 = SLO violated — drop it in CI as a gate.

prefly.yaml

version: 1
name: my-api

artifact:
  type: docker
  image: myapp:latest
  port: 8080
  health: GET /healthz

target:
  mode: docker          # docker | staging | ephemeral (coming soon)

load:
  profile: ramp         # constant | ramp | spike | soak
  duration: 1m
  target_rps: 500

slo:
  p95_ms: 200
  error_rate_max: 0.01
  min_rps: 400

probes:
  security: true
  bot_resilience: true

Architecture

• Rust engine (prefly-engine) — async load generator with HDR histograms, honest RPS, no GC overhead
• Python layer — orchestration, provisioning, cost engine, reports, integrations
• JSON contract between them — stable, versioned schema

Roadmap

• M1: docker + staging modes, constant/ramp, SLO gate, HTML report, CLI
• M2: Hetzner ephemeral provider, cost engine ($/request, $/month), right-sizing
• M3: spike/soak/stairstep profiles, GitHub Action, baseline regression
• M4: ML adapters (vLLM/TGI/Triton), chaos-lite, Prometheus/OTel export

Ephemeral mode (M2)

Run against a real throwaway Hetzner VPS, not your laptop:

target:
  mode: ephemeral
  ephemeral:
    instance: cpx31   # 4 vCPU, 8 GB — $0.021/hr
    region: nbg1

cost:
  project_prod_rps: 500   # estimates your monthly infra cost

HETZNER_TOKEN=... prefly run -c ephemeral.yaml

The report includes test cost ($0.0007 for a 2-minute run) and estimated monthly prod cost based on your traffic.

GitHub Action (M3)

- uses: yakubka/prefly@main
  with:
    config: prefly.yaml
    baseline: .prefly-baseline.json   # optional: detect regressions

Exit code propagates to CI — merge blocked on SLO violation or latency regression.

Load profiles

Profile	Description
constant	Steady RPS for the full duration
ramp	Linear increase from ramp_from to target_rps
spike	Ramp to max in 10%, hold, decay back in last 10%
stairstep	Step up 25% every quarter of the duration
soak	Constant load, designed for long multi-hour runs

Baseline regression

# first run: saves baseline
prefly run --baseline .prefly-baseline.json

# subsequent runs: compares and fails if p95 regresses >20% or rps drops >10%
prefly run --baseline .prefly-baseline.json

ML serving load tests (M4)

ml:
  adapter: vllm       # vllm | tgi | triton
  model: meta-llama/Llama-3.1-8B-Instruct
  prompt: "Explain what a load balancer does in two sentences."
  max_tokens: 128

Builds OpenAI-compatible chat completion requests automatically. See examples/ml-vllm.yaml.

Chaos-lite

probes:
  chaos: true

Fires a handful of common client misbehaviors — oversized headers, malformed JSON, abrupt disconnects, a burst of malformed concurrent requests — then confirms the target still answers normally afterward. Not a fuzzer, a sanity check.

Metrics export

export:
  prometheus_file: metrics.prom      # node_exporter textfile format
  otlp_endpoint: http://otel-collector:4318

License

MIT OR Apache-2.0