Triton Inference Server — Multi-LoRA Hot-Swap

NVIDIA Triton with vLLM backend serving Llama 3.1 8B as a base model with hot-swappable LoRA adapters. Route requests to different fine-tuned models via the model field — one GPU serves multiple customers/use-cases.

GPU: 1x A10G or L4 (24GB VRAM) · Cold start: ~90s · OpenAI-compatible: Yes (via OpenAI frontend)

Prerequisites

• Convox rack v3.24.6+ with GPU-capable nodes (g5.xlarge recommended)
• A HuggingFace access token (Llama 3.1 is gated)

Five-Minute Quickstart

git clone https://github.com/convox-examples/inference-examples.git
cd inference-examples/triton-multi-lora

convox apps create triton-lora
convox env set HUGGING_FACE_HUB_TOKEN=hf_your_token_here -a triton-lora
convox deploy -a triton-lora

Get the Endpoint

convox services -a triton-lora

Test It

Request to base model:

ENDPOINT=$(convox services -a triton-lora | awk '$1 == "triton" {print $2}')

curl -s "https://$ENDPOINT/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama_3_1_8b",
    "messages": [{"role": "user", "content": "What is Convox?"}],
    "max_tokens": 100
  }' | jq .

Request to a LoRA adapter:

curl -s "https://$ENDPOINT/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "customer_a_finance_v3",
    "messages": [{"role": "user", "content": "Summarize Q4 earnings"}],
    "max_tokens": 200
  }' | jq .

The model field selects which LoRA adapter to apply. The base model handles all requests; LoRA adapters modify behavior per-tenant.

Adding LoRA Adapters

Place LoRA adapter weights in the model repository:

model_repository/
  llama_3_1_8b/
    1/model.json          # Base model config with enable_lora=true
    config.pbtxt          # Triton model config
  loras/
    customer_a_finance/   # LoRA adapter weights (adapter_model.safetensors)
    customer_b_legal/

Update model.json to reference LoRA paths, then redeploy.

AWS Instance Sizing

Instance	GPU	VRAM	LoRA Capacity
g5.xlarge	1x A10G	24 GB	Base + 3-4 small LoRAs
g5.2xlarge	1x A10G	24 GB	More CPU for preprocessing

Budget Controls

convox budget set triton-lora --monthly-cap-usd 400 --at-cap-action alert-only

Scaling

Default scales to zero when idle. Cold start is ~90s (base model load + LoRA adapter registration).

For always-on (no cold start penalty): set scale.min: 1 in your convox.yml. Each replica loads all registered LoRA adapters.

GPU Observability

Enable GPU telemetry in Rack Settings to surface per-app GPU utilization, memory, temperature, and inference throughput in the Console GPU Dashboard.

Troubleshooting

Symptom	Cause	Fix
model_repository not found	Model repository path misconfigured	Verify --model-repository points to the correct mount path in convox.yml
LoRA adapter not loading	Adapter path not registered in model.json	Check that adapter directory exists and model.json references it correctly
OOM with multiple LoRAs	Too many adapters for available VRAM	Reduce number of concurrent LoRAs or upgrade to g5.2xlarge
Health check timeout	Base model still loading	Increase readiness probe timeout; Triton + vLLM backend needs ~90s