diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c769c6d..a8be10b 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -20,8 +20,8 @@ You are an expert Rust developer building **Nexus**, a distributed LLM orchestra |---------|-------|--------| | v0.1 | Foundation — Registry, Health, Router, mDNS, CLI, Aliases, Fallbacks | ✅ Released | | v0.2 | Observability — Prometheus metrics, Web Dashboard, Structured logging | ✅ Released | -| v0.3 | Cloud Hybrid — Cloud backends, Privacy zones, Budget management | Next | -| v0.4 | Intelligence — Speculative router, Quality tracking, Embeddings, Queuing | Planned | +| v0.3 | Cloud Hybrid — Cloud backends, Privacy zones, Budget management | ✅ Released | +| v0.4 | Intelligence — Speculative router, Quality tracking, Embeddings, Queuing | ✅ Released | | v0.5 | Orchestration — Pre-warming, Model lifecycle, Multi-tenant, Rate limiting | Planned | | v1.0 | Complete Product — Management UI, full web-based control plane | Planned | @@ -77,6 +77,7 @@ src/ ├── health/ # Background health checker with backend-specific endpoints ├── logging/ # Structured logging middleware (JSON/pretty, per-component levels) ├── metrics/ # Prometheus metrics and JSON stats API (/metrics, /v1/stats) +├── queue/ # Request queuing with priority support and configurable timeout ├── registry/ # In-memory backend/model registry (DashMap, source of truth) ├── routing/ # Intelligent request routing (strategies, scoring, requirements) ├── lib.rs # Module declarations @@ -88,6 +89,7 @@ src/ | Method | Path | Description | |--------|------|-------------| | `POST` | `/v1/chat/completions` | Chat completion (streaming and non-streaming) | +| `POST` | `/v1/embeddings` | Generate embeddings (OpenAI-compatible) | | `GET` | `/v1/models` | List available models from healthy backends (per-backend entries) | | `GET` | `/v1/stats` | JSON stats: uptime, request counts, per-backend metrics | | `GET` | `/metrics` | Prometheus metrics (counters, histograms, gauges) | @@ -120,6 +122,8 @@ src/ | `StatsResponse` | `metrics/types.rs` | JSON response for `/v1/stats` with uptime, requests, backends, models | | `BackendStats` | `metrics/types.rs` | Per-backend stats: id, name, requests, latency, pending | | `WebSocketUpdate` | `dashboard/types.rs` | Real-time dashboard updates: backend status, request history | +| `RequestQueue` | `queue/mod.rs` | Bounded priority queue with configurable timeout and drain loop | +| `QueuedRequest` | `queue/mod.rs` | Pending request: oneshot sender, model, priority, deadline | ### Key Patterns diff --git a/README.md b/README.md index e23961e..c6268e9 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,9 @@ Nexus is a distributed LLM orchestrator that unifies heterogeneous inference bac - 🔒 **Privacy Zones** — Structural enforcement prevents data from reaching cloud backends - 💰 **Budget Management** — Token-aware cost tracking with automatic spend limits - 📊 **Real-time Dashboard** — Monitor backends, models, and requests in your browser +- 🧠 **Quality Tracking** — Profiles backend response quality to inform routing decisions +- 📐 **Embeddings API** — OpenAI-compatible `/v1/embeddings` with capability-aware routing +- 📋 **Request Queuing** — Holds requests when backends are busy, with priority support ## Supported Backends @@ -64,19 +67,20 @@ Point any OpenAI-compatible client to `http://localhost:8000/v1` — Claude Code ## Architecture ``` -┌─────────────────────────────────────────────┐ -│ Nexus Orchestrator │ -│ - Discovers backends via mDNS │ -│ - Tracks model capabilities │ -│ - Routes to best available backend │ -│ - OpenAI-compatible API │ -└─────────────────────────────────────────────┘ - │ │ │ - ▼ ▼ ▼ - ┌────────┐ ┌────────┐ ┌────────┐ - │ Ollama │ │ vLLM │ │ exo │ - │ 7B │ │ 70B │ │ 32B │ - └────────┘ └────────┘ └────────┘ +┌──────────────────────────────────────────────────┐ +│ Nexus Orchestrator │ +│ - Discovers backends via mDNS │ +│ - Tracks model capabilities & quality │ +│ - Routes to best available backend │ +│ - Queues requests when backends are busy │ +│ - OpenAI-compatible API + Embeddings │ +└──────────────────────────────────────────────────┘ + │ │ │ │ + ▼ ▼ ▼ ▼ + ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ + │ Ollama │ │ vLLM │ │ exo │ │ OpenAI │ + │ 7B │ │ 70B │ │ 32B │ │ cloud │ + └────────┘ └────────┘ └────────┘ └────────┘ ``` ## Documentation diff --git a/docs/architecture.md b/docs/architecture.md index 958d2ab..460cb9f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -11,6 +11,7 @@ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ │ │ │ │ /v1/chat/ │ │ /v1/models │ │ /health │ │ │ │ │ │ completions │ │ /v1/stats │ │ /metrics │ │ │ +│ │ │ /v1/embeds │ │ │ │ │ │ │ │ │ └──────┬───────┘ └──────┬───────┘ └──────────┬───────────┘ │ │ │ │ │ │ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ @@ -46,6 +47,10 @@ │ │ │ Health Checker │ │ mDNS Discovery │ │ │ │ │ │ (30s interval) │ │ (continuous) │ │ │ │ │ └──────────────────┘ └──────────────────────────────────┘ │ │ +│ │ ┌──────────────────┐ ┌──────────────────────────────────┐ │ │ +│ │ │ Quality Tracker │ │ Request Queue (drain loop) │ │ │ +│ │ │ (30s interval) │ │ (continuous) │ │ │ +│ │ └──────────────────┘ └──────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ │ @@ -76,6 +81,7 @@ pub mod health; pub fn router(state: AppState) -> Router { Router::new() .route("/v1/chat/completions", post(chat::completions)) + .route("/v1/embeddings", post(embeddings::create)) .route("/v1/models", get(models::list)) .route("/health", get(health::check)) .route("/v1/stats", get(stats::handle)) @@ -90,6 +96,7 @@ pub fn router(state: AppState) -> Router { | Endpoint | Method | Description | |----------|--------|-------------| | `/v1/chat/completions` | POST | Chat completion (streaming or not) | +| `/v1/embeddings` | POST | Generate embeddings (OpenAI-compatible) | | `/v1/models` | GET | List all available models (per-backend entries with `owned_by` = backend name) | | `/v1/stats` | GET | JSON stats: uptime, request counts, per-backend metrics | | `/metrics` | GET | Prometheus metrics (counters, histograms, gauges) | @@ -432,6 +439,21 @@ Structured logging via `tracing` (`src/logging/`). `backend`, `backend_type`, `status`, `status_code`, `latency_ms`, `tokens_prompt`, `tokens_completion`, `tokens_total`, `stream`, `route_reason`, `retry_count`. +### 11. Request Queue + +Bounded priority queue for holding requests when all backends are busy (`src/queue/`). + +| File | Purpose | +|------|---------| +| `mod.rs` | `RequestQueue` — CAS-based bounded enqueue, priority channels, drain loop | + +**Key behaviors:** +- Atomic `compare_exchange` loop prevents TOCTOU race on queue depth +- Dual priority channels (high/normal) with single drain loop +- Configurable timeout with deadline tracking per request +- Re-runs reconciler pipeline on drain to find newly-available backends +- Queue depth exposed via Prometheus gauge (`nexus_queue_depth`) + --- ## Data Flow Diagrams @@ -858,8 +880,8 @@ Request └──────────┬───────────────────┘ ▼ ┌──────────────────────────────┐ -│ 5. QualityReconciler │ Reserved for future quality metrics -│ (pass-through Phase 1) │ (latency, accuracy, user ratings) +│ 5. QualityReconciler │ Deprioritize backends with high error rates +│ │ and slow TTFT using rolling window metrics └──────────┬───────────────────┘ ▼ ┌──────────────────────────────┐ @@ -888,7 +910,7 @@ RequestAnalyzer: resolved_model, candidate_agents PrivacyReconciler: privacy_constraint, excluded_agents, rejection_reasons BudgetReconciler: cost_estimate, budget_status, excluded_agents, rejection_reasons TierReconciler: min_capability_tier, excluded_agents, rejection_reasons -QualityReconciler: (none — pass-through) +QualityReconciler: quality_scores, excluded_agents (high error rate), rejection_reasons SchedulerReconciler: candidate_agents (→ single winner), route_reason ``` @@ -1024,7 +1046,7 @@ Per-backend tokenizer for audit-grade token counting and budget management: └─────────────────────────────────────────────────────────┘ ``` -### Speculative Router (v0.4) +### Speculative Router (v0.4) — Implemented Zero-ML request-content routing via JSON payload inspection: diff --git a/docs/getting-started.md b/docs/getting-started.md index a2a965f..7be43b9 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -228,6 +228,23 @@ strategy = "smart" # smart | round_robin | priority_only | random # monthly_limit_usd = 50.0 # soft_limit_percent = 75 # At 75%: prefer local backends # hard_limit_action = "block_cloud" # At 100%: warn | block_cloud | block_all + +# --- Quality Tracking --- + +# [quality] +# enabled = true +# window_1h = 3600 # Rolling window for 1-hour metrics (seconds) +# window_24h = 86400 # Rolling window for 24-hour metrics (seconds) +# update_interval = 30 # How often to recompute quality scores (seconds) +# error_rate_threshold = 0.5 # Exclude backends with error rate above this + +# --- Request Queuing --- + +# [queue] +# enabled = true +# max_queue_size = 100 # Maximum queued requests before rejecting +# default_timeout_seconds = 30 # How long to wait before 503 +# priority_header = "X-Nexus-Priority" # Header for request priority (high | normal) ``` ### Environment Variables diff --git a/docs/roadmap.md b/docs/roadmap.md index e98bc6d..6cb1300 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -13,7 +13,7 @@ Nexus is a control plane for heterogeneous LLM inference — it routes requests | **v0.1** | Foundation | Registry, Health, Router, mDNS, CLI, Aliases, Fallbacks | ✅ Released | | **v0.2** | Observability | Prometheus metrics, Web Dashboard, Structured logging | ✅ Released | | **v0.3** | Cloud Hybrid | Cloud backends, Privacy zones, Capability tiers, Budget management | ✅ Released | -| **v0.4** | Intelligence | Speculative router, Quality tracking, Embeddings, Queuing | 🚧 In Progress | +| **v0.4** | Intelligence | Speculative router, Quality tracking, Embeddings, Queuing | ✅ Complete | | **v0.5** | Orchestration | Pre-warming, Model lifecycle, Multi-tenant, Rate limiting | Planned | | **v1.0** | Complete Product | Management UI — full web-based control plane | Planned | @@ -40,10 +40,10 @@ Nexus is a control plane for heterogeneous LLM inference — it routes requests | F13 | Privacy Zones & Capability Tiers | v0.3 | ✅ Complete | [specs/015-privacy-zones-capability-tiers](../specs/015-privacy-zones-capability-tiers/) | | F14 | Inference Budget Management | v0.3 | ✅ Complete | [specs/016-inference-budget-mgmt](../specs/016-inference-budget-mgmt/) | | — | **Quality + Queuing (Phase 2.5)** | **v0.4** | ✅ Complete | [specs/017-quality-tracking-embeddings-queuing](../specs/017-quality-tracking-embeddings-queuing/) | -| F15 | Speculative Router | v0.4 | Planned | - | -| F16 | Quality Tracking & Backend Profiling | v0.4 | ✅ Complete | [#173](https://github.com/leocamello/nexus/issues/173) | -| F17 | Embeddings API | v0.4 | ✅ Complete | [#174](https://github.com/leocamello/nexus/issues/174) | -| F18 | Request Queuing & Prioritization | v0.4 | ✅ Complete | [#176](https://github.com/leocamello/nexus/issues/176), [#177](https://github.com/leocamello/nexus/issues/177) | +| F15 | Speculative Router | v0.4 | ✅ Complete | [specs/018-speculative-router](../specs/018-speculative-router/) | +| F16 | Quality Tracking & Backend Profiling | v0.4 | ✅ Complete | [specs/019-quality-tracking](../specs/019-quality-tracking/) | +| F17 | Embeddings API | v0.4 | ✅ Complete | [specs/020-embeddings-api](../specs/020-embeddings-api/) | +| F18 | Request Queuing & Prioritization | v0.4 | ✅ Complete | [specs/021-request-queuing](../specs/021-request-queuing/) | | — | **Fleet Intelligence (Phase 3)** | **v0.5** | Planned | - | | F19 | Pre-warming & Fleet Intelligence | v0.5 | Planned | - | | F20 | Model Lifecycle Management | v0.5 | Planned | - | @@ -71,15 +71,18 @@ Extended Nexus from a local-only gateway to a hybrid local+cloud control plane. ### v0.4 (Phase 2.5) — Quality, Embeddings & Queuing -Added intelligence features that make Nexus smarter about backend quality and request handling. Quality tracking ([#173](https://github.com/leocamello/nexus/issues/173)) profiles backend response quality — tracking success rates, error rates, and latency per model to inform routing decisions. The Embeddings API ([#174](https://github.com/leocamello/nexus/issues/174)) adds `/v1/embeddings` support with capability-aware routing to Ollama and OpenAI backends. Request queuing ([#176](https://github.com/leocamello/nexus/issues/176), [#177](https://github.com/leocamello/nexus/issues/177)) holds requests when all capable backends are at capacity instead of rejecting immediately, with priority support via the `X-Nexus-Priority` header. +Added intelligence features that make Nexus smarter about backend quality and request handling. The speculative router ([specs/018-speculative-router](../specs/018-speculative-router/)) pre-analyzes request patterns and model history to predict the optimal backend before full payload inspection. Quality tracking ([specs/019-quality-tracking](../specs/019-quality-tracking/)) profiles backend response quality — tracking success rates, error rates, and latency per model to inform routing decisions. The Embeddings API ([specs/020-embeddings-api](../specs/020-embeddings-api/)) adds `/v1/embeddings` support with capability-aware routing to Ollama and OpenAI backends. Request queuing ([specs/021-request-queuing](../specs/021-request-queuing/)) holds requests when all capable backends are at capacity instead of rejecting immediately, with priority support via the `X-Nexus-Priority` header. --- -## What's Next (v0.4 — remaining) +## What's Next (v0.5 — Orchestration) -The remaining **Intelligence** features for v0.4: +The next phase focuses on **fleet-level intelligence and operational control**: -- **Speculative Router (F15)** — Pre-analyze requests to predict optimal backend before payload inspection +- **Pre-warming & Fleet Intelligence (F19)** — Predictive model loading based on usage patterns +- **Model Lifecycle Management (F20)** — Load/unload/migrate models via API +- **Multi-Tenant Support (F21)** — API keys and per-tenant quotas +- **Rate Limiting (F22)** — Per-backend and per-tenant protection --- diff --git a/specs/020-embeddings-api/research.md b/specs/020-embeddings-api/research.md index f29a258..6892c44 100644 --- a/specs/020-embeddings-api/research.md +++ b/specs/020-embeddings-api/research.md @@ -329,7 +329,7 @@ if !agent.profile().capabilities.embeddings { **Nexus Implementation**: - ✅ Iterates per-input for batches -- ✅ Hardcoded to `all-minilm` model (TODO: make configurable) +- ✅ Uses model name from request (configurable per-request) - ✅ Transforms response to OpenAI format - ℹ️ Sequential processing (could be parallelized) @@ -417,11 +417,11 @@ if !agent.profile().capabilities.embeddings { - **Mitigation**: Backend reports actual tokens, can be monitored - **Status**: Accepted trade-off for performance (< 1ms routing) -**Risk 3: Model Name Hardcoding** -- **Impact**: OpenAI always uses `ada-002`, Ollama always uses `all-minilm` -- **Likelihood**: High (not configurable in initial implementation) -- **Mitigation**: Model name in request is cosmetic (backend overrides) -- **Status**: Documented as future enhancement +**Risk 3: Model Name Handling** +- **Impact**: Each agent type forwards the model name from the request to its backend +- **Likelihood**: Low (model name is now configurable per-request) +- **Mitigation**: Agent implementations pass through the requested model name +- **Status**: Resolved — model parameter forwarded correctly ### Security Considerations