docs: v0.4 pre-release documentation polish

.github/copilot-instructions.md

@@ -20,8 +20,8 @@ You are an expert Rust developer building **Nexus**, a distributed LLM orchestra
 |---------|-------|--------|
 | v0.1 | Foundation — Registry, Health, Router, mDNS, CLI, Aliases, Fallbacks | ✅ Released |
 | v0.2 | Observability — Prometheus metrics, Web Dashboard, Structured logging | ✅ Released |
-| v0.3 | Cloud Hybrid — Cloud backends, Privacy zones, Budget management | Next |
-| v0.4 | Intelligence — Speculative router, Quality tracking, Embeddings, Queuing | Planned |
+| v0.3 | Cloud Hybrid — Cloud backends, Privacy zones, Budget management | ✅ Released |
+| v0.4 | Intelligence — Speculative router, Quality tracking, Embeddings, Queuing | ✅ Released |
 | v0.5 | Orchestration — Pre-warming, Model lifecycle, Multi-tenant, Rate limiting | Planned |
 | v1.0 | Complete Product — Management UI, full web-based control plane | Planned |
 
@@ -77,6 +77,7 @@ src/
 ├── health/       # Background health checker with backend-specific endpoints
 ├── logging/      # Structured logging middleware (JSON/pretty, per-component levels)
 ├── metrics/      # Prometheus metrics and JSON stats API (/metrics, /v1/stats)
+├── queue/        # Request queuing with priority support and configurable timeout
 ├── registry/     # In-memory backend/model registry (DashMap, source of truth)
 ├── routing/      # Intelligent request routing (strategies, scoring, requirements)
 ├── lib.rs        # Module declarations
@@ -88,6 +89,7 @@ src/
 | Method | Path | Description |
 |--------|------|-------------|
 | `POST` | `/v1/chat/completions` | Chat completion (streaming and non-streaming) |
+| `POST` | `/v1/embeddings` | Generate embeddings (OpenAI-compatible) |
 | `GET` | `/v1/models` | List available models from healthy backends (per-backend entries) |
 | `GET` | `/v1/stats` | JSON stats: uptime, request counts, per-backend metrics |
 | `GET` | `/metrics` | Prometheus metrics (counters, histograms, gauges) |
@@ -120,6 +122,8 @@ src/
 | `StatsResponse` | `metrics/types.rs` | JSON response for `/v1/stats` with uptime, requests, backends, models |
 | `BackendStats` | `metrics/types.rs` | Per-backend stats: id, name, requests, latency, pending |
 | `WebSocketUpdate` | `dashboard/types.rs` | Real-time dashboard updates: backend status, request history |
+| `RequestQueue` | `queue/mod.rs` | Bounded priority queue with configurable timeout and drain loop |
+| `QueuedRequest` | `queue/mod.rs` | Pending request: oneshot sender, model, priority, deadline |
 
 ### Key Patterns
 

README.md

@@ -24,6 +24,9 @@ Nexus is a distributed LLM orchestrator that unifies heterogeneous inference bac
 - 🔒 **Privacy Zones** — Structural enforcement prevents data from reaching cloud backends
 - 💰 **Budget Management** — Token-aware cost tracking with automatic spend limits
 - 📊 **Real-time Dashboard** — Monitor backends, models, and requests in your browser
+- 🧠 **Quality Tracking** — Profiles backend response quality to inform routing decisions
+- 📐 **Embeddings API** — OpenAI-compatible `/v1/embeddings` with capability-aware routing
+- 📋 **Request Queuing** — Holds requests when backends are busy, with priority support
 
 ## Supported Backends
 
@@ -64,19 +67,20 @@ Point any OpenAI-compatible client to `http://localhost:8000/v1` — Claude Code
 ## Architecture
 
 ```
-┌─────────────────────────────────────────────┐
-│           Nexus Orchestrator                │
-│  - Discovers backends via mDNS              │
-│  - Tracks model capabilities                │
-│  - Routes to best available backend         │
-│  - OpenAI-compatible API                    │
-└─────────────────────────────────────────────┘
-        │           │           │
-        ▼           ▼           ▼
-   ┌────────┐  ┌────────┐  ┌────────┐
-   │ Ollama │  │  vLLM  │  │  exo   │
-   │  7B    │  │  70B   │  │  32B   │
-   └────────┘  └────────┘  └────────┘
+┌──────────────────────────────────────────────────┐
+│              Nexus Orchestrator                   │
+│  - Discovers backends via mDNS                   │
+│  - Tracks model capabilities & quality           │
+│  - Routes to best available backend              │
+│  - Queues requests when backends are busy        │
+│  - OpenAI-compatible API + Embeddings            │
+└──────────────────────────────────────────────────┘
+        │           │           │           │
+        ▼           ▼           ▼           ▼
+   ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐
+   │ Ollama │  │  vLLM  │  │  exo   │  │ OpenAI │
+   │  7B    │  │  70B   │  │  32B   │  │ cloud  │
+   └────────┘  └────────┘  └────────┘  └────────┘
 ```
 
 ## Documentation

docs/architecture.md

@@ -11,6 +11,7 @@
 │  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐  │ │
 │  │  │ /v1/chat/    │  │ /v1/models   │  │ /health              │  │ │
 │  │  │ completions  │  │ /v1/stats    │  │ /metrics             │  │ │
+│  │  │ /v1/embeds   │  │              │  │                      │  │ │
 │  │  └──────┬───────┘  └──────┬───────┘  └──────────┬───────────┘  │ │
 │  │                                                                  │ │
 │  │  ┌──────────────────────────────────────────────────────────┐   │ │
@@ -46,6 +47,10 @@
 │  │  │ Health Checker   │  │ mDNS Discovery                    │   │ │
 │  │  │ (30s interval)   │  │ (continuous)                      │   │ │
 │  │  └──────────────────┘  └──────────────────────────────────┘   │ │
+│  │  ┌──────────────────┐  ┌──────────────────────────────────┐   │ │
+│  │  │ Quality Tracker  │  │ Request Queue (drain loop)        │   │ │
+│  │  │ (30s interval)   │  │ (continuous)                      │   │ │
+│  │  └──────────────────┘  └──────────────────────────────────┘   │ │
 │  └──────────────────────────────────────────────────────────────────┘ │
 └─────────────────────────────────────────────────────────────────────┘
                                  │
@@ -76,6 +81,7 @@ pub mod health;
 pub fn router(state: AppState) -> Router {
     Router::new()
         .route("/v1/chat/completions", post(chat::completions))
+        .route("/v1/embeddings", post(embeddings::create))
         .route("/v1/models", get(models::list))
         .route("/health", get(health::check))
         .route("/v1/stats", get(stats::handle))
@@ -90,6 +96,7 @@ pub fn router(state: AppState) -> Router {
 | Endpoint | Method | Description |
 |----------|--------|-------------|
 | `/v1/chat/completions` | POST | Chat completion (streaming or not) |
+| `/v1/embeddings` | POST | Generate embeddings (OpenAI-compatible) |
 | `/v1/models` | GET | List all available models (per-backend entries with `owned_by` = backend name) |
 | `/v1/stats` | GET | JSON stats: uptime, request counts, per-backend metrics |
 | `/metrics` | GET | Prometheus metrics (counters, histograms, gauges) |
@@ -432,6 +439,21 @@ Structured logging via `tracing` (`src/logging/`).
 `backend`, `backend_type`, `status`, `status_code`, `latency_ms`, `tokens_prompt`,
 `tokens_completion`, `tokens_total`, `stream`, `route_reason`, `retry_count`.
 
+### 11. Request Queue
+
+Bounded priority queue for holding requests when all backends are busy (`src/queue/`).
+
+| File | Purpose |
+|------|---------|
+| `mod.rs` | `RequestQueue` — CAS-based bounded enqueue, priority channels, drain loop |
+
+**Key behaviors:**
+- Atomic `compare_exchange` loop prevents TOCTOU race on queue depth
+- Dual priority channels (high/normal) with single drain loop
+- Configurable timeout with deadline tracking per request
+- Re-runs reconciler pipeline on drain to find newly-available backends
+- Queue depth exposed via Prometheus gauge (`nexus_queue_depth`)
+
 ---
 
 ## Data Flow Diagrams
@@ -858,8 +880,8 @@ Request
 └──────────┬───────────────────┘
            ▼
 ┌──────────────────────────────┐
-│  5. QualityReconciler        │  Reserved for future quality metrics
-│     (pass-through Phase 1)   │  (latency, accuracy, user ratings)
+│  5. QualityReconciler        │  Deprioritize backends with high error rates
+│                              │  and slow TTFT using rolling window metrics
 └──────────┬───────────────────┘
            ▼
 ┌──────────────────────────────┐
@@ -888,7 +910,7 @@ RequestAnalyzer:    resolved_model, candidate_agents
 PrivacyReconciler:  privacy_constraint, excluded_agents, rejection_reasons
 BudgetReconciler:   cost_estimate, budget_status, excluded_agents, rejection_reasons
 TierReconciler:     min_capability_tier, excluded_agents, rejection_reasons
-QualityReconciler:  (none — pass-through)
+QualityReconciler:  quality_scores, excluded_agents (high error rate), rejection_reasons
 SchedulerReconciler: candidate_agents (→ single winner), route_reason
 ```
 
@@ -1024,7 +1046,7 @@ Per-backend tokenizer for audit-grade token counting and budget management:
 └─────────────────────────────────────────────────────────┘
 ```
 
-### Speculative Router (v0.4)
+### Speculative Router (v0.4) — Implemented
 
 Zero-ML request-content routing via JSON payload inspection:
 

docs/getting-started.md

@@ -228,6 +228,23 @@ strategy = "smart"    # smart | round_robin | priority_only | random
 # monthly_limit_usd = 50.0
 # soft_limit_percent = 75              # At 75%: prefer local backends
 # hard_limit_action = "block_cloud"    # At 100%: warn | block_cloud | block_all
+
+# --- Quality Tracking ---
+
+# [quality]
+# enabled = true
+# window_1h = 3600                     # Rolling window for 1-hour metrics (seconds)
+# window_24h = 86400                   # Rolling window for 24-hour metrics (seconds)
+# update_interval = 30                 # How often to recompute quality scores (seconds)
+# error_rate_threshold = 0.5           # Exclude backends with error rate above this
+
+# --- Request Queuing ---
+
+# [queue]
+# enabled = true
+# max_queue_size = 100                 # Maximum queued requests before rejecting
+# default_timeout_seconds = 30         # How long to wait before 503
+# priority_header = "X-Nexus-Priority" # Header for request priority (high | normal)
 ```
 
 ### Environment Variables

docs/roadmap.md

@@ -13,7 +13,7 @@ Nexus is a control plane for heterogeneous LLM inference — it routes requests
 | **v0.1** | Foundation | Registry, Health, Router, mDNS, CLI, Aliases, Fallbacks | ✅ Released |
 | **v0.2** | Observability | Prometheus metrics, Web Dashboard, Structured logging | ✅ Released |
 | **v0.3** | Cloud Hybrid | Cloud backends, Privacy zones, Capability tiers, Budget management | ✅ Released |
-| **v0.4** | Intelligence | Speculative router, Quality tracking, Embeddings, Queuing | 🚧 In Progress |
+| **v0.4** | Intelligence | Speculative router, Quality tracking, Embeddings, Queuing | ✅ Complete |
 | **v0.5** | Orchestration | Pre-warming, Model lifecycle, Multi-tenant, Rate limiting | Planned |
 | **v1.0** | Complete Product | Management UI — full web-based control plane | Planned |
 
@@ -40,10 +40,10 @@ Nexus is a control plane for heterogeneous LLM inference — it routes requests
 | F13 | Privacy Zones & Capability Tiers | v0.3 | ✅ Complete | [specs/015-privacy-zones-capability-tiers](../specs/015-privacy-zones-capability-tiers/) |
 | F14 | Inference Budget Management | v0.3 | ✅ Complete | [specs/016-inference-budget-mgmt](../specs/016-inference-budget-mgmt/) |
 | — | **Quality + Queuing (Phase 2.5)** | **v0.4** | ✅ Complete | [specs/017-quality-tracking-embeddings-queuing](../specs/017-quality-tracking-embeddings-queuing/) |
-| F15 | Speculative Router | v0.4 | Planned | - |
-| F16 | Quality Tracking & Backend Profiling | v0.4 | ✅ Complete | [#173](https://github.com/leocamello/nexus/issues/173) |
-| F17 | Embeddings API | v0.4 | ✅ Complete | [#174](https://github.com/leocamello/nexus/issues/174) |
-| F18 | Request Queuing & Prioritization | v0.4 | ✅ Complete | [#176](https://github.com/leocamello/nexus/issues/176), [#177](https://github.com/leocamello/nexus/issues/177) |
+| F15 | Speculative Router | v0.4 | ✅ Complete | [specs/018-speculative-router](../specs/018-speculative-router/) |
+| F16 | Quality Tracking & Backend Profiling | v0.4 | ✅ Complete | [specs/019-quality-tracking](../specs/019-quality-tracking/) |
+| F17 | Embeddings API | v0.4 | ✅ Complete | [specs/020-embeddings-api](../specs/020-embeddings-api/) |
+| F18 | Request Queuing & Prioritization | v0.4 | ✅ Complete | [specs/021-request-queuing](../specs/021-request-queuing/) |
 | — | **Fleet Intelligence (Phase 3)** | **v0.5** | Planned | - |
 | F19 | Pre-warming & Fleet Intelligence | v0.5 | Planned | - |
 | F20 | Model Lifecycle Management | v0.5 | Planned | - |
@@ -71,15 +71,18 @@ Extended Nexus from a local-only gateway to a hybrid local+cloud control plane.
 
 ### v0.4 (Phase 2.5) — Quality, Embeddings & Queuing
 
-Added intelligence features that make Nexus smarter about backend quality and request handling. Quality tracking ([#173](https://github.com/leocamello/nexus/issues/173)) profiles backend response quality — tracking success rates, error rates, and latency per model to inform routing decisions. The Embeddings API ([#174](https://github.com/leocamello/nexus/issues/174)) adds `/v1/embeddings` support with capability-aware routing to Ollama and OpenAI backends. Request queuing ([#176](https://github.com/leocamello/nexus/issues/176), [#177](https://github.com/leocamello/nexus/issues/177)) holds requests when all capable backends are at capacity instead of rejecting immediately, with priority support via the `X-Nexus-Priority` header.
+Added intelligence features that make Nexus smarter about backend quality and request handling. The speculative router ([specs/018-speculative-router](../specs/018-speculative-router/)) pre-analyzes request patterns and model history to predict the optimal backend before full payload inspection. Quality tracking ([specs/019-quality-tracking](../specs/019-quality-tracking/)) profiles backend response quality — tracking success rates, error rates, and latency per model to inform routing decisions. The Embeddings API ([specs/020-embeddings-api](../specs/020-embeddings-api/)) adds `/v1/embeddings` support with capability-aware routing to Ollama and OpenAI backends. Request queuing ([specs/021-request-queuing](../specs/021-request-queuing/)) holds requests when all capable backends are at capacity instead of rejecting immediately, with priority support via the `X-Nexus-Priority` header.
 
 ---
 
-## What's Next (v0.4 — remaining)
+## What's Next (v0.5 — Orchestration)
 
-The remaining **Intelligence** features for v0.4:
+The next phase focuses on **fleet-level intelligence and operational control**:
 
-- **Speculative Router (F15)** — Pre-analyze requests to predict optimal backend before payload inspection
+- **Pre-warming & Fleet Intelligence (F19)** — Predictive model loading based on usage patterns
+- **Model Lifecycle Management (F20)** — Load/unload/migrate models via API
+- **Multi-Tenant Support (F21)** — API keys and per-tenant quotas
+- **Rate Limiting (F22)** — Per-backend and per-tenant protection
 
 ---
 

specs/020-embeddings-api/research.md

@@ -329,7 +329,7 @@ if !agent.profile().capabilities.embeddings {
 
 **Nexus Implementation**:
 - ✅ Iterates per-input for batches
-- ✅ Hardcoded to `all-minilm` model (TODO: make configurable)
+- ✅ Uses model name from request (configurable per-request)
 - ✅ Transforms response to OpenAI format
 - ℹ️ Sequential processing (could be parallelized)
 
@@ -417,11 +417,11 @@ if !agent.profile().capabilities.embeddings {
 - **Mitigation**: Backend reports actual tokens, can be monitored
 - **Status**: Accepted trade-off for performance (< 1ms routing)
 
-**Risk 3: Model Name Hardcoding**
-- **Impact**: OpenAI always uses `ada-002`, Ollama always uses `all-minilm`
-- **Likelihood**: High (not configurable in initial implementation)
-- **Mitigation**: Model name in request is cosmetic (backend overrides)
-- **Status**: Documented as future enhancement
+**Risk 3: Model Name Handling**
+- **Impact**: Each agent type forwards the model name from the request to its backend
+- **Likelihood**: Low (model name is now configurable per-request)
+- **Mitigation**: Agent implementations pass through the requested model name
+- **Status**: Resolved — model parameter forwarded correctly
 
 ### Security Considerations