Skip to content

07-context-engineering.md

Latest commit

 

History

History
243 lines (182 loc) · 9.64 KB

07-context-engineering.md

File metadata and controls

243 lines (182 loc) · 9.64 KB
Raw

Journey 7 — Context Engineering & Observability

You want to actively manage the context window itself: see what's in it, measure cost, decide what to keep or evict, plan a budget, and reach for the deeper "power" tools. This journey documents the advanced and meta tools that don't belong to a single everyday flow — the ones that make lean-ctx a context runtime, not just a compressor.

Most tools here are in the power profile. Enable them with lean-ctx tools power, or load just one category at runtime with ctx_load_tools (see §6).

Source files referenced here:

  • rust/src/tools/registered/ctx_radar.rs, ctx_metrics.rs, ctx_cost.rs, ctx_feedback.rs, ctx_verify.rs, ctx_proof.rs
  • rust/src/tools/ctx_control.rs
  • rust/src/tools/ctx_preload.rs, ctx_prefetch.rs, ctx_dedup.rs, ctx_compose.rs
  • rust/src/cli/context_cmd.rs, ledger_cmd.rs

3.11.5: the MCP tools ctx_plan, ctx_compile, ctx_fill, ctx_ledger, ctx_overview, ctx_smart_read, ctx_symbol, ctx_retrieve, ctx_workflow, ctx_response, ctx_benchmark, ctx_analyze and ctx_compare were removed; the CLI commands below remain. Successors: docs/migration/focus-2026-10.md.

1. See what's in the context — observability

Before managing context, you measure it.

Tool CLI What it answers
ctx_radar — Full budget breakdown: prompt, messages, tools, reads, shell
ctx_metrics lean-ctx stats Session token stats, cache hit-rates, per-tool savings
ctx_context — Session-context overview: cache, files seen, current state
ctx_cost lean-ctx gain --cost Local cost attribution per agent / per tool
ctx_heatmap lean-ctx heatmap File-access heatmap (hot vs. cold files) 
ctx_radar format=display      # human-readable context budget
ctx_radar format=json         # machine-readable, for dashboards

ctx_radar is the single best "where are my tokens going?" view: it attributes the live context window across system prompt, message history, tool schemas, file reads, and shell output. Pair it with ctx_metrics for cumulative savings.

Read metrics are observations, not task acceptance. CLI and MCP reads retain mode statistics, token measurements and access history, but neither delivery nor bounce rate supplies task-completion feedback. The CLI's OCLA read observation has no acceptance or quality score; ETPAO requires an actual acceptance signal. Legacy feedback files are preserved, not rewritten by reads. A read does not supply the validated execution protocol required by canonical outcome learning. For local MCP diagnostics, LEAN_CTX_LOG=lean_ctx::read_observation=debug emits read_observation_worker_completed after the detached observation worker returns without a panic. This fixed event contains no paths or content and does not assert successful persistence or acceptance; it is not product telemetry.

2. Context Field Theory — actively shape the window

lean-ctx models the context window as a field you can manipulate with overlays (exclude, pin, prioritize) rather than only react to.

ctx_control / lean-ctx control

Overlay-based manipulation. Overlays apply at a scope (call, session, or project) and are reversible.

lean-ctx control pin src/auth.rs --reason "active task"
lean-ctx control exclude vendor/ --scope session
lean-ctx control set_priority src/main.rs --value high
lean-ctx control list            # current overlays
lean-ctx control history         # what changed and why
lean-ctx control reset           # drop overlays

Actions: exclude, include, pin, unpin, evict, set_view, set_priority, mark_outdated, reset, list, history.

lean-ctx ledger — pressure management

The ledger tracks per-file context "pressure" (token cost vs. recency vs. use) and lets you evict the expensive, stale entries.

lean-ctx ledger status           # pressure table
lean-ctx ledger evict big.json large.log
lean-ctx ledger prune            # drop low-value entries
lean-ctx ledger reset

Over MCP, evict with ctx_control action=evict targets=big.json,large.log.

lean-ctx plan — budget a task up front

Phi-scored planning: given a task and a token budget, it allocates the budget across the files/symbols most worth loading.

lean-ctx plan "add OAuth login" --budget=4000

lean-ctx compile — fill the budget optimally

Knapsack + Boltzmann view-selection: compiles the actual context to send under a budget, choosing per-file views (handles, compressed, or full).

lean-ctx compile --mode compressed --budget=6000

For both CLI commands, --budget N and --budget=N accept a nonnegative integer that fits the platform; zero is an explicit zero budget. Missing values, invalid, overflowing or duplicate budget values exit with invalid_budget instead of falling back to the default. Daemon and local fallback receive the same numeric budget. Pinned items cannot exceed the compilation budget: when no declared view fits, they are excluded with an explicit reason.

Together these form a pipeline: radar (measure) → plan (allocate) → compile (materialize) → control/ledger (adjust). Over MCP, ctx_compose selects under the turn budget and ctx_read loads the chosen files.

3. Proactive context — load before you're asked

Tool What it does
ctx_preload Load task-relevant files now; compact L-curve summary (~50–100 tok)
ctx_prefetch Predictive prefetch of blast-radius files for changed files
ctx_compose One call: keywords + ranked files + matches + top symbol inline
ctx_dedup Detect (and optionally remove) duplicated content across files 
ctx_preload task="refactor the auth module"
ctx_compose task="where is rate limiting enforced?"
ctx_prefetch changed_files=["src/auth.rs"] budget_tokens=3000
ctx_dedup action=analyze        # then action=apply to reclaim

ctx_compose is the highest-leverage everyday power tool: it replaces the typical search → read → outline → read chain (3–5 calls) with one rich response.

Golden output — the compact search primitive. ctx_compose builds on ctx_search, whose results are deliberately terse — a header plus one line per hit (path:line code), so a match costs a handful of tokens instead of pages of grep context:

1 matches in 805 files:
hooks/mod.rs:153 pub fn refresh_installed_hooks() {

ctx_compose then ranks the surrounding files and inlines the top symbol, so the agent gets the answer — not just the location — in the same call.

4. Advanced reads & symbols

Beyond ctx_read (Journey 2):

Tool What it does
ctx_read mode="auto" Auto-pick the optimal read mode for a file (paths=[…] batch-reads)
ctx_search action="symbol" Find one named symbol (fn/struct/class) by name
ctx_outline List all symbols of a file with signatures
ctx_compress_memory Compress memory/config files (CLAUDE.md, .cursorrules)
ctx_expand Zero-loss retrieval of an archived tool output or withheld content by id

ctx_expand retrieves archived tool output (retrieve, list, search_all) while the archive remains available and the caller retains permission. Expired, deleted, or inaccessible sources and archives cannot be recovered by this path.

5. Execution

Tool What it does
ctx_execute Sandboxed code execution (11 languages); only stdout enters context 
ctx_execute language=python code="print(sum(range(100)))"

Workflow and task execution belong to the host; ctx_work_graph keeps the shared coordination state between agents.

6. Dynamic tool loading — keep the surface small

You don't need the complete tool set loaded to use one. Lazy clients (and minimal/ standard profiles) reach deeper tools on demand:

ctx_discover_tools query="impact analysis"   # find tools by keyword
ctx_call name=ctx_graph arguments={...}       # call any tool by name
ctx_load_tools action=load category=arch      # load a category at runtime
ctx_load_tools action=list                    # what's loaded

Categories: arch, debug, memory, metrics, session. This is how you keep per-call overhead low (small visible tool list) without losing access to the full runtime.

7. Verification & proofs (CI / audit)

Tool CLI What it does
ctx_verify lean-ctx verify Verification observability + ContextProofV2
ctx_proof lean-ctx proof Export machine-readable ContextProofV1
ctx_feedback — Record LLM output tokens + latency for harness feedback
— lean-ctx benchmark Benchmark compression modes for a file/project
— lean-ctx compress diff <file> Preview compression — original vs the bytes lean-ctx would emit (read-only) 
lean-ctx benchmark run            # compare read modes on this repo
lean-ctx verify --format both
lean-ctx proof export             # ContextProof artifact for audit

These exist so the savings are provable, not just claimed — useful in CI to assert a context budget, or for an audit trail (lean-ctx audit).

UX notes captured during this walkthrough

  • This layer is genuinely advanced; it's gated behind the power profile on purpose so new users aren't overwhelmed. The radar → plan → compile pipeline is the through-line that makes the CFT tools coherent rather than a grab-bag.
  • ctx_compose deserves promotion: it's the one power tool worth using daily, so it's also cross-linked from Journey 2.