0.11.0

[Ghassan-elsman]

Release v0.11.0 — Eye AI Compliance & Correlation Engine Upgrade

This release is a deep reliability, extensibility, and accuracy upgrade. We focused heavily on enhancing the Eye AI Agent with detailed compliance for its thinking and tooling processes, alongside a production-grade upgrade of the Correlation Engine and **Dynamic Linking

(Note: This release incorporates the unreleased 0.10.2 correlation work, shipping everything together as 0.11.0).

Eye AI — Detailed Compliance & Enhancements
Eye is now a much more robust, verifiable, and capable forensic assistant, with a heavy focus on transparent reasoning and strict compliance.

• ThinkingStep Protocol & Tooling Compliance: A new structured bridge (eye_bridge.py) drives the AI's reasoning steps, streaming thinking → rag → tool_call → synthesis directly to the new React-based UI. Every tool call and context decision is tracked in a new truncation_audit.log.
• Evidence Seal (Chain of Custody): Every payload sent to the AI model is sealed with the SHA-256 hash of the exact bytes, token count, and per-row provenance. Seals are append-only and hash-chained—ensuring verifiable integrity.
• Self-Healing Context: Eye now measures the full payload and reserves reply room. If evidence overflows, it heals the context via a summarize/drop pass without ever touching protected evidence. It refuses silent truncation entirely.
• Living Report Workspace: Every AI answer is produced twice: once as a chat reply, and once as a structured block in the case dossier. Includes new tools to generate tables, charts, timelines, and heatmaps.
• Governance Enforcement: Eye-authored rules are strictly governed. The AI must provide reasoning and evidence references when editing rules, and it is blocked from modifying human-authored built-in rules.

Correlation Engine: Accuracy & Evidence-Completeness Update

19 A focused accuracy pass after extensive end-to-end validation against a real ~700K-record Windows case, layered on top of the reliability work. Every fix below is locked by the 96-test pytest suite and verified by a holistic validation harness,

Identity engine now captures all the evidence:

• Fixed: identity engine was iterating only the FIRST row of every feather when a time filter was active due to a timezone-aware vs naive datetime comparison. Records-seen jumped from 3,558 → 745,615 on the validation case.
• Fixed: log records collapsed every event to its event PROVIDER as the identity. All 33,855 SecurityLogs records previously extracted as Microsoft-Windows-Security-Auditing. The engine now prioritizes real per-row entities (User, ComputerName, NewProcessName). It falls back to feather_metadata.artifact_type so artifact-aware mapping functions correctly.
• Fixed: placeholder strings became false identities. 'N/A', 'Unknown', '-', nil-GUIDs, and 30+ placeholder variants are now rejected.
• Result: The Execution Proof wing now surfaces 2,856 cross-feather (High) matches in the identity engine and 643 cross-feather matches in the time engine. Identity-engine total matches per wing jumped to 82K–99K.
• Fix: Single-feather vs multi-feather matches. Matches with feather_count == 1 now properly receive confidence_category="Low - single feather" / confidence_score=0.5. The High view now focuses purely on real cross-feather correlation.
• Fix: Path-aware composite key split identical identities. Because feathers store paths differently (BAM /device/..., LNK c:/..., MFT 8.3 short paths), the key is now name-only (identity_grouping.identity_key), enabling proper cross-feather correlation.
• Impersonation detection via path classification: Replaces path-aware keying. The engine walks all records' path fields and classifies them as TRUSTED (Program Files, System32, WinSxS) or SUSPICIOUS (Temp,Downloads, Recycle Bin, removable roots).
• A match with records in BOTH classifications gets match.semantic_data['impersonation_alert']. The validation case produced ~50 alerts per wing (≈0.05% rate) of highly relevant candidates.
• Drop Ledger: A per-window drop ledger logs named buckets (no_identity_field, normalize_failure) with sample identities to transparently state what was lost.
• Pipeline Summary: Every pipeline outputs a summary block: records seen, high/low confidence emitted, drop buckets. Every record lands in a match or a named drop bucket. low_confidence_review_mode is now ON by default.
• Timeless Enrichment: Feathers without per-row timestamps (AutoStartPrograms, MUICache) no longer receive fake synthetic timestamps. They now dynamically join formed matches by identity, providing supplementary evidence without altering temporal accuracy.
• Consolidated identity registry & Semantic fixes:
  • Single Source of Truth: config/standard_fields/identities.json now acts as the sole registry with 98 categories and 1,146 column synonyms. Adding a new path column synonym is now a JSON edit, not a code change.
  • Semantic Mappings: Fixed logic bugs (4625 AND 4624), appropriately gated multi-indicator rules (default-data-exfiltration-pattern requires _min_indicators=2), and converted wildcard tool definitions (af_wiper_tool_run) to exact regexes. Baseline activity severity inflation was demoted from high/critical to info/low. Global mappings with artifact_type are no longer evaluated twice.
• Time-Engine Carry-Over: Method 0 schema-driven timestamp detection reads from feather_schemas.json. Implemented per column WHERE-clause format binding. A Year 1990–2100 plausibility filter drops NTFS / LNK placeholder dates (1601/1980/2000).

Correlation Engine: Reliability & Extensibility Update

No dropped evidence & Smarter grouping: closes the most common "where did my evidence go?" gaps and lays the groundwork for parallel correlation.

• Multi-timestamp fan-out: Prefetch's run_times JSON list (up to 8 historical executions) expands into one virtual record per timestamp. mft_usn went from 2 records returned to 913,809 virtual records.
• Tolerant Timestamp Parser: Windows FILETIME, YYYYMMDD, ISO 8601, Unix epoch, and trailing annotations are all classified on the first try.
• Duplicates Preserved: The dedup signature keys on the raw identity. Chrome.exe and Chrome.dll share an identity bucket but each row is counted. Case/extension differences collapse, while version (v1.0 vs v2.0) and architecture (x86 vs x64) stay split as signal. Wider identity-field coverage recognizes 290+ name/path synonyms.
• Performance foundation & Feather generation:
  • O(1) Streaming: Rebuilt on a streaming generator (query_time_range_iter()) fetching records in 1,000-row batches via a thread-safe RLock-protected cache.
  • FeatherWriter API: A new canonical write contract with WAL journaling, explicit BEGIN/COMMIT transactions, and executemany() batched inserts (5000/batch) for 50–200× faster imports. Schema metadata and multi-timestamp JSON column declarations (declare_multi_timestamp_json) are stamped into the feather DB.
  • Diagnostics: Every correlation window emits an INFO line detailing records in/dropped/emitted.

Dynamic Linking (Intelligence Engine)

• Crow_Intelligence.db Orchestration: The engine now handles intelligence gathering from artifacts, IOC file ingestion (CSV/JSON), and dynamic rule mapping.
• Non-Destructive Enrichment: Tables are visually enriched in-place (appending [Dynamic_Key] to matched cells) via a cached background lookup against Crow_Intelligence.db, ensuring the original evidence database is never altered.

Core Application & UI Upgrades

• Virtualized Table Views: Introduced a truly virtualized QTableView with lazy loading (256 rows/page, ordered rowid index). Deep scrolling is now O(page) instead of SQLite's O(offset), allowing Crow-Eye to scale smoothly to multi-million-row MFT, USN, and SRUM tables without UI lag.
• React Timeline: Fully migrated to the modern react-timeline. All legacy PyQt timeline widgets have been removed.

---

This discussion was created from the release 0.11.0.