Cortex Strategic Architecture Implementation Notes

Source document: roadmap.md, section "Strategic Architecture".

This document converts each strategic architecture point into an implementation tasklist. It assumes the current Cortex technology baseline:

TypeScript/Node.js runtime on the Matbot plugin API.
Ordered plugins that register tools, services, stores, hooks, providers, and frontend surfaces.
Per-workspace config, secrets, .data stores, sessions, skills, memories, and RAG configuration.
Local HTTP services for file indexing and file access.
Postgres/pgvector for workspace RAG vectors, metadata, chunks, and search.
Neo4j is already present in the local Docker stack.
MCP and MCP-over-HTTP plugins exist but are not active by default.
The background plugin exists for recurring prompt jobs, but durable workflow semantics are not yet separated from freeform model prompts.
Hooks already provide useful governance points: toolcall can deny or gate a tool call, toolresult can observe/redact/audit results, and screen can add source context or markers.
A principal carrier exists and can attribute operations to a user, agent, or system principal. It grants nothing by itself; policy services must enforce authorization decisions.

Evidence Base

These sources inform the implementation options below:

Model Context Protocol 2025-06-18 defines a standard way for LLM hosts, clients, and servers to expose resources, prompts, and tools, while warning that hosts must provide consent, access controls, tool confirmation, timeouts, output sanitization, and audit logging: https://modelcontextprotocol.io/specification/2025-06-18
MCP tools are model-controlled, schema-described functions. The spec supports structured output and says sensitive operations should keep a human in the loop: https://modelcontextprotocol.io/specification/2025-06-18/server/tools
Microsoft Graph delta query is a proven incremental sync pattern for change tracking without full re-reads: https://learn.microsoft.com/en-us/graph/delta-query-overview
Slack's Events API guidance recommends fast acknowledgement and queue-based asynchronous processing: https://docs.slack.dev/apis/events-api/
Airbyte has a large open-source connector catalog, useful as an optional ELT path for sources where Cortex does not need interactive tool semantics: https://docs.airbyte.com/platform/move-data/sources-destinations-connectors
pgvector supports vector search inside Postgres, including HNSW indexing. This matches Cortex's existing Postgres/pgvector workspace RAG direction: https://github.com/pgvector/pgvector
Microsoft GraphRAG and Neo4j GraphRAG documentation support combining vector retrieval with graph relationships for multi-hop context: https://microsoft.github.io/graphrag/ https://neo4j.com/docs/neo4j-graphrag-python/current/user_guide_rag.html
dbt semantic models and Cube show the proven shape of semantic data layers: entities, measures, dimensions, joins, access rules, and shared metric definitions: https://docs.getdbt.com/docs/build/semantic-models https://docs.cube.dev/docs/introduction
SQLGlot provides parser/transpiler/optimizer machinery for validating SQL before execution: https://sqlglot.com/
Postgres row-level security is a database-native defense for row-scoped access enforcement: https://www.postgresql.org/docs/current/ddl-rowsecurity.html
W3C PROV and OpenLineage provide mature vocabulary for provenance, lineage, jobs, runs, datasets, entities, activities, and agents: https://www.w3.org/TR/prov-dm/ https://openlineage.io/docs/spec/object-model/
OpenTelemetry context propagation is the standard pattern for correlating traces, metrics, and logs across process boundaries: https://opentelemetry.io/docs/concepts/context-propagation/
OPA and Zanzibar are proven authorization patterns for policy-as-code and relationship-based authorization: https://www.openpolicyagent.org/ https://research.google/pubs/zanzibar-googles-consistent-global-authorization-system/
Temporal's SDK model is a proven option for durable workflows if Cortex outgrows an in-process/local runner: https://docs.temporal.io/encyclopedia/temporal-sdks
Ragas provides practical RAG evaluation metrics such as context precision: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/

Implementation Principles

Start as plugins, not core rewrites. Add new services through the existing Matbot service registry and make them available to tools, hooks, and WebUI.
Use typed stores first for iteration. Promote hot, relational, or analytical data to Postgres tables once schemas stabilize.
Keep MCP as an integration interface, not the policy boundary. Cortex must own connector grants, effective identity, approvals, source metadata, and audit.
Treat source metadata as product state. Retrieval hits, SQL results, graph nodes, dossiers, and workflow runs should all reference source ids and source versions.
Keep write actions behind explicit approval gates. Shadow mode and dry-run records should exist before approved execution.
Prefer deterministic code for sync, policy, SQL validation, health checks, and workflow transitions. Use models for extraction, summarization, and recommendations only where they add value.

1. Enterprise Connector Fabric

Target

Create a connector layer for SaaS systems, databases, BI tools, and local or network files. MCP should be the preferred external integration protocol where it fits, but every connector must be wrapped by Cortex policy and audit.

Implementation Options

Option	Fit	Pros	Cons	Recommendation
MCP-first connector wrapper	Tool-like SaaS and local integrations	Matches existing `mcp` and `mcp-http` packages; standard tool/resource discovery; good for interactive actions	MCP does not fully solve Cortex-specific identity, source freshness, approval, and audit	Use as default external connector contract, wrapped by Cortex policy
Native Cortex connector plugins	Core systems like local files, Postgres, SQL Server, Microsoft 365, Google Workspace	Full control over sync, identity, health, freshness, and source records	More connector code to maintain	Use for first-party/core connectors
ELT connector layer such as Airbyte	High-volume sync into local Postgres or warehouse	Broad connector coverage and mature data movement	Less suitable for live action tools or user-level approvals	Optional for read-heavy replication later
Prompt-only background jobs	Quick experiments	Uses existing `background` plugin	Weak contracts, weak retries, hard to audit	Prototype only; do not use for governed connectors

Recommended Architecture

Add a connector-fabric plugin that registers:

ConnectorRegistry service for connector definitions, instances, grants, sync cursors, and health.
connector_action tool for listing connectors, testing health, configuring sync cadence, and inspecting grants.
toolcall hook that enforces connector policy before a connector-backed tool executes.
toolresult hook that audits connector calls, records duration/status, and redacts sensitive output fields.

MCP tools should be proxied through this service rather than being exposed directly to the model. The wrapper should attach connector metadata to every tool:

type ConnectorToolBinding = {
  connectorInstanceId: string;
  toolName: string;
  capability: "read" | "write" | "admin";
  sourceTypes: string[];
  sensitivity: "public" | "internal" | "confidential" | "restricted";
  approvalPolicyId?: string;
};

Data Model

Start with typed stores:

connector_definitions
connector_instances
connector_grants
connector_sync_cursors
connector_health
connector_audit_events

Promote connector_audit_events to Postgres when volume grows.

type ConnectorInstance = {
  id: string;
  version: string;
  type: string;
  workspaceId: string;
  displayName: string;
  ownerPrincipalId: string;
  authMode: "none" | "api_key" | "oauth_user" | "oauth_service" | "windows";
  credentialRef?: string;
  scopes: string[];
  readEnabled: boolean;
  writeEnabled: boolean;
  syncCadence?: string;
  lastSyncAt?: string;
  nextSyncAt?: string;
  createdAt: string;
  updatedAt: string;
};

type ConnectorGrant = {
  id: string;
  version: string;
  connectorInstanceId: string;
  principalId: string;
  effectiveUserId?: string;
  scopes: string[];
  allowedTools: string[];
  deniedTools: string[];
  sensitiveFields: string[];
  approvalRules: string[];
  expiresAt?: string;
  createdAt: string;
  updatedAt: string;
};

Tasklist

MVP note: the implemented MCP adapter binds mcp_action and the delegated mcp__* tool-name prefix to connector policy and audit. Generated stable connector_<instance>_<tool> proxy tools remain a future hardening task once per-server MCP trust metadata exists. The Postgres read-only connector is represented as a connector definition/instance for policy continuity; executable read-only SQL support lands in the Structured Data Reasoning MVP.

Technical Notes

Use Vault for connector credentials. Store only credential references in connector records.
Use currentPrincipal() only as attribution input. The policy service decides access.
Add a standard connector error shape so models and UI can distinguish auth failure, permission denial, rate limiting, stale data, and source outage.
Do not allow raw MCP tool descriptions to become trusted policy metadata. Treat them as untrusted unless a connector owner has approved the server.
Keep WebUI bound to localhost until remote auth exists; connector tools make local HTTP endpoints more powerful.

2. Source Registry

Target

Every indexed source should have a first-class record with freshness, trust, permissions, sensitivity, citation policy, health, limitations, and provenance.

Implementation Options

Option	Fit	Pros	Cons	Recommendation
Store-backed source registry	MVP and small local installs	Fits current Matbot stores; fast to add; easy to test	Queries across many sources may become slow	Start here
Postgres source tables	Large source counts, dashboards, health queries	Relational constraints, indexing, reporting	Requires migrations and operational care	Promote once schemas stabilize
OpenLineage/PROV-compatible event export	Audit, data lineage, interoperability	Proven vocabulary for datasets/jobs/runs/entities/activities/agents	Not enough by itself for Cortex UX metadata	Align internal model with these concepts

Recommended Architecture

Add a source-registry plugin that registers a SourceRegistry service and a source_action tool. Existing systems should call the service when ingesting, retrieving, citing, or auditing:

workspace-rag creates or updates source records for markdown files and knowledge entries.
file-index and file-broker report local source metadata and access mode.
Connector adapters create records for SaaS objects, tables, dashboards, and files.
Structured data reasoning creates records for query results and result snapshots.
Workflow runs reference source ids and source versions in evidence requirements.

Data Model

type SourceRecord = {
  id: string;
  version: string;
  workspaceId: string;
  connectorInstanceId?: string;
  connectorType: string;
  externalId: string;
  uri: string;
  title: string;
  ownerPrincipalId?: string;
  businessDomain?: string;
  sourceKind: "document" | "table" | "metric" | "dashboard" | "message" | "ticket" | "artifact" | "query_result";
  schemaOrDocumentType?: string;
  sensitivity: "public" | "internal" | "confidential" | "restricted";
  permissionState: "unknown" | "allowed" | "denied" | "partial";
  effectiveUserId?: string;
  trustLevel: "unknown" | "low" | "medium" | "high";
  freshnessSlaSeconds?: number;
  lastObservedAt?: string;
  lastSuccessfulReadAt?: string;
  staleAfter?: string;
  stalenessState: "unknown" | "fresh" | "stale" | "expired";
  citationPolicy: "cite_path" | "cite_link" | "cite_query" | "do_not_cite";
  retentionPolicyId?: string;
  healthState: "unknown" | "healthy" | "degraded" | "down";
  knownLimitations: string[];
  createdAt: string;
  updatedAt: string;
};

type SourceVersion = {
  id: string;
  version: string;
  sourceId: string;
  contentHash?: string;
  schemaHash?: string;
  observedAt: string;
  validFrom?: string;
  validTo?: string;
  provenance: {
    activityId: string;
    connectorAuditEventId?: string;
    ingestionRunId?: string;
    modelProvider?: string;
  };
};

Tasklist

Technical Notes

Model source provenance with the W3C PROV shape: entities are sources and artifacts, activities are ingestion/retrieval/query/workflow runs, and agents are users, connectors, models, or workflow runners.
Model data lineage with OpenLineage-compatible concepts where helpful: jobs, runs, inputs, outputs, datasets, and extensible facets.
Add OpenTelemetry-style trace ids to audit and source events so tool calls, workflow steps, and source reads can be correlated later.
Do not block retrieval only because a source has unknown freshness during early rollout. Warn first, then make strict freshness policies opt-in per workflow or source class.

3. Context Graph

Target

Add a business context graph above vector search so Cortex can reason over entities, relationships, time validity, provenance, and confidence.

Implementation Options

Option	Fit	Pros	Cons	Recommendation
Neo4j graph storage	Multi-hop graph queries, entity maps, GraphRAG	Already in Docker stack; graph-native traversal; mature ecosystem	Adds driver/schema work and backup considerations	Use for graph projection and traversal
Postgres edge tables	Simple MVP and SQL reporting	Low operational overhead; same database as RAG	Graph traversal is less natural	Use as canonical assertion log if needed
Store-backed graph docs	Very small MVP	Fast to build	Hard to query and dedupe	Only for early prototypes
Microsoft GraphRAG-style batch pipeline	Narrative corpora and global summaries	Proven approach for private corpora	More expensive ingestion; not ideal for live source updates	Use later for large document collections

Recommended Architecture

Build a context-graph plugin with two layers:

Canonical assertions: append-only relationship assertions with source ids, source versions, confidence, extraction method, and validity windows.
Query projection: Neo4j nodes/relationships for graph traversal and GraphRAG-style retrieval.

Do not make extracted graph facts silently authoritative. Every entity and edge must carry provenance and confidence.

Data Model

type ContextEntity = {
  id: string;
  version: string;
  workspaceId: string;
  type: "person" | "team" | "customer" | "vendor" | "system" | "process" | "metric" | "ticket" | "decision" | "document" | "contract" | "task";
  canonicalName: string;
  aliases: string[];
  identifiers: Record<string, string>;
  sensitivity: "public" | "internal" | "confidential" | "restricted";
  createdAt: string;
  updatedAt: string;
};

type ContextRelationshipAssertion = {
  id: string;
  version: string;
  workspaceId: string;
  subjectEntityId: string;
  predicate: string;
  objectEntityId: string;
  validFrom?: string;
  validTo?: string;
  confidence: number;
  extractionMethod: "deterministic" | "connector_metadata" | "model_extracted" | "user_confirmed";
  sourceId: string;
  sourceVersionId?: string;
  evidenceSpan?: { start?: number; end?: number; text?: string };
  createdAt: string;
  updatedAt: string;
};

Tasklist

Technical Notes

Prefer deterministic extraction first. Use model extraction only for relationship candidates that cannot be derived reliably from source metadata.
Store time validity separately from ingestion time. "Customer owns contract" may be true only over a validity interval; ingestion time only says when Cortex observed the claim.
Use graph traversal budgets: max depth, max nodes, max relationships, and max sources. GraphRAG can become expensive without hard limits.
Keep graph facts explainable. Every edge returned to the model should be renderable as "relationship, confidence, source, observed at, limitation".

MVP note: this slice uses Matbot stores as the canonical graph and writes a durable, idempotent Neo4j projection operation log. It does not open a live Neo4j driver connection yet; replay/apply workers, model extraction, correction UI, and richer dictionary management remain future work.

4. Structured Data Reasoning

Target

Create a safe structured data path for tables, metrics, joins, SQL preview, approval, result citation, and trend/anomaly analysis. The goal is not arbitrary SQL generation against raw schemas; it is governed metric and table reasoning.

Implementation Options

Option	Fit	Pros	Cons	Recommendation
Direct read-only SQL connector	Early database prototype	Simple; matches existing `pg` dependency in workspace RAG	Unsafe without semantic layer and validation	Use only behind preview, validation, and row caps
Cortex semantic model	Business metrics and joins	Product-specific governance; model-provider neutral	New modeling work	Build MVP
Import dbt/Cube models	Teams already using semantic layers	Proven shape for metrics/entities/dimensions/joins/access rules	Requires mapping external metadata	Add import path after MVP
DuckDB for local CSV/spreadsheets	Local file analytics	Strong local-first fit	Adds dependency and separate SQL dialect	Consider after Postgres MVP

Recommended Architecture

Add a structured-data plugin with:

DataCatalog service for connections, tables, columns, metrics, dimensions, joins, and access constraints.
SqlPlanner service that generates a query plan from approved semantic objects, not raw arbitrary schemas.
structured_data_action tool for catalog inspection, query preview, approved execution, and result citation.

Use Postgres as the first connector because the stack already includes Postgres and the project already uses the pg package. Add SQL Server, Snowflake, BigQuery, and local CSV/spreadsheet paths later.

Data Model

type DataConnection = {
  id: string;
  version: string;
  workspaceId: string;
  connectorInstanceId: string;
  dialect: "postgres" | "sqlserver" | "snowflake" | "bigquery" | "duckdb";
  readOnly: boolean;
  credentialRef: string;
  defaultSchema?: string;
  rowLimitDefault: number;
  timeoutMsDefault: number;
  createdAt: string;
  updatedAt: string;
};

type MetricDefinition = {
  id: string;
  version: string;
  workspaceId: string;
  name: string;
  businessName: string;
  description?: string;
  baseTableId: string;
  expression: string;
  aggregation: "sum" | "avg" | "count" | "count_distinct" | "min" | "max" | "ratio" | "custom";
  allowedDimensions: string[];
  allowedFilters: string[];
  ownerPrincipalId?: string;
  sourceId?: string;
  createdAt: string;
  updatedAt: string;
};

type QueryRun = {
  id: string;
  version: string;
  workspaceId: string;
  dataConnectionId: string;
  principalId: string;
  status: "planned" | "approved" | "running" | "succeeded" | "failed" | "cancelled";
  sql: string;
  sqlHash: string;
  semanticInputs: string[];
  rowCount?: number;
  executedAt?: string;
  resultSourceId?: string;
  error?: string;
  createdAt: string;
  updatedAt: string;
};

Tasklist

MVP note: this slice uses a deterministic TypeScript SQL scanner and semantic planner instead of adding SQLGlot or a new SQL parser dependency. Live Postgres execution is implemented behind read-only transactions, statement timeout, row caps, connector policy, and approval tokens. Live database introspection, foreign-key/join planning, cost/freshness display in the SQL UI, and anomaly/trend analysis remain future hardening tasks.

Technical Notes

Use database-native row-level security where available. For Postgres, row-level security policies can provide defense in depth, but Cortex still needs application-level policy for tool exposure, query preview, and audit.
SQLGlot is useful for validation and dialect-aware parsing. Because Cortex is TypeScript-first, introduce it either through a small local Python service or choose a TypeScript parser for the MVP and leave SQLGlot as a later hardening option.
Do not let the model issue arbitrary SQL. The model can propose intent; deterministic planner code should produce SQL from semantic inputs.
Query summaries should cite the data connection, tables/metrics, SQL hash, execution timestamp, row count, and source freshness state.

5. Workflow And Governance Layer

Target

Represent automation as durable workflow definitions and workflow runs with typed inputs, triggers, sources, tools, evidence requirements, risk, approvals, dry-run behavior, tests, execution history, and success metrics.

Implementation Options

Option	Fit	Pros	Cons	Recommendation
Extend `background` plugin	Simple scheduled prompts	Already exists	Prompt-oriented, weak step ledger, weak approvals	Use only as scheduler adapter
Cortex local workflow runner	Local-first governed workflows	Fits product architecture; direct access to stores, tools, approvals, WebUI	New runner code	Build MVP
Temporal	Long-running distributed workflows	Proven durable execution and worker model	More infrastructure; larger conceptual shift	Consider after local runner proves workflow schema
BPMN engine	Process-heavy enterprises	Standard process modeling	Heavy for current local workbench	Not first choice

Recommended Architecture

Add a workflow-governance plugin that registers:

WorkflowRegistry service for definitions, versions, tests, and policies.
WorkflowRunner service for typed run execution.
workflow_action tool for draft, validate, dry-run, run, approve, label, list, and inspect.
workflow_policy hook that blocks tool calls outside the active workflow's allowed tool/source/risk envelope.
workflow_audit event writer for every transition.

The workflow runner must be separate from freeform chat. A chat can compile a workflow draft, but a workflow run is a typed state machine with explicit events.

Data Model

type WorkflowDefinition = {
  id: string;
  version: string;
  workspaceId: string;
  name: string;
  ownerPrincipalId: string;
  description?: string;
  inputSchema: Record<string, unknown>;
  triggerSchema?: Record<string, unknown>;
  allowedSourceIds: string[];
  allowedConnectorInstanceIds: string[];
  allowedTools: string[];
  requiredEvidence: Array<{
    name: string;
    sourceKind?: string;
    freshnessSlaSeconds?: number;
    minCitations?: number;
  }>;
  riskLevel: "low" | "medium" | "high" | "critical";
  approvalGates: ApprovalGate[];
  dryRunDefault: boolean;
  tests: WorkflowEvalCase[];
  successMetrics: string[];
  createdAt: string;
  updatedAt: string;
};

type WorkflowRun = {
  id: string;
  version: string;
  workflowId: string;
  workflowVersion: string;
  workspaceId: string;
  principalId: string;
  mode: "dry_run" | "shadow" | "approval_gated" | "execute";
  status: "created" | "running" | "waiting_for_approval" | "succeeded" | "failed" | "cancelled";
  inputs: Record<string, unknown>;
  evidenceSourceIds: string[];
  proposedActions: ActionProposal[];
  executedActions: ExecutedAction[];
  labels: string[];
  startedAt?: string;
  finishedAt?: string;
  createdAt: string;
  updatedAt: string;
};

type WorkflowRunEvent = {
  id: string;
  version: string;
  runId: string;
  sequence: number;
  eventType: string;
  timestamp: string;
  principalId?: string;
  toolCallId?: string;
  sourceIds?: string[];
  payload: Record<string, unknown>;
};

Tasklist

MVP note: this slice implements the workflow run ledger, schema validation, approval records, source evidence references, dry-run/shadow semantics, and workflow-scoped connector policy. It does not yet execute arbitrary multi-step workflow plans or call models inside workflow steps; those remain automation execution hardening work after the compiler MVP.

Technical Notes

Represent workflow execution as event-sourced state, even if the initial store is simple. The run record is the current projection; workflow_run_events is the audit trail.
Use the existing background plugin only to trigger a workflow run on a schedule. The scheduled prompt should become a thin call to workflow_action.start, not the workflow logic itself.
For local-first MVP, a single-process runner is acceptable if every transition is persisted before and after external calls. If users need crash-proof, multi-worker execution, map the workflow schema to Temporal activities later.
Evaluation should be part of the workflow definition. Start with deterministic assertions over output shape, cited source count, stale-source handling, and blocked unsafe actions. Add RAG-specific metrics such as context precision when retrieval quality becomes a release gate.
Policy implementation can begin with TypeScript code. If policies become customer-authored or complex, evaluate OPA/Rego or Cedar-style policy syntax. For relationship-heavy enterprise permissions, evaluate a Zanzibar-inspired service such as SpiceDB later.

6. Automation Shadow Mode MVP

Target

Use the workflow run ledger to compare Cortex recommendations with later human labels before enabling unattended execution. Shadow mode should preserve the exact recommendation, evidence, and proposed action metadata that was evaluated.

Tasklist

MVP note: this slice keeps shadow mode deterministic and store-backed. It does not execute automation actions or generate model explanations for mismatches; those remain future automation execution and review-surface work.

7. Workflow Compiler MVP

Target

Convert a selected chat, investigation, or repeated tool-use pattern into a typed workflow draft with inputs, evidence, allowed tools, approval gates, tests, versioning, and optional dry-run.

Tasklist

MVP note: this slice is deterministic and review-first. It does not call a model to invent hidden workflow behavior; callers provide selected transcript text, source ids, input hints, and tool calls, and the compiler produces a validated workflow definition plus an optional dry-run. Richer model-assisted inference, UI review, schedules, and editable diffs remain future work.

8. Enterprise Expert Panel Upgrade

Target

Expand the existing expert panel into a governed review layer for decisions, workflows, alerts, and investigations, with business-role experts and durable structured review artifacts.

Tasklist

MVP note: this slice keeps structured review extraction deterministic over the expert answers and citations already produced by the panel. It now includes WebUI review cards for durable review records, but reviewer assignment, manual status editing, and direct source-registry citation resolution remain future review-surface work.

Suggested Build Sequence

Current implementation status: Source Registry MVP, Connector Fabric MVP, Source Health Monitor primitives, Structured Data Reasoning MVP, Workflow Run Ledger, Automation Shadow Mode MVP, Context Graph MVP, and Workflow Compiler MVP are complete, and the Enterprise Expert Panel upgrade is now complete. All current build-sequence items in this document have been implemented as committable slices.

Source Registry MVP.
- Build the source model first because connectors, graph, SQL, dossiers, health, and workflows all depend on stable source ids.
Connector Fabric MVP. Complete.
- Wrap existing local file, workspace RAG, MCP, and Postgres read-only paths.
- Add grant enforcement and audit hooks.
Source Health Monitor primitives. Complete.
- Add source health states, stale warnings, and connector health checks.
Structured Data Reasoning MVP. Complete.
- Postgres read-only connector, catalog, semantic metric definitions, SQL preview, approval, execution, and citations.
Workflow Run Ledger. Complete.
- Definition schema, run state, run events, dry-run/proposed actions, approval gates, and tool policy hook.
Automation Shadow Mode MVP. Complete.
- Use the workflow ledger to compare recommendations with human labels.
Context Graph MVP. Complete.
- Entity and relationship extraction from source records and workspace RAG, Neo4j projection, graph retrieval with source permission filtering.
Workflow Compiler MVP. Complete.
- Compile selected chat/investigation evidence into workflow definitions and tests.
Enterprise Expert Panel upgrade. Complete.
- Structured expert outputs linked to dossiers, workflows, and approvals.
Evaluation, Observability, and ROI. Complete.

End-to-end trace lineage across agents, models, tools, retrieval, policy, evaluation, and workflows.
Redacted inspection and side-effect-free replay; versioned regression suites with deterministic and model-based scorers.
Retrieval, citation, action, policy, cost, latency, workflow completion, approval, escalation, time-saved, benefit, and ROI metrics.
Sponsor-facing WebUI evidence and a CI-oriented evaluation command.

Cross-Cutting Implementation Notes

Storage

Use Matbot typed stores for MVP records to match existing plugin patterns.
Use Postgres for:
- high-volume audit events;
- source health dashboards;
- workflow run events at scale;
- structured data catalog and query runs;
- context graph canonical assertions if store queries become limiting.
Keep pgvector for vector search. Do not introduce another vector database until Postgres/pgvector is proven insufficient.

Audit And Provenance

Add a small shared audit-log service or make it part of source-registry. Minimum event fields:

type CortexAuditEvent = {
  id: string;
  version: string;
  timestamp: string;
  traceId?: string;
  workspaceId: string;
  principalId?: string;
  eventType: string;
  sourceIds?: string[];
  connectorInstanceId?: string;
  workflowRunId?: string;
  toolCallId?: string;
  providerName?: string;
  payloadHash?: string;
  redactedPayload?: Record<string, unknown>;
};

The evaluation-observability plugin now supplies this shared event path through the Observability service. The runner creates root trace lineage and emits agent/model/tool spans; retrieval, connector policy, workflow governance, and evaluation components append domain spans and events. Sensitive attributes are redacted before persistence, and sink failures never fail the production run.

Trace replay is deliberately side-effect-free: it reconstructs the persisted event sequence and reports that writes were not executed. A regression suite may also target static fixtures, stored traces, or workflow dry runs. Live write re-execution requires a separately approved workflow and is not part of replay.

Permissions

Avoid a single all-powerful service identity for user-facing operations.
Store effective user/service account per connector call.
Enforce policy before retrieval and before graph expansion, not only before final answer generation.
Add warning states before hard blocking, then make strict enforcement opt-in for high-risk workflows and sensitive sources.

Testing

Node tests:
- typed stores and state transitions;
- policy decisions;
- source id and citation stability;
- SQL validation;
- graph dedupe and permissions;
- workflow event ordering.
Playwright tests:
- source panels and stale warnings;
- connector health display;
- SQL preview and approval;
- workflow approval queue;
- graph entity panels;
- expert review cards;
- shadow-mode labeling.
No tests should require live provider tokens by default. Use fake Matbot harness patterns already present in the WebUI test suite.

Definition Of Done For Each Strategic Architecture Point

It has typed records with versions and stable ids.
It records source ids and source versions for material evidence.
It enforces effective principal and connector grants.
It records audit events for reads, retrievals, tool calls, approvals, writes, generated artifacts, and workflow outcomes.
It has stale-source and unhealthy-source behavior.
It has Node tests for backend behavior and Playwright tests for user-visible workflows.
It can run locally with the existing Cortex launch path.

FilesExpand file tree

strategic_architecture.md

Latest commit

History

strategic_architecture.md

File metadata and controls

Cortex Strategic Architecture Implementation Notes

Evidence Base

Implementation Principles

1. Enterprise Connector Fabric

Target

Implementation Options

Recommended Architecture

Data Model

Tasklist

Technical Notes

2. Source Registry

Target

Implementation Options

Recommended Architecture

Data Model

Tasklist

Technical Notes

3. Context Graph

Target

Implementation Options

Recommended Architecture

Data Model

Tasklist

Technical Notes

4. Structured Data Reasoning

Target

Implementation Options

Recommended Architecture

Data Model

Tasklist

Technical Notes

5. Workflow And Governance Layer

Target

Implementation Options

Recommended Architecture

Data Model

Tasklist

Technical Notes

6. Automation Shadow Mode MVP

Target

Tasklist

7. Workflow Compiler MVP

Target

Tasklist

8. Enterprise Expert Panel Upgrade

Target

Tasklist

Suggested Build Sequence

Cross-Cutting Implementation Notes

Storage

Audit And Provenance

Permissions

Testing

Definition Of Done For Each Strategic Architecture Point