diff --git a/CLAUDE.md b/CLAUDE.md index bd87a7146..80ca34031 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -180,7 +180,7 @@ mcp-data-platform/ │ ├── connid/ # Connection identity: the instance a connection is stored under, the name a call binds it by, the toolkit serving it, and which half of the config owns it — one Resolver, distinct types │ ├── connview/ # Builds the list_connections view (configured + discovered) │ ├── contenttype/ # Media-type detection and normalization for every content write path -│ ├── database/ # Database utilities (migrate/ = golang-migrate runner + 118 embedded SQL migrations) +│ ├── database/ # Database utilities (migrate/ = golang-migrate runner + 119 embedded SQL migrations) │ ├── embedding/ # Text embedding generation for memory vector search │ ├── indexjobs/ # Postgres-backed, source-kind-agnostic background indexer │ ├── knowledge/ # Unified read path for platform knowledge (federation/ = live toolkit registry adapter) @@ -224,7 +224,7 @@ mcp-data-platform/ ├── internal/ # Non-exported implementation (not part of the supported library surface) │ ├── admin/ # Admin-API seams built only by pkg/admin: auditapi/ (events + metrics), callapi/ (the call catalog + its review actions), catalogapi/ (OpenAPI spec bundles + embedding jobs), connoauthapi/ (connection OAuth, unified + legacy per-kind), notifyapi/ (notification delivery history + status counts), settingsapi/ (SMTP + review-queue-alert settings REST) — extracted by #1078 │ ├── httpjson/ # RFC 9457 Problem Details responder + admin list-query param parsing, shared by the admin/portal decomposition seams (#1078) -│ ├── httpserver/ # HTTP composition root: mux/route assembly (MCP streamable+SSE, OAuth, admin/portal/resources/gateway/observability REST, portal UI), CORS, drain/shutdown sequencing — extracted from main.go (#895). Subpackages are the adapters it mounts: accessgate/, attachhttp/, datahubapi/, gatewayhttp/, health/, httpauth/, mentionhttp/, notifyhttp/ (self-scoped notification prefs), scripthttp/ (managed-script review + the approval action), sources/, unsubhttp/ (no-login unsubscribe + its tokens), versionhttp/ (#1076, #1080) +│ ├── httpserver/ # HTTP composition root: mux/route assembly (MCP streamable+SSE, OAuth, admin/portal/resources/gateway/observability REST, portal UI), CORS, drain/shutdown sequencing — extracted from main.go (#895). Subpackages are the adapters it mounts: accessgate/, attachhttp/, datahubapi/, gatewayhttp/, health/, httpauth/, mentionhttp/, notifyhttp/ (self-scoped notification prefs), scripthttp/ (managed-script admin + portal routes, including the administrator's owner transfer), sources/, unsubhttp/ (no-login unsubscribe + its tokens), versionhttp/ (#1076, #1080) │ ├── sqltables/ # The one lexical extractor of the tables a SQL statement reads (enrichment + call targets) │ ├── pglisten/ # Shared LISTEN adapter: one goroutine per pg_notify channel waking the workers registered on it (notification delivery, managed-script runs) │ ├── notification/ # Notification delivery layers built only by internal/platform/notifydelivery, extracted by #1080: notifyprefs/ (preference persistence), notifyqueue/ (queue persistence + LISTEN wakeup), notifyrender/ (branded templates), notifysend/ (SMTP transport), notifyworker/ (send worker) @@ -410,9 +410,12 @@ calls: Authoring needs no configuration and is available wherever there is a database. A saved script runs: `run_script` and a schedule execute the latest saved version, presenting the roles its author held at the save, and the persona -filter authorizes every call at run time. The knobs are how long the record of -a run is kept, whether this replica executes runs at all, and which bucket -destinations a script's output may be delivered to. +filter authorizes every call at run time. A script is personal: its owner sees +it, edits it, runs it, and schedules it, administrators do all four on every +script, and an administrator can move a script to another owner (which +re-captures the run identity from the administrator making the move). The knobs +are how long the record of a run is kept, whether this replica executes runs at +all, and which bucket destinations a script's output may be delivered to. ```yaml scripts: diff --git a/dev/seed.sql b/dev/seed.sql index d44255fd9..bd6bc8124 100644 --- a/dev/seed.sql +++ b/dev/seed.sql @@ -1845,7 +1845,7 @@ DELETE FROM script_versions WHERE script_id IN ( INSERT INTO scripts ( id, name, display_name, description, source_code, params, - scope, personas, owner_email, tags, enabled, status, version, created_at, updated_at + owner_email, tags, enabled, status, version, created_at, updated_at ) VALUES ( 'e1e1e1e1-0000-4000-8000-000000000001', @@ -1853,7 +1853,7 @@ INSERT INTO scripts ( 'Yesterday''s sales by region, exported for the morning review.', E'rows = platform.query(\n connection="acme",\n sql="SELECT region, sum(amount) AS revenue FROM warehouse.public.sales WHERE sale_date = :d GROUP BY region",\n params={"d": run.params["report_date"]},\n)["rows"]\n\nplatform.export(name="daily-sales", rows=rows, format="csv")\nprint("wrote %d regions for %s" % (len(rows), run.params["report_date"]))\n', '[{"name":"report_date","type":"date","description":"The business date to report on; the schedule pins it to the fire time.","required":true}]'::jsonb, - 'global', '{}', 'analyst@example.com', '{sales,reporting}', true, 'active', 2, + 'analyst@example.com', '{sales,reporting}', true, 'active', 2, NOW() - interval '40 days', NOW() - interval '30 days' ), ( @@ -1862,7 +1862,7 @@ INSERT INTO scripts ( 'Accounts with no orders since a cutoff date, for the retention review.', E'rows = platform.query(\n connection="acme",\n sql="SELECT account_id, last_order_at FROM warehouse.public.accounts WHERE last_order_at < :cutoff",\n params={"cutoff": run.params["cutoff"]},\n)["rows"]\n\nplatform.export(name="dormant-accounts", rows=rows, format="csv")\n', '[{"name":"cutoff","type":"date","description":"Accounts idle since this date.","required":true}]'::jsonb, - 'personal', '{}', 'analyst@example.com', '{retention}', true, 'active', 1, + 'analyst@example.com', '{retention}', true, 'active', 1, NOW() - interval '3 days', NOW() - interval '3 days' ), ( @@ -1871,13 +1871,13 @@ INSERT INTO scripts ( 'Row counts and max load timestamps per warehouse table.', E'rows = platform.query(\n connection="acme",\n sql="SELECT table_name, row_count, max_loaded_at FROM warehouse.public.table_stats",\n)["rows"]\n\nplatform.export(name="freshness", rows=rows, format="csv")\n', '[]'::jsonb, - 'global', '{}', 'admin@example.com', '{operations}', true, 'active', 5, + 'admin@example.com', '{operations}', true, 'active', 5, NOW() - interval '60 days', NOW() - interval '21 days' ) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, display_name = EXCLUDED.display_name, description = EXCLUDED.description, source_code = EXCLUDED.source_code, - params = EXCLUDED.params, scope = EXCLUDED.scope, personas = EXCLUDED.personas, + params = EXCLUDED.params, owner_email = EXCLUDED.owner_email, tags = EXCLUDED.tags, enabled = EXCLUDED.enabled, status = EXCLUDED.status, version = EXCLUDED.version, updated_at = EXCLUDED.updated_at; diff --git a/docs/images/screenshots/dark/admin-admin-script-detail-dark.webp b/docs/images/screenshots/dark/admin-admin-script-detail-dark.webp index fb5e47829..6653520b3 100644 Binary files a/docs/images/screenshots/dark/admin-admin-script-detail-dark.webp and b/docs/images/screenshots/dark/admin-admin-script-detail-dark.webp differ diff --git a/docs/images/screenshots/dark/admin-admin-script-owner-dark.webp b/docs/images/screenshots/dark/admin-admin-script-owner-dark.webp new file mode 100644 index 000000000..47323cf69 Binary files /dev/null and b/docs/images/screenshots/dark/admin-admin-script-owner-dark.webp differ diff --git a/docs/images/screenshots/dark/admin-admin-script-runs-dark.webp b/docs/images/screenshots/dark/admin-admin-script-runs-dark.webp index aedde6ff8..baf1def96 100644 Binary files a/docs/images/screenshots/dark/admin-admin-script-runs-dark.webp and b/docs/images/screenshots/dark/admin-admin-script-runs-dark.webp differ diff --git a/docs/images/screenshots/dark/admin-admin-scripts-dark.webp b/docs/images/screenshots/dark/admin-admin-scripts-dark.webp index b8e78b6cb..ca4a20dd3 100644 Binary files a/docs/images/screenshots/dark/admin-admin-scripts-dark.webp and b/docs/images/screenshots/dark/admin-admin-scripts-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-detail-dark.webp b/docs/images/screenshots/dark/user-script-detail-dark.webp index 7e40f550e..75d778491 100644 Binary files a/docs/images/screenshots/dark/user-script-detail-dark.webp and b/docs/images/screenshots/dark/user-script-detail-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-documentation-dark.webp b/docs/images/screenshots/dark/user-script-documentation-dark.webp index 80668d236..0ed81cbfe 100644 Binary files a/docs/images/screenshots/dark/user-script-documentation-dark.webp and b/docs/images/screenshots/dark/user-script-documentation-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-dry-run-dark.webp b/docs/images/screenshots/dark/user-script-dry-run-dark.webp index 89eb09d16..c454899dd 100644 Binary files a/docs/images/screenshots/dark/user-script-dry-run-dark.webp and b/docs/images/screenshots/dark/user-script-dry-run-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-run-log-dark.webp b/docs/images/screenshots/dark/user-script-run-log-dark.webp index 25ed98683..bb21fdd28 100644 Binary files a/docs/images/screenshots/dark/user-script-run-log-dark.webp and b/docs/images/screenshots/dark/user-script-run-log-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-run-now-dark.webp b/docs/images/screenshots/dark/user-script-run-now-dark.webp index adba9e5e2..fb24515cc 100644 Binary files a/docs/images/screenshots/dark/user-script-run-now-dark.webp and b/docs/images/screenshots/dark/user-script-run-now-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-runs-dark.webp b/docs/images/screenshots/dark/user-script-runs-dark.webp index ac69d9dc4..d62c22516 100644 Binary files a/docs/images/screenshots/dark/user-script-runs-dark.webp and b/docs/images/screenshots/dark/user-script-runs-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-schedule-paused-dark.webp b/docs/images/screenshots/dark/user-script-schedule-paused-dark.webp index 8d3cdf1ec..cec96e636 100644 Binary files a/docs/images/screenshots/dark/user-script-schedule-paused-dark.webp and b/docs/images/screenshots/dark/user-script-schedule-paused-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-source-dark.webp b/docs/images/screenshots/dark/user-script-source-dark.webp index 5a517738a..c5e9a0492 100644 Binary files a/docs/images/screenshots/dark/user-script-source-dark.webp and b/docs/images/screenshots/dark/user-script-source-dark.webp differ diff --git a/docs/images/screenshots/dark/user-script-versions-dark.webp b/docs/images/screenshots/dark/user-script-versions-dark.webp index 4dd42b2a2..b7f542784 100644 Binary files a/docs/images/screenshots/dark/user-script-versions-dark.webp and b/docs/images/screenshots/dark/user-script-versions-dark.webp differ diff --git a/docs/images/screenshots/dark/user-scripts-dark.webp b/docs/images/screenshots/dark/user-scripts-dark.webp index 6812614a3..bb42d5b1e 100644 Binary files a/docs/images/screenshots/dark/user-scripts-dark.webp and b/docs/images/screenshots/dark/user-scripts-dark.webp differ diff --git a/docs/images/screenshots/dark/user-scripts-empty-dark.webp b/docs/images/screenshots/dark/user-scripts-empty-dark.webp index 88afb8994..0a4fc8686 100644 Binary files a/docs/images/screenshots/dark/user-scripts-empty-dark.webp and b/docs/images/screenshots/dark/user-scripts-empty-dark.webp differ diff --git a/docs/images/screenshots/light/admin-admin-script-detail-light.webp b/docs/images/screenshots/light/admin-admin-script-detail-light.webp index 4d00810f7..8e8b23052 100644 Binary files a/docs/images/screenshots/light/admin-admin-script-detail-light.webp and b/docs/images/screenshots/light/admin-admin-script-detail-light.webp differ diff --git a/docs/images/screenshots/light/admin-admin-script-owner-light.webp b/docs/images/screenshots/light/admin-admin-script-owner-light.webp new file mode 100644 index 000000000..a3b1ecdad Binary files /dev/null and b/docs/images/screenshots/light/admin-admin-script-owner-light.webp differ diff --git a/docs/images/screenshots/light/admin-admin-script-runs-light.webp b/docs/images/screenshots/light/admin-admin-script-runs-light.webp index 769ea254d..95c50c050 100644 Binary files a/docs/images/screenshots/light/admin-admin-script-runs-light.webp and b/docs/images/screenshots/light/admin-admin-script-runs-light.webp differ diff --git a/docs/images/screenshots/light/admin-admin-scripts-light.webp b/docs/images/screenshots/light/admin-admin-scripts-light.webp index 5f6d93818..d9ad40935 100644 Binary files a/docs/images/screenshots/light/admin-admin-scripts-light.webp and b/docs/images/screenshots/light/admin-admin-scripts-light.webp differ diff --git a/docs/images/screenshots/light/user-script-detail-light.webp b/docs/images/screenshots/light/user-script-detail-light.webp index 06161930e..29da1bfbe 100644 Binary files a/docs/images/screenshots/light/user-script-detail-light.webp and b/docs/images/screenshots/light/user-script-detail-light.webp differ diff --git a/docs/images/screenshots/light/user-script-documentation-light.webp b/docs/images/screenshots/light/user-script-documentation-light.webp index cc29f29b6..fedd4716d 100644 Binary files a/docs/images/screenshots/light/user-script-documentation-light.webp and b/docs/images/screenshots/light/user-script-documentation-light.webp differ diff --git a/docs/images/screenshots/light/user-script-dry-run-light.webp b/docs/images/screenshots/light/user-script-dry-run-light.webp index 5ba8fe545..2a3b1d3c9 100644 Binary files a/docs/images/screenshots/light/user-script-dry-run-light.webp and b/docs/images/screenshots/light/user-script-dry-run-light.webp differ diff --git a/docs/images/screenshots/light/user-script-run-log-light.webp b/docs/images/screenshots/light/user-script-run-log-light.webp index 7f464f2d8..37209a79c 100644 Binary files a/docs/images/screenshots/light/user-script-run-log-light.webp and b/docs/images/screenshots/light/user-script-run-log-light.webp differ diff --git a/docs/images/screenshots/light/user-script-run-now-light.webp b/docs/images/screenshots/light/user-script-run-now-light.webp index 476fc422e..d1210f36d 100644 Binary files a/docs/images/screenshots/light/user-script-run-now-light.webp and b/docs/images/screenshots/light/user-script-run-now-light.webp differ diff --git a/docs/images/screenshots/light/user-script-runs-light.webp b/docs/images/screenshots/light/user-script-runs-light.webp index 2973306a3..4d9b14f8d 100644 Binary files a/docs/images/screenshots/light/user-script-runs-light.webp and b/docs/images/screenshots/light/user-script-runs-light.webp differ diff --git a/docs/images/screenshots/light/user-script-schedule-paused-light.webp b/docs/images/screenshots/light/user-script-schedule-paused-light.webp index 2d9a1b77e..17b48c063 100644 Binary files a/docs/images/screenshots/light/user-script-schedule-paused-light.webp and b/docs/images/screenshots/light/user-script-schedule-paused-light.webp differ diff --git a/docs/images/screenshots/light/user-script-source-light.webp b/docs/images/screenshots/light/user-script-source-light.webp index beff52f0d..bc0246be7 100644 Binary files a/docs/images/screenshots/light/user-script-source-light.webp and b/docs/images/screenshots/light/user-script-source-light.webp differ diff --git a/docs/images/screenshots/light/user-script-versions-light.webp b/docs/images/screenshots/light/user-script-versions-light.webp index 2ce88ea19..99a7e19d2 100644 Binary files a/docs/images/screenshots/light/user-script-versions-light.webp and b/docs/images/screenshots/light/user-script-versions-light.webp differ diff --git a/docs/images/screenshots/light/user-scripts-empty-light.webp b/docs/images/screenshots/light/user-scripts-empty-light.webp index 388854442..c9e9937f9 100644 Binary files a/docs/images/screenshots/light/user-scripts-empty-light.webp and b/docs/images/screenshots/light/user-scripts-empty-light.webp differ diff --git a/docs/images/screenshots/light/user-scripts-light.webp b/docs/images/screenshots/light/user-scripts-light.webp index 6f4f7dfd2..faec328f9 100644 Binary files a/docs/images/screenshots/light/user-scripts-light.webp and b/docs/images/screenshots/light/user-scripts-light.webp differ diff --git a/docs/llms-full.txt b/docs/llms-full.txt index 7b3f1b940..dcf2fd7d3 100644 --- a/docs/llms-full.txt +++ b/docs/llms-full.txt @@ -1820,7 +1820,7 @@ Captures are owned by the user's email (`memory_records.created_by`), the same k The universal, topology-free discovery entry point (renamed from `knowledge_search` in #645; corpus widened to API endpoints and connections). Call it FIRST. One query fans across every searchable source the caller can access and returns results grouped by source with a coverage summary, so the agent sees the shape of the answer space instead of tunneling into the first tool that comes to mind. Structured catalog navigation stays in `datahub_browse`; the scoped API drill-down stays in `api_list_endpoints`. Registered whenever at least one source is available. -Corpus (everything the persona can access): the technical catalog (DataHub, when configured), the governance vocabulary (DataHub glossary terms, tags, and domains as first-class entities, #1160), context documents (the non-dataset knowledge home that predates knowledge pages, surfaced both by relevance and by the entities they document, the same way knowledge pages are, #692), canonical knowledge pages (the internal-knowledge home for business/domain ontology, stored as markdown in the portal database and searched over their full content), the caller's personal memory (non-knowledge dimensions; captured/remembered knowledge surfaces via the insights provider, so a record is never double-listed), captured insights, the caller's feedback threads (lexical, since threads carry no embedding), saved assets, managed resources (human-uploaded reference material, indexed over their metadata AND a bounded text prefix extracted from the uploaded file, so a data dictionary is found by a column name that appears only inside it; #1012), prompts, managed scripts (found by name, description, tags, and typed parameter contract, never by their source code; #1302), the caller's own recorded calls (each hit carrying its derived outcome and its reuse count, so a proven statement outranks a guess; #1321), the caller's own sessions (matched against the purposes their calls stated and the names of the assets they saved, ranked lexically because a session is derived from the audit log and so has no row to carry an embedding; the two arms use the indexes that do exist — the audit purpose full-text index added by migration 000108, and portal_asset_fts; #1322), API endpoints (aggregated across every API gateway connection, reusing `api_list_endpoints`'s per-connection semantic/hybrid ranking, each gateway applying its own route policy fail-closed), and connections. API endpoints and connections are in the default corpus, not behind an opt-in. Memory, insights, feedback, and assets are per-user, scoped server-side to the caller (memory and insights by email `created_by`/`captured_by`, feedback by author email, assets by `owner_id`) and fail closed when their scope key is absent; the catalog, the governance vocabulary, knowledge pages, prompts, scripts, endpoints, and connections are shared. Managed resources and managed scripts are visibility-scoped rather than per-user: the provider derives the caller's visible (scope, scope_id) set exactly as the `resources/list` middleware does (global to everyone, persona to its members, user to its owner) and passes it into the SQL, so a resource the caller could not list is never ranked. A managed script is scoped the same way from `script.Script.VisibleTo` — global to everyone, persona to a caller BELONGING to it, personal to its owner — applied as a store predicate rather than a filter over the answer, and the ranking additionally skips dead ends (disabled, deprecated, superseded), which is a ranking rule and not an access rule: a caller holding an `mcp:script:` reference to a retired script still fetches its contract, which states plainly that nothing will run it. Persona visibility keys on MEMBERSHIP — `knowledge.Caller.Personas`, resolved from roles by the same resolver the resources middleware uses and bound via `search.Toolkit.SetPersonasForRoles` — never on `Caller.Persona`, the persona the request resolved to: resolution substitutes the configured default persona when a caller's roles match none, so using it would hand every unmatched caller the default persona's material. With no resolver bound the set is empty and the caller sees only global plus their own user-scoped material (fail closed). A resource hit additionally carries an MCP `resource_link` content block with the canonical `mcp://` URI so a client with native resource support can attach the file itself. Knowledge pages are org-shared and editable by personas with `apply_knowledge` access; everyone can read them, and anyone can add feedback (threads). A search never surfaces another user's private records or a route a persona could not invoke; an anonymous caller still sees shared sources but no per-user data. The three topology sources (catalog, connections, endpoints) are additionally narrowed by the caller's persona `connections.allow` rules through the same predicate that authorizes a tool call (#1108), so discovery and authorization cannot drift: a catalog dataset is attributed to a connection through its DataHub platform name and hidden when the persona reaches none of the candidates, while a dataset that maps to no configured connection stays visible rather than being hidden on a guess. `fetch` applies the identical boundary, returning not-found for a denied dataset URN or connection reference so a citation cannot read around what search omitted, and `list_connections` enumerates only granted connections. +Corpus (everything the persona can access): the technical catalog (DataHub, when configured), the governance vocabulary (DataHub glossary terms, tags, and domains as first-class entities, #1160), context documents (the non-dataset knowledge home that predates knowledge pages, surfaced both by relevance and by the entities they document, the same way knowledge pages are, #692), canonical knowledge pages (the internal-knowledge home for business/domain ontology, stored as markdown in the portal database and searched over their full content), the caller's personal memory (non-knowledge dimensions; captured/remembered knowledge surfaces via the insights provider, so a record is never double-listed), captured insights, the caller's feedback threads (lexical, since threads carry no embedding), saved assets, managed resources (human-uploaded reference material, indexed over their metadata AND a bounded text prefix extracted from the uploaded file, so a data dictionary is found by a column name that appears only inside it; #1012), prompts, managed scripts (found by name, description, tags, and typed parameter contract, never by their source code; #1302), the caller's own recorded calls (each hit carrying its derived outcome and its reuse count, so a proven statement outranks a guess; #1321), the caller's own sessions (matched against the purposes their calls stated and the names of the assets they saved, ranked lexically because a session is derived from the audit log and so has no row to carry an embedding; the two arms use the indexes that do exist — the audit purpose full-text index added by migration 000108, and portal_asset_fts; #1322), API endpoints (aggregated across every API gateway connection, reusing `api_list_endpoints`'s per-connection semantic/hybrid ranking, each gateway applying its own route policy fail-closed), and connections. API endpoints and connections are in the default corpus, not behind an opt-in. Memory, insights, feedback, and assets are per-user, scoped server-side to the caller (memory and insights by email `created_by`/`captured_by`, feedback by author email, assets by `owner_id`) and fail closed when their scope key is absent; the catalog, the governance vocabulary, knowledge pages, prompts, endpoints, and connections are shared. Managed scripts are per-user: a script is its owner's, so an unidentified caller reaches none. Managed resources are visibility-scoped rather than per-user: the provider derives the caller's visible (scope, scope_id) set exactly as the `resources/list` middleware does (global to everyone, persona to its members, user to its owner) and passes it into the SQL, so a resource the caller could not list is never ranked. A managed script is scoped the same way from `script.Script.VisibleTo` — global to everyone, persona to a caller BELONGING to it, personal to its owner — applied as a store predicate rather than a filter over the answer, and the ranking additionally skips dead ends (disabled, deprecated, superseded), which is a ranking rule and not an access rule: a caller holding an `mcp:script:` reference to a retired script still fetches its contract, which states plainly that nothing will run it. Persona visibility keys on MEMBERSHIP — `knowledge.Caller.Personas`, resolved from roles by the same resolver the resources middleware uses and bound via `search.Toolkit.SetPersonasForRoles` — never on `Caller.Persona`, the persona the request resolved to: resolution substitutes the configured default persona when a caller's roles match none, so using it would hand every unmatched caller the default persona's material. With no resolver bound the set is empty and the caller sees only global plus their own user-scoped material (fail closed). A resource hit additionally carries an MCP `resource_link` content block with the canonical `mcp://` URI so a client with native resource support can attach the file itself. Knowledge pages are org-shared and editable by personas with `apply_knowledge` access; everyone can read them, and anyone can add feedback (threads). A search never surfaces another user's private records or a route a persona could not invoke; an anonymous caller still sees shared sources but no per-user data. The three topology sources (catalog, connections, endpoints) are additionally narrowed by the caller's persona `connections.allow` rules through the same predicate that authorizes a tool call (#1108), so discovery and authorization cannot drift: a catalog dataset is attributed to a connection through its DataHub platform name and hidden when the persona reaches none of the candidates, while a dataset that maps to no configured connection stays visible rather than being hidden on a guess. `fetch` applies the identical boundary, returning not-found for a denied dataset URN or connection reference so a citation cannot read around what search omitted, and `list_connections` enumerates only granted connections. A query may be text (`intent`), entity-keyed (`entity_urns`), or both. The entity path unions every source linked to those DataHub URNs (the catalog entity, URN-linked insights, and the caller's URN-linked memory), expanded along lineage; per-user scope and the catalog's access rules still apply to each source. @@ -2552,11 +2552,11 @@ Structured feedback from reviewers (including non-agent subject-matter experts a The single home for the Memory to Insight to Knowledge lifecycle (formerly the separate Knowledge Pages, Knowledge & Memory, and admin Knowledge & Memory routes, which now redirect here). A header teaches the model: everything learned is a Memory; a memory others would benefit from becomes an Insight (a proposal awaiting review); whoever holds the `apply_knowledge` capability promotes good insights into Knowledge (business/domain facts become knowledge pages, technical/entity facts go to the DataHub catalog). Three tabs, with review/promote affordances gated on the `apply_knowledge` tool (a capability, not an admin role). Knowledge (default): unified search across every accessible source grouped by source with a coverage summary (the same federation as the `search` tool, over `GET /api/v1/portal/search`); with an empty query, browse of canonical knowledge pages (create/edit/remove for `apply_knowledge` holders; the platform's own built-in pages sit in the same corpus — reconciled from the binary at startup, badged Built-in, read-only where people edit, hidable per deployment with the hide respected across upgrades and reversible via the Knowledge list's Restore built-in / POST /api/v1/portal/knowledge-pages/restore-builtin) in either of two layouts, a card list or an interactive graph of the corpus (#1162) where every page and every entity a page references is a typed node and every stored reference is a directed edge. The graph is exploratory rather than a whole-corpus hairball: it opens on the corpus's strongest bridge and its neighbourhood, with a hops control to widen it and a whole-corpus overview on demand. It is analysed, not merely drawn - Louvain community detection partitions the corpus into clusters (tinted as regions in the overview, with a clustering force separating them, and reported with the partition's modularity) and betweenness centrality scores each node for how much of the graph it bridges, which sets node size and is reported per node with its percentile. Clicking a node opens a side inspector (references in each direction, bridge score and rank, cluster, the reference's URN, selectable neighbour lists) rather than navigating away; its actions are focus, expand, shortest-path tracing between any two nodes, and open. Selecting a catalog node resolves it against the DataHub catalog and reports what is there (description, domain, owners, tags) naming the connection queried, or states plainly that the cited dataset is not in that catalog - a page citing a dataset the catalog does not have is surfaced as the gap it is. Catalog entities are URL-addressable at /knowledge/catalog?urn=..., which is where a catalog reference links from anywhere in the portal. Plus hover neighbourhood highlighting, node drag with pinning, pan/zoom, type and tag filters, and search-as-focus; the graph read is `GET /api/v1/portal/knowledge-pages/graph`, access-filtered so an entity the viewer cannot see has neither node nor edge, and explicitly reporting any node/page cap instead of truncating silently; and for `apply_knowledge` holders the changesets (the record of insights promoted into knowledge, with rollback) since a changeset is created only at apply time and belongs with the promoted knowledge, not the unpromoted insights. Insights: the review pipeline only (insights are the one memory type that crosses between users, and they cross when applied) - your captured insights with status and relevance search, plus for `apply_knowledge` holders the full review queue (approve/reject); a pending-review count is badged on the sidebar Knowledge item and the Insights tab. Memory: personal, scoped to your own records (the cross-user unit is the insight), classified by lifecycle class (`sink_class`: Preference, Event, Business knowledge, Operational rule, Schema/entity). The Knowledge tab also has a Catalog sub-tab (#719/#720/#1156/#1157/#1158/#1194), a first-class route at `/knowledge/catalog`, which holds every DataHub-backed surface in the portal: the rule is that everything under Catalog is DataHub and anything the portal's own database backs (knowledge pages, changesets) stays outside it, which is what keeps the Knowledge sub-tab row at four (Search All, Knowledge Pages, Catalog, Changesets) while the catalog surfaces grow underneath. Catalog's own inner tabs are Tables, Context Docs, Tags, Domains, and Glossary - the described things first, the vocabularies that describe them second. The DataHub connection is picked once for the whole section and applies to every inner tab; changing it returns each tab to its list, since an open table, document, tag, domain, or glossary entity belongs to the connection it was read from. The inner tab is carried in the hash (`/knowledge/catalog#tags`, `#domains`, `#glossary`, `#context-docs`, `#tables`) rather than its own route, so the selection survives a refresh and back/forward without unmounting the container that holds the shared connection; there is no `/knowledge/tags` or `/knowledge/context-docs` (they were removed outright in #1194, with no redirect, since only the tab bar itself produced those URLs). Tables: browse/search the tables the connection catalogs, open one to see description, tags, owners, glossary terms, domain, and columns, and edit each facet inline when the persona grants `datahub_update` and the connection is writable (no table create/delete since tables originate in source systems). Context Docs: browse/search and full create/edit/delete of markdown context documents through a markdown editor, gated on `datahub_create`/`datahub_update`/`datahub_delete`; a document attaches only to Dataset/GlossaryTerm/GlossaryNode/Container. Tags: the tag vocabulary itself rather than one table's tags - list and name-filter a connection's tags, open one to see its description and the tables carrying it (each linking into the Tables entity editor), and create, describe, or retire a tag when the persona grants the matching datahub tool on a writable connection; the delete confirmation states how many tables carry the tag first. A tag description is plain text, not markdown - the one deliberate exception among the Catalog vocabularies (#1200), because DataHub's own tag page renders the field as plain text and formatting authored in the portal would show as raw source everywhere else in the catalog. Domains: the business areas the catalog is grouped into rather than one table's domain - list and name-filter a connection's domains, open one to see its description and the tables in it (each linking into the Tables entity editor), create, describe, or retire a domain, and move tables in and out of it, each gated on the matching datahub tool on a writable connection; the delete confirmation states how many tables are in the domain and that deleting leaves them without one, since it touches no table. A domain description is markdown (#1200): the domain view renders it formatted and the editor is the split source/preview markdown editor the Tables tab and Context Docs use. Glossary (#1158): the business vocabulary itself - a tree, so it is walked one branch at a time (the root shows the nodes and terms with no parent; opening a node shows what is inside it, and a node's browse view IS its detail view, carrying its definition, its attached context documents, and its children on one screen). A term shows its definition, a breadcrumb built from DataHub's parent chain (so it is the same wherever the term was reached from), the context documents attached to it, and the tables annotated with it, each linking into the Tables entity editor and marked when a COLUMN rather than the table carries the term. Create a term or a node, edit either's definition, and retire a term or an EMPTY node, each gated on the matching datahub tool on a writable connection; a new entity lands in the open branch and the form names it. A node that still holds entries is not offered a delete at all, because DataHub takes the node without taking what is inside it - the surface says to empty it first rather than showing a confirmation that cannot state the outcome. Term and node definitions are markdown (#1200), rendered formatted and edited through the same split source/preview markdown editor, so a definition can carry a heading, a list of the cases it includes and excludes, and a worked example; the field itself was never the constraint, since all of these kinds write through the one `PUT catalog/entity/description` route and markdown survives it byte-for-byte. One glossary backs both surfaces: a term defined here is immediately what the Tables tab's glossary picker offers. All five are backed by the portal DataHub REST API at `/api/v1/portal/datahub/{connection}/...` (`GET .../connections` lists connections with a writable flag): reads require DataHub access on the persona; a write requires the matching MCP tool grant AND a write-enabled connection (`read_only: false`), both enforced server-side and recorded in the audit log. Tag and glossary-term edits use batched add/remove sets (the clobber-safe write path, #721/#729). The same REST surface exposes the business glossary as a tree rather than only a flat name search (#1155, requires mcp-datahub v1.15.0): `GET catalog/glossary/roots` returns the nodes and terms with no parent (each paged with its own total, since DataHub pages the two independently), `GET catalog/glossary/children?urn=` returns one page of the nodes and terms directly under a node (DataHub pages a node's children as one mixed collection, so start/count/total describe the combined page rather than either slice), `GET catalog/glossary/parents?urn=` returns the ancestor nodes of a term or node direct-parent-first for a breadcrumb, `POST catalog/glossary/nodes` creates a node from {name, definition, parent_node} and returns its URN (empty parent_node creates it at the root), gated on `datahub_create` plus a write-enabled connection, and #1158 adds the rest of the editor: `POST catalog/glossary/terms` creates a term from the same body through the same handler (a term and a node differ only in which upstream call runs), `DELETE catalog/glossary/entity?urn=` retires either kind through the one route because upstream is one call (`datahub_delete`), and `GET catalog/entity/documents?urn=` returns the context documents attached to one entity - the one document read the corpus-wide browse and search cannot express. Two things the glossary needs are not routes of their own: a definition is edited with `PUT catalog/entity/description` (DataHub stores a glossary entity's text in the glossaryTermInfo/glossaryNodeInfo aspect's `definition` field, and the platform routes the write there by entity type), and the tables a term is applied to come from the catalog search's glossary filters, `GET catalog/search?q=*&glossary_term=` for every annotated table and `&column_glossary_term=` for those where a column carries it - two reads because DataHub's `glossaryTerms` index folds column-level annotations into the table's and only `fieldGlossaryTerms` isolates them, with no table-level-only field. Deleting a node does not delete what is inside it and deleting a term does not remove the term from the tables annotated with it, since upstream DeleteGlossaryEntity touches only the entity named, which is why the portal shows a node's children and a term's usage before offering the delete. Every node read carries `terms_count`/`nodes_count`, DataHub's own tally of its direct children, so a branch renders as expandable without first fetching it. A URN of the wrong kind is a 400 (children hang off a node only; a parent chain exists for either kind) and an unknown node is a 404, not a 502. Children are served from DataHub's asynchronously populated graph index, so a just-created entity may not appear under its parent yet; the parent chain reads the entity itself and is immediately consistent. Tag governance (#1156) adds only two routes, because its reads already exist: `POST catalog/tags` creates a tag from {name, description} and returns the URN DataHub assigned it (201, gated on `datahub_create`), and `DELETE catalog/tags?urn=` retires one (gated on `datahub_delete`); listing and name-filtering tags is `GET catalog/lookup/tags` (the picker's read), the datasets carrying a tag are `GET catalog/search?q=*&tags=` through the catalog search's tag filter, and a tag's description is edited with `PUT catalog/entity/description`, which takes any entity URN. A URN that is not a tag is a 400 before the call reaches DataHub, and a newly created tag is not immediately listable because the list read is served from DataHub's asynchronously populated search index. Domain governance (#1157) adds the same two routes for the same reason: `POST catalog/domains` creates a domain from {name, description} and returns its URN (201, `datahub_create`), `DELETE catalog/domains?urn=` retires one (`datahub_delete`), while listing domains is `GET catalog/lookup/domains`, the tables in a domain are `GET catalog/search?q=*&domain=`, the description edit is `PUT catalog/entity/description`, and membership is edited with `PUT catalog/entity/domain` aimed at the table rather than at the domain. Two limits the surface states rather than hides: the domain list is capped at 100 by DataHub's own `listDomains` query (the lookup route takes no limit), and a table has at most one domain, so adding a table already in another domain moves it. Knowledge pages link to governance entities as first-class references (#1159): the manual-reference picker searches glossary terms, tags, and domains by display name through the existing lookup routes (asking which DataHub connection to search, and offering the three catalog types only when a connection exists), and stores the entity's own URN. A stored governance reference renders with the name the catalog reports rather than the key inside its URN - DataHub generates a UUID key for anything created without an explicit id, so a chip built from the URN alone would read as `8f3c1a94` where the page meant `Net Revenue`. Names are resolved server-side in one batch on the existing refs/resolve path (an optional `CatalogLabeler` over the same DataHub bridge the REST surface uses, wired only when a connection is configured), because a tag and a domain have NO by-URN read upstream and are resolved by listing their vocabulary once per request and matching; only a glossary term has one, `GET catalog/glossary/term?urn=`, added by #1159 and also what opens a cited term in the portal. An unresolved or unreachable name falls back to the URN-derived label rather than failing the page, and resolution is gated on catalog access - the one rule `portal.HasCatalogAccess` now holds for both the DataHub REST surface and the labels (any `datahub_*` tool on the persona, or admin), so a persona denied the Catalog tab does not learn a governance entity's name through the reference list instead. A catalog reference links to the Catalog inner tab that manages that kind of entity (`/knowledge/catalog?urn=...#glossary|#tags|#domains`, everything else `#tables`); each inner tab claims only its own URN kinds, so a stale link opens the list rather than a read that cannot succeed, and going back drops the `?urn=`. A tag, domain, or term the connection does not list says so instead of opening a detail view assembled from the URN alone, and a failed read is reported as a failure rather than as a missing entity. Each governance detail view lists the knowledge pages that reference its entity, through the existing `GET /api/v1/portal/knowledge-pages/backlinks?urn=` reverse lookup keyed on `entity_urn` (no schema change). The MCP side already accepted these refs via ParseCitableRef, so `apply_knowledge` attaches them with no new input. ## Prompts -The organization's prompt library, presented as two buckets (#1010, #1124): My Prompts (every prompt the caller owns at any scope — shared scopes carry a scope badge — plus prompts shared with them, each attributed to its sharer; never another owner's personal prompt) and Library (the approved shared prompts visible to the caller). Both buckets are browsed grouped by collection, each group headed by the collection name, its prompt count, and the collection description beneath it; only search results are flat, holding their relevance order. The scope taxonomy (global/persona/personal) appears only inside the promote and admin flows, never in the user-facing library. Collections are named groups organizing the library by team, domain, or workflow (`prompt_collections` table; a prompt belongs to at most one via `prompts.collection_id`, released to the default General group when its collection is deleted). Any user creates collections; renaming/deleting is creator-or-admin; assignment follows the prompt's own mutation rule (owner for personal, admin for shared) and is organizational metadata: it never versions or triggers review. REST: `GET/POST /api/v1/portal/prompt-collections`, `PUT/DELETE /api/v1/portal/prompt-collections/{id}`, `PUT /api/v1/portal/prompts/{id}/collection` (admin-prefixed equivalents exist); agents assign with `manage_prompt` `collection_id` on create/update (an id from the `collections` array list returns; empty string clears; collections themselves are created in the portal). Facets narrow by collection, tag, status (My Prompts), owner (Library), and usage; rows show run count and last-run age with sorts by name, runs, and last run, and dead prompts carry a badge naming the exact usage condition ("never run" or "unused 60d+"; a prompt created within the last week carries no flag). Search ranks prompts by relevance to a phrase (semantic vector similarity when an embedding provider is configured, keyword fallback otherwise) across the caller's full visibility, best-first; browse mode keeps sortable columns. The same ranking backs the MCP tool `manage_prompt list query=...`. Every enabled prompt is embedded off the request path by the shared index-jobs framework (source_kind `prompts`) regardless of lifecycle status, since visibility is decided at query time; editing a prompt's title, description, body, or tags clears its vector so it re-embeds against the new text. Visibility is applied before ranking, so a prompt you cannot read is never returned, and it is the same rule browsing applies (#1124), so a prompt visible in a list is findable by search: an admin ranks across everything (any owner, any status); a non-admin ranks over approved global prompts, approved matching-persona prompts, and every prompt they own at any scope and status — the publication gate applies to other people's shared prompts, never to your own work. Sortable columns, expandable rows with full content and copy button. Scope badges: Personal, Global, Persona, System. Lifecycle status badge (draft, approved, deprecated, superseded) and comma-separated tags on create/edit. An admin creating a global or persona prompt is its approver: the prompt lands approved with the stamp set (#1124), on both the admin REST API and `manage_prompt`; personal creates stay draft and publish through the promotion flow. Request Promotion on a personal prompt asks an admin to promote it to a chosen persona or to global; the prompt stays personal with a "Promotion requested" badge until an admin approves or rejects it. Share sends a personal prompt directly to another user by email (owner-initiated, no approval): the recipient gets a real runnable prompt, not a markdown snapshot, and it appears in the Prompts page's My Prompts bucket with a shared-by attribution. Personal prompt names are unique per owner; when served over MCP, names carry a scope prefix computed at serve time: `personal-`, `-` (one per persona), `global-`, or `shared-` for prompts shared with the caller, keeping the surface collision-free by construction; every descriptor carries a `title` from display_name. Users never need machine names: agents resolve any handle (stored name, display name, `mcp:prompt:`, or free text) to a ready-to-run prompt with the `manage_prompt` `use` command, which returns rendered content, argument specs, and provenance (including version, approver, and approval time), or ranked candidates when ambiguous. Every database prompt is versioned (#1009): each mutation of content, display name, description, arguments, or tags snapshots an immutable `prompt_versions` row with its author, and approval stamps bind to the specific version approved (approving v5 never alters v4's recorded approval). Editing the content or arguments of an approved global or persona prompt does not change what is served: the edit lands as a pending draft version (manage_prompt update returns status pending_approval; a gated content edit cannot be combined with scope/status changes in one call) and the approved snapshot keeps serving until an admin approves the draft via `POST /api/v1/admin/prompts/{id}/versions/{version}/approve` (reject with `.../reject`; full history with author and content via `GET /api/v1/admin/prompts/{id}/versions`; `GET /api/v1/portal/prompts/{id}/versions` serves history to any caller who can view the prompt (own personal prompts and enabled shared prompts), since history is the library's verification surface; non-admin viewers of a shared prompt get the served history only: applied snapshots in full, the pending draft as a content-redacted stub, and rejected/superseded drafts omitted). The portal prompt page renders this history with per-version approval provenance, flags a pending draft (readers keep being served the approved version), and diffs any version against the current content as a line diff; it also shows point-of-use invocation help (a copyable natural-language invocation built from the stable name and required arguments, resolved by agents via `manage_prompt use`). Metadata-only edits (tags, category, description, display name) apply directly; personal prompts version silently. Served prompts carry provenance: `prompts/get` stamps `prompt_version` / `prompt_approved_by` / `prompt_approved_at` / `prompt_reference` into `_meta`. Usage stats are aggregated from `prompt_serve` audit events (emitted on every database-prompt `prompts/get` and resolved `use`, within the audit retention window): `manage_prompt get` and `list` report `run_count` and `last_run_at` per prompt (`list` also includes the shared `collections` list when the store supports collections, so MCP clients see the same organization model as the portal), and `GET /api/v1/admin/prompts/usage` / `GET /api/v1/portal/prompts/usage` return the per-prompt rollup (portal scoped to the caller's visible prompts, including prompts shared person-to-person with them) for library curation and dead-prompt detection. A prompt also carries the reference material its procedure depends on (#1013): attachments are ordered links from `prompt_resource_attachments` to managed resources, stored by resource id so editing the uploaded file updates every prompt that attaches it, and deliberately without a foreign key so deleting a resource leaves the link visible as broken rather than silently erasing the evidence that the SOP is incomplete. A resolved prompt delivers them after the prompt text: textual material at or below a 64 KiB threshold as an MCP `EmbeddedResource` carrying the contents, anything binary or larger as a `ResourceLink` the client reads on demand, with a framing message stating the material is authoritative (fill an attached template, follow an attached checklist); `manage_prompt use` additionally lists them in an `attachments` array (uri, media type, size, availability, and inline content for embedded items) so the agent can state what it received. An attachment must be at least as widely visible as the prompt that carries it: global resources attach anywhere, a persona resource to personal prompts and to persona prompts scoped to exactly that persona, and a user-scoped resource only to its owner's own personal prompts. The rule is enforced at attach time (`POST /api/v1/portal/prompts/{id}/attachments`, admin-prefixed equivalent exists; `PUT` reorders, `DELETE .../{resourceId}` detaches, `GET /api/v1/portal/resources/{id}/prompts` answers what depends on a resource) and again on every prompt write through the shared store wrapper, so a scope edit or promotion request that would strand an attachment is refused with a message naming the resource, whichever surface it arrives from. At serve time it is re-checked per caller with `resource.CanReadResource`: an unreadable or deleted attachment is reported only as an aggregate count of undelivered materials, never by name and never by contents, and the prompt still serves. A prompt also references the managed scripts its procedure depends on (#1289): ordered links in `prompt_script_attachments` storing the canonical `mcp:script:` reference rather than a bare id, because that reference is the platform's one way to name a script from outside `pkg/script` (the same string `search` emits and `fetch` dereferences), and again deliberately without a foreign key so deleting a script leaves the reference visible as broken. `manage_prompt attach_script` / `detach_script` take `script` as either that reference or a bare script id and normalize to the reference. Serving a prompt delivers each referenced script's contract plus the instruction to call `run_script` for fresh output, and `manage_prompt use` lists them in a `scripts` array; serving NEVER executes a script, because a prompt read is a read path and running code from it would blur audit attribution and turn every read into a potential asset write. The same audience rule attachments follow governs references, over the script's own scope, enforced at attach time, at promotion time, on every prompt write through the shared store wrapper, and once more per caller at serve time, where an unreadable or deleted reference is reported only as an aggregate count with the prompt still serving. The rule itself is one implementation for both kinds (`prompt.CheckAttachScope` over an `AttachmentScope` carrying a SET of audience ids), because a script may serve several personas where a resource serves exactly one, and a single id cannot answer whether the prompt's audience is contained in the material's. In MCP Apps-capable hosts, the built-in `prompt-browser` app (#1011) is bound to the presentation-only `show_prompts` tool: a host renders an app on every call to the tool it is bound to, so the browser lives on a tool whose only job is to open it for the human rather than on `manage_prompt`, which the agent calls throughout its own work. `show_prompts` opens the browser for the human (search, buckets, collection/tag facets, argument forms, preview with provenance, and a Run action that resolves via `use` and inserts the rendered prompt when the host supports `ui/message`); the rendered app populates itself from its own `manage_prompt` calls. `manage_prompt` carries no app and renders nothing, and its JSON results stand alone in non-app clients. +The organization's prompt library, presented as two buckets (#1010, #1124): My Prompts (every prompt the caller owns at any scope — shared scopes carry a scope badge — plus prompts shared with them, each attributed to its sharer; never another owner's personal prompt) and Library (the approved shared prompts visible to the caller). Both buckets are browsed grouped by collection, each group headed by the collection name, its prompt count, and the collection description beneath it; only search results are flat, holding their relevance order. The scope taxonomy (global/persona/personal) appears only inside the promote and admin flows, never in the user-facing library. Collections are named groups organizing the library by team, domain, or workflow (`prompt_collections` table; a prompt belongs to at most one via `prompts.collection_id`, released to the default General group when its collection is deleted). Any user creates collections; renaming/deleting is creator-or-admin; assignment follows the prompt's own mutation rule (owner for personal, admin for shared) and is organizational metadata: it never versions or triggers review. REST: `GET/POST /api/v1/portal/prompt-collections`, `PUT/DELETE /api/v1/portal/prompt-collections/{id}`, `PUT /api/v1/portal/prompts/{id}/collection` (admin-prefixed equivalents exist); agents assign with `manage_prompt` `collection_id` on create/update (an id from the `collections` array list returns; empty string clears; collections themselves are created in the portal). Facets narrow by collection, tag, status (My Prompts), owner (Library), and usage; rows show run count and last-run age with sorts by name, runs, and last run, and dead prompts carry a badge naming the exact usage condition ("never run" or "unused 60d+"; a prompt created within the last week carries no flag). Search ranks prompts by relevance to a phrase (semantic vector similarity when an embedding provider is configured, keyword fallback otherwise) across the caller's full visibility, best-first; browse mode keeps sortable columns. The same ranking backs the MCP tool `manage_prompt list query=...`. Every enabled prompt is embedded off the request path by the shared index-jobs framework (source_kind `prompts`) regardless of lifecycle status, since visibility is decided at query time; editing a prompt's title, description, body, or tags clears its vector so it re-embeds against the new text. Visibility is applied before ranking, so a prompt you cannot read is never returned, and it is the same rule browsing applies (#1124), so a prompt visible in a list is findable by search: an admin ranks across everything (any owner, any status); a non-admin ranks over approved global prompts, approved matching-persona prompts, and every prompt they own at any scope and status — the publication gate applies to other people's shared prompts, never to your own work. Sortable columns, expandable rows with full content and copy button. Scope badges: Personal, Global, Persona, System. Lifecycle status badge (draft, approved, deprecated, superseded) and comma-separated tags on create/edit. An admin creating a global or persona prompt is its approver: the prompt lands approved with the stamp set (#1124), on both the admin REST API and `manage_prompt`; personal creates stay draft and publish through the promotion flow. Request Promotion on a personal prompt asks an admin to promote it to a chosen persona or to global; the prompt stays personal with a "Promotion requested" badge until an admin approves or rejects it. Share sends a personal prompt directly to another user by email (owner-initiated, no approval): the recipient gets a real runnable prompt, not a markdown snapshot, and it appears in the Prompts page's My Prompts bucket with a shared-by attribution. Personal prompt names are unique per owner; when served over MCP, names carry a scope prefix computed at serve time: `personal-`, `-` (one per persona), `global-`, or `shared-` for prompts shared with the caller, keeping the surface collision-free by construction; every descriptor carries a `title` from display_name. Users never need machine names: agents resolve any handle (stored name, display name, `mcp:prompt:`, or free text) to a ready-to-run prompt with the `manage_prompt` `use` command, which returns rendered content, argument specs, and provenance (including version, approver, and approval time), or ranked candidates when ambiguous. Every database prompt is versioned (#1009): each mutation of content, display name, description, arguments, or tags snapshots an immutable `prompt_versions` row with its author, and approval stamps bind to the specific version approved (approving v5 never alters v4's recorded approval). Editing the content or arguments of an approved global or persona prompt does not change what is served: the edit lands as a pending draft version (manage_prompt update returns status pending_approval; a gated content edit cannot be combined with scope/status changes in one call) and the approved snapshot keeps serving until an admin approves the draft via `POST /api/v1/admin/prompts/{id}/versions/{version}/approve` (reject with `.../reject`; full history with author and content via `GET /api/v1/admin/prompts/{id}/versions`; `GET /api/v1/portal/prompts/{id}/versions` serves history to any caller who can view the prompt (own personal prompts and enabled shared prompts), since history is the library's verification surface; non-admin viewers of a shared prompt get the served history only: applied snapshots in full, the pending draft as a content-redacted stub, and rejected/superseded drafts omitted). The portal prompt page renders this history with per-version approval provenance, flags a pending draft (readers keep being served the approved version), and diffs any version against the current content as a line diff; it also shows point-of-use invocation help (a copyable natural-language invocation built from the stable name and required arguments, resolved by agents via `manage_prompt use`). Metadata-only edits (tags, category, description, display name) apply directly; personal prompts version silently. Served prompts carry provenance: `prompts/get` stamps `prompt_version` / `prompt_approved_by` / `prompt_approved_at` / `prompt_reference` into `_meta`. Usage stats are aggregated from `prompt_serve` audit events (emitted on every database-prompt `prompts/get` and resolved `use`, within the audit retention window): `manage_prompt get` and `list` report `run_count` and `last_run_at` per prompt (`list` also includes the shared `collections` list when the store supports collections, so MCP clients see the same organization model as the portal), and `GET /api/v1/admin/prompts/usage` / `GET /api/v1/portal/prompts/usage` return the per-prompt rollup (portal scoped to the caller's visible prompts, including prompts shared person-to-person with them) for library curation and dead-prompt detection. A prompt also carries the reference material its procedure depends on (#1013): attachments are ordered links from `prompt_resource_attachments` to managed resources, stored by resource id so editing the uploaded file updates every prompt that attaches it, and deliberately without a foreign key so deleting a resource leaves the link visible as broken rather than silently erasing the evidence that the SOP is incomplete. A resolved prompt delivers them after the prompt text: textual material at or below a 64 KiB threshold as an MCP `EmbeddedResource` carrying the contents, anything binary or larger as a `ResourceLink` the client reads on demand, with a framing message stating the material is authoritative (fill an attached template, follow an attached checklist); `manage_prompt use` additionally lists them in an `attachments` array (uri, media type, size, availability, and inline content for embedded items) so the agent can state what it received. An attachment must be at least as widely visible as the prompt that carries it: global resources attach anywhere, a persona resource to personal prompts and to persona prompts scoped to exactly that persona, and a user-scoped resource only to its owner's own personal prompts. The rule is enforced at attach time (`POST /api/v1/portal/prompts/{id}/attachments`, admin-prefixed equivalent exists; `PUT` reorders, `DELETE .../{resourceId}` detaches, `GET /api/v1/portal/resources/{id}/prompts` answers what depends on a resource) and again on every prompt write through the shared store wrapper, so a scope edit or promotion request that would strand an attachment is refused with a message naming the resource, whichever surface it arrives from. At serve time it is re-checked per caller with `resource.CanReadResource`: an unreadable or deleted attachment is reported only as an aggregate count of undelivered materials, never by name and never by contents, and the prompt still serves. A prompt also references the managed scripts its procedure depends on (#1289): ordered links in `prompt_script_attachments` storing the canonical `mcp:script:` reference rather than a bare id, because that reference is the platform's one way to name a script from outside `pkg/script` (the same string `search` emits and `fetch` dereferences), and again deliberately without a foreign key so deleting a script leaves the reference visible as broken. `manage_prompt attach_script` / `detach_script` take `script` as either that reference or a bare script id and normalize to the reference. Serving a prompt delivers each referenced script's contract plus the instruction to call `run_script` for fresh output, and `manage_prompt use` lists them in a `scripts` array; serving NEVER executes a script, because a prompt read is a read path and running code from it would blur audit attribution and turn every read into a potential asset write. A reference resolves for the script's owner and for nobody else; referencing is allowed from any prompt and the response names who it will resolve for when the prompt serves anybody else, and only a script the caller can see may be referenced, with the per-caller check applied again at serve time, where an unreadable or deleted reference is reported only as an aggregate count with the prompt still serving. The rule itself is one implementation for both kinds (`prompt.CheckAttachScope` over an `AttachmentScope` carrying a SET of audience ids), because a script may serve several personas where a resource serves exactly one, and a single id cannot answer whether the prompt's audience is contained in the material's. In MCP Apps-capable hosts, the built-in `prompt-browser` app (#1011) is bound to the presentation-only `show_prompts` tool: a host renders an app on every call to the tool it is bound to, so the browser lives on a tool whose only job is to open it for the human rather than on `manage_prompt`, which the agent calls throughout its own work. `show_prompts` opens the browser for the human (search, buckets, collection/tag facets, argument forms, preview with provenance, and a Run action that resolves via `use` and inserts the rendered prompt when the host supports `ui/message`); the rendered app populates itself from its own `manage_prompt` calls. `manage_prompt` carries no app and renders nothing, and its JSON results stand alone in non-app clients. ## Scripts -The portal's view of managed scripts, for the people who own the automations (#1290, #1307). The listing shows every script the caller may see with, per row, what it is executing (the latest saved version, or plainly that it is disabled or retired and nothing will run it), the cadence and next fire of its schedule, stated in the same words the schedule editor states them in rather than as the cron expression the platform stores, since this is the column an owner scans to answer what is running and when; an expression the cadence builder cannot express falls back to the expression itself, which is all there is to say about it, and both the listing and the editor read what the schedule is doing off one `scheduleState` so neither can call a paused schedule due (a paused schedule says so rather than showing a fire that will not happen; a script without one runs on demand) (#1358), and the state of its most recent run. Opening a script shows the contract — owner, visibility, the version that runs, cadence, next fire, lifecycle status, and the typed parameters a run binds against — which is the SAME document `search`, `fetch` on an `mcp:script:` reference, and a prompt that references the script serve, so the page and an agent describe a script identically; where the run gate would refuse a run requested now, the page carries the gate's own refusal (`script.RefuseRun`) rather than re-deriving runnability from a status and an enabled flag. For a script the caller owns, two more sections: the version history, each version with its author and the roles that author held at the save — which are the roles a run of that version presents — with the version that runs open by default; and the run history, each run with its trigger, the version it executed, its duration, its outputs, its failure reason where it failed, and, opened in place, its bound parameters, its cost in steps and queries and exports, its outputs, and the bounded log it printed. The run rows are composed rather than laid out flat: how a run ended and when it ran are one fact and are set as one, while the trigger (a short enumeration) and the version (the same number down the whole column) qualify it from underneath rather than each holding a column open, which is what lets the history fit the width the page has instead of scrolling sideways (#1362). A `skipped_overlap` row is shown as a distinct state, neither success nor failure: it names a fire that never executed because the previous run was still going, which is exactly what a report that stopped producing has to be able to show. An output written to the portal links to the asset version it produced — a recurring script writes new versions of ONE asset keyed `script::`, so that asset's version history is the automation's refresh history — while an object delivered to a configured bucket destination names its bucket and key and is deliberately not a link, because the platform wrote those bytes and does not hold them. Three visibility rules, shared with every other surface: the contract to anyone the scope rules admit (`Script.VisibleTo`, applied as a store predicate, never as a filter over the answer); the source and the run history to the script's owner and to administrators, because a log is free text the script printed while presenting its author's captured roles and may echo rows the reader has no access to of their own; and one particular run additionally to whoever requested it, since the result was handed to them when they asked for it. "Not yours" and "no such script" are answered identically. The pages write five things and none of them is an authority beyond the author's own (#1307, #1363, #1364, #1369). The cadence: on a script they own, at any scope, a caller sets or replaces the cron expression, the timezone it is read in, and the values every fire binds, and pauses or resumes it — over `GET`/`PUT /api/v1/portal/scripts/{id}/schedule` and the enable/disable pair, inside the portal's own authentication and CSRF handling, restricted to the owner and to administrators and refusing a non-owner exactly as it refuses a caller who may not see the script. Pausing is its own route rather than a field of the cadence, because re-sending the schedule to turn it off would re-base the fire it resumes on. A cadence saved against a disabled or retired script saves and stays inert, and the page says so in the gate's own words. The cadence is asked for in the terms a person has it in — hourly, daily, weekdays, chosen days of the week, a day of the month, a time, a timezone — and the cron expression is DERIVED and shown rather than asked for, because cron is a precise notation for people who already know it and the owner of a report is not required to be one of them; a Custom field keeps every expression reachable, and an expression an agent wrote through manage_script opens there as itself rather than being rewritten into something near it. The SOURCE: the code is editable in the portal's own editor with Starlark highlighted as the Python dialect it is, and the edit crosses `script.ApplyEdit` — the one gate every mutation surface crosses — landing on the live row as the version that runs and recording the roles its editor held, which is what a run of it presents; the source is parsed before anything is stored. A RUN of the latest saved version (`POST /api/v1/portal/scripts/{id}/runs`), which queues exactly what `run_script` queues under the same gate, worker and principal and records `portal` as the trigger. A DRAFT run of an edit, executed as the caller with the draft limits, persisting nothing it produced. And what the script SAYS about itself — display name, markdown description, category, tags — which is not an input to any decision the platform makes and is still captured as a version. `show_scripts` opens the pages for a human and performs no data work, following the `show_prompts` split (#1040) so an agent's own `manage_script` calls never render UI; it returns a confirmation and, where the deployment is configured with its public address, a link. +The portal's view of managed scripts, for the people who own the automations (#1290, #1307). The listing shows every script the caller may see with, per row, what it is executing (the latest saved version, or plainly that it is disabled or retired and nothing will run it), the cadence and next fire of its schedule, stated in the same words the schedule editor states them in rather than as the cron expression the platform stores, since this is the column an owner scans to answer what is running and when; an expression the cadence builder cannot express falls back to the expression itself, which is all there is to say about it, and both the listing and the editor read what the schedule is doing off one `scheduleState` so neither can call a paused schedule due (a paused schedule says so rather than showing a fire that will not happen; a script without one runs on demand) (#1358), and the state of its most recent run. Opening a script shows the contract — owner, visibility, the version that runs, cadence, next fire, lifecycle status, and the typed parameters a run binds against — which is the SAME document `search`, `fetch` on an `mcp:script:` reference, and a prompt that references the script serve, so the page and an agent describe a script identically; where the run gate would refuse a run requested now, the page carries the gate's own refusal (`script.RefuseRun`) rather than re-deriving runnability from a status and an enabled flag. For a script the caller owns, two more sections: the version history, each version with its author and the roles that author held at the save — which are the roles a run of that version presents — with the version that runs open by default; and the run history, each run with its trigger, the version it executed, its duration, its outputs, its failure reason where it failed, and, opened in place, its bound parameters, its cost in steps and queries and exports, its outputs, and the bounded log it printed. The run rows are composed rather than laid out flat: how a run ended and when it ran are one fact and are set as one, while the trigger (a short enumeration) and the version (the same number down the whole column) qualify it from underneath rather than each holding a column open, which is what lets the history fit the width the page has instead of scrolling sideways (#1362). A `skipped_overlap` row is shown as a distinct state, neither success nor failure: it names a fire that never executed because the previous run was still going, which is exactly what a report that stopped producing has to be able to show. An output written to the portal links to the asset version it produced — a recurring script writes new versions of ONE asset keyed `script::`, so that asset's version history is the automation's refresh history — while an object delivered to a configured bucket destination names its bucket and key and is deliberately not a link, because the platform wrote those bytes and does not hold them. Three visibility rules, shared with every other surface: the contract to the script's owner (`Script.OwnedBy`, applied as a store predicate, never as a filter over the answer); the source and the run history to that owner and to administrators, because a log is free text the script printed while presenting its author's captured roles and may echo rows the reader has no access to of their own; and one particular run additionally to whoever requested it, since the result was handed to them when they asked for it. "Not yours" and "no such script" are answered identically. The pages write five things and none of them is an authority beyond the author's own (#1307, #1363, #1364, #1369). The cadence: on a script they own, at any scope, a caller sets or replaces the cron expression, the timezone it is read in, and the values every fire binds, and pauses or resumes it — over `GET`/`PUT /api/v1/portal/scripts/{id}/schedule` and the enable/disable pair, inside the portal's own authentication and CSRF handling, restricted to the owner and to administrators and refusing a non-owner exactly as it refuses a caller who may not see the script. Pausing is its own route rather than a field of the cadence, because re-sending the schedule to turn it off would re-base the fire it resumes on. A cadence saved against a disabled or retired script saves and stays inert, and the page says so in the gate's own words. The cadence is asked for in the terms a person has it in — hourly, daily, weekdays, chosen days of the week, a day of the month, a time, a timezone — and the cron expression is DERIVED and shown rather than asked for, because cron is a precise notation for people who already know it and the owner of a report is not required to be one of them; a Custom field keeps every expression reachable, and an expression an agent wrote through manage_script opens there as itself rather than being rewritten into something near it. The SOURCE: the code is editable in the portal's own editor with Starlark highlighted as the Python dialect it is, and the edit crosses `script.ApplyEdit` — the one gate every mutation surface crosses — landing on the live row as the version that runs and recording the roles its editor held, which is what a run of it presents; the source is parsed before anything is stored. A RUN of the latest saved version (`POST /api/v1/portal/scripts/{id}/runs`), which queues exactly what `run_script` queues under the same gate, worker and principal and records `portal` as the trigger. A DRAFT run of an edit, executed as the caller with the draft limits, persisting nothing it produced. And what the script SAYS about itself — display name, markdown description, category, tags — which is not an input to any decision the platform makes and is still captured as a version. `show_scripts` opens the pages for a human and performs no data work, following the `show_prompts` split (#1040) so an agent's own `manage_script` calls never render UI; it returns a confirmation and, where the deployment is configured with its public address, a link. --- @@ -3750,13 +3750,13 @@ SQL parameters are BOUND, never spliced. `platform.query` takes `:name` placehol The authoring loop is the feature's other half, because the author is a model whose Python instincts produce exactly what Starlark lacks. `manage_script help` states the dialect contract in-context, and `manage_script get` retrieves seeded worked examples. `validate` parses and resolves without executing, extracts the capabilities, connections, and output destinations the code references (reporting `dynamic_connections` when a call computes its connection instead of naming one, so a reviewer is never shown a list that quietly omits one), runs a credential-literal scan that blocks the script, and answers the predictable Python-isms with targeted corrections rather than bare parse errors: `import` is not available (json and date are already predeclared), `try`/`except` does not exist (errors fail the run by design; use `fail("why")`), f-strings are not supported (use `.format()` or `%`), there is no clock (derive dates from `run.fire_time`), no randomness, no filesystem, no network. `run_draft` then executes for real, persisting nothing: `platform.export` serializes the output exactly as a platform run would — a tabular output through the same formatter, a document body measured verbatim — and reports the shape and size that produced, so the number an author sizes an output against is the number a real run writes rather than a format-independent estimate of it, and an output over the size ceiling is refused at that point rather than at the first scheduled fire. -Governance follows the prompt shape in its storage: a live `scripts` row plus immutable `script_versions` snapshots, and one `ApplyEdit` funnel every mutation surface crosses — the `manage_script` tool and the portal editor alike. It diverges in the one thing that matters most (#1403): there is no execution gate and no review step. A SAVED script runs. Every edit lands on the live row and is captured as an applied version, and `run_script`, the portal's run action, and a cron schedule all execute the latest saved version; `script.RefuseRun` is the one rule every path into execution answers to, and its only refusals are disabled, deprecated, and superseded. It is asked at enqueue and again by the worker at claim, so a script taken out of service refuses a run already on the queue, while a run executes the version it was queued against — loaded by its immutable id — so a save landing during a queue wait cannot swap code underneath a pending run. What bounds a save is not a reviewer but the authority the version captures: `script_versions.author_roles` records the roles its author held at that moment, no surface anywhere accepts roles as input, and a run of that version presents exactly those roles. A version written by a caller holding none produces runs that resolve to the deny-all persona and can call nothing. Who may save is the edit rule — a non-admin edits only their own personal scripts, and editing a `global` or `persona`-scoped script is an administrator's action — and a personal script is its owner's to DELETE outright, while a shared one is refused in favor of deprecating it because it may be executing on a schedule for somebody else. +Governance follows the prompt shape in its storage: a live `scripts` row plus immutable `script_versions` snapshots, and one `ApplyEdit` funnel every mutation surface crosses — the `manage_script` tool and the portal editor alike. It diverges in the one thing that matters most (#1403): there is no execution gate and no review step. A SAVED script runs. Every edit lands on the live row and is captured as an applied version, and `run_script`, the portal's run action, and a cron schedule all execute the latest saved version; `script.RefuseRun` is the one rule every path into execution answers to, and its only refusals are disabled, deprecated, and superseded. It is asked at enqueue and again by the worker at claim, so a script taken out of service refuses a run already on the queue, while a run executes the version it was queued against — loaded by its immutable id — so a save landing during a queue wait cannot swap code underneath a pending run. What bounds a save is not a reviewer but the authority the version captures: `script_versions.author_roles` records the roles its author held at that moment, no surface anywhere accepts roles as input, and a run of that version presents exactly those roles. A version written by a caller holding none produces runs that resolve to the deny-all persona and can call nothing. Who may save is the edit rule — a script is one person's, so its owner saves it and so does an administrator — and it is its owner's to DELETE outright, taking its schedule and history with it. An administrator can move a script to another owner (`PUT /api/v1/portal/scripts/{id}/owner`), which hands over everything at once and records a new version whose captured roles are the transferring administrator's, so moving a script to an administrator is how it comes to run with an administrator's reach; the move is refused of deprecating it because it may be executing on a schedule for somebody else. Execution identity: a run authenticates as the distinct principal `script:` (following the `apikey:` convention), injected with `middleware.WithPreAuthenticatedUser` and tagged `middleware.AuthTypeScript`, carrying the executing version's `author_roles` and the script owner's address alongside for accountability. The principal holds no authority of its own — the middleware resolves its roles to a persona exactly as it does for a person, so the persona is the authority of record, and it is resolved FRESH at every call: narrowing a persona's connection rules takes effect on that script's next run with no script-side action, and there is no stored per-script allowlist to drift out of step with the persona configuration it would otherwise duplicate. Destinations are the one axis configuration owns rather than the script: `scripts.destinations` declares each bucket destination as a complete address (the platform S3 connection, the bucket, an optional key prefix), validated at startup — a partial address, a duplicated name, or an attempt to redeclare the built-in `portal` is refused — and a run resolves the destination name a script writes against that list at run time, so repointing one is a configuration change that takes effect on the next run. An undeclared name is refused inside the interpreter, naming the configured set, and a draft run resolves through the same list so a destination a real run would refuse fails while the author is still iterating. Runs execute as the distinct principal `script:` (following the `apikey:` convention) with the executing version's captured author roles, over a per-run in-memory MCP session, so persona and connection authorization, rate limiting, and audit apply exactly as to an agent's call. Enforcement is layered and neither layer is load-bearing alone: the host facade refuses an undeclared destination inside the interpreter, naming the configured set, and the middleware chain enforces the persona those roles resolve to at every call, which is the authority of record. External DELIVERY is the sharpest case and is deliberately not a private route to object storage: it is one ordinary `s3_put_object` tool call over the run's own session, so the facade refuses a destination configuration does not declare and the middleware then refuses the write independently when the script's persona does not hold that connection. There is no arbitrary egress to have — a script supplies no endpoint, credential, bucket, or host name, and the only network it can reach is the operator-configured connection set. The configured prefix is the boundary: an absolute key or one containing `..` is REFUSED rather than normalized away, an output may be written once per destination per run (and two outputs may not land on ONE object key, since the second write would replace the first in a bucket the platform cannot read back), and a reclaimed run does not deliver twice. `destination` and `key` must be NAMED arguments: passed by position they would be invisible to the static read the capability diff is built from, and the review surface would state positively that a script writing to a bucket writes to the portal. Audited arguments are bounded at 16KB so a delivered report does not put a second copy of itself in the audit table on every fire. The gate is re-read at EXECUTION, not trusted from the queue row: between requesting a run and running it a script can be disabled, deprecated, or superseded, and each refuses the run. `platform.export` now persists — one asset per (script, output name), a new VERSION per run, so a daily report keeps its identity, shares, and history instead of minting 365 assets a year. The run queue follows the platform's existing shape (`FOR UPDATE SKIP LOCKED` claim, crashed-worker reclaim folded into the claim predicate via an expiring lease, no reaper and no leader election); every write is fenced on the lease it was taken under, so a worker whose run was reclaimed writes to nothing rather than overwriting the new holder's result, and a reclaimed run skips outputs it already wrote. Retry is classified by WHERE a failure happened, never by matching error text: platform faults outside the interpreter (session, store reads) retry with backoff under a small attempt budget, and everything the interpreter reports is final, because a Starlark error reproduces exactly and a script that already queried or wrote must not be replayed. Run history is kept a year by default (`scripts.run_retention_days`), far longer than a delivery queue, because a scheduled report's run history is its refresh history. WHERE a run executes is one key: `scripts.worker.enabled` is a `*bool` defaulting to on, so a single process serves and executes; setting it false leaves a replica serving MCP and portal traffic, registering `run_script`, enqueueing, and waiting on results while never claiming, and a separate deployment of the same image with the worker on drains the queue. A stopping worker stops claiming immediately, gives a run it holds a short capped window out of the shutdown budget (never more than half of what is left, since that budget belongs to every component the lifecycle stops) with the write that records the outcome bounded too, and releases anything unfinished back onto the queue rather than recording a verdict on it — a shutdown decides nothing about a run — so a rolling deploy neither strands a lease until it expires nor kills a run mid-write. `run_draft` stays in process on whichever replica the author is talking to: it is bounded interactive authoring under the author's own identity, not queue work. Audit carries two joined rows per run: the per-capability tool calls under the script principal, and one `script_run` lifecycle event, both keyed on the run id as their session. -Scheduling adds cadence and nothing else. A `script_schedules` row carries a cron expression (standard five fields or a descriptor), the IANA timezone it is read in, the parameter values every fire binds, and an enabled flag — no roles, connections, or destinations, because a schedule decides when the latest saved version runs and never what it may reach. Cron parsing is `robfig/cron/v3` PARSE-ONLY (`ParseStandard(...).Next(t)`); its goroutine runner is not adopted, because there is no scheduler process: materializing a due fire means inserting a `script_runs` row, and the queue's existing `scheduled_for <= NOW()` claim predicate does the rest. A script has at most one schedule (a second cadence is a second script), setting one again replaces it in place so the runs pointing at it point at the same automation, and there is no delete — disabling is the retirement path, so the row that explains a run is never removable on its own. A paused schedule reports no next fire on any surface: the stored due time survives the pause because resuming picks up the fire it was parked on, and stating it while paused would tell an operator reading the unattended inventory that a schedule nobody has re-enabled is about to run. Bound values may contain one token, `${fire_date}`, expanded at materialization into the run row in the schedule's own timezone: that is what makes a scheduled run reproducible, since a script computing today's date would answer differently every time it ran. Bindings are checked against the APPROVED contract when the schedule is set, not silently at the first fire, so a cadence that could never bind is refused while somebody is still looking at it; a cadence on a disabled or retired script saves and simply fires nothing. Setting one is the script OWNER's action at every scope, or an administrator's, on `manage_script` and on the portal alike (#1307). That is deliberately weaker than the edit rule, which confines a non-admin to their personal scripts: the run gate and the persona filter are re-read at every fire, so re-timing a global or persona script reaches nothing it could not already reach, and requiring an administrator would mean the owner of a shared report cannot pause their own report. Three policies are enforced by PostgreSQL rather than by code that checks first: single-fire is a unique index on `script_runs (schedule_id, fire_time)` — keyed on `fire_time`, NOT `scheduled_for`, because an infrastructure retry MOVES `scheduled_for` and would take a run out from under a key built on it — so every worker replica materializes with no leader and racing inserts collapse to exactly one run; overlap is a partial unique index of one OPEN run per schedule, and the refused fire is recorded as a terminal `skipped_overlap` run so a skip is visible rather than silent; misfire is fire-once-latest, one run for the most recent due fire with the rest counted on the schedule's `missed_fires`, because a catch-up burst after downtime would hit the warehouse with reports computing dates nobody is waiting on any more, and a backfill somebody wants is an explicit `run_script`. A cadence must not fire more often than once a minute, and an expression that never fires is refused when it is set. Materialization runs wherever the run worker runs (`scripts.worker.enabled`), since a replica that will not claim gains nothing by producing rows for one that will; the release image is built FROM scratch, so the binary embeds the IANA zone database (`_ "time/tzdata"`) or every named zone would resolve in development and fail in production. A FAILED SCHEDULED run mails the script's owner, carrying the run id, the failure, and the tail of what the script printed; a `run_script` failure never mails, because it is already in the response its caller is reading. That category has no per-user toggle, for the same reason the review-queue alert has none — it is addressed to a responsibility rather than an interest — and a recipient's own delivery mode is still their opt-out; the alert names the SCRIPT as its actor, which is what the enqueuer rate-limits on, so a night that fails forty schedules does not spend one person's budget and drop the rest. Every run is measured where it reaches a terminal state rather than where it is enqueued (#1307): `script_runs_total` by script, trigger and status, `script_run_duration_seconds`, a `script_runs_running` gauge bracketed AROUND the execution so a worker wedged on a run that never finishes is visible, and `script_missed_fires_total` — the one thing the run table cannot show, because a missed fire is precisely a run that does not exist. The admin portal's Runs tab draws them beside the exact recent history from the run rows: the metrics survive run retention and aggregate across replicas, the rows carry the reason a particular run failed, and neither can do the other's job. The platform changes a schedule on its own in exactly one case: an expression that no longer parses is disabled, because walking an uncomputable row every half minute forever is worse than a state its owner can see. A timezone that will not LOAD is deliberately not treated that way — the zone database is compiled into the binary, so that fault belongs to the build and disabling would retire every non-UTC schedule at once with nothing to re-enable them. +Scheduling adds cadence and nothing else. A `script_schedules` row carries a cron expression (standard five fields or a descriptor), the IANA timezone it is read in, the parameter values every fire binds, and an enabled flag — no roles, connections, or destinations, because a schedule decides when the latest saved version runs and never what it may reach. Cron parsing is `robfig/cron/v3` PARSE-ONLY (`ParseStandard(...).Next(t)`); its goroutine runner is not adopted, because there is no scheduler process: materializing a due fire means inserting a `script_runs` row, and the queue's existing `scheduled_for <= NOW()` claim predicate does the rest. A script has at most one schedule (a second cadence is a second script), setting one again replaces it in place so the runs pointing at it point at the same automation, and there is no delete — disabling is the retirement path, so the row that explains a run is never removable on its own. A paused schedule reports no next fire on any surface: the stored due time survives the pause because resuming picks up the fire it was parked on, and stating it while paused would tell an operator reading the unattended inventory that a schedule nobody has re-enabled is about to run. Bound values may contain one token, `${fire_date}`, expanded at materialization into the run row in the schedule's own timezone: that is what makes a scheduled run reproducible, since a script computing today's date would answer differently every time it ran. Bindings are checked against the APPROVED contract when the schedule is set, not silently at the first fire, so a cadence that could never bind is refused while somebody is still looking at it; a cadence on a disabled or retired script saves and simply fires nothing. Setting one is the script OWNER's action, or an administrator's, on `manage_script` and on the portal alike (#1307). It is the same rule reading and editing answer to: the run gate and the persona filter are re-read at every fire, so re-timing a script reaches nothing it could not already reach, and requiring an administrator would mean the owner of a shared report cannot pause their own report. Three policies are enforced by PostgreSQL rather than by code that checks first: single-fire is a unique index on `script_runs (schedule_id, fire_time)` — keyed on `fire_time`, NOT `scheduled_for`, because an infrastructure retry MOVES `scheduled_for` and would take a run out from under a key built on it — so every worker replica materializes with no leader and racing inserts collapse to exactly one run; overlap is a partial unique index of one OPEN run per schedule, and the refused fire is recorded as a terminal `skipped_overlap` run so a skip is visible rather than silent; misfire is fire-once-latest, one run for the most recent due fire with the rest counted on the schedule's `missed_fires`, because a catch-up burst after downtime would hit the warehouse with reports computing dates nobody is waiting on any more, and a backfill somebody wants is an explicit `run_script`. A cadence must not fire more often than once a minute, and an expression that never fires is refused when it is set. Materialization runs wherever the run worker runs (`scripts.worker.enabled`), since a replica that will not claim gains nothing by producing rows for one that will; the release image is built FROM scratch, so the binary embeds the IANA zone database (`_ "time/tzdata"`) or every named zone would resolve in development and fail in production. A FAILED SCHEDULED run mails the script's owner, carrying the run id, the failure, and the tail of what the script printed; a `run_script` failure never mails, because it is already in the response its caller is reading. That category has no per-user toggle, for the same reason the review-queue alert has none — it is addressed to a responsibility rather than an interest — and a recipient's own delivery mode is still their opt-out; the alert names the SCRIPT as its actor, which is what the enqueuer rate-limits on, so a night that fails forty schedules does not spend one person's budget and drop the rest. Every run is measured where it reaches a terminal state rather than where it is enqueued (#1307): `script_runs_total` by script, trigger and status, `script_run_duration_seconds`, a `script_runs_running` gauge bracketed AROUND the execution so a worker wedged on a run that never finishes is visible, and `script_missed_fires_total` — the one thing the run table cannot show, because a missed fire is precisely a run that does not exist. The admin portal's Runs tab draws them beside the exact recent history from the run rows: the metrics survive run retention and aggregate across replicas, the rows carry the reason a particular run failed, and neither can do the other's job. The platform changes a schedule on its own in exactly one case: an expression that no longer parses is disabled, because walking an uncomputable row every half minute forever is worse than a state its owner can see. A timezone that will not LOAD is deliberately not treated that way — the zone database is compiled into the binary, so that fault belongs to the build and disabling would retire every non-UTC schedule at once with nothing to re-enable them. The owner's loop is on the script's own page rather than only in an agent session, and the ADMIN section mounts the SAME page, so an administrator runs, edits, dry-runs, schedules and reads the history of every script exactly as its owner does — one detail surface rather than two that drift apart a feature at a time. diff --git a/docs/llms.txt b/docs/llms.txt index 19bd66902..81e747378 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -57,8 +57,8 @@ Three facts that are commonly assumed the other way around, stated here so an ag - [OAuth Server](https://mcp-data-platform.txn2.com/auth/oauth-server/): Built-in OAuth 2.1 authorization server for Claude Desktop and other MCP clients, with PKCE, Dynamic Client Registration, upstream IdP integration via OIDC discovery of the authorization/token endpoints (works with any OIDC-compliant provider; optional explicit-endpoint override), refresh tokens hashed at rest, HS256 access tokens with `kid`-based signing-key rotation (verify-only previous keys), and default-on rate limiting (trusted-proxy-aware per-IP + global backstop) on the `/token` and `/register` endpoints - [OAuth to Upstream MCPs](https://mcp-data-platform.txn2.com/auth/oauth-gateway/): Outbound OAuth to gateway upstreams: client_credentials and authorization_code + PKCE grants, encrypted refresh tokens that survive restarts, background refresh, endpoint URL validation, and a full auth-event history - [Threat Model](https://mcp-data-platform.txn2.com/security/threat-model/): The security model as a whole: a trust-boundary diagram (inbound surfaces, identity mechanisms, outbound dependencies, at-rest stores), STRIDE-style attacker analysis across six personas (unauthenticated network, low-privilege persona, malicious upstream, malicious query data, database reader, compromised downstream credential), the recorded identity-provider-outage decision (edge passes an unvalidatable credential through, protocol layer refuses as retryable, pinned by an end-to-end test), a threat-to-mechanism mitigations table with package/config citations, and explicit non-goals (stdio local-process trust, no defense against a malicious admin, best-effort async audit loss model, per-connection rather than per-user downstream identity stated as a design boundary with its rationale and its cost, no content sanitization, deployment-owned TLS/segmentation) -- [Managed Scripts: Security Model](https://mcp-data-platform.txn2.com/scripts/security/): The threat model for managed scripts, the agent-authored Starlark programs the platform stores, versions, and governs. States the authority claim structurally — a script can never do what the person who WROTE it could not do, because a draft runs as the caller and a platform run runs as the principal `script:` carrying the roles its author held, captured on the immutable version row (`script_versions.author_roles`) at the save and presented by the runner; no surface anywhere accepts roles as input. Covers the run gate (`script.RefuseRun`: a SAVED script runs, and the only refusals are disabled, deprecated, and superseded — re-read at enqueue and again at claim, so a script taken out of service refuses a run already on the queue; a run executes the version it was queued against, the latest saved at the moment of the request or the fire, loaded by its immutable id, so a save landing during a queue wait cannot swap code underneath it). The persona filter is the ENTIRE authorization boundary at run time: every host call is one MCP tool call over a per-run in-memory session against the assembled server, so authentication, persona and connection authorization, rate limiting and audit apply exactly as they do to an agent's call, none of it re-implemented, and the roles are resolved to a persona fresh at every call — narrowing a persona takes effect on the next run with no script-side action, and there is no stored per-script allowlist to drift out of step with the persona configuration it would duplicate. Destinations are CONFIGURATION rather than a per-version record: `scripts.destinations` declares each bucket destination as a complete address (the platform S3 connection, the bucket, an optional key prefix), a run resolves the name a script writes against that list at run time so repointing one takes effect on the next run, the portal is built in with its name reserved and configuration cannot redeclare it, an undeclared name is refused inside the interpreter naming the configured set, a draft resolves through the same list so a destination a real run would refuse fails while the author is iterating, and the write is still authorized by the middleware, so a destination whose connection the run's persona cannot reach is refused however configuration names it. Covers external DELIVERY as one ordinary audited tool call rather than a private route to object storage, with the explicit statement that arbitrary egress does not exist — a script supplies no endpoint, credential, bucket or host name, and there is no binding that opens a socket, so the only network it reaches is the operator-configured connection set — plus the prefix as a boundary a key cannot climb out of (an absolute key, a `..` segment or an empty segment is refused rather than normalized away), exactly-once per run per destination and one object per key, `destination` and `key` required as NAMED arguments because a positional one would be invisible to the static read that reports where a script writes, and audited argument values bounded at 16KB so a delivered report does not put a second copy of itself in the audit table. Covers the data-region refresh (`platform.publish_data`, which adds no authority — the author can already rewrite the whole document — and whose region confinement is a behavioral contract: the target is pinned by the export identity rule so the call reaches only this script's own portal outputs and creates nothing, the splice is structural through the one element matching `#data` with the payload's `<` `>` `&` written as \u escapes so it cannot corrupt the document, and the validator reports the refresh target names), the run queue (lease-based claiming with fencing on every write, crashed-worker recovery folded into the claim predicate so there is no reaper and no leader election, and no double-written output because each output is recorded as it lands), retry classified by WHERE a failure happened rather than by matching error text, audit under the script principal joined to a `script_run` lifecycle event by the run id, the sandbox (Starlark has no ambient clock, randomness, filesystem, network, or module system; `while` and recursion off; the predeclared set is exactly platform/json/date/run), the resource limits with the honest gap (no hard MEMORY cap in any embedded interpreter of this class) and the control that bounds what that gap COSTS rather than preventing it (`scripts.worker.enabled: false` on serving replicas plus a worker deployment of the same binary, so heap pressure lands on a pod that accepts no request and the worst case is a restarted worker whose run another replica reclaims), typed SQL parameter binding with a state-aware scanner instead of string concatenation, a write refused in the script surface's own vocabulary before it becomes a tool call, a truncated query result failing the run because silently wrong is the one outcome the determinism contract exists to exclude, the credential-literal scan (error on a credential FORMAT, warning on a naming convention, and a tripwire rather than a proof), unparseable source never stored, the three `SourceScript` middleware behaviors (exempt from the session and search-first gates because there is no model in a script run, an isolated per-run session identity so a run never advances the gate or provenance state of the person it runs for, and enrichment skipped), and the determinism contract stated exactly: same script version + same parameters + same underlying data produce the same output, which is reproducibility rather than identical forever. The scheduling posture: a schedule carries cadence, timezone, and parameters only, is set by the script's OWNER at every scope or by an administrator — deliberately a weaker rule than the edit rule, because the run gate and the persona filter are re-read at every fire, so re-timing reaches nothing new — and fires nothing on a script the gate refuses; the one-fire-a-minute floor and the one-open-run-per-schedule overlap policy are what bound unattended repetition, single-fire across replicas is a unique index on (schedule, fire time) rather than a leader, and a failed scheduled run mails the script's OWNER. Covers DISCOVERABILITY as a security-relevant widening: a script is addressable as `mcp:script:` and reachable from `search`, `fetch`, and a prompt that references it, each applying the script's own scope rule as a store predicate, returning the contract (name, parameters, whether a run would be admitted, cadence, last run) and never the source, and granting nothing; the semantic index embeds the description card and never the Starlark, because one vector per row cannot be split along the line that admits the contract to everyone the scope rules admit and the source only to the owner and to administrators, and both ranking arms apply the same scope predicate so the index widens nothing. Reading and writing in the portal grants nothing either: the script pages write five things — a cadence, the SOURCE through the same `ApplyEdit` funnel every mutation surface crosses, a run of the latest saved version under `RefuseRun`, a DRAFT run executed as the caller with the draft limits that persists nothing it produced, and what the script SAYS about itself (display name, markdown description, category, tags), which is not an input to any decision the platform makes — and apply the rules every surface shares: the contract to anyone the scope rules admit; the source and the run history to the script's owner and administrators; one particular run additionally to whoever requested it; and the cadence controls to the owner and administrators, refusing a caller who does not own the script with the same answer as one who may not see it. Residual risks are named rather than minimized: no hard memory cap; a save is unattended execution with no second reader (bounded by the roles being the author's own and never more, by the persona filter enforcing them at every call and re-resolving them at every run, by editing a shared script being an administrator's action, and by disable/deprecate/supersede stopping it at execution — a person can, through a script, arrange for their OWN access to be exercised on a schedule, which is the feature, and the audit trail under the script principal is its record); a version authored by an admin captures admin roles; standing authority outlives the author; a schedule multiplies what a save permitted; delivery is standing egress on a schedule once configuration declares a destination; a draft run has no per-request rate limit of its own; and a dry run's stored log is free text the script printed under its CALLER's access -- [Running Managed Scripts](https://mcp-data-platform.txn2.com/scripts/running/): How a managed script runs and what happens when it does. Covers the central rule — a SAVED script runs: `run_script`, the portal's run action, and a cron schedule all execute the script's latest saved version, there is no approval step and no state in which a script exists but nothing may execute it, and `manage_script run_draft` remains the way to execute an edit as yourself before saving it. Covers the authority a run carries (the script's own principal presenting the roles its author held at the save, captured on the immutable version row and settable no other way, resolved to a persona by the middleware at every call so the persona filter decides which connections a run reaches at run time and a persona change takes effect on the next run), who may save (a non-admin edits only their own personal scripts; editing a global or persona-scoped script is an administrator's action; a personal script is its owner's to delete outright while a shared one is deprecated instead), and where output may go (`scripts.destinations` declares each bucket destination by name and complete address — connection, bucket, optional prefix — resolved at run time so repointing one takes effect on the next run, with the portal built in). Covers `run_script` (arguments checked against the script's parameter contract, a queued run executed by a worker on whichever replica claims it, a bounded wait that hands back a run id and pending status rather than holding the call open, and the run executing the version it was queued against so a save during the wait does not swap code underneath it), stable output identity (one portal asset per script and output name, a new version per run, so a daily report accumulates versions instead of assets), the two content shapes an output takes (rows serialized in the declared format for csv/json/markdown/text, or a string body written verbatim so a script can compose a document — an HTML or JSX dashboard, a prose report — in markdown, text, html, or jsx) and external delivery for the other case (`platform.export` with a `destination` configuration declares as a bucket writes the same bytes out of the platform at a `key` beneath the configured prefix, so one computed result can refresh a dashboard AND hand a CSV to another system, once per destination per run), the DATA-REGION REFRESH of a semi-dynamic dashboard (`platform.publish_data(name, data)`: the presentation lives in the asset — an html, jsx, or markdown document marking exactly one element `id="data"`, conventionally a `