retrieve_entities and retrieve_entity_snapshot disagree on schema_version and field set for the same observation #2466

Summary

For the same unchanged observation, retrieve_entities (list) and
retrieve_entity_snapshot (single) disagree on both schema_version and the
set of fields returned. One reports schema_version: "2.0" and includes a
rationale field; the other reports "1.1.0" and omits it entirely.

I have not isolated the mechanism and am deliberately not guessing at one —
see "What I did not determine" below.

Reproduction

Entity type agent_policy. Any row written with a field not declared in the
1.1.0 schema will do; the one observed carried rationale.

1. List endpoint — retrieve_entities(entity_type="agent_policy"), reading
the row out of entities[]:

{
  "entity_id": "<id>",
  "schema_version": "2.0",
  "snapshot": {
    "scope": "global",
    "rule_kind": "mandatory",
    "rationale": "Allowed Autonomous Actions | Forbidden Autonomous Actions | ...",
    "status": "active"
  }
}

2. Single-entity endpoint — retrieve_entity_snapshot(entity_id="<id>"),
same row, same moment:

{
  "entity_id": "<id>",
  "schema_version": "1.1.0",
  "snapshot": {
    "scope": "global",
    "rule_kind": "mandatory",
    "status": "active"
  }
}

rationale is absent, and schema_version differs.

3. Confirming it is not correction-induced. The row had corrections applied
after first observation, so the obvious hypothesis is that a later observation
landed at a different schema and dropped the field. An as-of read at a cutoff
before any correction rules that out — this replays the original
observation alone:

retrieve_entity_snapshot(entity_id="<id>", at_ingested="<pre-correction UTC timestamp>")
  -> schema_version: "1.1.0"
  -> observation_count: 1
  -> no `rationale` key

So the single-entity endpoint has always projected this row at 1.1.0
without rationale, for the one original observation. Nothing was lost by any
write.

4. retrieve_field_provenance(entity_id="<id>", field="rationale") returns:

MCP error -32602: Field 'rationale' not found in entity snapshot

describe_entity_type("agent_policy") reports schema_version: "1.1.0" and
does not declare rationale among its 17 fields.

Why it matters

A caller that lists entities and a caller that reads one back see different
field sets for the same row. Concretely:

• Read-back verification is weakened. The standing practice is to write,
  then read back and assert the field holds the value written. If the two read
  paths project different field sets, "I read it back and the field is absent"
  is ambiguous between the write did not land and this endpoint does not
  project that field. That ambiguity cost real time here before the as-of read
  settled it.
• It looks like data loss when it is not. A governance row appearing to
  lose a field across a correction is alarming, and the first hypothesis
  (correction dropped it) was wrong. Anyone auditing via the list endpoint and
  verifying via the snapshot endpoint will reach a false conclusion.
• schema_version disagreeing between endpoints makes it unreliable for
  deciding how to interpret a payload.

What I did not determine

I did not isolate whether this is:

• the list endpoint hydrating raw observation JSON (including undeclared fields
  that were routed to raw_fragments at store time) while the single-entity
  endpoint projects strictly through the registered schema; or
• two different schema-resolution paths disagreeing about which version applies
  to the row; or
• something else entirely.

The 2.0 / 1.1.0 split suggests the two paths resolve the schema
differently, but that is an observation about the symptom, not a diagnosis. I
would rather hand this over as a reproduction with an honest unknown than a
plausible-sounding guess that sends someone down the wrong path.

Expected

The two read paths agree on schema_version and on the field set for the same
observation — or, if the divergence is deliberate (e.g. the list endpoint is
intentionally tolerant and returns undeclared fields), it is documented, so a
caller knows which endpoint to trust for a read-back assertion.

Found while implementing markmhendrickson/ateles#1115.

Swarm specification

This section is maintained by the Ateles swarm. Each lens agent owns exactly one subsection below; the human-written description above these markers is never modified.

Product / Scope (PM)

Clean store, unknown_fields_count: 0. Gate handoff is complete: pm signed off, current_owner routed to waxwing (arch), plan_contribution recorded.

🧠 Neotoma — neotoma#2466 PM scoping

• ✅ Created (1): issue ent_9835e4a5bd35892c570b1607 — neotoma#2466, gate_status.pm=signed_off, current_owner=waxwing
• ✅ Created (1): plan_contribution ent_33e19ed53ca2a67ba306f9e9 — pm sign_off record

Verdict comment (post to GitHub issue #2466):

🤖 Pavo — Ateles swarm, pm gate owner
SIGNED_OFF

• Problem, scope, and acceptance criteria defined (see spec section above)
• pm gate signed off; routed to arch (waxwing) — not the bug fast path

This is a read-path contract discrepancy across two MCP query surfaces (schema_version + field-set divergence between retrieve_entities and retrieve_entity_snapshot), which is an interface-surface change wearing a bug label — the arch gate should determine root cause and canonical resolution before implementation.

📎 Neotoma: issue ent_9835e4a5bd35892c570b1607

[pavo] acceptance_criteria: PM scope signed off for neotoma#2466. Problem: retrieve_entities and retrieve_entity_snapshot disagree on schema_version and field set for the same observation. Scope: diagnose root cause (raw-fragment hydration vs strict schema projection, or dual schema-resolution paths), fix or explicitly document the divergence, resolve retrieve_field_provenance's error on the affected field, add regression coverage on both MCP surfaces proving the actual effect (not just contract acceptance) is fixed, revert-tested red-to-green. Out of scope: write-path/raw_fragments redesign, retroactive row repair. Priority: high (undermines standing read-back-verification practice). Routed to arch (waxwing) per interface-surface override rather than the bug fast path, since this is a query/schema-projection contract discrepancy across two MCP tools. Gate writeback confirmed via read-back: gate_status.pm=signed_off, current_owner=waxwing.

Engineering

Design basis: docs/foundation/data_models.md (Entity Snapshot: schema_version is the version used for snapshot computation; snapshot is reducer output) and docs/foundation/schema_agnostic_design_rules.md (projection follows the active schema declaration, not the read path).

Contract. For one entity, retrieve_entities and retrieve_entity_snapshot (current read and at / at_ingested) return the same schema_version and the same snapshot keys. schema_version is loadActiveSchema(entity_type, userId) — the schema describe_entity_type loads (user scope, then global). snapshot contains only fields declared on that schema. Keys present on the stored row but not declared are copied onto the response raw_fragments object and nowhere else. Do not insert raw_fragments rows. Do not recompute or upsert entity_snapshots. When no active schema exists, both paths keep the stored field set and the stored schema_version. Remove the divergence. Do not document it as intentional.

Cause the PR implements against. Do not re-diagnose.

• ObservationReducer.computeSnapshot in src/reducers/observation_reducer.ts stamps schemaEntry.schema_version and drops undeclared fields when a schema loads. computeSnapshotWithDefaults stamps observations[0].schema_version and copies every observation field. A row written on the defaults path persists schema_version: "2.0" and rationale inside entity_snapshots.snapshot.
• Current reads return that stored row and do not re-project. queryEntities in src/services/entity_queries.ts (snapshot select around the entity_snapshots .in("entity_id", entityIds) call) does not filter by user_id. getEntityWithProvenance in the same file does filter by user_id, and MCP retrieveEntitySnapshot in src/server.ts uses it for the non-cutoff read.
• Cutoff reads recompute in computeEntitySnapshotAtTime (src/services/entity_snapshot_at_time.ts) through the reducer, so they already follow the active schema (1.1.0, no rationale). That is why an as-of read of the original observation matches describe_entity_type and disagrees with the list.
• retrieveFieldProvenance (src/server.ts) and POST /get_field_provenance (src/actions.ts) read provenance[field] on the stored snapshot only, so rationale errors with Field 'rationale' not found in entity snapshot / HTTP 404 even when the observation or raw_fragments still holds it.
• POST /get_entity_snapshot with no cutoff returns the raw entity_snapshots row, unscoped. The raw-id branch of retrieveEntityByIdentifierWithFallback (src/shared/action_handlers/entity_identifier_handler.ts) assigns that whole row to snapshot.

Files.

• Add src/services/snapshot_read_model.ts with alignSnapshotToActiveSchema and resolveFieldProvenance.
• src/services/entity_queries.ts: call alignSnapshotToActiveSchema from the queryEntities return mapping and from getEntityWithProvenance. When userId is set, add .eq("user_id", userId) to the queryEntities entity_snapshots select.
• src/services/entity_snapshot_at_time.ts: pass the recomputed result through alignSnapshotToActiveSchema before return.
• src/actions.ts POST /get_entity_snapshot, no-cutoff branch: stop returning the raw table row. Return the same object getEntityWithProvenance returns (user-scoped, aligned). The cutoff branch stays on computeEntitySnapshotAtTime.
• src/shared/action_handlers/entity_identifier_handler.ts raw-id branch: load the entity through getEntityWithProvenance so snapshot is the field map, not the table row.
• src/server.ts retrieveFieldProvenance and src/actions.ts POST /get_field_provenance: both call resolveFieldProvenance.
• Add tests/integration/snapshot_read_model_parity.test.ts.
• Do not change store, raw_fragments writes, schema registration, or the agent_policy schema.

alignSnapshotToActiveSchema. Input: entity_type, user_id, stored schema_version, stored snapshot, response raw_fragments, provenance. Load schemaRegistry.loadActiveSchema(entity_type, user_id). If a schema exists, set schema_version to schema.schema_version; keep only keys in schema.schema_definition.fields; copy dropped keys into response raw_fragments when that key is not already present; drop those keys from the returned provenance. If no schema exists, return the stored values unchanged. No writes.

resolveFieldProvenance. User-scope every lookup. If the aligned provenance has the field, return the current success payload plus projected: true. If any observation fields object or a raw_fragments row for that entity has the field, return that observation or fragment with projected: false and do not error. Otherwise keep the MCP text Field '<field>' not found in entity snapshot and the HTTP 404 text Field not found in provenance.

Data / contract. No schema migration. No new entity fields. projected is a new boolean on the field-provenance success payload (MCP and POST /get_field_provenance). POST /get_entity_snapshot without a cutoff changes shape from the raw table row to the MCP read model (snapshot is the field map). List and snapshot responses already have schema_version, snapshot, and raw_fragments; only their values change.

Effect test. Must fail if alignment is removed. Register a test type at schema 1.1.0 with only scope, rule_kind, and status. Insert one observation whose fields also include rationale and whose schema_version column is "2.0". Upsert entity_snapshots to schema_version: "2.0" with rationale inside snapshot. Insert a raw_fragments row for rationale. On every surface below, assert schema_version === "1.1.0", snapshot has the three declared fields and no rationale key, and raw_fragments.rationale is the stored value. Also assert the at_ingested read (one observation) has the same version and no rationale.

Surfaces, natural call shape:

• MCP: retrieve_entities({ entity_type }); retrieve_entity_snapshot({ entity_id, format: "json" }); retrieve_entity_snapshot({ entity_id, at_ingested }).
• HTTP: POST /entities/query; GET /entities; POST /get_entity_snapshot with and without at_ingested.
• CLI: neotoma entities list and neotoma request --operation getEntitySnapshot with the same bodies.
• Provenance: MCP retrieve_field_provenance, POST /get_field_provenance, and CLI getFieldProvenance. rationale returns projected: false and not an error. status returns projected: true.

Assert the field set and version, not HTTP 200.

Build steps.

1. Add the integration test and confirm it is red on the list-versus-snapshot disagreement.
2. Add snapshot_read_model.ts and route the call sites above through it, including the user_id scope on the list snapshot select.
3. Switch both field-provenance handlers to resolveFieldProvenance.
4. Confirm the test is green on every surface listed above.
5. Run npm run generate:test-catalog and npm run validate:test-catalog. Commit the generated catalog in the same PR.
6. If projected changes a generated field-provenance schema, regenerate openapi_types / contract_mappings and run the matching validate:*. No other generated files.

Out of scope. Store path and raw_fragments writes. Retroactive snapshot recompute. Adding rationale to agent_policy. describe_entity_type.

QA / Test Plan

The QA section is above. The qa gate stays pending until tests/integration/snapshot_read_model_parity.test.ts is committed and green.

Security / Arch

Design basis: docs/foundation/data_models.md (Entity Snapshot: schema_version is the version used for snapshot computation; snapshot is reducer output) and docs/foundation/schema_agnostic_design_rules.md (projection follows the active schema declaration, not the read path). Binding controls: docs/architecture/change_guardrails_rules.mdc MUST 1, 5, 6, and 11; docs/foundation/principles.md invariants 5 and 7.

Decision. Accept the Engineering contract: one read model, the active schema, no writes. Do not document the list/snapshot split as intentional. Tighten who the user id is, and what field provenance may return.

Options.

• Leave the stored row on list and re-project only on the single-entity read. That keeps list a cheap table read, and schema_version: "2.0" stays the historical computation stamp in data_models.md. Rejected: the two callers still disagree, which is the bug, and an unscoped row can be another tenant's snapshot.
• Recompute and upsert entity_snapshots on read so later reads need no aligner. Rejected: a read becomes a write, which needs idempotency_key (MUST 11) and is out of scope.

Chosen. Read-time alignment. schema_version is loadActiveSchema(entity_type, authenticated user id) — the schema describe_entity_type loads.

• Contract order. Edit openapi.yaml and run npm run openapi:generate before the handler (MUST 1, MUST 6). Declare projected (boolean) on getFieldProvenance 200. Declare the no-cutoff getEntitySnapshot body as the read model (schema_version, snapshot, raw_fragments), not the entity_snapshots row. Same object on MCP, HTTP, and CLI. No new operationId; keep the existing contract_mappings.ts rows. The no-cutoff response change is breaking and must be named in that release's Breaking changes section.
• Tenant isolation. Entity ids are ent_{sha256(type:name)} and are not tenant-unique (data_models.md). Every public read of entity_snapshots, observations, raw_fragments, and sources on these paths takes the id from getAuthenticatedUserId (HTTP) or NeotomaServer.getAuthenticatedUserId() (MCP). Never req.body.user_id (MUST NOT 5). Today the list snapshot select in queryEntities and the no-cutoff branch of POST /get_entity_snapshot are unscoped; the cutoff branch already resolves the user and returns 401. Match that. A missing authenticated id on a public handler is 401, not an unscoped .single(). queryEntities may still skip the filter when an internal caller passes no user id; public handlers must not. Add a row to tests/security/tenant_isolation_matrix.test.ts: user A, given user B's entity id, receives neither B's snapshot, nor B's raw_fragments, nor B's provenance (MUST 5, GHSA-wrr4-782v-jhwh).
• Idempotency. alignSnapshotToActiveSchema and resolveFieldProvenance do not insert, update, or upsert. No new mutating operation, so no new idempotency_key. A healing snapshot write in this PR violates MUST 11 and the Engineering out-of-scope line.
• Credentials. resolveFieldProvenance returns the requested field's value plus projected. It does not return select("*") observation rows and it does not return sources.storage_url (the current HTTP handler returns both). Do not log fragment or field values (MUST NOT 11). A schema-load throw fails the read with the existing error envelope. Only a resolved "no active schema" keeps the stored field set (principle 7). projected: false is success, not an error, and includes no sibling fields.
• Instructions. Read-back is ambiguous unless agents are told: missing from snapshot but present in raw_fragments, or projected: false, means stored but not declared — not a failed write. Mirror that rule in docs/developer/mcp/instructions.md and docs/developer/cli_agent_instructions.md. Update the retrieve_field_provenance line in docs/developer/mcp/tool_descriptions.yaml. Assumption: a caller that needs an undeclared value reads raw_fragments or projected: false. Revise if a user-scoped schema must not hide a field the operator has to see on every read.

Reversibility. Remove the read-time aligner. No schema migration and no row repair.

Verdict. SIGNED_OFF. current_owner stays pavo because live gate_status.pm is still pending.