Sidecar format improvements.

[AriESQ]

A couple of suggestions on sidecar format/schema.

• Consider choosing a siddecar format that has industry support. This will do a lot of the work of thinking about future use cases / edge cases for you.

• Move from markdown to a structured format, such as YAML. This allows for fast machine processing of the sidecars without going through an LLM inference step.

• Version your sidecar schemas. If you change the schema in the future, it will be helpful to know how to read past sidecar formats.

• Include file hashes for unique identification.

• Use relative paths instead of absolute paths.

• You might be able to use a container format like .mkv to embed the sidecar into a single file.
  👍

[niranjans]

Thanks! Good list.

Quick comments:

• Schema versioning — yes, adding a schema_version field.
• Relative paths — already relative internally; I'll check nothing absolute leaks into a sidecar.
• The sidecar is already YAML frontmatter + prose, so it's machine-readable without an LLM today.
• File hashes — like it for dedup, but hashing multi-GB clips fully is heavy. Leaning toward a cheap fingerprint (size + first chunk). Open to thoughts.
• Embedding into .mkv — thinking of passing on this. Separate sidecars are the point: originals never get rewritten, stays greppable.

Picking up versioning + the path check soon.

[AriESQ]

I saw this, which I think is an absolute path.
https://github.com/Simbastack-hq/framedex/blob/main/README.md?plain=1#L68

Relative paths are so that if you move the files around they still work.

.mkv was just an idea to embed the sidecar into the video file so they stay together rather than having to keep track of two files in pairs.

The whole sidecar should be structured data so that you can use traditional tools to operate over a corpus of files. But another way, what does markdown get you that a structured data format does not?

Fair point about file hashing times. Maybe could use some R&D as to what are best practice. Not just for dedup, but for keeping track of files as they move around. Fuzzy hash might be better for that, in case the file gets' trivial edit that break a full SHA1.

[niranjans]

Good catch on L68 - real bug, not just docs. The path: field is now stored relative to the scan root, and the query/master tools resolve it back when they run. Old sidecars still work.

On YAML-for-everything: the queryable fields are already YAML frontmatter - the tools never parse the body. The body is just the transcript and scene description, i.e. prose. YAML wouldn't make prose any more machine-readable; it'd just make it harder to grep and read. So I'd say we should keep the hybrid.

Hashing: not sold on fuzzy hashing. framedex never edits originals, so "trivial edit breaks SHA1" doesn't come up - and perceptual hashing is slow and gives false positives. Going with a cheap fingerprint (size + hashed head/mid/tail chunks). Near-duplicate detection is interesting but a separate feature - I'll open an issue to track it.

.mkv: passing. Embedding means rewriting originals, which breaks the non-destructive guarantee. Two files side by side is the point.

schema_version and the fingerprint are coming up.

Thanks for the review!