Skip to content

fix: publish scratch artifacts to content-addressed storage - #574

Open
harshitethic wants to merge 2 commits into
Hebbian-Robotics:mainfrom
harshitethic:fix/durable-enrichment-artifacts
Open

harshitethic wants to merge 2 commits into
Hebbian-Robotics:mainfrom
harshitethic:fix/durable-enrichment-artifacts

Conversation

@harshitethic

Copy link
Copy Markdown
Contributor

Fixes #545.

Summary

  • detect enrichment artifacts that still live under the run scratch directory;
  • publish those artifacts under durable artifacts/ storage instead of exposing scratch URIs;
  • include the artifact SHA-256 in the published key so changed bytes get a different durable URI;
  • tighten camera-video regression coverage to reject scratch-bound URIs.

Why

Scratch is rewritten across reprocessing. A catalog URI that points into scratch can therefore start resolving to different bytes or disappear entirely. Content-addressed durable keys keep prior catalog URIs bound to the bytes that were published.

Testing

  • updated camera-video path assertions to require durable, content-addressed artifact paths;
  • full repository CI can exercise the integration suite.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix itself is right, and both halves hold under mutation: forcing artifact_is_scratch_bound = False reddens 1, and dropping artifact_content_digest from the key reddens 1. publish does copy when source and destination differ, so the artifact really does leave scratch rather than just getting a nicer-looking URI.

Two things before this can land.

uv run ruff format --check fails on src/hflow/app.py, so this would turn main red:

-   artifact_name_digest = hashlib.sha256(artifact_name.encode()).hexdigest()[:8]
+   artifact_name_digest = hashlib.sha256(artifact_name.encode()).hexdigest()[
+       :8
+   ]

And step_directory and artifact_name_digest are now computed twice, identically, in the two branches. Only the key layout differs between them. Hoist those two out above the if so the next person changing the naming scheme cannot change one copy and miss the other.

Worth a sentence in the code on why the non-scratch fallback stays non-content-addressed, since a reader hitting the second branch will wonder whether that is deliberate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: camera_video publishes scratch-bound artifact URIs as durable, causing silent byte-replacement or hard deletion on sync rewrite

2 participants