Skip to content

Garbage-collect orphaned thumbnails via reference markers - #43

Open
fanchenggang wants to merge 1 commit into
longern:mainfrom
fanchenggang:fix/thumbnail-refcount
Open

Garbage-collect orphaned thumbnails via reference markers#43
fanchenggang wants to merge 1 commit into
longern:mainfrom
fanchenggang:fix/thumbnail-refcount

Conversation

@fanchenggang

Copy link
Copy Markdown

Fixes #16.

Problem

Thumbnails are content-addressed (_$flaredrive$/thumbnails/<digest>.png) and intentionally shared between files whose generated thumbnails have the same digest. delete.ts never removed them, so every deleted image left an orphaned thumbnail behind, forever.

Solution: per-path reference markers

Each referencing file keeps an immutable marker object at _$flaredrive$/thumbnails/refs/<digest>/<path>. A thumbnail is deleted only once no marker remains for its digest. No counters, no read-modify-write: every operation is a single object PUT or DELETE.

operation behavior
PUT / multipart create / COPY write the destination's marker before writing the referencing object
DELETE (file) delete the file, remove its marker, collect the thumbnail if no marker remains
DELETE (directory) remove one marker per deleted child, then collect each touched digest once (duplicates sharing a thumbnail are handled)
overwrite (PUT / COPY over an existing key) release the destination's previous reference after the successful write
delete-all (DELETE /) also wipe the internal _$flaredrive$/thumbnails/ subtree, which listAll skips

Why this is safe

  • Markers are written before the referencing object and removed after the reference is gone, so a concurrent delete of the last other file using a thumbnail can at worst over-retain it (today's behavior) — it can never collect a thumbnail that is still in use.
  • Thumbnails uploaded before this change have no markers, so they are conservatively retained rather than risk breaking legacy duplicates that share them. They only get collected once every file referencing them has been rewritten (or on delete-all).
  • PUT verifies the thumbnail object still exists after referencing it and returns 409 otherwise, closing the window where a concurrent GC removes the blob between the client's thumbnail upload and the file upload — the client can simply retry.
  • The fd-thumbnail header is now validated as a hex digest (16–128 chars) before it is ever used in an object key.

Notes

  • Third-party WebDAV clients benefit too: COPY/MOVE of thumbnail-carrying objects propagates markers, even though they never send fd-thumbnail themselves.
  • Failed multipart uploads leave their marker behind (over-retention only); the same applies to markers written by a PUT that fails on a precondition — those are cleaned up, but an aborted multipart flow is not.

Happy to adjust the marker layout or add a scheduled sweep for legacy orphans if you'd prefer.

Thumbnails are content-addressed and shared between files whose
generated thumbnails have the same digest, so deleting a file could
never remove its thumbnail and they accumulated forever (longern#16).

Each referencing file now keeps an immutable marker object under
_$flaredrive$/thumbnails/refs/<digest>/<path>:

- put / multipart create / copy write the marker before the
  referencing object, so a concurrent delete of the last other file
  using the thumbnail cannot collect it mid-upload
- delete (file or directory) removes the markers first and collects
  a thumbnail only once no marker remains, so duplicates sharing a
  thumbnail are handled and concurrent deletes can at worst
  over-retain (never break a thumbnail still in use)
- overwriting a file releases its previous thumbnail reference
- delete-all also wipes the internal thumbnails subtree, which
  listAll skips
- the fd-thumbnail header is validated as a hex digest
fanchenggang added a commit to fanchenggang/Davflare that referenced this pull request Sep 6, 2026
缩略图按内容摘要寻址并被内容相同的文件共享,原删除逻辑从不清理 _$flaredrive$/thumbnails/。每个引用文件现在用不可变 marker 做引用计数,清零才回收。同步上游 longern/FlareDrive#43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deleted image thumbnails were not deleted

1 participant