feat: add experimental on-demand pinning#11252
Conversation
Add a background checker that automatically pins content when DHT provider counts fall below a configurable replication target and unpins once enough providers exist again after a grace period.
Gated behind Experimental.OnDemandPinningEnabled.
New CLI commands: ipfs pin ondemand {add,rm,ls}
Safety measures:
- storage budget check (respects StorageMax/GCWatermark)
- idle timeout on recursive DAG fetches (2 min without progress)
- pin partitioning via pin name to distinguish on-demand pins from persitent pins.
|
Please focus the review on: 1. Core logic —
|
…adding v0.42 changelog entry
|
@guillaumemichel Please re-run the checks. I addressed the failing ones for overfull lines , typos, and changelog. |
| "github.com/ipfs/kubo/ondemandpin" | ||
| ) | ||
|
|
||
| const onDemandLiveOptionName = "live" |
There was a problem hiding this comment.
the ondemand ls command shows the locally stored pin records (CID, pinned status, timestamps). With ---live, it additionally queries the DHT in real-time to report how many providers currently exist
There was a problem hiding this comment.
DHT queries are sequential. Doing ipfs pin ondemand ls --live on hundreds of CIDs will take a long time. Still handy for debugging and development.
|
Triage note: we run out of time to review this in 0.41 iteration, but we've added it to 0.42. Thank you for your contribution and patience. 🙏 |
Summary
Automatically pin content when DHT provider counts fall below a configurable replication target, and unpin once
replication has been above target for a grace period.
Helps keeping critical data around, without wasting storage on overly replicated CIDs.
The feature is described in this draft: ipfs/specs#532
The feature is gated behind
Experimental.OnDemandPinningEnabled.CLI commands for
ipfs pin ondemand:add-- register CIDs for on-demand pinningrm-- deregister and unpinls-- list registered CIDsDesign
pins to avoid accidental deletion. This implementation uses boxo's pin name
field (
"on-demand") for that.StorageMax * StorageGCWatermark.without receiving new blocks (allows large downloads while skipping dead records).
susceptible to Sybil manipulation. Documented as a known limitation.
Configuration Parameters (TBD)
OnDemandPinning.ReplicationTarget5OnDemandPinning.CheckInterval"10m"OnDemandPinning.UnpinGracePeriod"24h"Tests
Unit tests for the checker and store in
ondemandpin/.Visualization of Feature
(out of scope for this CLI PR, but helpful to get the idea)
