Support for snapshot sync at mount by Musholic · Pull Request #69 · ccomb/buttervolume

Musholic · 2026-01-30T10:49:18Z

I've implemented the proposal in #68 with a few tests (base mount/umount logics + different usual scenarios). I'm keeping the changes as simple as possible, let me know what you think

The implementation follows this logic:

Add a new schedule named "snapshot_sync", which will be used to know if we want to automatically restore newer snapshots at mount. And at unmount, it will automatically snapshot and send it
This new schedule will work the same way as replicate, in addition to the mount/unmount logic
Support for adding schedules automatically at volume creation: may not be necessary, but it felt really better with it (and this was simple enough to implement). The only downside is that we have to delete the schedules manually if we don't want them anymore, but it should be ok for now
Refactor of several functions, especially since I apparently could not reuse directly the route method, which I suffixed with _req
Prevent the creation of empty snapshots, as it may conflict if the service is moved to another host, but we kept creating snapshots on other hosts. It's using a quick and efficient check based on btrfs send/receive
Prevent sending the same snapshot to the same remote (since we don't necessarily create new snapshots every time we replicate)

I plan to implement the support for multiple hosts in another PR, as I may have to deal with timeouts and connection errors to handle the case where one of the hosts is unresponsive.

I've tested the PR on my own setup, and I think it's ready to be merged. Let me know if you are willing to merge it, and then I can also update the documentation to finish the PR (or in a second PR).

TODO:

Document the new "snapshot_sync" schedule and the new "schedules" option
Fix snapshot still created on "inactive" hosts

…hot_sync

from possible data loss and reduce useless activity)

if we don't have the tracking snapshot

remote

Musholic · 2026-02-06T17:28:48Z

Note that I added several fixes that I felt were necessary. I started using it on several docker swarm services this week, and it's working really great (with the fixes). I'll keep monitoring and testing other common scenarios next week

unmount/mount

Musholic · 2026-03-06T12:59:29Z

There are still some improvements I will continue to work on for this feature (without the multi-host support) but this will make this PR too big to review. It's fine if you don't want to merge it btw. You may not wish to introduce this complexity and I will be maintaining a fork in the meantime for my usage.

Buttervolume could send a snapshot to another host and never fetch one back, so a container moved to a second machine started on the empty volume of that machine while its data sat next door. `buttervolume receive <host> <volume>` is the other direction. It is the piece a synchronization at mount time needs, extracted from #69 on its own, and it does not touch mount or unmount. It names a volume where `send` names a snapshot, because whoever receives does not know what the other host has, which is the question being asked. The most recent snapshot that host keeps is fetched, incrementally when the two sides still share an older one to build on, and its name is printed. It restores nothing: which snapshot becomes the volume is a separate decision, and this command is not entitled to make it. **A host that could not answer is never read as a host with nothing.** The version of this in #69 reads the standard output of ssh and never looks at its return code, so a host that is down answers an empty listing, which reads as a host that keeps no snapshot. Acting on that is how the good copy of a volume gets replaced by an older one while the machine holding it was simply unreachable. Here the listing raises when ssh fails and when it takes too long, and an empty answer only ever means a host that answered and keeps nothing. The whole directory is listed rather than a `volume@*` pattern, because `ls` leaves with the same non-zero status when nothing matches as when it failed, and because the pattern would carry a name into a shell on the other machine. **A trace is not a snapshot.** Choosing what to fetch is a computation on two lists of names, so it lives with the naming rules and is tested without ssh or BTRFS. Only the snapshots count: a host keeps its own `www@date@node3` next to them, that name sorts after the snapshot it was made from, and fetching it would write locally the trace of a send nobody made, which the next send to node3 would then build on. **What a receive leaves half written carries the name of a whole snapshot.** So the answer to "do we have it already" cannot be the presence of that name, the way the send side can read a trace it only writes after a transfer went through. A subvolume that is not read-only is what a receive left behind, and it is named in the error rather than deleted behind the back of whoever might want to look at it. The one this call created itself is taken away, so a failed transfer does not block every later one, and a lock is what makes "this one is mine" true. **The exchange is remembered.** The trace `<volume>@<datetime>@<host>` is written after a receive as it is after a send, since a snapshot that just arrived from a host is a snapshot that host holds. Without it, a host that received a volume would send the whole of it back the first time it changed, to the host it came from. Writing that trace now has one place instead of two. Verified: `96 passed, 10 skipped` under `test_local.sh` and `106 passed` under `./test.sh`. Twelve new tests, among them two that **restore** what came back and read the file, one that proves an unreachable host is an error and not an empty host, one that proves the incremental transfer asks for the parent both sides hold and never falls back to the whole volume, and one that proves a received snapshot is not sent straight back. One thing those tests cannot prove, and the comment in them says so: both hosts share a filesystem on the test bench, so the parent of an incremental receive is found there by its received UUID. Between two real machines the local parent is the original rather than a copy, and `btrfs receive` falls back to looking it up by plain UUID. Should that ever fail, the whole volume comes over instead, which is the same fallback the send side has always had.

Moving an application between two hosts was done by hand: stop it, send a last snapshot, restore that snapshot on the other host, start it there. Docker Swarm moves a service without asking, and its volume stayed behind. This is the proposal of #68 and the subject of #69, built on the `replicate:<host>` line rather than on a new verb, since what it does is replication with the handover automated, and "sync" is the name the README gives to the other thing. **What changes for a volume with a `replicate:<host>` line scheduled** - The first container to use the volume on a host asks that host for the last snapshot of the volume to appear there, receives it, and restores it when it is the last to have appeared here and came from another host. What the volume held is kept as a snapshot first. A host that does not answer refuses the mount, and the error says how to mount without asking: pause the line. - The last container to stop snapshots the volume and sends that snapshot, waiting for a scheduled replication under way so that what leaves is the final state. - While no container uses the volume here, each scheduled round fetches what appeared on the other host, without restoring it, so that a mount receives a difference and not a whole volume within the thirty seconds Docker gives it. - A send refuses to bury a history this host never saw: a host that crashed with unsent writes and comes back after the application ran elsewhere keeps those writes aside, and the work done elsewhere wins. **Every `replicate:<host>` line already in `schedule.csv` changes meaning with this version**, and the changelog says so. **What had to move underneath** - The order of snapshots is read from the order BTRFS created them in, and whether one came from another host from its `Received UUID`, never from the date in the name: a host whose clock runs ahead can no longer pass its copy off as the most recent one. `buttervolume receive` follows the same rule. - A restore keeps what the volume held the way a snapshot is kept: nothing new when the volume is unchanged since its last snapshot, nothing at all when it is empty, and nothing done when the volume already holds the snapshot asked for. `VolumeBackup` is always in the answer, and a `Restored` field says whether the volume was replaced. - A snapshot is compared with the last snapshot taken of the volume here, not with a received one. - `buttervolume replicate <host> <volume>` snapshots and sends in one step, through a new `/VolumeDriver.Replicate` endpoint the scheduler now calls; the replication lock lives in the plugin, where the unmount needs it. - A volume can ask for its scheduled jobs as it is created, `-o replicate:node2=1`, which is how a Swarm service says once what happens to its volume on whatever host it lands on. An option nobody reads is now refused. The README has a new section, "Move an application between hosts", and AGENTS.md says that a mount the user scheduled may replace a volume, keeping what it held. **Verified** - `./test_local.sh`: 129 passed, 11 skipped (the ones over ssh). - The whole suite in the plugin image, ssh included: 140 passed, nothing skipped, nothing failed. - Two BTRFS facts the design rests on were checked on a real filesystem before writing the code: a writable snapshot of a received subvolume clears its `Received UUID`, and the size of a BTRFS directory is zero exactly when it is empty. **Not verified here**, and worth doing on a machine where the plugin can be replaced: two containers on one volume as seen by Docker, a mount refused as seen by Docker and Swarm, and the duration of a first mount against `docker plugin enable --timeout`. The plugin enabled on the machine this was written on holds volumes, and a second one would share its directories and its ssh port. The design was reviewed before implementation by a separate session, which found the four problems the plan now records: dates versus creation order, "never written" versus "empty", one Mount per container, and the option read after the "volume already exists" return. One more surfaced while writing the tests: fetching ahead of the mount made the send barrier pass by mere presence of the name, so the barrier reads the BTRFS lineage instead. Supersedes #69, whose four side fixes were merged earlier as #110, #111, #112 and #113.

ccomb · 2026-09-02T11:33:17Z

Hi, I've restarted development very recently, after one year of experience on agentic engineering. It helped an fixed a lot of problems, bugs, race conditions, in a very short time. If you're still interested you can try building and using it, I've also tried to solve the topic of #68 and #69 with #120. It was really not possible to rebase the code so I've reimplemented the idea using replicate instead of another command.
I'm not doing a release yet, but it will eventually be released as 4.0. I'm also considering a migration path from the former anybox/buttervolume. And I'll start using it soon myself for critical applications.
However I'm not using Swarm and would be very interested in your feedback. Feel free to report here and close this PR. I'm leaving open in the meantime.

Musholic · 2026-09-03T19:56:51Z

Hello, that's good to hear, thanks for your work on this side :).
I'll have a look in detail when possible. I've been using my fork for my homelab, and I still need to work on some issues and features that I need.

You may have already fixed some of my issues (especially with the mounted var and the locks), I'll test and let you know.

I'll probably open issues later, but if you're curious about the additional features I would need, it would be:

The possibility to specify an alternate parent volume location (I personally started switching the schedule.csv to a SQLite DB with Peewee to save both schedules and volume locations in a metadata.db file. It's also safer for concurrent write operations)
Then (when we can specify the alternate location) the possibility of having the snapshot directory in another local btrfs partition, which would allow having the volumes on a fast SSD while the snapshots are on a slower hdd. (This would use local btrfs send/receive under the hood)
Optional: having the snapshots in volume subdirectories (/var/lib/buttervolume/snapshots/<volume_name>/<timestamp>). For easier manual snapshot management and to allow tools like btdu to aggregate the size for each directory. It may have other advantages, but not sure.
Improvement to the purge pattern to reduce the number of snapshots. I personally worked on the following commit which I've not tested yet: Musholic@234cd47#diff-3665d65394f4f58a56a256ad6dd8621c68118d90fe56a19387e251c19cec2d2eR885 . It allows to specify a max number of snapshots per bucket with a pattern like 1h/2:1d/3:1M/5. Otherwise, I end up with hundreds of snapshots per volume, and Btrfs tends to slow down a bit if there are a lot of subvolumes.

Musholic · 2026-09-06T19:23:50Z

I did a review of some parts of the code, and I have a few suggestions and optimisations to share.

I can post a breakdown of my notes right here, but if you prefer a quicker back-and-forth, I would be happy to contact you on another platform like Discord.

(By the way, I realized we're both French, so feel free to reach out in French privately if you prefer 🙂)

And thanks already for implementing my idea of the purge pattern improvement!

Musholic force-pushed the snapshot_sync_at_mount branch 4 times, most recently from 629458b to c3a589f Compare February 2, 2026 21:05

Musholic added 6 commits February 3, 2026 10:37

Support for snapshot sync at mount

c802fff

Make snapshot_sync also act in the same way as replicate

dee3bdb

Handle cases where remote and/or local has no snapshots yet for snaps…

3f17d60

…hot_sync

Prevent the creation of new snapshots when there was no actual changes

36260ef

Prevent sending a snapshot when it's already on remote

a429519

Automatically recover last local snapshot after volume removal

0d890f1

Musholic force-pushed the snapshot_sync_at_mount branch from c3a589f to 0d890f1 Compare February 3, 2026 09:39

Musholic marked this pull request as ready for review February 3, 2026 09:55

Musholic mentioned this pull request Feb 3, 2026

Questions/Proposal on how to achieve high availability with docker swarms #68

Open

Musholic marked this pull request as draft February 3, 2026 10:56

Pause the snapshot_sync schedule when the volume is not mounted (protect

d389362

from possible data loss and reduce useless activity)

Musholic marked this pull request as ready for review February 3, 2026 17:08

Musholic added 4 commits February 5, 2026 09:28

When sending, detect if the snapshot already exists on remote host even

da97766

if we don't have the tracking snapshot

When purging, keep tracking snapshots

397ebd2

Fix snapshot sync when the parent snapshot is no longer available on

1f7cbf9

remote

Fix unmount failure when the snapshot already exists on the remote

d222e12

Fix data loss when receiving new remote snapshots directly between

e4a5462

unmount/mount

ccomb added agent-reviewed pr-cycle ran and removed agent-reviewed pr-cycle ran labels Aug 26, 2026

ccomb mentioned this pull request Aug 27, 2026

Receive a snapshot from a host that has it #113

Merged

ccomb added agent-reviewed pr-cycle ran and removed agent-reviewed pr-cycle ran labels Aug 30, 2026

ccomb mentioned this pull request Sep 2, 2026

Let a replicated volume follow its container between hosts #120

Merged

Navigation Menu

Support for snapshot sync at mount - #69

Support for snapshot sync at mount#69
Musholic wants to merge 12 commits into
ccomb:mainfrom
Musholic:snapshot_sync_at_mount

Musholic commented Jan 30, 2026 •

edited

Loading

Uh oh!

Musholic commented Feb 6, 2026

Uh oh!

Musholic commented Mar 6, 2026

Uh oh!

ccomb commented Sep 2, 2026

Uh oh!

Musholic commented Sep 3, 2026

Uh oh!

Musholic commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Musholic commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Musholic commented Feb 6, 2026

Uh oh!

Musholic commented Mar 6, 2026

Uh oh!

ccomb commented Sep 2, 2026

Uh oh!

Musholic commented Sep 3, 2026

Uh oh!

Musholic commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Musholic commented Jan 30, 2026 •

edited

Loading