Receive a snapshot from a host that has it - #113
Merged
Conversation
The send endpoint wrote the trace of what the remote host now holds, then deleted the traces of the sends before it. A receive has to say the same thing, since a snapshot that has just arrived from a host is a snapshot that host holds, and two copies of that reasoning would drift apart. Nothing changes for a send. The old traces are read from the directory again rather than from the list gathered before the transfer, which is the same list: the endpoint refuses earlier when the snapshot is already there, so nothing writes a trace for that host in between.
Deciding what to fetch from another host starts with knowing what that host has. The version of this in the pull request that proposes a synchronization at mount time reads the standard output of ssh and never looks at its return code, so a host that is down answers an empty listing, which reads as a host that keeps no snapshot. Acting on that is how the good copy of a volume gets replaced by an older one while the machine holding it was simply unreachable. The listing therefore raises when ssh fails and when it takes too long, and an empty answer only ever means a host that answered and keeps nothing. The whole directory is listed rather than a volume@* pattern, because ls leaves with the same status when nothing matches as when it failed, and because the pattern would carry a name into a shell on the other machine. Choosing which snapshot to fetch, and which one both sides already hold to build an incremental transfer on, is a computation on two lists of names. It lives with the naming rules, where it is tested without ssh or BTRFS. Only the snapshots count, never the traces of a send: a host keeps its own www@date@node3 next to them, that name sorts after the snapshot it was made from, and fetching it would write the trace of a send nobody made.
Buttervolume could send a snapshot to another host and never fetch one back, so a container moved to a second machine started on the empty volume of that machine while its data sat next door. `buttervolume receive <host> <volume>` is the other direction. It names a volume where send names a snapshot, because whoever receives does not know what the other host has, which is the question being asked. The most recent snapshot that host keeps is fetched, incrementally when the two sides still share an older one to build on. It restores nothing: which snapshot becomes the volume is a separate decision, and this one is not entitled to make it. A btrfs receive that stops early leaves behind what it was writing, under the name the whole snapshot would have carried and still open to writing. So the answer to "do we have it already" cannot be the presence of that name, the way the send side can read a trace it only writes after a transfer went through. A subvolume that is not read-only is what a receive left half written, and it is named in the error rather than deleted behind the back of whoever might want to look at it. The one this call created itself is taken away, so a failed transfer does not block every later one, and the lock is what makes "this one is mine" true. The trace of the exchange is written afterwards, as after a send. Without it, a host that received a volume would send the whole of it back the first time it changed, to the host it came from.
Reading what we already hold went through the same strict parsing as the listing from the other host, so any stray file next to our own snapshots stopped every receive of that volume, saying the name was invalid while the other host had answered perfectly well. Nothing was gained by it. That listing is only read to know which snapshot both sides hold, and a name we could not have written says the same thing as none of the names over there. It is now compared as plain text, and a stray file stops nothing, the way it stops nothing anywhere else this directory is read. The other listing keeps its strictness, which is the whole point: a listing we could not read must never be answered as a host with nothing.
ccomb
added a commit
that referenced
this pull request
Sep 2, 2026
Moving an application between two hosts was done by hand: stop it, send a last snapshot, restore that snapshot on the other host, start it there. Docker Swarm moves a service without asking, and its volume stayed behind. This is the proposal of #68 and the subject of #69, built on the `replicate:<host>` line rather than on a new verb, since what it does is replication with the handover automated, and "sync" is the name the README gives to the other thing. **What changes for a volume with a `replicate:<host>` line scheduled** - The first container to use the volume on a host asks that host for the last snapshot of the volume to appear there, receives it, and restores it when it is the last to have appeared here and came from another host. What the volume held is kept as a snapshot first. A host that does not answer refuses the mount, and the error says how to mount without asking: pause the line. - The last container to stop snapshots the volume and sends that snapshot, waiting for a scheduled replication under way so that what leaves is the final state. - While no container uses the volume here, each scheduled round fetches what appeared on the other host, without restoring it, so that a mount receives a difference and not a whole volume within the thirty seconds Docker gives it. - A send refuses to bury a history this host never saw: a host that crashed with unsent writes and comes back after the application ran elsewhere keeps those writes aside, and the work done elsewhere wins. **Every `replicate:<host>` line already in `schedule.csv` changes meaning with this version**, and the changelog says so. **What had to move underneath** - The order of snapshots is read from the order BTRFS created them in, and whether one came from another host from its `Received UUID`, never from the date in the name: a host whose clock runs ahead can no longer pass its copy off as the most recent one. `buttervolume receive` follows the same rule. - A restore keeps what the volume held the way a snapshot is kept: nothing new when the volume is unchanged since its last snapshot, nothing at all when it is empty, and nothing done when the volume already holds the snapshot asked for. `VolumeBackup` is always in the answer, and a `Restored` field says whether the volume was replaced. - A snapshot is compared with the last snapshot taken of the volume here, not with a received one. - `buttervolume replicate <host> <volume>` snapshots and sends in one step, through a new `/VolumeDriver.Replicate` endpoint the scheduler now calls; the replication lock lives in the plugin, where the unmount needs it. - A volume can ask for its scheduled jobs as it is created, `-o replicate:node2=1`, which is how a Swarm service says once what happens to its volume on whatever host it lands on. An option nobody reads is now refused. The README has a new section, "Move an application between hosts", and AGENTS.md says that a mount the user scheduled may replace a volume, keeping what it held. **Verified** - `./test_local.sh`: 129 passed, 11 skipped (the ones over ssh). - The whole suite in the plugin image, ssh included: 140 passed, nothing skipped, nothing failed. - Two BTRFS facts the design rests on were checked on a real filesystem before writing the code: a writable snapshot of a received subvolume clears its `Received UUID`, and the size of a BTRFS directory is zero exactly when it is empty. **Not verified here**, and worth doing on a machine where the plugin can be replaced: two containers on one volume as seen by Docker, a mount refused as seen by Docker and Swarm, and the duration of a first mount against `docker plugin enable --timeout`. The plugin enabled on the machine this was written on holds volumes, and a second one would share its directories and its ssh port. The design was reviewed before implementation by a separate session, which found the four problems the plan now records: dates versus creation order, "never written" versus "empty", one Mount per container, and the option read after the "volume already exists" return. One more surfaced while writing the tests: fetching ahead of the mount made the send barrier pass by mere presence of the name, so the barrier reads the BTRFS lineage instead. Supersedes #69, whose four side fixes were merged earlier as #110, #111, #112 and #113.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Buttervolume could send a snapshot to another host and never fetch one back, so
a container moved to a second machine started on the empty volume of that
machine while its data sat next door.
buttervolume receive <host> <volume>isthe other direction. It is the piece a synchronization at mount time needs,
extracted from #69 on its own, and it does not touch mount or unmount.
It names a volume where
sendnames a snapshot, because whoever receives doesnot know what the other host has, which is the question being asked. The most
recent snapshot that host keeps is fetched, incrementally when the two sides
still share an older one to build on, and its name is printed. It restores
nothing: which snapshot becomes the volume is a separate decision, and this
command is not entitled to make it.
A host that could not answer is never read as a host with nothing. The
version of this in #69 reads the standard output of ssh and never looks at its
return code, so a host that is down answers an empty listing, which reads as a
host that keeps no snapshot. Acting on that is how the good copy of a volume
gets replaced by an older one while the machine holding it was simply
unreachable. Here the listing raises when ssh fails and when it takes too long,
and an empty answer only ever means a host that answered and keeps nothing.
The whole directory is listed rather than a
volume@*pattern, becauselsleaves with the same non-zero status when nothing matches as when it failed,
and because the pattern would carry a name into a shell on the other machine.
A trace is not a snapshot. Choosing what to fetch is a computation on two
lists of names, so it lives with the naming rules and is tested without ssh or
BTRFS. Only the snapshots count: a host keeps its own
www@date@node3next tothem, that name sorts after the snapshot it was made from, and fetching it
would write locally the trace of a send nobody made, which the next send to
node3 would then build on.
What a receive leaves half written carries the name of a whole snapshot. So
the answer to "do we have it already" cannot be the presence of that name, the
way the send side can read a trace it only writes after a transfer went
through. A subvolume that is not read-only is what a receive left behind, and
it is named in the error rather than deleted behind the back of whoever might
want to look at it. The one this call created itself is taken away, so a failed
transfer does not block every later one, and a lock is what makes "this one is
mine" true.
The exchange is remembered. The trace
<volume>@<datetime>@<host>iswritten after a receive as it is after a send, since a snapshot that just
arrived from a host is a snapshot that host holds. Without it, a host that
received a volume would send the whole of it back the first time it changed, to
the host it came from. Writing that trace now has one place instead of two.
Verified:
96 passed, 10 skippedundertest_local.shand106 passedunder./test.sh. Twelve new tests, among them two that restore what came backand read the file, one that proves an unreachable host is an error and not an
empty host, one that proves the incremental transfer asks for the parent both
sides hold and never falls back to the whole volume, and one that proves a
received snapshot is not sent straight back.
One thing those tests cannot prove, and the comment in them says so: both hosts
share a filesystem on the test bench, so the parent of an incremental receive
is found there by its received UUID. Between two real machines the local parent
is the original rather than a copy, and
btrfs receivefalls back to looking itup by plain UUID. Should that ever fail, the whole volume comes over instead,
which is the same fallback the send side has always had.