Skip to content

Receive a snapshot from a host that has it - #113

Merged
ccomb merged 4 commits into
masterfrom
a-snapshot-can-be-received-from-another-host
Aug 28, 2026
Merged

Receive a snapshot from a host that has it#113
ccomb merged 4 commits into
masterfrom
a-snapshot-can-be-received-from-another-host

Conversation

@ccomb

@ccomb ccomb commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Buttervolume could send a snapshot to another host and never fetch one back, so
a container moved to a second machine started on the empty volume of that
machine while its data sat next door. buttervolume receive <host> <volume> is
the other direction. It is the piece a synchronization at mount time needs,
extracted from #69 on its own, and it does not touch mount or unmount.

It names a volume where send names a snapshot, because whoever receives does
not know what the other host has, which is the question being asked. The most
recent snapshot that host keeps is fetched, incrementally when the two sides
still share an older one to build on, and its name is printed. It restores
nothing: which snapshot becomes the volume is a separate decision, and this
command is not entitled to make it.

A host that could not answer is never read as a host with nothing. The
version of this in #69 reads the standard output of ssh and never looks at its
return code, so a host that is down answers an empty listing, which reads as a
host that keeps no snapshot. Acting on that is how the good copy of a volume
gets replaced by an older one while the machine holding it was simply
unreachable. Here the listing raises when ssh fails and when it takes too long,
and an empty answer only ever means a host that answered and keeps nothing.
The whole directory is listed rather than a volume@* pattern, because ls
leaves with the same non-zero status when nothing matches as when it failed,
and because the pattern would carry a name into a shell on the other machine.

A trace is not a snapshot. Choosing what to fetch is a computation on two
lists of names, so it lives with the naming rules and is tested without ssh or
BTRFS. Only the snapshots count: a host keeps its own www@date@node3 next to
them, that name sorts after the snapshot it was made from, and fetching it
would write locally the trace of a send nobody made, which the next send to
node3 would then build on.

What a receive leaves half written carries the name of a whole snapshot. So
the answer to "do we have it already" cannot be the presence of that name, the
way the send side can read a trace it only writes after a transfer went
through. A subvolume that is not read-only is what a receive left behind, and
it is named in the error rather than deleted behind the back of whoever might
want to look at it. The one this call created itself is taken away, so a failed
transfer does not block every later one, and a lock is what makes "this one is
mine" true.

The exchange is remembered. The trace <volume>@<datetime>@<host> is
written after a receive as it is after a send, since a snapshot that just
arrived from a host is a snapshot that host holds. Without it, a host that
received a volume would send the whole of it back the first time it changed, to
the host it came from. Writing that trace now has one place instead of two.

Verified: 96 passed, 10 skipped under test_local.sh and 106 passed under
./test.sh. Twelve new tests, among them two that restore what came back
and read the file, one that proves an unreachable host is an error and not an
empty host, one that proves the incremental transfer asks for the parent both
sides hold and never falls back to the whole volume, and one that proves a
received snapshot is not sent straight back.

One thing those tests cannot prove, and the comment in them says so: both hosts
share a filesystem on the test bench, so the parent of an incremental receive
is found there by its received UUID. Between two real machines the local parent
is the original rather than a copy, and btrfs receive falls back to looking it
up by plain UUID. Should that ever fail, the whole volume comes over instead,
which is the same fallback the send side has always had.

ccomb added 4 commits August 27, 2026 23:22
The send endpoint wrote the trace of what the remote host now holds, then
deleted the traces of the sends before it. A receive has to say the same
thing, since a snapshot that has just arrived from a host is a snapshot that
host holds, and two copies of that reasoning would drift apart.

Nothing changes for a send. The old traces are read from the directory again
rather than from the list gathered before the transfer, which is the same
list: the endpoint refuses earlier when the snapshot is already there, so
nothing writes a trace for that host in between.
Deciding what to fetch from another host starts with knowing what that host
has. The version of this in the pull request that proposes a synchronization
at mount time reads the standard output of ssh and never looks at its return
code, so a host that is down answers an empty listing, which reads as a host
that keeps no snapshot. Acting on that is how the good copy of a volume gets
replaced by an older one while the machine holding it was simply unreachable.

The listing therefore raises when ssh fails and when it takes too long, and
an empty answer only ever means a host that answered and keeps nothing. The
whole directory is listed rather than a volume@* pattern, because ls leaves
with the same status when nothing matches as when it failed, and because the
pattern would carry a name into a shell on the other machine.

Choosing which snapshot to fetch, and which one both sides already hold to
build an incremental transfer on, is a computation on two lists of names. It
lives with the naming rules, where it is tested without ssh or BTRFS. Only
the snapshots count, never the traces of a send: a host keeps its own
www@date@node3 next to them, that name sorts after the snapshot it was made
from, and fetching it would write the trace of a send nobody made.
Buttervolume could send a snapshot to another host and never fetch one back,
so a container moved to a second machine started on the empty volume of that
machine while its data sat next door. `buttervolume receive <host> <volume>`
is the other direction.

It names a volume where send names a snapshot, because whoever receives does
not know what the other host has, which is the question being asked. The most
recent snapshot that host keeps is fetched, incrementally when the two sides
still share an older one to build on. It restores nothing: which snapshot
becomes the volume is a separate decision, and this one is not entitled to
make it.

A btrfs receive that stops early leaves behind what it was writing, under the
name the whole snapshot would have carried and still open to writing. So the
answer to "do we have it already" cannot be the presence of that name, the
way the send side can read a trace it only writes after a transfer went
through. A subvolume that is not read-only is what a receive left half
written, and it is named in the error rather than deleted behind the back of
whoever might want to look at it. The one this call created itself is taken
away, so a failed transfer does not block every later one, and the lock is
what makes "this one is mine" true.

The trace of the exchange is written afterwards, as after a send. Without it,
a host that received a volume would send the whole of it back the first time
it changed, to the host it came from.
Reading what we already hold went through the same strict parsing as the
listing from the other host, so any stray file next to our own snapshots
stopped every receive of that volume, saying the name was invalid while the
other host had answered perfectly well.

Nothing was gained by it. That listing is only read to know which snapshot
both sides hold, and a name we could not have written says the same thing as
none of the names over there. It is now compared as plain text, and a stray
file stops nothing, the way it stops nothing anywhere else this directory is
read.

The other listing keeps its strictness, which is the whole point: a listing we
could not read must never be answered as a host with nothing.
@ccomb ccomb added the agent-reviewed pr-cycle ran label Aug 27, 2026
@ccomb
ccomb merged commit b092e7c into master Aug 28, 2026
2 checks passed
@ccomb
ccomb deleted the a-snapshot-can-be-received-from-another-host branch August 28, 2026 08:32
ccomb added a commit that referenced this pull request Sep 2, 2026
Moving an application between two hosts was done by hand: stop it, send
a last snapshot, restore that snapshot on the other host, start it
there. Docker Swarm moves a service without asking, and its volume
stayed behind. This is the proposal of #68 and the subject of #69, built
on the `replicate:<host>` line rather than on a new verb, since what it
does is replication with the handover automated, and "sync" is the name
the README gives to the other thing.

**What changes for a volume with a `replicate:<host>` line scheduled**

- The first container to use the volume on a host asks that host for the
last snapshot of the volume to appear there, receives it, and restores
it when it is the last to have appeared here and came from another host.
What the volume held is kept as a snapshot first. A host that does not
answer refuses the mount, and the error says how to mount without
asking: pause the line.
- The last container to stop snapshots the volume and sends that
snapshot, waiting for a scheduled replication under way so that what
leaves is the final state.
- While no container uses the volume here, each scheduled round fetches
what appeared on the other host, without restoring it, so that a mount
receives a difference and not a whole volume within the thirty seconds
Docker gives it.
- A send refuses to bury a history this host never saw: a host that
crashed with unsent writes and comes back after the application ran
elsewhere keeps those writes aside, and the work done elsewhere wins.

**Every `replicate:<host>` line already in `schedule.csv` changes
meaning with this version**, and the changelog says so.

**What had to move underneath**

- The order of snapshots is read from the order BTRFS created them in,
and whether one came from another host from its `Received UUID`, never
from the date in the name: a host whose clock runs ahead can no longer
pass its copy off as the most recent one. `buttervolume receive` follows
the same rule.
- A restore keeps what the volume held the way a snapshot is kept:
nothing new when the volume is unchanged since its last snapshot,
nothing at all when it is empty, and nothing done when the volume
already holds the snapshot asked for. `VolumeBackup` is always in the
answer, and a `Restored` field says whether the volume was replaced.
- A snapshot is compared with the last snapshot taken of the volume
here, not with a received one.
- `buttervolume replicate <host> <volume>` snapshots and sends in one
step, through a new `/VolumeDriver.Replicate` endpoint the scheduler now
calls; the replication lock lives in the plugin, where the unmount needs
it.
- A volume can ask for its scheduled jobs as it is created, `-o
replicate:node2=1`, which is how a Swarm service says once what happens
to its volume on whatever host it lands on. An option nobody reads is
now refused.

The README has a new section, "Move an application between hosts", and
AGENTS.md says that a mount the user scheduled may replace a volume,
keeping what it held.

**Verified**

- `./test_local.sh`: 129 passed, 11 skipped (the ones over ssh).
- The whole suite in the plugin image, ssh included: 140 passed, nothing
skipped, nothing failed.
- Two BTRFS facts the design rests on were checked on a real filesystem
before writing the code: a writable snapshot of a received subvolume
clears its `Received UUID`, and the size of a BTRFS directory is zero
exactly when it is empty.

**Not verified here**, and worth doing on a machine where the plugin can
be replaced: two containers on one volume as seen by Docker, a mount
refused as seen by Docker and Swarm, and the duration of a first mount
against `docker plugin enable --timeout`. The plugin enabled on the
machine this was written on holds volumes, and a second one would share
its directories and its ssh port.

The design was reviewed before implementation by a separate session,
which found the four problems the plan now records: dates versus
creation order, "never written" versus "empty", one Mount per container,
and the option read after the "volume already exists" return. One more
surfaced while writing the tests: fetching ahead of the mount made the
send barrier pass by mere presence of the name, so the barrier reads the
BTRFS lineage instead.

Supersedes #69, whose four side fixes were merged earlier as #110, #111,
#112 and #113.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-reviewed pr-cycle ran

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant