Conversation
The date in a snapshot's name is the clock of whichever host took it, so two hosts whose clocks disagree disagree on which snapshot is the most recent. The filesystem knows: it hands out subvolume ids in creation order, and marks with a Received UUID the subvolumes that arrived through btrfs receive. This reads both, for one subvolume or for a whole directory, and also as one shell command a remote host can be asked to run over ssh, so that both sides describe a directory the same way. The output of btrfs subvolume show is now read field by field instead of by line number, since newer versions print more lines than the parser assumed, and the list of snapshots nobody used is no longer returned.
A receive asked the other host for the names it keeps and took the one whose date was the highest. That date is the clock of whichever host took the snapshot, so a host whose clock ran ahead passed its copy off as the most recent one, and the README could only advise keeping the clocks in agreement. The host is now asked to describe its snapshots the way btrfs numbers them, in the order they were created or received there, and the last one is fetched. The parent of the incremental transfer is read the same way: the last earlier one both sides hold.
A restore snapshotted the volume before replacing it, whatever the volume held: a volume unchanged since its last snapshot left a second, identical one behind, and an empty volume left an empty one. The copy now goes through the rule a snapshot goes through, and names the previous snapshot when nothing changed since it. A volume that already holds the snapshot asked for is left alone, so asking twice does the same as asking once, which is what a mount that restores will need. The snapshot of a volume is also compared with the last snapshot taken of it here, in creation order, rather than with whichever snapshot of that volume carries the latest date. A snapshot received from another host carries the date its host wrote, and was never taken of this volume: comparing with it made a host at rest snapshot its stale copy again the minute a fresher one arrived.
Two hosts can end up with two histories of the same volume: one crashes with writes it never sent, the application moves to the other, which sends its work to the replication host, then the first one comes back. Its next send would carry a snapshot dated after everything over there, and every host reading that directory would take its stale copy for the most recent one. Before sending, the remote host is now asked what the last snapshot of the volume to appear there is, and a send is refused when this host neither holds it nor holds the trace of having exchanged it, with the error saying what to receive first. A volume at rest sends nothing, so it still asks nothing.
A scheduled replication was three calls the scheduler made in turn, a snapshot, a send and a cleanup when the send failed, and the set of replications under way lived in the scheduler. The unmount of a volume is about to replicate too, from inside the plugin, and has to wait for a scheduled replication of the same volume before sending the final state, where the scheduled round skips its turn. So the three calls become one function in the plugin, with the lock beside it, reachable through a new endpoint that the scheduler calls and that gives the command line buttervolume replicate: what moving an application by hand needed as two commands.
Moving an application between two hosts was done by hand: stop it, send a last snapshot, restore that snapshot on the other host, start it there. Docker Swarm moves a service without asking, and its volume stayed behind. On a volume with a replication scheduled, the first mount now asks the replication host for the last snapshot of the volume to appear there, receives it and restores it when it is the last to have appeared here and came from another host, keeping what the volume held; the last unmount snapshots the volume and sends it. The last snapshot is read from the order BTRFS created them in, so a host whose clock runs ahead cannot pass its copy off as the newest, and a snapshot taken here since anything was received means the volume's own history goes on: a container restarting on the same host keeps what it wrote. A host that does not answer refuses the mount, since mounting on a silence is how a stale copy buries the work done elsewhere, and pausing the line is the way to mount without asking. Docker calls the plugin once per container, so the mounts are counted, and every scheduled round of a volume no container uses here fetches what appeared on the other host, because Docker gives a mount thirty seconds and a whole volume does not fit in them.
A Docker Swarm service lands on whatever host has room, and the volume is created there on the spot: nobody can write its replication in the schedule of that host ahead of time. An option named after an action, replicate:node2=1, now writes that line in the schedule when the volume is created, and when it already exists, since a service deployed again onto a host that kept the volume has to find its replication scheduled all the same. A line already there is left alone, paused or not. An option that is neither copyonwrite, compression nor an action is refused where it was ignored: an option nobody read would leave a replication unscheduled and nothing said, and the volume is not created before its options are read, so a bad one leaves nothing behind.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moving an application between two hosts was done by hand: stop it, send a last snapshot, restore that snapshot on the other host, start it there. Docker Swarm moves a service without asking, and its volume stayed behind. This is the proposal of #68 and the subject of #69, built on the
replicate:<host>line rather than on a new verb, since what it does is replication with the handover automated, and "sync" is the name the README gives to the other thing.What changes for a volume with a
replicate:<host>line scheduledEvery
replicate:<host>line already inschedule.csvchanges meaning with this version, and the changelog says so.What had to move underneath
Received UUID, never from the date in the name: a host whose clock runs ahead can no longer pass its copy off as the most recent one.buttervolume receivefollows the same rule.VolumeBackupis always in the answer, and aRestoredfield says whether the volume was replaced.buttervolume replicate <host> <volume>snapshots and sends in one step, through a new/VolumeDriver.Replicateendpoint the scheduler now calls; the replication lock lives in the plugin, where the unmount needs it.-o replicate:node2=1, which is how a Swarm service says once what happens to its volume on whatever host it lands on. An option nobody reads is now refused.The README has a new section, "Move an application between hosts", and AGENTS.md says that a mount the user scheduled may replace a volume, keeping what it held.
Verified
./test_local.sh: 129 passed, 11 skipped (the ones over ssh).Received UUID, and the size of a BTRFS directory is zero exactly when it is empty.Not verified here, and worth doing on a machine where the plugin can be replaced: two containers on one volume as seen by Docker, a mount refused as seen by Docker and Swarm, and the duration of a first mount against
docker plugin enable --timeout. The plugin enabled on the machine this was written on holds volumes, and a second one would share its directories and its ssh port.The design was reviewed before implementation by a separate session, which found the four problems the plan now records: dates versus creation order, "never written" versus "empty", one Mount per container, and the option read after the "volume already exists" return. One more surfaced while writing the tests: fetching ahead of the mount made the send barrier pass by mere presence of the name, so the barrier reads the BTRFS lineage instead.
Supersedes #69, whose four side fixes were merged earlier as #110, #111, #112 and #113.