Receive a snapshot from a host that has it

CHANGES.rst

@@ -4,6 +4,23 @@ CHANGELOG
 3.13 (unreleased)
 *****************
 
+- Buttervolume can now receive a snapshot from another host, where it could
+  only send one. ``buttervolume receive <host> <volume>`` fetches the most
+  recent snapshot that host keeps of that volume, incrementally when the two
+  still share an older one, and prints its name. It restores nothing: which
+  snapshot becomes the volume stays a separate decision.
+
+  A host that keeps no snapshot of the volume and a host that could not answer
+  are two different answers. The listing raises when ssh fails or takes too
+  long, so an empty answer only ever means a host that answered and holds
+  nothing. Reading silence as "there is nothing over there" is how the good
+  copy of a volume gets replaced by an older one.
+
+  The trace ``<volume>@<datetime>@<host>`` is written after a receive as it is
+  after a send, since a snapshot that just arrived from a host is a snapshot
+  that host holds. Without it, the first send back would carry the whole
+  volume again.
+
 - A purge no longer deletes what a replication needs. The trace of the last
   send to a host is a snapshot like any other, with a date the purge could
   read, so a retention pattern would delete it along with the snapshot it was

README.rst

@@ -425,6 +425,7 @@ When buttervolume is installed, it provides a command line tool
     restore             Restore a snapshot (optionally to a different volume)
     clone               Clone a volume as new volume
     send                Send a snapshot to another host
+    receive             Receive from another host its last snapshot of a volume
     sync                Synchronise a volume from a remote host volume
     rm                  Delete a snapshot
     purge               Purge old snapshot using a purge pattern
@@ -525,10 +526,12 @@ consuming a lot of bandwith or disk space::
 to live in ``/var/lib/buttervolume/snapshots`` and is replicated to the same path on
 the remote host.
 
-What the remote host already holds is read from the trace kept locally after
-each send, named ``<volume>@<datetime>@<host>``, and nothing is asked of the
-remote host. So a snapshot whose trace is there is not sent a second time, and
-a copy deleted on the remote host behind Buttervolume's back goes unnoticed:
+What the remote host already holds is read from the trace kept locally,
+named ``<volume>@<datetime>@<host>``, and nothing is asked of the remote host.
+That trace is written after each send, and after each receive from that host,
+since a snapshot that has just arrived from a host is a snapshot that host
+holds. So a snapshot whose trace is there is not sent a second time, and a
+copy deleted on the remote host behind Buttervolume's back goes unnoticed:
 delete the trace as well, and the next send carries the whole volume again.
 
 A replication scheduled on a volume at rest therefore costs nothing at all: the
@@ -552,6 +555,39 @@ The default SSH_PORT of the ssh server included in the plugin is **1122**. You c
 change it with `docker plugin set ccomb/buttervolume SSH_PORT=<PORT>` before
 enabling the plugin.
 
+Receive a snapshot from another host
+------------------------------------
+
+The other direction, for a host that wants back what another one holds::
+
+    buttervolume receive <host> <volume>
+
+It names a **volume**, where ``send`` names a snapshot: whoever receives does
+not know what the other host has, which is precisely the question this asks.
+The most recent snapshot that host keeps of that volume is fetched into
+``/var/lib/buttervolume/snapshots``, and its name is printed. Only the
+difference crosses the network when the two hosts still share an older
+snapshot to build on.
+
+The command **does not restore anything**. It brings a snapshot over, and
+which snapshot becomes the volume stays a separate, explicit decision::
+
+    buttervolume receive node2 www
+    buttervolume restore www
+
+A host that keeps no snapshot of that volume and a host that could not answer
+are two different answers, and never the same one. An unreachable host, a
+refused connection, an ssh that takes too long: each is reported as the error
+it is. Only a host that answered and holds nothing is reported as holding
+nothing. Reading silence as "there is nothing over there" is how the good copy
+of a volume gets replaced by an older one.
+
+A snapshot carries the moment it was taken on the machine that took it, and
+"the most recent" is read from that name. A host whose clock runs ahead
+therefore passes its copy off as the most recent one, so the hosts replicating
+to each other should agree on the time.
+
+
 Synchronize a volume from another host volume
 ---------------------------------------------
 

buttervolume/api.py

@@ -155,6 +155,16 @@ def send(snapshot, host, test=False):
     return get_from(_post("/VolumeDriver.Snapshot.Send", payload, test), "") is not False
 
 
+def receive(volume, host, test=False):
+    """Fetch the last snapshot another host has of a volume, and answer its name."""
+    payload = {"Name": volume, "Host": host}
+    if test:
+        # the plugin reads it to fetch the snapshot from next door rather than
+        # from another machine
+        payload["Test"] = True
+    return get_from(_post("/VolumeDriver.Snapshot.Receive", payload, test), "Snapshot")
+
+
 def sync(volumes, hosts, test=False):
     """Pull these volumes back from these hosts, and answer whether it went."""
     payload = {"Volumes": volumes, "Hosts": hosts}

buttervolume/cli.py

@@ -186,6 +186,13 @@ def send(args, test=False):
     return api.send(args.snapshot[0], args.host[0], test=test)
 
 
+def receive(args, test=False):
+    res = api.receive(args.volume[0], args.host[0], test=test)
+    if res:
+        print(res)
+    return res
+
+
 def sync(args, test=False):
     return api.sync(args.volumes, args.hosts, test=test)
 
@@ -356,6 +363,16 @@ def main():
     parser_send.add_argument("host", metavar="host", nargs=1, help="Host to send the snapshot to")
     parser_send.add_argument("snapshot", metavar="snapshot", nargs=1, help="Snapshot to send")
 
+    parser_receive = subparsers.add_parser(
+        "receive", help="Receive the last snapshot another host has of a volume"
+    )
+    parser_receive.add_argument(
+        "host", metavar="host", nargs=1, help="Host to receive the snapshot from"
+    )
+    parser_receive.add_argument(
+        "volume", metavar="volume", nargs=1, help="Volume whose snapshot to receive"
+    )
+
     parser_sync = subparsers.add_parser("sync", help="Sync a volume from other host(s)")
     parser_sync.add_argument("volumes", metavar="volumes", nargs=1, help="Volumes to sync (1 max)")
     parser_sync.add_argument(
@@ -423,6 +440,7 @@ def main():
     parser_restore.set_defaults(func=restore)
     parser_clone.set_defaults(func=clone)
     parser_send.set_defaults(func=send)
+    parser_receive.set_defaults(func=receive)
     parser_sync.set_defaults(func=sync)
     parser_remove.set_defaults(func=remove)
     parser_purge.set_defaults(func=purge)

buttervolume/config.py

@@ -60,11 +60,15 @@ def getconfig(config, var, default):
             continue  # Try next driver name
 
 TIMER = int(getconfig(config, "TIMER", 60))
-# How long each external command may legitimately take, in seconds. These two
+# How long each external command may legitimately take, in seconds. These three
 # are not read from the configuration file, but they are delays like the next
 # one and are read next to it.
 SYNC_TIMEOUT = 30
 RSYNC_TIMEOUT = 600
+# How long a remote host has to answer a question that is not a transfer, such
+# as saying which snapshots it keeps. Not SEND_TIMEOUT: waiting ten minutes for
+# a listing would hold up whoever asked for it just as long.
+REMOTE_TIMEOUT = 30
 # The send crosses the network, so its limit is configurable like the rest
 SEND_TIMEOUT = int(getconfig(config, "SEND_TIMEOUT", 600))
 DTFORMAT = getconfig(config, "DTFORMAT", "%Y-%m-%dT%H:%M:%S.%f")

buttervolume/names.py

@@ -173,3 +173,43 @@ def sent_snapshots(volume, host, names):
         (s for s in parsed(names) if s.volume == volume and s.host == host),
         key=str,
     )
+
+
+def _taken_snapshots(volume, names):
+    """The snapshots taken of this volume among these names, the traces left out.
+
+    A trace of a send is named after the snapshot it was made from, plus the
+    target, so `www@2026-08-26T10:00:00.000000@node3` sorts after the snapshot
+    itself and would pass for the most recent one.
+
+    Every name of this volume is read, and one that cannot be read raises
+    rather than being dropped. These names come from another machine, and a
+    listing we could not read must never be answered as a host with nothing.
+    """
+    return [s for s in map(Snapshot.parse, snapshots_of(volume, names)) if not s.host]
+
+
+def snapshot_to_fetch(volume, remote_names, local_names):
+    """What to ask that host for, and the parent both sides already hold.
+
+    The pair `(snapshot, parent)`, or None when the host holds no snapshot of
+    this volume. The parent is the most recent older snapshot the two sides
+    have in common, which is what an incremental transfer is built on; None
+    when they have none, and then the whole volume has to come over.
+
+    The parent is read from the two listings rather than from the trace of a
+    send, which says what we once sent there. The question here is what that
+    host has now, and it has just been asked.
+    """
+    remote = _taken_snapshots(volume, remote_names)
+    if not remote:
+        return None
+    # what we hold is read as plain names, none of which has to make sense: a
+    # name saying the same thing as one over there is the same snapshot, and a
+    # name we could not have written says the same thing as none of them. So a
+    # stray file in our own directory stops nothing, the way it stops nothing
+    # anywhere else this directory is read.
+    held = set(local_names)
+    # never the one being fetched, which cannot be its own parent
+    common = [s for s in remote[:-1] if str(s) in held]
+    return remote[-1], (common[-1] if common else None)