Files
spinifex/internal
Bryn Mailer 5120aceaf6 feat(ebs): drain the volume inside each provider's CreateSnapshot (#1481)
* feat(ebs): drain the volume inside each provider's CreateSnapshot

viperblockd now drains a mounted volume's export over snapshot.sock in the
background snapshot work, before reading the live checkpoint. A refused or
stale socket fails the snapshot; no socket with the volume lock free means no
client is connected and nothing is buffered; no socket with the lock held is a
client between connections and is reported as retry-later.

Write-through providers drain nothing: the control-plane DrainVolume and the
daemon's drain handler skip the snapshot.sock drain when the node's EBS
provider is not viperblockd, so a snapshot or CreateImage of an attached
volume on qemunbd no longer fails. The control-plane drain stays for
viperblockd nodes that predate the provider drain.

The conformance suite gains a case that CreateSnapshot answered on the queue
subject by a node other than the volume's holder is refused, while the
holder's own snapshot succeeds, run against viperblockd through a third node
reachable only on the queue subjects.

* fix(ebs): refuse a snapshot whose last export client left unsealed WAL

The plugin releases its volume lock even when the seal on disconnect fails,
so a free lock alone did not show the last client's writes reached the
backend. With no drain socket and the lock free, the snapshot now also
requires that the volume holds no local WAL, or that a seal receipt attests
to it.

A socket path of 108 bytes or more cannot be listened on, so a held lock
there is reported as a failed drain rather than as retry-later, and the path
bound is now inclusive. The drain logs with the request context.

* fix(qemunbdd): snapshot a published volume through its export

qemu-nbd holds a published volume's qcow2 open with the write lock, so
qemu-img convert of the file failed with "Failed to get shared write lock",
and reading it with -U would race the metadata qemu-nbd caches. A published
volume is now copied from the NBD export, which serves exactly what the
attached client wrote; an unpublished volume keeps copying the file.

The reader needs a second connection beside the attached client. Any
--shared limit above 1 makes qemu-nbd advertise multi-conn, after which a
client opening several connections blocks on those over the limit, so the
export now accepts any number of clients.

* fix(ebs): require retry-later from the queue-subject snapshot refusal

The conformance case now requires the refusal to carry ErrUnavailable rather
than any error. The viperblockd harness returns a struct instead of five
positional values and stands its queue-only node up only for the case that
asks for it.

* docs(ebs): say only the snapshot accept reply is freed of the drain

* fix(ec2): warn when a drain is skipped for a provider not known to write through

Only viperblockd serves the drain socket, so the skip itself is unchanged, but
it is logged at Info only for qemunbd and at Warn for any other provider. The
daemon logs the host-side skip under its own message.

* fix(ec2): log a provider snapshot refusal and surface retry-later as ServiceUnavailable

CreateSnapshot flattened every provider failure into an unlogged
ServerInternal. It now logs the provider error with the volume and snapshot
ids, and maps a provider ErrUnavailable to ServiceUnavailable as the delete
path already does. CreateImage's snapshot step shares the mapping.

* fix(qemunbdd): refuse a snapshot of a published volume

Reading a published volume through its export needed qemu-nbd to accept more
than one client, which drops the guard against a second writer, and the copy
it made of a live export was not point-in-time. CreateSnapshot now refuses a
published volume with volume_in_use before running qemu-img, and the export
is back to a single client. An unpublished volume still snapshots from its
file.

* refactor(ec2): drop the control-plane snapshot drain

Every viperblockd in a cluster drains the volume inside its own
CreateSnapshot, so CreateSnapshot and CreateImage no longer dial
snapshot.sock or route a DrainVolume command to the instance's host.

Removed: DrainVolume, DrainVolumeSocket, DrainsThroughSocket and the
skipped-drain logging in the snapshot domain; the state.json read and
the volumestate package only it used; the daemon's DrainVolume handler
and the DrainVolume command, payload and response in ec2v1. CreateImage
now snapshots the root volume through one provider path whether or not
the instance is running.

Tests assert that both EC2 paths snapshot an attached volume through the
provider without opening the drain socket or messaging the host, and
that viperblockd drains a mounted volume exactly once.

* fix(viperblock): prove no local writer before an unmounted snapshot

The unmounted branch of CreateSnapshot opened its own engine once the lease
was held, but the lease is reentrant on the volume's own host, so an nbdkit
left running unregistered after a failed recovery rebuild kept taking writes
the snapshot never saw. The branch now takes the volume lock before the open
and holds it until the snapshot is read; a held lock is drained through the
export's socket, or refused as retry-later when it serves none.

A seal receipt now approves local WAL only if it is newer than every change
under the WAL, so a receipt from an earlier connection no longer covers WAL a
later one failed to seal. A refusal for a held lock with no drain socket is
logged at WARN with the socket path.

* fix(ec2): map a provider volume-in-use snapshot refusal to IncorrectState

A provider that cannot snapshot an attached volume, as qemunbdd refuses a
published one, surfaced as ServerInternal. CreateSnapshot and the CreateImage
snapshot step now return IncorrectState telling the caller to detach the
volume first.

* docs(e2e): describe the stopped CreateImage path as the single snapshotVolume path

* chore(viperblock): order the io/fs import as gofmt does
2026-11-03 16:38:47 +10:00
..
2026-09-30 11:48:27 +10:00
2026-08-20 10:52:03 +10:00