mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
fix: pause a guest on a backend I/O error instead of handing it EIO (#977)
* fix: fail DRAIN when a volume seal failed
shutdownAndUnmount logged the volume unmount error and returned nothing, so
StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set.
The coordinator then advanced to STORAGE and stopped viperblock while the seal
was still in flight, tearing the VBState envelope and leaving the guest unable
to mount its root volume on the next boot. The deploy reported success.
shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not
seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup
and stopOne carry it without aborting the rest of the teardown, since QEMU is
already down by then, and StopAll aggregates the per-VM failures so
handleShutdownDrain ACKs with Error set and never reports vms_stopped.
volumeMounterAdapter.Unmount was the second swallow site: it logged each failed
seal and returned nil unconditionally. It now attempts every volume and returns
the aggregate, which is what the VolumeMounter contract already described.
Terminate still tolerates a failed seal — the volumes are deleted next, so an
unsealed block map loses nothing a caller can ask for.
This does not fix the seal itself. It converts silent data loss into a refused
upgrade that an operator can act on.
* test: add the storagefault e2e suite for backend-outage guest survival
Freezes predastore with SIGSTOP to hold connections open while answering
nothing, which is the fault a timeout-free path cannot distinguish from a
slow peer. On one node that is a total outage; on three it freezes every
node but the guest's own, breaking the RS(2,1) shard floor.
The suite writes and flushes a verifiable region before the fault and
churns a separate unverified region during it, so an interrupted write is
never mistaken for lost data. The root-filesystem load runs from the
user's home rather than /tmp, which is tmpfs on this image and would move
no bytes on the root device.
* test: verify the root device is under write pressure, and size the outage
The previous guard matched /^ *vda / against /proc/diskstats, where the
device name is the third field, and read $7 which is ms_reading rather
than sectors_written. It printed empty and asserted nothing, so a green
run could not distinguish a surviving guest from an idle disk. It now
samples sectors_written twice and fails unless it advances.
The outage is sized from viperblock's absorption rather than guessed:
a workload must fill DefaultMaxPendingBytes before backpressure engages
at all, so a short freeze measures the retry budget rather than the fault.
* test: add nbdkit fault injection to the storage-fault e2e suite
The predastore freeze takes the whole storage cluster away from every
guest on it, which is too broad to run beside anything else. Removing a
single volume's NBD server is the same fault at one-volume scope: one
host, one guest, nothing cluster-wide stopped.
Both signals are here because they sit on opposite sides of the defect.
SIGSTOP holds the socket open and answers nothing, so the guest blocks
and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at
the default reconnect-delay of 0 fails requests at once, which under
werror=report aborts the journal.
Signalling is guarded: a pid is matched on comm, our plugin and the
volume argument together, and re-confirmed immediately before the
signal, because these hosts run nbdkit for other volumes and pids are
reused.
* fix: pause a guest on a backend I/O error instead of handing it EIO
Every virtio-blk device was created with werror=report,rerror=report, so a
backend outage was delivered to the guest as EIO. ext4 aborts its journal on
that and remounts read-only, and the write that failed is simply gone. Under
werror=stop QEMU pauses the VM and holds the failed request, so a backend that
comes back resumes the guest with nothing lost.
The boot drive, the cold-boot data volume and the hot-plugged data volume all
move to stop. Alongside it the NBD transport gets reconnect-delay=30, which
pauses requests rather than failing them while a storage process restarts on
the same socket; on the legacy -drive form it has to be file.reconnect-delay,
because the bare name reaches the raw format driver, which has no such option.
The QMP heartbeat now recognises run state io-error and issues cont. After ten
attempts the instance is reported impaired and the cadence drops to one attempt
per ten heartbeats, but it never stops: QEMU holds the request for as long as
the guest exists, so a backend that returns an hour later can still be resumed.
The storagefault suite gains the fault this was written for — kill the volume's
own nbdkit, hold, then put it back on the same socket — and it now verifies a
sha256 region across the outage rather than trusting the guest to be reachable.
Three harness defects went with it: the socket-liveness poll ran unprivileged
against a 0770 directory and could never succeed, the relaunched socket was
left 0755 where QEMU needs group write, and the two variants shared one guest
so a fault in the first was diagnosed in the second.
* test: cover the io-error resume policy
* test: stop two parallel tests racing on the terminate backoff variable
* fix: report a guest paused on a backend I/O error as impaired
A guest held by werror=stop kept reporting ok on both status checks. QMP stays
responsive while the guest is paused -- cont is being sent and answered -- so
the QMP failure gate that DescribeInstanceStatus keys on never trips, and the
one state a customer most needs to see looked healthy.
Instance status now reports impaired/failed while the resume budget is
non-zero, timestamped from IOErrorSince so impairedSince is when the guest
paused rather than when it was escalated. System status reports impaired too:
a backend I/O error is an infrastructure fault, not a guest one, and that
distinction is what the two checks exist to draw. The counter is cleared by
the first poll that sees the guest running, so both clear on recovery.
* test: reap the relaunched nbdkit when the fault test ends
relaunchNbdkit detaches the server with setsid so it outlives the SSH session
that started it, and viperblockd never learned it exists. Nothing reaped it, so
every run left an nbdkit holding a socket and a cache for a volume that was
later detached or deleted -- four were found running on dev-prod against two
detached and two deleted volumes.
The PID is resolved from the socket path rather than the volume id, so a
legitimate viperblockd server for the same volume is never the one killed.
* test: prove the backend is serving before a fault is called cleared
Thawing the process a fault stopped is not the same as restoring the service.
The outage test treated SIGCONT as the end of the fault and went straight to
judging the guest, so it passed while the cluster it ran on stayed unable to
serve a single object for the next 35 minutes, and the next test would have
inherited that.
thawPredastore now puts and gets an object through every gate, with a budget,
and fails the test if the backend does not come back.
* fix: do not hand back a mount entry whose export is gone
A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit
was gone. Every later ebs.mount then matched that entry and returned its socket
path, so vmMgr waited for a socket nothing would ever create and the instance
could not be started again — on dev-prod a guest stopped against a frozen
backend was left permanently stopped, four start attempts failing identically
on a socket the node had already removed.
Drop the entry when there is positive evidence the export is dead: the socket
is absent, or the recorded pid is not running. Neither can be true of a live
export, so the double-writer hazard the guard exists to prevent is unaffected.
Also make the durability test's pattern helpers report failures. The read dd sat
in a pipeline, so its exit status was discarded and the harness hashed nothing
and returned the empty-string sha256 as a checksum.
* test: report the guest console when a restart never answers SSH
A restart that reaches running but never comes back is indistinguishable from
a dead root filesystem without the serial console, and the log is gone once
the fixture terminates the instance.
* fix: converge guest port datapaths concurrently, not one deadline after another
Each guest port that misses its SB binding burns the full convergence deadline,
and applyEIPs spent them in series. Three ports belonging to long-stopped
instances therefore held a newly launched guest's EIP dark for the sum of them
before its own probe started.
AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and
sb-cluster-state-reset is now serialised with a minimum interval so concurrent
probes cannot each fire a cluster-wide resync.
* fix: refuse a stale cross-node start after the owning node is gone
The volume lease already refused a cross-node mount, but it is a liveness
claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded
writes stops renewing, and 45 seconds later nothing refuses a start elsewhere
that loads a stale backend checkpoint.
A failed seal now writes a durable marker naming the node holding the only
current copy, cleared on a seal that reaches the backend and on delete. The
lease covers a live holder, the marker a dead one.
The refusal also reaches the caller instead of being flattened to
ServerInternal, which told an operator to retry something that cannot succeed
while the holder is down.
* fix: keep the daemon's message when a start is refused
Rebuilding the error from the code alone rendered the generic ErrorLookup
text, so a refusal naming the node holding the volume reached the caller as
boilerplate about attaching volumes.
* fix(ec2): terminate detaches every DeleteOnTermination=false volume
TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume
on the assumption that Stop's Unmount had already cleared its attachment.
Unmount only clears it when the seal succeeded, so a volume whose seal failed
stayed in-use pointing at an instance that terminate then deleted, leaving it
unreachable with no owner to release it.
The running-instance path already detaches every DoT=false volume; the stopped
path now matches it, and tolerates an already-gone volume the same way.
DeleteVolume also keeps its provider error on the way out. The terminate reaper
retries the delete every two minutes and logged a bare ServerInternal, which
made a blob node being down indistinguishable from a transient fault.
* fix(ec2): DetachVolume --force releases a volume whose instance is gone
The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a
volume left attached by a part-way teardown could never be detached: the
instance no longer runs anywhere and nothing answers. That also made it
undeletable, since DeleteVolume refuses an attached volume.
ForceDetachVolume already existed for this deadlock but was reachable only from
account teardown. Force now falls back to it when the instance has no responder,
which is the one case where clearing the attachment cannot strand a live guest.
* fix(viperblock): unsealed writes steer placement instead of pinning a volume
The dirty marker refused any mount from a node other than the one holding
unsealed writes. Instance start already forwards to the node that last ran the
instance and falls back after a 30s window, so that refusal turned a working
fallback into a volume that could run nowhere once its owner was gone. It
traded a rare data-loss risk for a permanent availability failure.
The marker is now a record and a warning, not a gate. A takeover logs the
previous holder and why it could not seal, then adopts the marker so the old
node is no longer treated as the better source. Live exclusion is unchanged and
still enforced by the lease.
It is also written on open rather than on seal failure. A node killed mid-write
never reaches its seal, so the old trigger left no trace in exactly the case the
marker exists to describe.
Adds 'spx admin volumes unsealed' to see which volumes would cost something if
started elsewhere.
* fix(viperblock): fence a volume whose lease moved instead of logging it
The lease was checked once, at mount, and never again. A node that lost it
kept its export up and kept writing into a volume another node owned, so
both copies advanced and the backend settled it by arrival order.
Losing a renewal now decides: another holder means tear the export down,
no holder means the entry merely aged out, so reclaim it and leave the
guest running. Fencing never seals — this node's copy is the stale one —
and leaves the dirty marker for the winner to take over.
This is not an epoch fence. It narrows the window from the 45s TTL to one
renewal interval at no write-path cost; the backend still cannot refuse a
stale writer.
* fix(network): advertise EIPs so a recycled address is reachable
The gateway switch port only asked ovn-northd to publish the router's NAT
addresses in centralised and routed modes, so on a distributed cluster nothing
ever announced an elastic IP. OVN still answers ARP for it, which is why a
first-time allocation works and hid this — but no gratuitous ARP is sent when
the address moves to a new ENI, so every host on the segment keeps resolving it
to the previous holder's MAC until its cache ages out. Reusing an address from
the pool then lands on a dead MAC and the instance is unreachable from outside.
Distributed mode needs the option as much as the others: northd scopes each
entry with is_chassis_resident(<vif port>), so the announcement leaves from the
chassis running the instance rather than the gateway, which is what distributed
NAT wants.
Converge on the already-attached path too. AttachIGW returns early once the
gateway port exists and nothing else rewrites it, so without that an existing
VPC would stay silent for its whole life.
Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to
carrying their correct external_mac, each scoped to its own ENI's port.
* fix(viperblock): make the volume fence and its dirty marker ownership-safe
Four corrections to the lease-loss fence, all found in review.
Drop the reacquire path. Losing a lease and finding no successor was treated as
a lapsed entry worth reclaiming, but leases.acquire returns the cached entry —
already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease
renewed nothing and the node went on writing a volume it could not prove it
owned. The two errors are not symmetric: a false fence costs one guest a
restart, a false reclaim leaves two nodes writing one volume. Fence in both
cases, and report which happened through the outcome attribute.
Surrender a lease this node cannot confirm. Renewal only noticed a loss when
the server rejected an update, so a node partitioned from NATS kept its export
up while the TTL expired and another node took the volume. Track the last
confirmed renewal and give the volume up once that ages past the point where
the server may admit another owner. Bound the renewal call too, so a hung
JetStream request cannot hold the goroutine past that point.
Do not act on a kill that did not take. Both the fence and ebs.unmount removed
the mount entry, released the lease and announced the result before knowing
nbdkit had exited — the fence then reported a writer stopped that was still
running, and the unmount sealed into a directory a live process was writing.
Wait for the exit, and on failure leave the volume mounted and say so. An
already-exited process now counts as success, which is what the caller asked.
Give the dirty marker a generation. It recorded an owner and nothing to order
two nodes' claims, so a node returning after a takeover could clear or overwrite
the winner's marker and point placement back at the stale copy. Stamp the lease
generation, refuse a write from an older one, and only clear a marker this node
still owns. A marker that cannot be written now fails the open rather than
leaving writes nothing records.
* test(viperblock): cover the stall surrender and the unmount kill guard
The two ownership cases the fence work left untested.
A holder partitioned from NATS while the winner keeps it never gets an update
rejected, so revision checking cannot see it — it has to give the volume up on
its own clock. Pins that validity sits below the server TTL with room for a
check in between, that a lapsed holder reports the loss as a stall rather than
as a takeover, and that releasing a surrendered lease leaves the entry for
whoever holds it now.
The unmount side of the kill guard: sealing under a live nbdkit writes a state
torn between two writers, so a kill that did not take has to fail the unmount
and leave the entry mounted for a retry.
* fix(e2e): keep the guest private key for as long as the key pair
EnsureKeyPair computed the PEM path from the caller's directory but wrote
the file inside the ensureOnce closure, which runs only on the first call.
Callers passed a per-test artifact dir, and ArtifactDir removes that dir
when the test passes — so the first test's success deleted the only copy
of the private key while the key pair stayed registered, and every later
call returned a path nothing had ever been written to.
Guests after the first were then launched against a public key whose
private half no longer existed, so the harness could never authenticate
and gave up after the full five-minute budget. The guests were healthy
and reachable on tcp/22 throughout, which is why this read as a storage
or network fault for so long.
The directory parameter was the defect, so it is gone: the PEM now lives
in a directory the fixture owns and removes with the key pair, and no
caller can supply one that is shorter-lived.
Also reorder the killed-nbdkit teardown. The reaper registered its own
cleanup after the detach registered one, so LIFO killed the block server
before the platform detached the disk and DetachVolume then failed with
"Node ... is in use". The detach cleanup now reaps once it is done.
* fix: stop leaking an nbdkit export when recovery cannot adopt it
Restart recovery adopts a guest's boot volume, then routinely fails the
-efi volume with "claim lease: context deadline exceeded" -- JetStream is
still catching up at daemon start and the KV claim runs on the client's
5s default. Recovery gave up on the first error and never retried, so the
export stayed live with no registry entry. At terminate the unmount found
nothing to seal, returned NotFound and killed nothing, and the process
outlived its guest holding the volume and socket open. EFI is always the
second volume adopted, which is why it was always the one that leaked.
Two changes, because either alone leaves a hole:
- rebuildMountedVolume retries the build (4 attempts, 3s apart) so a
JetStream that is merely slow no longer orphans an export for the life
of the process.
- unmountVolume reaps a corroborated export for the volume it was asked
to unmount when the registry has no entry for it, so an export orphaned
by any means is still cleaned up. The reap goes through
reapOrphanedNbdkit, which declines anything a live process references:
at unmount the guest's blockdev may not be torn down yet, and killing
the server underneath it strands QEMU.
* test: hold the backend-serving probe and assert predastore readiness
The backend-outage test declared the backend restored on one successful
round trip through each gate. A gate can answer the request that arrives
just after a thaw and then lose its blob peers under the load that
follows, which is a backend that never recovered reported as one that did
-- and that is what happened: the probe reported all three gates serving
minutes before the guest was still taking BLOCK_IO_ERROR on every resume.
The round trip has to hold across three rounds now, and the count restarts
rather than decays when one fails, so a gate that serves and then stops is
the failure the hold is looking for rather than noise it averages out.
It also checks /readyz on each host and reports any failed check by name.
The round trip only proves this workstation can write through a gate; it
says nothing about which blob peers that gate can reach, and a gate one
node short of the write floor serves a small object and fails a guest. A
host that serves no admin listener is logged and skipped, since the port
is optional configuration.
* fix: scope predastore's admin port to the cluster plane in the host policy
predastore's admin listener binds the host's cluster bind_addr and serves
/healthz and /readyz unauthenticated. Readiness now names which blob peers
a gate cannot reach, which is exactly the kind of topology detail 4432 is
peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the
public one.
* fix: keep resuming a paused guest at heartbeat rate after a long outage
Past ten attempts the cont dropped to one per ten heartbeats, so a guest
whose backend had come back could sit paused for another five minutes
before anything retried it. Measured on dev-prod: the backend was proved
serving through all three gates while the guest was still waiting on
attempt 60, five minutes after attempt 50.
The slowdown was there to avoid paying a failed I/O every thirty seconds
on a backend still down. That is nearly free, and five minutes of downtime
after recovery is not, so only the log line is throttled now.
* test: fail the run when a key pair name resolves to more than one pair
A create that lands twice leaves two key pairs with one name and different
material. RunInstances resolves the name to one of them while the harness
holds the private key of the other, and the only symptom is every guest in
the run refusing the key at the SSH gate — five minutes of waiting per
guest, four guests, and nothing anywhere naming the cause. Observed on
dev-prod: two pairs created a millisecond apart, both named
e2e-key-30de10ba.
One DescribeKeyPairs after the create says so outright instead.
* test: tolerate an already-released volume in the detach teardown
Teardown runs DetachVolumeWait after a terminate that releases the volume
itself, so the detach can arrive at a volume already on its way to
available. That is the outcome it wanted, and treating the IncorrectState
refusal as fatal failed a backend-outage run whose assertions had all
passed — the guest came back, every acknowledged write verified, and the
test still went red on the way out.
VolumeInUse is still fatal: that one means the volume is attached
somewhere it should not be.
* test: carry the merged suites onto the reworked key-pair and account-scoped seams
The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir
this branch removed, and DeleteVolume's metadata lookup is account-scoped now,
so the provider-failure test has to seed and delete under the same account.
* refactor: make the fence outcome a type in one const block
Four separate exported string constants with a paragraph of comment each,
passed to RecordVolumeFence as a bare string. Group them, give them a
FenceOutcome type, and take that type at both call sites so an arbitrary
value can no longer reach the metric attribute.
* refactor: move viperblockd's wire contract into a leaf vbwire package
The daemon imported viperblockd for exactly two symbols -- the fenced
volume subject and its event -- which put the storage service back on the
daemon's dependency graph to name something it only listens for.
vbwire holds the contract instead: the subject and event, the dirty
bucket name and record shape, and the read-only listing an operator tool
needs. It imports nothing from spinifex, so both the daemon and the admin
CLI drop their edge to viperblockd entirely. cmd/service.go keeps its
one, which is genuine: it constructs the service.
* docs: drop a specific host name from the start-refusal comment
Named a real dev-prod host in a comment and in test fixtures. Say what
the name is instead.
* ci: give storagefault a dedicated nightly cell
The suite is eligible for both topologies and runs on the PR gate, but no
nightly cell ran it: the permutation cells run a deliberately narrow
subset and none of the dedicated suite cells named it.
Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation
cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that
environment would report the outage as its own failure.
* ci: run storagefault nightly on three nodes, not one
The previous commit put its dedicated cell on a single-node runner. That was
the wrong topology and the wrong justification: single-node is RS(1,0), where
the shard floor cannot be broken and a cross-node takeover after a failed seal
does not exist, so the two assertions worth having a dedicated cell for were
the two it could not make. Cell 29 is now real-multi on ci-multi.
It also corrects the claim that the PR gate already covers the multi-node case.
The suite is eligible for both topologies, but e2e.yml's default suite list
never names it, so it has never run there. A full pass is ~63 minutes on
dev-prod bare metal, almost all of it deliberate waiting -- one test holds
predastore SIGSTOPped for seven minutes -- which is too long to add to a gate
that currently finishes well inside its budget. Nightly is where it belongs.
Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since
90 minutes of suite cannot run in a 50-minute job; the permutation steps gated
on matrix.suite == '' so the dedicated cells do not also run them; and a Run
Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts
the suite against E2E_SUITES_MULTI before running it.
* fix: stop a startup KV race leaving vpcd serving keyless guests
vpcd opens five KV buckets while NATS is still coming up, each attempt
carrying a ~5s JetStream API deadline, and applied three different policies to
the same failure: spinifex-vpc-enis was fatal, while instance-state,
security-groups, subnets and vpcs each degraded silently for the lifetime of
the process. A five-second blip during a cold start therefore decided how a
node served metadata until someone restarted it.
On dev-prod all three nodes hit this after a deploy. Two were saved by
accident: their ENI bucket also timed out, so they exited 1 and systemd
restarted them cleanly. The third's ENI bucket opened in time, so it never
crashed and ran for hours with no instance record loader. With the local state
file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's
crawl died on ami-id and fell back to DataSourceNone, and every guest launched
on that node booted with no authorized_keys.
kvstore.OpenWithRetry retries for a bounded window before any policy applies,
logs the cause per attempt and the wait on a late success, wraps the last error
with the bucket name, and returns at once on a cancelled context. All five
opens use it. Failing to open the instance record space is now fatal, matching
the ENI bucket: the dependency whose loss stops guests logging in should not be
treated more leniently than the one whose loss stops ENI resolution.
Also fixes the reason the fallback was load bearing at all. The local state
file was written 0600 and vpcd reads it as a different user in the shared
spinifex group, so the local-first path had never once worked and its
mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so
an inherited umask cannot undo it.
Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an
unresolvable instance as "no key pair" with nothing logged.
* fix: scope the ime-* firewall accept to IMDS and VPC DNS only
The per-tap OVS demux delivers any guest packet addressed to
169.254.169.254 or .253 onto an ime-* host port regardless of L4 port,
and iifname "ime-*" accept then admitted all of it. Every wildcard-bound
host service — sshd, the AWS gateway, northstar, the console — answered
guest packets on those addresses; only asymmetric reply routing kept the
handshake from completing, and a guest could still drive northstar to
recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS
(.253:53), letting everything else fall through to policy drop.
* docs: document the instance-to-host plane and how to protect it without the ISO
The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by
host services and is invisible to a NIC-level or cloud firewall, since the
traffic is intra-host on the OVS bridge. Explain that the ISO ships the
scoping rule armed, that the binary installer ships the firewall off, and
that a non-ISO install must enable it with setup.sh --firewall=on or
firewall_enabled=true. Include verification from an instance and the host.
* fix: withhold the IMDS meta-data listing when the instance does not resolve
Every omission from the meta-data root is a positive claim about the instance —
public-keys/ absent means "launched without a key pair", and cloud-init believes
it, writes no authorized_keys and boots the guest keyless. So resolving the
instance was best-effort for a listing whose whole meaning depends on it: a
backend error or a miss shortened the listing instead of failing it, and nothing
anywhere recorded that the answer was a guess.
Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot
retry a 200 that is merely wrong, so this converts a permanent silent failure
into a transient loud one. servePublicKeys already had this policy for the leaf;
the root is what contradicted it.
A profile lookup that fails still serves the listing minus iam/, and now logs.
The key pair is the field a guest cannot recover from losing, while a missing
role surfaces through the credentials leaf, which reports the miss itself.
Also stops the storagefault cross-node test failing on its own access path. It
tolerates the pattern changing across a takeover, but readPattern needs sudo,
and a volume opened from the backend checkpoint may predate cloud-init writing
its sudoers drop-in — indistinguishable from an unreadable device at the dd call
site. It now checks sudo first, and reads cloud-init's recorded datasource to
tell a rolled-back root filesystem from IMDS never having answered, failing
loudly on the latter rather than accepting it as storage loss.
---------
Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>
This commit is contained in:
Ben Duncan
@@ -28,10 +28,19 @@
|
||||
# that runs identically on either, and because a quota left switched on would
|
||||
# cap every suite sharing the environment — so it must be proved to restore
|
||||
# itself wherever it runs, not only on one node.
|
||||
E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota"
|
||||
# storagefault runs last in both lists and belongs to both topologies, for
|
||||
# different reasons. On one node it is RS(1,0), so freezing the only predastore
|
||||
# is a total outage and the guest-survival assertion is unambiguous. On three it
|
||||
# is RS(2,1) with degraded writes on, so two must be frozen to break the shard
|
||||
# floor — and only there is a cross-node restart possible at all, which is the
|
||||
# case that takes a volume over from a node holding writes the backend never
|
||||
# received. It is last because it SIGSTOPs a cluster-wide
|
||||
# service: anything sharing the environment would see the same outage and report
|
||||
# a failure that says nothing about itself.
|
||||
E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota storagefault"
|
||||
|
||||
# Suites runnable against a multi-node environment.
|
||||
E2E_SUITES_MULTI="multinode lb cert quota"
|
||||
E2E_SUITES_MULTI="multinode lb cert quota storagefault"
|
||||
|
||||
# grep -xE alternation form, for narrowing a requested set to the eligible one.
|
||||
E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
|
||||
@@ -44,6 +53,16 @@ E2E_SUITES_MULTI_RE="$(printf '%s' "${E2E_SUITES_MULTI}" | tr ' ' '|')"
|
||||
# nightly in their own dedicated cell, because each needs a guest AMI built
|
||||
# first and a service failure should not mask whether a host OS boots. These
|
||||
# are subsets, and e2e_suites_assert_subset below enforces that.
|
||||
#
|
||||
# storagefault has a dedicated cell for two reasons. It SIGSTOPs a cluster-wide
|
||||
# service, so in a permutation cell every suite sharing that environment would
|
||||
# report the outage as its own failure. And a full pass is ~63 minutes, nearly
|
||||
# all of it deliberate waiting, which is too long to add to the PR gate — where
|
||||
# it has never run, since the gate's default suite list does not name it.
|
||||
#
|
||||
# Its cell is three-node: the RS(2,1) shard floor and the cross-node takeover
|
||||
# after a failed seal do not exist on one node, and they are the assertions
|
||||
# worth a nightly.
|
||||
E2E_SUITES_NIGHTLY_SINGLE="single cert iam"
|
||||
E2E_SUITES_NIGHTLY_MULTI="multinode cert lb"
|
||||
|
||||
@@ -61,6 +80,10 @@ e2e_suite_timeout() {
|
||||
# creates first. One pass is ~20 minutes on the bare-metal cell and a
|
||||
# baseline capture at SPINIFEX_DISKPERF_REPS=3 is three times that.
|
||||
diskperf) echo "180m" ;;
|
||||
# storagefault is mostly deliberate waiting: one test alone holds predastore
|
||||
# SIGSTOPped for seven minutes to prove a guest survives it. A full pass on
|
||||
# dev-prod is ~63 minutes, so the 30m default truncated it mid-suite.
|
||||
storagefault) echo "90m" ;;
|
||||
*) echo "30m" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
@@ -6,7 +6,7 @@ on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
cells:
|
||||
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance"
|
||||
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection"
|
||||
required: false
|
||||
|
||||
permissions:
|
||||
@@ -47,9 +47,10 @@ jobs:
|
||||
- { cell: 25, install: binary, topo: single, net: static, host: debian-13, arch: x86_64, runner: ci-single, extra: "eks", suite: "eks" }
|
||||
- { cell: 26, install: binary, topo: single, net: static, host: debian-13, arch: x86_64, runner: ci-single, extra: "ecs", suite: "ecs" }
|
||||
- { cell: 27, install: binary, topo: single, net: static, host: debian-13, arch: x86_64, runner: ci-single, extra: "rds", suite: "rds" }
|
||||
- { cell: 29, install: binary, topo: real-multi, net: static, host: debian-13, arch: x86_64, runner: ci-multi, extra: "", suite: "storagefault", timeout_minutes: 100 }
|
||||
|
||||
runs-on: [ self-hosted, "${{ matrix.runner }}" ]
|
||||
timeout-minutes: 50
|
||||
timeout-minutes: ${{ matrix.timeout_minutes || 50 }}
|
||||
env:
|
||||
TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
|
||||
ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
|
||||
@@ -174,7 +175,7 @@ jobs:
|
||||
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
|
||||
|
||||
- name: Run Single Node E2E
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == ''
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == '' && matrix.suite == ''
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -224,7 +225,7 @@ jobs:
|
||||
if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
|
||||
|
||||
- name: Run Multi Node E2E
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi' && matrix.suite == ''
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -259,11 +260,11 @@ jobs:
|
||||
if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
|
||||
done; exit \$FAIL"
|
||||
|
||||
# Cells 25-27 only. Resolved once here rather than in each step below,
|
||||
# which would repeat the same three lines five times.
|
||||
# The dedicated suite cells only. Resolved once here rather than in each
|
||||
# step below, which would repeat the same three lines five times.
|
||||
- name: Resolve VM SSH details
|
||||
id: vm
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.suite != ''
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.suite != '' && matrix.topo == 'single'
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -407,7 +408,7 @@ jobs:
|
||||
done
|
||||
|
||||
- name: Run Service Suite E2E
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.suite != ''
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.suite != '' && matrix.topo == 'single'
|
||||
env:
|
||||
SUITE: ${{ matrix.suite }}
|
||||
run: |
|
||||
@@ -432,6 +433,46 @@ jobs:
|
||||
if [ \$rc -ne 0 ]; then echo \"${SUITE}: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; fi; \
|
||||
exit \$rc"
|
||||
|
||||
# The multi-node twin of the step above. Separate rather than merged
|
||||
# because the suite needs every node's address to reach the whole cluster,
|
||||
# which the single-node path has no way to supply.
|
||||
- name: Run Multi Suite E2E
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.suite != '' && matrix.topo == 'real-multi'
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
env:
|
||||
SUITE: ${{ matrix.suite }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
. "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
|
||||
e2e_suites_assert_subset "e2e-nightly cell-${{ matrix.cell }}" "${SUITE}" "${E2E_SUITES_MULTI}"
|
||||
TMO="$(e2e_suite_timeout "${SUITE}")"
|
||||
|
||||
INVENTORY=$(tofu output -json inventory)
|
||||
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
||||
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
||||
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
||||
|
||||
NODES_CSV=""
|
||||
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
|
||||
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
||||
ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
|
||||
NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
|
||||
done
|
||||
|
||||
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
|
||||
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
||||
echo \"::group::suite ${SUITE}\"; \
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
|
||||
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
|
||||
SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
|
||||
SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
|
||||
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
||||
./${SUITE}.test -test.v -test.timeout=${TMO} \
|
||||
2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log'; \
|
||||
rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
|
||||
if [ \$rc -ne 0 ]; then echo \"${SUITE}: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; fi; \
|
||||
exit \$rc"
|
||||
|
||||
- name: Import RDS PostgreSQL AMI
|
||||
if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
|
||||
@@ -40,6 +40,7 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/hostdns"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/host"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
@@ -218,6 +219,18 @@ decision made per volume.`,
|
||||
Run: runVolumesOrphansCmd,
|
||||
}
|
||||
|
||||
var volumesUnsealedCmd = &cobra.Command{
|
||||
Use: "unsealed",
|
||||
Short: "Report volumes whose writes have not been confirmed to the backend",
|
||||
Long: `List volumes a node has open, or had open, without a confirmed seal to the
|
||||
backend. The node named holds writes that predastore may not have.
|
||||
|
||||
Starting such a volume elsewhere is allowed and is preferred over leaving the
|
||||
instance unable to run, but it opens from the last checkpoint that did reach
|
||||
the backend. This is the list of volumes where that would cost something.`,
|
||||
Run: runVolumesUnsealedCmd,
|
||||
}
|
||||
|
||||
var accountCmd = &cobra.Command{
|
||||
Use: "account",
|
||||
Short: "Manage Spinifex accounts",
|
||||
@@ -318,6 +331,7 @@ func init() {
|
||||
|
||||
adminCmd.AddCommand(volumesCmd)
|
||||
volumesCmd.AddCommand(volumesOrphansCmd)
|
||||
volumesCmd.AddCommand(volumesUnsealedCmd)
|
||||
|
||||
adminCmd.AddCommand(accountCmd)
|
||||
accountCmd.AddCommand(accountCreateCmd)
|
||||
@@ -3697,3 +3711,37 @@ func printChecksumError(w io.Writer, imageFile, imageName string, image utils.Im
|
||||
fmt.Fprintln(w, "The cached file was left in place. To re-download and retry:")
|
||||
fmt.Fprintf(w, " spx admin images import --name %s --force\n", imageName)
|
||||
}
|
||||
|
||||
func runVolumesUnsealedCmd(_ *cobra.Command, _ []string) {
|
||||
_, nc, err := loadConfigAndConnect()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Could not connect to the cluster: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer nc.Close()
|
||||
|
||||
unsealed, err := vbwire.ListUnsealedVolumes(context.Background(), nc)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "Could not read the unsealed volumes:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if len(unsealed) == 0 {
|
||||
fmt.Println("No unsealed volumes: every volume's writes are confirmed to the backend.")
|
||||
return
|
||||
}
|
||||
|
||||
sort.Slice(unsealed, func(i, j int) bool { return unsealed[i].VolumeID < unsealed[j].VolumeID })
|
||||
fmt.Printf("%d volume(s) with writes the backend may not have:\n\n", len(unsealed))
|
||||
for _, volume := range unsealed {
|
||||
fmt.Printf(" %s\n", volume.VolumeID)
|
||||
fmt.Printf(" held by: %s\n", volume.Owner)
|
||||
if !volume.Since.IsZero() {
|
||||
fmt.Printf(" since: %s\n", volume.Since.Format(time.RFC3339))
|
||||
}
|
||||
fmt.Printf(" reason: %s\n", volume.Reason)
|
||||
}
|
||||
fmt.Println()
|
||||
fmt.Println("A volume open right now is listed here and that is normal — the entry")
|
||||
fmt.Println("clears on its next successful seal. One naming a node that is gone is")
|
||||
fmt.Println("the case worth acting on: starting it elsewhere loses writes made since.")
|
||||
}
|
||||
|
||||
@@ -795,6 +795,103 @@ aws ec2 run-instances --image-id $AMI --instance-type t3.small \
|
||||
|
||||
Rule changes take effect immediately — no instance restart needed.
|
||||
|
||||
## The Instance-to-Host Plane (Metadata and VPC DNS)
|
||||
|
||||
Security groups govern traffic **between instances** and **to the outside**. There
|
||||
is a third path they do not touch, and it is the one most easily left exposed:
|
||||
traffic from an instance to the **host it runs on**.
|
||||
|
||||
Every instance reaches two link-local addresses served by the hypervisor:
|
||||
|
||||
| Address | Service | Port |
|
||||
| --- | --- | --- |
|
||||
| `169.254.169.254` | Instance metadata (IMDS) — cloud-init, instance-role credentials | TCP 80 |
|
||||
| `169.254.169.253` | VPC DNS resolver | UDP/TCP 53 |
|
||||
|
||||
These are not guest addresses and not OVN routed. Each instance's tap has a
|
||||
capture rule on the hypervisor that intercepts any packet addressed to `.254` or
|
||||
`.253` and delivers it to a per-ENI internal port in the host network namespace
|
||||
(named `ime-*`). That interception is **by destination address only — it does not
|
||||
match the port.** A packet to `169.254.169.254:22` is delivered to the host just
|
||||
as readily as one to `169.254.169.254:80`.
|
||||
|
||||
That matters because host services — SSH, the AWS gateway on 9999, the console on
|
||||
3000, the DNS server — bind the wildcard address, so they also answer on these
|
||||
link-local addresses. Without a control on the `ime-*` path, an instance can
|
||||
reach every one of them, with none of the source scoping that protects the same
|
||||
services on the network. A resolver query sent straight to the host's DNS port
|
||||
this way also bypasses the per-instance DNS rate limit.
|
||||
|
||||
### This plane is invisible to a network firewall
|
||||
|
||||
The critical point for anyone securing a deployment: **this traffic never crosses
|
||||
the physical NIC.** It is intra-host, on the OVS bridge between the instance's tap
|
||||
and the `ime-*` port. A cloud security group, an upstream firewall, `ufw`, or any
|
||||
rule written against the WAN interface is not in this path and cannot filter it.
|
||||
The only place to enforce it is on the `ime-*` interface in the host itself.
|
||||
|
||||
### How it is protected
|
||||
|
||||
Spinifex's host firewall carries a rule that scopes the `ime-*` path to exactly
|
||||
the two legitimate endpoints and ports — IMDS on `.254:80`, DNS on `.253:53` —
|
||||
and drops everything else. Whether you have that rule depends entirely on whether
|
||||
the host firewall is on:
|
||||
|
||||
| Install path | Host firewall | This plane |
|
||||
| --- | --- | --- |
|
||||
| **From the ISO** | on | protected |
|
||||
| **Binary installer** (`curl \| bash`) or `setup.sh` | **off** | **exposed** |
|
||||
| **Binary installer with `--firewall=on`** | on | protected |
|
||||
|
||||
The ISO ships the firewall armed, so ISO deployments are protected out of the box.
|
||||
**The binary installer ships it off** — deliberately, because it runs on servers
|
||||
that may already have services the installer knows nothing about, and a
|
||||
default-deny policy could cut them off. The consequence is that a binary install
|
||||
left at its default has this plane wide open, and so does every host service on
|
||||
the WAN besides.
|
||||
|
||||
If you install any way other than the ISO, turn the host firewall on. At install
|
||||
time:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
|
||||
# or, from a source checkout:
|
||||
sudo /usr/local/share/spinifex/setup.sh --firewall=on
|
||||
```
|
||||
|
||||
or afterwards, in `/etc/spinifex/spinifex.toml`, followed by a daemon restart:
|
||||
|
||||
```toml
|
||||
[network]
|
||||
firewall_enabled = true
|
||||
```
|
||||
|
||||
Before enabling it, check what else the machine is serving — anything listening
|
||||
outside the public port group stops accepting new connections. See
|
||||
[Firewall and Cluster Membership](/docs/install-multi-node#firewall-and-cluster-membership)
|
||||
for the full port policy and the cluster-formation steps.
|
||||
|
||||
### Verifying it
|
||||
|
||||
On the hypervisor, the `ime-*` accept should be scoped, not blanket:
|
||||
|
||||
```bash
|
||||
sudo nft list chain inet spinifex_filter input | grep ime
|
||||
# Expect three rules naming 169.254.169.254 tcp dport 80 and 169.254.169.253
|
||||
# udp/tcp dport 53 — not a bare `iifname "ime-*" accept`.
|
||||
```
|
||||
|
||||
From an instance, the two service ports work and nothing else does:
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://169.254.169.254/latest/meta-data/ # 401 (IMDSv2)
|
||||
dig +short @169.254.169.253 example.com A # resolves
|
||||
nc -vz 169.254.169.254 22 # must fail
|
||||
```
|
||||
|
||||
A missing `spinifex_filter` table means the host firewall is off and this plane
|
||||
is unprotected regardless of the rule above.
|
||||
|
||||
## Elastic IPs
|
||||
|
||||
Elastic IPs are static public IPs that persist across instance stop/start cycles. Unlike auto-assigned public IPs (which change on stop/start), an Elastic IP stays with your instance.
|
||||
|
||||
@@ -102,6 +102,7 @@ behavior in other words.
|
||||
| 5300 | northstar | DNS (UDP + TCP) | Cluster | Forward target for every node's per-instance DNS shim, dialled cross-node. Binds the wildcard by design. | None — restrict by firewall |
|
||||
| 6660 | predastore (blob node) | QUIC / UDP | Cluster | Erasure-coded object shard transport between hosts. Multi-node clusters only — see *Predastore ports* below. | TLS 1.3, server certificate verified against the cluster CA |
|
||||
| 7660 | predastore (meta node) | QUIC / UDP | Cluster | Raft consensus over global state — buckets and the object index — between hosts. Multi-node clusters only. | TLS 1.3, server certificate verified against the cluster CA |
|
||||
| 8660 | predastore (admin) | HTTP | Cluster | `/healthz` and `/readyz`. Readiness names the blob peers a gate cannot reach, so it must not face the WAN. Binds the host's cluster `bind_addr`, never the wildcard. | None — unauthenticated by design; restrict by firewall |
|
||||
| 6641 | OVN Northbound DB (client) | OVSDB/TCP | Cluster | Logical network topology consumed by vpcd. Binds `127.0.0.1` plus the node's lan-plane address (`--lan-addr`), never the wildcard address. On a node with no separate lan plane that address is the public one, and a host firewall is the only remaining control. | Cluster network only; TLS planned |
|
||||
| 6642 | OVN Southbound DB (client) | OVSDB/TCP | Cluster | Chassis / port / MAC binding state. Binds `127.0.0.1` plus the node's lan-plane address (`--lan-addr`), never the wildcard address. On a node with no separate lan plane that address is the public one, and a host firewall is the only remaining control. | Cluster network only; TLS planned |
|
||||
| 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
|
||||
@@ -207,7 +208,7 @@ table inet spinifex_filter {
|
||||
|
||||
# Cluster, from peer nodes only. Replace with your nodes' lan-plane
|
||||
# addresses — a CIDR does not generalise to nodes on different subnets.
|
||||
ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644 } accept
|
||||
ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
|
||||
ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } udp dport { 5300, 6660, 7660 } accept
|
||||
|
||||
# Encap, from peer chassis only. Replace with your nodes' vpc-plane
|
||||
|
||||
+14
-3
@@ -371,7 +371,16 @@ table inet spinifex_filter {
|
||||
# here. Without this, cloud-init, instance-role credentials and all
|
||||
# guest DNS break. The ports exist only while instances run, so this
|
||||
# matches the prefix rather than an enumerated list.
|
||||
iifname "ime-*" accept
|
||||
#
|
||||
# Scoped to the two link-local endpoints and their ports, not a blanket
|
||||
# accept: the OVS demux delivers any guest packet to 169.254.169.254/.253
|
||||
# onto an ime-* port regardless of L4 port, so `iifname "ime-*" accept`
|
||||
# would expose every wildcard-bound host service (sshd, the AWS gateway,
|
||||
# northstar, the UI) to any tenant. IMDS is HTTP on .254:80; VPC DNS is
|
||||
# 53 on .253. Everything else on these ports falls through to policy drop.
|
||||
iifname "ime-*" ip daddr 169.254.169.254 tcp dport 80 accept
|
||||
iifname "ime-*" ip daddr 169.254.169.253 udp dport 53 accept
|
||||
iifname "ime-*" ip daddr 169.254.169.253 tcp dport 53 accept
|
||||
|
||||
# PMTUD is not optional under a Geneve overlay.
|
||||
icmp type { echo-request, destination-unreachable, time-exceeded, parameter-problem } accept
|
||||
@@ -412,8 +421,10 @@ table inet spinifex_filter {
|
||||
# than an interface or a CIDR. 4432 stays here rather than joining the
|
||||
# public rule above: outside formation it is the daemon cluster manager,
|
||||
# whose /health and /local/* routes report node topology, service
|
||||
# inventory and running instances to anyone who asks.
|
||||
ip saddr $spinifex_peers tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644 } accept
|
||||
# inventory and running instances to anyone who asks. 8660 is
|
||||
# predastore's unauthenticated /healthz and /readyz, which name the
|
||||
# peers a gate cannot reach, so it belongs here for the same reason.
|
||||
ip saddr $spinifex_peers tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
|
||||
ip saddr $spinifex_peers udp dport { 5300, 6660, 7660 } accept
|
||||
|
||||
# Encap plane, peer-scoped: Geneve, IKE, NAT-T and ESP. Geneve is a
|
||||
|
||||
@@ -69,6 +69,7 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
"github.com/mulgadc/spinifex/spinifex/preflight"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
@@ -1265,6 +1266,9 @@ func (d *Daemon) subscribeAll() error {
|
||||
// here, so the reconcile request flows daemon-ward.
|
||||
natsSub{dhcp.TopicLeaseChanged, d.handleDHCPLeaseChanged, "spinifex-workers"},
|
||||
natsSub{dhcp.TopicOwnerCheck, d.handleDHCPOwnerCheck, "spinifex-workers"},
|
||||
// Node-addressed with no queue group: the fenced guest is on this host,
|
||||
// so a worker on another node would find nothing to stop.
|
||||
natsSub{vbwire.VolumeFencedSubject(d.node), d.handleVolumeFenced, ""},
|
||||
)
|
||||
|
||||
return d.registerNatsSubs(subs)
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
package daemon
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
|
||||
// handleVolumeFenced stops the local guest whose volume viperblockd has just
|
||||
// torn the export out from under.
|
||||
//
|
||||
// By the time this arrives the disk is already gone, so the guest is doing no
|
||||
// useful work and its writes cannot reach the backend. What is left is a QEMU
|
||||
// process holding memory and taps, and an instance record claiming this node
|
||||
// runs something that runs somewhere else now.
|
||||
func (d *Daemon) handleVolumeFenced(msg *nats.Msg) string {
|
||||
ctx, span := utils.StartConsumerSpan(msg)
|
||||
defer span.End()
|
||||
|
||||
var event vbwire.VolumeFencedEvent
|
||||
if err := json.Unmarshal(msg.Data, &event); err != nil {
|
||||
slog.ErrorContext(ctx, "volume fenced: bad event", "err", err)
|
||||
return outcomeError
|
||||
}
|
||||
if event.Volume == "" {
|
||||
slog.ErrorContext(ctx, "volume fenced: event named no volume")
|
||||
return outcomeError
|
||||
}
|
||||
|
||||
instanceID := d.vmMgr.FenceVolume(ctx, event.Volume, event.Reason)
|
||||
if instanceID == "" {
|
||||
// The usual case: the volume outlived the guest that was using it, so
|
||||
// there is nothing local left to stop.
|
||||
slog.InfoContext(ctx, "volume fenced with no local guest using it",
|
||||
"volumeId", event.Volume, "winner", event.Winner)
|
||||
return outcomeSuccess
|
||||
}
|
||||
|
||||
slog.ErrorContext(ctx, "volume fenced, stopped the guest that was using it",
|
||||
"volumeId", event.Volume, "instanceId", instanceID, "winner", event.Winner)
|
||||
return outcomeSuccess
|
||||
}
|
||||
@@ -57,6 +57,11 @@ func MarshalLocalState(vms map[string]*vm.VM) ([]byte, error) {
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// LocalStateFileMode is the mode of the on-disk instance state file. The daemon
|
||||
// writes it and vpcd reads it to serve IMDS; they run as different users in the
|
||||
// shared spinifex group, so owner-only would deny every IMDS local lookup.
|
||||
const LocalStateFileMode = 0o640
|
||||
|
||||
// WriteLocalStateBytes atomically writes pre-marshalled state JSON to path.
|
||||
func WriteLocalStateBytes(path string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
@@ -65,11 +70,19 @@ func WriteLocalStateBytes(path string, data []byte) error {
|
||||
}
|
||||
|
||||
tmp := path + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, LocalStateFileMode)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open tmp %s: %w", tmp, err)
|
||||
}
|
||||
|
||||
// Explicitly, because OpenFile's mode is masked by the umask the service
|
||||
// happens to inherit, and a 0600 here silently costs IMDS its local lookup.
|
||||
if err := f.Chmod(LocalStateFileMode); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("chmod tmp %s: %w", tmp, err)
|
||||
}
|
||||
|
||||
if _, err := f.Write(data); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
|
||||
@@ -57,6 +57,19 @@ func TestWriteLocalState_MkdirParent(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// TestWriteLocalState_GroupReadable pins the mode. vpcd serves IMDS and reads
|
||||
// this file as a different user in the shared spinifex group, so an owner-only
|
||||
// mode denies every local lookup and each guest boot falls back to the record
|
||||
// space -- where a miss is served as "this instance has no key pair".
|
||||
func TestWriteLocalState_GroupReadable(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "instance-state.json")
|
||||
require.NoError(t, writeLocalState(path, map[string]*vm.VM{}))
|
||||
|
||||
info, err := os.Stat(path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, os.FileMode(0o640), info.Mode().Perm())
|
||||
}
|
||||
|
||||
func TestWriteLocalState_NoTmpLeftBehind(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "instance-state.json")
|
||||
|
||||
@@ -100,7 +100,7 @@ func startLiveInstance(ctx context.Context, natsConn *nats.Conn, instanceID, acc
|
||||
|
||||
if responseError, parseErr := utils.ValidateErrorPayload(msg.Data); parseErr != nil {
|
||||
slog.ErrorContext(ctx, "StartInstances: owner returned error", "instance_id", instanceID, "code", *responseError.Code)
|
||||
return nil, false, errors.New(*responseError.Code)
|
||||
return nil, false, startRefusalError(responseError)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "StartInstances: restarted via owner node", "instance_id", instanceID, "response", string(msg.Data))
|
||||
@@ -132,9 +132,27 @@ func startStoppedInstance(ctx context.Context, natsConn *nats.Conn, instanceID,
|
||||
|
||||
if responseError, parseErr := utils.ValidateErrorPayload(msg.Data); parseErr != nil {
|
||||
slog.ErrorContext(ctx, "StartInstances: Daemon returned error", "instance_id", instanceID, "code", *responseError.Code)
|
||||
return nil, errors.New(*responseError.Code)
|
||||
return nil, startRefusalError(responseError)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "StartInstances: Command sent successfully", "instance_id", instanceID, "response", string(msg.Data))
|
||||
return newStateChange(instanceID, 0, "pending", 80, "stopped"), nil
|
||||
}
|
||||
|
||||
// startRefusalError rebuilds the daemon's error, keeping the message it sent.
|
||||
// Reconstructing from the code alone renders the generic ErrorLookup text
|
||||
// instead, which replaces the name of the node holding the volume with
|
||||
// boilerplate about attaching volumes -- the one fact the operator needed.
|
||||
func startRefusalError(responseError ec2.ResponseError) error {
|
||||
var code, message string
|
||||
if responseError.Code != nil {
|
||||
code = *responseError.Code
|
||||
}
|
||||
if responseError.Message != nil {
|
||||
message = *responseError.Message
|
||||
}
|
||||
if message == "" || message == code {
|
||||
return errors.New(code)
|
||||
}
|
||||
return awserrors.Errorf(code, "%s", message)
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
handlers_ec2_volume "github.com/mulgadc/spinifex/spinifex/handlers/ec2/volume"
|
||||
@@ -66,6 +67,8 @@ func DetachVolume(ctx context.Context, input *ec2.DetachVolumeInput, natsConn *n
|
||||
device = *input.Device
|
||||
}
|
||||
|
||||
force := input.Force != nil && *input.Force
|
||||
|
||||
command := types.EC2InstanceCommand{
|
||||
ID: instanceID,
|
||||
Attributes: types.EC2CommandAttributes{
|
||||
@@ -74,7 +77,7 @@ func DetachVolume(ctx context.Context, input *ec2.DetachVolumeInput, natsConn *n
|
||||
DetachVolumeData: &types.DetachVolumeData{
|
||||
VolumeID: volumeID,
|
||||
Device: device,
|
||||
Force: input.Force != nil && *input.Force,
|
||||
Force: force,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -93,6 +96,9 @@ func DetachVolume(ctx context.Context, input *ec2.DetachVolumeInput, natsConn *n
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "DetachVolume: NATS request failed", "instanceId", instanceID, "volumeId", volumeID, "err", err)
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
if force {
|
||||
return forceDetach(ctx, natsConn, accountID, volumeID, instanceID)
|
||||
}
|
||||
return output, errors.New(awserrors.ErrorInvalidInstanceIDNotFound)
|
||||
}
|
||||
return output, errors.New(awserrors.ErrorServerInternal)
|
||||
@@ -114,3 +120,31 @@ func DetachVolume(ctx context.Context, input *ec2.DetachVolumeInput, natsConn *n
|
||||
slog.InfoContext(ctx, "DetachVolume completed", "instanceId", instanceID, "volumeId", volumeID)
|
||||
return output, nil
|
||||
}
|
||||
|
||||
// forceDetachSubject clears a volume's attachment in the control plane only and
|
||||
// is answered by any node, unlike the ordinary detach, which routes to the host
|
||||
// running the instance and so cannot help when that host is the problem.
|
||||
const forceDetachSubject = "ec2.ForceDetachVolume"
|
||||
|
||||
// forceDetach releases a volume whose instance has no responder.
|
||||
//
|
||||
// A volume outlives the instance it was attached to whenever teardown failed
|
||||
// part-way, and the ordinary detach then has nowhere to go: there is no guest
|
||||
// to unplug it from and no host to ask. Without this the volume is attached
|
||||
// forever, which also makes it undeletable. Reaching here needs Force, which is
|
||||
// the caller stating that no clean guest unmount is expected.
|
||||
func forceDetach(ctx context.Context, natsConn *nats.Conn, accountID, volumeID, instanceID string) (ec2.VolumeAttachment, error) {
|
||||
attachment, err := utils.NATSRequest[ec2.VolumeAttachment](ctx, natsConn, forceDetachSubject,
|
||||
&ec2.DetachVolumeInput{VolumeId: &volumeID, Force: aws.Bool(true)}, 30*time.Second, accountID)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "DetachVolume: force detach failed", "volumeId", volumeID, "instanceId", instanceID, "err", err)
|
||||
return ec2.VolumeAttachment{}, err
|
||||
}
|
||||
|
||||
slog.WarnContext(ctx, "DetachVolume: forced, the instance had no responder",
|
||||
"volumeId", volumeID, "instanceId", instanceID)
|
||||
if attachment == nil {
|
||||
return ec2.VolumeAttachment{}, nil
|
||||
}
|
||||
return *attachment, nil
|
||||
}
|
||||
|
||||
@@ -1,12 +1,17 @@
|
||||
package gateway_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestValidateDetachVolumeInput(t *testing.T) {
|
||||
@@ -92,3 +97,59 @@ func TestValidateDetachVolumeInput(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDetachVolume_ForceFallsBackWhenTheInstanceIsGone covers the deadlock the
|
||||
// control-plane-only detach exists for. A teardown that fails part-way leaves a
|
||||
// volume attached to an instance that no longer runs anywhere, and the ordinary
|
||||
// detach routes to that instance's own subject, so it can never be answered.
|
||||
// Without the fallback the volume is attached forever and therefore undeletable.
|
||||
func TestDetachVolume_ForceFallsBackWhenTheInstanceIsGone(t *testing.T) {
|
||||
_, nc := testutil.StartTestNATS(t)
|
||||
|
||||
var forced []byte
|
||||
sub, err := nc.Subscribe(forceDetachSubject, func(msg *nats.Msg) {
|
||||
forced = msg.Data
|
||||
data, err := json.Marshal(&ec2.VolumeAttachment{
|
||||
VolumeId: aws.String("vol-orphan"), State: aws.String("detached"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, msg.Respond(data))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, nc.Flush())
|
||||
|
||||
// No subscriber on ec2.cmd.i-gone, which is exactly a terminated instance.
|
||||
out, err := DetachVolume(context.Background(), &ec2.DetachVolumeInput{
|
||||
VolumeId: aws.String("vol-orphan"),
|
||||
InstanceId: aws.String("i-gone"),
|
||||
Force: aws.Bool(true),
|
||||
}, nc, "000000000001")
|
||||
require.NoError(t, err, "force must release a volume whose instance has no responder")
|
||||
|
||||
assert.Equal(t, "detached", aws.StringValue(out.State))
|
||||
assert.Contains(t, string(forced), "vol-orphan", "the force detach must name the volume")
|
||||
}
|
||||
|
||||
// TestDetachVolume_WithoutForceStillReportsTheMissingInstance guards the other
|
||||
// direction: the fallback clears the attachment without touching the guest, so
|
||||
// a caller who did not ask for force must keep getting the honest answer rather
|
||||
// than a silent success.
|
||||
func TestDetachVolume_WithoutForceStillReportsTheMissingInstance(t *testing.T) {
|
||||
_, nc := testutil.StartTestNATS(t)
|
||||
|
||||
sub, err := nc.Subscribe(forceDetachSubject, func(msg *nats.Msg) {
|
||||
require.Fail(t, "force detach must not run without Force")
|
||||
_ = msg.Respond(nil)
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, nc.Flush())
|
||||
|
||||
_, err = DetachVolume(context.Background(), &ec2.DetachVolumeInput{
|
||||
VolumeId: aws.String("vol-orphan"),
|
||||
InstanceId: aws.String("i-gone"),
|
||||
}, nc, "000000000001")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, err.Error())
|
||||
}
|
||||
|
||||
@@ -2323,6 +2323,9 @@ func (s *InstanceServiceImpl) StartStoppedInstance(ctx context.Context, input *S
|
||||
s.resourceMgr.Deallocate(instanceType)
|
||||
s.vmMgr.Delete(instance.ID)
|
||||
s.restoreClaimedStoppedInstance(ctx, instance)
|
||||
if refusal := volumeHeldElsewhereError(err); refusal != nil {
|
||||
return nil, refusal
|
||||
}
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
@@ -2333,6 +2336,32 @@ func (s *InstanceServiceImpl) StartStoppedInstance(ctx context.Context, input *S
|
||||
return &StartStoppedInstanceOutput{Status: "running", InstanceID: instance.ID}, nil
|
||||
}
|
||||
|
||||
// volumeExclusionMarkers are the refusals that mean another node is writing this
|
||||
// volume right now. Only the lease refuses: a volume whose writes are merely
|
||||
// unsealed is opened here with a warning rather than blocked, so the instance
|
||||
// still runs. The text arrives having crossed NATS from viperblockd.
|
||||
var volumeExclusionMarkers = []string{
|
||||
"volume is leased by another owner",
|
||||
}
|
||||
|
||||
// volumeHeldElsewhereError converts a mount refused because another node holds
|
||||
// the volume into a client-facing error carrying that reason. Returns nil for
|
||||
// anything else. Without it the caller gets ServerInternal and is told to
|
||||
// retry, which cannot succeed while the holder is down, and the text naming the
|
||||
// holder never leaves the node's log.
|
||||
func volumeHeldElsewhereError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
msg := err.Error()
|
||||
for _, marker := range volumeExclusionMarkers {
|
||||
if strings.Contains(msg, marker) {
|
||||
return awserrors.Errorf(awserrors.ErrorIncorrectState, "%s", msg)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// restoreClaimedStoppedInstance writes instance back to the shared
|
||||
// stopped-instance store after a downstream failure that follows a
|
||||
// successful ClaimStoppedInstance (which already removed it), so the claim
|
||||
@@ -2442,25 +2471,21 @@ func (s *InstanceServiceImpl) deleteInstanceVolumes(ctx context.Context, instanc
|
||||
continue
|
||||
}
|
||||
|
||||
// User-visible volumes: DeleteOnTermination=false survives terminate,
|
||||
// but terminate still implies detach (AWS semantics). Only the Boot
|
||||
// volume can still be attached here — Stop's Unmount (daemon/vm_adapters.go)
|
||||
// clears every non-Boot volume's attachment already, so a non-Boot
|
||||
// DoT=false volume is genuinely a no-op skip. A DoT=false Boot volume
|
||||
// is never touched by Unmount, so without this it would strand
|
||||
// attached to the now-terminated instance forever.
|
||||
// User-visible volumes: DeleteOnTermination=false survives terminate, but
|
||||
// terminate still implies detach (AWS semantics) for every volume, Boot or
|
||||
// not. Unmount clears a non-Boot attachment only when the seal succeeded,
|
||||
// so a volume whose seal failed is still attached to an instance that is
|
||||
// about to stop existing.
|
||||
if !ebsRequest.DeleteOnTermination {
|
||||
if !ebsRequest.Boot {
|
||||
slog.InfoContext(ctx, "TerminateStoppedInstance: volume has DeleteOnTermination=false, skipping", "name", ebsRequest.Name)
|
||||
continue
|
||||
}
|
||||
slog.InfoContext(ctx, "TerminateStoppedInstance: boot volume has DeleteOnTermination=false, detaching without deleting", "name", ebsRequest.Name)
|
||||
slog.InfoContext(ctx, "TerminateStoppedInstance: volume has DeleteOnTermination=false, detaching without deleting", "name", ebsRequest.Name)
|
||||
if s.volumeDeleter == nil {
|
||||
slog.WarnContext(ctx, "TerminateStoppedInstance: volume deleter not configured, skipping detach", "name", ebsRequest.Name)
|
||||
lastErr = errors.New("volume deleter not configured")
|
||||
continue
|
||||
}
|
||||
if err := s.volumeDeleter.DetachVolumeOnTerminate(ctx, ebsRequest.Name, instance.AccountID); err != nil {
|
||||
// A volume already gone is the detach's own goal, not a failure.
|
||||
if err := s.volumeDeleter.DetachVolumeOnTerminate(ctx, ebsRequest.Name, instance.AccountID); err != nil &&
|
||||
!awserrors.IsNotFound(err) && !objectstore.IsNoSuchKeyError(err) {
|
||||
slog.ErrorContext(ctx, "TerminateStoppedInstance: failed to detach volume", "name", ebsRequest.Name, "err", err)
|
||||
lastErr = err
|
||||
}
|
||||
@@ -2877,13 +2902,14 @@ func (s *InstanceServiceImpl) buildInstanceStatus(v *vm.VM, systemImpaired bool)
|
||||
}
|
||||
|
||||
// SystemStatus reflects host/node health, independent of the VM process: a
|
||||
// running VM's host is reachable unless under memory pressure; non-running
|
||||
// instances are not-applicable.
|
||||
// running VM's host is reachable unless under memory pressure. A guest held
|
||||
// on a backend I/O error is an infrastructure fault rather than a guest one,
|
||||
// which is the distinction this check exists to draw.
|
||||
systemStatus, systemReach := instanceStatusOK, instanceStatusPassed
|
||||
switch {
|
||||
case v.Status != vm.StateRunning:
|
||||
systemStatus, systemReach = instanceStatusNotApplicable, instanceStatusNotApplicable
|
||||
case systemImpaired:
|
||||
case systemImpaired, v.Health.IOErrorResumes > 0:
|
||||
systemStatus, systemReach = instanceStatusImpaired, instanceStatusFailed
|
||||
}
|
||||
|
||||
@@ -2928,6 +2954,18 @@ func instanceHealthSummary(v *vm.VM) (status, reachability string, impairedSince
|
||||
return instanceStatusImpaired, instanceStatusFailed, impPtr
|
||||
}
|
||||
|
||||
// Paused by werror=stop on a backend I/O error. The counter is cleared the
|
||||
// moment a poll sees the guest running again, so this reports for exactly
|
||||
// as long as the guest is actually held.
|
||||
if v.Health.IOErrorResumes > 0 {
|
||||
var impPtr *time.Time
|
||||
if !v.Health.IOErrorSince.IsZero() {
|
||||
since := v.Health.IOErrorSince
|
||||
impPtr = &since
|
||||
}
|
||||
return instanceStatusImpaired, instanceStatusFailed, impPtr
|
||||
}
|
||||
|
||||
// Grace period: freshly launched VMs report initializing until reachable.
|
||||
if v.Instance != nil && v.Instance.LaunchTime != nil &&
|
||||
time.Since(*v.Instance.LaunchTime) < instanceInitializingGrace {
|
||||
|
||||
@@ -1811,8 +1811,8 @@ func TestTerminateStoppedInstance_UserVolumeDeleted(t *testing.T) {
|
||||
|
||||
_, err := svc.TerminateStoppedInstance(context.Background(), &TerminateStoppedInstanceInput{InstanceID: id}, "acc")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"vol-user-001"}, vd.calls, "only DeleteOnTermination=true volumes deleted")
|
||||
assert.Equal(t, []string{"vol-user-001"}, vd.deleted)
|
||||
assert.Equal(t, []string{"vol-user-001"}, vd.deleted, "only DeleteOnTermination=true volumes deleted")
|
||||
assert.Equal(t, []string{"vol-keep-001"}, vd.detached, "terminate implies detach even for a volume that survives it")
|
||||
}
|
||||
|
||||
// TestTerminateStoppedInstance_StampsTeardownVolumesDone locks the
|
||||
@@ -1889,6 +1889,51 @@ func TestTerminateStoppedInstance_NonDoTBootVolumeDetachedNotDeleted(t *testing.
|
||||
assert.Equal(t, string(vm.TeardownDone), store.WroteTerminated[id].Teardown[vm.TeardownVolumes])
|
||||
}
|
||||
|
||||
// TestTerminateStoppedInstance_NonDoTDataVolumeDetachedNotDeleted covers the
|
||||
// non-Boot half. Stop's Unmount clears a data volume's attachment only when the
|
||||
// seal succeeded, so a volume whose seal failed is still in-use and pointing at
|
||||
// an instance that is about to stop existing. Skipping the detach here stranded
|
||||
// it with no owner left to release it.
|
||||
func TestTerminateStoppedInstance_NonDoTDataVolumeDetachedNotDeleted(t *testing.T) {
|
||||
id := "i-nondot-data"
|
||||
v := &vm.VM{ID: id, Status: vm.StateStopped, AccountID: "acc"}
|
||||
v.EBSRequests.Requests = []spxtypes.EBSRequest{
|
||||
{Name: "vol-data-nondot", DeleteOnTermination: false},
|
||||
}
|
||||
store := &vmmock.StateStore{Stopped: map[string]*vm.VM{id: v}}
|
||||
vd := &fakeVolumeDeleter{}
|
||||
svc := &InstanceServiceImpl{stoppedStore: store, volumeDeleter: vd}
|
||||
|
||||
_, err := svc.TerminateStoppedInstance(context.Background(), &TerminateStoppedInstanceInput{InstanceID: id}, "acc")
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, []string{"vol-data-nondot"}, vd.detached,
|
||||
"a data volume left attached by a failed seal must still be detached on terminate")
|
||||
assert.Empty(t, vd.deleted, "a DeleteOnTermination=false volume must never be deleted")
|
||||
}
|
||||
|
||||
// TestTerminateStoppedInstance_DetachOfAlreadyGoneVolumeIsNotAFailure pins the
|
||||
// idempotent half: a volume whose metadata doc is gone has already reached the
|
||||
// state the detach was asking for, so it must not stamp the teardown failed and
|
||||
// keep the instance record alive for the leak reaper.
|
||||
func TestTerminateStoppedInstance_DetachOfAlreadyGoneVolumeIsNotAFailure(t *testing.T) {
|
||||
id := "i-nondot-gone"
|
||||
v := &vm.VM{ID: id, Status: vm.StateStopped, AccountID: "acc"}
|
||||
v.EBSRequests.Requests = []spxtypes.EBSRequest{
|
||||
{Name: "vol-data-gone", DeleteOnTermination: false},
|
||||
}
|
||||
store := &vmmock.StateStore{Stopped: map[string]*vm.VM{id: v}}
|
||||
vd := &fakeVolumeDeleter{err: errors.New(awserrors.ErrorInvalidVolumeNotFound)}
|
||||
svc := &InstanceServiceImpl{stoppedStore: store, volumeDeleter: vd}
|
||||
|
||||
_, err := svc.TerminateStoppedInstance(context.Background(), &TerminateStoppedInstanceInput{InstanceID: id}, "acc")
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NotNil(t, store.WroteTerminated[id])
|
||||
assert.Equal(t, string(vm.TeardownDone), store.WroteTerminated[id].Teardown[vm.TeardownVolumes],
|
||||
"a volume that is already gone is the detach's goal, not a teardown failure")
|
||||
}
|
||||
|
||||
func TestTerminateStoppedInstance_NoVolumeDeleterSkipsGracefully(t *testing.T) {
|
||||
id := "i-no-vd"
|
||||
v := &vm.VM{ID: id, Status: vm.StateStopped, AccountID: "acc"}
|
||||
@@ -3967,6 +4012,58 @@ func TestBuildInstanceStatus_NeverSurfacesNonAWSName(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A guest werror=stop has paused on a backend I/O error is unreachable, and the
|
||||
// cause is ours rather than the guest's. Before this it reported ok/passed on
|
||||
// both checks, because QMP stays perfectly responsive while the guest is held —
|
||||
// so the one state a customer most needs to see was the one that looked fine.
|
||||
func TestBuildInstanceStatus_IOErrorPauseIsImpaired(t *testing.T) {
|
||||
owner := "111122223333"
|
||||
paused := runningVM("i-io", owner)
|
||||
paused.Health.IOErrorResumes = 1
|
||||
paused.Health.IOErrorSince = time.Now().Add(-45 * time.Second)
|
||||
svc := instanceStatusService(t, "az-a", map[string]*vm.VM{paused.ID: paused})
|
||||
|
||||
is := svc.buildInstanceStatus(paused, false)
|
||||
assert.Equal(t, "impaired", *is.InstanceStatus.Status)
|
||||
assert.Equal(t, "failed", *is.InstanceStatus.Details[0].Status)
|
||||
assert.Equal(t, "impaired", *is.SystemStatus.Status,
|
||||
"a backend I/O error is an infrastructure fault, so the system check must fail too")
|
||||
require.NotNil(t, is.InstanceStatus.Details[0].ImpairedSince)
|
||||
assert.Equal(t, paused.Health.IOErrorSince, *is.InstanceStatus.Details[0].ImpairedSince,
|
||||
"impairedSince must be when the guest first paused, not when it was escalated")
|
||||
}
|
||||
|
||||
// The counter is cleared by the first poll that sees the guest running, so
|
||||
// clearing it must take the instance back to ok on both checks. An impairment
|
||||
// that outlives the fault is as misleading as one that never appears.
|
||||
func TestBuildInstanceStatus_ResumedGuestClearsImpairment(t *testing.T) {
|
||||
owner := "111122223333"
|
||||
resumed := runningVM("i-io-ok", owner)
|
||||
resumed.Instance.LaunchTime = aws.Time(time.Now().Add(-time.Hour))
|
||||
resumed.Health.IOErrorResumes = 0
|
||||
svc := instanceStatusService(t, "az-a", map[string]*vm.VM{resumed.ID: resumed})
|
||||
|
||||
is := svc.buildInstanceStatus(resumed, false)
|
||||
assert.Equal(t, "ok", *is.InstanceStatus.Status)
|
||||
assert.Equal(t, "ok", *is.SystemStatus.Status)
|
||||
assert.Nil(t, is.InstanceStatus.Details[0].ImpairedSince)
|
||||
}
|
||||
|
||||
// A guest that pauses inside the launch grace window is impaired, not
|
||||
// initializing: the grace period exists to hide a guest that has not booted
|
||||
// yet, and one held on a failed write is not that.
|
||||
func TestBuildInstanceStatus_IOErrorBeatsLaunchGrace(t *testing.T) {
|
||||
owner := "111122223333"
|
||||
fresh := runningVM("i-io-new", owner)
|
||||
fresh.Instance.LaunchTime = aws.Time(time.Now().Add(-10 * time.Second))
|
||||
fresh.Health.IOErrorResumes = 2
|
||||
fresh.Health.IOErrorSince = time.Now().Add(-5 * time.Second)
|
||||
svc := instanceStatusService(t, "az-a", map[string]*vm.VM{fresh.ID: fresh})
|
||||
|
||||
is := svc.buildInstanceStatus(fresh, false)
|
||||
assert.Equal(t, "impaired", *is.InstanceStatus.Status)
|
||||
}
|
||||
|
||||
func TestDescribeInstanceStatus_IncludeAllSurfacesStopped(t *testing.T) {
|
||||
owner := "111122223333"
|
||||
stopped := runningVM("i-stop", owner)
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
package handlers_ec2_instance
|
||||
|
||||
//test:in-package — volumeHeldElsewhereError is unexported, and exporting it
|
||||
//would widen the handler API for a test rather than for a caller.
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// TestVolumeHeldElsewhere_NamesTheHolder covers both refusals that mean the
|
||||
// data is on another node. An operator who gets ServerInternal is told to retry
|
||||
// something that cannot succeed while that node is down, so the node name and
|
||||
// the reason have to survive the trip out of the daemon.
|
||||
func TestVolumeHeldElsewhere_NamesTheHolder(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
}{
|
||||
{
|
||||
name: "live holder, refused by the lease",
|
||||
err: errors.New("failed to mount volume: volume is leased by another owner: " +
|
||||
"held by node-b since 2026-09-02T17:04:39Z"),
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := volumeHeldElsewhereError(tc.err)
|
||||
require.Error(t, got, "a volume held elsewhere must not fall through to ServerInternal")
|
||||
|
||||
code := awserrors.ValidErrorCodeFromError(got)
|
||||
assert.NotEqual(t, awserrors.ErrorServerInternal, code,
|
||||
"ServerInternal reads as a transient fault and invites a retry that cannot succeed")
|
||||
assert.Contains(t, got.Error(), "node-b", "the caller has to learn where the data is")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestVolumeHeldElsewhere_IgnoresUnrelatedFailures guards the other direction:
|
||||
// a genuine internal fault must keep reporting as one rather than being
|
||||
// relabelled as somebody else's volume.
|
||||
func TestVolumeHeldElsewhere_IgnoresUnrelatedFailures(t *testing.T) {
|
||||
assert.NoError(t, volumeHeldElsewhereError(nil))
|
||||
assert.NoError(t, volumeHeldElsewhereError(errors.New("nbd endpoint not ready: timeout waiting for unix socket")))
|
||||
assert.NoError(t, volumeHeldElsewhereError(fmt.Errorf("qemu exited: %w", errors.New("no such file"))))
|
||||
|
||||
// A volume whose writes are merely unsealed is opened with a warning rather
|
||||
// than refused, so this text is not an exclusion and must not become one.
|
||||
assert.NoError(t, volumeHeldElsewhereError(errors.New(
|
||||
"took over from node-b, which held unsealed writes since 2026-09-02T17:04:39Z")))
|
||||
}
|
||||
@@ -1116,10 +1116,12 @@ func (s *VolumeServiceImpl) DeleteVolume(ctx context.Context, input *ec2.DeleteV
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, Handle: meta.ProviderHandle,
|
||||
}); err != nil {
|
||||
slog.ErrorContext(ctx, "DeleteVolume: provider deletion failed", "volumeId", volumeID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
// The terminate reaper retries this every two minutes and logs whatever
|
||||
// it gets back, so a bare code makes a persistent fault unreadable.
|
||||
return nil, awserrors.Errorf(awserrors.ErrorServerInternal, "delete volume %s: %v", volumeID, err)
|
||||
}
|
||||
if err := s.metadata.DeleteVolume(ctx, accountID, volumeID); err != nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
return nil, awserrors.Errorf(awserrors.ErrorServerInternal, "delete volume metadata %s: %v", volumeID, err)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "DeleteVolume completed", "volumeId", volumeID)
|
||||
|
||||
@@ -1796,6 +1796,46 @@ func TestDetachVolumeOnTerminate_AlreadyGoneMetadataIsSuccess(t *testing.T) {
|
||||
require.NoError(t, err, "a volume with no metadata document is already gone, not a teardown failure")
|
||||
}
|
||||
|
||||
// failingDeleteProvider makes the provider's delete fail while leaving every
|
||||
// other operation intact, which is the shape of a blob node that is down.
|
||||
type failingDeleteProvider struct {
|
||||
ebsprovider.EBSProvider
|
||||
|
||||
err error
|
||||
}
|
||||
|
||||
func (p *failingDeleteProvider) DeleteVolume(context.Context, ebsprovider.DeleteVolumeRequest) error {
|
||||
return p.err
|
||||
}
|
||||
|
||||
// TestDeleteVolume_ProviderFailureCarriesItsCause pins that a provider delete
|
||||
// failure keeps its cause on the way out. The terminate reaper retries this
|
||||
// every two minutes and logs whatever it is handed, so a bare ServerInternal
|
||||
// makes a persistent fault indistinguishable from a transient one.
|
||||
func TestDeleteVolume_ProviderFailureCarriesItsCause(t *testing.T) {
|
||||
kv := setupTestVolumeKV(t)
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
svc.snapshotKV = kv
|
||||
|
||||
volumeID := "vol-provider-down"
|
||||
createVolumeInStoreWithMeta(t, svc, store, volumeID, ebsmetadata.Volume{
|
||||
VolumeID: volumeID, CapacityGiB: 10, State: "available",
|
||||
})
|
||||
svc.SetEBSProvider(&failingDeleteProvider{
|
||||
EBSProvider: svc.provider,
|
||||
err: errors.New("dial node 6: quic dial 10.10.8.5:6660: timeout"),
|
||||
})
|
||||
|
||||
_, err := svc.DeleteVolume(context.Background(), &ec2.DeleteVolumeInput{VolumeId: aws.String(volumeID)}, testVolAccountID)
|
||||
require.Error(t, err)
|
||||
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, awserrors.ValidErrorCodeFromError(err),
|
||||
"the code stays ServerInternal so callers classify it unchanged")
|
||||
assert.Contains(t, err.Error(), "dial node 6",
|
||||
"an operator reading the reaper log must learn why the delete keeps failing")
|
||||
}
|
||||
|
||||
// TestDeleteVolumeOnTerminate_SurfacesDeleteFailure verifies a DeleteVolume
|
||||
// failure downstream of the attachment clear is returned, not swallowed — the
|
||||
// caller (deleteInstanceVolumes / instanceCleanerAdapter.DeleteVolumes) relies
|
||||
|
||||
@@ -22,7 +22,8 @@ func newCATestService(t *testing.T, pem string) (http.Handler, string) {
|
||||
if pem != "" {
|
||||
require.NoError(t, os.WriteFile(path, []byte(pem), 0o600))
|
||||
}
|
||||
svc, _ := newTestService(&fakeResolver{eni: testENI()}, &fakeIAM{}, &fakeAssumer{})
|
||||
// A resolvable instance, because the meta-data listing now 500s without one.
|
||||
svc, _ := newTestService(&fakeResolver{eni: testENI(), inst: &instanceFacts{}}, &fakeIAM{}, &fakeAssumer{})
|
||||
svc.caCert = newCACertCache(path)
|
||||
return withTapENI(svc.httpHandler(), testENI()), path
|
||||
}
|
||||
|
||||
@@ -260,6 +260,10 @@ func (s *IMDSServiceImpl) dispatch(w http.ResponseWriter, r *http.Request, eni *
|
||||
// mid-listing and falls back to DataSourceNone: public-hostname/public-ipv4 only
|
||||
// with a public IP, public-keys/ only with a key pair, iam/ only with an instance
|
||||
// profile — each omission matching real EC2, like the macs/ subtree in macKeys.
|
||||
//
|
||||
// Every omission is therefore a positive claim about the instance, which is why
|
||||
// this 500s rather than shortening the listing when it cannot resolve one: a
|
||||
// shorter listing is indistinguishable from a smaller instance and is believed.
|
||||
func (s *IMDSServiceImpl) serveMetaDataRoot(ctx context.Context, w http.ResponseWriter, eni *eniFacts) {
|
||||
keys := []string{
|
||||
"ami-id", "ami-launch-index", "hostname", "instance-id",
|
||||
@@ -270,17 +274,27 @@ func (s *IMDSServiceImpl) serveMetaDataRoot(ctx context.Context, w http.Response
|
||||
if eni.publicIP != "" {
|
||||
keys = append(keys, "public-hostname", "public-ipv4")
|
||||
}
|
||||
// Resolve the instance once: public-keys/ is listed only with a key pair, iam/
|
||||
// only with a resolvable instance profile. A backend error counts as absent so
|
||||
// the listing never advertises a child whose leaf would 404 and break the crawl.
|
||||
if inst, err := s.resolver.resolveInstance(ctx, eni); err == nil && inst != nil {
|
||||
if inst.keyName != "" {
|
||||
keys = append(keys, "public-keys/")
|
||||
}
|
||||
if inst.iamInstanceProfileArn != "" {
|
||||
if p, err := s.iam.ResolveInstanceProfile(ctx, eni.iamAccountID(), inst.iamInstanceProfileArn); err == nil && p != nil {
|
||||
keys = append(keys, "iam/")
|
||||
}
|
||||
// An unresolvable instance withholds the whole listing rather than shortening
|
||||
// it: omitting public-keys/ reads to cloud-init as "this instance has no key
|
||||
// pair", and the guest boots keyless with nothing anywhere calling it an error.
|
||||
inst, err := s.resolver.resolveInstance(ctx, eni)
|
||||
if err != nil || inst == nil {
|
||||
slog.ErrorContext(ctx, "IMDS: withholding meta-data listing, instance did not resolve",
|
||||
"instance_id", eni.instanceID, "err", err)
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if inst.keyName != "" {
|
||||
keys = append(keys, "public-keys/")
|
||||
}
|
||||
if inst.iamInstanceProfileArn != "" {
|
||||
p, err := s.iam.ResolveInstanceProfile(ctx, eni.iamAccountID(), inst.iamInstanceProfileArn)
|
||||
switch {
|
||||
case err != nil:
|
||||
slog.ErrorContext(ctx, "IMDS: instance profile resolution failed, withholding iam/ from the listing",
|
||||
"instance_id", eni.instanceID, "profile", inst.iamInstanceProfileArn, "err", err)
|
||||
case p != nil:
|
||||
keys = append(keys, "iam/")
|
||||
}
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
@@ -572,6 +572,43 @@ func TestHTTP_DirectoryListingNoPublicIP(t *testing.T) {
|
||||
assert.Contains(t, body, "public-keys/")
|
||||
}
|
||||
|
||||
// An instance that does not resolve withholds the listing entirely. Shortening it
|
||||
// instead would drop public-keys/, which cloud-init reads as "no key pair" — it
|
||||
// writes no authorized_keys, and the guest boots keyless with nothing logged.
|
||||
func TestHTTP_DirectoryListingUnresolvedInstanceIs500(t *testing.T) {
|
||||
for name, res := range map[string]*fakeResolver{
|
||||
"backend error": {eni: testENI(), instErr: errors.New("jetstream: context deadline exceeded")},
|
||||
"not found": {eni: testENI(), inst: nil},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
svc, _ := newTestService(res, &fakeIAM{}, &fakeAssumer{})
|
||||
h := withTapENI(svc.httpHandler(), testENI())
|
||||
token := issueToken(t, h)
|
||||
rec := get(t, h, pathMetaDataRoot, token)
|
||||
assert.Equal(t, http.StatusInternalServerError, rec.Code,
|
||||
"an unresolved instance must not be served as a shorter listing")
|
||||
assert.NotContains(t, rec.Body.String(), "ami-id")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A resolvable instance whose profile lookup fails still serves the listing, minus
|
||||
// iam/. The key pair is the field a guest cannot recover from losing; a missing
|
||||
// role surfaces through the credentials leaf, which reports the miss on its own.
|
||||
func TestHTTP_DirectoryListingProfileErrorKeepsTheListing(t *testing.T) {
|
||||
res := &fakeResolver{eni: testENI(), inst: &instanceFacts{
|
||||
keyName: "e2e-key",
|
||||
iamInstanceProfileArn: "arn:aws:iam::111122223333:instance-profile/app-profile",
|
||||
}}
|
||||
svc, _ := newTestService(res, &fakeIAM{profileErr: errors.New("iam unavailable")}, &fakeAssumer{})
|
||||
h := withTapENI(svc.httpHandler(), testENI())
|
||||
token := issueToken(t, h)
|
||||
rec := get(t, h, pathMetaDataRoot, token)
|
||||
assert.Equal(t, http.StatusOK, rec.Code)
|
||||
assert.Contains(t, rec.Body.String(), "public-keys/")
|
||||
assert.NotContains(t, rec.Body.String(), "iam/")
|
||||
}
|
||||
|
||||
// An instance launched without a key pair omits public-keys/, whose leaf would 404.
|
||||
func TestHTTP_DirectoryListingNoKeyPair(t *testing.T) {
|
||||
res := &fakeResolver{eni: testENI(), inst: &instanceFacts{}}
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
|
||||
"github.com/aws/aws-sdk-go/service/iam"
|
||||
handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
@@ -111,13 +112,21 @@ func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer
|
||||
return nil, fmt.Errorf("get JetStream context: %w", err)
|
||||
}
|
||||
|
||||
eniKV, err := js.KeyValue(ctx, kvBucketENIs)
|
||||
// Every bucket below is retried for the same window before its policy
|
||||
// applies. vpcd starts alongside NATS, so a bucket that is merely not ready
|
||||
// yet must not decide how this node serves metadata for its whole lifetime.
|
||||
openBucket := func(name string) (jetstream.KeyValue, error) {
|
||||
return kvstore.OpenWithRetry(ctx, name, kvstore.DefaultOpenWindow,
|
||||
func(ctx context.Context) (jetstream.KeyValue, error) { return js.KeyValue(ctx, name) })
|
||||
}
|
||||
|
||||
eniKV, err := openBucket(kvBucketENIs)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open %s bucket: %w", kvBucketENIs, err)
|
||||
}
|
||||
|
||||
// Open SG bucket best-effort; IMDS starts fine without it, degrading to raw IDs.
|
||||
sgKV, err := js.KeyValue(ctx, kvBucketSecurityGroups)
|
||||
sgKV, err := openBucket(kvBucketSecurityGroups)
|
||||
if err != nil {
|
||||
slog.Warn("IMDS: security-group bucket unavailable, /security-groups will serve IDs", "bucket", kvBucketSecurityGroups, "err", err)
|
||||
sgKV = nil
|
||||
@@ -125,12 +134,12 @@ func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer
|
||||
|
||||
// Open subnet/VPC buckets best-effort; the network-interfaces CIDR leaves 404
|
||||
// (and drop from the listing) when a bucket is unavailable, IMDS still starts.
|
||||
subnetKV, err := js.KeyValue(ctx, kvBucketSubnets)
|
||||
subnetKV, err := openBucket(kvBucketSubnets)
|
||||
if err != nil {
|
||||
slog.Warn("IMDS: subnet bucket unavailable, network-interfaces subnet CIDR will 404", "bucket", kvBucketSubnets, "err", err)
|
||||
subnetKV = nil
|
||||
}
|
||||
vpcKV, err := js.KeyValue(ctx, kvBucketVPCs)
|
||||
vpcKV, err := openBucket(kvBucketVPCs)
|
||||
if err != nil {
|
||||
slog.Warn("IMDS: vpc bucket unavailable, network-interfaces VPC CIDR will 404", "bucket", kvBucketVPCs, "err", err)
|
||||
vpcKV = nil
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
package kvstore
|
||||
|
||||
import "time"
|
||||
|
||||
// SetOpenRetryInterval shortens the OpenWithRetry pause for tests and returns a
|
||||
// function restoring the previous value.
|
||||
func SetOpenRetryInterval(d time.Duration) func() {
|
||||
prev := openRetryInterval
|
||||
openRetryInterval = d
|
||||
return func() { openRetryInterval = prev }
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package kvstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
)
|
||||
|
||||
// DefaultOpenWindow is how long OpenWithRetry keeps trying. Long enough to
|
||||
// cover JetStream electing a leader after a whole-cluster restart, which is the
|
||||
// window in which every service starts at once and races it.
|
||||
const DefaultOpenWindow = 90 * time.Second
|
||||
|
||||
// openRetryInterval is the pause between attempts. Each attempt carries its own
|
||||
// JetStream API deadline, so this only adds to it. A var so tests can shorten
|
||||
// it; nothing in production writes it.
|
||||
var openRetryInterval = 2 * time.Second
|
||||
|
||||
// OpenWithRetry calls open until it succeeds or window expires, and reports how
|
||||
// long it waited. A bucket that is merely not ready yet is the common case on a
|
||||
// cold cluster; the caller decides what an exhausted window means.
|
||||
func OpenWithRetry[T any](ctx context.Context, what string, window time.Duration, open func(context.Context) (T, error)) (T, error) {
|
||||
var zero T
|
||||
if window <= 0 {
|
||||
window = DefaultOpenWindow
|
||||
}
|
||||
|
||||
started := time.Now()
|
||||
deadline := started.Add(window)
|
||||
attempt := 0
|
||||
|
||||
for {
|
||||
attempt++
|
||||
v, err := open(ctx)
|
||||
if err == nil {
|
||||
if attempt > 1 {
|
||||
slog.InfoContext(ctx, "kvstore: opened after retrying", "what", what,
|
||||
"attempts", attempt, "waited_ms", otelsetup.Millis(time.Since(started)))
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
if ctx.Err() != nil {
|
||||
return zero, fmt.Errorf("open %s: %w", what, ctx.Err())
|
||||
}
|
||||
if !time.Now().Add(openRetryInterval).Before(deadline) {
|
||||
return zero, fmt.Errorf("open %s: giving up after %d attempts in %s: %w",
|
||||
what, attempt, window, err)
|
||||
}
|
||||
|
||||
slog.WarnContext(ctx, "kvstore: not ready, retrying", "what", what,
|
||||
"attempt", attempt, "window_ms", otelsetup.Millis(window), "err", err)
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return zero, fmt.Errorf("open %s: %w", what, ctx.Err())
|
||||
case <-time.After(openRetryInterval):
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package kvstore_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestOpenWithRetry_SucceedsFirstAttempt(t *testing.T) {
|
||||
calls := 0
|
||||
got, err := kvstore.OpenWithRetry(context.Background(), "bucket", time.Second,
|
||||
func(context.Context) (string, error) { calls++; return "kv", nil })
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "kv", got)
|
||||
assert.Equal(t, 1, calls, "a bucket that opens must not be retried")
|
||||
}
|
||||
|
||||
// TestOpenWithRetry_SucceedsAfterNotReady is the case the whole helper exists
|
||||
// for: vpcd starts alongside NATS, and a bucket that is not ready for the first
|
||||
// few seconds is normal rather than a reason to degrade for the process's life.
|
||||
func TestOpenWithRetry_SucceedsAfterNotReady(t *testing.T) {
|
||||
defer kvstore.SetOpenRetryInterval(time.Millisecond)()
|
||||
|
||||
calls := 0
|
||||
got, err := kvstore.OpenWithRetry(context.Background(), "bucket", time.Second,
|
||||
func(context.Context) (string, error) {
|
||||
calls++
|
||||
if calls < 3 {
|
||||
return "", errors.New("context deadline exceeded")
|
||||
}
|
||||
return "kv", nil
|
||||
})
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "kv", got)
|
||||
assert.Equal(t, 3, calls)
|
||||
}
|
||||
|
||||
func TestOpenWithRetry_GivesUpAndKeepsTheCause(t *testing.T) {
|
||||
defer kvstore.SetOpenRetryInterval(time.Millisecond)()
|
||||
cause := errors.New("no servers available")
|
||||
|
||||
_, err := kvstore.OpenWithRetry(context.Background(), "spinifex-instance-state", 20*time.Millisecond,
|
||||
func(context.Context) (string, error) { return "", cause })
|
||||
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, cause, "the caller has to be able to see why, not just that")
|
||||
assert.Contains(t, err.Error(), "spinifex-instance-state")
|
||||
}
|
||||
|
||||
func TestOpenWithRetry_StopsOnCancelledContext(t *testing.T) {
|
||||
defer kvstore.SetOpenRetryInterval(time.Millisecond)()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
|
||||
calls := 0
|
||||
_, err := kvstore.OpenWithRetry(ctx, "bucket", time.Minute,
|
||||
func(context.Context) (string, error) {
|
||||
calls++
|
||||
cancel()
|
||||
return "", errors.New("not ready")
|
||||
})
|
||||
|
||||
require.ErrorIs(t, err, context.Canceled)
|
||||
assert.Equal(t, 1, calls, "a cancelled shutdown must not sit out the window")
|
||||
}
|
||||
Vendored
+40
-8
@@ -157,7 +157,13 @@ func (m *igwManager) AttachIGW(ctx context.Context, spec IGWSpec) error {
|
||||
}
|
||||
|
||||
// The gateway switch port on the shared switch is the per-VPC attach marker.
|
||||
if _, err := m.ovn.GetLogicalSwitchPort(ctx, switchGWPortName); err == nil {
|
||||
if existing, err := m.ovn.GetLogicalSwitchPort(ctx, switchGWPortName); err == nil {
|
||||
// Converge rather than early-return: a VPC attached before EIPs were
|
||||
// advertised carries no nat-addresses, and nothing else recreates this
|
||||
// port, so its EIPs would stay invisible to the physical network.
|
||||
if err := m.ensureNATAdvertisement(ctx, existing); err != nil {
|
||||
return err
|
||||
}
|
||||
// OVN state survives reboots but host routes do not: re-run the routed
|
||||
// leg so reconcile's replayed attach re-ensures ingress + SNAT.
|
||||
if m.natMode == policy.NATModeRouted {
|
||||
@@ -200,10 +206,7 @@ func (m *igwManager) AttachIGW(ctx context.Context, spec IGWSpec) error {
|
||||
|
||||
// nat-addresses is only honoured on a router-type port; OVN silently ignores
|
||||
// it on the localnet, which leaves EIPs unadvertised on the physical network.
|
||||
gwPortOpts := map[string]string{"router-port": gwPortName}
|
||||
if m.gatewayOwnsNAT() {
|
||||
gwPortOpts["nat-addresses"] = "router"
|
||||
}
|
||||
gwPortOpts := map[string]string{"router-port": gwPortName, natAdvertiseOption: "router"}
|
||||
if err := m.ovn.CreateLogicalSwitchPort(ctx, extSwitchName, &nbdb.LogicalSwitchPort{
|
||||
Name: switchGWPortName,
|
||||
Type: "router",
|
||||
@@ -372,6 +375,35 @@ func (m *igwManager) gatewayLRPIP(ctx context.Context, vpcID string) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// natAdvertiseOption asks ovn-northd to publish the router's NAT external
|
||||
// addresses on this port, which is what makes ovn-controller send a gratuitous
|
||||
// ARP for an EIP.
|
||||
//
|
||||
// Needed in every NAT mode, not only the centralised ones. For a distributed
|
||||
// dnat_and_snat northd scopes each entry with is_chassis_resident(<vif port>),
|
||||
// so the GARP leaves from the chassis running the instance rather than the
|
||||
// gateway — which is exactly right, and only happens if this is set.
|
||||
const natAdvertiseOption = "nat-addresses"
|
||||
|
||||
// ensureNATAdvertisement adds the NAT advertisement option to a gateway switch
|
||||
// port that predates it. Without it nothing announces an EIP when it moves, so
|
||||
// any host on the segment keeps resolving it to the previous holder's MAC until
|
||||
// its ARP cache ages out — a recycled EIP is unreachable for that whole window.
|
||||
func (m *igwManager) ensureNATAdvertisement(ctx context.Context, port *nbdb.LogicalSwitchPort) error {
|
||||
if port == nil || port.Options[natAdvertiseOption] == "router" {
|
||||
return nil
|
||||
}
|
||||
if port.Options == nil {
|
||||
port.Options = map[string]string{}
|
||||
}
|
||||
port.Options[natAdvertiseOption] = "router"
|
||||
if err := m.ovn.UpdateLogicalSwitchPort(ctx, port); err != nil {
|
||||
return fmt.Errorf("set %s on %s: %w", natAdvertiseOption, port.Name, err)
|
||||
}
|
||||
slog.Info("external: enabled EIP advertisement on gateway port", "port", port.Name)
|
||||
return nil
|
||||
}
|
||||
|
||||
// ensureSharedExternal idempotently creates the singleton shared external switch
|
||||
// and its single localnet port. NAT advertisement lives on the per-VPC gateway
|
||||
// port, not here — OVN ignores nat-addresses on a localnet port.
|
||||
@@ -387,12 +419,12 @@ func (m *igwManager) ensureSharedExternal(ctx context.Context, switchName, portN
|
||||
// advertisement moved to the gateway port still carry a stale nat-addresses
|
||||
// here, and it must not survive an upgrade.
|
||||
if existing, err := m.ovn.GetLogicalSwitchPort(ctx, portName); err == nil {
|
||||
if _, stale := existing.Options["nat-addresses"]; !stale {
|
||||
if _, stale := existing.Options[natAdvertiseOption]; !stale {
|
||||
return nil
|
||||
}
|
||||
delete(existing.Options, "nat-addresses")
|
||||
delete(existing.Options, natAdvertiseOption)
|
||||
if err := m.ovn.UpdateLogicalSwitchPort(ctx, existing); err != nil {
|
||||
return fmt.Errorf("clear stale nat-addresses on %s: %w", portName, err)
|
||||
return fmt.Errorf("clear stale %s on %s: %w", natAdvertiseOption, portName, err)
|
||||
}
|
||||
slog.Info("external: cleared stale nat-addresses from localnet port", "port", portName)
|
||||
return nil
|
||||
|
||||
+38
-2
@@ -100,10 +100,13 @@ func TestAttachIGW_Distributed_LinkLocalLRP(t *testing.T) {
|
||||
_, hasNat := localnet.Options["nat-addresses"]
|
||||
assert.False(t, hasNat, "nat-addresses is never valid on a localnet port")
|
||||
|
||||
// Distributed mode needs this too. northd scopes each NAT entry with
|
||||
// is_chassis_resident(<vif port>), so the GARP leaves the chassis running
|
||||
// the instance — and without it nothing announces an EIP when it moves.
|
||||
gwPort, err := m.GetLogicalSwitchPort(ctx, topology.GatewaySwitchPort("vpc-1"))
|
||||
require.NoError(t, err)
|
||||
_, hasGwNat := gwPort.Options["nat-addresses"]
|
||||
assert.False(t, hasGwNat, "distributed mode must NOT set nat-addresses")
|
||||
assert.Equal(t, "router", gwPort.Options["nat-addresses"],
|
||||
"an unadvertised EIP keeps resolving to the previous holder's MAC until the ARP cache ages out")
|
||||
|
||||
// Gateway LRP exists with link-local network.
|
||||
lrp, err := m.GetLogicalRouterPort(ctx, topology.GatewayRouterPort("vpc-1"))
|
||||
@@ -195,6 +198,39 @@ func TestAttachIGW_ClearsStaleLocalnetNATAddresses(t *testing.T) {
|
||||
assert.Equal(t, "external", localnet.Options["network_name"], "converge must not drop network_name")
|
||||
}
|
||||
|
||||
// TestAttachIGW_ConvergesNATAdvertisementOnAnAttachedVPC covers the upgrade
|
||||
// path, which is the only path that matters on a running cluster. A VPC
|
||||
// attached before EIPs were advertised takes the already-attached early return,
|
||||
// and nothing else ever rewrites this port — so without a converge its EIPs
|
||||
// stay unannounced for the life of the VPC.
|
||||
func TestAttachIGW_ConvergesNATAdvertisementOnAnAttachedVPC(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
m := mock.New()
|
||||
seedVPCRouter(t, m, "vpc-1", "10.0.0.0/16")
|
||||
|
||||
extSwitchName := topology.ExternalSwitchShared()
|
||||
gwPortName := topology.GatewaySwitchPort("vpc-1")
|
||||
_, _, err := m.EnsureLogicalSwitch(ctx, &nbdb.LogicalSwitch{Name: extSwitchName})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, m.CreateLogicalSwitchPort(ctx, extSwitchName, &nbdb.LogicalSwitchPort{
|
||||
Name: gwPortName,
|
||||
Type: "router",
|
||||
Addresses: []string{"router"},
|
||||
Options: map[string]string{"router-port": topology.GatewayRouterPort("vpc-1")},
|
||||
}))
|
||||
|
||||
pool := &ExternalPoolConfig{Name: "p", Gateway: "192.168.1.1", PrefixLen: 24}
|
||||
mgr, _ := newTestIGWManager(t, m, policy.NATModeDistributed, pool, LinkLocalAllocator{}, []string{"chassis-a"})
|
||||
require.NoError(t, mgr.AttachIGW(ctx, IGWSpec{VPCID: "vpc-1", InternetGatewayID: "igw-1"}))
|
||||
|
||||
gwPort, err := m.GetLogicalSwitchPort(ctx, gwPortName)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "router", gwPort.Options["nat-addresses"],
|
||||
"an already-attached VPC must gain EIP advertisement, not keep silence")
|
||||
assert.Equal(t, topology.GatewayRouterPort("vpc-1"), gwPort.Options["router-port"],
|
||||
"converge must not drop router-port")
|
||||
}
|
||||
|
||||
// The LRP address is written once at attach, so a lease re-issued on a new IP
|
||||
// leaves the port answering ARP for an address nothing renews.
|
||||
func TestRebindGatewayIP_MovesLRPAndStamp(t *testing.T) {
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"maps"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
@@ -38,6 +39,10 @@ var (
|
||||
guestPortDatapathInterval = 5 * time.Second
|
||||
)
|
||||
|
||||
// How many guest ports may wait for convergence at once. Bounded because each
|
||||
// miss nudges a recompute, and an unbounded fan-out would nudge once per port.
|
||||
var guestPortDatapathConcurrency = 8
|
||||
|
||||
// After this many recompute misses with a binding still down, the readiness loops
|
||||
// check SB connectivity and, if the local ovn-controller is not "connected",
|
||||
// escalate once to sb-cluster-state-reset. A recompute re-evaluates flows from the
|
||||
@@ -45,6 +50,9 @@ var (
|
||||
// reset re-syncs that view. Package var so tests can shrink it.
|
||||
var sbResetEscalateAfter = 3
|
||||
|
||||
// Minimum gap between two sb-cluster-state-resets. Package var so tests shrink it.
|
||||
var sbResetMinInterval = 2 * time.Minute
|
||||
|
||||
// Backoff applied to a guest port that burned its convergence deadline. Doubles
|
||||
// per consecutive failure from the base and holds at the cap. Package vars so
|
||||
// tests shorten them.
|
||||
@@ -601,6 +609,14 @@ func (r *reconciler) ensureGatewayDatapath(ctx context.Context, vpcID, gwIP, eip
|
||||
// Returns true when a reset was issued (caller stops escalating); false when the SB
|
||||
// is connected (recompute is the right tool) or the probe failed (retry next miss).
|
||||
func (r *reconciler) escalateSBReset(ctx context.Context, logKV ...any) bool {
|
||||
r.sbResetMu.Lock()
|
||||
defer r.sbResetMu.Unlock()
|
||||
// One reset re-syncs the SB view for every port, so a second one moments
|
||||
// later answers a question the first already answered.
|
||||
if time.Since(r.sbResetLast) < sbResetMinInterval {
|
||||
return true
|
||||
}
|
||||
|
||||
status, err := r.gwClaim.SBConnectionState(ctx)
|
||||
if err != nil {
|
||||
slog.Warn("reconcile/apply: SB connection-status probe failed during escalation", append(logKV, "err", err)...)
|
||||
@@ -614,6 +630,7 @@ func (r *reconciler) escalateSBReset(ctx context.Context, logKV ...any) bool {
|
||||
if err := r.gwClaim.ResetSBClusterState(ctx); err != nil {
|
||||
slog.Warn("reconcile/apply: sb-cluster-state-reset failed", append(logKV, "err", err)...)
|
||||
}
|
||||
r.sbResetLast = time.Now()
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -672,13 +689,36 @@ func (r *reconciler) ensureGatewayClaimed(ctx context.Context, crPortName string
|
||||
// whose port has not converged (e.g. just after a host reboot) stays dark while
|
||||
// every other signal is green.
|
||||
func (r *reconciler) applyEIPs(ctx context.Context, intent IntentState, _ ActualState, res *passResult) {
|
||||
for _, spec := range r.floatingIPSpecs(intent) {
|
||||
specs := r.floatingIPSpecs(intent)
|
||||
|
||||
// The DNAT rows go in first and in order: AddEIP mutates NB and is the cheap
|
||||
// half, so serialising it costs nothing and keeps NB writes single-threaded.
|
||||
for _, spec := range specs {
|
||||
if err := r.nat.AddEIP(ctx, spec); err != nil {
|
||||
slog.Error("reconcile/apply: AddEIP failed", "external_ip", spec.ExternalIP, "logical_ip", spec.LogicalIP, "err", err)
|
||||
res.fail(classEIP, spec.ExternalIP, err)
|
||||
}
|
||||
r.ensureGuestPortDatapath(ctx, spec)
|
||||
}
|
||||
|
||||
// The convergence waits run together. Each one can burn the full
|
||||
// guestPortDatapathTimeout, so serialising them made every port wait out
|
||||
// every port ahead of it: a guest whose neighbours are gone was dark for
|
||||
// N x 45s before its own probe even started.
|
||||
sem := make(chan struct{}, guestPortDatapathConcurrency)
|
||||
var wg sync.WaitGroup
|
||||
for _, spec := range specs {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
wg.Wait()
|
||||
return
|
||||
case sem <- struct{}{}:
|
||||
}
|
||||
wg.Go(func() {
|
||||
defer func() { <-sem }()
|
||||
r.ensureGuestPortDatapath(ctx, spec)
|
||||
})
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// floatingIPSpecs is every dnat_and_snat the datapath must carry: user EIPs
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
package reconcile
|
||||
|
||||
//test:in-package — drives applyEIPs and the unexported convergence bounds
|
||||
//directly, neither of which is reachable from an external test package.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/network/ovn/mock"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/ovn/nbdb"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/policy"
|
||||
)
|
||||
|
||||
// concurrentClaimVerifier reports livePort up and every other port permanently
|
||||
// down, recording when the live port was first observed. Thread-safe because
|
||||
// the ports are probed concurrently, which is the point of the test.
|
||||
type concurrentClaimVerifier struct {
|
||||
livePort string
|
||||
|
||||
mu sync.Mutex
|
||||
nudges int
|
||||
liveSeen time.Time
|
||||
}
|
||||
|
||||
func (f *concurrentClaimVerifier) GuestPortUp(_ context.Context, lspName string) (bool, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if lspName != f.livePort {
|
||||
return false, nil
|
||||
}
|
||||
if f.liveSeen.IsZero() {
|
||||
f.liveSeen = time.Now()
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (f *concurrentClaimVerifier) NudgeRecompute(context.Context) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.nudges++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *concurrentClaimVerifier) whenLiveSeen() time.Time {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.liveSeen
|
||||
}
|
||||
|
||||
func (f *concurrentClaimVerifier) GatewayPortClaimed(context.Context, string) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
func (f *concurrentClaimVerifier) GatewayReachable(context.Context, string) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
func (f *concurrentClaimVerifier) EIPReachable(context.Context, string) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
func (f *concurrentClaimVerifier) RepairDatapath(context.Context) error { return nil }
|
||||
func (f *concurrentClaimVerifier) SBConnectionState(context.Context) (string, error) {
|
||||
return "connected", nil
|
||||
}
|
||||
func (f *concurrentClaimVerifier) ResetSBClusterState(context.Context) error { return nil }
|
||||
|
||||
// countingNAT accepts every AddEIP. Only that method is exercised by applyEIPs,
|
||||
// so the rest of NATManager is inherited and never called.
|
||||
type countingNAT struct {
|
||||
policy.NATManager
|
||||
|
||||
mu sync.Mutex
|
||||
n int
|
||||
}
|
||||
|
||||
func (c *countingNAT) AddEIP(context.Context, policy.EIPSpec) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.n++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *countingNAT) count() int {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.n
|
||||
}
|
||||
|
||||
// ovnWithGuestLSPs returns a mock OVN client carrying every named port, which
|
||||
// ensureGuestPortDatapath requires before it will probe.
|
||||
func ovnWithGuestLSPs(t *testing.T, lspNames ...string) *mock.Client {
|
||||
t.Helper()
|
||||
m := mock.New()
|
||||
if err := m.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("mock Connect: %v", err)
|
||||
}
|
||||
if err := m.CreateLogicalSwitch(context.Background(), &nbdb.LogicalSwitch{Name: "ls-test"}); err != nil {
|
||||
t.Fatalf("CreateLogicalSwitch: %v", err)
|
||||
}
|
||||
for _, name := range lspNames {
|
||||
if err := m.CreateLogicalSwitchPort(context.Background(), "ls-test", &nbdb.LogicalSwitchPort{Name: name}); err != nil {
|
||||
t.Fatalf("CreateLogicalSwitchPort %s: %v", name, err)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// TestApplyEIPs_HealthyPortDoesNotWaitForDeadOnes is the reproduction of the
|
||||
// dev-prod symptom: three ports belonging to long-stopped instances each burned
|
||||
// the full convergence deadline in turn, so a newly launched guest's EIP was
|
||||
// dark for the sum of them before its own probe even started. The journal shows
|
||||
// three consecutive 45s burns and the live guest's AddEIP landing only after
|
||||
// the first one finished.
|
||||
//
|
||||
// The assertion is wall time, so it holds whatever order the intent map yields.
|
||||
func TestApplyEIPs_HealthyPortDoesNotWaitForDeadOnes(t *testing.T) {
|
||||
withFastGuestPortBounds(t)
|
||||
|
||||
const live = "port-eni-live"
|
||||
dead := []string{"port-eni-dead1", "port-eni-dead2", "port-eni-dead3"}
|
||||
|
||||
f := &concurrentClaimVerifier{livePort: live}
|
||||
nat := &countingNAT{}
|
||||
r := &reconciler{
|
||||
gwClaim: f,
|
||||
nat: nat,
|
||||
ovn: ovnWithGuestLSPs(t, append([]string{live}, dead...)...),
|
||||
}
|
||||
|
||||
intent := IntentState{EIPs: map[string]policy.EIPSpec{
|
||||
"10.0.0.1": {VPCID: "vpc-a", ExternalIP: "192.0.2.1", LogicalIP: "10.0.0.1", PortName: live},
|
||||
}}
|
||||
for i, name := range dead {
|
||||
ip := "10.0.0." + string(rune('2'+i))
|
||||
intent.EIPs[ip] = policy.EIPSpec{VPCID: "vpc-a", ExternalIP: "192.0.2.9", LogicalIP: ip, PortName: name}
|
||||
}
|
||||
|
||||
res := &passResult{}
|
||||
start := time.Now()
|
||||
r.applyEIPs(context.Background(), intent, ActualState{}, res)
|
||||
elapsed := time.Since(start)
|
||||
|
||||
if got := nat.count(); got != len(dead)+1 {
|
||||
t.Errorf("AddEIP called %d times, want %d: every spec must still get its DNAT row", got, len(dead)+1)
|
||||
}
|
||||
|
||||
seen := f.whenLiveSeen()
|
||||
if seen.IsZero() {
|
||||
t.Fatal("the live port was never probed")
|
||||
}
|
||||
if waited := seen.Sub(start); waited >= guestPortDatapathTimeout {
|
||||
t.Errorf("the live port waited %s before its first probe, which is at least one full deadline (%s): "+
|
||||
"it is queued behind an unrelated port instead of converging on its own schedule",
|
||||
waited, guestPortDatapathTimeout)
|
||||
}
|
||||
|
||||
// Serially this is one deadline per dead port; concurrently it is one
|
||||
// deadline for all of them. Two is comfortably between the two outcomes.
|
||||
if budget := 2 * guestPortDatapathTimeout; elapsed >= budget {
|
||||
t.Errorf("applyEIPs took %s for %d dead ports, want under %s: the convergence waits are still serialised",
|
||||
elapsed, len(dead), budget)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEscalateSBReset_NotRepeatedWhileConcurrent covers the hazard the
|
||||
// concurrency introduces: sb-cluster-state-reset re-syncs every chassis, so
|
||||
// simultaneous guest-port probes must not each fire one.
|
||||
func TestEscalateSBReset_NotRepeatedWhileConcurrent(t *testing.T) {
|
||||
prev := sbResetMinInterval
|
||||
sbResetMinInterval = time.Hour
|
||||
t.Cleanup(func() { sbResetMinInterval = prev })
|
||||
|
||||
f := &countingSBVerifier{}
|
||||
r := &reconciler{gwClaim: f}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for range 8 {
|
||||
wg.Go(func() { r.escalateSBReset(context.Background()) })
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
if got := f.resets(); got != 1 {
|
||||
t.Errorf("ResetSBClusterState called %d times, want 1: a cluster-wide reset must not be fired once per port", got)
|
||||
}
|
||||
}
|
||||
|
||||
// countingSBVerifier reports the SB wedge and counts resets.
|
||||
type countingSBVerifier struct {
|
||||
concurrentClaimVerifier
|
||||
|
||||
resetMu sync.Mutex
|
||||
n int
|
||||
}
|
||||
|
||||
func (c *countingSBVerifier) SBConnectionState(context.Context) (string, error) {
|
||||
return "not connected", nil
|
||||
}
|
||||
|
||||
func (c *countingSBVerifier) ResetSBClusterState(context.Context) error {
|
||||
c.resetMu.Lock()
|
||||
defer c.resetMu.Unlock()
|
||||
c.n++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *countingSBVerifier) resets() int {
|
||||
c.resetMu.Lock()
|
||||
defer c.resetMu.Unlock()
|
||||
return c.n
|
||||
}
|
||||
@@ -151,6 +151,11 @@ type reconciler struct {
|
||||
// is gone stops paying the full nudge sequence every cycle.
|
||||
portBackoffMu sync.Mutex
|
||||
portBackoff map[string]portBackoffState
|
||||
|
||||
// Serialises sb-cluster-state-reset. It is cluster-wide and re-syncs every
|
||||
// chassis, so concurrent guest-port probes must not each fire one.
|
||||
sbResetMu sync.Mutex
|
||||
sbResetLast time.Time
|
||||
}
|
||||
|
||||
// portBackoffState is one guest port's consecutive failure count and the instant
|
||||
|
||||
@@ -6,7 +6,7 @@ policy {UUID:X Priority:1100 Match:inport == "rtr-subnet-pub" && ip4.dst == 0.0.
|
||||
policy {UUID:X Priority:1200 Match:inport == "rtr-subnet-pub" && ip4.src == 10.0.1.10/32 && ip4.dst == 0.0.0.0/0 && ip4.dst != 10.0.0.0/16 Action:allow Nexthop:<nil> Nexthops:[] Options:map[] ExternalIDs:map[spinifex:output_port:gw-vpc-a spinifex:role:system-instance-egress spinifex:src_ip:10.0.1.10 spinifex:subnet:subnet-pub]}
|
||||
policy {UUID:X Priority:900 Match:inport == "rtr-subnet-priv" && ip4.dst == 0.0.0.0/0 && ip4.dst != 10.0.0.0/16 && ip4.dst != 169.254.0.0/16 Action:reroute Nexthop:PTR Nexthops:[] Options:map[] ExternalIDs:map[spinifex:output_port:gw-vpc-a spinifex:subnet:subnet-priv]}
|
||||
port {UUID:lsp:ext-port-shared Name:ext-port-shared Type:localnet Addresses:[unknown] PortSecurity:[] DHCPv4Options:<nil> Enabled:<nil> Up:<nil> ExternalIDs:map[spinifex:role:external-localnet] Options:map[network_name:external]}
|
||||
port {UUID:lsp:gw-port-vpc-a Name:gw-port-vpc-a Type:router Addresses:[router] PortSecurity:[] DHCPv4Options:<nil> Enabled:<nil> Up:<nil> ExternalIDs:map[spinifex:igw_id:igw-a spinifex:role:gateway spinifex:vpc_id:vpc-a] Options:map[router-port:gw-vpc-a]}
|
||||
port {UUID:lsp:gw-port-vpc-a Name:gw-port-vpc-a Type:router Addresses:[router] PortSecurity:[] DHCPv4Options:<nil> Enabled:<nil> Up:<nil> ExternalIDs:map[spinifex:igw_id:igw-a spinifex:role:gateway spinifex:vpc_id:vpc-a] Options:map[nat-addresses:router router-port:gw-vpc-a]}
|
||||
port {UUID:lsp:port-eni-priv Name:port-eni-priv Type: Addresses:[02:00:00:00:00:02 10.0.2.10] PortSecurity:[02:00:00:00:00:02 10.0.2.10] DHCPv4Options:PTR Enabled:<nil> Up:<nil> ExternalIDs:map[spinifex:eni_id:eni-priv spinifex:subnet_id:subnet-priv spinifex:vpc_id:vpc-a] Options:map[]}
|
||||
port {UUID:lsp:port-eni-pub Name:port-eni-pub Type: Addresses:[02:00:00:00:00:01 10.0.1.10] PortSecurity:[02:00:00:00:00:01 10.0.1.10] DHCPv4Options:PTR Enabled:<nil> Up:<nil> ExternalIDs:map[spinifex:eni_id:eni-pub spinifex:subnet_id:subnet-pub spinifex:vpc_id:vpc-a] Options:map[]}
|
||||
port {UUID:lsp:rtr-port-subnet-priv Name:rtr-port-subnet-priv Type:router Addresses:[router] PortSecurity:[] DHCPv4Options:<nil> Enabled:<nil> Up:<nil> ExternalIDs:map[spinifex:subnet_id:subnet-priv spinifex:vpc_id:vpc-a] Options:map[router-port:rtr-subnet-priv]}
|
||||
|
||||
@@ -23,6 +23,36 @@ const actionAttrKey = "rpc.method"
|
||||
// IDs — those belong in the accompanying log line.
|
||||
const leakKindAttrKey = "resource.kind"
|
||||
|
||||
// fenceOutcomeAttrKey names what a node did after losing a volume lease. The
|
||||
// FenceOutcome values below are the whole domain, and none carries a volume or
|
||||
// node identity: those belong in the log line beside the metric.
|
||||
const fenceOutcomeAttrKey = "outcome"
|
||||
|
||||
// FenceOutcome is what a node did after losing a volume lease.
|
||||
type FenceOutcome string
|
||||
|
||||
const (
|
||||
// FenceOutcomeTaken means another node held the lease, so this one gave up
|
||||
// its export. The clearest signal in the set: a volume changed hands.
|
||||
FenceOutcomeTaken FenceOutcome = "taken"
|
||||
|
||||
// FenceOutcomeExpired means the entry was gone or unreadable with no
|
||||
// successor. The node could not show it was the only writer, which is
|
||||
// fenced on the same terms.
|
||||
FenceOutcomeExpired FenceOutcome = "expired"
|
||||
|
||||
// FenceOutcomeStalled means renewal could not be confirmed for long enough
|
||||
// that the server TTL was about to admit another owner. This is the
|
||||
// self-fence, and a rising rate is a NATS problem showing up as guest
|
||||
// restarts.
|
||||
FenceOutcomeStalled FenceOutcome = "stalled"
|
||||
|
||||
// FenceOutcomeKillFailed means the export could not be torn down: nbdkit
|
||||
// did not exit. Nothing downstream is safe after this -- the volume is
|
||||
// neither fenced nor released -- so it is the one value here that is a page.
|
||||
FenceOutcomeKillFailed FenceOutcome = "kill_failed"
|
||||
)
|
||||
|
||||
var (
|
||||
instrumentsOnce sync.Once
|
||||
requestCounter metric.Int64Counter
|
||||
@@ -30,6 +60,11 @@ var (
|
||||
|
||||
leakOnce sync.Once
|
||||
leakCounter metric.Int64Counter
|
||||
|
||||
fenceOnce sync.Once
|
||||
fenceCounter metric.Int64Counter
|
||||
takeoverOnce sync.Once
|
||||
takeoverCounter metric.Int64Counter
|
||||
)
|
||||
|
||||
// requestInstruments lazily creates the shared request instruments. The
|
||||
@@ -90,3 +125,47 @@ func RecordResourceLeak(ctx context.Context, kind string) {
|
||||
attribute.NewSet(attribute.String(leakKindAttrKey, kind))))
|
||||
}
|
||||
}
|
||||
|
||||
// RecordVolumeFence counts one volume whose lease this node stopped holding
|
||||
// while an engine was open on it. outcome is one of the FenceOutcome constants.
|
||||
//
|
||||
// Any non-zero rate means guests are being stopped to protect a volume somebody
|
||||
// else now owns, which is a correctness action with an availability cost and is
|
||||
// worth paging on. FenceOutcomeKillFailed is the urgent one: the guest was not
|
||||
// stopped, so the protection did not happen.
|
||||
func RecordVolumeFence(ctx context.Context, outcome FenceOutcome) {
|
||||
fenceOnce.Do(func() {
|
||||
var err error
|
||||
fenceCounter, err = otel.Meter(meterName).Int64Counter("mulga.volume.fence",
|
||||
metric.WithDescription("Count of volume leases lost while an engine was open, by what the node did next."),
|
||||
metric.WithUnit("{volume}"))
|
||||
if err != nil {
|
||||
otel.Handle(err)
|
||||
}
|
||||
})
|
||||
if fenceCounter != nil {
|
||||
fenceCounter.Add(ctx, 1, metric.WithAttributeSet(
|
||||
attribute.NewSet(attribute.String(fenceOutcomeAttrKey, string(outcome)))))
|
||||
}
|
||||
}
|
||||
|
||||
// RecordVolumeTakeover counts one volume opened from the backend checkpoint
|
||||
// while another node held writes the backend never received.
|
||||
//
|
||||
// This is the deliberate trade the placement design makes — an instance that
|
||||
// runs with an older copy beats one that runs nowhere — so it is not an error.
|
||||
// It is the only quantity that says how often that trade is being paid.
|
||||
func RecordVolumeTakeover(ctx context.Context) {
|
||||
takeoverOnce.Do(func() {
|
||||
var err error
|
||||
takeoverCounter, err = otel.Meter(meterName).Int64Counter("mulga.volume.takeover",
|
||||
metric.WithDescription("Count of volumes opened from the backend while another node held unsealed writes."),
|
||||
metric.WithUnit("{volume}"))
|
||||
if err != nil {
|
||||
otel.Handle(err)
|
||||
}
|
||||
})
|
||||
if takeoverCounter != nil {
|
||||
takeoverCounter.Add(ctx, 1)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
package otelsetup_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/sdk/metric"
|
||||
"go.opentelemetry.io/otel/sdk/metric/metricdata"
|
||||
)
|
||||
|
||||
// collect installs an in-memory meter provider and returns a function that
|
||||
// reads what has been recorded on it so far.
|
||||
func collect(t *testing.T) func() metricdata.ResourceMetrics {
|
||||
t.Helper()
|
||||
|
||||
reader := metric.NewManualReader()
|
||||
previous := otel.GetMeterProvider()
|
||||
otel.SetMeterProvider(metric.NewMeterProvider(metric.WithReader(reader)))
|
||||
t.Cleanup(func() { otel.SetMeterProvider(previous) })
|
||||
|
||||
return func() metricdata.ResourceMetrics {
|
||||
var out metricdata.ResourceMetrics
|
||||
require.NoError(t, reader.Collect(t.Context(), &out))
|
||||
return out
|
||||
}
|
||||
}
|
||||
|
||||
// sumFor returns the total recorded on the named instrument, and whether the
|
||||
// instrument exists at all.
|
||||
func sumFor(rm metricdata.ResourceMetrics, name string) (int64, bool) {
|
||||
for _, scope := range rm.ScopeMetrics {
|
||||
for _, m := range scope.Metrics {
|
||||
if m.Name != name {
|
||||
continue
|
||||
}
|
||||
sum, ok := m.Data.(metricdata.Sum[int64])
|
||||
if !ok {
|
||||
return 0, false
|
||||
}
|
||||
var total int64
|
||||
for _, dp := range sum.DataPoints {
|
||||
total += dp.Value
|
||||
}
|
||||
return total, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// outcomesFor returns the set of outcome attribute values seen on a counter.
|
||||
// The fence's whole diagnostic value is in that attribute: it separates a
|
||||
// volume that changed hands from a node that lost contact with NATS.
|
||||
func outcomesFor(rm metricdata.ResourceMetrics, name string) []string {
|
||||
var seen []string
|
||||
for _, scope := range rm.ScopeMetrics {
|
||||
for _, m := range scope.Metrics {
|
||||
if m.Name != name {
|
||||
continue
|
||||
}
|
||||
if sum, ok := m.Data.(metricdata.Sum[int64]); ok {
|
||||
for _, dp := range sum.DataPoints {
|
||||
if v, found := dp.Attributes.Value("outcome"); found {
|
||||
seen = append(seen, v.AsString())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return seen
|
||||
}
|
||||
|
||||
// TestRecordVolumeFence_SeparatesTheOutcomes pins that the four fence outcomes
|
||||
// stay distinguishable. They call for different responses — a takeover is
|
||||
// working as designed, a kill_failed means the protection did not happen — so
|
||||
// collapsing them into one number would make the metric unactionable.
|
||||
func TestRecordVolumeFence_SeparatesTheOutcomes(t *testing.T) {
|
||||
read := collect(t)
|
||||
|
||||
otelsetup.RecordVolumeFence(t.Context(), otelsetup.FenceOutcomeTaken)
|
||||
otelsetup.RecordVolumeFence(t.Context(), otelsetup.FenceOutcomeExpired)
|
||||
otelsetup.RecordVolumeFence(t.Context(), otelsetup.FenceOutcomeStalled)
|
||||
otelsetup.RecordVolumeFence(t.Context(), otelsetup.FenceOutcomeKillFailed)
|
||||
|
||||
rm := read()
|
||||
total, ok := sumFor(rm, "mulga.volume.fence")
|
||||
require.True(t, ok, "the fence counter must be registered once a fence is recorded")
|
||||
assert.EqualValues(t, 4, total)
|
||||
assert.ElementsMatch(t,
|
||||
[]string{"taken", "expired", "stalled", "kill_failed"},
|
||||
outcomesFor(rm, "mulga.volume.fence"))
|
||||
}
|
||||
|
||||
// TestRecordVolumeTakeover_Counts covers the one quantity that says how often
|
||||
// the platform is choosing availability over the newer copy of a volume.
|
||||
func TestRecordVolumeTakeover_Counts(t *testing.T) {
|
||||
read := collect(t)
|
||||
|
||||
otelsetup.RecordVolumeTakeover(t.Context())
|
||||
otelsetup.RecordVolumeTakeover(t.Context())
|
||||
|
||||
total, ok := sumFor(read(), "mulga.volume.takeover")
|
||||
require.True(t, ok, "the takeover counter must be registered once a takeover is recorded")
|
||||
assert.EqualValues(t, 2, total)
|
||||
}
|
||||
|
||||
// TestRecordResourceLeak_CountsByKind covers the leak counter's attribute,
|
||||
// which is what makes a rising number point at a resource class rather than
|
||||
// just going up.
|
||||
func TestRecordResourceLeak_CountsByKind(t *testing.T) {
|
||||
read := collect(t)
|
||||
|
||||
otelsetup.RecordResourceLeak(t.Context(), "public_ip")
|
||||
|
||||
total, ok := sumFor(read(), "mulga.resource.leaked")
|
||||
require.True(t, ok)
|
||||
assert.EqualValues(t, 1, total)
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package viperblockd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -19,10 +20,13 @@ func TestMountVolume_AlreadyMountedReturnsExistingExport(t *testing.T) {
|
||||
|
||||
const volumeName = "vol-mountidempotent01"
|
||||
const existingURI = "nbd://127.0.0.1:10809"
|
||||
// A live pid, because the guard only hands back an export it can still see
|
||||
// running: an entry whose process is gone describes a dead export and is
|
||||
// dropped instead, which is what stops a volume becoming unstartable.
|
||||
cfg.MountedVolumes = append(cfg.MountedVolumes, MountedVolume{
|
||||
Name: volumeName,
|
||||
NBDURI: existingURI,
|
||||
PID: 4242,
|
||||
PID: os.Getpid(),
|
||||
ReadOnly: false,
|
||||
})
|
||||
|
||||
|
||||
@@ -16,8 +16,10 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
|
||||
@@ -125,13 +127,42 @@ func corroborateNbdkit(cfg *Config, disc discoveredNbdkit) bool {
|
||||
return err == nil && info.Mode()&os.ModeSocket != 0
|
||||
}
|
||||
|
||||
// defaultRecoveryAttempts is how many times recovery tries to build a
|
||||
// survivor's VB before giving up on it, and defaultRecoveryBackoff is the
|
||||
// pause between those attempts.
|
||||
//
|
||||
// Recovery runs at daemon start, when JetStream is routinely still catching up
|
||||
// and a KV claim can time out on its own default. One shot there orphans a
|
||||
// running guest's data path for the life of the process, which is a far worse
|
||||
// outcome than waiting a few seconds.
|
||||
const (
|
||||
defaultRecoveryAttempts = 4
|
||||
defaultRecoveryBackoff = 3 * time.Second
|
||||
)
|
||||
|
||||
// rebuildMountedVolume re-derives disc's NBD URI, builds its daemon-side VB
|
||||
// (cfg.buildVB, mountVolume's construction) and registers the same
|
||||
// config/owner subscriptions a fresh mount would. Never touches the process.
|
||||
func rebuildMountedVolume(ctx context.Context, cfg *Config, nc *nats.Conn, disc discoveredNbdkit) (MountedVolume, error) {
|
||||
vb, _, lease, err := cfg.buildVB(ctx, disc.Volume)
|
||||
if err != nil {
|
||||
return MountedVolume{}, fmt.Errorf("construct VB: %w", err)
|
||||
attempts, backoff := cfg.recoveryBuildPolicy()
|
||||
var vb *viperblock.VB
|
||||
var lease *volumeLease
|
||||
var err error
|
||||
for attempt := 1; ; attempt++ {
|
||||
vb, _, lease, err = cfg.buildVB(ctx, disc.Volume)
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if attempt >= attempts {
|
||||
return MountedVolume{}, fmt.Errorf("construct VB after %d attempts: %w", attempt, err)
|
||||
}
|
||||
slog.WarnContext(ctx, "recovery: could not build VB for a surviving nbdkit, retrying",
|
||||
"pid", disc.PID, "volume", disc.Volume, "attempt", attempt, "err", err)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return MountedVolume{}, fmt.Errorf("construct VB: %w", ctx.Err())
|
||||
case <-time.After(backoff):
|
||||
}
|
||||
}
|
||||
|
||||
var nbdURI string
|
||||
@@ -205,6 +236,9 @@ type reapReason string
|
||||
const (
|
||||
reapReasonUnadoptable reapReason = "unadoptable"
|
||||
reapReasonDuplicate reapReason = "duplicate"
|
||||
// reapReasonUnregistered is an export still serving a volume the registry
|
||||
// has no entry for, found when that volume is unmounted.
|
||||
reapReasonUnregistered reapReason = "unregistered"
|
||||
)
|
||||
|
||||
// reapOrphanedNbdkit decides whether disc -- a corroborated nbdkit process
|
||||
@@ -213,35 +247,36 @@ const (
|
||||
// only ever signalled once a scan of every other process on the host finds no
|
||||
// reference to its NBD endpoint. A referencing process, or a scan that could
|
||||
// not complete, both leave it running: the caller gets a loud log either way.
|
||||
func reapOrphanedNbdkit(procRoot string, disc discoveredNbdkit, reason reapReason) {
|
||||
// Reports whether disc was signalled.
|
||||
func reapOrphanedNbdkit(procRoot string, disc discoveredNbdkit, reason reapReason) bool {
|
||||
if hidden, herr := procHidepidInvisible(procRoot); hidden {
|
||||
slog.Warn("recovery: proc mount hides other users' process entries; a live referencer could be invisible to this scan, leaving unclaimed nbdkit running",
|
||||
"pid", disc.PID, "volume", disc.Volume, "reason", reason, "err", herr)
|
||||
return
|
||||
return false
|
||||
}
|
||||
|
||||
referencingPID, referenced, err := endpointReferencer(procRoot, disc)
|
||||
if err != nil {
|
||||
slog.Warn("recovery: could not scan for processes using an unclaimed nbdkit's endpoint, leaving it running",
|
||||
"pid", disc.PID, "volume", disc.Volume, "reason", reason, "err", err)
|
||||
return
|
||||
return false
|
||||
}
|
||||
if referenced {
|
||||
slog.Error("recovery: unclaimed nbdkit process is still referenced by a live process; left running deliberately, needs operator attention",
|
||||
"pid", disc.PID, "volume", disc.Volume, "reason", reason, "referencing_pid", referencingPID)
|
||||
return
|
||||
return false
|
||||
}
|
||||
|
||||
proc, err := os.FindProcess(disc.PID)
|
||||
if err != nil {
|
||||
slog.Error("recovery: failed to locate unclaimed nbdkit process to reap",
|
||||
"pid", disc.PID, "volume", disc.Volume, "reason", reason, "err", err)
|
||||
return
|
||||
return false
|
||||
}
|
||||
if err := proc.Signal(syscall.SIGTERM); err != nil {
|
||||
slog.Error("recovery: failed to signal unclaimed nbdkit process",
|
||||
"pid", disc.PID, "volume", disc.Volume, "reason", reason, "err", err)
|
||||
return
|
||||
return false
|
||||
}
|
||||
|
||||
if disc.Socket != "" {
|
||||
@@ -253,6 +288,7 @@ func reapOrphanedNbdkit(procRoot string, disc discoveredNbdkit, reason reapReaso
|
||||
|
||||
slog.Info("recovery: reaped unclaimed nbdkit process with no live referencer",
|
||||
"pid", disc.PID, "volume", disc.Volume, "reason", reason)
|
||||
return true
|
||||
}
|
||||
|
||||
// procHidepidInvisible reports whether procRoot's mount hides another uid's
|
||||
@@ -358,3 +394,30 @@ func cmdlineReferencesEndpoint(cmdline []byte, disc discoveredNbdkit) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// reapUnregisteredNbdkit reaps any nbdkit this daemon owns for volumeName that
|
||||
// is not in cfg.MountedVolumes, and reports whether it reaped one.
|
||||
//
|
||||
// The registry can be missing an entry for a live export: recovery gives up on
|
||||
// a survivor whose VB it could not build, and nothing reaps it afterwards, so
|
||||
// it outlives the guest and holds its socket and volume open. Unmount is the
|
||||
// point the volume is going away, so anything still serving it here is a leak.
|
||||
//
|
||||
// Only corroborated processes are considered -- our plugin, our base dir, that
|
||||
// exact volume -- and the reap itself still declines anything a live process
|
||||
// references, because the guest's disk may not be torn down yet.
|
||||
func reapUnregisteredNbdkit(ctx context.Context, cfg *Config, volumeName string) bool {
|
||||
procRoot := cfg.procScanRoot()
|
||||
var reaped bool
|
||||
for _, disc := range scanNbdkitProcs(procRoot) {
|
||||
if disc.Volume != volumeName || !corroborateNbdkit(cfg, disc) {
|
||||
continue
|
||||
}
|
||||
slog.WarnContext(ctx, "ebs.unmount: found an nbdkit serving this volume with no registry entry",
|
||||
"volume", volumeName, "pid", disc.PID, "socket", disc.Socket)
|
||||
if reapOrphanedNbdkit(procRoot, disc, reapReasonUnregistered) {
|
||||
reaped = true
|
||||
}
|
||||
}
|
||||
return reaped
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ package viperblockd
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
@@ -681,3 +682,168 @@ func TestRecoverMountedVolumes_UnadoptableReapedAcrossRestarts(t *testing.T) {
|
||||
_, ok = findMountedVolume(cfg, volumeName)
|
||||
assert.False(t, ok)
|
||||
}
|
||||
|
||||
// --- adoption failures must not orphan a live export ---
|
||||
|
||||
// TestRebuildMountedVolume_RetriesATransientBuildFailure pins the retry.
|
||||
// Recovery runs at daemon start, where a JetStream still catching up fails the
|
||||
// lease claim on its own 5s default. Giving up on the first error orphaned the
|
||||
// export for the life of the process, and the -efi volume -- always adopted
|
||||
// after the boot volume -- was the one that hit the window.
|
||||
func TestRebuildMountedVolume_RetriesATransientBuildFailure(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
cfg := setupTestConfig(t, natsURL)
|
||||
cfg.recoveryAttempts = 3
|
||||
cfg.recoveryBackoff = time.Millisecond
|
||||
nc := startProviderSubjects(t, cfg, natsURL)
|
||||
|
||||
var calls int
|
||||
build := fileBackedConstructVB(t)
|
||||
cfg.constructVB = func(ctx context.Context, volumeName string) (*viperblock.VB, int, error) {
|
||||
calls++
|
||||
if calls < 3 {
|
||||
return nil, 0, errors.New("context deadline exceeded")
|
||||
}
|
||||
return build(ctx, volumeName)
|
||||
}
|
||||
|
||||
disc := discoveredNbdkit{PID: 4243, Volume: "vol-transient001-efi", Socket: filepath.Join(t.TempDir(), "efi.sock")}
|
||||
mv, err := rebuildMountedVolume(context.Background(), cfg, nc, disc)
|
||||
require.NoError(t, err, "a build that succeeds on the third attempt must be adopted, not abandoned")
|
||||
assert.Equal(t, 3, calls, "every attempt up to the first success should have been made")
|
||||
assert.Equal(t, disc.Volume, mv.Name)
|
||||
}
|
||||
|
||||
// TestRebuildMountedVolume_GivesUpAfterTheLastAttempt keeps the retry bounded,
|
||||
// so a genuinely unbuildable volume still reaches the reap path.
|
||||
func TestRebuildMountedVolume_GivesUpAfterTheLastAttempt(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
cfg := setupTestConfig(t, natsURL)
|
||||
cfg.recoveryAttempts = 2
|
||||
cfg.recoveryBackoff = time.Millisecond
|
||||
nc := startProviderSubjects(t, cfg, natsURL)
|
||||
|
||||
var calls int
|
||||
cfg.constructVB = func(context.Context, string) (*viperblock.VB, int, error) {
|
||||
calls++
|
||||
return nil, 0, errors.New("permanently broken")
|
||||
}
|
||||
|
||||
disc := discoveredNbdkit{PID: 4244, Volume: "vol-permafail001", Socket: filepath.Join(t.TempDir(), "x.sock")}
|
||||
_, err := rebuildMountedVolume(context.Background(), cfg, nc, disc)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "after 2 attempts")
|
||||
assert.Equal(t, 2, calls, "the retry must be bounded by the configured attempts")
|
||||
}
|
||||
|
||||
// --- unmount reaping an export the registry never claimed ---
|
||||
|
||||
// unregisteredFixture stages the leak this exists for: a live, corroborated
|
||||
// nbdkit for volumeName that cfg.MountedVolumes has no entry for, as recovery
|
||||
// leaves behind when it cannot build the volume's VB.
|
||||
func unregisteredFixture(t *testing.T, volumeName string) (*Config, int, string) {
|
||||
t.Helper()
|
||||
pid := spawnLongRunningProcess(t)
|
||||
socket := filepath.Join(t.TempDir(), volumeName+".sock")
|
||||
listenUnix(t, socket)
|
||||
|
||||
procRoot := t.TempDir()
|
||||
writeProcFixture(t, procRoot, pid, "nbdkit", nbdkitArgv(t.TempDir(), socket, 0, volumeName))
|
||||
|
||||
cfg := &Config{procRoot: procRoot}
|
||||
cfg.BaseDir = discoveredBaseDir(t, procRoot, volumeName)
|
||||
return cfg, pid, socket
|
||||
}
|
||||
|
||||
// discoveredBaseDir reads back the base dir the fixture's argv encodes, so the
|
||||
// Config agrees with it and corroboration passes.
|
||||
func discoveredBaseDir(t *testing.T, procRoot, volumeName string) string {
|
||||
t.Helper()
|
||||
for _, disc := range scanNbdkitProcs(procRoot) {
|
||||
if disc.Volume == volumeName {
|
||||
return disc.BaseDir
|
||||
}
|
||||
}
|
||||
t.Fatalf("fixture nbdkit for %s was not discoverable", volumeName)
|
||||
return ""
|
||||
}
|
||||
|
||||
// TestReapUnregisteredNbdkit_ReapsAnExportTheRegistryNeverClaimed is the leak
|
||||
// itself: the EFI volume's nbdkit outlives its guest because recovery could
|
||||
// not adopt it, and unmount is the last point anything knows it should die.
|
||||
func TestReapUnregisteredNbdkit_ReapsAnExportTheRegistryNeverClaimed(t *testing.T) {
|
||||
const volumeName = "vol-unreg0001-efi"
|
||||
cfg, pid, socket := unregisteredFixture(t, volumeName)
|
||||
|
||||
logs := captureLogs(t)
|
||||
assert.True(t, reapUnregisteredNbdkit(context.Background(), cfg, volumeName),
|
||||
"an export serving a volume with no registry entry must be reported as reaped")
|
||||
|
||||
require.Eventually(t, func() bool { return !utils.ProcessAlive(pid) }, 2*time.Second, 20*time.Millisecond,
|
||||
"the unregistered export must be signalled and exit")
|
||||
_, err := os.Stat(socket)
|
||||
assert.True(t, os.IsNotExist(err), "the reaped export's socket file must be removed")
|
||||
assert.Contains(t, logs.String(), "no registry entry")
|
||||
assert.Contains(t, logs.String(), "reason=unregistered")
|
||||
}
|
||||
|
||||
// TestReapUnregisteredNbdkit_LeavesAnotherVolumeAlone guards the obvious way
|
||||
// this could destroy a running guest: unmounting one volume must not touch the
|
||||
// export for any other, however similar the name.
|
||||
func TestReapUnregisteredNbdkit_LeavesAnotherVolumeAlone(t *testing.T) {
|
||||
const volumeName = "vol-unreg0002-efi"
|
||||
cfg, pid, socket := unregisteredFixture(t, volumeName)
|
||||
|
||||
assert.False(t, reapUnregisteredNbdkit(context.Background(), cfg, "vol-unreg0002"),
|
||||
"the boot volume's unmount must not claim to have reaped the EFI volume's export")
|
||||
assert.True(t, utils.ProcessAlive(pid), "an export for a different volume must be left running")
|
||||
_, err := os.Stat(socket)
|
||||
assert.NoError(t, err, "an export for a different volume must keep its socket")
|
||||
}
|
||||
|
||||
// TestReapUnregisteredNbdkit_LeavesAForeignExportAlone proves the daemon only
|
||||
// reaps its own: another daemon's export for the same volume name is not ours
|
||||
// to signal, so corroboration must reject it on the base dir alone.
|
||||
func TestReapUnregisteredNbdkit_LeavesAForeignExportAlone(t *testing.T) {
|
||||
const volumeName = "vol-unreg0003-efi"
|
||||
cfg, pid, socket := unregisteredFixture(t, volumeName)
|
||||
cfg.BaseDir = filepath.Join(t.TempDir(), "someone-elses-data-dir")
|
||||
|
||||
assert.False(t, reapUnregisteredNbdkit(context.Background(), cfg, volumeName),
|
||||
"an export for a different data dir belongs to a different daemon")
|
||||
assert.True(t, utils.ProcessAlive(pid), "a foreign export must be left running")
|
||||
_, err := os.Stat(socket)
|
||||
assert.NoError(t, err, "a foreign export must keep its socket")
|
||||
}
|
||||
|
||||
// TestReapUnregisteredNbdkit_LeavesAReferencedExportAlone is the safety rule
|
||||
// carried over from recovery's reap: at unmount the guest's blockdev may not
|
||||
// be torn down yet, and killing the server underneath it strands QEMU.
|
||||
func TestReapUnregisteredNbdkit_LeavesAReferencedExportAlone(t *testing.T) {
|
||||
const volumeName = "vol-unreg0004-efi"
|
||||
cfg, pid, socket := unregisteredFixture(t, volumeName)
|
||||
writeReferencerFixture(t, cfg.procRoot, 919191, "driver=nbd,server.path="+socket)
|
||||
|
||||
logs := captureLogs(t)
|
||||
assert.False(t, reapUnregisteredNbdkit(context.Background(), cfg, volumeName),
|
||||
"an export something still references must not be reported as reaped")
|
||||
assert.True(t, utils.ProcessAlive(pid), "an export a live process references must be left running")
|
||||
assert.Contains(t, logs.String(), "still referenced by a live process")
|
||||
}
|
||||
|
||||
// TestUnmountVolume_ReapsAnUnregisteredExport is the wiring: unmount is the
|
||||
// only event that names the volume after its guest is gone, so it is where the
|
||||
// leaked export has to be caught. NotFound stays true -- there is still no
|
||||
// registry entry to seal -- and Reaped tells the caller it was not a no-op.
|
||||
func TestUnmountVolume_ReapsAnUnregisteredExport(t *testing.T) {
|
||||
const volumeName = "vol-unreg0005-efi"
|
||||
cfg, pid, _ := unregisteredFixture(t, volumeName)
|
||||
cfg.MountedVolumes = []MountedVolume{}
|
||||
|
||||
resp, err := unmountVolume(context.Background(), cfg, volumeName)
|
||||
require.Error(t, err, "the volume is genuinely not mounted here, reaped or not")
|
||||
assert.True(t, resp.NotFound, "an unregistered volume has no entry to seal")
|
||||
assert.True(t, resp.Reaped, "the export the unmount reaped must be reported")
|
||||
require.Eventually(t, func() bool { return !utils.ProcessAlive(pid) }, 2*time.Second, 20*time.Millisecond,
|
||||
"unmount must leave no nbdkit serving the volume it just removed")
|
||||
}
|
||||
|
||||
@@ -67,7 +67,18 @@ func registerProviderSubjects(cfg *Config, nc *nats.Conn) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("volume leases: %w", err)
|
||||
}
|
||||
cfg.leases = leases
|
||||
cfg.leases = cfg.bindLeaseFence(leases)
|
||||
}
|
||||
cfg.nc = nc
|
||||
|
||||
// Same reasoning as the lease store: a mount that cannot consult the dirty
|
||||
// marker cannot tell a stale cross-node start from a routine one.
|
||||
if cfg.dirty == nil {
|
||||
dirty, err := newVolumeDirty(context.Background(), nc, cfg.leaseOwner())
|
||||
if err != nil {
|
||||
return fmt.Errorf("volume dirty markers: %w", err)
|
||||
}
|
||||
cfg.dirty = dirty
|
||||
}
|
||||
|
||||
subs := []struct {
|
||||
@@ -858,6 +869,10 @@ func handleDeleteVolume(ctx context.Context, cfg *Config, nc *nats.Conn, msg *na
|
||||
}
|
||||
}
|
||||
|
||||
// The volume is gone, so a marker pinning it to this node pins nothing.
|
||||
// Left behind it would outlive every copy of the data it describes.
|
||||
cfg.purgeVolumeDirty(ctx, req.VolumeID)
|
||||
|
||||
slog.Info("ebs.provider.volume.delete: deleted", "volume", req.VolumeID)
|
||||
respondProvider(ctx, msg, ebsprovider.DeleteVolumeResponse{Versioned: ebsprovider.NewVersioned()})
|
||||
}
|
||||
@@ -1324,14 +1339,24 @@ func mountVolume(ctx context.Context, cfg *Config, nc *nats.Conn, volumeName str
|
||||
// ebs.mount subject is the route production actually takes: a retried
|
||||
// attach racing a fresh one would otherwise start two real exports.
|
||||
if mv, ok := findMountedVolume(cfg, volumeName); ok {
|
||||
// Access mode is fixed when nbdkit starts, so a remount asking for the
|
||||
// other mode cannot be answered with the running export.
|
||||
if mv.ReadOnly != readOnly {
|
||||
err := fmt.Errorf("volume %s is already mounted read_only=%t on this node", volumeName, mv.ReadOnly)
|
||||
return types.EBSMountResponse{Error: err.Error()}, err
|
||||
// An entry whose socket is gone describes an export that no longer
|
||||
// exists, and returning its URI makes the volume unstartable forever:
|
||||
// every caller waits for a socket nothing will create. Dropping it and
|
||||
// mounting again is safe because an absent socket cannot be serving.
|
||||
if stale, why := mountEntryIsStale(mv); stale {
|
||||
slog.WarnContext(ctx, "ebs.mount: dropping stale mount entry and remounting",
|
||||
"volume", volumeName, "uri", mv.NBDURI, "reason", why)
|
||||
releaseStaleMount(ctx, cfg, volumeName)
|
||||
} else {
|
||||
// Access mode is fixed when nbdkit starts, so a remount asking for
|
||||
// the other mode cannot be answered with the running export.
|
||||
if mv.ReadOnly != readOnly {
|
||||
err := fmt.Errorf("volume %s is already mounted read_only=%t on this node", volumeName, mv.ReadOnly)
|
||||
return types.EBSMountResponse{Error: err.Error()}, err
|
||||
}
|
||||
slog.InfoContext(ctx, "ebs.mount: already mounted, returning existing export", "volume", volumeName, "uri", mv.NBDURI)
|
||||
return types.EBSMountResponse{URI: mv.NBDURI, Mounted: true}, nil
|
||||
}
|
||||
slog.InfoContext(ctx, "ebs.mount: already mounted, returning existing export", "volume", volumeName, "uri", mv.NBDURI)
|
||||
return types.EBSMountResponse{URI: mv.NBDURI, Mounted: true}, nil
|
||||
}
|
||||
|
||||
// Clear any receipt left by a previous mount before anything else can
|
||||
@@ -1588,8 +1613,15 @@ func unmountVolume(ctx context.Context, cfg *Config, volumeName string) (types.E
|
||||
matched.VB.Detach()
|
||||
}
|
||||
|
||||
if err := utils.KillProcess(matched.PID); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to kill nbdkit process", "pid", matched.PID, "err", err)
|
||||
// The seal below rewrites the directory nbdkit writes, so a kill that
|
||||
// did not take makes it a concurrent write to that directory. Fail the
|
||||
// unmount instead: the entry stays mounted and a retry re-attempts
|
||||
// both, which is what a failed seal already does.
|
||||
if err := utils.ForceKillProcess(matched.PID, fenceKillTimeout); err != nil {
|
||||
slog.ErrorContext(ctx, "ebs.unmount: nbdkit did not exit, refusing to seal underneath a live writer",
|
||||
"volume", matched.Name, "pid", matched.PID, "err", err)
|
||||
ebsResponse.Error = fmt.Sprintf("kill nbdkit for %s: %v", matched.Name, err)
|
||||
return ebsResponse, errors.New(ebsResponse.Error)
|
||||
}
|
||||
|
||||
// nbdkit is now dead, so no process writes the shared BaseDir: seal
|
||||
@@ -1604,13 +1636,18 @@ func unmountVolume(ctx context.Context, cfg *Config, volumeName string) (types.E
|
||||
if err := cfg.seal(ctx, matched.Name); err != nil {
|
||||
slog.ErrorContext(ctx, "ebs.unmount: failed to seal volume to predastore", "volume", matched.Name, "err", err)
|
||||
ebsResponse.Error = fmt.Sprintf("seal volume: %v", err)
|
||||
// The un-uploaded writes are now this node's alone, and the
|
||||
// lease saying so dies with the node. Record it durably.
|
||||
cfg.markVolumeDirtyAfterFailedSeal(ctx, matched.Name, matched.leaseGeneration(), err.Error())
|
||||
} else {
|
||||
slog.InfoContext(ctx, "ebs.unmount: volume sealed to predastore", "volume", matched.Name)
|
||||
cfg.clearVolumeDirty(ctx, matched.Name, matched.leaseGeneration())
|
||||
}
|
||||
} else if consumeSealReceipt(cfg.BaseDir, matched.Name) {
|
||||
// Healthy path: the plugin sealed to predastore and removed
|
||||
// its local state itself, leaving this receipt as proof.
|
||||
slog.InfoContext(ctx, "ebs.unmount: volume already sealed by nbdkit plugin", "volume", matched.Name)
|
||||
cfg.clearVolumeDirty(ctx, matched.Name, matched.leaseGeneration())
|
||||
} else {
|
||||
// A durable volume reached unmount with no local WAL and no
|
||||
// seal receipt: this node never held its state, so there is
|
||||
@@ -1650,10 +1687,17 @@ func unmountVolume(ctx context.Context, cfg *Config, volumeName string) (types.E
|
||||
}
|
||||
|
||||
if matchIdx < 0 {
|
||||
// No registry entry, but the export can still be live: recovery gives
|
||||
// up on a survivor it could not adopt, and nothing else ever reaps it.
|
||||
// Unmount is the point the volume is going away, so an nbdkit this
|
||||
// daemon owns for this volume is a leak whatever left it behind.
|
||||
reaped := reapUnregisteredNbdkit(ctx, cfg, volumeName)
|
||||
|
||||
ebsResponse = types.EBSUnMountResponse{
|
||||
Volume: volumeName,
|
||||
Error: fmt.Sprintf("Volume %s not found", volumeName),
|
||||
NotFound: true,
|
||||
Reaped: reaped,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1773,3 +1817,64 @@ func handleUnpublishVolume(ctx context.Context, cfg *Config, msg *nats.Msg) {
|
||||
slog.Info("ebs.provider.volume.unpublish: unpublished", "volume", req.VolumeID, "node", cfg.NodeName)
|
||||
respondProvider(ctx, msg, ebsprovider.UnpublishVolumeResponse{Versioned: ebsprovider.NewVersioned()})
|
||||
}
|
||||
|
||||
// mountEntryIsStale reports whether a MountedVolume describes an export that is
|
||||
// no longer running, and why.
|
||||
//
|
||||
// Only positive evidence of death counts. The entry guards against a second
|
||||
// nbdkit for one volume, so treating a live export as stale would create the
|
||||
// double-writer hazard it exists to prevent — an absent socket or a dead pid
|
||||
// are the two conditions under which no export can be serving.
|
||||
func mountEntryIsStale(mv MountedVolume) (bool, string) {
|
||||
if mv.Socket != "" {
|
||||
if _, err := os.Stat(mv.Socket); errors.Is(err, os.ErrNotExist) {
|
||||
return true, "socket is gone"
|
||||
}
|
||||
}
|
||||
if mv.PID > 0 && !utils.ProcessAlive(mv.PID) {
|
||||
return true, "nbdkit process is gone"
|
||||
}
|
||||
return false, ""
|
||||
}
|
||||
|
||||
// releaseStaleMount drops a dead export's entry and everything it held, so the
|
||||
// remount that follows starts from the same state a clean unmount would leave.
|
||||
func releaseStaleMount(ctx context.Context, cfg *Config, volumeName string) {
|
||||
cfg.mu.Lock()
|
||||
var matched MountedVolume
|
||||
found := false
|
||||
for i, volume := range cfg.MountedVolumes {
|
||||
if volume.Name == volumeName {
|
||||
matched, found = volume, true
|
||||
cfg.MountedVolumes = append(cfg.MountedVolumes[:i], cfg.MountedVolumes[i+1:]...)
|
||||
break
|
||||
}
|
||||
}
|
||||
cfg.mu.Unlock()
|
||||
if !found {
|
||||
return
|
||||
}
|
||||
|
||||
if matched.ConfigSub != nil {
|
||||
if err := matched.ConfigSub.Unsubscribe(); err != nil {
|
||||
slog.ErrorContext(ctx, "stale mount: unsubscribe config topic", "volume", volumeName, "err", err)
|
||||
}
|
||||
}
|
||||
for _, sub := range matched.OwnerSubs {
|
||||
if sub == nil {
|
||||
continue
|
||||
}
|
||||
if err := sub.Unsubscribe(); err != nil {
|
||||
slog.ErrorContext(ctx, "stale mount: unsubscribe owner topic", "volume", volumeName, "err", err)
|
||||
}
|
||||
}
|
||||
if matched.VB != nil {
|
||||
matched.VB.Detach()
|
||||
}
|
||||
if matched.Socket != "" {
|
||||
if err := os.Remove(matched.Socket); err != nil && !os.IsNotExist(err) {
|
||||
slog.ErrorContext(ctx, "stale mount: remove socket", "volume", volumeName, "socket", matched.Socket, "err", err)
|
||||
}
|
||||
}
|
||||
cfg.releaseVolumeLease(ctx, matched.Lease)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
package viperblockd
|
||||
|
||||
//test:in-package — mountEntryIsStale and releaseStaleMount are unexported, and
|
||||
//exporting them would widen the package API for a test rather than for a caller.
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// A mount entry pointing at a socket that no longer exists describes an export
|
||||
// that is gone. Returning its URI is what makes a volume permanently
|
||||
// unstartable: every caller then waits for a socket nothing will create.
|
||||
func TestMountEntryIsStaleWhenSocketIsGone(t *testing.T) {
|
||||
socket := filepath.Join(t.TempDir(), "nbd.sock")
|
||||
|
||||
stale, why := mountEntryIsStale(MountedVolume{Name: "vol-1", Socket: socket, PID: os.Getpid()})
|
||||
assert.True(t, stale, "an absent socket means the export is gone")
|
||||
assert.Equal(t, "socket is gone", why)
|
||||
|
||||
require.NoError(t, os.WriteFile(socket, nil, 0o600))
|
||||
stale, _ = mountEntryIsStale(MountedVolume{Name: "vol-1", Socket: socket, PID: os.Getpid()})
|
||||
assert.False(t, stale, "a live export must never be treated as stale: that is the double-writer hazard")
|
||||
}
|
||||
|
||||
// The pid is the second piece of positive evidence, for an export whose socket
|
||||
// file outlived the process that owned it.
|
||||
func TestMountEntryIsStaleWhenProcessIsGone(t *testing.T) {
|
||||
socket := filepath.Join(t.TempDir(), "nbd.sock")
|
||||
require.NoError(t, os.WriteFile(socket, nil, 0o600))
|
||||
|
||||
// PID 0 is "unknown", not dead, so it must not be read as evidence.
|
||||
stale, _ := mountEntryIsStale(MountedVolume{Name: "vol-1", Socket: socket, PID: 0})
|
||||
assert.False(t, stale, "an unknown pid is not evidence of death")
|
||||
|
||||
stale, why := mountEntryIsStale(MountedVolume{Name: "vol-1", Socket: socket, PID: deadPID(t)})
|
||||
assert.True(t, stale, "a dead nbdkit means the export is gone")
|
||||
assert.Equal(t, "nbdkit process is gone", why)
|
||||
}
|
||||
|
||||
// releaseStaleMount must drop the entry so the remount behind it can proceed,
|
||||
// and must be safe to call for a volume that is not mounted at all.
|
||||
func TestReleaseStaleMountDropsTheEntry(t *testing.T) {
|
||||
cfg := &Config{MountedVolumes: []MountedVolume{{Name: "vol-1"}, {Name: "vol-2"}}}
|
||||
|
||||
releaseStaleMount(t.Context(), cfg, "vol-1")
|
||||
require.Len(t, cfg.MountedVolumes, 1)
|
||||
assert.Equal(t, "vol-2", cfg.MountedVolumes[0].Name)
|
||||
|
||||
releaseStaleMount(t.Context(), cfg, "vol-absent")
|
||||
assert.Len(t, cfg.MountedVolumes, 1, "releasing an unmounted volume must be a no-op")
|
||||
}
|
||||
|
||||
// deadPID returns a pid that has certainly exited.
|
||||
func deadPID(t *testing.T) int {
|
||||
t.Helper()
|
||||
proc, err := os.StartProcess("/bin/true", []string{"/bin/true"}, &os.ProcAttr{})
|
||||
require.NoError(t, err)
|
||||
state, err := proc.Wait()
|
||||
require.NoError(t, err)
|
||||
return state.Pid()
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
// Package vbwire is the contract between viperblockd and everything that has
|
||||
// to read what it publishes: the NATS subjects it addresses, the shapes it puts
|
||||
// on them, and the KV bucket recording which node holds a volume's only current
|
||||
// copy.
|
||||
//
|
||||
// It is a leaf on purpose. A consumer of the contract -- the daemon stopping a
|
||||
// fenced guest, an operator tool listing unsealed volumes -- should not have to
|
||||
// link the storage service to name what it is listening for.
|
||||
package vbwire
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// VolumeFencedSubject is where a fenced volume is announced so the daemon on
|
||||
// this node can stop the guest that was using it. Node-addressed: the guest is
|
||||
// local, and every other node's daemon has nothing to do about it.
|
||||
func VolumeFencedSubject(node string) string {
|
||||
if node == "" {
|
||||
return "ebs.fenced"
|
||||
}
|
||||
return "ebs." + node + ".fenced"
|
||||
}
|
||||
|
||||
// VolumeFencedEvent announces that a node tore down a volume's export because
|
||||
// the lease moved. Winner is who holds it now, for the operator reading the
|
||||
// guest's stop reason.
|
||||
type VolumeFencedEvent struct {
|
||||
Volume string `json:"volume"`
|
||||
Node string `json:"node"`
|
||||
Winner string `json:"winner"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// DirtyBucket names the volumes whose writes are not confirmed to have reached
|
||||
// the backend, so the node holding them is still known after that node is gone.
|
||||
//
|
||||
// Deliberately no TTL. The volume lease answers "who is writing this right now"
|
||||
// and expires with its holder; this answers "whose copy is ahead of the
|
||||
// backend", which has to outlive the holder to be worth anything.
|
||||
//
|
||||
// It is a placement input, not a gate. Instance start already prefers the node
|
||||
// that last ran the instance and falls back after a window, and refusing the
|
||||
// fallback here would trade a rare data-loss risk for a volume that cannot run
|
||||
// anywhere.
|
||||
const DirtyBucket = "VIPERBLOCK_VOLUME_DIRTY"
|
||||
|
||||
// DirtyRecord is what a node publishes about writes the backend may not have.
|
||||
// Reason says how it got that way, so an operator reading a takeover warning
|
||||
// does not have to correlate journals across nodes.
|
||||
//
|
||||
// Generation is the lease revision the writer held. It orders two nodes' claims
|
||||
// on one volume, which is the whole reason a returning node cannot quietly
|
||||
// overwrite or clear a marker that has moved on without it.
|
||||
type DirtyRecord struct {
|
||||
Owner string `json:"owner"`
|
||||
Generation uint64 `json:"generation"`
|
||||
Since time.Time `json:"since"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// UnsealedVolume reports a volume whose writes are not confirmed to have
|
||||
// reached the backend, and which node holds them.
|
||||
type UnsealedVolume struct {
|
||||
VolumeID string
|
||||
Owner string
|
||||
Since time.Time
|
||||
Reason string
|
||||
}
|
||||
|
||||
// BindDirtyBucket opens the dirty bucket read-only for an operator tool. It does
|
||||
// not create the bucket: a cluster where no volume has ever been opened has no
|
||||
// bucket, and reporting that as an error would read as a fault. A nil bucket
|
||||
// with a nil error is that case.
|
||||
func BindDirtyBucket(ctx context.Context, nc *nats.Conn) (jetstream.KeyValue, error) {
|
||||
js, err := jetstream.New(nc)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("jetstream: %w", err)
|
||||
}
|
||||
kv, err := js.KeyValue(ctx, DirtyBucket)
|
||||
if errors.Is(err, jetstream.ErrBucketNotFound) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("volume dirty bucket: %w", err)
|
||||
}
|
||||
return kv, nil
|
||||
}
|
||||
|
||||
// ListUnsealedVolumes reports every volume a node holds writes for that the
|
||||
// backend may not have.
|
||||
//
|
||||
// Starting one elsewhere is allowed and preferred over an instance that cannot
|
||||
// run, but it opens from the last checkpoint that did reach the backend. This
|
||||
// is the list of volumes where that trade would cost something.
|
||||
func ListUnsealedVolumes(ctx context.Context, nc *nats.Conn) ([]UnsealedVolume, error) {
|
||||
kv, err := BindDirtyBucket(ctx, nc)
|
||||
if err != nil || kv == nil {
|
||||
return nil, err
|
||||
}
|
||||
keys, err := kv.Keys(ctx)
|
||||
if errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list unsealed volumes: %w", err)
|
||||
}
|
||||
|
||||
unsealed := make([]UnsealedVolume, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
entry, err := kv.Get(ctx, key)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var record DirtyRecord
|
||||
if err := json.Unmarshal(entry.Value(), &record); err != nil {
|
||||
// Something holds writes for this volume even if the record cannot
|
||||
// be read, so report it rather than hide it.
|
||||
unsealed = append(unsealed, UnsealedVolume{VolumeID: key, Reason: "marker is unreadable"})
|
||||
continue
|
||||
}
|
||||
unsealed = append(unsealed, UnsealedVolume{
|
||||
VolumeID: key, Owner: record.Owner, Since: record.Since, Reason: record.Reason,
|
||||
})
|
||||
}
|
||||
return unsealed, nil
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package vbwire_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// TestVolumeFencedSubject_IsNodeAddressed pins the routing. A fenced guest is on
|
||||
// the node that lost the volume, so a subject without the node in it would reach
|
||||
// daemons with nothing to stop and, on a queue group, miss the one that has.
|
||||
func TestVolumeFencedSubject_IsNodeAddressed(t *testing.T) {
|
||||
assert.Equal(t, "ebs.node-a.fenced", vbwire.VolumeFencedSubject("node-a"))
|
||||
assert.Equal(t, "ebs.fenced", vbwire.VolumeFencedSubject(""),
|
||||
"a single-node daemon has no node name, and still has to hear its own fences")
|
||||
}
|
||||
@@ -110,6 +110,17 @@ type MountedVolume struct {
|
||||
// export is up and given back when this entry leaves cfg.MountedVolumes.
|
||||
Lease *volumeLease
|
||||
}
|
||||
|
||||
// leaseGeneration is the epoch this export writes under, or 0 for a mount that
|
||||
// predates the lease store. Zero never matches a real marker, so a clear that
|
||||
// falls back to it declines rather than removing somebody else's.
|
||||
func (v MountedVolume) leaseGeneration() uint64 {
|
||||
if v.Lease == nil {
|
||||
return 0
|
||||
}
|
||||
return v.Lease.generation
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
ConfigPath string
|
||||
PluginPath string
|
||||
@@ -167,6 +178,12 @@ type Config struct {
|
||||
// select the production defaults.
|
||||
loadStateRetryAttempts int
|
||||
loadStateRetryBaseDelay time.Duration
|
||||
recoveryAttempts int
|
||||
recoveryBackoff time.Duration
|
||||
|
||||
// procRoot is where process scans look. Empty means /proc; tests point it
|
||||
// at a fabricated directory.
|
||||
procRoot string
|
||||
|
||||
// sealVolume overrides how a detached volume is sealed to predastore.
|
||||
// Nil means sealVolumeVB, the real seal. Tests that need a seal to FAIL
|
||||
@@ -186,6 +203,16 @@ type Config struct {
|
||||
// refuses rather than proceeding blind.
|
||||
leases *volumeLeases
|
||||
|
||||
// dirty names the node holding a volume's only current copy after a failed
|
||||
// seal. The lease covers a live holder and expires with it; this covers a
|
||||
// dead one and does not expire.
|
||||
dirty *volumeDirty
|
||||
|
||||
// nc announces a fenced volume to this node's daemon. Held here because the
|
||||
// fence runs from a lease-renewal goroutine, which has no handler's
|
||||
// connection to borrow. Nil in tests that never fence.
|
||||
nc *nats.Conn
|
||||
|
||||
// ready is closed once every subscription is registered on the server.
|
||||
// Nil in production; tests set it to wait for the real event instead of
|
||||
// guessing at a sleep.
|
||||
@@ -239,6 +266,28 @@ func (cfg *Config) loadStateRetryPolicy() (int, time.Duration) {
|
||||
return attempts, baseDelay
|
||||
}
|
||||
|
||||
// recoveryBuildPolicy is how hard recovery tries to build a survivor's VB.
|
||||
// Zero values select the production defaults.
|
||||
func (cfg *Config) recoveryBuildPolicy() (int, time.Duration) {
|
||||
attempts := cfg.recoveryAttempts
|
||||
if attempts <= 0 {
|
||||
attempts = defaultRecoveryAttempts
|
||||
}
|
||||
backoff := cfg.recoveryBackoff
|
||||
if backoff <= 0 {
|
||||
backoff = defaultRecoveryBackoff
|
||||
}
|
||||
return attempts, backoff
|
||||
}
|
||||
|
||||
// procScanRoot is the directory process scans walk, defaulting to /proc.
|
||||
func (cfg *Config) procScanRoot() string {
|
||||
if cfg.procRoot == "" {
|
||||
return "/proc"
|
||||
}
|
||||
return cfg.procRoot
|
||||
}
|
||||
|
||||
type Service struct {
|
||||
Config *Config
|
||||
}
|
||||
@@ -657,7 +706,8 @@ func launchService(cfg *Config) (err error) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("volume leases: %w", err)
|
||||
}
|
||||
cfg.leases = leases
|
||||
cfg.leases = cfg.bindLeaseFence(leases)
|
||||
cfg.nc = nc
|
||||
|
||||
// Rebuild MountedVolumes from any nbdkit processes that survived a
|
||||
// restart before the daemon accepts a single request, so a handler can
|
||||
|
||||
@@ -94,6 +94,10 @@ func setupTestConfig(t *testing.T, natsURL string) *Config {
|
||||
Debug: true,
|
||||
MountedVolumes: []MountedVolume{},
|
||||
NodeName: "test-node",
|
||||
// One shot: the production retry exists for a JetStream still catching
|
||||
// up at daemon start, and every test that fails construction does so
|
||||
// deliberately and permanently.
|
||||
recoveryAttempts: 1,
|
||||
}
|
||||
installTestVolumeLeases(t, cfg, natsURL)
|
||||
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
package viperblockd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// errDirtyMarkerSuperseded reports a write refused because the marker names a
|
||||
// later generation. The caller holds a lease it no longer owns.
|
||||
var errDirtyMarkerSuperseded = errors.New("dirty marker belongs to a later generation")
|
||||
|
||||
// volumeDirty records and answers "which node holds this volume's only current
|
||||
// copy", from a JetStream KV bucket that outlives the node.
|
||||
type volumeDirty struct {
|
||||
kv jetstream.KeyValue
|
||||
owner string
|
||||
}
|
||||
|
||||
// newVolumeDirty binds the dirty bucket, creating it if this is the first node
|
||||
// up. owner identifies this node in the entries it writes.
|
||||
func newVolumeDirty(ctx context.Context, nc *nats.Conn, owner string) (*volumeDirty, error) {
|
||||
js, err := jetstream.New(nc)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("jetstream: %w", err)
|
||||
}
|
||||
kv, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
|
||||
Bucket: vbwire.DirtyBucket,
|
||||
Description: "volumes whose last seal failed, keyed by volume, naming the node holding the current copy",
|
||||
History: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("volume dirty bucket: %w", err)
|
||||
}
|
||||
return &volumeDirty{kv: kv, owner: owner}, nil
|
||||
}
|
||||
|
||||
// mark records that this node holds writes for volumeName at generation.
|
||||
// Conditional on the existing entry: a marker written by a later generation
|
||||
// belongs to a node that has taken the volume over, and overwriting it would
|
||||
// name this node as the current copy after it stopped being one.
|
||||
//
|
||||
// Writing at the same generation is the ordinary case — every open marks, and a
|
||||
// later failed seal on the same open refines the reason.
|
||||
func (d *volumeDirty) mark(ctx context.Context, volumeName string, generation uint64, reason string) error {
|
||||
if !volumeLeaseKeyPattern.MatchString(volumeName) {
|
||||
return fmt.Errorf("volume name %q cannot be a dirty key", volumeName)
|
||||
}
|
||||
payload, err := json.Marshal(vbwire.DirtyRecord{
|
||||
Owner: d.owner,
|
||||
Generation: generation,
|
||||
Since: time.Now().UTC(),
|
||||
Reason: reason,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal dirty record: %w", err)
|
||||
}
|
||||
|
||||
entry, err := d.kv.Get(ctx, volumeName)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
if _, err := d.kv.Create(ctx, volumeName, payload); err != nil {
|
||||
return fmt.Errorf("mark %s dirty: %w", volumeName, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("read dirty mark on %s: %w", volumeName, err)
|
||||
}
|
||||
|
||||
var current vbwire.DirtyRecord
|
||||
if err := json.Unmarshal(entry.Value(), ¤t); err == nil && current.Generation > generation {
|
||||
return fmt.Errorf("%w: %s holds generation %d, this node has %d",
|
||||
errDirtyMarkerSuperseded, current.Owner, current.Generation, generation)
|
||||
}
|
||||
|
||||
// Conditioned on the revision just read, so a concurrent takeover between
|
||||
// the read and the write loses this update rather than overwriting theirs.
|
||||
if _, err := d.kv.Update(ctx, volumeName, payload, entry.Revision()); err != nil {
|
||||
return fmt.Errorf("mark %s dirty: %w", volumeName, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// clear drops the marker after a seal that reached the backend. Absent is the
|
||||
// normal case, so a missing key is not an error.
|
||||
//
|
||||
// Only clears a marker this node wrote at this generation. A node returning
|
||||
// after a takeover still has local state to seal, and sealing it says nothing
|
||||
// about the copy that moved on — clearing the winner's marker there would erase
|
||||
// the record that the winner's own writes are unconfirmed.
|
||||
func (d *volumeDirty) clear(ctx context.Context, volumeName string, generation uint64) error {
|
||||
entry, err := d.kv.Get(ctx, volumeName)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("read dirty mark on %s: %w", volumeName, err)
|
||||
}
|
||||
|
||||
var current vbwire.DirtyRecord
|
||||
if err := json.Unmarshal(entry.Value(), ¤t); err != nil {
|
||||
return fmt.Errorf("unreadable dirty mark on %s: %w", volumeName, err)
|
||||
}
|
||||
if current.Owner != d.owner || current.Generation != generation {
|
||||
slog.WarnContext(ctx, "not clearing a dirty marker this node does not own, the volume moved on",
|
||||
"volume", volumeName, "marker_owner", current.Owner, "marker_generation", current.Generation,
|
||||
"this_owner", d.owner, "this_generation", generation)
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := d.kv.Delete(ctx, volumeName, jetstream.LastRevision(entry.Revision())); err != nil &&
|
||||
!errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return fmt.Errorf("clear dirty mark on %s: %w", volumeName, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// purge removes the marker for a volume that no longer exists. Unconditional
|
||||
// because delete is permanent: there is no copy left for an owner to hold, so a
|
||||
// marker naming one describes nothing.
|
||||
func (d *volumeDirty) purge(ctx context.Context, volumeName string) error {
|
||||
if err := d.kv.Delete(ctx, volumeName); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return fmt.Errorf("purge dirty mark on %s: %w", volumeName, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// holder returns the record for volumeName and whether one exists. An entry
|
||||
// that cannot be read is reported as absent: refusing every mount because the
|
||||
// bucket is unreadable would convert a degraded cluster into a stopped one.
|
||||
func (d *volumeDirty) holder(ctx context.Context, volumeName string) (vbwire.DirtyRecord, bool) {
|
||||
entry, err := d.kv.Get(ctx, volumeName)
|
||||
if err != nil {
|
||||
if !errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
slog.Warn("volume dirty: could not read marker", "volume", volumeName, "err", err)
|
||||
}
|
||||
return vbwire.DirtyRecord{}, false
|
||||
}
|
||||
var record vbwire.DirtyRecord
|
||||
if err := json.Unmarshal(entry.Value(), &record); err != nil {
|
||||
slog.Warn("volume dirty: unreadable marker", "volume", volumeName, "err", err)
|
||||
return vbwire.DirtyRecord{}, false
|
||||
}
|
||||
return record, true
|
||||
}
|
||||
|
||||
// markVolumeDirty records that this node holds writes for volumeName.
|
||||
//
|
||||
// The error is the caller's to honour on the open path. A volume opened without
|
||||
// a marker is one a later takeover cannot warn about, so a node that cannot
|
||||
// write the marker cannot show it is safe to write the volume.
|
||||
func (cfg *Config) markVolumeDirty(ctx context.Context, volumeName string, generation uint64, reason string) error {
|
||||
if cfg.dirty == nil {
|
||||
return nil
|
||||
}
|
||||
if err := cfg.dirty.mark(ctx, volumeName, generation, reason); err != nil {
|
||||
slog.ErrorContext(ctx, "could not record that this node holds unconfirmed writes",
|
||||
"volume", volumeName, "generation", generation, "err", err)
|
||||
return err
|
||||
}
|
||||
// Every open marks, so this is the routine case. The events worth a human's
|
||||
// attention — a failed seal, a takeover — are logged by their own callers.
|
||||
slog.DebugContext(ctx, "volume writes not yet confirmed to the backend",
|
||||
"volume", volumeName, "owner", cfg.dirty.owner, "generation", generation, "reason", reason)
|
||||
return nil
|
||||
}
|
||||
|
||||
// markVolumeDirtyAfterFailedSeal records a seal that did not reach the backend.
|
||||
// Best effort, unlike the open path: the seal has already failed and the
|
||||
// caller's own error is the one that matters.
|
||||
func (cfg *Config) markVolumeDirtyAfterFailedSeal(ctx context.Context, volumeName string, generation uint64, reason string) {
|
||||
if err := cfg.markVolumeDirty(ctx, volumeName, generation, reason); err != nil {
|
||||
slog.ErrorContext(ctx, "could not record the failed seal, so a start elsewhere will not warn",
|
||||
"volume", volumeName, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// clearVolumeDirty drops the marker after a successful seal, if this node still
|
||||
// owns it at this generation.
|
||||
func (cfg *Config) clearVolumeDirty(ctx context.Context, volumeName string, generation uint64) {
|
||||
if cfg.dirty == nil {
|
||||
return
|
||||
}
|
||||
if err := cfg.dirty.clear(ctx, volumeName, generation); err != nil {
|
||||
slog.ErrorContext(ctx, "could not clear the dirty mark, so later starts stay marked as unconfirmed",
|
||||
"volume", volumeName, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// purgeVolumeDirty drops the marker for a deleted volume.
|
||||
func (cfg *Config) purgeVolumeDirty(ctx context.Context, volumeName string) {
|
||||
if cfg.dirty == nil {
|
||||
return
|
||||
}
|
||||
if err := cfg.dirty.purge(ctx, volumeName); err != nil {
|
||||
slog.ErrorContext(ctx, "could not remove the dirty mark for a deleted volume",
|
||||
"volume", volumeName, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// reportVolumeTakeover records that this node is opening a volume another node
|
||||
// holds unsealed writes for, and takes ownership of the marker.
|
||||
//
|
||||
// It never refuses. Instance start forwards to the node that last ran the
|
||||
// instance and only falls back here once that node has failed its window, so by
|
||||
// the time this runs the alternative is not "wait for the better copy", it is
|
||||
// "the instance runs nowhere". The warning is the deliverable: this is the one
|
||||
// moment a human can be told which writes are about to be left behind.
|
||||
// It reports whether it took the marker over, so the caller does not then
|
||||
// overwrite the detail with a generic one.
|
||||
func (cfg *Config) reportVolumeTakeover(ctx context.Context, volumeName string, generation uint64) (bool, error) {
|
||||
if cfg.dirty == nil {
|
||||
return false, nil
|
||||
}
|
||||
record, ok := cfg.dirty.holder(ctx, volumeName)
|
||||
if !ok || record.Owner == "" || record.Owner == cfg.dirty.owner {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
slog.WarnContext(ctx, "opening a volume whose writes were last held by another node, from the backend checkpoint instead",
|
||||
"volume", volumeName, "previous_owner", record.Owner,
|
||||
"unsealed_since", record.Since.Format(time.RFC3339), "previous_reason", record.Reason)
|
||||
otelsetup.RecordVolumeTakeover(ctx)
|
||||
|
||||
// Take the marker over. The previous owner's copy is now behind this one,
|
||||
// so if that node returns it must not be treated as the better source.
|
||||
reason := fmt.Sprintf("took over from %s, which held unsealed writes since %s (%s)",
|
||||
record.Owner, record.Since.Format(time.RFC3339), record.Reason)
|
||||
return true, cfg.markVolumeDirty(ctx, volumeName, generation, reason)
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
package viperblockd
|
||||
|
||||
//test:in-package — the dirty marker, its bucket and acquireVolumeLease are all
|
||||
//unexported, and the takeover they implement has no exported surface.
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// newTestDirty binds a dirty-marker store for owner against natsURL.
|
||||
func newTestDirty(t *testing.T, natsURL, owner string) *volumeDirty {
|
||||
t.Helper()
|
||||
|
||||
nc, err := nats.Connect(natsURL)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(nc.Close)
|
||||
|
||||
dirty, err := newVolumeDirty(t.Context(), nc, owner)
|
||||
require.NoError(t, err)
|
||||
return dirty
|
||||
}
|
||||
|
||||
// TestVolumeDirty_MarkedOnOpenNotOnSealFailure is the property the marker is
|
||||
// built for. A node killed mid-write never reaches its seal, so a marker written
|
||||
// only when a seal fails is absent in exactly the case it exists to describe.
|
||||
func TestVolumeDirty_MarkedOnOpenNotOnSealFailure(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtymarkedonopen"
|
||||
|
||||
cfg := &Config{
|
||||
leases: newTestLeases(t, natsURL, "node-a"),
|
||||
dirty: newTestDirty(t, natsURL, "node-a"),
|
||||
}
|
||||
|
||||
_, err := cfg.acquireVolumeLease(t.Context(), volumeName)
|
||||
require.NoError(t, err)
|
||||
|
||||
record, ok := cfg.dirty.holder(t.Context(), volumeName)
|
||||
require.True(t, ok, "opening a volume must record that its writes are not yet on the backend")
|
||||
assert.Equal(t, "node-a", record.Owner)
|
||||
}
|
||||
|
||||
// TestVolumeDirty_DoesNotRefuseAnotherNode locks the availability half. Instance
|
||||
// start forwards to the node that last ran the instance and only falls back once
|
||||
// that node has failed its window, so refusing here would leave the instance
|
||||
// unable to run anywhere rather than running with a slightly older copy.
|
||||
func TestVolumeDirty_DoesNotRefuseAnotherNode(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtytakeoverallowed"
|
||||
|
||||
gone := newTestDirty(t, natsURL, "node-a")
|
||||
require.NoError(t, gone.mark(t.Context(), volumeName, 1, "seal volume: predastore unreachable"))
|
||||
|
||||
survivor := &Config{
|
||||
leases: newTestLeases(t, natsURL, "node-b"),
|
||||
dirty: newTestDirty(t, natsURL, "node-b"),
|
||||
}
|
||||
|
||||
lease, err := survivor.acquireVolumeLease(t.Context(), volumeName)
|
||||
require.NoError(t, err, "a volume must not be pinned to a node that is gone")
|
||||
require.NotNil(t, lease)
|
||||
}
|
||||
|
||||
// TestVolumeDirty_TakeoverNamesThePreviousHolder pins the record left behind. A
|
||||
// takeover silently loses whatever node-a never sealed, so the marker has to
|
||||
// carry who held it and why for the operator who reads it afterwards.
|
||||
func TestVolumeDirty_TakeoverNamesThePreviousHolder(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtytakeoverrecorded"
|
||||
|
||||
gone := newTestDirty(t, natsURL, "node-a")
|
||||
require.NoError(t, gone.mark(t.Context(), volumeName, 1, "seal volume: predastore unreachable"))
|
||||
|
||||
survivor := &Config{
|
||||
leases: newTestLeases(t, natsURL, "node-b"),
|
||||
dirty: newTestDirty(t, natsURL, "node-b"),
|
||||
}
|
||||
_, err := survivor.acquireVolumeLease(t.Context(), volumeName)
|
||||
require.NoError(t, err)
|
||||
|
||||
record, ok := survivor.dirty.holder(t.Context(), volumeName)
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, "node-b", record.Owner, "the node now holding the writes owns the marker")
|
||||
assert.Contains(t, record.Reason, "node-a", "the takeover must name the node whose writes were left behind")
|
||||
assert.Contains(t, record.Reason, "predastore unreachable",
|
||||
"why the previous holder could not seal is the operator's only lead")
|
||||
}
|
||||
|
||||
// TestVolumeDirty_OwnerReopeningIsNotATakeover guards against a node warning
|
||||
// about itself. Reopening a volume this node already holds is the ordinary
|
||||
// restart path, not a lost-writes event.
|
||||
func TestVolumeDirty_OwnerReopeningIsNotATakeover(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtyownerreopens"
|
||||
|
||||
cfg := &Config{
|
||||
leases: newTestLeases(t, natsURL, "node-a"),
|
||||
dirty: newTestDirty(t, natsURL, "node-a"),
|
||||
}
|
||||
require.NoError(t, cfg.dirty.mark(t.Context(), volumeName, 1, "seal volume: boom"))
|
||||
|
||||
_, err := cfg.acquireVolumeLease(t.Context(), volumeName)
|
||||
require.NoError(t, err)
|
||||
|
||||
record, ok := cfg.dirty.holder(t.Context(), volumeName)
|
||||
require.True(t, ok)
|
||||
assert.NotContains(t, record.Reason, "took over",
|
||||
"a node reopening its own volume has taken over from nobody")
|
||||
}
|
||||
|
||||
// TestVolumeDirty_ClearedAfterASuccessfulSeal pins that the marker is not a
|
||||
// one-way door: once the writes reach the backend the volume is clean again and
|
||||
// carries no placement preference.
|
||||
func TestVolumeDirty_ClearedAfterASuccessfulSeal(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtyclearsonseal"
|
||||
|
||||
owner := &Config{dirty: newTestDirty(t, natsURL, "node-a")}
|
||||
require.NoError(t, owner.dirty.mark(t.Context(), volumeName, 7, "seal volume: boom"))
|
||||
|
||||
owner.clearVolumeDirty(t.Context(), volumeName, 7)
|
||||
|
||||
_, ok := owner.dirty.holder(t.Context(), volumeName)
|
||||
assert.False(t, ok, "a sealed volume must leave no marker behind")
|
||||
}
|
||||
|
||||
// TestVolumeDirty_ClearIsIdempotent covers the ordinary path, where every
|
||||
// successful unmount clears a marker that may already be gone.
|
||||
func TestVolumeDirty_ClearIsIdempotent(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
dirty := newTestDirty(t, natsURL, "node-a")
|
||||
require.NoError(t, dirty.clear(t.Context(), "vol-dirtynevermarked", 1),
|
||||
"clearing an unmarked volume is the normal case, not an error")
|
||||
}
|
||||
|
||||
// TestVolumeDirty_ClearDeclinesAMarkerAnotherNodeOwns is the safety the
|
||||
// generation buys. A node that comes back after a takeover still has local
|
||||
// state to seal, and its seal says nothing about the copy that moved on —
|
||||
// clearing there would erase the record that the winner's writes are unsealed.
|
||||
func TestVolumeDirty_ClearDeclinesAMarkerAnotherNodeOwns(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtyclearnotours"
|
||||
|
||||
winner := newTestDirty(t, natsURL, "node-b")
|
||||
require.NoError(t, winner.mark(t.Context(), volumeName, 9, "took over from node-a"))
|
||||
|
||||
returned := &Config{dirty: newTestDirty(t, natsURL, "node-a")}
|
||||
returned.clearVolumeDirty(t.Context(), volumeName, 4)
|
||||
|
||||
record, ok := winner.holder(t.Context(), volumeName)
|
||||
require.True(t, ok, "a returning node must not clear the marker that moved past it")
|
||||
assert.Equal(t, "node-b", record.Owner)
|
||||
assert.EqualValues(t, 9, record.Generation)
|
||||
}
|
||||
|
||||
// TestVolumeDirty_ClearDeclinesAnOlderGenerationOfTheSameNode covers the same
|
||||
// node reopening a volume it already held. The stale export's clear must not
|
||||
// remove the marker the newer one wrote.
|
||||
func TestVolumeDirty_ClearDeclinesAnOlderGenerationOfTheSameNode(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtyclearoldgen"
|
||||
|
||||
owner := &Config{dirty: newTestDirty(t, natsURL, "node-a")}
|
||||
require.NoError(t, owner.dirty.mark(t.Context(), volumeName, 12, "reopened"))
|
||||
|
||||
owner.clearVolumeDirty(t.Context(), volumeName, 5)
|
||||
|
||||
record, ok := owner.dirty.holder(t.Context(), volumeName)
|
||||
require.True(t, ok, "an older export clearing would leave the live one unrecorded")
|
||||
assert.EqualValues(t, 12, record.Generation)
|
||||
}
|
||||
|
||||
// TestVolumeDirty_MarkRefusesToOverwriteALaterGeneration stops a returning node
|
||||
// renaming itself as the current copy. Placement reads this marker, so an
|
||||
// overwrite here would point a later takeover at the stale node.
|
||||
func TestVolumeDirty_MarkRefusesToOverwriteALaterGeneration(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtymarkstale"
|
||||
|
||||
winner := newTestDirty(t, natsURL, "node-b")
|
||||
require.NoError(t, winner.mark(t.Context(), volumeName, 9, "took over from node-a"))
|
||||
|
||||
stale := newTestDirty(t, natsURL, "node-a")
|
||||
err := stale.mark(t.Context(), volumeName, 4, "seal volume: boom")
|
||||
require.ErrorIs(t, err, errDirtyMarkerSuperseded)
|
||||
|
||||
record, ok := winner.holder(t.Context(), volumeName)
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, "node-b", record.Owner, "the later generation still owns the marker")
|
||||
}
|
||||
|
||||
// TestVolumeDirty_OpenFailsWhenTheMarkerCannotBeWritten pins the ordering the
|
||||
// marker depends on. A volume opened without one is a volume whose writes
|
||||
// nothing records, so a later takeover could not tell they existed — and the
|
||||
// lease has to go back, or the volume is stranded on a node not writing it.
|
||||
func TestVolumeDirty_OpenFailsWhenTheMarkerCannotBeWritten(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-dirtymarkunwritable"
|
||||
|
||||
// A generation no lease in a fresh bucket can reach, so this node's own
|
||||
// mark is refused as stale.
|
||||
blocker := newTestDirty(t, natsURL, "node-a")
|
||||
require.NoError(t, blocker.mark(t.Context(), volumeName, 1<<40, "held by a later generation"))
|
||||
|
||||
cfg := &Config{
|
||||
leases: newTestLeases(t, natsURL, "node-a"),
|
||||
dirty: newTestDirty(t, natsURL, "node-a"),
|
||||
}
|
||||
|
||||
_, err := cfg.acquireVolumeLease(t.Context(), volumeName)
|
||||
require.Error(t, err, "a volume whose writes cannot be recorded must not be opened")
|
||||
|
||||
owner, held := cfg.leases.currentOwner(t.Context(), volumeName)
|
||||
assert.False(t, held && owner == "node-a",
|
||||
"the lease has to be released, or the volume is locked to a node that is not writing it")
|
||||
}
|
||||
|
||||
// TestVolumeDirty_NoStoreDoesNotBlockMounts pins the degraded case. A daemon
|
||||
// with no marker store still has the lease for live exclusion, and refusing
|
||||
// every mount because the bucket is missing is a worse failure than the one
|
||||
// being described.
|
||||
func TestVolumeDirty_NoStoreDoesNotBlockMounts(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
cfg := &Config{leases: newTestLeases(t, natsURL, "node-b")}
|
||||
|
||||
lease, err := cfg.acquireVolumeLease(t.Context(), "vol-dirtynostore")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, lease)
|
||||
}
|
||||
|
||||
// TestVolumeDirty_ListReportsTheHolder covers the operator surface behind
|
||||
// 'spx admin volumes unsealed', which is the only way to see a volume whose
|
||||
// writes are stranded on a node that is not coming back.
|
||||
func TestVolumeDirty_ListReportsTheHolder(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
nc, err := nats.Connect(natsURL)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(nc.Close)
|
||||
|
||||
dirty := newTestDirty(t, natsURL, "node-a")
|
||||
require.NoError(t, dirty.mark(t.Context(), "vol-dirtylisted", 1, "seal volume: predastore unreachable"))
|
||||
|
||||
unsealed, err := vbwire.ListUnsealedVolumes(t.Context(), nc)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, unsealed, 1)
|
||||
assert.Equal(t, "vol-dirtylisted", unsealed[0].VolumeID)
|
||||
assert.Equal(t, "node-a", unsealed[0].Owner)
|
||||
assert.Contains(t, unsealed[0].Reason, "predastore unreachable")
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
package viperblockd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// The fence is what makes the volume lease more than a request. Acquiring it is
|
||||
// checked once, at mount; nothing rechecks it afterwards, so a node that loses
|
||||
// its lease while an engine is open goes on writing into a volume another node
|
||||
// now owns. Both copies then advance, and whichever reaches the backend last
|
||||
// wins by arrival order rather than by ownership.
|
||||
//
|
||||
// Stopping the loser locally is the cheap half of that problem. It does not
|
||||
// make a stale write impossible — only an epoch the backend enforces does that
|
||||
// — but it closes the window from the lease TTL down to one renewal interval,
|
||||
// and it costs nothing on the write path.
|
||||
|
||||
// fenceKillTimeout bounds the wait for nbdkit to exit after SIGKILL. Generous
|
||||
// for an uninterruptible sleep in the storage path; short enough that a fence
|
||||
// that will never complete is reported rather than waited on.
|
||||
const fenceKillTimeout = 30 * time.Second
|
||||
|
||||
// onVolumeLeaseLost fences a volume this node can no longer show it owns.
|
||||
// Called from the renewal goroutine's failure path, on its own goroutine:
|
||||
// fencing releases the lease, and releasing waits for that goroutine to exit.
|
||||
//
|
||||
// It never tries to reclaim. Reclaiming looks attractive — an entry that merely
|
||||
// aged out under JetStream pressure has no new holder, and stopping a healthy
|
||||
// guest for that is availability spent on nothing — but the two errors are not
|
||||
// symmetric. A false fence costs one guest a restart. A false reclaim leaves two
|
||||
// nodes writing one volume. Reclaiming correctly needs a state transition that
|
||||
// atomically replaces the lost in-memory lease and starts a new renewal loop,
|
||||
// and that is an optimisation, not a safety property.
|
||||
//
|
||||
// The holder is read only to say who won, which is what the operator needs and
|
||||
// not something the decision turns on.
|
||||
func (cfg *Config) onVolumeLeaseLost(ctx context.Context, volumeName string, kind leaseLossKind) {
|
||||
if cfg.leases == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Read only to name the winner for the operator. No holder, or one this node
|
||||
// cannot read, still fences: it cannot show it is the only writer, and
|
||||
// continuing to write is the one option that can corrupt.
|
||||
winner := "unknown"
|
||||
if owner, held := cfg.leases.currentOwner(ctx, volumeName); held && owner != cfg.leases.owner {
|
||||
winner = owner
|
||||
}
|
||||
|
||||
outcome := otelsetup.FenceOutcomeExpired
|
||||
switch {
|
||||
case kind == leaseLostStalled:
|
||||
outcome = otelsetup.FenceOutcomeStalled
|
||||
case winner != "unknown":
|
||||
outcome = otelsetup.FenceOutcomeTaken
|
||||
}
|
||||
cfg.fenceVolume(ctx, volumeName, winner, outcome)
|
||||
}
|
||||
|
||||
// fenceVolume tears this node's export down because the volume belongs to
|
||||
// somebody else now. The guest loses its disk immediately, which is the point:
|
||||
// an export left up is a second writer.
|
||||
//
|
||||
// Deliberately does not seal. This node's copy is the stale one, and sealing it
|
||||
// would publish an older state over the winner's. For the same reason the dirty
|
||||
// marker is left alone — the winner has taken it over and it now names them.
|
||||
func (cfg *Config) fenceVolume(ctx context.Context, volumeName, winner string, outcome otelsetup.FenceOutcome) {
|
||||
cfg.mu.Lock()
|
||||
var matched MountedVolume
|
||||
found := false
|
||||
for _, volume := range cfg.MountedVolumes {
|
||||
if volume.Name == volumeName {
|
||||
matched, found = volume, true
|
||||
break
|
||||
}
|
||||
}
|
||||
cfg.mu.Unlock()
|
||||
|
||||
if !found {
|
||||
// The export went away on its own between losing the lease and getting
|
||||
// here, which is the ordinary unmount racing this path. Nothing to do.
|
||||
slog.InfoContext(ctx, "volume lease lost after the export had already gone", "volume", volumeName)
|
||||
return
|
||||
}
|
||||
|
||||
slog.ErrorContext(ctx, "fencing volume: the lease moved while this node had it open, so the guest is losing its disk",
|
||||
"volume", volumeName, "previous_owner", cfg.leases.owner, "winner", winner,
|
||||
"pid", matched.PID)
|
||||
|
||||
if matched.ConfigSub != nil {
|
||||
if err := matched.ConfigSub.Unsubscribe(); err != nil {
|
||||
slog.ErrorContext(ctx, "fence: unsubscribe config topic", "volume", volumeName, "err", err)
|
||||
}
|
||||
}
|
||||
unsubscribeOwnerSubjects(matched.Name, matched.OwnerSubs)
|
||||
|
||||
// Detach before the kill so the state-tracking VB stops its background
|
||||
// flushes. Killing nbdkit stops the data path; this stops the metadata one.
|
||||
if matched.VB != nil {
|
||||
matched.VB.Detach()
|
||||
}
|
||||
|
||||
// Everything below assumes the writer is gone, so nothing below may run
|
||||
// until it is observed gone. A fence that has not killed nbdkit has fenced
|
||||
// nothing, and reporting it as done is the failure that lets a second
|
||||
// writer keep going while the cluster believes it stopped.
|
||||
if err := utils.ForceKillProcess(matched.PID, fenceKillTimeout); err != nil {
|
||||
slog.ErrorContext(ctx, "fence FAILED: nbdkit did not exit, so this node is still a writer on a volume it does not own",
|
||||
"volume", volumeName, "pid", matched.PID, "winner", winner, "err", err)
|
||||
otelsetup.RecordVolumeFence(ctx, otelsetup.FenceOutcomeKillFailed)
|
||||
return
|
||||
}
|
||||
|
||||
cfg.mu.Lock()
|
||||
for i, volume := range cfg.MountedVolumes {
|
||||
if volume.Name == matched.Name {
|
||||
cfg.MountedVolumes = append(cfg.MountedVolumes[:i], cfg.MountedVolumes[i+1:]...)
|
||||
break
|
||||
}
|
||||
}
|
||||
cfg.mu.Unlock()
|
||||
|
||||
if matched.Socket != "" {
|
||||
if err := os.Remove(matched.Socket); err != nil && !os.IsNotExist(err) {
|
||||
slog.ErrorContext(ctx, "fence: could not remove nbd socket", "socket", matched.Socket, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Stops the renewal goroutine and drops the local entry. The lease was
|
||||
// already flagged lost, so this cannot delete the winner's key.
|
||||
cfg.releaseVolumeLease(ctx, matched.Lease)
|
||||
|
||||
otelsetup.RecordVolumeFence(ctx, outcome)
|
||||
cfg.publishVolumeFenced(ctx, volumeName, winner)
|
||||
}
|
||||
|
||||
// publishVolumeFenced tells the local daemon a guest is now running against a
|
||||
// disk that is gone. Best effort: the fence has already happened, and a guest
|
||||
// left running with dead I/O is a worse outcome to report than to cause.
|
||||
func (cfg *Config) publishVolumeFenced(ctx context.Context, volumeName, winner string) {
|
||||
if cfg.nc == nil {
|
||||
return
|
||||
}
|
||||
event := vbwire.VolumeFencedEvent{
|
||||
Volume: volumeName,
|
||||
Node: cfg.leaseOwner(),
|
||||
Winner: winner,
|
||||
Reason: fmt.Sprintf("volume lease moved to %s while this node had the volume open", winner),
|
||||
}
|
||||
payload, err := json.Marshal(event)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "fence: marshal fenced event", "volume", volumeName, "err", err)
|
||||
return
|
||||
}
|
||||
subject := vbwire.VolumeFencedSubject(cfg.NodeName)
|
||||
if err := cfg.nc.Publish(subject, payload); err != nil {
|
||||
slog.ErrorContext(ctx, "fence: could not announce the fenced volume, the guest will be left with dead I/O",
|
||||
"volume", volumeName, "subject", subject, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// currentOwner reports who holds volumeName's lease now, and whether anyone
|
||||
// does. An unreadable entry is reported as unheld: the caller fences on that,
|
||||
// which is the safe reading of "cannot tell".
|
||||
func (l *volumeLeases) currentOwner(ctx context.Context, volumeName string) (string, bool) {
|
||||
entry, err := l.kv.Get(ctx, volumeName)
|
||||
if err != nil {
|
||||
if !errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
slog.WarnContext(ctx, "volume lease: could not read the holder after losing it",
|
||||
"volume", volumeName, "err", err)
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
var record volumeLeaseRecord
|
||||
if err := json.Unmarshal(entry.Value(), &record); err != nil {
|
||||
slog.WarnContext(ctx, "volume lease: unreadable holder record", "volume", volumeName, "err", err)
|
||||
return "", false
|
||||
}
|
||||
return record.Owner, record.Owner != ""
|
||||
}
|
||||
|
||||
// bindLeaseFence points a lease store's loss callback at this Config, so a
|
||||
// renewal that discovers the lease has moved reaches the fence.
|
||||
func (cfg *Config) bindLeaseFence(leases *volumeLeases) *volumeLeases {
|
||||
if leases != nil {
|
||||
leases.onLost = cfg.onVolumeLeaseLost
|
||||
}
|
||||
return leases
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
package viperblockd
|
||||
|
||||
//test:in-package — the fence, the lease store and MountedVolumes are all
|
||||
//unexported, and the decision the fence makes has no exported surface.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// stubExport starts a real process to stand in for nbdkit. The fence must
|
||||
// observe the writer gone before it does anything else, so a fixture with a
|
||||
// fake PID would exercise the failure path and prove nothing about the kill.
|
||||
//
|
||||
// Reaped on its own goroutine as the mount path does, because kill(pid,0)
|
||||
// succeeds against a zombie: an unreaped child never looks like it exited.
|
||||
func stubExport(t *testing.T) int {
|
||||
t.Helper()
|
||||
|
||||
cmd := exec.Command("sleep", "300")
|
||||
require.NoError(t, cmd.Start())
|
||||
reaped := make(chan struct{})
|
||||
go func() { defer close(reaped); _ = cmd.Wait() }()
|
||||
t.Cleanup(func() {
|
||||
_ = cmd.Process.Kill()
|
||||
<-reaped
|
||||
})
|
||||
return cmd.Process.Pid
|
||||
}
|
||||
|
||||
// fencedConfig builds a Config owning volumeName with one mounted entry, as a
|
||||
// node that has an export up would have.
|
||||
func fencedConfig(t *testing.T, natsURL, owner, volumeName string) *Config {
|
||||
t.Helper()
|
||||
|
||||
cfg := &Config{
|
||||
NodeName: owner,
|
||||
leases: newTestLeases(t, natsURL, owner),
|
||||
dirty: newTestDirty(t, natsURL, owner),
|
||||
}
|
||||
cfg.leases = cfg.bindLeaseFence(cfg.leases)
|
||||
|
||||
lease, err := cfg.acquireVolumeLease(t.Context(), volumeName)
|
||||
require.NoError(t, err)
|
||||
cfg.MountedVolumes = []MountedVolume{{Name: volumeName, Lease: lease, PID: stubExport(t)}}
|
||||
|
||||
// Stop the renewal goroutine before the embedded server goes away, or it
|
||||
// spends the TTL surrendering a lease no test is looking at any more.
|
||||
t.Cleanup(func() { cfg.releaseVolumeLease(context.Background(), lease) })
|
||||
return cfg
|
||||
}
|
||||
|
||||
// TestVolumeFence_TearsDownWhenAnotherNodeHoldsTheLease is the property the
|
||||
// fence exists for. A node that has lost its lease is a second writer, and the
|
||||
// only way to stop being one without a backend that can refuse is to give up
|
||||
// the export.
|
||||
func TestVolumeFence_TearsDownWhenAnotherNodeHoldsTheLease(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-fencelost"
|
||||
|
||||
cfg := fencedConfig(t, natsURL, "node-a", volumeName)
|
||||
pid := cfg.MountedVolumes[0].PID
|
||||
|
||||
// node-b takes the volume, which is what the renewal would discover.
|
||||
winner := newTestLeases(t, natsURL, "node-b")
|
||||
require.NoError(t, winner.kv.Delete(t.Context(), volumeName))
|
||||
_, err := winner.acquire(t.Context(), volumeName)
|
||||
require.NoError(t, err)
|
||||
|
||||
cfg.onVolumeLeaseLost(t.Context(), volumeName, leaseLostToPeer)
|
||||
|
||||
cfg.mu.Lock()
|
||||
mounted := len(cfg.MountedVolumes)
|
||||
cfg.mu.Unlock()
|
||||
assert.Zero(t, mounted, "a fenced volume must not be left exported: the export is the second writer")
|
||||
assert.False(t, utils.ProcessAlive(pid), "the export process has to be gone, not merely forgotten")
|
||||
}
|
||||
|
||||
// TestVolumeFence_FencesAnEntryNobodyHolds covers the lapsed case. An entry that
|
||||
// aged out with no successor is not proof this node is safe to keep writing —
|
||||
// it is proof this node cannot tell. Stopping one guest is the cheaper error
|
||||
// than leaving two nodes writing one volume.
|
||||
func TestVolumeFence_FencesAnEntryNobodyHolds(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-fencelapsed"
|
||||
|
||||
cfg := fencedConfig(t, natsURL, "node-a", volumeName)
|
||||
require.NoError(t, cfg.leases.kv.Delete(t.Context(), volumeName))
|
||||
|
||||
cfg.onVolumeLeaseLost(t.Context(), volumeName, leaseLostToPeer)
|
||||
|
||||
cfg.mu.Lock()
|
||||
mounted := len(cfg.MountedVolumes)
|
||||
cfg.mu.Unlock()
|
||||
assert.Zero(t, mounted, "a lease this node cannot prove it holds must not stay exported")
|
||||
}
|
||||
|
||||
// TestVolumeFence_KillFailureLeavesTheVolumeMounted pins the ordering the fence
|
||||
// depends on. Releasing the lease or announcing the fence after a kill that did
|
||||
// not take would tell the cluster a writer stopped that is still running, which
|
||||
// is worse than not fencing at all.
|
||||
func TestVolumeFence_KillFailureLeavesTheVolumeMounted(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-fencekillfails"
|
||||
|
||||
cfg := fencedConfig(t, natsURL, "node-a", volumeName)
|
||||
cfg.MountedVolumes[0].PID = 0 // rejected by ForceKillProcess, so the kill cannot succeed
|
||||
|
||||
cfg.fenceVolume(t.Context(), volumeName, "node-b", "taken")
|
||||
|
||||
cfg.mu.Lock()
|
||||
mounted := len(cfg.MountedVolumes)
|
||||
cfg.mu.Unlock()
|
||||
require.Equal(t, 1, mounted, "a fence that could not stop the writer must not report the volume released")
|
||||
|
||||
owner, held := cfg.leases.currentOwner(t.Context(), volumeName)
|
||||
require.True(t, held, "the lease must not be released while this node is still exporting")
|
||||
assert.Equal(t, "node-a", owner)
|
||||
}
|
||||
|
||||
// TestVolumeFence_DoesNotSealOnTheWayOut pins the one thing the fence must
|
||||
// never do. This node's copy is the stale one, so sealing it would publish an
|
||||
// older state over the winner's — the corruption the fence exists to prevent.
|
||||
func TestVolumeFence_DoesNotSealOnTheWayOut(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-fencenoseal"
|
||||
|
||||
sealed := false
|
||||
cfg := fencedConfig(t, natsURL, "node-a", volumeName)
|
||||
cfg.sealVolume = func(context.Context, string) error { sealed = true; return nil }
|
||||
|
||||
cfg.fenceVolume(t.Context(), volumeName, "node-b", "taken")
|
||||
|
||||
assert.False(t, sealed, "fencing must never seal: this node's copy is behind the winner's")
|
||||
}
|
||||
|
||||
// TestVolumeFence_LeavesTheDirtyMarkerToTheWinner covers the record an operator
|
||||
// reads afterwards. The winner takes the marker over when it opens the volume,
|
||||
// so a fencing node clearing it would erase the only note that writes were lost.
|
||||
func TestVolumeFence_LeavesTheDirtyMarkerToTheWinner(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-fencekeepsmarker"
|
||||
|
||||
cfg := fencedConfig(t, natsURL, "node-a", volumeName)
|
||||
|
||||
cfg.fenceVolume(t.Context(), volumeName, "node-b", "taken")
|
||||
|
||||
record, ok := cfg.dirty.holder(t.Context(), volumeName)
|
||||
require.True(t, ok, "fencing must leave the marker behind for the winner to take over")
|
||||
assert.Equal(t, "node-a", record.Owner)
|
||||
}
|
||||
|
||||
// TestVolumeFence_AlreadyUnmountedIsNotAnError covers the ordinary race, where
|
||||
// a volume is unmounted normally between losing its lease and the fence running.
|
||||
func TestVolumeFence_AlreadyUnmountedIsNotAnError(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
cfg := &Config{
|
||||
NodeName: "node-a",
|
||||
leases: newTestLeases(t, natsURL, "node-a"),
|
||||
dirty: newTestDirty(t, natsURL, "node-a"),
|
||||
}
|
||||
|
||||
cfg.fenceVolume(t.Context(), "vol-fencenotmounted", "node-b", "taken")
|
||||
}
|
||||
|
||||
// TestUnmountVolume_RefusesToSealUnderALiveWriter is the unmount side of the
|
||||
// same ordering. The seal rewrites the directory nbdkit writes, so a kill that
|
||||
// did not take makes it a concurrent write to that directory — and the seal
|
||||
// would publish a state torn between two writers. The entry has to stay
|
||||
// mounted so a retry re-attempts both.
|
||||
func TestUnmountVolume_RefusesToSealUnderALiveWriter(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
|
||||
const volumeName = "vol-unmountkillfails"
|
||||
|
||||
sealed := false
|
||||
cfg := fencedConfig(t, natsURL, "node-a", volumeName)
|
||||
cfg.sealVolume = func(context.Context, string) error { sealed = true; return nil }
|
||||
cfg.MountedVolumes[0].PID = 0 // rejected by ForceKillProcess, so the kill cannot succeed
|
||||
|
||||
response, err := unmountVolume(t.Context(), cfg, volumeName)
|
||||
|
||||
require.Error(t, err, "an unmount that could not stop the writer must not report success")
|
||||
assert.Contains(t, response.Error, "kill nbdkit",
|
||||
"the caller needs to know the writer is still up, not that some seal failed")
|
||||
assert.False(t, response.NotFound, "the volume is mounted here; reporting it missing would read as an already-completed unmount")
|
||||
assert.False(t, sealed, "sealing under a live nbdkit writes a state torn between two writers")
|
||||
|
||||
cfg.mu.Lock()
|
||||
mounted := len(cfg.MountedVolumes)
|
||||
cfg.mu.Unlock()
|
||||
assert.Equal(t, 1, mounted, "a failed unmount must leave the entry so a retry re-attempts the kill and the seal")
|
||||
}
|
||||
@@ -29,6 +29,22 @@ const volumeLeaseTTL = 45 * time.Second
|
||||
// TTL so one missed renewal does not surrender the lease.
|
||||
const volumeLeaseRenewInterval = 15 * time.Second
|
||||
|
||||
// volumeLeaseRenewTimeout bounds one renewal attempt. Without it a JetStream
|
||||
// call that hangs holds the renewal goroutine past the point where the server
|
||||
// has already given the entry away, and the holder never notices.
|
||||
const volumeLeaseRenewTimeout = 5 * time.Second
|
||||
|
||||
// volumeLeaseValidity is how long a holder may keep writing after its last
|
||||
// *confirmed* renewal. Set below volumeLeaseTTL on purpose: a server-side TTL
|
||||
// is not a lease unless the holder stops before the server may re-grant, so the
|
||||
// difference is the margin covering scheduling delay and request latency.
|
||||
const volumeLeaseValidity = 30 * time.Second
|
||||
|
||||
// volumeLeaseCheckInterval is how often validity is tested. Shorter than the
|
||||
// renewal interval so a lapsed holder is fenced on its own schedule rather than
|
||||
// on the next renewal that happens to come due.
|
||||
const volumeLeaseCheckInterval = 5 * time.Second
|
||||
|
||||
// errVolumeLeaseHeld reports that another claimant owns the volume. It is the
|
||||
// one lease failure a caller can do something about, so it is distinguishable.
|
||||
var errVolumeLeaseHeld = errors.New("volume is leased by another owner")
|
||||
@@ -52,12 +68,32 @@ type volumeLeaseRecord struct {
|
||||
AcquiredAt time.Time `json:"acquired_at"`
|
||||
}
|
||||
|
||||
// leaseLossKind is why a holder stopped being able to prove it owns a volume.
|
||||
// The two are not the same event and a fence reports them separately: one says
|
||||
// the cluster moved the volume, the other says this node lost contact with it.
|
||||
type leaseLossKind int
|
||||
|
||||
const (
|
||||
// leaseLostToPeer: a conditional update was rejected, so the entry changed
|
||||
// under this node and another may hold it now.
|
||||
leaseLostToPeer leaseLossKind = iota
|
||||
|
||||
// leaseLostStalled: renewal went unconfirmed for long enough that the server
|
||||
// TTL is about to admit another owner. Nobody has necessarily taken it yet.
|
||||
leaseLostStalled
|
||||
)
|
||||
|
||||
// volumeLeases hands out cluster-wide volume leases from a JetStream KV
|
||||
// bucket, and remembers which ones this node holds.
|
||||
type volumeLeases struct {
|
||||
kv jetstream.KeyValue
|
||||
owner string
|
||||
|
||||
// onLost is called when this node can no longer prove it owns a volume. Nil
|
||||
// leaves the loss logged and nothing else, which is what a unit test with no
|
||||
// export to tear down wants.
|
||||
onLost func(context.Context, string, leaseLossKind)
|
||||
|
||||
mu sync.Mutex
|
||||
held map[string]*volumeLease
|
||||
}
|
||||
@@ -98,6 +134,10 @@ type volumeLease struct {
|
||||
// lost records that the entry moved out from under this holder, which
|
||||
// means somebody else may now be writing the volume.
|
||||
lost bool
|
||||
// confirmed is when the last conditional update was acknowledged by the
|
||||
// server. An attempt still in flight does not count: only a write the
|
||||
// server accepted proves the entry is still ours.
|
||||
confirmed time.Time
|
||||
// refs counts opens on this node sharing the lease. The lease is released
|
||||
// when the last one lets go.
|
||||
refs int
|
||||
@@ -143,6 +183,7 @@ func (l *volumeLeases) acquire(ctx context.Context, volumeName string) (*volumeL
|
||||
volume: volumeName,
|
||||
generation: revision,
|
||||
revision: revision,
|
||||
confirmed: time.Now(),
|
||||
done: make(chan struct{}),
|
||||
refs: 1,
|
||||
}
|
||||
@@ -249,11 +290,19 @@ func (l *volumeLeases) release(ctx context.Context, lease *volumeLease) {
|
||||
}
|
||||
|
||||
// renewLoop refreshes the entry so the bucket TTL does not expire a live
|
||||
// holder, and notices when the lease has been taken away.
|
||||
// holder, notices when the lease has been taken away, and — the part that makes
|
||||
// it a lease rather than a hint — gives the volume up when it can no longer
|
||||
// prove the entry is still ours.
|
||||
//
|
||||
// The validity check is separate from the renewal because the two answer
|
||||
// different questions. Renewal asks "can I still reach JetStream"; validity
|
||||
// asks "may I still write". A holder partitioned from NATS keeps failing the
|
||||
// first indefinitely while the server hands the entry to somebody else, and
|
||||
// only the second stops it.
|
||||
func (lease *volumeLease) renewLoop(ctx context.Context) {
|
||||
defer close(lease.done)
|
||||
|
||||
ticker := time.NewTicker(volumeLeaseRenewInterval)
|
||||
ticker := time.NewTicker(volumeLeaseCheckInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
@@ -261,6 +310,13 @@ func (lease *volumeLease) renewLoop(ctx context.Context) {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if lease.expiredLocally() {
|
||||
lease.surrender(ctx)
|
||||
return
|
||||
}
|
||||
if time.Since(lease.lastConfirmed()) < volumeLeaseRenewInterval {
|
||||
continue
|
||||
}
|
||||
if !lease.renew(ctx) {
|
||||
return
|
||||
}
|
||||
@@ -268,12 +324,43 @@ func (lease *volumeLease) renewLoop(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// expiredLocally reports that this holder has gone longer than
|
||||
// volumeLeaseValidity without a confirmed renewal, so the server may be about
|
||||
// to grant the entry to somebody else.
|
||||
func (lease *volumeLease) expiredLocally() bool {
|
||||
return time.Since(lease.lastConfirmed()) >= volumeLeaseValidity
|
||||
}
|
||||
|
||||
func (lease *volumeLease) lastConfirmed() time.Time {
|
||||
lease.mu.Lock()
|
||||
defer lease.mu.Unlock()
|
||||
return lease.confirmed
|
||||
}
|
||||
|
||||
// surrender gives the volume up because this holder can no longer prove it owns
|
||||
// it. Marked lost so the release path cannot delete an entry that may already
|
||||
// belong to another node.
|
||||
func (lease *volumeLease) surrender(ctx context.Context) {
|
||||
lease.mu.Lock()
|
||||
lease.lost = true
|
||||
since := time.Since(lease.confirmed)
|
||||
lease.mu.Unlock()
|
||||
|
||||
slog.Error("volume lease could not be confirmed before its TTL, surrendering the volume",
|
||||
"volume", lease.volume, "generation", lease.generation,
|
||||
"unconfirmed_for_ms", otelsetup.Millis(since), "ttl_ms", otelsetup.Millis(volumeLeaseTTL))
|
||||
|
||||
if onLost := lease.leases.onLost; onLost != nil {
|
||||
go onLost(context.WithoutCancel(ctx), lease.volume, leaseLostStalled)
|
||||
}
|
||||
}
|
||||
|
||||
// renew rewrites the entry conditioned on the revision this holder last saw,
|
||||
// and reports whether the lease is still ours.
|
||||
//
|
||||
// Losing it is logged rather than enforced: until the object store can refuse
|
||||
// a stale writer's PUT, failing live guest I/O here would add a failure mode
|
||||
// without stopping anything.
|
||||
// Losing it hands off to onLost, which fences the export. That runs on its own
|
||||
// goroutine because fencing releases the lease, and releasing waits for this
|
||||
// one to exit.
|
||||
func (lease *volumeLease) renew(ctx context.Context) bool {
|
||||
record := volumeLeaseRecord{Owner: lease.leases.owner, Generation: lease.generation, AcquiredAt: time.Now().UTC()}
|
||||
payload, err := json.Marshal(record)
|
||||
@@ -286,11 +373,17 @@ func (lease *volumeLease) renew(ctx context.Context) bool {
|
||||
revision := lease.revision
|
||||
lease.mu.Unlock()
|
||||
|
||||
// Bounded so a hung JetStream call cannot hold this goroutine past the
|
||||
// point where the server has already re-granted the entry.
|
||||
ctx, cancel := context.WithTimeout(ctx, volumeLeaseRenewTimeout)
|
||||
defer cancel()
|
||||
|
||||
renewed, err := lease.leases.kv.Update(ctx, lease.key, payload, revision)
|
||||
switch {
|
||||
case err == nil:
|
||||
lease.mu.Lock()
|
||||
lease.revision = renewed
|
||||
lease.confirmed = time.Now()
|
||||
lease.mu.Unlock()
|
||||
return true
|
||||
case errors.Is(err, context.Canceled):
|
||||
@@ -302,6 +395,11 @@ func (lease *volumeLease) renew(ctx context.Context) bool {
|
||||
lease.lost = true
|
||||
lease.mu.Unlock()
|
||||
slog.Error("volume lease lost: another opener may hold this volume", "volume", lease.volume, "generation", lease.generation, "err", err)
|
||||
if onLost := lease.leases.onLost; onLost != nil {
|
||||
// WithoutCancel: release cancels this context, and the fence has
|
||||
// KV reads and a teardown to finish after that.
|
||||
go onLost(context.WithoutCancel(ctx), lease.volume, leaseLostToPeer)
|
||||
}
|
||||
return false
|
||||
default:
|
||||
// A transient JetStream error is not a lost lease. Keep renewing; the
|
||||
@@ -328,7 +426,26 @@ func (cfg *Config) acquireVolumeLease(ctx context.Context, volumeName string) (*
|
||||
if cfg.leases == nil {
|
||||
return nil, fmt.Errorf("%w: cannot establish exclusive access to %s", errNoVolumeLeaseStore, volumeName)
|
||||
}
|
||||
return cfg.leases.acquire(ctx, volumeName)
|
||||
lease, err := cfg.leases.acquire(ctx, volumeName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Mark before a single write lands, not when a seal fails. A node killed
|
||||
// mid-write never reaches its seal, so marking there leaves no trace of the
|
||||
// case the marker exists for. A takeover writes its own richer reason.
|
||||
took, err := cfg.reportVolumeTakeover(ctx, volumeName, lease.generation)
|
||||
if err == nil && !took {
|
||||
err = cfg.markVolumeDirty(ctx, volumeName, lease.generation,
|
||||
"volume open, writes not yet confirmed to the backend")
|
||||
}
|
||||
if err != nil {
|
||||
// Opening anyway would leave writes nothing records, so a later takeover
|
||||
// could not tell this node ever held them. Give the lease back instead.
|
||||
cfg.leases.release(ctx, lease)
|
||||
return nil, fmt.Errorf("record unconfirmed writes for %s: %w", volumeName, err)
|
||||
}
|
||||
return lease, nil
|
||||
}
|
||||
|
||||
// releaseVolumeLease gives up a lease taken by acquireVolumeLease.
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/nats-io/nats.go"
|
||||
@@ -224,3 +225,85 @@ func TestVolumeLease_RejectsUnsafeKeys(t *testing.T) {
|
||||
require.Errorf(t, err, "volume name %q must not become a lease key", name)
|
||||
}
|
||||
}
|
||||
|
||||
// TestVolumeLease_ValidityIsBoundedBelowTheServerTTL pins the arithmetic that
|
||||
// makes this a lease rather than a hint. The holder has to stop writing before
|
||||
// the server may grant the entry to somebody else, and it has to get at least
|
||||
// one check in between giving up on renewal and reaching that point.
|
||||
func TestVolumeLease_ValidityIsBoundedBelowTheServerTTL(t *testing.T) {
|
||||
require.Less(t, volumeLeaseValidity, volumeLeaseTTL,
|
||||
"a holder that stops only when the server expires the entry has already overlapped its successor")
|
||||
require.Less(t, volumeLeaseRenewInterval, volumeLeaseValidity,
|
||||
"a renewal that comes due after validity lapses can never confirm one in time")
|
||||
require.Less(t, volumeLeaseCheckInterval, volumeLeaseValidity-volumeLeaseRenewInterval,
|
||||
"validity has to be tested at least once between a missed renewal and the deadline it enforces")
|
||||
require.Less(t, volumeLeaseRenewTimeout, volumeLeaseCheckInterval+volumeLeaseRenewInterval,
|
||||
"an unbounded renewal holds the goroutine past the point the entry may have been re-granted")
|
||||
}
|
||||
|
||||
// TestVolumeLease_UnconfirmedRenewalSurrendersTheVolume is the partition case
|
||||
// revision checking cannot see. A node that loses NATS while the winner keeps
|
||||
// it never gets an update rejected, so nothing tells it the volume moved. It
|
||||
// has to give the volume up on its own clock, and report the reason as a stall
|
||||
// rather than as a peer taking it — nobody has necessarily taken it yet.
|
||||
func TestVolumeLease_UnconfirmedRenewalSurrendersTheVolume(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
leases := newTestLeases(t, natsURL, "node-a")
|
||||
|
||||
const volumeName = "vol-leasestalled"
|
||||
lost := make(chan leaseLossKind, 1)
|
||||
leases.onLost = func(_ context.Context, volume string, kind leaseLossKind) {
|
||||
assert.Equal(t, volumeName, volume)
|
||||
lost <- kind
|
||||
}
|
||||
|
||||
lease, err := leases.acquire(t.Context(), volumeName)
|
||||
require.NoError(t, err)
|
||||
lease.stop()
|
||||
<-lease.done
|
||||
|
||||
require.False(t, lease.expiredLocally(), "a lease confirmed just now is still valid")
|
||||
|
||||
// Age the last confirmation past the validity window, which is what a node
|
||||
// that cannot reach JetStream looks like from inside.
|
||||
lease.mu.Lock()
|
||||
lease.confirmed = time.Now().Add(-volumeLeaseValidity - time.Second)
|
||||
lease.mu.Unlock()
|
||||
require.True(t, lease.expiredLocally(), "an unconfirmed holder must not still consider itself valid")
|
||||
|
||||
lease.surrender(t.Context())
|
||||
|
||||
select {
|
||||
case kind := <-lost:
|
||||
assert.Equal(t, leaseLostStalled, kind,
|
||||
"a stall is not a takeover: reporting it as one would name a winner that may not exist")
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("a surrendered lease must fence its export, and nothing was called")
|
||||
}
|
||||
|
||||
lease.mu.Lock()
|
||||
defer lease.mu.Unlock()
|
||||
assert.True(t, lease.lost, "a surrendered lease must be marked lost so release cannot delete the successor's entry")
|
||||
}
|
||||
|
||||
// TestVolumeLease_SurrenderedLeaseIsNotDeletedOnRelease covers what happens
|
||||
// after. The entry may already belong to another node, so deleting it on the
|
||||
// way out would evict a live holder and hand the volume to a third opener.
|
||||
func TestVolumeLease_SurrenderedLeaseIsNotDeletedOnRelease(t *testing.T) {
|
||||
_, natsURL := setupEmbeddedNATS(t)
|
||||
leases := newTestLeases(t, natsURL, "node-a")
|
||||
|
||||
const volumeName = "vol-leasestalledrelease"
|
||||
lease, err := leases.acquire(t.Context(), volumeName)
|
||||
require.NoError(t, err)
|
||||
|
||||
lease.mu.Lock()
|
||||
lease.confirmed = time.Now().Add(-volumeLeaseValidity - time.Second)
|
||||
lease.mu.Unlock()
|
||||
lease.surrender(t.Context())
|
||||
|
||||
leases.release(t.Context(), lease)
|
||||
|
||||
_, err = leases.kv.Get(t.Context(), volumeName)
|
||||
require.NoError(t, err, "releasing a surrendered lease must leave the entry alone: it may be the successor's now")
|
||||
}
|
||||
|
||||
@@ -68,6 +68,9 @@ type EBSUnMountResponse struct {
|
||||
// after a request that timed out client-side but completed server-side).
|
||||
// The caller treats this as a successful, idempotent seal.
|
||||
NotFound bool `json:"NotFound"`
|
||||
// Reaped signals the handler found and killed an export for this volume
|
||||
// that had no registry entry, which is a leak whatever left it behind.
|
||||
Reaped bool `json:"Reaped,omitempty"`
|
||||
}
|
||||
|
||||
type EBSSyncRequest struct {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package utils
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"runtime"
|
||||
@@ -66,7 +67,13 @@ func ForceKillProcess(pid int, timeout time.Duration) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// A process that has already exited is the outcome asked for, not a failure.
|
||||
// Callers use this to establish that a writer is gone, and one that died on
|
||||
// its own is as gone as one this killed.
|
||||
if err := process.Signal(syscall.SIGKILL); err != nil {
|
||||
if errors.Is(err, os.ErrProcessDone) || errors.Is(err, syscall.ESRCH) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
return WaitForProcessExit(pid, timeout)
|
||||
|
||||
@@ -684,6 +684,20 @@ func TestForceKillProcess(t *testing.T) {
|
||||
assert.Error(t, ForceKillProcess(0, time.Second), "invalid pid must error")
|
||||
}
|
||||
|
||||
// TestForceKillProcess_AlreadyExitedIsSuccess pins the contract callers rely on.
|
||||
// This is asked "make sure this process is gone", and one that died on its own
|
||||
// is as gone as one it killed — reporting a failure there makes a caller treat
|
||||
// a stopped writer as a live one.
|
||||
func TestForceKillProcess_AlreadyExitedIsSuccess(t *testing.T) {
|
||||
cmd := exec.Command("true")
|
||||
require.NoError(t, cmd.Start())
|
||||
pid := cmd.Process.Pid
|
||||
require.NoError(t, cmd.Wait())
|
||||
|
||||
assert.NoError(t, ForceKillProcess(pid, time.Second),
|
||||
"an already-exited process satisfies the request, it does not fail it")
|
||||
}
|
||||
|
||||
func TestStopProcess(t *testing.T) {
|
||||
// Create and start a test process
|
||||
cmd := exec.Command("sleep", "60")
|
||||
|
||||
+11
-7
@@ -173,8 +173,7 @@ func HotplugEBSBus(port int) string {
|
||||
//
|
||||
// werror/rerror mirror the boot drive's on-error policy (see Drive.Werror):
|
||||
// a data volume has no -drive-backed BlockBackend to carry it, so the qdev
|
||||
// device itself sets werror=report,rerror=report to report backend ENOSPC to
|
||||
// the guest instead of letting QEMU's default werror=enospc pause the VM.
|
||||
// device itself sets werror=stop,rerror=stop.
|
||||
func VolumeBlkDeviceArgs(volumeID, nodeName, iothreadID, bus string) []string {
|
||||
return []string{
|
||||
fmt.Sprintf("id=%s", VolumeDeviceID(volumeID)),
|
||||
@@ -182,8 +181,8 @@ func VolumeBlkDeviceArgs(volumeID, nodeName, iothreadID, bus string) []string {
|
||||
fmt.Sprintf("iothread=%s", iothreadID),
|
||||
fmt.Sprintf("serial=%s", VolumeSerial(volumeID)),
|
||||
fmt.Sprintf("bus=%s", bus),
|
||||
"werror=report",
|
||||
"rerror=report",
|
||||
"werror=stop",
|
||||
"rerror=stop",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -205,11 +204,16 @@ func VolumeBlkDeviceQMPArgs(volumeID, nodeName, iothreadID, bus string) map[stri
|
||||
"iothread": iothreadID,
|
||||
"serial": VolumeSerial(volumeID),
|
||||
"bus": bus,
|
||||
"werror": "report",
|
||||
"rerror": "report",
|
||||
"werror": "stop",
|
||||
"rerror": "stop",
|
||||
}
|
||||
}
|
||||
|
||||
// NBDReconnectDelaySeconds is how long an NBD disconnect pauses requests before
|
||||
// they start failing. It covers a storage process restarting on the same socket;
|
||||
// past it the error surfaces and werror=stop pauses the VM instead.
|
||||
const NBDReconnectDelaySeconds = 30
|
||||
|
||||
// NBDServerOpts holds the server.* fields QEMU's nbd blockdev driver needs,
|
||||
// broken out of a parsed NBD URI (see utils.ParseNBDURI) so both the QMP
|
||||
// blockdev-add nested "server" object and the command-line -blockdev
|
||||
@@ -244,6 +248,6 @@ func (o NBDServerOpts) CommandLineArgs() []string {
|
||||
// which only creates a BlockBackend with an auto-generated node-name).
|
||||
func VolumeBlockdev(nodeName string, server NBDServerOpts) Blockdev {
|
||||
opts := append([]string{"driver=nbd", fmt.Sprintf("node-name=%s", nodeName)}, server.CommandLineArgs()...)
|
||||
opts = append(opts, "export=")
|
||||
opts = append(opts, "export=", fmt.Sprintf("reconnect-delay=%d", NBDReconnectDelaySeconds))
|
||||
return Blockdev{Value: strings.Join(opts, ",")}
|
||||
}
|
||||
|
||||
+10
-11
@@ -167,21 +167,20 @@ func TestRngDevice_MMIO_WithOptions(t *testing.T) {
|
||||
assert.Equal(t, "virtio-rng-device", d.Value)
|
||||
}
|
||||
|
||||
// TestVolumeBlkDevice_OnErrorReport pins the cold-boot data-volume device
|
||||
// string to werror=report,rerror=report, mirroring the boot drive's existing
|
||||
// on-error policy so a full backend reports ENOSPC to the guest instead of
|
||||
// pausing the VM.
|
||||
func TestVolumeBlkDevice_OnErrorReport(t *testing.T) {
|
||||
// TestVolumeBlkDevice_OnErrorStop pins the cold-boot data-volume device string
|
||||
// to werror=stop,rerror=stop. Reporting the error handed a backend outage to
|
||||
// the guest as EIO, which aborts its journal; stop holds the request instead.
|
||||
func TestVolumeBlkDevice_OnErrorStop(t *testing.T) {
|
||||
d := VolumeBlkDevice("vol-data-a", "nbd-vol-data-a", "ioth-vol-data-a", "hotplug-ebs3")
|
||||
assert.Equal(t, "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=report,rerror=report", d.Value)
|
||||
assert.Equal(t, "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=stop,rerror=stop", d.Value)
|
||||
}
|
||||
|
||||
// TestVolumeBlkDeviceQMPArgs_OnErrorReport is the hotplug counterpart: the
|
||||
// QMP device_add argument map AttachVolume sends must carry the same policy.
|
||||
func TestVolumeBlkDeviceQMPArgs_OnErrorReport(t *testing.T) {
|
||||
// TestVolumeBlkDeviceQMPArgs_OnErrorStop is the hotplug counterpart: the QMP
|
||||
// device_add argument map AttachVolume sends must carry the same policy.
|
||||
func TestVolumeBlkDeviceQMPArgs_OnErrorStop(t *testing.T) {
|
||||
args := VolumeBlkDeviceQMPArgs("vol-data-a", "nbd-vol-data-a", "ioth-vol-data-a", "hotplug-ebs3")
|
||||
assert.Equal(t, "report", args["werror"])
|
||||
assert.Equal(t, "report", args["rerror"])
|
||||
assert.Equal(t, "stop", args["werror"])
|
||||
assert.Equal(t, "stop", args["rerror"])
|
||||
}
|
||||
|
||||
// host_mtu is how a guest that never takes a DHCP lease — statically addressed,
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
package vm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
)
|
||||
|
||||
// fenceStateReasonCode is the StateReason a fenced instance carries. Distinct
|
||||
// from Server.RecoveryFailed so an operator can tell "this node could not
|
||||
// recover the guest" from "another node owns this guest's disks now".
|
||||
const fenceStateReasonCode = "Server.VolumeFenced"
|
||||
|
||||
// FenceVolume stops the guest running against volumeID on this node, because
|
||||
// the volume's lease has moved and this node is no longer its owner.
|
||||
//
|
||||
// It is deliberately the narrowest teardown in the package. The guest process
|
||||
// is killed and its taps are removed, and nothing else: the volumes, ENIs and
|
||||
// addresses all belong to whichever node holds the instance now, so releasing
|
||||
// them here would tear down resources in live use elsewhere. For the same
|
||||
// reason the volumes are not unmounted, which would seal this node's stale copy
|
||||
// over the winner's.
|
||||
//
|
||||
// Returns the instance it stopped, or "" when no local guest was using the
|
||||
// volume — the ordinary case, since a volume is usually fenced after its
|
||||
// instance has already gone.
|
||||
func (m *Manager) FenceVolume(ctx context.Context, volumeID, reason string) string {
|
||||
instance := m.instanceUsingVolume(volumeID)
|
||||
if instance == nil {
|
||||
return ""
|
||||
}
|
||||
|
||||
skip := false
|
||||
var observed InstanceState
|
||||
m.Inspect(instance, func(v *VM) {
|
||||
observed = v.Status
|
||||
if v.Status == StateError || v.Status == StateShuttingDown || v.Status == StateTerminated {
|
||||
skip = true
|
||||
return
|
||||
}
|
||||
if v.Instance != nil {
|
||||
v.Instance.StateReason = &ec2.StateReason{
|
||||
Code: aws.String(fenceStateReasonCode),
|
||||
Message: aws.String(reason),
|
||||
}
|
||||
}
|
||||
})
|
||||
if skip {
|
||||
slog.InfoContext(ctx, "FenceVolume: instance already in a terminal state",
|
||||
"instanceId", instance.ID, "volumeId", volumeID, "status", string(observed))
|
||||
return instance.ID
|
||||
}
|
||||
|
||||
if err := m.transitionWithPrecheck(instance, StateError); err != nil {
|
||||
slog.ErrorContext(ctx, "FenceVolume transition failed", "instanceId", instance.ID, "err", err)
|
||||
if m.Status(instance) != StateError {
|
||||
return instance.ID
|
||||
}
|
||||
}
|
||||
slog.ErrorContext(ctx, "Instance fenced: another node owns its volume, so the guest is being stopped here",
|
||||
"instanceId", instance.ID, "volumeId", volumeID, "reason", reason)
|
||||
|
||||
m.goroutineWg.Go(func() {
|
||||
m.shutdownQEMU(instance)
|
||||
m.cleanupTapDevices(instance)
|
||||
m.Inspect(instance, func(v *VM) { v.LastNode = m.deps.NodeID })
|
||||
if err := m.writeRunningState(); err != nil {
|
||||
slog.ErrorContext(ctx, "FenceVolume: failed to persist state after fencing",
|
||||
"instanceId", instance.ID, "err", err)
|
||||
}
|
||||
})
|
||||
return instance.ID
|
||||
}
|
||||
|
||||
// instanceUsingVolume returns the local instance holding volumeID, or nil. The
|
||||
// map is this node's own running set, so a hit means the guest is here.
|
||||
func (m *Manager) instanceUsingVolume(volumeID string) *VM {
|
||||
var found *VM
|
||||
m.View(func(vms map[string]*VM) {
|
||||
for _, v := range vms {
|
||||
for _, req := range v.EBSRequests.Requests {
|
||||
if req.Name == volumeID {
|
||||
found = v
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
return found
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
package vm
|
||||
|
||||
//test:in-package — FenceVolume's decision is about unexported manager state
|
||||
//(instance status, StateReason, the local VM map) with no exported surface.
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// vmUsingVolume builds a running instance with volumeID attached, as a guest
|
||||
// that viperblockd has an export open for would look.
|
||||
func vmUsingVolume(id, volumeID string) *VM {
|
||||
return &VM{
|
||||
ID: id,
|
||||
Status: StateRunning,
|
||||
Instance: &ec2.Instance{},
|
||||
EBSRequests: types.EBSRequests{Requests: []types.EBSRequest{{Name: volumeID}}},
|
||||
}
|
||||
}
|
||||
|
||||
// TestFenceVolume_StopsTheGuestUsingTheVolume is the property the fence exists
|
||||
// for on this side. viperblockd has already torn the export down, so the guest
|
||||
// is running against a disk that is gone and its writes can never land.
|
||||
func TestFenceVolume_StopsTheGuestUsingTheVolume(t *testing.T) {
|
||||
m := NewManagerWithDeps(Deps{NodeID: "node-a"})
|
||||
instance := vmUsingVolume("i-fenced", "vol-fenced")
|
||||
m.Insert(instance)
|
||||
|
||||
got := m.FenceVolume(t.Context(), "vol-fenced", "lease moved to node-b")
|
||||
m.goroutineWg.Wait()
|
||||
|
||||
assert.Equal(t, "i-fenced", got)
|
||||
assert.Equal(t, StateError, m.Status(instance),
|
||||
"a guest whose disk was taken away is not running, whatever the record says")
|
||||
}
|
||||
|
||||
// TestFenceVolume_RecordsWhyItStopped covers what an operator sees afterwards.
|
||||
// A guest in error with no reason is indistinguishable from a crash, and the
|
||||
// distinction is the whole point: another node owns this volume now.
|
||||
func TestFenceVolume_RecordsWhyItStopped(t *testing.T) {
|
||||
m := NewManagerWithDeps(Deps{NodeID: "node-a"})
|
||||
instance := vmUsingVolume("i-fencedreason", "vol-fencedreason")
|
||||
m.Insert(instance)
|
||||
|
||||
m.FenceVolume(t.Context(), "vol-fencedreason", "lease moved to node-b")
|
||||
m.goroutineWg.Wait()
|
||||
|
||||
require.NotNil(t, instance.Instance.StateReason)
|
||||
assert.Equal(t, fenceStateReasonCode, *instance.Instance.StateReason.Code)
|
||||
assert.Equal(t, "lease moved to node-b", *instance.Instance.StateReason.Message)
|
||||
}
|
||||
|
||||
// TestFenceVolume_RecordsTheNodeItRanOn keeps placement working after a fence.
|
||||
// Start prefers the node that last ran the instance, so a fence that did not
|
||||
// stamp LastNode would lose the only hint about where its state is.
|
||||
func TestFenceVolume_RecordsTheNodeItRanOn(t *testing.T) {
|
||||
m := NewManagerWithDeps(Deps{NodeID: "node-a"})
|
||||
instance := vmUsingVolume("i-fencedlastnode", "vol-fencedlastnode")
|
||||
m.Insert(instance)
|
||||
|
||||
m.FenceVolume(t.Context(), "vol-fencedlastnode", "lease moved")
|
||||
m.goroutineWg.Wait()
|
||||
|
||||
assert.Equal(t, "node-a", instance.LastNode)
|
||||
}
|
||||
|
||||
// TestFenceVolume_NoLocalGuestIsNotAnError covers the ordinary case. A volume
|
||||
// is usually fenced after the instance using it has already gone, and there is
|
||||
// nothing left here to stop.
|
||||
func TestFenceVolume_NoLocalGuestIsNotAnError(t *testing.T) {
|
||||
m := NewManagerWithDeps(Deps{NodeID: "node-a"})
|
||||
m.Insert(vmUsingVolume("i-other", "vol-somethingelse"))
|
||||
|
||||
assert.Empty(t, m.FenceVolume(t.Context(), "vol-nobodyhas", "lease moved"))
|
||||
}
|
||||
|
||||
// TestFenceVolume_LeavesATerminalInstanceAlone stops the fence undoing a
|
||||
// teardown already in flight. Driving a shutting-down instance to error would
|
||||
// strand its volumes and ENIs, which is worse than the state it came from.
|
||||
func TestFenceVolume_LeavesATerminalInstanceAlone(t *testing.T) {
|
||||
for _, state := range []InstanceState{StateError, StateShuttingDown, StateTerminated} {
|
||||
t.Run(string(state), func(t *testing.T) {
|
||||
m := NewManagerWithDeps(Deps{NodeID: "node-a"})
|
||||
instance := vmUsingVolume("i-fencedterminal", "vol-fencedterminal")
|
||||
instance.Status = state
|
||||
m.Insert(instance)
|
||||
|
||||
got := m.FenceVolume(t.Context(), "vol-fencedterminal", "lease moved")
|
||||
m.goroutineWg.Wait()
|
||||
|
||||
assert.Equal(t, "i-fencedterminal", got)
|
||||
assert.Equal(t, state, m.Status(instance), "a terminal instance must not be re-driven")
|
||||
assert.Nil(t, instance.Instance.StateReason,
|
||||
"the reason belongs to the teardown already under way, not to this fence")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestInstanceUsingVolume_MatchesOnAnyAttachedVolume covers the lookup. A guest
|
||||
// is fenced for its data disk as readily as its root, so a match on any
|
||||
// attached volume has to count.
|
||||
func TestInstanceUsingVolume_MatchesOnAnyAttachedVolume(t *testing.T) {
|
||||
m := NewManagerWithDeps(Deps{})
|
||||
instance := &VM{
|
||||
ID: "i-multi",
|
||||
Status: StateRunning,
|
||||
EBSRequests: types.EBSRequests{Requests: []types.EBSRequest{
|
||||
{Name: "vol-root"}, {Name: "vol-data"},
|
||||
}},
|
||||
}
|
||||
m.Insert(instance)
|
||||
|
||||
assert.Equal(t, instance, m.instanceUsingVolume("vol-data"))
|
||||
assert.Equal(t, instance, m.instanceUsingVolume("vol-root"))
|
||||
assert.Nil(t, m.instanceUsingVolume("vol-absent"))
|
||||
}
|
||||
@@ -890,6 +890,19 @@ const (
|
||||
// mark an instance impaired and trigger a process-liveness check (90s of
|
||||
// unresponsiveness). Exported so DescribeInstanceStatus uses the same gate.
|
||||
QMPMaxConsecutiveFailures = 3
|
||||
// qmpStatusIOError is the run state QEMU reports for a VM paused by
|
||||
// werror=stop or rerror=stop.
|
||||
qmpStatusIOError = "io-error"
|
||||
// qmpMaxIOErrorResumes is how many cont attempts are made at heartbeat rate
|
||||
// before the instance is called impaired. It is a reporting threshold, not
|
||||
// a stopping one: QEMU holds the failed request for as long as the guest
|
||||
// exists, so there is always something to recover.
|
||||
qmpMaxIOErrorResumes = 10
|
||||
// qmpIOErrorLogEvery is the heartbeat count between log lines once the
|
||||
// instance is impaired. Only the logging slows: a retry costs one replayed
|
||||
// I/O, and a guest that waited minutes to find out its backend had returned
|
||||
// was paying far more for it than the retry ever saved.
|
||||
qmpIOErrorLogEvery = 10
|
||||
)
|
||||
|
||||
// qmpHeartbeat polls query-status every qmpHeartbeatInterval. A dead process
|
||||
@@ -922,6 +935,15 @@ func (m *Manager) qmpHeartbeatPoll(instance *VM) bool {
|
||||
slog.Debug("QMP heartbeat", "instance", instance.ID)
|
||||
ctx := context.WithValue(context.Background(), noTraceKey{}, true)
|
||||
qmpStatus, err := queryQMPStatus(ctx, instance, qmpCommandTimeout)
|
||||
|
||||
// A VM paused by werror=stop is not a failure to recover from: the guest is
|
||||
// intact and its failed request is still held, so the answer is to re-submit
|
||||
// it once the backend is back rather than to count it as unresponsive.
|
||||
if err == nil && qmpStatus.Status == qmpStatusIOError && m.Status(instance) == StateRunning {
|
||||
m.resumeAfterIOError(ctx, instance)
|
||||
return true
|
||||
}
|
||||
|
||||
if err == nil && !qmpStatus.Running && m.Status(instance) == StateRunning {
|
||||
err = fmt.Errorf("QMP reported non-running status %q", qmpStatus.Status)
|
||||
}
|
||||
@@ -948,6 +970,44 @@ func (m *Manager) qmpHeartbeatPoll(instance *VM) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// resumeAfterIOError re-submits the held request on a VM that werror=stop
|
||||
// paused. cont replays the failed I/O rather than discarding it, so a backend
|
||||
// that has come back leaves the guest with nothing to notice but latency.
|
||||
//
|
||||
// Retries run at heartbeat rate for as long as the guest exists. After
|
||||
// qmpMaxIOErrorResumes the instance is reported impaired and the log line is
|
||||
// throttled, but the attempts are not: a cont costs one replayed I/O, so
|
||||
// retrying a backend that is still down is nearly free, while a guest still
|
||||
// paused minutes after its backend returned is an outage of our own making.
|
||||
func (m *Manager) resumeAfterIOError(ctx context.Context, instance *VM) {
|
||||
var attempt int
|
||||
m.UpdateState(instance.ID, func(v *VM) {
|
||||
v.Health.IOErrorResumes++
|
||||
attempt = v.Health.IOErrorResumes
|
||||
if attempt == 1 {
|
||||
v.Health.IOErrorSince = time.Now()
|
||||
}
|
||||
if attempt == qmpMaxIOErrorResumes {
|
||||
v.Health.ImpairedSince = time.Now()
|
||||
}
|
||||
})
|
||||
|
||||
if attempt == qmpMaxIOErrorResumes {
|
||||
slog.Error("Guest still paused on I/O error; reporting impaired and slowing retries",
|
||||
"instance", instance.ID, "attempts", attempt)
|
||||
}
|
||||
// A paused guest is polled for as long as it exists, so logging every
|
||||
// attempt would bury the transition that matters in copies of itself.
|
||||
if attempt <= qmpMaxIOErrorResumes || attempt%qmpIOErrorLogEvery == 0 {
|
||||
slog.Warn("Guest paused on a backend I/O error, resuming",
|
||||
"instance", instance.ID, "attempt", attempt)
|
||||
}
|
||||
if _, err := sendQMPCommandWithTimeout(ctx, instance.QMPClient,
|
||||
qmp.QMPCommand{Execute: "cont"}, instance.ID, qmpCommandTimeout); err != nil {
|
||||
slog.Error("Failed to resume guest after I/O error", "instance", instance.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// queryQMPStatus decodes query-status so monitor responsiveness is not
|
||||
// mistaken for a running guest.
|
||||
func queryQMPStatus(ctx context.Context, instance *VM, timeout time.Duration) (qmp.Status, error) {
|
||||
@@ -988,6 +1048,8 @@ func (m *Manager) recordQMPSuccess(instance *VM) {
|
||||
v.Health.QMPConsecutiveFailures = 0
|
||||
v.Health.ImpairedSince = time.Time{}
|
||||
v.Health.LastQMPSuccess = time.Now()
|
||||
v.Health.IOErrorResumes = 0
|
||||
v.Health.IOErrorSince = time.Time{}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1310,12 +1372,12 @@ func buildDrives(requests []types.EBSRequest, cpuCount int, machineType string)
|
||||
ID: "os",
|
||||
Cache: "none",
|
||||
|
||||
// Report backend write errors to the guest rather than letting
|
||||
// QEMU's default werror=enospc pause the VM: when the storage
|
||||
// pool exhausts, the guest fs then returns a clean ENOSPC to
|
||||
// userspace and the VM stays reachable instead of freezing.
|
||||
Werror: "report",
|
||||
Rerror: "report",
|
||||
// Pause the VM and hold the failed request rather than
|
||||
// delivering EIO to the guest, which aborts its journal. A
|
||||
// transient backend loss is then recovered with cont.
|
||||
Werror: "stop",
|
||||
Rerror: "stop",
|
||||
ReconnectDelay: NBDReconnectDelaySeconds,
|
||||
}
|
||||
|
||||
iothreadID := "ioth-os"
|
||||
|
||||
+136
-13
@@ -7,6 +7,7 @@ import (
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -127,7 +128,7 @@ func TestBuildDrives(t *testing.T) {
|
||||
},
|
||||
cpuCount: 4,
|
||||
wantDrives: []Drive{
|
||||
{File: "nbd:unix:/tmp/boot.sock", Format: "raw", If: "none", Media: "disk", ID: "os", Cache: "none", Werror: "report", Rerror: "report"},
|
||||
{File: "nbd:unix:/tmp/boot.sock", Format: "raw", If: "none", Media: "disk", ID: "os", Cache: "none", Werror: "stop", Rerror: "stop", ReconnectDelay: NBDReconnectDelaySeconds},
|
||||
},
|
||||
wantIOThreads: []IOThread{{ID: "ioth-os"}},
|
||||
wantDevices: []Device{
|
||||
@@ -168,10 +169,10 @@ func TestBuildDrives(t *testing.T) {
|
||||
cpuCount: 2,
|
||||
wantIOThreads: []IOThread{{ID: "ioth-vol-data-a"}},
|
||||
wantBlockdevs: []Blockdev{
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/tmp/data-a.sock,export="},
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/tmp/data-a.sock,export=,reconnect-delay=30"},
|
||||
},
|
||||
wantDevices: []Device{
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=report,rerror=report"},
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=stop,rerror=stop"},
|
||||
},
|
||||
wantHotplugPorts: []int{3},
|
||||
},
|
||||
@@ -183,10 +184,10 @@ func TestBuildDrives(t *testing.T) {
|
||||
cpuCount: 2,
|
||||
wantIOThreads: []IOThread{{ID: "ioth-vol-data-b"}},
|
||||
wantBlockdevs: []Blockdev{
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-b,server.type=unix,server.path=/tmp/data-b.sock,export="},
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-b,server.type=unix,server.path=/tmp/data-b.sock,export=,reconnect-delay=30"},
|
||||
},
|
||||
wantDevices: []Device{
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-b,drive=nbd-vol-data-b,iothread=ioth-vol-data-b,serial=voldatab,bus=hotplug-ebs1,werror=report,rerror=report"},
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-b,drive=nbd-vol-data-b,iothread=ioth-vol-data-b,serial=voldatab,bus=hotplug-ebs1,werror=stop,rerror=stop"},
|
||||
},
|
||||
wantHotplugPorts: []int{1},
|
||||
},
|
||||
@@ -202,12 +203,12 @@ func TestBuildDrives(t *testing.T) {
|
||||
{ID: "ioth-vol-data-d"},
|
||||
},
|
||||
wantBlockdevs: []Blockdev{
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-c,server.type=unix,server.path=/tmp/data-c.sock,export="},
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-d,server.type=unix,server.path=/tmp/data-d.sock,export="},
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-c,server.type=unix,server.path=/tmp/data-c.sock,export=,reconnect-delay=30"},
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-d,server.type=unix,server.path=/tmp/data-d.sock,export=,reconnect-delay=30"},
|
||||
},
|
||||
wantDevices: []Device{
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-c,drive=nbd-vol-data-c,iothread=ioth-vol-data-c,serial=voldatac,bus=hotplug-ebs1,werror=report,rerror=report"},
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-d,drive=nbd-vol-data-d,iothread=ioth-vol-data-d,serial=voldatad,bus=hotplug-ebs2,werror=report,rerror=report"},
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-c,drive=nbd-vol-data-c,iothread=ioth-vol-data-c,serial=voldatac,bus=hotplug-ebs1,werror=stop,rerror=stop"},
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-d,drive=nbd-vol-data-d,iothread=ioth-vol-data-d,serial=voldatad,bus=hotplug-ebs2,werror=stop,rerror=stop"},
|
||||
},
|
||||
wantHotplugPorts: []int{1, 2},
|
||||
},
|
||||
@@ -227,7 +228,7 @@ func TestBuildDrives(t *testing.T) {
|
||||
wantErr: "no free EBS hot-plug port for volume vol-overflow",
|
||||
},
|
||||
{
|
||||
name: "mixed boot + EFI + data: boot and EFI stay byte-identical to the legacy shape",
|
||||
name: "mixed boot + EFI + data: boot and EFI keep the legacy -drive shape",
|
||||
requests: []types.EBSRequest{
|
||||
{Name: "vol-boot", NBDURI: "nbd:unix:/tmp/boot.sock", Boot: true},
|
||||
{Name: "vol-efi", NBDURI: "nbd:unix:/tmp/efi.sock", EFI: true},
|
||||
@@ -235,7 +236,7 @@ func TestBuildDrives(t *testing.T) {
|
||||
},
|
||||
cpuCount: 4,
|
||||
wantDrives: []Drive{
|
||||
{File: "nbd:unix:/tmp/boot.sock", Format: "raw", If: "none", Media: "disk", ID: "os", Cache: "none", Werror: "report", Rerror: "report"},
|
||||
{File: "nbd:unix:/tmp/boot.sock", Format: "raw", If: "none", Media: "disk", ID: "os", Cache: "none", Werror: "stop", Rerror: "stop", ReconnectDelay: NBDReconnectDelaySeconds},
|
||||
{File: "nbd:unix:/tmp/efi.sock", Format: "raw", If: "pflash", Unit: 1},
|
||||
},
|
||||
wantIOThreads: []IOThread{
|
||||
@@ -243,11 +244,11 @@ func TestBuildDrives(t *testing.T) {
|
||||
{ID: "ioth-vol-data-a"},
|
||||
},
|
||||
wantBlockdevs: []Blockdev{
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/tmp/data-a.sock,export="},
|
||||
{Value: "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/tmp/data-a.sock,export=,reconnect-delay=30"},
|
||||
},
|
||||
wantDevices: []Device{
|
||||
{Value: "virtio-blk-pci,drive=os,iothread=ioth-os,num-queues=4,bootindex=1"},
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=report,rerror=report"},
|
||||
{Value: "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=stop,rerror=stop"},
|
||||
},
|
||||
wantHotplugPorts: []int{0, 0, 3},
|
||||
},
|
||||
@@ -1379,3 +1380,125 @@ func TestNewQMPClientWithHandshake_RetriesTransientConnect(t *testing.T) {
|
||||
require.NotNil(t, client)
|
||||
_ = client.Conn.Close()
|
||||
}
|
||||
|
||||
// startRecordingQMPListener is startWorkingQMPListener with a tally: it counts
|
||||
// the commands it was asked to execute, which is how a test says how many times
|
||||
// the manager tried to resume a paused guest.
|
||||
func startRecordingQMPListener(t *testing.T) (string, func(string) int) {
|
||||
t.Helper()
|
||||
sockPath := filepath.Join(t.TempDir(), "qmp.sock")
|
||||
ln, err := net.Listen("unix", sockPath)
|
||||
require.NoError(t, err)
|
||||
|
||||
var mu sync.Mutex
|
||||
seen := map[string]int{}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for {
|
||||
conn, err := ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
go func(c net.Conn) {
|
||||
defer func() { _ = c.Close() }()
|
||||
greeting := `{"QMP":{"version":{"qemu":{"major":8,"minor":0}},"capabilities":[]}}` + "\n"
|
||||
if _, err := c.Write([]byte(greeting)); err != nil {
|
||||
return
|
||||
}
|
||||
dec := json.NewDecoder(c)
|
||||
for {
|
||||
var cmd struct {
|
||||
Execute string `json:"execute"`
|
||||
}
|
||||
if err := dec.Decode(&cmd); err != nil {
|
||||
return
|
||||
}
|
||||
mu.Lock()
|
||||
seen[cmd.Execute]++
|
||||
mu.Unlock()
|
||||
if _, err := c.Write([]byte(`{"return":{}}` + "\n")); err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}(conn)
|
||||
}
|
||||
}()
|
||||
t.Cleanup(func() {
|
||||
_ = ln.Close()
|
||||
<-done
|
||||
})
|
||||
|
||||
return sockPath, func(execute string) int {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return seen[execute]
|
||||
}
|
||||
}
|
||||
|
||||
// pausedGuest wires a manager holding one running VM whose QMP monitor answers,
|
||||
// which is the state a werror=stop pause leaves behind.
|
||||
func pausedGuest(t *testing.T) (*Manager, *VM, func(string) int) {
|
||||
t.Helper()
|
||||
sockPath, count := startRecordingQMPListener(t)
|
||||
client, err := qmp.NewQMPClient(sockPath)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = client.Conn.Close() })
|
||||
|
||||
instance := &VM{ID: "i-paused", Status: StateRunning, QMPClient: client}
|
||||
m := NewManager()
|
||||
m.Insert(instance)
|
||||
|
||||
return m, instance, count
|
||||
}
|
||||
|
||||
// The first attempts run at heartbeat rate, because a backend that comes back
|
||||
// quickly should return the guest quickly.
|
||||
func TestResumeAfterIOErrorRetriesEveryHeartbeatAtFirst(t *testing.T) {
|
||||
m, instance, count := pausedGuest(t)
|
||||
|
||||
for range qmpMaxIOErrorResumes {
|
||||
m.resumeAfterIOError(t.Context(), instance)
|
||||
}
|
||||
|
||||
assert.Equal(t, qmpMaxIOErrorResumes, count("cont"),
|
||||
"every heartbeat inside the budget must re-submit the held request")
|
||||
assert.Equal(t, qmpMaxIOErrorResumes, instance.Health.IOErrorResumes)
|
||||
assert.False(t, instance.Health.IOErrorSince.IsZero(), "the first pause must be stamped")
|
||||
assert.False(t, instance.Health.ImpairedSince.IsZero(),
|
||||
"spending the budget is what DescribeInstanceStatus reports as impaired")
|
||||
}
|
||||
|
||||
// Past the budget the guest is reported impaired, and the retries keep running
|
||||
// at heartbeat rate. They used to drop to one per ten heartbeats, which meant a
|
||||
// guest could sit paused for five minutes after its backend had come back --
|
||||
// an outage of our own making, to save one replayed I/O per thirty seconds.
|
||||
func TestResumeAfterIOErrorKeepsRetryingPastTheBudget(t *testing.T) {
|
||||
m, instance, count := pausedGuest(t)
|
||||
|
||||
const beyond = qmpMaxIOErrorResumes + 3*qmpIOErrorLogEvery
|
||||
for range beyond {
|
||||
m.resumeAfterIOError(t.Context(), instance)
|
||||
}
|
||||
|
||||
assert.Equal(t, beyond, count("cont"),
|
||||
"every heartbeat must re-submit the held request, however long the outage has run")
|
||||
assert.Equal(t, beyond, instance.Health.IOErrorResumes,
|
||||
"the count keeps rising, so the impaired report stays anchored to it")
|
||||
}
|
||||
|
||||
// A guest that resumes must start from zero, or a second unrelated outage
|
||||
// inherits a spent budget and is written off on its first heartbeat.
|
||||
func TestQMPSuccessClearsTheIOErrorBudget(t *testing.T) {
|
||||
m, instance, _ := pausedGuest(t)
|
||||
|
||||
for range qmpMaxIOErrorResumes + 1 {
|
||||
m.resumeAfterIOError(t.Context(), instance)
|
||||
}
|
||||
require.NotZero(t, instance.Health.IOErrorResumes)
|
||||
|
||||
m.recordQMPSuccess(instance)
|
||||
|
||||
assert.Zero(t, instance.Health.IOErrorResumes)
|
||||
assert.True(t, instance.Health.IOErrorSince.IsZero())
|
||||
}
|
||||
|
||||
+26
-18
@@ -490,6 +490,32 @@ func (m *Manager) terminateCleanup(instance *VM) {
|
||||
// that is the one step whose failure means data loss. QMP, PID-file, fw_cfg
|
||||
// and telemetry cleanup stay best-effort and are logged only.
|
||||
func (m *Manager) shutdownAndUnmount(instance *VM) error {
|
||||
m.shutdownQEMU(instance)
|
||||
|
||||
var sealErr error
|
||||
if m.deps.VolumeMounter != nil {
|
||||
if err := m.deps.VolumeMounter.Unmount(context.Background(), instance); err != nil {
|
||||
slog.Error("Volume unmount failed", "id", instance.ID, "err", err)
|
||||
sealErr = fmt.Errorf("%w for instance %s: %w", ErrVolumeSealFailed, instance.ID, err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, fw := range instance.Config.FwCfg {
|
||||
if err := os.Remove(fw.File); err != nil && !os.IsNotExist(err) {
|
||||
slog.Warn("Failed to remove fw_cfg temp file", "file", fw.File, "id", instance.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
removeTelemetryArtifacts(instance)
|
||||
|
||||
return sealErr
|
||||
}
|
||||
|
||||
// shutdownQEMU takes the guest process down and nothing else. Separated from
|
||||
// shutdownAndUnmount for the fence path, which must stop the guest without
|
||||
// unmounting: a fenced node's volumes belong to another node now, and an
|
||||
// unmount would seal this node's stale copy over theirs.
|
||||
func (m *Manager) shutdownQEMU(instance *VM) {
|
||||
if instance.QMPClient != nil {
|
||||
if _, err := sendQMPCommand(context.Background(), instance.QMPClient, qmp.QMPCommand{Execute: "system_powerdown"}, instance.ID); err != nil {
|
||||
slog.Warn("QMP system_powerdown failed (VM may already be stopped)",
|
||||
@@ -514,24 +540,6 @@ func (m *Manager) shutdownAndUnmount(instance *VM) error {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var sealErr error
|
||||
if m.deps.VolumeMounter != nil {
|
||||
if err := m.deps.VolumeMounter.Unmount(context.Background(), instance); err != nil {
|
||||
slog.Error("Volume unmount failed", "id", instance.ID, "err", err)
|
||||
sealErr = fmt.Errorf("%w for instance %s: %w", ErrVolumeSealFailed, instance.ID, err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, fw := range instance.Config.FwCfg {
|
||||
if err := os.Remove(fw.File); err != nil && !os.IsNotExist(err) {
|
||||
slog.Warn("Failed to remove fw_cfg temp file", "file", fw.File, "id", instance.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
removeTelemetryArtifacts(instance)
|
||||
|
||||
return sealErr
|
||||
}
|
||||
|
||||
// cleanupTapDevices removes the primary VPC tap, every extra ENI tap, and
|
||||
|
||||
+17
-7
@@ -32,6 +32,12 @@ type InstanceHealthState struct {
|
||||
QMPConsecutiveFailures int `json:"qmp_consecutive_failures"`
|
||||
LastQMPSuccess time.Time `json:"last_qmp_success,omitzero"`
|
||||
ImpairedSince time.Time `json:"impaired_since,omitzero"`
|
||||
|
||||
// IOErrorResumes counts consecutive cont attempts made against a VM paused
|
||||
// by werror=stop. It clears on the first running poll, so it measures a
|
||||
// backend that keeps failing rather than one that failed once.
|
||||
IOErrorResumes int `json:"io_error_resumes,omitempty"`
|
||||
IOErrorSince time.Time `json:"io_error_since,omitzero"`
|
||||
}
|
||||
|
||||
// ExtraENI describes an additional VPC network interface attached to a VM
|
||||
@@ -290,15 +296,15 @@ type Drive struct {
|
||||
Media string `json:"media"`
|
||||
ID string `json:"id"`
|
||||
Cache string `json:"cache,omitempty"`
|
||||
// Werror and Rerror set QEMU's block on-error policy. Left empty, QEMU
|
||||
// applies its default werror=enospc, which pauses the whole VM on an
|
||||
// out-of-space write rather than reporting it — so a guest never sees the
|
||||
// backend's ENOSPC. Setting "report" delivers the error to the guest,
|
||||
// letting its filesystem surface a clean out-of-space to userspace and stay
|
||||
// alive. Set on the drive: with if=none the virtio-blk device inherits the
|
||||
// BlockBackend's on-error policy.
|
||||
// Werror and Rerror set QEMU's block on-error policy. "stop" pauses the VM
|
||||
// and holds the failed request, so a transient backend loss is resumed with
|
||||
// cont instead of reaching the guest as EIO and corrupting its filesystem.
|
||||
Werror string `json:"werror,omitempty"`
|
||||
Rerror string `json:"rerror,omitempty"`
|
||||
// ReconnectDelay emits file.reconnect-delay, the seconds an NBD disconnect
|
||||
// pauses requests before failing them. The file. prefix is required: the
|
||||
// bare name reaches the format driver, which has no such option.
|
||||
ReconnectDelay int `json:"reconnect_delay,omitempty"`
|
||||
// Unit selects the pflash slot when If=="pflash": 0 is the CODE blob,
|
||||
// 1 is the per-VM VARS volume. Ignored for non-pflash drives.
|
||||
Unit int `json:"unit,omitempty"`
|
||||
@@ -513,6 +519,10 @@ func (cfg *Config) Execute() (*exec.Cmd, error) {
|
||||
opts = append(opts, fmt.Sprintf("rerror=%s", drive.Rerror))
|
||||
}
|
||||
|
||||
if drive.ReconnectDelay > 0 {
|
||||
opts = append(opts, fmt.Sprintf("file.reconnect-delay=%d", drive.ReconnectDelay))
|
||||
}
|
||||
|
||||
args = append(args, "-drive", strings.Join(opts, ","))
|
||||
}
|
||||
|
||||
|
||||
@@ -255,7 +255,7 @@ func TestExecute_Blockdev_UnixServer(t *testing.T) {
|
||||
}
|
||||
}
|
||||
require.Greater(t, blockdevIdx, -1, "-blockdev must be emitted")
|
||||
assert.Equal(t, "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/run/spinifex/nbd/nbd-vol-data-a.sock,export=", args[blockdevIdx+1])
|
||||
assert.Equal(t, "driver=nbd,node-name=nbd-vol-data-a,server.type=unix,server.path=/run/spinifex/nbd/nbd-vol-data-a.sock,export=,reconnect-delay=30", args[blockdevIdx+1])
|
||||
|
||||
deviceIdx := -1
|
||||
for i, a := range args {
|
||||
@@ -266,7 +266,7 @@ func TestExecute_Blockdev_UnixServer(t *testing.T) {
|
||||
}
|
||||
require.Greater(t, deviceIdx, -1)
|
||||
assert.Greater(t, deviceIdx, blockdevIdx, "-blockdev must precede the -device referencing it")
|
||||
assert.Equal(t, "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=report,rerror=report", args[deviceIdx+1])
|
||||
assert.Equal(t, "virtio-blk-pci,id=vdisk-vol-data-a,drive=nbd-vol-data-a,iothread=ioth-vol-data-a,serial=voldataa,bus=hotplug-ebs3,werror=stop,rerror=stop", args[deviceIdx+1])
|
||||
}
|
||||
|
||||
// TestExecute_Blockdev_InetServer asserts the TCP NBD form renders
|
||||
@@ -285,7 +285,7 @@ func TestExecute_Blockdev_InetServer(t *testing.T) {
|
||||
cmd, err := cfg.Execute()
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, "driver=nbd,node-name=nbd-vol-data-b,server.type=inet,server.host=10.0.0.5,server.port=10809,export=",
|
||||
assert.Equal(t, "driver=nbd,node-name=nbd-vol-data-b,server.type=inet,server.host=10.0.0.5,server.port=10809,export=,reconnect-delay=30",
|
||||
argValue(cmd.Args[1:], "-blockdev"))
|
||||
}
|
||||
|
||||
|
||||
@@ -166,11 +166,12 @@ func (m *Manager) AttachVolume(ctx context.Context, id, volumeID, device string)
|
||||
if _, err := sendQMPCommand(ctx, instance.QMPClient, qmp.QMPCommand{
|
||||
Execute: "blockdev-add",
|
||||
Arguments: map[string]any{
|
||||
"node-name": nodeName,
|
||||
"driver": "nbd",
|
||||
"server": serverArg,
|
||||
"export": "",
|
||||
"read-only": false,
|
||||
"node-name": nodeName,
|
||||
"driver": "nbd",
|
||||
"server": serverArg,
|
||||
"export": "",
|
||||
"read-only": false,
|
||||
"reconnect-delay": NBDReconnectDelaySeconds,
|
||||
},
|
||||
}, instance.ID); err != nil {
|
||||
slog.ErrorContext(ctx, "AttachVolume: QMP blockdev-add failed", "volumeId", volumeID, "err", err)
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
package vpcd
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/daemon"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
@@ -79,18 +81,19 @@ type recordLoader interface {
|
||||
LoadInstanceRecord(instanceID string) (*vm.InstanceRecord, error)
|
||||
}
|
||||
|
||||
// newInstanceRecordLoader opens the shared instance record space best-effort:
|
||||
// IMDS still serves local hits without it, and it only backs the fallback for
|
||||
// an instance that has moved off this node. Returns nil on any failure.
|
||||
func newInstanceRecordLoader(nc *nats.Conn) recordLoader {
|
||||
// newInstanceRecordLoader opens the shared instance record space, retrying while
|
||||
// JetStream comes up. Failing here is fatal by design: without it an instance
|
||||
// this node does not hold locally resolves to nothing, and IMDS serves that as
|
||||
// "no such instance" — a guest that boots with no key and no error anywhere.
|
||||
func newInstanceRecordLoader(ctx context.Context, nc *nats.Conn) (recordLoader, error) {
|
||||
jsm, err := daemon.NewJetStreamManager(nc, 1)
|
||||
if err != nil {
|
||||
slog.Warn("vpcd: JetStream manager unavailable, IMDS instance lookup will not fall back off this node", "err", err)
|
||||
return nil
|
||||
return nil, fmt.Errorf("jetstream manager: %w", err)
|
||||
}
|
||||
if err := jsm.InitKVBucket(); err != nil {
|
||||
slog.Warn("vpcd: instance-state bucket unavailable, IMDS instance lookup will not fall back off this node", "err", err)
|
||||
return nil
|
||||
if _, err := kvstore.OpenWithRetry(ctx, daemon.InstanceStateBucket, kvstore.DefaultOpenWindow,
|
||||
func(context.Context) (struct{}, error) { return struct{}{}, jsm.InitKVBucket() },
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return jsm
|
||||
return jsm, nil
|
||||
}
|
||||
|
||||
@@ -539,6 +539,10 @@ func launchService(cfg *Config) error {
|
||||
// vpcd is the composition root for both planes IMDS straddles: it wires the
|
||||
// on-disk VM state reader and the shared record space, so handlers/imds
|
||||
// itself never needs to import the compute plane beyond vm.VM.
|
||||
records, err := newInstanceRecordLoader(ctx, nc)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open instance record space: %w", err)
|
||||
}
|
||||
imdsSvc, err := handlers_imds.NewIMDSServiceImpl(
|
||||
ctx,
|
||||
nc,
|
||||
@@ -546,7 +550,7 @@ func launchService(cfg *Config) error {
|
||||
handlers_imds.NewNATSProfileLookup(nc),
|
||||
handlers_imds.NewNATSPublicKeyLookup(nc),
|
||||
newLocalVMStateReader(cfg.DataDir),
|
||||
newInstanceRecordLoader(nc),
|
||||
records,
|
||||
listTaps,
|
||||
cfg.NorthstarBaseDomain,
|
||||
cfg.NorthstarInternalDomain,
|
||||
|
||||
+5
-2
@@ -18,9 +18,9 @@ SUITE_TAG ?= $(notdir $(patsubst %/...,%,$(firstword $(PKG))))
|
||||
# BIN_DIR/<suite>.test so CI can ship binaries (not source + go toolchain) to
|
||||
# the remote VM.
|
||||
BIN_DIR ?= $(CURDIR)/_bin
|
||||
SUITES_BUILD ?= cert lb eks ecs single iam ecr bedrock gpu natuplink multinode storagegrowth partialblock rds quota diskperf
|
||||
SUITES_BUILD ?= cert lb eks ecs single iam ecr bedrock gpu natuplink multinode storagegrowth partialblock rds quota diskperf storagefault
|
||||
|
||||
.PHONY: e2e e2e-quota e2e-cert e2e-lb e2e-eks e2e-ecs e2e-single e2e-iam e2e-ecr e2e-bedrock e2e-multinode e2e-bm e2e-reboot e2e-tofu e2e-gpu e2e-natuplink e2e-storagegrowth e2e-partialblock e2e-diskperf e2e-rds e2e-all clean e2e-build
|
||||
.PHONY: e2e e2e-quota e2e-cert e2e-lb e2e-eks e2e-ecs e2e-single e2e-iam e2e-ecr e2e-bedrock e2e-multinode e2e-bm e2e-reboot e2e-tofu e2e-gpu e2e-natuplink e2e-storagegrowth e2e-partialblock e2e-diskperf e2e-rds e2e-storagefault e2e-all clean e2e-build
|
||||
|
||||
RUN_FLAG = $(if $(RUN),-run '$(RUN)',)
|
||||
|
||||
@@ -65,6 +65,9 @@ e2e-diskperf: ; $(MAKE) e2e PKG=./diskperf/... TIMEOUT=180m
|
||||
# most of a 30m package timeout before the tests still running get to report.
|
||||
e2e-rds: ; $(MAKE) e2e PKG=./rds/... TIMEOUT=45m
|
||||
e2e-quota: ; $(MAKE) e2e PKG=./quota/...
|
||||
# Freezes predastore, so its budget covers a deliberate outage plus the
|
||||
# recovery and re-verification that follow it, twice over for the two tests.
|
||||
e2e-storagefault: ; $(MAKE) e2e PKG=./storagefault/... TIMEOUT=45m
|
||||
e2e-all: ; $(MAKE) e2e
|
||||
|
||||
e2e-build:
|
||||
|
||||
@@ -288,7 +288,7 @@ func ensureDiskPerfInstance(t *testing.T, fix *Fixture) (instanceID string, tgt
|
||||
t.Helper()
|
||||
instType, arch := harness.DiscoverNanoInstanceType(t, fix.Harness)
|
||||
ami := harness.DiscoverUbuntuAMI(t, fix.Harness, arch)
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness, fix.ArtifactDir(t))
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness)
|
||||
vpc := harness.EnsureDefaultVPC(t, fix.Harness)
|
||||
harness.AuthorizeSSHIngress(t, fix.AWS, vpc.SGID)
|
||||
|
||||
|
||||
@@ -97,14 +97,13 @@ func discoverECSGPUNodeAMI(c *harness.AWSClient, vendor string) (amiID, reason s
|
||||
// in for asserting on the nvidia-smi table text directly.
|
||||
func TestECSGPUTaskExposure(t *testing.T) {
|
||||
fx := requireECSGPUFixture(t)
|
||||
artifacts := harness.ArtifactDir(t, fx.Env)
|
||||
|
||||
harness.Phase(t, "Ensuring ecsInstanceRole instance profile")
|
||||
ensureECSGPUInstanceProfile(t, fx.AWS)
|
||||
|
||||
vpc := harness.EnsureDefaultVPC(t, fx.Harness)
|
||||
harness.AuthorizeSSHIngress(t, fx.AWS, vpc.SGID)
|
||||
keyName, _ := harness.EnsureKeyPair(t, fx.Harness, artifacts)
|
||||
keyName, _ := harness.EnsureKeyPair(t, fx.Harness)
|
||||
|
||||
clusterName := fmt.Sprintf("%s-%d", ecsGPUClusterPfx, time.Now().Unix())
|
||||
harness.Phase(t, "Creating ECS cluster %q", clusterName)
|
||||
|
||||
@@ -24,7 +24,7 @@ func TestGPUPassthrough(t *testing.T) {
|
||||
harness.Phase(t, "GPU — Passthrough: launch, device visibility, driver access with %s", fix.GPUInstanceType)
|
||||
|
||||
vpc := harness.EnsureDefaultVPC(t, fix.Harness)
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness, fix.ArtifactDir(t))
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness)
|
||||
harness.AuthorizeSSHIngress(t, fix.AWS, vpc.SGID)
|
||||
|
||||
instanceID := launchGPUInstance(t, fix, vpc.SubnetID, []string{vpc.SGID}, keyName)
|
||||
@@ -72,7 +72,7 @@ func TestGPUPassthrough_Release(t *testing.T) {
|
||||
harness.Phase(t, "GPU — Release: GPU returns to pool after termination")
|
||||
|
||||
vpc := harness.EnsureDefaultVPC(t, fix.Harness)
|
||||
keyName, _ := harness.EnsureKeyPair(t, fix.Harness, fix.ArtifactDir(t))
|
||||
keyName, _ := harness.EnsureKeyPair(t, fix.Harness)
|
||||
sgIDs := []*string{aws.String(vpc.SGID)}
|
||||
|
||||
harness.Step(t, "launching first GPU instance")
|
||||
|
||||
@@ -264,13 +264,27 @@ func AttachVolumeWait(t *testing.T, c *AWSClient, volID, instanceID, device stri
|
||||
// DetachVolumeWait detaches volID and blocks until it reaches "available".
|
||||
func DetachVolumeWait(t *testing.T, c *AWSClient, volID string) {
|
||||
t.Helper()
|
||||
_, err := c.EC2.DetachVolume(&ec2.DetachVolumeInput{VolumeId: aws.String(volID)})
|
||||
if err != nil {
|
||||
t.Fatalf("detach-volume %s: %v", volID, err)
|
||||
// A volume that is already on its way to available is the outcome asked
|
||||
// for, not a failure. Teardown runs this after a terminate that releases
|
||||
// the volume itself, so refusing an already-released volume fails a test
|
||||
// whose assertions all passed.
|
||||
if _, err := c.EC2.DetachVolume(&ec2.DetachVolumeInput{VolumeId: aws.String(volID)}); err != nil {
|
||||
if !isIncorrectVolumeState(err) {
|
||||
t.Fatalf("detach-volume %s: %v", volID, err)
|
||||
}
|
||||
t.Logf("detach-volume %s: already released, waiting for available", volID)
|
||||
}
|
||||
WaitForVolumeState(t, c, volID, "available", WithPoll(500*time.Millisecond))
|
||||
}
|
||||
|
||||
// isIncorrectVolumeState reports whether err is the refusal a detach gets for a
|
||||
// volume that is not attached. It is the answer for one already released, and
|
||||
// for one still mid-detach, both of which are what a teardown wanted.
|
||||
func isIncorrectVolumeState(err error) bool {
|
||||
var aerr awserr.Error
|
||||
return asErr(err, &aerr) && aerr.Code() == "IncorrectState"
|
||||
}
|
||||
|
||||
// RegisterVolumeTeardown force-detaches then deletes volID at test end. It does
|
||||
// not call t.Fatal, so a cleanup hiccup can't mask the test's real result, but a
|
||||
// teardown that actually leaves the volume behind fails the test — a silent leak
|
||||
|
||||
@@ -67,6 +67,10 @@ type Fixture struct {
|
||||
cleanups map[string]struct{}
|
||||
sf singleflight.Group
|
||||
|
||||
// keyDir holds private keys for the process. Created on first use and
|
||||
// removed with the key pair it belongs to.
|
||||
keyDir string
|
||||
|
||||
// processCleanups holds teardown callbacks for process-mode fixtures.
|
||||
// Close() runs them LIFO. Unused (and untouched) when parent != nil.
|
||||
//
|
||||
@@ -319,14 +323,40 @@ func randHex(nBytes int) (string, error) {
|
||||
// EnsureKeyPair
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
// EnsureKeyPair creates (or returns the cached) named EC2 key pair. The PEM
|
||||
// is written to artifactsDir/<name>.pem with 0600. Returns (keyName, pemPath).
|
||||
func EnsureKeyPair(t *testing.T, fx *Fixture, artifactsDir string) (string, string) {
|
||||
// keyPairDir returns the process-scoped directory private keys are written
|
||||
// to, creating it on first use.
|
||||
//
|
||||
// Deliberately not a caller-supplied directory. The key pair lives as long as
|
||||
// the fixture, so a private key kept anywhere shorter-lived goes missing while
|
||||
// the registered public key is still being launched against.
|
||||
func (f *Fixture) keyPairDir() (string, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.keyDir != "" {
|
||||
return f.keyDir, nil
|
||||
}
|
||||
dir, err := os.MkdirTemp("", "spinifex-e2e-keys-")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("private key dir: %w", err)
|
||||
}
|
||||
f.keyDir = dir
|
||||
return dir, nil
|
||||
}
|
||||
|
||||
// EnsureKeyPair creates (or returns the cached) named EC2 key pair. The PEM is
|
||||
// written with 0600 into a directory owned by the fixture, and both are
|
||||
// removed together. Returns (keyName, pemPath).
|
||||
func EnsureKeyPair(t *testing.T, fx *Fixture) (string, string) {
|
||||
t.Helper()
|
||||
name := fx.resourceName("e2e-key")
|
||||
key := "keypair:" + name
|
||||
|
||||
pemPath := filepath.Join(artifactsDir, name+".pem")
|
||||
dir, err := fx.keyPairDir()
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureKeyPair: %v", err)
|
||||
}
|
||||
pemPath := filepath.Join(dir, name+".pem")
|
||||
|
||||
id, err := fx.ensureOnce(t, key, func() (string, func() error, error) {
|
||||
out, err := fx.EC2.CreateKeyPair(&ec2.CreateKeyPairInput{
|
||||
KeyName: aws.String(name),
|
||||
@@ -334,15 +364,18 @@ func EnsureKeyPair(t *testing.T, fx *Fixture, artifactsDir string) (string, stri
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("CreateKeyPair %s: %w", name, err)
|
||||
}
|
||||
if err := os.MkdirAll(artifactsDir, 0o750); err != nil {
|
||||
return "", nil, fmt.Errorf("mkdir %s: %w", artifactsDir, err)
|
||||
}
|
||||
if err := os.WriteFile(pemPath, []byte(aws.StringValue(out.KeyMaterial)), 0o600); err != nil {
|
||||
return "", nil, fmt.Errorf("write pem %s: %w", pemPath, err)
|
||||
}
|
||||
if err := assertOneKeyPair(fx, name); err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
fx.tagRunResources(aws.StringValue(out.KeyPairId))
|
||||
return aws.StringValue(out.KeyName), func() error {
|
||||
_, derr := fx.EC2.DeleteKeyPair(&ec2.DeleteKeyPairInput{KeyName: aws.String(name)})
|
||||
if rerr := os.RemoveAll(dir); rerr != nil && derr == nil {
|
||||
derr = rerr
|
||||
}
|
||||
return derr
|
||||
}, nil
|
||||
})
|
||||
@@ -352,6 +385,33 @@ func EnsureKeyPair(t *testing.T, fx *Fixture, artifactsDir string) (string, stri
|
||||
return id, pemPath
|
||||
}
|
||||
|
||||
// assertOneKeyPair fails when the cluster holds more than one key pair under
|
||||
// name, which it should not be able to.
|
||||
//
|
||||
// A create that lands twice leaves two pairs with one name and different
|
||||
// material. RunInstances then resolves the name to one of them while we hold
|
||||
// the private key of the other, and the only symptom is every guest in the run
|
||||
// rejecting the key at the SSH gate five minutes later, once per guest. Cheap
|
||||
// to check here, and it names the cause outright.
|
||||
func assertOneKeyPair(fx *Fixture, name string) error {
|
||||
out, err := fx.EC2.DescribeKeyPairs(&ec2.DescribeKeyPairsInput{
|
||||
KeyNames: []*string{aws.String(name)},
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("DescribeKeyPairs %s: %w", name, err)
|
||||
}
|
||||
if len(out.KeyPairs) == 1 {
|
||||
return nil
|
||||
}
|
||||
|
||||
got := make([]string, 0, len(out.KeyPairs))
|
||||
for _, kp := range out.KeyPairs {
|
||||
got = append(got, fmt.Sprintf("%s/%s", aws.StringValue(kp.KeyPairId), aws.StringValue(kp.KeyFingerprint)))
|
||||
}
|
||||
return fmt.Errorf("key pair %s resolves to %d pairs (%s); a launch cannot be told which one it got",
|
||||
name, len(out.KeyPairs), strings.Join(got, " "))
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// EnsureAMI
|
||||
// ----------------------------------------------------------------------------
|
||||
@@ -606,6 +666,10 @@ func EnsureSG(t *testing.T, fx *Fixture, vpcID, namePrefix string) string {
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
// InstanceSpec captures the inputs to RunInstances. UserData is optional.
|
||||
//
|
||||
// Scope names a group of callers that must not share a guest. It reaches the
|
||||
// memo key and nothing else, so two identical specs with different scopes get
|
||||
// an instance each — what a test needs when it damages the guest on purpose.
|
||||
type InstanceSpec struct {
|
||||
AMIID string
|
||||
InstanceType string
|
||||
@@ -613,6 +677,7 @@ type InstanceSpec struct {
|
||||
SubnetID string
|
||||
SGID string
|
||||
UserData string
|
||||
Scope string
|
||||
}
|
||||
|
||||
// EnsureInstance launches a single instance matching spec, polls to
|
||||
@@ -627,8 +692,8 @@ func EnsureInstance(t *testing.T, fx *Fixture, spec InstanceSpec) string {
|
||||
sum := sha256.Sum256([]byte(spec.UserData))
|
||||
udHash = hex.EncodeToString(sum[:8])
|
||||
}
|
||||
key := fmt.Sprintf("instance:%s:%s:%s:%s:%s:%s",
|
||||
spec.AMIID, spec.InstanceType, spec.KeyName, spec.SubnetID, spec.SGID, udHash)
|
||||
key := fmt.Sprintf("instance:%s:%s:%s:%s:%s:%s:%s",
|
||||
spec.AMIID, spec.InstanceType, spec.KeyName, spec.SubnetID, spec.SGID, udHash, spec.Scope)
|
||||
id, err := fx.ensureOnce(t, key, func() (string, func() error, error) {
|
||||
input := &ec2.RunInstancesInput{
|
||||
ImageId: aws.String(spec.AMIID),
|
||||
|
||||
@@ -4,7 +4,8 @@ package harness
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
@@ -32,6 +33,10 @@ type fakeEC2 struct {
|
||||
// the concurrent-callers test deterministically.
|
||||
holdCreate chan struct{}
|
||||
|
||||
// keyPairsForName is how many pairs DescribeKeyPairs reports for a name.
|
||||
// Zero means one, which is the only healthy answer.
|
||||
keyPairsForName atomic.Int64
|
||||
|
||||
// Volume teardown state. volState is what DescribeVolumes reports;
|
||||
// DetachVolume flips it to "available", modelling a node that releases the
|
||||
// volume once the attachment is forced off. deleteVolumeErr, when set, is
|
||||
@@ -43,6 +48,22 @@ type fakeEC2 struct {
|
||||
deleteVolumeErr error
|
||||
}
|
||||
|
||||
func (f *fakeEC2) DescribeKeyPairs(in *ec2.DescribeKeyPairsInput) (*ec2.DescribeKeyPairsOutput, error) {
|
||||
n := int(f.keyPairsForName.Load())
|
||||
if n == 0 {
|
||||
n = 1
|
||||
}
|
||||
out := &ec2.DescribeKeyPairsOutput{}
|
||||
for i := range n {
|
||||
out.KeyPairs = append(out.KeyPairs, &ec2.KeyPairInfo{
|
||||
KeyName: in.KeyNames[0],
|
||||
KeyPairId: aws.String(fmt.Sprintf("key-%d", i)),
|
||||
KeyFingerprint: aws.String(fmt.Sprintf("fingerprint-%d", i)),
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (f *fakeEC2) CreateVolume(*ec2.CreateVolumeInput) (*ec2.Volume, error) {
|
||||
return &ec2.Volume{VolumeId: aws.String("vol-fake")}, nil
|
||||
}
|
||||
@@ -122,20 +143,43 @@ func newFakeFixture(t *testing.T) (*Fixture, *fakeEC2) {
|
||||
}
|
||||
|
||||
// TestEnsureKeyPair_FirstCallCreates verifies first invocation calls
|
||||
// CreateKeyPair once and writes PEM to the artifact dir.
|
||||
// CreateKeyPair once and writes the PEM where it says it did.
|
||||
func TestEnsureKeyPair_FirstCallCreates(t *testing.T) {
|
||||
fx, ec2c := newFakeFixture(t)
|
||||
dir := t.TempDir()
|
||||
|
||||
name, pemPath := EnsureKeyPair(t, fx, dir)
|
||||
name, pemPath := EnsureKeyPair(t, fx)
|
||||
if name == "" {
|
||||
t.Fatalf("EnsureKeyPair returned empty name")
|
||||
}
|
||||
if got := ec2c.createKeyPair.Load(); got != 1 {
|
||||
t.Fatalf("CreateKeyPair calls = %d, want 1", got)
|
||||
}
|
||||
if filepath.Dir(pemPath) != dir {
|
||||
t.Fatalf("pemPath dir = %s, want %s", filepath.Dir(pemPath), dir)
|
||||
if _, err := os.Stat(pemPath); err != nil {
|
||||
t.Fatalf("pemPath %s: %v", pemPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEnsureKeyPair_PEMOutlivesAPassingTest is the regression. The private key
|
||||
// used to be written into the caller's per-test artifact directory, which is
|
||||
// removed when that test passes. Every later test then received a path to a
|
||||
// file that no longer existed while the key pair was still registered, so its
|
||||
// guest was launched against a public key nothing held the other half of and
|
||||
// SSH could never succeed.
|
||||
func TestEnsureKeyPair_PEMOutlivesAPassingTest(t *testing.T) {
|
||||
fx, _ := newFakeFixture(t)
|
||||
|
||||
var firstPath string
|
||||
t.Run("passes", func(it *testing.T) {
|
||||
_ = ArtifactDir(it, &Env{ArtifactDir: it.TempDir()})
|
||||
_, firstPath = EnsureKeyPair(it, fx)
|
||||
})
|
||||
|
||||
_, secondPath := EnsureKeyPair(t, fx)
|
||||
if secondPath != firstPath {
|
||||
t.Fatalf("second call returned %s, want the path the PEM was written to (%s)", secondPath, firstPath)
|
||||
}
|
||||
if _, err := os.Stat(secondPath); err != nil {
|
||||
t.Fatalf("private key vanished when the first test passed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -143,10 +187,9 @@ func TestEnsureKeyPair_FirstCallCreates(t *testing.T) {
|
||||
// return the cached name without re-calling CreateKeyPair.
|
||||
func TestEnsureKeyPair_SecondCallCached(t *testing.T) {
|
||||
fx, ec2c := newFakeFixture(t)
|
||||
dir := t.TempDir()
|
||||
|
||||
first, _ := EnsureKeyPair(t, fx, dir)
|
||||
second, _ := EnsureKeyPair(t, fx, dir)
|
||||
first, _ := EnsureKeyPair(t, fx)
|
||||
second, _ := EnsureKeyPair(t, fx)
|
||||
if first != second {
|
||||
t.Fatalf("second call returned %q, want cached %q", second, first)
|
||||
}
|
||||
@@ -160,7 +203,6 @@ func TestEnsureKeyPair_SecondCallCached(t *testing.T) {
|
||||
func TestEnsureKeyPair_ConcurrentCallers(t *testing.T) {
|
||||
fx, ec2c := newFakeFixture(t)
|
||||
ec2c.holdCreate = make(chan struct{})
|
||||
dir := t.TempDir()
|
||||
|
||||
const N = 10
|
||||
var wg sync.WaitGroup
|
||||
@@ -171,7 +213,7 @@ func TestEnsureKeyPair_ConcurrentCallers(t *testing.T) {
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
started <- struct{}{}
|
||||
n, _ := EnsureKeyPair(t, fx, dir)
|
||||
n, _ := EnsureKeyPair(t, fx)
|
||||
names[i] = n
|
||||
}(i)
|
||||
}
|
||||
@@ -342,11 +384,10 @@ func TestEnsureKeyPair_CleanupRunsOnce(t *testing.T) {
|
||||
t.Run("inner", func(it *testing.T) {
|
||||
var fx *Fixture
|
||||
fx, ec2c = newFakeFixture(it)
|
||||
dir := it.TempDir()
|
||||
|
||||
EnsureKeyPair(it, fx, dir)
|
||||
EnsureKeyPair(it, fx, dir)
|
||||
EnsureKeyPair(it, fx, dir)
|
||||
EnsureKeyPair(it, fx)
|
||||
EnsureKeyPair(it, fx)
|
||||
EnsureKeyPair(it, fx)
|
||||
})
|
||||
|
||||
if got := ec2c.deleteKeyPair.Load(); got != 1 {
|
||||
@@ -390,7 +431,7 @@ func TestPollUntil_TimeoutWrapsLastErr(t *testing.T) {
|
||||
//
|
||||
//nolint:unused,deadcode // exists only to fail compile when an Ensure* signature drifts
|
||||
func _ensureCompileCheck(t *testing.T, fx *Fixture) {
|
||||
_, _ = EnsureKeyPair(t, fx, "/tmp")
|
||||
_, _ = EnsureKeyPair(t, fx)
|
||||
_ = EnsureAMI(t, fx, AMISource{Existing: "ami-0"})
|
||||
_ = EnsureDefaultVPC(t, fx)
|
||||
_ = EnsureSubnet(t, fx, "vpc-0", "10.0.0.0/24", "ap-southeast-2a")
|
||||
@@ -400,3 +441,26 @@ func _ensureCompileCheck(t *testing.T, fx *Fixture) {
|
||||
_ = EnsureSnapshot(t, fx, SnapshotSpec{VolumeID: "vol-0"})
|
||||
_ = EnsureNATGateway(t, fx, "subnet-0", "")
|
||||
}
|
||||
|
||||
// TestAssertOneKeyPair_RejectsADuplicateName is the five-minute-per-guest
|
||||
// failure made immediate. Two pairs under one name means RunInstances picks one
|
||||
// and we hold the private key of the other, and the only other evidence is
|
||||
// every guest in the run refusing the key at the SSH gate.
|
||||
func TestAssertOneKeyPair_RejectsADuplicateName(t *testing.T) {
|
||||
fx, ec2c := newFakeFixture(t)
|
||||
|
||||
if err := assertOneKeyPair(fx, "e2e-key-abc"); err != nil {
|
||||
t.Fatalf("one pair under the name must be accepted: %v", err)
|
||||
}
|
||||
|
||||
ec2c.keyPairsForName.Store(2)
|
||||
err := assertOneKeyPair(fx, "e2e-key-abc")
|
||||
if err == nil {
|
||||
t.Fatal("two pairs under one name must fail: a launch cannot be told which one it got")
|
||||
}
|
||||
for _, want := range []string{"e2e-key-abc", "2 pairs", "fingerprint-0", "fingerprint-1"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("error %q does not name %q", err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,11 +6,15 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -173,3 +177,198 @@ func newPredastoreS3(host string) (*s3.S3, error) {
|
||||
}
|
||||
return s3.New(sess), nil
|
||||
}
|
||||
|
||||
// servingBucketPrefix names the per-check bucket AssertPredastoreServing owns.
|
||||
const servingBucketPrefix = "predastore-serving-"
|
||||
|
||||
// servingHoldRounds is how many consecutive round trips have to succeed before
|
||||
// the backend is called restored, and servingHoldGap is the pause between them.
|
||||
//
|
||||
// One is not enough. A gate can answer the round trip that arrives just after a
|
||||
// thaw and then lose its blob peers under the load that follows, which is a
|
||||
// backend that never recovered reported as one that did. Three rounds over
|
||||
// ~20s is long enough to cross that window and short enough to stay in budget.
|
||||
const (
|
||||
servingHoldRounds = 3
|
||||
servingHoldGap = 10 * time.Second
|
||||
)
|
||||
|
||||
// AssertPredastoreServing waits until every host round-trips an object, holds
|
||||
// that result across servingHoldRounds, and fails the test if it never does
|
||||
// within budget.
|
||||
//
|
||||
// Restoring the process a fault stopped is not the same as restoring the
|
||||
// service. A backend can answer systemctl, hold its listeners and still serve
|
||||
// nothing, so a fault injector that stops at the process leaves whatever runs
|
||||
// next to discover the difference.
|
||||
//
|
||||
// The round trip writes as well as reads, deliberately. Under RS(2,1) a read
|
||||
// reconstructs from a single surviving shard while a write needs two durable,
|
||||
// so a read-only probe passes on a cluster that cannot serve a guest at all.
|
||||
func AssertPredastoreServing(ctx context.Context, t *testing.T, hosts []string, budget time.Duration) {
|
||||
t.Helper()
|
||||
if len(hosts) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
bucket := fmt.Sprintf("%s%d", servingBucketPrefix, time.Now().UnixNano())
|
||||
payload := bytes.Repeat([]byte("s"), 64<<10)
|
||||
deadline := time.Now().Add(budget)
|
||||
|
||||
var lastErr error
|
||||
var held int
|
||||
for {
|
||||
lastErr = predastoreRoundTrip(ctx, hosts, bucket, payload)
|
||||
switch {
|
||||
case lastErr == nil:
|
||||
held++
|
||||
if held >= servingHoldRounds {
|
||||
Detail(t, "backendServingOn", strings.Join(hosts, ","))
|
||||
Detail(t, "backendServingHeldRounds", strconv.Itoa(held))
|
||||
AssertPredastoreReady(ctx, t, hosts)
|
||||
return
|
||||
}
|
||||
case held > 0:
|
||||
// A gate that served and then stopped is the failure this hold is
|
||||
// here to catch, so the count starts again rather than decaying.
|
||||
t.Logf("predastore served %d round(s) then failed, restarting the hold: %v", held, lastErr)
|
||||
held = 0
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) || ctx.Err() != nil {
|
||||
t.Errorf("predastore did not hold %d serving round trips through %v within %s (last error: %v)",
|
||||
servingHoldRounds, hosts, budget, lastErr)
|
||||
return
|
||||
}
|
||||
time.Sleep(servingHoldGap)
|
||||
}
|
||||
}
|
||||
|
||||
// predastoreAdminPort serves /healthz and /readyz on each host's cluster plane.
|
||||
const predastoreAdminPort = 8660
|
||||
|
||||
// AssertPredastoreReady checks each host's readiness probe and reports any
|
||||
// failed check by name.
|
||||
//
|
||||
// The round trip above proves this workstation can write through a gate. It
|
||||
// says nothing about which blob peers that gate can reach, and a gate one node
|
||||
// short of the write floor serves a small object today and fails a guest
|
||||
// tomorrow. /readyz is where that is answerable, so a host that serves it and
|
||||
// reports a failed check fails here; one that does not serve it is reported and
|
||||
// skipped, since the admin listener is optional configuration.
|
||||
func AssertPredastoreReady(ctx context.Context, t *testing.T, hosts []string) {
|
||||
t.Helper()
|
||||
for _, host := range hosts {
|
||||
failed, err := predastoreReadyFailures(ctx, predastoreReadyURL(host))
|
||||
if err != nil {
|
||||
t.Logf("predastore readiness on %s is unavailable, skipping: %v", host, err)
|
||||
continue
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
t.Errorf("predastore on %s reports failed readiness checks after the fault was cleared: %v",
|
||||
host, failed)
|
||||
continue
|
||||
}
|
||||
Detail(t, "backendReady", host)
|
||||
}
|
||||
}
|
||||
|
||||
// predastoreReadyURL is the readiness probe on a host's cluster plane.
|
||||
func predastoreReadyURL(host string) string {
|
||||
return fmt.Sprintf("http://%s:%d/readyz", host, predastoreAdminPort)
|
||||
}
|
||||
|
||||
// predastoreReadyFailures names the checks /readyz reports as failed. An
|
||||
// unready process is not itself an error here -- the failed check names are the
|
||||
// finding, and an empty list from a 503 would be the probe hiding one.
|
||||
func predastoreReadyFailures(ctx context.Context, url string) ([]string, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := (&http.Client{Timeout: 10 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
var body struct {
|
||||
Status string `json:"status"`
|
||||
Checks map[string]string `json:"checks"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
|
||||
return nil, fmt.Errorf("decode %s: %w", url, err)
|
||||
}
|
||||
if len(body.Checks) == 0 && body.Status != "ready" {
|
||||
return nil, fmt.Errorf("%s reported %q with no checks", url, body.Status)
|
||||
}
|
||||
|
||||
failed := make([]string, 0, len(body.Checks))
|
||||
for name, state := range body.Checks {
|
||||
if state != "ok" {
|
||||
failed = append(failed, name+"="+state)
|
||||
}
|
||||
}
|
||||
sort.Strings(failed)
|
||||
return failed, nil
|
||||
}
|
||||
|
||||
// predastoreRoundTrip puts and gets one object through every host, so a single
|
||||
// unusable node fails the check whichever gate it is reached through.
|
||||
func predastoreRoundTrip(ctx context.Context, hosts []string, bucket string, payload []byte) error {
|
||||
first, err := newPredastoreS3(hosts[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := first.CreateBucketWithContext(ctx, &s3.CreateBucketInput{
|
||||
Bucket: aws.String(bucket),
|
||||
}); err != nil && !isBucketExists(err) {
|
||||
return fmt.Errorf("create bucket %s on %s: %w", bucket, hosts[0], err)
|
||||
}
|
||||
|
||||
for _, host := range hosts {
|
||||
cli, err := newPredastoreS3(host)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
key := "serving/" + host
|
||||
if _, err := cli.PutObjectWithContext(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(bucket),
|
||||
Key: aws.String(key),
|
||||
Body: bytes.NewReader(payload),
|
||||
}); err != nil {
|
||||
return fmt.Errorf("put through %s: %w", host, err)
|
||||
}
|
||||
got, err := getObjectBytes(ctx, cli, bucket, key)
|
||||
if err != nil {
|
||||
return fmt.Errorf("get through %s: %w", host, err)
|
||||
}
|
||||
if !bytes.Equal(got, payload) {
|
||||
return fmt.Errorf("get through %s returned %d bytes, want %d", host, len(got), len(payload))
|
||||
}
|
||||
if _, err := cli.DeleteObjectWithContext(ctx, &s3.DeleteObjectInput{
|
||||
Bucket: aws.String(bucket),
|
||||
Key: aws.String(key),
|
||||
}); err != nil {
|
||||
return fmt.Errorf("delete through %s: %w", host, err)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := first.DeleteBucketWithContext(ctx, &s3.DeleteBucketInput{
|
||||
Bucket: aws.String(bucket),
|
||||
}); err != nil {
|
||||
return fmt.Errorf("delete bucket %s: %w", bucket, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// isBucketExists reports whether err is the S3 error for a bucket the caller
|
||||
// already owns, which a retried round trip is expected to hit.
|
||||
func isBucketExists(err error) bool {
|
||||
var aerr awserr.Error
|
||||
if errors.As(err, &aerr) {
|
||||
return aerr.Code() == s3.ErrCodeBucketAlreadyExists ||
|
||||
aerr.Code() == s3.ErrCodeBucketAlreadyOwnedByYou
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
//go:build e2e
|
||||
|
||||
package harness
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws/awserr"
|
||||
)
|
||||
|
||||
// readyzStub serves one canned /readyz response.
|
||||
func readyzStub(t *testing.T, status int, body string) string {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_, _ = w.Write([]byte(body))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv.URL + "/readyz"
|
||||
}
|
||||
|
||||
// TestPredastoreReadyFailures_NamesTheUnreachableBlobNode is the check the
|
||||
// backend-outage test was missing. A gate that has lost a blob peer still
|
||||
// reconstructs every read, so the round trip passes; the failed check name is
|
||||
// the only place the missing peer is visible.
|
||||
func TestPredastoreReadyFailures_NamesTheUnreachableBlobNode(t *testing.T) {
|
||||
url := readyzStub(t, http.StatusServiceUnavailable,
|
||||
`{"status":"unready","checks":{"meta_leader":"ok","blob_nodes":"failed","blob_node_5":"failed","blob_node_6":"ok"}}`)
|
||||
|
||||
failed, err := predastoreReadyFailures(context.Background(), url)
|
||||
if err != nil {
|
||||
t.Fatalf("predastoreReadyFailures: %v", err)
|
||||
}
|
||||
if got, want := strings.Join(failed, ","), "blob_node_5=failed,blob_nodes=failed"; got != want {
|
||||
t.Errorf("failed checks = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPredastoreReadyFailures_ReadyReportsNothing(t *testing.T) {
|
||||
url := readyzStub(t, http.StatusOK,
|
||||
`{"status":"ready","checks":{"meta_leader":"ok","blob_nodes":"ok","blob_node_5":"ok"}}`)
|
||||
|
||||
failed, err := predastoreReadyFailures(context.Background(), url)
|
||||
if err != nil {
|
||||
t.Fatalf("predastoreReadyFailures: %v", err)
|
||||
}
|
||||
if len(failed) != 0 {
|
||||
t.Errorf("failed checks = %v, want none", failed)
|
||||
}
|
||||
}
|
||||
|
||||
// An unready answer carrying no checks tells us nothing about why, and
|
||||
// reporting it as "no failures" would be the probe hiding the finding.
|
||||
func TestPredastoreReadyFailures_UnreadyWithNoChecksIsAnError(t *testing.T) {
|
||||
url := readyzStub(t, http.StatusServiceUnavailable, `{"status":"unready","checks":{}}`)
|
||||
|
||||
if _, err := predastoreReadyFailures(context.Background(), url); err == nil {
|
||||
t.Error("an unready answer with no checks must not read as no failures")
|
||||
}
|
||||
}
|
||||
|
||||
// A host that serves no admin listener is a configuration state, not a
|
||||
// failure, so the caller has to be able to tell the two apart.
|
||||
func TestPredastoreReadyFailures_NoListenerIsAnError(t *testing.T) {
|
||||
if _, err := predastoreReadyFailures(context.Background(), "http://127.0.0.1:1/readyz"); err == nil {
|
||||
t.Error("a host with no admin listener must report an error, not readiness")
|
||||
}
|
||||
}
|
||||
|
||||
// TestIsIncorrectVolumeState covers the one refusal teardown tolerates. A
|
||||
// volume already on its way to available is the outcome the detach wanted, and
|
||||
// failing on it fails a test whose assertions all passed.
|
||||
func TestIsIncorrectVolumeState(t *testing.T) {
|
||||
if !isIncorrectVolumeState(awserr.New("IncorrectState", "not attached", nil)) {
|
||||
t.Error("a volume that is not attached must read as already released")
|
||||
}
|
||||
if isIncorrectVolumeState(awserr.New("VolumeInUse", "attached elsewhere", nil)) {
|
||||
t.Error("VolumeInUse is a real failure and must not be swallowed")
|
||||
}
|
||||
if isIncorrectVolumeState(errors.New("connection refused")) {
|
||||
t.Error("a non-AWS error must not read as already released")
|
||||
}
|
||||
}
|
||||
@@ -111,7 +111,7 @@ func RDSClientVMIn(t *testing.T, c *AWSClient, fx *Fixture, env *Env, p RDSClien
|
||||
// The suite-level artifact root, not the calling test's subdirectory: the key
|
||||
// outlives the test that first asked for the VM, and ArtifactDir prunes a
|
||||
// passing test's directory — taking the PEM the next caller needs with it.
|
||||
keyName, keyPath := EnsureKeyPair(t, fx, env.ArtifactDir)
|
||||
keyName, keyPath := EnsureKeyPair(t, fx)
|
||||
|
||||
instanceID := EnsureInstance(t, fx, InstanceSpec{
|
||||
AMIID: ami,
|
||||
|
||||
@@ -32,7 +32,7 @@ func needAMI(t *testing.T, fix *Fixture) string {
|
||||
// needKeyPair returns the memoized primary EC2 key pair name + PEM path.
|
||||
func needKeyPair(t *testing.T, fix *Fixture) (name, pemPath string) {
|
||||
t.Helper()
|
||||
return harness.EnsureKeyPair(t, fix.Harness, fix.TmpDir)
|
||||
return harness.EnsureKeyPair(t, fix.Harness)
|
||||
}
|
||||
|
||||
// needInstance bootstraps the suite's primary running instance and returns
|
||||
|
||||
@@ -42,7 +42,7 @@ func needAMI(t *testing.T, fix *Fixture, arch string) string {
|
||||
// pruned by t.Cleanup as soon as whichever test first calls this one passes.
|
||||
func needKeyPair(t *testing.T, fix *Fixture) (name, pemPath string) {
|
||||
t.Helper()
|
||||
return harness.EnsureKeyPair(t, fix.Harness, fix.TmpDir)
|
||||
return harness.EnsureKeyPair(t, fix.Harness)
|
||||
}
|
||||
|
||||
// Package-scoped trio. Whichever Test* runs first triggers the launch; every other caller of needInstanceTrio reuses the same IDs.
|
||||
|
||||
@@ -320,7 +320,7 @@ func phaseInstanceEgress(t *testing.T, fix *fixture, def harness.VPCInfo) egress
|
||||
|
||||
instType, arch := harness.DiscoverNanoInstanceType(t, fix.harness)
|
||||
amiID := harness.DiscoverUbuntuAMI(t, fix.harness, arch)
|
||||
keyName, _ := harness.EnsureKeyPair(t, fix.harness, fix.artifacts)
|
||||
keyName, _ := harness.EnsureKeyPair(t, fix.harness)
|
||||
harness.Detail(t, "type", instType, "ami", amiID, "key", keyName)
|
||||
|
||||
harness.Step(t, "run-instances (default subnet, egress-probe user-data)")
|
||||
|
||||
@@ -253,7 +253,7 @@ func ensurePartialBlockInstance(t *testing.T, fix *Fixture) (instanceID string,
|
||||
t.Helper()
|
||||
instType, arch := harness.DiscoverNanoInstanceType(t, fix.Harness)
|
||||
ami := harness.DiscoverUbuntuAMI(t, fix.Harness, arch)
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness, fix.ArtifactDir(t))
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness)
|
||||
vpc := harness.EnsureDefaultVPC(t, fix.Harness)
|
||||
harness.AuthorizeSSHIngress(t, fix.AWS, vpc.SGID)
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ func needAMI(t *testing.T, fix *Fixture) string {
|
||||
// needKeyPair returns the memoized primary EC2 key pair name + PEM path.
|
||||
func needKeyPair(t *testing.T, fix *Fixture) (name, pemPath string) {
|
||||
t.Helper()
|
||||
return harness.EnsureKeyPair(t, fix.Harness, fix.TmpDir)
|
||||
return harness.EnsureKeyPair(t, fix.Harness)
|
||||
}
|
||||
|
||||
// customAMIName is the well-known AMI name used by CreateImage tests.
|
||||
|
||||
@@ -0,0 +1,557 @@
|
||||
//go:build e2e
|
||||
|
||||
package storagefault
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/tests/e2e/harness"
|
||||
)
|
||||
|
||||
const (
|
||||
// faultDevice is the guest-visible attach point for the workload volume.
|
||||
faultDevice = "/dev/sdf"
|
||||
|
||||
// faultVolumeSizeGiB sizes the scratch volume. The working set is smaller
|
||||
// still: this test is about what happens to the filesystem, not about
|
||||
// moving enough data to measure anything.
|
||||
faultVolumeSizeGiB = 2
|
||||
|
||||
// faultMount is where the guest mounts the workload filesystem.
|
||||
faultMount = "/mnt/fault"
|
||||
|
||||
// verifyMiB is the region written, flushed and later re-verified. The
|
||||
// verdict is taken from this and nothing else.
|
||||
verifyMiB = 256
|
||||
|
||||
// loadBlocks is how many 64 KiB blocks each pass of the sustained load
|
||||
// writes: 512 MiB, to a file of its own so an interrupted load write is
|
||||
// never mistaken for lost data.
|
||||
loadBlocks = 8192
|
||||
|
||||
// rootLoadDir is where the root-filesystem load runs. The user's home is on
|
||||
// the root volume; /tmp is tmpfs on this image, so a load there would move
|
||||
// no bytes on vda and the root would never see the fault at all.
|
||||
rootLoadDir = "/home/ubuntu/.storagefault-load"
|
||||
|
||||
// verifyFile holds the region the verdict is taken from, and the pid and log
|
||||
// files track the two loads. All are on paths the guest always has.
|
||||
verifyFile = faultMount + "/verify.dat"
|
||||
loadPID = "/tmp/storagefault-load.pid"
|
||||
loadLog = "/tmp/storagefault-load.log"
|
||||
rootLoadPID = "/tmp/storagefault-rootload.pid"
|
||||
rootLoadLog = "/tmp/storagefault-rootload.log"
|
||||
|
||||
// sha256HexLen is the length of a sha256 digest in hex, used to reject a
|
||||
// truncated or error-laden reading rather than compare it.
|
||||
sha256HexLen = 64
|
||||
|
||||
// rootLoadBlocks is how many 32 KiB blocks each root-load file gets, kept
|
||||
// deliberately small: the point is journal traffic on the root volume, not
|
||||
// filling a nano guest's disk.
|
||||
rootLoadBlocks = 256
|
||||
|
||||
// rootDevice is the guest's root block device, whose writes are the ones
|
||||
// that decide whether the fault could reach the root filesystem.
|
||||
rootDevice = "vda"
|
||||
|
||||
// rootWriteSample is the gap between the two sectors_written readings that
|
||||
// prove the root load is live. Long enough to clear a quiet moment between
|
||||
// fio's own fsyncs, short enough not to eat into the pre-fault settle.
|
||||
rootWriteSample = 3 * time.Second
|
||||
|
||||
// freezeHold is how long predastore stays frozen. It is derived, not tuned:
|
||||
// viperblock tolerates maxDrainFailures=10 consecutive drain failures, and a
|
||||
// drain against a frozen backend returns at the 30s backend timeout, so the
|
||||
// earliest a write can fail is ~300s. Below that the test measures the retry
|
||||
// budget rather than the outage and passes for the wrong reason.
|
||||
freezeHold = 420 * time.Second
|
||||
|
||||
// loadSettle is how long to let the load establish real in-flight I/O before
|
||||
// the fault. A freeze that lands before the first write proves nothing.
|
||||
loadSettle = 20 * time.Second
|
||||
|
||||
// recoverySettle is how long to wait after the thaw for the backend to
|
||||
// answer again before judging the guest.
|
||||
recoverySettle = 30 * time.Second
|
||||
|
||||
// guestRecoveryTimeout bounds the wait for a guest paused on an I/O error
|
||||
// to answer again. The resume rides the control plane's heartbeat, so this
|
||||
// covers a poll period plus the guest catching up on held requests.
|
||||
guestRecoveryTimeout = 3 * time.Minute
|
||||
)
|
||||
|
||||
// corruptionSignatures are the kernel messages that mean the guest's
|
||||
// filesystem took the fault as data loss rather than as a pause. Each is the
|
||||
// guest reacting to an error it should never have been shown.
|
||||
var corruptionSignatures = []*regexp.Regexp{
|
||||
regexp.MustCompile(`EXT4-fs error`),
|
||||
regexp.MustCompile(`Aborting journal`),
|
||||
regexp.MustCompile(`Remounting filesystem read-only`),
|
||||
regexp.MustCompile(`I/O error, dev \w+`),
|
||||
}
|
||||
|
||||
// TestGuestSurvivesBackendOutage freezes predastore under a running fio
|
||||
// workload and asserts the guest comes back to an intact, still-writable
|
||||
// filesystem.
|
||||
//
|
||||
// Expected to fail against the current engine: every volume is realized with
|
||||
// werror=report, so QEMU hands the guest EIO, ext4 aborts its journal and
|
||||
// remounts read-only. That failure is the reproduction.
|
||||
func TestGuestSurvivesBackendOutage(t *testing.T) {
|
||||
fix := requireStorageFaultFixture(t)
|
||||
requireNoneFrozen(t, fix)
|
||||
|
||||
instanceID, tgt := ensureFaultInstance(t, fix)
|
||||
hostNode := harness.InstanceHostingNode(t, fix.Cluster, instanceID)
|
||||
if hostNode == nil && len(fix.Cluster.Nodes) > 1 {
|
||||
t.Fatalf("cannot identify the node hosting %s, so the freeze set cannot spare it", instanceID)
|
||||
}
|
||||
|
||||
az := harness.DiscoverDefaultAZ(t, fix.Harness)
|
||||
volID := harness.EnsureVolume(t, fix.Harness, az, faultVolumeSizeGiB)
|
||||
|
||||
before := harness.GuestDiskSet(t, tgt)
|
||||
harness.AttachVolumeWait(t, fix.AWS, volID, instanceID, faultDevice)
|
||||
dev := harness.WaitForNewGuestDisk(t, tgt, before, 90*time.Second)
|
||||
t.Cleanup(func() { harness.DetachVolumeWait(t, fix.AWS, volID) })
|
||||
|
||||
prepareWorkloadFilesystem(t, tgt, dev)
|
||||
quiesceWorkload(t, tgt)
|
||||
|
||||
freezeSet, why := freezeSetFor(fix, hostNode)
|
||||
harness.Detail(t, "instance", instanceID, "volume", volID, "guest_device", dev,
|
||||
"host_node", nodeName(hostNode), "freeze", why)
|
||||
|
||||
harness.Step(t, "writing the %d MiB verifiable region and flushing it", verifyMiB)
|
||||
digest := writeVerifiableRegion(t, tgt)
|
||||
|
||||
loadRuntime := loadSettle + freezeHold + recoverySettle
|
||||
harness.Step(t, "starting sustained load for %s, on the volume and the root filesystem", loadRuntime)
|
||||
startLoad(t, tgt, loadRuntime)
|
||||
startRootLoad(t, tgt, loadRuntime)
|
||||
time.Sleep(loadSettle)
|
||||
if !loadRunning(t, tgt) {
|
||||
t.Fatalf("the load exited before the fault was injected, so nothing was under load: %s",
|
||||
loadLogTail(t, tgt))
|
||||
}
|
||||
assertRootLoadIsWriting(t, tgt)
|
||||
|
||||
harness.Step(t, "freezing predastore on %s for %s", nodeNames(freezeSet), freezeHold)
|
||||
freezePredastore(t, fix, freezeSet)
|
||||
|
||||
observeDuringOutage(t, fix, instanceID, tgt)
|
||||
|
||||
harness.Step(t, "thawing predastore and waiting %s for the backend to answer", recoverySettle)
|
||||
thawPredastore(t, fix, freezeSet)
|
||||
awaitGuestRecovered(t, tgt)
|
||||
|
||||
// Evidence is gathered from the console as well as the guest because a
|
||||
// guest whose root went read-only may not answer SSH, and its silence is
|
||||
// the very outcome under test.
|
||||
console, consoleErr := harness.InstanceConsole(fix.AWS, instanceID)
|
||||
if consoleErr != nil {
|
||||
t.Logf("console unavailable (guest evidence only): %v", consoleErr)
|
||||
}
|
||||
guestDmesg := bestEffort(tgt, "sudo dmesg | tail -n 400")
|
||||
evidence := console + "\n" + guestDmesg
|
||||
|
||||
assertNoCorruption(t, evidence, instanceID, fix)
|
||||
assertStillWritable(t, tgt, dev)
|
||||
assertWorkloadIntegrity(t, tgt, digest)
|
||||
}
|
||||
|
||||
// prepareWorkloadFilesystem lays an ext4 filesystem on the scratch device and
|
||||
// mounts it. errors=remount-ro is ext4's default and is left in place on
|
||||
// purpose: it is the guest's own protection, and the test is about whether the
|
||||
// guest is ever given a reason to invoke it.
|
||||
func prepareWorkloadFilesystem(t *testing.T, tgt harness.SSHTarget, dev string) {
|
||||
t.Helper()
|
||||
cmd := fmt.Sprintf(
|
||||
"sudo mkfs.ext4 -q -F /dev/%s && sudo mkdir -p %s && sudo mount -o errors=remount-ro /dev/%s %s && sudo chmod 777 %s",
|
||||
dev, faultMount, dev, faultMount, faultMount)
|
||||
if out, err := harness.GuestExecTimeout(tgt, cmd, 3*time.Minute); err != nil {
|
||||
t.Fatalf("prepare workload filesystem on /dev/%s: %v\n%s", dev, err, out)
|
||||
}
|
||||
}
|
||||
|
||||
// The workload is built from dd, sha256sum and a shell loop, all of which are
|
||||
// in the base image. That is a deliberate choice over fio: this suite needs
|
||||
// real in-flight I/O and a byte-exact integrity check, not a benchmark, and
|
||||
// installing a package over the internet inside the guest was by some distance
|
||||
// the least reliable step in the whole run.
|
||||
|
||||
// loadLoop keeps real I/O in flight across the fault so it lands on a busy
|
||||
// device. It rewrites one file in place, which is deliberately not verified: a
|
||||
// time-bounded loop is cut off mid-write by design, and judging those blocks
|
||||
// would report the interruption as corruption.
|
||||
func loadLoop(runtime time.Duration) string {
|
||||
return fmt.Sprintf(
|
||||
"end=$(( $(date +%%s) + %d )); "+
|
||||
"while [ \"$(date +%%s)\" -lt $end ]; do "+
|
||||
"dd if=/dev/zero of=%s/load.dat bs=64k count=%d oflag=direct conv=notrunc "+
|
||||
"status=none || exit 1; done",
|
||||
int(runtime.Seconds()), faultMount, loadBlocks)
|
||||
}
|
||||
|
||||
// rootLoadLoop drives the root filesystem so the fault lands on vda as well as
|
||||
// on the attached volume. conv=fsync forces a flush after every file, which is
|
||||
// the ext4 journal commit that aborts the journal when it fails — a failed read
|
||||
// of a clean page is merely retried and proves much less.
|
||||
func rootLoadLoop(runtime time.Duration) string {
|
||||
return fmt.Sprintf(
|
||||
"end=$(( $(date +%%s) + %d )); i=0; "+
|
||||
"while [ \"$(date +%%s)\" -lt $end ]; do "+
|
||||
"dd if=/dev/zero of=%s/load.$((i%%4)) bs=32k count=%d conv=fsync "+
|
||||
"status=none || exit 1; i=$((i+1)); done",
|
||||
int(runtime.Seconds()), rootLoadDir, rootLoadBlocks)
|
||||
}
|
||||
|
||||
// startRootLoad launches the root-filesystem load detached and removes its
|
||||
// files afterwards, so a nano guest's 8 GiB root is not left consumed.
|
||||
func startRootLoad(t *testing.T, tgt harness.SSHTarget, runtime time.Duration) {
|
||||
t.Helper()
|
||||
t.Cleanup(func() {
|
||||
_, _ = harness.GuestExecTimeout(tgt, "sudo rm -rf "+rootLoadDir, 60*time.Second)
|
||||
})
|
||||
|
||||
cmd := fmt.Sprintf("mkdir -p %s && rm -f %s; "+
|
||||
"sudo setsid bash -c 'echo $$ > %s; %s' >%s 2>&1 </dev/null &",
|
||||
rootLoadDir, rootLoadPID, rootLoadPID, rootLoadLoop(runtime), rootLoadLog)
|
||||
if out, err := harness.GuestExec(tgt, cmd); err != nil {
|
||||
t.Fatalf("start root load: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
|
||||
// writeVerifiableRegion lays down the region the verdict is taken from and
|
||||
// returns its digest. It completes before any fault is injected, so every byte
|
||||
// was acknowledged and flushed — a mismatch afterwards is lost data and never
|
||||
// an interrupted write. /dev/urandom rather than zeroes so a hole or a
|
||||
// short read cannot pass by looking like the data it replaced.
|
||||
func writeVerifiableRegion(t *testing.T, tgt harness.SSHTarget) string {
|
||||
t.Helper()
|
||||
write := fmt.Sprintf(
|
||||
"sudo dd if=/dev/urandom of=%s bs=1M count=%d oflag=direct conv=fsync status=none && "+
|
||||
"sync && sudo sha256sum %s | cut -d' ' -f1",
|
||||
verifyFile, verifyMiB, verifyFile)
|
||||
out, err := harness.GuestExecTimeout(tgt, write, 10*time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("could not write the verifiable region before the fault: %v\n%s", err, out)
|
||||
}
|
||||
digest := strings.TrimSpace(out)
|
||||
if len(digest) != sha256HexLen {
|
||||
t.Fatalf("expected a sha256 of the verifiable region, got %q", out)
|
||||
}
|
||||
return digest
|
||||
}
|
||||
|
||||
// startLoad launches the sustained workload detached, so it keeps running while
|
||||
// the test injects the fault. setsid survives the SSH session closing, and the
|
||||
// loop records its own pid because the dd processes it spawns come and go.
|
||||
func startLoad(t *testing.T, tgt harness.SSHTarget, runtime time.Duration) {
|
||||
t.Helper()
|
||||
cmd := fmt.Sprintf("rm -f %s; sudo setsid bash -c 'echo $$ > %s; %s' >%s 2>&1 </dev/null &",
|
||||
loadPID, loadPID, loadLoop(runtime), loadLog)
|
||||
if out, err := harness.GuestExec(tgt, cmd); err != nil {
|
||||
t.Fatalf("start load: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
|
||||
// quiesceWorkload stops the load and unmounts the fault volume before it is
|
||||
// detached. This is teardown, not part of any assertion, and it matters:
|
||||
// once a drive is deleted the block layer returns EIO to the guest regardless
|
||||
// of werror/rerror, so hot-detaching a mounted filesystem with writes still in
|
||||
// flight damages the guest whatever on-error policy is configured. A test that
|
||||
// did that would be injecting a second, unrelated fault during its own
|
||||
// cleanup, and the guest it left behind would fail the next subtest.
|
||||
//
|
||||
// Registered as a cleanup after the detach so it runs before it: t.Cleanup is
|
||||
// LIFO. Every step is best effort, because a guest that is already unreachable
|
||||
// is the failure under test rather than a reason to fail here.
|
||||
func quiesceWorkload(t *testing.T, tgt harness.SSHTarget) {
|
||||
t.Helper()
|
||||
t.Cleanup(func() {
|
||||
// Kill the loop first, then any dd it is currently inside: the other
|
||||
// order lets the loop start a replacement.
|
||||
bestEffort(tgt, fmt.Sprintf(
|
||||
"sudo kill $(cat %s) 2>/dev/null; sudo pkill -x dd 2>/dev/null; true", loadPID))
|
||||
for range 15 {
|
||||
if !loadRunning(t, tgt) {
|
||||
break
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
bestEffort(tgt, fmt.Sprintf("sync; sudo umount %s 2>/dev/null; true", faultMount))
|
||||
})
|
||||
}
|
||||
|
||||
// loadRunning reports whether the load loop is still alive. It tracks the loop
|
||||
// rather than dd, which exits and restarts on every pass.
|
||||
func loadRunning(t *testing.T, tgt harness.SSHTarget) bool {
|
||||
t.Helper()
|
||||
out := bestEffort(tgt, fmt.Sprintf(
|
||||
"sudo kill -0 $(cat %s 2>/dev/null) 2>/dev/null && echo RUNNING || echo GONE", loadPID))
|
||||
return strings.Contains(out, "RUNNING")
|
||||
}
|
||||
|
||||
func loadLogTail(t *testing.T, tgt harness.SSHTarget) string {
|
||||
t.Helper()
|
||||
return bestEffort(tgt, "sudo tail -n 40 "+loadLog)
|
||||
}
|
||||
|
||||
// observeDuringOutage records what the control plane says while the backend is
|
||||
// down. These are logged rather than asserted mid-flight: the assertions that
|
||||
// decide the test are made after recovery, where they are not racing a
|
||||
// transient.
|
||||
func observeDuringOutage(t *testing.T, fix *Fixture, instanceID string, tgt harness.SSHTarget) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(freezeHold)
|
||||
for time.Now().Before(deadline) {
|
||||
time.Sleep(15 * time.Second)
|
||||
alive := "unreachable"
|
||||
if _, err := harness.GuestExecTimeout(tgt, "true", 10*time.Second); err == nil {
|
||||
alive = "reachable"
|
||||
}
|
||||
t.Logf("during outage: guest_ssh=%s instance_state=%s", alive, instanceStateOf(fix, instanceID))
|
||||
}
|
||||
}
|
||||
|
||||
// assertNoCorruption is the assertion this suite exists for. A guest that was
|
||||
// paused for the outage shows none of these; a guest that was handed EIO shows
|
||||
// several.
|
||||
func assertNoCorruption(t *testing.T, evidence, instanceID string, fix *Fixture) {
|
||||
t.Helper()
|
||||
var hits []string
|
||||
for _, re := range corruptionSignatures {
|
||||
if m := re.FindString(evidence); m != "" {
|
||||
hits = append(hits, m)
|
||||
}
|
||||
}
|
||||
if len(hits) == 0 {
|
||||
harness.Step(t, "no corruption signatures in the guest console or dmesg")
|
||||
return
|
||||
}
|
||||
|
||||
harness.DumpInstanceConsole(t, fix.AWS, instanceID, fix.ArtifactDir(t), "storagefault-console")
|
||||
t.Errorf("the guest was shown I/O errors and reacted to them: %s\n"+
|
||||
"This is the bug: the backend outage reached the guest as EIO instead of pausing it, "+
|
||||
"so ext4 aborted and protected itself by going read-only. The volume is now only as "+
|
||||
"consistent as the journal managed to be.", strings.Join(hits, ", "))
|
||||
}
|
||||
|
||||
// assertStillWritable checks the workload filesystem is mounted rw. A guest
|
||||
// that survived the outage correctly needs no remount to keep working.
|
||||
func assertStillWritable(t *testing.T, tgt harness.SSHTarget, dev string) {
|
||||
t.Helper()
|
||||
mounts, err := harness.GuestExecTimeout(tgt, "cat /proc/mounts", 30*time.Second)
|
||||
if err != nil {
|
||||
t.Errorf("cannot read /proc/mounts, so the guest did not come back usable: %v", err)
|
||||
return
|
||||
}
|
||||
for _, line := range strings.Split(mounts, "\n") {
|
||||
if !strings.Contains(line, faultMount) {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case strings.Contains(line, " ro,"), strings.HasSuffix(strings.TrimSpace(line), " ro"):
|
||||
t.Errorf("%s is mounted read-only after recovery: %q", faultMount, strings.TrimSpace(line))
|
||||
default:
|
||||
harness.Step(t, "%s is still mounted read-write", faultMount)
|
||||
}
|
||||
return
|
||||
}
|
||||
t.Errorf("%s is no longer mounted at all after the outage", faultMount)
|
||||
}
|
||||
|
||||
// assertWorkloadIntegrity re-reads the verifiable region and compares its
|
||||
// digest to the one taken when it was written. This is what distinguishes "the
|
||||
// guest went read-only" from "the volume lost an acknowledged write", which are
|
||||
// different failures with different fixes.
|
||||
//
|
||||
// The cache is dropped first, so the comparison reads the volume rather than
|
||||
// the page cache that has been holding the same bytes since before the fault.
|
||||
func assertWorkloadIntegrity(t *testing.T, tgt harness.SSHTarget, want string) {
|
||||
t.Helper()
|
||||
check := fmt.Sprintf(
|
||||
"sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null; "+
|
||||
"sudo sha256sum %s | cut -d' ' -f1", verifyFile)
|
||||
out, err := harness.GuestExecTimeout(tgt, check, 10*time.Minute)
|
||||
if err != nil {
|
||||
t.Errorf("could not re-read the verifiable region after recovery: %v\n%s", err, out)
|
||||
return
|
||||
}
|
||||
if got := strings.TrimSpace(out); got != want {
|
||||
t.Errorf("the verifiable region changed across the outage — an acknowledged write did not survive:\n"+
|
||||
" wrote %s\n read %s", want, got)
|
||||
return
|
||||
}
|
||||
harness.Step(t, "sha256 verification passed: every acknowledged write survived")
|
||||
}
|
||||
|
||||
// bestEffort runs a guest command and returns its output, or a marker when the
|
||||
// guest cannot answer. Used for evidence gathering, where an unreachable guest
|
||||
// is itself information and must not abort the run.
|
||||
func bestEffort(tgt harness.SSHTarget, cmd string) string {
|
||||
out, err := harness.GuestExecTimeout(tgt, cmd, 30*time.Second)
|
||||
if err != nil {
|
||||
return fmt.Sprintf("<guest unreachable: %v>", err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// awaitGuestRecovered waits for a guest that the fault paused to answer again.
|
||||
// It never fails the test: whether the guest came back is what the assertions
|
||||
// after it are for, and they say far more about why than a timeout here would.
|
||||
func awaitGuestRecovered(t *testing.T, tgt harness.SSHTarget) {
|
||||
t.Helper()
|
||||
time.Sleep(recoverySettle)
|
||||
if harness.TryGuestSSHReady(tgt.Host, tgt.Port, tgt.User, tgt.KeyPath, guestRecoveryTimeout) {
|
||||
return
|
||||
}
|
||||
harness.Step(t, "guest still not answering %s after the backend came back", tgt.Host)
|
||||
}
|
||||
|
||||
// lastLines keeps the tail of a console log. The whole buffer is 64 KiB of
|
||||
// boot messages, and what matters is what the guest said most recently.
|
||||
func lastLines(s string, n int) string {
|
||||
lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// instanceStateOf reports the EC2 state, or why it could not be read. It never
|
||||
// fails the test: the control plane being unreachable during the outage is an
|
||||
// observation, and the assertions that decide the run are made after recovery.
|
||||
func instanceStateOf(fix *Fixture, instanceID string) string {
|
||||
out, err := fix.AWS.EC2.DescribeInstances(&ec2.DescribeInstancesInput{
|
||||
InstanceIds: []*string{aws.String(instanceID)},
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Sprintf("<describe failed: %v>", err)
|
||||
}
|
||||
for _, r := range out.Reservations {
|
||||
for _, inst := range r.Instances {
|
||||
if inst.State != nil && inst.State.Name != nil {
|
||||
return *inst.State.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return "<unknown>"
|
||||
}
|
||||
|
||||
func nodeName(n *harness.Node) string {
|
||||
if n == nil {
|
||||
return "<unidentified>"
|
||||
}
|
||||
return n.Name
|
||||
}
|
||||
|
||||
func nodeNames(nodes []harness.Node) string {
|
||||
out := make([]string, 0, len(nodes))
|
||||
for _, n := range nodes {
|
||||
out = append(out, n.Name)
|
||||
}
|
||||
return strings.Join(out, ", ")
|
||||
}
|
||||
|
||||
// ensureFaultInstance launches the guest this suite drives. The smallest
|
||||
// catalog entry is enough: the workload is bounded by the volume, not by the
|
||||
// guest's CPU or memory.
|
||||
//
|
||||
// Each test gets its own guest. Every fault here is meant to hurt, and a guest
|
||||
// that survived one is not a clean starting point for the next: a wedged jbd2
|
||||
// or a filesystem the kernel has given up on outlives the test that caused it,
|
||||
// and the next test then fails on damage it did not do.
|
||||
func ensureFaultInstance(t *testing.T, fix *Fixture) (instanceID string, tgt harness.SSHTarget) {
|
||||
t.Helper()
|
||||
instType, arch := harness.DiscoverNanoInstanceType(t, fix.Harness)
|
||||
ami := harness.DiscoverUbuntuAMI(t, fix.Harness, arch)
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness)
|
||||
vpc := harness.EnsureDefaultVPC(t, fix.Harness)
|
||||
harness.AuthorizeSSHIngress(t, fix.AWS, vpc.SGID)
|
||||
|
||||
instanceID = harness.EnsureInstance(t, fix.Harness, harness.InstanceSpec{
|
||||
AMIID: ami,
|
||||
InstanceType: instType,
|
||||
KeyName: keyName,
|
||||
SubnetID: vpc.SubnetID,
|
||||
SGID: vpc.SGID,
|
||||
Scope: t.Name(),
|
||||
})
|
||||
inst := harness.WaitForInstanceState(t, fix.AWS, instanceID, "running")
|
||||
host, port := harness.InstancePublicSSHHost(t, inst)
|
||||
|
||||
harness.Step(t, "waiting for guest SSH on %s:%d (instance_type=%s)", host, port, instType)
|
||||
if !harness.TryGuestSSHReady(host, port, "ubuntu", keyPath, 5*time.Minute) {
|
||||
// The instance is shared across this package's tests, so an unreachable
|
||||
// guest is usually something an earlier test left behind. The console
|
||||
// is the only evidence, and terminating it takes that away.
|
||||
console, err := harness.InstanceConsole(fix.AWS, instanceID)
|
||||
if err != nil {
|
||||
console = fmt.Sprintf("console unavailable: %v", err)
|
||||
}
|
||||
t.Fatalf("guest %s SSH %s:%d not ready after 5m\nconsole tail:\n%s",
|
||||
instanceID, host, port, lastLines(console, 60))
|
||||
}
|
||||
return instanceID, harness.SSHTarget{User: "ubuntu", Host: host, Port: port, KeyPath: keyPath}
|
||||
}
|
||||
|
||||
// rootSectorsWritten reads sectors_written for the root device. In
|
||||
// /proc/diskstats the device name is the third field and sectors_written the
|
||||
// tenth, counted from the start of the line rather than from the name.
|
||||
func rootSectorsWritten(tgt harness.SSHTarget) (int64, error) {
|
||||
out, err := harness.GuestExecTimeout(tgt,
|
||||
"awk '$3==\""+rootDevice+"\"{print $10}' /proc/diskstats", 30*time.Second)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read /proc/diskstats: %w (%s)", err, out)
|
||||
}
|
||||
field := strings.TrimSpace(out)
|
||||
if field == "" {
|
||||
return 0, fmt.Errorf("no %s row in /proc/diskstats:\n%s", rootDevice,
|
||||
bestEffort(tgt, "cat /proc/diskstats"))
|
||||
}
|
||||
n, err := strconv.ParseInt(field, 10, 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("parse sectors_written %q: %w", field, err)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// assertRootLoadIsWriting confirms the root load is both running and actually
|
||||
// moving bytes on the root device. A load that produced no writes would make a
|
||||
// clean run mean nothing, since the fault could never have reached the root
|
||||
// filesystem at all. Two samples, because a non-zero total is only the boot's
|
||||
// own writes; an increase is what proves the load is live right now.
|
||||
func assertRootLoadIsWriting(t *testing.T, tgt harness.SSHTarget) {
|
||||
t.Helper()
|
||||
out, err := harness.GuestExecTimeout(tgt, "pgrep -f 'name=rootload' >/dev/null && echo RUNNING", 30*time.Second)
|
||||
if err != nil || !strings.Contains(out, "RUNNING") {
|
||||
t.Fatalf("the root-filesystem load is not running, so %s is idle and the fault cannot reach it:\n%s",
|
||||
rootDevice, bestEffort(tgt, "tail -n 30 ~/rootfio.log"))
|
||||
}
|
||||
|
||||
before, err := rootSectorsWritten(tgt)
|
||||
if err != nil {
|
||||
t.Fatalf("cannot confirm the root load is writing: %v", err)
|
||||
}
|
||||
time.Sleep(rootWriteSample)
|
||||
after, err := rootSectorsWritten(tgt)
|
||||
if err != nil {
|
||||
t.Fatalf("cannot confirm the root load is writing: %v", err)
|
||||
}
|
||||
if after <= before {
|
||||
t.Fatalf("%s took no writes over %s (sectors_written stayed at %d), so a clean run would prove nothing:\n%s",
|
||||
rootDevice, rootWriteSample, after, bestEffort(tgt, "tail -n 30 ~/rootfio.log"))
|
||||
}
|
||||
harness.Step(t, "root load is writing: %s gained %d sectors (%d KiB) in %s",
|
||||
rootDevice, after-before, (after-before)/2, rootWriteSample)
|
||||
}
|
||||
@@ -0,0 +1,302 @@
|
||||
//go:build e2e
|
||||
|
||||
// Package storagefault proves what a guest does when its storage backend stops
|
||||
// answering, and what a stop/start does when the seal to that backend failed.
|
||||
//
|
||||
// The claim is about corruption, not availability. Nobody expects a guest to
|
||||
// keep serving I/O while predastore is down; the question is whether it comes
|
||||
// back to a filesystem that is intact and a volume whose acknowledged writes
|
||||
// are all present. Those are the two assertions here, and both are expected to
|
||||
// fail against the engine as it stands.
|
||||
//
|
||||
// The fault is SIGSTOP rather than a stop of the unit. A stopped process holds
|
||||
// its TCP connections open and sends no RST, so peers see a node that is
|
||||
// present and answering nothing — which is the fault that defeats code with no
|
||||
// timeout, and the one a clean shutdown cannot reproduce. SIGCONT restores it
|
||||
// with no state lost, so the cluster is left exactly as it was found.
|
||||
//
|
||||
// It is its own package rather than a case inside single/ because it must
|
||||
// freeze a cluster-wide service. Everything sharing the process would see the
|
||||
// same outage, so a suite that memoizes one instance across many tests cannot
|
||||
// host this without making every sibling's failure ambiguous.
|
||||
package storagefault
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/tests/e2e/harness"
|
||||
)
|
||||
|
||||
var (
|
||||
pkgFixOnce sync.Once
|
||||
pkgFix *Fixture
|
||||
pkgFixErr error
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
code := m.Run()
|
||||
if pkgFix != nil && pkgFix.Harness != nil {
|
||||
if err := pkgFix.Harness.Close(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "e2e teardown: %v\n", err)
|
||||
code = 1
|
||||
}
|
||||
}
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
// Fixture carries per-process state shared across this package's tests.
|
||||
type Fixture struct {
|
||||
Env *harness.Env
|
||||
AWS *harness.AWSClient
|
||||
Harness *harness.Fixture
|
||||
Cluster *harness.Cluster
|
||||
SSH harness.SSH
|
||||
}
|
||||
|
||||
func (f *Fixture) ArtifactDir(t *testing.T) string {
|
||||
t.Helper()
|
||||
return harness.ArtifactDir(t, f.Env)
|
||||
}
|
||||
|
||||
// requireStorageFaultFixture returns the package-scoped Fixture singleton,
|
||||
// building it on first call. Skips when SPINIFEX_E2E is unset or the cluster
|
||||
// is a size this suite has no defined freeze set for.
|
||||
func requireStorageFaultFixture(t *testing.T) *Fixture {
|
||||
t.Helper()
|
||||
pkgFixOnce.Do(func() {
|
||||
if os.Getenv("SPINIFEX_E2E") == "" {
|
||||
return
|
||||
}
|
||||
env := harness.LoadEnv(t)
|
||||
cluster, err := harness.ClusterFromEnv()
|
||||
if err != nil {
|
||||
pkgFixErr = fmt.Errorf("this suite needs node SSH to inject the fault: %w", err)
|
||||
return
|
||||
}
|
||||
sshCli, err := harness.NewSSH(cluster)
|
||||
if err != nil {
|
||||
pkgFixErr = err
|
||||
return
|
||||
}
|
||||
awsCli := harness.NewAWSClient(t, env)
|
||||
h, err := harness.NewProcessFixture(awsCli)
|
||||
if err != nil {
|
||||
pkgFixErr = err
|
||||
return
|
||||
}
|
||||
harness.EnsureDefaultSGOpen(t, awsCli)
|
||||
pkgFix = &Fixture{Env: env, AWS: awsCli, Harness: h, Cluster: cluster, SSH: sshCli}
|
||||
})
|
||||
if pkgFixErr != nil {
|
||||
t.Fatalf("storagefault fixture init: %v", pkgFixErr)
|
||||
}
|
||||
if pkgFix == nil {
|
||||
t.Skip("SPINIFEX_E2E unset")
|
||||
}
|
||||
if n := len(pkgFix.Cluster.Nodes); n != 1 && n < 3 {
|
||||
t.Skipf("no defined freeze set for a %d-node cluster: 1 node is RS(1,0) and 3+ leaves a floor to break, 2 is neither", n)
|
||||
}
|
||||
return pkgFix
|
||||
}
|
||||
|
||||
// --- Fault injection -------------------------------------------------------
|
||||
//
|
||||
// Every signal below is sent to a PID systemd reports as the predastore unit's
|
||||
// own MainPID, and only after its /proc cmdline has been confirmed. Nothing
|
||||
// here matches a process by name. That is not caution for its own sake: every
|
||||
// Spinifex service is the same spx binary, so `pgrep -f spx` matches NATS and
|
||||
// the daemon too, and these hosts also run unrelated QEMU guests whose comm is
|
||||
// indistinguishable from the platform's own.
|
||||
|
||||
// predastoreUnit is the systemd unit whose MainPID is the only process this
|
||||
// suite ever signals.
|
||||
const predastoreUnit = "spinifex-predastore"
|
||||
|
||||
// predastoreCmdline is the substring the resolved PID's cmdline must contain.
|
||||
// A MainPID that does not match is a unit that has been redefined, and the
|
||||
// safe response is to refuse rather than to signal an unknown process.
|
||||
const predastoreCmdline = "service predastore"
|
||||
|
||||
// freezeSettle is how long to wait after signalling before treating the state
|
||||
// change as in effect, so an assertion cannot race the signal.
|
||||
const freezeSettle = 2 * time.Second
|
||||
|
||||
// backendRecoveryBudget is how long the backend has to serve object I/O again
|
||||
// once a fault is cleared. Generous because reconnection and meta re-election
|
||||
// both have to happen; a wedge does not clear at all, so the budget only has to
|
||||
// tell the two apart.
|
||||
const backendRecoveryBudget = 3 * time.Minute
|
||||
|
||||
// predastorePID resolves the predastore unit's MainPID on node and confirms the
|
||||
// process is the one expected. Returns 0 with no error when the unit is not
|
||||
// running, which is a fact the caller decides what to do about.
|
||||
func predastorePID(ctx context.Context, ssh harness.SSH, node harness.Node) (int, error) {
|
||||
raw, err := ssh.Run(ctx, node, "systemctl show "+predastoreUnit+" -p MainPID --value")
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("resolve %s MainPID on %s: %w", predastoreUnit, node.Name, err)
|
||||
}
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(string(raw)))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("MainPID on %s was %q, not a number", node.Name, strings.TrimSpace(string(raw)))
|
||||
}
|
||||
if pid <= 1 {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
cmdline, err := ssh.Run(ctx, node, fmt.Sprintf("tr '\\0' ' ' < /proc/%d/cmdline", pid))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read cmdline of pid %d on %s: %w", pid, node.Name, err)
|
||||
}
|
||||
if !strings.Contains(string(cmdline), predastoreCmdline) {
|
||||
return 0, fmt.Errorf("refusing to signal pid %d on %s: cmdline %q does not contain %q",
|
||||
pid, node.Name, strings.TrimSpace(string(cmdline)), predastoreCmdline)
|
||||
}
|
||||
return pid, nil
|
||||
}
|
||||
|
||||
// predastoreStopped reports whether the predastore process on node is in state
|
||||
// T. Used both to refuse a run that starts against a frozen cluster and to
|
||||
// confirm a thaw actually took.
|
||||
func predastoreStopped(ctx context.Context, ssh harness.SSH, node harness.Node) (bool, error) {
|
||||
pid, err := predastorePID(ctx, ssh, node)
|
||||
if err != nil || pid == 0 {
|
||||
return false, err
|
||||
}
|
||||
state, err := ssh.Run(ctx, node, fmt.Sprintf("ps -o state= -p %d", pid))
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("read state of pid %d on %s: %w", pid, node.Name, err)
|
||||
}
|
||||
return strings.HasPrefix(strings.TrimSpace(string(state)), "T"), nil
|
||||
}
|
||||
|
||||
// requireNoneFrozen fails the run when any node's predastore is already
|
||||
// stopped. A leftover from an aborted run would make every assertion below
|
||||
// meaningless, and thawing someone else's fault injection is not this suite's
|
||||
// call to make.
|
||||
func requireNoneFrozen(t *testing.T, f *Fixture) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
for _, n := range f.Cluster.Nodes {
|
||||
stopped, err := predastoreStopped(ctx, f.SSH, n)
|
||||
if err != nil {
|
||||
t.Fatalf("preflight: %v", err)
|
||||
}
|
||||
if stopped {
|
||||
t.Fatalf("preflight: predastore on %s is already stopped — a previous run left it frozen. "+
|
||||
"Thaw it with: ssh %s 'sudo kill -CONT $(systemctl show %s -p MainPID --value)'",
|
||||
n.Name, n.Addr, predastoreUnit)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// freezePredastore SIGSTOPs predastore on each node and registers an
|
||||
// unconditional thaw. The cleanup is registered before the first signal, so a
|
||||
// failure part-way through the set still restores the nodes already frozen.
|
||||
func freezePredastore(t *testing.T, f *Fixture, nodes []harness.Node) {
|
||||
t.Helper()
|
||||
t.Cleanup(func() { thawPredastore(t, f, nodes) })
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
for _, n := range nodes {
|
||||
pid, err := predastorePID(ctx, f.SSH, n)
|
||||
if err != nil {
|
||||
t.Fatalf("freeze: %v", err)
|
||||
}
|
||||
if pid == 0 {
|
||||
t.Fatalf("freeze: %s is not running on %s", predastoreUnit, n.Name)
|
||||
}
|
||||
if _, err := f.SSH.Run(ctx, n, fmt.Sprintf("sudo kill -STOP %d", pid)); err != nil {
|
||||
t.Fatalf("freeze: SIGSTOP pid %d on %s: %v", pid, n.Name, err)
|
||||
}
|
||||
t.Logf("froze predastore pid %d on %s", pid, n.Name)
|
||||
}
|
||||
time.Sleep(freezeSettle)
|
||||
}
|
||||
|
||||
// thawPredastore SIGCONTs predastore on each node and confirms it left state T.
|
||||
// A node that cannot be thawed fails the test even if every assertion passed:
|
||||
// leaving one frozen wedges the cluster for whatever runs next.
|
||||
func thawPredastore(t *testing.T, f *Fixture, nodes []harness.Node) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
|
||||
defer cancel()
|
||||
|
||||
for _, n := range nodes {
|
||||
pid, err := predastorePID(ctx, f.SSH, n)
|
||||
if err != nil || pid == 0 {
|
||||
t.Errorf("thaw: cannot resolve predastore on %s (pid=%d): %v", n.Name, pid, err)
|
||||
continue
|
||||
}
|
||||
if _, err := f.SSH.Run(ctx, n, fmt.Sprintf("sudo kill -CONT %d", pid)); err != nil {
|
||||
t.Errorf("thaw: SIGCONT pid %d on %s: %v", pid, n.Name, err)
|
||||
continue
|
||||
}
|
||||
stopped, err := predastoreStopped(ctx, f.SSH, n)
|
||||
if err != nil {
|
||||
t.Errorf("thaw: confirm %s: %v", n.Name, err)
|
||||
continue
|
||||
}
|
||||
if stopped {
|
||||
t.Errorf("thaw: predastore on %s is still stopped after SIGCONT", n.Name)
|
||||
continue
|
||||
}
|
||||
t.Logf("thawed predastore pid %d on %s", pid, n.Name)
|
||||
}
|
||||
time.Sleep(freezeSettle)
|
||||
|
||||
requireBackendServing(t, f)
|
||||
}
|
||||
|
||||
// requireBackendServing proves the whole cluster answers object I/O again, not
|
||||
// just the nodes that were frozen: the fault reaches every node through the
|
||||
// stripe, and a wedged one is reached through whichever gate is asked.
|
||||
func requireBackendServing(t *testing.T, f *Fixture) {
|
||||
t.Helper()
|
||||
if f == nil || f.Cluster == nil {
|
||||
return
|
||||
}
|
||||
|
||||
hosts := make([]string, 0, len(f.Cluster.Nodes))
|
||||
for _, n := range f.Cluster.Nodes {
|
||||
hosts = append(hosts, n.Addr)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), backendRecoveryBudget+time.Minute)
|
||||
defer cancel()
|
||||
harness.AssertPredastoreServing(ctx, t, hosts, backendRecoveryBudget)
|
||||
}
|
||||
|
||||
// freezeSetFor returns the nodes to freeze so the backend is definitively below
|
||||
// its floor for a guest on hostNode, and a label describing why.
|
||||
//
|
||||
// Every node but the guest's own is frozen. On three nodes that is two of
|
||||
// three: one shard is left, below DataShards=2 with degraded writes on, and
|
||||
// meta raft loses quorum. Leaving the guest's own node up is deliberate — the
|
||||
// local gate stays reachable, so what is being tested is the backend having no
|
||||
// quorum rather than viperblock having no gate to talk to.
|
||||
func freezeSetFor(f *Fixture, hostNode *harness.Node) ([]harness.Node, string) {
|
||||
all := f.Cluster.Nodes
|
||||
if len(all) == 1 {
|
||||
return all, "the only node: RS(1,0) has no redundancy, so any freeze is total"
|
||||
}
|
||||
|
||||
var out []harness.Node
|
||||
for _, n := range all {
|
||||
if hostNode != nil && n.Index == hostNode.Index {
|
||||
continue
|
||||
}
|
||||
out = append(out, n)
|
||||
}
|
||||
return out, fmt.Sprintf("%d of %d nodes, every one but the guest's own host", len(out), len(all))
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
//go:build e2e
|
||||
|
||||
package storagefault
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/tests/e2e/harness"
|
||||
)
|
||||
|
||||
// nbdkitComm is the exact /proc/<pid>/comm for an nbdkit process, short enough
|
||||
// that Linux's 15-byte truncation never applies.
|
||||
const nbdkitComm = "nbdkit"
|
||||
|
||||
// nbdkitPluginSuffix identifies our own plugin in a discovered argv, so an
|
||||
// nbdkit serving something else is never signalled.
|
||||
const nbdkitPluginSuffix = "viperblock-plugin.so"
|
||||
|
||||
// nbdkitSettle is how long to wait after signalling nbdkit before judging the
|
||||
// guest, so an assertion cannot race the signal.
|
||||
const nbdkitSettle = 5 * time.Second
|
||||
|
||||
// qemuUser is the account the guests run as. The relaunched socket has to be
|
||||
// writable by it, because that is who reconnects to it.
|
||||
const qemuUser = "spinifex-daemon"
|
||||
|
||||
// nbdkitUser is the account viperblockd starts nbdkit as. The relaunch matches
|
||||
// it so the process keeps the same access to the volume's cache and key.
|
||||
const nbdkitUser = "spinifex-viperblock"
|
||||
|
||||
// nbdkitRelaunchTimeout bounds the wait for a relaunched nbdkit to rebind its
|
||||
// socket. Opening a volume reads its state from the backend, so this is not
|
||||
// instant.
|
||||
const nbdkitRelaunchTimeout = 90 * time.Second
|
||||
|
||||
// nbdkitPIDForVolume resolves the pid of the nbdkit serving exactly volumeID on
|
||||
// node. It matches on three things together — the comm, our plugin, and the
|
||||
// volume argument — because every other nbdkit on these hosts belongs to
|
||||
// another volume or another tenant, and a name match would hit all of them.
|
||||
// Returns 0 with no error when nothing serves that volume here.
|
||||
func nbdkitPIDForVolume(ctx context.Context, ssh harness.SSH, node harness.Node, volumeID string) (int, error) {
|
||||
// pgrep -x on the comm, then filter each candidate's own cmdline. Done on
|
||||
// the far side because one round trip per pid over SSH is needlessly slow.
|
||||
script := fmt.Sprintf(
|
||||
`for p in $(pgrep -x %s 2>/dev/null); do `+
|
||||
`c=$(tr '\0' ' ' < /proc/$p/cmdline 2>/dev/null); `+
|
||||
`case "$c" in *%s*volume=%s*|*volume=%s*%s*) echo "$p"; esac; `+
|
||||
`done`,
|
||||
nbdkitComm, nbdkitPluginSuffix, volumeID, volumeID, nbdkitPluginSuffix)
|
||||
|
||||
raw, err := ssh.Run(ctx, node, script)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("scan nbdkit for %s on %s: %w", volumeID, node.Name, err)
|
||||
}
|
||||
fields := strings.Fields(string(raw))
|
||||
if len(fields) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if len(fields) > 1 {
|
||||
return 0, fmt.Errorf("refusing to signal: %d nbdkit processes serve %s on %s (%v)",
|
||||
len(fields), volumeID, node.Name, fields)
|
||||
}
|
||||
pid, err := strconv.Atoi(fields[0])
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("nbdkit pid for %s on %s was %q, not a number", volumeID, node.Name, fields[0])
|
||||
}
|
||||
return pid, nil
|
||||
}
|
||||
|
||||
// requireNbdkitForVolume resolves the pid and fails the test when nothing is
|
||||
// serving the volume. A test that signalled nothing would pass for the wrong
|
||||
// reason, so the absence is fatal rather than skipped.
|
||||
func requireNbdkitForVolume(t *testing.T, f *Fixture, node harness.Node, volumeID string) int {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
pid, err := nbdkitPIDForVolume(ctx, f.SSH, node, volumeID)
|
||||
if err != nil {
|
||||
t.Fatalf("cannot resolve nbdkit for %s on %s: %v", volumeID, node.Name, err)
|
||||
}
|
||||
if pid == 0 {
|
||||
t.Fatalf("no nbdkit serves %s on %s, so there is no fault to inject", volumeID, node.Name)
|
||||
}
|
||||
return pid
|
||||
}
|
||||
|
||||
// signalNbdkit sends sig to pid, having re-confirmed the pid still belongs to
|
||||
// the volume. The re-check is not paranoia: pids are reused, and the gap
|
||||
// between resolving one and signalling it is exactly where a reused pid would
|
||||
// land us on an unrelated process.
|
||||
func signalNbdkit(ctx context.Context, f *Fixture, node harness.Node, volumeID string, pid int, sig string) error {
|
||||
current, err := nbdkitPIDForVolume(ctx, f.SSH, node, volumeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if current != pid {
|
||||
return fmt.Errorf("refusing to signal pid %d on %s: it no longer serves %s (now %d)",
|
||||
pid, node.Name, volumeID, current)
|
||||
}
|
||||
if _, err := f.SSH.Run(ctx, node, fmt.Sprintf("sudo kill -%s %d", sig, pid)); err != nil {
|
||||
return fmt.Errorf("kill -%s %d on %s: %w", sig, pid, node.Name, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// freezeNbdkit SIGSTOPs the volume's nbdkit and registers the thaw before the
|
||||
// signal is sent, so an abort between the two still recovers. A frozen nbdkit
|
||||
// holds its NBD socket open and answers nothing, which the guest sees as a
|
||||
// stall rather than an error.
|
||||
func freezeNbdkit(t *testing.T, f *Fixture, node harness.Node, volumeID string, pid int) {
|
||||
t.Helper()
|
||||
t.Cleanup(func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
// Best effort by design: on the kill variant the process is already
|
||||
// gone, and a failed SIGCONT there is not a test failure.
|
||||
_, _ = f.SSH.Run(ctx, node, fmt.Sprintf("sudo kill -CONT %d 2>/dev/null || true", pid))
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if err := signalNbdkit(ctx, f, node, volumeID, pid, "STOP"); err != nil {
|
||||
t.Fatalf("freeze nbdkit for %s: %v", volumeID, err)
|
||||
}
|
||||
t.Logf("froze nbdkit pid %d serving %s on %s", pid, volumeID, node.Name)
|
||||
time.Sleep(nbdkitSettle)
|
||||
}
|
||||
|
||||
// thawNbdkit SIGCONTs the volume's nbdkit.
|
||||
func thawNbdkit(t *testing.T, f *Fixture, node harness.Node, pid int) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if _, err := f.SSH.Run(ctx, node, fmt.Sprintf("sudo kill -CONT %d", pid)); err != nil {
|
||||
t.Errorf("thaw nbdkit pid %d on %s: %v", pid, node.Name, err)
|
||||
return
|
||||
}
|
||||
t.Logf("thawed nbdkit pid %d on %s", pid, node.Name)
|
||||
}
|
||||
|
||||
// nbdkitCmdline reads a running nbdkit's argv, so the process can be recreated
|
||||
// exactly. Capturing it before the kill is what makes the relaunch faithful:
|
||||
// the socket path carries a launch-time nonce that cannot be derived from the
|
||||
// volume id.
|
||||
func nbdkitCmdline(ctx context.Context, f *Fixture, node harness.Node, pid int) (string, error) {
|
||||
raw, err := f.SSH.Run(ctx, node, fmt.Sprintf("sudo tr '\\0' '\\n' < /proc/%d/cmdline", pid))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read cmdline of pid %d on %s: %w", pid, node.Name, err)
|
||||
}
|
||||
args := strings.Fields(string(raw))
|
||||
if len(args) == 0 {
|
||||
return "", fmt.Errorf("pid %d on %s has an empty cmdline", pid, node.Name)
|
||||
}
|
||||
return strings.Join(args, " "), nil
|
||||
}
|
||||
|
||||
// nbdkitSocketPath pulls the --unix argument out of a captured argv. The
|
||||
// relaunch must bind the same path or QEMU has nothing to reconnect to.
|
||||
func nbdkitSocketPath(cmdline string) (string, error) {
|
||||
args := strings.Fields(cmdline)
|
||||
for i, a := range args {
|
||||
if a == "--unix" && i+1 < len(args) {
|
||||
return args[i+1], nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("no --unix argument in nbdkit cmdline %q", cmdline)
|
||||
}
|
||||
|
||||
// killedNbdkit is what a kill leaves behind: enough to put the process back.
|
||||
// envFile is a path on the node, never the credentials themselves — see
|
||||
// captureNbdkitEnv.
|
||||
type killedNbdkit struct {
|
||||
cmdline string
|
||||
socket string
|
||||
envFile string
|
||||
}
|
||||
|
||||
// captureNbdkitEnv copies the credential variables out of a running nbdkit's
|
||||
// environment into a file on the same node. viperblockd passes them this way
|
||||
// rather than on the command line so they stay out of the world-readable
|
||||
// /proc/<pid>/cmdline, so a relaunch built from argv alone fails with
|
||||
// "access_key parameter is required".
|
||||
//
|
||||
// The values are never read back into the test: the file is written on the
|
||||
// node, consumed on the node and deleted there, so they stay exactly as
|
||||
// confined as the original design intended.
|
||||
func captureNbdkitEnv(ctx context.Context, f *Fixture, node harness.Node, volumeID string, pid int) (string, error) {
|
||||
path := fmt.Sprintf("/tmp/nbdkit-env-%s", volumeID)
|
||||
capture := fmt.Sprintf(
|
||||
"sudo sh -c 'umask 077; tr \"\\0\" \"\\n\" < /proc/%d/environ | grep -E \"^VB_(ACCESS|SECRET)_KEY=\" > %s' && "+
|
||||
"sudo chown %s %s && sudo test -s %s",
|
||||
pid, path, nbdkitUser, path, path)
|
||||
if _, err := f.SSH.Run(ctx, node, capture); err != nil {
|
||||
return "", fmt.Errorf("capture nbdkit credentials on %s: %w", node.Name, err)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
// killNbdkit SIGKILLs the volume's nbdkit, first capturing what is needed to
|
||||
// bring it back. Nothing restarts it on its own — viperblockd re-adopts
|
||||
// survivors after its own restart and never starts a process — so the volume
|
||||
// stays unserved until relaunchNbdkit puts it back.
|
||||
//
|
||||
// The guest reaches this by a different route to a freeze: the socket closes
|
||||
// rather than going quiet, so requests fail instead of blocking.
|
||||
func killNbdkit(t *testing.T, f *Fixture, node harness.Node, volumeID string, pid int) killedNbdkit {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cmdline, err := nbdkitCmdline(ctx, f, node, pid)
|
||||
if err != nil {
|
||||
t.Fatalf("capture nbdkit cmdline before killing it: %v", err)
|
||||
}
|
||||
socket, err := nbdkitSocketPath(cmdline)
|
||||
if err != nil {
|
||||
t.Fatalf("locate the nbd socket for %s: %v", volumeID, err)
|
||||
}
|
||||
envFile, err := captureNbdkitEnv(ctx, f, node, volumeID, pid)
|
||||
if err != nil {
|
||||
t.Fatalf("capture the environment before killing nbdkit for %s: %v", volumeID, err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
cleanCtx, cancelClean := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancelClean()
|
||||
_, _ = f.SSH.Run(cleanCtx, node, fmt.Sprintf("sudo rm -f %s", envFile))
|
||||
})
|
||||
|
||||
if err := signalNbdkit(ctx, f, node, volumeID, pid, "KILL"); err != nil {
|
||||
t.Fatalf("kill nbdkit for %s: %v", volumeID, err)
|
||||
}
|
||||
t.Logf("killed nbdkit pid %d serving %s on %s (socket %s)", pid, volumeID, node.Name, socket)
|
||||
time.Sleep(nbdkitSettle)
|
||||
return killedNbdkit{cmdline: cmdline, socket: socket, envFile: envFile}
|
||||
}
|
||||
|
||||
// reapRelaunchedNbdkit kills the relaunched server. setsid detaches it from
|
||||
// the SSH session that started it, and viperblockd never learned it exists, so
|
||||
// nothing else will ever reap it.
|
||||
//
|
||||
// Call this only once the volume is detached. It is the server behind an
|
||||
// attached disk, so killing it first leaves QEMU holding the blockdev node and
|
||||
// the platform's own detach then fails with "Node ... is in use".
|
||||
//
|
||||
// The PID is resolved from the socket path rather than the volume id: a
|
||||
// legitimate viperblockd nbdkit for the same volume must not be killed, and
|
||||
// only this one is bound to this path.
|
||||
func reapRelaunchedNbdkit(t *testing.T, f *Fixture, node harness.Node, socket string) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
|
||||
defer cancel()
|
||||
out, err := f.SSH.Run(ctx, node,
|
||||
fmt.Sprintf("sudo pgrep -a nbdkit | grep -F -- %q | awk '{print $1}'", socket))
|
||||
if err != nil {
|
||||
t.Logf("could not look up relaunched nbdkit on %s: %v", node.Name, err)
|
||||
return
|
||||
}
|
||||
for _, pid := range strings.Fields(string(out)) {
|
||||
if _, err := f.SSH.Run(ctx, node, fmt.Sprintf("sudo kill %s", pid)); err != nil {
|
||||
t.Logf("could not stop relaunched nbdkit %s on %s: %v", pid, node.Name, err)
|
||||
continue
|
||||
}
|
||||
t.Logf("stopped relaunched nbdkit %s on %s", pid, node.Name)
|
||||
}
|
||||
if _, err := f.SSH.Run(ctx, node, fmt.Sprintf("sudo rm -f %s", socket)); err != nil {
|
||||
t.Logf("could not remove %s on %s: %v", socket, node.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// relaunchNbdkit puts the volume's NBD server back on the same socket path,
|
||||
// which is what turns the kill into a recoverable fault rather than a
|
||||
// destructive one. The stale socket is removed first: nbdkit will not bind a
|
||||
// path that already exists, and a SIGKILL leaves the file behind.
|
||||
//
|
||||
// It runs as the same user under setsid, because nbdkit was started with -f
|
||||
// and would otherwise die with the SSH session that launched it.
|
||||
func relaunchNbdkit(t *testing.T, f *Fixture, node harness.Node, volumeID string, k killedNbdkit) {
|
||||
t.Helper()
|
||||
// Budgeted to outlast the poll below with room for its round trips. Sharing
|
||||
// a shorter context with the poll left it expired by the time the
|
||||
// diagnostics ran, so a failure reported nothing at all.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), nbdkitRelaunchTimeout+3*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
// Output goes to a file rather than /dev/null: an nbdkit that starts and
|
||||
// exits is the interesting failure, and its reason is only on stderr.
|
||||
// The credentials are sourced inside the target user's shell rather than
|
||||
// placed on the command line, so they are no more exposed than viperblockd
|
||||
// leaves them. Reading with `export "$line"` keeps any character in the
|
||||
// value intact, which quoting into a command string would not.
|
||||
log := fmt.Sprintf("/tmp/nbdkit-relaunch-%s.log", volumeID)
|
||||
start := fmt.Sprintf(
|
||||
"sudo rm -f %s; sudo touch %s && sudo chmod 666 %s && "+
|
||||
"sudo -u %s bash -c 'while IFS= read -r l; do export \"$l\"; done < %s; exec setsid %s' "+
|
||||
"</dev/null >%s 2>&1 & sleep 5; cat %s",
|
||||
k.socket, log, log, nbdkitUser, k.envFile, k.cmdline, log, log)
|
||||
// The early output is read here rather than only on failure: a later query
|
||||
// can fail on its own and take the reason with it.
|
||||
early, err := f.SSH.Run(ctx, node, start)
|
||||
if err != nil {
|
||||
t.Fatalf("relaunch nbdkit for %s on %s: %v", volumeID, node.Name, err)
|
||||
}
|
||||
if len(strings.TrimSpace(string(early))) > 0 {
|
||||
t.Logf("nbdkit relaunch said: %s", strings.TrimSpace(string(early)))
|
||||
}
|
||||
|
||||
// QEMU reconnects to the path, not to the process, so the socket being
|
||||
// back and writable by the account QEMU runs as is the whole signal. The
|
||||
// chmod is viperblockd's own step, not a workaround: nbdkit binds the
|
||||
// socket 0755 and a QEMU with no group-write bit can never reconnect.
|
||||
ready := fmt.Sprintf("sudo test -S %s && sudo chmod 0770 %s && sudo -u %s test -w %s",
|
||||
k.socket, k.socket, qemuUser, k.socket)
|
||||
deadline := time.Now().Add(nbdkitRelaunchTimeout)
|
||||
for time.Now().Before(deadline) {
|
||||
if _, err := f.SSH.Run(ctx, node, ready); err == nil {
|
||||
t.Logf("relaunched nbdkit for %s on %s, socket %s is back", volumeID, node.Name, k.socket)
|
||||
return
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
|
||||
out, _ := f.SSH.Run(ctx, node, fmt.Sprintf(
|
||||
"cat %s 2>/dev/null; echo '--- socket ---'; sudo ls -la %s 2>&1; echo '--- procs ---'; pgrep -ax nbdkit | head",
|
||||
log, k.socket))
|
||||
t.Fatalf("nbdkit for %s did not rebind %s on %s within %s\ncommand: %s\n%s",
|
||||
volumeID, k.socket, node.Name, nbdkitRelaunchTimeout, k.cmdline, out)
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
//go:build e2e
|
||||
|
||||
package storagefault
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/tests/e2e/harness"
|
||||
)
|
||||
|
||||
// nbdkitHold is how long the volume stays unserved. Unlike the predastore
|
||||
// freeze this needs no retry budget outlasting: a killed nbdkit closes its
|
||||
// socket and QEMU fails the request at once, and a frozen one stalls for as
|
||||
// long as it is held. Short enough that both variants fit in one run.
|
||||
const nbdkitHold = 60 * time.Second
|
||||
|
||||
// TestGuestSurvivesNbdkitLoss removes the volume's own NBD server rather than
|
||||
// the storage cluster behind it, which is a far narrower fault: one volume on
|
||||
// one host, with no cluster-wide service stopped and nothing for another suite
|
||||
// to trip over.
|
||||
//
|
||||
// The two signals are deliberately both here, because they are on opposite
|
||||
// sides of the defect and running only one would prove the wrong thing:
|
||||
//
|
||||
// - SIGSTOP holds the NBD socket open and answers nothing. The guest blocks.
|
||||
// No error is ever delivered, so ext4 has no reason to abort.
|
||||
// - SIGKILL closes the socket, so requests fail rather than block. Under the
|
||||
// nbd driver's default reconnect-delay of 0 they fail instantly, and under
|
||||
// werror=report that EIO reaches the guest and aborts its journal.
|
||||
//
|
||||
// The killed variant is put back on the same socket path afterwards, because a
|
||||
// volume left dead tests the fault and not the recovery. Under
|
||||
// reconnect-delay plus werror=stop,rerror=stop both must become stalls that
|
||||
// resume, and neither may show a corruption signature.
|
||||
func TestGuestSurvivesNbdkitLoss(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
kill bool
|
||||
}{
|
||||
{name: "Frozen", kill: false},
|
||||
{name: "Killed", kill: true},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
fix := requireStorageFaultFixture(t)
|
||||
|
||||
instanceID, tgt := ensureFaultInstance(t, fix)
|
||||
hostNode := harness.InstanceHostingNode(t, fix.Cluster, instanceID)
|
||||
if hostNode == nil {
|
||||
t.Fatalf("cannot identify the node hosting %s, so its nbdkit cannot be found", instanceID)
|
||||
}
|
||||
|
||||
az := harness.DiscoverDefaultAZ(t, fix.Harness)
|
||||
volID := harness.EnsureVolume(t, fix.Harness, az, faultVolumeSizeGiB)
|
||||
|
||||
before := harness.GuestDiskSet(t, tgt)
|
||||
harness.AttachVolumeWait(t, fix.AWS, volID, instanceID, faultDevice)
|
||||
dev := harness.WaitForNewGuestDisk(t, tgt, before, 90*time.Second)
|
||||
|
||||
// relaunchedSocket names an nbdkit the killed variant put back by
|
||||
// hand. Reaped here rather than on its own cleanup so it happens
|
||||
// after the detach: it is the server behind an attached disk, and
|
||||
// killing it first leaves the detach fighting a pinned blockdev.
|
||||
var relaunchedSocket string
|
||||
t.Cleanup(func() {
|
||||
harness.DetachVolumeWait(t, fix.AWS, volID)
|
||||
if relaunchedSocket != "" {
|
||||
reapRelaunchedNbdkit(t, fix, *hostNode, relaunchedSocket)
|
||||
}
|
||||
})
|
||||
|
||||
prepareWorkloadFilesystem(t, tgt, dev)
|
||||
quiesceWorkload(t, tgt)
|
||||
|
||||
pid := requireNbdkitForVolume(t, fix, *hostNode, volID)
|
||||
harness.Detail(t, "instance", instanceID, "volume", volID, "guest_device", dev,
|
||||
"host_node", nodeName(hostNode), "nbdkit_pid", pid)
|
||||
|
||||
harness.Step(t, "writing the %d MiB verifiable region and flushing it", verifyMiB)
|
||||
digest := writeVerifiableRegion(t, tgt)
|
||||
|
||||
loadRuntime := loadSettle + nbdkitHold + recoverySettle
|
||||
harness.Step(t, "starting sustained load for %s", loadRuntime)
|
||||
startLoad(t, tgt, loadRuntime)
|
||||
time.Sleep(loadSettle)
|
||||
if !loadRunning(t, tgt) {
|
||||
t.Fatalf("the load exited before the fault was injected, so nothing was under load: %s",
|
||||
loadLogTail(t, tgt))
|
||||
}
|
||||
|
||||
var killed killedNbdkit
|
||||
if tc.kill {
|
||||
harness.Step(t, "killing nbdkit for %s, then holding %s", volID, nbdkitHold)
|
||||
killed = killNbdkit(t, fix, *hostNode, volID, pid)
|
||||
} else {
|
||||
harness.Step(t, "freezing nbdkit for %s for %s", volID, nbdkitHold)
|
||||
freezeNbdkit(t, fix, *hostNode, volID, pid)
|
||||
}
|
||||
|
||||
observeDuringOutage(t, fix, instanceID, tgt)
|
||||
|
||||
if tc.kill {
|
||||
harness.Step(t, "relaunching nbdkit on %s, then waiting %s", killed.socket, recoverySettle)
|
||||
relaunchNbdkit(t, fix, *hostNode, volID, killed)
|
||||
relaunchedSocket = killed.socket
|
||||
} else {
|
||||
harness.Step(t, "thawing nbdkit and waiting %s", recoverySettle)
|
||||
thawNbdkit(t, fix, *hostNode, pid)
|
||||
}
|
||||
awaitGuestRecovered(t, tgt)
|
||||
|
||||
console, consoleErr := harness.InstanceConsole(fix.AWS, instanceID)
|
||||
if consoleErr != nil {
|
||||
t.Logf("console unavailable (guest evidence only): %v", consoleErr)
|
||||
}
|
||||
evidence := console + "\n" + bestEffort(tgt, "sudo dmesg | tail -n 400")
|
||||
|
||||
assertNoCorruption(t, evidence, instanceID, fix)
|
||||
assertStillWritable(t, tgt, dev)
|
||||
assertWorkloadIntegrity(t, tgt, digest)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,466 @@
|
||||
//go:build e2e
|
||||
|
||||
package storagefault
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/tests/e2e/harness"
|
||||
)
|
||||
|
||||
const (
|
||||
// patternDevice is the guest-visible attach point for the pattern volume.
|
||||
patternDevice = "/dev/sdg"
|
||||
|
||||
// patternVolumeSizeGiB sizes the volume holding the known pattern.
|
||||
patternVolumeSizeGiB = 1
|
||||
|
||||
// patternMiB is how much of the device the pattern covers. Large enough to
|
||||
// span many 4 KiB blocks and several 4 MiB chunks, small enough to write
|
||||
// and re-read in seconds.
|
||||
patternMiB = 64
|
||||
|
||||
// stopStartTimeout bounds a StopInstances or StartInstances round trip.
|
||||
// Generous because the stop is deliberately made to run against a backend
|
||||
// that cannot answer, which is the slowest this path ever gets.
|
||||
stopStartTimeout = 10 * time.Minute
|
||||
)
|
||||
|
||||
// viperblockBaseDir is where a node keeps per-volume local state. Overridable
|
||||
// for a cluster laid out differently, but this is the installed default and
|
||||
// what the seal receipt sits beside.
|
||||
func viperblockBaseDir() string {
|
||||
if v := os.Getenv("SPINIFEX_VIPERBLOCK_BASEDIR"); v != "" {
|
||||
return v
|
||||
}
|
||||
return "/var/lib/spinifex/viperblock"
|
||||
}
|
||||
|
||||
// TestStopStartPreservesWritesOnSameNode proves the recovery path the whole
|
||||
// dirty-volume design rests on: when a seal fails, the local WAL is kept and a
|
||||
// restart on the same node recovers from it with nothing lost.
|
||||
//
|
||||
// Expected to pass against the current engine. It is here because the
|
||||
// cross-node test below is only meaningful if this one holds — a fix that
|
||||
// refused every start would make that test green while making the platform
|
||||
// worse.
|
||||
func TestStopStartPreservesWritesOnSameNode(t *testing.T) {
|
||||
fix := requireStorageFaultFixture(t)
|
||||
requireNoneFrozen(t, fix)
|
||||
|
||||
instanceID, tgt := ensureFaultInstance(t, fix)
|
||||
hostNode := harness.InstanceHostingNode(t, fix.Cluster, instanceID)
|
||||
if hostNode == nil {
|
||||
t.Fatalf("cannot identify the node hosting %s, so its local state cannot be inspected", instanceID)
|
||||
}
|
||||
|
||||
az := harness.DiscoverDefaultAZ(t, fix.Harness)
|
||||
volID := harness.EnsureVolume(t, fix.Harness, az, patternVolumeSizeGiB)
|
||||
|
||||
before := harness.GuestDiskSet(t, tgt)
|
||||
harness.AttachVolumeWait(t, fix.AWS, volID, instanceID, patternDevice)
|
||||
dev := harness.WaitForNewGuestDisk(t, tgt, before, 90*time.Second)
|
||||
t.Cleanup(func() { harness.DetachVolumeWait(t, fix.AWS, volID) })
|
||||
|
||||
want := writePattern(t, tgt, dev)
|
||||
harness.Detail(t, "instance", instanceID, "volume", volID, "host_node", hostNode.Name, "sha256", want)
|
||||
|
||||
freezeSet, why := freezeSetFor(fix, hostNode)
|
||||
harness.Step(t, "freezing predastore on %s (%s) so the seal cannot complete", nodeNames(freezeSet), why)
|
||||
freezePredastore(t, fix, freezeSet)
|
||||
|
||||
harness.Step(t, "stopping %s with the backend frozen", instanceID)
|
||||
stopErr := stopInstance(t, fix, instanceID)
|
||||
t.Logf("StopInstances returned: %v", errOrOK(stopErr))
|
||||
|
||||
assertLocalStateKept(t, fix, hostNode, volID)
|
||||
|
||||
thawPredastore(t, fix, freezeSet)
|
||||
time.Sleep(recoverySettle)
|
||||
|
||||
harness.Step(t, "starting %s again on the same node", instanceID)
|
||||
startInstance(t, fix, instanceID)
|
||||
inst := harness.WaitForInstanceState(t, fix.AWS, instanceID, "running")
|
||||
host, port := harness.InstancePublicSSHHost(t, inst)
|
||||
if !harness.TryGuestSSHReady(host, port, "ubuntu", tgt.KeyPath, 5*time.Minute) {
|
||||
t.Fatalf("guest did not come back after a same-node restart%s", consoleTail(fix, instanceID))
|
||||
}
|
||||
tgt = harness.SSHTarget{User: "ubuntu", Host: host, Port: port, KeyPath: tgt.KeyPath}
|
||||
|
||||
got := readPattern(t, tgt, dev)
|
||||
if got != want {
|
||||
t.Errorf("a same-node restart after a failed seal lost data: sha256 was %s, now %s.\n"+
|
||||
"This breaks the premise the dirty-volume design rests on — the local WAL is supposed "+
|
||||
"to be kept and recovered from on exactly this path.", want, got)
|
||||
return
|
||||
}
|
||||
harness.Step(t, "same-node restart recovered every acknowledged write from local state")
|
||||
}
|
||||
|
||||
// TestCrossNodeStartAfterFailedSealTakesOverLoudly pins the trade the design
|
||||
// makes. A volume whose seal failed has its freshest copy on one node, and
|
||||
// starting it elsewhere opens an older checkpoint from the backend.
|
||||
//
|
||||
// Starting elsewhere is nonetheless the right outcome: instance start forwards
|
||||
// to the node that last ran the instance and only falls back once that node has
|
||||
// failed its window, so refusing here would leave the instance unable to run at
|
||||
// all. What must never happen is the fallback being silent, so this asserts the
|
||||
// takeover is logged and names the node whose writes were left behind.
|
||||
func TestCrossNodeStartAfterFailedSealTakesOverLoudly(t *testing.T) {
|
||||
fix := requireStorageFaultFixture(t)
|
||||
if len(fix.Cluster.Nodes) < 3 {
|
||||
t.Skip("a cross-node start needs somewhere else to start: single-node clusters cannot reproduce this")
|
||||
}
|
||||
requireNoneFrozen(t, fix)
|
||||
|
||||
instanceID, tgt := ensureFaultInstance(t, fix)
|
||||
hostNode := harness.InstanceHostingNode(t, fix.Cluster, instanceID)
|
||||
if hostNode == nil {
|
||||
t.Fatalf("cannot identify the node hosting %s", instanceID)
|
||||
}
|
||||
|
||||
az := harness.DiscoverDefaultAZ(t, fix.Harness)
|
||||
volID := harness.EnsureVolume(t, fix.Harness, az, patternVolumeSizeGiB)
|
||||
|
||||
before := harness.GuestDiskSet(t, tgt)
|
||||
harness.AttachVolumeWait(t, fix.AWS, volID, instanceID, patternDevice)
|
||||
dev := harness.WaitForNewGuestDisk(t, tgt, before, 90*time.Second)
|
||||
t.Cleanup(func() { harness.DetachVolumeWait(t, fix.AWS, volID) })
|
||||
|
||||
want := writePattern(t, tgt, dev)
|
||||
harness.Detail(t, "instance", instanceID, "volume", volID, "host_node", hostNode.Name, "sha256", want)
|
||||
|
||||
freezeSet, _ := freezeSetFor(fix, hostNode)
|
||||
freezePredastore(t, fix, freezeSet)
|
||||
|
||||
harness.Step(t, "stopping %s with the backend frozen, so the seal fails", instanceID)
|
||||
_ = stopInstance(t, fix, instanceID)
|
||||
assertLocalStateKept(t, fix, hostNode, volID)
|
||||
|
||||
thawPredastore(t, fix, freezeSet)
|
||||
time.Sleep(recoverySettle)
|
||||
|
||||
// The node has to be gone, not merely quiet. A live viperblockd goes on
|
||||
// renewing the lease on a volume whose seal failed, and that refusal is
|
||||
// correct: the node could still be writing.
|
||||
harness.Step(t, "taking %s out so the start has to happen elsewhere", hostNode.Name)
|
||||
isolateNode(t, fix, *hostNode)
|
||||
|
||||
if startErr := startInstance(t, fix, instanceID); startErr != nil {
|
||||
t.Fatalf("the start was refused, so the instance can run nowhere while %s is down: %v\n"+
|
||||
"Falling back to the backend checkpoint is the intended behaviour here.", hostNode.Name, startErr)
|
||||
}
|
||||
|
||||
inst := harness.WaitForInstanceState(t, fix.AWS, instanceID, "running")
|
||||
landed := harness.InstanceHostingNode(t, fix.Cluster, instanceID)
|
||||
if landed != nil && landed.Index == hostNode.Index {
|
||||
t.Skipf("the instance started on %s again, so no cross-node start happened and this test proved nothing", hostNode.Name)
|
||||
}
|
||||
harness.Step(t, "the instance started on %s while %s was down", nodeName(landed), hostNode.Name)
|
||||
|
||||
// The loss is accepted; being unable to see it is not. This is the only
|
||||
// record that the volume opened from an older checkpoint than existed.
|
||||
assertTakeoverLogged(t, fix, landed, hostNode, volID)
|
||||
|
||||
host, port := harness.InstancePublicSSHHost(t, inst)
|
||||
if !harness.TryGuestSSHReady(host, port, "ubuntu", tgt.KeyPath, 5*time.Minute) {
|
||||
t.Errorf("the cross-node start was allowed and the guest never came back: opening from the "+
|
||||
"backend checkpoint has to yield a bootable volume, not an unusable one.%s",
|
||||
consoleTail(fix, instanceID))
|
||||
return
|
||||
}
|
||||
tgt = harness.SSHTarget{User: "ubuntu", Host: host, Port: port, KeyPath: tgt.KeyPath}
|
||||
|
||||
// Reading the pattern needs sudo, which the checkpoint may predate. Losing it
|
||||
// is the same accepted rollback, but only once cloud-init is ruled out: a guest
|
||||
// that found no datasource is the keyless-boot bug, not a storage outcome.
|
||||
if !guestSudoReady(tgt) {
|
||||
ds := guestDatasource(tgt)
|
||||
if strings.Contains(ds, "DataSourceNone") || ds == "" {
|
||||
t.Errorf("the guest came back unable to sudo and cloud-init found no datasource, so this "+
|
||||
"is IMDS failing to answer, not the checkpoint rolling back: %q%s", ds,
|
||||
consoleTail(fix, instanceID))
|
||||
return
|
||||
}
|
||||
harness.Step(t, "the pattern cannot be read: the root filesystem opened from a checkpoint "+
|
||||
"predating cloud-init's sudoers drop-in, the same documented cost as the pattern "+
|
||||
"changing. cloud-init did find its datasource (%s), so metadata was served.", ds)
|
||||
return
|
||||
}
|
||||
|
||||
if got := readPattern(t, tgt, dev); got == want {
|
||||
harness.Step(t, "the pattern also survived the takeover: %s had sealed enough of it", hostNode.Name)
|
||||
} else {
|
||||
harness.Step(t, "the pattern changed across the takeover (was %s, now %s), which is the "+
|
||||
"documented cost of starting from the backend checkpoint", want, got)
|
||||
}
|
||||
}
|
||||
|
||||
// guestSudoReady reports whether the guest can still sudo without a password.
|
||||
// A volume opened from the backend checkpoint may predate cloud-init writing
|
||||
// /etc/sudoers.d/90-cloud-init-users, which no read of the device can tell apart.
|
||||
func guestSudoReady(tgt harness.SSHTarget) bool {
|
||||
_, err := harness.GuestExecTimeout(tgt, "sudo -n true", 60*time.Second)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// guestDatasource returns what cloud-init recorded for this boot. result.json is
|
||||
// the file that names the datasource and is world-readable; status --long is the
|
||||
// fallback. This is the discriminator between a rollback and IMDS never answering.
|
||||
func guestDatasource(tgt harness.SSHTarget) string {
|
||||
out, err := harness.GuestExecTimeout(tgt,
|
||||
"cat /run/cloud-init/result.json 2>/dev/null; cloud-init status --long 2>/dev/null",
|
||||
60*time.Second)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.Join(strings.Fields(out), " ")
|
||||
}
|
||||
|
||||
// assertTakeoverLogged fails unless the node that opened the volume said so.
|
||||
//
|
||||
// This is the whole guarantee. Losing the previous holder's unsealed writes is
|
||||
// a deliberate trade against the instance not running at all, and it is only
|
||||
// defensible while an operator can find out it happened.
|
||||
func assertTakeoverLogged(t *testing.T, fix *Fixture, landed, previous *harness.Node, volID string) {
|
||||
t.Helper()
|
||||
if landed == nil {
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cmd := "sudo journalctl -u spinifex-viperblock -u spinifex-daemon --since '-10 min' --no-pager 2>/dev/null | " +
|
||||
"grep -F " + volID + " | grep -i 'last held by another node' | tail -5"
|
||||
out, err := fix.SSH.Run(ctx, *landed, cmd)
|
||||
if err != nil {
|
||||
t.Errorf("could not read %s's journal to confirm the takeover was logged: %v", landed.Name, err)
|
||||
return
|
||||
}
|
||||
journal := string(out)
|
||||
if strings.TrimSpace(journal) == "" {
|
||||
t.Errorf("%s opened %s from the backend checkpoint while %s held unsealed writes, and logged "+
|
||||
"nothing about it. The loss is acceptable; a silent loss is not.",
|
||||
landed.Name, volID, previous.Name)
|
||||
return
|
||||
}
|
||||
// The marker records the spinifex node name, which is the host's own name
|
||||
// and not the harness's nodeN label, so accept either identity.
|
||||
if !strings.Contains(journal, previous.Name) && !strings.Contains(journal, previous.Addr) {
|
||||
t.Errorf("%s logged a takeover for %s but did not name %s (%s), so an operator cannot tell "+
|
||||
"whose writes were left behind:\n%s", landed.Name, volID, previous.Name, previous.Addr, journal)
|
||||
return
|
||||
}
|
||||
harness.Step(t, "%s logged the takeover and named %s", landed.Name, previous.Name)
|
||||
}
|
||||
|
||||
// assertLocalStateKept checks the node kept the volume's local files and wrote
|
||||
// no seal receipt. Both are what a correctly failed seal leaves behind, and
|
||||
// together they are the ground truth a cluster-visible marker would index.
|
||||
func assertLocalStateKept(t *testing.T, fix *Fixture, node *harness.Node, volID string) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
base := viperblockBaseDir()
|
||||
dirCmd := fmt.Sprintf("sudo test -d %s/%s && echo PRESENT || echo ABSENT", base, volID)
|
||||
dirOut, err := fix.SSH.Run(ctx, *node, dirCmd)
|
||||
if err != nil {
|
||||
t.Fatalf("inspect local volume state on %s: %v", node.Name, err)
|
||||
}
|
||||
receiptCmd := fmt.Sprintf("sudo test -f %s/%s.sealed && echo PRESENT || echo ABSENT", base, volID)
|
||||
receiptOut, err := fix.SSH.Run(ctx, *node, receiptCmd)
|
||||
if err != nil {
|
||||
t.Fatalf("inspect seal receipt on %s: %v", node.Name, err)
|
||||
}
|
||||
|
||||
local := strings.TrimSpace(string(dirOut))
|
||||
receipt := strings.TrimSpace(string(receiptOut))
|
||||
harness.Detail(t, "node", node.Name, "local_state", local, "seal_receipt", receipt)
|
||||
|
||||
if receipt == "PRESENT" {
|
||||
t.Errorf("a seal receipt exists for %s on %s even though the backend was frozen: "+
|
||||
"the seal reported success it cannot have had", volID, node.Name)
|
||||
}
|
||||
if local == "ABSENT" {
|
||||
t.Errorf("%s kept no local state for %s after a failed seal. The un-uploaded writes were the "+
|
||||
"only current copy and they have been deleted — this is unrecoverable data loss, not a routing bug.",
|
||||
node.Name, volID)
|
||||
}
|
||||
}
|
||||
|
||||
// writePattern fills the head of the device with a reproducible pattern,
|
||||
// flushes it to the backend and returns its checksum. O_DIRECT plus an
|
||||
// explicit flush is what makes the write acknowledged rather than merely
|
||||
// buffered, which is the only kind this suite makes claims about.
|
||||
func writePattern(t *testing.T, tgt harness.SSHTarget, dev string) string {
|
||||
t.Helper()
|
||||
cmd := fmt.Sprintf(
|
||||
"sudo dd if=/dev/urandom of=/dev/%s bs=1M count=%d oflag=direct status=none && "+
|
||||
"sudo blockdev --flushbufs /dev/%s && sync",
|
||||
dev, patternMiB, dev)
|
||||
out, err := harness.GuestExecTimeout(tgt, cmd, 5*time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("write pattern to /dev/%s: %v\n%s\n%s", dev, err, out, deviceGeometry(tgt, dev))
|
||||
}
|
||||
return readPattern(t, tgt, dev)
|
||||
}
|
||||
|
||||
// readPattern re-reads the pattern region and returns its checksum. The page
|
||||
// cache is dropped first so the read comes from the device rather than from
|
||||
// what was written before the restart.
|
||||
//
|
||||
// Not O_DIRECT: this image ships uutils dd, whose O_DIRECT read path only
|
||||
// handles 512-byte transfers and fails EINVAL at 4k and above. Dropping the
|
||||
// cache gives the same guarantee and works under either dd.
|
||||
func readPattern(t *testing.T, tgt harness.SSHTarget, dev string) string {
|
||||
t.Helper()
|
||||
cmd := fmt.Sprintf(
|
||||
"set -o pipefail; sudo blockdev --flushbufs /dev/%[1]s && sync && "+
|
||||
"echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null && "+
|
||||
"sudo dd if=/dev/%[1]s bs=1M count=%[2]d status=none | sha256sum | cut -d' ' -f1",
|
||||
dev, patternMiB)
|
||||
out, err := harness.GuestExecTimeout(tgt, cmd, 5*time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("read pattern back from /dev/%s: %v\n%s\n%s", dev, err, out, deviceGeometry(tgt, dev))
|
||||
}
|
||||
sum := strings.TrimSpace(out)
|
||||
if len(sum) != 64 {
|
||||
t.Fatalf("expected a sha256 from /dev/%s, got %q\n%s", dev, sum, deviceGeometry(tgt, dev))
|
||||
}
|
||||
return sum
|
||||
}
|
||||
|
||||
// stopInstance stops the instance and waits for it to settle. The error is
|
||||
// returned rather than fatal: whether a stop against a dead backend succeeds
|
||||
// or fails is one of the things under test.
|
||||
func stopInstance(t *testing.T, fix *Fixture, instanceID string) error {
|
||||
t.Helper()
|
||||
if _, err := fix.AWS.EC2.StopInstances(&ec2.StopInstancesInput{
|
||||
InstanceIds: []*string{aws.String(instanceID)},
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
deadline := time.Now().Add(stopStartTimeout)
|
||||
for time.Now().Before(deadline) {
|
||||
switch instanceStateOf(fix, instanceID) {
|
||||
case "stopped":
|
||||
return nil
|
||||
case "error":
|
||||
return fmt.Errorf("instance %s entered error state during stop", instanceID)
|
||||
}
|
||||
time.Sleep(10 * time.Second)
|
||||
}
|
||||
return fmt.Errorf("instance %s did not reach stopped within %s", instanceID, stopStartTimeout)
|
||||
}
|
||||
|
||||
// startInstance starts the instance, returning the API error if the start is
|
||||
// refused. A refusal is the desired behaviour for a dirty volume, so this is a
|
||||
// value the caller inspects rather than a failure.
|
||||
func startInstance(t *testing.T, fix *Fixture, instanceID string) error {
|
||||
t.Helper()
|
||||
_, err := fix.AWS.EC2.StartInstances(&ec2.StartInstancesInput{
|
||||
InstanceIds: []*string{aws.String(instanceID)},
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// isolatedUnits are the two services that have to be down for a node to look
|
||||
// gone. The daemon so ec2.start.{lastNode} has no responder, and viperblock so
|
||||
// nothing on this node keeps renewing the lease it holds on the volume.
|
||||
//
|
||||
// Not the whole target: predastore stays up so the node keeps serving its shard,
|
||||
// which is what a node whose control plane died looks like from the cluster.
|
||||
var isolatedUnits = []string{"spinifex-daemon", "spinifex-viperblock"}
|
||||
|
||||
// leaseExpiryBudget is how long to wait for an unrenewed volume lease to age
|
||||
// out of JetStream once its holder is gone. Comfortably past the 45s TTL, since
|
||||
// the entry only disappears on the bucket's own schedule.
|
||||
const leaseExpiryBudget = 75 * time.Second
|
||||
|
||||
// isolateNode stops the control-plane services on node and restores them on
|
||||
// cleanup, then waits for the volume lease the node held to expire.
|
||||
//
|
||||
// Waiting is the point. Without it the start elsewhere is refused by a lease
|
||||
// that is simply still current, which says nothing about what happens when its
|
||||
// holder is not coming back.
|
||||
func isolateNode(t *testing.T, fix *Fixture, node harness.Node) {
|
||||
t.Helper()
|
||||
t.Cleanup(func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 180*time.Second)
|
||||
defer cancel()
|
||||
for _, unit := range isolatedUnits {
|
||||
if _, err := fix.SSH.Run(ctx, node, "sudo systemctl start "+unit); err != nil {
|
||||
t.Errorf("could not restart %s on %s — the node is left without it: %v", unit, node.Name, err)
|
||||
continue
|
||||
}
|
||||
t.Logf("restarted %s on %s", unit, node.Name)
|
||||
}
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 180*time.Second)
|
||||
defer cancel()
|
||||
for _, unit := range isolatedUnits {
|
||||
if _, err := fix.SSH.Run(ctx, node, "sudo systemctl stop "+unit); err != nil {
|
||||
t.Fatalf("stop %s on %s: %v", unit, node.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
harness.Step(t, "waiting %s for %s's volume lease to age out", leaseExpiryBudget, node.Name)
|
||||
time.Sleep(leaseExpiryBudget)
|
||||
}
|
||||
|
||||
func errOrOK(err error) string {
|
||||
if err == nil {
|
||||
return "success"
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
// consoleTail returns the guest's serial console for a restart that never
|
||||
// answered SSH, formatted to append to a failure message. Whether the kernel
|
||||
// booted and stalled or never got that far is only visible here, and the log is
|
||||
// gone once the fixture terminates the instance.
|
||||
func consoleTail(fix *Fixture, instanceID string) string {
|
||||
console, err := harness.InstanceConsole(fix.AWS, instanceID)
|
||||
if err != nil {
|
||||
return fmt.Sprintf("\nconsole unavailable: %v", err)
|
||||
}
|
||||
if strings.TrimSpace(console) == "" {
|
||||
return "\nconsole was empty, so the guest produced no serial output at all"
|
||||
}
|
||||
const tailBytes = 4096
|
||||
if len(console) > tailBytes {
|
||||
console = console[len(console)-tailBytes:]
|
||||
}
|
||||
return "\nguest console (last " + fmt.Sprint(len(console)) + " bytes):\n" + console
|
||||
}
|
||||
|
||||
// deviceGeometry reports what the guest kernel thinks the device is, for a
|
||||
// pattern write or read that failed. An alignment or size mismatch is the usual
|
||||
// cause of an EINVAL on an O_DIRECT transfer, and neither is visible in dd's
|
||||
// own message.
|
||||
func deviceGeometry(tgt harness.SSHTarget, dev string) string {
|
||||
cmd := fmt.Sprintf(
|
||||
"for p in getsize64 getss getpbsz getiomin getioopt; do "+
|
||||
"printf '%%s=%%s ' $p $(sudo blockdev --$p /dev/%[1]s 2>&1); done; echo; "+
|
||||
"dd --version | head -1; "+
|
||||
"echo -n 'direct 512: '; sudo dd if=/dev/%[1]s bs=512 count=1 iflag=direct of=/dev/null 2>&1 | tail -1; "+
|
||||
"echo -n 'direct 4k: '; sudo dd if=/dev/%[1]s bs=4k count=1 iflag=direct of=/dev/null 2>&1 | tail -1; "+
|
||||
"echo -n 'direct 1M: '; sudo dd if=/dev/%[1]s bs=1M count=1 iflag=direct of=/dev/null 2>&1 | tail -1; "+
|
||||
"echo -n 'buffered 1M: '; sudo dd if=/dev/%[1]s bs=1M count=1 of=/dev/null 2>&1 | tail -1",
|
||||
dev)
|
||||
out, err := harness.GuestExecTimeout(tgt, cmd, time.Minute)
|
||||
if err != nil {
|
||||
return fmt.Sprintf("device geometry unavailable: %v\n%s", err, out)
|
||||
}
|
||||
return "device geometry:\n" + out
|
||||
}
|
||||
@@ -568,7 +568,7 @@ func ensureImagePullInstance(t *testing.T, fix *Fixture, launch func(ami, instTy
|
||||
t.Helper()
|
||||
instType, arch := imagePullInstanceType(t, fix)
|
||||
ami := harness.DiscoverUbuntuAMI(t, fix.Harness, arch)
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness, fix.ArtifactDir(t))
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness)
|
||||
|
||||
vpcID := harness.CreateVPC(t, fix.AWS, imagePullVPCCIDR)
|
||||
t.Cleanup(func() { harness.DeleteVPC(t, fix.AWS, vpcID) })
|
||||
|
||||
@@ -390,7 +390,7 @@ func ensureWorkloadInstance(t *testing.T, fix *Fixture) (instanceID string, tgt
|
||||
t.Helper()
|
||||
instType, arch := harness.DiscoverNanoInstanceType(t, fix.Harness)
|
||||
ami := harness.DiscoverUbuntuAMI(t, fix.Harness, arch)
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness, fix.ArtifactDir(t))
|
||||
keyName, keyPath := harness.EnsureKeyPair(t, fix.Harness)
|
||||
vpc := harness.EnsureDefaultVPC(t, fix.Harness)
|
||||
harness.AuthorizeSSHIngress(t, fix.AWS, vpc.SGID)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user