From 021f439baba7958ef78ae3a9d059034c3f6038ae Mon Sep 17 00:00:00 2001 From: Ben Duncan <5178243+benduncan@users.noreply.github.com> Date: Sun, 27 Sep 2026 15:56:09 -0500 Subject: [PATCH] feat: recover instances automatically from a node that stopped heartbeating (#1111) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: recover instances automatically from a node that stopped heartbeating A node that dies hard takes its guests with it, and nothing brings them back until an operator brings that node back. Every daemon now runs the same small reconciler: it reads the instance records, selects desired-running instances whose owner is stale, and relaunches them here. The claim is a CAS on Status.LastNode and that is the whole of the coordination. Several survivors race one instance, exactly one revision wins, and the losers find a record naming a live owner and stop considering it. A claimant that dies mid-launch leaves the record naming a node that is itself stale, which is the same candidate the next pass derives again, so there is no lease to expire and nothing to reconcile by hand. Ownership is not permission to write. The volume leases still decide whether the guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt that loses the volume race has touched no tap, no OVS port and no OVN binding. A storage failure is not a host failure and wants the opposite response. A guest paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is refused: the volumes are the same objects in the same store from anywhere, so relaunching moves the pause rather than ending it. A node whose own object store is not answering stands down entirely, which separates a store broken here from a store broken everywhere without either having to be diagnosed. A launch that does not complete backs this node off that instance, doubling to a cap, because nothing on the mount path can yet say which of the two it hit. Recovery is off by default and inert below three nodes. It also waits until it has seen its peers live, or five minutes, so the first node back from a coordinated shutdown does not recover guests from nodes that are still booting. * test: prove instance recovery moves a host failure and refuses a storage failure The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves. Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live. harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down. * fix: make automatic instance recovery a cluster-wide setting Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file. Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it. * test: drive real I/O into the storage-fault case, and freeze the store rather than stop it Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted. The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them. The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three. * test: give instance recovery its own suite and its own nightly cell Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five. Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set. * fix: give a guest long enough to open a volume the object store is slow to serve QEMU opens every drive before its monitor answers a greeting, so the QMP greeting deadline is really a deadline on opening the guest's volumes. At 30s it was shorter than a cold viperblock open costs on a cluster that has lost a node, where every object read is served degraded: measured at 35s with one node of three away. The launch killed the guest five seconds short of ready and then did it again on every retry, which is a recovery that can never complete. A network-backed drive now gets its own deadline for the same reason VFIO already had one — synchronous work before the monitor answers — and the instancerecovery suite is what found it. Also make the suite's predastore thaw best effort per node. It runs after a node has been taken away, where there is no process to signal, and a node it cannot reach must not stop it restoring the rest. * docs: name the two newest cells in the dispatch input The cells input is the operator's index of what a number means, and it stopped at 29 while 30 and 31 were added. * test: prove the guest is writing before freezing the store in the recovery suite The storage-fault test froze the object store as soon as RunInstances reported running, which is when QEMU has launched rather than when the guest has booted. On the nightly cluster the freeze landed while the guest was still in UEFI, so the only drive it could reach was the EFI varstore — and pflash has no werror=stop to pause on, so the guest never paused and the test reported a product failure it had caused itself. The load driver now prints to the serial console after each pass it has written and synced, and the test waits for that before freezing. The console is read through GetConsoleOutput, so this still needs no login and still runs on cells whose guests have no public address. * feat: recover instances automatically with nothing to configure The switch is gone. An instance whose host has failed is down whether or not recovery is enabled, so the only outcome a disabled reconciler can produce is leaving it down — which is the state recovery exists to end, not a safer one to fall back to. Against that, the switch cost a cluster that can be configured into never recovering anything, with no symptom until a node dies. This feature already hit that once: a top-level table parsed into nothing and every node reported recovery off with the setting plainly in the file. What makes recovery inert is now only facts about the cluster — no cluster config, no JetStream, or fewer than three nodes — and each says in the log which one applied. The e2e no longer writes config or restarts daemons: it waits for every node to report the reconciler active on a cluster nothing configured, so reintroducing a switch fails there as well as in the unit tests. * feat: stop retrying a recovery that is never going to work Every failed attempt used to go to the same exponential backoff and stay there. An instance nothing could launch was retried every fifteen minutes forever, still observed-running on a node that is gone, with no account of it anywhere the customer or an operator would look. A failure is now classified and counted. Two classes are read from what the launch actually reports: the node it was taken from still holds a volume lease, which clears on that lease's clock and is retried on a faster one, and this node has no room, which another survivor may. There is no terminal class, because nothing on this path can prove one — the launch collapses everything else into ServerInternal, so a verdict of "never going to work" would be a guess, and the guess that abandons a recoverable instance is the expensive one. The budget bounds the unknown case instead. Every class counts against it, since a class that did not would be the old unbounded retry under a narrower condition. Exhausting it leaves the instance where AWS leaves one whose host failed and could not be brought back: stopped, carrying a StateReason that names the cause and the last error, one StartInstances from running again. Writing the desired state down is also what makes one node's decision cluster-wide, because selection reads desired state — the same property the claim already has. A record another node has since claimed is never touched, so a node exhausting its attempts cannot stop a guest that is now running elsewhere. * fix: give a guest the time for every cost it pays before its monitor answers A GPU instance booting from a network-backed root volume pays both costs: QEMU opens every drive across the object store and then pins the whole of guest RAM, one after the other, all before the monitor answers. The deadline took whichever was larger, so such a guest was given less time than it needs. That the two floors are both 180s is why nothing has been seen yet — the VFIO branch has always returned at least the network-drive deadline. It is the same defect either way, and the symptom when it lands is a GPU instance SIGKILLed on every attempt while its plain sibling launches, which reads as a GPU fault rather than as a deadline. The costs are now added, the plain QEMU startup is counted once rather than replaced, and the cap covers both floors together so it cannot clip a guest before either cost has been waited out. * test: assert the AWS-visible arc of a host failure end to end A customer cannot see which host an instance is on, and AWS never tells them. What they can see is DescribeInstanceStatus, so that is where a recovery has to be provable, and the suite never looked at it. Three frames are now required, in order: running with system-status ok before anything is broken, system-status impaired while the owning node is gone, and ok again once a survivor is running it. The middle frame is the one worth having — the window opens when the owner's heartbeat goes stale, which is strictly before any survivor may claim, and closes when the relaunch registers, so it is bounded rather than lucky. Reporting ok throughout would be the more comfortable lie. The availability zone is compared across the first and last frames as well. A recovery that crossed one would break every placement promise made to the customer, and nothing else in the suite would have noticed. * test: prove a half-healthy node fences itself and is never a second writer The two existing recovery cases take a node away by stopping its services, which is the failure this feature handles least interestingly: a node that is gone has no opinion about who owns its volumes. The one that matters is the node that is still up, still running QEMU, still answering its own local clients, and simply cannot reach consensus. That is the state in which two nodes can each believe they own a volume, and the only thing between that and a corrupted disk is that the partitioned node gives up first, on a clock it can evaluate without reaching anything. So this drops NATS on 4222 and 4248 between one node and its peers and nothing else, then asserts the ordering the safety argument rests on rather than the outcome it produces. A watch samples every node every three seconds for the whole run, including the heal, and fails on any single sample naming two hosts. Two guests writing one volume through one object store is data loss already under way, not a race that resolves, and no point-in-time assertion can see one that appeared between its own samples. The injector takes its own nft table rather than touching inet spinifex_filter, because a test that flushed the node's real firewall would leave it either unfirewalled or misfirewalled with no sign of which, and a separate table at priority -300 heals in one atomic delete. A transient systemd timer removes it independently, so a cancelled workflow does not leave a node cut off from its cluster until somebody notices. It proves SSH still works and proves a peer port is genuinely unreachable before anything is asserted, since a rule set that installed cleanly and dropped nothing would read as the system tolerating a partition. The suite budget goes to 90m and cell 31 to 105m to fit the third case. * fix: move an instance's boot state, volumes, logical port and address with it A record-driven launch assumes the node reading the record is the node that wrote it. On a daemon restart that holds. On a recovery onto a survivor it does not, and five things follow from it — one that stops a moved instance booting at all, and four that stop it running anywhere. A load balancer never comes back from a host failure. Its boot configuration is a set of files under the runtime directory and the record holds their paths, so the survivor restores paths that exist only on the node that died. The hook that rebuilds them was reachable from Restore alone, and the cross-node path goes straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert that are not there. The gateway URL baked into that env names the creating node's management bridge as well, which the rebuild corrects because it rebuilds from whichever node is running it. Firing the hook on both paths fixes both, and a rebuild that fails now refuses the launch: a guest that boots without its network configuration is worse than one that does not boot, because the records would say it recovered. The export outlives the guest and holds the volume lease. A returning node is healthy, so it renews that lease indefinitely and the new owner never acquires: the guest is stopped here and refused there, recovery spends its attempt budget on a volume that was never going to be free, and gives up on an instance that was never unrecoverable. So the export is given up too, over a new node-addressed request that tears it down exactly as the fence does. Deliberately not an unmount — an unmount seals, this node's block map is the stale one, and sealing publishes it over the winner's. Unlike the fence it deletes the lease key rather than leaving it to expire, because the lease is still this node's to hand back. The tap outlives the guest, and the tap is what claims the instance's OVN logical port. The binding names one chassis, so two nodes offering the same iface-id do not split the traffic — they contend, and the address answers from whichever won last. It also blocks the layer above: the external IP's host ingress is pruned on the evidence that the owning guest sits on another chassis, and that stays false while this node is still claiming the port. The management address outlives the guest as a cluster-wide reservation, and the release was unconditional. br-mgmt is one flat L2 segment, so the node an instance left was freeing an address still in use and offering it to the next instance anywhere. Only the node an entry names may release it now, with the takeover recorded on arrival so the node actually running the instance can still give the address back. An entry written before the node field existed names nobody and stays releasable, because there is no other holder to collide with. The fence path already cleaned up the tap, so the release fix repairs that path too. * test: prove a load balancer keeps its name, its address and its traffic when its host dies Takes away the node running an internet-facing ALB and asserts the four things a customer of one actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again without anybody being called, and the backends are healthy again from the recovered load balancer's point of view. The backends are placed around the load balancer rather than beside it. A load balancer lands wherever a node answers its queue group, so its node is discovered and the backends are relaunched until both are elsewhere; one on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve. DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve through the same server, so the claim under test is what this cluster answers for a name rather than what the machine running the tests was configured to forward to. A watch samples every surviving node every five seconds for the whole run, because an answer that went missing or pointed elsewhere for one reconcile pass is the outage, and no single sample reaches it. The returning node then has to give up what it no longer owns: no second copy of the guest, and no tap still claiming the load balancer's logical port, since OVN binds a port to one chassis and a second claimant contends for it rather than splitting the traffic. The last assertion is that the name still serves with that node back in the cluster. * docs: name the new cell in the nightly's cell list * test: stop the disk-load guest from filling the console it is judged by The load generator announced every write, once a second, for as long as the test ran. The console is a ring buffer this suite reads back, so an hour of that pushes out everything explaining a failure, including the kernel's own account of the I/O error the test exists to produce. Announce twenty times, then keep writing quietly. * docs: say what the OCI guide has been proven on, and what a host failure does The guide read as a support statement for OCI rather than as an account of one tested path, promised seven steps and then gave eight, and said nothing about what happens to a guest or its public address when the node under it fails. State the evidence, state that the nightly guards the datapath and not the cloud, and give the failure behaviour as a table with the reason codes a customer will actually see. Also say plainly what the volume lease is: one running guest and a bounded, logged window, not a write barrier the object store enforces. Someone sizing a workload against this should read that before they read the table. * docs: say why the mount classifier refuses to retry a lease another node holds It reads as an omission — a lease held elsewhere is the most transient refusal there is. It is not. This classifier drives the restore path, where the holder is a node that is running the guest, so retrying would relaunch against a live owner. Recovery wants the opposite and reads the refusal itself. * feat: claim a recovered guest's public address in line, and verify it arrived A recovered instance was reported running before anything moved its public address off the node that died, so for up to fifteen seconds the records said recovered and the address answered nothing. The recovery path now runs the affinity pass itself, which is idempotent and is the same pass the ticker runs. Two things the pass could not do before. It abandoned every remaining address on the first API failure, though a throttle or a denial is per-object and the guests behind the others are just as dark. And it trusted the move's own response, which is OCI agreeing to the request rather than OCI having done it, so an accepted move that never landed read as success. It now re-reads what OCI delivers here and reports the difference. An address that will not arrive makes the guest a system-status failure with the time it went dark, the same shape AWS uses for a host-level fault. The guest passes every check it can be asked about itself, which is exactly why reporting it healthy was the worst available answer. * feat: give a recovery attempt one trace across all three of its phases The volumes opening, QEMU starting and its monitor answering already trace themselves, but nothing tied them to the attempt that asked for them: three orphan traces per recovery, no way to ask which phase a slow one spent its time in, and nothing naming the node the guest came from. A span over the attempt makes them its children, and the reconciler's own log lines carry its trace ID because the handler stamps it from the context. A failed attempt records the fault and the attempt count on the span, so the budget running out is readable as a sequence rather than as twelve unrelated warnings. * fix: finish a fence once its writer finally exits A kill that did not take effect was logged, counted and dropped. The export stayed registered and the lease stayed held, which is correct while the writer lives and wrong forever after it dies: a lease a healthy node keeps renewing never expires, so the volume could not start anywhere. The failure is now re-reported every 30 s until it resolves, and the teardown completes as soon as the writer is confirmed gone. Only a delivered kill is waited on — a refused one says nothing about what became of the writer. * test: stop judging a detached stop or terminate on a transient state Both tests read the instance state immediately after the ack, but the transition runs on its own goroutine, so the read races it in both directions: too early sees running, too late sees the final state. The race gate caught the first of those on a loaded runner. * fix: replicate the volume lease and dirty buckets across the cluster Both were created with the default single replica, so both lived on one node. That node holds the answer to "who may write this volume" and the marker every mount reads, and losing it takes volume leasing and mounting down cluster-wide. A partitioned node was fenced correctly and then no survivor could relaunch its guest: every attempt failed on "read dirty mark: no responders available", for ten minutes, until the budget ran out. The replica count is now the cluster's node count, as the gateway's own buckets already do. Existing buckets are raised in place rather than requiring a wipe. The e2e diagnostics that should have said this are fixed too. The fence and the surrender are logged by the storage service, not the daemon, so the assertions read the wrong journal; and Eventuallyf evaluates its message at the call site, so a ten-minute wait was explained by a log that stopped before it began. --- .github/e2e-cells.json | 24 + .github/scripts/suite-sets.sh | 26 +- .github/workflows/e2e-nightly.yml | 2 +- cmd/spinifex/cmd/service.go | 1 + docs/coverage/README.md | 6 +- docs/coverage/coverage.json | 9 +- docs/coverage/internal-report.md | 5 +- docs/coverage/s3/README.md | 5 +- docs/oci-integration/README.md | 34 +- docs/testing/e2e-coverage.md | 32 +- spinifex/daemon/daemon.go | 71 ++- spinifex/daemon/daemon_handlers_test.go | 24 +- spinifex/daemon/daemon_mgmt_ip.go | 100 +++- spinifex/daemon/daemon_mgmt_ip_test.go | 195 ++++++ spinifex/daemon/daemon_system_instance.go | 18 + spinifex/daemon/instance_claim_test.go | 239 ++++++++ spinifex/daemon/instance_membership.go | 35 ++ spinifex/daemon/instance_recovery.go | 560 ++++++++++++++++++ spinifex/daemon/instance_recovery_test.go | 410 +++++++++++++ .../daemon/instance_recovery_trace_test.go | 42 ++ spinifex/daemon/jetstream.go | 115 ++++ spinifex/daemon/vm_adapters.go | 51 +- spinifex/daemon/vm_adapters_test.go | 21 +- .../instance/launch_claimed_relaunch_test.go | 81 +++ .../handlers/ec2/instance/service_impl.go | 155 +++-- .../ec2/instance/service_impl_test.go | 30 + .../ec2/instance/start_public_ip_test.go | 1 + .../instance/volume_held_elsewhere_test.go | 6 + spinifex/network/external/ocinet/affinity.go | 64 +- .../network/external/ocinet/affinity_test.go | 57 ++ .../services/viperblockd/provider_handlers.go | 9 +- .../viperblockd/provider_handlers_test.go | 5 + .../services/viperblockd/vbwire/vbwire.go | 35 ++ spinifex/services/viperblockd/viperblockd.go | 50 +- .../viperblockd_integration_test.go | 2 +- .../viperblockd/volume_abandon_test.go | 131 ++++ .../volume_bucket_replicas_test.go | 62 ++ spinifex/services/viperblockd/volume_dirty.go | 10 +- .../services/viperblockd/volume_dirty_test.go | 2 +- spinifex/services/viperblockd/volume_fence.go | 192 +++++- .../services/viperblockd/volume_fence_test.go | 69 ++- spinifex/services/viperblockd/volume_lease.go | 10 +- .../services/viperblockd/volume_lease_test.go | 4 +- spinifex/vm/address_health.go | 54 ++ spinifex/vm/address_health_test.go | 86 +++ spinifex/vm/errors.go | 6 + spinifex/vm/forget_superseded_test.go | 127 ++++ spinifex/vm/lifecycle.go | 64 +- spinifex/vm/lifecycle_test.go | 54 +- spinifex/vm/manager.go | 8 +- spinifex/vm/manager_deps_test.go | 10 + spinifex/vm/restore.go | 34 +- spinifex/vm/restore_test.go | 1 + spinifex/vm/shutdown.go | 45 ++ spinifex/vm/vm.go | 7 + spinifex/vm/volume_mounter.go | 10 + tests/e2e/Makefile | 10 +- tests/e2e/harness/dns.go | 45 ++ tests/e2e/harness/instance_status.go | 60 ++ tests/e2e/harness/multinode_distribution.go | 31 +- tests/e2e/harness/partition.go | 207 +++++++ tests/e2e/harness/traffic.go | 8 +- tests/e2e/instancerecovery/main_test.go | 104 ++++ tests/e2e/instancerecovery/partition_test.go | 249 ++++++++ tests/e2e/instancerecovery/recovery_test.go | 491 +++++++++++++++ tests/e2e/lbrecovery/alb_recovery_test.go | 389 ++++++++++++ tests/e2e/lbrecovery/fixture_test.go | 507 ++++++++++++++++ tests/e2e/lbrecovery/main_test.go | 149 +++++ .../lbrecovery/testdata/target-userdata.sh | 9 + 69 files changed, 5598 insertions(+), 167 deletions(-) create mode 100644 spinifex/daemon/instance_claim_test.go create mode 100644 spinifex/daemon/instance_recovery.go create mode 100644 spinifex/daemon/instance_recovery_test.go create mode 100644 spinifex/daemon/instance_recovery_trace_test.go create mode 100644 spinifex/handlers/ec2/instance/launch_claimed_relaunch_test.go create mode 100644 spinifex/services/viperblockd/volume_abandon_test.go create mode 100644 spinifex/services/viperblockd/volume_bucket_replicas_test.go create mode 100644 spinifex/vm/address_health.go create mode 100644 spinifex/vm/address_health_test.go create mode 100644 spinifex/vm/forget_superseded_test.go create mode 100644 tests/e2e/harness/instance_status.go create mode 100644 tests/e2e/harness/partition.go create mode 100644 tests/e2e/instancerecovery/main_test.go create mode 100644 tests/e2e/instancerecovery/partition_test.go create mode 100644 tests/e2e/instancerecovery/recovery_test.go create mode 100644 tests/e2e/lbrecovery/alb_recovery_test.go create mode 100644 tests/e2e/lbrecovery/fixture_test.go create mode 100644 tests/e2e/lbrecovery/main_test.go create mode 100644 tests/e2e/lbrecovery/testdata/target-userdata.sh diff --git a/.github/e2e-cells.json b/.github/e2e-cells.json index 0649da823..b895186d2 100644 --- a/.github/e2e-cells.json +++ b/.github/e2e-cells.json @@ -154,5 +154,29 @@ "runner": "ci-multi", "extra": "nat-multi", "suite": "" + }, + { + "cell": 31, + "install": "binary", + "topo": "real-multi", + "net": "static", + "host": "debian-13", + "arch": "x86_64", + "runner": "ci-multi", + "extra": "", + "suite": "instancerecovery", + "timeout_minutes": 105 + }, + { + "cell": 32, + "install": "binary", + "topo": "real-multi", + "net": "static", + "host": "debian-13", + "arch": "x86_64", + "runner": "ci-multi", + "extra": "", + "suite": "lbrecovery", + "timeout_minutes": 90 } ] diff --git a/.github/scripts/suite-sets.sh b/.github/scripts/suite-sets.sh index 5cc68eb4c..6818406ed 100644 --- a/.github/scripts/suite-sets.sh +++ b/.github/scripts/suite-sets.sh @@ -40,7 +40,18 @@ E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota storagefault" # Suites runnable against a multi-node environment. -E2E_SUITES_MULTI="multinode lb cert quota storagefault" +# +# instancerecovery is multi-only and needs three: recovery is inert below that, +# because there is no survivor to move a guest to that is not the node that +# failed. Like storagefault it takes a node away and freezes a cluster-wide +# service, so it runs last and owns its environment. +# +# lbrecovery is multi-only and needs three for two reasons rather than one: a +# survivor to move the load balancer to, and a node for its backends that is not +# the load balancer's, so taking that node away does not take the backends with +# it. It runs beside instancerecovery at the end because it also takes a node +# away and owns its environment while it does. +E2E_SUITES_MULTI="multinode lb cert quota lbrecovery instancerecovery storagefault" # grep -xE alternation form, for narrowing a requested set to the eligible one. E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')" @@ -88,6 +99,19 @@ e2e_suite_timeout() { # SIGSTOPped for seven minutes to prove a guest survives it. A full pass on # dev-prod is ~63 minutes, so the 30m default truncated it mid-suite. storagefault) echo "90m" ;; + # instancerecovery is almost entirely waiting, and none of it is avoidable: + # a dead owner has to be seen stale twice, a guest has to boot, and the + # storage case waits for a pause measured at 4m47s and then holds four + # minutes to prove nothing moved. Three tests, each taking a node away and + # bringing it back, and the partition case adds a lease validity, a lease + # TTL and a boot on top of everything the other two wait for. + instancerecovery) echo "90m" ;; + # lbrecovery pays for a load balancer twice: once to stand one up with two + # backends placed around it, and once for the recovery itself — a stale + # owner, a lease, a boot, HAProxy's own start, and its health checks against + # the backends before it forwards anything. Then the node comes back and has + # to be shown not to answer for an address it no longer holds. + lbrecovery) echo "75m" ;; *) echo "30m" ;; esac } diff --git a/.github/workflows/e2e-nightly.yml b/.github/workflows/e2e-nightly.yml index 4039720e8..bd706cc95 100644 --- a/.github/workflows/e2e-nightly.yml +++ b/.github/workflows/e2e-nightly.yml @@ -10,7 +10,7 @@ on: workflow_dispatch: inputs: cells: - description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes" + description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes" required: false permissions: diff --git a/cmd/spinifex/cmd/service.go b/cmd/spinifex/cmd/service.go index 2026f1b4f..edfa0ace9 100644 --- a/cmd/spinifex/cmd/service.go +++ b/cmd/spinifex/cmd/service.go @@ -384,6 +384,7 @@ var viperblockStartCmd = &cobra.Command{ SecretKey: nodeConfig.Predastore.SecretKey, BaseDir: nodeConfig.Predastore.BaseDir, NodeName: clusterConfig.Node, + KVReplicas: len(clusterConfig.Nodes), ShardWAL: shardWAL, GCEnabled: gcEnabled, WALBaseDir: nodeConfig.Viperblock.WALBaseDir, diff --git a/docs/coverage/README.md b/docs/coverage/README.md index 884439dc8..4f95333e6 100644 --- a/docs/coverage/README.md +++ b/docs/coverage/README.md @@ -17,7 +17,7 @@ tags: ## Overview -The platform serves **389 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand. +The platform serves **392 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand. | Service | Operations | |---|---:| @@ -29,6 +29,6 @@ The platform serves **389 operations** across the AWS APIs below. Every page nam | [ELBv2](/coverage/elbv2) | 34 | | [IAM](/coverage/iam) | 76 | | [RDS](/coverage/rds) | 26 | -| [S3](/coverage/s3) | 23 | +| [S3](/coverage/s3) | 26 | | [STS](/coverage/sts) | 5 | -| **Total** | **389** | +| **Total** | **392** | diff --git a/docs/coverage/coverage.json b/docs/coverage/coverage.json index 22acc4c5b..71e56c318 100644 --- a/docs/coverage/coverage.json +++ b/docs/coverage/coverage.json @@ -3049,15 +3049,18 @@ "DeleteObjects", "GetBucketLocation", "GetBucketTagging", + "GetBucketVersioning", "GetObject", "HeadBucket", "HeadObject", "ListBuckets", "ListMultipartUploads", + "ListObjectVersions", "ListObjects", "ListObjectsV2", "ListParts", "PutBucketTagging", + "PutBucketVersioning", "PutObject", "UploadPart", "UploadPartCopy" @@ -3074,15 +3077,18 @@ "DeleteObjects", "GetBucketLocation", "GetBucketTagging", + "GetBucketVersioning", "GetObject", "HeadBucket", "HeadObject", "ListBuckets", "ListMultipartUploads", + "ListObjectVersions", "ListObjects", "ListObjectsV2", "ListParts", "PutBucketTagging", + "PutBucketVersioning", "PutObject", "UploadPart", "UploadPartCopy" @@ -3122,7 +3128,6 @@ "GetBucketPolicyStatus", "GetBucketReplication", "GetBucketRequestPayment", - "GetBucketVersioning", "GetBucketWebsite", "GetObjectAcl", "GetObjectAttributes", @@ -3137,7 +3142,6 @@ "ListBucketInventoryConfigurations", "ListBucketMetricsConfigurations", "ListDirectoryBuckets", - "ListObjectVersions", "PutBucketAccelerateConfiguration", "PutBucketAcl", "PutBucketAnalyticsConfiguration", @@ -3155,7 +3159,6 @@ "PutBucketPolicy", "PutBucketReplication", "PutBucketRequestPayment", - "PutBucketVersioning", "PutBucketWebsite", "PutObjectAcl", "PutObjectLegalHold", diff --git a/docs/coverage/internal-report.md b/docs/coverage/internal-report.md index fe2dfaa5f..b896310c5 100644 --- a/docs/coverage/internal-report.md +++ b/docs/coverage/internal-report.md @@ -969,7 +969,7 @@ option-groups: Option groups configure engine add-ons for engines this platform ## s3 -`2006-03-01` — 23 of 99 modelled operations implemented (23.2%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model. +`2006-03-01` — 26 of 99 modelled operations implemented (26.3%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model. ### Not implemented @@ -1002,7 +1002,6 @@ option-groups: Option groups configure engine add-ons for engines this platform - `GetBucketPolicy` - `GetBucketPolicyStatus` - `GetBucketReplication` -- `GetBucketVersioning` - `GetBucketWebsite` - `GetObjectAcl` - `GetObjectAttributes` @@ -1017,7 +1016,6 @@ option-groups: Option groups configure engine add-ons for engines this platform - `ListBucketInventoryConfigurations` - `ListBucketMetricsConfigurations` - `ListDirectoryBuckets` -- `ListObjectVersions` - `PutBucketAcl` - `PutBucketAnalyticsConfiguration` - `PutBucketCors` @@ -1033,7 +1031,6 @@ option-groups: Option groups configure engine add-ons for engines this platform - `PutBucketOwnershipControls` - `PutBucketPolicy` - `PutBucketReplication` -- `PutBucketVersioning` - `PutBucketWebsite` - `PutObjectAcl` - `PutObjectLegalHold` diff --git a/docs/coverage/s3/README.md b/docs/coverage/s3/README.md index dba82bee4..e63151c3e 100644 --- a/docs/coverage/s3/README.md +++ b/docs/coverage/s3/README.md @@ -18,7 +18,7 @@ tags: ## Overview -Predastore implements **23 operations** in the S3 `2006-03-01` API model. +Predastore implements **26 operations** in the S3 `2006-03-01` API model. ### Predastore serves this endpoint @@ -45,15 +45,18 @@ That behaviour is measured separately, against the `ceph/s3-tests` suite Ceph RG | `DeleteObjects` | | `GetBucketLocation` | | `GetBucketTagging` | +| `GetBucketVersioning` | | `GetObject` | | `HeadBucket` | | `HeadObject` | | `ListBuckets` | | `ListMultipartUploads` | +| `ListObjectVersions` | | `ListObjects` | | `ListObjectsV2` | | `ListParts` | | `PutBucketTagging` | +| `PutBucketVersioning` | | `PutObject` | | `UploadPart` | | `UploadPartCopy` | diff --git a/docs/oci-integration/README.md b/docs/oci-integration/README.md index 1eae89cb3..bf6e7043c 100644 --- a/docs/oci-integration/README.md +++ b/docs/oci-integration/README.md @@ -27,6 +27,8 @@ resources: ## Table of Contents - [Overview](#overview) +- [Support status](#support-status) + - [What happens when a node fails](#what-happens-when-a-node-fails) - [Why run Spinifex on OCI](#why-run-spinifex-on-oci) - [What OCI changes](#what-oci-changes) - [How it fits together](#how-it-fits-together) @@ -52,10 +54,35 @@ Spinifex is an open-source infrastructure platform that brings core AWS services This guide runs that same stack on Oracle Cloud Infrastructure. Everything in [Single-Node Install](/docs/install) and [Multi-Node Install](/docs/install-multi-node) still applies: one node is a working install, three is the minimum for a cluster that can lose a node, and formation, storage and OVN behave exactly as they do on hardware you own. What changes is the layer underneath — an OCI VCN is not an Ethernet segment, so the external datapath is wired differently, and public addresses come from OCI's API rather than from a range you write in a config file. -**Read this guide end to end and you have a working cluster.** [Deploying](#deploying) is one sequence of seven steps, from an empty compartment to a node that can launch an instance with a publicly reachable address. It builds the infrastructure with Terraform, then installs and forms Spinifex the same way [Single-Node Install](/docs/install) and [Multi-Node Install](/docs/install-multi-node) do — those steps are inlined here, so you do not need to read three documents at once. The only choice to make is one node or three, and it is one line in Step 1. +**Read this guide end to end and you have a working cluster.** [Deploying](#deploying) is one sequence of eight steps, from an empty compartment to a node that can launch an instance with a publicly reachable address, and then to the Oracle Linux guest image to put on it. It builds the infrastructure with Terraform, then installs and forms Spinifex the same way [Single-Node Install](/docs/install) and [Multi-Node Install](/docs/install-multi-node) do — those steps are inlined here, so you do not need to read three documents at once. The only choice to make is one node or three, and it is one line in Step 1. Every command and every output below was run against a live OCI tenancy, on both paths. +## Support status + +**What is written here was proven on one shape, in one region, on one guest OS.** Three nodes of `VM.Standard.E6.Flex` in `ap-sydney-1`, Ubuntu hosts, routed NAT, OCI reserved public IPs over secondary private IPs, with a single-node pass of the same guide beside it. Nothing about the integration is region-specific or shape-specific by design, but nothing else has been run, so treat another region, a bare-metal shape or a different host image as untested rather than unsupported — and record the shape, region, fault domains, quotas, kernel and `spx version` when you reproduce it, because that is the only way a later difference can be attributed. + +**OCI is not in the nightly test matrix.** The routed-NAT datapath this integration depends on is guarded nightly, on three nodes, on our own hardware — the behaviour is covered, the cloud underneath it is not. So an OCI-specific regression would be found by the next person to run this guide rather than by CI. That is a real gap and it is tracked; it is not a reason to avoid the integration, but it is a reason not to read "proven" as "continuously proven". + +### What happens when a node fails + +A three-node cluster recovers a guest from a node that has stopped heartbeating, onto a survivor, with no operator action. There is no setting for it: an instance whose host is gone is down either way, and the only thing a switch could buy is leaving it down. It is inert below three nodes, because a single node has nowhere to move a guest to. + +The cluster is deliberately slow to believe a node has gone. A daemon takes no recovery action until it has seen every peer live or five minutes have passed, so a coordinated restart does not look like three simultaneous host failures; a node has to be seen stale across two passes fifteen seconds apart; and the guest's volume lease has to lapse before anything else may open it. Expect a recovery to take minutes, not seconds. + +| Condition | What the cluster does | What you see | +| --- | --- | --- | +| A node stops heartbeating and a survivor can reach its own object store | A survivor claims the instance and relaunches it | The same instance ID, the same volume and the same private address on a new host. `DescribeInstanceStatus` reports the move, as it would on EC2 | +| The volume is still leased by the node that had it | Retries every 15–90 seconds until the lease lapses | Recovery takes about a lease TTL longer. The guest never runs in two places | +| A survivor's own object store is not answering | That node stands down and lets the others try | If the object store is broken everywhere, nothing moves and nothing claims to have moved. An instance that no node could have run stays where it is | +| No survivor has capacity | Each survivor tries in its own order, with backoff | After the attempt budget the instance is stopped carrying `Server.InsufficientInstanceCapacity`, one `StartInstances` from running | +| Anything else — a torn volume, an OCI API denial, a launch that will not finish | Retries with doubling backoff, bounded at twelve attempts, roughly an hour and a half | The instance is stopped carrying `Server.HostRecoveryFailed` and the last error. It is never left looping with nothing visible to you | +| A planned whole-cluster restart | The settle window suppresses recovery while nodes return | Guests are not migrated out from under a deploy | + +**The public address follows the guest, and it keeps its identity while it does.** An OCI public address is a reserved public IP object that OCI 1:1-NATs onto a secondary private IP on a VNIC, and the guest only ever sees the private half. Recovery moves the private half between VNICs in the same subnet — one OCI call, server-side, keeping the OCID — so the reserved public IP is never released, never reallocated and never renumbered. The customer's address is the same address before and after. What does change is which node OCI delivers it to, and each node claims its own guests' addresses on a fifteen-second pass, so there is a window after the guest is running in which it is not yet reachable on its public address. Private VPC addresses have no such window: they are node-independent. + +**What this is not.** A fenced guest is stopped locally, and the volume lease is what stops it — it is not a write barrier the storage backend enforces. A node that loses contact with the cluster gives its volume up on a clock it can evaluate without reaching anything, which bounds the exposure to one renewal interval rather than to the lease TTL, and a survivor does not open the volume until the lease has lapsed. But a write the losing node had already issued is not refused by the object store, and the failure to stop the local writer is reported rather than corrected. So the guarantee is "one running guest, and a bounded, logged window" — not "a second writer is impossible". If your workload cannot tolerate that, it wants a replicated datastore rather than a recovered instance. + ## Why run Spinifex on OCI **Lower cost for the same shape of workload.** OCI's compute and block storage are priced well below the large US clouds for equivalent OCPU and IOPS, and Spinifex turns one large instance into a whole EC2/EBS/S3 surface — so a tenant's twenty small instances are twenty QEMU guests on hardware you are billed for once, not twenty billable cloud instances. @@ -242,7 +269,7 @@ oci --version **Terraform builds the infrastructure; the standard Spinifex installer builds the node.** That split is deliberate — formation on OCI is the same path prod and bare metal take, and diverging it for one cloud would mean two formation paths to keep correct. -There is one sequence of seven steps below, and the only thing that differs between a single node and a cluster is `node_count` in Step 1 and which variant of Steps 4 and 5 you follow. Decide now: +There is one sequence of eight steps below, and the only thing that differs between a single node and a cluster is `node_count` in Step 1 and which variant of Steps 4 and 5 you follow. Decide now: | | Single node | Three nodes | | --- | --- | --- | @@ -994,12 +1021,13 @@ ip -br addr show | grep ime- # holds 169.254.42.x, not .169.x - **IPv4 only.** IPv6 on OCI is materially simpler — no pools, prefixes assign directly to VCNs — but is not implemented. - **No BYOIP.** The integration accepts a public IP pool OCID in config so BYOIP becomes a config change rather than a code change, but the RIR validation and Oracle's own validation window both precede any use of it. - **OCI Block Volumes are not an EBS provider.** They back `/var/lib/spinifex` and therefore viperblock, which captures most of the performance benefit, but there is no native OCI block provider. +- **A recovered guest is briefly unreachable on its public address.** The address is never renumbered, but the node it is delivered to is corrected by a background pass every fifteen seconds rather than by the recovery itself. Private VPC addresses are unaffected. [Support status](#support-status) has the detail. - **Terraform builds infrastructure, not Spinifex.** Formation is still the standard install path, by design: it is the same path prod and bare metal take, and diverging it for one cloud would mean two formation paths to keep correct. ### What is no longer a limit Recorded because earlier versions of this guide said otherwise: -- **Multi-node works.** The `--nodes=1` cap on routed mode is gone, three-node formation and guest-to-guest traffic are proven on OCI, and the routed-NAT datapath is guarded nightly on three nodes. +- **Multi-node works.** The `--nodes=1` cap on routed mode is gone, three-node formation and guest-to-guest traffic are proven on OCI, and the routed-NAT datapath is guarded nightly on three nodes — on our own hardware, not on OCI. See [Support status](#support-status). - **Public addresses are not pinned to one node.** Each node allocates on its own VNIC, and an affinity pass reassigns an address's private half to whichever node the guest is on — one OCI call, keeping the OCID, so the customer's public address never changes. No node needs another node's VNIC OCID, and `oci_vnic_iface` is correct on every node. - **Addresses are released when an instance stops.** An auto-assigned address goes back to the pool on stop and the reserved public IP object is deleted, so a stopped fleet stops costing you addresses. diff --git a/docs/testing/e2e-coverage.md b/docs/testing/e2e-coverage.md index 014b323aa..b0b0847d1 100644 --- a/docs/testing/e2e-coverage.md +++ b/docs/testing/e2e-coverage.md @@ -7,11 +7,11 @@ Suites live in `tests/e2e//` and build to `tests/e2e/_bin/.test`. Ev | Set | Suites | | --- | --- | | `E2E_SUITES_SINGLE` | `single iam cert eks ecs storagegrowth partialblock rds quota storagefault` | -| `E2E_SUITES_MULTI` | `multinode lb cert quota storagefault` | +| `E2E_SUITES_MULTI` | `multinode lb cert quota lbrecovery instancerecovery storagefault` | | `E2E_SUITES_NIGHTLY_SINGLE` | `single cert iam` | | `E2E_SUITES_NIGHTLY_MULTI` | `multinode cert lb` | -The nightly permutation sets are deliberately narrower than the full ones: those cells have a ~35 minute budget and exist to prove every install / network / host-OS combination boots and serves. `eks`, `ecs`, `rds` and `storagefault` each get a dedicated cell instead. +The nightly permutation sets are deliberately narrower than the full ones: those cells have a ~35 minute budget and exist to prove every install / network / host-OS combination boots and serves. `eks`, `ecs`, `rds`, `storagefault`, `instancerecovery` and `lbrecovery` each get a dedicated cell instead. ## The suites @@ -42,6 +42,34 @@ Two nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30). Cell `VPCSetup`, `SpansMultipleNodes`, `SpreadPlacement`, `EveryRunningInstanceReported`, `BastionSSH`, and its own NAT Gateway lane: `PreNATIsolation`, `NATGatewayInternet`, `NATCleanupOrdering`. +### `instancerecovery` — a guest whose node goes away (multi-node) + +**Asserted from both sides, and the pair is the point.** `InstanceAutoRecovery` takes a node down and requires the guest to come back on exactly one survivor, identified by the qemu process rather than by what the API says, then brings the node back and requires it to drop the local copy instead of relaunching it. `InstanceRecoveryRefusesStorageFault` freezes predastore cluster-wide first, so the guest is paused by `werror=stop` when its node goes away, and requires that **nothing moves** — a store that refuses every node cannot be fixed by moving a guest, and the relaunch would cost it the request QEMU is holding. A reconciler that cannot tell a host failure from a storage failure passes the first and fails the second. Neither turns recovery on, because there is no setting for it: both wait for every node to report the reconciler active on a cluster nothing configured, so a build that reintroduced a switch fails here. + +**`InstancePartitionRecovery` is the fault this feature exists for, and the only one of the three that produces it rather than approximating it.** It drops NATS on 4222 and 4248 between one node and its peers and nothing else, so that node keeps its services, keeps its QEMU and keeps answering its own local clients while it cannot reach consensus — half healthy, which is what a real network fault looks like and what a service stop cannot imitate. It then requires that the node gives up the volume on a clock it can evaluate without reaching anything, that a survivor takes the guest over, and that the rejoining node forgets its copy rather than relaunching it. **The assertion it exists for is the one no single sample can reach:** a watch samples every node every three seconds for the whole run and fails on any sample naming two hosts, because two guests writing one volume through one object store is data loss already under way rather than a race that resolves. The evidence for *why* the guest stopped is the surrender and the fence in the node's own journal — a guest killed for any other reason satisfies the process count and means the opposite. + +**The partition gets its own nft table and a systemd deadman timer, for reasons worth keeping.** The node's real firewall is `inet spinifex_filter`, so a test that flushed or edited it would leave the node unfirewalled with no sign of which rules it lost; a separate table at priority -300 says exactly "these packets, dropped" and heals in one atomic delete. `systemd-run --on-active` removes it independently, so a cancelled workflow or a panicking test does not leave a node cut off from its cluster until somebody notices. It also proves SSH still works and proves a peer port is actually unreachable before asserting anything — a rule set that installed cleanly and dropped nothing would otherwise read as the system tolerating a partition. + +**The storage-fault victim launches with cloud-init user-data that writes continuously, and that is load bearing.** An idle guest issues no I/O that reaches the object store, so a version of this test without a workload left three guests running happily through twelve minutes of dead predastore and proved nothing. Driving the load from user-data rather than over SSH is what lets it run on a routed-NAT cell, where guests have no public address. The pause itself takes minutes — 4m47s when measured — so the budget is generous on purpose. + +**Its own suite and its own nightly cell, for storagefault's reasons.** It freezes a cluster-wide service and removes a node, so anything sharing the environment reports that outage as its own failure; and it is slow by nature, which is why it is not in a permutation cell budgeted at thirty-five minutes. Both behaviours were first proven by hand against a real hypervisor power-off on the three-node OCI cluster. + +### `lbrecovery` — a load balancer whose node goes away (multi-node, needs three) + +**The customer-visible half of host failure, asserted from outside the cluster rather than from the records.** `ALBSurvivesItsHostFailing` stands up an internet-facing ALB with two HTTP backends, discovers which node the load balancer's guest landed on, and takes that node away. It then requires the four things a customer of a load balancer actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again within a bounded time, and the backends — which never moved — are healthy again from the recovered load balancer's point of view. + +**The backends are placed around the load balancer, not beside it.** A load balancer is launched over a queue group and lands wherever a node answers, so its node is discovered and the backends are then launched and relaunched until both are somewhere else. A backend on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve. + +**DNS is queried against a node's own northstar, not through the runner's resolver.** Go's resolver dials port 53 on a named node — which is open on every node deliberately, because northstar serves public names — so the assertion is about what this cluster answers rather than about what the machine running the tests was configured to forward to. The same resolver drives the HTTP probes, so traffic goes through the load balancer's name end to end and not through an address resolved once at the start. `dig` is deliberately not used: it is proven present in the guest image and nowhere else. + +**The assertion no single sample can reach is the name.** A watch queries every surviving node every five seconds for the whole run, and fails on any sample where the record is missing, where a node refuses the query, or where it answers an address that is not the load balancer's. An answer naming another address is the worst of the three, because the traffic goes there; an empty answer outlives the fix for as long as anything cached it. An address that moves is asserted separately and before any traffic, since a passing probe against a renumbered address would hide it. + +**Then the node comes back, and has to give up what it no longer owns.** The returning node must not relaunch its local copy, and must no longer carry the tap that claims the load balancer's logical port — OVN binds a port to one chassis, so a second claimant does not split the traffic, it contends for it and the address answers from whichever won last. That same stale tap is also what keeps the external address's host ingress from being pruned, since the prune waits on evidence the guest sits elsewhere. The last assertion is that the name still serves traffic with the node back in the cluster, which is the end-user form of the same question. + +**`DescribeLoadBalancers` is required to keep reporting `active` throughout, and that is AWS parity rather than an oversight.** AWS never marks a load balancer degraded because a host under it failed: the state describes the customer's configuration, not our hardware. The honest signals about the outage are target health and the traffic itself, which is why both are asserted and the state is not trusted as evidence of anything. + +**Its own suite and its own cell, for instancerecovery's reasons plus one.** It takes a node away, so anything sharing the environment reports that outage as its own failure; it needs its own VPC, public subnet and a backend placement no shared fixture describes; and a failure here has to read as "the load balancer did not survive its host" rather than as one more red subtest in a suite about load balancer features. + ### `iam` — everything IAM and STS Policy evaluation against live requests rather than unit-level: a deny applying to the next request without a new credential, resource-scoped grants naming one key pair, `aws:SourceIp` seeing the real client address, assumed-role sessions carrying role policies, and revocation stopping one key alone. diff --git a/spinifex/daemon/daemon.go b/spinifex/daemon/daemon.go index f53c75352..a4965be22 100644 --- a/spinifex/daemon/daemon.go +++ b/spinifex/daemon/daemon.go @@ -171,9 +171,12 @@ type Daemon struct { routeTableService *handlers_ec2_routetable.RouteTableServiceImpl natGatewayService *handlers_ec2_natgw.NatGatewayServiceImpl externalIPAM *handlers_ec2_vpc.ExternalIPAM - ctx context.Context - cancel context.CancelFunc - shutdownWg sync.WaitGroup + // ociAllocators is every source="oci" pool on this node. Written once during + // startup, before anything reads it. + ociAllocators []ociPool + ctx context.Context + cancel context.CancelFunc + shutdownWg sync.WaitGroup // recoveryRetryWg tracks the post-restore retry loop, which writes the // local state file when it finishes. Tests wait on it before cleanup. @@ -1497,11 +1500,23 @@ func (d *Daemon) installOCIAllocators(ipam *handlers_ec2_vpc.ExternalIPAM, js je } slog.Info("OCI allocator ready", "pool", p.Name, "collected", len(res.Collected), "stale_bindings", len(res.Stale), "skipped", res.Skipped) - go d.runOCIAffinityLoop(alloc, p.Name) + d.ociAllocators = append(d.ociAllocators, ociPool{name: p.Name, alloc: alloc}) + } + if len(d.ociAllocators) > 0 { + go d.runOCIAffinityLoop() } return nil } +// ociPool pairs a pool's allocator with its name, so one pass can cover every +// pool and reach a single verdict about each guest. Per-pool loops could not: a +// guest is impaired or it is not, and two loops writing that answer from half the +// evidence would take turns contradicting each other. +type ociPool struct { + name string + alloc *ocinet.PoolAllocator +} + // ociAffinityInterval is how often a node checks that OCI delivers its guests' // addresses to it. A guest that has just started elsewhere is dark until the // pass runs, so it is short relative to a boot — the address is back before the @@ -1513,7 +1528,7 @@ const ociAffinityInterval = 15 * time.Second // loop and for the same reason: the question is about this host's guests, and a // node that never wins the reconcile lease would otherwise never claim the // addresses of the instances it is running. -func (d *Daemon) runOCIAffinityLoop(alloc *ocinet.PoolAllocator, poolName string) { +func (d *Daemon) runOCIAffinityLoop() { ticker := time.NewTicker(ociAffinityInterval) defer ticker.Stop() for { @@ -1522,16 +1537,54 @@ func (d *Daemon) runOCIAffinityLoop(alloc *ocinet.PoolAllocator, poolName string return case <-ticker.C: } - res, err := alloc.ClaimLocalAddresses(d.ctx) + d.ClaimOCIAddresses(d.ctx) + } +} + +// ClaimOCIAddresses runs one affinity pass over every OCI pool and records which +// guests OCI is not delivering an address to. +// +// Exported because the recovery path calls it in line: a guest relaunched here is +// reported running the moment it boots, and until this has run OCI still delivers +// its address to the node that died. Waiting for the ticker would mean up to a +// full interval in which the records say recovered and the address says nothing. +// +// A pass that could not read its own inputs marks nothing. It learned that no +// guest is reachable, not that every guest is, and clearing on that would turn an +// OCI outage into a clean bill of health. +func (d *Daemon) ClaimOCIAddresses(ctx context.Context) { + if len(d.ociAllocators) == 0 { + return + } + + complete := true + stalled := make(map[string]string) + for _, p := range d.ociAllocators { + res, err := p.alloc.ClaimLocalAddresses(ctx) if err != nil { slog.Error("OCI address affinity pass failed; a guest here may be unreachable on its public address", - "pool", poolName, "err", err) + "pool", p.name, "err", err) + complete = false continue } if len(res.Claimed) > 0 { - slog.Info("OCI addresses moved to this node", "pool", poolName, + slog.Info("OCI addresses moved to this node", "pool", p.name, "claimed", res.Claimed, "local_bindings", res.Local) } + for _, s := range res.Stalled { + slog.Error("OCI is not delivering an address to the node running its guest", + "pool", p.name, "public_ip", s.PublicIP, "eni_id", s.ENIID, "reason", s.Reason) + if s.ENIID != "" { + stalled[s.ENIID] = s.Reason + } + } + } + if !complete || d.vmMgr == nil { + return + } + if marked := d.vmMgr.SetUnreachableAddresses(stalled); len(marked) > 0 { + slog.Error("Instances are running and unreachable on their public address", + "node", d.node, "instances", marked) } } @@ -2165,6 +2218,7 @@ func (d *Daemon) startCluster() error { d.startHeartbeat() d.startRecordRepair() + d.startInstanceRecovery() d.vmMgr.StartPendingWatchdog(d.ctx) // Reality→desired GC backstop (ADR-0003 §3): finish teardown interrupted by @@ -2561,6 +2615,7 @@ func (d *Daemon) LoadState() error { // it is recovering are already down, and blocking here would keep the API from // coming up while it waits. func (d *Daemon) restoreInstances() error { + d.forgetSupersededInstances() d.vmMgr.Restore() if err := d.WriteState(); err != nil { slog.Error("Failed to persist local state after restore", "error", err) diff --git a/spinifex/daemon/daemon_handlers_test.go b/spinifex/daemon/daemon_handlers_test.go index 7a5651e3a..26765ef83 100644 --- a/spinifex/daemon/daemon_handlers_test.go +++ b/spinifex/daemon/daemon_handlers_test.go @@ -727,10 +727,22 @@ func TestHandleEC2Events_StopInstance(t *testing.T) { // Should get immediate {} response assert.Equal(t, `{}`, string(reply.Data)) - // State should transition to stopping - var status vm.InstanceState - daemon.vmMgr.UpdateState(instanceID, func(v *vm.VM) { status = v.Status }) - assert.Equal(t, vm.StateStopping, status) + // The ack comes first and the transition runs detached, so the state is a + // later fact than the reply — and it does not stop at stopping. + assertLeavesRunning(t, daemon, instanceID, vm.StateStopping, vm.StateStopped) +} + +// assertLeavesRunning waits for a detached stop or terminate to move the +// instance out of running. Asserting on the transient state alone is a race in +// both directions: too early reads running, too late reads the final state. +func assertLeavesRunning(t *testing.T, daemon *Daemon, instanceID string, want ...vm.InstanceState) { + t.Helper() + + require.EventuallyWithT(t, func(c *assert.CollectT) { + var status vm.InstanceState + daemon.vmMgr.UpdateState(instanceID, func(v *vm.VM) { status = v.Status }) + assert.Contains(c, want, status) + }, 10*time.Second, 10*time.Millisecond) } func TestHandleEC2Events_TerminateInstance(t *testing.T) { @@ -771,9 +783,7 @@ func TestHandleEC2Events_TerminateInstance(t *testing.T) { assert.Equal(t, `{}`, string(reply.Data)) - var status vm.InstanceState - daemon.vmMgr.UpdateState(instanceID, func(v *vm.VM) { status = v.Status }) - assert.Equal(t, vm.StateShuttingDown, status) + assertLeavesRunning(t, daemon, instanceID, vm.StateShuttingDown, vm.StateTerminated) } func TestHandleEC2Events_RebootRunningInstance(t *testing.T) { diff --git a/spinifex/daemon/daemon_mgmt_ip.go b/spinifex/daemon/daemon_mgmt_ip.go index e5bd2e195..a79928691 100644 --- a/spinifex/daemon/daemon_mgmt_ip.go +++ b/spinifex/daemon/daemon_mgmt_ip.go @@ -133,9 +133,11 @@ func NewMgmtIPAllocator(bridgeIP string) (*MgmtIPAllocator, error) { // BindKV attaches the cluster-wide KV backing store. Called once from // daemon.startCluster after JetStream is up — never from startLocal (DDIL -// §1e-audit). node identifies this node's entries in the shared record for -// audit; it plays no part in allocation, which considers every entry in the -// record regardless of which node owns it. +// §1e-audit). +// +// node plays no part in allocation, which considers every entry in the record +// regardless of which node owns it. It does decide releases: only the node an +// entry names may give that address back. func (a *MgmtIPAllocator) BindKV(jsManager *JetStreamManager, node string) { a.mu.Lock() defer a.mu.Unlock() @@ -239,7 +241,7 @@ func (a *MgmtIPAllocator) Release(instanceID string) { a.mu.Lock() ip, hadLocal := a.allocated[instanceID] delete(a.allocated, instanceID) - jsManager, subnet := a.jsManager, a.subnet + jsManager, node, subnet := a.jsManager, a.node, a.subnet a.mu.Unlock() if hadLocal { @@ -254,18 +256,92 @@ func (a *MgmtIPAllocator) Release(instanceID string) { // (or that no node has a record for) must not conjure an empty record // into existence — jetstream.ErrKeyNotFound just means there was nothing to // release. + // + // Only the node the entry names may free it. br-mgmt is one flat L2 segment + // across the cluster, so a node that gives back an address still in use puts + // it up for the next instance anywhere — and this path runs on exactly the + // node that should not: the one dropping a guest another node now runs, + // whether it was fenced or found superseded on its return. + foreign := "" _, err := updateMgmtIPAMWithRetry(jsManager, subnet, func(r *MgmtIPRecord) { + foreign = "" for i, e := range r.Allocated { - if e.InstanceID == instanceID { - r.Allocated = append(r.Allocated[:i], r.Allocated[i+1:]...) + if e.InstanceID != instanceID { + continue + } + // An entry written before the node field existed names nobody, and + // refusing those would leak every pre-upgrade address forever. + if e.Node != "" && e.Node != node { + foreign = e.Node return } + r.Allocated = append(r.Allocated[:i], r.Allocated[i+1:]...) + return } }, false) if err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) { slog.Warn("Failed to release mgmt IP in cluster KV; address remains reserved until reconciled", "instance", instanceID, "err", err) } + if foreign != "" { + slog.Info("Left a management IP reserved: it belongs to the node running this instance now", + "instance", instanceID, "ip", ip, "holder", foreign, "node", node) + } +} + +// Claim records this node as the holder of an instance's management IP, for an +// instance that arrived here from somewhere else. +// +// It is the other half of a node-scoped release. Without it the entry keeps +// naming the node the instance left, so the node actually running it could never +// give the address back and it would stay reserved for good. +// +// Idempotent, and it does not allocate: a record with no entry for the instance +// gets one for the address the instance already carries, because that address is +// in use on the segment whether or not anything wrote it down. +func (a *MgmtIPAllocator) Claim(instanceID, mgmtIP string) { + if instanceID == "" || mgmtIP == "" { + return + } + + a.mu.Lock() + a.allocated[instanceID] = mgmtIP + jsManager, node, subnet := a.jsManager, a.node, a.subnet + a.mu.Unlock() + + if jsManager == nil { + return + } + + moved := "" + _, err := updateMgmtIPAMWithRetry(jsManager, subnet, func(r *MgmtIPRecord) { + moved = "" + if r.Subnet == "" { + r.Subnet = subnet + } + for i, e := range r.Allocated { + if e.InstanceID != instanceID { + continue + } + if e.Node == node && e.IP == mgmtIP { + return + } + moved = e.Node + r.Allocated[i].Node = node + r.Allocated[i].IP = mgmtIP + return + } + r.Allocated = append(r.Allocated, MgmtIPEntry{IP: mgmtIP, InstanceID: instanceID, Node: node}) + }, true) + if err != nil { + slog.Warn("Failed to claim mgmt IP in cluster KV; the address stays recorded against its previous holder", + "instance", instanceID, "ip", mgmtIP, "err", err) + return + } + if moved != "" { + slog.Info("Took over a management IP from the node the instance left", + "instance", instanceID, "ip", mgmtIP, "from", moved, "node", node) + } } // Rebuild reconciles the allocator against vms (this node's own VM @@ -309,10 +385,16 @@ func (a *MgmtIPAllocator) Rebuild(vms map[string]*vm.VM) { if r.Subnet == "" { r.Subnet = subnet } - for _, e := range r.Allocated { - if e.InstanceID == id { - return // already reconciled + for i, e := range r.Allocated { + if e.InstanceID != id { + continue } + // Re-stamped rather than left alone: this node is running the + // instance, so it holds the address, and an entry still naming + // where the instance came from is one nobody may release. + r.Allocated[i].Node = node + r.Allocated[i].IP = mgmtIP + return } r.Allocated = append(r.Allocated, MgmtIPEntry{IP: mgmtIP, InstanceID: id, Node: node}) }, true) diff --git a/spinifex/daemon/daemon_mgmt_ip_test.go b/spinifex/daemon/daemon_mgmt_ip_test.go index d8731737a..3aabeb0a7 100644 --- a/spinifex/daemon/daemon_mgmt_ip_test.go +++ b/spinifex/daemon/daemon_mgmt_ip_test.go @@ -756,3 +756,198 @@ func TestPrimeMgmtNeighEntry_NoMACIsNoOp(t *testing.T) { t.Errorf("prime calls = %d, want 0", len(*primed)) } } + +// mgmtIPEntryFor reads the cluster record's entry for one instance, so a test +// can assert who is recorded as holding an address rather than inferring it from +// whether the next allocation happens to collide. +func mgmtIPEntryFor(t *testing.T, jsm *JetStreamManager, a *MgmtIPAllocator, instanceID string) (MgmtIPEntry, bool) { + t.Helper() + record, err := updateMgmtIPAMWithRetry(jsm, a.subnet, func(*MgmtIPRecord) {}, false) + if err != nil { + t.Fatalf("read mgmt ipam record: %v", err) + } + for _, e := range record.Allocated { + if e.InstanceID == instanceID { + return e, true + } + } + return MgmtIPEntry{}, false +} + +// The node an instance left runs the teardown for it, and br-mgmt is one flat L2 +// segment across the cluster. A release there would put an address still in use +// back in the pool for any node's next instance, so only the node the entry +// names may free it. +func TestMgmtIPAllocator_ReleaseWillNotFreeAnAddressAnotherNodeHolds(t *testing.T) { + jsm := newTestMgmtJSM(t) + + node1, err := NewMgmtIPAllocator("10.31.8.1") + if err != nil { + t.Fatal(err) + } + node1.BindKV(jsm, "node-1") + cleanupMgmtIPAM(t, jsm, node1) + + node2, err := NewMgmtIPAllocator("10.31.8.2") + if err != nil { + t.Fatal(err) + } + node2.BindKV(jsm, "node-2") + + ip, err := node1.Allocate("i-moved") + if err != nil { + t.Fatal(err) + } + + // The instance is recovered onto node2, which takes the reservation over. + node2.Claim("i-moved", ip) + + // node1 then reaches its own teardown for the copy it is dropping. + node1.Release("i-moved") + + entry, present := mgmtIPEntryFor(t, jsm, node1, "i-moved") + if !present { + t.Fatalf("node-1 freed %s while node-2 was running the instance; "+ + "on a flat segment the next instance anywhere can now be given an address in use", ip) + } + if entry.Node != "node-2" || entry.IP != ip { + t.Errorf("entry after the takeover = %+v, want node-2 holding %s", entry, ip) + } +} + +// Without the claim the entry keeps naming the node the instance left, so the +// node actually running it could never give the address back and it would stay +// reserved for good. +func TestMgmtIPAllocator_ClaimLetsTheNewHolderReleaseItLater(t *testing.T) { + jsm := newTestMgmtJSM(t) + + node1, err := NewMgmtIPAllocator("10.32.8.1") + if err != nil { + t.Fatal(err) + } + node1.BindKV(jsm, "node-1") + cleanupMgmtIPAM(t, jsm, node1) + + node2, err := NewMgmtIPAllocator("10.32.8.2") + if err != nil { + t.Fatal(err) + } + node2.BindKV(jsm, "node-2") + + ip, err := node1.Allocate("i-moved") + if err != nil { + t.Fatal(err) + } + + node2.Claim("i-moved", ip) + node2.Release("i-moved") + + if entry, present := mgmtIPEntryFor(t, jsm, node1, "i-moved"); present { + t.Errorf("the holder could not release its own address: entry still %+v", entry) + } + + // And the address is genuinely back, not merely unrecorded. + reused, err := node1.Allocate("i-next") + if err != nil { + t.Fatal(err) + } + if reused != ip { + t.Errorf("next allocation = %q, want the freed address %q", reused, ip) + } +} + +// Claim is idempotent and does not allocate: an instance carrying an address +// nothing wrote down still holds it on the segment, so the record has to say so. +func TestMgmtIPAllocator_ClaimRecordsAnAddressWithNoEntry(t *testing.T) { + jsm := newTestMgmtJSM(t) + + a, err := NewMgmtIPAllocator("10.33.8.1") + if err != nil { + t.Fatal(err) + } + a.BindKV(jsm, "node-a") + cleanupMgmtIPAM(t, jsm, a) + + a.Claim("i-orphan", "10.33.8.77") + a.Claim("i-orphan", "10.33.8.77") + + entry, present := mgmtIPEntryFor(t, jsm, a, "i-orphan") + if !present { + t.Fatal("an address in use on the segment must be recorded, or another instance is given it") + } + if entry.IP != "10.33.8.77" || entry.Node != "node-a" { + t.Errorf("entry = %+v, want node-a holding 10.33.8.77", entry) + } + + // Allocate must now route around it rather than handing it out again. + ip, err := a.Allocate("i-next") + if err != nil { + t.Fatal(err) + } + if ip == "10.33.8.77" { + t.Error("Allocate handed out an address Claim had already recorded as in use") + } +} + +// A record written before the node field existed names nobody. Refusing those +// would leak every pre-upgrade address forever, which is worse than the +// collision the check exists to prevent — there is no other holder to collide +// with. +func TestMgmtIPAllocator_ReleaseFreesAnEntryThatNamesNoNode(t *testing.T) { + jsm := newTestMgmtJSM(t) + + a, err := NewMgmtIPAllocator("10.34.8.1") + if err != nil { + t.Fatal(err) + } + a.BindKV(jsm, "node-a") + cleanupMgmtIPAM(t, jsm, a) + + if _, err := updateMgmtIPAMWithRetry(jsm, a.subnet, func(r *MgmtIPRecord) { + r.Subnet = a.subnet + r.Allocated = append(r.Allocated, MgmtIPEntry{IP: "10.34.8.10", InstanceID: "i-legacy"}) + }, true); err != nil { + t.Fatal(err) + } + + a.Release("i-legacy") + + if entry, present := mgmtIPEntryFor(t, jsm, a, "i-legacy"); present { + t.Errorf("a pre-upgrade entry was left reserved forever: %+v", entry) + } +} + +// Rebuild runs on a node that is already running the instance, so it is the +// other place the record can be told the address moved — the case where the +// recovery happened while this node was down. +func TestMgmtIPAllocator_RebuildTakesOverAnEntryFromAnotherNode(t *testing.T) { + jsm := newTestMgmtJSM(t) + + node1, err := NewMgmtIPAllocator("10.35.8.1") + if err != nil { + t.Fatal(err) + } + node1.BindKV(jsm, "node-1") + cleanupMgmtIPAM(t, jsm, node1) + + node2, err := NewMgmtIPAllocator("10.35.8.2") + if err != nil { + t.Fatal(err) + } + node2.BindKV(jsm, "node-2") + + ip, err := node1.Allocate("i-moved") + if err != nil { + t.Fatal(err) + } + + node2.Rebuild(map[string]*vm.VM{"i-moved": {MgmtIP: ip}}) + + entry, present := mgmtIPEntryFor(t, jsm, node1, "i-moved") + if !present { + t.Fatal("the entry vanished; Rebuild must re-stamp it, not remove it") + } + if entry.Node != "node-2" { + t.Errorf("holder after rebuild = %q, want node-2, or nothing can ever release this address", entry.Node) + } +} diff --git a/spinifex/daemon/daemon_system_instance.go b/spinifex/daemon/daemon_system_instance.go index a1eeaa126..d94600c23 100644 --- a/spinifex/daemon/daemon_system_instance.go +++ b/spinifex/daemon/daemon_system_instance.go @@ -648,6 +648,24 @@ func (d *Daemon) terminateSystemInstanceLocal(instanceID string) error { return nil } +// prepareInstanceRelaunch makes this node's world match an instance record it is +// about to launch from. +// +// Both callers start from a record rather than a request, and a record names +// things by reference: paths to files, an address held in a cluster-wide +// reservation. On a same-node restart the references still resolve; on a recovery +// onto a survivor they point at another node. So both concerns below are the same +// question asked twice — what does this record assume that is not true here yet. +func (d *Daemon) prepareInstanceRelaunch(inst *vm.VM) error { + // Before the launch, and not fatal. The instance is running on this node + // either way, and the reservation naming the node it left is what stops that + // node's own teardown from handing the address to somebody else. + if d.mgmtIPAllocator != nil { + d.mgmtIPAllocator.Claim(inst.ID, inst.MgmtIP) + } + return d.refreshSystemInstanceState(inst) +} + // refreshSystemInstanceState regenerates the tmpfs-backed fw_cfg blobs that // QEMU loads at boot. The blobs live under utils.RuntimeDir() (tmpfs on // production hosts) and are wiped on host reboot while the persisted diff --git a/spinifex/daemon/instance_claim_test.go b/spinifex/daemon/instance_claim_test.go new file mode 100644 index 000000000..6dc9f8ba3 --- /dev/null +++ b/spinifex/daemon/instance_claim_test.go @@ -0,0 +1,239 @@ +package daemon_test + +import ( + "sync" + "testing" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/mulgadc/spinifex/spinifex/daemon" + "github.com/mulgadc/spinifex/spinifex/resource" + "github.com/mulgadc/spinifex/spinifex/vm" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// recoverableRecord is what a node that died hard leaves behind: observed +// running, still wanted running, and named to a node that will never write +// again. +func recoverableRecord(id, node string) *vm.InstanceRecord { + return &vm.InstanceRecord{ + Metadata: resource.Metadata{Name: id, AccountID: "111122223333"}, + Spec: vm.InstanceSpec{InstanceType: "t3.micro", DesiredState: vm.DesiredRunning}, + Status: vm.InstanceStatus{Status: vm.StateRunning, LastNode: node}, + } +} + +func TestClaimRecoverableInstance_TakesOwnership(t *testing.T) { + m := newRecordManager(t) + require.NoError(t, m.WriteInstanceRecord("i-1", recoverableRecord("i-1", "node-1"))) + + claimed, err := m.ClaimRecoverableInstance("i-1", "node-1", "node-2") + require.NoError(t, err) + require.NotNil(t, claimed) + assert.Equal(t, "i-1", claimed.ID) + + stored, err := m.LoadInstanceRecord("i-1") + require.NoError(t, err) + assert.Equal(t, "node-2", stored.Status.LastNode, + "the claim is the ownership transfer, so it has to be durable before the launch starts") +} + +// The CAS is the whole of the mutual exclusion. Several survivors derive the +// same candidate and race it; exactly one revision may win. +func TestClaimRecoverableInstance_OnlyOneSurvivorWins(t *testing.T) { + m := newRecordManager(t) + require.NoError(t, m.WriteInstanceRecord("i-1", recoverableRecord("i-1", "node-1"))) + + var ( + wg sync.WaitGroup + mu sync.Mutex + winners []string + ) + for _, claimant := range []string{"node-2", "node-3", "node-4"} { + wg.Go(func() { + if _, err := m.ClaimRecoverableInstance("i-1", "node-1", claimant); err == nil { + mu.Lock() + winners = append(winners, claimant) + mu.Unlock() + } + }) + } + wg.Wait() + + require.Lenf(t, winners, 1, "more than one node claimed the same instance: %v", winners) + + stored, err := m.LoadInstanceRecord("i-1") + require.NoError(t, err) + assert.Equal(t, winners[0], stored.Status.LastNode) +} + +// A loser reads a record naming a live owner, which is how the candidate +// disappears from its view without anything telling it so. +func TestClaimRecoverableInstance_LostWhenTheOwnerAlreadyMoved(t *testing.T) { + m := newRecordManager(t) + require.NoError(t, m.WriteInstanceRecord("i-1", recoverableRecord("i-1", "node-3"))) + + _, err := m.ClaimRecoverableInstance("i-1", "node-1", "node-2") + assert.ErrorIs(t, err, vm.ErrRecoveryClaimLost) +} + +func TestClaimRecoverableInstance_LostWhenTheRecordIsGone(t *testing.T) { + m := newRecordManager(t) + + _, err := m.ClaimRecoverableInstance("i-never-written", "node-1", "node-2") + assert.ErrorIs(t, err, vm.ErrRecoveryClaimLost) +} + +// Ownership is checked and eligibility is checked again, at the point the +// transfer happens rather than only when the candidate was derived. A terminate +// that started in between must not be undone by a claim. +func TestClaimRecoverableInstance_RefusesAnInstanceNobodyWantsRunning(t *testing.T) { + m := newRecordManager(t) + + stopped := recoverableRecord("i-stopped", "node-1") + stopped.Spec.DesiredState = vm.DesiredStopped + stopped.Status.Status = vm.StateStopped + require.NoError(t, m.WriteInstanceRecord("i-stopped", stopped)) + + terminating := recoverableRecord("i-terminating", "node-1") + terminating.Status.Status = vm.StateShuttingDown + require.NoError(t, m.WriteInstanceRecord("i-terminating", terminating)) + + paused := recoverableRecord("i-paused", "node-1") + paused.Status.Health.IOErrorSince = time.Now() + require.NoError(t, m.WriteInstanceRecord("i-paused", paused)) + + for _, id := range []string{"i-stopped", "i-terminating", "i-paused"} { + _, err := m.ClaimRecoverableInstance(id, "node-1", "node-2") + assert.ErrorIsf(t, err, vm.ErrRecoveryClaimLost, "%s should not be claimable", id) + } +} + +// Without the release a failed attempt parks the instance on a node that has +// just proved it cannot run it, and no other survivor looks at it again. +func TestReleaseRecoveredInstance_HandsTheRecordBack(t *testing.T) { + m := newRecordManager(t) + require.NoError(t, m.WriteInstanceRecord("i-1", recoverableRecord("i-1", "node-1"))) + + _, err := m.ClaimRecoverableInstance("i-1", "node-1", "node-2") + require.NoError(t, err) + + require.NoError(t, m.ReleaseRecoveredInstance("i-1", "node-2", "node-1")) + + stored, err := m.LoadInstanceRecord("i-1") + require.NoError(t, err) + assert.Equal(t, "node-1", stored.Status.LastNode) +} + +// A release that arrives after someone else has taken the record must not pull +// it back: the current owner may already be running the guest. +func TestReleaseRecoveredInstance_LeavesAnotherNodesRecordAlone(t *testing.T) { + m := newRecordManager(t) + require.NoError(t, m.WriteInstanceRecord("i-1", recoverableRecord("i-1", "node-3"))) + + require.NoError(t, m.ReleaseRecoveredInstance("i-1", "node-2", "node-1")) + + stored, err := m.LoadInstanceRecord("i-1") + require.NoError(t, err) + assert.Equal(t, "node-3", stored.Status.LastNode) +} + +// The release is best-effort by design — the node making it may itself be +// failing — so an instance that has been terminated meanwhile is not an error. +func TestReleaseRecoveredInstance_AbsentRecordIsNotAnError(t *testing.T) { + m := newRecordManager(t) + + assert.NoError(t, m.ReleaseRecoveredInstance("i-never-written", "node-2", "node-1")) +} + +// Recovery is never inferred from a manager that has no record bucket: the +// answer would be "no instances to recover" and look identical to a healthy +// cluster. +func TestClaimAndRelease_RefuseWithoutABucket(t *testing.T) { + m := &daemon.JetStreamManager{} + + _, err := m.ClaimRecoverableInstance("i-1", "node-1", "node-2") + assert.Error(t, err) + assert.NotErrorIs(t, err, vm.ErrRecoveryClaimLost, + "an uninitialised bucket is a fault here, not an ordinary lost race") + + assert.Error(t, m.ReleaseRecoveredInstance("i-1", "node-2", "node-1")) +} + +// AbandonRecovery is what stops an instance being retried forever, and the +// shape it leaves behind is the whole of its contract: the retrying ends on +// every node because they all derive their work from desired state, and the +// customer's own StartInstances is the retry. +func TestAbandonRecovery_LeavesTheInstanceStoppedWithAReason(t *testing.T) { + m := newRecordManager(t) + record := recoverableRecord("i-1", "node-1") + record.Status.Instance = &ec2.Instance{InstanceId: aws.String("i-1")} + require.NoError(t, m.WriteInstanceRecord("i-1", record)) + + abandoned, err := m.AbandonRecovery("i-1", "node-1", "Server.HostRecoveryFailed", "nbdkit exited 1") + require.NoError(t, err) + require.True(t, abandoned) + + stored, err := m.LoadInstanceRecord("i-1") + require.NoError(t, err) + assert.Equal(t, vm.DesiredStopped, stored.Spec.DesiredState, + "desired state is what every node's selection reads, so this is what ends the retrying cluster-wide") + assert.Equal(t, vm.StateStopped, stored.Status.Status, + "AWS leaves a host-failed instance stopped, not running on a node that is gone") + require.NotNil(t, stored.Status.Instance.StateReason) + assert.Equal(t, "Server.HostRecoveryFailed", aws.StringValue(stored.Status.Instance.StateReason.Code)) + assert.Equal(t, "nbdkit exited 1", aws.StringValue(stored.Status.Instance.StateReason.Message), + "the reason is the only account of this the customer ever gets") + assert.Equal(t, "node-1", stored.Status.LastNode, + "where it last ran is still true and is the only pointer to the evidence") +} + +// The give-up races the success. A node that exhausts its attempts after another +// survivor has already claimed and launched the instance must not stop a guest +// that is now running. +func TestAbandonRecovery_WillNotStopAnInstanceAnotherNodeTook(t *testing.T) { + m := newRecordManager(t) + require.NoError(t, m.WriteInstanceRecord("i-1", recoverableRecord("i-1", "node-3"))) + + abandoned, err := m.AbandonRecovery("i-1", "node-1", "Server.HostRecoveryFailed", "gave up") + require.NoError(t, err) + assert.False(t, abandoned, "the caller has to be able to tell it did not apply") + + stored, err := m.LoadInstanceRecord("i-1") + require.NoError(t, err) + assert.Equal(t, vm.DesiredRunning, stored.Spec.DesiredState) + assert.Equal(t, vm.StateRunning, stored.Status.Status) +} + +// A record with no EC2 projection yet still has to stop being retried. The +// reason is lost, which is a worse outcome than having one and a better one than +// retrying forever. +func TestAbandonRecovery_WorksWithoutAnEC2Projection(t *testing.T) { + m := newRecordManager(t) + require.NoError(t, m.WriteInstanceRecord("i-1", recoverableRecord("i-1", "node-1"))) + + abandoned, err := m.AbandonRecovery("i-1", "node-1", "Server.HostRecoveryFailed", "gave up") + require.NoError(t, err) + assert.True(t, abandoned) + + stored, err := m.LoadInstanceRecord("i-1") + require.NoError(t, err) + assert.Equal(t, vm.DesiredStopped, stored.Spec.DesiredState) +} + +func TestAbandonRecovery_AbsentRecordIsNotAnError(t *testing.T) { + m := newRecordManager(t) + + abandoned, err := m.AbandonRecovery("i-never-written", "node-1", "Server.HostRecoveryFailed", "gave up") + assert.NoError(t, err) + assert.False(t, abandoned) +} + +func TestAbandonRecovery_RefusesWithoutABucket(t *testing.T) { + m := &daemon.JetStreamManager{} + + _, err := m.AbandonRecovery("i-1", "node-1", "Server.HostRecoveryFailed", "gave up") + assert.Error(t, err) +} diff --git a/spinifex/daemon/instance_membership.go b/spinifex/daemon/instance_membership.go index 438c664bc..d7460a08b 100644 --- a/spinifex/daemon/instance_membership.go +++ b/spinifex/daemon/instance_membership.go @@ -35,3 +35,38 @@ func runsOn(record *vm.InstanceRecord, nodeID string) bool { } return record.Status.LastNode == nodeID && !operatorStopped(record) } + +// recoverable reports whether a record is one another node may take over when +// the node named on it stops heartbeating. +// +// Desired state is the test, not observed state. A node that dies hard leaves +// its instances observed-running and never writes anything again, and a node +// drained for maintenance leaves them observed-stopped but still wanted — both +// are recoveries. What is excluded is an instance nobody wants running: an +// operator stop, or a terminate already under way. +func recoverable(record *vm.InstanceRecord) bool { + if record == nil || record.Spec.DesiredState != vm.DesiredRunning { + return false + } + if record.Metadata.MarkedForDeletion() { + return false + } + switch record.Status.Status { + case vm.StateTerminated, vm.StateShuttingDown: + return false + } + return !storageFaulted(record) +} + +// storageFaulted reports whether the last health an instance published was its +// storage refusing I/O, which werror=stop turns into a paused guest. +// +// This is the failure another node cannot fix. The volumes are the same objects +// in the same object store from anywhere, so a backend refusing them refuses +// them everywhere, and relaunching elsewhere moves the pause rather than ending +// it. Left alone the guest keeps its held request and resumes when the backend +// returns, which is the better outcome than a relaunch that loses its RAM and +// then pauses too. +func storageFaulted(record *vm.InstanceRecord) bool { + return !record.Status.Health.IOErrorSince.IsZero() +} diff --git a/spinifex/daemon/instance_recovery.go b/spinifex/daemon/instance_recovery.go new file mode 100644 index 000000000..21f4297bb --- /dev/null +++ b/spinifex/daemon/instance_recovery.go @@ -0,0 +1,560 @@ +package daemon + +import ( + "cmp" + "context" + "errors" + "fmt" + "hash/fnv" + "log/slog" + "slices" + "time" + + "github.com/mulgadc/spinifex/spinifex/awserrors" + handlers_ec2_instance "github.com/mulgadc/spinifex/spinifex/handlers/ec2/instance" + "github.com/mulgadc/spinifex/spinifex/instancecache" + "github.com/mulgadc/spinifex/spinifex/otelsetup" + "github.com/mulgadc/spinifex/spinifex/vm" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/codes" + "go.opentelemetry.io/otel/trace" +) + +// recoveryPassInterval is the gap between passes. It is well under the volume +// lease TTL, so a candidate refused because the old owner's lease has not +// expired is retried within that window rather than after it. +const recoveryPassInterval = 15 * time.Second + +// recoverySettleWindow is how long after this daemon started recovery stays +// inert when peers are still missing. +// +// It is the whole of the coordinated-shutdown suppression. The documented +// deploy path stops every node and then starts them, so the first one back +// finds every peer stale and every guest on them a candidate. Waiting until +// the peers are seen, or until this window expires, turns that into nothing +// happening. It is also why the window is generous: the cost of waiting is a +// slower recovery after a genuine failure, and the cost of not waiting is +// recovering guests from nodes that are merely still booting. +const recoverySettleWindow = 5 * time.Minute + +// recoveryMinimumNodes is the cluster size below which recovery cannot mean +// anything: on one or two nodes there is either nowhere to move a guest to, or +// no KV quorum left to move it with. +const recoveryMinimumNodes = 3 + +// recoveryBackoffBase and recoveryBackoffMax bound how often one node retries +// an instance whose launch failed. +// +// Some failures are not the node's to fix. A guest whose object store is +// refusing reads fails to mount identically on every survivor, and without a +// backoff each of them would claim it, fail, hand it back and claim it again +// every pass. Backing off turns that into a slow probe that costs nothing and +// still finishes the recovery the moment the store returns. +const ( + recoveryBackoffBase = 30 * time.Second + recoveryBackoffMax = 15 * time.Minute +) + +// recoveryLeaseBackoffBase and recoveryLeaseBackoffMax are the same bounds for +// the one failure that clears on a clock rather than on somebody fixing +// something: the node this instance was taken from still holds its volume +// lease, and will stop holding it within a TTL. Retrying that on the general +// backoff would spend minutes waiting for something already over, so it is +// faster and it is capped near two TTLs — past which the cause is no longer the +// one being waited for, and the attempt budget should be spent finding out. +const ( + recoveryLeaseBackoffBase = 15 * time.Second + recoveryLeaseBackoffMax = 90 * time.Second +) + +// recoveryMaxAttempts is how many times one node tries an instance before it +// stops and says so. +// +// Retrying forever was the previous behaviour and it is the worse failure. The +// instance stays observed-running on a node that is gone, nothing is visible to +// the customer beyond an instance that does not answer, and no operator is ever +// told. A budget converts that into a stopped instance carrying the reason, on +// the same path AWS leaves a host-failed instance on. At these delays the budget +// is roughly an hour and a half of trying, so anything that was going to clear +// has had far longer than the clocks it depends on. +const recoveryMaxAttempts = 12 + +// recoveryCandidate is one instance this node could take over, and why. +type recoveryCandidate struct { + record *vm.InstanceRecord + from string +} + +// startInstanceRecovery runs the recovery reconciler on this daemon. +// +// Every node runs the same loop with no leader between them. A leader would +// add failover state and no safety: the claim is a CAS on the instance record, +// so several nodes racing one instance already resolve to one winner. +// +// There is no setting for this. An instance whose host is gone is down either +// way, so the only thing a switch could buy is leaving it down, and a cluster +// configured into that state by accident is a worse outcome than any this +// recovers from. The conditions below are what make it inert, and each is a +// fact about the cluster rather than a choice about it. +func (d *Daemon) startInstanceRecovery() { + if d.clusterConfig == nil { + slog.Warn("Instance recovery not started: no cluster config") + return + } + if d.jsManager == nil || d.instanceService == nil { + slog.Warn("Instance recovery not started: JetStream or the instance service is unavailable") + return + } + if n := len(d.clusterConfig.Nodes); n < recoveryMinimumNodes { + slog.Info("Instance recovery is inert on a cluster this small", + "node", d.node, "nodes", n, "minimum", recoveryMinimumNodes) + return + } + + liveness := d.newLiveness() + if liveness == nil { + slog.Warn("Instance recovery not started: no liveness view") + return + } + rec := &instanceRecovery{ + daemon: d, + liveness: liveness, + startedAt: time.Now(), + staleOnce: map[string]struct{}{}, + backoff: map[string]recoveryBackoff{}, + } + + d.shutdownWg.Go(func() { + ticker := time.NewTicker(recoveryPassInterval) + defer ticker.Stop() + for { + select { + case <-d.ctx.Done(): + slog.Info("Instance recovery stopping") + return + case <-ticker.C: + rec.pass(d.ctx) + } + } + }) + + slog.Info("Instance recovery started", "node", d.node, + "interval_ms", otelsetup.Millis(recoveryPassInterval), + "settle_ms", otelsetup.Millis(recoverySettleWindow)) +} + +// instanceRecovery is one daemon's recovery loop. The only state it keeps is +// which instances were already seen stale, which is what makes a second pass a +// condition rather than a duplicate of the first. +type instanceRecovery struct { + daemon *Daemon + liveness *instancecache.Liveness + + startedAt time.Time + settledOnce bool + staleOnce map[string]struct{} + backoff map[string]recoveryBackoff +} + +// recoveryBackoff is how long this node has agreed to leave an instance alone +// after a launch that did not complete, and how many attempts it has made. +type recoveryBackoff struct { + until time.Time + attempts int +} + +// pass derives the work from the records once, then attempts what this node can +// take. Nothing is stored between passes but the stale-once set, so a daemon +// that restarts derives exactly the same work again. +func (r *instanceRecovery) pass(ctx context.Context) { + state := r.peerState(ctx) + if !r.settled(state) || !r.canHostRecoveries(ctx) { + return + } + + records, err := r.daemon.jsManager.ListInstanceRecords() + if err != nil { + slog.Warn("Instance recovery could not read the instance records", "err", err) + return + } + + for _, candidate := range r.selectCandidates(records, state) { + if ctx.Err() != nil { + return + } + if r.backingOff(candidate.record.Metadata.Name, time.Now()) { + continue + } + r.attempt(ctx, candidate) + } +} + +// canHostRecoveries reports whether this node's own object store is answering, +// which is the difference between the two failures that look alike from here. +// +// A node cannot mount a volume its object store will not serve, so claiming one +// would take an instance off its dead owner only to fail, and hold it away from +// a survivor that could have run it. Standing down separates the cases without +// anything having to diagnose them: if the store is broken only here, the other +// survivors carry on and the instance moves; if it is broken everywhere, every +// node stands down and the instance stays where it is, which is the honest +// answer because no node could have run it. +func (r *instanceRecovery) canHostRecoveries(ctx context.Context) bool { + verdict := probePredastore(ctx, r.daemon) + if verdict == predastoreHealthOK { + return true + } + slog.Warn("Instance recovery is standing down: this node's object store is not answering", + "node", r.daemon.node, "predastore", verdict) + return false +} + +// backingOff reports whether this node has agreed to leave an instance alone +// for now, and forgets the agreement once it has lapsed. +func (r *instanceRecovery) backingOff(id string, now time.Time) bool { + held, ok := r.backoff[id] + if !ok { + return false + } + return now.Before(held.until) +} + +// deferRetry backs this node off an instance whose launch did not complete, and +// reports how many attempts it has now made. +// +// The delay doubles per attempt because the two reasons a launch fails want +// opposite things. A volume lease the old owner has not released yet clears +// within one TTL and wants a prompt retry; an object store refusing reads clears +// when an operator fixes it and wants no traffic at all in the meantime. Doubling +// serves the first without spending the cluster on the second, and the bounds it +// doubles between come from the fault, which is the other half of the same +// distinction. +// +// Every fault counts against the same budget. A class that did not count would +// be the old unbounded behaviour under a narrower condition, and the conditions +// this can read are not precise enough to be trusted with that. +func (r *instanceRecovery) deferRetry(id string, fault recoveryFault, now time.Time) int { + base, limit := fault.backoff() + held := r.backoff[id] + held.attempts++ + delay := min(base<= recoverySettleWindow { + r.settledOnce = true + return true + } + for name := range r.daemon.clusterConfig.Nodes { + if name == r.daemon.node { + continue + } + if state(name) != instancecache.NodeLive { + return false + } + } + r.settledOnce = true + slog.Info("Instance recovery has seen every peer live and is now active", "node", r.daemon.node) + return true +} + +// selectCandidates is the whole of the policy, and is pure over what it is +// given so the matrix is testable without a cluster. +// +// NodeUnknown is never a candidate. An unreadable heartbeat is not evidence +// that a node is gone, and treating it as one turns a KV outage into a +// cluster-wide relaunch. +func (r *instanceRecovery) selectCandidates(records []*vm.InstanceRecord, state func(string) instancecache.NodeState) []recoveryCandidate { + var out []recoveryCandidate + seen := map[string]struct{}{} + + for _, record := range records { + owner := record.Status.LastNode + if owner == "" || owner == r.daemon.node || !recoverable(record) { + continue + } + if state(owner) != instancecache.NodeStale { + continue + } + + // Two consecutive passes, because liveness is computed from the + // timestamp the owner wrote into its own heartbeat rather than from + // when the store received it. One sample is therefore one node's clock; + // two are a condition that outlived a pass. + id := record.Metadata.Name + seen[id] = struct{}{} + if _, twice := r.staleOnce[id]; !twice { + continue + } + out = append(out, recoveryCandidate{record: record, from: owner}) + } + + for id := range r.staleOnce { + if _, still := seen[id]; !still { + delete(r.staleOnce, id) + } + } + for id := range seen { + r.staleOnce[id] = struct{}{} + } + + // Every survivor derives the same candidates, so a shared order would have + // them all race the same instance first and lose to the same winner. Hashing + // the pair gives each node its own order for free: no coordination, no + // randomness to make a failure unreproducible, and the recoveries spread. + slices.SortFunc(out, func(a, b recoveryCandidate) int { + return cmp.Compare(r.preference(a.record.Metadata.Name), r.preference(b.record.Metadata.Name)) + }) + return out +} + +// preference orders one node's candidates differently from every other node's. +func (r *instanceRecovery) preference(instanceID string) uint64 { + h := fnv.New64a() + _, _ = h.Write([]byte(r.daemon.node)) + _, _ = h.Write([]byte(instanceID)) + return h.Sum64() +} + +// peerState is the liveness view as a function, so everything that decides on +// it stays pure over the answer and testable without a cluster. +func (r *instanceRecovery) peerState(ctx context.Context) func(string) instancecache.NodeState { + return func(node string) instancecache.NodeState { + return r.liveness.State(ctx, node) + } +} + +// attempt claims one instance and launches it here. +// +// Claim before admission, not after: the claim is what stops two survivors +// doing the same expensive setup, and a node that turns out not to fit hands +// the record straight back. +// A recovery is three phases that already trace themselves — the volumes +// opening, QEMU starting, and its monitor answering — and nothing tied them to +// the attempt that asked for them. This span is that tie: the phases become its +// children, so one trace ID reads end to end across an attempt and a phase that +// took the whole of it can be named rather than guessed at. +func (r *instanceRecovery) attempt(ctx context.Context, candidate recoveryCandidate) { + id := candidate.record.Metadata.Name + + ctx, span := otel.Tracer(daemonTracerName).Start(ctx, "recovery.attempt", + trace.WithAttributes( + attribute.String("instance.id", id), + attribute.String("recovery.from", candidate.from), + attribute.String("recovery.to", r.daemon.node), + )) + defer span.End() + + instance, err := r.daemon.jsManager.ClaimRecoverableInstance(id, candidate.from, r.daemon.node) + if err != nil { + if !errors.Is(err, vm.ErrRecoveryClaimLost) { + slog.WarnContext(ctx, "Instance recovery could not claim an instance", "instanceId", id, "err", err) + } + return + } + + undo := func() { + if err := r.daemon.jsManager.ReleaseRecoveredInstance(id, r.daemon.node, candidate.from); err != nil { + slog.ErrorContext(ctx, "Instance recovery could not hand a claimed instance back", + "instanceId", id, "to", candidate.from, "err", err) + } + } + + slog.InfoContext(ctx, "Recovering an instance from a node that stopped heartbeating", + "instanceId", id, "from", candidate.from, "to", r.daemon.node) + + if err := r.daemon.instanceService.RecoverInstance(ctx, instance, undo); err != nil { + // Expected while the old owner's volume lease is still valid: its own + // self-fence has to run before anyone else may open the volume. + fault := classifyRecoveryFault(err) + attempts := r.deferRetry(id, fault, time.Now()) + span.SetAttributes( + attribute.String("recovery.fault", fault.String()), + attribute.Int("recovery.attempts", attempts)) + span.RecordError(err) + span.SetStatus(codes.Error, err.Error()) + slog.WarnContext(ctx, "Instance recovery attempt did not complete", + "instanceId", id, "from", candidate.from, "fault", fault.String(), + "attempts", attempts, "budget", recoveryMaxAttempts, "err", err) + if attempts >= recoveryMaxAttempts { + r.abandon(ctx, id, candidate.from, fault, err, attempts) + } + return + } + + delete(r.staleOnce, id) + delete(r.backoff, id) + + // The guest is running here and the cloud underneath still delivers its + // public address to the node that died, so this is the last thing between a + // recovered instance and a reachable one. It runs in line rather than waiting + // for the affinity ticker, and a guest whose address does not arrive is + // marked impaired by the pass rather than reported healthy. + r.daemon.ClaimOCIAddresses(ctx) + + slog.InfoContext(ctx, "Recovered an instance onto this node", + "instanceId", id, "from", candidate.from, "node", r.daemon.node) +} + +// abandon stops trying and leaves the instance stopped with the reason it +// stopped for, which is the only outcome here that is visible to the customer. +// +// This is the same place AWS leaves an instance whose host failed and could not +// be brought back: stopped, carrying a StateReason, one StartInstances away from +// running. So the customer's retry is the ordinary API call on whichever node +// has room, and nothing here needs an operator-only path to undo it. +// +// A record another node has since claimed and launched is not touched, and a +// write that does not land leaves the backoff in place — the next pass tries the +// launch again rather than the give-up, which is the safer of the two to repeat. +func (r *instanceRecovery) abandon(ctx context.Context, id, from string, fault recoveryFault, cause error, attempts int) { + reason := fmt.Sprintf("recovery onto %s failed %d times, last: %v", r.daemon.node, attempts, cause) + + abandoned, err := r.daemon.jsManager.AbandonRecovery(id, from, fault.stateReasonCode(), reason) + if err != nil { + slog.ErrorContext(ctx, "Instance recovery could not record that it has given up", + "instanceId", id, "from", from, "err", err) + return + } + if !abandoned { + slog.InfoContext(ctx, "Instance recovery gave up on an instance that is no longer its to give up on", + "instanceId", id, "from", from) + delete(r.backoff, id) + delete(r.staleOnce, id) + return + } + + delete(r.backoff, id) + delete(r.staleOnce, id) + slog.ErrorContext(ctx, "Instance recovery has given up: the instance is stopped and needs a start", + "instanceId", id, "from", from, "node", r.daemon.node, "attempts", attempts, + "fault", fault.String(), "code", fault.stateReasonCode(), "err", cause) +} + +// newLiveness opens the daemon's own read-only view of the heartbeat store. +// The gateway builds one of these too; they share the bucket and neither owns +// it, so a second reader costs one memoised list per pass. +func (d *Daemon) newLiveness() *instancecache.Liveness { + if d.jsManager == nil || d.jsManager.js == nil { + return nil + } + return instancecache.NewLiveness(d.jsManager.js, clusterStateConfig(d.jsManager.replicas)) +} + +// forgetSupersededInstances drops any local copy of an instance the records say +// another node now runs, before restore can relaunch it. +// +// A returning node loads its own state file, and an instance recovered while it +// was away is still in there. Nothing else stops it starting a second copy: the +// cluster's view of this node no longer lists that instance, but the local file +// is folded in on top of it, and the running-set writer would then stamp this +// node back over the winner. +// +// A record that cannot be read leaves the instance alone. An unreadable store +// is not evidence that an instance moved, and the volume lease refuses a second +// writer either way. +func (d *Daemon) forgetSupersededInstances() { + if d.jsManager == nil { + return + } + + for _, instance := range d.vmMgr.Snapshot() { + record, err := d.jsManager.LoadInstanceRecord(instance.ID) + if err != nil { + slog.Warn("Could not tell whether an instance moved while this node was away", + "instanceId", instance.ID, "err", err) + continue + } + if record == nil || record.Status.LastNode == "" || record.Status.LastNode == d.node { + continue + } + + slog.Info("An instance here is owned by another node now; dropping the local copy", + "instanceId", instance.ID, "owner", record.Status.LastNode, "node", d.node) + d.vmMgr.ForgetSuperseded(instance) + } +} diff --git a/spinifex/daemon/instance_recovery_test.go b/spinifex/daemon/instance_recovery_test.go new file mode 100644 index 000000000..621307509 --- /dev/null +++ b/spinifex/daemon/instance_recovery_test.go @@ -0,0 +1,410 @@ +//test:in-package — selectCandidates, settled and the backoff are the whole of +//the policy and are deliberately unexported. Exporting them to test them would +//make the reconciler's internals part of the daemon's API for no other reason. + +package daemon + +import ( + "bytes" + "context" + "errors" + "fmt" + "log/slog" + "reflect" + "strings" + "testing" + "time" + + "github.com/mulgadc/spinifex/spinifex/awserrors" + "github.com/mulgadc/spinifex/spinifex/config" + handlers_ec2_instance "github.com/mulgadc/spinifex/spinifex/handlers/ec2/instance" + "github.com/mulgadc/spinifex/spinifex/instancecache" + "github.com/mulgadc/spinifex/spinifex/resource" + "github.com/mulgadc/spinifex/spinifex/vm" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// recoveryFixture is the reconciler with no daemon behind it. Selection is the +// whole of the policy and is pure over the records and the liveness answer, so +// none of these tests needs a cluster. +func recoveryFixture(self string) *instanceRecovery { + return &instanceRecovery{ + daemon: &Daemon{node: self}, + staleOnce: map[string]struct{}{}, + backoff: map[string]recoveryBackoff{}, + } +} + +func runningOn(id, node string) *vm.InstanceRecord { + return &vm.InstanceRecord{ + Metadata: resource.Metadata{Name: id, AccountID: "111122223333"}, + Spec: vm.InstanceSpec{InstanceType: "t3.micro", DesiredState: vm.DesiredRunning}, + Status: vm.InstanceStatus{Status: vm.StateRunning, LastNode: node}, + } +} + +func stale(node string) instancecache.NodeState { + return instancecache.NodeStale +} + +// twice runs two passes, which is what the reconciler requires before it acts. +func twice(r *instanceRecovery, records []*vm.InstanceRecord, state func(string) instancecache.NodeState) []recoveryCandidate { + r.selectCandidates(records, state) + return r.selectCandidates(records, state) +} + +func TestARecordOwnedByAStaleNodeIsRecovered(t *testing.T) { + r := recoveryFixture("node-2") + got := twice(r, []*vm.InstanceRecord{runningOn("i-1", "node-1")}, stale) + + require.Len(t, got, 1) + assert.Equal(t, "i-1", got[0].record.Metadata.Name) + assert.Equal(t, "node-1", got[0].from, "the candidate has to name who it is being taken from") +} + +// One sample is one node's clock. Liveness compares the timestamp the owner +// wrote into its own heartbeat, so a single stale reading is as likely to be +// skew as a failure. +func TestOneStaleSampleIsNotEnough(t *testing.T) { + r := recoveryFixture("node-2") + assert.Empty(t, r.selectCandidates([]*vm.InstanceRecord{runningOn("i-1", "node-1")}, stale)) +} + +// A node that came back between the two passes must not still be a candidate +// on the strength of the first one. +func TestAnOwnerThatRecoversBetweenPassesIsForgotten(t *testing.T) { + r := recoveryFixture("node-2") + records := []*vm.InstanceRecord{runningOn("i-1", "node-1")} + + r.selectCandidates(records, stale) + assert.Empty(t, r.selectCandidates(records, func(string) instancecache.NodeState { + return instancecache.NodeLive + })) + assert.Empty(t, r.selectCandidates(records, stale), + "the run of stale passes has to start again, not resume where it left off") +} + +// An unreadable heartbeat store is not evidence that a node is gone. Treating +// it as one turns a KV outage into a cluster-wide relaunch. +func TestAnUnknownOwnerIsNeverACandidate(t *testing.T) { + r := recoveryFixture("node-2") + got := twice(r, []*vm.InstanceRecord{runningOn("i-1", "node-1")}, func(string) instancecache.NodeState { + return instancecache.NodeUnknown + }) + assert.Empty(t, got) +} + +func TestThisNodesOwnInstancesAreNeverCandidates(t *testing.T) { + r := recoveryFixture("node-1") + assert.Empty(t, twice(r, []*vm.InstanceRecord{runningOn("i-1", "node-1")}, stale)) +} + +// Desired state is what separates a recovery from a stop. A drain leaves an +// instance stopped but still wanted, and that is a recovery; an operator stop +// leaves it stopped and not wanted, and that is not. +func TestWhatDesiredStateAdmits(t *testing.T) { + drained := runningOn("i-drained", "node-1") + drained.Status.Status = vm.StateStopped + + stopped := runningOn("i-stopped", "node-1") + stopped.Status.Status = vm.StateStopped + stopped.Spec.DesiredState = vm.DesiredStopped + + terminating := runningOn("i-terminating", "node-1") + terminating.Status.Status = vm.StateShuttingDown + + deleting := runningOn("i-deleting", "node-1") + now := time.Now() + deleting.Metadata.DeletionTimestamp = &now + + r := recoveryFixture("node-2") + got := twice(r, []*vm.InstanceRecord{drained, stopped, terminating, deleting}, stale) + + require.Len(t, got, 1) + assert.Equal(t, "i-drained", got[0].record.Metadata.Name, + "a drain-stopped instance is still wanted running, so it is recovered") +} + +func TestARecordWithNoOwnerIsNotACandidate(t *testing.T) { + r := recoveryFixture("node-2") + assert.Empty(t, twice(r, []*vm.InstanceRecord{runningOn("i-1", "")}, stale)) +} + +// A guest paused because its storage stopped answering is not a host failure, +// and it is the one failure a different host cannot fix: the volumes are the +// same objects in the same store from anywhere. +func TestAnInstancePausedOnItsStorageIsNotMoved(t *testing.T) { + faulted := runningOn("i-faulted", "node-1") + faulted.Status.Health.IOErrorSince = time.Now().Add(-2 * time.Minute) + faulted.Status.Health.IOErrorResumes = 4 + + healthy := runningOn("i-healthy", "node-1") + + r := recoveryFixture("node-2") + got := twice(r, []*vm.InstanceRecord{faulted, healthy}, stale) + + require.Len(t, got, 1) + assert.Equal(t, "i-healthy", got[0].record.Metadata.Name, + "only the guest whose storage was answering is worth moving") +} + +// A guest that had recovered from an earlier pause is an ordinary candidate. +// IOErrorSince clears on the first running poll, so a non-zero value means the +// storage was still failing when the owner last wrote. +func TestAPastStoragePauseThatClearedDoesNotBlockRecovery(t *testing.T) { + recovered := runningOn("i-1", "node-1") + recovered.Status.Health.IOErrorResumes = 0 + + r := recoveryFixture("node-2") + assert.Len(t, twice(r, []*vm.InstanceRecord{recovered}, stale), 1) +} + +// The settle window asks whether this node is still coming up alongside its +// peers, which is answered once. Asked afresh every pass it would become +// "are the peers live now" and refuse to act on the failure it exists for. +func TestTheSettleWindowLatchesOnceThePeersHaveBeenSeen(t *testing.T) { + r := recoveryFixture("node-1") + r.startedAt = time.Now() + r.daemon.clusterConfig = &config.ClusterConfig{ + Nodes: map[string]config.Config{"node-1": {}, "node-2": {}, "node-3": {}}, + } + + live := map[string]instancecache.NodeState{ + "node-2": instancecache.NodeLive, "node-3": instancecache.NodeLive, + } + state := func(node string) instancecache.NodeState { return live[node] } + require.True(t, r.settled(state), "every peer is live, so the cluster is not mid-restart") + + live["node-2"] = instancecache.NodeStale + assert.True(t, r.settled(state), + "a peer going away afterwards is the failure to recover from, not a reason to stand down") +} + +func TestTheSettleWindowHoldsWhileAPeerHasNotBeenSeen(t *testing.T) { + r := recoveryFixture("node-1") + r.startedAt = time.Now() + r.daemon.clusterConfig = &config.ClusterConfig{ + Nodes: map[string]config.Config{"node-1": {}, "node-2": {}}, + } + assert.False(t, r.settled(stale), + "a node restarted alongside its peers must not recover their guests while they boot") +} + +func TestTheSettleWindowReleasesWhenItExpires(t *testing.T) { + r := recoveryFixture("node-1") + r.startedAt = time.Now().Add(-recoverySettleWindow - time.Second) + r.daemon.clusterConfig = &config.ClusterConfig{ + Nodes: map[string]config.Config{"node-1": {}, "node-2": {}}, + } + assert.True(t, r.settled(stale), + "a peer that never returns is the genuine failure, with nothing to compare against") +} + +// Every refusal that stops the loop starting, and none of them is a setting. +// Each is a fact about the cluster — not wired up, or not big enough for +// recovery to mean anything — so no cluster can be configured into leaving a +// dead node's guests down. +func TestWhatStopsTheLoopStarting(t *testing.T) { + threeNodes := map[string]config.Config{"node-1": {}, "node-2": {}, "node-3": {}} + + tests := []struct { + name string + reason string + build func() *Daemon + }{ + {"no cluster config to read the node list from", "no cluster config", func() *Daemon { + return &Daemon{node: "node-1"} + }}, + {"no JetStream to read records from", "JetStream or the instance service is unavailable", func() *Daemon { + return &Daemon{node: "node-1", clusterConfig: &config.ClusterConfig{Nodes: threeNodes}} + }}, + {"too few nodes to move a guest between", "inert on a cluster this small", func() *Daemon { + return &Daemon{ + node: "node-1", + clusterConfig: &config.ClusterConfig{Nodes: map[string]config.Config{"node-1": {}, "node-2": {}}}, + jsManager: &JetStreamManager{}, + instanceService: &handlers_ec2_instance.InstanceServiceImpl{}, + } + }}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + var logs bytes.Buffer + prev := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelInfo}))) + t.Cleanup(func() { slog.SetDefault(prev) }) + + d := tc.build() + require.NotPanics(t, d.startInstanceRecovery, + "a refusal to start has to return, not fail") + assert.Contains(t, logs.String(), tc.reason, + "the operator has to be told which condition made recovery inert here") + }) + } +} + +// TestRecoveryHasNoOffSwitch pins the absence itself, because the cheapest way +// to reintroduce one is to add a field nobody notices. A guest whose host is +// gone is down either way, so a switch could only buy leaving it down. +func TestRecoveryHasNoOffSwitch(t *testing.T) { + for field := range reflect.TypeFor[config.ClusterConfig]().Fields() { + assert.NotContains(t, strings.ToLower(field.Name), "recover", + "ClusterConfig.%s reads as a recovery setting, and recovery is not configurable", field.Name) + } +} + +// The difference between a store broken here and a store broken everywhere is +// made by every node asking only about itself. A node that cannot serve objects +// cannot mount a volume, so claiming one would take an instance off its dead +// owner only to fail, and hold it away from a survivor that could have run it. +func TestANodeWhoseObjectStoreIsDownClaimsNothing(t *testing.T) { + for _, verdict := range []string{predastoreHealthUnreachable, predastoreHealthNoLeader} { + t.Run(verdict, func(t *testing.T) { + r := recoveryFixture("node-1") + r.daemon.predastoreHealth.at = time.Now() + r.daemon.predastoreHealth.result = verdict + + assert.False(t, r.canHostRecoveries(context.Background())) + }) + } +} + +func TestANodeWhoseObjectStoreIsHealthyMayClaim(t *testing.T) { + r := recoveryFixture("node-1") + r.daemon.predastoreHealth.at = time.Now() + r.daemon.predastoreHealth.result = predastoreHealthOK + + assert.True(t, r.canHostRecoveries(context.Background())) +} + +// A pass with nothing wired behind it must return rather than fail. The store +// verdict is read before the records are, so a node standing down never reaches +// the reader at all. +func TestAPassStandsDownBeforeReadingAnything(t *testing.T) { + r := recoveryFixture("node-1") + r.settledOnce = true + r.daemon.predastoreHealth.at = time.Now() + r.daemon.predastoreHealth.result = predastoreHealthUnreachable + + assert.NotPanics(t, func() { r.pass(context.Background()) }, + "jsManager is nil here, so reaching the record read would panic") +} + +// An unreadable record is not evidence that an instance moved, so a returning +// node with no way to ask leaves its local copies alone. +func TestForgettingSupersededInstancesNeedsAStoreToAsk(t *testing.T) { + d := &Daemon{node: "node-1"} + assert.NotPanics(t, d.forgetSupersededInstances) +} + +// A launch that cannot complete has to cost less each time it fails. Some +// causes clear in seconds and some need an operator, and nothing here can tell +// which, so the retry rate has to suit the second. +func TestRetriesBackOffAndCapAfterAFailedLaunch(t *testing.T) { + r := recoveryFixture("node-2") + now := time.Now() + + assert.False(t, r.backingOff("i-1", now), "an instance never tried is not backed off") + + require.Equal(t, 1, r.deferRetry("i-1", recoveryFaultUnknown, now)) + assert.True(t, r.backingOff("i-1", now.Add(recoveryBackoffBase-time.Second))) + assert.False(t, r.backingOff("i-1", now.Add(recoveryBackoffBase))) + + require.Equal(t, 2, r.deferRetry("i-1", recoveryFaultUnknown, now)) + assert.True(t, r.backingOff("i-1", now.Add(2*recoveryBackoffBase-time.Second)), + "the second failure has to wait longer than the first") + + for range 20 { + r.deferRetry("i-1", recoveryFaultUnknown, now) + } + assert.False(t, r.backingOff("i-1", now.Add(recoveryBackoffMax)), + "the delay is capped, so a fault that is fixed is noticed without an operator nudge") +} + +// TestALeaseStillHeldIsRetriedFasterThanAnythingElse pins the one distinction +// the backoff makes. The old owner's lease lapses on a clock nobody has to act +// on, so waiting minutes for it is time an instance spends down for nothing. +func TestALeaseStillHeldIsRetriedFasterThanAnythingElse(t *testing.T) { + r := recoveryFixture("node-2") + now := time.Now() + + require.Equal(t, 1, r.deferRetry("i-lease", recoveryFaultLeaseHeld, now)) + assert.False(t, r.backingOff("i-lease", now.Add(recoveryLeaseBackoffBase))) + + for range 20 { + r.deferRetry("i-lease", recoveryFaultLeaseHeld, now) + } + assert.False(t, r.backingOff("i-lease", now.Add(recoveryLeaseBackoffMax))) + assert.Less(t, recoveryLeaseBackoffMax, recoveryBackoffMax, + "a cause that clears on a lease TTL must not be waited on like one that needs an operator") +} + +// TestTheAttemptBudgetIsSharedByEveryFault is the whole reason there is a +// budget. A class of failure that did not count against it would be the old +// unbounded retry under a narrower condition, and the conditions this code can +// read are not precise enough to be trusted with that. +func TestTheAttemptBudgetIsSharedByEveryFault(t *testing.T) { + r := recoveryFixture("node-2") + now := time.Now() + + faults := []recoveryFault{recoveryFaultLeaseHeld, recoveryFaultCapacity, recoveryFaultUnknown} + for i := range recoveryMaxAttempts { + attempts := r.deferRetry("i-1", faults[i%len(faults)], now) + require.Equal(t, i+1, attempts, + "every fault counts, so the budget cannot be extended by failing a different way") + } + assert.Equal(t, recoveryMaxAttempts, r.backoff["i-1"].attempts, + "reaching the budget is what makes the give-up happen, so the count has to be exact") +} + +// TestWhatTheLaunchRefusalMeansForTheNextAttempt is the classifier. The two +// cases it names are the two the launch reports distinguishably; everything else +// is unknown on purpose, because a fault named "will never succeed" on this +// evidence would be a guess, and the guess that abandons a recoverable instance +// is the expensive one. +func TestWhatTheLaunchRefusalMeansForTheNextAttempt(t *testing.T) { + tests := []struct { + name string + err error + want recoveryFault + code string + }{ + { + name: "the old owner still holds the volume lease", + err: fmt.Errorf("mount refused: %w", fmt.Errorf("%w: %w", + handlers_ec2_instance.ErrVolumeHeldElsewhere, + awserrors.Errorf(awserrors.ErrorIncorrectState, "volume is leased by another owner: node-1"))), + want: recoveryFaultLeaseHeld, + code: "Server.HostRecoveryFailed", + }, + { + name: "this node has no room for the instance", + err: errors.New(awserrors.ErrorInsufficientInstanceCapacity), + want: recoveryFaultCapacity, + code: "Server.InsufficientInstanceCapacity", + }, + { + name: "everything the launch collapses into ServerInternal", + err: errors.New(awserrors.ErrorServerInternal), + want: recoveryFaultUnknown, + code: "Server.HostRecoveryFailed", + }, + { + name: "an error carrying no AWS code at all", + err: errors.New("nbdkit exited 1"), + want: recoveryFaultUnknown, + code: "Server.HostRecoveryFailed", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := classifyRecoveryFault(tc.err) + assert.Equal(t, tc.want, got, "classified as %s", got) + assert.Equal(t, tc.code, got.stateReasonCode(), + "the state reason is what the customer sees, so it has to follow the classification") + }) + } +} diff --git a/spinifex/daemon/instance_recovery_trace_test.go b/spinifex/daemon/instance_recovery_trace_test.go new file mode 100644 index 000000000..1eb52122b --- /dev/null +++ b/spinifex/daemon/instance_recovery_trace_test.go @@ -0,0 +1,42 @@ +//test:in-package — attempt and the span it opens are unexported, and the span is +//the whole point: it is what ties a recovery's phases to the attempt that asked +//for them. + +package daemon + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel/attribute" +) + +// The phases of a launch already trace themselves. Without a span over the +// attempt they are orphans: three unrelated traces per recovery, no way to ask +// which phase a slow recovery spent its time in, and nothing naming the node it +// came from. So the span is the correlation, and its attributes are the answer to +// "what was this". +func TestARecoveryAttemptIsOneTraceFromClaimToLaunch(t *testing.T) { + sr := withRecordedSpans(t) + + r := recoveryFixture("node-2") + // No KV bucket, so the claim fails immediately. The span is opened before the + // claim and ended by defer, which is exactly what has to hold on the path + // that gives up earliest. + r.daemon.jsManager = &JetStreamManager{} + r.attempt(t.Context(), recoveryCandidate{record: runningOn("i-1", "node-1"), from: "node-1"}) + + spans := sr.Ended() + require.Len(t, spans, 1) + assert.Equal(t, "recovery.attempt", spans[0].Name()) + + got := map[attribute.Key]string{} + for _, kv := range spans[0].Attributes() { + got[kv.Key] = kv.Value.AsString() + } + assert.Equal(t, "i-1", got["instance.id"]) + assert.Equal(t, "node-1", got["recovery.from"], + "a recovery is only readable if the trace says where the guest came from") + assert.Equal(t, "node-2", got["recovery.to"]) +} diff --git a/spinifex/daemon/jetstream.go b/spinifex/daemon/jetstream.go index 210808f75..6f278f2d7 100644 --- a/spinifex/daemon/jetstream.go +++ b/spinifex/daemon/jetstream.go @@ -9,6 +9,8 @@ import ( "strings" "time" + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/ec2" "github.com/mulgadc/spinifex/spinifex/kvstore" "github.com/mulgadc/spinifex/spinifex/kvutil" "github.com/mulgadc/spinifex/spinifex/migrate" @@ -688,6 +690,119 @@ func (m *JetStreamManager) ClaimStoppedInstance(instanceID string) (*vm.VM, erro return vm.VMFromRecord(&claimed), nil } +// ClaimRecoverableInstance takes an instance off a node that has stopped +// heartbeating, by CASing Status.LastNode from that node to this one. +// +// The CAS is the whole of the coordination. Several survivors see the same +// candidate and race; exactly one revision wins, and the losers find a record +// that names a live owner and stop considering it. Nothing else is needed: a +// claimant that dies mid-launch leaves the record naming a node that is itself +// stale, which is the same candidate the next pass derives again. +// +// Ownership is not permission to write. The volume leases still decide whether +// the guest can actually move, and a claimant that cannot open them hands the +// record back with ReleaseRecoveredInstance. +func (m *JetStreamManager) ClaimRecoverableInstance(instanceID, from, to string) (*vm.VM, error) { + if m.records == nil { + return nil, errors.New("KV bucket not initialized") + } + + key := instanceRecordKey(instanceID) + record, rev, err := m.records.Get(context.Background(), key) + if err != nil { + if errors.Is(err, kvstore.ErrNotFound) { + return nil, vm.ErrRecoveryClaimLost + } + return nil, err + } + if record.Status.LastNode != from || !recoverable(record) { + return nil, vm.ErrRecoveryClaimLost + } + + claimed := *record + claimed.Status.LastNode = to + if _, err := m.records.CompareAndSet(context.Background(), key, &claimed, rev); err != nil { + if errors.Is(err, kvstore.ErrConflict) { + return nil, vm.ErrRecoveryClaimLost + } + return nil, err + } + + slog.Info("Claimed an instance from a node that stopped heartbeating", + "instanceId", instanceID, "from", from, "to", to) + return vm.VMFromRecord(&claimed), nil +} + +// ReleaseRecoveredInstance hands a claimed record back to the node it was taken +// from, after a launch this node could not complete. +// +// Without it a failed attempt parks the instance on a node that has just proved +// it cannot run it, and no other survivor will look at it again: the record +// would name a live owner. Best-effort — the claimant may itself be failing — +// and a release that does not land is corrected by the next pass once this node +// stops heartbeating, or by an operator. +func (m *JetStreamManager) ReleaseRecoveredInstance(instanceID, self, to string) error { + if m.records == nil { + return errors.New("KV bucket not initialized") + } + _, err := m.UpdateInstanceRecord(instanceID, func(record *vm.InstanceRecord) { + if record.Status.LastNode == self { + record.Status.LastNode = to + } + }) + if err != nil && !errors.Is(err, kvstore.ErrNotFound) { + return err + } + return nil +} + +// AbandonRecovery stops an instance being a recovery candidate on every node, +// by writing the giving-up into the record they all derive their work from. +// +// The record is left in the shape a stop leaves — observed stopped, wanted +// stopped, carrying a StateReason that names why — which is what AWS leaves +// behind when a host fails and the instance cannot be brought back. So the +// customer's own StartInstances is the retry and needs no special case, and +// nothing has to remember a cluster-wide decision: recoverable() tests desired +// state, so one node's write ends the retrying everywhere at once. That is the +// same property the claim has, for the same reason. +// +// from is checked, so a node that gives up after another has already claimed +// and launched the instance cannot stop a guest that is now running. Reports +// whether the record was still the one being given up on. +func (m *JetStreamManager) AbandonRecovery(instanceID, from, code, reason string) (bool, error) { + if m.records == nil { + return false, errors.New("KV bucket not initialized") + } + + applied := false + _, err := m.UpdateInstanceRecord(instanceID, func(record *vm.InstanceRecord) { + // Reset per attempt: the mutation is replayed on a revision conflict, + // and a retry that finds the record moved must not report the first + // read's answer. + applied = false + if record.Status.LastNode != from || !recoverable(record) { + return + } + applied = true + record.Spec.DesiredState = vm.DesiredStopped + record.Status.Status = vm.StateStopped + if record.Status.Instance != nil { + record.Status.Instance.StateReason = &ec2.StateReason{ + Code: aws.String(code), + Message: aws.String(reason), + } + } + }) + if err != nil { + if errors.Is(err, kvstore.ErrNotFound) { + return false, nil + } + return false, err + } + return applied, nil +} + // UpdateStoppedInstance atomically applies mutate to the current KV-stored // stopped record for instanceID and writes it back using optimistic // concurrency (CAS), retrying on a concurrent writer's revision conflict. diff --git a/spinifex/daemon/vm_adapters.go b/spinifex/daemon/vm_adapters.go index 2f7c4aba2..d31cd9cf4 100644 --- a/spinifex/daemon/vm_adapters.go +++ b/spinifex/daemon/vm_adapters.go @@ -16,6 +16,7 @@ import ( "github.com/mulgadc/spinifex/spinifex/network/topology" "github.com/mulgadc/spinifex/spinifex/objectstore" "github.com/mulgadc/spinifex/spinifex/otelsetup" + "github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire" "github.com/mulgadc/spinifex/spinifex/tags" "github.com/mulgadc/spinifex/spinifex/types" "github.com/mulgadc/spinifex/spinifex/utils" @@ -268,6 +269,54 @@ func (a *volumeMounterAdapter) Unmount(ctx context.Context, instance *vm.VM) err return errors.Join(sealErrs...) } +// abandonTimeout bounds one ebs.abandon request. Generous enough to cover +// viperblockd's SIGKILL wait for an nbdkit stuck in the storage path, and far +// short of the unmount's budget because nothing here seals. +const abandonTimeout = 45 * time.Second + +// Abandon tears down this node's export of every volume the instance had open, +// without sealing any of them. +// +// Every volume is attempted and the failures are aggregated, because a volume +// left exported is one the new owner cannot open — the caller is dropping a +// superseded instance and needs to know whether it actually let go. +func (a *volumeMounterAdapter) Abandon(ctx context.Context, instance *vm.VM, reason string) error { + instance.EBSRequests.Mu.Lock() + defer instance.EBSRequests.Mu.Unlock() + + var errs []error + for _, ebsRequest := range instance.EBSRequests.Requests { + payload, err := json.Marshal(vbwire.VolumeAbandonRequest{Volume: ebsRequest.Name, Reason: reason}) + if err != nil { + errs = append(errs, fmt.Errorf("marshal abandon request for %s: %w", ebsRequest.Name, err)) + continue + } + + msg, err := ebsRequestWithTrace(ctx, a.nc, instance.AccountID, + vbwire.VolumeAbandonSubject(a.node), payload, abandonTimeout) + if err != nil { + errs = append(errs, fmt.Errorf("abandon %s: %w", ebsRequest.Name, err)) + continue + } + + var resp vbwire.VolumeAbandonResponse + if err := json.Unmarshal(msg.Data, &resp); err != nil { + errs = append(errs, fmt.Errorf("unmarshal abandon response for %s: %w", ebsRequest.Name, err)) + continue + } + if resp.Error != "" { + errs = append(errs, fmt.Errorf("abandon %s: %s", ebsRequest.Name, resp.Error)) + continue + } + if resp.Abandoned { + slog.WarnContext(ctx, "Gave up a volume export to the node that owns the instance now", + "instance", instance.ID, "volume", ebsRequest.Name, "reason", reason) + } + } + + return errors.Join(errs...) +} + // MountOne sends ebs.mount for a single request and writes the resolved // NBDURI back into req.NBDURI. Used by hot-attach (Manager.AttachVolume). func (a *volumeMounterAdapter) MountOne(ctx context.Context, accountID string, req *types.EBSRequest) error { @@ -637,7 +686,7 @@ func (d *Daemon) buildVMManagerDeps() vm.Deps { OnInstanceUp: d.onInstanceUpHook(), OnInstanceDown: d.onInstanceDownHook(), OnInstanceRecovering: d.onInstanceRecoveringHook(), - BeforeInstanceRelaunch: d.refreshSystemInstanceState, + BeforeInstanceRelaunch: d.prepareInstanceRelaunch, }, ShutdownSignal: d.shuttingDown.Load, CrashHandler: d.vmMgr.HandleCrash, diff --git a/spinifex/daemon/vm_adapters_test.go b/spinifex/daemon/vm_adapters_test.go index aa6a52329..2278cc873 100644 --- a/spinifex/daemon/vm_adapters_test.go +++ b/spinifex/daemon/vm_adapters_test.go @@ -398,15 +398,15 @@ func TestRemoveFromSpotRequest_NoService_NoOp(t *testing.T) { // TestBuildVMManagerDeps_WiresBeforeInstanceRelaunch guards the single line // in buildVMManagerDeps that routes the recovery hook to -// refreshSystemInstanceState. Dropping it would surface only in cell-18. +// prepareInstanceRelaunch. Dropping it would surface only in cell-18. func TestBuildVMManagerDeps_WiresBeforeInstanceRelaunch(t *testing.T) { d := &Daemon{config: &config.Config{}, vmMgr: vm.NewManager()} deps := d.buildVMManagerDeps() require.NotNil(t, deps.Hooks.BeforeInstanceRelaunch) - wantPC := reflect.ValueOf(d.refreshSystemInstanceState).Pointer() + wantPC := reflect.ValueOf(d.prepareInstanceRelaunch).Pointer() gotPC := reflect.ValueOf(deps.Hooks.BeforeInstanceRelaunch).Pointer() - assert.Equal(t, wantPC, gotPC, "hook must point at refreshSystemInstanceState") + assert.Equal(t, wantPC, gotPC, "hook must point at prepareInstanceRelaunch") // Sanity: the wired hook is callable and returns nil for non-ELBv2 VMs. require.NoError(t, deps.Hooks.BeforeInstanceRelaunch(&vm.VM{ID: "i-noop", ManagedBy: ""})) @@ -414,6 +414,21 @@ func TestBuildVMManagerDeps_WiresBeforeInstanceRelaunch(t *testing.T) { "ELBv2 VM with nil elbv2Service must error rather than silently no-op") } +// The hook is what both relaunch paths pass through, so it is where an instance +// arriving from another node takes its management address over. Without that the +// reservation keeps naming the node it left, and the node running it could never +// give the address back. +func TestPrepareInstanceRelaunch_ClaimsTheManagementAddress(t *testing.T) { + alloc, err := NewMgmtIPAllocator("10.36.8.1") + require.NoError(t, err) + + d := &Daemon{config: &config.Config{}, vmMgr: vm.NewManager(), mgmtIPAllocator: alloc} + require.NoError(t, d.prepareInstanceRelaunch(&vm.VM{ID: "i-moved", MgmtIP: "10.36.8.42"})) + + assert.Equal(t, 1, alloc.AllocatedCount(), + "an instance that arrived here holds its address here, and the allocator has to know before the next launch") +} + // --- DetachAndDeleteENI: post-launch attach enumeration --- // TestInstanceCleanerAdapter_DetachAndDeleteENI_ReleasesPostLaunchAttach locks diff --git a/spinifex/handlers/ec2/instance/launch_claimed_relaunch_test.go b/spinifex/handlers/ec2/instance/launch_claimed_relaunch_test.go new file mode 100644 index 000000000..57f905994 --- /dev/null +++ b/spinifex/handlers/ec2/instance/launch_claimed_relaunch_test.go @@ -0,0 +1,81 @@ +// Builds an InstanceServiceImpl field by field and calls the claim path +// directly, because a cross-node relaunch is not reachable from the exported +// surface: it is driven by the recovery reconciler, not by a request. +// +//test:in-package — the service's dependencies and launchClaimedInstance itself are unexported. +package handlers_ec2_instance + +import ( + "testing" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/mulgadc/spinifex/spinifex/vm" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// claimedRelaunchService builds the service and manager a cross-node recovery +// launch runs through, with hook the pre-relaunch hook under test. +// +// The launch itself is expected to fail: the manager's resolver is unwired, so +// nothing spawns a qemu process. What matters is what happened before it. +func claimedRelaunchService(t *testing.T, hook func(*vm.VM) error) (*InstanceServiceImpl, *fakeResourceCapacityProvider) { + t.Helper() + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + + instanceType := &ec2.InstanceTypeInfo{InstanceType: aws.String("t3.micro")} + resources := &fakeResourceCapacityProvider{ + instanceTypes: map[string]*ec2.InstanceTypeInfo{"t3.micro": instanceType}, + } + + mgr := vm.NewManager() + mgr.SetDeps(vm.Deps{ + NodeID: "node-b", + VolumeMounter: raceVolumeMounter{}, + Hooks: vm.ManagerHooks{BeforeInstanceRelaunch: hook}, + }) + + return &InstanceServiceImpl{resourceMgr: resources, vmMgr: mgr}, resources +} + +// The record names on-host state by path, and this host is not the one that +// wrote it. A system instance's boot configuration is a set of files under the +// runtime directory, so a recovery that does not rebuild them hands QEMU paths +// that exist only on the node the instance is being recovered from. +// +// This is the whole reason a load balancer never came back from a host failure: +// the hook that rebuilds those files was reachable only from the same-node +// restart path. +func TestLaunchClaimedInstance_RebuildsThisNodesCopyOfTheBootState(t *testing.T) { + var prepared []string + svc, _ := claimedRelaunchService(t, func(v *vm.VM) error { + prepared = append(prepared, v.ID) + return nil + }) + + instance := &vm.VM{ID: "i-moved", AccountID: "acc", InstanceType: "t3.micro"} + _ = svc.launchClaimedInstance(t.Context(), instance, func() {}) + + assert.Equal(t, []string{"i-moved"}, prepared, + "a recovery onto another node has to rebuild what the record only names, or the guest boots with none of it") +} + +// A guest that cannot be given its boot state would come up unreachable, which +// is worse than not coming up: the records would say it recovered. So the +// refusal is a launch failure, with everything the attempt took put back. +func TestLaunchClaimedInstance_RefusesWhenTheBootStateCannotBeRebuilt(t *testing.T) { + svc, resources := claimedRelaunchService(t, func(*vm.VM) error { + return assert.AnError + }) + + undone := false + instance := &vm.VM{ID: "i-moved", AccountID: "acc", InstanceType: "t3.micro"} + err := svc.launchClaimedInstance(t.Context(), instance, func() { undone = true }) + + require.Error(t, err, "a guest that cannot be given its boot state must not be reported as recovered") + assert.True(t, undone, "the claim has to be given back, or no other node will try") + assert.Len(t, resources.deallocated, 1, "the capacity the attempt took has to be returned") + _, present := svc.vmMgr.Get("i-moved") + assert.False(t, present, "a refused launch must not leave the instance in this node's view") +} diff --git a/spinifex/handlers/ec2/instance/service_impl.go b/spinifex/handlers/ec2/instance/service_impl.go index a2f18d89f..7469b3f51 100644 --- a/spinifex/handlers/ec2/instance/service_impl.go +++ b/spinifex/handlers/ec2/instance/service_impl.go @@ -2593,19 +2593,35 @@ func (s *InstanceServiceImpl) StartStoppedInstance(ctx context.Context, input *S } // Reset node-local fields that are stale after cross-node migration. + if err := s.launchClaimedInstance(ctx, instance, func() { s.restoreClaimedStoppedInstance(ctx, instance) }); err != nil { + return nil, err + } + + slog.InfoContext(ctx, "Started stopped instance from shared KV", "instanceId", instance.ID) + return &StartStoppedInstanceOutput{Status: "running", InstanceID: instance.ID}, nil +} + +// launchClaimedInstance runs an instance this node has already won the right +// to run, and is the whole of the launch both claims share. +// +// undo hands the record back to whoever should hold it if the launch does not +// happen, which is the one thing the two callers disagree about: an explicit +// start returns it to the stopped store, a recovery returns it to the node it +// was taken from. +func (s *InstanceServiceImpl) launchClaimedInstance(ctx context.Context, instance *vm.VM, undo func()) error { instance.ResetNodeLocalState() instanceType, ok := s.resourceMgr.InstanceTypes()[instance.InstanceType] if !ok { - slog.ErrorContext(ctx, "StartStoppedInstance: instance type not available on this node", - "instanceId", input.InstanceID, "instanceType", instance.InstanceType) - s.restoreClaimedStoppedInstance(ctx, instance) - return nil, errors.New(awserrors.ErrorInsufficientInstanceCapacity) + slog.ErrorContext(ctx, "launchClaimedInstance: instance type not available on this node", + "instanceId", instance.ID, "instanceType", instance.InstanceType) + undo() + return errors.New(awserrors.ErrorInsufficientInstanceCapacity) } if err := s.resourceMgr.Allocate(instanceType); err != nil { - slog.ErrorContext(ctx, "StartStoppedInstance: failed to allocate resources", "instanceId", input.InstanceID, "err", err) - s.restoreClaimedStoppedInstance(ctx, instance) - return nil, errors.New(awserrors.ErrorInsufficientInstanceCapacity) + slog.ErrorContext(ctx, "launchClaimedInstance: failed to allocate resources", "instanceId", instance.ID, "err", err) + undo() + return errors.New(awserrors.ErrorInsufficientInstanceCapacity) } // Add to local map + clear the stop intent before launch. @@ -2621,40 +2637,40 @@ func (s *InstanceServiceImpl) StartStoppedInstance(ctx context.Context, input *S // field wired at a different moment. Starting a GPU type without a // claimer would hand the customer a running, GPU-less instance. if s.gpuClaimer == nil { - slog.ErrorContext(ctx, "StartStoppedInstance: GPU type admitted with no GPU claimer wired", - "instanceId", input.InstanceID, "instanceType", instance.InstanceType) + slog.ErrorContext(ctx, "launchClaimedInstance: GPU type admitted with no GPU claimer wired", + "instanceId", instance.ID, "instanceType", instance.InstanceType) s.resourceMgr.Deallocate(instanceType) s.vmMgr.Delete(instance.ID) - s.restoreClaimedStoppedInstance(ctx, instance) - return nil, errors.New(awserrors.ErrorInsufficientInstanceCapacity) + undo() + return errors.New(awserrors.ErrorInsufficientInstanceCapacity) } profileName := instancetypes.MIGProfileFromType(aws.StringValue(instanceType.InstanceType)) gpuCount := instancetypes.GPUCountForType(instance.InstanceType) if recorded := len(instance.GPUAttachments); recorded > 0 && recorded != gpuCount { - slog.WarnContext(ctx, "StartStoppedInstance: recorded GPU count differs from the instance type", - "instanceId", input.InstanceID, "instanceType", instance.InstanceType, + slog.WarnContext(ctx, "launchClaimedInstance: recorded GPU count differs from the instance type", + "instanceId", instance.ID, "instanceType", instance.InstanceType, "recorded", recorded, "reclaiming", gpuCount) } attachments, gpuErr := s.gpuClaimer.Claim(instance.ID, profileName, gpuCount) if gpuErr == nil && len(attachments) != gpuCount { - slog.ErrorContext(ctx, "StartStoppedInstance: GPU claim returned the wrong count", - "instanceId", input.InstanceID, "requested", gpuCount, "got", len(attachments)) + slog.ErrorContext(ctx, "launchClaimedInstance: GPU claim returned the wrong count", + "instanceId", instance.ID, "requested", gpuCount, "got", len(attachments)) if relErr := s.gpuClaimer.Release(instance.ID); relErr != nil { - slog.ErrorContext(ctx, "StartStoppedInstance: GPU release after short claim failed", - "instanceId", input.InstanceID, "err", relErr) + slog.ErrorContext(ctx, "launchClaimedInstance: GPU release after short claim failed", + "instanceId", instance.ID, "err", relErr) } gpuErr = errors.New("GPU claim returned fewer GPUs than the instance type requires") } if gpuErr != nil { - slog.ErrorContext(ctx, "StartStoppedInstance: GPU claim failed", "instanceId", input.InstanceID, "count", gpuCount, "err", gpuErr) + slog.ErrorContext(ctx, "launchClaimedInstance: GPU claim failed", "instanceId", instance.ID, "count", gpuCount, "err", gpuErr) s.resourceMgr.Deallocate(instanceType) s.vmMgr.Delete(instance.ID) - s.restoreClaimedStoppedInstance(ctx, instance) - return nil, errors.New(awserrors.ErrorInsufficientInstanceCapacity) + undo() + return errors.New(awserrors.ErrorInsufficientInstanceCapacity) } instance.GPUAttachments = attachments gpuClaimed = true - slog.InfoContext(ctx, "GPUs claimed for instance", "instanceId", input.InstanceID, + slog.InfoContext(ctx, "GPUs claimed for instance", "instanceId", instance.ID, "count", len(attachments), "attachments", attachments) } @@ -2662,44 +2678,75 @@ func (s *InstanceServiceImpl) StartStoppedInstance(ctx context.Context, input *S // booting a guest the customer cannot reach. addressAllocated, err := s.reassignAutoAssignedPublicIP(ctx, instance) if err != nil { - slog.ErrorContext(ctx, "StartStoppedInstance: public IP allocation failed — refusing the start", - "instanceId", input.InstanceID, "err", err) + slog.ErrorContext(ctx, "launchClaimedInstance: public IP allocation failed — refusing the start", + "instanceId", instance.ID, "err", err) if gpuClaimed { if relErr := s.gpuClaimer.Release(instance.ID); relErr != nil { - slog.ErrorContext(ctx, "StartStoppedInstance: GPU release failed after address allocation failure", - "instanceId", input.InstanceID, "err", relErr) + slog.ErrorContext(ctx, "launchClaimedInstance: GPU release failed after address allocation failure", + "instanceId", instance.ID, "err", relErr) } } s.resourceMgr.Deallocate(instanceType) s.vmMgr.Delete(instance.ID) - s.restoreClaimedStoppedInstance(ctx, instance) - return nil, errors.New(awserrors.ValidErrorCodeFromError(err)) + undo() + return errors.New(awserrors.ValidErrorCodeFromError(err)) } - if err := s.vmMgr.Run(ctx, instance); err != nil { - slog.ErrorContext(ctx, "StartStoppedInstance: vmMgr.Run failed", "instanceId", input.InstanceID, "err", err) + // The record names on-host state by path, and this host is not the one that + // wrote it. A system instance whose boot blobs are not rebuilt here is handed + // paths that exist only on the node it is being recovered from, so it either + // refuses to start or starts with no network configuration at all. + launchErr := s.vmMgr.PrepareRelaunch(instance) + if launchErr != nil { + slog.ErrorContext(ctx, "launchClaimedInstance: could not rebuild this node's copy of the instance's boot state", + "instanceId", instance.ID, "managedBy", instance.ManagedBy, "err", launchErr) + } else { + launchErr = s.vmMgr.Run(ctx, instance) + } + if err := launchErr; err != nil { + slog.ErrorContext(ctx, "launchClaimedInstance: relaunch failed", "instanceId", instance.ID, "err", err) if addressAllocated { s.rollbackStartPublicIP(ctx, instance) } if gpuClaimed { if relErr := s.gpuClaimer.Release(instance.ID); relErr != nil { - slog.ErrorContext(ctx, "StartStoppedInstance: GPU release failed after launch failure", - "instanceId", input.InstanceID, "err", relErr) + slog.ErrorContext(ctx, "launchClaimedInstance: GPU release failed after launch failure", + "instanceId", instance.ID, "err", relErr) } } s.resourceMgr.Deallocate(instanceType) s.vmMgr.Delete(instance.ID) - s.restoreClaimedStoppedInstance(ctx, instance) + undo() if refusal := volumeHeldElsewhereError(err); refusal != nil { - return nil, refusal + return refusal } - return nil, errors.New(awserrors.ErrorServerInternal) + return errors.New(awserrors.ErrorServerInternal) } s.vmMgr.LogGuestDeviceMap(instance) - slog.InfoContext(ctx, "Started stopped instance from shared KV", "instanceId", instance.ID) - return &StartStoppedInstanceOutput{Status: "running", InstanceID: instance.ID}, nil + return nil +} + +// RecoverInstance launches an instance whose record this node has already +// claimed from a node that stopped heartbeating. +// +// It makes no decision of its own: the claim decided who launches, and the +// volume leases decide whether the launch may proceed. A refusal because the +// old owner still holds a lease is ordinary and retryable — it means the +// source's own self-fence has not finished yet — so it is returned as-is for +// the caller to try again rather than turned into a failure. +func (s *InstanceServiceImpl) RecoverInstance(ctx context.Context, instance *vm.VM, undo func()) error { + if s.resourceMgr == nil || s.vmMgr == nil { + return errors.New(awserrors.ErrorServerInternal) + } + + // The record was written by a node that never got to stop the instance, so + // it still says running and a launch would read that as a terminate racing + // it. Pending is the same shape a drain-stop leaves behind, which restore + // already relaunches from. + instance.Status = vm.StatePending + return s.launchClaimedInstance(ctx, instance, undo) } // volumeExclusionMarkers are the refusals that mean another node is writing this @@ -2710,6 +2757,15 @@ var volumeExclusionMarkers = []string{ "volume is leased by another owner", } +// ErrVolumeHeldElsewhere reports a launch refused because another node still +// holds one of the instance's volume leases. +// +// Recovery treats it differently from every other failure — it clears on the +// lease's own clock rather than needing anything fixed — so it has to be +// recognisable with errors.Is rather than inferred from text that has already +// crossed NATS once. +var ErrVolumeHeldElsewhere = errors.New("a volume is leased by another node") + // volumeHeldElsewhereError converts a mount refused because another node holds // the volume into a client-facing error carrying that reason. Returns nil for // anything else. Without it the caller gets ServerInternal and is told to @@ -2722,7 +2778,8 @@ func volumeHeldElsewhereError(err error) error { msg := err.Error() for _, marker := range volumeExclusionMarkers { if strings.Contains(msg, marker) { - return awserrors.Errorf(awserrors.ErrorIncorrectState, "%s", msg) + return fmt.Errorf("%w: %w", ErrVolumeHeldElsewhere, + awserrors.Errorf(awserrors.ErrorIncorrectState, "%s", msg)) } } return nil @@ -3376,15 +3433,28 @@ func buildInstanceStatus(v *vm.VM, systemImpaired bool, az string) *ec2.Instance // SystemStatus reflects host/node health, independent of the VM process: a // running VM's host is reachable unless under memory pressure. A guest held // on a backend I/O error is an infrastructure fault rather than a guest one, - // which is the distinction this check exists to draw. + // which is the distinction this check exists to draw. So is a guest whose + // public address the cloud underneath is not delivering here: it answers on + // its private address and on nothing else, and reporting that as healthy is + // the one outcome worse than reporting it as down. systemStatus, systemReach := instanceStatusOK, instanceStatusPassed + addressDark := !v.Health.AddressUnreachableSince.IsZero() switch { case v.Status != vm.StateRunning: systemStatus, systemReach = instanceStatusNotApplicable, instanceStatusNotApplicable - case systemImpaired, v.Health.IOErrorResumes > 0: + case systemImpaired, v.Health.IOErrorResumes > 0, addressDark: systemStatus, systemReach = instanceStatusImpaired, instanceStatusFailed } + sysDetail := &ec2.InstanceStatusDetails{ + Name: aws.String(reachabilityDetailName), + Status: aws.String(systemReach), + } + if addressDark && v.Status == vm.StateRunning { + since := v.Health.AddressUnreachableSince + sysDetail.ImpairedSince = &since + } + return &ec2.InstanceStatus{ AvailabilityZone: aws.String(az), InstanceId: aws.String(v.ID), @@ -3394,11 +3464,8 @@ func buildInstanceStatus(v *vm.VM, systemImpaired bool, az string) *ec2.Instance Details: []*ec2.InstanceStatusDetails{instDetail}, }, SystemStatus: &ec2.InstanceStatusSummary{ - Status: aws.String(systemStatus), - Details: []*ec2.InstanceStatusDetails{{ - Name: aws.String(reachabilityDetailName), - Status: aws.String(systemReach), - }}, + Status: aws.String(systemStatus), + Details: []*ec2.InstanceStatusDetails{sysDetail}, }, } } diff --git a/spinifex/handlers/ec2/instance/service_impl_test.go b/spinifex/handlers/ec2/instance/service_impl_test.go index 1ec4309e6..94afb698e 100644 --- a/spinifex/handlers/ec2/instance/service_impl_test.go +++ b/spinifex/handlers/ec2/instance/service_impl_test.go @@ -2677,6 +2677,7 @@ type raceVolumeMounter struct{} func (raceVolumeMounter) Mount(context.Context, *vm.VM) error { return nil } func (raceVolumeMounter) Unmount(context.Context, *vm.VM) error { return nil } +func (raceVolumeMounter) Abandon(context.Context, *vm.VM, string) error { return nil } func (raceVolumeMounter) MountOne(context.Context, string, *spxtypes.EBSRequest) error { return nil } func (raceVolumeMounter) UnmountOne(context.Context, string, spxtypes.EBSRequest) error { return nil } @@ -4857,6 +4858,35 @@ func TestDescribeInstanceStatus_NoPressureSystemOK(t *testing.T) { assert.Equal(t, "ok", *out.InstanceStatuses[0].SystemStatus.Status) } +// A guest the cloud underneath is not delivering a public address to is running +// and answering on nothing. Reporting that as healthy is the one outcome worse +// than reporting it as down, and it is a system fault rather than a guest one: +// the instance passes every check it can be asked about itself. +func TestDescribeInstanceStatus_AnAddressTheCloudWillNotDeliverIsSystemImpaired(t *testing.T) { + owner := "111122223333" + v := runningVM("i-dark", owner) + started := time.Now().Add(-90 * time.Second) + v.Health.AddressUnreachableSince = started + v.Health.AddressUnreachableReason = "OCI does not deliver 10.200.0.7 to this VNIC" + svc := &InstanceServiceImpl{ + config: &config.Config{AZ: "az-a"}, + vmMgr: mgrWith(map[string]*vm.VM{v.ID: v}), + resourceMgr: &fakeResourceCapacityProvider{}, + } + + out, err := svc.DescribeInstanceStatus(context.Background(), &ec2.DescribeInstanceStatusInput{}, owner) + require.NoError(t, err) + require.Len(t, out.InstanceStatuses, 1) + + s := out.InstanceStatuses[0] + assert.Equal(t, "ok", *s.InstanceStatus.Status, "the guest is fine; the path to it is not") + assert.Equal(t, "impaired", *s.SystemStatus.Status) + assert.Equal(t, "failed", *s.SystemStatus.Details[0].Status) + require.NotNil(t, s.SystemStatus.Details[0].ImpairedSince, + "an operator needs to know how long it has been dark, not just that it is") + assert.WithinDuration(t, started, *s.SystemStatus.Details[0].ImpairedSince, time.Second) +} + // TestInstanceArchitecture pins the safe-extraction contract: malformed // InstanceTypeInfo returns "" rather than panicking, and the firmware probe // surfaces "" as a clear error on the launch path. diff --git a/spinifex/handlers/ec2/instance/start_public_ip_test.go b/spinifex/handlers/ec2/instance/start_public_ip_test.go index 0cd46c965..d0fa69697 100644 --- a/spinifex/handlers/ec2/instance/start_public_ip_test.go +++ b/spinifex/handlers/ec2/instance/start_public_ip_test.go @@ -31,6 +31,7 @@ func (refusingMounter) Unmount(context.Context, *vm.VM) error { return nil } func (refusingMounter) MountOne(context.Context, string, *types.EBSRequest) error { return nil } +func (refusingMounter) Abandon(context.Context, *vm.VM, string) error { return nil } func (refusingMounter) UnmountOne(context.Context, string, types.EBSRequest) error { return nil } // startFixture builds a stopped instance whose address went back to the pool, diff --git a/spinifex/handlers/ec2/instance/volume_held_elsewhere_test.go b/spinifex/handlers/ec2/instance/volume_held_elsewhere_test.go index bbddbf69e..74882175e 100644 --- a/spinifex/handlers/ec2/instance/volume_held_elsewhere_test.go +++ b/spinifex/handlers/ec2/instance/volume_held_elsewhere_test.go @@ -38,6 +38,12 @@ func TestVolumeHeldElsewhere_NamesTheHolder(t *testing.T) { assert.NotEqual(t, awserrors.ErrorServerInternal, code, "ServerInternal reads as a transient fault and invites a retry that cannot succeed") assert.Contains(t, got.Error(), "node-b", "the caller has to learn where the data is") + + assert.ErrorIs(t, got, ErrVolumeHeldElsewhere, + "recovery retries this on a shorter clock than anything else, so it has to be matchable") + resolved, _, ok := awserrors.ResolveErrorDetail(got) + require.True(t, ok, "the sentinel must not hide the code from the layer that returns it") + assert.Equal(t, awserrors.ErrorIncorrectState, resolved) }) } } diff --git a/spinifex/network/external/ocinet/affinity.go b/spinifex/network/external/ocinet/affinity.go index a235d3be0..e76c5b4e0 100644 --- a/spinifex/network/external/ocinet/affinity.go +++ b/spinifex/network/external/ocinet/affinity.go @@ -17,6 +17,22 @@ type ClaimResult struct { Claimed []string // Local is how many bindings this node already owned, moved or not. Local int + // Stalled is every address whose guest runs here and which OCI is not + // delivering here, after the pass has tried to move it and re-read the + // answer. Empty is the healthy result, and a non-empty entry names a guest + // that is running and unreachable on its public address. + Stalled []StalledAddress +} + +// StalledAddress is one address the pass could not put where its guest is. +// +// It carries the ENI rather than the instance because that is what this layer +// knows: a caller that wants to name the guest has the port-to-instance map and +// this package does not. +type StalledAddress struct { + PublicIP string + ENIID string + Reason string } // ClaimLocalAddresses makes OCI agree with where the guests actually are. @@ -48,6 +64,12 @@ type ClaimResult struct { // the same answer from a different local fact: it is a logical router's SNAT // source, so it belongs on the VPC's gateway chassis, and whether that is this // node is something the chassisredirect port binding says locally. +// +// What the pass could not do is reported rather than returned as one error. A +// denial or a throttle is per-address, the guests behind the others are just as +// dark, and the caller needs to know which guest to call impaired — so every +// address that did not arrive comes back in Stalled, verified against OCI's own +// answer rather than against its acknowledgement of the request. func (a *PoolAllocator) ClaimLocalAddresses(ctx context.Context) (ClaimResult, error) { var res ClaimResult if a.localPorts == nil && a.localGateway == nil { @@ -103,6 +125,9 @@ func (a *PoolAllocator) ClaimLocalAddresses(ctx context.Context) (ClaimResult, e return res, err } + // One address failing must not abandon the rest: a throttle or a denial is + // per-object, and the guests behind the other addresses are as unreachable + // as this one if the pass gives up here. for key := range mine { b := rec.Bindings[key] if _, ok := held[b.PrivateIPID]; ok { @@ -110,8 +135,9 @@ func (a *PoolAllocator) ClaimLocalAddresses(ctx context.Context) (ClaimResult, e } moved, err := a.client.MovePrivateIP(ctx, b.PrivateIPID, a.cfg.VNICID) if err != nil { - return res, fmt.Errorf("ocinet affinity: move %s (%s) to %s: %w", - b.PrivateAddr, b.PrivateIPID, a.cfg.VNICID, err) + res.Stalled = append(res.Stalled, StalledAddress{PublicIP: key, ENIID: b.ENIID, + Reason: fmt.Sprintf("move %s (%s) to %s: %v", b.PrivateAddr, b.PrivateIPID, a.cfg.VNICID, err)}) + continue } slog.WarnContext(ctx, "ocinet claimed an address whose datapath runs here", "pool", a.cfg.Pool.Name, "public_ip", key, "private_ip", b.PrivateAddr, @@ -123,9 +149,43 @@ func (a *PoolAllocator) ClaimLocalAddresses(ctx context.Context) (ClaimResult, e } res.Claimed = append(res.Claimed, key) } + + if err := a.verifyClaimed(ctx, rec, &res); err != nil { + return res, err + } return res, nil } +// verifyClaimed re-reads what OCI delivers here and checks that every address +// this pass moved actually arrived. +// +// The move's own response is OCI agreeing to the request, not OCI having done +// it, and the difference is the whole failure this guards: a guest that is +// running, reported healthy, and answering on nothing. One extra call, and only +// when something moved. +func (a *PoolAllocator) verifyClaimed(ctx context.Context, rec Record, res *ClaimResult) error { + if len(res.Claimed) == 0 { + return nil + } + held, err := a.heldPrivateIPIDs(ctx) + if err != nil { + return err + } + landed := res.Claimed[:0:0] + for _, key := range res.Claimed { + b := rec.Bindings[key] + if _, ok := held[b.PrivateIPID]; ok { + landed = append(landed, key) + continue + } + res.Stalled = append(res.Stalled, StalledAddress{PublicIP: key, ENIID: b.ENIID, + Reason: fmt.Sprintf("OCI accepted the move of %s (%s) and does not deliver it to %s", + b.PrivateAddr, b.PrivateIPID, a.cfg.VNICID)}) + } + res.Claimed = landed + return nil +} + // ownsBinding answers whether this node is where b's traffic is handled, and so // whose VNIC OCI has to deliver it to. // diff --git a/spinifex/network/external/ocinet/affinity_test.go b/spinifex/network/external/ocinet/affinity_test.go index 57e875bff..e70280daf 100644 --- a/spinifex/network/external/ocinet/affinity_test.go +++ b/spinifex/network/external/ocinet/affinity_test.go @@ -205,6 +205,63 @@ func TestClaimReportsAFailureToRecordTheNewOwner(t *testing.T) { assert.Contains(t, err.Error(), "record new VNIC") } +// acceptWithoutMoving is OCI agreeing to a reassignment and not performing it. +// It is the failure the verification exists for and the one nothing else can +// produce: every other error is reported by the call that failed. +type acceptWithoutMoving struct{ *oci.Fake } + +func (c acceptWithoutMoving) MovePrivateIP(ctx context.Context, privateIPID, _ string) (oci.PrivateIP, error) { + return c.GetPrivateIP(ctx, privateIPID) +} + +// A move's own response is OCI accepting the request. Trusting it would leave a +// guest running, reported healthy, and answering on nothing at all. +func TestClaimReportsAnAddressOCIAcceptedAndDidNotDeliver(t *testing.T) { + ctx := context.Background() + fake := oci.NewFake() + store := ocinet.NewMemStore() + a, err := ocinet.New(acceptWithoutMoving{fake}, store, ocinet.Config{ + Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI}, + VNICID: thisVNIC, + CompartmentID: "ocid1.compartment.oc1..comp1", + Schedule: []time.Duration{time.Millisecond}, + Budget: time.Second, + Sleep: func(context.Context, time.Duration) error { return nil }, + LocalPorts: func(context.Context) (map[string]struct{}, error) { + return map[string]struct{}{"port-eni-1": {}}, nil + }, + }) + require.NoError(t, err) + pubAddr, _ := seedForeignBinding(t, fake, store, "eni-1") + + res, err := a.ClaimLocalAddresses(ctx) + require.NoError(t, err, "a guest that cannot be reached is a per-address fault, not a failed pass") + assert.Empty(t, res.Claimed, "an address that did not arrive must not be reported as claimed") + require.Len(t, res.Stalled, 1) + assert.Equal(t, pubAddr, res.Stalled[0].PublicIP) + assert.Equal(t, "eni-1", res.Stalled[0].ENIID, + "the caller names the guest from the ENI, so an entry without one cannot be acted on") + assert.Contains(t, res.Stalled[0].Reason, "does not deliver") +} + +// A throttle or a denial is per-object. Abandoning the pass on the first one +// leaves every later guest dark for no reason of its own. +func TestOneAddressFailingDoesNotAbandonTheRest(t *testing.T) { + ctx := context.Background() + fake := oci.NewFake() + a, store := newAffinityAllocator(t, fake, "port-eni-1", "port-eni-2") + seedForeignBinding(t, fake, store, "eni-1") + seedForeignBinding(t, fake, store, "eni-2") + + fake.FailOp("MovePrivateIP", errors.New("429 too many requests")) + res, err := a.ClaimLocalAddresses(ctx) + require.NoError(t, err) + assert.Len(t, res.Claimed, 1, "the address that could move should have moved") + require.Len(t, res.Stalled, 1) + assert.Contains(t, res.Stalled[0].Reason, "429") + assert.NotEmpty(t, res.Stalled[0].ENIID) +} + // newGatewayAffinityAllocator builds an allocator with no guests at all and a // fixed answer to "am I the gateway chassis for this VPC". func newGatewayAffinityAllocator(t *testing.T, fake *oci.Fake, local bool, err error) (*ocinet.PoolAllocator, *ocinet.MemStore) { diff --git a/spinifex/services/viperblockd/provider_handlers.go b/spinifex/services/viperblockd/provider_handlers.go index c606e3548..5522804ba 100644 --- a/spinifex/services/viperblockd/provider_handlers.go +++ b/spinifex/services/viperblockd/provider_handlers.go @@ -68,7 +68,7 @@ func registerProviderSubjects(cfg *Config, nc *nats.Conn) error { // that reaches an engine open without one refuses, and refusing every // publish is a worse failure than not starting. if cfg.leases == nil { - leases, err := newVolumeLeases(context.Background(), nc, cfg.leaseOwner()) + leases, err := newVolumeLeases(context.Background(), nc, cfg.leaseOwner(), cfg.KVReplicas) if err != nil { return fmt.Errorf("volume leases: %w", err) } @@ -79,7 +79,7 @@ func registerProviderSubjects(cfg *Config, nc *nats.Conn) error { // Same reasoning as the lease store: a mount that cannot consult the dirty // marker cannot tell a stale cross-node start from a routine one. if cfg.dirty == nil { - dirty, err := newVolumeDirty(context.Background(), nc, cfg.leaseOwner()) + dirty, err := newVolumeDirty(context.Background(), nc, cfg.leaseOwner(), cfg.KVReplicas) if err != nil { return fmt.Errorf("volume dirty markers: %w", err) } @@ -1334,6 +1334,11 @@ func constructMountedVB(ctx context.Context, cfg *Config, volumeName string) (*v // and its absence is what made a routine cluster update destructive: the lease // is claimed before any state is read, so a node coming up alongside NATS fails // there first and never reaches the conditions these sentinels describe. +// +// errVolumeLeaseHeld deliberately does not. This classifier drives the restore +// path, where a lease held by another node means that node owns the guest now, +// and retrying would fight the owner rather than wait for one. The recovery +// reconciler wants the opposite and reads that refusal itself. func mountErrRetryable(err error) bool { return errors.Is(err, viperblock.ErrStateNotFound) || errors.Is(err, viperblock.ErrStateBackendUnavailable) || diff --git a/spinifex/services/viperblockd/provider_handlers_test.go b/spinifex/services/viperblockd/provider_handlers_test.go index 38f55106d..695193e33 100644 --- a/spinifex/services/viperblockd/provider_handlers_test.go +++ b/spinifex/services/viperblockd/provider_handlers_test.go @@ -56,6 +56,11 @@ func TestMountErrRetryable(t *testing.T) { true, }, {"plain error", errors.New("some other mount failure"), false}, + // Restoring this node's own guests, a lease another node holds means that + // node is running the guest. Retrying would relaunch against an owner, + // which is the outcome the lease exists to prevent. The recovery + // reconciler treats the same refusal as retryable because there it means + // the dead owner has not let go yet, and it classifies it itself. {"lease held by another node is not retryable", errVolumeLeaseHeld, false}, } for _, tt := range tests { diff --git a/spinifex/services/viperblockd/vbwire/vbwire.go b/spinifex/services/viperblockd/vbwire/vbwire.go index 1dc3e5e4d..7dac617fc 100644 --- a/spinifex/services/viperblockd/vbwire/vbwire.go +++ b/spinifex/services/viperblockd/vbwire/vbwire.go @@ -39,6 +39,41 @@ type VolumeFencedEvent struct { Reason string `json:"reason"` } +// VolumeAbandonSubject is where the local daemon asks for an export to be torn +// down because the instance using it belongs to another node now. +// +// It exists because an unmount is the wrong request and killing the guest is not +// enough. An unmount seals, and this node's copy is the stale one, so sealing +// would publish an older state over the winner's. Meanwhile the export holds the +// volume lease, and a lease this node keeps renewing is one the winner can never +// acquire — the guest would be stopped here and unable to start anywhere. +// +// Node-addressed, like mount and unmount: the export is local, and no other +// node's viperblockd has anything it could tear down. +func VolumeAbandonSubject(node string) string { + if node == "" { + return "ebs.abandon" + } + return "ebs." + node + ".abandon" +} + +// VolumeAbandonRequest asks for one volume's export to go away without sealing. +// Reason is logged, so a node that dropped an export can be told apart from one +// that was fenced. +type VolumeAbandonRequest struct { + Volume string `json:"volume"` + Reason string `json:"reason"` +} + +// VolumeAbandonResponse reports what happened. Abandoned is false with no error +// when the volume was not exported here, which is the ordinary case and not a +// failure: the request is idempotent by design, since the caller is reconciling +// rather than transacting. +type VolumeAbandonResponse struct { + Abandoned bool `json:"abandoned"` + Error string `json:"error,omitempty"` +} + // DirtyBucket names the volumes whose writes are not confirmed to have reached // the backend, so the node holding them is still known after that node is gone. // diff --git a/spinifex/services/viperblockd/viperblockd.go b/spinifex/services/viperblockd/viperblockd.go index 39fcee22d..1c347ef14 100644 --- a/spinifex/services/viperblockd/viperblockd.go +++ b/spinifex/services/viperblockd/viperblockd.go @@ -19,6 +19,7 @@ import ( "github.com/mulgadc/bluebottle/pkg/masterkey" "github.com/mulgadc/spinifex/spinifex/admin" "github.com/mulgadc/spinifex/spinifex/otelsetup" + "github.com/mulgadc/spinifex/spinifex/services/viperblockd/vbwire" "github.com/mulgadc/spinifex/spinifex/types" "github.com/mulgadc/spinifex/spinifex/utils" "github.com/mulgadc/viperblock/viperblock" @@ -141,6 +142,11 @@ type Config struct { // If empty, falls back to generic ebs.mount / ebs.unmount with queue group (single-node compat). NodeName string + // KVReplicas is the replica count for the volume lease and dirty buckets, + // which is the cluster's node count. 0 means one, for a single node and for + // tests. + KVReplicas int + // NBDTransport controls the transport type: "socket" (default) or "tcp" // Socket is faster for local connections, TCP required for remote/DPU scenarios NBDTransport types.NBDTransport @@ -203,6 +209,12 @@ type Config struct { // standing up a real S3 backend. constructVB func(ctx context.Context, volumeName string) (*viperblock.VB, int, error) + // fenceWatchEvery and processAlive drive the watch a failed fence keeps on + // its writer. Zero means the production interval and utils.ProcessAlive: a + // SIGKILL that does not take effect cannot be staged with a real process. + fenceWatchEvery time.Duration + processAlive func(pid int) bool + // leases excludes a second viperblock engine on a volume this node has // open. Nil means exclusion cannot be established, and every engine open // refuses rather than proceeding blind. @@ -708,7 +720,7 @@ func launchService(cfg *Config) (err error) { // Bound before recovery, which opens engines: without the store every // engine open refuses, and the daemon would come up unable to adopt the // exports that outlived it. - leases, err := newVolumeLeases(context.Background(), nc, cfg.leaseOwner()) + leases, err := newVolumeLeases(context.Background(), nc, cfg.leaseOwner(), cfg.KVReplicas) if err != nil { return fmt.Errorf("volume leases: %w", err) } @@ -846,6 +858,42 @@ func launchService(cfg *Config) (err error) { return fmt.Errorf("failed to subscribe to %s: %w", unmountTopic, err) } + // Never a queue group, even in single-node mode. Abandoning an export is an + // instruction to one specific node about its own volume, and a queue group + // would let it land on a node that has nothing to abandon while the one that + // does goes on holding the lease. + abandonTopic := vbwire.VolumeAbandonSubject(cfg.NodeName) + if _, err := nc.Subscribe(abandonTopic, func(msg *nats.Msg) { + ctx, span := utils.StartConsumerSpan(msg) + defer span.End() + + var req vbwire.VolumeAbandonRequest + if err := json.Unmarshal(msg.Data, &req); err != nil { + slog.ErrorContext(ctx, "failed to unmarshal volume abandon request", "err", err) + utils.MarkSpanError(span, err) + respondJSON(msg, vbwire.VolumeAbandonResponse{Error: fmt.Sprintf("bad request: %v", err)}) + return + } + // No path is built from this: the name is matched against exports this + // node recorded itself. An empty one would match nothing and report + // success, which is the wrong answer to a malformed request. + if req.Volume == "" { + slog.ErrorContext(ctx, "volume abandon: request names no volume") + respondJSON(msg, vbwire.VolumeAbandonResponse{Error: "request names no volume"}) + return + } + + abandoned, err := cfg.abandonVolume(ctx, req.Volume, req.Reason) + response := vbwire.VolumeAbandonResponse{Abandoned: abandoned} + if err != nil { + utils.MarkSpanError(span, err) + response.Error = err.Error() + } + respondJSON(msg, response) + }); err != nil { + return fmt.Errorf("failed to subscribe to %s: %w", abandonTopic, err) + } + if _, err := nc.QueueSubscribe("ebs.sync", "spinifex-workers", func(msg *nats.Msg) { ctx, span := utils.StartConsumerSpan(msg) defer span.End() diff --git a/spinifex/services/viperblockd/viperblockd_integration_test.go b/spinifex/services/viperblockd/viperblockd_integration_test.go index 01bbc85a2..ccd0dc951 100644 --- a/spinifex/services/viperblockd/viperblockd_integration_test.go +++ b/spinifex/services/viperblockd/viperblockd_integration_test.go @@ -54,7 +54,7 @@ func installTestVolumeLeases(t *testing.T, cfg *Config, natsURL string) { require.NoError(t, err) t.Cleanup(nc.Close) - leases, err := newVolumeLeases(t.Context(), nc, cfg.leaseOwner()) + leases, err := newVolumeLeases(t.Context(), nc, cfg.leaseOwner(), 1) require.NoError(t, err) cfg.leases = leases } diff --git a/spinifex/services/viperblockd/volume_abandon_test.go b/spinifex/services/viperblockd/volume_abandon_test.go new file mode 100644 index 000000000..672a66580 --- /dev/null +++ b/spinifex/services/viperblockd/volume_abandon_test.go @@ -0,0 +1,131 @@ +package viperblockd + +//test:in-package — abandonVolume, the lease store and MountedVolumes are all +//unexported, and the reason this differs from an unmount has no exported form. + +import ( + "context" + "testing" + + "github.com/mulgadc/spinifex/spinifex/utils" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// The returning node's half of the fence. Its lease is not lost — nothing is +// wrong with this node — so the export would be renewed indefinitely and the +// instance would be stopped here and unable to start anywhere. +func TestVolumeAbandon_GivesUpTheExportAndTheLease(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + const volumeName = "vol-abandoned" + + cfg := fencedConfig(t, natsURL, "node-a", volumeName) + pid := cfg.MountedVolumes[0].PID + + abandoned, err := cfg.abandonVolume(t.Context(), volumeName, "the instance moved") + require.NoError(t, err) + assert.True(t, abandoned) + + cfg.mu.Lock() + mounted := len(cfg.MountedVolumes) + cfg.mu.Unlock() + assert.Zero(t, mounted, "an export left up is a second writer, whoever decided to stop") + assert.False(t, utils.ProcessAlive(pid), "the export process has to be gone, not merely forgotten") +} + +// The difference from a fence, and the reason this path is worth having. A +// fenced node has lost its lease and may not delete the key, so the winner waits +// out the TTL. Here the node still holds it and hands it back, so the winner can +// open the volume on its next attempt. +func TestVolumeAbandon_ReleasesTheKeySoTheNewOwnerNeedNotWaitOutTheTTL(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + const volumeName = "vol-abandonreleases" + + cfg := fencedConfig(t, natsURL, "node-a", volumeName) + + _, err := cfg.abandonVolume(t.Context(), volumeName, "the instance moved") + require.NoError(t, err) + + owner, held := cfg.leases.currentOwner(t.Context(), volumeName) + assert.False(t, held, + "the lease was this node's to give back, and leaving it to expire makes the new owner wait for nothing") + assert.Empty(t, owner) +} + +// The one thing it must never do, for the fence's reason: this node's block map +// is behind the node that took the volume, and sealing publishes it over theirs. +func TestVolumeAbandon_DoesNotSeal(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + const volumeName = "vol-abandonnoseal" + + sealed := false + cfg := fencedConfig(t, natsURL, "node-a", volumeName) + cfg.sealVolume = func(context.Context, string) error { sealed = true; return nil } + + _, err := cfg.abandonVolume(t.Context(), volumeName, "the instance moved") + require.NoError(t, err) + + assert.False(t, sealed, "abandoning must never seal: this node's copy is behind the new owner's") +} + +// Reporting success on a kill that did not take would tell the caller a writer +// stopped that is still running, which is worse than refusing. +func TestVolumeAbandon_KillFailureIsReportedAndLeavesTheVolumeMounted(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + const volumeName = "vol-abandonkillfails" + + cfg := fencedConfig(t, natsURL, "node-a", volumeName) + cfg.MountedVolumes[0].PID = 0 // rejected by ForceKillProcess, so the kill cannot succeed + + abandoned, err := cfg.abandonVolume(t.Context(), volumeName, "the instance moved") + require.Error(t, err, "a writer that is still running must not be reported as stopped") + assert.False(t, abandoned) + + cfg.mu.Lock() + mounted := len(cfg.MountedVolumes) + cfg.mu.Unlock() + require.Equal(t, 1, mounted, "the entry has to stay so a retry re-attempts the kill") + + owner, held := cfg.leases.currentOwner(t.Context(), volumeName) + require.True(t, held, "the lease must not be handed back while this node is still exporting") + assert.Equal(t, "node-a", owner) +} + +// The ordinary case. The caller is reconciling a whole instance's volumes, and +// most of them were never exported here — so this is not a failure, and treating +// it as one would make every recovery look broken. +func TestVolumeAbandon_NotExportedHereIsNotAnError(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + cfg := &Config{ + NodeName: "node-a", + leases: newTestLeases(t, natsURL, "node-a"), + dirty: newTestDirty(t, natsURL, "node-a"), + } + + abandoned, err := cfg.abandonVolume(t.Context(), "vol-neverhere", "the instance moved") + require.NoError(t, err) + assert.False(t, abandoned, "nothing was torn down, and the caller is entitled to know that") +} + +// The marker names whose writes the backend may be missing. The new owner takes +// it over when it opens the volume, so clearing it here would erase the only +// note that writes were lost. +func TestVolumeAbandon_LeavesTheDirtyMarkerToTheNewOwner(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + const volumeName = "vol-abandonkeepsmarker" + + cfg := fencedConfig(t, natsURL, "node-a", volumeName) + + _, err := cfg.abandonVolume(t.Context(), volumeName, "the instance moved") + require.NoError(t, err) + + record, ok := cfg.dirty.holder(t.Context(), volumeName) + require.True(t, ok, "abandoning must leave the marker behind for the new owner to take over") + assert.Equal(t, "node-a", record.Owner) +} diff --git a/spinifex/services/viperblockd/volume_bucket_replicas_test.go b/spinifex/services/viperblockd/volume_bucket_replicas_test.go new file mode 100644 index 000000000..ec207fff7 --- /dev/null +++ b/spinifex/services/viperblockd/volume_bucket_replicas_test.go @@ -0,0 +1,62 @@ +package viperblockd + +//test:in-package — both bucket constructors are unexported, and the replica +//count they are given has no exported surface to read it back from. + +import ( + "testing" + + "github.com/nats-io/nats.go" + "github.com/nats-io/nats.go/jetstream" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// The lease bucket decides which node may write a volume, and the dirty bucket +// is read on every mount. Created on one replica, both live on one node, and a +// cluster that loses that node can neither fence nor mount anywhere — which is +// the failure the whole recovery path is built to survive. So the replica count +// has to reach NATS, and these assert that it does in both directions: a count +// the server cannot satisfy fails loudly rather than quietly creating one +// replica, and the count asked for is the count the bucket is created with. +func TestVolumeBuckets_AreCreatedWithTheReplicaCountTheyAreGiven(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + nc, err := nats.Connect(natsURL) + require.NoError(t, err) + t.Cleanup(nc.Close) + + leases, err := newVolumeLeases(t.Context(), nc, "node-a", 1) + require.NoError(t, err) + assert.Equal(t, 1, bucketReplicas(t, leases.kv)) + + dirty, err := newVolumeDirty(t.Context(), nc, "node-a", 1) + require.NoError(t, err) + assert.Equal(t, 1, bucketReplicas(t, dirty.kv)) +} + +// A single embedded server cannot hold three replicas, so asking for them is +// how a test proves the number is not being dropped on the way. +func TestVolumeBuckets_RefuseAReplicaCountTheServerCannotHold(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + nc, err := nats.Connect(natsURL) + require.NoError(t, err) + t.Cleanup(nc.Close) + + _, err = newVolumeLeases(t.Context(), nc, "node-a", 3) + require.Error(t, err, "a lease bucket that cannot be made quorate must fail the service, not serve unreplicated") + + _, err = newVolumeDirty(t.Context(), nc, "node-a", 3) + require.Error(t, err, "a dirty bucket that cannot be made quorate must fail the service, not serve unreplicated") +} + +func bucketReplicas(t *testing.T, kv jetstream.KeyValue) int { + t.Helper() + + status, err := kv.Status(t.Context()) + require.NoError(t, err) + bucket, ok := status.(*jetstream.KeyValueBucketStatus) + require.True(t, ok, "expected a JetStream-backed bucket") + return bucket.StreamInfo().Config.Replicas +} diff --git a/spinifex/services/viperblockd/volume_dirty.go b/spinifex/services/viperblockd/volume_dirty.go index e95f508f4..11745c2b4 100644 --- a/spinifex/services/viperblockd/volume_dirty.go +++ b/spinifex/services/viperblockd/volume_dirty.go @@ -26,16 +26,22 @@ type volumeDirty struct { } // newVolumeDirty binds the dirty bucket, creating it if this is the first node -// up. owner identifies this node in the entries it writes. -func newVolumeDirty(ctx context.Context, nc *nats.Conn, owner string) (*volumeDirty, error) { +// up. owner identifies this node in the entries it writes, and replicas is the +// cluster size: this is read on the mount path, so a single replica would make +// every mount depend on one node being up. +func newVolumeDirty(ctx context.Context, nc *nats.Conn, owner string, replicas int) (*volumeDirty, error) { js, err := jetstream.New(nc) if err != nil { return nil, fmt.Errorf("jetstream: %w", err) } + // Create-or-update rather than create: a bucket left behind by a build that + // made it single-replica is raised here, and a cluster cannot be asked to + // lose its markers to be repaired. kv, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{ Bucket: vbwire.DirtyBucket, Description: "volumes whose last seal failed, keyed by volume, naming the node holding the current copy", History: 1, + Replicas: max(replicas, 1), }) if err != nil { return nil, fmt.Errorf("volume dirty bucket: %w", err) diff --git a/spinifex/services/viperblockd/volume_dirty_test.go b/spinifex/services/viperblockd/volume_dirty_test.go index 546697e02..12fee61d7 100644 --- a/spinifex/services/viperblockd/volume_dirty_test.go +++ b/spinifex/services/viperblockd/volume_dirty_test.go @@ -20,7 +20,7 @@ func newTestDirty(t *testing.T, natsURL, owner string) *volumeDirty { require.NoError(t, err) t.Cleanup(nc.Close) - dirty, err := newVolumeDirty(t.Context(), nc, owner) + dirty, err := newVolumeDirty(t.Context(), nc, owner, 1) require.NoError(t, err) return dirty } diff --git a/spinifex/services/viperblockd/volume_fence.go b/spinifex/services/viperblockd/volume_fence.go index 2b6dcd53c..92441b4c5 100644 --- a/spinifex/services/viperblockd/volume_fence.go +++ b/spinifex/services/viperblockd/volume_fence.go @@ -76,17 +76,7 @@ func (cfg *Config) onVolumeLeaseLost(ctx context.Context, volumeName string, kin // would publish an older state over the winner's. For the same reason the dirty // marker is left alone — the winner has taken it over and it now names them. func (cfg *Config) fenceVolume(ctx context.Context, volumeName, winner string, outcome otelsetup.FenceOutcome) { - cfg.mu.Lock() - var matched MountedVolume - found := false - for _, volume := range cfg.MountedVolumes { - if volume.Name == volumeName { - matched, found = volume, true - break - } - } - cfg.mu.Unlock() - + matched, found := cfg.lookupMountedVolume(volumeName) if !found { // The export went away on its own between losing the lease and getting // here, which is the ordinary unmount racing this path. Nothing to do. @@ -98,9 +88,160 @@ func (cfg *Config) fenceVolume(ctx context.Context, volumeName, winner string, o "volume", volumeName, "previous_owner", cfg.leases.owner, "winner", winner, "pid", matched.PID) + if err := cfg.tearDownExport(ctx, matched, "fence"); err != nil { + slog.ErrorContext(ctx, "fence FAILED: nbdkit did not exit, so this node is still a writer on a volume it does not own", + "volume", volumeName, "pid", matched.PID, "winner", winner, "err", err) + otelsetup.RecordVolumeFence(ctx, otelsetup.FenceOutcomeKillFailed) + if !cfg.writerAlive(matched.PID) { + // The kill was refused rather than ignored, so nothing here says what + // happened to the writer. The export stays registered and the lease + // stays held, which is the only reading of "unknown" that is safe. + slog.ErrorContext(ctx, "fence FAILED with no writer to watch: the export stays registered and the lease held", + "volume", volumeName, "pid", matched.PID, "winner", winner) + return + } + cfg.awaitFencedWriterExit(ctx, volumeName, matched, winner, outcome) + return + } + + otelsetup.RecordVolumeFence(ctx, outcome) + cfg.publishVolumeFenced(ctx, volumeName, winner) +} + +// defaultFenceWatchInterval is how often a fence that did not complete re-checks +// and re-reports. One log line and one metric sample per interval, for a +// condition that must not be reported once and then look resolved. +const defaultFenceWatchInterval = 30 * time.Second + +// writerAlive reports whether an export's process is still running, honouring a +// test's injected probe if there is one. +func (cfg *Config) writerAlive(pid int) bool { + if cfg.processAlive != nil { + return cfg.processAlive(pid) + } + return utils.ProcessAlive(pid) +} + +// fenceWatchInterval is how often a failed fence re-checks its writer. +func (cfg *Config) fenceWatchInterval() time.Duration { + if cfg.fenceWatchEvery > 0 { + return cfg.fenceWatchEvery + } + return defaultFenceWatchInterval +} + +// awaitFencedWriterExit keeps a failed fence open until its writer is gone. It +// is entered only with that writer observed alive, because a PID this node +// cannot see says nothing about whether it exited. +// +// A SIGKILL that has not taken effect is a process in uninterruptible sleep with +// the signal already pending. It exits when its I/O completes, so there is +// nothing to re-send and nothing else to try — and because it has not exited, its +// PID cannot have been reused, which is what makes waiting safe where re-killing +// would not be. +// +// What there is to do is not stop watching. The teardown deliberately left the +// registry entry and the lease in place: releasing a lease while still a writer +// would hand the volume on with this node still on it. Both are finished here, +// once the exit is confirmed. Until then the failure is re-reported every +// interval, because a second writer that is only in one old log line is a second +// writer nobody knows about. +func (cfg *Config) awaitFencedWriterExit(ctx context.Context, volumeName string, + matched MountedVolume, winner string, outcome otelsetup.FenceOutcome) { + began := time.Now() + for { + select { + case <-ctx.Done(): + slog.ErrorContext(ctx, "fence never completed: stopping with a live writer on a volume this node does not own", + "volume", volumeName, "pid", matched.PID, "winner", winner, + "unfenced_ms", otelsetup.Millis(time.Since(began))) + return + case <-time.After(cfg.fenceWatchInterval()): + } + + if cfg.writerAlive(matched.PID) { + slog.ErrorContext(ctx, "fence still FAILED: this node is writing a volume it does not own", + "volume", volumeName, "pid", matched.PID, "winner", winner, + "unfenced_ms", otelsetup.Millis(time.Since(began))) + otelsetup.RecordVolumeFence(ctx, otelsetup.FenceOutcomeKillFailed) + continue + } + + cfg.finishTeardown(ctx, matched, "fence") + slog.WarnContext(ctx, "fence completed late: the writer finally exited", + "volume", volumeName, "pid", matched.PID, "winner", winner, + "unfenced_ms", otelsetup.Millis(time.Since(began))) + otelsetup.RecordVolumeFence(ctx, outcome) + cfg.publishVolumeFenced(ctx, volumeName, winner) + return + } +} + +// abandonVolume tears down an export because the instance that was using it +// belongs to another node now, and this node has just been told so. +// +// It is the returning node's half of the fence, and it is needed for a reason +// that is easy to miss: stopping the local guest is not enough. The export +// outlives it, and the export holds the volume lease, which this node goes on +// renewing because nothing is wrong with it. A lease held by a healthy node +// never expires, so the winner can never acquire, and the instance ends up +// stopped here and unable to start anywhere — recovery retries until its budget +// runs out and then gives up on a guest that was never unrecoverable. +// +// Like the fence it does not seal, for the same reason: this node's copy is the +// stale one, and sealing would publish an older state over the winner's. +// +// Unlike the fence, the lease is not lost here, so releasing it deletes the key +// rather than leaving it to expire — the winner gets the volume on the next +// attempt instead of waiting out the TTL. +func (cfg *Config) abandonVolume(ctx context.Context, volumeName, reason string) (bool, error) { + matched, found := cfg.lookupMountedVolume(volumeName) + if !found { + // The ordinary case: the caller is reconciling, and most volumes it asks + // about were never exported here or have already gone. + slog.InfoContext(ctx, "volume abandon: not exported here, nothing to tear down", + "volume", volumeName, "reason", reason) + return false, nil + } + + slog.WarnContext(ctx, "abandoning volume: the instance using it is owned by another node now", + "volume", volumeName, "owner", cfg.leaseOwner(), "pid", matched.PID, "reason", reason) + + if err := cfg.tearDownExport(ctx, matched, "abandon"); err != nil { + slog.ErrorContext(ctx, "volume abandon FAILED: nbdkit did not exit, so this node still holds the lease "+ + "and the volume cannot start anywhere else", + "volume", volumeName, "pid", matched.PID, "err", err) + return false, err + } + + slog.InfoContext(ctx, "volume abandoned", "volume", volumeName, "reason", reason) + return true, nil +} + +// lookupMountedVolume reports this node's export for volumeName, if it has one. +func (cfg *Config) lookupMountedVolume(volumeName string) (MountedVolume, bool) { + cfg.mu.Lock() + defer cfg.mu.Unlock() + for _, volume := range cfg.MountedVolumes { + if volume.Name == volumeName { + return volume, true + } + } + return MountedVolume{}, false +} + +// tearDownExport removes an export without sealing it: subscriptions, the +// metadata flushes, nbdkit, the registry entry, the socket and the lease. +// +// Shared by the fence and the abandon because the teardown is the same act +// either way — what differs is who decided and what it means. The kill is the +// step nothing may run ahead of: everything after it assumes the writer is gone, +// and reporting a teardown that has not killed nbdkit is the failure that lets a +// second writer keep going while the cluster believes it stopped. +func (cfg *Config) tearDownExport(ctx context.Context, matched MountedVolume, why string) error { if matched.ConfigSub != nil { if err := matched.ConfigSub.Unsubscribe(); err != nil { - slog.ErrorContext(ctx, "fence: unsubscribe config topic", "volume", volumeName, "err", err) + slog.ErrorContext(ctx, why+": unsubscribe config topic", "volume", matched.Name, "err", err) } } unsubscribeOwnerSubjects(matched.Name, matched.OwnerSubs) @@ -111,17 +252,18 @@ func (cfg *Config) fenceVolume(ctx context.Context, volumeName, winner string, o matched.VB.Detach() } - // Everything below assumes the writer is gone, so nothing below may run - // until it is observed gone. A fence that has not killed nbdkit has fenced - // nothing, and reporting it as done is the failure that lets a second - // writer keep going while the cluster believes it stopped. if err := utils.ForceKillProcess(matched.PID, fenceKillTimeout); err != nil { - slog.ErrorContext(ctx, "fence FAILED: nbdkit did not exit, so this node is still a writer on a volume it does not own", - "volume", volumeName, "pid", matched.PID, "winner", winner, "err", err) - otelsetup.RecordVolumeFence(ctx, otelsetup.FenceOutcomeKillFailed) - return + return err } + cfg.finishTeardown(ctx, matched, why) + return nil +} + +// finishTeardown is everything after the writer is confirmed gone. Split out +// because a fence whose kill did not take effect has to run it later rather than +// not at all, and because every line of it assumes the process has exited. +func (cfg *Config) finishTeardown(ctx context.Context, matched MountedVolume, why string) { cfg.mu.Lock() for i, volume := range cfg.MountedVolumes { if volume.Name == matched.Name { @@ -133,16 +275,14 @@ func (cfg *Config) fenceVolume(ctx context.Context, volumeName, winner string, o if matched.Socket != "" { if err := os.Remove(matched.Socket); err != nil && !os.IsNotExist(err) { - slog.ErrorContext(ctx, "fence: could not remove nbd socket", "socket", matched.Socket, "err", err) + slog.ErrorContext(ctx, why+": could not remove nbd socket", "socket", matched.Socket, "err", err) } } - // Stops the renewal goroutine and drops the local entry. The lease was - // already flagged lost, so this cannot delete the winner's key. + // Stops the renewal goroutine and drops the local entry. It deletes the key + // only when this node still holds it, so a lost lease cannot evict the + // winner and a held one does not make the volume wait out its TTL. cfg.releaseVolumeLease(ctx, matched.Lease) - - otelsetup.RecordVolumeFence(ctx, outcome) - cfg.publishVolumeFenced(ctx, volumeName, winner) } // publishVolumeFenced tells the local daemon a guest is now running against a diff --git a/spinifex/services/viperblockd/volume_fence_test.go b/spinifex/services/viperblockd/volume_fence_test.go index 869c557dc..05413da03 100644 --- a/spinifex/services/viperblockd/volume_fence_test.go +++ b/spinifex/services/viperblockd/volume_fence_test.go @@ -6,7 +6,9 @@ package viperblockd import ( "context" "os/exec" + "sync/atomic" "testing" + "time" "github.com/mulgadc/spinifex/spinifex/utils" "github.com/stretchr/testify/assert" @@ -113,16 +115,79 @@ func TestVolumeFence_KillFailureLeavesTheVolumeMounted(t *testing.T) { cfg := fencedConfig(t, natsURL, "node-a", volumeName) cfg.MountedVolumes[0].PID = 0 // rejected by ForceKillProcess, so the kill cannot succeed + cfg.processAlive = func(int) bool { return true } + cfg.fenceWatchEvery = 10 * time.Millisecond + + // The watch outlives the fence by design, so it is cancelled and joined here + // rather than left to the test's own cleanup. + ctx, cancel := context.WithCancel(t.Context()) + watched := make(chan struct{}) + go func() { defer close(watched); cfg.fenceVolume(ctx, volumeName, "node-b", "taken") }() + defer func() { cancel(); <-watched }() + + assert.Never(t, func() bool { + cfg.mu.Lock() + defer cfg.mu.Unlock() + return len(cfg.MountedVolumes) == 0 + }, 20*cfg.fenceWatchEvery, cfg.fenceWatchEvery, + "a fence that could not stop the writer must not report the volume released") + + owner, held := cfg.leases.currentOwner(t.Context(), volumeName) + require.True(t, held, "the lease must not be released while this node is still exporting") + assert.Equal(t, "node-a", owner) +} + +// TestVolumeFence_CompletesOnceTheWriterFinallyExits is the other half of a +// failed kill. The SIGKILL is pending on a process in uninterruptible sleep, so +// the writer does stop — and the teardown has to run then rather than never, or +// the lease is held by a node that is no longer writing and the volume cannot +// start anywhere. +func TestVolumeFence_CompletesOnceTheWriterFinallyExits(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + const volumeName = "vol-fencekilllate" + + cfg := fencedConfig(t, natsURL, "node-a", volumeName) + cfg.MountedVolumes[0].PID = 0 // rejected by ForceKillProcess, so the kill cannot succeed + // Alive for the guard and one watch pass, gone on the next: the fence has to + // see the failure persist and then resolve. + var probes atomic.Int32 + cfg.processAlive = func(int) bool { return probes.Add(1) <= 2 } + cfg.fenceWatchEvery = 10 * time.Millisecond cfg.fenceVolume(t.Context(), volumeName, "node-b", "taken") cfg.mu.Lock() mounted := len(cfg.MountedVolumes) cfg.mu.Unlock() - require.Equal(t, 1, mounted, "a fence that could not stop the writer must not report the volume released") + assert.Zero(t, mounted, "an export whose writer has exited must not stay registered") + + _, held := cfg.leases.currentOwner(t.Context(), volumeName) + assert.False(t, held, "a fence that completed late must still release the lease, or the volume starts nowhere") +} + +// TestVolumeFence_DoesNotCompleteForAWriterItCannotSee covers a kill refused +// rather than ignored. Nothing then says what became of the writer, so treating +// the PID's absence as an exit would release the lease on a guess. +func TestVolumeFence_DoesNotCompleteForAWriterItCannotSee(t *testing.T) { + _, natsURL := setupEmbeddedNATS(t) + + const volumeName = "vol-fencekillrefused" + + cfg := fencedConfig(t, natsURL, "node-a", volumeName) + cfg.MountedVolumes[0].PID = 0 // rejected by ForceKillProcess, so the kill cannot succeed + cfg.processAlive = func(int) bool { return false } + cfg.fenceWatchEvery = 10 * time.Millisecond + + cfg.fenceVolume(t.Context(), volumeName, "node-b", "taken") + + cfg.mu.Lock() + mounted := len(cfg.MountedVolumes) + cfg.mu.Unlock() + assert.Equal(t, 1, mounted, "an unproven writer must leave the export registered") owner, held := cfg.leases.currentOwner(t.Context(), volumeName) - require.True(t, held, "the lease must not be released while this node is still exporting") + require.True(t, held, "the lease must not move on the strength of a PID this node never saw") assert.Equal(t, "node-a", owner) } diff --git a/spinifex/services/viperblockd/volume_lease.go b/spinifex/services/viperblockd/volume_lease.go index 670cad268..175bbeed9 100644 --- a/spinifex/services/viperblockd/volume_lease.go +++ b/spinifex/services/viperblockd/volume_lease.go @@ -128,17 +128,23 @@ type volumeLeases struct { } // newVolumeLeases binds the lease bucket, creating it if this is the first -// node up. owner identifies this node in the entries it writes. -func newVolumeLeases(ctx context.Context, nc *nats.Conn, owner string) (*volumeLeases, error) { +// node up. owner identifies this node in the entries it writes, and replicas is +// the cluster size: this bucket decides who may write a volume, so a single +// replica would put every volume in the cluster behind one node staying up. +func newVolumeLeases(ctx context.Context, nc *nats.Conn, owner string, replicas int) (*volumeLeases, error) { js, err := jetstream.New(nc) if err != nil { return nil, fmt.Errorf("jetstream: %w", err) } + // Create-or-update rather than create: a bucket left behind by a build that + // made it single-replica is raised here, and a cluster cannot be asked to + // lose its leases to be repaired. kv, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{ Bucket: volumeLeaseBucket, Description: "one entry per volume with a viperblock engine open on it", TTL: volumeLeaseTTL, History: 1, + Replicas: max(replicas, 1), }) if err != nil { return nil, fmt.Errorf("volume lease bucket: %w", err) diff --git a/spinifex/services/viperblockd/volume_lease_test.go b/spinifex/services/viperblockd/volume_lease_test.go index 62dc02b8c..4aee0012b 100644 --- a/spinifex/services/viperblockd/volume_lease_test.go +++ b/spinifex/services/viperblockd/volume_lease_test.go @@ -24,7 +24,7 @@ func newTestLeases(t *testing.T, natsURL, owner string) *volumeLeases { require.NoError(t, err) t.Cleanup(nc.Close) - leases, err := newVolumeLeases(t.Context(), nc, owner) + leases, err := newVolumeLeases(t.Context(), nc, owner, 1) require.NoError(t, err) return leases } @@ -561,7 +561,7 @@ func TestVolumeLease_UnreachableStoreIsRetryable(t *testing.T) { require.NoError(t, err) t.Cleanup(nc.Close) - leases, err := newVolumeLeases(t.Context(), nc, "node-a") + leases, err := newVolumeLeases(t.Context(), nc, "node-a", 1) require.NoError(t, err) // Close the connection under the bound store, which is the shape of a diff --git a/spinifex/vm/address_health.go b/spinifex/vm/address_health.go new file mode 100644 index 000000000..5e13f92f7 --- /dev/null +++ b/spinifex/vm/address_health.go @@ -0,0 +1,54 @@ +package vm + +import "time" + +// SetUnreachableAddresses replaces this node's record of which guests the cloud +// underneath is not delivering a public address to, keyed by ENI. +// +// It takes the whole set rather than one guest at a time, because that is what +// clears a guest whose address has since arrived: a marker that only ever gets +// set would outlive the fault and report a healthy instance as impaired forever. +// Callers that learned nothing this pass must not call it at all — an empty map +// means "every guest is reachable", not "no information". +// +// Returns the IDs of the instances now marked, so a caller can say which guests +// are running and answering on nothing. +func (m *Manager) SetUnreachableAddresses(byENI map[string]string) []string { + var marked []string + now := time.Now() + + m.mu.Lock() + defer m.mu.Unlock() + for _, v := range m.vms { + reason, unreachable := unreachableReason(v, byENI) + if !unreachable { + v.Health.AddressUnreachableSince = time.Time{} + v.Health.AddressUnreachableReason = "" + continue + } + // The timestamp is when the fault started, not when it was last + // observed, so a pass that re-confirms it must not push it forward. + if v.Health.AddressUnreachableSince.IsZero() { + v.Health.AddressUnreachableSince = now + } + v.Health.AddressUnreachableReason = reason + marked = append(marked, v.ID) + } + return marked +} + +// unreachableReason reports whether any of v's ENIs is in byENI, and why. +func unreachableReason(v *VM, byENI map[string]string) (string, bool) { + if len(byENI) == 0 { + return "", false + } + if reason, ok := byENI[v.ENIId]; ok { + return reason, true + } + for _, eni := range v.ExtraENIs { + if reason, ok := byENI[eni.ENIID]; ok { + return reason, true + } + } + return "", false +} diff --git a/spinifex/vm/address_health_test.go b/spinifex/vm/address_health_test.go new file mode 100644 index 000000000..41a799692 --- /dev/null +++ b/spinifex/vm/address_health_test.go @@ -0,0 +1,86 @@ +package vm_test + +import ( + "testing" + "time" + + "github.com/mulgadc/spinifex/spinifex/vm" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// managerWithGuests returns a manager holding one VM per argument, each with a +// primary ENI named after it. +func managerWithGuests(t *testing.T, ids ...string) *vm.Manager { + t.Helper() + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + m := vm.NewManager() + for _, id := range ids { + m.Insert(&vm.VM{ID: id, ENIId: "eni-" + id, Status: vm.StateRunning}) + } + return m +} + +// The guest named is the guest marked, and no other. A pass that marked more +// than it found would report healthy instances as unreachable, which is the +// mirror of the fault it exists to catch. +func TestOnlyTheGuestWhoseAddressIsDarkIsMarked(t *testing.T) { + m := managerWithGuests(t, "i-1", "i-2") + + marked := m.SetUnreachableAddresses(map[string]string{"eni-i-1": "429 too many requests"}) + assert.Equal(t, []string{"i-1"}, marked) + + dark, ok := m.Get("i-1") + require.True(t, ok) + assert.False(t, dark.Health.AddressUnreachableSince.IsZero()) + assert.Equal(t, "429 too many requests", dark.Health.AddressUnreachableReason) + + fine, ok := m.Get("i-2") + require.True(t, ok) + assert.True(t, fine.Health.AddressUnreachableSince.IsZero()) +} + +// A load balancer carries more than one ENI, and its public address hangs off +// one of the extras. Reading only the primary would report it healthy. +func TestAnAddressOnASecondENICountsAsTheGuestsOwn(t *testing.T) { + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + m := vm.NewManager() + m.Insert(&vm.VM{ID: "i-alb", ENIId: "eni-primary", Status: vm.StateRunning, + ExtraENIs: []vm.ExtraENI{{ENIID: "eni-public"}}}) + + marked := m.SetUnreachableAddresses(map[string]string{"eni-public": "denied"}) + assert.Equal(t, []string{"i-alb"}, marked) +} + +// The marker has to clear on its own evidence. One that only ever gets set +// would leave a recovered guest reported impaired for the rest of its life. +func TestAnAddressThatArrivesClearsTheMarker(t *testing.T) { + m := managerWithGuests(t, "i-1") + + require.Len(t, m.SetUnreachableAddresses(map[string]string{"eni-i-1": "429"}), 1) + assert.Empty(t, m.SetUnreachableAddresses(nil)) + + v, ok := m.Get("i-1") + require.True(t, ok) + assert.True(t, v.Health.AddressUnreachableSince.IsZero()) + assert.Empty(t, v.Health.AddressUnreachableReason) +} + +// ImpairedSince is when the fault started. Re-confirming it every fifteen +// seconds must not keep resetting the clock, or the field reports the age of the +// last observation and a long outage looks brand new. +func TestReconfirmingAFaultDoesNotResetWhenItStarted(t *testing.T) { + m := managerWithGuests(t, "i-1") + + m.SetUnreachableAddresses(map[string]string{"eni-i-1": "429"}) + v, ok := m.Get("i-1") + require.True(t, ok) + first := v.Health.AddressUnreachableSince + require.False(t, first.IsZero()) + + time.Sleep(time.Millisecond) + m.SetUnreachableAddresses(map[string]string{"eni-i-1": "still 429"}) + assert.Equal(t, first, v.Health.AddressUnreachableSince) + assert.Equal(t, "still 429", v.Health.AddressUnreachableReason, + "the reason is the current one even though the timestamp is the original") +} diff --git a/spinifex/vm/errors.go b/spinifex/vm/errors.go index c5077310e..bc445f41c 100644 --- a/spinifex/vm/errors.go +++ b/spinifex/vm/errors.go @@ -58,3 +58,9 @@ var ErrQMPUnavailable = errors.New("QMP unavailable for instance") // lost the race and must not proceed to allocate resources or launch qemu // for that instance. var ErrStoppedInstanceClaimed = errors.New("stopped instance already claimed") + +// ErrRecoveryClaimLost is returned when the CAS that takes an instance off a +// node that stopped heartbeating did not land: another survivor won it, or the +// record changed underneath the attempt. Losing is ordinary, so the caller +// moves on rather than retrying. +var ErrRecoveryClaimLost = errors.New("instance recovery claim lost") diff --git a/spinifex/vm/forget_superseded_test.go b/spinifex/vm/forget_superseded_test.go new file mode 100644 index 000000000..bc0840ae9 --- /dev/null +++ b/spinifex/vm/forget_superseded_test.go @@ -0,0 +1,127 @@ +//test:in-package — ForgetSuperseded is asserted against Manager's own map and +//the in-package fake deps, which is where the rest of the shutdown suite lives. + +package vm + +import ( + "testing" + + "github.com/mulgadc/spinifex/spinifex/types" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// A returning node has to drop the instances the cluster moved while it was +// away, before its restore relaunches them from its own state file. +func TestForgetSuperseded_RemovesTheLocalCopy(t *testing.T) { + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + m := NewManager() + m.SetDeps(Deps{NodeID: "node-1", StateStore: newFakeStateStore()}) + + instance := &VM{ID: "i-moved", InstanceType: "t3.micro"} + m.Insert(instance) + _, present := m.Get("i-moved") + require.True(t, present) + + m.ForgetSuperseded(instance) + + _, present = m.Get("i-moved") + assert.False(t, present, + "an instance another node owns now must not survive in this node's view") +} + +// Called on whatever the caller found, and what it found may be nothing. +func TestForgetSuperseded_NilIsANoOp(t *testing.T) { + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + m := NewManager() + + assert.NotPanics(t, func() { m.ForgetSuperseded(nil) }) +} + +// Dropping the guest is the easy half. The export outlives it and holds the +// volume lease, and this node is healthy, so it would renew that lease forever — +// leaving the instance stopped here and unable to start anywhere. +func TestForgetSuperseded_GivesUpTheVolumesWithoutSealingThem(t *testing.T) { + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + mounter := &fakeVolumeMounter{} + m := NewManager() + m.SetDeps(Deps{NodeID: "node-1", StateStore: newFakeStateStore(), VolumeMounter: mounter}) + + instance := &VM{ID: "i-moved", InstanceType: "t3.micro"} + instance.EBSRequests.Requests = []types.EBSRequest{{Name: "vol-root"}} + m.Insert(instance) + + m.ForgetSuperseded(instance) + + assert.Equal(t, []string{"i-moved"}, mounter.abandoned, + "the export has to be given up, or the new owner waits on a lease this node keeps renewing") + assert.Empty(t, mounter.unmounted, + "an unmount seals, and this node's block map is the stale one: sealing publishes it over the winner's") +} + +// An export this node could not let go of is one the new owner cannot open, so +// the failure is reported rather than swallowed — but the instance still leaves, +// because it is running somewhere else either way. +func TestForgetSuperseded_DropsTheInstanceEvenIfTheVolumesRefuse(t *testing.T) { + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + mounter := &fakeVolumeMounter{abandonErr: assert.AnError} + m := NewManager() + m.SetDeps(Deps{NodeID: "node-1", StateStore: newFakeStateStore(), VolumeMounter: mounter}) + + instance := &VM{ID: "i-moved", InstanceType: "t3.micro"} + instance.EBSRequests.Requests = []types.EBSRequest{{Name: "vol-root"}} + m.Insert(instance) + + m.ForgetSuperseded(instance) + + assert.Equal(t, []string{"i-moved"}, mounter.abandoned) + _, present := m.Get("i-moved") + assert.False(t, present, + "a volume that would not let go does not make the instance this node's again") +} + +// The logical port is bound to whichever chassis offers a tap carrying its +// iface-id, and the binding names one chassis. Two nodes offering the same +// iface-id contend for it, so the address answers from whichever won last — +// which is the one failure a load balancer's customers would actually see. +func TestForgetSuperseded_StopsClaimingTheInstanceLogicalPort(t *testing.T) { + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + plumber := &fakeNetworkPlumber{} + cleaner := &recordingInstanceCleaner{} + m := NewManager() + m.SetDeps(Deps{ + NodeID: "node-1", StateStore: newFakeStateStore(), + NetworkPlumber: plumber, InstanceCleaner: cleaner, + }) + + instance := &VM{ID: "i-moved", InstanceType: "t3.micro", ENIId: "eni-abc"} + m.Insert(instance) + + m.ForgetSuperseded(instance) + + assert.Contains(t, plumber.cleanupCalls, TapDeviceName("eni-abc"), + "a tap left behind keeps this node contending for a logical port another node now serves") + assert.Equal(t, []string{"eni-abc"}, plumber.imdsDetachCalls, + "the IMDS patch port claims the same iface-id, so it has to go with the tap") + assert.Equal(t, []string{"i-moved"}, cleaner.cleanupMgmt, + "the mgmt tap is this node's own and nothing else will remove it") +} + +// The instance is not being terminated, only forgotten here. Anything that acts +// on the cluster's record of it would be this node deciding for the node that +// runs it now. +func TestForgetSuperseded_TouchesNothingTheNewOwnerOwns(t *testing.T) { + t.Setenv("XDG_RUNTIME_DIR", t.TempDir()) + cleaner := &recordingInstanceCleaner{} + m := NewManager() + m.SetDeps(Deps{NodeID: "node-1", StateStore: newFakeStateStore(), InstanceCleaner: cleaner}) + + instance := &VM{ID: "i-moved", InstanceType: "t3.micro", ENIId: "eni-abc", PublicIP: "203.0.113.5"} + m.Insert(instance) + + m.ForgetSuperseded(instance) + + assert.Empty(t, cleaner.deleteVolumes, "the volumes belong to the instance, and the instance moved") + assert.Empty(t, cleaner.releasePublicIP, "the address follows the instance; releasing it here takes it away") + assert.Empty(t, cleaner.detachAndDeleteENI, "the ENI is what the new owner plugs its own tap into") +} diff --git a/spinifex/vm/lifecycle.go b/spinifex/vm/lifecycle.go index 76c79d9ec..b9150943e 100644 --- a/spinifex/vm/lifecycle.go +++ b/spinifex/vm/lifecycle.go @@ -849,24 +849,60 @@ const ( // exceeds the plain-VM default even for a small guest (a 16GB EKS GPU worker // takes >30s on wattle). Small guests land here. qmpVFIOGreetingFloor = 180 * time.Second - // qmpVFIOGreetingCap bounds the scaled wait so a mis-sized guest cannot - // wedge a launch indefinitely. - qmpVFIOGreetingCap = 600 * time.Second + // qmpGreetingCap bounds the whole wait so a mis-sized guest cannot wedge a + // launch indefinitely. It has to clear the sum of the two floors with room + // to spare, or it would clip a guest that has both costs before either had + // been waited out. + qmpGreetingCap = 900 * time.Second + + // qmpNBDGreetingTimeout is the deadline for a guest whose drives are + // network-backed. QEMU opens every drive before its monitor answers, and a + // cold volume open reads its state out of the object store — which is + // slowest precisely when a node has just been lost and the store is serving + // degraded reads. Measured at 35s with one node of three away, against a 30s + // deadline that killed the guest five seconds short of ready and then did it + // again on every retry. + qmpNBDGreetingTimeout = 180 * time.Second ) -// qmpGreetingTimeout picks the QMP greeting deadline for a VM: the plain default -// unless the guest has GPU/VFIO passthrough, whose synchronous RAM pin delays the -// monitor. For VFIO the deadline is base + perGiB*RAM, floored so small guests -// keep the proven deadline and capped so a huge guest cannot wedge a launch. +// qmpGreetingTimeout is how long QEMU may take to answer its monitor. Both +// costs it allows for exist for the same reason: QEMU does work synchronously +// before the monitor answers, and a deadline shorter than that work SIGKILLs a +// guest that would have come up. +// +// They are added, not chosen between, because a GPU guest booting from a network +// drive pays both — QEMU opens every drive over the object store and pins the +// whole of guest RAM, one after the other. Taking whichever is larger would give +// such a guest a deadline that cannot cover what it has to do, and the symptom +// would be a GPU instance that is killed on every attempt while its plain +// sibling launches, which reads as a GPU fault rather than a deadline. +// +// The VFIO term scales with RAM because the pin is synchronous and grows with +// it; the drive term does not, because opening a volume's state out of the store +// costs the same whatever the guest is. func qmpGreetingTimeout(v *VM) time.Duration { - if len(v.GPUAttachments) == 0 { - return qmp.DefaultGreetingTimeout + deadline := qmp.DefaultGreetingTimeout + if hasNetworkBackedDrive(v) { + deadline = qmpNBDGreetingTimeout } - memGiB := v.Config.Memory / 1024 - scaled := qmpVFIOGreetingBase + time.Duration(memGiB)*qmpVFIOGreetingPerGiB - scaled = max(scaled, qmpVFIOGreetingFloor) - scaled = min(scaled, qmpVFIOGreetingCap) - return scaled + if len(v.GPUAttachments) > 0 { + memGiB := v.Config.Memory / 1024 + deadline += max(qmpVFIOGreetingBase+time.Duration(memGiB)*qmpVFIOGreetingPerGiB, qmpVFIOGreetingFloor) + } + return min(deadline, qmpGreetingCap) +} + +// hasNetworkBackedDrive reports whether any of the guest's drives is served +// over NBD, and so has to be opened across the object store before QEMU's +// monitor answers. Every production guest boots from one; a VM without is a +// unit test. +func hasNetworkBackedDrive(v *VM) bool { + for _, req := range v.EBSRequests.Requests { + if req.NBDURI != "" { + return true + } + } + return false } // removeStaleQMPSocket unlinks a leftover QMP socket inode from a prior QEMU so diff --git a/spinifex/vm/lifecycle_test.go b/spinifex/vm/lifecycle_test.go index a4eea9586..2de5024bb 100644 --- a/spinifex/vm/lifecycle_test.go +++ b/spinifex/vm/lifecycle_test.go @@ -1259,13 +1259,25 @@ func TestQMPGreetingTimeout(t *testing.T) { assert.Equal(t, qmp.DefaultGreetingTimeout, qmpGreetingTimeout(plain), "a plain VM keeps the default greeting deadline") + // A guest with a network-backed root volume, which is every real one. QEMU + // opens the drive across the object store before its monitor answers, and + // the default deadline is shorter than a cold open costs on a cluster that + // has just lost a node — which is the only time a recovery launch happens. + nbd := &VM{EBSRequests: types.EBSRequests{Requests: []types.EBSRequest{ + {Name: "vol-boot", NBDURI: "nbd+unix:///?socket=/run/spinifex/nbd/boot.sock", Boot: true}, + }}} + assert.Equal(t, qmpNBDGreetingTimeout, qmpGreetingTimeout(nbd), + "a guest with an NBD drive waits for the volume open, not the plain default") + assert.Greater(t, qmpNBDGreetingTimeout, 35*time.Second, + "the deadline must exceed the 35s cold open measured with one node of three away") + // A small GPU guest (<1 GiB scaling contribution) sits at the floor. smallGPU := &VM{ GPUAttachments: []gpu.GPUAttachment{{PCIAddress: "0000:5e:00.0"}}, Config: Config{Memory: 4096}, } - assert.Equal(t, qmpVFIOGreetingFloor, qmpGreetingTimeout(smallGPU), - "a small VFIO guest gets the floor greeting deadline") + assert.Equal(t, qmp.DefaultGreetingTimeout+qmpVFIOGreetingFloor, qmpGreetingTimeout(smallGPU), + "a small VFIO guest gets the floor on top of the plain startup it still pays") assert.Greater(t, qmpVFIOGreetingFloor, qmp.DefaultGreetingTimeout, "the VFIO floor must exceed the plain default") @@ -1274,7 +1286,7 @@ func TestQMPGreetingTimeout(t *testing.T) { GPUAttachments: []gpu.GPUAttachment{{PCIAddress: "0000:5e:00.0"}}, Config: Config{Memory: 64 * 1024}, } - assert.Equal(t, qmpVFIOGreetingBase+64*qmpVFIOGreetingPerGiB, qmpGreetingTimeout(largeGPU), + assert.Equal(t, qmp.DefaultGreetingTimeout+qmpVFIOGreetingBase+64*qmpVFIOGreetingPerGiB, qmpGreetingTimeout(largeGPU), "a large VFIO guest scales its deadline with RAM") // An oversized guest is clamped to the cap, not left unbounded. @@ -1282,10 +1294,44 @@ func TestQMPGreetingTimeout(t *testing.T) { GPUAttachments: []gpu.GPUAttachment{{PCIAddress: "0000:5e:00.0"}}, Config: Config{Memory: 1024 * 1024}, } - assert.Equal(t, qmpVFIOGreetingCap, qmpGreetingTimeout(hugeGPU), + assert.Equal(t, qmpGreetingCap, qmpGreetingTimeout(hugeGPU), "a huge VFIO guest is clamped to the cap") } +// TestQMPGreetingTimeoutAddsBothCosts is the case a real GPU instance is: it +// boots from a network-backed root volume and it passes through a GPU, so QEMU +// opens the drive over the object store and then pins the whole of guest RAM, +// one after the other, all before the monitor answers. +// +// Taking whichever cost is larger would give it a deadline that cannot cover +// what it has to do. The symptom would be a GPU instance SIGKILLed on every +// attempt while its plain sibling launches, which reads as a GPU fault. +func TestQMPGreetingTimeoutAddsBothCosts(t *testing.T) { + // EBSRequests carries a mutex, so each VM gets its own rather than a copy. + withDrive := func(v *VM) *VM { + v.EBSRequests.Requests = []types.EBSRequest{ + {Name: "vol-boot", NBDURI: "nbd+unix:///?socket=/run/spinifex/nbd/boot.sock", Boot: true}, + } + return v + } + attachment := []gpu.GPUAttachment{{PCIAddress: "0000:5e:00.0"}} + + gpuOnly := &VM{GPUAttachments: attachment, Config: Config{Memory: 16 * 1024}} + driveOnly := withDrive(&VM{}) + both := withDrive(&VM{GPUAttachments: attachment, Config: Config{Memory: 16 * 1024}}) + + // 16 GiB scales to 158s, so the VFIO term is its floor. + assert.Equal(t, qmpNBDGreetingTimeout+qmpVFIOGreetingFloor, qmpGreetingTimeout(both), + "the two costs are sequential, so the deadline that covers both is their sum") + assert.Greater(t, qmpGreetingTimeout(both), qmpGreetingTimeout(gpuOnly), + "a GPU guest that also has to open a volume over the network must be given longer than one that does not") + assert.Greater(t, qmpGreetingTimeout(both), qmpGreetingTimeout(driveOnly), + "and longer than a guest that only has the volume to open") + + assert.Greater(t, qmpGreetingCap, qmpVFIOGreetingFloor+qmpNBDGreetingTimeout, + "the cap has to clear both floors together, or it clips a guest before either cost has been waited out") +} + // TestRemoveStaleQMPSocket covers the SIGKILL-leftover unlink: a stale socket // inode is removed, and a missing file is a no-op (not an error). func TestRemoveStaleQMPSocket(t *testing.T) { diff --git a/spinifex/vm/manager.go b/spinifex/vm/manager.go index 1a793893e..2251f0de6 100644 --- a/spinifex/vm/manager.go +++ b/spinifex/vm/manager.go @@ -15,9 +15,11 @@ type ManagerHooks struct { // OnInstanceRecovering fires from Restore before each relaunch so the daemon // can early-subscribe ec2.cmd. and handle concurrent terminates in flight. OnInstanceRecovering func(*VM) - // BeforeInstanceRelaunch fires from Restore immediately before m.Run. Daemons - // use this to refresh ephemeral on-host state that did not survive a reboot. - // A non-nil error aborts the relaunch and marks the instance recovery_failed. + // BeforeInstanceRelaunch fires immediately before m.Run on any launch from a + // persisted record rather than from a request. Daemons use this to rebuild + // ephemeral on-host state the record only names: a path to a file this host + // does not have is the same problem whether the file went with a reboot or + // with the node that wrote it. A non-nil error aborts the relaunch. BeforeInstanceRelaunch func(*VM) error } diff --git a/spinifex/vm/manager_deps_test.go b/spinifex/vm/manager_deps_test.go index 8d7b076fa..c18709a5c 100644 --- a/spinifex/vm/manager_deps_test.go +++ b/spinifex/vm/manager_deps_test.go @@ -144,6 +144,8 @@ type fakeVolumeMounter struct { mu sync.Mutex mounted, unmounted []string mountedOne, unmountedOne []string + abandoned []string + abandonErr error mountErr error mountOneErr error unmountErr error @@ -175,6 +177,14 @@ func (f *fakeVolumeMounter) Unmount(_ context.Context, v *VM) error { return err } +func (f *fakeVolumeMounter) Abandon(_ context.Context, v *VM, _ string) error { + f.mu.Lock() + f.abandoned = append(f.abandoned, v.ID) + err := f.abandonErr + f.mu.Unlock() + return err +} + func (f *fakeVolumeMounter) MountOne(_ context.Context, _ string, req *types.EBSRequest) error { f.mu.Lock() f.mountedOne = append(f.mountedOne, req.Name) diff --git a/spinifex/vm/restore.go b/spinifex/vm/restore.go index 6bf52be17..d26fc444e 100644 --- a/spinifex/vm/restore.go +++ b/spinifex/vm/restore.go @@ -358,14 +358,12 @@ func (m *Manager) relaunchAll(toLaunch []*VM) { "instanceId", inst.ID, "status", string(status)) return } - if m.deps.Hooks.BeforeInstanceRelaunch != nil { - if err := m.deps.Hooks.BeforeInstanceRelaunch(inst); err != nil { - slog.Error("Pre-relaunch hook failed", - "instanceId", inst.ID, "managedBy", inst.ManagedBy, - "instanceType", inst.InstanceType, "err", err) - m.MarkRecoveryFailed(inst, "pre_relaunch_hook_failed") - return - } + if err := m.PrepareRelaunch(inst); err != nil { + slog.Error("Pre-relaunch hook failed", + "instanceId", inst.ID, "managedBy", inst.ManagedBy, + "instanceType", inst.InstanceType, "err", err) + m.MarkRecoveryFailed(inst, "pre_relaunch_hook_failed") + return } slog.Info("Launching instance (recovery)", "instance", inst.ID, "managedBy", inst.ManagedBy, "instanceType", inst.InstanceType) @@ -390,6 +388,26 @@ func (m *Manager) relaunchAll(toLaunch []*VM) { wg.Wait() } +// PrepareRelaunch rebuilds whatever on-host state an instance's record only +// names, for a launch driven by that record rather than by a request. +// +// Every such launch needs it, not only the same-node one. A system instance's +// boot configuration is a set of files under the runtime directory, and the +// record holds their paths — so a host reboot and a move to another node leave +// the record saying exactly the same wrong thing. Recovery onto a survivor is +// the case where nobody notices, because the guest simply never comes up and the +// service it was providing has no other symptom. +// +// Exported so the cross-node claim path can call it. Nothing about this is +// specific to Restore, and the hook being reachable only from there is what made +// a recovered load balancer boot with no network configuration. +func (m *Manager) PrepareRelaunch(inst *VM) error { + if m.deps.Hooks.BeforeInstanceRelaunch == nil { + return nil + } + return m.deps.Hooks.BeforeInstanceRelaunch(inst) +} + // relaunchWithRetry calls m.Run, retrying only when the failure is // ErrMountRetryable (the backing store is not yet ready). Any other error // returns immediately after a single attempt. Bounded by relaunchMaxAttempts diff --git a/spinifex/vm/restore_test.go b/spinifex/vm/restore_test.go index 628f93117..3f06cf88a 100644 --- a/spinifex/vm/restore_test.go +++ b/spinifex/vm/restore_test.go @@ -647,6 +647,7 @@ func (f *recoveryMounter) Mount(_ context.Context, v *VM) error { } func (f *recoveryMounter) Unmount(context.Context, *VM) error { return nil } +func (f *recoveryMounter) Abandon(context.Context, *VM, string) error { return nil } func (f *recoveryMounter) MountOne(context.Context, string, *types.EBSRequest) error { return nil } func (f *recoveryMounter) UnmountOne(context.Context, string, types.EBSRequest) error { return nil } diff --git a/spinifex/vm/shutdown.go b/spinifex/vm/shutdown.go index 80a9b4d1d..dd978814f 100644 --- a/spinifex/vm/shutdown.go +++ b/spinifex/vm/shutdown.go @@ -642,6 +642,51 @@ func (m *Manager) shutdownAndUnmount(instance *VM) error { return sealErr } +// ForgetSuperseded stops any local copy of an instance another node now owns +// and removes it from this node's view. +// +// It stops the guest without unmounting, for the reason shutdownQEMU is +// separate at all: the volumes belong to the new owner, and an unmount here +// would seal this node's stale copy over theirs. The local resources the +// instance was admitted against are returned, since it is no longer here. +// +// Stopping the guest is not sufficient, and this is the part that is easy to get +// wrong. The export outlives the guest and holds the volume lease, and this node +// is healthy, so it renews that lease indefinitely — the new owner would never +// acquire it, and the instance would be stopped here and unable to start +// anywhere. So the export is given up too, without sealing. +// An instance's OVN logical port is bound to whichever chassis has a tap +// carrying its iface-id, and the column holding that binding names one chassis. +// Two nodes offering the same iface-id therefore do not split the traffic — they +// contend for the binding, and the address answers from whichever won last. So +// the tap goes, on this node, for the same reason the guest does. +// +// It also unblocks the layer above: the external IP's host ingress is pruned on +// the evidence that the owning guest sits on another chassis, which stays false +// for as long as this node is still claiming the port. +func (m *Manager) ForgetSuperseded(instance *VM) { + if instance == nil { + return + } + m.shutdownQEMU(instance) + m.cleanupTapDevices(instance) + + if m.deps.VolumeMounter != nil { + if err := m.deps.VolumeMounter.Abandon(context.Background(), instance, + "the instance is owned by another node now"); err != nil { + // Logged rather than returned because there is nothing better to do + // here and the instance is going either way. The lease is the cost: a + // volume this node did not let go of is one the new owner waits on. + slog.Error("Could not give up the volumes of an instance that moved, "+ + "so this node may still hold leases its new owner needs", + "id", instance.ID, "err", err) + } + } + + m.deallocateResources(instance) + m.Delete(instance.ID) +} + // shutdownQEMU takes the guest process down and nothing else. Separated from // shutdownAndUnmount for the fence path, which must stop the guest without // unmounting: a fenced node's volumes belong to another node now, and an diff --git a/spinifex/vm/vm.go b/spinifex/vm/vm.go index 24b6d9d53..81b4d6484 100644 --- a/spinifex/vm/vm.go +++ b/spinifex/vm/vm.go @@ -39,6 +39,13 @@ type InstanceHealthState struct { // backend that keeps failing rather than one that failed once. IOErrorResumes int `json:"io_error_resumes,omitempty"` IOErrorSince time.Time `json:"io_error_since,omitzero"` + + // AddressUnreachableSince marks a guest the cloud underneath is not + // delivering a public address to. The guest is healthy and the infrastructure + // is not, which is why it reports as a system-status failure rather than an + // instance one. + AddressUnreachableSince time.Time `json:"address_unreachable_since,omitzero"` + AddressUnreachableReason string `json:"address_unreachable_reason,omitempty"` } // ExtraENI describes an additional VPC network interface attached to a VM diff --git a/spinifex/vm/volume_mounter.go b/spinifex/vm/volume_mounter.go index 1d847c8e3..9607bc563 100644 --- a/spinifex/vm/volume_mounter.go +++ b/spinifex/vm/volume_mounter.go @@ -23,6 +23,16 @@ type VolumeMounter interface { // failures are aggregated and returned, and mean the data is not durable. Unmount(ctx context.Context, v *VM) error + // Abandon gives up this node's export of every attached volume without + // sealing, for an instance another node owns now. + // + // It is not an unmount and must never become one. A seal here publishes this + // node's stale block map over the winner's. It is also not optional: the + // export holds the volume lease, and a healthy node renews that lease + // forever, so a guest stopped here without this is a guest that cannot start + // anywhere. + Abandon(ctx context.Context, v *VM, reason string) error + // MountOne sends ebs.mount for a single request and writes the resolved // NBDURI back into req.NBDURI on success. Used by hot-attach. accountID // names the owner, which a lone request carries no instance to supply. diff --git a/tests/e2e/Makefile b/tests/e2e/Makefile index 1200808b5..413ba5e22 100644 --- a/tests/e2e/Makefile +++ b/tests/e2e/Makefile @@ -18,9 +18,9 @@ SUITE_TAG ?= $(notdir $(patsubst %/...,%,$(firstword $(PKG)))) # BIN_DIR/.test so CI can ship binaries (not source + go toolchain) to # the remote VM. BIN_DIR ?= $(CURDIR)/_bin -SUITES_BUILD ?= cert lb eks ecs single iam ecr bedrock gpu natuplink multinode storagegrowth partialblock rds quota diskperf storagefault +SUITES_BUILD ?= cert lb eks ecs single iam ecr bedrock gpu natuplink multinode storagegrowth partialblock rds quota diskperf storagefault instancerecovery lbrecovery -.PHONY: e2e e2e-quota e2e-cert e2e-lb e2e-eks e2e-ecs e2e-single e2e-iam e2e-ecr e2e-bedrock e2e-multinode e2e-bm e2e-reboot e2e-tofu e2e-gpu e2e-natuplink e2e-storagegrowth e2e-partialblock e2e-diskperf e2e-rds e2e-storagefault e2e-all clean e2e-build +.PHONY: e2e e2e-quota e2e-cert e2e-lb e2e-eks e2e-ecs e2e-single e2e-iam e2e-ecr e2e-bedrock e2e-multinode e2e-bm e2e-reboot e2e-tofu e2e-gpu e2e-natuplink e2e-storagegrowth e2e-partialblock e2e-diskperf e2e-rds e2e-storagefault e2e-instancerecovery e2e-lbrecovery e2e-all clean e2e-build RUN_FLAG = $(if $(RUN),-run '$(RUN)',) @@ -68,6 +68,12 @@ e2e-quota: ; $(MAKE) e2e PKG=./quota/... # Freezes predastore, so its budget covers a deliberate outage plus the # recovery and re-verification that follow it, twice over for the two tests. e2e-storagefault: ; $(MAKE) e2e PKG=./storagefault/... TIMEOUT=45m +# Takes a node away twice and freezes predastore once. Almost all of its budget +# is waiting for staleness, a boot and a pause that is measured in minutes. +e2e-instancerecovery: ; $(MAKE) e2e PKG=./instancerecovery/... TIMEOUT=60m +# Stands a load balancer and its backends up before it breaks anything, then +# takes the load balancer's node away and brings it back. +e2e-lbrecovery: ; $(MAKE) e2e PKG=./lbrecovery/... TIMEOUT=75m e2e-all: ; $(MAKE) e2e e2e-build: diff --git a/tests/e2e/harness/dns.go b/tests/e2e/harness/dns.go index 4b164530a..acc60e816 100644 --- a/tests/e2e/harness/dns.go +++ b/tests/e2e/harness/dns.go @@ -5,6 +5,8 @@ package harness import ( "context" "fmt" + "net" + "net/http" "path/filepath" "slices" "strings" @@ -26,6 +28,11 @@ const ( // systemdUplinkResolvConf is the uplink view systemd-resolved maintains // alongside the stub, listing the nameservers it actually forwards to. systemdUplinkResolvConf = "/run/systemd/resolve/resolv.conf" + + // How long a query to one node's northstar is given. Short, because the + // caller is usually polling and a node that has stopped answering is the + // answer rather than something to wait out. + nodeDNSTimeout = 5 * time.Second ) // NorthstarBaseDomain returns the cluster's authoritative DNS base domain @@ -112,6 +119,44 @@ func RequireDNSEnabled(t *testing.T, env *Env) string { return domain } +// NodeResolver returns a resolver that asks one node's northstar directly, +// rather than whatever the runner's resolver has cached or forwards to. Port 53 +// is open on every node deliberately, because northstar serves public names. +// +// Go's own resolver rather than dig: dig is proven present in the guest image +// and nowhere else, and a test that shells out to a tool a host does not carry +// reports a DNS failure that is nothing of the kind. +func NodeResolver(node Node) *net.Resolver { + return &net.Resolver{ + PreferGo: true, + Dial: func(ctx context.Context, network, _ string) (net.Conn, error) { + d := net.Dialer{Timeout: nodeDNSTimeout} + return d.DialContext(ctx, network, net.JoinHostPort(node.Addr, "53")) + }, + } +} + +// ResolveViaNode returns the addresses one node's northstar answers for name. +func ResolveViaNode(ctx context.Context, node Node, name string) ([]string, error) { + addrs, err := NodeResolver(node).LookupHost(ctx, name) + if err != nil { + return nil, fmt.Errorf("resolve %s via %s: %w", name, node.Name, err) + } + return addrs, nil +} + +// NodeResolverHTTPClient returns an HTTP client whose hostnames are resolved by +// one node's northstar, so a probe can be driven through a name this cluster +// serves without the runner having been configured to forward to it. +func NodeResolverHTTPClient(node Node, timeout time.Duration) *http.Client { + resolver := NodeResolver(node) + dialer := &net.Dialer{Timeout: timeout, Resolver: resolver} + return &http.Client{ + Timeout: timeout, + Transport: &http.Transport{DialContext: dialer.DialContext}, + } +} + // GuestResolvers returns the nameservers a guest forwards DNS queries to, // unwrapping the systemd-resolved stub on images that run one. // diff --git a/tests/e2e/harness/instance_status.go b/tests/e2e/harness/instance_status.go new file mode 100644 index 000000000..9ff401051 --- /dev/null +++ b/tests/e2e/harness/instance_status.go @@ -0,0 +1,60 @@ +//go:build e2e + +package harness + +import ( + "fmt" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/ec2" +) + +// InstanceStatusSummary is what DescribeInstanceStatus says about one instance, +// flattened to the three labels a customer reads. Empty strings mean the field +// was absent, which is a different answer from any of the labels. +type InstanceStatusSummary struct { + State string + InstanceStatus string + SystemStatus string + AZ string +} + +func (s InstanceStatusSummary) String() string { + return fmt.Sprintf("state=%s instance-status=%s system-status=%s az=%s", + s.State, s.InstanceStatus, s.SystemStatus, s.AZ) +} + +// InstanceStatus returns DescribeInstanceStatus for one instance. +// +// IncludeAllInstances is set because the default is running-only, and a +// host-failure test has to be able to ask about an instance whose state is in +// question — which is exactly the case the default hides. Absent from the answer +// is reported as a zero summary with ok=false rather than an error: a gateway +// that returns no frame is a finding, not a failed call. +func InstanceStatus(c *AWSClient, instanceID string) (summary InstanceStatusSummary, ok bool, err error) { + out, err := c.EC2.DescribeInstanceStatus(&ec2.DescribeInstanceStatusInput{ + InstanceIds: []*string{aws.String(instanceID)}, + IncludeAllInstances: aws.Bool(true), + }) + if err != nil { + return InstanceStatusSummary{}, false, fmt.Errorf("describe-instance-status %s: %w", instanceID, err) + } + + for _, status := range out.InstanceStatuses { + if status == nil || aws.StringValue(status.InstanceId) != instanceID { + continue + } + summary.AZ = aws.StringValue(status.AvailabilityZone) + if status.InstanceState != nil { + summary.State = aws.StringValue(status.InstanceState.Name) + } + if status.InstanceStatus != nil { + summary.InstanceStatus = aws.StringValue(status.InstanceStatus.Status) + } + if status.SystemStatus != nil { + summary.SystemStatus = aws.StringValue(status.SystemStatus.Status) + } + return summary, true, nil + } + return InstanceStatusSummary{}, false, nil +} diff --git a/tests/e2e/harness/multinode_distribution.go b/tests/e2e/harness/multinode_distribution.go index 275aab827..03ee499d8 100644 --- a/tests/e2e/harness/multinode_distribution.go +++ b/tests/e2e/harness/multinode_distribution.go @@ -6,6 +6,7 @@ import ( "context" "fmt" "os/exec" + "slices" "strings" "testing" "time" @@ -15,20 +16,42 @@ import ( // qemu process (ps auxw | grep qemu-system). Returns nil if no node owns it. func InstanceHostingNode(t *testing.T, c *Cluster, instanceID string) *Node { t.Helper() - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + hosting := InstanceHostingNodes(t, c, instanceID, nil) + if len(hosting) == 0 { + return nil + } + return &hosting[0] +} + +// InstanceHostingNodes returns every node running the named instance's qemu +// process, so a caller can assert there is exactly one rather than assume it. +// Nodes in skip are not contacted, for a node deliberately taken down. +// +// A node that cannot be reached is skipped rather than fatal: this is used to +// prove an instance did not come back in two places at once, and a node that +// answers nothing is running nothing that matters to that question. +func InstanceHostingNodes(t *testing.T, c *Cluster, instanceID string, skip []string) []Node { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) defer cancel() ssh := NewPeerSSH() + + var hosting []Node for i := range c.Nodes { n := c.Nodes[i] + if slices.Contains(skip, n.Name) { + continue + } out, err := runPSGrep(ctx, ssh, n, instanceID) if err != nil { - t.Fatalf("discover instance %s on %s: %v", instanceID, n.Name, err) + t.Logf("InstanceHostingNodes: %s did not answer for %s: %v", n.Name, instanceID, err) + continue } if strings.Contains(out, instanceID) { - return &c.Nodes[i] + hosting = append(hosting, c.Nodes[i]) } } - return nil + return hosting } // runPSGrep runs the ps+grep pipeline on a node and returns combined output. diff --git a/tests/e2e/harness/partition.go b/tests/e2e/harness/partition.go new file mode 100644 index 000000000..4ee892ec3 --- /dev/null +++ b/tests/e2e/harness/partition.go @@ -0,0 +1,207 @@ +//go:build e2e + +package harness + +import ( + "context" + "fmt" + "strconv" + "strings" + "testing" + "time" +) + +// partitionTable is the nft table a partition lives in, and it is its own table +// on purpose. +// +// The node's real firewall is inet spinifex_filter, built by setup.sh, and a +// test that flushed or edited it would leave the node either unfirewalled or +// misfirewalled with no sign of which. nftables evaluates every table at each +// hook and a drop in any of them wins, so a separate table expresses exactly +// "these packets, dropped, and nothing else" — and removing it is one atomic +// delete that cannot take a rule the node needed with it. +const partitionTable = "spx_e2e_partition" + +// partitionHealUnit is a transient systemd timer that removes the table on its +// own, so a run killed between the partition and the heal — a cancelled +// workflow, a panicking test, a severed orchestrator — does not leave a node +// cut off from its cluster until somebody notices. +const partitionHealUnit = "spx-e2e-partition-heal" + +// partitionDeadman is how long that timer waits. Longer than any scenario that +// installs a partition, short enough that an abandoned one is gone before the +// next nightly reaches this cell. +const partitionDeadman = 30 * time.Minute + +// PortSet is a named group of TCP ports a partition cuts. Named because the +// name is what the test output says was severed, and "nats" is a fault a reader +// can reason about where a list of numbers is not. +type PortSet struct { + Name string + Ports []int +} + +// The port sets worth cutting on their own. Each is a different failure, and +// they are deliberately not combined into one "isolate the node": a node that +// has lost consensus but can still read its volumes behaves differently from one +// that has lost both, and a test that could not tell them apart would be +// asserting whichever happened. +var ( + // NATSPorts is the client port and the cluster route port. Cutting both + // leaves the node's own NATS server running and its local clients connected + // while severing the Raft routes, which is the half-healthy state a real + // network fault produces: services up, consensus unreachable. + NATSPorts = PortSet{Name: "nats", Ports: []int{4222, 4248}} + + // ObjectStorePorts is predastore's gate and its admin surface. A node that + // cannot reach these cannot read its guests' volumes, which is a storage + // fault rather than a partition and is answered differently. + ObjectStorePorts = PortSet{Name: "predastore", Ports: []int{8443, 8660}} +) + +// PartitionPeerPorts drops TCP traffic on the given ports between target and +// every other node in the cluster, in both directions, and returns the heal. +// +// Both directions and both sport and dport, because a node both dials its peers +// and accepts their connections, and a partition that cut only one of those +// would leave half the conversation working — which is a fault, but not the one +// being asked about, and not one any production network produces. +// +// SSH is proved still working afterwards. If the orchestrator reaches these +// nodes on an address that is also a peer address, the partition would sever the +// test's own control path, and failing there names that rather than timing out +// later with every assertion unexplained. +func PartitionPeerPorts(t *testing.T, cluster *Cluster, target Node, sets ...PortSet) (heal func()) { + t.Helper() + if len(sets) == 0 { + t.Fatalf("PartitionPeerPorts %s: no port sets given, so nothing would be severed", target.Name) + } + + peers := cluster.Peers(target) + if len(peers) == 0 { + t.Fatalf("PartitionPeerPorts %s: no peers in the cluster, so there is nothing to partition from", target.Name) + } + + var names, ports []string + for _, set := range sets { + names = append(names, set.Name) + for _, port := range set.Ports { + ports = append(ports, strconv.Itoa(port)) + } + } + var addrs []string + for _, peer := range peers { + addrs = append(addrs, ShellQuote(peer.Addr)) + } + + Step(t, "partition %s from %v on %s (ports %s)", + target.Name, nodeNames(peers), strings.Join(names, "+"), strings.Join(ports, ",")) + + // Resolved on the node, not here: it is the node's own resolution that + // decides where its packets go, and the orchestrator may not share it. + script := fmt.Sprintf(`set -e +peers= +for host in %s; do + addr=$(getent ahostsv4 "$host" | awk 'NR==1{print $1}') + [ -n "$addr" ] || { echo "cannot resolve $host on this node" >&2; exit 1; } + peers="${peers:+$peers, }$addr" +done +sudo systemctl stop %s.timer 2>/dev/null || true +sudo nft -f - </dev/null 2>&1 +echo "$peers"`, + strings.Join(addrs, " "), + partitionHealUnit, + partitionTable, partitionTable, partitionTable, + strings.Join(ports, ", "), + partitionHealUnit, int(partitionDeadman.Seconds()), partitionTable) + + healed := false + heal = func() { + if healed { + return + } + healed = true + Step(t, "heal the partition on %s", target.Name) + out, err := partitionRun(target, fmt.Sprintf( + "sudo systemctl stop %s.timer 2>/dev/null || true; "+ + "sudo nft delete table inet %s 2>/dev/null || true; "+ + "sudo nft list tables | grep -c %s || true", + partitionHealUnit, partitionTable, partitionTable)) + if err != nil { + t.Errorf("could not heal the partition on %s, which leaves it cut off from its cluster "+ + "until the deadman timer fires: %v\n%s", target.Name, err, out) + return + } + if remaining := strings.TrimSpace(out); remaining != "0" && remaining != "" { + t.Errorf("the partition table is still present on %s after the heal: %q", target.Name, remaining) + } + } + t.Cleanup(heal) + + out, err := partitionRun(target, script) + if err != nil { + t.Fatalf("could not partition %s: %v\n%s", target.Name, err, out) + } + Detail(t, "partitioned_from", strings.TrimSpace(out)) + + if _, err := partitionRun(target, "echo ok"); err != nil { + t.Fatalf("the partition severed the orchestrator's own SSH to %s, so the test cannot observe "+ + "anything it does; the orchestrator is probably reachable on a peer address: %v", target.Name, err) + } + + // The fault is the premise of everything after it, so it is proved rather + // than assumed. A rule set that installed cleanly and dropped nothing would + // otherwise read as the system tolerating a partition. + requirePeerPortUnreachable(t, target, peers[0], sets[0].Ports[0]) + return heal +} + +// requirePeerPortUnreachable proves the partition is actually dropping packets. +func requirePeerPortUnreachable(t *testing.T, target, peer Node, port int) { + t.Helper() + + out, err := partitionRun(target, fmt.Sprintf( + "timeout 5 bash -c 'cat /dev/tcp/%s/%d' && echo REACHED || echo BLOCKED", + ShellQuote(peer.Addr), port)) + if err != nil { + t.Fatalf("could not test the partition from %s: %v\n%s", target.Name, err, out) + } + if !strings.Contains(out, "BLOCKED") { + t.Fatalf("%s can still reach %s:%d after the partition, so nothing below would be a test of one\n%s", + target.Name, peer.Name, port, out) + } +} + +func partitionRun(node Node, cmd string) (string, error) { + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + out, err := NewPeerSSH().Run(ctx, node.Addr, cmd) + return string(out), err +} + +func nodeNames(nodes []Node) []string { + out := make([]string, 0, len(nodes)) + for _, node := range nodes { + out = append(out, node.Name) + } + return out +} diff --git a/tests/e2e/harness/traffic.go b/tests/e2e/harness/traffic.go index d4289a6a1..e1a4fe74d 100644 --- a/tests/e2e/harness/traffic.go +++ b/tests/e2e/harness/traffic.go @@ -27,8 +27,14 @@ func (r TrafficResult) Unique() int { return len(r.Distribution) } // JSON {"instance_id":"..."}, and counts responders. Matches the app-userdata // HTTP responder in run-lb-e2e.sh. func HTTPRoundRobin(url string, n int, timeout time.Duration) TrafficResult { + return HTTPRoundRobinWithClient(&http.Client{Timeout: timeout}, url, n) +} + +// HTTPRoundRobinWithClient is HTTPRoundRobin over a caller-supplied client, for +// probes that have to reach the URL by a particular path — a client whose +// hostnames are resolved by one named server rather than the runner's resolver. +func HTTPRoundRobinWithClient(client *http.Client, url string, n int) TrafficResult { r := TrafficResult{Distribution: map[string]int{}, Total: n} - client := &http.Client{Timeout: timeout} for i := 0; i < n; i++ { resp, err := client.Get(url) if err != nil { diff --git a/tests/e2e/instancerecovery/main_test.go b/tests/e2e/instancerecovery/main_test.go new file mode 100644 index 000000000..0b345c6a4 --- /dev/null +++ b/tests/e2e/instancerecovery/main_test.go @@ -0,0 +1,104 @@ +//go:build e2e + +// Package instancerecovery proves what happens to a running guest when the +// node under it stops answering, from both sides: a host failure moves it, and +// a storage failure must not. +// +// It is its own suite for the reason storagefault is: both tests take a node +// away and one of them freezes a cluster-wide service, so anything sharing the +// environment would see the same outage and report a failure that says nothing +// about itself. It is also slow by nature — a guest has to be seen stale, then +// claimed, then booted, and the storage case has to wait minutes for a pause +// that is not prompt — which is why it does not belong in a permutation cell +// budgeted at thirty-five minutes. +package instancerecovery + +import ( + "fmt" + "os" + "sync" + "testing" + + "github.com/mulgadc/spinifex/tests/e2e/harness" +) + +var ( + pkgFixOnce sync.Once + pkgFix *Fixture + pkgFixErr error +) + +func TestMain(m *testing.M) { + code := m.Run() + if pkgFix != nil && pkgFix.Harness != nil { + if err := pkgFix.Harness.Close(); err != nil { + fmt.Fprintf(os.Stderr, "e2e teardown: %v\n", err) + code = 1 + } + } + os.Exit(code) +} + +// Fixture carries per-process state shared across this package's tests. +type Fixture struct { + Env *harness.Env + AWS *harness.AWSClient + Harness *harness.Fixture + Cluster *harness.Cluster +} + +// requireFixture returns the package-scoped Fixture, building it on first +// call. Skips below three nodes, where recovery is inert by design: there is +// no survivor to move a guest to that is not the node that failed. +func requireFixture(t *testing.T) *Fixture { + t.Helper() + pkgFixOnce.Do(func() { + if os.Getenv("SPINIFEX_E2E") == "" { + return + } + env := harness.LoadEnv(t) + cluster, err := harness.ClusterFromEnv() + if err != nil { + pkgFixErr = fmt.Errorf("this suite needs node SSH to take a node away: %w", err) + return + } + awsCli := harness.NewAWSClient(t, env) + h, err := harness.NewProcessFixture(awsCli) + if err != nil { + pkgFixErr = err + return + } + harness.EnsureDefaultSGOpen(t, awsCli) + pkgFix = &Fixture{Env: env, AWS: awsCli, Harness: h, Cluster: cluster} + }) + if pkgFixErr != nil { + t.Fatalf("instancerecovery fixture init: %v", pkgFixErr) + } + if pkgFix == nil { + t.Skip("SPINIFEX_E2E unset") + } + if n := len(pkgFix.Cluster.Nodes); n < 3 { + t.Skipf("recovery is inert below three nodes, have %d", n) + } + return pkgFix +} + +// TestInstanceAutoRecovery is sequential and runs first: it leaves the cluster +// whole again before the storage case takes a node away for a second time. +func TestInstanceAutoRecovery(t *testing.T) { + runInstanceAutoRecovery(t, requireFixture(t)) +} + +// TestInstanceRecoveryRefusesStorageFault is sequential and declared after the +// recovery test, because it asserts the opposite outcome and a reconciler that +// cannot tell the two faults apart passes one and fails the other. +func TestInstanceRecoveryRefusesStorageFault(t *testing.T) { + runInstanceRecoveryRefusesStorageFault(t, requireFixture(t)) +} + +// TestInstancePartitionRecovery is last because it is the only one that leaves +// firewall state behind if it fails, and running it after the other two means a +// failure here cannot be mistaken for the cause of theirs. +func TestInstancePartitionRecovery(t *testing.T) { + runInstancePartitionRecovery(t, requireFixture(t)) +} diff --git a/tests/e2e/instancerecovery/partition_test.go b/tests/e2e/instancerecovery/partition_test.go new file mode 100644 index 000000000..ee85d177e --- /dev/null +++ b/tests/e2e/instancerecovery/partition_test.go @@ -0,0 +1,249 @@ +//go:build e2e + +package instancerecovery + +import ( + "fmt" + "strings" + "sync" + "testing" + "time" + + "github.com/mulgadc/spinifex/tests/e2e/harness" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const ( + // How long the partitioned node is given to notice and stop its own guest. + // Its lease validity is 30s and the check runs every 5s, so this is several + // times what it should take; the value is generous because failing it means + // a node kept writing to a volume it could not prove it owned, and that is + // worth waiting for rather than guessing at. + partitionFenceBudget = 4 * time.Minute + + // How long a survivor is given to claim and relaunch afterwards. The + // server-side lease TTL has to expire on top of everything the ordinary + // recovery waits for, so this is longer than recoveryBudget. + partitionRecoveryBudget = 10 * time.Minute + + // How often the split-brain watch samples every node. Fast enough that a + // second writer lasting one lease check would be seen. + splitBrainPollInterval = 3 * time.Second + + // The fence and the surrender, as the node writes them. Matched because the + // count of qemu processes says the guest stopped and these say why, and a + // guest that stopped for another reason would prove nothing about fencing. + partitionSurrenderLog = "volume lease could not be confirmed before its TTL, surrendering the volume" + partitionFenceLog = "fencing volume: the lease moved while this node had it open" +) + +// runInstancePartitionRecovery is the production failure this whole feature +// exists for, and the only test here that produces it rather than approximating +// it with a service stop. +// +// A node loses its network to the rest of the cluster and nothing else. Its +// services stay up, its QEMU keeps running, its local NATS keeps answering its +// own clients — it is half healthy, and it cannot reach consensus. That is the +// state in which two nodes can each believe they own a volume, and the only +// thing standing between that and a corrupted disk is that the partitioned node +// gives up first, on a clock it can evaluate without reaching anything. +// +// So the assertion that matters is not that the guest comes back. It is that at +// no moment is it running in two places. The ordering the safety argument rests +// on — local validity (30s) lapses strictly before the server-side TTL (45s) +// lets anyone else in — is observable, and this is what observes it. +func runInstancePartitionRecovery(t *testing.T, fix *Fixture) { + harness.Phase(t, "Recovery From a Network Partition") + recoveryWaitActive(t, fix) + + victim, instanceID := recoveryGuestOnAPeer(t, fix, diskLoadUserData) + harness.Detail(t, "victim_node", victim.Name) + harness.Detail(t, "instance", instanceID) + + // The guest has to be writing before the partition. A guest that never + // touched its volume has no lease renewal to lose and nothing to fence, so + // the test would pass having proved nothing. + recoveryWaitForDiskLoad(t, fix, instanceID) + + cli := harness.AWSClientForGateway(t, fix.Env, fix.Cluster.Nodes[0]) + healthy := recoveryRequireStatus(t, cli, instanceID, func(s harness.InstanceStatusSummary) bool { + return s.State == "running" && s.SystemStatus == "ok" + }, 3*time.Minute, "state=running with system-status=ok before the partition") + + watch := newSplitBrainWatch(t, fix, instanceID) + defer watch.stop() + + heal := harness.PartitionPeerPorts(t, fix.Cluster, victim, harness.NATSPorts) + + // Proving the node is still itself is what separates this from stopping it. + // If its services had fallen over, the rest would be the ordinary recovery + // test with extra steps. + harness.Step(t, "confirm %s is still running its services and its guest", victim.Name) + require.NotEmptyf(t, harness.InstanceHostingNodes(t, fix.Cluster, instanceID, nil), + "%s is not running anywhere immediately after the partition, so the guest died for some other reason", instanceID) + assert.Contains(t, recoveryRun(t, victim, "systemctl is-active spinifex-daemon spinifex-viperblock || true"), + "active", "the partitioned node's services must still be up, or this is not a half-healthy node") + + harness.Step(t, "wait for %s to give up the volume lease it can no longer confirm and stop its own guest", victim.Name) + require.Eventuallyf(t, func() bool { + return len(harness.InstanceHostingNodes(t, fix.Cluster, instanceID, peerNames(fix, victim))) == 0 + }, partitionFenceBudget, 5*time.Second, + "%s is still running %s after %s with no way to confirm it owns the volume; "+ + "a node that cannot prove it is the only writer and keeps writing is the corruption this fence exists to prevent\n%s", + victim.Name, instanceID, partitionFenceBudget, + lazyWhy(func() string { return partitionWhy(t, victim, instanceID) })) + + // Why it stopped, not just that it did. A guest killed by the OOM killer + // would satisfy the count above and mean the opposite. + journal := recoveryRun(t, victim, recoveryStorageJournalThisRun+" | grep -iE 'lease|fenc' | tail -40") + assert.Containsf(t, journal, partitionSurrenderLog, + "%s stopped the guest without recording that its lease went unconfirmed, so it stopped for some other reason\n%s", + victim.Name, journal) + assert.Contains(t, journal, partitionFenceLog, + "the surrender has to reach the export: a lease given up while nbdkit keeps serving has fenced nothing") + + harness.Step(t, "wait for a survivor to take %s over", instanceID) + var hosting []harness.Node + require.Eventuallyf(t, func() bool { + hosting = harness.InstanceHostingNodes(t, fix.Cluster, instanceID, []string{victim.Name}) + return len(hosting) > 0 + }, partitionRecoveryBudget, 15*time.Second, + "%s never came back on a survivor while %s was partitioned\n%s", + instanceID, victim.Name, lazyWhy(func() string { return recoveryWhy(t, fix, victim, instanceID) })) + + require.Lenf(t, hosting, 1, "%s is running on more than one survivor: %v", + instanceID, recoveryNodeNames(hosting)) + harness.Detail(t, "recovered_onto", hosting[0].Name) + + recovered := recoveryRequireStatus(t, cli, instanceID, func(s harness.InstanceStatusSummary) bool { + return s.State == "running" && s.SystemStatus == "ok" && s.InstanceStatus != "impaired" + }, 5*time.Minute, "system-status back to ok once a survivor is running it") + assert.Equal(t, healthy.AZ, recovered.AZ, + "a recovery must not move an instance between availability zones") + + harness.Step(t, "heal the partition and assert %s does not bring the guest back", victim.Name) + heal() + fix.Cluster.WaitNATSPeers(t, len(fix.Cluster.Nodes)-1, + harness.WithTimeout(3*time.Minute), harness.WithPoll(2*time.Second)) + + // A rejoining node has a local state file that still lists the guest, and + // nothing but the record stops it starting a second copy. Long enough that + // it would have. + time.Sleep(90 * time.Second) + after := harness.InstanceHostingNodes(t, fix.Cluster, instanceID, nil) + require.Lenf(t, after, 1, + "%s is running on %v after the partition healed; a node that rejoins must drop a copy the records say moved", + instanceID, recoveryNodeNames(after)) + assert.NotEqualf(t, victim.Name, after[0].Name, + "%s went back to %s, so the rejoining node relaunched it rather than forgetting it", + instanceID, victim.Name) + + // Last, because it covers the whole run including the heal. Nothing above + // could have seen a second writer that appeared between its own samples. + watch.stop() + watch.requireNeverTwoWriters(t) +} + +// splitBrainWatch samples which nodes are running a guest, continuously, for as +// long as a scenario lasts. +// +// The point-in-time assertions in the test above each answer "is it in two +// places now". This answers "was it ever", which is the question a corruption +// would be the consequence of, and the one no single sample can reach. +type splitBrainWatch struct { + stopOnce sync.Once + done chan struct{} + finished chan struct{} + + mu sync.Mutex + worst []string + samples int + failures int +} + +func newSplitBrainWatch(t *testing.T, fix *Fixture, instanceID string) *splitBrainWatch { + t.Helper() + w := &splitBrainWatch{done: make(chan struct{}), finished: make(chan struct{})} + + go func() { + defer close(w.finished) + ticker := time.NewTicker(splitBrainPollInterval) + defer ticker.Stop() + for { + select { + case <-w.done: + return + case <-ticker.C: + hosting := harness.InstanceHostingNodes(t, fix.Cluster, instanceID, nil) + w.mu.Lock() + w.samples++ + if len(hosting) > 1 { + w.failures++ + if len(hosting) > len(w.worst) { + w.worst = recoveryNodeNames(hosting) + } + } + w.mu.Unlock() + } + } + }() + return w +} + +func (w *splitBrainWatch) stop() { + w.stopOnce.Do(func() { + close(w.done) + <-w.finished + }) +} + +// requireNeverTwoWriters is the assertion the scenario exists for. A single +// sample naming two nodes is a failure — the guests write to the same volume +// through the same object store, so two of them is data loss already under way, +// not a race that might resolve. +func (w *splitBrainWatch) requireNeverTwoWriters(t *testing.T) { + t.Helper() + w.mu.Lock() + defer w.mu.Unlock() + + require.Positivef(t, w.samples, + "the split-brain watch took no samples, so it proved nothing about two writers") + harness.Detail(t, "split_brain_samples", fmt.Sprintf("%d", w.samples)) + require.Zerof(t, w.failures, + "the instance was running on more than one node in %d of %d samples, worst case %v; "+ + "two guests writing one volume through one object store is data loss already under way", + w.failures, w.samples, w.worst) +} + +// peerNames is every node but victim, for a poll that must ask only the victim. +func peerNames(fix *Fixture, victim harness.Node) []string { + var out []string + for _, node := range fix.Cluster.Nodes { + if node.Name != victim.Name { + out = append(out, node.Name) + } + } + return out +} + +// partitionWhy gathers the partitioned node's own account, for a failure that +// would otherwise send the reader to a journal on a node they have to remember +// is partitioned. +func partitionWhy(t *testing.T, victim harness.Node, instanceID string) string { + t.Helper() + var b strings.Builder + for _, query := range []string{ + recoveryJournalThisRun + " | grep -iE 'lease|fenc|nats' | tail -30", + recoveryStorageJournalThisRun + " | grep -iE 'lease|fenc|nats' | tail -30", + "ps auxw | grep -F " + harness.ShellQuote(instanceID) + " | grep -v grep || true", + "sudo nft list table inet " + "spx_e2e_partition" + " 2>&1 | head -30", + } { + out, err := recoveryRunErr(victim, query) + if err != nil { + out = "(" + err.Error() + ")" + } + fmt.Fprintf(&b, " %s: %s\n%s\n", victim.Name, query, strings.TrimSpace(out)) + } + return b.String() +} diff --git a/tests/e2e/instancerecovery/recovery_test.go b/tests/e2e/instancerecovery/recovery_test.go new file mode 100644 index 000000000..abeb048b9 --- /dev/null +++ b/tests/e2e/instancerecovery/recovery_test.go @@ -0,0 +1,491 @@ +//go:build e2e + +package instancerecovery + +import ( + "context" + "encoding/base64" + "fmt" + "strings" + "sync" + "testing" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/mulgadc/spinifex/tests/e2e/harness" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const ( + // recoveryJournalThisRun reads the daemon's log for its current invocation + // only. Scoping it that way is what lets a line logged once at startup be + // found on a daemon that has been up for days. + recoveryJournalThisRun = `sudo journalctl --no-pager ` + + `_SYSTEMD_INVOCATION_ID="$(systemctl show -p InvocationID --value spinifex-daemon)"` + + // The same for the storage service. The lease, the surrender and the fence + // are its work: the daemon only hears the announcement afterwards, so its + // journal carries the consequence and never the reason. + recoveryStorageJournalThisRun = `sudo journalctl --no-pager ` + + `_SYSTEMD_INVOCATION_ID="$(systemctl show -p InvocationID --value spinifex-viperblock)"` + + // The owner has to be seen stale across two passes before anything is + // claimed, then the volume leases have to expire before the launch can open + // them. That is roughly a minute of unavoidable waiting before the first + // attempt can succeed, and a boot on top of it. + recoveryBudget = 8 * time.Minute + + // How long the guest is given to notice its storage has gone and pause. + // Generous because nothing about it is prompt: viperblock serves what it + // has cached, nbdkit waits out its reconnect delay, and only then does the + // error reach QEMU. Measured at 4m47s on a three-node cluster. + recoveryPauseBudget = 10 * time.Minute + + // How long the guest is given to boot, run cloud-init, and announce that its + // first write landed. The announcement is the premise of the whole test, so + // this is generous and failing it is a hard error rather than a skip. + recoveryDiskLoadBudget = 6 * time.Minute + + // diskLoadMarker is printed to the serial console by the load driver once a + // full 32 MiB pass has been written and synced, so it is proof the guest is + // driving its root volume rather than a claim that it was asked to. + diskLoadMarker = "spx-e2e-diskload: writing" + + // diskLoadUserData starts a guest writing to its root volume and never + // stopping. An idle guest issues no I/O that reaches the object store, so + // freezing the store leaves it running and this test proves nothing; + // measured on a real cluster, an idle guest never paused at all. + // + // It is cloud-init rather than SSH into the guest on purpose: routed-NAT + // clusters give guests no public address, so a test that needed to log in + // could not run on half the cells. The console is readable the same way, by + // the API, so the marker travels the same path. + // + // The writing never stops and the announcing does. The console is a ring the + // whole test reads back, and a line a second for an hour pushes everything + // that explains a failure out of it — including the kernel's own account of + // the I/O error this test exists to produce. + diskLoadUserData = `#!/bin/bash +cat > /usr/local/sbin/spx-e2e-diskload <<'EOF' +#!/bin/bash +announce=20 +while true; do + dd if=/dev/urandom of=/var/tmp/spx-e2e-load bs=1M count=32 oflag=direct conv=fsync 2>/dev/null + sync + if [ "$announce" -gt 0 ]; then + echo "spx-e2e-diskload: writing" > /dev/console + announce=$((announce - 1)) + [ "$announce" -eq 0 ] && echo "spx-e2e-diskload: still writing, quietly from here" > /dev/console + fi + sleep 1 +done +EOF +chmod +x /usr/local/sbin/spx-e2e-diskload +systemd-run --unit=spx-e2e-diskload /usr/local/sbin/spx-e2e-diskload +` +) + +// runInstanceAutoRecovery proves a guest on a node that stops answering comes +// back on a survivor with no operator action. +// +// This is the one behaviour the feature exists for, and the only honest way to +// assert it is to look for the qemu process on the other nodes: the API +// reporting an instance running says where the records think it is, not where +// it is. +func runInstanceAutoRecovery(t *testing.T, fix *Fixture) { + harness.Phase(t, "Instance Auto-Recovery") + recoveryWaitActive(t, fix) + + victim, instanceID := recoveryGuestOnAPeer(t, fix, "") + harness.Detail(t, "victim_node", victim.Name) + harness.Detail(t, "instance", instanceID) + + cli := harness.AWSClientForGateway(t, fix.Env, fix.Cluster.Nodes[0]) + healthy := recoveryRequireStatus(t, cli, instanceID, func(s harness.InstanceStatusSummary) bool { + return s.State == "running" && s.SystemStatus == "ok" + }, 3*time.Minute, "state=running with system-status=ok before anything is broken") + + harness.Step(t, "stop the spinifex services on %s — the guest keeps running, nothing drains it", victim.Name) + harness.StopNode(t, victim) + t.Cleanup(func() { harness.StartNode(t, victim) }) + + fix.Cluster.WaitNATSPeers(t, 1, harness.WithTimeout(60*time.Second), + harness.WithPoll(2*time.Second), harness.WithSkipNodes(victim.Name)) + + // The AWS-visible half of a host failure, and the reason this is asserted + // before the recovery rather than after it: the window is real and bounded — + // it opens when the owner's heartbeat goes stale, which is strictly before any + // survivor may claim, and closes when the relaunch registers. A customer + // watching DescribeInstanceStatus is told the host is impaired, which is what + // AWS reports for exactly this fault and is all AWS ever discloses about a + // host. Reporting ok throughout would be the more comfortable lie. + recoveryRequireStatus(t, cli, instanceID, func(s harness.InstanceStatusSummary) bool { + return s.SystemStatus == "impaired" || s.SystemStatus == "insufficient-data" + }, 4*time.Minute, "system-status=impaired while its host is gone") + + harness.Step(t, "wait for a survivor to claim and relaunch %s", instanceID) + var hosting []harness.Node + require.Eventuallyf(t, func() bool { + hosting = harness.InstanceHostingNodes(t, fix.Cluster, instanceID, []string{victim.Name}) + return len(hosting) > 0 + }, recoveryBudget, 15*time.Second, + "%s never came back on a survivor after %s stopped answering\n%s", + instanceID, victim.Name, lazyWhy(func() string { return recoveryWhy(t, fix, victim, instanceID) })) + + require.Lenf(t, hosting, 1, "%s is running on more than one survivor: %v", + instanceID, recoveryNodeNames(hosting)) + harness.Detail(t, "recovered_onto", hosting[0].Name) + + harness.Step(t, "the gateway agrees the instance is running and names the new owner") + harness.WaitForInstanceState(t, cli, instanceID, "running") + + // The other half of the parity, and the part that says the move finished + // rather than merely started. AWS never discloses which host an instance is + // on, so the customer-visible evidence that it moved is that the impairment + // ended while the instance kept running — and that the AZ did not change, + // because a recovery that crossed one would break every placement promise + // made to the customer. + recovered := recoveryRequireStatus(t, cli, instanceID, func(s harness.InstanceStatusSummary) bool { + return s.State == "running" && s.SystemStatus == "ok" && s.InstanceStatus != "impaired" + }, 5*time.Minute, "system-status back to ok on the survivor, with the instance still running") + assert.Equalf(t, healthy.AZ, recovered.AZ, + "%s came back in %s having been launched in %s; a recovery must not move an instance between availability zones", + instanceID, recovered.AZ, healthy.AZ) + + harness.Step(t, "bring %s back and assert it does not relaunch what moved", victim.Name) + harness.StartNode(t, victim) + harness.WaitNodeServiceReady(t, victim, harness.WithTimeout(3*time.Minute)) + fix.Cluster.WaitNATSPeers(t, 2, harness.WithTimeout(2*time.Minute), harness.WithPoll(2*time.Second)) + + // Give the returning node long enough to have got it wrong. Its restore runs + // early in startup, so a second copy would already be visible here. + time.Sleep(60 * time.Second) + after := harness.InstanceHostingNodes(t, fix.Cluster, instanceID, nil) + require.Lenf(t, after, 1, + "%s is running on %v after %s returned; a returning node must drop a local copy the records say moved", + instanceID, recoveryNodeNames(after), victim.Name) + assert.NotEqualf(t, victim.Name, after[0].Name, + "%s went back to %s, so the returning node relaunched it rather than forgetting it", + instanceID, victim.Name) +} + +// runInstanceRecoveryRefusesStorageFault proves the opposite behaviour, and it +// is the one a reconciler that cannot tell its faults apart gets wrong. +// +// A guest whose object store has stopped answering is paused by werror=stop +// with its failed request held. Moving it relaunches it somewhere that will +// refuse it identically, having thrown away its RAM on the way. So nothing must +// move, and when the store comes back the guest must resume where it was. +func runInstanceRecoveryRefusesStorageFault(t *testing.T, fix *Fixture) { + harness.Phase(t, "Recovery Refuses a Storage Fault") + recoveryWaitActive(t, fix) + + victim, instanceID := recoveryGuestOnAPeer(t, fix, diskLoadUserData) + harness.Detail(t, "victim_node", victim.Name) + harness.Detail(t, "instance", instanceID) + + recoveryWaitForDiskLoad(t, fix, instanceID) + + harness.Step(t, "freeze predastore on every node — the store is now broken cluster-wide") + restore := recoveryFreezeStore(t, fix) + + harness.Step(t, "wait for %s to pause on its storage and publish that it did", instanceID) + require.Eventuallyf(t, func() bool { + return recoveryGuestIsPaused(t, victim, instanceID) + }, recoveryPauseBudget, 10*time.Second, + "%s never paused on io-error, so this run proves nothing about a storage fault", instanceID) + + harness.Step(t, "stop the spinifex services on %s, leaving a stale owner and a broken store", victim.Name) + harness.StopNode(t, victim) + t.Cleanup(func() { harness.StartNode(t, victim) }) + + fix.Cluster.WaitNATSPeers(t, 1, harness.WithTimeout(60*time.Second), + harness.WithPoll(2*time.Second), harness.WithSkipNodes(victim.Name)) + + // Long enough that a reconciler which was going to move it would have. Two + // stale passes plus a lease TTL is about a minute; this is four. + harness.Step(t, "hold for %s and assert no survivor claims it", 4*time.Minute) + deadline := time.Now().Add(4 * time.Minute) + for time.Now().Before(deadline) { + hosting := harness.InstanceHostingNodes(t, fix.Cluster, instanceID, []string{victim.Name}) + require.Emptyf(t, hosting, + "%s was relaunched on %v while the object store was refusing every node; "+ + "moving a guest cannot fix a store that is broken everywhere, and it costs the guest its held request", + instanceID, recoveryNodeNames(hosting)) + time.Sleep(20 * time.Second) + } + + harness.Step(t, "bring the store back and assert the guest resumes rather than having moved") + restore() + harness.StartNode(t, victim) + harness.WaitNodeServiceReady(t, victim, harness.WithTimeout(3*time.Minute)) + fix.Cluster.WaitNATSPeers(t, 2, harness.WithTimeout(2*time.Minute), harness.WithPoll(2*time.Second)) + + cli := harness.AWSClientForGateway(t, fix.Env, fix.Cluster.Nodes[0]) + harness.WaitForInstanceState(t, cli, instanceID, "running") + + require.Eventuallyf(t, func() bool { + return len(harness.InstanceHostingNodes(t, fix.Cluster, instanceID, nil)) == 1 + }, 3*time.Minute, 15*time.Second, + "%s is not running on exactly one node once the store returned", instanceID) +} + +// recoveryWaitActive waits until every node has recovery running, and nothing +// here turns it on: there is no setting to turn on. +// +// That absence is the assertion. A cluster installed by the ordinary path has +// recovery active on every node with no config written and no daemon restarted, +// so a build that reintroduced a switch would fail here rather than quietly +// leaving a dead node's guests down. +// +// Every node, not just the survivors, because any of them may be the one that +// claims. The reconciler is also inert until it has seen every peer live, so a +// run that started measuring before that would be measuring the settle window. +func recoveryWaitActive(t *testing.T, fix *Fixture) { + t.Helper() + harness.Step(t, "wait for all %d nodes to report recovery active, with no config written", len(fix.Cluster.Nodes)) + + for _, node := range fix.Cluster.Nodes { + n := node + require.Eventuallyf(t, func() bool { + out, err := recoveryRunErr(n, recoveryJournalThisRun) + return err == nil && strings.Contains(out, "recovery has seen every peer live") + }, 6*time.Minute, 5*time.Second, + "%s never reported recovery active, so either the settle window has not passed "+ + "or recovery is off on a cluster nothing configured — and nothing below would be a test of it", n.Name) + } +} + +// recoveryFreezeStore SIGSTOPs predastore on every node and returns the thaw, +// which is also registered as a cleanup so a failure anywhere restores it. +// +// SIGSTOP rather than stopping the unit, for the reason the storagefault suite +// gives: a stopped process holds its connections open and answers nothing, +// which is the outage that defeats code with no timeout. A clean stop refuses +// connections instead and is a different fault. +func recoveryFreezeStore(t *testing.T, fix *Fixture) func() { + t.Helper() + + // Best effort per node, and never fatal. The thaw runs after a node has been + // taken away, where there is no process to signal — and a node it cannot + // reach must not stop it restoring the others, which is how a failure here + // would leave the cluster frozen for every test after it. + signal := func(sig string) { + for _, node := range fix.Cluster.Nodes { + if _, err := recoveryRunErr(node, + "PID=$(systemctl show spinifex-predastore -p MainPID --value); "+ + "if [ \"$PID\" != 0 ]; then sudo kill -"+sig+" \"$PID\"; fi"); err != nil { + t.Logf("predastore %s on %s: %v", sig, node.Name, err) + } + } + } + + var once sync.Once + thaw := func() { once.Do(func() { signal("CONT") }) } + t.Cleanup(thaw) + signal("STOP") + return thaw +} + +// recoveryGuestOnAPeer launches guests until one lands on a node other than the +// first, and returns that node and instance. A non-empty userData is delivered +// to cloud-init. +// +// Not the first node: it is the operator gateway every assertion here is made +// through, and taking it down would remove the means of observing the result +// rather than test anything. +func recoveryGuestOnAPeer(t *testing.T, fix *Fixture, userData string) (harness.Node, string) { + t.Helper() + + instType, arch := harness.DiscoverNanoInstanceType(t, fix.Harness) + amiID := harness.DiscoverUbuntuAMI(t, fix.Harness, arch) + keyName, _ := harness.EnsureKeyPair(t, fix.Harness) + def := harness.EnsureDefaultVPC(t, fix.Harness) + require.NotEmpty(t, def.SGID, "default SG required") + + in := &ec2.RunInstancesInput{ + ImageId: aws.String(amiID), + InstanceType: aws.String(instType), + KeyName: aws.String(keyName), + SubnetId: aws.String(def.SubnetID), + SecurityGroupIds: []*string{aws.String(def.SGID)}, + MinCount: aws.Int64(1), + MaxCount: aws.Int64(1), + } + if userData != "" { + in.UserData = aws.String(base64.StdEncoding.EncodeToString([]byte(userData))) + } + + for attempt := 1; attempt <= 6; attempt++ { + // A shared fixture cannot serve this: the guest has to be on a named node + // so that node can be taken away, and losing it is what is under test. + // e2e:allow-create + out, err := fix.AWS.EC2.RunInstances(in) + if err != nil { + if strings.Contains(err.Error(), "InsufficientInstanceCapacity") { + time.Sleep(10 * time.Second) + continue + } + t.Fatalf("launch a guest to recover: %v", err) + } + require.NotEmpty(t, out.Instances, "RunInstances returned no instance") + + id := aws.StringValue(out.Instances[0].InstanceId) + t.Cleanup(func() { + _, _ = fix.AWS.EC2.TerminateInstances(&ec2.TerminateInstancesInput{ + InstanceIds: []*string{aws.String(id)}, + }) + }) + harness.WaitForInstanceState(t, fix.AWS, id, "running") + + host := harness.InstanceHostingNode(t, fix.Cluster, id) + require.NotNilf(t, host, "%s is running but no node is hosting its qemu", id) + if host.Name != fix.Cluster.Nodes[0].Name { + return *host, id + } + t.Logf("recoveryGuestOnAPeer: %s landed on the gateway node, launching another", id) + } + + t.Fatalf("no guest landed on a node other than %s in six attempts", fix.Cluster.Nodes[0].Name) + return harness.Node{}, "" +} + +// recoveryWaitForDiskLoad blocks until the guest says on its console that it has +// written and synced a pass to its root volume. +// +// This is the test's own premise, and a run that froze the store before it was +// true proved nothing while reporting a product failure: the guest was still in +// UEFI, so the only thing the frozen store could reach was the EFI varstore, and +// pflash has no werror=stop to pause on. So a guest that never says it is +// writing fails here, where the message names the real problem. +func recoveryWaitForDiskLoad(t *testing.T, fix *Fixture, instanceID string) { + t.Helper() + harness.Step(t, "wait for %s to boot and start driving its root volume", instanceID) + + var last string + require.Eventuallyf(t, func() bool { + console, err := harness.InstanceConsole(fix.AWS, instanceID) + if err != nil { + return false + } + last = console + return strings.Contains(console, diskLoadMarker) + }, recoveryDiskLoadBudget, 10*time.Second, + "%s never reported %q on its console, so its load driver never wrote to the root volume "+ + "and freezing the store would prove nothing\nconsole tail:\n%s", + instanceID, diskLoadMarker, recoveryTail(last, 2000)) +} + +// recoveryTail returns the last n bytes of s, for a failure message that needs +// the end of a console rather than all of it. +func recoveryTail(s string, n int) string { + if len(s) <= n { + return s + } + return s[len(s)-n:] +} + +// recoveryGuestIsPaused reports whether the node's daemon has seen this guest +// pause on an I/O error, which is what a broken object store looks like from +// the control plane. BLOCK_IO_ERROR is the QMP event and the other is the +// daemon's own line; either is the pause, and it stamps Health.IOErrorSince. +func recoveryGuestIsPaused(t *testing.T, node harness.Node, instanceID string) bool { + t.Helper() + out, err := recoveryRunErr(node, + "sudo journalctl -u spinifex-daemon --no-pager -n 2000 | grep -F "+harness.ShellQuote(instanceID)) + if err != nil { + return false + } + return strings.Contains(out, "BLOCK_IO_ERROR") || + strings.Contains(out, "Guest paused on a backend I/O error") +} + +// recoveryWhy gathers what decides a recovery, for a failure message that does +// not send the reader to three journals to find out which step did not happen. +func recoveryWhy(t *testing.T, fix *Fixture, victim harness.Node, instanceID string) string { + t.Helper() + var b strings.Builder + for _, node := range fix.Cluster.Nodes { + if node.Name == victim.Name { + continue + } + for label, query := range map[string]string{ + "recovery log": recoveryJournalThisRun + " | grep -iE 'recovery|claim|stopped heartbeating' | tail -20", + // A claim that succeeds and then cannot mount fails in the storage + // service, and the daemon records only that the attempt did not + // complete. Without this the reason is not in the failure at all. + "storage log": recoveryStorageJournalThisRun + " | grep -iE 'lease|dirty|mount' | tail -20", + } { + out, err := recoveryRunErr(node, query) + if err != nil { + out = "(" + err.Error() + ")" + } + fmt.Fprintf(&b, " %s %s:\n%s\n", node.Name, label, strings.TrimSpace(out)) + } + } + return b.String() +} + +// lazyWhy defers a diagnostic until the assertion formats it. require.Eventuallyf +// evaluates its message arguments at the call site, so a diagnostic passed +// directly describes the moment the wait began and never the failure — which is +// how a ten-minute wait came to be explained by a log that stopped before it. +type lazyWhy func() string + +func (f lazyWhy) String() string { return f() } + +func recoveryNodeNames(nodes []harness.Node) []string { + names := make([]string, 0, len(nodes)) + for _, n := range nodes { + names = append(names, n.Name) + } + return names +} + +func recoveryRun(t *testing.T, node harness.Node, cmd string) string { + t.Helper() + out, err := recoveryRunErr(node, cmd) + require.NoErrorf(t, err, "%s on %s: %s", cmd, node.Name, out) + return out +} + +func recoveryRunErr(node harness.Node, cmd string) (string, error) { + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + out, err := harness.NewPeerSSH().Run(ctx, node.Addr, cmd) + return string(out), err +} + +// recoveryRequireStatus polls DescribeInstanceStatus until want is satisfied and +// returns the frame that satisfied it, failing the test with the last frame seen. +// +// The last frame is the whole value of this over a bare Eventually: "the status +// never became impaired" is not a finding on its own, and "it stayed ok while +// its host was gone" is. +func recoveryRequireStatus(t *testing.T, cli *harness.AWSClient, instanceID string, + want func(harness.InstanceStatusSummary) bool, budget time.Duration, describe string, +) harness.InstanceStatusSummary { + t.Helper() + harness.Step(t, "require %s reports %s", instanceID, describe) + + var last harness.InstanceStatusSummary + var lastErr error + missing := false + + require.Eventuallyf(t, func() bool { + summary, ok, err := harness.InstanceStatus(cli, instanceID) + lastErr, missing = err, !ok && err == nil + if err != nil || !ok { + return false + } + last = summary + return want(summary) + }, budget, 3*time.Second, + "%s never reported %s\nlast frame: %s\nabsent from the answer: %v\nlast error: %v", + instanceID, describe, last, missing, lastErr) + + harness.Detail(t, "instance_status", last.String()) + return last +} diff --git a/tests/e2e/lbrecovery/alb_recovery_test.go b/tests/e2e/lbrecovery/alb_recovery_test.go new file mode 100644 index 000000000..6cf162674 --- /dev/null +++ b/tests/e2e/lbrecovery/alb_recovery_test.go @@ -0,0 +1,389 @@ +//go:build e2e + +package lbrecovery + +import ( + "context" + "fmt" + "slices" + "strings" + "sync" + "testing" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/aws/aws-sdk-go/service/elbv2" + "github.com/mulgadc/spinifex/spinifex/vm" + "github.com/mulgadc/spinifex/tests/e2e/harness" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// runALBSurvivesItsHostFailing takes away the node running a load balancer and +// asserts the only things a customer of one holds: the name, the address behind +// it, and that requests to it are answered again without anybody being called. +// +// The name is the part with no second chance. An address that moves is an +// outage; a name that stops resolving, or resolves somewhere else, is an outage +// that outlives the fix for as long as anything cached the answer. So it is +// watched continuously rather than sampled at the end. +func runALBSurvivesItsHostFailing(t *testing.T, fix *Fixture) { + harness.Phase(t, "A Load Balancer Survives Its Host Failing") + + f := setupALBFixture(t, fix) + victim := f.Host + observer := lbrSurvivor(t, fix, victim) + + // Every assertion after the node goes away is made through a survivor's + // gateway, because the load balancer is free to have landed on the node this + // suite is about to take down — including the one the environment names as + // the endpoint. + cli := harness.AWSClientForGateway(t, fix.Env, observer) + harness.Detail(t, "victim_node", victim.Name, "observer_node", observer.Name) + + harness.Step(t, "prove the name serves traffic before anything is broken") + lbrRequireResolvesEverywhere(t, fix, harness.Node{}, f, "before the fault") + lbrRequireServing(t, fix, observer, harness.Node{}, f, "before the fault", 90*time.Second) + + watch := newDNSWatch(t, fix, victim, f) + defer watch.stop() + + harness.Step(t, "stop the spinifex services on %s — the load balancer guest keeps running, nothing drains it", + victim.Name) + harness.StopNode(t, victim) + t.Cleanup(func() { harness.StartNode(t, victim) }) + + fix.Cluster.WaitNATSPeers(t, len(fix.Cluster.Nodes)-2, harness.WithTimeout(90*time.Second), + harness.WithPoll(2*time.Second), harness.WithSkipNodes(victim.Name)) + + // AWS never reports a load balancer as degraded because a host under it + // failed — the service is the abstraction, and its state describes the + // customer's configuration, not our hardware. So active is the right answer + // here, and the honest signal about the outage is target health and the + // traffic itself, both asserted below. + harness.Step(t, "the load balancer record survives the outage unchanged") + lbrRequireLBRecord(t, cli, f) + + harness.Step(t, "wait for a survivor to claim and relaunch the load balancer") + var hosting []harness.Node + require.Eventuallyf(t, func() bool { + hosting = harness.InstanceHostingNodes(t, fix.Cluster, f.SysInstanceID, []string{victim.Name}) + return len(hosting) > 0 + }, albRecoveryBudget, 15*time.Second, + "the load balancer's guest %s never came back on a survivor after %s stopped answering\n%s", + f.SysInstanceID, victim.Name, lbrWhy(t, fix, victim, f)) + + require.Lenf(t, hosting, 1, "the load balancer is running on more than one survivor: %v", + lbrNodeNames(hosting)) + harness.Detail(t, "recovered_onto", hosting[0].Name) + + // The address is the part a customer may have written down, put in a + // firewall rule, or handed to a partner. A recovery that renumbers it has + // moved the outage rather than ended it, so this is asserted before any + // traffic — a passing probe against a new address would hide it. + harness.Step(t, "the address did not move with the node") + lbrRequireAddressUnchanged(t, cli, f) + + harness.Step(t, "wait for the name to serve traffic again") + lbrRequireServing(t, fix, observer, victim, f, "after the recovery", albServingBudget) + + // The backends never moved, so anything unhealthy here is the recovered load + // balancer's view of them rather than the backends themselves. + harness.WaitForTargetsHealthy(t, cli, f.TGArn, lbrTargets, "backends after the recovery", targetsHealthyBudget) + + harness.Step(t, "bring %s back and assert it gives up what it no longer owns", victim.Name) + harness.StartNode(t, victim) + harness.WaitNodeServiceReady(t, victim, harness.WithTimeout(3*time.Minute)) + fix.Cluster.WaitNATSPeers(t, len(fix.Cluster.Nodes)-1, harness.WithTimeout(3*time.Minute), + harness.WithPoll(2*time.Second)) + + // Long enough for the returning node to have got it wrong. Its restore runs + // early in startup, so a second copy of the guest, or a tap still claiming + // the logical port, would already be visible. + time.Sleep(90 * time.Second) + + after := harness.InstanceHostingNodes(t, fix.Cluster, f.SysInstanceID, nil) + require.Lenf(t, after, 1, + "the load balancer is running on %v after %s returned; a returning node must drop a local copy the records say moved", + lbrNodeNames(after), victim.Name) + assert.NotEqualf(t, victim.Name, after[0].Name, + "the load balancer went back to %s, so the returning node relaunched it rather than forgetting it", victim.Name) + + lbrRequireNoStaleTap(t, victim, f) + + // Last, and the assertion the returning node exists in this test for. Its + // tap carried the load balancer's logical port, and the column that binds a + // port names one chassis — so a returning node still offering that port does + // not split the traffic, it contends for it, and the address answers from + // whichever won last. Serving after the return is what says it did not. + harness.Step(t, "the name still serves traffic with %s back in the cluster", victim.Name) + lbrRequireServing(t, fix, observer, harness.Node{}, f, "after the node returned", 3*time.Minute) + + watch.stop() + watch.requireNameNeverBroke(t, f) +} + +// lbrSurvivor returns a node that is not the victim, for the gateway every +// assertion during the outage is made through. +func lbrSurvivor(t *testing.T, fix *Fixture, victim harness.Node) harness.Node { + t.Helper() + for _, node := range fix.Cluster.Nodes { + if node.Name != victim.Name { + return node + } + } + t.Fatalf("no node other than %s to observe from", victim.Name) + return harness.Node{} +} + +// lbrPeers is every node but victim. +func lbrPeers(fix *Fixture, victim harness.Node) []harness.Node { + out := make([]harness.Node, 0, len(fix.Cluster.Nodes)-1) + for _, node := range fix.Cluster.Nodes { + if node.Name != victim.Name { + out = append(out, node) + } + } + return out +} + +func lbrNodeNames(nodes []harness.Node) []string { + names := make([]string, 0, len(nodes)) + for _, n := range nodes { + names = append(names, n.Name) + } + return names +} + +// lbrRequireResolvesEverywhere asserts every surviving node's northstar answers +// the load balancer's name with its address. +// +// Every node, not one: the zone is cluster-wide and a client may ask any of +// them, so a record served by one node and missing on another is a customer who +// resolves it or not depending on which server they reached. +func lbrRequireResolvesEverywhere(t *testing.T, fix *Fixture, skip harness.Node, f *albFixture, when string) { + t.Helper() + for _, node := range lbrPeers(fix, skip) { + n := node + var last []string + var lastErr error + require.Eventuallyf(t, func() bool { + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + last, lastErr = harness.ResolveViaNode(ctx, n, f.DNSName) + return lastErr == nil && slices.Contains(last, f.PublicIP) + }, 90*time.Second, 3*time.Second, + "%s's northstar does not answer %s with %s %s (last=%v err=%v)", + n.Name, f.DNSName, f.PublicIP, when, last, lastErr) + } + harness.Detail(t, "dns_name", f.DNSName, "resolves_to", f.PublicIP) +} + +// lbrRequireServing drives HTTP through the load balancer's name until both +// backends answer, resolving the name through a node's own northstar. +// +// Through the name rather than the address, because that is what a customer +// uses, and through a node's resolver rather than the runner's, because the +// runner's is not part of this product. +func lbrRequireServing(t *testing.T, fix *Fixture, via, down harness.Node, f *albFixture, when string, budget time.Duration) { + t.Helper() + url := fmt.Sprintf("http://%s:%d/", f.DNSName, lbrHTTPPort) + client := harness.NodeResolverHTTPClient(via, 5*time.Second) + + var last harness.TrafficResult + require.Eventuallyf(t, func() bool { + last = harness.HTTPRoundRobinWithClient(client, url, lbrProbes) + return last.Unique() >= lbrTargets && last.Successful >= lbrProbes/2 + }, budget, 10*time.Second, + "%s did not serve traffic from %d backends %s: %d/%d probes succeeded across %d responders %v\n%s", + f.DNSName, lbrTargets, when, last.Successful, last.Total, last.Unique(), last.Distribution, + lbrWhy(t, fix, down, f)) + + harness.AssertRoundRobin(t, last, lbrTargets, lbrProbes/2, "ALB via its name "+when) +} + +// lbrRequireLBRecord asserts the load balancer still describes itself the same +// way while its host is gone: same name, same address, still active. +func lbrRequireLBRecord(t *testing.T, cli *harness.AWSClient, f *albFixture) { + t.Helper() + out, err := cli.ELBv2.DescribeLoadBalancers(&elbv2.DescribeLoadBalancersInput{ + LoadBalancerArns: []*string{aws.String(f.LBArn)}, + }) + require.NoError(t, err, "describe the load balancer through a survivor") + require.NotEmpty(t, out.LoadBalancers, "the load balancer vanished from the API when its host did") + + lb := out.LoadBalancers[0] + assert.Equalf(t, f.DNSName, aws.StringValue(lb.DNSName), + "the load balancer's name changed while its host was gone, so every client holding the old one is stranded") + assert.Equalf(t, "active", aws.StringValue(lb.State.Code), + "the load balancer reports state %q while a host failure is being recovered; a customer's configuration did not change", + aws.StringValue(lb.State.Code)) +} + +// lbrRequireAddressUnchanged asserts the recovered load balancer answers on the +// address it was created with, and is the same instance rather than a new one. +func lbrRequireAddressUnchanged(t *testing.T, cli *harness.AWSClient, f *albFixture) { + t.Helper() + out, err := cli.EC2.DescribeNetworkInterfaces(&ec2.DescribeNetworkInterfacesInput{ + NetworkInterfaceIds: []*string{aws.String(f.ENIID)}, + }) + require.NoError(t, err, "describe the load balancer's interface") + require.NotEmptyf(t, out.NetworkInterfaces, "the load balancer's interface %s is gone", f.ENIID) + + eni := out.NetworkInterfaces[0] + require.NotNilf(t, eni.Association, "%s no longer has a public address at all", f.ENIID) + assert.Equalf(t, f.PublicIP, aws.StringValue(eni.Association.PublicIp), + "the load balancer came back on %s having been reachable on %s; an address a customer may have written down "+ + "must survive a host failure", aws.StringValue(eni.Association.PublicIp), f.PublicIP) + if eni.Attachment != nil { + assert.Equalf(t, f.SysInstanceID, aws.StringValue(eni.Attachment.InstanceId), + "the interface is attached to %s rather than the load balancer's own guest, so this is a different load balancer", + aws.StringValue(eni.Attachment.InstanceId)) + } +} + +// lbrRequireNoStaleTap asserts the returning node no longer offers the load +// balancer's logical port. +// +// This is the mechanism behind the assertion above it. OVN binds a logical port +// to one chassis, and a tap carrying that port's iface-id is how a node claims +// it, so a returning node that kept its tap is a second claimant — and it is +// also what keeps the external address's host ingress from being pruned, since +// that prune waits on evidence the guest sits elsewhere. +func lbrRequireNoStaleTap(t *testing.T, victim harness.Node, f *albFixture) { + t.Helper() + tap := vm.TapDeviceName(f.ENIID) + harness.Step(t, "%s must no longer carry %s, the tap that claims the load balancer's port", victim.Name, tap) + + out, err := lbrRunErr(victim, "ip -o link show "+harness.ShellQuote(tap)+" 2>/dev/null || true") + require.NoErrorf(t, err, "ask %s about %s: %s", victim.Name, tap, out) + assert.Emptyf(t, strings.TrimSpace(out), + "%s still carries %s after the load balancer moved, so two nodes are claiming one logical port and the "+ + "address answers from whichever won last:\n%s", victim.Name, tap, out) +} + +// dnsWatch samples what every surviving node's northstar answers for the load +// balancer's name, continuously, for as long as the scenario lasts. +// +// The point-in-time checks answer "does the name resolve now". This answers "did +// it ever stop, or ever point somewhere else", which is the question a customer +// with a cached answer is really asking, and the one no single sample reaches. +type dnsWatch struct { + stopOnce sync.Once + done chan struct{} + finished chan struct{} + + mu sync.Mutex + samples int + missing int + wrong map[string]int + errs map[string]int +} + +func newDNSWatch(t *testing.T, fix *Fixture, victim harness.Node, f *albFixture) *dnsWatch { + t.Helper() + w := &dnsWatch{ + done: make(chan struct{}), + finished: make(chan struct{}), + wrong: map[string]int{}, + errs: map[string]int{}, + } + peers := lbrPeers(fix, victim) + + go func() { + defer close(w.finished) + ticker := time.NewTicker(dnsWatchInterval) + defer ticker.Stop() + for { + select { + case <-w.done: + return + case <-ticker.C: + for _, node := range peers { + w.sample(node, f) + } + } + } + }() + return w +} + +// sample records one node's answer. A node that cannot be reached is counted +// separately from one that answers the wrong thing: the first is this harness +// losing contact, the second is the product being wrong. +func (w *dnsWatch) sample(node harness.Node, f *albFixture) { + ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second) + defer cancel() + addrs, err := harness.ResolveViaNode(ctx, node, f.DNSName) + + w.mu.Lock() + defer w.mu.Unlock() + w.samples++ + switch { + case err != nil: + w.errs[node.Name]++ + case len(addrs) == 0: + w.missing++ + case !slices.Contains(addrs, f.PublicIP): + w.wrong[node.Name+" -> "+strings.Join(addrs, ",")]++ + } +} + +func (w *dnsWatch) stop() { + w.stopOnce.Do(func() { + close(w.done) + <-w.finished + }) +} + +// requireNameNeverBroke is the assertion the watch exists for. An answer naming +// another address is a hard failure: it sends traffic somewhere that is not this +// load balancer. An empty answer, or a query the node refused, is the same +// outage from the client's side, so both are failures too — and they are +// reported apart because they point at different code. +func (w *dnsWatch) requireNameNeverBroke(t *testing.T, f *albFixture) { + t.Helper() + w.mu.Lock() + defer w.mu.Unlock() + + require.Positivef(t, w.samples, "the DNS watch took no samples, so it proved nothing about the name") + harness.Detail(t, "dns_samples", fmt.Sprintf("%d", w.samples)) + + require.Emptyf(t, w.wrong, + "%s resolved to an address that is not the load balancer's %s during the outage: %v; "+ + "a name pointing somewhere else is worse than one that does not resolve, because the traffic goes there", + f.DNSName, f.PublicIP, w.wrong) + require.Zerof(t, w.missing, + "%s resolved to nothing in %d of %d samples; the record must survive the loss of the node the load balancer was on", + f.DNSName, w.missing, w.samples) + require.Emptyf(t, w.errs, + "a surviving node's northstar refused to answer for %s during the outage: %v; "+ + "the zone is cluster-wide, so a client reaching that node gets no address at all", + f.DNSName, w.errs) +} + +// lbrWhy gathers what decides a load balancer recovery, for a failure message +// that does not send the reader to three journals to find which step did not +// happen. +func lbrWhy(t *testing.T, fix *Fixture, skip harness.Node, f *albFixture) string { + t.Helper() + var b strings.Builder + for _, node := range lbrPeers(fix, skip) { + out, err := lbrRunErr(node, + "sudo journalctl -u spinifex-daemon --no-pager -n 800 | "+ + "grep -iE 'recovery|claim|stopped heartbeating|"+f.SysInstanceID+"|elb|haproxy' | tail -25") + if err != nil { + out = "(" + err.Error() + ")" + } + fmt.Fprintf(&b, " %s:\n%s\n", node.Name, strings.TrimSpace(out)) + } + return b.String() +} + +func lbrRunErr(node harness.Node, cmd string) (string, error) { + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + out, err := harness.NewPeerSSH().Run(ctx, node.Addr, cmd) + return string(out), err +} diff --git a/tests/e2e/lbrecovery/fixture_test.go b/tests/e2e/lbrecovery/fixture_test.go new file mode 100644 index 000000000..723b14ec7 --- /dev/null +++ b/tests/e2e/lbrecovery/fixture_test.go @@ -0,0 +1,507 @@ +//go:build e2e + +package lbrecovery + +import ( + _ "embed" + "encoding/base64" + "errors" + "fmt" + "strings" + "testing" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/aws/aws-sdk-go/service/elbv2" + "github.com/mulgadc/spinifex/tests/e2e/harness" + "github.com/stretchr/testify/require" +) + +const ( + // Its own CIDR, not the lb suite's 10.200.0.0/16: a run that overlapped one + // left behind by a failed lb run would fail on the create rather than on + // anything this suite is about. + lbrVPCCIDR = "10.210.0.0/16" + lbrSubnetCIDR = "10.210.1.0/24" + + lbrLBName = "lbr-e2e-alb" + lbrTGName = "lbr-e2e-tg" + lbrHTTPPort = int64(80) + + // Two targets, on nodes other than the ALB's. One would not distinguish + // "the ALB is serving" from "the ALB is serving the one backend it has". + lbrTargets = 2 + + // Probes per round. Enough that a round-robin across two backends is + // unambiguous, small enough that a round costs seconds rather than a minute + // while a recovery is in flight. + lbrProbes = 12 + + // How many times an LB that lands in a terminal state is recreated before + // the suite gives up, matching the lb suite: on a loaded cluster this is + // usually the previous run's sys.micro slot still being reclaimed. + lbrLBAttempts = 3 +) + +//go:embed testdata/target-userdata.sh +var targetUserData string + +// albFixture is the topology the scenario needs: one internet-facing ALB with a +// node to itself, and its backends on other nodes, so taking the ALB's node away +// leaves the backends untouched and the only thing that has to move is the ALB. +type albFixture struct { + VPCID, SubnetID, IGWID, RouteTableID string + SGID, LBSGID string + AMIID, InstanceType, KeyName string + + TGArn, LBArn, LBID, ListenerArn string + DNSName, PublicIP string + ENIID, SysInstanceID string + Host harness.Node + + TargetIDs []string +} + +// setupALBFixture builds the VPC, the ALB, and then the targets — in that order +// because the ALB's node is not chosen by the test, it is discovered, and the +// targets are the half of the topology that can be placed around it. +func setupALBFixture(t *testing.T, fix *Fixture) *albFixture { + t.Helper() + f := &albFixture{} + + harness.Phase(t, "Building the ALB fixture") + instanceType, arch := harness.DiscoverNanoInstanceType(t, fix.Harness) + f.InstanceType = instanceType + f.AMIID = harness.DiscoverUbuntuAMI(t, fix.Harness, arch) + f.KeyName, _ = harness.EnsureKeyPair(t, fix.Harness) + + lbrCreateVPC(t, fix.AWS, f) + lbrCreateIGW(t, fix.AWS, f) + lbrCreateSubnet(t, fix.AWS, f) + lbrAddPublicRoute(t, fix.AWS, f) + lbrOpenDefaultSG(t, fix.AWS, f) + + f.TGArn = lbrCreateTargetGroup(t, fix.AWS, f) + lbrCreateActiveLB(t, fix.AWS, f) + lbrDiscoverLBPlacement(t, fix, f) + + lbrLaunchTargetsAvoiding(t, fix, f) + lbrRegisterTargets(t, fix.AWS, f) + harness.WaitForTargetsHealthy(t, fix.AWS, f.TGArn, lbrTargets, "ALB targets", targetsHealthyBudget) + + return f +} + +func lbrCreateVPC(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + // The ALB needs a public subnet of its own, and a shared fixture VPC cannot + // be given one without changing what every other suite sees. + // e2e:allow-create + out, err := c.EC2.CreateVpc(&ec2.CreateVpcInput{CidrBlock: aws.String(lbrVPCCIDR)}) + require.NoError(t, err, "create VPC %s", lbrVPCCIDR) + f.VPCID = aws.StringValue(out.Vpc.VpcId) + harness.Detail(t, "vpc", f.VPCID) + t.Cleanup(func() { + if _, err := c.EC2.DeleteVpc(&ec2.DeleteVpcInput{VpcId: aws.String(f.VPCID)}); err != nil { + t.Logf("delete VPC %s: %v", f.VPCID, err) + } + }) +} + +func lbrCreateIGW(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + // Belongs to this suite's own VPC. + // e2e:allow-create + out, err := c.EC2.CreateInternetGateway(&ec2.CreateInternetGatewayInput{}) + require.NoError(t, err, "create IGW") + f.IGWID = aws.StringValue(out.InternetGateway.InternetGatewayId) + _, err = c.EC2.AttachInternetGateway(&ec2.AttachInternetGatewayInput{ + InternetGatewayId: aws.String(f.IGWID), + VpcId: aws.String(f.VPCID), + }) + require.NoError(t, err, "attach IGW %s to %s", f.IGWID, f.VPCID) + t.Cleanup(func() { + _, _ = c.EC2.DetachInternetGateway(&ec2.DetachInternetGatewayInput{ + InternetGatewayId: aws.String(f.IGWID), + VpcId: aws.String(f.VPCID), + }) + if _, err := c.EC2.DeleteInternetGateway(&ec2.DeleteInternetGatewayInput{ + InternetGatewayId: aws.String(f.IGWID), + }); err != nil { + t.Logf("delete IGW %s: %v", f.IGWID, err) + } + }) +} + +func lbrCreateSubnet(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + // See lbrCreateVPC. + // e2e:allow-create + out, err := c.EC2.CreateSubnet(&ec2.CreateSubnetInput{ + VpcId: aws.String(f.VPCID), + CidrBlock: aws.String(lbrSubnetCIDR), + }) + require.NoError(t, err, "create subnet %s", lbrSubnetCIDR) + f.SubnetID = aws.StringValue(out.Subnet.SubnetId) + _, err = c.EC2.ModifySubnetAttribute(&ec2.ModifySubnetAttributeInput{ + SubnetId: aws.String(f.SubnetID), + MapPublicIpOnLaunch: &ec2.AttributeBooleanValue{Value: aws.Bool(true)}, + }) + require.NoError(t, err, "set MapPublicIpOnLaunch on %s", f.SubnetID) + harness.Detail(t, "subnet", f.SubnetID) + t.Cleanup(func() { + if _, err := c.EC2.DeleteSubnet(&ec2.DeleteSubnetInput{SubnetId: aws.String(f.SubnetID)}); err != nil { + t.Logf("delete subnet %s: %v", f.SubnetID, err) + } + }) +} + +// lbrAddPublicRoute installs 0.0.0.0/0 to the IGW on the main route table, which +// is what makes the daemon classify the subnet as public. Without it the logical +// router gets an egress DROP and the ALB is unreachable for a reason that has +// nothing to do with recovery. +func lbrAddPublicRoute(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + out, err := c.EC2.DescribeRouteTables(&ec2.DescribeRouteTablesInput{ + Filters: []*ec2.Filter{ + {Name: aws.String("vpc-id"), Values: []*string{aws.String(f.VPCID)}}, + {Name: aws.String("association.main"), Values: []*string{aws.String("true")}}, + }, + }) + require.NoError(t, err, "describe main route table") + require.NotEmpty(t, out.RouteTables, "VPC %s has no main route table", f.VPCID) + f.RouteTableID = aws.StringValue(out.RouteTables[0].RouteTableId) + _, err = c.EC2.CreateRoute(&ec2.CreateRouteInput{ + RouteTableId: aws.String(f.RouteTableID), + DestinationCidrBlock: aws.String("0.0.0.0/0"), + GatewayId: aws.String(f.IGWID), + }) + require.NoError(t, err, "create default route to %s", f.IGWID) + t.Cleanup(func() { + if _, err := c.EC2.DeleteRoute(&ec2.DeleteRouteInput{ + RouteTableId: aws.String(f.RouteTableID), + DestinationCidrBlock: aws.String("0.0.0.0/0"), + }); err != nil { + t.Logf("delete default route: %v", err) + } + }) +} + +// lbrOpenDefaultSG opens HTTP on the VPC's default security group, which is what +// the targets get, and mints a separate one for the ALB itself. +// +// The structured IpPermissions form is required: vpcd ignores the top-level +// shorthand fields, so the OVN ACL would never be installed. +func lbrOpenDefaultSG(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + out, err := c.EC2.DescribeSecurityGroups(&ec2.DescribeSecurityGroupsInput{ + Filters: []*ec2.Filter{ + {Name: aws.String("vpc-id"), Values: []*string{aws.String(f.VPCID)}}, + {Name: aws.String("group-name"), Values: []*string{aws.String("default")}}, + }, + }) + require.NoError(t, err, "describe default SG") + require.NotEmpty(t, out.SecurityGroups, "VPC %s has no default SG", f.VPCID) + f.SGID = aws.StringValue(out.SecurityGroups[0].GroupId) + lbrAuthorizeHTTP(t, c, f.SGID) + + // The ALB's own group, so a rule change on the targets cannot silently + // change what reaches the load balancer. + // e2e:allow-create + sg, err := c.EC2.CreateSecurityGroup(&ec2.CreateSecurityGroupInput{ + VpcId: aws.String(f.VPCID), + GroupName: aws.String("lbrecovery-e2e-lb-sg"), + Description: aws.String("lbrecovery e2e ALB-facing security group"), + }) + require.NoError(t, err, "create LB SG") + f.LBSGID = aws.StringValue(sg.GroupId) + lbrAuthorizeHTTP(t, c, f.LBSGID) + t.Cleanup(func() { + if _, err := c.EC2.DeleteSecurityGroup(&ec2.DeleteSecurityGroupInput{ + GroupId: aws.String(f.LBSGID), + }); err != nil { + t.Logf("delete LB SG %s: %v", f.LBSGID, err) + } + }) +} + +func lbrAuthorizeHTTP(t *testing.T, c *harness.AWSClient, sgID string) { + t.Helper() + _, err := c.EC2.AuthorizeSecurityGroupIngress(&ec2.AuthorizeSecurityGroupIngressInput{ + GroupId: aws.String(sgID), + IpPermissions: []*ec2.IpPermission{{ + IpProtocol: aws.String("tcp"), + FromPort: aws.Int64(lbrHTTPPort), + ToPort: aws.Int64(lbrHTTPPort), + IpRanges: []*ec2.IpRange{{CidrIp: aws.String("0.0.0.0/0")}}, + }}, + }) + if err == nil { + return + } + var aerr awserr.Error + if errors.As(err, &aerr) && aerr.Code() == "InvalidPermission.Duplicate" { + return + } + t.Fatalf("authorize tcp/%d on %s: %v", lbrHTTPPort, sgID, err) +} + +func lbrCreateTargetGroup(t *testing.T, c *harness.AWSClient, f *albFixture) string { + t.Helper() + // e2e:allow-create + out, err := c.ELBv2.CreateTargetGroup(&elbv2.CreateTargetGroupInput{ + Name: aws.String(lbrTGName), + Protocol: aws.String("HTTP"), + Port: aws.Int64(lbrHTTPPort), + VpcId: aws.String(f.VPCID), + HealthCheckPath: aws.String("/index.html"), + HealthCheckIntervalSeconds: aws.Int64(5), + HealthyThresholdCount: aws.Int64(2), + UnhealthyThresholdCount: aws.Int64(2), + }) + require.NoError(t, err, "create target group") + arn := aws.StringValue(out.TargetGroups[0].TargetGroupArn) + harness.Detail(t, "target_group", arn) + t.Cleanup(func() { + if _, err := c.ELBv2.DeleteTargetGroup(&elbv2.DeleteTargetGroupInput{ + TargetGroupArn: aws.String(arn), + }); err != nil { + t.Logf("delete target group: %v", err) + } + }) + return arn +} + +// lbrCreateActiveLB creates an internet-facing ALB with an HTTP listener and +// waits for it to go active, recreating it when it lands in a terminal state. +func lbrCreateActiveLB(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + var lastErr error + for attempt := 1; attempt <= lbrLBAttempts; attempt++ { + if attempt > 1 { + wait := time.Duration(attempt-1) * 15 * time.Second + harness.Step(t, "waiting %s before recreating the ALB", wait) + time.Sleep(wait) + } + lbrCreateLB(t, c, f) + lbrCreateListener(t, c, f) + lastErr = harness.WaitForLBActiveErr(t, c, f.LBArn, "recovery ALB", albActiveBudget) + if lastErr == nil { + harness.Detail(t, "load_balancer", f.LBArn, "dns_name", f.DNSName) + return + } + if !errors.Is(lastErr, harness.ErrLBTerminalFailed) && !errors.Is(lastErr, harness.ErrLBProvisioningTimeout) { + t.Fatalf("ALB never became active: %v", lastErr) + } + t.Logf("ALB attempt %d/%d: %v — tearing down and retrying", attempt, lbrLBAttempts, lastErr) + lbrDeleteListener(t, c, f) + lbrDeleteLB(t, c, f) + } + t.Fatalf("ALB never became active in %d attempts: %v", lbrLBAttempts, lastErr) +} + +func lbrCreateLB(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + // The load balancer's own placement and recovery are the subject, so it can + // never be one another test is sharing. + // e2e:allow-create + out, err := c.ELBv2.CreateLoadBalancer(&elbv2.CreateLoadBalancerInput{ + Name: aws.String(lbrLBName), + Subnets: []*string{aws.String(f.SubnetID)}, + Scheme: aws.String("internet-facing"), + SecurityGroups: []*string{aws.String(f.LBSGID)}, + }) + require.NoError(t, err, "create load balancer") + require.NotEmpty(t, out.LoadBalancers, "CreateLoadBalancer returned nothing") + lb := out.LoadBalancers[0] + f.LBArn = aws.StringValue(lb.LoadBalancerArn) + f.DNSName = aws.StringValue(lb.DNSName) + parts := strings.Split(f.LBArn, "/") + f.LBID = parts[len(parts)-1] + t.Cleanup(func() { lbrDeleteLB(t, c, f) }) +} + +func lbrDeleteLB(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + if f.LBArn == "" { + return + } + if _, err := c.ELBv2.DeleteLoadBalancer(&elbv2.DeleteLoadBalancerInput{ + LoadBalancerArn: aws.String(f.LBArn), + }); err != nil { + t.Logf("delete LB %s: %v", f.LBArn, err) + } +} + +func lbrCreateListener(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + // e2e:allow-create + out, err := c.ELBv2.CreateListener(&elbv2.CreateListenerInput{ + LoadBalancerArn: aws.String(f.LBArn), + Protocol: aws.String("HTTP"), + Port: aws.Int64(lbrHTTPPort), + DefaultActions: []*elbv2.Action{{ + Type: aws.String("forward"), + TargetGroupArn: aws.String(f.TGArn), + }}, + }) + require.NoError(t, err, "create listener") + f.ListenerArn = aws.StringValue(out.Listeners[0].ListenerArn) + t.Cleanup(func() { lbrDeleteListener(t, c, f) }) +} + +func lbrDeleteListener(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + if f.ListenerArn == "" { + return + } + if _, err := c.ELBv2.DeleteListener(&elbv2.DeleteListenerInput{ + ListenerArn: aws.String(f.ListenerArn), + }); err != nil { + t.Logf("delete listener: %v", err) + } +} + +// lbrDiscoverLBPlacement records where the ALB actually is, which is not the +// test's choice: a load balancer is launched over a queue group and lands on +// whichever node answers. Everything after this is arranged around the answer. +// +// The ENI is how the guest is found at all — a system instance is filtered out +// of DescribeInstances, so its id comes from the attachment on the ENI the LB +// describes itself by. +func lbrDiscoverLBPlacement(t *testing.T, fix *Fixture, f *albFixture) { + t.Helper() + desc := fmt.Sprintf("ELB app/%s/%s", lbrLBName, f.LBID) + + var eni *ec2.NetworkInterface + harness.EventuallyErr(t, func() error { + out, err := fix.AWS.EC2.DescribeNetworkInterfaces(&ec2.DescribeNetworkInterfacesInput{ + Filters: []*ec2.Filter{{ + Name: aws.String("description"), + Values: []*string{aws.String(desc)}, + }}, + }) + if err != nil { + return err + } + if len(out.NetworkInterfaces) == 0 { + return fmt.Errorf("no ENI described as %q", desc) + } + eni = out.NetworkInterfaces[0] + if eni.Attachment == nil || aws.StringValue(eni.Attachment.InstanceId) == "" { + return fmt.Errorf("ENI %s is not attached to an instance yet", aws.StringValue(eni.NetworkInterfaceId)) + } + if eni.Association == nil || aws.StringValue(eni.Association.PublicIp) == "" { + return fmt.Errorf("ENI %s has no public address yet", aws.StringValue(eni.NetworkInterfaceId)) + } + return nil + }, 60*time.Second, 3*time.Second) + + f.ENIID = aws.StringValue(eni.NetworkInterfaceId) + f.SysInstanceID = aws.StringValue(eni.Attachment.InstanceId) + f.PublicIP = aws.StringValue(eni.Association.PublicIp) + + host := harness.InstanceHostingNode(t, fix.Cluster, f.SysInstanceID) + require.NotNilf(t, host, "the ALB is active but no node is running %s, so there is nothing to take away", + f.SysInstanceID) + f.Host = *host + + harness.Detail(t, "lb_instance", f.SysInstanceID, "lb_eni", f.ENIID, + "lb_public_ip", f.PublicIP, "lb_node", f.Host.Name) +} + +// lbrLaunchTargetsAvoiding launches the backends on nodes other than the ALB's, +// relaunching one that lands beside it. +// +// The whole point of the topology: a backend on the node that is about to be +// taken away would be recovered too, so the run could not tell an ALB that came +// back serving from one that came back with nothing to serve. +func lbrLaunchTargetsAvoiding(t *testing.T, fix *Fixture, f *albFixture) { + t.Helper() + harness.Step(t, "launch %d backends on nodes other than %s", lbrTargets, f.Host.Name) + + for len(f.TargetIDs) < lbrTargets { + placed := false + for attempt := 1; attempt <= 4 && !placed; attempt++ { + id := lbrLaunchTarget(t, fix, f) + harness.WaitForInstanceRunning(t, fix.AWS, id, 3*time.Minute) + host := harness.InstanceHostingNode(t, fix.Cluster, id) + if host != nil && host.Name != f.Host.Name { + harness.Detail(t, "target", id, "target_node", host.Name) + f.TargetIDs = append(f.TargetIDs, id) + placed = true + continue + } + where := "nowhere" + if host != nil { + where = host.Name + } + t.Logf("target %s landed on %s, which is the ALB's node — terminating and launching another", id, where) + lbrTerminate(t, fix.AWS, []string{id}) + } + require.Truef(t, placed, + "no backend landed on a node other than %s in four attempts, so this cluster cannot host the topology "+ + "this scenario needs", f.Host.Name) + } +} + +func lbrLaunchTarget(t *testing.T, fix *Fixture, f *albFixture) string { + t.Helper() + // A backend has to be placed relative to the ALB's node, which a shared + // instance fixture cannot express. + // e2e:allow-create + out, err := fix.AWS.EC2.RunInstances(&ec2.RunInstancesInput{ + ImageId: aws.String(f.AMIID), + InstanceType: aws.String(f.InstanceType), + KeyName: aws.String(f.KeyName), + SubnetId: aws.String(f.SubnetID), + SecurityGroupIds: []*string{aws.String(f.SGID)}, + MinCount: aws.Int64(1), + MaxCount: aws.Int64(1), + UserData: aws.String(base64.StdEncoding.EncodeToString([]byte(targetUserData))), + }) + require.NoError(t, err, "launch a backend") + require.NotEmpty(t, out.Instances, "RunInstances returned no instance") + id := aws.StringValue(out.Instances[0].InstanceId) + t.Cleanup(func() { lbrTerminate(t, fix.AWS, []string{id}) }) + return id +} + +func lbrTerminate(t *testing.T, c *harness.AWSClient, ids []string) { + t.Helper() + if len(ids) == 0 { + return + } + awsIDs := make([]*string, len(ids)) + for i, id := range ids { + awsIDs[i] = aws.String(id) + } + if _, err := c.EC2.TerminateInstances(&ec2.TerminateInstancesInput{InstanceIds: awsIDs}); err != nil { + t.Logf("terminate %v: %v", ids, err) + return + } + harness.WaitForInstanceTerminated(t, c, ids, 2*time.Minute) +} + +func lbrRegisterTargets(t *testing.T, c *harness.AWSClient, f *albFixture) { + t.Helper() + targets := make([]*elbv2.TargetDescription, len(f.TargetIDs)) + for i, id := range f.TargetIDs { + targets[i] = &elbv2.TargetDescription{Id: aws.String(id)} + } + _, err := c.ELBv2.RegisterTargets(&elbv2.RegisterTargetsInput{ + TargetGroupArn: aws.String(f.TGArn), + Targets: targets, + }) + require.NoError(t, err, "register targets") + t.Cleanup(func() { + if _, err := c.ELBv2.DeregisterTargets(&elbv2.DeregisterTargetsInput{ + TargetGroupArn: aws.String(f.TGArn), + Targets: targets, + }); err != nil { + t.Logf("deregister targets: %v", err) + } + }) +} diff --git a/tests/e2e/lbrecovery/main_test.go b/tests/e2e/lbrecovery/main_test.go new file mode 100644 index 000000000..85dc45c78 --- /dev/null +++ b/tests/e2e/lbrecovery/main_test.go @@ -0,0 +1,149 @@ +//go:build e2e + +// Package lbrecovery proves what a customer sees when the node under a load +// balancer fails: the name keeps resolving, the address does not move, and +// traffic comes back on its own. +// +// It is its own suite for the reason instancerecovery is. It takes a node away, +// so anything sharing the environment would see the same outage and report a +// failure that says nothing about itself; and it needs its own VPC, its own +// public subnet and backends placed relative to the load balancer's node, which +// is a topology no shared fixture describes. A failure here also has to be +// readable as "the load balancer did not survive its host" rather than as one +// more red subtest in a suite about something else. +package lbrecovery + +import ( + "bytes" + "errors" + "fmt" + "os" + "path/filepath" + "sync" + "testing" + "time" + + "github.com/mulgadc/spinifex/tests/e2e/harness" +) + +const ( + // How long the ALB is given to reach active on a healthy cluster. A system + // instance has to be launched, booted, and reported before anything here + // starts. + albActiveBudget = 5 * time.Minute + + // How long the backends are given to pass their health checks, on a 5s + // interval with a healthy threshold of 2. + targetsHealthyBudget = 3 * time.Minute + + // How long a survivor is given to claim and relaunch the ALB. The owner has + // to be seen stale across two passes, the volume leases have to expire, and + // then a guest has to boot — the same arithmetic as an ordinary instance + // recovery, with HAProxy's own start on top. + albRecoveryBudget = 10 * time.Minute + + // How long traffic is given to come back once a survivor is running the ALB. + // The load balancer has to re-establish its own health checks against the + // backends before it will forward anything. + albServingBudget = 6 * time.Minute + + // How often the DNS watch samples every surviving node. The record is the + // only thing a customer holds, so the sampling is fast enough that a gap + // lasting one reconcile pass would be seen. + dnsWatchInterval = 5 * time.Second +) + +// errSkipDevNetworking marks an environment where the scenario has nothing to +// assert: a dev-networking cluster allocates no external addresses, so an +// internet-facing load balancer has no address to keep across a recovery. +var errSkipDevNetworking = errors.New("dev networking") + +// devNetworking reports whether this cluster is configured for dev networking. +func devNetworking(env *harness.Env) bool { + if env == nil || env.ConfigDir == "" { + return false + } + raw, err := os.ReadFile(filepath.Join(env.ConfigDir, "spinifex.toml")) + if err != nil { + return false + } + return bytes.Contains(raw, []byte("dev_networking = true")) +} + +var ( + pkgFixOnce sync.Once + pkgFix *Fixture + pkgFixErr error +) + +func TestMain(m *testing.M) { + code := m.Run() + if pkgFix != nil && pkgFix.Harness != nil { + if err := pkgFix.Harness.Close(); err != nil { + fmt.Fprintf(os.Stderr, "e2e teardown: %v\n", err) + code = 1 + } + } + os.Exit(code) +} + +// Fixture carries per-process state shared across this package's tests. +type Fixture struct { + Env *harness.Env + AWS *harness.AWSClient + Harness *harness.Fixture + Cluster *harness.Cluster +} + +// requireFixture returns the package-scoped Fixture, building it on first call. +// +// Three nodes is the floor, and for one more reason than instancerecovery's: +// recovery needs a survivor that is not the node that failed, and this scenario +// also needs a node for the backends that is not the load balancer's, so that +// taking the load balancer's node away does not take its backends with it. +func requireFixture(t *testing.T) *Fixture { + t.Helper() + pkgFixOnce.Do(func() { + if os.Getenv("SPINIFEX_E2E") == "" { + return + } + env := harness.LoadEnv(t) + if devNetworking(env) { + pkgFixErr = errSkipDevNetworking + return + } + harness.RequireDNSEnabled(t, env) + cluster, err := harness.ClusterFromEnv() + if err != nil { + pkgFixErr = fmt.Errorf("this suite needs node SSH to take a node away: %w", err) + return + } + awsCli := harness.NewAWSClient(t, env) + h, err := harness.NewProcessFixture(awsCli) + if err != nil { + pkgFixErr = err + return + } + pkgFix = &Fixture{Env: env, AWS: awsCli, Harness: h, Cluster: cluster} + }) + if errors.Is(pkgFixErr, errSkipDevNetworking) { + t.Skip("dev_networking is on, so this cluster has no external address for a load balancer to keep") + } + if pkgFixErr != nil { + t.Fatalf("lbrecovery fixture init: %v", pkgFixErr) + } + if pkgFix == nil { + t.Skip("SPINIFEX_E2E unset") + } + if n := len(pkgFix.Cluster.Nodes); n < 3 { + t.Skipf("a load balancer needs a survivor to move to and a node for its backends, have %d", n) + } + return pkgFix +} + +// TestALBSurvivesItsHostFailing is the whole suite. It is one test because it is +// one story, and splitting it would mean taking a node away twice to assert two +// halves of the same outage. +func TestALBSurvivesItsHostFailing(t *testing.T) { + runALBSurvivesItsHostFailing(t, requireFixture(t)) +} diff --git a/tests/e2e/lbrecovery/testdata/target-userdata.sh b/tests/e2e/lbrecovery/testdata/target-userdata.sh new file mode 100644 index 000000000..67c864aa1 --- /dev/null +++ b/tests/e2e/lbrecovery/testdata/target-userdata.sh @@ -0,0 +1,9 @@ +#!/bin/bash +# Backend for the load balancer recovery suite: one HTTP responder on :80 that +# names itself, so a round of probes shows which backend answered and whether +# both are still being reached. +INSTANCE_ID=$(hostname) + +mkdir -p /tmp/httpd && cd /tmp/httpd +echo "{\"instance_id\": \"${INSTANCE_ID}\"}" > index.html +nohup python3 -m http.server 80 --bind 0.0.0.0 > /dev/null 2>&1 &