mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Ben Duncan
ea3e01bb20
* fix: reboot a guest by asking it, not by pulling its reset line RebootInstances was QMP system_reset alone. The guest kernel is never told, so it does not sync, does not unmount and loses every dirty page - which is how a reboot issued seconds after a write loses that write, and how a guest rebooted during cloud-init can fail to come back at all. Guests now launch with -action shutdown=pause, so a clean guest poweroff leaves QEMU paused rather than gone. Reboot presses the power button, waits for the guest to reach that state, then resets and resumes the same process: no volume, tap or state churn, and no window where the guest is missing. A guest that will not go quietly is hard-reset, which is what AWS documents. The button is pressed again every 10s while the guest ignores it. One signal is not a shutdown request on a guest whose acpid has not started, which is the same boot race that produced the failure. Two consequences handled here: a guest that powers itself off now leaves QEMU paused, so the heartbeat quits it; and a guest launched before the flag exits on powerdown rather than pausing, so it is reset without being asked. * fix: answer a reboot when it is accepted, not when the guest is back Asking a guest to shut down takes as long as the guest takes. RebootInstances gives the daemon five seconds to reply, so a reboot that waited for the guest reported a reboot that had in fact succeeded as InvalidInstanceID.NotFound. Manager.Reboot now admits the reboot — the instance exists and is running — and runs the powerdown and reset in the background, which is the contract EC2 documents. The two admission failures are still reported; nothing after them can be, so a reset that fails is logged and left to the heartbeat. The gateway also stops calling a NATS timeout NotFound. Only the absence of a subscriber says anything about where the instance is, which is how the sibling start and terminate paths already read it. * test: prove the churn reboot by its boot ID, not by how fast it happened The subtest waited ten seconds and then read /proc/uptime. A reboot is asynchronous, so the guest answering at that point may be the one that has not shut down yet, and its uptime is whatever it was before — which the test reads as "reboot may not have occurred". Waiting for the boot ID to change is positive proof the guest rebooted and does not depend on how long the guest takes to go down. The uptime assertion stays, now measured against a boot that is known to be new. * test: prove the graceful reboot against real guests and a real engine Two tests the reboot design asked for and the unit tests cannot give. Cell 18 launches two guests that differ only in how they answer the power button. One installs a shutdown hook writing an fsynced line a second, so the file that survives measures how much of its shutdown it was allowed to run — a guest reset instead of asked leaves none of it. The other ignores the power key entirely and must be reset once its budget expires, with the daemon saying so. Each variant asserts the evidence only it can produce, and the slow one asserts the absence of the line the wedged one requires. Cell 27 loads a PostgreSQL instance past its shared buffers with pgbench and reboots it dirty. The probe is an UNLOGGED table: PostgreSQL writes no WAL for one, so a clean shutdown checkpoints it and crash recovery truncates it unconditionally. Rows committed normally survive a hard reset too, so the obvious assertion would pass against the bug. The rds suite and cell 27 both gain budget for the load phase. * fix: stop the process-group test misreading a reaped child as alive The test polled until the child was gone, then checked once more. A task being reaped shows state X for an instant, which processRunning counted as running, so that second look failed about 1 run in 50 under -race. * fix: wait for the recovery retry loop before removing a test's data dir restoreInstances starts the retry loop untracked, and it rewrites the local state file when it ends. A write landing during t.TempDir cleanup failed TestRestoreInstances_* with RemoveAll: directory not empty.