Update request and EC2 error-code counts after the suite artefacts were removed, describe how the sweep now reaches its targets, and point at the plan that traces each finding.
Salts restart per operation, so tag, untag and update cases only reached a resource when the same-numbered create case had made one. Create an IAM, ECS and ECR fixture per run and point every operation but a create or delete at it, and give ECR's policy and manifest documents values AWS accepts.
Clear the warnings the conformance suite caused itself, so what remains is a real divergence from AWS:
- Seed conditionally required members into the base request of RDS CreateDBInstance and RestoreDBInstanceFromDBSnapshot, ECS CreateCapacityProvider and CreateService, and EC2 CreateCapacityReservation.
- Give loosely modelled members values AWS accepts: ELBv2 ProtocolVersion and Matcher.HttpCode, EKS policyArn, addonName and access-entry type, EC2 SourceRegion and STS AccessKeyId.
- Import a CA-signed certificate, key and chain in ACM ImportCertificate.
- Rewrite ACM's lookahead domain pattern as RE2 so RequestCertificate is generated, and give the integration ACM service a tenant CA permitting example.com.
- Run the service daemon-lite behind the cross-tenant boundary test.
- Accept ELBv2's common parameter validation errors in responses.
TestServeQueueGroupBalancesSharedSubjects read both workers' handler counts
immediately after waiting for the mount to land on both of them, on the
argument that ordering on the shared connection meant the earlier capabilities
message must already have been delivered to everyone getting it.
That ordering does not exist. Each async subscription dispatches on its own
goroutine, so a worker's mount handler and its capabilities handler race, and
the mount having landed says nothing about whether the capabilities message has
been counted yet. Race Detection caught it: publishes reached 2 while
capabilities was still 1.
Wait for the expected count and then confirm it does not move. Both halves
matter — how many handlers a message reaches is the server's decision, so the
count converges from below and cannot overshoot, but a broken queue group
delivering to both members would pass through the right value on its way to the
wrong one.
* fix: wait for JetStream quorum before deciding viperblock has no storage
On a cold multi-node start every service comes up at once against a cluster
that has not elected a meta leader yet. IAM retries that window; viperblock did
not. Its lease and dirty bucket opens are both on the path to the daemon
running at all, so the first refusal took the process down with "volume lease
bucket: context deadline exceeded" and systemd crash-looped it into a failed
bootstrap.
The start path now opens both through kvstore.OpenWithRetry, which is what vpcd
and IMDS already use for the same window. The plain constructors are untouched:
they are what the replica-count and cluster-size tests assert fail closed, and
an operator tool reading a refusal as a hang would be worse than the crash.
OpenWithRetry now stops on an undeclared cluster size, matching the IAM and
kvlease loops. Waiting cannot declare one, so retrying it would turn a clear
configuration fault into ninety seconds of silence followed by the same error.
* test: mark the viperblock bucket startup test as in-package
test-package-check requires new test files to be external unless the file says
why it cannot be. The start-path openers and the constructors beneath them are
both unexported, and which of the two waits is the whole subject.
* fix: assert the takeover against the volume that was actually left unsealed
TestCrossNodeStartAfterFailedSealTakesOverLoudly picked its subject in advance
— the attached pattern volume — and treated the stop returning an error as
proof that volume's seal had failed. It is not. The unmount is a NATS request,
and the caller giving up says nothing about the seal behind it; local state
kept with no seal receipt is equally what an in-flight seal looks like.
In the nightly the test then thawed predastore, the pattern volume's seal
finished against a backend that was answering again, and the marker cleared. By
the time the volume moved there were no unsealed writes, so viperblock was
right to say nothing and the test failed it as a silent loss.
The product was loud where it mattered. The instance's root volume has to read
its AMI snapshot to seal, which the freeze does break, so its seal genuinely
failed and the node it landed on logged the takeover naming the node that had
been holding the writes. A blank attached volume has no such read and can seal
straight through the outage.
So the test now reads which volumes the cluster records as unsealed, after the
thaw has settled rather than before, and asserts the takeover against those. If
nothing was left behind it skips: the fault injection did not reproduce, and
asserting a warning there would be asserting a wrong one.
* fix: publish the instance cache resync mark with its readiness, not after it
installSync set ready under c.mu and markResynced ran after the unlock, so
there was a window where the cache was ready and serving records while
resyncAge reported that no sync had ever succeeded. A caller that observes a
ready cache and then asks how stale it is got "never", which reads as a cache
that cannot be trusted, and the resync-age metric was skipped for as long as
the window lasted.
The mark now goes up under the same lock that publishes readiness, so any
reader that takes the lock and sees a ready cache sees the mark too. Both
callers dropped their trailing markResynced, which installSync now does.
Widened under the race detector, which is where TestPeriodicResync_Undecodable
RecordMarksDegraded had been failing intermittently: it reads resyncAge
straight after waiting for readiness.
* fix: replicate every JetStream KV bucket across the cluster
A KV bucket created without a replica count gets one replica, and one
replica lives on one JetStream-chosen server recorded in no config.
Losing, restarting or partitioning that server makes the bucket return
"no responders available" from every node at once, so a single-node event
becomes a cluster-wide outage. The reconciler leader-lease buckets were
the worst case: acquiring a lease is a write, so a lease bucket without a
quorum cannot be acquired by anyone, and every reconciler sharing it stops
cluster-wide at exactly the moment a node has failed.
The cluster's node count is now declared once, in config.LoadConfig, and
read by the one helper every bucket creation goes through. Every process
that creates a bucket loads the cluster config first, so no service can
forget to pass it and no signature has to carry it. An undeclared size is
an error at creation rather than a silent fall back to one replica.
Existing buckets are raised rather than left behind: a service opening one
raises it, the daemon sweeps them all after formation, and
"spx admin kv replicas --repair" does it on demand. Raising reads the live
stream config and changes only the replica count, and never lowers one.
Removes the ~30 replicas/clusterSize parameters that only forwarded a
number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no
clamp and would have failed outright on a cluster of six or more.
Proved by a repo-wide invariant test pinning bucket creation to the one
helper, a whole-cluster sweep in every multi-node nightly cell, and a new
kvquorum suite (cell 33) that takes away the node leading the most buckets
and requires the survivors to keep answering for all of them.
* chore: name cell 33 in the nightly dispatch input
* fix: a KV bucket that cannot be raised must still open
The first version of the replica fix treated a raise it could not place as a
hard error, and on a real three-node cluster that turned the fix into the
outage it prevents. A node whose config named a fourth server wanted four
replicas, NATS answered "no suitable peers for placement" for three, and the
daemon retried for its full five-minute budget without finishing bootstrap.
The bucket was present, quorate and serving on three nodes throughout. A
config naming a node before that node serves is the ordinary middle of
growing a cluster, so that is a state to tolerate rather than refuse.
The line is now drawn between the two cases instead of around both. Creating
a bucket below the cluster's replica count still fails, because that is the
defect and a fallback there reintroduces it where it is hardest to notice.
Failing to raise an existing bucket warns and hands the bucket back, and the
raise is retried on every open besides the daemon sweep and --repair. This
matters most in kvlease, which runs on every reconcile tick: refusing there
would stop every reconciler sharing a lease bucket that was working.
getOrCreateBucket now attaches before it creates. A create asking for more
replicas than the cluster can place is refused before the server looks at the
name, so with create-first "this bucket already exists" was an inference from
which error came back, and the wrong one comes back on a single-node server.
RaiseAllBucketReplicas attempts every bucket and joins the failures, and
--repair keeps going too. Stopping at the first would leave the rest of a
cluster short because one bucket could not be placed.
Also adds nats.ServerName/NodeFromServerName beside the template that renders
server_name, with a test tying them together. The kvquorum suite had mapped
stream peers to nodes by comparing them to node names, which they are not:
they are the node name behind a "spinifex-nats-" prefix, so nothing matched
and the node-loss test could not choose a victim.
* test: wait for the right NATS peer count after a node rejoins
/routez reports a node's routes to its peers, not its own membership, so a
fully formed three-node cluster is two peers per node. The kvquorum suite
waited for three and spent its whole five-minute rejoin budget on a cluster
that had already reformed. Every other caller passes len(Nodes)-1; the
helper's doc comment now says so.
The assertion the test exists for had already passed: node1 was leading 18 of
47 buckets when it was taken away, and node2 and node3 each still answered for
all 47.
* fix: replicate KV buckets across the largest odd count a cluster can place
A bucket was replicated across min(nodes, 5) servers, which stores a fourth
full copy on a four-node cluster and still survives exactly one loss — the
same tolerance R3 gives for three quarters of the write amplification. A Raft
majority needs an odd count, so the rule becomes the largest odd count the
cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two
entry points, so no caller can hold a stale copy of the policy and keep passing
after the policy changes.
Three retry loops could not tell a configuration fault from a cluster that had
yet to form. An undeclared cluster size never becomes declared by waiting, so
each of them spent its whole budget on one and then reported quorum as the
cause. They now stop on a permanent error and say what is actually wrong. This
is what the integration suite was hitting: it builds its gateway by hand and
never loads a cluster config, so the size was never declared and the IAM retry
sat in its sleep until the test timed out. Both ends are fixed.
The audit reported configuration alone. A bucket at three replicas with one
peer offline and one lagging is one failure from unavailable and read as ok.
BucketReport now carries the leader and a count of the peers JetStream itself
calls current and reachable, quorum is measured against that count, and
spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem
--repair fixes from one needing a human.
Declaration is write-once: daemon.Start refuses a conflicting redeclaration
because it is about to create buckets, and it declares before JetStream init
rather than after it. config.LoadConfig warns instead, because it also reads
config files that are not this process's own cluster.
spx admin init --nodes 2 is refused — no replica count on two nodes both
survives a loss and accepts writes — and single-node replication is asserted in
the single e2e suite, so it runs in every nightly host permutation.
* fix: withhold the vLLM serving image until its artifact is published
The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names
404s on iso.mulgadc.com, so spx admin images import would fail for every
operator who tried it. Its GPU siblings are published; this one never was.
The entry moves to WithheldImages rather than being deleted, because the entry
is the definition — the URL, the checksum and the two tags resolveServingAMI
filters on. Publishing the artifact is then a move back, not a retype.
Bedrock already handles the image not being registered: resolveServingAMI
returns ErrServingAMINotFound, which is the honest answer while there is
nothing to launch.
The bedrock tag contract is still pinned, now against the map the definition
lives in, and it additionally asserts the entry is not offered for import. The
e2e reachability walk covers AvailableImages alone, so a withheld entry is out
of it by construction.
* fix: drop the cluster size from the call sites dev added
Two constructors on dev still pass a replica count that no longer exists as a
parameter: the cluster size is a declared process fact, so EKSServiceDeps has
no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is
textually clean and breaks the build, which is the shape of collision a
signature removal always has.
* docs: explain what an operator sees when a cluster server fails
The replication section said how many copies a bucket has and stopped there,
which leaves the question an operator actually has during an incident: one
server is gone, what still works. It now covers the majority side carrying on,
an isolated server refusing changes on purpose and why that is the behaviour
you want, guests continuing to run while anything that has to be recorded does
not, the risk of reading state from a server that is cut off, and a returning
server catching up by itself.
The surrounding prose reads from the operator's side rather than the
implementation's: what you see and what to do, not which consensus protocol is
underneath.
CreateRole, GetRole, ListRoles and instance-profile roles each get their own projection; GetRole carries an empty RoleLastUsed. Policies carry UpdateDate and PermissionsBoundaryUsageCount, CreatePolicy drops Description and ListPolicies drops Tags. CreateUser echoes Tags, list instance-profile actions drop profile Tags, and GetAccountSummary gains AWS's seven missing keys and loses SigningCertificatesPerUser.
AWS answers a bad name charset, a path not bounded by slashes and an
enum violation such as UpdateAccessKey's status with ValidationError.
Length and tag violations keep InvalidInput until AWS's code for them
is captured.
AWS percent-encodes every policy document it returns. Spinifex returned
raw JSON, so a client decoding once read '+' as a space and '%41' as 'A',
giving perpetual Terraform drift, and a lone '%' failed the read.
Encode in the gateway wrappers for CreateRole, GetRole, ListRoles,
GetRolePolicy, GetUserPolicy, GetGroupPolicy, GetPolicyVersion and the
instance-profile reads. The service layer keeps raw JSON because STS and
spx admin parse it in-process. The web UI now decodes role inline
policies like every other document.
Tests added since main used in-package test packages, failing the
test-package check. Convert them to <pkg>_test with export_test.go
hooks; the ecs-agent main package keeps an in-package marker.
A request whose body cannot be read to hash failed through the catch-all
branch, logging a malformed envelope, returning IncompleteSignature and
recording an anonymous lockout strike, so ten dropped connections locked
out an IP. Match sigv4.ErrReadingBody and answer InternalError without
recording a failure. Bumps bluebottle for the new sentinel.
* feat: recover instances automatically from a node that stopped heartbeating
A node that dies hard takes its guests with it, and nothing brings them back
until an operator brings that node back. Every daemon now runs the same small
reconciler: it reads the instance records, selects desired-running instances
whose owner is stale, and relaunches them here.
The claim is a CAS on Status.LastNode and that is the whole of the
coordination. Several survivors race one instance, exactly one revision wins,
and the losers find a record naming a live owner and stop considering it. A
claimant that dies mid-launch leaves the record naming a node that is itself
stale, which is the same candidate the next pass derives again, so there is no
lease to expire and nothing to reconcile by hand.
Ownership is not permission to write. The volume leases still decide whether the
guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt
that loses the volume race has touched no tap, no OVS port and no OVN binding.
A storage failure is not a host failure and wants the opposite response. A guest
paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is
refused: the volumes are the same objects in the same store from anywhere, so
relaunching moves the pause rather than ending it. A node whose own object store
is not answering stands down entirely, which separates a store broken here from
a store broken everywhere without either having to be diagnosed. A launch that
does not complete backs this node off that instance, doubling to a cap, because
nothing on the mount path can yet say which of the two it hit.
Recovery is off by default and inert below three nodes. It also waits until it
has seen its peers live, or five minutes, so the first node back from a
coordinated shutdown does not recover guests from nodes that are still booting.
* test: prove instance recovery moves a host failure and refuses a storage failure
The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves.
Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live.
harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down.
* fix: make automatic instance recovery a cluster-wide setting
Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file.
Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it.
* test: drive real I/O into the storage-fault case, and freeze the store rather than stop it
Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted.
The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them.
The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three.
* test: give instance recovery its own suite and its own nightly cell
Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five.
Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set.
* fix: give a guest long enough to open a volume the object store is slow to serve
QEMU opens every drive before its monitor answers a greeting, so the QMP
greeting deadline is really a deadline on opening the guest's volumes. At 30s
it was shorter than a cold viperblock open costs on a cluster that has lost a
node, where every object read is served degraded: measured at 35s with one node
of three away. The launch killed the guest five seconds short of ready and then
did it again on every retry, which is a recovery that can never complete.
A network-backed drive now gets its own deadline for the same reason VFIO
already had one — synchronous work before the monitor answers — and the
instancerecovery suite is what found it.
Also make the suite's predastore thaw best effort per node. It runs after a
node has been taken away, where there is no process to signal, and a node it
cannot reach must not stop it restoring the rest.
* docs: name the two newest cells in the dispatch input
The cells input is the operator's index of what a number means, and it stopped
at 29 while 30 and 31 were added.
* test: prove the guest is writing before freezing the store in the recovery suite
The storage-fault test froze the object store as soon as RunInstances reported
running, which is when QEMU has launched rather than when the guest has booted.
On the nightly cluster the freeze landed while the guest was still in UEFI, so
the only drive it could reach was the EFI varstore — and pflash has no
werror=stop to pause on, so the guest never paused and the test reported a
product failure it had caused itself.
The load driver now prints to the serial console after each pass it has written
and synced, and the test waits for that before freezing. The console is read
through GetConsoleOutput, so this still needs no login and still runs on cells
whose guests have no public address.
* feat: recover instances automatically with nothing to configure
The switch is gone. An instance whose host has failed is down whether or
not recovery is enabled, so the only outcome a disabled reconciler can
produce is leaving it down — which is the state recovery exists to end,
not a safer one to fall back to.
Against that, the switch cost a cluster that can be configured into
never recovering anything, with no symptom until a node dies. This
feature already hit that once: a top-level table parsed into nothing and
every node reported recovery off with the setting plainly in the file.
What makes recovery inert is now only facts about the cluster — no
cluster config, no JetStream, or fewer than three nodes — and each says
in the log which one applied. The e2e no longer writes config or
restarts daemons: it waits for every node to report the reconciler
active on a cluster nothing configured, so reintroducing a switch fails
there as well as in the unit tests.
* feat: stop retrying a recovery that is never going to work
Every failed attempt used to go to the same exponential backoff and stay
there. An instance nothing could launch was retried every fifteen
minutes forever, still observed-running on a node that is gone, with no
account of it anywhere the customer or an operator would look.
A failure is now classified and counted. Two classes are read from what
the launch actually reports: the node it was taken from still holds a
volume lease, which clears on that lease's clock and is retried on a
faster one, and this node has no room, which another survivor may. There
is no terminal class, because nothing on this path can prove one — the
launch collapses everything else into ServerInternal, so a verdict of
"never going to work" would be a guess, and the guess that abandons a
recoverable instance is the expensive one.
The budget bounds the unknown case instead. Every class counts against
it, since a class that did not would be the old unbounded retry under a
narrower condition. Exhausting it leaves the instance where AWS leaves
one whose host failed and could not be brought back: stopped, carrying a
StateReason that names the cause and the last error, one StartInstances
from running again. Writing the desired state down is also what makes
one node's decision cluster-wide, because selection reads desired state
— the same property the claim already has.
A record another node has since claimed is never touched, so a node
exhausting its attempts cannot stop a guest that is now running
elsewhere.
* fix: give a guest the time for every cost it pays before its monitor answers
A GPU instance booting from a network-backed root volume pays both
costs: QEMU opens every drive across the object store and then pins the
whole of guest RAM, one after the other, all before the monitor answers.
The deadline took whichever was larger, so such a guest was given less
time than it needs.
That the two floors are both 180s is why nothing has been seen yet — the
VFIO branch has always returned at least the network-drive deadline. It
is the same defect either way, and the symptom when it lands is a GPU
instance SIGKILLed on every attempt while its plain sibling launches,
which reads as a GPU fault rather than as a deadline.
The costs are now added, the plain QEMU startup is counted once rather
than replaced, and the cap covers both floors together so it cannot clip
a guest before either cost has been waited out.
* test: assert the AWS-visible arc of a host failure end to end
A customer cannot see which host an instance is on, and AWS never tells
them. What they can see is DescribeInstanceStatus, so that is where a
recovery has to be provable, and the suite never looked at it.
Three frames are now required, in order: running with system-status ok
before anything is broken, system-status impaired while the owning node
is gone, and ok again once a survivor is running it. The middle frame is
the one worth having — the window opens when the owner's heartbeat goes
stale, which is strictly before any survivor may claim, and closes when
the relaunch registers, so it is bounded rather than lucky. Reporting ok
throughout would be the more comfortable lie.
The availability zone is compared across the first and last frames as
well. A recovery that crossed one would break every placement promise
made to the customer, and nothing else in the suite would have noticed.
* test: prove a half-healthy node fences itself and is never a second writer
The two existing recovery cases take a node away by stopping its services,
which is the failure this feature handles least interestingly: a node that is
gone has no opinion about who owns its volumes. The one that matters is the node
that is still up, still running QEMU, still answering its own local clients, and
simply cannot reach consensus. That is the state in which two nodes can each
believe they own a volume, and the only thing between that and a corrupted disk
is that the partitioned node gives up first, on a clock it can evaluate without
reaching anything.
So this drops NATS on 4222 and 4248 between one node and its peers and nothing
else, then asserts the ordering the safety argument rests on rather than the
outcome it produces. A watch samples every node every three seconds for the
whole run, including the heal, and fails on any single sample naming two hosts.
Two guests writing one volume through one object store is data loss already
under way, not a race that resolves, and no point-in-time assertion can see one
that appeared between its own samples.
The injector takes its own nft table rather than touching inet spinifex_filter,
because a test that flushed the node's real firewall would leave it either
unfirewalled or misfirewalled with no sign of which, and a separate table at
priority -300 heals in one atomic delete. A transient systemd timer removes it
independently, so a cancelled workflow does not leave a node cut off from its
cluster until somebody notices. It proves SSH still works and proves a peer port
is genuinely unreachable before anything is asserted, since a rule set that
installed cleanly and dropped nothing would read as the system tolerating a
partition.
The suite budget goes to 90m and cell 31 to 105m to fit the third case.
* fix: move an instance's boot state, volumes, logical port and address with it
A record-driven launch assumes the node reading the record is the node that wrote
it. On a daemon restart that holds. On a recovery onto a survivor it does not, and
five things follow from it — one that stops a moved instance booting at all, and
four that stop it running anywhere.
A load balancer never comes back from a host failure. Its boot configuration is a
set of files under the runtime directory and the record holds their paths, so the
survivor restores paths that exist only on the node that died. The hook that
rebuilds them was reachable from Restore alone, and the cross-node path goes
straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert
that are not there. The gateway URL baked into that env names the creating node's
management bridge as well, which the rebuild corrects because it rebuilds from
whichever node is running it. Firing the hook on both paths fixes both, and a
rebuild that fails now refuses the launch: a guest that boots without its network
configuration is worse than one that does not boot, because the records would say
it recovered.
The export outlives the guest and holds the volume lease. A returning node is
healthy, so it renews that lease indefinitely and the new owner never acquires:
the guest is stopped here and refused there, recovery spends its attempt budget on
a volume that was never going to be free, and gives up on an instance that was
never unrecoverable. So the export is given up too, over a new node-addressed
request that tears it down exactly as the fence does. Deliberately not an unmount
— an unmount seals, this node's block map is the stale one, and sealing publishes
it over the winner's. Unlike the fence it deletes the lease key rather than
leaving it to expire, because the lease is still this node's to hand back.
The tap outlives the guest, and the tap is what claims the instance's OVN logical
port. The binding names one chassis, so two nodes offering the same iface-id do
not split the traffic — they contend, and the address answers from whichever won
last. It also blocks the layer above: the external IP's host ingress is pruned on
the evidence that the owning guest sits on another chassis, and that stays false
while this node is still claiming the port.
The management address outlives the guest as a cluster-wide reservation, and the
release was unconditional. br-mgmt is one flat L2 segment, so the node an instance
left was freeing an address still in use and offering it to the next instance
anywhere. Only the node an entry names may release it now, with the takeover
recorded on arrival so the node actually running the instance can still give the
address back. An entry written before the node field existed names nobody and
stays releasable, because there is no other holder to collide with.
The fence path already cleaned up the tap, so the release fix repairs that path
too.
* test: prove a load balancer keeps its name, its address and its traffic when its host dies
Takes away the node running an internet-facing ALB and asserts the four things a
customer of one actually holds: the name keeps resolving, the address behind it
does not move, requests through the name are answered again without anybody being
called, and the backends are healthy again from the recovered load balancer's
point of view.
The backends are placed around the load balancer rather than beside it. A load
balancer lands wherever a node answers its queue group, so its node is discovered
and the backends are relaunched until both are elsewhere; one on the node about to
be taken away would be recovered too, and the run could not then tell an ALB that
came back serving from one that came back with nothing to serve.
DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve
through the same server, so the claim under test is what this cluster answers for
a name rather than what the machine running the tests was configured to forward
to. A watch samples every surviving node every five seconds for the whole run,
because an answer that went missing or pointed elsewhere for one reconcile pass is
the outage, and no single sample reaches it.
The returning node then has to give up what it no longer owns: no second copy of
the guest, and no tap still claiming the load balancer's logical port, since OVN
binds a port to one chassis and a second claimant contends for it rather than
splitting the traffic. The last assertion is that the name still serves with that
node back in the cluster.
* docs: name the new cell in the nightly's cell list
* test: stop the disk-load guest from filling the console it is judged by
The load generator announced every write, once a second, for as long as the
test ran. The console is a ring buffer this suite reads back, so an hour of
that pushes out everything explaining a failure, including the kernel's own
account of the I/O error the test exists to produce. Announce twenty times,
then keep writing quietly.
* docs: say what the OCI guide has been proven on, and what a host failure does
The guide read as a support statement for OCI rather than as an account of one
tested path, promised seven steps and then gave eight, and said nothing about
what happens to a guest or its public address when the node under it fails.
State the evidence, state that the nightly guards the datapath and not the
cloud, and give the failure behaviour as a table with the reason codes a
customer will actually see.
Also say plainly what the volume lease is: one running guest and a bounded,
logged window, not a write barrier the object store enforces. Someone sizing a
workload against this should read that before they read the table.
* docs: say why the mount classifier refuses to retry a lease another node holds
It reads as an omission — a lease held elsewhere is the most transient refusal
there is. It is not. This classifier drives the restore path, where the holder
is a node that is running the guest, so retrying would relaunch against a live
owner. Recovery wants the opposite and reads the refusal itself.
* feat: claim a recovered guest's public address in line, and verify it arrived
A recovered instance was reported running before anything moved its public
address off the node that died, so for up to fifteen seconds the records said
recovered and the address answered nothing. The recovery path now runs the
affinity pass itself, which is idempotent and is the same pass the ticker runs.
Two things the pass could not do before. It abandoned every remaining address
on the first API failure, though a throttle or a denial is per-object and the
guests behind the others are just as dark. And it trusted the move's own
response, which is OCI agreeing to the request rather than OCI having done it,
so an accepted move that never landed read as success. It now re-reads what OCI
delivers here and reports the difference.
An address that will not arrive makes the guest a system-status failure with the
time it went dark, the same shape AWS uses for a host-level fault. The guest
passes every check it can be asked about itself, which is exactly why reporting
it healthy was the worst available answer.
* feat: give a recovery attempt one trace across all three of its phases
The volumes opening, QEMU starting and its monitor answering already trace
themselves, but nothing tied them to the attempt that asked for them: three
orphan traces per recovery, no way to ask which phase a slow one spent its time
in, and nothing naming the node the guest came from. A span over the attempt
makes them its children, and the reconciler's own log lines carry its trace ID
because the handler stamps it from the context.
A failed attempt records the fault and the attempt count on the span, so the
budget running out is readable as a sequence rather than as twelve unrelated
warnings.
* fix: finish a fence once its writer finally exits
A kill that did not take effect was logged, counted and dropped. The export
stayed registered and the lease stayed held, which is correct while the writer
lives and wrong forever after it dies: a lease a healthy node keeps renewing
never expires, so the volume could not start anywhere.
The failure is now re-reported every 30 s until it resolves, and the teardown
completes as soon as the writer is confirmed gone. Only a delivered kill is
waited on — a refused one says nothing about what became of the writer.
* test: stop judging a detached stop or terminate on a transient state
Both tests read the instance state immediately after the ack, but the transition
runs on its own goroutine, so the read races it in both directions: too early
sees running, too late sees the final state. The race gate caught the first of
those on a loaded runner.
* fix: replicate the volume lease and dirty buckets across the cluster
Both were created with the default single replica, so both lived on one node.
That node holds the answer to "who may write this volume" and the marker every
mount reads, and losing it takes volume leasing and mounting down cluster-wide.
A partitioned node was fenced correctly and then no survivor could relaunch its
guest: every attempt failed on "read dirty mark: no responders available", for
ten minutes, until the budget ran out.
The replica count is now the cluster's node count, as the gateway's own buckets
already do. Existing buckets are raised in place rather than requiring a wipe.
The e2e diagnostics that should have said this are fixed too. The fence and the
surrender are logged by the storage service, not the daemon, so the assertions
read the wrong journal; and Eventuallyf evaluates its message at the call site,
so a ten-minute wait was explained by a log that stopped before it began.
Julian Sommer