The storage-growth suite measures backend bytes. Local WAL retention sits
upstream of that boundary and nothing under tests/e2e ever looked at it, so the
condition this suite produces on every run has never once appeared in a result.
TestContinuousChurn is the only workload shaped to show it, because it holds a
single volume open for minutes while writing concurrently. The measurement has
to be taken before the detach: Close calls RemoveLocalFiles and deletes the
per-volume directory, so afterwards the number always reads zero.
It reports rather than asserts. Retention is unconditional today, so an
assertion would be red on every branch and would teach readers to scroll past
it. The readout is a banner plus a GitHub workflow warning annotation, so it
lands on the run summary and not only in the log body. It goes quiet on its own
once reclaim lands, at which point it can become a hard assertion.
Every probe step runs under sudo, the directory test included. The viperblock
base dir is 0700 spinifex-viperblock, so an unprivileged [ -d ] reads false even
when the volume is sitting right there, which would have silently downgraded
this to "no local state found" on every run.
Verified against a real retained WAL on a live node: 14853396868 bytes across
2028 files, rendered as 13.8 GiB at 99% of bytes written.
BlockDeviceMappings[].DeviceName was overwritten with the path the guest kernel assigns, on every launch, start and hot-attach. The name is ForceNew on aws_instance.ebs_block_device and root_block_device, so a value that never reads back as it was sent is not drift to reconcile: every plan destroyed and recreated the instance and everything on its volumes, and a stock terraform-aws-modules/ec2-instance configuration never converged.
The convention it served — lsblk inside the guest showing the same names as DescribeInstances — is not one AWS keeps either. On Nitro the API reports /dev/sda1 while the guest shows /dev/nvme0n1p1, and IMDS block-device-mapping reports the API name too. The API name is a label, and the wire contract owes the caller the label it supplied.
UpdateGuestDeviceNames becomes LogGuestDeviceMap: the guest paths are still discovered and logged for an operator correlating a volume against what the guest sees, and no longer recorded as the device the caller asked for. AttachVolume records the API name it already returns and already persists to volume metadata. IMDS projects the same field and so becomes AWS-shaped with it.
The Terraform AWS provider compares the stored url against the configured one through suppressOpenIDURL, which parses the stored value and forces the https scheme onto it. An issuer carrying a port is not parseable without a scheme — Go reads the host as one and rejects the rest — so the parse fails, the suppressor returns false, and the ForceNew url attribute replaces the provider on every plan. Every issuer this platform mints carries a port, so the AWS-shaped stripped form is the one that cannot round-trip here. The ARN keeps the stripped host-and-path form, which is what AWS uses and what issuerFromOIDCProviderARN reverses.
An instance whose relaunch fails during daemon-startup recovery goes to
StateError, and nothing ever retried it: every later restart read the same
StateError and skipped it again, permanently. The usual cause is a dependency
that was not ready at boot, which is over seconds later, so the instance stayed
down for a condition that had already cleared.
The manager already bounds restarts for a crashing instance, through
Health.CrashCount and Health.FirstCrashTime against MaxRestartsInWindow. A
recovery failure wrote none of it, so it was invisible to the only budget there
is. MarkRecoveryFailed now counts one, the window's two rules move into
recordCrash and rollRestartWindow, and MaybeRestart is rewritten over them with
the same constants and comparisons it had.
With a failure counted, restore can retry: an instance in StateError with budget
left returns to Pending and goes through the existing relaunch, including its
mount-not-ready backoff. With the budget spent it stays in StateError for the
operator and is only announced to the recovering hook, as before. The window
expires, so a later restart tries again rather than writing the instance off.
StartStoppedInstance is the operator's remedy and it was refusing to work. The
shared stopped store never holds a StateError record, so the load missed, and
the fallback treated the presence of the dead local record as proof the instance
was already running. It reads the record's state now. StateError is started
locally through StartInstance, which already accepts it; terminated answers
InvalidInstanceID.NotFound; everything else keeps IncorrectInstanceState,
because a claim inserts the instance before it launches, so a local record in a
startable state is a start another caller is part-way through.
The agent resolved the container runtime once, in New. A containerd socket that was not accepting during the first second of boot left the agent runtime-less for the life of the process, and nothing retried. It registered and heartbeated normally, so the scheduler saw a healthy instance and kept placing tasks that died immediately at the nil runtime check.
The runtime now lives behind a gate that a watch loop keeps current: it dials while the gate is empty, and probes with a container list while it is full, clearing the gate when the probe fails so the next tick re-dials. That covers a socket that is not ready yet at boot and a containerd restarted underneath a running agent alike, and everything that used the puller or the runner reads through the gate, so a runtime arriving late is picked up without a restart.
An agent with no runtime advertises no capacity. Registration and the heartbeat report zero CPU and zero memory while the gate is empty, so placement skips the host, and the control plane reads that pair of zeros as an involuntary drain: the instance becomes DRAINING with agentConnected false and zero registered resources, which is visible through DescribeContainerInstances rather than only in the node's log. Capacity returning clears it through the branch a reaped instance already used. A host with no CPU cannot exist, so the zero needs no new wire field. An operator drain is no longer cleared by a bus registration either, which it previously was.
A deployment whose tasks keep failing to start now backs off. Deployment gains NextLaunchAt, and past the failure count the circuit breaker already tracks, the reconciler widens the gap between launch passes up to a cap. It is keyed on that counter but applies whether or not the breaker is enabled, because AWS leaves it off by default and an unbraked service relaunches a task that cannot start on every pass, forever.
An instance left in StateError could not be terminated through any path, and the gateway reported success anyway, so a Terraform destroy waited on a record the platform had already claimed was gone.
classifyRestoredInstances skipped a StateError instance with the comment that the operator must retry or terminate it, but continued before firing OnInstanceRecovering, which is what binds ec2.cmd.<id>. The topic the terminate travels on therefore had no responder and the stated remedy was unreachable.
With no responder there, TerminateInstances fell back to the ec2.terminate topic, whose handler reads the stopped KV bucket only and answered InvalidInstanceID.NotFound for a record held in the active registry. The gateway discarded that payload without reading its code and took the idempotent-absence branch, returning terminated for a record it had not removed.
The StateError branch now announces the instance to the recovering hook before skipping relaunch, and the gateway reads the refusal: only a genuine absence falls through to the idempotent branch, and any other code is logged and returned.
RunInstances created exactly one volume, the root, plus the EFI variable store, and refused a request naming any other EBS device. An instance could not be launched with a data volume attached, so terraform-aws-modules/ec2-instance with an ebs_block_device failed the apply and the workaround was CreateVolume plus AttachVolume afterwards.
The launcher was never the constraint: buildDrives already has a data-volume branch and already allocates a hot-plug port for a volume attached at RunInstances time. The control plane now creates a volume for every EBS mapping through the volume service, which is where CreateVolume's validation and metadata write already live, and records each as a non-boot request at the device name the caller named.
A data volume is created empty, so a mapping naming a snapshot or naming no size is still refused by name rather than served as something it did not ask for. A launch that fails partway deletes the volumes it created.
The filter was never read, so ACTIVE, DRAINING and no filter at all returned every registration in the cluster. A caller could not ask which container instances can take work without describing every one of them, and a cluster carrying a drained node read as if every node were live.
The status a record holds is the one DescribeContainerInstances reports, so a filtered list and a describe of what it returns agree. REGISTERING, DEREGISTERING and REGISTRATION_FAILED are accepted and match nothing, because no record here ever holds them; a status outside the enum is refused.
An unserved object sub-resource fell through to PutObject, GetObject or DeleteObject, so a write to one stored the object itself. The pin also carries the meta leader health check fix and the router's single query parse.
Both install guides had grown sections that were not install steps, which is what made the multi-node one hard to follow. Each now ends on a step that hands off to Setting Up Your Cluster, and the reference material moves under Administration where it can be read on its own.
- Single-node and multi-node both end on a "Set Up Your Cluster" step rather than a trailing sentence.
- The multi-node cluster-state check via `spx admin kv digest` is gone. Verification is now every node Ready, capacity pooled, and the AWS API answering.
- "How Multi-Node Storage Works" becomes docs/admin/predastore-distributed-storage, expanded with the host and node model, the write and read paths, degraded writes, hinted handoff, repair, encryption at rest, and what each class of failure costs.
- "Firewall and Cluster Membership" becomes docs/admin/host-firewall, expanded with the full public, cluster and encap port groups, how to narrow SSH, and which files under /etc/spinifex/firewall survive an upgrade.
- Setting Up Your Cluster gains a closing step that arms the firewall and narrows SSH, so the securing work lands after a cluster that works rather than during the install.
install-usb and vpc-networking followed the two moved sections to their new paths.
Three omissions left the launched root volume reading back as something the caller never asked for.
DescribeInstances returned a null RootDeviceName and RootDeviceType while DescribeImages populated both for the same AMI, so the two answers disagreed about the same image. The launch path now projects both onto the instance.
The root volume's EBS request carried no device name, so the launcher wrote an empty string to the volume's metadata and DescribeVolumes reported an attached volume with an empty Attachments[].Device. The name parseVolumeParams already resolved now travels with the request.
A volume-scoped TagSpecification was discarded: only instance-scoped tags were read, and the comment claimed volume specs were handled elsewhere, which nothing did. They are now written to the root volume's metadata document.
BlockDeviceMappings[].DeviceName still reports the guest path rather than the requested name. That rewrite is deliberate and is left alone here.
* fix: route the EKS update reads so they refuse instead of reading as a typo
UpdateClusterVersion was registered at POST /clusters/{name}/update-version, which is the nodegroup pattern and not a path the SDK ever calls; AWS puts it at POST /clusters/{name}/updates. The refusal written for it was therefore unreachable, and the router answered InvalidAction, which tells a caller they typed something that does not exist.
ListUpdates and DescribeUpdate had no route at all and answered the same way. Both now refuse with NotImplemented at the gateway rather than through a service method: the update surface they would read is absent, so a round-trip to reach a constant refusal is surface for its own sake.
The absence of the cluster-level update surface is unchanged. This only stops the platform blaming the caller's spelling for its own routing.
* test: account for the two EKS update reads in the cross-tenant table
Both name their cluster by a path segment that carries no account segment, so
neither can name another tenant's resource and neither needs a case of its own.
The agent acked a stop in the same loop iteration that launched the reap, so
the directive was deleted before the reap had done anything. When the reap
could not run — no container runtime, or a failing List — nothing re-posted it,
and the task held its ENI, its capacity reservation and its target-group
registrations forever. The in-memory `stopping` map made that permanent: it was
set at dispatch and never cleared, so the agent would not re-dispatch even if
the directive did come back.
A stop ledger now separates a reap in flight from a reap that reported STOPPED.
Only the second is acked, so a failed reap leaves the directive in the inbox to
be redelivered and retried, while the assign loop keeps reading both halves and
so still refuses to start a task that is stopping or stopped.
* fix: persist security group rule tags and report -1 ports for all-protocol rules
A security group rule accepted a security-group-rule TagSpecification and kept
none of it, so DescribeSecurityGroupRules answered with no tags and every
subsequent plan offered to add them again. SGRule now carries a Tags map,
populated on both authorize paths and projected into the central tag store, and
DescribeSecurityGroupRules evaluates tag: and tag-key filters against it instead
of excluding every rule.
CreateTags and DeleteTags on an sgr- id were a silent no-op because
updateRecordTags routed only subnet-, vpc-, sg- and eni-. It now finds the rule
inside its parent group's record and mutates it there, and a revoked rule's
central entry is cleared with it.
An all-protocol rule reported the ports it was stored with, 0 and 0, where AWS
reports -1 and -1, which is the same never-converging plan by another route.
* fix: recognise sgr- as a security-group-rule resource id
DescribeTags reported ResourceType "unknown" for a security group rule, so a
resource-type filter naming security-group-rule matched none of them, and the
authorizer could build no ARN for a rule id and fell back to the any-resource
wildcard. Both read the same prefix table, which had no entry for sgr-.
RunInstances read BlockDeviceMappings[0] and nothing else, so a request carrying a root mapping and a data mapping created one volume sized from whichever happened to be first, and the first mapping became the root whatever device it named. A module emitting the data mapping first produced a root volume of the data volume's size and no data volume at all, with no error.
The root is now the mapping naming the AMI root device, with position deciding only when nothing names it, so the single /dev/vda mapping ECS and EKS launch with still works. Any further EBS mapping is refused in ValidateRunInstancesInput with InvalidParameterValue naming BlockDeviceMappings and the devices it cannot attach, rather than being discarded silently. Ephemeral mappings carry no volume and are ignored.
Creating the additional volumes at launch remains unimplemented; this makes its absence visible.
The deploy builds with GOWORK=off, so spx links the pin rather than the sibling
checkout, and the versioning handlers are absent from every deployed binary
until it moves.
GitHub registers a schedule: only from a repository's default branch, so this cron could drive exactly one branch and always executed main's copy of this file whatever it tested. main was never exercised by the nightly, and a fix to the workflow itself could not be proved until it had already shipped — which is how cell-24's unguarded pkill stayed live on the schedule after its fix merged to dev.
A Cloudflare Worker now dispatches this workflow once on dev and once on main. A dispatch takes both the workflow file and the source under test from its ref, so GIT_BRANCH is simply github.ref_name and every schedule-only expression goes with the cron.
cell-24 clears leftover predastore nodes with `pkill -KILL -f '/bin/s3d
-config'`, which matches every predastore node on the runner and not only the
ones this run left behind. The predastore repository runs the same harnesses on
the same machine, so "still running" does not mean "stale" — on 2026-09-16 this
step killed a predastore benchmark 32 seconds into its workload, and the run
failed with 61 connection-refused errors that look nothing like their cause.
Both kills now run under the host benchmark lock the harnesses themselves take,
so anything alive while it is held really is a leftover. The second one needs it
as much as the first: by the time cleanup runs the harness has released the
lock, so another repository's benchmark can already be up.
Every deployed binary links whatever go.mod pins, so the routing fix for bucket sub-resources is absent from the gateway until this moves. Without it a provider-level default_tags block still leaves no bucket creatable by Terraform.
DescribeTags kept answering for resources that no longer exist: the central tag store's DeleteAllTags had one production caller, the instance terminate path, so every other resource type left its entry behind forever.
CentralTagWriter becomes CentralTagStore and gains DeleteAllTags, which TagsServiceImpl already implemented, so the daemon wiring is a rename rather than new plumbing. clearRecordTags mirrors projectRecordTags: a nil store is a no-op and a failure is logged rather than returned.
The clear runs after the record delete has succeeded, never before, so a delete that fails leaves a live resource with its tags intact. deleteNetworkInterface covers both the public delete and the forced instance teardown, and deleteSecurityGroupInternal covers the default group that only the DeleteVpc cascade ever removes.
An AWS SDK's default endpoint resolution builds
{service}.{region}.{services/domain}. IMDS previously answered that
leaf with the literal amazonaws.com, so a workload that forgot its
endpoint override silently reached real AWS instead of failing.
Now that service-endpoint DNS resolves cluster-side, serve the
cluster's configured AWS.ServicesDomain instead, threaded from
config through vpcd.Config into NewIMDSServiceImpl. An empty value
fails closed with a 404 rather than ever falling back to
amazonaws.com. services/partition is unchanged: it still returns
aws, since AWS SDKs carry hardcoded partition tables.
A tag set at creation time through TagSpecifications was written only onto the resource's own KV record. DescribeTags reads a separate central index at tags/{accountID}/{resourceID}.json, which only CreateTags writes, so a resource born with tags described as having none until something tagged it again.
Terraform finds this immediately: terraform-aws-modules/eks reads a cluster security group's tags back after creating it, and aws_ec2_tag fails against a resource whose tags the tag API cannot see.
The four VPC create paths now project their stored tags into the central store through an optional CentralTagWriter, injected after construction like the external IPAM. A nil writer leaves create behaviour unchanged, and a write failure is logged rather than failing the create — the resource exists either way, and failing a create over a tag index would be worse than the divergence it repairs.
Only vpc, subnet, security group and network interface are covered here. Image, volume, snapshot, route table, internet gateway, egress-only internet gateway, elastic IP, NAT gateway, key pair and launch template have the same split and are follow-up work.
* fix(ecs): make awsvpc ENI slots a placement resource
A container instance carries a finite number of awsvpc task ENIs — the hot-plug PCIe root ports the daemon pre-allocates from the instance type's cap — and nothing outside the daemon knew the limit existed. The scheduler's capacity model was CPU, memory and GPU, so every over-capacity task was placed, created an ENI, failed to hot-plug it and rolled it back. A service asking for more tasks than the cluster has slots churned ENI create/delete indefinitely instead of reporting a placement failure once. On a t3.micro the third task and everything after it failed with "attach task ENI eni-… -> i-…: ServerInternal".
The instance record now carries the instance type, reported the way AWS carries it, as the ecs.instance-type container-instance attribute; the ENI cap is derived from it rather than stored, so a corrected limit table applies on the next read. An instance whose type has not been reported is not gated at all: a recognised type always has at least one slot, so zero can only mean unknown, and refusing every awsvpc placement against an older agent would turn a mixed-version cluster into an outage.
Both placement paths gate on it. A task declined for want of an interface is reported as RESOURCE:ENI, spelled as AWS spells it, and separately from a plain capacity refusal. The caller-visible detail is a fixed sentence; the wrapped error keeps its identifiers and server error codes in the log, where someone can act on them.
DescribeInstanceTypes now reports NetworkInfo.MaximumNetworkInterfaces, from the same table the slot allocator enforces, so a caller sizing an autoscaling group has something to size against. Fields this stack does not implement are left unset rather than guessed.
Finally, the daemon records the slot it actually allocated as the ENI's device index. The requested index was stored verbatim, so every task ENI on an instance reported index 1 while landing on a different PCIe slot; on AWS a device index is unique per instance.
* fix(ecs): send an ENI capacity refusal as the reason alone
AWS returns RESOURCE:* placement failures with no detail. Carrying a sentence
there gives a caller a second string to parse that says nothing the reason does
not already say.
CreateTags is a read-modify-write of one resource's whole tag map. It was
serialised by a sync.Mutex on the service struct, which exists once per
process, while the handler answers on the spinifex-workers queue group and so
runs on whichever node NATS picks. Two nodes read the same map, each added its
own key, and the later write won. Every call returned success.
Six parallel writers against one security group on a three-node cluster left
four tags. On a single node the mutex hides it entirely, which is why no test
caught it.
The tag index moves from one S3 object per resource to a JetStream KV bucket,
where the merge runs inside kvutil.Update and commits at the revision it read.
A writer that loses the race re-reads the winner's keys and adds its own on
top. S3 could not host the fix: predastore's PutObject has no If-Match, so
there is nothing to compare and swap against.
Tags written before the move are not migrated in a batch. A batch that
half-failed would leave a resource looking untagged, and the next write would
commit that as the truth. Instead the CAS seeds a missing entry from the old
object inside the attempt, so a resource migrates on its first write and a
failure to read the old object fails that one call. DescribeTags reads the
union of both key sets, preferring the bucket, and DeleteAllTags clears both so
old tags cannot reappear through the fallback.
The process-local mutex is removed rather than left beside the CAS.
The regression test drives six separate service values over one bucket,
because that is what makes it able to fail: with the old code every caller
inside one process was already serialised, and only a second node could lose an
update.
kvutil.BucketNames ranged the KeyValueStoreNames lister channel using whatever context the caller supplied, and the EKS desired-set builder supplies one that lives as long as the daemon. A lost stream-names reply therefore left the channel open with nothing ever sent on it and parked the caller for the life of the process. kvutil.Keys, immediately below it, already bounds exactly this case and says why; BucketNames never got the same treatment.
The DNS reconcile made that hang cluster-wide. A pass takes the reconcile lease before computing its batch and releases it on a deferred call, and the lease renews on a background timer for as long as the holder lives, so a pass that never returns holds it forever and the other nodes never take over. Observed on a three-node cluster: the zone object stopped being written after a rolling deploy, and every name published afterwards stayed NXDOMAIN until the holder was restarted.
The listing now carries its own deadline and reports a timeout as an error, never as a short list, because callers prune on absence. The pass bounds the batch computation as well, so any other deadline-free call reached from the desired-set builders costs a cycle rather than the cluster. Only the read side is bounded: it has no side effects, so abandoning it is safe, while the apply stays inline under the lease.
ListTasks read only the cluster and returned every task record under it, discarding DesiredStatus, Family, ServiceName, StartedBy and ContainerInstance. On a cluster with one running task and a hundred stopped ones, both status filters returned the same hundred-and-one ARNs.
A taskFilter now carries the input's matchable fields and is applied per record. An omitted desiredStatus means RUNNING, as it does on AWS, so an unfiltered call no longer hands back everything the cluster has ever run.
recordTaskState moves a self-exited task's desired status to STOPPED alongside its last status. Without that the filter would be honoured and still wrong: a task that died on its own would answer the RUNNING filter for as long as its record survived.
Container-instance counts came from len(PlacedTasks), which is the set of tasks still owing capacity rather than a status tally, so a PENDING task counted as running and a task whose release did not complete counted as running forever. PendingTasksCount was never set at all. Both are now tallied from the task records, the way the cluster-level counts already were. DeregisterContainerInstance passes zero deliberately — the instance is gone, and counting there would report the tasks it just force-stopped.
launchType is not filtered because TaskRecord has no launch-type field to match on, and maxResults/nextToken are left alone because no List call in this package paginates; adding a token scheme to one would leave the other four inconsistent.
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs
ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.
The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.
admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.
Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.
* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain
ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.
The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.
Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.
* chore: pin northstar to the merged set-valued write path
* fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host
DescribeRegions hardcoded https://localhost:9999 as the Region's endpoint
regardless of where the gateway actually runs, so a workload doing endpoint
discovery through this API was pointed at its own loopback instead of a
reachable address. The endpoint is now resolved by the caller from
GatewayConfig.RegistryHost/RegistryPort, the same advertised host:port ECR
already uses (concrete AWSGW bind host, else AdvertiseIP), and passed into
DescribeRegions rather than invented there. Falls back to localhost:9999
only when no concrete host is configured, which is also the only case where
that value is actually correct.
Closes mulga-0bqih.
* style: trim DescribeRegions endpoint comments
* feat(imds): serve block-device-mapping from instance block device data
The aws-ebs-csi-driver's IMDS client fails outright on this 404,
aborting IMDSInstanceInfo before it reaches the Kubernetes fallback the
driver otherwise uses. Serve ami, root, and one ebsN per attached EBS
volume from the instance's existing BlockDeviceMappings, root device
first, matching how the driver counts "ebs" occurrences in the listing.
* style: trim block-device-mapping comments
AuthorizeSecurityGroupIngress and AuthorizeSecurityGroupEgress rejected any
IpPermission carrying an Ipv6Ranges entry. The security-group module defaults
egress_ipv6_cidr_blocks to ::/0 and the provider revokes the group's own
default egress before authorizing the replacement, so the rejection left the
group with no egress rules at all, which is treated as deny.
SGRule gains CidrIpv6. The rule is stored, deduped, revoked and projected like
any other, and the two policy.Rule adapters drop it before the ACL builder,
which is IPv4-only. No interface here has an IPv6 address, so the rule matches
nothing - which is what an IPv6 rule does on AWS in a VPC with no IPv6 CIDR.
Revoking an IPv6 rule that does not exist now returns InvalidPermission.NotFound
rather than succeeding as a no-op. The no-op existed only because authorize
refused to store the rule, so a caller's state could never match.
Closes mulga-6bu36.
DescribeTasks returned an empty attachments array for every STOPPED awsvpc task. On env19, 134 of 135 tasks in a cluster reported attachments: [], and only the one PENDING task carried an ElasticNetworkInterface block. On AWS a stopped task keeps its attachment for the retention window, with status DELETED and the networkInterfaceId still readable, because it is part of the forensic record of a task that has stopped.
taskToAWS gates the attachment on ENIID being set, and the ENI release retry had made a successful release clear that field so a non-empty ENIID would mean "still owed" and the sweep could retry from it. The signal was right; carrying it in the emptiness of ENIID was not, because the same field is the projection's only source.
The state moves to an ENIReleased flag of its own. ENIID stays put, so the attachment survives as long as the record does, and the sweep's owed test, its refusal to prune, and the placement rollback all read the flag instead. A released record ages out on retention exactly as it did before the retry landed. The attachment reports DELETED once released rather than echoing the task's own status.
This also restores the evidence path the ENI-leak probe depends on: it follows a stopped task to its ENI, and with no attachment to follow it had been reporting skipped on every run.
Closes mulga-5bnx3.
Releasing an awsvpc task ENI was best effort: reclaimTaskENI logged a failure and returned, and nothing ever tried again. Its own comment said the ENI fields stayed on the record for a later retry, but no later retry existed, and the stopped-task sweep pruned the record an hour later without looking at those fields — deleting the only pointer to the leaked interface. An orphaned task ENI still references its security group, so DeleteSecurityGroup fails with DependencyViolation and no awsvpc stack can be destroyed without deleting the ENIs by hand.
A non-empty ENIID on a stopped record now means exactly one thing: this ENI is still owed. reclaimTaskENI clears the identity when Release succeeds and leaves it when it does not, forceStopTask reclaims before its single write so the cleared fields land in it, and the stopped-task sweep gains the job of paying the debt off — retrying with a backoff that doubles from 30s to a 15m ceiling, capped at twenty attempts per pass because Release is a 30s NATS round trip and a bucket full of them would stall the sweep for the whole fleet.
A record that still owes an ENI is never pruned, however far past retention it sits. It is the only pointer to the interface, so dropping it converts a recoverable leak into a permanent one.
The placement rollback had the same hole from the other end: an attach failure cleared the ENI fields unconditionally and RunTask never persisted the record, so a failed release there leaked with no trace at all. It now clears only when the release succeeded, and otherwise persists the record as STOPPED so the sweep owns it like any other.
Closes mulga-1ysjx.
An AMI imported on a node whose timezone is not UTC was stamped with local wall-clock time carrying a Z suffix, so on a +1000 node an 11:42Z import reported 21:42Z — ten hours in the future. The layout "2006-01-02T15:04:05.000Z" treats the trailing Z as a literal character rather than a zone directive, so Format printed whatever zone the value carried and then asserted UTC regardless.
DescribeImages now converts before formatting, which also corrects records already stored, since the stored time.Time still carries its original zone. The four sites that stamp a new image now store time.Now().UTC() as well, so the value is right on both sides of the round trip.
Terraform's most_recent sorts on this field, so before this an AMI imported on a +1000 node outranked a genuinely newer one imported on a UTC node for ten hours.
Closes mulga-il7w1.
DescribeServices returned no availabilityZoneRebalancing field, so the terraform provider read the computed attribute as empty and planned an in-place update to DISABLED on every plan, including one taken immediately after a clean apply.
The pinned aws-sdk-go v1.55.8 is the last v1 release and its ECS shapes predate the field, so it can be neither parsed from an input nor set on an output through the SDK types. Embedding ecs.Service in a wrapper does not work either: jsonutil nests an embedded struct rather than flattening it, so the service arrives under its own key.
The field is read off the raw request body and written onto the encoded response instead. checkAZRebalancing refuses anything but DISABLED with InvalidParameterException, because nothing here rebalances tasks across zones and a silently dropped ENABLED would report a service as rebalancing when nothing does. addSDKGapFields runs from WriteJSONResponse so every service-carrying action is covered from one place, decodes with UseNumber to keep the epoch-second times jsonutil emitted, and leaves a value already present alone.
Closes mulga-t239t.
oci.WithTTY put Terminal in the OCI spec while the task was created with a cio
creator that asked for no terminal, so the shim made no console socket and runc
refused to start the container at all. The terminal flag now reaches the FIFO
set, and a terminal carries no separate stderr stream.
* fix(ecs): enforce or refuse container runtime fields (TFC-0040)
user, mountPoints, volumesFrom, readonlyRootFilesystem, privileged,
pseudoTerminal, interactive, linuxParameters, systemControls,
startTimeout and stopTimeout registered successfully but were never
stored, sent to the agent, enforced, or echoed back on describe.
Store and echo the fields the agent can enforce (user, readonly root
filesystem, privileged, pseudo terminal, interactive, systemControls,
capability add/drop, start/stop timeout), apply them through containerd
via OCI spec options and a graceful Stop, and refuse at
RegisterTaskDefinition the fields the agent cannot honor (mountPoints,
volumesFrom, linuxParameters.devices/initProcessEnabled/
sharedMemorySize/tmpfs). Refusal only triggers on a value that asks
for the restriction; empty lists and false/unset flags are accepted
and echoed as before.
* test: move the bus wire-contract test to an external package
* fix(ecs-agent): release an interactive container's stdin so its copier exits
* fix(ecs): return a caller-supplied empty collection instead of omitting it
CreateNetworkInterface accepted a caller-supplied PrivateIpAddress
verbatim, with no subnet-range check and no IPAM reservation, so a
direct call could collide with a concurrent AllocateIP handing out the
same address. Add IPAM.ClaimIP, a CAS-guarded sibling of AllocateIP
that validates the address against the subnet and rejects it if
already allocated, and wire it into CreateNetworkInterface with the
matching AWS error codes.
RunInstances --private-ip-address silently returned a different,
pool-assigned address instead of honouring the request. Forward
PrivateIpAddress (including the NetworkInterfaces[].PrivateIpAddresses
primary entry) into the auto-created ENI's CreateNetworkInterfaceInput,
and reject a pinned address on a launch with MaxCount > 1 since AWS
only allows it for a single instance.
* fix: decompress single-file zst/bz2/xz/gz disk images into tmpdir
ExtractDiskImageFromFile only handled tar archives and a broken plain
.xz branch that ran xz -dk beside the source and then tried to parse
tar chatter for a path, so it silently returned an empty string. Talos
(.raw.zst, .raw.xz), Fedora CoreOS (.qcow2.xz, .raw.xz) and Flatcar
(.img.bz2) publish only single-file compressed artifacts, so none of
them could be imported.
Add streaming decompression for .zst, .bz2, .xz and .gz that writes
into the caller's tmpdir and returns that path directly, instead of
inferring it from tar output. A decompressed .qcow2 payload now flows
through the existing qcow2-to-raw qemu-img conversion via a shared
finalizeDiskImage helper. zstd uses klauspost/compress, already a
transitive dependency via containerd, for a pure-Go streaming decoder.
bz2 and gzip use the standard library. xz shells out to the xz binary
since no pure-Go xz decoder is already a dependency, and the existing
.tar.xz branch already requires that binary on the host via tar's -J
flag, so this adds no new host requirement.
* fix: write sparse holes when decompressing single-file disk images
Verified live on env13: a Talos raw.zst imported successfully but the
decompressed output was written literally rather than sparsely, an
11 GiB file on disk for content that decompresses to 183 MiB with
xz -dc | dd conv=sparse. This is a regression relative to the
qcow2-to-raw path beside it, which already writes sparse via
qemu-img convert.
decompressToTemp now reads in 1 MiB blocks, seeks past all-zero
blocks instead of writing them, and truncates to the final size so a
trailing zero run is still recorded as a hole. 1 MiB amortises the
read/seek syscall overhead over a multi-GiB image while still being
small enough to catch the large contiguous zero runs that dominate a
sparse raw disk; a zero run shorter than one block is written
literally, which is the safe direction for a false negative to fail
in.
zstd, bzip2 and gzip all route through decompressToTemp, so all three
gain sparse output from this change. The plain-.xz path shells out to
the xz binary with output piped straight to a file and does not go
through decompressToTemp, so it is unaffected by this fix and remains
a known follow-up.
* fix: make plain-.xz decompression sparse too
decompressXz shelled its output straight to a file, bypassing the
sparse-hole handling just added to decompressToTemp for zstd, bzip2
and gzip. .raw.xz is the format Talos and Flatcar actually publish
alongside .zst, so this path is the one most likely hit in practice,
and left dense the exact disk-size regression this fix targets still
fires for it.
exec.Cmd already runs its own copier goroutine when Stdout is not an
*os.File, and Wait joins it. decompressXz now hands the xz process an
io.Pipe write end and reuses decompressToTemp on the read end, so
plain .xz gets identical sparse-hole handling to the other codecs. A
separate goroutine closes the pipe with the process's exit error once
Wait returns, so a read past the end of a failed decompression
surfaces that error, stderr included, instead of a bare EOF; the
error path also closes the read end explicitly so a stuck copier
cannot wedge Wait forever if decompressToTemp gives up early.
AssociateAccessPolicy recorded a policy association but never granted
it in the cluster. Bake static ClusterRoleBindings for the mulga:
groups into the eks-node image, project a principal's cluster-scoped
associated policies onto those groups in the TokenReview webhook, and
reject namespace scope at association time since there is no
namespace-scoped RBAC writer.
* feat(iam): evaluate trust-policy conditions and accept aws:SourceAccount
evalTrustPolicy read Effect, Action and Principal but never Condition, so a
stored aws:SourceAccount clause was a control nothing enforced. Give it a
condition arm evaluated in both the Deny and Allow passes, following the
StringEquals-only, fail-closed-on-unknown-key shape
evalTrustPolicyForWebIdentity already uses. The value is the caller ARN's own
account, derived server-side rather than caller-asserted.
Relax validateWebIdentityCondition from a single-action gate to a per-action
allowlist: sts:AssumeRoleWithWebIdentity keeps its existing IRSA keys, and
sts:AssumeRole now accepts aws:SourceAccount under StringEquals. Everything
else still fails at write time, naming the operator, key and action rather
than a bare MalformedPolicyDocument, and CreateRole/UpdateAssumeRolePolicy now
carry that reason to the caller instead of swallowing it into slog.Debug.
Extend the identity-policy refusal at validateStatementRestrictions with the
reason for aws:SourceArn and aws:RequestTag/*, the two keys a companion change
records as divergences in rfc0-7_aws-divergence.md.
* test(iam): fix trust-policy reason tests after aws:SourceAccount now valid
PR #1046 merged into dev with fixtures asserting aws:SourceAccount plus
StringEquals on sts:AssumeRole is refused. Part B of mulga-yxd9p makes
that combination valid, so the fixtures now exercise an unsupported
action and an unsupported key instead.
* docs: drop bead and plan-decision references from test comments
Bump bluebottle for the new key. requestConditionKeys now maps this
door's internal principal-type spelling to the canonical AWS value
through the bluebottle constants, and omits the key for an
unrecognized type rather than supplying an empty string.
ValidateTrustPolicyDocument builds precise errors naming the statement index
and the offending element, and both callers discarded them into slog.Debug
and returned a bare MalformedPolicyDocument. Debug is below the default
level, so the reason reached nobody: not the caller, and not the operator
reading the gateway log.
The contrast with the identity-policy path was stark. CreatePolicy names the
operator and the key exactly; CreateRole answered only 'The policy document
is malformed.', so an operator could not tell whether Principal, Action, a
size limit or a Condition block was at fault. The trust gate is the one the
terraform ECS module hits, which is what made this worth fixing on its own.
Both sites now use awserrors.Errorf, the same mechanism the identity path
uses. The separate Debug line is dropped rather than duplicated: the gateway
already logs a returned error at Error level, so one message now serves both
the client and the operator.
* feat(ecs): populate service event ring on DescribeServices
Events was always projected as null since ServiceRecord carried no event
history. Add a capped, newest-first event ring populated at the rollout
transitions the reconciler already computes: steady state reached, a
circuit-breaker trip, and a placement shortfall it cannot resolve this pass.
Emission is edge-triggered on the state change already guarding each site, so
the repeated reconcile loop does not refill the ring with one repeated
message every pass.
* feat(ecs): project createdAt on services and deployments
CreatedAt was already computed and persisted on ServiceRecord and Deployment
at creation time, but serviceToAWS never echoed it, so DescribeServices could
never answer which build wrote a given record. Project ServiceRecord.CreatedAt
and Deployment.CreatedAt/UpdatedAt, omitting the key entirely for a legacy
record whose timestamp predates the field rather than emitting the 1970 epoch.
CreateCluster refused accessConfig.authenticationMode = API_AND_CONFIG_MAP,
which is both AWS's default since 1.23 and the terraform-aws-eks module
default, and named no parameter in the refusal. The mode is now stored on
the cluster record and echoed by DescribeCluster, so a caller does not see
a permanent access_config diff on every subsequent plan; bare CONFIG_MAP
still fails, since Spinifex has no aws-auth ConfigMap path. The refusal
now names accessConfig.authenticationMode via awserrors.Errorf.
nodegroupRecordToAWS also only set Health when a node group had a status
reason, so a healthy node group described back with health: null instead
of the empty-issues object AWS always returns. Health is now always
constructed with a non-nil issues slice.
The unserved-service gate ran before signature verification but answered
SignatureDoesNotMatch, falsely blaming credentials that were never checked.
It now returns InvalidAction, the same code an unimplemented action on a
served service already gets, so the behaviour is consistent rather than
novel.
The envelope was also always XML, because jsonErrorService only lists
scopes the gateway serves and an unserved scope is by definition absent
from it. A JSON-protocol caller such as sqs or dynamodb therefore could
not deserialize the body and never saw the true error. writeSigV4Error now
falls back to the request's own X-Amz-Target and Content-Type headers to
pick JSON when the scope is not in that list, leaving every served scope's
envelope selection unchanged.
RegisterTargets read the target group, appended in memory and wrote the whole record back with no revision guard, so two concurrent registrations lost one. Terraform parallelises by default, so any target group with more than one member hit it, and the apply exited 0 with half the fleet not behind the load balancer.
Every writer of the record now goes through the store's existing CAS rather than a wholesale Put, because a CAS loop on one writer is defeated by a non-CAS writer on the same key. Contention stays per target group; no global lock is introduced.
RequestCertificate returned InternalErrorException for a missing tenant CA, which is a client-actionable precondition rather than a server fault. Worse, that code is classified retryable, so the provider backed off and retried instead of failing, and a single certificate sat at Still creating for over nine minutes. It now returns ACM's not-found code, which is terminal, and carries a message naming the missing CA and how to create one.
ListCertificates omitted CertificateSummaryList entirely on an empty account, since the SDK marshaller drops a nil slice unconditionally.
Closes: mulga-c6khe
Closes: mulga-wbjal
Closes: mulga-l6yft
Closes: mulga-61h3h
* fix(iam): persist policy tags and count attachments across all principal kinds
CreatePolicy built its record with an empty tag list and never read input.Tags, and GetPolicy, ListPolicies and CreatePolicy's own response projected no Tags field at all. Both halves are fixed, matching the role path.
buildAttachmentCounts scanned only the users bucket, so a policy attached to a role or group counted as unattached. That is not just a misleading number: DeletePolicy gates its conflict check on the same count, so an attached policy could be deleted, leaving the principal holding a dangling ARN.
The count and the reverse lookup are the same traversal with the names kept or discarded, so both now derive from one buildPolicyAttachments over users, roles and groups. Nothing is stored, so no counter can drift from the attachments it counts.
ListEntitiesForPolicy is implemented on that traversal, honouring EntityFilter and PathPrefix.
Closes: mulga-fvhlc
Closes: mulga-9i6uh
Closes: mulga-ryu30
Closes: mulga-8ranb
* test(iam): account for ListEntitiesForPolicy in the cross-tenant and policy-ARN gates
The action resolves PolicyArn through the same branch as GetPolicy, so it
belongs with that helper's siblings; it was in neither gate, which is what
the accounting test exists to catch.
* fix(ecr): store and project repository encryption, scanning and tags
CreateRepository parsed none of encryptionConfiguration, imageScanningConfiguration or tags, so none were stored and none came back. Both blocks are now parsed, persisted on RepoMeta and projected, defaulting to AES256 and scanOnPush false as AWS does.
encryptionType KMS is rejected rather than echoed. Predastore seals every fragment with AES-256-GCM under a required master key, so reporting AES256 states something true, but no customer-managed key material is plumbed anywhere and accepting a KMS key ARN would assert a protection that does not exist.
Repository tags are stored on the same record, and TagResource and UntagResource now work instead of returning NotImplemented.
PutImageScanningConfiguration gets its own handler: asking for scanning off succeeds as a no-op, which is the already-true state, while asking for it on still returns OperationNotSupportedException because no scanner exists. Reporting scanning as enabled when nothing scans would read as a false all-clear.
The AWS Repository shape was built in three places with no shared helper, so every new field would have been added three times. They now share one buildRepository projection.
Closes: mulga-ywvtu
Closes: mulga-j2y54
Closes: mulga-3whvb
* fix: accept and store repository tags on ECR CreateRepository
The AWS API takes a tags parameter on CreateRepository; the gateway accepted
the request and discarded them, leaving the repository untagged with no error.
Decode them into the RepoMeta tag map at create time, rejecting an empty key
the same way TagResource does.
DescribeTaskDefinition now echoes portMappings[].name and runtimePlatform,
DescribeServices projects the stored networkConfiguration and
enableECSManagedTags, DescribeClusters returns the four instance, task and
service counts, and the five list operations return their ARN key as a
present empty list instead of omitting it.
UpdateService now honours networkConfiguration and enableECSManagedTags so a
changed value converges instead of producing a permanent diff.
The eleven container runtime fields the task definition also drops are
deliberately left absent: nothing between the daemon and the on-host agent
carries them, and reporting a container as privileged or read-only-root when
nothing enforces it is worse than reporting nothing.
Closes: mulga-0t9q2
Closes: mulga-7thn7
Closes: mulga-d8kif
Closes: mulga-5z8z9
Three defects at the gateway's HTTP error boundary, none of which change an authorisation outcome.
xmlErrorBody defaulted to the EC2 query envelope, so any scope it did not enumerate, including every service the gateway does not serve, received a shape a REST-XML client cannot deserialize and the real code never surfaced. The generic ErrorResponse is now the default and the query-protocol services are named explicitly. An unparsed signature keeps the EC2 shape, since no service is known there and that path is not what this changes.
The supportedServices gate logged the truthful reason and then emitted the canned credentials text, because writeSigV4Error had no parameter to carry a message. It takes one now, and the gate names the service it refused. The signature is never verified at that point, so blaming the caller's credentials was actively misleading.
ctxAction was set only for the query protocol, so every JSON-1.1 request shared the throttle bucket "<account>:unknown" and no per-action limit could ever match. It now resolves from X-Amz-Target, and for the path-routed rest-json services from the route's own lookup function.
Also sets X-Amzn-Errortype on the four JSON emitters, which JSON-protocol SDKs read to resolve the modelled exception type.
Recovery tested whether the QEMU process was still running before it looked at the instance's state, so a live process short-circuited the transitional-state handling and reconnectInstance then set the instance back to running. A stop or terminate the caller had already been told was under way was silently discarded.
Check for StateStopping and StateShuttingDown first, kill a QEMU that outlived the transition, and finalize. reconnectInstance now refuses a transitional instance outright rather than promoting it.
Whether br-mgmt is a per-host island or one L2 segment spanning the cluster
decides whether a system instance can reach the gateway address it was baked
with. No artifact recorded which one a run built, and the environment is
destroyed before it can be asked, so the decisive evidence for an LB agent
heartbeat failure was gone by the time anyone looked.
Records ports, address and fail_mode per node, then probes every node's
br-mgmt address from every node. Islands reach only themselves and spanned
segments reach everything; a mixed result is the failure this catches.
* fix: carry a handler's own refusal message to the client without leaking wrapper context
A daemon that explains a refusal had that explanation discarded one step short of the client, so every NATS-backed error surfaced as the generic ErrorLookup blurb for its code. The message crossed NATS intact; resolveErrorDetail then dropped it because decodedNATSError was not the shape it recognises.
Rebuild the decoded response through awserrors.Errorf so the walk finds the message, and send only ResolveErrorDetail's curated message from the two encode sites instead of err.Error(). The latter rendered every %w layer, including context a call site added purely for its own logs, so restoring the message without also fixing the encode side would have turned a fidelity bug into an information disclosure.
Two existing tests pinned that wrapper context onto the wire, directly contradicting the gateway's own no-leak assertion; both are rewritten to pin the sanitized contract instead.
Also name the rejected filter in the twenty-four ParseFilters sites that logged the detail and returned a bare code, and give the ACM force-renew refusals a registered code so their reason still reaches the caller.
* fix: stop the filter rejection repeating its own error code to the client
ParseFilters spelled the code into its message text, so once the code began travelling separately the client saw it twice: "InvalidParameterValue: InvalidParameterValue: The filter 'x' is invalid".
Give ParseFilters the code directly via awserrors.Errorf and have its twenty-five callers return that error unwrapped rather than re-wrapping it. All of them already mapped it to InvalidParameterValue, so nothing changes but the duplication.
* fix: surface why a security group rule was rejected
ipPermissionsToSGRules already names the reason, "IPv6 rules are not supported" or "IpPermission must specify at least one IpRange or UserIdGroupPair", and both Authorize call sites logged it and returned a bare InvalidParameterValue, so the caller saw only the canned blurb.
This is the case mulga-e9qqx names specifically. It does not make IPv6 rules work; it makes the refusal legible.
CreateCluster and CreateNodegroup accepted fields that DescribeCluster and DescribeNodegroup never returned. The unmodified terraform-aws-modules/eks/aws module reads those fields back, finds them empty, and either aborts the apply or proposes a destroy of a cluster that is already correct.
Two of the omissions aborted the apply outright. A module precondition on data.aws_eks_cluster failed with cluster_service_cidr is "" because kubernetesNetworkConfig was absent, and the tag resource address built from clusterSecurityGroupId came out as (,suite) because that field was empty.
Store what the caller asked for on ClusterMeta and NodegroupRecord at create time, and project it back. Both records are encoding/json with omitempty, so records already in KV stay readable and project exactly as they did before.
The read-backs echo the request rather than synthesising an AWS default, because UpdateClusterConfig is not implemented — a value the caller did not ask for would be a diff they could never converge. Where a field does report a default it is the value genuinely in force: 10.43.0.0/16 is k3s's own service CIDR, API is the only authentication mode implemented, and logging reports every type disabled because no control-plane log export exists.
resourcesVpcConfig.securityGroupIds previously overwrote the caller's own groups with the platform's pair, which live in a VPC the caller cannot see. It now echoes the caller. The customer-facing cluster security group is created in the caller's account and VPC at cluster-create time, so the id reported by clusterSecurityGroupId resolves for the tenant that owns it. That creation is idempotent with the one launchNodegroupInfra already makes, and matches AWS, which creates the group with the cluster.
updateNodegroupConfig accepted UpdateConfig and Taints, ignored both, and returned SUCCESSFUL. It now applies them under the existing compare-and-swap, so a plan converges instead of asking forever. Caller-supplied labels and taints still never reach kubelet; that gap predates this change and is tracked separately.
Verified on env19: the module applies clean and a re-plan reports no changes.
Six defects in the EC2 describe path, found by running unmodified Terraform Registry modules against a live cluster and planning immediately after a clean apply. They share one shape: the write succeeds, the read-back is incomplete, and the caller cannot tell.
Populate Association on the primary network interface. The public IP was projected onto the instance but never onto the NIC, so the AWS provider read associate_public_ip_address as false for an instance that had a working public address. That attribute is force-new, so a no-op plan proposed destroying and recreating the instance.
Stamp EnclaveOptions. AWS always returns it, and its absence rendered as a permanent enclave_options block in every plan.
Stop emitting empty Reservations. The reservation map entry was created before the filter predicate ran, so a filter that excluded every instance in a reservation still emitted the reservation with no instances. Projection and filtering now run before the insert, in both the live and the KV path.
Merge reservations by ID in the gateway. Response frames were concatenated rather than merged, so one reservation could appear twice in a single response and which copy landed first varied between calls.
Mint the reservation ID once per RunInstances call. Each node minted its own, so a launch spread across nodes came back as several reservations. The gateway now allocates the ID and passes it to every node in a NATS header; a node that does not read the header mints its own as before, so the change is additive in both directions.
Answer an incomplete DescribeInstances sweep with a retryable ServiceUnavailable. Suppressing a false InvalidInstanceID.NotFound when a node had not answered was correct, but no branch was ever written for that case, so the caller received an empty 200 that read as authoritative. A filters-only query with no matches is still a legitimate empty result.
Accept the AWS spellings of the DescribeVpcs filters. The filters were implemented but named cidr-block and is-default where AWS uses cidr and isDefault, so Terraform's aws_vpc data source was rejected outright. Both spellings now work, since an internal caller depends on the old one. The parse error also names the offending filter to the client instead of only the log.
Resolve image owner aliases before filtering on them. describe-images --owners self compared an alias against a numeric account ID and matched nothing; the correct resolution already existed but ran after the filter.
Also builds the segscan test oracle with -buildvcs=false. That is unrelated to the above and fails on current toolchains regardless of this change, but it blocks preflight.
Host reservations, ECR image scanning and the flow-logs integration template
were declared not applicable, and the reserved instance family with them. None
of those are ruled out: reservations and scheduled instances can be mocked over
capacity reservations the way spot instances already are, scanning needs only a
scanner behind the API, and the template needs CloudFormation.
What stays is what the platform cannot have: retired AWS features, operations
whose content is an AWS price or counterparty, and AWS-hosted services with
nothing behind them here.
The roadmap carries IPv4 pools, local zones as an installation option, and an
S3 gap closure naming inventory and intelligent tiering. All three were
published as operations this platform will never serve.
Twenty-nine EC2 operations — local gateways, CoIP pools, BYOIP and the zone
group opt-in — and nine S3 operations are no longer declared. Carrier gateways
stay, as a route into a telco's 5G network in a Wavelength Zone.
The internal report listed every unimplemented operation together, so the 87
EC2 operations declared not applicable sat in the same list as the work, where
anything reading it for candidates would pick them up.
They are now listed apart, grouped under the reason they were declared with,
and the counts line reports them as not applicable rather than as gaps.
EC2 published one not-applicable operation against 625 in the model, so the
page was silent about a large surface that is absent for a structural reason
rather than an unfinished one.
Eighty-six more are declared across nine families: EC2-Classic, the 2016
resource ID migration, FPGA images, Elastic Graphics, Outposts and the edge
zone gateways, capacity purchasing, AWS backbone telemetry, Marketplace
product codes and BYOIP. S3 loses its replication and analytics declarations,
which named work that is not ruled out.
Multi-AZ clusters, read replicas and point-in-time restore were declared not
applicable, which says the platform will never serve them. All three are
planned, so the pages claimed something the roadmap contradicts.
Only a declaration now publishes that claim. A handler that refuses an
operation today establishes nothing about tomorrow, so it reads as the gap it
is rather than a permanent one.
Splitting them into a second table repeated a reason on every row that shared
it and separated an operation from the rest of the surface it belongs to. One
table carries every operation the platform answers, served or refused, and the
reasons return to numbered footnotes below it.
S3 was the one served surface with no coverage page. The gateway has no
operation-name dispatch table for it, so there was nothing to compare against
the model.
Predastore now declares the operations each of its S3 routes answers, so the
generator compares that inventory the same way it compares the other nine
services. The page credits Predastore rather than Spinifex, and its intro
separates what is routed from what conforms, pointing at the ceph/s3-tests
report for the second.
The index listed a count per service and left the reader to add them up. It
now opens with the total and closes the table with it, counting only the
services it links.
The coverage pages led with a percentage and an "x of y" count, and every
table listed the modelled operations Spinifex does not implement, which reads
as a scorecard against AWS rather than a statement of what the platform serves.
A published page now opens with the operation count alone, tables the
operations the gateway serves, and carries a separate not-applicable table with
each reason inline instead of footnotes. The unimplemented operations and the
counts move to docs/coverage/internal-report.md, which no docs-site slug points
at. A service with no implemented operations has no publishable page.
Every node's startup pass, the account.created event and the gateway's
EnsureDefaultVpc request can each build an account's defaults on a different
node. Each checked for an existing record, then wrote a fresh random ID with
Put, so racing nodes built a VPC, security group, route table and internet
gateway each. A per-account claim, written with Create, now fixes the IDs
before anything is built; every record is created under those IDs with
Create, and only the creator announces it to vpcd. nextVNI retries its CAS
instead of failing the loser.
spx admin init now discards the node's pre-formation JetStream store by default for --nodes >= 2; --discard-jetstream=false restores the refusal. spx admin kv digest gains --json with stream replica, retention and limit settings and per-sequence subject, time and TTL. The new spx admin kv compare reads every node's digest and fails only on real divergence: different content at one sequence, a missing message nothing explains, or wrong replica placement. install-node.sh gates formation on it and gains --ipsec on|off, verified on every node. The multi-node guide documents both.
A weekend gap is two runs of evidence lost on a suite whose remaining
failures are a rate rather than a bug. GitHub queues scheduled runs behind
every other top-of-hour cron and sheds them under load, which is why
2026-09-14 never fired and why every run started 3h15m-3h45m late.
* fix: reboot a guest by asking it, not by pulling its reset line
RebootInstances was QMP system_reset alone. The guest kernel is never told,
so it does not sync, does not unmount and loses every dirty page - which is
how a reboot issued seconds after a write loses that write, and how a guest
rebooted during cloud-init can fail to come back at all.
Guests now launch with -action shutdown=pause, so a clean guest poweroff
leaves QEMU paused rather than gone. Reboot presses the power button, waits
for the guest to reach that state, then resets and resumes the same process:
no volume, tap or state churn, and no window where the guest is missing. A
guest that will not go quietly is hard-reset, which is what AWS documents.
The button is pressed again every 10s while the guest ignores it. One signal
is not a shutdown request on a guest whose acpid has not started, which is
the same boot race that produced the failure.
Two consequences handled here: a guest that powers itself off now leaves
QEMU paused, so the heartbeat quits it; and a guest launched before the flag
exits on powerdown rather than pausing, so it is reset without being asked.
* fix: answer a reboot when it is accepted, not when the guest is back
Asking a guest to shut down takes as long as the guest takes. RebootInstances
gives the daemon five seconds to reply, so a reboot that waited for the guest
reported a reboot that had in fact succeeded as InvalidInstanceID.NotFound.
Manager.Reboot now admits the reboot — the instance exists and is running — and
runs the powerdown and reset in the background, which is the contract EC2
documents. The two admission failures are still reported; nothing after them
can be, so a reset that fails is logged and left to the heartbeat.
The gateway also stops calling a NATS timeout NotFound. Only the absence of a
subscriber says anything about where the instance is, which is how the sibling
start and terminate paths already read it.
* test: prove the churn reboot by its boot ID, not by how fast it happened
The subtest waited ten seconds and then read /proc/uptime. A reboot is
asynchronous, so the guest answering at that point may be the one that has
not shut down yet, and its uptime is whatever it was before — which the test
reads as "reboot may not have occurred".
Waiting for the boot ID to change is positive proof the guest rebooted and
does not depend on how long the guest takes to go down. The uptime assertion
stays, now measured against a boot that is known to be new.
* test: prove the graceful reboot against real guests and a real engine
Two tests the reboot design asked for and the unit tests cannot give.
Cell 18 launches two guests that differ only in how they answer the power
button. One installs a shutdown hook writing an fsynced line a second, so the
file that survives measures how much of its shutdown it was allowed to run —
a guest reset instead of asked leaves none of it. The other ignores the power
key entirely and must be reset once its budget expires, with the daemon saying
so. Each variant asserts the evidence only it can produce, and the slow one
asserts the absence of the line the wedged one requires.
Cell 27 loads a PostgreSQL instance past its shared buffers with pgbench and
reboots it dirty. The probe is an UNLOGGED table: PostgreSQL writes no WAL for
one, so a clean shutdown checkpoints it and crash recovery truncates it
unconditionally. Rows committed normally survive a hard reset too, so the
obvious assertion would pass against the bug.
The rds suite and cell 27 both gain budget for the load phase.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly
* fix: stop probing and nudging the EIP datapath of a stopped instance's port
* fix: log the real client IP, request ID and duration_ms in the awsgw access line
* test: mark the UI proxy header test as in-package
* fix: discard pre-formation JetStream data when forming a multi-node cluster
A node that ran as its own single-node cluster carries KV streams into
formation. nats-server adopts a local stream with the same name as the
clustered one, and raft catch-up only fetches sequences beyond a
follower's last, so the conflicting prefix is never repaired and replicas
diverge permanently.
- spx admin join removes the local JetStream store once formation
succeeds. A store holding streams now needs --force, and the
identity-discard warning names it.
- spx admin init --nodes >= 2 refuses a store that holds streams unless
--discard-jetstream is given, and removes it once every node has joined.
Keeping it as the seed is unsafe: stream leadership ignores who holds
data, and unassigned local streams are deleted as orphans.
- Neither removes a store while `spx service nats` runs, --force included.
- install-node.sh passes --discard-jetstream and waits for every
`spx service` process to exit before formation.
* feat: add spx admin kv digest for per-replica JetStream consistency checks
Replicas of a stream can hold different content while raft reports them
current, and nothing in the cluster detects it. A direct get over NATS is
no way to inspect one replica: a client's request goes to a random member
of the direct-get queue group, routed replicas included.
kv digest copies this node's $G streams tree (without consumer state) to a
temporary directory, opens the copy in a private in-process server with no
listener, and prints first/last sequence, message count and a SHA-256 over
every stored message per stream. Running it on each node and diffing the
output shows which streams diverge; --seqs locates the first differing
sequence. The live store is only ever read.
* fix: let spx admin kv digest read replicated streams
A replicated stream's meta.inf carries num_replicas > 1, which a standalone
server refuses to recover, so the digest could not read any production store.
Rewrite the copy's config to one replica with no expiry and a recomputed
checksum, and test it against a three-node cluster and an expired bucket.
* test: use assert.ErrorIs for the removed JetStream store
* test: mark the JetStream store and digest tests as in-package
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: retry a recovery relaunch that lost a race with a starting dependency
A guest drain-stopped by a coordinated cluster shutdown could be lost for good.
The relaunch ran before JetStream was serving, the volume lease claim timed out,
and the failure was recorded as permanent.
The backoff this needed already existed. relaunchWithRetry retries five times,
but only on ErrMountRetryable, and mountErrRetryable recognised exactly two
viperblock state-load sentinels. A lease claim returning context deadline
exceeded matched neither, so the mount was reported as permanently failed and
the backoff never engaged once.
- Classify a lease store that cannot answer as ErrLeaseStoreUnavailable, and
include it in mountErrRetryable. A lease held by another node is deliberately
excluded: that is an answer, not an outage.
- Probe readiness rather than connectedness. checkViperblockReady tested
natsConn.IsConnected(), which is true against a server whose JetStream is
still starting; it now calls AccountInfo with a timeout.
- Reconsider a latched failure. RetryRecoveryFailed relaunches instances parked
in error by a dependency-unavailable reason, once the backing store is
confirmed ready, for a bounded number of passes, then leaves them alone.
Only mount-state and launch failures qualify; a failed reconnect or
pre-relaunch hook can mean the guest itself needs looking at.
Closes the e2e gap that made this feel covered. The node-failure tests stop the
service units directly so guests keep running and recovery is a QMP re-attach to
a live process. Nothing stopped a guest, so nothing relaunched one from a sealed
volume against dependencies that were still starting.
TestMultinodeClusterRestartResumesGuests does, including the wait for every spx
process to exit that makes it a real restart rather than a stop that returned
early.
Beads mulga-v7kp4, mulga-kyb1d.
* fix: stop every spx process by unit in the cluster-restart helper
The first CI run of TestMultinodeClusterRestartResumesGuests failed at the
process wait with "node1 still has 2 spx process(es)", and the same thing
happened by hand on a prod node.
Every service is the same binary, and the stop was driven from a hard-coded
list. StopNode names seven units and omits spinifex-northstar and
spinifex-qmp-collector, which is exactly the two survivors. The unit name now
comes from the surviving process's cgroup, so the set that stops is whoever is
still running the binary and the list cannot go stale when a service is added.
StopNode itself is left alone: its list is deliberate for the hard outage it
simulates, where the target's drain must not fire.
Also marks recovery_retry_test.go as //test:in-package — the retry policy is
unexported and the tests drive it through the seams restore_test.go uses.
* fix: stop spinifex.target in the cluster-restart helper, not the units
The second CI run reformed NATS to 2 peers of 3: node3's NATS was stopped and
never started again.
StopNode stops the service units directly and leaves spinifex.target active.
That is right for the hard outage it simulates and wrong for a real shutdown —
the target then reads as satisfied while its units are dead, and the later start
raced a stop job that was still queued.
The helper now runs what the operator procedure and update-nodes.sh run: stop
spinifex.target, wait for every spx process to exit, and additionally wait for
systemd to have no spinifex job left queued before starting. That last wait is
the one that closes this race.
* fix: wait for the right NATS peer count in the cluster-restart helper
WaitNATSPeers counts remote peers, so a healthy three-node cluster reports two.
Asking for three could never be satisfied, and the run that hit it had in fact
restarted all three NATS correctly.
The QEMU guard never fired. pgrep -x and pkill -x match comm, which the kernel
truncates to 15 characters, so qemu-system-x86_64 is stored as qemu-system-x86
and neither the -x name nor the -x alternation could ever match. The script
therefore never signalled a guest and never waited, and wiped viperblock state
out from under live guests. Match the 11-character prefix against comm, the same
rule qemuProcessPrefix already applies in vm/orphan_scan.go, and scope it to the
user the daemon runs guests as so a host carrying libvirt guests alongside
Spinifex ones does not lose the wrong set. Not -f: that matches the whole
command line, where argv[0] may carry a path.
Nothing waited for spx either. Stopping spinifex.target returns once the target
is inactive, but spinifex-shutdown.service is ordered After= the storage
services, so their stop jobs queue behind its ExecStop and NATS can still be
running. JetStream then rewrites its index after the rm and before the
verification, failing the reset on a file the script had already deleted. Add
the wait update-nodes.sh has carried since d9bc6e35.
The firewall restore was gated on a mode file the failed run had already
deleted, so a retry read the absence as "never armed", skipped the restore, and
install-node.sh honoured that and formed the cluster around an open node. Stash
the mode outside every wiped tree, and fail rather than warn when a node that
arrived armed cannot be put back.
install-node.sh had the other half of that cascade: it disarms every host to let
the OVN raft ports through and only re-arms at the end, so any abort in between
left them all open. Re-arm from an EXIT trap, and say so loudly when a host
arrives with no policy.
The wipe loop deletes files but keeps every directory, so one empty
$G/streams/KV_<name> directory per stream survived the reset. NATS enumerates
those on start and reports them as HA assets it must restore, leaving the meta
group at size=4 with no leader over hundreds of assets that have no data behind
them. Every service then fails to reach JetStream and the daemon aborts its
bootstrap.
Remove the tree outright, mirroring the predastore/cluster removal above it.
The existing docker push e2e test pushes a single 4KB scratch layer, too
small to exercise predastore's storage path for a real multi-hundred-MB
blob. Add a 3-layer (100+80+60MiB) push/pull case that verifies re-pulled
layer digests against the pushed image and directly checks Content-Length
on a raw HTTP GET of the largest blob, targeting a reported defect where
large blob GETs returned 200 with content-length: 0. Re-pull uses
`docker builder prune -af`, not just `rmi`, since buildx's build cache can
otherwise satisfy the pull without touching the registry.
* ci: benchmark Predastore across a real 3-node Spinifex env
The nightly throughput job runs three s3d processes on loopback aliases on a
single box against local NVMe, so nothing measures Predastore in the shape it
ships in: three machines, with reads crossing a real NIC between hosts.
This provisions a 3-node env the way the multi-node e2e job does, then hands
its three gates to the existing harness through PERF_EXTERNAL_HOSTS. The
harness skips its own start and stop and measures them directly, so the
workloads and analysis are unchanged and the two numbers stay comparable.
It runs on a systemd LAN runner rather than the ephemeral pool because a pool
VM sits behind the geneve overlay at MTU 1408, which would put the runner's
own network path inside the measurement.
* ci: trigger the 3-node benchmark on its own branch for a first run
A workflow_dispatch only resolves against the default branch, so this cannot
be dispatched until it is merged. The push trigger is scoped to this branch
and comes back out once the workflow is on dev.
* fix: measure as the identity the provisioned cluster actually mints
spx admin init generates a random access key pair at bootstrap and writes it
to node1's ~/.aws/credentials, so the benchmark's built-in key is one this
cluster has never heard of and every run died on the first CreateBucket.
The pair is read and used inside a single step, the way e2e-ddil.yml does it:
a step-level env: block is echoed verbatim into the run log, a shell variable
is not.
* ci: default the 3-node benchmark to the compare preset
smoke was the right default while the pipeline was unproven. It is green now,
and compare is what the nightly loopback run uses, so it is the preset whose
numbers can be set beside it.
* ci: make the cluster's ref a knob, pinned to v1.18.0 for one run
The harness stays on this branch while the cluster is built from whatever ref
is asked for, so an old release is measured with the same ruler as the current
one. run-info.txt records the deployed ref rather than the harness checkout.
The v1.18.0 default is temporary: a push event carries no dispatch inputs, and
this cannot be dispatched until it is on dev. It comes back out after the run.
* test: probe ranged GETs, which no workload here exercises
Warp reads whole objects, so the throughput figures stay healthy while ranged
GETs read the whole object per request. The probe reads the same boundary
crossing range from a 64 MiB and a 256 MiB object: under read amplification
the cost tracks object size, and otherwise the two match.
Cluster ref is pinned to the fix branch for this run and comes back out after.
* test: run the ranged GET probe against dev to prove it fails there
A probe that has only ever been green proves nothing about the fix. Pointing
the cluster at dev, which still has the defect, is the control that shows the
probe detects it. The pin comes back out after.
* ci: drop the branch push trigger and the cluster ref pin
Both were scaffolding to run this before it existed on dev. The workflow is
dispatch only again, and cluster_ref defaults to the branch under test.
All three need unexported access and cannot move to an external test
package: the daemon suite drives shared package fixtures and reads vmMgr,
resourceMgr and localStatePath, while the other two assert on buildArgs
and volumeVBConfig directly.
They were added on dev without the marker, so check-test-package.sh
failed for anyone running preflight on dev, where the base ref is
origin/main rather than origin/dev.
The controlplane egress selector also carries the authentication
token-webhook dial, which targets 127.0.0.1:8443 on the control-plane VM.
Tunnelling it through konnectivity hands that dial to a randomly chosen
agent, where nothing is listening, so the authenticator returns an error
rather than a decision and every bearer-token request fails closed with a
bare 401.
It only surfaced once a second agent existed. The control-plane node runs
an agent itself, so while it is the only backend every tunnelled dial
lands back home and authentication works by luck. The apiserver's webhook
client also holds its connection for the 90s IdleConnTimeout, so there is
one backend selection per connection rather than per request.
TestEKS/IRSAPod hit both conditions every run: it builds a kubeconfig and
then waits ~105s for its nodegroup to go ACTIVE before issuing any
request, which both expires the pooled connection and brings up the
cluster's second agent.
Only the cluster egress needs the tunnel, for kubelet exec/logs/
portforward. This matches the upstream reference configuration.
The three ci-single runners map to env3/bottlebrush, env10/banksia and
env11/wattle, and the cell landed on whichever was free. Only bottlebrush
has NVMe; banksia and wattle are Dells with none, which is where every
etcd flush measurement to date was taken.
The eks suite boots its control plane from the eks-node AMI, which only
the nightly built. Bare metal had no EKS or mkosi step at all, so naming
the suite there produced a run that could never pass.
dev did not build: the WALBaseDir call sites landed while go.mod still
pinned viperblock v1.18.0, which predates the field.
constructMountedVB passed WALBaseDir to the nbdkit plugin but not to its
own engine, so the daemon would look for WALs under BaseDir while the
plugin wrote them under WALBaseDir.
A WAL fsync shares the device queue with predastore, the chunk cache and
every other volume on the node. Measured on the e2e stack, competing
filesystem traffic alone moves fsync p99 from 4.7ms to 22ms with no
change to viperblock, which is the tail an etcd guest reports as its
commit latency.
wal_base_dir under [nodes.<n>.viperblock] routes the WAL elsewhere. It
reaches both the short-lived VBs the service opens and, through the
nbdkit argv, the plugin backing each mounted volume. Unset keeps the WAL
under base_dir.
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic
An unhealthy control plane reported readyz, etcd reachability and etcd-disk
free space. When etcd goes unreachable with disk:ok and no OOM kill, that set
cannot distinguish a starved guest from a genuine etcd fault, so add the
loadavg and MemAvailable that separate them.
An empty /readyz body also rendered as readyz:[none], which reads as "no
subcheck failed" when it means the probe never answered. Report that as
unreachable instead.
* fix: sample the node's load, not just the runner host's
Every runner agent runs on one host while their nodes sit on several, so the
existing sample measured the runner's host and was labelled as the hypervisor.
It happens to coincide when the node is on that same host and misleads
otherwise. Relabel it and add a node-side sample, which is where the guests are
actually scheduled and so where a starved guest shows up.
* feat: build the mkosi EKS image in CI and give its control plane telemetry
The Ubuntu/systemd EKS image is what customers run, and nothing built it: the
composition existed in mkosi-build.sh's image_profiles table but had no caller,
so a change that broke it was found by hand or not at all. Cell-25 builds the
Alpine image, which leaves the published one with no gate. Add a Makefile target
beside the Alpine one and a job in the existing path-filtered image workflow.
The guest also collected nothing. k3s has served embedded-etcd metrics on
127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no
collector ever read them, so an etcd stall was only ever visible as
"etcd:unreachable" in the state report — which cannot separate a slow disk from
a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already
run, scraping etcd and guest host metrics.
The unit is gated on ConditionPathExists for its endpoint file rather than
started unconditionally: whether the control-plane guest can reach the sink from
the isolated CP VPC is unverified, and a collector with nowhere to export is a
restart loop, not telemetry.
* fix: report etcd fsync latency in the control-plane health diagnostic
disk:ok measures free space, which says nothing about how long a write takes.
etcd stalls on fsync latency long before it runs out of room, and this guest's
disk is an EBS volume served by viperblock, so a fsync is a network round trip.
An unhealthy control plane reporting healthy memory and disk was uninterpretable
without it.
Derived from the histogram's sum and count rather than a bucket percentile: two
greps against an interpolation this script has no business doing, and the
difference between a healthy single-digit reading and a stalled one is an order
of magnitude, not a tail. Rides the state report's existing gateway path, so it
needs no route the CP VPC does not already grant, and lands on both images
because the script is shared.
The scrape's failure is guarded explicitly. Under set -e a bare assignment from
a curl against a port nothing answers aborts the whole diagnosis, dropping the
reason field exactly when it is wanted.
* feat: publish etcd fsync on every control-plane report, not only failing ones
A figure that appears only once the apiserver is already unhealthy has no
baseline to be read against, which is the same gap a passing cell that uploads
no journal leaves: the number is unanchored when it finally arrives. Move the
scrape into its own helper, emit it on healthy reports too, and carry it as a
typed field rather than only inside the reason string.
Slowness is logged, never gated. observe() decides cluster health, and a working
cluster on a slow disk is working — returning an issue there would fail
customers' clusters on a latency threshold. The warning is latched on the
transition instead, because the control plane republishes on its own timer and a
line every interval is how a real signal gets tuned out.
An absent sample is not a fast one. Too few fsyncs to average, an etcd serving
no metrics and an older AMI all arrive as nil, and the field is omitted rather
than defaulted so none of them can be read as healthy. The minimum sample count
is the same lesson as the slow-shard floor: a mean over three fsyncs on a
just-started etcd is a number with no meaning.
* fix: log an etcd fsync baseline and name the two no-sample cases apart
A healthy cluster logs no health lines, so the fsync figure reached the
daemon on every report and was never surfaced — a passing cell left nothing
for a later slow reading to be compared against. Log the first known sample.
Split the "unknown" sentinel into nometrics (etcd served none) and warmup
(too few fsyncs to average): different faults, different fixes. The payload
guard now tests for a number rather than the sentinel word, so a future
sentinel cannot leak into the JSON as a bare value.
* feat: measure fdatasync latency in the diskperf gate
The two pinned profiles are buffered-bypassing throughput jobs at queue
depth 32. Neither measures the cost of making a write durable, which is what
a consensus datastore pays per raft entry and stalls on long before it runs
short of bandwidth. A 5ms barrier on bare metal went unseen as a result.
syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1,
fdatasync every write, buffered psync so the sync has a page cache to flush.
fio reports the sync distribution apart from the write stream, so the parser
reads it separately — gating on write clat would score the time to reach the
cache and call it durability.
No figures captured on this hardware yet, so the baseline entry is zero and
the job records without gating.
* fix: use the real otelcol checksums asset name in the EKS server image
The checksums file is named for the release repo and the distribution and
carries no version, unlike the tarball beside it. The guessed name 404'd and
took the image build with it.
* fix: extract otelcol as root in the EKS server image
The upstream tarball records its build uid (runner/docker, 1001:118), which
does not exist in the image. tar running as root tries to restore it and
fails outright rather than degrading, taking the build with it.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
* fix: stop the e2e analyser inventing failures and let a detach outlast its seal
go-junit-report emits <error message="No test result found"> for a === RUN with
no matching result line, and attaches the test's own captured log as the body.
The analyser counted those as failures, so the baremetal cell reported 25
failures against a junit file recording 1, with log output reading as
assertions. Separate the marker from a real error in both the parse and the
roll-up, count the unresolved cases and say so in the summary.
The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount
seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a
WAL was reported as ServerInternal while it went on to succeed.
Also switch four nightly job gates from always() to !cancelled(): always()
overrides cancellation, so a cancelled run kept starting jobs.
* fix: serialise the e2e workflows on one group and retry a dropped libvirt
e2e.yml had no concurrency block at all, so every push to dev could provision
alongside every other push and alongside the nightly. All four workflows that
build on these hypervisors now share e2e-hypervisors, queued rather than
cancelled. A per-environment group is not expressible: GitHub picks the group
before it picks a runner, and the environment is a property of the runner.
tofu apply also failed outright on a transient libvirt connect, which fails a
45-minute job in 90 seconds and reports it as a test failure. retry_run retries
only when that attempt's own output names a connection failure, so a real
failure still fails once.
Also collect the unfiltered journal from the matrix cells and the node journal
from cell-21: the unit filter drops the kernel and OOM lines that explain why
one node behaved unlike its siblings, and a passing run's journal is what makes
a failing one readable.
* fix: build the baremetal junit from the unfiltered suite log
run-baremetal-e2e.sh streams the full suite output to suite-output.log and a
grep-filtered summary to its stdout, which the workflow tees into
test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents
every subtest result by four spaces, so the junit was generated from a log with
no subtest results in it: go-junit-report saw each === RUN with no outcome and
emitted <error message="No test result found">.
17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating
cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors.
* fix: converge released EIPs instead of trusting a fire-and-forget teardown
A disassociate leaves the ENI running, so the orphan sweep's owning-port test
never fires for it. The teardown then rests entirely on vpc.delete-nat, which is
core NATS and is lost whenever vpcd is restarting — and the intent KV watch is
UpdatesOnly, so the same window hides the write that would have driven a repair
pass. The dnat_and_snat row and its host /32 route to a released public address
survived until something else happened to touch them, which for a live ENI was
nothing at all.
Give the sweep the second test it was missing: AddEIP is the only writer of
dnat_and_snat, so the floating IPs intent asks for are the complete set of
addresses that may hold one, and a row for any other address is stale however
alive its port. Ports and external IPs travel together in LiveEIPs, both unioned
from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is
still spared. The host unbind now runs even when the row had already gone, which
is the state a crash between the OVN delete and the unbind leaves.
Run that sweep on the startup pass too. Startup is the pass that follows the
blind window, so skipping it deferred repair of exactly the changes that were
missed. The port-group and ENI-port sweeps stay gated there for the in-flight
create they were written for; the EIP sweep re-reads intent immediately before
deciding and abandons the whole sweep if that read fails, which is the same
protection by a stronger route.
* fix: keep a release verify off the hypervisors an e2e run is using
release.yml is the fifth workflow that provisions on the ci-single pool and had
no concurrency block at all, so a tag push could land a tofu apply on the same
libvirt host a nightly cell was mid-provision on. The group goes on the verify
job rather than the workflow: the build jobs touch no hypervisor and should not
queue behind a nightly.
* fix: stop cell-27 handing a 50-minute suite a 50-minute job
The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix
default of 50 minutes for the whole job, so provisioning, install, the suite and
teardown had to fit inside the suite's allowance alone. It never could: the last
measured run spent 6 minutes reaching the suite and was killed 44 minutes into
it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled"
and nothing else, which is why this cell has no verdict in any nightly rather
than a failure anyone could read.
cell-27 gets 75 minutes, and the select job now refuses to start a run whose job
budget does not clear its suite budget plus a setup allowance. The inner timeout
firing first is the property that matters: go test says where it stopped and
what failed, and the job timeout says nothing at all. cell-29 already had the
same fix applied by hand; nothing stopped the next suite from repeating it.
* fix: converge host EIP plumbing against intent, not against the NAT row
The released-EIP sweep landed and did not fire. Its premise was that a stranded
dnat_and_snat row is what a stranded host route can be found by, and on the path
that actually strands one there is no row left. AddEIP scrubs the predecessor
row sharing the private IP before installing the new one, so when reconcile
re-added the instance's auto-assigned address it silently deleted the released
EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table
with nothing stale in it while the /32 route and its proxy-ARP stayed up.
Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from
the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is
found by what installed it rather than by a row some other path is free to
delete. Anything plumbed that intent no longer asks for is torn down, which also
covers a crash between the OVN delete and the unbind. A failed inventory fails
the pass: an empty list and an unreadable one are indistinguishable, and reading
the second as the first is a sweep that reports success having done nothing.
Close the window inside the pass too. The predecessor scrub now reads the row it
is about to drop and releases that address's host state, so the strand does not
exist between applyEIPs and the sweep rather than merely being repaired a step
later.
* fix: retry the release verify's libvirt calls like every other job does
The verify job drives the same libvirt provider on the same hypervisors as the
e2e cells, where a dropped connection under concurrent load is the steady state
and is retried. Here it was a hard failure that reads as a broken release. It
was skipped earlier only because the job checks out mulga and not spinifex, so
the shared helpers were not on disk — a sparse checkout of .github/scripts costs
nothing and makes apply and destroy behave like their e2e counterparts.
* fix: do not report a rebooting DB instance available on the pre-reboot agent's health
RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at
the value the previous agent process reported. After a reboot the new agent
registers before it has probed anything, so the record read as a healthy engine
with a beat newer than the transition, and the reconciler moved the instance to
available while PostgreSQL was still starting up. A client that connects the
moment the status turns green then gets "the database system is starting up".
A register now discards the previous process's verdict and records when this
agent process started. The restart reconciler requires that registration to be
newer than the transition, which also closes the window where a beat lands
between the engine being stopped and the VM going down.
* test: record the credential kubectl sent when an EKS call comes back Unauthorized
The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands
before the pod dumps are registered, so a 401 there left no evidence at all. A
verbose read-only probe reports whether an Authorization header was sent, which
separates a token the apiserver rejected from one the exec plugin never made.
* fix: report a starting engine promptly, and budget CI jobs from the suite
The agent waited a full heartbeat interval before its first beat, so a restart
could not be observed as finished for up to 30 seconds after the engine was
already serving. Beat at startup instead, and report every 5 seconds until the
engine has answered once — the control plane is actively waiting on that beat,
and the steady-state cadence it hands back is for a database already serving.
The rds suite is the one most sensitive to what else the hypervisor is doing:
its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how
long it waits for the budget. The same pass ran 45m beside four nightly cells
and overran 50m beside nine, panicking with two tests still running. Raise its
budget to 60m.
That budget was then larger than the jobs meant to contain it. A job cancelled
on timeout reports only "The operation was canceled" and takes the artifacts
with it, so the inner `go test` timeout must always fire first — and it could
not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for
90m and 180m inside a 60m job since they were added. Both workflows now resolve
their job budget from suite-sets.sh rather than carrying a literal: e2e.yml
from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is
bounded by its own suite and not by the longest one beside it. The default push
set resolves to exactly the timeouts both workflows had before.
* fix: check a permutation cell's job budget against the suites it runs
The guard only saw cells that name a suite, so the nine that run a topology's
nightly subset were never checked against it — the same class of mistake the
guard exists to catch, in the majority of the cells. Bound each by the longest
suite in its subset: max() and not sum(), because whichever suite hangs must
panic on its own timeout first, and a sum would demand a budget the set has
never come close to using. Every cell clears it today.
* fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed
Three defects, all found by the first rds pass that ever ran deep enough to
reach them.
A wait that saw an instance reach `failed` called t.Fatalf from inside the
polling condition, and testify runs that condition on its own goroutine. The
Goexit killed the tick and left the collector empty, which reads as the
condition being satisfied — so the wait returned successfully on an instance it
had just declared broken, logged "reached status available" about one that was
failed, and let the test carry on against it. Six conditions across four suites
call Fatal or require the same way. EventuallyErr now runs its condition on the
test's goroutine, which makes all six behave as written.
A DB VM stop has two drivers: the API call and the reconciler resuming one whose
caller it assumes has died. When they overlap the loser is told the VM is not in
a state to be stopped — which is the state the stop was heading for — and the
customer's StopDBInstance returned InvalidDBInstanceState for a stop that had
just completed. Both paths now treat a refused command the way they already
treat an unanswered one: it says nothing about the VM, so the fleet view
settles it.
PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it
was added, so a postmaster that refuses a setting produced "engine did not
respond on /run/postgresql:5432" and nothing else — on a guest with no SSH,
whose console records only that pg_ctl could not start the server. The probe now
quotes the tail of the log the image already directs the postmaster to, and the
reboot and modify paths carry the agent's reason into the failure the way the
create path always has. The reason reaches the customer through StatusInfos.
The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is
what the DB-VM queue costs rather than what the tests do — single tests logged
11m, 22m and 26m waiting for budget — and the pass above still had three of
seventeen tests running at 60m.
* fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot
An rds-agent that failed to construct exited, so a transient IMDS credential
resolution took the guest down for good: rds-datadir and rds-init cascade on the
missing handoff, the engine never starts, and the control plane sees only silence
until it declares the instance failed ten minutes later. Construction is retried
like every other boot step instead.
OpenRC skips start_post when the start itself fails, which is the one case with
no diagnostic at all — the console carried nothing but "died". The log mirror
moves out of the handoff wait so a failed start uses it too.
A snapshot quiesced the engine and nothing else. The engine repairs its own files
from the WAL, but postgresql.conf sits beside them and it does not: a restore
came up on a torn tail and the postmaster stopped on a syntax error at the
unterminated include hook. The quiesce now flushes the whole data mount, and the
hook is renamed into place rather than appended, so a boot that finds a half
written one replaces it instead of appending after its open quote.
* fix: flush the config cloud-init wrote before a reset can take it
A DB instance rebooted seconds after it went available came back with no
RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine
never started. cloud-init writes agent.env once, per-instance, and never again;
a reboot is a QMP system_reset with no guest shutdown, so the file was still
dirty page cache when the machine reset under it and the next boot read nothing.
The handoff wait now syncs, which is the point at which the agent has proved it
read that file.
* fix: stop dropping an EIP teardown published while vpcd is restarting
DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run
34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement
subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and
went nowhere. ReleaseAddress then returned the address to the pool two minutes
later with its /32 host route still delivering to the guest that had held it.
vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on
no-responders — which returns immediately, so the cost is paid solely when
nothing is listening. A teardown that still fails is logged as an error naming
the address rather than lost silently. Neither is fatal to the caller: the
reconciler's orphan sweep remains the backstop.
vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it
on the flows barrier for up to 45s, and associating an address should not
inherit that.
Two reconcile passes in that run had the address absent from intent and did not
sweep its host route, and the journal cannot say why: the sweep logged only when
it acted, so a route it kept was indistinguishable from one it never saw. It now
logs the bindings it found, kept and wanted.
* ci: render one run-level table for the e2e nightly
Reading a nightly meant opening nineteen jobs, and the Actions UI does not
separate a cell that failed from one cancelled out from under it — a
distinction every investigation of this suite has needed, and one the run
34046316024 analysis had to reach for the jobs API to get.
A final job, needs: every cell and if: always(), renders cell, topology,
result, duration, runner and per-cell test counts into the step summary, with
run-level totals, the three slowest cells and the failing test names.
Result, duration and runner come from the jobs API so a cell that died before
writing junit still gets a row. Counts come from a new junit-only artifact per
cell rather than the full cell artifact, so the roll-up does not pull nineteen
journals it has no use for.
Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the
two cancellations shown as cancelled rather than folded into failure.
* fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path
natuplink asserted that an instance's auto-assigned public IP is still plumbed
after an EIP associated to the same instance is torn down. EC2 does the
opposite: associating an EIP releases the auto-assigned address back into the
pool, and it cannot be reused. Our datapath already does that — run 34059879666
shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was
installed — so the test was asserting behaviour neither AWS nor we have.
It now asserts the release at the point it happens, and that no public delivery
remains after the teardown. This surfaced only once the teardown itself started
working: the run's other assertions passed, including .73's removal, which was
the failure before.
Our records still disagree with our datapath — the ENI keeps the released
address, which is why termination tears the old one down twice and never the
current one. That is the remaining half of mulga-7zc9w and is not fixed here;
it needs an IPAM release on the associate path.
Separately, AuthDiagnostics read /version, which is anonymously readable, so its
200 was consistent with both branches it claimed to distinguish. It now runs
auth whoami, reads kube-system, and invokes the credential plugin directly,
reporting exit code, stderr and whether a token was produced — never the token,
since CI logs outlive it.
* test: capture per-guest state when a spread guest never answers SSH
TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects
OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The
failure it actually hits is a guest that never accepted SSH — run 34040324968,
the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port
binding and no tap state. There was no way to tell from the artifact whether the
guest booted at all.
Now dumps, per guest: the console, the NB logical switch port, the SB port
binding, the OVS interface backing its tap, and br-int's port list. Probed on
every node rather than the expected one, because a port bound on the wrong
chassis is exactly what a hosting-node-only dump cannot show.
Covers all three guests, not just the failing one, so a working guest is
available as the control.
* ci: sample hypervisor load either side of every cell
Eight runner names map onto three hypervisors, so cells that never share a
runner still share a host: when cell 10's SSH wait expired on run 34040324968,
cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing
in the artifacts measured host load, which left co-tenancy as an explanation
that could be neither confirmed nor ruled out.
Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the
QEMU process count before and after each cell, into the cell artifact.
* ci: note that main's missing concurrency block is why the nightly runs ungrouped
Three refs carry three different groupings: main has no concurrency block at
all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load
the workflow from the default branch, so every cron nightly runs ungrouped and
cannot queue against anything — which is how a dispatch of 34050630627 ran
alongside the scheduled 34046316024 and cost it cells 24 and 27.
e2e-hypervisors is the right group and deliberately broader than dev's, since
e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing
on a feature branch can fix this; the block has to reach main.
* fix: stop the detach seal, EIP teardown and key-pair races failing the nightly
Four independent faults the full-suite nightly surfaced, each of which had
been masked while the heavy cells were skipped.
unmountVolume SIGKILLed nbdkit the moment the request arrived. Removing the
guest device has already started the plugin's own Close, which drains to the
backend and leaves a seal receipt, so the kill landed 61ms into that flush and
threw it away — the receipt path never fired once across a whole run. What was
left was 74 orphaned WAL files for the fallback seal to replay, which ran past
the caller's 180s deadline and returned DetachVolume ServerInternal with the
volume still attached. It now terminates gracefully first, bounded by the
plugin's own drain budget, and only escalates if that does not take.
DeleteEIP returned early when the dnat_and_snat row was already absent, before
the neighbour flush and the host unbind. An absent row is not a reassignment —
nobody owns the address — so the /32 route and proxy-ARP entry were orphaned
and a disassociate left the address still answering from the host. The
reassignment cases keep their early return.
EnsureKeyPair was not atomic: a failure after CreateKeyPair left the pair on
the cluster with no cleanup registered, so every later caller failed with
InvalidKeyPair.Duplicate in a test that never ran. The describe that follows a
create also had no tolerance for the propagation window on a multi-node
cluster; it now retries, and still reports a duplicate the moment it sees one.
An admission refusal could not name its cause. The static vCPU/memory budget
and the live MemAvailable clamp both surface as InsufficientInstanceCapacity,
and they have different fixes; the subscription transition now records which
gate closed.
* fix: identify the default-deny by priority, not by being a drop
DenyIsNotEthertypeScoped bucketed every drop ACL by direction, so the last one
seen won each direction. The platform WAN egress block added at priority 1049
is also a drop and is legitimately ip4-scoped, so it displaced the default-deny
in the map and the assertion fired on its qualifier instead:
from-lport deny on sg_c94870c0d68288dc6 is IPv4-scoped:
"inport == @sg_c94870c0d68288dc6 && ip4 && ip4.dst != {10.0.0.0/8, ...}"
Select the two default-denies by their own priorities. What the test asserts is
unchanged — the default-deny still must carry no ethertype qualifier, for the
reason denyIngressACL states: OVN allows on no-match, so an ip4-scoped deny
lets every other ethertype through.
* fix: drain every EIP on an ENI at teardown, not just the first
findByENI returned the first matching record and stopped, so an ENI holding
more than one EIP had all but one left marked associated to an interface being
deleted — with the host route still installed and no record left naming the ENI
to find them. Teardown now clears them all.
* ci: serialise e2e-nightly runs so two do not share the hypervisors
Only iso-single and iso-multi were guarded, by the iso-qemu-rack group. Every
other job could overlap a second run of the workflow, with both sets of cells
building clusters on the same five hosts. Queue rather than cancel: a dispatch
is usually a verification run whose result someone is waiting on.
* fix: point cell-24 stress work root at the runner nvme
The fault-injection harness ignores TMPDIR for large-object and
multipart-upload, defaulting them to $HOME/.cache/predastore-e2e because
TMPDIR is commonly tmpfs. On this runner $HOME is a 110G root disk with
14G free while TMPDIR is a 938G nvme, so the preflight guard passed on a
filesystem the scenarios that need disk never touch.
Set STRESS_WORK_ROOT alongside TMPDIR and check free space on both at a
threshold that covers the 1GiBx8 multipart case, so a run that would skip
a size for want of disk fails preflight instead of reporting a pass with
less coverage.
* fix: give cell-24 room for the full thirteen-scenario stress suite
The scenarios run serially and are ~35 minutes of work on their own,
measured from the per-scenario predastore matrix. The job allowed 30, so
it could not have passed whatever the disk did.
The matrix job gated all 32 of its steps on a step output. A job whose every
step is skipped still completes green, so a dispatch that ran one cell reported
as a nineteen-job pass and was indistinguishable in the UI from a full one. The
only tell was duration: a deselected cell finished in 11-12 seconds.
Every green run in the recent history is such a dispatch, and every run that
actually executed the suite has failed. There was no known-good baseline; there
was a display bug.
A job-level if cannot fix it — matrix is not available in that context, since
it is evaluated before the matrix expands. So the selection moves ahead of the
matrix: a select job filters the cell table and the matrix is built from what
comes back. A cell that was not selected produces no job at all, which is
plainer than skipped and cannot be read as a pass.
The table moves to .github/e2e-cells.json so both the workflow and the filter
read one copy.
* feat: block outbound SMTP egress via platform OVN ACL
Add a platform egress ACL that drops outbound TCP to 25/465/587 for public
destinations on every guest, matching AWS's default outbound-SMTP block. The
control is global by default with an operator-held per-VPC exemption; tenants
cannot open the ports via a security group because the block sits above the
tenant-allow priority band. Private destinations are always exempt.
Config exposes blocked_ports_wan (nil defaults to [25,465,587], [] disables)
and egress_block_exempt_vpcs. Adds unit tests for the ACL builder, the SG
manager wiring and the config resolution, plus an e2e that digs gmail.com MX
and confirms a 5s SMTP connect is blocked while other WAN egress works.
* docs: note the multi-node sync + restart requirement for the egress block
* feat: map virtio-blk virtqueues across an IOThread pool
Boot volumes ran every virtqueue through a single IOThread. Size a pool
from the vCPU count and hand QEMU an iothread-vq-mapping so the queues
spread across it. Falls back to the scalar iothread= form for MMIO
machines, single-queue devices and a pool of one.
SPINIFEX_BLK_IOTHREADS is experiment scaffolding for sweeping the pool
size without a rebuild, and comes out before this merges.
* feat: name QEMU guests with debug-threads=on
Config.Name held the instance ID but was never emitted, so QEMU left
every thread called qemu-system-x86 and a host-side pidstat -t could
not tell an IOThread from a vCPU thread. Emit -name guest=<id> with
debug-threads=on, which is what libvirt does for the same reason.
* refactor: drop the IOThread pool-size experiment override
SPINIFEX_BLK_IOTHREADS existed to sweep the pool size on dev-prod
without a rebuild and redeploy per value. The sweep is done, so the
derived size is the only path and buildDrives keeps its own doc comment.
The public, formation and DNS accept rules carried no address family, so
on a node that autoconfigures a global IPv6 from an upstream Router
Advertisement they served the console, predastore S3 gate, AWS gateway
and DNS on that public address with none of the source scoping applied on
IPv4. Qualify them with meta nfproto ipv4 so v6 falls to the default
drop; the scoped rules are already v4-only via ip saddr.
* refactor: extract DescribeInstanceStatus selection and projection
The request semantics — the instance-ID filter, filter parsing, the visibility
check, state membership defaulting to running-only, and the status filters that
can only run against a built frame — live inside the node handler's vmMgr.View
closure, so nothing else can apply them. A second caller answering the same
request from cached records would have to restate them, and the copy that
drifts is the one no customer request exercises until it does.
StatusSelection parses the request once; StatusFor applies it to one VM and
projects, returning nil when the request does not cover it. Per-VM rather than
per-slice so the node keeps reading VM fields inside its lock exactly as before.
buildInstanceStatus takes az rather than reading s.config.AZ, because a caller
projecting another node's record must use the AZ on that record. The node
passes its own config value, so behaviour is unchanged. systemImpaired stays a
parameter: it is node-local memory pressure only an answering node can report.
No behaviour change; the twenty existing status tests pass untouched.
* feat: read node liveness from the heartbeat store
A gateway answering from cached records has no way to tell a node that is
quietly healthy from one that has died, because an idle node writes no records
and record age measures activity rather than liveness. Liveness reads the
heartbeats the daemon already publishes and judges a node stale after three and
a half missed beats.
Unknown is a distinct answer from Stale and the two must not collapse. A node
nobody has ever heard from may simply never have published, and a store the
gateway cannot reach is not evidence that anything died — both report Unknown,
and a failed read clears the previous answers rather than keeping them.
The threshold is deliberately tighter than the IPsec barrier's six beats. That
one gates a cluster encryption decision where calling a live node dead churns a
security barrier; this one decides whether to admit the control plane cannot
confirm a host, where the costly error is the opposite one. The IPsec window is
left alone. A test in the daemon asserts the publish interval and the gateway's
copy of it have not drifted, since the gateway cannot import that package.
* feat: report impaired status for instances on a node that stopped answering
DescribeInstanceStatus fans out over NATS and returns whatever the nodes send
back. A node that is down sends nothing, so its instances disappear from the
answer entirely and a dead host reads as no host.
Backfill from the instance record cache: for any cached instance no node
answered for, synthesise a status frame and mark the system status from the
heartbeat store. A stale node reports impaired; an unreadable heartbeat store
reports insufficient-data, because not knowing a host is healthy is not knowing
it is not.
A frame an answering node returned is never touched. The covered set is what
guarantees that, and it is asserted directly rather than only through the
handler, where dedupStatuses being first-writer-wins makes either guard alone
sufficient. A live node's omission is its own answer and is not backfilled, and
a cache that has not finished its first whole-set sync contributes nothing.
The zero-value StatusSynthesis disables the whole path, so a gateway wired
without a cache answers exactly as it did before.
* fix: use fan-out responder identity, not heartbeat age, to gate status synthesis
DescribeInstanceStatus previously skipped synthesising an instance whenever
its owning node's heartbeat was still live, even when that node did not
answer this particular fan-out. A node that goes down loses its instances
from the response entirely until its heartbeat crosses the stale threshold.
Switch the Gather call to ExpectedResponders so it tracks per-node identity,
and use SuccessResponders to decide whether an omission was deliberate.
Heartbeat age now only selects the label (impaired vs insufficient-data),
never whether to synthesise. Falls back to the old heartbeat-only rule when
any frame cannot be attributed to a node.
* test: tag existing status-synthesis tests with node identity
answerWith never set X-Node-ID, so every existing test in
status_synthesis_test.go silently ran down the Unidentified fallback
branch instead of the responder-set path real daemons take. Give
answerWith a node-ID parameter (reusing subscribeAsNode) and retag each
call to match the record it is about, so these tests exercise the
success-responder guard they are named for. Keep exactly one test,
via a new answerAnonymously helper, on the fallback branch deliberately.
* test: tag DescribeInstanceStatus_test.go replies with node identity
Same rot as status_synthesis_test.go: raw nc.Subscribe replies carry no
X-Node-ID, so every describe here also sat on the 3s Gather deadline.
These tests mostly pass StatusSynthesis{}, so synthesize bails at
Records == nil before the fallback branch is ever reached — the payoff
is dropping the per-test timeout, not fixing a correctness gap.
ec2rolecreds implements aws.HandleFailRefreshCredentialsCacheStrategy, which
aws.CredentialsCache prefers over returning the refresh error. When the cached
credential is expired or within five minutes of expiring and the refresh fails,
it hands back the same access key, secret and session token with Expires pushed
a random five to fifteen minutes out, reported as a success. Every later
refresh fails the same way, so ecs-agent signs with a credential the control
plane has already rejected, indefinitely, and nothing surfaces it.
The cache picks its strategies by asserting on the provider's dynamic type, so
returning aws.CredentialsProvider from NewProvider does not disarm it. Wrap the
provider in a type whose method set omits HandleFailToRefresh, and the cache
falls through to returning the error. AdjustExpiresBy is forwarded on purpose:
it only fires under an ExpiryWindow, which no caller sets today, but dropping
it silently would change behaviour for whoever sets one.
Fixed in imdscreds rather than at the one exposed call site, so a future caller
that wraps this provider in a cache is safe by construction.
ecr-credential-provider was never affected: it calls Retrieve uncached.
Two comments claimed the hazard was ec2rolecreds capping Expires to an hour.
That cap only ever shortens a too-far-future expiry and is unrelated.
* feat: time the local and KV halves of a state write separately
RunInstances responds before vmMgr.Insert and WriteState, so the cost of
gating the respond on the local write is measurable today: it is the
ec2.RecordRunInstances span. Splitting persistState's timing by store says
which half that cost is, and the summed instance count says whether the
local write scales with the node's set size.
Duration is a sum plus an op count rather than a histogram, because ES|QL
cannot aggregate a native histogram field.
* fix: count only the records a KV state write actually touched
WriteRunningSet is digest-guarded and writes just the changed records, so
attributing the node's whole VM count to it overstated the per-record cost.
Use Written+Retired, and rename the metric to records.sum with a description
that spells out the two meanings: the whole set locally, the changed ones in
KV.
* feat: record a launch before replying to the caller
RunInstances replied with the reservation and only then inserted the records
and wrote state, so a caller holding an instance ID could describe it and get
nothing back, and a guest could read IMDS before this node's state file knew
the instance existed. Move the reply after the insert, the state write and the
per-instance subscribe loop, so by the time an ID reaches the customer the node
can answer for it and a terminate issued immediately finds a responder.
A local write that fails now fails the request: the file is what this node
restores from, so a launch it did not record is a launch nothing can recover.
The handler undoes the insert with DeleteIf and returns the capacity, rather
than running instances the caller was told had failed.
Measured on env19 first: the newly-serialised work is 7-23ms against a 2.3-10.8s
end-to-end launch, under 1%, so no split of the state write was needed.
* fix: fail DRAIN when a volume seal failed
shutdownAndUnmount logged the volume unmount error and returned nothing, so
StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set.
The coordinator then advanced to STORAGE and stopped viperblock while the seal
was still in flight, tearing the VBState envelope and leaving the guest unable
to mount its root volume on the next boot. The deploy reported success.
shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not
seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup
and stopOne carry it without aborting the rest of the teardown, since QEMU is
already down by then, and StopAll aggregates the per-VM failures so
handleShutdownDrain ACKs with Error set and never reports vms_stopped.
volumeMounterAdapter.Unmount was the second swallow site: it logged each failed
seal and returned nil unconditionally. It now attempts every volume and returns
the aggregate, which is what the VolumeMounter contract already described.
Terminate still tolerates a failed seal — the volumes are deleted next, so an
unsealed block map loses nothing a caller can ask for.
This does not fix the seal itself. It converts silent data loss into a refused
upgrade that an operator can act on.
* test: add the storagefault e2e suite for backend-outage guest survival
Freezes predastore with SIGSTOP to hold connections open while answering
nothing, which is the fault a timeout-free path cannot distinguish from a
slow peer. On one node that is a total outage; on three it freezes every
node but the guest's own, breaking the RS(2,1) shard floor.
The suite writes and flushes a verifiable region before the fault and
churns a separate unverified region during it, so an interrupted write is
never mistaken for lost data. The root-filesystem load runs from the
user's home rather than /tmp, which is tmpfs on this image and would move
no bytes on the root device.
* test: verify the root device is under write pressure, and size the outage
The previous guard matched /^ *vda / against /proc/diskstats, where the
device name is the third field, and read $7 which is ms_reading rather
than sectors_written. It printed empty and asserted nothing, so a green
run could not distinguish a surviving guest from an idle disk. It now
samples sectors_written twice and fails unless it advances.
The outage is sized from viperblock's absorption rather than guessed:
a workload must fill DefaultMaxPendingBytes before backpressure engages
at all, so a short freeze measures the retry budget rather than the fault.
* test: add nbdkit fault injection to the storage-fault e2e suite
The predastore freeze takes the whole storage cluster away from every
guest on it, which is too broad to run beside anything else. Removing a
single volume's NBD server is the same fault at one-volume scope: one
host, one guest, nothing cluster-wide stopped.
Both signals are here because they sit on opposite sides of the defect.
SIGSTOP holds the socket open and answers nothing, so the guest blocks
and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at
the default reconnect-delay of 0 fails requests at once, which under
werror=report aborts the journal.
Signalling is guarded: a pid is matched on comm, our plugin and the
volume argument together, and re-confirmed immediately before the
signal, because these hosts run nbdkit for other volumes and pids are
reused.
* fix: pause a guest on a backend I/O error instead of handing it EIO
Every virtio-blk device was created with werror=report,rerror=report, so a
backend outage was delivered to the guest as EIO. ext4 aborts its journal on
that and remounts read-only, and the write that failed is simply gone. Under
werror=stop QEMU pauses the VM and holds the failed request, so a backend that
comes back resumes the guest with nothing lost.
The boot drive, the cold-boot data volume and the hot-plugged data volume all
move to stop. Alongside it the NBD transport gets reconnect-delay=30, which
pauses requests rather than failing them while a storage process restarts on
the same socket; on the legacy -drive form it has to be file.reconnect-delay,
because the bare name reaches the raw format driver, which has no such option.
The QMP heartbeat now recognises run state io-error and issues cont. After ten
attempts the instance is reported impaired and the cadence drops to one attempt
per ten heartbeats, but it never stops: QEMU holds the request for as long as
the guest exists, so a backend that returns an hour later can still be resumed.
The storagefault suite gains the fault this was written for — kill the volume's
own nbdkit, hold, then put it back on the same socket — and it now verifies a
sha256 region across the outage rather than trusting the guest to be reachable.
Three harness defects went with it: the socket-liveness poll ran unprivileged
against a 0770 directory and could never succeed, the relaunched socket was
left 0755 where QEMU needs group write, and the two variants shared one guest
so a fault in the first was diagnosed in the second.
* test: cover the io-error resume policy
* test: stop two parallel tests racing on the terminate backoff variable
* fix: report a guest paused on a backend I/O error as impaired
A guest held by werror=stop kept reporting ok on both status checks. QMP stays
responsive while the guest is paused -- cont is being sent and answered -- so
the QMP failure gate that DescribeInstanceStatus keys on never trips, and the
one state a customer most needs to see looked healthy.
Instance status now reports impaired/failed while the resume budget is
non-zero, timestamped from IOErrorSince so impairedSince is when the guest
paused rather than when it was escalated. System status reports impaired too:
a backend I/O error is an infrastructure fault, not a guest one, and that
distinction is what the two checks exist to draw. The counter is cleared by
the first poll that sees the guest running, so both clear on recovery.
* test: reap the relaunched nbdkit when the fault test ends
relaunchNbdkit detaches the server with setsid so it outlives the SSH session
that started it, and viperblockd never learned it exists. Nothing reaped it, so
every run left an nbdkit holding a socket and a cache for a volume that was
later detached or deleted -- four were found running on dev-prod against two
detached and two deleted volumes.
The PID is resolved from the socket path rather than the volume id, so a
legitimate viperblockd server for the same volume is never the one killed.
* test: prove the backend is serving before a fault is called cleared
Thawing the process a fault stopped is not the same as restoring the service.
The outage test treated SIGCONT as the end of the fault and went straight to
judging the guest, so it passed while the cluster it ran on stayed unable to
serve a single object for the next 35 minutes, and the next test would have
inherited that.
thawPredastore now puts and gets an object through every gate, with a budget,
and fails the test if the backend does not come back.
* fix: do not hand back a mount entry whose export is gone
A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit
was gone. Every later ebs.mount then matched that entry and returned its socket
path, so vmMgr waited for a socket nothing would ever create and the instance
could not be started again — on dev-prod a guest stopped against a frozen
backend was left permanently stopped, four start attempts failing identically
on a socket the node had already removed.
Drop the entry when there is positive evidence the export is dead: the socket
is absent, or the recorded pid is not running. Neither can be true of a live
export, so the double-writer hazard the guard exists to prevent is unaffected.
Also make the durability test's pattern helpers report failures. The read dd sat
in a pipeline, so its exit status was discarded and the harness hashed nothing
and returned the empty-string sha256 as a checksum.
* test: report the guest console when a restart never answers SSH
A restart that reaches running but never comes back is indistinguishable from
a dead root filesystem without the serial console, and the log is gone once
the fixture terminates the instance.
* fix: converge guest port datapaths concurrently, not one deadline after another
Each guest port that misses its SB binding burns the full convergence deadline,
and applyEIPs spent them in series. Three ports belonging to long-stopped
instances therefore held a newly launched guest's EIP dark for the sum of them
before its own probe started.
AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and
sb-cluster-state-reset is now serialised with a minimum interval so concurrent
probes cannot each fire a cluster-wide resync.
* fix: refuse a stale cross-node start after the owning node is gone
The volume lease already refused a cross-node mount, but it is a liveness
claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded
writes stops renewing, and 45 seconds later nothing refuses a start elsewhere
that loads a stale backend checkpoint.
A failed seal now writes a durable marker naming the node holding the only
current copy, cleared on a seal that reaches the backend and on delete. The
lease covers a live holder, the marker a dead one.
The refusal also reaches the caller instead of being flattened to
ServerInternal, which told an operator to retry something that cannot succeed
while the holder is down.
* fix: keep the daemon's message when a start is refused
Rebuilding the error from the code alone rendered the generic ErrorLookup
text, so a refusal naming the node holding the volume reached the caller as
boilerplate about attaching volumes.
* fix(ec2): terminate detaches every DeleteOnTermination=false volume
TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume
on the assumption that Stop's Unmount had already cleared its attachment.
Unmount only clears it when the seal succeeded, so a volume whose seal failed
stayed in-use pointing at an instance that terminate then deleted, leaving it
unreachable with no owner to release it.
The running-instance path already detaches every DoT=false volume; the stopped
path now matches it, and tolerates an already-gone volume the same way.
DeleteVolume also keeps its provider error on the way out. The terminate reaper
retries the delete every two minutes and logged a bare ServerInternal, which
made a blob node being down indistinguishable from a transient fault.
* fix(ec2): DetachVolume --force releases a volume whose instance is gone
The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a
volume left attached by a part-way teardown could never be detached: the
instance no longer runs anywhere and nothing answers. That also made it
undeletable, since DeleteVolume refuses an attached volume.
ForceDetachVolume already existed for this deadlock but was reachable only from
account teardown. Force now falls back to it when the instance has no responder,
which is the one case where clearing the attachment cannot strand a live guest.
* fix(viperblock): unsealed writes steer placement instead of pinning a volume
The dirty marker refused any mount from a node other than the one holding
unsealed writes. Instance start already forwards to the node that last ran the
instance and falls back after a 30s window, so that refusal turned a working
fallback into a volume that could run nowhere once its owner was gone. It
traded a rare data-loss risk for a permanent availability failure.
The marker is now a record and a warning, not a gate. A takeover logs the
previous holder and why it could not seal, then adopts the marker so the old
node is no longer treated as the better source. Live exclusion is unchanged and
still enforced by the lease.
It is also written on open rather than on seal failure. A node killed mid-write
never reaches its seal, so the old trigger left no trace in exactly the case the
marker exists to describe.
Adds 'spx admin volumes unsealed' to see which volumes would cost something if
started elsewhere.
* fix(viperblock): fence a volume whose lease moved instead of logging it
The lease was checked once, at mount, and never again. A node that lost it
kept its export up and kept writing into a volume another node owned, so
both copies advanced and the backend settled it by arrival order.
Losing a renewal now decides: another holder means tear the export down,
no holder means the entry merely aged out, so reclaim it and leave the
guest running. Fencing never seals — this node's copy is the stale one —
and leaves the dirty marker for the winner to take over.
This is not an epoch fence. It narrows the window from the 45s TTL to one
renewal interval at no write-path cost; the backend still cannot refuse a
stale writer.
* fix(network): advertise EIPs so a recycled address is reachable
The gateway switch port only asked ovn-northd to publish the router's NAT
addresses in centralised and routed modes, so on a distributed cluster nothing
ever announced an elastic IP. OVN still answers ARP for it, which is why a
first-time allocation works and hid this — but no gratuitous ARP is sent when
the address moves to a new ENI, so every host on the segment keeps resolving it
to the previous holder's MAC until its cache ages out. Reusing an address from
the pool then lands on a dead MAC and the instance is unreachable from outside.
Distributed mode needs the option as much as the others: northd scopes each
entry with is_chassis_resident(<vif port>), so the announcement leaves from the
chassis running the instance rather than the gateway, which is what distributed
NAT wants.
Converge on the already-attached path too. AttachIGW returns early once the
gateway port exists and nothing else rewrites it, so without that an existing
VPC would stay silent for its whole life.
Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to
carrying their correct external_mac, each scoped to its own ENI's port.
* fix(viperblock): make the volume fence and its dirty marker ownership-safe
Four corrections to the lease-loss fence, all found in review.
Drop the reacquire path. Losing a lease and finding no successor was treated as
a lapsed entry worth reclaiming, but leases.acquire returns the cached entry —
already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease
renewed nothing and the node went on writing a volume it could not prove it
owned. The two errors are not symmetric: a false fence costs one guest a
restart, a false reclaim leaves two nodes writing one volume. Fence in both
cases, and report which happened through the outcome attribute.
Surrender a lease this node cannot confirm. Renewal only noticed a loss when
the server rejected an update, so a node partitioned from NATS kept its export
up while the TTL expired and another node took the volume. Track the last
confirmed renewal and give the volume up once that ages past the point where
the server may admit another owner. Bound the renewal call too, so a hung
JetStream request cannot hold the goroutine past that point.
Do not act on a kill that did not take. Both the fence and ebs.unmount removed
the mount entry, released the lease and announced the result before knowing
nbdkit had exited — the fence then reported a writer stopped that was still
running, and the unmount sealed into a directory a live process was writing.
Wait for the exit, and on failure leave the volume mounted and say so. An
already-exited process now counts as success, which is what the caller asked.
Give the dirty marker a generation. It recorded an owner and nothing to order
two nodes' claims, so a node returning after a takeover could clear or overwrite
the winner's marker and point placement back at the stale copy. Stamp the lease
generation, refuse a write from an older one, and only clear a marker this node
still owns. A marker that cannot be written now fails the open rather than
leaving writes nothing records.
* test(viperblock): cover the stall surrender and the unmount kill guard
The two ownership cases the fence work left untested.
A holder partitioned from NATS while the winner keeps it never gets an update
rejected, so revision checking cannot see it — it has to give the volume up on
its own clock. Pins that validity sits below the server TTL with room for a
check in between, that a lapsed holder reports the loss as a stall rather than
as a takeover, and that releasing a surrendered lease leaves the entry for
whoever holds it now.
The unmount side of the kill guard: sealing under a live nbdkit writes a state
torn between two writers, so a kill that did not take has to fail the unmount
and leave the entry mounted for a retry.
* fix(e2e): keep the guest private key for as long as the key pair
EnsureKeyPair computed the PEM path from the caller's directory but wrote
the file inside the ensureOnce closure, which runs only on the first call.
Callers passed a per-test artifact dir, and ArtifactDir removes that dir
when the test passes — so the first test's success deleted the only copy
of the private key while the key pair stayed registered, and every later
call returned a path nothing had ever been written to.
Guests after the first were then launched against a public key whose
private half no longer existed, so the harness could never authenticate
and gave up after the full five-minute budget. The guests were healthy
and reachable on tcp/22 throughout, which is why this read as a storage
or network fault for so long.
The directory parameter was the defect, so it is gone: the PEM now lives
in a directory the fixture owns and removes with the key pair, and no
caller can supply one that is shorter-lived.
Also reorder the killed-nbdkit teardown. The reaper registered its own
cleanup after the detach registered one, so LIFO killed the block server
before the platform detached the disk and DetachVolume then failed with
"Node ... is in use". The detach cleanup now reaps once it is done.
* fix: stop leaking an nbdkit export when recovery cannot adopt it
Restart recovery adopts a guest's boot volume, then routinely fails the
-efi volume with "claim lease: context deadline exceeded" -- JetStream is
still catching up at daemon start and the KV claim runs on the client's
5s default. Recovery gave up on the first error and never retried, so the
export stayed live with no registry entry. At terminate the unmount found
nothing to seal, returned NotFound and killed nothing, and the process
outlived its guest holding the volume and socket open. EFI is always the
second volume adopted, which is why it was always the one that leaked.
Two changes, because either alone leaves a hole:
- rebuildMountedVolume retries the build (4 attempts, 3s apart) so a
JetStream that is merely slow no longer orphans an export for the life
of the process.
- unmountVolume reaps a corroborated export for the volume it was asked
to unmount when the registry has no entry for it, so an export orphaned
by any means is still cleaned up. The reap goes through
reapOrphanedNbdkit, which declines anything a live process references:
at unmount the guest's blockdev may not be torn down yet, and killing
the server underneath it strands QEMU.
* test: hold the backend-serving probe and assert predastore readiness
The backend-outage test declared the backend restored on one successful
round trip through each gate. A gate can answer the request that arrives
just after a thaw and then lose its blob peers under the load that
follows, which is a backend that never recovered reported as one that did
-- and that is what happened: the probe reported all three gates serving
minutes before the guest was still taking BLOCK_IO_ERROR on every resume.
The round trip has to hold across three rounds now, and the count restarts
rather than decays when one fails, so a gate that serves and then stops is
the failure the hold is looking for rather than noise it averages out.
It also checks /readyz on each host and reports any failed check by name.
The round trip only proves this workstation can write through a gate; it
says nothing about which blob peers that gate can reach, and a gate one
node short of the write floor serves a small object and fails a guest. A
host that serves no admin listener is logged and skipped, since the port
is optional configuration.
* fix: scope predastore's admin port to the cluster plane in the host policy
predastore's admin listener binds the host's cluster bind_addr and serves
/healthz and /readyz unauthenticated. Readiness now names which blob peers
a gate cannot reach, which is exactly the kind of topology detail 4432 is
peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the
public one.
* fix: keep resuming a paused guest at heartbeat rate after a long outage
Past ten attempts the cont dropped to one per ten heartbeats, so a guest
whose backend had come back could sit paused for another five minutes
before anything retried it. Measured on dev-prod: the backend was proved
serving through all three gates while the guest was still waiting on
attempt 60, five minutes after attempt 50.
The slowdown was there to avoid paying a failed I/O every thirty seconds
on a backend still down. That is nearly free, and five minutes of downtime
after recovery is not, so only the log line is throttled now.
* test: fail the run when a key pair name resolves to more than one pair
A create that lands twice leaves two key pairs with one name and different
material. RunInstances resolves the name to one of them while the harness
holds the private key of the other, and the only symptom is every guest in
the run refusing the key at the SSH gate — five minutes of waiting per
guest, four guests, and nothing anywhere naming the cause. Observed on
dev-prod: two pairs created a millisecond apart, both named
e2e-key-30de10ba.
One DescribeKeyPairs after the create says so outright instead.
* test: tolerate an already-released volume in the detach teardown
Teardown runs DetachVolumeWait after a terminate that releases the volume
itself, so the detach can arrive at a volume already on its way to
available. That is the outcome it wanted, and treating the IncorrectState
refusal as fatal failed a backend-outage run whose assertions had all
passed — the guest came back, every acknowledged write verified, and the
test still went red on the way out.
VolumeInUse is still fatal: that one means the volume is attached
somewhere it should not be.
* test: carry the merged suites onto the reworked key-pair and account-scoped seams
The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir
this branch removed, and DeleteVolume's metadata lookup is account-scoped now,
so the provider-failure test has to seed and delete under the same account.
* refactor: make the fence outcome a type in one const block
Four separate exported string constants with a paragraph of comment each,
passed to RecordVolumeFence as a bare string. Group them, give them a
FenceOutcome type, and take that type at both call sites so an arbitrary
value can no longer reach the metric attribute.
* refactor: move viperblockd's wire contract into a leaf vbwire package
The daemon imported viperblockd for exactly two symbols -- the fenced
volume subject and its event -- which put the storage service back on the
daemon's dependency graph to name something it only listens for.
vbwire holds the contract instead: the subject and event, the dirty
bucket name and record shape, and the read-only listing an operator tool
needs. It imports nothing from spinifex, so both the daemon and the admin
CLI drop their edge to viperblockd entirely. cmd/service.go keeps its
one, which is genuine: it constructs the service.
* docs: drop a specific host name from the start-refusal comment
Named a real dev-prod host in a comment and in test fixtures. Say what
the name is instead.
* ci: give storagefault a dedicated nightly cell
The suite is eligible for both topologies and runs on the PR gate, but no
nightly cell ran it: the permutation cells run a deliberately narrow
subset and none of the dedicated suite cells named it.
Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation
cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that
environment would report the outage as its own failure.
* ci: run storagefault nightly on three nodes, not one
The previous commit put its dedicated cell on a single-node runner. That was
the wrong topology and the wrong justification: single-node is RS(1,0), where
the shard floor cannot be broken and a cross-node takeover after a failed seal
does not exist, so the two assertions worth having a dedicated cell for were
the two it could not make. Cell 29 is now real-multi on ci-multi.
It also corrects the claim that the PR gate already covers the multi-node case.
The suite is eligible for both topologies, but e2e.yml's default suite list
never names it, so it has never run there. A full pass is ~63 minutes on
dev-prod bare metal, almost all of it deliberate waiting -- one test holds
predastore SIGSTOPped for seven minutes -- which is too long to add to a gate
that currently finishes well inside its budget. Nightly is where it belongs.
Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since
90 minutes of suite cannot run in a 50-minute job; the permutation steps gated
on matrix.suite == '' so the dedicated cells do not also run them; and a Run
Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts
the suite against E2E_SUITES_MULTI before running it.
* fix: stop a startup KV race leaving vpcd serving keyless guests
vpcd opens five KV buckets while NATS is still coming up, each attempt
carrying a ~5s JetStream API deadline, and applied three different policies to
the same failure: spinifex-vpc-enis was fatal, while instance-state,
security-groups, subnets and vpcs each degraded silently for the lifetime of
the process. A five-second blip during a cold start therefore decided how a
node served metadata until someone restarted it.
On dev-prod all three nodes hit this after a deploy. Two were saved by
accident: their ENI bucket also timed out, so they exited 1 and systemd
restarted them cleanly. The third's ENI bucket opened in time, so it never
crashed and ran for hours with no instance record loader. With the local state
file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's
crawl died on ami-id and fell back to DataSourceNone, and every guest launched
on that node booted with no authorized_keys.
kvstore.OpenWithRetry retries for a bounded window before any policy applies,
logs the cause per attempt and the wait on a late success, wraps the last error
with the bucket name, and returns at once on a cancelled context. All five
opens use it. Failing to open the instance record space is now fatal, matching
the ENI bucket: the dependency whose loss stops guests logging in should not be
treated more leniently than the one whose loss stops ENI resolution.
Also fixes the reason the fallback was load bearing at all. The local state
file was written 0600 and vpcd reads it as a different user in the shared
spinifex group, so the local-first path had never once worked and its
mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so
an inherited umask cannot undo it.
Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an
unresolvable instance as "no key pair" with nothing logged.
* fix: scope the ime-* firewall accept to IMDS and VPC DNS only
The per-tap OVS demux delivers any guest packet addressed to
169.254.169.254 or .253 onto an ime-* host port regardless of L4 port,
and iifname "ime-*" accept then admitted all of it. Every wildcard-bound
host service — sshd, the AWS gateway, northstar, the console — answered
guest packets on those addresses; only asymmetric reply routing kept the
handshake from completing, and a guest could still drive northstar to
recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS
(.253:53), letting everything else fall through to policy drop.
* docs: document the instance-to-host plane and how to protect it without the ISO
The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by
host services and is invisible to a NIC-level or cloud firewall, since the
traffic is intra-host on the OVS bridge. Explain that the ISO ships the
scoping rule armed, that the binary installer ships the firewall off, and
that a non-ISO install must enable it with setup.sh --firewall=on or
firewall_enabled=true. Include verification from an instance and the host.
* fix: withhold the IMDS meta-data listing when the instance does not resolve
Every omission from the meta-data root is a positive claim about the instance —
public-keys/ absent means "launched without a key pair", and cloud-init believes
it, writes no authorized_keys and boots the guest keyless. So resolving the
instance was best-effort for a listing whose whole meaning depends on it: a
backend error or a miss shortened the listing instead of failing it, and nothing
anywhere recorded that the answer was a guess.
Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot
retry a 200 that is merely wrong, so this converts a permanent silent failure
into a transient loud one. servePublicKeys already had this policy for the leaf;
the root is what contradicted it.
A profile lookup that fails still serves the listing minus iam/, and now logs.
The key pair is the field a guest cannot recover from losing, while a missing
role surfaces through the credentials leaf, which reports the miss itself.
Also stops the storagefault cross-node test failing on its own access path. It
tolerates the pattern changing across a takeover, but readPattern needs sudo,
and a volume opened from the backend checkpoint may predate cloud-init writing
its sudoers drop-in — indistinguishable from an unreadable device at the dd call
site. It now checks sudo first, and reads cloud-init's recorded datasource to
tell a rolled-back root filesystem from IMDS never having answered, failing
loudly on the latter rather than accepting it as storage loss.
---------
Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>
* feat: add read-only instance record cache (Phase 2)
Adds spinifex/instancecache: a revision-fenced snapshot+watch informer
over the instance record KV space, with a per-account index and
OTel metrics. Wired up and running in awsgw at startup, but not yet
consulted by any read path.
* fix: cover the fence with real syncs, bound decode-failure cardinality, expose degraded
Adds end-to-end fencing tests that drive freshSync/periodicResync
against the embedded NATS server via a test-only postSnapshotHook
seam, since drainBuffered unit tests alone don't exercise the
buffer-before-snapshot ordering. Drops the per-key attribute from
the decode_failures counter to keep it O(1) time series; the key
is still logged where the decode happens. Adds a Degraded accessor
set on sync failure and cleared on the next success, per D7,
without changing what List or Get answer.
* refactor: replace instancecache's buffering fence with ResumeFromRevision
The watch/snapshot fence no longer needs a side buffer, watcher modes, or
an overflow failure mode. Every sync now snapshots first, then opens a
watcher with kvstore.Bucket.WatchFrom (ResumeFromRevision), and replays
onto a private candidate map until the end-of-replay marker arrives.
Only a fully caught-up candidate is swapped in as the live map, together
with its watcher becoming active; the previous watcher is stopped and
joined only after, so two watchers never apply to the same map.
kvstore.Bucket gains WatchFrom as a sibling to Watch, opened through the
same withKV wrapper, for a caller that needs to replay from a snapshot's
high-water mark rather than an UpdatesOnly live feed.
* fix: cover the fence with production entry points, not a reimplementation
The four fence tests recomputed hw+1 and called kvstore.WatchFrom/replay
themselves instead of going through syncOnce, so they verified nats.go's
resume behavior (already covered by TestResumeFromRevision_...) rather
than the cache's own revision handling. A mutation inside syncOnce
(hw+1 -> hw+1_000_000) passed all four.
Reinstate a postSnapshotHook seam (nil in production) so the initial-sync
pair can call syncOnce/installSync directly and the periodic-resync pair
can call periodicResync directly, landing the write inside syncOnce's
internal snapshot-to-watch window without duplicating it. The same
mutation now fails all four; reverting passes all four.
* feat: count the records a resumed watch replays, so the fence is observable live
* fix: render northstar's recursion allowlist so a node is not an open resolver
northstar.toml is written with allow_recursion = false and an allow_recursion_from
list built from cluster membership, so a freshly formed node serves recursion to
the cluster and refuses it to the internet.
The list is the node's own bind and advertise addresses plus every peer's host,
with any port stripped. Both local addresses matter: a node's resolv.conf points
at its own advertised address, so the query arrives from that public IP over lo
rather than from 127.0.0.1, and an allowlist of loopback alone would stop every
node resolving internal names.
Guests are covered without appearing in the list. vpcd's per-tap DNS shim relays
guest queries to a node's :5300 from a node address, so trusting the nodes trusts
every guest behind them and no guest CIDR is needed.
The template comment carries the caveats for allow_recursion = true, since the
cost of setting it is remote from the setting: an open resolver is found by
scanners within hours and gets the node's whole address range blocklisted, which
is an outage for every tenant on it.
The list is rendered at formation, so a node added later is not in its peers'
files. Regenerating it live from membership, as the daemon already does for
peers.nft, is follow-on work tracked on mulga-t3mub.
* chore: pin northstar to the recursion ACL and atomic zone reload
The recursion allowlist this branch renders is enforced by northstar's
RecursionPolicy, which lands in dev at a0e741e. CI resolves northstar from
go.mod rather than the workspace, so the build failed on NewRecursionPolicy
being undefined until this pin moved. The AWS SDK bumps come with it.
The EgressSelectorConfiguration written to every EKS control-plane VM
defined only the `cluster` egress selection, which carries kubelet
exec/logs/portforward. Admission webhooks and aggregated apiservers dial
via the `controlplane` selection instead, and an undefined selection
falls back to a direct dial. The control-plane VM sits alone in its own
VPC with no route to the service CIDR, so every webhook call timed out
after 10s and konnectivity never saw a single dial request for one.
Define both selections on the konnectivity-server UDS so webhook traffic
rides the agent-initiated reverse tunnel alongside kubelet traffic.
natsMetricsHandler records after the handler has already replied, so a
test that collects the moment nc.Request returns can beat the point to
the reader. Matches the poll the ec2.cmd outcome tests already use.
* feat: repair instance records on a ticker, not only on state change
A record write is best-effort, and a failed one forgets its digest so the
next reconcile retries it. But the only caller is persistState, reached
from WriteState, which fires on state change. An instance that launches,
has its record write fail, and then simply runs has no next state change,
so its record is absent for as long as it lives.
A 60s sweep on the node now reconciles the live instance set against the
digest of what it has published. The digest guard means a node with
nothing outstanding writes nothing, so the cost tracks failures rather
than instance count. The interval is the point of the change — it is what
turns an unbounded window into a bounded one — so it is asserted directly
rather than inferred from a sweep.
The sweep does not go through persistState: the local state file is
already correct and rewriting it would advance the state revision for no
reason. Only the record half needs repairing.
WriteRunningSet now reports what it did, so a sweep that found work can
say so and a healthy node stays silent.
* fix: keep the QMP liveness stamp out of record change detection
Found on env19: the node owning a running instance rewrote its record on
every sweep, forever. The digest guard was working exactly as coded — it
correctly found a real difference each tick, because Health.LastQMPSuccess
advances on every successful QMP poll and the collector runs on its own
cadence. Left alone this is a KV write per running instance per node per
minute, indefinitely.
The stamp is now held out of the digest. Nothing reads it back: impairment
is judged from QMPConsecutiveFailures and ImpairedSince, and node liveness
comes from the heartbeat store, so holding it out costs no reader anything
and restores the publication cadence the record had before the sweep
existed. QMPConsecutiveFailures and ImpairedSince stay in, because an
impairment is a real transition that has to reach the record space.
Both halves are covered: a stamp moving on its own must not write, and an
impairment must still write. The first fails without this change.
Snapshot metadata is now keyed by owning account, so a snapshot outside
the caller's prefix is absent rather than forbidden. The e2e assertion
still encoded the pre-partitioning UnauthorizedOperation contract.
ModifyLoadBalancerAttributes and ModifyTargetGroupAttributes reject any
key absent from the per-type default maps with ValidationError, and the
rejection fails the whole call rather than the single key. Terraform and
the AWS Load Balancer Controller send the full attribute set on every
apply, so one missing key breaks the resource outright.
Adds the 17 keys the ELBv2 API reference documents but the default maps
lacked:
ALB: health_check_logs.s3.{enabled,bucket,prefix}, ipv6.deny_all_igw_traffic
NLB: secondary_ips.auto_assigned.per_subnet
Target group: load_balancing.algorithm.anomaly_mitigation,
stickiness.app_cookie.{cookie_name,duration_seconds},
lambda.multi_value_headers.enabled, target_group_health.* (4),
deregistration_delay.connection_termination.enabled,
preserve_client_ip.enabled, target_health_state.unhealthy.* (2),
target_failover.{on_deregistration,on_unhealthy}
health_check_logs.s3.* is what surfaced this: it is ALB-only, and an
aws_lb apply against the sandbox failed on it.
The target group map stays a single flat union rather than splitting by
protocol, so it remains a permissive superset. Values are store-and-echo
only; nothing in the data plane reads them.
Filling the filesystem behind the JetStream store to zero free bytes leaves
nats-server with nowhere to write while it recovers a stream. Every error
return inside the locked region of newFileStoreWithCreated assigns nil to the
named return value fs, and the deferred unlock then dereferences it, so a
recovery write failure surfaces as a nil-pointer panic instead of an error.
That structure is unchanged in v2.14.3, in the pinned v2.14.5 and in the
current v2.14.6, so no available version bump addresses it.
The nats service now keeps a preallocated reserve file beside the jetstream
directory and hands it back at startup when the filesystem can no longer
supply that headroom on its own, re-arming once space returns. Recovery
therefore always has somewhere to write, whichever process filled the disk.
* feat: add the e2e guest disk-performance gate
Nothing measured guest disk performance in CI, which is why a quadratic
index scan shipped in viperblock 35dd9f2 and survived three subsequent
fixes to adjacent code. It is also why the two availability clauses at
the end of the ebs-performance plan's Stage 0 have never been evaluated:
a sustained random-write workload took the guest, and then the host,
down, and nothing since has demonstrated that it no longer does.
tests/e2e/diskperf/ launches an instance, attaches a dedicated blank
volume per repetition, and drives a pinned fio profile over guest SSH.
Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit
criterion names, and randrw-70-30-4k, the mixed workload the headline
IOPS came from. The target is a raw device with direct=1 rather than a
file on the boot disk, so guest page cache, filesystem allocation and
journal traffic are out of the path.
Two classes of assertion, kept apart because folding them together lets
a run that left the host unreachable pass on a respectable IOPS figure.
Availability is hard pass/fail and absolute, so it gates from the first
run: the job completes inside its budget, a guest responsiveness probe
answers throughout on a second SSH channel, and the host's own sshd
answers throughout. The guest probe reads the root volume while the
workload runs on a different one, so a pass is evidence of isolation
between volumes rather than of the job merely being slow. The host probe
is a TCP connect plus an SSH banner read, which needs no credential and
still proves the kernel scheduled sshd.
Throughput and latency are banded against a committed baseline: fail
above 25% regression, warn above 10%, per-metric median across
repetitions. baseline.json is committed and embedded, with every figure
zeroed, and a zero reference records and warns rather than failing. Only
the availability class gates until numbers are captured on the bare-metal
cell; a throughput band nobody measured would be a fabricated gate.
A fresh volume per repetition is the reset. Extent count grows
monotonically, so reusing a volume makes the gate order-dependent and
drifts its own numbers upward over time, slowly normalising the
regression it exists to catch. Each job is its own guest invocation and
fio's JSON is fetched by a separate command, so a wedged job loses only
itself and reports as a budget breach rather than a parse failure.
e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable
workflow serves both the correctness cell and nightly cell 28, which
queues behind cell 20 on the shared concurrency group. Cells 25-27 are
the dedicated eks, ecs and rds cells. The per-suite Go test timeout is
resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares
its 180-minute budget beside rds's rather than in a second place.
The gate's own arithmetic runs in preflight via test-harness: band edges,
the unmeasured cases, per-stream metric selection, fio JSON parsing, and
the probe accounting including a slow-but-successful answer counting as a
breach. A gate whose comparison logic is untested is the failure mode the
plan is otherwise full of.
The extent-count ceiling named as a fourth availability assertion is not
implemented: it needs viperblock.index.entries from S1.5, which does not
exist.
* feat: let a manual bare-metal run choose its e2e suites
A dispatch run only sees inputs declared under workflow_dispatch, which had
none, so suites and suite-env resolved empty and fell back to the correctness
set. Capturing a diskperf baseline by hand was therefore unreachable without a
workflow_call caller.
* fix: build the bare-metal suites the run step will execute
The build step packs pre-built binaries into the tarball and did not see the
suite selection, so a selected suite was absent from the node unless it
happened to be one of the three the build script hardcoded.
* feat: commit the measured disk-performance baseline
Medians of three repetitions on the bare-metal cell. The throughput and
latency bands now gate instead of warning, so a 2x regression in either
job fails the suite.
randwrite-4k's write tail is two orders above the mixed job's. It is
recorded as measured rather than zeroed: an ungated metric would miss the
regression class the gate exists to catch.
* feat: stamp the owning node's AZ on the instance record
RunInstance never persists a node's AZ onto the stored instance, so
today only the launching node's own ProjectInstance call can produce
a correct AZ for a describe. Add VM.AZ, carry it on InstanceStatus
since it is placement observed by the running node, and stamp it in
WriteRunningSet alongside Status.LastNode from the node's own
config.AZ.
* docs: trim the AZ field comment to the project's three-line limit
* fix: identify Gather responders by node, not by frame count
utils.Gather previously counted reply frames toward completeness, so
two subscriptions replying from one host (a daemon mid-restart) could
satisfy a fan-out while a genuinely configured node stayed silent.
Every daemon reply now carries an X-Node-ID header set by the shared
response helpers via nats.Msg.RespondMsg, so Gather can attribute each
frame to the node that sent it. Gather gains a CollectionMode
(CollectServeData preserves today's early-exit behavior;
CollectUntilDeadline collects to the full deadline for absence-proof
callers) and returns node-identified Frames plus Responders/
SuccessResponders/ErrorResponders sets. Identity mode retains only the
first payload seen per node; a disagreeing later frame from the same
node flags a conflict but never overwrites the retained data.
Node identity uses one namespace across the reply header,
config.ClusterConfig.Nodes keys, daemon.Heartbeat.Node, and
vm.VM.LastNode, all traced back to Daemon.node - confirmed to already
agree, with a new test asserting it at runtime.
This is a mechanical caller-side follow-on: every existing
utils.Gather caller now reads frame.Data instead of frame directly.
* feat: thread configured node set to DescribeInstances
GatewayConfig gains NodeIDs, the keys of config.ClusterConfig.Nodes,
distinct from the existing ExpectedNodes count. awsgw builds it from
cluster config and wires it into GatewayConfig. The DescribeInstances
action passes gw.NodeIDs through instead of relying on
gw.DiscoverActiveNodes(ctx) at that call site, so completeness can be
judged against the nodes that are actually configured rather than the
nodes that happened to answer a discovery probe.
* fix: judge DescribeInstances completeness by node identity
DescribeInstancesChecked now judges completeness on decoded frames
against the configured node set (ValidResponders) rather than on
Gather's frame/byte counts, which cannot know the describe schema. A
node that answers with a nil Reservations list is a valid responder,
not an invalid one - the existing merge guard on Reservations != nil
governs how frames combine, not whether the node counted as having
answered.
The explicit-instance-ID 404 path now collects with
CollectUntilDeadline so a node that hasn't replied yet cannot be
mistaken for a node that has nothing to report; general listing stays
on CollectServeData. Any ambiguity - an unidentified frame, a node
appearing in both the success and error sets, a node replying twice
with disagreeing frames, a Gather cap hit, or an empty/unset node set
- suppresses the NotFound rather than risking a false one.
* fix: let a describe by instance id stop once every id is found
CollectUntilDeadline made every DescribeInstances with explicit ids wait the
full 3s fan-out deadline, measured at 4650ms against 1655ms before, on a
healthy cluster where all three nodes answer in about a millisecond.
Gather gains a Settled predicate, consulted only under CollectUntilDeadline,
that ends collection once the replies already answer the question. Absence
still pays the whole deadline, and an instance the caller cannot see is
filtered out of every frame so it takes the absence path too, which keeps the
two NotFound answers indistinguishable in time.
* refactor: resolve IMDS instance metadata local-first, drop the fan-out
IMDS only ever serves a guest on its own node, so a DescribeInstances
fan-out to the whole cluster is unnecessary latency and load. Resolve
from the node's own on-disk VM state first, falling back to a direct
record-space read for an instance that has moved off this node.
Neither path issues a NATS request.
* perf: cache the local instance state parse for IMDS lookups
IMDS request rate is guest-driven and unbounded, so re-reading and
re-parsing the whole on-disk VM state on every request does not
scale with VM count on a node. Cache the parsed state and only
re-read when the file's mtime or size changes; a stat or parse
failure clears the cache and falls back to the record space rather
than serving a stale hit.
* fix: share ManagedBy visibility rule between DescribeInstances and IMDS
Extracts IsInstanceVisibleToCaller so account ownership and the
platform-managed hide rule live in one place instead of two
independently maintained copies.
* refactor: decouple IMDS instance lookup from the daemon package
handlers/imds no longer imports daemon: NewIMDSServiceImpl now takes a
localStateReader and a recordLoader interface instead of a dataDir
string, and vpcd (the composition root for both planes) constructs
both adapters, including the stat-guarded local-state cache that used
to live inline in handlers/imds. Either adapter may be nil, degrading
that source to a miss rather than failing IMDS to start.
* test: move the local state reader test to an external package
* chore: exempt export_test.go from the external test package check
The check's own remedy text tells you to move unexported hooks into an
export_test.go, then flags that file for being in-package — which it is by
definition, since exporting internals is its whole purpose.
* fix: bound how many describes may hold a full fan-out deadline
A describe naming an instance the caller cannot see is filtered out of every
frame, so it never settles early and holds a gateway goroutine and inbox
subscription for the whole 3s deadline. Issuing those is free for the caller,
so nothing stopped a burst of them pinning unbounded gateway state.
Absence proofs now draw on a fixed budget. Past it, callers get a retryable
RequestLimitExceeded rather than being queued, since queueing would let the
backlog grow. The decision is taken before any reply is read, so a nonexistent
id and another account's id throttle identically and the throttle cannot become
the timing oracle the deadline exists to close.
* perf: settle describe-by-id on responder coverage, not the full deadline
Every describe-by-id was paying the whole 3s fan-out deadline whenever the
requested instance was not in a running-instance frame — which includes a
stopped instance, a terminated one, an id belonging to another account, and
an id that does not exist.
Gather gains ResponderGrace: once every ExpectedResponders node has
answered, collection closes after a short grace window instead of running to
the deadline. Core NATS gives a request only to the subscribers present when
it was published, so an overlapping subscription replies alongside the first
and one that arrives later never receives the request at all. Waiting past
full coverage cannot find a reply that full coverage missed. The window is
armed once and never extended, so a node replying repeatedly cannot hold the
fan-out open. The full deadline is now paid only when a node stays silent,
which is the case it was always meant to cover.
The stopped and terminated bucket queries now run alongside the fan-out
rather than after it, so their latency overlaps it instead of adding to it.
They feed a shared instanceSettler that ends collection as soon as every
requested id has been seen, skipping even the grace window.
The timing oracle is unchanged. An instance the caller cannot see is
filtered out of every source before the reply is sent, so it takes the same
path as an id that does not exist: both answer NotFound after full responder
coverage, and remain indistinguishable in time.
* fix: stop parallel tests racing the terminate backoff seam
terminateRetrySleep is a package global, and every test that needed it
neutered swapped it and restored the old value from t.Cleanup. Those tests
run in parallel, so two cleanups could write the seam at once — a real data
race the detector caught once enough parallel tests shared the binary.
Installed once in TestMain instead. Nothing in this package wants the real
backoff, so there is no value in restoring it per test.
The timeout was written out separately in each workflow and drifted: rds
ran 50m on the PR gate and 30m nightly, where it panicked and took six
in-flight subtests with it. e2e-suites.yml had the same hardcoded 30m on
its single path, which also runs rds.
A new e2e_suite_timeout helper sits beside the eligibility lists it
already guards, and all four call sites resolve through it. e2e.yml loops
over suites on the remote VM, which has no checkout, so it resolves the
pairs on the runner and looks them up in the loop.
Also lift the nightly cell ceiling to 50 minutes so the rds cell is not
cut short by the job before its own timeout applies.
checkSGDependencies knew which ENI or referencing security group blocked the
delete and discarded it, returning a bare DependencyViolation with no log line,
so the blocking id had to be recovered by cross-referencing the ENI table.
Return it via awserrors.Errorf and log it at the point of refusal.
shutdownAndUnmount logged the volume unmount error and returned nothing, so
StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set.
The coordinator then advanced to STORAGE and stopped viperblock while the seal
was still in flight, tearing the VBState envelope and leaving the guest unable
to mount its root volume on the next boot. The deploy reported success.
shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not
seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup
and stopOne carry it without aborting the rest of the teardown, since QEMU is
already down by then, and StopAll aggregates the per-VM failures so
handleShutdownDrain ACKs with Error set and never reports vms_stopped.
volumeMounterAdapter.Unmount was the second swallow site: it logged each failed
seal and returned nil unconditionally. It now attempts every volume and returns
the aggregate, which is what the VolumeMounter contract already described.
Terminate still tolerates a failed seal — the volumes are deleted next, so an
unsealed block map loses nothing a caller can ask for.
This does not fix the seal itself. It converts silent data loss into a refused
upgrade that an operator can act on.
The policy gate had the cluster and nodegroup name but not the trailing
UUID, so it evaluated nodegroup/<cluster>/<name>/*. That value matches the
AWS-documented wildcard policy spelling but not one naming the exact stored
ARN, which is what a tenant gets by pasting DescribeNodegroup output, so
that Deny stayed inert.
The discriminator is now derived from (account, cluster, name) as an RFC
9562 version-8 UUID, so the gateway and the handler compute the same value
independently and the gate names the stored ARN exactly with no lookup.
Nodegroups created before this keep their random discriminator and their
existing wildcard-only behaviour.
Drives every vpc.* topology topic end to end against the mock OVN client and
asserts the resulting rows: router, subnet switch and router port, logical
switch port addresses and port-group membership, gateway router port, NAT
rules and egress policies. Invalid CIDR, IP and MAC events are pinned to a
failed reply envelope with no partial write.
Moves the elbv2 dispatch table to an external test package.
subscribers 53.5% -> 73.4%.
Adds a 35-row dispatch table covering every NATS-backed elbv2 gateway
wrapper. Each row subscribes a stub only to the expected subject, so a
mis-wired subject fails as no-responders, and asserts the returned value,
the forwarded request body and the account header. A second table covers
the 28 validation guards that had no test, each pinned to its AWS error
code.
Package coverage 49.5% -> 93.8%.
Tables the twelve subjects Serve subscribes through the malformed-body,
version-skew and provider-failure arms, and round-trips every verb through
NATSProvider against a served MemoryProvider so client encoding and server
decoding are asserted against each other.
Covers MemoryProvider.ListVolumes, ListSnapshots and CopySnapshot, all of
which were at zero, plus NATSProvider's nil-payload and subject-token guards.
natsserve 28.8% -> 91.3%, ebsprovider 67.5% -> 86.3%, nullprovider 0% -> 100%.
Drives the model through Update with real key messages: a full walkthrough
asserting the config it produces, every validation gate keeping the operator on
the screen with the cursor on the field at fault, and the rendered output for
facts that appear nowhere else (role layout, pool geometry, folded planes).
The cross-host ICMP probe fired once, with no convergence budget, the moment
sshd first answered on the source guest. The default SG self-ingress rule
matches on the northd-derived $<pg>_ip4 address set, which only reaches the
remote chassis after northd regenerates it from port-group membership and
ovn-controller installs the flows there, so a first-packet burst can be
swallowed by the priority-900 default drop with no log line anywhere.
Give the probe the same 45s/2s budget the single-node suite already uses, and
snapshot NB/SB address sets, port groups, ACLs, port bindings and br-int flows
from every node plus the source guest on the first failed burst rather than at
teardown, so the state is captured while the drop is still observable. Emit
attempt count and convergence time so a pass that needed most of the budget
still surfaces as a propagation delay instead of a silent green.
One behavioural suite runs against LiveClient over the in-memory NB server and
against mock.Client, so the semantics reconcile, policy, external, topology,
subscribers and vpcd rely on are pinned in both. Covers NAT router scoping and
duplicate deletion, the not-found sentinels, gateway chassis idempotence,
ReplaceACLs no-churn, port group membership and DHCP lookups.
The readiness loop terminates on wall clock, so a stubbed sleep alone still
costs two minutes; clusterReadyMaxWait becomes a var alongside clusterReadySleep
so the timeout branch is reachable. Adds the in-package marker to the EIP
association tests, which drive setupTestEIP and read svc.natsConn.
setupTestEIP built the EIP service with a nil VPCService, so every ENI
lookup AssociateAddress performs was structurally unreachable: lookupENI,
lookupENIByInstance and AssociatedPublicIPForInstance had no test at all
and any test reaching them would have panicked. Wire a real VPC service
over the same embedded NATS and cover association by ENI id and by
instance id, the private-IP override, both not-found codes, and the
add-nat/delete-nat events the NAT rule is built from.
sgRulesToPolicyRules was never carried a tenant security-group rule: the
golden scenarios build IntentState by hand and bypass the adapter, so a
transposed FromPort/ToPort or a dropped SourceSG would have shipped
silently. Load real SG records through KV and pin the field mapping.
ECS and ECR take the existing both-directions shape against their Actions
tables. RDS carries its scopes on the dispatch table itself, so the gate is the
account-level half instead: every served action is either scoped or carries a
written reason, and one in both sets or neither fails. HasScope and
ScopedActions are added to drive it.
The condition-key gate pins the door's key set and asserts the IAM write path
accepts exactly the operator/key pairs the evaluator enforces.
The seven sourcePolicyARN actions resolved to the PolicyArn the caller sent,
so the gate decided against an ARN whose account the caller chose. The scope
now takes the ARN's path and name and rebuilds it on the caller's account, as
sourceOIDCARN beside it already does, and widens to account-wide when the ARN
carries no policy component. Lands with the cross-tenant integration suite that
found it.
snapshotVolumeEngine's unmounted branch built its engine by hand instead of
going through openVolumeVB, so it was the last control-plane path that opened
an engine over a volume without holding the cluster-wide lease. A snapshot
request landing on a node that does not have the volume mounted would open a
second engine over storage another node was writing.
Two engines on one encrypted volume read the same SeqNumHighWater and issue the
same AES-GCM nonces, which discloses the XOR of the plaintexts and permits
recovery of the authentication subkey, so this is not only a consistency
question.
Route the branch through openVolumeVB and give the lease the engine's lifetime.
The tail both branches shared moves into createSnapshotWithVB, mirroring
copySnapshotOnVolume, which was converted earlier and is what this one was
measured against.
A refusal is reported as unavailable rather than internal: the holder gives the
volume up on detach, so the caller should retry instead of treating a live
mount elsewhere as permanent.
The gateway does not serve S3, so an S3-scoped credential is rejected on
scope by the SigV4 middleware. That rejection was written through
GenerateEC2ErrorResponse, giving the client the EC2 query envelope
<Response><Errors><Error>. SDKs look for a flat top-level <Error><Code>
on an S3 response, find nothing, and report an empty code and an empty
message, which reads as a credential fault rather than a wrong endpoint.
Add the flat S3 envelope and resolve the envelope from the service the
client actually signed with: GetService reports an unserved scope as
empty, which is exactly the case that has to render as S3.
The service-to-envelope decision was copied into four emitters and the
copies had already drifted - writeClusterUnavailable put ELBv2 on the
EC2 envelope while ErrorHandler and writeThrottleError put it on the IAM
one. Collapse them onto a single xmlErrorBody so a new service cannot be
wired into some emitters and missed by others.
s3 is deliberately not added to supportedServices: only the shape of the
rejection changes, not the rejection.
* fix: ECS RunTask and CreateLaunchTemplateVersion honour ClientToken
A retried RunTask placed the tasks again: it mints a fresh task ID per task,
reserves CPU/memory/GPU, hot-plugs an ENI in awsvpc mode and starts the
containers, so a resend with Count 5 left ten running tasks, double the reserved
capacity and five leaked ENIs. CreateLaunchTemplateVersion added a second
identical version and moved $Latest.
RunTask gets a per-action wrapper over idempotency.Do rather than a dispatch
decorator: it is the only ECS action that needs one. A partial placement
finalizes rather than aborts, since re-running would add tasks on top of the
ones already started.
CreateLaunchTemplateVersion reuses the EC2 dispatch decorator, so it is an
opt-in rather than new machinery.
CreateLaunchTemplate, ModifyLaunchTemplate and ECS CreateService accept a token
they do not store; each is name-keyed or a CAS and cannot duplicate, which is
now recorded at the handler.
* test: cover the RunTask client-token binding layer
Split the token layer out of RunTask's NATS binding as runTaskIdempotent, the
same seam runTask already has, so the no-token, store-failure and replay paths
are reachable without a live connection.
CreateVolume, CreateNetworkInterface, CreateNatGateway, CreateRouteTable,
CreateEgressOnlyInternetGateway, CopyImage and CreateCapacityReservation parsed
a ClientToken and never read it, so a retried request created a second resource.
Adds ec2IdempotentHandler at the EC2 dispatch table rather than seven
hand-written wrappers, keyed <action>.<account>.<token> in the bucket
RunInstances already uses. RunInstances keeps its unprefixed keys: re-keying
live records would make a retry re-launch instead of replay.
Promotes the shared orchestration to idempotency.Do, now that eight callers
share it, and wraps store failures in ErrUnavailable so callers map every
non-mismatch token failure onto one API error.
ServeNATSRequestCtx ends its consumer span in a defer that runs after the
reply is published, so a test returning from NATSRequest could read the
span recorder before the consumer span had ended and see no account
attribute. The subscriber callbacks now signal completion and both tests
wait on that signal instead of on the reply.
The cache was a union that only grew: each run restored everything, added
what it compiled and saved the result, so objects superseded by later
commits stayed for Go's full five-day window. Deleting every entry and
letting it rebuild took Build and Test from 3434 MB to 478 MB, restore
from 75s to 13s and save from 64s to 7s.
Dropping entries nothing has touched in four hours holds it near that.
Go bumps an entry's mtime when a run uses it, so the working set
survives; the step runs after the tests, so a pruned entry costs a later
run a partial recompile at worst and never the current one.
Restore is what makes CI fast; save is what got expensive. A same-era A/B
across the switch to this cache shows what restore buys: wall time went
6:50 to 3:52, Build Spinifex 63s to 10s, test-race 385s to 188s. Since
then the cache has grown to its steady state and the work has not moved,
so Build and Test now spends 75s restoring and 64s saving where it once
spent 13s and 8s.
A PR's save is read by nothing else. Splitting restore from save and
gating the save on dev and main keeps every branch reading a cache a dev
merge refreshes minutes later, and takes the upload off the 56% of runs
that are pull requests.
pass and keyPass logged every failure at WARN as "retrying next cycle" and
returned the callee's deadline, which for a failed callee is zero. Zero means
no deadline, so the next attempt was the resync — five minutes by default.
Two things were wrong with that. A fault that clears in seconds left the world
unconverged for minutes, when the loop already had the mechanism to ask for
sooner. And a terminal failure was announced as recoverable: "retrying next
cycle" is false for a 400, which will be a 400 every cycle, forever, at WARN.
Both now go through failed(), which classifies with awserrors.IsTerminal — the
predicate added in #923 for exactly this decision and until now called by
nothing. A terminal error is logged at ERROR and asks for no early retry; any
other error asks to be revisited after DefaultErrorRetry. A deadline the callee
named survives when it is sooner, and clampRevisit still caps every request at
the resync, so a loop can only become more eager than it is today.
It answered its question: of 18 GB on disk, 1.45 GB is untouched for a
day and nothing at all survives three. Go's own trim and the rate CI
runs already keep the cache live, so pruning before the save would shed
under 4% of the bytes at a two-day cut and risk recompiles for it.
The lifecycle handlers published record changes onto the dns.recordset.change
queue group, which load-balances rather than serialises, so any daemon could be
read-modify-writing a zone object at any moment. zonelock.go existed to make
that safe. With the reconcile pass now applying its own batch on the node that
won the election, the desired set is the only path into a zone and the elected
node is its only writer, so both the publish path and the lock go.
Deleted: the four lifecycle publish call sites and their helpers, publish.go,
zonelock.go, SubjectRecordsetChange and QueueGroup, and the lock from
applyZone. Every record they published is already in the desired set the
reconcile rebuilds each pass, and every record they withdrew is already pruned
by it, so the classes DNS carries are unchanged.
The build cache key rolls per run, so each job restores the previous
run's cache, adds to it and uploads the result. Nothing removes entries,
and Go only trims at 5 days, so every restore and save carries whatever
dead weight has accumulated. Build and Test moves 3.2 GB per run: 59s
down, 56s up.
Whether pruning that is worth doing depends on how much of the 3.2 GB is
still live, which is not visible from outside the runner. This reports
the age distribution and deletes nothing. It comes out once the number
is known.
The reconnect tests paid the client's one-second default sleep between
attempts, which ConnectNATS already lets a caller override; the retry test
scales its floor with the delay it sets, so it still proves a retry happened.
7.9s to 5.0s.
The reconcile elected a leader, computed a converging batch, then published it
to the queue group, which load-balances rather than serialises. The batch was
therefore applied by an arbitrary peer, not by the node that won the election.
That split the read that decides which records are stale from the write that
removes them. A lifecycle upsert landing between the two was invisible to the
second, so a record re-added after the reconcile read it was deleted by a
decision that predated it. Applying locally closes the window, and makes the
elected node the only writer the reconcile puts on a zone.
Deleted with their tests: the batch splitter, the payload limits and the
partial-acknowledgement check, none of which have anything left to bound now
that no batch crosses a NATS connection. The lifecycle publishers and the zone
lock are untouched: they are still a second writer, and retiring them is
separate.
Every test already owns its embedded JetStream server on an ephemeral port
under its own temp dir, so nothing here has to run serially. The recovery
failure test also left its doomed Set on a deadline-free context, which made
it wait out the jetstream client's own five-second default.
9.5s to 0.8s, and 11.8s to 2.0s under -race.
A unit test cannot change what e2e exercises, so a push touching only _test.go
files no longer provisions the tofu VMs. tests/e2e/** is listed after the
exclusion so the e2e tests themselves still trigger their own suites.
The guard only reads types for the syntax it walks, which resolves from
dependency export data; NeedDeps type-checked the whole graph from source for
no extra coverage. Verified identical detection either way (160 packages, 993
files, 3684 slog call sites, and both violation classes still caught).
19.0s to 7.3s under -race.
Each test starts its own embedded JetStream server and bucket, so they were
already independent; running them serially just stacked up the fixed settle
sleeps. 13.3s to 2.8s under -race.
* fix: build the EC2 DNS desired set from the instance records so stale records prune
* fix: keep DNS records for stopped instances in the reconcile desired set
The desired set filtered on Running, but the lifecycle publish deliberately
does not withdraw a stopped instance's records: stop and start retain the
address, so they retain the name. With EC2 pruning enabled, absence from the
desired set became a deletion, so a stopped instance would have lost both its
names until the next publish.
Both sides now read one predicate, InstanceRetainsRecords, which replaces
needsDNSWithdrawal and lives next to the names it governs. Records carrying a
DeletionTimestamp are skipped as well, so a re-upsert cannot race the
withdrawal the terminate path has already published.
* refactor: wake the vpcd drift loop on intent changes, not a fixed 5-minute ticker
DriftLoop was the last ticker-driven reconcile loop in the event-driven
reconciliation work. It runs on reconciler.Run now, watching the eight KV
buckets LoadIntentFromKV reads, with DriftInterval kept as the resync
backstop and the existing backoff ladder returned as the pass deadline.
Repair latency for a dropped vpc.* event drops from a uniform 0-300s to
the debounce plus a 5s floor. The floor is new and deliberate: a pass
loads eight buckets and scans the OVN NB DB, so without it a launch storm
would turn a five-minute scan into a continuous one.
The buckets are enumerated rather than fixed so watching never creates
one: the EC2 handlers own them and set their history depth.
The bootstrap ReconcileApplyOnly pass stays the startup scan. The loop's
own startup pass does no work and returns the seed deadline instead, so a
clean boot still waits a full interval and a failed one still requeues at
5s.
* fix: keep the intent watchers up when a bucket lookup blips
A per-bucket resolve failure was skipped, which drops the bucket out of
the enumerated set and has the reconciler tear its watcher down until the
next resync. Only a genuine ErrBucketNotFound is a skip now; anything else
fails the enumeration, which is what tells the caller to leave the
watchers it already has in place.
* fix: measure the drift pass floor from the end of a pass, not its start
Stamping before the scan means a pass slower than the floor leaves no gap
at all behind it. The floor is meant to be the minimum time between
passes, so it is stamped once the scan returns.
* test: wait for the intent watcher before the write it must catch
* fix: follow ListObjectsV2 continuation tokens on every production caller
Six ListObjectsV2 call sites (DeleteSnapshot, DescribeSnapshots, EKS
etcd-snapshot selection, DescribeTags, and both key-pair listings) issued a
single request and never read IsTruncated or NextContinuationToken. This was
latent while predastore ignored max-keys and always answered IsTruncated:
false; predastore#117 pages listings at 1000 keys, turning a truncated
listing beyond that into a silently wrong answer rather than a short one.
DeleteSnapshot in particular could delete 1000 objects, report success, and
orphan the rest, and the EKS restore path could resolve to a stale snapshot
if the newest keys fell past the first page.
Add objectstore.ListAll, a shared helper that drains a listing to
exhaustion, accumulating Contents and CommonPrefixes across pages and
erroring instead of returning a partial result if the server claims
truncation with no token to resume from or never finishes within a bounded
page budget. All six call sites, plus the ebsfake test double with the same
shape, now use it. Bump the predastore pin to dev head ff6a1dc4 so a real
cluster picks up the server-side pagination fix this depends on.
* fix: use external test packages and exclude test fakes from diff coverage
test-package-check requires new test files to bind to the exported API so
internals can change without rewriting tests. Move list_test.go and
delete_snapshot_pagination_test.go to their _test packages, adding a small
export_test.go for the one unexported constant a test needs to assert
against. Simplify list_test.go's pagingStore to embed MemoryObjectStore
instead of hand-stubbing unused methods.
Exclude spinifex/testutil from diff-coverage.sh: those files are fakes and
harnesses driven only from other packages' _test.go files, so go test
records them as zero-covered under the package they live in however hard
another package's tests exercise them, which is not a real coverage gap.
* refactor: drive the ECS scheduler from the bus and deadlines, not four tickers
The scheduler ran four tickers — service reconcile every 10s, heartbeat reaper
every 30s, stopped-task sweep every 60s, instance-role converge every 5m — and
the 10s one rewrote every service record whether or not anything changed.
A KV watch is the wrong tool here: the account bucket takes a heartbeat write
per instance per interval and the key is one NATS token, so no filter can
exclude them. Instead the service reconcile is triggered from the task-state
subscription, which is already separate from the heartbeat one at the bus
layer, and the three timed jobs return their own deadlines.
The service write becomes conditional so an idle service is no longer churned,
and a mutex keeps the two loops from overlapping, which the four tickers got
from sharing one goroutine.
* fix: wake the ECS service reconcile from the path the agent actually uses
The agent stopped publishing to ecs.bus when it moved onto the gateway, so
nothing publishes those subjects any more: the scheduler's task-state
subscription is dead in production and an in-process trigger off it would
never have fired on a live cluster.
A task state report now arrives as SubmitTaskStateChange on whichever worker
serves the call, so that path publishes a payload-free wake on the cluster's
service-reconcile subject and the leader — usually another node — subscribes
to it.
The pass also names its own deadline rather than trusting the wake to arrive:
a service with a task still in flight asks to be revisited at the old 10s
cadence, and only a settled fleet asks for nothing.
listDocuments issued a single ListObjectsV2 and iterated result.Contents.
It never read IsTruncated or NextContinuationToken, so above the server's
page limit the listing returned a prefix of the truth and reported success.
Every caller of ListVolumes, ListVolumesStrict, ListAMIs and ListAMIsStrict
inherited it. The strict variants are the worse case: their whole premise
is that a document they did not see is not evidence of absence, and a
truncated page gives them exactly that with no error to notice.
Key collection now walks the prefix to exhaustion before any document is
fetched. A page that claims truncation without a token to resume from is an
error rather than a short answer, since it is the one truncation a client
cannot honour, and a page count bounds a server that keeps claiming there
is more so it fails instead of hanging.
Nil keys are dropped while collecting rather than carried as empty slots
through the fetch, which removes the empty flag from the result struct.
The loop polled every 30s for two things it was already being told about: the
bootstrap artifacts are keys in the account bucket a watch can see land, and the
control plane's health arrives over core NATS into a field the reconciler read
only on the next tick.
It now runs on reconciler.Run over a watch of the account bucket, plus a new
Config.Trigger fed by the state-report subscription, and returns a revisit
deadline for what elapsed time alone decides: the create timeout, and the instant
the newest report goes stale. A degraded cluster keeps the old interval rather
than restating the three recovery ladders' grace and backoff arithmetic in a
second place.
The trigger only fires on a report that changes health. The control plane
publishes on its own timer regardless, so forwarding every report would have been
the same 30s tick under another name.
applyPorts refuses a port whose SG policy cannot be programmed, but applyEIPs
still ran ensureGuestPortDatapath for every public-IP guest it skipped. GuestPortUp
reports false, not an error, for a port that does not exist, so each refused port
burned the full 45s deadline and ~9 ovn-controller recomputes before giving up.
* feat: give a reconcile pass its own revisit deadline, and convert the RDS loop
The reconciler assumed every loop was a pure function of the buckets it
watches. Quota and DNS are; RDS, EKS and ECS are not. They also wait on an
EC2 VM reaching running, an endpoint answering, and an agent falling silent,
none of which write to KV, so a watch never sees them.
Converting RDS on the watch alone would have traded traffic for detection
latency: an agent going silent writes nothing, so a dead database would have
been noticed on the 5 minute resync rather than within 15s of its heartbeat
expiring.
Reconcile and ReconcileKey now return a revisit deadline, clamped to the
resync. The RDS reconciler returns the earliest across the fleet: the old
interval for a transitional record, the instant the heartbeat expires for a
settled one, the resolve timeout for a snapshot stuck in creating, and
nothing at all for a region with no databases.
* fix: keep a deadline while a dark instance runs down its failure grace
revisitFor measured a settled instance against its heartbeat expiry, and
returned that difference directly. Once the beat expired the difference went
non-positive, which the loop reads as no deadline at all.
That is the window the detector lives in. Between the beat expiring and the
grace closing nothing writes to KV, so there was nothing left to schedule the
pass that does the failing, and a dead database would have waited for the
resync — slower than the ticker this replaced.
The deadline is now the beat expiry while it is still ahead, and whatever
remains of the failure grace once it is not. A failed instance asks for
nothing: it is terminal, and recovery arrives as a heartbeat, which is a write
the watch already delivers. Already due but still not failed is the degraded
reading, dark agent over a live VM, so it retries at the old interval rather
than immediately.
* feat: gate the Ochre console behind a runtime flag
Add OchreEnabled to the spinifexui service config and /api/config
body, defaulting to false so the console and every /bedrock/* route
stay hidden until a node explicitly turns it on.
* feat: wire the Ochre flag through to the spinifex-ui service unit
Add a spinifex-ui-ochre-enabled flag (SPINIFEX_UI_OCHRE_ENABLED, off
by default) so the console flag can actually be set on a deployed
node, and bump the service unit so the reconciler picks it up.
* feat: add knowledge base and data source create/delete to the console
Ochre's KB pages were read-only. Adds create + delete for knowledge
bases and their S3 data sources, mirroring the guardrail CRUD pattern.
* feat: note data source is added after KB creation
Users expect to pick a data source at create time; the create form now
says up front that S3 data sources are attached from the detail page
after the knowledge base exists, then ingested separately.
applyPorts dropped any security group whose OVN port group was missing and
carried on. When that removed the last one, the ENI's logical switch port was
created belonging to no port group, so no SG ACL applied — including the
priority 900/800 default-denies, which are attached per port group — and the
port was fully unrestricted until a later pass healed it.
Fail closed instead: a port whose policy cannot be programmed in full is
refused and reported unconverged, so the drift loop retries it once applySGs
catches up. The update path is skipped too, leaving a live guest on a stale
policy rather than stripping it bare.
EnsurePort and SetPortSecurityGroups get the same guard for an empty SG list.
Every ENI record written by CreateNetworkInterface has at least one SG, but the
field is omitempty with no migration, so a legacy row would otherwise yield an
unrestricted port.
The vCPU sweep ran on a 30s ticker and called the account-filtered
DescribeInstances fan-out once per account per pass, costing
passes x accounts x (nodes + 2) and measured at ~62,000 instance
describes a day, none of it user traffic and none of it tracking how
often anything actually changed.
The account is on the instance record and vCPUs derive from the instance
type, which is spec, so the whole sweep is one read of the record space.
The loop moves onto reconciler.Run: a change to one instance recomputes
that one account, and the resync sweeps every account as the backstop.
The completeness flag has to stay, for a different reason than the one it
was written for. nats.go hardcodes AllowDirect on KV buckets, so a get at
R3 may be served by a lagging replica, and ChargeLaunch increments the
counter while reconcile overwrites it absolutely. A scan that misses a
just-charged launch would lower the counter and lift the cap. The lower
is now written only while the counter still stands at the revision read
before the scan started, so a charge landing mid-scan wins and the scan
is dropped for the next pass.
Two behaviours change with the source. A failed sweep was a per-account
describe error the pass continued past; it is now one read covering every
account, so a failure moves no counter at all. And an account holding
nothing is absent from the scan rather than reporting empty, so the
caller's account list is what zeroes it.
The retype gate and the volume delete guard still use the fan-out; only
the sweep comes off it.
The loop watches KV buckets and runs a whole-set recompute when anything
in them changes, discarding the update it already holds. A caller whose
work is per resource can now set ReconcileKey and be handed the key that
changed, so a change to one instance costs one instance's work.
Reconcile stays required. A per-key queue only converges if something
behind it covers what the queue never learned: a key deleted while the
watch was down is enqueued by nothing, because nobody observes an
absence. Keeping the whole-set pass as the startup pass and the resync
backstop is what makes that real, and it makes the change additive — a
caller that sets only Reconcile behaves exactly as before.
KeyFor maps an update onto the caller's unit of work, which is not
always the KV key: fifty instances changing in one account are one
recompute. Reporting ok=false falls back to a whole-set pass, which is
the honest answer for a delete — a tombstone carries no value, so a
caller whose work key lives in the value cannot name it.
A dropped and re-established watch queues the whole set rather than any
key, since UpdatesOnly hides the gap. One worker drains the queue, and a
key taken off it is no longer pending, so a change arriving mid-reconcile
is queued again rather than folded into the pass already running.
* feat: decide instance set membership from the record, not the key prefix
* feat: cut instance records over to one key per instance
The record space becomes the only copy. Which set an instance belongs to
was structural before this — a stopped instance sat at instance.<id> and a
running one inside the node.<nodeID> blob, so it could not be wrong — and
is now a predicate over the record, decided in one place rather than at
each call site.
The old key spaces are frozen, not deleted. A migration that removed them
would make the crossing unrollbackable, and the node blob is the only
record of what a node was running that a build predating the cutover can
read. instance.<id> and terminated.<id> drain on the delete path.
The per-node key survives as a presence marker at its own prefix. Reads
need it because a node with no instances and a node the cluster has no
record of both scan empty, and only one of them may replace what the node
has locally. It carries no instances, so a state change no longer rewrites
the node's whole running set.
Claiming a stopped instance becomes a compare-and-set. It cannot wait for
a later change: with one key holding an instance for its whole life, a
claim that deletes is deleting the instance it is about to start. The
winner clears the stop intent and leaves LastNode alone, so an instance
whose launch never happens is still recoverable by the node that ran it.
A build predating the cutover now refuses to open a bucket stamped past
what it understands, rather than reading the keys it knows and writing
back a view assembled from half a key space.
* fix: carry node ownership out of the blob key during the cutover migration
The node.<id> key carried two facts in its name: that the node exists, and
which instances are its. The 4->5 migration moved the first and not the
second.
A blob member's own LastNode is empty, because the key was the owner, and so
is last_node on the record copied forward from it. After the cutover
ownership is read off the record, so an unowned record is one no node lists
- and with the presence marker seeded, LoadRunningState returns found=true
with an empty set, which restore adopts over the instances the node is
actually running.
The migration now stamps last_node from the blob's key onto every record in
it that has no owner, and leaves a record that already names one alone.
Found on a live cluster before crossing it. The test helpers built blob
members with LastNode already set, which production blobs never have; the
new tests seed them the way the bucket actually holds them.
* feat: add guardrail create, edit and delete to the console
Adds CRUD for Bedrock guardrails alongside the existing read-only view:
create and edit forms covering topic, content, word, PII, regex and
contextual-grounding policies, a delete confirmation flow, and a
create-version action, all built on the same real-SDK client pattern
already used for guardrail reads and runtime testing.
* fix: mount the guardrail edit form instead of no-opping the link
The guardrail detail route file dot-nested under the edit route
(detail as parent, edit as child), but the detail page renders no
outlet, so navigating to the edit URL matched the route without ever
mounting the edit form. Renaming the detail route file to an index
sibling of edit fixes the mount without changing any full path.
* feat: render predastore admin_port and consult /readyz in health probe
Spinifex now allocates predastore's admin listener port (8660, after the
existing gate/blob/meta bases) and renders admin_port into both the
single-node and multi-node predastore.toml templates. The daemon's health
probe consults /readyz over that port on the two weak paths (single-host
gate fallback and hosts running no meta node), falling back to the prior
behavior when no admin port is configured or the listener refuses.
* build: pin predastore to the release carrying Host.AdminPort
The admin port field this branch reads is new in predastore, so CI could not
compile against the pinned version: go.work hid the gap locally by resolving
predastore to the checkout instead of the module.
* feat: add admin Model Access console page
Lets an admin grant or revoke a provider-tier model for a tenant
account from the console, cross-referencing the model catalog with
that account's current grants via the existing signed-proxy actions.
* feat: replace free-text account id with a real account picker
Sources the Model Access account selector from /admin/ListAccounts
via a new signed JSON admin-API fetch helper, sorted by name and
excluding the global account. Falls back to a validated 12-digit
free-text account id when enumeration is unauthorized or empty, since
that call requires a long-lived IAM user in the super-admin account.
* feat: mirror each node's running set onto the i/<id> key space
* refactor: re-key instance records onto a dot separator
A reconciler Filter is a NATS subject filter, where "*" matches one
dot-delimited token. "i/<id>" is a single token, so "i/*" matches nothing
and a watch over the record space would have had to take the whole bucket
with ">" — waking on every key rather than on instance state, which is
worse than the "node.*" it replaces. "i.<id>" is two tokens, so "i.*" is
the filter the DNS reconcile can narrow to at the cutover.
The space stays disjoint from the three it replaces for the same reason
the slash did: List and DeletePrefix match a plain string prefix, and
"instance.i-1" does not begin "i.".
Each bucket gains a re-key step that moves "i/<id>" to "i.<id>" and
deletes the old key. Deleting is safe under the shadow rule: a node still
writing slashes also reads them, so what it loses is a mirror, and a
missing mirror falls back to the key it mirrors rather than erroring.
A pair of tests asserts both halves — that "i.*" delivers a record write
and that "i/*" does not — so the reasoning that chose the separator is
checked rather than trusted.
* test: benchmark what the node blob costs per state change
The case for splitting the blob is that one instance changing state
rewrites every instance on the node, and that cost grows with N while the
record's does not. That was asserted rather than measured.
Two benchmarks measure both sides on a record sized from env19, which
averages around 10KB terminated: marshalling the blob at N of 1, 10, 50
and 100, and marshalling one record. Each reports the bytes it produced
alongside the time, so the amplification is readable from the output
rather than needing a second calculation.
There is no assertion, because there is no threshold worth failing a
build on. Benchmarks compile in CI and run only when asked, which is what
a measurement wants — a test that cannot fail would run every pass and
gate nothing.
* fix: cover the embedder cold-start window with retry and Retry-After
Guardrail and knowledge-base embed calls hard-failed with a raw 503
whenever the embedding endpoint's connection was refused mid-relaunch,
instead of riding out the short window until it accepted connections.
embedBatch now retries a bounded number of times on connection-refused
or connection-reset, and any resulting 503 carries a Retry-After header
so callers back off instead of hammering the endpoint. A background
warm-up probe gates the embed path for statically pinned endpoints,
logging once when the endpoint first becomes reachable. Guardrail and
embedder infra logs now carry service/action labels for easier triage.
Fail-closed guardrail semantics are unchanged: an unavailable embedder
still denies unverified content, this only changes how quickly it
recovers once the endpoint is reachable again.
* fix: re-verify a persisted READY endpoint's liveness before resolving it
DynamicEndpointResolver's StateReady branch trusted the KV record's base
URL unconditionally. After an awsgw restart the record can still say
READY while the co-resident engine on that VM is mid-rebind, so the
resolver handed callers an address nothing was listening on yet.
describeAndEnsure now re-probes a READY member's own readiness route
before trusting it, short-TTL cached so a warm model still resolves at
no extra cost per request. A member that is not answering falls through
to the same not-ready path as a launch already in flight, so guardrail
and embed callers get the existing retryable answer instead of dialing
a dead address. This applies at the resolver seam, so it covers every
co-served member, not just the embedder.
The spinifex-ui reverse proxy copied only the decoded URL path and dropped
RawPath, so a percent-encoded path segment (a ':' in a bedrock modelId) was
re-encoded before reaching the gateway. The gateway then canonicalised a
different path than the one the browser SDK signed, and SigV4 rejected the
request with a signature mismatch — surfacing as an unparseable '<?xml'
deserialization error in the console Playground.
Carry RawPath through the rewrite so the escaped path arrives byte-identical.
Also route the auth, cluster-unavailable, and throttle error emitters through a
shared jsonErrorService helper so the bedrock family receives AWS JSON 1.1
errors the SDK can parse, and stamp the signed service onto the request before
verification so a rejection renders in the client's format.
* feat: add Ochre console Playground page
A Converse chat surface with model picker, system prompt, inference params,
optional guardrail, per-turn tokens, raw request/response JSON, and copy-as-CLI.
The transport lives in one module so a later ConverseStream swap edits a single
file; the message list appends and mutates the last turn in place. A cold model
degrades to a retryable warming-up turn instead of an error.
* feat: pin the playground composer, send on Enter, render markdown
The chat composer sat below the message list in page flow, so it slid down as
the conversation grew and the reader had to scroll to reach it. Bound the
chat + composer to a fixed-height column: the message pane scrolls internally
and autoscrolls to the newest turn while the composer stays pinned.
Enter now sends (Shift+Enter inserts a newline), and assistant replies render
markdown — lists, inline code, code blocks, and links — instead of raw text.
Adds the accessors for the per-resource instance record key space, over both
buckets that carry instance records. Nothing calls them yet.
The prefix is disjoint from the three it replaces: List and DeletePrefix match
on a plain string prefix rather than on NATS subject tokens, so i/ shares a
bucket with instance., node. and terminated. without colliding with any of them.
mutateVM and listVMs become generic so the record accessors reuse them rather
than carrying a second copy.
InstanceRecord is resource.Object[InstanceSpec, InstanceStatus], the persisted
form of an instance under the per-resource key space the re-key introduces.
Nothing reads or writes it yet: the boundary is settled and tested before the
keys move onto it.
The wrappers around the EBS and ENI request slices are dropped on the way in
and rebuilt on the way out, so the record does not carry a mutex.
Instance and Reservation are carried whole. They are SDK types returned
verbatim by DescribeInstances and hold both halves in one object, so the
boundary does not hold there and the comment says so.
Object[Spec, Status] separates what an operator asked for from what a node
observed, with the identity and lifecycle bookkeeping common to both.
Generation is bumped only through MutateSpec so NeedsSync means something, and
ObserveGeneration records the generation the node acted on rather than the
current one, so a spec change landing mid-reconcile is not marked done.
No ARN or tag helpers: the arn package already builds ARNs per service, and the
tag helpers in utils already cover the map every record carries.
StateReason was cleared by ModifyInstanceAttribute, so the API wrote a field
the node owns. Move the clear onto the launch path.
ExtraHostfwd held the requested guest port and the bound host port in one map,
with zero standing in for both 'not bound yet' and a real value. Split it into
HostfwdPorts and HostfwdPortMap, rebuilt per launch, and fold legacy records on
decode.
The command an instance was last handled with was stored on the record, but
only stop_instance and delete_instance were ever read back. Replace them with
DesiredState and DeletionTimestamp, which say what the read sites actually
mean, and drop the other twelve booleans.
Legacy records are folded on decode so an operator-stopped instance is not
relaunched on upgrade.
A staged self-host model is granted to every account without a per-account
grant, making the admin import the only v1 gate. Wrap the access resolver so
every gate opens together; the admin catalog reports self-host state
weights-first. Per-account grants remain as a dormant seam.
Operator-only catalog page listing every model with modality, streaming,
price, VRAM/instance, co-serve group, and availability with a fix-affordance
hint per reason. Reads the ListOchreCatalog admin action, mirroring the
existing admin-service query pattern.
Expose per-model availability and reason on an operator-only ListOchreCatalog
action for the console, keeping the tenant ListFoundationModels omission
intact. Remove the unbuilt provider-direct entry from the v1 catalog.
The server queues the API audit advisory after the reply the caller
waits on, so subscribing once the bucket exists could still catch the
create the first call made. Watch from the start and account for it.
The handler records its request point after it replies, so asserting as
soon as Request returns could read the metric before it landed. Poll for
the outcome the way the sibling drain test does.
The 40ms apply-params timeout was shared by the tests that expect the
agent to answer and those that expect it not to, so a parallel coverage
run could miss the reply. Only the unreachable-agent cases keep the
short budget.
* feat: semantic topic detection for guardrails via embedding similarity
Guardrail topic policy previously only blocked on an exact word-boundary
match against a denied topic's name or listed examples, so a paraphrase or
synonym for a blocked topic sailed straight through. This adds an
embedding-similarity check: each denied topic's name, definition, and
examples are embedded and cached, incoming text is embedded and compared by
max cosine similarity against those phrase vectors, and a topic blocks when
that similarity clears a conservative threshold. The literal exact-match
check still runs first and always blocks on its own. If no embedder is
configured, or an embed call fails, the check falls back to the existing
literal match alone rather than ever failing open.
* fix: fail closed when guardrail topic embedding is unavailable
The topic policy's semantic check silently fell back to its literal-only
matcher whenever the embedder was configured but temporarily unreachable
(for example during model cold-start), so a paraphrase of a denied topic
could pass through unblocked while the embedding endpoint was down. Now that
case is distinguished from an unconfigured embedder: if literal matching
already caught the request, that block still wins outright. Otherwise, when
the embedder is wired but the embed call itself fails, the guardrail check
returns an error and the caller aborts the request rather than letting
unverified content through.
* feat: converge DNS on change rather than on a timer
The drift backstop now watches the four stores dnsDesiredSet reads: instance state, ELBv2 load balancers, and the per-account EKS and RDS buckets. Only the trigger changes; the leader election, computeBatch and the publish path are untouched.
The interval becomes the backstop for drift the buckets cannot report, such as a record edited directly in the zone object, rather than the path by which a lifecycle event reaches DNS.
* test: cover the DNS watch wiring and reconciler run path
ListUsers, ListRoles, ListInstanceProfiles and ListGroupsForUser built their
results from a nil slice when the filtering loop matched nothing. xmlutil
drops a nil slice entirely rather than emitting an empty element, so each
response omitted a member its output shape marks required.
Each operation's empty-bucket shortcut already returned an empty non-nil
slice, so only the filtering path was affected. Start those slices empty and
non-nil too.
ListAttachedGroupPolicies, ListAttachedRolePolicies, ListPolicies and
ListAttachedUserPolicies declare no required output members, so their nil
slices are legal and are left alone.
DescribeImages projected BootMode through aws.String unconditionally, so an
AMI recording none emitted "", which is outside the model enum. RegisterImage
takes BootMode as an optional input, so this is the normal path for an AMI
registered without one, not stale data. Set the member only when present.
ListAccessKeys and ListGroups built their results from a nil slice when
nothing matched. xmlutil drops a nil slice entirely rather than emitting an
empty element, so the response lost a member the model marks required. Start
both slices empty and non-nil.
Integration conformance drops from 18 violations to 15; the remainder are the
EC2 AccessDenied error-code findings, which are unrelated.
The conformance subtest asserting the clamp ran against a provider holding
no volumes, so len(resp.Volumes) <= MaxListResults held whatever PageSize
did. Removing the clamp branch left the suite green.
Add a unit test on PageSize covering above the cap, at it, below it, and at
or below zero, and give the conformance subtests a fixture so the claim they
keep, that an over-cap request returns a page rather than an error, can fail.
buildInstanceStatus read vm.EC2StateCodes directly, so a StateError instance
surfaced the non-AWS name "error" while DescribeInstances returned "stopped"
for the same instance.
Ubuntu 22.04 was accepted by detect_os but the runtime apt list has never
resolved on jammy, which ships dhcpcd5 rather than dhcpcd-base. No CI cell
or E2E matrix installs on it, and the install docs already claim Ubuntu
26.04 and Debian 13 only, so drop the claim instead of carrying a
jammy-conditional package branch for an untested suite.
Watch is used zero times in spinifex against 33 ticker loops. This adds the primitive on Bucket, which already owns the memoised handle, plus the watch/debounce/resync/re-establish loop, so each converted caller changes only its trigger.
Watches are established before the first pass: a change landing between the two would otherwise wait for the next resync, since the pass that would have seen it ran before the watch existed.
Service.bucket() resolved a JetStream client and did a get-or-create round trip on every KV operation; kvstore.Store memoises the handle per Service.
Add Store.Purge, since the endpoint records are deleted and re-created under the same key and Create is the single-writer claim: a delete marker is not the same starting state as an absent key.
Persist ingestion progress and bound per-document work with a timeout so a
job on a dead backend fails fast and visibly instead of sitting silently
IN_PROGRESS; add StopIngestionJob so a stuck job can be cancelled; map a
backend-down Retrieve/RetrieveAndGenerate error to 503 instead of an opaque
500.
The local helper masked x-amz-security-token by line prefix, which matched the
header form and missed the presigned query form entirely, leaking the token into
the exported logs on every signature mismatch. Defer to bluebottle's
RedactedCanonicalRequest, which knows the structure and masks both forms.
Both files only ever called exported identifiers, so binding them to the
exported API needs just an import alias and qualified call sites. Clears
the test-package-check failure that surfaced once dev's base ref moved to
origin/main.
releaseReservation returned nil for every kv.Get error, so a NATS
timeout or connection drop was reported as a successful release and the
stopped task's CPU/memory/GPU capacity leaked on that instance. Narrow
the nil return to jetstream.ErrKeyNotFound, matching the pattern already
used in service.go and task_stop.go, and return anything else.
The generated policy accepted ssh from any source and setup.sh rewrote
spinifex.nft on every run, so there was nowhere to record an allowlist that
survived an upgrade.
custom.nft is created once, never rewritten, and kept across a node reset.
It holds a define rather than rules, so an operator supplies a value and
cannot write a rule that fails the ruleset. The default accepts every source,
which is what the policy already did.
Teardown no longer purges the index registry or KB/DataSource records by
default, mirroring how they already survive a restart; a --purge-metadata
flag restores the old full wipe for an intentional destroy. On the next
appliance connect, a new reconcile pass re-creates any registry index whose
pg table is missing and re-enqueues ingest from its persisted SourceSpecs,
so a torn-down appliance heals without operator action and keeps its KB id.
getResourceType and ec2ARN both encoded the same resource-type knowledge on
opposite sides of the gateway. The unknown sentinel becomes ok == false, since a
sentinel that still looks like a valid ARN segment is how a wrong ARN gets built;
DescribeTags keeps emitting "unknown" on the wire.
The RDS customer endpoint ENI is configured statically in the guest, but OVN still attached dhcpv4_options to its logical switch port. The guest's cloud-init dhcpcd then took an OVN DHCP lease on that NIC and flushed the static address at the 3600s lease expiry, roughly an hour after every boot. Thread a SuppressDHCP flag from the customer-ENI creation through the ENI record, port event, and PortSpec, and gate the dhcpv4_options assignment at both create sites on it. The primary NIC keeps DHCP for IMDS; a delete and recreate never re-adds DHCP because the reconciler intent carries the flag too.
Multi-node install
- Lead with the fact that the guide builds a three-node cluster, and say
how to go beyond three. The callout sits inside ## Overview because the
docs site discards everything above the first section heading.
- Replace the OVN-only sizing prose with a table covering all three
distributed layers, and move the OVN control-plane detail into the VPC
networking guide, linked from here.
- Add minimum and recommended hardware.
- Move the object storage layout to the end as background; it was the
second thing a reader hit.
- Collapse the ISO conversion notes from a section into one skippable
box, now that --force is unconditional in Step 4 and the formation
port needs no manual intervention.
- Expand verification with spx get nodes and spx top nodes, including
what to look for in each.
- Drop the install-node.sh shortcut (internal tooling) and the
"adding a node later" section (no good story yet).
Corrections found while writing
- Servers 4+ were described as running no database. They run predastore
and viperblock like every other node; only the OVN database cluster is
capped at three. Fixed in four places.
- Step 4 never needed the firewall down: spx admin init opens the
formation port itself for the formation window. Only Step 3's
peer-scoped OVN ports are affected.
- `nft list table inet spinifex_filter | grep spinifex_peers` could never
match — the peer list is an nft define, expanded at load time. Anyone
following it would have seen empty output and assumed the worst.
- Disambiguate "database": with predastore, NATS and OVN all in play it
needed qualifying throughout.
- Align the example output region with the AWS_REGION the guide exports.
Cross-doc links
- Rewrite relative and category-prefixed links to the canonical
/docs/<slug> form. The site publishes docs flat, so ../foo/README.md
and /docs/category/foo both 404.
- Point the six links to unpublished docs at the source repo instead.
- Fix four fragments written against GitHub's anchor algorithm.
* feat: implement EC2 MonitorInstances and UnmonitorInstances
The telemetry collector already supports both EC2 monitoring tiers and
already switches a running poller between them when its discovery file
changes, but nothing could make that change: Monitoring.Enabled was only
ever written at launch, and both API actions returned InvalidAction.
Route through the owning daemon over ec2.cmd.<id> rather than the
spinifex-workers queue group, so an instance running on any node is
reachable; no responders falls back to the shared stopped store, where the
write is a CAS so a racing start-claim cannot resurrect a deleted record.
Also report Monitoring on DescribeInstances. It is derived in
ProjectInstance from the same field the collector's period reads, so a
describe cannot disagree with the tier actually being collected and
instances launched before this change report disabled without a backfill.
* fix: wake the telemetry poller when its monitoring tier changes
poller.run re-read its period inside the tick loop, so a reset waited out
the interval being replaced: moving a 300s instance to the detailed tier
took up to five minutes to have any effect, which reads as the API call
not working. Found verifying monitor-instances on env13.
updateMeta now signals a buffered retier channel when PeriodSeconds moves
and run selects on it alongside the ticker. The signal only resets the
ticker rather than forcing a tick, so a published delta still spans the
period its batch declares. The send is non-blocking and the channel
coalesces, so a slow poller cannot stall the reconcile loop.
* fix: join the daemon host-port ENI to the appliance security group
The customer ENI's self-referencing SG drops all ingress except its own
members, so the daemon's host-port ENI could open TCP but never complete a
postgres handshake. Carry the appliance's security groups onto the daemon ENI
at create, and re-associate an already-created one in place.
* fix: complete teardown GC when a volume's metadata doc is already gone
DeleteVolumeOnTerminate and DetachVolumeOnTerminate propagated the
NoSuchKey error from a missing volume metadata doc as a hard failure, so
the teardown reaper backed off forever and never purged the terminated
record. A missing metadata doc means the volume is already gone, so treat
it as success via objectstore.IsNoSuchKeyError at both call sites, with a
matching tolerance in the instance-cleaner adapter.
* fix: statically address the RDS customer ENI so it survives the DHCP lease
The customer ENI was configured via a one-shot udhcpc with no renewal
against OVN's 3600s lease, so the DB endpoint went address-less an hour
after every boot. Resolve the customer subnet's prefix and gateway at
launch and emit the ENI as a static NIC (CIDR + gateway, never the
default route), mirroring the mgmt NIC, so it no longer depends on the
lease being renewed. The guest static branch now installs the same
source-based return-path policy a non-default DHCP NIC gets.
* fix: delete the OVN logical switch port when an ENI is deleted
An ENI delete published vpc.delete-port fire-and-forget, so a lost or slow
message let the ENI record and its IP free through with the logical switch
port still in OVN. A recreate onto the same IP then had two ports carrying
it and the router latched the stale MAC, blackholing the datapath. Publish
the delete synchronously so the port is confirmed gone before reuse, and
make DeletePort idempotent so an already-absent port is a no-op.
Both the endpoint ENI and the management NIC carry the rds-db-instance tag,
so a first-match pick non-deterministically dialed the unreachable management
NIC. Select the endpoint ENI by its RDS description, falling back to any usable
tagged NIC.
The ELBv2 LB microVM boots with -m 128 (sys.micro). After kernel
reservations the 37.7 MiB initramfs — 45% of it the Go 1.27 lb-agent —
leaves under 512 KiB of headroom, so the guest panics with 'Initramfs
unpacking failed' before init and lb-agent never heartbeats.
Bump sys.micro to 256 MiB so the initramfs unpacks with margin, and add
an uncompressed-initramfs gate to build-microvm-image.sh (the compressed
gate was blind to the guest-RAM footprint) so a future bump fails the
build loudly instead of panicking in production.
A class-change ModifyDBInstance is the intended remedy for a failed DB
instance, but applyPendingModifications applied the parameter group through the
live instance agent first. A dead agent returned ErrCommandUnreachable and
aborted the modify before replaceInstanceVM could run, so the one recovery
lever the API offers could not run for the case that needs it most.
The replacement VM's fresh agent applies the resolved set itself on boot, so
the pre-replace apply against the old dead agent is redundant for a class
change. Tolerate ErrCommandUnreachable there: resolve the class-correct set
(a KV read, no agent), persist it for the fresh agent, and skip only the dead
agent's apply. Parameter-group-only modifies still fail on a dead agent.
* feat: gate self-host inference behind capacity-based ThrottlingException
Add a process-scoped concurrency limiter keyed by (servingAccountID,
modelID) with capacity = catalog MaxConcurrency x max(ModelUnits, 1),
and apply it to every self-host Converse/ConverseStream/InvokeModel/
InvokeModelWithResponseStream path. Over-capacity requests reject
immediately with ThrottlingException instead of queuing inside vLLM,
matching real Bedrock. Managed (Anthropic) providers are untouched.
* test: mark admission_test as in-package for the unexported limiter
* test: cover capacity admission gate and throttling
* feat: back up the ochre pgvector schema to object storage
Adds account-scoped pg_dump/restore of the per-account kb_<accountID>
schema to predastore, gzip-compressed and keyed under the account's
own prefix. Backup shells out to pg_dump (--no-owner --no-privileges
--clean --if-exists) behind a PgDumper seam; restore runs psql then
repairs the account role's grants via EnsureAccount/RegrantAccount
since the dump carries no ownership. Exposed as
`spx admin ochre vector backup/restore --account <id>`.
* test: cover ochre vector backup dump/restore and error paths
* feat: co-served GPU model bundles (catalog, lifecycle, embed/rerank routing)
Serve the generative LLM, embedder and reranker as one co-resident GPU
bundle: EMBEDDING/RERANK catalog modalities + co-serve group, bundle-aware
admission (one GPU claim sized to summed member VRAM), single-flight one-VM
launch with a member->port EndpointRecord, DynamicEndpointResolver member
routing. Ochrevector embed/rerank/token-introspection resolve through the
resolver (static single-endpoint map retired); rerank degrades to plain KNN on
absent/failure. Public InvokeModel-embeddings and top-level Rerank deferred.
* feat: co-serve vLLM+TEI bundle image with per-engine readiness
Replace the single-model vllm-serve unit with a generic bedrock-serve@
systemd template driven by per-member BEDROCK_* env files, dispatching to
vLLM or text-embeddings-router by engine and binding each member to its own
port and weights device. TEI serves directly on its serve port; readiness
probes each member on the route its engine answers (vLLM /v1/models, TEI
/health) with the engine family carried end-to-end from the catalog.
* fix: serve nomic embedder correctly in the co-serve bundle
Bound embeddings requests to the TEI client batch limit so a large
document's chunks are split across requests instead of 422ing. Serve
nomic with explicit mean pooling, since the flat weights layout omits
1_Pooling/config.json and TEI would otherwise default to CLS. Add
bge-base-en-v1.5 as a standalone alternative embedder.
* fix: make the bedrock reaper family-aware and pin-safe
A pinned endpoint is now exempt from terminate-and-relaunch on scrape
failure, mirroring how it is already exempt from idle reclaim, so a
transient unreachability can no longer tear down operator- or
commitment-managed serving.
A bundle with no generative vLLM member exposes no vLLM load metrics, so
it is liveness-probed on /health and kept warm rather than scraped for
idleness on a route it does not serve and reaped in a loop.
* test: lock in co-serve group-member convergence to one VM
* fix: pin self-host HF repo/revision in catalog and stop nested config flattening
Carry HFRepo/HFRevision per self-host catalog entry; weights pull defaults
--hf-repo and --revision from them so a bare pull cannot silently grab a
regressed upstream main (the nomic v5 config.json duplicate-field break).
Restrict fixed config filenames to the repo root so a nested config.json
(sentence-transformers 1_Pooling/) is dropped instead of flattening onto the
model config, and refuse any two source paths that collide at one dest key.
* fix: let endpoint delete clear an account-scoped pinned record and report honestly
A bare 'endpoint delete' resolved the Global key, which never held a pinned,
account-scoped record, then printed 'torn down' over the idempotent no-op while
the record survived. Add --account to the delete command so an operator can
target the account 'endpoint list' shows, and return DeleteEndpointOutput.Removed
so the CLI names a real teardown versus a lookup that found nothing and points
at --account. A gone VM already tolerates NotFound, so a scoped delete clears a
record whose instance was terminated out of band.
Real Bedrock emits contentBlockStart only for toolUse blocks; text flows
messageStart -> contentBlockDelta -> contentBlockStop -> messageStop. An
empty start:{} union violates Smithy's exactly-one-member rule and breaks
strict SDK stream consumers (boto3 errors, the AWS Go SDK stalls).
The drift loop tests launched DriftLoop with a bare go statement and never
joined it, so shrinkDriftTiming's cleanup wrote DriftInterval and
driftBackoffBase while the loop was still reading them in nextDriftBackoff.
Cleanups run LIFO, so registering the join after shrinkDriftTiming makes it
run first and closes the window.
Go 1.27's encoding/json v2 decoder stops at the closing brace and leaves
Encoder.Encode's trailing newline unread. net.Pipe is unbuffered and
synchronous, so the writer blocks on that undelivered byte while the peer
blocks writing its reply, deadlocking on the first QMP command.
TestTryBlockdevDel hung until the 180s package timeout. A real unix socket
is kernel-buffered and matches production QMP transport, which is already
the pattern newMockQMPClientWithEvents uses.
The replacement lease is persisted before releaseStaleUpstream runs, so
the KV predicate did not prove the release had completed and the count
assertion could observe zero under race-detector scheduling.
The RDS reconciler and the ECS scheduler call InitLeaderBucket on every
tick, and it created before attaching. CreateKeyValue against a bucket that
already exists is only a no-op when the stored config matches the requested
one; otherwise JetStream answers 10058 and the caller silently falls back to
an attach.
On a multi-node cluster it never matches. The stored buckets are replicated
while the RDS and ECS calls ask for the default replica count, so every tick
billed an error. That is the whole of the 14-18% JetStream API error rate
seen on env1 and env19, and it landed entirely on the meta leader because
that is the node that serves STREAM.CREATE. Single-node clusters matched and
so sat at 0%, which is why this never showed up in CI.
Attaching first removes the per-tick create outright, so it no longer matters
whether the configs agree. EKS already passed a replica count and so did not
error, but it made the same wasted call every tick and is changed with them;
its replica count now applies to a genuine first creation, which is the only
time it was ever read.
The regression test asserts on the API traffic rather than the return value.
Both orders reach the same handle, so idempotency cannot tell them apart, and
on a single-node test server the old order raised no error at all - it just
issued the create. Watching for the STREAM.CREATE advisory catches it there.
Adds per-account resolution of the [quota] limits from a KV override record,
an admin surface and CLI to set one, RDS and load-balancer dimensions, and an
e2e suite that drives every gate on single and multi node.
Also fixes external IPAM initialising without retry: a node that lost a race
against JetStream convergence silently served the disabled EIP stub for the
life of the process, answering cluster-wide with an empty address list.
predastore pinned MULGA_ROOT_TRACE_RATIO=0.05 so guest block I/O, which is
locally rooted and has no caller trace, would not drown request traces. That
worked, but TraceIDRatioBased decides when the root span is created, before
any error exists, so it discarded 95% of predastore failures along with the
successes. An RCA on a rare locally-rooted failure had a 1-in-20 chance of
having the span at all.
The sink now runs a tail-sampling tier in front of APM Server that keeps every
errored trace, keeps every trace slower than 2s, and samples the rest at 10%.
It judges a trace once it is whole, so it can shed the same volume while
keeping the traces worth having. Dropping the head ratio is what lets it see
them: at 0.05 the tail sampler only ever saw 5% of predastore roots, so its
error rule could keep at most 5% of the failures.
Sized before changing: predastore's true local root rate is ~434 spans/s
across six active CI environments, against 21.7/s reaching the sink today.
Driving the tier at 508 spans/s held it at 88 MiB of a 1.5 GiB cap and 1.8%
CPU with zero drops, and trace storage goes from 14.9 to 22.5 GB per 7 days
on a volume with 735 GB free.
* docs: bring doc titles and descriptions into SEO length limits
Rewrite the frontmatter title and description of every published doc so the
rendered <title> lands in 50-60 characters (including the site suffix) and the
meta description lands in 150-160 characters.
* docs: move SEO titles to a seoTitle field and restore display titles
The frontmatter title renders as the page H1 and the sidebar label, so tuning it
for the title tag shortened nothing and lengthened the visible navigation. Put
the search-facing string in seoTitle and give title back its original value.
* docs: document the frontmatter length rules for the docs site
The artifact bundle carried the guest console and the spinifex journals but
nothing from OVN, so a phase 6 egress failure could not be attributed to an
unresolved gateway nexthop MAC versus a return-path or upstream problem.
Register an OnFailure at the top of phase 6 that writes
static-mac-binding-list, the SB MAC_Binding table and the VPC router's
lr-route-list into the bundle.
The nightly collected only spinifex-* journal units, so a cross-chassis
black-hole could not be attributed to IPsec SA establishment versus SB
port binding. ovn-controller, ovn-northd and ovs-vswitchd log to files
rather than journald, so they were absent from the bundle entirely.
Adds the openvswitch-ipsec.service journal per node, mirroring e2e.yml,
and pulls /var/log/ovn and /var/log/openvswitch off each node.
* docs: bring doc titles and descriptions into SEO length limits
Rewrite the frontmatter title and description of every published doc so the
rendered <title> lands in 50-60 characters (including the site suffix) and the
meta description lands in 150-160 characters.
* docs: move SEO titles to a seoTitle field and restore display titles
The frontmatter title renders as the page H1 and the sidebar label, so tuning it
for the title tag shortened nothing and lengthened the visible navigation. Put
the search-facing string in seoTitle and give title back its original value.
The guest fail-loop ran 36 iterations at ~28 s each (two 10 s curl timeouts,
a 3 s ping, a 5 s sleep), so NAT-E2E-EGRESS-FAIL first printed at ~16m50s
while the console wait gave up at 8m. The marker was structurally
unreachable and an egress failure reported only a byte count.
Derive the wait budget from the loop instead of hard-coding both ends: 10
iterations at a 30 s per-iteration allowance plus 3m for boot and cloud-init,
keeping the same 8m ceiling. The user-data script is now generated from those
constants and the existing marker consts. Dump the last console read via
OnFailure so a budget overrun also leaves the console in artifacts.
TestSGPolicyDatapath/AllowedTraffic waited 60s for curl to reach the
target's HTTP server but nothing ever waited for that server to start.
Its cloud-init runs later than the client's sshd, which is what the
preceding stage actually gates on, so the budget was measured from the
wrong baseline and the gap between the two is unbounded on a loaded
node. The failures reported curl exit 7, a refused connection: a
missing SG allow is a drop ACL here, which would time out as exit 28.
The target guest now reports its listener on the serial console, the
only channel back to a guest whose SG has no SSH ingress, and a new
TargetHTTPReady stage waits for that marker before AllowedTraffic runs
at all. AllowedTraffic keeps its 60s budget, which now only has to
cover chassis flow install, and dumps VPC flow diagnostics when it does
fail so a genuine propagation failure is attributable from CI artifacts.
test-package-check fails on dev without it: the test mutates the
unexported guestPortBackoffBase / guestPortBackoffMax and drives
ensureGuestPortDatapath and recordPortFailure directly.
A node whose NewExternalIPAM call lost the race with the JetStream meta
leader logs a WARN and carries on with d.externalIPAM nil. That nil
concrete pointer was handed straight to the PublicIPAllocator interface,
where it reads as non-nil, so the guard in PrepareRunInstances passed and
findPool derefed a nil receiver. RunInstances into a subnet with
MapPublicIpOnLaunch took the daemon down with SIGSEGV, and because the ENI
was already persisted and the gateway saw only a NATS timeout, nothing
rolled it back: the orphan ENI later rejected DeleteSecurityGroup with
DependencyViolation.
The concrete value now crosses into PublicIPAllocator and PublicIPReleaser
only once init succeeded, and AllocateIP/ReleaseIP tolerate a nil receiver
so another caller repeating the mistake degrades instead of panicking.
Making the interface genuinely nil is not enough on its own: the existing
nil check would then have skipped assignment altogether and booted an
instance with no public address on a subnet that promised one. The
wantPublic decision now runs unconditionally and a missing allocator takes
the allocation-failure path, which detaches and deletes the ENI, returns
capacity and reports InsufficientAddressCapacity.
Launches that do not ask for a public IP are untouched: subnets default to
MapPublicIpOnLaunch only when the cluster has public pools, which is the
same condition that initialises the IPAM.
* feat: ship per-test JUnit results to the Elasticsearch sink
Every e2e workflow already converts its test log to JUnit XML and renders
it into the job summary, then discards it with the artifact. mulga-ci-runs
records which suite failed; nothing records which test failed, so
root-causing a failure means reading raw Actions logs across historical
runs.
The junit-to-es action bulk-indexes one doc per leaf test case into
mulga-ci-test-results: name, subtest, status, duration, failure text, run
id, cell, suite, branch and sha. Parent t.Run wrappers are dropped so a
failing subtest counts once, matching the job summary renderer. The doc id
is deterministic, so a replayed post overwrites rather than
double-counting.
Called from every workflow that produces JUnit XML, and tolerated on
failure: an unreachable sink must not turn a green test run red.
* feat: classify unparsed JUnit cases as unknown, not error
go-junit-report emits <error message="No test result found"> when a
=== RUN line has no matching result line, which means the log it was fed
was truncated or filtered. Recording that as a failure would put 78
phantom failures per green bare-metal run into the pass rate.
Two ports on env19 have no tap behind them anywhere in the cluster:
their NB Logical_Switch_Port reads up: false and the SB Port_Binding
has chassis: [], so nothing can ever claim them. ensureGuestPortDatapath
is already bounded, but the outer reconcile loop calls it again every
cycle, so each dead port pays the full 45s nudge sequence and files
another ERROR roughly twelve times an hour, which is 1500 of vpcd's 1691
daily WARN lines and all of its ERRORs.
A port that burns its convergence deadline now backs off, doubling from
a minute to a thirty-minute cap, and clears the moment it binds. The
failure and recovery lines also carry the ENI, the EIP and the guest IP:
they previously named only the VPC and the LSP, so the guest behind a
stuck port could not be identified from telemetry.
slog renders a bare time.Duration as an int64 nanosecond count and names
the unit nowhere, so every duration field in the fleet read like
"elapsed":135000000000. The repo idiom of .Round(time.Second) changed
the value without changing the shape and truncated every sub-second
measurement to zero.
otelsetup.Millis converts a duration for logging under a snake_case key
ending in _ms, and TestSlogDurationConvention type-checks ./spinifex/...
and ./cmd/... to fail the build when a raw Duration reaches an slog call
or a Millis value is logged under a key without the suffix. It found 55
sites across 34 files, well past the sites a grep for time.Since shows,
because most durations come from a config field or a package constant.
A handler-level ReplaceAttr would fix every site at once but can only
emit d.String(), which Elasticsearch maps as a keyword and nothing can
aggregate over.
2026-08-18 20:38:13 +10:00
2279 changed files with 135547 additions and 26149 deletions
# Connection-layer failures from the libvirt tofu provider. The socket drops
# when a hypervisor is carrying several environments at once, which is the
# steady state here. A resource that failed to build says something else.
CI_LIBVIRT_TRANSIENT='Unable to Connect to Libvirt|error connecting to libvirt|procedure interrupted|Cannot recv data|Cannot write data|End of file while reading data'
# retry_run LABEL LOG_FILE -- CMD ARGS …
# quiet_run with a bounded retry, taken only when that attempt's own output
# names a libvirt connection failure. Every other failure is reported on the
# first attempt, so a real bug never becomes a slow flake.
# Mirrors the suite selection of workflow_call so a baseline capture can be
# driven by hand. A dispatch run sees only the inputs declared here, so
# omitting these would silently fall back to the correctness set.
workflow_dispatch:
inputs:
suites:
description:"Space-separated e2e suites to run on the node. Default is the correctness set; a performance run names its own so its numbers come off an otherwise idle cell"
required:false
type:string
default:"single cert lb"
suite-env:
description:"Extra environment for the suite invocation, e.g. SPINIFEX_DISKPERF_REPS=3"
required:false
type:string
default:""
cell:
description:"Nightly cell number this run reports as. Names the artifact bundle and the ES dimension, so two callers of this workflow stay distinguishable in the sink"
required:false
type:string
default:"20"
workflow_call:
inputs:
ref:
@@ -14,6 +33,21 @@ on:
required:false
type:string
default:""
suites:
description:"Space-separated e2e suites to run on the node. Default is the correctness set; a performance caller names its own so its numbers come off an otherwise idle cell"
required:false
type:string
default:"single cert lb"
suite-env:
description:"Extra environment for the suite invocation, e.g. SPINIFEX_DISKPERF_REPS=3"
required:false
type:string
default:""
cell:
description:"Nightly cell number this run reports as. Names the artifact bundle and the ES dimension, so two callers of this workflow stay distinguishable in the sink"
required:false
type:string
default:"20"
permissions:
contents:read
@@ -21,25 +55,21 @@ permissions:
env:
GIT_BRANCH:${{ github.ref_name }}
# Same central ES/APM sink as the other four e2e workflows. Bare-metal had
# no default here before — this cell's telemetry never reached the sink at
# all (not just missing ci.cell), see bm-bootstrap.sh's telemetry.env step.
# -count=1 is load-bearing: the conformance report is written by the test binary,
# so a cached pass skips the run, leaves no report, and the cat below fails the
# target. It also keeps the conformance gate honest — a cached result would mean
@@ -186,14 +195,13 @@ test-integration:
$(_Q)LOG_IGNORE=1AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE)AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
@@ -29,11 +31,9 @@ Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your
Spinifex is an open-source infrastructure platform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering underneath: distributed object storage with erasure coding, block storage built to survive failure, and compute on bare metal. That depth is the reason your software runs unchanged, instead of rewritten.
Built for teams that need:
It's built for teams that need:
- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- The AWS tooling you already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
@@ -50,15 +50,15 @@ From commodity hardware up to unmodified AWS tooling, every layer is replaceable
## Three Ways To Deploy
One AWS-compatible surface, three deployment shapes. Pick the one that matches your reality; the platform on top is the same.
The same AWS-compatible platform runs in three shapes. Pick the one that matches your hardware.
### Neocloud
Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and onto our Neocloud partner ecosystem without rewriting them. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
Lift and shift onto a partner Neocloud without rewriting workloads. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
### On-premise
Bring your own hardware and host Spinifex in your own data centre. A real multi-node HA cluster integrated with your storage and networking, with full control of stack, data, and jurisdiction. A predictable bill, no egress surprises, an auditable open-source core.
Host Spinifex on your own hardware in your own data centre. You get a multi-node HA cluster integrated with your storage and networking, so the stack, the data, and the jurisdiction stay yours.
### Edge
@@ -66,7 +66,7 @@ Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factorie
## AWS Compatibility
Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform: everything you deploy on AWS deploys on Spinifex unchanged. At the edge, on-premise, or on a partner Neocloud.
Spinifex speaks the AWS API surface. The AWS SDKs, AWS CLI, and Terraform all work unchanged, whether you deploy at the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
| --- | --- | --- |
@@ -80,19 +80,37 @@ Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform: every
| **ECR** | Container Registry | Available |
| **ECS** | Container Service | Available |
| **RDS** | Databases | Available |
| **Bedrock** | AI Deployment | Q3 2026 |
| **Bedrock** | AI Deployment | Q4 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Live Demo
Create a live sandbox in under a minute. The fastest way to experience the stack. Try Spinifex against one of your own workloads on infrastructure we run for you.
* Live in under a minute. No card, no install.
* Yours for 72 hours to test and experiment.
* Real AWS-compatible endpoint, access keys and console.
* EC2, EBS, S3, VPC, IAM, EKS and RDS, ready to call.
* Point your existing AWS CLI or Terraform straight at it.
Sign up at [https://mulgadc.com/signup](https://mulgadc.com/signup)
## Core Components
### Spinifex (Compute Service – EC2 Alternative)
### Spinifex (Compute and Platform Services– EC2, EKS, ECS, RDS)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing an EC2 compatible API. It manages lifecycle operations like start, stop, and terminate through QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs with the AWS CLI, SDKs, or Terraform, without Kubernetes or a heavyweight orchestrator.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
-Works with standard AWS tooling
The managed services build on that same instance layer. An EKS node group, an ECS container instance, and an RDS database are Spinifex VMs on your own hardware, each driven through its own AWS API.
-**EC2** – VM lifecycle on bare metal, cloud-init and IMDS metadata, key pairs, tags, spot instances, capacity reservations, and GPU passthrough
- **EKS** – Managed Kubernetes on k3s, with HA control planes, managed node groups, cluster add-ons, an OIDC provider, and IAM access entries
- **ECS** – Task and service scheduling across container instances, with awsvpc ENIs, capacity providers, rolling deployments, and GPU-aware placement
- **ECR** – Private container registry backed by Predastore, with content-addressed blobs deduplicated per account and lifecycle policies
- **RDS** – Managed PostgreSQL and MariaDB, with parameter groups, automated backups, and DNS endpoints served by Northstar
- **ALB / NLB** – Layer 7 load balancing on HAProxy and layer 4 on nginx `stream`, with health checks and ACM-issued listener certificates
- **VPC and IAM** – OVN-backed networking and security groups, with roles, policies, and SigV4 auth applied across every service
### Viperblock (Block Storage – EBS Alternative)
@@ -101,12 +119,12 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
-Supports high performance WAL logs using local NVMe drives to reduce IO traffic to S3.
- Inmemory read/write block cache for blazing performance.
-High performance WAL logs on local NVMe drives, reducing IO traffic to S3
- In-memory read/write block cache
### Predastore (Object Storage – S3-Compatible)
[Predastore](https://github.com/mulgadc/predastore) is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Data is chunked and distributed across nodes using Reed-Solomon erasure coding, making it fault-tolerant and ideal for large-scale or low-bandwidth scenarios.
[Predastore](https://github.com/mulgadc/predastore) is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Predastore chunks data and distributes it across nodes using Reed-Solomon erasure coding, so it survives node loss and suits large-scale or low-bandwidth sites.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
@@ -128,16 +146,7 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Key Features
- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
- **Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
- **Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
- **Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
- **Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
## Installation
## Self-hosted Installation
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for installing Spinifex.
- **[Northstar](https://github.com/mulgadc/northstar)** - Authoritative DNS server
Each component can be developed independently. See component-specific documentation for focused development guides.
## Spinifex UI
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
Spinifex ships with a built-in web console, an optional alternative to the AWS CLI, SDKs, and Terraform. If you know the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
<p align="center">
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console — dashboard view" width="900">
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console dashboard" width="900">
</p>
The console is served by each node on port `3000` over TLS, and becomes available as soon as`spinifex.target` is up:
Each node serves the console on port `3000` over TLS, and it comes up with`spinifex.target`:
- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
- **Same API, different surface.** The UI issues the same AWS SigV4 calls the CLI does, so RBAC, audit trails, and IAM policies apply the same way.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed. There is no separate user database.
- **Self-hosted, works offline.** The Spinifex binary embeds the UI and each node serves it directly, with no external CDN or analytics calls.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
For the full walkthrough, covering first-time TLS certificate trust, login, and a feature tour, see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
## Development Philosophy
@@ -205,16 +204,9 @@ For the full walkthrough — first-time TLS certificate trust, login, and featur
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
**Real-World Experience:**
- Production AWS service development and operations
- Large-scale infrastructure deployment and management
- Edge computing and resource-constrained environments
- Enterprise security and compliance requirements
### AI-Assisted Development
While Spinifex is architected and implemented by experienced engineers, we leverage**Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
Spinifex is architected and implemented by experienced engineers, with**Claude Code** accelerating certain development tasks. The split keeps the quality and reliability standards that production infrastructure demands:
**How We Use Claude Code:**
@@ -230,12 +222,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
- **Performance Optimization**: Real-world performance tuning and benchmarking
- **Production Operations**: Deployment strategies and operational procedures
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE), with a commercial license available. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Please do not open a public issue for a security vulnerability.
Report it privately, either way works:
- [Open a private security advisory](https://github.com/mulgadc/spinifex/security/advisories/new) on this repository.
- Email <security@mulgadc.com>.
Please include enough for us to reproduce it: the version or commit, the configuration involved, and the steps that trigger it. If you have a proof of concept, include that too.
## What Happens Next
We aim to acknowledge a report within three business days. We will confirm whether we can reproduce it, agree a fix and a disclosure timeline with you, and credit you in the advisory unless you would rather we did not.
## Scope
Spinifex is one part of the [Mulga](https://mulgadc.com) stack. If a vulnerability spans more than one component, report it once against whichever repository you found it in and we will handle the rest.
## Supported Versions
Fixes land on the latest release and on `main`. Older releases are not patched.
adminInitCmd.Flags().Bool("discard-jetstream",true,"With --nodes >= 2, remove this node's existing JetStream streams once every node has joined; =false refuses to form over a non-empty store instead. Ignored with --nodes 1")
adminInitCmd.Flags().String("region","ap-southeast-2","Mulga region to create")
adminInitCmd.Flags().String("az","ap-southeast-2a","Mulga AZ to create")
adminInitCmd.Flags().String("node","node1","Node name, increment for additional nodes (default, node1)")
@@ -415,15 +431,11 @@ func init() {
imagesRemoveCmd.Flags().String("image-id","","AMI ID to remove (required)")
imagesRemoveCmd.Flags().Bool("force",false,"Bypass dependency, ownership and config-corrupt checks (salvage mode)")
fmt.Printf("\nOverride set by %s at %s\n",resp.Overrides.UpdatedBy,resp.Overrides.UpdatedAt)
}
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
joshsiv-mulga
