mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
20b3eae728eea2008588f0eb7eefe7f6f2338836
2364 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ben Duncan
|
20b3eae728 |
fix: record the OCI SDK's own dependencies in go.sum (#1115)
spinifex/cloud/oci imports oci-go-sdk directly, but go.mod listed it as indirect and go.sum was missing three of the modules it needs — sony/gobreaker/v2, gofrs/flock and youmark/pkcs8. The module therefore only type-checked inside the go.work workspace, whose go.work.sum happened to carry them, so a standalone checkout failed to build and the linter reported it as a typecheck error against an unrelated file. |
||
|
Ben Duncan
|
fcda031117 |
test(e2e): make the workbook teardown a gate, not a warning (#1110)
* test(e2e): make the workbook teardown a gate, not a warning A customer who cannot remove what they created is as stuck as one who could not create it, so the teardown is part of what a workbook proves. It logged WARN and passed the suite regardless, and a bucket that could never be emptied went unnoticed for weeks behind exactly that line. The failure it hides is a wedge rather than an error: the AWS provider retries DeleteBucket against a bucket it cannot empty indefinitely. So the run is bounded by DESTROY_TIMEOUT, defaulting to 900s, and the last forty lines of its output are printed when it fails. The cleanup after a failed apply is bounded the same way for the same reason. * test(e2e): print a workbook's PASS after its teardown, not before The verdict now depends on the teardown, so a workbook that built everything correctly and then could not delete it printed PASS and FAIL for the same run. |
||
|
Ben Duncan
|
cacd2c2c85 |
fix: re-read a rejected volume lease renewal before fencing the guest (#1116)
A renewal that times out is treated as transient and keeps the lease, correctly, but the server may have applied it anyway. It is then a revision ahead of what the holder recorded, so every later conditional update is refused for that reason alone — and a refusal was read as a peer having taken the volume. The export was fenced, nbdkit killed and the guest lost its disk, with the log naming a winner of "unknown" because no peer ever existed. A ten-second JetStream stall destroyed four running guests this way. A rejected update now re-reads the entry. One that still carries this lease's owner and generation is this node's own write coming back, so the revision is adopted and the lease continues; anything else, or an entry that has gone, is a real loss. A re-read that cannot be answered keeps the lease without extending it, because local validity is the bound that depends on reading nothing. The rule is stated in its own test: a revision mismatch may not fence before a re-read says the entry is no longer ours, and lapsed local validity always fences whether or not a winner can be named. The weaker rule — that an unnameable winner never fences — would leave a partitioned node writing for as long as it stayed partitioned. |
||
|
Ben Duncan
|
adb3d9876d |
fix: read a tagged bucket's tags through S3 instead of S3 Control (#1109)
The AWS provider reads a bucket's tags with `ListTagsForResource` on S3 Control when it knows the account ID. S3 Control puts the account in the hostname — `https://{account}.{endpoint}` — so against an IP-addressed endpoint the request fails a DNS lookup before a packet leaves, and no endpoint configuration can help. A tagged bucket therefore fails its first read after being created. `skip_requesting_account_id` alone does not fix it: `aws-sdk-go-base` takes the account ID from the STS `GetCallerIdentity` credentials check before it consults that flag, so `skip_credentials_validation` is needed alongside it. With no account ID the provider falls back to `GetBucketTagging`, which predastore serves. `s3-webapp` is the only workbook that creates a bucket, and it uses no `aws_caller_identity` and references no other security group, so a blank account ID costs it nothing. The bucket also gets `force_destroy`, since the webapp uploads into it and a destroy of a used bucket fails `BucketNotEmpty` without it. The e2e harness now asserts the bucket carries the tags Terraform set, which is what catches this regressing, and no longer aborts the remaining workbooks on a failure — `s3-webapp` failing says nothing about whether RDS works, and four nights of `rds-quickstart` coverage were lost that way. `STOP_ON_FAIL=1` restores the early abort. Every workbook's provider versions are pinned, so a run reproduces rather than picking up whatever the registry published that morning. |
||
|
Ben Duncan
|
7163f7d62d |
feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108)
* feat: ship uninstall-spx.sh and fix the OCI bring-up path
Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the
software where node-reset.sh clears state and leaves the node installed. An
operator who installed with `curl ... | bash` had no teardown path at all.
State teardown is delegated to node-reset.sh rather than reimplemented, and
setup.sh now ships both to /usr/local/share/spinifex so they are reachable
without a checkout.
It refuses on a multi-node cluster and --yes does not override that: a flag
meaning "don't ask me" must not also mean "ignore a safety property".
/var/lib/spinifex and every apt package are kept unless asked for, and both
opt-ins confirm separately even under --yes.
Proved by a full uninstall/reinstall round trip on dev-oci, which found two
bugs neither prod nor dev-prod can reach, because both are ISO-installed:
node-reset.sh exited 2, silently, at its last step and after the wipe.
setup.sh creates the spinifex *group* and the spinifex-* service users but
never a *user* named spinifex, so `getent passwd spinifex` exits 2 and
pipefail carried it into set -e.
The uninstaller broke openvswitch-switch on its first run by removing the
socket-perms helper while leaving the drop-in referencing it, failing the
unit at step EXEC with status 203. Fixed on both sides: the uninstaller
removes drop-ins it wrote into other units' namespaces and restarts anything
it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can
never stop another package's daemon.
setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux
bridge it links it to OVS with a veth pair instead of enslaving the NIC, so
the VNIC keeps its MAC identity and netplan keeps owning the addresses. It
now fails on OVS datapath errors instead of reporting success, and the
--management demotion guard keys on /etc/default/ovn-central rather than on
database presence, which the ovn-central package creates on every node.
Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts
along with shellcheck over the four scripts that ship to every node.
* feat: allocate external IPs from the Oracle Cloud API
An OCI VNIC drops any packet whose source is not a registered private IP
object on it, so an external pool cannot be range math there — the address
has to be created through the provider before it exists at all. Add
source = "oci" and the allocator behind it.
One allocation is two OCI objects: a secondary private IP that rides the
wire, and a RESERVED public IP attached to it that the internet reaches.
Allocate returns the public address because that is what AWS semantics are
about, and records the pair so the host plumbing and release path can find
the private half.
The OCI objects are created before the binding is written, and dropped
after it on release — both orders chosen so a crash leaves a discoverable
leak rather than a record that lies. Reconcile squares the two on startup
and is required, not optional: a leaked reserved public IP bills and holds
one of the 50 per-region slots with nothing referencing it. It never
touches an object without our display-name prefix, because the reference
host carries operator-created secondaries on the same VNIC.
Resolve the VNIC from an interface name by MAC through IMDS rather than by
Oracle's VNIC ordering, which is not a documented contract.
Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring
is not a second single-purpose installer.
* fix: refuse source = "oci" outside routed mode, and cover the SDK path
An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound
frame to it regardless of which of its addresses the packet is for.
Measured on the reference host: an internet source reaches 10.200.0.144 as
router-MAC to VNIC-MAC, the public address never appears on the wire at
all, and every off-subnet destination resolves to the same next hop.
Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on
the uplink and ARPs for the address, which policy.go:50 justifies with "the
Linux bridge learns the per-rule external MAC on the veth port like any
other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode
guest's egress is dropped and its ingress never reaches OVN. Routed mode
keeps the address on the host, which already wears the VNIC's MAC.
Pinning every LRP to that MAC does not rescue pool mode: they share one
external logical switch and OVN resolves inbound frames by an L2 lookup on
it, so duplicate MACs work for one VPC and silently misroute past that.
The refusal names the MAC reason, because a refusal that only says "use nat
mode" reads as arbitrary and gets worked around.
Also test the SDK request shapes against a local endpoint. Two of them are
wrong in ways reading does not catch: pagination stopping early reports a
partial VNIC to reconcile, which then drops live bindings, and a nil
privateIpId on detach is omitted from the body entirely, so OCI reads the
update as a no-op, returns 200, and leaves the address attached and billing.
* fix: install host firewall rules where a distro's catch-all reject cannot precede them
Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD.
Two Spinifex datapaths were silently dead behind it on any such host.
EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules
landed after the REJECT and never matched: 24113 packets on the reject, 0 on
both accepts, and guests had no egress. The comment named the case it did not
handle, anticipating a FORWARD policy of DROP — which an append does survive —
rather than an explicit REJECT rule, which it cannot. Insert at the head, and
drain existing copies first, because a rule an earlier build appended still
satisfies -C and would otherwise stay stranded across an upgrade.
Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254
and .253 per instance on the ime- devices, so guest requests are locally
delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init
spent 490s timing out and reported the endpoint as disabled. With one wildcarded
accept the same image boots in 18.6s and finds its datasource.
The natuplink e2e check had the same blind spot the code did, asserting -C and
therefore presence but never position. It now asserts the accepts precede the
first reject.
* feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata
An OCI external address is two objects, not one: a RESERVED public IP that
OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP
on the VNIC that the wire actually carries. Every AWS-facing record holds the
public half; every datapath object must hold the private one.
The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP,
PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile
rebuilds the rule set from AWS-facing intent on every pass — so a correctly
mapped rule installed further upstream is swept on the next reconcile and
replaced with one matching an address that never arrives. The seam is a
DatapathResolver defaulting to identity, so every non-OCI deployment is
unchanged. A resolve failure is fatal: falling back to the public address
would install a rule that reads correctly in ovn-nbctl and blackholes.
ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no
credentials, because vpcd is a different process from the daemon that owns
the allocator and the bindings record is all either would consult.
Two live bugs the in-memory test store was hiding, both found by making the
fake report ErrNotFound the way the real store does:
- KVStore.Mutate could not create, so every pool failed its own first
allocation after OCI had already created and billed the pair.
- Reconcile treated a missing record as a failure, aborting the one pass
that exists to find an object created before its binding was written.
Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's
own metadata service lives — the first guest launch made it a local address
and took the host's metadata and DNS with it. The datapath is OpenFlow
steered, so the guest-facing addresses stay put and only the host-side
endpoint address moves, with a PREROUTING DNAT on ime-+ translating and
conntrack un-NATing the reply. Off by default; identity when unset.
The IMDS firewall rules move to vpcd. They are wildcarded over every
endpoint rather than per-launch work, and spinifex-daemon holds no iptables
grant and no CAP_NET_ADMIN by design, so attempting them aborted every
launch.
* fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package
The error string pointed at DefaultHostBindAddrs, which does not exist — a
caller following it would have looked for a symbol that was never written.
nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries
the //test:in-package marker the package check asks for.
* docs: index what each e2e suite proves
Written because a datapath comment was read as an end-to-end behavioural claim
and the suite that already answered it was found only after the wrong answer
had been given. TestVPCEgressPaths/NATBaseline is the authority on private
subnet egress, and nothing pointed at it.
* fix: source-route an EIP's reply out the VNIC that holds it
An EIP's reply carries the EIP as its source. On a host that pins its
uplink with a policy rule, the reply took the main table instead and left
by the wrong interface — which a cloud VNIC drops, because the address is
registered on a different one. Ingress reached the guest and nothing came
back.
Every bind now mirrors the uplink's rule onto the EIP. The selector is the
rule rather than the interface: both OCI VNICs sit in one subnet, so "the
interface owning the EIP's subnet" names two of them, and two candidate
rules is a refusal rather than a guess. A host with no source routing gets
no rule and no change.
* docs: name the setup-ovn recipe routed mode actually needs on OCI
The OCI guide told operators to run `--wan-bridge=br-wan`, which is
mutually exclusive with `--nat-uplink` and deletes the spx-nat transit
veth routed mode runs on. The node then refuses to bring the uplink up,
with an error naming the flag the guide should have used.
Several messages also claimed routed mode is outbound-only with no
public IPs. That stopped being true when the routed EIP path landed, and
it reads as "what you are about to do cannot work" during the one init
the OCI path documents.
* feat: let routed NAT form a multi-node cluster, and carry every pool through formation
Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no
technical justification -- it was a v1 scope decision -- but routed NAT is the
only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become
"Spinifex on a cloud host is single-node".
It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms,
all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but
every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only
the gateway chassis reaches its own guests. init now says so on stderr rather
than refusing, because the refusal made the defects unreachable and therefore
unfixable. Distributing the EIP datapath is the actual fix.
Lifting it exposed a second defect that stops formation outright. NetworkConfig
held nine flattened Pool* scalars and the init node set PoolName = "wan"
unconditionally; routed NAT renders two pools and the one that matters --
nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a
single pool named "wan" holding the transit gateway and the detected WAN prefix,
and the leader re-rendered its own config from the same struct, so all three
nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData
gains json tags because it is on the wire now rather than a template input.
* feat(install-node): add --external-mode so a cluster can be formed in routed NAT
install-node.sh could not express routed NAT at all. It required
--external-pool/--external-gateway/--external-prefix-len and passed
--external-mode=pool to every spx admin init unconditionally, so the only
supported way to form a routed-NAT cluster was by hand.
--external-mode nat refuses the three pool flags rather than ignoring them, and
threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check
rejects a physical NIC during formation, which is correct in pool mode and
wrong in routed mode, where nothing is bridged.
* fix(ocinet): stop each node deleting its peers' OCI addresses on startup
Reconcile compares the binding record against the private IPs on one VNIC. The
record is one key per pool and therefore cluster-wide, so on more than one node
every peer's binding looks exactly like a binding whose objects OCI has lost.
Each node dropped them, and direction 1 on the owning node then collected the
now-unclaimed objects as leaks -- deleting the reserved public IP and private IP
of a running instance.
Observed on the three-node OCI cluster: node2's startup pass logged "collected a
leaked address" for an address it had allocated ten minutes earlier and "dropped
a stale binding" for one living on node3's VNIC.
The damage did not stop at the OCI objects. With the binding gone,
Lookup.DatapathIP returns the public address unchanged, so the OVN
dnat_and_snat row carries the public address instead of the private half. Guest
egress then leaves the VNIC sourced from a reserved public IP, which OCI drops
because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute
no-ops too, since a public address shares no subnet with any host address. One
destroyed binding takes out both directions.
Binding.VNICID already exists for exactly this ("how a node that comes back from
the dead tells mine from moved on without me") and Reconcile never consulted it.
A binding is now only dropped when it names this node's VNIC; one naming no VNIC
cannot be attributed and is left alone rather than guessed at.
* fix(host): find the EIP's source-routing table when the uplink is held as a /32
uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding
a host address configured with a prefix that contains it. On OCI both VNICs sit
in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts
the subnet's on-link route in the main table at metric 0 and takes the whole
subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was
ever found, no source rule was installed, and the reply left by the primary
VNIC where OCI dropped it for carrying an unregistered source.
The address is not the only record of who owns the subnet. The rule names a
table, and that table holds the on-link route. Consulted as a fallback, only
when the address prefixes matched nobody, so a host that already resolved this
the direct way keeps its existing answer -- prod and dev-prod are unaffected.
Only scope-link routes count. A default route covers every address and would
make every source-routing table an equally good answer.
* docs(oci): correct the multi-node story and name the oci_vnic_id constraint
The nat-mode warning said EIPs and default egress were both broken on more
than one node. Default egress is fine: delete a guest's dnat_and_snat row and
it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the
public-IP path was broken, and the three defects behind it are now fixed.
What remains is not a bug but the shape of the datapath: routed NAT centralises
the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other
nodes cannot reach it, and OCI drops any frame whose source is not registered on
the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway
chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator
writing the pool will read it, and in §9, where the cost is that a gateway
change kills every public IP until the pools are repointed.
The single-node cap itself is gone from the doc: a three-node cluster has been
formed and measured, and the warning proceeds rather than refusing.
* feat(nat): distribute the routed-NAT EIP path so it works on more than one node
Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is
a localnet, so the transit segment a /32 route pointed into existed only on
that one node. Measured on three nodes: all three claimed all three EIPs, the
nodes disagreed on the gateway LRP, and only guests on the gateway chassis were
reachable.
The mechanism to fix it is not new. Pool mode already distributes EIPs by
putting external_mac and logical_port on the dnat_and_snat row and letting
OVN's ARP responder answer for the external IP on the external segment. Routed
mode has an external segment too, the spx-nat veth pair, and it being per node
is the distribution: only the node running the instance can answer ARP on it.
So a routed rule is now distributed on the same terms as a pool one, and its
host route becomes on-link on the transit veth instead of via a gateway LRP
that may be on another chassis. bindHostEIP stops demanding that address for a
distributed rule, which is what previously failed the bind on every node but
one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS
rather than the Southbound DB because the question is about this host.
The gateway LRP keeps its transit-pool address. Default egress routes through
it and works across three nodes today, so making it link-local as pool mode
does would break a working path to fix one distribution has already fixed.
Pool and static are unmoved, and a test asserts that against the same input
rather than leaving it to inspection. A rule with no MAC stays centralised, so
a cluster part-way through an upgrade is never in a state neither shape handles.
Beads: mulga-opyl7, mulga-la0pe
* fix(nat): make routed-EIP host plumbing work on every node
Three guests on three OCI nodes, one guest per node: only one was
reachable. Two defects, both invisible on a single node.
The transit veth took a random MAC per node, but OVN holds one MAC
binding for the transit nexthop across the whole cluster — so every
chassis addressed its egress to whichever node seeded the binding, and
the other kernels dropped it at L2. The pair is node-local, so one
locally-administered address everywhere is unambiguous; setup-ovn.sh
pins it at creation and in the netdev unit, and EnsureUplinkPort repairs
a veth made before that.
Host routes and proxy-ARP also rode the leader-gated reconcile pass, so
a node that never won the lease never plumbed its own guests' EIPs.
BindHostEIPs does that half on its own loop on every node; Bind already
leaves alone an EIP whose ENI is elsewhere, so two nodes never contend.
An unresolvable address skips the sweep rather than shrinking it.
Pool and static are untouched: the new pass is a no-op outside routed
mode, and a test asserts it.
Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on
a node that is already tangled. It deliberately spares the NB/SB
databases, /etc/spinifex, the system-id and the packages, and refuses to
del-br a Linux bridge.
* fix(nat): assert the transit MAC where vpcd actually runs it
EnsureUplinkPort is never called in production — host.Wiring is
constructed nowhere — so the repair landed in a path the deployed
daemon does not take, and three nodes came back with three different
MACs. vpcd re-asserts it beside the NAT egress rules instead, which is
the routed-NAT branch that does run on every start.
* fix(vpcd): make uplink-mode detection say what it saw, and check it
detectBridgeMode logged only the branch it took, so a node detected as
veth when it was wired for nat looked identical to one that was right.
It now reports both probes every time, errors when a node carries both
uplink veths (set up twice, and --teardown is the way out), and errors
when external_mode and the wiring disagree — the datapath follows the
wiring, so the config silently loses that argument. An explicitly
configured bridge_mode gets the same cross-check, since nothing else
ever compared it against the wiring.
The vpc-networking doc now opens nat with "start with pool unless one
of these applies" and gains a section on nat across more than one node;
the admin warning for --external-mode=nat on two or more nodes says
what was verified by hand and what is still uncovered.
* test(nat): prove routed NAT delivers each node's own guests, on a cluster
natuplink ran single-node only, which is exactly where the two defects
that broke three nodes are invisible. It now admits ModeMultinode, and
phase 9 takes a second shape there: a guest per node in a spread
placement group, and for each one — its public IP plumbed by the node
running it and by no other node, the route on-link with proxy-ARP
answered there, a dnat_and_snat row carrying external_mac and
logical_port, a TCP handshake in from another node, and the guest's own
console verdict for egress. Every node's transit veth is checked
against the one cluster-wide MAC before any guest boots.
Single-node phase 9 gains the same row and on-link assertions, so the
shape is held on both topologies.
Two gates come from what three OCI nodes actually do: every node holds
a gateway_chassis row for every VPC (the active one is the highest
priority) and every node installs the VPC ingress route, so only phase
7 needs gating — it runs where the gateway port and the guest are both
local, and logs which one is missing otherwise.
Cell 20 is real-multi x nat with its own nightly step, leaving the
permutation cells' suite set alone.
* test(nat): run the VPC egress paths in nat mode, not pool mode only
The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on
external_mode = "pool", with a comment saying nat clusters have no
EIP/NAT-GW support. They have had EIPs since the routed path landed, so
the gate now asks whether the cluster can allocate public addresses at
all — pool, or nat with a non-transit pool configured.
The nat-single cell runs TestVPCEgressPaths beside natuplink, which is
the only nightly coverage of a private subnet's isolation and its NAT
gateway in routed mode. natuplink never creates a private subnet.
* docs(config): list nat among external_mode values
* fix(ci): renumber the multi-node nat cell off the bare-metal job
Cell 20 is already the bare-metal single-node job, gated on the same
cells input but declared in the workflow rather than e2e-cells.json.
A matrix entry with the same number runs two unrelated jobs under one
name, and naming 20 in a dispatch selects both.
* feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests
Oracle publishes its qcow2 checksums inline on an HTML page and ships no
sums file, so the four new entries carry the expected digest in the catalog
rather than a URL to fetch one from. Images gains ChecksumDigest for that,
and the catalog import path verifies against it and says which of the two
sources it used, so a recorded digest never reads as a fetched sums file.
Pinning is safe here and only here: each URL names an immutable build
(b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build
alongside the plain -kvm- one; the cloud build is the cloud-init image and
is the one that matches x86_64's -kvm-.
TestImageCatalogReachable HEADs every URL in the catalog from the general
single-node suite. These artifacts live on hosts nobody here controls, so
an upstream that retires a build breaks `spx admin images import` for a
customer and nothing else in the suite would notice. It also asserts that
every entry sets exactly one checksum source, and that no pinned entry
points at a "latest" URL whose bytes would change under it.
* fix(formation): carry bridge_mode onto the leader and every joiner
applyNetworkConfig copied external_mode, external_interface and the pools
out of the formation NetworkConfig but never bridge_mode, so a multi-node
cluster wrote a spinifex.toml without the key while single-node init wrote
it. Every node then fell back to autodetecting a mode that init had pinned.
Nightly cell 30 caught it: TestNATUplink failed at the config phase on a
spinifex.toml that did not contain bridge_mode = "nat". Deriving it from
the mode rather than adding a NetworkConfig field keeps the leader and the
joiners on the same rule single-node init already uses.
* feat(ui): show how to ssh into an instance, and which user to use
An instance detail page gave the addresses and the key pair name but never
the login user, which is the one part of the command nobody can derive. It
belongs to the AMI, not to Spinifex: nothing in the launch path creates or
renames an account, so stock cloud-init applies the distro's own
default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is
ubuntu, and guessing from the distro name gets two of those three wrong.
The card resolves the user from the AMI's name and description, and an
unrecognised image yields a placeholder rather than a guess, because a
wrong username in a copy-paste command is worse than an honest gap. A
private-only instance is called out as reachable from inside the VPC only,
rather than presented as a command that will simply time out.
* fix(nat): plumb host state for a NAT gateway's public address
AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT
node the host is what delivers a public address, so an egressing private guest
left with a source the host held no route back to, and the masquerade rule —
which matches the transit 100.127.0.0/24 alone — never saw the packet. The
gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD
accepts an EIP gets.
hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway
address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs
deliberately: pruneOrphanEIPs and addLive read that set as the complete list of
addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so
widening it would stop the sweep reclaiming a stale row on an address later
reused as a gateway's.
AddNATGateway also binds directly, on both the create and idempotent-skip
paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP
pass — and so a reboot that keeps the OVN row still replays the volatile half.
Documents NAT gateways in nat mode in vpc-networking, and says at the head of
that section that nat is a supported mode covered by nightly cells 19 and 30.
* docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user
Four catalog entries, all UEFI. The login user is the one that catches
people: opc is the default on the images Oracle publishes into OCI itself,
while these are the generic KVM cloud images from yum.oracle.com, whose
cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle
and ec2-user are all refused.
Also records why these four pin a checksum digest inline (Oracle ships no sums
file, and each URL names an immutable build), why arm64 takes the -kvm-cloud-
build, and that SELinux ships enforcing.
* fix(reconcile): stop reading an unreadable intent as an empty one
A wedged JetStream KV returned no keys for the ENI bucket while writes were
still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts
read every guest LSP as an orphan against it, and every guest on the node lost
DHCP and IMDS while the API kept answering and the instances stayed "running".
The existing re-read guard did not help: it re-reads through the same path.
Three defences, because no single one covers every shape of the fault.
A failed kv.Get now fails the pass instead of dropping the record. Dropping one
VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads
as permission to delete every guest port in that VPC — one failed read, same
outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that
is a poison record rather than an unhealthy store, and failing on it would
wedge reconcile forever.
The port sweep now asks local OVS which taps are plugged in here and never
deletes a port that has one. A running guest's port cannot be an orphan
whatever intent says, and OVS is the one liveness signal that does not come
through the KV read path the sweep is otherwise entirely driven by. Wired from
vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the
sweep rather than deciding on intent alone.
Each sweep also refuses outright when the keep set it would delete against
holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That
is the case local taps cannot cover, because the drift sweep is leader-gated
and lists NB cluster-wide, so the leader holds no tap for another node's guest.
The cost is real: a genuinely orphaned row now survives until one live record
of its class exists. Leaking a stale row is the recoverable half.
Three existing prune tests drove their fixtures to a wholly empty intent, which
is now the refusal case; they keep one live record of the class so they still
prove the prune. The Get-failure path has no unit test — there is no
jetstream.JetStream fake in the tree and an embedded server cannot be made to
list a key it will then refuse.
* docs(oci): separate the two address quotas, and the principal that can read them
The regional reserved-public-IP ceiling is tenancy-wide and does not grow with
node count, so a cluster hits it before the per-VNIC 64, which does. Records
the commands that read both, and that a detached reserved public IP still
bills.
Reading limits or usage needs a tenancy-level grant, which cannot be
compartment-scoped the way the node's policy is. That is a second, read-only
principal kept off every node, not an addition to SpinifexOperators.
* docs(oci): give quotas their own troubleshooting section, and say we enforce none
Spinifex hardcodes no address ceiling: it attempts the allocation and maps
OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with
Oracle and nothing here is reconfigured or restarted — which is what an
operator needs to know before it bites, not after.
Separates the two quotas by how they scale: the regional reserved-public-IP
default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and
is not.
* fix(oci): follow a guest to its new node when its EIP moves
OCI delivers a public address to whichever VNIC holds the private IP behind
it, and nothing ever moved that private IP. Binding.VNICID was written once at
allocate time, as the VNIC of whichever queue-group daemon happened to serve
the AWS request, and never updated — so an address landed on the right node
only by chance, and a guest that stopped and started elsewhere went dark while
every OVN row, host route and firewall rule on the new node was correct.
ClaimLocalAddresses closes the gap from the node that should hold the address.
Local OVS decides ownership, the same authority the host EIP binder already
trusts for the same question, and UpdatePrivateIp with a new VnicId does the
move in one server-side call. The private IP OCID survives it, so the reserved
public IP attached to it follows without being touched and the customer's
address never changes.
A loop per node, not leader-gated: the question is about this host's guests,
and a node that never wins the reconcile lease would otherwise never claim the
addresses of the instances it is running.
* fix(oci): record which ENI an EIP is associated with
An EIP is allocated bare and attached by a later AssociateAddress, so the OCI
binding written at allocate time named no ENI and nothing ever filled it in.
The address then stayed on whichever node happened to serve the allocation:
the affinity pass could follow an auto-assigned address, which is allocated
with its ENI, but never an EIP — the case customers actually use.
BindOwner is on an optional OwnerTracker interface rather than Allocator,
because only OCI has a notion of an address being delivered to one node. A
static or DHCP address is on the wire everywhere its node can reach, those
allocators do not implement it, and ExternalIPAM treats a pool that does not
as a success with nothing to do.
The same gap made the release owner guard inert for EIPs: it compares against
a binding ENI that was always empty, so a stale teardown could free an address
already handed to another instance.
* fix(ec2): report the address an instance is actually reachable on
Three faults kept DescribeInstances answering with the address launch assigned,
whatever happened afterwards.
The launch-time stamp lives on the stored ec2.Instance, and the stopped path
only declined to add an address rather than removing one, so a stopped instance
went on advertising an address it had released and that may already belong to
someone else. The projection now clears it, and the NIC association with it,
working on its copy so a restart can still report the address.
The running path filled the public address only when it was nil, which the
stamp never is, so an EIP could not be reported however correctly it was
recorded. The runtime address now wins.
And nothing told the rest of the system about an association: AssociateAddress
wrote neither the ENI record, which is what IMDS and the network reconcile
read, nor the instance record, which is what a describe answers from. The ENI
write is direct; the instance write is a fan-out, because the queue-group
worker that serves AssociateAddress is rarely the node running the instance.
* fix(net): take the host EIP plumbing off a node its guest has left
Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept
anything still wanted cluster-wide. A guest that moved therefore left the old
node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with
nothing that would ever remove them. On a shared segment that node goes on
answering ARP for an address another node is now serving.
The two halves now ask the same question by the same authority, in the loop
that already runs per node. An unreadable local OVS is not a yes: tearing down
a live address on a guess would fire everywhere at once, so it keeps the
binding and the next pass tries again. A binder with no chassis test behaves
exactly as before, and a centralised EIP is exempt because it hairpins through
the gateway chassis and every node plumbs it legitimately.
* fix(net): refuse to free an Elastic IP on behalf of a departing interface
Releasing an instance's public address on teardown releases it by address,
scoped to the ENI holding it. Since an association now writes the EIP onto the
instance record, that address is the EIP — so terminating an instance gave a
customer's Elastic IP back to the pool while describe-addresses still reported
it as theirs, free to be handed to another guest.
The existing ownership guard cannot catch this. It only fires when the stored
allocation already names an ENI, and an EIP is allocated bare and associated
later, so the stored ENI is empty and the guard passes straight through.
The distinction that holds is who is asking. ReleaseAddress is the address's own
owner and names no interface; instance teardown names the interface it is tearing
down. So a release that names an owner may free an auto-assigned address and
never an EIP, which is enforced in each of the three allocators.
DHCP needed the scope carried on the wire, since it releases by address and
resolves the lease afterwards. Absent on an older caller, which reads as false
and keeps the previous behaviour.
Worst on the two providers where it is not recoverable: on DHCP the lease is the
only thing holding the address and the server promises nothing about handing it
back, and on OCI the address is a reserved public IP object that the release
destroys outright.
* test(e2e): prove an Elastic IP survives a stop and start
Instance teardown releases the public address by value, and since the
association writes the EIP onto the instance record, that release aims
straight at the customer's Elastic IP. The allocator guards now refuse it,
but nothing end to end proved the address comes back on a restart.
The step goes inside EIPFlip, which already runs last on the shared public
guest for the same reason a stop is safe there: the address the guest
returns on is the allocation's, not the pool's. It runs on the static
cells and the DHCP ones alike, and on DHCP it doubles as the only outside
view of the lease still renewing while the guest is down.
* feat: release an auto-assigned public address on stop, take a new one on start
AWS lends an instance a public address for as long as it runs and gives it a
different one on the next start. Spinifex kept the address across a stop, which
holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a
lease nothing is using.
A stop now offers the address back through the instance cleaner, which refuses
anything an Elastic IP is behind — the EIP association record answers that, the
same question an ENI delete already asks — and records on the VM that the start
owes it a new one. A host drain keeps the address: Restore relaunches the guest
on the next boot, and changing its address under it is an outage nobody asked
for.
The start allocates before the launch, so an exhausted pool refuses the start
rather than booting a guest the customer cannot reach, adopts an Elastic IP
associated while the instance was stopped instead of assigning a second address
beside it, and hands the address back if the launch then fails.
e2e: a StopStartAddress stage on the public guest asserts a stopped instance
reports no address and comes back on a different one from a static pool, and a
StopStartKeepsTheEIP step asserts the opposite for an Elastic IP.
* docs: restructure the OCI guide around two deployment paths, with diagrams
The guide had grown into a numbered wall with no shape: no frontmatter, no
contents, and a reader with an OCI account had no way to tell where to start.
It now follows the multi-node install template — overview, prerequisites, then
the deployment path — and splits that path in two: Terraform for one node or
three, which is the preferred route, and a manual single-node walkthrough for
an instance that already exists. Both meet at one shared configuration section
rather than repeating it.
Two ASCII diagrams carry what the prose could not: one OCI instance with the
whole AWS surface inside it, and three nodes with the same surface distributed,
showing which plane each VNIC carries and how an address follows a guest that
moves.
The limits section was materially out of date and said so in the places that
mattered most. Multi-node is no longer experimental, public addresses are no
longer pinned to one node — each allocates on its own VNIC and the affinity
pass moves the private half to wherever the guest is — and an auto-assigned
address is now returned to OCI on stop, so a stopped fleet stops paying for
addresses. What changed is recorded rather than quietly edited.
The Terraform configuration moves in from the monorepo as
scripts/terraform/oci-spx, so the guide points at something in this repository.
Also marks the two new lifecycle test files in-package, with the reason each
one needs to be: both substitute unexported dependencies.
* docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan
Replaces the two ASCII diagrams with SVGs in the house style, one for a
single node and one for three. Renames the two deployment sections to
Terraform deployment and Manual deployment, documents what the Terraform
builds per node and every variable worth setting, and explains that the
data volume is an iSCSI attachment rather than a disk.
The manual path now covers the iSCSI login, the fstab entry and the
netplan source routing that Terraform writes, and states what br-wan is
for on a routed-NAT node: it names the VNIC for the OCI allocator and
holds the registered addresses, but is never bridged into OVS.
Opens with what Spinifex is and why OCI is worth running it on.
* fix: make gateway convergence work on a multi-node routed-NAT cluster
Three defects each stopped a multi-node cluster from ever attaching an
internet gateway, and all three produced the same uninformative symptom:
`terraform apply` sitting on `aws_internet_gateway.igw: Still creating...`
forever while every service reported healthy.
ovn-northd was given only its local OVSDB socket as a client connection.
A RAFT follower does not forward writes -- it answers "not cluster leader,
trying another server" -- so with one entry in the list, northd stops
translating Northbound intent into Southbound flows the moment database
leadership moves to another node. Measured on the OCI cluster:
SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no
chassisredirect port for any VPC created after leadership moved.
setup-ovn.sh wrote the member list only when the caller passed
--ovn-remote, which only install-node.sh did; it now derives the list from
the live RAFT membership otherwise, and says so when it finds a single
member, because that node needs a second pass once the others join.
The gateway datapath probe ran wherever the reconcile lease landed rather
than on the chassis owning the port. Both probes leave from the local
uplink, and under routed NAT the transit subnet exists identically on every
node, so a remote probe reported a healthy gateway as dead and aimed a
repair at a datapath the node does not own. GatewayPortLocal gates it.
On OCI the probe then ARPed the tenant-visible public IP, which the cloud
NATs and which appears on no interface anywhere. The reconciler now maps
the address through the same DatapathResolver the NAT rules are built from.
Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC
delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed
the destroy and stranded the VPC.
The OCI guide gains a Terraform section recording which shipped workbooks
were validated against a three-node OCI cluster and which are still open.
* docs: record the validated Terraform workbook results for the three-node OCI cluster
Six of the seven workbooks run against the OCI cluster pass, including the
three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the
clean eks-quickstart re-run, and names the two lifecycle gaps that bit during
the run so an operator recognises them rather than reading them as a broken
workbook.
* fix: route the NAT gateway's SNAT source through the datapath resolver
AddNATGateway was the one NAT path that did not map its external address to
the one that rides the wire. Where the public half of an allocation is NAT'd by
the cloud it is on no interface anywhere, so every private-subnet guest egressed
with a source that was dropped, while the gateway reported available and DNS
kept working because the VPC resolver is host-local.
The unmapped address also failed the host EIP sweep, which works in resolved
addresses, so the bind was pruned as foreign on the next pass and reinstalled on
the one after. The public half now survives as an external ID instead.
Also restructures the OCI guide around numbered steps through to running
Terraform, and moves the cloud-init and metadata sections into troubleshooting
where they are read when something is wrong rather than followed when it is not.
* fix: place a NAT gateway's OCI address on the node that forwards for it
OCI drops a packet whose source is not a registered private IP on the VNIC it
leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis.
Two independent placements decided those and nothing reconciled them: the
address was created by whichever worker served AllocateAddress, while the
chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster
that made private-subnet egress a coin flip, and the same Terraform passed or
died depending on which node answered.
The affinity pass already solves this for guests, by asking local OVS where the
tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which
is exactly and only this address. The equivalent local fact is the VPC's
chassisredirect binding, so ownership now has two answers rather than one, and
an address that is neither a guest's nor a gateway's is still left alone.
The VPC is recorded from the reconcile pass rather than from CreateNatGateway,
because the answer is the gateway chassis and reconcile is what decides it.
* fix: plumb a NAT gateway's host ingress only on its gateway chassis
A NAT gateway's address has no ENI, so neither the bind's locality test nor
the prune's could ever fire for it and every node installed its /32 route into
OVN. Latent on bare metal, where pool addresses are disjoint from the node
subnet; on a cloud that hands out the address from the nodes' own subnet it
outranks the connected route and swallows node-to-node traffic.
Give that address the same second authority the ocinet affinity pass has: the
VPC's chassisredirect binding, read from the local ovn-controller. An
unanswerable question still leaves the binding alone, so a single node whose
gateway port is not yet claimed behaves exactly as before.
The orphan EIP prune built its live set from ENI and user EIPs alone while also
sweeping host ingress, so every drift pass tore down the ingress of every live
NAT gateway and left it dark until the next host EIP tick.
* fix: ask the gateway chassis only about a NAT gateway's address
gatewayElsewhere skipped the NAT gateway marker it was named for and answered
for every spec, so on a node that is not the VPC's gateway chassis every EIP —
including a guest's, whose locality is its own port — was left unplumbed. Two
of three nodes bound nothing.
* test: assert the two private-subnet EKS workbooks in the tofu harness
eks-https-ingress and eks-gitops-argocd put their workers in private subnets
behind a NAT gateway, so a worker only registers Ready if its egress to the
cluster endpoint and to ECR survives that path. That makes them the workbooks
that catch a NAT gateway fault, and neither had an assertion, so neither could
be run at all — run_workbook calls assert_<name> by construction.
Both share eks-quickstart's readiness check, extracted unchanged. They also
name the worker size node_instance_type rather than instance_type, and tofu
rejects a -var for an undeclared variable outright, so the name is read from
the config instead of kept in a list here.
* test: count agents, not every node, when asserting EKS workers Ready
The control plane is three K3s servers, so an unfiltered node count reaches a
one-worker cluster's desired size before any worker joins — the assertion
passed on exactly the failure it exists to catch, reporting "4/1 workers
Ready". Filtering on the control-plane role label makes it 1/1, which is the
number that says a node-group worker registered.
Matters most for the two private-subnet workbooks, where the worker joining is
the whole proof that egress through the NAT gateway works.
* docs: record the two private-subnet EKS workbooks passing on OCI
|
||
|
Julian Sommer
|
6d4a66be90 | fix(sts): reject session policies and tags as unsupported, and echo SourceIdentity (#1105) | ||
|
Julian Sommer
|
cb83f9b193 | fix(bedrock): GetFoundationModel answers an unknown model with ValidationException (#1104) | ||
|
Julian Sommer
|
372991c304 | feat: implement EC2 ModifySecurityGroupRules (#1103) | ||
|
Julian Sommer
|
28963e28af | feat: verify local image imports with --checksum and record AMI source digests (#1102) | ||
|
Julian Sommer
|
85dd1dcd75 | docs: stop skipping the account ID lookup in Terraform provider blocks (#1101) | ||
|
Julian Sommer
|
da2e6c5c7b | chore: update coverage | ||
|
Julian Sommer
|
71059834e7 | feat(gpu): multi-GPU instance types (#1099) | ||
|
Josh Siviour
|
5c5a2bc0b7 |
fix: omit the user data attribute value when an instance has none (#1097)
DescribeInstanceAttribute always returned a Value for userData, an empty string when the instance was launched without any. AWS returns the attribute with no Value at all in that case. The Terraform provider reads user_data only when Value is non-nil, so an empty string made it record the hash of the empty string in state and propose a change on every plan for an instance whose configuration names no user data. |
||
|
Josh Siviour
|
259b737f7a |
fix: size each EBS provider deadline to the work its subject names (#1096)
One 30s timeout bounded every ebs.provider.v1.* request. capabilities, volume.describe and volume.list answer from state already held; volume.create clones an AMI snapshot. The cheap calls set the budget and the clone inherited it. On a freshly restarted cluster the first clone of an AMI does not finish inside that budget, so RunInstances took a context deadline for a preparation failure and terminated the instance, taking its ENI and volumes with it. A retry of the identical request succeeded once the snapshot had been read once. The subjects that move or reference data now carry 180s, matching unmountSealTimeout, the existing budget here for an EBS operation the caller cannot bound. The classification lives with the subject constants, so a caller constructing a provider cannot reintroduce one flat value, and a new subject stays fast until someone classifies it. Retrying a timed-out create is deliberately not attempted: the responder may still be running one, and that needs idempotency this change does not establish. |
||
|
Julian Sommer
|
0ae2826cca | chore: update pnpm | ||
|
Josh Siviour
|
2dee58d749 |
fix: default the root block device to the name the AMI declares (#1095)
RunInstances defaulted the root device to /dev/vda, the path the guest sees, so an instance launched without a block device mapping reported a DeviceName that contradicted its own RootDeviceName of /dev/sda1. ECS capacity and EKS nodegroups named the same guest path explicitly, so every internally launched instance carried it too. All three now name ebsmetadata.RootDeviceName, which is what AWS returns and what the guest already resolves through its own device links. |
||
|
Julian Sommer
|
b53ac2d148 | chore: frontend deps | ||
|
dependabot[bot]
|
7cc8f59a7d | chore: bump the go-dependencies group with 5 updates (#1094) | ||
|
Josh Siviour
|
4fc99a5e3e |
fix: report the block device name the caller asked for (#1093)
BlockDeviceMappings[].DeviceName was overwritten with the path the guest kernel assigns, on every launch, start and hot-attach. The name is ForceNew on aws_instance.ebs_block_device and root_block_device, so a value that never reads back as it was sent is not drift to reconcile: every plan destroyed and recreated the instance and everything on its volumes, and a stock terraform-aws-modules/ec2-instance configuration never converged. The convention it served — lsblk inside the guest showing the same names as DescribeInstances — is not one AWS keeps either. On Nitro the API reports /dev/sda1 while the guest shows /dev/nvme0n1p1, and IMDS block-device-mapping reports the API name too. The API name is a label, and the wire contract owes the caller the label it supplied. UpdateGuestDeviceNames becomes LogGuestDeviceMap: the guest paths are still discovered and logged for an operator correlating a volume against what the guest sees, and no longer recorded as the device the caller asked for. AttachVolume records the API name it already returns and already persists to volume metadata. IMDS projects the same field and so becomes AWS-shaped with it. |
||
|
Josh Siviour
|
8b5e23da70 |
fix: return the OIDC provider URL with the scheme it was created with (#1092)
The Terraform AWS provider compares the stored url against the configured one through suppressOpenIDURL, which parses the stored value and forces the https scheme onto it. An issuer carrying a port is not parseable without a scheme — Go reads the host as one and rejects the rest — so the parse fails, the suppressor returns false, and the ForceNew url attribute replaces the provider on every plan. Every issuer this platform mints carries a port, so the AWS-shaped stripped form is the one that cannot round-trip here. The ARN keeps the stripped host-and-path form, which is what AWS uses and what issuerFromOIDCProviderARN reverses. |
||
|
Josh Siviour
|
c2332746f3 |
fix: retry a recovery-failed instance and stop calling a dead record running (#1091)
An instance whose relaunch fails during daemon-startup recovery goes to StateError, and nothing ever retried it: every later restart read the same StateError and skipped it again, permanently. The usual cause is a dependency that was not ready at boot, which is over seconds later, so the instance stayed down for a condition that had already cleared. The manager already bounds restarts for a crashing instance, through Health.CrashCount and Health.FirstCrashTime against MaxRestartsInWindow. A recovery failure wrote none of it, so it was invisible to the only budget there is. MarkRecoveryFailed now counts one, the window's two rules move into recordCrash and rollRestartWindow, and MaybeRestart is rewritten over them with the same constants and comparisons it had. With a failure counted, restore can retry: an instance in StateError with budget left returns to Pending and goes through the existing relaunch, including its mount-not-ready backoff. With the budget spent it stays in StateError for the operator and is only announced to the recovering hook, as before. The window expires, so a later restart tries again rather than writing the instance off. StartStoppedInstance is the operator's remedy and it was refusing to work. The shared stopped store never holds a StateError record, so the load missed, and the fallback treated the presence of the dead local record as proof the instance was already running. It reads the record's state now. StateError is started locally through StartInstance, which already accepts it; terminated answers InvalidInstanceID.NotFound; everything else keeps IncorrectInstanceState, because a claim inserts the instance before it launches, so a local record in a startable state is a start another caller is part-way through. |
||
|
Josh Siviour
|
a74052fd6f |
fix: keep the ecs-agent dialling containerd and brake a deployment that cannot start (#1090)
The agent resolved the container runtime once, in New. A containerd socket that was not accepting during the first second of boot left the agent runtime-less for the life of the process, and nothing retried. It registered and heartbeated normally, so the scheduler saw a healthy instance and kept placing tasks that died immediately at the nil runtime check. The runtime now lives behind a gate that a watch loop keeps current: it dials while the gate is empty, and probes with a container list while it is full, clearing the gate when the probe fails so the next tick re-dials. That covers a socket that is not ready yet at boot and a containerd restarted underneath a running agent alike, and everything that used the puller or the runner reads through the gate, so a runtime arriving late is picked up without a restart. An agent with no runtime advertises no capacity. Registration and the heartbeat report zero CPU and zero memory while the gate is empty, so placement skips the host, and the control plane reads that pair of zeros as an involuntary drain: the instance becomes DRAINING with agentConnected false and zero registered resources, which is visible through DescribeContainerInstances rather than only in the node's log. Capacity returning clears it through the branch a reaped instance already used. A host with no CPU cannot exist, so the zero needs no new wire field. An operator drain is no longer cleared by a bus registration either, which it previously was. A deployment whose tasks keep failing to start now backs off. Deployment gains NextLaunchAt, and past the failure count the circuit breaker already tracks, the reconciler widens the gap between launch passes up to a cap. It is keyed on that counter but applies whether or not the breaker is enabled, because AWS leaves it off by default and an unbraked service relaunches a task that cannot start on every pass, forever. |
||
|
Josh Siviour
|
32bcada227 |
fix: make an error-state instance terminable and stop reporting a refusal as terminated (#1089)
An instance left in StateError could not be terminated through any path, and the gateway reported success anyway, so a Terraform destroy waited on a record the platform had already claimed was gone. classifyRestoredInstances skipped a StateError instance with the comment that the operator must retry or terminate it, but continued before firing OnInstanceRecovering, which is what binds ec2.cmd.<id>. The topic the terminate travels on therefore had no responder and the stated remedy was unreachable. With no responder there, TerminateInstances fell back to the ec2.terminate topic, whose handler reads the stopped KV bucket only and answered InvalidInstanceID.NotFound for a record held in the active registry. The gateway discarded that payload without reading its code and took the idempotent-absence branch, returning terminated for a record it had not removed. The StateError branch now announces the instance to the recovering hook before skipping relaunch, and the gateway reads the refusal: only a genuine absence falls through to the idempotent branch, and any other code is logged and returned. |
||
|
Josh Siviour
|
db5e446338 |
feat: create the volumes a launch names beyond the root device (#1088)
RunInstances created exactly one volume, the root, plus the EFI variable store, and refused a request naming any other EBS device. An instance could not be launched with a data volume attached, so terraform-aws-modules/ec2-instance with an ebs_block_device failed the apply and the workaround was CreateVolume plus AttachVolume afterwards. The launcher was never the constraint: buildDrives already has a data-volume branch and already allocates a hot-plug port for a volume attached at RunInstances time. The control plane now creates a volume for every EBS mapping through the volume service, which is where CreateVolume's validation and metadata write already live, and records each as a non-boot request at the device name the caller named. A data volume is created empty, so a mapping naming a snapshot or naming no size is still refused by name rather than served as something it did not ask for. A launch that fails partway deletes the volumes it created. |
||
|
Josh Siviour
|
2ee22e0f66 |
fix: honour the status filter on ListContainerInstances (#1087)
The filter was never read, so ACTIVE, DRAINING and no filter at all returned every registration in the cluster. A caller could not ask which container instances can take work without describing every one of them, and a cluster carrying a drained node read as if every node were live. The status a record holds is the one DescribeContainerInstances reports, so a filtered list and a describe of what it returns agree. REGISTERING, DEREGISTERING and REGISTRATION_FAILED are accepted and match nothing, because no record here ever holds them; a status outside the enum is refused. |
||
|
Josh Siviour
|
eb365eca7b |
fix: pin predastore to the object sub-resource guard (#1086)
An unserved object sub-resource fell through to PutObject, GetObject or DeleteObject, so a write to one stored the object itself. The pin also carries the meta leader health check fix and the router's single query parse. |
||
|
Ben Duncan
|
2ba8f5e4ab |
docs: separate the install guides from the reference material they had absorbed
Both install guides had grown sections that were not install steps, which is what made the multi-node one hard to follow. Each now ends on a step that hands off to Setting Up Your Cluster, and the reference material moves under Administration where it can be read on its own. - Single-node and multi-node both end on a "Set Up Your Cluster" step rather than a trailing sentence. - The multi-node cluster-state check via `spx admin kv digest` is gone. Verification is now every node Ready, capacity pooled, and the AWS API answering. - "How Multi-Node Storage Works" becomes docs/admin/predastore-distributed-storage, expanded with the host and node model, the write and read paths, degraded writes, hinted handoff, repair, encryption at rest, and what each class of failure costs. - "Firewall and Cluster Membership" becomes docs/admin/host-firewall, expanded with the full public, cluster and encap port groups, how to narrow SSH, and which files under /etc/spinifex/firewall survive an upgrade. - Setting Up Your Cluster gains a closing step that arms the firewall and narrows SSH, so the securing work lands after a cluster that works rather than during the install. install-usb and vpc-networking followed the two moved sections to their new paths. |
||
|
Josh Siviour
|
e035842503 |
fix: report the root volume as it was requested (#1082)
Three omissions left the launched root volume reading back as something the caller never asked for. DescribeInstances returned a null RootDeviceName and RootDeviceType while DescribeImages populated both for the same AMI, so the two answers disagreed about the same image. The launch path now projects both onto the instance. The root volume's EBS request carried no device name, so the launcher wrote an empty string to the volume's metadata and DescribeVolumes reported an attached volume with an empty Attachments[].Device. The name parseVolumeParams already resolved now travels with the request. A volume-scoped TagSpecification was discarded: only instance-scoped tags were read, and the comment claimed volume specs were handled elsewhere, which nothing did. They are now written to the root volume's metadata document. BlockDeviceMappings[].DeviceName still reports the guest path rather than the requested name. That rewrite is deliberate and is left alone here. |
||
|
Josh Siviour
|
0b93b1a957 |
fix: route the EKS update reads so they refuse instead of reading as a typo (#1084)
* fix: route the EKS update reads so they refuse instead of reading as a typo
UpdateClusterVersion was registered at POST /clusters/{name}/update-version, which is the nodegroup pattern and not a path the SDK ever calls; AWS puts it at POST /clusters/{name}/updates. The refusal written for it was therefore unreachable, and the router answered InvalidAction, which tells a caller they typed something that does not exist.
ListUpdates and DescribeUpdate had no route at all and answered the same way. Both now refuse with NotImplemented at the gateway rather than through a service method: the update surface they would read is absent, so a round-trip to reach a constant refusal is surface for its own sake.
The absence of the cluster-level update surface is unchanged. This only stops the platform blaming the caller's spelling for its own routing.
* test: account for the two EKS update reads in the cross-tenant table
Both name their cluster by a path segment that carries no account segment, so
neither can name another tenant's resource and neither needs a case of its own.
|
||
|
Josh Siviour
|
664db686e1 |
fix: ack an ECS stop directive only after the reap reports STOPPED (#1085)
The agent acked a stop in the same loop iteration that launched the reap, so the directive was deleted before the reap had done anything. When the reap could not run — no container runtime, or a failing List — nothing re-posted it, and the task held its ENI, its capacity reservation and its target-group registrations forever. The in-memory `stopping` map made that permanent: it was set at dispatch and never cleared, so the agent would not re-dispatch even if the directive did come back. A stop ledger now separates a reap in flight from a reap that reported STOPPED. Only the second is acked, so a failed reap leaves the directive in the inbox to be redelivered and retried, while the assign loop keeps reading both halves and so still refuses to start a task that is stopping or stopped. |
||
|
Josh Siviour
|
e2db95d746 |
fix: persist security group rule tags and report -1 ports for all-protocol rules (#1083)
* fix: persist security group rule tags and report -1 ports for all-protocol rules A security group rule accepted a security-group-rule TagSpecification and kept none of it, so DescribeSecurityGroupRules answered with no tags and every subsequent plan offered to add them again. SGRule now carries a Tags map, populated on both authorize paths and projected into the central tag store, and DescribeSecurityGroupRules evaluates tag: and tag-key filters against it instead of excluding every rule. CreateTags and DeleteTags on an sgr- id were a silent no-op because updateRecordTags routed only subnet-, vpc-, sg- and eni-. It now finds the rule inside its parent group's record and mutates it there, and a revoked rule's central entry is cleared with it. An all-protocol rule reported the ports it was stored with, 0 and 0, where AWS reports -1 and -1, which is the same never-converging plan by another route. * fix: recognise sgr- as a security-group-rule resource id DescribeTags reported ResourceType "unknown" for a security group rule, so a resource-type filter naming security-group-rule matched none of them, and the authorizer could build no ARN for a rule id and fell back to the any-resource wildcard. Both read the same prefix table, which had no entry for sgr-. |
||
|
Josh Siviour
|
ccc7478f19 |
fix: select the root block device mapping by name and refuse the rest (#1081)
RunInstances read BlockDeviceMappings[0] and nothing else, so a request carrying a root mapping and a data mapping created one volume sized from whichever happened to be first, and the first mapping became the root whatever device it named. A module emitting the data mapping first produced a root volume of the data volume's size and no data volume at all, with no error. The root is now the mapping naming the AMI root device, with position deciding only when nothing names it, so the single /dev/vda mapping ECS and EKS launch with still works. Any further EBS mapping is refused in ValidateRunInstancesInput with InvalidParameterValue naming BlockDeviceMappings and the devices it cannot attach, rather than being discarded silently. Ephemeral mappings carry no volume and are ignored. Creating the additional volumes at launch remains unimplemented; this makes its absence visible. |
||
|
Josh Siviour
|
f536b1a0cb |
chore: pin predastore to the revision that serves bucket versioning (#1080)
The deploy builds with GOWORK=off, so spx links the pin rather than the sibling checkout, and the versioning handlers are absent from every deployed binary until it moves. |
||
|
Ben Duncan
|
deff3a1e1e |
chore: dispatch the E2E Nightly from Cloudflare instead of a GitHub schedule (#1076)
GitHub registers a schedule: only from a repository's default branch, so this cron could drive exactly one branch and always executed main's copy of this file whatever it tested. main was never exercised by the nightly, and a fix to the workflow itself could not be proved until it had already shipped — which is how cell-24's unguarded pkill stayed live on the schedule after its fix merged to dev. A Cloudflare Worker now dispatches this workflow once on dev and once on main. A dispatch takes both the workflow file and the source under test from its ref, so GIT_BRANCH is simply github.ref_name and every schedule-only expression goes with the cron. |
||
|
Ben Duncan
|
2039c05c17 |
fix(ci): take the host benchmark lock before killing predastore nodes (#1065)
cell-24 clears leftover predastore nodes with `pkill -KILL -f '/bin/s3d -config'`, which matches every predastore node on the runner and not only the ones this run left behind. The predastore repository runs the same harnesses on the same machine, so "still running" does not mean "stale" — on 2026-09-16 this step killed a predastore benchmark 32 seconds into its workload, and the run failed with 61 connection-refused errors that look nothing like their cause. Both kills now run under the host benchmark lock the harnesses themselves take, so anything alive while it is held really is a leftover. The second one needs it as much as the first: by the time cleanup runs the harness has released the lock, so another repository's benchmark can already be up. |
||
|
Josh Siviour
|
451df1a8cd | docs: publish AWS operation coverage inventory (#1075) | ||
|
Josh Siviour
|
b26478bd06 |
chore: pin predastore to the bucket sub-resource dispatch fix (#1074)
Every deployed binary links whatever go.mod pins, so the routing fix for bucket sub-resources is absent from the gateway until this moves. Without it a provider-level default_tags block still leaves no bucket creatable by Terraform. |
||
|
Josh Siviour
|
c9460628fb |
fix(ec2): clear a resource's central tags when it is deleted (#1073)
DescribeTags kept answering for resources that no longer exist: the central tag store's DeleteAllTags had one production caller, the instance terminate path, so every other resource type left its entry behind forever. CentralTagWriter becomes CentralTagStore and gains DeleteAllTags, which TagsServiceImpl already implemented, so the daemon wiring is a rename rather than new plumbing. clearRecordTags mirrors projectRecordTags: a nil store is a no-op and a failure is logged rather than returned. The clear runs after the record delete has succeeded, never before, so a delete that fails leaves a live resource with its tags intact. deleteNetworkInterface covers both the public delete and the forced instance teardown, and deleteSecurityGroupInternal covers the default group that only the DeleteVpc cascade ever removes. |
||
|
Josh Siviour
|
e88691e64d |
feat(imds): serve the cluster services domain, not amazonaws.com (#1072)
An AWS SDK's default endpoint resolution builds
{service}.{region}.{services/domain}. IMDS previously answered that
leaf with the literal amazonaws.com, so a workload that forgot its
endpoint override silently reached real AWS instead of failing.
Now that service-endpoint DNS resolves cluster-side, serve the
cluster's configured AWS.ServicesDomain instead, threaded from
config through vpcd.Config into NewIMDSServiceImpl. An empty value
fails closed with a 404 rather than ever falling back to
amazonaws.com. services/partition is unchanged: it still returns
aws, since AWS SDKs carry hardcoded partition tables.
|
||
|
Josh Siviour
|
c674e4b5ce |
fix(ec2): project creation-time tags into the central tag store (#1063)
A tag set at creation time through TagSpecifications was written only onto the resource's own KV record. DescribeTags reads a separate central index at tags/{accountID}/{resourceID}.json, which only CreateTags writes, so a resource born with tags described as having none until something tagged it again.
Terraform finds this immediately: terraform-aws-modules/eks reads a cluster security group's tags back after creating it, and aws_ec2_tag fails against a resource whose tags the tag API cannot see.
The four VPC create paths now project their stored tags into the central store through an optional CentralTagWriter, injected after construction like the external IPAM. A nil writer leaves create behaviour unchanged, and a write failure is logged rather than failing the create — the resource exists either way, and failing a create over a tag index would be worse than the divergence it repairs.
Only vpc, subnet, security group and network interface are covered here. Image, volume, snapshot, route table, internet gateway, egress-only internet gateway, elastic IP, NAT gateway, key pair and launch template have the same split and are follow-up work.
|
||
|
Josh Siviour
|
bef0780b10 |
fix(ecs): make awsvpc ENI slots a placement resource (#1064)
* fix(ecs): make awsvpc ENI slots a placement resource A container instance carries a finite number of awsvpc task ENIs — the hot-plug PCIe root ports the daemon pre-allocates from the instance type's cap — and nothing outside the daemon knew the limit existed. The scheduler's capacity model was CPU, memory and GPU, so every over-capacity task was placed, created an ENI, failed to hot-plug it and rolled it back. A service asking for more tasks than the cluster has slots churned ENI create/delete indefinitely instead of reporting a placement failure once. On a t3.micro the third task and everything after it failed with "attach task ENI eni-… -> i-…: ServerInternal". The instance record now carries the instance type, reported the way AWS carries it, as the ecs.instance-type container-instance attribute; the ENI cap is derived from it rather than stored, so a corrected limit table applies on the next read. An instance whose type has not been reported is not gated at all: a recognised type always has at least one slot, so zero can only mean unknown, and refusing every awsvpc placement against an older agent would turn a mixed-version cluster into an outage. Both placement paths gate on it. A task declined for want of an interface is reported as RESOURCE:ENI, spelled as AWS spells it, and separately from a plain capacity refusal. The caller-visible detail is a fixed sentence; the wrapped error keeps its identifiers and server error codes in the log, where someone can act on them. DescribeInstanceTypes now reports NetworkInfo.MaximumNetworkInterfaces, from the same table the slot allocator enforces, so a caller sizing an autoscaling group has something to size against. Fields this stack does not implement are left unset rather than guessed. Finally, the daemon records the slot it actually allocated as the ENI's device index. The requested index was stored verbatim, so every task ENI on an instance reported index 1 while landing on a different PCIe slot; on AWS a device index is unique per instance. * fix(ecs): send an ENI capacity refusal as the reason alone AWS returns RESOURCE:* placement failures with no detail. Carrying a sentence there gives a caller a second string to parse that says nothing the reason does not already say. |
||
|
Josh Siviour
|
86014ac08a |
fix(ec2): serialise tag writes across the cluster (#1070)
CreateTags is a read-modify-write of one resource's whole tag map. It was serialised by a sync.Mutex on the service struct, which exists once per process, while the handler answers on the spinifex-workers queue group and so runs on whichever node NATS picks. Two nodes read the same map, each added its own key, and the later write won. Every call returned success. Six parallel writers against one security group on a three-node cluster left four tags. On a single node the mutex hides it entirely, which is why no test caught it. The tag index moves from one S3 object per resource to a JetStream KV bucket, where the merge runs inside kvutil.Update and commits at the revision it read. A writer that loses the race re-reads the winner's keys and adds its own on top. S3 could not host the fix: predastore's PutObject has no If-Match, so there is nothing to compare and swap against. Tags written before the move are not migrated in a batch. A batch that half-failed would leave a resource looking untagged, and the next write would commit that as the truth. Instead the CAS seeds a missing entry from the old object inside the attempt, so a resource migrates on its first write and a failure to read the old object fails that one call. DescribeTags reads the union of both key sets, preferring the bucket, and DeleteAllTags clears both so old tags cannot reappear through the fallback. The process-local mutex is removed rather than left beside the CAS. The regression test drives six separate service values over one bucket, because that is what makes it able to fail: with the old code every caller inside one process was already serialised, and only a second node could lose an update. |
||
|
Josh Siviour
|
fd22c518bf |
fix: bound the KV bucket listing so a stuck DNS reconcile cannot wedge the cluster (#1071)
kvutil.BucketNames ranged the KeyValueStoreNames lister channel using whatever context the caller supplied, and the EKS desired-set builder supplies one that lives as long as the daemon. A lost stream-names reply therefore left the channel open with nothing ever sent on it and parked the caller for the life of the process. kvutil.Keys, immediately below it, already bounds exactly this case and says why; BucketNames never got the same treatment. The DNS reconcile made that hang cluster-wide. A pass takes the reconcile lease before computing its batch and releases it on a deferred call, and the lease renews on a background timer for as long as the holder lives, so a pass that never returns holds it forever and the other nodes never take over. Observed on a three-node cluster: the zone object stopped being written after a rolling deploy, and every name published afterwards stayed NXDOMAIN until the holder was restarted. The listing now carries its own deadline and reports a timeout as an error, never as a short list, because callers prune on absence. The pass bounds the batch computation as well, so any other deadline-free call reached from the desired-set builders costs a cycle rather than the cluster. Only the read side is bounded: it has no side effects, so abandoning it is safe, while the apply stays inline under the lease. |
||
|
Josh Siviour
|
4a9dda0b99 |
fix(ecs): honour ListTasks filters and count instance tasks by status (#1062)
ListTasks read only the cluster and returned every task record under it, discarding DesiredStatus, Family, ServiceName, StartedBy and ContainerInstance. On a cluster with one running task and a hundred stopped ones, both status filters returned the same hundred-and-one ARNs. A taskFilter now carries the input's matchable fields and is applied per record. An omitted desiredStatus means RUNNING, as it does on AWS, so an unfiltered call no longer hands back everything the cluster has ever run. recordTaskState moves a self-exited task's desired status to STOPPED alongside its last status. Without that the filter would be honoured and still wrong: a task that died on its own would answer the RUNNING filter for as long as its record survived. Container-instance counts came from len(PlacedTasks), which is the set of tasks still owing capacity rather than a status tally, so a PENDING task counted as running and a task whose release did not complete counted as running forever. PendingTasksCount was never set at all. Both are now tallied from the task records, the way the cluster-level counts already were. DeregisterContainerInstance passes zero deliberately — the instance is gone, and counting there would report the tasks it just force-stopped. launchType is not filtered because TaskRecord has no launch-type field to match on, and maxResults/nextToken are left alone because no List call in this package paginates; adding a token scheme to one would leave the other four inconsistent. |
||
|
Josh Siviour
|
c10b6732d5 |
feat: publish AWS service endpoint names in DNS and match gateway cert SANs (#1068)
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs
ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.
The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.
admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.
Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.
* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain
ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.
The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.
Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.
* chore: pin northstar to the merged set-valued write path
|
||
|
Tom Newton
|
c4b380b2ed | chore: update cisco docs | ||
|
Josh Siviour
|
ef9bfb4c1d |
fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host (#1066)
* fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host DescribeRegions hardcoded https://localhost:9999 as the Region's endpoint regardless of where the gateway actually runs, so a workload doing endpoint discovery through this API was pointed at its own loopback instead of a reachable address. The endpoint is now resolved by the caller from GatewayConfig.RegistryHost/RegistryPort, the same advertised host:port ECR already uses (concrete AWSGW bind host, else AdvertiseIP), and passed into DescribeRegions rather than invented there. Falls back to localhost:9999 only when no concrete host is configured, which is also the only case where that value is actually correct. Closes mulga-0bqih. * style: trim DescribeRegions endpoint comments |
||
|
Josh Siviour
|
969163c114 |
feat(imds): serve block-device-mapping from instance block device data (#1067)
* feat(imds): serve block-device-mapping from instance block device data The aws-ebs-csi-driver's IMDS client fails outright on this 404, aborting IMDSInstanceInfo before it reaches the Kubernetes fallback the driver otherwise uses. Serve ami, root, and one ebsN per attached EBS volume from the instance's existing BlockDeviceMappings, root device first, matching how the driver counts "ebs" occurrences in the listing. * style: trim block-device-mapping comments |
||
|
Julian Sommer
|
ff41bcf369 | fix: read only the named documents on explicit-ID describes (#1069) | ||
|
Julian Sommer
|
5991b92b06 | chore: concurrent preflight |