mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-21 11:46:37 +00:00
f67a326b9750e6d72fd2bd831714d2e1ddaf5002
197 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
dd9c9666f2 | release: update go.mod for v1.21.0 | ||
|
Julian Sommer
|
94488dc399 |
fix(gateway): don't count body-read failures toward the auth lockout
A request whose body cannot be read to hash failed through the catch-all branch, logging a malformed envelope, returning IncompleteSignature and recording an anonymous lockout strike, so ten dropped connections locked out an IP. Match sigv4.ErrReadingBody and answer InternalError without recording a failure. Bumps bluebottle for the new sentinel. |
||
|
dependabot[bot]
|
b7b1ae562a |
chore: bump the go-dependencies group with 2 updates (#1112)
Bumps the go-dependencies group with 2 updates: [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) and [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go). Updates `github.com/nats-io/nats-server/v2` from 2.14.7 to 2.15.0 - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/RELEASES.md) - [Commits](https://github.com/nats-io/nats-server/compare/v2.14.7...v2.15.0) Updates `github.com/nats-io/nats.go` from 1.53.1 to 1.54.0 - [Release notes](https://github.com/nats-io/nats.go/releases) - [Commits](https://github.com/nats-io/nats.go/compare/v1.53.1...v1.54.0) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-version: 2.15.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/nats-io/nats.go dependency-version: 1.54.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Ben Duncan
|
20b3eae728 |
fix: record the OCI SDK's own dependencies in go.sum (#1115)
spinifex/cloud/oci imports oci-go-sdk directly, but go.mod listed it as indirect and go.sum was missing three of the modules it needs — sony/gobreaker/v2, gofrs/flock and youmark/pkcs8. The module therefore only type-checked inside the go.work workspace, whose go.work.sum happened to carry them, so a standalone checkout failed to build and the linter reported it as a typecheck error against an unrelated file. |
||
|
Ben Duncan
|
7163f7d62d |
feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108)
* feat: ship uninstall-spx.sh and fix the OCI bring-up path
Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the
software where node-reset.sh clears state and leaves the node installed. An
operator who installed with `curl ... | bash` had no teardown path at all.
State teardown is delegated to node-reset.sh rather than reimplemented, and
setup.sh now ships both to /usr/local/share/spinifex so they are reachable
without a checkout.
It refuses on a multi-node cluster and --yes does not override that: a flag
meaning "don't ask me" must not also mean "ignore a safety property".
/var/lib/spinifex and every apt package are kept unless asked for, and both
opt-ins confirm separately even under --yes.
Proved by a full uninstall/reinstall round trip on dev-oci, which found two
bugs neither prod nor dev-prod can reach, because both are ISO-installed:
node-reset.sh exited 2, silently, at its last step and after the wipe.
setup.sh creates the spinifex *group* and the spinifex-* service users but
never a *user* named spinifex, so `getent passwd spinifex` exits 2 and
pipefail carried it into set -e.
The uninstaller broke openvswitch-switch on its first run by removing the
socket-perms helper while leaving the drop-in referencing it, failing the
unit at step EXEC with status 203. Fixed on both sides: the uninstaller
removes drop-ins it wrote into other units' namespaces and restarts anything
it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can
never stop another package's daemon.
setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux
bridge it links it to OVS with a veth pair instead of enslaving the NIC, so
the VNIC keeps its MAC identity and netplan keeps owning the addresses. It
now fails on OVS datapath errors instead of reporting success, and the
--management demotion guard keys on /etc/default/ovn-central rather than on
database presence, which the ovn-central package creates on every node.
Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts
along with shellcheck over the four scripts that ship to every node.
* feat: allocate external IPs from the Oracle Cloud API
An OCI VNIC drops any packet whose source is not a registered private IP
object on it, so an external pool cannot be range math there — the address
has to be created through the provider before it exists at all. Add
source = "oci" and the allocator behind it.
One allocation is two OCI objects: a secondary private IP that rides the
wire, and a RESERVED public IP attached to it that the internet reaches.
Allocate returns the public address because that is what AWS semantics are
about, and records the pair so the host plumbing and release path can find
the private half.
The OCI objects are created before the binding is written, and dropped
after it on release — both orders chosen so a crash leaves a discoverable
leak rather than a record that lies. Reconcile squares the two on startup
and is required, not optional: a leaked reserved public IP bills and holds
one of the 50 per-region slots with nothing referencing it. It never
touches an object without our display-name prefix, because the reference
host carries operator-created secondaries on the same VNIC.
Resolve the VNIC from an interface name by MAC through IMDS rather than by
Oracle's VNIC ordering, which is not a documented contract.
Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring
is not a second single-purpose installer.
* fix: refuse source = "oci" outside routed mode, and cover the SDK path
An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound
frame to it regardless of which of its addresses the packet is for.
Measured on the reference host: an internet source reaches 10.200.0.144 as
router-MAC to VNIC-MAC, the public address never appears on the wire at
all, and every off-subnet destination resolves to the same next hop.
Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on
the uplink and ARPs for the address, which policy.go:50 justifies with "the
Linux bridge learns the per-rule external MAC on the veth port like any
other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode
guest's egress is dropped and its ingress never reaches OVN. Routed mode
keeps the address on the host, which already wears the VNIC's MAC.
Pinning every LRP to that MAC does not rescue pool mode: they share one
external logical switch and OVN resolves inbound frames by an L2 lookup on
it, so duplicate MACs work for one VPC and silently misroute past that.
The refusal names the MAC reason, because a refusal that only says "use nat
mode" reads as arbitrary and gets worked around.
Also test the SDK request shapes against a local endpoint. Two of them are
wrong in ways reading does not catch: pagination stopping early reports a
partial VNIC to reconcile, which then drops live bindings, and a nil
privateIpId on detach is omitted from the body entirely, so OCI reads the
update as a no-op, returns 200, and leaves the address attached and billing.
* fix: install host firewall rules where a distro's catch-all reject cannot precede them
Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD.
Two Spinifex datapaths were silently dead behind it on any such host.
EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules
landed after the REJECT and never matched: 24113 packets on the reject, 0 on
both accepts, and guests had no egress. The comment named the case it did not
handle, anticipating a FORWARD policy of DROP — which an append does survive —
rather than an explicit REJECT rule, which it cannot. Insert at the head, and
drain existing copies first, because a rule an earlier build appended still
satisfies -C and would otherwise stay stranded across an upgrade.
Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254
and .253 per instance on the ime- devices, so guest requests are locally
delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init
spent 490s timing out and reported the endpoint as disabled. With one wildcarded
accept the same image boots in 18.6s and finds its datasource.
The natuplink e2e check had the same blind spot the code did, asserting -C and
therefore presence but never position. It now asserts the accepts precede the
first reject.
* feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata
An OCI external address is two objects, not one: a RESERVED public IP that
OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP
on the VNIC that the wire actually carries. Every AWS-facing record holds the
public half; every datapath object must hold the private one.
The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP,
PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile
rebuilds the rule set from AWS-facing intent on every pass — so a correctly
mapped rule installed further upstream is swept on the next reconcile and
replaced with one matching an address that never arrives. The seam is a
DatapathResolver defaulting to identity, so every non-OCI deployment is
unchanged. A resolve failure is fatal: falling back to the public address
would install a rule that reads correctly in ovn-nbctl and blackholes.
ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no
credentials, because vpcd is a different process from the daemon that owns
the allocator and the bindings record is all either would consult.
Two live bugs the in-memory test store was hiding, both found by making the
fake report ErrNotFound the way the real store does:
- KVStore.Mutate could not create, so every pool failed its own first
allocation after OCI had already created and billed the pair.
- Reconcile treated a missing record as a failure, aborting the one pass
that exists to find an object created before its binding was written.
Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's
own metadata service lives — the first guest launch made it a local address
and took the host's metadata and DNS with it. The datapath is OpenFlow
steered, so the guest-facing addresses stay put and only the host-side
endpoint address moves, with a PREROUTING DNAT on ime-+ translating and
conntrack un-NATing the reply. Off by default; identity when unset.
The IMDS firewall rules move to vpcd. They are wildcarded over every
endpoint rather than per-launch work, and spinifex-daemon holds no iptables
grant and no CAP_NET_ADMIN by design, so attempting them aborted every
launch.
* fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package
The error string pointed at DefaultHostBindAddrs, which does not exist — a
caller following it would have looked for a symbol that was never written.
nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries
the //test:in-package marker the package check asks for.
* docs: index what each e2e suite proves
Written because a datapath comment was read as an end-to-end behavioural claim
and the suite that already answered it was found only after the wrong answer
had been given. TestVPCEgressPaths/NATBaseline is the authority on private
subnet egress, and nothing pointed at it.
* fix: source-route an EIP's reply out the VNIC that holds it
An EIP's reply carries the EIP as its source. On a host that pins its
uplink with a policy rule, the reply took the main table instead and left
by the wrong interface — which a cloud VNIC drops, because the address is
registered on a different one. Ingress reached the guest and nothing came
back.
Every bind now mirrors the uplink's rule onto the EIP. The selector is the
rule rather than the interface: both OCI VNICs sit in one subnet, so "the
interface owning the EIP's subnet" names two of them, and two candidate
rules is a refusal rather than a guess. A host with no source routing gets
no rule and no change.
* docs: name the setup-ovn recipe routed mode actually needs on OCI
The OCI guide told operators to run `--wan-bridge=br-wan`, which is
mutually exclusive with `--nat-uplink` and deletes the spx-nat transit
veth routed mode runs on. The node then refuses to bring the uplink up,
with an error naming the flag the guide should have used.
Several messages also claimed routed mode is outbound-only with no
public IPs. That stopped being true when the routed EIP path landed, and
it reads as "what you are about to do cannot work" during the one init
the OCI path documents.
* feat: let routed NAT form a multi-node cluster, and carry every pool through formation
Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no
technical justification -- it was a v1 scope decision -- but routed NAT is the
only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become
"Spinifex on a cloud host is single-node".
It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms,
all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but
every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only
the gateway chassis reaches its own guests. init now says so on stderr rather
than refusing, because the refusal made the defects unreachable and therefore
unfixable. Distributing the EIP datapath is the actual fix.
Lifting it exposed a second defect that stops formation outright. NetworkConfig
held nine flattened Pool* scalars and the init node set PoolName = "wan"
unconditionally; routed NAT renders two pools and the one that matters --
nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a
single pool named "wan" holding the transit gateway and the detected WAN prefix,
and the leader re-rendered its own config from the same struct, so all three
nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData
gains json tags because it is on the wire now rather than a template input.
* feat(install-node): add --external-mode so a cluster can be formed in routed NAT
install-node.sh could not express routed NAT at all. It required
--external-pool/--external-gateway/--external-prefix-len and passed
--external-mode=pool to every spx admin init unconditionally, so the only
supported way to form a routed-NAT cluster was by hand.
--external-mode nat refuses the three pool flags rather than ignoring them, and
threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check
rejects a physical NIC during formation, which is correct in pool mode and
wrong in routed mode, where nothing is bridged.
* fix(ocinet): stop each node deleting its peers' OCI addresses on startup
Reconcile compares the binding record against the private IPs on one VNIC. The
record is one key per pool and therefore cluster-wide, so on more than one node
every peer's binding looks exactly like a binding whose objects OCI has lost.
Each node dropped them, and direction 1 on the owning node then collected the
now-unclaimed objects as leaks -- deleting the reserved public IP and private IP
of a running instance.
Observed on the three-node OCI cluster: node2's startup pass logged "collected a
leaked address" for an address it had allocated ten minutes earlier and "dropped
a stale binding" for one living on node3's VNIC.
The damage did not stop at the OCI objects. With the binding gone,
Lookup.DatapathIP returns the public address unchanged, so the OVN
dnat_and_snat row carries the public address instead of the private half. Guest
egress then leaves the VNIC sourced from a reserved public IP, which OCI drops
because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute
no-ops too, since a public address shares no subnet with any host address. One
destroyed binding takes out both directions.
Binding.VNICID already exists for exactly this ("how a node that comes back from
the dead tells mine from moved on without me") and Reconcile never consulted it.
A binding is now only dropped when it names this node's VNIC; one naming no VNIC
cannot be attributed and is left alone rather than guessed at.
* fix(host): find the EIP's source-routing table when the uplink is held as a /32
uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding
a host address configured with a prefix that contains it. On OCI both VNICs sit
in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts
the subnet's on-link route in the main table at metric 0 and takes the whole
subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was
ever found, no source rule was installed, and the reply left by the primary
VNIC where OCI dropped it for carrying an unregistered source.
The address is not the only record of who owns the subnet. The rule names a
table, and that table holds the on-link route. Consulted as a fallback, only
when the address prefixes matched nobody, so a host that already resolved this
the direct way keeps its existing answer -- prod and dev-prod are unaffected.
Only scope-link routes count. A default route covers every address and would
make every source-routing table an equally good answer.
* docs(oci): correct the multi-node story and name the oci_vnic_id constraint
The nat-mode warning said EIPs and default egress were both broken on more
than one node. Default egress is fine: delete a guest's dnat_and_snat row and
it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the
public-IP path was broken, and the three defects behind it are now fixed.
What remains is not a bug but the shape of the datapath: routed NAT centralises
the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other
nodes cannot reach it, and OCI drops any frame whose source is not registered on
the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway
chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator
writing the pool will read it, and in §9, where the cost is that a gateway
change kills every public IP until the pools are repointed.
The single-node cap itself is gone from the doc: a three-node cluster has been
formed and measured, and the warning proceeds rather than refusing.
* feat(nat): distribute the routed-NAT EIP path so it works on more than one node
Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is
a localnet, so the transit segment a /32 route pointed into existed only on
that one node. Measured on three nodes: all three claimed all three EIPs, the
nodes disagreed on the gateway LRP, and only guests on the gateway chassis were
reachable.
The mechanism to fix it is not new. Pool mode already distributes EIPs by
putting external_mac and logical_port on the dnat_and_snat row and letting
OVN's ARP responder answer for the external IP on the external segment. Routed
mode has an external segment too, the spx-nat veth pair, and it being per node
is the distribution: only the node running the instance can answer ARP on it.
So a routed rule is now distributed on the same terms as a pool one, and its
host route becomes on-link on the transit veth instead of via a gateway LRP
that may be on another chassis. bindHostEIP stops demanding that address for a
distributed rule, which is what previously failed the bind on every node but
one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS
rather than the Southbound DB because the question is about this host.
The gateway LRP keeps its transit-pool address. Default egress routes through
it and works across three nodes today, so making it link-local as pool mode
does would break a working path to fix one distribution has already fixed.
Pool and static are unmoved, and a test asserts that against the same input
rather than leaving it to inspection. A rule with no MAC stays centralised, so
a cluster part-way through an upgrade is never in a state neither shape handles.
Beads: mulga-opyl7, mulga-la0pe
* fix(nat): make routed-EIP host plumbing work on every node
Three guests on three OCI nodes, one guest per node: only one was
reachable. Two defects, both invisible on a single node.
The transit veth took a random MAC per node, but OVN holds one MAC
binding for the transit nexthop across the whole cluster — so every
chassis addressed its egress to whichever node seeded the binding, and
the other kernels dropped it at L2. The pair is node-local, so one
locally-administered address everywhere is unambiguous; setup-ovn.sh
pins it at creation and in the netdev unit, and EnsureUplinkPort repairs
a veth made before that.
Host routes and proxy-ARP also rode the leader-gated reconcile pass, so
a node that never won the lease never plumbed its own guests' EIPs.
BindHostEIPs does that half on its own loop on every node; Bind already
leaves alone an EIP whose ENI is elsewhere, so two nodes never contend.
An unresolvable address skips the sweep rather than shrinking it.
Pool and static are untouched: the new pass is a no-op outside routed
mode, and a test asserts it.
Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on
a node that is already tangled. It deliberately spares the NB/SB
databases, /etc/spinifex, the system-id and the packages, and refuses to
del-br a Linux bridge.
* fix(nat): assert the transit MAC where vpcd actually runs it
EnsureUplinkPort is never called in production — host.Wiring is
constructed nowhere — so the repair landed in a path the deployed
daemon does not take, and three nodes came back with three different
MACs. vpcd re-asserts it beside the NAT egress rules instead, which is
the routed-NAT branch that does run on every start.
* fix(vpcd): make uplink-mode detection say what it saw, and check it
detectBridgeMode logged only the branch it took, so a node detected as
veth when it was wired for nat looked identical to one that was right.
It now reports both probes every time, errors when a node carries both
uplink veths (set up twice, and --teardown is the way out), and errors
when external_mode and the wiring disagree — the datapath follows the
wiring, so the config silently loses that argument. An explicitly
configured bridge_mode gets the same cross-check, since nothing else
ever compared it against the wiring.
The vpc-networking doc now opens nat with "start with pool unless one
of these applies" and gains a section on nat across more than one node;
the admin warning for --external-mode=nat on two or more nodes says
what was verified by hand and what is still uncovered.
* test(nat): prove routed NAT delivers each node's own guests, on a cluster
natuplink ran single-node only, which is exactly where the two defects
that broke three nodes are invisible. It now admits ModeMultinode, and
phase 9 takes a second shape there: a guest per node in a spread
placement group, and for each one — its public IP plumbed by the node
running it and by no other node, the route on-link with proxy-ARP
answered there, a dnat_and_snat row carrying external_mac and
logical_port, a TCP handshake in from another node, and the guest's own
console verdict for egress. Every node's transit veth is checked
against the one cluster-wide MAC before any guest boots.
Single-node phase 9 gains the same row and on-link assertions, so the
shape is held on both topologies.
Two gates come from what three OCI nodes actually do: every node holds
a gateway_chassis row for every VPC (the active one is the highest
priority) and every node installs the VPC ingress route, so only phase
7 needs gating — it runs where the gateway port and the guest are both
local, and logs which one is missing otherwise.
Cell 20 is real-multi x nat with its own nightly step, leaving the
permutation cells' suite set alone.
* test(nat): run the VPC egress paths in nat mode, not pool mode only
The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on
external_mode = "pool", with a comment saying nat clusters have no
EIP/NAT-GW support. They have had EIPs since the routed path landed, so
the gate now asks whether the cluster can allocate public addresses at
all — pool, or nat with a non-transit pool configured.
The nat-single cell runs TestVPCEgressPaths beside natuplink, which is
the only nightly coverage of a private subnet's isolation and its NAT
gateway in routed mode. natuplink never creates a private subnet.
* docs(config): list nat among external_mode values
* fix(ci): renumber the multi-node nat cell off the bare-metal job
Cell 20 is already the bare-metal single-node job, gated on the same
cells input but declared in the workflow rather than e2e-cells.json.
A matrix entry with the same number runs two unrelated jobs under one
name, and naming 20 in a dispatch selects both.
* feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests
Oracle publishes its qcow2 checksums inline on an HTML page and ships no
sums file, so the four new entries carry the expected digest in the catalog
rather than a URL to fetch one from. Images gains ChecksumDigest for that,
and the catalog import path verifies against it and says which of the two
sources it used, so a recorded digest never reads as a fetched sums file.
Pinning is safe here and only here: each URL names an immutable build
(b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build
alongside the plain -kvm- one; the cloud build is the cloud-init image and
is the one that matches x86_64's -kvm-.
TestImageCatalogReachable HEADs every URL in the catalog from the general
single-node suite. These artifacts live on hosts nobody here controls, so
an upstream that retires a build breaks `spx admin images import` for a
customer and nothing else in the suite would notice. It also asserts that
every entry sets exactly one checksum source, and that no pinned entry
points at a "latest" URL whose bytes would change under it.
* fix(formation): carry bridge_mode onto the leader and every joiner
applyNetworkConfig copied external_mode, external_interface and the pools
out of the formation NetworkConfig but never bridge_mode, so a multi-node
cluster wrote a spinifex.toml without the key while single-node init wrote
it. Every node then fell back to autodetecting a mode that init had pinned.
Nightly cell 30 caught it: TestNATUplink failed at the config phase on a
spinifex.toml that did not contain bridge_mode = "nat". Deriving it from
the mode rather than adding a NetworkConfig field keeps the leader and the
joiners on the same rule single-node init already uses.
* feat(ui): show how to ssh into an instance, and which user to use
An instance detail page gave the addresses and the key pair name but never
the login user, which is the one part of the command nobody can derive. It
belongs to the AMI, not to Spinifex: nothing in the launch path creates or
renames an account, so stock cloud-init applies the distro's own
default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is
ubuntu, and guessing from the distro name gets two of those three wrong.
The card resolves the user from the AMI's name and description, and an
unrecognised image yields a placeholder rather than a guess, because a
wrong username in a copy-paste command is worse than an honest gap. A
private-only instance is called out as reachable from inside the VPC only,
rather than presented as a command that will simply time out.
* fix(nat): plumb host state for a NAT gateway's public address
AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT
node the host is what delivers a public address, so an egressing private guest
left with a source the host held no route back to, and the masquerade rule —
which matches the transit 100.127.0.0/24 alone — never saw the packet. The
gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD
accepts an EIP gets.
hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway
address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs
deliberately: pruneOrphanEIPs and addLive read that set as the complete list of
addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so
widening it would stop the sweep reclaiming a stale row on an address later
reused as a gateway's.
AddNATGateway also binds directly, on both the create and idempotent-skip
paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP
pass — and so a reboot that keeps the OVN row still replays the volatile half.
Documents NAT gateways in nat mode in vpc-networking, and says at the head of
that section that nat is a supported mode covered by nightly cells 19 and 30.
* docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user
Four catalog entries, all UEFI. The login user is the one that catches
people: opc is the default on the images Oracle publishes into OCI itself,
while these are the generic KVM cloud images from yum.oracle.com, whose
cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle
and ec2-user are all refused.
Also records why these four pin a checksum digest inline (Oracle ships no sums
file, and each URL names an immutable build), why arm64 takes the -kvm-cloud-
build, and that SELinux ships enforcing.
* fix(reconcile): stop reading an unreadable intent as an empty one
A wedged JetStream KV returned no keys for the ENI bucket while writes were
still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts
read every guest LSP as an orphan against it, and every guest on the node lost
DHCP and IMDS while the API kept answering and the instances stayed "running".
The existing re-read guard did not help: it re-reads through the same path.
Three defences, because no single one covers every shape of the fault.
A failed kv.Get now fails the pass instead of dropping the record. Dropping one
VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads
as permission to delete every guest port in that VPC — one failed read, same
outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that
is a poison record rather than an unhealthy store, and failing on it would
wedge reconcile forever.
The port sweep now asks local OVS which taps are plugged in here and never
deletes a port that has one. A running guest's port cannot be an orphan
whatever intent says, and OVS is the one liveness signal that does not come
through the KV read path the sweep is otherwise entirely driven by. Wired from
vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the
sweep rather than deciding on intent alone.
Each sweep also refuses outright when the keep set it would delete against
holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That
is the case local taps cannot cover, because the drift sweep is leader-gated
and lists NB cluster-wide, so the leader holds no tap for another node's guest.
The cost is real: a genuinely orphaned row now survives until one live record
of its class exists. Leaking a stale row is the recoverable half.
Three existing prune tests drove their fixtures to a wholly empty intent, which
is now the refusal case; they keep one live record of the class so they still
prove the prune. The Get-failure path has no unit test — there is no
jetstream.JetStream fake in the tree and an embedded server cannot be made to
list a key it will then refuse.
* docs(oci): separate the two address quotas, and the principal that can read them
The regional reserved-public-IP ceiling is tenancy-wide and does not grow with
node count, so a cluster hits it before the per-VNIC 64, which does. Records
the commands that read both, and that a detached reserved public IP still
bills.
Reading limits or usage needs a tenancy-level grant, which cannot be
compartment-scoped the way the node's policy is. That is a second, read-only
principal kept off every node, not an addition to SpinifexOperators.
* docs(oci): give quotas their own troubleshooting section, and say we enforce none
Spinifex hardcodes no address ceiling: it attempts the allocation and maps
OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with
Oracle and nothing here is reconfigured or restarted — which is what an
operator needs to know before it bites, not after.
Separates the two quotas by how they scale: the regional reserved-public-IP
default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and
is not.
* fix(oci): follow a guest to its new node when its EIP moves
OCI delivers a public address to whichever VNIC holds the private IP behind
it, and nothing ever moved that private IP. Binding.VNICID was written once at
allocate time, as the VNIC of whichever queue-group daemon happened to serve
the AWS request, and never updated — so an address landed on the right node
only by chance, and a guest that stopped and started elsewhere went dark while
every OVN row, host route and firewall rule on the new node was correct.
ClaimLocalAddresses closes the gap from the node that should hold the address.
Local OVS decides ownership, the same authority the host EIP binder already
trusts for the same question, and UpdatePrivateIp with a new VnicId does the
move in one server-side call. The private IP OCID survives it, so the reserved
public IP attached to it follows without being touched and the customer's
address never changes.
A loop per node, not leader-gated: the question is about this host's guests,
and a node that never wins the reconcile lease would otherwise never claim the
addresses of the instances it is running.
* fix(oci): record which ENI an EIP is associated with
An EIP is allocated bare and attached by a later AssociateAddress, so the OCI
binding written at allocate time named no ENI and nothing ever filled it in.
The address then stayed on whichever node happened to serve the allocation:
the affinity pass could follow an auto-assigned address, which is allocated
with its ENI, but never an EIP — the case customers actually use.
BindOwner is on an optional OwnerTracker interface rather than Allocator,
because only OCI has a notion of an address being delivered to one node. A
static or DHCP address is on the wire everywhere its node can reach, those
allocators do not implement it, and ExternalIPAM treats a pool that does not
as a success with nothing to do.
The same gap made the release owner guard inert for EIPs: it compares against
a binding ENI that was always empty, so a stale teardown could free an address
already handed to another instance.
* fix(ec2): report the address an instance is actually reachable on
Three faults kept DescribeInstances answering with the address launch assigned,
whatever happened afterwards.
The launch-time stamp lives on the stored ec2.Instance, and the stopped path
only declined to add an address rather than removing one, so a stopped instance
went on advertising an address it had released and that may already belong to
someone else. The projection now clears it, and the NIC association with it,
working on its copy so a restart can still report the address.
The running path filled the public address only when it was nil, which the
stamp never is, so an EIP could not be reported however correctly it was
recorded. The runtime address now wins.
And nothing told the rest of the system about an association: AssociateAddress
wrote neither the ENI record, which is what IMDS and the network reconcile
read, nor the instance record, which is what a describe answers from. The ENI
write is direct; the instance write is a fan-out, because the queue-group
worker that serves AssociateAddress is rarely the node running the instance.
* fix(net): take the host EIP plumbing off a node its guest has left
Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept
anything still wanted cluster-wide. A guest that moved therefore left the old
node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with
nothing that would ever remove them. On a shared segment that node goes on
answering ARP for an address another node is now serving.
The two halves now ask the same question by the same authority, in the loop
that already runs per node. An unreadable local OVS is not a yes: tearing down
a live address on a guess would fire everywhere at once, so it keeps the
binding and the next pass tries again. A binder with no chassis test behaves
exactly as before, and a centralised EIP is exempt because it hairpins through
the gateway chassis and every node plumbs it legitimately.
* fix(net): refuse to free an Elastic IP on behalf of a departing interface
Releasing an instance's public address on teardown releases it by address,
scoped to the ENI holding it. Since an association now writes the EIP onto the
instance record, that address is the EIP — so terminating an instance gave a
customer's Elastic IP back to the pool while describe-addresses still reported
it as theirs, free to be handed to another guest.
The existing ownership guard cannot catch this. It only fires when the stored
allocation already names an ENI, and an EIP is allocated bare and associated
later, so the stored ENI is empty and the guard passes straight through.
The distinction that holds is who is asking. ReleaseAddress is the address's own
owner and names no interface; instance teardown names the interface it is tearing
down. So a release that names an owner may free an auto-assigned address and
never an EIP, which is enforced in each of the three allocators.
DHCP needed the scope carried on the wire, since it releases by address and
resolves the lease afterwards. Absent on an older caller, which reads as false
and keeps the previous behaviour.
Worst on the two providers where it is not recoverable: on DHCP the lease is the
only thing holding the address and the server promises nothing about handing it
back, and on OCI the address is a reserved public IP object that the release
destroys outright.
* test(e2e): prove an Elastic IP survives a stop and start
Instance teardown releases the public address by value, and since the
association writes the EIP onto the instance record, that release aims
straight at the customer's Elastic IP. The allocator guards now refuse it,
but nothing end to end proved the address comes back on a restart.
The step goes inside EIPFlip, which already runs last on the shared public
guest for the same reason a stop is safe there: the address the guest
returns on is the allocation's, not the pool's. It runs on the static
cells and the DHCP ones alike, and on DHCP it doubles as the only outside
view of the lease still renewing while the guest is down.
* feat: release an auto-assigned public address on stop, take a new one on start
AWS lends an instance a public address for as long as it runs and gives it a
different one on the next start. Spinifex kept the address across a stop, which
holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a
lease nothing is using.
A stop now offers the address back through the instance cleaner, which refuses
anything an Elastic IP is behind — the EIP association record answers that, the
same question an ENI delete already asks — and records on the VM that the start
owes it a new one. A host drain keeps the address: Restore relaunches the guest
on the next boot, and changing its address under it is an outage nobody asked
for.
The start allocates before the launch, so an exhausted pool refuses the start
rather than booting a guest the customer cannot reach, adopts an Elastic IP
associated while the instance was stopped instead of assigning a second address
beside it, and hands the address back if the launch then fails.
e2e: a StopStartAddress stage on the public guest asserts a stopped instance
reports no address and comes back on a different one from a static pool, and a
StopStartKeepsTheEIP step asserts the opposite for an Elastic IP.
* docs: restructure the OCI guide around two deployment paths, with diagrams
The guide had grown into a numbered wall with no shape: no frontmatter, no
contents, and a reader with an OCI account had no way to tell where to start.
It now follows the multi-node install template — overview, prerequisites, then
the deployment path — and splits that path in two: Terraform for one node or
three, which is the preferred route, and a manual single-node walkthrough for
an instance that already exists. Both meet at one shared configuration section
rather than repeating it.
Two ASCII diagrams carry what the prose could not: one OCI instance with the
whole AWS surface inside it, and three nodes with the same surface distributed,
showing which plane each VNIC carries and how an address follows a guest that
moves.
The limits section was materially out of date and said so in the places that
mattered most. Multi-node is no longer experimental, public addresses are no
longer pinned to one node — each allocates on its own VNIC and the affinity
pass moves the private half to wherever the guest is — and an auto-assigned
address is now returned to OCI on stop, so a stopped fleet stops paying for
addresses. What changed is recorded rather than quietly edited.
The Terraform configuration moves in from the monorepo as
scripts/terraform/oci-spx, so the guide points at something in this repository.
Also marks the two new lifecycle test files in-package, with the reason each
one needs to be: both substitute unexported dependencies.
* docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan
Replaces the two ASCII diagrams with SVGs in the house style, one for a
single node and one for three. Renames the two deployment sections to
Terraform deployment and Manual deployment, documents what the Terraform
builds per node and every variable worth setting, and explains that the
data volume is an iSCSI attachment rather than a disk.
The manual path now covers the iSCSI login, the fstab entry and the
netplan source routing that Terraform writes, and states what br-wan is
for on a routed-NAT node: it names the VNIC for the OCI allocator and
holds the registered addresses, but is never bridged into OVS.
Opens with what Spinifex is and why OCI is worth running it on.
* fix: make gateway convergence work on a multi-node routed-NAT cluster
Three defects each stopped a multi-node cluster from ever attaching an
internet gateway, and all three produced the same uninformative symptom:
`terraform apply` sitting on `aws_internet_gateway.igw: Still creating...`
forever while every service reported healthy.
ovn-northd was given only its local OVSDB socket as a client connection.
A RAFT follower does not forward writes -- it answers "not cluster leader,
trying another server" -- so with one entry in the list, northd stops
translating Northbound intent into Southbound flows the moment database
leadership moves to another node. Measured on the OCI cluster:
SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no
chassisredirect port for any VPC created after leadership moved.
setup-ovn.sh wrote the member list only when the caller passed
--ovn-remote, which only install-node.sh did; it now derives the list from
the live RAFT membership otherwise, and says so when it finds a single
member, because that node needs a second pass once the others join.
The gateway datapath probe ran wherever the reconcile lease landed rather
than on the chassis owning the port. Both probes leave from the local
uplink, and under routed NAT the transit subnet exists identically on every
node, so a remote probe reported a healthy gateway as dead and aimed a
repair at a datapath the node does not own. GatewayPortLocal gates it.
On OCI the probe then ARPed the tenant-visible public IP, which the cloud
NATs and which appears on no interface anywhere. The reconciler now maps
the address through the same DatapathResolver the NAT rules are built from.
Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC
delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed
the destroy and stranded the VPC.
The OCI guide gains a Terraform section recording which shipped workbooks
were validated against a three-node OCI cluster and which are still open.
* docs: record the validated Terraform workbook results for the three-node OCI cluster
Six of the seven workbooks run against the OCI cluster pass, including the
three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the
clean eks-quickstart re-run, and names the two lifecycle gaps that bit during
the run so an operator recognises them rather than reading them as a broken
workbook.
* fix: route the NAT gateway's SNAT source through the datapath resolver
AddNATGateway was the one NAT path that did not map its external address to
the one that rides the wire. Where the public half of an allocation is NAT'd by
the cloud it is on no interface anywhere, so every private-subnet guest egressed
with a source that was dropped, while the gateway reported available and DNS
kept working because the VPC resolver is host-local.
The unmapped address also failed the host EIP sweep, which works in resolved
addresses, so the bind was pruned as foreign on the next pass and reinstalled on
the one after. The public half now survives as an external ID instead.
Also restructures the OCI guide around numbered steps through to running
Terraform, and moves the cloud-init and metadata sections into troubleshooting
where they are read when something is wrong rather than followed when it is not.
* fix: place a NAT gateway's OCI address on the node that forwards for it
OCI drops a packet whose source is not a registered private IP on the VNIC it
leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis.
Two independent placements decided those and nothing reconciled them: the
address was created by whichever worker served AllocateAddress, while the
chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster
that made private-subnet egress a coin flip, and the same Terraform passed or
died depending on which node answered.
The affinity pass already solves this for guests, by asking local OVS where the
tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which
is exactly and only this address. The equivalent local fact is the VPC's
chassisredirect binding, so ownership now has two answers rather than one, and
an address that is neither a guest's nor a gateway's is still left alone.
The VPC is recorded from the reconcile pass rather than from CreateNatGateway,
because the answer is the gateway chassis and reconcile is what decides it.
* fix: plumb a NAT gateway's host ingress only on its gateway chassis
A NAT gateway's address has no ENI, so neither the bind's locality test nor
the prune's could ever fire for it and every node installed its /32 route into
OVN. Latent on bare metal, where pool addresses are disjoint from the node
subnet; on a cloud that hands out the address from the nodes' own subnet it
outranks the connected route and swallows node-to-node traffic.
Give that address the same second authority the ocinet affinity pass has: the
VPC's chassisredirect binding, read from the local ovn-controller. An
unanswerable question still leaves the binding alone, so a single node whose
gateway port is not yet claimed behaves exactly as before.
The orphan EIP prune built its live set from ENI and user EIPs alone while also
sweeping host ingress, so every drift pass tore down the ingress of every live
NAT gateway and left it dark until the next host EIP tick.
* fix: ask the gateway chassis only about a NAT gateway's address
gatewayElsewhere skipped the NAT gateway marker it was named for and answered
for every spec, so on a node that is not the VPC's gateway chassis every EIP —
including a guest's, whose locality is its own port — was left unplumbed. Two
of three nodes bound nothing.
* test: assert the two private-subnet EKS workbooks in the tofu harness
eks-https-ingress and eks-gitops-argocd put their workers in private subnets
behind a NAT gateway, so a worker only registers Ready if its egress to the
cluster endpoint and to ECR survives that path. That makes them the workbooks
that catch a NAT gateway fault, and neither had an assertion, so neither could
be run at all — run_workbook calls assert_<name> by construction.
Both share eks-quickstart's readiness check, extracted unchanged. They also
name the worker size node_instance_type rather than instance_type, and tofu
rejects a -var for an undeclared variable outright, so the name is read from
the config instead of kept in a list here.
* test: count agents, not every node, when asserting EKS workers Ready
The control plane is three K3s servers, so an unfiltered node count reaches a
one-worker cluster's desired size before any worker joins — the assertion
passed on exactly the failure it exists to catch, reporting "4/1 workers
Ready". Filtering on the control-plane role label makes it 1/1, which is the
number that says a node-group worker registered.
Matters most for the two private-subnet workbooks, where the worker joining is
the whole proof that egress through the NAT gateway works.
* docs: record the two private-subnet EKS workbooks passing on OCI
|
||
|
dependabot[bot]
|
7cc8f59a7d | chore: bump the go-dependencies group with 5 updates (#1094) | ||
|
Josh Siviour
|
eb365eca7b |
fix: pin predastore to the object sub-resource guard (#1086)
An unserved object sub-resource fell through to PutObject, GetObject or DeleteObject, so a write to one stored the object itself. The pin also carries the meta leader health check fix and the router's single query parse. |
||
|
Josh Siviour
|
f536b1a0cb |
chore: pin predastore to the revision that serves bucket versioning (#1080)
The deploy builds with GOWORK=off, so spx links the pin rather than the sibling checkout, and the versioning handlers are absent from every deployed binary until it moves. |
||
|
Josh Siviour
|
b26478bd06 |
chore: pin predastore to the bucket sub-resource dispatch fix (#1074)
Every deployed binary links whatever go.mod pins, so the routing fix for bucket sub-resources is absent from the gateway until this moves. Without it a provider-level default_tags block still leaves no bucket creatable by Terraform. |
||
|
Josh Siviour
|
c10b6732d5 |
feat: publish AWS service endpoint names in DNS and match gateway cert SANs (#1068)
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs
ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.
The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.
admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.
Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.
* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain
ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.
The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.
Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.
* chore: pin northstar to the merged set-valued write path
|
||
|
Josh Siviour
|
578185f533 |
feat: supply the aws:PrincipalType condition key at the AWS gateway door (#1052)
Bump bluebottle for the new key. requestConditionKeys now maps this door's internal principal-type spelling to the canonical AWS value through the bluebottle constants, and omits the key for an unrecognized type rather than supplying an empty string. |
||
|
Julian Sommer
|
840b6e93c5 | release: update go.mod for v1.20.0 | ||
|
Julian Sommer
|
442444a274 | chore: pin to latest predastore | ||
|
dependabot[bot]
|
e3ea38ccf1 |
chore: bump the go-dependencies group with 10 updates (#1022)
Bumps the go-dependencies group with 10 updates: | Package | From | To | | --- | --- | --- | | [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.45.1` | `1.47.0` | | [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.33.2` | `1.33.4` | | [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.20.2` | `1.20.4` | | [github.com/aws/aws-sdk-go-v2/feature/ec2/imds](https://github.com/aws/aws-sdk-go-v2) | `1.19.1` | `1.20.0` | | [github.com/jackc/pgx/v5](https://github.com/jackc/pgx) | `5.10.0` | `5.11.0` | | [golang.org/x/crypto](https://github.com/golang/crypto) | `0.56.0` | `0.57.0` | | [golang.org/x/sync](https://github.com/golang/sync) | `0.22.0` | `0.23.0` | | [golang.org/x/sys](https://github.com/golang/sys) | `0.47.0` | `0.48.0` | | [golang.org/x/time](https://github.com/golang/time) | `0.15.0` | `0.16.0` | | [golang.org/x/tools](https://github.com/golang/tools) | `0.49.0` | `0.50.0` | Updates `github.com/aws/aws-sdk-go-v2` from 1.45.1 to 1.47.0 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.45.1...v1.47.0) Updates `github.com/aws/aws-sdk-go-v2/config` from 1.33.2 to 1.33.4 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.2...config/v1.33.4) Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.20.2 to 1.20.4 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.20.2...service/mq/v1.20.4) Updates `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` from 1.19.1 to 1.20.0 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/v1.20.0/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.19.1...v1.20.0) Updates `github.com/jackc/pgx/v5` from 5.10.0 to 5.11.0 - [Release notes](https://github.com/jackc/pgx/releases) - [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md) - [Commits](https://github.com/jackc/pgx/compare/v5.10.0...v5.11.0) Updates `golang.org/x/crypto` from 0.56.0 to 0.57.0 - [Commits](https://github.com/golang/crypto/compare/v0.56.0...v0.57.0) Updates `golang.org/x/sync` from 0.22.0 to 0.23.0 - [Commits](https://github.com/golang/sync/compare/v0.22.0...v0.23.0) Updates `golang.org/x/sys` from 0.47.0 to 0.48.0 - [Commits](https://github.com/golang/sys/compare/v0.47.0...v0.48.0) Updates `golang.org/x/time` from 0.15.0 to 0.16.0 - [Commits](https://github.com/golang/time/compare/v0.15.0...v0.16.0) Updates `golang.org/x/tools` from 0.49.0 to 0.50.0 - [Release notes](https://github.com/golang/tools/releases) - [Commits](https://github.com/golang/tools/compare/v0.49.0...v0.50.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-version: 1.47.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.33.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.20.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/feature/ec2/imds dependency-version: 1.20.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/jackc/pgx/v5 dependency-version: 5.11.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/crypto dependency-version: 0.57.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/sync dependency-version: 0.23.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/sys dependency-version: 0.48.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/time dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/tools dependency-version: 0.50.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Julian Sommer
|
5c367cac43 | chore: update github.com/containerd/containerd/v2 | ||
|
dependabot[bot]
|
393f78a9cd | chore: bump google.golang.org/grpc (#1013) | ||
|
Julian Sommer
|
ea51cd4d7c | release: update go.mod for v1.19.0 | ||
|
joshsiv-mulga
|
befdb1ca10 |
fix: pin viperblock for WALBaseDir and set it on the mounted volume path
dev did not build: the WALBaseDir call sites landed while go.mod still pinned viperblock v1.18.0, which predates the field. constructMountedVB passed WALBaseDir to the nbdkit plugin but not to its own engine, so the daemon would look for WALs under BaseDir while the plugin wrote them under WALBaseDir. |
||
|
dependabot[bot]
|
7d639cf283 | chore: bump the go-dependencies group with 6 updates (#1005) | ||
|
Ben Duncan
|
b6d01370dd |
fix: render northstar's recursion allowlist so a node is not an open resolver (#993)
* fix: render northstar's recursion allowlist so a node is not an open resolver northstar.toml is written with allow_recursion = false and an allow_recursion_from list built from cluster membership, so a freshly formed node serves recursion to the cluster and refuses it to the internet. The list is the node's own bind and advertise addresses plus every peer's host, with any port stripped. Both local addresses matter: a node's resolv.conf points at its own advertised address, so the query arrives from that public IP over lo rather than from 127.0.0.1, and an allowlist of loopback alone would stop every node resolving internal names. Guests are covered without appearing in the list. vpcd's per-tap DNS shim relays guest queries to a node's :5300 from a node address, so trusting the nodes trusts every guest behind them and no guest CIDR is needed. The template comment carries the caveats for allow_recursion = true, since the cost of setting it is remote from the setting: an open resolver is found by scanners within hours and gets the node's whole address range blocklisted, which is an outage for every tenant on it. The list is rendered at formation, so a node added later is not in its peers' files. Regenerating it live from membership, as the daemon already does for peers.nft, is follow-on work tracked on mulga-t3mub. * chore: pin northstar to the recursion ACL and atomic zone reload The recursion allowlist this branch renders is enforced by northstar's RecursionPolicy, which lands in dev at a0e741e. CI resolves northstar from go.mod rather than the workspace, so the build failed on NewRecursionPolicy being undefined until this pin moved. The AWS SDK bumps come with it. |
||
|
Julian Sommer
|
a5fc2b517c | fix: supply aws:userid on the gateway authorization path (#987) | ||
|
Julian Sommer
|
ab167a192c | fix: reject an unresolvable policy variable on the IAM write path (#986) | ||
|
Julian Sommer
|
1fd4373b35 | fix: verify caller-supplied role ARNs against the stored canonical ARN (#972) | ||
|
Josh Siviour
|
4c948512d1 |
fix: follow ListObjectsV2 continuation tokens on every production caller (#958)
* fix: follow ListObjectsV2 continuation tokens on every production caller Six ListObjectsV2 call sites (DeleteSnapshot, DescribeSnapshots, EKS etcd-snapshot selection, DescribeTags, and both key-pair listings) issued a single request and never read IsTruncated or NextContinuationToken. This was latent while predastore ignored max-keys and always answered IsTruncated: false; predastore#117 pages listings at 1000 keys, turning a truncated listing beyond that into a silently wrong answer rather than a short one. DeleteSnapshot in particular could delete 1000 objects, report success, and orphan the rest, and the EKS restore path could resolve to a stale snapshot if the newest keys fell past the first page. Add objectstore.ListAll, a shared helper that drains a listing to exhaustion, accumulating Contents and CommonPrefixes across pages and erroring instead of returning a partial result if the server claims truncation with no token to resume from or never finishes within a bounded page budget. All six call sites, plus the ebsfake test double with the same shape, now use it. Bump the predastore pin to dev head ff6a1dc4 so a real cluster picks up the server-side pagination fix this depends on. * fix: use external test packages and exclude test fakes from diff coverage test-package-check requires new test files to bind to the exported API so internals can change without rewriting tests. Move list_test.go and delete_snapshot_pagination_test.go to their _test packages, adding a small export_test.go for the one unexported constant a test needs to assert against. Simplify list_test.go's pagingStore to embed MemoryObjectStore instead of hand-stubbing unused methods. Exclude spinifex/testutil from diff-coverage.sh: those files are fakes and harnesses driven only from other packages' _test.go files, so go test records them as zero-covered under the package they live in however hard another package's tests exercise them, which is not a real coverage gap. |
||
|
dependabot[bot]
|
7ff125a343 | chore: bump the go-dependencies group with 12 updates (#941) | ||
|
Josh Siviour
|
3680e5b5bc |
feat: render predastore admin_port and gate deploys on /readyz (#936)
* feat: render predastore admin_port and consult /readyz in health probe Spinifex now allocates predastore's admin listener port (8660, after the existing gate/blob/meta bases) and renders admin_port into both the single-node and multi-node predastore.toml templates. The daemon's health probe consults /readyz over that port on the two weak paths (single-host gate fallback and hosts running no meta node), falling back to the prior behavior when no admin port is configured or the listener refuses. * build: pin predastore to the release carrying Host.AdminPort The admin port field this branch reads is new in predastore, so CI could not compile against the pinned version: go.work hid the gap locally by resolving predastore to the checkout instead of the module. |
||
|
Julian Sommer
|
8c7e3dfa7b | refactor: use bluebottle/pkg/safecast for the numeric conversions (#913) | ||
|
Julian Sommer
|
95afec6a76 | refactor: use bluebottle/pkg/fipsboot for the FIPS 140-3 startup guard (#911) | ||
|
Julian Sommer
|
97acfbbd80 | refactor: use bluebottle/pkg/tlsconfig for the TLS curve allowlist (#910) | ||
|
Julian Sommer
|
d82ee71b54 | refactor: drop duplicated bluebottle logic and hand-rolled token buckets (#905) | ||
|
Julian Sommer
|
0362cdf371 | refactor: drop the forked HTTP middleware for the shared one (#904) | ||
|
Julian Sommer
|
46d8c03293 |
fix: pin predastore to the dev build that drops ExtractPolicyName
The bluebottle bump in #887 removed pkg/auth.ExtractPolicyName, which predastore v1.18.0 still calls, breaking e2e-build on the runner. |
||
|
Julian Sommer
|
08da638375 | refactor: route policy ARN parsing through the shared bluebottle parser (#887) | ||
|
Julian Sommer
|
a865ffc48e | release: update go.mod for v1.18.0 | ||
|
Julian Sommer
|
d80542ac80 | fix: pin predastore dev version | ||
|
Julian Sommer
|
58e1bd2c5a | fix: reject and enforce IAM Condition blocks on identity policies (#864) | ||
|
dependabot[bot]
|
9b5acc738c | chore: bump the go-dependencies group with 8 updates (#855) | ||
|
Julian Sommer
|
7b1890eb8a | refactor: use the stdlib uuid package instead of github.com/google/uuid (#841) | ||
|
Julian Sommer
|
7f6557d55d | fix: make state retry policy injectable (#836) | ||
|
Julian Sommer
|
a4a644c558 | release: update go.mod for v1.17.0 | ||
|
Ben Duncan
|
225bf65a8e |
feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800)
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut. |
||
|
Julian Sommer
|
4716ea306f | chore: bump insomniacslk/dhcp to fix panic | ||
|
dependabot[bot]
|
8fe299ba40 | chore: bump the go-dependencies group with 8 updates (#801) | ||
|
joshsiv-mulga
|
1e00425bb3 |
fix: add missing puddle/v2 dependency for pgxpool
The Ochre vector store change imported github.com/jackc/pgx/v5/pgxpool without running go mod tidy, so github.com/jackc/puddle/v2 was absent from go.mod and go.sum. Any build under -mod=readonly failed, breaking the multi-node E2E job at 'go test -c'. Also promotes pgx/v5 from the indirect block, since it is imported directly. |
||
|
Josh Siviour
|
d52245001e |
feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788)
* feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate |
||
|
Josh Siviour
|
560b437f3b |
fix(health): probe predastore raft state instead of echoing the config (#786)
/health reported service_health.predastore == "ok" for every node simply because predastore was named in spinifex.toml, without contacting it — on env19 all three nodes stayed green for the whole time raft had no leader and volume creates were failing. A probe registry now backs /health's predastore entry with a real raft status dial (ok/no_leader/unreachable, cached briefly and bounded by a short timeout), while every other service keeps its unprobed "ok" default unchanged. spx admin storage status had the same root problem from the other side: it dialed HTTPS /health and /status endpoints predastore's meta nodes never served (they speak an opcode-multiplexed RPC), so every meta node always reported unhealthy with blank raft fields. It now dials the same raft status RPC predastore#82 exposes, verified against the cluster CA, with the JSON output shape unchanged. |
||
|
Josh Siviour
|
f4623658e5 |
refactor: detach abandoned viperblock engines instead of stopping their goroutines by hand (#778)
* refactor: detach abandoned viperblock engines instead of stopping their goroutines by hand viperblockd opens an engine for a single operation in ten places and ends it by stopping the background goroutines, never reaching Close. That leaves the chunk GC sweeper running on an engine the caller has finished with, and leaves the volume recorded as still held, so nothing can tell an engine that was handed over from two engines holding one volume at once. Replace each with vb.Detach(), which stops all three goroutines and reports the release. One site keeps its bare StopChunkUploader: the daemon-side VB that tracks state while the nbdkit plugin owns the data path is not finished with the volume, and must not report otherwise. An invariants test scans the package for a goroutine stopped outside Detach, so the pattern cannot come back one call site at a time. * chore: bump the viperblock pin to the release carrying Detach Detach does not exist in the previous pin, so the detach work in this branch cannot build without it. Targeted go get of one module, no tidy — viperblock is the only requirement that moves. |
||
|
Josh Siviour
|
98ec781629 |
feat: decouple the EC2 control plane from the viperblock engine behind an EBSProvider contract (#770)
* feat: introduce the EBSProvider interface and control-plane metadata store
Phase 1 of decoupling the control plane from viperblock. Adds a
provider-neutral block-storage contract that spinifex owns end to end.
ebsprovider defines EBSProvider, whose operation set and idempotency rules
follow the CSI controller service, with NATS request/reply as the transport
rather than CSI's gRPC. Every request and response carries an explicit schema
version so wire skew fails loudly at the boundary instead of silently decoding
a changed payload. The package deliberately exposes no viperblock types:
provider-specific state is an opaque handle. Ships with a NATS client against
the ebs.provider.v1.* subjects and an in-memory fake for tests.
ebsmetadata gives spinifex its own volume and AMI documents under a spinifex
key prefix, so the control plane can answer DescribeVolumes and DescribeImages
without reading the storage engine's persisted state.
The EC2 volume and snapshot handlers gain provider-backed paths for create,
describe, modify, attach-state and delete. Provider injection is optional for
now, so the existing engine path stays the default until the viperblockd side
serves the new subjects and the metadata migration lands.
vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol
it used and is now a local helper.
* feat: serve the EBS provider contract from viperblockd
Gives the ebsprovider NATS client a server. Volume create, describe, expand
and delete, plus snapshot create and delete, are now handled by the daemon
that owns BaseDir and the mounted-volume registry, which is where viperblock
engine construction belongs.
Every handler prefers an already-mounted engine over opening a second one on
the same volume, and expand refuses outright while the volume is mounted.
Opening a second engine on a live volume is the double-writer hazard this
decoupling exists to remove, so the boundary enforces it rather than trusting
callers. Requests carrying an unrecognised schema version are rejected at the
boundary instead of being decoded on a best-effort basis.
Snapshot creation replies immediately with an operation ID and completes
asynchronously on a per-snapshot subject, since a large volume will outrun the
request timeout. Draining the volume stays with the caller: it coordinates
with the guest and is not something the storage boundary can reach.
The snapshot create subject gains an explicit create verb. Addressing it as
ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject
and left the wildcard unsubscribable.
Publish and unpublish are not served yet; the mount and unmount logic they
would front is still inline in launchService and needs extracting first.
Also adds a conformance suite that runs one set of contract assertions against
both the in-memory fake and the NATS client, so each side is checked against
the contract rather than against the other's structs.
* feat: let an operator select the EBS provider
Adds an [ebs] provider key and wires the NATS provider into the EC2 volume,
snapshot, image and instance services at the daemon's composition root. Until
now nothing injected a provider, so the provider-backed code paths in those
services were unreachable.
The key is provider-neutral rather than nested under [viperblock], because it
names the boundary rather than one provider's settings; a second provider will
not need it renamed. An empty value resolves to "embedded", so deployed config
files that predate the key keep today's behaviour, and an unrecognised value
is a startup error rather than a silent fallback.
Startup probes the provider's capability verb and warns when nothing answers.
The daemon and viperblockd start independently, so an unanswered probe is not
fatal, but without it a misconfigured node fails much later as an unexplained
request timeout on the first EBS call.
"viperblockd" stays experimental: under it volume metadata lives in the
control plane's own document, and the leak reaper, GC and admin paths still
read viperblock's config.json, so they cannot see tenant or tag data for
volumes created that way.
* fix: make every volume-metadata consumer provider-aware
Several readers and writers of volume metadata were never given a provider
branch, so under the viperblockd provider they silently used the wrong store.
Each of these is a correctness bug rather than a missing feature, because the
control plane had already stopped writing the document they read.
Modifications had nowhere to live: ModifyVolume built its result in memory and
dropped it, and DescribeVolumesModifications read it back out of viperblock's
config.json, so a describe straight after a successful modify returned nothing.
Volume metadata now carries its own modification record.
The leak reaper read attachment and tags from config.json and wrote orphan
markers to tags.json, neither of which exists for a provider-managed volume, so
a leaked volume was invisible to it and any mark it did write could not be read
back. Tag mutations had the same split: a tag applied after creation never
reached the document DescribeVolumes reads.
Snapshot deletion could not see a provider-managed clone, because a clone
records its source only in the control-plane document. That let a delete strip
chunks a live clone still reads, so it is fixed here rather than left for the
migration.
Volume enumeration and DescribeVolumeStatus were likewise scanning object
prefixes that a provider-managed volume does not create.
Also adds an encryption flag to the volume document, which DescribeVolumes had
been reporting as false unconditionally and snapshots could not carry at all,
and a list operation for AMIs so the image path can follow later.
The legacy path is unchanged; its invariant tests pass unmodified.
* feat: backfill ebsmetadata documents from legacy volume and AMI state
DescribeVolumes under the viperblockd provider enumerates only ebsmetadata
documents, so a volume created before the switch had no document and was
invisible rather than merely slow to read. A per-object read fallback cannot
fix enumeration, so both a backfill and a fallback are needed.
Add an object-store migration kind to the migrate framework. Predastore has no
conditional write, so an object-based version stamp would be last-writer-wins
across nodes; RunObject stamps progress in a caller-supplied JetStream KV
bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a
read-then-write rather than a compare-and-swap, so correctness rests on every
ObjectMigration.Run being safe to re-execute.
The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the
numbered config migrations: it needs viperblock's VBState schema to decode the
legacy config.json, and migrate is imported by every handler that stamps a KV
bucket version. It converts what it can reach and skips unreadable objects with
a log, so one corrupt volume cannot hide the rest of the fleet.
ebsmetadata gains the legacy read fallback for the window before the migration
has run everywhere. It stays free of viperblock by taking the decode as an
injected function value, wired at the composition root, and the fallback is
retired by leaving it unset. A fallback read never writes the converted value
back, which would race the migration.
Prefix enumeration pages to exhaustion. Stopping at the first page would let
the backfill stamp itself complete having converted only the volumes it
happened to see, reintroducing the invisible-volume failure it exists to
prevent.
The migration runs from configureEBSProvider, so only when the provider is
selected; backfilling while the embedded engine is still authoritative would
produce documents that immediately go stale. Failure is fatal rather than
advisory, unlike the capability probe beside it. This makes the switch to
viperblockd one-way, which is now recorded on the config field.
* feat: serve volume publish and unpublish from viperblockd
PublishVolume and UnpublishVolume were the last unserved verbs on the provider
contract, and they are the ones that attach a volume to a VM. Until they are
served the provider path cannot boot an instance, so provider = "viperblockd"
cannot be verified end to end.
The mount and unmount logic was two large anonymous handler bodies inline in
launchService. Rather than write a second mount path, extract each into a named
function and have both the legacy ebs.mount/ebs.unmount handlers and the new
provider handlers call it. Two divergent implementations of nbdkit spawning and
the seal decision would be far worse than the inline bodies. The extraction
relocates code without rewriting it; response publishing stays in each handler
so the provider never publishes to ebs.mount.response.
Publish is idempotent: a volume this node already has mounted returns its
existing attachment rather than starting a second nbdkit against it, which is
the double-writer hazard the boundary exists to prevent. Unpublish preserves
the invariant that a failed seal leaves the volume mounted so a retry can
re-attempt it, and reports an error rather than success.
Both verbs are node-addressed, so they subscribe directly rather than through
the spinifex-workers queue group, and are registered only when NodeName is set.
There is no queue-group fallback: a subject that already names one node has
nothing sensible to fall back to.
Read-only attachment is refused rather than silently granted read-write, since
nbdkit is started read-write or not at all.
Also fixes a latent panic carried by the extracted mount path: viperblock.New
returns a nil VB alongside its error, and SetCacheSize was called on it before
the error was checked, so a failing New would have taken down the handler.
* feat: read and write AMI metadata through ebsmetadata
The backfill migration wrote ebsmetadata AMI documents that nothing read: every
AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at
{imageID}/config.json. Give the image service provider branches for the AMI
read, write and enumerate paths, with enumeration going through ListAMIs rather
than an object-prefix scan, as DescribeVolumes already does.
AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a
viperblock type on an interface boundary in the instance package. It now returns
ebsmetadata.AMI on both paths, the embedded one converting before it returns, so
the boundary names no viperblock type whichever provider is selected.
The legacy-to-document mapping had accumulated three copies. Consolidate them in
ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata
itself stays free of viperblock, and cover it with a reflective test that fails
when a field is added to either schema without being wired through. Sixteen
fields copied by hand in three places would have dropped one silently.
The admin image paths read whichever representation exists and write back to
that same one, so a provider-managed AMI never gains a spurious config.json.
FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan:
missing a dependent there deletes an AMI a live volume still needs, the same
class of bug as the snapshot clone path fixed earlier on this branch.
That union is built by hand rather than by wiring Store.ListVolumes' fallback,
which swallows per-ID read errors. This path deliberately fails closed on a
transport error so a delete cannot proceed on a partial picture, and the
fallback would have quietly turned that into failing open.
The CLI image import now also writes the document, best-effort, so an AMI
imported after the migration is not left reachable only through the fallback.
* fix(ec2): resolve volume config through ebsmetadata when a provider is configured
A provider-created volume has no vol-*/config.json — its metadata lives in
the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy
key unconditionally, so AttachVolume reported a volume that DescribeVolumes
had just listed as available as InvalidVolume.NotFound.
Routing through ebsmetadata.Store also puts this path behind the legacy
fallback, so a volume created before the provider switch stays attachable.
The NoSuchKey remap now names the key it missed; its silence made the
original failure invisible in the daemon logs on every node.
* test(bench): add an EC2 control-plane latency benchmark
Measures the EBS control-plane operations either side of the provider
boundary, so the cost of the NATS hop can be stated rather than guessed.
Reports median and p95 with a bootstrap CI on the median difference, and
labels a delta indicative unless that CI excludes zero.
* fix(ec2): drain the volume before a provider snapshot
The legacy CreateSnapshot branch drains first so the snapshot reads a
current checkpoint; the provider branch did not, and viperblockd documents
the drain as the caller's responsibility. Both sides deferred to the other,
so an attached volume snapshotted stale data with no error raised.
DrainVolume and volumeAttachment now take the fallback state and instance
as strings rather than a viperblock.VolumeMetadata, removing a viperblock
type from an exported signature.
* feat(ec2): route the instance root volume through the EBS provider
RunInstances built its own viperblock engine to clone the AMI snapshot into
the root volume, which made every launch a second writer against a volume
viperblockd already serves. The launch now calls the provider's CreateVolume
with the AMI's snapshot and source volume, and records the result in
ebsmetadata so DescribeVolumes and attach can see it.
A snapshot clone needs both the snapshot ID and its source volume ID to
resolve base blocks, so CreateVolume in the volume handler now sends
SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected
every create-volume --snapshot-id outright.
The gp3 IOPS and throughput constants move to spinifex/types so the instance
and volume handlers share one definition rather than duplicating literals.
* feat(ebsprovider): add a bounded volume seed to CreateVolume
The EFI variable store must be initialised from the launching node's firmware
VARS template, byte for byte. Passing a host path would not work: CreateVolume
is queue-subscribed, so any node can service it, and a foreign node's OVMF
package would bind the guest's pflash region to the wrong firmware build.
CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd
writes them at offset 0 in the engine it already builds. The seed applies only
when the create actually allocates the volume, so a relaunch never overwrites
a BootOrder the guest has since written.
JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes
bounds the seed well inside the NATS max_payload the cluster runs with and
both ends reject an oversized seed with a clear error rather than letting the
publish fail. VolumeSeeding advertises the behaviour so a provider that cannot
seed says so instead of silently dropping the bytes.
* feat(ec2): route the EFI variable store through the EBS provider
prepareEFIVolume built a control-plane viperblock engine to seed the variable
store from the firmware VARS template, making every UEFI launch a second
writer against a volume viperblockd already serves. It now calls the
provider's CreateVolume with the template as seed bytes, so the store is
allocated and written by the daemon that owns the volume.
The template is still read on the launching node and shipped with the request.
CreateVolume is queue-subscribed, so any node can service it, and a VARS
region seeded from a foreign node's OVMF build would bind the guest's pflash
to the wrong firmware.
pflash refuses a VARS region that is not byte-exact, so the returned capacity
is checked against the template length; a provider that rounded up to a GiB
boundary would otherwise yield a guest that cannot boot.
The engine path stays behind a nil-provider guard until the legacy branch goes.
* feat(ec2): route image snapshots through the EBS provider
snapshotRunningVolume and snapshotStoppedVolume each built a control-plane
viperblock engine to take the snapshot backing a new AMI, making CreateImage
a second writer against a volume viperblockd already serves. Both now call
the provider's CreateSnapshot with the volume's provider handle.
The running path keeps its drain, which is what makes the snapshot current
rather than one predating a guest-triggered write. The stopped path needs no
drain, and the daemon already prefers a live mounted engine over opening a
second one, so it handles both cases itself.
getVolumeConfig read the legacy config.json, which a provider-created volume
does not have, so every CreateImage against one failed. It now reads the
ebsmetadata document under a provider, the same way attach does.
* fix(daemon): only the owning node creates an image for a stopped instance
handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which
only the node holding the VM passes, so a running instance is imaged once.
A stopped instance lives in the cluster-replicated KV instead, which every
node reads successfully, so all of them ran the full snapshot and AMI
creation. The caller saw the first reply and the rest leaked: on a three-node
cluster one create-image produced three AMIs and three snapshots.
A stopped instance records its owner in LastNode, so a node that is not the
owner now declines. Gather tallies error frames and keeps waiting, returning
only on a success, so declining cannot preempt the owner's reply.
An instance stopped by a build that never set LastNode elects an owner
through the KV's compare-and-swap, since having every node decline would make
create-image fail permanently for those records.
* perf(ec2): serve DescribeVolumes by ID from the volume documents
The provider branch called ListVolumes unconditionally and filtered in
memory, so asking for one volume read and decoded every volume document in
the bucket. The non-provider branch already had the equivalent fast path.
Explicit IDs now fetch their documents directly. The tenant check moves with
them: a document belonging to another account reports not-found rather than
being returned, matching what the enumeration filter did.
Measured against a memory object store, the fast path stays flat near 20us
regardless of bucket size while the enumeration grows linearly, reaching
4.6ms at 500 volumes.
* chore(lint): ban the viperblock import from the EC2 control plane
The control plane reaches block storage through ebsprovider now, but nothing
stopped a handler from importing the engine again. That is not hypothetical:
the instance handler first imported viperblock for a config struct alone, and
the engine construction followed thirteen days later.
depguard denies the import across handlers/, which matters because
viperblock.New starts a chunk uploader — every construction is a writer
against a volume viperblockd already serves. A types-only import is denied
too, since that is exactly how the last one started.
The four files still carrying the pre-provider engine path behind a
nil-provider guard are exempted by name. The exemption list is the remaining
work: it shrinks to nothing when that path goes.
* fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot
CopySnapshot wrote a control-plane snapshot document under the new ID and
nothing else. The copy described correctly and CreateVolume against it
succeeded, so it looked like it worked, but the new ID had no viperblock
snapshot metadata behind it and the volume failed when the guest attached it.
Producing the metadata means re-sealing it under the destination ID. On an
encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the
sealing helpers are unexported, so spinifex cannot write it. Copying only the
unencrypted form would be dead code on any cluster with encryption enabled.
Refusing with UnsupportedOperation fails at the call the caller can act on
rather than at attach time. Source lookup and the ownership check still run
first, so a bad ID or another tenant's snapshot reports that instead.
* test(ec2): give handler test fixtures their own legacy config.json shape
Handler tests built the legacy config.json fixture by importing
viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly.
Mirror that shape in ebsmetadata/vblegacy, next to the production code
that already reads it (ebsmetadatabackfill), and switch the
fixture-only usages across the volume/snapshot/image handler tests to
it. A byte-for-byte equivalence test against the real viperblock types
guards the mirror from drifting out of sync.
Call sites that hand a viperblock type into putVolumeConfig,
mergeVolumeConfig, or buildVBConfig are untouched, since those
functions still name the real type in their signature.
* feat(ebsprovider): add CopySnapshot to the EBS provider contract
Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy
NATS subject, MemoryProvider and NATSProvider implementations, conformance
coverage) and a viperblockd handler that duplicates a snapshot's metadata via
viperblock.CopySnapshotMeta on the source volume's VB.
* feat(ec2): serve CopySnapshot through the EBS provider
CopySnapshot has been refusing with UnsupportedOperation because a copy needs
its own snapshot metadata on the backend under the new ID, and on an encrypted
volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the
control plane cannot re-seal it. The provider contract now carries a copy
operation, so the storage engine does it.
The provider path generates a new snapshot ID, has the provider duplicate the
backend metadata under it, then writes the control-plane document. Rollback
mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot
rather than leaving one nothing references. Ownership is still checked against
the source before anything is created.
The no-provider path keeps refusing. It has no way to produce the backend
metadata, and the copy it used to write described fine and then failed at
attach — a resource that fails as far as possible from the call that made it.
Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot
reached the provider rather than existing only as a control-plane document.
That assertion is what catches the old behaviour.
* refactor: remove the embedded EBS engine path from the EC2 control plane
viperblockd is now the only EBS provider. The pre-provider path that
constructed a viperblock engine in-process is deleted, and with it every
github.com/mulgadc/viperblock import under spinifex/handlers/ (tests
included). The depguard ec2-control-plane rule drops all four file
exemptions and the !$test carve-out, so the rule now reads simply as
"nothing under spinifex/handlers/** may import viperblock".
viperblock.VolumeConfig was also a data carrier on the provider path. It
is replaced by the spinifex-native ebsmetadata.Volume rather than the
vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is
decoding old objects, while ebsmetadata.Volume already carries every field
the call sites read, so the conversion step disappears instead of being
reimplemented against a legacy shape.
The embedded-only helpers (volumeConfigWrapper, putVolumeConfig,
mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached,
deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json
prefix scan) had exactly one production caller chain each, all through a
nil-provider branch, and are removed. The snapshot-dependency guard on
DeleteVolume is provider-agnostic and is kept, now running on the provider
path where it previously did not.
config: viperblockd is the default and an empty Provider resolves to it.
An explicit provider = "embedded" is rejected at config load rather than
silently upgraded, because starting anyway would migrate a node's volumes
to ebsmetadata one-way without the operator asking. The generated
spinifex.toml template emits viperblockd.
* fix(images): write the imported AMI document from the config the import wrote
ImportDiskImage creates the AMI's snapshot and records it on its own
*viperblock.VB, which viperblock.New allocates fresh, so the value lands only
in {imageID}/config.json. The import command was persisting the ebsmetadata
document from its pre-import manifest copy, whose SnapshotID is still empty.
Store.GetAMI only consults the legacy config.json fallback when the document
is missing, so that empty-SnapshotID document permanently shadowed the real
value and every launch from an imported AMI failed the zero-copy clone check
with ServerInternal.
Rebuild the document by reading back the config.json the import just wrote. A
missing or unreadable config is skipped rather than written from the stale
manifest: no document still resolves correctly through the legacy fallback.
* test(e2e): read NODE from spx get vms instead of VCPU in the spread check
spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to
field 8. The spread placement check still read field 6, so every instance
reported its vCPU count instead of its node, the unique-node count always
collapsed to 1, and the check failed on clusters that had spread correctly.
Read field 8 and strip the ANSI colour codes wrapping each cell.
* feat(viperblockd): add a --debug flag so viperblock debug logging is reachable
viperblockd hardcoded Info, and viperblock's VB.logger() falls back to
slog.Default() because vb.log is only ever set by SetDebug. That left the
engine's debug records, including the chunk uploader's start and disabled
lines emitted during New(), unreachable in a deployed environment.
Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG
rather than the bare debug key, which several commands share. Also sets the
Config.Debug field, which existed but was set only by tests.
* fix(viperblockd): release the chunk uploader when openVolumeVB fails
viperblock.New starts a chunk uploader and a WAL syncer before the backend is
touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the
caller while leaving both running with no handle to stop them. Repeated backend
failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders
on a 30s ticker for the life of the process.
Adds invariant tests for both halves of the ownership rule: a mounted volume's
snapshot create and copy reuse the live engine and log zero volume opens, an
unmounted volume opens exactly one, and a failed open leaves no uploader behind.
The open counts are read from the same log record a production alarm watches.
* fix(ebsprovider): route requests to the node that owns the mount
Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one
spinifex-workers group, so NATS delivered each request to a random node. Only
the node holding the mount can reuse the live engine through findMountedVolume;
any other node constructed a second viperblock engine, with a live 30s chunk
uploader, against a volume nbdkit was serving on a different host. Measured on
a three-node cluster, this was roughly two snapshots in three.
Rather than resolve the owner, let the owner answer. A node subscribes
ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts
a volume and drops it on unmount, mirroring the per-volume ebs.config
subscription. Clients try that subject first and fall back to the queue group
only on ErrNoResponders, which means nothing is mounted and opening an engine
is correct. A timeout deliberately does not fall back, since the owner may
still be running the operation.
Applies to snapshot create and copy, expand and describe, the four verbs that
reuse a mounted engine. Delete keeps queue dispatch.
* fix: rebuild viperblockd mounted-volume registry after a restart
Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a
viperblockd restart while a VM keeps running leaves nbdkit serving a volume
the daemon has forgotten. Owner routing then infers from ErrNoResponders that
nothing is mounted and falls back to the queue group, which opens a second
viperblock engine, with a live chunk uploader, on another node. The same empty
registry also defeats handlePublishVolume's double-mount guard, so a republish
could start a second nbdkit.
Discover surviving nbdkit processes from /proc at startup and rebuild the
registry from them, following orphan_scan.go's precedent: match comm exactly,
parse the argv for volume=<id> and --unix/-p, and adopt a PID only when
nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or
killed; anything that fails corroboration is logged and skipped. Recovery runs
before the first subscription so no handler can race it.
Extract mountVolume's engine construction into constructMountedVB so mount and
recovery cannot drift, and release the WAL syncer viperblock.New starts when
construction fails after it.
* fix: corroborate recovered nbdkit processes from argv, not a pidfile
nbdkit is launched with -f (required for the Go plugin) and never writes the
--pidfile it is given, so no such file exists on any node; the daemon's own
/tmp is a PrivateTmp namespace besides. Corroborating against it rejected
every genuine mount, making recovery a no-op in production.
Corroborate against what the kernel actually has: the process must serve our
plugin, name this daemon's base dir, and still hold its socket. Adopt a volume
once, so two live nbdkits for it leave one registry entry and a warning rather
than two conflicting ones.
* feat: add a second EBS provider backed by qcow2 and qemu-nbd
The decoupling was justified by swappability that nothing demonstrated: the
conformance suite ran against MemoryProvider and against a NATSProvider wired
to a MemoryProvider, so both arms were the same implementation behind
different transports. A fake is not a second implementation.
Add three pieces. natsserve serves the ebs.provider.v1.* contract from any
EBSProvider, depending on nothing but the interface, and replaces the
test-only server twin so conformance now exercises production code. The
conformance suite moves to its own package and is exported so implementations
outside ebsprovider can run it, with no assertion changed. qemunbdd is the
second implementation: one qcow2 file per volume, qemu-nbd for publish, real
snapshot copies instead of chunk references, and every exec behind a runner
seam so its tests need no qemu.
Owner-subject routing is deliberately not served neutrally: it needs a
per-node mounted-volume registry the interface does not model, so it stays in
viperblockd and a neutral server's clients correctly fall back to the queue
group.
* test: run the conformance suite against the qcow2 provider
The suite passes unmodified against an implementation sharing no code with
viperblock, both in-process and through natsserve over the real wire
contract, which is the swappability claim the provider boundary was built to
support.
Two real defects it caught. ExpandVolume and GetVolume inspected the image
before consulting the publication map, so qemu-img could not open a volume
qemu-nbd held the write lock on and the refusal surfaced as an internal
error instead of ErrVolumeInUse; qemu-img info now runs with --force-share,
so describing a volume no longer depends on whether it is attached. Publish
also failed with qemu-nbd's raw exit status when the socket path exceeded the
108-byte AF_UNIX limit, and now reports that reason itself.
* feat: make the EBS provider selectable by config
Add the qemunbd selector alongside viperblockd, a spx service command that
serves the qcow2 provider over natsserve, and service registration. Config
validation accepts the new value everywhere viperblockd is accepted, while
embedded stays rejected by name.
The daemon logged a hard-coded viperblockd constant as the active provider
rather than the resolved one, in configureEBSProvider and in both of
probeEBSProvider's lines; all three now report what is actually configured.
Both daemons answer the same subjects, so exactly one may run on a cluster.
* test: add a live lifecycle check for the qcow2 provider
Drives create, publish, write, snapshot, restore and read-back entirely
through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a
cluster node, so the client under test is the same shape the control plane
uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE
is set, so it stays out of preflight.
* fix: publish NBD URIs in the form standard clients accept
PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy
pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright
with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever
consume what the EBS provider boundary hands back.
Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit
all accept. ParseNBDURI now takes both that and the TCP form, and still
accepts the legacy nbd:unix: syntax for state written by older daemons, so
a conformant third-party provider's URI is no longer rejected on the way
in either.
Verified against qemu 10.0.11: qemu-img and a real -drive option string
both open the URI form, and qemu reports both forms as nbd+unix://?socket=,
so it already treats the URI as canonical.
* test: validate published exports with libnbd's client tools
Every other assertion about the EBS provider goes through our own Go
client, so nothing has ever checked whether the export we publish is usable
by an NBD client that knows nothing about this codebase. That is the layer
neither the conformance suite nor a CSI-shaped contract can reach.
RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume:
the URI connects, the export advertises its size and block-size
constraints, data written externally reads back byte for byte, a ReadOnly
publish sets the read-only transmission flag, and UnpublishVolume tears the
export down. It skips when libnbd-bin is absent rather than passing
vacuously.
Wired to qemunbdd, where all five failed before the URI fix. The publish
assertion in the main suite now requires a dialable URI instead of merely
a non-empty string, which is what let the wrong form through.
* test: run the external NBD client suite against a live viperblockd
The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit
export is the comparison worth having: one external client, two providers it
cannot tell apart, and every difference is a candidate leak in the boundary.
RunNBDClientSuiteWithConfig takes the node name and volume prefix a live
cluster needs, and publishForNBD now unpublishes and deletes what it created
so a run against a real cluster leaves nothing behind. Subtests dial a
normalised URI so a provider emitting the legacy syntax still gets its export
tested rather than every subtest failing on the same string; whether the URI
itself is standard is now its own subtest.
Adds a reconnect check, since publication holds the volume rather than any one
connection. It retries the real operation because a cheaper liveness probe is
itself a connection and re-triggers whatever refused it.
* feat: make the conformance suite capability-driven
The suite asserted a fixed Capabilities literal, so it could only ever pass
against a provider advertising exactly that set. Every implementation had to
match the suite rather than the suite testing the contract. RunSuite now reads
GetCapabilities from the provider under test and gates optional behaviour on
what it advertises, and the fixed literal is demoted to ReferenceCapabilities,
a convenience for callers constructing a MemoryProvider.
Add ReadOnlyPublish to Capabilities. A provider that leaves it false must
refuse a ReadOnly publish rather than hand back a writable export, which is
what viperblockd already does; it now has a way to say so up front.
Add ErrUnsupportedCapability and the unsupported_capability error code. A
request for optional behaviour a provider does not advertise is well formed,
so classifying it as invalid_argument told the caller to fix a request that
was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly.
Declared capabilities now match observed behaviour: qemunbdd advertises
SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with
nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises
neither: the nbdkit Go binding exposes no extents callback, and the mount path
never passes -r.
The external NBD client suite gains a sparse-extent subtest that checks the
export's advertised contexts against the capability, so SparseExtentReporting
is falsifiable from outside rather than taken on trust, and its read-only
subtest now branches on the capability instead of assuming refusal.
Cover the new branches with a MemoryProvider advertising no optional
capabilities and one advertising OnlineExpansion, the branch no provider we
ship exercises.
* refactor: rename SourceVolumeID to SourceSnapshotVolumeID
CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd
required it whenever SourceSnapshotID was set and used it to resolve the
snapshot's chunk references, so there it named the snapshot's origin volume.
MemoryProvider ignored it entirely. qemunbdd implemented it as an independent
volume-to-volume clone, which is what the name and doc comment implied. The
conformance suite could not catch the divergence because the reference
provider had no branch for the field at all.
The contract already has a field meaning "the volume this snapshot came from":
Snapshot.SourceVolumeID, which every provider populates. So the request field
is the same fact travelling back the other way, and its name was the problem.
Rename it to SourceSnapshotVolumeID, document it as the source snapshot's
origin volume, and state that it is required with SourceSnapshotID and
meaningless without it.
All three providers now enforce one rule: a source snapshot must be
accompanied by its origin volume, or the request is invalid_argument.
MemoryProvider additionally rejects an origin that does not match the
snapshot's own SourceVolumeID, so a wrong name fails instead of resolving
blocks against a volume the snapshot never came from. qemunbdd loses its
volume-to-volume clone branch, which implemented behaviour the contract no
longer offers and no caller ever requested.
The suite pins both the requirement and a full create-snapshot-restore round
trip, so the field is exercised end to end rather than only on the error path.
* feat: make the conformance suite runnable against a live provider
The suite had only ever run against MemoryProvider and qemunbdd, both of which
are discarded with the test binary. viperblockd, the provider production
actually uses, was the one implementation the suite never judged, because the
suite assumed each subtest got fresh state: hard-coded volume IDs collide with
an earlier run's leftovers on a store that keeps them, and a hard-coded node ID
addresses a node that does not exist.
SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records
every volume, snapshot and publication the suite creates and tears them down in
dependency order, logging rather than failing so a teardown problem cannot mask
the assertion a subtest was making. Two new tests cover this by running the
whole suite twice against one shared provider instance, which passes only
because cleanup works, and once per prefix to cover concurrent runs sharing a
daemon.
TestLive_ViperblockdConformance wires this to a real cluster behind
SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite.
* fix(viperblockd): close the four contract gaps the live conformance run found
Unpublishing a volume this node has no mount for now succeeds instead of
returning not_found, matching the legacy unmount path: nothing mounted here is
already the state the request asks for.
CreateSnapshot refuses a snapshot ID that already exists against a different
source volume with already_exists, rather than overwriting it.
DeleteVolume refuses a published volume with volume_in_use instead of tearing
the mount down, on this node or on whichever node answers the volume's owner
subject. The export and the guest writing through it survive the metadata
deletion, so the old behaviour surfaced later as corruption.
CreateVolume reads the source snapshot before writing the volume, returning
not_found when it is absent and invalid_argument when it was taken from a
different volume than the request names. Previously both produced a blank
volume and a successful response.
CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument
now that it carries a sentinel.
* feat(ebsprovider): pin the source snapshot's origin volume in the contract
The suite only required source_snapshot_volume_id to be present, so a caller
naming a volume the snapshot never came from was accepted everywhere. The
snapshot records its own origin, so this is checkable: creating from a snapshot
with a mismatched source volume is now invalid_argument for every provider,
enforced by qemunbdd against its snapshot metadata and by viperblockd against
the snapshot's persisted SourceVolumeName.
* test(ebsprovider): run the contract suite against viperblockd in CI
CI installed neither qemu nor nbdkit, so every conformance test that needs a
real implementation skipped silently and MemoryProvider was the only provider
the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and
sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails
instead of skipping.
TestViperblockdConformance stands the provider handlers up over an embedded
NATS and a real predastore and runs the same suite against them, which is what
caught CopySnapshot accepting a foreign volume's snapshot on unencrypted
volumes.
* feat(viperblockd): honour read-only publish and advertise block size constraints
ReadOnlyPublish was advertised false and read_only=true was rejected,
so the capability was true only by omission. nbdkit -r now starts the
export when the request asks for it, which sets the NBD read-only
transmission flag and makes the plugin refuse writes. Access mode is
fixed when nbdkit starts, so a republish asking for the other mode is
refused with VolumeInUse rather than handed the running export.
Block size constraints were absent entirely: the nbdkit Go binding has
no block-size callback, so the plugin cannot advertise them. The
blocksize-policy filter can, without a binding callback. Minimum 1
because viperblock's WriteAt handles unaligned requests with a
read-modify-write, preferred 4096 to match its block size, maximum 32M
to match what qemu-nbd advertises.
* test(conformance): judge sparse reporting by the allocation map, not the context
The subtest asserted that base:allocation was offered. nbdkit offers
that context whether or not the plugin can compute extents, answering
'all allocated' when it cannot, so the assertion passed for a server
with no extent support at all — it could not fail. What separates the
two is the answer: a volume with nothing written to it is entirely
holes to a server that really tracks allocation.
Read the map with nbdinfo --map --json and compare holes against the
advertised capability. qemunbdd reports holes and advertises true;
viperblockd reports none and advertises false.
* test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime
The predastore fixture is one daemon per test binary, but its cert was
exported with t.Setenv, which reverts when the first test that called
Start finishes. In-process clients did not notice because crypto/x509
caches the pool behind a sync.Once, loaded while the variable was still
set. Child processes did: nbdkit spawned by a later test inherited an
environment with no SSL_CERT_FILE and could not verify the daemon, so
the whole NBD client suite failed when the package ran as a whole while
passing when run alone.
Also install libnbd-bin in CI, so the NBD client suite runs there
instead of failing on missing tools.
* ci: install nbdkit-plugin-dev so the viperblock plugin can build
* test(volume): give the snapshot fixture an owner
CreateVolume now fails closed on a snapshot whose owner it cannot match,
including an unset owner_id. The fixture predates that and wrote none,
so the provider test failed with InvalidSnapshot.NotFound before ever
reaching the provider.
* test(integration): give the volume harness a real provider
The control plane no longer builds volumes itself, so CreateVolume in
the integration harness failed with "no EBS provider configured": the
harness wired a VolumeServiceImpl with nothing behind it. It now serves
viperblockd's provider subjects against the same predastore fixture, so
the blocks are still written by real viperblock to a real daemon.
Three persisted-state assertions moved with it. TenantID, VolumeType
and AvailabilityZone are control-plane facts, held in the ebsmetadata
document rather than viperblock's own state, so the storage oracle now
asserts their absence and DescribeVolumes -- whose answer is
account-scoped -- is what checks them.
* refactor: route image import through the EBS provider and drop the legacy metadata path
The control plane could still only run against viperblock: the AMI import and
the ochre weights import both linked the storage library directly, and the
ebsmetadata store kept a read fallback onto viperblock's config.json.
Import now goes through the provider contract with no new verb: CreateVolume,
PublishVolume, a bulk copy over the published NBD export with qemu-img, then
UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS,
whose request size the contract caps far below a disk image.
The migration scaffolding goes with it. ebsmetadata documents are now the only
thing the control plane reads: no legacy prefix scan, no config.json fallback,
no backfill on daemon start. config.json belongs to the provider, which is the
single writer of it.
ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is
right for DescribeImages, where one corrupt document must not hide the fleet,
and wrong for a name-uniqueness check, which cannot rule out a name it failed
to read.
spinifex and cmd no longer import viperblock outside services/viperblockd and
testutil/vbscan.
* feat: add a volume enumeration verb so a lost document cannot strand blocks
Making ebsmetadata documents the sole index of volumes left one failure with
no recovery: a volume whose document is gone disappears from DescribeVolumes,
from the leak reaper (which enumerates documents), and from the provider,
whose contract had no verb that asks what it holds.
ListVolumes closes that. It is capability-gated on VolumeEnumeration and
paginated, following CSI's optional LIST_VOLUMES rather than inventing a
shape. Refs are deliberately not Volumes: enumeration answers what exists,
and a provider must not open every volume's engine to say so.
Each implementation reads what actually exists rather than what it remembers.
qemunbdd walks its volumes directory, not its in-memory map, which does not
survive a restart; viperblockd walks the bucket's top-level prefixes. A
volume the provider has forgotten is precisely what a caller is looking for.
The consumer is a read-only orphan report: provider volumes minus documented
ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is
excused only when its base volume is documented, so a stranded pair is still
reported. Listing documents stays strict, because reporting a volume as
orphaned on the strength of a document that failed to decode would point an
operator at live data.
* fix: stop the orphan scan accusing storage of holding things it does not
A live env19 run reported three orphans against a healthy cluster: an AMI's
backing volume, a snapshot, and the cluster key store. All three were wrong,
and a report that cries wolf on a clean cluster is worse than no report.
Enumeration treated every top-level bucket prefix as a volume. Snapshots sit
beside volumes at that level and have a config of their own, so presence of a
config cannot separate them; the snap- name is what does, which is already how
the AMI removal path reads them back. keys/ and spinifex/ are not volumes at
all.
An AMI's blocks are a genuine provider volume, so enumeration keeps reporting
them. What was missing is on the reading side: an AMI is recorded as an image
document, not a volume one, so the scan now treats a documented image as
known. Listing images is strict for the same reason listing volumes is —
unreadable and absent are different answers, and only one means orphaned.
* fix: let an orphaned image's blocks be reclaimed
The orphan report tells an operator that reclaiming a stranded volume is their
decision, and images remove --force is the tool it points them at. On env19
that tool failed on exactly those volumes: removal drops the ebsmetadata
document first, and predastore answers NoSuchKey for a document that is
already gone, which is the definition of an orphan.
Deleting the document stays first and stays fatal. That ordering is the
barrier that stops a launch landing on blocks about to be deleted, and an AMI
still advertised after its blocks are gone is the worse failure. Only an
already-absent document is now tolerated: the barrier is satisfied, and the
blocks are the sole thing left to reclaim.
S3ObjectStore.DeleteObject normalises a missing key the way GetObject already
does, so this is recognised rather than matched on error text.
The obvious unit test passed without the fix, because the in-memory store is
silently tolerant on delete and real S3 answers 204 — neither behaves like the
backend this runs against. The test now drives a store that reports the miss,
and fails without the fix.
* feat: add per-verb OTel spans to the EBS provider contract
The ten provider verbs cross a single NATS seam and nothing measured it.
The only spans on the storage path were the legacy ebs.mount/ebs.unmount
pair, so control-plane storage latency could not be attributed to a verb
or compared across provider implementations.
NATSProvider.request is the one client-side funnel every verb passes
through, including both legs of an owner-subject fallback, so a single
span there covers the whole contract and measures what the control plane
experiences regardless of which provider answers.
Span names come from a closed verb set rather than the subject. Owner
and snapshot-create subjects embed a volume ID and the mount/unmount
subjects embed a node ID; naming spans after them would give every
volume a span name of its own. The IDs move to attributes and an
unrecognised subject collapses to a single unknown verb.
Trace context rides the message headers, so the serving provider joins
the caller's trace instead of rooting one of its own. The carrier is a
copy of the one in spinifex/utils rather than an import: that package
pulls in the aws-sdk and the cluster key store, which the wire contract
must not depend on.
Responses carry no shared accessor for their error field, so both sides
re-read just that field from the encoded reply, and only when the span
is recording. Without it a failed verb looks like a successful round
trip.
viperblockd serves the contract with its own handlers rather than
natsserve, so its handlers now take the per-message context and its
publish/unpublish pair drops the subject-named consumer span it opened
for itself.
* feat: add null and hostile providers, and a per-verb bench suite
The provider boundary made it possible to put something other than
viperblock behind the control plane, and nothing yet did.
nullprovider is the floor. It is not a new type: MemoryProvider already
answers every verb from a map and natsserve already serves any
EBSProvider, so the package names that composition and states what it
measures. On loopback with no storage behind it a round trip costs
around 195us against 74ns in process, which is the seam's own price and
is not attributable to any storage engine.
conformance.RunBenchSuite prices one round trip per verb and drives
every implementation through the same calls in the same order, so two
providers' numbers are comparable. It measures control operations only:
guest I/O never crosses these verbs.
hostile wraps any provider in one that misbehaves. Faults derive from
the seed and a per-target sequence rather than a global counter, so
concurrent callers on different volumes cannot reorder each other's
faults and a failure found deep into a soak run replays exactly. Error
and error-after-work are separate faults on purpose: the second reports
failure after the work has already happened, which is how an orphan is
made, and is what makes an enumeration scan the right oracle for a soak.
Enumeration itself is never faulted, since an oracle that can be told
what to say cannot falsify anything.
The soak asserts that every volume the provider still holds is
attributable to an injected fault. A leftover with nothing behind it is
the control plane losing track of storage on its own.
Two bugs were found by falsifying the new tests rather than by running
them. The FNV-derived draw clustered badly enough that two fault bands
were never reachable in 1200 draws, so the draw now goes through a
SplitMix64 finalizer. And the first version of the lie test was
worthless: MemoryProvider returns copies, so a lie could not corrupt it
whether or not the injector copied, and the test now drives a provider
that hands out pointers into its own state.
* test: bench the provider contract against qemunbdd and a live viperblockd
RunBenchSuite was only ever pointed at the null provider, so the numbers
priced the seam and nothing else. Point it at the two real implementations.
BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a
provider that stores nothing and wrong against one that does, so qemunbdd and
viperblockd are both measured at 64 MiB.
liveProvider splits into a testing.TB flavour. Benchmarks need the same
connection and the same node and cannot be handed a *testing.T.
* fix(viperblockd): carry the trace context into every viperblock engine call
Both S3 clients only emit a span when the request already carries one, so
calling the no-context Init, LoadState, SaveState, LoadBlockState and Close
left the whole engine open untraceable. Every provider verb traced as one flat
server span, and the 342ms describe measured on env19 could not be attributed
to anything.
viperblock already exposes the Ctx variants and Backend already declares
InitCtx, so this is threading, not new instrumentation. openVolumeVB,
openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine,
copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and
mount/unmount keep the context their own span derives instead of dropping it.
completeCreateSnapshot gets WithoutCancel: the background half of
accept-then-publish has to outlive the request it was accepted by and still be
attributable to it.
The per-volume ebs.config handler opens a consumer span like the other legacy
subjects, so a config update is no longer the one path with no trace at all.
* perf: describe a volume without opening an engine on it
describeVolumeEngine fell through to openVolumeVB for any volume not mounted
on the node serving the request, which constructed a full viperblock engine
to read one integer. On env19 that decomposed as a 22 ms reachability list, a
3 ms state read and a 16 ms PutObject of the volume's own config.json — the
write coming from the SeqNum bootstrap a read-write open performs.
Use viperblock's read-only state load instead. A describe is now the 3 ms
read: no background goroutines, no reachability probe, no SeqNum window
claimed, and no write issued by a worker that does not own the volume.
Falsified: restoring the openVolumeVB call fails
TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the
list, with the fake backend answering each so neither assertion hides behind
the other.
* perf: share one HTTP client across every viperblock backend
Four sites built the same S3Config by hand, each leaving HTTPClient unset, so
every engine open and state read got its own connection pool and paid its own
TLS handshake. That was invisible while Backend.Init issued a whole-bucket
list — the list paid for the connection and the state read rode it — and it
became the dominant cost once describe was reduced to a single request.
Route all four through cfg.volumeS3Config, which builds the client once.
Falsified: dropping the HTTPClient field takes
TestDescribeVolume_ReusesOneConnection from one connection to three.
* perf: wait for nbdkit to be ready instead of waiting a fixed second
mountVolume spawned nbdkit and then ran a confirm loop until a one-second
deadline passed. The loop only broke early when the process exited, so the
fast-failure path worked and the success path always cost the full second.
That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so
the timings never contradicted the assumption that it was process spawn.
Wait for the endpoint to accept a connection instead. That is also the
question the caller needs answered, since it is about to hand the URI to
something that will connect to it — where 'still alive after a second' only
correlates with it. A nbdkit that exits during startup still fails the mount
immediately, and one that never comes up now fails on a deadline rather than
being assumed healthy.
Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness
timing assertion and the fast-failure one.
* test: measure the data path, not just the control verbs
The conformance benchmarks price create, publish and describe. Guest
I/O never crosses those verbs, so none of those numbers say anything
about what a guest experiences. This drives the published export with
qemu-img bench, which sits on the same block layer virtio-blk does.
It asserts only that the I/O completes and logs a table: what a given
store is capable of is not a contract, and pinning a number here would
only produce a flaky test.
Read profiles stay inside the region the write profiles covered, or
they would be timing how fast the export answers with zeroes, and one
write profile flushes, because an unflushed write measures how fast the
WAL accepts data rather than how fast the store takes it.
Profiles are spaced apart because the plugin tears an engine down per
disconnect and removes the volume's local directory as it goes, so a
connection opened while the previous one is closing fails its
handshake.
* test: run data-path profiles back to back
The spacing was a workaround for a volume lock that did not hold across
close. With that fixed, consecutive profiles are worth more than the
workaround was: each is its own NBD connection, so a run that completes
has also shown that an arriving connection waits out the previous one's
close rather than racing it.
* fix: refuse a delete that cannot establish whether a volume is published
volumeMountedElsewhere returned false on every failure path, including a
request timeout, so a busy or partitioned owner was indistinguishable
from no owner at all and the delete proceeded. Deleting a published
volume leaves the nbdkit export and the guest writing through it alive
over storage that is going away.
Report the three outcomes separately. NATS distinguishes no-responders
from no-answer, so only the former means unmounted; an unanswered probe
is refused and the caller may retry once the owner is reachable.
* feat: exclude a second viperblock engine with a JetStream volume lease
Nothing in the stack stopped two viperblock engines opening one volume from
different nodes. For an encrypted volume that is not just a consistency risk:
the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte,
and the SeqNum window claim is a read-then-write with no CAS, so two engines
opening concurrently issue overlapping nonces under the same key.
Every engine open now takes a lease first, keyed by volume in a JetStream KV
bucket. A lease is a KV create, which JetStream serialises, so a second
claimant loses instead of both winning. The revision that won is the
generation: monotonic across the bucket, so a later opener always carries a
higher one than the writer it displaced. Entries carry a TTL and a renewal
loop, so a node that dies mid-mount does not wedge its volumes forever.
Losing a lease is logged, not enforced. Until the object store can refuse a
stale writer's PUT, failing live guest I/O on lease loss would add a failure
mode without stopping the zombie it is aimed at.
A daemon with no lease store refuses to open rather than opening blind, so
nbdkit recovery binds the store before it adopts surviving exports. Repeat
acquisitions on one node share a lease: the unmount seal opens a detached
engine while the mount being torn down still holds it, and cross-node
exclusion is what the lease is for. A restart reclaims this node's own stale
entries, since the exports that outlived the daemon are still running.
* refactor: consume masterkey from bluebottle in the test this branch added
* feat(ebsprovider): classify refusals a caller may retry
Every ErrorCode said what went wrong and none said whether the same
request could succeed later, so a caller could not tell a permanent
refusal from a transient one. Transient conditions shipped as internal,
which tells a caller to give up when it should retry.
Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(),
which callers branch on instead of on the code itself. DeleteVolume's
"could not establish whether the volume is published" refusal moves off
internal onto it: the owner that did not answer may answer later.
TestErrorCodeRetryable names every code, so adding one without deciding
its retry meaning fails rather than silently defaulting to permanent.
TestClassifyError pins the sentinel-to-wire-code mapping, which had no
coverage at all; a sentinel missing from classifyError falls through to
internal and loses both the reason and the retry signal.
* fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for
Four providers advertised the capability and nothing consumed it: there
is no clone verb on the interface. CreateVolume's SourceSnapshotID
restores from a snapshot and CopySnapshot duplicates snapshot metadata;
neither is a volume clone, and the field documenting SourceSnapshotID
says so explicitly. CSI expresses clone as a volume_content_source with
a volume source, which this contract does not implement.
Advertising it told a caller a feature was available and then gave them
no way to use it. Removed rather than backed by a method: adding a clone
verb is a feature, not a contract fix, and the capability can come back
alongside it. The Capabilities doc comment now states the rule, and
qemunbdd's stale comment about sparse extents is corrected while there.
* feat(ebsprovider): add ListSnapshots so snapshots are reconcilable
The contract could create, copy and delete snapshots but never say which
ones it held. That left the control-plane metadata document as the only
index of them, so losing one stranded the data it pointed at with no way
to find it again. This is the same exposure ListVolumes closed for
volumes, and it stayed invisible because ListSnapshots is optional in
CSI too.
ListSnapshots pages exactly as ListVolumes does, gated on a new
SnapshotEnumeration capability. SnapshotRef is deliberately not a
Snapshot: enumeration answers what exists, and reading a snapshot's size
or state can cost far more than listing it.
viperblockd walks the bucket's snap- prefixes, sharing the top-level
walk with ListVolumes. It leaves SourceVolumeID empty because the link
lives inside the snapshot's own encrypted state and reading it would
mean a fetch per snapshot listed. qemunbdd walks its snapshots
directory rather than its in-memory map, so a snapshot that outlived the
process still appears; its SourceVolumeID is empty in that case for the
same reason CreateSnapshot refuses to reuse the ID.
Five conformance subtests cover it, and they run green against
MemoryProvider, qemunbdd and viperblockd over a real predastore.
* feat(ebsprovider): state the exclusion guarantee in the contract
A caller could not tell which providers prevent two writers from holding
one volume, at what scope, or for how long. The guarantee existed in
viperblockd and was invisible from the outside, which is why the
conformance suite had no concurrency dimension at all.
Capabilities.Exclusion states it, and deliberately separates three
questions the contract used to run together: how far exclusion reaches
(none, node, cluster), whether a dead owner's claim ever expires, and
whether a writer that lost its claim is actually stopped. That last one
matters most and is the least flattering: exclusion refuses the second
opener, fencing stops the first. viperblockd advertises cluster scope
with a 45s TTL and FencesLostClaim false, because a partitioned node
whose lease lapsed keeps writing until the object store can refuse it.
Four new conformance subtests. Two required a second viperblockd node in
the harness: cluster exclusion cannot be exercised by one node, because
the second publish goes to a subject nobody serves and times out rather
than being refused. Both nodes share the NATS the leases live in and the
bucket the volumes live in, so a refusal comes from the lease.
Standing that second node up immediately found a real gap, in a test
that had been silently skipping for want of one. A publish refused by
another node's lease came back as internal rather than volume_in_use, so
a caller was told something had broken when the answer was that someone
else held the volume. Mapped in handlePublishVolume.
* build: pin viperblock to the release carrying the provider's dependencies
The branch has been building against a local replace pointing at a
viperblock worktree, so its committed pin never actually built: it named
a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch,
NewStateReader or s3.NewHTTPClient, all of which this branch calls.
Points at the merged viperblock dev instead and drops the replace.
GOWORK=off now builds clean, which is the deploy path ansible uses.
* fix: let viperblockd's delete sweep tolerate an object another sweep removed
deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and
give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it
returned on any DeleteObject error. Two overlapping deletes for one volume left
the loser reporting an internal failure once the winner had removed a chunk
between the list and the delete.
The control-plane copy that carried this fix is gone with the embedded engine
path, so the provider path reintroduced the bug it had just fixed.
* fix: stop CreateVolume leaving local state on whichever node served it
A state read prefers this node's local copy over the backend, and provider RPCs
are not owner-routed, so the node that happened to serve a create kept a copy
that answered describe as if the volume still existed after another node
deleted it. The two-node conformance harness caught it as DeleteVolume/success
failing about one run in three.
The seeded path already removed its local files after closing. The unseeded
path saved state and left them, which was the only difference between the two.
* fix: guard the mount path itself against a second export of one volume
handlePublishVolume refused a volume this node already exported, but
mountVolume did not, and the legacy ebs.<node>.mount subject is the route
production takes for instance launch and hot-attach. A retried attach racing a
fresh one started two real nbdkit against one volume, with no refusal and no
warning: the volume lease only refuses a different node, and a same-node
re-acquire finds the volume already held and increments its refcount.
That also made the advertised exclusion inaccurate. viperblockd claims
cluster-scoped exclusion on the grounds that a second opener is refused, while
the conformance subtests reach it only through PublishVolume, so they passed
with the legacy route unguarded. Moving the check into mountVolume gives both
entry points the same behaviour and makes the claim true for the path
production uses.
* fix: tolerate a missing legacy AMI config on deregister
DeregisterImage deletes the AMI document, then the pre-provider
{imageID}/config.json. Predastore answers a delete of an absent key with
NoSuchKey, and that was treated as fatal, so every AMI created since the
metadata store landed failed to deregister after its document was already
gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had
just been told it could not delete.
The regression test uses a store that answers a missing-key delete the way
Predastore does; MemoryObjectStore silently succeeds, which is why the unit
suite stayed green.
* fix: reap ENIs left behind by an abandoned instance launch
A launch creates the instance's ENI and then attaches it. When the daemon
dies in between, the record survives with no instance and nothing ever looks
at it again. The security group dependency check counts an ENI that lists the
group, so the group cannot be deleted for as long as the record exists.
The in-process launch path already rolls back on error; a process that dies
cannot. This adds a cluster-wide sweep over records that are launch-created,
never attached, and older than the launch RPC timeout by a wide margin. The
description prefix is now a shared constant because it is the only thing
separating an abandoned launch from a standalone ENI a caller asked for.
|
||
|
Julian Sommer
|
2a3d6de7d5 |
fix: pin bluebottle, predastore, viperblock and northstar to dev
The bare-checkout build on the CI runner resolves these from go.mod alone, where the tagged v1.16.0 releases still carry the pre-bluebottle masterkey and otelsetup layouts. |
||
|
Julian Sommer
|
8f930e6275 |
refactor: consume the shared packages from bluebottle
Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather than predastore, so the control plane no longer depends on the S3 server for shared auth and policy code. |