mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
dev
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ben Duncan
|
8ab96b29a1 |
fix: replicate every JetStream KV bucket across the cluster (#1117)
* fix: replicate every JetStream KV bucket across the cluster A KV bucket created without a replica count gets one replica, and one replica lives on one JetStream-chosen server recorded in no config. Losing, restarting or partitioning that server makes the bucket return "no responders available" from every node at once, so a single-node event becomes a cluster-wide outage. The reconciler leader-lease buckets were the worst case: acquiring a lease is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed. The cluster's node count is now declared once, in config.LoadConfig, and read by the one helper every bucket creation goes through. Every process that creates a bucket loads the cluster config first, so no service can forget to pass it and no signature has to carry it. An undeclared size is an error at creation rather than a silent fall back to one replica. Existing buckets are raised rather than left behind: a service opening one raises it, the daemon sweeps them all after formation, and "spx admin kv replicas --repair" does it on demand. Raising reads the live stream config and changes only the replica count, and never lowers one. Removes the ~30 replicas/clusterSize parameters that only forwarded a number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no clamp and would have failed outright on a cluster of six or more. Proved by a repo-wide invariant test pinning bucket creation to the one helper, a whole-cluster sweep in every multi-node nightly cell, and a new kvquorum suite (cell 33) that takes away the node leading the most buckets and requires the survivors to keep answering for all of them. * chore: name cell 33 in the nightly dispatch input * fix: a KV bucket that cannot be raised must still open The first version of the replica fix treated a raise it could not place as a hard error, and on a real three-node cluster that turned the fix into the outage it prevents. A node whose config named a fourth server wanted four replicas, NATS answered "no suitable peers for placement" for three, and the daemon retried for its full five-minute budget without finishing bootstrap. The bucket was present, quorate and serving on three nodes throughout. A config naming a node before that node serves is the ordinary middle of growing a cluster, so that is a state to tolerate rather than refuse. The line is now drawn between the two cases instead of around both. Creating a bucket below the cluster's replica count still fails, because that is the defect and a fallback there reintroduces it where it is hardest to notice. Failing to raise an existing bucket warns and hands the bucket back, and the raise is retried on every open besides the daemon sweep and --repair. This matters most in kvlease, which runs on every reconcile tick: refusing there would stop every reconciler sharing a lease bucket that was working. getOrCreateBucket now attaches before it creates. A create asking for more replicas than the cluster can place is refused before the server looks at the name, so with create-first "this bucket already exists" was an inference from which error came back, and the wrong one comes back on a single-node server. RaiseAllBucketReplicas attempts every bucket and joins the failures, and --repair keeps going too. Stopping at the first would leave the rest of a cluster short because one bucket could not be placed. Also adds nats.ServerName/NodeFromServerName beside the template that renders server_name, with a test tying them together. The kvquorum suite had mapped stream peers to nodes by comparing them to node names, which they are not: they are the node name behind a "spinifex-nats-" prefix, so nothing matched and the node-loss test could not choose a victim. * test: wait for the right NATS peer count after a node rejoins /routez reports a node's routes to its peers, not its own membership, so a fully formed three-node cluster is two peers per node. The kvquorum suite waited for three and spent its whole five-minute rejoin budget on a cluster that had already reformed. Every other caller passes len(Nodes)-1; the helper's doc comment now says so. The assertion the test exists for had already passed: node1 was leading 18 of 47 buckets when it was taken away, and node2 and node3 each still answered for all 47. * fix: replicate KV buckets across the largest odd count a cluster can place A bucket was replicated across min(nodes, 5) servers, which stores a fourth full copy on a four-node cluster and still survives exactly one loss — the same tolerance R3 gives for three quarters of the write amplification. A Raft majority needs an odd count, so the rule becomes the largest odd count the cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two entry points, so no caller can hold a stale copy of the policy and keep passing after the policy changes. Three retry loops could not tell a configuration fault from a cluster that had yet to form. An undeclared cluster size never becomes declared by waiting, so each of them spent its whole budget on one and then reported quorum as the cause. They now stop on a permanent error and say what is actually wrong. This is what the integration suite was hitting: it builds its gateway by hand and never loads a cluster config, so the size was never declared and the IAM retry sat in its sleep until the test timed out. Both ends are fixed. The audit reported configuration alone. A bucket at three replicas with one peer offline and one lagging is one failure from unavailable and read as ok. BucketReport now carries the leader and a count of the peers JetStream itself calls current and reachable, quorum is measured against that count, and spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem --repair fixes from one needing a human. Declaration is write-once: daemon.Start refuses a conflicting redeclaration because it is about to create buckets, and it declares before JetStream init rather than after it. config.LoadConfig warns instead, because it also reads config files that are not this process's own cluster. spx admin init --nodes 2 is refused — no replica count on two nodes both survives a loss and accepts writes — and single-node replication is asserted in the single e2e suite, so it runs in every nightly host permutation. * fix: withhold the vLLM serving image until its artifact is published The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names 404s on iso.mulgadc.com, so spx admin images import would fail for every operator who tried it. Its GPU siblings are published; this one never was. The entry moves to WithheldImages rather than being deleted, because the entry is the definition — the URL, the checksum and the two tags resolveServingAMI filters on. Publishing the artifact is then a move back, not a retype. Bedrock already handles the image not being registered: resolveServingAMI returns ErrServingAMINotFound, which is the honest answer while there is nothing to launch. The bedrock tag contract is still pinned, now against the map the definition lives in, and it additionally asserts the entry is not offered for import. The e2e reachability walk covers AvailableImages alone, so a withheld entry is out of it by construction. * fix: drop the cluster size from the call sites dev added Two constructors on dev still pass a replica count that no longer exists as a parameter: the cluster size is a declared process fact, so EKSServiceDeps has no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is textually clean and breaks the build, which is the shape of collision a signature removal always has. * docs: explain what an operator sees when a cluster server fails The replication section said how many copies a bucket has and stopped there, which leaves the question an operator actually has during an incident: one server is gone, what still works. It now covers the majority side carrying on, an isolated server refusing changes on purpose and why that is the behaviour you want, guests continuing to run while anything that has to be recorded does not, the risk of reading state from a server that is cut off, and a returning server catching up by itself. The surrounding prose reads from the operator's side rather than the implementation's: what you see and what to do, not which consensus protocol is underneath. |
||
|
joshsiv-mulga
|
5fa12ec358 |
fix(e2e): skip stopped node in multinode NATS peer poll
StopNode now stops spinifex-nats.service, so WaitNATSPeers polling that node's dead 127.0.0.1:8222 monitor port fails with curl exit 7 and never converges. Add WithSkipNodes to exclude the stopped node from the poll; TestMultinodeNodeFailure skips node2. |
||
|
Julian Sommer
|
50b5495d26 | chore: more comment cleanup | ||
|
Josh Siviour
|
bedc35344f |
feat(e2e/multinode) (#242)
* feat(e2e/harness): multinode cluster wait helpers + AWSClientForGateway (Stage A)
Stage A of mulga-siv-90. Compile-only; no tests wired yet.
multinode_cluster.go:
- Cluster.WaitNATSPeers polls /routez per node, dedupes by remote_id.
want=2 on healthy 3-node, =1 after a node-down event.
- Cluster.WaitPredastoreHealthy / WaitGatewayHealthy probe HTTPS per node
via insecureHTTPClient (multinode certs are self-signed across peers).
- Cluster.WaitDaemonReady runs DescribeInstanceTypes through every
node's gateway, not just env.ServiceIPs[0] — catches a daemon that's
only registered on one node.
multinode_aws_via.go:
- AWSClientForGateway clones *Env, swaps ServiceIPs to a single node's
Addr, returns a fresh AWSClient. Lets phases 6/7 issue describe/
stop/start across every node in turn (bash aws_via equivalent).
Init/Join helpers deferred to the pseudo-multinode bead (mulga-siv-91)
where they belong — real-multinode bootstrap is owned by tofu +
bootstrap-install.sh before the Go suite starts.
* feat(e2e/multinode): TestMultinodePreflight + TestMultinodeClusterHealth (Stage B)
Stage B of mulga-siv-90. First runnable Test* against a 3-node cluster.
multinode/main_test.go:
- Package-scoped singleton fixture, mirrors single/main_test.go.
- requireMultiNodeFixture skips when SPINIFEX_E2E unset or mode != multinode.
- Fixture = {Env, AWS, Harness, Cluster, Artifacts, TmpDir} — adds Cluster
to the single-node shape via harness.ClusterFromEnv (reads SPINIFEX_NODES
+ SPINIFEX_SSH_USER/KEY).
- TestMain drains harness.Fixture.Close + removes TmpDir at process exit.
multinode/helpers_test.go:
- needAZ / needInstanceTypeArch / needAMI shims (memoized via harness).
- readyNodeCount substring counter — matches bash `grep -c "Ready"`.
multinode/tests_test.go:
- Top-level Test* wrappers (TestMultinodePreflight, TestMultinodeClusterHealth).
multinode/preflight_test.go (phase 1):
- /dev/kvm writable check (no /dev/kvm = test environment is wrong, fail fast).
- peer_ssh hostname on every remote node. Skips node[0] to match bash —
the runner's loopback SSHd is not part of the contract.
multinode/cluster_health_test.go (phase 2):
- Cluster.WaitNATSPeers >= 2 per node.
- Cluster.WaitPredastoreHealthy / WaitGatewayHealthy across nodes.
- Cluster.WaitDaemonReady (DescribeInstanceTypes per gateway).
- spx get nodes shows N Ready, where N = cluster size.
- spx get vms returns (length only — content may be empty).
* feat(e2e/multinode): TestMultinodeInstanceDistribution + GuestSSH (Stage C)
Stage C of mulga-siv-90. Phase 3 (distribution) + phase 4 (guest SSH).
harness/multinode_distribution.go:
- InstanceHostingNode: SSH each peer, ps grep qemu-system | <id>.
Reuses NewPeerSSH; returns nil on no match (caller decides fatality).
- CountInstancesPerNode + AssertSpreadAcrossNodes: fan-out + minNodes
gate. Bash phase 3 warns-on-stacking; Go matches by accepting
minNodes=1 + caller-side log.
harness/multinode_guest_ssh.go:
- GuestSSHPort polls hosting node's qemu cmdline for hostfwd=tcp:...
reusing the existing hostfwdRE from ec2helpers.go (single regex, two
capture groups: host + port).
- GuestSSHEndpoint prefers PublicIpAddress, falls back to hosting-node
+ hostfwd. Mirrors the phase 4 conditional.
- GuestSSHReady polls `echo ready` via SSH with bash-equivalent -o flags.
multinode/helpers_test.go:
- needKeyPair (EnsureKeyPair shim).
- needInstanceTrio: pkg-level sync.Once launches 3 nano instances on
default VPC, registers terminate-on-process-exit. Phase 4+ reuse the
same trio without re-launch.
multinode/distribution_test.go (phase 3):
- launch trio + assert >= 1 node hosting (log warn on stacking).
- spx get vms contains every trio ID.
multinode/guest_ssh_test.go (phase 4):
- For each trio instance: resolve endpoint, poll SSH ready, assert id
reports ec2-user, log lsblk line count.
multinode/tests_test.go: TestMultinodeInstanceDistribution + TestMultinodeGuestSSH.
* feat(e2e/multinode): TestMultinode{Volume,CrossNodeGateway,CrossNodeOps} (Stage D)
Stage D of mulga-siv-90. Phases 5/6/7 — pure AWS API exercise (no
node-state churn).
volume_test.go (phase 5):
- Create 10GiB volume in discovered AZ, attach to trio[0] at /dev/sdf,
detach, delete. Inline lifecycle — DeleteVolume in-test would race a
registered Ensure* cleanup, surface as InvalidVolume.NotFound.
gateway_test.go (phase 6):
- Baseline DescribeInstances via the local gateway, then fan out
AWSClientForGateway per non-local node. Require.Equal across all —
cross-node fan-out regression surfaces immediately.
crossnode_ops_test.go (phase 7):
- Find trio[0]'s hosting node, stop via a different gateway, start via
a third (or same other on 2-node fallback). Returns to running so
later Test*s (phase 4 SSH, etc.) don't cascade-fail. otherTwoGateways
mirrors the bash fallback chain.
* feat(e2e/multinode): TestMultinodeNode{Failure,Recovery} (Stage E)
Stage E of mulga-siv-90. Phase 8 (failure) + phase 9 (recovery) +
diagnostic primitives.
harness/multinode_node_lifecycle.go:
- StopNode SSHes `sudo systemctl stop spinifex.target`. Non-fatal on
exit error (bash matches — cluster shutdown races the SSH conn).
- StartNode mirrors with sudo systemctl start.
- WaitNodeServiceReady polls a single node's HTTPS gateway. Distinct
from cluster-wide WaitGatewayHealthy so a recovering node can be
tracked without re-checking peers.
harness/multinode_diag.go:
- DumpAllNodeLogs fans out journalctl -u 'spinifex-*' per node into
artifactsDir/journal-<node>.log. Tolerates per-node SSH failure.
- SpxGetNodesAcrossCluster strips trailing whitespace off SpxGetNodes.
multinode/node_failure_test.go (phase 8):
- Stop node2, register t.Cleanup restart safety net.
- WaitNATSPeers want=1 (degraded), DescribeInstanceTypes via node1+node3,
DescribeInstances still returns the trio.
multinode/node_recovery_test.go (phase 9):
- StartNode node2 (idempotent — phase 9 in isolation still works).
- WaitNATSPeers want=2, WaitNodeServiceReady, spx get nodes 3 Ready,
DescribeInstanceTypes via the recovered gateway.
* feat(e2e/multinode): TestMultinodeVPCNetworking (Stage F)
Ports phase 10 from run-multinode-e2e.sh. Stands up a fresh non-default
VPC + subnet (10.200.0.0/16 + 10.200.1.0/24), launches 3 instances into
the subnet, then asserts:
- PrivateIpAddress is non-empty and matches 10.200.1.0/24
- All 3 IPs are unique
- SubnetId / VpcId on the instance match the freshly-created ids
- IPs persist across stop -> start cycles
Independent of the package-level trio (which uses the default VPC) —
this phase needs full control over subnet allocation so resources are
created and cleaned up inline via t.Cleanup (LIFO: instances ->
subnet -> VPC).
* feat(e2e/multinode): TestMultinodePlacementAndNAT (Stage G)
Ports phase 11 from run-multinode-e2e.sh — the largest single phase
(~340 bash lines covering spread placement + NAT Gateway egress). Eight
sub-steps preserved with matching harness.Step markers so the Go run
cross-references cleanly with the bash log:
Step 1: VPC + pub/priv subnets + IGW + main RT route + 2 SGs
(bastion: tcp/22 anywhere; private: tcp/22 from bastion-sg,
icmp from VPC)
Step 2: bastion launch, wait SSH, scp key
Step 3: spread placement group + 3 private instances via the group
Step 4: validate spread (>=2 unique hosting nodes; >=3 = full PASS)
Step 5: pre-NAT baseline — confirm NO internet from private VMs
Step 6: allocate EIP + create NAT GW + private RT + 0.0.0.0/0 -> NGW
Step 7: post-NAT — every private VM reaches 8.8.8.8 via NAT GW
Step 8: inline teardown of NAT GW + private RT + EIP
Each resource is owned by the test (no Ensure* memoization) so a failure
mid-flight tears the full VPC graph down via t.Cleanup in LIFO order,
matching the bash teardown sequence. NAT GW state polling is duplicated
locally (waitForNATGatewayStateFatal/Best); the equivalents in
single/natgw_test.go are unexported. Promote to harness/poll.go once a
third consumer lands.
* ci(e2e-go): add multi-node Go E2E job + cert/lb on 3-node cluster (Stage H)
Adds e2e_go_multinode as a parallel job alongside e2e_go_single on the
e2e-go workflow. The single-node job is untouched apart from extending
its SUITES filter to drop the multinode suite (which only runs on the
ci-multi cluster).
What the new job does:
- Provisions a 3-node tofu cluster via the same build-install-artifacts
+ bootstrap-install path as e2e_real_multi (bash).
- Builds multinode.test / cert.test / lb.test on the runner, ships
them to node1.
- Runs each suite with SPINIFEX_MODE=multinode and the cluster csv
in SPINIFEX_NODES / SPINIFEX_NODE_IPS. cert / lb branch internally
on env.Mode so the same binaries cover both single + multi.
- Pulls per-node journals (spinifex-journal-node1..N.log) for the
e2e-analyze action so a node2/3 failure isn't blind.
- Same JUnit + analyze + artifact-upload + status pipeline as the
single-node job.
Other moves:
- Default SUITES now includes e2e-multinode so push triggers fan
out across single + multi automatically. e2e_go_reboot stays
explicit-opt-in (unchanged).
- tests/e2e/Makefile: e2e-multinode now sets MODE=multinode so
`make e2e-multinode` doesn't fall back to single (which would
skip every TestMultinode* on the SPINIFEX_MODE check).
- tests/e2e/TEST_COVERAGE.md: cross-reference the Go port at the
head of the Real Multi-Node section. The bash driver stays in
place as executable spec until the Go run is soak-tested.
- e2e_go_status now needs e2e_go_multinode; multi=skipped counts
as success so a single-only dispatch still posts green.
* fix(e2e/multinode): NATS routez via peer_ssh + StartNode timeout + spx best-effort
CI run 26161792146 surfaced three regressions in the freshly-landed
multinode suite:
1. WaitNATSPeers dialled node.Addr:8222 directly. NATS monitor binds
127.0.0.1:8222 only (cmd/spinifex/cmd/templates/nats.conf:65), so
every routez poll returned "connection refused". Switched to
peer_ssh + curl http://127.0.0.1:8222/routez per node — matches
the bash run-multinode-e2e.sh phase 2 pattern.
2. StartNode / StopNode capped ssh at a 30s context. spinifex.target
has 6+ dependent units (predastore, NATS, awsgw, vpcd, daemon, ui)
and a clean systemctl start runs 60-90s; the 30s budget was
killing ssh mid-command ("signal: killed"). Bump StartNode to 5min
and StopNode to 3min.
3. SpxGetVMs is best-effort in bash (`spx get vms --timeout 5s
2>/dev/null` with no exit-code check) because the CLI's NATS dial
can race the cluster join shortly after bootstrap. The Go port
fatal'd on non-zero exit and broke TestMultinodeClusterHealth +
TestMultinodeInstanceDistribution downstream. Added
harness.SpxRunBestEffort and rewired both call sites.
* fix(e2e/multinode): spx CLI lenience + trio capacity retry + diag
Run 26163994815 cascaded from two issues that bash silently tolerated.
The Go port hard-failed where bash counts the failure and continues, so
3+ tests cascaded off transient first-failures.
- spx get nodes: cluster_health + node_recovery now downgrade <Ready
count to WARN with SpxRunBestEffort, matching bash phase 2
fail_test counter and phase 9 `|| echo ""`. Gateway+daemon health
checks above already prove end-to-end NATS routing.
- needInstanceTrio: retry RunInstances up to 6× on
InsufficientInstanceCapacity (10s backoff). Daemon resourceMgr can
briefly report 0 capacity right after node bootstrap before its
first inventory scan completes — AWS itself surfaces this
transiently. Stagger 2s between launches to match bash phase 3.
- CI workflow: new "Pre-flight diagnostics on node1" step dumps
groups/perms/spx output/systemctl status/journal tail to
preflight-diag.log so next run's artifacts surface what tf-user
sees before the test binary starts.
* fix(e2e): strip SPINIFEX_* from spx child env
Root cause of `spx get nodes` returning "nats: no servers available for
connection" in the multinode Go suite (mulga-siv-90 runs 26163994815,
26195522455): spx CLI calls viper.AutomaticEnv() with prefix "SPINIFEX"
inside config.LoadConfig. The CI workflow exports SPINIFEX_NODES (csv
of cluster WAN IPs) so harness.ClusterFromEnv can build the *Cluster.
viper picks SPINIFEX_NODES up as the `nodes` config key and silently
clobbers the loaded `[nodes.*]` map → nodeConfig.NATS.Host = "" →
nats.Connect("") → ErrNoServers.
Bash never set SPINIFEX_NODES so the bash port was unaffected. Direct
ssh `spx get nodes` from the runner also worked because the env was
clean. Only the test-binary-spawned spx hit the collision.
Fix: SpxRun / SpxRunBestEffort build a sanitized child env that
preserves the small SPINIFEX_* set spx's root command explicitly
BindEnvs (config path, AWS keys, host, base-dir, NATS dial overrides)
and drops everything else.
* fix(vpcd): make handleAddNAT idempotent (mulga-siv-124)
PR #232 (commit
|