mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
b505887b36a70bc6eebb2e3d97276d2ba35786e8
1572 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
b505887b36 | chore: add nilaway | ||
|
Julian Sommer
|
1d55404a73 |
revert: host DNS setup on resolved/resolvconf hosts (#580)
This reverts commit
|
||
|
Julian Sommer
|
5001a04b6a | chore: lint config | ||
|
Julian Sommer
|
f595129c7a | chore: enable SA1019 | ||
|
Julian Sommer
|
a36c6d819e | chore: bump image versions | ||
|
Josh Siviour
|
528bb1e483 |
fix(lbagent): quiet expected startup heartbeat failures below ERROR (#579)
A freshly launched LB agent begins heartbeating the instant the VM boots, but the gateway node fielding that first heartbeat may not yet have the LB record — it is still replicating from the node that created it — so the gateway answers 400 LoadBalancerNotFound. The IMDS credential chain is likewise not always warm on the first tick. Both self-heal within a few ticks, yet every early failure was logged at ERROR, flooding the console telemetry with startup noise (46 such lines across 8 agents in one run). Bound these to a registration grace window: before any heartbeat has ever succeeded, and only within the window, a failure logs at INFO. Past the window with no success — the fingerprint of a genuinely blackholed agent — or after any prior success, a failure logs at ERROR as before, so the serial-console telemetry still surfaces a real blackhole. The only change for a persistent failure is a bounded delay before it escalates. |
||
|
Josh Siviour
|
8989fc9ba8 |
fix: make the seal-failure unmount test deterministic (#578)
* fix: make the seal-failure unmount test deterministic The test forced the seal to fail by pointing S3Host at an unreachable endpoint, so the handler only returned once the S3 client had exhausted its jittered backoff. Measured handler time spanned 1.8s to 5.5s against a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and worse on a loaded CI runner. Config gains a sealVolume seam, nil meaning the real seal, and the test injects an immediate failure instead. It now runs in 0.53s, 50 for 50, and still asserts what it was written to assert -- plus that the handler actually attempted the seal, which the old form could not distinguish. * chore: bump viperblock to dev tip (GC observability + scratch sweep) Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info) and #48 (TestMain reclaims scratch dirs abandoned by a panicking run). |
||
|
Julian Sommer
|
a55544b7d4 | chore: remove integration from e2e | ||
|
Julian Sommer
|
f9c9f0b38d | fix: configure host DNS for Spinifex zones on resolved and resolvconf hosts (#580) | ||
|
joshsiv-mulga
|
d6b356b658 | Stop GPU E2E on push and merge | ||
|
joshsiv-mulga
|
e0259b7638 | update preflight | ||
|
Josh Siviour
|
88b57b1805 |
chore: give the integration tier its own CI job and drop it from preflight (#577)
The tier ran inside the Build and Test job, serialised ahead of test-cover, and again in preflight. Neither placement bought signal that a parallel job does not: it provisions nothing and finishes in seconds, so the only effect was wall time on the two paths developers wait for most. It stays PR-blocking as its own job, which is what distinguishes it from e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so never gates a PR before merge. |
||
|
Josh Siviour
|
318e43c66f |
fix: allocate mgmt IPs cluster-wide via NATS KV CAS (#576)
MgmtIPAllocator allocated from a process-local map, but br-mgmt is trunked onto a shared physical NIC making the /24 one flat L2 segment across the whole cluster. Every node independently allocated from .10, causing collisions and ARP-race losers being blackholed. Rewrite the allocator on the spinifex-cluster-state NATS KV bucket using compare-and-swap, keyed by mgmt-ipam.<subnet>, modeled on IPAM. Bind the KV handle in startCluster (not startLocal) to respect the DDIL boundary. Serve already-allocated instance IDs from the local cache when KV is unhealthy, but refuse new allocations until it recovers. Rebuild now reconciles this node's running VMs into the cluster-wide record instead of just refreshing the local cache. |
||
|
Josh Siviour
|
7e8cbeef6e |
feat(config): enable viperblock chunk GC in rendered node configs (#563)
* feat(config): enable viperblock chunk GC in rendered node configs The gc_enabled toggle was plumbed through every VB construction site but nothing ever set it. GCEnabled is a *bool that every consumer treats as off when nil, and the config template rendered shardwal while omitting gc_enabled, so no deployed node ever carried the key and chunk GC was off everywhere by construction. Superseded chunks were never reclaimed under overwrite churn. Render gc_enabled = true alongside shardwal, and cover it with a test that loads the rendered file through config.LoadConfig rather than matching the template text. A string match would still pass if the value failed to parse into a non-nil pointer, which is the exact way the setting reaches its consumers. Existing deployments keep the old rendering until their config is regenerated; the template governs install and bootstrap only. * chore: pin viperblock to the build with the cross-process GC snapshot marker Enabling chunk GC in rendered node configs is only safe once the sweeping engine can see a snapshot taken by another process. viperblock 3b56d61 adds the per-volume snapshot marker that closes that window, and 2a34add fixes the concurrent-stop panic in the background goroutine lifecycle that GC's own ticker would otherwise widen. Pin both before turning GC on. |
||
|
Josh Siviour
|
d2efbee043 |
fix(daemon): fail the system-instance launch when the VM never reaches running (#574)
LaunchRunInstances is a best-effort batch launcher that reports failure only through VM state: volume preparation, GPU claim and vmMgr.Run errors each call MarkFailed and continue. launchAMISystemInstance ignored that and always returned a SystemInstanceOutput, so an EKS control-plane server, ELB or Bedrock serving VM whose launch failed handed its caller an instance ID for a VM that was already being torn down. Check the instance reached running state before returning. Cleanup stays with the MarkFailed chain LaunchRunInstances has already run. |
||
|
Josh Siviour
|
344a490a70 |
fix(vm): give relaunched data volumes named block nodes so detach can complete (#575)
buildDrives emitted every non-boot data volume as a bare -drive with no id= and no node-name=, so a stop/start (or reboot/resize) relaunch left the volume anonymous: device_del/blockdev-del had nothing to address, QEMU kept the NBD socket open, and DetachVolume stalled through nbdkit's full kill grace before giving up. Add -blockdev support to the QEMU config builder and use it for non-boot, non-EFI volumes, sharing the node-name/device-id/iothread-id naming and virtio-blk-pci device arguments with AttachVolume's hot-plug path so cold-boot and hot-attach can no longer drift apart. A data volume with no persisted HotplugPort now gets one allocated and written back during buildDrives instead of landing on an unplugged bus. Also raise unmountSealTimeout past viperblockd's kill grace (it runs the seal after the SIGKILL, so matching the grace meant an unmount that used the whole grace could never be observed as successful), and bound the e2e harness's forced teardown detach by the teardown deadline so a slow detach is reported as slow rather than as an unexplained leak. |
||
|
Josh Siviour
|
62e7f983ee |
test: assert the gateway->vpcd->OVN crossing in the integration tier (#572)
* test: assert the gateway->vpcd->OVN crossing in the integration tier
The tier already ran the real VPCServiceImpl for the AWS-facing half, but
nothing consumed the vpc.* events it published: the SG topics were answered
by a canned {"success":true} and the fire-and-forget topology topics were
dropped. Both sides of that wire were tested only against stubs, so the join
was asserted nowhere below the live e2e tier.
Wire a real subscribers.Subscriber over a real in-process OVN Northbound DB
(ovntest.StartNB), opt-in per test so only tests that assert OVN state pay
the startup cost. StartDaemonLite gains WithRealVPCD to suppress its stubs,
since a stub and the subscriber both answering one request-reply would race.
Scope is the crossing, not NB row shapes — those stay with the reconcile
scenario tests. Everything host-dependent (ovn-controller, OVS, taps, netns,
nftables, packet paths) remains the live tier's claim.
* fix: make the integration tier re-runnable and quiet the ensure probe
Two defects the vpcd/OVN work surfaced.
The tier was green at -count=1 and red package-wide at -count=2. Test
accounts were keyed by t.Name(), which is unique among tests but repeats
across passes in one TestMain, so a second pass inherited the first pass's
JetStream namespace — including an ECR signing key sealed with a master key
StartGateway regenerates per call. Key each account off a counter instead.
TestRunInstances_ClientTokenIdempotency then collided with itself for the
same reason: its dedup store outlives a pass, so its fixed ClientToken made
the second pass's opening launch look like a replay. Suffix the tokens per
invocation.
ensureNamedRowOps' wait-op is built to fail fast when the row it guards
already exists, which is the ordinary idempotent path, but transactOps
reported it as "OVSDB operation failed" at ERROR. Every re-ensure emitted
one, drowning real faults in the log sink. Classify that one result as the
probe outcome it is and log it at Debug.
* test: cover transactOps' failed-result paths
The classifier split one log line into two branches and neither was
exercised: transactOps sat at 31% because EnsureLogicalRouter's cache
lookup absorbs a re-ensure before it ever transacts, so no unit test
reached a failed operation result.
Drive both kinds directly against a real NB — the guard probe finding an
existing row, and two same-named inserts violating the schema's name
index — since only the second is a fault and they now log at different
levels. transactOps goes to 94%; the remainder is the transport-error
return, which needs a broken connection to reach.
|
||
|
Josh Siviour
|
68abc0bfb3 |
fix(ci): scope e2e telemetry per cell/job and define suite topology eligibility once (#573)
* feat(ci): extend per-cell telemetry scoping to every e2e workflow ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil and gpu on their single-job workflows. The best-effort numeric job id lookup moves out of e2e-nightly into a shared composite action, so the four other provisioning jobs get ci.job_id without copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather than interpolating the name into the jq filter, which broke on names carrying quotes or non-ASCII separators, and no-ops when telemetry is disabled. gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job bakes the attrs into a daemon, so the API call would be paid for an env var no exporter reads. * fix(ci): define e2e suite topology eligibility once Three workflows carried three different answers to which suites a topology can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where every internet-facing subtest skips for want of a peer, so a fraction of the suite reported as a green lb run. e2e-suites' multi leg ran multinode alone, dropping the lb and cert coverage that only a multi-node env can complete — so full lb coverage never ran there at all. .github/scripts/suite-sets.sh is now the single definition, sourced by all three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives both matrices from it (multi_provision gains the suite-matrix step single_provision already had), and e2e-nightly's deliberately narrower per-cell lists are declared there too, guarded by an assert that fails the cell if a subset ever names a suite its topology cannot fully exercise. The nightly stays narrow on purpose: its cells have a 35-minute budget and exist to prove each install/network/host-OS permutation boots and serves. Suite breadth is the PR gate's job and already runs on every push. That is now stated in one place rather than implied by a third divergent list. |
||
|
Josh Siviour
|
cae76ba1b6 |
test: complete the e2e live-vs-mock rebalance (epic mulga-8oyg3) (#566)
* test(volume): assert chunk GC stays wired on in CreateVolume GCEnabled was zero-valued at every viperblock construction site for a long time and no test noticed. It is not part of viperblock.VBState, so it never round-trips through a saved config.json and cannot be observed by reading volume state back. Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so the GC wiring is assertable directly against the returned config, without a live viperblock or S3 backend. The config is returned by pointer because viperblock.VB carries atomic.Uint64 fields that vet's copylocks check rightly refuses to see copied. * test(e2e): snapshot backend byte accounting before volume purge DeleteVolume purges a volume's entire backend prefix, including any chunks a leak left behind, before anything measures it. That is the same masking that hid the storage-growth leak in production: a passing E2E cannot fail on backend growth when the evidence is deleted first. Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup closure, immediately before DeleteVolume, so every volume the shared teardown path purges is measured rather than silently discarded. The snapshot is always taken and logged; the hard ratio check additionally runs for callers that registered a guest-written-bytes baseline, since most callers never attach or write to the volume they create and so have no baseline to compare against. The tolerance is the RS(2,1) erasure floor widened for per-chunk header and tail-chunk overhead. It is sized to distinguish unbounded growth from a footprint settling at the floor, not to catch bounded residual garbage. The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit would abort the DeleteVolume call queued behind it and turn a reported leak into a real one. * test(e2e): add memory-constrained concurrent-write durability regression The existing durability tests write one 4MiB sentinel, sequentially, on a comfortably sized guest, which is the exact inverse of the known-open data-loss bug: acked writes read back zero-filled after a cold reread when a guest is memory-starved during concurrent writeback. No happy-path test can reach it, because the fault lives in the intersection of memory pressure, write concurrency and cache-cold reread. Add a suite that encodes those three conditions together: a guest discovered at or below the working set, N parallel large writers launched as background jobs in a single command so they genuinely overlap, swap disabled, then sync and drop_caches before rereading. Assertions are per stream rather than aggregate so a failure names the losing writer instead of leaving whoever reads it to re-derive which of N writers lost data from one combined digest. GuestExec gains a timeout-parameterised variant. Its fixed two-minute default is sized for sub-second payloads, and a multi-gigabyte write to a network-backed volume would exceed it, so the suite would have failed by timeout while looking like it had caught the corruption. The budget scales with the working set, and a deadline is reported distinctly from a data mismatch so the two red states are never confused. The suite is dispatchable but deliberately absent from the default SUITES: the bug it reproduces is still open, so it is expected to fail and must not sit in a merge-gating path. It is registered in the workflow allow-list so it stays discoverable rather than being buried behind a skip. Also record on the live IAM instance-profile test why its assertions were not moved down to the integration tier, so the trace is not repeated. * test(e2e): drop backend byte accounting, assert a node-wide delta instead The accounting helper listed the internal predastore bucket to attribute bytes per volume. That listing is refused unconditionally: predastore authorises bucket access by same-account ownership, and the bucket belongs to the service account, so a test caller gets 403 no matter what it holds. Granting the test a service credential would invert the trust boundary to buy a measurement, which is the wrong trade. Replace it with what the S3 surface actually permits: sample node-wide allocated bytes either side of the write and bound the delta against the bytes the guest wrote. Coarser, but it measures something real rather than depending on a call that cannot succeed. The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x expansion floor plus headroom; a cluster running the recommended RS(3,2) has a 1.667x floor and only ~12% of margin left, so the constant needs revisiting if the encoding changes. * revert: drop memdurability e2e suite The suite asserted the guest read back what it wrote after a memory-constrained concurrent write, but the assertion could not fail for the reason it claimed: it compared the guest's own read against the guest's own write with the page cache still warm, so it passed whether or not the data reached the backend. A test that is green on a build with a known-open durability bug is worse than no test — it advertises coverage that does not exist. Removes the suite and its Makefile/workflow wiring, plus DiscoverInstanceTypeAtLeastMemory, which had no other caller. GuestExecTimeout stays: GuestExec now delegates to it, and the old two-minute hard cap it replaced was a latent trap for any long guest command. * test(e2e): add partial-block concurrent-write durability suite Replaces the removed memdurability suite, which passed on a build that still carried the bug it claimed to test. The engine reaches its read-modify-write path only for a partial block; a fully covered block takes a full-overwrite branch that reads nothing and can lose nothing. The old workload wrote through ext4 with 1 MiB buffers, which the filesystem coalesced into aligned full-block requests: block-layer tracing recorded zero unaligned and zero sub-4096 requests, so the path under test never ran, and writers 4 MiB apart could not collide on a block even in principle. This suite writes the shape the defect needs. Two writers issue concurrent O_DIRECT writes into disjoint halves of the same 4096-byte blocks on a raw, unformatted device, stamping every half with the round number, then drop the page cache and reread with O_DIRECT asserting every half carries the final round. One half reading back at an earlier round is the signature of the lost update. The old suite also could not fail even had it reached the path: its write phase ended in 'wait; sync; sha256sum' and reported sha256sum's exit status, while bare wait returns 0 unconditionally, so writer errors were invisible and a backend ENOSPC presented as a pass. Here each writer's exit status and exact byte count are asserted before any durability claim is made. Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on teardown; the predecessor left roughly 3 GiB of backend per run and twice drove a node under the nearfull watermark. Dispatchable but not in the default suite list until a green run and a demonstrated fail-first against a pre-fix engine exist. * test(e2e): let the fixture own the working-set volume delete The suite deleted the volume in its own cleanup and the fixture deleted it again on close, so every run ended with an InvalidVolume.NotFound from the second attempt. Detach stays — DeleteVolume against a still attached volume is what leaks volumes on teardown — and the delete is left to the fixture so the volume is released exactly once. * test(e2e): gate merges on the partial-block durability suite Verified on one node with the same test binary and workload, varying only the engine: 1758 of 8192 halves reverted a generation against the pre-fix build, 0 of 8192 against the fixed one. That is the fail-first evidence the suite needed before it could be trusted to gate anything. Adding it to the default suite list rather than leaving it dispatchable is the point. Its predecessor sat outside every path that runs, which is how it survived long enough to be discovered green on a build that still lost data. * test(integration): cover the ECR data plane in the front gate The live ecr/ suite drove docker, crane and skopeo against a real registry, yet none of it boots a guest: it needs a registry, an object store and an HTTP client. The in-process gateway already serves the OCI Distribution v2 surface on an ephemeral port with a memory-backed object store, so these run here in 0.18s instead of behind full provisioning. Covers the three claims the live tests made beyond authorization, which is already down here: a push/pull round trip asserting blobs and manifests read back byte-identical, refusal to auto-create a repository on push, and retagging an existing manifest without disturbing the original tag. Content is compared rather than status codes alone. A registry that stores a blob under the wrong digest, truncates it, or serves a different one answers 200 to every request in the sequence. The capacity/load leg deliberately stays live: it exists to put real parallel pressure on a real predastore, and a memory-backed object store cannot make that claim. * fix(e2e): make volume teardown release the volume before deleting it Fixture cleanups drain LIFO, so a suite that creates its instance before its volume tears the volume down first, while it is still attached. DeleteVolume answered VolumeInUse, nothing retried, and the volume was stranded on the node. Reordering the chain cannot fix this — registration order has no way to express the instance/volume dependency. Instead teardown is now self-sufficient for any creation order: it force-detaches, waits for the volume to reach available, and only then deletes. Instance teardown waits for terminated for the same reason, since TerminateInstances returns as soon as the request is accepted. Two things kept the leak invisible. A failing cleanup was logged and forgotten, so a run stayed green with resources left behind; failures are now reported as leaks, failing the bound test or surfacing through Close() for TestMain to fail the run. And a panic in one cleanup aborted the whole chain, stranding every resource behind it; each callback is now isolated. RegisterVolumeTeardown and the fixture had grown separate teardown paths, only one of which detached. Both now share it. |
||
|
Julian Sommer
|
c054b0213e | fix(ec2): render ED25519 key fingerprints as EC2 does (#571) | ||
|
Julian Sommer
|
a3885db916 | fix(ec2): fingerprint key pairs over the encodings AWS hashes (#570) | ||
|
Julian Sommer
|
369c237f01 | refactor(gateway): parse AWS query bodies with url.ParseQuery (#569) | ||
|
Julian Sommer
|
2119981924 | refactor(ec2): use x/crypto/ssh for key parsing and fingerprints (#567) | ||
|
Josh Siviour
|
5fd74a8212 |
fix(eks): don't baseline a nodegroup's Ready-gate on its prior incarnation (#568)
Re-creating a nodegroup of the same name could never reach ACTIVE. The Ready-gate took its baseline from the live Ready count for that name, so a terraform replace after CREATE_FAILED — which deletes then re-creates — captured the previous incarnation's workers, demanded old+new, then watched the count fall as the old workers were terminated. The target was unreachable by construction. Observed on env19 with a perfectly healthy cluster: three new workers all joined and the create still failed with reports 3 Ready nodes, want >= 5 (baseline 2 + 3 workers) and each retry raised the bar again (baseline 3 => want 6), so the only way out was untainting the resource by hand. A create means every worker under that name is new, so the gate is just "desired Ready nodes carrying this nodegroup label" — baseline 0. The trade-off is stated in the comment: a stale incarnation's Ready nodes can briefly inflate the count and satisfy the gate early, but they go NotReady within about a minute of termination, whereas baselining on them fails every replacement permanently. The test models the real sequence — delete, stale Ready count still reported, re-create — and fails without the fix with the same shape as the field error. |
||
|
Josh Siviour
|
291e550e8a |
fix(dhcp): honour the caller's retransmission budget and encode option 61 correctly (#565)
* fix(dhcp): derive nclient4 read deadline from the caller's context
nclient4 races its own read deadline against the caller's context and the
shorter one ends the attempt. Acquire and Renew passed a fixed 5s, so every
window in Manager's 4s/8s/16s/32s retransmission schedule past the first was
truncated to 5s: the client listened for ~19s of its 90s budget and a server
that went quiet for longer was never waited out.
Measured in the sink, the split was total — attempt 1 (jittered under 5s)
always failed with the caller's context deadline, attempts 2 to 4 always
failed with nclient4's ErrNoResponse — and only 27% of clients that lost
attempt 1 were rescued by attempt 2.
Track the context's remaining time instead, keeping the configured value as
the fallback its doc comment already describes.
* fix(dhcp): keep the socket deadline strictly behind the caller's
Setting the nclient4 read deadline to exactly the context's remaining time
makes the two expire together, so a timed-out attempt reports either
context.DeadlineExceeded or nclient4's ErrNoResponse depending on which
goroutine wins. Observed live on a settled bridge: three attempts reported the
context deadline and the fourth reported ErrNoResponse, which reads as a
packet-matching fault rather than the plain timeout it is.
Give the socket a second of grace so ctx.Done() is always what fires.
* fix(dhcp): encode option 61 with its hardware-type byte
The client identifier was sent as a bare string, so the server read the
identifier's first character as the hardware type — "dhcp-gw-lrp-vpc-x"
arrived as type 100 ('d') with the identifier truncated. Since that is not
Ethernet, the server recorded no hardware address, and the upstream lease
table listed our leases with a MAC of all zeroes.
Send the chaddr typed as Ethernet, falling back to a type-0 identifier when
no hardware address is available. RELEASE now encodes identically, since the
server matches a release to a lease by client identifier.
Option 61 no longer carries the readable id, so restore it via hostname and
vendor class, which were specified but never populated. dhcptest builds its
options through the same helper rather than its own copy, which had drifted
into reproducing the broken encoding.
* fix(dhcptest): derive the probe MAC the way vpcd does
The probe carried its own copy of the MAC derivation, still on the 24-bit
02:00:00:xx:xx:xx hash that vpcd moved off when widening it to a SHA-256
prefix. So --spinifex-id leased under a MAC the daemon would never use:
"gateway-wan1" resolved to 02:00:00:7c:b7:b6 here and 02:54:df:71:22:5c in
vpcd, which makes the probe unable to reproduce a daemon lease at all.
Call DeriveMAC instead and drop the local copy.
|
||
|
Julian Sommer
|
826a441beb | chore: use stdlib sort + ip conversion | ||
|
Josh Siviour
|
2e7136c893 |
feat(ci): stamp per-cell scoping labels on e2e telemetry (#564)
The nightly matrix runs every cell under a single ci.run_id, and each cell names its hosts node1/node2/node3, so telemetry from all cells collapses into one undifferentiated bucket in the ES sink. Root-causing a single failed cell means aggregating across every other cell in the run. Stamp ci.cell and ci.matrix alongside the existing ci.run_id. Workflow-level env cannot reference the matrix context, so the attrs are overridden in each matrixed job's own env block, leaving the workflow-level value as the non-matrix fallback. The numeric GitHub job id is not exposed in any context (github.job is the YAML key, not the id), so resolve it best-effort via the jobs API and append it if found. The lookup never fails the job. e2e-suites can only distinguish single from multi: daemon telemetry is baked into the VM at provision time, before the suite matrix exists. Baremetal shipped no telemetry at all — neither the endpoint nor the attrs crossed the workflow_call boundary. Pass them through as an explicit input so the cell reaches the sink for the first time. |
||
|
Josh Siviour
|
4793b5ac09 |
fix(eks): node route MTU blackholing image pulls, and teardown wedged in DELETING (#562)
* fix(eks-node): stop the DHCP MTU overriding the pinned link MTU
Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which
honours DHCP option 26 and stamps the advertised MTU onto every route it
installs:
default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \
metric 1002 mtu 1442
Linux prefers a route's MTU metric over the device MTU, so the node kept
advertising an MSS derived from 1442 while the real path MTU through the
NAT gateway is 1408. Small packets pass and full-MSS inbound segments —
a TLS ServerHello with its certificate chain, image layers — are
silently dropped. It presents as `net/http: TLS handshake timeout` on
image pulls, not as anything MTU-shaped, and it only affects the host
netns, so containerd pulls fail while pod traffic sized off cni0 is
fine. Per-destination PMTU exceptions expiring in and out of the route
cache made it look intermittent and host-specific.
Three changes, innermost first:
- dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a
route MTU. mulga-vpc-mtu is the single source of truth for the MTU
of these nodes.
- mulga-vpc-mtu re-stamps any route that already carries a different
MTU, covering an image built before this fix or a dhcpcd already
running. Routes with no MTU metric inherit the device MTU and are
left alone.
- the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and
ignores the IPsec ESP overhead on the NAT egress path, so it
overstates the usable MTU for every guest, not just these nodes.
Also corrects the header comment, which claimed Alpine's udhcpc ignores
option 26 — this image runs dhcpcd, which honours it. That wrong premise
is why the route metric was never considered.
Adds a stubbed-`ip` test covering the rewrite, the dev token that
`ip route show dev X` omits, idempotency, and routes without an MTU
metric.
* fix(eks): sweep the CP VPC by identity so teardown can finish
DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs
by role tag. A create that fails partway — the first control-plane
server failing to launch, say — leaves resources behind that never got
tagged, so the sweep cannot see them.
DeleteVpc rejects while any subnet or non-default SG remains, and
teardown is re-driven on a timer, so the cluster sits in DELETING
forever retrying the same doomed delete every couple of minutes:
eks-deleting: re-driving wedged DELETING teardown cluster=toc
eks-deleting: re-drive teardown failed
err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation"
Nothing converges: the cluster name cannot be reused and terraform's
delete wait can never pass. Observed on env19, wedged for ~11 hours
until this change went in, at which point the next re-drive cleared it.
After the tagged sweep, clear anything still in the VPC scoped by
vpc-id: subnets, and non-default SGs (the default SG goes with the VPC
cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG
provisioner — nil is tolerated, since only teardown uses it.
* chore: pin viperblock + predastore to the merged storage fixes
viperblock f8cfd77 (#42) makes read-modify-write atomic per block,
removing the lost update that silently reverted a 4096-byte block to a
previous generation of itself. Confirmed as the env19 corruption cause
by an A/B: the pre-fix build corrupted image layers within ~15 minutes
(failed to extract layer / invalid tar header / unpigz: corrupted) while
the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW
conflicts on the worker root disks.
predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore
restart on a store whose metadata set outgrew one badger transaction
fails with "Txn is too big to fit into one request" on every node at
once, taking the metadata plane — and with it the AMI catalogue and
cluster creation — down permanently.
Both were verified in the built artifact rather than from go.mod alone:
go version -m reports the new pseudo-versions, and the rmw_conflicts
counter plus the batched-restore error strings are present in the
binary. Pinning by go.mod while the submodule pointer says otherwise is
how an earlier analysis ended up reading dead code.
* test: record the corrected DHCP MTU in the kitchen-sink golden
The NB golden pins the DHCP options published per subnet, including the
advertised MTU. Dropping the subnet option from 1442 to 1408 changes
what the reconciler writes, so the golden diverged and the live scenario
test failed — the test doing exactly its job.
Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408
on the two subnet DHCP records; nothing else in the snapshot moves.
|
||
|
Julian Sommer
|
3e0aab174a | refactor: migrate instance describe handlers (#561) | ||
|
Julian Sommer
|
10ce0fba8f | fix: resolve wrapped AWS error codes (#560) | ||
|
Julian Sommer
|
77c0e337a6 | fix: return Placement and Spot lineage for stopped and terminated instances (#558) | ||
|
Julian Sommer
|
4274184ba4 | refactor(daemon): collapse one-line NATS handler wrappers into the subscription table (#557) | ||
|
joshsiv-mulga
|
220141b145 | release: update go.mod for v1.13.0 v1.13.0 | ||
|
tomnewton-mulga
|
4670d09d58 | chore: RTX-6000 Supermicro demo doc draft | ||
|
Josh Siviour
|
48749576c3 |
test: rebalance E2E into in-process integration tier + shared-guest live scenarios (#556)
* test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — revert |
||
|
dependabot[bot]
|
8c8dc173bd |
chore: bump the npm-dependencies group (#554)
Bumps the npm-dependencies group in /spinifex/services/spinifexui/frontend with 21 updates: | Package | From | To | | --- | --- | --- | | [@aws-sdk/client-acm](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-acm) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-ec2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ec2) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-ecr](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecr) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-ecs](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecs) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-eks](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-eks) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-elastic-load-balancing-v2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-elastic-load-balancing-v2) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-iam](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-iam) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-s3](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-s3) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/client-sts](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-sts) | `3.1084.0` | `3.1089.0` | | [@aws-sdk/lib-storage](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/lib/lib-storage) | `3.1084.0` | `3.1089.0` | | [@smithy/protocol-http](https://github.com/smithy-lang/smithy-typescript/tree/HEAD/packages/protocol-http) | `5.5.7` | `5.5.10` | | [@smithy/signature-v4](https://github.com/smithy-lang/smithy-typescript/tree/HEAD/packages/signature-v4) | `5.6.3` | `5.6.6` | | [@tanstack/react-router](https://github.com/TanStack/router/tree/HEAD/packages/react-router) | `1.170.17` | `1.170.18` | | [@tailwindcss/vite](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite) | `4.3.2` | `4.3.3` | | [@tanstack/router-plugin](https://github.com/TanStack/router/tree/HEAD/packages/router-plugin) | `1.168.19` | `1.168.20` | | [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) | `0.58.0` | `0.59.0` | | [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) | `1.73.0` | `1.74.0` | | [oxlint-tsgolint](https://github.com/oxc-project/tsgolint) | `0.24.0` | `0.25.0` | | [tailwindcss](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss) | `4.3.2` | `4.3.3` | | [ultracite](https://github.com/haydenbleasel/ultracite) | `7.9.3` | `7.9.4` | | [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.1.4` | `8.1.5` | Updates `@aws-sdk/client-acm` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-acm/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-acm) Updates `@aws-sdk/client-ec2` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ec2/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-ec2) Updates `@aws-sdk/client-ecr` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecr/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-ecr) Updates `@aws-sdk/client-ecs` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecs/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-ecs) Updates `@aws-sdk/client-eks` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-eks/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-eks) Updates `@aws-sdk/client-elastic-load-balancing-v2` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-elastic-load-balancing-v2/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-elastic-load-balancing-v2) Updates `@aws-sdk/client-iam` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-iam/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-iam) Updates `@aws-sdk/client-s3` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-s3) Updates `@aws-sdk/client-sts` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/clients/client-sts) Updates `@aws-sdk/lib-storage` from 3.1084.0 to 3.1089.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1089.0/lib/lib-storage) Updates `@smithy/protocol-http` from 5.5.7 to 5.5.10 - [Release notes](https://github.com/smithy-lang/smithy-typescript/releases) - [Changelog](https://github.com/smithy-lang/smithy-typescript/blob/main/packages/protocol-http/CHANGELOG.md) - [Commits](https://github.com/smithy-lang/smithy-typescript/commits/@smithy/protocol-http@5.5.10/packages/protocol-http) Updates `@smithy/signature-v4` from 5.6.3 to 5.6.6 - [Release notes](https://github.com/smithy-lang/smithy-typescript/releases) - [Changelog](https://github.com/smithy-lang/smithy-typescript/blob/main/packages/signature-v4/CHANGELOG.md) - [Commits](https://github.com/smithy-lang/smithy-typescript/commits/@smithy/signature-v4@5.6.6/packages/signature-v4) Updates `@tanstack/react-router` from 1.170.17 to 1.170.18 - [Release notes](https://github.com/TanStack/router/releases) - [Changelog](https://github.com/TanStack/router/blob/main/packages/react-router/CHANGELOG.md) - [Commits](https://github.com/TanStack/router/commits/@tanstack/react-router@1.170.18/packages/react-router) Updates `@tailwindcss/vite` from 4.3.2 to 4.3.3 - [Release notes](https://github.com/tailwindlabs/tailwindcss/releases) - [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.3.3/packages/@tailwindcss-vite) Updates `@tanstack/router-plugin` from 1.168.19 to 1.168.20 - [Release notes](https://github.com/TanStack/router/releases) - [Changelog](https://github.com/TanStack/router/blob/main/packages/router-plugin/CHANGELOG.md) - [Commits](https://github.com/TanStack/router/commits/@tanstack/router-plugin@1.168.20/packages/router-plugin) Updates `oxfmt` from 0.58.0 to 0.59.0 - [Release notes](https://github.com/oxc-project/oxc/releases) - [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md) - [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.59.0/npm/oxfmt) Updates `oxlint` from 1.73.0 to 1.74.0 - [Release notes](https://github.com/oxc-project/oxc/releases) - [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md) - [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.74.0/npm/oxlint) Updates `oxlint-tsgolint` from 0.24.0 to 0.25.0 - [Release notes](https://github.com/oxc-project/tsgolint/releases) - [Commits](https://github.com/oxc-project/tsgolint/compare/v0.24.0...v0.25.0) Updates `tailwindcss` from 4.3.2 to 4.3.3 - [Release notes](https://github.com/tailwindlabs/tailwindcss/releases) - [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.3.3/packages/tailwindcss) Updates `ultracite` from 7.9.3 to 7.9.4 - [Release notes](https://github.com/haydenbleasel/ultracite/releases) - [Commits](https://github.com/haydenbleasel/ultracite/compare/ultracite@7.9.3...ultracite@7.9.4) Updates `vite` from 8.1.4 to 8.1.5 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.1.5/packages/vite) --- updated-dependencies: - dependency-name: "@aws-sdk/client-acm" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-ec2" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-ecr" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-ecs" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-eks" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-elastic-load-balancing-v2" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-iam" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-s3" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/client-sts" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@aws-sdk/lib-storage" dependency-version: 3.1089.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: "@smithy/protocol-http" dependency-version: 5.5.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: npm-dependencies - dependency-name: "@smithy/signature-v4" dependency-version: 5.6.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: npm-dependencies - dependency-name: "@tanstack/react-router" dependency-version: 1.170.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: npm-dependencies - dependency-name: "@tailwindcss/vite" dependency-version: 4.3.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: npm-dependencies - dependency-name: "@tanstack/router-plugin" dependency-version: 1.168.20 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: npm-dependencies - dependency-name: oxfmt dependency-version: 0.59.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: oxlint dependency-version: 1.74.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: oxlint-tsgolint dependency-version: 0.25.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: npm-dependencies - dependency-name: tailwindcss dependency-version: 4.3.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: npm-dependencies - dependency-name: ultracite dependency-version: 7.9.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: npm-dependencies - dependency-name: vite dependency-version: 8.1.5 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: npm-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
f02d9a6c02 |
chore: bump the go-dependencies group with 2 updates (#553)
Bumps the go-dependencies group with 2 updates: [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2). Updates `github.com/aws/aws-sdk-go-v2/config` from 1.32.29 to 1.32.30 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.29...config/v1.32.30) Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.19.28 to 1.19.29 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.28...credentials/v1.19.29) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.32.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.19.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
5dc945a6df |
chore: bump the github-actions group with 2 updates (#552)
Bumps the github-actions group with 2 updates: [actions/setup-go](https://github.com/actions/setup-go) and [actions/setup-node](https://github.com/actions/setup-node). Updates `actions/setup-go` from 6 to 7 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v6...v7) Updates `actions/setup-node` from 6 to 7 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Josh Siviour
|
a01e58d90c |
feat: deliver backend ENOSPC to the guest and enable chunk GC in production (#549)
* feat: plumb GCEnabled config through to every VB construction site viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never set it, so every deployed node ran with GC off by construction regardless of which viperblock binary shipped. Add a [viperblock] gc_enabled config key (default false, mirrors shardwal) and thread it through to the three places spinifex builds a VB: the nbdkit plugin it spawns per mounted volume (via a new gc_enabled plugin param), viperblockd's detached-volume opens, and the volume service's CreateVolume. * chore: repin viperblock to GC-enable lineage The gc-enable plumbing references viperblock.VB.GCEnabled, which is absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable commit so a standalone (GOWORK=off) build — the path the ansible deploy uses — resolves the field, and spx plus the nbdkit plugin build from the same GC-bearing viperblock lineage. * fix(storagegrowth): stop launching a nano for image-pull repro ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the synthetic npass suite. A 512MB guest OOM-thrashes containerd before any disk-driven ENOSPC and conflates the storage leak with guest-memory pressure. Add a package-local imagePullInstanceType helper: an exact override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default 16384 MiB). npass keeps using the nano selector unchanged. * feat: report guest block write errors so backend ENOSPC reaches the guest Set werror=report and rerror=report on the guest boot drive so a viperblock ENOSPC (or any other backend write failure) is surfaced to the guest as a real I/O error instead of QEMU silently pausing the VM. Without this the guest kernel never learns its disk is out of space, so it can't fail writes cleanly or let an operator react. * chore: repin predastore and viperblock to the storage-capacity-safety lineage Drop the local-path replace for predastore (a test shortcut) and pin both predastore and viperblock to their feat/storage-capacity-safety commits, so a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC classification, and backpressure plumbing that spinifex's werror/rerror change depends on. * chore: bump viperblock to 054a44a3f0f5 Realigns spx's linked viperblock with the submodule working tree, which the build role's source-drift gate compares against. * chore: repin predastore and viperblock to their dev lineage Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work has now landed on dev in each repo, and both PRs were squash-merged, so the branch commits those pins named are unreachable from any branch. Repin to the dev commits carrying the same code: predastore 5699329 and viperblock 35dd9f2. |
||
|
Josh Siviour
|
08fc5e4c58 |
fix(northstar): initialise OTel SDK and source the telemetry environment (#555)
Northstar carried working northstar.dns.* instruments but shipped no telemetry, so its dashboard was empty and the only northstar data in Elasticsearch was journald scraped under the systemd unit name. Two gaps, both outside the instrumentation. northstarStartCmd never called initTelemetry, unlike every other service command, so the instruments recorded into a no-op provider. The unit also did not source /etc/spinifex/telemetry.env, which every other application unit reads for OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV and MULGA_SOURCE — so even an initialised SDK would have had no endpoint and no env label. Add an invariant covering the second gap: any unit launching an spx service must source the telemetry drop-in. A service missing it starts healthy and silently reports nothing, which only surfaces when someone notices an empty dashboard. Verified the test fails against the unpatched unit. |
||
|
Josh Siviour
|
ffc34b735b |
ci: emit per-run ES summary doc from e2e workflow (#551)
Add e2e_es_summary, a self-hosted job that aggregates suite-status.txt from both e2e legs and POSTs one doc per run to the Elastic sink, keyed by labels.ci_run_id, so Kibana can answer "did run X pass, and where are its artifacts" without hand-aggregating traces-apm*/logs-apm.error*. Runs on ci-single (LAN route to the sink) since e2e_status stays on GitHub-hosted ubuntu-26.04 for the PR status-report check. State is computed with the same single_ok/multi_ok logic as e2e_status so the two can never disagree. |
||
|
Josh Siviour
|
055fa26f37 |
feat(bedrock): Ochre Phase 2 — ConverseStream + InvokeModelWithResponseStream (#526)
* feat(bedrock): Ochre streaming — ConverseStream + InvokeModelWithResponseStream
Adds the two streaming bedrock-runtime operations behind a shared
eventstream.Encoder frame writer: ConverseStream reframes vLLM/Anthropic SSE
into the normalized Bedrock event taxonomy (messageStart -> contentBlock*
-> messageStop -> metadata), InvokeModelWithResponseStream wraps each
provider-native chunk in a PayloadPart frame. Fixes otelsetup's
statusRecorder to implement Flusher/Unwrap so flushes actually reach the
socket through the middleware chain, which streaming depends on.
* fix(bedrock): keep vLLM ConverseStream taxonomy ordered on edge cases
Route contentBlockStop/messageStop through an idempotent ensureStarted so a
usage-only chunk arriving before any finish_reason no longer emits metadata
ahead of messageStop, and a zero-content stream no longer emits stop events
without a preceding messageStart. Regression tests for both.
* fix(bedrock): route /model/ data-plane calls to bedrock-runtime
AWS SDKs sign bedrock-runtime with the SigV4 signing name "bedrock", the
same as the control plane; the two services are separated only by endpoint
hostname. The gateway serves one endpoint and routed purely on the signing
name, so every /model/{id}/converse*|invoke* call landed in Bedrock_Request
(control plane) and returned InvalidAction — no standard SDK/CLI could reach
any bedrock-runtime endpoint.
Disambiguate by path in GetService: a "bedrock"-scoped request to a /model/
path (data-plane-exclusive) resolves to bedrock-runtime. Control-plane paths
(/foundation-models, /custom-models, /model-invocation-job) are unaffected.
* test(bedrock): satisfy testifylint go-require/empty in streaming tests
Use assert (guarded where a value is dereferenced afterward) instead of
require inside httptest handler closures, and assert.Empty for empty-string
checks, per dev's stricter testifylint config.
|
||
|
Josh Siviour
|
8205f88b04 |
ci: build sibling repos from a co-named branch, falling back to dev (#550)
The Build/Lint/Race jobs checked out viperblock and predastore at a hardcoded ref: dev and overlaid them with a go workspace, so the go.mod pins were ignored and a spinifex change depending on an unmerged sibling change could never compile in CI. Resolve each sibling to a branch of the same name when it exists on that repo, else dev, mirroring the e2e path in scripts/tofu-cluster/build-install-artifacts.sh so unit CI and e2e CI agree. |
||
|
Josh Siviour
|
908bfbfc27 |
fix(ecr): enforce IAM authorization on OCI registry operations (#548)
Treat the ECR registry JWT as a signed identity pointer rather than a permission grant. Every /v2/* request now rehydrates the token's access key against current IAM/STS state, maps the OCI route to an ecr: action and repository ARN, and runs the identity-policy evaluator before dispatch. PullOnly/ReadOnly principals can no longer push, overwrite tags, or delete manifests, and key/session/user/role/account/policy revocation takes effect on the next request instead of at token expiry. Cross-repository blob mounts now require source-read permission and proof the digest is referenced by a manifest in the named source repo, closing an account-wide blob-fishing gap. |
||
|
Josh Siviour
|
95631bf57b |
fix(elbv2): roll back managed SG on sync LB launch failure and surface the cause (#547)
A CreateLoadBalancerSync whose data-plane launch failed returned without rolling back the NLB managed security group. The orphaned SG sat in the EKS managed control-plane VPC and blocked DeleteVpc with DependencyViolation indefinitely, wedging the cluster in DELETING. The failure was also flattened to ServerInternal, discarding the only useful diagnostic. The sync path now unwinds fully on launch failure (ENIs, managed SG, name claim, record) and maps the launch error onto its real cause: capacity shortfalls pass through as their own code, anything else stays ServerInternal. The async path deliberately keeps its failed record — the caller holds the ARN and can reclaim it via DeleteLoadBalancer. |
||
|
joshsiv-mulga
|
9ccf147fc8 |
Merge remote-tracking branch 'origin/feat/storagegrowth-rootvol' into harness-merge-dev
# Conflicts: # tests/e2e/Makefile |
||
|
Julian Sommer
|
501f787c36 | refactor(ui): extract AttachedPoliciesPanel and SystemImageRequired (#546) | ||
|
Julian Sommer
|
5b03cf72c4 | fix: prevent live volume tag clobbering (#545) | ||
|
joshsiv-mulga
|
72353e476c |
test(e2e): read the live checkpoint on-node and target the root volume
The image-pull storage-growth workload HEADed <volID>/checkpoints/blocks.live.bin with the operator box's gateway identity, which has no S3 read grant on the bucket and was refused 403 on every sample, so checkpoint_bytes read 0 across a whole run. It now HEADs the object on the node over PeerSSH using predastore's own [[auth]] credentials from predastore.toml (root-owned, so read via passwordless sudo; the CA is readable through the spinifex group). The credentials are read into shell locals, used inline for one aws s3api head-object, then unset — never echoed, never leaving the node. A probe failure that is not a legitimate 404 voids the sample (checkpoint_bytes -1 plus a probe error) rather than reporting a false zero, so a dead instrument can never look like no growth. The dead operator-side S3 client path is removed. |
||
|
Julian Sommer
|
3b44d3969e | remove git hook |