* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
CI e2e-build compiles the spinifex module with GOWORK=off, resolving the
go.mod pins instead of the go.work local submodule redirect. v1.7.0 of both
modules predates symbols spinifex now calls (viperblock.StateBody,
predastore auth.ParseRoleARN), so the build failed with undefined-symbol
errors that go.work masked locally. Pin both to current dev-branch
pseudo-versions until a coordinated release is cut.
* feat(sts): EKS Q4 STS extensions — interface + JWKS reader + stubs
Lays the structural slice of AssumeRoleWithWebIdentity + presigned-URL
GetCallerIdentity verify for IRSA. The verify-path bodies (JWT signature
check against fetched JWKS, claims validation, Federated-principal trust
policy evaluation, SigV4 presigned-URL canonical-request reconstruction)
land in follow-up commits on this branch.
Adds:
- handlers/sts/oidc_jwks.go — JWK/JWKS types, ParseEKSIssuerURL (defensive
iss parsing), FetchClusterJWKS (KV-backed read from
eks-account-{accountID}/clusters/{name}/oidc-jwks.json via the EKS store
helpers).
- handlers/sts/oidc_jwks_test.go — unit tests over a real JetStream
test fixture.
- handlers/sts/assume_role_with_web_identity.go — interface entry, stub
returns ErrorNotImplemented.
- handlers/sts/presigned_url_verify.go — PresignedCallerIdentity type +
VerifyPresignedGetCallerIdentity stub.
- gateway/sts/AssumeRoleWithWebIdentity.go — dispatch shell.
Modifies:
- handlers/sts/service.go — STSService interface gains
AssumeRoleWithWebIdentity + VerifyPresignedGetCallerIdentity.
- gateway/sts.go — registers AssumeRoleWithWebIdentity in stsActions +
stsSkipPolicyCheck (anonymous; trust-policy-gated in handler).
- gateway/sts_request_test.go, gateway/sts/handler_test.go,
gateway/auth_test.go — stub implementations updated to satisfy
extended STSService interface.
Anonymous routing for AssumeRoleWithWebIdentity (eks-v1.md Q5 mux rework)
lands in Sprint 6e — for now the action is registered on the standard STS
gateway and rejects non-SigV4 requests upstream.
Refs: cs-eks-sts-q4 (mulga monorepo bead)
* feat(sts): AssumeRoleWithWebIdentity + presigned GetCallerIdentity verify
Implements EKS Q4 STS extensions for IRSA + EKS token-webhook flows.
AssumeRoleWithWebIdentity:
- ES256 JWT verification against per-cluster JWKS (eks-account-{accountID}
KV bucket), keyed by issuer host-path.
- Federated trust-policy evaluator (explicit-deny-wins, exact-match
Principal.Federated, fail-closed on Principal: "*").
- Condition support scoped to StringEquals on {iss}:sub (exact) and
{iss}:aud (set-membership). All other operators rejected at validator.
- OIDC provider registry check against iam-account-{roleAccountID}; v1
does not support cross-account registration.
- New awserrors: InvalidIdentityToken, IDPRejectedClaim,
IDPCommunicationError.
VerifyPresignedGetCallerIdentity:
- Hand-rolled SigV4 canonical-request reconstruction (no sdk dep).
- Mandatory X-K8s-Aws-Id in SignedHeaders + value pinned to
expectedClusterName during canonicalisation — cross-cluster replay
surfaces as a signature mismatch.
- AKID-prefix dispatch (ASIA → session credential, AKIA → IAM user).
- Strict envelope checks: algorithm, credential scope service/terminator,
X-Amz-Date / credential-scope date agreement, expiry within
X-Amz-Expires, 5-minute wire skew.
Tests:
- 14 web-identity scenarios (happy paths, tampering, expiry, wrong aud,
unknown kid, HS256 rejection, federated-wildcard fail-closed,
explicit-deny-wins, provider-not-registered).
- 12 presigned-URL scenarios including cross-cluster replay defence,
missing-x-k8s-aws-id, tampered query, bad service in credential, AKIA
+ ASIA happy paths.
- Cross-cluster anti-replay invariant test (rejects matrix of variant
cluster names — case-different, prefix, suffix).
- AWS SigV4 known-vector test for deriveSigningKey.
Gateway:
- TODO documents that AssumeRoleWithWebIdentity is currently dispatched
on the SigV4-gated path; a pre-auth route is needed so a projected
ServiceAccount token alone can authenticate the caller.
IAM:
- Relaxes trust-policy Condition validator: now permitted only when
Action is exactly [sts:AssumeRoleWithWebIdentity] and operator is
StringEquals. Preserves loud failure for sts:AssumeRole.
* feat(viperblockd): retry LoadState on transient backend errors
ebs.mount used to call vb.LoadState once and treat any failure as fatal.
When predastore was still warming up after a coordinated daemon restart,
the backend Read came back with a network/timeout error, viperblock
folded it into "block size is 0 in both local and backend state", and
the daemon's recovery path then ran the cleanup branch — destroying
DeleteOnTermination=true volumes (mulga-siv-25).
Wrap LoadState in retryLoadState: 5 attempts, 200ms * 1.5^n backoff
(total worst-case ~3.7s, well inside the 30s ebs.mount NATS deadline).
Retry only on viperblock.ErrStateBackendUnavailable; ErrStateNotFound
and any unclassified error fail fast so genuine problems are still
visible.
Retry loop is parameterised over sleep and loadFn so the retry policy
itself is unit-tested without spinning up a real VB.
* chore: bump viperblock to v1.2.2-0.20260511064619-1aa54f0dc1f6
Pulls in the ErrStateNotFound / ErrStateBackendUnavailable sentinels
introduced by mulgadc/viperblock#11 (merged on dev). Required for the
retry wrapper added in the previous commit to compile against the
public viperblock module.
* fix(installer): emit kernel cmdline to ttyS0 alongside tty0 (mulga-siv-33)
GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the
installed kernel printed nothing on serial after the GRUB menu. Headless
installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a
black screen forever and had no path to debug boot failures.
Append console=ttyS0,115200 — the last `console=` on the cmdline becomes
the system console, so ttyS0 wins on serial-only hardware while tty0
still receives output on local displays.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(ipam): widen vpc.dhcp.acquire retry budget to 60 s (mulga-siv-34)
ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard
error. On a cold ISO boot, NATS comes up well before vpcd's reconcile
leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological
~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's
dhcp.acquire subscription and fails permanently — d.externalIPAM stays
nil for the daemon's lifetime, daemon_handlers_instance.go silently
skips all auto-public-IP allocation, and instances launch with no
public IP, no dnat_and_snat, and no NAT event ever published.
Bump maxAttempts to 60 with 1 s spacing; expose both as package vars
so tests can shrink the budget for fast timeout-path coverage. Pure
widening of the existing retry loop — no behaviour change on the
success path, only the worst-case wallclock when vpcd never comes up.
The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd
not enabled in the ISO → veth pair vanishes on reboot → vpcd
crashloops) is fixed alongside this. A 60 s budget fits comfortably
above all observed healthy-startup times.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(setup-ovn): declare Linux WAN bridge as NetDev so networkd-managed veth resolves Bridge= on reboot (mulga-siv-33)
The veth-mode reboot persistence drop-ins added in mulga-998.b were
incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with
`Bridge=br-wan`, but never declared br-wan itself to networkd.
br-wan is created by `networking.service` (ifupdown via
`/etc/network/interfaces`, generated by `install.go writeNetworkConfig`),
not networkd. networkd's `Bridge=` directive requires the target to be a
known NetDev — when it isn't, networkd silently logs
`br-wan NetDev could not be found, ignoring assignment` and the veth
ends up with no master. vpcd's bridge-mode sanity check then fails on
every reboot.
Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as
`Kind=bridge`. networkd matches the existing kernel bridge by name+kind
(the `Failed to create netdev: File exists` log is harmless), and
`Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the
IP — networkd just needs the NetDev declaration to enslave the veth.
The write is gated to avoid clobbering operator-managed networkd config:
if any other `.netdev` under `/etc/systemd/network`,
`/run/systemd/network`, or `/usr/lib/systemd/network` already declares
the WAN bridge by name, the write is skipped with a log line pointing
at the operator file. Cleanup hook also removes the new file when
switching away from veth mode.
Verified end-to-end via QEMU repro: clean install + first boot + reboot,
veth-wan-br master persists across reboots, vpcd starts cleanly.
* fix(vpcd/dhcp): enable AF_PACKET promisc on bridge for DORA (mulga-siv-33)
Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and
reply with a unicast OFFER addressed to the synthetic per-tenant chaddr
(e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports
but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set,
so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times
out at 15s ("unable to receive an offer: context deadline exceeded").
Dev/CI didn't hit this:
- CI uses spx_wan_mode=static — no DORA via vpcd.
- QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so
their replies are L2 broadcast which the bridge always passes up.
Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/
Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET
DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket
(per-DORA, ~15s) and refcounted by the kernel — auto-clears on close.
* revert(vpcd/dhcp): drop AF_PACKET promisc wrapper, restore nclient4.New
ecfbd4f swapped nclient4.New for openPromiscBridgeConn + NewWithConn to
catch unicast OFFER replies that bypass the bridge netdev. On dev hosts
where upstream honors the BOOTP BROADCAST flag, the wrapper regressed
DORA — public-IP allocation failed where main works. Reverting until
the promisc path is validated against both broadcast- and unicast-OFFER
servers.
* fix(setup-ovn): mark br-int/br-ext Unmanaged for systemd-networkd (mulga-siv-37)
Fresh ISO install on Trixie+systemd-networkd left the OVS internal netdevs
admin-down: networkd has no matching .network so it tags them as unmanaged
but never brings them up. OVS dataplane works but ovn-controller flow
programming and any tooling that probes link state misbehaves.
Drop /etc/systemd/network/05-spinifex-ovs-internal.network with
Unmanaged=yes so OVS itself owns the link state — `ovs-vsctl add-br`
already runs the netdev up at create time.
* fix(vpcd): allocate gateway LRP IP from pool in centralized NAT (mulga-siv-36)
In veth/macvlan modes the gateway LRP carries all egress, so it must hold
a WAN-subnet IP — otherwise the upstream router silently drops ARP for
the WAN nexthop (RFC 826) and the default route never resolves. The
hardcoded 169.254.0.1/30 was correct for distributed NAT but regressed
centralized NAT (introduced under mulga-siv-26).
Add gw_lrp_range_start/end to ExternalPool. In centralized mode at IGW
attach the gateway LRP gets a fresh IP from that range, persisted as
external_ids:spinifex:gateway_ip on the LRP for idempotent reconcile and
retrofit. Direct mode keeps link-local. Per-VM EIP allocator skips the
gw range; ValidatePoolConfig rejects overlap with the EIP range.
Adds TestTopologyHandler_IGWAttach_CentralizedNATAllocatesGwLrpIP and
extends RetrofitAllGatewayPortNetworks to migrate stale link-local LRPs
to allocated pool IPs at startup.
* fix(vpcd): drop link-local fallback, auto-derive gw_lrp_range (mulga-siv-36)
In centralized NAT (veth/macvlan) the gateway LRP must hold a WAN-subnet
IP — link-local cannot ARP-resolve the upstream nexthop, so the upstream
router silently drops egress. Hardware deployment confirmed the previous
fallback to 169.254.0.1/30 was always reached when gw_lrp_range was not
explicitly configured.
Auto-derive a 16-IP range at the top of the WAN subnet (broadcast - 16
.. broadcast - 1) when gw_lrp_range is unset; shift below RangeStart on
overlap with the per-VM EIP range; clamp inside the subnet; guard the
upstream gateway IP. IGW attach now errors out instead of falling back
to link-local when allocation fails — surfaces misconfiguration loudly.
Adds focused unit tests for gwLrpRange, ValidatePoolConfig and
nextAvailableExternalIP gw-lrp skip; updates the multi-VPC and
PoolWithGatewayIP regression tests.
* feat(vpcd): DHCP-acquire gateway LRP IPs (mulga-siv-38)
Per-VPC OVN gateway LRPs now obtain their WAN-subnet IP via vpc.dhcp.acquire
when the pool is source="dhcp", instead of carving from a static gw_lrp_range.
The upstream DHCP server becomes the single arbiter of LAN address space; a
cluster grows by N nodes without operator-side range carving in spinifex.toml.
- Consolidate ObtainDHCPLease/ReleaseDHCPLease into the dhcp package as
RequestAcquire/RequestRelease so vpcd and the daemon-side ExternalIPAM
share one helper. handlers/ec2/vpc keeps its existing ENI/EIP DHCP path.
- TopologyHandler dispatches by pool source: DHCP pools call DHCPManager
via NATS (client-id "gw-lrp-<vpcId>", idempotent on retry); static pools
keep the siv-36 gw_lrp_range / auto-derive allocator. Direct mode and
no-pool deployments still use link-local.
- IGW detach issues vpc.dhcp.release so the upstream server reclaims the
IP. Best-effort — server-side expiry is the safety net.
- Tests: TestTopologyHandler_IGWAttach_DHCPGwLrpIP (3 VPCs share one DHCP
pool, distinct IPs, idempotent attach, release on detach),
IGWAttach_DHCPAcquireFailureAborts (no link-local fallback when DHCP
errors), TestPrefixFromMask. Existing static / auto-derive tests
preserved.
* fix(vpcd): subscribe DHCPManager before reconcile to fix boot race (mulga-siv-38)
DHCPManager.Subscribe ran AFTER ReconcileFromKV / RetrofitAllGatewayPortNetworks,
so retrofit's vpc.dhcp.acquire requests for existing IGWs hit "no responders" and
burned the 60-attempt × 1 s cold-boot retry budget per VPC before failing. On a
dev box with one bootstrap VPC the symptom is a 60 s vpcd startup hang followed
by "allocate gw LRP IP failed", and the bootstrap IGW topology missing from OVN
until the next reconcile pass.
Move dhcpManager.Subscribe ahead of the two reconcile passes so the subscription
is live before any topology code can fan out an acquire.
* fix(vpcd): also subscribe DHCPManager before bootstrap reconcile pass
Previous fix moved DHCPManager.Subscribe ahead of ReconcileFromKV but
missed Pass 1 (the bootstrap-config Reconcile that runs even earlier).
On a dev box with a source="dhcp" pool and a bootstrap IGW, that earlier
pass still hit "no responders" and burned the 60-retry cold-boot budget
before the Manager came up. Move both reconcile passes after Subscribe.
* fix(vpcd): RFC 2131 §4.1 DORA exponential backoff (mulga-siv-39)
DHCPManager.Acquire previously did one DISCOVER and gave up after 15s
because the outer ctx.WithTimeout matched the per-attempt timeout —
nclient4's WithRetry(3) could never fire. Real DHCP servers (consumer
routers, ISC dhcpd, dnsmasq) silently drop OFFERs under load; the spec
mandates retransmission with exponential backoff, and a single lost
DISCOVER was killing legitimate boots.
Move retransmission into DHCPManager.acquireWithBackoff with a
4s/8s/16s/32s schedule (±1s jitter, ~60s worst case). NClient4Client
becomes single-attempt; manager owns retry policy. Default
acquireTimeout lifts from 15s to 90s (sum of schedule + slack).
* fix(vpcd): retry NewDHCPManager on JetStream-not-ready (mulga-siv-40)
NewDHCPManager creates the spinifex-dhcp-leases KV bucket via
utils.GetOrCreateKVBucket → js.CreateKeyValue with no retry. After the
mulga-siv-38 launchService reorder, this runs before any other JetStream
user, so on multi-node cold-boot vpcd hits "JetStream system temporarily
unavailable" and exits before cluster quorum forms. Pseudo-multinode CI
calls start_node_services sequentially, so node2's vpcd failure aborts
the script before node3 ever launches and quorum then never forms.
Wrap NewDHCPManager in newDHCPManagerWithRetry (500ms→10s, 5 min cap,
ctx-cancellable) mirroring daemon.initJetStream so vpcd waits as
patiently as the daemon for the cluster to come up.
* fix(daemon): idempotent DetachVolume QMP path (mulga-siv-41)
DetachVolume's QMP sequence was not retry-safe. First call: device_del
succeeds, blockdev-del fails with "Node nbd-vol-X is in use" because NBD
client teardown / in-flight I/O still hold the block node. Handler bails
ServerInternal "leaving volume state intact" but the guest device is
actually gone. AWS-CLI retries: device_del returns DeviceNotFound,
handler bails again. After 2 retries aws cli exits 254 — Single-Node
Phase 8 Step 3 reproducibly fails this way.
Phase 1: treat DeviceNotFound on device_del as already-detached, log and
continue. Detach is now idempotent across retries.
Phase 2: tryBlockdevDel polls blockdev-del up to 20×detachDelay on "in
use" errors so the I/O drain can complete instead of failing the whole
detach. Other QMP errors fall through immediately as before.
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)
Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:
- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
/local/instances (placeholder types until daemon-local-autonomy lands),
plus /health for scaffolding validation
Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.
* feat: add DDIL harness fault injection and link profiles
Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:
- harness/fault.go — PartitionNode (per-peer iptables DROP with
orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
RestartDaemonOnly, WaitForMode (polls /local/status for typed
DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
SATCOM, HFData, Flapping constants (unvalidated approximations per
plan §2 table; validation tracking lives in TEST_COVERAGE.md)
All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.
* feat: add DDIL harness witness VM helpers
LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.
* feat: add DDIL harness snapshot and state assertions
TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.
* feat: add DDIL harness reset, retry, and quarantine
ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.
AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.
Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.
bd: mulga-siv-5.5
* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)
Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.
* feat: add DDIL E2E CI workflow (workflow_dispatch only)
Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
Introduces the spinifex/services/vpcd/dhcp package with a Client
interface, typed Lease struct, nclient4-backed implementation, and a
Fake for unit tests. Adds DHCPManager in the vpcd package: subscribes
to vpc.dhcp.acquire / vpc.dhcp.release NATS requests, runs a renewal
goroutine per lease with ±10% jitter on T1/T2, persists leases in the
new spinifex-dhcp-leases KV bucket, and fans out vpc.dhcp.lease-expired
when a lease permanently fails to renew. Wires the Manager into
launchService so it starts on vpcd boot and rehydrates persisted leases
via Bootstrap.
Handler-side wiring (ExternalIPAM → NATS) and the systemd capability
move land separately in mulga-991.
See docs/development/bugs/dhcp-allocate-error.md.
Picks up multipart upload wedge fixes from predastore v1.1.5:
- 76fdbf1 fix: pool cleanup must not reap connections with in-flight streams (Bug A)
- 7df9645 fix: add mutex to bufferedWALFile to prevent bufio race (Bug B)
- 9dcd3dd fix: drain shard body before taking wal.mu in handlePUTShard (Bug C)
- 43e20f6 fix: widen QUIC flow-control windows to mitigate multipart upload deadlock
plus regression tests for all three bugs (predastore PR #8).
Replaces charmbracelet/huh with a custom bubbletea wizard for the
installer TUI. This gives full control over terminal I/O so the
installer attaches directly to the correct console device (ttyS0 for
serial, tty1 for VGA) as directed by the new spinifex-init boot script.
- branding/branding.go: centralised logo (ASCII placeholder), color
constants, and publisher string ready for Mulga asset swap
- ui/ui.go: multi-screen bubbletea model (welcome → disk → network →
identity → join → CA cert → confirm) styled with lipgloss
- main.go: reads SPINIFEX_CONSOLE from /proc/cmdline to open the right
tty before launching tea.Program
- go.mod: bubbletea/lipgloss/bubbles promoted to direct deps; huh removed
Adds cmd/installer with three packages:
- ui: huh-based TUI collecting disk, network, node, and cluster config
- install: sequential install steps (partition, debootstrap, chroot, bootloader)
- firstboot: oneshot systemd service that runs setup-ovn and spx admin init/join on first real boot
Supports both single-node init and multi-node join flows.
Julian Sommer