mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
7d5ebabecc4a8b90a3b5b3612dfd60acf4cffaef
1650 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
7d5ebabecc |
fix: explicit stop timeouts, graceful-shutdown exit code, and QEMU orphan classification (#640)
* fix(systemd): give every unit an explicit TimeoutStopSec 9 of 12 spinifex systemd units had no TimeoutStopSec and inherited systemd's 90s default, which is either far more or far less time than their own SIGTERM handling needs. Size each one to what its shutdown path actually does: predastore matches viperblock's 30s WAL-drain budget since viperblock flushes through it (After= already sequences viperblock's stop first; this is the backstop if that's ever bypassed), spinifex-ui covers its 30s internal shutdown context plus margin for long-lived streaming proxy connections, nats covers its own JetStream-shutdown/raft-stepdown/client-drain signal handler, and services with no SIGTERM handler at all (awsgw) get a short bound since they exit immediately regardless. Extends the existing unit-file invariant test with a check that every .service file declares TimeoutStopSec, so a newly added unit that forgets one fails CI instead of silently falling back to 90s. * fix(vm): reconcile QEMU processes with no instance record on restore classifyRestoredInstances only walks Manager.Snapshot(), so a qemu-system process with no instance record at all -- a crash mid-launch, a lost KV write, a stale process surviving a wiped state file -- was never handed to killOrphanedQEMU and stayed permanently invisible to recovery. Add a host-process scan that diffs live qemu-system PIDs (matched via /proc/<pid>/comm) against the PIDs claimed by known instances' own -pidfile files. It only reports what it finds, never kills: this daemon cannot distinguish "record not loaded yet" or "owned by another daemon/tenant on this host" from a genuine orphan, and killing a running customer VM to fix an invisibility gap is the wrong trade. Runs at the end of Restore, once every known instance has been classified and relaunched, so a live process still unaccounted for at that point genuinely has no record anywhere. * fix(vm): distinguish recordless QEMU orphans by pidfile ownership A recordless qemu-system process with a stale spinifex pidfile is positively this daemon's own orphan; one with no pidfile at all could belong to another tenant on a shared host. Log the two cases distinctly instead of one blanket warning, so an operator can tell them apart. Also sandbox XDG_RUNTIME_DIR in the shared daemon test harness so Restore()-driven tests never read this box's real runtime directory. * fix(spinifexui): treat ErrServerClosed as a clean shutdown, not a failure Serve returns http.ErrServerClosed after Shutdown is called deliberately (the SIGTERM handler above does exactly that), but Start propagated it as an error, making systemd log a graceful stop as exit code 1 and Result: exit-code -- indistinguishable from a real crash. |
||
|
Josh Siviour
|
99323672bc |
fix: enrich REST-JSON spans with resolved AWS action (#639)
Query-protocol services (EC2, IAM, STS...) get their action from the SigV4 Action= form param before dispatch, so traceActionEnricher can rename the span and set aws.action/aws.service from that context. REST-JSON services (EKS, ECS, ACM, tagging, ECR, bedrock...) only resolve their action inside their own dispatcher, well after that middleware runs, so the enrichment never fired for them. checkPolicyResource is the one call every dispatcher already routes its resolved (service, action) pair through, so move the enrichment there via a shared recordResolvedAction helper instead of duplicating it per REST-JSON handler. |
||
|
Josh Siviour
|
59db7d1e52 |
fix: stop calling RemoveSystemImage's deleted bytes "freed" (#638)
BytesFreed counted logical object sizes deleted from the S3 namespace, but predastore only reclaims that disk space later via background compaction. The CLI printed "freed X GiB" immediately on delete, which made a compaction outage that never reclaimed the space invisible. Rename to BytesDeleted and reword the CLI output to say the bytes are marked for deletion and reclaimed by background compaction, not freed immediately. |
||
|
Josh Siviour
|
9f5f877e50 |
fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637)
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console. |
||
|
Josh Siviour
|
614ab09866 |
fix(ec2): render empty result containers and return NotFound for missing named resources (#636)
* fix(ec2): render empty Describe result containers and add requestId xmlutil.BuildXML omits a nil slice's XML container entirely but renders an empty one for a non-nil empty slice, so every EC2 Describe with zero results (DescribeKeyPairs, DescribeVolumes, DescribeInstances, etc.) came back as a bare "<FooResponse></FooResponse>" instead of AWS's "<FooResponse><requestId/><fooSet/></FooResponse>". botocore parses that body as having no result key at all, so the CLI silently prints nothing and exits 0 — this broke the ansible key-import guard that gates cluster bootstrap on describe-key-pairs' output. Fix normalizes nil result slices to empty ones and synthesizes a requestId in the single marshal chokepoint shared by every EC2 action (marshalEC2Response, used by both ec2Handler and ec2HandlerWithReq), rather than patching each handler individually. * fix(ec2): return InvalidKeyPair.NotFound / InvalidSnapshot.NotFound for missing explicit IDs DescribeKeyPairs and DescribeSnapshots silently returned an empty list when the caller named a specific, nonexistent KeyName/KeyPairId or SnapshotId, instead of erroring like real AWS (and like the existing DescribeSecurityGroups/DescribeImages/DescribeVolumes/DescribeSubnets handlers already do). botocore treats the empty-list response as success, so ansible's cluster-post key-import gate — which checks `describe-key-pairs --key-names spinifex-key` for a nonzero exit code — never fires, and spinifex-key is never imported during bootstrap. Mirrors the two-pass found-set pattern already used by DescribeSecurityGroups: build the set of names/IDs actually present in the filtered results, then error if any explicitly requested name/ID is absent. An unfiltered list or a --filters query that matches nothing is unaffected and still returns an empty list with no error. DescribeInstances has the same gap but needs a design decision on how a NotFound check interacts with the fan-out's existing partial-result leniency, so it is left for separate follow-up. |
||
|
Josh Siviour
|
044d7af85e |
fix(ui): pin router-plugin version and commit deterministic route tree (#635)
TanStack router-plugin <1.168.21 sorted routes by coercing route nodes to strings, so output order followed filesystem-walk order and flipped depending on which generator version a contributor's node_modules resolved. Pin the version so it cannot drift, commit the sorted output from the fixed generator, and fail CI if a build produces a different route tree than the one committed. |
||
|
Josh Siviour
|
6873875248 |
fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634)
On a zero-worker or GPU-only cluster no node tolerates the DaemonSet, so it has zero desired pods. The apiserver then has no tunnel to kube-system pods (including metrics-server), v1beta1.metrics.k8s.io never goes Available, and namespace GC hangs on the kubernetes finalizer for every namespace, not just ones using metrics. Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already used by every other control-plane-critical addon in this repo (aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent also schedules onto the control plane. |
||
|
Josh Siviour
|
6ce8b9ff18 |
fix: fail preflight when a gate tool is killed instead of reporting it clean (#633)
A tool whose child dies by signal can still exit 0 through go tool, so govulncheck printed 'ok' after being OOM-killed and preflight passed without a vulnerability scan having run. Route lint, govulncheck and nilaway through scripts/run-gate.sh, which fails on a non-zero exit or on signal-death output. |
||
|
Josh Siviour
|
d89ac24af2 |
fix(gateway): enforce iam:PassRole on ECS task and service role ARNs (#632)
RegisterTaskDefinition let any caller with ecs:RegisterTaskDefinition name an arbitrary taskRoleArn/executionRoleArn, and RunTask/StartTask/CreateService/ UpdateService could launch tasks under a previously registered definition's roles, all without holding iam:PassRole on those roles. Mirror the EC2 gateway's PassRoleChecker closure pattern to close both paths. |
||
|
Josh Siviour
|
ccc726e0ba |
fix: close the LB name-claim orphan-reclaim race window (#631)
ClaimLBName's crash-orphan reclaim treated any owner with no persisted LB record as abandoned, but a legitimate in-flight create has no record yet either during the gap between claiming the name and PutLoadBalancer. Two concurrent creates for the same name could both win. Track a claim timestamp and only allow reclaim once it exceeds a TTL, so a fresh in-flight claim blocks a second claimer like a live LB would, while a claim abandoned by a genuine crash is still reclaimed. |
||
|
Josh Siviour
|
ee0bf68d71 |
fix(gateway): prefer a call site's own error message over the generic lookup (#630)
gateway.ErrorHandler unconditionally overwrote every error with ErrorLookup's static per-code text, discarding both a caller's own formatted message and any per-service distinction for wire codes shared across services (e.g. ACM and EKS both use ResourceInUseException). ResolveErrorDetail now also returns a message when the producing call site used awserrors.Errorf, and LookupErrorMessage resolves service-scoped wording before falling back to the global default. DeleteVpc's DependencyViolation call sites now name the blocking subnet or security group. |
||
|
Josh Siviour
|
135da2371e |
fix: scope golangci-lint cache per worktree, correct eks-node-gpu catalog description (#629)
* fix: scope golangci-lint cache per worktree golangci-lint's default cache under ~/.cache/golangci-lint is keyed by module path rather than worktree path, so parallel worktrees of this module collide and surface each other's lint issues, including from worktrees that have since been deleted. * fix: correct eks-node-gpu catalog description to match agent-only image The GPU node image never installs eks-token-webhook or a server/agent role selector, per scripts/images/eks-node-gpu/manifest.conf; it ships ecr-credential-provider only and is agent-only because GPU nodes never run the EKS control plane. |
||
|
Josh Siviour
|
bca63a4ede |
fix(vm): detach non-DeleteOnTermination volumes on terminate, not just Boot ones (#628)
instanceCleanerAdapter.DeleteVolumes only called DetachVolumeOnTerminate for a still-attached Boot volume, assuming shutdownAndUnmount's Unmount already cleared a non-Boot volume's attachment. That assumption breaks whenever the unmount seal fails (Unmount swallows the error) or never runs at all, as on the stuck-terminate force-complete path. A leaked volume then stays in-use with no recovery path once the instance record is deleted: detach-volume --force fails InvalidInstanceID.NotFound and delete-volume fails VolumeInUse. Call the same explicit, idempotent detach for every non-DeleteOnTermination volume regardless of Boot. |
||
|
Josh Siviour
|
0628b5f20d |
fix(ec2): project AMIMetadata.State into DescribeImages instead of hardcoding available (#627)
DescribeImages hardcoded every AMI as "available" regardless of import progress, so a failed backend import that never reached the flush was indistinguishable from a complete one. Project the AMI's own State, treating empty as available so pre-existing AMIs registered before the field existed stay visible. CreateImageFromInstance, RegisterImage, and CopyImage now set State explicitly since each completes synchronously before the config write. |
||
|
Josh Siviour
|
82d6b2044c |
fix(vm): report ENOSPC to the guest instead of pausing the VM, and let the stuck-terminate backstop fire (#626)
* fix(vm): report ENOSPC to the guest on data volumes instead of pausing the VM QEMU's default werror=enospc pauses the whole VM when a data volume's backend runs out of space. The boot drive already avoids this via werror=report,rerror=report; data volumes had no on-error policy at all in either the cold-boot -device or the hotplug device_add. Set werror=report,rerror=report on the virtio-blk-pci device itself, since a data volume's blockdev-add NBD node has no BlockBackend to carry it the way -drive does for the boot drive. * fix(vm): stamp ShuttingDownAt even when the state transition write fails transitionWithPrecheck returned before stampShuttingDownAt whenever TransitionState failed, even though the in-memory status had already moved to the target. On a full disk this leaves ShuttingDownAt zero, and StuckTerminateReaper.Sweep skips any VM whose ShuttingDownAt is zero — so the backstop meant to bound a wedged terminate is disabled by exactly the condition it exists to catch. Stamp on the persistence-failure branch too, using the same helper and locking as the success path, while still returning the original error to the caller. The raced branch (TransitionState failed and the in-memory status never reached target) is left unstamped, since the target state was never actually entered there. |
||
|
Josh Siviour
|
d455f3e569 |
fix(daemon): exit on SIGTERM instead of deadlocking on its own wait group (#625)
Start registers startCluster as a member of shutdownWg and then waits on that group. startCluster's tail waited on the same group, so it could not return until the counter it was waiting on reached zero, and the counter could not reach zero until it returned. The SIGTERM handler completed and logged "Shutdown complete" in ~20ms, but the process never exited: it sat until systemd's TimeoutStopSec=15 and was SIGKILLed, on every stop. Because the member stop chain is serialized, that 15s also gated viperblock, predastore and nats, which each stop in milliseconds once reached. The duplicate setupShutdown in the same tail registered a second signal.Notify and a second handler goroutine, which is where the duplicated shutdown log lines and NATS unsubscribe pass came from. Drop both. Start already installs the handler and owns the single wait. The invariant test pins each helper to one call site, since a second call site is the defect rather than a symptom of it. |
||
|
Josh Siviour
|
a104f73306 |
test(dhcp): assert on persisted KV state, not the acquire counter (#624)
Five manager tests waited on a fake DHCP counter and then read the KV store. Fake.Acquire increments that counter before Manager.doAcquire persists the entry, so the assertion ran in the gap between the two and read the pre-acquire address whenever the write lost the race. Idle machines always won the race, which is why this only ever failed under load: the last three Build and Test failures on dev were this test twice, on a diff that never touched the package. Make the persisted KV state the barrier, matching the shape already used by TestManagerStartKeepsNewLeaseWhenStaleReleaseFails, and keep the counter as a plain assertion once the write is observed. |
||
|
Josh Siviour
|
d38db57912 |
fix(ec2): release post-launch ENIs on instance terminate (#623)
* fix(ec2): release post-launch ENIs on instance terminate Terminate released only the ENI recorded at RunInstances time. Attachments made after boot — ECS awsvpc task ENIs, or any direct AttachNetworkInterface — are written to the spinifex-vpc-enis KV and never reach instance.ENIId, so both terminate paths walked straight past them. The survivors held their security group, blocking the SG, subnet and VPC deletes behind DependencyViolation. Both paths now sweep the KV for records naming the instance, independent of the scalar. The sweep is additive, so an instance with no secondary ENIs behaves exactly as before, and best-effort per ENI so one failure neither aborts the rest nor blocks convergence on re-run. DeleteOnTermination was advertised by DescribeNetworkInterfaces but backed by nothing; ENIRecord now carries it, defaulting to true, and false detaches without deleting. * test(ec2): cover ENI release branches Add table-driven and NATS-backed tests for the terminate-time ENI sweep: DetachENI/ForceDeleteInstanceENI failure tolerance, DeleteOnTermination skip, multi-ENI happy path, and ListInstanceENIs error mapping on both the daemon and instance service adapters. |
||
|
Josh Siviour
|
67a8b4409f |
chore(deps): bump viperblock for the nearfull throttle fix (#622)
viperblock#64 stops WriteAtCtx returning ErrNoSpace on predastore's nearfull pressure signal, which rides on a successful write and means slow down rather than stop. Without this, spx refused every write once the store fell below 15% free, 10 points above the watermark at which predastore actually rejects. Needed by spx itself: image import runs in-process against the module-cache build, so the plugin-side fix alone left the control plane on the old behaviour. |
||
|
Josh Siviour
|
e4cd9ed527 |
chore(deps): bump viperblock for the durability batch (#621)
Six merged viperblock changes behind one repin: local files kept when the WAL cannot be uploaded, stranded WAL generations retried after a failed chunk upload, the flush-covers-acked-writes invariant pinned, the nbdkit plugin's shutdown drain bounded at 20s, seal receipts written after a clean seal, and FLUSH fsyncing the WAL before it reports success. The seal receipts pair with the consumer already on dev, and the 20s drain bound pairs with TimeoutStopSec=30 on spinifex-viperblock.service. |
||
|
Josh Siviour
|
59b19192c4 |
fix(viperblockd): branch on the plugin's seal receipt, not on a deleted directory (#620)
The unmount path decided whether to seal by stat-ing BaseDir/<volume>, but the nbdkit plugin deletes exactly that directory as the last step of a successful seal. So the fallback seal was unreachable whenever the plugin exited cleanly, and a WARN claiming there was no local state fired on every healthy unmount. Branch on the receipt the plugin now leaves instead. Local state present still means the plugin's seal failed or was cut short, so the fallback runs and is finally reachable; a receipt means the plugin sealed and cleaned up; neither keeps the original WARN, now meaning what it says. The aux-volume check moves out of volumeNeedsSeal and to the front of the chain: split out but left second, an -efi volume with local state would newly reach a seal it has always been excluded from. Stale receipts are cleared at mount rather than matched by PID at unmount, so nothing depends on nbdkit's pid surviving to plugin load. |
||
|
Josh Siviour
|
70dda56fc3 |
fix(systemd): backstop viperblock stop at 30s instead of the 90s default (#619)
In-use nbdkit children are SIGTERMed with this cgroup and run the plugin's shutdown drain. That drain is now bounded at 20s in the plugin itself, so 30s here lets the plugin's own bound fire first and keeps SIGKILL exceptional. At the 90s default a slow or unreachable backend stalled the whole target-stop transaction. |
||
|
tomnewton-mulga
|
d9fb17155f | chore: fix EKS doc | ||
|
tomnewton-mulga
|
168b8eb83e | chore: RTX doc fix | ||
|
tomnewton-mulga
|
4a87f19ca0 | chore: Updated RTX-6000 doc | ||
|
Josh Siviour
|
034c786916 |
chore(deps): bump viperblock for the nbdkit plugin stderr fix (#618)
Picks up viperblock #57, which routes the JSON slog handler to stderr so the nbdkit plugin's logs reach journald — nbdkit repoints a plugin's fd 1 at /dev/null, so the whole volume seal path was discarded. Also picks up #56, removing a per-call allocation from the per-block cache instrument. No spinifex code depends on either change: spx does not import viperblock/telemetry. The bump is needed because the build role compares the viperblock source behind spx (module cache, GOWORK=off) against the one behind the nbdkit plugin (the submodule tree) and fails on a mismatch. |
||
|
Josh Siviour
|
2983160374 |
fix(lbagent): emit JSON logs so console attributes reach the sink as fields (#617)
lb-agent configured no logger at all, so it fell back to slog's default text handler. That folds attributes into the message, leaving the cause of a failure trapped in a string: 'Heartbeat failed err="gateway returned 400"' arrived greppable but never queryable. No OTLP export: the agent's only egress is the gateway it is watching, so the host-side serial console remains the path that survives a blackout. |
||
|
Josh Siviour
|
df6f797971 |
fix(otel): change log level in place so the gateway keeps shipping to the sink (#616)
Init bolts the OTLP bridge onto whatever slog default is installed at the time. SetupRoutes then called SetDefaultJSONLogger purely to set a level, and that bare slog.SetDefault replaced the fanout with a stdout-only handler, so the gateway went dark to the sink for the rest of the process. One nightly cell shipped 9 app-log docs, all within 30ms of startup, while its journal held 574 request lines. Backs the stdout handler with a LevelVar and adds SetLevel, so verbosity changes no longer touch the handler chain. |
||
|
Josh Siviour
|
b6d43d8b51 |
feat(gateway): log the canonical request and action on signature mismatch (#615)
* feat(gateway): log the canonical request and action on signature mismatch A SignatureDoesNotMatch told us nothing beyond err="signature mismatch": the canonical request the server reconstructed was never logged, and the APM trace is a bare "POST /" because auth fails before the dispatcher parses the action. An intermittent mismatch was therefore undiagnosable. Logs the server-side canonical request, with the session token masked, plus the recovered action. Warn rather than Debug because the failures worth catching never reproduce on demand; the path is already rate-limited. * chore(deps): bump predastore for SignedRequest.CanonicalRequest Picks up the exported canonical request the mismatch logging needs. |
||
|
Josh Siviour
|
545ca94a8e |
fix(ci): report a cell that never ran a test as failed, not clean (#614)
Render summed FailCount across suites and printed "No failures across any suite" whenever the total was zero, without asking whether any test had run. A cell that dies during provisioning emits junit with no testcases at all, so it took the clean path. Nightly run 30467638631 cell 20 failed with "install callback never fired after 3 attempts; PXE never triggered install" and its analysis.md read "No failures across any suite" — absence of evidence rendered as positive evidence. Zero testcases now renders a warning naming the layer above the suites. A run with tests and no failures is unchanged. |
||
|
Josh Siviour
|
0cdf591c7c |
ci(nightly): move ubuntu cells to the 26.04 gold image (#611)
Cells 2, 3, 9 and 10 ran on spinifex-image-builder-ubuntu-24.04.qcow2, built 2026-04-20 — five weeks before openvswitch-ipsec and strongswan-charon were added to the image builder. Those cells therefore had no OVN IPsec support and ran intra-AZ Geneve in plaintext. Every other workflow is already on 26.04. The host_image_override guards now match ubuntu-* rather than a pinned version. An exact match is what let this drift go unnoticed: bumping the matrix release silently stops overriding the image and drops the cell onto the debian gold. |
||
|
Josh Siviour
|
5fc6a137ec |
fix(e2e): fail TestMultinodeIPSec when a cluster that requested IPsec has none (#612)
* fix(e2e): fail TestMultinodeIPSec when a cluster that requested IPsec has none An empty ipsec_encapsulation was treated as "bootstrapped without IPsec" and skipped. On ubuntu-24.04 openvswitch-ipsec.service is inactive at daemon start, so the enable guard correctly refuses to flip the flag and the cluster runs plaintext intra-AZ Geneve — but the test then skipped and the cell reported green, masking the gap on 2 of 3 multinode cells. Cross-check network.ipsec_enabled from the node config: an explicit false still skips, anything else fails with the plaintext condition named. * fix(e2e): read only the ipsec_enabled value, not the whole config line Detail() rendered "ipsec_enabled=ipsec_enabled = true" because the raw grep line was logged as the value. Take the first token after "=" so a trailing comment cannot be read as the value either, and compare exactly rather than by substring. |
||
|
Josh Siviour
|
f973b306c4 |
fix: serialise DNS zone writes behind a per-zone lock (#610)
Every daemon read-modify-wrote the same zone object in s3://northstar with no mutual exclusion. A NATS queue group balances load, it does not serialise distinct messages, so concurrent daemons lost each other's records and could leave the object unparseable as TOML. Once unparseable it stayed that way: both the writer and the drift reconciler must parse the zone before they can repair it, so DNS changes stopped applying until the object was edited by hand. The exposure did not need user activity. The drift backstop ran on every node, unelected, on a 60s ticker, so an idle three-node cluster raced its own zone once per tick. Five publisher families feed the same object, so ordinary concurrent API calls collided too. - Add a per-zone JetStream KV lock, following the CAS patterns already used by the vpcd reconcile leader and the vCPU quota store. One key per zone, so unrelated zones never block. The wait budget stays under requestTimeout so a producer's ack is not lost to contention, and a write aborts rather than land outside an expired lease. - Rebuild a zone whose stored bytes do not parse instead of failing every write forever. The reconciler reports a corrupt zone as absent, which suppresses pruning while still publishing the full desired set. Detection matches the decoder's error type so it works against the pinned northstar release. - Elect one node per drift cycle, removing the idle-cluster self-race. - Correct three comments claiming the queue group serialised zone writes; they are why the hazard went unnoticed. - Run the LB e2e internet-facing subtests in parallel. Their serialisation workaround exempted internal LBs on the false premise that those use a different DNS path; ELBChanges sets the same base-domain zone for both. |
||
|
Josh Siviour
|
628fbbc0c3 |
fix: group-scope the OVS/OVN sockets and retire the vpcd sudoers grants (#609)
* fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh, with a systemd override re-applying it on every openvswitch-switch restart. Write access to db.sock is full control of the local datapath, so this handed the datapath to any account on the box. Both service users run with Group=spinifex, so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller alike — the latter is new, and matters because the ovn-controller ctl socket name embeds the pid and is recreated on every start. That also fixes the health probe reporting ovn-controller as not_running on a healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is granted; as spinifex-daemon sudo demanded a password and the swallowed error read as OVN being down. With the ctl socket group-owned the probe needs no privilege, so health.go moves to a non-escalating Runner rather than the alternative of granting the daemon a root-equivalent ovn-appctl rule to read a status. ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick, so 1+N sudo invocations every tick, each one logging a pam_limits warning about the core rlimit the unit deliberately pins to 0. It now issues a single JSON query per pass, and the sudoers drop-in disables the PAM session for both service users, which also leaves sudo children inheriting the unit's rlimits. An absent iface-id on a patch port is now skipped with a warning instead of aborting the pass: it is the only record of the full ENI, so such a port cannot join the live set either way, and failing the pass stalled reconcile for every healthy port behind it. A transient query failure still aborts. * fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`: it resolves the pid-named socket through the pidfile, and OVS opens that O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which also tightens them from the 0644 they ship with. openvswitch-ipsec joins the override list because setup-ovn.sh restarts it after the sweep runs, leaving its ctl socket as the one file still at the shipped mode. Verified on the dev node: every socket and pidfile is 0660 root:spinifex and survives a restart of all three units, an unprivileged user is refused, and /health reports ovn-controller ok where the sudo probe read not_running. * refactor: stop escalating the OVS/OVN socket clients Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the service user. utils.NeedsPrivilege carries that policy and both command builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate while the socket clients no longer do. Their sudoers grants are removed: each took unrestricted args and all of them accept --log-file=PATH, so every one was a root-equivalent grant handed out to run a query. ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that ovs-vswitchd creates when the bridge appears, including bridges spinifex creates at runtime, so the provisioning sweep cannot group-own it up front. The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl. vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing the policy entirely. It survived only because the grants existed and broke the OVN flows-ready barrier the moment they were removed, so it now delegates to utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another package reintroduces a local copy. Verified on the dev node: an instance launches, IMDS is served, the flows-ready barrier completes, and no sudo denial appears anywhere in the journal. * refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant alone was root-equivalent: sudoers cannot constrain its arguments, so `sudo ip netns exec <ns> /bin/sh` was a root shell for the service account. The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel passes an ambient set to each child on exec, so those tools work as the service user with no escalation at all. Verified on a live node: all sixteen ip subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient set. spinifex-vpcd now has no sudoers rules. NeedsPrivilege had to become capability-aware to do this. spx is one binary shared by both service units, and the daemon holds CAP_SYS_ADMIN and CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express "vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb from /proc/self/status once and matches it against the capability each invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the net.* trees and nothing else. The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which runs as the provisioning user, and the in-guest ecs-agent. TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant cannot be re-added without also explaining why the capability stopped working. * chore: restore routeTree.gen.ts swept in by mistake * fix: probe ovn-controller with ovn-appctl, not ovs-appctl checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl, which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's socket and pidfile are in /var/run/ovn, so the probe could never succeed and logged "ovn-controller.pid: open: No such file or directory" on every check. Not a functional break: the next probe globs the full /var/run/ovn path and succeeds, so the function still returned nil. It cost an exec per check and made a healthy node look like it was failing an OVN liveness probe. ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches first. The glob and systemctl fallbacks stay for older layouts. |
||
|
Julian Sommer
|
e7f4ee46b0 | feat: enforce rds:* authorization by principal class and resource (#607) | ||
|
Josh Siviour
|
eff11b611b |
fix: stop upstream DHCP leases leaking out of the external pool (#602)
* fix(dhcp): release stranded leases and label them with the owning node Two paths dropped a lease from KV without returning it to the upstream server, so a source="dhcp" pool leaked addresses permanently. Manager.Start deleted an expired entry outright. Dropping the entry alone abandons the binding rather than returning it, and a server that ages leases by liveness holds the address for as long as anything answers ARP for it. Adoption now sends a bounded, best-effort RELEASE first, skipping leases with no stored offer/ack because nclient4 cannot rebuild those. Release-by-IP answered SUCCESS when the IP was absent from KV, telling the caller the address was freed when nothing went on the wire. It now returns an error. Every ReleaseIP call site logs and continues, so this surfaces the strand without wedging teardown. Leases were also unattributable upstream: chaddr is a derived HashMAC and option 61 carries that same MAC, so no field identified the host that took the lease. Option 12 is now "<node>-<client-id>" and option 60 is "mulga-spinifex/<node>", giving the lease table a per-lease identity and a per-host column to group on. HashMAC is untouched — encoding the node there would cut per-node entropy and reverse a documented decision. Observed live: 20 office-LAN addresses held by one node's orphaned OVN gateway LRPs, with no lease records and no renewals. * fix: rebind the datapath when a DHCP lease moves to a new address A lease re-issued on a different IP was persisted to KV with no notification of any kind. doAcquire replaced the lease after an expiry and doRenew accepted whatever a rebind answered with, while the gateway LRP's address is written once at attach and never revisited. Each such event burned one address permanently: the old one squatted by OVN's ARP responder with no renewer and no DHCPRELEASE, the new one renewed forever attached to nothing. Route both paths through applyLease, which releases the superseded address and invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg so the VPC SNAT and host ingress route follow, and waits on the flows barrier. EIP, ENI-public and NATGW addresses are recorded outside vpcd in DescribeAddresses and DescribeInstances, so the hook reports those rather than repointing a datapath the control plane no longer agrees with. The release half still applies to every purpose, so the leak itself is closed either way. A failed rebind does not fail the lease loop: the new address is held regardless, and dropping it would strand both. * fix: reconcile API-visible records when their DHCP lease moves An EIP or auto-assigned public IP whose lease was re-issued elsewhere left DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had already released. The datapath kept the old dnat_and_snat too, so the address was dead on both sides while the upstream server was free to hand it to someone else. vpcd cannot fix this alone: it holds the lease, but the records naming those addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and the daemon serves, dispatching to the service owning the moved address. RebindPublicIP and RebindENIPublicIP commit the new external IP through vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP, so one commit is enough — and only then move the record, so a NAT failure never leaves a record advertising an address that does not route. Both refuse to move a record naming neither address, so a stale request cannot overwrite an allocation the lease does not own, and both are idempotent under duplicate delivery. A failed or unanswered reconcile propagates back to the lease loop rather than reading as success. vpc.dhcp.lease-expired, listed in the service manifest but never implemented, is replaced by the subject that now exists. * fix: dedupe retried AllocateAddress so one call takes one address EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can outlast the AWS client read timeout, so every SDK-level retry minted a fresh allocation ID and drew another upstream DHCP lease. The caller kept one EIP and the rest stayed leased with no record naming them, so nothing ever released them. Retries are now identified by the SDK's amz-sdk-invocation-id, which is stable across attempts of one call. The gateway lifts it off the request, NATSRequest forwards it as a header, and the daemon puts it back on the context, so the EIP service sees the same token every attempt. singleflight joins a retry to the call still in flight — the common case — and a TTL'd KV record answers retries that arrive after the first call completed. * fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones Three leaks that all end the same way: an address stays leased upstream with nothing renewing it on purpose and nothing that will ever release it. Manager.Start deleted any lease that expired while vpcd was down, after releasing it. The resource it was taken for still existed, so the datapath kept forwarding on an address the upstream server now considers free — a duplicate-IP conflict waiting for the server to reissue it — and nothing ever re-acquired. Expired leases now get a renewal loop like any other, which re-DORAs on its first pass; the existing IP-change hook rebinds the datapath when the new address differs. The reaffirm RENEW is skipped for these, since a lease the server has already aged out will only fail it. Nothing reconciled leases against the resources that justify them, so a resource deleted out from under its lease left the address renewed forever. ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check whether the EIP, ENI or VPC still exists, and releases the lease when it does not. A gateway lease also has a router port to remove, so DetachIGW runs before the address goes back — releasing it while OVN still answers ARP for it invites the same conflict. Anything the daemon cannot resolve is reported unknown and kept: an unreachable daemon must never read as a fleet-wide deletion. The acquire ladder ran to ~62s, just past the 60s read timeout botocore defaults to, which is what turned one AllocateAddress into two leases. It now totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the outcome instead of giving up and retrying. This is a mitigation, not a fix, for callers that send no retry token: botocore emits no amz-sdk-invocation-id, so the dedupe added earlier cannot see their retries at all. Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj. * feat: add spx get dhcp-leases so held external addresses are visible The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV bucket with no read path, so working out which addresses were held, by what, and whether the owner still existed meant writing a NATS client or reading journals. That was the main obstacle to diagnosing every leak in this series. The command lists each lease with its purpose, VPC, address, MAC, holding node and remaining time, and resolves the owner over vpc.dhcp.owner-check so an orphan is visible as such. --orphans narrows the output to leaks; --skip-owner-check drops the lookup when only the addresses matter. Fixes mulga-655iu. * fix(vpcd): reconcile gateway ports onto their lease address at startup A gateway LRP configured from a static pool keeps its address when the pool is switched to source="dhcp", so the VPC forwards on an address nothing leases while the lease it holds goes unused. No lease event ever fires to correct it. vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the address the lease actually holds. RebindGatewayIP is idempotent, so a port already in agreement costs one OVSDB read. * test: cover dhcp lease owner-check handlers and the lease CLI The owner-check path decides whether the reaper releases an address, so every inconclusive branch is asserted to report unknown rather than gone. * fix(dhcp): make the outcome of a stale lease release observable A stale RELEASE was logged only when the transport failed. A lease with no raw offer/ack was skipped in silence, and a RELEASE the server ignored looked identical to one it honoured, so an address left bound upstream could not be traced to either cause. The skip now warns, and a successful release logs the chaddr it used, which is what a server matches a RELEASE against. * fix(dhcp): take the RELEASE chaddr from the lease being released A server matches a RELEASE to a binding on chaddr, and the release path re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed since, that sends an address the binding never had: the server drops the RELEASE and says nothing, so the address stays held with no renewer. The chaddr now comes from the lease record, which carries what took the lease out, falling back to live resolution only when the record has none. |
||
|
Josh Siviour
|
3e195506d3 |
feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605)
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
* refactor(images): extract eks-node initd logic into init-agnostic helpers
The OpenRC runscripts carried their start_pre logic inline, so porting a
service to systemd meant re-deriving that logic under a distro change
rather than just rewriting the init wiring. Move it into /bin/sh helpers
that either init system can drive, keeping eks-node on Alpine/OpenRC.
k3s-prestart takes the restore-block guard, config.yaml derivation,
konnectivity-agent and CoreDNS staging, and the token-webhook wait.
konnectivity-server-prestart takes the CA wait and cert mint, plus the
server-count derivation that used to run at parse time. Both keep every
guard, bound and message as-is.
Values flow back the way the GPU image already does it: the helper writes
shell assignments to a /run file and the init system sources them, so
command_args is built in start_pre where it used to be a top-level
assignment.
k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU
manifest claimed its helpers were reused verbatim from eks-node while
hand-copying this one; it now points at the shared file, so the two
cannot drift.
k3s-agent.initd still sources agent.env and exports itself rather than
delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon
inherits only what the runscript exports, so a helper subprocess sourcing
it would not reach k3s.
The helpers run under set -eu, which the runscripts did not. The awk over
first-boot.env is guarded on the konnectivity manifest, not on the env
file, so a missing env file exits awk nonzero; without || true that turned
a warn-and-continue into a refusal to start k3s. Covered by a test that
fails on rc=2 if the guard is dropped.
* feat(images): build system images in a pinned mkosi container
Move the image-build toolchain into a container so a runner needs only
Docker. The toolchain cannot drift between runners, and the same build
runs identically on a developer's machine.
mkosi builds in a user namespace, which two of Docker's default
confinements block: the seccomp profile masks CLONE_NEWUSER, and the
docker-default AppArmor profile separately blocks unsharing the mount
namespace. Both are relaxed; neither is a privilege grant, as
capabilities stay dropped and no devices are exposed. mkosi also runs as
an ordinary user rather than root, since as root it takes a path that
needs real CAP_SYS_ADMIN.
Verified end to end: an Ubuntu 26.04 Format=disk image builds from the
Debian container and the artefact is a valid GPT disk, with the package
cache on a named volume cutting a forced rebuild from ~2min to ~34s.
* feat(images): publish system images from the build container
Convert mkosi's raw output to a compressed qcow2, write the .sha256
sidecar the spx catalog verifier matches, and upload both to R2 using the
same pinned toolchain as the build, so publishing needs nothing on the
host either. The toolchain image build moves to a shared helper rather
than being duplicated across the two entry points.
Publishing needs no relaxed confinement, unlike the build: qemu-img and
aws create no namespaces, so it keeps Docker's default seccomp and
AppArmor profiles.
The object path carries no version and the catalog resolves newest-wins,
so an upload overwrites the live image with no way back. Copy the current
object to a timestamped archive key first, making a bad publish a restore
rather than an unrecoverable loss.
* fix(images): build the image on the output's mount, not the cache volume
mkosi assembles the image in its workspace and moves the result to the
output directory. The workspace lived under the cache volume, which is a
different mount from the output bind mount even when both sit on one
filesystem, so the move failed EXDEV and mkosi fell back to copying —
requiring the finished image to exist twice at once.
That is invisible on a small image and fatal on a real one: a ~4G GPU
image needed ~8G to land and died with ENOSPC after a successful build.
Point the workspace at the output's own mount so the move is a rename
again and the image only exists once. The package cache stays on the
volume, which is the part worth persisting across runs.
* feat(images): add mkosi base config and gpu-nvidia profile
Establish images/ with a base mkosi.conf plus a gpu-nvidia profile,
replacing the throwaway smoke-test config. The base builds a bootable
UEFI disk; the profile adds the NVIDIA stack.
Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version
hunt, the do_bootloader=no kernel-img.conf hack and the manual
update-initramfs all exist only because that pipeline chroots into a
prebuilt cloud image that already has a kernel. mkosi installs the kernel
and owns the bootloader, so the version is known rather than detected, and
the postinst asserts exactly one kernel instead of searching for it. The
static configuration (nouveau blacklist, unattended-upgrades blacklist,
cloud-init docker group) becomes files under mkosi.extra/ rather than
heredocs.
linux-image-generic drags in every linux-firmware split package, none of
it relevant to a headless VM guest; left at the default the modules initrd
embeds all of it into the UKI, which grew to ~880MB and overflowed the
512M ESP. The initrd module set is trimmed to mkosi-initrd's curated
default, which is a boot fix and not only a size one.
Verified in the built artefact, not just the build log: nvidia.ko.zst and
the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the
nouveau blacklist, and the nvidia runtime in docker's daemon.json.
* fix(images): unblock guest boot on wait-online and firstboot
A guest built from these images stalled twice on the way to a login prompt,
both times on a unit waiting for input that never arrives in a VM.
systemd-networkd-wait-online holds network-online.target until every managed
link is up, and docker.service pulls that target in with Wants=, so any image
carrying docker inherits the wait whether or not it wants it. A guest whose
links are slow or absent ran the unit to its full 120s default with
multi-user.target queued behind it. Override it to --any with a short timeout:
one link up is what a single-NIC guest actually needs.
systemd-firstboot separately prompts on the console for anything the image
does not already carry, before multi-user.target. With no timezone baked in,
boot stopped dead at "Please enter the new timezone name" and waited for a
keystroke. Setting locale, keymap and timezone at build time leaves firstboot
with nothing to ask.
Both are baked into the base rather than delivered as first-boot drop-ins:
an image built from scratch has no cloud-init to write them.
Measured on wattle with an A1000 passed through: login at 42.8s, down from
~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports
"NVIDIA RTX A1000, 595.71.05, 8188 MiB".
Also widen the images gitignore to output-*/ so a scratch build cannot leave
multi-GB artefacts stageable.
* feat(images): make the base image reachable — cloud-init, sshd, console
An image built from scratch launched unreachable. The images this replaces
start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a
network renderer, dbus and a serial console from it; mkosi installs only what
is listed, so none of it was there and nothing said so.
Guests get their SSH key and user-data from IMDS via cloud-init, so without
the package a launched instance has no authorized_keys at all — no error, just
no way in. It also made any cloud-init-delivered config inert, including the
gpu profile's docker-group drop-in.
Each addition is load-bearing on its own:
cloud-init consumes IMDS: keys, user-data, hostname, network
openssh-server the key needs a sshd to be installed into
netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is
never configured, IMDS is never reached, and cloud-init
falls back to DataSourceNone
dbus systemd only Recommends it and mkosi installs no Recommends,
so systemctl could not reach the system bus — and the EKS and
ECS bootstraps drive guests with systemctl from user-data
lsb-release cloud-init shells out to it; without it every boot reports
"degraded" forever and the status stops being a usable signal
The kernel also needs console=ttyS0 baked in. The launcher gives every guest a
serial port and get-console-output reads it, but a kernel with no console=
writes to none of it: the console stopped after the UEFI bootloader and every
boot failure was invisible. ttyS0 goes last because Linux makes the final
console= the controlling /dev/console.
manage_etc_hosts is set because cloud-init renames the guest per launch but at
its default never writes a matching hosts entry, so sudo warned on every call.
datasource_list is deliberately NOT pinned: the guest identifies the platform
from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on
its own, and pinning per image is the customisation this migration deletes.
Verified on wattle with an A1000 passed through: ssh in as ubuntu with
spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed
units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s.
* feat(images): let the guest grow its root into the volume it was given
mkosi sizes the disk to its content, so the image ships with a root
partition that fills the disk to its last sector and ~1.4G free. Nothing
in it can expand that: cloud-init's growpart module is enabled, but the
growpart binary lives in cloud-guest-utils, which is not installed and is
not pulled in as a Recommends by anything in the package set. systemd-repart
is not an alternative — Ubuntu's systemd package carries no repart binary,
unit or definitions at all.
The result was a node handed a larger volume that it could only ever use
the first few GiB of, with the rest allocated and unreachable. That is
fatal for a worker whose job is to pull container images.
gdisk is deliberately left out despite being a cloud-init Recommends:
growpart resizes GPT through sfdisk, which arrives via the hard fdisk
dependency, and e2fsprogs brings the resize2fs that the resizefs module
runs afterwards.
Verified on wattle: the gpu-nvidia image launched with a 20G root reports
20G on df -h / (was 4.4G), via
"'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with
nvidia-smi still reporting the A1000 and no failed units.
* feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles
Adds the eks-common, eks-agent and ecs profiles, so both GPU node images
build from the composed base rather than a hand-authored tree, and the three
verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone.
Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the
standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS
node images run k3s or containerd and reach the GPU through CDI. Fusing the
two was fatal for ECS rather than merely wasteful: gpu-nvidia installed
containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu
containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at
all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs
nvidia-ctk to generate its CDI spec.
Closes three gaps the pilot left, each of which only shows up on a booted
guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the
modules on a headless node and CDI generation finds no GPU;
mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on
the toolkit packages, which the unattended-upgrades blacklist does not cover
for a plain `apt upgrade`.
mkosi-build.sh gains --image, naming an output and expanding it to an ordered
profile list. The order is load-bearing: mkosi runs each profile's postinst in
the order the profiles are passed, so docker's nvidia-ctk wiring must follow
gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a
healthy-looking image with no GPU wiring, so docker's postinst also fails
loudly when gpu-nvidia is selected but nvidia-ctk is absent.
Profiles reference build inputs where they already live via ExtraTrees=
instead of copying them, keeping one copy of the helpers shared with the
Alpine image. Go binaries are built on the host into a staging tree, since the
builder container carries no Go on purpose: go.mod is the single pin for the
toolchain and a second pin in the Dockerfile could disagree with it silently.
* fix(images): install systemd-resolved so the guest has a DNS resolver
mkosi installs only the packages named in Packages= plus their hard dependency
closure, and systemd-resolved is merely a Recommends of systemd. So it was never
installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated,
because netplan renders to systemd-networkd and networkd alone does not write
resolv.conf. glibc then fell back to localhost and every lookup failed with
"[::1]:53 connection refused", which reads as a network fault rather than a
missing package.
A node that cannot resolve cannot pull a container image. The ECS GPU node
booted, registered as a container instance, then failed every task at image
pull. The EKS image carries the same defect but is masked by the cloud-init
bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS
launch path has no equivalent, so only ECS exposed it.
* fix(images): name build output by image and never report a skipped build as success
Two defects in the build wrapper that let the wrong bytes reach a boot test.
Every composition wrote the same spinifex.raw, taken from the base config's
ImageId, so an artefact carried no record of which profiles produced it and an
EKS image was distinguishable from an ECS one only by which build ran last.
Passing --image-id names the output after the image, and as a side effect the
name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses
its default input: --name is the published object and --image the local
artefact, and with nothing tying the two together a defaulted input meant
whatever raw happened to be in the output directory could be uploaded under the
name the caller asked for.
mkosi also skips the build whenever the output path merely exists. It does not
compare inputs, so it never notices an edited mkosi.conf or a restaged binary,
prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable
from a fresh one. Builds are now always forced, and the run additionally asserts
the expected raw was written during this invocation rather than trusting the
exit status. The cache worth keeping is the package download volume, which
--force does not touch.
* feat: add gpu-amd mkosi profile for the ROCm AMI
Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under
mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to
stay alive for one remaining image.
Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04
universe ships its own vendored rocminfo, which unpinned apt preferred over
AMD's, mixing release trains within one image.
Uses a systemd unit for render/video group membership instead of a cloud-init
runcmd, because cloud-init's config merge takes a duplicated runcmd key
wholesale from one fragment and silently discards the other, which would have
dropped the docker profile's own usermod.
Runtime GPU verification is deferred — no AMD hardware is available.
* feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images
Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine
systemd units, a templated etcd-snapshot service with daily and 15-minute
timers, the addon/coredns/konnectivity assets, and the six server binaries.
konnectivity-server is not an in-repo package, so it gets a pinned host-side
clone-and-build rather than an entry in profile_binaries().
Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any
attached volume and mulga-mgmt-net configures any multi-NIC system VM, so
neither belongs behind an EKS profile that a database image would have to
compose to reach them.
Start-at-boot moves into preset files. mkosi runs preset-all after every
postinst, so a postinst enable is discarded before the image is finished, and
with no policy file the fallthrough is enable — which built an image that ran a
control plane and a worker at once.
Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base
ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the
directory in reverse and the merge keeps the first source, so a 90- prefixed
file lost the key silently. The postinst now asserts the merged result.
* refactor(eks-node): make the role selector init-agnostic
eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image
the selector enabled nothing and the node never started k3s. Both now dispatch
through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT
overriding detection so either branch is testable on a host that has only one.
Detection prefers rc-update over systemctl: the test suite stubs rc-update on
PATH, and preferring systemctl would have silently rerouted those cases to a
path they were not written for.
Enable and start stay two separate passes. The server path starts
mulga-eks-k3s-recovery before k3s so a recovery directive applies before the
apiserver comes up, and collapsing to systemctl enable --now would interleave
them.
The dispatch is duplicated in both scripts rather than shared: they are
single-file deliverables with no library-path convention between them.
Tests cover both branches across all three roles, including that server-join
still gets no k3s-first-boot, and that rc-update wins when both inits are
present.
* fix(eks-node): enqueue role-selected units instead of blocking on them
eks-node-role.service declares Before=k3s.service, and
mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a
blocking systemctl start deadlocks: systemd will not run the queued job while
the selector is still activating, and the selector cannot finish until that job
runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie
and the node wedges mid-boot with the role written but no control plane. On a
booted guest the selector sat in activating for 261s until the VM was torn down.
Start with --no-block on systemd. The ordering the blocking call was implicitly
providing is already declared between the units themselves, so systemd
sequences them correctly on its own. OpenRC runs the script directly rather
than scheduling it, so that branch keeps blocking and keeps surfacing failures
through set -e.
* fix: escape $ in konnectivity-server ExecStart so server-count survives
systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before
the command runs, substituting an empty string for a name it does not know.
KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes
it to /run/konnectivity-server-args.env, which the wrapper shell sources at
runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw
it. konnectivity-server then died on flag parsing, which is why the unit logged
Started and produced nothing despite --logtostderr=true and journal+console,
and why the apiserver's cluster egress selector found no UDS for the life of
the cluster.
This is a port artefact. konnectivity-server.initd is flag-for-flag identical
and shares the same prestart script, but under OpenRC the shell was the only
thing reading ${KONN_SERVER_COUNT}, so the same text was correct.
Escape it as $$ to defer expansion to the wrapper shell, and add
lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of
a profile's units. The linter joins line continuations first, since the
offending reference sat mid-way through a wrapped multi-line ExecStart, and
runs on the host before the container starts so a bad unit fails the build in
seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd
leaves it literal.
Verified on env12 with a rebuilt image: TestEKS/IRSAPod and
TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed
out` through the egress tunnel, now pass, with zero UDS connect errors across
the run.
* fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw
mkosi only emits the raw. A qcow2 next to it came from a previous run's hand
conversion, and nothing about its name says which build it belongs to, so it
survives a rebuild and silently ships the old image to a node while the raw
beside it carries the new one. Observed on a rebuild that left the raw at 15:34
carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit.
This is the same failure the existing freshness assertion guards against, one
file over, so extend that block rather than add a second check. Refresh rather
than delete: the qcow2 is what gets copied to a node and imported, so removing
it would push the manual conversion, and this trap, back onto the caller.
Convert via a temp name and rename so an interrupted run leaves the old file
rather than a half-written one that looks current.
* fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc
ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns,
which systemd-networkd does not manage, so it shells out to a DHCP client. That
call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not.
`ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the
failure surfaced as an opaque "network setup failed" with the task stuck in
STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and
ServiceWithELB both failed on the mkosi image, the latter as runningCount=0
because its target group targets task ENI IPs.
Resolve the client at setup time, preferring udhcpc so Alpine behaviour is
unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not
sufficient to select it: Debian and Ubuntu's busybox ship the applet without
/usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits
nonzero having configured nothing. Same order and same reasoning as
dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no
daemon behind in the netns.
Install dhcpcd-base in base, because neither client was present in any profile.
That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved
to nothing on Ubuntu and would have failed the first time a cross-host
control-plane data NIC needed it — the S5 path the step-6 plan notes has never
been exercised.
The DHCP-client choice is injected so the tests assert a named platform's
branch rather than whatever happens to be installed on the machine running
them; without that the same test exercises different code on an Alpine CI box
and an Ubuntu dev box.
* refactor: retire the ec2-node profile, fold its helpers into the base image
ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition
had to remember to list it — eks-common even guarded on its presence and
aborted the build without it. ECS did not list it, so the ECS image shipped
with no by-id symlinks for hot-attached volumes at all.
Neither helper is answerable at build time: any image may later be attached a
volume, and any image may later be launched as a control plane. Both are inert
when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its
fw_cfg blob is absent), so the base image is where they belong. That deletes
the composition rule, the guard, and the ECS gap in one move.
Also drops ec2-node from the two eks image compositions in mkosi-build.sh.
Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both
carry the helpers and the udev rule, mulga-ebs-byid is enabled via
multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants.
* fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link
mulga-vpc-mtu only won the boot race by ordering: netplan renders
UseMTU=true (it does not inherit networkd's own false default), so a
DHCP lease renewal at T1 could silently re-apply the advertised MTU to
the link. usemtu_off() forces UseMTU=false via a runtime drop-in in
the systemd-documented [DHCPv4] section for every rendered unit and
reloads networkd, removing the dependency on boot ordering entirely.
assert_pinned() then verifies the link and its routes actually match
before exiting, so any regression is a loud boot-time failure instead
of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has
no systemd-networkd.
* fix(images): state UseMTU=false under both the current section and the legacy alias
netplan renders the DHCP settings under [DHCP] while systemd documents
[DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the
running systemd disagrees, which is the silent no-op the drop-in exists to
prevent.
* fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch
credendpoint.go DNATs the ECS task credential endpoint via iptables; neither
the mkosi ecs profile nor the legacy Alpine manifest installed it, so the
endpoint silently failed to start and any task with a task role could not
fetch credentials while the instance still looked healthy.
Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only,
which this platform's HTTPS AssumeRole path can never match) and adds a
TaskRoleCredentials subtest that fetches real credentials from inside a task
container, so a dead endpoint fails the suite instead of passing unnoticed.
* fix(images): ship preset files for base, eks-common and ecs start-at-boot
mkosi runs systemctl preset-all after every profile's postinst, discarding
whatever systemctl enable a postinst performed. eks-server and eks-agent
already carry preset files for this reason; base, eks-common and ecs still
relied on the default-enable fallthrough, which happens to match today but
is one upstream package change from reversing.
Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net),
30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and
40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables
cni-dhcp.service/socket, which containernetworking-plugins enables by
default with nothing to serve under the baked host-local IPAM conflist).
Removes the now-redundant systemctl enable calls from each postinst,
matching the eks-server/eks-agent precedent.
* fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers
The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal
denied every Service clause other than the IMDS-only EC2 one. A role carrying the
trust policy AWS documents for an ECS task role was unassumable by construction.
Service attribution is now derived from the authenticated SigV4 caller: an
assumed-role session of the account's ecsInstanceRole — the only credential the
in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles
in its own account. Nothing in the request body contributes, so a task holding
its own task-role session cannot claim the service principal.
* fix(images): remove baked SSH host keys, pin Seed= for reproducibility
openssh-server's postinst bakes an identical host key pair into every
build, making every guest from one build mutually MITM-able; sshd-keygen
already regenerates keys on first boot (ConditionFirstBoot=yes), so the
baked pair is pure non-determinism with no purpose. Also pin Seed= so
systemd-repart stops assigning fresh partition UUIDs on every build.
Verified on a built spinifex-ecs-node image: no ssh_host_* files ship,
cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service
is wired via ssh.service.wants/ (redundant regeneration path). A two-build
comparison dropped content-differing files from 9 to 1 (only the UKI,
whose zstd internals remain sensitive to unrelated input changes).
---------
Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>
|
||
|
Josh Siviour
|
2db1034436 |
feat(acm): managed certificate issuance and renewal from a tenant Private CA (#606)
* fix: fan out certificate re-import to referencing load balancers Re-importing new material under an existing ACM ARN previously updated only the stored CertRecord; no load balancer referencing that ARN ever re-rendered its config, so HAProxy kept serving the old leaf until it expired while the API reported success. Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff around every listener create/modify/delete and cert attach/detach in handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook (wired in daemon.go to avoid an import cycle) that re-runs updateStoredConfig for every load balancer in the ARN's InUseBy set, mirroring the existing target-group fan-out. lbagent is unchanged: it already polls ConfigHash and reloads on change. * fix: make InUseBy index updates CAS-safe under concurrent writers AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which loses concurrent writers silently: two listeners on different load balancers attaching the same certificate at once could drop one entry with no error. Because InUseBy is the sole mechanism that fans a renewed certificate out to the data plane, a lost entry means that load balancer never re-renders and its certificate silently expires in HAProxy while ACM reports it as healthy. Switch both to a JetStream KV revision-based compare-and-swap loop with jittered backoff, retrying on a concurrent writer instead of overwriting it. Replace the old concurrency test, which only asserted no errors/panics and so could not detect a lost index entry, with one that asserts exact final state across N concurrent writers. * fix: encrypt ACM certificate private keys at rest CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key. Store.PutCert encrypts when a master key is set; GetCert/ListCerts decrypt, falling back to a narrowly-bounded legacy-plaintext read (only when the stored value is unambiguously a PEM private key block) so pre-existing records keep working and are re-encrypted on next write. A Store never given a key (ELBv2's independent read-only acmStore) behaves exactly as before — encryption is opt-in per Store instance via the new SetMasterKey method, so no other package needs to change. NewACMServiceImplWithNATS now requires a master key and fails construction without one; daemon.go loads master.key inside the initServiceWithRetry closure so a missing key fails daemon startup after the retry window instead of leaving the service permanently degraded. * fix: wire ELBv2's ACM store with the deployment master key ELBv2 constructed its own handlers_acm.Store without a master key, so once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could no longer resolve listener certificates: it read back ciphertext where it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a key so an unkeyed Store can no longer be constructed, thread the deployment master key through NewELBv2ServiceImplWithNATS and the daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM store construction failure fatal instead of silently degrading every HTTPS listener. * feat(acm): tenant Private CA with x509 name constraints Adds an independent tenant root CA (separate from the platform CA) that signs leaf certificates only within its baked-in permitted domains. Name constraints (PermittedDNSDomains, excluded IP ranges, excluded email/URI) are frozen at creation and drift is rejected rather than silently regenerated, so a device's trust store is never invalidated without an explicit operator action. * docs(acm): record OpenSSL verification of the empty email/URI exclusions Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not just Go's own verifier, since this CA is meant for device trust stores. * feat: implement RequestCertificate with PRIVATE_CA managed issuance Adds the ACM RequestCertificate API surface: mode derivation (PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for the deferred modes. Extends CertRecord with issuance/validation/renewal state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md. * feat: wire tenant CA into the running spinifex daemon Add LoadTenantCA as a load-only entry point (never creates), refactor LoadOrCreateTenantCA to share the load path. Add 'spx admin cert create-tenant-ca' to create the tenant CA with explicit permitted domains, idempotent unless --regenerate is passed. Wire tenant CA loading into daemon startup: missing tenant CA logs a warning and degrades PRIVATE_CA issuance only, it does not block startup like the master key does. Fold tenant CA paths into admin's CA path helpers, keeping filenames distinct from the platform CA (tenant-ca.pem/.key vs ca.pem/.key) so the two roots can never be confused. Also give ACM's DeleteCertificate in-use refusal its own awserrors alias instead of reusing the EKS-labelled one directly, and correct COMMANDS.md to reflect that the tagging endpoints are implemented. * fix: derive ACM validation mode from the requested domain, not a global flag Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that northstar is configured, never whether it is authoritative for the domain being requested. Any deployment with a default domain set therefore returned MANUAL_TXT for every request, including domains hosted nowhere, with a TXT record the caller could not publish. Resolve the requested domain against northstar's actual hosted zones, and put MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly delegated so a public CA can resolve the challenge - which is never true of an internal northstar zone. A tenant CA covering every requested domain now wins, and a configured DNS provider still takes precedence over the manual path so operators with a real domain are not downgraded to a private certificate. * feat: renew PRIVATE_CA certificates automatically before they expire Managed issuance signed a leaf once and never renewed it. With 90-day leaves every certificate silently expired a quarter after the first apply, which is the treadmill managed issuance exists to remove. Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully offline, so this needs no ACME client and no network access. The window is proportional - two thirds of the certificate's own lifetime, floored at a minimum - so it self-tunes as issuance lifetimes change rather than degenerating into 'renew immediately, forever' the way a fixed window would. A deterministic per-ARN jitter keeps certificates issued in one apply from all renewing in the same minute. Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes through the ELBv2 fan-out. The push is mandatory rather than incidental: GetLBConfig returns stored configuration and never re-resolves certificate material, so a renewal that did not push would leave HAProxy serving the old leaf until something unrelated changed the config hash. A per-certificate lease keyed on the ARN means one node drives a given renewal without serialising all renewals behind a single leader. Certificates whose domains have fallen outside the tenant CA's name constraints record a failure reason instead of retrying in a tight loop. The renewal scan reads metadata only and never decrypts stored private keys; the lease is a raw compare-and-swap that leaves the encrypted key untouched. * fix: carry handler error messages over the daemon NATS transport handleNATSRequest responded with the sanitized error code alone, so any handler error without a registered code collapsed to a bare ServerInternal and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx has always preserved the message; the two implementations had drifted. Adds respondWithServiceError and routes the 16 service-error sites through it, plus transport-level tests pinning that the message survives for an uncoded error, that a wrapped coded error keeps its real code, and that a bare code carries no message. * feat: implement ACM GetCertificate and stop decrypting keys on read paths GetCertificate was unimplemented end to end: it now returns the PEM certificate body and chain for an owned ARN, ResourceNotFound for an unknown or cross-account ARN, and RequestInProgressException for a certificate that has a record but no body yet, wired through the service interface, the NATS client, the daemon subscription and the gateway action map. Store gains GetCertMetadata alongside GetCert, and ListCerts becomes ListCertMetadata. Every read that answers a question about a certificate rather than serving it - ownership checks, Describe, Delete, tag reads, GetCertificate and listing - now leaves PrivateKey empty instead of running an AES-GCM open per record to produce plaintext key material no caller reads. * fix: reconcile the ACM master-key wiring with dev's new tests The InUseBy and cert fan-out tests that landed on dev construct the ACM and ELBv2 services directly, and both constructors now take the deployment master key. Pass one shared key in the fan-out test - two services over the same bucket disagreeing on the key is the exact failure the wiring exists to stop. A Store is never unkeyed now, so the two test helpers collapse into one. * fix: rebuild the ACM InUseBy index at daemon startup The index that drives certificate fan-out is written only by listener mutations, so a listener whose certificates have not been touched since it was created has no entry. A certificate with an empty index then neither fans out on renewal nor trips DeleteCertificate's in-use guard, and both failures are silent: the renewal reports success while the load balancer keeps serving the old leaf, and a certificate a live listener depends on can be deleted. Reconcile the index from the persisted listener records once at startup. Walking every certificate rather than only the referenced ones makes it a rebuild rather than a backfill, so it also drops entries naming a load balancer that no longer exists. Writing only differences keeps it idempotent, and the index writes are already CAS-guarded, so the nodes of a cluster reconciling the same bucket converge. * fix: deliver listener SNI certificates to HAProxy Add/RemoveListenerCertificates persisted the listener record and then returned, so the config was never re-rendered and lbagent had nothing to poll for. Re-render through reloadListenerLB, the same helper the rule mutations use. That alone changed nothing, because the renderer dropped SNI certificates: resolveFrontendCert returned only the default certificate and the bind line emitted a single crt, with the PEM path keyed on the listener so a second certificate had nowhere to live. Render every attached certificate instead, default first so it stays the no-SNI fallback, each with its own path. HAProxy accepts crt more than once on a bind line and selects by SNI. The default certificate keeps its existing filename so an upgrade does not move a deployed PEM, and the nginx renderer keeps the single-certificate resolver it still relies on. * fix: prune stale TLS certs from the lb-agent cert dir A delivered PEM is cert + chain + private key. writeCertFiles only ever wrote, so every certificate detached from a listener left its key on the load balancer VM's disk indefinitely. Prune after a successful reload, scoped to .pem files in the cert dir's immediate entries. * fix: serve every attached certificate on the nginx NLB engine nginx allows one ssl_certificate per key type in a server block, and the NLB config is a stream block where server_name does not exist. Select by SNI instead: give ssl_certificate a variable and map $ssl_server_name to each certificate's path, keyed on the leaf's DNS SANs. A single-certificate listener keeps a literal path, since a variable costs a per-handshake read. The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so they now take the dir and both engines share one multi-cert resolver. * fix: emit ValidationMethod in DomainValidationOptions DescribeCertificate returned each domain validation entry without a ValidationMethod, so the AWS provider had nothing to compare against aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes. PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits the method on a private certificate, where the provider reads it as NONE whatever the request asked for. |
||
|
Julian Sommer
|
edf2113f54 | feat: schedule RDS automated backups and sweep them past retention (#604) | ||
|
Josh Siviour
|
2638a792c5 |
fix(ecs): task-definition round-trip fidelity and a task that dies while the agent is down (#603)
Persist and echo requiresCompatibilities on task definitions, so a Terraform round trip no longer reads null as drift and force-replaces the definition on every apply. Report a task whose containers all exited while the ecs-agent was down. reconcile only re-adopted running containers, so the gateway held such a task RUNNING forever, its service never replaced it, and the load balancer target stayed unhealthy. |
||
|
Julian Sommer
|
d2eb3b11dd | feat: add manual DB snapshots and restore (#601) | ||
|
Julian Sommer
|
d4d9db9e0b | feat: add DB subnet groups and DB parameter groups (#600) | ||
|
Josh Siviour
|
a0659ae7d0 |
fix: gate ovn-controller on SB RAFT readiness and force recompute (#599)
Single-node OVN RAFT bring-up gated only on the Northbound DB socket, then restarted ovn-controller without waiting for the Southbound RAFT to elect a leader, serve clients, and for northd to write the gateway logical flows. With no join step to absorb the boot window the controller attached mid-election, programmed a partial datapath (SNAT ct-commit without output/delivery) and never reconverged, so NAT-GW egress and EIP ingress intermittently black-holed on the single-node baremetal CI host. Add an SB readiness gate before ovn-controller starts (SB reachable and, when clustered, an elected leader), and after ovn-controller restart wait for its SB connection then force inc-engine/recompute to rebuild all OpenFlow from the converged Southbound. Applies to 1/2/3-node RAFT alike. |
||
|
Julian Sommer
|
53c0671512 |
feat: rds-6 failure detection — an honest health state for a DB instance
The failed state existed and was unreachable: nothing drove available → failed, so DescribeDBInstances reported available for a database that had been dead for a week. Add the failure classifier on top of the rds-4 reconciler. An instance is failed when its system VM is not running AND its agent heartbeat is stale — never either alone, since a stale beat under a live VM is an agent fault rather than a dead database. The transition records a timestamped reason that DescribeDBInstances and DescribeEvents can explain, and a healthy heartbeat clears it again, which is the recovery the AMI's in-guest Restart=on-failure and EC2's VM auto-restart provide underneath. The recovery ladder and its fencing gate stay deferred to v1.1. The failure clock is persisted rather than held in leader memory, so a leader change does not restart the window, and it resets only on a healthy heartbeat so a VM that boots and immediately wedges cannot mask a persistent fault. Every lifecycle-owned state is skipped outright, which is what keeps the classifier from racing rds-5a/rds-5b over the same VM. A persisted heartbeat is judged against the stale window plus the persist floor: beats are queue-group delivered, so the leader mostly sees them only through the record, which a healthy agent refreshes no more often than every five beats. The steady-state path is answered from the record alone, so a healthy fleet costs no VM describe fan-outs. |
||
|
Julian Sommer
|
23de2ad85d | feat: ModifyDBInstance and the shared VM-replace helper (#597) | ||
|
Josh Siviour
|
7d2106bfb5 |
fix(test): reclaim fixture scratch dirs stranded in /tmp (#598)
* fix(test): reclaim fixture scratch dirs stranded in /tmp The shared predastore fixture and the integration suite's JetStream store both outlive any individual test, so neither can use t.TempDir() - the first test to finish would delete files the daemon still has open. Both used os.MkdirTemp with nothing removing the directory afterwards, so every run stranded one, and a run killed by a timeout panic or the OOM killer stranded it regardless. /tmp is a tmpfs on the dev boxes, so the leak is charged to RAM and swap, and swapped tmpfs pages have no backing store to be dropped to. Add a shared sweep that reclaims a directory only when it matches both the prefix and an age no live run can reach. The NATS store also has a clean shutdown point, so TestMain now removes it outright and the sweep is left to cover the killed-run case. The predastore daemon runs until the process exits and has no such point, so the sweep is its only mechanism. * chore: repin viperblock to the dev tip carrying the scratch-dir fix Test-only upstream change, so the build output is unchanged - this keeps go.mod aligned with the submodule pointer rather than delivering a fix. |
||
|
Julian Sommer
|
fc6e055ca0 | feat: RDS lifecycle core — reboot, start, stop, delete, events (#596) | ||
|
Julian Sommer
|
6eb76b439b | feat: RDS tag actions, --tags on create, and the ARN parser (#595) | ||
|
tomnewton-mulga
|
51ecddbf81 | routeTree file | ||
|
tomnewton-mulga
|
27aa2394b3 | chore: Updated RTX-6000 RA Doc |