mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
2db103443653ea64b81908bc3fc4ac07610fa11f
68 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
2db1034436 |
feat(acm): managed certificate issuance and renewal from a tenant Private CA (#606)
* fix: fan out certificate re-import to referencing load balancers Re-importing new material under an existing ACM ARN previously updated only the stored CertRecord; no load balancer referencing that ARN ever re-rendered its config, so HAProxy kept serving the old leaf until it expired while the API reported success. Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff around every listener create/modify/delete and cert attach/detach in handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook (wired in daemon.go to avoid an import cycle) that re-runs updateStoredConfig for every load balancer in the ARN's InUseBy set, mirroring the existing target-group fan-out. lbagent is unchanged: it already polls ConfigHash and reloads on change. * fix: make InUseBy index updates CAS-safe under concurrent writers AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which loses concurrent writers silently: two listeners on different load balancers attaching the same certificate at once could drop one entry with no error. Because InUseBy is the sole mechanism that fans a renewed certificate out to the data plane, a lost entry means that load balancer never re-renders and its certificate silently expires in HAProxy while ACM reports it as healthy. Switch both to a JetStream KV revision-based compare-and-swap loop with jittered backoff, retrying on a concurrent writer instead of overwriting it. Replace the old concurrency test, which only asserted no errors/panics and so could not detect a lost index entry, with one that asserts exact final state across N concurrent writers. * fix: encrypt ACM certificate private keys at rest CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key. Store.PutCert encrypts when a master key is set; GetCert/ListCerts decrypt, falling back to a narrowly-bounded legacy-plaintext read (only when the stored value is unambiguously a PEM private key block) so pre-existing records keep working and are re-encrypted on next write. A Store never given a key (ELBv2's independent read-only acmStore) behaves exactly as before — encryption is opt-in per Store instance via the new SetMasterKey method, so no other package needs to change. NewACMServiceImplWithNATS now requires a master key and fails construction without one; daemon.go loads master.key inside the initServiceWithRetry closure so a missing key fails daemon startup after the retry window instead of leaving the service permanently degraded. * fix: wire ELBv2's ACM store with the deployment master key ELBv2 constructed its own handlers_acm.Store without a master key, so once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could no longer resolve listener certificates: it read back ciphertext where it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a key so an unkeyed Store can no longer be constructed, thread the deployment master key through NewELBv2ServiceImplWithNATS and the daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM store construction failure fatal instead of silently degrading every HTTPS listener. * feat(acm): tenant Private CA with x509 name constraints Adds an independent tenant root CA (separate from the platform CA) that signs leaf certificates only within its baked-in permitted domains. Name constraints (PermittedDNSDomains, excluded IP ranges, excluded email/URI) are frozen at creation and drift is rejected rather than silently regenerated, so a device's trust store is never invalidated without an explicit operator action. * docs(acm): record OpenSSL verification of the empty email/URI exclusions Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not just Go's own verifier, since this CA is meant for device trust stores. * feat: implement RequestCertificate with PRIVATE_CA managed issuance Adds the ACM RequestCertificate API surface: mode derivation (PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for the deferred modes. Extends CertRecord with issuance/validation/renewal state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md. * feat: wire tenant CA into the running spinifex daemon Add LoadTenantCA as a load-only entry point (never creates), refactor LoadOrCreateTenantCA to share the load path. Add 'spx admin cert create-tenant-ca' to create the tenant CA with explicit permitted domains, idempotent unless --regenerate is passed. Wire tenant CA loading into daemon startup: missing tenant CA logs a warning and degrades PRIVATE_CA issuance only, it does not block startup like the master key does. Fold tenant CA paths into admin's CA path helpers, keeping filenames distinct from the platform CA (tenant-ca.pem/.key vs ca.pem/.key) so the two roots can never be confused. Also give ACM's DeleteCertificate in-use refusal its own awserrors alias instead of reusing the EKS-labelled one directly, and correct COMMANDS.md to reflect that the tagging endpoints are implemented. * fix: derive ACM validation mode from the requested domain, not a global flag Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that northstar is configured, never whether it is authoritative for the domain being requested. Any deployment with a default domain set therefore returned MANUAL_TXT for every request, including domains hosted nowhere, with a TXT record the caller could not publish. Resolve the requested domain against northstar's actual hosted zones, and put MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly delegated so a public CA can resolve the challenge - which is never true of an internal northstar zone. A tenant CA covering every requested domain now wins, and a configured DNS provider still takes precedence over the manual path so operators with a real domain are not downgraded to a private certificate. * feat: renew PRIVATE_CA certificates automatically before they expire Managed issuance signed a leaf once and never renewed it. With 90-day leaves every certificate silently expired a quarter after the first apply, which is the treadmill managed issuance exists to remove. Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully offline, so this needs no ACME client and no network access. The window is proportional - two thirds of the certificate's own lifetime, floored at a minimum - so it self-tunes as issuance lifetimes change rather than degenerating into 'renew immediately, forever' the way a fixed window would. A deterministic per-ARN jitter keeps certificates issued in one apply from all renewing in the same minute. Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes through the ELBv2 fan-out. The push is mandatory rather than incidental: GetLBConfig returns stored configuration and never re-resolves certificate material, so a renewal that did not push would leave HAProxy serving the old leaf until something unrelated changed the config hash. A per-certificate lease keyed on the ARN means one node drives a given renewal without serialising all renewals behind a single leader. Certificates whose domains have fallen outside the tenant CA's name constraints record a failure reason instead of retrying in a tight loop. The renewal scan reads metadata only and never decrypts stored private keys; the lease is a raw compare-and-swap that leaves the encrypted key untouched. * fix: carry handler error messages over the daemon NATS transport handleNATSRequest responded with the sanitized error code alone, so any handler error without a registered code collapsed to a bare ServerInternal and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx has always preserved the message; the two implementations had drifted. Adds respondWithServiceError and routes the 16 service-error sites through it, plus transport-level tests pinning that the message survives for an uncoded error, that a wrapped coded error keeps its real code, and that a bare code carries no message. * feat: implement ACM GetCertificate and stop decrypting keys on read paths GetCertificate was unimplemented end to end: it now returns the PEM certificate body and chain for an owned ARN, ResourceNotFound for an unknown or cross-account ARN, and RequestInProgressException for a certificate that has a record but no body yet, wired through the service interface, the NATS client, the daemon subscription and the gateway action map. Store gains GetCertMetadata alongside GetCert, and ListCerts becomes ListCertMetadata. Every read that answers a question about a certificate rather than serving it - ownership checks, Describe, Delete, tag reads, GetCertificate and listing - now leaves PrivateKey empty instead of running an AES-GCM open per record to produce plaintext key material no caller reads. * fix: reconcile the ACM master-key wiring with dev's new tests The InUseBy and cert fan-out tests that landed on dev construct the ACM and ELBv2 services directly, and both constructors now take the deployment master key. Pass one shared key in the fan-out test - two services over the same bucket disagreeing on the key is the exact failure the wiring exists to stop. A Store is never unkeyed now, so the two test helpers collapse into one. * fix: rebuild the ACM InUseBy index at daemon startup The index that drives certificate fan-out is written only by listener mutations, so a listener whose certificates have not been touched since it was created has no entry. A certificate with an empty index then neither fans out on renewal nor trips DeleteCertificate's in-use guard, and both failures are silent: the renewal reports success while the load balancer keeps serving the old leaf, and a certificate a live listener depends on can be deleted. Reconcile the index from the persisted listener records once at startup. Walking every certificate rather than only the referenced ones makes it a rebuild rather than a backfill, so it also drops entries naming a load balancer that no longer exists. Writing only differences keeps it idempotent, and the index writes are already CAS-guarded, so the nodes of a cluster reconciling the same bucket converge. * fix: deliver listener SNI certificates to HAProxy Add/RemoveListenerCertificates persisted the listener record and then returned, so the config was never re-rendered and lbagent had nothing to poll for. Re-render through reloadListenerLB, the same helper the rule mutations use. That alone changed nothing, because the renderer dropped SNI certificates: resolveFrontendCert returned only the default certificate and the bind line emitted a single crt, with the PEM path keyed on the listener so a second certificate had nowhere to live. Render every attached certificate instead, default first so it stays the no-SNI fallback, each with its own path. HAProxy accepts crt more than once on a bind line and selects by SNI. The default certificate keeps its existing filename so an upgrade does not move a deployed PEM, and the nginx renderer keeps the single-certificate resolver it still relies on. * fix: prune stale TLS certs from the lb-agent cert dir A delivered PEM is cert + chain + private key. writeCertFiles only ever wrote, so every certificate detached from a listener left its key on the load balancer VM's disk indefinitely. Prune after a successful reload, scoped to .pem files in the cert dir's immediate entries. * fix: serve every attached certificate on the nginx NLB engine nginx allows one ssl_certificate per key type in a server block, and the NLB config is a stream block where server_name does not exist. Select by SNI instead: give ssl_certificate a variable and map $ssl_server_name to each certificate's path, keyed on the leaf's DNS SANs. A single-certificate listener keeps a literal path, since a variable costs a per-handshake read. The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so they now take the dir and both engines share one multi-cert resolver. * fix: emit ValidationMethod in DomainValidationOptions DescribeCertificate returned each domain validation entry without a ValidationMethod, so the AWS provider had nothing to compare against aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes. PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits the method on a private certificate, where the provider reads it as NONE whatever the request asked for. |
||
|
Julian Sommer
|
e79ca59d4b | fix: configure host DNS from spx admin init/join (#582) | ||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Julian Sommer
|
b40d538c61 | refactor(admin): remove dead --no-external flag from spx admin init (#537) | ||
|
Julian Sommer
|
744cb1929d | fix: admin init --force preserves keys, credentials and CA (#536) | ||
|
Josh Siviour
|
183a08353e |
feat(network): harden routed-NAT v1 (VPC cap, e2e gate, docs) (#504)
Three related nat-mode fixes: - ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template so operators can widen any pool. - 724mm: detectPoolMode returns true only for external_mode=pool. nat and disabled clusters now skip Phase 8b/8d, which otherwise failed on AllocateAddress UnsupportedOperation. - pnz2i: document that system instances (ECS/EKS/LB agents) are unsupported in nat v1 (require pool mode) in the vpc-networking README and COMMANDS.md; document the gw_lrp_range_* fields. |
||
|
Josh Siviour
|
d663760dec |
feat(eks): control-plane disaster recovery — etcd snapshot + restore-snapshot (#503)
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for
|
||
|
Julian Sommer
|
8fe8a9436c | feat(ec2): EC2 Launch Templates (#498) | ||
|
Julian Sommer
|
5d199361b6 | feat: implement IAM GetAccountSummary (#484) | ||
|
Josh Siviour
|
e1c4f85a36 |
feat: routed-NAT external mode with two-tier ingress (#475)
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets. |
||
|
Josh Siviour
|
a7b689fec3 |
feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479)
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits |
||
|
tomnewton-mulga
|
4f5fd88c86 |
feat: snapshot ami (#279)
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation - Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe) - Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels - Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type) - Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles - GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors * fix(daemon): enforce CPU/memory limits for MIG instance types Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate). MIG types now fall through to canAllocateCount so host CPU and RAM are checked before launch — prevents over-committing a host where one physical GPU backs multiple concurrent slices, each with non-trivial vCPU/memory allocation. * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * fixed testing * more tests * fix: added new graceful process killer in utils, bumped NATS timeout for testing * fix: complete at-rest encryption launch + volume lifecycle (#361) * chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against the fix for the encrypted AMI snapshot metadata tag-verify failure (cipher: message authentication failed) on instance launch. Pairs with viperblock PR #22. * fix: forward viperblock master key to nbdkit mount path viperblockd loaded the master key but never passed it to the nbdkit plugin it spawns for volume mounts, so encrypted boot volumes opened in cleartext and failed LoadState with an encryption-state mismatch. Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's encryption_key_file= argument when set. * chore: bump viperblock to fresh-volume VolumeUUID mint fix * chore: bump viperblock to snapshot-link LoadState ordering fix * fix: route encrypted-volume config writes through viperblock keyholder config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived nonce can only be advanced by the master-key holder that owns the volume. The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to refuse encrypted volumes (rewriting would strip the tag) — bricking UpdateVolumeState (attach in-use / detach available) and ModifyVolume. Route encrypted-volume config updates through viperblockd, mirroring the ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config queue group opens the detached volume exclusively and reseals. Cleartext volumes keep the direct-S3 fast path. No crypto leaves viperblockd. mulga-siv-323 * fix: unwrap encryption envelope when snapshot handler reads config.json CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into a VBState. For encrypted volumes config.json is an at-rest envelope {payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so CreateSnapshot 500s on the size guard and the in-use scan silently drops encrypted volumes' snapshots. Unwrap via viperblock.StateBody first, mirroring the volume handler's mergeVolumeConfig path. mulga-siv-324 * test: cover viperblockd ebs.config handlers Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler (ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only size reconcile, live-VB apply, invalid JSON, apply errors, and the detached-volume open path. Clears the diff-coverage gate (68.8%). * chore: pin viperblock to dev tip (#22 merged) * fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) * feat: cloud-init EC2 datasource (#411) * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * merge changes * chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never set post-rebase. Removes the field from MountedVolume and the two nil-checked unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig. * fix: post-rebase gofmt fixes and remove dead snapshot handler code Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go, viperblockd.go, volumes.go, and volume_durability_test.go. Removes the dead unix-socket snapshot handler tests from viperblockd_handlers_test.go. * test: add viperblock snapshot S3 coverage tests Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume past the nil-config guard using an httptest server that returns 403. The AWS SDK credential check fails synchronously without network I/O, keeping the tests fast while covering the S3Config construction, master key load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%. * fix: cloneAMIToVolume reads config via format-agnostic path LoadStateRequest expects a viperblock binary envelope (encrypted), but CreateImageFromInstance writes the AMI config as plain JSON via putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig which uses viperblock.StateBody to handle both formats, matching the path used by ListImages and all other AMI readers. * fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint. CI builds without go.work so it resolved to the published version and failed. Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged. * fix: replace indexOf with slices.Index, revert e2e runner and viperblock version - nbd_test.go: remove local indexOf helper, use slices.Index from stdlib - e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting - go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges) * fix: remove pseudo-version pin from go.sum * feat: drain viperblock via snapshot socket before CreateSnapshot Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock and wait for DrainToBackend() to complete (30s timeout). This ensures WAL chunks and the live checkpoint are current in S3 before the snapshot VB reads them. If the socket is absent the instance is stopped and Close() already drained. fix: socket path used s.config.WalDir which is empty in the daemon config; changed to s.config.DataDir + "/viperblock" to match the path the NBD plugin uses when creating the socket. * feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint - Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot) - Deregister from image detail (DeregisterImage with navigate-away on success) - Promote to System from image detail (admin-only, irreversible via UI) - NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage - signedFetch extended with optional params to support action-keyed extra fields * refactor: address PR review on PromoteImage types and subscription ordering - Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult (add JSON tags; drop the duplicate pair) - Inline the input variable in gateway/spx/promote_image.go - Move spinifex.image.promote subscription to the spinifex.* group in daemon.go * test: cover snapshotVolume/Running/Stopped error paths for diff coverage Add encryption-key-load-error and LoadState-failure tests for the three new viperblock snapshot paths (snapshotVolume in snapshot service, snapshotRunningVolume and snapshotStoppedVolume in image service). Diff coverage: 58.8% → 65.5%. --------- Co-authored-by: Josh Siviour <mrk@mulgadc.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Julian Sommer
|
81212fe91b | feat: EC2 central-record resource tagging (#469) | ||
|
Julian Sommer
|
b756798ed6 | feat: IAM resource tagging (#467) | ||
|
Julian Sommer
|
8641629313 | feat: IAM user inline policies (#462) | ||
|
Julian Sommer
|
63437f759b | feat: IAM Group inline policies (#455) | ||
|
Julian Sommer
|
c9629d3bf6 | feat: IAM Groups v1 (#449) | ||
|
Julian Sommer
|
988a9b305c | feat: EC2 Spot Instances v1 (mock SIRs over RunInstances) (#448) | ||
|
Julian Sommer
|
485cca941f | feat: IAM inline role policies (Put/Get/DeleteRolePolicy + real ListRolePolicies) (#429) | ||
|
Julian Sommer
|
7ffe12ac82 | feat: EC2 instance metadata-options API (IMDSv2-only, faithfully reported) (#428) | ||
|
Julian Sommer
|
8a94886b13 | feat: cloud-init EC2 datasource (#411) | ||
|
Josh Siviour
|
9e333c8a6c |
feat: graceful guest drain on systemctl stop / host shutdown (#412)
* fix: drop removed predastore s3.Config.Compaction from admin_test The predastore submodule bump removed the Compaction field from s3.Config. Parse the rendered [compaction] block into a local struct so the template round-trip test no longer references the deleted field. * feat: graceful guest drain on systemctl stop / host shutdown systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU was still writing, corrupting guest filesystems. The ordered drain existed only over NATS via cluster shutdown, never on the systemd/host path. Add spx admin node drain --local (GATE+DRAIN against the local node, leaving STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new spinifex-shutdown.service oneshot ordered After= all services so it stops first and drains guests while every service is still up. DDIL preserved: the oneshot fires only on target/host shutdown, not on a daemon restart. Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an nbdkit a guest is still attached to. Add a units ordering invariant. * fix: ExecStop drain timeout needs duration unit (120 -> 120s) The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin node drain's --timeout is a cobra Duration flag and rejects a bare integer (time: missing unit in duration "120"). The drain exited 1 without running, so a systemctl stop tore nbdkit out from under the running guest and corrupted its ext4 (the exact failure this feature prevents). Pass 120s. * fix: nbdkitInUse read ss state from correct column + cover it ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0] (field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit read as idle and the SIGTERM path would reap it under a live guest. Add tests exercising a real connected unix socket, an absent socket, and TCP transport. |
||
|
Julian Sommer
|
98db2c4e0c | feat(imds): serve network/interfaces/macs/<mac> subtree (single-NIC) (#410) | ||
|
Julian Sommer
|
ad05544075 | feat: EC2 capacity reservations — launch into reservation (#397) | ||
|
Julian Sommer
|
83e538f2f2 | feat: IMDS version discovery, instance-identity document, and cheap metadata routes (#382) | ||
|
Julian Sommer
|
996eb370f3 | feat: Capacity Reservations (#377) | ||
|
Julian Sommer
|
89ec47a69b | chore: update COMMANDS.md | ||
|
Julian Sommer
|
c4926d423f | feat: IMDS public-keys (#350) | ||
|
Julian Sommer
|
19de3b0994 | chore: update COMMANDS.md | ||
|
Julian Sommer
|
05e66bd6a0 | feat: ListPolicyVersions | ||
|
Josh Siviour
|
c7f337229f |
feat(vpcd): drain held DHCP leases on teardown (#346)
DHCP-source external-pool leases are DHCPRELEASEd only on explicit instance/EIP terminate. Manager.Stop() preserves leases for adopt-on-reboot, so env teardown/reset kills vpcd and destroys the per-AZ lease KV without releasing — stranding every live lease (instance ENIs, EIP allocations, per-VPC LRP gateways) on the upstream DHCP server until TTL. Across CI churn with fresh client-ids the upstream scope fills and new launches get no public IP. Add Manager.DrainAll + a vpc.dhcp.drain RPC and a standalone 'spx admin cluster drain-dhcp' command that releases every held lease. Wired into both teardown paths (single-node teardown role and multi-node cluster-teardown.yml) before services stop. Stop() is unchanged, so normal restarts still adopt leases on reboot. |
||
|
Julian Sommer
|
15cd1a1ad7 | feat: sts get session token (#321) | ||
|
Josh Siviour
|
554c5d96c7 |
feat(elbv2): NLB L4 data plane on nginx stream + UI parity (#295)
* feat(elbv2): NLB L4 data plane on nginx stream (mulga-siv-952) Network load balancers render an nginx `stream` config (TCP/UDP/TLS/ TCP_UDP) instead of HAProxy, which cannot load-balance UDP in any release. The gateway tags GetLBConfig with Engine=nginx; the LB agent dispatches to the nginx engine (config/pid/cert paths + reload) and skips HAProxy stats polling. ALB stays on HAProxy. nginx is added to the microvm image. * docs(elbv2): note NLB runs nginx stream, ALB runs HAProxy (mulga-siv-952) * fix(elbv2): load nginx stream module for NLB data plane Alpine's `nginx` apk is built without the stream module — it ships separately as `nginx-mod-stream`. Without it, the generated NLB config fails `nginx -t` with [emerg] unknown directive "stream", so the agent never applies it. Add the package to the microvm image and emit a `load_module .../ngx_stream_module.so;` directive at the top of the generated config. * fix(elbv2-ui): allow single-subnet LB + full NLB protocols in listener edit Create-LB form required min 2 subnets while the CLI/backend accept 1; relax to min(1) to match. Listener add/edit only offered HTTP/HTTPS — add the NLB protocol set (TCP, UDP, TLS, TCP_UDP), driven by LB type, with cert/ssl-policy fields gated on secure protocols (HTTPS/TLS). * feat(elbv2): active target health checks for nginx NLBs nginx stream has no active upstream probing, so NLB targets never advanced past initial — DescribeTargetHealth stayed initial forever and WaitForTargetsHealthy timed out for every NLB. GetLBConfig now delivers per-target HealthTarget specs (daemon-computed server name, probe address, protocol/path) for network LBs; the lb-agent probes each target every tick (TCP dial or HTTP/HTTPS GET) and reports UP/DOWN via the existing heartbeat path, so the daemon's threshold hysteresis advances targets to healthy. ALBs are unchanged (HAProxy stats). Adds an Internal_NLB_UDP e2e suite gating on WaitForTargetsHealthy plus a UDP datagram round-trip through the VIP. |
||
|
Julian Sommer
|
bcb49cd771 | feat: IMDS v1 (#266) | ||
|
Josh Siviour
|
f14fb64562 |
feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288)
* feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first. |
||
|
Josh Siviour
|
bbff004b45 |
feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280)
* feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep
Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first
read/write helpers against the eks-account-{accountID} bucket so later
stages (oidc keypair, NLB wiring, cluster reconciler) can persist state
without re-inventing CAS/sweep logic.
- ClusterMeta struct holds the DescribeCluster source of truth (status,
endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN).
- ClusterStatus enum mirrors the AWS EKS string set verbatim.
- PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta
returns ErrClusterNotFound on key absence so the service boundary can
map to ResourceNotFoundException.
- SetClusterStatus is revision-checked with bounded CAS retries so the
reconciler and CreateCluster body cannot wedge each other.
- DeleteClusterPrefix sweeps every clusters/{name}/* key; called from
DeleteCluster after the OIDC private key is zeroized.
Tests cover round-trip, missing/corrupt blob, idempotent status set,
DELETING transition from every prior state, concurrent revision bump
recovery, sibling-cluster preservation, empty-bucket no-op.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize
Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body
will call before launching the K3s server VM. Pure crypto helpers — no
daemon, no launcher, no NATS bus.
- GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster
(matches eks-v1.md Q8), PEM-marshals the private as PKCS8,
envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM,
AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and
publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS
bytes so CreateCluster can hand them straight to user-data without
a follow-up KV read.
- JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA
verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER,
alg ES256, use sig). Types are defined locally to avoid a
handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire
compat is the contract.
- LoadClusterOIDCPrivateKey is the recovery-path read for future
stages (reconciler restart, restore-snapshot tooling). Validates
PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else.
- ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of
the same length, then deletes. DeleteCluster calls this *first* so a
crash mid-teardown leaves no recoverable key material. Idempotent
on missing key — supports retry semantics.
Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1
coordinates), per-cluster key uniqueness, round-trip private-key
decrypt matches JWKS public, wrong-master-key fails, missing-key
ErrClusterNotFound mapping, zeroize idempotency, JWKS survives
zeroize (carries no secret material), padCoord left-pad correctness.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): cluster SG + NLB ensure/delete helpers
Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account
SG provisioning + per-cluster NLB plumbing the CreateCluster body will
call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure
interface-call helpers — no daemon, no KV, no NATS bus.
- EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg
and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged
spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role.
Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort
with continue-on-error sweep.
- EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the
eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md
Q13), and the TCP :443 forward listener. Returns ClusterNLB
{LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster
can persist every ARN in ClusterMeta. Idempotent on cluster name;
resource-name length validated against the AWS 32-char cap so a too-long
cluster name fails with a clear error instead of a generic
InvalidParameterValue downstream.
- RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM
ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its
ENI IP; DeleteCluster calls Deregister before VM termination.
- DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already
cascades listener cleanup inside the ELBv2 impl. LB sweep error takes
precedence so DeleteCluster surfaces the more visible failure first
while the TG sweep still runs.
Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests
hand-rolled recorder fakes; the production VPCService + ELBv2Service
satisfy them structurally so no adapter is needed at Stage 6 wire-up.
Tests cover fresh-creates with right names/tags/ports, idempotent reuse,
mixed-existence (control-plane exists, nodegroup created), name-too-long
rejection, empty-inputs rejection, missing-resource no-op on delete, and
first-error-surfaced sweep continuation on delete.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(ec2): RunInstances honors pre-created NetworkInterfaceId
Closes the AWS-parity gap where RunInstances always auto-created a
primary ENI even when the caller specified
NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster
launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the
ENI can be tagged/owned distinctly from the launcher's instance; the
existing path silently created a second ENI and left the caller's
unattached.
Changes:
- ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo
is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status,
SecurityGroupIDs) so handlers/ec2/instance stays free of an import on
handlers/ec2/vpc.ENIRecord.
- daemon/vpc_adapter.go gains daemonENICreator.GetENI translating
VPCServiceImpl.GetENIRecord into ENIInfo.
- PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId
after the existing subnet/SG lift block. If non-empty, the loop
dispatches to a new attachPreCreatedENI helper:
- Looks up the ENI under accountID.
- Rejects with InvalidNetworkInterface.InUse if Status == "in-use".
- Calls AttachENI to mark the ENI in-use and bind it to the new VM.
- Populates instance.ENIId / ENIMac and ec2.Instance network fields
(PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]).
- Skips the auto-create + public-IP-auto-assign paths; the caller is
expected to manage public IP out-of-band.
- Auto-create path is left untouched for every other code path: callers
that pass SubnetId without a NetworkInterfaceId still get a fresh ENI
with the existing semantics.
Tests cover the four shapes: happy path (auto-create not called, attach
called once, instance populated), in-use ENI rejected with the canonical
AWS error code, lookup error surfaces verbatim, attach error surfaces
verbatim.
* feat(eks): K3s control-plane VM launcher with cloud-init user-data
LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the
customer-account control-plane ENI in the supplied subnet with the
control-plane SG, renders cloud-init user-data (first-boot env, OIDC
private key, K3s config.yaml with tls-san + service-account args), then
calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing
at the pre-created ENI. RunInstances failures roll back the ENI so the
caller does not leak a customer-account resource.
TerminateK3sServerVM tears the instance and ENI down for DeleteCluster;
missing identifiers are a no-op so retries stay safe.
ENI + VM both live in the customer account (deviation from plan §7 which
specified a SystemAccount-owned VM); cross-account ENI attach is deferred
to a follow-up bead.
* feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease
NATSBootstrap subscribes to the four single-shot subjects the K3s server VM
publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca)
on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are
envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The
published JWKS is cross-checked against the controller-generated keypair by
kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64
via the new SetClusterCertificateAuthority CAS updater.
ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/
{clusterName} (60s TTL, 20s refresh) and drives:
- CREATING -> ACTIVE once the four bootstrap KV artifacts are present and
NLB /healthz responds 200.
- ACTIVE -> probe /healthz on the configured interval, warn on failure.
- DELETING / FAILED -> exit (DeleteCluster body owns teardown).
Loss of the leader lease (CAS conflict or different holderID) returns
ErrReconcilerLeaseLost so the caller can re-acquire from another node.
Tested with embedded JetStream + a stub HTTPDoer; race-clean.
* feat(eks): OIDC issuer URL builder + reconciler registry
ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/
{clusterName} URL kube-apiserver advertises as service-account-issuer and
that STS / external IRSA validators dereference. Validates URL safety on the
path segments so a hostile cluster name cannot inject scheme/path fragments.
ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines
with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait
for release), and StopAll (graceful daemon shutdown). RunClusterReconciler
is the production SpawnReconcilerFn: constructs a ClusterReconciler,
acquires the leader lease, runs Run in a goroutine. Both helpers consumed
by the upcoming service_impl CreateCluster body.
* feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies
Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB,
Instance launcher, AMI resolver, master key, gateway base URL, region,
holder ID, reconciler registry) and implements the four lifecycle
methods:
- CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer +
keypair → K3s server VM (cloud-init carries OIDC private PEM) →
register ENI IP with NLB target group → spawn NATS bootstrap
subscriber + per-cluster reconciler via the registry.
- DescribeCluster: KV meta read → eks.Cluster shape, with
ErrClusterNotFound translated to ResourceNotFoundException at the
service boundary.
- ListClusters: KV key-prefix scan + sorted maxResults/nextToken
pagination.
- DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop
reconciler via registry → deregister NLB target + delete NLB →
terminate K3s VM → delete SGs → delete cluster KV prefix.
Daemon wiring:
- buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key
best-effort, resolves the gateway base URL via the same precedence the
LB-agent block applies later, and passes VPCService + ELBv2Service +
ImageService directly while adapting InstanceServiceImpl via
daemonEKSInstanceLauncher (RunInstances passes through; Terminate-
Instances loops vmMgr.Get + StopOrTerminateInstance in-process).
- SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping
VPCService.GetSubnet.
- Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster
via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so
every per-cluster goroutine releases its lease before exit.
Test surface:
- service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers
the four lifecycle methods when constructed via the back-compat shim
(CreateCluster/DeleteCluster → ServerInternal, DescribeCluster →
ResourceNotFound on empty bucket, ListClusters → empty list).
- daemon_handlers_eks_test: drop the four lifecycle subjects from the
NotImplemented routing loop; handler-side coverage now lives in
handlers/eks.
- awserrors: add ResourceNotFound (404) for the DescribeCluster path.
* fix(eks): load master.key from config dir not service data dir
eks_deps read master.key from {BaseDir}/config/master.key, but the daemon
systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service
data dir). admin init writes master.key to the shared config dir, so the
daemon looked in a path that only resolved when a {BaseDir}/config symlink
existed — awsgw has one, the daemon's nested data dir does not. Result:
MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster
collapse to ServerInternal.
master.key is a config artifact written next to spinifex.toml. Read it from
filepath.Dir(d.configPath), matching how the daemon already resolves
predastore.toml and TLS certs. No symlink or provisioning change needed.
* fix(eks): register + resolve eks-server AMI via tag, add import --ami-name
CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every
clean install. Two defects:
1. spx admin images import had no way to name a locally-built appliance AMI:
--name is a catalog lookup (rejects non-catalog values) and the AMI name
was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran
'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all
failed 'Image name not found in available images'.
2. The launcher resolved DescribeImages by name=='eks-server', a name the
pipeline could never produce.
Add --ami-name to override the registered AMI name on --file imports (fixes
eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the
spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on
multiple matches. Tag-based resolution survives the planned server+agent →
single EKS AMI unify untouched (role chosen per-instance at launch).
build-system-image.sh now passes --ami-name; Makefile gains
import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when
DEV_INSTALL_EKS=1.
* fix(eks): give K3s server VM NATS token + CA so bootstrap can publish
The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback,
unreachable from the VM) and no auth material, while spinifex NATS requires a
shared token over TLS. The VM could not connect to NATS, so it never published
its four one-shot bootstrap messages and the cluster stayed in CREATING.
Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL
(mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys
hierarchy to scope per-principal creds against).
- K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert,
required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN,
SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600.
- EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert.
- buildEKSServiceDeps: read CA from config.NATS.CACert; extract
resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port).
- k3s-first-boot.sh: build nats CLI args with --token + --tlsca.
* fix(eks): retry transient bootstrap persists, fail cluster on permanent error
The NATS bootstrap subscriber tore down on the first persist error and the
cluster sat in CREATING forever. Core-NATS delivers a publication once, so a
transient KV blip during persist was unrecoverable and a malformed payload
(bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster.
- persistWithRetry rides out transient persist failures in-handler (3 attempts,
200ms backoff) instead of unsubscribing and waiting for a re-publish that
never comes.
- errBootstrapPermanent classifies validation failures (malformed envelope,
empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they
return immediately.
- spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a
non-context error, so DescribeCluster surfaces the cause and DeleteCluster
can reclaim resources.
- MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a
late bootstrap error cannot clobber a concurrent delete or healthy cluster.
- casUpdateMeta extracts the revision-checked RMW loop shared by the meta
setters.
* fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart
SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the
bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM
still booting) lost the subscriptions; the VM published its four one-shot
core-NATS messages to nobody and the cluster hung in CREATING forever.
- BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are
still missing from KV. JWKS is excluded: it persists no artifact and
re-waiting on a once-published subject would deadlock — its cross-check is
best-effort across a restart.
- RunForKinds subscribes to a subject subset (nil = all four); Run delegates to
it. subjectHandlers maps each kind to its handler and rejects unknown kinds.
- SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting
only on the missing artifacts, so a mid-CREATING restart still drives the
cluster to ACTIVE.
* fix(eks): DeleteCluster must not wipe KV or report success on teardown failure
DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue
then unconditionally swept the per-cluster KV prefix and returned success. A
failed NLB or VM teardown leaked billable resources with no owning record (the
meta holding their ARNs/IDs was erased) and told the operator the cluster was
gone. Zeroize — a stated security guarantee of no recoverable key material —
had its failure ignored too.
- Zeroize, NLB delete, and VM terminate are now blocking: their failures
accumulate (errors.Join) and abort the call before the KV sweep, leaving the
cluster DELETING and returning the error so a retry can still reach the ARNs.
- KV prefix sweep runs only after NLB + VM teardown confirm.
- SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning
record — and target deregister stays best-effort since DeleteClusterNLB tears
down the whole NLB + target group.
* fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak
CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and
persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any
failure between EnsureClusterNLB and that persist left the NLB + target group +
listener + backing LB VM created but unrecorded; markFailed did no rollback and
DeleteCluster keys teardown off the persisted ARNs, so the resources leaked.
- Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and
again after LaunchK3sServerVM (records instance + ENI), before the next
fallible step.
- A failed create now leaves a FAILED meta carrying the resource references, so
the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM.
- Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete
reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture
extracted from the delete-cluster fixture.
* fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens
ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment
path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves —
https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b
cluster's iss claim was rejected with "issuer host must start with oidc.eks",
so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them.
Reconcile on the path-based shape, which is what is actually reachable on
bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from
the IAM OIDC-provider registration (the full issuer string is the registry key)
and the JWKS signature check, not a host pattern — so the parser drops the host
prefix rule and instead pins accountID/clusterName at fixed positions in a
strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted
iss with extra or missing segments still fails closed.
Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss
parses back to the same (accountID, clusterName).
* fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors
A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the
preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently
in use...") for a cluster-exists condition, and because that maps to a
retryable code the SDK's auto-retry surfaced ResourceInUseException instead of
the original failure. FAILED clusters were also a dead state — visible to
neither a clean recreate nor a clear lifecycle.
- Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with
cluster-scoped messages; CreateCluster's exists check and DescribeCluster /
DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes.
- CreateCluster reclaims a FAILED cluster: it purges the prior attempt's
recorded resources and clears KV before recreating, so a retry (including the
SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces
rather than recreating on top of orphans.
- Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path
(blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown
confirms).
- FAILED clusters remain observable via Describe (status=FAILED) and List — no
state where create blocks but describe/list show nothing.
* fix(eks): fail CREATING clusters that exceed create timeout
A per-cluster reconciler that never observes a ready control plane (bootstrap
artifacts missing or healthz unreachable) previously stayed CREATING forever.
Add a configurable create timeout (default 15m): once meta.CreatedAt is older
than the deadline and the cluster is still not ready, the reconciler marks it
FAILED with a reason, exiting the loop. Within the deadline it stays CREATING.
* fix(eks): surface ACTIVE cluster health in describe-cluster
The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE
cluster, so describe-cluster kept reporting a dead control plane as healthy.
Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE
reconcile branch (recording the failure reason, clearing on recovery), and map
a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in
describe-cluster.
* test(eks): add plan-mandated CreateCluster/DeleteCluster service cases
service_impl_test only covered shim mode. Add the four cases the plan's Testing
section requires: CONFIG_MAP auth mode rejected (and API accepted), missing
subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim
short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven
by forcing a teardown failure and asserting the OIDC key is already wiped.
* refactor(eks): return OIDC private-key PEM from generator, drop re-read
CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second
KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the
generator already had in hand. Return the plaintext PEM (alongside the JWKS)
from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now
-unused crypto/x509 + encoding/pem imports in service_impl.
* fix(eks): make CreateCluster/DeleteCluster failures diagnosable
Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare
ServerInternal with no request-time log, leaving the client a 'contact AWS
re:Post' message and zero cause. Enumerate the specific missing deps and log
them at ERROR, returning ServiceUnavailable (an operator/config fault, not an
unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a
missing control-plane AMI is logged and surfaced as ServiceUnavailable too.
* fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them
WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json,
which ignores their locationName tags and emits Go PascalCase keys. EKS
clients parse restjson on the lowercase locationName, so every field of
list/describe/create/delete responses was silently dropped (empty endpoint,
null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same
marshaler the SDK uses for restjson bodies — which honors locationName.
Adds wire-shape tests asserting lowercase keys present and PascalCase absent.
* feat(eks): launch K3s server VM via boot-agnostic system-instance path
The EKS K3s control-plane VM launched through the plain tenant RunInstances
path, so it had only its VPC ENI and no management-bridge NIC -- no route to
the daemon's NATS endpoint. It never published its bootstrap subjects and the
cluster stayed CREATING.
Generalize LaunchSystemInstance to a boot-agnostic contract supporting both
system-instance boot styles:
- BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent
- BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM
The launch contract moves to a neutral handlers/sysinstance package so EKS can
launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports
the types as aliases. The BootAMI branch mirrors the daemon's
Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record
between Insert and Launch, so the cloud-init network-config renders a static
mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the
on-link address the VM's mgmt NIC can reach), falling back to the gateway-host
resolution when no mgmt bridge exists.
Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC
(10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the
bridge IP. No AMI rebuild required.
* fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge
TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every
TerminateSystemInstance error -- as blocking. Because instance terminate is
async, the immediate ENI delete races it: the first attempt fails
InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases
the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into
purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep
and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING
permanently, leaking its control-plane + nodegroup SGs and KV meta.
Classify the teardown errors instead of blanket-propagating:
- ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound
-> idempotent success (the cascade or a prior retry already deleted it);
must not block the downstream SG + KV sweep.
- ENI InvalidNetworkInterface.InUse (VM still terminating) and any other
error -> retryable; surfaced so the cluster stays DELETING and the
reconciler retries once the instance releases the ENI.
- Instance already gone -> idempotent. TerminateSystemInstance now returns a
sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates
via errors.Is, so a retried delete after the VM drained completes.
Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier
(substring match through %w wrapping) and route the lone ad-hoc strings.Contains
guard in daemon/vm_adapters.go through it.
Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI +
VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak.
* fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key
The controller pre-seeds OIDCJWKSKey at create time, so its presence
proved nothing about the running cluster's actual signing key. A cluster
could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's
published JWKS was ever cross-checked against the generated keypair.
Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after
assertJWKSMatch passes. The reconciler's ACTIVE gate and
BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS.
assertJWKSMatch is map-based and also rejects kty mismatch.
* fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost
startCluster reimplemented the 5-case gateway-host precedence inline,
duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL).
If one drifted, the OIDC issuer host embedded in cluster tokens would
diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch
and cluster-wide token rejection.
startCluster now calls resolveGatewayHost and derives the multi-node
mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost
carries the canonical case enumeration; a table test pins all five
branches plus the case-1 skip guards and the no-host fallthrough.
* fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash
Three issues found while diagnosing why K3s clusters never leave CREATING:
1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in
default_kernel_opts, so Linux binds /dev/console to the framebuffer and the
host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and
k3s-first-boot output were invisible. Reorder so console=ttyS0 is last.
2. k3s-first-boot.initd: command_background was "no" while output_log/error_log
were set; start-stop-daemon rejects --stdout/--stderr without --background
("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and
the cluster wedged in CREATING. Background it (the publisher waits up to 5min
on K3s /healthz and must not block the runlevel anyway) and route its output
to the serial console, which the orchestrator captures (the VM has no login).
3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log
to the console so the apiserver's own failure is visible host-side.
* fix(eks): seed K3s apiserver a public OIDC key, not the private key
buildK3sUserData pointed both kube-apiserver service-account args at the same
private-key file. --service-account-key-file requires a PUBLIC key — k8s
ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any
valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never
reaches /healthz, first-boot never publishes, and the cluster wedges in
CREATING.
Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and
seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point
service-account-key-file at the public file; keep service-account-signing-key-file
on the private file. Runtime-verified: the apiserver crash is gone and K3s now
starts and schedules system pods.
* feat(eks): egress-only SNAT + DNS for K3s control-plane VM
The K3s server VM (a hidden system instance) could not pull system-pod
images: it had no internet egress and no working resolver, so
coredns/traefik/metrics-server never started and /healthz timed out.
Egress: give the single control-plane VM egress-only internet, confined
to its /32, with no inbound exposure and no customer-visible resource.
- policy/route.go: SystemInstanceEgressPriority (1200, above the drop
gate) + AddSystemInstanceEgress/Delete — a reroute scoped to
ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected.
- policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not
dnat_and_snat, so no inbound path), idempotent on its pool external IP.
- external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the
gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW
egress path; ADR-0006 S1/S4/S5 preserved).
- subscribers: vpc.add/delete-system-egress topics + handlers.
- handlers/eks: allocate a hidden (ManagedBy=eks) pool address on
CreateCluster and publish add; release + publish delete on teardown.
EIP dep wired through EKSServiceDeps + the daemon.
DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a
dangling symlink, so DHCP-supplied resolvers are never persisted. Write
a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not
write_files — a write_files entry follows the dead symlink, fails, and
aborts the whole block, dropping first-boot.env + the k3s config).
* feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes
Switch the K3s server VM from embedded etcd to embedded SQLite (kine)
for the single-node v1, so CreateCluster reaches a healthy control plane.
etcd's per-commit fsync stalls past the apiserver deadline on viperblock's
deferred-durability block path; SQLite tolerates it.
- k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true
so the reconciler's anonymous /healthz NLB probe returns 200.
- first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth
=false) -> authenticated 'kubectl get --raw /readyz'.
- first-boot publish: emit a JSON BootstrapEnvelope for all four subjects
(token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before.
- nats pub: --stdin -> --force-stdin (nats CLI 0.4.0).
- manifest: add jq (builds the bootstrap-message JSON envelopes).
* feat: EKS reconciler probes control-plane mgmt IP until DNS lands
The host-side cluster reconciler cannot resolve or route to the VPC-internal
NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed
out in CREATING and was marked FAILED even though the control plane was healthy.
Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver
endpoint reachable from the daemon. Capture the mgmt IP from the launch result
(SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta.
ControlPlaneMgmtIP) and build the reconciler health URL as
https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify,
so the mgmt IP need not be in the apiserver cert SAN.
Workaround until authoritative DNS + a host-reachable NLB front-end land, at
which point spawnReconciler reverts to the NLB endpoint.
|
||
|
Julian Sommer
|
94aa482931 | feat: sts v1 (#261) | ||
|
tomnewton-mulga
|
78d703da78 |
Feat/mig capable (#260)
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit
|
||
|
Julian Sommer
|
3f3ba32317 | chore: cleanup COMMANDS.md | ||
|
Julian Sommer
|
f525006d87 | feat: IAM Roles part 2 (#256) | ||
|
Julian Sommer
|
51c48d8c0e | feat: IAM Roles part 1 (#253) | ||
|
Julian Sommer
|
b2dde0a66d | feat: spx admin images remove (#239) | ||
|
Julian Sommer
|
20a8a39608 | feat: flip images to UEFI default (#237) | ||
|
Julian Sommer
|
f7efc69344 | feat: disable api termination (#224) | ||
|
Julian Sommer
|
89ddd06846 | chore: update COMMANDS.md | ||
|
Julian Sommer
|
97701737cb | feat: describe instance status (#221) | ||
|
tomnewton-mulga
|
43fa6eae59 |
feat: Updated GPU-Passthrough - AMD support, GPU images, /demo folder (#217)
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * installer grub image fixes * made gpu-passthrough more user facing * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * Fix: AMD GPU edge cases * script for running 8x instances on SMCI computer * fix: increase disk space on instances * feat: multi-GPU passthrough support for g7e.12xlarge (2× MI350X) - vm.VM: GPUPCIAddress string → GPUPCIAddresses []string for multi-GPU - instancetypes: add g7e.12xlarge (2 GPUs), GPUCountForType(), AMDmi350x model - instancetypes/generate: use GPUCountForType for GpuInfo.Count and TotalGpuMemoryInMiB - gpu/manager: Release() loops all pool entries for instanceID (not just first) - vm/lifecycle: loop over GPUPCIAddresses, emit gpu0/gpu1/... QEMU devices - daemon/handlers: claim N GPUs in loop for both RunInstances and StartStoppedInstance - daemon/vm_adapters: ReclaimByAddress loops GPUPCIAddresses; ReleaseGPU checks len > 0 - scripts/launch-gpu-fleet: fixed fleet — 6× g7e.4xlarge + 1× g7e.12xlarge * feat: per-instance-type disk size in GPU fleet (300 GB single, 600 GB dual) * fix: multi-gpu instance types * fix: kill QEMU synchronously when QMP handshake fails to free VFIO device When startQEMU succeeds but newQMPClientWithHandshake times out, the instance has not yet been inserted into the manager map so vmMgr.Delete in the rollback path is a no-op. QEMU keeps running and holds the VFIO group open. The GPU pool entry gets freed, the next Claim returns the same PCI address, and every subsequent QEMU fails with 'device or resource busy'. Fix: SIGKILL QEMU and wait for it to exit before returning the QMP error, ensuring the VFIO device is released before the caller recycles the GPU pool slot. * fix: quarantine GPU after QEMU launch failure to prevent re-assignment cascade When QEMU crashes during startup (e.g. pci_irq_handler assertion on a specific GPU), the pool Release frees that address and the next NATS request immediately re-claims the same broken device. Add MarkFailed() to gpu.Manager which sets Available=false for a PCI address after a launch failure, preventing re-assignment until the daemon restarts. Call MarkFailed for each GPU address in both RunInstances and StartStoppedInstance rollback paths after the Release succeeds. * fixed instance fleet launch script * remove known hosts and stagger provisioning * improvements to launch script * more upgrades * single instance gpu instance provisioning * fixed single instance provision * smci computer demo dashboard * fixes to launch script * updated smci dashboard_server * fixed demo dashboard * updated dashboard.py * feat: GPU Passthrough fixes to passthrough process - builds proper AMD or NVIDA ready images, generalised launch script * feat: gpu-passthrough updating image builder to include gpu-ready ubuntu images * fixing image builder process * fixed image builder to remove /tmp/ file before import to free space * fix: Updated COMMANDS.md to reflect new GPU management admin commands * Update docs, update internal launch script to work with new SGs, test tweaks * Updated Docs, fixed internal launch-gpu.sh script, test tweaks * fxing CI race due to GPU load |
||
|
tomnewton-mulga
|
cc3c77fdb7 | Merge commit '0a7832440504442f685e85048d545848a8f2b8f6' into feat/iso-installer | ||
|
Julian Sommer
|
0a78324405 | feat: missing image commands (#156) | ||
|
Ben Duncan
|
27d7fdd12b | Latest changes re ISO and binary install path |