mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
2db1034436
* fix: fan out certificate re-import to referencing load balancers Re-importing new material under an existing ACM ARN previously updated only the stored CertRecord; no load balancer referencing that ARN ever re-rendered its config, so HAProxy kept serving the old leaf until it expired while the API reported success. Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff around every listener create/modify/delete and cert attach/detach in handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook (wired in daemon.go to avoid an import cycle) that re-runs updateStoredConfig for every load balancer in the ARN's InUseBy set, mirroring the existing target-group fan-out. lbagent is unchanged: it already polls ConfigHash and reloads on change. * fix: make InUseBy index updates CAS-safe under concurrent writers AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which loses concurrent writers silently: two listeners on different load balancers attaching the same certificate at once could drop one entry with no error. Because InUseBy is the sole mechanism that fans a renewed certificate out to the data plane, a lost entry means that load balancer never re-renders and its certificate silently expires in HAProxy while ACM reports it as healthy. Switch both to a JetStream KV revision-based compare-and-swap loop with jittered backoff, retrying on a concurrent writer instead of overwriting it. Replace the old concurrency test, which only asserted no errors/panics and so could not detect a lost index entry, with one that asserts exact final state across N concurrent writers. * fix: encrypt ACM certificate private keys at rest CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key. Store.PutCert encrypts when a master key is set; GetCert/ListCerts decrypt, falling back to a narrowly-bounded legacy-plaintext read (only when the stored value is unambiguously a PEM private key block) so pre-existing records keep working and are re-encrypted on next write. A Store never given a key (ELBv2's independent read-only acmStore) behaves exactly as before — encryption is opt-in per Store instance via the new SetMasterKey method, so no other package needs to change. NewACMServiceImplWithNATS now requires a master key and fails construction without one; daemon.go loads master.key inside the initServiceWithRetry closure so a missing key fails daemon startup after the retry window instead of leaving the service permanently degraded. * fix: wire ELBv2's ACM store with the deployment master key ELBv2 constructed its own handlers_acm.Store without a master key, so once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could no longer resolve listener certificates: it read back ciphertext where it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a key so an unkeyed Store can no longer be constructed, thread the deployment master key through NewELBv2ServiceImplWithNATS and the daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM store construction failure fatal instead of silently degrading every HTTPS listener. * feat(acm): tenant Private CA with x509 name constraints Adds an independent tenant root CA (separate from the platform CA) that signs leaf certificates only within its baked-in permitted domains. Name constraints (PermittedDNSDomains, excluded IP ranges, excluded email/URI) are frozen at creation and drift is rejected rather than silently regenerated, so a device's trust store is never invalidated without an explicit operator action. * docs(acm): record OpenSSL verification of the empty email/URI exclusions Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not just Go's own verifier, since this CA is meant for device trust stores. * feat: implement RequestCertificate with PRIVATE_CA managed issuance Adds the ACM RequestCertificate API surface: mode derivation (PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for the deferred modes. Extends CertRecord with issuance/validation/renewal state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md. * feat: wire tenant CA into the running spinifex daemon Add LoadTenantCA as a load-only entry point (never creates), refactor LoadOrCreateTenantCA to share the load path. Add 'spx admin cert create-tenant-ca' to create the tenant CA with explicit permitted domains, idempotent unless --regenerate is passed. Wire tenant CA loading into daemon startup: missing tenant CA logs a warning and degrades PRIVATE_CA issuance only, it does not block startup like the master key does. Fold tenant CA paths into admin's CA path helpers, keeping filenames distinct from the platform CA (tenant-ca.pem/.key vs ca.pem/.key) so the two roots can never be confused. Also give ACM's DeleteCertificate in-use refusal its own awserrors alias instead of reusing the EKS-labelled one directly, and correct COMMANDS.md to reflect that the tagging endpoints are implemented. * fix: derive ACM validation mode from the requested domain, not a global flag Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that northstar is configured, never whether it is authoritative for the domain being requested. Any deployment with a default domain set therefore returned MANUAL_TXT for every request, including domains hosted nowhere, with a TXT record the caller could not publish. Resolve the requested domain against northstar's actual hosted zones, and put MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly delegated so a public CA can resolve the challenge - which is never true of an internal northstar zone. A tenant CA covering every requested domain now wins, and a configured DNS provider still takes precedence over the manual path so operators with a real domain are not downgraded to a private certificate. * feat: renew PRIVATE_CA certificates automatically before they expire Managed issuance signed a leaf once and never renewed it. With 90-day leaves every certificate silently expired a quarter after the first apply, which is the treadmill managed issuance exists to remove. Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully offline, so this needs no ACME client and no network access. The window is proportional - two thirds of the certificate's own lifetime, floored at a minimum - so it self-tunes as issuance lifetimes change rather than degenerating into 'renew immediately, forever' the way a fixed window would. A deterministic per-ARN jitter keeps certificates issued in one apply from all renewing in the same minute. Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes through the ELBv2 fan-out. The push is mandatory rather than incidental: GetLBConfig returns stored configuration and never re-resolves certificate material, so a renewal that did not push would leave HAProxy serving the old leaf until something unrelated changed the config hash. A per-certificate lease keyed on the ARN means one node drives a given renewal without serialising all renewals behind a single leader. Certificates whose domains have fallen outside the tenant CA's name constraints record a failure reason instead of retrying in a tight loop. The renewal scan reads metadata only and never decrypts stored private keys; the lease is a raw compare-and-swap that leaves the encrypted key untouched. * fix: carry handler error messages over the daemon NATS transport handleNATSRequest responded with the sanitized error code alone, so any handler error without a registered code collapsed to a bare ServerInternal and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx has always preserved the message; the two implementations had drifted. Adds respondWithServiceError and routes the 16 service-error sites through it, plus transport-level tests pinning that the message survives for an uncoded error, that a wrapped coded error keeps its real code, and that a bare code carries no message. * feat: implement ACM GetCertificate and stop decrypting keys on read paths GetCertificate was unimplemented end to end: it now returns the PEM certificate body and chain for an owned ARN, ResourceNotFound for an unknown or cross-account ARN, and RequestInProgressException for a certificate that has a record but no body yet, wired through the service interface, the NATS client, the daemon subscription and the gateway action map. Store gains GetCertMetadata alongside GetCert, and ListCerts becomes ListCertMetadata. Every read that answers a question about a certificate rather than serving it - ownership checks, Describe, Delete, tag reads, GetCertificate and listing - now leaves PrivateKey empty instead of running an AES-GCM open per record to produce plaintext key material no caller reads. * fix: reconcile the ACM master-key wiring with dev's new tests The InUseBy and cert fan-out tests that landed on dev construct the ACM and ELBv2 services directly, and both constructors now take the deployment master key. Pass one shared key in the fan-out test - two services over the same bucket disagreeing on the key is the exact failure the wiring exists to stop. A Store is never unkeyed now, so the two test helpers collapse into one. * fix: rebuild the ACM InUseBy index at daemon startup The index that drives certificate fan-out is written only by listener mutations, so a listener whose certificates have not been touched since it was created has no entry. A certificate with an empty index then neither fans out on renewal nor trips DeleteCertificate's in-use guard, and both failures are silent: the renewal reports success while the load balancer keeps serving the old leaf, and a certificate a live listener depends on can be deleted. Reconcile the index from the persisted listener records once at startup. Walking every certificate rather than only the referenced ones makes it a rebuild rather than a backfill, so it also drops entries naming a load balancer that no longer exists. Writing only differences keeps it idempotent, and the index writes are already CAS-guarded, so the nodes of a cluster reconciling the same bucket converge. * fix: deliver listener SNI certificates to HAProxy Add/RemoveListenerCertificates persisted the listener record and then returned, so the config was never re-rendered and lbagent had nothing to poll for. Re-render through reloadListenerLB, the same helper the rule mutations use. That alone changed nothing, because the renderer dropped SNI certificates: resolveFrontendCert returned only the default certificate and the bind line emitted a single crt, with the PEM path keyed on the listener so a second certificate had nowhere to live. Render every attached certificate instead, default first so it stays the no-SNI fallback, each with its own path. HAProxy accepts crt more than once on a bind line and selects by SNI. The default certificate keeps its existing filename so an upgrade does not move a deployed PEM, and the nginx renderer keeps the single-certificate resolver it still relies on. * fix: prune stale TLS certs from the lb-agent cert dir A delivered PEM is cert + chain + private key. writeCertFiles only ever wrote, so every certificate detached from a listener left its key on the load balancer VM's disk indefinitely. Prune after a successful reload, scoped to .pem files in the cert dir's immediate entries. * fix: serve every attached certificate on the nginx NLB engine nginx allows one ssl_certificate per key type in a server block, and the NLB config is a stream block where server_name does not exist. Select by SNI instead: give ssl_certificate a variable and map $ssl_server_name to each certificate's path, keyed on the leaf's DNS SANs. A single-certificate listener keeps a literal path, since a variable costs a per-handshake read. The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so they now take the dir and both engines share one multi-cert resolver. * fix: emit ValidationMethod in DomainValidationOptions DescribeCertificate returned each domain validation entry without a ValidationMethod, so the AWS provider had nothing to compare against aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes. PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits the method on a private certificate, where the provider reads it as NONE whatever the request asked for.
Spinifex Documentation
This directory contains the Spinifex documentation. This is available in individual markdown files, and a generated website to aid navigation and browing documentation at: https://docs.mulgadc.com