mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
bbff004b456a6c714ac0aac2c80eda589d317d96
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
bbff004b45 |
feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280)
* feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep
Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first
read/write helpers against the eks-account-{accountID} bucket so later
stages (oidc keypair, NLB wiring, cluster reconciler) can persist state
without re-inventing CAS/sweep logic.
- ClusterMeta struct holds the DescribeCluster source of truth (status,
endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN).
- ClusterStatus enum mirrors the AWS EKS string set verbatim.
- PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta
returns ErrClusterNotFound on key absence so the service boundary can
map to ResourceNotFoundException.
- SetClusterStatus is revision-checked with bounded CAS retries so the
reconciler and CreateCluster body cannot wedge each other.
- DeleteClusterPrefix sweeps every clusters/{name}/* key; called from
DeleteCluster after the OIDC private key is zeroized.
Tests cover round-trip, missing/corrupt blob, idempotent status set,
DELETING transition from every prior state, concurrent revision bump
recovery, sibling-cluster preservation, empty-bucket no-op.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize
Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body
will call before launching the K3s server VM. Pure crypto helpers — no
daemon, no launcher, no NATS bus.
- GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster
(matches eks-v1.md Q8), PEM-marshals the private as PKCS8,
envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM,
AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and
publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS
bytes so CreateCluster can hand them straight to user-data without
a follow-up KV read.
- JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA
verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER,
alg ES256, use sig). Types are defined locally to avoid a
handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire
compat is the contract.
- LoadClusterOIDCPrivateKey is the recovery-path read for future
stages (reconciler restart, restore-snapshot tooling). Validates
PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else.
- ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of
the same length, then deletes. DeleteCluster calls this *first* so a
crash mid-teardown leaves no recoverable key material. Idempotent
on missing key — supports retry semantics.
Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1
coordinates), per-cluster key uniqueness, round-trip private-key
decrypt matches JWKS public, wrong-master-key fails, missing-key
ErrClusterNotFound mapping, zeroize idempotency, JWKS survives
zeroize (carries no secret material), padCoord left-pad correctness.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): cluster SG + NLB ensure/delete helpers
Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account
SG provisioning + per-cluster NLB plumbing the CreateCluster body will
call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure
interface-call helpers — no daemon, no KV, no NATS bus.
- EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg
and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged
spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role.
Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort
with continue-on-error sweep.
- EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the
eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md
Q13), and the TCP :443 forward listener. Returns ClusterNLB
{LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster
can persist every ARN in ClusterMeta. Idempotent on cluster name;
resource-name length validated against the AWS 32-char cap so a too-long
cluster name fails with a clear error instead of a generic
InvalidParameterValue downstream.
- RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM
ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its
ENI IP; DeleteCluster calls Deregister before VM termination.
- DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already
cascades listener cleanup inside the ELBv2 impl. LB sweep error takes
precedence so DeleteCluster surfaces the more visible failure first
while the TG sweep still runs.
Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests
hand-rolled recorder fakes; the production VPCService + ELBv2Service
satisfy them structurally so no adapter is needed at Stage 6 wire-up.
Tests cover fresh-creates with right names/tags/ports, idempotent reuse,
mixed-existence (control-plane exists, nodegroup created), name-too-long
rejection, empty-inputs rejection, missing-resource no-op on delete, and
first-error-surfaced sweep continuation on delete.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(ec2): RunInstances honors pre-created NetworkInterfaceId
Closes the AWS-parity gap where RunInstances always auto-created a
primary ENI even when the caller specified
NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster
launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the
ENI can be tagged/owned distinctly from the launcher's instance; the
existing path silently created a second ENI and left the caller's
unattached.
Changes:
- ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo
is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status,
SecurityGroupIDs) so handlers/ec2/instance stays free of an import on
handlers/ec2/vpc.ENIRecord.
- daemon/vpc_adapter.go gains daemonENICreator.GetENI translating
VPCServiceImpl.GetENIRecord into ENIInfo.
- PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId
after the existing subnet/SG lift block. If non-empty, the loop
dispatches to a new attachPreCreatedENI helper:
- Looks up the ENI under accountID.
- Rejects with InvalidNetworkInterface.InUse if Status == "in-use".
- Calls AttachENI to mark the ENI in-use and bind it to the new VM.
- Populates instance.ENIId / ENIMac and ec2.Instance network fields
(PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]).
- Skips the auto-create + public-IP-auto-assign paths; the caller is
expected to manage public IP out-of-band.
- Auto-create path is left untouched for every other code path: callers
that pass SubnetId without a NetworkInterfaceId still get a fresh ENI
with the existing semantics.
Tests cover the four shapes: happy path (auto-create not called, attach
called once, instance populated), in-use ENI rejected with the canonical
AWS error code, lookup error surfaces verbatim, attach error surfaces
verbatim.
* feat(eks): K3s control-plane VM launcher with cloud-init user-data
LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the
customer-account control-plane ENI in the supplied subnet with the
control-plane SG, renders cloud-init user-data (first-boot env, OIDC
private key, K3s config.yaml with tls-san + service-account args), then
calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing
at the pre-created ENI. RunInstances failures roll back the ENI so the
caller does not leak a customer-account resource.
TerminateK3sServerVM tears the instance and ENI down for DeleteCluster;
missing identifiers are a no-op so retries stay safe.
ENI + VM both live in the customer account (deviation from plan §7 which
specified a SystemAccount-owned VM); cross-account ENI attach is deferred
to a follow-up bead.
* feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease
NATSBootstrap subscribes to the four single-shot subjects the K3s server VM
publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca)
on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are
envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The
published JWKS is cross-checked against the controller-generated keypair by
kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64
via the new SetClusterCertificateAuthority CAS updater.
ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/
{clusterName} (60s TTL, 20s refresh) and drives:
- CREATING -> ACTIVE once the four bootstrap KV artifacts are present and
NLB /healthz responds 200.
- ACTIVE -> probe /healthz on the configured interval, warn on failure.
- DELETING / FAILED -> exit (DeleteCluster body owns teardown).
Loss of the leader lease (CAS conflict or different holderID) returns
ErrReconcilerLeaseLost so the caller can re-acquire from another node.
Tested with embedded JetStream + a stub HTTPDoer; race-clean.
* feat(eks): OIDC issuer URL builder + reconciler registry
ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/
{clusterName} URL kube-apiserver advertises as service-account-issuer and
that STS / external IRSA validators dereference. Validates URL safety on the
path segments so a hostile cluster name cannot inject scheme/path fragments.
ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines
with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait
for release), and StopAll (graceful daemon shutdown). RunClusterReconciler
is the production SpawnReconcilerFn: constructs a ClusterReconciler,
acquires the leader lease, runs Run in a goroutine. Both helpers consumed
by the upcoming service_impl CreateCluster body.
* feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies
Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB,
Instance launcher, AMI resolver, master key, gateway base URL, region,
holder ID, reconciler registry) and implements the four lifecycle
methods:
- CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer +
keypair → K3s server VM (cloud-init carries OIDC private PEM) →
register ENI IP with NLB target group → spawn NATS bootstrap
subscriber + per-cluster reconciler via the registry.
- DescribeCluster: KV meta read → eks.Cluster shape, with
ErrClusterNotFound translated to ResourceNotFoundException at the
service boundary.
- ListClusters: KV key-prefix scan + sorted maxResults/nextToken
pagination.
- DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop
reconciler via registry → deregister NLB target + delete NLB →
terminate K3s VM → delete SGs → delete cluster KV prefix.
Daemon wiring:
- buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key
best-effort, resolves the gateway base URL via the same precedence the
LB-agent block applies later, and passes VPCService + ELBv2Service +
ImageService directly while adapting InstanceServiceImpl via
daemonEKSInstanceLauncher (RunInstances passes through; Terminate-
Instances loops vmMgr.Get + StopOrTerminateInstance in-process).
- SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping
VPCService.GetSubnet.
- Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster
via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so
every per-cluster goroutine releases its lease before exit.
Test surface:
- service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers
the four lifecycle methods when constructed via the back-compat shim
(CreateCluster/DeleteCluster → ServerInternal, DescribeCluster →
ResourceNotFound on empty bucket, ListClusters → empty list).
- daemon_handlers_eks_test: drop the four lifecycle subjects from the
NotImplemented routing loop; handler-side coverage now lives in
handlers/eks.
- awserrors: add ResourceNotFound (404) for the DescribeCluster path.
* fix(eks): load master.key from config dir not service data dir
eks_deps read master.key from {BaseDir}/config/master.key, but the daemon
systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service
data dir). admin init writes master.key to the shared config dir, so the
daemon looked in a path that only resolved when a {BaseDir}/config symlink
existed — awsgw has one, the daemon's nested data dir does not. Result:
MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster
collapse to ServerInternal.
master.key is a config artifact written next to spinifex.toml. Read it from
filepath.Dir(d.configPath), matching how the daemon already resolves
predastore.toml and TLS certs. No symlink or provisioning change needed.
* fix(eks): register + resolve eks-server AMI via tag, add import --ami-name
CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every
clean install. Two defects:
1. spx admin images import had no way to name a locally-built appliance AMI:
--name is a catalog lookup (rejects non-catalog values) and the AMI name
was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran
'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all
failed 'Image name not found in available images'.
2. The launcher resolved DescribeImages by name=='eks-server', a name the
pipeline could never produce.
Add --ami-name to override the registered AMI name on --file imports (fixes
eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the
spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on
multiple matches. Tag-based resolution survives the planned server+agent →
single EKS AMI unify untouched (role chosen per-instance at launch).
build-system-image.sh now passes --ami-name; Makefile gains
import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when
DEV_INSTALL_EKS=1.
* fix(eks): give K3s server VM NATS token + CA so bootstrap can publish
The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback,
unreachable from the VM) and no auth material, while spinifex NATS requires a
shared token over TLS. The VM could not connect to NATS, so it never published
its four one-shot bootstrap messages and the cluster stayed in CREATING.
Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL
(mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys
hierarchy to scope per-principal creds against).
- K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert,
required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN,
SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600.
- EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert.
- buildEKSServiceDeps: read CA from config.NATS.CACert; extract
resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port).
- k3s-first-boot.sh: build nats CLI args with --token + --tlsca.
* fix(eks): retry transient bootstrap persists, fail cluster on permanent error
The NATS bootstrap subscriber tore down on the first persist error and the
cluster sat in CREATING forever. Core-NATS delivers a publication once, so a
transient KV blip during persist was unrecoverable and a malformed payload
(bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster.
- persistWithRetry rides out transient persist failures in-handler (3 attempts,
200ms backoff) instead of unsubscribing and waiting for a re-publish that
never comes.
- errBootstrapPermanent classifies validation failures (malformed envelope,
empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they
return immediately.
- spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a
non-context error, so DescribeCluster surfaces the cause and DeleteCluster
can reclaim resources.
- MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a
late bootstrap error cannot clobber a concurrent delete or healthy cluster.
- casUpdateMeta extracts the revision-checked RMW loop shared by the meta
setters.
* fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart
SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the
bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM
still booting) lost the subscriptions; the VM published its four one-shot
core-NATS messages to nobody and the cluster hung in CREATING forever.
- BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are
still missing from KV. JWKS is excluded: it persists no artifact and
re-waiting on a once-published subject would deadlock — its cross-check is
best-effort across a restart.
- RunForKinds subscribes to a subject subset (nil = all four); Run delegates to
it. subjectHandlers maps each kind to its handler and rejects unknown kinds.
- SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting
only on the missing artifacts, so a mid-CREATING restart still drives the
cluster to ACTIVE.
* fix(eks): DeleteCluster must not wipe KV or report success on teardown failure
DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue
then unconditionally swept the per-cluster KV prefix and returned success. A
failed NLB or VM teardown leaked billable resources with no owning record (the
meta holding their ARNs/IDs was erased) and told the operator the cluster was
gone. Zeroize — a stated security guarantee of no recoverable key material —
had its failure ignored too.
- Zeroize, NLB delete, and VM terminate are now blocking: their failures
accumulate (errors.Join) and abort the call before the KV sweep, leaving the
cluster DELETING and returning the error so a retry can still reach the ARNs.
- KV prefix sweep runs only after NLB + VM teardown confirm.
- SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning
record — and target deregister stays best-effort since DeleteClusterNLB tears
down the whole NLB + target group.
* fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak
CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and
persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any
failure between EnsureClusterNLB and that persist left the NLB + target group +
listener + backing LB VM created but unrecorded; markFailed did no rollback and
DeleteCluster keys teardown off the persisted ARNs, so the resources leaked.
- Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and
again after LaunchK3sServerVM (records instance + ENI), before the next
fallible step.
- A failed create now leaves a FAILED meta carrying the resource references, so
the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM.
- Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete
reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture
extracted from the delete-cluster fixture.
* fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens
ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment
path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves —
https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b
cluster's iss claim was rejected with "issuer host must start with oidc.eks",
so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them.
Reconcile on the path-based shape, which is what is actually reachable on
bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from
the IAM OIDC-provider registration (the full issuer string is the registry key)
and the JWKS signature check, not a host pattern — so the parser drops the host
prefix rule and instead pins accountID/clusterName at fixed positions in a
strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted
iss with extra or missing segments still fails closed.
Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss
parses back to the same (accountID, clusterName).
* fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors
A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the
preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently
in use...") for a cluster-exists condition, and because that maps to a
retryable code the SDK's auto-retry surfaced ResourceInUseException instead of
the original failure. FAILED clusters were also a dead state — visible to
neither a clean recreate nor a clear lifecycle.
- Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with
cluster-scoped messages; CreateCluster's exists check and DescribeCluster /
DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes.
- CreateCluster reclaims a FAILED cluster: it purges the prior attempt's
recorded resources and clears KV before recreating, so a retry (including the
SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces
rather than recreating on top of orphans.
- Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path
(blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown
confirms).
- FAILED clusters remain observable via Describe (status=FAILED) and List — no
state where create blocks but describe/list show nothing.
* fix(eks): fail CREATING clusters that exceed create timeout
A per-cluster reconciler that never observes a ready control plane (bootstrap
artifacts missing or healthz unreachable) previously stayed CREATING forever.
Add a configurable create timeout (default 15m): once meta.CreatedAt is older
than the deadline and the cluster is still not ready, the reconciler marks it
FAILED with a reason, exiting the loop. Within the deadline it stays CREATING.
* fix(eks): surface ACTIVE cluster health in describe-cluster
The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE
cluster, so describe-cluster kept reporting a dead control plane as healthy.
Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE
reconcile branch (recording the failure reason, clearing on recovery), and map
a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in
describe-cluster.
* test(eks): add plan-mandated CreateCluster/DeleteCluster service cases
service_impl_test only covered shim mode. Add the four cases the plan's Testing
section requires: CONFIG_MAP auth mode rejected (and API accepted), missing
subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim
short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven
by forcing a teardown failure and asserting the OIDC key is already wiped.
* refactor(eks): return OIDC private-key PEM from generator, drop re-read
CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second
KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the
generator already had in hand. Return the plaintext PEM (alongside the JWKS)
from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now
-unused crypto/x509 + encoding/pem imports in service_impl.
* fix(eks): make CreateCluster/DeleteCluster failures diagnosable
Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare
ServerInternal with no request-time log, leaving the client a 'contact AWS
re:Post' message and zero cause. Enumerate the specific missing deps and log
them at ERROR, returning ServiceUnavailable (an operator/config fault, not an
unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a
missing control-plane AMI is logged and surfaced as ServiceUnavailable too.
* fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them
WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json,
which ignores their locationName tags and emits Go PascalCase keys. EKS
clients parse restjson on the lowercase locationName, so every field of
list/describe/create/delete responses was silently dropped (empty endpoint,
null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same
marshaler the SDK uses for restjson bodies — which honors locationName.
Adds wire-shape tests asserting lowercase keys present and PascalCase absent.
* feat(eks): launch K3s server VM via boot-agnostic system-instance path
The EKS K3s control-plane VM launched through the plain tenant RunInstances
path, so it had only its VPC ENI and no management-bridge NIC -- no route to
the daemon's NATS endpoint. It never published its bootstrap subjects and the
cluster stayed CREATING.
Generalize LaunchSystemInstance to a boot-agnostic contract supporting both
system-instance boot styles:
- BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent
- BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM
The launch contract moves to a neutral handlers/sysinstance package so EKS can
launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports
the types as aliases. The BootAMI branch mirrors the daemon's
Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record
between Insert and Launch, so the cloud-init network-config renders a static
mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the
on-link address the VM's mgmt NIC can reach), falling back to the gateway-host
resolution when no mgmt bridge exists.
Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC
(10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the
bridge IP. No AMI rebuild required.
* fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge
TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every
TerminateSystemInstance error -- as blocking. Because instance terminate is
async, the immediate ENI delete races it: the first attempt fails
InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases
the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into
purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep
and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING
permanently, leaking its control-plane + nodegroup SGs and KV meta.
Classify the teardown errors instead of blanket-propagating:
- ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound
-> idempotent success (the cascade or a prior retry already deleted it);
must not block the downstream SG + KV sweep.
- ENI InvalidNetworkInterface.InUse (VM still terminating) and any other
error -> retryable; surfaced so the cluster stays DELETING and the
reconciler retries once the instance releases the ENI.
- Instance already gone -> idempotent. TerminateSystemInstance now returns a
sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates
via errors.Is, so a retried delete after the VM drained completes.
Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier
(substring match through %w wrapping) and route the lone ad-hoc strings.Contains
guard in daemon/vm_adapters.go through it.
Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI +
VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak.
* fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key
The controller pre-seeds OIDCJWKSKey at create time, so its presence
proved nothing about the running cluster's actual signing key. A cluster
could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's
published JWKS was ever cross-checked against the generated keypair.
Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after
assertJWKSMatch passes. The reconciler's ACTIVE gate and
BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS.
assertJWKSMatch is map-based and also rejects kty mismatch.
* fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost
startCluster reimplemented the 5-case gateway-host precedence inline,
duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL).
If one drifted, the OIDC issuer host embedded in cluster tokens would
diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch
and cluster-wide token rejection.
startCluster now calls resolveGatewayHost and derives the multi-node
mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost
carries the canonical case enumeration; a table test pins all five
branches plus the case-1 skip guards and the no-host fallthrough.
* fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash
Three issues found while diagnosing why K3s clusters never leave CREATING:
1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in
default_kernel_opts, so Linux binds /dev/console to the framebuffer and the
host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and
k3s-first-boot output were invisible. Reorder so console=ttyS0 is last.
2. k3s-first-boot.initd: command_background was "no" while output_log/error_log
were set; start-stop-daemon rejects --stdout/--stderr without --background
("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and
the cluster wedged in CREATING. Background it (the publisher waits up to 5min
on K3s /healthz and must not block the runlevel anyway) and route its output
to the serial console, which the orchestrator captures (the VM has no login).
3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log
to the console so the apiserver's own failure is visible host-side.
* fix(eks): seed K3s apiserver a public OIDC key, not the private key
buildK3sUserData pointed both kube-apiserver service-account args at the same
private-key file. --service-account-key-file requires a PUBLIC key — k8s
ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any
valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never
reaches /healthz, first-boot never publishes, and the cluster wedges in
CREATING.
Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and
seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point
service-account-key-file at the public file; keep service-account-signing-key-file
on the private file. Runtime-verified: the apiserver crash is gone and K3s now
starts and schedules system pods.
* feat(eks): egress-only SNAT + DNS for K3s control-plane VM
The K3s server VM (a hidden system instance) could not pull system-pod
images: it had no internet egress and no working resolver, so
coredns/traefik/metrics-server never started and /healthz timed out.
Egress: give the single control-plane VM egress-only internet, confined
to its /32, with no inbound exposure and no customer-visible resource.
- policy/route.go: SystemInstanceEgressPriority (1200, above the drop
gate) + AddSystemInstanceEgress/Delete — a reroute scoped to
ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected.
- policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not
dnat_and_snat, so no inbound path), idempotent on its pool external IP.
- external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the
gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW
egress path; ADR-0006 S1/S4/S5 preserved).
- subscribers: vpc.add/delete-system-egress topics + handlers.
- handlers/eks: allocate a hidden (ManagedBy=eks) pool address on
CreateCluster and publish add; release + publish delete on teardown.
EIP dep wired through EKSServiceDeps + the daemon.
DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a
dangling symlink, so DHCP-supplied resolvers are never persisted. Write
a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not
write_files — a write_files entry follows the dead symlink, fails, and
aborts the whole block, dropping first-boot.env + the k3s config).
* feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes
Switch the K3s server VM from embedded etcd to embedded SQLite (kine)
for the single-node v1, so CreateCluster reaches a healthy control plane.
etcd's per-commit fsync stalls past the apiserver deadline on viperblock's
deferred-durability block path; SQLite tolerates it.
- k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true
so the reconciler's anonymous /healthz NLB probe returns 200.
- first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth
=false) -> authenticated 'kubectl get --raw /readyz'.
- first-boot publish: emit a JSON BootstrapEnvelope for all four subjects
(token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before.
- nats pub: --stdin -> --force-stdin (nats CLI 0.4.0).
- manifest: add jq (builds the bootstrap-message JSON envelopes).
* feat: EKS reconciler probes control-plane mgmt IP until DNS lands
The host-side cluster reconciler cannot resolve or route to the VPC-internal
NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed
out in CREATING and was marked FAILED even though the control plane was healthy.
Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver
endpoint reachable from the daemon. Capture the mgmt IP from the launch result
(SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta.
ControlPlaneMgmtIP) and build the reconciler health URL as
https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify,
so the mgmt IP need not be in the apiserver cert SAN.
Workaround until authoritative DNS + a host-reachable NLB front-end land, at
which point spawnReconciler reverts to the NLB endpoint.
|
||
|
Julian Sommer
|
b2dde0a66d | feat: spx admin images remove (#239) | ||
|
Josh Siviour
|
4ae2a4ac53 |
feat(elbv2): QEMU microvm direct-boot for LB system instances (#228)
* feat(vm): direct-boot foundations + config/input extensions + image build pipeline
vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go
config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput
image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts
* feat(daemon): direct-boot launch path for microvm system instances
When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration
PC-machine path (DirectBoot=false) unchanged.
Closes mulga-siv-65
* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service
When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer
When ELBv2Enabled=false: existing PC-machine path unchanged.
Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.
Closes mulga-siv-66
* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold
* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag
- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note
Closes mulga-siv-67
* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed
* fix(microvm): end-to-end fixes for direct-boot LB path
Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).
virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.
sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).
BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.
wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.
Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.
* fix(build): auto-detect docker/podman when apk unavailable for microvm image build
* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs
tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.
Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.
* fix(microvm): fix mgmt NIC routing for all LB schemes
Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:
1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
'ip route add 192.168.1.33 via 10.15.8.1'.
2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
blob and skipping interface setup in init.sh entirely.
Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.
* feat(microvm): Phase 2 kernel stripping + boot optimisation
- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches
* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip
The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.
Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.
* fix(build): keep failover modules in initramfs strip
virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.
Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.
* fix(daemon): restore slog default after viperblock.New mutates it
viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.
Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.
Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.
Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).
* perf(microvm): hit Phase 2 boot/memory gates
t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.
Root causes:
1. build-microvm-image.sh staged the entire chroot into the cpio
without dropping /boot, embedding a duplicate vmlinuz, Alpine's
stock initramfs-virt, System.map, and a "boot → ." symlink loop
(~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
the kernel is copied out, before the cpio. Initramfs drops from
31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
42 MiB → 22 MiB.
2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
no firmware, ROM, or block layer). Both extracted to
directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
qemuStartSettleWait to 50 ms.
3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
Added boot-time perf flags: quiet loglevel=3 (silence printk over
115200-baud isa-serial), mitigations=off (KVM guest on trusted
host, single-tenant LB workload), tsc=reliable, no_timer_check,
reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).
4. init.sh dropped modprobe virtio_mmio (built-in:
CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).
Phase 2 gates:
- t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms
- artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script)
- ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure,
unrelated to microvm path)
* build(microvm): wire build into deploy + distro + release pipeline
Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).
Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.
Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.
* build(ansible): include microvm artifacts in build role tarball
Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).
* feat(config): enable microvm direct-boot LB path by default
Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.
Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).
* fix(microvm): bootstrap apk.static when no host rootfs tool exists
Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.
When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.
No-op when apk, docker, or podman is already present.
Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244
* revert: drop apk.static bootstrap in build-microvm-image
Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.
* docs(workbook): drop lb-alpine import for microvm direct-boot LB
LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.
* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN
When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").
Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.
* fix(e2e): redirect lb probe chatter to stderr so stalls are visible
test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.
Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.
* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node
buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.
resolveMgmtRoute already encodes the correct policy:
- multi-node any scheme → explicit MgmtRoute{Gateway,Target}
- single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP)
- single-node internet-facing → empty (reach AWSGW via VPC SNAT)
Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.
Test covers all three branches.
* feat(ansible): dev-deploy playbook for in-place artifact swap
Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.
Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.
Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.
* chore(microvm): remove legacy LB Alpine image build pipeline
Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.
build-system-image stays — other manifests (e.g. GPU images) still
use it.
* refactor(microvm): remove legacy PC-machine LB boot path
Phase 3 — direct-boot becomes the only ELBv2 LB launch path.
- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
[daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
paths) and refresh stale comments that still mentioned lbVMUserData.
ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.
* docs: drop lb-alpine catalog entry from cluster setup guide
LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.
* feat(ui): remove LB image import banner + LB-AMI gating
Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:
- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
system-managed.ts (isSystemManagedImage stays — still used by the
EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
wizard pages
- The corresponding tests that asserted the banner appears
* revert: restore lb.conf + lb-setup.sh for mulga install pipeline
Phase 3 microvm removal (commit
|
||
|
tomnewton-mulga
|
28ca689130 |
feat: VFIO GPU passthrough (#188)
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Ben Duncan
|
36425a48eb |
refactor: rewrite reset-dev-env.sh against production layout
Reset now targets /etc/spinifex, /var/lib/spinifex, /var/log/spinifex, /run/spinifex (the privilege-separated layout installed by setup.sh) and drives re-install from a source build via dev-install.sh instead of the legacy in-tree $HOME/spinifex + start-dev.sh flow. Capture region, AZ, external_mode, pool range, gateway, and prefix from the existing spinifex.toml before the wipe so the reset restores the current topology rather than applying defaults. Fresh boxes fall back to ap-southeast-2 and a pool derived from the WAN subnet. Drop the macvlan branch entirely — bridge-only from here on. Operators on single-NIC hosts must run reset from the console (documented in the script header). dev-install.sh gains DEV_INSTALL_SKIP_INIT so reset can own the post- setup.sh steps (setup-ovn, spx admin init with preserved args, CA trust, service start, LB image import). Also honors SPINIFEX_REGION / SPINIFEX_AZ for non-reset callers, and scopes systemctl reset-failed to spinifex-* so unrelated failed units aren't silently cleared. Closes mulga-983. |
||
|
Julian Sommer
|
1ad03bb518 | feat(gateway): per-IP auth failure rate limiting (#138) | ||
|
Julian Sommer
|
0547bdd1fe | feat: loadbalancer frontend (#136) | ||
|
Josh Siviour
|
ab95566795 |
feat: initial ELBv2 implementation (#122)
* feat: add ALB target health reporting via HAProxy stats Agent-side: periodically query HAProxy stats socket and publish target health status to NATS (elbv2.alb.<id>.health topic). Control-plane: healthChecker subscribes to health reports from all ALB agents and updates target health state in the KV store. Wire into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup. E2E: remove health check skip — targets must now become healthy. * feat: allocate public IP for internet-facing ALB VMs in daemon Add Scheme field to SystemInstanceInput so the daemon knows whether an ALB is internet-facing or internal. For internet-facing, allocate a public IP via externalIPAM and publish vpc.add-nat for the OVN dnat_and_snat rule. Internal ALBs skip public IP assignment entirely. * fix: release auto-assigned public IPs on DeleteNetworkInterface DeleteNetworkInterface previously leaked public IPs and NAT rules. Now checks if the ENI has a public IP, verifies it's not EIP-owned, and releases via externalIPAM + publishes vpc.delete-nat. Daemon injects externalIPAM and EIP KV into VPC service at init time. Tests cover: auto-assigned IP release with NAT event, EIP-owned IP skipped, and no-op when externalIPAM is not configured. * feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770) In CreateLoadBalancer, when DevNetworking is false (pool mode), require 2+ subnets in different AZs for internet-facing scheme. Dev mode and internal scheme skip the validation. * feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771) Add 9 unit tests covering scheme-dependent ALB behavior: - DNS name uses internal- prefix for internal scheme (AWS-compatible) - Internet-facing ALBs include public IP in AvailabilityZones response - Internal ALBs have no public IP in AZ response - Launcher receives correct scheme, delete cleans up VM + ENIs Also implements the code changes these tests exercise: - AvailZoneInfo.PublicIP field populated from launcher output - lbRecordToSDK populates LoadBalancerAddresses for internet-facing * feat: add custom AMI for ALB to images list * feat: add internet-facing ALB data plane E2E test (mulga-772) New E2E script that validates internet-facing ALBs get public IPs and are reachable from a peer cluster node (external validation). Skips in dev mode. * fix: set ENDPOINT to runner node IP in ALB E2E workflow * fix: correct paths filter to reference alb-e2e.yml * fix: alb ami name and workflow step to import the image * fix: spx path * fix: remove ALB AMI import from workflow and added to bootstrap.sh * fix: use two subnets in different AZs for internet-facing ALB E2E Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each. * refactor: remove multi-AZ enforcement for internet-facing ALBs Team decision: ALBs must work with single-AZ and single-node clusters. Removes the multi-AZ validation, pool mode helper, associated tests, and error constant. E2E script reverted to single subnet. * fix: use correct account ID for ALB ENI public IP assignment The ENI is created under the caller's account but LaunchSystemInstance was looking it up with GlobalAccountID (system account), causing the UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's account ID via SystemInstanceInput.AccountID. * fix: add debug logging for ALB public IP assignment failure * fix: expand debug logging to include peer node and more keywords * fix: lazy system AMI discovery for ALB VMs The daemon was caching the system AMI at startup, but the ALB image is imported after services start (bootstrap step 9). Replace SetSystemAMI with SetSystemAMIFunc so the AMI is resolved at request time when CreateLoadBalancer actually needs it. * fix: resolve ALB VM instance type dynamically from resource manager The instance type was hardcoded to t3.nano which may not exist on all clusters. Now the daemon provides a resolver that picks the smallest available instance type from the resource manager. * push interim changes * refactor: rewrite ALB agent from NATS client to HTTP server Replace NATS subscriptions with an HTTP server on :8405 exposing POST /config, GET /ping, and GET /health. The daemon will push config and poll health over HTTP through the VPC network, removing the agent's direct NATS dependency. - agent.go: net/http server replaces nats.Conn/Subscriptions - health.go: remove push-based reportHealth; /health handler calls queryHAProxyStats synchronously on each request - cmd/alb-agent: --nats flag replaced with --listen - tests: rewritten with httptest instead of NATS test server - health_checker_test: inline topic strings (HealthTopic removed) * refactor: replace NATS with HTTP client for daemon-to-ALB agent comms Daemon now pushes config via POST and polls ping via GET to the ALB agent's HTTP server on the VPC IP, eliminating the agent's NATS dependency from the daemon side. - service_impl: pushConfig uses http.Post, waitForAgentReady uses http.Get, both target http://<vpcIP>:8405 - albVMUserData: drop natsURL param (agent no longer needs NATS) - types: add VPCIP field to LoadBalancerRecord - daemon: remove SetNATSURL call and 10.0.2.2 translation block - arn_test: update user data tests for new signature * refactor: rewrite ALB health checker from NATS subscription to HTTP polling Health checker now polls each active ALB's GET /health endpoint on a 10s ticker instead of subscribing to a NATS wildcard topic. Removes the NATS connection dependency from the health checker. - health_checker: replace nc/sub with httpClient + stopCh polling loop - newHealthChecker: drop nc param, add agentURLFn for testability - service_impl: update newHealthChecker call site - tests: rewrite with httptest, test pollAll/pollOne paths * refactor: remove NATS wiring from ALB image and make hostfwd dev-only - alb-setup.sh: remove --nats flag, use --listen=:8405; remove placeholder HAProxy frontend on 8405 (agent owns the port now); update description and comments - service_impl: HostfwdPorts (80/443) now only set when DevNetworking is enabled — production ALBs are reached via VPC IP directly * feat: system credentials file + daemon wiring for ALB agent SigV4 auth Write config/system-credentials.json during admin init/join with the system access key. Daemon reads it at startup, resolves the gateway URL from external IPAM (or 10.0.2.2 in dev mode), and wires both into the ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY, and ALB_SECRET_KEY into cloud-init for ALB VMs. * feat: ALB agent gateway actions + daemon handlers (heartbeat, config) Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB agent can communicate via the AWS gateway instead of direct HTTP. - Add agent_types.go with input/output types for both actions - Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active on first contact, updates LastHeartbeat, processes health report - Add GetALBConfig: returns pre-computed HAProxy config text and hash - Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores config + SHA256 hash on LB record for agent pull) - Remove waitForAgentReady polling goroutine (heartbeat handles transition) - Remove HTTP client from ELBv2ServiceImpl (no longer needed) - Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health reports now arrive directly via ALBAgentHeartbeat - Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord - Wire NATS subscriptions + daemon handlers for both new actions - Add gateway handler files and elbv2Actions map entries - Comprehensive tests for heartbeat, config, and updateStoredConfig * feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r) Replace the HTTP server agent with a poll-based client that communicates with the gateway using SigV4-signed requests. The agent now sends ALBAgentHeartbeat every 5s with health reports and fetches config via GetALBConfig when the config hash changes. * feat: add VPC service route for ALB agent gateway access ALB VMs need to reach the gateway for heartbeat/config polling, but VPCs without an IGW have no external routing. Instead of requiring an IGW, the daemon now publishes a vpc.ensure-service-route event before launching ALB VMs. vpcd handles this by creating minimal external connectivity (external switch, localnet port, gateway router port, chassis bindings) if not already present, then adds a /32 static route for the gateway IP. Refactors handleIGWAttach to share the external connectivity setup via ensureExternalConnectivity helper. * Revert "feat: add VPC service route for ALB agent gateway access" This reverts commit |
||
|
Julian Sommer
|
a56a2baf1f | feat: privilege separation (#114) |