mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-16 01:22:00 +00:00
c9a4e904a802441db3d55cda124bb30521ab4261
1114 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 joshsiv-mulga
|
c9a4e904a8 |
test(e2e): fresh-install reachability baselines + drop qemu hostfwd fallback
Add baseline tests that confirm two fresh-install regressions the existing
suite masks (it authorizes tcp/22 on the default SG before every SSH probe):
- default SG default-deny gates external reach of instances in the
default subnet (SG, not routing)
- a newly-created public subnet never gets its OVN IGW egress LRP, so it
has no route to the VPC IGW despite MapPublicIpOnLaunch
Plus a positive test: instances sharing the default SG can communicate via
ICMP (same-host single-node, cross-host multinode).
Single-node baselines run after TestClusterStatsCLI (0-VM gate); multinode
after TestMultinodePreflight (fixture init). Instances are self-cleaning so
they don't perturb the singleton-VM ordering.
Remove the qemu hostfwd SSH fallback from the harness (InstancePublicSSHHost,
GuestSSHEndpoint) - it bypasses the OVN datapath and masks these regressions.
Endpoints now t.Fatal when no public IP is assigned.
|
||
|
Julian Sommer
|
ff0dc42562 | feat: IMDS subnet level (#292) | ||
|
Josh Siviour
|
50bc38c8fd |
feat(ui): NLB support in create/edit load balancer + target group flows (#293)
* feat(ui): NLB support + protocol/type-dependent create-LB fields The Create Load Balancer page was hardcoded to ALB. Add a load-balancer Type selector (application/network) and switch the listener + target-group form options on the selected type, mirroring the server's validation: - ALB: HTTP/HTTPS listeners and target groups. - NLB: TCP/UDP/TLS/TCP_UDP listeners and target groups; no security groups. - Secure protocols (HTTPS/TLS) reveal a certificate selector; HTTPS also a security-policy selector. Cert is required for both; SSL policy is HTTPS-only. - The default target-group protocol + health-check protocol stay in sync with the listener (TCP health checks omit HTTP path/matcher). - The wizard mutation now sends the chosen Type and threads cert/SSL-policy through CreateListener (previously hardcoded Type=application and dropped certificates). * feat(ui): NLB-aware target groups and LB detail Standalone Create Target Group offers the full protocol set (HTTP/HTTPS/TCP/UDP/TLS/TCP_UDP) and syncs the health check to TCP for L4 groups, dropping path/matcher from the request and CLI preview. Hide the Security groups tab on the load balancer detail page for network LBs. |
||
|
Josh Siviour
|
9fa596001f |
fix: EKS 6b create-cluster follow-ups (security + cleanup + tests) (#291)
* fix(admin): write ~/.aws credentials 0600 via atomic temp+rename UpdateAWSINIFile ended with ini.SaveTo, which uses os.Create (0644 & ~umask) and left ~/.aws/credentials world-readable. Render to a 0600 temp file and rename into place so secrets are never exposed. * fix(eks): purge OIDC signing key on zeroize; pin account bucket History=1 ZeroizeClusterOIDCKey used kv.Delete (a tombstone) which leaves prior revisions of the encrypted PEM in KV history. Use kv.Purge to drop the key's whole history. Decouple the account bucket's History from the schema version (KVBucketEKSAccountHistory=1) so a version bump can't silently widen history and resurrect ciphertext. * fix(reconcile): get-or-create leader bucket in AcquireLeader CreateKeyValue returns 'stream name already in use' once the bucket exists, so every reconcile after the first hit the 60s deadline and never elected a leader. Attach to the existing bucket first, create only when genuinely absent. * fix(eks): drop registry entry on lease-held instead of phantom holder RunClusterReconciler returned (nil, nil) when another node held the lease; Spawn recorded a holder with a nil release, so Has() stayed true and a later Spawn no-op'd — the cluster was never re-reconciled until daemon restart. Return ErrLeaseHeld so Spawn drops the entry and retries after the holder's TTL lapses. * refactor: tidy EKS cluster state, dedup, and not-found error wiring - listClusterNames: O(n) map dedup instead of O(n^2) slices.Contains - drop dead store *Store field + NewStore call from EKSServiceImpl - ClusterMeta.LastHealthProbe: drop misleading omitzero tag (stdlib encoding/json ignores it; json/v2 only) and document why it always serializes - clarify ErrorResourceNotFound is the ACM code; EKS uses ErrorEKSResourceNotFound (ResourceNotFoundException) * test: EKS 6b pagination, restart-resume, and bootstrap bind hardening - T7: ListClusters NextToken/MaxResults paging + out-of-range clamp - T4: SpawnRegisteredReconcilers resumes CREATING/ACTIVE on restart, skips terminal clusters, no-ops when deps absent - T-quality: replace sleep-based subscription binding with polling NumSubscriptions, poll KV for the one-shot drop, and assert Run tears down its subscriptions on return T1 (restjson wire shape), T5 (JWKS kty mismatch), and T6 (resolveGatewayHost branches) were already covered. * test(reconcile): cover AcquireLeader get-or-create election First acquire creates the bucket and wins; a second acquire while the lock is held attaches to the existing bucket and loses cleanly (the get-or-create fix, not a CreateKeyValue dead-end); after release a later acquire takes over. |
||
|
Josh Siviour
|
216b9c7682 |
feat(ui): HTTPS listeners — ACM certificate + SSL policy (Phase C) (#290)
* feat(ui): HTTPS listeners with ACM certificate + SSL policy Expose HTTPS in the load balancer listener UI. The ListenerForm gains a HTTPS protocol option that reveals a certificate selector (backed by a new ACM ListCertificates query), an inline certificate import dialog (ImportCertificate), and an SSL security-policy selector (DescribeSSLPolicies). CreateListener/ModifyListener now carry Certificates + SslPolicy for HTTPS; the listeners table summarises the policy and certificate per HTTPS listener. Adds an ACM client (getAcmClient) and @aws-sdk/client-acm. Server, gateway and ACM endpoints already exist (epic phases A + B); this is frontend-only. * fix: ACM JSON timestamps, react-compiler watch reactivity, scroll-chaining - gateway/acm: serialise responses with jsonutil.BuildJSON so time.Time fields emit epoch seconds (AWS JSON 1.1 wire format). encoding/json emitted RFC3339 strings the JS SDK rejected with 'Expected real number', crashing the LB detail Listeners tab once any cert exists. - frontend: replace watch() with useWatch across forms so the React Compiler tracks the subscription and re-renders on field change (HTTPS listener cert/policy fields now reveal). Add toggle regression test for listener-form. - frontend: scope overscroll-none to html,body instead of *, restoring wheel scroll-chaining over nested scroll containers (sidebar, tables, dialogs). |
||
|
Julian Sommer
|
bcb49cd771 | feat: IMDS v1 (#266) | ||
|
Josh Siviour
|
f14fb64562 |
feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288)
* feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first. |
||
|
Josh Siviour
|
bfacca7789 |
feat(eks): 6c token webhook + AccessEntry (IAM auth → kubectl) (#284)
* feat(eks): 6c AccessEntry CRUD + access-policy association
Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real
KV-backed handlers (Q9, API auth mode):
- AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash};
the principal ARN is hashed into the key (ARNs carry ':', illegal in
NATS KV keys) while the record keeps the plaintext ARN.
- AssociateAccessPolicy/Disassociate/ListAssociated against the four
supported AWS-managed policies; unsupported policy ARN or bad access
scope -> InvalidParameterValue.
- ListAccessPolicies returns the static supported catalogue.
- Cluster-existence gate -> ResourceNotFoundException; duplicate create
-> ResourceInUseException; node-type entries rejected until 6d.
Updates the daemon routing test for the now-implemented handlers.
* feat(eks): plumb caller principal ARN + bootstrap-creator-admin
Thread the caller's resolved IAM principal ARN from the gateway to the
EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster
can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9):
- utils.NATSRequest takes optional extra headers; PrincipalARNHeader +
PrincipalARNFromMsg added.
- Gateway resolves the caller ARN best-effort (buildCallerARN) and passes
it down the EKS route table; only CreateCluster consumes it.
- CreateCluster signature gains callerPrincipalARN; default-true bootstrap
flag mints a system:masters entry keyed by the caller's exact ARN (the
token webhook resolves the same ARN from STS). '' degrades gracefully.
- Dedicated daemon handler reads both headers.
* feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring
The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl
requests that present an `aws eks get-token` bearer token: decode the presigned
STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster
AccessEntry, and answer the apiserver TokenReview with the mapped username +
groups.
STS is hosted in-process in the awsgw service and is not otherwise on the bus,
so awsgw now serves an `eks.VerifyToken` NATS subject that wraps
VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces
the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly
over its own NATS connection.
Apiserver wiring: the k3s config skeleton sets
authentication-token-webhook-config-file; the webhook writes its own self-signed
serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a
bounded wait in the k3s start_pre so the apiserver never loads a missing file).
* fix(eks): omit empty spec from token webhook TokenReview response
* fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD
VerifyPresignedGetCallerIdentity built the SigV4 canonical request with
"UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore
and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws
eks get-token` produces) with the SHA256 of the empty string. Every real
get-token therefore failed signature verification with InvalidIdentityToken,
breaking the entire EKS IAM token-auth flow end to end.
Use the empty-string SHA256. The existing hand-rolled presign tests masked the
bug by signing with the same wrong constant; they now sign correctly, and a new
test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the
CLI uses) so this can't regress silently.
Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to
the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid
token presented for a different cluster is still rejected (anti-replay intact).
* fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve
IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI
percent-encodes in the request path. The dispatcher matched r.URL.Path
(decoded), so %2F became a literal '/' and split the ARN across the
([^/]+) capture, dropping every describe/update/delete-access-entry and
associate/disassociate/list-access-policy call to InvalidActionException.
Match r.URL.EscapedPath() so the encoded ARN stays one segment, then
url.PathUnescape each captured param before it reaches the handler.
* test(eks): unit-cover access-entry scope validation, AWS converters, store guards
|
||
|
Josh Siviour
|
b0227e7eeb |
chore(ansible): remove ansible tree, uplift to mulga/ansible/ (#287)
Ansible orchestrates the whole stack (spinifex + predastore + viperblock + OVN + tofu hosts) and its spec/beads live in mulga. Moving it to top-level mulga/ansible/ ends the per-change submodule-pointer-bump friction. - git rm -r scripts/ansible/ - Makefile: drop ansible-dev-* / ansible-cluster-bootstrap targets + .PHONY The tree lands at mulga/ansible/ (flat copy, Option B). History pivot: tag feat/uplift-ansible-snapshot-pre-move marks the pre-move HEAD. Plan: docs/development/improvements/ansible-uplift-to-mulga.md (in mulga). |
||
|
Josh Siviour
|
61d73d1d4e |
fix: reject unknown ACM cert ARN at listener API boundary (#289)
An unresolvable certificate ARN was accepted into a listener record and only failed later at config-render time, which aborted updateStoredConfig wholesale and silently froze the load balancer's data-plane convergence. Validate every certificate ARN against the ACM store (existence + account ownership) in CreateListener, ModifyListener and AddListenerCertificates, returning CertificateNotFound up front. Skipped when the ACM store is unavailable, matching the nil-safe resolve path. |
||
|
Josh Siviour
|
470954bbe8 |
feat(elbv2): HTTPS data plane — TLS termination via ssl crt + PEM delivery (#286)
Resolve listener certificate ARNs to PEM via the ACM store (shared JetStream KV, no cross-service hop), render HAProxy 'ssl crt' for secure frontends, and deliver the combined cert+chain+key PEM to the LB agent alongside the config. ConfigHash folds in cert content so rotation triggers a reload; the agent writes each cert 0600 (path-constrained to its cert dir) before the config + reload. Phase B of the ELBv2 HTTPS epic (mulga-siv-180). |
||
|
Josh Siviour
|
38f5dd0d41 |
feat(acm): minimal ACM-alike certificate service (#285)
Stand up acm as its own AWS JSON 1.1 gateway service wiring the BYO-cert import/store path ELBv2 HTTPS listeners need. ImportCertificate validates tls.X509KeyPair, parses the leaf for domain/serial/expiry/key-algorithm, mints arn:aws:acm:<region>:<acct>:certificate/<uuid>, and stores PEM (cert, chain, key) in JetStream KV. DescribeCertificate/ListCertificates/ DeleteCertificate round-trip; access is account-scoped. New gateway dispatch by X-Amz-Target (CertificateManager.<Action>); errors share the AWS JSON 1.1 envelope with EKS. Wired through gateway/acm ops, handlers/acm (service iface, store, NATS client), daemon handlers + acm.* subscriptions, and service-interfaces.yaml. Out of scope (future, same epic): RequestCertificate + DNS validation, managed renewal, Get/ExportCertificate, tagging, ACM-PCA. |
||
|
Josh Siviour
|
e06d0c0865 |
feat(elbv2): HTTPS listener certificates + SSL policies (Phase A) (#283)
* feat(elbv2): HTTPS listener certificates + SSL policies Add Certificates[]/SslPolicy to ListenerRecord. CreateListener and ModifyListener validate secure protocols (HTTPS/TLS) require >=1 certificate with exactly one default, non-secure protocols reject certs/policy, and default+validate the SSL policy. DescribeListeners echoes certificates and SslPolicy. New endpoints: AddListenerCertificates, RemoveListenerCertificates, DescribeListenerCertificates, DescribeSSLPolicies (gateway ops, NATS subjects, daemon handlers, service-interfaces.yaml). No TLS termination yet. * test(elbv2): cover listener-cert gateway ops + handler error paths |
||
|
Josh Siviour
|
bbff004b45 |
feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280)
* feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep
Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first
read/write helpers against the eks-account-{accountID} bucket so later
stages (oidc keypair, NLB wiring, cluster reconciler) can persist state
without re-inventing CAS/sweep logic.
- ClusterMeta struct holds the DescribeCluster source of truth (status,
endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN).
- ClusterStatus enum mirrors the AWS EKS string set verbatim.
- PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta
returns ErrClusterNotFound on key absence so the service boundary can
map to ResourceNotFoundException.
- SetClusterStatus is revision-checked with bounded CAS retries so the
reconciler and CreateCluster body cannot wedge each other.
- DeleteClusterPrefix sweeps every clusters/{name}/* key; called from
DeleteCluster after the OIDC private key is zeroized.
Tests cover round-trip, missing/corrupt blob, idempotent status set,
DELETING transition from every prior state, concurrent revision bump
recovery, sibling-cluster preservation, empty-bucket no-op.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize
Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body
will call before launching the K3s server VM. Pure crypto helpers — no
daemon, no launcher, no NATS bus.
- GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster
(matches eks-v1.md Q8), PEM-marshals the private as PKCS8,
envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM,
AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and
publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS
bytes so CreateCluster can hand them straight to user-data without
a follow-up KV read.
- JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA
verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER,
alg ES256, use sig). Types are defined locally to avoid a
handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire
compat is the contract.
- LoadClusterOIDCPrivateKey is the recovery-path read for future
stages (reconciler restart, restore-snapshot tooling). Validates
PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else.
- ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of
the same length, then deletes. DeleteCluster calls this *first* so a
crash mid-teardown leaves no recoverable key material. Idempotent
on missing key — supports retry semantics.
Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1
coordinates), per-cluster key uniqueness, round-trip private-key
decrypt matches JWKS public, wrong-master-key fails, missing-key
ErrClusterNotFound mapping, zeroize idempotency, JWKS survives
zeroize (carries no secret material), padCoord left-pad correctness.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): cluster SG + NLB ensure/delete helpers
Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account
SG provisioning + per-cluster NLB plumbing the CreateCluster body will
call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure
interface-call helpers — no daemon, no KV, no NATS bus.
- EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg
and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged
spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role.
Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort
with continue-on-error sweep.
- EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the
eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md
Q13), and the TCP :443 forward listener. Returns ClusterNLB
{LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster
can persist every ARN in ClusterMeta. Idempotent on cluster name;
resource-name length validated against the AWS 32-char cap so a too-long
cluster name fails with a clear error instead of a generic
InvalidParameterValue downstream.
- RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM
ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its
ENI IP; DeleteCluster calls Deregister before VM termination.
- DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already
cascades listener cleanup inside the ELBv2 impl. LB sweep error takes
precedence so DeleteCluster surfaces the more visible failure first
while the TG sweep still runs.
Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests
hand-rolled recorder fakes; the production VPCService + ELBv2Service
satisfy them structurally so no adapter is needed at Stage 6 wire-up.
Tests cover fresh-creates with right names/tags/ports, idempotent reuse,
mixed-existence (control-plane exists, nodegroup created), name-too-long
rejection, empty-inputs rejection, missing-resource no-op on delete, and
first-error-surfaced sweep continuation on delete.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(ec2): RunInstances honors pre-created NetworkInterfaceId
Closes the AWS-parity gap where RunInstances always auto-created a
primary ENI even when the caller specified
NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster
launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the
ENI can be tagged/owned distinctly from the launcher's instance; the
existing path silently created a second ENI and left the caller's
unattached.
Changes:
- ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo
is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status,
SecurityGroupIDs) so handlers/ec2/instance stays free of an import on
handlers/ec2/vpc.ENIRecord.
- daemon/vpc_adapter.go gains daemonENICreator.GetENI translating
VPCServiceImpl.GetENIRecord into ENIInfo.
- PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId
after the existing subnet/SG lift block. If non-empty, the loop
dispatches to a new attachPreCreatedENI helper:
- Looks up the ENI under accountID.
- Rejects with InvalidNetworkInterface.InUse if Status == "in-use".
- Calls AttachENI to mark the ENI in-use and bind it to the new VM.
- Populates instance.ENIId / ENIMac and ec2.Instance network fields
(PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]).
- Skips the auto-create + public-IP-auto-assign paths; the caller is
expected to manage public IP out-of-band.
- Auto-create path is left untouched for every other code path: callers
that pass SubnetId without a NetworkInterfaceId still get a fresh ENI
with the existing semantics.
Tests cover the four shapes: happy path (auto-create not called, attach
called once, instance populated), in-use ENI rejected with the canonical
AWS error code, lookup error surfaces verbatim, attach error surfaces
verbatim.
* feat(eks): K3s control-plane VM launcher with cloud-init user-data
LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the
customer-account control-plane ENI in the supplied subnet with the
control-plane SG, renders cloud-init user-data (first-boot env, OIDC
private key, K3s config.yaml with tls-san + service-account args), then
calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing
at the pre-created ENI. RunInstances failures roll back the ENI so the
caller does not leak a customer-account resource.
TerminateK3sServerVM tears the instance and ENI down for DeleteCluster;
missing identifiers are a no-op so retries stay safe.
ENI + VM both live in the customer account (deviation from plan §7 which
specified a SystemAccount-owned VM); cross-account ENI attach is deferred
to a follow-up bead.
* feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease
NATSBootstrap subscribes to the four single-shot subjects the K3s server VM
publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca)
on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are
envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The
published JWKS is cross-checked against the controller-generated keypair by
kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64
via the new SetClusterCertificateAuthority CAS updater.
ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/
{clusterName} (60s TTL, 20s refresh) and drives:
- CREATING -> ACTIVE once the four bootstrap KV artifacts are present and
NLB /healthz responds 200.
- ACTIVE -> probe /healthz on the configured interval, warn on failure.
- DELETING / FAILED -> exit (DeleteCluster body owns teardown).
Loss of the leader lease (CAS conflict or different holderID) returns
ErrReconcilerLeaseLost so the caller can re-acquire from another node.
Tested with embedded JetStream + a stub HTTPDoer; race-clean.
* feat(eks): OIDC issuer URL builder + reconciler registry
ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/
{clusterName} URL kube-apiserver advertises as service-account-issuer and
that STS / external IRSA validators dereference. Validates URL safety on the
path segments so a hostile cluster name cannot inject scheme/path fragments.
ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines
with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait
for release), and StopAll (graceful daemon shutdown). RunClusterReconciler
is the production SpawnReconcilerFn: constructs a ClusterReconciler,
acquires the leader lease, runs Run in a goroutine. Both helpers consumed
by the upcoming service_impl CreateCluster body.
* feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies
Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB,
Instance launcher, AMI resolver, master key, gateway base URL, region,
holder ID, reconciler registry) and implements the four lifecycle
methods:
- CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer +
keypair → K3s server VM (cloud-init carries OIDC private PEM) →
register ENI IP with NLB target group → spawn NATS bootstrap
subscriber + per-cluster reconciler via the registry.
- DescribeCluster: KV meta read → eks.Cluster shape, with
ErrClusterNotFound translated to ResourceNotFoundException at the
service boundary.
- ListClusters: KV key-prefix scan + sorted maxResults/nextToken
pagination.
- DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop
reconciler via registry → deregister NLB target + delete NLB →
terminate K3s VM → delete SGs → delete cluster KV prefix.
Daemon wiring:
- buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key
best-effort, resolves the gateway base URL via the same precedence the
LB-agent block applies later, and passes VPCService + ELBv2Service +
ImageService directly while adapting InstanceServiceImpl via
daemonEKSInstanceLauncher (RunInstances passes through; Terminate-
Instances loops vmMgr.Get + StopOrTerminateInstance in-process).
- SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping
VPCService.GetSubnet.
- Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster
via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so
every per-cluster goroutine releases its lease before exit.
Test surface:
- service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers
the four lifecycle methods when constructed via the back-compat shim
(CreateCluster/DeleteCluster → ServerInternal, DescribeCluster →
ResourceNotFound on empty bucket, ListClusters → empty list).
- daemon_handlers_eks_test: drop the four lifecycle subjects from the
NotImplemented routing loop; handler-side coverage now lives in
handlers/eks.
- awserrors: add ResourceNotFound (404) for the DescribeCluster path.
* fix(eks): load master.key from config dir not service data dir
eks_deps read master.key from {BaseDir}/config/master.key, but the daemon
systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service
data dir). admin init writes master.key to the shared config dir, so the
daemon looked in a path that only resolved when a {BaseDir}/config symlink
existed — awsgw has one, the daemon's nested data dir does not. Result:
MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster
collapse to ServerInternal.
master.key is a config artifact written next to spinifex.toml. Read it from
filepath.Dir(d.configPath), matching how the daemon already resolves
predastore.toml and TLS certs. No symlink or provisioning change needed.
* fix(eks): register + resolve eks-server AMI via tag, add import --ami-name
CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every
clean install. Two defects:
1. spx admin images import had no way to name a locally-built appliance AMI:
--name is a catalog lookup (rejects non-catalog values) and the AMI name
was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran
'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all
failed 'Image name not found in available images'.
2. The launcher resolved DescribeImages by name=='eks-server', a name the
pipeline could never produce.
Add --ami-name to override the registered AMI name on --file imports (fixes
eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the
spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on
multiple matches. Tag-based resolution survives the planned server+agent →
single EKS AMI unify untouched (role chosen per-instance at launch).
build-system-image.sh now passes --ami-name; Makefile gains
import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when
DEV_INSTALL_EKS=1.
* fix(eks): give K3s server VM NATS token + CA so bootstrap can publish
The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback,
unreachable from the VM) and no auth material, while spinifex NATS requires a
shared token over TLS. The VM could not connect to NATS, so it never published
its four one-shot bootstrap messages and the cluster stayed in CREATING.
Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL
(mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys
hierarchy to scope per-principal creds against).
- K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert,
required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN,
SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600.
- EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert.
- buildEKSServiceDeps: read CA from config.NATS.CACert; extract
resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port).
- k3s-first-boot.sh: build nats CLI args with --token + --tlsca.
* fix(eks): retry transient bootstrap persists, fail cluster on permanent error
The NATS bootstrap subscriber tore down on the first persist error and the
cluster sat in CREATING forever. Core-NATS delivers a publication once, so a
transient KV blip during persist was unrecoverable and a malformed payload
(bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster.
- persistWithRetry rides out transient persist failures in-handler (3 attempts,
200ms backoff) instead of unsubscribing and waiting for a re-publish that
never comes.
- errBootstrapPermanent classifies validation failures (malformed envelope,
empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they
return immediately.
- spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a
non-context error, so DescribeCluster surfaces the cause and DeleteCluster
can reclaim resources.
- MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a
late bootstrap error cannot clobber a concurrent delete or healthy cluster.
- casUpdateMeta extracts the revision-checked RMW loop shared by the meta
setters.
* fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart
SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the
bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM
still booting) lost the subscriptions; the VM published its four one-shot
core-NATS messages to nobody and the cluster hung in CREATING forever.
- BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are
still missing from KV. JWKS is excluded: it persists no artifact and
re-waiting on a once-published subject would deadlock — its cross-check is
best-effort across a restart.
- RunForKinds subscribes to a subject subset (nil = all four); Run delegates to
it. subjectHandlers maps each kind to its handler and rejects unknown kinds.
- SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting
only on the missing artifacts, so a mid-CREATING restart still drives the
cluster to ACTIVE.
* fix(eks): DeleteCluster must not wipe KV or report success on teardown failure
DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue
then unconditionally swept the per-cluster KV prefix and returned success. A
failed NLB or VM teardown leaked billable resources with no owning record (the
meta holding their ARNs/IDs was erased) and told the operator the cluster was
gone. Zeroize — a stated security guarantee of no recoverable key material —
had its failure ignored too.
- Zeroize, NLB delete, and VM terminate are now blocking: their failures
accumulate (errors.Join) and abort the call before the KV sweep, leaving the
cluster DELETING and returning the error so a retry can still reach the ARNs.
- KV prefix sweep runs only after NLB + VM teardown confirm.
- SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning
record — and target deregister stays best-effort since DeleteClusterNLB tears
down the whole NLB + target group.
* fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak
CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and
persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any
failure between EnsureClusterNLB and that persist left the NLB + target group +
listener + backing LB VM created but unrecorded; markFailed did no rollback and
DeleteCluster keys teardown off the persisted ARNs, so the resources leaked.
- Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and
again after LaunchK3sServerVM (records instance + ENI), before the next
fallible step.
- A failed create now leaves a FAILED meta carrying the resource references, so
the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM.
- Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete
reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture
extracted from the delete-cluster fixture.
* fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens
ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment
path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves —
https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b
cluster's iss claim was rejected with "issuer host must start with oidc.eks",
so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them.
Reconcile on the path-based shape, which is what is actually reachable on
bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from
the IAM OIDC-provider registration (the full issuer string is the registry key)
and the JWKS signature check, not a host pattern — so the parser drops the host
prefix rule and instead pins accountID/clusterName at fixed positions in a
strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted
iss with extra or missing segments still fails closed.
Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss
parses back to the same (accountID, clusterName).
* fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors
A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the
preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently
in use...") for a cluster-exists condition, and because that maps to a
retryable code the SDK's auto-retry surfaced ResourceInUseException instead of
the original failure. FAILED clusters were also a dead state — visible to
neither a clean recreate nor a clear lifecycle.
- Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with
cluster-scoped messages; CreateCluster's exists check and DescribeCluster /
DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes.
- CreateCluster reclaims a FAILED cluster: it purges the prior attempt's
recorded resources and clears KV before recreating, so a retry (including the
SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces
rather than recreating on top of orphans.
- Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path
(blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown
confirms).
- FAILED clusters remain observable via Describe (status=FAILED) and List — no
state where create blocks but describe/list show nothing.
* fix(eks): fail CREATING clusters that exceed create timeout
A per-cluster reconciler that never observes a ready control plane (bootstrap
artifacts missing or healthz unreachable) previously stayed CREATING forever.
Add a configurable create timeout (default 15m): once meta.CreatedAt is older
than the deadline and the cluster is still not ready, the reconciler marks it
FAILED with a reason, exiting the loop. Within the deadline it stays CREATING.
* fix(eks): surface ACTIVE cluster health in describe-cluster
The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE
cluster, so describe-cluster kept reporting a dead control plane as healthy.
Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE
reconcile branch (recording the failure reason, clearing on recovery), and map
a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in
describe-cluster.
* test(eks): add plan-mandated CreateCluster/DeleteCluster service cases
service_impl_test only covered shim mode. Add the four cases the plan's Testing
section requires: CONFIG_MAP auth mode rejected (and API accepted), missing
subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim
short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven
by forcing a teardown failure and asserting the OIDC key is already wiped.
* refactor(eks): return OIDC private-key PEM from generator, drop re-read
CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second
KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the
generator already had in hand. Return the plaintext PEM (alongside the JWKS)
from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now
-unused crypto/x509 + encoding/pem imports in service_impl.
* fix(eks): make CreateCluster/DeleteCluster failures diagnosable
Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare
ServerInternal with no request-time log, leaving the client a 'contact AWS
re:Post' message and zero cause. Enumerate the specific missing deps and log
them at ERROR, returning ServiceUnavailable (an operator/config fault, not an
unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a
missing control-plane AMI is logged and surfaced as ServiceUnavailable too.
* fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them
WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json,
which ignores their locationName tags and emits Go PascalCase keys. EKS
clients parse restjson on the lowercase locationName, so every field of
list/describe/create/delete responses was silently dropped (empty endpoint,
null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same
marshaler the SDK uses for restjson bodies — which honors locationName.
Adds wire-shape tests asserting lowercase keys present and PascalCase absent.
* feat(eks): launch K3s server VM via boot-agnostic system-instance path
The EKS K3s control-plane VM launched through the plain tenant RunInstances
path, so it had only its VPC ENI and no management-bridge NIC -- no route to
the daemon's NATS endpoint. It never published its bootstrap subjects and the
cluster stayed CREATING.
Generalize LaunchSystemInstance to a boot-agnostic contract supporting both
system-instance boot styles:
- BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent
- BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM
The launch contract moves to a neutral handlers/sysinstance package so EKS can
launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports
the types as aliases. The BootAMI branch mirrors the daemon's
Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record
between Insert and Launch, so the cloud-init network-config renders a static
mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the
on-link address the VM's mgmt NIC can reach), falling back to the gateway-host
resolution when no mgmt bridge exists.
Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC
(10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the
bridge IP. No AMI rebuild required.
* fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge
TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every
TerminateSystemInstance error -- as blocking. Because instance terminate is
async, the immediate ENI delete races it: the first attempt fails
InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases
the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into
purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep
and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING
permanently, leaking its control-plane + nodegroup SGs and KV meta.
Classify the teardown errors instead of blanket-propagating:
- ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound
-> idempotent success (the cascade or a prior retry already deleted it);
must not block the downstream SG + KV sweep.
- ENI InvalidNetworkInterface.InUse (VM still terminating) and any other
error -> retryable; surfaced so the cluster stays DELETING and the
reconciler retries once the instance releases the ENI.
- Instance already gone -> idempotent. TerminateSystemInstance now returns a
sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates
via errors.Is, so a retried delete after the VM drained completes.
Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier
(substring match through %w wrapping) and route the lone ad-hoc strings.Contains
guard in daemon/vm_adapters.go through it.
Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI +
VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak.
* fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key
The controller pre-seeds OIDCJWKSKey at create time, so its presence
proved nothing about the running cluster's actual signing key. A cluster
could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's
published JWKS was ever cross-checked against the generated keypair.
Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after
assertJWKSMatch passes. The reconciler's ACTIVE gate and
BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS.
assertJWKSMatch is map-based and also rejects kty mismatch.
* fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost
startCluster reimplemented the 5-case gateway-host precedence inline,
duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL).
If one drifted, the OIDC issuer host embedded in cluster tokens would
diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch
and cluster-wide token rejection.
startCluster now calls resolveGatewayHost and derives the multi-node
mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost
carries the canonical case enumeration; a table test pins all five
branches plus the case-1 skip guards and the no-host fallthrough.
* fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash
Three issues found while diagnosing why K3s clusters never leave CREATING:
1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in
default_kernel_opts, so Linux binds /dev/console to the framebuffer and the
host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and
k3s-first-boot output were invisible. Reorder so console=ttyS0 is last.
2. k3s-first-boot.initd: command_background was "no" while output_log/error_log
were set; start-stop-daemon rejects --stdout/--stderr without --background
("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and
the cluster wedged in CREATING. Background it (the publisher waits up to 5min
on K3s /healthz and must not block the runlevel anyway) and route its output
to the serial console, which the orchestrator captures (the VM has no login).
3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log
to the console so the apiserver's own failure is visible host-side.
* fix(eks): seed K3s apiserver a public OIDC key, not the private key
buildK3sUserData pointed both kube-apiserver service-account args at the same
private-key file. --service-account-key-file requires a PUBLIC key — k8s
ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any
valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never
reaches /healthz, first-boot never publishes, and the cluster wedges in
CREATING.
Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and
seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point
service-account-key-file at the public file; keep service-account-signing-key-file
on the private file. Runtime-verified: the apiserver crash is gone and K3s now
starts and schedules system pods.
* feat(eks): egress-only SNAT + DNS for K3s control-plane VM
The K3s server VM (a hidden system instance) could not pull system-pod
images: it had no internet egress and no working resolver, so
coredns/traefik/metrics-server never started and /healthz timed out.
Egress: give the single control-plane VM egress-only internet, confined
to its /32, with no inbound exposure and no customer-visible resource.
- policy/route.go: SystemInstanceEgressPriority (1200, above the drop
gate) + AddSystemInstanceEgress/Delete — a reroute scoped to
ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected.
- policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not
dnat_and_snat, so no inbound path), idempotent on its pool external IP.
- external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the
gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW
egress path; ADR-0006 S1/S4/S5 preserved).
- subscribers: vpc.add/delete-system-egress topics + handlers.
- handlers/eks: allocate a hidden (ManagedBy=eks) pool address on
CreateCluster and publish add; release + publish delete on teardown.
EIP dep wired through EKSServiceDeps + the daemon.
DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a
dangling symlink, so DHCP-supplied resolvers are never persisted. Write
a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not
write_files — a write_files entry follows the dead symlink, fails, and
aborts the whole block, dropping first-boot.env + the k3s config).
* feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes
Switch the K3s server VM from embedded etcd to embedded SQLite (kine)
for the single-node v1, so CreateCluster reaches a healthy control plane.
etcd's per-commit fsync stalls past the apiserver deadline on viperblock's
deferred-durability block path; SQLite tolerates it.
- k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true
so the reconciler's anonymous /healthz NLB probe returns 200.
- first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth
=false) -> authenticated 'kubectl get --raw /readyz'.
- first-boot publish: emit a JSON BootstrapEnvelope for all four subjects
(token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before.
- nats pub: --stdin -> --force-stdin (nats CLI 0.4.0).
- manifest: add jq (builds the bootstrap-message JSON envelopes).
* feat: EKS reconciler probes control-plane mgmt IP until DNS lands
The host-side cluster reconciler cannot resolve or route to the VPC-internal
NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed
out in CREATING and was marked FAILED even though the control plane was healthy.
Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver
endpoint reachable from the daemon. Capture the mgmt IP from the launch result
(SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta.
ControlPlaneMgmtIP) and build the reconciler health URL as
https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify,
so the mgmt IP need not be in the apiserver cert SAN.
Workaround until authoritative DNS + a host-reachable NLB front-end land, at
which point spawnReconciler reverts to the NLB endpoint.
|
||
|
Josh Siviour
|
7b9eb9cc4e |
feat(elbv2): tag/SG/IP-type setters, ip targets, redirect actions, delete-cascade hardening (#282)
* fix(elbv2): make resource deletes idempotent on absent
DeleteLoadBalancer, DeleteTargetGroup, DeleteListener and DeleteRule
returned a NotFound error when the target was already gone. Real AWS
ELBv2 deletes are idempotent (success on absent), so terraform-provider-aws
fails a destroy step on retry. Return an empty success output instead,
mirroring the KV store which already tolerates absent keys.
Adds TestRLC1_ELBv2DeleteIdempotentOnAbsent enforcing the contract across
the ELBv2 delete surface; converts the three _NotFound delete tests to
assert idempotent success.
Refs mulga-siv-172 (idempotency clause; cascade-composition clause stays open).
* feat: ELBv2 AddTags/RemoveTags, SetSecurityGroups, SetIpAddressType
- O(1) GetListenerByArn via ARN-parse short-ID lookup + ARN equality check
- persist tags on listeners/rules; seed at create time
- AddTags/RemoveTags across LB, target group, listener, rule resources
- SetSecurityGroups (ALB only) reattaches SGs to each ENI via live port update
- SetIpAddressType (ipv4-only, idempotent)
- wire all four ops through gateway, NATS client, and daemon handlers
- frontend: editable Tags tab on LB/TG detail, Security groups tab on LB detail
* feat(elbv2): redirect listener/rule actions
- add redirect action type; parse/store/read-back RedirectActionConfig
- validate status code + render-safe fields (strip AWS #{...} placeholders)
- accept redirect and fixed-response on listener rules (not just defaults)
- HAProxy: http-request redirect scheme (HTTP->HTTPS fast path) or rebuilt
location with placeholder->fetch translation; synthetic per-rule backends
* refactor: dedupe ELBv2 attribute handlers and harden store lookups
Collapse the four near-identical Modify/Describe attribute handlers (target
group and load balancer mirror pairs) onto generic helpers in both the service
impl and the gateway, and table-drive the mirror-pair tests over a TG/LB kind.
- service_impl_attributes.go: generic attrResource/modifyResourceAttributes/
describeResourceAttributes plus sdkAttrsIn/sdkAttrsOut; folds in nil-input
guards, explicit empty-accountID rejection, fresh-string output, and a
no-op skip-persist for unchanged Modify calls.
- gateway/elbv2/attributes.go: generic attributeOp + shared validateAttrArn,
replacing four per-endpoint files.
- types.go: default attribute sets are now package-level maps cloned on access
instead of rebuilt per Describe; document the access_logs empty-string defaults.
- store.go: drop the stale 'linear scan' docstring and log ARN mismatches on
GetLoadBalancerByArn/GetTargetGroupByArn instead of silently returning nil.
Adds cross-account isolation, wrong-account no-mutation, and empty-string-clear
coverage.
* feat: support ip target type in ELBv2 RegisterTargets
RegisterTargets resolved every target ID through an instance ENI lookup, so
ip-type target groups silently no-op'd with an empty PrivateIP and never
received traffic.
- types.go: add TargetTypeInstance/TargetTypeIP constants.
- CreateTargetGroup: default to instance; reject target types other than
instance/ip with InvalidParameterValue.
- RegisterTargets: resolve PrivateIP per target type via resolveRegisteredTargetIP
— ip targets use the supplied IP directly (validated with net.ParseIP),
instance targets keep the ENI lookup and reject IP-shaped IDs.
* fix: cascade rule deletion in ELBv2 DeleteLoadBalancer
DeleteLoadBalancer deleted listeners via store.DeleteListener directly, which
bypasses the rule cascade in the DeleteListener handler. Orphan rules survived
LB teardown and pinned their target groups as ResourceInUse, leaving the TG
permanently undeletable and breaking tofu destroy on multinode.
Extract deleteListenerCascade (rules then listener record) and call it from both
DeleteListener and the DeleteLoadBalancer cleanup loop.
Regression: TestDeleteLoadBalancer_CascadesRuleDeletion.
* fix: harden ELBv2 delete lifecycle against orphaned cascades
- DeleteLoadBalancer: fail-fast on listener-list/cascade error and run the
rule cascade before backing-artifact teardown, so a transient failure
leaves the record intact for retry instead of orphaning rules.
- DeleteTargetGroup: in-use scan skips listeners/rules whose owning load
balancer no longer exists, so a partial-teardown orphan can't pin a target
group as ResourceInUse permanently.
- DeleteRule: propagate the listener lookup error instead of discarding it,
so a failed HAProxy reload is no longer reported as success.
- Add lifecycle invariants for no-orphan completeness, no store-bypass
cascade, and target-group deletability after LB teardown.
- Drop plan/bead references from comments.
* test: add NATS e2e round-trips for new ELBv2 ops
Cover the client→NATS→handler marshalling path that unit tests skip:
tags lifecycle (AddTags/DescribeTags/RemoveTags), target-group and
load-balancer attribute round-trips, redirect-action default listener,
ip target-type registration, and SetSecurityGroups/SetIpAddressType.
* fix(elbv2): correct misleading InvalidConfigurationRequest message
The error code is shared by five unrelated conditions (dualstack
rejection in SetIpAddressType, cross-listener rule moves, unsupported
rule action types, and the two genuine NLB security-group cases), but
its canonical message claimed security groups were unsupported on
network load balancers. Callers hitting it from SetIpAddressType or the
rule paths got a message about security groups and NLBs that did not
match their request.
Neutralize to a generic, condition-agnostic message. The AWS error code
is unchanged.
* style(ui): oxfmt security-groups-editor test
|
||
|
Josh Siviour
|
b3f7251239 |
refactor: migrate lbagent signing to predastore/auth.SignReq (#281)
agent.go signed gateway requests with aws-sdk-go v1's v4.Signer. Switch to predastore/auth.SignReq (a thin wrapper over aws-sdk-go-v2's v4.Signer) so the monorepo runs a single signer version and drops the last direct dependency on aws-sdk-go v1's signer subpackage. - Replace the *v4.Signer field with the raw access/secret keys. - signedPost computes the hex SHA-256 payload hash and calls auth.SignReq. - Add TestSignedPost_ProducesVerifiableSignature: the signed request parses and verifies server-side via auth.ParseReq/Verify with a matching body hash. |
||
|
Josh Siviour
|
de5ee4339a |
fix(elbv2): render fixed-response listener defaults + DescribeTags listener-rules (#278)
Shared-ingress ALBs use a fixed-response default action (HTTP 404) with host-header rules forwarding to per-app target groups. Two gaps broke this: - The HAProxy generator read DefaultActions[0].TargetGroupArn unconditionally, so a fixed-response default (empty ARN) rendered `default_backend bk_`, a dangling reference that made HAProxy fail to start — the listener never bound :80 and the LB served no traffic. Store and render the fixed-response config as a synthetic backend (`http-request return status N ...`), with status, content-type, and body validated before interpolation. - ListenerAction dropped the fixed-response config on create/modify, so DescribeListeners read back an empty action and tofu saw a perpetual diff. Preserve it through CreateListener/ModifyListener and the SDK conversion. - DescribeTags returned InvalidParameterValue for listener-rule ARNs; resolve the rule and return an (untagged) descriptor instead. Adds generator tests for the fixed-response default (with host-header rules) and unsafe-body rejection, plus DescribeTags listener-rule coverage. |
||
|
joshsiv-mulga
|
3de1d6d7c5 |
fix(microvm): build initramfs device nodes via fakeroot, drop sudo mknod
sudo mknod created real /dev/console and /dev/null char-device nodes on the host under $CHROOT_DIR. Real device nodes on the host filesystem are a footgun: a stale-image cleanup, tmp reaper, or an empty $CHROOT_DIR (which would redirect the recursive /dev removal at the host root) could clobber host /dev/null and force a reboot, breaking CI servers. Write the two static nodes the kernel needs into the cpio archive inside a fakeroot session instead. mknod/cpio see faked device entries and emit real char-device records into the archive, while the host filesystem never gets an actual node. init mounts devtmpfs over /dev at runtime for everything else. Also guard $CHROOT_DIR against empty/unexpected values before any destructive removal, and add fakeroot to the required-tools check. |
||
|
Julian Sommer
|
044d35737e | feat: remove bash e2e for e2e-go | ||
|
Julian Sommer
|
2dada2cf9d | release: update go.mod for v1.6.0 v1.6.0 | ||
|
Julian Sommer
|
9508aaacce | chore: bump go 1.26.4 | ||
|
joshsiv-mulga
|
40deb1e4c2 |
fix(e2e): add explicit IGW default route in run-lb-e2e
After PR #270 the VPC-LR-wide 0.0.0.0/0 -> IGW static route is no longer installed implicitly at attach-internet-gateway. Subnets whose effective route table lacks an internet-bound route get an OVN drop policy at SubnetEgressPriorityDrop (1100), which kills LB target reply traffic before un-DNAT can fire and stalls the LB internal-client probe until job timeout. Add the standard AWS pattern: describe-route-tables -> create-route 0.0.0.0/0 -> IGW. This fires vpc.add-igw-route + vpc.ungate-subnet-egress, removing the drop policy and installing a reroute policy at SubnetEgressPriorityIGW (1000). |
||
|
joshsiv-mulga
|
7cd054cbf1 | chore(e2e): dump Logical_Router_Policy + Logical_Router_Static_Route in W5 diag | ||
|
joshsiv-mulga
|
41f840ee85 |
fix(setup): grant spinifex-vpcd sudo for ovn-appctl (siv-167)
The W2 inject-garp fix landed in
|
||
|
joshsiv-mulga
|
ba7253e2c6 |
ci(e2e): fix Start Packet Capture EACCES + PID-tracking on tcpdump (siv-167)
Two bugs in
|
||
|
joshsiv-mulga
|
afd63ed967 |
ci(e2e): Phase 1 diag — tcpdump br-wan + bridge fdb + hairpin modes (siv-167)
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB single-host return-path drop class: e2e_single only: - New step 'Start Packet Capture' between Bootstrap and Run E2E Tests. SSHes to test VM and starts background tcpdump on br-wan capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap. PID stored at /tmp/spinifex-diag/tcpdump.pid. All three jobs (single, pseudo-multi, real-multi): - Collect Diagnostics: kill tcpdump (single only — PID file guarded) -INT for graceful flush, falls through to -KILL after 5s. - Adds journal-spinifex-full.log (no -n truncation) alongside the existing -n 10000 capped one. - Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood for every br-wan brif — needed to confirm/disconfirm the hairpin-loss hypothesis in the plan doc. - Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap (created by sudo tcpdump as root) is readable by tf-user. bash -n verified on all four rendered scripts. YAML parses. Zero new gold-image deps (tcpdump, bridge present in base image). |
||
|
joshsiv-mulga
|
2574956f08 | ansible cluster-bootstrap | ||
|
joshsiv-mulga
|
3f4613c28b |
ci(e2e): capture OVN/OVS/journal diagnostics on every run
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:
- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list
Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).
OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.
Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
|
||
|
joshsiv-mulga
|
72a6a59d80 |
fix(network): inject GARP after AddEIP to refresh upstream ARP on EIP recycle
ovn-controller emits gratuitous ARPs automatically when a NAT-bound LSP migrates chassis, but not when an external IP is reassigned to a different LR whose chassisredirect MAC differs from the prior occupant's. In CI single-host runs the WAN pool slice is small enough that the same EIP recycles across VPCs within the 60-300s kernel ARP timeout, so upstream caches keep pointing at the dead chassis-redirect MAC and packets to the new VM never arrive. Fix: add a post-AddEIP hook that runs `ovn-appctl -t ovn-controller inject-garp <port>` (OVN 22.09+). Port is the EIP's LSP in distributed NAT or "cr-gw-<vpcID>" in centralized NAT. Best-effort — failure logs a warning and the NAT install still succeeds. - network/host/garp.go: new L0 shell-out helper InjectGARP. - network/policy/nat.go: GARPEmitter option type, called after the flows-ready barrier in AddEIP. - vpcd: wire the production emitter that derives the right OVN port from the EIPSpec. - invariants: add ovn-appctl to the host-tool ban list so future callers are forced into network/host/. |
||
|
Josh Siviour
|
dd65a2dfa6 |
feat(elbv2): ListenerRules CRUD with HAProxy ACL routing (#275)
* feat(elbv2): ListenerRules CRUD with HAProxy ACL routing Implement CreateRule, ModifyRule, DeleteRule, DescribeRules, and SetRulePriorities for ALB listeners. Rules support host-header, path-pattern, http-header, http-request-method, source-ip, and query-string condition fields. Actions limited to forward (single target group); redirect and fixed-response deferred. HAProxy frontends emit per-rule ACLs with `use_backend ... if <acls>` ordered by priority. Wildcards in host/path map to HAProxy match modes (`-m beg/end/sub/str`). Condition values are validated at store-time and re-validated at render-time to prevent config injection. Target groups referenced by rule actions are protected against deletion. Deleting a listener cascades its rules. Frontend adds a Rules tab beneath each listener row showing rules with conditions and target groups, plus add/edit/delete dialogs with field-typed condition inputs. mulga-950 * test(elbv2): rule CRUD e2e + render coverage for all condition fields Adds Internal_ALB_ListenerRules subtest exercising CreateRule, ModifyRule, DeleteRule, and DescribeRules over a real ALB with HAProxy. Two TGs with disjoint backends (app#0 -> tgA, app#1 -> tgB) let the probe responder identify which TG actually served the request, so path-pattern routing can be asserted without inspecting internal state. Extends app-userdata.sh with alpha/ and beta/ sub-paths and extends client-userdata.sh trigger to accept optional http_path and host fields so probes can target specific paths and Host headers. Both changes are backward-compatible: existing callers default to / with no Host override. Adds unit-test coverage for every condition-field render path (http-header, http-request-method, query-string with and without key, source-ip) plus error-path validation tests for CreateRule, ModifyRule, DeleteRule, DescribeRules, and SetRulePriorities. Lifts diff coverage above the 60% gate. * test(elbv2): wait for tgB healthy after rule creation, not before Spinifex only health-checks a target group once it is referenced by a listener default action or a rule action. The previous ordering waited for tgB healthy before CreateRule wired it, so health checks never ran and the wait timed out. Move the WaitForTargetsHealthy(tgB) call after createPathRule. * fix(elbv2): include rule-action TGs in TargetGroupsForLB healthChecker uses Store.TargetGroupsForLB to map HAProxy server statuses back to TargetGroupRecord entries. The previous implementation walked only listener DefaultActions, so a target group referenced only by a CreateRule action was invisible to the health checker — HAProxy probed its backends but the resulting health never propagated to the store. DescribeTargetHealth reported the targets as 'initial' forever. Extend TargetGroupsForLB to walk rule actions per listener as well. Adds a unit test that wires a TG via CreateRule and asserts the resolver returns it. * test(elbv2): fix wait-for-rule logic to poll away from default backend waitForPathRouted required the caller to name the expected new host, but at the post-CreateRule wait point we only know the default host (appA); the rule target hostname (appB) is discovered by the subsequent probe. The previous code passed appA as both "new" and "old", so the wait checked "see appA" against "/alpha/" responses — which never matched because the rule had successfully routed appA out. Replace with two narrower helpers: waitForPathRoutedAway (poll until host X stops appearing) and waitForPathRoutedTo (poll until host Y appears). Update the three call sites to use the variant that matches what the caller actually knows at that point. * fix(elbv2): IAM-style error envelope + ModifyTargetGroup + listener attr stubs ELBv2 success responses use the IAM-style XML envelope (<ErrorResponse>), but error responses were emitted with the EC2 envelope (<Response>), which aws-sdk-go-v2 cannot deserialize for ELBv2 and surfaces as "UnknownError: UnknownError" — defeating provider-side InvalidAction ignores (e.g. terraform-provider-aws v5.100's DescribeCapacityReservation swallow). Route elasticloadbalancing errors through GenerateIAMErrorResponse. Add ModifyTargetGroup end-to-end (interface, impl, NATS shim, daemon handler, gateway dispatch) so terraform-provider-aws can set health-check fields after CreateTargetGroup without infinite drift. Persist HealthCheckEnabled on TargetGroupRecord and emit it in tgRecordToSDK so the provider sees the configured value across refresh cycles. Stub DescribeListenerAttributes / ModifyListenerAttributes with hand-rolled input/output structs (aws-sdk-go v1.55 lacks these types). Returns empty attribute list — sufficient for default-config workflows. * test(elbv2): cover ModifyTargetGroup + listener attribute stubs Extend handler tests with all-fields, invalid path/matcher, wrong-account ModifyTargetGroup cases plus HealthCheckEnabled assertion on CreateTargetGroup. Add gateway-layer validation tests for the new DescribeListenerAttributes, ModifyListenerAttributes, and ModifyTargetGroup dispatchers. Raises diff coverage from 58.8% to 63.7%. |
||
|
brynmailer-mdc
|
0c577da6e2 |
test: cover encryption wiring on feat/encryption-at-rest
Lifts diff-coverage on PR #276 from 37.5% to 63.5% (threshold 60%) so CI's Check Diff Coverage step passes. - utils: LoadViperblockMasterKey — empty path, happy path + cache reuse, load error, and bad-perm failure-not-cached. - handlers/ec2/image: clusterEncryptionEnabled — nil config, empty key file, masterkey load error (warn + false), and key-loaded (true). - handlers/ec2/volume: - GetVolumeConfig surfaces "failed to unmarshal config" when the body is neither a valid VBState nor a wrapper. - mergeVolumeConfig refuses to re-marshal an encrypted VBState (would strip the AES-GCM tag and brick the volume). - CreateVolume returns ServerInternal when EncryptionKeyFile is set but unreadable. |
||
|
brynmailer-mdc
|
557d3e4d59 |
feat: derive Encrypted from VBState.EncryptionEnabled, drop IsEncrypted reads
VolumeMetadata.IsEncrypted is being removed from viperblock — it was hardcoded false everywhere in spinifex and misrepresented the per-volume encryption status to AWS SDK callers. The authoritative flag is now VBState.EncryptionEnabled, persisted by viperblock at New() when MasterKey != nil. - volume CreateVolume drops the dead IsEncrypted struct member and surfaces Encrypted=(mkey != nil) on the EC2 response. - volume getVolumeByID now decodes the full VBState (falling back to a wrapper-only blob for legacy pre-encryption volumes) so it can read state.EncryptionEnabled directly. - snapshot CreateSnapshot pulls Encrypted from the source volume's VBState.EncryptionEnabled (already loaded one level up). - image synthesizeRootBlockDeviceMapping takes the cluster encryption posture (mkey != nil) since AMI metadata carries no per-image flag; cached masterkey load keeps the hot path cheap. - Test fixtures drop the now-defunct IsEncrypted lines; the volume fixture that asserts Encrypted=true switches to a full VBState seed with EncryptionEnabled=true. |
||
|
brynmailer-mdc
|
b5220996e1 |
fix: only clone AMI on os.ErrNotExist in prepareRootVolume
LoadStateRequest can fail for many reasons — HMAC integrity, key mismatch, transient backend 5xx, JSON parse failure. The previous fall-through treated every error as "volume doesn't exist, clone from AMI", which would overwrite the legitimate encrypted volume's config.json with AMI base-map state on a tamper or transient backend fault. Discriminate os.ErrNotExist explicitly and abort otherwise. |
||
|
brynmailer-mdc
|
ef50392f9e |
feat: wire viperblock at-rest encryption key through daemon and AWS handlers
Adds EncryptionKeyFile to ViperblockConfig (cluster config) and viperblockd.Config. viperblockd loads the master key once at launchService start via masterkey.LoadShared and passes MasterKey + EncryptionEnabled into the VB constructed for every mounted volume. AWS handlers (volume.CreateVolume, instance.newViperblock, image.snapshotStoppedVolume) and the spx admin AMI-import path now resolve the key via utils.LoadViperblockMasterKey — a process-wide cache keyed by path so the per-VB lookup is one map hit after the first load. volume.mergeVolumeConfig: switch from json.Unmarshal to json.Decoder so a trailing AES-GCM tag cannot route the body to the wrapper-only path, and refuse outright when state.EncryptionEnabled — re-marshaling an encrypted VBState here would strip the tag and brick the volume. cmd/spinifex viperblock CLI grows a --encryption-key-file flag (env SPINIFEX_VIPERBLOCK_ENCRYPTION_KEY_FILE) under a distinct viper key so it does not collide with predastore's identically-named flag. |
||
|
Josh Siviour
|
8883984f3f |
feat(eks): eks-server + eks-agent AMI manifests (#273)
* feat(eks): eks-server + eks-agent AMI manifests + builder INSTALL_FILES
Manifest-driven AMI build for the two EKS image variants (eks-v1.md Q12).
Lifted forward from cs-eks-6b so the K3s boot path can be smoke-tested in
isolation before CreateCluster orchestration lands.
Builder:
- build-system-image.sh gains optional INSTALL_FILES (src:dst pairs,
mode 0644, parent dirs auto-created). Symmetric with INSTALL_BINARIES.
- ENABLE_SERVICES moves after INSTALL_FILES + SETUP_SCRIPT so init scripts
the manifest references are on disk before rc-update enables them.
eks-server image (Alpine 3.21.7, q35, 4G):
- K3s v1.32.5+k3s1 (downloaded at bake time, SHA256 verified against the
upstream sha256sum-amd64.txt trust anchor).
- nats-cli v0.4.0 (not in Alpine main/community — pulled from upstream
release zip; unzip installed + purged in same setup step).
- OpenRC init scripts: k3s, eks-token-webhook, k3s-first-boot.
- Cron periodics under /etc/periodic/{15min,daily}/ for state-report +
etcd-snapshot.
- k3s-first-boot publishes node-token + admin kubeconfig over NATS on
first successful K3s healthz, then self-disables (sentinel + rc-update
del).
- config.yaml.skel without webhook-token-auth — wiring it before the real
webhook lands (cs-eks-6c) would block anonymous probes against /healthz.
Orchestrator drops the live config + restarts k3s.
eks-agent image (Alpine 3.21.7, q35, 2G):
- Same pinned K3s release for server/agent lock-step.
- k3s-agent OpenRC service refuses to start without K3S_URL+K3S_TOKEN —
fail-closed pre-orchestration.
eks-token-webhook stub (cmd/eks-token-webhook/main.go):
- 503 + Retry-After:30 on every path. Statically linked. Lets the AMI
build succeed and kube-apiserver boot cleanly. Replaced wholesale by
cs-eks-6c.
Init system divergence from spec wording: Alpine ships OpenRC, not
systemd. Unit files use .initd extensions and cron periodics instead of
.service / .timer. Bead-level intent (auto-enable on boot, periodic
execution) preserved.
Smoke-tested:
- eks-server: builds 300M qcow2; qemu q35 boot reaches K3s apiserver on
6443 within ~120s; HTTPS probes return valid k8s Status JSON (the 401
is K3s v1.32 default RBAC behaviour against anonymous, not a regression).
- eks-agent: builds 280M qcow2; qemu boot mounts root cleanly.
* feat(eks): make targets for eks-server + eks-agent AMI builds
- build-system-image gains subdir-manifest fallback: tries
scripts/images/$(IMAGE).conf first, then scripts/images/$(IMAGE)/manifest.conf
so the new EKS layout works alongside the existing flat lb manifest.
- build-eks-server-image / build-eks-agent-image — explicit convenience
targets that delegate to the generic one.
|
||
|
Josh Siviour
|
785a5655e4 |
feat(sts): EKS Q4 — AssumeRoleWithWebIdentity + presigned GetCallerIdentity verify (#272)
* feat(sts): EKS Q4 STS extensions — interface + JWKS reader + stubs
Lays the structural slice of AssumeRoleWithWebIdentity + presigned-URL
GetCallerIdentity verify for IRSA. The verify-path bodies (JWT signature
check against fetched JWKS, claims validation, Federated-principal trust
policy evaluation, SigV4 presigned-URL canonical-request reconstruction)
land in follow-up commits on this branch.
Adds:
- handlers/sts/oidc_jwks.go — JWK/JWKS types, ParseEKSIssuerURL (defensive
iss parsing), FetchClusterJWKS (KV-backed read from
eks-account-{accountID}/clusters/{name}/oidc-jwks.json via the EKS store
helpers).
- handlers/sts/oidc_jwks_test.go — unit tests over a real JetStream
test fixture.
- handlers/sts/assume_role_with_web_identity.go — interface entry, stub
returns ErrorNotImplemented.
- handlers/sts/presigned_url_verify.go — PresignedCallerIdentity type +
VerifyPresignedGetCallerIdentity stub.
- gateway/sts/AssumeRoleWithWebIdentity.go — dispatch shell.
Modifies:
- handlers/sts/service.go — STSService interface gains
AssumeRoleWithWebIdentity + VerifyPresignedGetCallerIdentity.
- gateway/sts.go — registers AssumeRoleWithWebIdentity in stsActions +
stsSkipPolicyCheck (anonymous; trust-policy-gated in handler).
- gateway/sts_request_test.go, gateway/sts/handler_test.go,
gateway/auth_test.go — stub implementations updated to satisfy
extended STSService interface.
Anonymous routing for AssumeRoleWithWebIdentity (eks-v1.md Q5 mux rework)
lands in Sprint 6e — for now the action is registered on the standard STS
gateway and rejects non-SigV4 requests upstream.
Refs: cs-eks-sts-q4 (mulga monorepo bead)
* feat(sts): AssumeRoleWithWebIdentity + presigned GetCallerIdentity verify
Implements EKS Q4 STS extensions for IRSA + EKS token-webhook flows.
AssumeRoleWithWebIdentity:
- ES256 JWT verification against per-cluster JWKS (eks-account-{accountID}
KV bucket), keyed by issuer host-path.
- Federated trust-policy evaluator (explicit-deny-wins, exact-match
Principal.Federated, fail-closed on Principal: "*").
- Condition support scoped to StringEquals on {iss}:sub (exact) and
{iss}:aud (set-membership). All other operators rejected at validator.
- OIDC provider registry check against iam-account-{roleAccountID}; v1
does not support cross-account registration.
- New awserrors: InvalidIdentityToken, IDPRejectedClaim,
IDPCommunicationError.
VerifyPresignedGetCallerIdentity:
- Hand-rolled SigV4 canonical-request reconstruction (no sdk dep).
- Mandatory X-K8s-Aws-Id in SignedHeaders + value pinned to
expectedClusterName during canonicalisation — cross-cluster replay
surfaces as a signature mismatch.
- AKID-prefix dispatch (ASIA → session credential, AKIA → IAM user).
- Strict envelope checks: algorithm, credential scope service/terminator,
X-Amz-Date / credential-scope date agreement, expiry within
X-Amz-Expires, 5-minute wire skew.
Tests:
- 14 web-identity scenarios (happy paths, tampering, expiry, wrong aud,
unknown kid, HS256 rejection, federated-wildcard fail-closed,
explicit-deny-wins, provider-not-registered).
- 12 presigned-URL scenarios including cross-cluster replay defence,
missing-x-k8s-aws-id, tampered query, bad service in credential, AKIA
+ ASIA happy paths.
- Cross-cluster anti-replay invariant test (rejects matrix of variant
cluster names — case-different, prefix, suffix).
- AWS SigV4 known-vector test for deriveSigningKey.
Gateway:
- TODO documents that AssumeRoleWithWebIdentity is currently dispatched
on the SigV4-gated path; a pre-auth route is needed so a projected
ServiceAccount token alone can authenticate the caller.
IAM:
- Relaxes trust-policy Condition validator: now permitted only when
Action is exactly [sts:AssumeRoleWithWebIdentity] and operator is
StringEquals. Preserves loud failure for sts:AssumeRole.
|
||
|
tomnewton-mulga
|
9d8597317a |
Feat/mig capable (#271)
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit
|
||
|
Josh Siviour
|
e7a25316f0 |
feat(eks): gateway + handler scaffolding for EKS control plane (#267)
* feat(eks): gateway + handler scaffolding for EKS control plane
Lands the EKS service contract — gateway REST-JSON dispatcher, handler
interface + stub impl, JetStream KV bucket layout, and daemon wiring —
ahead of the K3s control-plane bodies.
Every action currently returns NotImplemented (HTTP 501 + JSON
NotImplementedException) so downstream work can call into a stable
contract while real bodies land sprint by sprint.
- Gateway: 34-action REST-JSON routing table under svc=eks, per-action
closures dispatch to handlers_eks.NATSEKSService. ErrorHandler /
writeClusterUnavailable / writeThrottleError gain EKS JSON branches so
clients see application/x-amz-json-1.1 envelopes.
- Handler: EKSService interface + NATS wrapper + stub impl with
var _ EKSService compile checks on both implementations.
- KV: eks-account-{accountID} lazy per-account bucket factory plus the
shared spinifex-eks-leader bucket (TTL 60s, History 1) bootstrapped
at daemon start. Key-path helpers cover the full Q2 layout
(cluster meta, nodegroups, access entries, OIDC providers + signing
key + JWKS, admin kubeconfig, K3s node-token, events).
- Daemon: 34 eks.<Action> subscriptions on the spinifex-workers queue
group, init via initServiceWithRetry, mirrors the ELBv2 layout.
- awserrors: new ErrorNotImplemented constant (HTTP 501).
- tags: ManagedByEKS constant for downstream SG/NLB/VM tagging.
* test(eks): cover scaffolding wrappers + daemon NATS handlers
Adds two test files closing the diff-coverage gap on the EKS
scaffolding commit:
- spinifex/gateway/eks/wrappers_test.go drives every per-action
gateway wrapper (cluster, nodegroup, access, addons, oidc, tags —
34 actions, 50+ subtest invocations) through an embedded NATS
server with a `eks.>` catch-all stub responder. Covers the
wrappers themselves plus handlers/eks/service_nats.go via the
NewNATSEKSService construction inside each wrapper. Bad-JSON
branches in unmarshalIfBody are also covered.
- spinifex/daemon/daemon_handlers_eks_test.go constructs a minimal
Daemon{eksService: real impl} and dispatches each of the 34
handleEKS* handlers via embedded NATS, asserting the NotImplemented
error code rides back through handleNATSRequest.
Lifts diff coverage from 27.3% to 62.6%, above the 60% threshold.
|
||
|
Josh Siviour
|
3415d9f3de |
fix(network): IGW route-table AWS parity + per-subnet egress gating (#270)
* fix(network): IGW default route AWS-parity via per-subnet LR policy
ADR-0006's IGWManager.AttachIGW installed a VPC-router-wide 0.0.0.0/0
static route at attach time, violating AWS semantics: every subnet on
the LR saw egress to the IGW regardless of its route table. Private
subnets that should be IGW-isolated were not.
Move IGW egress lifecycle onto the route-table API. CreateRoute(IGW,
0.0.0.0/0) emits vpc.add-igw-route per associated subnet; the network
subscriber installs an OVN Logical_Router_Policy matching
`inport == "rtr-<subnetID>" && ip4.dst == 0.0.0.0/0` with action
`reroute` via the IGW gateway port. DeleteRoute / DisassociateRouteTable
/ ReplaceRouteTableAssociation tear it down. AssociateRouteTable emits
add events so the terraform ordering (CreateRoute -> Associate) is
honored.
Additions:
- nbdb.LogicalRouterPolicy + Logical_Router_Policy registration
- ovn.Client {Add,Delete,Find,List}LogicalRouterPolicy + live + mock
- policy.RouteManager.{Add,Delete}SubnetEgress with priority constants
- external.IGWManager.{Ensure,Remove}SubnetEgress resolving nexthop +
output port from the gateway state
- subscribers handleAddIGWRoute / handleDeleteIGWRoute on
vpc.{add,delete}-igw-route
- routetable publishIGWRouteEvents / *ForAssociation / *ForSubnet
Test updates:
- network/policy: AddSubnetEgress idempotency, drift-replace, per-subnet
separation, DeleteSubnetEgress
- network/external: AttachIGW must NOT install a router-wide default
route; EnsureSubnetEgress installs the per-subnet policy with the
resolved nexthop (link-local fallback + allocator override paths)
- handlers/ec2/routetable: CreateRoute(IGW), Associate, DeleteRoute,
Disassociate publish events with the expected subnetID + igwID
- ovn/nbdb: table-count + Logical_Router_Policy presence
- subscribers: TestSubscribe_RegistersAllTopics covers the two new topics
* fix(network): NATGW per-subnet egress + DeleteRoute event symmetry
handlers/ec2/routetable:
- DeleteRoute now publishes vpc.delete-nat-gateway per associated subnet
(mirror of the IGW DeleteRoute path) so the subscriber tears down both
the SNAT rule and the per-subnet egress policy.
- publishNatGatewayEvent* signatures carry SubnetId + DestinationCidr so
the subscriber can install the matching Logical_Router_Policy.
network/external:
- IGWManager: EnsureNATGatewaySubnetEgress / RemoveNATGatewaySubnetEgress
install/remove a per-subnet LR policy at priority SubnetEgressPriorityNATGW.
Same nexthop and output port as the IGW priority sibling; lower priority
so an IGW route wins when both apply to the same subnet.
- EnsureSubnetEgress delegates to ensureSubnetEgressAtPriority for shared
validation, lookup, and slog tagging.
network/subscribers:
- handleAddNATGateway: after SNAT install, parse DestinationCidr (defaults
to 0.0.0.0/0) and call EnsureNATGatewaySubnetEgress. Without this the LR
has no route to egress NAT'd packets and private-subnet traffic is
dropped (E2E Phase 8d regression).
- handleDeleteNATGateway: remove the per-subnet policy FIRST, then detach
SNAT, so packets are never routed unmasqueraded after SNAT removal.
- NATGatewayEvent gains SubnetId + DestinationCidr.
Tests:
- policy.AddSubnetEgress validation + drift-replace + propagation paths.
- mock LR-policy methods reject missing routers and tolerate dangling UUIDs.
- subscribers cover NATGW add/delete with invalid CIDR, bad JSON, missing
router, IGW-route bad JSON + invalid CIDR. Full end-to-end add+delete
via NATS asserting policy lifecycle.
- IGW manager covers RemoveSubnetEgress / RemoveNATGatewaySubnetEgress
empty-arg rejection + delete error propagation.
- routetable: DeleteRoute(NATGW) publishes per associated subnet.
* fix(network): exclude VPC CIDR from per-subnet egress reroute predicate
In-VPC traffic (e.g. bastion -> private SSH) was being intercepted by the
NATGW per-subnet egress LR policy on `ip4.dst == 0.0.0.0/0` and forwarded
out the gateway with SNAT, breaking peer-subnet connectivity. Append
`&& ip4.dst != <vpcCIDR>` to the match so in-VPC destinations skip the
reroute and follow the directly-connected route. Apply to both IGW (1000)
and NATGW (900) priorities. CIDR is read from the LR ExternalIDs label
populated at VPC create.
DeleteSubnetEgress takes the same excludes so the match string is
identical on cleanup. igwManager.vpcExcludeCIDRs fetches from the router
each call; lookup failure returns nil and the policy installs without
exclusions (preserves prior behaviour as fallback).
* fix(routetable): publish per-subnet events for implicit main-RT membership
E2E run 26620082197 RCA: subnets with no explicit RT association inherit
the VPC main route table per AWS semantics, but publishIGWRouteEvents and
publishNatGatewayEvents iterated only record.Associations and skipped
Main:true/SubnetId:"" sentinels. Main RT carries exactly one such
sentinel, so CreateRoute(igw)/CreateRoute(natgw) on the main RT fanned
out to ZERO subnets — subscriber never called EnsureSubnetEgress, OVN
LR policy never installed, ping 8.8.8.8 from default-VPC VMs dropped at
the VPC router. Caused TestVPCSubnetE2E, TestNATGateway,
TestMultinodeSpread/NATGatewayInternet to fail.
Add subnetsImplicitlyOnMainRT helper that enumerates VPC subnets
without explicit non-main associations, and fan publish to them when
record.IsMain. Apply to all three publish helpers (IGW, NATGW,
NATGW-delete).
Symmetric fixes for effective-RT changes:
- AssociateRouteTable: subnet leaves implicit main → emit deletes for
main RT's routes before adds for explicit RT
- DisassociateRouteTable: subnet returns to implicit main → emit adds
for main RT's routes after deletes for departing RT
* chore(e2e): capture per-LR lr-route-list/lr-policy-list/lr-nat-list
ovn-nbctl show does not dump static routes or policies. Adds three
per-LR captures so RCA can verify policy/route install state after
the IGW per-subnet refactor.
* chore(e2e): tcpdump br-wan + neigh/fdb/conntrack capture during tests
EIP egress regression on feat/igw-route-table-aws-parity shows SNAT
packets present (n_packets>0) but pings still time out. Post-teardown
ovn.* snapshots can't tell whether replies actually arrived on br-wan
or got dropped at L2 / by upstream router.
start-net-capture.sh: nohup tcpdump on br-wan filtered to 192.168.0.0/23
(EIP pool) + ARP for same range, 200-byte snaplen, 64 MiB cap. Pair:
capture-net-diag.sh stops it via /tmp/spinifex-e2e-tcpdump.pid and copies
wan.pcap into net-diag/.
Also added neigh/bridge-fdb/conntrack dumps to capture-net-diag.sh so
L2/L3 binding state is visible alongside the existing OVN dumps.
Wired both single-VM and multinode workflows to start capture before
the e2e run; teardown step picks up the pcap automatically.
* fix(reconcile): rebuild IGW/NATGW per-subnet egress from KV
Publisher fires vpc.add-igw-route during default-VPC bootstrap before the
subscriber daemon attaches to the topic. NATS core pub/sub drops the
message, leaving the LR without a 0.0.0.0/0 policy and off-subnet egress
broken on the affected VPC for the lifetime of the process. Reproduced
in CI: TestVPCSubnetE2E bastion EIP could ARP and DNS its on-LAN
neighbour but never reach 8.8.8.8.
LoadIntentFromKV now fans every IGW or NAT-gateway RT route out over
its explicit + implicit-main subnets and the reconcile pass invokes
EnsureSubnetEgress / EnsureNATGatewaySubnetEgress for each one. Same
shape as publishIGWRouteEvents so the bootstrap and the runtime event
path produce identical OVN state.
* fix(routes): serialize OVN egress wiring via request-reply
vpc.igw-attach, vpc.add-igw-route and vpc.add-nat-gateway were fire-and-forget
Publish, racing the subscriber's AttachIGW barrier. AttachInternetGateway
returned before vpcd finished wiring the OVN gateway LRP, so a follow-up
CreateRoute(IGW) published an add-igw-route event that the subscriber
processed in a parallel goroutine — installing a Logical_Router_Policy with a
nexthop the LRP didn't yet expose. northd compiled an inert reroute that
never recovered, leaving the bastion's egress traffic blackholed (no packets
dst 8.8.8.8 from the failing VPC's EIP even though local DNS succeeded).
Switch the three add-* publishers to utils.RequestEvent. The subscriber ACKs
once OVN state is committed, so the gateway handler can't issue subsequent
route events against a half-wired router. delete-* paths stay Publish —
teardown ordering doesn't matter.
handleAddNATGateway now calls respond() at every exit so requesters don't
hang. Route-table tests gain an ackSub wrapper (async subscribe + auto-Reply)
so request-based publishers complete during unit tests.
* Revert "fix(routes): serialize OVN egress wiring via request-reply"
This reverts commit bf6f77d5bc8b21f72ed5867175d5a67c010066c8.
* fix(igw): restore default static route in AttachIGW/DetachIGW
OVN 25.03 LR pipeline runs lr_in_ip_routing (table 15) BEFORE lr_in_policy
(table 17). When this branch swapped the per-VPC default static route for
the per-subnet Logical_Router_Policy reroute installed by EnsureSubnetEgress,
external destinations started dropping at routing priority-0 before the
policy table ran. Reproduced locally: ovn-trace stops at table 15 drop, ping
8.8.8.8 from inside a fresh-VPC bastion produced zero packets on br-wan
until `ovn-nbctl lr-route-add <router> 0.0.0.0/0 <wan-gw> <gw-port>` was
issued manually — after which the same ping succeeded immediately. dev still
green because dev calls AddDefaultRoute on AttachIGW.
Restore the original AddDefaultRoute / DeleteDefaultRoute calls; keep the
per-subnet EnsureSubnetEgress machinery intact so future per-subnet egress
gating (DROP policies for subnets whose RT lacks a 0.0.0.0/0 route) can
layer on top. Flip the two TestAttachIGW assertions that pinned the
"AttachIGW must not install a default static route" expectation.
Pairs with revert of bf6f77d5 in the previous commit (request-reply
serialization treated a non-existent race; real bug was missing routing
fallback).
* feat(network): per-subnet egress drop gating for AWS RT parity
VPC LRs carry a router-wide 0.0.0.0/0 default static route while an IGW is
attached (lr_in_ip_routing runs before lr_in_policy on OVN 25.03, so per-
subnet reroute alone is insufficient). Subnets whose effective route table
lacks 0.0.0.0/0 were previously able to egress via that default route,
violating AWS semantics.
Install an LR_Policy DROP at SubnetEgressPriorityDrop=1100 (above IGW=1000
and NATGW=900) for unrouted subnets. Match excludes VPC CIDR, link-local
(169.254/16) and multicast (224/4) so intra-VPC, DHCP/metadata and
multicast still flow.
Wiring:
- policy.RouteManager gains AddSubnetEgressDrop / DeleteSubnetEgressDrop.
- external.IGWManager gains EnsureSubnetEgressDrop / RemoveSubnetEgressDrop.
- New events: SubnetEgressGateEvent / SubnetEgressUngateEvent on topics
vpc.gate-subnet-egress / vpc.ungate-subnet-egress.
- routetable handler publishes gate/ungate alongside existing add/delete
IGW/NATGW route events on every CreateRoute/DeleteRoute/Associate/
Disassociate/Replace mutation, based on subnet's recomputed effective RT.
- reconciler bootstrap path emits DropGates intent for subnets in IGW-
attached VPCs whose effective RT lacks 0.0.0.0/0.
Tests cover primitive idempotency, drift recovery, event publishing per
mutation surface, implicit-main fan-out, and intent computation.
* fix(igw): fan out subnet egress gate decisions on IGW attach/detach
The reconciler computes per-subnet DROP policies for any subnet in an
IGW-attached VPC whose effective route table lacks 0.0.0.0/0, but those
decisions were only re-evaluated by the periodic drift tick (every
5 min). AttachInternetGateway / DetachInternetGateway now publish
gate/ungate events for every subnet in the VPC immediately, closing
the window where a freshly-attached IGW could leave unrouted subnets
unprotected.
Wiring: IGWServiceImpl gains a nil-safe GatePublisher hook satisfied
by RouteTableServiceImpl. Daemon installs the hook once both services
exist. RouteTableServiceImpl.PublishGateDecisionsForVPC enumerates the
VPC's subnets and reuses publishSubnetEgressGateDecision per subnet.
Also fixes the LB e2e suite, which built VPC + IGW + subnet without
ever creating a 0.0.0.0/0 → IGW route — relying on the pre-DROP-gate
behaviour where unrouted subnets still had silent egress via the
router-wide static route. The fixture now installs the explicit
public route on the main RT (AWS-faithful semantics for a public
subnet).
* fix(igw): skip gate fan-out on attach to avoid bootstrap race
AttachInternetGateway fan-out races with the default-VPC bootstrap path
in ensureDefaultVPCInfrastructureFor, which attaches the IGW and
immediately publishes a 0.0.0.0/0 → IGW CreateRoute event. The two
events land on distinct NATS subjects (vpc.gate-subnet-egress vs
vpc.ungate-subnet-egress) with no cross-subject ordering guarantee.
When ungate arrived first the subscriber became a no-op and the
subsequent gate installed a permanent DROP on the default subnet,
killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to the instance public IP).
Detach fan-out is retained — no follow-up CreateRoute fires after
DetachInternetGateway so the path is race-free.
Subnets without a default route still get DROPped during the
reconciler's periodic drift pass (loadSubnetDropGates). The window is
re-opened to ≤5 min on user-driven AttachInternetGateway, which is
the original behaviour pre-mulga-siv-163. A follow-up bead can close
that gap properly by converting the gate/ungate subjects into a
single recompute trigger that re-reads effective RT state.
* fix(vpc): prevent duplicate main route table from bootstrap race
Concurrent multi-node daemons running EnsureDefaultVPC for the admin
account (bootstrap-pinned VpcId) each minted a fresh random rtb-ID and
Put it with IsMain=true. The KV ended up with two main route table
records for the same VPC.
CreateRoute's post-write gate recompute (per-subnet egress drop gating)
then resolved effectiveRouteTable non-deterministically across the two
IsMain=true records; when the orphan (route-less) RT won, the gate
decision became "no default egress" and vpc.gate-subnet-egress fired.
The subscriber installed a Priority-1100 DROP on the default subnet's
egress, killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to 192.168.0.161 in run
26699045536).
Two-part fix:
1. createMainRouteTable scans rtbKV for an existing IsMain=true record
on the same VPC before Put-ing a new one. Race window narrows from
"always duplicates under concurrent bootstrap" to a sub-ms TOCTOU
between scan and Put.
2. effectiveRouteTable / mainRouteTable use a deterministic preferMain
tiebreak when multiple IsMain=true records exist: more routes >
oldest CreatedAt > smaller RouteTableId. Defensive — any duplicate
that slips through (cross-AZ restore, manual KV edit, residual race)
still produces consistent egress gate decisions instead of flapping
on KV scan order.
* fix(policy): atomic ACL replacement for SG mutations
Add ovn.Client.ReplaceACLs — detach + delete existing ACL rows and
attach freshly-created rows in one OVSDB transaction. EnsureSG and
UpdateSG now funnel through ReplaceACLs instead of ClearACLs +
AddACLs, closing the mid-flight window where the port group held
zero ACLs and tenant LSP traffic defaulted to drop (observable on
ICMP and on TCP SYNs that miss conntrack).
Mock client exposes ClearACLCalls / ReplaceACLCalls counters so
tests can assert atomicity. New TestSGManager_UpdateSG_AtomicReplace
fails if SG mutations ever fall back to the split path.
|
||
|
Josh Siviour
|
287487a4f2 |
feat(elbv2): implement ModifyListener (#268)
* feat(elbv2): implement ModifyListener
Adds AWS-compatible ModifyListener to the spinifex ELBv2 stack so
listeners can be edited in place — port, protocol, and default
actions — without the current delete+create dance that briefly drops
the listener from HAProxy.
- Handler interface, impl, and NATS wrapper get ModifyListener;
validation mirrors CreateListener (LB type ↔ protocol, dup-port
excluding self, target-group existence + protocol compatibility).
- Daemon subscribes to elbv2.ModifyListener on the spinifex-workers
queue and a new gateway action dispatches it.
- Frontend exposes a per-row edit button that pre-fills a dialog from
the existing listener and calls ModifyListenerCommand, replacing
the "Listeners cannot be edited" note.
- Unit + NATS round-trip + frontend tests cover the new flow.
Cert / SslPolicy / Alpn handling stays out of scope and lands with
HTTPS listener support.
* test(e2e): cover ModifyListener in lb harness
Adds an Internal_ALB_ModifyListener subtest to TestLoadBalancer that
exercises in-place listener edits end-to-end:
- Provisions tgA + tgB sharing the same app instances.
- Brings up an internal ALB with HTTP:80 → tgA, baselines round-robin.
- ModifyListener port=8090, asserts DescribeListeners shows the new
port and that probes on 8090 still round-robin — proves HAProxy is
reconciled without dropping the listener (the regression mulga-944
fixes; the frontend previously delete+created on every edit).
- ModifyListener DefaultActions → tgB, asserts traffic still flows.
Also factors a probeAtPort helper so the order JSON can target any
listener port, and updates TEST_COVERAGE.md with phase 3e.
* fix(ui): widen listener add/edit dialog so content stays inside the box
AlertDialogContent defaults to max-w-xs (20rem) which clipped the listener
form on the right edge. Use a 32rem cap with grid-cols-[minmax(0,1fr)] +
min-w-0 on the form so CSS grid tracks shrink instead of expanding to
max-content.
* test: cover ModifyListener + system instance dispatch
Lifts ModifyListener diff coverage by exercising nil/empty arn paths,
NLB protocol acceptance + rejection, protocol-only revalidation against
existing target groups, and the launcher-nats wire format error paths
(missing input, missing instance type, decode failures, no responder,
default timeout fallback).
Adds a new spinifex/daemon/daemon_system_dispatch_test.go covering
splitSubjectTail edge cases, respondWithSystem{Launch,Terminate}{Output,
OK,Error} envelope shape, handler entry points for invalid JSON and
missing instance ID, and subscribeSystemTerminate nil-conn + idempotent
re-subscribe behaviour.
Fixes the flaky TestHandleEC2DescribeInstanceTypes/NoFilter… subtest by
selecting a 2-vCPU instance type that also fits schedulable memory, so
hosts with smaller schedulableMem (CI 4 vCPU / ~13 GiB) don't pick a
16 GiB-mem type that fails allocation.
* test(e2e): wait for tgB after listener swap, not before
Spinifex health-checks only target groups wired into a listener's
HAProxy config. Waiting for tgB to report healthy before
modifyListenerDefaultTG attaches it to the listener is impossible —
the agent never probes its targets, so they stay in initial and the
2m EventuallyErr trips.
Drop the pre-swap tgB wait. After modifyListenerDefaultTG, replace
the fixed 3s sleep with WaitForTargetsHealthy on tgB (post-swap),
which is now the listener's DefaultAction.
* test(e2e): derive LB name from ARN in lbENI/deleteLB ENI filter
The ENI description filter was building the LB-name token from a
hard-coded "alb"/"nlb" template (e.g. `ELB app/lb-e2e-alb-int/<id>`),
which only matched LBs whose name happened to be `lb-e2e-alb-*` or
`lb-e2e-nlb-*`. The ModifyListener suite names its LB `lb-e2e-mod`,
so lbENI's EventuallyErr could never find the ENI and tripped after
30s. deleteLB had the same bug: sysInstanceID stayed empty and
WaitForENICleanup waited for a non-existent filter, allowing the
next suite's createLB to race the sys.micro VM reclaim.
Parse the LB name straight out of lb.ARN (penultimate path segment,
same shape createLB already relies on) so the filter matches any
caller-chosen name. Drop the now-redundant scheme arg from lbENI.
* test(e2e): poll for listener serving on new port after ModifyListener
The post-modifyListenerPort settle was a hard 3s sleep, but lb-agent
polls for new HAProxy config on a 5s tick. After the API returns, the
agent has typically not yet detected the config-hash change, so
HAProxy is still bound to the old port — every probe to the new port
RSTs and probeAtPort sees 0/0 successful.
Replace the sleep with waitForListenerServing: a polling loop that
fires 2 probes via the shared trigger client every 2s until at least
one returns a parsable instance_id, with a 60s ceiling. Factor the
trigger-and-fetch out of probeAtPort into probeOnce so the polling
helper can reuse it without the require.* asserts.
|
||
|
Josh Siviour
|
c1f44b36da |
feat: vendor-agnostic instance types + sys.* NATS fan-out (#269)
* feat(instancetypes): emit cross-vendor sibling families on x86_64 hosts
Each x86_64 cpuGeneration's family list is now extended at generate time
with its cross-vendor counterpart family (Intel t3 ↔ AMD t3a,
c5 ↔ c5a, m6i ↔ m6a, etc.) via the new vendorSiblingFamily map in
definitions.go. Sibling families share an identical vCPU/memory shape
per size (same instanceFamilyDef size tables) and the same x86_64 arch
tag, so any host can schedule either vendor's request without changes
to canAllocate or the subscription path.
ResourceManager.updateInstanceSubscriptions iterates the type catalog
unchanged; it now picks up both vendors automatically and subscribes to
ec2.RunInstances.t3.* AND ec2.RunInstances.t3a.* on the same node.
RunInstances for either family load-balances across mixed Intel + AMD
clusters via the existing spinifex-workers queue group.
ARM (Graviton t4g/c?g/m?g/r?g) and legacy Intel-only families (t2, c4,
m4, r4) have no cross-vendor sibling and are unaffected.
Bead: mulga-siv-157 (vendor aliasing portion). sys.* NATS fan-out for
ALB-VMs is a separate change in the same plan.
* feat(daemon,elbv2): NATS fan-out for system instance launch/terminate
ELBv2 ALB-VMs (sys.micro) previously launched via a direct Go method call
from ELBv2ServiceImpl into the local Daemon, so every ALB-VM piled on
whichever node handled the upstream CreateLoadBalancer call. Mixed
clusters tripped 'insufficient capacity for sys.micro' once that node's
host-reserve trimmed schedulable vCPU below the next launch.
Route system-instance lifecycle through a dedicated NATS subject root:
system.LaunchInstance.{type} (queue group spinifex-workers)
system.LaunchInstance.{type}.{nodeID} (targeted)
system.TerminateInstance.{instanceID} (owning-node only)
ResourceManager.updateInstanceSubscriptions now treats sys.* types under
the new subject root with a separate handler hook (systemHandler) and
the same capacity-aware subscribe/unsubscribe behaviour customer types
already have — sys.micro subscriptions drop when the host saturates,
naturally re-distributing load.
The daemon side keeps LaunchSystemInstance / TerminateSystemInstance as
local methods; the new handleSystemLaunchInstance / handleSystemTerminate
handlers decode the JSON envelope, invoke the local method, and reply
with output or error. The terminate subscription is bound on launch
success so future terminate requests reach the owning node, and dropped
on terminate success.
ELBv2ServiceImpl now consumes a NATSSystemInstanceLauncher (handlers/elbv2/
system_instance_launcher_nats.go) instead of a direct daemon pointer, so
any node in the cluster can carry an ALB-VM.
SystemInstanceInput / Output / NICConfig / ExtraENIInput get explicit
JSON tags because the wire crosses NATS.
Tests cover JSON round-trip, remote-error propagation, terminate routing,
and no-responder failure mode.
Bead: mulga-siv-157 (sys.* fan-out portion completes the bead).
|
||
|
Julian Sommer
|
fbd5e3a607 | fix: reconciler emits configured dns_server instead of hardcoded 8.8.8.8 (#265) | ||
|
Julian Sommer
|
10728391af | chore: revert test-cover Makefile | ||
|
Julian Sommer
|
e4b7834b2b | chore: move input validation into gateway (#264) | ||
|
Julian Sommer
|
94aa482931 | feat: sts v1 (#261) | ||
|
Josh Siviour
|
65ba60e8bc |
fix(gateway/ec2): fan out DescribeInstanceAttribute across daemons (#263)
* fix(gateway/ec2): fan out DescribeInstanceAttribute across daemons
DescribeInstanceAttribute was queue-grouped on `spinifex-workers`, so each
request landed on one daemon at random. The handler only consults per-daemon
state (vmMgr / stoppedStore), so 1/N requests succeeded and the rest returned
InvalidInstanceID.NotFound. Terraform's AWS provider refreshes aws_instance
with three attribute reads per cycle, which on a 5-node cluster fails ~99% of
the time.
Move the daemon subscription into the fan-out block (no queue group) and
rewrite the gateway to mirror DescribeInstances: inbox + N replies + drop
NotFound. The aggregator returns the first parsed success, falls back to
NotFound only when every daemon confirms absence, and propagates deterministic
client errors (InvalidParameterValue, MissingParameter) when no daemon
succeeds.
Tests:
- New: MultipleNodes_OwnerWins, AllNodesNotFound, ClientErrorPropagates,
NoResponders, SingleNode (rewritten without queue group).
- daemon_handlers_test.go: drop the `spinifex-workers` queue-group string from
the ten DescribeInstanceAttribute subscriptions so the tests match the new
production wiring.
* fix(ec2): DescribeInstanceTypes returns supported set, not capacity-gated
The no-filter DescribeInstanceTypes response was filtered to types with
at least one free slot. AWS-compatible callers (e.g. the Terraform AWS
provider's post-RunInstances metadata lookup) expect a stable supported
set: when a launch consumes the last free slot of its type, the next
DescribeInstanceTypes for that type must still return it, or the apply
fails with "reading EC2 Instance Type (X): empty result".
Split the API into two ResourceManager methods so capacity semantics
stay strict for the capacity=true filter (used for cluster aggregation)
while default callers get the supported answer:
- GetAvailableInstanceTypeInfos(showCapacity bool) — unchanged; gates on
remaining schedulable CPU/memory and (for GPU types) free GPU pool
slots. Used when capacity=true is requested.
- GetSupportedInstanceTypeInfos() — new; lists every loaded instance
type once, regardless of current allocation or GPU pool state. Still
skips system types ("sys.*") and entries with incomplete CPU/memory
metadata.
The handler picks between the two based on the capacity filter. The
gateway aggregator (DescribeInstanceTypes.go) needs no change — it
already deduplicates by type name when capacity=false, so requesting a
specific name now resolves from whichever node supports it.
Tests:
- Daemon unit tests for the new method cover the three filter rules
(no free slots, system/incomplete skip, GPU without manager).
- Handler tests assert the supported path is hit by default and the
capacity-gated path is hit only when capacity=true.
- The NATS-driven integration test is split into two subtests: one
proves the no-filter response is invariant under allocation, the
other proves capacity=true still gates on remaining resources.
* fix: persist API device name in volume metadata on AttachVolume
aws_volume_attachment polls DescribeVolumes with
attachment.device=/dev/sd[f-p], the API-form name. Spinifex was
persisting the in-guest virtio path (/dev/vd*) via UpdateVolumeState
after AttachVolume, so the filter rejected every attached volume and
the post-attach wait loop failed with "couldn't find resource".
UpdateVolumeState now receives the API-form device name and the
daemon's AttachVolume response echoes DeviceName instead of
GuestDevice. BlockDeviceMappings continues to carry the guest path
(mulga-599 / PR #55) since UpdateGuestDeviceNames re-applies that
convention on every Launch/Start; only the volume-metadata path
diverges to the API form for the AWS-spec round-trip.
Adds fakeVolumeStateUpdater to manager_deps_test.go and a regression
test that asserts the API-form name flows through
UpdateVolumeState while BDM keeps the guest path.
Resolves mulga-siv-154.
|
||
|
Josh Siviour
|
f877f5edef |
feat(network/external): restore source=dhcp on external pools (#262)
* refactor(network/external): introduce Allocator interface + StaticPoolAllocator
Carve external IP allocation out of handlers/ec2/vpc/external_ipam.go into
network/external/static_pool.go behind a typed Allocator interface. The KV
bucket name, schema, CAS retry count, and gateway-IP reservation semantics
are unchanged so existing static-pool data carries over with no manual
migration.
ExternalIPAM stays in handlers/ec2/vpc (L5 cleanup deferred per plan) but
becomes a thin facade that constructs a StaticPoolAllocator and delegates
Allocate / Release / GetPoolRecord through the new interface. Q4 will plug
DHCPPoolAllocator into the same Allocator slot.
Plan: docs/development/improvements/external-pool-dhcp-source.md (Q1).
* feat(network/external/dhcp): package skeleton — Client + Fake + KV store
New network/external/dhcp/ package, restored from the structure deleted
in Slice C (commit
|
||
|
Julian Sommer
|
621e49405e | fix: UI bug fixes | ||
|
tomnewton-mulga
|
88881ba04c | fix: Updated Trinity and SMCI docs | ||
|
Julian Sommer
|
7ff02a6020 | fix: consistent UI capitlisation |