mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
a4a4288d7d37eeafeba9052d0a9d2ed1fdcb7813
1275 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
a4a4288d7d |
feat(ecr): auth bridge + GetAuthorizationToken JWT (Sprint 2c) (#380)
* feat(ecr): auth bridge + GetAuthorizationToken JWT (Sprint 2c)
Mint a self-contained ES256 JWT from GetAuthorizationToken and verify it on
the /v2 OCI registry via a new ECRAuthBridgeMiddleware (Bearer | Basic AWS:jwt).
Signing keys live in the cluster-replicated awsgw-keys KV bucket under
jwt-signing/{kid}, encrypted with the IAM master key.
The registry is now scoped per-request to the token's account (replacing the
fixed bootstrap account); bucket-ensure is cached per account. Cross-account
token/host mismatch is rejected 403; unauthenticated /v2 access gets a 401
Bearer challenge.
* fix(ecr): use Basic auth challenge on /v2 (not Bearer token-endpoint flow)
ECR authenticates the OCI registry with Basic AWS:<token>; a Bearer realm
made docker chase an unimplemented /v2/token endpoint and fail login.
|
||
|
Julian Sommer
|
3f592b4a5e | fix: alpine image ssh (#384) | ||
|
Julian Sommer
|
038927d18e | fix: stop NAT Gateway teardown leaking SNAT rule (#383) | ||
|
Julian Sommer
|
83e538f2f2 | feat: IMDS version discovery, instance-identity document, and cheap metadata routes (#382) | ||
|
Josh Siviour
|
3e6dfde7dd |
feat(ui): inline EKS cluster IAM role creation in wizard (#379)
* feat(ui): inline EKS cluster IAM role creation in wizard * feat(ui): show trust policy + attached managed policy in EKS role dialog Render the trust policy with PolicyDocumentViewer (pretty-printed, safe fallback) instead of a raw read-only textarea, matching the IAM detail pages. Surface the attached AmazonEKSClusterPolicy ARN so users can see exactly which permissions the cluster role grants. |
||
|
Josh Siviour
|
833456ae09 | fix(ui): EKS image banner shows spx admin images import command (#378) | ||
|
Julian Sommer
|
941978a28c | refactor: consolidate NATS fan-out onto a single utils.Gather (#381) | ||
|
tomnewton-mulga
|
b0810bd5a1 | Merge branch 'dev' of github.com:mulgadc/spinifex into dev | ||
|
tomnewton-mulga
|
116a80f0ca | fixed up SMCI MIG doc | ||
|
Josh Siviour
|
2acb37342d |
feat(eks): aws-load-balancer-controller addon — real ALB Ingress (L7, IRSA) (#376)
* fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution
DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer
Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ
info ("cannot resolve AZ info for AZs: []") and never built the ALB
model. Stamp the single zone-id on every subnet and share it with
DescribeAvailabilityZones through one SingleZoneID constant so the two
responses can never drift.
* feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle
Baked manifests for the LBC managed addon (CRDs, controller Deployment/
RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two
wirings make it work against the spinifex gateway:
- --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/
acm. The aws-sdk-go v1 credential bootstrap resolves the STS client
from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the
web-identity AssumeRoleWithWebIdentity must be pointed at the gateway
here or it falls back to real AWS STS.
- --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet
count from two to one so an ALB provisions in a single-AZ deployment.
* feat(eks): render IRSA into managed addons + node-minted webhook cert
Extend the addon-sync renderer to inject the projected-token / role-arn /
gateway STS-endpoint + CA block into managed-addon pod specs via opt-in
markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA
placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl,
so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the
admission webhook, cached on disk so the render is byte-stable across
ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES
for the LBC bundle.
* feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller
Thread the control-plane VPC id through K3sServerInput.VpcID into the
first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller
can pass --aws-vpc-id to the gateway elbv2/ec2 handlers.
* fix(eks): point the LB controller at the customer VPC + customer-facing gateway
The aws-load-balancer-controller ran with two control-plane-scoped values
that customer worker pods cannot use:
- --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID),
which the gateway's DescribeVpcs does not return for customer calls, so
backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from
meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers
and their subnets live and where the ALB is provisioned.
- The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL /
SystemGatewayURL) into pod specs, unreachable from worker pods. Add a
distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same
host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in
managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own
bootstrap, state-report and token-webhook paths.
* fix(elbv2): resolve forward target group from ForwardConfig
listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a
forward action carrying its target group via ForwardConfig.TargetGroups (the
shape the aws-load-balancer-controller emits) left the action target empty and
CreateRule/CreateListener rejected it with MissingParameter. Flatten the first
target group from ForwardConfig when the flat field is absent.
* fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records
The generic S3 tag store and the per-resource KV records are disjoint:
DescribeSubnets/DescribeVpcs read tags from their KV record, while
CreateTags only wrote the generic store. Subnet tags applied after
create were therefore invisible to tag-filtered describes, breaking
aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb).
Write through CreateTags/DeleteTags to the owning subnet/vpc record so
tag-filtered describes observe them. Resource types the VPC service does
not own are skipped; the generic store stays their record of truth.
* test(daemon): pin synthetic capacity in host-dependent allocate tests
TestCanAllocate_CountEdgeCases/Mixed_instance_types and
TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation
called rm.allocate against live-detected host capacity. allocate gates on
readMemAvailableGB (live /proc MemAvailable), so a box already running VMs
fails the allocation spuriously, breaking the preflight gate locally.
Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic
host, matching the sibling Capacity_decreases_after_allocation subtest.
* fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster)
In the managed-CP topology the apiserver VM sits in the system CP VPC with
no route to the worker pod network in the unpeered customer VPC. k3s default
egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl
logs/exec work but apiserver->pod traffic (admission webhooks, aggregated
APIs) is attempted directly and times out -- the aws-load-balancer-controller
webhooks fail with context deadline exceeded, blocking Service/Ingress
creation cluster-wide.
Set egress-selector-mode=cluster so apiserver->pod/service traffic routes
through the agent's outbound tunnel, selected by the node hosting the
destination, without requiring inbound CP->worker reachability.
* feat: bump aws-load-balancer-controller addon bundle to v2.11.0
v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex
clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2)
and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true.
Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings),
controller manifest, and image tag v2.11.0, re-applying the spinifex deltas
(IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA
secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig,
alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1.
* fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2)
LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK
service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase
names the bundle passed. Those overrides were silently ignored, so every
non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the
gateway CA -- no ALB could provision.
Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global
base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for
STS) and drop the dead --aws-api-endpoints flag from the controller args.
* fix: synthesize taggable ARN for default listener rule
Gateway DescribeRules returned the synthetic IsDefault rule with a null
RuleArn. LBC read the arn-less default rule after creating a listener and
issued DescribeTags with an empty resource ARN, which the gateway rejected
with 400 MissingParameter, aborting reconcile.
Set a deterministic RuleArn on the default rule, resolve it back to its
parent listener in DescribeTags (empty tags) and DescribeRules by-arn.
Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls.
* feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB
LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a
single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts
Ingress reconcile. Only the explicit-subnet path honors the gate.
Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface
them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb
IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path.
An empty set falls back to auto-discovery, preserving the native UX.
* fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams
Live verification showed two gaps in the LBC addon bundle:
LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL
is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS).
Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs
(EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase
names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the
web-identity STS client still resolves via AWS_ENDPOINT_URL_STS.
The injected IngressClassParams was orphaned: the alb IngressClass had no
.spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a
single-AZ cluster). Link the IngressClass to the IngressClassParams so the
explicit-subnet path is taken.
* fix: stamp EKS-parity providerID on worker nodes for LBC ip targets
k3s registers workers with providerID k3s://<name>, which the AWS Load
Balancer Controller rejects (providerID invalid for EC2 instances) when
resolving nodes for ip-mode target registration. Resolve the instance-id and
AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id>
so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps
the default providerID and still starts.
* fix(elbv2): honor SubnetMappings in CreateLoadBalancer
LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list.
createLoadBalancer only read input.Subnets, so ENI allocation was skipped,
VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the
ALB went active but could not serve traffic. Add flattenSubnetIDs to read
both sources and reuse it in createLoadBalancer and desiredSubnetSet.
* fix(vm): capture GCInterval before spawning GC goroutine
TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global
GCInterval and restores it in a defer, racing the Start goroutine's read at
time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's
goroutine into a local before spawning so the global access is ordered with
the test's defer; the goroutine touches only the captured local.
|
||
|
Josh Siviour
|
332087fccc |
feat: OCI Distribution v2 blob + manifest storage (#375)
* feat: implement OCI Distribution v2 blob and manifest storage Replace the /v2/* 501 stub with a real path-parsing dispatcher and content-addressable storage for the ECR registry surface: - Blob upload (monolithic + chunked PATCH), HEAD/GET, cross-repo mount, and cancel, with server-side sha256 verification on finalize. - Manifest PUT with type-aware shallow validation (image manifest blob existence; image index child-manifest existence + mediaType match), byte-for-byte GET with stored Content-Type, HEAD, and tag delete. - Tag list and account-scoped catalog. Blob and manifest bytes stream straight to predastore; repo, tag, manifest and in-progress-upload metadata live in a per-account JetStream KV bucket with optimistic-concurrency CAS on uploads. Chunks are buffered to a temp upload object and reassembled on finalize, since predastore exposes no append primitive through the shared object-store interface. Adds HeadObject to the objectstore abstraction. Refs: mulgadc/mulga#73 * refactor(ecr): route registry metadata through NATS daemon handlers The gateway no longer talks JetStream KV directly. The daemon owns the per-account KV via a MetaService bound to ecr.* request/reply subjects; the gateway is a NATS client (NATSMetaStore). Blob and manifest bytes still stream gateway-to-predastore directly. Also fixes four registry review findings: - finishUpload: clean up the temp object and upload record on a blob store failure so the uuid is reusable and no partial blob lingers. - patchUpload: commit the appended bytes and advanced hash state under a single KV CAS keyed per write, so a racing PATCH is rejected (409) rather than corrupting the assembled bytes; the verified digest always matches the stored bytes. - cancelUpload: delete both the temp bytes and the record, 204 on success, 404 on unknown uuid, no dead branch. - getManifest: return MANIFEST_INVALID (not MANIFEST_UNKNOWN) for a 406 Accept media-type mismatch. Refs: mulgadc/mulga#73 * fix(ecr): provision account predastore bucket on demand OCI blob/manifest writes target ecr-{accountID}, which nothing created, so every push failed with a backend 403. Add idempotent EnsureBucket to the object store (HeadBucket -> CreateBucket, swallowing already-owned) and ensure it once per process at the top of the registry ServeHTTP, so ECR storage appears transparently like AWS-managed ECR. * fix(ecr): sanitize ':' from predastore blob/manifest object keys predastore mis-canonicalizes SigV4 for object keys containing ':', so writes/reads of content-addressable keys (sha256:<hex>) were rejected 403 (reproducible with the aws CLI too). Route BlobKey and ManifestKey digests through DigestToken so the object key carries no ':' separator. |
||
|
tomnewton-mulga
|
aa46943e0a | fixed up SMCI MIG doc | ||
|
Julian Sommer
|
996eb370f3 | feat: Capacity Reservations (#377) | ||
|
Julian Sommer
|
28c6e8ccff | chore: bump CI time | ||
|
Josh Siviour
|
167c93b1c7 |
feat: ECR TLS SAN wildcard + hostmatch host-routing middleware (#374)
Implement the registry host-routing middleware so /v2/* requests addressed
at {accountID}.dkr.ecr.{region}.{suffix} carry the parsed target account and
region on the request context; non-registry hosts pass through unchanged.
Add cluster-wide AWS-parity config (aws.region, aws.internal_suffix) with
defaults, and append the AWS-parity ECR SANs (exact control-plane host plus
*.dkr wildcard) to the awsgw server certificate during cert generation.
Refs: mulgadc/mulga#78
|
||
|
Josh Siviour
|
a1c307a128 |
feat: scaffold ECR control-plane + OCI registry gateway (501 stubs) (#373)
* feat: scaffold ECR control-plane + OCI registry gateway with 501 stubs Add the ECR HTTP surface on awsgw as compiling skeletons that return NotImplemented, so later work can call into the packages against a fixed contract. - gateway/ecrapi: ECR control plane (AWS JSON 1.1, X-Amz-Target). Full action namespace registered; every action resolves to a shared 501 stub. - gateway/ecr: OCI Distribution v2 (/v2/*). Catch-all 501 with the OCI error envelope; GET /v2/ is live for the registry version-check probe. - gateway/hostmatch: no-op host-routing middleware skeleton for the registry host group. - handlers/ecr/storage: per-account bucket + object-key layout constants. - gateway wiring: ecr in supportedServices, dispatch case, JSON 1.1 error branch, and /v2 mounted outside the SigV4 group. * test: cover ECR scaffolding packages for diff-coverage gate In-package tests for gateway_ecr, gateway_ecrapi, and handlers/ecr so the new code reports coverage under its own package profile. Diff coverage 76.5%. |
||
|
tomnewton-mulga
|
42db84184d | Merge branch 'dev' of github.com:mulgadc/spinifex into dev | ||
|
tomnewton-mulga
|
a396e1bd30 | updating SMCI docs | ||
|
Julian Sommer
|
d88783d5bf | feat: ui json formatting | ||
|
Julian Sommer
|
eced3f733a | feat: split IAM e2e (#372) | ||
|
Julian Sommer
|
b99b46c18c | chore: consistent ui malformed json rejection | ||
|
Julian Sommer
|
9288ecc7dc | chore: delete no op tests + speedup e2e | ||
|
Julian Sommer
|
5c0c9ed173 | chore: fix favicon | ||
|
Julian Sommer
|
058da3eff4 | fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) | ||
|
Julian Sommer
|
89ec47a69b | chore: update COMMANDS.md | ||
|
Julian Sommer
|
2e9501524b | feat(e2e): more e2e coverage (#365) | ||
|
Josh Siviour
|
67092d40be |
fix(workbooks): split EKS demo workloads into separate root module (#370)
The kubernetes provider in the eks-quickstart, eks-addons-access and eks-https-ingress workbooks was configured from the managed cluster resource (host = aws_eks_cluster.this.endpoint). On destroy the endpoint resolved empty, the provider fell back to localhost:80, and refreshing kubernetes_deployment_v1.hello / kubernetes_service_v1.hello failed with 'dial tcp [::1]:80: connect: connection refused'. Move the kubernetes provider and the hello Deployment + Service into a nested workloads/ root module with its own state. It reads cluster identity from the parent via terraform_remote_state and the live endpoint/CA from a data aws_eks_cluster source, so the provider is only ever configured while the cluster exists. Apply parent then workloads; destroy workloads then parent. |
||
|
Josh Siviour
|
da0f10015c |
fix(eks): stop leaking cluster SGs that pin the customer VPC on teardown (#369)
purgeClusterInfra stranded the cluster security groups two ways, leaving
the nodegroup/control-plane SGs (which live in the customer VPC) behind to
pin aws_vpc destroy on DependencyViolation:
* an early return before the SG cleanup block skipped DeleteClusterSGs
whenever any earlier best-effort step (NLB / ENI / VM / EIP) errored;
* DeleteClusterSGs failures were warn-and-swallowed, so a nodegroup SG
still referenced by a terminating worker's ENI leaked while the cluster
completed deletion — no reaper retry ever fired.
Surface DeleteClusterSGs / DeleteClusterCPVPC / DeleteClusterIGW failures
into teardownErrs and gate the KV sweep on them, so a leaked VPC-pinning SG
keeps the cluster DELETING for a retry. Add a bounded DependencyViolation
retry to the SG delete (mirroring deleteENIAwaitingTerminate) to ride out
the worker-ENI-detach race in the common case.
Fixes the manual 'revoke rules + delete SGs before the VPC will destroy'
teardown step.
|
||
|
Josh Siviour
|
bc9556d1a9 |
fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368)
* fix: persist instance TagSpecifications in RunInstance RunInstance built the ec2.Instance metadata but never read input.TagSpecifications, so DescribeInstances returned untagged instances and tag filters matched nothing. EKS node groups discover their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup tags, so scale-up convergence and worker reconciliation depended on tags that were silently dropped. Apply instance-scoped TagSpecifications onto the built instance so the tags surface through DescribeInstances and tag filters. * fix: serialize nodegroup reconcile via CAS to prevent scale overshoot UpdateNodegroupConfig is dispatched on a queue group, so two overlapping requests (e.g. an SDK retry mid-flight) each read current = len(InstanceIDs) and each launched desired-current workers, overshooting the target (1 -> 5 for desired=3) and stranding untracked VMs via last-writer-wins on the unconditional PutNodegroupRecord. Make every durable nodegroup-record mutation a compare-and-swap on the KV revision. Scale-up launches one worker at a time and CAS-appends it only while the record is below the live ScalingDesired, terminating any surplus VM launched into an already-full record. Scale-down terminates before the CAS truncation so a crash or retry leaves reclaimable dead IDs rather than orphaned VMs. A concurrent regression test asserts two overlapping scale-ups converge to exactly desired under -race. * fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress The AWS provider's aws_vpc_security_group_ingress_rule reads SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset panics the plugin (Plugin did not respond), though the rule applies server-side. Echo the created rules via sgRuleToSecurityGroupRule, the same shape DescribeSecurityGroupRules already returns. * fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500 launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the gateway ErrorHandler's exact-match against ErrorLookup missed and remapped to InternalError (500). An out-of-capacity scale now surfaces the real InsufficientInstanceCapacity code (400); opaque internal failures still wrap and stay 500. Adds awserrors.HasErrorCode for the membership test. * fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu Worker/server user-data runcmd ended with `[ rc-service, local, start ]`, which runs inside cloud-final while the OpenRC `local` service is ordered after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling the k3s join. Run the IMDS route script directly instead and keep `rc-update add local default` for reboot persistence. setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the menu stays interruptible over serial). Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final` stall, apiserver ready ~66s (was ~109s+). mulga-siv-332 * feat: add argocd managed addon + bake bundle into eks-node AMI The addon catalog advertised coredns and aws-load-balancer-controller, but the only baked manifest shipped in the eks-node AMI was the spinifex-noop fixture. Installing any real addon failed with 'no baked bundle for version' once the VM-side renderer found nothing on disk. Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no IRSA), a flattened namespaced install manifest baked into the AMI, and the manifest.conf bake line. The eks-addons-access workbook switches from coredns (redundant on K3s, never baked) to argocd. The baked bundle is generated from upstream install.yaml via kubectl kustomize with namespace: argocd plus a prepended Namespace, since upstream is namespace-agnostic and K3s auto-deploy applies with no -n flag. * fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint and strands itself on the control-plane node. In the managed-CP topology the worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot reach a CP-resident CoreDNS and lose all cluster + external DNS. Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik) and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP both work). Image-side only; no handler change. * fix(eks): revoke cross-referencing SG rules before delete on teardown DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a security group still referenced by another SG's rule. Both deletes failed with DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu destroy hung on DeleteVpc). Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's DependencyViolation. * fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP In the managed-control-plane topology the apiserver advertised its Set-B node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the unpeered CP VPC and is unreachable from worker pods, so every worker pod using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers, argocd redis secret-init) failed. - k3s_server_vm.go: set advertise-address to the worker-reachable NLB front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs. - nlb.go: add a second :6443 listener on the cluster NLB (same CP target group) so PrivateEndpointIP:6443 — the advertised endpoint — is served. - security_groups.go: admit :6443 on the private-endpoint SG. Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd pods Running, kubectl logs on worker pods working. |
||
|
Josh Siviour
|
cba1ecceaf |
fix: route EKS DisassociateAccessPolicy policyArn path segment (#367)
The DisassociateAccessPolicy route regex terminated at
/access-entries/{principalArn}/access-policies, but the AWS EKS REST API
sends the policy ARN as the final path segment:
DELETE /clusters/{name}/access-entries/{principalArn}/access-policies/{policyArn}
The route never matched, so the gateway returned InvalidActionException
(seen as a 400 on 'tofu destroy' of aws_eks_access_policy_association).
The handler also read policyArn from the request body, which AWS leaves
empty for this DELETE.
Capture policyArn as a third path param and pass it through to the input,
dropping the body read. Update route and wrapper tests accordingly.
|
||
|
Josh Siviour
|
c22a6f4ffc |
fix: install chrony on host via curl|bash setup path (#366)
The host clock needs an NTP daemon; without one it drifts and SigV4 request signatures fall outside the skew window, kicking the web console back to /login. Mirrors the ISO rootfs (iso-builder packages.list) so both host-install paths ship the same software. Guest VMs track the host via kvm-clock. Bug: mulga-siv-335 |
||
|
Ben Duncan
|
0dc5c97ba9 | Draft doc feedback | ||
|
tomnewton-mulga
|
362e1f42c1 | Merge branch 'dev' of github.com:mulgadc/spinifex into dev | ||
|
tomnewton-mulga
|
e2782ede44 | chore: first draft of SMCI MIG writeup | ||
|
Josh Siviour
|
9a474220ce |
fix: preserve terminated-instance describe-visibility window in GC reaper (#363)
* fix: preserve terminated-instance describe-visibility window in GC reaper The TerminatedTeardownReaper purged a terminated record the moment it drove the instance's pending NAT/OVN teardown to done. Normal terminations always arrive NAT/OVN-pending, so the first 2m GC sweep completed and purged them immediately, removing the instance from DescribeInstances before clients could observe state=terminated — an AWS-compat divergence that flaked e2e-single TestRunInstancesMultiCount. Stamp TerminatedAt on transition and purge a completed record only once it is older than a 15m visibility window; fresher records are left to the terminated bucket's 1h TTL, keeping the instance describable as terminated. The ADR-0003 §3 purge-on-completion invariants still hold (their seeded records predate the window). Closes mulga-siv-328. * test(e2e): poll intra-SG east-west ping until L2 converges TestSameSGComms fired a single un-retried 'ping -c 3 -W 2' immediately after the source instance's SSH became reachable, but the ICMP target is the destination's east-west data plane. Two instances boot in parallel and the destination guest NIC / OVN port flows come up several seconds after the EC2 running transition the launch waits on. When the probe wins that race the source kernel reports 'Destination Host Unreachable' (ARP unresolved), not an ACL drop, and the test flaked. Replace the one-shot ping with pingConverged, a bounded poll that retries until 0% packet loss or 45s. A genuine self-ingress/datapath break still stays at 100% loss for the full window and fails, so coverage is preserved. |
||
|
Julian Sommer
|
1f7adcc4dd | chore: more e2e tests for IAM + STS + IMDS (#364) | ||
|
Julian Sommer
|
30871edc05 | release: update go.mod for v1.8.0 v1.8.0 | ||
|
Julian Sommer
|
5cb73d83db | chore: rebuild frontend with latest deps | ||
|
dependabot[bot]
|
5f166b8197 | chore: bump the npm-dependencies group (#360) | ||
|
dependabot[bot]
|
4bc788ca51 | chore: bump the go-dependencies group with 4 updates (#359) | ||
|
Josh Siviour
|
129bfc0fb5 |
feat(eks): join nodegroup workers via cluster NLB endpoint + private endpoint access (#362)
* feat: join nodegroup workers via cluster NLB endpoint Under the Set-B managed-CP-VPC topology the control plane runs in the system-account CP VPC while nodegroup workers launch in the customer VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP (:6443), which sits in a different, unpeered VPC and is unreachable, so the join path was dead. Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint, https://<frontend-ip>:443, the :443->:6443 listener kubectl already uses). Guard CreateNodegroup on a non-empty endpoint. mulga-siv-298 * feat: gate nodegroup ACTIVE on workers registering Ready launchNodegroupInfra flipped the nodegroup to ACTIVE the moment RunInstances returned, so a worker that booted but never joined left the nodegroup falsely ACTIVE with no usable capacity. Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed by the cluster reconciler from the CP NATS state report — Ready nodes only) rising by the desired worker count over the create-time baseline. On timeout the nodegroup is marked CREATE_FAILED with its instance IDs retained for the reclaim path. Mirrors the CP reconciler's healthy-observe gating. Bounded by injectable timeout/poll knobs. mulga-siv-299 * feat(eks): private endpoint access via cross-account Set A ENI Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the control plane through a customer-VPC (Set A) resident ENI on the cluster NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path. Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not its primary, so the instance-terminate cascade never reclaims it. Detach (store-clear) the stale attachment before deleting, so private clusters delete cleanly instead of looping on InvalidNetworkInterface.InUse. Refs: mulga-siv-301 * fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule resources by SecurityGroupRuleId with empty IpPermissions. The handlers only read IpPermissions, so they returned success while removing nothing; the surviving SG-reference rule then pinned both SGs undeletable (DependencyViolation), deadlocking destroy of mutually-referencing SGs. Resolve SecurityGroupRuleIds to their stored rules and remove them (malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound). Refs: mulga-siv-321 * fix(vpc): reap the main route table on DeleteVpc CreateVpc auto-creates a main route table and DeleteRouteTable refuses to delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps the VPC's main route table before deleting the VPC record. Refs: mulga-siv-322 * fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS Concurrent writes to a single role / route table record used a blind read-modify-Put with no optimistic-concurrency guard, so parallel writers racing from the same revision clobbered each other and lost updates. Verified on env19: three parallel policy attachments to one node role kept only one; two parallel subnet associations to one route table kept only one (the provider then timed out waiting for state 'associated'). Route the mutating paths (AttachRolePolicy/DetachRolePolicy, AssociateRouteTable/DisassociateRouteTable) through a revision-guarded Update + bounded retry, matching the existing accountCounterBucket CAS idiom. Add concurrent regression tests covering both lost updates. * docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress) Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a browsable load-balanced demo app: - eks-quickstart: minimal single-node cluster + NodePort demo - eks-addons-access: CoreDNS managed addon + access-entry RBAC - eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS with an ACM-imported self-signed certificate Each covers the Spinifex-specific endpoints, the auto-managed worker SG NodePort pattern, and the omitted addon_version constraint. |
||
|
Josh Siviour
|
1873f45d26 |
Resilient resource lifecycle (ADR-0001..0006): EKS multi-node teardown, idempotent deletes, GC backstops (#357)
* fix: idempotent Delete* across VPC family + volume (RLC1) Delete handlers now return empty success when the resource is already absent (nats.ErrKeyNotFound), instead of *NotFound. Transient read errors still map to ServerInternal. Covers DeleteVpc, DeleteSubnet, DeleteSecurityGroup, DeleteNetworkInterface, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable and DeleteVolume. Makes destroy retries converge instead of failing mid-graph, removing a source of orphaned/zombie resources between deploys. Adds a TestRLC1_*DeleteIdempotentOnAbsent invariant per package and updates the daemon delegation table; supersedes the old absent->NotFound tests. * feat: prune orphan ENI LSPs in reconcile applyPorts (ADR-0003 §4) applyPorts was create-or-update only, so a guest LSP whose spinifex:eni_id no longer maps to an intent ENI survived instance terminate and VM-host reinstall — the orphan-LSP capacity-drift root cause. It now takes pruneOrphans (mirroring applySGs): on a full reconcile it lists LSPs and tears down each managed ENI port absent from intent via the composed topology.DeletePort cascade (clears port-group memberships, then deletes the LSP). The apply-only startup path does not prune, preserving in-flight ports until subscribers converge. Adds ListLogicalSwitchPorts to the OVN client (live + mock). Invariant TestRLC5_OrphanLSPPrunedAfterReconcile guards ADR-0003 §4; a startup-race test guards the apply-only path. * feat: idempotent + tracked instance terminate (ADR-0003 §1/§2) Make terminate idempotent across all three layers and break the un-terminable-ENI deadlock, and record per-dependency teardown so the GC backstop can later tell a fully-reaped instance from a dangling one. - vm.Manager.Terminate: absent / already-terminated / shutting-down now return nil instead of NotFound / invalid-transition (ADR-0003 §2). - VM.Teardown map + teardown.go marks: sync steps stamped Done/Failed via cleaner error returns; fire-and-forget OVN LSP and NAT rule stamped Pending for the reconciler / GC reaper (ADR-0003 §1). - InstanceCleaner: DeleteVolumes / ReleasePublicIP / DetachAndDeleteENI / RemoveFromPlacementGroup / ReleaseGPU now return error. - vpc ForceDeleteInstanceENI bypasses the in-use guard for the owning instance, breaking the failed-detach deadlock; plain delete still rejects an in-use ENI held by another instance. - gateway TerminateInstances: KV-health-gated idempotent absence (a failed terminated-bucket query does not fabricate success, ADR-0003 §3). - Invariants: TestRLC1_TerminateIdempotentOnAbsent, TestRLC4 teardown-map stamping + failed-dependency, TestRLC_ENINonDeadlock. * feat: reconciler GC backstop framework + terminated-teardown reaper (ADR-0003 §3) Add the shared reality->desired garbage-collection framework (inverse of Restore) and its first reaper, closing the gap 295.3 deferred: nothing retried pending/failed teardown or purged the terminated record when its dependents reached done. - vm/gc.go: GarbageCollector + Reaper interface (Class/Scope/Sweep). KV-health gated — holds and alarms when the desired-state cannot be trusted, never reaps against it. ClusterWide reapers run only on the elected leader (optional elector); NodeLocal run on every node. Start sweeps immediately then every GCInterval. - vm/teardown_reaper.go: TerminatedTeardownReaper (node-local). Scans this node's terminated records, re-drives each not-done dependent through the idempotent cleaner, and purges the record once all reach done. Records already complete on arrival are left to the bucket's 1h TTL. - StateStore gains DeleteTerminatedInstance (JetStream impl already existed). - daemon: wires the GC after the pending watchdog, gated on a cheap JetStream AccountInfo health probe. - Cluster OVN-LSP arm already lands via the reconciler drift prune (295.2). - Invariants: TestRLC5_GCRetriesPendingTeardownToDoneThenPurges, TestRLC5_GCNeverPurgesIncompleteRecord, TestRLC5_GCHoldsWhenKVDegraded, TestRLC5_GCSkipsOtherNodesRecords. * feat: VPC-family dependency guards (ADR-0004 rule #3) DeleteSubnet blocks while a live ENI attachment (a resident instance) remains; available/orphan ENIs do not pin it. DeleteNetworkInterface guards on the structured attachment (eniIsLiveAttachment), not a bare in-use string. DeleteNatGateway blocks while a live route table forwards to it and disassociates its EIP keeping the allocation (AWS parity); CreateNatGateway now marks the EIP associated so the disassociate is real and a second NAT GW cannot reuse the allocation. Binding invariants: TestRLC3 dependency-guard completeness across the subnet/ENI/NAT-gateway handlers. * feat: EBS volume data-safety GC override (ADR-0005) The shared GC backstop reaps actual-desired for every other class; for volumes that default is forbidden because they carry customer data. Add VolumeLeakReaper (node-local, rides the 295.5 GC): it marks a volume left attached to a definitively-gone instance as orphaned (a non-destructive tag) and alarms, but NEVER issues a delete. The daemon feeds it only this node's terminated instances whose volumes-teardown failed, so the shared predastore scan can never false-mark another node's live-instance volume. DeleteVolume idempotency + the detach/snapshot guards + the DeleteOnTermination teardown retry already landed (295.1/295.3/295.5). Binding invariants: TestRLC5 (GC marks+alarms, never deletes data), TestRLC3 (detach-before-delete). * test(elbv2): lock ADR-0002 §3 live-only TG in-use guard (mulga-siv-295.4) * feat(eks): idempotent DeleteCluster on absent + ADR-0006 lifecycle invariants (mulga-siv-295.8) DeleteCluster returns empty success when the cluster meta is already swept (Common Resource Lifecycle Contract #1), so tofu destroy retries converge. Adds TestRLC1 (idempotent delete), TestRLC2 (billable-before-KV-sweep ordering) and TestRLC3 (NLB target-group no-orphan, riding ADR-0002/172). * fix(eks): GC reaper terminates orphan control-plane VM after cluster delete (mulga-siv-294) A normal DeleteCluster can leave a running control-plane VM behind (terminate completes async; a daemon restart re-launches it from saved running-state) after the cluster meta is swept — a billable orphan with no owner. Add EKSBillableReaper on the 0003 GC backstop: a running managed-by=eks CP VM whose cluster meta is definitively gone is terminated. The CP ENI now carries the customer-account tag so a system-account VM can be mapped back to its cluster meta; the reaper never reaps on uncertainty (ENI gone/untagged or meta present). * feat: reclaim orphan EKS NLB + NAT-GW EIP in billable reaper The billable GC reaper terminated an orphan control-plane VM but left the cluster's NAT-gateway EIP and NLB front-end NAT behind when the meta was gone. After a successful terminate, run the name/tag-driven DeleteClusterNLB + DeleteClusterCPVPC teardown (idempotent, meta-independent) so the billable NAT-GW EIP and NLB NAT are reclaimed too. ADR-0006 §5. * fix: persist NAT-GW & route-table tags and reorder CP-VPC teardown to reclaim EKS NAT-GW EIP CreateNatGateway and CreateRouteTable both dropped input.TagSpecifications, so the tag-driven DeleteClusterCPVPC teardown could not discover the cluster NAT gateway or its private route table. The explicit DeleteNatGateway + ReleaseAddress were skipped and the billable NAT-GW EIP leaked under the system account on every clean DeleteCluster. Three fixes: - CreateNatGateway persists ExtractTags(TagSpecifications, natgateway). - CreateRouteTable persists ExtractTags(TagSpecifications, route-table). - DeleteClusterCPVPC tears down route tables before the NAT gateway, so the rule #3 route-ref guard (DependencyViolation / InvalidAddress.Locked) is not tripped and the EIP releases. Adds tag-persistence regression tests for both create paths and a teardown ordering invariant (TestRLC5). Validated live: NAT-GW EIP released cleanly, no DependencyViolation/Locked. * fix: route EKS control-plane VM terminate cross-node + DELETING teardown backstop Multi-node EKS cluster delete wedged in DELETING and stranded the billable NAT-GW EIP. Two root causes, both invisible on a single-node box: Cross-node terminate was never routed. TerminateSystemInstance was node-local, so terminating a CP VM owned by another node returned NotFound (treated as idempotent) and the teardown deleted a still-attached ENI (InvalidNetworkInterface.InUse). That error short-circuited purgeClusterInfra before the managed CP-VPC teardown that releases the NAT-GW EIP. Split TerminateSystemInstance into a local body plus a routing wrapper that issues system.TerminateInstance.<id> to the owning node. Bind that subscription for every system-managed VM (elbv2 + eks) via the OnInstanceUp funnel, so CP VMs placed on the coordinator's own node (and recovered after a daemon restart) are reachable too, not only remotely-launched ones. No DELETING backstop. A failed synchronous DeleteCluster left the cluster DELETING with nothing to re-drive it. Add EKSDeletingReaper (vm.Reaper, class eks-deleting) that re-runs purgeClusterInfra for clusters wedged past a min-age, gated by the per-cluster reconciler leader lease. Stamp ClusterMeta.DeletingSince on the transition into DELETING. TerminateK3sServerVM additionally waits out any residual ENI-InUse with a bounded budget as defense-in-depth. Validated live on env19 3-node: first delete-cluster converges in 29s, both remote CP VMs terminated cross-node, NAT-GW EIP released, backstop never fires. * fix(network): gate gateway reconcile on real datapath reachability Post-reboot the external OVN datapath can stay dead while every control-plane signal is green: the chassisredirect SB binding is claimed (up=true), NAT rows are present and the host neigh is primed, yet ovn-controller leaves stale gateway/localnet flows so EIP north-south and the VM->gateway heartbeat forward nothing. ensureGatewayClaimed only nudges recompute when the binding is unclaimed, so a claimed-but-dead datapath is never detected or repaired. Add ensureGatewayDatapath: after the SB claim converges, probe the gateway LRP IP (OVN answers ICMP natively, no guest or security-group dependency), and on a miss force one ovn-controller recompute and re-probe until a short deadline. Reuses the existing recompute nudge and no-ops when no verifier is wired or no gateway IP resolves. * fix: self-heal external uplink after reboot boot-race Post-reboot, ovs-vswitchd/ovn-controller order after the passive network.target rather than network-online.target, so they start concurrently with systemd-networkd. Under a heavy boot the WAN-glue veth (veth-wan-ovs) can come up admin-down before networkd brings it up, leaving br-wan with no carrier and the external datapath dead while every control-plane signal reconciles green. Broaden the datapath health-gate remediation from recompute-only to RepairDatapath: re-assert the veth admin state (covers the admin-down link, which a flow recompute cannot revive) then recompute (covers a stale gateway ofport). Idempotent; the veth ops are skipped in physical-uplink mode where the device is absent. Also capture ip -d link (veth admin/carrier) and ovs-ofctl show br-ext (ofport) in the reboot diag bundle to disambiguate the two failure shapes from CI artifacts. * fix: restore AWS-faithful Delete* NotFound; move idempotency to teardown callers (mulga-siv-306) RLC1 (f9218982) made EC2/VPC/EBS Delete* handlers return success on an absent resource, breaking AWS compatibility and the cross-account isolation signal (a cross-account delete reads as absent -> success instead of *.NotFound). Restore the AWS contract at the API while keeping lifecycle convergence: - Handlers return canonical *.NotFound on absent: DeleteVpc, DeleteSubnet, DeleteSecurityGroup, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable, DeleteVolume, and the public DeleteNetworkInterface (ForceDeleteInstanceENI keeps its idempotent force path, ADR-0003 section 2). - Add awserrors.IsNotFound; destroy orchestration (instance terminate, EKS CP-VPC teardown, EKS IGW/SG cleanup, k3s rollback, ELBv2 ENI/SG teardown) tolerates NotFound so re-issued/double-targeted deletes converge. - Flip the RLC1 unit invariants to assert *.NotFound on absent; revert the daemon round-trip cases; add IsNotFound + CP-VPC convergence tests. ELBv2 deletes and TerminateInstances stay idempotent (AWS itself is). * fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence TestMultinodeGuestSSH intermittently times out (Connection timed out, not #355's reset-by-peer) connecting to a fresh trio instance's public IP after the suite churns instances across all three chassis. The instance is running and AddEIP has installed the DNAT row, flushed/primed the host neigh, and barriered the gateway-chassis flow — but none of that proves the gatewayLRP->guest geneve hop: until the guest ENI's southbound Port_Binding is up on its hosting chassis the cross-node flow is missing and the DNAT-translated packet blackholes. Add ensureGuestPortDatapath, the guest-side analog of ensureGatewayDatapath: after AddEIP, probe the guest LSP's SB Port_Binding (ovn-sbctl up column); on a miss force one ovn-controller recompute (binds the port / installs the remote flow) and re-probe until a short deadline. Recompute, not RepairDatapath: this is flow/binding convergence, not a WAN-uplink fault. No-op when no verifier is wired or the EIP carries no guest port. spec.PortName is populated for every associated EIP regardless of NAT mode. Extends GatewayClaimVerifier with GuestPortUp and wires the gate into applyEIPs so it runs in the continuous reconcile loop on every node. * Revert "fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence" This reverts commit 728b1a9cec6eaf5c7ef5e9eda4469f239b4d8fc0. * fix(network): key EIP DNAT delete on external IP, not logical IP A dnat_and_snat row's identity on the VPC gateway router is its external IP. The logical (private) IP is not a stable key: the EIP pool round-robins and instance private IPs are reused as instances recycle, so the same logical IP is held by different EIPs over time. DeleteEIP deleted the row by logical IP, so a stale or retried delete for a freed EIP clobbered whichever live EIP now held that recycled private IP -- the next owner's row vanished with no re-add and its public IP went dark (SSH SYN never left br-wan: Connection timed out). Lifecycle-resilience teardown/GC retries amplified this by firing vpc.delete-nat repeatedly. Key the delete on the external IP via DeleteNATByExternalIP and skip when no external IP is supplied (no EIP means no dnat_and_snat row). A stale delete now targets the freed EIP's own external IP (ErrNATNotFound no-op) and can no longer touch the live row. * refactor: relax DeleteNatGateway to AWS-faithful Remove the route-reference dependency guard (checkNatGatewayRouteRefs) so a NAT gateway deletes even while route tables still forward to it, matching AWS (the route blackholes). The reconciler's loadNATGWs drops the SNAT once the record leaves the active bucket, so a forwarding route is not a delete-blocking dependent. EIP is still disassociated on delete (no leak). * test: invert NAT-GW route-ref invariant to AWS-faithful TestRLC3_NatGatewayBlocksWhileRouted asserted DependencyViolation while a route forwarded to the NAT gateway, which is AWS-infaithful. Replace with TestRLC3_NatGatewayDeletesWhileRouted: delete must succeed whether or not a live route still targets it (the route blackholes; the reconciler drops the SNAT). Per ADR-0004 §2. * fix: tear down NAT-GW SNAT on delete (restore delete-time fan-out) The guard commit (53ed55a9) dropped publishDeleteEventsForNatGateway on the assumption a routed NAT GW could never be deleted. With the route-ref guard removed, deleting a NAT GW while routes still target it left the SNAT in place (applyNATGWs is create-only, so the reconciler never prunes it) and egress kept working — TestNATGateway's post-teardown 'lost internet' check failed. Restore the fan-out: on delete, scan route tables and publish vpc.delete-nat-gateway per associated subnet so egress stops, while the route entry survives as a blackhole (AWS-faithful). * fix: probe EIP datapath in reboot self-heal, not the gateway LRP IP The post-reboot gateway datapath self-heal probed the gateway LRP IP for liveness, which OVN answers natively at the router — so it stayed green even when the EIP NAT pipeline was stranded (ofproto/trace dropped EIP traffic at table 0). RepairDatapath never fired and TestRebootResilience timed out, with the drift loop re-probing the same false-green every cycle. Prefer probing an associated EIP: force a fresh ARP of the dnat_and_snat external IP (flush neigh, ping to trigger, read the resolved MAC). OVN answers ARP for the external IP from the gateway chassis independent of the guest, so a resolved MAC proves the WAN uplink forwards and the NAT flows are installed — the signal the LRP IP cannot give. Fall back to the LRP IP for VPCs with no EIP. The existing drift-loop cadence now re-heals a later flow strand. * fix: lb-agent heartbeats over mgmt port, not the WAN/EIP datapath The lb-agent heartbeat to awsgw is control-plane and must survive a host reboot that strands the OVN/EIP north-south data plane. Customer (internet-facing) lb-agents heartbeated over the WAN GatewayURL, riding the EIP datapath, so after a reboot the ALB never reported healthy until the datapath self-healed. System LBs already heartbeated over the on-link br-mgmt NIC; this extends that path to every lb-agent. buildLBAgentEnv now always uses mgmtGatewayURL (on-link MgmtBridgeIP, the mgmt NIC the lb-agent VM already carries), falling back to the WAN URL only when no mgmt bridge exists. Drops the account-id branch. * fix: re-heal guest ENI SB Port_Binding post-reboot so EIP ingress converges Post-reboot every EIP in a VPC goes dark north-south: the DNAT row, host neigh and gateway-chassis flow are all in place (ARP to the EIP resolves) but the gatewayLRP->guest flow is missing until ovn-controller binds the guest ENI's SB Port_Binding, so the DNAT-translated packet blackholes and the public IP is unreachable on an otherwise-running instance. Restore the guest-port convergence gate in applyEIPs: probe the guest LSP's SB Port_Binding up column; on a miss force an ovn-controller recompute and re-probe until a deadline. Unlike the earlier reverted form, recompute on every miss, not once: after a host reboot the guest tap is replumbed several seconds after the reconcile runs, so a single early nudge fires before the port exists and never binds it. The window is widened to span guest boot. This is the southbound analog of the gateway datapath self-heal; the EIP ARP probe cannot detect it because the gateway chassis answers ARP for the external IP even while the guest delivery flow is absent. * fix(network): force post-reboot EIP recompute on guest-port bind, startup-only The reboot suite passed only by accident: the pre-#351 gateway-claim verifier polled the always-chassis-less gateway LRP, so it fired an ovn-controller inc-engine/recompute on every reconcile. That spurious recompute was load-bearing - it reinstalled the centralised gateway DNAT flows that ovn-controller's incremental engine misses on a cold (post-reboot) start. #351 correctly switched the check to the chassisredirect port, eliminating the every-cycle recompute, and the EIP datapath went dark post-reboot with every control-plane signal green. Restore the kick, scoped to the post-reboot startup path: applyEIPs now gates a single recompute on the guest ENI's SB Port_Binding chassis claim - the reliable signal the tap is back, unlike the up column which stays stale-true across a reboot. Drift never recomputes, preserving the #351 fix for freshly attached EIPs. Probe renamed GuestPortUp -> GuestPortBound (chassis, not up). * Revert "fix(network): force post-reboot EIP recompute on guest-port bind, startup-only" This reverts commit 792e318211ef03efbe24febeb470749d88c0b03c. * test(e2e/reboot): capture dataplane diag on ALB serve failure The post-reboot ALB-serve assertion (8.6) failed with no flow/ARP evidence: the OVN/OVS dataplane dumps only ran on the 8.5 targets-unhealthy path, but 8.5 passes on the daemon's own health view while the north-south EIP datapath is dark, so the serve failure (8.6) left nothing to localise the drop. On a serve failure, capture dumpALBProbe/dumpOVNDataplane/dumpOVNClaimState/ dumpHeartbeatDiagnostics/dumpAppDiagnostics before failing. Add to the OVN dataplane dump: dump-flows br-ext and fdb/show br-ext (north-south bridge, previously only br-int), and a real ARP probe that flushes the neigh entry first so primeReachability's pre-seeded reachable entry can no longer mask an unanswered ARP. Pure diagnostics; no production change. * test(e2e/reboot): decisive datapath diag — untruncate br-int, real-ingress ofproto/trace, conntrack + tap capture * test(e2e/reboot): add south->north return-path ofproto/trace to localize egress drop * test(e2e/reboot): derive LRP MAC from realized flow + remediation experiment (recompute vs ovn-controller restart) * test(e2e/reboot): dump persisted SB MAC_Binding + external MACs; flush experiments (conntrack vs MAC_Binding) to isolate recompute-resistant return-path drop * test(e2e/reboot): untruncate Port_Binding + enumerate DGP/chassisredirect claim state (does cr-gw row exist + resident chassis) * test(e2e/reboot): dump NB source of return-path drop (lflow stage, NAT external_mac/options, LR policy, LSP port_security) * fix: exempt public-IP instances from subnet egress drop gate (mulga-siv-311) Post-reboot reconcile installs the priority-1100 subnet egress drop gate for an IGW-attached subnet that has no 0.0.0.0/0 route, but nothing punched the subnet's public-IP instances through it. lr_in_policy runs before lr_out un-DNAT/SNAT, so an inbound connection's reply still carries its private source when it hits the gate and is dropped. The ALB/EIP datapath goes dark south->north post-reboot while NAT, ARP, chassis and guest-port binding all read green; the pre-reboot path survives only because the event-driven gate never evaluates a subnet created after IGW attach. Install the existing priority-1200 /32 reroute (above the gate) for every public-IP instance during reconcile. Reroute only: the instance's dnat_and_snat already supplies SNAT, unlike EnsureSystemInstanceEgress which also adds a plain snat. Public IPs come from two disjoint sources -- auto-assigned/ELB addresses on the ENI record and user EIPs in the EIP bucket -- so drive the exemption from both intent.Ports (new PortSpec.PublicIP) and intent.EIPs. * refactor: scope public-IP drop-gate exemption to gated subnets Only install the priority-1200 reroute when the instance's subnet actually carries a drop gate. Routed subnets egress via their priority-1000 reroute and need no exemption, so bounding on gate presence keeps the change inert for IGW-routed and NATGW-routed subnets (no redundant or overriding policy). Adds a negative test. * refactor: drop unused RemoveEIPInstanceEgress No production caller; the exemption is install-only in reconcile. Removed the interface method and impl rather than leave dead API surface. * test: dump guest console + OVN/datapath diag on SG-to-SG client SSH timeout TestSGToSGDatapath waited for client-vm SSH via EventuallyErr and reported only 'exit status 255' on timeout — no guest-side or datapath state. When the fresh public-IP VM is unreachable for the full window the failure was undiagnosable from CI artifacts. Swap the wait to the non-fatal trySSHReady probe and, on timeout, call DumpVPCFlowDiagnostics (guest console tail, ovn-nbctl/sbctl show, NAT rows, br-int flows / conntrack / host ARP keyed on the public + private IP) before Fatal, mirroring vpc_test/baseline_test. Captures persist to the artifact dir for the Stage 2 analyzer bundle. * fix: ownership-scope public-IP teardown to survive IP recycling A live instance's dnat_and_snat rule, host ARP, and IPAM lease were torn down by the delayed garbage-collection teardown of a different, long-dead instance that previously held the same recycled public IP. The 2-min vm/gc instance-teardown reaper re-runs ReleasePublicIP on a stale terminated record; because the external IPAM round-robins a small pool, the IP had already been reassigned to a live instance, and both teardown primitives acted on the IP by value with no ownership check: - policy.DeleteEIP deleted the dnat_and_snat row by external IP alone, discarding the logical IP it receives. - StaticPoolAllocator.Release freed the pool entry by IP value. Scope both to the current owner so a stale or duplicated teardown for a recycled IP is a no-op: - DeleteEIP looks up the row by external IP and skips the delete (and the ARP flush) when its logical IP no longer matches, mirroring AddEIP's idempotency lookup. Defeats both external- and logical-IP reuse races. - Allocator.Release / ExternalIPAM.ReleaseIP take an ownerENIID; the static allocator no-ops a release whose ENI differs from the live allocation, and treats an already-freed owner-scoped release as idempotent. Teardown and rollback callers pass their ENI; user-driven EIP release passes none. Proven via E2E single TestSGToSGDatapath (run 27515293563): client-vm SSH black-holed after its NAT was deleted by the dead instance's GC sweep. * fix(eks): single-flight DeleteCluster teardown per cluster SDK retries of one DeleteCluster get fanned across nodes by the worker queue group, so purgeClusterInfra raced the same cluster on multiple nodes: duplicate ENI/NLB teardown, duplicate TargetGroupNotFound deregisters, and a real double-release risk on the billable NAT-GW EIP. Idempotency kept it correct but noisy. Add a dedicated per-cluster teardown lease (teardownLeaderKey = reconcilerLeaderKey + "/teardown"), distinct from the reconciler lease so a delete never contends with the still-live reconciler that holds the reconciler lease until it observes DELETING and exits. DeleteCluster acquires it after the NotFound-idempotency check: the loser returns the cluster as DELETING (AWS-async semantics) without running the teardown; the winner drives purgeClusterInfra and releases. The DELETING backstop reaper shares the same lease, so at most one purgeClusterInfra runs per cluster cluster-wide. A nil leaderKV (single-node) skips gating, so local behavior is unchanged. * fix: env-driven SSH-ready budget for slow baremetal guests Add SPINIFEX_SSH_READY_TIMEOUT override for the single-node SSH-handshake probes (default unchanged at 3m). Baremetal boots q35 guests slower (OVMF + NBD-backed disks) than CI loop devices and intermittently misses the 3m window; the harness can now widen it to 5m without affecting shared-runner e2e. Applied at all single-node probe sites; failure messages now report the actual budget. * fix: shared external switch to stop per-VPC localnet ARP collision Per-VPC external switches each attached an untagged localnet port to the same physical network (br-ext). OVN cannot keep multiple untagged localnets on one physical network L2-separate, so with two or more concurrent public-IP VPCs ARP for a public IP landed on the wrong VPC's switch and the owning gateway router's ARP responder never fired, leaving the public IP unreachable. Move to a single shared external switch (provider-network pattern): one switch with one localnet on network_name=external. Each VPC's gateway LRP attaches via its own router-type switch port; per-VPC NAT isolation is unchanged. AttachIGW ensures the shared switch idempotently and is migration-safe; DetachIGW removes only the per-VPC port + LRP and preserves the shared switch. DeleteVPC removes the per-VPC gateway switch port from the shared switch so it does not dangle. Reconcile detects attachment via the gateway LRP since the shared switch carries no vpc_id. * fix: owner-scope DeleteEIP by ENI port to survive identical pair recycling A stale or GC-re-emitted vpc.delete-nat for a terminated instance tore down a live instance's dnat_and_snat rule (and flushed the host ARP) when the public IP and private IP both recycled, together, to that live instance in the same VPC. The (external_ip, logical_ip) pair-key matches under identical recycling, so the existing logical-IP guard fell through. Stamp the owning ENI port (spinifex:logical_port) on the dnat_and_snat row at AddEIP, thread the port through handleDeleteNAT -> DetachEIP -> DeleteEIP, and skip the delete when the row's stamped port differs from the incoming one. The IPAM release was already owner-scoped; this mirrors that guard at the NAT layer so any stale delete for a recycled pair is a no-op. Empty port preserves the prior pair-key behaviour. |
||
|
Josh Siviour
|
465fa1e6c7 |
fix: complete at-rest encryption launch + volume lifecycle (#361)
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against the fix for the encrypted AMI snapshot metadata tag-verify failure (cipher: message authentication failed) on instance launch. Pairs with viperblock PR #22. * fix: forward viperblock master key to nbdkit mount path viperblockd loaded the master key but never passed it to the nbdkit plugin it spawns for volume mounts, so encrypted boot volumes opened in cleartext and failed LoadState with an encryption-state mismatch. Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's encryption_key_file= argument when set. * chore: bump viperblock to fresh-volume VolumeUUID mint fix * chore: bump viperblock to snapshot-link LoadState ordering fix * fix: route encrypted-volume config writes through viperblock keyholder config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived nonce can only be advanced by the master-key holder that owns the volume. The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to refuse encrypted volumes (rewriting would strip the tag) — bricking UpdateVolumeState (attach in-use / detach available) and ModifyVolume. Route encrypted-volume config updates through viperblockd, mirroring the ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config queue group opens the detached volume exclusively and reseals. Cleartext volumes keep the direct-S3 fast path. No crypto leaves viperblockd. mulga-siv-323 * fix: unwrap encryption envelope when snapshot handler reads config.json CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into a VBState. For encrypted volumes config.json is an at-rest envelope {payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so CreateSnapshot 500s on the size guard and the in-use scan silently drops encrypted volumes' snapshots. Unwrap via viperblock.StateBody first, mirroring the volume handler's mergeVolumeConfig path. mulga-siv-324 * test: cover viperblockd ebs.config handlers Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler (ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only size reconcile, live-VB apply, invalid JSON, apply errors, and the detached-volume open path. Clears the diff-coverage gate (68.8%). * chore: pin viperblock to dev tip (#22 merged) |
||
|
Josh Siviour
|
17c83cb3ad |
fix: build microVM rootfs via buildx to stop overlay2 export leak (#358)
docker export of a container leaks its overlay mount on Docker's overlay2 driver — the mount survives docker rm, so the layer dirs are never reaped except by a daemon restart. The lb-agent microVM build ran docker export on every e2e job across ~9 runners, accumulating ~69G of orphaned overlay2 layers on the CI host (disk hit 94%). Build the rootfs with `docker buildx --output type=local` from a small generated Dockerfile instead, which writes the image filesystem to a directory and releases all mounts — nothing leaks. Keep the podman path (its store is unaffected). Tighten the e2e cleanup image prune to dangling-only so the alpine base stays cached instead of re-pulling. |
||
|
joshsiv-mulga
|
3835f39d35 |
fix: bump predastore + viperblock pins to dev-tip pseudo-versions
CI e2e-build compiles the spinifex module with GOWORK=off, resolving the go.mod pins instead of the go.work local submodule redirect. v1.7.0 of both modules predates symbols spinifex now calls (viperblock.StateBody, predastore auth.ParseRoleARN), so the build failed with undefined-symbol errors that go.work masked locally. Pin both to current dev-branch pseudo-versions until a coordinated release is cut. |
||
|
tomnewton-mulga
|
eed57ee084 | Merge branch 'dev' of github.com:mulgadc/spinifex into dev | ||
|
tomnewton-mulga
|
69276f9a7a | fix: Added better intro to Reference Architecture docs | ||
|
tomnewton-mulga
|
1f22c96b3d | fix: Added better intro to Reference Architecture docs | ||
|
Bryn Mailer
|
f6f3956952 |
feat: enable viperblock at-rest encryption by default on fresh installs (#323)
* feat: enable viperblock at-rest encryption by default on fresh installs
Fresh installs left nodes.<node>.viperblock.encryption_key_file empty, which
viperblockd treats as legacy cleartext mode. The encryption machinery existed
but the install path never generated a key or set the field.
Provision a cluster-wide viperblock master key in all three fresh-install
flows and render its path into spinifex.toml:
- admin init (single node): generate the key locally.
- admin init --multi-node (leader): generate once, save locally, and
distribute via the formation server (mirrors IAM master key distribution).
- admin join (joiner): receive the shared key from the leader rather than
minting its own, so volumes sealed on any node open everywhere. Lenient on
absence — an older leader returns no key, so the joiner warns and stays
cleartext instead of failing the join.
The key file is omitted from the template when unset, so upgrades with no
existing key keep running cleartext.
* fix: make viperblock encryption key group-readable for both readers
The key is loaded by two services per node — viperblockd (spinifex-viperblock)
and the awsgw EC2 handlers (spinifex-gw), both in the spinifex group — so it
must be group-readable, not owner-only like the per-node predastore key.
The previous commit generated the key but never registered its path with the
deployment ownership machinery, leaving /etc/spinifex/viperblock as root:root
0700 (non-traversable) and the key as root:root 0600. The daemon failed at
startup with "stat ... permission denied".
Mirror master.key: setup.sh creates /etc/spinifex/viperblock (0750), and
SetServiceOwnership chowns the dir to spinifex-viperblock and promotes the key
to root:spinifex 0640.
* fix: read viperblock config.json through StateBody for encrypted volumes
The metadata layer parsed config.json with json.Unmarshal/NewDecoder directly,
which fails on encrypted volumes/AMIs now that viperblock wraps VBState in a
valid-JSON envelope ({"v":1,"payload":...,"authtag":...}). DescribeImages
silently skipped encrypted AMIs (count=0 → "no AMIs available"), bricking the
whole launch path.
Route every config.json read through viperblock.StateBody, which unwraps the
envelope (no key needed — the metadata is authenticated, not encrypted) and is
a no-op for unencrypted volumes:
- image: DescribeImages, getVolumeConfig, amiNameExists, GetAMIConfig
- volume: getVolumeConfigAndEncryption, and mergeVolumeConfig (so the
refuse-to-merge-encrypted guard sees the real VBState instead of clobbering
the envelope with a tag-less blob)
- admin: readAMIConfig, readVolumeConfig
|
||
|
Josh Siviour
|
b23fa3bf60 |
feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356)
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
|