The AWS provider reads a bucket's tags with `ListTagsForResource` on S3 Control when it knows the account ID. S3 Control puts the account in the hostname — `https://{account}.{endpoint}` — so against an IP-addressed endpoint the request fails a DNS lookup before a packet leaves, and no endpoint configuration can help. A tagged bucket therefore fails its first read after being created.
`skip_requesting_account_id` alone does not fix it: `aws-sdk-go-base` takes the account ID from the STS `GetCallerIdentity` credentials check before it consults that flag, so `skip_credentials_validation` is needed alongside it. With no account ID the provider falls back to `GetBucketTagging`, which predastore serves.
`s3-webapp` is the only workbook that creates a bucket, and it uses no `aws_caller_identity` and references no other security group, so a blank account ID costs it nothing.
The bucket also gets `force_destroy`, since the webapp uploads into it and a destroy of a used bucket fails `BucketNotEmpty` without it.
The e2e harness now asserts the bucket carries the tags Terraform set, which is what catches this regressing, and no longer aborts the remaining workbooks on a failure — `s3-webapp` failing says nothing about whether RDS works, and four nights of `rds-quickstart` coverage were lost that way. `STOP_ON_FAIL=1` restores the early abort.
Every workbook's provider versions are pinned, so a run reproduces rather than picking up whatever the registry published that morning.
Multi-node install
- Lead with the fact that the guide builds a three-node cluster, and say
how to go beyond three. The callout sits inside ## Overview because the
docs site discards everything above the first section heading.
- Replace the OVN-only sizing prose with a table covering all three
distributed layers, and move the OVN control-plane detail into the VPC
networking guide, linked from here.
- Add minimum and recommended hardware.
- Move the object storage layout to the end as background; it was the
second thing a reader hit.
- Collapse the ISO conversion notes from a section into one skippable
box, now that --force is unconditional in Step 4 and the formation
port needs no manual intervention.
- Expand verification with spx get nodes and spx top nodes, including
what to look for in each.
- Drop the install-node.sh shortcut (internal tooling) and the
"adding a node later" section (no good story yet).
Corrections found while writing
- Servers 4+ were described as running no database. They run predastore
and viperblock like every other node; only the OVN database cluster is
capped at three. Fixed in four places.
- Step 4 never needed the firewall down: spx admin init opens the
formation port itself for the formation window. Only Step 3's
peer-scoped OVN ports are affected.
- `nft list table inet spinifex_filter | grep spinifex_peers` could never
match — the peer list is an nft define, expanded at load time. Anyone
following it would have seen empty output and assumed the worst.
- Disambiguate "database": with predastore, NATS and OVN all in play it
needed qualifying throughout.
- Align the example output region with the AWS_REGION the guide exports.
Cross-doc links
- Rewrite relative and category-prefixed links to the canonical
/docs/<slug> form. The site publishes docs flat, so ../foo/README.md
and /docs/category/foo both 404.
- Point the six links to unpublished docs at the source repo instead.
- Fix four fragments written against GitHub's anchor algorithm.
* docs: bring doc titles and descriptions into SEO length limits
Rewrite the frontmatter title and description of every published doc so the
rendered <title> lands in 50-60 characters (including the site suffix) and the
meta description lands in 150-160 characters.
* docs: move SEO titles to a seoTitle field and restore display titles
The frontmatter title renders as the page H1 and the sidebar label, so tuning it
for the title tag shortened nothing and lengthened the visible navigation. Put
the search-facing string in seoTitle and give title back its original value.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
* docs: add Containers section with EKS and ECR service pages
New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks.
* docs(eks): rework Terraform workbooks into a capability ladder
Rework the three EKS workbooks into a coherent ladder and publish them to
the Containers section. All tiers are single- or three-node capable via
node_desired_size and serve a shared Spinifex-themed demo app from ECR,
replacing nginxdemos/hello.
- demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/
region + hit counter; optional /data persistence), distroless image.
- eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort.
- eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer-
controller addon + ACM, public/private VPC with NAT, managed-ingress=false.
- eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from
a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access
entry RBAC, HTTPS Ingress carried over.
Pending live e2e verification (image build/push, apply, ALB + PVC bind).
* fix(ecr): implement ListTagsForResource (empty tags) for client parity
The AWS provider calls ecr:ListTagsForResource on every repository read, which
returned 501 NotImplementedException and broke 'tofu apply' of any workbook that
creates an aws_ecr_repository. Implement it to return an empty tag set (resource
tagging stays a deferred feature) so reads succeed. Drop the tags block from the
three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed
TagResource on create.
* feat(eks): pull from internal ECR via kubelet credential provider
Worker nodes could not pull custom images from the internal Spinifex ECR
registry: the eks-node had no DNS for the registry host, no containerd CA
trust, no port rewrite to :9999, and no pull credentials. Only public
images worked.
Add cmd/ecr-credential-provider, a kubelet exec credential provider that
fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken
against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic
auth for the registry host. Thread the gateway URL, account, region, and
internal suffix into the agent userdata, which now writes the gateway CA,
a registries.yaml (CA trust + :9999 endpoint), the kubelet
credential-provider config and config.yaml.d drop-in, and an /etc/hosts
entry mapping the registry host to the gateway IP. Ship the provider binary
in the eks-node AMI.
Also remove a stray paragraph from the admin-cli README.
* fix: dial EKS ECR registry mirror by gateway IP
cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on
every boot, wiping the registry-host mapping appended via bootcmd, so worker
nodes could not resolve the internal ECR host and pulls failed with no such
host. Point the k3s registries.yaml mirror endpoint and tls config at the
gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image
ref port-less and dropping the DNS leg until northstar lands. Fall back to the
hostname endpoint when the gateway is not an IP.
* fix: attach node-role instance profile to EKS workers for ECR
EKS workers launched with no IAM instance profile, so IMDS served no role and
the ECR credential provider could not mint a registry token, leaving internal
ECR pulls failing with 401. Real EKS implicitly creates an instance profile for
the node role; do the same here. Wire a KV-backed IAM service into the EKS deps
and, at worker launch, find-or-create an instance profile named for the node
role, attach the role, and pass it to RunInstances so IMDS exposes the role to
the credential provider.
* feat: use mulga logo in EKS demo app
* feat: add ACM certificate tag support (List/Add/RemoveTags)
Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate.
* test: cover ACM tag gateway and NATS wrapper paths
* feat: support EKS nodegroup tags (Tag/Untag/ListTags)
Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException.
* fix: provision EKS ALB ingress via node credentials
The aws-load-balancer-controller addon could not provision an ALB. Three fixes:
- TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own
BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE,
falling back to the Go system trust pool and rejecting the gateway cert as
'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool.
- Credentials: an empty AWS_ROLE_ARN beside the web-identity token file
half-arms the SDK web-identity provider, which then signs anonymously
(MissingAuthenticationToken). Inject the IRSA env only when a service-account
role ARN is staged; otherwise omit it so the default chain falls back to the
node instance-profile credentials served over IMDS.
- Authorization: the node role lacked the ec2/elbv2 actions the controller
needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController
managed policy to the node role (CreatePolicy + AttachRolePolicy).
* fix: add force_delete to EKS workbook demo ECR repos
tofu destroy of an EKS workbook left the demo ECR repository behind: it
still holds the pushed demo image, so DeleteRepository (force=false)
returns RepositoryNotEmptyException. Add force_delete = true to each
workbook's aws_ecr_repository.demo so teardown removes the repo with its
images.
* fix: force-delete EKS control-plane ENIs on teardown
TerminateK3sServerVM deleted the CP ENI with a plain force=false
DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the
record's Status/InstanceId/AttachmentId are set, so when the K3s VM is
gone but the ENI record's attachment fields were never cleared the delete
returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper
re-drove the same non-idempotent call every 90s, wedging the cluster in
DELETING.
Teardown owns its CP ENIs and has already terminated the VM, so the
attachment is authoritatively dead. Replace the InUse-retry budget loop
with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the
stale attachment fields, then DeleteNetworkInterface passes the in-use
guard. Both calls tolerate an already-gone ENI (NotFound) so a race with
the async instance-terminate cascade resolves to idempotent success.
purgeClusterInfra's private-endpoint ENI path reuses the same helper.
* fix: detach+delete worker ENIs on EKS nodegroup teardown
A terminated nodegroup worker could leave its primary ENI in
spinifex-vpc-enis as in-use/attached, and a dangling ENI throws
DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two
gaps:
TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped
any worker owned by another node, stranding both the VM and its ENI on
multi-node clusters. Route the terminate per-instance via ec2.cmd.<id>
(mirroring the gateway TerminateInstances path) so the owning daemon runs
the full teardown — detach + force-delete the primary ENI and clear its
record. No responder / NotFound payload is treated as already-gone so a
retried DeleteNodegroup stays idempotent.
deleteInstanceENI (stopped-instance terminate) issued a force=false
DeleteNetworkInterface with no prior detach: a stale in-use attachment
made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind
after the instance was already written to the terminated bucket. Detach
first, then delete, tolerating NotFound.
* fix: heal volume State drift on detach/terminate teardown
A detach/terminate could leave a volume's State empty instead of
'available'. DeleteVolume gated on State=='available' exactly, so it
returned VolumeInUse for an unattached volume nothing was using, leaving
it undeletable and blocking stack teardown. DescribeVolumes cosmetically
rendered an empty State as 'available', masking the drift.
- DeleteVolume now treats an empty State with no attachment as deletable
(an attached volume still returns VolumeInUse).
- DescribeVolumes derives an empty State from the attachment (attached ->
in-use, unattached -> available) instead of blindly rendering
available, so an empty-but-attached volume is no longer hidden.
- UpdateVolumeState normalizes an unattached volume with an empty state to
available, so a detach writeback that omits the state cannot strand the
volume in drift.
* fix: break all SG cross-references on EKS teardown
DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked
only ingress (IpPermissions), never egress (IpPermissionsEgress), so a
cross-reference via an egress rule kept the SG pinned on
DependencyViolation. It also resolved only the two named cluster SGs, so
any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG,
the private-endpoint SG, user/TF SGs — was never revoked and pinned the
referenced SG on delete, leaving the VPC stuck.
Enumerate every non-default SG in the VPC and revoke BOTH ingress and
egress on each before deleting, breaking any cycle order-independently,
then delete only the cluster-owned SGs (referrer SGs are deleted by their
own teardown). The default SG is left untouched. Since DeleteClusterSGs
runs against both the customer VPC and the managed control-plane VPC, the
broadened sweep covers both teardown paths.
* feat: support TargetGroup ProtocolVersion in elbv2
CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target
groups to HTTP1) and DescribeTargetGroups/Create output return it. Without
this the aws-load-balancer-controller sees a perpetual immutable-field
mismatch (desired HTTP1 vs empty) and recreates the target group every
reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup
reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it
omitted to match AWS.
* fix: omit IRSA env from ebs-csi addon when no role staged
The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}}
plus the web-identity token volume. With no service-account role staged the
placeholder rendered empty, half-arming the SDK web-identity provider, which
panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs
Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}}
markers like the LBC bundle so the IRSA env and token volume are omitted when
no role is staged, falling back to node instance-profile credentials over IMDS.
The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes.
Requires an eks-node AMI rebake to take effect.
* docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI
Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup
with host routing (app./argocd. on a wildcard cert) instead of one ALB per
ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the
access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers
need (Spinifex wires controller creds at the node level, not IRSA) and an
ignore_changes block for the cluster's access_config and
vpc_config security_group_ids, which Spinifex DescribeCluster does not echo
back, so the provider stops trying to replace the cluster on every plan.
* fix: stop addon-sync substituting IRSA markers named in comments
The ebs-csi controller manifest's header comment spelled out the literal
{{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync
matched each marker by substring, so the comment lines were replaced with the
env/mount blocks, injecting a bare YAML sequence between two documents. K3s'
auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal
array into unstructured), so the ebs-csi Addon CR never applied and the EKS
addon hung in CREATING. Reword the comment to not contain the literal tokens,
and anchor the marker substitution to lines that are exactly the marker
(whitespace aside) so a future comment mentioning a marker can't retrigger it.
* fix: disable K3s local-storage on EKS clusters in parity mode
In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server
disabled traefik and servicelb but left local-storage enabled. Its local-path
provisioner ships a StorageClass marked default, which races the EBS CSI
ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind
nondeterministically. EKS has no local-path, so disable local-storage too and
let ebs-gp3 be the sole default. The built-in-ingress path is unchanged.
* fix: re-encrypt HTTPS target groups in haproxy ALB backends
An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which
serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's
own check then spoke HTTP to a TLS port, failed, and marked the server DOWN,
returning 503 — even though the lb-agent's separate HTTPS health probe reported
the target healthy, so describe-target-health masked the fault.
Emit check-ssl ssl verify none on the server line for HTTPS target groups so
both the check and proxied traffic re-encrypt to the backend.
* fix: reconnect wedged QMP client on next command
A timed-out QMP decode under host contention left the cached client's
shared json.Decoder mid-message, wedging every later command and breaking
EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead
on encode/decode error and redial the retained socket path (re-running the
qmp_capabilities handshake in place) on the next send.
* fix: round-trip security-group rule descriptions
SG ingress/egress rules dropped the per-rule Description on store and
describe. The AWS Load Balancer Controller tags the worker-node SG rules it
manages with a UserIdGroupPair description and identifies them by that value
to revoke them on teardown. Without the round-trip the controller cannot
match its rules, never revokes them, and the backend security group stays
referenced — DeleteSecurityGroup returns DependencyViolation forever,
hanging the ingress finalizer and tofu destroy with orphaned security groups.
Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not
identity) and render it back through describe.
* fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown
In the managed control-plane VPC topology, launchNodegroupInfra creates the
cluster control-plane and nodegroup security groups in the customer VPC for
worker<->control-plane networking, but purgeClusterInfra only reclaimed the
security groups in the managed CP VPC. The customer-VPC SGs orphaned,
cross-referenced each other, and pinned the customer VPC with
DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the
managed-CP-VPC branch too, matching the legacy customer-VPC branch.
* fix: reclaim ALB EIP when system VM already gone on terminate
TerminateSystemInstance only released an internet-facing ALB's Elastic IP on
the local path, where the VM is still present in the manager. When the VM was
already gone (health-reaped, crashed, or raced by ENI teardown), it took the
remote/no-owner path and returned idempotent success without reconciling the
EIP, leaking the wan-pool address still associated to a deleted ENI.
Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic
scan that disassociates then releases any EIP recorded against an instance, and
call it as a backstop in TerminateSystemInstance for both branches.
* fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook
The demo Application carried no resources finalizer, so tofu destroy removed
the Application CR without pruning the synced PVC. The PVC survived, the cluster
was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the
resources-finalizer.argocd.argoproj.io finalizer so deleting the Application
cascades to the PVC and ebs-csi reaps the volume before cluster teardown.
* feat: raw-IP access + documented Argo CD creds in eks-gitops workbook
The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser
without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add
a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and
https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password
command as a tofu output and document the credential + IP URLs in the README.
* fix: reclaim ALB EIP on the owning-node terminate path
The EIP backstop lived only in TerminateSystemInstance, but the ALB
delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal
directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's
cached allocation which is often empty. Extract reclaimSystemInstanceEIP
(authoritative KV reconcile via ReleaseAddressByInstanceID) and call it
from terminateSystemInstanceLocal as well as the no-owner backstop, so an
internet-facing ALB's EIP is released on every terminate route.
* chore(eks): drop coredns from the managed addon catalog
Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as
a managed addon — there is no bundled coredns manifest, so CreateAddon
coredns would stage but fail VM-side render. Remove the phantom catalog
entry so DescribeAddonVersions and CreateAddon no longer advertise it;
repoint the addon tests that used coredns as their sample to argocd.
* feat(ui): group EKS and ECR under a Containers nav section
Fold the separate EKS and ECR sidebar groups into one Containers group
with EKS (clusters) and ECR (repositories) entries.
* fix(ui): satisfy strict null checks in ECR component tests
Index mock.calls via optional chaining and narrow the getAllByRole result
before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix.
* refactor(eks): hide internal fixture addons from the public catalog
Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays
creatable (lookupAddon finds it, so the delivery-transport e2e still drives
it) but no longer surfaces in the unfiltered DescribeAddonVersions listing.
* refactor(eks): remove K3s built-in ingress; LBC-only
Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik,
servicelb, and local-storage are now always disabled on EKS nodes (AWS
parity — Service type=LoadBalancer and Ingress are the Load Balancer
Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd
traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in
the state reporter.
* fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation
Parallelize the independent subnet-create and route-table-association loops
in the create-VPC mutation with Promise.all (each task records its resource
in the rollback list before any further call), and switch the CIDR-validation
regex to named capture groups. Clears the no-await-in-loop and
prefer-named-capture-group lint errors.
* fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal
The ECR registry is served on the gateway port (:9999), but proxyEndpoint and
repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and
failed. Derive the port from the awsgw listen host and append it to the registry
host (omitted for 443). Also rename the default internal DNS suffix.
* docs: drop verbose ECR :9999 host note
* docs: fix literal-asterisk emphasis (single-* not rendered)
* docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust
* feat: default ECR registry URIs to the gateway host (no-DNS interim)
The registry authenticates by token, not hostname, so the per-account
<acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port
(already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need
no DNS; the parity host still works server-side where DNS exists.
* fix: serve ECR registry on a DNS-free gateway host end to end
Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW
bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable
host:9999 carried in the server cert SANs rather than the unresolvable
per-account parity name. Docker push then dials it directly with no DNS.
Mirror the same host on EKS workers: the registries.yaml mirror keys and the
credential provider matchImages now cover the port-less parity name, the
port-ful parity name, and the gateway IP:9999, all redirecting to the dialed
gateway endpoint. A port-ful pull from the API-returned repository_url matches
the mirror instead of bypassing it into a failed DNS lookup.
Net: docker push host, pod image host, and the API's repository_url are the
same value, with no DNS or /etc/hosts.
* docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS)
* docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS)
* docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks
The kubernetes provider in the eks-quickstart, eks-addons-access and
eks-https-ingress workbooks was configured from the managed cluster
resource (host = aws_eks_cluster.this.endpoint). On destroy the endpoint
resolved empty, the provider fell back to localhost:80, and refreshing
kubernetes_deployment_v1.hello / kubernetes_service_v1.hello failed with
'dial tcp [::1]:80: connect: connection refused'.
Move the kubernetes provider and the hello Deployment + Service into a
nested workloads/ root module with its own state. It reads cluster
identity from the parent via terraform_remote_state and the live
endpoint/CA from a data aws_eks_cluster source, so the provider is only
ever configured while the cluster exists. Apply parent then workloads;
destroy workloads then parent.
* fix: persist instance TagSpecifications in RunInstance
RunInstance built the ec2.Instance metadata but never read
input.TagSpecifications, so DescribeInstances returned untagged
instances and tag filters matched nothing. EKS node groups discover
their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup
tags, so scale-up convergence and worker reconciliation depended on
tags that were silently dropped.
Apply instance-scoped TagSpecifications onto the built instance so the
tags surface through DescribeInstances and tag filters.
* fix: serialize nodegroup reconcile via CAS to prevent scale overshoot
UpdateNodegroupConfig is dispatched on a queue group, so two
overlapping requests (e.g. an SDK retry mid-flight) each read
current = len(InstanceIDs) and each launched desired-current workers,
overshooting the target (1 -> 5 for desired=3) and stranding untracked
VMs via last-writer-wins on the unconditional PutNodegroupRecord.
Make every durable nodegroup-record mutation a compare-and-swap on the
KV revision. Scale-up launches one worker at a time and CAS-appends it
only while the record is below the live ScalingDesired, terminating any
surplus VM launched into an already-full record. Scale-down terminates
before the CAS truncation so a crash or retry leaves reclaimable dead
IDs rather than orphaned VMs. A concurrent regression test asserts two
overlapping scale-ups converge to exactly desired under -race.
* fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress
The AWS provider's aws_vpc_security_group_ingress_rule reads
SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset
panics the plugin (Plugin did not respond), though the rule applies
server-side. Echo the created rules via sgRuleToSecurityGroupRule, the
same shape DescribeSecurityGroupRules already returns.
* fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500
launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the
gateway ErrorHandler's exact-match against ErrorLookup missed and remapped
to InternalError (500). An out-of-capacity scale now surfaces the real
InsufficientInstanceCapacity code (400); opaque internal failures still
wrap and stay 500. Adds awserrors.HasErrorCode for the membership test.
* fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu
Worker/server user-data runcmd ended with `[ rc-service, local, start ]`,
which runs inside cloud-final while the OpenRC `local` service is ordered
after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling
the k3s join. Run the IMDS route script directly instead and keep
`rc-update add local default` for reboot persistence.
setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the
menu stays interruptible over serial).
Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final`
stall, apiserver ready ~66s (was ~109s+).
mulga-siv-332
* feat: add argocd managed addon + bake bundle into eks-node AMI
The addon catalog advertised coredns and aws-load-balancer-controller, but
the only baked manifest shipped in the eks-node AMI was the spinifex-noop
fixture. Installing any real addon failed with 'no baked bundle for version'
once the VM-side renderer found nothing on disk.
Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no
IRSA), a flattened namespaced install manifest baked into the AMI, and the
manifest.conf bake line. The eks-addons-access workbook switches from coredns
(redundant on K3s, never baked) to argocd.
The baked bundle is generated from upstream install.yaml via kubectl kustomize
with namespace: argocd plus a prepended Namespace, since upstream is
namespace-agnostic and K3s auto-deploy applies with no -n flag.
* fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS
k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint
and strands itself on the control-plane node. In the managed-CP topology the
worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot
reach a CP-resident CoreDNS and lose all cluster + external DNS.
Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik)
and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns
internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune
to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP
both work). Image-side only; no handler change.
* fix(eks): revoke cross-referencing SG rules before delete on teardown
DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they
cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a
security group still referenced by another SG's rule. Both deletes failed with
DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu
destroy hung on DeleteVpc).
Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then
delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on
VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's
DependencyViolation.
* fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP
In the managed-control-plane topology the apiserver advertised its Set-B
node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the
unpeered CP VPC and is unreachable from worker pods, so every worker pod
using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers,
argocd redis secret-init) failed.
- k3s_server_vm.go: set advertise-address to the worker-reachable NLB
front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs.
- nlb.go: add a second :6443 listener on the cluster NLB (same CP target
group) so PrivateEndpointIP:6443 — the advertised endpoint — is served.
- security_groups.go: admit :6443 on the private-endpoint SG.
Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd
pods Running, kubectl logs on worker pods working.
* feat: join nodegroup workers via cluster NLB endpoint
Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.
Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.
mulga-siv-298
* feat: gate nodegroup ACTIVE on workers registering Ready
launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.
Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.
mulga-siv-299
* feat(eks): private endpoint access via cross-account Set A ENI
Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.
Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.
Refs: mulga-siv-301
* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress
Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.
Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).
Refs: mulga-siv-321
* fix(vpc): reap the main route table on DeleteVpc
CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.
Refs: mulga-siv-322
* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS
Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').
Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.
* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)
Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
with an ACM-imported self-signed certificate
Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.
After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute
instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on
the new image. Make them detect at runtime so v3 and v6 envs both work.
- tests/e2e/run-e2e.sh: DescribeImages filter accepts both
ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match.
- tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img,
fall back to ubuntu-24.04.img; pass detected version to spx import.
- .github/workflows/e2e-baremetal.yml: download resolute (26.04) into
BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path.
- docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}:
AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values=
array is ORed).
Companion mulga changes (bootstrap-install.sh, bootstrap.sh, tofu-iso/*,
baremetal/*, iso-builder/spx-test-gpu.sh) land under bead mulga-siv-114.
Drop the 24.04 fallbacks once all envs are on the v6 gold image and 24.04
entries are removed from spinifex/utils/images.go.
The companion image_test.go fix (walk catalog name candidates) lives on
feat/e2e-go-harness — that test file doesn't exist on dev yet.
* feat(vm): direct-boot foundations + config/input extensions + image build pipeline
vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go
config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput
image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts
* feat(daemon): direct-boot launch path for microvm system instances
When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration
PC-machine path (DirectBoot=false) unchanged.
Closes mulga-siv-65
* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service
When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer
When ELBv2Enabled=false: existing PC-machine path unchanged.
Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.
Closes mulga-siv-66
* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold
* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag
- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note
Closes mulga-siv-67
* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed
* fix(microvm): end-to-end fixes for direct-boot LB path
Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).
virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.
sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).
BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.
wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.
Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.
* fix(build): auto-detect docker/podman when apk unavailable for microvm image build
* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs
tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.
Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.
* fix(microvm): fix mgmt NIC routing for all LB schemes
Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:
1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
'ip route add 192.168.1.33 via 10.15.8.1'.
2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
blob and skipping interface setup in init.sh entirely.
Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.
* feat(microvm): Phase 2 kernel stripping + boot optimisation
- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches
* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip
The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.
Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.
* fix(build): keep failover modules in initramfs strip
virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.
Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.
* fix(daemon): restore slog default after viperblock.New mutates it
viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.
Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.
Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.
Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).
* perf(microvm): hit Phase 2 boot/memory gates
t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.
Root causes:
1. build-microvm-image.sh staged the entire chroot into the cpio
without dropping /boot, embedding a duplicate vmlinuz, Alpine's
stock initramfs-virt, System.map, and a "boot → ." symlink loop
(~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
the kernel is copied out, before the cpio. Initramfs drops from
31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
42 MiB → 22 MiB.
2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
no firmware, ROM, or block layer). Both extracted to
directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
qemuStartSettleWait to 50 ms.
3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
Added boot-time perf flags: quiet loglevel=3 (silence printk over
115200-baud isa-serial), mitigations=off (KVM guest on trusted
host, single-tenant LB workload), tsc=reliable, no_timer_check,
reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).
4. init.sh dropped modprobe virtio_mmio (built-in:
CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).
Phase 2 gates:
- t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms
- artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script)
- ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure,
unrelated to microvm path)
* build(microvm): wire build into deploy + distro + release pipeline
Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).
Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.
Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.
* build(ansible): include microvm artifacts in build role tarball
Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).
* feat(config): enable microvm direct-boot LB path by default
Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.
Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).
* fix(microvm): bootstrap apk.static when no host rootfs tool exists
Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.
When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.
No-op when apk, docker, or podman is already present.
Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244
* revert: drop apk.static bootstrap in build-microvm-image
Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.
* docs(workbook): drop lb-alpine import for microvm direct-boot LB
LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.
* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN
When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").
Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.
* fix(e2e): redirect lb probe chatter to stderr so stalls are visible
test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.
Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.
* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node
buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.
resolveMgmtRoute already encodes the correct policy:
- multi-node any scheme → explicit MgmtRoute{Gateway,Target}
- single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP)
- single-node internet-facing → empty (reach AWSGW via VPC SNAT)
Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.
Test covers all three branches.
* feat(ansible): dev-deploy playbook for in-place artifact swap
Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.
Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.
Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.
* chore(microvm): remove legacy LB Alpine image build pipeline
Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.
build-system-image stays — other manifests (e.g. GPU images) still
use it.
* refactor(microvm): remove legacy PC-machine LB boot path
Phase 3 — direct-boot becomes the only ELBv2 LB launch path.
- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
[daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
paths) and refresh stale comments that still mentioned lbVMUserData.
ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.
* docs: drop lb-alpine catalog entry from cluster setup guide
LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.
* feat(ui): remove LB image import banner + LB-AMI gating
Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:
- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
system-managed.ts (isSystemManagedImage stays — still used by the
EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
wizard pages
- The corresponding tests that asserted the banner appears
* revert: restore lb.conf + lb-setup.sh for mulga install pipeline
Phase 3 microvm removal (commit 6313d74) deleted these too early — the
mulga install/iso artifact builders (scripts/tofu-cluster/build-install
-artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke
scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with
"ERROR: Manifest not found: scripts/images/lb.conf".
Restore both files verbatim so the legacy pipeline keeps building until
the mulga-side migration to microvm-only artifacts lands post-merge.
Internet-facing ALBs allocated their public IP by calling
externalIPAM.AllocateIP() directly from LaunchSystemInstance, bypassing
the EIP service entirely. No EIPRecord landed in the eips KV bucket, the
tag was "auto_assign" (EIP service uses "elastic_ip"), and
DescribeAddresses never returned the IP — so any OpenTofu flow that
expects an AllocateAddress-observable record hung waiting.
LaunchSystemInstance now calls eipService.AllocateAddress +
AssociateAddress when the EIP service is wired (falls back to the old
direct IPAM path otherwise). TerminateSystemInstance releases through
the service so the KV record is cleaned up and vpc.delete-nat is
published. The generic stopInstance teardown is left untouched so
customer RunInstances flows behave identically.
Also refreshes the nginx-alb terraform example to use public subnets
with auto-assigned public IPs (the configuration that actually exercises
this path end-to-end) and tightens the awsgwBindIP loopback guard in
wireLBAgentConfig so the gateway URL is usable by LB agents.
Julian Sommer