* fix: drop removed predastore s3.Config.Compaction from admin_test
The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.
* feat: graceful guest drain on systemctl stop / host shutdown
systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU
was still writing, corrupting guest filesystems. The ordered drain existed
only over NATS via cluster shutdown, never on the systemd/host path.
Add spx admin node drain --local (GATE+DRAIN against the local node, leaving
STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new
spinifex-shutdown.service oneshot ordered After= all services so it stops
first and drains guests while every service is still up. DDIL preserved: the
oneshot fires only on target/host shutdown, not on a daemon restart.
Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an
nbdkit a guest is still attached to. Add a units ordering invariant.
* fix: ExecStop drain timeout needs duration unit (120 -> 120s)
The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin
node drain's --timeout is a cobra Duration flag and rejects a bare integer
(time: missing unit in duration "120"). The drain exited 1 without running,
so a systemctl stop tore nbdkit out from under the running guest and corrupted
its ext4 (the exact failure this feature prevents). Pass 120s.
* fix: nbdkitInUse read ss state from correct column + cover it
ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0]
(field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit
read as idle and the SIGTERM path would reap it under a live guest. Add tests
exercising a real connected unix socket, an absent socket, and TCP transport.
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
* docs: add Containers section with EKS and ECR service pages
New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks.
* docs(eks): rework Terraform workbooks into a capability ladder
Rework the three EKS workbooks into a coherent ladder and publish them to
the Containers section. All tiers are single- or three-node capable via
node_desired_size and serve a shared Spinifex-themed demo app from ECR,
replacing nginxdemos/hello.
- demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/
region + hit counter; optional /data persistence), distroless image.
- eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort.
- eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer-
controller addon + ACM, public/private VPC with NAT, managed-ingress=false.
- eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from
a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access
entry RBAC, HTTPS Ingress carried over.
Pending live e2e verification (image build/push, apply, ALB + PVC bind).
* fix(ecr): implement ListTagsForResource (empty tags) for client parity
The AWS provider calls ecr:ListTagsForResource on every repository read, which
returned 501 NotImplementedException and broke 'tofu apply' of any workbook that
creates an aws_ecr_repository. Implement it to return an empty tag set (resource
tagging stays a deferred feature) so reads succeed. Drop the tags block from the
three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed
TagResource on create.
* feat(eks): pull from internal ECR via kubelet credential provider
Worker nodes could not pull custom images from the internal Spinifex ECR
registry: the eks-node had no DNS for the registry host, no containerd CA
trust, no port rewrite to :9999, and no pull credentials. Only public
images worked.
Add cmd/ecr-credential-provider, a kubelet exec credential provider that
fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken
against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic
auth for the registry host. Thread the gateway URL, account, region, and
internal suffix into the agent userdata, which now writes the gateway CA,
a registries.yaml (CA trust + :9999 endpoint), the kubelet
credential-provider config and config.yaml.d drop-in, and an /etc/hosts
entry mapping the registry host to the gateway IP. Ship the provider binary
in the eks-node AMI.
Also remove a stray paragraph from the admin-cli README.
* fix: dial EKS ECR registry mirror by gateway IP
cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on
every boot, wiping the registry-host mapping appended via bootcmd, so worker
nodes could not resolve the internal ECR host and pulls failed with no such
host. Point the k3s registries.yaml mirror endpoint and tls config at the
gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image
ref port-less and dropping the DNS leg until northstar lands. Fall back to the
hostname endpoint when the gateway is not an IP.
* fix: attach node-role instance profile to EKS workers for ECR
EKS workers launched with no IAM instance profile, so IMDS served no role and
the ECR credential provider could not mint a registry token, leaving internal
ECR pulls failing with 401. Real EKS implicitly creates an instance profile for
the node role; do the same here. Wire a KV-backed IAM service into the EKS deps
and, at worker launch, find-or-create an instance profile named for the node
role, attach the role, and pass it to RunInstances so IMDS exposes the role to
the credential provider.
* feat: use mulga logo in EKS demo app
* feat: add ACM certificate tag support (List/Add/RemoveTags)
Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate.
* test: cover ACM tag gateway and NATS wrapper paths
* feat: support EKS nodegroup tags (Tag/Untag/ListTags)
Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException.
* fix: provision EKS ALB ingress via node credentials
The aws-load-balancer-controller addon could not provision an ALB. Three fixes:
- TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own
BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE,
falling back to the Go system trust pool and rejecting the gateway cert as
'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool.
- Credentials: an empty AWS_ROLE_ARN beside the web-identity token file
half-arms the SDK web-identity provider, which then signs anonymously
(MissingAuthenticationToken). Inject the IRSA env only when a service-account
role ARN is staged; otherwise omit it so the default chain falls back to the
node instance-profile credentials served over IMDS.
- Authorization: the node role lacked the ec2/elbv2 actions the controller
needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController
managed policy to the node role (CreatePolicy + AttachRolePolicy).
* fix: add force_delete to EKS workbook demo ECR repos
tofu destroy of an EKS workbook left the demo ECR repository behind: it
still holds the pushed demo image, so DeleteRepository (force=false)
returns RepositoryNotEmptyException. Add force_delete = true to each
workbook's aws_ecr_repository.demo so teardown removes the repo with its
images.
* fix: force-delete EKS control-plane ENIs on teardown
TerminateK3sServerVM deleted the CP ENI with a plain force=false
DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the
record's Status/InstanceId/AttachmentId are set, so when the K3s VM is
gone but the ENI record's attachment fields were never cleared the delete
returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper
re-drove the same non-idempotent call every 90s, wedging the cluster in
DELETING.
Teardown owns its CP ENIs and has already terminated the VM, so the
attachment is authoritatively dead. Replace the InUse-retry budget loop
with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the
stale attachment fields, then DeleteNetworkInterface passes the in-use
guard. Both calls tolerate an already-gone ENI (NotFound) so a race with
the async instance-terminate cascade resolves to idempotent success.
purgeClusterInfra's private-endpoint ENI path reuses the same helper.
* fix: detach+delete worker ENIs on EKS nodegroup teardown
A terminated nodegroup worker could leave its primary ENI in
spinifex-vpc-enis as in-use/attached, and a dangling ENI throws
DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two
gaps:
TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped
any worker owned by another node, stranding both the VM and its ENI on
multi-node clusters. Route the terminate per-instance via ec2.cmd.<id>
(mirroring the gateway TerminateInstances path) so the owning daemon runs
the full teardown — detach + force-delete the primary ENI and clear its
record. No responder / NotFound payload is treated as already-gone so a
retried DeleteNodegroup stays idempotent.
deleteInstanceENI (stopped-instance terminate) issued a force=false
DeleteNetworkInterface with no prior detach: a stale in-use attachment
made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind
after the instance was already written to the terminated bucket. Detach
first, then delete, tolerating NotFound.
* fix: heal volume State drift on detach/terminate teardown
A detach/terminate could leave a volume's State empty instead of
'available'. DeleteVolume gated on State=='available' exactly, so it
returned VolumeInUse for an unattached volume nothing was using, leaving
it undeletable and blocking stack teardown. DescribeVolumes cosmetically
rendered an empty State as 'available', masking the drift.
- DeleteVolume now treats an empty State with no attachment as deletable
(an attached volume still returns VolumeInUse).
- DescribeVolumes derives an empty State from the attachment (attached ->
in-use, unattached -> available) instead of blindly rendering
available, so an empty-but-attached volume is no longer hidden.
- UpdateVolumeState normalizes an unattached volume with an empty state to
available, so a detach writeback that omits the state cannot strand the
volume in drift.
* fix: break all SG cross-references on EKS teardown
DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked
only ingress (IpPermissions), never egress (IpPermissionsEgress), so a
cross-reference via an egress rule kept the SG pinned on
DependencyViolation. It also resolved only the two named cluster SGs, so
any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG,
the private-endpoint SG, user/TF SGs — was never revoked and pinned the
referenced SG on delete, leaving the VPC stuck.
Enumerate every non-default SG in the VPC and revoke BOTH ingress and
egress on each before deleting, breaking any cycle order-independently,
then delete only the cluster-owned SGs (referrer SGs are deleted by their
own teardown). The default SG is left untouched. Since DeleteClusterSGs
runs against both the customer VPC and the managed control-plane VPC, the
broadened sweep covers both teardown paths.
* feat: support TargetGroup ProtocolVersion in elbv2
CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target
groups to HTTP1) and DescribeTargetGroups/Create output return it. Without
this the aws-load-balancer-controller sees a perpetual immutable-field
mismatch (desired HTTP1 vs empty) and recreates the target group every
reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup
reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it
omitted to match AWS.
* fix: omit IRSA env from ebs-csi addon when no role staged
The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}}
plus the web-identity token volume. With no service-account role staged the
placeholder rendered empty, half-arming the SDK web-identity provider, which
panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs
Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}}
markers like the LBC bundle so the IRSA env and token volume are omitted when
no role is staged, falling back to node instance-profile credentials over IMDS.
The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes.
Requires an eks-node AMI rebake to take effect.
* docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI
Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup
with host routing (app./argocd. on a wildcard cert) instead of one ALB per
ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the
access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers
need (Spinifex wires controller creds at the node level, not IRSA) and an
ignore_changes block for the cluster's access_config and
vpc_config security_group_ids, which Spinifex DescribeCluster does not echo
back, so the provider stops trying to replace the cluster on every plan.
* fix: stop addon-sync substituting IRSA markers named in comments
The ebs-csi controller manifest's header comment spelled out the literal
{{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync
matched each marker by substring, so the comment lines were replaced with the
env/mount blocks, injecting a bare YAML sequence between two documents. K3s'
auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal
array into unstructured), so the ebs-csi Addon CR never applied and the EKS
addon hung in CREATING. Reword the comment to not contain the literal tokens,
and anchor the marker substitution to lines that are exactly the marker
(whitespace aside) so a future comment mentioning a marker can't retrigger it.
* fix: disable K3s local-storage on EKS clusters in parity mode
In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server
disabled traefik and servicelb but left local-storage enabled. Its local-path
provisioner ships a StorageClass marked default, which races the EBS CSI
ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind
nondeterministically. EKS has no local-path, so disable local-storage too and
let ebs-gp3 be the sole default. The built-in-ingress path is unchanged.
* fix: re-encrypt HTTPS target groups in haproxy ALB backends
An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which
serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's
own check then spoke HTTP to a TLS port, failed, and marked the server DOWN,
returning 503 — even though the lb-agent's separate HTTPS health probe reported
the target healthy, so describe-target-health masked the fault.
Emit check-ssl ssl verify none on the server line for HTTPS target groups so
both the check and proxied traffic re-encrypt to the backend.
* fix: reconnect wedged QMP client on next command
A timed-out QMP decode under host contention left the cached client's
shared json.Decoder mid-message, wedging every later command and breaking
EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead
on encode/decode error and redial the retained socket path (re-running the
qmp_capabilities handshake in place) on the next send.
* fix: round-trip security-group rule descriptions
SG ingress/egress rules dropped the per-rule Description on store and
describe. The AWS Load Balancer Controller tags the worker-node SG rules it
manages with a UserIdGroupPair description and identifies them by that value
to revoke them on teardown. Without the round-trip the controller cannot
match its rules, never revokes them, and the backend security group stays
referenced — DeleteSecurityGroup returns DependencyViolation forever,
hanging the ingress finalizer and tofu destroy with orphaned security groups.
Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not
identity) and render it back through describe.
* fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown
In the managed control-plane VPC topology, launchNodegroupInfra creates the
cluster control-plane and nodegroup security groups in the customer VPC for
worker<->control-plane networking, but purgeClusterInfra only reclaimed the
security groups in the managed CP VPC. The customer-VPC SGs orphaned,
cross-referenced each other, and pinned the customer VPC with
DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the
managed-CP-VPC branch too, matching the legacy customer-VPC branch.
* fix: reclaim ALB EIP when system VM already gone on terminate
TerminateSystemInstance only released an internet-facing ALB's Elastic IP on
the local path, where the VM is still present in the manager. When the VM was
already gone (health-reaped, crashed, or raced by ENI teardown), it took the
remote/no-owner path and returned idempotent success without reconciling the
EIP, leaking the wan-pool address still associated to a deleted ENI.
Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic
scan that disassociates then releases any EIP recorded against an instance, and
call it as a backstop in TerminateSystemInstance for both branches.
* fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook
The demo Application carried no resources finalizer, so tofu destroy removed
the Application CR without pruning the synced PVC. The PVC survived, the cluster
was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the
resources-finalizer.argocd.argoproj.io finalizer so deleting the Application
cascades to the PVC and ebs-csi reaps the volume before cluster teardown.
* feat: raw-IP access + documented Argo CD creds in eks-gitops workbook
The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser
without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add
a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and
https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password
command as a tofu output and document the credential + IP URLs in the README.
* fix: reclaim ALB EIP on the owning-node terminate path
The EIP backstop lived only in TerminateSystemInstance, but the ALB
delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal
directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's
cached allocation which is often empty. Extract reclaimSystemInstanceEIP
(authoritative KV reconcile via ReleaseAddressByInstanceID) and call it
from terminateSystemInstanceLocal as well as the no-owner backstop, so an
internet-facing ALB's EIP is released on every terminate route.
* chore(eks): drop coredns from the managed addon catalog
Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as
a managed addon — there is no bundled coredns manifest, so CreateAddon
coredns would stage but fail VM-side render. Remove the phantom catalog
entry so DescribeAddonVersions and CreateAddon no longer advertise it;
repoint the addon tests that used coredns as their sample to argocd.
* feat(ui): group EKS and ECR under a Containers nav section
Fold the separate EKS and ECR sidebar groups into one Containers group
with EKS (clusters) and ECR (repositories) entries.
* fix(ui): satisfy strict null checks in ECR component tests
Index mock.calls via optional chaining and narrow the getAllByRole result
before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix.
* refactor(eks): hide internal fixture addons from the public catalog
Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays
creatable (lookupAddon finds it, so the delivery-transport e2e still drives
it) but no longer surfaces in the unfiltered DescribeAddonVersions listing.
* refactor(eks): remove K3s built-in ingress; LBC-only
Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik,
servicelb, and local-storage are now always disabled on EKS nodes (AWS
parity — Service type=LoadBalancer and Ingress are the Load Balancer
Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd
traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in
the state reporter.
* fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation
Parallelize the independent subnet-create and route-table-association loops
in the create-VPC mutation with Promise.all (each task records its resource
in the rollback list before any further call), and switch the CIDR-validation
regex to named capture groups. Clears the no-await-in-loop and
prefer-named-capture-group lint errors.
* fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal
The ECR registry is served on the gateway port (:9999), but proxyEndpoint and
repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and
failed. Derive the port from the awsgw listen host and append it to the registry
host (omitted for 443). Also rename the default internal DNS suffix.
* docs: drop verbose ECR :9999 host note
* docs: fix literal-asterisk emphasis (single-* not rendered)
* docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust
* feat: default ECR registry URIs to the gateway host (no-DNS interim)
The registry authenticates by token, not hostname, so the per-account
<acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port
(already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need
no DNS; the parity host still works server-side where DNS exists.
* fix: serve ECR registry on a DNS-free gateway host end to end
Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW
bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable
host:9999 carried in the server cert SANs rather than the unresolvable
per-account parity name. Docker push then dials it directly with no DNS.
Mirror the same host on EKS workers: the registries.yaml mirror keys and the
credential provider matchImages now cover the port-less parity name, the
port-ful parity name, and the gateway IP:9999, all redirecting to the dialed
gateway endpoint. A port-ful pull from the API-returned repository_url matches
the mirror instead of bypassing it into a failed DNS lookup.
Net: docker push host, pod image host, and the API's repository_url are the
same value, with no DNS or /etc/hosts.
* docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS)
* docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS)
* docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks
* feat(ecr): return OperationNotSupportedException for image-scanning actions
The ECR image-scan control-plane actions resolved to the generic
NotImplemented 501 stub, indistinguishable from unbuilt features and
surfaced as a server fault. mulga runs no vulnerability scanner, so the
scan surface now returns OperationNotSupportedException (400):
- awserrors: add OperationNotSupportedException (400) ECR code.
- ecrapi: ScanningNotSupported handler; repoint the four image-scan
actions and register the three registry-scanning actions at it.
- spinifexui: Scan tab on the repository detail page renders a static
'scanning not supported' panel rather than a request that errors.
* refactor(config): default internal DNS suffix to spinifex.internal
Rename the AWS-parity internal DNS suffix from mulga.internal to
spinifex.internal across the config default, cert SAN builder, ECR
endpoint/audience construction, and their tests. Single runtime knob is
config.DefaultAWSInternalSuffix; no DNS backend exists yet, so this is a
naming change with no functional impact.
The ECR auth bridge minted ES256 tokens from a key that was created once and
never rotated. Add a background rotator (ecr-v1 Q3): mint a fresh key every 30
days, retain the previous keys for 24h so in-flight 12h tokens stay verifiable,
then prune. The active key is now the newest by KV creation time (not the
lexically-greatest kid) so a minted key takes over and all nodes converge. The
Issuer and Verifier became hot-swappable so the rotator updates the live
gateway as keys roll.
* feat: bake aws-ebs-csi-driver addon + mdev by-id bridge
Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and
ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet,
default gp3 StorageClass) under the eks-node image.
The node plugin locates EBS volumes via /dev/disk/by-id, which busybox
mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that
symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk
serial, plus a boot oneshot that points the kernel hotplug helper at
mdev and coldplugs volumes present at boot.
* test: e2e EBS CSI volume — PVC bind + mount + reschedule survival
Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver
managed addon bound to an IRSA role, provisions a default gp3 PVC + pod,
and asserts a marker written by the first pod survives a reschedule onto a
fresh pod backed by the same PVC. Exercises the full data path:
CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to
the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount.
* feat: eks-node provider-id + topology/instance-type labels for EBS CSI
mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in
from IMDS before K3s starts: kubelet provider-id plus the region, zone
and instance-type node labels. The aws-ebs-csi-driver node plugin reads
this metadata for NodeGetInfo (instance id from providerID, region/zone
from topology labels, instance type for the volume-limit family lookup).
k3s runs no AWS cloud-provider so none of it is set otherwise, which left
the node plugin's metadata init failing, NodeGetInfo panicking in
IsNitroInstanceType, and the CSINode carrying no topology key. Wire the
service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and
chmod it in setup.
* test: e2e EBS CSI controller IRSA policy + failure diagnostics
awsgw enforces IAM on assumed-role EC2 calls, so the controller's
CreateVolume/AttachVolume need a permission policy on the IRSA role.
Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and
grant nothing) covering the EBS CSI action set; CreateVolume then
succeeds. Add an OnFailure dump of the CSI control path (provisioner and
plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume
subtest so provisioning/attach failures are diagnosable post-teardown.
* feat: EBS CSI obeys control-plane taint; e2e worker nodegroup
Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI
controller + node manifests so the driver runs only on customer worker
nodes, never the system control-plane VM. The e2e creates a 1-node
worker nodegroup and the writer/reader pods drop their CP tolerations so
the customer-owned volume attaches in the customer space.
* fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback
virtio-blk-pci hot-plug bus was derived from the AWS device-name letter
(/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that
map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug —
and the attach hung. A failed attach also left the per-volume iothread object
behind, so every retry died on object-add 'duplicate property'.
nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live
query-block state, independent of the device name. delIothreadBestEffort
removes the iothread on every attach rollback path.
* fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link
Alpine eks-node runs busybox mdev (no eudev), which stops at the first
matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after
the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on
hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the
upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>.
Swap the stock vd* rule in place to route through mulga-ebs-byid, which now
delegates to persistent-storage (preserving stock links) before minting the
nvme-style link the node plugin expects.
* test: EKS e2e — NAT Gateway egress for private worker nodegroup
The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS:
an IGW serves only public-IP instances), so an IGW-only fixture left the worker
unable to reach the public NLB endpoint or pull public CSI images — the
nodegroup never joined.
createWorkerEgress now builds the real topology: an IGW-fronted public subnet
hosting a NAT Gateway, with the worker subnet default-routed to it. Associating
the worker route table triggers the subnet-wide SNAT egress reroute. Teardown
drains the NAT Gateway before releasing the public subnet + EIP it pins.
The OCI /v2 registry auto-created a repository on first push (ensureRepo).
AWS ECR does not: a push or PutImage to a repository that was never created
with CreateRepository is rejected. Replace ensureRepo with requireRepo, which
returns OCI NAME_UNKNOWN on the data plane and RepositoryNotFoundException on
the JSON PutImage path. Enforcement sits at the single push choke point
(startUpload + StoreManifest).
* test(e2e): add ECR end-to-end suite (control plane + docker/crane/skopeo)
New tests/e2e/ecr/ suite plus harness support:
- harness: ECR SDK client, control-plane helpers (repo CRUD, get-login-password,
describe-images), and docker/crane/skopeo CLI wrappers that skip when the
binary or registry-host resolution is absent.
- suite: control-plane CRUD + policy + lifecycle round-trip; docker login/push/
pull; crane append/pull and skopeo copy exercising the Bearer /v2/token flow.
- registered e2e-ecr in the Makefile + service manifest.
Predastore capacity/load leg is a skipped placeholder pending mulga-siv-359/356.
* test(e2e): carry awsgw port in ECR registry host for dev clusters
Dev clusters bind awsgw on a non-standard port (e.g. 9999); production uses
:443. ECRRegistryHost now appends SPINIFEX_ECR_PORT when set so docker/crane/
skopeo address the right endpoint, and RequireRegistryResolves strips the port
before DNS lookup.
getManifest read only the first Accept header line via r.Header.Get, so
clients that send each acceptable manifest media type as a separate Accept
header (skopeo) never matched the stored type and got 406 MANIFEST_INVALID.
Join all Accept header values before negotiation.
Add the Docker Registry v2 token endpoint (GET /v2/token) and switch the
/v2 401 challenge from Basic to a Bearer realm pointing at it. The endpoint
verifies the caller's existing AWS:<jwt> credential and re-mints a fresh
full-TTL token, providing the token-refresh path for OCI clients (docker,
crane, skopeo). The auth bridge still accepts Basic AWS:<jwt> directly.
Stamp the dash-stripped volume-id (vol+17hex = 20 bytes, the virtio-blk
serial limit) as the hotplug disk serial so the EBS CSI node plugin can
locate the device via /dev/disk/by-id and match lsblk SERIAL.
* feat: ECR background lifecycle expiry sweep
Add a gateway-side periodic sweep (LifecycleSweeper) that applies each repo's
stored lifecycle policy: it evaluates the shared rule engine against the repo's
current images and deletes the expired set through the existing registry GC
reclaim path (DeleteImage -> reclaimManifest). It runs in the gateway because
only the gateway holds a Registry (object store + meta); the daemon owns the KV
but cannot reclaim predastore bytes. Launched on the STS janitor's lifetime
context in awsgw, enumerating ACTIVE IAM accounts.
Per repo: skip when no policy; evaluate; then before deleting, re-read the
current tag set and skip any image whose tags changed since evaluation
(optimistic-concurrency guard against concurrent push/re-tag). Cross-repo blob
mount safety is provided by reclaimManifest/referencedDigests. Per-item errors
are logged and skipped so one bad repo never aborts the cycle.
* test: cover lifecycle sweeper Run loop and active-account filter
Add tests for LifecycleSweeper.Run ticker loop, NewLifecycleSweeper
interval fallback, and sweepRepo error branches. Extract accountLister
interface so activeAccountIDs is unit-testable. Raises diff coverage
on the sweep changes from 53.4% to 76.9%.
* feat: ECR lifecycle policies (CRUD + evaluation engine + preview)
Implement PutLifecyclePolicy/GetLifecyclePolicy/DeleteLifecyclePolicy and
StartLifecyclePolicyPreview/GetLifecyclePolicyPreview, replacing the
NotImplemented stubs. A pure rule-evaluation engine parses the AWS
lifecycle-policy JSON and, given a repo's images, returns the ordered set that
would expire; each image is claimed by at most one rule (lowest rulePriority
first). Preview is served synchronously and deletes nothing.
Lifecycle-policy documents persist as a per-repo KV doc (repos/<name>/lifecycle)
over the ecr.lifecycle.* NATS subjects, mirroring the repo-policy storage stack,
and cascade on DeleteRepository. The console gains a Lifecycle tab with a policy
editor and a preview table over the saved policy.
The background expiry sweep that applies the engine and deletes images via GC is
deferred to mulga-siv-387.
* feat: client-side JSON validation in ECR policy editors
Swap the raw Textarea in the repository-policy and lifecycle-policy editors for
the shared JsonEditor (blur-time pretty-print) and gate Save on isValidJson,
showing an inline error for malformed JSON — matching the IAM/EKS editors. The
backend remains the source of truth for schema validation.
Honor, persist and echo imageTagMutability on CreateRepository and
DescribeRepositories instead of the hardcoded MUTABLE, and implement
PutImageTagMutability to flip a repository between MUTABLE and IMMUTABLE.
Enforce immutability at the shared StoreManifest chokepoint: on an
IMMUTABLE repo, re-tagging an existing tag onto a different digest is
rejected before any write (OCI 409 TAG_IMMUTABLE; JSON PutImage maps to
ImageTagAlreadyExistsException). Same-digest re-pushes, new tags and
digest references are still allowed.
Console: migrate the repository detail page to Summary/Images/Permissions
tabs and add a tag-immutability toggle on the Summary tab.
DELETE /v2/{name}/manifests/{digest} was a 202 no-op. Now delete-by-digest
removes the manifest record, deletes the manifest object, and sweeps the
account-wide blob pool: each child blob no longer referenced by any live
manifest is reclaimed, while shared blobs survive.
- DeleteImage (digest path) captures ChildDigests, deletes the manifest object,
then reclaimManifest() drops orphaned blobs via referencedDigests() (a
mark-and-sweep over the live manifest set across the account's repos).
- OCI deleteManifest by-digest wired to DeleteImage (real work + 202; 404 on
unknown). Tag delete stays KV-only so untagged images persist (AWS-faithful).
- Predastore reclaim is best-effort: a failure leaks bytes but never undoes the
logical delete.
Serve the five image-management JSON 1.1 actions inline in ECR_Request,
reusing the gateway-side OCI Registry for manifest bytes:
- ListImages / DescribeImages enumerate images by tag + digest (KV meta).
- BatchGetImage returns verbatim manifest bytes with Q14 partial-failure
semantics (cap 100, digest wins over tag, 200 + failures array).
- PutImage stores a manifest + tag via a shared StoreManifest core that the
OCI manifest PUT now delegates to.
- BatchDeleteImage removes tag pointers / manifest meta (blob GC deferred).
Adds DeleteManifestMeta across the MetaStore relay chain and ECR image
error codes.
Add the two missing repository-lifecycle actions, served inline in
ECR_Request (like DescribeRepositories) since both return the repository
ARN/URI built from the gateway region + internal suffix.
CreateRepository validates the name, guards cross-account registryId,
rejects an existing repo (RepositoryAlreadyExistsException), and writes
the per-account KV meta (predastore bucket stays lazy on first push).
DeleteRepository rejects a non-empty repo without force
(RepositoryNotEmptyException) and cascades the KV records (meta, policy,
tags, manifests). Predastore blob GC is deferred to a separate pass.
Adds MetaStore DeleteRepo + ListManifests with the matching daemon-relayed
subjects, service methods, and NATS client, plus two new awserrors codes.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
CreateTags/DeleteTags wrote only the standalone tag store, which
DescribeImages does not read (it reads AMI metadata), so tag mutations on
an AMI were invisible — DeleteTags appeared to be a silent no-op. Mirror
tag add/remove into the owning AMI config (ownership-checked; absent or
foreign AMIs skipped) so DescribeImages reflects them, matching the
existing subnet/vpc mirror.
* feat(ecr): auth bridge + GetAuthorizationToken JWT (Sprint 2c)
Mint a self-contained ES256 JWT from GetAuthorizationToken and verify it on
the /v2 OCI registry via a new ECRAuthBridgeMiddleware (Bearer | Basic AWS:jwt).
Signing keys live in the cluster-replicated awsgw-keys KV bucket under
jwt-signing/{kid}, encrypted with the IAM master key.
The registry is now scoped per-request to the token's account (replacing the
fixed bootstrap account); bucket-ensure is cached per account. Cross-account
token/host mismatch is rejected 403; unauthenticated /v2 access gets a 401
Bearer challenge.
* fix(ecr): use Basic auth challenge on /v2 (not Bearer token-endpoint flow)
ECR authenticates the OCI registry with Basic AWS:<token>; a Bearer realm
made docker chase an unimplemented /v2/token endpoint and fail login.
* feat(ui): inline EKS cluster IAM role creation in wizard
* feat(ui): show trust policy + attached managed policy in EKS role dialog
Render the trust policy with PolicyDocumentViewer (pretty-printed, safe
fallback) instead of a raw read-only textarea, matching the IAM detail
pages. Surface the attached AmazonEKSClusterPolicy ARN so users can see
exactly which permissions the cluster role grants.
Josh Siviour