25 Commits

Author	SHA1	Message	Date
Josh Siviour	b5879f44ff	ADR-0001 package-boundary migration checkpoint (#1190) ... * refactor: move telemetry file naming to contract * docs: record telemetry contract move * refactor: move aws xml response helpers * docs: record aws xml helper move * refactor: move aws tag projections to foundation * docs: record aws tag projection move * refactor: move account key to state foundation * docs: record account key state move * refactor: move host privilege escalation to runtime * docs: record host privilege move * chore: resolve lint findings from package-boundary moves * refactor: move process supervision helpers to runtime host * docs: record process supervision move * refactor: move nbd endpoint helpers to compute runtime * docs: record nbd endpoint move * refactor: move image catalogue to operator * docs: record image catalogue move * refactor: move resource id generation to aws foundation * docs: record resource id move * refactor: move predastore role to runtime roles * docs: point service manifest at acm domain * docs: record predastore role move * refactor: move northstar role to runtime roles * docs: record northstar role move * refactor: move awsgw role to runtime roles * docs: record awsgw role move * refactor: move error payload helpers to aws errors * docs: record error payload move * refactor: move account identifiers to aws identifiers * docs: record account identifier move * refactor: move NATS transport to messaging foundation * docs: record NATS transport move * refactor: move system instance contract to EC2 domain * docs: record system instance move * refactor: move system VPC builder to network domain * docs: record system VPC move * refactor: move instance metadata service to EC2 domain * docs: record guest metadata move and dependency-rule debt * refactor: move DNS control plane to DNS domain * docs: record DNS domain move * refactor: move base zone bootstrap to DNS domain * docs: record base zone bootstrap move * refactor: replace SystemAccountID with GlobalAccountID * docs: record SystemAccountID removal * refactor: extract CA key-pair loader to foundation pki * docs: record foundation pki extraction * refactor: move tenant CA paths to ACM domain * docs: record tenant CA path move * refactor: move IPsec certificate handling to network host * docs: record IPsec certificate move * refactor: move Ochre serving lifecycle to the Ochre domain handlers/bedrock becomes domains/ochre. Ochre is the platform service; Bedrock names only its AWS API surface. Persisted and observable identifiers keep their bedrock names. * docs: record Ochre domain move * refactor: move Ochre vector store to the Ochre domain handlers/ochrevector becomes domains/ochre/vector. Callers import it as ochrevector because vector is a common local name in the gateway. * docs: record Ochre vector store move * refactor: move Hugging Face Hub client to the Ochre domain gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is the spx admin ochre weights pull. * docs: record Hugging Face Hub client move and Ochre gateway debt * refactor: move EC2 tag mutation helpers to the EC2 domain utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag mutation behaviour, and the tags resource package cannot host it without a resource-to-resource import cycle. * docs: record EC2 tag mutation move and migrate split finding * refactor: move network packages under the network domain Move spinifex/network/* to spinifex/domains/network/* as one enclosing move so the L0-L5 layer invariants keep classifying every package from a single root. Package names and dependency direction are unchanged; invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and the repo-root walk in the IMDS netns unit test follow the new path. * docs: record network domain move * refactor: move standalone EC2 resource packages to the EC2 domain Move handlers/ec2/{account,key,launchtemplate,placementgroup, spotinstance,volumestate} to domains/ec2/<resource>. Each package is named after its directory and callers import it as ec2<resource>; volumestate already carried its directory name. * docs: record standalone EC2 resource moves * refactor: move EC2 VPC networking packages to the EC2 domain Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to domains/ec2/<resource>, each named after its directory and imported as ec2<resource>. The reconcile drift-watch source scan now matches the ec2<resource> bucket qualifiers; its rule is unchanged. * docs: record EC2 VPC networking moves * refactor: move EC2 compute and storage packages to the EC2 domain Move handlers/ec2/{instance,volume,snapshot,image,tags} to domains/ec2/<resource>, each named after its directory and imported as ec2<resource>, which empties handlers/ec2. The daemon's e2e suite selection now lists the moved EC2 resource paths in place of handlers/ec2, so EC2-only changes select the daemon suites again. * docs: record EC2 compute and storage moves * refactor: move ECR credential provider to the EKS agent tree Move the kubelet exec credential provider implementation from cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing one Main(in, out) entry point. The binary keeps its name and FIPS boot import as a thin main. Moving out of cmd/ brings the code under gosec; the one G101 hit on the kubelet API version constant is annotated. * docs: record ECR credential provider move * refactor: move EKS gateway publish helper to the EKS agent tree Move the eks-gateway-publish implementation to agents/eks/gatewaypublish behind Run(ctx, Config) error. The binary keeps flag definitions, environment defaults, parsing, its log prefix and exit code; each former fatal site returns an error with the same text. The retry wait now also honours context cancellation, which the binary never triggers. * docs: record EKS gateway publish move * refactor: move EKS gateway fetch helper to the EKS agent tree Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary keeps flag definitions, environment defaults, parsing, its log prefix and exit code; each former fatal site returns an error with the same text. * docs: record EKS gateway fetch move * refactor: move EKS webhook cert helper to the EKS agent tree cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit; agents/eks/webhookcert owns Config and Run. Output, error text, file modes and exit codes are unchanged. * docs: record EKS webhook cert move * refactor: move EKS konnectivity cert helper to the EKS agent tree cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the exit; agents/eks/konnectivitycert owns Config and Run. Output, error text, file modes and exit codes are unchanged. * docs: record EKS konnectivity cert move * refactor: move EKS token webhook to the EKS agent tree cmd/eks-token-webhook keeps the -addr flag, environment-derived config, logging setup and exits; agents/eks/tokenwebhook owns Config and Run. The handlers/eks TokenReview result type stays as recorded temporary debt. * docs: record EKS token webhook move * refactor: move ECS agent runtime and credentials packages to the ECS agent tree * docs: record ECS agent subpackage move * refactor: move ECS agent implementation to the ECS agent tree cmd/ecs-agent keeps process lifecycle: signal context, the separate New and Run calls, both failure log lines, the exit and the ldflags-set build version, now passed in through Config.AgentVersion. agents/ecs/agent owns Config, LoadConfig and the agent. * docs: record ECS agent move * refactor: move RDS guest agent to agents/rds/agent * docs: record RDS guest agent move * refactor: move installer packages to operator/installer * docs: record installer move * refactor: move spx command tree to operator/cli Moving the command tree out of cmd/ brings it under errcheck and gosec. Flag and config wiring now fails loudly instead of silently dropping a binding, listing commands fail when they cannot write their table, and the formation server's unauthenticated TLS is annotated as a known gap. * docs: record spx command tree move * chore: accept known vulnerabilities with an expiry The vulnerability scan failed preflight on a containerd advisory that dev already carries and that is deferred to a separate dependency update. The gate now passes only reachable findings listed with an expiry date, fails on any new one or on an expired acceptance, and warns when an acceptance is no longer needed. * refactor: move EC2 account action adapters to domains/ec2/awsapi * refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi * refactor: move EC2 key action adapters to domains/ec2/awsapi * refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi * refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi * docs: record EC2 standalone action adapter moves * refactor: move EC2 zone action adapters to domains/ec2/awsapi * refactor: move EC2 vpc action adapters to domains/ec2/awsapi * refactor: move EC2 eigw action adapters to domains/ec2/awsapi * refactor: move EC2 eip action adapters to domains/ec2/awsapi * refactor: move EC2 igw action adapters to domains/ec2/awsapi * refactor: move EC2 natgw action adapters to domains/ec2/awsapi * refactor: move EC2 routetable action adapters to domains/ec2/awsapi * docs: record EC2 VPC networking action adapter moves * refactor: move EC2 image action adapters to domains/ec2/awsapi * refactor: move EC2 snapshot action adapters to domains/ec2/awsapi * refactor: move EC2 volume action adapters to domains/ec2/awsapi * refactor: move EC2 tags action adapters to domains/ec2/awsapi * docs: record EC2 storage and tag action adapter moves * refactor: move EC2 resource authorization scopes to domains/ec2/awsapi * refactor: move EC2 idempotency adapter to domains/ec2/awsapi * refactor: move EC2 instance action adapters to domains/ec2/awsapi * refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi * docs: record EC2 instance action adapter moves * refactor: move AWS REST-JSON router to ingress/aws/rest * refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit * docs: record AWS ingress helper moves and tooling conformance * docs: correct historical lifecycle references * test: characterize ECR control-plane dispatch before registration seam * feat: add AWS ingress service registration contract * docs: record ingress registration phases 0 and 1 * refactor: route registered AWS services through the ingress dispatch registry The gateway now consumes ingress/aws/dispatch. A registered service is accepted by the auth gate, resolves its own action label, renders errors in its declared envelope and is dispatched with an authenticated Invocation (account, region, policy authorisation and a lazy verified actor). Every other service keeps the existing switch, now named as the legacy path. The awsgw role wires an empty registry and refuses to start if a registered name is also served by the legacy table. No service is registered yet, so behaviour for every existing service is unchanged. * docs: record ingress dispatch registry consumption * refactor: register the ECR control plane from its domain domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the ingress dispatch registry: action resolution, resource scope, authorisation through the invocation, the composed capabilities and the success writer all live with the domain. The awsgw role is the only serving composition point; the gateway no longer imports the ECR API package and ECR leaves the legacy service table, JSON error table and dispatch switch. The OCI /v2 route is unchanged. Operation coverage reads ECR's declared inventory, which the registration uses verbatim; coverage.json is byte-identical. * docs: record ECR control-plane registration * fix: compose the ECR control plane in the integration harness The harness stopped wiring ECR's composed capabilities when the authorization-token action was composed, so every integration test that creates a repository or mints a token got ServerInternal. It now composes ECR as the awsgw role does, sharing the OCI registry, meta store and token issuer the /v2 data plane uses. * docs: record ingress pipeline deferral * refactor: extract AWS error envelope rendering to ingress ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header detector and the two error content types. Each shape keeps its own builder. The gateway error writers, ErrorHandler and the legacy per-service envelope selection stay in gateway and call the new package; gateway/eks renders its JSON errors through it directly. A 140-case status/header/body golden comparison across services and error writers was identical before and after. * docs: record error envelope extraction * refactor: move AWS service selection into ingress dispatch Selector answers served, action, error protocol, registered entry and startup overlap. The gateway supplies its legacy tables through the dispatch.Legacy interface; legacy dispatch and invocation building stay in gateway. * docs: record ingress service selection move * test: characterize ACM control-plane dispatch before registration Pins full-pipeline error envelopes, a successful action, target parsing agreement between dispatch and telemetry, authorization before the NATS call, 413 and 503 paths, throttle/trace/audit action and the inventory. * refactor: register ACM control plane from its domain domains/acm/awsapi registers the acm JSON 1.1 surface through the ingress dispatch registry, wired by the awsgw role. The gateway drops ACM_Request, the ACM action table and every legacy acm entry; the import invariant now covers each registered domain. * docs: record ACM registration * docs: record ECS, EKS and RDS source inventory and first-resource ranking * docs: record EKS control-plane secrets persisted in the cluster record * chore: drop containerd advisory acceptance now that dev ships v2.4.1 * test: characterize RDS DB subnet group lifecycle before extraction * docs: record RDS DB subnet group lifecycle contract * refactor: extract the RDS DB subnet group lifecycle owner domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member subnet validation and the delete in-use guard. handlers/rds keeps AWS validation, projection and paging and binds two consumer-owned capabilities: the owner's Dependants (instances naming the group) and the instance side's subnetGroupPlacement. Stored shape and behaviour are unchanged. * docs: record RDS DB subnet group owner extraction * docs: correct RDS subnet group tagging debt and conformance scope * test: characterize ECS task definition lifecycle before extraction * docs: record ECS task definition lifecycle contract * refactor: extract ECS task definition owner into domains/ecs/taskdefinition Move the task definition record, container types, keys, reference parsing, ARN helpers and the register/deregister/resolve/list logic into a resource owner package. Stored JSON field names and KV keys are unchanged, and revision allocation keeps its existing non-atomic read-then-write. The owner declares its own PortMapping and SystemControl persisted types so it no longer depends on handlers/ecs/bus; the handler translates them onto the assign payload. Task and service launches reach task definitions only through a launch-side taskDefinitionResolver. AWS conversion, the guest projection and tags stay in handlers/ecs, and gateway authz only switches to the moved reference/ARN helpers. List handlers/ecs and the new package under spinifex_daemon so ECS-only changes select e2e-ecs. * docs: record ECS task definition owner extraction * test: characterize EKS access entry lifecycle before extraction * docs: record EKS access entry lifecycle contract * refactor: extract EKS access entry owner into domains/eks/access Move the access entry record, associated policy and scope types, keys, PrincipalARNHash, the policy catalogue and group mapping into a resource owner package. Stored JSON field names and KV keys are unchanged. Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate, Disassociate and Delete carry the existing logic verbatim, and the put, compare-and-set and delete primitives are private. The handler keeps AWS validation, the cluster existence check, error mapping and projection. Token review still reads records through the exported Get, and the cluster prefix purge still sweeps entry keys. List handlers/eks and the new package under spinifex_daemon so EKS-only changes select the EKS suites. * docs: record EKS access entry owner extraction * test: characterize RDS engine catalogue before extraction * fix: return InvalidParameterValue for a missing RDS master password ValidateMasterUserPassword built the empty-password error with a bare errors.New carrying the code only in its text, which awserrors does not resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500 ServerInternal. Build it with awserrors.Errorf like the other branches so the gateway returns 400 InvalidParameterValue, and cover the full gateway to RDS service path with a regression test. * docs: record RDS engine catalogue boundary and evidence * refactor: extract RDS engine catalogue into domains/rds/engine Move engine identity and validation, the per-engine parameter catalogues, effective-parameter resolution, db.* instance classes and engine/orderable selection into an engine package. Behaviour, error codes and ordering are unchanged. The package imports no EC2 package: instance-class memory comes from an immutable Sizing value built from an engine-declared InstanceTypes interface, backed by the fixed instance-type table in handlers/rds. Effective parameters leave as an engine-owned Setting, converted to Parameter in handlers/rds. AWS output building, paging, apply-method mapping, storage limits and the gateway runnable check stay in the adapter path. The RDS agent declares its own EngineCatalog interface and types and receives an engine-backed implementation from cmd/rds-agent, so agent code does not import the catalogue. Cross-engine tests use the two real engines instead of a registered fake. * docs: record RDS engine catalogue owner move * test: characterize RDS DB parameter group lifecycle before extraction * docs: record RDS DB parameter group lifecycle contract * refactor: extract RDS DB parameter group owner Move the parameter group and override records, keys and listing into domains/rds/parametergroup behind an Owner. Delete asks an owner-declared Dependants interface; instance-side resolution reads through its own parameterGroupValues interface. SetOverrides refuses an unknown group so no override key is written without its group. Stored shape, keys, public errors and propagation are unchanged. * docs: record RDS DB parameter group owner extraction * test: characterize EKS add-on lifecycle and guest wire Pin the persisted record and manifest layout, create/update/delete and status-report transitions, the ignored client token and report version, the missing serviceAccountRoleArn PassRole check, and the exact internal add-ons HTTP body and agent TSV output as they stand today. * docs: record EKS add-on lifecycle contract * fix: require iam:PassRole for EKS add-on service account role CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup. * docs: record EKS add-on PassRole check * refactor: extract EKS add-on owner Move the add-on catalogue, record and manifest keys, CAS, status transitions and staging into domains/eks/addon behind an Owner that is the sole writer. The handler keeps AWS validation, error mapping and projection; the cluster reconciler delegates reports through the narrow addonReports interface; node groups call EnsureGPUDevicePlugin. Stored keys, JSON shape, wire types and lifecycle behaviour are unchanged. * docs: record EKS add-on owner extraction * refactor: extract EKS add-on guest wire contract Move the add-on status subject, delivery phase, status report, staged manifest and internal add-ons response into contracts/eks/v1 and use it from the handler, gateway and gatewayfetch, replacing the agent's duplicate. Wire bytes are unchanged, including the internal add-ons body that the SDK renderer emits with Go field names; compatibility tests pin the JSON, the guest printf form and the TSV. * docs: record EKS add-on wire contract extraction * docs: update EKS add-on inventory and remaining edges * docs: draft EKS add-on lifecycle correction design for review * chore: run contracts tests in the default test, coverage and race targets * docs: apply review decisions to EKS add-on lifecycle design * test: characterize EKS add-on v1 compatibility with added fields Pin which owner writes drop unknown record and manifest fields, that reads and deletes ignore or erase them, that one undecodable entry fails a listing, and that v1 wire consumers ignore extra fields. * docs: add EKS add-on executor, bundle, v1 compatibility and release inventories * docs: draft EKS add-on v2 guest contract design for review * docs: qualify historical architecture references * docs: apply review decisions and ADR-0006 to EKS add-on v2 design * fix(eks): bind internal report publishing to the caller's cluster PublishInternal was authorised by IAM policy alone, with its resource built from the body accountId and the URL cluster. The control-plane role grants it on "*", so any control-plane VM session, any tenant holding eks:*, or static system credentials could publish cluster state, bootstrap and add-on reports for another cluster or account. PublishInternal now passes the existing internal-route gate: the caller must be a control-plane instance-role session that is a member of the URL cluster in the body account. The gate and the handler decode the body through one shared decoder, so the account checked is the account the subject is built from. Every unauthorised target returns the same AccessDenied, and invalid account or cluster names are treated as not found. Valid publishes keep their subjects and payload bytes. * fix(eks): bind webhook token review to the caller's cluster WebhookTokenReview took its account from the request body and was authorised by IAM policy alone, with the control-plane role granting it on "*". A control-plane VM, any tenant holding eks:*, or static system credentials could resolve a token against another cluster or account and learn the Kubernetes groups it maps to there. Error responses also distinguished absent accounts from absent clusters. WebhookTokenReview now passes the same internal-route gate as PublishInternal: a control-plane instance-role session that is a member of the URL cluster in the body account. The gate and the handler share one body decoder, each body-scoped internal route must declare its reader, and every unauthorised target returns the same AccessDenied without reaching STS verification. Same-cluster responses are unchanged. * fix(eks): validate add-on serviceAccountRoleArn syntax CreateAddon and UpdateAddon stored serviceAccountRoleArn without checking it was an ARN, and the guest renders the value into the add-on's manifests. A non-empty value must now be syntactically an IAM role ARN (12-digit account, no region, role resource, IAM name and path rules); anything else returns InvalidParameterException before any record is written or manifest staged. The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at the gateway remains the permission and existence check. Stored values are not rewritten, and an UpdateAddon that omits the field still succeeds on a record holding a legacy value. * docs: record EKS internal-route and role-ARN security corrections * docs: add EKS add-on release membership and activation case study * fix: point UI build, CI and lint rules at moved package paths * docs: record EKS add-on activation review decisions * refactor: move quota admission to domains/admission/quota Relocate the quota service, live counters and vCPU records from handlers/quota to domains/admission/quota. Package renamed from handlers_quota to quota; callers with a local quota identifier import it as admissionquota. KV buckets, subjects and error codes are unchanged, and the package's existing debt edges move with it unchanged. * docs: reconcile package-boundary record for ADR-0001 close-out Add the missing slice rows (the commits that dissolved spinifex/types, the ECR repository projection, the stale-path fix and the quota move), correct stale entries, record the dependency-rule breaks found in the import graph with removal conditions, add the close-out table of remaining legacy paths with their decided target owners, recount the mulga-side follow-ups, and fix stale old-path references in docs and comments. * test: move KV migration engine tests to foundation/state/migrate The Registry and RunKV tests exercise foundation/state/migrate exclusively but lived in spinifex/migrate's test file. Relocate them, with their NATS helpers, to the package that owns the engine. No test name, assertion or behaviour changes. * refactor: move config and object-store upgrade workflow to operator/upgrade Relocate spinifex/migrate, the operator-driven config-file and object-store upgrade workflow behind spx admin upgrade, to operator/upgrade. Package renamed to upgrade (imported as opupgrade where a local upgrade identifier exists). It stays distinct from foundation/state/migrate, the generic KV migration engine; no registry types, migration names, versions, file formats or command names change. * docs: record migrate move to operator/upgrade * test: enforce ADR-0001 layering with a blocking ratchet Add spinifex/architecture/layering, a test-only package that parses the module's import graph, maps every production package to an ADR-0001 layer and checks each dependency rule as its own subtest. Existing violations are allow-listed against their recorded debt; a new forbidden edge, an unmapped package, a stale allowlist entry or a new package under a frozen legacy root fails local preflight and CI. Record the nine edges the test found missing from the migration record (quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2 cross-domain, installer UI -> admin), name the importers of the utils residuals and gateway/ec2_idempotency.go, and correct the handlers/iam consumer list. * refactor: drop the legacy handlers_ecr import alias Import domains/ecr under its package name, or as ecrdomain where the file also imports the AWS SDK ecr package, matching acmdomain. Also fix a stale daemon file reference in the integration ECR daemon comment. * chore: mark spf13/pflag as a direct dependency operator/cli/wiring.go imports pflag directly. Same version; no upgrade. * docs: record test-package-check and nilaway as unverified gates * refactor: move the layering test to module-root architecture/ The ADR-0001 layering ratchet is repository governance tooling, not Spinifex product code, so it sits beside tests/ rather than under the spinifex/ product tree. make test, test-cover and test-race now include ./architecture/..., and the graph walk excludes architecture/. * fix: update paths left stale by the package moves The MariaDB image stamp test read the RDS engine from handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads domains/rds/engine/mariadb.go. Comments citing moved files now name their new locations. * refactor: move the default-VPC subject into contracts/ec2/v1 utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the request/reply route between gateway account creation and the daemon. Only the subject moves; the request and reply bodies are still anonymous structs on each side and are recorded as such in the contract README. The layering freeze now allows two utils files. * docs: qualify the migrate split and record admin and Viperblock findings The migrate row is a split, not an exact ADR-0001 map row, and needs a map clarification. Correct the stale admin consumers, record the operator-only admin files as movable, the vbwire prerequisite for the Viperblock adapter and the capability decision encryption.go needs. * refactor: move operator-only admin code to operator/admin Config generation, certificate issuance, host discovery, telemetry, email validation and image import/removal/orphan-volume tooling move to operator/admin. Legacy admin keeps only the symbols with non-operator consumers: default account identifiers, AWS credential generators and system image promotion. foundation/pki and the tenant ACM CA are unchanged. The readAMI helper and the removal test fixtures are duplicated privately in legacy admin rather than shared across the split. The now-stale operator/installer/ui -> admin layering allowlist entry is removed. * docs: record operator/admin split in package-boundary migration * test: pin viperblockd fence, abandon and dirty-marker wire shapes Characterization tests against the current vbwire package, before it moves to contracts/viperblockd/legacy/v1: both subject forms for the fenced and abandon routes, golden JSON for VolumeFencedEvent, VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty bucket name, and the volume-key pattern shared by the lease and dirty stores. * refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts Move the fence and abandon subjects/DTOs, the dirty bucket name, DirtyRecord and the shared volume-key pattern out of vbwire and into contracts/viperblockd/legacy/v1, alongside the package's existing mount/unmount/delete/sync/config-update routes. vbwire keeps only the I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the contract vocabulary instead of defining it. Record the approved dirty-marker ownership boundary in the contract README: the Viperblock adapter is the sole producer, viperblockd was only its transitional custodian. Document the fence and abandon routes and the dirty bucket in docs/service-interfaces.yaml. * refactor: split the Viperblock EBS adapter out of services/viperblockd Move the EBS provider adapter (volume lease, fence, dirty marker, nbdkit recovery, provider handlers, encryption key loading) to providers/ebs/viperblock. services/viperblockd keeps only process lifecycle and the NATS subscription wiring that starts and stops it. The dirty-bucket read path moves to providers/ebs/viperblock/dirty as a read-only ListUnsealedVolumes capability; the adapter's write, clear and purge logic stays unexported in the adapter package. Delete services/viperblockd/vbwire now that its vocabulary lives in contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty. Update operator/cli, the runtime service registry, qemunbdd's config comment and the integration test harness to the new import paths, and drop the now-stale architecture/layering allowlist and frozen-package entries for the deleted package. Behaviour, persisted keys and NATS subjects are unchanged. * refactor: move the viperblockd role to runtime/roles/viperblockd With the EBS adapter in providers/ebs/viperblock, services/viperblockd holds only role supervision and the opt-in live data-path tests, so it joins the other runtime roles. It was the last package under services/; the frozen legacy list drops it. * docs: record viperblockd contract, adapter and role moves * test: characterise vpc.* network-realisation routes Pin every vpc.* subject, JSON payload shape and the request/reply {success,error} ack envelope exactly as they exist today, ahead of introducing contracts/network/v1. Covers the 22 subjects registered by domains/network/subscribers, the vpc.igw-attach/detach subjects and InternetGatewayEvent already owned by contracts/ec2/v1, the utils.AddNAT/PublishNATEvent client wire behaviour including the delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent and SGEvent/SGRule duplicates in domains/ec2/vpc. Also adds the six vpc.* route-table and system-egress subjects that were missing from service-interfaces.yaml despite being live production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet- egress, vpc.add/delete-system-egress). * feat: add contracts/network/v1 for vpc.* realisation routes Declare the versioned wire vocabulary for the vpc.* control-plane-to- network-realisation boundary: subjects, payload types, the shared {success,error} ack envelope and the vpcd-workers queue group, one file per route family. No NATS I/O, no consumer switched over yet; the package coexists with the fragmented originals pinned in the prior characterization commit until slice 3 migrates call sites. * refactor: use contracts/network/v1 for vpc.* producers and consumers Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw, routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the network/subscribers consumer onto contracts/network/v1 subjects, event types and the shared AckEnvelope. contracts/network/v1 becomes the sole owner of vpc.igw-attach/detach, replacing the duplicate definitions in contracts/ec2/v1. Deletes the duplicated local subject constants and event structs: subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent, eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent. domains/ec2/vpc.SGRule survives because EC2 persists it directly in SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded in slice 2 to match it field-for-field so wire bytes are unchanged. No lifecycle or reliability changes: the NAT add barrier, delete retry-on-ErrNoResponders, and the inconsistent per-call-site NAT delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay fire-and-forget with no ack) are preserved exactly, as are the golden wire literals pinned in the characterization tests. * refactor: replace utils/vpcd_event.go with a typed network projection client Every vpc.* call site now goes through domains/network/projection's operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent or utils.RequestEvent, so no caller passes a bare subject string. Each method states today's delivery mode by name: AddNAT stays strict, AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those call sites already had, and RemoveNAT keeps the ErrNoResponders retry. utils/vpcd_event.go and its tests are deleted; the behaviour they pinned (wire shape, ack envelope, retry-on-no-responders) moved into the projection package's own golden and nat tests, unchanged. utils/ now holds only encryption.go, which this does not touch. * refactor: route the remaining fire-and-forget vpc.* producers through projection CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/ DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/ RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress were still publishing through natsmsg.PublishEvent or a raw nc.Publish call with a bare subject string, left over from the mechanical slice 3 pass. Each now goes through its named projection.Client method instead, so every vpc.* producer in the tree uses the typed client. * refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils The master-key loader was used only by EC2 image, instance and volume to derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled, EC2's temporary reading of the provider's key-file setting; the cache, empty-path meaning, error text and each caller's error handling are unchanged (image still reports unencrypted when the key will not load). Its successor is a providers/ebs capability reporting volume encryption posture. utils/ is now empty and deleted. The layering test drops utils and services from the legacy roots, removes their allowlist entries and fails if either retired root regains files. The S7 subject scan follows the vpc.* literals from utils to contracts/network so its gap list stays complete. * docs: record the network contract, projection client and utils removal * test: characterise instance-state persistence, recovery and generation handling Pin the current local-file bytes and schema failure modes, the instance bucket configuration, record key and wire form, the bucket version gate, generation preservation across every KV writer and the VM conversion, and what a returning node does for a missing, foreign, terminal or unreadable canonical record. These describe current behaviour, not the ADR-0007 target. * test: freeze direct instance-state consumers Add an AST ratchet over production code that fails when a new file names the daemon's instance-state bucket, record keys, local-file API, the raw vm.InstanceRecord or its conversion, or a JetStreamManager instance-state method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0 inventory; stale entries fail, and a synthetic-source self-test proves the matcher detects each kind of use and ignores declarations. * docs: record ADR-0007 slice 0 instance-state inventory Record the instance-state persistence formats, the nineteen direct consumers by process and access kind, the characterised recovery and schema outcomes, the generation-preservation matrix and the ratchet that freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays blocked. * test: narrow the instance-state ratchet to instance-state access Stop freezing the JetStreamManager type and constructor, which also serve cluster state; flag only instance-state methods, bucket and key identifiers, record codecs and local-state access. The cluster-state-only consumers in daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67). The self-test now proves a cluster-state use is ignored and an instance-state method call is caught. Record the owner's disposition of the KV-unavailable restore result in the migration record. * docs: record the narrowed instance-state ratchet commit * docs: design the instance-state transition record and activation gate * docs: record the instance-state transition-record design commit * docs: record slice 1a activation-gate decisions * feat: add instance transition-record codec with format marker Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner package. A top-level format member selects the shape: absent or 1 decodes the legacy record and is never given a marker on write; 2 also decodes the new assignment and observation sections; an integer above 2 decodes its legacy fields read-only and Encode refuses it; anything else is refused on decode. Encode applies only the typed paths that differ from the decode-time snapshot onto the input, so unknown members at every object level and unchanged known fields keep their read form. A changed array is replaced whole. Decode refuses malformed JSON, duplicate keys at any depth, keys that alias a known field by case, trailing data and a non-object record, each with its own sentinel. No production reader or writer uses the codec yet; the new types are not part of vm.InstanceRecord, so every existing record encodes unchanged. The codec file is registered as an owner file of the instance-state ratchet. * docs: record slice 1b step 1 codec and awsgw rollout qualification Records the codec added in e62f921d2 as design Section 10 (C1 to C7): its surface, preservation at every object level, the changed-array limitation, marker handling, the rejection classes and its capable-writers-only scope. C2 supersedes P1's top-level-only preservation for the codec. G19 records that old read-only gateway consumers remain safe but DescribeInstanceStatus gains unconfirmed/degraded visibility only after awsgw upgrades. The migration record gains the commit row and states that no production path uses the codec yet. * refactor: move the instance transition-record codec to domains/ec2/instance/state The codec defines how the canonical persisted instance record evolves, with its assignment and observation sections, which is EC2 instance ownership rather than node-local VM supervision. It moves unchanged into its own package and keeps wrapping the legacy vm.InstanceRecord alias and its spec and status types, which stay in runtime/compute/vm for now. The domain-to-runtime import is recorded as new dependency-rule debt with its removal condition and allow-listed against that entry. The instance-state ratchet now names the new file as an owner file; its allowlist is unchanged. * fix: refuse array changes that would drop element extensions A changed array is replaced whole by the writer's value, which silently lost any member a newer writer had added inside its elements. Encode now inspects the array as read: if any element carries a member its typed schema does not know, at any depth, it returns ErrRecordExtensionConflict naming the array and the first such member. Arrays without such members are still replaced whole, unchanged arrays are still kept verbatim, and arrays inside unknown members are never inspected. * docs: record the codec move and the array extension-conflict rule Records 8db9771fb and c617914a1. T1 and Section 10 now name domains/ec2/instance/state as the eventual owner of the persisted instance record and its evolution, with vm.InstanceRecord as the temporary legacy type it wraps. C3 replaces the changed-array limitation with the refusal: a changed array with element extensions fails with ErrRecordExtensionConflict and is never silently dropped. P2 stays a conservative rule until finalisation. * fix: refuse removals that would drop record extension fields A capable writer may no longer remove or replace a known object, map entry or array when its stored value carries members the writer's schema does not know. The check that guarded changed arrays now runs wherever the merge drops or overwrites a stored value instead of merging into it: a known member deleted or set to null, a map entry removed or stored under a key the typed decode rewrote, a value of another JSON kind, and a changed array. Encode returns ErrRecordExtensionConflict naming the changed container and the first unknown member. Removal stays allowed when the stored value has no extensions, and a change merged member by member still keeps unknown members. * docs: record the uniform extension-conflict rule Adds design clause C8 for 0d7b4266d: a capable writer may not remove or replace a known object, map entry or array whose stored value carries unknown structured extension fields; removal without extensions stays allowed. C3 and C5 point at it. The migration record gains the commit row, updated subsection wording and a pointer-bump follow-up deferring make docs-portal to the Mulga gitlink PR so the portal evaluates the proposed Spinifex revision. * refactor: inject vpcd's IMDS instance-state readers from the instance state boundary * docs: cite the vpcd reader injection commit in the migration record * refactor: move vpcd to runtime/roles/vpcd * docs: cite the vpcd move commit in the migration record * refactor: move the ECS agent wire vocabulary to contracts/ecs/v1 Move handlers/ecs/bus to contracts/ecs/v1 (package ecsv1) with no legacy shim. Subject strings, JSON fields and types are byte-identical; the old-agent compatibility test moves unchanged and golden pins cover the remaining message types. The README route table records that only service-reconcile has a live producer today; the agent reuses these payload types through the gateway actions. service-interfaces.yaml lists ecs.bus.* under the daemon. * refactor: move system-image promotion to domains/ec2/image Move images_promote.go and its tests from admin to domains/ec2/image: SystemOwnerAlias, PromoteImageOpts/PromoteImageResult, PromoteSystemImage, GetAMIMetadata and the private snapshot reader. Storage, ordering, errors and promotion behaviour are unchanged; the daemon, gateway/spx and operator CLI callers import ec2image. admin now holds only account IDs and key generators. * docs: record the ECS contract and image-promotion moves and the remaining-roots matrix * refactor: move the bootstrap account identifiers to foundation/aws/identifiers DefaultAccountID and DefaultAccountName join GlobalAccountID as the fixed, platform-reserved identity of the first human-facing bootstrap account. Names and values are unchanged and stay fixed. Callers in daemon, gateway, awsgw, accountteardown and the operator CLI use awsidentifiers; admin now holds only the IAM credential generators. * docs: cite the bootstrap account identifier move in the migration record * refactor: inject the STS cluster JWKS lookup from handlers/eks handlers/sts declares ClusterJWKSResolver and keeps JWKS decoding and InvalidIdentityToken mapping; handlers/eks implements it as OIDCJWKSReader beside the bucket and key layout helpers. awsgw and the integration harness inject the reader. STS no longer imports any EKS package, including tests, and the handlers/sts -> handlers/eks layering exception is removed. * docs: cite the STS JWKS resolver cut in the migration record * refactor: move the AWS credential-format generators to foundation/aws/credentials and retire admin GenerateAWSAccessKey and GenerateAWSSecretKey move unchanged to foundation/aws/credentials as a stateless format primitive. handlers/iam and operator/cli import it directly. The admin legacy root is deleted and joins the retired roots so it cannot return. * docs: cite the admin retirement in the migration record * revert: remove the uncalled ADR-0007 codec and design stack from the ADR-0001 checkpoint The instance transition-record codec, its tests and the slice 1a design had no production caller. They leave this package-boundary checkpoint and will be reapplied on a dedicated ADR-0007 branch. Slice 0 characterisation and the vpcd IMDS readers in domains/ec2/instance/state stay. * docs: cite the ADR-0007 codec revert in the migration record * fix: declare the ECS action subjects in service-interfaces.yaml Declaring ecs.bus.* made ecs a managed prefix for manifest-lint, which then failed on the 34 ECS action subjects the daemon registers. Declare them as ecs.*, the same wildcard convention as rds.*. * chore: run manifest-check and manifest-lint in preflight CI runs both, but preflight did not, so manifest drift could pass local checks and still fail Build and Test. * docs: record resumed preflight enforcement and dev merge ports * fix: restore dev changes dropped by earlier merges Reapplies the slices.ContainsFunc rewrite in nodeFitsCustomerInstance and the DeleteDBSubnetGroup doc comment, both lost when their files were refactored. * docs: record accountteardown as a frozen legacy root pending cleanup capabilities --------- Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-10-15 14:42:53 +10:00
Ben Duncan	18853cef31	fix: close the four defects keeping the e2e nightly red (#1007) ... * fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main.	2026-09-06 16:53:32 -05:00
Josh Siviour	f0ccf0d0c3	Fix/ecs scheduler leader cas (#861) ... * fix(ecs): guard scheduler leader refresh and release with a revision CAS * Ignore .cache * update comment	2026-08-24 11:15:58 +10:00
Ben Duncan	b834910e64	UCS tweaks	2026-08-21 09:53:04 -05:00
Julian Sommer	74117e5b26	fix: remove the broken golangci-lint cache path	2026-08-20 11:37:10 +10:00
Josh Siviour	135da2371e	fix: scope golangci-lint cache per worktree, correct eks-node-gpu catalog description (#629) ... * fix: scope golangci-lint cache per worktree golangci-lint's default cache under ~/.cache/golangci-lint is keyed by module path rather than worktree path, so parallel worktrees of this module collide and surface each other's lint issues, including from worktrees that have since been deleted. * fix: correct eks-node-gpu catalog description to match agent-only image The GPU node image never installs eks-token-webhook or a server/agent role selector, per scripts/images/eks-node-gpu/manifest.conf; it ships ecr-credential-provider only and is agent-only because GPU nodes never run the EKS control plane.	2026-08-02 08:01:45 +10:00
Julian Sommer	b1b7d4196f	chore: cleanup .gitignore	2026-06-19 11:13:16 +10:00
Josh Siviour	4ae2a4ac53	feat(elbv2): QEMU microvm direct-boot for LB system instances (#228) ... * feat(vm): direct-boot foundations + config/input extensions + image build pipeline vm package (mulga-siv-62): - Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config - Emit -kernel/-initrd/-append/-fw_cfg args when fields set - Relax drive invariant: accept kernel image in place of drives - Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers - Refactor lifecycle.go + network_helpers.go call sites to use helpers - shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits - Add devices_test.go, microvm_test.go; extend vm_test.go config/input extensions (mulga-siv-63): - Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig - ImagePath falls back to /usr/share/spinifex/microvm when empty - Add commented [microvm] section to spinifex.toml template - Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput image build pipeline (mulga-siv-64): - scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz - build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert) - build/microvm/inittab: BusyBox init with lb-agent respawn - make build-microvm-image target - setup.sh: install microvm/* to /usr/share/spinifex/microvm/ - .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts * feat(daemon): direct-boot launch path for microvm system instances When SystemInstanceInput.DirectBoot=true: - Skip GetAMIConfig and GenerateVolumes - Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports) - MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on - Resolve kernel/initrd from Microvm.ImagePath config field - Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert) - Validate exactly one NICConfig.IsDefault=true before writing netcfg blob - Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration PC-machine path (DirectBoot=false) unchanged. Closes mulga-siv-65 * feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service When ELBv2Enabled=true: - Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead - Set SystemInstanceInput.DirectBoot=true; omit ImageID - Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs) - Validate exactly one IsDefault NIC at service layer When ELBv2Enabled=false: existing PC-machine path unchanged. Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers. Tests in arn_test.go cover both flag-on and flag-off paths. Closes mulga-siv-66 * test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold * feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag - Log t1→t2 boot timing via slog on the direct-boot path - measureQEMURSSMiB helper reads /proc/<pid>/status (tested) - Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25 - run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note Closes mulga-siv-67 * fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed * fix(microvm): end-to-end fixes for direct-boot LB path Serial output was silent (x-option-roms=off suppresses BIOS serial init); fixed by removing it from the machine string and using -serial chardev:console0 to attach to the machine-created ISA serial device instead of adding a duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0). virtio-net interfaces were invisible to the kernel because QEMU's auto-kernel-cmdline does not append virtio_mmio.device= params when -append is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net in init.sh. sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs decompression (~68 MiB uncompressed). BusyBox init ::respawn: entries do not source or export env files; fixed inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec. wireLBAgentConfig was documented as reading the CA cert but never set elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with x509: certificate signed by unknown authority. Fixed by reading d.config.NATS.CACert (same local CA that signs the AWSGW server cert) and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a non-empty cacert blob into fw_cfg. Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test to assert SerialSocket is intentionally empty on direct-boot VMs. * fix(build): auto-detect docker/podman when apk unavailable for microvm image build * fix(build): create /dev/console and /dev/null as proper device nodes in initramfs tar extraction without root silently demotes device files to 0-byte regular files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring init's stdio to the serial console; a regular file there causes all init output to be silently discarded to RAM — the VM appears stuck at 'Run /init as init process' with no subsequent console output and lb-agent never heartbeats. Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime once init.sh mounts it. * fix(microvm): fix mgmt NIC routing for all LB schemes Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface: 1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing 'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct 'ip route add 192.168.1.33 via 10.15.8.1'. 2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC after buildMicrovmNICs returns, but didn't inject them into input.NICs[1] before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg blob and skipping interface setup in init.sh entirely. Also extends mgmt routing to internet-facing scheme in the microvm path: VPC SNAT is not reliable across all VPC topologies for control-plane heartbeats; mgmt is always a direct L2 path to the host. * feat(microvm): Phase 2 kernel stripping + boot optimisation - cpu: switch to host,-pmu (disable PMU — not needed in microvm guest) - machine type: make isa-serial configurable via microvm.isa_serial config field; default true (on) preserves console log path - build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input, HID, bluetooth, staging, sound, wireless) from initramfs before cpio - build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var during development - perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants with canary tests to catch unintentional gate changes - tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths - tests: cover microvmMachineType nil/true/false branches * fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip The directory-only keep-list (drivers/virtio, drivers/firmware) missed virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/, and virtio-rng in drivers/char/hw_random/. Switch to find-based approach: collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then restore them into their original relative paths under kernel/. Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes. * fix(build): keep failover modules in initramfs strip virtio_net depends on net_failover/failover for live-migration support. Stripping them left virtio_net with unresolved symbols (net_failover_create/destroy), so the module failed to load and the guest came up with only lo. init.sh then skipped every NIC ("no interface found for MAC ..."), leaving lb-agent with no network path to AWSGW and the LB stuck in provisioning. Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB (under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and 2/2 targets healthy for both ALB and NLB internal LBs. * fix(daemon): restore slog default after viperblock.New mutates it viperblock.New unconditionally calls vb.SetDebug, which calls slog.SetDefault with a TextHandler at LevelError — globally silencing every slog.Info and slog.Warn in the caller process for the rest of its life. Discovered while validating microvm Phase 2 (mulga-siv-68): the daemon stopped emitting Info logs the moment the first customer VM with volumes launched, hiding direct-boot timing, LaunchSystemInstance completed, and CreateLoadBalancer completed lines that we need for boot-time tuning. Workaround in newViperblock: re-install an Info-level TextHandler on os.Stderr after each viperblock.New call. Also drop the three explicit vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume / prepareCloudInitVolume — they were no-ops (viperblock.New already runs SetDebug internally) and each one re-triggered the global mutation we are trying to undo. Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug should not call slog.SetDefault. Until then, this restore call is the shortest path to getting the daemon's observability back so we can measure t1_to_t2_ms and tune Phase 2. Preflight note: golangci-lint not available on this dev host, so the project preflight target cannot run end-to-end. Verified with go vet ./spinifex/handlers/ec2/instance/... (clean) and go test ./spinifex/handlers/ec2/instance/... (pass). * perf(microvm): hit Phase 2 boot/memory gates t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots cleanly without OOM, reverting the temporary 512 MiB bump. Root causes: 1. build-microvm-image.sh staged the entire chroot into the cpio without dropping /boot, embedding a duplicate vmlinuz, Alpine's stock initramfs-virt, System.map, and a "boot → ." symlink loop (~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after the kernel is copied out, before the cpio. Initramfs drops from 31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact 42 MiB → 22 MiB. 2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and 1 s post-fork settle (QEMU binds QMP within ~10 ms when there is no firmware, ROM, or block layer). Both extracted to directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and qemuStartSettleWait to 50 ms. 3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`. Added boot-time perf flags: quiet loglevel=3 (silence printk over 115200-baud isa-serial), mitigations=off (KVM guest on trusted host, single-tenant LB workload), tsc=reliable, no_timer_check, reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2). 4. init.sh dropped modprobe virtio_mmio (built-in: CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms). Phase 2 gates: - t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms - artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script) - ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure, unrelated to microvm path) * build(microvm): wire build into deploy + distro + release pipeline Adds install-microvm Makefile target (with grouped-target staleness detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when present, and makes 'make distro-amd64' stage build/microvm/* into the release tarball staging dir. arm64 distro stays unchanged (no aarch64 microvm artifact yet — Phase 4). Release workflow gains a setup-go step and a 'make build-microvm-image' step before 'make distro-amd64' on the amd64 runner so the resulting tarball includes the kernel + initramfs without a manual copy. Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69). Closes mulga-siv-71. * build(ansible): include microvm artifacts in build role tarball Adds a build-microvm-image task to the ansible build role and stages build/microvm/* into the role's release tarball so the install role's setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped gracefully when the build script is absent (older branches without direct-boot LB wiring). * feat(config): enable microvm direct-boot LB path by default Uncomments the [nodes.<node>.daemon.microvm] section in the generated spinifex.toml template and sets elbv2_enabled = true so all new clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators can opt back to the legacy PC-machine path by flipping the flag to false in the rendered config. Required for end-to-end CI on this branch to exercise the direct-boot path. Existing clusters' configs are unchanged (the template only affects fresh 'spx admin init' renders). * fix(microvm): bootstrap apk.static when no host rootfs tool exists Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often have none of apk, docker, or podman installed. apk-tools is not in the Debian 13 (trixie) repositories. Without one of these the rootfs build fails up front with 'no rootfs build tool available'. When all three are absent, fetch apk-tools-static from the configured Alpine mirror, extract it to a tempdir, and prepend it to PATH so the existing native-apk strategy below picks it up transparently. Requires only curl + tar, both of which are present on every CI runner that can already check out source. No-op when apk, docker, or podman is already present. Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244 * revert: drop apk.static bootstrap in build-microvm-image Reverts 0bd76b7. Self-hosted runners will get docker.io installed out of band; existing Strategy 2 (docker run alpine:3.20) handles the rootfs build without host sudo or unsigned package install. * docs(workbook): drop lb-alpine import for microvm direct-boot LB LBs now launch via direct-boot QEMU microvm using kernel + initramfs bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system AMI import is no longer required. * fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN When the daemon runs non-root (production: User=spinifex-daemon), child QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun attach check then requires the tap owner uid to match the caller's euid. Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation not permitted"). Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo runner where the kernel enforces strictly; legacy EC2 launches on ci-single runners happened to attach successfully despite the same code path. Adding `user $current group $gid` to `ip tuntap add` ensures the tap is owned by whatever uid the daemon (and its qemu children) run as. Root daemons skip the flag — they have CAP_NET_ADMIN ambient. * fix(e2e): redirect lb probe chatter to stderr so stalls are visible test_http_from / test_tcp_from echoed progress to stdout, which the callers captured into *_OK vars via $(...). Result: 6+ minute retry loops ran silently in CI, and the "$ok" comparison never matched "true" because the captured value contained the full multi-line chatter. Reachability pass/fail counters were also lost in the subshell. Redirect helper output to stderr, leave only the true/false flag on stdout, and move pass/fail to the caller so they actually count. * fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return for the single-node internet-facing case by falling back to (MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP via the mgmt NIC inside the LB microvm. On single-node where the AWSGW host IP equals the WAN client IP, the /32 also catches reply traffic for inbound HTTP/TCP to the LB's floating IP — replies leave via the mgmt NIC with src=data-NIC IP, the host kernel drops the asymmetric flow, and same-chassis ingress hangs. resolveMgmtRoute already encodes the correct policy: - multi-node any scheme → explicit MgmtRoute{Gateway,Target} - single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP) - single-node internet-facing → empty (reach AWSGW via VPC SNAT) Honor it. Matches the legacy lbVMUserData path, whose own comment flagged the stolen-WAN-return-path symptom this introduced. Test covers all three branches. * feat(ansible): dev-deploy playbook for in-place artifact swap Fills the gap between ansible-dev-install (skips when already installed) and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors Makefile `deploy` target: rebuild + replace binaries and microvm artifacts on a running node, restart spinifex.target, wait for awsgw. Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so iteration is fast. Use ansible-dev-reset when changing systemd units, helper scripts, logrotate, or setup.sh itself. Adds: - roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm artifacts, with plugindir auto-detection via `nbdkit --dump-config`. - playbooks/dev-deploy.yml — chains build + deploy roles. - Makefile ansible-dev-deploy target. - README playbook table + "when to use which" guidance. * chore(microvm): remove legacy LB Alpine image build pipeline Phase 3 microvm removal — strip the PC-machine LB image build artifacts that direct-boot no longer needs. The microvm initramfs already carries lb-agent + HAProxy via build-microvm-image, so the lb.conf manifest, lb-setup.sh chroot script, build-lb-image target, dev-install / reset-dev-env import steps, and ansible LB-image tasks all become dead. build-system-image stays — other manifests (e.g. GPU images) still use it. * refactor(microvm): remove legacy PC-machine LB boot path Phase 3 — direct-boot becomes the only ELBv2 LB launch path. - Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB caller set DirectBoot=true; legacy cloud-init userdata is gone). - Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc, getSystemAMI, and the lazy-AMI wiring in daemon.startCluster. - Collapse CreateLoadBalancer to a single direct-boot launch input. - Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID on RunInstancesInput, UserData base64 encode, GenerateVolumes branch. buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and isa-serial=on (microvmMachineType no longer takes a flag). - Drop MicrovmConfig from config.DaemonConfig and the [daemon.microvm] section from the spinifex.toml template. - Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver paths) and refresh stale comments that still mentioned lbVMUserData. ARM hosts have no LB launch path until Phase 4 adds direct-boot on virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec. * docs: drop lb-alpine catalog entry from cluster setup guide LB system instances no longer use an AMI — direct-boot microvm artifacts ship in the spinifex package. * feat(ui): remove LB image import banner + LB-AMI gating Direct-boot microvm artifacts ship in the spinifex package, so the LB system AMI no longer exists — the "Load balancer image not imported" banner and the Create-button disable on the LB list / wizard pages can never fire. Strip: - lb-image-missing-banner.tsx component - hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from system-managed.ts (isSystemManagedImage stays — still used by the EC2 Images pages to hide system AMIs) - Banner gating + ec2ImagesQueryOptions imports from the LB list and wizard pages - The corresponding tests that asserted the banner appears * revert: restore lb.conf + lb-setup.sh for mulga install pipeline Phase 3 microvm removal (commit 6313d74) deleted these too early — the mulga install/iso artifact builders (scripts/tofu-cluster/build-install -artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with "ERROR: Manifest not found: scripts/images/lb.conf". Restore both files verbatim so the legacy pipeline keeps building until the mulga-side migration to microvm-only artifacts lands post-merge.	2026-05-18 15:48:04 +10:00
Josh Siviour	b7dea45ee0	feat: Ansible dev lifecycle (parallel to bash scripts) (#178) ... Playbooks for preflight / teardown / install / reset covering the same ground as dev-install.sh, dev-setup.sh, reset-dev-env.sh. Existing bash scripts untouched — decide which to keep after parity testing. - roles/teardown + teardown catalog README track every known state source; add a task + row when a leak is reported. - roles/init gates smoketest on awsgw readiness (describe-regions poll). - roles/smoketest is dup-tolerant, retry-wrapped, auto-terminates the launched instance. - dev-install.yml skips build/install/init when a healthy environment is already present (4-signal health check), always runs smoketest. - dev-reset.yml captures region/az/external-mode scalars from the existing spinifex.toml and replays them into init. Plan: docs/development/improvements/ansible-dev-lifecycle.md (mulga). Bead: mulga-siv-9.	2026-05-01 10:04:37 +10:00
Josh Siviour	6238988a8f	feat: add Terraform nginx-alb example (#129) ... Deploys two Nginx EC2 instances behind an Application Load Balancer on Spinifex, exercising VPC, public subnets, IGW, security group, key pair + cloud-init, ALB, target group, and HTTP listener end-to-end. Also gitignore generated *.pem files from tofu local_file outputs.	2026-04-10 15:56:50 +10:00
Ben Duncan	9e8e944faa	fix: pass private IP in vpc.delete-nat event to clean up OVN rules (mulga-764) ... Root cause: when terminating instances, the vpc.delete-nat event was published with an empty logicalIP. vpcd's DeleteNAT looked for a dnat_and_snat rule with logicalIP="" — found nothing — and the real rule was left as a stale entry in OVN. When the IPAM later recycled that public IP for an EIP/NAT Gateway, OVN had conflicting rules: stale dnat_and_snat AND new snat, both using the same external IP. Fix: extract PrivateIpAddress from instance.Instance before publishing vpc.delete-nat, in both running and stopped termination paths. Also adds 14 unit tests for NAT Gateway service.	2026-03-27 07:43:53 +10:00
Julian Sommer	4a2ae33dd9	fix: add missing assets	2026-03-26 09:56:38 +10:00
Julian Sommer	a09a8efe3c	build frontend with new branding	2026-03-18 13:38:47 +10:00
Ben Duncan	a458a28406	new binary mode and systemd process	2026-03-16 11:57:52 +00:00
Ben Duncan	dd7bce20e6	Move internal dev docs to private repo, trim CLAUDE.md for public use ... Internal development plans, roadmap, and detailed dev process docs moved to private mulga repo. CLAUDE.md trimmed to remove SSH credentials, node IPs, and internal deployment workflows. Gosec fixes for file permissions and unhandled error.	2026-02-17 09:38:28 +10:00
Ben Duncan	962d85edf2	Improved IAC automation testing	2026-02-14 19:27:13 +10:00
Ben Duncan	db0d46ca5b	Improved multi-server node testing and tweaks	2026-02-13 20:31:45 +10:00
Ben Duncan	cf12a3272e	IAC for Proxmox / AWS terraform examples	2026-02-13 11:42:43 +10:00
Julian	7dca292e67	fix: web-ui to use hive certs	2026-01-28 09:55:01 +10:00
Julian Sommer	52b8550788	feat: hive-ui (#10)	2026-01-27 17:05:08 +11:00
Julian Sommer	b9b8c65e79	(fix) Linter fixes ... * Fix go vet mutex copy issues and remove report files from git * remove redundant nil checks (len of nil slice is 0) * handle error * add missing error handling * run modernize to change interface{} to any * run other modernise commands * use crypto instead of rand.Intn	2026-01-06 14:20:56 +11:00
benduncan	8336374837	Expanded error coverage	2025-10-09 16:14:53 +08:00
benduncan	9de19be3f0	Added viperblock unit testing and integration tests	2025-10-05 20:45:07 +08:00
Ben Duncan	a59988d614	Simplified development install method	2025-09-29 19:32:44 +08:00
benduncan	a909fc6753	Alpha hive daemon for compute	2025-07-01 15:10:37 +09:30