29 Commits

Author	SHA1	Message	Date
Josh Siviour	b5879f44ff	ADR-0001 package-boundary migration checkpoint (#1190) ... * refactor: move telemetry file naming to contract * docs: record telemetry contract move * refactor: move aws xml response helpers * docs: record aws xml helper move * refactor: move aws tag projections to foundation * docs: record aws tag projection move * refactor: move account key to state foundation * docs: record account key state move * refactor: move host privilege escalation to runtime * docs: record host privilege move * chore: resolve lint findings from package-boundary moves * refactor: move process supervision helpers to runtime host * docs: record process supervision move * refactor: move nbd endpoint helpers to compute runtime * docs: record nbd endpoint move * refactor: move image catalogue to operator * docs: record image catalogue move * refactor: move resource id generation to aws foundation * docs: record resource id move * refactor: move predastore role to runtime roles * docs: point service manifest at acm domain * docs: record predastore role move * refactor: move northstar role to runtime roles * docs: record northstar role move * refactor: move awsgw role to runtime roles * docs: record awsgw role move * refactor: move error payload helpers to aws errors * docs: record error payload move * refactor: move account identifiers to aws identifiers * docs: record account identifier move * refactor: move NATS transport to messaging foundation * docs: record NATS transport move * refactor: move system instance contract to EC2 domain * docs: record system instance move * refactor: move system VPC builder to network domain * docs: record system VPC move * refactor: move instance metadata service to EC2 domain * docs: record guest metadata move and dependency-rule debt * refactor: move DNS control plane to DNS domain * docs: record DNS domain move * refactor: move base zone bootstrap to DNS domain * docs: record base zone bootstrap move * refactor: replace SystemAccountID with GlobalAccountID * docs: record SystemAccountID removal * refactor: extract CA key-pair loader to foundation pki * docs: record foundation pki extraction * refactor: move tenant CA paths to ACM domain * docs: record tenant CA path move * refactor: move IPsec certificate handling to network host * docs: record IPsec certificate move * refactor: move Ochre serving lifecycle to the Ochre domain handlers/bedrock becomes domains/ochre. Ochre is the platform service; Bedrock names only its AWS API surface. Persisted and observable identifiers keep their bedrock names. * docs: record Ochre domain move * refactor: move Ochre vector store to the Ochre domain handlers/ochrevector becomes domains/ochre/vector. Callers import it as ochrevector because vector is a common local name in the gateway. * docs: record Ochre vector store move * refactor: move Hugging Face Hub client to the Ochre domain gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is the spx admin ochre weights pull. * docs: record Hugging Face Hub client move and Ochre gateway debt * refactor: move EC2 tag mutation helpers to the EC2 domain utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag mutation behaviour, and the tags resource package cannot host it without a resource-to-resource import cycle. * docs: record EC2 tag mutation move and migrate split finding * refactor: move network packages under the network domain Move spinifex/network/* to spinifex/domains/network/* as one enclosing move so the L0-L5 layer invariants keep classifying every package from a single root. Package names and dependency direction are unchanged; invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and the repo-root walk in the IMDS netns unit test follow the new path. * docs: record network domain move * refactor: move standalone EC2 resource packages to the EC2 domain Move handlers/ec2/{account,key,launchtemplate,placementgroup, spotinstance,volumestate} to domains/ec2/<resource>. Each package is named after its directory and callers import it as ec2<resource>; volumestate already carried its directory name. * docs: record standalone EC2 resource moves * refactor: move EC2 VPC networking packages to the EC2 domain Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to domains/ec2/<resource>, each named after its directory and imported as ec2<resource>. The reconcile drift-watch source scan now matches the ec2<resource> bucket qualifiers; its rule is unchanged. * docs: record EC2 VPC networking moves * refactor: move EC2 compute and storage packages to the EC2 domain Move handlers/ec2/{instance,volume,snapshot,image,tags} to domains/ec2/<resource>, each named after its directory and imported as ec2<resource>, which empties handlers/ec2. The daemon's e2e suite selection now lists the moved EC2 resource paths in place of handlers/ec2, so EC2-only changes select the daemon suites again. * docs: record EC2 compute and storage moves * refactor: move ECR credential provider to the EKS agent tree Move the kubelet exec credential provider implementation from cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing one Main(in, out) entry point. The binary keeps its name and FIPS boot import as a thin main. Moving out of cmd/ brings the code under gosec; the one G101 hit on the kubelet API version constant is annotated. * docs: record ECR credential provider move * refactor: move EKS gateway publish helper to the EKS agent tree Move the eks-gateway-publish implementation to agents/eks/gatewaypublish behind Run(ctx, Config) error. The binary keeps flag definitions, environment defaults, parsing, its log prefix and exit code; each former fatal site returns an error with the same text. The retry wait now also honours context cancellation, which the binary never triggers. * docs: record EKS gateway publish move * refactor: move EKS gateway fetch helper to the EKS agent tree Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary keeps flag definitions, environment defaults, parsing, its log prefix and exit code; each former fatal site returns an error with the same text. * docs: record EKS gateway fetch move * refactor: move EKS webhook cert helper to the EKS agent tree cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit; agents/eks/webhookcert owns Config and Run. Output, error text, file modes and exit codes are unchanged. * docs: record EKS webhook cert move * refactor: move EKS konnectivity cert helper to the EKS agent tree cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the exit; agents/eks/konnectivitycert owns Config and Run. Output, error text, file modes and exit codes are unchanged. * docs: record EKS konnectivity cert move * refactor: move EKS token webhook to the EKS agent tree cmd/eks-token-webhook keeps the -addr flag, environment-derived config, logging setup and exits; agents/eks/tokenwebhook owns Config and Run. The handlers/eks TokenReview result type stays as recorded temporary debt. * docs: record EKS token webhook move * refactor: move ECS agent runtime and credentials packages to the ECS agent tree * docs: record ECS agent subpackage move * refactor: move ECS agent implementation to the ECS agent tree cmd/ecs-agent keeps process lifecycle: signal context, the separate New and Run calls, both failure log lines, the exit and the ldflags-set build version, now passed in through Config.AgentVersion. agents/ecs/agent owns Config, LoadConfig and the agent. * docs: record ECS agent move * refactor: move RDS guest agent to agents/rds/agent * docs: record RDS guest agent move * refactor: move installer packages to operator/installer * docs: record installer move * refactor: move spx command tree to operator/cli Moving the command tree out of cmd/ brings it under errcheck and gosec. Flag and config wiring now fails loudly instead of silently dropping a binding, listing commands fail when they cannot write their table, and the formation server's unauthenticated TLS is annotated as a known gap. * docs: record spx command tree move * chore: accept known vulnerabilities with an expiry The vulnerability scan failed preflight on a containerd advisory that dev already carries and that is deferred to a separate dependency update. The gate now passes only reachable findings listed with an expiry date, fails on any new one or on an expired acceptance, and warns when an acceptance is no longer needed. * refactor: move EC2 account action adapters to domains/ec2/awsapi * refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi * refactor: move EC2 key action adapters to domains/ec2/awsapi * refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi * refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi * docs: record EC2 standalone action adapter moves * refactor: move EC2 zone action adapters to domains/ec2/awsapi * refactor: move EC2 vpc action adapters to domains/ec2/awsapi * refactor: move EC2 eigw action adapters to domains/ec2/awsapi * refactor: move EC2 eip action adapters to domains/ec2/awsapi * refactor: move EC2 igw action adapters to domains/ec2/awsapi * refactor: move EC2 natgw action adapters to domains/ec2/awsapi * refactor: move EC2 routetable action adapters to domains/ec2/awsapi * docs: record EC2 VPC networking action adapter moves * refactor: move EC2 image action adapters to domains/ec2/awsapi * refactor: move EC2 snapshot action adapters to domains/ec2/awsapi * refactor: move EC2 volume action adapters to domains/ec2/awsapi * refactor: move EC2 tags action adapters to domains/ec2/awsapi * docs: record EC2 storage and tag action adapter moves * refactor: move EC2 resource authorization scopes to domains/ec2/awsapi * refactor: move EC2 idempotency adapter to domains/ec2/awsapi * refactor: move EC2 instance action adapters to domains/ec2/awsapi * refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi * docs: record EC2 instance action adapter moves * refactor: move AWS REST-JSON router to ingress/aws/rest * refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit * docs: record AWS ingress helper moves and tooling conformance * docs: correct historical lifecycle references * test: characterize ECR control-plane dispatch before registration seam * feat: add AWS ingress service registration contract * docs: record ingress registration phases 0 and 1 * refactor: route registered AWS services through the ingress dispatch registry The gateway now consumes ingress/aws/dispatch. A registered service is accepted by the auth gate, resolves its own action label, renders errors in its declared envelope and is dispatched with an authenticated Invocation (account, region, policy authorisation and a lazy verified actor). Every other service keeps the existing switch, now named as the legacy path. The awsgw role wires an empty registry and refuses to start if a registered name is also served by the legacy table. No service is registered yet, so behaviour for every existing service is unchanged. * docs: record ingress dispatch registry consumption * refactor: register the ECR control plane from its domain domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the ingress dispatch registry: action resolution, resource scope, authorisation through the invocation, the composed capabilities and the success writer all live with the domain. The awsgw role is the only serving composition point; the gateway no longer imports the ECR API package and ECR leaves the legacy service table, JSON error table and dispatch switch. The OCI /v2 route is unchanged. Operation coverage reads ECR's declared inventory, which the registration uses verbatim; coverage.json is byte-identical. * docs: record ECR control-plane registration * fix: compose the ECR control plane in the integration harness The harness stopped wiring ECR's composed capabilities when the authorization-token action was composed, so every integration test that creates a repository or mints a token got ServerInternal. It now composes ECR as the awsgw role does, sharing the OCI registry, meta store and token issuer the /v2 data plane uses. * docs: record ingress pipeline deferral * refactor: extract AWS error envelope rendering to ingress ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header detector and the two error content types. Each shape keeps its own builder. The gateway error writers, ErrorHandler and the legacy per-service envelope selection stay in gateway and call the new package; gateway/eks renders its JSON errors through it directly. A 140-case status/header/body golden comparison across services and error writers was identical before and after. * docs: record error envelope extraction * refactor: move AWS service selection into ingress dispatch Selector answers served, action, error protocol, registered entry and startup overlap. The gateway supplies its legacy tables through the dispatch.Legacy interface; legacy dispatch and invocation building stay in gateway. * docs: record ingress service selection move * test: characterize ACM control-plane dispatch before registration Pins full-pipeline error envelopes, a successful action, target parsing agreement between dispatch and telemetry, authorization before the NATS call, 413 and 503 paths, throttle/trace/audit action and the inventory. * refactor: register ACM control plane from its domain domains/acm/awsapi registers the acm JSON 1.1 surface through the ingress dispatch registry, wired by the awsgw role. The gateway drops ACM_Request, the ACM action table and every legacy acm entry; the import invariant now covers each registered domain. * docs: record ACM registration * docs: record ECS, EKS and RDS source inventory and first-resource ranking * docs: record EKS control-plane secrets persisted in the cluster record * chore: drop containerd advisory acceptance now that dev ships v2.4.1 * test: characterize RDS DB subnet group lifecycle before extraction * docs: record RDS DB subnet group lifecycle contract * refactor: extract the RDS DB subnet group lifecycle owner domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member subnet validation and the delete in-use guard. handlers/rds keeps AWS validation, projection and paging and binds two consumer-owned capabilities: the owner's Dependants (instances naming the group) and the instance side's subnetGroupPlacement. Stored shape and behaviour are unchanged. * docs: record RDS DB subnet group owner extraction * docs: correct RDS subnet group tagging debt and conformance scope * test: characterize ECS task definition lifecycle before extraction * docs: record ECS task definition lifecycle contract * refactor: extract ECS task definition owner into domains/ecs/taskdefinition Move the task definition record, container types, keys, reference parsing, ARN helpers and the register/deregister/resolve/list logic into a resource owner package. Stored JSON field names and KV keys are unchanged, and revision allocation keeps its existing non-atomic read-then-write. The owner declares its own PortMapping and SystemControl persisted types so it no longer depends on handlers/ecs/bus; the handler translates them onto the assign payload. Task and service launches reach task definitions only through a launch-side taskDefinitionResolver. AWS conversion, the guest projection and tags stay in handlers/ecs, and gateway authz only switches to the moved reference/ARN helpers. List handlers/ecs and the new package under spinifex_daemon so ECS-only changes select e2e-ecs. * docs: record ECS task definition owner extraction * test: characterize EKS access entry lifecycle before extraction * docs: record EKS access entry lifecycle contract * refactor: extract EKS access entry owner into domains/eks/access Move the access entry record, associated policy and scope types, keys, PrincipalARNHash, the policy catalogue and group mapping into a resource owner package. Stored JSON field names and KV keys are unchanged. Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate, Disassociate and Delete carry the existing logic verbatim, and the put, compare-and-set and delete primitives are private. The handler keeps AWS validation, the cluster existence check, error mapping and projection. Token review still reads records through the exported Get, and the cluster prefix purge still sweeps entry keys. List handlers/eks and the new package under spinifex_daemon so EKS-only changes select the EKS suites. * docs: record EKS access entry owner extraction * test: characterize RDS engine catalogue before extraction * fix: return InvalidParameterValue for a missing RDS master password ValidateMasterUserPassword built the empty-password error with a bare errors.New carrying the code only in its text, which awserrors does not resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500 ServerInternal. Build it with awserrors.Errorf like the other branches so the gateway returns 400 InvalidParameterValue, and cover the full gateway to RDS service path with a regression test. * docs: record RDS engine catalogue boundary and evidence * refactor: extract RDS engine catalogue into domains/rds/engine Move engine identity and validation, the per-engine parameter catalogues, effective-parameter resolution, db.* instance classes and engine/orderable selection into an engine package. Behaviour, error codes and ordering are unchanged. The package imports no EC2 package: instance-class memory comes from an immutable Sizing value built from an engine-declared InstanceTypes interface, backed by the fixed instance-type table in handlers/rds. Effective parameters leave as an engine-owned Setting, converted to Parameter in handlers/rds. AWS output building, paging, apply-method mapping, storage limits and the gateway runnable check stay in the adapter path. The RDS agent declares its own EngineCatalog interface and types and receives an engine-backed implementation from cmd/rds-agent, so agent code does not import the catalogue. Cross-engine tests use the two real engines instead of a registered fake. * docs: record RDS engine catalogue owner move * test: characterize RDS DB parameter group lifecycle before extraction * docs: record RDS DB parameter group lifecycle contract * refactor: extract RDS DB parameter group owner Move the parameter group and override records, keys and listing into domains/rds/parametergroup behind an Owner. Delete asks an owner-declared Dependants interface; instance-side resolution reads through its own parameterGroupValues interface. SetOverrides refuses an unknown group so no override key is written without its group. Stored shape, keys, public errors and propagation are unchanged. * docs: record RDS DB parameter group owner extraction * test: characterize EKS add-on lifecycle and guest wire Pin the persisted record and manifest layout, create/update/delete and status-report transitions, the ignored client token and report version, the missing serviceAccountRoleArn PassRole check, and the exact internal add-ons HTTP body and agent TSV output as they stand today. * docs: record EKS add-on lifecycle contract * fix: require iam:PassRole for EKS add-on service account role CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup. * docs: record EKS add-on PassRole check * refactor: extract EKS add-on owner Move the add-on catalogue, record and manifest keys, CAS, status transitions and staging into domains/eks/addon behind an Owner that is the sole writer. The handler keeps AWS validation, error mapping and projection; the cluster reconciler delegates reports through the narrow addonReports interface; node groups call EnsureGPUDevicePlugin. Stored keys, JSON shape, wire types and lifecycle behaviour are unchanged. * docs: record EKS add-on owner extraction * refactor: extract EKS add-on guest wire contract Move the add-on status subject, delivery phase, status report, staged manifest and internal add-ons response into contracts/eks/v1 and use it from the handler, gateway and gatewayfetch, replacing the agent's duplicate. Wire bytes are unchanged, including the internal add-ons body that the SDK renderer emits with Go field names; compatibility tests pin the JSON, the guest printf form and the TSV. * docs: record EKS add-on wire contract extraction * docs: update EKS add-on inventory and remaining edges * docs: draft EKS add-on lifecycle correction design for review * chore: run contracts tests in the default test, coverage and race targets * docs: apply review decisions to EKS add-on lifecycle design * test: characterize EKS add-on v1 compatibility with added fields Pin which owner writes drop unknown record and manifest fields, that reads and deletes ignore or erase them, that one undecodable entry fails a listing, and that v1 wire consumers ignore extra fields. * docs: add EKS add-on executor, bundle, v1 compatibility and release inventories * docs: draft EKS add-on v2 guest contract design for review * docs: qualify historical architecture references * docs: apply review decisions and ADR-0006 to EKS add-on v2 design * fix(eks): bind internal report publishing to the caller's cluster PublishInternal was authorised by IAM policy alone, with its resource built from the body accountId and the URL cluster. The control-plane role grants it on "*", so any control-plane VM session, any tenant holding eks:*, or static system credentials could publish cluster state, bootstrap and add-on reports for another cluster or account. PublishInternal now passes the existing internal-route gate: the caller must be a control-plane instance-role session that is a member of the URL cluster in the body account. The gate and the handler decode the body through one shared decoder, so the account checked is the account the subject is built from. Every unauthorised target returns the same AccessDenied, and invalid account or cluster names are treated as not found. Valid publishes keep their subjects and payload bytes. * fix(eks): bind webhook token review to the caller's cluster WebhookTokenReview took its account from the request body and was authorised by IAM policy alone, with the control-plane role granting it on "*". A control-plane VM, any tenant holding eks:*, or static system credentials could resolve a token against another cluster or account and learn the Kubernetes groups it maps to there. Error responses also distinguished absent accounts from absent clusters. WebhookTokenReview now passes the same internal-route gate as PublishInternal: a control-plane instance-role session that is a member of the URL cluster in the body account. The gate and the handler share one body decoder, each body-scoped internal route must declare its reader, and every unauthorised target returns the same AccessDenied without reaching STS verification. Same-cluster responses are unchanged. * fix(eks): validate add-on serviceAccountRoleArn syntax CreateAddon and UpdateAddon stored serviceAccountRoleArn without checking it was an ARN, and the guest renders the value into the add-on's manifests. A non-empty value must now be syntactically an IAM role ARN (12-digit account, no region, role resource, IAM name and path rules); anything else returns InvalidParameterException before any record is written or manifest staged. The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at the gateway remains the permission and existence check. Stored values are not rewritten, and an UpdateAddon that omits the field still succeeds on a record holding a legacy value. * docs: record EKS internal-route and role-ARN security corrections * docs: add EKS add-on release membership and activation case study * fix: point UI build, CI and lint rules at moved package paths * docs: record EKS add-on activation review decisions * refactor: move quota admission to domains/admission/quota Relocate the quota service, live counters and vCPU records from handlers/quota to domains/admission/quota. Package renamed from handlers_quota to quota; callers with a local quota identifier import it as admissionquota. KV buckets, subjects and error codes are unchanged, and the package's existing debt edges move with it unchanged. * docs: reconcile package-boundary record for ADR-0001 close-out Add the missing slice rows (the commits that dissolved spinifex/types, the ECR repository projection, the stale-path fix and the quota move), correct stale entries, record the dependency-rule breaks found in the import graph with removal conditions, add the close-out table of remaining legacy paths with their decided target owners, recount the mulga-side follow-ups, and fix stale old-path references in docs and comments. * test: move KV migration engine tests to foundation/state/migrate The Registry and RunKV tests exercise foundation/state/migrate exclusively but lived in spinifex/migrate's test file. Relocate them, with their NATS helpers, to the package that owns the engine. No test name, assertion or behaviour changes. * refactor: move config and object-store upgrade workflow to operator/upgrade Relocate spinifex/migrate, the operator-driven config-file and object-store upgrade workflow behind spx admin upgrade, to operator/upgrade. Package renamed to upgrade (imported as opupgrade where a local upgrade identifier exists). It stays distinct from foundation/state/migrate, the generic KV migration engine; no registry types, migration names, versions, file formats or command names change. * docs: record migrate move to operator/upgrade * test: enforce ADR-0001 layering with a blocking ratchet Add spinifex/architecture/layering, a test-only package that parses the module's import graph, maps every production package to an ADR-0001 layer and checks each dependency rule as its own subtest. Existing violations are allow-listed against their recorded debt; a new forbidden edge, an unmapped package, a stale allowlist entry or a new package under a frozen legacy root fails local preflight and CI. Record the nine edges the test found missing from the migration record (quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2 cross-domain, installer UI -> admin), name the importers of the utils residuals and gateway/ec2_idempotency.go, and correct the handlers/iam consumer list. * refactor: drop the legacy handlers_ecr import alias Import domains/ecr under its package name, or as ecrdomain where the file also imports the AWS SDK ecr package, matching acmdomain. Also fix a stale daemon file reference in the integration ECR daemon comment. * chore: mark spf13/pflag as a direct dependency operator/cli/wiring.go imports pflag directly. Same version; no upgrade. * docs: record test-package-check and nilaway as unverified gates * refactor: move the layering test to module-root architecture/ The ADR-0001 layering ratchet is repository governance tooling, not Spinifex product code, so it sits beside tests/ rather than under the spinifex/ product tree. make test, test-cover and test-race now include ./architecture/..., and the graph walk excludes architecture/. * fix: update paths left stale by the package moves The MariaDB image stamp test read the RDS engine from handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads domains/rds/engine/mariadb.go. Comments citing moved files now name their new locations. * refactor: move the default-VPC subject into contracts/ec2/v1 utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the request/reply route between gateway account creation and the daemon. Only the subject moves; the request and reply bodies are still anonymous structs on each side and are recorded as such in the contract README. The layering freeze now allows two utils files. * docs: qualify the migrate split and record admin and Viperblock findings The migrate row is a split, not an exact ADR-0001 map row, and needs a map clarification. Correct the stale admin consumers, record the operator-only admin files as movable, the vbwire prerequisite for the Viperblock adapter and the capability decision encryption.go needs. * refactor: move operator-only admin code to operator/admin Config generation, certificate issuance, host discovery, telemetry, email validation and image import/removal/orphan-volume tooling move to operator/admin. Legacy admin keeps only the symbols with non-operator consumers: default account identifiers, AWS credential generators and system image promotion. foundation/pki and the tenant ACM CA are unchanged. The readAMI helper and the removal test fixtures are duplicated privately in legacy admin rather than shared across the split. The now-stale operator/installer/ui -> admin layering allowlist entry is removed. * docs: record operator/admin split in package-boundary migration * test: pin viperblockd fence, abandon and dirty-marker wire shapes Characterization tests against the current vbwire package, before it moves to contracts/viperblockd/legacy/v1: both subject forms for the fenced and abandon routes, golden JSON for VolumeFencedEvent, VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty bucket name, and the volume-key pattern shared by the lease and dirty stores. * refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts Move the fence and abandon subjects/DTOs, the dirty bucket name, DirtyRecord and the shared volume-key pattern out of vbwire and into contracts/viperblockd/legacy/v1, alongside the package's existing mount/unmount/delete/sync/config-update routes. vbwire keeps only the I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the contract vocabulary instead of defining it. Record the approved dirty-marker ownership boundary in the contract README: the Viperblock adapter is the sole producer, viperblockd was only its transitional custodian. Document the fence and abandon routes and the dirty bucket in docs/service-interfaces.yaml. * refactor: split the Viperblock EBS adapter out of services/viperblockd Move the EBS provider adapter (volume lease, fence, dirty marker, nbdkit recovery, provider handlers, encryption key loading) to providers/ebs/viperblock. services/viperblockd keeps only process lifecycle and the NATS subscription wiring that starts and stops it. The dirty-bucket read path moves to providers/ebs/viperblock/dirty as a read-only ListUnsealedVolumes capability; the adapter's write, clear and purge logic stays unexported in the adapter package. Delete services/viperblockd/vbwire now that its vocabulary lives in contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty. Update operator/cli, the runtime service registry, qemunbdd's config comment and the integration test harness to the new import paths, and drop the now-stale architecture/layering allowlist and frozen-package entries for the deleted package. Behaviour, persisted keys and NATS subjects are unchanged. * refactor: move the viperblockd role to runtime/roles/viperblockd With the EBS adapter in providers/ebs/viperblock, services/viperblockd holds only role supervision and the opt-in live data-path tests, so it joins the other runtime roles. It was the last package under services/; the frozen legacy list drops it. * docs: record viperblockd contract, adapter and role moves * test: characterise vpc.* network-realisation routes Pin every vpc.* subject, JSON payload shape and the request/reply {success,error} ack envelope exactly as they exist today, ahead of introducing contracts/network/v1. Covers the 22 subjects registered by domains/network/subscribers, the vpc.igw-attach/detach subjects and InternetGatewayEvent already owned by contracts/ec2/v1, the utils.AddNAT/PublishNATEvent client wire behaviour including the delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent and SGEvent/SGRule duplicates in domains/ec2/vpc. Also adds the six vpc.* route-table and system-egress subjects that were missing from service-interfaces.yaml despite being live production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet- egress, vpc.add/delete-system-egress). * feat: add contracts/network/v1 for vpc.* realisation routes Declare the versioned wire vocabulary for the vpc.* control-plane-to- network-realisation boundary: subjects, payload types, the shared {success,error} ack envelope and the vpcd-workers queue group, one file per route family. No NATS I/O, no consumer switched over yet; the package coexists with the fragmented originals pinned in the prior characterization commit until slice 3 migrates call sites. * refactor: use contracts/network/v1 for vpc.* producers and consumers Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw, routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the network/subscribers consumer onto contracts/network/v1 subjects, event types and the shared AckEnvelope. contracts/network/v1 becomes the sole owner of vpc.igw-attach/detach, replacing the duplicate definitions in contracts/ec2/v1. Deletes the duplicated local subject constants and event structs: subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent, eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent. domains/ec2/vpc.SGRule survives because EC2 persists it directly in SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded in slice 2 to match it field-for-field so wire bytes are unchanged. No lifecycle or reliability changes: the NAT add barrier, delete retry-on-ErrNoResponders, and the inconsistent per-call-site NAT delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay fire-and-forget with no ack) are preserved exactly, as are the golden wire literals pinned in the characterization tests. * refactor: replace utils/vpcd_event.go with a typed network projection client Every vpc.* call site now goes through domains/network/projection's operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent or utils.RequestEvent, so no caller passes a bare subject string. Each method states today's delivery mode by name: AddNAT stays strict, AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those call sites already had, and RemoveNAT keeps the ErrNoResponders retry. utils/vpcd_event.go and its tests are deleted; the behaviour they pinned (wire shape, ack envelope, retry-on-no-responders) moved into the projection package's own golden and nat tests, unchanged. utils/ now holds only encryption.go, which this does not touch. * refactor: route the remaining fire-and-forget vpc.* producers through projection CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/ DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/ RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress were still publishing through natsmsg.PublishEvent or a raw nc.Publish call with a bare subject string, left over from the mechanical slice 3 pass. Each now goes through its named projection.Client method instead, so every vpc.* producer in the tree uses the typed client. * refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils The master-key loader was used only by EC2 image, instance and volume to derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled, EC2's temporary reading of the provider's key-file setting; the cache, empty-path meaning, error text and each caller's error handling are unchanged (image still reports unencrypted when the key will not load). Its successor is a providers/ebs capability reporting volume encryption posture. utils/ is now empty and deleted. The layering test drops utils and services from the legacy roots, removes their allowlist entries and fails if either retired root regains files. The S7 subject scan follows the vpc.* literals from utils to contracts/network so its gap list stays complete. * docs: record the network contract, projection client and utils removal * test: characterise instance-state persistence, recovery and generation handling Pin the current local-file bytes and schema failure modes, the instance bucket configuration, record key and wire form, the bucket version gate, generation preservation across every KV writer and the VM conversion, and what a returning node does for a missing, foreign, terminal or unreadable canonical record. These describe current behaviour, not the ADR-0007 target. * test: freeze direct instance-state consumers Add an AST ratchet over production code that fails when a new file names the daemon's instance-state bucket, record keys, local-file API, the raw vm.InstanceRecord or its conversion, or a JetStreamManager instance-state method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0 inventory; stale entries fail, and a synthetic-source self-test proves the matcher detects each kind of use and ignores declarations. * docs: record ADR-0007 slice 0 instance-state inventory Record the instance-state persistence formats, the nineteen direct consumers by process and access kind, the characterised recovery and schema outcomes, the generation-preservation matrix and the ratchet that freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays blocked. * test: narrow the instance-state ratchet to instance-state access Stop freezing the JetStreamManager type and constructor, which also serve cluster state; flag only instance-state methods, bucket and key identifiers, record codecs and local-state access. The cluster-state-only consumers in daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67). The self-test now proves a cluster-state use is ignored and an instance-state method call is caught. Record the owner's disposition of the KV-unavailable restore result in the migration record. * docs: record the narrowed instance-state ratchet commit * docs: design the instance-state transition record and activation gate * docs: record the instance-state transition-record design commit * docs: record slice 1a activation-gate decisions * feat: add instance transition-record codec with format marker Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner package. A top-level format member selects the shape: absent or 1 decodes the legacy record and is never given a marker on write; 2 also decodes the new assignment and observation sections; an integer above 2 decodes its legacy fields read-only and Encode refuses it; anything else is refused on decode. Encode applies only the typed paths that differ from the decode-time snapshot onto the input, so unknown members at every object level and unchanged known fields keep their read form. A changed array is replaced whole. Decode refuses malformed JSON, duplicate keys at any depth, keys that alias a known field by case, trailing data and a non-object record, each with its own sentinel. No production reader or writer uses the codec yet; the new types are not part of vm.InstanceRecord, so every existing record encodes unchanged. The codec file is registered as an owner file of the instance-state ratchet. * docs: record slice 1b step 1 codec and awsgw rollout qualification Records the codec added in e62f921d2 as design Section 10 (C1 to C7): its surface, preservation at every object level, the changed-array limitation, marker handling, the rejection classes and its capable-writers-only scope. C2 supersedes P1's top-level-only preservation for the codec. G19 records that old read-only gateway consumers remain safe but DescribeInstanceStatus gains unconfirmed/degraded visibility only after awsgw upgrades. The migration record gains the commit row and states that no production path uses the codec yet. * refactor: move the instance transition-record codec to domains/ec2/instance/state The codec defines how the canonical persisted instance record evolves, with its assignment and observation sections, which is EC2 instance ownership rather than node-local VM supervision. It moves unchanged into its own package and keeps wrapping the legacy vm.InstanceRecord alias and its spec and status types, which stay in runtime/compute/vm for now. The domain-to-runtime import is recorded as new dependency-rule debt with its removal condition and allow-listed against that entry. The instance-state ratchet now names the new file as an owner file; its allowlist is unchanged. * fix: refuse array changes that would drop element extensions A changed array is replaced whole by the writer's value, which silently lost any member a newer writer had added inside its elements. Encode now inspects the array as read: if any element carries a member its typed schema does not know, at any depth, it returns ErrRecordExtensionConflict naming the array and the first such member. Arrays without such members are still replaced whole, unchanged arrays are still kept verbatim, and arrays inside unknown members are never inspected. * docs: record the codec move and the array extension-conflict rule Records 8db9771fb and c617914a1. T1 and Section 10 now name domains/ec2/instance/state as the eventual owner of the persisted instance record and its evolution, with vm.InstanceRecord as the temporary legacy type it wraps. C3 replaces the changed-array limitation with the refusal: a changed array with element extensions fails with ErrRecordExtensionConflict and is never silently dropped. P2 stays a conservative rule until finalisation. * fix: refuse removals that would drop record extension fields A capable writer may no longer remove or replace a known object, map entry or array when its stored value carries members the writer's schema does not know. The check that guarded changed arrays now runs wherever the merge drops or overwrites a stored value instead of merging into it: a known member deleted or set to null, a map entry removed or stored under a key the typed decode rewrote, a value of another JSON kind, and a changed array. Encode returns ErrRecordExtensionConflict naming the changed container and the first unknown member. Removal stays allowed when the stored value has no extensions, and a change merged member by member still keeps unknown members. * docs: record the uniform extension-conflict rule Adds design clause C8 for 0d7b4266d: a capable writer may not remove or replace a known object, map entry or array whose stored value carries unknown structured extension fields; removal without extensions stays allowed. C3 and C5 point at it. The migration record gains the commit row, updated subsection wording and a pointer-bump follow-up deferring make docs-portal to the Mulga gitlink PR so the portal evaluates the proposed Spinifex revision. * refactor: inject vpcd's IMDS instance-state readers from the instance state boundary * docs: cite the vpcd reader injection commit in the migration record * refactor: move vpcd to runtime/roles/vpcd * docs: cite the vpcd move commit in the migration record * refactor: move the ECS agent wire vocabulary to contracts/ecs/v1 Move handlers/ecs/bus to contracts/ecs/v1 (package ecsv1) with no legacy shim. Subject strings, JSON fields and types are byte-identical; the old-agent compatibility test moves unchanged and golden pins cover the remaining message types. The README route table records that only service-reconcile has a live producer today; the agent reuses these payload types through the gateway actions. service-interfaces.yaml lists ecs.bus.* under the daemon. * refactor: move system-image promotion to domains/ec2/image Move images_promote.go and its tests from admin to domains/ec2/image: SystemOwnerAlias, PromoteImageOpts/PromoteImageResult, PromoteSystemImage, GetAMIMetadata and the private snapshot reader. Storage, ordering, errors and promotion behaviour are unchanged; the daemon, gateway/spx and operator CLI callers import ec2image. admin now holds only account IDs and key generators. * docs: record the ECS contract and image-promotion moves and the remaining-roots matrix * refactor: move the bootstrap account identifiers to foundation/aws/identifiers DefaultAccountID and DefaultAccountName join GlobalAccountID as the fixed, platform-reserved identity of the first human-facing bootstrap account. Names and values are unchanged and stay fixed. Callers in daemon, gateway, awsgw, accountteardown and the operator CLI use awsidentifiers; admin now holds only the IAM credential generators. * docs: cite the bootstrap account identifier move in the migration record * refactor: inject the STS cluster JWKS lookup from handlers/eks handlers/sts declares ClusterJWKSResolver and keeps JWKS decoding and InvalidIdentityToken mapping; handlers/eks implements it as OIDCJWKSReader beside the bucket and key layout helpers. awsgw and the integration harness inject the reader. STS no longer imports any EKS package, including tests, and the handlers/sts -> handlers/eks layering exception is removed. * docs: cite the STS JWKS resolver cut in the migration record * refactor: move the AWS credential-format generators to foundation/aws/credentials and retire admin GenerateAWSAccessKey and GenerateAWSSecretKey move unchanged to foundation/aws/credentials as a stateless format primitive. handlers/iam and operator/cli import it directly. The admin legacy root is deleted and joins the retired roots so it cannot return. * docs: cite the admin retirement in the migration record * revert: remove the uncalled ADR-0007 codec and design stack from the ADR-0001 checkpoint The instance transition-record codec, its tests and the slice 1a design had no production caller. They leave this package-boundary checkpoint and will be reapplied on a dedicated ADR-0007 branch. Slice 0 characterisation and the vpcd IMDS readers in domains/ec2/instance/state stay. * docs: cite the ADR-0007 codec revert in the migration record * fix: declare the ECS action subjects in service-interfaces.yaml Declaring ecs.bus.* made ecs a managed prefix for manifest-lint, which then failed on the 34 ECS action subjects the daemon registers. Declare them as ecs.*, the same wildcard convention as rds.*. * chore: run manifest-check and manifest-lint in preflight CI runs both, but preflight did not, so manifest drift could pass local checks and still fail Build and Test. * docs: record resumed preflight enforcement and dev merge ports * fix: restore dev changes dropped by earlier merges Reapplies the slices.ContainsFunc rewrite in nodeFitsCustomerInstance and the DeleteDBSubnetGroup doc comment, both lost when their files were refactored. * docs: record accountteardown as a frozen legacy root pending cleanup capabilities --------- Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-10-15 14:42:53 +10:00
Julian Sommer	28963e28af	feat: verify local image imports with --checksum and record AMI source digests (#1102)	2026-09-23 15:15:23 +10:00
Ben Duncan	828b5df4aa	fix: pause a guest on a backend I/O error instead of handing it EIO (#977) ... * fix: fail DRAIN when a volume seal failed shutdownAndUnmount logged the volume unmount error and returned nothing, so StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set. The coordinator then advanced to STORAGE and stopped viperblock while the seal was still in flight, tearing the VBState envelope and leaving the guest unable to mount its root volume on the next boot. The deploy reported success. shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup and stopOne carry it without aborting the rest of the teardown, since QEMU is already down by then, and StopAll aggregates the per-VM failures so handleShutdownDrain ACKs with Error set and never reports vms_stopped. volumeMounterAdapter.Unmount was the second swallow site: it logged each failed seal and returned nil unconditionally. It now attempts every volume and returns the aggregate, which is what the VolumeMounter contract already described. Terminate still tolerates a failed seal — the volumes are deleted next, so an unsealed block map loses nothing a caller can ask for. This does not fix the seal itself. It converts silent data loss into a refused upgrade that an operator can act on. * test: add the storagefault e2e suite for backend-outage guest survival Freezes predastore with SIGSTOP to hold connections open while answering nothing, which is the fault a timeout-free path cannot distinguish from a slow peer. On one node that is a total outage; on three it freezes every node but the guest's own, breaking the RS(2,1) shard floor. The suite writes and flushes a verifiable region before the fault and churns a separate unverified region during it, so an interrupted write is never mistaken for lost data. The root-filesystem load runs from the user's home rather than /tmp, which is tmpfs on this image and would move no bytes on the root device. * test: verify the root device is under write pressure, and size the outage The previous guard matched /^ *vda / against /proc/diskstats, where the device name is the third field, and read $7 which is ms_reading rather than sectors_written. It printed empty and asserted nothing, so a green run could not distinguish a surviving guest from an idle disk. It now samples sectors_written twice and fails unless it advances. The outage is sized from viperblock's absorption rather than guessed: a workload must fill DefaultMaxPendingBytes before backpressure engages at all, so a short freeze measures the retry budget rather than the fault. * test: add nbdkit fault injection to the storage-fault e2e suite The predastore freeze takes the whole storage cluster away from every guest on it, which is too broad to run beside anything else. Removing a single volume's NBD server is the same fault at one-volume scope: one host, one guest, nothing cluster-wide stopped. Both signals are here because they sit on opposite sides of the defect. SIGSTOP holds the socket open and answers nothing, so the guest blocks and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at the default reconnect-delay of 0 fails requests at once, which under werror=report aborts the journal. Signalling is guarded: a pid is matched on comm, our plugin and the volume argument together, and re-confirmed immediately before the signal, because these hosts run nbdkit for other volumes and pids are reused. * fix: pause a guest on a backend I/O error instead of handing it EIO Every virtio-blk device was created with werror=report,rerror=report, so a backend outage was delivered to the guest as EIO. ext4 aborts its journal on that and remounts read-only, and the write that failed is simply gone. Under werror=stop QEMU pauses the VM and holds the failed request, so a backend that comes back resumes the guest with nothing lost. The boot drive, the cold-boot data volume and the hot-plugged data volume all move to stop. Alongside it the NBD transport gets reconnect-delay=30, which pauses requests rather than failing them while a storage process restarts on the same socket; on the legacy -drive form it has to be file.reconnect-delay, because the bare name reaches the raw format driver, which has no such option. The QMP heartbeat now recognises run state io-error and issues cont. After ten attempts the instance is reported impaired and the cadence drops to one attempt per ten heartbeats, but it never stops: QEMU holds the request for as long as the guest exists, so a backend that returns an hour later can still be resumed. The storagefault suite gains the fault this was written for — kill the volume's own nbdkit, hold, then put it back on the same socket — and it now verifies a sha256 region across the outage rather than trusting the guest to be reachable. Three harness defects went with it: the socket-liveness poll ran unprivileged against a 0770 directory and could never succeed, the relaunched socket was left 0755 where QEMU needs group write, and the two variants shared one guest so a fault in the first was diagnosed in the second. * test: cover the io-error resume policy * test: stop two parallel tests racing on the terminate backoff variable * fix: report a guest paused on a backend I/O error as impaired A guest held by werror=stop kept reporting ok on both status checks. QMP stays responsive while the guest is paused -- cont is being sent and answered -- so the QMP failure gate that DescribeInstanceStatus keys on never trips, and the one state a customer most needs to see looked healthy. Instance status now reports impaired/failed while the resume budget is non-zero, timestamped from IOErrorSince so impairedSince is when the guest paused rather than when it was escalated. System status reports impaired too: a backend I/O error is an infrastructure fault, not a guest one, and that distinction is what the two checks exist to draw. The counter is cleared by the first poll that sees the guest running, so both clear on recovery. * test: reap the relaunched nbdkit when the fault test ends relaunchNbdkit detaches the server with setsid so it outlives the SSH session that started it, and viperblockd never learned it exists. Nothing reaped it, so every run left an nbdkit holding a socket and a cache for a volume that was later detached or deleted -- four were found running on dev-prod against two detached and two deleted volumes. The PID is resolved from the socket path rather than the volume id, so a legitimate viperblockd server for the same volume is never the one killed. * test: prove the backend is serving before a fault is called cleared Thawing the process a fault stopped is not the same as restoring the service. The outage test treated SIGCONT as the end of the fault and went straight to judging the guest, so it passed while the cluster it ran on stayed unable to serve a single object for the next 35 minutes, and the next test would have inherited that. thawPredastore now puts and gets an object through every gate, with a budget, and fails the test if the backend does not come back. * fix: do not hand back a mount entry whose export is gone A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit was gone. Every later ebs.mount then matched that entry and returned its socket path, so vmMgr waited for a socket nothing would ever create and the instance could not be started again — on dev-prod a guest stopped against a frozen backend was left permanently stopped, four start attempts failing identically on a socket the node had already removed. Drop the entry when there is positive evidence the export is dead: the socket is absent, or the recorded pid is not running. Neither can be true of a live export, so the double-writer hazard the guard exists to prevent is unaffected. Also make the durability test's pattern helpers report failures. The read dd sat in a pipeline, so its exit status was discarded and the harness hashed nothing and returned the empty-string sha256 as a checksum. * test: report the guest console when a restart never answers SSH A restart that reaches running but never comes back is indistinguishable from a dead root filesystem without the serial console, and the log is gone once the fixture terminates the instance. * fix: converge guest port datapaths concurrently, not one deadline after another Each guest port that misses its SB binding burns the full convergence deadline, and applyEIPs spent them in series. Three ports belonging to long-stopped instances therefore held a newly launched guest's EIP dark for the sum of them before its own probe started. AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and sb-cluster-state-reset is now serialised with a minimum interval so concurrent probes cannot each fire a cluster-wide resync. * fix: refuse a stale cross-node start after the owning node is gone The volume lease already refused a cross-node mount, but it is a liveness claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded writes stops renewing, and 45 seconds later nothing refuses a start elsewhere that loads a stale backend checkpoint. A failed seal now writes a durable marker naming the node holding the only current copy, cleared on a seal that reaches the backend and on delete. The lease covers a live holder, the marker a dead one. The refusal also reaches the caller instead of being flattened to ServerInternal, which told an operator to retry something that cannot succeed while the holder is down. * fix: keep the daemon's message when a start is refused Rebuilding the error from the code alone rendered the generic ErrorLookup text, so a refusal naming the node holding the volume reached the caller as boilerplate about attaching volumes. * fix(ec2): terminate detaches every DeleteOnTermination=false volume TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume on the assumption that Stop's Unmount had already cleared its attachment. Unmount only clears it when the seal succeeded, so a volume whose seal failed stayed in-use pointing at an instance that terminate then deleted, leaving it unreachable with no owner to release it. The running-instance path already detaches every DoT=false volume; the stopped path now matches it, and tolerates an already-gone volume the same way. DeleteVolume also keeps its provider error on the way out. The terminate reaper retries the delete every two minutes and logged a bare ServerInternal, which made a blob node being down indistinguishable from a transient fault. * fix(ec2): DetachVolume --force releases a volume whose instance is gone The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a volume left attached by a part-way teardown could never be detached: the instance no longer runs anywhere and nothing answers. That also made it undeletable, since DeleteVolume refuses an attached volume. ForceDetachVolume already existed for this deadlock but was reachable only from account teardown. Force now falls back to it when the instance has no responder, which is the one case where clearing the attachment cannot strand a live guest. * fix(viperblock): unsealed writes steer placement instead of pinning a volume The dirty marker refused any mount from a node other than the one holding unsealed writes. Instance start already forwards to the node that last ran the instance and falls back after a 30s window, so that refusal turned a working fallback into a volume that could run nowhere once its owner was gone. It traded a rare data-loss risk for a permanent availability failure. The marker is now a record and a warning, not a gate. A takeover logs the previous holder and why it could not seal, then adopts the marker so the old node is no longer treated as the better source. Live exclusion is unchanged and still enforced by the lease. It is also written on open rather than on seal failure. A node killed mid-write never reaches its seal, so the old trigger left no trace in exactly the case the marker exists to describe. Adds 'spx admin volumes unsealed' to see which volumes would cost something if started elsewhere. * fix(viperblock): fence a volume whose lease moved instead of logging it The lease was checked once, at mount, and never again. A node that lost it kept its export up and kept writing into a volume another node owned, so both copies advanced and the backend settled it by arrival order. Losing a renewal now decides: another holder means tear the export down, no holder means the entry merely aged out, so reclaim it and leave the guest running. Fencing never seals — this node's copy is the stale one — and leaves the dirty marker for the winner to take over. This is not an epoch fence. It narrows the window from the 45s TTL to one renewal interval at no write-path cost; the backend still cannot refuse a stale writer. * fix(network): advertise EIPs so a recycled address is reachable The gateway switch port only asked ovn-northd to publish the router's NAT addresses in centralised and routed modes, so on a distributed cluster nothing ever announced an elastic IP. OVN still answers ARP for it, which is why a first-time allocation works and hid this — but no gratuitous ARP is sent when the address moves to a new ENI, so every host on the segment keeps resolving it to the previous holder's MAC until its cache ages out. Reusing an address from the pool then lands on a dead MAC and the instance is unreachable from outside. Distributed mode needs the option as much as the others: northd scopes each entry with is_chassis_resident(<vif port>), so the announcement leaves from the chassis running the instance rather than the gateway, which is what distributed NAT wants. Converge on the already-attached path too. AttachIGW returns early once the gateway port exists and nothing else rewrites it, so without that an existing VPC would stay silent for its whole life. Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to carrying their correct external_mac, each scoped to its own ENI's port. * fix(viperblock): make the volume fence and its dirty marker ownership-safe Four corrections to the lease-loss fence, all found in review. Drop the reacquire path. Losing a lease and finding no successor was treated as a lapsed entry worth reclaiming, but leases.acquire returns the cached entry — already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease renewed nothing and the node went on writing a volume it could not prove it owned. The two errors are not symmetric: a false fence costs one guest a restart, a false reclaim leaves two nodes writing one volume. Fence in both cases, and report which happened through the outcome attribute. Surrender a lease this node cannot confirm. Renewal only noticed a loss when the server rejected an update, so a node partitioned from NATS kept its export up while the TTL expired and another node took the volume. Track the last confirmed renewal and give the volume up once that ages past the point where the server may admit another owner. Bound the renewal call too, so a hung JetStream request cannot hold the goroutine past that point. Do not act on a kill that did not take. Both the fence and ebs.unmount removed the mount entry, released the lease and announced the result before knowing nbdkit had exited — the fence then reported a writer stopped that was still running, and the unmount sealed into a directory a live process was writing. Wait for the exit, and on failure leave the volume mounted and say so. An already-exited process now counts as success, which is what the caller asked. Give the dirty marker a generation. It recorded an owner and nothing to order two nodes' claims, so a node returning after a takeover could clear or overwrite the winner's marker and point placement back at the stale copy. Stamp the lease generation, refuse a write from an older one, and only clear a marker this node still owns. A marker that cannot be written now fails the open rather than leaving writes nothing records. * test(viperblock): cover the stall surrender and the unmount kill guard The two ownership cases the fence work left untested. A holder partitioned from NATS while the winner keeps it never gets an update rejected, so revision checking cannot see it — it has to give the volume up on its own clock. Pins that validity sits below the server TTL with room for a check in between, that a lapsed holder reports the loss as a stall rather than as a takeover, and that releasing a surrendered lease leaves the entry for whoever holds it now. The unmount side of the kill guard: sealing under a live nbdkit writes a state torn between two writers, so a kill that did not take has to fail the unmount and leave the entry mounted for a retry. * fix(e2e): keep the guest private key for as long as the key pair EnsureKeyPair computed the PEM path from the caller's directory but wrote the file inside the ensureOnce closure, which runs only on the first call. Callers passed a per-test artifact dir, and ArtifactDir removes that dir when the test passes — so the first test's success deleted the only copy of the private key while the key pair stayed registered, and every later call returned a path nothing had ever been written to. Guests after the first were then launched against a public key whose private half no longer existed, so the harness could never authenticate and gave up after the full five-minute budget. The guests were healthy and reachable on tcp/22 throughout, which is why this read as a storage or network fault for so long. The directory parameter was the defect, so it is gone: the PEM now lives in a directory the fixture owns and removes with the key pair, and no caller can supply one that is shorter-lived. Also reorder the killed-nbdkit teardown. The reaper registered its own cleanup after the detach registered one, so LIFO killed the block server before the platform detached the disk and DetachVolume then failed with "Node ... is in use". The detach cleanup now reaps once it is done. * fix: stop leaking an nbdkit export when recovery cannot adopt it Restart recovery adopts a guest's boot volume, then routinely fails the -efi volume with "claim lease: context deadline exceeded" -- JetStream is still catching up at daemon start and the KV claim runs on the client's 5s default. Recovery gave up on the first error and never retried, so the export stayed live with no registry entry. At terminate the unmount found nothing to seal, returned NotFound and killed nothing, and the process outlived its guest holding the volume and socket open. EFI is always the second volume adopted, which is why it was always the one that leaked. Two changes, because either alone leaves a hole: - rebuildMountedVolume retries the build (4 attempts, 3s apart) so a JetStream that is merely slow no longer orphans an export for the life of the process. - unmountVolume reaps a corroborated export for the volume it was asked to unmount when the registry has no entry for it, so an export orphaned by any means is still cleaned up. The reap goes through reapOrphanedNbdkit, which declines anything a live process references: at unmount the guest's blockdev may not be torn down yet, and killing the server underneath it strands QEMU. * test: hold the backend-serving probe and assert predastore readiness The backend-outage test declared the backend restored on one successful round trip through each gate. A gate can answer the request that arrives just after a thaw and then lose its blob peers under the load that follows, which is a backend that never recovered reported as one that did -- and that is what happened: the probe reported all three gates serving minutes before the guest was still taking BLOCK_IO_ERROR on every resume. The round trip has to hold across three rounds now, and the count restarts rather than decays when one fails, so a gate that serves and then stops is the failure the hold is looking for rather than noise it averages out. It also checks /readyz on each host and reports any failed check by name. The round trip only proves this workstation can write through a gate; it says nothing about which blob peers that gate can reach, and a gate one node short of the write floor serves a small object and fails a guest. A host that serves no admin listener is logged and skipped, since the port is optional configuration. * fix: scope predastore's admin port to the cluster plane in the host policy predastore's admin listener binds the host's cluster bind_addr and serves /healthz and /readyz unauthenticated. Readiness now names which blob peers a gate cannot reach, which is exactly the kind of topology detail 4432 is peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the public one. * fix: keep resuming a paused guest at heartbeat rate after a long outage Past ten attempts the cont dropped to one per ten heartbeats, so a guest whose backend had come back could sit paused for another five minutes before anything retried it. Measured on dev-prod: the backend was proved serving through all three gates while the guest was still waiting on attempt 60, five minutes after attempt 50. The slowdown was there to avoid paying a failed I/O every thirty seconds on a backend still down. That is nearly free, and five minutes of downtime after recovery is not, so only the log line is throttled now. * test: fail the run when a key pair name resolves to more than one pair A create that lands twice leaves two key pairs with one name and different material. RunInstances resolves the name to one of them while the harness holds the private key of the other, and the only symptom is every guest in the run refusing the key at the SSH gate — five minutes of waiting per guest, four guests, and nothing anywhere naming the cause. Observed on dev-prod: two pairs created a millisecond apart, both named e2e-key-30de10ba. One DescribeKeyPairs after the create says so outright instead. * test: tolerate an already-released volume in the detach teardown Teardown runs DetachVolumeWait after a terminate that releases the volume itself, so the detach can arrive at a volume already on its way to available. That is the outcome it wanted, and treating the IncorrectState refusal as fatal failed a backend-outage run whose assertions had all passed — the guest came back, every acknowledged write verified, and the test still went red on the way out. VolumeInUse is still fatal: that one means the volume is attached somewhere it should not be. * test: carry the merged suites onto the reworked key-pair and account-scoped seams The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir this branch removed, and DeleteVolume's metadata lookup is account-scoped now, so the provider-failure test has to seed and delete under the same account. * refactor: make the fence outcome a type in one const block Four separate exported string constants with a paragraph of comment each, passed to RecordVolumeFence as a bare string. Group them, give them a FenceOutcome type, and take that type at both call sites so an arbitrary value can no longer reach the metric attribute. * refactor: move viperblockd's wire contract into a leaf vbwire package The daemon imported viperblockd for exactly two symbols -- the fenced volume subject and its event -- which put the storage service back on the daemon's dependency graph to name something it only listens for. vbwire holds the contract instead: the subject and event, the dirty bucket name and record shape, and the read-only listing an operator tool needs. It imports nothing from spinifex, so both the daemon and the admin CLI drop their edge to viperblockd entirely. cmd/service.go keeps its one, which is genuine: it constructs the service. * docs: drop a specific host name from the start-refusal comment Named a real dev-prod host in a comment and in test fixtures. Say what the name is instead. * ci: give storagefault a dedicated nightly cell The suite is eligible for both topologies and runs on the PR gate, but no nightly cell ran it: the permutation cells run a deliberately narrow subset and none of the dedicated suite cells named it. Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that environment would report the outage as its own failure. * ci: run storagefault nightly on three nodes, not one The previous commit put its dedicated cell on a single-node runner. That was the wrong topology and the wrong justification: single-node is RS(1,0), where the shard floor cannot be broken and a cross-node takeover after a failed seal does not exist, so the two assertions worth having a dedicated cell for were the two it could not make. Cell 29 is now real-multi on ci-multi. It also corrects the claim that the PR gate already covers the multi-node case. The suite is eligible for both topologies, but e2e.yml's default suite list never names it, so it has never run there. A full pass is ~63 minutes on dev-prod bare metal, almost all of it deliberate waiting -- one test holds predastore SIGSTOPped for seven minutes -- which is too long to add to a gate that currently finishes well inside its budget. Nightly is where it belongs. Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since 90 minutes of suite cannot run in a 50-minute job; the permutation steps gated on matrix.suite == '' so the dedicated cells do not also run them; and a Run Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts the suite against E2E_SUITES_MULTI before running it. * fix: stop a startup KV race leaving vpcd serving keyless guests vpcd opens five KV buckets while NATS is still coming up, each attempt carrying a ~5s JetStream API deadline, and applied three different policies to the same failure: spinifex-vpc-enis was fatal, while instance-state, security-groups, subnets and vpcs each degraded silently for the lifetime of the process. A five-second blip during a cold start therefore decided how a node served metadata until someone restarted it. On dev-prod all three nodes hit this after a deploy. Two were saved by accident: their ENI bucket also timed out, so they exited 1 and systemd restarted them cleanly. The third's ENI bucket opened in time, so it never crashed and ran for hours with no instance record loader. With the local state file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's crawl died on ami-id and fell back to DataSourceNone, and every guest launched on that node booted with no authorized_keys. kvstore.OpenWithRetry retries for a bounded window before any policy applies, logs the cause per attempt and the wait on a late success, wraps the last error with the bucket name, and returns at once on a cancelled context. All five opens use it. Failing to open the instance record space is now fatal, matching the ENI bucket: the dependency whose loss stops guests logging in should not be treated more leniently than the one whose loss stops ENI resolution. Also fixes the reason the fallback was load bearing at all. The local state file was written 0600 and vpcd reads it as a different user in the shared spinifex group, so the local-first path had never once worked and its mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so an inherited umask cannot undo it. Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an unresolvable instance as "no key pair" with nothing logged. * fix: scope the ime-* firewall accept to IMDS and VPC DNS only The per-tap OVS demux delivers any guest packet addressed to 169.254.169.254 or .253 onto an ime-* host port regardless of L4 port, and iifname "ime-*" accept then admitted all of it. Every wildcard-bound host service — sshd, the AWS gateway, northstar, the console — answered guest packets on those addresses; only asymmetric reply routing kept the handshake from completing, and a guest could still drive northstar to recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS (.253:53), letting everything else fall through to policy drop. * docs: document the instance-to-host plane and how to protect it without the ISO The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by host services and is invisible to a NIC-level or cloud firewall, since the traffic is intra-host on the OVS bridge. Explain that the ISO ships the scoping rule armed, that the binary installer ships the firewall off, and that a non-ISO install must enable it with setup.sh --firewall=on or firewall_enabled=true. Include verification from an instance and the host. * fix: withhold the IMDS meta-data listing when the instance does not resolve Every omission from the meta-data root is a positive claim about the instance — public-keys/ absent means "launched without a key pair", and cloud-init believes it, writes no authorized_keys and boots the guest keyless. So resolving the instance was best-effort for a listing whose whole meaning depends on it: a backend error or a miss shortened the listing instead of failing it, and nothing anywhere recorded that the answer was a guess. Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot retry a 200 that is merely wrong, so this converts a permanent silent failure into a transient loud one. servePublicKeys already had this policy for the leaf; the root is what contradicted it. A profile lookup that fails still serves the listing minus iam/, and now logs. The key pair is the field a guest cannot recover from losing, while a missing role surfaces through the credentials leaf, which reports the miss itself. Also stops the storagefault cross-node test failing on its own access path. It tolerates the pattern changing across a takeover, but readPattern needs sudo, and a volume opened from the backend checkpoint may predate cloud-init writing its sudoers drop-in — indistinguishable from an unreadable device at the dd call site. It now checks sudo first, and reads cloud-init's recorded datasource to tell a rolled-back root filesystem from IMDS never having answered, failing loudly on the latter rather than accepting it as storage loss. --------- Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>	2026-09-03 19:45:33 -05:00
Ben Duncan	8b3c706ea3	docs: rework the multi-node install guide, fix cross-doc links ... Multi-node install - Lead with the fact that the guide builds a three-node cluster, and say how to go beyond three. The callout sits inside ## Overview because the docs site discards everything above the first section heading. - Replace the OVN-only sizing prose with a table covering all three distributed layers, and move the OVN control-plane detail into the VPC networking guide, linked from here. - Add minimum and recommended hardware. - Move the object storage layout to the end as background; it was the second thing a reader hit. - Collapse the ISO conversion notes from a section into one skippable box, now that --force is unconditional in Step 4 and the formation port needs no manual intervention. - Expand verification with spx get nodes and spx top nodes, including what to look for in each. - Drop the install-node.sh shortcut (internal tooling) and the "adding a node later" section (no good story yet). Corrections found while writing - Servers 4+ were described as running no database. They run predastore and viperblock like every other node; only the OVN database cluster is capped at three. Fixed in four places. - Step 4 never needed the firewall down: spx admin init opens the formation port itself for the formation window. Only Step 3's peer-scoped OVN ports are affected. - `nft list table inet spinifex_filter | grep spinifex_peers` could never match — the peer list is an nft define, expanded at load time. Anyone following it would have seen empty output and assumed the worst. - Disambiguate "database": with predastore, NATS and OVN all in play it needed qualifying throughout. - Align the example output region with the AWS_REGION the guide exports. Cross-doc links - Rewrite relative and category-prefixed links to the canonical /docs/<slug> form. The site publishes docs flat, so ../foo/README.md and /docs/category/foo both 404. - Point the six links to unpublished docs at the source repo instead. - Fix four fragments written against GitHub's anchor algorithm.	2026-08-21 14:14:28 -05:00
Josh Siviour	e84753f9b9	docs: bring doc titles and descriptions into SEO length limits (#831) ... * docs: bring doc titles and descriptions into SEO length limits Rewrite the frontmatter title and description of every published doc so the rendered <title> lands in 50-60 characters (including the site suffix) and the meta description lands in 150-160 characters. * docs: move SEO titles to a seoTitle field and restore display titles The frontmatter title renders as the page H1 and the sidebar label, so tuning it for the title tag shortened nothing and lengthened the visible navigation. Put the search-facing string in seoTitle and give title back its original value.	2026-08-19 12:05:59 +10:00
Josh Siviour	b93f6c258a	fix(e2e): resolve the listener inventory without a Go toolchain on the target node (#787) ... * fix(e2e): resolve the listener inventory without a Go toolchain on the target node tests/e2e/multinode/listener_invariant_test.go resolved docs/security/network-connections/README.md at runtime via `go env GOMOD`, which works under `go test` but not inside a compiled .test binary shipped to a node with no Go toolchain. Embed the doc in a new docs/security/network-connections package instead, so the content is compiled into the binary and every caller parses bytes rather than a resolved path. RepoRoot and DocPath are now unused and removed. * docs: trim the embed package comment to the three-line standard	2026-08-13 14:18:02 +10:00
Josh Siviour	57bfbc20df	test: listener invariant static guard + e2e runtime check (#782) ... * test: add the listener invariant static guard and e2e runtime check Adds both halves the plan called for: a static test in spinifex/network/invariants that scans install scripts and config templates for a bare OVSDB ptcp remote, a wildcard bind on a Cluster/Encap port, or a bind port absent from the inventory, and an e2e test in tests/e2e/multinode that asserts the same on live ss -tulnp output. Both read docs/security/network-connections/README.md "## 1. Inbound Listeners" as their fixture through a shared, non-test parser package (spinifex/network/listenerinventory), so a wildcard exception has to be declared in the doc's own prose rather than hardcoded in either test. * fix: make the listener wildcard exception fail closed on an explicit marker The prior heuristic granted an exception for any unnegated "wildcard" or "0.0.0.0" mention within 40 characters of the port's row. That fails open: a future row phrased as "avoids the wildcard" or with a negation further upstream than the window would silently exempt itself. Replace it with a single exact literal phrase, "binds the wildcard by design", that a row must contain to grant the exception — no other wording counts, so an unrecognized mention denies rather than grants. Update the two rows that need it (5300, 500/4500) and document the marker's role in the inventory's own prose. * docs: record that the Geneve tunnel socket binds the wildcard by design	2026-08-13 12:29:26 +10:00
Josh Siviour	93a15c1a8f	docs: reconcile the listener inventory with the listeners that exist (#767) ... The inventory was missing northstar (53 on the advertise address, 5300 on the wildcard and dialled cross-node by every node's per-instance DNS shim), charon (500/4500 plus ESP), Geneve 6081, the per-instance IMDS and VPC-DNS listeners, and chronyd. Section 6 asks the operator to compare the table against ss -tulnp, which was not possible while six listeners were absent. Two classifications were wrong. 8443 was Cluster, but predastore binds the S3 gate to 0.0.0.0 deliberately because S3 is a public plane. 4432 was External, but it binds the node's --bind address and is transient. The scope vocabulary predated the three-plane model, so it could not express the encap plane or record that a node with fewer NICs collapses the planes onto its public address. Cluster and Encap now state intent rather than a guarantee. Replaces the reference nftables ruleset. The previous one had no conntrack, loopback, ICMP or ime-* rules and ended in a blanket drop, so an operator who applied it lost guest metadata, guest DNS, all egress return traffic and their own SSH session. The replacement is input-only in a dedicated table, because nftables verdicts are per-table and a default-deny forward chain would override the accepts vpcd writes to ip filter and break every routed-NAT instance.	2026-08-12 12:47:47 +10:00
Josh Siviour	80faf59a43	fix(ovn): bind the NB/SB client listeners to loopback and the lan plane (#765) ... setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB control plane was reachable from the public interface. Write access to the Southbound database is control of the whole tenant network fabric. ovsdb-server takes remotes from two additive sources: the Connection table, which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That split is what the fix uses. Loopback goes in the replicated row, because 127.0.0.1 is correct on every node, so replication is harmless and local clients keep working. The lan address goes on the per-node command line via --db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so each node binds its own address without pushing it to its peers. A new --lan-addr flag carries that address, falling back to --db-cluster-local-addr so clustered nodes stay correct without caller changes. The standalone branch now writes /etc/default/ovn-central too, since a two-node install still has a remote ovn-controller that needs the lan listener. The guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting create-insecure-remote without a resolved address would recreate the bug. install-node.sh passes the lan address on both branches and warns when the lan plane has collapsed onto the public address, since the listener then lands on the public interface and a firewall is the only remaining control.	2026-08-12 10:37:01 +10:00
brynmailer-mdc	1394fbf488	docs: describe the predastore host and node model ... The port tables were wrong in both directions: inter-node traffic is QUIC over UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are unique within a host rather than across the cluster, so every machine uses the same three, and a single-node install opens only the S3 port because colocated nodes talk over an in-process pipe. The reference architectures scripted a move of node data out of a directory layout that no longer exists, which would have broken any install following them. The update guide now says object storage has no migration path.	2026-08-11 17:23:42 +10:00
Julian Sommer	871513dc6c	Reapply "feat: match new predastore config surface area (#591)" ... This reverts commit b4701c78de.	2026-08-04 16:13:28 +10:00
Julian Sommer	b4701c78de	Revert "feat: match new predastore config surface area (#591)" ... This reverts commit c2b80a9b93.	2026-08-04 15:02:39 +10:00
Bryn Mailer	c2b80a9b93	feat: match new predastore config surface area (#591) ... * match new predastore config surface area * fix(migrate): carry existing installs onto the predastore v5 topology The cluster topology refactor breaks an install three ways at once. The old config carries a top-level host key, which the new schema binds to the [[host]] table array, so TOML rejects the document and the daemon dies before it reads any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host data directory keyed by a node id that is now unique across roles. And because object metadata records which node holds each shard, the shard nodes had to keep their ids, so the state replicas took new ones and every replica's persisted raft configuration stopped listing it. predastore.toml gains a 4 to 5 migration covering all three: it rewrites the tables, relocates whichever node directories exist on this machine, and recovers each relocated state replica onto the new member set. The rewrite is line-oriented rather than a re-render from the template, so buckets, auth, rate limits and iam settings an operator edited survive. spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to host_id, which predastore-start.sh greps to select the host this machine runs. The predastore migration lives in its own package rather than in migrate, which every handler that stamps a KV bucket version imports; reaching the cluster runtime from there would pull raft, badger and the QUIC client into all of them. admin_upgrade.go blank-imports it, so only the binary that runs the upgrade links it, and a test pins both registrations. * test(predastore): cover the service runtime, gateway and shutdown paths The service wrapper's new seams had no unit tests: buildRuntime, serve and the errgroup Start runs the cluster and the S3 gateway under. CI's diff coverage gate measured the PR at 34.6% against a 60% threshold as a result. Cover every buildRuntime error wrap and its host selection, both ways the gateway can fail serve, and the full lifecycle: Start brings the nodes and the gateway up together, Shutdown drains both by cancelling the context they share. Topologies are written per test as real config, with node ids from a process-wide counter — the pipe transport's listener registry is global, and this package's integration tests hold a fixture daemon on the low ids for the life of the test binary.	2026-08-03 09:59:25 +10:00
Julian Sommer	9cec5a50c6	fix: pin isos to /latest so they stop getting pruned	2026-07-27 09:33:44 +10:00
Julian Sommer	ee2ab8e952	chore: cleanup docs	2026-07-09 09:43:31 +10:00
Josh Siviour	0823bf8c16	feat: OVN OVSDB RAFT clustering for multinode Spinifex (epic mulga-siv-486) (#458) ... * feat: add OVSDB RAFT cluster primitives to setup-ovn.sh Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a management node can run the OVN NB/SB databases as a native OVSDB RAFT cluster instead of a single non-clustered ovn-central. When a local addr is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since the ovn-central aggregator is ExecStart=/bin/true and won't restart its children. Empty remote addr creates the cluster (init node); a set remote addr joins it. NB/SB client set-connection runs only on the create node (RAFT replicates it cluster-wide). The single-node --management path is unchanged when no cluster addrs are supplied, so dev boxes and existing single-node clusters keep working. --ovn-remote already passes through verbatim, so a comma-separated SB endpoint list works for compute nodes; documented in usage. * feat: OVN client multi-endpoint failover for RAFT clusters Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a clustered DB. config.ParseEndpoints splits the list (trimming whitespace, dropping empties); live.go Connect passes one libovsdb WithEndpoint per endpoint, so the client tries them in order, fails over on disconnect, and follows RAFT leader redirects. Single-endpoint strings still yield one endpoint, so single-node dev is unchanged. ovn-sbctl --db= already accepts the raw comma list, so the SB CLI consumers (discoverChassis, gateway_claim) need no change. * feat: write clustered OVN NB/SB endpoints across the RAFT quorum spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback or a single leader address, so vpcd fails over across the cluster. firstboot defers OVN bring-up to the provisioning controller when it owns formation (SkipFormation): a standalone single-node .db would otherwise block the controller's clustered create/join. * test: add multinode OVN RAFT cluster + failover e2e; document clustered topology ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with one leader on every DB node, then stops the NB leader's ovn-central and proves an NB write still lands via a surviving endpoint. Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT cluster-status debugging section; network-connections lists the 6643/6644 RAFT ports.	2026-07-09 09:43:29 +10:00
Julian Sommer	756a5bb31f	feat: media-sanitization doc	2026-05-22 15:38:10 +10:00
Julian Sommer	85f248be39	chore: bump debian and ubuntu images	2026-05-20 09:48:26 +10:00
Julian Sommer	e32b267f18	chore: docs debian 12 -> 13	2026-05-19 15:57:49 +10:00
Julian Sommer	f3b00eb85a	feat: ubuntu 26.04 image	2026-05-05 09:32:29 +10:00
Julian Sommer	fa071795d0	fix: use debian 13	2026-05-05 09:10:59 +10:00
Julian Sommer	f00a376d0b	tweak: docs	2026-04-17 16:09:06 +10:00
Julian Sommer	c469eecfa1	chore: doc tweaks	2026-04-17 15:27:14 +10:00
Julian Sommer	4f991095ee	feat: physical security guide	2026-04-17 14:28:43 +10:00
Julian Sommer	14efe44125	feat: flaw remediation policy doc (#151)	2026-04-17 14:15:44 +10:00
Julian Sommer	e131caf9ec	fix: doc links	2026-04-17 13:41:58 +10:00
Julian Sommer	e46b427938	feat: network-connections docs (#150)	2026-04-17 12:44:48 +10:00
Julian Sommer	1a41731ef5	chore: tweak malware-protection doc	2026-04-17 11:59:10 +10:00
Julian Sommer	6f58a3f190	feat(docs): malware protection guide (#149)	2026-04-17 11:17:16 +10:00