mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
dev
71 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
b5879f44ff |
ADR-0001 package-boundary migration checkpoint (#1190)
* refactor: move telemetry file naming to contract
* docs: record telemetry contract move
* refactor: move aws xml response helpers
* docs: record aws xml helper move
* refactor: move aws tag projections to foundation
* docs: record aws tag projection move
* refactor: move account key to state foundation
* docs: record account key state move
* refactor: move host privilege escalation to runtime
* docs: record host privilege move
* chore: resolve lint findings from package-boundary moves
* refactor: move process supervision helpers to runtime host
* docs: record process supervision move
* refactor: move nbd endpoint helpers to compute runtime
* docs: record nbd endpoint move
* refactor: move image catalogue to operator
* docs: record image catalogue move
* refactor: move resource id generation to aws foundation
* docs: record resource id move
* refactor: move predastore role to runtime roles
* docs: point service manifest at acm domain
* docs: record predastore role move
* refactor: move northstar role to runtime roles
* docs: record northstar role move
* refactor: move awsgw role to runtime roles
* docs: record awsgw role move
* refactor: move error payload helpers to aws errors
* docs: record error payload move
* refactor: move account identifiers to aws identifiers
* docs: record account identifier move
* refactor: move NATS transport to messaging foundation
* docs: record NATS transport move
* refactor: move system instance contract to EC2 domain
* docs: record system instance move
* refactor: move system VPC builder to network domain
* docs: record system VPC move
* refactor: move instance metadata service to EC2 domain
* docs: record guest metadata move and dependency-rule debt
* refactor: move DNS control plane to DNS domain
* docs: record DNS domain move
* refactor: move base zone bootstrap to DNS domain
* docs: record base zone bootstrap move
* refactor: replace SystemAccountID with GlobalAccountID
* docs: record SystemAccountID removal
* refactor: extract CA key-pair loader to foundation pki
* docs: record foundation pki extraction
* refactor: move tenant CA paths to ACM domain
* docs: record tenant CA path move
* refactor: move IPsec certificate handling to network host
* docs: record IPsec certificate move
* refactor: move Ochre serving lifecycle to the Ochre domain
handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.
* docs: record Ochre domain move
* refactor: move Ochre vector store to the Ochre domain
handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.
* docs: record Ochre vector store move
* refactor: move Hugging Face Hub client to the Ochre domain
gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.
* docs: record Hugging Face Hub client move and Ochre gateway debt
* refactor: move EC2 tag mutation helpers to the EC2 domain
utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.
* docs: record EC2 tag mutation move and migrate split finding
* refactor: move network packages under the network domain
Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.
* docs: record network domain move
* refactor: move standalone EC2 resource packages to the EC2 domain
Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.
* docs: record standalone EC2 resource moves
* refactor: move EC2 VPC networking packages to the EC2 domain
Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.
* docs: record EC2 VPC networking moves
* refactor: move EC2 compute and storage packages to the EC2 domain
Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.
* docs: record EC2 compute and storage moves
* refactor: move ECR credential provider to the EKS agent tree
Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.
* docs: record ECR credential provider move
* refactor: move EKS gateway publish helper to the EKS agent tree
Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.
* docs: record EKS gateway publish move
* refactor: move EKS gateway fetch helper to the EKS agent tree
Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.
* docs: record EKS gateway fetch move
* refactor: move EKS webhook cert helper to the EKS agent tree
cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.
* docs: record EKS webhook cert move
* refactor: move EKS konnectivity cert helper to the EKS agent tree
cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.
* docs: record EKS konnectivity cert move
* refactor: move EKS token webhook to the EKS agent tree
cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.
* docs: record EKS token webhook move
* refactor: move ECS agent runtime and credentials packages to the ECS agent tree
* docs: record ECS agent subpackage move
* refactor: move ECS agent implementation to the ECS agent tree
cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.
* docs: record ECS agent move
* refactor: move RDS guest agent to agents/rds/agent
* docs: record RDS guest agent move
* refactor: move installer packages to operator/installer
* docs: record installer move
* refactor: move spx command tree to operator/cli
Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.
* docs: record spx command tree move
* chore: accept known vulnerabilities with an expiry
The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.
* refactor: move EC2 account action adapters to domains/ec2/awsapi
* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi
* refactor: move EC2 key action adapters to domains/ec2/awsapi
* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi
* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi
* docs: record EC2 standalone action adapter moves
* refactor: move EC2 zone action adapters to domains/ec2/awsapi
* refactor: move EC2 vpc action adapters to domains/ec2/awsapi
* refactor: move EC2 eigw action adapters to domains/ec2/awsapi
* refactor: move EC2 eip action adapters to domains/ec2/awsapi
* refactor: move EC2 igw action adapters to domains/ec2/awsapi
* refactor: move EC2 natgw action adapters to domains/ec2/awsapi
* refactor: move EC2 routetable action adapters to domains/ec2/awsapi
* docs: record EC2 VPC networking action adapter moves
* refactor: move EC2 image action adapters to domains/ec2/awsapi
* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi
* refactor: move EC2 volume action adapters to domains/ec2/awsapi
* refactor: move EC2 tags action adapters to domains/ec2/awsapi
* docs: record EC2 storage and tag action adapter moves
* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi
* refactor: move EC2 idempotency adapter to domains/ec2/awsapi
* refactor: move EC2 instance action adapters to domains/ec2/awsapi
* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi
* docs: record EC2 instance action adapter moves
* refactor: move AWS REST-JSON router to ingress/aws/rest
* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit
* docs: record AWS ingress helper moves and tooling conformance
* docs: correct historical lifecycle references
* test: characterize ECR control-plane dispatch before registration seam
* feat: add AWS ingress service registration contract
* docs: record ingress registration phases 0 and 1
* refactor: route registered AWS services through the ingress dispatch registry
The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.
The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.
* docs: record ingress dispatch registry consumption
* refactor: register the ECR control plane from its domain
domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.
Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.
* docs: record ECR control-plane registration
* fix: compose the ECR control plane in the integration harness
The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.
* docs: record ingress pipeline deferral
* refactor: extract AWS error envelope rendering to ingress
ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.
The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.
* docs: record error envelope extraction
* refactor: move AWS service selection into ingress dispatch
Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.
* docs: record ingress service selection move
* test: characterize ACM control-plane dispatch before registration
Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.
* refactor: register ACM control plane from its domain
domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.
* docs: record ACM registration
* docs: record ECS, EKS and RDS source inventory and first-resource ranking
* docs: record EKS control-plane secrets persisted in the cluster record
* chore: drop containerd advisory acceptance now that dev ships v2.4.1
* test: characterize RDS DB subnet group lifecycle before extraction
* docs: record RDS DB subnet group lifecycle contract
* refactor: extract the RDS DB subnet group lifecycle owner
domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.
* docs: record RDS DB subnet group owner extraction
* docs: correct RDS subnet group tagging debt and conformance scope
* test: characterize ECS task definition lifecycle before extraction
* docs: record ECS task definition lifecycle contract
* refactor: extract ECS task definition owner into domains/ecs/taskdefinition
Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.
The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.
List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.
* docs: record ECS task definition owner extraction
* test: characterize EKS access entry lifecycle before extraction
* docs: record EKS access entry lifecycle contract
* refactor: extract EKS access entry owner into domains/eks/access
Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.
Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.
List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.
* docs: record EKS access entry owner extraction
* test: characterize RDS engine catalogue before extraction
* fix: return InvalidParameterValue for a missing RDS master password
ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.
* docs: record RDS engine catalogue boundary and evidence
* refactor: extract RDS engine catalogue into domains/rds/engine
Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.
The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.
The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.
* docs: record RDS engine catalogue owner move
* test: characterize RDS DB parameter group lifecycle before extraction
* docs: record RDS DB parameter group lifecycle contract
* refactor: extract RDS DB parameter group owner
Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.
* docs: record RDS DB parameter group owner extraction
* test: characterize EKS add-on lifecycle and guest wire
Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.
* docs: record EKS add-on lifecycle contract
* fix: require iam:PassRole for EKS add-on service account role
CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.
* docs: record EKS add-on PassRole check
* refactor: extract EKS add-on owner
Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.
* docs: record EKS add-on owner extraction
* refactor: extract EKS add-on guest wire contract
Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.
* docs: record EKS add-on wire contract extraction
* docs: update EKS add-on inventory and remaining edges
* docs: draft EKS add-on lifecycle correction design for review
* chore: run contracts tests in the default test, coverage and race targets
* docs: apply review decisions to EKS add-on lifecycle design
* test: characterize EKS add-on v1 compatibility with added fields
Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.
* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
* docs: draft EKS add-on v2 guest contract design for review
* docs: qualify historical architecture references
* docs: apply review decisions and ADR-0006 to EKS add-on v2 design
* fix(eks): bind internal report publishing to the caller's cluster
PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.
PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.
* fix(eks): bind webhook token review to the caller's cluster
WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.
WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.
* fix(eks): validate add-on serviceAccountRoleArn syntax
CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.
The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.
* docs: record EKS internal-route and role-ARN security corrections
* docs: add EKS add-on release membership and activation case study
* fix: point UI build, CI and lint rules at moved package paths
* docs: record EKS add-on activation review decisions
* refactor: move quota admission to domains/admission/quota
Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.
* docs: reconcile package-boundary record for ADR-0001 close-out
Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.
* test: move KV migration engine tests to foundation/state/migrate
The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.
* refactor: move config and object-store upgrade workflow to operator/upgrade
Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.
* docs: record migrate move to operator/upgrade
* test: enforce ADR-0001 layering with a blocking ratchet
Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.
Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.
* refactor: drop the legacy handlers_ecr import alias
Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.
* chore: mark spf13/pflag as a direct dependency
operator/cli/wiring.go imports pflag directly. Same version; no upgrade.
* docs: record test-package-check and nilaway as unverified gates
* refactor: move the layering test to module-root architecture/
The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.
* fix: update paths left stale by the package moves
The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.
* refactor: move the default-VPC subject into contracts/ec2/v1
utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.
* docs: qualify the migrate split and record admin and Viperblock findings
The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.
* refactor: move operator-only admin code to operator/admin
Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.
foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.
* docs: record operator/admin split in package-boundary migration
* test: pin viperblockd fence, abandon and dirty-marker wire shapes
Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.
* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts
Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.
Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.
* refactor: split the Viperblock EBS adapter out of services/viperblockd
Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.
The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.
Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.
* refactor: move the viperblockd role to runtime/roles/viperblockd
With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.
* docs: record viperblockd contract, adapter and role moves
* test: characterise vpc.* network-realisation routes
Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.
Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).
* feat: add contracts/network/v1 for vpc.* realisation routes
Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.
* refactor: use contracts/network/v1 for vpc.* producers and consumers
Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.
Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.
No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.
* refactor: replace utils/vpcd_event.go with a typed network projection client
Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.
utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.
* refactor: route the remaining fire-and-forget vpc.* producers through projection
CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.
* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils
The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.
utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.
* docs: record the network contract, projection client and utils removal
* test: characterise instance-state persistence, recovery and generation handling
Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.
* test: freeze direct instance-state consumers
Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.
* docs: record ADR-0007 slice 0 instance-state inventory
Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.
* test: narrow the instance-state ratchet to instance-state access
Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.
* docs: record the narrowed instance-state ratchet commit
* docs: design the instance-state transition record and activation gate
* docs: record the instance-state transition-record design commit
* docs: record slice 1a activation-gate decisions
* feat: add instance transition-record codec with format marker
Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.
Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.
No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.
* docs: record slice 1b step 1 codec and awsgw rollout qualification
Records the codec added in
|
||
|
Julian Sommer
|
b3b066562e | chore: add comment to all exported types | ||
|
Julian Sommer
|
36d5788923 | fix: require iam:PassRole for EKS cluster and node roles (#1139) | ||
|
Julian Sommer
|
2b4ddad78a |
chore: drop dead and redundant nolint directives
Removes directives for linters that are not enabled (noctx, exhaustive, deadcode) and gosec suppressions already covered by config exclusions. Tags ochrevector QueryResult and Filter with their existing field names so musttag passes without changing the NATS wire format, and bounds the RDS window slot conversion instead of suppressing G115. |
||
|
Julian Sommer
|
a3ea962d1e | chore: shared e2e ssh harness | ||
|
Julian Sommer
|
5f21fd27e3 | chore: inline more | ||
|
Julian Sommer
|
810aa019bf | chore: use strings.Cut | ||
|
Julian Sommer
|
610ee9bab0 | chore: use slices.Sorted | ||
|
Ben Duncan
|
8ab96b29a1 |
fix: replicate every JetStream KV bucket across the cluster (#1117)
* fix: replicate every JetStream KV bucket across the cluster A KV bucket created without a replica count gets one replica, and one replica lives on one JetStream-chosen server recorded in no config. Losing, restarting or partitioning that server makes the bucket return "no responders available" from every node at once, so a single-node event becomes a cluster-wide outage. The reconciler leader-lease buckets were the worst case: acquiring a lease is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed. The cluster's node count is now declared once, in config.LoadConfig, and read by the one helper every bucket creation goes through. Every process that creates a bucket loads the cluster config first, so no service can forget to pass it and no signature has to carry it. An undeclared size is an error at creation rather than a silent fall back to one replica. Existing buckets are raised rather than left behind: a service opening one raises it, the daemon sweeps them all after formation, and "spx admin kv replicas --repair" does it on demand. Raising reads the live stream config and changes only the replica count, and never lowers one. Removes the ~30 replicas/clusterSize parameters that only forwarded a number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no clamp and would have failed outright on a cluster of six or more. Proved by a repo-wide invariant test pinning bucket creation to the one helper, a whole-cluster sweep in every multi-node nightly cell, and a new kvquorum suite (cell 33) that takes away the node leading the most buckets and requires the survivors to keep answering for all of them. * chore: name cell 33 in the nightly dispatch input * fix: a KV bucket that cannot be raised must still open The first version of the replica fix treated a raise it could not place as a hard error, and on a real three-node cluster that turned the fix into the outage it prevents. A node whose config named a fourth server wanted four replicas, NATS answered "no suitable peers for placement" for three, and the daemon retried for its full five-minute budget without finishing bootstrap. The bucket was present, quorate and serving on three nodes throughout. A config naming a node before that node serves is the ordinary middle of growing a cluster, so that is a state to tolerate rather than refuse. The line is now drawn between the two cases instead of around both. Creating a bucket below the cluster's replica count still fails, because that is the defect and a fallback there reintroduces it where it is hardest to notice. Failing to raise an existing bucket warns and hands the bucket back, and the raise is retried on every open besides the daemon sweep and --repair. This matters most in kvlease, which runs on every reconcile tick: refusing there would stop every reconciler sharing a lease bucket that was working. getOrCreateBucket now attaches before it creates. A create asking for more replicas than the cluster can place is refused before the server looks at the name, so with create-first "this bucket already exists" was an inference from which error came back, and the wrong one comes back on a single-node server. RaiseAllBucketReplicas attempts every bucket and joins the failures, and --repair keeps going too. Stopping at the first would leave the rest of a cluster short because one bucket could not be placed. Also adds nats.ServerName/NodeFromServerName beside the template that renders server_name, with a test tying them together. The kvquorum suite had mapped stream peers to nodes by comparing them to node names, which they are not: they are the node name behind a "spinifex-nats-" prefix, so nothing matched and the node-loss test could not choose a victim. * test: wait for the right NATS peer count after a node rejoins /routez reports a node's routes to its peers, not its own membership, so a fully formed three-node cluster is two peers per node. The kvquorum suite waited for three and spent its whole five-minute rejoin budget on a cluster that had already reformed. Every other caller passes len(Nodes)-1; the helper's doc comment now says so. The assertion the test exists for had already passed: node1 was leading 18 of 47 buckets when it was taken away, and node2 and node3 each still answered for all 47. * fix: replicate KV buckets across the largest odd count a cluster can place A bucket was replicated across min(nodes, 5) servers, which stores a fourth full copy on a four-node cluster and still survives exactly one loss — the same tolerance R3 gives for three quarters of the write amplification. A Raft majority needs an odd count, so the rule becomes the largest odd count the cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two entry points, so no caller can hold a stale copy of the policy and keep passing after the policy changes. Three retry loops could not tell a configuration fault from a cluster that had yet to form. An undeclared cluster size never becomes declared by waiting, so each of them spent its whole budget on one and then reported quorum as the cause. They now stop on a permanent error and say what is actually wrong. This is what the integration suite was hitting: it builds its gateway by hand and never loads a cluster config, so the size was never declared and the IAM retry sat in its sleep until the test timed out. Both ends are fixed. The audit reported configuration alone. A bucket at three replicas with one peer offline and one lagging is one failure from unavailable and read as ok. BucketReport now carries the leader and a count of the peers JetStream itself calls current and reachable, quorum is measured against that count, and spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem --repair fixes from one needing a human. Declaration is write-once: daemon.Start refuses a conflicting redeclaration because it is about to create buckets, and it declares before JetStream init rather than after it. config.LoadConfig warns instead, because it also reads config files that are not this process's own cluster. spx admin init --nodes 2 is refused — no replica count on two nodes both survives a loss and accepts writes — and single-node replication is asserted in the single e2e suite, so it runs in every nightly host permutation. * fix: withhold the vLLM serving image until its artifact is published The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names 404s on iso.mulgadc.com, so spx admin images import would fail for every operator who tried it. Its GPU siblings are published; this one never was. The entry moves to WithheldImages rather than being deleted, because the entry is the definition — the URL, the checksum and the two tags resolveServingAMI filters on. Publishing the artifact is then a move back, not a retype. Bedrock already handles the image not being registered: resolveServingAMI returns ErrServingAMINotFound, which is the honest answer while there is nothing to launch. The bedrock tag contract is still pinned, now against the map the definition lives in, and it additionally asserts the entry is not offered for import. The e2e reachability walk covers AvailableImages alone, so a withheld entry is out of it by construction. * fix: drop the cluster size from the call sites dev added Two constructors on dev still pass a replica count that no longer exists as a parameter: the cluster size is a declared process fact, so EKSServiceDeps has no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is textually clean and breaks the build, which is the shape of collision a signature removal always has. * docs: explain what an operator sees when a cluster server fails The replication section said how many copies a bucket has and stopped there, which leaves the question an operator actually has during an incident: one server is gone, what still works. It now covers the majority side carrying on, an isolated server refusing changes on purpose and why that is the behaviour you want, guests continuing to run while anything that has to be recorded does not, the risk of reading state from a server that is cut off, and a returning server catching up by itself. The surrounding prose reads from the operator's side rather than the implementation's: what you see and what to do, not which consensus protocol is underneath. |
||
|
Ben Duncan
|
021f439bab |
feat: recover instances automatically from a node that stopped heartbeating (#1111)
* feat: recover instances automatically from a node that stopped heartbeating A node that dies hard takes its guests with it, and nothing brings them back until an operator brings that node back. Every daemon now runs the same small reconciler: it reads the instance records, selects desired-running instances whose owner is stale, and relaunches them here. The claim is a CAS on Status.LastNode and that is the whole of the coordination. Several survivors race one instance, exactly one revision wins, and the losers find a record naming a live owner and stop considering it. A claimant that dies mid-launch leaves the record naming a node that is itself stale, which is the same candidate the next pass derives again, so there is no lease to expire and nothing to reconcile by hand. Ownership is not permission to write. The volume leases still decide whether the guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt that loses the volume race has touched no tap, no OVS port and no OVN binding. A storage failure is not a host failure and wants the opposite response. A guest paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is refused: the volumes are the same objects in the same store from anywhere, so relaunching moves the pause rather than ending it. A node whose own object store is not answering stands down entirely, which separates a store broken here from a store broken everywhere without either having to be diagnosed. A launch that does not complete backs this node off that instance, doubling to a cap, because nothing on the mount path can yet say which of the two it hit. Recovery is off by default and inert below three nodes. It also waits until it has seen its peers live, or five minutes, so the first node back from a coordinated shutdown does not recover guests from nodes that are still booting. * test: prove instance recovery moves a host failure and refuses a storage failure The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves. Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live. harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down. * fix: make automatic instance recovery a cluster-wide setting Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file. Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it. * test: drive real I/O into the storage-fault case, and freeze the store rather than stop it Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted. The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them. The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three. * test: give instance recovery its own suite and its own nightly cell Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five. Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set. * fix: give a guest long enough to open a volume the object store is slow to serve QEMU opens every drive before its monitor answers a greeting, so the QMP greeting deadline is really a deadline on opening the guest's volumes. At 30s it was shorter than a cold viperblock open costs on a cluster that has lost a node, where every object read is served degraded: measured at 35s with one node of three away. The launch killed the guest five seconds short of ready and then did it again on every retry, which is a recovery that can never complete. A network-backed drive now gets its own deadline for the same reason VFIO already had one — synchronous work before the monitor answers — and the instancerecovery suite is what found it. Also make the suite's predastore thaw best effort per node. It runs after a node has been taken away, where there is no process to signal, and a node it cannot reach must not stop it restoring the rest. * docs: name the two newest cells in the dispatch input The cells input is the operator's index of what a number means, and it stopped at 29 while 30 and 31 were added. * test: prove the guest is writing before freezing the store in the recovery suite The storage-fault test froze the object store as soon as RunInstances reported running, which is when QEMU has launched rather than when the guest has booted. On the nightly cluster the freeze landed while the guest was still in UEFI, so the only drive it could reach was the EFI varstore — and pflash has no werror=stop to pause on, so the guest never paused and the test reported a product failure it had caused itself. The load driver now prints to the serial console after each pass it has written and synced, and the test waits for that before freezing. The console is read through GetConsoleOutput, so this still needs no login and still runs on cells whose guests have no public address. * feat: recover instances automatically with nothing to configure The switch is gone. An instance whose host has failed is down whether or not recovery is enabled, so the only outcome a disabled reconciler can produce is leaving it down — which is the state recovery exists to end, not a safer one to fall back to. Against that, the switch cost a cluster that can be configured into never recovering anything, with no symptom until a node dies. This feature already hit that once: a top-level table parsed into nothing and every node reported recovery off with the setting plainly in the file. What makes recovery inert is now only facts about the cluster — no cluster config, no JetStream, or fewer than three nodes — and each says in the log which one applied. The e2e no longer writes config or restarts daemons: it waits for every node to report the reconciler active on a cluster nothing configured, so reintroducing a switch fails there as well as in the unit tests. * feat: stop retrying a recovery that is never going to work Every failed attempt used to go to the same exponential backoff and stay there. An instance nothing could launch was retried every fifteen minutes forever, still observed-running on a node that is gone, with no account of it anywhere the customer or an operator would look. A failure is now classified and counted. Two classes are read from what the launch actually reports: the node it was taken from still holds a volume lease, which clears on that lease's clock and is retried on a faster one, and this node has no room, which another survivor may. There is no terminal class, because nothing on this path can prove one — the launch collapses everything else into ServerInternal, so a verdict of "never going to work" would be a guess, and the guess that abandons a recoverable instance is the expensive one. The budget bounds the unknown case instead. Every class counts against it, since a class that did not would be the old unbounded retry under a narrower condition. Exhausting it leaves the instance where AWS leaves one whose host failed and could not be brought back: stopped, carrying a StateReason that names the cause and the last error, one StartInstances from running again. Writing the desired state down is also what makes one node's decision cluster-wide, because selection reads desired state — the same property the claim already has. A record another node has since claimed is never touched, so a node exhausting its attempts cannot stop a guest that is now running elsewhere. * fix: give a guest the time for every cost it pays before its monitor answers A GPU instance booting from a network-backed root volume pays both costs: QEMU opens every drive across the object store and then pins the whole of guest RAM, one after the other, all before the monitor answers. The deadline took whichever was larger, so such a guest was given less time than it needs. That the two floors are both 180s is why nothing has been seen yet — the VFIO branch has always returned at least the network-drive deadline. It is the same defect either way, and the symptom when it lands is a GPU instance SIGKILLed on every attempt while its plain sibling launches, which reads as a GPU fault rather than as a deadline. The costs are now added, the plain QEMU startup is counted once rather than replaced, and the cap covers both floors together so it cannot clip a guest before either cost has been waited out. * test: assert the AWS-visible arc of a host failure end to end A customer cannot see which host an instance is on, and AWS never tells them. What they can see is DescribeInstanceStatus, so that is where a recovery has to be provable, and the suite never looked at it. Three frames are now required, in order: running with system-status ok before anything is broken, system-status impaired while the owning node is gone, and ok again once a survivor is running it. The middle frame is the one worth having — the window opens when the owner's heartbeat goes stale, which is strictly before any survivor may claim, and closes when the relaunch registers, so it is bounded rather than lucky. Reporting ok throughout would be the more comfortable lie. The availability zone is compared across the first and last frames as well. A recovery that crossed one would break every placement promise made to the customer, and nothing else in the suite would have noticed. * test: prove a half-healthy node fences itself and is never a second writer The two existing recovery cases take a node away by stopping its services, which is the failure this feature handles least interestingly: a node that is gone has no opinion about who owns its volumes. The one that matters is the node that is still up, still running QEMU, still answering its own local clients, and simply cannot reach consensus. That is the state in which two nodes can each believe they own a volume, and the only thing between that and a corrupted disk is that the partitioned node gives up first, on a clock it can evaluate without reaching anything. So this drops NATS on 4222 and 4248 between one node and its peers and nothing else, then asserts the ordering the safety argument rests on rather than the outcome it produces. A watch samples every node every three seconds for the whole run, including the heal, and fails on any single sample naming two hosts. Two guests writing one volume through one object store is data loss already under way, not a race that resolves, and no point-in-time assertion can see one that appeared between its own samples. The injector takes its own nft table rather than touching inet spinifex_filter, because a test that flushed the node's real firewall would leave it either unfirewalled or misfirewalled with no sign of which, and a separate table at priority -300 heals in one atomic delete. A transient systemd timer removes it independently, so a cancelled workflow does not leave a node cut off from its cluster until somebody notices. It proves SSH still works and proves a peer port is genuinely unreachable before anything is asserted, since a rule set that installed cleanly and dropped nothing would read as the system tolerating a partition. The suite budget goes to 90m and cell 31 to 105m to fit the third case. * fix: move an instance's boot state, volumes, logical port and address with it A record-driven launch assumes the node reading the record is the node that wrote it. On a daemon restart that holds. On a recovery onto a survivor it does not, and five things follow from it — one that stops a moved instance booting at all, and four that stop it running anywhere. A load balancer never comes back from a host failure. Its boot configuration is a set of files under the runtime directory and the record holds their paths, so the survivor restores paths that exist only on the node that died. The hook that rebuilds them was reachable from Restore alone, and the cross-node path goes straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert that are not there. The gateway URL baked into that env names the creating node's management bridge as well, which the rebuild corrects because it rebuilds from whichever node is running it. Firing the hook on both paths fixes both, and a rebuild that fails now refuses the launch: a guest that boots without its network configuration is worse than one that does not boot, because the records would say it recovered. The export outlives the guest and holds the volume lease. A returning node is healthy, so it renews that lease indefinitely and the new owner never acquires: the guest is stopped here and refused there, recovery spends its attempt budget on a volume that was never going to be free, and gives up on an instance that was never unrecoverable. So the export is given up too, over a new node-addressed request that tears it down exactly as the fence does. Deliberately not an unmount — an unmount seals, this node's block map is the stale one, and sealing publishes it over the winner's. Unlike the fence it deletes the lease key rather than leaving it to expire, because the lease is still this node's to hand back. The tap outlives the guest, and the tap is what claims the instance's OVN logical port. The binding names one chassis, so two nodes offering the same iface-id do not split the traffic — they contend, and the address answers from whichever won last. It also blocks the layer above: the external IP's host ingress is pruned on the evidence that the owning guest sits on another chassis, and that stays false while this node is still claiming the port. The management address outlives the guest as a cluster-wide reservation, and the release was unconditional. br-mgmt is one flat L2 segment, so the node an instance left was freeing an address still in use and offering it to the next instance anywhere. Only the node an entry names may release it now, with the takeover recorded on arrival so the node actually running the instance can still give the address back. An entry written before the node field existed names nobody and stays releasable, because there is no other holder to collide with. The fence path already cleaned up the tap, so the release fix repairs that path too. * test: prove a load balancer keeps its name, its address and its traffic when its host dies Takes away the node running an internet-facing ALB and asserts the four things a customer of one actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again without anybody being called, and the backends are healthy again from the recovered load balancer's point of view. The backends are placed around the load balancer rather than beside it. A load balancer lands wherever a node answers its queue group, so its node is discovered and the backends are relaunched until both are elsewhere; one on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve. DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve through the same server, so the claim under test is what this cluster answers for a name rather than what the machine running the tests was configured to forward to. A watch samples every surviving node every five seconds for the whole run, because an answer that went missing or pointed elsewhere for one reconcile pass is the outage, and no single sample reaches it. The returning node then has to give up what it no longer owns: no second copy of the guest, and no tap still claiming the load balancer's logical port, since OVN binds a port to one chassis and a second claimant contends for it rather than splitting the traffic. The last assertion is that the name still serves with that node back in the cluster. * docs: name the new cell in the nightly's cell list * test: stop the disk-load guest from filling the console it is judged by The load generator announced every write, once a second, for as long as the test ran. The console is a ring buffer this suite reads back, so an hour of that pushes out everything explaining a failure, including the kernel's own account of the I/O error the test exists to produce. Announce twenty times, then keep writing quietly. * docs: say what the OCI guide has been proven on, and what a host failure does The guide read as a support statement for OCI rather than as an account of one tested path, promised seven steps and then gave eight, and said nothing about what happens to a guest or its public address when the node under it fails. State the evidence, state that the nightly guards the datapath and not the cloud, and give the failure behaviour as a table with the reason codes a customer will actually see. Also say plainly what the volume lease is: one running guest and a bounded, logged window, not a write barrier the object store enforces. Someone sizing a workload against this should read that before they read the table. * docs: say why the mount classifier refuses to retry a lease another node holds It reads as an omission — a lease held elsewhere is the most transient refusal there is. It is not. This classifier drives the restore path, where the holder is a node that is running the guest, so retrying would relaunch against a live owner. Recovery wants the opposite and reads the refusal itself. * feat: claim a recovered guest's public address in line, and verify it arrived A recovered instance was reported running before anything moved its public address off the node that died, so for up to fifteen seconds the records said recovered and the address answered nothing. The recovery path now runs the affinity pass itself, which is idempotent and is the same pass the ticker runs. Two things the pass could not do before. It abandoned every remaining address on the first API failure, though a throttle or a denial is per-object and the guests behind the others are just as dark. And it trusted the move's own response, which is OCI agreeing to the request rather than OCI having done it, so an accepted move that never landed read as success. It now re-reads what OCI delivers here and reports the difference. An address that will not arrive makes the guest a system-status failure with the time it went dark, the same shape AWS uses for a host-level fault. The guest passes every check it can be asked about itself, which is exactly why reporting it healthy was the worst available answer. * feat: give a recovery attempt one trace across all three of its phases The volumes opening, QEMU starting and its monitor answering already trace themselves, but nothing tied them to the attempt that asked for them: three orphan traces per recovery, no way to ask which phase a slow one spent its time in, and nothing naming the node the guest came from. A span over the attempt makes them its children, and the reconciler's own log lines carry its trace ID because the handler stamps it from the context. A failed attempt records the fault and the attempt count on the span, so the budget running out is readable as a sequence rather than as twelve unrelated warnings. * fix: finish a fence once its writer finally exits A kill that did not take effect was logged, counted and dropped. The export stayed registered and the lease stayed held, which is correct while the writer lives and wrong forever after it dies: a lease a healthy node keeps renewing never expires, so the volume could not start anywhere. The failure is now re-reported every 30 s until it resolves, and the teardown completes as soon as the writer is confirmed gone. Only a delivered kill is waited on — a refused one says nothing about what became of the writer. * test: stop judging a detached stop or terminate on a transient state Both tests read the instance state immediately after the ack, but the transition runs on its own goroutine, so the read races it in both directions: too early sees running, too late sees the final state. The race gate caught the first of those on a loaded runner. * fix: replicate the volume lease and dirty buckets across the cluster Both were created with the default single replica, so both lived on one node. That node holds the answer to "who may write this volume" and the marker every mount reads, and losing it takes volume leasing and mounting down cluster-wide. A partitioned node was fenced correctly and then no survivor could relaunch its guest: every attempt failed on "read dirty mark: no responders available", for ten minutes, until the budget ran out. The replica count is now the cluster's node count, as the gateway's own buckets already do. Existing buckets are raised in place rather than requiring a wipe. The e2e diagnostics that should have said this are fixed too. The fence and the surrender are logged by the storage service, not the daemon, so the assertions read the wrong journal; and Eventuallyf evaluates its message at the call site, so a ten-minute wait was explained by a log that stopped before it began. |
||
|
Josh Siviour
|
c10b6732d5 |
feat: publish AWS service endpoint names in DNS and match gateway cert SANs (#1068)
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs
ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.
The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.
admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.
Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.
* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain
ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.
The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.
Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.
* chore: pin northstar to the merged set-valued write path
|
||
|
Ben Duncan
|
ea3e01bb20 |
fix: reboot a guest by asking it, not by pulling its reset line (#1028)
* fix: reboot a guest by asking it, not by pulling its reset line RebootInstances was QMP system_reset alone. The guest kernel is never told, so it does not sync, does not unmount and loses every dirty page - which is how a reboot issued seconds after a write loses that write, and how a guest rebooted during cloud-init can fail to come back at all. Guests now launch with -action shutdown=pause, so a clean guest poweroff leaves QEMU paused rather than gone. Reboot presses the power button, waits for the guest to reach that state, then resets and resumes the same process: no volume, tap or state churn, and no window where the guest is missing. A guest that will not go quietly is hard-reset, which is what AWS documents. The button is pressed again every 10s while the guest ignores it. One signal is not a shutdown request on a guest whose acpid has not started, which is the same boot race that produced the failure. Two consequences handled here: a guest that powers itself off now leaves QEMU paused, so the heartbeat quits it; and a guest launched before the flag exits on powerdown rather than pausing, so it is reset without being asked. * fix: answer a reboot when it is accepted, not when the guest is back Asking a guest to shut down takes as long as the guest takes. RebootInstances gives the daemon five seconds to reply, so a reboot that waited for the guest reported a reboot that had in fact succeeded as InvalidInstanceID.NotFound. Manager.Reboot now admits the reboot — the instance exists and is running — and runs the powerdown and reset in the background, which is the contract EC2 documents. The two admission failures are still reported; nothing after them can be, so a reset that fails is logged and left to the heartbeat. The gateway also stops calling a NATS timeout NotFound. Only the absence of a subscriber says anything about where the instance is, which is how the sibling start and terminate paths already read it. * test: prove the churn reboot by its boot ID, not by how fast it happened The subtest waited ten seconds and then read /proc/uptime. A reboot is asynchronous, so the guest answering at that point may be the one that has not shut down yet, and its uptime is whatever it was before — which the test reads as "reboot may not have occurred". Waiting for the boot ID to change is positive proof the guest rebooted and does not depend on how long the guest takes to go down. The uptime assertion stays, now measured against a boot that is known to be new. * test: prove the graceful reboot against real guests and a real engine Two tests the reboot design asked for and the unit tests cannot give. Cell 18 launches two guests that differ only in how they answer the power button. One installs a shutdown hook writing an fsynced line a second, so the file that survives measures how much of its shutdown it was allowed to run — a guest reset instead of asked leaves none of it. The other ignores the power key entirely and must be reset once its budget expires, with the daemon saying so. Each variant asserts the evidence only it can produce, and the slow one asserts the absence of the line the wedged one requires. Cell 27 loads a PostgreSQL instance past its shared buffers with pgbench and reboots it dirty. The probe is an UNLOGGED table: PostgreSQL writes no WAL for one, so a clean shutdown checkpoints it and crash recovery truncates it unconditionally. Rows committed normally survive a hard reset too, so the obvious assertion would pass against the bug. The rds suite and cell 27 both gain budget for the load phase. * fix: stop the process-group test misreading a reaped child as alive The test polled until the child was gone, then checked once more. A task being reaped shows state X for an instant, which processRunning counted as running, so that second look failed about 1 run in 50 under -race. * fix: wait for the recovery retry loop before removing a test's data dir restoreInstances starts the retry loop untracked, and it rewrites the local state file when it ends. A write landing during t.TempDir cleanup failed TestRestoreInstances_* with RemoveAll: directory not empty. |
||
|
Ben Duncan
|
1afc7fa8aa |
fix: retry a recovery relaunch that lost a race with a starting dependency (#1020)
* fix: retry a recovery relaunch that lost a race with a starting dependency A guest drain-stopped by a coordinated cluster shutdown could be lost for good. The relaunch ran before JetStream was serving, the volume lease claim timed out, and the failure was recorded as permanent. The backoff this needed already existed. relaunchWithRetry retries five times, but only on ErrMountRetryable, and mountErrRetryable recognised exactly two viperblock state-load sentinels. A lease claim returning context deadline exceeded matched neither, so the mount was reported as permanently failed and the backoff never engaged once. - Classify a lease store that cannot answer as ErrLeaseStoreUnavailable, and include it in mountErrRetryable. A lease held by another node is deliberately excluded: that is an answer, not an outage. - Probe readiness rather than connectedness. checkViperblockReady tested natsConn.IsConnected(), which is true against a server whose JetStream is still starting; it now calls AccountInfo with a timeout. - Reconsider a latched failure. RetryRecoveryFailed relaunches instances parked in error by a dependency-unavailable reason, once the backing store is confirmed ready, for a bounded number of passes, then leaves them alone. Only mount-state and launch failures qualify; a failed reconnect or pre-relaunch hook can mean the guest itself needs looking at. Closes the e2e gap that made this feel covered. The node-failure tests stop the service units directly so guests keep running and recovery is a QMP re-attach to a live process. Nothing stopped a guest, so nothing relaunched one from a sealed volume against dependencies that were still starting. TestMultinodeClusterRestartResumesGuests does, including the wait for every spx process to exit that makes it a real restart rather than a stop that returned early. Beads mulga-v7kp4, mulga-kyb1d. * fix: stop every spx process by unit in the cluster-restart helper The first CI run of TestMultinodeClusterRestartResumesGuests failed at the process wait with "node1 still has 2 spx process(es)", and the same thing happened by hand on a prod node. Every service is the same binary, and the stop was driven from a hard-coded list. StopNode names seven units and omits spinifex-northstar and spinifex-qmp-collector, which is exactly the two survivors. The unit name now comes from the surviving process's cgroup, so the set that stops is whoever is still running the binary and the list cannot go stale when a service is added. StopNode itself is left alone: its list is deliberate for the hard outage it simulates, where the target's drain must not fire. Also marks recovery_retry_test.go as //test:in-package — the retry policy is unexported and the tests drive it through the seams restore_test.go uses. * fix: stop spinifex.target in the cluster-restart helper, not the units The second CI run reformed NATS to 2 peers of 3: node3's NATS was stopped and never started again. StopNode stops the service units directly and leaves spinifex.target active. That is right for the hard outage it simulates and wrong for a real shutdown — the target then reads as satisfied while its units are dead, and the later start raced a stop job that was still queued. The helper now runs what the operator procedure and update-nodes.sh run: stop spinifex.target, wait for every spx process to exit, and additionally wait for systemd to have no spinifex job left queued before starting. That last wait is the one that closes this race. * fix: wait for the right NATS peer count in the cluster-restart helper WaitNATSPeers counts remote peers, so a healthy three-node cluster reports two. Asking for three could never be satisfied, and the run that hit it had in fact restarted all three NATS correctly. |
||
|
Ben Duncan
|
18853cef31 |
fix: close the four defects keeping the e2e nightly red (#1007)
* fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main. |
||
|
Ben Duncan
|
c2aad92044 |
fix: stop the detach seal, EIP teardown and key-pair races failing the nightly (#1003)
* fix: stop the detach seal, EIP teardown and key-pair races failing the nightly
Four independent faults the full-suite nightly surfaced, each of which had
been masked while the heavy cells were skipped.
unmountVolume SIGKILLed nbdkit the moment the request arrived. Removing the
guest device has already started the plugin's own Close, which drains to the
backend and leaves a seal receipt, so the kill landed 61ms into that flush and
threw it away — the receipt path never fired once across a whole run. What was
left was 74 orphaned WAL files for the fallback seal to replay, which ran past
the caller's 180s deadline and returned DetachVolume ServerInternal with the
volume still attached. It now terminates gracefully first, bounded by the
plugin's own drain budget, and only escalates if that does not take.
DeleteEIP returned early when the dnat_and_snat row was already absent, before
the neighbour flush and the host unbind. An absent row is not a reassignment —
nobody owns the address — so the /32 route and proxy-ARP entry were orphaned
and a disassociate left the address still answering from the host. The
reassignment cases keep their early return.
EnsureKeyPair was not atomic: a failure after CreateKeyPair left the pair on
the cluster with no cleanup registered, so every later caller failed with
InvalidKeyPair.Duplicate in a test that never ran. The describe that follows a
create also had no tolerance for the propagation window on a multi-node
cluster; it now retries, and still reports a duplicate the moment it sees one.
An admission refusal could not name its cause. The static vCPU/memory budget
and the live MemAvailable clamp both surface as InsufficientInstanceCapacity,
and they have different fixes; the subscription transition now records which
gate closed.
* fix: identify the default-deny by priority, not by being a drop
DenyIsNotEthertypeScoped bucketed every drop ACL by direction, so the last one
seen won each direction. The platform WAN egress block added at priority 1049
is also a drop and is legitimately ip4-scoped, so it displaced the default-deny
in the map and the assertion fired on its qualifier instead:
from-lport deny on sg_c94870c0d68288dc6 is IPv4-scoped:
"inport == @sg_c94870c0d68288dc6 && ip4 && ip4.dst != {10.0.0.0/8, ...}"
Select the two default-denies by their own priorities. What the test asserts is
unchanged — the default-deny still must carry no ethertype qualifier, for the
reason denyIngressACL states: OVN allows on no-match, so an ip4-scoped deny
lets every other ethertype through.
* fix: drain every EIP on an ENI at teardown, not just the first
findByENI returned the first matching record and stopped, so an ENI holding
more than one EIP had all but one left marked associated to an interface being
deleted — with the host route still installed and no record left naming the ENI
to find them. Teardown now clears them all.
* ci: serialise e2e-nightly runs so two do not share the hypervisors
Only iso-single and iso-multi were guarded, by the iso-qemu-rack group. Every
other job could overlap a second run of the workflow, with both sets of cells
building clusters on the same five hosts. Queue rather than cancel: a dispatch
is usually a verification run whose result someone is waiting on.
|
||
|
Ben Duncan
|
828b5df4aa |
fix: pause a guest on a backend I/O error instead of handing it EIO (#977)
* fix: fail DRAIN when a volume seal failed
shutdownAndUnmount logged the volume unmount error and returned nothing, so
StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set.
The coordinator then advanced to STORAGE and stopped viperblock while the seal
was still in flight, tearing the VBState envelope and leaving the guest unable
to mount its root volume on the next boot. The deploy reported success.
shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not
seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup
and stopOne carry it without aborting the rest of the teardown, since QEMU is
already down by then, and StopAll aggregates the per-VM failures so
handleShutdownDrain ACKs with Error set and never reports vms_stopped.
volumeMounterAdapter.Unmount was the second swallow site: it logged each failed
seal and returned nil unconditionally. It now attempts every volume and returns
the aggregate, which is what the VolumeMounter contract already described.
Terminate still tolerates a failed seal — the volumes are deleted next, so an
unsealed block map loses nothing a caller can ask for.
This does not fix the seal itself. It converts silent data loss into a refused
upgrade that an operator can act on.
* test: add the storagefault e2e suite for backend-outage guest survival
Freezes predastore with SIGSTOP to hold connections open while answering
nothing, which is the fault a timeout-free path cannot distinguish from a
slow peer. On one node that is a total outage; on three it freezes every
node but the guest's own, breaking the RS(2,1) shard floor.
The suite writes and flushes a verifiable region before the fault and
churns a separate unverified region during it, so an interrupted write is
never mistaken for lost data. The root-filesystem load runs from the
user's home rather than /tmp, which is tmpfs on this image and would move
no bytes on the root device.
* test: verify the root device is under write pressure, and size the outage
The previous guard matched /^ *vda / against /proc/diskstats, where the
device name is the third field, and read $7 which is ms_reading rather
than sectors_written. It printed empty and asserted nothing, so a green
run could not distinguish a surviving guest from an idle disk. It now
samples sectors_written twice and fails unless it advances.
The outage is sized from viperblock's absorption rather than guessed:
a workload must fill DefaultMaxPendingBytes before backpressure engages
at all, so a short freeze measures the retry budget rather than the fault.
* test: add nbdkit fault injection to the storage-fault e2e suite
The predastore freeze takes the whole storage cluster away from every
guest on it, which is too broad to run beside anything else. Removing a
single volume's NBD server is the same fault at one-volume scope: one
host, one guest, nothing cluster-wide stopped.
Both signals are here because they sit on opposite sides of the defect.
SIGSTOP holds the socket open and answers nothing, so the guest blocks
and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at
the default reconnect-delay of 0 fails requests at once, which under
werror=report aborts the journal.
Signalling is guarded: a pid is matched on comm, our plugin and the
volume argument together, and re-confirmed immediately before the
signal, because these hosts run nbdkit for other volumes and pids are
reused.
* fix: pause a guest on a backend I/O error instead of handing it EIO
Every virtio-blk device was created with werror=report,rerror=report, so a
backend outage was delivered to the guest as EIO. ext4 aborts its journal on
that and remounts read-only, and the write that failed is simply gone. Under
werror=stop QEMU pauses the VM and holds the failed request, so a backend that
comes back resumes the guest with nothing lost.
The boot drive, the cold-boot data volume and the hot-plugged data volume all
move to stop. Alongside it the NBD transport gets reconnect-delay=30, which
pauses requests rather than failing them while a storage process restarts on
the same socket; on the legacy -drive form it has to be file.reconnect-delay,
because the bare name reaches the raw format driver, which has no such option.
The QMP heartbeat now recognises run state io-error and issues cont. After ten
attempts the instance is reported impaired and the cadence drops to one attempt
per ten heartbeats, but it never stops: QEMU holds the request for as long as
the guest exists, so a backend that returns an hour later can still be resumed.
The storagefault suite gains the fault this was written for — kill the volume's
own nbdkit, hold, then put it back on the same socket — and it now verifies a
sha256 region across the outage rather than trusting the guest to be reachable.
Three harness defects went with it: the socket-liveness poll ran unprivileged
against a 0770 directory and could never succeed, the relaunched socket was
left 0755 where QEMU needs group write, and the two variants shared one guest
so a fault in the first was diagnosed in the second.
* test: cover the io-error resume policy
* test: stop two parallel tests racing on the terminate backoff variable
* fix: report a guest paused on a backend I/O error as impaired
A guest held by werror=stop kept reporting ok on both status checks. QMP stays
responsive while the guest is paused -- cont is being sent and answered -- so
the QMP failure gate that DescribeInstanceStatus keys on never trips, and the
one state a customer most needs to see looked healthy.
Instance status now reports impaired/failed while the resume budget is
non-zero, timestamped from IOErrorSince so impairedSince is when the guest
paused rather than when it was escalated. System status reports impaired too:
a backend I/O error is an infrastructure fault, not a guest one, and that
distinction is what the two checks exist to draw. The counter is cleared by
the first poll that sees the guest running, so both clear on recovery.
* test: reap the relaunched nbdkit when the fault test ends
relaunchNbdkit detaches the server with setsid so it outlives the SSH session
that started it, and viperblockd never learned it exists. Nothing reaped it, so
every run left an nbdkit holding a socket and a cache for a volume that was
later detached or deleted -- four were found running on dev-prod against two
detached and two deleted volumes.
The PID is resolved from the socket path rather than the volume id, so a
legitimate viperblockd server for the same volume is never the one killed.
* test: prove the backend is serving before a fault is called cleared
Thawing the process a fault stopped is not the same as restoring the service.
The outage test treated SIGCONT as the end of the fault and went straight to
judging the guest, so it passed while the cluster it ran on stayed unable to
serve a single object for the next 35 minutes, and the next test would have
inherited that.
thawPredastore now puts and gets an object through every gate, with a budget,
and fails the test if the backend does not come back.
* fix: do not hand back a mount entry whose export is gone
A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit
was gone. Every later ebs.mount then matched that entry and returned its socket
path, so vmMgr waited for a socket nothing would ever create and the instance
could not be started again — on dev-prod a guest stopped against a frozen
backend was left permanently stopped, four start attempts failing identically
on a socket the node had already removed.
Drop the entry when there is positive evidence the export is dead: the socket
is absent, or the recorded pid is not running. Neither can be true of a live
export, so the double-writer hazard the guard exists to prevent is unaffected.
Also make the durability test's pattern helpers report failures. The read dd sat
in a pipeline, so its exit status was discarded and the harness hashed nothing
and returned the empty-string sha256 as a checksum.
* test: report the guest console when a restart never answers SSH
A restart that reaches running but never comes back is indistinguishable from
a dead root filesystem without the serial console, and the log is gone once
the fixture terminates the instance.
* fix: converge guest port datapaths concurrently, not one deadline after another
Each guest port that misses its SB binding burns the full convergence deadline,
and applyEIPs spent them in series. Three ports belonging to long-stopped
instances therefore held a newly launched guest's EIP dark for the sum of them
before its own probe started.
AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and
sb-cluster-state-reset is now serialised with a minimum interval so concurrent
probes cannot each fire a cluster-wide resync.
* fix: refuse a stale cross-node start after the owning node is gone
The volume lease already refused a cross-node mount, but it is a liveness
claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded
writes stops renewing, and 45 seconds later nothing refuses a start elsewhere
that loads a stale backend checkpoint.
A failed seal now writes a durable marker naming the node holding the only
current copy, cleared on a seal that reaches the backend and on delete. The
lease covers a live holder, the marker a dead one.
The refusal also reaches the caller instead of being flattened to
ServerInternal, which told an operator to retry something that cannot succeed
while the holder is down.
* fix: keep the daemon's message when a start is refused
Rebuilding the error from the code alone rendered the generic ErrorLookup
text, so a refusal naming the node holding the volume reached the caller as
boilerplate about attaching volumes.
* fix(ec2): terminate detaches every DeleteOnTermination=false volume
TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume
on the assumption that Stop's Unmount had already cleared its attachment.
Unmount only clears it when the seal succeeded, so a volume whose seal failed
stayed in-use pointing at an instance that terminate then deleted, leaving it
unreachable with no owner to release it.
The running-instance path already detaches every DoT=false volume; the stopped
path now matches it, and tolerates an already-gone volume the same way.
DeleteVolume also keeps its provider error on the way out. The terminate reaper
retries the delete every two minutes and logged a bare ServerInternal, which
made a blob node being down indistinguishable from a transient fault.
* fix(ec2): DetachVolume --force releases a volume whose instance is gone
The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a
volume left attached by a part-way teardown could never be detached: the
instance no longer runs anywhere and nothing answers. That also made it
undeletable, since DeleteVolume refuses an attached volume.
ForceDetachVolume already existed for this deadlock but was reachable only from
account teardown. Force now falls back to it when the instance has no responder,
which is the one case where clearing the attachment cannot strand a live guest.
* fix(viperblock): unsealed writes steer placement instead of pinning a volume
The dirty marker refused any mount from a node other than the one holding
unsealed writes. Instance start already forwards to the node that last ran the
instance and falls back after a 30s window, so that refusal turned a working
fallback into a volume that could run nowhere once its owner was gone. It
traded a rare data-loss risk for a permanent availability failure.
The marker is now a record and a warning, not a gate. A takeover logs the
previous holder and why it could not seal, then adopts the marker so the old
node is no longer treated as the better source. Live exclusion is unchanged and
still enforced by the lease.
It is also written on open rather than on seal failure. A node killed mid-write
never reaches its seal, so the old trigger left no trace in exactly the case the
marker exists to describe.
Adds 'spx admin volumes unsealed' to see which volumes would cost something if
started elsewhere.
* fix(viperblock): fence a volume whose lease moved instead of logging it
The lease was checked once, at mount, and never again. A node that lost it
kept its export up and kept writing into a volume another node owned, so
both copies advanced and the backend settled it by arrival order.
Losing a renewal now decides: another holder means tear the export down,
no holder means the entry merely aged out, so reclaim it and leave the
guest running. Fencing never seals — this node's copy is the stale one —
and leaves the dirty marker for the winner to take over.
This is not an epoch fence. It narrows the window from the 45s TTL to one
renewal interval at no write-path cost; the backend still cannot refuse a
stale writer.
* fix(network): advertise EIPs so a recycled address is reachable
The gateway switch port only asked ovn-northd to publish the router's NAT
addresses in centralised and routed modes, so on a distributed cluster nothing
ever announced an elastic IP. OVN still answers ARP for it, which is why a
first-time allocation works and hid this — but no gratuitous ARP is sent when
the address moves to a new ENI, so every host on the segment keeps resolving it
to the previous holder's MAC until its cache ages out. Reusing an address from
the pool then lands on a dead MAC and the instance is unreachable from outside.
Distributed mode needs the option as much as the others: northd scopes each
entry with is_chassis_resident(<vif port>), so the announcement leaves from the
chassis running the instance rather than the gateway, which is what distributed
NAT wants.
Converge on the already-attached path too. AttachIGW returns early once the
gateway port exists and nothing else rewrites it, so without that an existing
VPC would stay silent for its whole life.
Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to
carrying their correct external_mac, each scoped to its own ENI's port.
* fix(viperblock): make the volume fence and its dirty marker ownership-safe
Four corrections to the lease-loss fence, all found in review.
Drop the reacquire path. Losing a lease and finding no successor was treated as
a lapsed entry worth reclaiming, but leases.acquire returns the cached entry —
already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease
renewed nothing and the node went on writing a volume it could not prove it
owned. The two errors are not symmetric: a false fence costs one guest a
restart, a false reclaim leaves two nodes writing one volume. Fence in both
cases, and report which happened through the outcome attribute.
Surrender a lease this node cannot confirm. Renewal only noticed a loss when
the server rejected an update, so a node partitioned from NATS kept its export
up while the TTL expired and another node took the volume. Track the last
confirmed renewal and give the volume up once that ages past the point where
the server may admit another owner. Bound the renewal call too, so a hung
JetStream request cannot hold the goroutine past that point.
Do not act on a kill that did not take. Both the fence and ebs.unmount removed
the mount entry, released the lease and announced the result before knowing
nbdkit had exited — the fence then reported a writer stopped that was still
running, and the unmount sealed into a directory a live process was writing.
Wait for the exit, and on failure leave the volume mounted and say so. An
already-exited process now counts as success, which is what the caller asked.
Give the dirty marker a generation. It recorded an owner and nothing to order
two nodes' claims, so a node returning after a takeover could clear or overwrite
the winner's marker and point placement back at the stale copy. Stamp the lease
generation, refuse a write from an older one, and only clear a marker this node
still owns. A marker that cannot be written now fails the open rather than
leaving writes nothing records.
* test(viperblock): cover the stall surrender and the unmount kill guard
The two ownership cases the fence work left untested.
A holder partitioned from NATS while the winner keeps it never gets an update
rejected, so revision checking cannot see it — it has to give the volume up on
its own clock. Pins that validity sits below the server TTL with room for a
check in between, that a lapsed holder reports the loss as a stall rather than
as a takeover, and that releasing a surrendered lease leaves the entry for
whoever holds it now.
The unmount side of the kill guard: sealing under a live nbdkit writes a state
torn between two writers, so a kill that did not take has to fail the unmount
and leave the entry mounted for a retry.
* fix(e2e): keep the guest private key for as long as the key pair
EnsureKeyPair computed the PEM path from the caller's directory but wrote
the file inside the ensureOnce closure, which runs only on the first call.
Callers passed a per-test artifact dir, and ArtifactDir removes that dir
when the test passes — so the first test's success deleted the only copy
of the private key while the key pair stayed registered, and every later
call returned a path nothing had ever been written to.
Guests after the first were then launched against a public key whose
private half no longer existed, so the harness could never authenticate
and gave up after the full five-minute budget. The guests were healthy
and reachable on tcp/22 throughout, which is why this read as a storage
or network fault for so long.
The directory parameter was the defect, so it is gone: the PEM now lives
in a directory the fixture owns and removes with the key pair, and no
caller can supply one that is shorter-lived.
Also reorder the killed-nbdkit teardown. The reaper registered its own
cleanup after the detach registered one, so LIFO killed the block server
before the platform detached the disk and DetachVolume then failed with
"Node ... is in use". The detach cleanup now reaps once it is done.
* fix: stop leaking an nbdkit export when recovery cannot adopt it
Restart recovery adopts a guest's boot volume, then routinely fails the
-efi volume with "claim lease: context deadline exceeded" -- JetStream is
still catching up at daemon start and the KV claim runs on the client's
5s default. Recovery gave up on the first error and never retried, so the
export stayed live with no registry entry. At terminate the unmount found
nothing to seal, returned NotFound and killed nothing, and the process
outlived its guest holding the volume and socket open. EFI is always the
second volume adopted, which is why it was always the one that leaked.
Two changes, because either alone leaves a hole:
- rebuildMountedVolume retries the build (4 attempts, 3s apart) so a
JetStream that is merely slow no longer orphans an export for the life
of the process.
- unmountVolume reaps a corroborated export for the volume it was asked
to unmount when the registry has no entry for it, so an export orphaned
by any means is still cleaned up. The reap goes through
reapOrphanedNbdkit, which declines anything a live process references:
at unmount the guest's blockdev may not be torn down yet, and killing
the server underneath it strands QEMU.
* test: hold the backend-serving probe and assert predastore readiness
The backend-outage test declared the backend restored on one successful
round trip through each gate. A gate can answer the request that arrives
just after a thaw and then lose its blob peers under the load that
follows, which is a backend that never recovered reported as one that did
-- and that is what happened: the probe reported all three gates serving
minutes before the guest was still taking BLOCK_IO_ERROR on every resume.
The round trip has to hold across three rounds now, and the count restarts
rather than decays when one fails, so a gate that serves and then stops is
the failure the hold is looking for rather than noise it averages out.
It also checks /readyz on each host and reports any failed check by name.
The round trip only proves this workstation can write through a gate; it
says nothing about which blob peers that gate can reach, and a gate one
node short of the write floor serves a small object and fails a guest. A
host that serves no admin listener is logged and skipped, since the port
is optional configuration.
* fix: scope predastore's admin port to the cluster plane in the host policy
predastore's admin listener binds the host's cluster bind_addr and serves
/healthz and /readyz unauthenticated. Readiness now names which blob peers
a gate cannot reach, which is exactly the kind of topology detail 4432 is
peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the
public one.
* fix: keep resuming a paused guest at heartbeat rate after a long outage
Past ten attempts the cont dropped to one per ten heartbeats, so a guest
whose backend had come back could sit paused for another five minutes
before anything retried it. Measured on dev-prod: the backend was proved
serving through all three gates while the guest was still waiting on
attempt 60, five minutes after attempt 50.
The slowdown was there to avoid paying a failed I/O every thirty seconds
on a backend still down. That is nearly free, and five minutes of downtime
after recovery is not, so only the log line is throttled now.
* test: fail the run when a key pair name resolves to more than one pair
A create that lands twice leaves two key pairs with one name and different
material. RunInstances resolves the name to one of them while the harness
holds the private key of the other, and the only symptom is every guest in
the run refusing the key at the SSH gate — five minutes of waiting per
guest, four guests, and nothing anywhere naming the cause. Observed on
dev-prod: two pairs created a millisecond apart, both named
e2e-key-30de10ba.
One DescribeKeyPairs after the create says so outright instead.
* test: tolerate an already-released volume in the detach teardown
Teardown runs DetachVolumeWait after a terminate that releases the volume
itself, so the detach can arrive at a volume already on its way to
available. That is the outcome it wanted, and treating the IncorrectState
refusal as fatal failed a backend-outage run whose assertions had all
passed — the guest came back, every acknowledged write verified, and the
test still went red on the way out.
VolumeInUse is still fatal: that one means the volume is attached
somewhere it should not be.
* test: carry the merged suites onto the reworked key-pair and account-scoped seams
The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir
this branch removed, and DeleteVolume's metadata lookup is account-scoped now,
so the provider-failure test has to seed and delete under the same account.
* refactor: make the fence outcome a type in one const block
Four separate exported string constants with a paragraph of comment each,
passed to RecordVolumeFence as a bare string. Group them, give them a
FenceOutcome type, and take that type at both call sites so an arbitrary
value can no longer reach the metric attribute.
* refactor: move viperblockd's wire contract into a leaf vbwire package
The daemon imported viperblockd for exactly two symbols -- the fenced
volume subject and its event -- which put the storage service back on the
daemon's dependency graph to name something it only listens for.
vbwire holds the contract instead: the subject and event, the dirty
bucket name and record shape, and the read-only listing an operator tool
needs. It imports nothing from spinifex, so both the daemon and the admin
CLI drop their edge to viperblockd entirely. cmd/service.go keeps its
one, which is genuine: it constructs the service.
* docs: drop a specific host name from the start-refusal comment
Named a real dev-prod host in a comment and in test fixtures. Say what
the name is instead.
* ci: give storagefault a dedicated nightly cell
The suite is eligible for both topologies and runs on the PR gate, but no
nightly cell ran it: the permutation cells run a deliberately narrow
subset and none of the dedicated suite cells named it.
Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation
cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that
environment would report the outage as its own failure.
* ci: run storagefault nightly on three nodes, not one
The previous commit put its dedicated cell on a single-node runner. That was
the wrong topology and the wrong justification: single-node is RS(1,0), where
the shard floor cannot be broken and a cross-node takeover after a failed seal
does not exist, so the two assertions worth having a dedicated cell for were
the two it could not make. Cell 29 is now real-multi on ci-multi.
It also corrects the claim that the PR gate already covers the multi-node case.
The suite is eligible for both topologies, but e2e.yml's default suite list
never names it, so it has never run there. A full pass is ~63 minutes on
dev-prod bare metal, almost all of it deliberate waiting -- one test holds
predastore SIGSTOPped for seven minutes -- which is too long to add to a gate
that currently finishes well inside its budget. Nightly is where it belongs.
Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since
90 minutes of suite cannot run in a 50-minute job; the permutation steps gated
on matrix.suite == '' so the dedicated cells do not also run them; and a Run
Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts
the suite against E2E_SUITES_MULTI before running it.
* fix: stop a startup KV race leaving vpcd serving keyless guests
vpcd opens five KV buckets while NATS is still coming up, each attempt
carrying a ~5s JetStream API deadline, and applied three different policies to
the same failure: spinifex-vpc-enis was fatal, while instance-state,
security-groups, subnets and vpcs each degraded silently for the lifetime of
the process. A five-second blip during a cold start therefore decided how a
node served metadata until someone restarted it.
On dev-prod all three nodes hit this after a deploy. Two were saved by
accident: their ENI bucket also timed out, so they exited 1 and systemd
restarted them cleanly. The third's ENI bucket opened in time, so it never
crashed and ran for hours with no instance record loader. With the local state
file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's
crawl died on ami-id and fell back to DataSourceNone, and every guest launched
on that node booted with no authorized_keys.
kvstore.OpenWithRetry retries for a bounded window before any policy applies,
logs the cause per attempt and the wait on a late success, wraps the last error
with the bucket name, and returns at once on a cancelled context. All five
opens use it. Failing to open the instance record space is now fatal, matching
the ENI bucket: the dependency whose loss stops guests logging in should not be
treated more leniently than the one whose loss stops ENI resolution.
Also fixes the reason the fallback was load bearing at all. The local state
file was written 0600 and vpcd reads it as a different user in the shared
spinifex group, so the local-first path had never once worked and its
mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so
an inherited umask cannot undo it.
Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an
unresolvable instance as "no key pair" with nothing logged.
* fix: scope the ime-* firewall accept to IMDS and VPC DNS only
The per-tap OVS demux delivers any guest packet addressed to
169.254.169.254 or .253 onto an ime-* host port regardless of L4 port,
and iifname "ime-*" accept then admitted all of it. Every wildcard-bound
host service — sshd, the AWS gateway, northstar, the console — answered
guest packets on those addresses; only asymmetric reply routing kept the
handshake from completing, and a guest could still drive northstar to
recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS
(.253:53), letting everything else fall through to policy drop.
* docs: document the instance-to-host plane and how to protect it without the ISO
The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by
host services and is invisible to a NIC-level or cloud firewall, since the
traffic is intra-host on the OVS bridge. Explain that the ISO ships the
scoping rule armed, that the binary installer ships the firewall off, and
that a non-ISO install must enable it with setup.sh --firewall=on or
firewall_enabled=true. Include verification from an instance and the host.
* fix: withhold the IMDS meta-data listing when the instance does not resolve
Every omission from the meta-data root is a positive claim about the instance —
public-keys/ absent means "launched without a key pair", and cloud-init believes
it, writes no authorized_keys and boots the guest keyless. So resolving the
instance was best-effort for a listing whose whole meaning depends on it: a
backend error or a miss shortened the listing instead of failing it, and nothing
anywhere recorded that the answer was a guess.
Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot
retry a 200 that is merely wrong, so this converts a permanent silent failure
into a transient loud one. servePublicKeys already had this policy for the leaf;
the root is what contradicted it.
A profile lookup that fails still serves the listing minus iam/, and now logs.
The key pair is the field a guest cannot recover from losing, while a missing
role surfaces through the credentials leaf, which reports the miss itself.
Also stops the storagefault cross-node test failing on its own access path. It
tolerates the pattern changing across a takeover, but readPattern needs sudo,
and a volume opened from the backend checkpoint may predate cloud-init writing
its sudoers drop-in — indistinguishable from an unreadable device at the dd call
site. It now checks sudo first, and reads cloud-init's recorded datasource to
tell a rolled-back root filesystem from IMDS never having answered, failing
loudly on the latter rather than accepting it as storage loss.
---------
Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>
|
||
|
Julian Sommer
|
ee258fd248 |
refactor: back the e2e Eventually helpers with testify
Delegate the hand-rolled deadline loops to require.Eventually and require.EventuallyWithT instead of open-coding the polling. |
||
|
Ben Duncan
|
ae2a1830ef | Merge remote-tracking branch 'origin/dev' into account-quota | ||
|
Ben Duncan
|
f41f5530a0 | test: add an e2e suite proving quotas gate every capped resource | ||
|
Julian Sommer
|
17cea6cd4b |
fix: gate the SG datapath probe on the target's HTTP listener
TestSGPolicyDatapath/AllowedTraffic waited 60s for curl to reach the target's HTTP server but nothing ever waited for that server to start. Its cloud-init runs later than the client's sshd, which is what the preceding stage actually gates on, so the budget was measured from the wrong baseline and the gap between the two is unbounded on a loaded node. The failures reported curl exit 7, a refused connection: a missing SG allow is a drop ACL here, which would time out as exit 28. The target guest now reports its listener on the serial console, the only channel back to a guest whose SG has no SSH ingress, and a new TargetHTTPReady stage waits for that marker before AllowedTraffic runs at all. AllowedTraffic keeps its 60s budget, which now only has to cover chassis flow install, and dumps VPC flow diagnostics when it does fail so a genuine propagation failure is attributable from CI artifacts. |
||
|
Julian Sommer
|
bdc79108e9 | fix: make the DB endpoint the only path to the engine (#830) | ||
|
Julian Sommer
|
eac0f175ba | feat: require TLS on RDS connections by default, on both engines (#790) | ||
|
Julian Sommer
|
65d2501e6b | feat: add MariaDB as the second RDS engine (#762) | ||
|
Julian Sommer
|
1c4d41bffc | fix: reach the DB endpoint from any subnet of the VPC (#738) | ||
|
Julian Sommer
|
2e09461080 | chore: cleanup code comments | ||
|
Julian Sommer
|
5e17169661 | feat(e2e): complete the RDS end-to-end suite (#608) | ||
|
Julian Sommer
|
d4d9db9e0b | feat: add DB subnet groups and DB parameter groups (#600) | ||
|
Julian Sommer
|
2569601e3d | feat: implement RDS CreateDBInstance, reconciler and endpoint DNS (#594) | ||
|
Julian Sommer
|
5b8f20b89f | test: assert ModifyVolume resize reaches the guest block device (#585) | ||
|
Julian Sommer
|
95973eeb3c | refactor: migrate spinifex off legacy nats.JetStreamContext onto nats.go/jetstream (#584) | ||
|
Josh Siviour
|
344a490a70 |
fix(vm): give relaunched data volumes named block nodes so detach can complete (#575)
buildDrives emitted every non-boot data volume as a bare -drive with no id= and no node-name=, so a stop/start (or reboot/resize) relaunch left the volume anonymous: device_del/blockdev-del had nothing to address, QEMU kept the NBD socket open, and DetachVolume stalled through nbdkit's full kill grace before giving up. Add -blockdev support to the QEMU config builder and use it for non-boot, non-EFI volumes, sharing the node-name/device-id/iothread-id naming and virtio-blk-pci device arguments with AttachVolume's hot-plug path so cold-boot and hot-attach can no longer drift apart. A data volume with no persisted HotplugPort now gets one allocated and written back during buildDrives instead of landing on an unplugged bus. Also raise unmountSealTimeout past viperblockd's kill grace (it runs the seal after the SIGKILL, so matching the grace meant an unmount that used the whole grace could never be observed as successful), and bound the e2e harness's forced teardown detach by the teardown deadline so a slow detach is reported as slow rather than as an unexplained leak. |
||
|
Josh Siviour
|
cae76ba1b6 |
test: complete the e2e live-vs-mock rebalance (epic mulga-8oyg3) (#566)
* test(volume): assert chunk GC stays wired on in CreateVolume GCEnabled was zero-valued at every viperblock construction site for a long time and no test noticed. It is not part of viperblock.VBState, so it never round-trips through a saved config.json and cannot be observed by reading volume state back. Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so the GC wiring is assertable directly against the returned config, without a live viperblock or S3 backend. The config is returned by pointer because viperblock.VB carries atomic.Uint64 fields that vet's copylocks check rightly refuses to see copied. * test(e2e): snapshot backend byte accounting before volume purge DeleteVolume purges a volume's entire backend prefix, including any chunks a leak left behind, before anything measures it. That is the same masking that hid the storage-growth leak in production: a passing E2E cannot fail on backend growth when the evidence is deleted first. Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup closure, immediately before DeleteVolume, so every volume the shared teardown path purges is measured rather than silently discarded. The snapshot is always taken and logged; the hard ratio check additionally runs for callers that registered a guest-written-bytes baseline, since most callers never attach or write to the volume they create and so have no baseline to compare against. The tolerance is the RS(2,1) erasure floor widened for per-chunk header and tail-chunk overhead. It is sized to distinguish unbounded growth from a footprint settling at the floor, not to catch bounded residual garbage. The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit would abort the DeleteVolume call queued behind it and turn a reported leak into a real one. * test(e2e): add memory-constrained concurrent-write durability regression The existing durability tests write one 4MiB sentinel, sequentially, on a comfortably sized guest, which is the exact inverse of the known-open data-loss bug: acked writes read back zero-filled after a cold reread when a guest is memory-starved during concurrent writeback. No happy-path test can reach it, because the fault lives in the intersection of memory pressure, write concurrency and cache-cold reread. Add a suite that encodes those three conditions together: a guest discovered at or below the working set, N parallel large writers launched as background jobs in a single command so they genuinely overlap, swap disabled, then sync and drop_caches before rereading. Assertions are per stream rather than aggregate so a failure names the losing writer instead of leaving whoever reads it to re-derive which of N writers lost data from one combined digest. GuestExec gains a timeout-parameterised variant. Its fixed two-minute default is sized for sub-second payloads, and a multi-gigabyte write to a network-backed volume would exceed it, so the suite would have failed by timeout while looking like it had caught the corruption. The budget scales with the working set, and a deadline is reported distinctly from a data mismatch so the two red states are never confused. The suite is dispatchable but deliberately absent from the default SUITES: the bug it reproduces is still open, so it is expected to fail and must not sit in a merge-gating path. It is registered in the workflow allow-list so it stays discoverable rather than being buried behind a skip. Also record on the live IAM instance-profile test why its assertions were not moved down to the integration tier, so the trace is not repeated. * test(e2e): drop backend byte accounting, assert a node-wide delta instead The accounting helper listed the internal predastore bucket to attribute bytes per volume. That listing is refused unconditionally: predastore authorises bucket access by same-account ownership, and the bucket belongs to the service account, so a test caller gets 403 no matter what it holds. Granting the test a service credential would invert the trust boundary to buy a measurement, which is the wrong trade. Replace it with what the S3 surface actually permits: sample node-wide allocated bytes either side of the write and bound the delta against the bytes the guest wrote. Coarser, but it measures something real rather than depending on a call that cannot succeed. The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x expansion floor plus headroom; a cluster running the recommended RS(3,2) has a 1.667x floor and only ~12% of margin left, so the constant needs revisiting if the encoding changes. * revert: drop memdurability e2e suite The suite asserted the guest read back what it wrote after a memory-constrained concurrent write, but the assertion could not fail for the reason it claimed: it compared the guest's own read against the guest's own write with the page cache still warm, so it passed whether or not the data reached the backend. A test that is green on a build with a known-open durability bug is worse than no test — it advertises coverage that does not exist. Removes the suite and its Makefile/workflow wiring, plus DiscoverInstanceTypeAtLeastMemory, which had no other caller. GuestExecTimeout stays: GuestExec now delegates to it, and the old two-minute hard cap it replaced was a latent trap for any long guest command. * test(e2e): add partial-block concurrent-write durability suite Replaces the removed memdurability suite, which passed on a build that still carried the bug it claimed to test. The engine reaches its read-modify-write path only for a partial block; a fully covered block takes a full-overwrite branch that reads nothing and can lose nothing. The old workload wrote through ext4 with 1 MiB buffers, which the filesystem coalesced into aligned full-block requests: block-layer tracing recorded zero unaligned and zero sub-4096 requests, so the path under test never ran, and writers 4 MiB apart could not collide on a block even in principle. This suite writes the shape the defect needs. Two writers issue concurrent O_DIRECT writes into disjoint halves of the same 4096-byte blocks on a raw, unformatted device, stamping every half with the round number, then drop the page cache and reread with O_DIRECT asserting every half carries the final round. One half reading back at an earlier round is the signature of the lost update. The old suite also could not fail even had it reached the path: its write phase ended in 'wait; sync; sha256sum' and reported sha256sum's exit status, while bare wait returns 0 unconditionally, so writer errors were invisible and a backend ENOSPC presented as a pass. Here each writer's exit status and exact byte count are asserted before any durability claim is made. Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on teardown; the predecessor left roughly 3 GiB of backend per run and twice drove a node under the nearfull watermark. Dispatchable but not in the default suite list until a green run and a demonstrated fail-first against a pre-fix engine exist. * test(e2e): let the fixture own the working-set volume delete The suite deleted the volume in its own cleanup and the fixture deleted it again on close, so every run ended with an InvalidVolume.NotFound from the second attempt. Detach stays — DeleteVolume against a still attached volume is what leaks volumes on teardown — and the delete is left to the fixture so the volume is released exactly once. * test(e2e): gate merges on the partial-block durability suite Verified on one node with the same test binary and workload, varying only the engine: 1758 of 8192 halves reverted a generation against the pre-fix build, 0 of 8192 against the fixed one. That is the fail-first evidence the suite needed before it could be trusted to gate anything. Adding it to the default suite list rather than leaving it dispatchable is the point. Its predecessor sat outside every path that runs, which is how it survived long enough to be discovered green on a build that still lost data. * test(integration): cover the ECR data plane in the front gate The live ecr/ suite drove docker, crane and skopeo against a real registry, yet none of it boots a guest: it needs a registry, an object store and an HTTP client. The in-process gateway already serves the OCI Distribution v2 surface on an ephemeral port with a memory-backed object store, so these run here in 0.18s instead of behind full provisioning. Covers the three claims the live tests made beyond authorization, which is already down here: a push/pull round trip asserting blobs and manifests read back byte-identical, refusal to auto-create a repository on push, and retagging an existing manifest without disturbing the original tag. Content is compared rather than status codes alone. A registry that stores a blob under the wrong digest, truncates it, or serves a different one answers 200 to every request in the sequence. The capacity/load leg deliberately stays live: it exists to put real parallel pressure on a real predastore, and a memory-backed object store cannot make that claim. * fix(e2e): make volume teardown release the volume before deleting it Fixture cleanups drain LIFO, so a suite that creates its instance before its volume tears the volume down first, while it is still attached. DeleteVolume answered VolumeInUse, nothing retried, and the volume was stranded on the node. Reordering the chain cannot fix this — registration order has no way to express the instance/volume dependency. Instead teardown is now self-sufficient for any creation order: it force-detaches, waits for the volume to reach available, and only then deletes. Instance teardown waits for terminated for the same reason, since TerminateInstances returns as soon as the request is accepted. Two things kept the leak invisible. A failing cleanup was logged and forgotten, so a run stayed green with resources left behind; failures are now reported as leaks, failing the bound test or surfacing through Close() for TestMain to fail the run. And a panic in one cleanup aborted the whole chain, stranding every resource behind it; each callback is now isolated. RegisterVolumeTeardown and the fixture had grown separate teardown paths, only one of which detached. Both now share it. |
||
|
Josh Siviour
|
48749576c3 |
test: rebalance E2E into in-process integration tier + shared-guest live scenarios (#556)
* test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — revert |
||
|
Josh Siviour
|
908bfbfc27 |
fix(ecr): enforce IAM authorization on OCI registry operations (#548)
Treat the ECR registry JWT as a signed identity pointer rather than a permission grant. Every /v2/* request now rehydrates the token's access key against current IAM/STS state, maps the OCI route to an ecr: action and repository ARN, and runs the identity-policy evaluator before dispatch. PullOnly/ReadOnly principals can no longer push, overwrite tags, or delete manifests, and key/session/user/role/account/policy revocation takes effect on the next request instead of at token expiry. Cross-repository blob mounts now require source-read permission and proof the digest is referenced by a manifest in the named source repo, closing an account-wide blob-fishing gap. |
||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Josh Siviour
|
369025f86e |
fix(e2e): fail fast on reaped instance launch + emit APM error on MarkFailed (#531)
WaitForInstanceState now aborts immediately with the recorded StateReason message when it observes a reaped state (shutting-down/terminated) while waiting for a live target, instead of burning the full 5-minute timeout on an instance that can never reach it. MarkFailed takes a context and records the failure on the active span via span.RecordError so a reaped launch surfaces in the APM error stream, correlated by trace id. Only the LaunchRunInstances path carries a live span; other callers pass context.Background() and the record is a no-op. Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2). |
||
|
Josh Siviour
|
4b877c6a93 |
fix(e2e): resolve WANHost from spinifex.toml advertise on single-node (#527)
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs() in undefined order, so WANHost=nodeIPs[0] was loopback and the guest gateway URL helpers fell through to the first non-loopback interface addr — an arbitrary co-tenant bridge on multi-bridge shared hosts (wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the advertised gateway to register. Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back to that node's host), mirroring awsgwBindIP's toml scan; fall back to the first non-loopback discovered addr only when config is unreadable. This removes the need for the SPINIFEX_WAN_IP override on shared hosts. |
||
|
Josh Siviour
|
cee73671ff |
feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct (#520)
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime) following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel over a static connectivity-tiered catalog, and non-streaming Converse routed per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire) and Anthropic-direct (Anthropic Messages wire). Per-account provider API keys are a master-key-encrypted JetStream KV secret with an optional platform default; the advertised catalog gates external models on a resolvable credential. Self-host endpoints are pinned via config (OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam. Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters, credential store), awserrors REST-JSON codes, and the bedrock E2E suite (list/get light; Converse gated on live backends). Daemon/NATS control plane and dynamic endpoint lifecycle are deferred to Phase 3. * test(bedrock): cover dispatch, router, credentials KV + endpoint config Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request / BedrockRuntime_Request end-to-end dispatch, provider router, JSON response/error writers, upstream-status mapping, the credential-store KV round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints. Lifts PR diff coverage above the 60% gate. |
||
|
Josh Siviour
|
9f349e86ca |
feat(eks): expose NVIDIA GPUs to pods (Epic B) — VFIO QMP, worker registration, device-plugin GPU access (#507)
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that addon-sync renders into the K3s auto-deploy dir. The device plugin runs with --device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector + toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes and get the device injected via the CDI spec baked by mulga-cdi-generate. Relies on containerd 2.0.5 default enable_cdi=true (no config fork). * fix(vm): scale QMP greeting deadline for VFIO passthrough guests A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map its entire guest RAM through the IOMMU before QEMU's QMP monitor answers, which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout) and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s default and reconnects against an already-running QEMU keep it too. * fix(eks): register GPU workers reliably and expose GPUs to pods Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU: - k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree and ContainerManager.Start fails before Node registration. - The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label + node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label the nvidia-device-plugin nodeSelector never matches. - Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot. Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test. * fix(eks): install ecr-credential-provider on the GPU node AMI The GPU worker's kubelet is started (from nodegroup userdata) with --image-credential-provider-config + --image-credential-provider-bin-dir= /usr/local/bin, but the eks-node-gpu AMI never built or installed the ecr-credential-provider binary. kubelet treats a referenced-but-missing provider binary as a fatal RegisterCredentialProviderPlugins error, so the node crash-loops and never registers (while the Alpine eks-node worker, which ships the binary, joins fine). Build + install it in the GPU manifest; the server-only binaries stay omitted since GPU nodes are agent-only. Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial console: Ubuntu logs only to journald, which is off-box-invisible on a worker in an isolated customer VPC, so a kubelet that fails to register left no trace. * fix(eks): give the nvidia-device-plugin its own GPU access The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run nvml.Init and generate the CDI spec in its own container, but the DS had no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted read-only at /driver-root so it can locate the driver libs + ld.so.cache, and /var/run/cdi mounted writable so the generated spec is visible to containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd 2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu. Live-validated end-to-end on the single-GPU host (RTX A1000): baked control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi pod, zero manual intervention. * test(ec2): drop obsolete restoreSlogDefault test Rebased onto dev, which dropped the restoreSlogDefault workaround in #506 (viperblock no longer mutates the global slog). The test referenced the now-deleted helper and no longer compiles. Real GPU/registration diff is independently covered (diff coverage 95.3%). |
||
|
Josh Siviour
|
e262ee0064 |
fix(elbv2): reap LBs wedged in provisioning + harden IMDS bootstrap (#496)
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never delivers a first heartbeat previously stayed in provisioning forever with an empty StateReason: promotion to active happens only on the first heartbeat and the only provisioning->failed transitions were at create/launch time. A single wedged LB in the multi-node e2e lb suite leaked its listener/target-group and cascaded into ResourceInUse and DependencyViolation on VPC teardown. - Add a lifecycle reaper that marks failed any LB still provisioning past lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and populate a provisioning StateReason so a stuck LB is self-describing. Started once at daemon startup; stopped on service Close. - Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout as retryable in createActiveLB; register idempotent LB/listener teardown up-front so no exit path leaks into the shared VPC. - IMDS: bounded retry+backoff on the instance-fact and user-data lookups so a transient gather timeout under multi-node load does not starve the lb-agent bootstrap. |
||
|
joshsiv-mulga
|
96909d0e9a |
fix(e2e): retry LB creation on sys.micro capacity reclaim race in lb suite
The internal lb subtests raced the previous subtest's sys.micro VM deallocation: fixed 15s sleeps were not enough on capacity-tight single nodes, so the LB entered terminal state failed (insufficient capacity, then nats no-responders after the daemon unsubscribed the full topic). Replace the sleeps with createActiveLB: create LB + listener, wait for active via the new non-fatal harness WaitForLBActiveErr, and on a terminal failed state (wraps ErrLBTerminalFailed) tear down fully and retry with backoff, up to 3 attempts. |
||
|
Josh Siviour
|
491f5fa7b9 |
fix: claim gateway cr-port on every reconcile miss + surface node launch failures (siv-499) (#463)
* test(e2e): capture SNAT datapath on NAT GW egress failure TestNATGateway captured nothing when the private-VM egress probe failed, so the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in the failure bundle to root-cause it. Register harness.OnFailure before the egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP, matching the eip/vpc datapath tests. Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss the egress hop where SNAT'd traffic leaves for the physical uplink, which is exactly where an egress black-hole would show. * test(e2e): add OVN pipeline diagnostics to NAT GW egress capture Capture the gateway router's per-stage logical-flow packet counts (ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's logical switch port alongside the existing OVS/conntrack/ARP dumps. When the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint which pipeline stage drops the packet upstream of lr_out_snat. * test(e2e): dump full lflow-list + datapath ct/flows for egress RCA The IP-filtered lflow dump hides the default-drop ACL flow (its match carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the allow-flows at n_packets=0, never the deny that ate the packets. Replace it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack (the standalone conntrack binary is absent on the baremetal runner) and add dpctl/dump-flows -m for kernel megaflow hit counts. * fix: claim gateway cr-port on every reconcile miss and stop masking launch failures ensureGatewayClaimed nudged ovn-controller recompute only once then polled until timeout, so a fresh VPC bring-up or chassis flap that fired the single nudge too early left the gateway chassisredirect port permanently unclaimed. Recompute on every miss instead, mirroring ensureGuestPortDatapath. aggregateResults and distributeInstancesSpread masked any node launch failure (including the resulting 5-minute launch timeout) as InsufficientInstance Capacity. Surface the real node error when one exists; keep the capacity code only for genuine capacity shortfalls. |
||
|
Julian Sommer
|
63437f759b | feat: IAM Group inline policies (#455) | ||
|
Julian Sommer
|
c9629d3bf6 | feat: IAM Groups v1 (#449) | ||
|
Josh Siviour
|
47c9808a96 |
test(e2e): add ECS end-to-end suite (awsvpc, bridge, host, ELB service) (#439)
Drives the ECS data plane against the local awsgw: a customer VPC plus a real container instance launched from the spinifex-ecs-node AMI, then standalone tasks in all three network modes and an awsvpc service fronted by an ALB target group. Adds an ECS client to the harness, a Makefile target, and a service-interfaces suite entry. bridge/host tasks share the host netns where the agent credential endpoint owns 169.254.170.2:80, so they run a no-port command and assert placement + mode; awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks after a settle so a transient RUNNING cannot pass. |
||
|
Julian Sommer
|
8a94886b13 | feat: cloud-init EC2 datasource (#411) | ||
|
joshsiv-mulga
|
5fa12ec358 |
fix(e2e): skip stopped node in multinode NATS peer poll
StopNode now stops spinifex-nats.service, so WaitNATSPeers polling that node's dead 127.0.0.1:8222 monitor port fails with curl exit 7 and never converges. Add WithSkipNodes to exclude the stopped node from the poll; TestMultinodeNodeFailure skips node2. |
||
|
Julian Sommer
|
fa066a64d7 | fix | ||
|
Julian Sommer
|
e11437cec9 | feat(imds): per-tap responder datapath (#408) | ||
|
Bryn Mailer
|
15dede64ad |
feat: predastore compaction E2E (#407)
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
|