mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 02:43:17 +00:00
dev
61 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
71301983bd |
feat: add read-only EC2 platform inventory for account termination (#1212)
* feat: add read-only EC2 platform inventory for account termination * fix: drop placement-group member instance IDs from the platform inventory reply |
||
|
Josh Siviour
|
b5879f44ff |
ADR-0001 package-boundary migration checkpoint (#1190)
* refactor: move telemetry file naming to contract
* docs: record telemetry contract move
* refactor: move aws xml response helpers
* docs: record aws xml helper move
* refactor: move aws tag projections to foundation
* docs: record aws tag projection move
* refactor: move account key to state foundation
* docs: record account key state move
* refactor: move host privilege escalation to runtime
* docs: record host privilege move
* chore: resolve lint findings from package-boundary moves
* refactor: move process supervision helpers to runtime host
* docs: record process supervision move
* refactor: move nbd endpoint helpers to compute runtime
* docs: record nbd endpoint move
* refactor: move image catalogue to operator
* docs: record image catalogue move
* refactor: move resource id generation to aws foundation
* docs: record resource id move
* refactor: move predastore role to runtime roles
* docs: point service manifest at acm domain
* docs: record predastore role move
* refactor: move northstar role to runtime roles
* docs: record northstar role move
* refactor: move awsgw role to runtime roles
* docs: record awsgw role move
* refactor: move error payload helpers to aws errors
* docs: record error payload move
* refactor: move account identifiers to aws identifiers
* docs: record account identifier move
* refactor: move NATS transport to messaging foundation
* docs: record NATS transport move
* refactor: move system instance contract to EC2 domain
* docs: record system instance move
* refactor: move system VPC builder to network domain
* docs: record system VPC move
* refactor: move instance metadata service to EC2 domain
* docs: record guest metadata move and dependency-rule debt
* refactor: move DNS control plane to DNS domain
* docs: record DNS domain move
* refactor: move base zone bootstrap to DNS domain
* docs: record base zone bootstrap move
* refactor: replace SystemAccountID with GlobalAccountID
* docs: record SystemAccountID removal
* refactor: extract CA key-pair loader to foundation pki
* docs: record foundation pki extraction
* refactor: move tenant CA paths to ACM domain
* docs: record tenant CA path move
* refactor: move IPsec certificate handling to network host
* docs: record IPsec certificate move
* refactor: move Ochre serving lifecycle to the Ochre domain
handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.
* docs: record Ochre domain move
* refactor: move Ochre vector store to the Ochre domain
handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.
* docs: record Ochre vector store move
* refactor: move Hugging Face Hub client to the Ochre domain
gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.
* docs: record Hugging Face Hub client move and Ochre gateway debt
* refactor: move EC2 tag mutation helpers to the EC2 domain
utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.
* docs: record EC2 tag mutation move and migrate split finding
* refactor: move network packages under the network domain
Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.
* docs: record network domain move
* refactor: move standalone EC2 resource packages to the EC2 domain
Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.
* docs: record standalone EC2 resource moves
* refactor: move EC2 VPC networking packages to the EC2 domain
Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.
* docs: record EC2 VPC networking moves
* refactor: move EC2 compute and storage packages to the EC2 domain
Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.
* docs: record EC2 compute and storage moves
* refactor: move ECR credential provider to the EKS agent tree
Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.
* docs: record ECR credential provider move
* refactor: move EKS gateway publish helper to the EKS agent tree
Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.
* docs: record EKS gateway publish move
* refactor: move EKS gateway fetch helper to the EKS agent tree
Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.
* docs: record EKS gateway fetch move
* refactor: move EKS webhook cert helper to the EKS agent tree
cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.
* docs: record EKS webhook cert move
* refactor: move EKS konnectivity cert helper to the EKS agent tree
cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.
* docs: record EKS konnectivity cert move
* refactor: move EKS token webhook to the EKS agent tree
cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.
* docs: record EKS token webhook move
* refactor: move ECS agent runtime and credentials packages to the ECS agent tree
* docs: record ECS agent subpackage move
* refactor: move ECS agent implementation to the ECS agent tree
cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.
* docs: record ECS agent move
* refactor: move RDS guest agent to agents/rds/agent
* docs: record RDS guest agent move
* refactor: move installer packages to operator/installer
* docs: record installer move
* refactor: move spx command tree to operator/cli
Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.
* docs: record spx command tree move
* chore: accept known vulnerabilities with an expiry
The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.
* refactor: move EC2 account action adapters to domains/ec2/awsapi
* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi
* refactor: move EC2 key action adapters to domains/ec2/awsapi
* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi
* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi
* docs: record EC2 standalone action adapter moves
* refactor: move EC2 zone action adapters to domains/ec2/awsapi
* refactor: move EC2 vpc action adapters to domains/ec2/awsapi
* refactor: move EC2 eigw action adapters to domains/ec2/awsapi
* refactor: move EC2 eip action adapters to domains/ec2/awsapi
* refactor: move EC2 igw action adapters to domains/ec2/awsapi
* refactor: move EC2 natgw action adapters to domains/ec2/awsapi
* refactor: move EC2 routetable action adapters to domains/ec2/awsapi
* docs: record EC2 VPC networking action adapter moves
* refactor: move EC2 image action adapters to domains/ec2/awsapi
* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi
* refactor: move EC2 volume action adapters to domains/ec2/awsapi
* refactor: move EC2 tags action adapters to domains/ec2/awsapi
* docs: record EC2 storage and tag action adapter moves
* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi
* refactor: move EC2 idempotency adapter to domains/ec2/awsapi
* refactor: move EC2 instance action adapters to domains/ec2/awsapi
* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi
* docs: record EC2 instance action adapter moves
* refactor: move AWS REST-JSON router to ingress/aws/rest
* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit
* docs: record AWS ingress helper moves and tooling conformance
* docs: correct historical lifecycle references
* test: characterize ECR control-plane dispatch before registration seam
* feat: add AWS ingress service registration contract
* docs: record ingress registration phases 0 and 1
* refactor: route registered AWS services through the ingress dispatch registry
The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.
The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.
* docs: record ingress dispatch registry consumption
* refactor: register the ECR control plane from its domain
domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.
Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.
* docs: record ECR control-plane registration
* fix: compose the ECR control plane in the integration harness
The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.
* docs: record ingress pipeline deferral
* refactor: extract AWS error envelope rendering to ingress
ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.
The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.
* docs: record error envelope extraction
* refactor: move AWS service selection into ingress dispatch
Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.
* docs: record ingress service selection move
* test: characterize ACM control-plane dispatch before registration
Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.
* refactor: register ACM control plane from its domain
domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.
* docs: record ACM registration
* docs: record ECS, EKS and RDS source inventory and first-resource ranking
* docs: record EKS control-plane secrets persisted in the cluster record
* chore: drop containerd advisory acceptance now that dev ships v2.4.1
* test: characterize RDS DB subnet group lifecycle before extraction
* docs: record RDS DB subnet group lifecycle contract
* refactor: extract the RDS DB subnet group lifecycle owner
domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.
* docs: record RDS DB subnet group owner extraction
* docs: correct RDS subnet group tagging debt and conformance scope
* test: characterize ECS task definition lifecycle before extraction
* docs: record ECS task definition lifecycle contract
* refactor: extract ECS task definition owner into domains/ecs/taskdefinition
Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.
The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.
List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.
* docs: record ECS task definition owner extraction
* test: characterize EKS access entry lifecycle before extraction
* docs: record EKS access entry lifecycle contract
* refactor: extract EKS access entry owner into domains/eks/access
Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.
Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.
List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.
* docs: record EKS access entry owner extraction
* test: characterize RDS engine catalogue before extraction
* fix: return InvalidParameterValue for a missing RDS master password
ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.
* docs: record RDS engine catalogue boundary and evidence
* refactor: extract RDS engine catalogue into domains/rds/engine
Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.
The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.
The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.
* docs: record RDS engine catalogue owner move
* test: characterize RDS DB parameter group lifecycle before extraction
* docs: record RDS DB parameter group lifecycle contract
* refactor: extract RDS DB parameter group owner
Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.
* docs: record RDS DB parameter group owner extraction
* test: characterize EKS add-on lifecycle and guest wire
Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.
* docs: record EKS add-on lifecycle contract
* fix: require iam:PassRole for EKS add-on service account role
CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.
* docs: record EKS add-on PassRole check
* refactor: extract EKS add-on owner
Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.
* docs: record EKS add-on owner extraction
* refactor: extract EKS add-on guest wire contract
Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.
* docs: record EKS add-on wire contract extraction
* docs: update EKS add-on inventory and remaining edges
* docs: draft EKS add-on lifecycle correction design for review
* chore: run contracts tests in the default test, coverage and race targets
* docs: apply review decisions to EKS add-on lifecycle design
* test: characterize EKS add-on v1 compatibility with added fields
Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.
* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
* docs: draft EKS add-on v2 guest contract design for review
* docs: qualify historical architecture references
* docs: apply review decisions and ADR-0006 to EKS add-on v2 design
* fix(eks): bind internal report publishing to the caller's cluster
PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.
PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.
* fix(eks): bind webhook token review to the caller's cluster
WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.
WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.
* fix(eks): validate add-on serviceAccountRoleArn syntax
CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.
The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.
* docs: record EKS internal-route and role-ARN security corrections
* docs: add EKS add-on release membership and activation case study
* fix: point UI build, CI and lint rules at moved package paths
* docs: record EKS add-on activation review decisions
* refactor: move quota admission to domains/admission/quota
Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.
* docs: reconcile package-boundary record for ADR-0001 close-out
Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.
* test: move KV migration engine tests to foundation/state/migrate
The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.
* refactor: move config and object-store upgrade workflow to operator/upgrade
Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.
* docs: record migrate move to operator/upgrade
* test: enforce ADR-0001 layering with a blocking ratchet
Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.
Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.
* refactor: drop the legacy handlers_ecr import alias
Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.
* chore: mark spf13/pflag as a direct dependency
operator/cli/wiring.go imports pflag directly. Same version; no upgrade.
* docs: record test-package-check and nilaway as unverified gates
* refactor: move the layering test to module-root architecture/
The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.
* fix: update paths left stale by the package moves
The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.
* refactor: move the default-VPC subject into contracts/ec2/v1
utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.
* docs: qualify the migrate split and record admin and Viperblock findings
The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.
* refactor: move operator-only admin code to operator/admin
Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.
foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.
* docs: record operator/admin split in package-boundary migration
* test: pin viperblockd fence, abandon and dirty-marker wire shapes
Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.
* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts
Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.
Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.
* refactor: split the Viperblock EBS adapter out of services/viperblockd
Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.
The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.
Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.
* refactor: move the viperblockd role to runtime/roles/viperblockd
With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.
* docs: record viperblockd contract, adapter and role moves
* test: characterise vpc.* network-realisation routes
Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.
Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).
* feat: add contracts/network/v1 for vpc.* realisation routes
Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.
* refactor: use contracts/network/v1 for vpc.* producers and consumers
Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.
Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.
No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.
* refactor: replace utils/vpcd_event.go with a typed network projection client
Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.
utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.
* refactor: route the remaining fire-and-forget vpc.* producers through projection
CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.
* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils
The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.
utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.
* docs: record the network contract, projection client and utils removal
* test: characterise instance-state persistence, recovery and generation handling
Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.
* test: freeze direct instance-state consumers
Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.
* docs: record ADR-0007 slice 0 instance-state inventory
Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.
* test: narrow the instance-state ratchet to instance-state access
Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.
* docs: record the narrowed instance-state ratchet commit
* docs: design the instance-state transition record and activation gate
* docs: record the instance-state transition-record design commit
* docs: record slice 1a activation-gate decisions
* feat: add instance transition-record codec with format marker
Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.
Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.
No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.
* docs: record slice 1b step 1 codec and awsgw rollout qualification
Records the codec added in
|
||
|
Julian Sommer
|
19c35326a3 | chore: fix lint issues | ||
|
Julian Sommer
|
b3b066562e | chore: add comment to all exported types | ||
|
Julian Sommer
|
810aa019bf | chore: use strings.Cut | ||
|
Julian Sommer
|
610ee9bab0 | chore: use slices.Sorted | ||
|
Julian Sommer
|
410faf3c83 | fix: EC2 UnauthorizedOperation, MaxResults ranges and tag resource types as AWS answers them (#1134) | ||
|
Julian Sommer
|
bfad1dc004 |
test: target fixture resources in generated conformance requests
Salts restart per operation, so tag, untag and update cases only reached a resource when the same-numbered create case had made one. Create an IAM, ECS and ECR fixture per run and point every operation but a create or delete at it, and give ECR's policy and manifest documents values AWS accepts. |
||
|
Julian Sommer
|
b3be052311 |
test: remove integration conformance suite artefacts
Clear the warnings the conformance suite caused itself, so what remains is a real divergence from AWS: - Seed conditionally required members into the base request of RDS CreateDBInstance and RestoreDBInstanceFromDBSnapshot, ECS CreateCapacityProvider and CreateService, and EC2 CreateCapacityReservation. - Give loosely modelled members values AWS accepts: ELBv2 ProtocolVersion and Matcher.HttpCode, EKS policyArn, addonName and access-entry type, EC2 SourceRegion and STS AccessKeyId. - Import a CA-signed certificate, key and chain in ACM ImportCertificate. - Rewrite ACM's lookahead domain pattern as RE2 so RequestCertificate is generated, and give the integration ACM service a tenant CA permitting example.com. - Run the service daemon-lite behind the cross-tenant boundary test. - Accept ELBv2's common parameter validation errors in responses. |
||
|
Julian Sommer
|
2898181837 | fix: test packages | ||
|
Julian Sommer
|
2d2d2f27ed | feat: model-driven request conformance in the integration suite (#1122) | ||
|
Julian Sommer
|
0c3403889c | feat: load AWS conformance models from api-models-aws (#1121) | ||
|
Julian Sommer
|
c473365f1a |
refactor: move new test files to external test packages
Tests added since main used in-package test packages, failing the test-package check. Convert them to <pkg>_test with export_test.go hooks; the ecs-agent main package keeps an in-package marker. |
||
|
Josh Siviour
|
451df1a8cd | docs: publish AWS operation coverage inventory (#1075) | ||
|
Julian Sommer
|
c0d54e6131 | docs: publish one column of implemented operations and nothing else | ||
|
Julian Sommer
|
881a66ab57 | docs: publish only the operations the platform serves | ||
|
Julian Sommer
|
b89334d76f |
docs: keep the not-applicable operations out of the internal gap list
The internal report listed every unimplemented operation together, so the 87 EC2 operations declared not applicable sat in the same list as the work, where anything reading it for candidates would pick them up. They are now listed apart, grouped under the reason they were declared with, and the counts line reports them as not applicable rather than as gaps. |
||
|
Julian Sommer
|
3b523498fc |
docs: stop publishing planned RDS work as never
Multi-AZ clusters, read replicas and point-in-time restore were declared not applicable, which says the platform will never serve them. All three are planned, so the pages claimed something the roadmap contradicts. Only a declaration now publishes that claim. A handler that refuses an operation today establishes nothing about tomorrow, so it reads as the gap it is rather than a permanent one. |
||
|
Julian Sommer
|
c24636c734 |
docs: put the not-applicable operations back in the one table
Splitting them into a second table repeated a reason on every row that shared it and separated an operation from the rest of the surface it belongs to. One table carries every operation the platform answers, served or refused, and the reasons return to numbered footnotes below it. |
||
|
Julian Sommer
|
c8d42ac160 |
docs: publish S3 API coverage
S3 was the one served surface with no coverage page. The gateway has no operation-name dispatch table for it, so there was nothing to compare against the model. Predastore now declares the operations each of its S3 routes answers, so the generator compares that inventory the same way it compares the other nine services. The page credits Predastore rather than Spinifex, and its intro separates what is routed from what conforms, pointing at the ceph/s3-tests report for the second. |
||
|
Julian Sommer
|
a3fbf82b9b | chore: fix coverage text | ||
|
Julian Sommer
|
05830f3499 |
docs: total the served operations on the coverage index
The index listed a count per service and left the reader to add them up. It now opens with the total and closes the table with it, counting only the services it links. |
||
|
Julian Sommer
|
e2fb37c0ac |
docs: publish only the AWS operations Spinifex serves
The coverage pages led with a percentage and an "x of y" count, and every table listed the modelled operations Spinifex does not implement, which reads as a scorecard against AWS rather than a statement of what the platform serves. A published page now opens with the operation count alone, tables the operations the gateway serves, and carries a separate not-applicable table with each reason inline instead of footnotes. The unimplemented operations and the counts move to docs/coverage/internal-report.md, which no docs-site slug points at. A service with no implemented operations has no publishable page. |
||
|
Julian Sommer
|
ba98509c5b | fix: simplify mulga-docs build step by collapsing coverage paths + hardware | ||
|
Julian Sommer
|
613231fd18 | feat: hide internal commands from compatibility page | ||
|
Julian Sommer
|
19a047bccb | fix: footnotes instead of notes for compatibility | ||
|
Julian Sommer
|
bba735497f | fix: add not applicable to some actions | ||
|
Julian Sommer
|
29faaa78fd | chore: update compatibility pages | ||
|
Julian Sommer
|
fb6494da01 | feat: coverage docs | ||
|
Julian Sommer
|
2265aee120 | feat: add eks to aws coverage | ||
|
Josh Siviour
|
31eb3bc672 |
fix: fail a credential refresh instead of extending a dead key's expiry (#995)
ec2rolecreds implements aws.HandleFailRefreshCredentialsCacheStrategy, which aws.CredentialsCache prefers over returning the refresh error. When the cached credential is expired or within five minutes of expiring and the refresh fails, it hands back the same access key, secret and session token with Expires pushed a random five to fifteen minutes out, reported as a success. Every later refresh fails the same way, so ecs-agent signs with a credential the control plane has already rejected, indefinitely, and nothing surfaces it. The cache picks its strategies by asserting on the provider's dynamic type, so returning aws.CredentialsProvider from NewProvider does not disarm it. Wrap the provider in a type whose method set omits HandleFailToRefresh, and the cache falls through to returning the error. AdjustExpiresBy is forwarded on purpose: it only fires under an ExpiryWindow, which no caller sets today, but dropping it silently would change behaviour for whoever sets one. Fixed in imdscreds rather than at the one exposed call site, so a future caller that wraps this provider in a cache is safe by construction. ecr-credential-provider was never affected: it calls Retrieve uncached. Two comments claimed the hazard was ec2rolecreds capping Expires to an hour. That cap only ever shortens a too-far-future expiry and is unrelated. |
||
|
Julian Sommer
|
95afec6a76 | refactor: use bluebottle/pkg/fipsboot for the FIPS 140-3 startup guard (#911) | ||
|
Julian Sommer
|
97acfbbd80 | refactor: use bluebottle/pkg/tlsconfig for the TLS curve allowlist (#910) | ||
|
Julian Sommer
|
8e01e76dc0 | fix: disable embedlit | ||
|
Julian Sommer
|
f21db26816 | chore: block panic | ||
|
Julian Sommer
|
6b71fb876d | fix: use stdlib sort | ||
|
Josh Siviour
|
fd8be8a814 |
refactor(test): route duplicated NATS bootstraps through testutil (#776)
* refactor(test): route duplicated NATS bootstraps through testutil Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that spinifex/testutil already provides, and one seedKV clone duplicated testutil.SeedKV. Route them through the shared helpers. Sites whose options genuinely diverge are left alone: fixed ports needed for reconnect and restart scenarios, custom TLS and token auth, a custom MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup. The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than CreateKeyValue, so their failure behaviour differs and they stay as they are. Also moves eight test files to external test packages where they declare no helper used by a sibling in the same package. Coverage is unaffected: Go compiles internal and external test packages into one binary per package, so attribution is identical either way. * chore(test): require new test files to use an external test package Adds a preflight check that fails when a newly added test file declares an in-package test. External test packages bind tests to the exported API and let packages share test doubles, which an in-package double cannot be. Only added files are checked, so the existing in-package tests are untouched. e2e and integration suites are exempt: both share a package-level fixture built in main_test.go, so their files cannot move individually. A file that needs unexported access can opt out with a //test:in-package comment. |
||
|
Julian Sommer
|
8f930e6275 |
refactor: consume the shared packages from bluebottle
Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather than predastore, so the control plane no longer depends on the S3 server for shared auth and policy code. |
||
|
Julian Sommer
|
456a60717b | feat: add RDS to awsmodel coverage | ||
|
Julian Sommer
|
e3b5701f15 | fix: authorize RDS data-volume formatting by exact volume identity (#737) | ||
|
Julian Sommer
|
8c07f80032 | chore: remove the half line breaks | ||
|
Josh Siviour
|
688ce5ccfd | feat: add AWS model conformance coverage (#736) | ||
|
Julian Sommer
|
615578a2e7 | refactor: simplify RDS internals (#692) | ||
|
Josh Siviour
|
5de598e85a |
ci: trim unused E2E builds, move IMDS to the SDK, derive predastore bind in spx (#656)
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs These two jobs immediately rebuild suite-aware binaries after Build Install Artifacts and never consume the tar's _bin contents, so the cross-compiled suite build was pure waste. build-install-artifacts.sh gains an optional --skip-e2e-build flag (positional contract unchanged); e2e-nightly.yml and the other callers keep building the full set. * refactor: replace hand-rolled IMDS client with AWS SDK v2 internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials provider used a hand-written IMDSv2 token/GET flow with manual RFC3339 expiry parsing. A malformed Expiration silently fell back to "never expires" instead of failing. Replace it with feature/ec2/imds, credentials/ec2rolecreds and aws.CredentialsCache, following the internal/gwsign/signer.go precedent. An unparseable Expiration is now a hard error from Fetch, covered by TestFetch_BadExpirationErrors. Output shapes to callers are unchanged, and IMDS base URLs ending in /latest keep working. * refactor: derive predastore bind config from spinifex.toml in spx itself predastore-start.sh awk-parsed spinifex.toml for the bind host/port and node_id, exported them as env vars, then exec'd spx. Move that derivation into predastoreStartCmd via derivePredastoreBind, which reads the raw (non-DIAL-normalized) nodes.<node>.predastore section, defaulting node_id to -1 (co-located) when the key is absent, same as the script. An explicit --host/--port/--node-id flag or SPINIFEX_PREDASTORE_* env var still overrides the derived value. Delete the now-unused script and point the systemd unit's ExecStart directly at spx service predastore start, matching the spinifex-viperblock.service precedent. |
||
|
Josh Siviour
|
6ce8b9ff18 |
fix: fail preflight when a gate tool is killed instead of reporting it clean (#633)
A tool whose child dies by signal can still exit 0 through go tool, so govulncheck printed 'ok' after being OOM-killed and preflight passed without a vulnerability scan having run. Route lint, govulncheck and nilaway through scripts/run-gate.sh, which fails on a non-zero exit or on signal-death output. |
||
|
Julian Sommer
|
32d80b473a | feat: RDS v1 foundations (#592) | ||
|
Bryn Mailer
|
6cdb6000ce | feat: presigned urls (#533) | ||
|
Josh Siviour
|
e47baf321c |
feat(eks): HA control-plane recovery + reconciliation, VM restart hardening, viperblock AEAD pin (#480)
* feat(eks): reconciler in-place control-plane restart on sustained degradation Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only recorded health; a CP VM that died (OOM/host crash) left the cluster wedged ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past a grace window, StartInstance is issued (backoff-spaced, capped) when the instance is stopped/error, re-mounting the same root volume so embedded etcd survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path. Wiring of a real CPInstanceControl in the daemon and the error-state manager-eviction fix are the remaining Phase-1 integration steps; Phase 2 (etcd snapshot cadence + restore-snapshot) is tracked separately. * feat(eks): wire control-plane restart into the reconciler via instance service Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1). cpControlAdapter binds the daemon instance service + account: InstanceState reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped CP via StartStoppedInstance, which loads from the shared stopped store and re-mounts the same root volume so embedded etcd survives. Wired as EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler. Recovers a cleanly-stopped control plane co-located with the reconciler leader. Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an aggregating describe; and an error/crashed CP (not in the stopped store) needs a dedicated error-state restart path — both tracked as Phase-1 follow-ups. * feat(eks): cross-node CP describe + error-state control-plane restart The reconciler's control-plane recovery was wired to the node-local instance service: DescribeInstances only saw the leader's node and StartStoppedInstance only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in error state, still resident in its owning node's manager map (not the stopped store), so recovery never reached it — the actual incident in the recovery plan. Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances fans out across every host (gateway scatter-gather) so the reconciler observes a CP on any node. RecoverInstance drives the gateway StartInstances two-step: ec2.cmd.<id> recovers a live error/running owner in place (the owner stays subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting the etcd root volume), falling back to ec2.start shared-KV rehydration for a stopped CP. The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that hide so the owning account resolves its own managed CP when targeting it by explicit instance ID (bulk listings still hide it). EC2APIState already maps the internal error state to the AWS closed-enum stopped, which cpRestartable accepts. * fix(eks): reconcile CP as system account, not customer The control-plane VMs are launched under the system account (admin.SystemAccountID()), not the customer account that owns the cluster record. spawnReconciler was binding cpControlAdapter to the customer account, so InstanceState/RecoverInstance ran as the customer — which cannot see or own its own cluster's system-owned CP VMs. DescribeInstances returned zero and the reconciler logged 'control-plane instance not found', never restarting a wedged CP. Bind the adapter to the system account instead. Revert the earlier DescribeInstances managed-VM relaxation: it was based on the mistaken assumption that CP VMs are customer-owned. Visibility is resolved by the system-account adapter, so the explicit-ID escape hatch is dead code. * fix(vm): pre-create mgmt tap on disk-boot restart path The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap (owned by the daemon euid), but the disk-boot/else branch only appended the QEMU tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the disk-boot path QEMU tried to create /dev/net/tun itself and failed with 'Operation not permitted', failing every stopped-instance restart (StartInstances returned ServerInternal). This blocked EKS control-plane recovery: the reconciler found the stopped CP and issued the restart, but the VM never booted. Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot. * feat(eks): recover all HA control-plane members, not just primary The reconciler restarted only ControlPlaneInstanceID (the primary CP member), so on an HA cluster the other server VMs stayed stopped and etcd never regained quorum (1/3) — apiserver healthz stayed fail even after recovery ran. maybeRecoverControlPlane now iterates every member in ControlPlaneNodes (falling back to the scalar ID for single-node clusters), restarting each stopped/error member per pass while preserving the grace window, attempt cap, and backoff gates. Refs mulga-siv-509 * fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data The eks-node system VM's user-data is delivered only via IMDS (the NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap br-imds datapath reconciles, which on the first-launched (primary) CP can lag cloud-init's stock 120s probe window. With the default max_wait cloud-init falls back to DataSourceNone, permanently losing the launch user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster in CREATING. Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to 600s so cloud-init keeps polling until the datapath is ready instead of falling through to None. Worst case a slow boot, never a lost role. * fix(eks): warm IMDS instance-role creds before first SigV4 sign On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook, eks-gateway-publish, mulga-eks-state-report) sign via the SDK default chain, which resolves in-VM IMDS instance-role creds. The per-tap br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's IMDS client fast-fails each probe (250ms dial), so early Retrieve calls return 'context deadline exceeded' / 'no EC2 IMDS role found'. eks-gateway-publish only retries for 30x5s=150s, shorter than the datapath warm time, so it exhausts its budget and drops the bootstrap/state publish -> cluster hangs in CREATING. The credential chain is sound (the CP role trusts ec2.amazonaws.com and AssumeRoleForInstance evaluates that principal); the failure is timing. Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the instance-role creds first retrieve (2s interval, 10m ceiling aligned with cloud-init max_wait), so every helper's first signed request already holds creds instead of racing a cold datapath. Static-key mode is unaffected. * fix(eks): retry CP data-NIC DHCP until cross-host datapath is up A single ~16s DHCP one-shot on the primary data NIC lost the race against the cross-host OVN datapath (logical-port binding + per-tap reconcile on the remote placement host), stranding the NIC IP-less so the Ec2 datasource never reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet address before success. Each attempt still exits leaving no daemon, so cloud-init's later ENI management is unaffected. * fix(eks): drop reconciler registry entry when Run self-exits ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel (Stop/StopAll). When a ClusterReconciler's Run returned on its own — a terminal cluster state or a lost lease — nothing cancelled the ctx, so the holder leaked and a later same-name create silently no-opped at the head of Spawn, never reconciling to ACTIVE. Thread a finished channel from RunClusterReconciler's Run goroutine back to Spawn's watcher, which now cleans up on whichever fires first: an external cancel or Run exiting. On self-exit it deletes its own holder and releases the lease, so a fresh same-name create re-spawns and reconciles. * fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local CP user-data is delivered only via IMDS since the NoCloud seed was retired (#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net DHCPs the data NIC that makes IMDS reachable, but it was added to the default runlevel and only declared `before cloud-init` (the network-stage service). cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC route and burned its whole probe window (~4 min) before mulga-mgmt-net brought eth0 up, regressing CP bringup from ~1m50s to ~5m54s. Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag. * fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch RestartCrashedInstance re-launched a crashed VM without refreshing Instance.LaunchTime. The pending watchdog measures now-LaunchTime against PendingWatchdogTimeout, so the stale original launch time tripped launch_timeout on the next tick and terminated the crash-restart mid-flight. Refresh LaunchTime under the state lock, mirroring the daemon-restart restore path (restore.go), so the relaunch gets a fresh timeout window. Adds a regression test asserting LaunchTime advances. * feat(eks): reconcile control-plane member count after a lost member The cluster reconciler self-heals a stopped/error control-plane VM in place but never replaced a terminated/gone member, so an HA cluster ran below haControlPlaneCount permanently. Add a member-count reconcile pass that provisions one replacement CP joining the surviving etcd quorum. - CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the persisted create template on a free host, the single-node analogue of the spread join branch, with rotating creds re-derived. - ControlPlaneTemplate persisted on ClusterMeta at create (per-node and rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead node for the replacement and refreshes the [0] scalar mirrors. - maybeReplaceControlPlaneMember fires only when no member is restartable and one is truly lost, a healthy majority survives, bounded by grace, backoff and an attempt cap; placement excludes live-member hosts. - Replacement prunes the superseded member from etcd once joined: k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum width returns to N. json tags added to K3sServerInput (now persisted). * feat(eks): in-guest control-plane self-diagnosis on healthz fail When a control-plane apiserver reports unhealthy, the reconciler only saw healthz=fail with no cause, and the guest has no on-demand command channel (serial root login is password-gated, no guest agent). Make the CP self-diagnose and relay the cause through the one-way state report. - mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free space; on fail write a verbose block to the host-captured serial console and add a JSON-safe reason to the state payload. - ServerStateReport: add Reason (omitempty; empty from a healthy CP or an older AMI predating the field). - cluster_reconciler observe(): surface the reason in the health issue. - Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests for the reason-bearing and terse health-issue paths. * feat(eks): 15-min etcd snapshot cadence alongside nightly Nightly alone is too coarse a recovery point for a young cluster. Install the same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir (/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3 lifecycle rule can retain the frequent rolling window shallow and the nightly copies deep. Server-role only (agents exit early); no restore-path change. Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily key prefixes, local-snapshot prune, and quiet no-op without creds. * feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset An HA k3s control plane does not self-heal after a simultaneous full-CP restart: embedded etcd never reforms quorum (etcd:unreachable sustained), and in-place VM restarts cannot fix it. Add a last-resort escalation that drives the documented k3s cluster-reset recovery, pull-based so the host never needs an inbound channel into the guest. Reconciler: maybeReformEtcdQuorum escalates only when every CP member is VM-running but the health issue carries etcd:unreachable, the cluster is HA, and a grace window has elapsed. It sets per-member recovery directives (seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by instance ID, then restarts the members. Bounded by grace/backoff/attempts; a recovered cluster clears the directives once. Enabled via WithEtcdResetRecovery alongside the existing CP control wiring. Directive plane: per-member KV directive {epoch, action, snapshot} with store helpers, an EKS service Get/Set pair over NATS-RPC, and an internal-recovery gateway route (system-cred carve-out, same as internal-addons). The CP server instance role gains eks:GetRecoveryDirective. Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that fetches its directive through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guards one-shot application, then runs `k3s server --cluster-reset` (optionally restoring a predastore snapshot) or wipes the local etcd datastore to rejoin the reset seed. Wired into the image manifest, setup.sh perms, and the eks-node-role selector. Tests: reconciler escalation gates + recovery clear, directive store epoch increment + service validation, and a POSIX unit test for the guest agent. * fix(eks): reboot running CP members to apply etcd cluster-reset directive The etcd quorum-reformation escalation drove StartInstance on the seed and followers, but the reset gate only fires when every member is VM-running. The StartInstances path rejects a running instance with IncorrectInstanceState, so the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin directive was never applied — the wedged control plane stayed wedged. Reboot the running members (QMP system_reset) instead, re-entering the boot sequence so the on-VM recovery agent applies the directive. Adds RebootInstance to the CP control surface (interface, adapter, NATS-backed daemon impl) and switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path keeps StartInstance. * fix(eks): graceful stop for etcd reset recovery The etcd quorum-reformation reset path forced running CP members through a hard QMP system_reset to re-enter the boot sequence and apply their recovery directives. A hard reset skips a clean unmount: on the next boot e2fsck cleared the recovery and k3s runlevel entries (dangling inodes), so the cluster-reset never ran and k3s was unstartable — the reset defeated recovery. Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean unmount, SIGKILL fallback). The existing in-place restart path (maybeRecoverControlPlane) then boots each stopped member clean on a later tick, and the on-VM recovery agent applies its directive against an intact filesystem. Reuses existing lifecycle machinery; the stop also tolerates a wedged QMP client via the SIGKILL fallback. Swaps RebootInstance -> StopInstance across CPInstanceControl, cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl (gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes. * fix(vm): retry QMP dial and unlink stale socket on relaunch A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the QMP dial both raced that dead listener, so a single dial returned 'connection refused': StartStoppedInstance then reported ServerInternal and left the instance KV-stopped even though the guest booted fully. The wedged member stayed invisible to the reconciler's cpLive gate, stalling recovery. Unlink the stale socket before launch so the inode that appears is the fresh QEMU's, and retry the QMP connect with a short backoff to cover the window where the socket is bound but not yet listening. Handshake failures are not retried — a listener answered, so the error surfaces immediately. * feat(eks-ui): surface cluster Health.Issues on the EKS console DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in Health.Issues), but the cluster detail page only rendered the status badge, so a degraded control plane still showed as ACTIVE with no visible signal. Add a destructive health badge next to the status and a Health row that lists the issue messages, mirroring the addons tab. Healthy clusters read OK. * chore: rebuild ui * fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber The daemon-side VB created in the ebs.mount handler is stored in MountedVolume.VB for the whole mount, and its background chunk uploader was only stopped at daemon shutdown. That state-only VB ran DrainToBackend every 30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path writer) keeps updating, leaving SeqNums inconsistent with sealed chunk ciphertext and causing transient AEAD auth failures on cloned encrypted volumes. Stop its uploader right after New, matching the other short-lived VB call sites. Also bump viperblock to the atomic chunk-ObjectID allocation fix. * build(deps): pin viperblock to landed AEAD concurrency fix Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that eliminates "cipher: message authentication failed" on encrypted volumes under concurrent read-modify-write + background chunk drain. Removes the local-path replace used while the fix was in flight. |
||
|
Josh Siviour
|
2d4f0fa94c |
feat: IMDS instance-role creds for lb-agent + EKS server VMs (siv-231.6) (#450)
* feat: IMDS-capable gateway signer for lb-agent + EKS clients Add internal/gwsign, a SigV4 signer that draws credentials from either fixed static keys or the AWS SDK default credential chain. In a guest VM the chain falls back to the in-VM IMDS endpoint, yielding scoped, auto-rotating instance-role credentials; a session token is signed as X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds. Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client (internal/eksgw). When static keys are absent the clients now sign with IMDS credentials instead of failing; when present, behaviour is unchanged (static path). This is the client-side foundation for dropping long-lived secrets from VM user-data. Credentials are retrieved per request so rotated IMDS creds take effect without an agent restart. * feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C) Attach a Spinifex-managed system instance profile to the k3s control-plane VM and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating IMDS instance-role credentials instead of a long-lived static secret baked into user-data. - Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a role (EC2 trust) plus its instance profile and asserts an inline permission policy. The gateway enforces per-action authz on assumed-role principals, so the roles carry exactly the actions each VM calls: - EKS server: eks:PublishInternal, eks:ListInternalAddons. - LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig. - Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves the role. - Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/ EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from the lb-agent fw_cfg env; relax k3s server validation to accept profile-only; drop the EKS key requires from the first-boot and state-report scripts. - Both verticals fall back to baked static creds when IAM is unwired, so an absent master key never blocks a launch. * fix: resolve IMDS instance-role creds under the instance account for system VMs Three boot-path defects kept the LB VM's lb-agent from authenticating to the gateway with IMDS-served instance-role creds; each produced a silent empty IMDS role list and a heartbeat failing with "unexpected empty EC2 IMDS role list". Found and fixed against env19. - Build the system IAM ensurer lazily on the first system-VM launch instead of eagerly at daemon startup, where it raced the NATS KV backend and fell back to baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider resolved at launch time. - Create the LB instance profile in the system account the LB VM runs under (utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile under the instance's account, so a customer-account role was invisible to it. - Record the attached instance's owning account on the ENI (InstanceOwnerId, mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every IAM/STS lookup under it. The LB ENI lives in the customer account while the system LB VM and its role live in the system account; keying the whole IMDS chain off the ENI account left the instance invisible and the role lookup missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account. A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role principal with no static keys in user-data. * test(e2e): assert LB VM authenticates via IMDS instance-role creds The LB suite's targets-healthy assertion already depends on the lb-agent authenticating to the gateway, and with static keys dropped from user-data that auth is IMDS-only. Add an explicit check that a recent gateway authentication for the LB used an assumed-role (ASIA) principal, so a regression to the instance-account IMDS resolution fails with a clear message instead of a mysterious datapath timeout. The LB VM is a system-account instance invisible to the customer AWS API, so the assertion observes the gateway journal on the cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and skips when no node is SSH-reachable. |
||
|
Josh Siviour
|
9a6bf692ca |
feat(ecs): EC2 capacity provisioning + auto-EIP service tasks + task/service detail pages (#431)
* feat(ecs): provision EC2 container-instance capacity via IMDS instance-role Adds a ProvisionCapacity gateway action that launches spinifex-ecs-node EC2 instances which register as ECS container instances, fixing RESOURCE:placement when a cluster has no capacity. Agent control-plane now SigV4-signs gateway calls with instance-role credentials read from IMDS (no static keys seeded in user-data). The gateway already verifies STS/temp creds, so the instance role carries an ecs:* policy + ec2 trust; ensureECSInstanceProfile find-or-creates the ecsInstanceRole instance profile attached at launch. Backend resolves the spinifex-ecs-node AMI by spinifex:managed-by=ecs tag, renders keyless user-data, and launches via the customer RunInstances path. Frontend adds a hand-signed JSON-1.1 ProvisionCapacity call, a provision-capacity dialog on the container-instances tab, and the ECS console section nav + system-image gating. * feat: auto-assign EIPs to awsvpc service tasks; add ECS task/service detail pages Wire AssignPublicIp=ENABLED through the service reconciler so awsvpc service tasks automatically allocate and associate an Elastic IP on the RUNNING transition, and release it on STOP. assignTaskPublicIP/releaseTaskPublicIP mirror the target-group registration hooks in recordTaskState and forceStopTask; the public IP and allocation ID are persisted on the task record and surfaced in DescribeTasks attachments as publicIPv4Address. Add AWS-parity ECS task and service detail pages showing private and public endpoints, container state, and networking configuration, reachable by clicking a task or service in the cluster tabs. * test: cover natsEIPManager allocate/associate/release over NATS |