mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-23 04:36:40 +00:00
ADR-0001 package-boundary migration checkpoint (#1190)
* refactor: move telemetry file naming to contract
* docs: record telemetry contract move
* refactor: move aws xml response helpers
* docs: record aws xml helper move
* refactor: move aws tag projections to foundation
* docs: record aws tag projection move
* refactor: move account key to state foundation
* docs: record account key state move
* refactor: move host privilege escalation to runtime
* docs: record host privilege move
* chore: resolve lint findings from package-boundary moves
* refactor: move process supervision helpers to runtime host
* docs: record process supervision move
* refactor: move nbd endpoint helpers to compute runtime
* docs: record nbd endpoint move
* refactor: move image catalogue to operator
* docs: record image catalogue move
* refactor: move resource id generation to aws foundation
* docs: record resource id move
* refactor: move predastore role to runtime roles
* docs: point service manifest at acm domain
* docs: record predastore role move
* refactor: move northstar role to runtime roles
* docs: record northstar role move
* refactor: move awsgw role to runtime roles
* docs: record awsgw role move
* refactor: move error payload helpers to aws errors
* docs: record error payload move
* refactor: move account identifiers to aws identifiers
* docs: record account identifier move
* refactor: move NATS transport to messaging foundation
* docs: record NATS transport move
* refactor: move system instance contract to EC2 domain
* docs: record system instance move
* refactor: move system VPC builder to network domain
* docs: record system VPC move
* refactor: move instance metadata service to EC2 domain
* docs: record guest metadata move and dependency-rule debt
* refactor: move DNS control plane to DNS domain
* docs: record DNS domain move
* refactor: move base zone bootstrap to DNS domain
* docs: record base zone bootstrap move
* refactor: replace SystemAccountID with GlobalAccountID
* docs: record SystemAccountID removal
* refactor: extract CA key-pair loader to foundation pki
* docs: record foundation pki extraction
* refactor: move tenant CA paths to ACM domain
* docs: record tenant CA path move
* refactor: move IPsec certificate handling to network host
* docs: record IPsec certificate move
* refactor: move Ochre serving lifecycle to the Ochre domain
handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.
* docs: record Ochre domain move
* refactor: move Ochre vector store to the Ochre domain
handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.
* docs: record Ochre vector store move
* refactor: move Hugging Face Hub client to the Ochre domain
gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.
* docs: record Hugging Face Hub client move and Ochre gateway debt
* refactor: move EC2 tag mutation helpers to the EC2 domain
utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.
* docs: record EC2 tag mutation move and migrate split finding
* refactor: move network packages under the network domain
Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.
* docs: record network domain move
* refactor: move standalone EC2 resource packages to the EC2 domain
Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.
* docs: record standalone EC2 resource moves
* refactor: move EC2 VPC networking packages to the EC2 domain
Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.
* docs: record EC2 VPC networking moves
* refactor: move EC2 compute and storage packages to the EC2 domain
Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.
* docs: record EC2 compute and storage moves
* refactor: move ECR credential provider to the EKS agent tree
Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.
* docs: record ECR credential provider move
* refactor: move EKS gateway publish helper to the EKS agent tree
Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.
* docs: record EKS gateway publish move
* refactor: move EKS gateway fetch helper to the EKS agent tree
Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.
* docs: record EKS gateway fetch move
* refactor: move EKS webhook cert helper to the EKS agent tree
cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.
* docs: record EKS webhook cert move
* refactor: move EKS konnectivity cert helper to the EKS agent tree
cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.
* docs: record EKS konnectivity cert move
* refactor: move EKS token webhook to the EKS agent tree
cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.
* docs: record EKS token webhook move
* refactor: move ECS agent runtime and credentials packages to the ECS agent tree
* docs: record ECS agent subpackage move
* refactor: move ECS agent implementation to the ECS agent tree
cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.
* docs: record ECS agent move
* refactor: move RDS guest agent to agents/rds/agent
* docs: record RDS guest agent move
* refactor: move installer packages to operator/installer
* docs: record installer move
* refactor: move spx command tree to operator/cli
Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.
* docs: record spx command tree move
* chore: accept known vulnerabilities with an expiry
The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.
* refactor: move EC2 account action adapters to domains/ec2/awsapi
* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi
* refactor: move EC2 key action adapters to domains/ec2/awsapi
* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi
* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi
* docs: record EC2 standalone action adapter moves
* refactor: move EC2 zone action adapters to domains/ec2/awsapi
* refactor: move EC2 vpc action adapters to domains/ec2/awsapi
* refactor: move EC2 eigw action adapters to domains/ec2/awsapi
* refactor: move EC2 eip action adapters to domains/ec2/awsapi
* refactor: move EC2 igw action adapters to domains/ec2/awsapi
* refactor: move EC2 natgw action adapters to domains/ec2/awsapi
* refactor: move EC2 routetable action adapters to domains/ec2/awsapi
* docs: record EC2 VPC networking action adapter moves
* refactor: move EC2 image action adapters to domains/ec2/awsapi
* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi
* refactor: move EC2 volume action adapters to domains/ec2/awsapi
* refactor: move EC2 tags action adapters to domains/ec2/awsapi
* docs: record EC2 storage and tag action adapter moves
* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi
* refactor: move EC2 idempotency adapter to domains/ec2/awsapi
* refactor: move EC2 instance action adapters to domains/ec2/awsapi
* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi
* docs: record EC2 instance action adapter moves
* refactor: move AWS REST-JSON router to ingress/aws/rest
* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit
* docs: record AWS ingress helper moves and tooling conformance
* docs: correct historical lifecycle references
* test: characterize ECR control-plane dispatch before registration seam
* feat: add AWS ingress service registration contract
* docs: record ingress registration phases 0 and 1
* refactor: route registered AWS services through the ingress dispatch registry
The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.
The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.
* docs: record ingress dispatch registry consumption
* refactor: register the ECR control plane from its domain
domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.
Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.
* docs: record ECR control-plane registration
* fix: compose the ECR control plane in the integration harness
The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.
* docs: record ingress pipeline deferral
* refactor: extract AWS error envelope rendering to ingress
ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.
The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.
* docs: record error envelope extraction
* refactor: move AWS service selection into ingress dispatch
Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.
* docs: record ingress service selection move
* test: characterize ACM control-plane dispatch before registration
Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.
* refactor: register ACM control plane from its domain
domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.
* docs: record ACM registration
* docs: record ECS, EKS and RDS source inventory and first-resource ranking
* docs: record EKS control-plane secrets persisted in the cluster record
* chore: drop containerd advisory acceptance now that dev ships v2.4.1
* test: characterize RDS DB subnet group lifecycle before extraction
* docs: record RDS DB subnet group lifecycle contract
* refactor: extract the RDS DB subnet group lifecycle owner
domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.
* docs: record RDS DB subnet group owner extraction
* docs: correct RDS subnet group tagging debt and conformance scope
* test: characterize ECS task definition lifecycle before extraction
* docs: record ECS task definition lifecycle contract
* refactor: extract ECS task definition owner into domains/ecs/taskdefinition
Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.
The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.
List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.
* docs: record ECS task definition owner extraction
* test: characterize EKS access entry lifecycle before extraction
* docs: record EKS access entry lifecycle contract
* refactor: extract EKS access entry owner into domains/eks/access
Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.
Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.
List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.
* docs: record EKS access entry owner extraction
* test: characterize RDS engine catalogue before extraction
* fix: return InvalidParameterValue for a missing RDS master password
ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.
* docs: record RDS engine catalogue boundary and evidence
* refactor: extract RDS engine catalogue into domains/rds/engine
Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.
The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.
The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.
* docs: record RDS engine catalogue owner move
* test: characterize RDS DB parameter group lifecycle before extraction
* docs: record RDS DB parameter group lifecycle contract
* refactor: extract RDS DB parameter group owner
Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.
* docs: record RDS DB parameter group owner extraction
* test: characterize EKS add-on lifecycle and guest wire
Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.
* docs: record EKS add-on lifecycle contract
* fix: require iam:PassRole for EKS add-on service account role
CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.
* docs: record EKS add-on PassRole check
* refactor: extract EKS add-on owner
Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.
* docs: record EKS add-on owner extraction
* refactor: extract EKS add-on guest wire contract
Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.
* docs: record EKS add-on wire contract extraction
* docs: update EKS add-on inventory and remaining edges
* docs: draft EKS add-on lifecycle correction design for review
* chore: run contracts tests in the default test, coverage and race targets
* docs: apply review decisions to EKS add-on lifecycle design
* test: characterize EKS add-on v1 compatibility with added fields
Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.
* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
* docs: draft EKS add-on v2 guest contract design for review
* docs: qualify historical architecture references
* docs: apply review decisions and ADR-0006 to EKS add-on v2 design
* fix(eks): bind internal report publishing to the caller's cluster
PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.
PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.
* fix(eks): bind webhook token review to the caller's cluster
WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.
WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.
* fix(eks): validate add-on serviceAccountRoleArn syntax
CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.
The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.
* docs: record EKS internal-route and role-ARN security corrections
* docs: add EKS add-on release membership and activation case study
* fix: point UI build, CI and lint rules at moved package paths
* docs: record EKS add-on activation review decisions
* refactor: move quota admission to domains/admission/quota
Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.
* docs: reconcile package-boundary record for ADR-0001 close-out
Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.
* test: move KV migration engine tests to foundation/state/migrate
The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.
* refactor: move config and object-store upgrade workflow to operator/upgrade
Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.
* docs: record migrate move to operator/upgrade
* test: enforce ADR-0001 layering with a blocking ratchet
Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.
Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.
* refactor: drop the legacy handlers_ecr import alias
Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.
* chore: mark spf13/pflag as a direct dependency
operator/cli/wiring.go imports pflag directly. Same version; no upgrade.
* docs: record test-package-check and nilaway as unverified gates
* refactor: move the layering test to module-root architecture/
The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.
* fix: update paths left stale by the package moves
The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.
* refactor: move the default-VPC subject into contracts/ec2/v1
utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.
* docs: qualify the migrate split and record admin and Viperblock findings
The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.
* refactor: move operator-only admin code to operator/admin
Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.
foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.
* docs: record operator/admin split in package-boundary migration
* test: pin viperblockd fence, abandon and dirty-marker wire shapes
Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.
* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts
Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.
Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.
* refactor: split the Viperblock EBS adapter out of services/viperblockd
Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.
The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.
Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.
* refactor: move the viperblockd role to runtime/roles/viperblockd
With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.
* docs: record viperblockd contract, adapter and role moves
* test: characterise vpc.* network-realisation routes
Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.
Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).
* feat: add contracts/network/v1 for vpc.* realisation routes
Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.
* refactor: use contracts/network/v1 for vpc.* producers and consumers
Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.
Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.
No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.
* refactor: replace utils/vpcd_event.go with a typed network projection client
Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.
utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.
* refactor: route the remaining fire-and-forget vpc.* producers through projection
CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.
* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils
The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.
utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.
* docs: record the network contract, projection client and utils removal
* test: characterise instance-state persistence, recovery and generation handling
Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.
* test: freeze direct instance-state consumers
Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.
* docs: record ADR-0007 slice 0 instance-state inventory
Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.
* test: narrow the instance-state ratchet to instance-state access
Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.
* docs: record the narrowed instance-state ratchet commit
* docs: design the instance-state transition record and activation gate
* docs: record the instance-state transition-record design commit
* docs: record slice 1a activation-gate decisions
* feat: add instance transition-record codec with format marker
Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.
Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.
No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.
* docs: record slice 1b step 1 codec and awsgw rollout qualification
Records the codec added in e62f921d2 as design Section 10 (C1 to C7): its
surface, preservation at every object level, the changed-array limitation,
marker handling, the rejection classes and its capable-writers-only scope. C2
supersedes P1's top-level-only preservation for the codec. G19 records that
old read-only gateway consumers remain safe but DescribeInstanceStatus gains
unconfirmed/degraded visibility only after awsgw upgrades. The migration record
gains the commit row and states that no production path uses the codec yet.
* refactor: move the instance transition-record codec to domains/ec2/instance/state
The codec defines how the canonical persisted instance record evolves, with
its assignment and observation sections, which is EC2 instance ownership rather
than node-local VM supervision. It moves unchanged into its own package and
keeps wrapping the legacy vm.InstanceRecord alias and its spec and status
types, which stay in runtime/compute/vm for now.
The domain-to-runtime import is recorded as new dependency-rule debt with its
removal condition and allow-listed against that entry. The instance-state
ratchet now names the new file as an owner file; its allowlist is unchanged.
* fix: refuse array changes that would drop element extensions
A changed array is replaced whole by the writer's value, which silently lost
any member a newer writer had added inside its elements. Encode now inspects
the array as read: if any element carries a member its typed schema does not
know, at any depth, it returns ErrRecordExtensionConflict naming the array and
the first such member. Arrays without such members are still replaced whole,
unchanged arrays are still kept verbatim, and arrays inside unknown members are
never inspected.
* docs: record the codec move and the array extension-conflict rule
Records 8db9771fb and c617914a1. T1 and Section 10 now name
domains/ec2/instance/state as the eventual owner of the persisted instance
record and its evolution, with vm.InstanceRecord as the temporary legacy type
it wraps. C3 replaces the changed-array limitation with the refusal: a changed
array with element extensions fails with ErrRecordExtensionConflict and is never
silently dropped. P2 stays a conservative rule until finalisation.
* fix: refuse removals that would drop record extension fields
A capable writer may no longer remove or replace a known object, map entry or
array when its stored value carries members the writer's schema does not know.
The check that guarded changed arrays now runs wherever the merge drops or
overwrites a stored value instead of merging into it: a known member deleted or
set to null, a map entry removed or stored under a key the typed decode
rewrote, a value of another JSON kind, and a changed array. Encode returns
ErrRecordExtensionConflict naming the changed container and the first unknown
member. Removal stays allowed when the stored value has no extensions, and a
change merged member by member still keeps unknown members.
* docs: record the uniform extension-conflict rule
Adds design clause C8 for 0d7b4266d: a capable writer may not remove or
replace a known object, map entry or array whose stored value carries unknown
structured extension fields; removal without extensions stays allowed. C3 and
C5 point at it. The migration record gains the commit row, updated subsection
wording and a pointer-bump follow-up deferring make docs-portal to the Mulga
gitlink PR so the portal evaluates the proposed Spinifex revision.
* refactor: inject vpcd's IMDS instance-state readers from the instance state boundary
* docs: cite the vpcd reader injection commit in the migration record
* refactor: move vpcd to runtime/roles/vpcd
* docs: cite the vpcd move commit in the migration record
* refactor: move the ECS agent wire vocabulary to contracts/ecs/v1
Move handlers/ecs/bus to contracts/ecs/v1 (package ecsv1) with no legacy
shim. Subject strings, JSON fields and types are byte-identical; the old-agent
compatibility test moves unchanged and golden pins cover the remaining
message types. The README route table records that only service-reconcile
has a live producer today; the agent reuses these payload types through the
gateway actions. service-interfaces.yaml lists ecs.bus.* under the daemon.
* refactor: move system-image promotion to domains/ec2/image
Move images_promote.go and its tests from admin to domains/ec2/image:
SystemOwnerAlias, PromoteImageOpts/PromoteImageResult, PromoteSystemImage,
GetAMIMetadata and the private snapshot reader. Storage, ordering, errors and
promotion behaviour are unchanged; the daemon, gateway/spx and operator CLI
callers import ec2image. admin now holds only account IDs and key generators.
* docs: record the ECS contract and image-promotion moves and the remaining-roots matrix
* refactor: move the bootstrap account identifiers to foundation/aws/identifiers
DefaultAccountID and DefaultAccountName join GlobalAccountID as the fixed,
platform-reserved identity of the first human-facing bootstrap account.
Names and values are unchanged and stay fixed. Callers in daemon, gateway,
awsgw, accountteardown and the operator CLI use awsidentifiers; admin now
holds only the IAM credential generators.
* docs: cite the bootstrap account identifier move in the migration record
* refactor: inject the STS cluster JWKS lookup from handlers/eks
handlers/sts declares ClusterJWKSResolver and keeps JWKS decoding and
InvalidIdentityToken mapping; handlers/eks implements it as
OIDCJWKSReader beside the bucket and key layout helpers. awsgw and the
integration harness inject the reader. STS no longer imports any EKS
package, including tests, and the handlers/sts -> handlers/eks layering
exception is removed.
* docs: cite the STS JWKS resolver cut in the migration record
* refactor: move the AWS credential-format generators to foundation/aws/credentials and retire admin
GenerateAWSAccessKey and GenerateAWSSecretKey move unchanged to
foundation/aws/credentials as a stateless format primitive. handlers/iam
and operator/cli import it directly. The admin legacy root is deleted
and joins the retired roots so it cannot return.
* docs: cite the admin retirement in the migration record
* revert: remove the uncalled ADR-0007 codec and design stack from the ADR-0001 checkpoint
The instance transition-record codec, its tests and the slice 1a design
had no production caller. They leave this package-boundary checkpoint
and will be reapplied on a dedicated ADR-0007 branch. Slice 0
characterisation and the vpcd IMDS readers in
domains/ec2/instance/state stay.
* docs: cite the ADR-0007 codec revert in the migration record
* fix: declare the ECS action subjects in service-interfaces.yaml
Declaring ecs.bus.* made ecs a managed prefix for manifest-lint, which then
failed on the 34 ECS action subjects the daemon registers. Declare them as
ecs.*, the same wildcard convention as rds.*.
* chore: run manifest-check and manifest-lint in preflight
CI runs both, but preflight did not, so manifest drift could pass local
checks and still fail Build and Test.
* docs: record resumed preflight enforcement and dev merge ports
* fix: restore dev changes dropped by earlier merges
Reapplies the slices.ContainsFunc rewrite in nodeFitsCustomerInstance and the DeleteDBSubnetGroup doc comment, both lost when their files were refactored.
* docs: record accountteardown as a frozen legacy root pending cleanup capabilities
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
This commit is contained in:
Josh Siviour
@@ -7,7 +7,10 @@ import (
|
||||
|
||||
"github.com/mulgadc/predastore/s3api"
|
||||
"github.com/mulgadc/spinifex/internal/awsmodel"
|
||||
acmawsapi "github.com/mulgadc/spinifex/spinifex/domains/acm/awsapi"
|
||||
awsapi "github.com/mulgadc/spinifex/spinifex/domains/ecr/awsapi"
|
||||
"github.com/mulgadc/spinifex/spinifex/gateway"
|
||||
"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
)
|
||||
@@ -43,10 +46,15 @@ func fail(err error) {
|
||||
}
|
||||
|
||||
func compareAll() ([]awsmodel.OperationCoverage, error) {
|
||||
dispatch := gateway.AWSOperationInventory()
|
||||
// Registered services contribute their declared inventory, which needs no
|
||||
// live dependencies; nothing here serves a request.
|
||||
inventories := gateway.AWSOperationInventory(map[string]dispatch.Inventory{
|
||||
awsapi.ServiceName: awsapi.OperationInventory(),
|
||||
acmawsapi.ServiceName: acmawsapi.OperationInventory(),
|
||||
})
|
||||
coverages := make([]awsmodel.OperationCoverage, 0, len(awsmodel.Services()))
|
||||
for _, service := range awsmodel.Services() {
|
||||
inventory, ok := dispatch[string(service)]
|
||||
inventory, ok := inventories[string(service)]
|
||||
var modelInventory awsmodel.DispatchInventory
|
||||
switch {
|
||||
case ok:
|
||||
|
||||
@@ -23,7 +23,7 @@ import (
|
||||
|
||||
"github.com/insomniacslk/dhcp/dhcpv4"
|
||||
"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
|
||||
"github.com/mulgadc/spinifex/spinifex/domains/network/external/dhcp"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
)
|
||||
|
||||
@@ -1,175 +1,15 @@
|
||||
// ecr-credential-provider is a kubelet exec credential provider that lets a
|
||||
// Spinifex EKS worker node pull images from the internal Spinifex ECR registry.
|
||||
//
|
||||
// kubelet invokes it per-pull, feeding a CredentialProviderRequest on STDIN and
|
||||
// reading a CredentialProviderResponse on STDOUT. The provider fetches the
|
||||
// node IAM-role credentials from IMDS, SigV4-signs GetAuthorizationToken against
|
||||
// the AWS gateway's ECR endpoint, and returns the minted AWS:<jwt> as Basic auth
|
||||
// for the requested registry host.
|
||||
//
|
||||
// Static node config (gateway URL, CA, region) is read from the cloud-init env
|
||||
// file /etc/spinifex-eks/agent.env (KEY=value); real env vars override it.
|
||||
// ecr-credential-provider is the kubelet exec credential provider for Spinifex
|
||||
// EKS worker nodes; the implementation is in agents/eks/credentialprovider.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/ecrauth"
|
||||
"github.com/mulgadc/spinifex/internal/guestenv"
|
||||
"github.com/mulgadc/spinifex/internal/imdscreds"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/agents/eks/credentialprovider"
|
||||
)
|
||||
|
||||
const (
|
||||
credProviderAPIVersion = "credentialprovider.kubelet.k8s.io/v1"
|
||||
defaultCacheDuration = "10h"
|
||||
// expiryMargin trims the token's real TTL so kubelet re-invokes the provider
|
||||
// before the JWT actually expires mid-run.
|
||||
expiryMargin = 30 * time.Minute
|
||||
|
||||
defaultEnvFile = "/etc/spinifex-eks/agent.env"
|
||||
defaultGatewayCA = "/etc/spinifex-eks/gateway-ca.pem"
|
||||
defaultIMDSBase = "http://169.254.169.254/latest"
|
||||
)
|
||||
|
||||
// CredentialProviderRequest is the kubelet exec-provider request on STDIN.
|
||||
type CredentialProviderRequest struct {
|
||||
APIVersion string `json:"apiVersion"`
|
||||
Kind string `json:"kind"`
|
||||
Image string `json:"image"`
|
||||
}
|
||||
|
||||
// AuthConfig is a single registry credential entry.
|
||||
type AuthConfig struct {
|
||||
Username string `json:"username"`
|
||||
Password string `json:"password"`
|
||||
}
|
||||
|
||||
// CredentialProviderResponse is the kubelet exec-provider response on STDOUT.
|
||||
type CredentialProviderResponse struct {
|
||||
APIVersion string `json:"apiVersion"`
|
||||
Kind string `json:"kind"`
|
||||
CacheKeyType string `json:"cacheKeyType"`
|
||||
CacheDuration string `json:"cacheDuration"`
|
||||
Auth map[string]AuthConfig `json:"auth"`
|
||||
}
|
||||
|
||||
// config holds the static node settings the provider needs.
|
||||
type config struct {
|
||||
GatewayURL string
|
||||
GatewayCA string
|
||||
Region string
|
||||
IMDSBase string
|
||||
}
|
||||
|
||||
func main() {
|
||||
cfg := loadConfig(defaultEnvFile)
|
||||
|
||||
httpClient, err := ecrauth.GatewayHTTPClient(cfg.GatewayCA)
|
||||
if err != nil {
|
||||
slog.Error("ecr-credential-provider: build gateway client", "err", err)
|
||||
emitEmpty(os.Stdout)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if err := run(os.Stdin, os.Stdout, cfg, httpClient); err != nil {
|
||||
slog.Error("ecr-credential-provider: run failed", "err", err)
|
||||
// Still emit a valid empty-auth response so kubelet has a parseable reply.
|
||||
emitEmpty(os.Stdout)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// run is the testable core: it reads a CredentialProviderRequest from in,
|
||||
// resolves credentials via IMDS + the gateway, and writes a
|
||||
// CredentialProviderResponse to out. The httpClient must trust the gateway CA.
|
||||
func run(in io.Reader, out io.Writer, cfg config, httpClient *http.Client) error {
|
||||
var req CredentialProviderRequest
|
||||
if err := json.NewDecoder(in).Decode(&req); err != nil {
|
||||
return fmt.Errorf("decode request: %w", err)
|
||||
}
|
||||
if req.Image == "" {
|
||||
return fmt.Errorf("request image is empty")
|
||||
}
|
||||
imageHost := ecrauth.HostFromImage(req.Image)
|
||||
|
||||
creds, err := imdscreds.Fetch(httpClient, cfg.IMDSBase)
|
||||
if err != nil {
|
||||
return fmt.Errorf("fetch IMDS credentials: %w", err)
|
||||
}
|
||||
|
||||
tok, err := ecrauth.GetAuthorizationToken(cfg.Region, cfg.GatewayURL, httpClient,
|
||||
creds.AccessKeyID, creds.SecretAccessKey, creds.SessionToken)
|
||||
if err != nil {
|
||||
return fmt.Errorf("get authorization token: %w", err)
|
||||
}
|
||||
|
||||
auth := AuthConfig{Username: tok.Username, Password: tok.Password}
|
||||
authMap := map[string]AuthConfig{imageHost: auth}
|
||||
if tok.ProxyHost != "" && tok.ProxyHost != imageHost {
|
||||
authMap[tok.ProxyHost] = auth
|
||||
}
|
||||
|
||||
resp := CredentialProviderResponse{
|
||||
APIVersion: credProviderAPIVersion,
|
||||
Kind: "CredentialProviderResponse",
|
||||
CacheKeyType: "Registry",
|
||||
CacheDuration: cacheDurationFrom(tok.ExpiresAt),
|
||||
Auth: authMap,
|
||||
}
|
||||
return json.NewEncoder(out).Encode(&resp)
|
||||
}
|
||||
|
||||
// cacheDurationFrom derives a Go-duration string from the token expiry minus a
|
||||
// safety margin, falling back to the default when expiry is unknown or too near.
|
||||
func cacheDurationFrom(expiresAt time.Time) string {
|
||||
if expiresAt.IsZero() {
|
||||
return defaultCacheDuration
|
||||
}
|
||||
d := time.Until(expiresAt) - expiryMargin
|
||||
if d <= 0 {
|
||||
return defaultCacheDuration
|
||||
}
|
||||
return d.Round(time.Second).String()
|
||||
}
|
||||
|
||||
// loadConfig reads the cloud-init env file then lets real env vars override. The
|
||||
// IMDS base is fixed (not in the env file) but overridable for tests.
|
||||
func loadConfig(envFile string) config {
|
||||
get := guestenv.Load(envFile).Get
|
||||
|
||||
cfg := config{
|
||||
GatewayURL: get("EKS_GATEWAY_URL"),
|
||||
GatewayCA: get("EKS_GATEWAY_CA"),
|
||||
Region: get("EKS_REGION"),
|
||||
IMDSBase: get("EKS_IMDS_BASE"),
|
||||
}
|
||||
if cfg.GatewayCA == "" {
|
||||
cfg.GatewayCA = defaultGatewayCA
|
||||
}
|
||||
if cfg.IMDSBase == "" {
|
||||
cfg.IMDSBase = defaultIMDSBase
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
// emitEmpty writes a valid CredentialProviderResponse with no auth entries.
|
||||
func emitEmpty(out io.Writer) {
|
||||
resp := CredentialProviderResponse{
|
||||
APIVersion: credProviderAPIVersion,
|
||||
Kind: "CredentialProviderResponse",
|
||||
CacheKeyType: "Registry",
|
||||
CacheDuration: defaultCacheDuration,
|
||||
Auth: map[string]AuthConfig{},
|
||||
}
|
||||
if err := json.NewEncoder(out).Encode(&resp); err != nil {
|
||||
slog.Error("ecr-credential-provider: encode empty response", "err", err)
|
||||
}
|
||||
os.Exit(credentialprovider.Main(os.Stdin, os.Stdout))
|
||||
}
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeBackend serves both the IMDS role/credentials endpoints and the gateway
|
||||
// ECR GetAuthorizationToken on one TLS server, routed by request path.
|
||||
func fakeBackend(t *testing.T) *httptest.Server {
|
||||
t.Helper()
|
||||
mux := http.NewServeMux()
|
||||
|
||||
// IMDSv2 token. Real IMDS echoes the requested TTL back as a response
|
||||
// header; the SDK client parses it from there, not the body.
|
||||
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPut {
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
|
||||
_, _ = w.Write([]byte("imds-v2-token"))
|
||||
})
|
||||
// IMDS role name.
|
||||
mux.HandleFunc("/latest/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/latest/meta-data/iam/security-credentials/" {
|
||||
if r.Header.Get("X-Aws-Ec2-Metadata-Token") != "imds-v2-token" {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte("node-role"))
|
||||
return
|
||||
}
|
||||
// Role credentials.
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{
|
||||
"Code": "Success",
|
||||
"Type": "AWS-HMAC",
|
||||
"AccessKeyId": "AKIATEST",
|
||||
"SecretAccessKey": "secrettest",
|
||||
"Token": "sessiontoken",
|
||||
"AccountId": "123456789012",
|
||||
})
|
||||
})
|
||||
// Gateway ECR GetAuthorizationToken (X-Amz-Target dispatch).
|
||||
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !strings.Contains(r.Header.Get("X-Amz-Target"), "GetAuthorizationToken") {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
authToken := base64.StdEncoding.EncodeToString([]byte("AWS:testjwt"))
|
||||
w.Header().Set("Content-Type", "application/x-amz-json-1.1")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"authorizationData": []map[string]any{{
|
||||
"authorizationToken": authToken,
|
||||
"proxyEndpoint": "https://123456789012.dkr.ecr.us-east-1.spinifex.internal",
|
||||
}},
|
||||
})
|
||||
})
|
||||
|
||||
srv := httptest.NewTLSServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func clientTrusting(srv *httptest.Server) *http.Client {
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(srv.Certificate())
|
||||
return &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
|
||||
}
|
||||
|
||||
func TestRun_ReturnsBasicAuth(t *testing.T) {
|
||||
srv := fakeBackend(t)
|
||||
client := clientTrusting(srv)
|
||||
|
||||
cfg := config{
|
||||
GatewayURL: srv.URL,
|
||||
Region: "us-east-1",
|
||||
IMDSBase: srv.URL + "/latest",
|
||||
}
|
||||
|
||||
const image = "123456789012.dkr.ecr.us-east-1.spinifex.internal/spinifex-demo:latest"
|
||||
reqBody, err := json.Marshal(CredentialProviderRequest{
|
||||
APIVersion: credProviderAPIVersion,
|
||||
Kind: "CredentialProviderRequest",
|
||||
Image: image,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal request: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
if err := run(bytes.NewReader(reqBody), &out, cfg, client); err != nil {
|
||||
t.Fatalf("run: %v", err)
|
||||
}
|
||||
|
||||
var resp CredentialProviderResponse
|
||||
if err := json.Unmarshal(out.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("unmarshal response: %v\n%s", err, out.String())
|
||||
}
|
||||
|
||||
if resp.Kind != "CredentialProviderResponse" {
|
||||
t.Errorf("Kind = %q, want CredentialProviderResponse", resp.Kind)
|
||||
}
|
||||
if resp.CacheKeyType != "Registry" {
|
||||
t.Errorf("CacheKeyType = %q, want Registry", resp.CacheKeyType)
|
||||
}
|
||||
|
||||
host := "123456789012.dkr.ecr.us-east-1.spinifex.internal"
|
||||
auth, ok := resp.Auth[host]
|
||||
if !ok {
|
||||
t.Fatalf("auth map missing host %q; got %v", host, resp.Auth)
|
||||
}
|
||||
if auth.Username != "AWS" {
|
||||
t.Errorf("username = %q, want AWS", auth.Username)
|
||||
}
|
||||
if auth.Password != "testjwt" {
|
||||
t.Errorf("password = %q, want testjwt", auth.Password)
|
||||
}
|
||||
}
|
||||
@@ -1,214 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
goruntime "runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
// version is the agent build version, reported in the register message.
|
||||
// Overridable via -ldflags "-X main.version=...".
|
||||
var version = "dev"
|
||||
|
||||
// Agent wires the ecs-agent's runtime seams: a gateway control-plane client, a
|
||||
// container runtime, and an ECR resolver. It registers the host as a container
|
||||
// instance at boot, heartbeats (re-registers) while alive, polls the gateway for
|
||||
// task assignments, and runs them through containerd, reporting state back over
|
||||
// the gateway. It never connects to NATS — the bus stays host-internal.
|
||||
type Agent struct {
|
||||
cfg config
|
||||
id identity
|
||||
cp controlPlane
|
||||
resolver ctrruntime.Resolver
|
||||
|
||||
// rt holds the container runtime, which is resolved by a loop rather than
|
||||
// once at boot. Empty until the containerd socket answers, and emptied again
|
||||
// if it stops answering.
|
||||
rt *runtimeGate
|
||||
|
||||
reg *registrar
|
||||
hb *heartbeater
|
||||
|
||||
// netns builds awsvpc task network namespaces (nil-safe; bridge/host skip it).
|
||||
netns *taskNetns
|
||||
|
||||
// cred serves task IAM role credentials at 169.254.170.2 (nil in unit tests).
|
||||
cred *credEndpoint
|
||||
|
||||
// gpu is the local pinning ledger over the host's discovered GPU UUIDs.
|
||||
// Always non-nil (newAgent seeds an empty one); New() reseeds it from
|
||||
// nvidia-smi discovery.
|
||||
gpu *gpuLedger
|
||||
|
||||
// dialRuntime connects to the container runtime. nil in unit tests, which
|
||||
// drive the gate directly.
|
||||
dialRuntime func() (ctrruntime.Runtime, error)
|
||||
|
||||
closers []func() error
|
||||
}
|
||||
|
||||
// newAgent assembles an Agent from already-built seams. Tests use this directly
|
||||
// with fakes; New builds the production seams and delegates here. rt may be nil
|
||||
// when containerd is unavailable; the assign path then reports the task STOPPED
|
||||
// rather than crashing, and the watch loop keeps dialling for it.
|
||||
func newAgent(cfg config, id identity, cp controlPlane, rt ctrruntime.Runtime, resolver ctrruntime.Resolver) *Agent {
|
||||
gate := newRuntimeGate(rt)
|
||||
a := &Agent{
|
||||
cfg: cfg,
|
||||
id: id,
|
||||
cp: cp,
|
||||
resolver: resolver,
|
||||
rt: gate,
|
||||
netns: newTaskNetns(execNetRunner{}),
|
||||
gpu: newGPULedger(id.Capacity.GPUIDs),
|
||||
}
|
||||
// Registration reports the host's capacity only while a runtime is there to
|
||||
// use it, so an agent that cannot run a container advertises none and takes
|
||||
// no placement.
|
||||
a.reg = newRegistrar(cp, id, gate.available)
|
||||
a.hb = newHeartbeater(cp, id, cfg.Heartbeat, gate.available)
|
||||
return a
|
||||
}
|
||||
|
||||
// New builds an Agent from config: it resolves the host identity from IMDS,
|
||||
// builds the gateway control-plane client, the ECR resolver and (best-effort)
|
||||
// the containerd runtime. A containerd connect failure is logged, not fatal —
|
||||
// registration and heartbeat still run so the instance is visible while the
|
||||
// runtime recovers. The ECR gateway client is built lazily on first image pull
|
||||
// (not here), so a missing or malformed gateway CA does not stop registration.
|
||||
func New(cfg config) (*Agent, error) {
|
||||
imdsClient := &http.Client{Timeout: 5 * time.Second}
|
||||
|
||||
meta, err := fetchInstanceMetadata(imdsClient, cfg.IMDSBase)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("instance metadata: %w", err)
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
id := identity{
|
||||
AccountID: meta.AccountID,
|
||||
ClusterName: cfg.ClusterName,
|
||||
InstanceID: meta.InstanceID,
|
||||
AZ: meta.AZ,
|
||||
InstanceType: meta.InstanceType,
|
||||
Hostname: host,
|
||||
Capacity: detectCapacity(discoverNvidiaGPUs(execCommandRunner)),
|
||||
AgentVersion: version,
|
||||
}
|
||||
|
||||
creds := credentials.NewIMDSProvider(imdsClient, cfg.IMDSBase)
|
||||
resolver := newLazyECRResolver(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA)
|
||||
|
||||
// A dial that fails here is not fatal and not final: the node starts the
|
||||
// container daemon and the agent back to back, so the socket is often not
|
||||
// accepting yet. Run's watch loop keeps dialling.
|
||||
rt, perr := ctrruntime.New(cfg.ContainerdSocket)
|
||||
if perr != nil {
|
||||
slog.Warn("ecs-agent: containerd unavailable at boot, will keep retrying", "err", perr)
|
||||
}
|
||||
|
||||
cp, err := newGatewayControlPlane(cfg, creds)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build gateway control-plane: %w", err)
|
||||
}
|
||||
|
||||
a := newAgent(cfg, id, cp, rt, resolver)
|
||||
a.cred = newCredEndpoint(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA,
|
||||
cfg.CredEndpointIP, cfg.CredEndpointPort, execNetRunner{})
|
||||
a.dialRuntime = func() (ctrruntime.Runtime, error) { return ctrruntime.New(cfg.ContainerdSocket) }
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// Run registers the instance, starts the heartbeat and assignment-poll loops,
|
||||
// and blocks until ctx is cancelled, then tears down.
|
||||
func (a *Agent) Run(ctx context.Context) error {
|
||||
if err := a.reg.Register(); err != nil {
|
||||
slog.Error("ecs-agent: initial registration failed", "err", err)
|
||||
} else {
|
||||
slog.Info("ecs-agent: registered", "cluster", a.id.ClusterName, "instance", a.id.InstanceID)
|
||||
}
|
||||
|
||||
if a.cred != nil {
|
||||
if err := a.cred.Start(); err != nil {
|
||||
slog.Error("ecs-agent: credential endpoint start failed", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
go a.hb.Run(ctx)
|
||||
|
||||
// Keeps the runtime current for the life of the process: dials while there
|
||||
// is none, and drops one that stops answering.
|
||||
if a.dialRuntime != nil {
|
||||
go a.rt.watch(ctx, a.dialRuntime, a.cfg.RuntimeRetry)
|
||||
}
|
||||
|
||||
// Re-adopt containers still running from before this restart, then poll with
|
||||
// their tasks pre-seeded so re-delivered assignments are acked, not re-run.
|
||||
go a.pollAssignments(ctx, a.reconcile(ctx))
|
||||
|
||||
<-ctx.Done()
|
||||
return a.Stop()
|
||||
}
|
||||
|
||||
// Stop closes the runtime. Safe to call once.
|
||||
func (a *Agent) Stop() error {
|
||||
var firstErr error
|
||||
if a.cred != nil {
|
||||
_ = a.cred.Stop()
|
||||
}
|
||||
for _, c := range a.closers {
|
||||
if err := c(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
a.rt.clear()
|
||||
return firstErr
|
||||
}
|
||||
|
||||
// detectCapacity reports the host's total CPU units (1 vCPU = 1024), memory,
|
||||
// and GPU count/UUIDs from the caller's nvidia-smi discovery (nil on a
|
||||
// non-GPU host).
|
||||
func detectCapacity(gpuUUIDs []string) bus.InstanceCapacity {
|
||||
return bus.InstanceCapacity{
|
||||
CPU: goruntime.NumCPU() * 1024,
|
||||
MemoryMiB: memTotalMiB(),
|
||||
GPU: len(gpuUUIDs),
|
||||
GPUIDs: gpuUUIDs,
|
||||
}
|
||||
}
|
||||
|
||||
// memTotalMiB reads MemTotal from /proc/meminfo, returning 0 if unavailable.
|
||||
func memTotalMiB() int {
|
||||
f, err := os.Open("/proc/meminfo")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
defer f.Close()
|
||||
scanner := bufio.NewScanner(f)
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if !strings.HasPrefix(line, "MemTotal:") {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 2 {
|
||||
return 0
|
||||
}
|
||||
kb, err := strconv.Atoi(fields[1])
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return kb / 1024
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -1,66 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
)
|
||||
|
||||
func TestAgent_RunRegistersThenStopsOnContext(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
cfg := config{Heartbeat: 5 * time.Millisecond, PollInterval: 5 * time.Millisecond}
|
||||
puller := &ctrruntime.FakePuller{}
|
||||
a := newAgent(cfg, testIdentity(), cp, puller, nil)
|
||||
a.closers = append(a.closers, puller.Close)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- a.Run(ctx) }()
|
||||
|
||||
// Give it time to register + re-register (heartbeat) a few times.
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
cancel()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Fatalf("Run returned error: %v", err)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("Run did not return after cancel")
|
||||
}
|
||||
|
||||
// One boot register + at least one heartbeat re-register.
|
||||
if cp.registerCount() < 2 {
|
||||
t.Errorf("registers = %d, want >= 2 (boot + heartbeat)", cp.registerCount())
|
||||
}
|
||||
if !puller.Closed {
|
||||
t.Error("Stop did not close the puller")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectCapacity_Positive(t *testing.T) {
|
||||
c := detectCapacity(nil)
|
||||
if c.CPU <= 0 {
|
||||
t.Errorf("CPU = %d, want > 0", c.CPU)
|
||||
}
|
||||
// MemoryMiB may be 0 on platforms without /proc/meminfo; just assert non-negative.
|
||||
if c.MemoryMiB < 0 {
|
||||
t.Errorf("MemoryMiB = %d, want >= 0", c.MemoryMiB)
|
||||
}
|
||||
if c.GPU != 0 || c.GPUIDs != nil {
|
||||
t.Errorf("no GPU UUIDs given: want GPU=0/GPUIDs=nil, got %d/%v", c.GPU, c.GPUIDs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectCapacity_WithGPUUUIDs(t *testing.T) {
|
||||
c := detectCapacity([]string{"GPU-aaa", "GPU-bbb"})
|
||||
if c.GPU != 2 {
|
||||
t.Errorf("GPU = %d, want 2", c.GPU)
|
||||
}
|
||||
if len(c.GPUIDs) != 2 {
|
||||
t.Errorf("GPUIDs = %v, want 2 entries", c.GPUIDs)
|
||||
}
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/guestenv"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultEnvFile = "/etc/spinifex-ecs/agent.env"
|
||||
defaultGatewayCA = "/etc/spinifex-ecs/gateway-ca.pem"
|
||||
defaultIMDSBase = "http://169.254.169.254/latest"
|
||||
defaultContainerdSocket = "/run/containerd/containerd.sock"
|
||||
defaultHeartbeat = 30 * time.Second
|
||||
defaultPollInterval = 5 * time.Second
|
||||
)
|
||||
|
||||
// config holds the static settings the agent reads at boot. Cluster identity
|
||||
// (account ID, instance ID) is discovered at runtime from IMDS, not configured;
|
||||
// only the cluster *name* the instance was launched into is static. AccessKey /
|
||||
// SecretKey are legacy seeded creds, optional now that the agent SigV4-signs
|
||||
// gateway calls with instance-role credentials from IMDS; the agent never holds
|
||||
// a NATS token.
|
||||
type config struct {
|
||||
GatewayURL string
|
||||
GatewayCA string
|
||||
Region string
|
||||
ClusterName string
|
||||
AccessKey string
|
||||
SecretKey string
|
||||
IMDSBase string
|
||||
ContainerdSocket string
|
||||
Heartbeat time.Duration
|
||||
PollInterval time.Duration
|
||||
RuntimeRetry time.Duration
|
||||
CredEndpointIP string
|
||||
CredEndpointPort int
|
||||
}
|
||||
|
||||
// loadConfig reads the cloud-init env file then lets real env vars override.
|
||||
func loadConfig(envFile string) config {
|
||||
get := guestenv.Load(envFile).Get
|
||||
|
||||
cfg := config{
|
||||
GatewayURL: get("ECS_GATEWAY_URL"),
|
||||
GatewayCA: get("ECS_GATEWAY_CA"),
|
||||
Region: get("ECS_REGION"),
|
||||
ClusterName: get("ECS_CLUSTER"),
|
||||
AccessKey: get("ECS_ACCESS_KEY"),
|
||||
SecretKey: get("ECS_SECRET_KEY"),
|
||||
IMDSBase: get("ECS_IMDS_BASE"),
|
||||
ContainerdSocket: get("ECS_CONTAINERD_SOCKET"),
|
||||
}
|
||||
if cfg.GatewayCA == "" {
|
||||
cfg.GatewayCA = defaultGatewayCA
|
||||
}
|
||||
if cfg.IMDSBase == "" {
|
||||
cfg.IMDSBase = defaultIMDSBase
|
||||
}
|
||||
if cfg.ContainerdSocket == "" {
|
||||
cfg.ContainerdSocket = defaultContainerdSocket
|
||||
}
|
||||
if cfg.ClusterName == "" {
|
||||
cfg.ClusterName = "default"
|
||||
}
|
||||
cfg.Heartbeat = defaultHeartbeat
|
||||
if v := get("ECS_HEARTBEAT_INTERVAL"); v != "" {
|
||||
if d, err := time.ParseDuration(v); err == nil && d > 0 {
|
||||
cfg.Heartbeat = d
|
||||
}
|
||||
}
|
||||
cfg.PollInterval = defaultPollInterval
|
||||
if v := get("ECS_POLL_INTERVAL"); v != "" {
|
||||
if d, err := time.ParseDuration(v); err == nil && d > 0 {
|
||||
cfg.PollInterval = d
|
||||
}
|
||||
}
|
||||
cfg.RuntimeRetry = defaultRuntimeRetry
|
||||
if v := get("ECS_RUNTIME_RETRY"); v != "" {
|
||||
if d, err := time.ParseDuration(v); err == nil && d > 0 {
|
||||
cfg.RuntimeRetry = d
|
||||
}
|
||||
}
|
||||
cfg.CredEndpointIP = get("ECS_CRED_ENDPOINT_IP")
|
||||
if cfg.CredEndpointIP == "" {
|
||||
cfg.CredEndpointIP = defaultCredEndpointIP
|
||||
}
|
||||
cfg.CredEndpointPort = defaultCredEndpointPort
|
||||
if v := get("ECS_CRED_ENDPOINT_PORT"); v != "" {
|
||||
if p, err := strconv.Atoi(v); err == nil && p > 0 {
|
||||
cfg.CredEndpointPort = p
|
||||
}
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
@@ -1,73 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestLoadConfig_Defaults(t *testing.T) {
|
||||
// Ensure env vars don't leak in from the host.
|
||||
for _, k := range []string{"ECS_GATEWAY_URL", "ECS_GATEWAY_CA", "ECS_REGION",
|
||||
"ECS_CLUSTER", "ECS_ACCESS_KEY", "ECS_SECRET_KEY", "ECS_IMDS_BASE",
|
||||
"ECS_CONTAINERD_SOCKET", "ECS_HEARTBEAT_INTERVAL", "ECS_POLL_INTERVAL"} {
|
||||
t.Setenv(k, "")
|
||||
}
|
||||
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
|
||||
if cfg.GatewayCA != defaultGatewayCA {
|
||||
t.Errorf("GatewayCA = %q, want default", cfg.GatewayCA)
|
||||
}
|
||||
if cfg.IMDSBase != defaultIMDSBase {
|
||||
t.Errorf("IMDSBase = %q, want default", cfg.IMDSBase)
|
||||
}
|
||||
if cfg.ContainerdSocket != defaultContainerdSocket {
|
||||
t.Errorf("ContainerdSocket = %q, want default", cfg.ContainerdSocket)
|
||||
}
|
||||
if cfg.ClusterName != "default" {
|
||||
t.Errorf("ClusterName = %q, want default", cfg.ClusterName)
|
||||
}
|
||||
if cfg.Heartbeat != defaultHeartbeat {
|
||||
t.Errorf("Heartbeat = %v, want %v", cfg.Heartbeat, defaultHeartbeat)
|
||||
}
|
||||
if cfg.PollInterval != defaultPollInterval {
|
||||
t.Errorf("PollInterval = %v, want %v", cfg.PollInterval, defaultPollInterval)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfig_FileThenEnvOverride(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
envFile := filepath.Join(dir, "agent.env")
|
||||
body := "# comment\nECS_GATEWAY_URL=https://gw.file\nECS_REGION=us-west-2\nECS_CLUSTER=prod\n"
|
||||
if err := os.WriteFile(envFile, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("ECS_GATEWAY_URL", "https://gw.env")
|
||||
t.Setenv("ECS_ACCESS_KEY", "AKIAENV")
|
||||
|
||||
cfg := loadConfig(envFile)
|
||||
if cfg.GatewayURL != "https://gw.env" {
|
||||
t.Errorf("GatewayURL = %q, want env override", cfg.GatewayURL)
|
||||
}
|
||||
if cfg.Region != "us-west-2" {
|
||||
t.Errorf("Region = %q, want from file", cfg.Region)
|
||||
}
|
||||
if cfg.ClusterName != "prod" {
|
||||
t.Errorf("ClusterName = %q, want from file", cfg.ClusterName)
|
||||
}
|
||||
if cfg.AccessKey != "AKIAENV" {
|
||||
t.Errorf("AccessKey = %q, want env", cfg.AccessKey)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfig_HeartbeatOverride(t *testing.T) {
|
||||
t.Setenv("ECS_HEARTBEAT_INTERVAL", "5s")
|
||||
if cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env")); cfg.Heartbeat != 5*time.Second {
|
||||
t.Errorf("Heartbeat = %v, want 5s", cfg.Heartbeat)
|
||||
}
|
||||
// Garbage value falls back to the default, not zero.
|
||||
t.Setenv("ECS_HEARTBEAT_INTERVAL", "not-a-duration")
|
||||
if cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env")); cfg.Heartbeat != defaultHeartbeat {
|
||||
t.Errorf("Heartbeat = %v, want default on bad value", cfg.Heartbeat)
|
||||
}
|
||||
}
|
||||
@@ -1,159 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ecs"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
|
||||
"github.com/mulgadc/spinifex/internal/ecsgw"
|
||||
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
// controlPlane is the agent's channel to the AWS gateway. It replaces the direct
|
||||
// NATS publisher: every method is a SigV4-signed gateway request, so the NATS bus
|
||||
// stays host-internal. Tests inject a fake; production uses gatewayControlPlane.
|
||||
type controlPlane interface {
|
||||
// Register (re-)registers the container instance. Called at boot and on every
|
||||
// heartbeat tick — the gateway's RegisterContainerInstance is idempotent and
|
||||
// refreshes LastSeen, folding liveness into registration (no bus.Heartbeat).
|
||||
Register(id identity) error
|
||||
// SubmitTaskState reports a task's RUNNING/STOPPED transition.
|
||||
SubmitTaskState(st bus.TaskState) error
|
||||
// PollAssignments drains this instance's assignment + stop inboxes, acking the
|
||||
// assign taskIDs and stop taskIDs accepted on the previous poll. Returns the
|
||||
// still-pending assignments and stop directives.
|
||||
PollAssignments(cluster, instance string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error)
|
||||
// ReportTaskGPU reports the local ledger's pinned device UUIDs for a task's
|
||||
// containers, so the control plane can populate DescribeTasks gpuIds.
|
||||
ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error
|
||||
}
|
||||
|
||||
// gatewayControlPlane implements controlPlane over the SigV4 ECS gateway client.
|
||||
type gatewayControlPlane struct {
|
||||
client *ecsgw.Client
|
||||
}
|
||||
|
||||
var _ controlPlane = (*gatewayControlPlane)(nil)
|
||||
|
||||
// pollTimeout bounds a single PollAssignments request; larger than the register
|
||||
// timeout to leave room for a future server-side long-poll.
|
||||
const pollTimeout = 30 * time.Second
|
||||
|
||||
// newGatewayControlPlane builds the SigV4 client signing with the instance-role
|
||||
// credentials from IMDS (fetched per call so rotation is transparent) + pinned CA.
|
||||
func newGatewayControlPlane(cfg config, creds credentials.CredentialsProvider) (*gatewayControlPlane, error) {
|
||||
client, err := ecsgw.New(cfg.GatewayURL, cfg.GatewayCA, credentialsFunc(creds), cfg.Region, pollTimeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &gatewayControlPlane{client: client}, nil
|
||||
}
|
||||
|
||||
// credentialsFunc adapts the agent's IMDS CredentialsProvider to the ecsgw
|
||||
// per-call credentials hook.
|
||||
func credentialsFunc(p credentials.CredentialsProvider) ecsgw.CredentialsFunc {
|
||||
return func(ctx context.Context) (string, string, string, error) {
|
||||
c, err := p.Retrieve(ctx)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
return c.AccessKeyID, c.SecretAccessKey, c.SessionToken, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) Register(id identity) error {
|
||||
in := &ecs.RegisterContainerInstanceInput{
|
||||
Cluster: aws.String(id.ClusterName),
|
||||
InstanceIdentityDocument: aws.String(id.InstanceID),
|
||||
TotalResources: []*ecs.Resource{
|
||||
{Name: aws.String("CPU"), Type: aws.String("INTEGER"), IntegerValue: aws.Int64(int64(id.Capacity.CPU))},
|
||||
{Name: aws.String("MEMORY"), Type: aws.String("INTEGER"), IntegerValue: aws.Int64(int64(id.Capacity.MemoryMiB))},
|
||||
},
|
||||
VersionInfo: &ecs.VersionInfo{AgentVersion: aws.String(id.AgentVersion)},
|
||||
// The built-in container-instance attributes AWS's own agent reports.
|
||||
// The scheduler sizes the host's awsvpc ENI capacity from the instance
|
||||
// type, so a registration without one is not ENI-gated.
|
||||
Attributes: instanceAttributes(id),
|
||||
}
|
||||
if len(id.Capacity.GPUIDs) > 0 {
|
||||
in.TotalResources = append(in.TotalResources, &ecs.Resource{
|
||||
Name: aws.String("GPU"), Type: aws.String("STRINGSET"), StringSetValue: aws.StringSlice(id.Capacity.GPUIDs),
|
||||
})
|
||||
}
|
||||
body, err := json.Marshal(in)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal register: %w", err)
|
||||
}
|
||||
if _, err := g.client.Call("RegisterContainerInstance", body); err != nil {
|
||||
return fmt.Errorf("register: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) SubmitTaskState(st bus.TaskState) error {
|
||||
in := &ecs.SubmitTaskStateChangeInput{
|
||||
Cluster: aws.String(st.ClusterName),
|
||||
Task: aws.String(st.TaskID),
|
||||
Status: aws.String(st.LastStatus),
|
||||
Reason: aws.String(st.Reason),
|
||||
}
|
||||
for _, c := range st.Containers {
|
||||
csc := &ecs.ContainerStateChange{
|
||||
ContainerName: aws.String(c.Name),
|
||||
Status: aws.String(c.Status),
|
||||
RuntimeId: aws.String(c.ContainerID),
|
||||
}
|
||||
if c.ExitCode != nil {
|
||||
csc.ExitCode = aws.Int64(int64(*c.ExitCode))
|
||||
}
|
||||
in.Containers = append(in.Containers, csc)
|
||||
}
|
||||
body, err := json.Marshal(in)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal task-state: %w", err)
|
||||
}
|
||||
if _, err := g.client.Call("SubmitTaskStateChange", body); err != nil {
|
||||
return fmt.Errorf("submit task-state: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) PollAssignments(cluster, instance string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error) {
|
||||
in := &handlers_ecs.PollAssignmentsInput{
|
||||
Cluster: cluster,
|
||||
ContainerInstance: instance,
|
||||
AckTaskIDs: ackAssigns,
|
||||
AckStopIDs: ackStops,
|
||||
}
|
||||
body, err := json.Marshal(in)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("marshal poll: %w", err)
|
||||
}
|
||||
resp, err := g.client.Call("PollAssignments", body)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("poll: %w", err)
|
||||
}
|
||||
var out handlers_ecs.PollAssignmentsOutput
|
||||
if err := json.Unmarshal(resp, &out); err != nil {
|
||||
return nil, nil, fmt.Errorf("decode poll: %w", err)
|
||||
}
|
||||
return out.Assignments, out.Stops, nil
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error {
|
||||
in := &handlers_ecs.ReportTaskGPUInput{Cluster: cluster, Task: task, Containers: containers}
|
||||
body, err := json.Marshal(in)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal report-task-gpu: %w", err)
|
||||
}
|
||||
if _, err := g.client.Call("ReportTaskGPU", body); err != nil {
|
||||
return fmt.Errorf("report task gpu: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,136 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/service/ecs"
|
||||
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
|
||||
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
|
||||
)
|
||||
|
||||
// fixedCredsProvider is a static credentials.CredentialsProvider for tests
|
||||
// that only need a valid SigV4 signature, not real IMDS rotation.
|
||||
type fixedCredsProvider struct{}
|
||||
|
||||
func (fixedCredsProvider) Retrieve(context.Context) (credentials.Credentials, error) {
|
||||
return credentials.Credentials{AccessKeyID: "AKIATEST", SecretAccessKey: "secret"}, nil
|
||||
}
|
||||
|
||||
func newTestGatewayControlPlane(t *testing.T, srv *httptest.Server) *gatewayControlPlane {
|
||||
t.Helper()
|
||||
cp, err := newGatewayControlPlane(config{GatewayURL: srv.URL, Region: "us-east-1"}, fixedCredsProvider{})
|
||||
if err != nil {
|
||||
t.Fatalf("newGatewayControlPlane: %v", err)
|
||||
}
|
||||
return cp
|
||||
}
|
||||
|
||||
// Register carries the discovered GPU UUIDs as a STRINGSET resource, matching
|
||||
// the control plane's RegisterContainerInstance GPU parsing (Epic C3).
|
||||
func TestGatewayControlPlane_Register_SendsGPUStringSet(t *testing.T) {
|
||||
var body []byte
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, _ = io.ReadAll(r.Body)
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
cp := newTestGatewayControlPlane(t, srv)
|
||||
id := identity{
|
||||
ClusterName: "default", InstanceID: "i-1",
|
||||
Capacity: detectCapacity([]string{"GPU-aaa", "GPU-bbb"}),
|
||||
}
|
||||
if err := cp.Register(id); err != nil {
|
||||
t.Fatalf("Register: %v", err)
|
||||
}
|
||||
|
||||
var in ecs.RegisterContainerInstanceInput
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
t.Fatalf("decode body: %v", err)
|
||||
}
|
||||
var gpu *ecs.Resource
|
||||
for _, r := range in.TotalResources {
|
||||
if r.Name != nil && *r.Name == "GPU" {
|
||||
gpu = r
|
||||
}
|
||||
}
|
||||
if gpu == nil {
|
||||
t.Fatal("no GPU resource in TotalResources")
|
||||
}
|
||||
if *gpu.Type != "STRINGSET" {
|
||||
t.Errorf("GPU resource type = %q, want STRINGSET", *gpu.Type)
|
||||
}
|
||||
got := make([]string, len(gpu.StringSetValue))
|
||||
for i, v := range gpu.StringSetValue {
|
||||
got[i] = *v
|
||||
}
|
||||
if len(got) != 2 || got[0] != "GPU-aaa" || got[1] != "GPU-bbb" {
|
||||
t.Errorf("GPU UUIDs = %v, want [GPU-aaa GPU-bbb]", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A non-GPU host (no discovered UUIDs) registers with no GPU resource at all.
|
||||
func TestGatewayControlPlane_Register_NoGPUOmitsResource(t *testing.T) {
|
||||
var body []byte
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, _ = io.ReadAll(r.Body)
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
cp := newTestGatewayControlPlane(t, srv)
|
||||
id := identity{ClusterName: "default", InstanceID: "i-1", Capacity: detectCapacity(nil)}
|
||||
if err := cp.Register(id); err != nil {
|
||||
t.Fatalf("Register: %v", err)
|
||||
}
|
||||
|
||||
var in ecs.RegisterContainerInstanceInput
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
t.Fatalf("decode body: %v", err)
|
||||
}
|
||||
for _, r := range in.TotalResources {
|
||||
if r.Name != nil && *r.Name == "GPU" {
|
||||
t.Fatalf("unexpected GPU resource on a non-GPU host: %+v", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ReportTaskGPU posts the per-container device UUIDs under the ReportTaskGPU
|
||||
// action, decodable as the handlers_ecs internal shape.
|
||||
func TestGatewayControlPlane_ReportTaskGPU(t *testing.T) {
|
||||
var gotTarget string
|
||||
var body []byte
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotTarget = r.Header.Get("X-Amz-Target")
|
||||
body, _ = io.ReadAll(r.Body)
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
cp := newTestGatewayControlPlane(t, srv)
|
||||
err := cp.ReportTaskGPU("web", "t-1", []handlers_ecs.ContainerGPUReport{
|
||||
{Name: "trainer", GPUIDs: []string{"GPU-aaa"}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("ReportTaskGPU: %v", err)
|
||||
}
|
||||
if gotTarget == "" {
|
||||
t.Fatal("no X-Amz-Target header sent")
|
||||
}
|
||||
|
||||
var in handlers_ecs.ReportTaskGPUInput
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
t.Fatalf("decode body: %v", err)
|
||||
}
|
||||
if in.Cluster != "web" || in.Task != "t-1" {
|
||||
t.Errorf("cluster/task = %q/%q, want web/t-1", in.Cluster, in.Task)
|
||||
}
|
||||
if len(in.Containers) != 1 || in.Containers[0].Name != "trainer" || len(in.Containers[0].GPUIDs) != 1 {
|
||||
t.Errorf("containers = %+v, want one trainer entry with one UUID", in.Containers)
|
||||
}
|
||||
}
|
||||
@@ -1,401 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
|
||||
"github.com/mulgadc/spinifex/internal/ecrauth"
|
||||
"github.com/mulgadc/spinifex/internal/stsauth"
|
||||
)
|
||||
|
||||
const (
|
||||
// defaultCredEndpointIP is the AWS-reserved link-local address tasks reach for
|
||||
// container credentials; defaultCredEndpointPort is the standard HTTP port.
|
||||
defaultCredEndpointIP = "169.254.170.2"
|
||||
defaultCredEndpointPort = 80
|
||||
// defaultCredProxyPort is the high port the listener actually binds; a nat
|
||||
// DNAT rule rewrites 169.254.170.2:80 to it so nothing holds a socket on :80
|
||||
// and host/bridge tasks sharing the netns can bind :80 themselves.
|
||||
defaultCredProxyPort = 51679
|
||||
// credRefreshMargin refreshes a cached credential set this long before expiry
|
||||
// so a container never starts a request with a token about to lapse.
|
||||
credRefreshMargin = 5 * time.Minute
|
||||
// credDummyIface is the dummy interface the endpoint IP is added to in the host
|
||||
// netns so the agent can bind it.
|
||||
credDummyIface = "ecs-cred"
|
||||
)
|
||||
|
||||
// credResponse is the ECS task-credentials JSON the container's AWS SDK expects
|
||||
// at AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.
|
||||
type credResponse struct {
|
||||
RoleArn string `json:"RoleArn"`
|
||||
AccessKeyID string `json:"AccessKeyId"`
|
||||
SecretAccessKey string `json:"SecretAccessKey"`
|
||||
Token string `json:"Token"`
|
||||
Expiration string `json:"Expiration"`
|
||||
}
|
||||
|
||||
// credEntry caches one task's assumed-role credentials alongside the role it was
|
||||
// assumed from.
|
||||
type credEntry struct {
|
||||
roleARN string
|
||||
creds stsauth.Credentials
|
||||
}
|
||||
|
||||
// credEndpoint is the host-netns HTTP server that hands a task its IAM role
|
||||
// credentials. It maps credID -> taskRoleARN (registered as tasks start) and
|
||||
// mints/caches credentials on demand by assuming the role over the gateway with
|
||||
// the agent's instance credentials.
|
||||
type credEndpoint struct {
|
||||
provider credentials.CredentialsProvider
|
||||
region string
|
||||
gatewayURL string
|
||||
caPath string
|
||||
ip string
|
||||
port int
|
||||
proxyPort int
|
||||
run netCmdRunner
|
||||
|
||||
// assume mints fresh credentials for a role; defaults to AssumeRole over the
|
||||
// gateway and is overridable in tests.
|
||||
assume func(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error)
|
||||
|
||||
mu sync.Mutex
|
||||
roles map[string]string // credID -> taskRoleARN
|
||||
cache map[string]credEntry // credID -> cached creds
|
||||
httpClient *http.Client
|
||||
|
||||
srv *http.Server
|
||||
ln net.Listener
|
||||
}
|
||||
|
||||
// newCredEndpoint builds the endpoint. An empty ip/port falls back to the
|
||||
// AWS-reserved 169.254.170.2:80; tests pass loopback + an ephemeral port. run
|
||||
// adds the link-local address to the dummy interface (real or fake).
|
||||
func newCredEndpoint(provider credentials.CredentialsProvider, region, gatewayURL, caPath, ip string, port int, run netCmdRunner) *credEndpoint {
|
||||
if ip == "" {
|
||||
ip = defaultCredEndpointIP
|
||||
}
|
||||
if port == 0 {
|
||||
port = defaultCredEndpointPort
|
||||
}
|
||||
c := &credEndpoint{
|
||||
provider: provider,
|
||||
region: region,
|
||||
gatewayURL: gatewayURL,
|
||||
caPath: caPath,
|
||||
ip: ip,
|
||||
port: port,
|
||||
proxyPort: defaultCredProxyPort,
|
||||
run: run,
|
||||
roles: map[string]string{},
|
||||
cache: map[string]credEntry{},
|
||||
}
|
||||
c.assume = c.assumeOverGateway
|
||||
return c
|
||||
}
|
||||
|
||||
// Register associates a credID with the task role it serves; Deregister removes
|
||||
// it and drops any cached credentials. Both are no-ops for an empty credID.
|
||||
func (c *credEndpoint) Register(credID, roleARN string) {
|
||||
if credID == "" || roleARN == "" {
|
||||
return
|
||||
}
|
||||
c.mu.Lock()
|
||||
c.roles[credID] = roleARN
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
func (c *credEndpoint) Deregister(credID string) {
|
||||
if credID == "" {
|
||||
return
|
||||
}
|
||||
c.mu.Lock()
|
||||
delete(c.roles, credID)
|
||||
delete(c.cache, credID)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// relativeURI is the value injected into a container's
|
||||
// AWS_CONTAINER_CREDENTIALS_RELATIVE_URI for the given credID.
|
||||
func credRelativeURI(credID string) string { return "/v2/credentials/" + credID }
|
||||
|
||||
// ServeHTTP serves GET /v2/credentials/{credID}. Unknown paths/credIDs return
|
||||
// 404; non-GET returns 405; an AssumeRole failure returns 502.
|
||||
func (c *credEndpoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
credID, ok := strings.CutPrefix(r.URL.Path, "/v2/credentials/")
|
||||
if !ok || credID == "" || strings.Contains(credID, "/") {
|
||||
http.Error(w, "not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
creds, roleARN, err := c.fetch(r.Context(), credID)
|
||||
if err != nil {
|
||||
if errors.Is(err, errUnknownCredID) {
|
||||
http.Error(w, "not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
slog.Error("ecs-agent: assume task role failed", "credId", credID, "err", err)
|
||||
http.Error(w, "credentials unavailable", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if err := json.NewEncoder(w).Encode(credResponse{
|
||||
RoleArn: roleARN,
|
||||
AccessKeyID: creds.AccessKeyID,
|
||||
SecretAccessKey: creds.SecretAccessKey,
|
||||
Token: creds.SessionToken,
|
||||
Expiration: creds.Expiration.UTC().Format(time.RFC3339),
|
||||
}); err != nil {
|
||||
slog.Warn("ecs-agent: encode credential response", "credId", credID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// errUnknownCredID signals a credID with no registered task role.
|
||||
var errUnknownCredID = fmt.Errorf("unknown credId")
|
||||
|
||||
// fetch returns valid credentials for credID, reusing the cache until within the
|
||||
// refresh margin and otherwise assuming the task role over the gateway.
|
||||
func (c *credEndpoint) fetch(ctx context.Context, credID string) (stsauth.Credentials, string, error) {
|
||||
c.mu.Lock()
|
||||
roleARN, known := c.roles[credID]
|
||||
if !known {
|
||||
c.mu.Unlock()
|
||||
return stsauth.Credentials{}, "", errUnknownCredID
|
||||
}
|
||||
if e, ok := c.cache[credID]; ok && e.roleARN == roleARN && credValid(e.creds, credRefreshMargin) {
|
||||
c.mu.Unlock()
|
||||
return e.creds, roleARN, nil
|
||||
}
|
||||
c.mu.Unlock()
|
||||
|
||||
creds, err := c.assume(ctx, roleARN, sessionName(credID))
|
||||
if err != nil {
|
||||
return stsauth.Credentials{}, "", err
|
||||
}
|
||||
|
||||
c.mu.Lock()
|
||||
c.cache[credID] = credEntry{roleARN: roleARN, creds: creds}
|
||||
c.mu.Unlock()
|
||||
return creds, roleARN, nil
|
||||
}
|
||||
|
||||
// AssumeProvider returns a CredentialsProvider that mints credentials by assuming
|
||||
// roleARN over the gateway — the execution-role path for ECR image pulls. It
|
||||
// caches the assumed set and re-assumes within the refresh margin. session scopes
|
||||
// the STS session name.
|
||||
func (c *credEndpoint) AssumeProvider(roleARN, session string) credentials.CredentialsProvider {
|
||||
return &assumeProvider{ep: c, roleARN: roleARN, session: session}
|
||||
}
|
||||
|
||||
// assumeProvider adapts a credEndpoint's AssumeRole path to the agent's
|
||||
// CredentialsProvider interface, caching the assumed credentials.
|
||||
type assumeProvider struct {
|
||||
ep *credEndpoint
|
||||
roleARN string
|
||||
session string
|
||||
|
||||
mu sync.Mutex
|
||||
cached stsauth.Credentials
|
||||
}
|
||||
|
||||
var _ credentials.CredentialsProvider = (*assumeProvider)(nil)
|
||||
|
||||
func (p *assumeProvider) Retrieve(ctx context.Context) (credentials.Credentials, error) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if !credValid(p.cached, credRefreshMargin) {
|
||||
creds, err := p.ep.assume(ctx, p.roleARN, p.session)
|
||||
if err != nil {
|
||||
return credentials.Credentials{}, err
|
||||
}
|
||||
p.cached = creds
|
||||
}
|
||||
return credentials.Credentials{
|
||||
AccessKeyID: p.cached.AccessKeyID,
|
||||
SecretAccessKey: p.cached.SecretAccessKey,
|
||||
SessionToken: p.cached.SessionToken,
|
||||
Expiration: p.cached.Expiration,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// assumeOverGateway mints credentials for roleARN by SigV4-signing AssumeRole
|
||||
// against the gateway with the agent's instance credentials.
|
||||
func (c *credEndpoint) assumeOverGateway(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error) {
|
||||
instCreds, err := c.provider.Retrieve(ctx)
|
||||
if err != nil {
|
||||
return stsauth.Credentials{}, fmt.Errorf("instance credentials: %w", err)
|
||||
}
|
||||
client, err := c.client()
|
||||
if err != nil {
|
||||
return stsauth.Credentials{}, err
|
||||
}
|
||||
return stsauth.AssumeRole(c.region, c.gatewayURL, client,
|
||||
instCreds.AccessKeyID, instCreds.SecretAccessKey, instCreds.SessionToken,
|
||||
roleARN, sessionName)
|
||||
}
|
||||
|
||||
// client builds the pinned-CA gateway HTTP client on first use and caches it.
|
||||
func (c *credEndpoint) client() (*http.Client, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.httpClient != nil {
|
||||
return c.httpClient, nil
|
||||
}
|
||||
hc, err := ecrauth.GatewayHTTPClient(c.caPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.httpClient = hc
|
||||
return hc, nil
|
||||
}
|
||||
|
||||
// Start adds the endpoint IP to a dummy interface in the host netns, serves HTTP
|
||||
// on the high proxy port, and DNATs the advertised :80 to it. A bind, address,
|
||||
// or nat failure is returned so the caller can log it without aborting
|
||||
// register/heartbeat.
|
||||
func (c *credEndpoint) Start() error {
|
||||
if err := c.addEndpointAddr(); err != nil {
|
||||
return err
|
||||
}
|
||||
ln, err := net.Listen("tcp", net.JoinHostPort(c.ip, strconv.Itoa(c.listenPort())))
|
||||
if err != nil {
|
||||
return fmt.Errorf("listen %s:%d: %w", c.ip, c.listenPort(), err)
|
||||
}
|
||||
if err := c.addRedirect(); err != nil {
|
||||
_ = ln.Close()
|
||||
return err
|
||||
}
|
||||
c.ln = ln
|
||||
c.srv = &http.Server{Handler: c, ReadHeaderTimeout: 5 * time.Second}
|
||||
go func() {
|
||||
if err := c.srv.Serve(ln); err != nil && err != http.ErrServerClosed {
|
||||
slog.Error("ecs-agent: credential endpoint serve", "err", err)
|
||||
}
|
||||
}()
|
||||
slog.Info("ecs-agent: credential endpoint listening", "addr", ln.Addr().String())
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop shuts the server down and removes the dummy interface. Safe on a nil/
|
||||
// never-started endpoint.
|
||||
func (c *credEndpoint) Stop() error {
|
||||
if c == nil {
|
||||
return nil
|
||||
}
|
||||
if c.srv != nil {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
_ = c.srv.Shutdown(ctx)
|
||||
}
|
||||
c.delRedirect()
|
||||
c.delEndpointAddr()
|
||||
return nil
|
||||
}
|
||||
|
||||
// listenPort is the port the socket binds. Loopback test mode binds the
|
||||
// caller-supplied (ephemeral) port directly; the real endpoint binds the high
|
||||
// proxy port and reaches :80 via the DNAT rules.
|
||||
func (c *credEndpoint) listenPort() int {
|
||||
if c.ip == "127.0.0.1" {
|
||||
return c.port
|
||||
}
|
||||
return c.proxyPort
|
||||
}
|
||||
|
||||
// addRedirect installs nat DNAT rules rewriting the advertised :80 to the proxy
|
||||
// port, for both locally generated traffic (host/bridge task containers, OUTPUT)
|
||||
// and traffic arriving over the awsvpc credential veth (PREROUTING). The dst IP
|
||||
// is left on the endpoint IP so no route_localnet handling is needed. Skipped in
|
||||
// loopback test mode, which binds the port directly.
|
||||
func (c *credEndpoint) addRedirect() error {
|
||||
if c.run == nil || c.ip == "127.0.0.1" {
|
||||
return nil
|
||||
}
|
||||
for _, chain := range []string{"OUTPUT", "PREROUTING"} {
|
||||
if _, err := c.run.Run("iptables", c.natArgs("-C", chain)...); err == nil {
|
||||
continue // rule already present
|
||||
}
|
||||
if _, err := c.run.Run("iptables", c.natArgs("-A", chain)...); err != nil {
|
||||
return fmt.Errorf("add nat redirect %s: %w", chain, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *credEndpoint) delRedirect() {
|
||||
if c.run == nil || c.ip == "127.0.0.1" {
|
||||
return
|
||||
}
|
||||
for _, chain := range []string{"OUTPUT", "PREROUTING"} {
|
||||
_, _ = c.run.Run("iptables", c.natArgs("-D", chain)...)
|
||||
}
|
||||
}
|
||||
|
||||
// natArgs builds the iptables nat rule for op (-A/-C/-D) on chain: DNAT the
|
||||
// advertised endpoint :80 to the proxy port on the same IP.
|
||||
func (c *credEndpoint) natArgs(op, chain string) []string {
|
||||
return []string{
|
||||
"-t", "nat", op, chain,
|
||||
"-d", c.ip, "-p", "tcp", "--dport", strconv.Itoa(c.port),
|
||||
"-j", "DNAT", "--to-destination", net.JoinHostPort(c.ip, strconv.Itoa(c.proxyPort)),
|
||||
}
|
||||
}
|
||||
|
||||
// addEndpointAddr brings up the dummy interface holding the endpoint IP. Skipped
|
||||
// for loopback (tests bind 127.0.0.1 directly, no interface plumbing needed).
|
||||
func (c *credEndpoint) addEndpointAddr() error {
|
||||
if c.run == nil || c.ip == "127.0.0.1" {
|
||||
return nil
|
||||
}
|
||||
_, _ = c.run.Run("ip", "link", "add", credDummyIface, "type", "dummy")
|
||||
if _, err := c.run.Run("ip", "addr", "add", c.ip+"/32", "dev", credDummyIface); err != nil &&
|
||||
!strings.Contains(err.Error(), "exists") {
|
||||
return fmt.Errorf("add endpoint addr %s: %w", c.ip, err)
|
||||
}
|
||||
if _, err := c.run.Run("ip", "link", "set", credDummyIface, "up"); err != nil {
|
||||
return fmt.Errorf("bring up %s: %w", credDummyIface, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *credEndpoint) delEndpointAddr() {
|
||||
if c.run == nil || c.ip == "127.0.0.1" {
|
||||
return
|
||||
}
|
||||
_, _ = c.run.Run("ip", "link", "del", credDummyIface)
|
||||
}
|
||||
|
||||
// credValid reports whether creds are usable now with margin to spare.
|
||||
func credValid(c stsauth.Credentials, margin time.Duration) bool {
|
||||
if c.AccessKeyID == "" || c.SecretAccessKey == "" {
|
||||
return false
|
||||
}
|
||||
if c.Expiration.IsZero() {
|
||||
return true
|
||||
}
|
||||
return time.Now().Add(margin).Before(c.Expiration)
|
||||
}
|
||||
|
||||
// sessionName derives an AssumeRole RoleSessionName from credID, satisfying the
|
||||
// STS [\w+=,.@-]{2,64} constraint.
|
||||
func sessionName(credID string) string {
|
||||
name := "ecs-" + credID
|
||||
if len(name) > 64 {
|
||||
name = name[:64]
|
||||
}
|
||||
return name
|
||||
}
|
||||
@@ -1,197 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/stsauth"
|
||||
)
|
||||
|
||||
// newTestCredEndpoint builds an endpoint whose assume seam returns canned creds,
|
||||
// counting calls so cache behaviour is observable. run is nil (no iface plumbing).
|
||||
func newTestCredEndpoint(creds stsauth.Credentials) (*credEndpoint, *int32) {
|
||||
c := newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
|
||||
var calls int32
|
||||
c.assume = func(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error) {
|
||||
atomic.AddInt32(&calls, 1)
|
||||
return creds, nil
|
||||
}
|
||||
return c, &calls
|
||||
}
|
||||
|
||||
func TestCredEndpoint_ServesRegisteredCredID(t *testing.T) {
|
||||
exp := time.Now().Add(time.Hour).UTC().Truncate(time.Second)
|
||||
c, _ := newTestCredEndpoint(stsauth.Credentials{
|
||||
AccessKeyID: "AKIA", SecretAccessKey: "secret", SessionToken: "tok", Expiration: exp,
|
||||
})
|
||||
c.Register("cred-1", "arn:aws:iam::111122223333:role/task")
|
||||
|
||||
rr := httptest.NewRecorder()
|
||||
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200; body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var got credResponse
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if got.AccessKeyID != "AKIA" || got.SecretAccessKey != "secret" || got.Token != "tok" {
|
||||
t.Errorf("bad creds in response: %+v", got)
|
||||
}
|
||||
if got.RoleArn != "arn:aws:iam::111122223333:role/task" {
|
||||
t.Errorf("RoleArn = %q", got.RoleArn)
|
||||
}
|
||||
if got.Expiration != exp.Format(time.RFC3339) {
|
||||
t.Errorf("Expiration = %q, want %q", got.Expiration, exp.Format(time.RFC3339))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_UnknownCredIDIs404(t *testing.T) {
|
||||
c, calls := newTestCredEndpoint(stsauth.Credentials{AccessKeyID: "x", SecretAccessKey: "y"})
|
||||
rr := httptest.NewRecorder()
|
||||
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/nope", nil))
|
||||
if rr.Code != http.StatusNotFound {
|
||||
t.Fatalf("status = %d, want 404", rr.Code)
|
||||
}
|
||||
if *calls != 0 {
|
||||
t.Errorf("assume called %d times for unknown credID, want 0", *calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_WrongMethodIs405(t *testing.T) {
|
||||
c, _ := newTestCredEndpoint(stsauth.Credentials{AccessKeyID: "x", SecretAccessKey: "y"})
|
||||
c.Register("cred-1", "arn:role")
|
||||
rr := httptest.NewRecorder()
|
||||
c.ServeHTTP(rr, httptest.NewRequest(http.MethodPost, "/v2/credentials/cred-1", nil))
|
||||
if rr.Code != http.StatusMethodNotAllowed {
|
||||
t.Fatalf("status = %d, want 405", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_CachesUntilExpiry(t *testing.T) {
|
||||
c, calls := newTestCredEndpoint(stsauth.Credentials{
|
||||
AccessKeyID: "AKIA", SecretAccessKey: "secret", Expiration: time.Now().Add(time.Hour),
|
||||
})
|
||||
c.Register("cred-1", "arn:role")
|
||||
for hit := range 3 {
|
||||
rr := httptest.NewRecorder()
|
||||
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("hit %d: status %d", hit, rr.Code)
|
||||
}
|
||||
}
|
||||
if *calls != 1 {
|
||||
t.Errorf("assume called %d times, want 1 (cached)", *calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_RefreshesNearExpiry(t *testing.T) {
|
||||
// Expiry inside the refresh margin -> every hit re-assumes.
|
||||
c, calls := newTestCredEndpoint(stsauth.Credentials{
|
||||
AccessKeyID: "AKIA", SecretAccessKey: "secret", Expiration: time.Now().Add(time.Minute),
|
||||
})
|
||||
c.Register("cred-1", "arn:role")
|
||||
for range 2 {
|
||||
rr := httptest.NewRecorder()
|
||||
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
|
||||
}
|
||||
if *calls != 2 {
|
||||
t.Errorf("assume called %d times, want 2 (expiry within margin)", *calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_DeregisterDropsCredID(t *testing.T) {
|
||||
c, _ := newTestCredEndpoint(stsauth.Credentials{AccessKeyID: "x", SecretAccessKey: "y"})
|
||||
c.Register("cred-1", "arn:role")
|
||||
c.Deregister("cred-1")
|
||||
rr := httptest.NewRecorder()
|
||||
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
|
||||
if rr.Code != http.StatusNotFound {
|
||||
t.Fatalf("status = %d, want 404 after deregister", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_AssumeFailureIs502(t *testing.T) {
|
||||
c, _ := newTestCredEndpoint(stsauth.Credentials{})
|
||||
c.assume = func(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error) {
|
||||
return stsauth.Credentials{}, context.DeadlineExceeded
|
||||
}
|
||||
c.Register("cred-1", "arn:role")
|
||||
rr := httptest.NewRecorder()
|
||||
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
|
||||
if rr.Code != http.StatusBadGateway {
|
||||
t.Fatalf("status = %d, want 502", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_StartServesOverLoopback(t *testing.T) {
|
||||
c, _ := newTestCredEndpoint(stsauth.Credentials{
|
||||
AccessKeyID: "AKIA", SecretAccessKey: "secret", Expiration: time.Now().Add(time.Hour),
|
||||
})
|
||||
c.Register("cred-1", "arn:role")
|
||||
c.port = 0 // bind an ephemeral port instead of privileged 80
|
||||
if err := c.Start(); err != nil {
|
||||
t.Fatalf("start: %v", err)
|
||||
}
|
||||
defer c.Stop()
|
||||
|
||||
url := "http://" + c.ln.Addr().String() + credRelativeURI("cred-1")
|
||||
resp, err := http.Get(url)
|
||||
if err != nil {
|
||||
t.Fatalf("get: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_RedirectInstallsAndRemovesDNAT(t *testing.T) {
|
||||
// failOn the -C existence check so addRedirect falls through to -A.
|
||||
f := &fakeNetRunner{failOn: "-t nat -C"}
|
||||
c := newCredEndpoint(nil, "us-east-1", "https://gw", "", "169.254.170.2", 80, f)
|
||||
if c.listenPort() != defaultCredProxyPort {
|
||||
t.Fatalf("listenPort = %d, want %d (proxy)", c.listenPort(), defaultCredProxyPort)
|
||||
}
|
||||
if err := c.addRedirect(); err != nil {
|
||||
t.Fatalf("addRedirect: %v", err)
|
||||
}
|
||||
target := "-d 169.254.170.2 -p tcp --dport 80 -j DNAT --to-destination 169.254.170.2:51679"
|
||||
for _, chain := range []string{"-A OUTPUT", "-A PREROUTING"} {
|
||||
if !f.sawAny("iptables -t nat " + chain + " " + target) {
|
||||
t.Errorf("missing add rule for %s; calls=%v", chain, f.joined())
|
||||
}
|
||||
}
|
||||
c.delRedirect()
|
||||
for _, chain := range []string{"-D OUTPUT", "-D PREROUTING"} {
|
||||
if !f.sawAny("iptables -t nat " + chain + " " + target) {
|
||||
t.Errorf("missing delete rule for %s; calls=%v", chain, f.joined())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredEndpoint_RedirectSkippedOnLoopback(t *testing.T) {
|
||||
f := &fakeNetRunner{}
|
||||
c := newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, f)
|
||||
if err := c.addRedirect(); err != nil {
|
||||
t.Fatalf("addRedirect: %v", err)
|
||||
}
|
||||
c.delRedirect()
|
||||
if f.sawAny("iptables") {
|
||||
t.Errorf("loopback mode must not touch iptables; calls=%v", f.joined())
|
||||
}
|
||||
if c.listenPort() != c.port {
|
||||
t.Errorf("loopback listenPort = %d, want caller port %d", c.listenPort(), c.port)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredRelativeURI(t *testing.T) {
|
||||
if got := credRelativeURI("abc"); got != "/v2/credentials/abc" {
|
||||
t.Errorf("credRelativeURI = %q", got)
|
||||
}
|
||||
}
|
||||
@@ -1,41 +0,0 @@
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
|
||||
"github.com/aws/aws-sdk-go-v2/aws"
|
||||
"github.com/mulgadc/spinifex/internal/imdscreds"
|
||||
)
|
||||
|
||||
// IMDSProvider fetches instance-role credentials from IMDS via the AWS SDK's
|
||||
// EC2 IMDS client and ec2rolecreds provider, cached with aws.CredentialsCache,
|
||||
// which refetches once the real expiry passes. A failed refresh is an error
|
||||
// here, not an extended expiry. Safe for concurrent use.
|
||||
type IMDSProvider struct {
|
||||
cache *aws.CredentialsCache
|
||||
}
|
||||
|
||||
var _ CredentialsProvider = (*IMDSProvider)(nil)
|
||||
|
||||
// NewIMDSProvider builds a provider against the given IMDS base URL (e.g.
|
||||
// "http://169.254.169.254/latest"). A nil client uses the SDK IMDS client's
|
||||
// default HTTP client.
|
||||
func NewIMDSProvider(client *http.Client, base string) *IMDSProvider {
|
||||
provider := imdscreds.NewProvider(client, base)
|
||||
return &IMDSProvider{cache: aws.NewCredentialsCache(provider)}
|
||||
}
|
||||
|
||||
// Retrieve returns cached credentials when still valid, else fetches fresh ones.
|
||||
func (p *IMDSProvider) Retrieve(ctx context.Context) (Credentials, error) {
|
||||
creds, err := p.cache.Retrieve(ctx)
|
||||
if err != nil {
|
||||
return Credentials{}, err
|
||||
}
|
||||
return Credentials{
|
||||
AccessKeyID: creds.AccessKeyID,
|
||||
SecretAccessKey: creds.SecretAccessKey,
|
||||
SessionToken: creds.SessionToken,
|
||||
Expiration: creds.Expires,
|
||||
}, nil
|
||||
}
|
||||
@@ -1,164 +0,0 @@
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// imdsStub serves the IMDSv2 token + role + credentials endpoints under the
|
||||
// real /latest prefix the SDK IMDS client always requests.
|
||||
func imdsStub(t *testing.T, hits *int32, creds map[string]string) *httptest.Server {
|
||||
t.Helper()
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
|
||||
// Real IMDS echoes the requested TTL back as a response header; the SDK
|
||||
// client parses it from there, not the body.
|
||||
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
|
||||
_, _ = w.Write([]byte("v2-token"))
|
||||
})
|
||||
mux.HandleFunc("/latest/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/latest/meta-data/iam/security-credentials/" {
|
||||
_, _ = w.Write([]byte("node-role"))
|
||||
return
|
||||
}
|
||||
if hits != nil {
|
||||
atomic.AddInt32(hits, 1)
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(creds)
|
||||
})
|
||||
srv := httptest.NewServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func TestRetrieve_FetchesAndCaches(t *testing.T) {
|
||||
var hits int32
|
||||
// Kept under an hour: ec2rolecreds caps Expires to now+1h.
|
||||
exp := time.Now().Add(30 * time.Minute).UTC().Round(time.Second)
|
||||
srv := imdsStub(t, &hits, map[string]string{
|
||||
"Code": "Success",
|
||||
"AccessKeyId": "AKIA",
|
||||
"SecretAccessKey": "secret",
|
||||
"Token": "session",
|
||||
"Expiration": exp.Format(time.RFC3339),
|
||||
})
|
||||
|
||||
p := NewIMDSProvider(srv.Client(), srv.URL+"/latest")
|
||||
got, err := p.Retrieve(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Retrieve: %v", err)
|
||||
}
|
||||
if got.AccessKeyID != "AKIA" || !got.Expiration.Equal(exp) {
|
||||
t.Errorf("creds mismatch: %+v", got)
|
||||
}
|
||||
|
||||
// Second call well within validity must not re-hit IMDS.
|
||||
if _, err := p.Retrieve(context.Background()); err != nil {
|
||||
t.Fatalf("Retrieve 2: %v", err)
|
||||
}
|
||||
if hits != 1 {
|
||||
t.Errorf("IMDS credential hits = %d, want 1 (cached)", hits)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetrieve_RefetchesWhenExpired(t *testing.T) {
|
||||
var hits int32
|
||||
// Already expired: aws.CredentialsCache refetches once the stored
|
||||
// credentials' real expiry has passed, regardless of ExpiryWindow.
|
||||
srv := imdsStub(t, &hits, map[string]string{
|
||||
"Code": "Success",
|
||||
"AccessKeyId": "AKIA",
|
||||
"SecretAccessKey": "secret",
|
||||
"Expiration": time.Now().Add(-time.Minute).UTC().Format(time.RFC3339),
|
||||
})
|
||||
p := NewIMDSProvider(srv.Client(), srv.URL+"/latest")
|
||||
if _, err := p.Retrieve(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := p.Retrieve(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if hits != 2 {
|
||||
t.Errorf("hits = %d, want 2 (expired refetch)", hits)
|
||||
}
|
||||
}
|
||||
|
||||
// imdsStubFailAfterFirst serves creds on the first credential request and 500s
|
||||
// on every one after it, so a test can drive a refresh that fails against a
|
||||
// credential the cache has already stored.
|
||||
func imdsStubFailAfterFirst(t *testing.T, creds map[string]string) *httptest.Server {
|
||||
t.Helper()
|
||||
var hits atomic.Int32
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
|
||||
_, _ = w.Write([]byte("v2-token"))
|
||||
})
|
||||
mux.HandleFunc("/latest/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/latest/meta-data/iam/security-credentials/" {
|
||||
_, _ = w.Write([]byte("node-role"))
|
||||
return
|
||||
}
|
||||
if hits.Add(1) > 1 {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(creds)
|
||||
})
|
||||
srv := httptest.NewServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
// A refresh that fails must surface as an error. Left to itself the SDK's
|
||||
// ec2rolecreds implements HandleFailToRefresh, which hands back the same dead
|
||||
// key with Expires pushed 5-15 minutes out, so the agent would sign requests
|
||||
// with a credential the control plane has already rejected — indefinitely,
|
||||
// since every subsequent refresh fails the same way.
|
||||
func TestRetrieve_FailedRefreshDoesNotExtendExpiry(t *testing.T) {
|
||||
srv := imdsStubFailAfterFirst(t, map[string]string{
|
||||
"Code": "Success",
|
||||
"AccessKeyId": "AKIA",
|
||||
"SecretAccessKey": "secret",
|
||||
"Token": "session",
|
||||
// Already expired, so the next Retrieve must go back to IMDS.
|
||||
"Expiration": time.Now().Add(-time.Minute).UTC().Format(time.RFC3339),
|
||||
})
|
||||
|
||||
p := NewIMDSProvider(srv.Client(), srv.URL+"/latest")
|
||||
first, err := p.Retrieve(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("first Retrieve: %v", err)
|
||||
}
|
||||
|
||||
got, err := p.Retrieve(context.Background())
|
||||
if err == nil {
|
||||
t.Fatalf("refresh failure was reported as success: %+v", got)
|
||||
}
|
||||
if got.Expiration.After(first.Expiration) {
|
||||
t.Errorf("expiry extended on refresh failure: %v -> %v", first.Expiration, got.Expiration)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetrieve_CancelledContext(t *testing.T) {
|
||||
srv := imdsStub(t, nil, map[string]string{"Code": "Success", "AccessKeyId": "A", "SecretAccessKey": "B"})
|
||||
p := NewIMDSProvider(srv.Client(), srv.URL+"/latest")
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if _, err := p.Retrieve(ctx); err == nil {
|
||||
t.Fatal("expected error from cancelled context")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetrieve_CodeNotSuccessErrors(t *testing.T) {
|
||||
srv := imdsStub(t, nil, map[string]string{"Code": "Failure", "AccessKeyId": "A", "SecretAccessKey": "B"})
|
||||
p := NewIMDSProvider(srv.Client(), srv.URL+"/latest")
|
||||
if _, err := p.Retrieve(context.Background()); err == nil {
|
||||
t.Fatal("expected error when Code != Success")
|
||||
}
|
||||
}
|
||||
@@ -1,24 +0,0 @@
|
||||
// Package credentials supplies AWS SigV4 credentials to the ecs-agent. The agent
|
||||
// signs gateway calls (ECR GetAuthorizationToken, instance register/heartbeat)
|
||||
// with the host's instance-role credentials, fetched from IMDS and cached until
|
||||
// near expiry.
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Credentials is a SigV4 credential set with an expiry.
|
||||
type Credentials struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
SessionToken string
|
||||
Expiration time.Time
|
||||
}
|
||||
|
||||
// CredentialsProvider yields current credentials, refreshing as needed. Retrieve
|
||||
// must be safe for concurrent use.
|
||||
type CredentialsProvider interface {
|
||||
Retrieve(ctx context.Context) (Credentials, error)
|
||||
}
|
||||
@@ -1,55 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
func TestTaskCredID(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
as bus.Assign
|
||||
want string
|
||||
}{
|
||||
{"no role", bus.Assign{TaskID: "t1"}, ""},
|
||||
{"role, no credID -> taskID", bus.Assign{TaskID: "t1", TaskRoleARN: "arn:role"}, "t1"},
|
||||
{"role + credID", bus.Assign{TaskID: "t1", CredID: "c9", TaskRoleARN: "arn:role"}, "c9"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := taskCredID(&tc.as); got != tc.want {
|
||||
t.Errorf("%s: taskCredID = %q, want %q", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithCredEnv(t *testing.T) {
|
||||
base := map[string]string{"FOO": "bar"}
|
||||
|
||||
// No credID -> env returned unchanged (same map).
|
||||
if got := withCredEnv(base, ""); got["FOO"] != "bar" || got["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"] != "" {
|
||||
t.Errorf("blank credID mutated env: %v", got)
|
||||
}
|
||||
|
||||
got := withCredEnv(base, "cred-1")
|
||||
if got["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"] != "/v2/credentials/cred-1" {
|
||||
t.Errorf("missing relative-uri: %v", got)
|
||||
}
|
||||
if got["FOO"] != "bar" {
|
||||
t.Errorf("dropped existing env: %v", got)
|
||||
}
|
||||
// Original map must be untouched (copy semantics).
|
||||
if _, ok := base["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]; ok {
|
||||
t.Errorf("source env mutated: %v", base)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionName(t *testing.T) {
|
||||
if got := sessionName("abc"); got != "ecs-abc" {
|
||||
t.Errorf("sessionName = %q", got)
|
||||
}
|
||||
long := sessionName(string(make([]byte, 100)))
|
||||
if len(long) != 64 {
|
||||
t.Errorf("sessionName not truncated to 64: len=%d", len(long))
|
||||
}
|
||||
}
|
||||
@@ -1,109 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
|
||||
"github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
"github.com/mulgadc/spinifex/internal/ecrauth"
|
||||
)
|
||||
|
||||
// ecrResolver implements runtime.Resolver by minting an ECR authorization token
|
||||
// per pull: it fetches the host's IMDS credentials, SigV4-signs
|
||||
// GetAuthorizationToken against the gateway, and returns the AWS:<jwt> as a
|
||||
// user/password pair for containerd.
|
||||
//
|
||||
// The gateway HTTP client is built lazily on first Authorize, not at agent boot:
|
||||
// a missing or malformed gateway CA only matters when an image is actually
|
||||
// pulled, so the agent can still register + heartbeat on a host with absent or
|
||||
// wrong ECR config.
|
||||
type ecrResolver struct {
|
||||
creds credentials.CredentialsProvider
|
||||
region string
|
||||
gatewayURL string
|
||||
caPath string
|
||||
|
||||
mu sync.Mutex
|
||||
httpClient *http.Client
|
||||
}
|
||||
|
||||
var _ runtime.Resolver = (*ecrResolver)(nil)
|
||||
|
||||
// newECRResolver builds a resolver with an injected HTTP client (tests). The
|
||||
// client is used as-is and never rebuilt from caPath.
|
||||
func newECRResolver(creds credentials.CredentialsProvider, region, gatewayURL string, httpClient *http.Client) *ecrResolver {
|
||||
return &ecrResolver{creds: creds, region: region, gatewayURL: gatewayURL, httpClient: httpClient}
|
||||
}
|
||||
|
||||
// newLazyECRResolver builds a resolver that constructs its gateway HTTP client
|
||||
// from caPath on first use (production path).
|
||||
func newLazyECRResolver(creds credentials.CredentialsProvider, region, gatewayURL, caPath string) *ecrResolver {
|
||||
return &ecrResolver{creds: creds, region: region, gatewayURL: gatewayURL, caPath: caPath}
|
||||
}
|
||||
|
||||
// client returns the gateway HTTP client, building it from caPath on first call
|
||||
// and caching the result. An injected client (newECRResolver) is returned as-is.
|
||||
func (e *ecrResolver) client() (*http.Client, error) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
if e.httpClient != nil {
|
||||
return e.httpClient, nil
|
||||
}
|
||||
c, err := ecrauth.GatewayHTTPClient(e.caPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
e.httpClient = c
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// gatewayRegistryHost returns the host[:port] of the co-located ECR registry
|
||||
// from the gateway URL; empty/unparseable yields "" (matches no ref). The
|
||||
// registry shares the gateway listener, so its port is the gateway port.
|
||||
func gatewayRegistryHost(gatewayURL string) string {
|
||||
u, err := url.Parse(gatewayURL)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return u.Host
|
||||
}
|
||||
|
||||
// isECRRef reports whether ref targets an ECR registry: an "<acct>.dkr.ecr."
|
||||
// host, or one equal to gatewayHost (the agent's own co-located registry
|
||||
// advertised as a bare host/IP). Other refs pull anonymously, no token.
|
||||
func isECRRef(ref, gatewayHost string) bool {
|
||||
host, _, _ := strings.Cut(ref, "/")
|
||||
if gatewayHost != "" && host == gatewayHost {
|
||||
return true
|
||||
}
|
||||
if !strings.Contains(host, ".") && !strings.Contains(host, ":") {
|
||||
return false // no registry host -> docker.io default
|
||||
}
|
||||
return strings.Contains(host, ".dkr.ecr.")
|
||||
}
|
||||
|
||||
// Authorize returns the ECR token's user/password and proxy endpoint for ref.
|
||||
// Non-ECR refs resolve to anonymous pull (empty credentials).
|
||||
func (e *ecrResolver) Authorize(ctx context.Context, ref string) (user, pass, endpoint string, err error) {
|
||||
if !isECRRef(ref, gatewayRegistryHost(e.gatewayURL)) {
|
||||
return "", "", "", nil
|
||||
}
|
||||
c, err := e.creds.Retrieve(ctx)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
httpClient, err := e.client()
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
tok, err := ecrauth.GetAuthorizationToken(e.region, e.gatewayURL, httpClient,
|
||||
c.AccessKeyID, c.SecretAccessKey, c.SessionToken)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
return tok.Username, tok.Password, tok.ProxyHost, nil
|
||||
}
|
||||
@@ -1,158 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
|
||||
)
|
||||
|
||||
// stubCreds is a credentials.CredentialsProvider returning fixed creds or error.
|
||||
type stubCreds struct {
|
||||
c credentials.Credentials
|
||||
err error
|
||||
}
|
||||
|
||||
func (s stubCreds) Retrieve(context.Context) (credentials.Credentials, error) {
|
||||
return s.c, s.err
|
||||
}
|
||||
|
||||
func gatewayStub(t *testing.T, token, proxy string, status int) *httptest.Server {
|
||||
t.Helper()
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if !strings.Contains(r.Header.Get("X-Amz-Target"), "GetAuthorizationToken") {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
w.WriteHeader(status)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/x-amz-json-1.1")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"authorizationData": []map[string]any{{
|
||||
"authorizationToken": base64.StdEncoding.EncodeToString([]byte(token)),
|
||||
"proxyEndpoint": proxy,
|
||||
}},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func trusting(srv *httptest.Server) *http.Client {
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(srv.Certificate())
|
||||
return &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
|
||||
}
|
||||
|
||||
func TestECRResolver_Authorize(t *testing.T) {
|
||||
srv := gatewayStub(t, "AWS:jwt", "https://123456789012.dkr.ecr.us-east-1.spinifex.internal", http.StatusOK)
|
||||
creds := stubCreds{c: credentials.Credentials{
|
||||
AccessKeyID: "AKIA", SecretAccessKey: "secret", SessionToken: "sess",
|
||||
Expiration: time.Now().Add(time.Hour),
|
||||
}}
|
||||
r := newECRResolver(creds, "us-east-1", srv.URL, trusting(srv))
|
||||
|
||||
user, pass, endpoint, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1")
|
||||
if err != nil {
|
||||
t.Fatalf("Authorize: %v", err)
|
||||
}
|
||||
if user != "AWS" || pass != "jwt" {
|
||||
t.Errorf("got %q:%q, want AWS:jwt", user, pass)
|
||||
}
|
||||
if endpoint != "123456789012.dkr.ecr.us-east-1.spinifex.internal" {
|
||||
t.Errorf("endpoint = %q", endpoint)
|
||||
}
|
||||
}
|
||||
|
||||
func TestECRResolver_CredsError(t *testing.T) {
|
||||
r := newECRResolver(stubCreds{err: errors.New("no imds")}, "us-east-1", "https://x", http.DefaultClient)
|
||||
if _, _, _, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1"); err == nil {
|
||||
t.Fatal("expected creds error to propagate")
|
||||
}
|
||||
}
|
||||
|
||||
func TestECRResolver_GatewayError(t *testing.T) {
|
||||
srv := gatewayStub(t, "", "", http.StatusInternalServerError)
|
||||
creds := stubCreds{c: credentials.Credentials{AccessKeyID: "A", SecretAccessKey: "B"}}
|
||||
r := newECRResolver(creds, "us-east-1", srv.URL, trusting(srv))
|
||||
if _, _, _, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1"); err == nil {
|
||||
t.Fatal("expected gateway error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestECRResolver_LazyClient_BadCADeferredToAuthorize(t *testing.T) {
|
||||
creds := stubCreds{c: credentials.Credentials{AccessKeyID: "A", SecretAccessKey: "B"}}
|
||||
// Construction with a bogus CA path must not fail — only Authorize does.
|
||||
r := newLazyECRResolver(creds, "us-east-1", "https://gw", filepath.Join(t.TempDir(), "absent.pem"))
|
||||
if _, _, _, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1"); err == nil {
|
||||
t.Fatal("expected lazy gateway-client build to fail on missing CA")
|
||||
}
|
||||
}
|
||||
|
||||
// TestECRResolver_NonECRAnonymous asserts that a public-registry ref pulls
|
||||
// anonymously: Authorize returns empty creds without ever touching IMDS, even
|
||||
// when the credentials provider would error.
|
||||
func TestECRResolver_NonECRAnonymous(t *testing.T) {
|
||||
r := newECRResolver(stubCreds{err: errors.New("imds must not be called")}, "us-east-1", "https://x", http.DefaultClient)
|
||||
for _, ref := range []string{"nginx:alpine", "docker.io/library/nginx:alpine", "ghcr.io/org/app:1"} {
|
||||
user, pass, endpoint, err := r.Authorize(context.Background(), ref)
|
||||
if err != nil || user != "" || pass != "" || endpoint != "" {
|
||||
t.Errorf("ref %q: got (%q,%q,%q,%v), want anonymous", ref, user, pass, endpoint, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsECRRef(t *testing.T) {
|
||||
cases := []struct {
|
||||
ref, gatewayHost string
|
||||
want bool
|
||||
}{
|
||||
{"nginx:alpine", "192.168.1.13:9999", false},
|
||||
{"docker.io/library/nginx", "192.168.1.13:9999", false},
|
||||
{"ghcr.io/org/app:1", "192.168.1.13:9999", false},
|
||||
{"registry:5000/app:1", "192.168.1.13:9999", false},
|
||||
{"123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1", "192.168.1.13:9999", true},
|
||||
{"123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/x", "192.168.1.13:9999", true},
|
||||
// Co-located gateway registry advertised as a bare host/IP.
|
||||
{"192.168.1.13:9999/bunyip:latest", "192.168.1.13:9999", true},
|
||||
{"192.168.1.13:9999/bunyip:latest", "", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := isECRRef(c.ref, c.gatewayHost); got != c.want {
|
||||
t.Errorf("isECRRef(%q, %q) = %v, want %v", c.ref, c.gatewayHost, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestECRResolver_Authorize_BareHostGateway asserts that a ref whose host
|
||||
// equals the agent's own gateway host authenticates via the co-located ECR
|
||||
// registry, even though it lacks a ".dkr.ecr." name.
|
||||
func TestECRResolver_Authorize_BareHostGateway(t *testing.T) {
|
||||
srv := gatewayStub(t, "AWS:jwt", "https://192.168.1.13:9999", http.StatusOK)
|
||||
creds := stubCreds{c: credentials.Credentials{
|
||||
AccessKeyID: "AKIA", SecretAccessKey: "secret", SessionToken: "sess",
|
||||
Expiration: time.Now().Add(time.Hour),
|
||||
}}
|
||||
r := newECRResolver(creds, "us-east-1", srv.URL, trusting(srv))
|
||||
|
||||
host := strings.TrimPrefix(srv.URL, "https://")
|
||||
user, pass, _, err := r.Authorize(context.Background(), host+"/bunyip:latest")
|
||||
if err != nil {
|
||||
t.Fatalf("Authorize: %v", err)
|
||||
}
|
||||
if user != "AWS" || pass != "jwt" {
|
||||
t.Errorf("got %q:%q, want AWS:jwt", user, pass)
|
||||
}
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os/exec"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// commandRunner abstracts exec.Command so discovery is unit-testable without
|
||||
// shelling out. Production uses execCommandRunner; tests inject a fake.
|
||||
type commandRunner func(name string, args ...string) ([]byte, error)
|
||||
|
||||
// execCommandRunner runs a real host command and returns its stdout.
|
||||
func execCommandRunner(name string, args ...string) ([]byte, error) {
|
||||
return exec.Command(name, args...).Output()
|
||||
}
|
||||
|
||||
// discoverNvidiaGPUs enumerates local NVIDIA GPU device UUIDs via nvidia-smi.
|
||||
// A missing binary or any run failure (no GPU host) yields an empty list, not
|
||||
// an error — the agent still registers with zero GPU capacity.
|
||||
func discoverNvidiaGPUs(run commandRunner) []string {
|
||||
out, err := run("nvidia-smi", "--query-gpu=uuid", "--format=csv,noheader")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return parseNvidiaSMIUUIDs(string(out))
|
||||
}
|
||||
|
||||
// parseNvidiaSMIUUIDs splits nvidia-smi's newline-delimited UUID output,
|
||||
// dropping blank lines. Exported as a pure function so fixture strings can be
|
||||
// tested without exec.
|
||||
func parseNvidiaSMIUUIDs(out string) []string {
|
||||
lines := strings.Split(strings.TrimSpace(out), "\n")
|
||||
uuids := make([]string, 0, len(lines))
|
||||
for _, line := range lines {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
uuids = append(uuids, line)
|
||||
}
|
||||
return uuids
|
||||
}
|
||||
@@ -1,71 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseNvidiaSMIUUIDs(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
in string
|
||||
want []string
|
||||
}{
|
||||
{"empty", "", nil},
|
||||
{"whitespace only", " \n\n ", nil},
|
||||
{"single", "GPU-11111111-1111-1111-1111-111111111111\n", []string{"GPU-11111111-1111-1111-1111-111111111111"}},
|
||||
{"multi", "GPU-aaa\nGPU-bbb\nGPU-ccc\n", []string{"GPU-aaa", "GPU-bbb", "GPU-ccc"}},
|
||||
{"blank lines between", "GPU-aaa\n\nGPU-bbb\n", []string{"GPU-aaa", "GPU-bbb"}},
|
||||
{"trailing whitespace per line", "GPU-aaa \r\nGPU-bbb\r\n", []string{"GPU-aaa", "GPU-bbb"}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := parseNvidiaSMIUUIDs(c.in)
|
||||
if !reflect.DeepEqual(got, c.want) && (len(got) != 0 || len(c.want) != 0) {
|
||||
t.Errorf("parseNvidiaSMIUUIDs(%q) = %v, want %v", c.in, got, c.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverNvidiaGPUs_AbsentBinaryReturnsEmptyNoError(t *testing.T) {
|
||||
run := func(name string, args ...string) ([]byte, error) {
|
||||
return nil, errors.New(`exec: "nvidia-smi": executable file not found in $PATH`)
|
||||
}
|
||||
got := discoverNvidiaGPUs(run)
|
||||
if len(got) != 0 {
|
||||
t.Errorf("want empty list on absent nvidia-smi, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverNvidiaGPUs_ParsesFixtureOutput(t *testing.T) {
|
||||
var gotName string
|
||||
var gotArgs []string
|
||||
run := func(name string, args ...string) ([]byte, error) {
|
||||
gotName = name
|
||||
gotArgs = args
|
||||
return []byte("GPU-aaa\nGPU-bbb\n"), nil
|
||||
}
|
||||
got := discoverNvidiaGPUs(run)
|
||||
want := []string{"GPU-aaa", "GPU-bbb"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
if gotName != "nvidia-smi" {
|
||||
t.Errorf("command = %q, want nvidia-smi", gotName)
|
||||
}
|
||||
if len(gotArgs) != 2 || gotArgs[0] != "--query-gpu=uuid" || gotArgs[1] != "--format=csv,noheader" {
|
||||
t.Errorf("args = %v, want the uuid/csv,noheader query", gotArgs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverNvidiaGPUs_NoDevicesReturnsEmpty(t *testing.T) {
|
||||
run := func(name string, args ...string) ([]byte, error) {
|
||||
return []byte(""), nil
|
||||
}
|
||||
got := discoverNvidiaGPUs(run)
|
||||
if len(got) != 0 {
|
||||
t.Errorf("want empty list, got %v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,62 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// gpuLedger is the agent's local source of truth for which of the host's
|
||||
// discovered GPU UUIDs are free vs pinned to a running task's container. C4
|
||||
// reads a task's pinned UUIDs to build the CDI device list for containerd.
|
||||
type gpuLedger struct {
|
||||
mu sync.Mutex
|
||||
free []string
|
||||
pinned map[string][]string // key ("taskID/containerName") -> uuids
|
||||
}
|
||||
|
||||
// newGPULedger seeds the ledger with the host's discovered device UUIDs. A nil
|
||||
// or empty slice is a valid non-GPU host: every Pin then fails, which callers
|
||||
// treat as best-effort (no device, no CDI injection later).
|
||||
func newGPULedger(uuids []string) *gpuLedger {
|
||||
free := append([]string(nil), uuids...)
|
||||
return &gpuLedger{free: free, pinned: make(map[string][]string)}
|
||||
}
|
||||
|
||||
// gpuKey builds the ledger key for one container's pin/release.
|
||||
func gpuKey(taskID, containerName string) string {
|
||||
return taskID + "/" + containerName
|
||||
}
|
||||
|
||||
// Pin reserves n free UUIDs under key, returning them. An error means fewer
|
||||
// than n devices were free; the key is left unpinned so a retry (or a
|
||||
// best-effort caller) can proceed without a partial reservation.
|
||||
func (l *gpuLedger) Pin(key string, n int) ([]string, error) {
|
||||
if n <= 0 {
|
||||
return nil, nil
|
||||
}
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if len(l.free) < n {
|
||||
return nil, fmt.Errorf("gpu ledger: requested %d GPUs, %d free", n, len(l.free))
|
||||
}
|
||||
uuids := append([]string(nil), l.free[:n]...)
|
||||
l.free = l.free[n:]
|
||||
l.pinned[key] = uuids
|
||||
return uuids, nil
|
||||
}
|
||||
|
||||
// ReleaseTask frees every UUID pinned under any container key belonging to
|
||||
// taskID, called once the task's terminal STOPPED state is reported.
|
||||
func (l *gpuLedger) ReleaseTask(taskID string) {
|
||||
prefix := taskID + "/"
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
for key, uuids := range l.pinned {
|
||||
if !strings.HasPrefix(key, prefix) {
|
||||
continue
|
||||
}
|
||||
l.free = append(l.free, uuids...)
|
||||
delete(l.pinned, key)
|
||||
}
|
||||
}
|
||||
@@ -1,155 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGPULedger_PinReturnsDistinctUUIDs(t *testing.T) {
|
||||
l := newGPULedger([]string{"GPU-a", "GPU-b", "GPU-c"})
|
||||
got, err := l.Pin("t-1/web", 2)
|
||||
if err != nil {
|
||||
t.Fatalf("Pin: %v", err)
|
||||
}
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("want 2 UUIDs, got %v", got)
|
||||
}
|
||||
if got[0] == got[1] {
|
||||
t.Errorf("pinned UUIDs must be distinct, got %v", got)
|
||||
}
|
||||
if len(l.free) != 1 {
|
||||
t.Errorf("free = %d, want 1 remaining", len(l.free))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGPULedger_PinInsufficientFreeErrors(t *testing.T) {
|
||||
l := newGPULedger([]string{"GPU-a"})
|
||||
_, err := l.Pin("t-1/web", 2)
|
||||
if err == nil {
|
||||
t.Fatal("want error when requesting more GPUs than free")
|
||||
}
|
||||
if len(l.free) != 1 {
|
||||
t.Errorf("failed pin must not consume any device, free = %d", len(l.free))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGPULedger_PinZeroIsNoop(t *testing.T) {
|
||||
l := newGPULedger([]string{"GPU-a"})
|
||||
got, err := l.Pin("t-1/web", 0)
|
||||
if err != nil || got != nil {
|
||||
t.Errorf("Pin(0) = %v, %v; want nil, nil", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGPULedger_ReleaseTaskFreesOnlyThatTasksDevices(t *testing.T) {
|
||||
l := newGPULedger([]string{"GPU-a", "GPU-b", "GPU-c", "GPU-d"})
|
||||
if _, err := l.Pin(gpuKey("t-1", "web"), 2); err != nil {
|
||||
t.Fatalf("pin t-1: %v", err)
|
||||
}
|
||||
if _, err := l.Pin(gpuKey("t-2", "worker"), 1); err != nil {
|
||||
t.Fatalf("pin t-2: %v", err)
|
||||
}
|
||||
if len(l.free) != 1 {
|
||||
t.Fatalf("free = %d, want 1 before release", len(l.free))
|
||||
}
|
||||
|
||||
l.ReleaseTask("t-1")
|
||||
if len(l.free) != 3 {
|
||||
t.Errorf("free = %d, want 3 after releasing t-1's two devices", len(l.free))
|
||||
}
|
||||
if _, ok := l.pinned[gpuKey("t-2", "worker")]; !ok {
|
||||
t.Error("t-2's pin should survive releasing t-1")
|
||||
}
|
||||
if _, ok := l.pinned[gpuKey("t-1", "web")]; ok {
|
||||
t.Error("t-1's pin should be gone after release")
|
||||
}
|
||||
}
|
||||
|
||||
// Multi-device pinning across concurrent task assigns must hand out disjoint
|
||||
// UUID sets — no two tasks may ever share a device.
|
||||
func TestGPULedger_ConcurrentPinsHandOutDistinctUUIDs(t *testing.T) {
|
||||
const nDevices = 20
|
||||
uuids := make([]string, nDevices)
|
||||
for i := range uuids {
|
||||
uuids[i] = "GPU-" + string(rune('a'+i))
|
||||
}
|
||||
l := newGPULedger(uuids)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
seen := make(map[string]bool)
|
||||
errs := 0
|
||||
for i := range nDevices {
|
||||
wg.Add(1)
|
||||
go func(n int) {
|
||||
defer wg.Done()
|
||||
key := gpuKey("t-concurrent", string(rune('A'+n)))
|
||||
got, err := l.Pin(key, 1)
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if err != nil {
|
||||
errs++
|
||||
return
|
||||
}
|
||||
for _, u := range got {
|
||||
if seen[u] {
|
||||
t.Errorf("UUID %s handed out to more than one concurrent pin", u)
|
||||
}
|
||||
seen[u] = true
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
if errs != 0 {
|
||||
t.Errorf("want all %d pins to succeed (exactly enough devices), got %d errors", nDevices, errs)
|
||||
}
|
||||
if len(seen) != nDevices {
|
||||
t.Errorf("distinct UUIDs handed out = %d, want %d", len(seen), nDevices)
|
||||
}
|
||||
if len(l.free) != 0 {
|
||||
t.Errorf("free = %d, want 0 after all devices pinned", len(l.free))
|
||||
}
|
||||
}
|
||||
|
||||
// Releasing a task's devices concurrently with new pins never double-hands a
|
||||
// UUID: a device is only ever free or pinned to exactly one key at a time.
|
||||
func TestGPULedger_ConcurrentPinAndReleaseNoDoubleAssignment(t *testing.T) {
|
||||
l := newGPULedger([]string{"GPU-a", "GPU-b"})
|
||||
var wg sync.WaitGroup
|
||||
for round := range 50 {
|
||||
wg.Add(2)
|
||||
taskID := "t-" + string(rune('a'+round%26))
|
||||
go func(id string) {
|
||||
defer wg.Done()
|
||||
_, _ = l.Pin(gpuKey(id, "web"), 1)
|
||||
}(taskID)
|
||||
go func(id string) {
|
||||
defer wg.Done()
|
||||
l.ReleaseTask(id)
|
||||
}(taskID)
|
||||
}
|
||||
wg.Wait()
|
||||
// Invariant: every currently-pinned UUID appears in exactly one pinned entry
|
||||
// and never simultaneously in free.
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
freeSet := make(map[string]bool, len(l.free))
|
||||
for _, u := range l.free {
|
||||
if freeSet[u] {
|
||||
t.Errorf("duplicate UUID %s in free list", u)
|
||||
}
|
||||
freeSet[u] = true
|
||||
}
|
||||
pinnedSet := make(map[string]bool)
|
||||
for _, uuids := range l.pinned {
|
||||
for _, u := range uuids {
|
||||
if pinnedSet[u] {
|
||||
t.Errorf("UUID %s pinned under more than one key", u)
|
||||
}
|
||||
if freeSet[u] {
|
||||
t.Errorf("UUID %s is both free and pinned", u)
|
||||
}
|
||||
pinnedSet[u] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,51 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
)
|
||||
|
||||
// heartbeater keeps the container instance alive by periodically re-registering
|
||||
// it through the gateway. Real ECS folds liveness into ACS; v1 folds it into an
|
||||
// idempotent RegisterContainerInstance, which refreshes the instance's LastSeen
|
||||
// so the scheduler can tell live instances from dead ones (no bus.Heartbeat).
|
||||
type heartbeater struct {
|
||||
cp controlPlane
|
||||
id identity
|
||||
interval time.Duration
|
||||
// runtimeReady reports whether a container runtime is resolved, so a beat
|
||||
// advertises capacity only while there is something to run it with. nil
|
||||
// means always ready.
|
||||
runtimeReady func() bool
|
||||
}
|
||||
|
||||
func newHeartbeater(cp controlPlane, id identity, interval time.Duration, runtimeReady func() bool) *heartbeater {
|
||||
if interval <= 0 {
|
||||
interval = defaultHeartbeat
|
||||
}
|
||||
return &heartbeater{cp: cp, id: id, interval: interval, runtimeReady: runtimeReady}
|
||||
}
|
||||
|
||||
// beat re-registers the instance once, refreshing its LastSeen.
|
||||
func (h *heartbeater) beat() error {
|
||||
ready := h.runtimeReady == nil || h.runtimeReady()
|
||||
return h.cp.Register(advertisedIdentity(h.id, ready))
|
||||
}
|
||||
|
||||
// Run beats every interval until ctx is cancelled. A failed beat is logged and
|
||||
// retried on the next tick rather than killing the loop.
|
||||
func (h *heartbeater) Run(ctx context.Context) {
|
||||
ticker := time.NewTicker(h.interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if err := h.beat(); err != nil {
|
||||
slog.Warn("ecs-agent: heartbeat (re-register) failed", "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+13
-13
@@ -1,13 +1,6 @@
|
||||
// ecs-agent runs inside a Spinifex ECS container instance (a guest VM booted from
|
||||
// the ECS-AMI, with containerd baked in). It registers the host with the ECS
|
||||
// scheduler through the AWS gateway over TLS+SigV4 (never NATS), heartbeats by
|
||||
// re-registering, polls the gateway for task assignments, runs them through
|
||||
// containerd, and reports state back over the gateway.
|
||||
//
|
||||
// Static config (gateway URL, CA, region, cluster, seeded IAM creds, containerd
|
||||
// socket) is read from the cloud-init env file /etc/spinifex-ecs/agent.env
|
||||
// (KEY=value); real env vars override it. Host identity (account, instance, AZ)
|
||||
// comes from IMDS at boot.
|
||||
// Command ecs-agent runs inside a Spinifex ECS container instance; the
|
||||
// implementation is in agents/ecs/agent. This binary owns process lifecycle:
|
||||
// signal handling, startup and run failure logging, and the exit code.
|
||||
package main
|
||||
|
||||
import (
|
||||
@@ -18,12 +11,19 @@ import (
|
||||
"syscall"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
|
||||
ecsagent "github.com/mulgadc/spinifex/spinifex/agents/ecs/agent"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cfg := loadConfig(defaultEnvFile)
|
||||
// version is the agent build version, reported in the register message.
|
||||
// Overridable via -ldflags "-X main.version=...".
|
||||
var version = "dev"
|
||||
|
||||
agent, err := New(cfg)
|
||||
func main() {
|
||||
cfg := ecsagent.LoadConfig(ecsagent.DefaultEnvFile)
|
||||
cfg.AgentVersion = version
|
||||
|
||||
agent, err := ecsagent.New(cfg)
|
||||
if err != nil {
|
||||
slog.Error("ecs-agent: startup failed", "err", err)
|
||||
os.Exit(1)
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/aws/aws-sdk-go-v2/feature/ec2/imds"
|
||||
"github.com/mulgadc/spinifex/internal/imdscreds"
|
||||
)
|
||||
|
||||
// instanceMetadata is the IMDS-derived identity of the host the agent runs on.
|
||||
type instanceMetadata struct {
|
||||
AccountID string
|
||||
InstanceID string
|
||||
AZ string
|
||||
InstanceType string
|
||||
}
|
||||
|
||||
// fetchInstanceMetadata reads the instance-id, AZ and account-id from IMDSv2
|
||||
// via the SDK IMDS client. base is the IMDS root, e.g.
|
||||
// "http://169.254.169.254/latest".
|
||||
func fetchInstanceMetadata(client *http.Client, base string) (instanceMetadata, error) {
|
||||
imdsClient := imdscreds.NewClient(client, base)
|
||||
ctx := context.Background()
|
||||
|
||||
instanceID, err := getMetadata(ctx, imdsClient, "instance-id")
|
||||
if err != nil {
|
||||
return instanceMetadata{}, fmt.Errorf("instance-id: %w", err)
|
||||
}
|
||||
az, err := getMetadata(ctx, imdsClient, "placement/availability-zone")
|
||||
if err != nil {
|
||||
return instanceMetadata{}, fmt.Errorf("availability-zone: %w", err)
|
||||
}
|
||||
|
||||
doc, err := imdsClient.GetInstanceIdentityDocument(ctx, &imds.GetInstanceIdentityDocumentInput{})
|
||||
if err != nil {
|
||||
return instanceMetadata{}, fmt.Errorf("identity document: %w", err)
|
||||
}
|
||||
|
||||
// The instance type comes off the identity document already in hand rather
|
||||
// than a further meta-data read; the scheduler sizes the host's awsvpc ENI
|
||||
// capacity from it.
|
||||
return instanceMetadata{
|
||||
AccountID: doc.AccountID,
|
||||
InstanceID: strings.TrimSpace(instanceID),
|
||||
AZ: strings.TrimSpace(az),
|
||||
InstanceType: strings.TrimSpace(doc.InstanceType),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// getMetadata reads a single meta-data path and returns its body as a string.
|
||||
func getMetadata(ctx context.Context, imdsClient *imds.Client, path string) (string, error) {
|
||||
out, err := imdsClient.GetMetadata(ctx, &imds.GetMetadataInput{Path: path})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer out.Content.Close()
|
||||
body, err := io.ReadAll(out.Content)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(body), nil
|
||||
}
|
||||
@@ -1,82 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func metadataStub(t *testing.T, enforceV2 bool) *httptest.Server {
|
||||
t.Helper()
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPut {
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
// Real IMDS echoes the requested TTL back as a response header; the SDK
|
||||
// client parses it from there, not the body.
|
||||
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
|
||||
_, _ = w.Write([]byte("v2-token"))
|
||||
})
|
||||
guard := func(w http.ResponseWriter, r *http.Request) bool {
|
||||
if enforceV2 && r.Header.Get("X-Aws-Ec2-Metadata-Token") != "v2-token" {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
mux.HandleFunc("/latest/meta-data/instance-id", func(w http.ResponseWriter, r *http.Request) {
|
||||
if guard(w, r) {
|
||||
_, _ = w.Write([]byte("i-0abc123"))
|
||||
}
|
||||
})
|
||||
mux.HandleFunc("/latest/meta-data/placement/availability-zone", func(w http.ResponseWriter, r *http.Request) {
|
||||
if guard(w, r) {
|
||||
_, _ = w.Write([]byte("us-east-1a"))
|
||||
}
|
||||
})
|
||||
mux.HandleFunc("/latest/dynamic/instance-identity/document", func(w http.ResponseWriter, r *http.Request) {
|
||||
if guard(w, r) {
|
||||
_, _ = w.Write([]byte(`{"accountId":"123456789012","region":"us-east-1"}`))
|
||||
}
|
||||
})
|
||||
srv := httptest.NewServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func TestFetchInstanceMetadata_OK(t *testing.T) {
|
||||
srv := metadataStub(t, true)
|
||||
got, err := fetchInstanceMetadata(srv.Client(), srv.URL+"/latest")
|
||||
if err != nil {
|
||||
t.Fatalf("fetch: %v", err)
|
||||
}
|
||||
if got.InstanceID != "i-0abc123" || got.AZ != "us-east-1a" || got.AccountID != "123456789012" {
|
||||
t.Errorf("metadata mismatch: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchInstanceMetadata_TokenRequired(t *testing.T) {
|
||||
// Server returns 405 for non-PUT token; a GET-only client would 401 downstream.
|
||||
srv := metadataStub(t, true)
|
||||
if _, err := fetchInstanceMetadata(srv.Client(), srv.URL+"/latest"); err != nil {
|
||||
t.Fatalf("v2 path should succeed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchInstanceMetadata_BadDocument(t *testing.T) {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
|
||||
_, _ = w.Write([]byte("t"))
|
||||
})
|
||||
mux.HandleFunc("/latest/meta-data/instance-id", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("i-1")) })
|
||||
mux.HandleFunc("/latest/meta-data/placement/availability-zone", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("z")) })
|
||||
mux.HandleFunc("/latest/dynamic/instance-identity/document", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("not json")) })
|
||||
srv := httptest.NewServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
if _, err := fetchInstanceMetadata(srv.Client(), srv.URL+"/latest"); err == nil {
|
||||
t.Fatal("expected parse error on bad identity document")
|
||||
}
|
||||
}
|
||||
@@ -1,216 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"hash/crc32"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// netnsRunDir is the iproute2 netns mount directory; a name here is the netns
|
||||
// path containerd joins via the OCI network namespace path.
|
||||
const netnsRunDir = "/var/run/netns"
|
||||
|
||||
// Credentials path constants: each task netns gets a /30 veth into the host netns
|
||||
// from 169.254.172.0/22 and a host-routed /32 to the credential endpoint, keeping
|
||||
// the endpoint off the task ENI's VPC route. credTaskVeth is the task-side name
|
||||
// (unique per netns); the host side is per-task (credVethName).
|
||||
const (
|
||||
credTaskVeth = "ecsc0"
|
||||
credEndpointCIDR = defaultCredEndpointIP + "/32"
|
||||
)
|
||||
|
||||
// udhcpcScript is the config script busybox udhcpc execs to apply a lease. It
|
||||
// is the reason presence of the udhcpc binary is not enough to use it: Debian
|
||||
// and Ubuntu's busybox ship the applet without this script, and udhcpc then
|
||||
// takes the lease and exits nonzero having configured nothing.
|
||||
const udhcpcScript = "/usr/share/udhcpc/default.script"
|
||||
|
||||
// netCmdRunner executes a host networking command. Abstracted so unit tests can
|
||||
// assert the ip/netns sequence without touching the kernel.
|
||||
type netCmdRunner interface {
|
||||
Run(name string, args ...string) (string, error)
|
||||
}
|
||||
|
||||
// execNetRunner runs real commands.
|
||||
type execNetRunner struct{}
|
||||
|
||||
func (execNetRunner) Run(name string, args ...string) (string, error) {
|
||||
out, err := exec.Command(name, args...).CombinedOutput()
|
||||
return string(out), err
|
||||
}
|
||||
|
||||
// taskNetns builds and tears down per-task awsvpc network namespaces. The ENI is
|
||||
// already hot-plugged into the VM (scheduler did that); the agent finds the NIC
|
||||
// by MAC, moves it into a task netns, and DHCPs the OVN-assigned ENI IP — the
|
||||
// same way the primary NIC obtains its address.
|
||||
type taskNetns struct {
|
||||
run netCmdRunner
|
||||
nicWait time.Duration
|
||||
poll time.Duration
|
||||
// Injected so the DHCP-client choice below is unit-testable on a host that
|
||||
// has neither client installed.
|
||||
lookPath func(string) (string, error)
|
||||
fileIsExec func(string) bool
|
||||
}
|
||||
|
||||
func newTaskNetns(run netCmdRunner) *taskNetns {
|
||||
return &taskNetns{
|
||||
run: run,
|
||||
nicWait: 30 * time.Second,
|
||||
poll: time.Second,
|
||||
lookPath: exec.LookPath,
|
||||
fileIsExec: func(p string) bool { fi, err := os.Stat(p); return err == nil && fi.Mode()&0o111 != 0 },
|
||||
}
|
||||
}
|
||||
|
||||
// dhcpStep returns the in-netns DHCP command that leases the task ENI's
|
||||
// OVN-assigned address.
|
||||
//
|
||||
// This image ships on two distributions and they do not agree on a DHCP client.
|
||||
// Alpine has busybox udhcpc; Ubuntu has no udhcpc at all, and `ip netns exec`
|
||||
// exits 1 — not 127 — when it cannot exec the binary, so a hardcoded udhcpc
|
||||
// fails as an opaque "network setup failed" with the task stuck in STOPPED.
|
||||
// Resolve the client at setup time instead, preferring udhcpc so Alpine keeps
|
||||
// its existing behaviour exactly, then falling back to dhcpcd. Same order and
|
||||
// same reasoning as dhcp_oneshot() in mulga-mgmt-net.sh.
|
||||
//
|
||||
// Both invocations are one-shot: udhcpc -q exits once it has the lease and -n
|
||||
// gives up rather than backgrounding, and dhcpcd -1 exits after configuring the
|
||||
// interface. Neither leaves a daemon behind in the netns.
|
||||
func (n *taskNetns) dhcpStep(name, iface string) ([]string, error) {
|
||||
if _, err := n.lookPath("udhcpc"); err == nil && n.fileIsExec(udhcpcScript) {
|
||||
return []string{"ip", "netns", "exec", name, "udhcpc", "-i", iface, "-q", "-n"}, nil
|
||||
}
|
||||
// Lease state is keyed by interface name, which is unique per task here:
|
||||
// each task ENI is a separate hot-plug, so it lands on its own PCI slot and
|
||||
// gets its own predictable name before being moved into the netns.
|
||||
if _, err := n.lookPath("dhcpcd"); err == nil {
|
||||
return []string{"ip", "netns", "exec", name, "dhcpcd", "-q", "-1", "-t", "20", iface}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("no DHCP client for %s: need udhcpc (with %s) or dhcpcd", iface, udhcpcScript)
|
||||
}
|
||||
|
||||
func netnsName(taskID string) string { return "ecs-" + taskID }
|
||||
func netnsPathFor(taskID string) string {
|
||||
return netnsRunDir + "/" + netnsName(taskID)
|
||||
}
|
||||
|
||||
// Setup creates the task netns, moves the ENI NIC (matched by MAC) into it, and
|
||||
// brings it up with a DHCP lease. It returns the netns path for containerd. On
|
||||
// any failure it tears the partial netns down so a retry starts clean.
|
||||
func (n *taskNetns) Setup(taskID, mac string) (string, error) {
|
||||
iface, err := n.waitForNIC(mac)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
name := netnsName(taskID)
|
||||
if _, err := n.run.Run("ip", "netns", "add", name); err != nil {
|
||||
return "", fmt.Errorf("netns add %s: %w", name, err)
|
||||
}
|
||||
hostVeth := credVethName(taskID)
|
||||
hostIP, taskIP := credSubnet(taskID)
|
||||
dhcp, err := n.dhcpStep(name, iface)
|
||||
if err != nil {
|
||||
_ = n.Teardown(taskID)
|
||||
return "", err
|
||||
}
|
||||
steps := [][]string{
|
||||
{"ip", "link", "set", iface, "netns", name},
|
||||
{"ip", "-n", name, "link", "set", "lo", "up"},
|
||||
{"ip", "-n", name, "link", "set", iface, "up"},
|
||||
dhcp,
|
||||
// Credentials path: a veth into the host netns plus a host-routed /32 to the
|
||||
// credential endpoint, so the container reaches 169.254.170.2 without it
|
||||
// leaking onto the task ENI's VPC route.
|
||||
{"ip", "link", "add", hostVeth, "type", "veth", "peer", "name", credTaskVeth, "netns", name},
|
||||
{"ip", "addr", "add", hostIP + "/30", "dev", hostVeth},
|
||||
{"ip", "link", "set", hostVeth, "up"},
|
||||
{"ip", "-n", name, "addr", "add", taskIP + "/30", "dev", credTaskVeth},
|
||||
{"ip", "-n", name, "link", "set", credTaskVeth, "up"},
|
||||
{"ip", "-n", name, "route", "add", credEndpointCIDR, "via", hostIP},
|
||||
}
|
||||
for _, s := range steps {
|
||||
if _, err := n.run.Run(s[0], s[1:]...); err != nil {
|
||||
_ = n.Teardown(taskID)
|
||||
return "", fmt.Errorf("netns setup %v: %w", s, err)
|
||||
}
|
||||
}
|
||||
return netnsPathFor(taskID), nil
|
||||
}
|
||||
|
||||
// credVethName is the host-side veth name for a task's credential path. Derived
|
||||
// from the taskID and kept within the 15-char interface-name limit.
|
||||
func credVethName(taskID string) string {
|
||||
return fmt.Sprintf("ecsv%08x", crc32.ChecksumIEEE([]byte(taskID)))
|
||||
}
|
||||
|
||||
// credSubnet derives a unique /30 in 169.254.172.0/22 for a task's credential
|
||||
// veth, returning the host-side and task-side addresses. The host side is the
|
||||
// container's gateway to the credential endpoint.
|
||||
func credSubnet(taskID string) (hostIP, taskIP string) {
|
||||
slot := crc32.ChecksumIEEE([]byte(taskID)) % 256 // 256 /30s in the /22
|
||||
third := 172 + (slot*4)/256
|
||||
base := (slot * 4) % 256
|
||||
hostIP = fmt.Sprintf("169.254.%d.%d", third, base+1)
|
||||
taskIP = fmt.Sprintf("169.254.%d.%d", third, base+2)
|
||||
return hostIP, taskIP
|
||||
}
|
||||
|
||||
// Teardown deletes the task netns (releasing the moved NIC). Idempotent enough
|
||||
// for the stop path: a missing netns is reported but not fatal to the caller.
|
||||
func (n *taskNetns) Teardown(taskID string) error {
|
||||
name := netnsName(taskID)
|
||||
// Deleting the netns destroys the task-side veth and its host peer; remove the
|
||||
// host side explicitly too so a leaked half from a partial setup is reaped.
|
||||
_, _ = n.run.Run("ip", "link", "del", credVethName(taskID))
|
||||
if _, err := n.run.Run("ip", "netns", "del", name); err != nil {
|
||||
return fmt.Errorf("netns del %s: %w", name, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// waitForNIC polls until a NIC with the given MAC appears in the host netns,
|
||||
// returning its interface name. The hot-plug pipeline is async, so the device
|
||||
// may lag the assign by a few seconds.
|
||||
func (n *taskNetns) waitForNIC(mac string) (string, error) {
|
||||
want := strings.ToLower(strings.TrimSpace(mac))
|
||||
deadline := time.Now().Add(n.nicWait)
|
||||
for {
|
||||
iface, err := n.findNICByMAC(want)
|
||||
if err == nil {
|
||||
return iface, nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return "", fmt.Errorf("nic with mac %s not found within %s", want, n.nicWait)
|
||||
}
|
||||
time.Sleep(n.poll)
|
||||
}
|
||||
}
|
||||
|
||||
// findNICByMAC parses `ip -o link` and returns the interface whose link/ether
|
||||
// matches mac.
|
||||
func (n *taskNetns) findNICByMAC(mac string) (string, error) {
|
||||
out, err := n.run.Run("ip", "-o", "link")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("ip -o link: %w", err)
|
||||
}
|
||||
for line := range strings.SplitSeq(out, "\n") {
|
||||
if !strings.Contains(strings.ToLower(line), mac) {
|
||||
continue
|
||||
}
|
||||
// "<idx>: <ifname>: <flags> ... link/ether <mac> ..."
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 2 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimSuffix(strings.TrimSuffix(fields[1], ":"), "@")
|
||||
if i := strings.IndexByte(name, '@'); i >= 0 {
|
||||
name = name[:i]
|
||||
}
|
||||
return name, nil
|
||||
}
|
||||
return "", fmt.Errorf("no interface with mac %s", mac)
|
||||
}
|
||||
@@ -1,244 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// fakeNetRunner records every command and serves canned `ip -o link` output. A
|
||||
// failOn substring forces the matching command to error so teardown paths run.
|
||||
type fakeNetRunner struct {
|
||||
mu sync.Mutex
|
||||
calls [][]string
|
||||
linkOut string
|
||||
failOn string
|
||||
}
|
||||
|
||||
func (f *fakeNetRunner) Run(name string, args ...string) (string, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
cmd := append([]string{name}, args...)
|
||||
f.calls = append(f.calls, cmd)
|
||||
joined := strings.Join(cmd, " ")
|
||||
if f.failOn != "" && strings.Contains(joined, f.failOn) {
|
||||
return "", errors.New("boom: " + joined)
|
||||
}
|
||||
if name == "ip" && len(args) >= 2 && args[0] == "-o" && args[1] == "link" {
|
||||
return f.linkOut, nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (f *fakeNetRunner) joined() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
out := make([]string, len(f.calls))
|
||||
for i, c := range f.calls {
|
||||
out[i] = strings.Join(c, " ")
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (f *fakeNetRunner) sawAny(sub string) bool {
|
||||
for _, j := range f.joined() {
|
||||
if strings.Contains(j, sub) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
const linkWithENI = `1: lo: <LOOPBACK,UP> mtu 65536 ... link/loopback 00:00:00:00:00:00 ...
|
||||
2: eth0: <BROADCAST,UP> mtu 1500 ... link/ether 52:54:00:aa:bb:cc ...
|
||||
3: eth1: <BROADCAST,UP> mtu 1500 ... link/ether 52:54:00:de:ad:01 ...`
|
||||
|
||||
// withDHCPClients pins which DHCP clients the netns sees, so a test asserts the
|
||||
// branch it names rather than whatever happens to be installed on the machine
|
||||
// running it. Without this the DHCP step is host-dependent: an Alpine CI box and
|
||||
// an Ubuntu dev box would exercise different code from the same test.
|
||||
func withDHCPClients(n *taskNetns, present map[string]bool, scriptOK bool) {
|
||||
n.lookPath = func(f string) (string, error) {
|
||||
if present[f] {
|
||||
return "/sbin/" + f, nil
|
||||
}
|
||||
return "", errors.New("not found")
|
||||
}
|
||||
n.fileIsExec = func(string) bool { return scriptOK }
|
||||
}
|
||||
|
||||
// newTestNetns defaults to the Alpine arrangement — udhcpc plus its config
|
||||
// script — so the sequence assertions below keep describing that platform.
|
||||
func newTestNetns(f *fakeNetRunner) *taskNetns {
|
||||
n := newTaskNetns(f)
|
||||
n.nicWait = 50 * time.Millisecond
|
||||
n.poll = 5 * time.Millisecond
|
||||
withDHCPClients(n, map[string]bool{"udhcpc": true}, true)
|
||||
return n
|
||||
}
|
||||
|
||||
func TestNetns_FindNICByMAC(t *testing.T) {
|
||||
n := newTestNetns(&fakeNetRunner{linkOut: linkWithENI})
|
||||
iface, err := n.findNICByMAC("52:54:00:de:ad:01")
|
||||
if err != nil || iface != "eth1" {
|
||||
t.Fatalf("want eth1, got %q err=%v", iface, err)
|
||||
}
|
||||
if _, err := n.findNICByMAC("52:54:00:ff:ff:ff"); err == nil {
|
||||
t.Fatal("want error for absent mac")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNetns_SetupSequence(t *testing.T) {
|
||||
f := &fakeNetRunner{linkOut: linkWithENI}
|
||||
n := newTestNetns(f)
|
||||
path, err := n.Setup("t-001", "52:54:00:DE:AD:01")
|
||||
if err != nil {
|
||||
t.Fatalf("setup: %v", err)
|
||||
}
|
||||
if path != netnsPathFor("t-001") {
|
||||
t.Fatalf("want %q, got %q", netnsPathFor("t-001"), path)
|
||||
}
|
||||
hostIP, taskIP := credSubnet("t-001")
|
||||
hostVeth := credVethName("t-001")
|
||||
want := []string{
|
||||
"ip netns add ecs-t-001",
|
||||
"ip link set eth1 netns ecs-t-001",
|
||||
"ip -n ecs-t-001 link set lo up",
|
||||
"ip -n ecs-t-001 link set eth1 up",
|
||||
"ip netns exec ecs-t-001 udhcpc -i eth1 -q -n",
|
||||
"ip link add " + hostVeth + " type veth peer name ecsc0 netns ecs-t-001",
|
||||
"ip addr add " + hostIP + "/30 dev " + hostVeth,
|
||||
"ip -n ecs-t-001 addr add " + taskIP + "/30 dev ecsc0",
|
||||
"ip -n ecs-t-001 route add 169.254.170.2/32 via " + hostIP,
|
||||
}
|
||||
for _, w := range want {
|
||||
if !f.sawAny(w) {
|
||||
t.Errorf("missing command %q in %v", w, f.joined())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCredSubnet_UniquePer30(t *testing.T) {
|
||||
hostA, taskA := credSubnet("task-a")
|
||||
hostB, taskB := credSubnet("task-b")
|
||||
if hostA == hostB {
|
||||
t.Errorf("distinct tasks share a host IP: %s", hostA)
|
||||
}
|
||||
for _, ip := range []string{hostA, taskA, hostB, taskB} {
|
||||
if !strings.HasPrefix(ip, "169.254.17") {
|
||||
t.Errorf("cred IP %s outside 169.254.172.0/22", ip)
|
||||
}
|
||||
}
|
||||
// host and task ends of one task sit in the same /30 (differ in last octet).
|
||||
if hostA[:strings.LastIndexByte(hostA, '.')] != taskA[:strings.LastIndexByte(taskA, '.')] {
|
||||
t.Errorf("host %s and task %s not in same /30", hostA, taskA)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNetns_SetupNICNotFound(t *testing.T) {
|
||||
f := &fakeNetRunner{linkOut: linkWithENI}
|
||||
n := newTestNetns(f)
|
||||
if _, err := n.Setup("t-001", "52:54:00:00:00:99"); err == nil {
|
||||
t.Fatal("want timeout error for absent NIC")
|
||||
}
|
||||
if f.sawAny("netns add") {
|
||||
t.Error("should not create netns when NIC never appears")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNetns_SetupTearsDownOnStepFailure(t *testing.T) {
|
||||
f := &fakeNetRunner{linkOut: linkWithENI, failOn: "udhcpc"}
|
||||
n := newTestNetns(f)
|
||||
if _, err := n.Setup("t-001", "52:54:00:de:ad:01"); err == nil {
|
||||
t.Fatal("want error when udhcpc fails")
|
||||
}
|
||||
if !f.sawAny("ip netns del ecs-t-001") {
|
||||
t.Errorf("expected teardown after step failure, got %v", f.joined())
|
||||
}
|
||||
}
|
||||
|
||||
// The image runs on two distributions with different DHCP clients, and picking
|
||||
// the wrong one fails as an opaque "network setup failed" with the task stuck
|
||||
// in STOPPED — `ip netns exec` exits 1, not 127, when it cannot exec the
|
||||
// binary. Each branch is asserted so that misresolution is a test failure.
|
||||
func TestNetns_DHCPStepPerPlatform(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
present map[string]bool
|
||||
scriptOK bool
|
||||
want string
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "alpine: udhcpc with its config script",
|
||||
present: map[string]bool{"udhcpc": true, "dhcpcd": true},
|
||||
scriptOK: true,
|
||||
want: "ip netns exec ecs-t-001 udhcpc -i eth1 -q -n",
|
||||
},
|
||||
{
|
||||
// Ubuntu ships no udhcpc at all — the case that broke awsvpc tasks.
|
||||
name: "ubuntu: no udhcpc, falls back to dhcpcd",
|
||||
present: map[string]bool{"dhcpcd": true},
|
||||
want: "ip netns exec ecs-t-001 dhcpcd -q -1 -t 20 eth1",
|
||||
},
|
||||
{
|
||||
// Debian/Ubuntu busybox ships the applet without the script, so
|
||||
// presence of the binary alone must not select it.
|
||||
name: "udhcpc present but config script missing: still dhcpcd",
|
||||
present: map[string]bool{"udhcpc": true, "dhcpcd": true},
|
||||
scriptOK: false,
|
||||
want: "ip netns exec ecs-t-001 dhcpcd -q -1 -t 20 eth1",
|
||||
},
|
||||
{
|
||||
name: "neither client: explicit error, not a silent bad command",
|
||||
present: map[string]bool{},
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
n := newTestNetns(&fakeNetRunner{linkOut: linkWithENI})
|
||||
withDHCPClients(n, tc.present, tc.scriptOK)
|
||||
got, err := n.dhcpStep("ecs-t-001", "eth1")
|
||||
if tc.wantErr {
|
||||
if err == nil {
|
||||
t.Fatalf("want error with no DHCP client, got %v", got)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("dhcpStep: %v", err)
|
||||
}
|
||||
if joined := strings.Join(got, " "); joined != tc.want {
|
||||
t.Errorf("want %q, got %q", tc.want, joined)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A missing DHCP client must not leave the half-built netns behind for the next
|
||||
// attempt to trip over, same as any other failed setup step.
|
||||
func TestNetns_SetupTearsDownWhenNoDHCPClient(t *testing.T) {
|
||||
f := &fakeNetRunner{linkOut: linkWithENI}
|
||||
n := newTestNetns(f)
|
||||
withDHCPClients(n, map[string]bool{}, false)
|
||||
if _, err := n.Setup("t-001", "52:54:00:de:ad:01"); err == nil {
|
||||
t.Fatal("want error when no DHCP client is available")
|
||||
}
|
||||
if !f.sawAny("ip netns del ecs-t-001") {
|
||||
t.Errorf("expected teardown, got %v", f.joined())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNetns_Teardown(t *testing.T) {
|
||||
f := &fakeNetRunner{}
|
||||
n := newTestNetns(f)
|
||||
if err := n.Teardown("t-001"); err != nil {
|
||||
t.Fatalf("teardown: %v", err)
|
||||
}
|
||||
if !f.sawAny("ip netns del ecs-t-001") {
|
||||
t.Errorf("expected netns del, got %v", f.joined())
|
||||
}
|
||||
}
|
||||
@@ -1,115 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
// reconcile re-syncs the gateway with what this instance is actually running
|
||||
// after an agent restart. It lists the runtime's containers and, for each task
|
||||
// in this cluster, either re-adopts it (still running: re-register IAM
|
||||
// credentials, re-attach the exit-wait, refresh RUNNING) or reports it STOPPED
|
||||
// and clears its leftovers. The returned set seeds pollAssignments so the
|
||||
// gateway's re-delivered assignments for these tasks are acked but not re-run.
|
||||
//
|
||||
// Reporting the dead ones is the load-bearing half. The exit-wait is cancelled
|
||||
// when the agent shuts down, so a container that dies while the agent is down
|
||||
// has nobody to report its exit; the gateway holds the task RUNNING forever,
|
||||
// its service never replaces it, and the target never recovers.
|
||||
//
|
||||
// It is best-effort: a nil runner or a List error returns an empty set and the
|
||||
// agent falls back to its pre-reconcile behaviour (poll + run).
|
||||
func (a *Agent) reconcile(ctx context.Context) map[string]bool {
|
||||
adopted := map[string]bool{}
|
||||
runner := a.rt.get()
|
||||
if runner == nil {
|
||||
return adopted
|
||||
}
|
||||
containers, err := runner.List(ctx)
|
||||
if err != nil {
|
||||
slog.Warn("ecs-agent: reconcile list failed", "err", err)
|
||||
return adopted
|
||||
}
|
||||
|
||||
// Group this cluster's containers back into their tasks, running or not. The
|
||||
// Assign is reconstructed from labels — enough to re-register creds and tear
|
||||
// down netns.
|
||||
type task struct {
|
||||
as *bus.Assign
|
||||
running []ctrruntime.Container
|
||||
dead []ctrruntime.Container
|
||||
}
|
||||
tasks := map[string]*task{}
|
||||
for _, c := range containers {
|
||||
if c.Labels[labelClusterName] != a.id.ClusterName {
|
||||
continue
|
||||
}
|
||||
taskID := c.Labels[labelTaskID]
|
||||
if taskID == "" {
|
||||
continue
|
||||
}
|
||||
t := tasks[taskID]
|
||||
if t == nil {
|
||||
t = &task{as: &bus.Assign{
|
||||
TaskID: taskID,
|
||||
ClusterName: a.id.ClusterName,
|
||||
CredID: c.Labels[labelCredID],
|
||||
TaskRoleARN: c.Labels[labelTaskRoleARN],
|
||||
ENIMacAddress: c.Labels[labelENIMac],
|
||||
}}
|
||||
tasks[taskID] = t
|
||||
}
|
||||
if c.Running {
|
||||
t.running = append(t.running, c)
|
||||
} else {
|
||||
t.dead = append(t.dead, c)
|
||||
}
|
||||
}
|
||||
|
||||
for taskID, t := range tasks {
|
||||
// A task with nothing running is over, whatever the gateway still believes.
|
||||
// Mixed running/dead is left to the running path: the exit-wait re-attached
|
||||
// below reports the individual container when it is reaped.
|
||||
if len(t.running) == 0 {
|
||||
a.reportStoppedTask(ctx, t.as, t.dead)
|
||||
adopted[taskID] = true
|
||||
continue
|
||||
}
|
||||
|
||||
a.registerTaskCreds(t.as)
|
||||
statuses := make([]bus.ContainerStatus, 0, len(t.running))
|
||||
for _, c := range t.running {
|
||||
name := c.Labels[labelContainerName]
|
||||
go a.waitContainer(ctx, t.as, name, c.ID)
|
||||
statuses = append(statuses, bus.ContainerStatus{Name: name, Status: bus.TaskStatusRunning, ContainerID: c.ID})
|
||||
}
|
||||
a.reportTaskState(t.as, bus.TaskStatusRunning, "", statuses)
|
||||
adopted[taskID] = true
|
||||
slog.Info("ecs-agent: re-adopted running task", "task", taskID, "containers", len(t.running))
|
||||
}
|
||||
return adopted
|
||||
}
|
||||
|
||||
// reportStoppedTask clears a task's leftover containers and tells the gateway it
|
||||
// is STOPPED, so the scheduler stops counting a task this instance is not
|
||||
// running and its service can place a replacement.
|
||||
func (a *Agent) reportStoppedTask(ctx context.Context, as *bus.Assign, dead []ctrruntime.Container) {
|
||||
runner := a.rt.get()
|
||||
statuses := make([]bus.ContainerStatus, 0, len(dead))
|
||||
for _, c := range dead {
|
||||
if runner == nil {
|
||||
break
|
||||
}
|
||||
if err := runner.Remove(ctx, c.ID); err != nil {
|
||||
slog.Warn("ecs-agent: reconcile remove failed", "task", as.TaskID, "container", c.ID, "err", err)
|
||||
}
|
||||
statuses = append(statuses, bus.ContainerStatus{
|
||||
Name: c.Labels[labelContainerName], Status: bus.TaskStatusStopped, ContainerID: c.ID,
|
||||
})
|
||||
}
|
||||
a.reportTaskState(as, bus.TaskStatusStopped, "Container exited while the agent was not running", statuses)
|
||||
slog.Info("ecs-agent: reported stopped task", "task", as.TaskID, "containers", len(dead))
|
||||
}
|
||||
@@ -1,250 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
const testTaskRole = "arn:aws:iam::111122223333:role/task"
|
||||
|
||||
// adoptedContainer builds a runtime.Container as the reconciler would discover it,
|
||||
// stamping only the mulga.ecs.* labels that are non-empty.
|
||||
func adoptedContainer(taskID, name, cluster, credID, role, mac string, running bool) ctrruntime.Container {
|
||||
labels := map[string]string{
|
||||
labelTaskID: taskID,
|
||||
labelContainerName: name,
|
||||
labelClusterName: cluster,
|
||||
}
|
||||
if credID != "" {
|
||||
labels[labelCredID] = credID
|
||||
}
|
||||
if role != "" {
|
||||
labels[labelTaskRoleARN] = role
|
||||
}
|
||||
if mac != "" {
|
||||
labels[labelENIMac] = mac
|
||||
}
|
||||
return ctrruntime.Container{ID: containerID(taskID, name), Labels: labels, Running: running}
|
||||
}
|
||||
|
||||
func waitedFor(t *testing.T, rt *ctrruntime.FakePuller, id string) {
|
||||
t.Helper()
|
||||
deadline := time.After(time.Second)
|
||||
for !slices.Contains(rt.Waits(), id) {
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatalf("exit-wait never re-attached for %s; waited=%v", id, rt.Waits())
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A running, correctly-labeled container is re-adopted: its credentials are
|
||||
// re-registered, its RUNNING state refreshed, and its exit-wait re-attached.
|
||||
func TestReconcile_AdoptsRunningLabeledContainer(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{
|
||||
WaitErr: errors.New("blocked"),
|
||||
Containers: []ctrruntime.Container{
|
||||
adoptedContainer("t-001", "web", "default", "t-001", testTaskRole, "52:54:00:de:ad:01", true),
|
||||
},
|
||||
}
|
||||
a := newAgent(config{}, testIdentity(), cp, rt, nil)
|
||||
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
|
||||
|
||||
adopted := a.reconcile(context.Background())
|
||||
|
||||
if !adopted["t-001"] {
|
||||
t.Fatalf("t-001 not adopted; got %v", adopted)
|
||||
}
|
||||
if got := a.cred.roles["t-001"]; got != testTaskRole {
|
||||
t.Errorf("cred not re-registered: roles[t-001] = %q, want %q", got, testTaskRole)
|
||||
}
|
||||
if n := countStatus(cp.taskStates(), "t-001", bus.TaskStatusRunning); n != 1 {
|
||||
t.Errorf("RUNNING reports for t-001 = %d, want 1", n)
|
||||
}
|
||||
waitedFor(t, rt, containerID("t-001", "web"))
|
||||
}
|
||||
|
||||
// A task whose container died while the agent was down must be reported STOPPED,
|
||||
// not silently skipped. The exit-wait is cancelled on agent shutdown, so nothing
|
||||
// else ever reports that exit: skipping it leaves the gateway holding the task
|
||||
// RUNNING forever, its service never places a replacement, and the load
|
||||
// balancer target stays unhealthy with no path back.
|
||||
func TestReconcile_ReportsTaskWhoseContainerDied(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{
|
||||
Containers: []ctrruntime.Container{
|
||||
adoptedContainer("t-dead", "web", "default", "t-dead", testTaskRole, "52:54:00:de:ad:02", false),
|
||||
},
|
||||
}
|
||||
a := newAgent(config{}, testIdentity(), cp, rt, nil)
|
||||
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
|
||||
|
||||
adopted := a.reconcile(context.Background())
|
||||
|
||||
if n := countStatus(cp.taskStates(), "t-dead", bus.TaskStatusStopped); n != 1 {
|
||||
t.Errorf("STOPPED reports for t-dead = %d, want 1", n)
|
||||
}
|
||||
if n := countStatus(cp.taskStates(), "t-dead", bus.TaskStatusRunning); n != 0 {
|
||||
t.Errorf("t-dead must not be reported RUNNING, got %d such reports", n)
|
||||
}
|
||||
// Seeded so a re-delivered assignment for the dead task is acked, not re-run.
|
||||
if !adopted["t-dead"] {
|
||||
t.Errorf("t-dead not seeded into the adopted set; got %v", adopted)
|
||||
}
|
||||
if want := containerID("t-dead", "web"); !slices.Contains(rt.Removed, want) {
|
||||
t.Errorf("leftover container %s not removed; removed=%v", want, rt.Removed)
|
||||
}
|
||||
}
|
||||
|
||||
// A task with one live and one dead container is still running: the live half is
|
||||
// re-adopted and the re-attached exit-wait reports the dead one when it is
|
||||
// reaped, so reconcile must not declare the whole task STOPPED.
|
||||
func TestReconcile_PartiallyDeadTaskStaysRunning(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{
|
||||
WaitErr: errors.New("blocked"),
|
||||
Containers: []ctrruntime.Container{
|
||||
adoptedContainer("t-mixed", "web", "default", "t-mixed", testTaskRole, "52:54:00:de:ad:03", true),
|
||||
adoptedContainer("t-mixed", "sidecar", "default", "t-mixed", testTaskRole, "52:54:00:de:ad:03", false),
|
||||
},
|
||||
}
|
||||
a := newAgent(config{}, testIdentity(), cp, rt, nil)
|
||||
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
|
||||
|
||||
a.reconcile(context.Background())
|
||||
|
||||
if n := countStatus(cp.taskStates(), "t-mixed", bus.TaskStatusRunning); n != 1 {
|
||||
t.Errorf("RUNNING reports for t-mixed = %d, want 1", n)
|
||||
}
|
||||
if n := countStatus(cp.taskStates(), "t-mixed", bus.TaskStatusStopped); n != 0 {
|
||||
t.Errorf("t-mixed must not be reported STOPPED while a container still runs, got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// pollAssignments seeded with an adopted task acks its re-delivered assignment
|
||||
// but does not re-run it, while a genuinely new task still dispatches.
|
||||
func TestPollAssignments_SeededTaskNotRerun(t *testing.T) {
|
||||
mk := func(id string) bus.Assign {
|
||||
return bus.Assign{
|
||||
AccountID: "123456789012", ClusterName: "default", InstanceID: "i-abc123", TaskID: id,
|
||||
Containers: []bus.AssignContainer{{Name: "web", Image: "registry/web:1", Command: []string{"/bin/true"}}},
|
||||
}
|
||||
}
|
||||
adopted, fresh := mk("t-001"), mk("t-002")
|
||||
cp := &fakeCP{pollReplies: [][]bus.Assign{
|
||||
{adopted, fresh},
|
||||
{adopted, fresh},
|
||||
nil,
|
||||
}}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go a.pollAssignments(ctx, map[string]bool{"t-001": true})
|
||||
|
||||
deadline := time.After(time.Second)
|
||||
for countStatus(cp.taskStates(), "t-002", bus.TaskStatusRunning) < 1 {
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatal("fresh task t-002 was never dispatched")
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
cancel()
|
||||
|
||||
if n := countStatus(cp.taskStates(), "t-001", bus.TaskStatusRunning); n != 0 {
|
||||
t.Errorf("adopted task t-001 was re-run %d times, want 0", n)
|
||||
}
|
||||
if n := countStatus(cp.taskStates(), "t-002", bus.TaskStatusRunning); n != 1 {
|
||||
t.Errorf("fresh task t-002 dispatched %d times, want 1", n)
|
||||
}
|
||||
var t1Acked bool
|
||||
for _, ack := range cp.acks() {
|
||||
for _, id := range ack {
|
||||
if id == "t-001" {
|
||||
t1Acked = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !t1Acked {
|
||||
t.Errorf("adopted task t-001 was never acked; acks=%v", cp.acks())
|
||||
}
|
||||
}
|
||||
|
||||
// Containers belonging to another cluster are ignored entirely: this agent
|
||||
// speaks for its own cluster only, and must neither adopt nor report them.
|
||||
//
|
||||
// A stopped container of this cluster IS handled — see
|
||||
// TestReconcile_ReportsTaskWhoseContainerDied. It lands in the adopted set so
|
||||
// its re-delivered assignment is acked rather than re-run, after reconcile has
|
||||
// reported the task STOPPED.
|
||||
func TestReconcile_IgnoresOtherClusterContainers(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{
|
||||
WaitErr: errors.New("blocked"),
|
||||
Containers: []ctrruntime.Container{
|
||||
adoptedContainer("t-other", "web", "other", "", "", "", true),
|
||||
adoptedContainer("t-other-dead", "web", "other", "", "", "", false),
|
||||
adoptedContainer("t-ok", "web", "default", "", "", "", true),
|
||||
},
|
||||
}
|
||||
a := newAgent(config{}, testIdentity(), cp, rt, nil)
|
||||
|
||||
adopted := a.reconcile(context.Background())
|
||||
|
||||
if len(adopted) != 1 || !adopted["t-ok"] {
|
||||
t.Fatalf("adopted = %v, want only t-ok", adopted)
|
||||
}
|
||||
for _, id := range []string{"t-other", "t-other-dead"} {
|
||||
if n := countStatus(cp.taskStates(), id, bus.TaskStatusStopped); n != 0 {
|
||||
t.Errorf("reported %d STOPPED states for another cluster's task %s, want 0", n, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A nil runner or a List error degrades to an empty set without panicking.
|
||||
func TestReconcile_DegradesGracefully(t *testing.T) {
|
||||
noRunner := newAgent(config{}, testIdentity(), &fakeCP{}, nil, nil)
|
||||
if got := noRunner.reconcile(context.Background()); len(got) != 0 {
|
||||
t.Errorf("nil runner: adopted = %v, want empty", got)
|
||||
}
|
||||
|
||||
rt := &ctrruntime.FakePuller{ListErr: errors.New("boom")}
|
||||
listErr := newAgent(config{}, testIdentity(), &fakeCP{}, rt, nil)
|
||||
if got := listErr.reconcile(context.Background()); len(got) != 0 {
|
||||
t.Errorf("list error: adopted = %v, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
// taskLabels emits the optional cred/role/MAC labels only when set.
|
||||
func TestTaskLabels_OmitsEmptyOptionalLabels(t *testing.T) {
|
||||
full := taskLabels(&bus.Assign{
|
||||
TaskID: "t1", ClusterName: "c", CredID: "x", TaskRoleARN: testTaskRole, ENIMacAddress: "mac",
|
||||
}, "web")
|
||||
for k, want := range map[string]string{
|
||||
labelCredID: "x", labelTaskRoleARN: testTaskRole, labelENIMac: "mac",
|
||||
} {
|
||||
if full[k] != want {
|
||||
t.Errorf("full labels[%q] = %q, want %q", k, full[k], want)
|
||||
}
|
||||
}
|
||||
|
||||
bare := taskLabels(&bus.Assign{TaskID: "t1", ClusterName: "c"}, "web")
|
||||
if len(bare) != 3 {
|
||||
t.Errorf("bare labels = %v, want only taskID/containerName/clusterName", bare)
|
||||
}
|
||||
for _, k := range []string{labelCredID, labelTaskRoleARN, labelENIMac} {
|
||||
if _, ok := bare[k]; ok {
|
||||
t.Errorf("bare labels unexpectedly carry %q", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,78 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ecs"
|
||||
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
// identity is the container instance's stable identity, assembled at boot from
|
||||
// IMDS (account, instance, AZ) and config (cluster). It is the join key the
|
||||
// scheduler uses to track this instance.
|
||||
type identity struct {
|
||||
AccountID string
|
||||
ClusterName string
|
||||
InstanceID string
|
||||
AZ string
|
||||
InstanceType string
|
||||
Hostname string
|
||||
Capacity bus.InstanceCapacity
|
||||
AgentVersion string
|
||||
}
|
||||
|
||||
// instanceAttributes builds the built-in container-instance attributes from the
|
||||
// host's identity. Only attributes whose value was actually discovered are
|
||||
// reported, so an empty one is absent rather than present and blank.
|
||||
func instanceAttributes(id identity) []*ecs.Attribute {
|
||||
var attrs []*ecs.Attribute
|
||||
add := func(name, value string) {
|
||||
if value == "" {
|
||||
return
|
||||
}
|
||||
attrs = append(attrs, &ecs.Attribute{Name: aws.String(name), Value: aws.String(value)})
|
||||
}
|
||||
add(handlers_ecs.AttrInstanceType, id.InstanceType)
|
||||
add(handlers_ecs.AttrAvailabilityZone, id.AZ)
|
||||
return attrs
|
||||
}
|
||||
|
||||
// advertisedIdentity is the identity to register with, given whether a
|
||||
// container runtime is available. A host with no runtime can run nothing, so it
|
||||
// advertises no capacity: the scheduler places nothing on it, and the control
|
||||
// plane reads the zero as an involuntary drain. The identity is otherwise
|
||||
// unchanged, so the instance stays visible rather than disappearing.
|
||||
func advertisedIdentity(id identity, runtimeReady bool) identity {
|
||||
if runtimeReady {
|
||||
return id
|
||||
}
|
||||
id.Capacity = bus.InstanceCapacity{}
|
||||
return id
|
||||
}
|
||||
|
||||
// registrar registers the host as a container instance through the gateway.
|
||||
type registrar struct {
|
||||
cp controlPlane
|
||||
id identity
|
||||
// runtimeReady reports whether a container runtime is resolved. nil means
|
||||
// always ready, which is what a test with no gate wants.
|
||||
runtimeReady func() bool
|
||||
}
|
||||
|
||||
func newRegistrar(cp controlPlane, id identity, runtimeReady func() bool) *registrar {
|
||||
return ®istrar{cp: cp, id: id, runtimeReady: runtimeReady}
|
||||
}
|
||||
|
||||
// ready reports the runtime state, defaulting to ready when no gate is wired.
|
||||
func (r *registrar) ready() bool { return r.runtimeReady == nil || r.runtimeReady() }
|
||||
|
||||
// Register calls the gateway's RegisterContainerInstance. The scheduler records
|
||||
// (or refreshes) the container instance on receipt.
|
||||
func (r *registrar) Register() error {
|
||||
if err := r.cp.Register(advertisedIdentity(r.id, r.ready())); err != nil {
|
||||
return fmt.Errorf("register instance %s: %w", r.id.InstanceID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,257 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
// fakeCP is a test double for the gateway control-plane. It records registers,
|
||||
// task-state reports and GPU reports, and replays a scripted queue of poll
|
||||
// responses.
|
||||
type fakeCP struct {
|
||||
mu sync.Mutex
|
||||
|
||||
registers int
|
||||
lastRegistered identity
|
||||
states []bus.TaskState
|
||||
gpuReports []gpuReport
|
||||
pollAcks [][]string
|
||||
pollStopAcks [][]string
|
||||
pollReplies [][]bus.Assign
|
||||
stopReplies [][]bus.StopDirective
|
||||
stopPending []bus.StopDirective
|
||||
pollCalls int
|
||||
|
||||
registerErr bool
|
||||
submitErr bool
|
||||
}
|
||||
|
||||
// gpuReport is one recorded ReportTaskGPU call.
|
||||
type gpuReport struct {
|
||||
cluster string
|
||||
task string
|
||||
containers []handlers_ecs.ContainerGPUReport
|
||||
}
|
||||
|
||||
var _ controlPlane = (*fakeCP)(nil)
|
||||
|
||||
func (f *fakeCP) Register(id identity) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.registerErr {
|
||||
return errors.New("register boom")
|
||||
}
|
||||
f.registers++
|
||||
f.lastRegistered = id
|
||||
return nil
|
||||
}
|
||||
|
||||
// registered returns the identity of the most recent Register call.
|
||||
func (f *fakeCP) registered() identity {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.lastRegistered
|
||||
}
|
||||
|
||||
func (f *fakeCP) SubmitTaskState(st bus.TaskState) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.submitErr {
|
||||
return errors.New("submit boom")
|
||||
}
|
||||
f.states = append(f.states, st)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeCP) PollAssignments(_, _ string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.pollAcks = append(f.pollAcks, append([]string(nil), ackAssigns...))
|
||||
f.pollStopAcks = append(f.pollStopAcks, append([]string(nil), ackStops...))
|
||||
var out []bus.Assign
|
||||
if f.pollCalls < len(f.pollReplies) {
|
||||
out = f.pollReplies[f.pollCalls]
|
||||
}
|
||||
var stops []bus.StopDirective
|
||||
if f.pollCalls < len(f.stopReplies) {
|
||||
stops = f.stopReplies[f.pollCalls]
|
||||
}
|
||||
// stopPending is the durable inbox the gateway keeps: an acked directive is
|
||||
// deleted, and everything still in it is redelivered on every poll.
|
||||
f.stopPending = slices.DeleteFunc(f.stopPending, func(sd bus.StopDirective) bool {
|
||||
return slices.Contains(ackStops, sd.TaskID)
|
||||
})
|
||||
stops = slices.Concat(stops, f.stopPending)
|
||||
f.pollCalls++
|
||||
return out, stops, nil
|
||||
}
|
||||
|
||||
func (f *fakeCP) ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.gpuReports = append(f.gpuReports, gpuReport{cluster: cluster, task: task, containers: containers})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeCP) gpuReportsFor(taskID string) []gpuReport {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
var out []gpuReport
|
||||
for _, r := range f.gpuReports {
|
||||
if r.task == taskID {
|
||||
out = append(out, r)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (f *fakeCP) registerCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.registers
|
||||
}
|
||||
|
||||
func (f *fakeCP) taskStates() []bus.TaskState {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]bus.TaskState(nil), f.states...)
|
||||
}
|
||||
|
||||
func (f *fakeCP) acks() [][]string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([][]string(nil), f.pollAcks...)
|
||||
}
|
||||
|
||||
func (f *fakeCP) stopAcks() [][]string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([][]string(nil), f.pollStopAcks...)
|
||||
}
|
||||
|
||||
func testIdentity() identity {
|
||||
return identity{
|
||||
AccountID: "123456789012",
|
||||
ClusterName: "default",
|
||||
InstanceID: "i-abc123",
|
||||
AZ: "us-east-1a",
|
||||
Hostname: "ecs-host",
|
||||
Capacity: bus.InstanceCapacity{CPU: 2048, MemoryMiB: 4096},
|
||||
AgentVersion: "test",
|
||||
}
|
||||
}
|
||||
|
||||
// testIdentityWithGPUs is testIdentity seeded with discovered GPU UUIDs, so
|
||||
// newAgent's ledger has real devices to pin from.
|
||||
func testIdentityWithGPUs(uuids ...string) identity {
|
||||
id := testIdentity()
|
||||
id.Capacity.GPU = len(uuids)
|
||||
id.Capacity.GPUIDs = uuids
|
||||
return id
|
||||
}
|
||||
|
||||
func TestRegistrar_Register(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
if err := newRegistrar(cp, testIdentity(), nil).Register(); err != nil {
|
||||
t.Fatalf("Register: %v", err)
|
||||
}
|
||||
if cp.registerCount() != 1 {
|
||||
t.Errorf("registers = %d, want 1", cp.registerCount())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistrar_RegisterError(t *testing.T) {
|
||||
cp := &fakeCP{registerErr: true}
|
||||
if err := newRegistrar(cp, testIdentity(), nil).Register(); err == nil {
|
||||
t.Fatal("expected register error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeartbeater_Beat(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
if err := newHeartbeater(cp, testIdentity(), time.Second, nil).beat(); err != nil {
|
||||
t.Fatalf("beat: %v", err)
|
||||
}
|
||||
if cp.registerCount() != 1 {
|
||||
t.Errorf("beat should re-register once, got %d", cp.registerCount())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeartbeater_DefaultInterval(t *testing.T) {
|
||||
h := newHeartbeater(&fakeCP{}, testIdentity(), 0, nil)
|
||||
if h.interval != defaultHeartbeat {
|
||||
t.Errorf("interval = %v, want default", h.interval)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeartbeater_RunStopsOnContext(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
h := newHeartbeater(cp, testIdentity(), 5*time.Millisecond, nil)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() { h.Run(ctx); close(done) }()
|
||||
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
cancel()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("Run did not stop on context cancel")
|
||||
}
|
||||
if cp.registerCount() == 0 {
|
||||
t.Error("expected at least one re-register before cancel")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRegistrar_NoRuntimeAdvertisesNoCapacity is the placement half of the boot
|
||||
// bug: an agent with no container runtime kept registering its full CPU and
|
||||
// memory, so the scheduler placed tasks it could only fail to start. Capacity is
|
||||
// advertised only while a runtime is there to use it.
|
||||
func TestRegistrar_NoRuntimeAdvertisesNoCapacity(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
ready := false
|
||||
r := newRegistrar(cp, testIdentity(), func() bool { return ready })
|
||||
|
||||
if err := r.Register(); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
if got := cp.registered().Capacity; got.CPU != 0 || got.MemoryMiB != 0 {
|
||||
t.Fatalf("advertised capacity with no runtime: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
|
||||
}
|
||||
|
||||
ready = true
|
||||
if err := r.Register(); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
want := testIdentity().Capacity
|
||||
if got := cp.registered().Capacity; got.CPU != want.CPU || got.MemoryMiB != want.MemoryMiB {
|
||||
t.Fatalf("capacity not restored once the runtime resolved: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeater_NoRuntimeAdvertisesNoCapacity is the same rule on the beat,
|
||||
// which is how a runtime lost after boot reaches the control plane.
|
||||
func TestHeartbeater_NoRuntimeAdvertisesNoCapacity(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
h := newHeartbeater(cp, testIdentity(), time.Second, func() bool { return false })
|
||||
|
||||
if err := h.beat(); err != nil {
|
||||
t.Fatalf("beat: %v", err)
|
||||
}
|
||||
if got := cp.registered().Capacity; got.CPU != 0 || got.MemoryMiB != 0 {
|
||||
t.Fatalf("beat advertised capacity with no runtime: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
|
||||
}
|
||||
// The instance still identifies itself, so it stays visible as drained rather
|
||||
// than vanishing from the cluster.
|
||||
if cp.registered().InstanceID != testIdentity().InstanceID {
|
||||
t.Error("beat dropped the instance identity")
|
||||
}
|
||||
}
|
||||
@@ -1,124 +0,0 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
containerd "github.com/containerd/containerd/v2/client"
|
||||
"github.com/containerd/containerd/v2/core/remotes"
|
||||
"github.com/containerd/containerd/v2/core/remotes/docker"
|
||||
"github.com/containerd/containerd/v2/pkg/namespaces"
|
||||
"github.com/distribution/reference"
|
||||
)
|
||||
|
||||
// ecsNamespace isolates ecs-agent images/containers in containerd, mirroring the
|
||||
// "moby"/"k8s.io" convention other runtimes use.
|
||||
const ecsNamespace = "ecs"
|
||||
|
||||
// servingProbeTimeout bounds the boot-time daemon liveness check.
|
||||
const servingProbeTimeout = 3 * time.Second
|
||||
|
||||
// containerdPuller drives a real containerd daemon over its unix socket.
|
||||
type containerdPuller struct {
|
||||
client *containerd.Client
|
||||
|
||||
// stdinDone holds one channel per interactive container, closed when that
|
||||
// container is stopped or removed so its stdin reader returns EOF.
|
||||
mu sync.Mutex
|
||||
stdinDone map[string]chan struct{}
|
||||
}
|
||||
|
||||
var _ ImagePuller = (*containerdPuller)(nil)
|
||||
|
||||
// New connects to the containerd daemon at socket (e.g.
|
||||
// "/run/containerd/containerd.sock") and returns an ImagePuller. The containerd
|
||||
// client dials lazily, so New actively probes the daemon with IsServing and
|
||||
// returns an error if it is unreachable — otherwise a dead socket would only
|
||||
// surface on the first pull, and the agent's boot-time "containerd unavailable"
|
||||
// log would never fire.
|
||||
func New(socket string) (Runtime, error) {
|
||||
client, err := containerd.New(socket, containerd.WithDefaultNamespace(ecsNamespace))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("connect containerd %s: %w", socket, err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), servingProbeTimeout)
|
||||
defer cancel()
|
||||
serving, err := client.IsServing(ctx)
|
||||
if err != nil || !serving {
|
||||
_ = client.Close()
|
||||
return nil, fmt.Errorf("containerd %s not serving: %w", socket, err)
|
||||
}
|
||||
return &containerdPuller{client: client}, nil
|
||||
}
|
||||
|
||||
// Pull resolves credentials via r, then pulls and unpacks the image.
|
||||
func (p *containerdPuller) Pull(ctx context.Context, spec PullSpec, r Resolver) (Image, error) {
|
||||
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
|
||||
|
||||
ref, err := normalizeRef(spec.Ref)
|
||||
if err != nil {
|
||||
return Image{}, err
|
||||
}
|
||||
|
||||
resolver, err := newDockerResolver(ctx, ref, r)
|
||||
if err != nil {
|
||||
return Image{}, err
|
||||
}
|
||||
|
||||
img, err := p.client.Pull(ctx, ref,
|
||||
containerd.WithPullUnpack,
|
||||
containerd.WithResolver(resolver),
|
||||
)
|
||||
if err != nil {
|
||||
return Image{}, fmt.Errorf("pull %s: %w", ref, err)
|
||||
}
|
||||
|
||||
target := img.Target()
|
||||
return Image{
|
||||
Ref: ref,
|
||||
Digest: target.Digest.String(),
|
||||
Size: target.Size,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// normalizeRef expands a task-definition image string into the fully-qualified
|
||||
// reference containerd's resolver requires: a bare "nginx:alpine" becomes
|
||||
// "docker.io/library/nginx:alpine" and an untagged ref gets ":latest". Without
|
||||
// this, containerd parses the bare ref as "dummy://nginx:alpine" and rejects it.
|
||||
func normalizeRef(ref string) (string, error) {
|
||||
named, err := reference.ParseNormalizedNamed(ref)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("parse image ref %q: %w", ref, err)
|
||||
}
|
||||
return reference.TagNameOnly(named).String(), nil
|
||||
}
|
||||
|
||||
// Close releases the containerd client connection.
|
||||
func (p *containerdPuller) Close() error {
|
||||
if p.client == nil {
|
||||
return nil
|
||||
}
|
||||
return p.client.Close()
|
||||
}
|
||||
|
||||
// newDockerResolver builds a containerd remote resolver whose per-host
|
||||
// credentials come from r (the ECR token resolver). Credentials are fetched once
|
||||
// for ref and reused for every host containerd asks about during the pull.
|
||||
func newDockerResolver(ctx context.Context, ref string, r Resolver) (remotes.Resolver, error) {
|
||||
creds := func(string) (string, string, error) { return "", "", nil }
|
||||
if r != nil {
|
||||
user, pass, _, err := r.Authorize(ctx, ref)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("authorize %s: %w", ref, err)
|
||||
}
|
||||
creds = func(string) (string, string, error) { return user, pass, nil }
|
||||
}
|
||||
|
||||
authorizer := docker.NewDockerAuthorizer(docker.WithAuthCreds(creds))
|
||||
return docker.NewResolver(docker.ResolverOptions{
|
||||
Hosts: docker.ConfigureDefaultRegistries(docker.WithAuthorizer(authorizer)),
|
||||
}), nil
|
||||
}
|
||||
@@ -1,326 +0,0 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
containerd "github.com/containerd/containerd/v2/client"
|
||||
"github.com/containerd/containerd/v2/core/containers"
|
||||
"github.com/containerd/containerd/v2/pkg/cdi"
|
||||
"github.com/containerd/containerd/v2/pkg/cio"
|
||||
"github.com/containerd/containerd/v2/pkg/namespaces"
|
||||
"github.com/containerd/containerd/v2/pkg/oci"
|
||||
specs "github.com/opencontainers/runtime-spec/specs-go"
|
||||
)
|
||||
|
||||
// cdiVendorGPU is the NVIDIA CDI vendor/class prefix for whole-GPU devices, per
|
||||
// the NVIDIA CDI naming convention (nvidia.com/gpu=<uuid>).
|
||||
const cdiVendorGPU = "nvidia.com/gpu"
|
||||
|
||||
// cdiDeviceNames maps a container's pinned GPU device UUIDs to CDI device
|
||||
// names. Empty input (non-GPU container) returns nil so no CDI opt is added.
|
||||
func cdiDeviceNames(uuids []string) []string {
|
||||
if len(uuids) == 0 {
|
||||
return nil
|
||||
}
|
||||
names := make([]string, len(uuids))
|
||||
for i, id := range uuids {
|
||||
names[i] = cdiVendorGPU + "=" + id
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
// cdiSpecOpts returns the oci.SpecOpts needed to inject gpuIDs (via
|
||||
// containerd's CDI machinery) as devices into a container's OCI spec. It
|
||||
// returns nil for a non-GPU container so Run adds nothing to the opts list.
|
||||
func cdiSpecOpts(gpuIDs []string) []oci.SpecOpts {
|
||||
devices := cdiDeviceNames(gpuIDs)
|
||||
if len(devices) == 0 {
|
||||
return nil
|
||||
}
|
||||
return []oci.SpecOpts{cdi.WithCDIDevices(devices...)}
|
||||
}
|
||||
|
||||
// ociCapName renders an ECS Linux capability (Docker's bare name, e.g.
|
||||
// "SYS_ADMIN") as the OCI runtime-spec name ("CAP_SYS_ADMIN") capabilities.Add/
|
||||
// Drop expect.
|
||||
func ociCapName(name string) string {
|
||||
if strings.HasPrefix(name, "CAP_") {
|
||||
return name
|
||||
}
|
||||
return "CAP_" + name
|
||||
}
|
||||
|
||||
// ociCapNames maps a list of ECS capability names to their OCI form.
|
||||
func ociCapNames(names []string) []string {
|
||||
if len(names) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, len(names))
|
||||
for i, n := range names {
|
||||
out[i] = ociCapName(n)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// withSysctls sets Linux.Sysctl on the OCI spec from systemControls namespace/
|
||||
// value pairs. containerd has no built-in SpecOpts for sysctls.
|
||||
func withSysctls(ctls []SystemControl) oci.SpecOpts {
|
||||
return func(_ context.Context, _ oci.Client, _ *containers.Container, s *oci.Spec) error {
|
||||
if len(ctls) == 0 {
|
||||
return nil
|
||||
}
|
||||
if s.Linux == nil {
|
||||
s.Linux = &specs.Linux{}
|
||||
}
|
||||
if s.Linux.Sysctl == nil {
|
||||
s.Linux.Sysctl = map[string]string{}
|
||||
}
|
||||
for _, c := range ctls {
|
||||
s.Linux.Sysctl[c.Namespace] = c.Value
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// heldStdin backs an interactive container's stdin: reads block instead of
|
||||
// hitting an immediate EOF, which is the "stdin held open" behaviour
|
||||
// interactive=true asks for, until done closes and the read reports EOF.
|
||||
type heldStdin struct{ done <-chan struct{} }
|
||||
|
||||
func (h heldStdin) Read([]byte) (int, error) {
|
||||
<-h.done
|
||||
return 0, io.EOF
|
||||
}
|
||||
|
||||
// ioStreamOpts describes the task IO for a container. A terminal container needs
|
||||
// a terminal FIFO set: oci.WithTTY alone puts Terminal in the OCI spec while the
|
||||
// shim creates no console socket, and runc refuses that pair outright. A
|
||||
// terminal carries no separate stderr, so that stream stays nil and the shim
|
||||
// merges it into the console.
|
||||
func ioStreamOpts(stdin io.Reader, terminal bool) []cio.Opt {
|
||||
if terminal {
|
||||
return []cio.Opt{cio.WithStreams(stdin, io.Discard, nil), cio.WithTerminal}
|
||||
}
|
||||
return []cio.Opt{cio.WithStreams(stdin, io.Discard, io.Discard)}
|
||||
}
|
||||
|
||||
// containerIOCreator returns cio.NullIO for a container needing neither stdin
|
||||
// nor a terminal, else a creator carrying whichever it asked for. An interactive
|
||||
// container's stdin is held open until releaseStdin; containerd copies stdin on
|
||||
// a goroutine that exits only once the read returns, so a reader that never
|
||||
// returned would park it for the life of the agent. A terminal container that is
|
||||
// not interactive gets no stdin stream at all, which is what a nil reader means.
|
||||
func (p *containerdPuller) containerIOCreator(containerID string, interactive, terminal bool) cio.Creator {
|
||||
if !interactive && !terminal {
|
||||
return cio.NullIO
|
||||
}
|
||||
var stdin io.Reader
|
||||
if interactive {
|
||||
done := make(chan struct{})
|
||||
p.mu.Lock()
|
||||
if p.stdinDone == nil {
|
||||
p.stdinDone = make(map[string]chan struct{})
|
||||
}
|
||||
p.stdinDone[containerID] = done
|
||||
p.mu.Unlock()
|
||||
stdin = heldStdin{done: done}
|
||||
}
|
||||
return cio.NewCreator(ioStreamOpts(stdin, terminal)...)
|
||||
}
|
||||
|
||||
// releaseStdin closes a held stdin so containerd's copier goroutine exits. Safe
|
||||
// for a container that never held one, and safe to call more than once.
|
||||
func (p *containerdPuller) releaseStdin(containerID string) {
|
||||
p.mu.Lock()
|
||||
done, ok := p.stdinDone[containerID]
|
||||
delete(p.stdinDone, containerID)
|
||||
p.mu.Unlock()
|
||||
if ok {
|
||||
close(done)
|
||||
}
|
||||
}
|
||||
|
||||
var _ Runner = (*containerdPuller)(nil)
|
||||
|
||||
// Run creates a container from an already-pulled image and starts its task. The
|
||||
// container joins the host network namespace (bridge/CNI task networking is a
|
||||
// later sprint) and carries spec.Labels for the reboot reconciler. id must be
|
||||
// unique within the "ecs" namespace.
|
||||
func (p *containerdPuller) Run(ctx context.Context, id string, spec RunSpec) (string, error) {
|
||||
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
|
||||
|
||||
ref, err := normalizeRef(spec.Image)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
image, err := p.client.GetImage(ctx, ref)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("get image %s: %w", ref, err)
|
||||
}
|
||||
|
||||
specOpts := []oci.SpecOpts{
|
||||
oci.WithImageConfig(image),
|
||||
oci.WithHostHostsFile,
|
||||
oci.WithHostResolvconf,
|
||||
}
|
||||
if spec.NetnsPath != "" {
|
||||
// awsvpc: join the task ENI netns built by the agent.
|
||||
specOpts = append(specOpts, oci.WithLinuxNamespace(specs.LinuxNamespace{
|
||||
Type: specs.NetworkNamespace,
|
||||
Path: spec.NetnsPath,
|
||||
}))
|
||||
} else {
|
||||
// bridge/host: share the VM (host) netns.
|
||||
specOpts = append(specOpts, oci.WithHostNamespace(specs.NetworkNamespace))
|
||||
}
|
||||
if len(spec.Command) > 0 {
|
||||
specOpts = append(specOpts, oci.WithProcessArgs(spec.Command...))
|
||||
}
|
||||
if len(spec.Env) > 0 {
|
||||
specOpts = append(specOpts, oci.WithEnv(envSlice(spec.Env)))
|
||||
}
|
||||
specOpts = append(specOpts, cdiSpecOpts(spec.GPUIDs)...)
|
||||
if spec.User != "" {
|
||||
specOpts = append(specOpts, oci.WithUser(spec.User))
|
||||
}
|
||||
if spec.ReadonlyRootFilesystem != nil && *spec.ReadonlyRootFilesystem {
|
||||
specOpts = append(specOpts, oci.WithRootFSReadonly())
|
||||
}
|
||||
if spec.Privileged != nil && *spec.Privileged {
|
||||
specOpts = append(specOpts, oci.WithPrivileged)
|
||||
}
|
||||
terminal := spec.PseudoTerminal != nil && *spec.PseudoTerminal
|
||||
if terminal {
|
||||
specOpts = append(specOpts, oci.WithTTY)
|
||||
}
|
||||
specOpts = append(specOpts, withSysctls(spec.SystemControls))
|
||||
if len(spec.CapAdd) > 0 {
|
||||
specOpts = append(specOpts, oci.WithAddedCapabilities(ociCapNames(spec.CapAdd)))
|
||||
}
|
||||
if len(spec.CapDrop) > 0 {
|
||||
specOpts = append(specOpts, oci.WithDroppedCapabilities(ociCapNames(spec.CapDrop)))
|
||||
}
|
||||
|
||||
container, err := p.client.NewContainer(ctx, id,
|
||||
containerd.WithNewSnapshot(id+"-snapshot", image),
|
||||
containerd.WithNewSpec(specOpts...),
|
||||
containerd.WithContainerLabels(spec.Labels),
|
||||
)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create container %s: %w", id, err)
|
||||
}
|
||||
|
||||
interactive := spec.Interactive != nil && *spec.Interactive
|
||||
task, err := container.NewTask(ctx, p.containerIOCreator(id, interactive, terminal))
|
||||
if err != nil {
|
||||
p.releaseStdin(id)
|
||||
_ = container.Delete(ctx, containerd.WithSnapshotCleanup)
|
||||
return "", fmt.Errorf("create task %s: %w", id, err)
|
||||
}
|
||||
if err := task.Start(ctx); err != nil {
|
||||
p.releaseStdin(id)
|
||||
_, _ = task.Delete(ctx)
|
||||
_ = container.Delete(ctx, containerd.WithSnapshotCleanup)
|
||||
return "", fmt.Errorf("start task %s: %w", id, err)
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// Wait blocks until the container's task exits and returns its exit code.
|
||||
func (p *containerdPuller) Wait(ctx context.Context, containerID string) (RunStatus, error) {
|
||||
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
|
||||
container, err := p.client.LoadContainer(ctx, containerID)
|
||||
if err != nil {
|
||||
return RunStatus{}, fmt.Errorf("load container %s: %w", containerID, err)
|
||||
}
|
||||
task, err := container.Task(ctx, nil)
|
||||
if err != nil {
|
||||
return RunStatus{}, fmt.Errorf("attach task %s: %w", containerID, err)
|
||||
}
|
||||
statusC, err := task.Wait(ctx)
|
||||
if err != nil {
|
||||
return RunStatus{}, fmt.Errorf("wait task %s: %w", containerID, err)
|
||||
}
|
||||
status := <-statusC
|
||||
code, _, err := status.Result()
|
||||
return RunStatus{ExitCode: int(code)}, err
|
||||
}
|
||||
|
||||
// List enumerates every container in the ecs namespace with its labels and
|
||||
// whether its task is running, so the agent can re-adopt live containers after a
|
||||
// restart. Per-container label/task errors are treated as "not running" rather
|
||||
// than failing the whole list.
|
||||
func (p *containerdPuller) List(ctx context.Context) ([]Container, error) {
|
||||
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
|
||||
containers, err := p.client.Containers(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list containers: %w", err)
|
||||
}
|
||||
out := make([]Container, 0, len(containers))
|
||||
for _, c := range containers {
|
||||
labels, _ := c.Labels(ctx)
|
||||
running := false
|
||||
if task, terr := c.Task(ctx, nil); terr == nil {
|
||||
if status, serr := task.Status(ctx); serr == nil {
|
||||
running = status.Status == containerd.Running
|
||||
}
|
||||
}
|
||||
out = append(out, Container{ID: c.ID(), Labels: labels, Running: running})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Stop enforces a container's stopTimeout: SIGTERM the task, wait up to
|
||||
// timeout for it to exit on its own, then fall through to Remove's forced
|
||||
// kill+delete regardless of whether it exited in time. A container with no
|
||||
// live task (already gone) or that fails to accept the signal goes straight to
|
||||
// Remove, matching Remove's own already-gone tolerance.
|
||||
func (p *containerdPuller) Stop(ctx context.Context, containerID string, timeout time.Duration) error {
|
||||
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
|
||||
container, err := p.client.LoadContainer(ctx, containerID)
|
||||
if err != nil {
|
||||
return nil //nolint:nilerr // load failure means the container is already gone
|
||||
}
|
||||
task, terr := container.Task(ctx, nil)
|
||||
if terr != nil {
|
||||
return p.Remove(ctx, containerID)
|
||||
}
|
||||
statusC, werr := task.Wait(ctx)
|
||||
if werr == nil && task.Kill(ctx, syscall.SIGTERM) == nil {
|
||||
select {
|
||||
case <-statusC:
|
||||
case <-time.After(timeout):
|
||||
}
|
||||
}
|
||||
return p.Remove(ctx, containerID)
|
||||
}
|
||||
|
||||
// Remove kills and deletes the container's task, then the container + snapshot.
|
||||
func (p *containerdPuller) Remove(ctx context.Context, containerID string) error {
|
||||
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
|
||||
p.releaseStdin(containerID)
|
||||
container, err := p.client.LoadContainer(ctx, containerID)
|
||||
if err != nil {
|
||||
return nil //nolint:nilerr // load failure means the container is already gone
|
||||
}
|
||||
if task, terr := container.Task(ctx, nil); terr == nil {
|
||||
_, _ = task.Delete(ctx, containerd.WithProcessKill)
|
||||
}
|
||||
return container.Delete(ctx, containerd.WithSnapshotCleanup)
|
||||
}
|
||||
|
||||
// envSlice renders an env map as sorted "K=V" entries for deterministic specs.
|
||||
func envSlice(env map[string]string) []string {
|
||||
out := make([]string, 0, len(env))
|
||||
for k, v := range env {
|
||||
out = append(out, k+"="+v)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
@@ -1,217 +0,0 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/containerd/containerd/v2/pkg/cio"
|
||||
specs "github.com/opencontainers/runtime-spec/specs-go"
|
||||
)
|
||||
|
||||
// cdiDeviceNames maps a container's pinned GPU UUIDs to CDI device names
|
||||
// (nvidia.com/gpu=<uuid>); Run wires the result into oci.WithCDIDevices when
|
||||
// non-empty, so this is the injection logic's unit-testable core.
|
||||
func TestCDIDeviceNames(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
uuids []string
|
||||
want []string
|
||||
}{
|
||||
{name: "empty", uuids: nil, want: nil},
|
||||
{name: "single", uuids: []string{"GPU-aaaaaaaa-1111-2222-3333-444444444444"},
|
||||
want: []string{"nvidia.com/gpu=GPU-aaaaaaaa-1111-2222-3333-444444444444"}},
|
||||
{name: "multiple preserves order", uuids: []string{"GPU-aaa", "GPU-bbb"},
|
||||
want: []string{"nvidia.com/gpu=GPU-aaa", "nvidia.com/gpu=GPU-bbb"}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := cdiDeviceNames(tc.uuids)
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("cdiDeviceNames(%v) = %v, want %v", tc.uuids, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestCDISpecOpts_Wiring confirms Run's specOpts list picks up exactly one CDI
|
||||
// opt when GPUIDs is non-empty, and none for a non-GPU container — the wiring
|
||||
// Run relies on (a live oci.Spec is needed to unit test the opt's effect).
|
||||
func TestCDISpecOpts_Wiring(t *testing.T) {
|
||||
if opts := cdiSpecOpts(nil); len(opts) != 0 {
|
||||
t.Errorf("non-GPU container: want no spec opts, got %d", len(opts))
|
||||
}
|
||||
opts := cdiSpecOpts([]string{"GPU-aaa", "GPU-bbb"})
|
||||
if len(opts) != 1 {
|
||||
t.Fatalf("GPU container: want exactly one CDI spec opt, got %d", len(opts))
|
||||
}
|
||||
if opts[0] == nil {
|
||||
t.Error("GPU container: spec opt is nil")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOCICapNames verifies ECS's bare capability names (Docker convention,
|
||||
// e.g. "SYS_PTRACE") are prefixed to the OCI runtime-spec form the process
|
||||
// capability sets expect, and a name already carrying the prefix passes
|
||||
// through unchanged.
|
||||
func TestOCICapNames(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
in []string
|
||||
want []string
|
||||
}{
|
||||
{name: "empty", in: nil, want: nil},
|
||||
{name: "bare", in: []string{"SYS_PTRACE"}, want: []string{"CAP_SYS_PTRACE"}},
|
||||
{name: "already prefixed", in: []string{"CAP_NET_ADMIN"}, want: []string{"CAP_NET_ADMIN"}},
|
||||
{name: "mixed", in: []string{"SYS_PTRACE", "CAP_NET_ADMIN"}, want: []string{"CAP_SYS_PTRACE", "CAP_NET_ADMIN"}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := ociCapNames(tc.in)
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("ociCapNames(%v) = %v, want %v", tc.in, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestWithSysctls verifies systemControls namespace/value pairs land on
|
||||
// Linux.Sysctl, and an empty list leaves the spec's Linux section untouched
|
||||
// rather than allocating an empty map (the "requests nothing" case).
|
||||
func TestWithSysctls(t *testing.T) {
|
||||
s := &specs.Spec{}
|
||||
if err := withSysctls(nil)(context.Background(), nil, nil, s); err != nil {
|
||||
t.Fatalf("empty ctls: %v", err)
|
||||
}
|
||||
if s.Linux != nil {
|
||||
t.Errorf("empty ctls: want no Linux section, got %+v", s.Linux)
|
||||
}
|
||||
|
||||
ctls := []SystemControl{{Namespace: "net.core.somaxconn", Value: "1024"}}
|
||||
if err := withSysctls(ctls)(context.Background(), nil, nil, s); err != nil {
|
||||
t.Fatalf("with ctls: %v", err)
|
||||
}
|
||||
if s.Linux == nil || s.Linux.Sysctl["net.core.somaxconn"] != "1024" {
|
||||
t.Errorf("want sysctl net.core.somaxconn=1024, got %+v", s.Linux)
|
||||
}
|
||||
}
|
||||
|
||||
// An interactive container's stdin read blocks until the container is released,
|
||||
// then reports EOF so containerd's copier goroutine exits instead of parking
|
||||
// for the life of the agent.
|
||||
func TestHeldStdin_ReadsEOFOnceReleased(t *testing.T) {
|
||||
p := &containerdPuller{}
|
||||
p.containerIOCreator("c1", true, false)
|
||||
|
||||
p.mu.Lock()
|
||||
done := p.stdinDone["c1"]
|
||||
p.mu.Unlock()
|
||||
if done == nil {
|
||||
t.Fatal("interactive container registered no stdin channel")
|
||||
}
|
||||
|
||||
read := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := heldStdin{done: done}.Read(make([]byte, 1))
|
||||
read <- err
|
||||
}()
|
||||
|
||||
select {
|
||||
case err := <-read:
|
||||
t.Fatalf("read returned before release: %v", err)
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
}
|
||||
|
||||
p.releaseStdin("c1")
|
||||
|
||||
select {
|
||||
case err := <-read:
|
||||
if !errors.Is(err, io.EOF) {
|
||||
t.Fatalf("want io.EOF after release, got %v", err)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("read did not return after release")
|
||||
}
|
||||
}
|
||||
|
||||
// A non-interactive container uses cio.NullIO and registers nothing to release.
|
||||
func TestContainerIOCreator_NonInteractiveRegistersNothing(t *testing.T) {
|
||||
p := &containerdPuller{}
|
||||
p.containerIOCreator("c1", false, false)
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if len(p.stdinDone) != 0 {
|
||||
t.Fatalf("want no registered stdin channels, got %d", len(p.stdinDone))
|
||||
}
|
||||
}
|
||||
|
||||
// applyIOOpts resolves the cio options onto a Streams the way cio.NewCreator
|
||||
// does, which is the only way to observe what the creator was built with.
|
||||
func applyIOOpts(opts []cio.Opt) cio.Streams {
|
||||
var s cio.Streams
|
||||
for _, o := range opts {
|
||||
o(&s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// A terminal container needs the terminal flag set on its FIFO set, else the
|
||||
// shim creates no console socket and runc refuses to start the container at all.
|
||||
// The terminal carries no separate stderr stream.
|
||||
func TestIOStreamOpts_TerminalRequestsAConsole(t *testing.T) {
|
||||
s := applyIOOpts(ioStreamOpts(nil, true))
|
||||
if !s.Terminal {
|
||||
t.Error("terminal container: want Terminal set")
|
||||
}
|
||||
if s.Stderr != nil {
|
||||
t.Errorf("terminal container: want no stderr stream, got %#v", s.Stderr)
|
||||
}
|
||||
if s.Stdout == nil {
|
||||
t.Error("terminal container: want a stdout stream")
|
||||
}
|
||||
}
|
||||
|
||||
// A container with stdin but no terminal keeps both output streams and asks for
|
||||
// no console.
|
||||
func TestIOStreamOpts_NonTerminalKeepsStderr(t *testing.T) {
|
||||
s := applyIOOpts(ioStreamOpts(heldStdin{}, false))
|
||||
if s.Terminal {
|
||||
t.Error("non-terminal container: want Terminal unset")
|
||||
}
|
||||
if s.Stderr == nil {
|
||||
t.Error("non-terminal container: want a stderr stream")
|
||||
}
|
||||
if s.Stdin == nil {
|
||||
t.Error("non-terminal interactive container: want a stdin stream")
|
||||
}
|
||||
}
|
||||
|
||||
// A terminal container that is not interactive still gets a creator rather than
|
||||
// cio.NullIO, and registers no stdin to release.
|
||||
func TestContainerIOCreator_TerminalWithoutInteractive(t *testing.T) {
|
||||
p := &containerdPuller{}
|
||||
if c := p.containerIOCreator("c1", false, true); c == nil {
|
||||
t.Fatal("terminal container: want a creator")
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if len(p.stdinDone) != 0 {
|
||||
t.Fatalf("want no registered stdin channels, got %d", len(p.stdinDone))
|
||||
}
|
||||
}
|
||||
|
||||
// Remove calls releaseStdin unconditionally and Stop falls through to Remove,
|
||||
// so an unknown container and a second release must both be no-ops.
|
||||
func TestReleaseStdin_UnknownAndRepeatedAreSafe(t *testing.T) {
|
||||
p := &containerdPuller{}
|
||||
p.releaseStdin("never-registered")
|
||||
|
||||
p.containerIOCreator("c1", true, false)
|
||||
p.releaseStdin("c1")
|
||||
p.releaseStdin("c1")
|
||||
}
|
||||
@@ -1,117 +0,0 @@
|
||||
// Package runtime is the ecs-agent's container-runtime seam. The agent pulls
|
||||
// task images and (from Sprint 4e) runs containers through an ImagePuller; the
|
||||
// real implementation drives containerd, but the interface lets tests and the
|
||||
// scheduler-only build run against a fake.
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PullSpec describes one image to pull.
|
||||
type PullSpec struct {
|
||||
// Ref is the full image reference, e.g.
|
||||
// "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:latest".
|
||||
Ref string
|
||||
}
|
||||
|
||||
// Image identifies a pulled image in the runtime's content store.
|
||||
type Image struct {
|
||||
Ref string
|
||||
Digest string
|
||||
Size int64
|
||||
}
|
||||
|
||||
// Resolver supplies registry credentials for a given image reference. The
|
||||
// ecs-agent backs this with ECR GetAuthorizationToken (see ecrresolver.go).
|
||||
type Resolver interface {
|
||||
// Authorize returns the username, password and registry endpoint to use for
|
||||
// ref. An empty user/pass means anonymous pull.
|
||||
Authorize(ctx context.Context, ref string) (user, pass, endpoint string, err error)
|
||||
}
|
||||
|
||||
// ImagePuller pulls images into the local runtime. Close releases the runtime
|
||||
// client connection.
|
||||
type ImagePuller interface {
|
||||
Pull(ctx context.Context, spec PullSpec, r Resolver) (Image, error)
|
||||
Close() error
|
||||
}
|
||||
|
||||
// SystemControl is a sysctl namespace/value pair applied to the OCI spec.
|
||||
type SystemControl struct {
|
||||
Namespace string
|
||||
Value string
|
||||
}
|
||||
|
||||
// RunSpec describes a single container the agent must create and start. v1 runs
|
||||
// the container in the host network namespace (bridge/CNI task networking lands
|
||||
// in a later sprint); Labels carry the mulga.ecs.* task identity for the reboot
|
||||
// reconciler.
|
||||
type RunSpec struct {
|
||||
Image string
|
||||
Command []string
|
||||
Env map[string]string
|
||||
Labels map[string]string
|
||||
// NetnsPath, when set, joins the container to a pre-built network namespace
|
||||
// (awsvpc task ENI). Empty keeps the container in the host (VM) netns —
|
||||
// bridge/host mode behaviour.
|
||||
NetnsPath string
|
||||
// GPU is the whole-GPU count requested for this container.
|
||||
GPU int
|
||||
// GPUIDs are the ledger-pinned device UUIDs assigned to this container, one
|
||||
// per GPU requested. Empty for non-GPU containers or if pinning fell short.
|
||||
// The runner injects these as CDI devices into the OCI spec.
|
||||
GPUIDs []string
|
||||
// User is the OCI process user (oci.WithUser). Empty runs as the image
|
||||
// default.
|
||||
User string
|
||||
// ReadonlyRootFilesystem, Privileged, PseudoTerminal and Interactive are
|
||||
// pointers so the runner can tell "not requested" (nil) from an explicit
|
||||
// false, which still needs echoing but applies nothing.
|
||||
ReadonlyRootFilesystem *bool
|
||||
Privileged *bool
|
||||
PseudoTerminal *bool
|
||||
Interactive *bool
|
||||
// SystemControls are sysctls set on the OCI spec's Linux.Sysctl map.
|
||||
SystemControls []SystemControl
|
||||
// CapAdd / CapDrop are Linux capabilities added/dropped from the OCI
|
||||
// spec's process capability sets (linuxParameters.capabilities).
|
||||
CapAdd []string
|
||||
CapDrop []string
|
||||
}
|
||||
|
||||
// RunStatus is a finished container's outcome.
|
||||
type RunStatus struct {
|
||||
ExitCode int
|
||||
}
|
||||
|
||||
// Container is a container the runtime already knows about, discovered by List.
|
||||
// Labels carry the mulga.ecs.* task identity so the agent can re-adopt running
|
||||
// containers after a restart; Running reports whether its task is live.
|
||||
type Container struct {
|
||||
ID string
|
||||
Labels map[string]string
|
||||
Running bool
|
||||
}
|
||||
|
||||
// Runner creates and starts containers from already-pulled images. id is a
|
||||
// caller-unique container ID; Wait blocks until the container exits; Stop
|
||||
// requests a graceful exit (SIGTERM, then a forced kill+remove once timeout
|
||||
// elapses) — the agent's enforcement of a container's stopTimeout; Remove
|
||||
// tears down the container + its task immediately, with no grace period; List
|
||||
// enumerates known containers so the reboot reconciler can re-adopt the ones
|
||||
// still running.
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, id string, spec RunSpec) (containerID string, err error)
|
||||
Wait(ctx context.Context, containerID string) (RunStatus, error)
|
||||
Stop(ctx context.Context, containerID string, timeout time.Duration) error
|
||||
Remove(ctx context.Context, containerID string) error
|
||||
List(ctx context.Context) ([]Container, error)
|
||||
}
|
||||
|
||||
// Runtime is the full container runtime the ecs-agent drives: pull + run.
|
||||
type Runtime interface {
|
||||
ImagePuller
|
||||
Runner
|
||||
}
|
||||
@@ -1,168 +0,0 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// FakePuller is an in-memory ImagePuller for tests and the scheduler-only build.
|
||||
// It records pull requests and replays a programmed result or error.
|
||||
type FakePuller struct {
|
||||
mu sync.Mutex
|
||||
Pulls []PullSpec
|
||||
Authzd []string
|
||||
Result Image
|
||||
Err error
|
||||
Closed bool
|
||||
OnPull func(PullSpec) (Image, error)
|
||||
|
||||
// Run bookkeeping.
|
||||
Runs []RunSpec
|
||||
RunErr error
|
||||
WaitCode int
|
||||
WaitErr error
|
||||
Waited []string
|
||||
Removed []string
|
||||
OnRun func(string, RunSpec) (string, error)
|
||||
|
||||
// Stop bookkeeping: StopTimeouts pairs the container ID with the timeout it
|
||||
// was stopped with, in call order, so a test can assert both the target and
|
||||
// the enforced stopTimeout without a fake containerd.
|
||||
Stopped []string
|
||||
StopTimeouts []time.Duration
|
||||
StopErr error
|
||||
|
||||
// List bookkeeping: Containers is replayed by List; ListErr forces a failure,
|
||||
// limited to the first ListErrFor calls when that is non-zero, so a test can
|
||||
// model a runtime that recovers. Listed and ListCalls record the calls.
|
||||
Containers []Container
|
||||
ListErr error
|
||||
ListErrFor int
|
||||
Listed bool
|
||||
ListCalls int
|
||||
}
|
||||
|
||||
var (
|
||||
_ ImagePuller = (*FakePuller)(nil)
|
||||
_ Runner = (*FakePuller)(nil)
|
||||
)
|
||||
|
||||
// Pull records the spec, exercises the resolver, and returns the programmed
|
||||
// result. If OnPull is set it takes precedence over Result/Err.
|
||||
func (f *FakePuller) Pull(ctx context.Context, spec PullSpec, r Resolver) (Image, error) {
|
||||
f.mu.Lock()
|
||||
f.Pulls = append(f.Pulls, spec)
|
||||
f.mu.Unlock()
|
||||
|
||||
if r != nil {
|
||||
user, _, _, err := r.Authorize(ctx, spec.Ref)
|
||||
if err != nil {
|
||||
return Image{}, fmt.Errorf("authorize %s: %w", spec.Ref, err)
|
||||
}
|
||||
f.mu.Lock()
|
||||
f.Authzd = append(f.Authzd, user)
|
||||
f.mu.Unlock()
|
||||
}
|
||||
|
||||
if f.OnPull != nil {
|
||||
return f.OnPull(spec)
|
||||
}
|
||||
if f.Err != nil {
|
||||
return Image{}, f.Err
|
||||
}
|
||||
res := f.Result
|
||||
if res.Ref == "" {
|
||||
res.Ref = spec.Ref
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// Close marks the puller closed.
|
||||
func (f *FakePuller) Close() error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.Closed = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// Run records the spec and returns id (or the programmed result/error).
|
||||
func (f *FakePuller) Run(_ context.Context, id string, spec RunSpec) (string, error) {
|
||||
f.mu.Lock()
|
||||
f.Runs = append(f.Runs, spec)
|
||||
f.mu.Unlock()
|
||||
if f.OnRun != nil {
|
||||
return f.OnRun(id, spec)
|
||||
}
|
||||
if f.RunErr != nil {
|
||||
return "", f.RunErr
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// Wait records the container ID and returns the programmed exit code/error.
|
||||
func (f *FakePuller) Wait(_ context.Context, containerID string) (RunStatus, error) {
|
||||
f.mu.Lock()
|
||||
f.Waited = append(f.Waited, containerID)
|
||||
f.mu.Unlock()
|
||||
return RunStatus{ExitCode: f.WaitCode}, f.WaitErr
|
||||
}
|
||||
|
||||
// Stop records the container ID and the timeout it was stopped with.
|
||||
func (f *FakePuller) Stop(_ context.Context, containerID string, timeout time.Duration) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.Stopped = append(f.Stopped, containerID)
|
||||
f.StopTimeouts = append(f.StopTimeouts, timeout)
|
||||
return f.StopErr
|
||||
}
|
||||
|
||||
// Remove records the container ID.
|
||||
func (f *FakePuller) Remove(_ context.Context, containerID string) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.Removed = append(f.Removed, containerID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Waits returns a copy of the container IDs passed to Wait, for assertions.
|
||||
func (f *FakePuller) Waits() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.Waited...)
|
||||
}
|
||||
|
||||
// RunCalls returns a copy of the specs passed to Run, for assertions made while
|
||||
// the agent's goroutines may still be live.
|
||||
func (f *FakePuller) RunCalls() []RunSpec {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]RunSpec(nil), f.Runs...)
|
||||
}
|
||||
|
||||
// Reaped returns a copy of the container IDs passed to Stop.
|
||||
func (f *FakePuller) Reaped() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.Stopped...)
|
||||
}
|
||||
|
||||
// List replays the programmed container set (or error).
|
||||
func (f *FakePuller) List(_ context.Context) ([]Container, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.Listed = true
|
||||
f.ListCalls++
|
||||
if f.ListErr != nil && (f.ListErrFor == 0 || f.ListCalls <= f.ListErrFor) {
|
||||
return nil, f.ListErr
|
||||
}
|
||||
return f.Containers, nil
|
||||
}
|
||||
|
||||
// Lists reports how many times List was called.
|
||||
func (f *FakePuller) Lists() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.ListCalls
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestNormalizeRef(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"nginx:alpine": "docker.io/library/nginx:alpine",
|
||||
"nginx": "docker.io/library/nginx:latest",
|
||||
"ghcr.io/org/app:1": "ghcr.io/org/app:1",
|
||||
"123456789012.dkr.ecr.ap-southeast-2.spinifex.internal/app:1": "123456789012.dkr.ecr.ap-southeast-2.spinifex.internal/app:1",
|
||||
}
|
||||
for in, want := range cases {
|
||||
got, err := normalizeRef(in)
|
||||
if err != nil {
|
||||
t.Errorf("normalizeRef(%q): %v", in, err)
|
||||
continue
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("normalizeRef(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
if _, err := normalizeRef("Bad::Ref"); err == nil {
|
||||
t.Error("normalizeRef accepted invalid ref")
|
||||
}
|
||||
}
|
||||
@@ -1,116 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
)
|
||||
|
||||
// runtimeProbeTimeout bounds the liveness probe against a runtime the gate is
|
||||
// already holding, so a wedged socket cannot stall the watch loop.
|
||||
const runtimeProbeTimeout = 5 * time.Second
|
||||
|
||||
// defaultRuntimeRetry is how often the gate dials a runtime it does not have,
|
||||
// and probes one it does. The node starts containerd and the agent back to
|
||||
// back, so the first few dials are expected to fail.
|
||||
const defaultRuntimeRetry = 10 * time.Second
|
||||
|
||||
// runtimeGate holds the container runtime the agent has resolved, or nothing
|
||||
// when the socket has not answered. The runtime is resolved by a loop rather
|
||||
// than once at boot: the node's service manager reports the container daemon
|
||||
// started when the process spawned, not when its socket began accepting, so a
|
||||
// dial that fails in the first second of boot is a transient condition.
|
||||
//
|
||||
// Every read of the runtime goes through the gate, so a runtime that arrives
|
||||
// after boot is picked up on the next task without restarting the agent.
|
||||
type runtimeGate struct {
|
||||
mu sync.RWMutex
|
||||
rt ctrruntime.Runtime
|
||||
}
|
||||
|
||||
// newRuntimeGate returns a gate holding rt, which may be nil.
|
||||
func newRuntimeGate(rt ctrruntime.Runtime) *runtimeGate {
|
||||
return &runtimeGate{rt: rt}
|
||||
}
|
||||
|
||||
// get returns the current runtime, or nil when none is resolved.
|
||||
func (g *runtimeGate) get() ctrruntime.Runtime {
|
||||
g.mu.RLock()
|
||||
defer g.mu.RUnlock()
|
||||
return g.rt
|
||||
}
|
||||
|
||||
// available reports whether a runtime is resolved. It is what registration
|
||||
// consults to decide whether the host has any capacity to advertise.
|
||||
func (g *runtimeGate) available() bool { return g.get() != nil }
|
||||
|
||||
// set replaces the held runtime, closing the one it displaces.
|
||||
func (g *runtimeGate) set(rt ctrruntime.Runtime) {
|
||||
g.mu.Lock()
|
||||
old := g.rt
|
||||
g.rt = rt
|
||||
g.mu.Unlock()
|
||||
if old != nil && old != rt {
|
||||
_ = old.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// clear drops and closes the held runtime, returning whether it held one.
|
||||
func (g *runtimeGate) clear() bool {
|
||||
g.mu.Lock()
|
||||
old := g.rt
|
||||
g.rt = nil
|
||||
g.mu.Unlock()
|
||||
if old == nil {
|
||||
return false
|
||||
}
|
||||
_ = old.Close()
|
||||
return true
|
||||
}
|
||||
|
||||
// watch keeps the gate current until ctx is cancelled: it dials while the gate
|
||||
// is empty, and probes while it is full, clearing the gate when the probe fails
|
||||
// so the next tick re-dials. A containerd stopped and restarted underneath a
|
||||
// running agent is therefore picked up the same way a slow boot is.
|
||||
func (g *runtimeGate) watch(ctx context.Context, dial func() (ctrruntime.Runtime, error), interval time.Duration) {
|
||||
if interval <= 0 {
|
||||
interval = defaultRuntimeRetry
|
||||
}
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
g.step(ctx, dial)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// step performs one pass of the watch loop: dial when empty, probe when full.
|
||||
func (g *runtimeGate) step(ctx context.Context, dial func() (ctrruntime.Runtime, error)) {
|
||||
if rt := g.get(); rt != nil {
|
||||
probeCtx, cancel := context.WithTimeout(ctx, runtimeProbeTimeout)
|
||||
// A container list is what reconcile already asks the runtime at boot and
|
||||
// the cheapest thing it answers, so it doubles as the liveness probe.
|
||||
_, err := rt.List(probeCtx)
|
||||
cancel()
|
||||
if err != nil && ctx.Err() == nil {
|
||||
slog.Warn("ecs-agent: container runtime stopped answering, will re-dial", "err", err)
|
||||
g.clear()
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
rt, err := dial()
|
||||
if err != nil {
|
||||
slog.Debug("ecs-agent: container runtime still unavailable", "err", err)
|
||||
return
|
||||
}
|
||||
slog.Info("ecs-agent: container runtime available")
|
||||
g.set(rt)
|
||||
}
|
||||
@@ -1,124 +0,0 @@
|
||||
//test:in-package — package main cannot be imported by an external test package.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
)
|
||||
|
||||
// TestRuntimeGate_DialsUntilTheSocketAnswers is the boot case the agent used to
|
||||
// lose: the node starts containerd and the agent together, so the first dials
|
||||
// fail. Resolving once left the agent runtime-less for the life of the process.
|
||||
func TestRuntimeGate_DialsUntilTheSocketAnswers(t *testing.T) {
|
||||
g := newRuntimeGate(nil)
|
||||
if g.available() {
|
||||
t.Fatal("empty gate reported a runtime")
|
||||
}
|
||||
|
||||
rt := &ctrruntime.FakePuller{}
|
||||
dials := 0
|
||||
dial := func() (ctrruntime.Runtime, error) {
|
||||
dials++
|
||||
if dials < 3 {
|
||||
return nil, errors.New("dial containerd: no such file or directory")
|
||||
}
|
||||
return rt, nil
|
||||
}
|
||||
|
||||
for range 2 {
|
||||
g.step(t.Context(), dial)
|
||||
if g.available() {
|
||||
t.Fatalf("gate resolved a runtime after %d failed dials", dials)
|
||||
}
|
||||
}
|
||||
|
||||
g.step(t.Context(), dial)
|
||||
if g.get() != rt {
|
||||
t.Fatalf("gate did not pick up the runtime once the socket answered (dials=%d)", dials)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuntimeGate_DropsARuntimeThatStopsAnswering covers a containerd restarted
|
||||
// underneath a live agent: the held runtime stops answering, so the gate clears
|
||||
// it and the next pass re-dials rather than handing out a dead client forever.
|
||||
func TestRuntimeGate_DropsARuntimeThatStopsAnswering(t *testing.T) {
|
||||
dead := &ctrruntime.FakePuller{ListErr: errors.New("connection refused")}
|
||||
fresh := &ctrruntime.FakePuller{}
|
||||
g := newRuntimeGate(dead)
|
||||
|
||||
dial := func() (ctrruntime.Runtime, error) { return fresh, nil }
|
||||
|
||||
g.step(t.Context(), dial)
|
||||
if g.available() {
|
||||
t.Fatal("gate kept a runtime whose probe failed")
|
||||
}
|
||||
if !dead.Closed {
|
||||
t.Error("displaced runtime was not closed")
|
||||
}
|
||||
|
||||
g.step(t.Context(), dial)
|
||||
if g.get() != fresh {
|
||||
t.Fatal("gate did not re-dial after clearing")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuntimeGate_KeepsAHealthyRuntime guards the other direction: a probe that
|
||||
// succeeds must not churn the runtime, or every pass would drop a working client.
|
||||
func TestRuntimeGate_KeepsAHealthyRuntime(t *testing.T) {
|
||||
rt := &ctrruntime.FakePuller{}
|
||||
g := newRuntimeGate(rt)
|
||||
|
||||
dial := func() (ctrruntime.Runtime, error) {
|
||||
t.Error("dialled while holding a healthy runtime")
|
||||
return nil, errors.New("unexpected dial")
|
||||
}
|
||||
g.step(t.Context(), dial)
|
||||
|
||||
if g.get() != rt {
|
||||
t.Fatal("gate dropped a runtime whose probe succeeded")
|
||||
}
|
||||
if rt.Closed {
|
||||
t.Error("healthy runtime was closed")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuntimeGate_ClearIsIdempotent keeps Stop safe to call on an agent that
|
||||
// never resolved a runtime.
|
||||
func TestRuntimeGate_ClearIsIdempotent(t *testing.T) {
|
||||
g := newRuntimeGate(nil)
|
||||
if g.clear() {
|
||||
t.Error("clear reported dropping a runtime the gate never held")
|
||||
}
|
||||
|
||||
rt := &ctrruntime.FakePuller{}
|
||||
g.set(rt)
|
||||
if !g.clear() {
|
||||
t.Error("clear did not report dropping the held runtime")
|
||||
}
|
||||
if !rt.Closed {
|
||||
t.Error("cleared runtime was not closed")
|
||||
}
|
||||
if g.clear() {
|
||||
t.Error("second clear reported dropping a runtime")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuntimeGate_WatchStopsWithContext keeps the watch loop from outliving the
|
||||
// agent it belongs to.
|
||||
func TestRuntimeGate_WatchStopsWithContext(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
g := newRuntimeGate(nil)
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
g.watch(ctx, func() (ctrruntime.Runtime, error) { return nil, errors.New("no socket") }, 0)
|
||||
close(done)
|
||||
}()
|
||||
|
||||
cancel()
|
||||
<-done
|
||||
}
|
||||
@@ -1,469 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
// defaultStopTimeout is applied when a container carries no explicit
|
||||
// stopTimeout, matching the EC2 launch-type default ECS itself uses.
|
||||
const defaultStopTimeout = 30 * time.Second
|
||||
|
||||
// pollAssignments drains the instance's assignment inbox on a ticker until ctx is
|
||||
// cancelled. Each assign is dispatched to runTask exactly once; the taskIDs seen
|
||||
// on a poll are acked on the next poll so the gateway can drop them — a crash
|
||||
// before ack re-delivers (at-least-once), matching ACS. A failed poll is logged
|
||||
// and retried on the next tick rather than killing the loop. dispatched is seeded
|
||||
// by the startup reconcile with tasks already adopted from running containers, so
|
||||
// their re-delivered assignments are acked but not re-run.
|
||||
func (a *Agent) pollAssignments(ctx context.Context, dispatched map[string]bool) {
|
||||
ticker := time.NewTicker(a.cfg.PollInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
reaps := newStopLedger()
|
||||
var ackAssigns, ackStops []string
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
assigns, stops, err := a.cp.PollAssignments(a.id.ClusterName, a.id.InstanceID, ackAssigns, ackStops)
|
||||
if err != nil {
|
||||
slog.Warn("ecs-agent: poll assignments failed", "err", err)
|
||||
continue
|
||||
}
|
||||
// Dispatch stops first so a task stopped in the same poll it was assigned
|
||||
// is never started.
|
||||
ackStops = ackStops[:0]
|
||||
for i := range stops {
|
||||
sd := stops[i]
|
||||
switch {
|
||||
case reaps.done(sd.TaskID):
|
||||
ackStops = append(ackStops, sd.TaskID)
|
||||
case reaps.begin(sd.TaskID):
|
||||
slog.Info("ecs-agent: task stop requested", "task", sd.TaskID, "reason", sd.Reason)
|
||||
go func() { reaps.end(sd.TaskID, a.stopTask(ctx, sd)) }()
|
||||
}
|
||||
}
|
||||
ackAssigns = ackAssigns[:0]
|
||||
for i := range assigns {
|
||||
as := assigns[i]
|
||||
if !dispatched[as.TaskID] && !reaps.seen(as.TaskID) {
|
||||
dispatched[as.TaskID] = true
|
||||
slog.Info("ecs-agent: task assigned", "task", as.TaskID, "containers", len(as.Containers))
|
||||
go a.runTask(ctx, &as)
|
||||
}
|
||||
ackAssigns = append(ackAssigns, as.TaskID)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// stopLedger tracks reaps in flight and reaps that reported STOPPED. The two
|
||||
// are separate because only the second may be acked: acking the first deletes a
|
||||
// directive that was never performed, and nothing re-posts it.
|
||||
type stopLedger struct {
|
||||
mu sync.Mutex
|
||||
inflight map[string]bool
|
||||
stopped map[string]bool
|
||||
}
|
||||
|
||||
func newStopLedger() *stopLedger {
|
||||
return &stopLedger{inflight: map[string]bool{}, stopped: map[string]bool{}}
|
||||
}
|
||||
|
||||
// begin claims a task for a reap, reporting false when one is already running.
|
||||
func (l *stopLedger) begin(taskID string) bool {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if l.inflight[taskID] {
|
||||
return false
|
||||
}
|
||||
l.inflight[taskID] = true
|
||||
return true
|
||||
}
|
||||
|
||||
// end releases the claim, recording the task as stopped only if the reap got
|
||||
// that far. A failed reap leaves nothing behind, so the next poll redelivers
|
||||
// the directive and dispatches it again.
|
||||
func (l *stopLedger) end(taskID string, reported bool) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
delete(l.inflight, taskID)
|
||||
if reported {
|
||||
l.stopped[taskID] = true
|
||||
}
|
||||
}
|
||||
|
||||
// done reports whether the task's reap reported STOPPED, which is the only
|
||||
// condition under which its directive may be acked.
|
||||
func (l *stopLedger) done(taskID string) bool {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.stopped[taskID]
|
||||
}
|
||||
|
||||
// seen reports whether a stop for the task is in flight or complete, so an
|
||||
// assignment arriving alongside its own stop is never started.
|
||||
func (l *stopLedger) seen(taskID string) bool {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inflight[taskID] || l.stopped[taskID]
|
||||
}
|
||||
|
||||
// stopTask reaps a task's containers on a control-plane stop directive: it
|
||||
// lists the runtime's containers, gracefully stops (SIGTERM, wait up to the
|
||||
// container's stopTimeout, then force kill + delete) the ones labeled with
|
||||
// this task, then reports the task STOPPED with the directive's reason.
|
||||
// Idempotent — a task with no live containers still reports STOPPED so the
|
||||
// scheduler releases its capacity. Reports whether it got as far as saying
|
||||
// STOPPED: a caller may only ack the directive when it did.
|
||||
func (a *Agent) stopTask(ctx context.Context, sd bus.StopDirective) bool {
|
||||
runner := a.rt.get()
|
||||
if runner == nil {
|
||||
slog.Warn("ecs-agent: stop deferred, no container runtime", "task", sd.TaskID)
|
||||
return false
|
||||
}
|
||||
containers, err := runner.List(ctx)
|
||||
if err != nil {
|
||||
slog.Warn("ecs-agent: stop list failed", "task", sd.TaskID, "err", err)
|
||||
return false
|
||||
}
|
||||
statuses := make([]bus.ContainerStatus, 0)
|
||||
for _, c := range containers {
|
||||
if c.Labels[labelTaskID] != sd.TaskID {
|
||||
continue
|
||||
}
|
||||
if rerr := runner.Stop(ctx, c.ID, containerStopTimeout(c.Labels)); rerr != nil {
|
||||
slog.Warn("ecs-agent: stop failed", "task", sd.TaskID, "container", c.ID, "err", rerr)
|
||||
}
|
||||
statuses = append(statuses, bus.ContainerStatus{
|
||||
Name: c.Labels[labelContainerName], Status: bus.TaskStatusStopped, ContainerID: c.ID,
|
||||
})
|
||||
}
|
||||
reason := sd.Reason
|
||||
if reason == "" {
|
||||
reason = "Task stopped"
|
||||
}
|
||||
a.reportTaskState(&bus.Assign{TaskID: sd.TaskID}, bus.TaskStatusStopped, reason, statuses)
|
||||
slog.Info("ecs-agent: task reaped", "task", sd.TaskID, "containers", len(statuses))
|
||||
return true
|
||||
}
|
||||
|
||||
// runTask pulls each container image, starts the containers, and reports task
|
||||
// state on the bus. A missing runtime or any per-container failure reports the
|
||||
// task STOPPED with a reason; success reports RUNNING and waits for exit.
|
||||
func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
|
||||
rt := a.rt.get()
|
||||
if rt == nil {
|
||||
a.reportTaskState(as, bus.TaskStatusStopped, "containerd unavailable on agent", nil)
|
||||
return
|
||||
}
|
||||
|
||||
// awsvpc: build the task netns from the hot-plugged ENI before any container.
|
||||
netnsPath, err := a.setupTaskNetns(as)
|
||||
if err != nil {
|
||||
slog.Error("ecs-agent: task netns setup failed", "task", as.TaskID, "err", err)
|
||||
a.reportTaskState(as, bus.TaskStatusStopped, "network setup failed: "+err.Error(), nil)
|
||||
return
|
||||
}
|
||||
|
||||
credID := a.registerTaskCreds(as)
|
||||
resolver := a.pullResolver(as)
|
||||
|
||||
statuses := make([]bus.ContainerStatus, 0, len(as.Containers))
|
||||
for _, c := range as.Containers {
|
||||
// startTimeout bounds pull-through-start as a unit: real ECS measures a
|
||||
// container's time-to-RUNNING from the same starting point.
|
||||
runCtx, cancel := containerStartCtx(ctx, c.StartTimeout)
|
||||
|
||||
if _, err := rt.Pull(runCtx, ctrruntime.PullSpec{Ref: c.Image}, resolver); err != nil {
|
||||
cancel()
|
||||
slog.Error("ecs-agent: pull failed", "task", as.TaskID, "image", c.Image, "err", err)
|
||||
a.teardownTaskNetns(as)
|
||||
a.reportTaskState(as, bus.TaskStatusStopped, "image pull failed: "+err.Error(), statuses)
|
||||
return
|
||||
}
|
||||
|
||||
cid := containerID(as.TaskID, c.Name)
|
||||
gpuIDs := a.pinContainerGPUs(as.TaskID, c.Name, c.GPU)
|
||||
labels := taskLabels(as, c.Name)
|
||||
if c.StopTimeout != nil {
|
||||
labels[labelStopTimeout] = strconv.FormatInt(*c.StopTimeout, 10)
|
||||
}
|
||||
spec := ctrruntime.RunSpec{
|
||||
Image: c.Image,
|
||||
Command: c.Command,
|
||||
Env: withCredEnv(c.Environment, credID),
|
||||
Labels: labels,
|
||||
NetnsPath: netnsPath,
|
||||
GPU: c.GPU,
|
||||
GPUIDs: gpuIDs,
|
||||
User: c.User,
|
||||
ReadonlyRootFilesystem: c.ReadonlyRootFilesystem,
|
||||
Privileged: c.Privileged,
|
||||
PseudoTerminal: c.PseudoTerminal,
|
||||
Interactive: c.Interactive,
|
||||
SystemControls: toRuntimeSystemControls(c.SystemControls),
|
||||
CapAdd: c.CapAdd,
|
||||
CapDrop: c.CapDrop,
|
||||
}
|
||||
id, err := rt.Run(runCtx, cid, spec)
|
||||
cancel()
|
||||
if err != nil {
|
||||
slog.Error("ecs-agent: run failed", "task", as.TaskID, "container", c.Name, "err", err)
|
||||
a.teardownTaskNetns(as)
|
||||
a.reportTaskState(as, bus.TaskStatusStopped, "container start failed: "+err.Error(), statuses)
|
||||
return
|
||||
}
|
||||
statuses = append(statuses, bus.ContainerStatus{
|
||||
Name: c.Name, Status: bus.TaskStatusRunning, ContainerID: id, GPUIDs: gpuIDs,
|
||||
})
|
||||
go a.waitContainer(ctx, as, c.Name, id)
|
||||
}
|
||||
|
||||
a.reportTaskState(as, bus.TaskStatusRunning, "", statuses)
|
||||
slog.Info("ecs-agent: task running", "task", as.TaskID, "containers", len(statuses))
|
||||
}
|
||||
|
||||
// waitContainer blocks until a container exits, then reports the task STOPPED
|
||||
// with the exit code. v1 stops the whole task when its first container exits.
|
||||
func (a *Agent) waitContainer(ctx context.Context, as *bus.Assign, name, containerID string) {
|
||||
runner := a.rt.get()
|
||||
if runner == nil {
|
||||
slog.Warn("ecs-agent: cannot wait on container, no container runtime", "task", as.TaskID, "container", name)
|
||||
return
|
||||
}
|
||||
status, err := runner.Wait(ctx, containerID)
|
||||
if err != nil {
|
||||
slog.Warn("ecs-agent: wait container failed", "task", as.TaskID, "container", name, "err", err)
|
||||
return
|
||||
}
|
||||
exit := status.ExitCode
|
||||
a.teardownTaskNetns(as)
|
||||
statuses := []bus.ContainerStatus{{Name: name, Status: bus.TaskStatusStopped, ContainerID: containerID, ExitCode: &exit}}
|
||||
a.reportTaskState(as, bus.TaskStatusStopped, fmt.Sprintf("container %s exited (%d)", name, exit), statuses)
|
||||
slog.Info("ecs-agent: task stopped", "task", as.TaskID, "container", name, "exitCode", exit)
|
||||
}
|
||||
|
||||
// setupTaskNetns builds the awsvpc task netns from the hot-plugged ENI. Bridge/
|
||||
// host tasks (no ENI MAC) and a nil controller are no-ops returning an empty path.
|
||||
func (a *Agent) setupTaskNetns(as *bus.Assign) (string, error) {
|
||||
if as.ENIMacAddress == "" || a.netns == nil {
|
||||
return "", nil
|
||||
}
|
||||
return a.netns.Setup(as.TaskID, as.ENIMacAddress)
|
||||
}
|
||||
|
||||
// teardownTaskNetns releases the task's IAM credential registration, then removes
|
||||
// the awsvpc task netns (the latter is a no-op for bridge/host tasks). It runs at
|
||||
// every task-stop path so credentials never outlive the task that owns them.
|
||||
func (a *Agent) teardownTaskNetns(as *bus.Assign) {
|
||||
if a.cred != nil {
|
||||
a.cred.Deregister(taskCredID(as))
|
||||
}
|
||||
if as.ENIMacAddress == "" || a.netns == nil {
|
||||
return
|
||||
}
|
||||
if err := a.netns.Teardown(as.TaskID); err != nil {
|
||||
slog.Warn("ecs-agent: task netns teardown", "task", as.TaskID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// pullResolver returns the ECR resolver used for a task's image pulls. When the
|
||||
// assign carries an execution role, it authorizes pulls as that role (assumed
|
||||
// over the gateway); an empty role or a nil credential endpoint (unit tests)
|
||||
// falls back to the instance-role resolver, so a pull is never worse off.
|
||||
func (a *Agent) pullResolver(as *bus.Assign) ctrruntime.Resolver {
|
||||
if as.ExecutionRoleARN == "" || a.cred == nil {
|
||||
return a.resolver
|
||||
}
|
||||
prov := a.cred.AssumeProvider(as.ExecutionRoleARN, sessionName(as.TaskID))
|
||||
return newLazyECRResolver(prov, a.cfg.Region, a.cfg.GatewayURL, a.cfg.GatewayCA)
|
||||
}
|
||||
|
||||
// registerTaskCreds registers the task's credID -> role mapping with the
|
||||
// credential endpoint and returns the credID (empty when the task has no role).
|
||||
func (a *Agent) registerTaskCreds(as *bus.Assign) string {
|
||||
credID := taskCredID(as)
|
||||
if credID == "" || a.cred == nil {
|
||||
return credID
|
||||
}
|
||||
a.cred.Register(credID, as.TaskRoleARN)
|
||||
return credID
|
||||
}
|
||||
|
||||
// taskCredID is the credential ID a task's containers fetch credentials under:
|
||||
// the scheduler-assigned CredID, falling back to the taskID. Empty when the task
|
||||
// carries no IAM role.
|
||||
func taskCredID(as *bus.Assign) string {
|
||||
if as.TaskRoleARN == "" {
|
||||
return ""
|
||||
}
|
||||
if as.CredID != "" {
|
||||
return as.CredID
|
||||
}
|
||||
return as.TaskID
|
||||
}
|
||||
|
||||
// withCredEnv returns env with AWS_CONTAINER_CREDENTIALS_RELATIVE_URI set for
|
||||
// credID, copying the map so the assign's container env is left untouched. A
|
||||
// blank credID (no task role) returns env unchanged.
|
||||
func withCredEnv(env map[string]string, credID string) map[string]string {
|
||||
if credID == "" {
|
||||
return env
|
||||
}
|
||||
out := make(map[string]string, len(env)+1)
|
||||
maps.Copy(out, env)
|
||||
out["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"] = credRelativeURI(credID)
|
||||
return out
|
||||
}
|
||||
|
||||
// containerStartCtx bounds pull+run by a container's startTimeout, if set. A
|
||||
// nil startTimeout (the common case) returns ctx unchanged with a no-op
|
||||
// cancel, matching today's unbounded behaviour.
|
||||
func containerStartCtx(ctx context.Context, startTimeout *int64) (context.Context, context.CancelFunc) {
|
||||
if startTimeout == nil {
|
||||
return ctx, func() {}
|
||||
}
|
||||
return context.WithTimeout(ctx, time.Duration(*startTimeout)*time.Second)
|
||||
}
|
||||
|
||||
// containerStopTimeout reads the stopTimeout stamped on a container's labels
|
||||
// by runTask, falling back to defaultStopTimeout when absent or unparseable.
|
||||
func containerStopTimeout(labels map[string]string) time.Duration {
|
||||
v, ok := labels[labelStopTimeout]
|
||||
if !ok {
|
||||
return defaultStopTimeout
|
||||
}
|
||||
secs, err := strconv.ParseInt(v, 10, 64)
|
||||
if err != nil {
|
||||
return defaultStopTimeout
|
||||
}
|
||||
return time.Duration(secs) * time.Second
|
||||
}
|
||||
|
||||
// toRuntimeSystemControls maps the assign's sysctl pairs to the runtime
|
||||
// package's own type, keeping cmd/ecs-agent/runtime decoupled from the ecs
|
||||
// handlers' bus package.
|
||||
func toRuntimeSystemControls(in []bus.SystemControl) []ctrruntime.SystemControl {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]ctrruntime.SystemControl, len(in))
|
||||
for i, sc := range in {
|
||||
out[i] = ctrruntime.SystemControl{Namespace: sc.Namespace, Value: sc.Value}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// pinContainerGPUs reserves n device UUIDs for a container from the local
|
||||
// ledger, before the container is started so the runner can inject them as
|
||||
// CDI devices. A short ledger (fewer free devices than requested, e.g.
|
||||
// discovery found none) is logged and treated as best-effort: the container
|
||||
// still runs without pinned UUIDs (and without GPU access) rather than
|
||||
// failing the task.
|
||||
func (a *Agent) pinContainerGPUs(taskID, containerName string, n int) []string {
|
||||
if n <= 0 || a.gpu == nil {
|
||||
return nil
|
||||
}
|
||||
uuids, err := a.gpu.Pin(gpuKey(taskID, containerName), n)
|
||||
if err != nil {
|
||||
slog.Warn("ecs-agent: gpu pin short", "task", taskID, "container", containerName, "err", err)
|
||||
}
|
||||
return uuids
|
||||
}
|
||||
|
||||
// reportTaskState reports a task transition through the gateway's
|
||||
// SubmitTaskStateChange action, then (RUNNING with pinned GPUs) reports the
|
||||
// per-container device UUIDs, and (STOPPED) releases them back to the ledger.
|
||||
func (a *Agent) reportTaskState(as *bus.Assign, status, reason string, containers []bus.ContainerStatus) {
|
||||
if a.cp == nil {
|
||||
return
|
||||
}
|
||||
msg := bus.TaskState{
|
||||
AccountID: a.id.AccountID,
|
||||
ClusterName: a.id.ClusterName,
|
||||
InstanceID: a.id.InstanceID,
|
||||
TaskID: as.TaskID,
|
||||
LastStatus: status,
|
||||
Containers: containers,
|
||||
Reason: reason,
|
||||
ReportedAt: time.Now().UTC(),
|
||||
}
|
||||
if err := a.cp.SubmitTaskState(msg); err != nil {
|
||||
slog.Error("ecs-agent: report task-state failed", "task", as.TaskID, "err", err)
|
||||
}
|
||||
a.reportTaskGPUs(as.TaskID, containers)
|
||||
if status == bus.TaskStatusStopped && a.gpu != nil {
|
||||
a.gpu.ReleaseTask(as.TaskID)
|
||||
}
|
||||
}
|
||||
|
||||
// reportTaskGPUs sends the control plane the pinned device UUIDs for any
|
||||
// container in the report that has them, so DescribeTasks can surface gpuIds.
|
||||
// A no-op when nothing was pinned (the common non-GPU-task case).
|
||||
func (a *Agent) reportTaskGPUs(taskID string, containers []bus.ContainerStatus) {
|
||||
reports := make([]handlers_ecs.ContainerGPUReport, 0, len(containers))
|
||||
for _, c := range containers {
|
||||
if len(c.GPUIDs) > 0 {
|
||||
reports = append(reports, handlers_ecs.ContainerGPUReport{Name: c.Name, GPUIDs: c.GPUIDs})
|
||||
}
|
||||
}
|
||||
if len(reports) == 0 {
|
||||
return
|
||||
}
|
||||
if err := a.cp.ReportTaskGPU(a.id.ClusterName, taskID, reports); err != nil {
|
||||
slog.Error("ecs-agent: report task-gpu failed", "task", taskID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// containerID composes a containerd-valid container ID from a task + container.
|
||||
func containerID(taskID, name string) string {
|
||||
return fmt.Sprintf("%s-%s", taskID, name)
|
||||
}
|
||||
|
||||
// mulga.ecs.* label keys stamped on every container. The reboot reconciler reads
|
||||
// them back to re-associate running containers with their task on restart; the
|
||||
// cred/role/MAC labels make a container self-describing enough to re-register its
|
||||
// credentials and tear down its netns without the original assignment.
|
||||
const (
|
||||
labelTaskID = "mulga.ecs.taskID"
|
||||
labelContainerName = "mulga.ecs.containerName"
|
||||
labelClusterName = "mulga.ecs.clusterName"
|
||||
labelCredID = "mulga.ecs.credID"
|
||||
labelTaskRoleARN = "mulga.ecs.taskRoleArn"
|
||||
labelENIMac = "mulga.ecs.eniMac"
|
||||
// labelStopTimeout carries a container's stopTimeout (seconds) so stopTask
|
||||
// can read it back from List's labels; it has no assign to consult there.
|
||||
labelStopTimeout = "mulga.ecs.stopTimeout"
|
||||
)
|
||||
|
||||
// taskLabels are the mulga.ecs.* labels stamped on a container. The cred/role/MAC
|
||||
// labels are omitted when empty so a container carries only what it needs.
|
||||
func taskLabels(as *bus.Assign, name string) map[string]string {
|
||||
labels := map[string]string{
|
||||
labelTaskID: as.TaskID,
|
||||
labelContainerName: name,
|
||||
labelClusterName: as.ClusterName,
|
||||
}
|
||||
if credID := taskCredID(as); credID != "" {
|
||||
labels[labelCredID] = credID
|
||||
}
|
||||
if as.TaskRoleARN != "" {
|
||||
labels[labelTaskRoleARN] = as.TaskRoleARN
|
||||
}
|
||||
if as.ENIMacAddress != "" {
|
||||
labels[labelENIMac] = as.ENIMacAddress
|
||||
}
|
||||
return labels
|
||||
}
|
||||
@@ -1,825 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
|
||||
)
|
||||
|
||||
func testAssign() *bus.Assign {
|
||||
return &bus.Assign{
|
||||
AccountID: "123456789012",
|
||||
ClusterName: "default",
|
||||
InstanceID: "i-abc123",
|
||||
TaskID: "t-001",
|
||||
Containers: []bus.AssignContainer{
|
||||
{Name: "web", Image: "registry/web:1", Command: []string{"/bin/true"},
|
||||
Environment: map[string]string{"FOO": "bar"}},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newRunAgent(cp controlPlane, rt *ctrruntime.FakePuller) *Agent {
|
||||
return newAgent(config{}, testIdentity(), cp, rt, nil)
|
||||
}
|
||||
|
||||
// runTask with no runtime reports the task STOPPED instead of crashing.
|
||||
func TestRunTask_NoRuntimeReportsStopped(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
a := newAgent(config{}, testIdentity(), cp, nil, nil)
|
||||
a.runTask(context.Background(), testAssign())
|
||||
|
||||
st := cp.taskStates()
|
||||
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
|
||||
t.Fatalf("want one STOPPED state, got %+v", st)
|
||||
}
|
||||
if st[0].Reason == "" {
|
||||
t.Error("STOPPED state missing reason")
|
||||
}
|
||||
}
|
||||
|
||||
// Happy path: pull + run reports RUNNING with the container ID; Wait blocking
|
||||
// (forced error) means no STOPPED overwrite, so we can assert the RUNNING frame.
|
||||
func TestRunTask_PullRunReportsRunning(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
as := testAssign()
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if len(rt.Pulls) != 1 || rt.Pulls[0].Ref != "registry/web:1" {
|
||||
t.Fatalf("expected one pull of registry/web:1, got %+v", rt.Pulls)
|
||||
}
|
||||
if len(rt.Runs) != 1 || rt.Runs[0].Image != "registry/web:1" {
|
||||
t.Fatalf("expected one run, got %+v", rt.Runs)
|
||||
}
|
||||
if rt.Runs[0].Labels["mulga.ecs.taskID"] != "t-001" {
|
||||
t.Errorf("missing task label: %+v", rt.Runs[0].Labels)
|
||||
}
|
||||
|
||||
st := cp.taskStates()
|
||||
if len(st) == 0 || st[len(st)-1].LastStatus != bus.TaskStatusRunning {
|
||||
t.Fatalf("want RUNNING final state, got %+v", st)
|
||||
}
|
||||
if len(st[len(st)-1].Containers) != 1 || st[len(st)-1].Containers[0].ContainerID != "t-001-web" {
|
||||
t.Errorf("RUNNING state container wrong: %+v", st[len(st)-1].Containers)
|
||||
}
|
||||
}
|
||||
|
||||
// Each enforced field maps onto RunSpec exactly, asserted through the fake
|
||||
// runtime — the mapping this change exists to wire, without a real containerd.
|
||||
func TestRunTask_RuntimeFieldsCarriedToRunSpec(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
as := testAssign()
|
||||
ro := true
|
||||
priv := false
|
||||
tty := true
|
||||
interactive := true
|
||||
as.Containers[0].User = "1000:1000"
|
||||
as.Containers[0].ReadonlyRootFilesystem = &ro
|
||||
as.Containers[0].Privileged = &priv
|
||||
as.Containers[0].PseudoTerminal = &tty
|
||||
as.Containers[0].Interactive = &interactive
|
||||
as.Containers[0].SystemControls = []bus.SystemControl{{Namespace: "net.core.somaxconn", Value: "1024"}}
|
||||
as.Containers[0].CapAdd = []string{"SYS_PTRACE"}
|
||||
as.Containers[0].CapDrop = []string{"NET_RAW"}
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if len(rt.Runs) != 1 {
|
||||
t.Fatalf("expected one run, got %+v", rt.Runs)
|
||||
}
|
||||
got := rt.Runs[0]
|
||||
if got.User != "1000:1000" {
|
||||
t.Errorf("User = %q, want 1000:1000", got.User)
|
||||
}
|
||||
if got.ReadonlyRootFilesystem == nil || !*got.ReadonlyRootFilesystem {
|
||||
t.Errorf("ReadonlyRootFilesystem = %v, want true", got.ReadonlyRootFilesystem)
|
||||
}
|
||||
if got.Privileged == nil || *got.Privileged {
|
||||
t.Errorf("Privileged = %v, want false (present, not applied)", got.Privileged)
|
||||
}
|
||||
if got.PseudoTerminal == nil || !*got.PseudoTerminal {
|
||||
t.Errorf("PseudoTerminal = %v, want true", got.PseudoTerminal)
|
||||
}
|
||||
if got.Interactive == nil || !*got.Interactive {
|
||||
t.Errorf("Interactive = %v, want true", got.Interactive)
|
||||
}
|
||||
if len(got.SystemControls) != 1 || got.SystemControls[0].Namespace != "net.core.somaxconn" {
|
||||
t.Errorf("SystemControls = %+v, want one net.core.somaxconn entry", got.SystemControls)
|
||||
}
|
||||
if len(got.CapAdd) != 1 || got.CapAdd[0] != "SYS_PTRACE" {
|
||||
t.Errorf("CapAdd = %+v, want [SYS_PTRACE]", got.CapAdd)
|
||||
}
|
||||
if len(got.CapDrop) != 1 || got.CapDrop[0] != "NET_RAW" {
|
||||
t.Errorf("CapDrop = %+v, want [NET_RAW]", got.CapDrop)
|
||||
}
|
||||
}
|
||||
|
||||
// An assign carrying none of the new fields leaves RunSpec's pointer fields
|
||||
// nil rather than defaulting to a false indistinguishable from "explicitly
|
||||
// not requested".
|
||||
func TestRunTask_NilRuntimeFieldsLeaveRunSpecUnset(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
a.runTask(context.Background(), testAssign())
|
||||
|
||||
if len(rt.Runs) != 1 {
|
||||
t.Fatalf("expected one run, got %+v", rt.Runs)
|
||||
}
|
||||
got := rt.Runs[0]
|
||||
if got.ReadonlyRootFilesystem != nil || got.Privileged != nil || got.PseudoTerminal != nil || got.Interactive != nil {
|
||||
t.Errorf("want all runtime-field pointers nil, got %+v", got)
|
||||
}
|
||||
if len(got.SystemControls) != 0 || len(got.CapAdd) != 0 || len(got.CapDrop) != 0 {
|
||||
t.Errorf("want no sysctls/capabilities, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTask_StampsStopTimeoutLabel verifies runTask stamps a container's
|
||||
// stopTimeout as a label — the only way stopTask (which only has List's
|
||||
// labels, not the original assign) can recover it later.
|
||||
func TestRunTask_StampsStopTimeoutLabel(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
as := testAssign()
|
||||
stop := int64(45)
|
||||
as.Containers[0].StopTimeout = &stop
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if len(rt.Runs) != 1 {
|
||||
t.Fatalf("expected one run, got %+v", rt.Runs)
|
||||
}
|
||||
if got := rt.Runs[0].Labels[labelStopTimeout]; got != "45" {
|
||||
t.Errorf("stopTimeout label = %q, want 45", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTask_NoStopTimeoutLabelWhenUnset verifies a container with no
|
||||
// stopTimeout carries no label, so stopTask falls back to the default rather
|
||||
// than reading a stray "0".
|
||||
func TestRunTask_NoStopTimeoutLabelWhenUnset(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
a.runTask(context.Background(), testAssign())
|
||||
|
||||
if len(rt.Runs) != 1 {
|
||||
t.Fatalf("expected one run, got %+v", rt.Runs)
|
||||
}
|
||||
if _, ok := rt.Runs[0].Labels[labelStopTimeout]; ok {
|
||||
t.Errorf("want no stopTimeout label, got %+v", rt.Runs[0].Labels)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopTask_UsesContainerStopTimeout verifies that stopTask reads a
|
||||
// container's stopTimeout label and calls the runner's graceful Stop (SIGTERM
|
||||
// + bounded wait) with it, instead of the hard kill+delete used before this
|
||||
// fix.
|
||||
func TestStopTask_UsesContainerStopTimeout(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{{
|
||||
ID: "t-001-web",
|
||||
Labels: map[string]string{
|
||||
labelTaskID: "t-001", labelContainerName: "web", labelStopTimeout: "45",
|
||||
},
|
||||
}}}
|
||||
a := newRunAgent(cp, rt)
|
||||
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
|
||||
|
||||
if len(rt.Stopped) != 1 || rt.Stopped[0] != "t-001-web" {
|
||||
t.Fatalf("want t-001-web stopped, got %+v", rt.Stopped)
|
||||
}
|
||||
if len(rt.StopTimeouts) != 1 || rt.StopTimeouts[0] != 45*time.Second {
|
||||
t.Fatalf("want stopTimeout 45s, got %+v", rt.StopTimeouts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopTask_DefaultStopTimeoutWhenUnset verifies a container with no
|
||||
// stopTimeout label gets the EC2 launch-type default rather than an unbounded
|
||||
// or zero grace period.
|
||||
func TestStopTask_DefaultStopTimeoutWhenUnset(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{{
|
||||
ID: "t-001-web",
|
||||
Labels: map[string]string{labelTaskID: "t-001", labelContainerName: "web"},
|
||||
}}}
|
||||
a := newRunAgent(cp, rt)
|
||||
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
|
||||
|
||||
if len(rt.StopTimeouts) != 1 || rt.StopTimeouts[0] != defaultStopTimeout {
|
||||
t.Fatalf("want default stopTimeout %v, got %+v", defaultStopTimeout, rt.StopTimeouts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTask_GPUCarriedToRunSpec verifies that the AssignContainer's GPU count
|
||||
// reaches the runtime RunSpec unchanged; a
|
||||
// container with no GPU request keeps RunSpec.GPU at zero (regression).
|
||||
func TestRunTask_GPUCarriedToRunSpec(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
as := testAssign()
|
||||
as.Containers = append(as.Containers, bus.AssignContainer{
|
||||
Name: "trainer", Image: "registry/trainer:1", GPU: 2,
|
||||
})
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if len(rt.Runs) != 2 {
|
||||
t.Fatalf("expected two runs, got %+v", rt.Runs)
|
||||
}
|
||||
if rt.Runs[0].GPU != 0 {
|
||||
t.Errorf("web container: want GPU=0, got %d", rt.Runs[0].GPU)
|
||||
}
|
||||
if rt.Runs[1].GPU != 2 {
|
||||
t.Errorf("trainer container: want GPU=2, got %d", rt.Runs[1].GPU)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTask_GPUPinsAndReportsUUIDs covers C3: a container's GPU request is
|
||||
// pinned from the agent's local ledger, carried on the RUNNING report's
|
||||
// ContainerStatus, and forwarded to the control plane via ReportTaskGPU.
|
||||
func TestRunTask_GPUPinsAndReportsUUIDs(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, nil)
|
||||
as := testAssign()
|
||||
as.Containers = append(as.Containers, bus.AssignContainer{
|
||||
Name: "trainer", Image: "registry/trainer:1", GPU: 1,
|
||||
})
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
st := cp.taskStates()
|
||||
if len(st) == 0 || st[len(st)-1].LastStatus != bus.TaskStatusRunning {
|
||||
t.Fatalf("want RUNNING final state, got %+v", st)
|
||||
}
|
||||
last := st[len(st)-1]
|
||||
var trainer *bus.ContainerStatus
|
||||
for i := range last.Containers {
|
||||
if last.Containers[i].Name == "trainer" {
|
||||
trainer = &last.Containers[i]
|
||||
}
|
||||
}
|
||||
if trainer == nil || len(trainer.GPUIDs) != 1 {
|
||||
t.Fatalf("trainer container missing pinned GPU UUID: %+v", last.Containers)
|
||||
}
|
||||
if trainer.GPUIDs[0] != "GPU-aaa" && trainer.GPUIDs[0] != "GPU-bbb" {
|
||||
t.Errorf("unexpected pinned UUID %q", trainer.GPUIDs[0])
|
||||
}
|
||||
|
||||
reports := cp.gpuReportsFor(as.TaskID)
|
||||
if len(reports) != 1 {
|
||||
t.Fatalf("want one ReportTaskGPU call, got %d", len(reports))
|
||||
}
|
||||
if reports[0].cluster != as.ClusterName {
|
||||
t.Errorf("report cluster = %q, want %q", reports[0].cluster, as.ClusterName)
|
||||
}
|
||||
if len(reports[0].containers) != 1 || reports[0].containers[0].Name != "trainer" {
|
||||
t.Errorf("report containers = %+v, want one trainer entry", reports[0].containers)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTask_GPUIDsReachRunSpec covers C4: the ledger's pinned UUIDs for a
|
||||
// container reach RunSpec.GPUIDs (not just the RunSpec.GPU count), so the
|
||||
// runner can inject them as CDI devices; a non-GPU container's RunSpec
|
||||
// carries no GPUIDs.
|
||||
func TestRunTask_GPUIDsReachRunSpec(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, nil)
|
||||
as := testAssign()
|
||||
as.Containers = append(as.Containers, bus.AssignContainer{
|
||||
Name: "trainer", Image: "registry/trainer:1", GPU: 1,
|
||||
})
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if len(rt.Runs) != 2 {
|
||||
t.Fatalf("expected two runs, got %+v", rt.Runs)
|
||||
}
|
||||
if len(rt.Runs[0].GPUIDs) != 0 {
|
||||
t.Errorf("web container: want no GPUIDs, got %+v", rt.Runs[0].GPUIDs)
|
||||
}
|
||||
trainer := rt.Runs[1]
|
||||
if len(trainer.GPUIDs) != 1 {
|
||||
t.Fatalf("trainer container: want one pinned GPUID, got %+v", trainer.GPUIDs)
|
||||
}
|
||||
if trainer.GPUIDs[0] != "GPU-aaa" && trainer.GPUIDs[0] != "GPU-bbb" {
|
||||
t.Errorf("unexpected pinned UUID %q", trainer.GPUIDs[0])
|
||||
}
|
||||
}
|
||||
|
||||
// A container with no GPU request never triggers a ReportTaskGPU call.
|
||||
func TestRunTask_NoGPUSkipsReport(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
as := testAssign()
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if reports := cp.gpuReportsFor(as.TaskID); len(reports) != 0 {
|
||||
t.Errorf("want no GPU report for a non-GPU task, got %+v", reports)
|
||||
}
|
||||
}
|
||||
|
||||
// A GPU request exceeding the ledger's free devices (discovery found none, or
|
||||
// fewer than requested) is best-effort: the container still runs, it just
|
||||
// carries no pinned UUIDs and so no CDI devices are injected.
|
||||
func TestRunTask_GPUShortLedgerStillRuns(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt) // testIdentity() has no discovered GPU UUIDs
|
||||
as := testAssign()
|
||||
as.Containers = append(as.Containers, bus.AssignContainer{
|
||||
Name: "trainer", Image: "registry/trainer:1", GPU: 2,
|
||||
})
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if len(rt.Runs) != 2 {
|
||||
t.Fatalf("expected both containers to run despite the short ledger, got %+v", rt.Runs)
|
||||
}
|
||||
if reports := cp.gpuReportsFor(as.TaskID); len(reports) != 0 {
|
||||
t.Errorf("want no GPU report when nothing was pinned, got %+v", reports)
|
||||
}
|
||||
}
|
||||
|
||||
// stopTask releases the task's pinned GPU UUIDs back to the ledger so a later
|
||||
// task on the same instance can reuse them.
|
||||
func TestStopTask_ReleasesGPUsBackToLedger(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
|
||||
{ID: "t-001-trainer", Running: true, Labels: map[string]string{
|
||||
"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "trainer",
|
||||
}},
|
||||
}}
|
||||
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa"), cp, rt, nil)
|
||||
if _, err := a.gpu.Pin(gpuKey("t-001", "trainer"), 1); err != nil {
|
||||
t.Fatalf("seed pin: %v", err)
|
||||
}
|
||||
if len(a.gpu.free) != 0 {
|
||||
t.Fatalf("setup: want ledger fully pinned, free = %v", a.gpu.free)
|
||||
}
|
||||
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
|
||||
|
||||
if len(a.gpu.free) != 1 {
|
||||
t.Errorf("want the GPU released back to the ledger, free = %v", a.gpu.free)
|
||||
}
|
||||
}
|
||||
|
||||
// A pull failure reports STOPPED and never starts the container.
|
||||
func TestRunTask_PullFailureReportsStopped(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{Err: errors.New("pull denied")}
|
||||
a := newRunAgent(cp, rt)
|
||||
a.runTask(context.Background(), testAssign())
|
||||
|
||||
if len(rt.Runs) != 0 {
|
||||
t.Errorf("container should not run after pull failure, got %+v", rt.Runs)
|
||||
}
|
||||
st := cp.taskStates()
|
||||
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
|
||||
t.Fatalf("want STOPPED, got %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// A run failure reports STOPPED.
|
||||
func TestRunTask_RunFailureReportsStopped(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{RunErr: errors.New("start boom")}
|
||||
a := newRunAgent(cp, rt)
|
||||
a.runTask(context.Background(), testAssign())
|
||||
|
||||
st := cp.taskStates()
|
||||
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
|
||||
t.Fatalf("want STOPPED, got %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// On container exit, waitContainer reports STOPPED with the exit code.
|
||||
func TestRunTask_ExitReportsStoppedWithCode(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitCode: 7}
|
||||
a := newRunAgent(cp, rt)
|
||||
a.runTask(context.Background(), testAssign())
|
||||
|
||||
deadline := time.After(time.Second)
|
||||
for {
|
||||
st := cp.taskStates()
|
||||
if len(st) > 0 && st[len(st)-1].LastStatus == bus.TaskStatusStopped {
|
||||
last := st[len(st)-1]
|
||||
if len(last.Containers) != 1 || last.Containers[0].ExitCode == nil || *last.Containers[0].ExitCode != 7 {
|
||||
t.Fatalf("want exit code 7, got %+v", last.Containers)
|
||||
}
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatalf("no STOPPED state after exit; got %+v", cp.taskStates())
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pollAssignments dispatches each assign once and acks it on the next poll.
|
||||
func TestPollAssignments_DispatchesOnceAndAcks(t *testing.T) {
|
||||
as := testAssign()
|
||||
cp := &fakeCP{pollReplies: [][]bus.Assign{
|
||||
{*as}, // poll 1: one pending assign
|
||||
{*as}, // poll 2: still present (not yet acked when poll 2 was issued)
|
||||
nil, // poll 3: gateway dropped it after the ack
|
||||
}}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go a.pollAssignments(ctx, map[string]bool{})
|
||||
|
||||
// runTask reports RUNNING through the (mutex-guarded) control plane. With the
|
||||
// assign returned on two consecutive polls, dedup means exactly one dispatch →
|
||||
// exactly one RUNNING report.
|
||||
deadline := time.After(time.Second)
|
||||
for {
|
||||
if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running >= 1 {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatal("assign was never dispatched (no RUNNING report)")
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
cancel()
|
||||
|
||||
if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running != 1 {
|
||||
t.Errorf("task dispatched %d times, want exactly 1 (dedup on taskID)", running)
|
||||
}
|
||||
// The taskID must appear in a later poll's ack list.
|
||||
acked := false
|
||||
for _, ack := range cp.acks() {
|
||||
for _, id := range ack {
|
||||
if id == as.TaskID {
|
||||
acked = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !acked {
|
||||
t.Errorf("task %s was never acked; acks=%v", as.TaskID, cp.acks())
|
||||
}
|
||||
}
|
||||
|
||||
// stopTask reaps the task's labeled containers and reports STOPPED with the
|
||||
// directive reason and container status.
|
||||
func TestStopTask_ReapsLabeledContainersAndReportsStopped(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
|
||||
{ID: "t-001-web", Running: true, Labels: map[string]string{
|
||||
"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "web",
|
||||
}},
|
||||
{ID: "t-999-other", Running: true, Labels: map[string]string{
|
||||
"mulga.ecs.taskID": "t-999", "mulga.ecs.containerName": "web",
|
||||
}},
|
||||
}}
|
||||
a := newRunAgent(cp, rt)
|
||||
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
|
||||
|
||||
if len(rt.Stopped) != 1 || rt.Stopped[0] != "t-001-web" {
|
||||
t.Fatalf("want only t-001-web stopped, got %+v", rt.Stopped)
|
||||
}
|
||||
st := cp.taskStates()
|
||||
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
|
||||
t.Fatalf("want one STOPPED report, got %+v", st)
|
||||
}
|
||||
if st[0].Reason != "bye" {
|
||||
t.Errorf("want reason %q, got %q", "bye", st[0].Reason)
|
||||
}
|
||||
if len(st[0].Containers) != 1 || st[0].Containers[0].Status != bus.TaskStatusStopped {
|
||||
t.Errorf("want one STOPPED container, got %+v", st[0].Containers)
|
||||
}
|
||||
}
|
||||
|
||||
// A stop for a task with no live containers still reports STOPPED so the
|
||||
// scheduler releases its capacity.
|
||||
func TestStopTask_NoContainersStillReportsStopped(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{}
|
||||
a := newRunAgent(cp, rt)
|
||||
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "gone"})
|
||||
|
||||
if len(rt.Removed) != 0 {
|
||||
t.Errorf("nothing to remove, got %+v", rt.Removed)
|
||||
}
|
||||
st := cp.taskStates()
|
||||
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
|
||||
t.Fatalf("want one STOPPED report, got %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// A stop delivered by poll is dispatched once, acked in the stop-ack list, and
|
||||
// suppresses a same-poll assign for the same task.
|
||||
func TestPollAssignments_StopReapsAndSuppressesAssign(t *testing.T) {
|
||||
as := testAssign()
|
||||
cp := &fakeCP{
|
||||
pollReplies: [][]bus.Assign{{*as}, {*as}, nil},
|
||||
stopPending: []bus.StopDirective{{TaskID: as.TaskID, Reason: "bye"}},
|
||||
}
|
||||
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
|
||||
{ID: "t-001-web", Running: true, Labels: map[string]string{"mulga.ecs.taskID": "t-001"}},
|
||||
}}
|
||||
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go a.pollAssignments(ctx, map[string]bool{})
|
||||
|
||||
deadline := time.After(time.Second)
|
||||
for {
|
||||
if stopped := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusStopped); stopped >= 1 {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatalf("stop never reaped; states=%+v", cp.taskStates())
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
cancel()
|
||||
|
||||
// The assign for the stopping task must never have started a container.
|
||||
if len(rt.Runs) != 0 {
|
||||
t.Errorf("assign for a stopping task should not run, got %+v", rt.Runs)
|
||||
}
|
||||
if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running != 0 {
|
||||
t.Errorf("stopping task should never report RUNNING, got %d", running)
|
||||
}
|
||||
acked := false
|
||||
for _, ack := range cp.stopAcks() {
|
||||
for _, id := range ack {
|
||||
if id == as.TaskID {
|
||||
acked = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !acked {
|
||||
t.Errorf("stop %s was never acked; stopAcks=%v", as.TaskID, cp.stopAcks())
|
||||
}
|
||||
}
|
||||
|
||||
// countStopAcks reports how many polls carried taskID in their stop-ack list.
|
||||
func countStopAcks(acks [][]string, taskID string) int {
|
||||
n := 0
|
||||
for _, ack := range acks {
|
||||
if slices.Contains(ack, taskID) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// A reap that cannot run is never acked, so the directive stays in the gateway's
|
||||
// inbox and is redelivered and retried until the runtime comes back.
|
||||
func TestPollAssignments_UnperformableStopIsNotAckedAndRetried(t *testing.T) {
|
||||
cp := &fakeCP{stopPending: []bus.StopDirective{{TaskID: "t-001", Reason: "bye"}}}
|
||||
rt := &ctrruntime.FakePuller{ListErr: errors.New("containerd is down")}
|
||||
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go a.pollAssignments(ctx, map[string]bool{})
|
||||
|
||||
deadline := time.After(time.Second)
|
||||
for rt.Lists() < 3 {
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatalf("failed reap was not retried; List calls=%d", rt.Lists())
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
cancel()
|
||||
|
||||
if n := countStopAcks(cp.stopAcks(), "t-001"); n != 0 {
|
||||
t.Errorf("a reap that never ran was acked %d times; stopAcks=%v", n, cp.stopAcks())
|
||||
}
|
||||
if stopped := countStatus(cp.taskStates(), "t-001", bus.TaskStatusStopped); stopped != 0 {
|
||||
t.Errorf("a reap that never ran reported STOPPED %d times", stopped)
|
||||
}
|
||||
}
|
||||
|
||||
// Once the runtime recovers, the redelivered directive is reaped, reported
|
||||
// STOPPED and acked exactly once, which deletes it from the inbox.
|
||||
func TestPollAssignments_RecoveredStopIsAckedExactlyOnce(t *testing.T) {
|
||||
cp := &fakeCP{stopPending: []bus.StopDirective{{TaskID: "t-001", Reason: "bye"}}}
|
||||
rt := &ctrruntime.FakePuller{
|
||||
ListErr: errors.New("containerd is down"),
|
||||
ListErrFor: 1,
|
||||
Containers: []ctrruntime.Container{
|
||||
{ID: "t-001-web", Running: true, Labels: map[string]string{"mulga.ecs.taskID": "t-001"}},
|
||||
},
|
||||
}
|
||||
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go a.pollAssignments(ctx, map[string]bool{})
|
||||
|
||||
deadline := time.After(2 * time.Second)
|
||||
for countStopAcks(cp.stopAcks(), "t-001") == 0 {
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatalf("recovered reap was never acked; states=%+v stopAcks=%v", cp.taskStates(), cp.stopAcks())
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
cancel()
|
||||
|
||||
if n := countStopAcks(cp.stopAcks(), "t-001"); n != 1 {
|
||||
t.Errorf("stop acked %d times, want exactly 1; stopAcks=%v", n, cp.stopAcks())
|
||||
}
|
||||
if stopped := countStatus(cp.taskStates(), "t-001", bus.TaskStatusStopped); stopped != 1 {
|
||||
t.Errorf("task reported STOPPED %d times, want exactly 1", stopped)
|
||||
}
|
||||
if reaped := rt.Reaped(); len(reaped) != 1 || reaped[0] != "t-001-web" {
|
||||
t.Errorf("want t-001-web reaped once, got %+v", reaped)
|
||||
}
|
||||
}
|
||||
|
||||
// A task whose reap completed is never started again by an assign the gateway
|
||||
// is still redelivering.
|
||||
func TestPollAssignments_StoppedTaskIsNotRerunByRedeliveredAssign(t *testing.T) {
|
||||
as := testAssign()
|
||||
cp := &fakeCP{
|
||||
pollReplies: [][]bus.Assign{{*as}, {*as}, {*as}, {*as}, {*as}, {*as}},
|
||||
stopPending: []bus.StopDirective{{TaskID: as.TaskID, Reason: "bye"}},
|
||||
}
|
||||
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
|
||||
{ID: "t-001-web", Running: true, Labels: map[string]string{"mulga.ecs.taskID": "t-001"}},
|
||||
}}
|
||||
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go a.pollAssignments(ctx, map[string]bool{})
|
||||
|
||||
deadline := time.After(time.Second)
|
||||
for countStopAcks(cp.stopAcks(), as.TaskID) == 0 {
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatalf("stop never completed; states=%+v", cp.taskStates())
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
// Let the remaining scripted assigns arrive after the reap has finished.
|
||||
time.Sleep(40 * time.Millisecond)
|
||||
cancel()
|
||||
|
||||
if runs := rt.RunCalls(); len(runs) != 0 {
|
||||
t.Errorf("stopped task was started by a redelivered assign: %+v", runs)
|
||||
}
|
||||
if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running != 0 {
|
||||
t.Errorf("stopped task reported RUNNING %d times", running)
|
||||
}
|
||||
}
|
||||
|
||||
// stopTask with no runtime is a no-op: nothing to reap, no state reported.
|
||||
func TestStopTask_NilRunnerNoOp(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
a := newAgent(config{}, testIdentity(), cp, nil, nil)
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "x"})
|
||||
if len(cp.taskStates()) != 0 {
|
||||
t.Fatalf("nil runner should report no state, got %+v", cp.taskStates())
|
||||
}
|
||||
}
|
||||
|
||||
// A List failure aborts the reap without reporting a (false) STOPPED state.
|
||||
func TestStopTask_ListErrorNoReport(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{ListErr: errors.New("list boom")}
|
||||
a := newRunAgent(cp, rt)
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "x"})
|
||||
if len(cp.taskStates()) != 0 {
|
||||
t.Fatalf("list error should report no state, got %+v", cp.taskStates())
|
||||
}
|
||||
}
|
||||
|
||||
// A directive with no reason reaps the matching container and reports STOPPED with
|
||||
// the default reason.
|
||||
func TestStopTask_EmptyReasonReapsWithDefault(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{{
|
||||
ID: "t-001-web",
|
||||
Labels: map[string]string{
|
||||
"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "web",
|
||||
},
|
||||
}}}
|
||||
a := newRunAgent(cp, rt)
|
||||
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001"})
|
||||
|
||||
if len(rt.Stopped) != 1 || rt.Stopped[0] != "t-001-web" {
|
||||
t.Fatalf("want container t-001-web reaped, got %+v", rt.Stopped)
|
||||
}
|
||||
st := cp.taskStates()
|
||||
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
|
||||
t.Fatalf("want one STOPPED report, got %+v", st)
|
||||
}
|
||||
if st[0].Reason != "Task stopped" {
|
||||
t.Errorf("want default reason %q, got %q", "Task stopped", st[0].Reason)
|
||||
}
|
||||
}
|
||||
|
||||
func countStatus(states []bus.TaskState, taskID, status string) int {
|
||||
n := 0
|
||||
for _, st := range states {
|
||||
if st.TaskID == taskID && st.LastStatus == status {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// awsvpc: an assign carrying an ENI MAC builds the task netns and passes its
|
||||
// path to the container RunSpec.
|
||||
func TestRunTask_AwsvpcBuildsNetns(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
f := &fakeNetRunner{linkOut: linkWithENI}
|
||||
a.netns = newTestNetns(f)
|
||||
|
||||
as := testAssign()
|
||||
as.ENIMacAddress = "52:54:00:de:ad:01"
|
||||
a.runTask(context.Background(), as)
|
||||
|
||||
if !f.sawAny("ip netns add ecs-t-001") {
|
||||
t.Fatalf("expected task netns build, got %v", f.joined())
|
||||
}
|
||||
if len(rt.Runs) != 1 || rt.Runs[0].NetnsPath != netnsPathFor("t-001") {
|
||||
t.Fatalf("want NetnsPath %q on RunSpec, got %+v", netnsPathFor("t-001"), rt.Runs)
|
||||
}
|
||||
}
|
||||
|
||||
// bridge/host: no ENI MAC means no netns is built and the container runs in the
|
||||
// host (VM) netns (empty NetnsPath).
|
||||
func TestRunTask_NoMacSkipsNetns(t *testing.T) {
|
||||
cp := &fakeCP{}
|
||||
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
|
||||
a := newRunAgent(cp, rt)
|
||||
f := &fakeNetRunner{linkOut: linkWithENI}
|
||||
a.netns = newTestNetns(f)
|
||||
|
||||
a.runTask(context.Background(), testAssign())
|
||||
|
||||
if len(f.joined()) != 0 {
|
||||
t.Fatalf("expected no netns commands, got %v", f.joined())
|
||||
}
|
||||
if len(rt.Runs) != 1 || rt.Runs[0].NetnsPath != "" {
|
||||
t.Fatalf("want empty NetnsPath, got %+v", rt.Runs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContainerID(t *testing.T) {
|
||||
if got := containerID("t-1", "web"); got != "t-1-web" {
|
||||
t.Errorf("containerID = %q, want t-1-web", got)
|
||||
}
|
||||
}
|
||||
|
||||
// sentinelResolver is an identity-comparable Resolver for pullResolver tests.
|
||||
type sentinelResolver struct{}
|
||||
|
||||
func (sentinelResolver) Authorize(context.Context, string) (string, string, string, error) {
|
||||
return "", "", "", nil
|
||||
}
|
||||
|
||||
// TestPullResolver verifies that an assign with no execution role (or no
|
||||
// credential endpoint) uses the instance resolver; with both, a distinct
|
||||
// exec-role resolver is built.
|
||||
func TestPullResolver(t *testing.T) {
|
||||
fake := sentinelResolver{}
|
||||
a := newAgent(config{Region: "us-east-1"}, testIdentity(), &fakeCP{}, nil, fake)
|
||||
|
||||
if got := a.pullResolver(&bus.Assign{TaskID: "t1"}); got != fake {
|
||||
t.Fatalf("no execution role: want instance resolver fallback")
|
||||
}
|
||||
execAssign := &bus.Assign{TaskID: "t1", ExecutionRoleARN: "arn:aws:iam::1:role/e"}
|
||||
if got := a.pullResolver(execAssign); got != fake {
|
||||
t.Fatalf("nil cred endpoint: want instance resolver fallback")
|
||||
}
|
||||
|
||||
a.cred = &credEndpoint{}
|
||||
got := a.pullResolver(execAssign)
|
||||
if got == nil || got == fake {
|
||||
t.Fatalf("execution role + cred endpoint: want distinct exec-role resolver, got %v", got)
|
||||
}
|
||||
}
|
||||
+17
-170
@@ -1,196 +1,43 @@
|
||||
// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
|
||||
// control-plane resource from the host through the AWS gateway, instead of
|
||||
// dialing core NATS directly. It is the read-side companion to
|
||||
// eks-gateway-publish.
|
||||
//
|
||||
// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
|
||||
// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
|
||||
// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
|
||||
// as tab-separated lines the on-VM addon-sync shell agent consumes without a
|
||||
// JSON parser:
|
||||
//
|
||||
// <addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
|
||||
//
|
||||
// The configuration values are base64-encoded so embedded tabs/newlines cannot
|
||||
// break the line framing.
|
||||
// control-plane resource from the host through the AWS gateway, writing it to
|
||||
// stdout as TSV; the implementation is in agents/eks/gatewayfetch.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// eks-gateway-fetch -resource addons # GET /clusters/{cluster}/internal-addons/{accountId}
|
||||
// eks-gateway-fetch -resource recovery -instance-id i-… # GET /clusters/{cluster}/internal-recovery/{accountId}/{instanceId}
|
||||
//
|
||||
// The recovery resource returns a single epoch\taction\tsnapshot line the on-VM
|
||||
// k3s-recovery agent applies before k3s starts (etcd cluster-reset / wipe-rejoin).
|
||||
//
|
||||
// Flags default to environment variables seeded by cloud-init:
|
||||
// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
|
||||
// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME, EKS_INSTANCE_ID.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/eksgw"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
)
|
||||
|
||||
// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
|
||||
// here to keep this tiny VM binary free of the handler package's dependency
|
||||
// graph.
|
||||
type stagedAddon struct {
|
||||
AddonName string `json:"addonName"`
|
||||
AddonVersion string `json:"addonVersion"`
|
||||
ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
|
||||
ConfigurationValues string `json:"configurationValues"`
|
||||
}
|
||||
|
||||
type internalAddonsResponse struct {
|
||||
Addons []stagedAddon `json:"addons"`
|
||||
}
|
||||
|
||||
const (
|
||||
maxAttempts = 30
|
||||
retryDelay = 5 * time.Second
|
||||
"github.com/mulgadc/spinifex/spinifex/agents/eks/gatewayfetch"
|
||||
)
|
||||
|
||||
func main() {
|
||||
var (
|
||||
gatewayURL string
|
||||
gatewayCA string
|
||||
accessKey string
|
||||
secretKey string
|
||||
region string
|
||||
accountID string
|
||||
clusterName string
|
||||
resource string
|
||||
instanceID string
|
||||
)
|
||||
|
||||
flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
|
||||
flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
|
||||
flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
|
||||
flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
|
||||
flag.StringVar(®ion, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
|
||||
flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
|
||||
flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
|
||||
flag.StringVar(&instanceID, "instance-id", os.Getenv("EKS_INSTANCE_ID"), "This member's EC2 instance ID (recovery resource)")
|
||||
flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons | recovery")
|
||||
cfg := gatewayfetch.Config{Out: os.Stdout}
|
||||
flag.StringVar(&cfg.GatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
|
||||
flag.StringVar(&cfg.GatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
|
||||
flag.StringVar(&cfg.AccessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
|
||||
flag.StringVar(&cfg.SecretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
|
||||
flag.StringVar(&cfg.Region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
|
||||
flag.StringVar(&cfg.AccountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
|
||||
flag.StringVar(&cfg.ClusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
|
||||
flag.StringVar(&cfg.InstanceID, "instance-id", os.Getenv("EKS_INSTANCE_ID"), "This member's EC2 instance ID (recovery resource)")
|
||||
flag.StringVar(&cfg.Resource, "resource", "addons", "Resource to fetch: addons | recovery")
|
||||
flag.Parse()
|
||||
|
||||
switch {
|
||||
case accountID == "":
|
||||
fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
|
||||
case clusterName == "":
|
||||
fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
|
||||
case resource != "addons" && resource != "recovery":
|
||||
fatal("--resource must be addons or recovery")
|
||||
case resource == "recovery" && instanceID == "":
|
||||
fatal("--instance-id is required for --resource recovery (or set EKS_INSTANCE_ID)")
|
||||
if err := gatewayfetch.Run(context.Background(), cfg); err != nil {
|
||||
slog.Error("eks-gateway-fetch: " + err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
|
||||
if err != nil {
|
||||
fatal(fmt.Sprintf("build gateway client: %v", err))
|
||||
}
|
||||
|
||||
var (
|
||||
path string
|
||||
emit func(io.Writer, []byte) error
|
||||
)
|
||||
switch resource {
|
||||
case "recovery":
|
||||
path = "/clusters/" + clusterName + "/internal-recovery/" + accountID + "/" + instanceID
|
||||
emit = emitRecoveryTSV
|
||||
default:
|
||||
path = "/clusters/" + clusterName + "/internal-addons/" + accountID
|
||||
emit = emitAddonsTSV
|
||||
}
|
||||
|
||||
var lastErr error
|
||||
for attempt := 1; attempt <= maxAttempts; attempt++ {
|
||||
body, err := client.Get(path)
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
|
||||
if attempt < maxAttempts {
|
||||
time.Sleep(retryDelay)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if werr := emit(os.Stdout, body); werr != nil {
|
||||
fatal(fmt.Sprintf("render response: %v", werr))
|
||||
}
|
||||
return
|
||||
}
|
||||
fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
|
||||
}
|
||||
|
||||
// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
|
||||
// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
|
||||
// every locally-rendered manifest), which is the correct steady state for a
|
||||
// cluster with no managed add-ons.
|
||||
func emitAddonsTSV(out io.Writer, body []byte) error {
|
||||
var resp internalAddonsResponse
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return fmt.Errorf("unmarshal internal-addons response: %w", err)
|
||||
}
|
||||
w := bufio.NewWriter(out)
|
||||
for _, a := range resp.Addons {
|
||||
cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
|
||||
if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return w.Flush()
|
||||
}
|
||||
|
||||
// recoveryDirective mirrors the gateway's RecoveryDirective wire shape; duplicated
|
||||
// here to keep this tiny VM binary free of the handler package's dependency graph.
|
||||
type recoveryDirective struct {
|
||||
Epoch int64 `json:"epoch"`
|
||||
Action string `json:"action"`
|
||||
Snapshot string `json:"snapshot"`
|
||||
SnapshotRequired bool `json:"snapshotRequired"`
|
||||
}
|
||||
|
||||
type internalRecoveryResponse struct {
|
||||
Directive recoveryDirective `json:"directive"`
|
||||
}
|
||||
|
||||
// emitRecoveryTSV decodes the recovery directive and writes a single
|
||||
// epoch\taction\tsnapshot\tsnapshotRequired line the on-VM k3s-recovery agent
|
||||
// consumes without a JSON parser. An unset action defaults to "none" (steady
|
||||
// state); snapshotRequired is 1 when the snapshot MUST restore (fresh DR seed
|
||||
// aborts boot on fetch failure) else 0.
|
||||
func emitRecoveryTSV(out io.Writer, body []byte) error {
|
||||
var resp internalRecoveryResponse
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return fmt.Errorf("unmarshal internal-recovery response: %w", err)
|
||||
}
|
||||
d := resp.Directive
|
||||
if d.Action == "" {
|
||||
d.Action = "none"
|
||||
}
|
||||
required := "0"
|
||||
if d.SnapshotRequired {
|
||||
required = "1"
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "%d\t%s\t%s\t%s\n", d.Epoch, d.Action, d.Snapshot, required); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fatal(msg string) {
|
||||
slog.Error("eks-gateway-fetch: " + msg)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/base64"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEmitAddonsTSV(t *testing.T) {
|
||||
body := []byte(`{"addons":[
|
||||
{"addonName":"spinifex-noop","addonVersion":"0.1.0"},
|
||||
{"addonName":"aws-load-balancer-controller","addonVersion":"2.8.1",
|
||||
"serviceAccountRoleArn":"arn:aws:iam::111122223333:role/alb",
|
||||
"configurationValues":"{\"replicaCount\":2}"}
|
||||
]}`)
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := emitAddonsTSV(&buf, body); err != nil {
|
||||
t.Fatalf("emitAddonsTSV: %v", err)
|
||||
}
|
||||
cfg := base64.StdEncoding.EncodeToString([]byte(`{"replicaCount":2}`))
|
||||
want := "spinifex-noop\t0.1.0\t\t\n" +
|
||||
"aws-load-balancer-controller\t2.8.1\tarn:aws:iam::111122223333:role/alb\t" + cfg + "\n"
|
||||
if buf.String() != want {
|
||||
t.Fatalf("TSV mismatch:\n got: %q\nwant: %q", buf.String(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmitAddonsTSV_Empty(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
if err := emitAddonsTSV(&buf, []byte(`{"addons":[]}`)); err != nil {
|
||||
t.Fatalf("emitAddonsTSV: %v", err)
|
||||
}
|
||||
if buf.Len() != 0 {
|
||||
t.Fatalf("empty addon set must emit nothing, got %q", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmitAddonsTSV_BadJSON(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
if err := emitAddonsTSV(&buf, []byte(`{not-json`)); err == nil {
|
||||
t.Fatal("expected error on malformed JSON")
|
||||
}
|
||||
}
|
||||
+17
-108
@@ -1,13 +1,6 @@
|
||||
// eks-gateway-publish runs inside the EKS K3s control-plane VM and relays a
|
||||
// single publication to the host through the AWS gateway, instead of dialing
|
||||
// core NATS directly.
|
||||
//
|
||||
// It reads a JSON payload on stdin, wraps it as
|
||||
// {accountId, channel, kind, payload}, SigV4-signs (service "eks") an HTTPS
|
||||
// POST to {gateway}/clusters/{cluster}/internal-publish, and retries with
|
||||
// backoff until the gateway returns 2xx or the attempt budget is exhausted —
|
||||
// so a degraded link surfaces as a non-zero exit rather than a silently
|
||||
// dropped message (the failure mode of fire-and-forget `nats pub`).
|
||||
// single JSON publication from stdin to the host through the AWS gateway; the
|
||||
// implementation is in agents/eks/gatewaypublish.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
@@ -20,115 +13,31 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/eksgw"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
)
|
||||
|
||||
const (
|
||||
// Retry budget: the control plane reaches readiness before this runs, so a
|
||||
// failing POST means a degraded link, not a cold start. Bounded so a stuck
|
||||
// boot still terminates the OpenRC service.
|
||||
maxAttempts = 30
|
||||
retryDelay = 5 * time.Second
|
||||
"github.com/mulgadc/spinifex/spinifex/agents/eks/gatewaypublish"
|
||||
)
|
||||
|
||||
type publishBody struct {
|
||||
AccountID string `json:"accountId"`
|
||||
Channel string `json:"channel"`
|
||||
Kind string `json:"kind"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
var (
|
||||
gatewayURL string
|
||||
gatewayCA string
|
||||
accessKey string
|
||||
secretKey string
|
||||
region string
|
||||
accountID string
|
||||
clusterName string
|
||||
channel string
|
||||
kind string
|
||||
)
|
||||
|
||||
flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
|
||||
flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
|
||||
flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
|
||||
flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
|
||||
flag.StringVar(®ion, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
|
||||
flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
|
||||
flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
|
||||
flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state|addon")
|
||||
flag.StringVar(&kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
|
||||
cfg := gatewaypublish.Config{Payload: os.Stdin}
|
||||
flag.StringVar(&cfg.GatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
|
||||
flag.StringVar(&cfg.GatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
|
||||
flag.StringVar(&cfg.AccessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
|
||||
flag.StringVar(&cfg.SecretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
|
||||
flag.StringVar(&cfg.Region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
|
||||
flag.StringVar(&cfg.AccountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
|
||||
flag.StringVar(&cfg.ClusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
|
||||
flag.StringVar(&cfg.Channel, "channel", "", "Publish channel: bootstrap|state|addon")
|
||||
flag.StringVar(&cfg.Kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
|
||||
flag.Parse()
|
||||
|
||||
switch {
|
||||
case accountID == "":
|
||||
fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
|
||||
case clusterName == "":
|
||||
fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
|
||||
case channel != "bootstrap" && channel != "state" && channel != "addon":
|
||||
fatal("--channel must be bootstrap, state or addon")
|
||||
case channel == "bootstrap" && kind == "":
|
||||
fatal("--kind is required for the bootstrap channel")
|
||||
if err := gatewaypublish.Run(context.Background(), cfg); err != nil {
|
||||
slog.Error("eks-gateway-publish: " + err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
payload, err := io.ReadAll(os.Stdin)
|
||||
if err != nil {
|
||||
fatal(fmt.Sprintf("read stdin payload: %v", err))
|
||||
}
|
||||
if len(bytes.TrimSpace(payload)) == 0 {
|
||||
fatal("empty stdin payload")
|
||||
}
|
||||
if !json.Valid(payload) {
|
||||
fatal("stdin payload is not valid JSON")
|
||||
}
|
||||
|
||||
body, err := json.Marshal(publishBody{
|
||||
AccountID: accountID,
|
||||
Channel: channel,
|
||||
Kind: kind,
|
||||
Payload: json.RawMessage(payload),
|
||||
})
|
||||
if err != nil {
|
||||
fatal(fmt.Sprintf("marshal request body: %v", err))
|
||||
}
|
||||
|
||||
client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
|
||||
if err != nil {
|
||||
fatal(fmt.Sprintf("build gateway client: %v", err))
|
||||
}
|
||||
path := "/clusters/" + clusterName + "/internal-publish"
|
||||
|
||||
var lastErr error
|
||||
for attempt := 1; attempt <= maxAttempts; attempt++ {
|
||||
if _, err := client.Post(path, body); err != nil {
|
||||
lastErr = err
|
||||
slog.Warn("eks-gateway-publish: attempt failed",
|
||||
"channel", channel, "kind", kind, "attempt", attempt, "err", err)
|
||||
if attempt < maxAttempts {
|
||||
time.Sleep(retryDelay)
|
||||
}
|
||||
continue
|
||||
}
|
||||
slog.Info("eks-gateway-publish: published", "channel", channel, "kind", kind)
|
||||
return
|
||||
}
|
||||
fatal(fmt.Sprintf("publish failed after %d attempts: %v", maxAttempts, lastErr))
|
||||
}
|
||||
|
||||
func fatal(msg string) {
|
||||
slog.Error("eks-gateway-publish: " + msg)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
@@ -1,16 +1,6 @@
|
||||
// Command eks-konnectivity-cert mints, then reuses, an ECDSA-P256 serving
|
||||
// certificate for the on-node konnectivity-server, signed by the K3s cluster CA.
|
||||
//
|
||||
// The konnectivity-agent (DaemonSet on workers) verifies the server it dials
|
||||
// using the in-pod ServiceAccount CA (/var/run/secrets/.../ca.crt), which in
|
||||
// K3s is the same server-ca that signs the apiserver. Signing the konnectivity
|
||||
// serving cert with that CA is therefore what lets the agent validate the
|
||||
// reverse-tunnel endpoint with no separate CA distribution.
|
||||
//
|
||||
// The eks-node AMI carries no openssl, so this is the Go equivalent: load the
|
||||
// K3s server CA cert+key, issue a leaf (ServerAuth) cert carrying the cluster's
|
||||
// konnectivity SANs (the NLB private-endpoint IP, public endpoint IP, NLB DNS),
|
||||
// and cache it under -dir so the OpenRC service is byte-stable across restarts.
|
||||
// Command eks-konnectivity-cert mints, then reuses, a K3s-CA-signed serving
|
||||
// certificate for the on-node konnectivity-server; the implementation is in
|
||||
// agents/eks/konnectivitycert.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
@@ -23,225 +13,27 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/pem"
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"math/big"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/agents/eks/konnectivitycert"
|
||||
)
|
||||
|
||||
func main() {
|
||||
dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
|
||||
caCert := flag.String("ca-cert", "", "PEM path of the signing (K3s server) CA cert")
|
||||
caKey := flag.String("ca-key", "", "PEM path of the signing CA private key")
|
||||
cn := flag.String("cn", "konnectivity-server", "certificate common name")
|
||||
sansCSV := flag.String("sans", "", "comma-separated SANs (IP or DNS)")
|
||||
cfg := konnectivitycert.Config{Out: os.Stdout}
|
||||
flag.StringVar(&cfg.Dir, "dir", "", "directory to cache tls.crt/tls.key in")
|
||||
flag.StringVar(&cfg.CACert, "ca-cert", "", "PEM path of the signing (K3s server) CA cert")
|
||||
flag.StringVar(&cfg.CAKey, "ca-key", "", "PEM path of the signing CA private key")
|
||||
flag.StringVar(&cfg.CN, "cn", "konnectivity-server", "certificate common name")
|
||||
flag.StringVar(&cfg.SANs, "sans", "", "comma-separated SANs (IP or DNS)")
|
||||
flag.Parse()
|
||||
|
||||
if *dir == "" || *caCert == "" || *caKey == "" || *sansCSV == "" {
|
||||
fatal("eks-konnectivity-cert: -dir, -ca-cert, -ca-key and -sans are required")
|
||||
if err := konnectivitycert.Run(context.Background(), cfg); err != nil {
|
||||
slog.Error("eks-konnectivity-cert: " + err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
dns, ips := splitSANs(*sansCSV)
|
||||
if len(dns) == 0 && len(ips) == 0 {
|
||||
fatal("eks-konnectivity-cert: -sans lists no usable names")
|
||||
}
|
||||
|
||||
certPath, keyPath, err := ensureCert(*dir, *caCert, *caKey, *cn, dns, ips)
|
||||
if err != nil {
|
||||
fatal("eks-konnectivity-cert: " + err.Error())
|
||||
}
|
||||
fmt.Printf("%s\t%s\n", certPath, keyPath)
|
||||
}
|
||||
|
||||
// ensureCert returns the cached cert/key paths under dir, minting and persisting
|
||||
// a fresh CA-signed pair when none is present, readable, and still SAN-current.
|
||||
func ensureCert(dir, caCertPath, caKeyPath, cn string, dns []string, ips []net.IP) (string, string, error) {
|
||||
certPath := filepath.Join(dir, "tls.crt")
|
||||
keyPath := filepath.Join(dir, "tls.key")
|
||||
|
||||
if certCoversSANs(certPath, keyPath, dns, ips) {
|
||||
return certPath, keyPath, nil
|
||||
}
|
||||
|
||||
caCert, caKey, err := loadCA(caCertPath, caKeyPath)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
certPEM, keyPEM, err := issueLeaf(caCert, caKey, cn, dns, ips)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
if err = os.MkdirAll(dir, 0o700); err != nil {
|
||||
return "", "", fmt.Errorf("mkdir %s: %w", dir, err)
|
||||
}
|
||||
if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
|
||||
return "", "", fmt.Errorf("write %s: %w", certPath, err)
|
||||
}
|
||||
if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
|
||||
return "", "", fmt.Errorf("write %s: %w", keyPath, err)
|
||||
}
|
||||
return certPath, keyPath, nil
|
||||
}
|
||||
|
||||
// certCoversSANs reports whether the cached cert exists, parses, has a key, and
|
||||
// already carries every requested SAN — so a cert minted before the endpoint
|
||||
// set changed is re-issued rather than served stale.
|
||||
func certCoversSANs(certPath, keyPath string, dns []string, ips []net.IP) bool {
|
||||
c, err := os.ReadFile(certPath)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if k, kerr := os.ReadFile(keyPath); kerr != nil || len(k) == 0 {
|
||||
return false
|
||||
}
|
||||
der := decodeFirst(c)
|
||||
if der == nil {
|
||||
return false
|
||||
}
|
||||
crt, err := x509.ParseCertificate(der)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
have := make(map[string]struct{}, len(crt.DNSNames)+len(crt.IPAddresses))
|
||||
for _, n := range crt.DNSNames {
|
||||
have[n] = struct{}{}
|
||||
}
|
||||
for _, ip := range crt.IPAddresses {
|
||||
have[ip.String()] = struct{}{}
|
||||
}
|
||||
for _, n := range dns {
|
||||
if _, ok := have[n]; !ok {
|
||||
return false
|
||||
}
|
||||
}
|
||||
for _, ip := range ips {
|
||||
if _, ok := have[ip.String()]; !ok {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// loadCA parses the PEM CA cert + EC/PKCS8 private key used to sign the leaf.
|
||||
func loadCA(certPath, keyPath string) (*x509.Certificate, *ecdsa.PrivateKey, error) {
|
||||
certPEM, err := os.ReadFile(certPath)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("read ca cert %s: %w", certPath, err)
|
||||
}
|
||||
der := decodeFirst(certPEM)
|
||||
if der == nil {
|
||||
return nil, nil, fmt.Errorf("ca cert %s: no PEM block", certPath)
|
||||
}
|
||||
caCert, err := x509.ParseCertificate(der)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("parse ca cert: %w", err)
|
||||
}
|
||||
|
||||
keyPEM, err := os.ReadFile(keyPath)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("read ca key %s: %w", keyPath, err)
|
||||
}
|
||||
keyDER := decodeFirst(keyPEM)
|
||||
if keyDER == nil {
|
||||
return nil, nil, fmt.Errorf("ca key %s: no PEM block", keyPath)
|
||||
}
|
||||
caKey, err := parseECKey(keyDER)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return caCert, caKey, nil
|
||||
}
|
||||
|
||||
// parseECKey accepts the EC SEC1 or PKCS8 encodings K3s may write for the CA key.
|
||||
func parseECKey(der []byte) (*ecdsa.PrivateKey, error) {
|
||||
if k, err := x509.ParseECPrivateKey(der); err == nil {
|
||||
return k, nil
|
||||
}
|
||||
k8, err := x509.ParsePKCS8PrivateKey(der)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse ca key (sec1 and pkcs8 failed): %w", err)
|
||||
}
|
||||
ec, ok := k8.(*ecdsa.PrivateKey)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("ca key is %T, want ecdsa", k8)
|
||||
}
|
||||
return ec, nil
|
||||
}
|
||||
|
||||
// issueLeaf mints an ECDSA-P256 ServerAuth leaf for cn + SANs, signed by caCert/caKey.
|
||||
func issueLeaf(caCert *x509.Certificate, caKey *ecdsa.PrivateKey, cn string, dns []string, ips []net.IP) (certPEM, keyPEM []byte, err error) {
|
||||
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("generate key: %w", err)
|
||||
}
|
||||
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("serial: %w", err)
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
tmpl := x509.Certificate{
|
||||
SerialNumber: serial,
|
||||
Subject: pkix.Name{CommonName: cn},
|
||||
NotBefore: now.Add(-time.Hour),
|
||||
NotAfter: now.AddDate(10, 0, 0),
|
||||
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
|
||||
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
||||
BasicConstraintsValid: true,
|
||||
DNSNames: dns,
|
||||
IPAddresses: ips,
|
||||
}
|
||||
der, err := x509.CreateCertificate(rand.Reader, &tmpl, caCert, &priv.PublicKey, caKey)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("create cert: %w", err)
|
||||
}
|
||||
keyDER, err := x509.MarshalECPrivateKey(priv)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("marshal key: %w", err)
|
||||
}
|
||||
certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
||||
keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
|
||||
return certPEM, keyPEM, nil
|
||||
}
|
||||
|
||||
// splitSANs partitions the CSV into DNS names and IPs (an entry that parses as an
|
||||
// IP goes to IPAddresses, else DNSNames), matching how SAN matching is typed.
|
||||
func splitSANs(s string) (dns []string, ips []net.IP) {
|
||||
for p := range strings.SplitSeq(s, ",") {
|
||||
p = strings.TrimSpace(p)
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
if ip := net.ParseIP(p); ip != nil {
|
||||
ips = append(ips, ip)
|
||||
continue
|
||||
}
|
||||
dns = append(dns, p)
|
||||
}
|
||||
return dns, ips
|
||||
}
|
||||
|
||||
// decodeFirst returns the DER bytes of the first PEM block, or nil.
|
||||
func decodeFirst(pemBytes []byte) []byte {
|
||||
blk, _ := pem.Decode(pemBytes)
|
||||
if blk == nil {
|
||||
return nil
|
||||
}
|
||||
return blk.Bytes
|
||||
}
|
||||
|
||||
func fatal(msg string) {
|
||||
slog.Error(msg)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
@@ -1,153 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/pem"
|
||||
"math/big"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// writeTestCA mints a self-signed EC CA (cert + SEC1 key PEM) into dir, returning
|
||||
// the two paths — a stand-in for K3s' /var/lib/rancher/k3s/server/tls/server-ca.*.
|
||||
func writeTestCA(t *testing.T, dir string) (certPath, keyPath string) {
|
||||
t.Helper()
|
||||
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
t.Fatalf("gen ca key: %v", err)
|
||||
}
|
||||
tmpl := x509.Certificate{
|
||||
SerialNumber: big.NewInt(1),
|
||||
Subject: pkix.Name{CommonName: "k3s-server-ca"},
|
||||
NotBefore: time.Now().Add(-time.Hour),
|
||||
NotAfter: time.Now().AddDate(10, 0, 0),
|
||||
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature,
|
||||
BasicConstraintsValid: true,
|
||||
IsCA: true,
|
||||
}
|
||||
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
|
||||
if err != nil {
|
||||
t.Fatalf("create ca cert: %v", err)
|
||||
}
|
||||
keyDER, err := x509.MarshalECPrivateKey(priv)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal ca key: %v", err)
|
||||
}
|
||||
certPath = filepath.Join(dir, "server-ca.crt")
|
||||
keyPath = filepath.Join(dir, "server-ca.key")
|
||||
if err := os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
|
||||
t.Fatalf("write ca cert: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}), 0o600); err != nil {
|
||||
t.Fatalf("write ca key: %v", err)
|
||||
}
|
||||
return certPath, keyPath
|
||||
}
|
||||
|
||||
func leafFrom(t *testing.T, certPath string) *x509.Certificate {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(certPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read leaf: %v", err)
|
||||
}
|
||||
crt, err := x509.ParseCertificate(decodeFirst(b))
|
||||
if err != nil {
|
||||
t.Fatalf("parse leaf: %v", err)
|
||||
}
|
||||
return crt
|
||||
}
|
||||
|
||||
func TestEnsureCertSignsLeafWithSANsThatChainsToCA(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
caCert, caKey := writeTestCA(t, dir)
|
||||
|
||||
dns, ips := splitSANs("10.32.100.4,203.0.113.9,eks-toc.example")
|
||||
certPath, keyPath, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips)
|
||||
if err != nil {
|
||||
t.Fatalf("ensureCert: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(keyPath); err != nil {
|
||||
t.Fatalf("key not written: %v", err)
|
||||
}
|
||||
|
||||
leaf := leafFrom(t, certPath)
|
||||
if leaf.IsCA {
|
||||
t.Error("konnectivity serving cert must be a leaf, not a CA")
|
||||
}
|
||||
// The agent dials the NLB private-endpoint IP and verifies via the in-pod CA,
|
||||
// so the IP must be a SAN or the tunnel TLS handshake fails.
|
||||
if err := leaf.VerifyHostname("10.32.100.4"); err != nil {
|
||||
t.Errorf("IP SAN 10.32.100.4 missing: %v", err)
|
||||
}
|
||||
|
||||
// The leaf must chain to the K3s server CA — that CA is the in-pod ca.crt the
|
||||
// agent trusts, so no separate CA distribution is needed.
|
||||
caPEM, _ := os.ReadFile(caCert)
|
||||
roots := x509.NewCertPool()
|
||||
if !roots.AppendCertsFromPEM(caPEM) {
|
||||
t.Fatal("append CA failed")
|
||||
}
|
||||
if _, err := leaf.Verify(x509.VerifyOptions{
|
||||
Roots: roots,
|
||||
DNSName: "eks-toc.example",
|
||||
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
||||
}); err != nil {
|
||||
t.Errorf("leaf does not chain to the K3s server CA: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureCertReusesThenReissuesOnSANChange(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
caCert, caKey := writeTestCA(t, dir)
|
||||
|
||||
dns, ips := splitSANs("10.32.100.4,eks-toc.example")
|
||||
c1, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips)
|
||||
if err != nil {
|
||||
t.Fatalf("first ensureCert: %v", err)
|
||||
}
|
||||
first, _ := os.ReadFile(c1)
|
||||
|
||||
// Same SANs → cached pair reused byte-for-byte (a churning cert would bounce
|
||||
// the konnectivity-server on every restart).
|
||||
if _, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips); err != nil {
|
||||
t.Fatalf("second ensureCert: %v", err)
|
||||
}
|
||||
if second, _ := os.ReadFile(c1); string(first) != string(second) {
|
||||
t.Error("ensureCert re-minted instead of reusing the SAN-current cached cert")
|
||||
}
|
||||
|
||||
// A new endpoint in the SAN set must force a re-issue.
|
||||
dns2, ips2 := splitSANs("10.32.100.4,eks-toc.example,10.32.100.99")
|
||||
if _, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns2, ips2); err != nil {
|
||||
t.Fatalf("third ensureCert: %v", err)
|
||||
}
|
||||
if third, _ := os.ReadFile(c1); string(first) == string(third) {
|
||||
t.Error("ensureCert served a stale cert that does not cover the new SAN")
|
||||
}
|
||||
if err := leafFrom(t, c1).VerifyHostname("10.32.100.99"); err != nil {
|
||||
t.Errorf("re-issued cert missing new SAN: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitSANsPartitionsIPsAndDNS(t *testing.T) {
|
||||
dns, ips := splitSANs(" 10.0.0.1 , host.example , , 2001:db8::1 ")
|
||||
if len(dns) != 1 || dns[0] != "host.example" {
|
||||
t.Errorf("dns = %v, want [host.example]", dns)
|
||||
}
|
||||
wantIPs := []net.IP{net.ParseIP("10.0.0.1"), net.ParseIP("2001:db8::1")}
|
||||
if len(ips) != len(wantIPs) {
|
||||
t.Fatalf("ips = %v, want %v", ips, wantIPs)
|
||||
}
|
||||
for i, ip := range ips {
|
||||
if !ip.Equal(wantIPs[i]) {
|
||||
t.Errorf("ips[%d] = %v, want %v", i, ip, wantIPs[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/base64"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"net"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/bluebottle/pkg/tlsconfig"
|
||||
)
|
||||
|
||||
// serverTLSConfig is the webhook's loopback serving TLS config.
|
||||
func serverTLSConfig(cert tls.Certificate) *tls.Config {
|
||||
return &tls.Config{
|
||||
Certificates: []tls.Certificate{cert},
|
||||
MinVersion: tls.VersionTLS13,
|
||||
CurvePreferences: tlsconfig.Curves,
|
||||
}
|
||||
}
|
||||
|
||||
// ensureServingCert loads the persisted self-signed serving cert/key, or mints
|
||||
// a fresh ECDSA-P256 pair (SAN 127.0.0.1 / localhost) and persists it. Reusing
|
||||
// across restarts keeps the CA the apiserver loaded into its webhook kubeconfig
|
||||
// valid even if the webhook process is restarted. Returns the parsed cert and
|
||||
// the certificate PEM (for embedding as the kubeconfig CA bundle).
|
||||
func ensureServingCert(certPath, keyPath string) (tls.Certificate, []byte, error) {
|
||||
// Fall through to regenerate on a missing/corrupt/incompatible persisted
|
||||
// pair. The cert PEM is re-read because tls.Certificate keeps only DER.
|
||||
if cert, err := tls.LoadX509KeyPair(certPath, keyPath); err == nil {
|
||||
if certPEM, err := os.ReadFile(certPath); err == nil {
|
||||
return cert, certPEM, nil
|
||||
}
|
||||
}
|
||||
|
||||
certPEM, keyPEM, err := generateSelfSigned()
|
||||
if err != nil {
|
||||
return tls.Certificate{}, nil, err
|
||||
}
|
||||
if err := os.WriteFile(certPath, certPEM, 0o644); err != nil {
|
||||
return tls.Certificate{}, nil, fmt.Errorf("write cert %s: %w", certPath, err)
|
||||
}
|
||||
if err := os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
|
||||
return tls.Certificate{}, nil, fmt.Errorf("write key %s: %w", keyPath, err)
|
||||
}
|
||||
cert, err := tls.X509KeyPair(certPEM, keyPEM)
|
||||
if err != nil {
|
||||
return tls.Certificate{}, nil, fmt.Errorf("parse minted keypair: %w", err)
|
||||
}
|
||||
return cert, certPEM, nil
|
||||
}
|
||||
|
||||
func generateSelfSigned() (certPEM, keyPEM []byte, err error) {
|
||||
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("generate key: %w", err)
|
||||
}
|
||||
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("serial: %w", err)
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
tmpl := x509.Certificate{
|
||||
SerialNumber: serial,
|
||||
Subject: pkix.Name{CommonName: "eks-token-webhook"},
|
||||
NotBefore: now.Add(-time.Hour),
|
||||
NotAfter: now.AddDate(10, 0, 0),
|
||||
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
|
||||
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
||||
BasicConstraintsValid: true,
|
||||
IsCA: true,
|
||||
IPAddresses: []net.IP{net.IPv4(127, 0, 0, 1), net.IPv6loopback},
|
||||
DNSNames: []string{"localhost"},
|
||||
}
|
||||
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("create cert: %w", err)
|
||||
}
|
||||
keyDER, err := x509.MarshalECPrivateKey(priv)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("marshal key: %w", err)
|
||||
}
|
||||
certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
||||
keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
|
||||
return certPEM, keyPEM, nil
|
||||
}
|
||||
|
||||
// writeAPIServerKubeconfig writes the kubeconfig kube-apiserver loads via
|
||||
// --authentication-token-webhook-config-file. It points the apiserver at the
|
||||
// loopback webhook and trusts its self-signed serving cert as the CA bundle.
|
||||
func writeAPIServerKubeconfig(path, addr string, certPEM []byte) error {
|
||||
caB64 := base64.StdEncoding.EncodeToString(certPEM)
|
||||
kubeconfig := fmt.Sprintf(`apiVersion: v1
|
||||
kind: Config
|
||||
clusters:
|
||||
- name: mulga-eks-token-webhook
|
||||
cluster:
|
||||
certificate-authority-data: %s
|
||||
server: https://%s/authenticate
|
||||
users:
|
||||
- name: kube-apiserver
|
||||
contexts:
|
||||
- name: webhook
|
||||
context:
|
||||
cluster: mulga-eks-token-webhook
|
||||
user: kube-apiserver
|
||||
current-context: webhook
|
||||
`, caB64, addr)
|
||||
if err := os.WriteFile(path, []byte(kubeconfig), 0o600); err != nil {
|
||||
return fmt.Errorf("write %s: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+24
-179
@@ -1,76 +1,45 @@
|
||||
// eks-token-webhook is the K8s TokenReview webhook authenticator baked into
|
||||
// the eks-server AMI. Per kubectl call the kube-apiserver POSTs the bearer
|
||||
// token (produced by `aws eks get-token`); the webhook relays it through the
|
||||
// AWS gateway broker (SigV4 HTTPS POST to /clusters/{name}/token-review), which
|
||||
// host-side runs the STS verify (cross-cluster x-k8s-aws-id pin) + AccessEntry
|
||||
// KV lookup and returns the resolved identity. The webhook never speaks core
|
||||
// NATS — same gateway-broker model as ELBv2's lb-agent and eks-gateway-publish.
|
||||
//
|
||||
// It binds 127.0.0.1 only; the kube-apiserver is the sole client (loopback).
|
||||
// On startup it writes the apiserver webhook kubeconfig (with its self-signed
|
||||
// serving CA) so k3s can be pointed at it via
|
||||
// --authentication-token-webhook-config-file.
|
||||
//
|
||||
// SigV4 creds come from the AWS SDK chain (IMDS instance-role) when static
|
||||
// EKS_ACCESS_KEY/EKS_SECRET_KEY are absent, matching the sibling helpers.
|
||||
// Command eks-token-webhook is the K8s TokenReview webhook authenticator baked
|
||||
// into the eks-server AMI; the implementation is in agents/eks/tokenwebhook.
|
||||
// It reads its configuration from the first-boot env cloud-init seeds onto the
|
||||
// VM and binds loopback only.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/eksgw"
|
||||
handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/agents/eks/tokenwebhook"
|
||||
)
|
||||
|
||||
// config is the webhook's runtime configuration, sourced from the first-boot
|
||||
// env (/etc/spinifex-eks/first-boot.env) cloud-init seeds onto the VM.
|
||||
type config struct {
|
||||
addr string
|
||||
gatewayURL string
|
||||
gatewayCA string
|
||||
accessKey string
|
||||
secretKey string
|
||||
region string
|
||||
accountID string
|
||||
clusterName string
|
||||
certPath string
|
||||
keyPath string
|
||||
kubeconfig string
|
||||
}
|
||||
|
||||
func loadConfig() (config, error) {
|
||||
c := config{
|
||||
addr: flagAddr,
|
||||
gatewayURL: os.Getenv("EKS_GATEWAY_URL"),
|
||||
gatewayCA: os.Getenv("EKS_GATEWAY_CA"),
|
||||
accessKey: os.Getenv("EKS_ACCESS_KEY"),
|
||||
secretKey: os.Getenv("EKS_SECRET_KEY"),
|
||||
region: os.Getenv("EKS_REGION"),
|
||||
accountID: os.Getenv("EKS_ACCOUNT_ID"),
|
||||
clusterName: os.Getenv("EKS_CLUSTER_NAME"),
|
||||
certPath: envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
|
||||
keyPath: envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
|
||||
kubeconfig: envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
|
||||
func loadConfig() (tokenwebhook.Config, error) {
|
||||
c := tokenwebhook.Config{
|
||||
Addr: flagAddr,
|
||||
GatewayURL: os.Getenv("EKS_GATEWAY_URL"),
|
||||
GatewayCA: os.Getenv("EKS_GATEWAY_CA"),
|
||||
AccessKey: os.Getenv("EKS_ACCESS_KEY"),
|
||||
SecretKey: os.Getenv("EKS_SECRET_KEY"),
|
||||
Region: os.Getenv("EKS_REGION"),
|
||||
AccountID: os.Getenv("EKS_ACCOUNT_ID"),
|
||||
ClusterName: os.Getenv("EKS_CLUSTER_NAME"),
|
||||
CertPath: envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
|
||||
KeyPath: envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
|
||||
Kubeconfig: envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
|
||||
}
|
||||
// Static SigV4 creds are optional: when absent, eksgw.New signs with the
|
||||
// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's
|
||||
// sibling helpers use. The CP VM launches on an instance profile, so
|
||||
// buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY by design.
|
||||
switch {
|
||||
case c.gatewayURL == "":
|
||||
case c.GatewayURL == "":
|
||||
return c, errors.New("EKS_GATEWAY_URL not set")
|
||||
case c.accountID == "":
|
||||
case c.AccountID == "":
|
||||
return c, errors.New("EKS_ACCOUNT_ID not set")
|
||||
case c.clusterName == "":
|
||||
case c.ClusterName == "":
|
||||
return c, errors.New("EKS_CLUSTER_NAME not set")
|
||||
}
|
||||
return c, nil
|
||||
@@ -97,132 +66,8 @@ func main() {
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if err := run(cfg); err != nil {
|
||||
if err := tokenwebhook.Run(context.Background(), cfg); err != nil {
|
||||
slog.Error("eks-token-webhook fatal", "err", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func run(cfg config) error {
|
||||
// Serving cert + apiserver kubeconfig. Persisted and reused across restarts
|
||||
// so a webhook crash does not invalidate the CA the apiserver already loaded.
|
||||
tlsCert, certPEM, err := ensureServingCert(cfg.certPath, cfg.keyPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("serving cert: %w", err)
|
||||
}
|
||||
if err := writeAPIServerKubeconfig(cfg.kubeconfig, cfg.addr, certPEM); err != nil {
|
||||
return fmt.Errorf("write apiserver kubeconfig: %w", err)
|
||||
}
|
||||
|
||||
client, err := eksgw.New(cfg.gatewayURL, cfg.gatewayCA, cfg.accessKey, cfg.secretKey, cfg.region)
|
||||
if err != nil {
|
||||
return fmt.Errorf("build gateway client: %w", err)
|
||||
}
|
||||
|
||||
authr := &authenticator{
|
||||
accountID: cfg.accountID,
|
||||
clusterName: cfg.clusterName,
|
||||
review: gatewayReviewer(client, cfg.clusterName, cfg.accountID),
|
||||
}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte("ok"))
|
||||
})
|
||||
mux.HandleFunc("/authenticate", authr.handle)
|
||||
|
||||
server := &http.Server{
|
||||
Addr: cfg.addr,
|
||||
Handler: mux,
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
TLSConfig: serverTLSConfig(tlsCert),
|
||||
}
|
||||
slog.Info("eks-token-webhook listening", "addr", cfg.addr, "cluster", cfg.clusterName)
|
||||
if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
return fmt.Errorf("serve: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// tokenReviewRequest is the body POSTed to /clusters/{name}/token-review. The
|
||||
// webhook holds system SigV4 creds (system account, not the cluster account),
|
||||
// so accountId names the cluster account explicitly — same as PublishInternal.
|
||||
type tokenReviewRequest struct {
|
||||
AccountID string `json:"accountId"`
|
||||
Token string `json:"token"`
|
||||
}
|
||||
|
||||
// gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
|
||||
// the gateway and decode the resolved identity. A non-2xx / transport error is
|
||||
// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
|
||||
// silent deny, distinguishing "webhook broker down" from "token rejected".
|
||||
func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (handlers_eks.WebhookTokenReviewResult, error) {
|
||||
path := "/clusters/" + clusterName + "/token-review"
|
||||
return func(token string) (handlers_eks.WebhookTokenReviewResult, error) {
|
||||
body, err := json.Marshal(tokenReviewRequest{AccountID: accountID, Token: token})
|
||||
if err != nil {
|
||||
return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("marshal request: %w", err)
|
||||
}
|
||||
respBody, err := client.Post(path, body)
|
||||
if err != nil {
|
||||
return handlers_eks.WebhookTokenReviewResult{}, err
|
||||
}
|
||||
var res handlers_eks.WebhookTokenReviewResult
|
||||
if err := json.Unmarshal(respBody, &res); err != nil {
|
||||
return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("decode response: %w", err)
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
}
|
||||
|
||||
// authenticator resolves a TokenReview by relaying to the gateway broker.
|
||||
type authenticator struct {
|
||||
accountID string
|
||||
clusterName string
|
||||
review func(token string) (handlers_eks.WebhookTokenReviewResult, error)
|
||||
}
|
||||
|
||||
func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
var review tokenReview
|
||||
if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
res, err := a.review(review.Spec.Token)
|
||||
if err != nil {
|
||||
// Broker fault (gateway down, transport error): a 5xx tells the apiserver
|
||||
// to retry rather than treating a transient outage as a hard deny.
|
||||
slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
|
||||
http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
|
||||
status := tokenReviewStatus{Authenticated: res.Authenticated}
|
||||
if res.Authenticated {
|
||||
status.User = userInfo{
|
||||
Username: res.Username,
|
||||
UID: res.UID,
|
||||
Groups: res.Groups,
|
||||
}
|
||||
// Confirm the token for the audiences the apiserver asked about; an empty
|
||||
// status.audiences is rejected when --api-audiences is configured.
|
||||
status.Audiences = review.Spec.Audiences
|
||||
}
|
||||
slog.Info("TokenReview decision", "authenticated", status.Authenticated, "username", status.User.Username, "audiences", status.Audiences)
|
||||
|
||||
resp := tokenReview{
|
||||
APIVersion: "authentication.k8s.io/v1",
|
||||
Kind: "TokenReview",
|
||||
Status: status,
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if err := json.NewEncoder(w).Encode(resp); err != nil {
|
||||
slog.Error("encode TokenReview response", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,127 +1,12 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// postReview drives authenticator.handle with a TokenReview carrying token and
|
||||
// audiences, returning the decoded response status and HTTP code.
|
||||
func postReview(t *testing.T, a *authenticator, token string, audiences []string) (tokenReviewStatus, int) {
|
||||
t.Helper()
|
||||
body, err := json.Marshal(tokenReview{
|
||||
APIVersion: "authentication.k8s.io/v1",
|
||||
Kind: "TokenReview",
|
||||
Spec: tokenReviewSpec{Token: token, Audiences: audiences},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/authenticate", strings.NewReader(string(body)))
|
||||
w := httptest.NewRecorder()
|
||||
a.handle(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
return tokenReviewStatus{}, w.Code
|
||||
}
|
||||
var resp tokenReview
|
||||
require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp))
|
||||
return resp.Status, w.Code
|
||||
}
|
||||
|
||||
func TestHandle_GrantsAndStampsAudiences(t *testing.T) {
|
||||
a := &authenticator{
|
||||
accountID: "111122223333",
|
||||
clusterName: "alpha",
|
||||
review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
|
||||
return handlers_eks.WebhookTokenReviewResult{
|
||||
Authenticated: true,
|
||||
Username: "arn:aws:iam::111122223333:role/admin",
|
||||
UID: "AROAEXAMPLE:session",
|
||||
Groups: []string{"system:masters"},
|
||||
}, nil
|
||||
},
|
||||
}
|
||||
|
||||
status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
|
||||
require.Equal(t, http.StatusOK, code)
|
||||
require.True(t, status.Authenticated)
|
||||
assert.Equal(t, "arn:aws:iam::111122223333:role/admin", status.User.Username)
|
||||
assert.Equal(t, "AROAEXAMPLE:session", status.User.UID)
|
||||
assert.Equal(t, []string{"system:masters"}, status.User.Groups)
|
||||
assert.Equal(t, []string{"https://kubernetes.default.svc"}, status.Audiences)
|
||||
}
|
||||
|
||||
func TestHandle_DeniesWithoutStampingAudiences(t *testing.T) {
|
||||
a := &authenticator{
|
||||
clusterName: "alpha",
|
||||
review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
|
||||
return handlers_eks.WebhookTokenReviewResult{Authenticated: false}, nil
|
||||
},
|
||||
}
|
||||
|
||||
status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
|
||||
require.Equal(t, http.StatusOK, code)
|
||||
assert.False(t, status.Authenticated)
|
||||
assert.Empty(t, status.User.Username)
|
||||
assert.Empty(t, status.Audiences, "unauthenticated review must not stamp audiences")
|
||||
}
|
||||
|
||||
// A broker fault is a 5xx (retryable), not a silent deny — a transient gateway
|
||||
// outage must not look identical to a rejected token.
|
||||
func TestHandle_BrokerFaultReturns5xx(t *testing.T) {
|
||||
a := &authenticator{
|
||||
clusterName: "alpha",
|
||||
review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
|
||||
return handlers_eks.WebhookTokenReviewResult{}, errors.New("gateway returned 503")
|
||||
},
|
||||
}
|
||||
|
||||
_, code := postReview(t, a, "tok", nil)
|
||||
assert.Equal(t, http.StatusServiceUnavailable, code)
|
||||
}
|
||||
|
||||
func TestEnsureServingCert_PersistsAndReuses(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
certPath := filepath.Join(dir, "wh.crt")
|
||||
keyPath := filepath.Join(dir, "wh.key")
|
||||
|
||||
c1, pem1, err := ensureServingCert(certPath, keyPath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, pem1)
|
||||
require.NotNil(t, serverTLSConfig(c1))
|
||||
|
||||
// Second call reuses the persisted pair (same cert bytes).
|
||||
_, pem2, err := ensureServingCert(certPath, keyPath)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, pem1, pem2)
|
||||
}
|
||||
|
||||
func TestWriteAPIServerKubeconfig(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "wh.kubeconfig")
|
||||
certPEM := []byte("-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n")
|
||||
|
||||
require.NoError(t, writeAPIServerKubeconfig(path, "127.0.0.1:8443", certPEM))
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
require.NoError(t, err)
|
||||
body := string(data)
|
||||
assert.Contains(t, body, "server: https://127.0.0.1:8443/authenticate")
|
||||
assert.Contains(t, body, base64.StdEncoding.EncodeToString(certPEM))
|
||||
assert.Contains(t, body, "current-context: webhook")
|
||||
}
|
||||
|
||||
// The CP VM launches on an instance profile, so cloud-init omits the static
|
||||
// EKS_ACCESS_KEY/EKS_SECRET_KEY and eksgw.New signs with IMDS creds. loadConfig
|
||||
// must accept absent static keys; gating on them aborts every IMDS-mode boot,
|
||||
@@ -148,8 +33,8 @@ func TestLoadConfig_AcceptsAbsentStaticCreds(t *testing.T) {
|
||||
})
|
||||
cfg, err := loadConfig()
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, cfg.accessKey)
|
||||
assert.Empty(t, cfg.secretKey)
|
||||
assert.Empty(t, cfg.AccessKey)
|
||||
assert.Empty(t, cfg.SecretKey)
|
||||
|
||||
// Genuinely-required vars are still enforced.
|
||||
setEnv(t, map[string]string{
|
||||
|
||||
@@ -1,34 +0,0 @@
|
||||
package main
|
||||
|
||||
// Minimal authentication.k8s.io/v1 TokenReview wire types. Hand-rolled to avoid
|
||||
// pulling k8s.io/api into the spinifex module for these few fields; the shapes
|
||||
// match what kube-apiserver sends and expects on the webhook contract.
|
||||
|
||||
type tokenReview struct {
|
||||
APIVersion string `json:"apiVersion"`
|
||||
Kind string `json:"kind"`
|
||||
Spec tokenReviewSpec `json:"spec,omitzero"`
|
||||
Status tokenReviewStatus `json:"status"`
|
||||
}
|
||||
|
||||
type tokenReviewSpec struct {
|
||||
Token string `json:"token"`
|
||||
Audiences []string `json:"audiences,omitempty"`
|
||||
}
|
||||
|
||||
type tokenReviewStatus struct {
|
||||
Authenticated bool `json:"authenticated"`
|
||||
User userInfo `json:"user,omitzero"`
|
||||
// Audiences echoes the request's spec.audiences on success. When the
|
||||
// apiserver runs with --api-audiences, it sends spec.audiences and discards
|
||||
// an authenticated=true response that does not confirm a matching audience —
|
||||
// so an empty Audiences here surfaces as a 401 despite a valid principal.
|
||||
Audiences []string `json:"audiences,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type userInfo struct {
|
||||
Username string `json:"username,omitempty"`
|
||||
UID string `json:"uid,omitempty"`
|
||||
Groups []string `json:"groups,omitempty"`
|
||||
}
|
||||
+13
-127
@@ -1,12 +1,6 @@
|
||||
// Command eks-webhook-cert mints, then reuses, a self-signed ECDSA-P256 serving
|
||||
// certificate for an in-cluster admission webhook and prints its base64 PEM so
|
||||
// the addon renderer can embed it as the webhook Secret + each caBundle.
|
||||
//
|
||||
// The eks-node AMI carries no openssl, so this is the Go equivalent of the
|
||||
// upstream Helm chart's install-time genSelfSignedCert: one cert per cluster,
|
||||
// cached on disk under -dir, stable across addon-sync ticks. The single cert is
|
||||
// its own CA (IsCA + ServerAuth), so the same PEM serves as both the TLS leaf
|
||||
// and the caBundle the apiserver trusts.
|
||||
// Command eks-webhook-cert mints, then reuses, a self-signed serving certificate
|
||||
// for an in-cluster admission webhook and prints its base64 PEM; the
|
||||
// implementation is in agents/eks/webhookcert.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
@@ -17,133 +11,25 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/base64"
|
||||
"encoding/pem"
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"math/big"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/agents/eks/webhookcert"
|
||||
)
|
||||
|
||||
func main() {
|
||||
dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
|
||||
cn := flag.String("cn", "", "certificate common name")
|
||||
dnsCSV := flag.String("dns", "", "comma-separated DNS SANs")
|
||||
cfg := webhookcert.Config{Out: os.Stdout}
|
||||
flag.StringVar(&cfg.Dir, "dir", "", "directory to cache tls.crt/tls.key in")
|
||||
flag.StringVar(&cfg.CN, "cn", "", "certificate common name")
|
||||
flag.StringVar(&cfg.DNS, "dns", "", "comma-separated DNS SANs")
|
||||
flag.Parse()
|
||||
|
||||
if *dir == "" || *cn == "" || *dnsCSV == "" {
|
||||
fatal("eks-webhook-cert: -dir, -cn and -dns are required")
|
||||
if err := webhookcert.Run(context.Background(), cfg); err != nil {
|
||||
slog.Error("eks-webhook-cert: " + err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
dnsNames := splitCSV(*dnsCSV)
|
||||
if len(dnsNames) == 0 {
|
||||
fatal("eks-webhook-cert: -dns lists no names")
|
||||
}
|
||||
|
||||
certPEM, keyPEM, err := ensureCert(*dir, *cn, dnsNames)
|
||||
if err != nil {
|
||||
fatal("eks-webhook-cert: " + err.Error())
|
||||
}
|
||||
b := base64.StdEncoding.EncodeToString
|
||||
fmt.Printf("%s\t%s\t%s\n", b(certPEM), b(certPEM), b(keyPEM))
|
||||
}
|
||||
|
||||
// ensureCert returns the cached cert/key under dir, minting and persisting a
|
||||
// fresh self-signed pair (for cn + dnsNames) when none is present or readable.
|
||||
func ensureCert(dir, cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
|
||||
certPath := filepath.Join(dir, "tls.crt")
|
||||
keyPath := filepath.Join(dir, "tls.key")
|
||||
|
||||
if c, cerr := os.ReadFile(certPath); cerr == nil {
|
||||
if k, kerr := os.ReadFile(keyPath); kerr == nil {
|
||||
if _, perr := x509.ParseCertificate(decodeFirst(c)); perr == nil && len(k) > 0 {
|
||||
return c, k, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
certPEM, keyPEM, err = generateSelfSigned(cn, dnsNames)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if err = os.MkdirAll(dir, 0o700); err != nil {
|
||||
return nil, nil, fmt.Errorf("mkdir %s: %w", dir, err)
|
||||
}
|
||||
if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
|
||||
return nil, nil, fmt.Errorf("write %s: %w", certPath, err)
|
||||
}
|
||||
if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
|
||||
return nil, nil, fmt.Errorf("write %s: %w", keyPath, err)
|
||||
}
|
||||
return certPEM, keyPEM, nil
|
||||
}
|
||||
|
||||
// generateSelfSigned mints an ECDSA-P256 self-signed cert valid as both a CA and
|
||||
// a server cert, with the given common name and DNS SANs.
|
||||
func generateSelfSigned(cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
|
||||
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("generate key: %w", err)
|
||||
}
|
||||
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("serial: %w", err)
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
tmpl := x509.Certificate{
|
||||
SerialNumber: serial,
|
||||
Subject: pkix.Name{CommonName: cn},
|
||||
NotBefore: now.Add(-time.Hour),
|
||||
NotAfter: now.AddDate(10, 0, 0),
|
||||
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
|
||||
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
||||
BasicConstraintsValid: true,
|
||||
IsCA: true,
|
||||
DNSNames: dnsNames,
|
||||
}
|
||||
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("create cert: %w", err)
|
||||
}
|
||||
keyDER, err := x509.MarshalECPrivateKey(priv)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("marshal key: %w", err)
|
||||
}
|
||||
certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
||||
keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
|
||||
return certPEM, keyPEM, nil
|
||||
}
|
||||
|
||||
// decodeFirst returns the DER bytes of the first PEM block, or nil.
|
||||
func decodeFirst(pemBytes []byte) []byte {
|
||||
blk, _ := pem.Decode(pemBytes)
|
||||
if blk == nil {
|
||||
return nil
|
||||
}
|
||||
return blk.Bytes
|
||||
}
|
||||
|
||||
func splitCSV(s string) []string {
|
||||
var out []string
|
||||
for p := range strings.SplitSeq(s, ",") {
|
||||
if p = strings.TrimSpace(p); p != "" {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func fatal(msg string) {
|
||||
slog.Error(msg)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/x509"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const (
|
||||
testCN = "aws-load-balancer-webhook-service.kube-system.svc"
|
||||
testDNS = "aws-load-balancer-webhook-service.kube-system.svc.cluster.local"
|
||||
)
|
||||
|
||||
func parseLeaf(t *testing.T, certPEM []byte) *x509.Certificate {
|
||||
t.Helper()
|
||||
leaf, err := x509.ParseCertificate(decodeFirst(certPEM))
|
||||
if err != nil {
|
||||
t.Fatalf("parse cert: %v", err)
|
||||
}
|
||||
return leaf
|
||||
}
|
||||
|
||||
func TestEnsureCertMintsServingCert(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
certPEM, keyPEM, err := ensureCert(dir, testCN, []string{testCN, testDNS})
|
||||
if err != nil {
|
||||
t.Fatalf("ensureCert: %v", err)
|
||||
}
|
||||
if len(keyPEM) == 0 {
|
||||
t.Fatal("empty key PEM")
|
||||
}
|
||||
leaf := parseLeaf(t, certPEM)
|
||||
|
||||
// Both webhook service DNS names must be present, or the apiserver's TLS
|
||||
// dial to the webhook fails SAN verification.
|
||||
for _, want := range []string{testCN, testDNS} {
|
||||
if err := leaf.VerifyHostname(want); err != nil {
|
||||
t.Errorf("SAN %q missing: %v", want, err)
|
||||
}
|
||||
}
|
||||
if !leaf.IsCA {
|
||||
t.Error("cert must be a CA so the same PEM serves as the webhook caBundle")
|
||||
}
|
||||
|
||||
// The single self-signed cert must verify against itself as a root — this is
|
||||
// exactly what the apiserver does with caBundle == the serving cert.
|
||||
roots := x509.NewCertPool()
|
||||
roots.AddCert(leaf)
|
||||
if _, err := leaf.Verify(x509.VerifyOptions{
|
||||
Roots: roots,
|
||||
DNSName: testCN,
|
||||
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
||||
}); err != nil {
|
||||
t.Errorf("self-signed cert does not verify as its own CA: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureCertReusesCachedPair(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cert1, key1, err := ensureCert(dir, testCN, []string{testCN})
|
||||
if err != nil {
|
||||
t.Fatalf("first ensureCert: %v", err)
|
||||
}
|
||||
cert2, key2, err := ensureCert(dir, testCN, []string{testCN})
|
||||
if err != nil {
|
||||
t.Fatalf("second ensureCert: %v", err)
|
||||
}
|
||||
if string(cert1) != string(cert2) || string(key1) != string(key2) {
|
||||
t.Error("ensureCert regenerated instead of reusing the cached pair; a churning cert re-applies the addon every tick")
|
||||
}
|
||||
|
||||
// The private key must not be world-readable on disk.
|
||||
info, err := os.Stat(filepath.Join(dir, "tls.key"))
|
||||
if err != nil {
|
||||
t.Fatalf("stat key: %v", err)
|
||||
}
|
||||
if info.Mode().Perm()&0o077 != 0 {
|
||||
t.Errorf("tls.key mode %o is group/other-accessible", info.Mode().Perm())
|
||||
}
|
||||
}
|
||||
@@ -1,461 +0,0 @@
|
||||
// Package autoinstall reads headless install parameters from environment
|
||||
// variables exported by spinifex-init.sh from the kernel cmdline.
|
||||
// Parameters are set in the GRUB "Headless" menu entry in grub.cfg.
|
||||
package autoinstall
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/installer/install"
|
||||
)
|
||||
|
||||
// listDisks is the block-device scan, indirected so tests can drive selection
|
||||
// without real hardware.
|
||||
var listDisks = install.ListDisks
|
||||
|
||||
// Load returns an install.Config built from SPINIFEX_* environment variables,
|
||||
// or nil if SPINIFEX_AUTO is not set to "1" (interactive mode).
|
||||
func Load() (*install.Config, error) {
|
||||
if os.Getenv("SPINIFEX_AUTO") != "1" {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
slog.Info("autoinstall: headless mode enabled via kernel cmdline")
|
||||
|
||||
cfg, err := buildConfig()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("autoinstall config: %w", err)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// EjectAndReboot ejects the boot device (best-effort) then reboots.
|
||||
func EjectAndReboot() {
|
||||
srcDev, _ := os.ReadFile("/run/iso-dev")
|
||||
if dev := strings.TrimSpace(string(srcDev)); dev != "" {
|
||||
slog.Info("autoinstall: ejecting boot device", "device", dev)
|
||||
_ = exec.Command("eject", dev).Run()
|
||||
}
|
||||
|
||||
fmt.Println()
|
||||
fmt.Println("Installation complete.")
|
||||
fmt.Println("Remove the USB drive now if it was not ejected automatically.")
|
||||
fmt.Println("Rebooting in 10 seconds...")
|
||||
time.Sleep(10 * time.Second)
|
||||
// Use the kernel syscall directly — the live environment runs spinifex-init
|
||||
// as PID 1 (not systemd), so exec("reboot") fails trying to reach D-Bus.
|
||||
if err := syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART); err != nil {
|
||||
slog.Error("autoinstall: reboot syscall failed", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
func buildConfig() (*install.Config, error) {
|
||||
password := os.Getenv("SPINIFEX_PASSWORD")
|
||||
if password == "" {
|
||||
return nil, fmt.Errorf("SPINIFEX_PASSWORD is required")
|
||||
}
|
||||
|
||||
hostname := os.Getenv("SPINIFEX_HOSTNAME")
|
||||
if hostname == "" {
|
||||
hostname = "spinifex-node"
|
||||
}
|
||||
|
||||
storage, err := resolveStorage()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("storage: %w", err)
|
||||
}
|
||||
|
||||
wanIface, err := resolveNIC(os.Getenv("SPINIFEX_WAN_IFACE"), "")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("WAN NIC: %w", err)
|
||||
}
|
||||
|
||||
cfg := &install.Config{
|
||||
Storage: storage,
|
||||
Hostname: hostname,
|
||||
RootPassword: password,
|
||||
// Optional on the headless path — empty means no --email passed to
|
||||
// spx admin init, which omits the operator identity from telemetry.
|
||||
Email: strings.TrimSpace(os.Getenv("SPINIFEX_EMAIL")),
|
||||
}
|
||||
|
||||
cfg.WAN, err = parseRole("WAN", wanIface)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cfg.WAN.DHCPMode {
|
||||
cfg.WAN.Gateway = ""
|
||||
} else if cfg.WAN.Gateway == "" {
|
||||
return nil, fmt.Errorf("SPINIFEX_WAN_IP, SPINIFEX_WAN_MASK, SPINIFEX_WAN_GW required for static mode")
|
||||
}
|
||||
|
||||
// lan and vpc are optional: an unset _IFACE leaves the role folded, which
|
||||
// is how a single- or two-NIC node collapses onto the planes above it.
|
||||
for _, r := range []struct {
|
||||
plane string
|
||||
dst *install.NetworkRole
|
||||
}{{"LAN", &cfg.LAN}, {"VPC", &cfg.VPC}} {
|
||||
name := os.Getenv("SPINIFEX_" + r.plane + "_IFACE")
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
iface, err := resolveNIC(name, wanIface)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s NIC: %w", r.plane, err)
|
||||
}
|
||||
if *r.dst, err = parseRole(r.plane, iface); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Only the wan plane installs a default route.
|
||||
r.dst.Gateway = ""
|
||||
}
|
||||
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return nil, fmt.Errorf("network roles: %w", err)
|
||||
}
|
||||
|
||||
if os.Getenv("SPINIFEX_GPU_PASSTHROUGH") == "1" {
|
||||
cfg.GPUPassthrough = true
|
||||
}
|
||||
|
||||
if os.Getenv("SPINIFEX_SKIP_FORMATION") == "1" {
|
||||
cfg.SkipFormation = true
|
||||
}
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// resolveStorage builds the disk configuration from the kernel cmdline.
|
||||
//
|
||||
// SPINIFEX_FS ext4 (default) or one of the zfs-* topologies
|
||||
// SPINIFEX_DISKS explicit ordered member list, required for multi-disk pools
|
||||
// SPINIFEX_DISK the OS disk, as before
|
||||
// SPINIFEX_DISK_SPINIFEX ext4 only: dedicated disk for /var/lib/spinifex
|
||||
// SPINIFEX_DISK_PREDASTORE ext4 only: dedicated disk for /var/lib/spinifex/predastore
|
||||
func resolveStorage() (install.DiskConfig, error) {
|
||||
cfg := install.DiskConfig{FS: install.FSExt4}
|
||||
if v := strings.TrimSpace(os.Getenv("SPINIFEX_FS")); v != "" {
|
||||
fs, err := install.ParseFSType(v)
|
||||
if err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
cfg.FS = fs
|
||||
}
|
||||
|
||||
var err error
|
||||
switch list := strings.TrimSpace(os.Getenv("SPINIFEX_DISKS")); {
|
||||
case list != "":
|
||||
if cfg.Disks, err = disksByName(strings.Split(list, ",")); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
case cfg.FS.MinDisks() > 1:
|
||||
// No auto-selection for a multi-disk pool. Choosing which disks to
|
||||
// erase is not a decision an unattended install gets to make, and the
|
||||
// member order determines the RAID10 pairing.
|
||||
return cfg, fmt.Errorf("%s needs at least %d disks — set SPINIFEX_DISKS to an explicit comma-separated list",
|
||||
cfg.FS.Label(), cfg.FS.MinDisks())
|
||||
default:
|
||||
d, derr := resolveDisk(os.Getenv("SPINIFEX_DISK"))
|
||||
if derr != nil {
|
||||
return cfg, derr
|
||||
}
|
||||
cfg.Disks = []install.Disk{d}
|
||||
}
|
||||
|
||||
if err := applyDiskRoles(&cfg); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
if cfg.ZFS, err = parseZFSOpts(); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
slog.Info("autoinstall: storage selected", "fs", cfg.FS, "disks", cfg.Paths())
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// roleVars maps each optional data role to the variable that assigns it. The OS
|
||||
// role is not here: it always comes from SPINIFEX_DISK.
|
||||
var roleVars = []struct {
|
||||
role install.DiskRole
|
||||
env string
|
||||
}{
|
||||
{install.RoleSpinifex, "SPINIFEX_DISK_SPINIFEX"},
|
||||
{install.RolePredastore, "SPINIFEX_DISK_PREDASTORE"},
|
||||
}
|
||||
|
||||
// applyDiskRoles attaches dedicated data drives to an ext4 install. Roles are
|
||||
// always set on ext4, even with no extra drives, so the OS assignment is
|
||||
// explicit rather than implied by position.
|
||||
func applyDiskRoles(cfg *install.DiskConfig) error {
|
||||
if cfg.FS.IsZFS() {
|
||||
for _, rv := range roleVars {
|
||||
if strings.TrimSpace(os.Getenv(rv.env)) != "" {
|
||||
return fmt.Errorf("%s applies to ext4 only — %s spans every member already", rv.env, cfg.FS.Label())
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if len(cfg.Disks) != 1 {
|
||||
return fmt.Errorf("ext4 takes one OS disk, %d given — set SPINIFEX_DISK, and SPINIFEX_DISK_SPINIFEX or SPINIFEX_DISK_PREDASTORE for dedicated drives",
|
||||
len(cfg.Disks))
|
||||
}
|
||||
|
||||
roles := []install.RoleMount{{Role: install.RoleOS, Disk: cfg.Disks[0]}}
|
||||
for _, rv := range roleVars {
|
||||
spec := strings.TrimSpace(os.Getenv(rv.env))
|
||||
if spec == "" {
|
||||
continue
|
||||
}
|
||||
// Resolved with the same fail-closed selector as the OS disk, so an
|
||||
// ambiguous value lists the candidates rather than picking one.
|
||||
d, err := resolveDisk(spec)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %w", rv.env, err)
|
||||
}
|
||||
roles = append(roles, install.RoleMount{Role: rv.role, Disk: d})
|
||||
}
|
||||
*cfg = cfg.WithRoles(roles)
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseZFSOpts reads the advanced pool tunables. Every one is optional; unset
|
||||
// values are computed from the hardware at install time.
|
||||
func parseZFSOpts() (install.ZFSOpts, error) {
|
||||
var o install.ZFSOpts
|
||||
ints := []struct {
|
||||
env string
|
||||
dst *int
|
||||
}{
|
||||
{"SPINIFEX_ZFS_ASHIFT", &o.Ashift},
|
||||
{"SPINIFEX_ZFS_COPIES", &o.Copies},
|
||||
{"SPINIFEX_ZFS_ARC_MAX_MIB", &o.ARCMaxMiB},
|
||||
}
|
||||
for _, f := range ints {
|
||||
v := strings.TrimSpace(os.Getenv(f.env))
|
||||
if v == "" {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.Atoi(v)
|
||||
if err != nil {
|
||||
return o, fmt.Errorf("%s: %w", f.env, err)
|
||||
}
|
||||
*f.dst = n
|
||||
}
|
||||
o.Compress = strings.TrimSpace(os.Getenv("SPINIFEX_ZFS_COMPRESS"))
|
||||
o.Checksum = strings.TrimSpace(os.Getenv("SPINIFEX_ZFS_CHECKSUM"))
|
||||
return o, nil
|
||||
}
|
||||
|
||||
// disksByName resolves an explicit member list. Names may be kernel names
|
||||
// ("sdb"), device paths or by-id paths; the order given is preserved because
|
||||
// RAID10 pairs members in it.
|
||||
func disksByName(names []string) ([]install.Disk, error) {
|
||||
available, err := listDisks()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]install.Disk, 0, len(names))
|
||||
for _, raw := range names {
|
||||
name := strings.TrimSpace(raw)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
// A by-id path or a symlink resolves to the kernel device the scan
|
||||
// reported, so both spellings name the same disk.
|
||||
resolved := name
|
||||
if !strings.HasPrefix(resolved, "/dev/") {
|
||||
resolved = "/dev/" + resolved
|
||||
}
|
||||
if target, err := filepath.EvalSymlinks(resolved); err == nil {
|
||||
resolved = target
|
||||
}
|
||||
|
||||
idx := slices.IndexFunc(available, func(d install.Disk) bool { return d.Path == resolved })
|
||||
if idx < 0 {
|
||||
return nil, fmt.Errorf("%q is not an available disk — found:\n%s", name, diskList(available))
|
||||
}
|
||||
out = append(out, available[idx])
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// resolveDisk maps the SPINIFEX_DISK value from grub.cfg to a disk.
|
||||
//
|
||||
// Supported values:
|
||||
//
|
||||
// auto — use the only non-removable disk; fail if multiple found
|
||||
// largest — largest non-removable disk (explicit opt-in)
|
||||
// smallest — smallest non-removable disk (typical OS-on-SSD pattern)
|
||||
// nvme — the only NVMe disk; fail if multiple found
|
||||
// /dev/sda (etc.) — exact device path; fail if not present
|
||||
func resolveDisk(target string) (install.Disk, error) {
|
||||
disks, err := candidateDisks()
|
||||
if err != nil {
|
||||
return install.Disk{}, err
|
||||
}
|
||||
switch strings.ToLower(strings.TrimSpace(target)) {
|
||||
case "", "auto":
|
||||
if len(disks) != 1 {
|
||||
return install.Disk{}, fmt.Errorf(
|
||||
"expected exactly one disk, found %d — set SPINIFEX_DISK to one of:\n%s\n"+
|
||||
" largest (largest disk)\n"+
|
||||
" smallest (smallest disk)\n"+
|
||||
" nvme (NVMe only)\n"+
|
||||
" /dev/sdX (exact path)",
|
||||
len(disks), diskList(disks))
|
||||
}
|
||||
return disks[0], nil
|
||||
case "largest":
|
||||
return pickBySize(disks, true)
|
||||
case "smallest":
|
||||
return pickBySize(disks, false)
|
||||
case "nvme":
|
||||
nvmes := slices.DeleteFunc(slices.Clone(disks), func(d install.Disk) bool {
|
||||
return !strings.HasPrefix(filepath.Base(d.Path), "nvme")
|
||||
})
|
||||
if len(nvmes) != 1 {
|
||||
return install.Disk{}, fmt.Errorf("expected exactly one NVMe disk, found %d:\n%s",
|
||||
len(nvmes), diskList(nvmes))
|
||||
}
|
||||
return nvmes[0], nil
|
||||
default:
|
||||
found, err := disksByName([]string{target})
|
||||
if err != nil {
|
||||
return install.Disk{}, err
|
||||
}
|
||||
return found[0], nil
|
||||
}
|
||||
}
|
||||
|
||||
// candidateDisks lists the disks an unattended install may erase: fixed, and
|
||||
// not the installer's own media.
|
||||
func candidateDisks() ([]install.Disk, error) {
|
||||
all, err := listDisks()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return slices.DeleteFunc(all, func(d install.Disk) bool {
|
||||
return d.Removable || d.LiveMedia
|
||||
}), nil
|
||||
}
|
||||
|
||||
func pickBySize(disks []install.Disk, largest bool) (install.Disk, error) {
|
||||
if len(disks) == 0 {
|
||||
return install.Disk{}, fmt.Errorf("no non-removable disks found")
|
||||
}
|
||||
sorted := slices.Clone(disks)
|
||||
slices.SortFunc(sorted, func(a, b install.Disk) int { return cmp.Compare(a.Bytes, b.Bytes) })
|
||||
if largest {
|
||||
return sorted[len(sorted)-1], nil
|
||||
}
|
||||
return sorted[0], nil
|
||||
}
|
||||
|
||||
// diskList renders candidates for an error message.
|
||||
func diskList(disks []install.Disk) string {
|
||||
lines := make([]string, len(disks))
|
||||
for i, d := range disks {
|
||||
lines[i] = fmt.Sprintf(" %s (%s, %s)", d.Path, d.SizeHuman(), d.Content)
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// virtualNICPrefixes lists interface name prefixes that identify non-physical
|
||||
// interfaces (bridges, tunnels, container/OVS veth pairs, etc.). These are
|
||||
// skipped when auto-selecting a NIC so we don't configure docker0 or
|
||||
// ovs-system as the management interface on a machine that previously ran
|
||||
// Docker or OVN.
|
||||
var virtualNICPrefixes = []string{
|
||||
"docker", "veth", "virbr", "br-", "ovs-", "vxlan", "genev", "tun", "tap",
|
||||
}
|
||||
|
||||
// resolveNIC returns the interface name to use. "auto" or empty picks the
|
||||
// first physical (non-loopback, non-virtual) interface that is not exclude.
|
||||
// parseRole reads the SPINIFEX_<PLANE>_* kernel-cmdline vars for one plane.
|
||||
// iface is the already-resolved interface name; _VLAN and _MTU are optional
|
||||
// and default to untagged at the link MTU.
|
||||
func parseRole(plane, iface string) (install.NetworkRole, error) {
|
||||
role := install.NetworkRole{
|
||||
Interface: iface,
|
||||
DHCPMode: strings.ToLower(os.Getenv("SPINIFEX_"+plane+"_MODE")) != "static",
|
||||
}
|
||||
if v := os.Getenv("SPINIFEX_" + plane + "_VLAN"); v != "" {
|
||||
id, err := strconv.Atoi(v)
|
||||
if err != nil {
|
||||
return role, fmt.Errorf("SPINIFEX_%s_VLAN: %w", plane, err)
|
||||
}
|
||||
role.VLAN = id
|
||||
}
|
||||
if v := os.Getenv("SPINIFEX_" + plane + "_MTU"); v != "" {
|
||||
mtu, err := strconv.Atoi(v)
|
||||
if err != nil {
|
||||
return role, fmt.Errorf("SPINIFEX_%s_MTU: %w", plane, err)
|
||||
}
|
||||
role.MTU = mtu
|
||||
}
|
||||
if !role.DHCPMode {
|
||||
role.Address = os.Getenv("SPINIFEX_" + plane + "_IP")
|
||||
role.Mask = os.Getenv("SPINIFEX_" + plane + "_MASK")
|
||||
role.Gateway = os.Getenv("SPINIFEX_" + plane + "_GW")
|
||||
if role.Address == "" || role.Mask == "" {
|
||||
return role, fmt.Errorf("SPINIFEX_%s_IP and SPINIFEX_%s_MASK required for static mode", plane, plane)
|
||||
}
|
||||
}
|
||||
if dns := os.Getenv("SPINIFEX_" + plane + "_DNS"); dns != "" {
|
||||
role.DNS = strings.Split(dns, ",")
|
||||
}
|
||||
return role, nil
|
||||
}
|
||||
|
||||
func resolveNIC(name, exclude string) (string, error) {
|
||||
if name != "" && name != "auto" {
|
||||
return name, nil
|
||||
}
|
||||
ifaces, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, iface := range ifaces {
|
||||
if iface.Flags&net.FlagLoopback != 0 {
|
||||
continue
|
||||
}
|
||||
// Require broadcast capability — filters out point-to-point tunnels.
|
||||
if iface.Flags&net.FlagBroadcast == 0 {
|
||||
continue
|
||||
}
|
||||
// Require a MAC address — virtual/tunnel interfaces have none.
|
||||
if len(iface.HardwareAddr) == 0 {
|
||||
continue
|
||||
}
|
||||
if iface.Name == exclude {
|
||||
continue
|
||||
}
|
||||
// Skip known virtual interface prefixes.
|
||||
virtual := false
|
||||
for _, pfx := range virtualNICPrefixes {
|
||||
if strings.HasPrefix(iface.Name, pfx) {
|
||||
virtual = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if virtual {
|
||||
continue
|
||||
}
|
||||
slog.Info("autoinstall: NIC resolved", "mode", "auto", "selected", iface.Name)
|
||||
return iface.Name, nil
|
||||
}
|
||||
return "", fmt.Errorf("no suitable physical NIC found (non-loopback, broadcast-capable, with MAC)")
|
||||
}
|
||||
@@ -1,688 +0,0 @@
|
||||
package autoinstall
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/installer/install"
|
||||
)
|
||||
|
||||
const gib = 1 << 30
|
||||
|
||||
// setEnv clears every SPINIFEX_* variable inherited from the test runner before
|
||||
// applying want, so one case cannot leak a kernel-cmdline value into the next.
|
||||
func setEnv(t *testing.T, want map[string]string) {
|
||||
t.Helper()
|
||||
for _, kv := range os.Environ() {
|
||||
if k, _, ok := strings.Cut(kv, "="); ok && strings.HasPrefix(k, "SPINIFEX_") {
|
||||
t.Setenv(k, "")
|
||||
}
|
||||
}
|
||||
for k, v := range want {
|
||||
t.Setenv(k, v)
|
||||
}
|
||||
}
|
||||
|
||||
// withDisks replaces the block-device scan. The clone matters: candidateDisks
|
||||
// filters with slices.DeleteFunc, which would otherwise mutate the fixture.
|
||||
func withDisks(t *testing.T, disks []install.Disk, err error) {
|
||||
t.Helper()
|
||||
prev := listDisks
|
||||
listDisks = func() ([]install.Disk, error) { return slices.Clone(disks), err }
|
||||
t.Cleanup(func() { listDisks = prev })
|
||||
}
|
||||
|
||||
func mkDisk(path string, bytes int64) install.Disk {
|
||||
return install.Disk{
|
||||
Path: path,
|
||||
ByID: "/dev/disk/by-id/test-" + strings.TrimPrefix(path, "/dev/"),
|
||||
Bytes: bytes,
|
||||
Model: "TESTDISK",
|
||||
LogicalBlockSize: 512,
|
||||
PhysicalBlockSize: 512,
|
||||
Content: "empty",
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadReturnsNilWhenNotAuto(t *testing.T) {
|
||||
setEnv(t, nil)
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if cfg != nil {
|
||||
t.Fatal("expected nil config when SPINIFEX_AUTO is not set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadBuildsConfigFromEnv(t *testing.T) {
|
||||
withDisks(t, []install.Disk{mkDisk("/dev/vdb", 40*gib)}, nil)
|
||||
setEnv(t, map[string]string{
|
||||
"SPINIFEX_AUTO": "1",
|
||||
"SPINIFEX_PASSWORD": "s3cret",
|
||||
"SPINIFEX_HOSTNAME": "node-7",
|
||||
"SPINIFEX_EMAIL": " ops@example.com ",
|
||||
"SPINIFEX_WAN_IFACE": "eno1",
|
||||
"SPINIFEX_WAN_DNS": "1.1.1.1,8.8.8.8",
|
||||
"SPINIFEX_LAN_IFACE": "eno2",
|
||||
"SPINIFEX_LAN_VLAN": "100",
|
||||
"SPINIFEX_LAN_MTU": "9000",
|
||||
"SPINIFEX_GPU_PASSTHROUGH": "1",
|
||||
})
|
||||
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
if cfg.Hostname != "node-7" || cfg.RootPassword != "s3cret" {
|
||||
t.Errorf("identity = %q/%q", cfg.Hostname, cfg.RootPassword)
|
||||
}
|
||||
if cfg.Email != "ops@example.com" {
|
||||
t.Errorf("Email = %q, want trimmed", cfg.Email)
|
||||
}
|
||||
if cfg.Storage.FS != install.FSExt4 || len(cfg.Storage.Disks) != 1 {
|
||||
t.Errorf("Storage = %v / %d disks", cfg.Storage.FS, len(cfg.Storage.Disks))
|
||||
}
|
||||
if cfg.WAN.Interface != "eno1" || !cfg.WAN.DHCPMode {
|
||||
t.Errorf("WAN = %+v, want eno1 on DHCP", cfg.WAN)
|
||||
}
|
||||
if !slices.Equal(cfg.WAN.DNS, []string{"1.1.1.1", "8.8.8.8"}) {
|
||||
t.Errorf("WAN.DNS = %v", cfg.WAN.DNS)
|
||||
}
|
||||
if cfg.LAN.Interface != "eno2" || cfg.LAN.VLAN != 100 || cfg.LAN.MTU != 9000 {
|
||||
t.Errorf("LAN = %+v", cfg.LAN)
|
||||
}
|
||||
// Only the wan plane installs a default route.
|
||||
if cfg.LAN.Gateway != "" {
|
||||
t.Errorf("LAN.Gateway = %q, want cleared", cfg.LAN.Gateway)
|
||||
}
|
||||
if !cfg.VPC.Folded() {
|
||||
t.Errorf("VPC = %+v, want folded when no _IFACE is set", cfg.VPC)
|
||||
}
|
||||
if !cfg.GPUPassthrough {
|
||||
t.Error("GPUPassthrough not enabled")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadDefaultsHostname(t *testing.T) {
|
||||
withDisks(t, []install.Disk{mkDisk("/dev/vdb", 40*gib)}, nil)
|
||||
setEnv(t, map[string]string{
|
||||
"SPINIFEX_AUTO": "1",
|
||||
"SPINIFEX_PASSWORD": "s3cret",
|
||||
"SPINIFEX_WAN_IFACE": "eno1",
|
||||
})
|
||||
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
if cfg.Hostname != "spinifex-node" {
|
||||
t.Errorf("Hostname = %q, want spinifex-node", cfg.Hostname)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRequiresPassword(t *testing.T) {
|
||||
setEnv(t, map[string]string{"SPINIFEX_AUTO": "1"})
|
||||
|
||||
cfg, err := Load()
|
||||
if err == nil {
|
||||
t.Fatalf("expected error, got config %+v", cfg)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "SPINIFEX_PASSWORD") {
|
||||
t.Errorf("err = %v, want it to name SPINIFEX_PASSWORD", err)
|
||||
}
|
||||
if !strings.HasPrefix(err.Error(), "autoinstall config:") {
|
||||
t.Errorf("err = %v, want the autoinstall wrapper", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildConfigStaticWAN(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "full static",
|
||||
env: map[string]string{
|
||||
"SPINIFEX_WAN_MODE": "static",
|
||||
"SPINIFEX_WAN_IP": "10.0.0.5",
|
||||
"SPINIFEX_WAN_MASK": "255.255.255.0",
|
||||
"SPINIFEX_WAN_GW": "10.0.0.1",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "missing gateway",
|
||||
env: map[string]string{
|
||||
"SPINIFEX_WAN_MODE": "static",
|
||||
"SPINIFEX_WAN_IP": "10.0.0.5",
|
||||
"SPINIFEX_WAN_MASK": "255.255.255.0",
|
||||
},
|
||||
wantErr: "SPINIFEX_WAN_GW required",
|
||||
},
|
||||
{
|
||||
name: "missing address",
|
||||
env: map[string]string{"SPINIFEX_WAN_MODE": "static"},
|
||||
wantErr: "SPINIFEX_WAN_IP and SPINIFEX_WAN_MASK required",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
withDisks(t, []install.Disk{mkDisk("/dev/vdb", 40*gib)}, nil)
|
||||
env := map[string]string{
|
||||
"SPINIFEX_PASSWORD": "s3cret",
|
||||
"SPINIFEX_WAN_IFACE": "eno1",
|
||||
}
|
||||
maps.Copy(env, tt.env)
|
||||
setEnv(t, env)
|
||||
|
||||
cfg, err := buildConfig()
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("err = %v, want %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("buildConfig: %v", err)
|
||||
}
|
||||
if cfg.WAN.DHCPMode {
|
||||
t.Error("WAN.DHCPMode = true, want static")
|
||||
}
|
||||
if cfg.WAN.Address != "10.0.0.5" || cfg.WAN.Gateway != "10.0.0.1" {
|
||||
t.Errorf("WAN = %+v", cfg.WAN)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// SPINIFEX_ROLE and SPINIFEX_JOIN_ADDR no longer exist: the installer cannot
|
||||
// form a multi-node cluster, so a stale cmdline from an older ISO has to be
|
||||
// inert rather than producing a half-configured node.
|
||||
func TestBuildConfigSkipFormation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
wantSkipForm bool
|
||||
}{
|
||||
{
|
||||
name: "default initializes a single-node cluster",
|
||||
wantSkipForm: false,
|
||||
},
|
||||
{
|
||||
name: "formation deferred to the provisioner",
|
||||
env: map[string]string{"SPINIFEX_SKIP_FORMATION": "1"},
|
||||
wantSkipForm: true,
|
||||
},
|
||||
{
|
||||
name: "stale join cmdline is ignored",
|
||||
env: map[string]string{"SPINIFEX_ROLE": "join", "SPINIFEX_JOIN_ADDR": "10.0.0.1:9999"},
|
||||
wantSkipForm: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
withDisks(t, []install.Disk{mkDisk("/dev/vdb", 40*gib)}, nil)
|
||||
env := map[string]string{
|
||||
"SPINIFEX_PASSWORD": "s3cret",
|
||||
"SPINIFEX_WAN_IFACE": "eno1",
|
||||
}
|
||||
maps.Copy(env, tt.env)
|
||||
setEnv(t, env)
|
||||
|
||||
cfg, err := buildConfig()
|
||||
if err != nil {
|
||||
t.Fatalf("buildConfig: %v", err)
|
||||
}
|
||||
if cfg.SkipFormation != tt.wantSkipForm {
|
||||
t.Errorf("SkipFormation = %v, want %v", cfg.SkipFormation, tt.wantSkipForm)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildConfigRejectsDuplicatePlaneBinding(t *testing.T) {
|
||||
withDisks(t, []install.Disk{mkDisk("/dev/vdb", 40*gib)}, nil)
|
||||
setEnv(t, map[string]string{
|
||||
"SPINIFEX_PASSWORD": "s3cret",
|
||||
"SPINIFEX_WAN_IFACE": "eno1",
|
||||
"SPINIFEX_LAN_IFACE": "eno1",
|
||||
})
|
||||
|
||||
if _, err := buildConfig(); err == nil || !strings.Contains(err.Error(), "network roles") {
|
||||
t.Fatalf("err = %v, want a network roles validation failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildConfigPropagatesStorageError(t *testing.T) {
|
||||
setEnv(t, map[string]string{
|
||||
"SPINIFEX_PASSWORD": "s3cret",
|
||||
"SPINIFEX_WAN_IFACE": "eno1",
|
||||
"SPINIFEX_FS": "btrfs",
|
||||
})
|
||||
|
||||
_, err := buildConfig()
|
||||
if err == nil || !strings.HasPrefix(err.Error(), "storage:") {
|
||||
t.Fatalf("err = %v, want a storage: prefix", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseRole(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
want install.NetworkRole
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "dhcp by default",
|
||||
want: install.NetworkRole{Interface: "eno1", DHCPMode: true},
|
||||
},
|
||||
{
|
||||
name: "vlan and mtu",
|
||||
env: map[string]string{"SPINIFEX_LAN_VLAN": "42", "SPINIFEX_LAN_MTU": "9000"},
|
||||
want: install.NetworkRole{Interface: "eno1", DHCPMode: true, VLAN: 42, MTU: 9000},
|
||||
},
|
||||
{
|
||||
name: "dns list is comma separated",
|
||||
env: map[string]string{"SPINIFEX_LAN_DNS": "1.1.1.1,9.9.9.9"},
|
||||
want: install.NetworkRole{Interface: "eno1", DHCPMode: true, DNS: []string{"1.1.1.1", "9.9.9.9"}},
|
||||
},
|
||||
{
|
||||
name: "static",
|
||||
env: map[string]string{
|
||||
"SPINIFEX_LAN_MODE": "STATIC",
|
||||
"SPINIFEX_LAN_IP": "192.168.1.2",
|
||||
"SPINIFEX_LAN_MASK": "255.255.255.0",
|
||||
"SPINIFEX_LAN_GW": "192.168.1.1",
|
||||
},
|
||||
want: install.NetworkRole{
|
||||
Interface: "eno1",
|
||||
Address: "192.168.1.2",
|
||||
Mask: "255.255.255.0",
|
||||
Gateway: "192.168.1.1",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "static without mask",
|
||||
env: map[string]string{"SPINIFEX_LAN_MODE": "static", "SPINIFEX_LAN_IP": "192.168.1.2"},
|
||||
wantErr: "SPINIFEX_LAN_IP and SPINIFEX_LAN_MASK required",
|
||||
},
|
||||
{
|
||||
name: "unparsable vlan",
|
||||
env: map[string]string{"SPINIFEX_LAN_VLAN": "trunk"},
|
||||
wantErr: "SPINIFEX_LAN_VLAN",
|
||||
},
|
||||
{
|
||||
name: "unparsable mtu",
|
||||
env: map[string]string{"SPINIFEX_LAN_MTU": "jumbo"},
|
||||
wantErr: "SPINIFEX_LAN_MTU",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
setEnv(t, tt.env)
|
||||
|
||||
got, err := parseRole("LAN", "eno1")
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("err = %v, want %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("parseRole: %v", err)
|
||||
}
|
||||
if fmt.Sprintf("%+v", got) != fmt.Sprintf("%+v", tt.want) {
|
||||
t.Errorf("role = %+v, want %+v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseZFSOpts(t *testing.T) {
|
||||
t.Run("unset leaves the computed defaults", func(t *testing.T) {
|
||||
setEnv(t, nil)
|
||||
got, err := parseZFSOpts()
|
||||
if err != nil {
|
||||
t.Fatalf("parseZFSOpts: %v", err)
|
||||
}
|
||||
if got != (install.ZFSOpts{}) {
|
||||
t.Errorf("opts = %+v, want zero", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("all tunables", func(t *testing.T) {
|
||||
setEnv(t, map[string]string{
|
||||
"SPINIFEX_ZFS_ASHIFT": " 12 ",
|
||||
"SPINIFEX_ZFS_COPIES": "2",
|
||||
"SPINIFEX_ZFS_ARC_MAX_MIB": "4096",
|
||||
"SPINIFEX_ZFS_COMPRESS": " lz4 ",
|
||||
"SPINIFEX_ZFS_CHECKSUM": "blake3",
|
||||
})
|
||||
got, err := parseZFSOpts()
|
||||
if err != nil {
|
||||
t.Fatalf("parseZFSOpts: %v", err)
|
||||
}
|
||||
want := install.ZFSOpts{Ashift: 12, Copies: 2, ARCMaxMiB: 4096, Compress: "lz4", Checksum: "blake3"}
|
||||
if got != want {
|
||||
t.Errorf("opts = %+v, want %+v", got, want)
|
||||
}
|
||||
})
|
||||
|
||||
for _, env := range []string{"SPINIFEX_ZFS_ASHIFT", "SPINIFEX_ZFS_COPIES", "SPINIFEX_ZFS_ARC_MAX_MIB"} {
|
||||
t.Run("unparsable "+env, func(t *testing.T) {
|
||||
setEnv(t, map[string]string{env: "auto"})
|
||||
if _, err := parseZFSOpts(); err == nil || !strings.Contains(err.Error(), env) {
|
||||
t.Fatalf("err = %v, want it to name %s", err, env)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveStorage(t *testing.T) {
|
||||
four := []install.Disk{
|
||||
mkDisk("/dev/vdb", 40*gib), mkDisk("/dev/vdc", 40*gib),
|
||||
mkDisk("/dev/vdd", 40*gib), mkDisk("/dev/vde", 40*gib),
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
disks []install.Disk
|
||||
env map[string]string
|
||||
wantFS install.FSType
|
||||
wantPaths []string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "ext4 single disk by default",
|
||||
disks: four[:1],
|
||||
wantFS: install.FSExt4,
|
||||
wantPaths: []string{"/dev/vdb"},
|
||||
},
|
||||
{
|
||||
name: "explicit member list preserves order",
|
||||
disks: four,
|
||||
env: map[string]string{"SPINIFEX_FS": "zfs-raid10", "SPINIFEX_DISKS": "vde,/dev/vdd,vdc, vdb "},
|
||||
wantFS: install.FSZFSRAID10,
|
||||
wantPaths: []string{"/dev/vde", "/dev/vdd", "/dev/vdc", "/dev/vdb"},
|
||||
},
|
||||
{
|
||||
name: "unknown filesystem",
|
||||
disks: four,
|
||||
env: map[string]string{"SPINIFEX_FS": "btrfs"},
|
||||
wantErr: "unknown filesystem",
|
||||
},
|
||||
{
|
||||
// Choosing which disks to erase is not an unattended decision.
|
||||
name: "multi-disk pool without an explicit list",
|
||||
disks: four,
|
||||
env: map[string]string{"SPINIFEX_FS": "zfs-raid1"},
|
||||
wantErr: "set SPINIFEX_DISKS to an explicit comma-separated list",
|
||||
},
|
||||
{
|
||||
name: "member that is not present",
|
||||
disks: four[:1],
|
||||
env: map[string]string{"SPINIFEX_DISKS": "vdz"},
|
||||
wantErr: `"vdz" is not an available disk`,
|
||||
},
|
||||
{
|
||||
name: "member list is for pools, not ext4 roles",
|
||||
disks: four,
|
||||
env: map[string]string{"SPINIFEX_DISKS": "vdb,vdc"},
|
||||
wantErr: "ext4 takes one OS disk",
|
||||
},
|
||||
{
|
||||
name: "dedicated role drives, os first",
|
||||
disks: four,
|
||||
env: map[string]string{
|
||||
"SPINIFEX_DISK": "vdb",
|
||||
"SPINIFEX_DISK_SPINIFEX": "vdc",
|
||||
"SPINIFEX_DISK_PREDASTORE": "vdd",
|
||||
},
|
||||
wantFS: install.FSExt4,
|
||||
wantPaths: []string{"/dev/vdb", "/dev/vdc", "/dev/vdd"},
|
||||
},
|
||||
{
|
||||
// The two-drive layout: predastore gets its own disk while
|
||||
// /var/lib/spinifex stays on the root filesystem.
|
||||
name: "predastore alone",
|
||||
disks: four,
|
||||
env: map[string]string{"SPINIFEX_DISK": "vdb", "SPINIFEX_DISK_PREDASTORE": "vdc"},
|
||||
wantFS: install.FSExt4,
|
||||
wantPaths: []string{"/dev/vdb", "/dev/vdc"},
|
||||
},
|
||||
{
|
||||
name: "role drive on a pool",
|
||||
disks: four,
|
||||
env: map[string]string{
|
||||
"SPINIFEX_FS": "zfs-raid1", "SPINIFEX_DISKS": "vdb,vdc",
|
||||
"SPINIFEX_DISK_SPINIFEX": "vdd",
|
||||
},
|
||||
wantErr: "SPINIFEX_DISK_SPINIFEX applies to ext4 only",
|
||||
},
|
||||
{
|
||||
name: "role drive that is not present",
|
||||
disks: four[:1],
|
||||
env: map[string]string{"SPINIFEX_DISK": "vdb", "SPINIFEX_DISK_PREDASTORE": "vdz"},
|
||||
wantErr: "SPINIFEX_DISK_PREDASTORE",
|
||||
},
|
||||
{
|
||||
// Two roles cannot share a drive; the disk is erased once and the
|
||||
// second mount would land on top of the first.
|
||||
name: "role drive repeats the os disk",
|
||||
disks: four,
|
||||
env: map[string]string{"SPINIFEX_DISK": "vdb", "SPINIFEX_DISK_SPINIFEX": "vdb"},
|
||||
wantErr: "selected more than once",
|
||||
},
|
||||
{
|
||||
name: "bad zfs tunable",
|
||||
disks: four[:1],
|
||||
env: map[string]string{"SPINIFEX_ZFS_ASHIFT": "big"},
|
||||
wantErr: "SPINIFEX_ZFS_ASHIFT",
|
||||
},
|
||||
{
|
||||
name: "scan failure",
|
||||
disks: nil,
|
||||
env: map[string]string{"SPINIFEX_DISKS": "vdb"},
|
||||
wantErr: "sysfs unavailable",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var scanErr error
|
||||
if tt.disks == nil {
|
||||
scanErr = errors.New("sysfs unavailable")
|
||||
}
|
||||
withDisks(t, tt.disks, scanErr)
|
||||
setEnv(t, tt.env)
|
||||
|
||||
got, err := resolveStorage()
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("err = %v, want %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("resolveStorage: %v", err)
|
||||
}
|
||||
if got.FS != tt.wantFS {
|
||||
t.Errorf("FS = %v, want %v", got.FS, tt.wantFS)
|
||||
}
|
||||
if !slices.Equal(got.Paths(), tt.wantPaths) {
|
||||
t.Errorf("Paths = %v, want %v", got.Paths(), tt.wantPaths)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveDisk(t *testing.T) {
|
||||
small := mkDisk("/dev/vdb", 40*gib)
|
||||
large := mkDisk("/dev/vdc", 400*gib)
|
||||
nvme := mkDisk("/dev/nvme0n1", 200*gib)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
disks []install.Disk
|
||||
target string
|
||||
wantPath string
|
||||
wantErr string
|
||||
}{
|
||||
{name: "auto with one candidate", disks: []install.Disk{small}, target: "auto", wantPath: "/dev/vdb"},
|
||||
{name: "empty target is auto", disks: []install.Disk{small}, wantPath: "/dev/vdb"},
|
||||
{
|
||||
name: "auto with several candidates",
|
||||
disks: []install.Disk{small, large},
|
||||
target: "auto",
|
||||
wantErr: "expected exactly one disk, found 2",
|
||||
},
|
||||
{name: "largest", disks: []install.Disk{small, large, nvme}, target: "LARGEST", wantPath: "/dev/vdc"},
|
||||
{name: "smallest", disks: []install.Disk{large, small, nvme}, target: " smallest ", wantPath: "/dev/vdb"},
|
||||
{name: "nvme", disks: []install.Disk{small, large, nvme}, target: "nvme", wantPath: "/dev/nvme0n1"},
|
||||
{
|
||||
name: "nvme with none present",
|
||||
disks: []install.Disk{small, large},
|
||||
target: "nvme",
|
||||
wantErr: "expected exactly one NVMe disk, found 0",
|
||||
},
|
||||
{name: "exact path", disks: []install.Disk{small, large}, target: "/dev/vdc", wantPath: "/dev/vdc"},
|
||||
{
|
||||
name: "path that is not present",
|
||||
disks: []install.Disk{small},
|
||||
target: "/dev/vdz",
|
||||
wantErr: "is not an available disk",
|
||||
},
|
||||
{name: "no candidates at all", disks: []install.Disk{}, target: "largest", wantErr: "no non-removable disks found"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
withDisks(t, tt.disks, nil)
|
||||
|
||||
got, err := resolveDisk(tt.target)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("err = %v, want %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("resolveDisk(%q): %v", tt.target, err)
|
||||
}
|
||||
if got.Path != tt.wantPath {
|
||||
t.Errorf("path = %q, want %q", got.Path, tt.wantPath)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCandidateDisksExcludesRemovableAndLiveMedia(t *testing.T) {
|
||||
fixed := mkDisk("/dev/vdb", 40*gib)
|
||||
usb := mkDisk("/dev/sdz", 16*gib)
|
||||
usb.Removable = true
|
||||
boot := mkDisk("/dev/sdy", 16*gib)
|
||||
boot.LiveMedia = true
|
||||
withDisks(t, []install.Disk{usb, fixed, boot}, nil)
|
||||
|
||||
got, err := candidateDisks()
|
||||
if err != nil {
|
||||
t.Fatalf("candidateDisks: %v", err)
|
||||
}
|
||||
if len(got) != 1 || got[0].Path != "/dev/vdb" {
|
||||
t.Fatalf("candidates = %v, want only /dev/vdb", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCandidateDisksPropagatesScanError(t *testing.T) {
|
||||
withDisks(t, nil, errors.New("read /sys/block: denied"))
|
||||
if _, err := candidateDisks(); err == nil {
|
||||
t.Fatal("expected the scan error to propagate")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPickBySizeRejectsEmptySet(t *testing.T) {
|
||||
if _, err := pickBySize(nil, true); err == nil {
|
||||
t.Fatal("expected an error for an empty disk set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPickBySizeLeavesInputOrderIntact(t *testing.T) {
|
||||
disks := []install.Disk{mkDisk("/dev/vdb", 40*gib), mkDisk("/dev/vdc", 400*gib)}
|
||||
if _, err := pickBySize(disks, false); err != nil {
|
||||
t.Fatalf("pickBySize: %v", err)
|
||||
}
|
||||
if disks[0].Path != "/dev/vdb" {
|
||||
t.Errorf("input reordered: %v", disks)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiskList(t *testing.T) {
|
||||
got := diskList([]install.Disk{mkDisk("/dev/vdb", 40*gib), mkDisk("/dev/vdc", 2048*gib)})
|
||||
want := " /dev/vdb (40.0G, empty)\n /dev/vdc (2.0T, empty)"
|
||||
if got != want {
|
||||
t.Errorf("diskList =\n%s\nwant\n%s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveNICExplicitName(t *testing.T) {
|
||||
got, err := resolveNIC("eno1", "")
|
||||
if err != nil {
|
||||
t.Fatalf("resolveNIC: %v", err)
|
||||
}
|
||||
if got != "eno1" {
|
||||
t.Errorf("nic = %q, want eno1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The auto path reads the host's real interfaces, so assert the selection rules
|
||||
// rather than a specific name.
|
||||
func TestResolveNICAutoSkipsVirtualInterfaces(t *testing.T) {
|
||||
ifaces, err := net.Interfaces()
|
||||
if err != nil {
|
||||
t.Skipf("no interface list on this host: %v", err)
|
||||
}
|
||||
|
||||
got, err := resolveNIC("auto", "")
|
||||
if err != nil {
|
||||
if len(ifaces) > 0 {
|
||||
t.Logf("no physical NIC among %d interfaces: %v", len(ifaces), err)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
idx := slices.IndexFunc(ifaces, func(i net.Interface) bool { return i.Name == got })
|
||||
if idx < 0 {
|
||||
t.Fatalf("resolveNIC returned %q, which is not a host interface", got)
|
||||
}
|
||||
sel := ifaces[idx]
|
||||
if sel.Flags&net.FlagLoopback != 0 || sel.Flags&net.FlagBroadcast == 0 || len(sel.HardwareAddr) == 0 {
|
||||
t.Errorf("selected %q with flags %v and MAC %q", got, sel.Flags, sel.HardwareAddr)
|
||||
}
|
||||
for _, pfx := range virtualNICPrefixes {
|
||||
if strings.HasPrefix(got, pfx) {
|
||||
t.Errorf("selected virtual interface %q (prefix %q)", got, pfx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveNICAutoHonoursExclude(t *testing.T) {
|
||||
first, err := resolveNIC("auto", "")
|
||||
if err != nil {
|
||||
t.Skipf("no physical NIC on this host: %v", err)
|
||||
}
|
||||
got, err := resolveNIC("", first)
|
||||
if err == nil && got == first {
|
||||
t.Errorf("resolveNIC returned the excluded interface %q", first)
|
||||
}
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
// Package branding centralises Mulga/Spinifex visual identity constants used
|
||||
// by the installer TUI.
|
||||
//
|
||||
// TODO: Replace placeholder values with official Mulga/Spinifex assets once
|
||||
// the brand guide and logo files are located.
|
||||
package branding
|
||||
|
||||
import "github.com/charmbracelet/lipgloss"
|
||||
|
||||
// Logo is the ASCII art rendered on the installer welcome screen.
|
||||
// TODO: Replace with official Spinifex/Mulga ASCII logo.
|
||||
const Logo = `
|
||||
███████╗██████╗ ██╗███╗ ██╗██╗███████╗███████╗██╗ ██╗
|
||||
██╔════╝██╔══██╗██║████╗ ██║██║██╔════╝██╔════╝╚██╗██╔╝
|
||||
███████╗██████╔╝██║██╔██╗ ██║██║█████╗ █████╗ ╚███╔╝
|
||||
╚════██║██╔═══╝ ██║██║╚██╗██║██║██╔══╝ ██╔══╝ ██╔██╗
|
||||
███████║██║ ██║██║ ╚████║██║██║ ███████╗██╔╝ ██╗
|
||||
╚══════╝╚═╝ ╚═╝╚═╝ ╚═══╝╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝`
|
||||
|
||||
// Subtitle is displayed beneath the logo on the welcome screen.
|
||||
const Subtitle = "Bare-Metal Node Provisioner"
|
||||
|
||||
// Publisher is shown in the welcome screen footer.
|
||||
const Publisher = "Mulga Defense Corporation"
|
||||
|
||||
// Mulga color palette.
|
||||
// TODO: Replace with official hex values from the Mulga brand guide.
|
||||
const (
|
||||
ColorPrimary = lipgloss.Color("#00A3E0") // Mulga blue (placeholder)
|
||||
ColorAccent = lipgloss.Color("#FF6B00") // Mulga orange (placeholder)
|
||||
ColorBackground = lipgloss.Color("#0D1117") // Dark background
|
||||
ColorSurface = lipgloss.Color("#161B22") // Slightly lighter surface
|
||||
ColorText = lipgloss.Color("#E6EDF3") // Light text
|
||||
ColorMuted = lipgloss.Color("#8B949E") // Muted / secondary text
|
||||
ColorSuccess = lipgloss.Color("#3FB950") // Green
|
||||
ColorWarning = lipgloss.Color("#D29922") // Amber
|
||||
ColorError = lipgloss.Color("#F85149") // Red
|
||||
ColorBorder = lipgloss.Color("#30363D") // Border
|
||||
)
|
||||
@@ -1,347 +0,0 @@
|
||||
// Package firstboot writes the oneshot systemd service and configuration that
|
||||
// completes Spinifex provisioning on the first real boot after installation.
|
||||
package firstboot
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/installer/systemd"
|
||||
)
|
||||
|
||||
// Config holds the values the firstboot service needs to configure the node.
|
||||
type Config struct {
|
||||
Hostname string
|
||||
// EncapIP is the Geneve tunnel IP for OVN, taken from the vpc plane after
|
||||
// collapsing (vpc <- lan <- wan). Empty when that plane uses DHCP —
|
||||
// setup-ovn.sh auto-detects the IP from the default route in that case.
|
||||
EncapIP string
|
||||
// LANIP is the internal cluster address, taken from the lan plane after
|
||||
// collapsing. Passed as --bind and --cluster-bind so predastore replication,
|
||||
// the NATS mesh and OVN control traffic stay off the public interface.
|
||||
// Empty leaves the 0.0.0.0 wildcard default, which is correct for a
|
||||
// single-NIC node where lan folds onto wan.
|
||||
LANIP string
|
||||
// WANIP is the public address, taken from the wan plane. Passed as
|
||||
// --advertise so northstar's :53 listener, the awsgw registry host and the
|
||||
// dial target recorded for off-host clients stay on the public interface.
|
||||
// Without it a concrete --bind is echoed back as the advertise address,
|
||||
// which would move all three onto the internal plane. Empty when wan uses
|
||||
// DHCP, leaving spx to auto-detect from the default route.
|
||||
WANIP string
|
||||
// Email is the operator email collected by the TUI or SPINIFEX_EMAIL on
|
||||
// the headless path. Passed to `spx admin init --email=<value>` when set;
|
||||
// omitted entirely when empty.
|
||||
Email string
|
||||
// GPUPassthrough enables VFIO GPU passthrough by passing --gpu-passthrough
|
||||
// to `spx admin init`, which writes gpu_passthrough = true in the daemon config.
|
||||
GPUPassthrough bool
|
||||
// InstallCallback, when non-empty, is curled once at the end of firstboot
|
||||
// after the success marker is written. Generic phone-home hook for
|
||||
// provisioning controllers (PXE/MAAS-style flows). Best-effort: non-2xx
|
||||
// and network failures are swallowed so the install is not gated on
|
||||
// controller reachability.
|
||||
InstallCallback string
|
||||
// SkipFormation skips spx admin init/join. Used when a provisioning
|
||||
// controller (e.g. bm-bootstrap.sh) owns cluster formation and will call
|
||||
// spx admin init/join itself with the correct multi-node parameters.
|
||||
SkipFormation bool
|
||||
}
|
||||
|
||||
// Write drops the firstboot script and systemd unit into root, which should be
|
||||
// the path of the installed system's root filesystem (e.g. /mnt/spinifex-install).
|
||||
func Write(root string, cfg Config) error {
|
||||
if err := writeScript(root, cfg); err != nil {
|
||||
return fmt.Errorf("firstboot script: %w", err)
|
||||
}
|
||||
if err := systemd.WriteFirstbootUnit(root); err != nil {
|
||||
return fmt.Errorf("firstboot unit: %w", err)
|
||||
}
|
||||
if err := systemd.EnableUnit(root, "spinifex-firstboot.service"); err != nil {
|
||||
return err
|
||||
}
|
||||
// Banner unit is written but NOT enabled here. firstboot enables it
|
||||
// (along with spinifex.target) at the end of its run, so on the first
|
||||
// boot the only spinifex unit running is firstboot itself — no implicit
|
||||
// pull of spinifex.target via the banner's Wants= relationship, which
|
||||
// was the root cause of the boot-time race against firstboot.
|
||||
if err := systemd.WriteBannerUnit(root); err != nil {
|
||||
return fmt.Errorf("banner unit: %w", err)
|
||||
}
|
||||
return systemd.WriteGettyDropIn(root)
|
||||
}
|
||||
|
||||
func writeScript(root string, cfg Config) error {
|
||||
clusterCmd := buildClusterCmd(cfg)
|
||||
|
||||
// --encap-ip is optional: when DHCP is used the IP is unknown at install time
|
||||
// and setup-ovn.sh auto-detects it from the default route at boot.
|
||||
setupOVN := "/usr/local/bin/setup-ovn.sh --management"
|
||||
if cfg.EncapIP != "" {
|
||||
setupOVN += fmt.Sprintf(" --encap-ip=%s", cfg.EncapIP)
|
||||
}
|
||||
|
||||
// Pre-start OVS and OVN central so their databases are initialised before
|
||||
// setup-ovn.sh runs. On physical hardware, first-boot DB initialisation takes
|
||||
// longer than setup-ovn.sh's internal 15-second timeout allows. Starting them
|
||||
// here and waiting until the NB DB is ready means setup-ovn.sh sees a live DB
|
||||
// the moment it starts — no races, no timeout failures.
|
||||
ovnPrestart := `systemctl start openvswitch-switch
|
||||
systemctl start ovn-central
|
||||
echo "Waiting for OVN NB DB to initialise..."
|
||||
for _i in $(seq 1 120); do
|
||||
if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
|
||||
echo "OVN NB DB ready (${_i}s)"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done`
|
||||
|
||||
// When a provisioning controller owns formation (SkipFormation) it also owns
|
||||
// the clustered OVN bring-up via setup-ovn.sh's RAFT flags. firstboot must
|
||||
// not start a standalone ovn-central here: the single-node .db it would leave
|
||||
// blocks ovn-ctl's create/join (which require a clean DB). Defer both.
|
||||
if cfg.SkipFormation {
|
||||
ovnPrestart = `echo "[firstboot] OVN bring-up deferred to provisioning controller"`
|
||||
setupOVN = `echo "[firstboot] setup-ovn deferred to provisioning controller"`
|
||||
}
|
||||
|
||||
callbackBlock := ""
|
||||
if cfg.InstallCallback != "" {
|
||||
callbackBlock = fmt.Sprintf(
|
||||
"\ncurl -fsS --max-time 10 --retry 2 --retry-delay 2 -o /dev/null %s || true\n",
|
||||
shellEscapeSingle(cfg.InstallCallback),
|
||||
)
|
||||
}
|
||||
|
||||
script := fmt.Sprintf(`#!/bin/bash
|
||||
# Spinifex firstboot — runs once after ISO installation. On success, writes
|
||||
# /var/lib/spinifex/.firstboot-done and the systemd unit's
|
||||
# ConditionPathExists=! prevents re-execution on subsequent boots. A partial
|
||||
# run leaves no marker, so the next reboot retries from the top — safe
|
||||
# because every step below is idempotent (hostnamectl, setup-ovn.sh, spx
|
||||
# admin init are all "set if not set"; systemctl enable on an already-enabled
|
||||
# unit is a no-op).
|
||||
set -euo pipefail
|
||||
|
||||
DONE_MARKER=/var/lib/spinifex/.firstboot-done
|
||||
|
||||
# Idempotency: bail early if a previous run completed successfully. The unit
|
||||
# also has ConditionPathExists=!$DONE_MARKER so we shouldn't get here on
|
||||
# subsequent boots, but defend in depth in case the unit is re-triggered
|
||||
# manually before the operator notices the marker exists.
|
||||
if [ -f "$DONE_MARKER" ]; then
|
||||
echo "[firstboot] already complete — skipping"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Set hostname
|
||||
hostnamectl set-hostname %s
|
||||
|
||||
%s
|
||||
|
||||
# Create default nameservers if DHCP server returns blank
|
||||
printf "nameserver 1.1.1.1\nnameserver 8.8.8.8\n" > /etc/resolvconf/resolv.conf.d/base
|
||||
resolvconf -u
|
||||
|
||||
# The RTC on a fresh machine can be badly wrong — a flat CMOS battery is routine
|
||||
# on refurbished hardware. Credentials and TLS certificates are minted further
|
||||
# down, and a certificate issued against a wrong clock carries a NotBefore that
|
||||
# outlives the correction, so the clock has to be right before that happens.
|
||||
# chronyd is already running; makestep forces an immediate correction instead of
|
||||
# waiting out the slew. Bounded and non-fatal: an air-gapped install must still
|
||||
# complete, and rtcsync writes the corrected time back so later boots start close.
|
||||
echo "[firstboot] syncing clock before credentials are minted"
|
||||
chronyc -a makestep >/dev/null 2>&1 || true
|
||||
if chronyc waitsync 12 0.1 0 5 >/dev/null 2>&1; then
|
||||
echo "[firstboot] clock synced: $(date -uIs)"
|
||||
else
|
||||
echo "[firstboot] WARNING: clock not synced, continuing with $(date -uIs)"
|
||||
fi
|
||||
|
||||
# Configure OVN networking.
|
||||
# br-wan (and br-lan if present) are Linux bridges managed by systemd-networkd
|
||||
# (declared in /etc/systemd/network/ by the installer). setup-ovn.sh
|
||||
# auto-detects br-wan as the default route device and wires it to OVS via a
|
||||
# veth pair — non-destructive, SSH-safe.
|
||||
%s
|
||||
|
||||
# Write the banner (first time, spinifex-banner service will do this on reboot)
|
||||
/usr/local/bin/spx admin banner --boot-check
|
||||
|
||||
# Cluster formation. Output goes to the journal via the systemd unit; spx
|
||||
# admin init writes credentials to disk and never prints them.
|
||||
%s 2>&1
|
||||
|
||||
# Fix ownership of files spx admin init wrote. spx runs as root under
|
||||
# systemd, so /etc/spinifex/{spinifex.toml,master.key,ca.key,*.pem} and any
|
||||
# per-service files written under /var/lib/spinifex/* land as root:root.
|
||||
# Delegate to setup.sh's fix_file_ownership (single source of truth) — it
|
||||
# knows the per-file modes (ca.key 0600, *.pem 0644, toml/key 0640) that a
|
||||
# blunt recursive chmod would clobber.
|
||||
SETUP_STAGES=fixown /usr/local/share/spinifex/setup.sh
|
||||
# The /var/lib/spinifex/awsgw/config symlink was created at build time by
|
||||
# setup.sh's create_directories — it resolves to /etc/spinifex, so
|
||||
# {BaseDir}/config/master.key automatically points at /etc/spinifex/master.key.
|
||||
|
||||
# Bootstrap the spinifex user's SSH directory with a generated key pair so
|
||||
# ~/.ssh exists and the operator can immediately use ssh-keygen / ssh-copy-id
|
||||
# without hitting "No such file or directory" errors documented in the setup guide.
|
||||
if [ ! -f /home/spinifex/.ssh/id_ed25519 ]; then
|
||||
mkdir -p /home/spinifex/.ssh
|
||||
chmod 700 /home/spinifex/.ssh
|
||||
ssh-keygen -t ed25519 -f /home/spinifex/.ssh/id_ed25519 -N "" -C "spinifex@$(hostname)"
|
||||
chown -R spinifex:spinifex /home/spinifex/.ssh
|
||||
chmod 600 /home/spinifex/.ssh/id_ed25519
|
||||
chmod 644 /home/spinifex/.ssh/id_ed25519.pub
|
||||
fi
|
||||
|
||||
# Copy AWS credentials to the spinifex user's home directory.
|
||||
# spx admin init runs with HOME=/root (set by the systemd unit), so credentials
|
||||
# land in /root/.aws/. Copy them to the spinifex user's home so the operator
|
||||
# can use the AWS CLI without sudo.
|
||||
if [ -f /root/.aws/credentials ]; then
|
||||
mkdir -p /home/spinifex/.aws
|
||||
cp /root/.aws/credentials /home/spinifex/.aws/credentials
|
||||
cp /root/.aws/config /home/spinifex/.aws/config 2>/dev/null || true
|
||||
chown -R spinifex:spinifex /home/spinifex/.aws
|
||||
chmod 700 /home/spinifex/.aws
|
||||
chmod 600 /home/spinifex/.aws/credentials
|
||||
[ -f /home/spinifex/.aws/config ] && chmod 600 /home/spinifex/.aws/config
|
||||
fi
|
||||
|
||||
# Enable + activate spinifex.target and the banner now that all configs are
|
||||
# in place. "enable --now" creates the multi-user.target.wants/ symlinks (so
|
||||
# they start directly on every subsequent boot — no longer dependent on
|
||||
# firstboot running each time, since firstboot is condition-skipped after
|
||||
# this run via the marker) and activates them in this boot.
|
||||
# NOTE: Moved to firstinstall to enable
|
||||
#systemctl enable --now spinifex.target spinifex-banner.service
|
||||
|
||||
# Enable services to start, on reboot
|
||||
systemctl enable spinifex.target spinifex-banner.service
|
||||
# setup.sh enables this on the package install path, but the ISO chroot returns
|
||||
# before that line (no systemd to talk to), so the ISO needs it here. An nft
|
||||
# ruleset is runtime state and does not survive a reboot; without this the unit
|
||||
# never reapplies the policy and the node comes back up open. Enabled, not
|
||||
# started: ConditionPathExists holds it inert until the daemon writes a peer file.
|
||||
systemctl enable spinifex-firewall.service
|
||||
# WantedBy=timers.target only self-activates once enabled — without this the
|
||||
# JetStream ENOSPC-latch watchdog never runs and a full disk needs a manual restart.
|
||||
systemctl enable --now spinifex-nats-watchdog.timer
|
||||
|
||||
# Mark complete only after every step above has succeeded. Until this point,
|
||||
# any failure (set -e) leaves the marker absent and the next reboot retries
|
||||
# firstboot from the top.
|
||||
mkdir -p "$(dirname "$DONE_MARKER")"
|
||||
touch "$DONE_MARKER"
|
||||
%s
|
||||
systemctl start spinifex.target
|
||||
|
||||
# Wait for the daemon to bring up external networking. When external_mode=pool
|
||||
# is configured the daemon creates br-ext (the OVN external bridge) during
|
||||
# startup; launching instances before it is ready results in no public IP being
|
||||
# assigned. ip link requires no root and avoids parsing daemon logs.
|
||||
if grep -q 'external_mode.*pool' /etc/spinifex/spinifex.toml 2>/dev/null; then
|
||||
echo "[firstboot] waiting for external networking (br-ext)..."
|
||||
for _i in $(seq 1 30); do
|
||||
if ip link show br-ext >/dev/null 2>&1; then
|
||||
echo "[firstboot] br-ext ready (${_i}s)"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
if ! ip link show br-ext >/dev/null 2>&1; then
|
||||
echo "[firstboot] warning: br-ext not up after 30s — external networking may be delayed"
|
||||
fi
|
||||
fi
|
||||
`, cfg.Hostname, ovnPrestart, setupOVN, clusterCmd, callbackBlock)
|
||||
|
||||
path := filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh")
|
||||
return os.WriteFile(path, []byte(script), 0o755)
|
||||
}
|
||||
|
||||
func buildClusterCmd(cfg Config) string {
|
||||
if cfg.SkipFormation {
|
||||
return `echo "[firstboot] cluster formation skipped — provisioning controller will handle it"`
|
||||
}
|
||||
emailFlag := ""
|
||||
if cfg.Email != "" {
|
||||
// shellEscapeSingle keeps the email safe if it ever contains a
|
||||
// character shell treats specially — belt-and-braces; the regex
|
||||
// validator already rejects whitespace and @-chains.
|
||||
emailFlag = " --email=" + shellEscapeSingle(cfg.Email)
|
||||
}
|
||||
gpuFlag := ""
|
||||
if cfg.GPUPassthrough {
|
||||
gpuFlag = " --gpu-passthrough"
|
||||
}
|
||||
// Without these the node takes the 0.0.0.0 wildcard default and every
|
||||
// internal service resolves onto the auto-detected WAN address, which is
|
||||
// exactly what the three-plane model exists to prevent.
|
||||
bindFlags := ""
|
||||
if cfg.LANIP != "" {
|
||||
bindFlags = fmt.Sprintf(" --bind %s --cluster-bind %s", cfg.LANIP, cfg.LANIP)
|
||||
}
|
||||
// --advertise must be explicit whenever --bind is: spx echoes a concrete
|
||||
// bind address straight back as the advertise address and never reaches its
|
||||
// WAN auto-detection, which would silently publish the internal plane as
|
||||
// this node's public dial target.
|
||||
preamble := ""
|
||||
switch {
|
||||
case bindFlags == "":
|
||||
// Nothing pinned, so spx auto-detects both and a guessed advertise
|
||||
// address would only get in the way.
|
||||
case cfg.WANIP != "":
|
||||
bindFlags += " --advertise " + cfg.WANIP
|
||||
default:
|
||||
// A DHCP wan has no address at install time, but it does by the time
|
||||
// firstboot runs, so read it off the bridge instead of shipping the
|
||||
// bind address as the public one.
|
||||
preamble = wanAdvertisePreamble
|
||||
bindFlags += ` $SPX_ADVERTISE`
|
||||
}
|
||||
|
||||
// Always a single-node cluster. The installer cannot form a multi-node one:
|
||||
// cluster membership decides the OVN database topology and the join token
|
||||
// only exists once the primary has booted, so neither is knowable while the
|
||||
// nodes are still being installed. Multi-node is a post-install conversion,
|
||||
// documented in the multi-node install guide.
|
||||
cmd := fmt.Sprintf("spx admin init --node %s --nodes 1%s%s%s", cfg.Hostname, bindFlags, emailFlag, gpuFlag)
|
||||
return preamble + cmd
|
||||
}
|
||||
|
||||
// wanAdvertisePreamble resolves the wan plane's DHCP lease into $SPX_ADVERTISE
|
||||
// just before formation. br-wan is always the wan bridge — that plane is the one
|
||||
// role that cannot fold — so its address is the node's public identity.
|
||||
//
|
||||
// Falling through with SPX_ADVERTISE empty is deliberate: no lease means there
|
||||
// is no public address to advertise yet, and spx echoing the bind address is
|
||||
// still better than formation failing outright.
|
||||
const wanAdvertisePreamble = `# The wan plane leases its address, so it is only knowable at boot. Without this
|
||||
# spx would echo --bind back as the advertise address and publish the internal
|
||||
# plane as this node's public dial target.
|
||||
SPX_ADVERTISE=""
|
||||
echo "[firstboot] waiting for the wan plane to acquire an address..."
|
||||
for _i in $(seq 1 60); do
|
||||
_wan_ip=$(ip -4 -o addr show br-wan scope global 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)
|
||||
if [ -n "$_wan_ip" ]; then
|
||||
SPX_ADVERTISE=" --advertise $_wan_ip"
|
||||
echo "[firstboot] wan address $_wan_ip (${_i}s)"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
if [ -z "$SPX_ADVERTISE" ]; then
|
||||
echo "[firstboot] warning: br-wan has no address after 60s — this node will advertise its bind address"
|
||||
fi
|
||||
`
|
||||
|
||||
// shellEscapeSingle wraps s in single quotes with any embedded single
|
||||
// quotes escaped. Minimal — we only need this because the email value is
|
||||
// interpolated into a shell script written by Write().
|
||||
func shellEscapeSingle(s string) string {
|
||||
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
||||
}
|
||||
@@ -1,284 +0,0 @@
|
||||
package firstboot
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func makeRootDirs(t *testing.T, root string) {
|
||||
t.Helper()
|
||||
for _, d := range []string{
|
||||
"usr/local/bin",
|
||||
"etc/systemd/system",
|
||||
"etc/systemd/system/multi-user.target.wants",
|
||||
} {
|
||||
if err := os.MkdirAll(filepath.Join(root, d), 0o755); err != nil {
|
||||
t.Fatalf("mkdir %s: %v", d, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteScriptNoCallbackWhenEmpty(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
makeRootDirs(t, root)
|
||||
|
||||
cfg := Config{Hostname: "test-node"}
|
||||
if err := Write(root, cfg); err != nil {
|
||||
t.Fatalf("Write: %v", err)
|
||||
}
|
||||
|
||||
script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
|
||||
if err != nil {
|
||||
t.Fatalf("read script: %v", err)
|
||||
}
|
||||
if strings.Contains(string(script), "curl") {
|
||||
t.Error("script should not contain curl when InstallCallback is empty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteScriptEmbedsCurlWhenCallbackSet(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
makeRootDirs(t, root)
|
||||
|
||||
const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
|
||||
cfg := Config{
|
||||
Hostname: "test-node",
|
||||
InstallCallback: callbackURL,
|
||||
}
|
||||
if err := Write(root, cfg); err != nil {
|
||||
t.Fatalf("Write: %v", err)
|
||||
}
|
||||
|
||||
script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
|
||||
if err != nil {
|
||||
t.Fatalf("read script: %v", err)
|
||||
}
|
||||
content := string(script)
|
||||
|
||||
if !strings.Contains(content, "curl") {
|
||||
t.Error("script missing curl command")
|
||||
}
|
||||
if !strings.Contains(content, callbackURL) {
|
||||
t.Errorf("script missing callback URL %q", callbackURL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteScriptRunsOVNWhenFormationOwned(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
makeRootDirs(t, root)
|
||||
|
||||
cfg := Config{Hostname: "test-node"}
|
||||
if err := Write(root, cfg); err != nil {
|
||||
t.Fatalf("Write: %v", err)
|
||||
}
|
||||
content := readScript(t, root)
|
||||
|
||||
if !strings.Contains(content, "setup-ovn.sh --management") {
|
||||
t.Error("script should run setup-ovn --management when firstboot owns formation")
|
||||
}
|
||||
if !strings.Contains(content, "systemctl start ovn-central") {
|
||||
t.Error("script should pre-start ovn-central when firstboot owns formation")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteScriptDefersOVNWhenSkipFormation(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
makeRootDirs(t, root)
|
||||
|
||||
cfg := Config{Hostname: "test-node", SkipFormation: true}
|
||||
if err := Write(root, cfg); err != nil {
|
||||
t.Fatalf("Write: %v", err)
|
||||
}
|
||||
content := readScript(t, root)
|
||||
|
||||
if strings.Contains(content, "setup-ovn.sh --management") {
|
||||
t.Error("script must not run setup-ovn --management when a controller owns OVN")
|
||||
}
|
||||
if strings.Contains(content, "systemctl start ovn-central") {
|
||||
t.Error("script must not pre-start ovn-central when a controller owns OVN")
|
||||
}
|
||||
if !strings.Contains(content, "setup-ovn deferred") {
|
||||
t.Error("script should note setup-ovn is deferred under SkipFormation")
|
||||
}
|
||||
}
|
||||
|
||||
func readScript(t *testing.T, root string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
|
||||
if err != nil {
|
||||
t.Fatalf("read script: %v", err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// TestWriteScriptEnablesNatsWatchdogTimer pins that firstboot activates the
|
||||
// JetStream ENOSPC-latch watchdog timer. The unit file's WantedBy=timers.target
|
||||
// only takes effect once enabled — without this line the watchdog is dropped
|
||||
// onto disk but never runs, and a full disk needs a manual nats restart forever.
|
||||
func TestWriteScriptEnablesNatsWatchdogTimer(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
makeRootDirs(t, root)
|
||||
|
||||
cfg := Config{Hostname: "test-node"}
|
||||
if err := Write(root, cfg); err != nil {
|
||||
t.Fatalf("Write: %v", err)
|
||||
}
|
||||
content := readScript(t, root)
|
||||
|
||||
if !strings.Contains(content, "systemctl enable --now spinifex-nats-watchdog.timer") {
|
||||
t.Error("firstboot script must enable --now spinifex-nats-watchdog.timer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteScriptCallbackAfterDoneMarker(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
makeRootDirs(t, root)
|
||||
|
||||
const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
|
||||
cfg := Config{
|
||||
Hostname: "node1",
|
||||
InstallCallback: callbackURL,
|
||||
}
|
||||
if err := Write(root, cfg); err != nil {
|
||||
t.Fatalf("Write: %v", err)
|
||||
}
|
||||
|
||||
script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
|
||||
if err != nil {
|
||||
t.Fatalf("read script: %v", err)
|
||||
}
|
||||
content := string(script)
|
||||
|
||||
doneIdx := strings.Index(content, "touch \"$DONE_MARKER\"")
|
||||
curlIdx := strings.Index(content, "curl")
|
||||
if doneIdx < 0 {
|
||||
t.Fatal("done marker not found in script")
|
||||
}
|
||||
if curlIdx < 0 {
|
||||
t.Fatal("curl not found in script")
|
||||
}
|
||||
if curlIdx < doneIdx {
|
||||
t.Error("curl must appear after done marker write")
|
||||
}
|
||||
}
|
||||
|
||||
// A multi-NIC node must keep cluster traffic on the internal plane while still
|
||||
// publishing the public one. --advertise has to be explicit to do that: spx
|
||||
// returns a concrete --bind verbatim as the advertise address and never reaches
|
||||
// its WAN auto-detection, so omitting it moves northstar's :53 listener and the
|
||||
// off-host dial target onto the internal plane.
|
||||
func TestBuildClusterCmdSeparatesBindFromAdvertise(t *testing.T) {
|
||||
cmd := buildClusterCmd(Config{
|
||||
Hostname: "hydrogen",
|
||||
LANIP: "10.0.0.3",
|
||||
WANIP: "216.218.163.99",
|
||||
})
|
||||
for _, want := range []string{
|
||||
"--bind 10.0.0.3",
|
||||
"--cluster-bind 10.0.0.3",
|
||||
"--advertise 216.218.163.99",
|
||||
} {
|
||||
if !strings.Contains(cmd, want) {
|
||||
t.Errorf("missing %q in:\n%s", want, cmd)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Without a bind address there is nothing to correct: spx auto-detects both,
|
||||
// and a stray --advertise would pin the node to an address the installer only
|
||||
// guessed at. A single-NIC node also collapses wan and lan onto one address,
|
||||
// where the split is meaningless.
|
||||
func TestBuildClusterCmdOmitsAdvertiseWithoutBind(t *testing.T) {
|
||||
cmd := buildClusterCmd(Config{
|
||||
Hostname: "node1",
|
||||
WANIP: "216.218.163.99",
|
||||
})
|
||||
if strings.Contains(cmd, "--advertise") {
|
||||
t.Errorf("advertise must not be passed without a bind address:\n%s", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
// A DHCP wan has no address at install time, so the advertise value has to be
|
||||
// resolved at boot. Shipping --bind alone here would republish the internal
|
||||
// plane as the node's public dial target, which is the case the static path
|
||||
// already guards against.
|
||||
func TestBuildClusterCmdResolvesDHCPWanAtBoot(t *testing.T) {
|
||||
cmd := buildClusterCmd(Config{
|
||||
Hostname: "hydrogen",
|
||||
LANIP: "10.0.0.3",
|
||||
// WANIP empty — the wan plane is on DHCP.
|
||||
})
|
||||
if !strings.Contains(cmd, "ip -4 -o addr show br-wan") {
|
||||
t.Errorf("must read the wan address off the bridge:\n%s", cmd)
|
||||
}
|
||||
if !strings.Contains(cmd, "$SPX_ADVERTISE") {
|
||||
t.Errorf("resolved address must reach the command:\n%s", cmd)
|
||||
}
|
||||
// The preamble has to run before the command that consumes it.
|
||||
if strings.Index(cmd, "SPX_ADVERTISE=\"\"") > strings.Index(cmd, "spx admin init") {
|
||||
t.Errorf("preamble must precede the spx invocation:\n%s", cmd)
|
||||
}
|
||||
if !strings.Contains(cmd, "--bind 10.0.0.3") {
|
||||
t.Errorf("bind still comes from the lan plane:\n%s", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
// With no bind address there is nothing to correct, so neither the literal flag
|
||||
// nor the boot-time lookup should appear — spx auto-detects both ends.
|
||||
func TestBuildClusterCmdSkipsWanLookupWithoutBind(t *testing.T) {
|
||||
cmd := buildClusterCmd(Config{Hostname: "node1"})
|
||||
if strings.Contains(cmd, "SPX_ADVERTISE") || strings.Contains(cmd, "--advertise") {
|
||||
t.Errorf("no bind means no advertise handling at all:\n%s", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
// The advertise handling injects a shell preamble into the generated script, so
|
||||
// the permutations that change its shape are syntax-checked rather than only
|
||||
// string-matched — a broken quote here bricks first boot with no installer error.
|
||||
func TestGeneratedScriptIsValidShell(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
cfg Config
|
||||
}{
|
||||
{"dhcp wan resolves at boot", Config{Hostname: "hydrogen", LANIP: "10.0.0.3"}},
|
||||
{"static wan is inlined", Config{Hostname: "hydrogen", LANIP: "10.0.0.3", WANIP: "216.218.163.99"}},
|
||||
{"single nic, nothing pinned", Config{Hostname: "node1"}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
makeRootDirs(t, root)
|
||||
if err := Write(root, tc.cfg); err != nil {
|
||||
t.Fatalf("Write: %v", err)
|
||||
}
|
||||
path := filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh")
|
||||
if out, err := exec.Command("bash", "-n", path).CombinedOutput(); err != nil {
|
||||
script, _ := os.ReadFile(path)
|
||||
t.Fatalf("generated script is not valid shell: %v\n%s\n---\n%s", err, out, script)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The installer cannot form a multi-node cluster: membership decides the OVN
|
||||
// database topology, and the join token only exists once the primary has
|
||||
// booted. Neither is knowable at install time, so firstboot must only ever
|
||||
// initialize a single-node cluster and leave multi-node to a post-install
|
||||
// conversion.
|
||||
func TestBuildClusterCmdNeverJoins(t *testing.T) {
|
||||
for _, cfg := range []Config{
|
||||
{Hostname: "node1"},
|
||||
{Hostname: "node1", LANIP: "10.0.0.3", WANIP: "216.218.163.99"},
|
||||
{Hostname: "node1", LANIP: "10.0.0.3"},
|
||||
} {
|
||||
cmd := buildClusterCmd(cfg)
|
||||
if strings.Contains(cmd, "spx admin join") {
|
||||
t.Errorf("firstboot must never join a cluster:\n%s", cmd)
|
||||
}
|
||||
if !strings.Contains(cmd, "--nodes 1") {
|
||||
t.Errorf("want a single-node init, got:\n%s", cmd)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,45 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Diversion wrapper for grub-install on a spinifex-boot-tool system.
|
||||
#
|
||||
# On ZFS root the bootloader lives on per-disk ESPs, not in /boot. A plain
|
||||
# grub-install writes a core image that tries to read the pool, which works
|
||||
# until the next OpenZFS feature flag lands and then silently does not. Worse,
|
||||
# the grub-pc and shim packages run grub-install from their own postinst, so an
|
||||
# ordinary `apt upgrade` would undo the layout with no visible error.
|
||||
#
|
||||
# Mirrors the wrapper Proxmox ships for the same reason.
|
||||
|
||||
set -e
|
||||
|
||||
REAL="/usr/sbin/grub-install.real"
|
||||
TOOL="/usr/sbin/spinifex-boot-tool"
|
||||
|
||||
# Not a managed system: behave exactly as the real grub-install.
|
||||
if ! "$TOOL" status --quiet 2>/dev/null; then
|
||||
exec "$REAL" "$@"
|
||||
fi
|
||||
|
||||
# dpkg sets this when a maintainer script is running. The bootloader packages
|
||||
# reinstalling themselves is expected and is redirected, not refused.
|
||||
case "${DPKG_MAINTSCRIPT_PACKAGE:-}" in
|
||||
grub-* | shim-*)
|
||||
echo "This system is managed by spinifex-boot-tool; redirecting grub-install to 'spinifex-boot-tool reinit'." >&2
|
||||
exec "$TOOL" reinit
|
||||
;;
|
||||
esac
|
||||
|
||||
cat >&2 <<EOF
|
||||
E: This system's bootloader is managed by spinifex-boot-tool.
|
||||
|
||||
Running grub-install directly would install GRUB somewhere the firmware does
|
||||
not read, and would leave the per-disk ESPs holding a stale kernel.
|
||||
|
||||
Use instead:
|
||||
spinifex-boot-tool refresh re-sync kernels and grub.cfg to every ESP
|
||||
spinifex-boot-tool reinit reinstall GRUB on every ESP
|
||||
spinifex-boot-tool status list the ESPs under management
|
||||
|
||||
To run the real grub-install anyway (you probably do not want this):
|
||||
$REAL $*
|
||||
EOF
|
||||
exit 1
|
||||
@@ -1,412 +0,0 @@
|
||||
#!/bin/bash
|
||||
# spinifex-boot-tool — keep a bootloader on every disk of a ZFS root pool.
|
||||
#
|
||||
# GRUB cannot be trusted to read a ZFS pool: its ZFS reader lags OpenZFS by
|
||||
# years, and enabling a new feature flag on the pool can make the machine
|
||||
# unbootable with no warning. So /boot on the pool is never read by firmware.
|
||||
# Instead each pool member carries a small FAT32 ESP holding its own copy of
|
||||
# GRUB, the kernels and the initramfs images, and this tool keeps them in step.
|
||||
#
|
||||
# Modelled directly on Proxmox's proxmox-boot-tool, which has run this design
|
||||
# across their entire installed base for years.
|
||||
|
||||
set -u
|
||||
set -o pipefail
|
||||
|
||||
ESP_LIST="/etc/kernel/spinifex-boot-uuids"
|
||||
ESPMOUNTS="/var/tmp/espmounts"
|
||||
MARKER="spinifex-boot-uuids"
|
||||
|
||||
# Kernels are only ever copied out of the real /boot, which lives on the pool.
|
||||
# Once an ESP is bind-mounted over /boot this variable is how we still find it.
|
||||
BOOTROOT="/boot"
|
||||
|
||||
warn() { echo "W: $*" >&2; }
|
||||
die() { echo "E: $*" >&2; exit 1; }
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $(basename "$0") <command>
|
||||
|
||||
format <partition> [--force] make <partition> a Spinifex ESP
|
||||
init <partition> register <partition> and install a bootloader on it
|
||||
clean [--dry-run] drop ESPs that are no longer present
|
||||
refresh [--hook <name>] re-sync kernels and grub.cfg to every ESP
|
||||
reinit re-run refresh, reinstalling GRUB on every ESP
|
||||
kernel <add|remove> <ver> pin or unpin a kernel version
|
||||
status [--quiet] list registered ESPs
|
||||
EOF
|
||||
}
|
||||
|
||||
# reexec_in_mountns re-runs this script inside a private mount namespace. Every
|
||||
# code path here bind-mounts an ESP over /boot; if the script is killed at the
|
||||
# wrong moment outside a namespace, the running system is left with its real
|
||||
# /boot masked and the next kernel upgrade writes into a FAT partition.
|
||||
reexec_in_mountns() {
|
||||
if [ "${SPINIFEX_BOOT_TOOL_NS:-0}" = "1" ]; then
|
||||
return 0
|
||||
fi
|
||||
export SPINIFEX_BOOT_TOOL_NS=1
|
||||
# --propagation private stops the bind mounts leaking back to the host.
|
||||
exec unshare --mount --propagation private -- "$0" "$@"
|
||||
}
|
||||
|
||||
# --- ESP registry -----------------------------------------------------------
|
||||
|
||||
# esp_list prints the registered ESP filesystem UUIDs.
|
||||
esp_list() {
|
||||
[ -f "$ESP_LIST" ] || return 0
|
||||
grep -v '^[[:space:]]*\(#\|$\)' "$ESP_LIST" || true
|
||||
}
|
||||
|
||||
# uuid_of prints the FAT filesystem UUID of a partition.
|
||||
uuid_of() {
|
||||
blkid "$1" -o value -s UUID 2>/dev/null
|
||||
}
|
||||
|
||||
# path_for_uuid resolves a registered UUID back to a device, or nothing if the
|
||||
# disk has been pulled.
|
||||
path_for_uuid() {
|
||||
local dev
|
||||
dev="/dev/disk/by-uuid/$1"
|
||||
[ -e "$dev" ] || return 1
|
||||
readlink -f "$dev"
|
||||
}
|
||||
|
||||
# mount_esp mounts the ESP for a UUID and echoes its mountpoint.
|
||||
mount_esp() {
|
||||
local uuid="$1" dev mp
|
||||
dev=$(path_for_uuid "$uuid") || return 1
|
||||
mp="$ESPMOUNTS/$uuid"
|
||||
mkdir -p "$mp"
|
||||
mountpoint -q "$mp" || mount "$dev" "$mp" || return 1
|
||||
echo "$mp"
|
||||
}
|
||||
|
||||
umount_esp() {
|
||||
local mp="$ESPMOUNTS/$1"
|
||||
mountpoint -q "$mp" && umount "$mp"
|
||||
rmdir "$mp" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# --- format / init ----------------------------------------------------------
|
||||
|
||||
do_format() {
|
||||
local part="$1" force="${2:-}"
|
||||
[ -b "$part" ] || die "$part is not a block device"
|
||||
|
||||
if [ "$force" != "--force" ]; then
|
||||
local existing
|
||||
existing=$(blkid -o value -s TYPE "$part" 2>/dev/null || true)
|
||||
[ -z "$existing" ] || die "$part already contains a $existing filesystem; pass --force to overwrite"
|
||||
fi
|
||||
mkfs.vfat -F 32 "$part" || die "mkfs.vfat failed on $part"
|
||||
}
|
||||
|
||||
do_init() {
|
||||
local part="$1"
|
||||
[ -b "$part" ] || die "$part is not a block device"
|
||||
|
||||
local uuid
|
||||
uuid=$(uuid_of "$part") || true
|
||||
[ -n "$uuid" ] || die "$part has no filesystem UUID — run '$(basename "$0") format $part' first"
|
||||
|
||||
mkdir -p "$(dirname "$ESP_LIST")"
|
||||
touch "$ESP_LIST"
|
||||
if ! grep -qx "$uuid" "$ESP_LIST"; then
|
||||
echo "$uuid" >>"$ESP_LIST"
|
||||
fi
|
||||
echo "Registered $part ($uuid)"
|
||||
|
||||
install_hooks
|
||||
# Only the ESP just registered: the others are already current, and
|
||||
# refreshing all of them here makes registering N disks an N-squared
|
||||
# sequence of grub-installs.
|
||||
refresh_one "$uuid" || die "could not make $part bootable"
|
||||
}
|
||||
|
||||
do_clean() {
|
||||
local dry="${1:-}" uuid keep=()
|
||||
for uuid in $(esp_list); do
|
||||
if path_for_uuid "$uuid" >/dev/null 2>&1; then
|
||||
keep+=("$uuid")
|
||||
else
|
||||
echo "Removing stale ESP $uuid"
|
||||
fi
|
||||
done
|
||||
if [ "$dry" = "--dry-run" ]; then
|
||||
return 0
|
||||
fi
|
||||
printf '%s\n' "${keep[@]+"${keep[@]}"}" >"$ESP_LIST"
|
||||
}
|
||||
|
||||
# --- refresh ----------------------------------------------------------------
|
||||
|
||||
# kernel_versions lists the kernels present in the real /boot, newest first, so
|
||||
# the default menu entry is the newest kernel.
|
||||
kernel_versions() {
|
||||
local img
|
||||
for img in "$BOOTROOT"/vmlinuz-*; do
|
||||
[ -e "$img" ] || continue
|
||||
basename "$img" | sed 's/^vmlinuz-//'
|
||||
done | sort -Vr
|
||||
}
|
||||
|
||||
# sync_esp copies the current kernels and initramfs images onto one ESP and
|
||||
# removes anything left from a kernel that is no longer installed — an ESP is
|
||||
# small and a stale 100MB initramfs is what makes the next upgrade fail.
|
||||
sync_esp() {
|
||||
local mp="$1" kver f base
|
||||
|
||||
local -A wanted=()
|
||||
for kver in $(kernel_versions); do
|
||||
wanted["vmlinuz-$kver"]=1
|
||||
wanted["initrd.img-$kver"]=1
|
||||
done
|
||||
|
||||
for f in "$mp"/vmlinuz-* "$mp"/initrd.img-*; do
|
||||
[ -e "$f" ] || continue
|
||||
base=$(basename "$f")
|
||||
[ -n "${wanted[$base]:-}" ] || rm -f "$f"
|
||||
done
|
||||
|
||||
for kver in $(kernel_versions); do
|
||||
for base in "vmlinuz-$kver" "initrd.img-$kver"; do
|
||||
[ -e "$BOOTROOT/$base" ] || continue
|
||||
# Copy to a temp name and rename: a half-written kernel on the ESP
|
||||
# is a machine that does not come back from a reboot.
|
||||
cp -f "$BOOTROOT/$base" "$mp/$base.tmp" || return 1
|
||||
mv -f "$mp/$base.tmp" "$mp/$base" || return 1
|
||||
done
|
||||
done
|
||||
|
||||
# GRUB reads its theme from $prefix, which is the ESP's own grub directory,
|
||||
# so the branding has to be copied out of /boot alongside the kernels.
|
||||
mkdir -p "$mp/grub/fonts"
|
||||
for f in grub/theme.txt grub/splash.png grub/fonts/unicode.pf2; do
|
||||
[ -e "$BOOTROOT/$f" ] && cp -f "$BOOTROOT/$f" "$mp/$f"
|
||||
done
|
||||
|
||||
# Marks the partition as ours, so a later `format` on the wrong disk is
|
||||
# obviously a mistake rather than a silent overwrite.
|
||||
echo "$MARKER" >"$mp/$MARKER"
|
||||
return 0
|
||||
}
|
||||
|
||||
# grub_install_bin echoes the real grub-install. On a managed system the normal
|
||||
# name is a wrapper that refuses to run, and it must not intercept this tool's
|
||||
# own calls — that would make the refusal a self-deadlock.
|
||||
grub_install_bin() {
|
||||
if [ -x /usr/sbin/grub-install.real ]; then
|
||||
echo /usr/sbin/grub-install.real
|
||||
else
|
||||
echo grub-install
|
||||
fi
|
||||
}
|
||||
|
||||
# install_grub puts GRUB itself on one ESP, for both firmware types.
|
||||
#
|
||||
# --boot-directory points at the ESP, not at /boot, so the GRUB core image
|
||||
# embeds a FAT reader and a search for the ESP's own UUID. It never needs to
|
||||
# read the pool, which is the entire point of this tool.
|
||||
install_grub() {
|
||||
local mp="$1" dev="$2" rc=0 grub
|
||||
grub="$(grub_install_bin)"
|
||||
|
||||
if [ ! -d /sys/firmware/efi ]; then
|
||||
"$grub" --target=i386-pc --boot-directory="$mp" --recheck "$dev" || rc=1
|
||||
return $rc
|
||||
fi
|
||||
|
||||
# NVRAM is written at install time only. A refresh runs on every kernel
|
||||
# upgrade, and rewriting boot variables that often has bricked firmware on
|
||||
# more than one vendor's boards.
|
||||
local nvram="--no-nvram"
|
||||
[ "${SPINIFEX_BOOT_TOOL_NVRAM:-0}" = "1" ] && nvram=""
|
||||
|
||||
# Twice: once as a named entry, and once at the removable path, which is all
|
||||
# some firmware will ever look at and is what keeps the disk bootable after
|
||||
# its NVRAM entry is lost or the drive is moved to another machine.
|
||||
# shellcheck disable=SC2086 # $nvram is a single optional flag, not a path
|
||||
"$grub" --target=x86_64-efi \
|
||||
--efi-directory="$mp" --boot-directory="$mp" \
|
||||
--bootloader-id=spinifex $nvram --recheck "$dev" || rc=1
|
||||
"$grub" --target=x86_64-efi \
|
||||
--efi-directory="$mp" --boot-directory="$mp" \
|
||||
--bootloader-id=spinifex --removable --no-nvram --recheck "$dev" || rc=1
|
||||
return $rc
|
||||
}
|
||||
|
||||
# update_grub_on regenerates grub.cfg for one ESP.
|
||||
#
|
||||
# The bind mount is the trick that makes this work: grub-mkconfig probes the
|
||||
# device behind /boot to decide how to find the kernel at boot. With the ESP
|
||||
# mounted there it emits a FAT UUID search and paths relative to the ESP root,
|
||||
# so the generated config matches what the embedded core image can actually do.
|
||||
update_grub_on() {
|
||||
local mp="$1" rc=0
|
||||
mount --bind "$mp" "$BOOTROOT" || return 1
|
||||
grub-mkconfig -o "$BOOTROOT/grub/grub.cfg" || rc=1
|
||||
umount "$BOOTROOT" || warn "could not unmount the bind mount over $BOOTROOT"
|
||||
return $rc
|
||||
}
|
||||
|
||||
# refresh_one makes a single registered ESP independently bootable: kernels,
|
||||
# bootloader, and a grub.cfg generated against that ESP's own UUID.
|
||||
refresh_one() {
|
||||
local uuid="$1" mp dev disk rc=0
|
||||
|
||||
if ! dev=$(path_for_uuid "$uuid"); then
|
||||
warn "ESP $uuid is not present, skipping"
|
||||
return 1
|
||||
fi
|
||||
if ! mp=$(mount_esp "$uuid"); then
|
||||
warn "could not mount ESP $uuid"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# The whole disk, not the partition: BIOS GRUB embeds into the boot
|
||||
# sector and the BIOS boot partition, both of which live on the disk.
|
||||
disk=$(lsblk -no pkname "$dev")
|
||||
disk="/dev/$disk"
|
||||
|
||||
if sync_esp "$mp" && install_grub "$mp" "$disk" && update_grub_on "$mp"; then
|
||||
echo "Synced $dev ($uuid)"
|
||||
else
|
||||
warn "failed to update ESP $uuid on $dev"
|
||||
rc=1
|
||||
fi
|
||||
umount_esp "$uuid"
|
||||
return $rc
|
||||
}
|
||||
|
||||
do_refresh() {
|
||||
local uuid failed=0 ok=0 total=0
|
||||
|
||||
local uuids
|
||||
uuids=$(esp_list)
|
||||
if [ -z "$uuids" ]; then
|
||||
echo "No ESPs registered, nothing to do."
|
||||
return 0
|
||||
fi
|
||||
|
||||
for uuid in $uuids; do
|
||||
total=$((total + 1))
|
||||
if refresh_one "$uuid"; then
|
||||
ok=$((ok + 1))
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$ok" -eq 0 ]; then
|
||||
die "no ESP could be updated ($failed of $total failed) — this system may not boot"
|
||||
fi
|
||||
if [ "$failed" -gt 0 ]; then
|
||||
# Strict mode is used by the installer, where a partly-updated set of
|
||||
# ESPs means the install is broken and should fail loudly now rather
|
||||
# than at the first reboot.
|
||||
if [ "${SPINIFEX_BOOT_TOOL_STRICT:-0}" = "1" ]; then
|
||||
die "$failed of $total ESPs failed to update"
|
||||
fi
|
||||
warn "$failed of $total ESPs failed to update; $ok succeeded, so the system will still boot"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
do_status() {
|
||||
local quiet="${1:-}" uuid dev
|
||||
if [ ! -s "$ESP_LIST" ]; then
|
||||
[ "$quiet" = "--quiet" ] || echo "No ESPs registered — this system does not use spinifex-boot-tool."
|
||||
return 1
|
||||
fi
|
||||
[ "$quiet" = "--quiet" ] && return 0
|
||||
for uuid in $(esp_list); do
|
||||
if dev=$(path_for_uuid "$uuid"); then
|
||||
echo "$uuid $dev"
|
||||
else
|
||||
echo "$uuid MISSING"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
do_kernel() {
|
||||
local action="$1" kver="${2:-}"
|
||||
local pinfile="/etc/kernel/spinifex-boot-pin"
|
||||
case "$action" in
|
||||
add)
|
||||
[ -n "$kver" ] || die "kernel add needs a version"
|
||||
touch "$pinfile"
|
||||
grep -qx "$kver" "$pinfile" || echo "$kver" >>"$pinfile"
|
||||
;;
|
||||
remove)
|
||||
[ -n "$kver" ] || die "kernel remove needs a version"
|
||||
[ -f "$pinfile" ] && sed -i "\|^$kver\$|d" "$pinfile"
|
||||
;;
|
||||
*) die "unknown kernel action: $action" ;;
|
||||
esac
|
||||
do_refresh
|
||||
}
|
||||
|
||||
# --- hooks ------------------------------------------------------------------
|
||||
|
||||
# install_hooks wires this tool into every package operation that would
|
||||
# otherwise write a bootloader to the wrong place.
|
||||
install_hooks() {
|
||||
mkdir -p /etc/kernel/postinst.d /etc/kernel/postrm.d /etc/initramfs/post-update.d
|
||||
|
||||
# Debian's own hook runs update-grub against the real /boot, which is on the
|
||||
# pool. Firmware never reads that, so it would produce a config nothing uses
|
||||
# while the ESPs silently kept an old kernel.
|
||||
if [ -e /etc/kernel/postinst.d/zz-update-grub ]; then
|
||||
dpkg-divert --divert /etc/kernel/postinst.d/zz-update-grub.distrib \
|
||||
--rename --add /etc/kernel/postinst.d/zz-update-grub >/dev/null 2>&1 || true
|
||||
fi
|
||||
|
||||
for hook in /etc/kernel/postinst.d/zz-spinifex-boot-tool \
|
||||
/etc/kernel/postrm.d/zz-spinifex-boot-tool \
|
||||
/etc/initramfs/post-update.d/spinifex-boot-tool; do
|
||||
cat >"$hook" <<'HOOK'
|
||||
#!/bin/sh
|
||||
# Re-sync the ESPs after a kernel or initramfs change.
|
||||
set -e
|
||||
exec /usr/sbin/spinifex-boot-tool refresh --hook "$(basename "$0")"
|
||||
HOOK
|
||||
chmod 0755 "$hook"
|
||||
done
|
||||
}
|
||||
|
||||
# --- dispatch ---------------------------------------------------------------
|
||||
|
||||
cmd="${1:-}"
|
||||
[ $# -gt 0 ] && shift
|
||||
|
||||
case "$cmd" in
|
||||
format) do_format "$@" ;;
|
||||
init)
|
||||
reexec_in_mountns init "$@"
|
||||
do_init "$@"
|
||||
;;
|
||||
refresh)
|
||||
# --hook <name> is passed by the kernel hooks purely so the reason for a
|
||||
# refresh is visible in the log.
|
||||
if [ "${1:-}" = "--hook" ]; then shift 2 || shift; fi
|
||||
reexec_in_mountns refresh
|
||||
do_refresh
|
||||
;;
|
||||
reinit)
|
||||
reexec_in_mountns reinit
|
||||
do_refresh
|
||||
;;
|
||||
clean) do_clean "$@" ;;
|
||||
kernel)
|
||||
reexec_in_mountns kernel "$@"
|
||||
do_kernel "$@"
|
||||
;;
|
||||
status) do_status "$@" ;;
|
||||
help | --help | -h | "") usage ;;
|
||||
*)
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -1,406 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Disk is one candidate block device with every attribute the installer needs
|
||||
// to validate a selection: sizes for the same-size rule, block sizes for ashift
|
||||
// and the 4Kn/BIOS check, rotational for autotrim, and ByID for pool members.
|
||||
type Disk struct {
|
||||
Path string // /dev/sda
|
||||
ByID string // /dev/disk/by-id/ata-… — stable across controller reordering
|
||||
Bytes int64
|
||||
Model string
|
||||
|
||||
LogicalBlockSize int
|
||||
PhysicalBlockSize int
|
||||
|
||||
Rotational bool
|
||||
Removable bool
|
||||
|
||||
// LiveMedia marks the device the installer itself booted from. Selecting it
|
||||
// would erase the running system mid-install.
|
||||
LiveMedia bool
|
||||
|
||||
// Content is a human-readable summary of what is on the disk today
|
||||
// ("ext4 filesystem", "empty"), shown before the operator confirms erasure.
|
||||
Content string
|
||||
}
|
||||
|
||||
// skipPrefixes are kernel block devices that are never install targets:
|
||||
// virtual, mapped, or removable media. Mirrors Proxmox's hd_list filter.
|
||||
var skipPrefixes = []string{"ram", "loop", "md", "dm-", "fd", "sr", "zram", "nbd"}
|
||||
|
||||
// The kernel-owned trees the scan reads, indirected so tests can point it at a
|
||||
// fixture instead of the running machine's hardware.
|
||||
var (
|
||||
sysBlockDir = "/sys/block"
|
||||
sysClassBlockDir = "/sys/class/block"
|
||||
devByIDDir = "/dev/disk/by-id"
|
||||
procMountsPath = "/proc/mounts"
|
||||
procSwapsPath = "/proc/swaps"
|
||||
)
|
||||
|
||||
// ListDisks enumerates install-candidate disks from /sys/block. It reads sysfs
|
||||
// and blkid only — pool detection is separate (ImportablePools) because it is
|
||||
// far slower and not every caller needs it.
|
||||
func ListDisks() ([]Disk, error) {
|
||||
entries, err := os.ReadDir(sysBlockDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read /sys/block: %w", err)
|
||||
}
|
||||
|
||||
byID := buildByIDIndex()
|
||||
live := liveMediaDevices()
|
||||
|
||||
var disks []Disk
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if hasAnyPrefix(name, skipPrefixes) {
|
||||
continue
|
||||
}
|
||||
// A zero-sized device is an empty card reader slot, not a disk.
|
||||
sectors := sysfsInt(name, "size")
|
||||
if sectors == 0 {
|
||||
continue
|
||||
}
|
||||
d := Disk{
|
||||
Path: "/dev/" + name,
|
||||
ByID: byID[name],
|
||||
Bytes: sectors * 512,
|
||||
Model: sysfsString(name, "device/model"),
|
||||
LogicalBlockSize: sysfsIntDefault(name, "queue/logical_block_size", 512),
|
||||
PhysicalBlockSize: sysfsIntDefault(name, "queue/physical_block_size", 512),
|
||||
Rotational: sysfsInt(name, "queue/rotational") == 1,
|
||||
Removable: sysfsInt(name, "removable") == 1,
|
||||
LiveMedia: live[name],
|
||||
}
|
||||
d.Content = describeContent(d.Path)
|
||||
disks = append(disks, d)
|
||||
}
|
||||
slices.SortFunc(disks, func(a, b Disk) int { return strings.Compare(a.Path, b.Path) })
|
||||
return disks, nil
|
||||
}
|
||||
|
||||
// SizeHuman renders the disk size for display, e.g. "1.7T".
|
||||
func (d Disk) SizeHuman() string {
|
||||
switch {
|
||||
case d.Bytes >= 1<<40:
|
||||
return fmt.Sprintf("%.1fT", float64(d.Bytes)/(1<<40))
|
||||
case d.Bytes >= 1<<30:
|
||||
return fmt.Sprintf("%.1fG", float64(d.Bytes)/(1<<30))
|
||||
case d.Bytes >= 1<<20:
|
||||
return fmt.Sprintf("%.1fM", float64(d.Bytes)/(1<<20))
|
||||
default:
|
||||
return fmt.Sprintf("%dB", d.Bytes)
|
||||
}
|
||||
}
|
||||
|
||||
// Stable returns the path to use when building a pool: the by-id path when one
|
||||
// exists, else the kernel name. Pools referencing /dev/sdX break when a
|
||||
// controller enumerates disks in a different order after a reboot.
|
||||
func (d Disk) Stable() string {
|
||||
if d.ByID != "" {
|
||||
return d.ByID
|
||||
}
|
||||
return d.Path
|
||||
}
|
||||
|
||||
// PartitionPath returns the device path for partition n of this disk. NVMe and
|
||||
// other devices whose name ends in a digit take a 'p' separator.
|
||||
func (d Disk) PartitionPath(n int) string { return partitionPath(d.Path, n) }
|
||||
|
||||
// StablePartitionPath is PartitionPath against the by-id path, which udev
|
||||
// publishes with a "-partN" suffix rather than the kernel's 'p' convention.
|
||||
func (d Disk) StablePartitionPath(n int) string {
|
||||
if d.ByID == "" {
|
||||
return d.PartitionPath(n)
|
||||
}
|
||||
return fmt.Sprintf("%s-part%d", d.ByID, n)
|
||||
}
|
||||
|
||||
func partitionPath(disk string, n int) string {
|
||||
if len(disk) > 0 && disk[len(disk)-1] >= '0' && disk[len(disk)-1] <= '9' {
|
||||
return fmt.Sprintf("%sp%d", disk, n)
|
||||
}
|
||||
return fmt.Sprintf("%s%d", disk, n)
|
||||
}
|
||||
|
||||
// readKernelPartitions returns the partition device names the kernel currently
|
||||
// publishes for a disk, sorted.
|
||||
//
|
||||
// It reads sysfs rather than /dev deliberately. A partition device node
|
||||
// outlives the table that created it, so a stale node is indistinguishable
|
||||
// from a fresh one by existence alone — which is how a disk whose old table
|
||||
// the kernel never dropped passes every check and then gets formatted at the
|
||||
// previous layout's offsets.
|
||||
func readKernelPartitions(disk string) ([]string, error) {
|
||||
name := filepath.Base(disk)
|
||||
entries, err := os.ReadDir(filepath.Join(sysBlockDir, name))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read the kernel's partition list for %s: %w", disk, err)
|
||||
}
|
||||
var out []string
|
||||
for _, e := range entries {
|
||||
// Partitions are the child directories carrying a "partition" attribute.
|
||||
// "queue", "device" and the plain attribute files sit alongside them.
|
||||
if _, err := os.Stat(filepath.Join(sysBlockDir, name, e.Name(), "partition")); err != nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
slices.Sort(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// kernelPartitions is indirected so tests can drive the partition-table
|
||||
// assertions without a real block device.
|
||||
var kernelPartitions = readKernelPartitions
|
||||
|
||||
// partitionDevice maps a kernel partition name back to its device node,
|
||||
// resolved alongside the disk it belongs to rather than assuming /dev.
|
||||
func partitionDevice(disk, part string) string {
|
||||
return filepath.Join(filepath.Dir(disk), part)
|
||||
}
|
||||
|
||||
// diskHolders names the md, LVM or device-mapper devices claiming this disk or
|
||||
// one of its partitions. A holder is the usual reason the kernel refuses to
|
||||
// re-read a partition table, so it is what the operator needs told when the
|
||||
// installer gives up on a disk.
|
||||
func diskHolders(disk string) []string {
|
||||
name := filepath.Base(disk)
|
||||
dirs := []string{filepath.Join(sysBlockDir, name, "holders")}
|
||||
parts, _ := kernelPartitions(disk)
|
||||
for _, p := range parts {
|
||||
dirs = append(dirs, filepath.Join(sysBlockDir, name, p, "holders"))
|
||||
}
|
||||
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
for _, dir := range dirs {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, e := range entries {
|
||||
if seen[e.Name()] {
|
||||
continue
|
||||
}
|
||||
seen[e.Name()] = true
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
}
|
||||
slices.Sort(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// diskMount is one filesystem mounted from a disk the installer is about to
|
||||
// erase.
|
||||
type diskMount struct {
|
||||
Device string
|
||||
Mountpoint string
|
||||
}
|
||||
|
||||
// devicesOnDisk is the set of device paths belonging to a disk: the whole disk
|
||||
// and every partition the kernel currently publishes. Matching /proc tables
|
||||
// against a set rather than a string prefix keeps /dev/nvme0n1 from claiming
|
||||
// /dev/nvme0n11.
|
||||
func devicesOnDisk(disk string) map[string]bool {
|
||||
out := map[string]bool{disk: true}
|
||||
parts, _ := kernelPartitions(disk)
|
||||
for _, p := range parts {
|
||||
out[partitionDevice(disk, p)] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// mountsOnDisk returns the filesystems mounted from a disk, deepest mountpoint
|
||||
// first so a nested mount is released before the one it sits inside.
|
||||
func mountsOnDisk(disk string) []diskMount {
|
||||
devs := devicesOnDisk(disk)
|
||||
var out []diskMount
|
||||
for _, fields := range procTable(procMountsPath, 0) {
|
||||
if len(fields) >= 2 && devs[fields[0]] {
|
||||
out = append(out, diskMount{Device: fields[0], Mountpoint: fields[1]})
|
||||
}
|
||||
}
|
||||
slices.SortFunc(out, func(a, b diskMount) int {
|
||||
return strings.Count(b.Mountpoint, "/") - strings.Count(a.Mountpoint, "/")
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// swapsOnDisk returns the active swap areas backed by a disk.
|
||||
func swapsOnDisk(disk string) []string {
|
||||
devs := devicesOnDisk(disk)
|
||||
var out []string
|
||||
// The first line of /proc/swaps is a column header, not an entry.
|
||||
for _, fields := range procTable(procSwapsPath, 1) {
|
||||
if len(fields) >= 1 && devs[fields[0]] {
|
||||
out = append(out, fields[0])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// procTable splits a whitespace-delimited /proc table into fields per line,
|
||||
// dropping the first skip lines and any blank ones.
|
||||
func procTable(path string, skip int) [][]string {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out [][]string
|
||||
n := 0
|
||||
for line := range strings.SplitSeq(string(data), "\n") {
|
||||
n++
|
||||
if n <= skip {
|
||||
continue
|
||||
}
|
||||
if fields := strings.Fields(line); len(fields) > 0 {
|
||||
out = append(out, fields)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// byIDPreference orders the /dev/disk/by-id/ namespaces we prefer to name a
|
||||
// pool member by. wwn- ids are last: they are stable but opaque, so a degraded
|
||||
// pool reports a number nobody can map to a drive bay.
|
||||
var byIDPreference = []string{"nvme-", "ata-", "scsi-", "virtio-", "usb-", "wwn-"}
|
||||
|
||||
// buildByIDIndex maps kernel device name to its preferred by-id path.
|
||||
func buildByIDIndex() map[string]string {
|
||||
entries, err := os.ReadDir(devByIDDir)
|
||||
if err != nil {
|
||||
return map[string]string{}
|
||||
}
|
||||
// Track the chosen preference rank per device so a later, less-preferred
|
||||
// id cannot displace a better one already found.
|
||||
rank := map[string]int{}
|
||||
out := map[string]string{}
|
||||
for _, e := range entries {
|
||||
id := e.Name()
|
||||
// Partition links describe a slice of the disk, not the disk.
|
||||
if strings.Contains(id, "-part") {
|
||||
continue
|
||||
}
|
||||
target, err := filepath.EvalSymlinks(filepath.Join(devByIDDir, id))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
dev := filepath.Base(target)
|
||||
r := prefixRank(id)
|
||||
if existing, ok := rank[dev]; ok && existing <= r {
|
||||
continue
|
||||
}
|
||||
rank[dev] = r
|
||||
out[dev] = filepath.Join(devByIDDir, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func prefixRank(id string) int {
|
||||
for i, p := range byIDPreference {
|
||||
if strings.HasPrefix(id, p) {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return len(byIDPreference)
|
||||
}
|
||||
|
||||
// liveMediaDevices returns the set of parent disk names backing the live
|
||||
// installer's own media, so they can never be selected as install targets.
|
||||
func liveMediaDevices() map[string]bool {
|
||||
out := map[string]bool{}
|
||||
data, err := os.ReadFile(procMountsPath)
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
liveMounts := []string{"/cdrom", "/run/live/medium", "/lib/live/mount/medium"}
|
||||
for line := range strings.SplitSeq(string(data), "\n") {
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 2 || !strings.HasPrefix(fields[0], "/dev/") {
|
||||
continue
|
||||
}
|
||||
if !slices.Contains(liveMounts, fields[1]) {
|
||||
continue
|
||||
}
|
||||
if parent := parentDisk(filepath.Base(fields[0])); parent != "" {
|
||||
out[parent] = true
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// parentDisk resolves a partition's kernel name to its whole-disk name by
|
||||
// walking /sys/class/block/<part>/.. — the sysfs parent of a partition is its
|
||||
// disk. Returns the input unchanged when it is already a whole disk.
|
||||
func parentDisk(dev string) string {
|
||||
link, err := filepath.EvalSymlinks(filepath.Join(sysClassBlockDir, dev))
|
||||
if err != nil {
|
||||
return dev
|
||||
}
|
||||
// A whole disk lives directly under a bus path; a partition sits one level
|
||||
// deeper, inside its disk's directory.
|
||||
if _, err := os.Stat(filepath.Join(link, "partition")); err != nil {
|
||||
return dev
|
||||
}
|
||||
return filepath.Base(filepath.Dir(link))
|
||||
}
|
||||
|
||||
// describeContent summarises what is currently on a disk for the confirmation
|
||||
// screen. Best-effort: an unreadable disk is reported as unknown, not empty,
|
||||
// because "empty" invites an operator to erase something that is not.
|
||||
func describeContent(dev string) string {
|
||||
out, err := exec.Command("blkid", "-p", "-o", "value", "-s", "TYPE", dev).Output()
|
||||
if err == nil {
|
||||
if t := strings.TrimSpace(string(out)); t != "" {
|
||||
return t + " filesystem"
|
||||
}
|
||||
}
|
||||
// No filesystem on the whole device: report the partition table instead,
|
||||
// since that is what tells the operator the disk is in use.
|
||||
out, err = exec.Command("blkid", "-p", "-o", "value", "-s", "PTTYPE", dev).Output()
|
||||
if err != nil {
|
||||
return "unknown"
|
||||
}
|
||||
if t := strings.TrimSpace(string(out)); t != "" {
|
||||
return t + " partition table"
|
||||
}
|
||||
return "empty"
|
||||
}
|
||||
|
||||
func sysfsString(dev, file string) string {
|
||||
data, err := os.ReadFile(filepath.Join(sysBlockDir, dev, file))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(data))
|
||||
}
|
||||
|
||||
func sysfsInt(dev, file string) int64 {
|
||||
n, err := strconv.ParseInt(sysfsString(dev, file), 10, 64)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func sysfsIntDefault(dev, file string, def int) int {
|
||||
if n := sysfsInt(dev, file); n > 0 {
|
||||
return int(n)
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func hasAnyPrefix(s string, prefixes []string) bool {
|
||||
return slices.ContainsFunc(prefixes, func(p string) bool { return strings.HasPrefix(s, p) })
|
||||
}
|
||||
@@ -1,484 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeDisk describes one device to lay down in the fixture sysfs tree.
|
||||
type fakeDisk struct {
|
||||
name string
|
||||
sectors int64
|
||||
model string
|
||||
logical int
|
||||
physical int
|
||||
rotational bool
|
||||
removable bool
|
||||
partitions []string
|
||||
}
|
||||
|
||||
// fakeSysfs builds a /sys/block, /sys/class/block, /dev/disk/by-id and
|
||||
// /proc/mounts fixture and points the scan at it for the duration of the test.
|
||||
type fakeSysfs struct {
|
||||
root string
|
||||
t *testing.T
|
||||
}
|
||||
|
||||
func newFakeSysfs(t *testing.T, disks ...fakeDisk) *fakeSysfs {
|
||||
t.Helper()
|
||||
f := &fakeSysfs{root: t.TempDir(), t: t}
|
||||
|
||||
swap := func(dst *string, v string) {
|
||||
prev := *dst
|
||||
*dst = v
|
||||
t.Cleanup(func() { *dst = prev })
|
||||
}
|
||||
swap(&sysBlockDir, filepath.Join(f.root, "sys/block"))
|
||||
swap(&sysClassBlockDir, filepath.Join(f.root, "sys/class/block"))
|
||||
swap(&devByIDDir, filepath.Join(f.root, "dev/disk/by-id"))
|
||||
swap(&procMountsPath, filepath.Join(f.root, "proc/mounts"))
|
||||
swap(&procSwapsPath, filepath.Join(f.root, "proc/swaps"))
|
||||
|
||||
f.mkdirAll(sysBlockDir)
|
||||
f.mkdirAll(sysClassBlockDir)
|
||||
for _, d := range disks {
|
||||
f.addDisk(d)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func (f *fakeSysfs) mkdirAll(dir string) {
|
||||
f.t.Helper()
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
f.t.Fatalf("mkdir %s: %v", dir, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fakeSysfs) write(path, content string) {
|
||||
f.t.Helper()
|
||||
f.mkdirAll(filepath.Dir(path))
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
f.t.Fatalf("write %s: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
// addDisk lays out one device the way the kernel does: the attribute files
|
||||
// under /sys/block/<dev>, and a /sys/class/block symlink per disk and
|
||||
// partition, since parentDisk walks the latter.
|
||||
func (f *fakeSysfs) addDisk(d fakeDisk) {
|
||||
f.t.Helper()
|
||||
dir := filepath.Join(sysBlockDir, d.name)
|
||||
f.write(filepath.Join(dir, "size"), strconv.FormatInt(d.sectors, 10)+"\n")
|
||||
if d.model != "" {
|
||||
f.write(filepath.Join(dir, "device/model"), d.model+"\n")
|
||||
}
|
||||
if d.logical > 0 {
|
||||
f.write(filepath.Join(dir, "queue/logical_block_size"), strconv.Itoa(d.logical)+"\n")
|
||||
}
|
||||
if d.physical > 0 {
|
||||
f.write(filepath.Join(dir, "queue/physical_block_size"), strconv.Itoa(d.physical)+"\n")
|
||||
}
|
||||
f.write(filepath.Join(dir, "queue/rotational"), boolAttr(d.rotational))
|
||||
f.write(filepath.Join(dir, "removable"), boolAttr(d.removable))
|
||||
|
||||
f.symlink(dir, filepath.Join(sysClassBlockDir, d.name))
|
||||
for _, part := range d.partitions {
|
||||
partDir := filepath.Join(dir, part)
|
||||
f.write(filepath.Join(partDir, "partition"), "1\n")
|
||||
f.symlink(partDir, filepath.Join(sysClassBlockDir, part))
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fakeSysfs) symlink(target, link string) {
|
||||
f.t.Helper()
|
||||
f.mkdirAll(filepath.Dir(link))
|
||||
if err := os.Symlink(target, link); err != nil {
|
||||
f.t.Fatalf("symlink %s -> %s: %v", link, target, err)
|
||||
}
|
||||
}
|
||||
|
||||
// addByID creates a by-id alias pointing at a kernel device name. The link
|
||||
// target must exist for EvalSymlinks, so a placeholder /dev tree is used.
|
||||
func (f *fakeSysfs) addByID(id, dev string) {
|
||||
f.t.Helper()
|
||||
devPath := filepath.Join(f.root, "dev", dev)
|
||||
f.write(devPath, "")
|
||||
f.symlink(devPath, filepath.Join(devByIDDir, id))
|
||||
}
|
||||
|
||||
func (f *fakeSysfs) setMounts(content string) {
|
||||
f.t.Helper()
|
||||
f.write(procMountsPath, content)
|
||||
}
|
||||
|
||||
func (f *fakeSysfs) setSwaps(content string) {
|
||||
f.t.Helper()
|
||||
f.write(procSwapsPath, content)
|
||||
}
|
||||
|
||||
// addHolder marks a partition as claimed by md, LVM or device-mapper — the
|
||||
// state that makes the kernel refuse to re-read the disk's partition table.
|
||||
func (f *fakeSysfs) addHolder(disk, part, holder string) {
|
||||
f.t.Helper()
|
||||
f.mkdirAll(filepath.Join(sysBlockDir, disk, part, "holders", holder))
|
||||
}
|
||||
|
||||
func boolAttr(b bool) string {
|
||||
if b {
|
||||
return "1\n"
|
||||
}
|
||||
return "0\n"
|
||||
}
|
||||
|
||||
func TestListDisksReadsSysfsAttributes(t *testing.T) {
|
||||
f := newFakeSysfs(t, fakeDisk{
|
||||
name: "sdb", sectors: 2 << 20, model: "SAMSUNG MZ7", logical: 512, physical: 4096, rotational: true,
|
||||
})
|
||||
f.setMounts("")
|
||||
|
||||
disks, err := ListDisks()
|
||||
if err != nil {
|
||||
t.Fatalf("ListDisks: %v", err)
|
||||
}
|
||||
if len(disks) != 1 {
|
||||
t.Fatalf("got %d disks, want 1: %+v", len(disks), disks)
|
||||
}
|
||||
d := disks[0]
|
||||
if d.Path != "/dev/sdb" || d.Bytes != (2<<20)*512 {
|
||||
t.Errorf("path/size = %s/%d", d.Path, d.Bytes)
|
||||
}
|
||||
if d.Model != "SAMSUNG MZ7" {
|
||||
t.Errorf("Model = %q", d.Model)
|
||||
}
|
||||
if d.LogicalBlockSize != 512 || d.PhysicalBlockSize != 4096 {
|
||||
t.Errorf("block sizes = %d/%d", d.LogicalBlockSize, d.PhysicalBlockSize)
|
||||
}
|
||||
if !d.Rotational || d.Removable || d.LiveMedia {
|
||||
t.Errorf("flags = rot %v, rem %v, live %v", d.Rotational, d.Removable, d.LiveMedia)
|
||||
}
|
||||
if d.Content == "" {
|
||||
t.Error("Content is empty; an unreadable device must report something")
|
||||
}
|
||||
}
|
||||
|
||||
func TestListDisksSkipsVirtualAndEmptyDevices(t *testing.T) {
|
||||
f := newFakeSysfs(t,
|
||||
fakeDisk{name: "sda", sectors: 1 << 20},
|
||||
fakeDisk{name: "loop0", sectors: 1 << 20},
|
||||
fakeDisk{name: "dm-0", sectors: 1 << 20},
|
||||
fakeDisk{name: "zram0", sectors: 1 << 20},
|
||||
fakeDisk{name: "sr0", sectors: 1 << 20},
|
||||
fakeDisk{name: "nbd0", sectors: 1 << 20},
|
||||
// An empty card-reader slot reports zero sectors.
|
||||
fakeDisk{name: "mmcblk0", sectors: 0},
|
||||
)
|
||||
f.setMounts("")
|
||||
|
||||
disks, err := ListDisks()
|
||||
if err != nil {
|
||||
t.Fatalf("ListDisks: %v", err)
|
||||
}
|
||||
var paths []string
|
||||
for _, d := range disks {
|
||||
paths = append(paths, d.Path)
|
||||
}
|
||||
if !slices.Equal(paths, []string{"/dev/sda"}) {
|
||||
t.Errorf("paths = %v, want only /dev/sda", paths)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListDisksSortsByPath(t *testing.T) {
|
||||
f := newFakeSysfs(t,
|
||||
fakeDisk{name: "sdc", sectors: 1 << 20},
|
||||
fakeDisk{name: "sda", sectors: 1 << 20},
|
||||
fakeDisk{name: "sdb", sectors: 1 << 20},
|
||||
)
|
||||
f.setMounts("")
|
||||
|
||||
disks, err := ListDisks()
|
||||
if err != nil {
|
||||
t.Fatalf("ListDisks: %v", err)
|
||||
}
|
||||
var paths []string
|
||||
for _, d := range disks {
|
||||
paths = append(paths, d.Path)
|
||||
}
|
||||
if !slices.Equal(paths, []string{"/dev/sda", "/dev/sdb", "/dev/sdc"}) {
|
||||
t.Errorf("paths = %v, want sorted", paths)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListDisksDefaultsBlockSizesWhenSysfsIsSilent(t *testing.T) {
|
||||
f := newFakeSysfs(t, fakeDisk{name: "vda", sectors: 1 << 20})
|
||||
f.setMounts("")
|
||||
|
||||
disks, err := ListDisks()
|
||||
if err != nil {
|
||||
t.Fatalf("ListDisks: %v", err)
|
||||
}
|
||||
if disks[0].LogicalBlockSize != 512 || disks[0].PhysicalBlockSize != 512 {
|
||||
t.Errorf("block sizes = %d/%d, want the 512 default",
|
||||
disks[0].LogicalBlockSize, disks[0].PhysicalBlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListDisksErrorsWhenSysfsIsUnreadable(t *testing.T) {
|
||||
newFakeSysfs(t)
|
||||
sysBlockDir = filepath.Join(t.TempDir(), "does-not-exist")
|
||||
|
||||
if _, err := ListDisks(); err == nil {
|
||||
t.Fatal("expected an error when /sys/block cannot be read")
|
||||
}
|
||||
}
|
||||
|
||||
func TestListDisksMarksTheLiveMedium(t *testing.T) {
|
||||
f := newFakeSysfs(t,
|
||||
fakeDisk{name: "sda", sectors: 1 << 20},
|
||||
fakeDisk{name: "sdz", sectors: 1 << 20, removable: true, partitions: []string{"sdz1"}},
|
||||
)
|
||||
// The installer boots from a partition; the whole disk must be excluded.
|
||||
f.setMounts("/dev/sdz1 /run/live/medium iso9660 ro 0 0\n" +
|
||||
"proc /proc proc rw 0 0\n" +
|
||||
"/dev/sda1 / ext4 rw 0 0\n")
|
||||
|
||||
disks, err := ListDisks()
|
||||
if err != nil {
|
||||
t.Fatalf("ListDisks: %v", err)
|
||||
}
|
||||
byPath := map[string]Disk{}
|
||||
for _, d := range disks {
|
||||
byPath[d.Path] = d
|
||||
}
|
||||
if !byPath["/dev/sdz"].LiveMedia {
|
||||
t.Error("/dev/sdz backs the live medium and was not marked")
|
||||
}
|
||||
if byPath["/dev/sda"].LiveMedia {
|
||||
t.Error("/dev/sda is not the live medium but was marked")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLiveMediaDevicesIsEmptyWithoutProcMounts(t *testing.T) {
|
||||
newFakeSysfs(t)
|
||||
if got := liveMediaDevices(); len(got) != 0 {
|
||||
t.Errorf("liveMediaDevices = %v, want empty when /proc/mounts is missing", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParentDiskLeavesWholeDisksAlone(t *testing.T) {
|
||||
newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 20, partitions: []string{"sda1"}})
|
||||
|
||||
if got := parentDisk("sda"); got != "sda" {
|
||||
t.Errorf("parentDisk(sda) = %q, want sda", got)
|
||||
}
|
||||
if got := parentDisk("sda1"); got != "sda" {
|
||||
t.Errorf("parentDisk(sda1) = %q, want sda", got)
|
||||
}
|
||||
// An unknown name cannot be resolved and is returned unchanged.
|
||||
if got := parentDisk("sdq9"); got != "sdq9" {
|
||||
t.Errorf("parentDisk(sdq9) = %q, want it unchanged", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildByIDIndexPrefersTheMostLegibleAlias(t *testing.T) {
|
||||
f := newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 20})
|
||||
// wwn- is stable but opaque, so ata- must win regardless of scan order.
|
||||
f.addByID("wwn-0x5000c500", "sda")
|
||||
f.addByID("ata-SAMSUNG_MZ7_S1", "sda")
|
||||
f.addByID("ata-SAMSUNG_MZ7_S1-part1", "sda")
|
||||
|
||||
idx := buildByIDIndex()
|
||||
want := filepath.Join(devByIDDir, "ata-SAMSUNG_MZ7_S1")
|
||||
if idx["sda"] != want {
|
||||
t.Errorf("by-id[sda] = %q, want %q", idx["sda"], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildByIDIndexIsEmptyWhenTheDirIsAbsent(t *testing.T) {
|
||||
newFakeSysfs(t)
|
||||
if got := buildByIDIndex(); len(got) != 0 {
|
||||
t.Errorf("buildByIDIndex = %v, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListDisksAttachesByIDPath(t *testing.T) {
|
||||
f := newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 20})
|
||||
f.addByID("ata-TEST_SERIAL0", "sda")
|
||||
f.setMounts("")
|
||||
|
||||
disks, err := ListDisks()
|
||||
if err != nil {
|
||||
t.Fatalf("ListDisks: %v", err)
|
||||
}
|
||||
want := filepath.Join(devByIDDir, "ata-TEST_SERIAL0")
|
||||
if disks[0].ByID != want {
|
||||
t.Errorf("ByID = %q, want %q", disks[0].ByID, want)
|
||||
}
|
||||
if disks[0].Stable() != want {
|
||||
t.Errorf("Stable() = %q, want the by-id path", disks[0].Stable())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrefixRank(t *testing.T) {
|
||||
if prefixRank("nvme-Foo") >= prefixRank("wwn-0x1") {
|
||||
t.Error("nvme- must outrank wwn-")
|
||||
}
|
||||
if prefixRank("md-uuid-x") != len(byIDPreference) {
|
||||
t.Error("an unknown namespace must rank last")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSizeHuman(t *testing.T) {
|
||||
tests := []struct {
|
||||
bytes int64
|
||||
want string
|
||||
}{
|
||||
{2 << 40, "2.0T"},
|
||||
{500 << 30, "500.0G"},
|
||||
{64 << 20, "64.0M"},
|
||||
{512, "512B"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := (Disk{Bytes: tt.bytes}).SizeHuman(); got != tt.want {
|
||||
t.Errorf("SizeHuman(%d) = %q, want %q", tt.bytes, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStableFallsBackToKernelName(t *testing.T) {
|
||||
d := Disk{Path: "/dev/sda"}
|
||||
if d.Stable() != "/dev/sda" {
|
||||
t.Errorf("Stable() = %q", d.Stable())
|
||||
}
|
||||
if d.StablePartitionPath(2) != "/dev/sda2" {
|
||||
t.Errorf("StablePartitionPath(2) = %q", d.StablePartitionPath(2))
|
||||
}
|
||||
}
|
||||
|
||||
func TestHasAnyPrefix(t *testing.T) {
|
||||
if !hasAnyPrefix("loop3", skipPrefixes) {
|
||||
t.Error("loop3 should match")
|
||||
}
|
||||
if hasAnyPrefix("sda", skipPrefixes) {
|
||||
t.Error("sda should not match")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadKernelPartitionsListsOnlyPartitions(t *testing.T) {
|
||||
newFakeSysfs(t, fakeDisk{
|
||||
name: "nvme0n1", sectors: 1 << 22,
|
||||
partitions: []string{"nvme0n1p3", "nvme0n1p1", "nvme0n1p2"},
|
||||
})
|
||||
|
||||
got, err := readKernelPartitions("/dev/nvme0n1")
|
||||
if err != nil {
|
||||
t.Fatalf("readKernelPartitions: %v", err)
|
||||
}
|
||||
// Sorted, and "queue" and "device" alongside them are not partitions.
|
||||
want := []string{"nvme0n1p1", "nvme0n1p2", "nvme0n1p3"}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadKernelPartitionsIsEmptyForAnUnpartitionedDisk(t *testing.T) {
|
||||
newFakeSysfs(t, fakeDisk{name: "nvme2n1", sectors: 1 << 22})
|
||||
|
||||
got, err := readKernelPartitions("/dev/nvme2n1")
|
||||
if err != nil {
|
||||
t.Fatalf("readKernelPartitions: %v", err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Errorf("got %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadKernelPartitionsFailsOnAnUnknownDisk(t *testing.T) {
|
||||
newFakeSysfs(t)
|
||||
|
||||
if _, err := readKernelPartitions("/dev/nvme9n1"); err == nil {
|
||||
t.Fatal("expected an error for a disk with no sysfs entry")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiskHoldersNamesEveryClaimant(t *testing.T) {
|
||||
f := newFakeSysfs(t, fakeDisk{
|
||||
name: "sda", sectors: 1 << 22, partitions: []string{"sda1", "sda2"},
|
||||
})
|
||||
f.addHolder("sda", "sda1", "md0")
|
||||
f.addHolder("sda", "sda2", "dm-0")
|
||||
|
||||
if got, want := diskHolders("/dev/sda"), []string{"dm-0", "md0"}; !slices.Equal(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiskHoldersIsEmptyWhenNothingClaimsTheDisk(t *testing.T) {
|
||||
newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1"}})
|
||||
|
||||
if got := diskHolders("/dev/sda"); len(got) != 0 {
|
||||
t.Errorf("got %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMountsOnDiskFindsOnlyThisDiskDeepestFirst(t *testing.T) {
|
||||
f := newFakeSysfs(t,
|
||||
fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1", "sda2"}},
|
||||
fakeDisk{name: "sdb", sectors: 1 << 22, partitions: []string{"sdb1"}},
|
||||
)
|
||||
f.setMounts(strings.Join([]string{
|
||||
"/dev/sda1 /oldroot ext4 rw 0 0",
|
||||
"/dev/sda2 /oldroot/boot/efi vfat rw 0 0",
|
||||
"/dev/sdb1 /other ext4 rw 0 0",
|
||||
"proc /proc proc rw 0 0",
|
||||
}, "\n") + "\n")
|
||||
|
||||
got := mountsOnDisk("/dev/sda")
|
||||
want := []diskMount{
|
||||
{Device: "/dev/sda2", Mountpoint: "/oldroot/boot/efi"},
|
||||
{Device: "/dev/sda1", Mountpoint: "/oldroot"},
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// /dev/nvme0n1 must not claim /dev/nvme0n11, which a string-prefix match would.
|
||||
func TestMountsOnDiskDoesNotMatchASiblingNamespace(t *testing.T) {
|
||||
f := newFakeSysfs(t,
|
||||
fakeDisk{name: "nvme0n1", sectors: 1 << 22, partitions: []string{"nvme0n1p1"}},
|
||||
fakeDisk{name: "nvme0n11", sectors: 1 << 22, partitions: []string{"nvme0n11p1"}},
|
||||
)
|
||||
f.setMounts("/dev/nvme0n11p1 /elsewhere ext4 rw 0 0\n")
|
||||
|
||||
if got := mountsOnDisk("/dev/nvme0n1"); len(got) != 0 {
|
||||
t.Errorf("got %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSwapsOnDiskSkipsTheHeaderAndOtherDisks(t *testing.T) {
|
||||
f := newFakeSysfs(t,
|
||||
fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1"}},
|
||||
fakeDisk{name: "sdb", sectors: 1 << 22, partitions: []string{"sdb1"}},
|
||||
)
|
||||
f.setSwaps(strings.Join([]string{
|
||||
"Filename\t\t\t\tType\t\tSize\tUsed\tPriority",
|
||||
"/dev/sda1 partition 8388604 0 -2",
|
||||
"/dev/sdb1 partition 8388604 0 -3",
|
||||
}, "\n") + "\n")
|
||||
|
||||
if got, want := swapsOnDisk("/dev/sda"), []string{"/dev/sda1"}; !slices.Equal(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSwapsOnDiskIsEmptyWithoutASwapsTable(t *testing.T) {
|
||||
newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1"}})
|
||||
|
||||
if got := swapsOnDisk("/dev/sda"); len(got) != 0 {
|
||||
t.Errorf("got %v, want none", got)
|
||||
}
|
||||
}
|
||||
@@ -1,202 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
//go:embed assets/spinifex-boot-tool
|
||||
var bootToolScript []byte
|
||||
|
||||
//go:embed assets/grub-install-wrapper
|
||||
var grubInstallWrapper []byte
|
||||
|
||||
const (
|
||||
bootToolPath = "/usr/sbin/spinifex-boot-tool"
|
||||
grubInstallPath = "/usr/sbin/grub-install"
|
||||
grubInstallReal = "/usr/sbin/grub-install.real"
|
||||
bootToolUUIDList = "/etc/kernel/spinifex-boot-uuids"
|
||||
)
|
||||
|
||||
// installBootToolZFS sets up per-disk bootloaders on a ZFS root system.
|
||||
//
|
||||
// Nothing in the boot path is allowed to read the pool: GRUB's ZFS support
|
||||
// trails OpenZFS, so a pool feature flag enabled years from now could make the
|
||||
// machine unbootable. Each disk instead carries a self-contained ESP with its
|
||||
// own GRUB, kernels and initramfs.
|
||||
func installBootToolZFS(cfg DiskConfig) error {
|
||||
if err := writeTargetFile(bootToolPath, bootToolScript, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := writeGrubDefaults(cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := bindChrootMounts(); err != nil {
|
||||
return err
|
||||
}
|
||||
defer unbindChrootMounts()
|
||||
|
||||
// Diverted before the first init: init runs a refresh, and refresh calls
|
||||
// grub-install. With the wrapper already in place that call has to reach
|
||||
// the real binary, which is what the diversion arranges.
|
||||
if err := divertGrubInstall(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// A partially-updated set of ESPs is an install that boots today and fails
|
||||
// after the next kernel upgrade, so every disk must succeed here.
|
||||
env := []string{"SPINIFEX_BOOT_TOOL_STRICT=1"}
|
||||
for _, d := range cfg.Disks {
|
||||
esp := d.PartitionPath(espPartNum)
|
||||
if err := runEnv(env, "chroot", mountRoot, bootToolPath, "init", esp); err != nil {
|
||||
return fmt.Errorf("spinifex-boot-tool init %s: %w", esp, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The one moment firmware boot variables are written. Refreshes run on
|
||||
// every kernel upgrade and deliberately leave NVRAM alone: repeatedly
|
||||
// rewriting boot entries has bricked more than one vendor's boards. Losing
|
||||
// the entry is survivable anyway, because each ESP also carries the
|
||||
// removable-path loader that firmware falls back to.
|
||||
if err := runEnv(append(env, "SPINIFEX_BOOT_TOOL_NVRAM=1"),
|
||||
"chroot", mountRoot, bootToolPath, "refresh"); err != nil {
|
||||
return fmt.Errorf("spinifex-boot-tool refresh: %w", err)
|
||||
}
|
||||
return verifyBootConfig(cfg)
|
||||
}
|
||||
|
||||
// divertGrubInstall moves the real grub-install aside and puts the wrapper in
|
||||
// its place. dpkg-divert makes the move survive package upgrades, which is the
|
||||
// point: grub-pc's postinst runs grub-install and would otherwise quietly
|
||||
// replace a working boot layout with one that reads the pool.
|
||||
func divertGrubInstall() error {
|
||||
if err := run("chroot", mountRoot, "dpkg-divert", "--divert", grubInstallReal,
|
||||
"--rename", "--add", grubInstallPath); err != nil {
|
||||
return fmt.Errorf("divert grub-install: %w", err)
|
||||
}
|
||||
return writeTargetFile(grubInstallPath, grubInstallWrapper, 0o755)
|
||||
}
|
||||
|
||||
// verifyBootConfig asserts that the config the machine will actually boot from
|
||||
// names the ZFS root. grub-mkconfig degrades quietly when grub-probe cannot
|
||||
// read the pool, emitting a config that gets as far as the GRUB menu and then
|
||||
// panics with "unable to mount root fs" — which looks like a hardware fault.
|
||||
func verifyBootConfig(cfg DiskConfig) error {
|
||||
uuids, err := os.ReadFile(filepath.Join(mountRoot, strings.TrimPrefix(bootToolUUIDList, "/")))
|
||||
if err != nil {
|
||||
return fmt.Errorf("read ESP list: %w", err)
|
||||
}
|
||||
if len(strings.Fields(string(uuids))) != len(cfg.Disks) {
|
||||
return fmt.Errorf("expected %d registered ESPs, found %d — some disks have no bootloader",
|
||||
len(cfg.Disks), len(strings.Fields(string(uuids))))
|
||||
}
|
||||
|
||||
// One ESP is enough to prove grub-mkconfig produced the right thing; they
|
||||
// are all generated by the same code path in the same run.
|
||||
mp := filepath.Join(mountRoot, "var/tmp/verify-esp")
|
||||
if err := os.MkdirAll(mp, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = os.Remove(mp) }()
|
||||
|
||||
esp := cfg.Disks[0].PartitionPath(espPartNum)
|
||||
if err := run("mount", esp, mp); err != nil {
|
||||
return fmt.Errorf("mount %s for verification: %w", esp, err)
|
||||
}
|
||||
defer func() { _ = run("umount", mp) }()
|
||||
|
||||
grubCfg, err := os.ReadFile(filepath.Join(mp, "grub/grub.cfg"))
|
||||
if err != nil {
|
||||
return fmt.Errorf("read grub.cfg from %s: %w", esp, err)
|
||||
}
|
||||
want := "root=ZFS=" + ZFSRootDataset
|
||||
if !strings.Contains(string(grubCfg), want) {
|
||||
return fmt.Errorf("grub.cfg on %s does not contain %q — the installed system would not find its root filesystem", esp, want)
|
||||
}
|
||||
if !strings.Contains(string(grubCfg), "vmlinuz-") {
|
||||
return fmt.Errorf("grub.cfg on %s references no kernel", esp)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeTargetFile writes a file into the install target.
|
||||
func writeTargetFile(path string, content []byte, mode os.FileMode) error {
|
||||
full := filepath.Join(mountRoot, strings.TrimPrefix(path, "/"))
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(full, content, mode); err != nil {
|
||||
return fmt.Errorf("write %s: %w", path, err)
|
||||
}
|
||||
// WriteFile does not apply the mode to a file that already exists, and this
|
||||
// path is re-run on a retried install.
|
||||
return os.Chmod(full, mode)
|
||||
}
|
||||
|
||||
// writeGrubDefaults writes /etc/default/grub and the branding snippet.
|
||||
//
|
||||
// Both consoles are listed: tty0 stays primary for a local install while ttyS0
|
||||
// mirrors kernel output for headless racks and CI. The last console= on the
|
||||
// cmdline becomes the system console, so serial-only hardware still works.
|
||||
func writeGrubDefaults(cfg DiskConfig) error {
|
||||
cmdline := "console=tty0 console=ttyS0,115200 systemd.show_status=1"
|
||||
if cfg.FS.IsZFS() {
|
||||
// Stated explicitly rather than relying on grub-mkconfig's probe. It
|
||||
// appears after the root= that 10_linux computes, so it wins if the
|
||||
// probe came back with something wrong.
|
||||
cmdline = fmt.Sprintf("root=ZFS=%s boot=zfs %s", ZFSRootDataset, cmdline)
|
||||
}
|
||||
grubDefault := fmt.Sprintf(`GRUB_DEFAULT=0
|
||||
GRUB_TIMEOUT=5
|
||||
GRUB_DISTRIBUTOR=Spinifex
|
||||
GRUB_CMDLINE_LINUX_DEFAULT=""
|
||||
GRUB_CMDLINE_LINUX="%s"
|
||||
`, cmdline)
|
||||
if err := os.WriteFile(filepath.Join(mountRoot, "etc/default/grub"), []byte(grubDefault), 0o644); err != nil {
|
||||
return fmt.Errorf("write /etc/default/grub: %w", err)
|
||||
}
|
||||
|
||||
// gfxterm must be activated before serial is appended — background_image
|
||||
// silently does nothing in text mode. $prefix is used rather than a literal
|
||||
// /boot/grub so the same snippet is correct whether GRUB's root is the ext4
|
||||
// filesystem or a per-disk ESP.
|
||||
grubTheme := `#!/bin/sh
|
||||
exec tail -n +3 $0
|
||||
insmod all_video
|
||||
insmod font
|
||||
if loadfont $prefix/fonts/unicode.pf2; then
|
||||
set gfxmode=auto
|
||||
insmod gfxterm
|
||||
terminal_output gfxterm
|
||||
fi
|
||||
insmod serial
|
||||
if serial --unit=0 --speed=115200 --timeout=1; then
|
||||
terminal_input --append serial
|
||||
terminal_output --append serial
|
||||
fi
|
||||
|
||||
# --- Branding ---
|
||||
insmod png
|
||||
set theme=$prefix/theme.txt
|
||||
export theme
|
||||
`
|
||||
if err := os.WriteFile(filepath.Join(mountRoot, "etc/grub.d/06_spinifex"), []byte(grubTheme), 0o755); err != nil {
|
||||
return fmt.Errorf("write /etc/grub.d/06_spinifex: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// refreshBootToolNote logs the command an operator needs after replacing a
|
||||
// disk. Nothing in the installer can do this for them later.
|
||||
func refreshBootToolNote(cfg DiskConfig) {
|
||||
if cfg.FS.IsZFS() {
|
||||
slog.Info("bootloader installed on every pool member",
|
||||
"disks", len(cfg.Disks),
|
||||
"after_disk_replacement", "spinifex-boot-tool format <esp> && spinifex-boot-tool init <esp>")
|
||||
}
|
||||
}
|
||||
@@ -1,235 +0,0 @@
|
||||
// Package install performs the disk installation steps after the UI has
|
||||
// collected configuration.
|
||||
package install
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Plane names the three traffic planes a node presents. Each plane is a role
|
||||
// bound to a target, not a fixed NIC: the same three roles collapse onto one,
|
||||
// two or three interfaces depending on the hardware.
|
||||
type Plane string
|
||||
|
||||
const (
|
||||
// PlaneWAN carries public north-south traffic: ingress/egress, EIPs and
|
||||
// SNAT/DNAT. It is the OVN external bridge-mapping.
|
||||
PlaneWAN Plane = "wan"
|
||||
|
||||
// PlaneLAN carries internal cluster traffic: predastore replication, the
|
||||
// NATS cluster mesh, OVN NB/SB control, Raft and S3DB.
|
||||
PlaneLAN Plane = "lan"
|
||||
|
||||
// PlaneVPC carries the tenant overlay: OVN Geneve tunnels between nodes.
|
||||
// It is the ovn-encap-ip endpoint.
|
||||
PlaneVPC Plane = "vpc"
|
||||
)
|
||||
|
||||
// Bridge returns the Linux bridge that carries this plane.
|
||||
func (p Plane) Bridge() string { return "br-" + string(p) }
|
||||
|
||||
// NetworkRole binds one plane to a network target. An empty Interface means
|
||||
// the role is folded onto the next role up the chain (vpc <- lan <- wan) and
|
||||
// no bridge of its own is created.
|
||||
//
|
||||
// VLAN is what allows two roles to share one physical NIC while staying
|
||||
// separate planes: each gets its own NIC.<vlan> subinterface and bridge. Two
|
||||
// roles on the same Interface with no VLAN are the same plane, not two.
|
||||
type NetworkRole struct {
|
||||
// Interface is the physical NIC this role binds to; empty means folded.
|
||||
Interface string
|
||||
|
||||
// VLAN is the 802.1Q tag, or 0 for untagged.
|
||||
VLAN int
|
||||
|
||||
// DHCPMode leaves addressing to the DHCP client; Address/Mask are then unset.
|
||||
DHCPMode bool
|
||||
|
||||
Address string
|
||||
Mask string
|
||||
|
||||
// Gateway is only meaningful on the wan role — the other planes are
|
||||
// link-local to the rack and must not install a default route.
|
||||
Gateway string
|
||||
|
||||
DNS []string
|
||||
|
||||
// MTU is written as MTUBytes= on the link; 0 leaves the kernel default.
|
||||
// East-west planes typically run 9000, wan stays at 1500.
|
||||
MTU int
|
||||
|
||||
// WiFiSSID and WiFiPass are set when the selected NIC is wireless.
|
||||
WiFiSSID string
|
||||
WiFiPass string
|
||||
}
|
||||
|
||||
// Folded reports whether this role has no interface of its own and inherits
|
||||
// the plane of the role above it.
|
||||
func (r NetworkRole) Folded() bool { return r.Interface == "" }
|
||||
|
||||
// Link returns the name of the interface carrying this role's address: the
|
||||
// physical NIC, or its VLAN subinterface when tagged. Empty when folded.
|
||||
func (r NetworkRole) Link() string {
|
||||
if r.Folded() {
|
||||
return ""
|
||||
}
|
||||
if r.VLAN > 0 {
|
||||
return fmt.Sprintf("%s.%d", r.Interface, r.VLAN)
|
||||
}
|
||||
return r.Interface
|
||||
}
|
||||
|
||||
// Config holds all values collected by the installer UI.
|
||||
type Config struct {
|
||||
// Storage is the filesystem choice and the ordered set of disks it is
|
||||
// built from. In ext4 mode it holds exactly one disk.
|
||||
Storage DiskConfig
|
||||
|
||||
// WAN is always present and can never be folded — a node needs an uplink.
|
||||
// The address lives on br-wan (a Linux bridge over the NIC), not on the
|
||||
// physical NIC itself, so OVN/OVS can attach to the bridge without
|
||||
// disrupting host connectivity.
|
||||
WAN NetworkRole
|
||||
|
||||
// LAN carries internal cluster traffic. Folded onto WAN on single-NIC nodes.
|
||||
LAN NetworkRole
|
||||
|
||||
// VPC carries Geneve overlay traffic and supplies ovn-encap-ip. Folded onto
|
||||
// LAN unless a node dedicates an interface or VLAN to it.
|
||||
VPC NetworkRole
|
||||
|
||||
// Node identity
|
||||
Hostname string
|
||||
|
||||
// SkipFormation skips spx admin init/join in firstboot; the provisioning
|
||||
// controller (e.g. bm-bootstrap.sh) owns cluster formation instead.
|
||||
SkipFormation bool
|
||||
|
||||
// CA certificate (PEM), optional.
|
||||
HasCACert bool
|
||||
CACert string
|
||||
|
||||
// RootPassword is the password to set for the root account on the installed system.
|
||||
RootPassword string
|
||||
|
||||
// Email is the operator's email address, used by the call-home telemetry
|
||||
// endpoint to notify of important system updates and security advisories.
|
||||
// Required on interactive installs; may be empty on headless/CI installs
|
||||
// when SPINIFEX_EMAIL is not supplied on the kernel cmdline.
|
||||
Email string
|
||||
|
||||
// GPUPassthrough enables VFIO GPU passthrough on this node.
|
||||
// Set via SPINIFEX_GPU_PASSTHROUGH=1 on headless installs.
|
||||
// Passes --gpu-passthrough to `spx admin init` during firstboot.
|
||||
GPUPassthrough bool
|
||||
}
|
||||
|
||||
// Resolve returns the role that actually carries the given plane after
|
||||
// collapsing folded roles up the chain vpc <- lan <- wan, along with the plane
|
||||
// it landed on. A folded vpc on a single-NIC node resolves to the wan role.
|
||||
func (c *Config) Resolve(p Plane) (NetworkRole, Plane) {
|
||||
switch p {
|
||||
case PlaneVPC:
|
||||
if !c.VPC.Folded() {
|
||||
return c.VPC, PlaneVPC
|
||||
}
|
||||
return c.Resolve(PlaneLAN)
|
||||
case PlaneLAN:
|
||||
if !c.LAN.Folded() {
|
||||
return c.LAN, PlaneLAN
|
||||
}
|
||||
return c.WAN, PlaneWAN
|
||||
default:
|
||||
return c.WAN, PlaneWAN
|
||||
}
|
||||
}
|
||||
|
||||
// PlaneAddress returns the configured address for a plane after collapsing.
|
||||
// Empty when the resolved role uses DHCP, since no static address is known
|
||||
// until the lease is taken.
|
||||
func (c *Config) PlaneAddress(p Plane) string {
|
||||
role, _ := c.Resolve(p)
|
||||
if role.DHCPMode {
|
||||
return ""
|
||||
}
|
||||
return role.Address
|
||||
}
|
||||
|
||||
// PlaneBridge returns the bridge carrying a plane after collapsing, so callers
|
||||
// asking for the vpc bridge on a single-NIC node correctly get br-wan.
|
||||
func (c *Config) PlaneBridge(p Plane) string {
|
||||
_, landed := c.Resolve(p)
|
||||
return landed.Bridge()
|
||||
}
|
||||
|
||||
// RoleBinding pairs a plane with the role that carries it.
|
||||
type RoleBinding struct {
|
||||
Plane Plane
|
||||
Role NetworkRole
|
||||
}
|
||||
|
||||
// Roles returns the bound roles in bridge-creation order, skipping folded ones.
|
||||
// wan comes first because br-lan and br-vpc are activated manually after it.
|
||||
func (c *Config) Roles() []RoleBinding {
|
||||
out := []RoleBinding{{PlaneWAN, c.WAN}}
|
||||
if !c.LAN.Folded() {
|
||||
out = append(out, RoleBinding{PlaneLAN, c.LAN})
|
||||
}
|
||||
if !c.VPC.Folded() {
|
||||
out = append(out, RoleBinding{PlaneVPC, c.VPC})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Validate enforces the sharing rules the UI and the headless path must both
|
||||
// obey: wan always needs a target, and two roles may only share an interface
|
||||
// when distinct VLAN ids keep them on separate broadcast domains.
|
||||
func (c *Config) Validate() error {
|
||||
if c.WAN.Folded() {
|
||||
return fmt.Errorf("wan role must be bound to an interface")
|
||||
}
|
||||
|
||||
seen := map[string]Plane{}
|
||||
for _, r := range c.Roles() {
|
||||
link := r.Role.Link()
|
||||
if prev, dup := seen[link]; dup {
|
||||
return fmt.Errorf("%s and %s both bind %s: give each a distinct VLAN id or fold one onto the other",
|
||||
prev, r.Plane, link)
|
||||
}
|
||||
seen[link] = r.Plane
|
||||
if r.Role.VLAN < 0 || r.Role.VLAN > 4094 {
|
||||
return fmt.Errorf("%s: VLAN id %d out of range (1-4094)", r.Plane, r.Role.VLAN)
|
||||
}
|
||||
if r.Role.MTU != 0 && (r.Role.MTU < 68 || r.Role.MTU > 9216) {
|
||||
return fmt.Errorf("%s: MTU %d out of range (68-9216)", r.Plane, r.Role.MTU)
|
||||
}
|
||||
// Resolvers belong to the plane that holds the default route. Accepting
|
||||
// one here and dropping it during generation would leave an unattended
|
||||
// install believing it had set a resolver that was never written.
|
||||
if r.Plane != PlaneWAN && len(r.Role.DNS) > 0 {
|
||||
return fmt.Errorf("%s: DNS servers belong to the wan plane, which holds the default route", r.Plane)
|
||||
}
|
||||
// Netgen converts the mask with net.ParseIP, so a prefix length reaches
|
||||
// it as garbage and fails the install after the disk is partitioned.
|
||||
// Catch it here, where the operator can still fix it.
|
||||
if m := strings.TrimSpace(r.Role.Mask); m != "" && !validDottedMask(m) {
|
||||
return fmt.Errorf("%s: subnet mask %q must be dotted-decimal, e.g. 255.255.255.0", r.Plane, m)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// validDottedMask reports whether s is a contiguous dotted-decimal IPv4 subnet
|
||||
// mask. Prefix-length form ("24", "/24") is deliberately not accepted: netgen
|
||||
// parses the mask with net.ParseIP, so anything else fails the install itself.
|
||||
func validDottedMask(s string) bool {
|
||||
ip := net.ParseIP(strings.TrimSpace(s))
|
||||
if ip == nil || ip.To4() == nil {
|
||||
return false
|
||||
}
|
||||
// Size reports zero bits for a non-contiguous mask, which netgen rejects too.
|
||||
_, bits := net.IPMask(ip.To4()).Size()
|
||||
return bits != 0
|
||||
}
|
||||
@@ -1,463 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Node layouts drawn from the us-west-1 racks, which between them exercise
|
||||
// every collapse permutation the installer has to express.
|
||||
func threeNIC() *Config { // hydrogen / radon
|
||||
return &Config{
|
||||
WAN: NetworkRole{Interface: "eno6", Address: "216.218.163.99", Mask: "255.255.255.224", Gateway: "216.218.163.97"},
|
||||
LAN: NetworkRole{Interface: "ens1f0np0", Address: "10.0.0.3", Mask: "255.255.0.0", MTU: 9000},
|
||||
VPC: NetworkRole{Interface: "ens1f1np1", Address: "10.1.0.3", Mask: "255.255.0.0", MTU: 9000},
|
||||
}
|
||||
}
|
||||
|
||||
func vlanSplit() *Config { // helium — one 100gbe carrying lan and vpc
|
||||
return &Config{
|
||||
WAN: NetworkRole{Interface: "enx6c1ff7c1475a", DHCPMode: true},
|
||||
LAN: NetworkRole{Interface: "enp129s0np0", VLAN: 10, Address: "10.0.0.2", Mask: "255.255.0.0", MTU: 9000},
|
||||
VPC: NetworkRole{Interface: "enp129s0np0", VLAN: 20, Address: "10.1.0.2", Mask: "255.255.0.0", MTU: 9000},
|
||||
}
|
||||
}
|
||||
|
||||
func dualPortTagged() *Config { // us-west-1-az2 — wan and lan share port 1
|
||||
return &Config{
|
||||
WAN: NetworkRole{Interface: "enp1s0f0", VLAN: 30, Address: "216.218.163.10", Mask: "255.255.255.224", Gateway: "216.218.163.97"},
|
||||
LAN: NetworkRole{Interface: "enp1s0f0", VLAN: 10, Address: "10.0.0.10", Mask: "255.255.0.0", MTU: 9000},
|
||||
VPC: NetworkRole{Interface: "enp1s0f1", VLAN: 20, Address: "10.1.0.10", Mask: "255.255.0.0", MTU: 9000},
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCollapsesUpTheChain(t *testing.T) {
|
||||
singleNIC := &Config{WAN: NetworkRole{Interface: "eth0", Address: "192.168.1.10", Mask: "255.255.255.0"}}
|
||||
twoNIC := &Config{
|
||||
WAN: NetworkRole{Interface: "eth0", Address: "192.168.1.10", Mask: "255.255.255.0"},
|
||||
LAN: NetworkRole{Interface: "eth1", Address: "10.0.0.5", Mask: "255.255.0.0"},
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg *Config
|
||||
plane Plane
|
||||
wantAddr string
|
||||
wantPlane Plane
|
||||
}{
|
||||
{"1 NIC: lan folds onto wan", singleNIC, PlaneLAN, "192.168.1.10", PlaneWAN},
|
||||
{"1 NIC: vpc folds all the way to wan", singleNIC, PlaneVPC, "192.168.1.10", PlaneWAN},
|
||||
{"2 NIC: vpc folds onto lan", twoNIC, PlaneVPC, "10.0.0.5", PlaneLAN},
|
||||
{"2 NIC: lan stays put", twoNIC, PlaneLAN, "10.0.0.5", PlaneLAN},
|
||||
{"3 NIC: vpc is its own plane", threeNIC(), PlaneVPC, "10.1.0.3", PlaneVPC},
|
||||
{"VLAN split: vpc is its own plane", vlanSplit(), PlaneVPC, "10.1.0.2", PlaneVPC},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := tt.cfg.PlaneAddress(tt.plane); got != tt.wantAddr {
|
||||
t.Errorf("PlaneAddress(%s) = %q, want %q", tt.plane, got, tt.wantAddr)
|
||||
}
|
||||
if _, landed := tt.cfg.Resolve(tt.plane); landed != tt.wantPlane {
|
||||
t.Errorf("Resolve(%s) landed on %s, want %s", tt.plane, landed, tt.wantPlane)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A DHCP plane has no address known at install time; setup-ovn.sh auto-detects
|
||||
// it from the default route at boot instead.
|
||||
func TestPlaneAddressEmptyUnderDHCP(t *testing.T) {
|
||||
if got := vlanSplit().PlaneAddress(PlaneWAN); got != "" {
|
||||
t.Errorf("PlaneAddress(wan) = %q, want empty for a DHCP role", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg *Config
|
||||
wantErr string
|
||||
}{
|
||||
{"three physical NICs", threeNIC(), ""},
|
||||
{"VLAN split on one NIC", vlanSplit(), ""},
|
||||
{"dual port, all tagged", dualPortTagged(), ""},
|
||||
{
|
||||
"wan cannot be folded",
|
||||
&Config{LAN: NetworkRole{Interface: "eth1"}},
|
||||
"wan role must be bound",
|
||||
},
|
||||
{
|
||||
"two roles share a NIC untagged",
|
||||
&Config{
|
||||
WAN: NetworkRole{Interface: "eth0"},
|
||||
LAN: NetworkRole{Interface: "eth0"},
|
||||
},
|
||||
"both bind eth0",
|
||||
},
|
||||
{
|
||||
"duplicate VLAN id on one NIC",
|
||||
&Config{
|
||||
WAN: NetworkRole{Interface: "eth0", VLAN: 10},
|
||||
LAN: NetworkRole{Interface: "eth0", VLAN: 10},
|
||||
},
|
||||
"both bind eth0.10",
|
||||
},
|
||||
{
|
||||
"VLAN id out of range",
|
||||
&Config{WAN: NetworkRole{Interface: "eth0", VLAN: 5000}},
|
||||
"out of range",
|
||||
},
|
||||
{
|
||||
"MTU out of range",
|
||||
&Config{WAN: NetworkRole{Interface: "eth0", MTU: 70000}},
|
||||
"MTU 70000 out of range",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := tt.cfg.Validate()
|
||||
if tt.wantErr == "" {
|
||||
if err != nil {
|
||||
t.Fatalf("Validate() = %v, want nil", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err == nil {
|
||||
t.Fatalf("Validate() = nil, want error containing %q", tt.wantErr)
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Errorf("Validate() = %v, want error containing %q", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Two roles sharing one NIC must produce a single parent .network carrying both
|
||||
// VLAN declarations — not two files racing to configure the same interface.
|
||||
func TestWriteParentNICTaggedSharesOneFile(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := &parentNIC{vlans: []string{"enp129s0np0.10", "enp129s0np0.20"}, mtu: 9000}
|
||||
if err := writeParentNIC(dir, "enp129s0np0", p); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got := readFile(t, dir, "05-spinifex-nic-enp129s0np0.network")
|
||||
for _, want := range []string{
|
||||
"Name=enp129s0np0",
|
||||
"MTUBytes=9000",
|
||||
"VLAN=enp129s0np0.10",
|
||||
"VLAN=enp129s0np0.20",
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("parent .network missing %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
// A tagged parent is never itself enslaved — the VLAN devices are.
|
||||
if strings.Contains(got, "Bridge=") {
|
||||
t.Errorf("tagged parent must not be bridged directly:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteNetworkdBridgeVLAN(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cfg := vlanSplit()
|
||||
if err := writeNetworkdBridge(dir, PlaneVPC, cfg.VPC, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
netdev := readFile(t, dir, "10-spinifex-vpc-vlan.netdev")
|
||||
if !strings.Contains(netdev, "Kind=vlan") || !strings.Contains(netdev, "Id=20") {
|
||||
t.Errorf("VLAN netdev wrong:\n%s", netdev)
|
||||
}
|
||||
if !strings.Contains(netdev, "Name=enp129s0np0.20") {
|
||||
t.Errorf("VLAN netdev name wrong:\n%s", netdev)
|
||||
}
|
||||
|
||||
vlanNet := readFile(t, dir, "11-spinifex-vpc-vlan.network")
|
||||
if !strings.Contains(vlanNet, "Bridge=br-vpc") {
|
||||
t.Errorf("VLAN device must be enslaved to br-vpc:\n%s", vlanNet)
|
||||
}
|
||||
|
||||
brNet := readFile(t, dir, "20-spinifex-vpc.network")
|
||||
if !strings.Contains(brNet, "Address=10.1.0.2/16") {
|
||||
t.Errorf("bridge address wrong:\n%s", brNet)
|
||||
}
|
||||
// Only wan installs a default route; an east-west plane must not.
|
||||
if strings.Contains(brNet, "Gateway=") {
|
||||
t.Errorf("vpc plane must not install a default route:\n%s", brNet)
|
||||
}
|
||||
if !strings.Contains(brNet, "ActivationPolicy=manual") {
|
||||
t.Errorf("east-west bridge must not auto-activate:\n%s", brNet)
|
||||
}
|
||||
}
|
||||
|
||||
// The wan plane is the one that carries a default route and auto-activates.
|
||||
func TestWriteNetworkdBridgeWANUntagged(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cfg := threeNIC()
|
||||
if err := writeNetworkdBridge(dir, PlaneWAN, cfg.WAN, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(filepath.Join(dir, "10-spinifex-wan-vlan.netdev")); !os.IsNotExist(err) {
|
||||
t.Error("untagged role must not produce a VLAN netdev")
|
||||
}
|
||||
brNet := readFile(t, dir, "20-spinifex-wan.network")
|
||||
if !strings.Contains(brNet, "Gateway=216.218.163.97") {
|
||||
t.Errorf("wan must install its default route:\n%s", brNet)
|
||||
}
|
||||
if strings.Contains(brNet, "ActivationPolicy=manual") {
|
||||
t.Errorf("wan bridge must auto-activate:\n%s", brNet)
|
||||
}
|
||||
}
|
||||
|
||||
func TestToFirstbootConfigSourcesPlanes(t *testing.T) {
|
||||
cfg := threeNIC()
|
||||
cfg.Hostname = "hydrogen"
|
||||
fb := cfg.toFirstbootConfig()
|
||||
|
||||
if fb.EncapIP != "10.1.0.3" {
|
||||
t.Errorf("EncapIP = %q, want the vpc plane address", fb.EncapIP)
|
||||
}
|
||||
if fb.LANIP != "10.0.0.3" {
|
||||
t.Errorf("LANIP = %q, want the lan plane address", fb.LANIP)
|
||||
}
|
||||
// The public address has to travel separately: spx echoes a concrete --bind
|
||||
// back as the advertise address, so leaving this empty would publish the
|
||||
// internal plane as the node's public dial target.
|
||||
if fb.WANIP != "216.218.163.99" {
|
||||
t.Errorf("WANIP = %q, want the wan plane address", fb.WANIP)
|
||||
}
|
||||
|
||||
// On a single-NIC node all three collapse onto wan.
|
||||
single := &Config{WAN: NetworkRole{Interface: "eth0", Address: "192.168.1.10", Mask: "255.255.255.0"}}
|
||||
fb = single.toFirstbootConfig()
|
||||
if fb.EncapIP != "192.168.1.10" || fb.LANIP != "192.168.1.10" || fb.WANIP != "192.168.1.10" {
|
||||
t.Errorf("single-NIC collapse wrong: EncapIP=%q LANIP=%q WANIP=%q", fb.EncapIP, fb.LANIP, fb.WANIP)
|
||||
}
|
||||
}
|
||||
|
||||
func readFile(t *testing.T, dir, name string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
t.Fatalf("reading %s: %v", name, err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// Resolvers belong to the plane holding the default route. The east-west
|
||||
// bridges are link-local and activate manually, so a resolver pinned to one is
|
||||
// unreachable while resolved still tries it — a timeout in front of every
|
||||
// lookup. The TUI cannot express this; the kernel cmdline can, so it is caught.
|
||||
func TestValidateRejectsDNSOffWAN(t *testing.T) {
|
||||
cfg := threeNIC()
|
||||
cfg.LAN.DNS = []string{"10.0.0.1"}
|
||||
if err := cfg.Validate(); err == nil {
|
||||
t.Fatal("lan must not carry DNS servers")
|
||||
}
|
||||
|
||||
cfg = threeNIC()
|
||||
cfg.VPC.DNS = []string{"10.1.0.1"}
|
||||
if err := cfg.Validate(); err == nil {
|
||||
t.Fatal("vpc must not carry DNS servers")
|
||||
}
|
||||
|
||||
cfg = threeNIC()
|
||||
cfg.WAN.DNS = []string{"8.8.8.8", "1.1.1.1"}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("wan carries the resolvers: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Even if a role somehow carries DNS, generation must not write it onto an
|
||||
// east-west bridge.
|
||||
func TestWriteNetworkdBridgeKeepsDNSOnWAN(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
lan := NetworkRole{Interface: "ens1f0np0", Address: "10.0.0.3", Mask: "255.255.0.0", DNS: []string{"10.0.0.1"}}
|
||||
if err := writeNetworkdBridge(dir, PlaneLAN, lan, true); err != nil {
|
||||
t.Fatalf("writeNetworkdBridge: %v", err)
|
||||
}
|
||||
if got := readFile(t, dir, "20-spinifex-lan.network"); strings.Contains(got, "DNS=") {
|
||||
t.Errorf("lan bridge must carry no resolver:\n%s", got)
|
||||
}
|
||||
|
||||
wan := NetworkRole{Interface: "eno6", Address: "216.218.163.99", Mask: "255.255.255.224", Gateway: "216.218.163.97", DNS: []string{"8.8.8.8"}}
|
||||
if err := writeNetworkdBridge(dir, PlaneWAN, wan, false); err != nil {
|
||||
t.Fatalf("writeNetworkdBridge: %v", err)
|
||||
}
|
||||
if got := readFile(t, dir, "20-spinifex-wan.network"); !strings.Contains(got, "DNS=8.8.8.8") {
|
||||
t.Errorf("wan bridge must carry the resolver:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSwapSize(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
// Without an override, RAM drives the size and the disk caps it.
|
||||
ram, err := totalRAM()
|
||||
if err != nil {
|
||||
t.Fatalf("totalRAM: %v", err)
|
||||
}
|
||||
got, err := swapSize(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("swapSize: %v", err)
|
||||
}
|
||||
want := ram / 2
|
||||
if ram >= swapRAMThreshold {
|
||||
want = swapLargeBytes
|
||||
}
|
||||
if got > want {
|
||||
t.Errorf("swapSize = %d, must not exceed what RAM implies (%d)", got, want)
|
||||
}
|
||||
if got != 0 && got < swapMinBytes {
|
||||
t.Errorf("swapSize = %d, must be 0 rather than below the %d floor", got, int64(swapMinBytes))
|
||||
}
|
||||
|
||||
// An explicit size wins, so a large node can be told to take the full 32G
|
||||
// and a constrained one can opt out entirely.
|
||||
for _, tt := range []struct {
|
||||
env string
|
||||
want int64
|
||||
}{
|
||||
{"32G", 32 << 30},
|
||||
{"512M", 512 << 20},
|
||||
{"0", 0},
|
||||
{"1073741824", 1 << 30},
|
||||
} {
|
||||
t.Setenv("SPINIFEX_SWAP_SIZE", tt.env)
|
||||
got, err := swapSize(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("swapSize(%s): %v", tt.env, err)
|
||||
}
|
||||
if got != tt.want {
|
||||
t.Errorf("swapSize(%s) = %d, want %d", tt.env, got, tt.want)
|
||||
}
|
||||
}
|
||||
|
||||
t.Setenv("SPINIFEX_SWAP_SIZE", "banana")
|
||||
if _, err := swapSize(dir); err == nil {
|
||||
t.Error("an unparseable size must be reported, not silently ignored")
|
||||
}
|
||||
}
|
||||
|
||||
// The swap line must only appear when the file backing it exists — systemd
|
||||
// fails the boot's swap unit on a missing file rather than skipping it.
|
||||
func TestFstabSwapLineTracksTheFile(t *testing.T) {
|
||||
if _, err := parseSize("8G"); err != nil {
|
||||
t.Fatalf("parseSize: %v", err)
|
||||
}
|
||||
if n, _ := parseSize("2G"); n != 2<<30 {
|
||||
t.Errorf("parseSize(2G) = %d, want %d", n, int64(2)<<30)
|
||||
}
|
||||
if _, err := parseSize("-5G"); err == nil {
|
||||
t.Error("negative sizes must be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
// Bridge ports and VLAN trunks never reach the "degraded" state
|
||||
// systemd-networkd-wait-online requires — the address sits on the bridge above
|
||||
// them. Leaving them required makes that unit fail on every boot and stall
|
||||
// until it times out, which is what a three-NIC node was doing.
|
||||
func TestSlaveLinksAreNotRequiredForOnline(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
// Untagged bridge port.
|
||||
if err := writeParentNIC(dir, "ens4", &parentNIC{bridge: "br-lan", mtu: 9000}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := readFile(t, dir, "05-spinifex-nic-ens4.network")
|
||||
if !strings.Contains(got, "RequiredForOnline=no") {
|
||||
t.Errorf("bridge port must not gate network-online:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "MTUBytes=9000") {
|
||||
t.Errorf("MTU must survive alongside it:\n%s", got)
|
||||
}
|
||||
|
||||
// VLAN trunk carrying tagged planes.
|
||||
if err := writeParentNIC(dir, "enp129s0np0", &parentNIC{vlans: []string{"enp129s0np0.10"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := readFile(t, dir, "05-spinifex-nic-enp129s0np0.network"); !strings.Contains(got, "RequiredForOnline=no") {
|
||||
t.Errorf("VLAN trunk must not gate network-online:\n%s", got)
|
||||
}
|
||||
|
||||
// The VLAN device itself is enslaved too.
|
||||
cfg := vlanSplit()
|
||||
if err := writeNetworkdBridge(dir, PlaneVPC, cfg.VPC, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := readFile(t, dir, "11-spinifex-vpc-vlan.network"); !strings.Contains(got, "RequiredForOnline=no") {
|
||||
t.Errorf("VLAN device must not gate network-online:\n%s", got)
|
||||
}
|
||||
|
||||
// The wan bridge is what should gate it, and still does.
|
||||
three := threeNIC()
|
||||
if err := writeNetworkdBridge(dir, PlaneWAN, three.WAN, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := readFile(t, dir, "20-spinifex-wan.network"); strings.Contains(got, "RequiredForOnline=no") {
|
||||
t.Errorf("wan bridge must remain required for network-online:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A prefix length reaches netgen's net.ParseIP as garbage and fails the install
|
||||
// after the disk has been partitioned, so it has to be rejected up front. The
|
||||
// TUI advertised "255.255.255.0 or 24" while only the first ever worked.
|
||||
func TestValidateRejectsPrefixLengthMask(t *testing.T) {
|
||||
for _, bad := range []string{"24", "/24", "16", "255.255.0", "255.0.255.0", "banana"} {
|
||||
cfg := threeNIC()
|
||||
cfg.LAN.Mask = bad
|
||||
if err := cfg.Validate(); err == nil {
|
||||
t.Errorf("mask %q must be rejected", bad)
|
||||
} else if !strings.Contains(err.Error(), "dotted-decimal") {
|
||||
t.Errorf("mask %q: error should name the expected form, got: %v", bad, err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, good := range []string{"255.255.255.0", "255.255.0.0", "255.255.255.252", "255.255.255.255"} {
|
||||
cfg := threeNIC()
|
||||
cfg.LAN.Mask = good
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Errorf("mask %q must be accepted: %v", good, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A DHCP role carries no mask, and that must stay valid.
|
||||
cfg := threeNIC()
|
||||
cfg.LAN = NetworkRole{Interface: "ens1f0np0", DHCPMode: true}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Errorf("a DHCP role has no mask to check: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A bridged uplink must present the enslaved NIC's address. Left to its default
|
||||
// policy, systemd-networkd generates one for the bridge, so the node appears on
|
||||
// the wire as an address belonging to no NIC — DHCP requests carry a source that
|
||||
// disagrees with their own chaddr and the unicast reply is never delivered.
|
||||
func TestBridgeInheritsPortMAC(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cfg := threeNIC()
|
||||
|
||||
for _, tc := range []struct {
|
||||
plane Plane
|
||||
role NetworkRole
|
||||
manual bool
|
||||
}{
|
||||
{PlaneWAN, cfg.WAN, false},
|
||||
{PlaneLAN, cfg.LAN, true},
|
||||
{PlaneVPC, cfg.VPC, true},
|
||||
} {
|
||||
if err := writeNetworkdBridge(dir, tc.plane, tc.role, tc.manual); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := readFile(t, dir, "20-spinifex-"+string(tc.plane)+".netdev")
|
||||
if !strings.Contains(got, "MACAddressPolicy=none") {
|
||||
t.Errorf("%s bridge must inherit its port's MAC:\n%s", tc.plane, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,278 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Partition numbers, identical on every disk the installer touches so that a
|
||||
// replacement drive can be cloned with `sgdisk -R` and nothing has to be
|
||||
// recomputed at 3am.
|
||||
const (
|
||||
biosPartNum = 1
|
||||
espPartNum = 2
|
||||
rootPartNum = 3
|
||||
|
||||
// dataPartNum is the sole partition on a role drive. Data drives carry no
|
||||
// bootloader, so they start at 1 rather than mirroring the boot layout.
|
||||
dataPartNum = 1
|
||||
)
|
||||
|
||||
// GPT partition type GUIDs, by their sgdisk short code.
|
||||
const (
|
||||
typeBIOSBoot = "EF02"
|
||||
typeESP = "EF00"
|
||||
typeLinuxFS = "8300"
|
||||
typeZFS = "BF00"
|
||||
)
|
||||
|
||||
// Partition geometry. The ESP is sized as Proxmox does: 512MiB is enough for
|
||||
// two kernels, but a 1GiB ESP on a real disk costs nothing and keeps room for
|
||||
// the extra kernels an operator pins during an upgrade.
|
||||
const (
|
||||
espSmallMiB = 512
|
||||
espLargeMiB = 1024
|
||||
espLargeDiskGiB = 100
|
||||
|
||||
// biosBootMiB is the BIOS Boot Partition GRUB's i386-pc core.img is embedded
|
||||
// into. GPT leaves no post-MBR gap, so without it there is nowhere to put it.
|
||||
biosBootMiB = 1
|
||||
|
||||
// tailReserveMiB is left unallocated at the end of every disk. A replacement
|
||||
// drive of the "same" size is routinely a few thousand sectors smaller, and
|
||||
// without slack `sgdisk -R` refuses to clone the table onto it — which is
|
||||
// discovered while standing in front of a degraded pool.
|
||||
tailReserveMiB = 1024
|
||||
)
|
||||
|
||||
// partitionSettleTimeout bounds the wait for the kernel and udev to publish a
|
||||
// disk's new partition table. A var so tests do not pay it.
|
||||
var partitionSettleTimeout = 15 * time.Second
|
||||
|
||||
// releaseDisk detaches whatever the live environment has attached to a disk
|
||||
// before it is erased. A mounted filesystem or an active swap area makes
|
||||
// BLKRRPART return EBUSY, and the kernel then keeps serving the disk's old
|
||||
// partition table while sgdisk writes the new one and reports success.
|
||||
func releaseDisk(d Disk) {
|
||||
for _, m := range mountsOnDisk(d.Path) {
|
||||
err := runQuiet("umount", m.Device)
|
||||
if err != nil {
|
||||
// Lazy detach: a busy mount must not stall an install that is about to
|
||||
// erase the filesystem underneath it anyway.
|
||||
err = runQuiet("umount", "-l", m.Device)
|
||||
}
|
||||
if err != nil {
|
||||
slog.Warn("could not unmount before erase", "device", m.Device, "mountpoint", m.Mountpoint, "err", err)
|
||||
continue
|
||||
}
|
||||
slog.Info("released mount before erase", "device", m.Device, "mountpoint", m.Mountpoint)
|
||||
}
|
||||
for _, s := range swapsOnDisk(d.Path) {
|
||||
if err := runQuiet("swapoff", s); err != nil {
|
||||
slog.Warn("could not disable swap before erase", "device", s, "err", err)
|
||||
continue
|
||||
}
|
||||
slog.Info("disabled swap before erase", "device", s)
|
||||
}
|
||||
}
|
||||
|
||||
// settlePartitions forces the kernel to re-read a disk's partition table and
|
||||
// blocks until its view matches want, naming the partitions by number.
|
||||
//
|
||||
// An empty want asserts the disk has no partitions at all. That is the only
|
||||
// state no leftover can satisfy: a disk whose previous table had the same
|
||||
// partition numbers as the one being written — an old OS install is p1, p2, p3,
|
||||
// exactly what the boot layout writes — passes an "are the expected partitions
|
||||
// there" check on stale nodes alone.
|
||||
func settlePartitions(d Disk, want ...int) error {
|
||||
if err := runQuiet("partprobe", d.Path); err != nil {
|
||||
// partprobe rescans the whole disk and fails if it cannot open any part of
|
||||
// it; the ioctl on its own sometimes still gets through.
|
||||
if err := runQuiet("blockdev", "--rereadpt", d.Path); err != nil {
|
||||
slog.Warn("could not force a partition table re-read", "disk", d.Path, "err", err)
|
||||
}
|
||||
}
|
||||
_ = runQuiet("udevadm", "settle", "--timeout=10")
|
||||
|
||||
wanted := make([]string, 0, len(want))
|
||||
for _, n := range want {
|
||||
wanted = append(wanted, filepath.Base(d.PartitionPath(n)))
|
||||
}
|
||||
slices.Sort(wanted)
|
||||
|
||||
deadline := time.Now().Add(partitionSettleTimeout)
|
||||
for {
|
||||
got, err := kernelPartitions(d.Path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if slices.Equal(got, wanted) {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return staleTableError(d, wanted, got)
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// staleTableError explains a kernel partition view that never caught up. The
|
||||
// erase case gets its own wording because it is the dangerous one — proceeding
|
||||
// would format the previous layout — and because the holder is usually the
|
||||
// whole diagnosis.
|
||||
func staleTableError(d Disk, wanted, got []string) error {
|
||||
if len(wanted) == 0 {
|
||||
msg := fmt.Sprintf("%s still shows partitions [%s] after its table was erased — the kernel is using the old table, so formatting would write at the previous layout's offsets",
|
||||
d.Path, strings.Join(got, ", "))
|
||||
if h := diskHolders(d.Path); len(h) > 0 {
|
||||
return fmt.Errorf("%s; it is held by %s, which must be stopped before this disk can be installed to",
|
||||
msg, strings.Join(h, ", "))
|
||||
}
|
||||
return fmt.Errorf("%s; something in the live environment is holding it open", msg)
|
||||
}
|
||||
return fmt.Errorf("%s: the kernel shows partitions [%s] but the new table has [%s] — the table was written but never picked up",
|
||||
d.Path, strings.Join(got, ", "), strings.Join(wanted, ", "))
|
||||
}
|
||||
|
||||
// espSizeMiB picks the ESP size for a disk.
|
||||
func espSizeMiB(d Disk) int {
|
||||
if d.Bytes >= espLargeDiskGiB<<30 {
|
||||
return espLargeMiB
|
||||
}
|
||||
return espSmallMiB
|
||||
}
|
||||
|
||||
// partitionDisks lays down an identical GPT on every selected disk. Each disk
|
||||
// gets a BIOS boot partition and an ESP whether or not this machine booted in
|
||||
// EFI mode: the cost is 1GiB, and it means a board swap or a firmware change
|
||||
// from BIOS to UEFI does not leave an unbootable pool.
|
||||
func partitionDisks(cfg DiskConfig) error {
|
||||
ptype := typeLinuxFS
|
||||
if cfg.FS.IsZFS() {
|
||||
ptype = typeZFS
|
||||
}
|
||||
for _, d := range cfg.bootDisks() {
|
||||
if err := partitionOne(d, ptype); err != nil {
|
||||
return fmt.Errorf("partition %s: %w", d.Path, err)
|
||||
}
|
||||
}
|
||||
for _, rm := range cfg.DataMounts() {
|
||||
if err := partitionDataDisk(rm); err != nil {
|
||||
return fmt.Errorf("partition %s (%s): %w", rm.Disk.Path, rm.Role, err)
|
||||
}
|
||||
}
|
||||
return waitForPartitions(cfg)
|
||||
}
|
||||
|
||||
// partitionDataDisk writes the single-partition GPT for a role drive. It gets
|
||||
// no bios_boot and no ESP: nothing installs a bootloader there, and a formatted
|
||||
// ESP no firmware entry points at only misleads whoever reads lsblk in an
|
||||
// incident.
|
||||
func partitionDataDisk(rm RoleMount) error {
|
||||
if err := clearDisk(rm.Disk); err != nil {
|
||||
return fmt.Errorf("clear: %w", err)
|
||||
}
|
||||
// Same tail reserve as a boot drive, for the same reason: a replacement of
|
||||
// the "same" size is routinely a few thousand sectors smaller.
|
||||
if err := run("sgdisk",
|
||||
"-n", fmt.Sprintf("%d:1M:-%dM", dataPartNum, tailReserveMiB),
|
||||
"-t", fmt.Sprintf("%d:%s", dataPartNum, typeLinuxFS),
|
||||
"-c", fmt.Sprintf("%d:%s", dataPartNum, rm.Role.Label()),
|
||||
rm.Disk.Path,
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := run("sgdisk", "-G", rm.Disk.Path); err != nil {
|
||||
slog.Warn("could not randomise GPT GUIDs", "disk", rm.Disk.Path, "err", err)
|
||||
}
|
||||
return settlePartitions(rm.Disk, dataPartNum)
|
||||
}
|
||||
|
||||
// partitionOne writes the GPT for a single disk.
|
||||
func partitionOne(d Disk, rootType string) error {
|
||||
// Stale ZFS labels and filesystem signatures survive a new partition table
|
||||
// and make a retried install fail with errors that read like hardware faults.
|
||||
if err := clearDisk(d); err != nil {
|
||||
return fmt.Errorf("clear: %w", err)
|
||||
}
|
||||
|
||||
esp := espSizeMiB(d)
|
||||
if err := run("sgdisk",
|
||||
"-n", fmt.Sprintf("%d:1M:+%dM", biosPartNum, biosBootMiB),
|
||||
"-t", fmt.Sprintf("%d:%s", biosPartNum, typeBIOSBoot),
|
||||
"-c", fmt.Sprintf("%d:bios_boot", biosPartNum),
|
||||
"-n", fmt.Sprintf("%d:0:+%dM", espPartNum, esp),
|
||||
"-t", fmt.Sprintf("%d:%s", espPartNum, typeESP),
|
||||
"-c", fmt.Sprintf("%d:ESP", espPartNum),
|
||||
"-n", fmt.Sprintf("%d:0:-%dM", rootPartNum, tailReserveMiB),
|
||||
"-t", fmt.Sprintf("%d:%s", rootPartNum, rootType),
|
||||
"-c", fmt.Sprintf("%d:root", rootPartNum),
|
||||
d.Path,
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
// A fresh disk-level GUID: cloning a table with `sgdisk -R` copies it, and
|
||||
// two disks sharing one leaves the pool ambiguous to udev.
|
||||
if err := run("sgdisk", "-G", d.Path); err != nil {
|
||||
slog.Warn("could not randomise GPT GUIDs", "disk", d.Path, "err", err)
|
||||
}
|
||||
return settlePartitions(d, biosPartNum, espPartNum, rootPartNum)
|
||||
}
|
||||
|
||||
// waitForPartitions ensures every partition device node the installer is about
|
||||
// to use exists. The kernel's own view was already asserted per disk as each
|
||||
// table was written; udev publishes the /dev nodes and by-id symlinks after
|
||||
// that, and Trixie is slow enough about it that mkfs races them.
|
||||
func waitForPartitions(cfg DiskConfig) error {
|
||||
deadline := time.Now().Add(partitionSettleTimeout)
|
||||
for _, d := range cfg.bootDisks() {
|
||||
wanted := []string{d.PartitionPath(espPartNum), d.PartitionPath(rootPartNum)}
|
||||
// The pool is built from by-id paths, so their symlinks have to exist
|
||||
// too — udev publishes them later than the kernel device node.
|
||||
if d.ByID != "" {
|
||||
wanted = append(wanted, d.StablePartitionPath(rootPartNum))
|
||||
}
|
||||
for _, part := range wanted {
|
||||
if err := waitForPath(part, deadline); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, rm := range cfg.DataMounts() {
|
||||
if err := waitForPath(rm.Disk.PartitionPath(dataPartNum), deadline); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func waitForPath(path string, deadline time.Time) error {
|
||||
for {
|
||||
if _, err := os.Stat(path); err == nil {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("device %s did not appear within timeout — the kernel or udev did not pick up the new partition table", path)
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// formatESPs makes a FAT32 filesystem on every disk's ESP. Each one is
|
||||
// independent — there is no RAID here, they are kept in step by content.
|
||||
func formatESPs(cfg DiskConfig) error {
|
||||
for _, d := range cfg.bootDisks() {
|
||||
esp := d.PartitionPath(espPartNum)
|
||||
// A distinct label per member so `lsblk -f` on a degraded machine tells
|
||||
// the operator which disk each ESP belongs to.
|
||||
if err := run("mkfs.fat", "-F", "32", "-n", "EFI", esp); err != nil {
|
||||
return fmt.Errorf("format ESP %s: %w", esp, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,483 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Pool and dataset names. rpool matches Proxmox, which is what every ZFS
|
||||
// recovery guide and forum answer assumes — the name an operator will be
|
||||
// reading off a serial console at 3am.
|
||||
const (
|
||||
ZFSPoolName = "rpool"
|
||||
ZFSRootDataset = ZFSPoolName + "/ROOT/spinifex"
|
||||
)
|
||||
|
||||
// FSType selects the root filesystem and, for ZFS, the vdev topology.
|
||||
type FSType string
|
||||
|
||||
const (
|
||||
FSExt4 FSType = "ext4"
|
||||
FSZFSRAID0 FSType = "zfs-raid0"
|
||||
FSZFSRAID1 FSType = "zfs-raid1"
|
||||
FSZFSRAID10 FSType = "zfs-raid10"
|
||||
FSZFSRAIDZ1 FSType = "zfs-raidz1"
|
||||
FSZFSRAIDZ2 FSType = "zfs-raidz2"
|
||||
FSZFSRAIDZ3 FSType = "zfs-raidz3"
|
||||
)
|
||||
|
||||
// AllFSTypes is the selector order presented by the UI. ext4 is first and is
|
||||
// the default, so ZFS — which holds the kernel at its shipped version and
|
||||
// reserves RAM for the ARC — is always an explicit choice.
|
||||
var AllFSTypes = []FSType{
|
||||
FSExt4, FSZFSRAID0, FSZFSRAID1, FSZFSRAID10, FSZFSRAIDZ1, FSZFSRAIDZ2, FSZFSRAIDZ3,
|
||||
}
|
||||
|
||||
// fsMeta carries the per-topology facts that drive display and validation.
|
||||
var fsMeta = map[FSType]struct {
|
||||
label string
|
||||
minDisks int
|
||||
redundant bool // subject to the same-size rule
|
||||
vdev string // zpool create keyword; empty means a plain stripe
|
||||
}{
|
||||
FSExt4: {"ext4", 1, false, ""},
|
||||
FSZFSRAID0: {"zfs (RAID0)", 1, false, ""},
|
||||
FSZFSRAID1: {"zfs (RAID1)", 2, true, "mirror"},
|
||||
FSZFSRAID10: {"zfs (RAID10)", 4, true, "mirror"},
|
||||
FSZFSRAIDZ1: {"zfs (RAIDZ-1)", 3, true, "raidz1"},
|
||||
FSZFSRAIDZ2: {"zfs (RAIDZ-2)", 4, true, "raidz2"},
|
||||
FSZFSRAIDZ3: {"zfs (RAIDZ-3)", 5, true, "raidz3"},
|
||||
}
|
||||
|
||||
// IsZFS reports whether this mode creates a pool rather than a plain filesystem.
|
||||
func (f FSType) IsZFS() bool { return strings.HasPrefix(string(f), "zfs-") }
|
||||
|
||||
// Label is the display name, matching proxinstall's wording so the screen is
|
||||
// recognisable to anyone who has installed Proxmox.
|
||||
func (f FSType) Label() string { return fsMeta[f].label }
|
||||
|
||||
// MinDisks is the smallest member count the topology can be built from.
|
||||
func (f FSType) MinDisks() int { return fsMeta[f].minDisks }
|
||||
|
||||
// Redundant reports whether the topology stores parity or mirrors, and so
|
||||
// requires members of matching size. A RAID0 stripe over unequal disks is
|
||||
// legitimate and is deliberately not covered.
|
||||
func (f FSType) Redundant() bool { return fsMeta[f].redundant }
|
||||
|
||||
// vdevKeyword is the zpool create keyword introducing each vdev.
|
||||
func (f FSType) vdevKeyword() string { return fsMeta[f].vdev }
|
||||
|
||||
// ParseFSType maps a SPINIFEX_FS value to an FSType.
|
||||
func ParseFSType(s string) (FSType, error) {
|
||||
f := FSType(strings.ToLower(strings.TrimSpace(s)))
|
||||
if _, ok := fsMeta[f]; !ok {
|
||||
names := make([]string, 0, len(AllFSTypes))
|
||||
for _, t := range AllFSTypes {
|
||||
names = append(names, string(t))
|
||||
}
|
||||
return "", fmt.Errorf("unknown filesystem %q — must be one of: %s", s, strings.Join(names, ", "))
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// DiskRole is the purpose a selected drive serves on an ext4 install. A ZFS
|
||||
// pool carries every workload on one set of members, so roles do not apply
|
||||
// there — the topology already spans all the disks.
|
||||
type DiskRole string
|
||||
|
||||
const (
|
||||
RoleOS DiskRole = "os"
|
||||
RoleSpinifex DiskRole = "spinifex"
|
||||
RolePredastore DiskRole = "predastore"
|
||||
)
|
||||
|
||||
// AllDiskRoles is canonical order, outermost mountpoint first. Mounting, fstab
|
||||
// emission and rsync protect filters all walk it forwards; teardown walks it
|
||||
// back, because predastore nests inside spinifex.
|
||||
var AllDiskRoles = []DiskRole{RoleOS, RoleSpinifex, RolePredastore}
|
||||
|
||||
// roleMountpoints is where each role lands on the installed system. A role left
|
||||
// unassigned simply stays on the filesystem above it.
|
||||
var roleMountpoints = map[DiskRole]string{
|
||||
RoleOS: "/",
|
||||
RoleSpinifex: "/var/lib/spinifex",
|
||||
RolePredastore: "/var/lib/spinifex/predastore",
|
||||
}
|
||||
|
||||
// Label is the role name as shown in the UI and written to partition labels.
|
||||
func (r DiskRole) Label() string { return string(r) }
|
||||
|
||||
// Mountpoint is the role's path on the installed system.
|
||||
func (r DiskRole) Mountpoint() string { return roleMountpoints[r] }
|
||||
|
||||
// RoleMount binds one drive to one role.
|
||||
type RoleMount struct {
|
||||
Role DiskRole
|
||||
Disk Disk
|
||||
}
|
||||
|
||||
// Mountpoint is where this drive's filesystem is mounted on the target.
|
||||
func (r RoleMount) Mountpoint() string { return r.Role.Mountpoint() }
|
||||
|
||||
// ZFSOpts are the advanced tunables. A zero value means "use the computed
|
||||
// default", which is why Ashift and ARCMaxMiB are plain ints — 0 is not legal
|
||||
// for either, so it is unambiguous as a sentinel.
|
||||
type ZFSOpts struct {
|
||||
Ashift int
|
||||
Compress string
|
||||
Checksum string
|
||||
Copies int
|
||||
ARCMaxMiB int
|
||||
}
|
||||
|
||||
// DiskConfig is the storage half of the installer's configuration. Disks is
|
||||
// ordered and the order is significant: RAID10 pairs members two at a time, and
|
||||
// on ext4 the first entry is the OS drive.
|
||||
type DiskConfig struct {
|
||||
FS FSType
|
||||
Disks []Disk
|
||||
ZFS ZFSOpts
|
||||
|
||||
// Roles assigns ext4 drives to mountpoints, in AllDiskRoles order. Empty
|
||||
// means a single-disk install with everything on the root filesystem.
|
||||
Roles []RoleMount
|
||||
}
|
||||
|
||||
// WithRoles returns a copy with ext4 drive roles assigned and Disks rebuilt
|
||||
// from them in canonical order, so the list of drives to erase and the list of
|
||||
// roles cannot drift apart — there is one place a drive can be named, not two.
|
||||
func (d DiskConfig) WithRoles(roles []RoleMount) DiskConfig {
|
||||
d.Roles, d.Disks = nil, nil
|
||||
add := func(rm RoleMount) {
|
||||
d.Roles = append(d.Roles, rm)
|
||||
d.Disks = append(d.Disks, rm.Disk)
|
||||
}
|
||||
for _, want := range AllDiskRoles {
|
||||
for _, rm := range roles {
|
||||
if rm.Role == want {
|
||||
add(rm)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Anything with an unknown or duplicate role is kept so Validate can name
|
||||
// it, rather than being dropped into a silently smaller selection.
|
||||
for _, rm := range roles {
|
||||
if _, known := roleMountpoints[rm.Role]; !known {
|
||||
add(rm)
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// RolesForDisks assigns roles positionally, which is the default the UI offers:
|
||||
// first disk picked is the OS, second is spinifex, third is predastore.
|
||||
func RolesForDisks(disks []Disk) []RoleMount {
|
||||
out := make([]RoleMount, 0, len(disks))
|
||||
for i, disk := range disks {
|
||||
var role DiskRole
|
||||
if i < len(AllDiskRoles) {
|
||||
role = AllDiskRoles[i]
|
||||
}
|
||||
out = append(out, RoleMount{Role: role, Disk: disk})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// DataMounts is every role except the OS drive, shallowest mountpoint first.
|
||||
// This is the traversal every stage of the install shares.
|
||||
func (d DiskConfig) DataMounts() []RoleMount {
|
||||
var out []RoleMount
|
||||
for _, rm := range d.Roles {
|
||||
if rm.Role != RoleOS {
|
||||
out = append(out, rm)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// bootDisks are the drives a bootloader is installed to. Every ZFS member gets
|
||||
// its own ESP; on ext4 only the OS drive does, because a data drive holding a
|
||||
// bootloader that nothing points at is a trap during recovery.
|
||||
func (d DiskConfig) bootDisks() []Disk {
|
||||
if d.FS.IsZFS() || len(d.Disks) == 0 {
|
||||
return d.Disks
|
||||
}
|
||||
return d.Disks[:1]
|
||||
}
|
||||
|
||||
// Primary is the first selected disk — the OS drive in ext4 mode, and the
|
||||
// disk shown in single-disk summaries.
|
||||
func (d DiskConfig) Primary() Disk {
|
||||
if len(d.Disks) == 0 {
|
||||
return Disk{}
|
||||
}
|
||||
return d.Disks[0]
|
||||
}
|
||||
|
||||
// Paths lists the selected device paths, for logs and confirmation screens.
|
||||
func (d DiskConfig) Paths() []string {
|
||||
out := make([]string, len(d.Disks))
|
||||
for i, disk := range d.Disks {
|
||||
out[i] = disk.Path
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Buildable reports whether the selection has enough members, of the right
|
||||
// count, to form the topology at all. Capacity and fault tolerance are
|
||||
// meaningless until it does — a 3-disk RAIDZ-3 has no geometry to describe.
|
||||
func (d DiskConfig) Buildable() bool {
|
||||
if _, ok := fsMeta[d.FS]; !ok {
|
||||
return false
|
||||
}
|
||||
if len(d.Disks) < d.FS.MinDisks() {
|
||||
return false
|
||||
}
|
||||
return d.FS != FSZFSRAID10 || len(d.Disks)%2 == 0
|
||||
}
|
||||
|
||||
// Requirement describes what the topology still needs, for the preview line
|
||||
// shown in place of a capacity that cannot be computed yet.
|
||||
func (d DiskConfig) Requirement() string {
|
||||
req := fmt.Sprintf("needs %d disks", d.FS.MinDisks())
|
||||
if d.FS == FSZFSRAID10 {
|
||||
req = "needs an even number of disks, at least 4"
|
||||
}
|
||||
if d.FS.Redundant() {
|
||||
req += " of the same size"
|
||||
}
|
||||
return req
|
||||
}
|
||||
|
||||
// UsableBytes estimates post-parity capacity, for the geometry preview. It is
|
||||
// an operator-facing approximation and deliberately ignores metadata overhead.
|
||||
// Returns 0 for a selection that cannot form the topology, since the parity
|
||||
// arithmetic below goes to zero and then negative as members run out.
|
||||
func (d DiskConfig) UsableBytes() int64 {
|
||||
n := int64(len(d.Disks))
|
||||
if n == 0 || !d.Buildable() {
|
||||
return 0
|
||||
}
|
||||
smallest := d.Disks[0].Bytes
|
||||
for _, disk := range d.Disks {
|
||||
smallest = min(smallest, disk.Bytes)
|
||||
}
|
||||
switch d.FS {
|
||||
case FSExt4, FSZFSRAID0:
|
||||
var total int64
|
||||
for _, disk := range d.Disks {
|
||||
total += disk.Bytes
|
||||
}
|
||||
return total
|
||||
case FSZFSRAID1:
|
||||
return smallest
|
||||
case FSZFSRAID10:
|
||||
return smallest * (n / 2)
|
||||
case FSZFSRAIDZ1:
|
||||
return smallest * (n - 1)
|
||||
case FSZFSRAIDZ2:
|
||||
return smallest * (n - 2)
|
||||
case FSZFSRAIDZ3:
|
||||
return smallest * (n - 3)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Tolerated returns the number of simultaneous disk failures the topology
|
||||
// survives, for the geometry preview.
|
||||
func (d DiskConfig) Tolerated() int {
|
||||
if !d.Buildable() {
|
||||
return 0
|
||||
}
|
||||
switch d.FS {
|
||||
case FSZFSRAID1:
|
||||
return max(len(d.Disks)-1, 0)
|
||||
case FSZFSRAID10:
|
||||
return 1 // one per mirror pair; the honest worst case is one
|
||||
case FSZFSRAIDZ1:
|
||||
return 1
|
||||
case FSZFSRAIDZ2:
|
||||
return 2
|
||||
case FSZFSRAIDZ3:
|
||||
return 3
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// sizeTolerance is the fraction by which pool members may differ in size.
|
||||
// Matches Proxmox's zfs_mirror_size_check: anything beyond 10% and the pool
|
||||
// silently truncates every member to the smallest.
|
||||
const sizeTolerance = 0.10
|
||||
|
||||
// bootedEFI reports whether firmware booted this machine in EFI mode. GRUB's
|
||||
// i386-pc target cannot read a 4Kn drive, so legacy BIOS constrains selection.
|
||||
func bootedEFI() bool {
|
||||
_, err := os.Stat("/sys/firmware/efi")
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// Validate checks a disk selection against every rule that must hold before any
|
||||
// destructive command runs. The UI calls it on each keystroke so failures block
|
||||
// Continue, rather than aborting an install that has already repartitioned.
|
||||
func (d DiskConfig) Validate() error {
|
||||
if _, ok := fsMeta[d.FS]; !ok {
|
||||
return fmt.Errorf("no filesystem selected")
|
||||
}
|
||||
if len(d.Disks) == 0 {
|
||||
return fmt.Errorf("%s: select at least one disk", d.FS.Label())
|
||||
}
|
||||
if err := d.validateRoles(); err != nil {
|
||||
return err
|
||||
}
|
||||
if n := len(d.Disks); n < d.FS.MinDisks() {
|
||||
return fmt.Errorf("%s: needs at least %d disks, %d selected", d.FS.Label(), d.FS.MinDisks(), n)
|
||||
}
|
||||
if d.FS == FSZFSRAID10 && len(d.Disks)%2 != 0 {
|
||||
return fmt.Errorf("%s: needs an even number of disks, %d selected", d.FS.Label(), len(d.Disks))
|
||||
}
|
||||
|
||||
seen := map[string]bool{}
|
||||
for _, disk := range d.Disks {
|
||||
if seen[disk.Path] {
|
||||
return fmt.Errorf("%s is selected more than once", disk.Path)
|
||||
}
|
||||
seen[disk.Path] = true
|
||||
|
||||
if disk.LiveMedia {
|
||||
return fmt.Errorf("%s is the installer's own boot media and cannot be erased", disk.Path)
|
||||
}
|
||||
if disk.Bytes < minRootBytes {
|
||||
return fmt.Errorf("%s is too small (%s, need at least %dGiB)",
|
||||
disk.Path, disk.SizeHuman(), minRootBytes>>30)
|
||||
}
|
||||
}
|
||||
|
||||
// GRUB's i386-pc target cannot read a 4Kn drive, so on legacy BIOS a drive
|
||||
// holding a bootloader would leave the installed system unreachable. Only
|
||||
// boot drives are constrained: a data role is never booted from.
|
||||
for _, disk := range d.bootDisks() {
|
||||
if !bootedEFI() && disk.LogicalBlockSize == 4096 {
|
||||
return fmt.Errorf("%s is a 4Kn drive (4096-byte sectors) and this machine booted in legacy BIOS mode — booting from it is not supported; enable UEFI in firmware",
|
||||
disk.Path)
|
||||
}
|
||||
}
|
||||
|
||||
return d.validateSizes()
|
||||
}
|
||||
|
||||
// validateRoles checks the ext4 role assignment. Roles are the only way to use
|
||||
// more than one disk without ZFS, so a multi-disk selection without them is
|
||||
// rejected rather than quietly installed onto the first drive.
|
||||
func (d DiskConfig) validateRoles() error {
|
||||
if d.FS.IsZFS() {
|
||||
if len(d.Roles) > 0 {
|
||||
return fmt.Errorf("%s: drive roles apply to ext4 only — a pool spans every member already", d.FS.Label())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if len(d.Roles) == 0 {
|
||||
if len(d.Disks) > 1 {
|
||||
return fmt.Errorf("%s: %d disks selected but no roles assigned — assign os, spinifex and predastore, or choose a zfs mode",
|
||||
d.FS.Label(), len(d.Disks))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if len(d.Roles) > len(AllDiskRoles) {
|
||||
return fmt.Errorf("%s: at most %d disks, %d selected — the roles are os, spinifex and predastore",
|
||||
d.FS.Label(), len(AllDiskRoles), len(d.Roles))
|
||||
}
|
||||
|
||||
seen := map[DiskRole]bool{}
|
||||
for _, rm := range d.Roles {
|
||||
if _, ok := roleMountpoints[rm.Role]; !ok {
|
||||
return fmt.Errorf("%s has no role assigned — every selected disk needs one", rm.Disk.Path)
|
||||
}
|
||||
if seen[rm.Role] {
|
||||
return fmt.Errorf("two disks are assigned the %s role", rm.Role)
|
||||
}
|
||||
seen[rm.Role] = true
|
||||
}
|
||||
if !seen[RoleOS] {
|
||||
return fmt.Errorf("%s: no disk assigned the os role — one disk must hold the operating system", d.FS.Label())
|
||||
}
|
||||
if len(d.Roles) != len(d.Disks) {
|
||||
return fmt.Errorf("internal: %d roles for %d disks", len(d.Roles), len(d.Disks))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// minRootBytes matches Proxmox's hard floor for a root disk.
|
||||
const minRootBytes = 2 << 30
|
||||
|
||||
// validateSizes enforces the same-size rule where the topology needs it:
|
||||
// across all members for mirrors and raidz, and per-pair for RAID10 — a
|
||||
// striped mirror only requires each pair to match, not every disk in the pool.
|
||||
func (d DiskConfig) validateSizes() error {
|
||||
if !d.FS.Redundant() {
|
||||
return nil
|
||||
}
|
||||
if d.FS == FSZFSRAID10 {
|
||||
for i := 0; i+1 < len(d.Disks); i += 2 {
|
||||
if err := sizesMatch(d.Disks[i], d.Disks[i+1]); err != nil {
|
||||
return fmt.Errorf("%s mirror pair %d: %w", d.FS.Label(), i/2+1, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
for _, disk := range d.Disks[1:] {
|
||||
if err := sizesMatch(d.Disks[0], disk); err != nil {
|
||||
return fmt.Errorf("%s: %w", d.FS.Label(), err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SizesWithinTolerance reports whether two disks could be members of the same
|
||||
// redundant pool. The UI uses it to decide whether suggesting ZFS is honest.
|
||||
func SizesWithinTolerance(a, b Disk) bool { return sizesMatch(a, b) == nil }
|
||||
|
||||
// sizesMatch reports whether two members are within the tolerance, naming both
|
||||
// disks and their sizes — that message is the operator's only diagnostic.
|
||||
func sizesMatch(a, b Disk) error {
|
||||
if math.Abs(float64(a.Bytes-b.Bytes)) > float64(a.Bytes)*sizeTolerance {
|
||||
return fmt.Errorf("disks must be the same size within %.0f%%: %s is %s, %s is %s",
|
||||
sizeTolerance*100, a.Path, a.SizeHuman(), b.Path, b.SizeHuman())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Warnings returns non-blocking advice for a valid selection. A RAID0 stripe
|
||||
// over mismatched disks is legal but almost always a mistake, so it is
|
||||
// surfaced without preventing the install — which is what Proxmox does.
|
||||
func (d DiskConfig) Warnings() []string {
|
||||
var out []string
|
||||
if d.FS == FSZFSRAID0 && len(d.Disks) > 1 {
|
||||
for _, disk := range d.Disks[1:] {
|
||||
if sizesMatch(d.Disks[0], disk) != nil {
|
||||
out = append(out, "striping disks of different sizes — capacity is usable but a failure of any disk loses the pool")
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if d.FS.IsZFS() && d.FS != FSZFSRAID0 && d.Buildable() && d.Tolerated() == 0 {
|
||||
out = append(out, "this topology stores no redundancy")
|
||||
}
|
||||
// A three-drive role layout looks like an array and is not one: each drive
|
||||
// is an independent point of failure, so the node loses whatever sat on it.
|
||||
if len(d.DataMounts()) > 0 {
|
||||
out = append(out, "role-assigned drives store no redundancy — losing any one drive loses what it held")
|
||||
}
|
||||
for _, rm := range d.DataMounts() {
|
||||
if rm.Role == RolePredastore && rm.Disk.Bytes < smallPredastoreBytes {
|
||||
out = append(out, fmt.Sprintf("%s is small for object storage (%s) — predastore will fill quickly",
|
||||
rm.Disk.Path, rm.Disk.SizeHuman()))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// smallPredastoreBytes is where a dedicated object-store drive stops being
|
||||
// worth the drive. Advisory only: the install is legal below it.
|
||||
const smallPredastoreBytes = 64 << 30
|
||||
@@ -1,267 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDiskConfigValidate(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg DiskConfig
|
||||
wantErr string
|
||||
}{
|
||||
{"ext4 on one disk", DiskConfig{FS: FSExt4, Disks: disks(1, 40)}, ""},
|
||||
{"raidz1 on three matching disks", DiskConfig{FS: FSZFSRAIDZ1, Disks: disks(3, 40)}, ""},
|
||||
{"raid10 on four matching disks", DiskConfig{FS: FSZFSRAID10, Disks: disks(4, 40)}, ""},
|
||||
// The single-drive ZFS case: no redundancy, but checksums, compression
|
||||
// and snapshots still apply.
|
||||
{"raid0 on one disk", DiskConfig{FS: FSZFSRAID0, Disks: disks(1, 40)}, ""},
|
||||
|
||||
{"no filesystem chosen", DiskConfig{Disks: disks(1, 40)}, "no filesystem selected"},
|
||||
{"no disks", DiskConfig{FS: FSZFSRAIDZ1}, "select at least one disk"},
|
||||
{"ext4 spans disks only with roles", DiskConfig{FS: FSExt4, Disks: disks(2, 40)}, "no roles assigned"},
|
||||
{"raidz1 needs three", DiskConfig{FS: FSZFSRAIDZ1, Disks: disks(2, 40)}, "at least 3 disks"},
|
||||
{"raidz2 needs four", DiskConfig{FS: FSZFSRAIDZ2, Disks: disks(3, 40)}, "at least 4 disks"},
|
||||
{"raidz3 needs five", DiskConfig{FS: FSZFSRAIDZ3, Disks: disks(4, 40)}, "at least 5 disks"},
|
||||
{"raid1 needs two", DiskConfig{FS: FSZFSRAID1, Disks: disks(1, 40)}, "at least 2 disks"},
|
||||
{"raid10 rejects an odd count", DiskConfig{FS: FSZFSRAID10, Disks: disks(5, 40)}, "even number"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := tt.cfg.Validate()
|
||||
switch {
|
||||
case tt.wantErr == "" && err != nil:
|
||||
t.Fatalf("Validate() = %v, want nil", err)
|
||||
case tt.wantErr != "" && err == nil:
|
||||
t.Fatalf("Validate() = nil, want error containing %q", tt.wantErr)
|
||||
case tt.wantErr != "" && !strings.Contains(err.Error(), tt.wantErr):
|
||||
t.Fatalf("Validate() = %v, want error containing %q", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsDuplicateDisk(t *testing.T) {
|
||||
d := disk("a", 40)
|
||||
err := DiskConfig{FS: FSZFSRAID1, Disks: []Disk{d, d}}.Validate()
|
||||
if err == nil || !strings.Contains(err.Error(), "more than once") {
|
||||
t.Fatalf("Validate() = %v, want a duplicate-selection error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsLiveMedia(t *testing.T) {
|
||||
d := disk("a", 40)
|
||||
d.LiveMedia = true
|
||||
err := DiskConfig{FS: FSExt4, Disks: []Disk{d}}.Validate()
|
||||
if err == nil || !strings.Contains(err.Error(), "boot media") {
|
||||
t.Fatalf("Validate() = %v, want the installer's own media to be refused", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsTooSmallDisk(t *testing.T) {
|
||||
err := DiskConfig{FS: FSExt4, Disks: []Disk{disk("a", 1)}}.Validate()
|
||||
if err == nil || !strings.Contains(err.Error(), "too small") {
|
||||
t.Fatalf("Validate() = %v, want a minimum-size error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A pool truncates every member to the smallest, so mismatched disks in a
|
||||
// redundant topology silently throw capacity away.
|
||||
func TestValidateSizeTolerance(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
fs FSType
|
||||
sizes []int64
|
||||
wantErr bool
|
||||
}{
|
||||
{"identical disks", FSZFSRAIDZ1, []int64{40, 40, 40}, false},
|
||||
{"within 10%", FSZFSRAIDZ1, []int64{100, 95, 92}, false},
|
||||
{"beyond 10%", FSZFSRAIDZ1, []int64{100, 100, 80}, true},
|
||||
{"mirror with a mismatched member", FSZFSRAID1, []int64{100, 50}, true},
|
||||
// A stripe has no parity to waste, so mismatched sizes are a warning
|
||||
// rather than a refusal — matching Proxmox.
|
||||
{"raid0 permits mismatched disks", FSZFSRAID0, []int64{100, 40}, false},
|
||||
{"raid10 checks each pair, not the whole pool", FSZFSRAID10, []int64{40, 40, 100, 100}, false},
|
||||
{"raid10 rejects a mismatched pair", FSZFSRAID10, []int64{40, 100, 100, 100}, true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
members := make([]Disk, len(tt.sizes))
|
||||
for i, gib := range tt.sizes {
|
||||
members[i] = disk(string(rune('a'+i)), gib)
|
||||
}
|
||||
err := DiskConfig{FS: tt.fs, Disks: members}.Validate()
|
||||
if tt.wantErr && err == nil {
|
||||
t.Fatalf("Validate() = nil, want a size-mismatch error")
|
||||
}
|
||||
if !tt.wantErr && err != nil {
|
||||
t.Fatalf("Validate() = %v, want nil", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWarningsFlagMismatchedStripe(t *testing.T) {
|
||||
cfg := DiskConfig{FS: FSZFSRAID0, Disks: []Disk{disk("a", 100), disk("b", 40)}}
|
||||
warnings := cfg.Warnings()
|
||||
if len(warnings) == 0 {
|
||||
t.Fatal("a stripe over mismatched disks should warn")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUsableBytes(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
fs FSType
|
||||
n int
|
||||
wantGiB int64
|
||||
}{
|
||||
{"stripe adds every disk", FSZFSRAID0, 3, 120},
|
||||
{"mirror gives one disk", FSZFSRAID1, 2, 40},
|
||||
{"raid10 gives half", FSZFSRAID10, 4, 80},
|
||||
{"raidz1 loses one to parity", FSZFSRAIDZ1, 3, 80},
|
||||
{"raidz2 loses two", FSZFSRAIDZ2, 4, 80},
|
||||
{"raidz3 loses three", FSZFSRAIDZ3, 5, 80},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := DiskConfig{FS: tt.fs, Disks: disks(tt.n, 40)}
|
||||
if got := cfg.UsableBytes() >> 30; got != tt.wantGiB {
|
||||
t.Errorf("UsableBytes() = %dG, want %dG", got, tt.wantGiB)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The parity arithmetic goes to zero and then negative as members run out, and
|
||||
// a preview reading "-40GiB usable" is worse than no figure at all.
|
||||
func TestUsableBytesIsZeroWhenTopologyCannotBeBuilt(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
fs FSType
|
||||
n int
|
||||
}{
|
||||
{"raidz3 with exactly its parity count", FSZFSRAIDZ3, 3},
|
||||
{"raidz3 with fewer disks than parity", FSZFSRAIDZ3, 2},
|
||||
{"raidz2 with two disks", FSZFSRAIDZ2, 2},
|
||||
{"raidz1 with one disk", FSZFSRAIDZ1, 1},
|
||||
{"mirror with one disk", FSZFSRAID1, 1},
|
||||
{"raid10 with an odd count", FSZFSRAID10, 3},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := DiskConfig{FS: tt.fs, Disks: disks(tt.n, 40)}
|
||||
if cfg.Buildable() {
|
||||
t.Fatalf("Buildable() = true for %s across %d disks", tt.fs, tt.n)
|
||||
}
|
||||
if got := cfg.UsableBytes(); got != 0 {
|
||||
t.Errorf("UsableBytes() = %d, want 0", got)
|
||||
}
|
||||
if got := cfg.Tolerated(); got != 0 {
|
||||
t.Errorf("Tolerated() = %d, want 0", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A selection that cannot be built must not also be described as unredundant:
|
||||
// the operator's problem is the disk count, not the topology.
|
||||
func TestWarningsStaySilentUntilBuildable(t *testing.T) {
|
||||
cfg := DiskConfig{FS: FSZFSRAIDZ3, Disks: disks(3, 40)}
|
||||
if got := cfg.Warnings(); len(got) != 0 {
|
||||
t.Errorf("Warnings() = %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRequirementNamesTheSameSizeRule(t *testing.T) {
|
||||
if got := (DiskConfig{FS: FSZFSRAIDZ2}).Requirement(); !strings.Contains(got, "4 disks") ||
|
||||
!strings.Contains(got, "same size") {
|
||||
t.Errorf("Requirement() = %q, want the disk count and the same-size rule", got)
|
||||
}
|
||||
if got := (DiskConfig{FS: FSZFSRAID10}).Requirement(); !strings.Contains(got, "even") {
|
||||
t.Errorf("Requirement() = %q, want the even-count rule", got)
|
||||
}
|
||||
// A stripe has no matching requirement, so claiming one would be wrong.
|
||||
if got := (DiskConfig{FS: FSZFSRAID0}).Requirement(); strings.Contains(got, "same size") {
|
||||
t.Errorf("Requirement() = %q, want no same-size rule for a stripe", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTolerated(t *testing.T) {
|
||||
tests := []struct {
|
||||
fs FSType
|
||||
n int
|
||||
want int
|
||||
}{
|
||||
{FSExt4, 1, 0},
|
||||
{FSZFSRAID0, 3, 0},
|
||||
{FSZFSRAID1, 2, 1},
|
||||
{FSZFSRAID1, 3, 2},
|
||||
{FSZFSRAID10, 4, 1},
|
||||
{FSZFSRAIDZ1, 3, 1},
|
||||
{FSZFSRAIDZ2, 4, 2},
|
||||
{FSZFSRAIDZ3, 5, 3},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(string(tt.fs), func(t *testing.T) {
|
||||
cfg := DiskConfig{FS: tt.fs, Disks: disks(tt.n, 40)}
|
||||
if got := cfg.Tolerated(); got != tt.want {
|
||||
t.Errorf("Tolerated() = %d, want %d", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseFSType(t *testing.T) {
|
||||
for _, fs := range AllFSTypes {
|
||||
got, err := ParseFSType(strings.ToUpper(string(fs)))
|
||||
if err != nil {
|
||||
t.Errorf("ParseFSType(%q): %v", fs, err)
|
||||
continue
|
||||
}
|
||||
if got != fs {
|
||||
t.Errorf("ParseFSType(%q) = %q", fs, got)
|
||||
}
|
||||
}
|
||||
// The error has to list the valid values: it is read off a serial console
|
||||
// by someone whose unattended install just stopped.
|
||||
_, err := ParseFSType("btrfs")
|
||||
if err == nil || !strings.Contains(err.Error(), "zfs-raidz1") {
|
||||
t.Fatalf("ParseFSType(btrfs) = %v, want an error listing the accepted values", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPartitionPathSeparator(t *testing.T) {
|
||||
tests := []struct{ dev, want string }{
|
||||
{"/dev/sda", "/dev/sda3"},
|
||||
// A trailing digit needs a 'p' or the name is ambiguous.
|
||||
{"/dev/nvme0n1", "/dev/nvme0n1p3"},
|
||||
{"/dev/vda", "/dev/vda3"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := (Disk{Path: tt.dev}).PartitionPath(3); got != tt.want {
|
||||
t.Errorf("PartitionPath(%s) = %s, want %s", tt.dev, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// udev names partition links with a -partN suffix regardless of the kernel's
|
||||
// separator, so the by-id form must not reuse the 'p' rule.
|
||||
func TestStablePartitionPathUsesPartSuffix(t *testing.T) {
|
||||
d := Disk{Path: "/dev/nvme0n1", ByID: "/dev/disk/by-id/nvme-SAMSUNG_1234"}
|
||||
if got := d.StablePartitionPath(3); got != "/dev/disk/by-id/nvme-SAMSUNG_1234-part3" {
|
||||
t.Errorf("StablePartitionPath() = %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestESPSizeMiB(t *testing.T) {
|
||||
if got := espSizeMiB(disk("a", 40)); got != espSmallMiB {
|
||||
t.Errorf("espSizeMiB(40G) = %d, want %d", got, espSmallMiB)
|
||||
}
|
||||
if got := espSizeMiB(disk("a", 500)); got != espLargeMiB {
|
||||
t.Errorf("espSizeMiB(500G) = %d, want %d", got, espLargeMiB)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,27 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestToFirstbootConfigIncludesInstallCallback(t *testing.T) {
|
||||
const url = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
|
||||
t.Setenv("SPINIFEX_INSTALL_CALLBACK", url)
|
||||
|
||||
cfg := &Config{Hostname: "node1"}
|
||||
fb := cfg.toFirstbootConfig()
|
||||
if fb.InstallCallback != url {
|
||||
t.Errorf("InstallCallback = %q, want %q", fb.InstallCallback, url)
|
||||
}
|
||||
}
|
||||
|
||||
func TestToFirstbootConfigEmptyCallbackWhenEnvUnset(t *testing.T) {
|
||||
os.Unsetenv("SPINIFEX_INSTALL_CALLBACK")
|
||||
|
||||
cfg := &Config{Hostname: "node1"}
|
||||
fb := cfg.toFirstbootConfig()
|
||||
if fb.InstallCallback != "" {
|
||||
t.Errorf("InstallCallback = %q, want empty", fb.InstallCallback)
|
||||
}
|
||||
}
|
||||
@@ -1,391 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// roleCfg builds an ext4 configuration from role/disk-name pairs, in whatever
|
||||
// order they are given — WithRoles is responsible for normalising them.
|
||||
func roleCfg(pairs ...any) DiskConfig {
|
||||
var roles []RoleMount
|
||||
for i := 0; i+1 < len(pairs); i += 2 {
|
||||
roles = append(roles, RoleMount{
|
||||
Role: pairs[i].(DiskRole),
|
||||
Disk: disk(pairs[i+1].(string), 100),
|
||||
})
|
||||
}
|
||||
return DiskConfig{FS: FSExt4}.WithRoles(roles)
|
||||
}
|
||||
|
||||
// fakeUUIDs makes partUUID return a value derived from the device path, so a
|
||||
// generated fstab can be checked without touching a real disk.
|
||||
func fakeUUIDs(t *testing.T) {
|
||||
t.Helper()
|
||||
prev := partUUID
|
||||
partUUID = func(dev string) (string, error) { return "uuid" + strings.ReplaceAll(dev, "/", "-"), nil }
|
||||
t.Cleanup(func() { partUUID = prev })
|
||||
}
|
||||
|
||||
func TestWithRolesNormalisesOrderAndDerivesDisks(t *testing.T) {
|
||||
// Given deepest-first, which is the order that breaks mounting.
|
||||
cfg := roleCfg(RolePredastore, "c", RoleSpinifex, "b", RoleOS, "a")
|
||||
|
||||
wantRoles := []DiskRole{RoleOS, RoleSpinifex, RolePredastore}
|
||||
for i, rm := range cfg.Roles {
|
||||
if rm.Role != wantRoles[i] {
|
||||
t.Fatalf("role %d = %s, want %s", i, rm.Role, wantRoles[i])
|
||||
}
|
||||
}
|
||||
// Disks is derived, so the OS drive is first and every role drive appears
|
||||
// exactly once in the list the installer erases.
|
||||
if got := cfg.Paths(); !slices.Equal(got, []string{"/dev/a", "/dev/b", "/dev/c"}) {
|
||||
t.Fatalf("Paths() = %v, want the roles in canonical order", got)
|
||||
}
|
||||
if cfg.Primary().Path != "/dev/a" {
|
||||
t.Errorf("Primary() = %s, want the os drive", cfg.Primary().Path)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDataMountsExcludeTheOSDriveAndNestShallowestFirst(t *testing.T) {
|
||||
got := roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c").DataMounts()
|
||||
want := []string{"/var/lib/spinifex", "/var/lib/spinifex/predastore"}
|
||||
for i, rm := range got {
|
||||
if rm.Mountpoint() != want[i] {
|
||||
t.Fatalf("mount %d = %s, want %s", i, rm.Mountpoint(), want[i])
|
||||
}
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %d data mounts, want %d", len(got), len(want))
|
||||
}
|
||||
}
|
||||
|
||||
// A predastore drive without a spinifex drive is the two-drive layout, and it
|
||||
// nests a mount inside the root filesystem rather than inside another mount.
|
||||
func TestPredastoreRoleWithoutSpinifexIsValid(t *testing.T) {
|
||||
cfg := roleCfg(RoleOS, "a", RolePredastore, "b")
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("Validate() = %v, want nil", err)
|
||||
}
|
||||
if got := cfg.DataMounts(); len(got) != 1 || got[0].Mountpoint() != "/var/lib/spinifex/predastore" {
|
||||
t.Fatalf("DataMounts() = %v, want predastore only", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRoles(t *testing.T) {
|
||||
twoRoles := func(a, b DiskRole) DiskConfig { return roleCfg(a, "a", b, "b") }
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg DiskConfig
|
||||
wantErr string
|
||||
}{
|
||||
{"single disk needs no roles", DiskConfig{FS: FSExt4, Disks: disks(1, 40)}, ""},
|
||||
{"os alone", roleCfg(RoleOS, "a"), ""},
|
||||
{"all three", roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c"), ""},
|
||||
{"no os role", twoRoles(RoleSpinifex, RolePredastore), "no disk assigned the os role"},
|
||||
{"duplicate role", twoRoles(RoleOS, RoleOS), "two disks are assigned the os role"},
|
||||
{"unknown role", roleCfg(RoleOS, "a", DiskRole("cache"), "b"), "no role assigned"},
|
||||
{"more disks than roles", roleCfg(
|
||||
RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c", DiskRole("cache"), "d"), "at most 3 disks"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := tt.cfg.Validate()
|
||||
switch {
|
||||
case tt.wantErr == "" && err != nil:
|
||||
t.Fatalf("Validate() = %v, want nil", err)
|
||||
case tt.wantErr != "" && (err == nil || !strings.Contains(err.Error(), tt.wantErr)):
|
||||
t.Fatalf("Validate() = %v, want error containing %q", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Roles describe where a workload lives on a plain filesystem. A pool already
|
||||
// spans every member, so accepting them there would silently do nothing.
|
||||
func TestValidateRejectsRolesOnZFS(t *testing.T) {
|
||||
cfg := DiskConfig{FS: FSZFSRAID1, Disks: disks(2, 40)}
|
||||
cfg.Roles = []RoleMount{{Role: RoleOS, Disk: cfg.Disks[0]}}
|
||||
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "ext4 only") {
|
||||
t.Fatalf("Validate() = %v, want a roles-are-ext4-only error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBootDisksAreOnlyTheOSDriveOnExt4(t *testing.T) {
|
||||
cfg := roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c")
|
||||
if got := cfg.bootDisks(); len(got) != 1 || got[0].Path != "/dev/a" {
|
||||
t.Fatalf("bootDisks() = %v, want the os drive only", got)
|
||||
}
|
||||
// Every ZFS member carries its own ESP, so all of them stay boot disks.
|
||||
zfs := DiskConfig{FS: FSZFSRAID1, Disks: disks(2, 40)}
|
||||
if got := zfs.bootDisks(); len(got) != 2 {
|
||||
t.Fatalf("bootDisks() = %v, want every pool member", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteFstabEmitsRoleMountsShallowestFirst(t *testing.T) {
|
||||
root := withMountRoot(t)
|
||||
targetSkeleton(t, root)
|
||||
fakeUUIDs(t)
|
||||
|
||||
cfg := roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c")
|
||||
if err := writeFstab(cfg); err != nil {
|
||||
t.Fatalf("writeFstab: %v", err)
|
||||
}
|
||||
got := readFile(t, root, "etc/fstab")
|
||||
|
||||
// Order matters: `mount -a` in a rescue shell walks the file top-down, and
|
||||
// predastore cannot be mounted before the filesystem it sits inside.
|
||||
wantOrder := []string{" / ", "/boot/efi", " /var/lib/spinifex ", " /var/lib/spinifex/predastore "}
|
||||
at := -1
|
||||
for _, want := range wantOrder {
|
||||
i := strings.Index(got, want)
|
||||
if i < 0 {
|
||||
t.Fatalf("fstab is missing %q:\n%s", want, got)
|
||||
}
|
||||
if i < at {
|
||||
t.Fatalf("%q is out of order:\n%s", want, got)
|
||||
}
|
||||
at = i
|
||||
}
|
||||
|
||||
for _, dev := range []string{"/dev/b1", "/dev/c1"} {
|
||||
if want := "UUID=uuid-dev-" + strings.TrimPrefix(dev, "/dev/"); !strings.Contains(got, want) {
|
||||
t.Errorf("fstab does not mount %s by UUID:\n%s", dev, got)
|
||||
}
|
||||
}
|
||||
// Data mounts are fsck'd after root, never alongside it.
|
||||
for line := range strings.SplitSeq(got, "\n") {
|
||||
if strings.Contains(line, "/var/lib/spinifex") && !strings.HasSuffix(line, "0 2") {
|
||||
t.Errorf("data mount should be fsck pass 2: %q", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fail-closed by design: a missing data drive must stop the boot for an admin,
|
||||
// not let predastore write objects onto the OS drive while everything reports
|
||||
// healthy. nofail would silently produce exactly that.
|
||||
func TestWriteFstabNeverMarksRoleMountsNofail(t *testing.T) {
|
||||
root := withMountRoot(t)
|
||||
targetSkeleton(t, root)
|
||||
fakeUUIDs(t)
|
||||
|
||||
if err := writeFstab(roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c")); err != nil {
|
||||
t.Fatalf("writeFstab: %v", err)
|
||||
}
|
||||
if got := readFile(t, root, "etc/fstab"); strings.Contains(got, "nofail") {
|
||||
t.Errorf("role mounts must not be nofail:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteFstabSingleDiskIsUnchanged(t *testing.T) {
|
||||
root := withMountRoot(t)
|
||||
targetSkeleton(t, root)
|
||||
fakeUUIDs(t)
|
||||
|
||||
if err := writeFstab(roleCfg(RoleOS, "a")); err != nil {
|
||||
t.Fatalf("writeFstab: %v", err)
|
||||
}
|
||||
if got := readFile(t, root, "etc/fstab"); strings.Contains(got, "/var/lib/spinifex") {
|
||||
t.Errorf("a single-disk install should mount nothing extra:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// rsync runs with --delete, so every mountpoint that exists before the copy
|
||||
// needs protecting or the copy can delete across a mount boundary on a retry.
|
||||
func TestMountProtectFiltersCoverEveryRoleMount(t *testing.T) {
|
||||
got := mountProtectFilters(roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c"))
|
||||
want := []string{"--filter=P /var/lib/spinifex", "--filter=P /var/lib/spinifex/predastore"}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Fatalf("mountProtectFilters() = %v, want %v", got, want)
|
||||
}
|
||||
if got := mountProtectFilters(roleCfg(RoleOS, "a")); len(got) != 0 {
|
||||
t.Errorf("a single-disk install has no mounts to protect, got %v", got)
|
||||
}
|
||||
// ZFS keeps its dataset filters — the union is by mode, not additive.
|
||||
if got := mountProtectFilters(DiskConfig{FS: FSZFSRAID1, Disks: disks(2, 40)}); len(got) == 0 {
|
||||
t.Error("zfs mode should still protect its dataset mountpoints")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMountPartitionsMountsRolesAfterRootAndBeforeTheCopy(t *testing.T) {
|
||||
f := fakeCommands(t)
|
||||
root := withMountRoot(t)
|
||||
|
||||
cfg := roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c")
|
||||
if err := mountPartitions(cfg); err != nil {
|
||||
t.Fatalf("mountPartitions: %v", err)
|
||||
}
|
||||
|
||||
// The nested mount cannot precede the one it sits inside, or it is masked.
|
||||
want := []string{
|
||||
"mount /dev/a3 " + root,
|
||||
"mount /dev/a2 " + root + "/boot/efi",
|
||||
"mount /dev/b1 " + root + "/var/lib/spinifex",
|
||||
"mount /dev/c1 " + root + "/var/lib/spinifex/predastore",
|
||||
}
|
||||
var mounts []string
|
||||
for _, c := range f.calls {
|
||||
if strings.HasPrefix(c, "mount ") {
|
||||
mounts = append(mounts, c)
|
||||
}
|
||||
}
|
||||
if !slices.Equal(mounts, want) {
|
||||
t.Fatalf("mount order:\n got %v\nwant %v", mounts, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupTargetUnmountsDeepestFirst(t *testing.T) {
|
||||
f := fakeCommands(t)
|
||||
root := withMountRoot(t)
|
||||
|
||||
cleanupTarget(roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c"))
|
||||
|
||||
// Unmounting the outer filesystem first fails with EBUSY and leaves a
|
||||
// half-mounted target behind for the next attempt.
|
||||
want := []string{
|
||||
"umount " + root + "/var/lib/spinifex/predastore",
|
||||
"umount " + root + "/var/lib/spinifex",
|
||||
"umount " + root + "/boot/efi",
|
||||
"umount " + root,
|
||||
}
|
||||
// The chroot binds are released first and are not part of the target order.
|
||||
var got []string
|
||||
for _, c := range f.calls {
|
||||
if strings.HasPrefix(c, "umount ") && slices.Contains(want, c) {
|
||||
got = append(got, c)
|
||||
}
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Fatalf("unmount order:\n got %v\nwant %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPartitionDisksGivesDataDrivesOnePartitionAndNoESP(t *testing.T) {
|
||||
f := fakeCommands(t)
|
||||
nodes := tempDisks(t, 2, 200)
|
||||
cfg := DiskConfig{FS: FSExt4}.WithRoles(
|
||||
[]RoleMount{{Role: RoleOS, Disk: nodes[0]}, {Role: RoleSpinifex, Disk: nodes[1]}})
|
||||
osDisk, data := nodes[0].Path, nodes[1].Path
|
||||
|
||||
if err := partitionDisks(cfg); err != nil {
|
||||
t.Fatalf("partitionDisks: %v", err)
|
||||
}
|
||||
|
||||
f.mustRun(t, "sgdisk", osDisk, "-n 3:0:-1024M", "-t 3:8300")
|
||||
f.mustRun(t, "sgdisk", osDisk, "-t 2:EF00")
|
||||
// A data drive is never booted from, so a bios_boot or an ESP there would
|
||||
// only mislead whoever reads lsblk during an incident.
|
||||
for _, c := range f.calls {
|
||||
if strings.Contains(c, data) && (strings.Contains(c, "EF02") || strings.Contains(c, "EF00")) {
|
||||
t.Errorf("data drive must get no bios_boot or ESP: %q", c)
|
||||
}
|
||||
}
|
||||
f.mustRun(t, "sgdisk", data, "-n 1:1M:-1024M", "-t 1:8300", "-c 1:spinifex")
|
||||
}
|
||||
|
||||
func TestFormatPartitionsLabelsEachRoleFilesystem(t *testing.T) {
|
||||
f := fakeCommands(t)
|
||||
cfg := roleCfg(RoleOS, "a", RoleSpinifex, "b", RolePredastore, "c")
|
||||
|
||||
if err := formatPartitions(cfg); err != nil {
|
||||
t.Fatalf("formatPartitions: %v", err)
|
||||
}
|
||||
// One ESP only: a data drive holding a bootloader nothing points at would
|
||||
// mislead whoever reads lsblk during an incident.
|
||||
esps := 0
|
||||
for _, c := range f.calls {
|
||||
if strings.HasPrefix(c, "mkfs.fat") {
|
||||
esps++
|
||||
}
|
||||
}
|
||||
if esps != 1 {
|
||||
t.Errorf("want one ESP, got %d:\n%v", esps, f.calls)
|
||||
}
|
||||
f.mustRun(t, "mkfs.ext4", "-L", "spinifex", "/dev/b1")
|
||||
f.mustRun(t, "mkfs.ext4", "-L", "predastore", "/dev/c1")
|
||||
}
|
||||
|
||||
func TestWarningsFlagTheAbsenceOfRedundancy(t *testing.T) {
|
||||
got := strings.Join(roleCfg(RoleOS, "a", RoleSpinifex, "b").Warnings(), "\n")
|
||||
if !strings.Contains(got, "no redundancy") {
|
||||
t.Errorf("a multi-drive role layout must warn it is not an array, got %q", got)
|
||||
}
|
||||
if got := roleCfg(RoleOS, "a").Warnings(); len(got) != 0 {
|
||||
t.Errorf("a single-disk install has nothing to warn about, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWarningsFlagASmallPredastoreDrive(t *testing.T) {
|
||||
cfg := DiskConfig{FS: FSExt4}.WithRoles([]RoleMount{
|
||||
{Role: RoleOS, Disk: disk("a", 500)},
|
||||
{Role: RolePredastore, Disk: disk("b", 8)},
|
||||
})
|
||||
got := strings.Join(cfg.Warnings(), "\n")
|
||||
if !strings.Contains(got, "small for object storage") {
|
||||
t.Errorf("an 8GiB predastore drive should be flagged, got %q", got)
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Errorf("the warning must not block the install: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRolesForDisksAssignsPositionally(t *testing.T) {
|
||||
got := RolesForDisks(disks(3, 40))
|
||||
for i, rm := range got {
|
||||
if want := AllDiskRoles[i]; rm.Role != want {
|
||||
t.Errorf("disk %d got role %s, want %s", i, rm.Role, want)
|
||||
}
|
||||
}
|
||||
// A fourth disk has no role, which Validate is left to reject by name.
|
||||
if extra := RolesForDisks(disks(4, 40)); extra[3].Role != "" {
|
||||
t.Errorf("disk 4 got role %q, want none", extra[3].Role)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFourKnDataDriveIsAllowedWhereABootDriveIsNot(t *testing.T) {
|
||||
if bootedEFI() {
|
||||
t.Skip("machine booted in EFI mode; the 4Kn rule does not apply")
|
||||
}
|
||||
fourKn := func(name string) Disk {
|
||||
d := disk(name, 100)
|
||||
d.LogicalBlockSize = 4096
|
||||
return d
|
||||
}
|
||||
// GRUB's i386-pc target cannot read a 4Kn drive, but nothing boots from a
|
||||
// data drive, so only the OS role is constrained.
|
||||
ok := DiskConfig{FS: FSExt4}.WithRoles(
|
||||
[]RoleMount{{Role: RoleOS, Disk: disk("a", 100)}, {Role: RoleSpinifex, Disk: fourKn("b")}})
|
||||
if err := ok.Validate(); err != nil {
|
||||
t.Errorf("a 4Kn data drive should be accepted: %v", err)
|
||||
}
|
||||
|
||||
bad := DiskConfig{FS: FSExt4}.WithRoles(
|
||||
[]RoleMount{{Role: RoleOS, Disk: fourKn("a")}, {Role: RoleSpinifex, Disk: disk("b", 100)}})
|
||||
if err := bad.Validate(); err == nil || !strings.Contains(err.Error(), "4Kn") {
|
||||
t.Errorf("Validate() = %v, want a 4Kn rejection for the os drive", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyRoleLayoutCatchesAMountThatNeverHappened(t *testing.T) {
|
||||
root := withMountRoot(t)
|
||||
// Plain directories on one filesystem: exactly what a mount performed after
|
||||
// the rootfs copy, or not at all, leaves behind.
|
||||
if err := os.MkdirAll(filepath.Join(root, "var/lib/spinifex/predastore"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
err := verifyRoleLayout(roleCfg(RoleOS, "a", RoleSpinifex, "b"))
|
||||
if err == nil || !strings.Contains(err.Error(), "not a separate filesystem") {
|
||||
t.Fatalf("verifyRoleLayout() = %v, want a missing-mount error", err)
|
||||
}
|
||||
if !strings.Contains(fmt.Sprint(err), "/dev/b") {
|
||||
t.Errorf("the error must name the drive that was not mounted: %v", err)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,373 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"math"
|
||||
"math/bits"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ARC bounds. Spinifex RAM is guest RAM: Proxmox defaults non-PVE products to
|
||||
// half of system memory, which on a 256GB compute node would remove 128GB of
|
||||
// schedulable instance capacity. 10% capped at 16GiB is their PVE behaviour and
|
||||
// the right trade for a hypervisor.
|
||||
const (
|
||||
arcMinMiB = 64
|
||||
arcMaxCapMiB = 16 * 1024
|
||||
arcFraction = 0.10
|
||||
arcHeadroomMi = 1024 // always leave the OS a GiB above the ARC ceiling
|
||||
)
|
||||
|
||||
// zfsDataset describes one dataset created at install time. Order matters:
|
||||
// parents must precede children so mountpoints nest correctly.
|
||||
type zfsDataset struct {
|
||||
name string
|
||||
mountpoint string
|
||||
props []string
|
||||
}
|
||||
|
||||
// datasets is the layout described in the plan. Recordsizes are matched to each
|
||||
// service's access pattern; 16K on the predastore index keeps RAIDZ-1 parity
|
||||
// overhead at 33% rather than the ~50% a 4-8K record would cost.
|
||||
var datasets = []zfsDataset{
|
||||
{ZFSPoolName + "/ROOT", "none", []string{"canmount=off"}},
|
||||
{ZFSRootDataset, "/", []string{"acltype=posix"}},
|
||||
{ZFSPoolName + "/data", "/var/lib/spinifex", []string{"canmount=off"}},
|
||||
{ZFSPoolName + "/data/viperblock", "/var/lib/spinifex/viperblock",
|
||||
[]string{"recordsize=128K", "logbias=throughput"}},
|
||||
{ZFSPoolName + "/data/predastore", "/var/lib/spinifex/predastore",
|
||||
[]string{"recordsize=1M"}},
|
||||
// Broken out so the later move of object data onto dedicated drives is a
|
||||
// zfs send | zfs recv rather than an rsync with downtime.
|
||||
{ZFSPoolName + "/data/predastore-nodes", "/var/lib/spinifex/predastore/distributed/nodes", nil},
|
||||
// BadgerDB index plus the Raft BoltDB: small random, fsync-heavy.
|
||||
{ZFSPoolName + "/data/predastore-db", "/var/lib/spinifex/predastore/distributed/db",
|
||||
[]string{"recordsize=16K", "logbias=latency"}},
|
||||
{ZFSPoolName + "/data/nats", "/var/lib/spinifex/nats", []string{"recordsize=16K"}},
|
||||
{ZFSPoolName + "/log", "/var/log/spinifex", []string{"compression=zstd"}},
|
||||
}
|
||||
|
||||
// buildVdev renders the vdev specification for zpool create. Pure: it is the
|
||||
// function that decides pool shape, so it is exhaustively unit-tested rather
|
||||
// than discovered to be wrong on hardware.
|
||||
func buildVdev(cfg DiskConfig) (string, error) {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !cfg.FS.IsZFS() {
|
||||
return "", fmt.Errorf("buildVdev: %s is not a zfs topology", cfg.FS)
|
||||
}
|
||||
|
||||
// Members are named by their partition-3 by-id path: a pool built on
|
||||
// /dev/sdX breaks the first time a controller enumerates in a new order.
|
||||
members := make([]string, len(cfg.Disks))
|
||||
for i, d := range cfg.Disks {
|
||||
members[i] = d.StablePartitionPath(rootPartNum)
|
||||
}
|
||||
|
||||
keyword := cfg.FS.vdevKeyword()
|
||||
if keyword == "" {
|
||||
return strings.Join(members, " "), nil // plain stripe
|
||||
}
|
||||
// RAID10 is a stripe of two-disk mirrors, paired in selection order — which
|
||||
// is why the UI numbers selected disks instead of just ticking them.
|
||||
if cfg.FS == FSZFSRAID10 {
|
||||
var parts []string
|
||||
for i := 0; i+1 < len(members); i += 2 {
|
||||
parts = append(parts, fmt.Sprintf("mirror %s %s", members[i], members[i+1]))
|
||||
}
|
||||
return strings.Join(parts, " "), nil
|
||||
}
|
||||
return keyword + " " + strings.Join(members, " "), nil
|
||||
}
|
||||
|
||||
// detectAshift derives the pool sector size from the widest sector any member
|
||||
// reports. ashift cannot be changed after creation, and one that is too small
|
||||
// costs read-modify-write on every block for the life of the pool, so the floor
|
||||
// is 12 (4K) even when every disk claims 512.
|
||||
func detectAshift(disks []Disk) int {
|
||||
widest := 4096
|
||||
for _, d := range disks {
|
||||
widest = max(widest, d.PhysicalBlockSize, d.LogicalBlockSize)
|
||||
}
|
||||
// ashift is the base-2 exponent of the sector size.
|
||||
return bits.Len(uint(widest)) - 1
|
||||
}
|
||||
|
||||
// defaultARCMaxMiB computes the ARC ceiling from total system memory.
|
||||
func defaultARCMaxMiB(totalMiB int) int {
|
||||
want := int(math.Round(float64(totalMiB) * arcFraction))
|
||||
want = min(want, arcMaxCapMiB)
|
||||
// Never let the ARC crowd the OS out of its last GiB on a small node.
|
||||
if ceiling := totalMiB - arcHeadroomMi; ceiling > 0 {
|
||||
want = min(want, ceiling)
|
||||
}
|
||||
return max(want, arcMinMiB)
|
||||
}
|
||||
|
||||
// totalMemoryMiB reads MemTotal from /proc/meminfo.
|
||||
func totalMemoryMiB() int {
|
||||
data, err := os.ReadFile("/proc/meminfo")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
for line := range strings.SplitSeq(string(data), "\n") {
|
||||
if !strings.HasPrefix(line, "MemTotal:") {
|
||||
continue
|
||||
}
|
||||
var kb int
|
||||
if _, err := fmt.Sscanf(line, "MemTotal: %d kB", &kb); err == nil {
|
||||
return kb / 1024
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// resolveZFSOpts fills unset advanced options with their computed defaults, so
|
||||
// every value written to the pool is explicit and auditable afterwards.
|
||||
func resolveZFSOpts(cfg DiskConfig) ZFSOpts {
|
||||
o := cfg.ZFS
|
||||
if o.Ashift == 0 {
|
||||
o.Ashift = detectAshift(cfg.Disks)
|
||||
}
|
||||
if o.Compress == "" {
|
||||
// Named explicitly rather than "on" so a future OpenZFS default change
|
||||
// cannot silently alter what this pool does.
|
||||
o.Compress = "lz4"
|
||||
}
|
||||
if o.Checksum == "" {
|
||||
o.Checksum = "on"
|
||||
}
|
||||
if o.Copies == 0 {
|
||||
o.Copies = 1
|
||||
}
|
||||
if o.ARCMaxMiB == 0 {
|
||||
o.ARCMaxMiB = defaultARCMaxMiB(totalMemoryMiB())
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// loadZFSModule brings up the ZFS kernel module in the live environment and
|
||||
// waits for /dev/zfs. The module is prebuilt against the shipped kernel at ISO
|
||||
// build time, so failure here means the ISO is broken, not the hardware.
|
||||
func loadZFSModule() error {
|
||||
if err := run("modprobe", "zfs"); err != nil {
|
||||
slog.Warn("modprobe zfs failed, checking for /dev/zfs anyway", "err", err)
|
||||
}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
if _, err := os.Stat("/dev/zfs"); err == nil {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("/dev/zfs did not appear after modprobe zfs — the ZFS module is missing or does not match the running kernel")
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// ImportablePools returns the names of pools visible to `zpool import`. There
|
||||
// is still no machine-readable output in OpenZFS 2.3, so the human format is
|
||||
// parsed — the same approach Proxmox::Sys::ZFS takes.
|
||||
func ImportablePools() ([]string, error) {
|
||||
out, err := exec.Command("zpool", "import").Output()
|
||||
if err != nil {
|
||||
// zpool exits non-zero when there is nothing to import, which is the
|
||||
// normal case. Anything else — no binary, no /dev/zfs — must surface, or
|
||||
// a machine with an existing pool looks empty to the caller.
|
||||
if _, ok := errors.AsType[*exec.ExitError](err); ok {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, fmt.Errorf("zpool import: %w", err)
|
||||
}
|
||||
return parseImportablePools(string(out)), nil
|
||||
}
|
||||
|
||||
// parseImportablePools extracts pool names from `zpool import` output. Split
|
||||
// out from the command call so it can be tested against captured fixtures.
|
||||
func parseImportablePools(out string) []string {
|
||||
var pools []string
|
||||
scanner := bufio.NewScanner(strings.NewReader(out))
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
// Only the "pool:" key introduces a pool; "id:", "state:" and the
|
||||
// config tree that follows are attributes of the one before it.
|
||||
if name, ok := strings.CutPrefix(line, "pool:"); ok {
|
||||
if n := strings.TrimSpace(name); n != "" {
|
||||
pools = append(pools, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
return pools
|
||||
}
|
||||
|
||||
// createPool builds the root pool across the selected disks. altroot keeps
|
||||
// every dataset mounted under the install target instead of over the live
|
||||
// filesystem, and cachefile=none stops the live environment from claiming it.
|
||||
func createPool(cfg DiskConfig, opts ZFSOpts) error {
|
||||
vdev, err := buildVdev(cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
args := []string{
|
||||
"create", "-f",
|
||||
"-o", "cachefile=none",
|
||||
"-o", fmt.Sprintf("ashift=%d", opts.Ashift),
|
||||
"-O", "atime=on", "-O", "relatime=on",
|
||||
"-O", "xattr=sa",
|
||||
"-O", fmt.Sprintf("compression=%s", opts.Compress),
|
||||
"-O", fmt.Sprintf("checksum=%s", opts.Checksum),
|
||||
"-O", fmt.Sprintf("copies=%d", opts.Copies),
|
||||
"-O", "mountpoint=none",
|
||||
"-R", mountRoot,
|
||||
ZFSPoolName,
|
||||
}
|
||||
args = append(args, strings.Fields(vdev)...)
|
||||
if err := run("zpool", args...); err != nil {
|
||||
return fmt.Errorf("zpool create: %w", err)
|
||||
}
|
||||
|
||||
// TRIM matters on a pool that churns EBS volumes: without it the drives'
|
||||
// garbage collection degrades write latency over months, and diagnosing
|
||||
// that after the fact is miserable. Detected, never prompted.
|
||||
if allFlash(cfg.Disks) {
|
||||
if err := run("zpool", "set", "autotrim=on", ZFSPoolName); err != nil {
|
||||
slog.Warn("could not enable autotrim", "err", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// allFlash reports whether every member is non-rotational.
|
||||
func allFlash(disks []Disk) bool {
|
||||
for _, d := range disks {
|
||||
if d.Rotational {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return len(disks) > 0
|
||||
}
|
||||
|
||||
// createDatasets lays out the service datasets. Mountpoints are set relative to
|
||||
// the pool's altroot, so they land under the install target now and at their
|
||||
// real paths once the installed system imports the pool.
|
||||
func createDatasets() error {
|
||||
for _, ds := range datasets {
|
||||
args := []string{"create", "-o", "mountpoint=" + ds.mountpoint}
|
||||
for _, p := range ds.props {
|
||||
args = append(args, "-o", p)
|
||||
}
|
||||
args = append(args, ds.name)
|
||||
if err := run("zfs", args...); err != nil {
|
||||
return fmt.Errorf("create dataset %s: %w", ds.name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// datasetProtectFilters keeps rsync --delete away from the dataset mountpoints.
|
||||
//
|
||||
// The datasets are mounted before the copy so files land on the right one, but
|
||||
// the live source has nothing at those paths, so --delete tries to rmdir a busy
|
||||
// mountpoint and fails the whole transfer. Protect rules exempt them.
|
||||
func datasetProtectFilters(cfg DiskConfig) []string {
|
||||
if !cfg.FS.IsZFS() {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, ds := range datasets {
|
||||
if ds.mountpoint == "/" || ds.mountpoint == "none" {
|
||||
continue
|
||||
}
|
||||
out = append(out, "--filter=P "+ds.mountpoint)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// writeZFSSystemConfig writes the target's ZFS tunables: the ARC ceiling, and
|
||||
// the matching host reserve so the daemon's admission gate stops handing out
|
||||
// memory the ARC is holding.
|
||||
func writeZFSSystemConfig(opts ZFSOpts) error {
|
||||
modprobeDir := filepath.Join(mountRoot, "etc/modprobe.d")
|
||||
if err := os.MkdirAll(modprobeDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
arcBytes := int64(opts.ARCMaxMiB) << 20
|
||||
conf := fmt.Sprintf("# Generated by the Spinifex installer.\n"+
|
||||
"# Guest RAM is the scarce resource on a hypervisor, so the ARC is\n"+
|
||||
"# capped well below the ZFS default of half of system memory.\n"+
|
||||
"options zfs zfs_arc_max=%d\n", arcBytes)
|
||||
if err := os.WriteFile(filepath.Join(modprobeDir, "zfs.conf"), []byte(conf), 0o644); err != nil {
|
||||
return fmt.Errorf("write zfs.conf: %w", err)
|
||||
}
|
||||
|
||||
// The daemon computes schedulable memory as host total minus this reserve
|
||||
// (defaultHostReserve in daemon/resource.go). ARC is invisible to it, so
|
||||
// the ceiling has to be declared or instances get admitted into memory the
|
||||
// ARC already owns.
|
||||
reserve := defaultHostReserveGB + float64(opts.ARCMaxMiB)/1024.0
|
||||
env := fmt.Sprintf("# Generated by the Spinifex installer.\n"+
|
||||
"# %.1fGB base reserve + %.1fGB ZFS ARC ceiling.\n"+
|
||||
"SPINIFEX_RESERVED_MEM_GB=%.1f\n",
|
||||
defaultHostReserveGB, float64(opts.ARCMaxMiB)/1024.0, reserve)
|
||||
confDir := filepath.Join(mountRoot, "etc/spinifex")
|
||||
if err := os.MkdirAll(confDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(filepath.Join(confDir, "host.env"), []byte(env), 0o644)
|
||||
}
|
||||
|
||||
// defaultHostReserveGB mirrors defaultHostReserve.memGB in
|
||||
// spinifex/daemon/resource.go. Changing one without the other silently
|
||||
// misreports how much memory a ZFS node can schedule.
|
||||
const defaultHostReserveGB = 2.0
|
||||
|
||||
// exportPool releases the pool so a retry after a failed install is not blocked
|
||||
// by "pool is busy". The live environment runs spinifex-init as PID 1, so
|
||||
// nothing else will ever clean this up.
|
||||
func exportPool() {
|
||||
if err := run("zpool", "export", ZFSPoolName); err != nil {
|
||||
slog.Warn("zpool export failed", "pool", ZFSPoolName, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// clearDisk takes a disk over unconditionally, whatever it holds: everything
|
||||
// attached to it is released, every signature on it is wiped, its partition
|
||||
// table is erased, and the kernel is made to agree the disk is now empty.
|
||||
//
|
||||
// The last part is the point. sgdisk writes to the platters and exits 0 even
|
||||
// when the kernel refuses to drop the old table, so without the assertion an
|
||||
// install proceeds against partition nodes describing the previous layout.
|
||||
//
|
||||
// The probes are quiet because a blank disk makes all of them fail, and their
|
||||
// stderr on the install console reads like the install is going wrong.
|
||||
func clearDisk(d Disk) error {
|
||||
releaseDisk(d)
|
||||
|
||||
// Every partition, not just the root slice. ZFS labels live at both ends of
|
||||
// a device and wipefs does not know about them, so a data drive that was
|
||||
// once a pool member keeps its label and fails a later zpool create.
|
||||
parts, err := kernelPartitions(d.Path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, p := range parts {
|
||||
dev := partitionDevice(d.Path, p)
|
||||
_ = runQuiet("zpool", "labelclear", "-f", dev)
|
||||
_ = runQuiet("wipefs", "-a", dev)
|
||||
}
|
||||
|
||||
_ = runQuiet("wipefs", "-a", d.Path)
|
||||
if err := runQuiet("sgdisk", "-Z", d.Path); err != nil {
|
||||
return fmt.Errorf("erase partition table: %w", err)
|
||||
}
|
||||
return settlePartitions(d)
|
||||
}
|
||||
@@ -1,311 +0,0 @@
|
||||
package install
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// disk builds a test disk with a by-id path, since that is what pool members
|
||||
// are actually named by.
|
||||
func disk(name string, gib int64) Disk {
|
||||
return Disk{
|
||||
Path: "/dev/" + name,
|
||||
ByID: "/dev/disk/by-id/virtio-" + name,
|
||||
Bytes: gib << 30,
|
||||
LogicalBlockSize: 512,
|
||||
PhysicalBlockSize: 512,
|
||||
}
|
||||
}
|
||||
|
||||
func disks(n int, gib int64) []Disk {
|
||||
out := make([]Disk, n)
|
||||
for i := range out {
|
||||
out[i] = disk(string(rune('a'+i)), gib)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestBuildVdev(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
fs FSType
|
||||
disks []Disk
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"raid0 is a bare stripe with no keyword",
|
||||
FSZFSRAID0, disks(2, 40),
|
||||
"/dev/disk/by-id/virtio-a-part3 /dev/disk/by-id/virtio-b-part3",
|
||||
},
|
||||
{
|
||||
"single-disk raid0 is the one-drive case",
|
||||
FSZFSRAID0, disks(1, 40),
|
||||
"/dev/disk/by-id/virtio-a-part3",
|
||||
},
|
||||
{
|
||||
"raid1 is one mirror vdev",
|
||||
FSZFSRAID1, disks(2, 40),
|
||||
"mirror /dev/disk/by-id/virtio-a-part3 /dev/disk/by-id/virtio-b-part3",
|
||||
},
|
||||
{
|
||||
"raid1 across three disks is a three-way mirror, not a stripe",
|
||||
FSZFSRAID1, disks(3, 40),
|
||||
"mirror /dev/disk/by-id/virtio-a-part3 /dev/disk/by-id/virtio-b-part3 /dev/disk/by-id/virtio-c-part3",
|
||||
},
|
||||
{
|
||||
"raid10 pairs members in selection order",
|
||||
FSZFSRAID10, disks(4, 40),
|
||||
"mirror /dev/disk/by-id/virtio-a-part3 /dev/disk/by-id/virtio-b-part3 " +
|
||||
"mirror /dev/disk/by-id/virtio-c-part3 /dev/disk/by-id/virtio-d-part3",
|
||||
},
|
||||
{
|
||||
"raidz1 across three disks",
|
||||
FSZFSRAIDZ1, disks(3, 40),
|
||||
"raidz1 /dev/disk/by-id/virtio-a-part3 /dev/disk/by-id/virtio-b-part3 /dev/disk/by-id/virtio-c-part3",
|
||||
},
|
||||
{
|
||||
"raidz2 needs four",
|
||||
FSZFSRAIDZ2, disks(4, 40),
|
||||
"raidz2 /dev/disk/by-id/virtio-a-part3 /dev/disk/by-id/virtio-b-part3 " +
|
||||
"/dev/disk/by-id/virtio-c-part3 /dev/disk/by-id/virtio-d-part3",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := buildVdev(DiskConfig{FS: tt.fs, Disks: tt.disks})
|
||||
if err != nil {
|
||||
t.Fatalf("buildVdev: %v", err)
|
||||
}
|
||||
if got != tt.want {
|
||||
t.Errorf("buildVdev()\n got: %s\nwant: %s", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A pool built on /dev/sdX silently breaks when the controller enumerates in a
|
||||
// different order, so the fallback only applies when there is no by-id path.
|
||||
func TestBuildVdevFallsBackToKernelNameWithoutByID(t *testing.T) {
|
||||
d := disk("a", 40)
|
||||
d.ByID = ""
|
||||
got, err := buildVdev(DiskConfig{FS: FSZFSRAID0, Disks: []Disk{d}})
|
||||
if err != nil {
|
||||
t.Fatalf("buildVdev: %v", err)
|
||||
}
|
||||
if got != "/dev/a3" {
|
||||
t.Errorf("buildVdev() = %q, want /dev/a3", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildVdevRejectsExt4(t *testing.T) {
|
||||
if _, err := buildVdev(DiskConfig{FS: FSExt4, Disks: disks(1, 40)}); err == nil {
|
||||
t.Fatal("buildVdev accepted ext4")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildVdevValidatesFirst(t *testing.T) {
|
||||
// Three disks cannot form striped mirrors, and producing a two-disk mirror
|
||||
// plus a silently dropped disk would be worse than an error.
|
||||
if _, err := buildVdev(DiskConfig{FS: FSZFSRAID10, Disks: disks(3, 40)}); err == nil {
|
||||
t.Fatal("buildVdev accepted an odd disk count for RAID10")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectAshift(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
logical int
|
||||
physical int
|
||||
want int
|
||||
}{
|
||||
// ashift cannot be changed after creation and one that is too small
|
||||
// costs a read-modify-write per block forever, so 512e drives are
|
||||
// deliberately treated as 4K.
|
||||
{"512n drive is floored at 4K", 512, 512, 12},
|
||||
{"512e drive reports 4K physical", 512, 4096, 12},
|
||||
{"4Kn drive", 4096, 4096, 12},
|
||||
{"8K native drive", 8192, 8192, 13},
|
||||
{"16K native drive", 16384, 16384, 14},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
d := Disk{LogicalBlockSize: tt.logical, PhysicalBlockSize: tt.physical}
|
||||
if got := detectAshift([]Disk{d}); got != tt.want {
|
||||
t.Errorf("detectAshift() = %d, want %d", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A pool cannot have two sector sizes, so the widest member wins — the reverse
|
||||
// would make every write to the large-sector disk a read-modify-write.
|
||||
func TestDetectAshiftTakesWidestMember(t *testing.T) {
|
||||
got := detectAshift([]Disk{
|
||||
{LogicalBlockSize: 512, PhysicalBlockSize: 512},
|
||||
{LogicalBlockSize: 512, PhysicalBlockSize: 8192},
|
||||
})
|
||||
if got != 13 {
|
||||
t.Errorf("detectAshift() = %d, want 13", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultARCMaxMiB(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
totalMi int
|
||||
want int
|
||||
}{
|
||||
{"tiny VM gets 10% of very little", 2048, 205},
|
||||
{"8GB node gets 10%", 8192, 819},
|
||||
{"64GB node gets 10%", 65536, 6554},
|
||||
// The point of the cap: guest RAM is what this machine sells, and half
|
||||
// of a 256GB node is 128GB of instance capacity the ARC would swallow.
|
||||
{"256GB node is capped at 16GiB", 262144, 16384},
|
||||
{"1TB node is still capped at 16GiB", 1048576, 16384},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := defaultARCMaxMiB(tt.totalMi); got != tt.want {
|
||||
t.Errorf("defaultARCMaxMiB(%d) = %d, want %d", tt.totalMi, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// An unreadable /proc/meminfo must not produce a zero or negative ceiling,
|
||||
// which ZFS would either reject or read as "no limit".
|
||||
func TestDefaultARCMaxMiBNeverGoesBelowFloor(t *testing.T) {
|
||||
for _, total := range []int{0, 128, 512} {
|
||||
if got := defaultARCMaxMiB(total); got < arcMinMiB {
|
||||
t.Errorf("defaultARCMaxMiB(%d) = %d, below the floor of %d", total, got, arcMinMiB)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveZFSOptsFillsDefaults(t *testing.T) {
|
||||
got := resolveZFSOpts(DiskConfig{FS: FSZFSRAIDZ1, Disks: disks(3, 40)})
|
||||
if got.Ashift != 12 {
|
||||
t.Errorf("Ashift = %d, want 12", got.Ashift)
|
||||
}
|
||||
if got.Compress != "lz4" {
|
||||
t.Errorf("Compress = %q, want lz4", got.Compress)
|
||||
}
|
||||
if got.Checksum != "on" {
|
||||
t.Errorf("Checksum = %q, want on", got.Checksum)
|
||||
}
|
||||
if got.Copies != 1 {
|
||||
t.Errorf("Copies = %d, want 1", got.Copies)
|
||||
}
|
||||
if got.ARCMaxMiB <= 0 {
|
||||
t.Errorf("ARCMaxMiB = %d, want a positive ceiling", got.ARCMaxMiB)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveZFSOptsKeepsOperatorChoices(t *testing.T) {
|
||||
in := DiskConfig{
|
||||
FS: FSZFSRAIDZ1,
|
||||
Disks: disks(3, 40),
|
||||
ZFS: ZFSOpts{Ashift: 13, Compress: "zstd", Checksum: "blake3", Copies: 2, ARCMaxMiB: 2048},
|
||||
}
|
||||
got := resolveZFSOpts(in)
|
||||
if got != in.ZFS {
|
||||
t.Errorf("resolveZFSOpts overrode explicit options: got %+v, want %+v", got, in.ZFS)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseImportablePools(t *testing.T) {
|
||||
// Captured from `zpool import` on a machine with a pool from a previous
|
||||
// install: the config tree below each header must not be mistaken for one.
|
||||
const out = ` pool: rpool
|
||||
id: 8543219876543210987
|
||||
state: ONLINE
|
||||
action: The pool can be imported using its name or numeric identifier.
|
||||
config:
|
||||
|
||||
rpool ONLINE
|
||||
raidz1-0 ONLINE
|
||||
virtio-spinifex-disk1-part3 ONLINE
|
||||
virtio-spinifex-disk2-part3 ONLINE
|
||||
|
||||
pool: tank
|
||||
id: 1111111111111111111
|
||||
state: ONLINE
|
||||
config:
|
||||
|
||||
tank ONLINE
|
||||
`
|
||||
got := parseImportablePools(out)
|
||||
want := []string{"rpool", "tank"}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("parseImportablePools() = %v, want %v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("pool %d = %q, want %q", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseImportablePoolsEmpty(t *testing.T) {
|
||||
if got := parseImportablePools("no pools available to import\n"); len(got) != 0 {
|
||||
t.Errorf("parseImportablePools() = %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Every dataset must be creatable in order: a child cannot be created before
|
||||
// its parent, and getting this wrong only shows up on real hardware.
|
||||
func TestDatasetsAreOrderedParentFirst(t *testing.T) {
|
||||
seen := map[string]bool{ZFSPoolName: true}
|
||||
for _, ds := range datasets {
|
||||
parent := ds.name[:strings.LastIndex(ds.name, "/")]
|
||||
if !seen[parent] {
|
||||
t.Errorf("dataset %s is created before its parent %s", ds.name, parent)
|
||||
}
|
||||
seen[ds.name] = true
|
||||
}
|
||||
}
|
||||
|
||||
// The root dataset has to be the one the bootloader names, or the machine
|
||||
// mounts an empty filesystem and panics.
|
||||
func TestRootDatasetIsCreated(t *testing.T) {
|
||||
for _, ds := range datasets {
|
||||
if ds.name == ZFSRootDataset {
|
||||
if ds.mountpoint != "/" {
|
||||
t.Fatalf("%s mountpoint = %q, want /", ZFSRootDataset, ds.mountpoint)
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatalf("%s is never created", ZFSRootDataset)
|
||||
}
|
||||
|
||||
// rsync --delete tries to rmdir a dataset mountpoint the live source does not
|
||||
// have, which fails on a busy mount and aborts the whole copy. Every dataset
|
||||
// with a real mountpoint must be protected.
|
||||
func TestDatasetProtectFilters(t *testing.T) {
|
||||
got := datasetProtectFilters(DiskConfig{FS: FSZFSRAIDZ1, Disks: disks(3, 40)})
|
||||
for _, ds := range datasets {
|
||||
if ds.mountpoint == "/" || ds.mountpoint == "none" {
|
||||
continue
|
||||
}
|
||||
want := "--filter=P " + ds.mountpoint
|
||||
if !slices.Contains(got, want) {
|
||||
t.Errorf("missing protect rule for %s (%s)", ds.name, ds.mountpoint)
|
||||
}
|
||||
}
|
||||
// The root dataset is the destination itself, and "none" datasets are never
|
||||
// mounted, so protecting either would be meaningless.
|
||||
for _, f := range got {
|
||||
if f == "--filter=P /" || f == "--filter=P none" {
|
||||
t.Errorf("unexpected protect rule %q", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetProtectFiltersSkipsExt4(t *testing.T) {
|
||||
if got := datasetProtectFilters(DiskConfig{FS: FSExt4, Disks: disks(1, 40)}); got != nil {
|
||||
t.Errorf("datasetProtectFilters() = %v on ext4, want none", got)
|
||||
}
|
||||
}
|
||||
@@ -6,9 +6,9 @@ import (
|
||||
"strings"
|
||||
|
||||
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
|
||||
"github.com/mulgadc/spinifex/cmd/installer/autoinstall"
|
||||
"github.com/mulgadc/spinifex/cmd/installer/install"
|
||||
"github.com/mulgadc/spinifex/cmd/installer/ui"
|
||||
"github.com/mulgadc/spinifex/spinifex/operator/installer/autoinstall"
|
||||
"github.com/mulgadc/spinifex/spinifex/operator/installer/install"
|
||||
"github.com/mulgadc/spinifex/spinifex/operator/installer/ui"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
@@ -1,426 +0,0 @@
|
||||
// Package netprobe identifies physical network interfaces by hardware rather
|
||||
// than kernel name, so an operator choosing between eno1, ens1f0np0 and
|
||||
// ens1f1np1 can tell the 1gbe Broadcom from the 25gbe Mellanox.
|
||||
package netprobe
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// sysClassNet is the kernel's per-interface attribute tree. A variable, not a
|
||||
// constant, so tests can point the probe at a fixture instead of the running
|
||||
// machine's hardware.
|
||||
var sysClassNet = "/sys/class/net"
|
||||
|
||||
// The external helpers the probe leans on, indirected so tests can exercise it
|
||||
// without the host's real tooling.
|
||||
var (
|
||||
listInterfaces = net.Interfaces
|
||||
commandOutput = func(name string, args ...string) ([]byte, error) {
|
||||
return exec.Command(name, args...).Output()
|
||||
}
|
||||
)
|
||||
|
||||
// NIC describes one physical interface as presented in the installer's
|
||||
// selection table.
|
||||
type NIC struct {
|
||||
Name string
|
||||
|
||||
// Vendor and Model come from the udev hardware database, e.g. "Mellanox
|
||||
// Technologies" and "MT27710 Family [ConnectX-4 Lx]". Empty when the
|
||||
// database carries no entry for the card.
|
||||
Vendor string
|
||||
Model string
|
||||
|
||||
// Driver is the kernel module that claimed the port, e.g. "mlx5_core". It
|
||||
// comes straight from sysfs, so it still identifies a card the hardware
|
||||
// database misses.
|
||||
Driver string
|
||||
|
||||
// DeviceID is the PCI or USB "vendor:device" pair, the last-resort identity
|
||||
// for hardware neither the database nor a driver name pins down.
|
||||
DeviceID string
|
||||
|
||||
// AltName is the first alternative name systemd assigns, which is what
|
||||
// operators often see in switch and cabling documentation.
|
||||
AltName string
|
||||
|
||||
// Speed is the negotiated link speed rendered for display ("25 Gbps") and
|
||||
// SpeedMbps the raw figure behind it. Both are zero on a port with no
|
||||
// carrier, which is why Probe brings links up before reading.
|
||||
Speed string
|
||||
SpeedMbps int
|
||||
|
||||
// Carrier reports a cable with link. It orders the picker so the port the
|
||||
// operator has just plugged in is the one offered first.
|
||||
Carrier bool
|
||||
|
||||
// State is the link summary shown in the table: "online", "no cable" or
|
||||
// "down".
|
||||
State string
|
||||
|
||||
MTU int
|
||||
IsWiFi bool
|
||||
}
|
||||
|
||||
// Label renders the NIC's identity line: kernel name, negotiated speed and link
|
||||
// state. The hardware detail is deliberately absent — it goes on its own line
|
||||
// via Hardware, where there is room to read it.
|
||||
func (n NIC) Label() string {
|
||||
speed := n.Speed
|
||||
if speed == "" {
|
||||
speed = "—"
|
||||
}
|
||||
state := n.State
|
||||
if state == "" {
|
||||
state = "unknown"
|
||||
}
|
||||
return fmt.Sprintf("%s %s %s", n.Name, speed, state)
|
||||
}
|
||||
|
||||
// Hardware renders the card's identity for the detail line, degrading from the
|
||||
// database name to the driver and device id rather than to "unknown". An
|
||||
// operator needs something to match against the card in their hand.
|
||||
func (n NIC) Hardware() string {
|
||||
if hw := strings.TrimSpace(n.Vendor + " " + n.Model); hw != "" {
|
||||
return hw
|
||||
}
|
||||
switch {
|
||||
case n.Driver != "" && n.DeviceID != "":
|
||||
return n.Driver + " [" + n.DeviceID + "]"
|
||||
case n.Driver != "":
|
||||
return n.Driver
|
||||
case n.DeviceID != "":
|
||||
return n.DeviceID
|
||||
}
|
||||
return "no hardware detail"
|
||||
}
|
||||
|
||||
// linkUpTimeout bounds how long Probe waits after bringing links up for speed
|
||||
// and carrier to settle. A 25gbe port negotiating against a cold switch can
|
||||
// take a second or two to report a speed.
|
||||
var linkUpTimeout = 2 * time.Second
|
||||
|
||||
// Probe enumerates non-loopback physical interfaces and enriches each from
|
||||
// sysfs and the udev hardware database, ordered so the ports worth assigning
|
||||
// come first. Links are brought up first: a down port reports no speed and no
|
||||
// carrier, which would leave every fast port unidentified at exactly the moment
|
||||
// the operator needs to pick one.
|
||||
//
|
||||
// Nothing the probe depends on needs a running systemd. The live installer
|
||||
// boots straight into spinifex-init with only udevd started, so networkctl —
|
||||
// which has to reach systemd-networkd — enriches gaps and is never the source.
|
||||
func Probe() ([]NIC, error) {
|
||||
ifaces, err := listInterfaces()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var names, physical []string
|
||||
for _, iface := range ifaces {
|
||||
if iface.Flags&net.FlagLoopback != 0 {
|
||||
continue
|
||||
}
|
||||
if len(iface.HardwareAddr) == 0 {
|
||||
continue
|
||||
}
|
||||
names = append(names, iface.Name)
|
||||
if hasDevice(iface.Name) {
|
||||
physical = append(physical, iface.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// Bridges, bonds and veth pairs carry a MAC like any port but have no
|
||||
// device behind them, and offering them would let a plane be bound to
|
||||
// something that does not exist until after the install. Keep them only if
|
||||
// the filter found nothing, so an unusual platform still gets a list.
|
||||
if len(physical) > 0 {
|
||||
names = physical
|
||||
}
|
||||
|
||||
brought := false
|
||||
for _, name := range names {
|
||||
if _, err := commandOutput("ip", "link", "set", "dev", name, "up"); err != nil {
|
||||
slog.Debug("netprobe: could not bring link up", "iface", name, "err", err)
|
||||
continue
|
||||
}
|
||||
brought = true
|
||||
}
|
||||
if brought {
|
||||
time.Sleep(linkUpTimeout)
|
||||
}
|
||||
|
||||
nics := make([]NIC, 0, len(names))
|
||||
for _, name := range names {
|
||||
nics = append(nics, describe(name))
|
||||
}
|
||||
slices.SortFunc(nics, byUsefulness)
|
||||
return nics, nil
|
||||
}
|
||||
|
||||
// describe reads everything about one interface that needs no daemon, then lets
|
||||
// networkctl fill the gaps if it happens to be answering.
|
||||
func describe(name string) NIC {
|
||||
nic := NIC{
|
||||
Name: name,
|
||||
Driver: sysLinkBase(name, "device/driver"),
|
||||
DeviceID: deviceID(name),
|
||||
MTU: sysInt(name, "mtu"),
|
||||
IsWiFi: isWiFi(name),
|
||||
}
|
||||
|
||||
// The kernel reports -1 for a port that cannot state its speed; treat that
|
||||
// as unknown rather than letting it sort below every real figure.
|
||||
nic.SpeedMbps = max(sysInt(name, "speed"), 0)
|
||||
nic.Speed = formatSpeed(nic.SpeedMbps)
|
||||
nic.Carrier = sysAttr(name, "carrier") == "1"
|
||||
nic.State = linkState(name, nic.Carrier)
|
||||
|
||||
props := udevProperties(name)
|
||||
nic.Vendor = props["ID_VENDOR_FROM_DATABASE"]
|
||||
nic.Model = props["ID_MODEL_FROM_DATABASE"]
|
||||
|
||||
// udev's predictable names are the ones that appear in cabling notes; the
|
||||
// path-based name is the one networkctl lists first.
|
||||
for _, key := range []string{"ID_NET_NAME_PATH", "ID_NET_NAME_ONBOARD", "ID_NET_NAME_SLOT"} {
|
||||
if alt := props[key]; alt != "" && alt != name {
|
||||
nic.AltName = alt
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if out, err := commandOutput("networkctl", "status", "--no-pager", name); err == nil {
|
||||
fill(&nic, ParseStatus(string(out)))
|
||||
} else {
|
||||
slog.Debug("netprobe: networkctl unavailable, using sysfs only", "iface", name, "err", err)
|
||||
}
|
||||
return nic
|
||||
}
|
||||
|
||||
// fill copies parsed hardware detail onto a NIC without overwriting anything
|
||||
// sysfs already established. State is never taken: the carrier bit is a truer
|
||||
// answer to "is this cable live" than networkd's configuration status.
|
||||
func fill(dst *NIC, src NIC) {
|
||||
if dst.Vendor == "" {
|
||||
dst.Vendor = src.Vendor
|
||||
}
|
||||
if dst.Model == "" {
|
||||
dst.Model = src.Model
|
||||
}
|
||||
if dst.AltName == "" {
|
||||
dst.AltName = src.AltName
|
||||
}
|
||||
if dst.Speed == "" {
|
||||
dst.Speed = src.Speed
|
||||
}
|
||||
if dst.MTU == 0 {
|
||||
dst.MTU = src.MTU
|
||||
}
|
||||
}
|
||||
|
||||
// byUsefulness orders the picker so the interfaces an operator would actually
|
||||
// assign come first: a live cable, then wired over wireless, then the fastest
|
||||
// port. The rest still appear, just below the plausible choices.
|
||||
func byUsefulness(a, b NIC) int {
|
||||
if a.Carrier != b.Carrier {
|
||||
return boolOrder(a.Carrier)
|
||||
}
|
||||
if a.IsWiFi != b.IsWiFi {
|
||||
return boolOrder(b.IsWiFi)
|
||||
}
|
||||
if a.SpeedMbps != b.SpeedMbps {
|
||||
return cmp.Compare(b.SpeedMbps, a.SpeedMbps)
|
||||
}
|
||||
return cmp.Compare(a.Name, b.Name)
|
||||
}
|
||||
|
||||
// boolOrder sorts true ahead of false.
|
||||
func boolOrder(first bool) int {
|
||||
if first {
|
||||
return -1
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
// formatSpeed renders the kernel's Mbit/s figure the way a datasheet does.
|
||||
func formatSpeed(mbps int) string {
|
||||
switch {
|
||||
case mbps <= 0:
|
||||
return ""
|
||||
case mbps%1000 == 0:
|
||||
return fmt.Sprintf("%d Gbps", mbps/1000)
|
||||
default:
|
||||
return fmt.Sprintf("%d Mbps", mbps)
|
||||
}
|
||||
}
|
||||
|
||||
// linkState collapses the kernel's flags into the three outcomes an operator
|
||||
// acts on: the port is live, the port is enabled but unplugged, or the port
|
||||
// never came up at all. operstate reports "down" for the latter two alike.
|
||||
func linkState(name string, carrier bool) string {
|
||||
switch {
|
||||
case carrier:
|
||||
return "online"
|
||||
case adminUp(name):
|
||||
return "no cable"
|
||||
default:
|
||||
return "down"
|
||||
}
|
||||
}
|
||||
|
||||
// adminUp reads the IFF_UP bit out of the interface flags.
|
||||
func adminUp(name string) bool {
|
||||
flags, err := strconv.ParseUint(strings.TrimPrefix(sysAttr(name, "flags"), "0x"), 16, 64)
|
||||
return err == nil && flags&0x1 != 0
|
||||
}
|
||||
|
||||
// deviceID reads the PCI or USB "vendor:device" pair. A NIC's device link
|
||||
// points at the PCI function directly; on USB it points at the interface, whose
|
||||
// parent carries the ids.
|
||||
func deviceID(name string) string {
|
||||
dev := filepath.Join(sysClassNet, name, "device")
|
||||
if id := idPair(dev, "vendor", "device"); id != "" {
|
||||
return id
|
||||
}
|
||||
return idPair(filepath.Join(dev, ".."), "idVendor", "idProduct")
|
||||
}
|
||||
|
||||
func idPair(dir, vendorAttr, productAttr string) string {
|
||||
vendor := readTrimmed(filepath.Join(dir, vendorAttr))
|
||||
product := readTrimmed(filepath.Join(dir, productAttr))
|
||||
if vendor == "" || product == "" {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimPrefix(vendor, "0x") + ":" + strings.TrimPrefix(product, "0x")
|
||||
}
|
||||
|
||||
// udevProperties reads the interface's udev database entry, which is where the
|
||||
// hardware-database vendor and model names live. udevd populates it during the
|
||||
// installer's device enumeration, so this works with no systemd running.
|
||||
func udevProperties(name string) map[string]string {
|
||||
out, err := commandOutput("udevadm", "info", "--query=property", "--path="+filepath.Join(sysClassNet, name))
|
||||
if err != nil {
|
||||
slog.Debug("netprobe: udevadm info failed", "iface", name, "err", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
props := make(map[string]string)
|
||||
for line := range strings.SplitSeq(string(out), "\n") {
|
||||
if key, value, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
|
||||
props[key] = value
|
||||
}
|
||||
}
|
||||
return props
|
||||
}
|
||||
|
||||
// ParseStatus extracts the hardware fields from `networkctl status <iface>`
|
||||
// output. The format is a set of right-aligned "Key: value" lines, with
|
||||
// Alternative Names continuing across subsequent indented lines.
|
||||
func ParseStatus(out string) NIC {
|
||||
var nic NIC
|
||||
var inAltNames bool
|
||||
|
||||
for line := range strings.SplitSeq(out, "\n") {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
if trimmed == "" {
|
||||
inAltNames = false
|
||||
continue
|
||||
}
|
||||
|
||||
key, value, found := strings.Cut(trimmed, ": ")
|
||||
if !found {
|
||||
// A continuation line under "Alternative Names:" — take the first
|
||||
// only, which is the one operators see in cabling notes.
|
||||
if inAltNames && nic.AltName == "" {
|
||||
nic.AltName = trimmed
|
||||
}
|
||||
continue
|
||||
}
|
||||
inAltNames = false
|
||||
|
||||
key = strings.TrimSpace(key)
|
||||
value = strings.TrimSpace(value)
|
||||
switch key {
|
||||
case "Vendor":
|
||||
nic.Vendor = value
|
||||
case "Model":
|
||||
nic.Model = value
|
||||
case "Speed":
|
||||
nic.Speed = value
|
||||
case "Online state":
|
||||
// Rendered as "online" or "online (configured)".
|
||||
if f := strings.Fields(value); len(f) > 0 {
|
||||
nic.State = f[0]
|
||||
}
|
||||
case "MTU":
|
||||
// Rendered as "9000 (min: 68, max: 9978)".
|
||||
if f := strings.Fields(value); len(f) > 0 {
|
||||
if mtu, err := strconv.Atoi(f[0]); err == nil {
|
||||
nic.MTU = mtu
|
||||
}
|
||||
}
|
||||
case "Alternative Names":
|
||||
inAltNames = true
|
||||
if f := strings.Fields(value); len(f) > 0 && nic.AltName == "" {
|
||||
nic.AltName = f[0]
|
||||
}
|
||||
}
|
||||
}
|
||||
return nic
|
||||
}
|
||||
|
||||
// hasDevice reports whether the interface is backed by real hardware, which
|
||||
// sysfs shows as a device link to the PCI, USB or platform node behind it.
|
||||
func hasDevice(name string) bool {
|
||||
_, err := os.Stat(filepath.Join(sysClassNet, name, "device"))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// isWiFi reports whether the interface has a wireless subdirectory in sysfs.
|
||||
func isWiFi(name string) bool {
|
||||
_, err := os.Stat(filepath.Join(sysClassNet, name, "wireless"))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// sysAttr reads one sysfs attribute. A missing or unreadable attribute is not
|
||||
// an error worth surfacing: speed and carrier both fail on a port with no link,
|
||||
// which the state column already conveys.
|
||||
func sysAttr(name, attr string) string {
|
||||
return readTrimmed(filepath.Join(sysClassNet, name, attr))
|
||||
}
|
||||
|
||||
func sysInt(name, attr string) int {
|
||||
n, err := strconv.Atoi(sysAttr(name, attr))
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// sysLinkBase resolves a sysfs symlink and returns its final element, which is
|
||||
// how the driver name is exposed.
|
||||
func sysLinkBase(name, attr string) string {
|
||||
target, err := os.Readlink(filepath.Join(sysClassNet, name, attr))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return filepath.Base(target)
|
||||
}
|
||||
|
||||
func readTrimmed(path string) string {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
@@ -1,292 +0,0 @@
|
||||
package netprobe
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Captured from hydrogen (HP DL325 Gen10, dual 25gbe Mellanox) in us-west-1-az1.
|
||||
const mellanox25g = `● 6: ens1f0np0
|
||||
Link File: /usr/lib/systemd/network/99-default.link
|
||||
Network File: /etc/systemd/network/12-spinifex-lan-nic.network
|
||||
State: enslaved (configured)
|
||||
Online state: online
|
||||
Type: ether
|
||||
Path: pci-0000:84:00.0
|
||||
Driver: mlx5_core
|
||||
Vendor: Mellanox Technologies
|
||||
Model: MT27710 Family [ConnectX-4 Lx]
|
||||
Alternative Names: enp132s0f0np0
|
||||
enx040973e5ce60
|
||||
Hardware Address: 04:09:73:e5:ce:60 (Hewlett Packard Enterprise)
|
||||
MTU: 9000 (min: 68, max: 9978)
|
||||
QDisc: mq
|
||||
Master: br-lan
|
||||
IPv6 Address Generation Mode: none
|
||||
Number of Queues (Tx/Rx): 384/48
|
||||
Auto negotiation: yes
|
||||
Speed: 25Gbps
|
||||
Duplex: full
|
||||
Activation Policy: up
|
||||
Required For Online: yes
|
||||
Connected To: MikroTik (MikroTik RouterOS 7.11.3 (stable) Sep/27/2023 13:09:44 CRS504-4XQ) on port bridge/qsfp28-3-3
|
||||
`
|
||||
|
||||
// A link with no carrier reports neither Speed nor an online state, which is
|
||||
// the case Probe brings links up to avoid.
|
||||
const linkDown = `● 3: eno6
|
||||
Link File: /usr/lib/systemd/network/99-default.link
|
||||
State: off
|
||||
Online state: off
|
||||
Type: ether
|
||||
Driver: bnxt_en
|
||||
Vendor: Broadcom Inc. and subsidiaries
|
||||
Model: NetXtreme BCM5720 Gigabit Ethernet PCIe
|
||||
Hardware Address: 04:09:73:e5:ce:5f
|
||||
MTU: 1500 (min: 60, max: 9000)
|
||||
`
|
||||
|
||||
func TestParseStatusMellanox(t *testing.T) {
|
||||
nic := ParseStatus(mellanox25g)
|
||||
|
||||
if nic.Vendor != "Mellanox Technologies" {
|
||||
t.Errorf("Vendor = %q, want %q", nic.Vendor, "Mellanox Technologies")
|
||||
}
|
||||
if nic.Model != "MT27710 Family [ConnectX-4 Lx]" {
|
||||
t.Errorf("Model = %q, want %q", nic.Model, "MT27710 Family [ConnectX-4 Lx]")
|
||||
}
|
||||
if nic.Speed != "25Gbps" {
|
||||
t.Errorf("Speed = %q, want %q", nic.Speed, "25Gbps")
|
||||
}
|
||||
if nic.State != "online" {
|
||||
t.Errorf("State = %q, want %q", nic.State, "online")
|
||||
}
|
||||
if nic.MTU != 9000 {
|
||||
t.Errorf("MTU = %d, want 9000", nic.MTU)
|
||||
}
|
||||
// Only the first alternative name is taken; enx040973e5ce60 is ignored.
|
||||
if nic.AltName != "enp132s0f0np0" {
|
||||
t.Errorf("AltName = %q, want %q", nic.AltName, "enp132s0f0np0")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStatusLinkDown(t *testing.T) {
|
||||
nic := ParseStatus(linkDown)
|
||||
|
||||
if nic.Speed != "" {
|
||||
t.Errorf("Speed = %q, want empty for a down link", nic.Speed)
|
||||
}
|
||||
if nic.State != "off" {
|
||||
t.Errorf("State = %q, want %q", nic.State, "off")
|
||||
}
|
||||
if nic.Model != "NetXtreme BCM5720 Gigabit Ethernet PCIe" {
|
||||
t.Errorf("Model = %q, want the Broadcom model", nic.Model)
|
||||
}
|
||||
if nic.MTU != 1500 {
|
||||
t.Errorf("MTU = %d, want 1500", nic.MTU)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStatusEmpty(t *testing.T) {
|
||||
// A truncated or unparseable status must not panic or invent values.
|
||||
nic := ParseStatus("● 2: eth0\n Online state:\n Alternative Names:\n")
|
||||
if nic.State != "" || nic.AltName != "" {
|
||||
t.Errorf("expected empty fields from valueless keys, got State=%q AltName=%q", nic.State, nic.AltName)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLabelFallsBackWhenSpeedUnknown(t *testing.T) {
|
||||
got := NIC{Name: "eth0", State: "no cable"}.Label()
|
||||
want := "eth0 — no cable"
|
||||
if got != want {
|
||||
t.Errorf("Label() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Hardware must never render "unknown": an operator standing at the console
|
||||
// needs something to match against the card in their hand, and sysfs always has
|
||||
// at least a driver or a device id.
|
||||
func TestHardwareDegradesRatherThanUnknown(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
nic NIC
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "database name wins",
|
||||
nic: NIC{Vendor: "Mellanox Technologies", Model: "MT27710", Driver: "mlx5_core", DeviceID: "15b3:1015"},
|
||||
want: "Mellanox Technologies MT27710",
|
||||
},
|
||||
{"vendor only", NIC{Vendor: "Intel Corporation"}, "Intel Corporation"},
|
||||
{"driver and id", NIC{Driver: "igb", DeviceID: "8086:1533"}, "igb [8086:1533]"},
|
||||
{"driver only", NIC{Driver: "virtio_net"}, "virtio_net"},
|
||||
{"id only", NIC{DeviceID: "0bda:8153"}, "0bda:8153"},
|
||||
{"nothing at all", NIC{Name: "eth0"}, "no hardware detail"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := tt.nic.Hardware(); got != tt.want {
|
||||
t.Errorf("Hardware() = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatSpeed(t *testing.T) {
|
||||
tests := []struct {
|
||||
mbps int
|
||||
want string
|
||||
}{
|
||||
{25000, "25 Gbps"},
|
||||
{1000, "1 Gbps"},
|
||||
{100, "100 Mbps"},
|
||||
{2500, "2500 Mbps"},
|
||||
{0, ""},
|
||||
{-1, ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
if got := formatSpeed(tt.mbps); got != tt.want {
|
||||
t.Errorf("formatSpeed(%d) = %q, want %q", tt.mbps, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A live cable outranks everything, then wired over wireless, then speed. This
|
||||
// is what puts the ports an operator would actually assign at indices 0..2,
|
||||
// where newModel pre-fills wan, lan and vpc from.
|
||||
func TestByUsefulnessOrdersPluggedPortsFirst(t *testing.T) {
|
||||
nics := []NIC{
|
||||
{Name: "eno1", SpeedMbps: 1000},
|
||||
{Name: "wlp2s0", Carrier: true, IsWiFi: true, SpeedMbps: 866},
|
||||
{Name: "ens1f1np1", SpeedMbps: 25000},
|
||||
{Name: "ens1f0np0", Carrier: true, SpeedMbps: 25000},
|
||||
{Name: "eno2", Carrier: true, SpeedMbps: 1000},
|
||||
}
|
||||
slices.SortFunc(nics, byUsefulness)
|
||||
|
||||
want := []string{"ens1f0np0", "eno2", "wlp2s0", "ens1f1np1", "eno1"}
|
||||
for i, name := range want {
|
||||
if nics[i].Name != name {
|
||||
t.Fatalf("order = %v, want %v", names(nics), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func names(nics []NIC) []string {
|
||||
out := make([]string, len(nics))
|
||||
for i, n := range nics {
|
||||
out[i] = n.Name
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestProbeReadsSysfsWithoutSystemd(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
sysClassNet = root
|
||||
linkUpTimeout = 0
|
||||
t.Cleanup(func() {
|
||||
sysClassNet = "/sys/class/net"
|
||||
linkUpTimeout = 2 * time.Second
|
||||
})
|
||||
|
||||
// ens1f0np0: a 25gbe Mellanox with a live cable, identified by the hardware
|
||||
// database. eno1: a 1gbe port that is up but unplugged, and absent from the
|
||||
// database, so only its driver and PCI id identify it.
|
||||
writeIface(t, root, "ens1f0np0", map[string]string{
|
||||
"speed": "25000", "carrier": "1", "mtu": "9000", "flags": "0x1003",
|
||||
"device/vendor": "0x15b3", "device/device": "0x1015",
|
||||
}, "mlx5_core")
|
||||
writeIface(t, root, "eno1", map[string]string{
|
||||
"speed": "1000", "carrier": "0", "mtu": "1500", "flags": "0x1003",
|
||||
"device/vendor": "0x8086", "device/device": "0x1533",
|
||||
}, "igb")
|
||||
|
||||
listInterfaces = func() ([]net.Interface, error) {
|
||||
return []net.Interface{
|
||||
{Name: "lo", Flags: net.FlagLoopback},
|
||||
{Name: "eno1", HardwareAddr: net.HardwareAddr{4, 9, 0x73, 0xe5, 0xce, 0x5f}},
|
||||
{Name: "ens1f0np0", HardwareAddr: net.HardwareAddr{4, 9, 0x73, 0xe5, 0xce, 0x60}},
|
||||
{Name: "veth0"}, // no hardware address, so not a physical port
|
||||
{Name: "br-lan", HardwareAddr: net.HardwareAddr{2, 0, 0, 0, 0, 1}}, // a bridge, so no device behind it
|
||||
}, nil
|
||||
}
|
||||
t.Cleanup(func() { listInterfaces = net.Interfaces })
|
||||
|
||||
// Only udevadm answers. networkctl fails the way it does in the live
|
||||
// installer, where there is no systemd-networkd for it to reach.
|
||||
commandOutput = func(name string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case name == "ip":
|
||||
return nil, nil
|
||||
case name == "udevadm" && strings.HasSuffix(args[len(args)-1], "ens1f0np0"):
|
||||
return []byte("ID_VENDOR_FROM_DATABASE=Mellanox Technologies\n" +
|
||||
"ID_MODEL_FROM_DATABASE=MT27710 Family [ConnectX-4 Lx]\n" +
|
||||
"ID_NET_NAME_PATH=enp132s0f0np0\n"), nil
|
||||
case name == "udevadm":
|
||||
return []byte("ID_NET_NAME_PATH=enp1s0f0\n"), nil
|
||||
default:
|
||||
return nil, errors.New("Failed to connect to bus: No such file or directory")
|
||||
}
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
commandOutput = func(name string, args ...string) ([]byte, error) {
|
||||
return exec.Command(name, args...).Output()
|
||||
}
|
||||
})
|
||||
|
||||
nics, err := Probe()
|
||||
if err != nil {
|
||||
t.Fatalf("Probe: %v", err)
|
||||
}
|
||||
if got := names(nics); !slices.Equal(got, []string{"ens1f0np0", "eno1"}) {
|
||||
t.Fatalf("probed %v, want the two physical ports with the plugged one first", got)
|
||||
}
|
||||
|
||||
mlx := nics[0]
|
||||
if mlx.Label() != "ens1f0np0 25 Gbps online" {
|
||||
t.Errorf("Label() = %q", mlx.Label())
|
||||
}
|
||||
if want := "Mellanox Technologies MT27710 Family [ConnectX-4 Lx]"; mlx.Hardware() != want {
|
||||
t.Errorf("Hardware() = %q, want %q", mlx.Hardware(), want)
|
||||
}
|
||||
if mlx.AltName != "enp132s0f0np0" || mlx.MTU != 9000 || !mlx.Carrier {
|
||||
t.Errorf("unexpected detail: %+v", mlx)
|
||||
}
|
||||
|
||||
// Up but unplugged, and unknown to the database: the driver and PCI id are
|
||||
// all the operator gets, which is still enough to identify the card.
|
||||
igb := nics[1]
|
||||
if igb.Label() != "eno1 1 Gbps no cable" {
|
||||
t.Errorf("Label() = %q", igb.Label())
|
||||
}
|
||||
if igb.Hardware() != "igb [8086:1533]" {
|
||||
t.Errorf("Hardware() = %q", igb.Hardware())
|
||||
}
|
||||
}
|
||||
|
||||
// writeIface lays out the sysfs attributes Probe reads for one interface,
|
||||
// including the driver symlink it resolves the module name from.
|
||||
func writeIface(t *testing.T, root, name string, attrs map[string]string, driver string) {
|
||||
t.Helper()
|
||||
for attr, value := range attrs {
|
||||
path := filepath.Join(root, name, attr)
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(value+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.Symlink("../../../bus/pci/drivers/"+driver, filepath.Join(root, name, "device/driver")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -1,161 +0,0 @@
|
||||
// Package systemd writes systemd unit files and drop-ins into an installed
|
||||
// system root during the Spinifex ISO installation process.
|
||||
package systemd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// WriteBridgeUnit installs a non-critical oneshot service that activates an
|
||||
// east-west bridge (br-lan, br-vpc) via networkctl *after* network-online.target.
|
||||
// Those bridges carry ActivationPolicy=manual in their networkd .network file,
|
||||
// so networkd creates them at boot but does not bring them up automatically.
|
||||
// This service triggers the activation once the WAN path is confirmed online,
|
||||
// ensuring a missing cable or DHCP timeout on a secondary bridge never stalls
|
||||
// the management interface or firstboot.
|
||||
func WriteBridgeUnit(root, bridge string) error {
|
||||
name := "spinifex-" + strings.TrimPrefix(bridge, "br-") + "-bridge.service"
|
||||
unit := fmt.Sprintf(`[Unit]
|
||||
Description=Spinifex %s bridge (non-critical)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/networkctl up %s
|
||||
RemainAfterExit=yes
|
||||
# Failure is non-critical — cable unplugged or switch not ready at boot.
|
||||
SuccessExitStatus=0 1
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`, bridge, bridge)
|
||||
unitPath := filepath.Join(root, "etc/systemd/system", name)
|
||||
if err := os.WriteFile(unitPath, []byte(unit), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
return EnableUnit(root, name)
|
||||
}
|
||||
|
||||
// WriteFirstbootUnit writes the spinifex-firstboot.service oneshot unit that
|
||||
// runs the firstboot provisioning script on the first real boot after
|
||||
// installation. The script writes /var/lib/spinifex/.firstboot-done on
|
||||
// success; ConditionPathExists=! on that marker condition-skips the unit on
|
||||
// every subsequent boot so it never re-runs (replacing the previous
|
||||
// "disable on EXIT" trap which couldn't distinguish success from failure).
|
||||
func WriteFirstbootUnit(root string) error {
|
||||
unit := `[Unit]
|
||||
Description=Spinifex first-boot provisioning
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
ConditionPathExists=/usr/local/bin/spinifex-firstboot.sh
|
||||
ConditionPathExists=!/var/lib/spinifex/.firstboot-done
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
Environment=HOME=/root
|
||||
ExecStart=/usr/local/bin/spinifex-firstboot.sh
|
||||
RemainAfterExit=yes
|
||||
# On non-zero exit, overwrite /etc/issue with a hint so the operator sees
|
||||
# the failure on the console — the banner unit is never enabled in that
|
||||
# case (firstboot didn't reach its enable --now line), so /etc/issue would
|
||||
# otherwise stay at the Debian default with no signal that anything is wrong.
|
||||
ExecStopPost=/bin/sh -c 'if [ "$EXIT_STATUS" != "0" ]; then printf "\n*** Spinifex firstboot FAILED ***\nLogin as spinifex, then run:\n sudo journalctl -u spinifex-firstboot.service\n\n" > /etc/issue; fi'
|
||||
# Cap total firstboot runtime so a hang in setup-ovn.sh / spx admin init /
|
||||
# ovn-central startup cannot wedge multi-user.target and keep getty from
|
||||
# ever reaching the login prompt.
|
||||
TimeoutStartSec=300s
|
||||
# Install diagnostics land in the journal, which persists for the life of the
|
||||
# node — nothing this script runs may print secrets to stdout or stderr.
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
path := filepath.Join(root, "etc/systemd/system/spinifex-firstboot.service")
|
||||
return os.WriteFile(path, []byte(unit), 0o644)
|
||||
}
|
||||
|
||||
// WriteBannerUnit writes the spinifex-banner.service unit that runs
|
||||
// `spx admin banner --boot-check` on every boot after spinifex.target is up.
|
||||
// Running After=spinifex.target ensures the banner reflects a settled system
|
||||
// state and that the IP check/restart happens once services are already running.
|
||||
//
|
||||
// The unit file is written by the installer but NOT enabled at install time.
|
||||
// firstboot enables it (along with spinifex.target) at the end of its first
|
||||
// run via `systemctl enable --now`. On every subsequent boot both units are
|
||||
// driven by the multi-user.target.wants/ symlinks firstboot installed.
|
||||
//
|
||||
// Note: only After=spinifex.target — no Wants=. A Wants= would implicitly
|
||||
// pull spinifex.target up at boot, which is exactly the race this whole
|
||||
// change is fixing. After= alone is a no-op when the target isn't started,
|
||||
// which is correct on first boot before firstboot has provisioned configs.
|
||||
func WriteBannerUnit(root string) error {
|
||||
unit := `[Unit]
|
||||
Description=Spinifex console banner and boot health check
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/spx admin banner --boot-check
|
||||
RemainAfterExit=yes
|
||||
# Banner is oneshot; cap it so a stuck boot-check (IP detection, try-restart)
|
||||
# cannot block getty via the spinifex-wait.conf drop-in.
|
||||
TimeoutStartSec=30s
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
path := filepath.Join(root, "etc/systemd/system/spinifex-banner.service")
|
||||
return os.WriteFile(path, []byte(unit), 0o644)
|
||||
}
|
||||
|
||||
// WriteGettyDropIn holds the primary consoles (tty1 and ttyS0) until
|
||||
// spinifex-banner.service completes (when banner is enabled), so the MOTD
|
||||
// banner is visible before the login prompt appears. Drop-ins are scoped to
|
||||
// named instances (getty@tty1, serial-getty@ttyS0) so tty2, tty3, etc.
|
||||
// remain available as rescue terminals.
|
||||
//
|
||||
// Only After=, not Wants=. A Wants= would implicitly start the (un-enabled)
|
||||
// banner on the very first boot, partially undoing the "only firstboot is
|
||||
// enabled on first boot" guarantee. On first boot, banner is not enabled,
|
||||
// so After= is a no-op and login appears immediately with the default
|
||||
// /etc/issue. Once firstboot completes and enables the banner, subsequent
|
||||
// boots have banner enabled at multi-user.target and After= correctly
|
||||
// orders getty after the banner has rendered /etc/issue.
|
||||
func WriteGettyDropIn(root string) error {
|
||||
dropIn := `[Unit]
|
||||
After=spinifex-firstboot.service
|
||||
`
|
||||
for _, svc := range []string{"getty@tty1", "serial-getty@ttyS0"} {
|
||||
dropInDir := filepath.Join(root, "etc/systemd/system/"+svc+".service.d")
|
||||
if err := os.MkdirAll(dropInDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dropInDir, "spinifex-wait.conf"), []byte(dropIn), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// EnableUnit creates the multi-user.target.wants symlink for serviceName,
|
||||
// equivalent to `systemctl enable` for a unit that targets multi-user.target.
|
||||
func EnableUnit(root, serviceName string) error {
|
||||
wantsDir := filepath.Join(root, "etc/systemd/system/multi-user.target.wants")
|
||||
if err := os.MkdirAll(wantsDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
link := filepath.Join(wantsDir, serviceName)
|
||||
target := "/etc/systemd/system/" + serviceName
|
||||
if err := os.Remove(link); err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("remove stale symlink %s: %w", link, err)
|
||||
}
|
||||
return os.Symlink(target, link)
|
||||
}
|
||||
@@ -1,696 +0,0 @@
|
||||
package ui
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/charmbracelet/bubbles/textinput"
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
"github.com/mulgadc/spinifex/cmd/installer/install"
|
||||
"github.com/mulgadc/spinifex/cmd/installer/netprobe"
|
||||
)
|
||||
|
||||
// foldedNIC marks a role with no interface of its own, which collapses onto
|
||||
// the plane above it (vpc <- lan <- wan).
|
||||
const foldedNIC = -1
|
||||
|
||||
// roleField identifies one editable element on the per-role screen. Which of
|
||||
// these are actually shown depends on the plane, the addressing mode, whether
|
||||
// the NIC is wireless, and whether Advanced is on.
|
||||
type roleField int
|
||||
|
||||
const (
|
||||
roleFieldNIC roleField = iota
|
||||
roleFieldMethod
|
||||
roleFieldIP
|
||||
roleFieldMask
|
||||
roleFieldGateway
|
||||
roleFieldDNS
|
||||
roleFieldVLAN
|
||||
roleFieldMTU
|
||||
roleFieldSSID
|
||||
roleFieldWiFiPass
|
||||
)
|
||||
|
||||
// roleForm is the editable state for one plane. The three forms are the whole
|
||||
// of the installer's network configuration.
|
||||
type roleForm struct {
|
||||
plane install.Plane
|
||||
|
||||
// nic indexes into model.nics, or foldedNIC when the role is folded.
|
||||
nic int
|
||||
dhcp bool
|
||||
focus roleField
|
||||
|
||||
ip textinput.Model
|
||||
mask textinput.Model
|
||||
gateway textinput.Model
|
||||
dns textinput.Model
|
||||
vlan textinput.Model
|
||||
mtu textinput.Model
|
||||
ssid textinput.Model
|
||||
wifiPass textinput.Model
|
||||
}
|
||||
|
||||
func newRoleForm(plane install.Plane, nic int) roleForm {
|
||||
text := func(placeholder string, limit int) textinput.Model {
|
||||
in := textinput.New()
|
||||
in.Prompt = ""
|
||||
in.Placeholder = placeholder
|
||||
if limit > 0 {
|
||||
in.CharLimit = limit
|
||||
}
|
||||
return in
|
||||
}
|
||||
|
||||
f := roleForm{
|
||||
plane: plane,
|
||||
nic: nic,
|
||||
dhcp: plane == install.PlaneWAN,
|
||||
ip: text(defaultIPPlaceholder(plane), 0),
|
||||
mask: text("255.255.255.0", 0),
|
||||
gateway: text("192.168.1.1", 0),
|
||||
dns: text("1.1.1.1, 8.8.8.8", 0),
|
||||
vlan: text("untagged", 4),
|
||||
mtu: text("1500", 5),
|
||||
ssid: text("Network SSID", 64),
|
||||
wifiPass: text("WiFi password", 128),
|
||||
}
|
||||
f.wifiPass.EchoMode = textinput.EchoPassword
|
||||
return f
|
||||
}
|
||||
|
||||
// defaultIPPlaceholder hints the addressing convention for each plane so an
|
||||
// operator filling in a rack does not have to remember it.
|
||||
func defaultIPPlaceholder(plane install.Plane) string {
|
||||
switch plane {
|
||||
case install.PlaneLAN:
|
||||
return "10.0.0.3"
|
||||
case install.PlaneVPC:
|
||||
return "10.1.0.3"
|
||||
default:
|
||||
return "192.168.1.10"
|
||||
}
|
||||
}
|
||||
|
||||
func (f *roleForm) folded() bool { return f.nic == foldedNIC }
|
||||
|
||||
// canFold reports whether this role may be left unbound. wan always needs an
|
||||
// uplink, so only lan and vpc can fold.
|
||||
func (f *roleForm) canFold() bool { return f.plane != install.PlaneWAN }
|
||||
|
||||
// visibleFields returns the fields shown for this form, in tab order. Hiding
|
||||
// rather than disabling keeps the default screen narrow: a DHCP role shows two
|
||||
// lines, and VLAN/MTU only appear once Advanced is on.
|
||||
func (f *roleForm) visibleFields(isWiFi, advanced bool) []roleField {
|
||||
if f.folded() {
|
||||
return []roleField{roleFieldNIC}
|
||||
}
|
||||
fields := []roleField{roleFieldNIC, roleFieldMethod}
|
||||
if !f.dhcp {
|
||||
fields = append(fields, roleFieldIP, roleFieldMask)
|
||||
// Default route and resolvers are properties of the node, not of each
|
||||
// plane, and wan is the only one that carries either: it holds the
|
||||
// default route, so it is the only plane that can reach an upstream
|
||||
// resolver. On DHCP the lease supplies both.
|
||||
if f.plane == install.PlaneWAN {
|
||||
fields = append(fields, roleFieldGateway, roleFieldDNS)
|
||||
}
|
||||
}
|
||||
if advanced {
|
||||
fields = append(fields, roleFieldVLAN, roleFieldMTU)
|
||||
}
|
||||
if isWiFi {
|
||||
fields = append(fields, roleFieldSSID, roleFieldWiFiPass)
|
||||
}
|
||||
return fields
|
||||
}
|
||||
|
||||
// input returns the textinput backing a field, or nil for the non-text fields.
|
||||
func (f *roleForm) input(field roleField) *textinput.Model {
|
||||
switch field {
|
||||
case roleFieldIP:
|
||||
return &f.ip
|
||||
case roleFieldMask:
|
||||
return &f.mask
|
||||
case roleFieldGateway:
|
||||
return &f.gateway
|
||||
case roleFieldDNS:
|
||||
return &f.dns
|
||||
case roleFieldVLAN:
|
||||
return &f.vlan
|
||||
case roleFieldMTU:
|
||||
return &f.mtu
|
||||
case roleFieldSSID:
|
||||
return &f.ssid
|
||||
case roleFieldWiFiPass:
|
||||
return &f.wifiPass
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// blurAll drops focus from every input so only the focused field has a cursor.
|
||||
func (f *roleForm) blurAll() {
|
||||
for _, field := range []roleField{
|
||||
roleFieldIP, roleFieldMask, roleFieldGateway, roleFieldDNS,
|
||||
roleFieldVLAN, roleFieldMTU, roleFieldSSID, roleFieldWiFiPass,
|
||||
} {
|
||||
f.input(field).Blur()
|
||||
}
|
||||
}
|
||||
|
||||
// focusCurrent gives the cursor to the focused field, if it is a text field.
|
||||
func (f *roleForm) focusCurrent() {
|
||||
f.blurAll()
|
||||
if in := f.input(f.focus); in != nil {
|
||||
in.Focus()
|
||||
}
|
||||
}
|
||||
|
||||
// toRole converts the form into the install package's role model.
|
||||
func (f *roleForm) toRole(nics []netprobe.NIC) install.NetworkRole {
|
||||
if f.folded() || f.nic >= len(nics) {
|
||||
return install.NetworkRole{}
|
||||
}
|
||||
role := install.NetworkRole{
|
||||
Interface: nics[f.nic].Name,
|
||||
DHCPMode: f.dhcp,
|
||||
VLAN: atoiOr(f.vlan.Value(), 0),
|
||||
MTU: atoiOr(f.mtu.Value(), 0),
|
||||
}
|
||||
if !f.dhcp {
|
||||
role.Address = strings.TrimSpace(f.ip.Value())
|
||||
role.Mask = strings.TrimSpace(f.mask.Value())
|
||||
role.DNS = parseDNS(f.dns.Value())
|
||||
if f.plane == install.PlaneWAN {
|
||||
role.Gateway = strings.TrimSpace(f.gateway.Value())
|
||||
}
|
||||
}
|
||||
if nics[f.nic].IsWiFi {
|
||||
role.WiFiSSID = strings.TrimSpace(f.ssid.Value())
|
||||
role.WiFiPass = f.wifiPass.Value()
|
||||
}
|
||||
return role
|
||||
}
|
||||
|
||||
func atoiOr(s string, fallback int) int {
|
||||
n, err := strconv.Atoi(strings.TrimSpace(s))
|
||||
if err != nil {
|
||||
return fallback
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// ── Roles overview screen ─────────────────────────────────────────────────────
|
||||
|
||||
// continueRow is the cursor position below the three role rows.
|
||||
const continueRow = 3
|
||||
|
||||
func (m model) handleRolesKey(key string) (tea.Model, tea.Cmd) {
|
||||
switch key {
|
||||
case "up", "k":
|
||||
if m.roleCursor > 0 {
|
||||
m.roleCursor--
|
||||
}
|
||||
case "down", "j":
|
||||
if m.roleCursor < continueRow {
|
||||
m.roleCursor++
|
||||
}
|
||||
case "a":
|
||||
// Advanced reveals VLAN and MTU on every role at once, rather than
|
||||
// per-role, so the operator flips it on for the whole node.
|
||||
m.advanced = !m.advanced
|
||||
m.validationErr = ""
|
||||
case "enter":
|
||||
if m.roleCursor == continueRow {
|
||||
if err := m.buildConfig().Validate(); err != nil {
|
||||
m.validationErr = capitalise(err.Error())
|
||||
return m, nil
|
||||
}
|
||||
m.validationErr = ""
|
||||
m.screen = screenIdentity
|
||||
m.hostnameInput.Focus()
|
||||
return m, nil
|
||||
}
|
||||
m.validationErr = ""
|
||||
m.screen = screenNetworkRole
|
||||
m.roles[m.roleCursor].focus = roleFieldNIC
|
||||
m.roles[m.roleCursor].blurAll()
|
||||
case "esc":
|
||||
m.screen = screenDiskConfirm
|
||||
m.eraseInput.Focus()
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (m model) viewNetworkRoles(w int) string {
|
||||
title := styleTitle.Render("Network Planes")
|
||||
subtitle := styleMuted.Render("A role left unbound folds onto the plane above it.")
|
||||
|
||||
adv := styleMuted.Render("Advanced: off")
|
||||
if m.advanced {
|
||||
adv = styleSuccess.Render("Advanced: on")
|
||||
}
|
||||
|
||||
header := fmt.Sprintf(" %-*s%-*s", colRole, "ROLE", colIface, "INTERFACE")
|
||||
if m.advanced {
|
||||
header += fmt.Sprintf("%-*s%-*s", colVLAN, "VLAN", colMTU, "MTU")
|
||||
}
|
||||
header += "ADDRESS"
|
||||
|
||||
lines := []string{title, subtitle, "", adv, "", styleMuted.Render(header)}
|
||||
|
||||
for i := range m.roles {
|
||||
row := m.renderRoleRow(i)
|
||||
if i == m.roleCursor {
|
||||
lines = append(lines, styleSelected.Render("> "+row))
|
||||
} else {
|
||||
lines = append(lines, " "+row)
|
||||
}
|
||||
if detail := m.roleHardware(i, bodyWidth(w)); detail != "" {
|
||||
lines = append(lines, detail)
|
||||
}
|
||||
}
|
||||
|
||||
lines = append(lines, "")
|
||||
cont := " Continue"
|
||||
if m.roleCursor == continueRow {
|
||||
cont = styleSelected.Render("> Continue")
|
||||
}
|
||||
lines = append(lines, cont)
|
||||
|
||||
if m.validationErr != "" {
|
||||
lines = append(lines, "", styleError.Render(m.validationErr))
|
||||
}
|
||||
lines = append(lines, styleHelp.Render("↑/↓ move • Enter edit • a advanced • Esc back"))
|
||||
|
||||
body := lipgloss.JoinVertical(lipgloss.Left, lines...)
|
||||
return lipgloss.Place(w, m.height, lipgloss.Center, lipgloss.Center,
|
||||
styleBox.Width(min(w-4, 78)).Render(body),
|
||||
)
|
||||
}
|
||||
|
||||
// renderRoleRow renders one line of the overview table. A folded role names
|
||||
// the plane it collapsed onto rather than showing a blank, so the operator can
|
||||
// always see where the traffic will actually go.
|
||||
func (m model) renderRoleRow(i int) string {
|
||||
f := &m.roles[i]
|
||||
name := fmt.Sprintf("%-5s", f.plane)
|
||||
|
||||
if f.folded() {
|
||||
_, landed := m.buildConfig().Resolve(f.plane)
|
||||
return fmt.Sprintf("%-*s%s", colRole, name, styleMuted.Render("folds onto "+string(landed)))
|
||||
}
|
||||
|
||||
iface := styleMuted.Render("—")
|
||||
if f.nic < len(m.nics) {
|
||||
iface = nicSummary(m.nics[f.nic])
|
||||
}
|
||||
|
||||
addr := "DHCP"
|
||||
if !f.dhcp {
|
||||
addr = formatCIDR(f.ip.Value(), f.mask.Value())
|
||||
if addr == "" {
|
||||
addr = styleWarning.Render("not set")
|
||||
}
|
||||
}
|
||||
|
||||
row := padCell(name, colRole) + padCell(iface, colIface)
|
||||
if m.advanced {
|
||||
row += padCell(valueOr(f.vlan.Value(), "—"), colVLAN) + padCell(valueOr(f.mtu.Value(), "—"), colMTU)
|
||||
}
|
||||
return row + addr
|
||||
}
|
||||
|
||||
// roleHardware is the vendor/model line beneath a role row, indented under the
|
||||
// interface column. It is what tells an operator that wan really is the 25gbe
|
||||
// Mellanox and not the onboard 1gbe.
|
||||
func (m model) roleHardware(i, width int) string {
|
||||
f := &m.roles[i]
|
||||
if f.folded() || f.nic >= len(m.nics) {
|
||||
return ""
|
||||
}
|
||||
indent := 2 + colRole
|
||||
return strings.Repeat(" ", indent) + styleMuted.Render(nicHardware(m.nics[f.nic], width-indent))
|
||||
}
|
||||
|
||||
// nicSummary renders name, speed and link state, colouring the state so a port
|
||||
// with no cable stands out rather than having to be read for.
|
||||
func nicSummary(n netprobe.NIC) string {
|
||||
state := styleWarning.Render(n.State)
|
||||
if n.Carrier {
|
||||
state = styleSuccess.Render(n.State)
|
||||
}
|
||||
speed := n.Speed
|
||||
if speed == "" {
|
||||
speed = "—"
|
||||
}
|
||||
return fmt.Sprintf("%s %s %s", n.Name, speed, state)
|
||||
}
|
||||
|
||||
// nicHardware is the detail line: what the card is, plus the predictable name
|
||||
// it also answers to, which is the one written on cabling notes. The alt name
|
||||
// is dropped rather than truncated when the two will not fit together.
|
||||
func nicHardware(n netprobe.NIC, width int) string {
|
||||
hw := n.Hardware()
|
||||
const separator = " · "
|
||||
if n.AltName != "" && len([]rune(hw))+len(separator)+len([]rune(n.AltName)) <= width {
|
||||
return hw + separator + n.AltName
|
||||
}
|
||||
return truncate(hw, width)
|
||||
}
|
||||
|
||||
// bodyWidth is the text width inside the framed body. Detail lines are trimmed
|
||||
// to it rather than left to wrap, which would push the table out of alignment.
|
||||
func bodyWidth(w int) int {
|
||||
return min(w-4, 78) - 4
|
||||
}
|
||||
|
||||
// padCell pads a cell to width by display width, which the %-*s verb cannot do
|
||||
// once a cell carries the ANSI escapes of a style.
|
||||
func padCell(s string, width int) string {
|
||||
return lipgloss.PlaceHorizontal(width, lipgloss.Left, s)
|
||||
}
|
||||
|
||||
// Column widths for the overview table, shared by the header and the rows.
|
||||
const (
|
||||
colRole = 6
|
||||
colIface = 32
|
||||
colVLAN = 7
|
||||
colMTU = 7
|
||||
)
|
||||
|
||||
// focusMark gives the focused text field a visible caret without every field
|
||||
// rendering its own prompt.
|
||||
func focusMark(focused bool) string {
|
||||
if focused {
|
||||
return styleLabel.Render("> ")
|
||||
}
|
||||
return " "
|
||||
}
|
||||
|
||||
// formatCIDR renders an address and mask the way an operator writes them,
|
||||
// falling back to the raw mask when it is not dotted-decimal.
|
||||
func formatCIDR(addr, mask string) string {
|
||||
addr = strings.TrimSpace(addr)
|
||||
mask = strings.TrimSpace(mask)
|
||||
if addr == "" {
|
||||
return ""
|
||||
}
|
||||
if mask == "" {
|
||||
return addr
|
||||
}
|
||||
if ip := net.ParseIP(mask); ip != nil && ip.To4() != nil {
|
||||
if ones, bits := net.IPMask(ip.To4()).Size(); bits != 0 {
|
||||
return fmt.Sprintf("%s/%d", addr, ones)
|
||||
}
|
||||
}
|
||||
return addr + "/" + strings.TrimPrefix(mask, "/")
|
||||
}
|
||||
|
||||
func valueOr(v, fallback string) string {
|
||||
if v = strings.TrimSpace(v); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
// ── Per-role editor screen ────────────────────────────────────────────────────
|
||||
|
||||
func (m model) handleRoleKey(key string, msg tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
f := &m.roles[m.roleCursor]
|
||||
fields := f.visibleFields(m.nicIsWiFi(f.nic), m.advanced)
|
||||
pos := fieldIndex(fields, f.focus)
|
||||
|
||||
switch key {
|
||||
case "esc":
|
||||
f.blurAll()
|
||||
m.screen = screenNetworkRoles
|
||||
m.validationErr = ""
|
||||
return m, nil
|
||||
|
||||
case "enter":
|
||||
if err := m.validateRole(m.roleCursor); err != "" {
|
||||
m.validationErr = err
|
||||
return m, nil
|
||||
}
|
||||
f.blurAll()
|
||||
m.screen = screenNetworkRoles
|
||||
m.validationErr = ""
|
||||
return m, nil
|
||||
|
||||
case "tab", "down":
|
||||
pos = (pos + 1) % len(fields)
|
||||
f.focus = fields[pos]
|
||||
f.focusCurrent()
|
||||
return m, nil
|
||||
|
||||
case "shift+tab", "up":
|
||||
pos = (pos - 1 + len(fields)) % len(fields)
|
||||
f.focus = fields[pos]
|
||||
f.focusCurrent()
|
||||
return m, nil
|
||||
|
||||
case "left", "right":
|
||||
// Only the two selector fields consume arrows; text fields need them
|
||||
// for cursor movement.
|
||||
switch f.focus {
|
||||
case roleFieldNIC:
|
||||
return m.cycleNIC(m.roleCursor, key == "right"), nil
|
||||
case roleFieldMethod:
|
||||
f.dhcp = key == "left"
|
||||
f.focusCurrent()
|
||||
return m, nil
|
||||
}
|
||||
}
|
||||
|
||||
if in := f.input(f.focus); in != nil {
|
||||
var cmd tea.Cmd
|
||||
*in, cmd = in.Update(msg)
|
||||
return m, cmd
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// cycleNIC steps the interface selection, including the fold option for lan
|
||||
// and vpc. Wrapping keeps a single key usable on a node with one NIC.
|
||||
func (m model) cycleNIC(idx int, forward bool) model {
|
||||
f := &m.roles[idx]
|
||||
low := 0
|
||||
if f.canFold() {
|
||||
low = foldedNIC
|
||||
}
|
||||
high := len(m.nics) - 1
|
||||
|
||||
if forward {
|
||||
if f.nic >= high {
|
||||
f.nic = low
|
||||
} else {
|
||||
f.nic++
|
||||
}
|
||||
return m
|
||||
}
|
||||
if f.nic <= low {
|
||||
f.nic = high
|
||||
} else {
|
||||
f.nic--
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func (m model) nicIsWiFi(idx int) bool {
|
||||
return idx >= 0 && idx < len(m.nics) && m.nics[idx].IsWiFi
|
||||
}
|
||||
|
||||
func (m model) viewNetworkRole(w int) string {
|
||||
f := &m.roles[m.roleCursor]
|
||||
title := styleTitle.Render(fmt.Sprintf("Configure %s plane", strings.ToUpper(string(f.plane))))
|
||||
subtitle := styleMuted.Render(planeDescription(f.plane))
|
||||
|
||||
lines := []string{title, subtitle, ""}
|
||||
for _, field := range f.visibleFields(m.nicIsWiFi(f.nic), m.advanced) {
|
||||
lines = append(lines, m.renderRoleField(f, field, bodyWidth(w)), "")
|
||||
}
|
||||
|
||||
if m.validationErr != "" {
|
||||
lines = append(lines, styleError.Render(m.validationErr), "")
|
||||
}
|
||||
lines = append(lines, styleHelp.Render("Tab/↑↓ move • ←/→ change • Enter done • Esc cancel"))
|
||||
|
||||
body := lipgloss.JoinVertical(lipgloss.Left, lines...)
|
||||
return lipgloss.Place(w, m.height, lipgloss.Center, lipgloss.Center,
|
||||
styleBox.Width(min(w-4, 78)).Render(body),
|
||||
)
|
||||
}
|
||||
|
||||
// planeDescription explains what rides each plane, so the choice is not just a
|
||||
// three-letter name.
|
||||
func planeDescription(p install.Plane) string {
|
||||
switch p {
|
||||
case install.PlaneWAN:
|
||||
return "Public ingress and egress, EIPs and NAT (br-wan)."
|
||||
case install.PlaneLAN:
|
||||
return "Storage replication, NATS mesh and OVN control (br-lan)."
|
||||
default:
|
||||
return "Geneve tunnels carrying EC2 traffic between nodes (br-vpc)."
|
||||
}
|
||||
}
|
||||
|
||||
// roleLabelWidth is the field-label column on the editor screen, shared so a
|
||||
// wrapped value can indent to line up underneath it.
|
||||
const roleLabelWidth = 14
|
||||
|
||||
func (m model) renderRoleField(f *roleForm, field roleField, width int) string {
|
||||
focused := f.focus == field
|
||||
label := styleLabel.Render(fmt.Sprintf("%-*s", roleLabelWidth, roleFieldLabel(field)))
|
||||
|
||||
switch field {
|
||||
case roleFieldNIC:
|
||||
return label + m.renderNICChoice(f, focused, width)
|
||||
|
||||
case roleFieldMethod:
|
||||
return label + renderToggle([]string{"DHCP (automatic)", "Static"}, boolToIdx(!f.dhcp), focused)
|
||||
|
||||
default:
|
||||
if in := f.input(field); in != nil {
|
||||
return label + focusMark(focused) + in.View()
|
||||
}
|
||||
return label
|
||||
}
|
||||
}
|
||||
|
||||
func roleFieldLabel(field roleField) string {
|
||||
switch field {
|
||||
case roleFieldNIC:
|
||||
return "Interface"
|
||||
case roleFieldMethod:
|
||||
return "IP method"
|
||||
case roleFieldIP:
|
||||
return "IP address"
|
||||
case roleFieldMask:
|
||||
return "Subnet mask"
|
||||
case roleFieldGateway:
|
||||
return "Gateway"
|
||||
case roleFieldDNS:
|
||||
return "DNS"
|
||||
case roleFieldVLAN:
|
||||
return "VLAN id"
|
||||
case roleFieldMTU:
|
||||
return "MTU"
|
||||
case roleFieldSSID:
|
||||
return "WiFi SSID"
|
||||
default:
|
||||
return "WiFi password"
|
||||
}
|
||||
}
|
||||
|
||||
// renderNICChoice shows the selected interface on the field line and its
|
||||
// hardware identity on a line of its own beneath. Vendor and model are what
|
||||
// make a port identifiable, and they do not fit alongside the selector.
|
||||
func (m model) renderNICChoice(f *roleForm, focused bool, width int) string {
|
||||
var text, detail string
|
||||
switch {
|
||||
case f.folded():
|
||||
_, landed := m.buildConfig().Resolve(f.plane)
|
||||
text = fmt.Sprintf("(fold onto %s)", landed)
|
||||
case f.nic < len(m.nics):
|
||||
text = nicSummary(m.nics[f.nic])
|
||||
detail = nicHardware(m.nics[f.nic], width-roleLabelWidth-2)
|
||||
default:
|
||||
text = "(none detected)"
|
||||
}
|
||||
|
||||
if focused {
|
||||
text = styleSelected.Render(" ← " + text + " → ")
|
||||
} else {
|
||||
text = styleLabel.Render("[ " + text + " ]")
|
||||
}
|
||||
if detail == "" {
|
||||
return text
|
||||
}
|
||||
return text + "\n" + strings.Repeat(" ", roleLabelWidth+2) + styleMuted.Render(detail)
|
||||
}
|
||||
|
||||
func renderToggle(options []string, selected int, focused bool) string {
|
||||
var parts []string
|
||||
for i, opt := range options {
|
||||
switch {
|
||||
case i == selected && focused:
|
||||
parts = append(parts, styleSelected.Render(" "+opt+" "))
|
||||
case i == selected:
|
||||
parts = append(parts, styleLabel.Render("["+opt+"]"))
|
||||
default:
|
||||
parts = append(parts, styleMuted.Render(opt))
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
func boolToIdx(b bool) int {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func fieldIndex(fields []roleField, want roleField) int {
|
||||
for i, f := range fields {
|
||||
if f == want {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// validateRole checks one plane's entries. Cross-role rules (a shared NIC
|
||||
// without distinct VLAN ids, an unbound wan) belong to install.Config.Validate
|
||||
// and run when the operator continues.
|
||||
func (m model) validateRole(idx int) string {
|
||||
f := &m.roles[idx]
|
||||
if f.folded() {
|
||||
return ""
|
||||
}
|
||||
if len(m.nics) == 0 {
|
||||
return "No network interfaces detected"
|
||||
}
|
||||
|
||||
if !f.dhcp {
|
||||
if net.ParseIP(strings.TrimSpace(f.ip.Value())) == nil {
|
||||
return "Enter a valid IP address"
|
||||
}
|
||||
if !validSubnetMask(f.mask.Value()) {
|
||||
return "Enter a dotted-decimal subnet mask, e.g. 255.255.255.0 or 255.255.0.0"
|
||||
}
|
||||
if f.plane == install.PlaneWAN && net.ParseIP(strings.TrimSpace(f.gateway.Value())) == nil {
|
||||
return "Enter a valid gateway address"
|
||||
}
|
||||
}
|
||||
|
||||
if v := strings.TrimSpace(f.vlan.Value()); v != "" {
|
||||
id, err := strconv.Atoi(v)
|
||||
if err != nil || id < 1 || id > 4094 {
|
||||
return "VLAN id must be between 1 and 4094"
|
||||
}
|
||||
}
|
||||
if v := strings.TrimSpace(f.mtu.Value()); v != "" {
|
||||
mtu, err := strconv.Atoi(v)
|
||||
if err != nil || mtu < 68 || mtu > 9216 {
|
||||
return "MTU must be between 68 and 9216"
|
||||
}
|
||||
}
|
||||
if m.nicIsWiFi(f.nic) && strings.TrimSpace(f.ssid.Value()) == "" {
|
||||
return "Enter the WiFi network SSID"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func capitalise(s string) string {
|
||||
if s == "" {
|
||||
return s
|
||||
}
|
||||
return strings.ToUpper(s[:1]) + s[1:]
|
||||
}
|
||||
@@ -1,357 +0,0 @@
|
||||
package ui
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/installer/install"
|
||||
"github.com/mulgadc/spinifex/cmd/installer/netprobe"
|
||||
)
|
||||
|
||||
func nics(n int) []netprobe.NIC {
|
||||
out := make([]netprobe.NIC, 0, n)
|
||||
names := []string{"eno1", "ens1f0np0", "ens1f1np1", "ens2f0np0"}
|
||||
for i := range n {
|
||||
out = append(out, netprobe.NIC{Name: names[i], Speed: "25Gbps", Vendor: "Mellanox", State: "online"})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The pre-fill is the whole of the simple path: an operator with a
|
||||
// conventional node should be able to press Continue without editing anything.
|
||||
func TestNewModelPrefillsRolesByNICCount(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
count int
|
||||
wantLAN, wantVPC int
|
||||
}{
|
||||
{"one NIC folds everything onto wan", 1, foldedNIC, foldedNIC},
|
||||
{"two NICs dedicate the second to lan, vpc folds", 2, 1, foldedNIC},
|
||||
{"three NICs give each plane its own", 3, 1, 2},
|
||||
{"four NICs still only bind three", 4, 1, 2},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(tt.count))
|
||||
|
||||
if got := m.roles[0].nic; got != 0 {
|
||||
t.Errorf("wan nic = %d, want 0", got)
|
||||
}
|
||||
if got := m.roles[1].nic; got != tt.wantLAN {
|
||||
t.Errorf("lan nic = %d, want %d", got, tt.wantLAN)
|
||||
}
|
||||
if got := m.roles[2].nic; got != tt.wantVPC {
|
||||
t.Errorf("vpc nic = %d, want %d", got, tt.wantVPC)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Hiding rather than disabling is what keeps the default screen narrow.
|
||||
func TestVisibleFields(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*roleForm)
|
||||
isWiFi bool
|
||||
advanced bool
|
||||
want []roleField
|
||||
}{
|
||||
{
|
||||
name: "wan on DHCP asks two questions",
|
||||
mutate: func(f *roleForm) { f.dhcp = true },
|
||||
want: []roleField{roleFieldNIC, roleFieldMethod},
|
||||
},
|
||||
{
|
||||
name: "wan static adds a gateway",
|
||||
mutate: func(f *roleForm) { f.dhcp = false },
|
||||
want: []roleField{roleFieldNIC, roleFieldMethod, roleFieldIP, roleFieldMask, roleFieldGateway, roleFieldDNS},
|
||||
},
|
||||
{
|
||||
name: "advanced reveals VLAN and MTU",
|
||||
mutate: func(f *roleForm) { f.dhcp = true },
|
||||
advanced: true,
|
||||
want: []roleField{roleFieldNIC, roleFieldMethod, roleFieldVLAN, roleFieldMTU},
|
||||
},
|
||||
{
|
||||
name: "wireless adds credentials",
|
||||
mutate: func(f *roleForm) { f.dhcp = true },
|
||||
isWiFi: true,
|
||||
want: []roleField{roleFieldNIC, roleFieldMethod, roleFieldSSID, roleFieldWiFiPass},
|
||||
},
|
||||
{
|
||||
name: "a folded role has nothing to configure",
|
||||
mutate: func(f *roleForm) { f.nic = foldedNIC },
|
||||
want: []roleField{roleFieldNIC},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
f := newRoleForm(install.PlaneWAN, 0)
|
||||
tt.mutate(&f)
|
||||
got := f.visibleFields(tt.isWiFi, tt.advanced)
|
||||
|
||||
if len(got) != len(tt.want) {
|
||||
t.Fatalf("visibleFields() = %v, want %v", got, tt.want)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tt.want[i] {
|
||||
t.Fatalf("visibleFields() = %v, want %v", got, tt.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A non-wan role must be able to reach the folded state, and wan must not.
|
||||
func TestCycleNICFoldReachableOnlyOffWAN(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(2))
|
||||
|
||||
// lan: 1 -> 0 -> folded -> wraps to the last NIC.
|
||||
m.roles[1].nic = 1
|
||||
for _, want := range []int{0, foldedNIC, 1} {
|
||||
m = m.cycleNIC(1, false)
|
||||
if m.roles[1].nic != want {
|
||||
t.Fatalf("lan nic = %d, want %d", m.roles[1].nic, want)
|
||||
}
|
||||
}
|
||||
|
||||
// wan cycles only through real interfaces.
|
||||
m.roles[0].nic = 0
|
||||
for range 4 {
|
||||
m = m.cycleNIC(0, false)
|
||||
if m.roles[0].nic == foldedNIC {
|
||||
t.Fatal("wan reached the folded state, but it always needs an uplink")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestToRoleReadsVLANAndMTU(t *testing.T) {
|
||||
f := newRoleForm(install.PlaneLAN, 0)
|
||||
f.dhcp = false
|
||||
f.ip.SetValue("10.0.0.2")
|
||||
f.mask.SetValue("255.255.0.0")
|
||||
f.vlan.SetValue("10")
|
||||
f.mtu.SetValue("9000")
|
||||
|
||||
role := f.toRole(nics(1))
|
||||
|
||||
if role.VLAN != 10 || role.MTU != 9000 {
|
||||
t.Errorf("VLAN/MTU = %d/%d, want 10/9000", role.VLAN, role.MTU)
|
||||
}
|
||||
if role.Link() != "eno1.10" {
|
||||
t.Errorf("Link() = %q, want eno1.10", role.Link())
|
||||
}
|
||||
// Only wan carries a default route, even if a gateway was typed earlier.
|
||||
f.gateway.SetValue("10.0.0.1")
|
||||
if got := f.toRole(nics(1)).Gateway; got != "" {
|
||||
t.Errorf("lan Gateway = %q, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestToRoleFoldedIsEmpty(t *testing.T) {
|
||||
f := newRoleForm(install.PlaneVPC, foldedNIC)
|
||||
if !f.toRole(nics(2)).Folded() {
|
||||
t.Error("a folded form must produce a folded role")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRole(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
plane install.Plane
|
||||
mutate func(*roleForm)
|
||||
wantErr string
|
||||
}{
|
||||
{"valid static lan", install.PlaneLAN, func(f *roleForm) {
|
||||
f.dhcp = false
|
||||
f.ip.SetValue("10.0.0.2")
|
||||
f.mask.SetValue("255.255.0.0")
|
||||
}, ""},
|
||||
{"bad ip", install.PlaneLAN, func(f *roleForm) {
|
||||
f.dhcp = false
|
||||
f.ip.SetValue("not-an-ip")
|
||||
f.mask.SetValue("255.255.0.0")
|
||||
}, "valid IP"},
|
||||
{"wan needs a gateway", install.PlaneWAN, func(f *roleForm) {
|
||||
f.dhcp = false
|
||||
f.ip.SetValue("216.218.163.99")
|
||||
f.mask.SetValue("255.255.255.224")
|
||||
}, "gateway"},
|
||||
{"VLAN out of range", install.PlaneLAN, func(f *roleForm) {
|
||||
f.dhcp = true
|
||||
f.vlan.SetValue("9000")
|
||||
}, "VLAN id must be"},
|
||||
{"MTU out of range", install.PlaneLAN, func(f *roleForm) {
|
||||
f.dhcp = true
|
||||
f.mtu.SetValue("70000")
|
||||
}, "MTU must be"},
|
||||
{"folded roles skip validation", install.PlaneVPC, func(f *roleForm) {
|
||||
f.nic = foldedNIC
|
||||
f.ip.SetValue("nonsense")
|
||||
}, ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(3))
|
||||
m.roles[0] = newRoleForm(tt.plane, 0)
|
||||
tt.mutate(&m.roles[0])
|
||||
|
||||
got := m.validateRole(0)
|
||||
if tt.wantErr == "" {
|
||||
if got != "" {
|
||||
t.Fatalf("validateRole() = %q, want no error", got)
|
||||
}
|
||||
return
|
||||
}
|
||||
if !strings.Contains(got, tt.wantErr) {
|
||||
t.Fatalf("validateRole() = %q, want it to mention %q", got, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Continue must refuse a layout install.Config.Validate would reject, rather
|
||||
// than failing later during the install itself.
|
||||
func TestContinueBlocksOnSharedNICWithoutVLANs(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(2))
|
||||
m.screen = screenNetworkRoles
|
||||
m.roleCursor = continueRow
|
||||
m.roles[1].nic = 0 // lan onto the wan NIC, both untagged
|
||||
|
||||
next, _ := m.handleRolesKey("enter")
|
||||
got := next.(model)
|
||||
|
||||
if got.screen != screenNetworkRoles {
|
||||
t.Error("expected to stay on the roles screen when the layout is invalid")
|
||||
}
|
||||
if !strings.Contains(got.validationErr, "both bind eno1") {
|
||||
t.Errorf("validationErr = %q, want it to name the shared interface", got.validationErr)
|
||||
}
|
||||
|
||||
// Distinct VLAN ids make the same pairing legal.
|
||||
got.roles[0].vlan.SetValue("30")
|
||||
got.roles[1].vlan.SetValue("10")
|
||||
next, _ = got.handleRolesKey("enter")
|
||||
if next.(model).screen != screenIdentity {
|
||||
t.Error("distinct VLAN ids should allow two roles to share one NIC")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdvancedToggle(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(3))
|
||||
m.screen = screenNetworkRoles
|
||||
|
||||
next, _ := m.handleRolesKey("a")
|
||||
if !next.(model).advanced {
|
||||
t.Fatal("'a' should turn Advanced on")
|
||||
}
|
||||
next, _ = next.(model).handleRolesKey("a")
|
||||
if next.(model).advanced {
|
||||
t.Fatal("'a' should turn Advanced back off")
|
||||
}
|
||||
}
|
||||
|
||||
// The default route and the resolvers are node-level, and wan is the only plane
|
||||
// that carries either, so the east-west editors must not ask for them. Their
|
||||
// bridges are link-local and activate manually — a resolver pinned there is
|
||||
// unreachable, and a gateway would fight the real default route.
|
||||
func TestVisibleFieldsOmitsRoutingOffWAN(t *testing.T) {
|
||||
for _, plane := range []install.Plane{install.PlaneLAN, install.PlaneVPC} {
|
||||
t.Run(string(plane), func(t *testing.T) {
|
||||
f := newRoleForm(plane, 1)
|
||||
f.dhcp = false
|
||||
got := f.visibleFields(false, false)
|
||||
|
||||
want := []roleField{roleFieldNIC, roleFieldMethod, roleFieldIP, roleFieldMask}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("visibleFields() = %v, want %v", got, want)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("visibleFields() = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The mask field must accept only what netgen can consume. It previously
|
||||
// advertised and accepted a prefix length, which passed the TUI and then failed
|
||||
// the install itself with "invalid mask: 24".
|
||||
func TestValidSubnetMaskRejectsPrefixLength(t *testing.T) {
|
||||
for _, bad := range []string{"24", "/24", "8", "", "255.255.0", "255.0.255.0", "::1"} {
|
||||
if validSubnetMask(bad) {
|
||||
t.Errorf("validSubnetMask(%q) = true, want false", bad)
|
||||
}
|
||||
}
|
||||
for _, good := range []string{"255.255.255.0", "255.255.0.0", "255.0.0.0", "255.255.255.252"} {
|
||||
if !validSubnetMask(good) {
|
||||
t.Errorf("validSubnetMask(%q) = false, want true", good)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The hardware detail is the point of the NIC table: an operator has to be able
|
||||
// to tell which physical port each plane landed on. It gets a line of its own,
|
||||
// under the interface column, rather than being squeezed into the row.
|
||||
func TestRoleHardwareRendersOnItsOwnLine(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, []netprobe.NIC{{
|
||||
Name: "ens1f0np0", Vendor: "Mellanox Technologies", Model: "MT27710",
|
||||
AltName: "enp132s0f0np0", Speed: "25 Gbps", Carrier: true, State: "online",
|
||||
}})
|
||||
|
||||
view := m.viewNetworkRoles(100)
|
||||
if !strings.Contains(view, "ens1f0np0 25 Gbps online") {
|
||||
t.Errorf("expected the identity line in:\n%s", view)
|
||||
}
|
||||
detail := m.roleHardware(0, bodyWidth(100))
|
||||
if !strings.Contains(detail, "Mellanox Technologies MT27710") {
|
||||
t.Errorf("roleHardware = %q, want the vendor and model", detail)
|
||||
}
|
||||
if !strings.Contains(detail, "enp132s0f0np0") {
|
||||
t.Errorf("roleHardware = %q, want the alternative name", detail)
|
||||
}
|
||||
// Every line has to fit the box, or lipgloss wraps it and the columns shear.
|
||||
for line := range strings.SplitSeq(view, "\n") {
|
||||
if len([]rune(line)) > 100 {
|
||||
t.Errorf("line exceeds the terminal width: %q", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A narrow box drops the alternative name before it truncates the model, since
|
||||
// half a model name identifies nothing.
|
||||
func TestNICHardwareDropsAltNameBeforeTruncating(t *testing.T) {
|
||||
n := netprobe.NIC{Vendor: "Broadcom Inc. and subsidiaries", Model: "NetXtreme BCM5720", AltName: "enp1s0f1"}
|
||||
|
||||
if got := nicHardware(n, 60); got != "Broadcom Inc. and subsidiaries NetXtreme BCM5720" {
|
||||
t.Errorf("nicHardware(60) = %q, want the name without the alt name", got)
|
||||
}
|
||||
if got := nicHardware(n, 20); got != "Broadcom Inc. and s…" {
|
||||
t.Errorf("nicHardware(20) = %q, want a marked truncation", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A role bound to a port with no hardware database entry still gets something
|
||||
// to identify the card by, never the bare word "unknown".
|
||||
func TestRoleHardwareFallsBackToDriver(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, []netprobe.NIC{{
|
||||
Name: "eno1", Driver: "bnxt_en", DeviceID: "14e4:165f", State: "no cable",
|
||||
}})
|
||||
|
||||
if got := m.roleHardware(0, bodyWidth(100)); !strings.Contains(got, "bnxt_en [14e4:165f]") {
|
||||
t.Errorf("roleHardware = %q, want the driver and device id", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A folded role has no interface, so it must not render an orphan detail line
|
||||
// under the plane it collapsed onto.
|
||||
func TestRoleHardwareEmptyWhenFolded(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(1))
|
||||
if got := m.roleHardware(1, bodyWidth(100)); got != "" {
|
||||
t.Errorf("roleHardware for a folded role = %q, want empty", got)
|
||||
}
|
||||
}
|
||||
@@ -1,139 +0,0 @@
|
||||
package ui
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/cmd/installer/install"
|
||||
)
|
||||
|
||||
// diskModel is a storage screen with n unselected disks and ext4 chosen, which
|
||||
// is where the wizard starts.
|
||||
func diskModel(n int) model {
|
||||
m := model{fsCursor: slices.Index(install.AllFSTypes, install.FSExt4)}
|
||||
for i := range n {
|
||||
m.disks = append(m.disks, install.Disk{
|
||||
Path: string(rune('a'+i)) + "-disk",
|
||||
Bytes: 200 << 30,
|
||||
LogicalBlockSize: 512,
|
||||
PhysicalBlockSize: 512,
|
||||
})
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func rolePaths(cfg install.DiskConfig) map[install.DiskRole]string {
|
||||
out := map[install.DiskRole]string{}
|
||||
for _, rm := range cfg.Roles {
|
||||
out[rm.Role] = rm.Disk.Path
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// A single-disk machine should need no storage input at all, so the disk that
|
||||
// newModel preselects has to arrive already holding the os role.
|
||||
func TestPreselectedDiskIsAValidConfiguration(t *testing.T) {
|
||||
m := newModel([]install.Disk{{Path: "/dev/sda", Bytes: 200 << 30, LogicalBlockSize: 512}}, nil)
|
||||
cfg := m.storage()
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("the preselected disk must be installable as-is: %v", err)
|
||||
}
|
||||
if cfg.Primary().Path != "/dev/sda" {
|
||||
t.Errorf("Primary() = %q, want the preselected disk", cfg.Primary().Path)
|
||||
}
|
||||
}
|
||||
|
||||
// Picking disks in order gives the layout an operator almost always wants, so
|
||||
// the role key is only needed for the exceptions.
|
||||
func TestSelectingDisksAssignsRolesInOrder(t *testing.T) {
|
||||
m := diskModel(3)
|
||||
for i := range 3 {
|
||||
m = m.toggleDisk(i)
|
||||
}
|
||||
got := rolePaths(m.storage())
|
||||
for role, want := range map[install.DiskRole]string{
|
||||
install.RoleOS: "a-disk", install.RoleSpinifex: "b-disk", install.RolePredastore: "c-disk",
|
||||
} {
|
||||
if got[role] != want {
|
||||
t.Errorf("%s = %q, want %q", role, got[role], want)
|
||||
}
|
||||
}
|
||||
if err := m.storage().Validate(); err != nil {
|
||||
t.Errorf("the default assignment must be valid: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Deselecting has to drop the role with the disk, or every later role shifts
|
||||
// onto the wrong drive.
|
||||
func TestDeselectingADiskReleasesItsRole(t *testing.T) {
|
||||
m := diskModel(3)
|
||||
for i := range 3 {
|
||||
m = m.toggleDisk(i)
|
||||
}
|
||||
m = m.toggleDisk(1) // drop the spinifex drive
|
||||
|
||||
got := rolePaths(m.storage())
|
||||
if got[install.RoleOS] != "a-disk" || got[install.RolePredastore] != "c-disk" {
|
||||
t.Fatalf("remaining roles moved: %v", got)
|
||||
}
|
||||
if _, ok := got[install.RoleSpinifex]; ok {
|
||||
t.Errorf("the spinifex role should be free again: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The two-drive predastore layout is only reachable by reassigning, so cycling
|
||||
// swaps rather than refusing a role another disk already holds.
|
||||
func TestCycleRoleSwapsWithTheDiskThatHoldsIt(t *testing.T) {
|
||||
m := diskModel(2)
|
||||
m = m.toggleDisk(0).toggleDisk(1)
|
||||
// b-disk: spinifex → predastore, which no disk holds, so no swap.
|
||||
m = m.cycleRole(1)
|
||||
|
||||
got := rolePaths(m.storage())
|
||||
if got[install.RoleOS] != "a-disk" || got[install.RolePredastore] != "b-disk" {
|
||||
t.Fatalf("roles = %v, want os on a-disk and predastore on b-disk", got)
|
||||
}
|
||||
|
||||
// Cycling past the end wraps onto os, which a-disk holds, so they trade.
|
||||
m = m.cycleRole(1)
|
||||
got = rolePaths(m.storage())
|
||||
if got[install.RoleOS] != "b-disk" || got[install.RolePredastore] != "a-disk" {
|
||||
t.Fatalf("roles = %v, want the two disks to have swapped", got)
|
||||
}
|
||||
if err := m.storage().Validate(); err != nil {
|
||||
t.Errorf("a swap must always leave a valid assignment: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ZFS pairs mirrors by selection order and has no roles to assign.
|
||||
func TestStorageLeavesRolesUnsetOnZFS(t *testing.T) {
|
||||
m := diskModel(2)
|
||||
m.fsCursor = slices.Index(install.AllFSTypes, install.FSZFSRAID1)
|
||||
m = m.toggleDisk(0).toggleDisk(1)
|
||||
|
||||
cfg := m.storage()
|
||||
if len(cfg.Roles) != 0 {
|
||||
t.Errorf("Roles = %v, want none on a pool", cfg.Roles)
|
||||
}
|
||||
if !slices.Equal(cfg.Paths(), []string{"a-disk", "b-disk"}) {
|
||||
t.Errorf("Paths() = %v, want the selection order preserved", cfg.Paths())
|
||||
}
|
||||
}
|
||||
|
||||
// A three-drive selection looks like an array and is not one, so the mapping
|
||||
// and the absence of redundancy both have to be on screen before the erase.
|
||||
func TestConfirmScreenStatesTheLayoutAndTheRisk(t *testing.T) {
|
||||
m := diskModel(3)
|
||||
m.height = 40
|
||||
for i := range 3 {
|
||||
m = m.toggleDisk(i)
|
||||
}
|
||||
got := m.viewDiskConfirm(100)
|
||||
|
||||
for _, want := range []string{"/var/lib/spinifex/predastore", "c-disk", "losing one loses"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("confirm screen is missing %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,896 +0,0 @@
|
||||
//test:in-package — the wizard model, its screen constants and the key handlers
|
||||
//under test are all unexported, and the tests read model fields directly to
|
||||
//assert which screen a key sequence landed on.
|
||||
|
||||
package ui
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"github.com/mulgadc/spinifex/cmd/installer/install"
|
||||
)
|
||||
|
||||
// namedKeys maps the key names the handlers switch on to their tea.KeyType.
|
||||
// Anything not listed is sent as runes.
|
||||
var namedKeys = map[string]tea.KeyType{
|
||||
"enter": tea.KeyEnter,
|
||||
"esc": tea.KeyEsc,
|
||||
"tab": tea.KeyTab,
|
||||
"shift+tab": tea.KeyShiftTab,
|
||||
"up": tea.KeyUp,
|
||||
"down": tea.KeyDown,
|
||||
"left": tea.KeyLeft,
|
||||
"right": tea.KeyRight,
|
||||
" ": tea.KeySpace,
|
||||
"backspace": tea.KeyBackspace,
|
||||
"ctrl+c": tea.KeyCtrlC,
|
||||
}
|
||||
|
||||
// keyMsg builds the tea.KeyMsg for a key name and checks it round-trips, so a
|
||||
// mis-built message fails here rather than silently exercising the wrong branch.
|
||||
func keyMsg(t *testing.T, key string) tea.KeyMsg {
|
||||
t.Helper()
|
||||
msg := tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(key)}
|
||||
if kt, ok := namedKeys[key]; ok {
|
||||
msg = tea.KeyMsg{Type: kt}
|
||||
}
|
||||
if got := msg.String(); got != key {
|
||||
t.Fatalf("built key %q, which the handlers see as %q", key, got)
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
// press feeds keys through Update, which is the only entry point the bubbletea
|
||||
// runtime uses, and returns the resulting model.
|
||||
func press(t *testing.T, m model, keys ...string) model {
|
||||
t.Helper()
|
||||
for _, k := range keys {
|
||||
next, _ := m.Update(keyMsg(t, k))
|
||||
typed, ok := next.(model)
|
||||
if !ok {
|
||||
t.Fatalf("Update returned %T, not a model", next)
|
||||
}
|
||||
m = typed
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// typeText sends a string one rune at a time, the way a terminal does.
|
||||
func typeText(t *testing.T, m model, s string) model {
|
||||
t.Helper()
|
||||
for _, r := range s {
|
||||
m = press(t, m, string(r))
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func wizardModel(t *testing.T, diskCount, nicCount int) model {
|
||||
t.Helper()
|
||||
disks := make([]install.Disk, 0, diskCount)
|
||||
for i := range diskCount {
|
||||
disks = append(disks, install.Disk{
|
||||
Path: "/dev/sd" + string(rune('a'+i)),
|
||||
Bytes: 200 << 30,
|
||||
LogicalBlockSize: 512,
|
||||
})
|
||||
}
|
||||
m := newModel(disks, nics(nicCount))
|
||||
m.width, m.height = 100, 40
|
||||
return m
|
||||
}
|
||||
|
||||
// The wizard has to be completable from the welcome screen using only the keys
|
||||
// the help lines advertise, and produce the config the operator entered.
|
||||
func TestWizardWalkthroughProducesTheEnteredConfig(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
|
||||
m = press(t, m, "enter") // welcome → disk
|
||||
if m.screen != screenDisk {
|
||||
t.Fatalf("screen = %v, want the disk screen", m.screen)
|
||||
}
|
||||
|
||||
m = press(t, m, "enter") // preselected disk is valid → erase confirmation
|
||||
if m.screen != screenDiskConfirm {
|
||||
t.Fatalf("screen = %v, want the erase confirmation", m.screen)
|
||||
}
|
||||
|
||||
m = typeText(t, m, "yes")
|
||||
m = press(t, m, "enter") // → network roles
|
||||
if m.screen != screenNetworkRoles {
|
||||
t.Fatalf("screen = %v, want the network roles screen", m.screen)
|
||||
}
|
||||
|
||||
m = press(t, m, "down", "down", "down", "enter") // Continue → identity
|
||||
if m.screen != screenIdentity {
|
||||
t.Fatalf("screen = %v, want the identity screen", m.screen)
|
||||
}
|
||||
|
||||
m = typeText(t, m, "node1")
|
||||
m = press(t, m, "enter") // → credentials
|
||||
if m.screen != screenPassword {
|
||||
t.Fatalf("screen = %v, want the credentials screen", m.screen)
|
||||
}
|
||||
|
||||
m = typeText(t, m, "admin@example.com")
|
||||
m = press(t, m, "enter") // email accepted → password
|
||||
m = typeText(t, m, "hunter2hunter2")
|
||||
m = press(t, m, "enter") // → confirm password
|
||||
m = typeText(t, m, "hunter2hunter2")
|
||||
m = press(t, m, "enter") // → final confirmation
|
||||
if m.screen != screenConfirm {
|
||||
t.Fatalf("screen = %v, want the final confirmation (err: %q)", m.screen, m.validationErr)
|
||||
}
|
||||
|
||||
m = press(t, m, "y")
|
||||
if m.screen != screenDone {
|
||||
t.Fatalf("screen = %v, want done", m.screen)
|
||||
}
|
||||
if m.result == nil {
|
||||
t.Fatal("result is nil after confirming")
|
||||
}
|
||||
|
||||
cfg := m.result
|
||||
if cfg.Hostname != "node1" {
|
||||
t.Errorf("Hostname = %q, want node1", cfg.Hostname)
|
||||
}
|
||||
if cfg.Email != "admin@example.com" {
|
||||
t.Errorf("Email = %q, want admin@example.com", cfg.Email)
|
||||
}
|
||||
if cfg.RootPassword != "hunter2hunter2" {
|
||||
t.Errorf("RootPassword = %q, want what was typed", cfg.RootPassword)
|
||||
}
|
||||
if !slices.Equal(cfg.Storage.Paths(), []string{"/dev/sda"}) {
|
||||
t.Errorf("Storage.Paths() = %v, want the preselected disk", cfg.Storage.Paths())
|
||||
}
|
||||
if cfg.WAN.Interface != "eno1" {
|
||||
t.Errorf("WAN interface = %q, want the first NIC", cfg.WAN.Interface)
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Errorf("the wizard produced a config it would refuse to install: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Esc has to walk back the way it came, or an operator who overshoots is stuck
|
||||
// with a wizard they can only cancel out of.
|
||||
func TestEscapeWalksBackThroughTheScreens(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m = press(t, m, "enter") // disk
|
||||
m = press(t, m, "enter") // erase confirmation
|
||||
|
||||
for _, step := range []struct {
|
||||
key string
|
||||
want screen
|
||||
}{
|
||||
{"esc", screenDisk},
|
||||
{"esc", screenWelcome},
|
||||
} {
|
||||
m = press(t, m, step.key)
|
||||
if m.screen != step.want {
|
||||
t.Fatalf("after esc: screen = %v, want %v", m.screen, step.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Ctrl+C is handled before any screen sees the key, so it works everywhere.
|
||||
func TestCtrlCCancelsWithAnError(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1)
|
||||
next, cmd := m.Update(keyMsg(t, "ctrl+c"))
|
||||
final, ok := next.(model)
|
||||
if !ok {
|
||||
t.Fatalf("Update returned %T", next)
|
||||
}
|
||||
if final.err == nil || !strings.Contains(final.err.Error(), "cancelled") {
|
||||
t.Fatalf("err = %v, want a cancellation", final.err)
|
||||
}
|
||||
if cmd == nil {
|
||||
t.Error("ctrl+c must return tea.Quit")
|
||||
}
|
||||
}
|
||||
|
||||
// A window resize has to reach the model, since every view lays out against it.
|
||||
func TestWindowResizeIsStored(t *testing.T) {
|
||||
m := model{}
|
||||
next, _ := m.Update(tea.WindowSizeMsg{Width: 120, Height: 50})
|
||||
resized, ok := next.(model)
|
||||
if !ok {
|
||||
t.Fatalf("Update returned %T", next)
|
||||
}
|
||||
if resized.width != 120 || resized.height != 50 {
|
||||
t.Errorf("size = %dx%d, want 120x50", resized.width, resized.height)
|
||||
}
|
||||
}
|
||||
|
||||
// The erase prompt is the last point of no return, so anything other than an
|
||||
// explicit "yes" has to keep the operator on it.
|
||||
func TestEraseConfirmationRequiresYes(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1)
|
||||
m = press(t, m, "enter", "enter") // → erase confirmation
|
||||
|
||||
m = typeText(t, m, "no")
|
||||
m = press(t, m, "enter")
|
||||
if m.screen != screenDiskConfirm {
|
||||
t.Fatalf("screen = %v, want to stay on the confirmation", m.screen)
|
||||
}
|
||||
if !strings.Contains(m.validationErr, "yes") {
|
||||
t.Errorf("validationErr = %q, want it to say what to type", m.validationErr)
|
||||
}
|
||||
if !strings.Contains(m.viewDiskConfirm(100), m.validationErr) {
|
||||
t.Error("the confirmation screen must show the error it set")
|
||||
}
|
||||
}
|
||||
|
||||
// The filesystem selector and the disk list share one screen, so the cursor has
|
||||
// to move between them and stop at both ends.
|
||||
func TestDiskScreenCursorStaysInRange(t *testing.T) {
|
||||
m := wizardModel(t, 2, 1)
|
||||
m.screen = screenDisk
|
||||
|
||||
m = press(t, m, "up", "up")
|
||||
if m.storageCursor != fsRow {
|
||||
t.Errorf("storageCursor = %d, want it clamped to the filesystem row", m.storageCursor)
|
||||
}
|
||||
|
||||
m = press(t, m, "down", "down", "down", "down")
|
||||
if m.storageCursor != len(m.disks) {
|
||||
t.Errorf("storageCursor = %d, want it clamped to the last disk", m.storageCursor)
|
||||
}
|
||||
}
|
||||
|
||||
// Left/right only cycle the filesystem while the cursor is on that row, and
|
||||
// they wrap, so one key reaches every topology.
|
||||
func TestFilesystemCyclesOnlyOnItsOwnRow(t *testing.T) {
|
||||
m := wizardModel(t, 2, 1)
|
||||
m.screen = screenDisk
|
||||
|
||||
m = press(t, m, "left")
|
||||
if got := m.fsType(); got != install.AllFSTypes[len(install.AllFSTypes)-1] {
|
||||
t.Errorf("fsType = %v, want the cycle to wrap backwards to the last", got)
|
||||
}
|
||||
m = press(t, m, "right")
|
||||
if got := m.fsType(); got != install.AllFSTypes[0] {
|
||||
t.Errorf("fsType = %v, want the cycle to wrap forwards to the first", got)
|
||||
}
|
||||
|
||||
m = press(t, m, "down") // onto the first disk
|
||||
before := m.fsType()
|
||||
m = press(t, m, "left", "right")
|
||||
if m.fsType() != before {
|
||||
t.Errorf("fsType changed to %v while the cursor was on a disk", m.fsType())
|
||||
}
|
||||
}
|
||||
|
||||
// Space toggles the disk under the cursor, and r cycles its role — the two keys
|
||||
// the storage screen exists for.
|
||||
func TestDiskScreenSelectionKeys(t *testing.T) {
|
||||
m := wizardModel(t, 2, 1)
|
||||
m.screen = screenDisk
|
||||
|
||||
// The second disk is unselected; the first arrives preselected as os.
|
||||
m = press(t, m, "down", "down", " ")
|
||||
if !slices.Contains(m.selected, 1) {
|
||||
t.Fatalf("selected = %v, want the second disk added", m.selected)
|
||||
}
|
||||
|
||||
roleBefore := m.roleOf[slices.Index(m.selected, 1)]
|
||||
m = press(t, m, "r")
|
||||
if got := m.roleOf[slices.Index(m.selected, 1)]; got == roleBefore {
|
||||
t.Errorf("role stayed %q after r", got)
|
||||
}
|
||||
|
||||
m = press(t, m, " ")
|
||||
if slices.Contains(m.selected, 1) {
|
||||
t.Fatalf("selected = %v, want the second disk removed again", m.selected)
|
||||
}
|
||||
if len(m.roleOf) != len(m.selected) {
|
||||
t.Errorf("roleOf (%d) and selected (%d) drifted", len(m.roleOf), len(m.selected))
|
||||
}
|
||||
}
|
||||
|
||||
// A selection that cannot be installed has to be reported on the screen the
|
||||
// operator can still fix it on, not part-way through the install.
|
||||
func TestDiskScreenRefusesAnInvalidSelection(t *testing.T) {
|
||||
m := wizardModel(t, 2, 1)
|
||||
m.screen = screenDisk
|
||||
m.fsCursor = slices.Index(install.AllFSTypes, install.FSZFSRAID1) // needs 2 disks
|
||||
|
||||
m = press(t, m, "enter")
|
||||
if m.screen != screenDisk {
|
||||
t.Fatalf("screen = %v, want to stay on the disk screen", m.screen)
|
||||
}
|
||||
if m.validationErr == "" {
|
||||
t.Fatal("an invalid selection must set a validation error")
|
||||
}
|
||||
if !strings.Contains(m.viewDisk(100), m.validationErr) {
|
||||
t.Error("the disk screen must render the error it set")
|
||||
}
|
||||
|
||||
// Selecting the second disk satisfies the mirror and clears the way.
|
||||
m = press(t, m, "down", "down", " ", "enter")
|
||||
if m.screen != screenDiskConfirm {
|
||||
t.Fatalf("screen = %v, want to advance once the mirror has two disks", m.screen)
|
||||
}
|
||||
}
|
||||
|
||||
// The ZFS options screen is only reachable on a pool, since its settings have
|
||||
// no meaning on ext4.
|
||||
func TestZFSOptionsScreenIsZFSOnly(t *testing.T) {
|
||||
m := wizardModel(t, 2, 1)
|
||||
m.screen = screenDisk
|
||||
|
||||
m = press(t, m, "a")
|
||||
if m.screen != screenDisk {
|
||||
t.Fatalf("screen = %v, want a to do nothing on ext4", m.screen)
|
||||
}
|
||||
|
||||
m.fsCursor = slices.Index(install.AllFSTypes, install.FSZFSRAID1)
|
||||
m = press(t, m, "a")
|
||||
if m.screen != screenZFSOptions {
|
||||
t.Fatalf("screen = %v, want the ZFS options screen", m.screen)
|
||||
}
|
||||
}
|
||||
|
||||
// Every option defaults to auto and cycles through its own choice list, with
|
||||
// the cursor clamped to the rows that exist.
|
||||
func TestZFSOptionsCycleTheirValues(t *testing.T) {
|
||||
m := wizardModel(t, 2, 1)
|
||||
m.screen = screenZFSOptions
|
||||
|
||||
if got := zfsOptions[0].value(&m.zfs); got != "auto" {
|
||||
t.Errorf("ashift starts at %q, want auto", got)
|
||||
}
|
||||
m = press(t, m, "right")
|
||||
if got := zfsOptions[0].value(&m.zfs); got == "auto" {
|
||||
t.Error("right must move ashift off auto")
|
||||
}
|
||||
m = press(t, m, "left")
|
||||
if got := zfsOptions[0].value(&m.zfs); got != "auto" {
|
||||
t.Errorf("ashift = %q after cycling back, want auto", got)
|
||||
}
|
||||
|
||||
m = press(t, m, "up", "up")
|
||||
if m.zfsCursor != 0 {
|
||||
t.Errorf("zfsCursor = %d, want it clamped to the first row", m.zfsCursor)
|
||||
}
|
||||
for range len(zfsOptions) + 2 {
|
||||
m = press(t, m, "down")
|
||||
}
|
||||
if m.zfsCursor != len(zfsOptions)-1 {
|
||||
t.Errorf("zfsCursor = %d, want it clamped to the last row", m.zfsCursor)
|
||||
}
|
||||
|
||||
// The compression choices are strings rather than ints, so they exercise
|
||||
// the other half of cycle.
|
||||
m.zfsCursor = 1
|
||||
m = press(t, m, "right")
|
||||
if m.zfs.Compress == "" {
|
||||
t.Error("right must set a compression algorithm")
|
||||
}
|
||||
|
||||
m = press(t, m, "enter")
|
||||
if m.screen != screenDisk {
|
||||
t.Fatalf("screen = %v, want enter to return to the disk screen", m.screen)
|
||||
}
|
||||
}
|
||||
|
||||
// The screen has to show what each option is currently set to, since that is
|
||||
// the only place the pool's geometry is stated before it is created.
|
||||
func TestZFSOptionsViewShowsValuesAndTheFocusedNote(t *testing.T) {
|
||||
m := wizardModel(t, 2, 1)
|
||||
m.screen = screenZFSOptions
|
||||
m.zfs.Compress = "zstd"
|
||||
|
||||
got := m.viewZFSOptions(100)
|
||||
for _, want := range []string{"ashift", "auto", "compression", "zstd", zfsOptions[0].note} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("ZFS options screen is missing %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
if strings.Contains(got, zfsOptions[1].note) {
|
||||
t.Error("only the focused row's note should be shown")
|
||||
}
|
||||
}
|
||||
|
||||
// Credentials are three fields on one screen, so focus has to cycle both ways
|
||||
// and land somewhere sensible after a validation failure.
|
||||
func TestCredentialsFocusCyclesBothWays(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1)
|
||||
m.screen = screenPassword
|
||||
m = m.setCredsFocus(0)
|
||||
|
||||
m = press(t, m, "tab")
|
||||
if m.credsFocus != 1 || !m.passwordInput.Focused() {
|
||||
t.Fatalf("credsFocus = %d, want the password field focused", m.credsFocus)
|
||||
}
|
||||
m = press(t, m, "tab", "tab")
|
||||
if m.credsFocus != 0 || !m.emailInput.Focused() {
|
||||
t.Fatalf("credsFocus = %d, want the cycle to wrap to email", m.credsFocus)
|
||||
}
|
||||
m = press(t, m, "shift+tab")
|
||||
if m.credsFocus != 2 || !m.passwordConfirmInput.Focused() {
|
||||
t.Fatalf("credsFocus = %d, want shift+tab to wrap backwards to confirm", m.credsFocus)
|
||||
}
|
||||
}
|
||||
|
||||
// Each credential rule has to stop the wizard on the screen and put the cursor
|
||||
// on the field at fault, not just refuse to advance.
|
||||
func TestCredentialValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
email string
|
||||
password string
|
||||
confirm string
|
||||
wantErr string
|
||||
wantFocus int
|
||||
wantScreen screen
|
||||
}{
|
||||
{
|
||||
name: "rejects a malformed email", email: "not-an-email",
|
||||
password: "hunter2hunter2", confirm: "hunter2hunter2",
|
||||
wantErr: "email", wantFocus: 0, wantScreen: screenPassword,
|
||||
},
|
||||
{
|
||||
name: "rejects an empty password", email: "admin@example.com",
|
||||
wantErr: "Password is required", wantFocus: 1, wantScreen: screenPassword,
|
||||
},
|
||||
{
|
||||
name: "rejects a mismatch", email: "admin@example.com",
|
||||
password: "hunter2hunter2", confirm: "hunter2hunter3",
|
||||
wantErr: "do not match", wantFocus: 2, wantScreen: screenPassword,
|
||||
},
|
||||
{
|
||||
name: "accepts a matching pair", email: "admin@example.com",
|
||||
password: "hunter2hunter2", confirm: "hunter2hunter2",
|
||||
wantScreen: screenConfirm,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1)
|
||||
m.screen = screenPassword
|
||||
m.emailInput.SetValue(tt.email)
|
||||
m.passwordInput.SetValue(tt.password)
|
||||
m.passwordConfirmInput.SetValue(tt.confirm)
|
||||
m = m.setCredsFocus(2) // enter on confirm runs every rule
|
||||
|
||||
m = press(t, m, "enter")
|
||||
|
||||
if m.screen != tt.wantScreen {
|
||||
t.Fatalf("screen = %v, want %v (err %q)", m.screen, tt.wantScreen, m.validationErr)
|
||||
}
|
||||
if tt.wantErr == "" {
|
||||
if m.validationErr != "" {
|
||||
t.Fatalf("validationErr = %q, want none", m.validationErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if !strings.Contains(strings.ToLower(m.validationErr), strings.ToLower(tt.wantErr)) {
|
||||
t.Errorf("validationErr = %q, want it to mention %q", m.validationErr, tt.wantErr)
|
||||
}
|
||||
if m.credsFocus != tt.wantFocus {
|
||||
t.Errorf("credsFocus = %d, want the cursor on field %d", m.credsFocus, tt.wantFocus)
|
||||
}
|
||||
if !strings.Contains(m.viewPassword(100), m.validationErr) {
|
||||
t.Error("the credentials screen must render the error it set")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The hostname ends up in the config and in DNS, so an empty one has to be
|
||||
// caught on the screen rather than installed.
|
||||
func TestHostnameIsRequired(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1)
|
||||
m.screen = screenIdentity
|
||||
m.hostnameInput.Focus()
|
||||
|
||||
m = press(t, m, "enter")
|
||||
if m.screen != screenIdentity {
|
||||
t.Fatalf("screen = %v, want to stay on identity", m.screen)
|
||||
}
|
||||
if !strings.Contains(m.validationErr, "Hostname") {
|
||||
t.Errorf("validationErr = %q, want it to name the field", m.validationErr)
|
||||
}
|
||||
if !m.hostnameInput.Focused() {
|
||||
t.Error("the hostname field must keep focus after a failed submit")
|
||||
}
|
||||
if !strings.Contains(m.viewIdentity(100), m.validationErr) {
|
||||
t.Error("the identity screen must render the error it set")
|
||||
}
|
||||
}
|
||||
|
||||
// Tab order follows the visible fields, which change with the addressing mode,
|
||||
// so it has to be derived from them rather than fixed.
|
||||
func TestRoleEditorTabsThroughVisibleFieldsOnly(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.screen = screenNetworkRole
|
||||
m.roleCursor = 0 // wan, DHCP by default
|
||||
|
||||
m = press(t, m, "tab")
|
||||
if m.roles[0].focus != roleFieldMethod {
|
||||
t.Fatalf("focus = %v, want the IP method field", m.roles[0].focus)
|
||||
}
|
||||
m = press(t, m, "tab")
|
||||
if m.roles[0].focus != roleFieldNIC {
|
||||
t.Fatalf("focus = %v, want the DHCP form to wrap after two fields", m.roles[0].focus)
|
||||
}
|
||||
|
||||
// Switching to static reveals the address fields, and tab must reach them.
|
||||
m = press(t, m, "tab", "right")
|
||||
if m.roles[0].dhcp {
|
||||
t.Fatal("right on the method field must select static addressing")
|
||||
}
|
||||
m = press(t, m, "tab")
|
||||
if m.roles[0].focus != roleFieldIP || !m.roles[0].ip.Focused() {
|
||||
t.Fatalf("focus = %v, want the IP field focused", m.roles[0].focus)
|
||||
}
|
||||
m = press(t, m, "shift+tab")
|
||||
if m.roles[0].focus != roleFieldMethod {
|
||||
t.Fatalf("focus = %v, want shift+tab to go back", m.roles[0].focus)
|
||||
}
|
||||
}
|
||||
|
||||
// A focused text field has to receive typed runes, since that is the only way
|
||||
// an address is entered.
|
||||
func TestRoleEditorTypesIntoTheFocusedField(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.screen = screenNetworkRole
|
||||
m.roles[0].dhcp = false
|
||||
m.roles[0].focus = roleFieldIP
|
||||
m.roles[0].focusCurrent()
|
||||
|
||||
m = typeText(t, m, "192.168.1.10")
|
||||
if got := m.roles[0].ip.Value(); got != "192.168.1.10" {
|
||||
t.Fatalf("ip = %q, want what was typed", got)
|
||||
}
|
||||
if got := m.roles[0].mask.Value(); got != "" {
|
||||
t.Errorf("mask = %q, want the keys to reach only the focused field", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The interface selector wraps and offers the fold option on lan and vpc only,
|
||||
// so a single-NIC node stays configurable with one key.
|
||||
func TestNICSelectorWrapsAndFoldsWhereAllowed(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.screen = screenNetworkRole
|
||||
|
||||
// wan cannot fold, so it wraps across the real NICs only.
|
||||
m.roleCursor = 0
|
||||
m = press(t, m, "left")
|
||||
if m.roles[0].nic != len(m.nics)-1 {
|
||||
t.Fatalf("wan nic = %d, want it to wrap to the last NIC", m.roles[0].nic)
|
||||
}
|
||||
m = press(t, m, "right")
|
||||
if m.roles[0].nic != 0 {
|
||||
t.Fatalf("wan nic = %d, want it to wrap back to the first", m.roles[0].nic)
|
||||
}
|
||||
|
||||
// lan may fold, so the cycle includes the unbound position.
|
||||
m.roleCursor = 1
|
||||
m.roles[1].nic = 0
|
||||
m = press(t, m, "left")
|
||||
if m.roles[1].nic != foldedNIC {
|
||||
t.Fatalf("lan nic = %d, want the fold position", m.roles[1].nic)
|
||||
}
|
||||
if !m.roles[1].folded() {
|
||||
t.Error("the lan role should read as folded")
|
||||
}
|
||||
}
|
||||
|
||||
// Per-role validation runs on Enter and keeps the operator on the editor, since
|
||||
// an invalid address here fails the install after the disk is partitioned.
|
||||
func TestRoleEditorValidatesBeforeLeaving(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.screen = screenNetworkRole
|
||||
m.roleCursor = 0
|
||||
m.roles[0].dhcp = false
|
||||
m.roles[0].ip.SetValue("not-an-ip")
|
||||
|
||||
m = press(t, m, "enter")
|
||||
if m.screen != screenNetworkRole {
|
||||
t.Fatalf("screen = %v, want to stay on the role editor", m.screen)
|
||||
}
|
||||
if !strings.Contains(m.validationErr, "IP address") {
|
||||
t.Errorf("validationErr = %q, want it to name the field", m.validationErr)
|
||||
}
|
||||
if !strings.Contains(m.viewNetworkRole(100), m.validationErr) {
|
||||
t.Error("the role editor must render the error it set")
|
||||
}
|
||||
|
||||
// A prefix length is rejected here because netgen parses the mask with
|
||||
// net.ParseIP and would fail the install itself.
|
||||
m.roles[0].ip.SetValue("192.168.1.10")
|
||||
m.roles[0].mask.SetValue("24")
|
||||
m = press(t, m, "enter")
|
||||
if !strings.Contains(m.validationErr, "dotted-decimal") {
|
||||
t.Errorf("validationErr = %q, want the mask rule", m.validationErr)
|
||||
}
|
||||
|
||||
m.roles[0].mask.SetValue("255.255.255.0")
|
||||
m.roles[0].gateway.SetValue("192.168.1.1")
|
||||
m = press(t, m, "enter")
|
||||
if m.screen != screenNetworkRoles {
|
||||
t.Fatalf("screen = %v, want a valid role to return to the overview (err %q)", m.screen, m.validationErr)
|
||||
}
|
||||
}
|
||||
|
||||
// Esc abandons the editor without validating, so a half-typed address is not a
|
||||
// trap the operator has to finish before backing out.
|
||||
func TestRoleEditorEscapeSkipsValidation(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.screen = screenNetworkRole
|
||||
m.roles[0].dhcp = false
|
||||
m.roles[0].ip.SetValue("192.")
|
||||
|
||||
m = press(t, m, "esc")
|
||||
if m.screen != screenNetworkRoles {
|
||||
t.Fatalf("screen = %v, want the overview", m.screen)
|
||||
}
|
||||
if m.validationErr != "" {
|
||||
t.Errorf("validationErr = %q, want esc to clear it", m.validationErr)
|
||||
}
|
||||
}
|
||||
|
||||
// The editor shows only the fields that apply, so the DHCP form stays two lines
|
||||
// and Advanced is what reveals VLAN and MTU.
|
||||
func TestRoleEditorViewShowsTheApplicableFields(t *testing.T) {
|
||||
m := wizardModel(t, 1, 3)
|
||||
m.screen = screenNetworkRole
|
||||
m.roleCursor = 0
|
||||
|
||||
dhcpView := m.viewNetworkRole(100)
|
||||
if !strings.Contains(dhcpView, "DHCP") || !strings.Contains(dhcpView, "WAN") {
|
||||
t.Errorf("DHCP view is missing the plane or the method:\n%s", dhcpView)
|
||||
}
|
||||
for _, hidden := range []string{"IP address", "Subnet mask", "VLAN id"} {
|
||||
if strings.Contains(dhcpView, hidden) {
|
||||
t.Errorf("DHCP view should not show %q:\n%s", hidden, dhcpView)
|
||||
}
|
||||
}
|
||||
|
||||
m.roles[0].dhcp = false
|
||||
m.advanced = true
|
||||
staticView := m.viewNetworkRole(100)
|
||||
for _, want := range []string{"IP address", "Subnet mask", "Gateway", "DNS", "VLAN id", "MTU"} {
|
||||
if !strings.Contains(staticView, want) {
|
||||
t.Errorf("static advanced view is missing %q:\n%s", want, staticView)
|
||||
}
|
||||
}
|
||||
|
||||
// Gateway and DNS belong to the plane holding the default route, so a
|
||||
// static lan must not offer them.
|
||||
m.roleCursor = 1
|
||||
m.roles[1].nic = 1
|
||||
m.roles[1].dhcp = false
|
||||
lanView := m.viewNetworkRole(100)
|
||||
for _, hidden := range []string{"Gateway", "DNS"} {
|
||||
if strings.Contains(lanView, hidden) {
|
||||
t.Errorf("static lan view should not show %q:\n%s", hidden, lanView)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A folded role names the plane it landed on, on both screens, so the operator
|
||||
// can see where the traffic actually goes before committing.
|
||||
func TestFoldedRolesAreNamedNotBlank(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1) // one NIC folds lan and vpc onto wan
|
||||
|
||||
overview := m.viewNetworkRoles(100)
|
||||
if !strings.Contains(overview, "folds onto") {
|
||||
t.Errorf("overview does not state where the folded roles land:\n%s", overview)
|
||||
}
|
||||
|
||||
m.screen = screenNetworkRole
|
||||
m.roleCursor = 2
|
||||
if got := m.viewNetworkRole(100); !strings.Contains(got, "fold onto") {
|
||||
t.Errorf("role editor does not offer the fold plainly:\n%s", got)
|
||||
}
|
||||
|
||||
m.screen = screenConfirm
|
||||
if got := m.viewConfirm(100); !strings.Contains(got, "folded onto") {
|
||||
t.Errorf("confirmation does not state the folded planes:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Continue is refused while the config would fail, and the message is the one
|
||||
// install.Config.Validate produced.
|
||||
func TestRolesOverviewRefusesAnInvalidConfig(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.screen = screenNetworkRoles
|
||||
m.roles[1].nic = 0 // lan on the same NIC as wan, no VLAN to separate them
|
||||
m.roleCursor = continueRow
|
||||
|
||||
m = press(t, m, "enter")
|
||||
if m.screen != screenNetworkRoles {
|
||||
t.Fatalf("screen = %v, want to stay on the overview", m.screen)
|
||||
}
|
||||
if !strings.Contains(m.validationErr, "VLAN") {
|
||||
t.Errorf("validationErr = %q, want the shared-NIC rule", m.validationErr)
|
||||
}
|
||||
// The message is wrapped to the box, so match its opening rather than
|
||||
// the whole line.
|
||||
if !strings.Contains(m.viewNetworkRoles(100), "both bind eno1") {
|
||||
t.Error("the overview must render the error it set")
|
||||
}
|
||||
|
||||
// a toggles Advanced for the whole node rather than per role.
|
||||
m = press(t, m, "a")
|
||||
if !m.advanced {
|
||||
t.Error("a must turn Advanced on")
|
||||
}
|
||||
if !strings.Contains(m.viewNetworkRoles(100), "Advanced: on") {
|
||||
t.Error("the overview must show Advanced is on")
|
||||
}
|
||||
}
|
||||
|
||||
// The final screen is the last chance to catch a wrong disk or address, so the
|
||||
// summary has to carry what was entered.
|
||||
func TestConfirmScreenSummarisesTheEnteredConfig(t *testing.T) {
|
||||
m := wizardModel(t, 1, 3)
|
||||
m.screen = screenConfirm
|
||||
m.hostnameInput.SetValue("node7")
|
||||
m.roles[1].dhcp = false
|
||||
m.roles[1].ip.SetValue("10.0.0.3")
|
||||
m.roles[1].mask.SetValue("255.255.255.0")
|
||||
|
||||
got := m.viewConfirm(100)
|
||||
for _, want := range []string{"node7", "/dev/sda", "ext4", "eno1", "10.0.0.3/255.255.255.0", "br-lan", "will erase"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("confirmation is missing %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// n on the confirmation cancels rather than installing.
|
||||
func TestConfirmScreenCancelsOnN(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1)
|
||||
m.screen = screenConfirm
|
||||
|
||||
m = press(t, m, "n")
|
||||
if m.err == nil {
|
||||
t.Fatal("n must set the cancellation error")
|
||||
}
|
||||
if m.result != nil {
|
||||
t.Error("a cancelled install must produce no config")
|
||||
}
|
||||
}
|
||||
|
||||
// View dispatches on the screen, so each one has to render its own content
|
||||
// rather than falling through to an empty frame.
|
||||
func TestViewRendersEveryScreen(t *testing.T) {
|
||||
tests := []struct {
|
||||
screen screen
|
||||
want string
|
||||
}{
|
||||
{screenWelcome, "Enter to begin"},
|
||||
{screenDisk, "Select Installation Disk"},
|
||||
{screenZFSOptions, "ZFS Options"},
|
||||
{screenDiskConfirm, "Confirm Disk Erasure"},
|
||||
{screenNetworkRoles, "Network Planes"},
|
||||
{screenNetworkRole, "Configure WAN plane"},
|
||||
{screenIdentity, "Node Identity"},
|
||||
{screenPassword, "Administrator Credentials"},
|
||||
{screenConfirm, "Confirm Installation"},
|
||||
{screenDone, "Installation complete"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.screen = tt.screen
|
||||
if got := m.View(); !strings.Contains(got, tt.want) {
|
||||
t.Errorf("screen %v does not render %q:\n%s", tt.screen, tt.want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The disk screen has to state what the selection actually builds, since the
|
||||
// cost of the redundancy choice is otherwise invisible until after the install.
|
||||
func TestDiskViewStatesTheResultingGeometry(t *testing.T) {
|
||||
m := wizardModel(t, 3, 1)
|
||||
m.screen = screenDisk
|
||||
|
||||
// ext4 with a role per drive: the mountpoints are the summary that matters.
|
||||
m = press(t, m, "down", "down", " ", "down", " ")
|
||||
if got := m.viewDisk(100); !strings.Contains(got, "/var/lib/spinifex") {
|
||||
t.Errorf("ext4 view does not state the role layout:\n%s", got)
|
||||
}
|
||||
// Matched spares are suggested, never applied.
|
||||
if got := m.viewDisk(100); !strings.Contains(got, "survive a disk failure") {
|
||||
t.Errorf("view does not suggest the pool the matched disks allow:\n%s", got)
|
||||
}
|
||||
|
||||
// A mirror with too few disks quotes the requirement, not a capacity.
|
||||
m = wizardModel(t, 3, 1)
|
||||
m.screen = screenDisk
|
||||
m.fsCursor = slices.Index(install.AllFSTypes, install.FSZFSRAID1)
|
||||
if got := m.viewDisk(100); !strings.Contains(got, "1 selected") {
|
||||
t.Errorf("view does not state what the mirror is missing:\n%s", got)
|
||||
}
|
||||
|
||||
// Once buildable it quotes usable capacity and the failures tolerated.
|
||||
m = press(t, m, "down", "down", " ")
|
||||
got := m.viewDisk(100)
|
||||
if !strings.Contains(got, "usable") || !strings.Contains(got, "survives 1 disk failure") {
|
||||
t.Errorf("view does not quote the pool geometry:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "A for ZFS options") {
|
||||
t.Errorf("the ZFS help line should replace the role key:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The overview names the card behind each port, which is what distinguishes the
|
||||
// 25gbe uplink from the onboard 1gbe.
|
||||
func TestRolesOverviewNamesTheHardware(t *testing.T) {
|
||||
m := wizardModel(t, 1, 2)
|
||||
m.nics[0].AltName = "enp1s0f0"
|
||||
m.advanced = true
|
||||
|
||||
got := m.viewNetworkRoles(100)
|
||||
for _, want := range []string{"eno1", "25Gbps", "Mellanox", "enp1s0f0", "VLAN", "MTU"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("overview is missing %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHumanBytes(t *testing.T) {
|
||||
tests := []struct {
|
||||
bytes int64
|
||||
want string
|
||||
}{
|
||||
{512, "512B"},
|
||||
{2 << 20, "2.0M"},
|
||||
{3 << 30, "3.0G"},
|
||||
{2 << 40, "2.0T"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := humanBytes(tt.bytes); got != tt.want {
|
||||
t.Errorf("humanBytes(%d) = %q, want %q", tt.bytes, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The overview writes an address the way an operator does, and falls back to
|
||||
// the raw mask rather than dropping it when it is not dotted-decimal.
|
||||
func TestFormatCIDR(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
addr, mask string
|
||||
want string
|
||||
}{
|
||||
{"dotted mask becomes a prefix", "10.0.0.3", "255.255.255.0", "10.0.0.3/24"},
|
||||
{"short mask keeps its prefix", "10.0.0.3", "255.255.0.0", "10.0.0.3/16"},
|
||||
{"no mask yet", "10.0.0.3", "", "10.0.0.3"},
|
||||
{"no address is blank", "", "255.255.255.0", ""},
|
||||
{"unparseable mask is shown as typed", "10.0.0.3", "24", "10.0.0.3/24"},
|
||||
{"whitespace is trimmed", " 10.0.0.3 ", " 255.255.255.0 ", "10.0.0.3/24"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := formatCIDR(tt.addr, tt.mask); got != tt.want {
|
||||
t.Errorf("formatCIDR(%q, %q) = %q, want %q", tt.addr, tt.mask, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Messages that are not keys reach the focused input, which is what keeps the
|
||||
// cursor blinking on the two single-field screens.
|
||||
func TestNonKeyMessagesReachTheActiveInput(t *testing.T) {
|
||||
m := wizardModel(t, 1, 1)
|
||||
m.screen = screenIdentity
|
||||
m.hostnameInput.Focus()
|
||||
|
||||
next, _ := m.Update(struct{ tea.Msg }{})
|
||||
if _, ok := next.(model); !ok {
|
||||
t.Fatalf("Update returned %T", next)
|
||||
}
|
||||
|
||||
m.screen = screenWelcome
|
||||
if _, cmd := m.Update(struct{ tea.Msg }{}); cmd != nil {
|
||||
t.Error("a screen with no input must issue no command")
|
||||
}
|
||||
}
|
||||
@@ -1,155 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/rdsgw"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
)
|
||||
|
||||
// The completion receipt rds-init writes once the master role is durably
|
||||
// applied. Deliberately outside PGDATA: clear_unfinished_datadir does an rm -rf
|
||||
// of PGDATA, so keeping the receipt out of it means no initdb-adjacent path can
|
||||
// touch it and the datadir stays a byte-for-byte stock one.
|
||||
const (
|
||||
receiptRelDir = ".spinifex-rds/bootstrap"
|
||||
receiptFile = "receipt.env"
|
||||
receiptPayload = "RDS_RECEIPT_PAYLOAD_ID"
|
||||
receiptDBID = "RDS_RECEIPT_DB_INSTANCE_IDENTIFIER"
|
||||
)
|
||||
|
||||
func (a *Agent) receiptPath() string {
|
||||
return filepath.Join(a.cfg.DataMount, receiptRelDir, receiptFile)
|
||||
}
|
||||
|
||||
// Confirms the initial bootstrap, which is what destroys the staged ciphertext.
|
||||
// A no-op on the attach path, and deliberately synchronous: it inherits the
|
||||
// cancellation semantics register, bootstrap and the data-mount wait already
|
||||
// share, and blocking the stateful loops behind it is correct because on a first
|
||||
// boot the engine is not bootstrapped until rds-init has finished anyway.
|
||||
//
|
||||
// It does not wait for the engine to be healthy. The receipt already proves the
|
||||
// master role was durably applied, and acknowledging early shortens the window
|
||||
// in which the ciphertext exists.
|
||||
func (a *Agent) acknowledgeBootstrap(ctx context.Context) error {
|
||||
pending := a.pending
|
||||
if pending == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Polled rather than waited on: rds-init runs after this agent registered,
|
||||
// so the file appears some time into the boot. There is no give-up deadline
|
||||
// here — the control plane's bootstrap timeout already fails the instance,
|
||||
// and a second one in the guest would only give the two a way to disagree.
|
||||
if err := retryObserved(ctx, "bootstrap receipt", func(context.Context) error {
|
||||
return a.checkBootstrapReceipt(pending)
|
||||
}, func(err error) {
|
||||
// An absent receipt is the expected state for the whole initdb window,
|
||||
// so reporting it would put a failure on every healthy create's beat and
|
||||
// then into the reason a timed-out create records.
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return
|
||||
}
|
||||
a.hb.setBootstrapFailure("bootstrap receipt", err)
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
a.hb.clearBootstrapFailure()
|
||||
|
||||
if err := retryObserved(ctx, "bootstrap acknowledgement", func(ctx context.Context) error {
|
||||
return terminalOnDenial(a.cp.AcknowledgeBootstrap(ctx, a.id, pending))
|
||||
}, func(err error) {
|
||||
a.hb.setBootstrapFailure("bootstrap acknowledgement", err)
|
||||
}); err != nil {
|
||||
if !errors.Is(err, errRetryTerminal) {
|
||||
return err
|
||||
}
|
||||
// The staged payload will never be delivered to this VM, and blocking the
|
||||
// command poller and the parameter guard on that would leave a serving
|
||||
// database no one can administer. The reason rides the heartbeat instead.
|
||||
slog.ErrorContext(ctx, "rds-agent: the initial bootstrap cannot be acknowledged",
|
||||
"payloadId", pending.payloadID, "err", err)
|
||||
return nil
|
||||
}
|
||||
a.hb.clearBootstrapFailure()
|
||||
|
||||
// AcknowledgedAt on the record is the durable audit trail, so the receipt is
|
||||
// removed rather than archived: keeping it would only grow the surface of
|
||||
// stale receipts riding along in data-volume snapshots. A removal failure is
|
||||
// harmless, since a later boot fetches attach and never consults one.
|
||||
if err := os.Remove(a.receiptPath()); err != nil && !os.IsNotExist(err) {
|
||||
slog.WarnContext(ctx, "rds-agent: could not remove the bootstrap receipt",
|
||||
"receipt", a.receiptPath(), "err", err)
|
||||
}
|
||||
slog.InfoContext(ctx, "rds-agent: initial bootstrap acknowledged", "payloadId", pending.payloadID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// The control plane resolved these from its own record — a superseded
|
||||
// generation, a payload it never staged, a data volume that is not this
|
||||
// instance's. Asking again cannot change any of them.
|
||||
func terminalOnDenial(err error) error {
|
||||
var apiErr *rdsgw.APIError
|
||||
if !errors.As(err, &apiErr) {
|
||||
return err
|
||||
}
|
||||
switch apiErr.Code {
|
||||
case awserrors.ErrorAccessDenied, awserrors.ErrorInvalidParameterValue:
|
||||
return fmt.Errorf("%w: %w", errRetryTerminal, err)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// A receipt naming another payload or another DB instance is treated as absent.
|
||||
// The receipt lives on the data volume, so it rides along in every snapshot of
|
||||
// it and a restored instance's volume carries the source instance's receipt.
|
||||
func (a *Agent) checkBootstrapReceipt(pending *pendingBootstrap) error {
|
||||
path := a.receiptPath()
|
||||
values, err := readReceipt(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if got := values[receiptPayload]; got != pending.payloadID {
|
||||
return fmt.Errorf("%s names bootstrap payload %q, not %q", path, got, pending.payloadID)
|
||||
}
|
||||
if got := values[receiptDBID]; got != a.id.DBInstanceIdentifier {
|
||||
return fmt.Errorf("%s names DB instance %q, not %q", path, got, a.id.DBInstanceIdentifier)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// KEY=value, the one format both readers can parse: rds-init in POSIX shell
|
||||
// with no guaranteed jq, and this.
|
||||
func readReceipt(path string) (map[string]string, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
values := make(map[string]string)
|
||||
scanner := bufio.NewScanner(f)
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
key, value, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
values[strings.TrimSpace(key)] = strings.TrimSpace(value)
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, fmt.Errorf("read %s: %w", path, err)
|
||||
}
|
||||
return values, nil
|
||||
}
|
||||
@@ -1,424 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/gwsign"
|
||||
"github.com/mulgadc/spinifex/internal/rdsgw"
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
)
|
||||
|
||||
// Overridable via -ldflags "-X main.version=...".
|
||||
var version = "dev"
|
||||
|
||||
// The gateway resolves the authoritative DB instance from the request's
|
||||
// credentials, so DBInstanceIdentifier is an assertion to be checked, not
|
||||
// trusted.
|
||||
type identity struct {
|
||||
DBInstanceIdentifier string
|
||||
AgentVersion string
|
||||
EngineVersion string
|
||||
}
|
||||
|
||||
// Every method is a SigV4-signed Query request, so the NATS bus stays
|
||||
// host-internal.
|
||||
type controlPlane interface {
|
||||
Register(ctx context.Context, id identity) (*handlers_rds.RegisterDBInstanceOutput, error)
|
||||
SubmitState(ctx context.Context, id identity, health handlers_rds.EngineHealth, message string) (*handlers_rds.SubmitDBStateChangeOutput, error)
|
||||
// Returns Mode=initialize with the master password for as long as a payload
|
||||
// staged for this VM generation is unacknowledged, and Mode=attach after.
|
||||
GetBootstrapConfig(ctx context.Context, id identity) (*handlers_rds.GetDBBootstrapConfigOutput, error)
|
||||
// Reports that PostgreSQL durably applied the staged password, which is what
|
||||
// lets the control plane destroy it.
|
||||
AcknowledgeBootstrap(ctx context.Context, id identity, pending *pendingBootstrap) error
|
||||
PollCommands(ctx context.Context, id identity, replies []handlers_rds.CommandReply, wait time.Duration) ([]handlers_rds.Command, error)
|
||||
}
|
||||
|
||||
// Bounds a register, heartbeat or bootstrap request. The long poll sets its own
|
||||
// deadline from the wait it asked the gateway to hold.
|
||||
const callTimeout = 15 * time.Second
|
||||
|
||||
// Keeps the poll's deadline past the requested wait, so a gateway answering at
|
||||
// the end of its window is not cut off by the client.
|
||||
const pollSlack = 10 * time.Second
|
||||
|
||||
type gatewayControlPlane struct {
|
||||
client *rdsgw.Client
|
||||
}
|
||||
|
||||
var _ controlPlane = (*gatewayControlPlane)(nil)
|
||||
|
||||
// Signs with the instance-role credentials the SDK chain resolves from IMDS,
|
||||
// against the pinned gateway CA.
|
||||
func newGatewayControlPlane(cfg config) (*gatewayControlPlane, error) {
|
||||
signer, err := gwsign.NewIMDS(context.Background(), cfg.Region)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build IMDS signer: %w", err)
|
||||
}
|
||||
// The client timeout is the ceiling for the longest call, the long poll;
|
||||
// shorter calls carry their own deadline.
|
||||
client, err := rdsgw.New(cfg.GatewayURL, cfg.GatewayCA, signer, cfg.Region, cfg.PollWait+pollSlack)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &gatewayControlPlane{client: client}, nil
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) Register(ctx context.Context, id identity) (*handlers_rds.RegisterDBInstanceOutput, error) {
|
||||
params := url.Values{}
|
||||
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
|
||||
setIfPresent(params, "AgentVersion", id.AgentVersion)
|
||||
setIfPresent(params, "EngineVersion", id.EngineVersion)
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, callTimeout)
|
||||
defer cancel()
|
||||
|
||||
var out handlers_rds.RegisterDBInstanceOutput
|
||||
if err := g.client.Call(ctx, "RegisterDBInstance", params, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) SubmitState(ctx context.Context, id identity, health handlers_rds.EngineHealth, message string) (*handlers_rds.SubmitDBStateChangeOutput, error) {
|
||||
params := url.Values{"EngineHealth": {string(health)}}
|
||||
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
|
||||
setIfPresent(params, "EngineVersion", id.EngineVersion)
|
||||
setIfPresent(params, "Message", message)
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, callTimeout)
|
||||
defer cancel()
|
||||
|
||||
var out handlers_rds.SubmitDBStateChangeOutput
|
||||
if err := g.client.Call(ctx, "SubmitDBStateChange", params, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) GetBootstrapConfig(ctx context.Context, id identity) (*handlers_rds.GetDBBootstrapConfigOutput, error) {
|
||||
params := url.Values{}
|
||||
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, callTimeout)
|
||||
defer cancel()
|
||||
|
||||
var out handlers_rds.GetDBBootstrapConfigOutput
|
||||
if err := g.client.Call(ctx, "GetDBBootstrapConfig", params, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) AcknowledgeBootstrap(ctx context.Context, id identity, pending *pendingBootstrap) error {
|
||||
params := url.Values{
|
||||
"PayloadId": {pending.payloadID},
|
||||
"VMGeneration": {strconv.FormatInt(pending.vmGeneration, 10)},
|
||||
}
|
||||
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
|
||||
setIfPresent(params, "DataVolumeId", pending.dataVolumeID)
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, callTimeout)
|
||||
defer cancel()
|
||||
|
||||
var out handlers_rds.AcknowledgeDBBootstrapOutput
|
||||
if err := g.client.Call(ctx, "AcknowledgeDBBootstrap", params, &out); err != nil {
|
||||
return err
|
||||
}
|
||||
if !out.Acknowledged {
|
||||
return fmt.Errorf("the control plane did not acknowledge bootstrap payload %s", pending.payloadID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Declared here so the in-guest binary does not link the server side of the API.
|
||||
type pollDBCommandsOutput struct {
|
||||
Commands []handlers_rds.Command `xml:"Commands>member"`
|
||||
}
|
||||
|
||||
func (g *gatewayControlPlane) PollCommands(ctx context.Context, id identity, replies []handlers_rds.CommandReply, wait time.Duration) ([]handlers_rds.Command, error) {
|
||||
params := url.Values{"WaitTimeSeconds": {strconv.FormatInt(int64(wait.Seconds()), 10)}}
|
||||
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
|
||||
// Replies ride the poll rather than a separate action, so replying and
|
||||
// asking for the next command are one round trip.
|
||||
for i, reply := range replies {
|
||||
member := fmt.Sprintf("Replies.member.%d.", i+1)
|
||||
params.Set(member+"CommandId", reply.CommandID)
|
||||
params.Set(member+"Status", reply.Status)
|
||||
setIfPresent(params, member+"Message", reply.Message)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, wait+pollSlack)
|
||||
defer cancel()
|
||||
|
||||
var out pollDBCommandsOutput
|
||||
if err := g.client.Call(ctx, "PollDBCommands", params, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out.Commands, nil
|
||||
}
|
||||
|
||||
// Only sets a parameter that has a value, so the gateway can tell "no
|
||||
// assertion" from "asserted as blank".
|
||||
func setIfPresent(params url.Values, key, value string) {
|
||||
if value != "" {
|
||||
params.Set(key, value)
|
||||
}
|
||||
}
|
||||
|
||||
type Agent struct {
|
||||
cfg config
|
||||
id identity
|
||||
cp controlPlane
|
||||
probe *engineProbe
|
||||
engine engine
|
||||
handoffWriter func(string, *handlers_rds.GetDBBootstrapConfigOutput) error
|
||||
dataMountWaiter func(context.Context, string, string) error
|
||||
|
||||
// Set by bootstrap when the fetch replayed a staged payload; nil leaves the
|
||||
// acknowledgement step a no-op, which is every attach boot.
|
||||
pending *pendingBootstrap
|
||||
|
||||
hb *heartbeater
|
||||
cmd *commander
|
||||
guard *paramGuard
|
||||
}
|
||||
|
||||
// Assembles from already-built parts, so tests can pass fakes; New builds the
|
||||
// production control plane and delegates here.
|
||||
func newAgent(cfg config, cp controlPlane, probe *engineProbe) (*Agent, error) {
|
||||
if cfg.MountsFile == "" {
|
||||
cfg.MountsFile = defaultMountsFile
|
||||
}
|
||||
id := identity{
|
||||
DBInstanceIdentifier: cfg.DBInstanceIdentifier,
|
||||
AgentVersion: version,
|
||||
EngineVersion: cfg.EngineVersion,
|
||||
}
|
||||
a := &Agent{
|
||||
cfg: cfg, id: id, cp: cp, probe: probe,
|
||||
handoffWriter: writeHandoff, dataMountWaiter: waitForDataMount,
|
||||
}
|
||||
eng, err := newEngine(cfg, execCommandRunner, execSessionRunner, probe)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a.engine = eng
|
||||
a.hb = newHeartbeater(cp, probe, a.engine, handlers_rds.HeartbeatInterval)
|
||||
a.cmd = newCommander(cp, newCommandRegistry(a.engine, newGuestStorage(cfg, execCommandRunner)), cfg.PollWait)
|
||||
a.guard = newParamGuard(a.engine, probe, cp)
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// New builds the agent from cfg, failing if no gateway URL is set. It does not
|
||||
// wait for IMDS: the register loop rides out a datapath still coming up.
|
||||
func New(cfg config) (*Agent, error) {
|
||||
if cfg.GatewayURL == "" {
|
||||
return nil, fmt.Errorf("no gateway URL configured (RDS_GATEWAY_URL)")
|
||||
}
|
||||
cp, err := newGatewayControlPlane(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
probe, err := newProbe(cfg, execProbeRunner)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return newAgent(cfg, cp, probe)
|
||||
}
|
||||
|
||||
func (a *Agent) Run(ctx context.Context) error {
|
||||
// Named before the first dial: when register cannot reach the gateway, the
|
||||
// address it is dialing is what separates a mgmt NIC that came up without one
|
||||
// from a control plane that is genuinely down.
|
||||
slog.Info("rds-agent: starting",
|
||||
"gatewayURL", a.cfg.GatewayURL, "dbInstanceIdentifier", a.id.DBInstanceIdentifier,
|
||||
"agentVersion", a.id.AgentVersion)
|
||||
|
||||
// Registration comes first: its response is the agent's identity.
|
||||
if err := a.register(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
a.hb.id, a.cmd.id, a.guard.id = a.id, a.id, a.id
|
||||
|
||||
// Ahead of the handoff, so rds-init never unblocks and no datadir is touched.
|
||||
// An agent running on the wrong image would initialise one engine over
|
||||
// another's data, coming up healthy and empty beside data nothing references.
|
||||
if err := a.refuseEngineMismatch(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Beating before the bootstrap keeps a stuck boot visible as a live VM with
|
||||
// a down engine rather than as silence.
|
||||
go a.hb.Run(ctx)
|
||||
|
||||
if err := a.bootstrap(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// The handoff must land before this wait: rds-datadir needs it to identify
|
||||
// and authorize the volume. Commands and the parameter guard wait because
|
||||
// their durable state belongs on that mount, never on the disposable boot disk.
|
||||
if err := a.dataMountWaiter(ctx, a.cfg.MountsFile, a.cfg.DataMount); err != nil {
|
||||
// A shutdown cancels this wait; main treats the wrapped context.Canceled
|
||||
// as a clean stop, exactly as it does for register and bootstrap above.
|
||||
return fmt.Errorf("wait for data mount %s: %w", a.cfg.DataMount, err)
|
||||
}
|
||||
|
||||
// The receipt it waits on lives on that mount, so this cannot run any earlier.
|
||||
// A cancelled wait is a clean stop here too.
|
||||
if err := a.acknowledgeBootstrap(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Directives are only meaningful against a bootstrapped, mounted engine.
|
||||
go a.cmd.Run(ctx)
|
||||
|
||||
// Started after the handoff, so the window it measures covers the engine's
|
||||
// own start rather than the time rds-init spent waiting for this agent.
|
||||
go a.guard.Run(ctx)
|
||||
|
||||
<-ctx.Done()
|
||||
return nil
|
||||
}
|
||||
|
||||
// The engine cloud-init says this VM was launched as, against the engine the
|
||||
// image bakes. An empty expectation is no assertion rather than a disagreement:
|
||||
// a VM launched before the control plane emitted RDS_ENGINE carries none.
|
||||
func (a *Agent) refuseEngineMismatch(ctx context.Context) error {
|
||||
expected := strings.ToLower(strings.TrimSpace(a.cfg.Engine))
|
||||
if expected == "" || expected == a.cfg.BakedEngine {
|
||||
return nil
|
||||
}
|
||||
|
||||
message := fmt.Sprintf("this VM was launched as engine %q but its image bakes %q; refusing to bootstrap",
|
||||
expected, a.cfg.BakedEngine)
|
||||
slog.ErrorContext(ctx, "rds-agent: "+message)
|
||||
// Reported before returning, so the control plane learns why the instance
|
||||
// never came up rather than only that it never did.
|
||||
if _, err := a.cp.SubmitState(ctx, a.id, handlers_rds.EngineHealthUnhealthy, message); err != nil {
|
||||
slog.ErrorContext(ctx, "rds-agent: reporting the engine mismatch failed", "err", err)
|
||||
}
|
||||
return errors.New(message)
|
||||
}
|
||||
|
||||
const dataMountPollInterval = time.Second
|
||||
|
||||
// waitForDataMount blocks the stateful agent loops until the kernel reports the
|
||||
// configured data mount. Registration, heartbeat, and bootstrap run before it.
|
||||
func waitForDataMount(ctx context.Context, mountsFile, target string) error {
|
||||
ticker := time.NewTicker(dataMountPollInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
mounted, err := mountTableContains(mountsFile, target)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "rds-agent: could not read mount table while waiting for data volume",
|
||||
"mountsFile", mountsFile, "dataMount", target, "err", err)
|
||||
} else if mounted {
|
||||
slog.InfoContext(ctx, "rds-agent: data volume mount is ready", "dataMount", target)
|
||||
return nil
|
||||
}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mountTableContains(path, target string) (bool, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
scanner := bufio.NewScanner(f)
|
||||
for scanner.Scan() {
|
||||
fields := strings.Fields(scanner.Text())
|
||||
if len(fields) >= 2 && unescapeMountField(fields[1]) == target {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// Linux mount tables encode whitespace and backslashes as octal escapes.
|
||||
func unescapeMountField(field string) string {
|
||||
replacer := strings.NewReplacer(`\134`, `\`, `\040`, " ", `\011`, "\t", `\012`, "\n")
|
||||
return replacer.Replace(field)
|
||||
}
|
||||
|
||||
// Boot-retry bounds. Tight to start, since the usual cause is a control plane
|
||||
// still creating this instance's record, and capped so an outage does not turn
|
||||
// a booting fleet into a retry storm. Vars so tests can shorten the wait.
|
||||
var (
|
||||
retryMin = 1 * time.Second
|
||||
retryMax = 30 * time.Second
|
||||
)
|
||||
|
||||
// With the production backoff, attempt five is about 15 seconds into the
|
||||
// failure and still precedes the first 30-second heartbeat.
|
||||
const retryErrorAttempt = 5
|
||||
|
||||
// Wrapping an error in this stops retryObserved. For the failures the control
|
||||
// plane decides from its own record, no retry can change the answer, and looping
|
||||
// on one only keeps the caller blocked.
|
||||
var errRetryTerminal = errors.New("terminal failure")
|
||||
|
||||
// Never gives up on its own: an agent that stopped trying would leave the
|
||||
// control plane with no signal at all.
|
||||
func retry(ctx context.Context, what string, fn func(context.Context) error) error {
|
||||
return retryObserved(ctx, what, fn, nil)
|
||||
}
|
||||
|
||||
// onFailure lets boot-critical callers publish the latest error while the retry
|
||||
// remains blocked. It must not block because it runs on the retry path itself.
|
||||
func retryObserved(ctx context.Context, what string, fn func(context.Context) error, onFailure func(error)) error {
|
||||
delay := retryMin
|
||||
for attempt := 1; ; attempt++ {
|
||||
err := fn(ctx)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return fmt.Errorf("%s: %w", what, ctx.Err())
|
||||
}
|
||||
if onFailure != nil {
|
||||
onFailure(err)
|
||||
}
|
||||
if errors.Is(err, errRetryTerminal) {
|
||||
slog.ErrorContext(ctx, "rds-agent: "+what+" failed terminally, not retrying", "err", err)
|
||||
return fmt.Errorf("%s: %w", what, err)
|
||||
}
|
||||
if attempt >= retryErrorAttempt {
|
||||
slog.ErrorContext(ctx, "rds-agent: "+what+" still failing, retrying",
|
||||
"attempt", attempt, "retry_in_ms", otelsetup.Millis(delay), "err", err)
|
||||
} else {
|
||||
slog.WarnContext(ctx, "rds-agent: "+what+" failed, retrying",
|
||||
"attempt", attempt, "retry_in_ms", otelsetup.Millis(delay), "err", err)
|
||||
}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return fmt.Errorf("%s: %w", what, ctx.Err())
|
||||
case <-time.After(delay):
|
||||
}
|
||||
delay = min(delay*2, retryMax)
|
||||
}
|
||||
}
|
||||
@@ -1,880 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/gwsign"
|
||||
"github.com/mulgadc/spinifex/internal/rdsgw"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
gateway_rds "github.com/mulgadc/spinifex/spinifex/gateway/rds"
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// fakeControlPlane is a scriptable controlPlane. Every call records what the
|
||||
// agent sent, so a test asserts on the request as well as the reaction.
|
||||
type fakeControlPlane struct {
|
||||
mu sync.Mutex
|
||||
|
||||
registerOut *handlers_rds.RegisterDBInstanceOutput
|
||||
registerErrs []error
|
||||
registerReqs []identity
|
||||
|
||||
bootstrapOut *handlers_rds.GetDBBootstrapConfigOutput
|
||||
bootstrapErrs []error
|
||||
// Returned once bootstrapErrs is drained, for a failure no retry heals.
|
||||
bootstrapErr error
|
||||
bootstrapReqs []identity
|
||||
// Scripted responses ahead of bootstrapOut, so a test can answer the first
|
||||
// fetch differently from the ones the agent retries with.
|
||||
bootstrapQueue []*handlers_rds.GetDBBootstrapConfigOutput
|
||||
|
||||
ackErrs []error
|
||||
ackReqs []pendingBootstrap
|
||||
|
||||
states []submittedState
|
||||
|
||||
pollReplies [][]handlers_rds.CommandReply
|
||||
pollQueue [][]handlers_rds.Command
|
||||
pollErr error
|
||||
polled chan struct{}
|
||||
}
|
||||
|
||||
type submittedState struct {
|
||||
id identity
|
||||
health handlers_rds.EngineHealth
|
||||
message string
|
||||
}
|
||||
|
||||
var _ controlPlane = (*fakeControlPlane)(nil)
|
||||
|
||||
func newFakeControlPlane() *fakeControlPlane {
|
||||
return &fakeControlPlane{
|
||||
registerOut: &handlers_rds.RegisterDBInstanceOutput{
|
||||
DBInstanceIdentifier: "db-resolved", HeartbeatIntervalSeconds: 30,
|
||||
},
|
||||
polled: make(chan struct{}, 64),
|
||||
}
|
||||
}
|
||||
|
||||
// nextErr pops the next scripted error, so a test can fail a call n times and
|
||||
// then let it succeed.
|
||||
func nextErr(errs *[]error) error {
|
||||
if len(*errs) == 0 {
|
||||
return nil
|
||||
}
|
||||
err := (*errs)[0]
|
||||
*errs = (*errs)[1:]
|
||||
return err
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) Register(_ context.Context, id identity) (*handlers_rds.RegisterDBInstanceOutput, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.registerReqs = append(f.registerReqs, id)
|
||||
if err := nextErr(&f.registerErrs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return f.registerOut, nil
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) SubmitState(_ context.Context, id identity, health handlers_rds.EngineHealth, message string) (*handlers_rds.SubmitDBStateChangeOutput, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.states = append(f.states, submittedState{id: id, health: health, message: message})
|
||||
// Zero means "keep the cadence you have", so a test's fast interval is not
|
||||
// reset by every beat.
|
||||
return &handlers_rds.SubmitDBStateChangeOutput{Acknowledged: true}, nil
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) GetBootstrapConfig(_ context.Context, id identity) (*handlers_rds.GetDBBootstrapConfigOutput, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.bootstrapReqs = append(f.bootstrapReqs, id)
|
||||
if err := nextErr(&f.bootstrapErrs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if f.bootstrapErr != nil {
|
||||
return nil, f.bootstrapErr
|
||||
}
|
||||
if len(f.bootstrapQueue) > 0 {
|
||||
out := f.bootstrapQueue[0]
|
||||
f.bootstrapQueue = f.bootstrapQueue[1:]
|
||||
return out, nil
|
||||
}
|
||||
return f.bootstrapOut, nil
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) AcknowledgeBootstrap(_ context.Context, _ identity, pending *pendingBootstrap) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.ackReqs = append(f.ackReqs, *pending)
|
||||
return nextErr(&f.ackErrs)
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) PollCommands(ctx context.Context, _ identity, replies []handlers_rds.CommandReply, _ time.Duration) ([]handlers_rds.Command, error) {
|
||||
f.mu.Lock()
|
||||
f.pollReplies = append(f.pollReplies, slices.Clone(replies))
|
||||
err := f.pollErr
|
||||
var out []handlers_rds.Command
|
||||
if len(f.pollQueue) > 0 {
|
||||
out, f.pollQueue = f.pollQueue[0], f.pollQueue[1:]
|
||||
}
|
||||
f.mu.Unlock()
|
||||
|
||||
select {
|
||||
case f.polled <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(out) > 0 {
|
||||
return out, nil
|
||||
}
|
||||
// An idle window: block until the agent hangs up, as the gateway's long poll
|
||||
// does, rather than spinning the caller's loop.
|
||||
<-ctx.Done()
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) ackRequests() []pendingBootstrap {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return slices.Clone(f.ackReqs)
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) snapshotStates() []submittedState {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]submittedState(nil), f.states...)
|
||||
}
|
||||
|
||||
func (f *fakeControlPlane) snapshotReplies() [][]handlers_rds.CommandReply {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
out := make([][]handlers_rds.CommandReply, len(f.pollReplies))
|
||||
for i := range f.pollReplies {
|
||||
out[i] = slices.Clone(f.pollReplies[i])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// testConfig stamps the engine a real rds-postgres image bakes, so the agent
|
||||
// resolves the same guest layout it would in a guest, and points its file output
|
||||
// at a temp dir.
|
||||
func testConfig(t *testing.T) config {
|
||||
t.Helper()
|
||||
cfg := testLoadConfig(t, enginePostgres)
|
||||
cfg.GatewayURL = "https://gw.test"
|
||||
cfg.HandoffDir = filepath.Join(t.TempDir(), "spinifex-rds")
|
||||
cfg.PollWait = time.Second
|
||||
return cfg
|
||||
}
|
||||
|
||||
// newTestAgent builds the agent over a probe the test drives directly.
|
||||
func newTestAgent(t *testing.T, cfg config, cp controlPlane, run probeRunner) *Agent {
|
||||
t.Helper()
|
||||
a, err := newAgent(cfg, cp, newPostgresProbe(cfg, run))
|
||||
if err != nil {
|
||||
t.Fatalf("newAgent: %v", err)
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// staticProbe is a probeRunner that always reports the same exit code.
|
||||
func staticProbe(code int) probeRunner {
|
||||
return func(context.Context, string, ...string) (int, string, error) { return code, "", nil }
|
||||
}
|
||||
|
||||
func bootstrapOutput(password *string) *handlers_rds.GetDBBootstrapConfigOutput {
|
||||
mode := handlers_rds.BootstrapModeAttach
|
||||
if password != nil {
|
||||
mode = handlers_rds.BootstrapModeInitialize
|
||||
}
|
||||
return &handlers_rds.GetDBBootstrapConfigOutput{
|
||||
Mode: mode,
|
||||
DBInstanceIdentifier: "db-resolved",
|
||||
Engine: "postgres",
|
||||
MasterUsername: "master",
|
||||
MasterUserPassword: password,
|
||||
Port: 5432,
|
||||
ListenAddress: testListenAddress,
|
||||
ClientCIDR: testClientCIDR,
|
||||
DataVolumeID: "vol-data-01",
|
||||
DataVolumeSerial: "voldata01",
|
||||
VMGeneration: 1,
|
||||
PayloadID: testPayloadID,
|
||||
BootstrapPending: password != nil,
|
||||
}
|
||||
}
|
||||
|
||||
const testPayloadID = "bp-0123456789abcdef"
|
||||
|
||||
// The customer ENI address the engine binds, and the VPC range its client
|
||||
// authentication admits. writeHandoff refuses a config without either.
|
||||
const (
|
||||
testListenAddress = "10.0.1.20"
|
||||
testClientCIDR = "10.0.0.0/16"
|
||||
)
|
||||
|
||||
// Writes the completion receipt rds-init leaves behind, so the acknowledgement
|
||||
// step finds what a finished initialization would have produced.
|
||||
func writeReceipt(t *testing.T, a *Agent, payloadID, dbInstanceIdentifier string) {
|
||||
t.Helper()
|
||||
path := a.receiptPath()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
t.Fatalf("mkdir receipt dir: %v", err)
|
||||
}
|
||||
body := fmt.Sprintf("%s=%s\n%s=%s\n", receiptPayload, payloadID, receiptDBID, dbInstanceIdentifier)
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatalf("write receipt: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// runAgent runs the agent until its bootstrap has landed, then cancels. It
|
||||
// returns once Run has exited so no goroutine outlives the test.
|
||||
func runAgent(t *testing.T, a *Agent) error {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- a.Run(ctx) }()
|
||||
|
||||
waitFor(t, func() bool {
|
||||
_, err := os.Stat(filepath.Join(a.cfg.HandoffDir, handoffEnvFile))
|
||||
return err == nil
|
||||
}, "handoff file to be written")
|
||||
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
// The cancellation above is this helper's own, so Run reporting it is the
|
||||
// expected clean stop rather than a failure, exactly as main treats it.
|
||||
if errors.Is(err, context.Canceled) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("Run did not exit after cancellation")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// waitFor polls cond until it holds or the test times out.
|
||||
func waitFor(t *testing.T, cond func() bool, what string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if cond() {
|
||||
return
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("timed out waiting for %s", what)
|
||||
}
|
||||
|
||||
func TestRun_RegistersThenDeliversBootstrap(t *testing.T) {
|
||||
password := "s3cret"
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(&password)
|
||||
|
||||
cfg := testConfig(t)
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
if err := runAgent(t, a); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
if len(cp.registerReqs) != 1 {
|
||||
t.Fatalf("registered %d times, want 1", len(cp.registerReqs))
|
||||
}
|
||||
// The agent had no configured identifier, so the gateway's resolution is
|
||||
// what every later call must carry.
|
||||
if got := cp.bootstrapReqs[0].DBInstanceIdentifier; got != "db-resolved" {
|
||||
t.Errorf("bootstrap sent identifier %q, want the registered db-resolved", got)
|
||||
}
|
||||
|
||||
env := readFile(t, filepath.Join(cfg.HandoffDir, handoffEnvFile))
|
||||
if !strings.Contains(env, "RDS_MODE='initialize'") || !strings.Contains(env, "RDS_MASTER_PASSWORD='s3cret'") {
|
||||
t.Errorf("handoff env = %q, want initialize mode carrying the password", env)
|
||||
}
|
||||
// The probe follows the port the control plane assigned, not the default.
|
||||
if got := a.probe.port.Load(); got != 5432 {
|
||||
t.Errorf("probe port = %d, want 5432", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A reboot re-fetches in attach mode. The agent must write that payload as it
|
||||
// stands rather than refusing it or inventing a password.
|
||||
func TestRun_AttachModeNeedsNoPassword(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
|
||||
cfg := testConfig(t)
|
||||
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
env := readFile(t, filepath.Join(cfg.HandoffDir, handoffEnvFile))
|
||||
if !strings.Contains(env, "RDS_MODE='attach'") {
|
||||
t.Errorf("handoff env = %q, want attach mode", env)
|
||||
}
|
||||
if strings.Contains(env, "RDS_MASTER_PASSWORD") {
|
||||
t.Errorf("handoff env = %q, want no password key at all in attach mode", env)
|
||||
}
|
||||
}
|
||||
|
||||
// The record may not exist yet when a freshly launched VM first calls, so both
|
||||
// boot-critical calls have to ride out failures instead of exiting.
|
||||
func TestRun_RetriesRegisterAndBootstrap(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
cp.registerErrs = []error{errors.New("DBInstanceNotFound")}
|
||||
cp.bootstrapErrs = []error{errors.New("ServerInternal")}
|
||||
|
||||
cfg := testConfig(t)
|
||||
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
if len(cp.registerReqs) != 2 {
|
||||
t.Errorf("register attempts = %d, want 2 (one failure, one success)", len(cp.registerReqs))
|
||||
}
|
||||
if len(cp.bootstrapReqs) != 2 {
|
||||
t.Errorf("bootstrap attempts = %d, want 2 (one failure, one success)", len(cp.bootstrapReqs))
|
||||
}
|
||||
}
|
||||
|
||||
// The fetch mutates nothing, so a handoff that cannot be written is retried
|
||||
// against the payload already in hand rather than costing a second fetch.
|
||||
func TestBootstrap_RetriesHandoffWithoutRefetching(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
password := "staged-secret"
|
||||
cp.bootstrapOut = bootstrapOutput(&password)
|
||||
cfg := testConfig(t)
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
|
||||
writes := 0
|
||||
a.handoffWriter = func(_ string, got *handlers_rds.GetDBBootstrapConfigOutput) error {
|
||||
writes++
|
||||
if got.MasterUserPassword == nil || *got.MasterUserPassword != password {
|
||||
t.Fatalf("handoff attempt %d lost the initialize password", writes)
|
||||
}
|
||||
if writes == 1 {
|
||||
return errors.New("temporary tmpfs write error")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := a.bootstrap(t.Context()); err != nil {
|
||||
t.Fatalf("bootstrap: %v", err)
|
||||
}
|
||||
if len(cp.bootstrapReqs) != 1 {
|
||||
t.Errorf("bootstrap fetches = %d, want 1", len(cp.bootstrapReqs))
|
||||
}
|
||||
if writes != 2 {
|
||||
t.Errorf("handoff writes = %d, want 2", writes)
|
||||
}
|
||||
if a.pending == nil || a.pending.payloadID != testPayloadID {
|
||||
t.Errorf("pending bootstrap = %+v, want the staged payload %s", a.pending, testPayloadID)
|
||||
}
|
||||
}
|
||||
|
||||
// A daemon that predates the encrypted payload answers initialize with nothing
|
||||
// to initialise with. Writing that handoff would let rds-init run initdb with an
|
||||
// empty master password, so it is retried until an upgraded node answers.
|
||||
func TestBootstrap_RetriesInitializeWithNoPassword(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
empty := ""
|
||||
password := "staged-secret"
|
||||
cp.bootstrapQueue = []*handlers_rds.GetDBBootstrapConfigOutput{{
|
||||
Mode: handlers_rds.BootstrapModeInitialize, DBInstanceIdentifier: "db-resolved",
|
||||
MasterUsername: "master", MasterUserPassword: &empty,
|
||||
}}
|
||||
cp.bootstrapOut = bootstrapOutput(&password)
|
||||
cfg := testConfig(t)
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
|
||||
var handed []*handlers_rds.GetDBBootstrapConfigOutput
|
||||
a.handoffWriter = func(_ string, got *handlers_rds.GetDBBootstrapConfigOutput) error {
|
||||
handed = append(handed, got)
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := a.bootstrap(t.Context()); err != nil {
|
||||
t.Fatalf("bootstrap: %v", err)
|
||||
}
|
||||
if len(handed) != 1 {
|
||||
t.Fatalf("handoff writes = %d, want 1 once a usable payload arrived", len(handed))
|
||||
}
|
||||
if handed[0].MasterUserPassword == nil || *handed[0].MasterUserPassword != password {
|
||||
t.Error("the agent wrote a handoff for a fetch that carried no master password")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAcknowledgeBootstrap_WaitsForTheReceiptThenRetiresIt(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cfg := testConfig(t)
|
||||
cfg.DataMount = t.TempDir()
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
a.id = identity{DBInstanceIdentifier: "db-resolved"}
|
||||
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1, dataVolumeID: "vol-data-01"}
|
||||
|
||||
// rds-init runs after the agent registers, so the receipt is absent when the
|
||||
// acknowledgement step first looks and appears some way into the boot.
|
||||
if err := a.checkBootstrapReceipt(a.pending); err == nil {
|
||||
t.Fatal("a missing receipt must not read as a completed bootstrap")
|
||||
}
|
||||
writeReceipt(t, a, testPayloadID, "db-resolved")
|
||||
|
||||
if err := a.acknowledgeBootstrap(t.Context()); err != nil {
|
||||
t.Fatalf("acknowledgeBootstrap: %v", err)
|
||||
}
|
||||
if len(cp.ackRequests()) != 1 {
|
||||
t.Fatalf("acknowledgements = %d, want 1", len(cp.ackRequests()))
|
||||
}
|
||||
if got := cp.ackRequests()[0]; got != *a.pending {
|
||||
t.Errorf("acknowledged %+v, want %+v", got, *a.pending)
|
||||
}
|
||||
if _, err := os.Stat(a.receiptPath()); !os.IsNotExist(err) {
|
||||
t.Error("the receipt must be removed once the record holds the audit trail")
|
||||
}
|
||||
}
|
||||
|
||||
// The receipt rides along in every snapshot of the data volume, so a restored
|
||||
// instance's volume carries the source instance's receipt. Acknowledging on one
|
||||
// would destroy the ciphertext this instance still needs.
|
||||
func TestAcknowledgeBootstrap_IgnoresAForeignReceipt(t *testing.T) {
|
||||
tests := map[string]struct{ payloadID, dbID string }{
|
||||
"another payload": {payloadID: "bp-somebody-else", dbID: "db-resolved"},
|
||||
"another instance": {payloadID: testPayloadID, dbID: "db-the-snapshot-source"},
|
||||
}
|
||||
|
||||
for name, tc := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cfg := testConfig(t)
|
||||
cfg.DataMount = t.TempDir()
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
a.id = identity{DBInstanceIdentifier: "db-resolved"}
|
||||
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
|
||||
writeReceipt(t, a, tc.payloadID, tc.dbID)
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := a.acknowledgeBootstrap(ctx); err == nil {
|
||||
t.Fatal("acknowledgeBootstrap returned nil, want it still waiting on a real receipt")
|
||||
}
|
||||
if n := len(cp.ackRequests()); n != 0 {
|
||||
t.Errorf("acknowledgements = %d, want 0", n)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// An attach boot has nothing staged, so it must not go looking for a receipt the
|
||||
// datadir will never grow.
|
||||
func TestAcknowledgeBootstrap_IsANoOpOnAttach(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cfg := testConfig(t)
|
||||
cfg.DataMount = t.TempDir()
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
|
||||
if err := a.acknowledgeBootstrap(t.Context()); err != nil {
|
||||
t.Fatalf("acknowledgeBootstrap: %v", err)
|
||||
}
|
||||
if n := len(cp.ackRequests()); n != 0 {
|
||||
t.Errorf("acknowledgements = %d, want 0 when nothing was staged", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The image is the authority on which engine the agent runs. A VM launched as
|
||||
// another engine has to stop ahead of the handoff: rds-init would otherwise
|
||||
// initialise one engine over the other's datadir and come up healthy and empty
|
||||
// beside data nothing references.
|
||||
func TestRun_RefusesAnEngineTheImageDoesNotBake(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
|
||||
cfg := testConfig(t)
|
||||
cfg.Engine = "mariadb"
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
|
||||
err := a.Run(t.Context())
|
||||
if err == nil {
|
||||
t.Fatal("Run returned nil for a VM launched as an engine this image does not bake")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "mariadb") {
|
||||
t.Errorf("error = %v, want it to name the engine the VM was launched as", err)
|
||||
}
|
||||
if _, statErr := os.Stat(filepath.Join(cfg.HandoffDir, handoffEnvFile)); !os.IsNotExist(statErr) {
|
||||
t.Error("the agent wrote a bootstrap handoff for an engine it refused to run")
|
||||
}
|
||||
if n := len(cp.bootstrapReqs); n != 0 {
|
||||
t.Errorf("bootstrap fetches = %d, want none before the refusal", n)
|
||||
}
|
||||
// Reported rather than only logged, so the control plane learns why the
|
||||
// instance never came up rather than only that it never did.
|
||||
states := cp.snapshotStates()
|
||||
if len(states) != 1 || states[0].health != handlers_rds.EngineHealthUnhealthy {
|
||||
t.Fatalf("submitted states = %+v, want a single unhealthy report", states)
|
||||
}
|
||||
}
|
||||
|
||||
// The same assertion when it agrees is not a gate: the boot goes on exactly as
|
||||
// it does for a VM launched before the control plane sent one at all.
|
||||
func TestRun_ProceedsWhenTheDeliveredEngineMatchesTheImage(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
|
||||
cfg := testConfig(t)
|
||||
cfg.Engine = enginePostgres
|
||||
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A configured identifier is asserted on the wire so a mis-provisioned VM is
|
||||
// rejected by the gateway instead of adopting whatever it resolves to.
|
||||
func TestRun_SendsConfiguredIdentifier(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
|
||||
cfg := testConfig(t)
|
||||
cfg.DBInstanceIdentifier = "db-configured"
|
||||
cfg.EngineVersion = "18.1"
|
||||
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
sent := cp.registerReqs[0]
|
||||
if sent.DBInstanceIdentifier != "db-configured" || sent.EngineVersion != "18.1" {
|
||||
t.Errorf("register sent %+v, want the configured identifier and engine version", sent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_WaitsForDataMountBeforePollingCommands(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
cfg := testConfig(t)
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(2))
|
||||
|
||||
waiting := make(chan struct{})
|
||||
release := make(chan struct{})
|
||||
a.dataMountWaiter = func(ctx context.Context, mountsFile, target string) error {
|
||||
if mountsFile != defaultMountsFile || target != cfg.DataMount {
|
||||
t.Errorf("mount waiter got %q/%q, want configured defaults", mountsFile, target)
|
||||
}
|
||||
close(waiting)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-release:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- a.Run(ctx) }()
|
||||
|
||||
select {
|
||||
case <-waiting:
|
||||
case <-ctx.Done():
|
||||
t.Fatal("agent did not reach the data-mount wait after writing the handoff")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(cfg.HandoffDir, handoffEnvFile)); err != nil {
|
||||
t.Fatalf("handoff was not present before mount wait: %v", err)
|
||||
}
|
||||
select {
|
||||
case <-cp.polled:
|
||||
t.Fatal("command poll started before the data volume was mounted")
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
}
|
||||
|
||||
close(release)
|
||||
select {
|
||||
case <-cp.polled:
|
||||
case <-ctx.Done():
|
||||
t.Fatal("command poll did not start after the data mount became ready")
|
||||
}
|
||||
cancel()
|
||||
if err := <-done; err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMountTableContains(t *testing.T) {
|
||||
mounts := filepath.Join(t.TempDir(), "mounts")
|
||||
require.NoError(t, os.WriteFile(mounts, []byte(
|
||||
"/dev/vda1 / ext4 rw 0 0\n/dev/vdb /var/lib/postgresql ext4 rw 0 0\n"+
|
||||
"/dev/vdc /path\\040with\\040spaces xfs rw 0 0\nmalformed\n"), 0o600))
|
||||
|
||||
tests := []struct {
|
||||
target string
|
||||
want bool
|
||||
}{
|
||||
{target: "/var/lib/postgresql", want: true},
|
||||
{target: "/path with spaces", want: true},
|
||||
{target: "/missing", want: false},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.target, func(t *testing.T) {
|
||||
got, err := mountTableContains(mounts, tc.target)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tc.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_HeartbeatIncludesBootstrapFailure(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
// Persistent, so every retry republishes the 403 rather than leaving it
|
||||
// visible to a heartbeat for a single retry interval.
|
||||
cp.bootstrapErr = errors.New("403 AccessDenied: instance profile is not authorised")
|
||||
cp.registerOut.HeartbeatIntervalSeconds = 0
|
||||
|
||||
cfg := testConfig(t)
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(2))
|
||||
a.hb.interval = 10 * time.Millisecond
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- a.Run(ctx) }()
|
||||
|
||||
waitFor(t, func() bool {
|
||||
for _, state := range cp.snapshotStates() {
|
||||
if strings.Contains(state.message, "bootstrap fetch failed") &&
|
||||
strings.Contains(state.message, "403 AccessDenied") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}, "the bootstrap failure to reach a heartbeat")
|
||||
cancel()
|
||||
<-done
|
||||
}
|
||||
|
||||
func TestRun_HeartbeatsEngineHealth(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
// Declining to set a cadence leaves the agent's own, which the test shortens
|
||||
// rather than waiting out the 30s production interval.
|
||||
cp.registerOut.HeartbeatIntervalSeconds = 0
|
||||
|
||||
cfg := testConfig(t)
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
a.hb.interval = 10 * time.Millisecond
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- a.Run(ctx) }()
|
||||
|
||||
waitFor(t, func() bool { return len(cp.snapshotStates()) > 0 }, "a heartbeat")
|
||||
cancel()
|
||||
<-done
|
||||
|
||||
beat := cp.snapshotStates()[0]
|
||||
if beat.health != handlers_rds.EngineHealthHealthy {
|
||||
t.Errorf("heartbeat health = %q, want healthy", beat.health)
|
||||
}
|
||||
if beat.id.DBInstanceIdentifier != "db-resolved" {
|
||||
t.Errorf("heartbeat identifier = %q, want db-resolved", beat.id.DBInstanceIdentifier)
|
||||
}
|
||||
}
|
||||
|
||||
// The cadence is the control plane's to set: it feeds the staleness window the
|
||||
// recovery reconciler judges an instance against, so the guest adopts it rather
|
||||
// than keeping its own.
|
||||
func TestRegister_AdoptsControlPlaneCadence(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
cp.registerOut.HeartbeatIntervalSeconds = 45
|
||||
|
||||
cfg := testConfig(t)
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
if err := runAgent(t, a); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if a.hb.interval != 45*time.Second {
|
||||
t.Errorf("heartbeat interval = %v, want the 45s the control plane returned", a.hb.interval)
|
||||
}
|
||||
}
|
||||
|
||||
// The agent being up says nothing about the engine. A down engine has to reach
|
||||
// the control plane as such, or auto-recovery has nothing to act on.
|
||||
func TestRun_HeartbeatReportsDownEngine(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.bootstrapOut = bootstrapOutput(nil)
|
||||
cp.registerOut.HeartbeatIntervalSeconds = 0
|
||||
|
||||
cfg := testConfig(t)
|
||||
probe := newPostgresProbe(cfg, staticProbe(2))
|
||||
// The engine has served before, so a silent one now is a failure rather
|
||||
// than a boot still in progress.
|
||||
probe.seenHealthy = true
|
||||
|
||||
a, err := newAgent(cfg, cp, probe)
|
||||
if err != nil {
|
||||
t.Fatalf("newAgent: %v", err)
|
||||
}
|
||||
a.hb.interval = 10 * time.Millisecond
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- a.Run(ctx) }()
|
||||
|
||||
waitFor(t, func() bool { return len(cp.snapshotStates()) > 0 }, "a heartbeat")
|
||||
cancel()
|
||||
<-done
|
||||
|
||||
beat := cp.snapshotStates()[0]
|
||||
if beat.health != handlers_rds.EngineHealthUnhealthy {
|
||||
t.Errorf("heartbeat health = %q, want unhealthy while the agent is up", beat.health)
|
||||
}
|
||||
if beat.message == "" {
|
||||
t.Error("unhealthy heartbeat carried no message explaining the probe result")
|
||||
}
|
||||
}
|
||||
|
||||
// The mirror the agent decodes poll responses into has to stay aligned with the
|
||||
// gateway type that produces them; nothing else in the build ties the two.
|
||||
func TestPollDBCommandsOutput_MatchesGatewayShape(t *testing.T) {
|
||||
issued := time.Date(2026, 7, 28, 1, 2, 3, 0, time.UTC)
|
||||
// Rendered exactly as the gateway's typed adapter renders it, so a change to
|
||||
// either side's tags fails here.
|
||||
payload := utils.GenerateIAMXMLPayload("PollDBCommands", &gateway_rds.PollDBCommandsOutput{
|
||||
Commands: []handlers_rds.Command{{
|
||||
CommandID: "cmd-1",
|
||||
Type: "reload-parameters",
|
||||
Parameters: []handlers_rds.Parameter{{Name: "work_mem", Value: "8MB"}},
|
||||
IssuedAt: &issued,
|
||||
}},
|
||||
})
|
||||
body, err := utils.MarshalToXML(payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal poll result: %v", err)
|
||||
}
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/xml")
|
||||
_, _ = w.Write(body)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
client, err := rdsgw.New(srv.URL, "", gwsign.NewStatic("AKIATEST", "secret"), "us-east-1", time.Second)
|
||||
if err != nil {
|
||||
t.Fatalf("rdsgw.New: %v", err)
|
||||
}
|
||||
|
||||
var got pollDBCommandsOutput
|
||||
if err := client.Call(context.Background(), "PollDBCommands", nil, &got); err != nil {
|
||||
t.Fatalf("Call: %v", err)
|
||||
}
|
||||
if len(got.Commands) != 1 {
|
||||
t.Fatalf("decoded %d commands, want 1", len(got.Commands))
|
||||
}
|
||||
cmd := got.Commands[0]
|
||||
if cmd.CommandID != "cmd-1" || cmd.Type != "reload-parameters" {
|
||||
t.Errorf("command = %+v, want cmd-1/reload-parameters", cmd)
|
||||
}
|
||||
if len(cmd.Parameters) != 1 || cmd.Parameters[0].Value != "8MB" {
|
||||
t.Errorf("Parameters = %+v, want the seeded member", cmd.Parameters)
|
||||
}
|
||||
if cmd.IssuedAt == nil || !cmd.IssuedAt.Equal(issued) {
|
||||
t.Errorf("IssuedAt = %v, want %v", cmd.IssuedAt, issued)
|
||||
}
|
||||
}
|
||||
|
||||
func readFile(t *testing.T, path string) string {
|
||||
t.Helper()
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", path, err)
|
||||
}
|
||||
return string(body)
|
||||
}
|
||||
|
||||
// AccessDenied is decided by the control plane's own record, so no retry can
|
||||
// change it. Looping on one would leave the command poller and the parameter
|
||||
// guard unstarted for the life of a VM whose engine is serving normally.
|
||||
func TestAcknowledgeBootstrap_StopsOnATerminalDenial(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.ackErrs = []error{&rdsgw.APIError{
|
||||
Action: "AcknowledgeDBBootstrap", StatusCode: 403,
|
||||
Code: awserrors.ErrorAccessDenied,
|
||||
Message: "bootstrap payload bp-x at generation 1 does not match the current state",
|
||||
}}
|
||||
cfg := testConfig(t)
|
||||
cfg.DataMount = t.TempDir()
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
a.id = identity{DBInstanceIdentifier: "db-resolved"}
|
||||
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
|
||||
writeReceipt(t, a, testPayloadID, "db-resolved")
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := a.acknowledgeBootstrap(ctx); err != nil {
|
||||
t.Fatalf("acknowledgeBootstrap = %v, want nil so the stateful loops still start", err)
|
||||
}
|
||||
if n := len(cp.ackRequests()); n != 1 {
|
||||
t.Errorf("acknowledgements = %d, want 1 attempt and no retry", n)
|
||||
}
|
||||
if a.hb.bootstrapFailure.Load() == nil {
|
||||
t.Error("a terminal denial must still reach the operator through the heartbeat")
|
||||
}
|
||||
if _, err := os.Stat(a.receiptPath()); err != nil {
|
||||
t.Error("a denied acknowledgement must leave the receipt for a later attempt")
|
||||
}
|
||||
}
|
||||
|
||||
// rds-init runs after this agent, so the receipt is absent for the whole initdb
|
||||
// window. Reporting that would put a failure on every healthy create's beat, and
|
||||
// then into the reason a create that timed out for some other cause records.
|
||||
func TestAcknowledgeBootstrap_DoesNotReportAnAbsentReceiptAsAFailure(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cfg := testConfig(t)
|
||||
cfg.DataMount = t.TempDir()
|
||||
a := newTestAgent(t, cfg, cp, staticProbe(0))
|
||||
a.id = identity{DBInstanceIdentifier: "db-resolved"}
|
||||
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := a.acknowledgeBootstrap(ctx); err == nil {
|
||||
t.Fatal("acknowledgeBootstrap returned nil, want it still waiting for rds-init")
|
||||
}
|
||||
if got := a.hb.bootstrapFailure.Load(); got != nil {
|
||||
t.Errorf("bootstrap failure = %q, want none while the receipt has simply not been written", *got)
|
||||
}
|
||||
|
||||
// A receipt that exists but names another instance is a real mismatch and is
|
||||
// reported, which is what separates the two cases.
|
||||
writeReceipt(t, a, testPayloadID, "db-the-snapshot-source")
|
||||
mismatch, cancelMismatch := context.WithTimeout(t.Context(), 50*time.Millisecond)
|
||||
defer cancelMismatch()
|
||||
if err := a.acknowledgeBootstrap(mismatch); err == nil {
|
||||
t.Fatal("a foreign receipt must not be acknowledged")
|
||||
}
|
||||
if a.hb.bootstrapFailure.Load() == nil {
|
||||
t.Error("a receipt naming another instance must be reported")
|
||||
}
|
||||
}
|
||||
@@ -1,237 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// The staged payload this boot has to acknowledge once PostgreSQL has applied
|
||||
// it. Nil on an attach boot, which makes the acknowledgement step a no-op.
|
||||
type pendingBootstrap struct {
|
||||
payloadID string
|
||||
vmGeneration int64
|
||||
dataVolumeID string
|
||||
}
|
||||
|
||||
// Fetches the boot material and writes the handoff rds-init is blocked waiting
|
||||
// for, then points the health probe at the assigned port.
|
||||
func (a *Agent) bootstrap(ctx context.Context) error {
|
||||
var cfg *handlers_rds.GetDBBootstrapConfigOutput
|
||||
if err := retryObserved(ctx, "bootstrap fetch", func(ctx context.Context) error {
|
||||
fetched, err := a.cp.GetBootstrapConfig(ctx, a.id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if fetched == nil {
|
||||
return fmt.Errorf("bootstrap fetch returned no config")
|
||||
}
|
||||
// A control plane predating the encrypted payload answers initialize with
|
||||
// nothing to initialise with. Retrying re-dials the queue group until an
|
||||
// upgraded node answers, so a mixed-version fleet heals within one boot.
|
||||
if fetched.Mode == handlers_rds.BootstrapModeInitialize &&
|
||||
(fetched.MasterUserPassword == nil || *fetched.MasterUserPassword == "") {
|
||||
return fmt.Errorf("bootstrap fetch returned mode=%s with no master password", fetched.Mode)
|
||||
}
|
||||
cfg = fetched
|
||||
return nil
|
||||
}, func(err error) {
|
||||
a.hb.setBootstrapFailure("bootstrap fetch", err)
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
a.hb.clearBootstrapFailure()
|
||||
|
||||
// The fetch mutates nothing, so a handoff that cannot be written is retried
|
||||
// against the same staged payload rather than against an attach response.
|
||||
if err := retryObserved(ctx, "bootstrap handoff", func(context.Context) error {
|
||||
return a.handoffWriter(a.cfg.HandoffDir, cfg)
|
||||
}, func(err error) {
|
||||
a.hb.setBootstrapFailure("bootstrap handoff", err)
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
a.hb.clearBootstrapFailure()
|
||||
if cfg.Port > 0 {
|
||||
a.probe.setPort(int(cfg.Port))
|
||||
}
|
||||
if cfg.BootstrapPending && cfg.PayloadID != "" {
|
||||
a.pending = &pendingBootstrap{
|
||||
payloadID: cfg.PayloadID,
|
||||
vmGeneration: cfg.VMGeneration,
|
||||
dataVolumeID: cfg.DataVolumeID,
|
||||
}
|
||||
}
|
||||
// Mode is logged but never branched on: rds-init decides whether to initdb
|
||||
// from the state of the datadir.
|
||||
slog.Info("rds-agent: bootstrap config delivered",
|
||||
"mode", cfg.Mode, "port", cfg.Port, "parameters", len(cfg.Parameters),
|
||||
"bootstrapPending", cfg.BootstrapPending)
|
||||
return nil
|
||||
}
|
||||
|
||||
// The handoff files rds-init reads. They live on tmpfs, so nothing survives a
|
||||
// reboot and the next boot re-fetches rather than reusing a stale password or a
|
||||
// re-minted cert.
|
||||
const (
|
||||
handoffEnvFile = "bootstrap.env"
|
||||
handoffParamsFile = "parameters.conf"
|
||||
handoffCertFile = "server.crt"
|
||||
handoffKeyFile = "server.key"
|
||||
|
||||
// Root-only: only rds-init reads these, and one holds the master password.
|
||||
handoffMode = 0o600
|
||||
handoffDirMode = 0o700
|
||||
)
|
||||
|
||||
// bootstrap.env is written last and renamed into place, so its appearance —
|
||||
// which rds-init waits on — means the whole handoff is complete.
|
||||
func writeHandoff(dir string, cfg *handlers_rds.GetDBBootstrapConfigOutput) error {
|
||||
if cfg.MasterUsername == "" {
|
||||
return fmt.Errorf("bootstrap config carries no master username")
|
||||
}
|
||||
// Half a handoff would leave rds-init deciding what to bind and who to admit,
|
||||
// and the only answers it could reach on its own are the wildcards this
|
||||
// contract exists to remove.
|
||||
if cfg.ListenAddress == "" {
|
||||
return fmt.Errorf("bootstrap config carries no listen address for the engine to bind")
|
||||
}
|
||||
if cfg.ClientCIDR == "" {
|
||||
return fmt.Errorf("bootstrap config carries no client CIDR to scope client authentication to")
|
||||
}
|
||||
if err := os.MkdirAll(dir, handoffDirMode); err != nil {
|
||||
return fmt.Errorf("create handoff dir %s: %w", dir, err)
|
||||
}
|
||||
// MkdirAll leaves an existing directory's mode alone, and this one may have
|
||||
// been created more permissively earlier in the boot.
|
||||
if err := os.Chmod(dir, handoffDirMode); err != nil {
|
||||
return fmt.Errorf("secure handoff dir %s: %w", dir, err)
|
||||
}
|
||||
|
||||
params, err := renderParameters(cfg.Engine, cfg.Parameters)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := writeHandoffFile(dir, handoffParamsFile, params); err != nil {
|
||||
return err
|
||||
}
|
||||
// Cert and key are written as a pair; half of one would start the engine
|
||||
// with TLS configured against a key it does not have.
|
||||
if cfg.ServingCertificate != "" && cfg.ServingPrivateKey != "" {
|
||||
if err := writeHandoffFile(dir, handoffCertFile, cfg.ServingCertificate); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := writeHandoffFile(dir, handoffKeyFile, cfg.ServingPrivateKey); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return writeHandoffFile(dir, handoffEnvFile, renderBootstrapEnv(cfg))
|
||||
}
|
||||
|
||||
// Every value is single-quoted because rds-init sources this with `.`: an
|
||||
// unquoted password holding a space, `$` or `;` would be word-split or executed.
|
||||
func renderBootstrapEnv(cfg *handlers_rds.GetDBBootstrapConfigOutput) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("# Written by rds-agent. Regenerated on every boot; edits are lost.\n")
|
||||
writeEnvLine(&b, "RDS_MODE", cfg.Mode)
|
||||
writeEnvLine(&b, "RDS_DB_INSTANCE_IDENTIFIER", cfg.DBInstanceIdentifier)
|
||||
writeEnvLine(&b, "RDS_MASTER_USERNAME", cfg.MasterUsername)
|
||||
writeEnvLine(&b, "RDS_LISTEN_ADDRESS", cfg.ListenAddress)
|
||||
writeEnvLine(&b, "RDS_CLIENT_CIDR", cfg.ClientCIDR)
|
||||
writeEnvLine(&b, "RDS_DATA_VOLUME_ID", cfg.DataVolumeID)
|
||||
writeEnvLine(&b, "RDS_DATA_VOLUME_SERIAL", cfg.DataVolumeSerial)
|
||||
writeEnvLine(&b, "RDS_VM_GENERATION", strconv.FormatInt(cfg.VMGeneration, 10))
|
||||
writeEnvLine(&b, "RDS_FORMAT_AUTHORIZED", strconv.FormatBool(cfg.FormatAuthorized))
|
||||
// Neither needs volume access, which is why the assertion is made here: the
|
||||
// agent runs before rds-datadir mounts the volume, so rds-init is the first
|
||||
// component that can compare it against the completion receipt on disk.
|
||||
writeEnvLine(&b, "RDS_BOOTSTRAP_PENDING", boolFlag(cfg.BootstrapPending))
|
||||
writeEnvLine(&b, "RDS_PAYLOAD_ID", cfg.PayloadID)
|
||||
// Present only in initialize mode: an attach fetch has no password to write
|
||||
// and rds-init must not find a stale one.
|
||||
if cfg.MasterUserPassword != nil {
|
||||
writeEnvLine(&b, "RDS_MASTER_PASSWORD", *cfg.MasterUserPassword)
|
||||
}
|
||||
if cfg.DBName != "" {
|
||||
writeEnvLine(&b, "RDS_DB_NAME", cfg.DBName)
|
||||
}
|
||||
if cfg.Port > 0 {
|
||||
writeEnvLine(&b, "RDS_PORT", strconv.FormatInt(cfg.Port, 10))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// rds-init compares against 1 rather than sourcing a Go boolean, so the two
|
||||
// halves of the guard cannot drift on spelling.
|
||||
func boolFlag(v bool) string {
|
||||
if v {
|
||||
return "1"
|
||||
}
|
||||
return "0"
|
||||
}
|
||||
|
||||
func writeEnvLine(b *strings.Builder, key, value string) {
|
||||
b.WriteString(key)
|
||||
b.WriteString("=")
|
||||
b.WriteString(shellQuote(value))
|
||||
b.WriteString("\n")
|
||||
}
|
||||
|
||||
// Single quotes suppress every shell expansion; an embedded quote is closed,
|
||||
// escaped and reopened.
|
||||
func shellQuote(s string) string {
|
||||
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
||||
}
|
||||
|
||||
// postgresql.conf syntax, which MariaDB's option files also parse. Values are
|
||||
// quoted so a space or unit suffix survives. Names are the engine's startup
|
||||
// spellings, not the customer's: the wrong one leaves the engine unable to boot.
|
||||
func renderParameters(engineName string, params []handlers_rds.Parameter) (string, error) {
|
||||
engine, err := handlers_rds.LookupEngine(engineName)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("render the resolved parameters: %w", err)
|
||||
}
|
||||
var b strings.Builder
|
||||
b.WriteString("# Resolved parameter group, written by rds-agent.\n")
|
||||
for _, p := range params {
|
||||
if p.Name == "" {
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, "%s = '%s'\n", engine.OptionFileName(p.Name), strings.ReplaceAll(p.Value, "'", "''"))
|
||||
}
|
||||
return b.String(), nil
|
||||
}
|
||||
|
||||
// Atomic: a temp file in the same directory, renamed over the target. The mode
|
||||
// is set at creation, so content is never briefly readable at the process umask.
|
||||
func writeHandoffFile(dir, name, content string) error {
|
||||
path := filepath.Join(dir, name)
|
||||
tmp := path + ".new"
|
||||
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, handoffMode)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create %s: %w", tmp, err)
|
||||
}
|
||||
// A failed write must not leave the temp file behind holding a password.
|
||||
if _, err := f.WriteString(content); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("write %s: %w", tmp, err)
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("close %s: %w", tmp, err)
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("install %s: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,280 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
func TestWriteHandoff_WritesEveryFileRootOnly(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "handoff")
|
||||
password := "s3cret"
|
||||
|
||||
err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
|
||||
Mode: handlers_rds.BootstrapModeInitialize,
|
||||
Engine: "postgres",
|
||||
MasterUsername: "master",
|
||||
ListenAddress: "10.0.1.20",
|
||||
ClientCIDR: "10.0.0.0/16",
|
||||
MasterUserPassword: &password,
|
||||
DBName: "appdb",
|
||||
Port: 5433,
|
||||
DataVolumeID: "vol-data-01",
|
||||
DataVolumeSerial: "voldata01",
|
||||
VMGeneration: 4,
|
||||
FormatAuthorized: true,
|
||||
Parameters: []handlers_rds.Parameter{
|
||||
{Name: "max_connections", Value: "200"},
|
||||
{Name: "shared_buffers", Value: "256MB"},
|
||||
},
|
||||
ServingCertificate: "CERT",
|
||||
ServingPrivateKey: "KEY",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("writeHandoff: %v", err)
|
||||
}
|
||||
|
||||
env := readFile(t, filepath.Join(dir, handoffEnvFile))
|
||||
for _, want := range []string{
|
||||
"RDS_MODE='initialize'",
|
||||
"RDS_MASTER_USERNAME='master'",
|
||||
"RDS_MASTER_PASSWORD='s3cret'",
|
||||
"RDS_DB_NAME='appdb'",
|
||||
"RDS_PORT='5433'",
|
||||
"RDS_DATA_VOLUME_ID='vol-data-01'",
|
||||
"RDS_DATA_VOLUME_SERIAL='voldata01'",
|
||||
"RDS_VM_GENERATION='4'",
|
||||
"RDS_FORMAT_AUTHORIZED='true'",
|
||||
} {
|
||||
if !strings.Contains(env, want) {
|
||||
t.Errorf("handoff env missing %s:\n%s", want, env)
|
||||
}
|
||||
}
|
||||
|
||||
params := readFile(t, filepath.Join(dir, handoffParamsFile))
|
||||
if !strings.Contains(params, "max_connections = '200'") || !strings.Contains(params, "shared_buffers = '256MB'") {
|
||||
t.Errorf("parameters.conf = %q, want both resolved settings", params)
|
||||
}
|
||||
if got := readFile(t, filepath.Join(dir, handoffCertFile)); got != "CERT" {
|
||||
t.Errorf("server.crt = %q, want CERT", got)
|
||||
}
|
||||
if got := readFile(t, filepath.Join(dir, handoffKeyFile)); got != "KEY" {
|
||||
t.Errorf("server.key = %q, want KEY", got)
|
||||
}
|
||||
|
||||
// The handoff holds a master password and a private key; anything readable
|
||||
// beyond root is a disclosure to every process in the guest.
|
||||
if mode := statMode(t, dir); mode != handoffDirMode {
|
||||
t.Errorf("handoff dir mode = %#o, want %#o", mode, handoffDirMode)
|
||||
}
|
||||
for _, name := range []string{handoffEnvFile, handoffParamsFile, handoffCertFile, handoffKeyFile} {
|
||||
if mode := statMode(t, filepath.Join(dir, name)); mode != handoffMode {
|
||||
t.Errorf("%s mode = %#o, want %#o", name, mode, handoffMode)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Attach mode carries no password, and the key must be absent rather than
|
||||
// present and empty — rds-init reads an empty value as "no password" only by
|
||||
// accident, and a later boot would inherit it.
|
||||
func TestWriteHandoff_AttachOmitsPassword(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "handoff")
|
||||
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
|
||||
Mode: handlers_rds.BootstrapModeAttach,
|
||||
Engine: "postgres",
|
||||
MasterUsername: "master",
|
||||
ListenAddress: "10.0.1.20",
|
||||
ClientCIDR: "10.0.0.0/16",
|
||||
Port: 5432,
|
||||
}); err != nil {
|
||||
t.Fatalf("writeHandoff: %v", err)
|
||||
}
|
||||
|
||||
env := readFile(t, filepath.Join(dir, handoffEnvFile))
|
||||
if strings.Contains(env, "RDS_MASTER_PASSWORD") {
|
||||
t.Errorf("handoff env = %q, want no RDS_MASTER_PASSWORD key", env)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"RDS_DATA_VOLUME_ID=''",
|
||||
"RDS_DATA_VOLUME_SERIAL=''",
|
||||
"RDS_VM_GENERATION='0'",
|
||||
"RDS_FORMAT_AUTHORIZED='false'",
|
||||
} {
|
||||
if !strings.Contains(env, want) {
|
||||
t.Errorf("handoff env missing fail-closed field %s: %q", want, env)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The env file is sourced by a shell, so a password holding shell metacharacters
|
||||
// must survive as data rather than being split, expanded or executed.
|
||||
func TestWriteHandoff_QuotesShellMetacharacters(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "handoff")
|
||||
password := `a b$(id);'"\` + "`whoami`"
|
||||
|
||||
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
|
||||
Mode: handlers_rds.BootstrapModeInitialize,
|
||||
Engine: "postgres",
|
||||
MasterUsername: "master",
|
||||
ListenAddress: "10.0.1.20",
|
||||
ClientCIDR: "10.0.0.0/16",
|
||||
MasterUserPassword: &password,
|
||||
Port: 5432,
|
||||
}); err != nil {
|
||||
t.Fatalf("writeHandoff: %v", err)
|
||||
}
|
||||
|
||||
env := readFile(t, filepath.Join(dir, handoffEnvFile))
|
||||
line := findLine(t, env, "RDS_MASTER_PASSWORD=")
|
||||
if got := shellSource(t, line, "RDS_MASTER_PASSWORD"); got != password {
|
||||
t.Errorf("sourced password = %q, want %q (line was %s)", got, password, line)
|
||||
}
|
||||
}
|
||||
|
||||
// A serving cert with no key would have rds-init configure TLS against a key
|
||||
// that is not there; the pair is delivered together or not at all.
|
||||
func TestWriteHandoff_SkipsHalfATLSPair(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "handoff")
|
||||
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
|
||||
Mode: handlers_rds.BootstrapModeAttach,
|
||||
Engine: "postgres",
|
||||
MasterUsername: "master",
|
||||
ListenAddress: "10.0.1.20",
|
||||
ClientCIDR: "10.0.0.0/16",
|
||||
Port: 5432,
|
||||
ServingCertificate: "CERT",
|
||||
}); err != nil {
|
||||
t.Fatalf("writeHandoff: %v", err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(filepath.Join(dir, handoffCertFile)); !os.IsNotExist(err) {
|
||||
t.Errorf("server.crt exists without a key; stat err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A config with no master username cannot bootstrap anything, and writing it
|
||||
// would only move the failure into rds-init after the boot has committed to it.
|
||||
func TestWriteHandoff_RejectsConfigWithNoMasterUser(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "handoff")
|
||||
err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
|
||||
Mode: handlers_rds.BootstrapModeAttach,
|
||||
Engine: "postgres",
|
||||
Port: 5432,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("writeHandoff accepted a config with no master username, want an error")
|
||||
}
|
||||
if _, statErr := os.Stat(filepath.Join(dir, handoffEnvFile)); !os.IsNotExist(statErr) {
|
||||
t.Errorf("handoff env was written despite the rejection; stat err = %v", statErr)
|
||||
}
|
||||
}
|
||||
|
||||
// A directory left over from an earlier boot at a looser mode must be tightened,
|
||||
// not accepted as it is found.
|
||||
func TestWriteHandoff_TightensExistingDirectory(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "handoff")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatalf("seed dir: %v", err)
|
||||
}
|
||||
|
||||
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
|
||||
Mode: handlers_rds.BootstrapModeAttach,
|
||||
Engine: "postgres",
|
||||
MasterUsername: "master",
|
||||
ListenAddress: "10.0.1.20",
|
||||
ClientCIDR: "10.0.0.0/16",
|
||||
Port: 5432,
|
||||
}); err != nil {
|
||||
t.Fatalf("writeHandoff: %v", err)
|
||||
}
|
||||
if mode := statMode(t, dir); mode != handoffDirMode {
|
||||
t.Errorf("handoff dir mode = %#o, want it tightened to %#o", mode, handoffDirMode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestShellQuote_EscapesEmbeddedQuotes(t *testing.T) {
|
||||
for _, in := range []string{"", "plain", "it's", `''`, "new\nline", `$HOME`} {
|
||||
if got := shellSource(t, "V="+shellQuote(in), "V"); got != in {
|
||||
t.Errorf("shellQuote(%q) sourced back as %q", in, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// shellSource evaluates one KEY=value assignment in a real shell and returns
|
||||
// what the variable ends up holding. The quoting is only correct if a shell
|
||||
// agrees, so the test asks one instead of re-implementing its rules.
|
||||
func shellSource(t *testing.T, line, name string) string {
|
||||
t.Helper()
|
||||
out, err := exec.Command("/bin/sh", "-c", line+"\nprintf %s \"$"+name+"\"\n").Output()
|
||||
if err != nil {
|
||||
t.Fatalf("sourcing %q: %v", line, err)
|
||||
}
|
||||
return string(out)
|
||||
}
|
||||
|
||||
func statMode(t *testing.T, path string) os.FileMode {
|
||||
t.Helper()
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
t.Fatalf("stat %s: %v", path, err)
|
||||
}
|
||||
return info.Mode().Perm()
|
||||
}
|
||||
|
||||
func findLine(t *testing.T, body, prefix string) string {
|
||||
t.Helper()
|
||||
for line := range strings.SplitSeq(body, "\n") {
|
||||
if strings.HasPrefix(line, prefix) {
|
||||
return line
|
||||
}
|
||||
}
|
||||
t.Fatalf("no line starting %q in:\n%s", prefix, body)
|
||||
return ""
|
||||
}
|
||||
|
||||
// The include rds-init installs is what the engine starts on, so the spelling
|
||||
// written here is the one that has to boot.
|
||||
func TestRenderParameters_UsesTheEngineStartupSpelling(t *testing.T) {
|
||||
params := []handlers_rds.Parameter{
|
||||
{Name: "time_zone", Value: "SYSTEM"},
|
||||
{Name: "max_connections", Value: "85"},
|
||||
}
|
||||
|
||||
rendered, err := renderParameters("mariadb", params)
|
||||
if err != nil {
|
||||
t.Fatalf("renderParameters: %v", err)
|
||||
}
|
||||
if !strings.Contains(rendered, "default_time_zone = 'SYSTEM'\n") {
|
||||
t.Errorf("time_zone was not written under its startup spelling:\n%s", rendered)
|
||||
}
|
||||
if strings.Contains(rendered, "\ntime_zone = ") {
|
||||
t.Errorf("time_zone reached the option file, which mariadbd refuses:\n%s", rendered)
|
||||
}
|
||||
if !strings.Contains(rendered, "max_connections = '85'\n") {
|
||||
t.Errorf("an unmapped parameter was not written unchanged:\n%s", rendered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderParameters_PostgresKeepsEveryName(t *testing.T) {
|
||||
rendered, err := renderParameters("postgres", []handlers_rds.Parameter{
|
||||
{Name: "shared_buffers", Value: "262144"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("renderParameters: %v", err)
|
||||
}
|
||||
if !strings.Contains(rendered, "shared_buffers = '262144'\n") {
|
||||
t.Errorf("postgres name was rewritten:\n%s", rendered)
|
||||
}
|
||||
}
|
||||
|
||||
// Failing closed beats writing a file the engine may refuse: the bad include
|
||||
// would already be on the data volume by the time the server rejected it.
|
||||
func TestRenderParameters_RefusesAnUnknownEngine(t *testing.T) {
|
||||
if _, err := renderParameters("", nil); err == nil {
|
||||
t.Fatal("renderParameters accepted a config naming no engine, want an error")
|
||||
}
|
||||
}
|
||||
@@ -1,166 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// Executes one control-plane directive inside the guest and returns the message
|
||||
// reported back to the issuer.
|
||||
type commandHandler func(ctx context.Context, cmd handlers_rds.Command) (string, error)
|
||||
|
||||
// A registry rather than a dispatch switch, so guest operations — password
|
||||
// apply, parameter reload, storage grow, snapshot quiesce — add an entry each.
|
||||
type commandRegistry map[string]commandHandler
|
||||
|
||||
// An unregistered type is replied to as unsupported, so a control plane ahead
|
||||
// of the guest gets a clear answer rather than a timeout.
|
||||
func newCommandRegistry(engine engineOps, storage storageOps) commandRegistry {
|
||||
return commandRegistry{
|
||||
handlers_rds.CommandSetPassword: func(ctx context.Context, cmd handlers_rds.Command) (string, error) {
|
||||
params := commandParams(cmd)
|
||||
return "", engine.SetPassword(ctx,
|
||||
params[handlers_rds.CommandParamMasterUsername],
|
||||
params[handlers_rds.CommandParamMasterUserPassword])
|
||||
},
|
||||
handlers_rds.CommandApplyParams: func(ctx context.Context, cmd handlers_rds.Command) (string, error) {
|
||||
pending, err := engine.ApplyParameters(ctx, cmd.Parameters)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// The reply carries no structured payload, so the settings still
|
||||
// awaiting a restart come back as the message itself.
|
||||
return strings.Join(pending, ","), nil
|
||||
},
|
||||
handlers_rds.CommandStopEngine: func(ctx context.Context, cmd handlers_rds.Command) (string, error) {
|
||||
return "", engine.Stop(ctx)
|
||||
},
|
||||
handlers_rds.CommandGrowFilesystem: func(ctx context.Context, cmd handlers_rds.Command) (string, error) {
|
||||
return storage.GrowFilesystem(ctx)
|
||||
},
|
||||
handlers_rds.CommandQuiesce: func(ctx context.Context, cmd handlers_rds.Command) (string, error) {
|
||||
params := commandParams(cmd)
|
||||
hold, err := quiesceHoldFrom(params[handlers_rds.CommandParamQuiesceDeadlineSeconds])
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := engine.Quiesce(ctx, params[handlers_rds.CommandParamQuiesceLabel], hold); err != nil {
|
||||
return "", err
|
||||
}
|
||||
// After the hold, so what is flushed is the engine's checkpointed
|
||||
// state. Its own files are not the risk — those it repairs from the
|
||||
// WAL — but the parameter file beside them it never opens.
|
||||
return "", storage.SyncDataMount(ctx)
|
||||
},
|
||||
handlers_rds.CommandUnquiesce: func(ctx context.Context, cmd handlers_rds.Command) (string, error) {
|
||||
return "", engine.Unquiesce(ctx)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// The control plane's hold deadline. A missing or unreadable one is refused
|
||||
// rather than defaulted: a hold the guest chose the length of would not be the
|
||||
// bound the control plane is relying on.
|
||||
func quiesceHoldFrom(value string) (time.Duration, error) {
|
||||
seconds, err := strconv.Atoi(strings.TrimSpace(value))
|
||||
if err != nil || seconds <= 0 {
|
||||
return 0, fmt.Errorf("%s must be a positive number of seconds, got %q",
|
||||
handlers_rds.CommandParamQuiesceDeadlineSeconds, value)
|
||||
}
|
||||
return time.Duration(seconds) * time.Second, nil
|
||||
}
|
||||
|
||||
// A later value wins, so a malformed command carrying a parameter twice cannot
|
||||
// leave the handler acting on the first of two conflicting values.
|
||||
func commandParams(cmd handlers_rds.Command) map[string]string {
|
||||
params := make(map[string]string, len(cmd.Parameters))
|
||||
for _, p := range cmd.Parameters {
|
||||
params[p.Name] = p.Value
|
||||
}
|
||||
return params
|
||||
}
|
||||
|
||||
// Spaces retries after a failed poll, so a broken channel is not re-polled at
|
||||
// line rate for as long as it stays broken.
|
||||
const pollErrorBackoff = 5 * time.Second
|
||||
|
||||
// The agent's command channel: a long poll that carries back the replies for
|
||||
// whatever the previous poll delivered.
|
||||
type commander struct {
|
||||
cp controlPlane
|
||||
id identity
|
||||
registry commandRegistry
|
||||
wait time.Duration
|
||||
// Cleared only once a poll has carried them, so a failed poll re-delivers
|
||||
// rather than dropping the result of work the guest actually did.
|
||||
pending []handlers_rds.CommandReply
|
||||
}
|
||||
|
||||
func newCommander(cp controlPlane, registry commandRegistry, wait time.Duration) *commander {
|
||||
if wait <= 0 {
|
||||
wait = defaultPollWait
|
||||
}
|
||||
return &commander{cp: cp, registry: registry, wait: wait}
|
||||
}
|
||||
|
||||
func (c *commander) Run(ctx context.Context) {
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
commands, err := c.cp.PollCommands(ctx, c.id, c.pending, c.wait)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
slog.Warn("rds-agent: command poll failed", "err", err)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(pollErrorBackoff):
|
||||
}
|
||||
continue
|
||||
}
|
||||
c.pending = nil
|
||||
|
||||
for _, cmd := range commands {
|
||||
c.pending = append(c.pending, c.execute(ctx, cmd))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Every command gets a reply: the issuer is blocked on this command ID, so a
|
||||
// silent drop costs it a full timeout.
|
||||
func (c *commander) execute(ctx context.Context, cmd handlers_rds.Command) handlers_rds.CommandReply {
|
||||
handler, ok := c.registry[cmd.Type]
|
||||
if !ok {
|
||||
slog.Warn("rds-agent: unsupported command", "commandID", cmd.CommandID, "type", cmd.Type)
|
||||
return failedReply(cmd, fmt.Sprintf("unsupported command type %q", cmd.Type))
|
||||
}
|
||||
|
||||
slog.Info("rds-agent: executing command", "commandID", cmd.CommandID, "type", cmd.Type)
|
||||
message, err := handler(ctx, cmd)
|
||||
if err != nil {
|
||||
slog.Error("rds-agent: command failed", "commandID", cmd.CommandID, "type", cmd.Type, "err", err)
|
||||
return failedReply(cmd, err.Error())
|
||||
}
|
||||
return handlers_rds.CommandReply{
|
||||
CommandID: cmd.CommandID,
|
||||
Status: handlers_rds.CommandStatusSucceeded,
|
||||
Message: message,
|
||||
}
|
||||
}
|
||||
|
||||
func failedReply(cmd handlers_rds.Command, message string) handlers_rds.CommandReply {
|
||||
return handlers_rds.CommandReply{
|
||||
CommandID: cmd.CommandID,
|
||||
Status: handlers_rds.CommandStatusFailed,
|
||||
Message: message,
|
||||
}
|
||||
}
|
||||
@@ -1,173 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestCommander_DispatchesByNameAndReplies(t *testing.T) {
|
||||
var ran handlers_rds.Command
|
||||
registry := commandRegistry{
|
||||
"reload-parameters": func(_ context.Context, cmd handlers_rds.Command) (string, error) {
|
||||
ran = cmd
|
||||
return "reloaded", nil
|
||||
},
|
||||
}
|
||||
|
||||
reply := newCommander(nil, registry, time.Second).execute(context.Background(), handlers_rds.Command{
|
||||
CommandID: "cmd-1",
|
||||
Type: "reload-parameters",
|
||||
Parameters: []handlers_rds.Parameter{{Name: "work_mem", Value: "8MB"}},
|
||||
})
|
||||
|
||||
if ran.CommandID != "cmd-1" || len(ran.Parameters) != 1 {
|
||||
t.Errorf("handler received %+v, want the command with its parameters", ran)
|
||||
}
|
||||
if reply.Status != handlers_rds.CommandStatusSucceeded || reply.Message != "reloaded" {
|
||||
t.Errorf("reply = %+v, want succeeded/reloaded", reply)
|
||||
}
|
||||
if reply.CommandID != "cmd-1" {
|
||||
t.Errorf("reply command ID = %q, want cmd-1", reply.CommandID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCommander_FailedHandlerRepliesWithItsError(t *testing.T) {
|
||||
registry := commandRegistry{
|
||||
"grow-storage": func(context.Context, handlers_rds.Command) (string, error) {
|
||||
return "", errors.New("no space left on device")
|
||||
},
|
||||
}
|
||||
|
||||
reply := newCommander(nil, registry, time.Second).execute(context.Background(),
|
||||
handlers_rds.Command{CommandID: "cmd-2", Type: "grow-storage"})
|
||||
|
||||
if reply.Status != handlers_rds.CommandStatusFailed {
|
||||
t.Errorf("reply status = %q, want failed", reply.Status)
|
||||
}
|
||||
if reply.Message != "no space left on device" {
|
||||
t.Errorf("reply message = %q, want the handler's error", reply.Message)
|
||||
}
|
||||
}
|
||||
|
||||
// A control plane ahead of the guest issues types this build has no handler
|
||||
// for. The issuer is blocked on the command ID, so it gets an answer rather
|
||||
// than a timeout.
|
||||
func TestCommander_UnknownTypeRepliesFailedNotDropped(t *testing.T) {
|
||||
reply := newCommander(nil, newCommandRegistry(&fakeEngine{}, &fakeStorage{}), time.Second).execute(context.Background(),
|
||||
handlers_rds.Command{CommandID: "cmd-3", Type: "quiesce"})
|
||||
|
||||
if reply.CommandID != "cmd-3" || reply.Status != handlers_rds.CommandStatusFailed {
|
||||
t.Errorf("reply = %+v, want a failed reply for cmd-3", reply)
|
||||
}
|
||||
if reply.Message == "" {
|
||||
t.Error("unsupported-command reply carried no message")
|
||||
}
|
||||
}
|
||||
|
||||
// The reply for work the guest actually did rides the next poll. A poll that
|
||||
// fails must not consume it, or the issuer never learns the outcome.
|
||||
func TestCommander_RunCarriesRepliesOnTheNextPoll(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.pollQueue = [][]handlers_rds.Command{
|
||||
{{CommandID: "cmd-1", Type: "reload-parameters"}},
|
||||
}
|
||||
registry := commandRegistry{
|
||||
"reload-parameters": func(context.Context, handlers_rds.Command) (string, error) { return "done", nil },
|
||||
}
|
||||
|
||||
c := newCommander(cp, registry, 10*time.Millisecond)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
done := make(chan struct{})
|
||||
go func() { c.Run(ctx); close(done) }()
|
||||
|
||||
waitFor(t, func() bool {
|
||||
replies := cp.snapshotReplies()
|
||||
return len(replies) >= 2 && len(replies[1]) == 1
|
||||
}, "the reply to ride the second poll")
|
||||
cancel()
|
||||
<-done
|
||||
|
||||
replies := cp.snapshotReplies()
|
||||
if len(replies[0]) != 0 {
|
||||
t.Errorf("first poll carried replies %+v, want none", replies[0])
|
||||
}
|
||||
if got := replies[1][0]; got.CommandID != "cmd-1" || got.Status != handlers_rds.CommandStatusSucceeded {
|
||||
t.Errorf("second poll carried %+v, want the succeeded reply for cmd-1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCommander_RetainsAndRetriesTheCompletePendingBatch(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
pending := []handlers_rds.CommandReply{
|
||||
{CommandID: "cmd-1", Status: handlers_rds.CommandStatusSucceeded},
|
||||
{CommandID: "cmd-2", Status: handlers_rds.CommandStatusFailed, Message: "disk full"},
|
||||
}
|
||||
c := newCommander(cp, commandRegistry{
|
||||
"reload-parameters": func(context.Context, handlers_rds.Command) (string, error) {
|
||||
return "reloaded", nil
|
||||
},
|
||||
}, 10*time.Millisecond)
|
||||
c.pending = append([]handlers_rds.CommandReply(nil), pending...)
|
||||
cp.pollErr = errors.New("reply publication failed")
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan struct{})
|
||||
go func() { c.Run(ctx); close(done) }()
|
||||
<-cp.polled
|
||||
cancel()
|
||||
<-done
|
||||
|
||||
assert.Equal(t, pending, c.pending)
|
||||
replies := cp.snapshotReplies()
|
||||
require.Len(t, replies, 1)
|
||||
assert.Equal(t, pending, replies[0])
|
||||
|
||||
cp.mu.Lock()
|
||||
cp.pollErr = nil
|
||||
cp.pollQueue = [][]handlers_rds.Command{{{
|
||||
CommandID: "cmd-3",
|
||||
Type: "reload-parameters",
|
||||
}}}
|
||||
cp.mu.Unlock()
|
||||
|
||||
ctx, cancel = context.WithCancel(context.Background())
|
||||
done = make(chan struct{})
|
||||
go func() { c.Run(ctx); close(done) }()
|
||||
waitFor(t, func() bool {
|
||||
return len(cp.snapshotReplies()) >= 3
|
||||
}, "the retained batch to succeed and the new reply to reach the following poll")
|
||||
cancel()
|
||||
<-done
|
||||
|
||||
replies = cp.snapshotReplies()
|
||||
assert.Equal(t, pending, replies[1], "the complete failed batch must be retried in order")
|
||||
require.Len(t, replies[2], 1)
|
||||
assert.Equal(t, "cmd-3", replies[2][0].CommandID)
|
||||
assert.Equal(t, handlers_rds.CommandStatusSucceeded, replies[2][0].Status)
|
||||
}
|
||||
|
||||
// A poll error must not spin the loop: without a backoff a gateway that is down
|
||||
// would be re-polled at line rate for as long as it stayed down.
|
||||
func TestCommander_BacksOffAfterAFailedPoll(t *testing.T) {
|
||||
cp := newFakeControlPlane()
|
||||
cp.pollErr = errors.New("gateway unreachable")
|
||||
|
||||
c := newCommander(cp, newCommandRegistry(&fakeEngine{}, &fakeStorage{}), 10*time.Millisecond)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
|
||||
defer cancel()
|
||||
done := make(chan struct{})
|
||||
go func() { c.Run(ctx); close(done) }()
|
||||
<-done
|
||||
|
||||
// pollErrorBackoff is 5s, so the window admits the first poll and no more.
|
||||
if polls := len(cp.snapshotReplies()); polls > 1 {
|
||||
t.Errorf("polled %d times in 200ms against a broken channel, want 1", polls)
|
||||
}
|
||||
}
|
||||
@@ -1,185 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/internal/guestenv"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultEnvFile = "/etc/spinifex-rds/agent.env"
|
||||
defaultGatewayCA = "/etc/spinifex-rds/gateway-ca.pem"
|
||||
defaultHandoffDir = "/run/spinifex-rds"
|
||||
// Where setup.sh stamps the engine the image bakes. The agent builds its
|
||||
// engine implementation from this file rather than from anything delivered.
|
||||
defaultEngineFile = "/etc/spinifex-rds/engine"
|
||||
defaultRCService = "rc-service"
|
||||
defaultEngineProbeTimeout = 5 * time.Second
|
||||
// The long-poll window the agent asks the gateway to hold a request open
|
||||
// for. The gateway caps it at 20s.
|
||||
defaultPollWait = 20 * time.Second
|
||||
)
|
||||
|
||||
// Static settings delivered per-instance by cloud-init. It carries no secrets:
|
||||
// IMDS is readable by anything in the guest, so the master password only
|
||||
// arrives via GetDBBootstrapConfig.
|
||||
type config struct {
|
||||
GatewayURL string
|
||||
GatewayCA string
|
||||
Region string
|
||||
// Optional — the gateway resolves the instance from the caller's
|
||||
// credentials. When set it is sent, so a mis-provisioned VM is rejected.
|
||||
DBInstanceIdentifier string
|
||||
// What this image actually ships. Empty leaves the control plane's recorded
|
||||
// version alone rather than clearing it.
|
||||
EngineVersion string
|
||||
HandoffDir string
|
||||
EnginePort int
|
||||
PollWait time.Duration
|
||||
|
||||
// The engine the control plane launched this VM as, delivered by cloud-init,
|
||||
// and the engine the image itself bakes. Two independent statements of the
|
||||
// same fact: the agent refuses to bootstrap when they disagree.
|
||||
Engine string
|
||||
EngineFile string
|
||||
BakedEngine string
|
||||
|
||||
// Where the command handlers reach the engine: the client binaries, the
|
||||
// datadir the parameter file is installed into, the socket they connect
|
||||
// over, the OS user they drop to, and the service the stop goes through.
|
||||
EngineBinDir string
|
||||
EngineDataDir string
|
||||
SocketDir string
|
||||
EngineUser string
|
||||
RCService string
|
||||
EngineService string
|
||||
// Where the engine records the pid its probe checks for liveness, for an
|
||||
// engine whose own client cannot tell a server still recovering from one that
|
||||
// is not running at all.
|
||||
EnginePidFile string
|
||||
EngineProbeTimeout time.Duration
|
||||
// Where the engine records why it would not start, which the probe quotes
|
||||
// when the server is up but not serving. Overridable so a test can point it
|
||||
// at a fixture.
|
||||
EngineErrorLog string
|
||||
|
||||
// Where the data volume is mounted, and the two kernel surfaces a storage
|
||||
// grow resolves its device from. Overridable so a test can point them at
|
||||
// fixtures rather than at the running host's.
|
||||
DataMount string
|
||||
MountsFile string
|
||||
SysBlock string
|
||||
}
|
||||
|
||||
// Reads the cloud-init env file, then lets real env vars override.
|
||||
func loadConfig(envFile string) config {
|
||||
get := guestenv.Load(envFile).Get
|
||||
|
||||
cfg := config{
|
||||
GatewayURL: get("RDS_GATEWAY_URL"),
|
||||
GatewayCA: get("RDS_GATEWAY_CA"),
|
||||
Region: get("RDS_REGION"),
|
||||
DBInstanceIdentifier: get("RDS_DB_INSTANCE_IDENTIFIER"),
|
||||
Engine: get("RDS_ENGINE"),
|
||||
EngineFile: get("RDS_ENGINE_FILE"),
|
||||
EngineVersion: get("RDS_ENGINE_VERSION"),
|
||||
HandoffDir: get("RDS_HANDOFF_DIR"),
|
||||
EngineBinDir: get("RDS_ENGINE_BIN"),
|
||||
EngineDataDir: get("RDS_ENGINE_DATA"),
|
||||
SocketDir: get("RDS_SOCKET_DIR"),
|
||||
EngineUser: get("RDS_ENGINE_USER"),
|
||||
RCService: get("RDS_RC_SERVICE"),
|
||||
EngineService: get("RDS_ENGINE_SERVICE"),
|
||||
EnginePidFile: get("RDS_ENGINE_PIDFILE"),
|
||||
EngineErrorLog: get("RDS_ENGINE_LOG"),
|
||||
DataMount: get("RDS_DATA_MOUNT"),
|
||||
PollWait: defaultPollWait,
|
||||
EngineProbeTimeout: defaultEngineProbeTimeout,
|
||||
}
|
||||
if cfg.GatewayCA == "" {
|
||||
cfg.GatewayCA = defaultGatewayCA
|
||||
}
|
||||
if cfg.HandoffDir == "" {
|
||||
cfg.HandoffDir = defaultHandoffDir
|
||||
}
|
||||
if cfg.EngineFile == "" {
|
||||
cfg.EngineFile = defaultEngineFile
|
||||
}
|
||||
if cfg.RCService == "" {
|
||||
cfg.RCService = defaultRCService
|
||||
}
|
||||
// The image's own stamp rather than the delivered engine: an agent that took
|
||||
// its layout from the payload could run one engine's implementation against
|
||||
// another engine's datadir, which is the failure the stamp exists to prevent.
|
||||
cfg.BakedEngine = readBakedEngine(cfg.EngineFile)
|
||||
cfg.applyLayout(engineLayouts[cfg.BakedEngine])
|
||||
|
||||
// rds-datadir reads the same RDS_DATA_MOUNT, so the two agree on where the
|
||||
// volume landed without either asserting it to the other.
|
||||
cfg.MountsFile = defaultMountsFile
|
||||
cfg.SysBlock = defaultSysBlock
|
||||
// The authoritative port comes from the bootstrap config; this is only what
|
||||
// the probe uses until that first fetch lands.
|
||||
if v := get("RDS_ENGINE_PORT"); v != "" {
|
||||
if p, err := strconv.Atoi(v); err == nil && p > 0 {
|
||||
cfg.EnginePort = p
|
||||
}
|
||||
}
|
||||
if v := get("RDS_POLL_WAIT"); v != "" {
|
||||
if d, err := time.ParseDuration(v); err == nil && d > 0 {
|
||||
cfg.PollWait = d
|
||||
}
|
||||
}
|
||||
if v := get("RDS_ENGINE_PROBE_TIMEOUT"); v != "" {
|
||||
if d, err := time.ParseDuration(v); err == nil && d > 0 {
|
||||
cfg.EngineProbeTimeout = d
|
||||
}
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
// Fills in whatever the delivered configuration left unset. An override always
|
||||
// wins, so a test can point any of these at a fixture. An unrecognised engine
|
||||
// leaves the layout empty, and New refuses rather than guessing at one.
|
||||
func (c *config) applyLayout(layout engineLayout) {
|
||||
if c.EngineBinDir == "" {
|
||||
c.EngineBinDir = layout.binDir
|
||||
}
|
||||
if c.EngineDataDir == "" {
|
||||
c.EngineDataDir = layout.dataDir
|
||||
}
|
||||
if c.SocketDir == "" {
|
||||
c.SocketDir = layout.socketDir
|
||||
}
|
||||
if c.EngineUser == "" {
|
||||
c.EngineUser = layout.osUser
|
||||
}
|
||||
if c.EngineService == "" {
|
||||
c.EngineService = layout.service
|
||||
}
|
||||
if c.EnginePidFile == "" {
|
||||
c.EnginePidFile = layout.pidFile
|
||||
}
|
||||
if c.EngineErrorLog == "" {
|
||||
c.EngineErrorLog = layout.errorLog
|
||||
}
|
||||
if c.DataMount == "" {
|
||||
c.DataMount = layout.dataMount
|
||||
}
|
||||
if c.EnginePort == 0 {
|
||||
c.EnginePort = layout.port
|
||||
}
|
||||
}
|
||||
|
||||
// The engine the image bakes. An absent or unreadable stamp reads as empty,
|
||||
// which New refuses: an agent that guessed would defeat the stamp.
|
||||
func readBakedEngine(path string) string {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.ToLower(strings.TrimSpace(string(raw)))
|
||||
}
|
||||
@@ -1,162 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Stamps the engine a real image bakes and loads the configuration against it,
|
||||
// which is what selects the guest layout the agent's defaults come from.
|
||||
func testLoadConfig(t *testing.T, engine string) config {
|
||||
t.Helper()
|
||||
stamp := filepath.Join(t.TempDir(), "engine")
|
||||
if err := os.WriteFile(stamp, []byte(engine+"\n"), 0o444); err != nil {
|
||||
t.Fatalf("write the engine stamp: %v", err)
|
||||
}
|
||||
t.Setenv("RDS_ENGINE_FILE", stamp)
|
||||
return loadConfig(filepath.Join(t.TempDir(), "absent.env"))
|
||||
}
|
||||
|
||||
func TestLoadConfig_Defaults(t *testing.T) {
|
||||
cfg := testLoadConfig(t, enginePostgres)
|
||||
|
||||
if cfg.GatewayCA != defaultGatewayCA || cfg.HandoffDir != defaultHandoffDir {
|
||||
t.Errorf("CA/handoff = %q/%q, want the built-in defaults", cfg.GatewayCA, cfg.HandoffDir)
|
||||
}
|
||||
if cfg.PollWait != defaultPollWait {
|
||||
t.Errorf("pollWait = %v, want the built-in default", cfg.PollWait)
|
||||
}
|
||||
if cfg.EngineProbeTimeout != defaultEngineProbeTimeout {
|
||||
t.Errorf("engine probe timeout = %v, want %v", cfg.EngineProbeTimeout, defaultEngineProbeTimeout)
|
||||
}
|
||||
// Identity is optional: the gateway resolves it from the caller's
|
||||
// credentials, so an agent with no configured identifier is normal.
|
||||
if cfg.DBInstanceIdentifier != "" {
|
||||
t.Errorf("DBInstanceIdentifier = %q, want empty when unconfigured", cfg.DBInstanceIdentifier)
|
||||
}
|
||||
}
|
||||
|
||||
// The guest layout is the image's to state, not the control plane's: everything
|
||||
// the agent reaches the engine through comes from the stamp it was built with.
|
||||
func TestLoadConfig_TakesTheLayoutFromTheBakedEngine(t *testing.T) {
|
||||
cfg := testLoadConfig(t, enginePostgres)
|
||||
|
||||
if cfg.BakedEngine != enginePostgres {
|
||||
t.Fatalf("baked engine = %q, want %q", cfg.BakedEngine, enginePostgres)
|
||||
}
|
||||
layout := engineLayouts[enginePostgres]
|
||||
if cfg.EngineBinDir != layout.binDir || cfg.EngineDataDir != layout.dataDir {
|
||||
t.Errorf("bin/data = %q/%q, want the PostgreSQL layout", cfg.EngineBinDir, cfg.EngineDataDir)
|
||||
}
|
||||
if cfg.EngineUser != layout.osUser || cfg.EngineService != layout.service {
|
||||
t.Errorf("user/service = %q/%q, want the PostgreSQL layout", cfg.EngineUser, cfg.EngineService)
|
||||
}
|
||||
if cfg.SocketDir != layout.socketDir || cfg.DataMount != layout.dataMount {
|
||||
t.Errorf("socket/mount = %q/%q, want the PostgreSQL layout", cfg.SocketDir, cfg.DataMount)
|
||||
}
|
||||
if cfg.EnginePort != layout.port {
|
||||
t.Errorf("port = %d, want the PostgreSQL default %d", cfg.EnginePort, layout.port)
|
||||
}
|
||||
}
|
||||
|
||||
// An image with no stamp cannot say which engine it is, and an agent that
|
||||
// guessed would be the failure the stamp exists to prevent.
|
||||
func TestNewEngine_RefusesAnImageWithNoEngineStamp(t *testing.T) {
|
||||
t.Setenv("RDS_ENGINE_FILE", filepath.Join(t.TempDir(), "absent-engine"))
|
||||
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
|
||||
|
||||
if _, err := newProbe(cfg, staticProbe(0)); err == nil {
|
||||
t.Error("newProbe built a probe for an image carrying no engine stamp")
|
||||
}
|
||||
if _, err := newEngine(cfg, nil, nil, nil); err == nil {
|
||||
t.Error("newEngine built an engine for an image carrying no engine stamp")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewEngine_RefusesAnEngineItDoesNotImplement(t *testing.T) {
|
||||
cfg := testLoadConfig(t, "oracle")
|
||||
|
||||
_, err := newEngine(cfg, nil, nil, nil)
|
||||
if err == nil {
|
||||
t.Fatal("newEngine built an engine this agent does not implement")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "oracle") {
|
||||
t.Errorf("error = %v, want it to name the engine the image bakes", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The implementation is the agent's, but the definition it validates names and
|
||||
// classifies parameters against is the control plane's, which now offers MariaDB.
|
||||
func TestNewEngine_BuildsMariaDBForAMariaDBImage(t *testing.T) {
|
||||
cfg := testLoadConfig(t, engineMariaDB)
|
||||
probe, err := newProbe(cfg, staticProbe(0))
|
||||
if err != nil {
|
||||
t.Fatalf("newProbe: %v", err)
|
||||
}
|
||||
|
||||
built, err := newEngine(cfg, nil, nil, probe)
|
||||
if err != nil {
|
||||
t.Fatalf("newEngine: %v", err)
|
||||
}
|
||||
if _, ok := built.(*mariadbEngine); !ok {
|
||||
t.Errorf("newEngine built %T for a MariaDB image", built)
|
||||
}
|
||||
}
|
||||
|
||||
// The probe reads the pidfile Alpine's packaged service starts mariadbd with,
|
||||
// because nothing in the generated configuration can move it: the service passes
|
||||
// --pid-file on the command line, which beats an option file.
|
||||
func TestLoadConfig_TakesTheMariaDBPidfileFromTheLayout(t *testing.T) {
|
||||
cfg := testLoadConfig(t, engineMariaDB)
|
||||
|
||||
if cfg.EnginePidFile != engineLayouts[engineMariaDB].pidFile {
|
||||
t.Errorf("pidfile = %q, want the MariaDB layout's", cfg.EnginePidFile)
|
||||
}
|
||||
if cfg.EnginePidFile == "" {
|
||||
t.Error("an engine whose probe needs a pidfile was given none")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfig_ReadsEnvFile(t *testing.T) {
|
||||
t.Setenv("RDS_ENGINE_FILE", filepath.Join(t.TempDir(), "absent-engine"))
|
||||
path := filepath.Join(t.TempDir(), "agent.env")
|
||||
body := "RDS_GATEWAY_URL=https://gw.internal:9999\n" +
|
||||
"RDS_REGION=ap-southeast-2\n" +
|
||||
"RDS_DB_INSTANCE_IDENTIFIER=db-1\n" +
|
||||
"RDS_ENGINE=postgres\n" +
|
||||
"RDS_ENGINE_VERSION=18.1\n" +
|
||||
"RDS_ENGINE_PORT=6543\n" +
|
||||
"RDS_POLL_WAIT=5s\n"
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatalf("write env file: %v", err)
|
||||
}
|
||||
|
||||
cfg := loadConfig(path)
|
||||
if cfg.GatewayURL != "https://gw.internal:9999" || cfg.Region != "ap-southeast-2" {
|
||||
t.Errorf("gateway/region = %q/%q, want the delivered values", cfg.GatewayURL, cfg.Region)
|
||||
}
|
||||
if cfg.DBInstanceIdentifier != "db-1" || cfg.EngineVersion != "18.1" {
|
||||
t.Errorf("identity = %q/%q, want db-1/18.1", cfg.DBInstanceIdentifier, cfg.EngineVersion)
|
||||
}
|
||||
if cfg.Engine != enginePostgres {
|
||||
t.Errorf("delivered engine = %q, want the one cloud-init carried", cfg.Engine)
|
||||
}
|
||||
if cfg.EnginePort != 6543 || cfg.PollWait != 5*time.Second {
|
||||
t.Errorf("port/pollWait = %d/%v, want 6543/5s", cfg.EnginePort, cfg.PollWait)
|
||||
}
|
||||
}
|
||||
|
||||
// An unparseable override falls back to the layout's own default rather than
|
||||
// leaving the agent with a zero port or a zero-length long poll.
|
||||
func TestLoadConfig_IgnoresUnusableOverrides(t *testing.T) {
|
||||
t.Setenv("RDS_ENGINE_PORT", "not-a-port")
|
||||
t.Setenv("RDS_POLL_WAIT", "forever")
|
||||
|
||||
cfg := testLoadConfig(t, enginePostgres)
|
||||
if cfg.EnginePort != engineLayouts[enginePostgres].port || cfg.PollWait != defaultPollWait {
|
||||
t.Errorf("port/pollWait = %d/%v, want the defaults", cfg.EnginePort, cfg.PollWait)
|
||||
}
|
||||
}
|
||||
@@ -1,73 +0,0 @@
|
||||
package main
|
||||
|
||||
//test:in-package — the agent is a main package, which has no external test
|
||||
// package to import it from, and this covers the unexported default parameter
|
||||
// set the image bakes.
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// The include the image build validates against a real mariadbd. It is checked
|
||||
// in rather than generated during the build because the build runs inside the
|
||||
// libguestfs appliance, where there is no Go toolchain to resolve the catalog.
|
||||
const mariadbDefaultParametersFixture = "../../scripts/images/rds-mariadb/default-parameters.cnf"
|
||||
|
||||
// Byte-for-byte what rds-init installs on a smallest-class instance that
|
||||
// overrides nothing. Regenerate with RDS_UPDATE_FIXTURES=1 go test ./cmd/rds-agent/.
|
||||
//
|
||||
// Keeping it in step matters because setup.sh feeds exactly this file to
|
||||
// mariadbd: a catalog that grew a name the server refuses at startup would
|
||||
// otherwise reach a customer as an instance that never boots.
|
||||
func TestMariaDBDefaultParametersFixture_MatchesTheCatalog(t *testing.T) {
|
||||
engine, err := handlers_rds.LookupEngine("mariadb")
|
||||
if err != nil {
|
||||
t.Fatalf("LookupEngine: %v", err)
|
||||
}
|
||||
class := handlers_rds.SmallestInstanceClass()
|
||||
params, err := engine.ResolveEffectiveParameters(class, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveEffectiveParameters(%s): %v", class, err)
|
||||
}
|
||||
body, err := renderParameters("mariadb", params)
|
||||
if err != nil {
|
||||
t.Fatalf("renderParameters: %v", err)
|
||||
}
|
||||
want := mariadbParametersHeader + body
|
||||
|
||||
path := filepath.FromSlash(mariadbDefaultParametersFixture)
|
||||
if os.Getenv("RDS_UPDATE_FIXTURES") == "1" {
|
||||
if err := os.WriteFile(path, []byte(want), 0o644); err != nil {
|
||||
t.Fatalf("update %s: %v", path, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
got, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", path, err)
|
||||
}
|
||||
if string(got) != want {
|
||||
t.Errorf("%s is stale for class %s.\nRegenerate with RDS_UPDATE_FIXTURES=1 go test ./cmd/rds-agent/\n\ngot:\n%s\nwant:\n%s",
|
||||
path, class, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The one thing the fixture exists to prove cannot regress without the image
|
||||
// build catching it, asserted here too so a plain preflight names it.
|
||||
func TestMariaDBDefaultParametersFixture_CarriesNoSetOnlyName(t *testing.T) {
|
||||
got, err := os.ReadFile(filepath.FromSlash(mariadbDefaultParametersFixture))
|
||||
if err != nil {
|
||||
t.Fatalf("read the fixture: %v", err)
|
||||
}
|
||||
for _, name := range []string{"\ntime_zone ", "\ntx_isolation ", "\ninnodb_buffer_pool_instances "} {
|
||||
if strings.Contains(string(got), name) {
|
||||
t.Errorf("the default include carries %q, which mariadbd refuses at startup", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,336 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// The in-guest half of the engine seam. Everything engine-shaped that a
|
||||
// control-plane command needs lives behind this, so the command registry stays
|
||||
// engine-agnostic.
|
||||
type engineOps interface {
|
||||
// Rotates the master role's password live. Never persisted anywhere in the
|
||||
// guest: it exists only for the length of this call.
|
||||
SetPassword(ctx context.Context, username, password string) error
|
||||
// Installs the resolved parameter set and reloads. Returns the settings the
|
||||
// engine accepted but will not honour until it restarts.
|
||||
ApplyParameters(ctx context.Context, params []handlers_rds.Parameter) ([]string, error)
|
||||
// Shuts the engine down cleanly, so the data volume is checkpointed before
|
||||
// the VM is stopped or snapshotted.
|
||||
Stop(ctx context.Context) error
|
||||
// Puts the engine into backup mode so the datadir can be snapshotted at a
|
||||
// checkpoint. Releases itself after hold, whatever happens to the caller.
|
||||
Quiesce(ctx context.Context, label string, hold time.Duration) error
|
||||
// Takes the engine back out of backup mode. Fails when no hold is active,
|
||||
// which is how a hold that expired mid-snapshot becomes visible.
|
||||
Unquiesce(ctx context.Context) error
|
||||
}
|
||||
|
||||
// The rollback half of the parameter path, kept off engineOps because it is not
|
||||
// a control-plane directive: nothing issues it, the agent runs it when the
|
||||
// engine fails to come up after a parameter change.
|
||||
type parameterRecovery interface {
|
||||
// Reports whether the installed set differed and was replaced. True with a
|
||||
// non-nil error is a set already on disk whose re-derived TLS enforcement did
|
||||
// not land: the caller still has to restart and report.
|
||||
RestoreLastKnownGoodParameters(ctx context.Context) (bool, error)
|
||||
Restart(ctx context.Context) error
|
||||
}
|
||||
|
||||
// Records the parameter file only after the probe observes the engine serving
|
||||
// it. This is separate from apply, where static values are installed but not
|
||||
// adopted until a later restart.
|
||||
type servingParameterRecorder interface {
|
||||
RecordServingParameters(ctx context.Context) error
|
||||
}
|
||||
|
||||
// Everything one engine implementation owns. The narrow interfaces above are
|
||||
// what the heartbeat, the commander and the rollback guard each hold; this is
|
||||
// only what the agent has to construct.
|
||||
type engine interface {
|
||||
engineOps
|
||||
parameterRecovery
|
||||
servingParameterRecorder
|
||||
}
|
||||
|
||||
// The guest layout an engine preset lays down, keyed by the engine the image
|
||||
// bakes. Every default that belongs to one engine lives here, so nothing
|
||||
// outside an implementation names its paths, service, port or probe.
|
||||
type engineLayout struct {
|
||||
binDir string
|
||||
dataDir string
|
||||
socketDir string
|
||||
osUser string
|
||||
service string
|
||||
dataMount string
|
||||
port int
|
||||
// Where the engine records the pid the probe checks for liveness. Empty for
|
||||
// an engine whose probe answers without one.
|
||||
pidFile string
|
||||
// Where the engine records why it would not start. Empty for an engine whose
|
||||
// image does not direct one, which leaves the probe reporting its own reason
|
||||
// alone rather than guessing at a path.
|
||||
errorLog string
|
||||
// The engine's own liveness signal, which is the only part of engineProbe
|
||||
// that is not shared.
|
||||
newProbe func(cfg config, run probeRunner) *engineProbe
|
||||
// The implementation every control-plane directive is served by. It fails
|
||||
// when the control plane this agent was built against does not offer the
|
||||
// engine the image bakes, rather than serving one it has no definition for.
|
||||
newEngine func(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error)
|
||||
}
|
||||
|
||||
// The names the control plane knows these implementations by, which are also
|
||||
// what setup.sh stamps into each image.
|
||||
const (
|
||||
enginePostgres = "postgres"
|
||||
engineMariaDB = "mariadb"
|
||||
)
|
||||
|
||||
var engineLayouts = map[string]engineLayout{
|
||||
enginePostgres: {
|
||||
binDir: "/usr/libexec/postgresql18",
|
||||
dataDir: "/var/lib/postgresql/18/data",
|
||||
socketDir: "/run/postgresql",
|
||||
osUser: "postgres",
|
||||
service: "postgresql",
|
||||
dataMount: "/var/lib/postgresql",
|
||||
port: 5432,
|
||||
// logfile in the image's /etc/conf.d/postgresql. The postmaster states
|
||||
// which setting it refused there and nowhere else — the console shows only
|
||||
// that pg_ctl could not start it, on a guest with no SSH.
|
||||
errorLog: "/var/log/postgresql/postmaster.log",
|
||||
newProbe: newPostgresProbe,
|
||||
newEngine: newPostgresEngineFromCatalog,
|
||||
},
|
||||
engineMariaDB: {
|
||||
binDir: "/usr/bin",
|
||||
dataDir: "/var/lib/mysql/data",
|
||||
socketDir: "/run/mysqld",
|
||||
osUser: "mysql",
|
||||
service: "mariadb",
|
||||
dataMount: "/var/lib/mysql",
|
||||
port: 3306,
|
||||
// Alpine's service passes --pid-file=/run/mysqld/$RC_SVCNAME.pid, and the
|
||||
// command line beats the option file, so this is the path rather than
|
||||
// anything the drop-in names. setup.sh asserts it at build time.
|
||||
pidFile: "/run/mysqld/mariadb.pid",
|
||||
// log_error in the platform drop-in rds-init writes. Without it
|
||||
// mysqld_safe would send the server's errors to syslog, which nothing
|
||||
// collects off a guest that has no SSH.
|
||||
errorLog: "/var/log/mysql/error.log",
|
||||
newProbe: newMariaDBProbe,
|
||||
newEngine: newMariaDBEngineFromCatalog,
|
||||
},
|
||||
}
|
||||
|
||||
// How much of the engine's error log a probe reason carries. The server names
|
||||
// what it refused on in its last few lines, and the reason reaches a customer
|
||||
// through StatusInfos, so this quotes the tail rather than the whole file.
|
||||
const (
|
||||
engineErrorLogTailBytes = 4096
|
||||
engineErrorLogTailLines = 4
|
||||
)
|
||||
|
||||
// Appends the last few lines of the engine's error log to reason. An unset path,
|
||||
// an unreadable file or an empty one leaves reason alone: the probe's own
|
||||
// statement is still true, and a guessed cause would be worse than none.
|
||||
func withEngineLogTail(reason, path string) string {
|
||||
tail := engineLogTail(path)
|
||||
if tail == "" {
|
||||
return reason
|
||||
}
|
||||
return reason + "; the engine's log ends: " + tail
|
||||
}
|
||||
|
||||
func engineLogTail(path string) string {
|
||||
if path == "" {
|
||||
return ""
|
||||
}
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
offset := max(info.Size()-engineErrorLogTailBytes, 0)
|
||||
buf := make([]byte, info.Size()-offset)
|
||||
if _, err := f.ReadAt(buf, offset); err != nil {
|
||||
return ""
|
||||
}
|
||||
|
||||
lines := strings.Split(string(buf), "\n")
|
||||
// A read that started mid-file almost certainly started mid-line, and half a
|
||||
// line is more misleading than one fewer.
|
||||
if offset > 0 && len(lines) > 0 {
|
||||
lines = lines[1:]
|
||||
}
|
||||
kept := make([]string, 0, engineErrorLogTailLines)
|
||||
for i := len(lines) - 1; i >= 0 && len(kept) < engineErrorLogTailLines; i-- {
|
||||
if line := strings.TrimSpace(lines[i]); line != "" {
|
||||
kept = append(kept, line)
|
||||
}
|
||||
}
|
||||
slices.Reverse(kept)
|
||||
return strings.Join(kept, " | ")
|
||||
}
|
||||
|
||||
// Builds the health probe the image's engine supplies.
|
||||
func newProbe(cfg config, run probeRunner) (*engineProbe, error) {
|
||||
layout, err := layoutFor(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return layout.newProbe(cfg, run), nil
|
||||
}
|
||||
|
||||
// Builds the implementation the image bakes. Nothing here consults the control
|
||||
// plane for which engine to be: the agent runs the engine it is made of, and the
|
||||
// delivered engine is only ever checked against it.
|
||||
func newEngine(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error) {
|
||||
layout, err := layoutFor(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return layout.newEngine(cfg, run, startSess, probe)
|
||||
}
|
||||
|
||||
func layoutFor(cfg config) (engineLayout, error) {
|
||||
if cfg.BakedEngine == "" {
|
||||
return engineLayout{}, fmt.Errorf("this image carries no engine stamp at %s", cfg.EngineFile)
|
||||
}
|
||||
layout, ok := engineLayouts[cfg.BakedEngine]
|
||||
if !ok {
|
||||
return engineLayout{}, fmt.Errorf("this image bakes engine %q, which this agent does not implement", cfg.BakedEngine)
|
||||
}
|
||||
return layout, nil
|
||||
}
|
||||
|
||||
// One child process. Env replaces the agent's own environment rather than
|
||||
// extending it, so a secret placed here reaches only the process that needs it.
|
||||
type command struct {
|
||||
Name string
|
||||
Args []string
|
||||
Env []string
|
||||
Stdin string
|
||||
// The OS user to drop to before exec. Empty runs as the agent itself.
|
||||
User string
|
||||
// The statement that makes this client print the session sentinel. Read only
|
||||
// when the command is started as a held session, whose reader has to know
|
||||
// when the engine has finished each script it was fed.
|
||||
SentinelStatement string
|
||||
}
|
||||
|
||||
// Returns stdout. stderr is folded into the error, because both clients report
|
||||
// the actual SQL failure there while exiting with a bare status.
|
||||
type commandRunner func(ctx context.Context, c command) (string, error)
|
||||
|
||||
func execCommandRunner(ctx context.Context, c command) (string, error) {
|
||||
cmd := exec.CommandContext(ctx, c.Name, c.Args...)
|
||||
cmd.Env = c.Env
|
||||
if c.Stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(c.Stdin)
|
||||
}
|
||||
if c.User != "" {
|
||||
credential, err := lookupCredential(c.User)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Credential: credential}
|
||||
}
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
message := strings.TrimSpace(stderr.String())
|
||||
if message == "" {
|
||||
message = err.Error()
|
||||
}
|
||||
return stdout.String(), fmt.Errorf("%s: %s", filepath.Base(c.Name), message)
|
||||
}
|
||||
return stdout.String(), nil
|
||||
}
|
||||
|
||||
func lookupCredential(username string) (*syscall.Credential, error) {
|
||||
u, err := user.Lookup(username)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve user %s: %w", username, err)
|
||||
}
|
||||
uid, err := strconv.ParseUint(u.Uid, 10, 32)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse uid of %s: %w", username, err)
|
||||
}
|
||||
gid, err := strconv.ParseUint(u.Gid, 10, 32)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse gid of %s: %w", username, err)
|
||||
}
|
||||
return &syscall.Credential{Uid: uint32(uid), Gid: uint32(gid)}, nil
|
||||
}
|
||||
|
||||
// Through the service manager rather than the engine's own control tool, so the
|
||||
// supervisor records the state and does not restart the engine underneath a VM
|
||||
// that is going down.
|
||||
func serviceAction(ctx context.Context, run commandRunner, rcService, service, action string) error {
|
||||
if _, err := run(ctx, command{
|
||||
Name: rcService,
|
||||
Args: []string{service, action},
|
||||
Env: []string{"PATH=" + defaultGuestPath},
|
||||
}); err != nil {
|
||||
return fmt.Errorf("%s the %s service: %w", action, service, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Whether the installed set requires TLS of client connections. Read back from
|
||||
// the file rather than taken from the set being applied, because the restore and
|
||||
// repair paths put a file in place without holding one. Both engines' generated
|
||||
// parameter files are written in the same syntax, so one reader serves them.
|
||||
func installedTLSEnforcement(name, installedPath string) (bool, error) {
|
||||
values, err := readOptionFile(installedPath)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return false, fmt.Errorf("read the installed parameters: %w", err)
|
||||
}
|
||||
value, ok := values[name]
|
||||
if !ok {
|
||||
// The ordinary state of an instance whose resolved set predates the
|
||||
// parameter, and the whole of the migration story: it begins enforcing at
|
||||
// its next boot with no control-plane work at all.
|
||||
return true, nil
|
||||
}
|
||||
switch value {
|
||||
case "1":
|
||||
return true, nil
|
||||
case "0":
|
||||
return false, nil
|
||||
}
|
||||
// The resolver canonicalises every boolean, so anything else means the file
|
||||
// was written by something other than the platform. The permissive reading is
|
||||
// the one that must not be chosen for a setting that turns TLS off.
|
||||
return false, fmt.Errorf("the installed parameters set %s to %q, which is neither 1 nor 0", name, value)
|
||||
}
|
||||
|
||||
// A minimal environment for a child that is not carrying secrets.
|
||||
const defaultGuestPath = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
|
||||
func redact(s, secret string) string {
|
||||
if secret == "" {
|
||||
return s
|
||||
}
|
||||
return strings.ReplaceAll(s, secret, "[REDACTED]")
|
||||
}
|
||||
@@ -1,333 +0,0 @@
|
||||
package main
|
||||
|
||||
//test:in-package — the agent is a main package, which has no external test
|
||||
// package to import it from, and these cases are built on the unexported
|
||||
// engine interface and each implementation's own constructor.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// A probe the test drives directly, in place of whichever binaries the engine's
|
||||
// own state function would have run. Each implementation's state function is
|
||||
// covered on its own; what these cases exercise is what the engine does with the
|
||||
// answer.
|
||||
func drivenProbe(state *engineState) *engineProbe {
|
||||
return newEngineProbe(0, func(context.Context, int64) (engineState, string) {
|
||||
return *state, "driven by the test"
|
||||
})
|
||||
}
|
||||
|
||||
// One engine behind the interfaces the heartbeat, the commander and the rollback
|
||||
// guard hold, plus the few facts a shared case needs to say what it means.
|
||||
type engineHarness struct {
|
||||
engine engine
|
||||
runner *recordingRunner
|
||||
params parameterStore
|
||||
service string
|
||||
// The state the probe reports, moved by the test rather than by the engine.
|
||||
state *engineState
|
||||
// Points the next quiesce at a session the test can inspect, and returns the
|
||||
// commands those sessions were opened with.
|
||||
useSession func(*fakeSession) *[]command
|
||||
// A setting this engine's catalog really carries, so a shared case installs
|
||||
// something the implementation would genuinely be handed.
|
||||
parameter handlers_rds.Parameter
|
||||
// What the engine's own include directive globs. Neither the rollback copy
|
||||
// nor the serving copy may end in it.
|
||||
includeSuffix string
|
||||
}
|
||||
|
||||
func engineHarnesses() []struct {
|
||||
name string
|
||||
build func(*testing.T) *engineHarness
|
||||
} {
|
||||
return []struct {
|
||||
name string
|
||||
build func(*testing.T) *engineHarness
|
||||
}{
|
||||
{name: enginePostgres, build: buildPostgresHarness},
|
||||
{name: engineMariaDB, build: buildMariaDBHarness},
|
||||
}
|
||||
}
|
||||
|
||||
func forEachEngine(t *testing.T, run func(*testing.T, *engineHarness)) {
|
||||
t.Helper()
|
||||
for _, fixture := range engineHarnesses() {
|
||||
t.Run(fixture.name, func(t *testing.T) { run(t, fixture.build(t)) })
|
||||
}
|
||||
}
|
||||
|
||||
func buildPostgresHarness(t *testing.T) *engineHarness {
|
||||
t.Helper()
|
||||
runner := &recordingRunner{}
|
||||
e := newTestEngine(t, runner.run)
|
||||
state := engineServing
|
||||
e.probe = drivenProbe(&state)
|
||||
|
||||
return &engineHarness{
|
||||
engine: e, runner: runner, params: e.params, service: e.service, state: &state,
|
||||
useSession: func(session *fakeSession) *[]command {
|
||||
started := &[]command{}
|
||||
e.startSess = func(_ context.Context, c command) (engineSession, error) {
|
||||
*started = append(*started, c)
|
||||
return session, nil
|
||||
}
|
||||
return started
|
||||
},
|
||||
parameter: handlers_rds.Parameter{Name: "work_mem", Value: "4096"},
|
||||
includeSuffix: ".conf",
|
||||
}
|
||||
}
|
||||
|
||||
func buildMariaDBHarness(t *testing.T) *engineHarness {
|
||||
t.Helper()
|
||||
runner := &recordingRunner{}
|
||||
e := newTestMariaDBEngine(t, runner.run)
|
||||
state := engineServing
|
||||
e.probe = drivenProbe(&state)
|
||||
|
||||
return &engineHarness{
|
||||
engine: e, runner: runner, params: e.params, service: e.service, state: &state,
|
||||
useSession: func(session *fakeSession) *[]command {
|
||||
started := &[]command{}
|
||||
e.startSess = func(_ context.Context, c command) (engineSession, error) {
|
||||
*started = append(*started, c)
|
||||
return session, nil
|
||||
}
|
||||
return started
|
||||
},
|
||||
parameter: handlers_rds.Parameter{Name: "max_connections", Value: "200"},
|
||||
includeSuffix: ".cnf",
|
||||
}
|
||||
}
|
||||
|
||||
// The parameters land where rds-init installs them, so the next boot overwrites
|
||||
// the file rather than shadowing it with a second copy.
|
||||
func TestEngineContract_ApplyParametersInstallsWhereRdsInitDoes(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
if _, err := h.engine.ApplyParameters(t.Context(), []handlers_rds.Parameter{h.parameter}); err != nil {
|
||||
t.Fatalf("ApplyParameters: %v", err)
|
||||
}
|
||||
|
||||
installed, err := os.ReadFile(h.params.installedPath())
|
||||
if err != nil {
|
||||
t.Fatalf("read the installed parameters: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(installed), h.parameter.Name+" = '"+h.parameter.Value+"'") {
|
||||
t.Errorf("installed parameters = %q, want the resolved value", installed)
|
||||
}
|
||||
if !strings.HasSuffix(h.params.installedPath(), h.includeSuffix) {
|
||||
t.Errorf("the installed set at %s is not a file the engine includes", h.params.installedPath())
|
||||
}
|
||||
// Comparison material, not a second set of settings: an include glob that
|
||||
// read either of these would apply a withdrawn configuration.
|
||||
copies := []string{h.params.lastGoodPath()}
|
||||
if h.params.serving != "" {
|
||||
copies = append(copies, h.params.servingPath())
|
||||
}
|
||||
for _, path := range copies {
|
||||
if strings.HasSuffix(path, h.includeSuffix) {
|
||||
t.Errorf("the copy at %s would be included by the engine", path)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// An engine still starting or replaying has not finished adopting the set it is
|
||||
// on, so replacing that set underneath it would be installing over a moving
|
||||
// target.
|
||||
func TestEngineContract_ApplyParametersRefusesWhileTheEngineIsRecovering(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
*h.state = engineRecovering
|
||||
|
||||
if _, err := h.engine.ApplyParameters(t.Context(), []handlers_rds.Parameter{h.parameter}); err == nil {
|
||||
t.Fatal("ApplyParameters succeeded against an engine that was still coming up")
|
||||
}
|
||||
if _, err := os.Stat(h.params.installedPath()); !os.IsNotExist(err) {
|
||||
t.Errorf("the refused set was installed anyway (stat err = %v)", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// The boot-time half of the parameter safety net: the last set the engine
|
||||
// accepted goes back only when the engine is genuinely down, and only once.
|
||||
func TestEngineContract_RestoreLastKnownGoodOnlyRollsBackADownEngine(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
accepted := h.parameter
|
||||
if _, err := h.engine.ApplyParameters(t.Context(), []handlers_rds.Parameter{accepted}); err != nil {
|
||||
t.Fatalf("ApplyParameters: %v", err)
|
||||
}
|
||||
if err := h.engine.RecordServingParameters(t.Context()); err != nil {
|
||||
t.Fatalf("RecordServingParameters: %v", err)
|
||||
}
|
||||
if _, err := h.engine.ApplyParameters(t.Context(),
|
||||
[]handlers_rds.Parameter{{Name: accepted.Name, Value: accepted.Value + "0"}}); err != nil {
|
||||
t.Fatalf("second ApplyParameters: %v", err)
|
||||
}
|
||||
|
||||
// A guard whose deadline expired while an apply brought the engine back must
|
||||
// not reverse that repair.
|
||||
restored, err := h.engine.RestoreLastKnownGoodParameters(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("RestoreLastKnownGoodParameters against a serving engine: %v", err)
|
||||
}
|
||||
if restored {
|
||||
t.Fatal("rolled the parameters back while the engine was serving")
|
||||
}
|
||||
|
||||
*h.state = engineAbsent
|
||||
restored, err = h.engine.RestoreLastKnownGoodParameters(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("RestoreLastKnownGoodParameters: %v", err)
|
||||
}
|
||||
if !restored {
|
||||
t.Fatal("did not roll back to a set that differs from the last accepted one")
|
||||
}
|
||||
installed, err := os.ReadFile(h.params.installedPath())
|
||||
if err != nil {
|
||||
t.Fatalf("read the installed parameters: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(installed), accepted.Name+" = '"+accepted.Value+"'") {
|
||||
t.Errorf("installed = %q, want the last accepted set", installed)
|
||||
}
|
||||
|
||||
// Repeating a rollback would only churn a cluster that is failing for
|
||||
// another reason.
|
||||
restored, err = h.engine.RestoreLastKnownGoodParameters(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("RestoreLastKnownGoodParameters: %v", err)
|
||||
}
|
||||
if restored {
|
||||
t.Error("rolled back again when the installed set already matched")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Through the service manager rather than the engine's own control tool, so the
|
||||
// supervisor records the engine as stopped and does not restart it underneath a
|
||||
// VM that is going down.
|
||||
func TestEngineContract_StopGoesThroughTheServiceManager(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
if err := h.engine.Stop(t.Context()); err != nil {
|
||||
t.Fatalf("Stop: %v", err)
|
||||
}
|
||||
if len(h.runner.calls) != 1 {
|
||||
t.Fatalf("ran %d commands, want the service stop", len(h.runner.calls))
|
||||
}
|
||||
call := h.runner.calls[0]
|
||||
if !slices.Equal(call.Args, []string{h.service, "stop"}) {
|
||||
t.Errorf("stop ran %s %v, want the service manager", call.Name, call.Args)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// One backup at a time, released by Unquiesce or by its own deadline, whichever
|
||||
// comes first. The deadline is what keeps a control plane that dies mid-snapshot
|
||||
// from leaving the engine held.
|
||||
func TestEngineContract_QuiesceHoldsUntilReleased(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
session := &fakeSession{}
|
||||
started := h.useSession(session)
|
||||
|
||||
if err := h.engine.Quiesce(t.Context(), "orders-db-pre-upgrade", time.Minute); err != nil {
|
||||
t.Fatalf("Quiesce: %v", err)
|
||||
}
|
||||
if len(*started) != 1 {
|
||||
t.Fatalf("started %d sessions, want 1", len(*started))
|
||||
}
|
||||
if session.closes() != 0 {
|
||||
t.Fatal("the session was closed while the backup was meant to be held")
|
||||
}
|
||||
|
||||
err := h.engine.Quiesce(t.Context(), "second", time.Minute)
|
||||
if err == nil {
|
||||
t.Fatal("a second quiesce was accepted while the first was held")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "orders-db-pre-upgrade") {
|
||||
t.Errorf("error %q does not name the backup already holding the engine", err)
|
||||
}
|
||||
if len(*started) != 1 {
|
||||
t.Errorf("started %d sessions, want the second refused before opening one", len(*started))
|
||||
}
|
||||
|
||||
if err := h.engine.Unquiesce(t.Context()); err != nil {
|
||||
t.Fatalf("Unquiesce: %v", err)
|
||||
}
|
||||
if session.closes() != 1 {
|
||||
t.Errorf("closed %d times, want the session ended exactly once", session.closes())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A hold that expired mid-snapshot means the snapshot was not taken against a
|
||||
// held checkpoint, so the release reports it rather than succeeding silently.
|
||||
func TestEngineContract_QuiesceReleasesItselfOnItsDeadline(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
session := &fakeSession{}
|
||||
h.useSession(session)
|
||||
|
||||
if err := h.engine.Quiesce(t.Context(), "orders-db-pre-upgrade", 10*time.Millisecond); err != nil {
|
||||
t.Fatalf("Quiesce: %v", err)
|
||||
}
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for session.closes() == 0 && time.Now().Before(deadline) {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
if session.closes() != 1 {
|
||||
t.Fatal("the backup session outlived its deadline")
|
||||
}
|
||||
|
||||
err := h.engine.Unquiesce(t.Context())
|
||||
if err == nil {
|
||||
t.Fatal("Unquiesce succeeded after the hold expired")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "expired") {
|
||||
t.Errorf("error %q does not say the hold had expired", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A failed start leaves nothing held, so a retry is not refused by a hold that
|
||||
// never took effect.
|
||||
func TestEngineContract_QuiesceEndsTheSessionWhenTheBackupWillNotStart(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
session := &fakeSession{execErr: errors.New("the engine refused the backup")}
|
||||
h.useSession(session)
|
||||
|
||||
if err := h.engine.Quiesce(t.Context(), "orders-db-pre-upgrade", time.Minute); err == nil {
|
||||
t.Fatal("Quiesce succeeded against an engine that refused the backup")
|
||||
}
|
||||
if session.closes() != 1 {
|
||||
t.Errorf("closed %d times, want the failed session ended", session.closes())
|
||||
}
|
||||
if err := h.engine.Unquiesce(t.Context()); err == nil {
|
||||
t.Error("a failed quiesce left a hold behind")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Every command the control plane can issue is served by both implementations,
|
||||
// so a directive cannot be answered as unsupported on one engine and honoured on
|
||||
// the other.
|
||||
func TestEngineContract_ServesEveryRegisteredCommand(t *testing.T) {
|
||||
forEachEngine(t, func(t *testing.T, h *engineHarness) {
|
||||
registry := newCommandRegistry(h.engine, &fakeStorage{})
|
||||
for _, name := range []string{
|
||||
handlers_rds.CommandSetPassword, handlers_rds.CommandApplyParams, handlers_rds.CommandStopEngine,
|
||||
handlers_rds.CommandGrowFilesystem, handlers_rds.CommandQuiesce, handlers_rds.CommandUnquiesce,
|
||||
} {
|
||||
if _, ok := registry[name]; !ok {
|
||||
t.Errorf("command %q has no handler", name)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -1,262 +0,0 @@
|
||||
package main
|
||||
|
||||
//test:in-package — the agent is a main package, which has no external test
|
||||
// package to import it from, and this covers the unexported engineLogTail and
|
||||
// probe-reason helpers directly.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const mariadbRDSInitScript = "../../scripts/images/rds-mariadb/rds-init"
|
||||
|
||||
func writeEngineLog(t *testing.T, body string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "error.log")
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write engine log: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
// The whole point of the log: "not answering on its socket" is the same message
|
||||
// for a server still recovering and for one that refused to start, and only the
|
||||
// engine's own log separates them.
|
||||
func TestMariaDBProbe_QuotesTheEngineLogWhenTheSocketIsSilent(t *testing.T) {
|
||||
cfg := mariadbTestProbeConfig(t)
|
||||
cfg.EngineErrorLog = writeEngineLog(t, strings.Join([]string{
|
||||
"2026-08-12 1:49:26 0 [Note] Starting MariaDB 11.8.0",
|
||||
"2026-08-12 1:49:26 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool",
|
||||
"2026-08-12 1:49:26 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.",
|
||||
"",
|
||||
}, "\n"))
|
||||
writePid(t, cfg.EnginePidFile, os.Getpid())
|
||||
|
||||
state, message := mariadbProbeState(cfg,
|
||||
func(context.Context, string, ...string) (int, string, error) { return 1, "", nil },
|
||||
func(int) bool { return true })(t.Context(), int64(cfg.EnginePort))
|
||||
|
||||
if state != engineRecovering {
|
||||
t.Errorf("state = %v, want recovering", state)
|
||||
}
|
||||
if !strings.Contains(message, "Cannot allocate memory for the buffer pool") {
|
||||
t.Errorf("message = %q, want the engine's own refusal quoted", message)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMariaDBProbe_SurvivesAnAbsentEngineLog(t *testing.T) {
|
||||
cfg := mariadbTestProbeConfig(t)
|
||||
cfg.EngineErrorLog = filepath.Join(t.TempDir(), "does-not-exist.log")
|
||||
writePid(t, cfg.EnginePidFile, os.Getpid())
|
||||
|
||||
state, message := mariadbProbeState(cfg,
|
||||
func(context.Context, string, ...string) (int, string, error) { return 1, "", nil },
|
||||
func(int) bool { return true })(t.Context(), int64(cfg.EnginePort))
|
||||
|
||||
if state != engineRecovering {
|
||||
t.Errorf("state = %v, want recovering", state)
|
||||
}
|
||||
if message != "engine is not answering on its socket yet (startup or crash recovery)" {
|
||||
t.Errorf("message = %q, want the bare reason when there is no log to quote", message)
|
||||
}
|
||||
}
|
||||
|
||||
// A client refused during the handshake is closed by the server without the
|
||||
// server ever logging a cause, so the engine log alone leaves the failure
|
||||
// unexplained and only the client's own stderr names it.
|
||||
func TestMariaDBProbe_QuotesTheProbeClientWhenItIsRefused(t *testing.T) {
|
||||
cfg := mariadbTestProbeConfig(t)
|
||||
cfg.EngineErrorLog = writeEngineLog(t, "2026-08-12 4:32:39 148 [Warning] Aborted connection 148\n")
|
||||
writePid(t, cfg.EnginePidFile, os.Getpid())
|
||||
|
||||
state, message := mariadbProbeState(cfg,
|
||||
func(context.Context, string, ...string) (int, string, error) {
|
||||
return 1, "ERROR 2026 (HY000): TLS/SSL error: self-signed certificate\n", nil
|
||||
},
|
||||
func(int) bool { return true })(t.Context(), int64(cfg.EnginePort))
|
||||
|
||||
if state != engineRecovering {
|
||||
t.Errorf("state = %v, want recovering", state)
|
||||
}
|
||||
if !strings.Contains(message, "self-signed certificate") {
|
||||
t.Errorf("message = %q, want the probe client's own error quoted", message)
|
||||
}
|
||||
if !strings.Contains(message, "Aborted connection") {
|
||||
t.Errorf("message = %q, want the engine log kept alongside the client's error", message)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollapseProbeStderr(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in string
|
||||
want string
|
||||
}{
|
||||
{name: "folds to one line", in: "ERROR 2026:\n TLS error\n", want: "ERROR 2026: TLS error"},
|
||||
{name: "empty stays empty", in: " \n", want: ""},
|
||||
{
|
||||
name: "bounded",
|
||||
in: strings.Repeat("y", mariadbProbeStderrMaxBytes+50),
|
||||
want: strings.Repeat("y", mariadbProbeStderrMaxBytes) + "...",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := collapseProbeStderr(tt.in); got != tt.want {
|
||||
t.Errorf("collapseProbeStderr() = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A probe that says nothing must not append an empty clause: the reason reaches
|
||||
// a customer, and a dangling "reported:" would read as truncated output.
|
||||
func TestWithProbeStderrLeavesReasonAloneWhenTheClientIsSilent(t *testing.T) {
|
||||
const reason = "engine is not answering on its socket yet"
|
||||
if got := withProbeStderr(reason, " \n "); got != reason {
|
||||
t.Errorf("withProbeStderr() = %q, want the reason unchanged", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineLogTail(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
body string
|
||||
want string
|
||||
wantMissing string
|
||||
}{
|
||||
{
|
||||
name: "keeps only the last lines",
|
||||
body: "one\ntwo\nthree\nfour\nfive\nsix\n",
|
||||
want: "three | four | five | six",
|
||||
},
|
||||
{
|
||||
name: "skips blank lines",
|
||||
body: "alpha\n\n\nbeta\n\n",
|
||||
want: "alpha | beta",
|
||||
},
|
||||
{
|
||||
name: "empty file yields nothing",
|
||||
body: "",
|
||||
want: "",
|
||||
},
|
||||
{
|
||||
name: "drops a line the byte cap cut in half",
|
||||
body: strings.Repeat("x", engineErrorLogTailBytes) + "\nlast\n",
|
||||
want: "last",
|
||||
wantMissing: "x",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := engineLogTail(writeEngineLog(t, tt.body))
|
||||
if got != tt.want {
|
||||
t.Errorf("engineLogTail() = %q, want %q", got, tt.want)
|
||||
}
|
||||
if tt.wantMissing != "" && strings.Contains(got, tt.wantMissing) {
|
||||
t.Errorf("engineLogTail() = %q, want no partial line", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineLogTailUnsetPath(t *testing.T) {
|
||||
if got := engineLogTail(""); got != "" {
|
||||
t.Errorf("engineLogTail(\"\") = %q, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The agent reads a path the image writes, and nothing at runtime would notice
|
||||
// the two drifting apart: the log would simply always be missing and every
|
||||
// refusal would go back unexplained again.
|
||||
func TestMariaDBErrorLogPathMatchesTheImage(t *testing.T) {
|
||||
raw, err := os.ReadFile(mariadbRDSInitScript)
|
||||
if err != nil {
|
||||
t.Fatalf("read rds-init: %v", err)
|
||||
}
|
||||
script := string(raw)
|
||||
|
||||
logDir := shellDefault(t, script, "LOG_DIR", "RDS_LOG_DIR")
|
||||
engineLog := shellDefault(t, script, "ENGINE_LOG", "RDS_ENGINE_LOG")
|
||||
want := strings.Replace(engineLog, "${LOG_DIR}", logDir, 1)
|
||||
|
||||
if got := engineLayouts[engineMariaDB].errorLog; got != want {
|
||||
t.Errorf("layout errorLog = %q, but rds-init writes log_error = %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Pulls the fallback out of a `NAME="${OVERRIDE:-fallback}"` assignment.
|
||||
func shellDefault(t *testing.T, script, name, override string) string {
|
||||
t.Helper()
|
||||
re := regexp.MustCompile(`(?m)^` + name + `="\$\{` + override + `:-(.*)\}"$`)
|
||||
m := re.FindStringSubmatch(script)
|
||||
if m == nil {
|
||||
t.Fatalf("rds-init has no %s assignment defaulting from %s", name, override)
|
||||
}
|
||||
return m[1]
|
||||
}
|
||||
|
||||
func TestWithEngineLogTailLeavesReasonAloneWithoutALog(t *testing.T) {
|
||||
const reason = "engine is not answering on its socket yet"
|
||||
if got := withEngineLogTail(reason, ""); got != reason {
|
||||
t.Errorf("withEngineLogTail() = %q, want the reason unchanged", got)
|
||||
}
|
||||
}
|
||||
|
||||
const postgresConfD = "../../scripts/images/rds-postgres/postgresql.confd"
|
||||
|
||||
// "engine did not respond" is where a postmaster that refused a setting lands,
|
||||
// and it is the same message as an engine that never got as far as trying. Only
|
||||
// the server's own log separates them, and the guest has no other way out.
|
||||
func TestPostgresProbe_QuotesTheEngineLogWhenTheEngineIsAbsent(t *testing.T) {
|
||||
log := writeEngineLog(t, strings.Join([]string{
|
||||
`2026-09-06 10:20:33 UTC [712] LOG: starting PostgreSQL 18.0`,
|
||||
`2026-09-06 10:20:33 UTC [712] FATAL: could not map anonymous shared memory`,
|
||||
"",
|
||||
}, "\n"))
|
||||
|
||||
state, message := postgresProbeState("/run/postgresql", log,
|
||||
func(context.Context, string, ...string) (int, string, error) { return 2, "", nil })(t.Context(), 5432)
|
||||
|
||||
if state != engineAbsent {
|
||||
t.Errorf("state = %v, want absent", state)
|
||||
}
|
||||
if !strings.Contains(message, "could not map anonymous shared memory") {
|
||||
t.Errorf("message = %q, want the postmaster's own refusal quoted", message)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresProbe_SurvivesAnAbsentEngineLog(t *testing.T) {
|
||||
state, message := postgresProbeState("/run/postgresql", filepath.Join(t.TempDir(), "nope.log"),
|
||||
func(context.Context, string, ...string) (int, string, error) { return 2, "", nil })(t.Context(), 5432)
|
||||
|
||||
if state != engineAbsent {
|
||||
t.Errorf("state = %v, want absent", state)
|
||||
}
|
||||
if message != "engine did not respond on /run/postgresql:5432" {
|
||||
t.Errorf("message = %q, want the bare reason when there is no log to quote", message)
|
||||
}
|
||||
}
|
||||
|
||||
// The layout names a path the image has to be directing the postmaster to, or
|
||||
// the tail above quotes nothing on the one failure it exists for.
|
||||
func TestPostgresErrorLogPathMatchesTheImage(t *testing.T) {
|
||||
raw, err := os.ReadFile(postgresConfD)
|
||||
if err != nil {
|
||||
t.Fatalf("read postgresql.confd: %v", err)
|
||||
}
|
||||
m := regexp.MustCompile(`(?m)^logfile="(.*)"$`).FindStringSubmatch(string(raw))
|
||||
if m == nil {
|
||||
t.Fatal("postgresql.confd has no logfile assignment")
|
||||
}
|
||||
if got := engineLayouts[enginePostgres].errorLog; got != m[1] {
|
||||
t.Errorf("layout errorLog = %q, but the image sets logfile = %q", got, m[1])
|
||||
}
|
||||
}
|
||||
@@ -1,762 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// The control plane's own statements about this engine, resolved once at
|
||||
// startup. Held as functions rather than as the Engine value so nothing here
|
||||
// re-derives a rule the API already owns.
|
||||
type controlPlaneRules struct {
|
||||
// The whole master-username rule rather than only the reserved set. MariaDB's
|
||||
// client has no identifier quoting, so the name is interpolated into SQL by
|
||||
// this process and every part of the rule is load-bearing here.
|
||||
validateUsername func(username string) error
|
||||
// Whether a setting takes effect only at a restart. Already customer-facing
|
||||
// and authoritative, since the API refuses ApplyMethod=immediate on one.
|
||||
isStatic func(name string) bool
|
||||
// The catalog name behind a spelling read back out of an option file. Only a
|
||||
// setting whose startup spelling differs from the customer's moves.
|
||||
catalogName func(optionFileName string) string
|
||||
// The catalog's data type for a setting, which decides how its value is
|
||||
// rendered into SQL. Empty for a name the catalog does not carry.
|
||||
dataType func(name string) string
|
||||
// The engine's own name for the setting that requires TLS of a client
|
||||
// connection. Held rather than spelled out here, so the guest derives
|
||||
// enforcement from the same key the control plane resolves it under.
|
||||
tlsEnforcementParameter string
|
||||
}
|
||||
|
||||
func controlPlaneRulesFrom(meta handlers_rds.Engine) controlPlaneRules {
|
||||
// Built once per engine: everything read back out of a generated file has to
|
||||
// return to the catalog's namespace before it is classified or reported, or a
|
||||
// startup spelling would read as an unknown name.
|
||||
catalogNames := map[string]string{}
|
||||
for _, name := range meta.CatalogParameterNames() {
|
||||
if optionFileName := meta.OptionFileName(name); optionFileName != name {
|
||||
catalogNames[optionFileName] = name
|
||||
}
|
||||
}
|
||||
|
||||
return controlPlaneRules{
|
||||
validateUsername: meta.ValidateMasterUsername,
|
||||
isStatic: func(name string) bool {
|
||||
spec, ok := meta.LookupParameter(name)
|
||||
// A name the catalog does not carry cannot be shown to have been adopted
|
||||
// without a restart, and is never issued as a live SET GLOBAL either.
|
||||
return !ok || spec.ApplyType == handlers_rds.ApplyTypeStatic
|
||||
},
|
||||
catalogName: func(optionFileName string) string {
|
||||
if name, ok := catalogNames[optionFileName]; ok {
|
||||
return name
|
||||
}
|
||||
return optionFileName
|
||||
},
|
||||
dataType: func(name string) string {
|
||||
spec, ok := meta.LookupParameter(name)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
return spec.DataType
|
||||
},
|
||||
tlsEnforcementParameter: meta.TLSEnforcementParameter(),
|
||||
}
|
||||
}
|
||||
|
||||
// Reached over its unix socket as root, which the datadir's unix_socket plugin
|
||||
// authenticates from the connecting process' own uid. That is why the agent
|
||||
// holds no password of its own.
|
||||
type mariadbEngine struct {
|
||||
quiesceState
|
||||
parameterManager
|
||||
|
||||
rules controlPlaneRules
|
||||
run commandRunner
|
||||
startSess sessionRunner
|
||||
client string
|
||||
socket string
|
||||
rcService string
|
||||
service string
|
||||
}
|
||||
|
||||
var _ engine = (*mariadbEngine)(nil)
|
||||
|
||||
const (
|
||||
mariadbClientBinary = "mariadb"
|
||||
mariadbAdminBinary = "mariadb-admin"
|
||||
// The account mariadb-install-db creates for the unix_socket plugin, which
|
||||
// rds-init keeps for the platform and never hands to the customer.
|
||||
mariadbSuperuser = "root"
|
||||
mariadbProbeConnectTimeoutSeconds = 3
|
||||
// How much of the probe client's stderr a reason carries, on the same grounds
|
||||
// as the log tail above.
|
||||
mariadbProbeStderrMaxBytes = 512
|
||||
// Named by the platform drop-in rds-init writes, so both halves reach the
|
||||
// same socket without either asserting it to the other.
|
||||
mariadbSocketFile = "mysqld.sock"
|
||||
// The customer reaches the instance over its customer ENI, so the master is
|
||||
// created and rotated on the wildcard host rather than on localhost.
|
||||
mariadbMasterHost = "%"
|
||||
)
|
||||
|
||||
// The generated drop-ins on the data volume. The rollback and serving copies
|
||||
// deliberately do not end in .cnf: !includedir reads *.cnf, and neither is a
|
||||
// second set of settings.
|
||||
const (
|
||||
mariadbParametersFile = "10-rds-parameters.cnf"
|
||||
mariadbLastGoodFile = "10-rds-parameters.last-good"
|
||||
mariadbServingFile = "10-rds-parameters.serving"
|
||||
// MariaDB treats a setting before any group as a fatal parsing error, so
|
||||
// every generated file carries the header rds-init writes.
|
||||
mariadbParametersHeader = "[mysqld]\n"
|
||||
)
|
||||
|
||||
// The layout's factory. The rules come from the control plane's own definition
|
||||
// of this engine, so a build whose control plane does not offer MariaDB refuses
|
||||
// to run this implementation rather than inventing a definition for it.
|
||||
func newMariaDBEngineFromCatalog(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error) {
|
||||
meta, err := handlers_rds.LookupEngine(engineMariaDB)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("this image bakes %s, which this build's control plane does not offer: %w", engineMariaDB, err)
|
||||
}
|
||||
return newMariaDBEngine(cfg, controlPlaneRulesFrom(meta), run, startSess, probe), nil
|
||||
}
|
||||
|
||||
func newMariaDBEngine(cfg config, rules controlPlaneRules, run commandRunner, startSess sessionRunner, probe *engineProbe) *mariadbEngine {
|
||||
return &mariadbEngine{
|
||||
rules: rules,
|
||||
run: run,
|
||||
startSess: startSess,
|
||||
client: filepath.Join(cfg.EngineBinDir, mariadbClientBinary),
|
||||
socket: mariadbSocketPath(cfg),
|
||||
rcService: cfg.RCService,
|
||||
service: cfg.EngineService,
|
||||
parameterManager: parameterManager{
|
||||
probe: probe,
|
||||
params: parameterStore{
|
||||
// The mount point rather than the datadir one level inside it: the
|
||||
// include directory has to outlive the sweep a failed bootstrap runs.
|
||||
dir: filepath.Join(cfg.DataMount, "conf.d"),
|
||||
installed: mariadbParametersFile,
|
||||
lastGood: mariadbLastGoodFile,
|
||||
serving: mariadbServingFile,
|
||||
header: mariadbParametersHeader,
|
||||
osUser: cfg.EngineUser,
|
||||
engine: engineMariaDB,
|
||||
},
|
||||
repairTimeout: parameterRepairTimeout,
|
||||
repairPoll: parameterRepairPoll,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func mariadbSocketPath(cfg config) string {
|
||||
return filepath.Join(cfg.SocketDir, mariadbSocketFile)
|
||||
}
|
||||
|
||||
// Every local connection, from the probe's ping to a held quiesce session.
|
||||
// --no-defaults so no option file can move it; --skip-ssl because the serving
|
||||
// cert names the endpoint rather than this socket, so a verifying client fails.
|
||||
func mariadbSocketConnectArgs(socket string) []string {
|
||||
return []string{
|
||||
"--no-defaults", "--protocol=socket", "--socket=" + socket,
|
||||
"--user=" + mariadbSuperuser, "--skip-ssl",
|
||||
}
|
||||
}
|
||||
|
||||
// Reports whether a pid is a live process. EPERM is a process this agent may not
|
||||
// signal rather than one that is gone, which matters because a stale pidfile
|
||||
// read as "alive" is far less dangerous than a live engine read as "absent".
|
||||
type processLivenessFn func(pid int) bool
|
||||
|
||||
func processAlive(pid int) bool {
|
||||
if pid <= 0 {
|
||||
return false
|
||||
}
|
||||
err := syscall.Kill(pid, 0)
|
||||
return err == nil || errors.Is(err, syscall.EPERM)
|
||||
}
|
||||
|
||||
func newMariaDBProbe(cfg config, run probeRunner) *engineProbe {
|
||||
return newEngineProbe(cfg.EnginePort, mariadbProbeState(cfg, run, processAlive))
|
||||
}
|
||||
|
||||
// Three stages: during InnoDB crash recovery mariadbd opens neither socket nor
|
||||
// port, so a ping cannot tell a recovering engine from an absent one — and
|
||||
// reading recovery as absent has the rollback guard restart a server mid-replay.
|
||||
func mariadbProbeState(cfg config, run probeRunner, alive processLivenessFn) probeStateFn {
|
||||
pidFile, socket := cfg.EnginePidFile, mariadbSocketPath(cfg)
|
||||
admin, client := filepath.Join(cfg.EngineBinDir, mariadbAdminBinary), filepath.Join(cfg.EngineBinDir, mariadbClientBinary)
|
||||
// The probe alone bounds the connect: it must come back and report, where a
|
||||
// statement the agent issues is allowed to take as long as the engine needs.
|
||||
connect := append(mariadbSocketConnectArgs(socket),
|
||||
fmt.Sprintf("--connect-timeout=%d", mariadbProbeConnectTimeoutSeconds))
|
||||
probeTimeout := cfg.EngineProbeTimeout
|
||||
if probeTimeout <= 0 {
|
||||
probeTimeout = defaultEngineProbeTimeout
|
||||
}
|
||||
runBounded := func(ctx context.Context, name string, args ...string) (int, string, error) {
|
||||
probeCtx, cancel := context.WithTimeout(ctx, probeTimeout)
|
||||
defer cancel()
|
||||
return run(probeCtx, name, args...)
|
||||
}
|
||||
|
||||
return func(ctx context.Context, _ int64) (engineState, string) {
|
||||
pid, err := readPidFile(pidFile)
|
||||
switch {
|
||||
case os.IsNotExist(err):
|
||||
return engineAbsent, fmt.Sprintf("the engine has written no pidfile at %s", pidFile)
|
||||
case err != nil:
|
||||
return engineAbsent, fmt.Sprintf("the engine pidfile %s could not be read: %v", pidFile, err)
|
||||
case !alive(pid):
|
||||
return engineAbsent, fmt.Sprintf("the engine pidfile %s names pid %d, which is not running", pidFile, pid)
|
||||
}
|
||||
|
||||
// The process is up from here on, so every remaining failure is an engine
|
||||
// not serving yet rather than one that is gone. Reporting absent against a
|
||||
// live server would have the rollback guard restart one making progress.
|
||||
switch code, stderr, err := runBounded(ctx, admin, append(slices.Clone(connect), "ping")...); {
|
||||
case err != nil:
|
||||
return engineRecovering, fmt.Sprintf("engine probe could not run: %v", err)
|
||||
case code != 0:
|
||||
// A server that is genuinely recovering and one that refused the client
|
||||
// are indistinguishable from the socket. The server states its side in
|
||||
// its own log, and the client states its side on stderr.
|
||||
return engineRecovering, withProbeStderr(withEngineLogTail(
|
||||
"engine is not answering on its socket yet (startup or crash recovery)",
|
||||
cfg.EngineErrorLog), stderr)
|
||||
}
|
||||
|
||||
// ping answers even on ER_ACCESS_DENIED, so on its own it certifies a server
|
||||
// that may be unable to execute anything. The statement rides argv rather
|
||||
// than stdin: it carries nothing secret and a probe reads no result back.
|
||||
query := append(slices.Clone(connect), "--batch", "--skip-column-names", "--execute=SELECT 1")
|
||||
switch code, stderr, err := runBounded(ctx, client, query...); {
|
||||
case err != nil:
|
||||
return engineRecovering, fmt.Sprintf("engine probe could not run: %v", err)
|
||||
case code != 0:
|
||||
return engineRecovering, withProbeStderr(
|
||||
"engine answered its socket but would not execute a statement", stderr)
|
||||
}
|
||||
return engineServing, ""
|
||||
}
|
||||
}
|
||||
|
||||
// Appends what the probe client said on stderr. A client that fails during the
|
||||
// handshake is refused by the server without the server ever logging why, so
|
||||
// this is the only account of that half of the exchange.
|
||||
func withProbeStderr(reason, stderr string) string {
|
||||
stderr = strings.TrimSpace(stderr)
|
||||
if stderr == "" {
|
||||
return reason
|
||||
}
|
||||
return reason + "; the probe client reported: " + collapseProbeStderr(stderr)
|
||||
}
|
||||
|
||||
// One line, bounded: the reason reaches a customer through StatusInfos, and a
|
||||
// client that repeats itself per attempt would otherwise crowd out the rest.
|
||||
func collapseProbeStderr(stderr string) string {
|
||||
line := strings.Join(strings.Fields(strings.ReplaceAll(stderr, "\n", " ")), " ")
|
||||
if len(line) > mariadbProbeStderrMaxBytes {
|
||||
return line[:mariadbProbeStderrMaxBytes] + "..."
|
||||
}
|
||||
return line
|
||||
}
|
||||
|
||||
func readPidFile(path string) (int, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(string(raw)))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%s does not hold a pid: %w", path, err)
|
||||
}
|
||||
return pid, nil
|
||||
}
|
||||
|
||||
func (e *mariadbEngine) Stop(ctx context.Context) error {
|
||||
return serviceAction(ctx, e.run, e.rcService, e.service, "stop")
|
||||
}
|
||||
|
||||
// Only the parameter rollback and a failed apply call this: a restart the
|
||||
// control plane wants goes through RebootDBInstance, which cycles the VM. The
|
||||
// serving copy is refreshed first, since the server starts on what is installed.
|
||||
func (e *mariadbEngine) Restart(ctx context.Context) error {
|
||||
if err := e.recordServingCopy(); err != nil {
|
||||
return err
|
||||
}
|
||||
return serviceAction(ctx, e.run, e.rcService, e.service, "restart")
|
||||
}
|
||||
|
||||
// The installed set copied to the name the pending-restart comparison reads. A
|
||||
// live SET GLOBAL apply deliberately does not touch it: that value was adopted
|
||||
// without a restart, which is what "not pending" means.
|
||||
func (e *mariadbEngine) recordServingCopy() error {
|
||||
content, err := os.ReadFile(e.params.installedPath())
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("read the installed parameters: %w", err)
|
||||
}
|
||||
return e.params.write(e.params.servingPath(), content)
|
||||
}
|
||||
|
||||
// A value safe to interpolate into a single-quoted SQL literal. The quote is
|
||||
// doubled, which holds in every sql_mode, and the backslash is doubled, which is
|
||||
// why every statement built here pins a mode with backslash escapes on.
|
||||
func sqlLiteral(value string) string {
|
||||
return strings.ReplaceAll(strings.ReplaceAll(value, `\`, `\\`), `'`, `''`)
|
||||
}
|
||||
|
||||
// The spellings MariaDB parses for a boolean system variable, and the only ones
|
||||
// the catalog's own enum narrows a boolean to.
|
||||
var mariadbBooleanValues = map[string]string{
|
||||
"on": "ON", "off": "OFF", "true": "ON", "false": "OFF", "1": "ON", "0": "OFF",
|
||||
}
|
||||
|
||||
// One SET GLOBAL right-hand side. MariaDB refuses a quoted literal for a numeric
|
||||
// or boolean with ER_WRONG_TYPE_FOR_VAR, so the catalog's data type picks the
|
||||
// form; re-rendering from the parsed value is what keeps it safe to interpolate.
|
||||
func mariadbSetValue(dataType, value string) (string, error) {
|
||||
switch dataType {
|
||||
case handlers_rds.ParamTypeInteger:
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(value), 10, 64)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("value %q is not an integer", value)
|
||||
}
|
||||
return strconv.FormatInt(n, 10), nil
|
||||
case handlers_rds.ParamTypeReal:
|
||||
f, err := strconv.ParseFloat(strings.TrimSpace(value), 64)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("value %q is not a real number", value)
|
||||
}
|
||||
return strconv.FormatFloat(f, 'g', -1, 64), nil
|
||||
case handlers_rds.ParamTypeBoolean:
|
||||
if literal, ok := mariadbBooleanValues[strings.ToLower(strings.TrimSpace(value))]; ok {
|
||||
return literal, nil
|
||||
}
|
||||
return "", fmt.Errorf("value %q is not a boolean", value)
|
||||
default:
|
||||
return "'" + sqlLiteral(value) + "'", nil
|
||||
}
|
||||
}
|
||||
|
||||
// Turns off the two logs that would otherwise copy a statement's text, and pins
|
||||
// the mode the escaping above assumes. All three are session-scoped: a rotation
|
||||
// must not leave the customer's own general log switched off behind it.
|
||||
const mariadbSessionGuard = `SET SESSION sql_log_off = 1;
|
||||
SET SESSION slow_query_log = 0;
|
||||
SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';
|
||||
`
|
||||
|
||||
// Rotates the master's password. The statement is built here rather than by the
|
||||
// client, which has neither psql's identifier interpolation nor its parameter
|
||||
// quoting, and it rides stdin so it is never visible in the process table.
|
||||
func (e *mariadbEngine) SetPassword(ctx context.Context, username, password string) error {
|
||||
if username == "" || password == "" {
|
||||
return fmt.Errorf("set-password requires both %s and %s",
|
||||
handlers_rds.CommandParamMasterUsername, handlers_rds.CommandParamMasterUserPassword)
|
||||
}
|
||||
// This runs as the server's own superuser, so a reserved name in the payload
|
||||
// would rotate the platform's account. The whole rule matters here, not just
|
||||
// the reserved set: the name is interpolated below rather than client-quoted.
|
||||
if err := e.rules.validateUsername(username); err != nil {
|
||||
return fmt.Errorf("refusing to set the password of role %q: %w", username, err)
|
||||
}
|
||||
|
||||
escaped := sqlLiteral(password)
|
||||
sql := fmt.Sprintf("%sALTER USER '%s'@'%s' IDENTIFIED BY '%s';\n",
|
||||
mariadbSessionGuard, username, mariadbMasterHost, escaped)
|
||||
if _, err := e.clientRun(ctx, sql); err != nil {
|
||||
// The client echoes the failing statement, which carries the escaped form
|
||||
// of the password rather than the raw one — so redacting only the raw one
|
||||
// would leak exactly the passwords that needed escaping.
|
||||
return fmt.Errorf("apply the master password: %s", redact(redact(err.Error(), password), escaped))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Installs the resolved set and applies the half a running server can adopt.
|
||||
// MariaDB re-reads no file while up, so SET GLOBAL is the only immediate path.
|
||||
// It has no offline parser, so the safety net is the boot-time rollback.
|
||||
func (e *mariadbEngine) ApplyParameters(ctx context.Context, params []handlers_rds.Parameter) ([]string, error) {
|
||||
e.paramMu.Lock()
|
||||
defer e.paramMu.Unlock()
|
||||
|
||||
initialState, _ := e.probe.state(ctx)
|
||||
if initialState == engineRecovering {
|
||||
return nil, errors.New("apply parameters while the engine is still starting or recovering")
|
||||
}
|
||||
if initialState == engineServing {
|
||||
// A command can arrive before the first heartbeat. Seed the configuration
|
||||
// currently being served before replacing its file in that window.
|
||||
if err := e.recordServingParametersLocked(ctx); err != nil {
|
||||
return nil, fmt.Errorf("record the parameters serving before the apply: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
restore, err := e.params.install(params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
state, _ := e.probe.state(ctx)
|
||||
switch state {
|
||||
case engineAbsent:
|
||||
return e.restartOnRepairSetLocked(ctx)
|
||||
case engineRecovering:
|
||||
return nil, errors.Join(errors.New("the engine entered startup or recovery during the parameter apply"), restore())
|
||||
}
|
||||
|
||||
// Before the set reaches the server: enforcement is one of the values below,
|
||||
// and a server told to require a transport it cannot offer would refuse every
|
||||
// client the statement after.
|
||||
if err := e.requireServingTLSForEnforcement(ctx); err != nil {
|
||||
return nil, errors.Join(err, restore())
|
||||
}
|
||||
|
||||
if err := e.applyDynamicParameters(ctx, params); err != nil {
|
||||
// The engine may have gone down after the first probe. In that case start it
|
||||
// on the newly installed set rather than restoring one that already left it
|
||||
// unable to serve.
|
||||
if state, _ := e.probe.state(ctx); state == engineAbsent {
|
||||
return e.restartOnRepairSetLocked(ctx)
|
||||
}
|
||||
return nil, errors.Join(err, restore())
|
||||
}
|
||||
// Last, and on its own: the batch above is not a transaction, so moving the
|
||||
// TLS posture inside it would leave enforcement changed on a set the server
|
||||
// went on to refuse, with only the parameter file rolled back.
|
||||
if err := e.applyTLSEnforcement(ctx); err != nil {
|
||||
if state, _ := e.probe.state(ctx); state == engineAbsent {
|
||||
return e.restartOnRepairSetLocked(ctx)
|
||||
}
|
||||
return nil, errors.Join(err, restore())
|
||||
}
|
||||
return e.pendingRestartParameters(ctx)
|
||||
}
|
||||
|
||||
// The dynamic half, in one invocation that stops at the first refusal so the
|
||||
// server names the setting it would not take. Only catalog names are emitted,
|
||||
// which is what makes the identifier safe to build into the statement.
|
||||
//
|
||||
// TLS enforcement is excluded: applyTLSEnforcement issues it after this batch
|
||||
// has succeeded.
|
||||
func (e *mariadbEngine) applyDynamicParameters(ctx context.Context, params []handlers_rds.Parameter) error {
|
||||
var b strings.Builder
|
||||
b.WriteString("SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';\n")
|
||||
applied := 0
|
||||
for _, p := range params {
|
||||
if p.Name == "" || p.Name == e.rules.tlsEnforcementParameter || e.rules.isStatic(p.Name) {
|
||||
continue
|
||||
}
|
||||
value, err := mariadbSetValue(e.rules.dataType(p.Name), p.Value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("apply the dynamic parameters: %s: %w", p.Name, err)
|
||||
}
|
||||
fmt.Fprintf(&b, "SET GLOBAL %s = %s;\n", p.Name, value)
|
||||
applied++
|
||||
}
|
||||
if applied == 0 {
|
||||
return nil
|
||||
}
|
||||
if _, err := e.clientRun(ctx, b.String()); err != nil {
|
||||
return fmt.Errorf("apply the dynamic parameters: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Puts the installed set's enforcement on the running server. The value comes
|
||||
// from the installed file rather than the set in hand, so a set that predates
|
||||
// the parameter enforces here rather than leaving the server on mariadbd's own
|
||||
// default of off while the API reports enforcement.
|
||||
func (e *mariadbEngine) applyTLSEnforcement(ctx context.Context) error {
|
||||
enforce, err := installedTLSEnforcement(e.rules.tlsEnforcementParameter, e.params.installedPath())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
previous, err := e.readTLSEnforcement(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if previous == enforce {
|
||||
return nil
|
||||
}
|
||||
applyErr := e.setTLSEnforcement(ctx, enforce)
|
||||
if applyErr == nil {
|
||||
return nil
|
||||
}
|
||||
// The client can fail without saying whether the server ran the statement, so
|
||||
// the posture it left behind is read rather than assumed unchanged: the one
|
||||
// outcome that must not survive a failed apply is plaintext nobody asked for.
|
||||
live, readErr := e.readTLSEnforcement(ctx)
|
||||
if readErr != nil {
|
||||
return errors.Join(applyErr, readErr)
|
||||
}
|
||||
if live == previous {
|
||||
return applyErr
|
||||
}
|
||||
return errors.Join(applyErr, e.setTLSEnforcement(ctx, previous))
|
||||
}
|
||||
|
||||
func (e *mariadbEngine) readTLSEnforcement(ctx context.Context) (bool, error) {
|
||||
name := e.rules.tlsEnforcementParameter
|
||||
out, err := e.clientRun(ctx, "SELECT @@global."+name+";\n")
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("read whether the engine requires TLS of clients: %w", err)
|
||||
}
|
||||
switch value := strings.TrimSpace(out); value {
|
||||
case "1":
|
||||
return true, nil
|
||||
case "0":
|
||||
return false, nil
|
||||
default:
|
||||
return false, fmt.Errorf("the engine reports %s as %q, which is neither 1 nor 0", name, value)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *mariadbEngine) setTLSEnforcement(ctx context.Context, enforce bool) error {
|
||||
name, value := e.rules.tlsEnforcementParameter, "0"
|
||||
if enforce {
|
||||
value = "1"
|
||||
}
|
||||
// Rendered the way the batch renders every other boolean, so the statement the
|
||||
// server sees does not depend on which path issued it.
|
||||
literal, err := mariadbSetValue(handlers_rds.ParamTypeBoolean, value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("render %s: %w", name, err)
|
||||
}
|
||||
sql := fmt.Sprintf("SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';\nSET GLOBAL %s = %s;\n", name, literal)
|
||||
if _, err := e.clientRun(ctx, sql); err != nil {
|
||||
return fmt.Errorf("set %s to %s: %w", name, literal, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Refuses to require TLS of clients from a server that is not serving it, which
|
||||
// would reject every TCP connection while the parameter path reported success.
|
||||
// Read back from the installed file rather than taken from the set in hand, so
|
||||
// what is checked is also what the server reads at its next start.
|
||||
func (e *mariadbEngine) requireServingTLSForEnforcement(ctx context.Context) error {
|
||||
enforce, err := installedTLSEnforcement(e.rules.tlsEnforcementParameter, e.params.installedPath())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !enforce {
|
||||
return nil
|
||||
}
|
||||
// DISABLED is a server built with TLS and started without a certificate,
|
||||
// which is what a boot that was handed none leaves behind. The agent's own
|
||||
// connections are unaffected either way: MariaDB counts a unix socket as a
|
||||
// secure transport.
|
||||
out, err := e.clientRun(ctx, "SELECT @@global.have_ssl;\n")
|
||||
if err != nil {
|
||||
return fmt.Errorf("read whether the engine is serving TLS: %w", err)
|
||||
}
|
||||
if serving := strings.TrimSpace(out); serving != "YES" {
|
||||
return fmt.Errorf("the engine is serving without TLS (have_ssl = %q), so requiring it of clients would reject every connection", serving)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *mariadbEngine) restartOnRepairSetLocked(ctx context.Context) ([]string, error) {
|
||||
return awaitRepairedEngine(ctx, e.probe, e.Restart, e.pendingRestartParameters, e.repairTimeout, e.repairPoll)
|
||||
}
|
||||
|
||||
// MariaDB has no pending_restart, so this is computed from the files: the
|
||||
// catalog-static keys whose installed value differs from the one the server
|
||||
// started on. Live values are not compared — mysqld silently rewrites them.
|
||||
func (e *mariadbEngine) pendingRestartParameters(context.Context) ([]string, error) {
|
||||
installed, err := readOptionFile(e.params.installedPath())
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
// Nothing is installed, so nothing is waiting to be adopted.
|
||||
return nil, nil
|
||||
}
|
||||
return nil, fmt.Errorf("read the installed parameters: %w", err)
|
||||
}
|
||||
serving, err := readOptionFile(e.params.servingPath())
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("read the parameters the engine started on: %w", err)
|
||||
}
|
||||
// Back to the catalog's names before anything is classified or reported: the
|
||||
// files are written in the engine's startup spellings, and the answer is a
|
||||
// customer-facing list the API refuses ApplyMethod=immediate against.
|
||||
installed = e.catalogKeyed(installed)
|
||||
serving = e.catalogKeyed(serving)
|
||||
|
||||
// An absent serving copy is a set the engine has not started on: rds-init
|
||||
// writes the two together on every boot, so the whole static half counts as
|
||||
// pending rather than being promoted on the strength of a missing file.
|
||||
var pending []string
|
||||
for name, value := range installed {
|
||||
if !e.rules.isStatic(name) {
|
||||
continue
|
||||
}
|
||||
if served, ok := serving[name]; !ok || served != value {
|
||||
pending = append(pending, name)
|
||||
}
|
||||
}
|
||||
// A static setting the group stopped naming reverts to its default at the next
|
||||
// start, which the running server has not adopted either.
|
||||
for name := range serving {
|
||||
if _, ok := installed[name]; !ok && e.rules.isStatic(name) {
|
||||
pending = append(pending, name)
|
||||
}
|
||||
}
|
||||
slices.Sort(pending)
|
||||
return pending, nil
|
||||
}
|
||||
|
||||
// Snapshots the installed set as the rollback target, but only when nothing in
|
||||
// it is still waiting for a restart. The parameter mutex keeps an apply from
|
||||
// replacing the file between that check and the copy.
|
||||
func (e *mariadbEngine) RecordServingParameters(ctx context.Context) error {
|
||||
e.paramMu.Lock()
|
||||
defer e.paramMu.Unlock()
|
||||
return e.recordServingParametersLocked(ctx)
|
||||
}
|
||||
|
||||
func (e *mariadbEngine) recordServingParametersLocked(ctx context.Context) error {
|
||||
return recordLastGood(ctx, e.params, e.pendingRestartParameters)
|
||||
}
|
||||
|
||||
// Puts the last set the engine accepted back in place, for a start that failed
|
||||
// after a parameter change.
|
||||
func (e *mariadbEngine) RestoreLastKnownGoodParameters(ctx context.Context) (bool, error) {
|
||||
e.paramMu.Lock()
|
||||
defer e.paramMu.Unlock()
|
||||
return restoreLastGood(ctx, e.params, e.probe)
|
||||
}
|
||||
|
||||
// The same settings under the names the catalog and the API know them by, since
|
||||
// a file is written in the engine's startup spellings and those are not always
|
||||
// the customer's.
|
||||
func (e *mariadbEngine) catalogKeyed(values map[string]string) map[string]string {
|
||||
out := make(map[string]string, len(values))
|
||||
for name, value := range values {
|
||||
out[e.rules.catalogName(name)] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The subset of MariaDB's option-file syntax the generated drop-ins are written
|
||||
// in: a group header, comments, and one `name = value` per line. Nothing else
|
||||
// reaches these two files, because only rds-init and this agent write them.
|
||||
func readOptionFile(path string) (map[string]string, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
values := make(map[string]string)
|
||||
for line := range strings.SplitSeq(string(raw), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, ";") || strings.HasPrefix(line, "[") {
|
||||
continue
|
||||
}
|
||||
// A bare option such as skip-log-bin carries no value; it is still a setting
|
||||
// whose presence or absence differs.
|
||||
name, value, _ := strings.Cut(line, "=")
|
||||
values[normaliseOptionName(name)] = strings.Trim(strings.TrimSpace(value), `"'`)
|
||||
}
|
||||
return values, nil
|
||||
}
|
||||
|
||||
// MariaDB reads - and _ as the same character in an option name, so the two
|
||||
// spellings of one setting must not compare as two settings.
|
||||
func normaliseOptionName(name string) string {
|
||||
return strings.ReplaceAll(strings.TrimSpace(name), "-", "_")
|
||||
}
|
||||
|
||||
// How long any one BACKUP STAGE waits for its metadata locks. Well under the
|
||||
// control plane's quiesce timeout, so a stage that cannot take its lock is
|
||||
// reported rather than left queued in front of live traffic.
|
||||
const mariadbQuiesceLockWait = 20 * time.Second
|
||||
|
||||
// MariaDB's own backup API rather than FLUSH TABLES WITH READ LOCK: FTWRL makes
|
||||
// the whole database read-only for the full hold, and its acquisition phase is
|
||||
// unbounded. BACKUP STAGE blocks commits instead, and covers Aria and MyISAM.
|
||||
func (e *mariadbEngine) Quiesce(ctx context.Context, label string, hold time.Duration) error {
|
||||
if err := validateQuiesceRequest(label, hold); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Held across the whole start, so a second quiesce waits and then finds the
|
||||
// first one's hold rather than opening a concurrent backup alongside it.
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
if e.held != nil {
|
||||
return fmt.Errorf("the engine is already quiesced for backup %s", e.held.label)
|
||||
}
|
||||
|
||||
// Deliberately not the command's context: the session has to outlive the call
|
||||
// that started it, and its own deadline is what bounds it instead.
|
||||
session, err := e.startSess(context.WithoutCancel(ctx), command{
|
||||
Name: e.client,
|
||||
Args: e.clientArgs(),
|
||||
Env: []string{"PATH=" + defaultGuestPath},
|
||||
SentinelStatement: "SELECT '" + sessionSentinel + "';\n",
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("open a backup session: %w", err)
|
||||
}
|
||||
|
||||
// In order and on one connection: MariaDB releases the hold with the session
|
||||
// that took it, which bounds a control plane that dies mid-snapshot. Fed one
|
||||
// at a time so a stage that cannot take its locks is named.
|
||||
stages := []string{
|
||||
fmt.Sprintf("SET SESSION lock_wait_timeout = %d;\n", int(mariadbQuiesceLockWait.Seconds())),
|
||||
"BACKUP STAGE START;\n",
|
||||
"BACKUP STAGE FLUSH;\n",
|
||||
"BACKUP STAGE BLOCK_DDL;\n",
|
||||
"BACKUP STAGE BLOCK_COMMIT;\n",
|
||||
}
|
||||
for _, sql := range stages {
|
||||
if err := session.Exec(ctx, sql); err != nil {
|
||||
if closeErr := session.Close(); closeErr != nil {
|
||||
slog.Warn("rds-agent: closing a failed backup session", "err", closeErr)
|
||||
}
|
||||
return fmt.Errorf("put the engine into backup mode at %q: %w", strings.TrimSpace(sql), err)
|
||||
}
|
||||
}
|
||||
|
||||
e.beginHoldLocked(label, session, hold)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *mariadbEngine) Unquiesce(ctx context.Context) error {
|
||||
return e.releaseHold(ctx, "BACKUP STAGE END;\n")
|
||||
}
|
||||
|
||||
// Feeds sql on stdin rather than as an argument, so a statement is never visible
|
||||
// in the process table.
|
||||
func (e *mariadbEngine) clientRun(ctx context.Context, sql string) (string, error) {
|
||||
return e.run(ctx, command{
|
||||
Name: e.client,
|
||||
Args: e.clientArgs(),
|
||||
Env: []string{"PATH=" + defaultGuestPath},
|
||||
Stdin: sql,
|
||||
})
|
||||
}
|
||||
|
||||
// --unbuffered so a held session's output reaches the reader as each statement
|
||||
// finishes rather than when a pipe buffer fills. No User on the command: the
|
||||
// connection args carry it, and the agent runs as the account they name.
|
||||
func (e *mariadbEngine) clientArgs() []string {
|
||||
return append(mariadbSocketConnectArgs(e.socket),
|
||||
"--batch", "--skip-column-names", "--unbuffered")
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,553 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// The engine is reached over its unix socket under peer authentication, so the
|
||||
// agent drops to the postgres OS user rather than holding a password of its own.
|
||||
type postgresEngine struct {
|
||||
quiesceState
|
||||
parameterManager
|
||||
|
||||
// The control plane's own metadata for this engine, resolved once at
|
||||
// startup: the live password apply runs as the cluster superuser, so it
|
||||
// re-checks the role name against the same reserved set the API validates.
|
||||
meta handlers_rds.Engine
|
||||
run commandRunner
|
||||
startSess sessionRunner
|
||||
psql string
|
||||
rcService string
|
||||
service string
|
||||
pgData string
|
||||
socketDir string
|
||||
osUser string
|
||||
}
|
||||
|
||||
var _ engine = (*postgresEngine)(nil)
|
||||
|
||||
// pg_isready gives PostgreSQL three states for free: exit 0 is serving, exit 1
|
||||
// is a postmaster up and rejecting connections, anything else is an engine that
|
||||
// is not there. Resolved on PATH, where the client package puts it.
|
||||
const postgresProbeBinary = "pg_isready"
|
||||
|
||||
// The socket directory rather than loopback: the engine binds the customer ENI
|
||||
// and nothing else, so there is no TCP path for the probe to take. It also keeps
|
||||
// liveness independent of the network path, of the pg_hba scope and of the TLS
|
||||
// enforcement rule — libpq reads a host beginning with / as a socket directory,
|
||||
// and the generated `local ... peer` line covers it.
|
||||
func newPostgresProbe(cfg config, run probeRunner) *engineProbe {
|
||||
return newEngineProbe(cfg.EnginePort, postgresProbeState(cfg.SocketDir, cfg.EngineErrorLog, run))
|
||||
}
|
||||
|
||||
func postgresProbeState(host, errorLog string, run probeRunner) probeStateFn {
|
||||
return func(ctx context.Context, port int64) (engineState, string) {
|
||||
portArg := strconv.FormatInt(port, 10)
|
||||
code, _, err := run(ctx, postgresProbeBinary, "-h", host, "-p", portArg, "-q")
|
||||
switch {
|
||||
case err != nil:
|
||||
// A missing binary or broken image. Reporting healthy on the strength of
|
||||
// nothing would hide it, so this reads as absent like an engine that did
|
||||
// not answer.
|
||||
return engineAbsent, fmt.Sprintf("engine probe could not run: %v", err)
|
||||
case code == 0:
|
||||
return engineServing, ""
|
||||
case code == 1:
|
||||
return engineRecovering, withEngineLogTail(
|
||||
"engine is rejecting connections (startup or recovery)", errorLog)
|
||||
default:
|
||||
// A postmaster that refused a setting never opens the socket, so this is
|
||||
// the state a bad parameter lands in. The reason for it is only in the
|
||||
// server's own log, and it reaches the customer through StatusInfos.
|
||||
return engineAbsent, withEngineLogTail(
|
||||
fmt.Sprintf("engine did not respond on %s:%s", host, portArg), errorLog)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The include the resolved set is rendered to, and the copy of the last one the
|
||||
// engine accepted. Both live beside the data rather than in /etc, matching
|
||||
// rds-init: a class change boots a fresh root volume and would revert them.
|
||||
const (
|
||||
postgresParametersFile = "10-rds-parameters.conf"
|
||||
// Deliberately not a .conf name: include_dir globs *.conf, so the rollback
|
||||
// copy must not be read as a second set of settings.
|
||||
postgresLastGoodFile = "10-rds-parameters.last-good"
|
||||
)
|
||||
|
||||
// The pg_hba rule that implements TLS enforcement, and the directory the
|
||||
// generated pg_hba includes it from. Its content is a constant, so the copy
|
||||
// rds-init writes in shell and the one written here have no template to drift
|
||||
// apart on: enforcement is the presence of the file and nothing more.
|
||||
const (
|
||||
postgresHBADir = "hba.d"
|
||||
postgresForceSSLRuleFile = "20-rds-force-ssl.conf"
|
||||
// An explicit reject rather than an omitted permit: both refuse the
|
||||
// connection, and only this one puts the reason in the server log and in the
|
||||
// client's error.
|
||||
postgresForceSSLRules = `# Managed by the platform: the rule that requires TLS of client connections.
|
||||
# Written by rds-init at boot and by rds-agent on a parameter apply.
|
||||
hostnossl all all 0.0.0.0/0 reject
|
||||
hostnossl all all ::/0 reject
|
||||
`
|
||||
// What the read-back must count once the file is in place. Asserting the shape
|
||||
// rather than only the absence of parse errors is what catches the failure
|
||||
// that will actually happen: the derivation above writing the wrong thing.
|
||||
postgresForceSSLRuleCount = 2
|
||||
)
|
||||
|
||||
// The layout's factory resolves the control plane metadata during startup.
|
||||
func newPostgresEngineFromCatalog(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error) {
|
||||
meta, err := handlers_rds.LookupEngine(enginePostgres)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("this image bakes %s, which this build's control plane does not offer: %w", enginePostgres, err)
|
||||
}
|
||||
return newPostgresEngine(cfg, meta, run, startSess, probe), nil
|
||||
}
|
||||
|
||||
func newPostgresEngine(cfg config, meta handlers_rds.Engine, run commandRunner, startSess sessionRunner, probe *engineProbe) *postgresEngine {
|
||||
return &postgresEngine{
|
||||
meta: meta,
|
||||
run: run,
|
||||
startSess: startSess,
|
||||
psql: filepath.Join(cfg.EngineBinDir, "psql"),
|
||||
rcService: cfg.RCService,
|
||||
service: cfg.EngineService,
|
||||
pgData: cfg.EngineDataDir,
|
||||
socketDir: cfg.SocketDir,
|
||||
osUser: cfg.EngineUser,
|
||||
parameterManager: parameterManager{
|
||||
probe: probe,
|
||||
params: parameterStore{
|
||||
dir: filepath.Join(cfg.EngineDataDir, "conf.d"),
|
||||
installed: postgresParametersFile,
|
||||
lastGood: postgresLastGoodFile,
|
||||
osUser: cfg.EngineUser,
|
||||
engine: enginePostgres,
|
||||
},
|
||||
repairTimeout: parameterRepairTimeout,
|
||||
repairPoll: parameterRepairPoll,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// The role name and the password ride the environment and are re-quoted by
|
||||
// psql, so neither reaches a shell word or an argv another process can read.
|
||||
func (e *postgresEngine) SetPassword(ctx context.Context, username, password string) error {
|
||||
if username == "" || password == "" {
|
||||
return fmt.Errorf("set-password requires both %s and %s",
|
||||
handlers_rds.CommandParamMasterUsername, handlers_rds.CommandParamMasterUserPassword)
|
||||
}
|
||||
// The statement below runs as the cluster superuser over the socket under
|
||||
// peer auth, so a reserved name in the command payload would hand the
|
||||
// customer the bootstrap superuser rather than rotate their own role.
|
||||
if err := e.meta.ValidateUsernameNotReserved(username); err != nil {
|
||||
return fmt.Errorf("refusing to set the password of a role the engine reserves: %w", err)
|
||||
}
|
||||
// psql interpolates the password into the ALTER ROLE before the server sees
|
||||
// it, and these three are what would write it to the log — the last on any
|
||||
// failure at its own default. SUSET, so the parameter group cannot win.
|
||||
const sql = `SET log_statement = 'none';
|
||||
SET log_min_duration_statement = -1;
|
||||
SET log_min_error_statement = 'panic';
|
||||
\getenv master RDS_MASTER_USERNAME
|
||||
\getenv password RDS_MASTER_PASSWORD
|
||||
ALTER ROLE :"master" WITH LOGIN PASSWORD :'password';
|
||||
`
|
||||
_, err := e.psqlRun(ctx, sql,
|
||||
"RDS_MASTER_USERNAME="+username,
|
||||
"RDS_MASTER_PASSWORD="+password,
|
||||
)
|
||||
if err != nil {
|
||||
// psql echoes the failing statement, which here would carry the new
|
||||
// password back to the control plane and into the event ring.
|
||||
return fmt.Errorf("apply the master password: %s", redact(err.Error(), password))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Installs the resolved set, validates it with the engine's own parser before
|
||||
// the engine adopts it, and reloads. A refused value is rolled back here rather
|
||||
// than left on the data volume, where it would survive every VM replace.
|
||||
func (e *postgresEngine) ApplyParameters(ctx context.Context, params []handlers_rds.Parameter) ([]string, error) {
|
||||
e.paramMu.Lock()
|
||||
defer e.paramMu.Unlock()
|
||||
|
||||
initialState, _ := e.probe.state(ctx)
|
||||
if initialState == engineRecovering {
|
||||
return nil, errors.New("apply parameters while the engine is still starting or recovering")
|
||||
}
|
||||
if initialState == engineServing {
|
||||
// A command can arrive before the first heartbeat. Seed the configuration
|
||||
// currently being served before replacing its file in that window.
|
||||
if err := e.recordServingParametersLocked(ctx); err != nil {
|
||||
return nil, fmt.Errorf("record the parameters serving before the apply: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The check runs against the file in place, because the engine parses the
|
||||
// datadir's own include_dir. The window that leaves is closed by the rollback
|
||||
// below and the last-known-good restore the agent runs at boot.
|
||||
restoreParameters, err := e.params.install(params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := e.checkConfig(ctx); err != nil {
|
||||
return nil, errors.Join(fmt.Errorf("the engine rejected the parameter set: %w", err), restoreParameters())
|
||||
}
|
||||
|
||||
state, _ := e.probe.state(ctx)
|
||||
switch state {
|
||||
case engineAbsent:
|
||||
return e.restartOnRepairSetLocked(ctx)
|
||||
case engineRecovering:
|
||||
return nil, errors.Join(errors.New("the engine entered startup or recovery during the parameter apply"), restoreParameters())
|
||||
}
|
||||
|
||||
// After the engine's own parser has passed the set and before the reload that
|
||||
// adopts both files at once.
|
||||
enforce, restoreEnforcement, err := e.syncTLSEnforcement(ctx, state)
|
||||
if err != nil {
|
||||
return nil, errors.Join(fmt.Errorf("apply the TLS enforcement of the parameter set: %w", err), restoreParameters())
|
||||
}
|
||||
// The two files carry one apply from here: a rejected set withdrawn without
|
||||
// its enforcement rule would leave the instance serving the old parameters
|
||||
// under the new set's TLS posture.
|
||||
restore := func() error { return errors.Join(restoreEnforcement(), restoreParameters()) }
|
||||
|
||||
if err := e.reload(ctx); err != nil {
|
||||
// The engine may have gone down after the first probe. In that case start
|
||||
// it on the new, parser-checked repair set rather than restoring the set
|
||||
// that already left it unable to serve.
|
||||
if state, _ := e.probe.state(ctx); state == engineAbsent {
|
||||
return e.restartOnRepairSetLocked(ctx)
|
||||
}
|
||||
return nil, errors.Join(err, restore())
|
||||
}
|
||||
if err := e.verifyTLSEnforcement(ctx, enforce); err != nil {
|
||||
// The reload already happened, so putting the files back is not enough: a
|
||||
// pg_hba that parsed but says the wrong thing has been adopted, and only a
|
||||
// second reload takes the engine off it.
|
||||
return nil, errors.Join(err, restore(), e.reload(ctx))
|
||||
}
|
||||
return e.pendingRestartParameters(ctx)
|
||||
}
|
||||
|
||||
// Re-derived before the restart: the engine reads pg_hba when it starts, so a
|
||||
// repair set started under the previous set's enforcement would serve one TLS
|
||||
// posture while the API reported the other.
|
||||
func (e *postgresEngine) restartOnRepairSetLocked(ctx context.Context) ([]string, error) {
|
||||
if _, _, err := e.syncTLSEnforcement(ctx, engineAbsent); err != nil {
|
||||
return nil, fmt.Errorf("apply the TLS enforcement of the repair set: %w", err)
|
||||
}
|
||||
return awaitRepairedEngine(ctx, e.probe, e.Restart, e.pendingRestartParameters, e.repairTimeout, e.repairPoll)
|
||||
}
|
||||
|
||||
// The function's own answer is the reload, not psql's exit status: it returns f
|
||||
// when it could not signal the postmaster, and the read-back that follows parses
|
||||
// pg_hba from disk, so an unsignalled reload would otherwise verify clean.
|
||||
func (e *postgresEngine) reload(ctx context.Context) error {
|
||||
out, err := e.psqlRun(ctx, "SELECT pg_reload_conf();\n")
|
||||
if err != nil {
|
||||
return fmt.Errorf("reload the engine configuration: %w", err)
|
||||
}
|
||||
if result := strings.TrimSpace(out); result != "t" {
|
||||
return fmt.Errorf("reload the engine configuration: pg_reload_conf() returned %q", result)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// The postmaster's own answer: a static setting it has stored but not adopted is
|
||||
// what it reports here, so nothing in the guest has to classify one.
|
||||
func (e *postgresEngine) pendingRestartParameters(ctx context.Context) ([]string, error) {
|
||||
out, err := e.psqlRun(ctx, "SELECT name FROM pg_settings WHERE pending_restart ORDER BY name;\n")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read the settings pending a restart: %w", err)
|
||||
}
|
||||
var pending []string
|
||||
for line := range strings.SplitSeq(out, "\n") {
|
||||
if name := strings.TrimSpace(line); name != "" {
|
||||
pending = append(pending, name)
|
||||
}
|
||||
}
|
||||
return pending, nil
|
||||
}
|
||||
|
||||
// Snapshots the include only when the running postmaster has no settings still
|
||||
// pending a restart. The parameter mutex keeps an apply from replacing the file
|
||||
// between that check and the copy.
|
||||
func (e *postgresEngine) RecordServingParameters(ctx context.Context) error {
|
||||
e.paramMu.Lock()
|
||||
defer e.paramMu.Unlock()
|
||||
return e.recordServingParametersLocked(ctx)
|
||||
}
|
||||
|
||||
func (e *postgresEngine) recordServingParametersLocked(ctx context.Context) error {
|
||||
return recordLastGood(ctx, e.params, e.pendingRestartParameters)
|
||||
}
|
||||
|
||||
// Puts the last set the engine accepted back in place, for a restart that failed
|
||||
// after a parameter change.
|
||||
func (e *postgresEngine) RestoreLastKnownGoodParameters(ctx context.Context) (bool, error) {
|
||||
e.paramMu.Lock()
|
||||
defer e.paramMu.Unlock()
|
||||
restored, err := restoreLastGood(ctx, e.params, e.probe)
|
||||
if err != nil || !restored {
|
||||
return restored, err
|
||||
}
|
||||
// A service restart does not re-run rds-init, so a restore that stopped at the
|
||||
// parameter file would bring the last known good set up under the enforcement
|
||||
// of the set that was just rejected. restoreLastGood only replaces the file of
|
||||
// an engine that is down.
|
||||
if _, _, err := e.syncTLSEnforcement(ctx, engineAbsent); err != nil {
|
||||
return true, fmt.Errorf("apply the TLS enforcement of the restored parameter set: %w", err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Turns the installed enforcement value into the pg_hba fact that implements it.
|
||||
// PostgreSQL has no server setting for this, so the parameter file on its own
|
||||
// would report the change applied and enforce nothing.
|
||||
//
|
||||
// The serving-TLS guard runs only against an engine that is up. The restore and
|
||||
// repair paths reach this with the engine down, where the rule is read by the
|
||||
// start that follows rather than by a reload.
|
||||
func (e *postgresEngine) syncTLSEnforcement(ctx context.Context, state engineState) (enforce bool, restore func() error, err error) {
|
||||
enforce, err = installedTLSEnforcement(e.meta.TLSEnforcementParameter(), e.params.installedPath())
|
||||
if err != nil {
|
||||
return false, nil, err
|
||||
}
|
||||
if enforce && state == engineServing {
|
||||
if err := e.requireServingTLS(ctx); err != nil {
|
||||
return false, nil, err
|
||||
}
|
||||
}
|
||||
restore, err = e.writeTLSEnforcement(enforce)
|
||||
if err != nil {
|
||||
return false, nil, err
|
||||
}
|
||||
return enforce, restore, nil
|
||||
}
|
||||
|
||||
// Writes or removes the enforcement rule, returning a func putting back whatever
|
||||
// was there. Removing matters as much as writing: a restored data volume carries
|
||||
// hba.d with it, so a stale rule would keep enforcing under a parameter group
|
||||
// that turns enforcement off.
|
||||
func (e *postgresEngine) writeTLSEnforcement(enforce bool) (func() error, error) {
|
||||
path := e.forceSSLRulePath()
|
||||
previous, readErr := os.ReadFile(path)
|
||||
if readErr != nil && !os.IsNotExist(readErr) {
|
||||
return nil, fmt.Errorf("read the TLS enforcement rule at %s: %w", path, readErr)
|
||||
}
|
||||
existed := readErr == nil
|
||||
restore := func() error {
|
||||
if !existed {
|
||||
return e.removeTLSEnforcement()
|
||||
}
|
||||
return e.params.write(path, previous)
|
||||
}
|
||||
|
||||
if !enforce {
|
||||
if err := e.removeTLSEnforcement(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return restore, nil
|
||||
}
|
||||
// rds-init lays the directory down on every boot, so this covers an agent that
|
||||
// reached an apply before one ran rather than the ordinary case.
|
||||
dir := filepath.Dir(path)
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return nil, fmt.Errorf("create %s: %w", dir, err)
|
||||
}
|
||||
if err := e.params.chownToEngine(dir); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := e.params.write(path, []byte(postgresForceSSLRules)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return restore, nil
|
||||
}
|
||||
|
||||
func (e *postgresEngine) removeTLSEnforcement() error {
|
||||
path := e.forceSSLRulePath()
|
||||
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("withdraw the TLS enforcement rule at %s: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *postgresEngine) forceSSLRulePath() string {
|
||||
return filepath.Join(e.pgData, postgresHBADir, postgresForceSSLRuleFile)
|
||||
}
|
||||
|
||||
// Refuses to require TLS of clients from an engine that is not serving it, which
|
||||
// would reject every connection while the parameter path reported success.
|
||||
func (e *postgresEngine) requireServingTLS(ctx context.Context) error {
|
||||
out, err := e.psqlRun(ctx, "SHOW ssl;\n")
|
||||
if err != nil {
|
||||
return fmt.Errorf("read whether the engine is serving TLS: %w", err)
|
||||
}
|
||||
if serving := strings.TrimSpace(out); serving != "on" {
|
||||
return fmt.Errorf("the engine is serving without TLS (ssl = %q), so requiring it of clients would reject every connection", serving)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PostgreSQL discards a pg_hba it cannot parse on reload and keeps the rules it
|
||||
// is already serving, which presents as an apply that succeeded and enforces
|
||||
// nothing. So the file is read back.
|
||||
//
|
||||
// pg_hba_file_rules re-parses from disk, so this proves the file would be
|
||||
// adopted rather than that the postmaster has adopted it. The inference holds
|
||||
// because a reloaded pg_hba is discarded only on a parse error.
|
||||
const postgresHBAReadBack = `SELECT count(*) FILTER (WHERE type = 'hostnossl' AND auth_method = 'reject'),
|
||||
count(*) FILTER (WHERE error IS NOT NULL) FROM pg_hba_file_rules;
|
||||
`
|
||||
|
||||
func (e *postgresEngine) verifyTLSEnforcement(ctx context.Context, enforce bool) error {
|
||||
out, err := e.psqlRun(ctx, postgresHBAReadBack)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read back the client authentication rules: %w", err)
|
||||
}
|
||||
rules, unparsable, err := parseHBAReadBack(out)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if unparsable > 0 {
|
||||
return fmt.Errorf("the client authentication file carries %d rules the engine could not parse, so it would keep the rules it is already serving", unparsable)
|
||||
}
|
||||
want := 0
|
||||
if enforce {
|
||||
want = postgresForceSSLRuleCount
|
||||
}
|
||||
if rules != want {
|
||||
return fmt.Errorf("the client authentication file carries %d TLS enforcement rules, want %d", rules, want)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func parseHBAReadBack(out string) (rules, unparsable int, err error) {
|
||||
answer := strings.TrimSpace(out)
|
||||
malformed := fmt.Errorf("the client authentication read-back returned %q, want two counts", answer)
|
||||
counts := strings.Split(answer, "|")
|
||||
if len(counts) != 2 {
|
||||
return 0, 0, malformed
|
||||
}
|
||||
parsed := make([]int, len(counts))
|
||||
for i, count := range counts {
|
||||
if parsed[i], err = strconv.Atoi(strings.TrimSpace(count)); err != nil {
|
||||
return 0, 0, malformed
|
||||
}
|
||||
}
|
||||
return parsed[0], parsed[1], nil
|
||||
}
|
||||
|
||||
// The engine's own parser, run offline against the datadir. Reading one setting
|
||||
// back is enough: postgres parses every include first and exits non-zero naming
|
||||
// the file and line of an unknown parameter or an out-of-range value.
|
||||
func (e *postgresEngine) checkConfig(ctx context.Context) error {
|
||||
if _, err := e.run(ctx, command{
|
||||
Name: filepath.Join(filepath.Dir(e.psql), "postgres"),
|
||||
Args: []string{"-D", e.pgData, "-C", "shared_buffers"},
|
||||
Env: []string{"PATH=" + defaultGuestPath},
|
||||
User: e.osUser,
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *postgresEngine) Stop(ctx context.Context) error {
|
||||
return serviceAction(ctx, e.run, e.rcService, e.service, "stop")
|
||||
}
|
||||
|
||||
// Only the parameter rollback calls this: a restart the control plane wants goes
|
||||
// through RebootDBInstance, which cycles the VM.
|
||||
func (e *postgresEngine) Restart(ctx context.Context) error {
|
||||
return serviceAction(ctx, e.run, e.rcService, e.service, "restart")
|
||||
}
|
||||
|
||||
// Feeds sql on stdin rather than as an argument, so a statement is never
|
||||
// visible in the process table.
|
||||
func (e *postgresEngine) psqlRun(ctx context.Context, sql string, env ...string) (string, error) {
|
||||
return e.run(ctx, command{
|
||||
Name: e.psql,
|
||||
Args: e.psqlArgs(),
|
||||
Env: append([]string{"PATH=" + defaultGuestPath}, env...),
|
||||
Stdin: sql,
|
||||
User: e.osUser,
|
||||
})
|
||||
}
|
||||
|
||||
// Reading the script from stdin is what lets one invocation serve both a
|
||||
// one-shot run and a session held open across several statements.
|
||||
func (e *postgresEngine) psqlArgs() []string {
|
||||
return []string{
|
||||
"--no-psqlrc", "--quiet", "--no-align", "--tuples-only",
|
||||
"-v", "ON_ERROR_STOP=1",
|
||||
"-h", e.socketDir,
|
||||
"-p", strconv.FormatInt(e.probe.port.Load(), 10),
|
||||
"-U", e.osUser,
|
||||
"-d", "postgres",
|
||||
"-f", "-",
|
||||
}
|
||||
}
|
||||
|
||||
// Puts the engine into backup mode: the datadir is checkpointed and the engine
|
||||
// stops writing over the pages a snapshot is about to read. The hold is released
|
||||
// by Unquiesce, or by its own deadline, whichever comes first.
|
||||
func (e *postgresEngine) Quiesce(ctx context.Context, label string, hold time.Duration) error {
|
||||
if err := validateQuiesceRequest(label, hold); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Held across the whole start, so a second quiesce waits and then finds the
|
||||
// first one's hold rather than opening a concurrent backup alongside it.
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
if e.held != nil {
|
||||
return fmt.Errorf("the engine is already quiesced for backup %s", e.held.label)
|
||||
}
|
||||
|
||||
// Deliberately not the command's context: the session has to outlive the call
|
||||
// that started it, and its own deadline is what bounds it instead.
|
||||
session, err := e.startSess(context.WithoutCancel(ctx), command{
|
||||
Name: e.psql,
|
||||
Args: e.psqlArgs(),
|
||||
Env: []string{"PATH=" + defaultGuestPath, "RDS_BACKUP_LABEL=" + label},
|
||||
User: e.osUser,
|
||||
SentinelStatement: `\echo ` + sessionSentinel + "\n",
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("open a backup session: %w", err)
|
||||
}
|
||||
|
||||
// fast forces an immediate checkpoint rather than spreading it over the
|
||||
// checkpoint interval, which would hold the snapshot open for minutes.
|
||||
const sql = `\getenv label RDS_BACKUP_LABEL
|
||||
SELECT pg_backup_start(:'label', fast => true);
|
||||
`
|
||||
if err := session.Exec(ctx, sql); err != nil {
|
||||
if closeErr := session.Close(); closeErr != nil {
|
||||
slog.Warn("rds-agent: closing a failed backup session", "err", closeErr)
|
||||
}
|
||||
return fmt.Errorf("put the engine into backup mode: %w", err)
|
||||
}
|
||||
|
||||
e.beginHoldLocked(label, session, hold)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *postgresEngine) Unquiesce(ctx context.Context) error {
|
||||
return e.releaseHold(ctx, "SELECT pg_backup_stop();\n")
|
||||
}
|
||||
@@ -1,697 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
|
||||
)
|
||||
|
||||
// Stands in for the engine in the registry tests, where what matters is that a
|
||||
// command reaches the right method with the right arguments.
|
||||
type fakeEngine struct {
|
||||
username string
|
||||
password string
|
||||
params []handlers_rds.Parameter
|
||||
pending []string
|
||||
stopped bool
|
||||
label string
|
||||
hold time.Duration
|
||||
released bool
|
||||
err error
|
||||
}
|
||||
|
||||
var _ engineOps = (*fakeEngine)(nil)
|
||||
|
||||
func (f *fakeEngine) SetPassword(_ context.Context, username, password string) error {
|
||||
f.username, f.password = username, password
|
||||
return f.err
|
||||
}
|
||||
|
||||
func (f *fakeEngine) ApplyParameters(_ context.Context, params []handlers_rds.Parameter) ([]string, error) {
|
||||
f.params = params
|
||||
return f.pending, f.err
|
||||
}
|
||||
|
||||
func (f *fakeEngine) Stop(context.Context) error {
|
||||
f.stopped = true
|
||||
return f.err
|
||||
}
|
||||
|
||||
func (f *fakeEngine) Quiesce(_ context.Context, label string, hold time.Duration) error {
|
||||
f.label, f.hold = label, hold
|
||||
return f.err
|
||||
}
|
||||
|
||||
func (f *fakeEngine) Unquiesce(context.Context) error {
|
||||
f.released = true
|
||||
return f.err
|
||||
}
|
||||
|
||||
func TestCommandRegistry_SetPasswordCarriesBothParameters(t *testing.T) {
|
||||
engine := &fakeEngine{}
|
||||
reply := newCommander(nil, newCommandRegistry(engine, &fakeStorage{}), 0).execute(context.Background(), handlers_rds.Command{
|
||||
CommandID: "cmd-1",
|
||||
Type: handlers_rds.CommandSetPassword,
|
||||
Parameters: []handlers_rds.Parameter{
|
||||
{Name: handlers_rds.CommandParamMasterUsername, Value: "mulgamaster"},
|
||||
{Name: handlers_rds.CommandParamMasterUserPassword, Value: "n3w-pw"},
|
||||
},
|
||||
})
|
||||
|
||||
if reply.Status != handlers_rds.CommandStatusSucceeded {
|
||||
t.Fatalf("reply = %+v, want succeeded", reply)
|
||||
}
|
||||
if engine.username != "mulgamaster" || engine.password != "n3w-pw" {
|
||||
t.Errorf("engine got %q/%q, want mulgamaster/n3w-pw", engine.username, engine.password)
|
||||
}
|
||||
// The reply is recorded by the control plane; the new password must not be
|
||||
// in it.
|
||||
if strings.Contains(reply.Message, "n3w-pw") {
|
||||
t.Errorf("reply message %q carries the new password", reply.Message)
|
||||
}
|
||||
}
|
||||
|
||||
// CommandReply has no structured payload, so the settings still awaiting a
|
||||
// restart come back as the message the issuer parses.
|
||||
func TestCommandRegistry_ApplyParamsReportsPendingRestart(t *testing.T) {
|
||||
engine := &fakeEngine{pending: []string{"shared_buffers", "max_connections"}}
|
||||
reply := newCommander(nil, newCommandRegistry(engine, &fakeStorage{}), 0).execute(context.Background(), handlers_rds.Command{
|
||||
CommandID: "cmd-2",
|
||||
Type: handlers_rds.CommandApplyParams,
|
||||
Parameters: []handlers_rds.Parameter{{Name: "shared_buffers", Value: "256MB"}},
|
||||
})
|
||||
|
||||
if reply.Status != handlers_rds.CommandStatusSucceeded {
|
||||
t.Fatalf("reply = %+v, want succeeded", reply)
|
||||
}
|
||||
if reply.Message != "shared_buffers,max_connections" {
|
||||
t.Errorf("reply message = %q, want the pending-restart list", reply.Message)
|
||||
}
|
||||
if len(engine.params) != 1 || engine.params[0].Name != "shared_buffers" {
|
||||
t.Errorf("engine got %+v, want the command's parameters", engine.params)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCommandRegistry_StopEngineReachesTheEngine(t *testing.T) {
|
||||
engine := &fakeEngine{}
|
||||
reply := newCommander(nil, newCommandRegistry(engine, &fakeStorage{}), 0).execute(context.Background(),
|
||||
handlers_rds.Command{CommandID: "cmd-3", Type: handlers_rds.CommandStopEngine})
|
||||
|
||||
if reply.Status != handlers_rds.CommandStatusSucceeded {
|
||||
t.Fatalf("reply = %+v, want succeeded", reply)
|
||||
}
|
||||
if !engine.stopped {
|
||||
t.Error("stop-engine did not reach the engine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCommandRegistry_FailureRepliesFailed(t *testing.T) {
|
||||
engine := &fakeEngine{err: errors.New("the engine did not stop")}
|
||||
reply := newCommander(nil, newCommandRegistry(engine, &fakeStorage{}), 0).execute(context.Background(),
|
||||
handlers_rds.Command{CommandID: "cmd-4", Type: handlers_rds.CommandStopEngine})
|
||||
|
||||
if reply.Status != handlers_rds.CommandStatusFailed {
|
||||
t.Fatalf("reply = %+v, want failed", reply)
|
||||
}
|
||||
if !strings.Contains(reply.Message, "the engine did not stop") {
|
||||
t.Errorf("reply message = %q, want the engine's error", reply.Message)
|
||||
}
|
||||
}
|
||||
|
||||
// Records what a postgresEngine would have run, so the SQL and the argv are
|
||||
// asserted without a live cluster.
|
||||
type recordingRunner struct {
|
||||
calls []command
|
||||
out string
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *recordingRunner) run(_ context.Context, c command) (string, error) {
|
||||
r.calls = append(r.calls, c)
|
||||
return r.out, r.err
|
||||
}
|
||||
|
||||
func testPostgresEngineMeta(t *testing.T) handlers_rds.Engine {
|
||||
t.Helper()
|
||||
meta, err := handlers_rds.LookupEngine(enginePostgres)
|
||||
if err != nil {
|
||||
t.Fatalf("LookupEngine(%s): %v", enginePostgres, err)
|
||||
}
|
||||
return meta
|
||||
}
|
||||
|
||||
func newTestEngine(t *testing.T, run commandRunner) *postgresEngine {
|
||||
t.Helper()
|
||||
cfg := testEngineConfig(t)
|
||||
return newPostgresEngine(cfg, testPostgresEngineMeta(t), withPostgresReadBacks(cfg.EngineDataDir, run), nil, newPostgresProbe(cfg, staticProbe(0)))
|
||||
}
|
||||
|
||||
// The same engine with the read-backs left to the runner, for the cases that are
|
||||
// about what the engine does with each answer rather than about the rest of an
|
||||
// apply.
|
||||
func newScriptedTestEngine(t *testing.T, run commandRunner) *postgresEngine {
|
||||
t.Helper()
|
||||
cfg := testEngineConfig(t)
|
||||
return newPostgresEngine(cfg, testPostgresEngineMeta(t), run, nil, newPostgresProbe(cfg, staticProbe(0)))
|
||||
}
|
||||
|
||||
func testEngineConfig(t *testing.T) config {
|
||||
t.Helper()
|
||||
cfg := testLoadConfig(t, enginePostgres)
|
||||
cfg.EngineDataDir = t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(cfg.EngineDataDir, "conf.d"), 0o700); err != nil {
|
||||
t.Fatalf("create conf.d: %v", err)
|
||||
}
|
||||
// The guest's postgres user does not exist here, and the chown of the
|
||||
// installed parameter file has to resolve to something real.
|
||||
current, err := user.Current()
|
||||
if err != nil {
|
||||
t.Fatalf("resolve the current user: %v", err)
|
||||
}
|
||||
cfg.EngineUser = current.Username
|
||||
return cfg
|
||||
}
|
||||
|
||||
// Answers the read-backs every apply runs against a live postmaster, from the
|
||||
// datadir the case is working in: an engine serving TLS, a reload the postmaster
|
||||
// took, and a pg_hba carrying whatever the agent last wrote. The call still
|
||||
// reaches the runner, so a case counting commands sees them; one that is about
|
||||
// the read-backs themselves builds its own engine.
|
||||
func withPostgresReadBacks(dataDir string, run commandRunner) commandRunner {
|
||||
rule := filepath.Join(dataDir, postgresHBADir, postgresForceSSLRuleFile)
|
||||
return func(ctx context.Context, c command) (string, error) {
|
||||
out, err := run(ctx, c)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
switch {
|
||||
case strings.Contains(c.Stdin, "SHOW ssl"):
|
||||
return "on\n", nil
|
||||
case strings.Contains(c.Stdin, "pg_reload_conf"):
|
||||
return "t\n", nil
|
||||
case strings.Contains(c.Stdin, "pg_hba_file_rules"):
|
||||
rules := 0
|
||||
if _, statErr := os.Stat(rule); statErr == nil {
|
||||
rules = postgresForceSSLRuleCount
|
||||
}
|
||||
return fmt.Sprintf("%d|0\n", rules), nil
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
|
||||
// The password reaches psql through the environment and is re-quoted there,
|
||||
// so it is never a shell word or an argv another process can read.
|
||||
func TestPostgresEngine_SetPasswordKeepsTheSecretOutOfArgv(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
|
||||
if err := engine.SetPassword(context.Background(), "mulgamaster", "n3w-pw"); err != nil {
|
||||
t.Fatalf("SetPassword: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 1 {
|
||||
t.Fatalf("ran %d commands, want 1", len(runner.calls))
|
||||
}
|
||||
call := runner.calls[0]
|
||||
for _, arg := range call.Args {
|
||||
if strings.Contains(arg, "n3w-pw") {
|
||||
t.Fatalf("argv %v carries the password", call.Args)
|
||||
}
|
||||
}
|
||||
if strings.Contains(call.Stdin, "n3w-pw") {
|
||||
t.Errorf("SQL %q carries the password literally, want it read from the environment", call.Stdin)
|
||||
}
|
||||
if !slices.Contains(call.Env, "RDS_MASTER_PASSWORD=n3w-pw") {
|
||||
t.Errorf("env %v does not carry the password", call.Env)
|
||||
}
|
||||
if !strings.Contains(call.Stdin, "ALTER ROLE") {
|
||||
t.Errorf("SQL %q does not alter the role", call.Stdin)
|
||||
}
|
||||
}
|
||||
|
||||
// psql interpolates the password into the ALTER ROLE before the server sees it,
|
||||
// so a parameter group that turns statement logging on would write the rotated
|
||||
// password into a postmaster.log every snapshot then carries.
|
||||
func TestPostgresEngine_SetPasswordSilencesStatementLogging(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
|
||||
if err := engine.SetPassword(context.Background(), "mulgamaster", "n3w-pw"); err != nil {
|
||||
t.Fatalf("SetPassword: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 1 {
|
||||
t.Fatalf("ran %d commands, want 1", len(runner.calls))
|
||||
}
|
||||
sql := runner.calls[0].Stdin
|
||||
alter := strings.Index(sql, "ALTER ROLE")
|
||||
if alter < 0 {
|
||||
t.Fatalf("SQL %q does not alter the role", sql)
|
||||
}
|
||||
for _, guard := range []string{
|
||||
"SET log_statement = 'none';",
|
||||
"SET log_min_duration_statement = -1;",
|
||||
"SET log_min_error_statement = 'panic';",
|
||||
} {
|
||||
at := strings.Index(sql, guard)
|
||||
if at < 0 {
|
||||
t.Errorf("SQL %q is missing %q", sql, guard)
|
||||
continue
|
||||
}
|
||||
if at > alter {
|
||||
t.Errorf("SQL %q sets %q only after the ALTER ROLE has already been logged", sql, guard)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// psql echoes the failing statement, which here would carry the new password
|
||||
// back into a reply the control plane records.
|
||||
func TestPostgresEngine_SetPasswordRedactsTheSecretFromAFailure(t *testing.T) {
|
||||
runner := &recordingRunner{err: errors.New(`ERROR: syntax error at or near "n3w-pw"`)}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
|
||||
err := engine.SetPassword(context.Background(), "mulgamaster", "n3w-pw")
|
||||
if err == nil {
|
||||
t.Fatal("SetPassword succeeded against a failing psql")
|
||||
}
|
||||
if strings.Contains(err.Error(), "n3w-pw") {
|
||||
t.Errorf("error %q leaks the password", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "[REDACTED]") {
|
||||
t.Errorf("error %q does not mark the redaction", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresEngine_SetPasswordRejectsAnIncompleteCommand(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
|
||||
if err := engine.SetPassword(context.Background(), "mulgamaster", ""); err == nil {
|
||||
t.Error("SetPassword accepted an empty password")
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("ran %d commands for an incomplete request, want 0", len(runner.calls))
|
||||
}
|
||||
}
|
||||
|
||||
// The apply runs as the cluster superuser under peer auth, so a reserved
|
||||
// name arriving in a command payload would rotate the bootstrap superuser's
|
||||
// password rather than the customer's own role.
|
||||
func TestPostgresEngine_SetPasswordRefusesReservedRoles(t *testing.T) {
|
||||
for _, username := range []string{"postgres", "rds_superuser", "rdsadmin", "pg_toast_owner", "PostGres"} {
|
||||
t.Run(username, func(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
|
||||
err := engine.SetPassword(context.Background(), username, "n3w-pw")
|
||||
if err == nil {
|
||||
t.Fatalf("SetPassword accepted the reserved role %q", username)
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("ran %d commands for a reserved role, want 0", len(runner.calls))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The parameters land where rds-init installs them, so the next boot overwrites
|
||||
// the file rather than shadowing it with a second copy.
|
||||
func TestPostgresEngine_ApplyParametersInstallsAndReloads(t *testing.T) {
|
||||
runner := &recordingRunner{out: "shared_buffers\nmax_connections\n"}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
|
||||
pending, err := engine.ApplyParameters(context.Background(),
|
||||
[]handlers_rds.Parameter{{Name: "shared_buffers", Value: "256MB"}})
|
||||
if err != nil {
|
||||
t.Fatalf("ApplyParameters: %v", err)
|
||||
}
|
||||
|
||||
installed, err := os.ReadFile(filepath.Join(engine.pgData, "conf.d", "10-rds-parameters.conf"))
|
||||
if err != nil {
|
||||
t.Fatalf("read the installed parameters: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(installed), "shared_buffers = '256MB'") {
|
||||
t.Errorf("installed parameters = %q, want the resolved value", installed)
|
||||
}
|
||||
if len(runner.calls) != 5 {
|
||||
t.Fatalf("ran %d commands, want a config check, the TLS guard, a reload, the rule read-back and a pending-restart read", len(runner.calls))
|
||||
}
|
||||
// The check and the TLS guard both run before the reload, so neither a value
|
||||
// the engine would refuse nor a rule rejecting every client reaches a running
|
||||
// cluster. The read-back of the rules comes after, since the reload is what
|
||||
// puts them in force.
|
||||
if !slices.Contains(runner.calls[0].Args, "-C") {
|
||||
t.Errorf("first command = %v, want the offline config check", runner.calls[0].Args)
|
||||
}
|
||||
if !strings.Contains(runner.calls[1].Stdin, "SHOW ssl") {
|
||||
t.Errorf("second statement = %q, want the serving-TLS guard", runner.calls[1].Stdin)
|
||||
}
|
||||
if !strings.Contains(runner.calls[2].Stdin, "pg_reload_conf") {
|
||||
t.Errorf("third statement = %q, want the reload", runner.calls[2].Stdin)
|
||||
}
|
||||
if !strings.Contains(runner.calls[3].Stdin, "pg_hba_file_rules") {
|
||||
t.Errorf("fourth statement = %q, want the client authentication read-back", runner.calls[3].Stdin)
|
||||
}
|
||||
if !strings.Contains(runner.calls[4].Stdin, "pending_restart") {
|
||||
t.Errorf("fifth statement = %q, want the pending-restart read", runner.calls[4].Stdin)
|
||||
}
|
||||
if len(pending) != 2 || pending[0] != "shared_buffers" || pending[1] != "max_connections" {
|
||||
t.Errorf("pending = %v, want both settings the engine reported", pending)
|
||||
}
|
||||
}
|
||||
|
||||
// A reload that failed leaves the engine on its old values, so reporting the
|
||||
// apply as successful would tell the customer a change took effect that did not.
|
||||
func TestPostgresEngine_ApplyParametersSurfacesAFailedReload(t *testing.T) {
|
||||
runner := &reloadFailingRunner{reloadErr: errors.New("reload was rejected")}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
|
||||
if _, err := engine.ApplyParameters(context.Background(),
|
||||
[]handlers_rds.Parameter{{Name: "work_mem", Value: "8MB"}}); err == nil {
|
||||
t.Fatal("ApplyParameters succeeded against a failing reload")
|
||||
}
|
||||
}
|
||||
|
||||
// pg_reload_conf() returns f when it could not signal the postmaster, and psql
|
||||
// exits 0 all the same. Every read-back after it parses files from disk, so an
|
||||
// apply that trusted the exit status would verify clean against a server still
|
||||
// serving the old rules.
|
||||
func TestPostgresEngine_ApplyParametersSurfacesAnUnsignalledReload(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on", ruleRows: postgresForceSSLRuleCount, reload: "f"}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
|
||||
_, err := engine.ApplyParameters(context.Background(),
|
||||
[]handlers_rds.Parameter{{Name: "work_mem", Value: "8MB"}})
|
||||
if err == nil || !strings.Contains(err.Error(), `pg_reload_conf() returned "f"`) {
|
||||
t.Fatalf("ApplyParameters error = %v, want a refusal naming the answer the postmaster gave", err)
|
||||
}
|
||||
}
|
||||
|
||||
type reloadFailingRunner struct {
|
||||
recordingRunner
|
||||
|
||||
reloadErr error
|
||||
}
|
||||
|
||||
func (r *reloadFailingRunner) run(ctx context.Context, c command) (string, error) {
|
||||
if strings.Contains(c.Stdin, "pg_reload_conf") {
|
||||
r.calls = append(r.calls, c)
|
||||
return "", r.reloadErr
|
||||
}
|
||||
return r.recordingRunner.run(ctx, c)
|
||||
}
|
||||
|
||||
// A failed instance cannot reload through psql. Its corrected, parser-checked
|
||||
// include must start the service and remain installed as the serving set.
|
||||
func TestPostgresEngine_ApplyParametersRestartsOnARepairSetWhileEngineIsDown(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
codes := []int{2, 2, 0}
|
||||
probeCall := 0
|
||||
cfg := testProbeConfig()
|
||||
engine.probe = newPostgresProbe(cfg, func(context.Context, string, ...string) (int, string, error) {
|
||||
code := codes[min(probeCall, len(codes)-1)]
|
||||
probeCall++
|
||||
return code, "", nil
|
||||
})
|
||||
engine.repairTimeout = 100 * time.Millisecond
|
||||
engine.repairPoll = time.Millisecond
|
||||
params := []handlers_rds.Parameter{
|
||||
{Name: "shared_buffers", Value: "32768"},
|
||||
{Name: "work_mem", Value: "8192"},
|
||||
}
|
||||
|
||||
pending, err := engine.ApplyParameters(t.Context(), params)
|
||||
if err != nil {
|
||||
t.Fatalf("ApplyParameters: %v", err)
|
||||
}
|
||||
if len(pending) != 0 {
|
||||
t.Errorf("pending = %v, want the restarted engine's actual pending set", pending)
|
||||
}
|
||||
if len(runner.calls) != 3 {
|
||||
t.Fatalf("calls = %+v, want config check, service restart and pending read", runner.calls)
|
||||
}
|
||||
if !slices.Contains(runner.calls[0].Args, "-C") {
|
||||
t.Errorf("first call = %+v, want the offline config check", runner.calls[0])
|
||||
}
|
||||
if !slices.Equal(runner.calls[1].Args, []string{"postgresql", "restart"}) {
|
||||
t.Errorf("second call args = %v, want the service restart", runner.calls[1].Args)
|
||||
}
|
||||
if !strings.Contains(runner.calls[2].Stdin, "pending_restart") {
|
||||
t.Errorf("third call = %+v, want the pending-restart read", runner.calls[2])
|
||||
}
|
||||
installed, err := os.ReadFile(engine.params.installedPath())
|
||||
if err != nil {
|
||||
t.Fatalf("read installed parameters: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(installed), "work_mem = '8192'") {
|
||||
t.Errorf("installed parameters = %q, want the repair set retained", installed)
|
||||
}
|
||||
}
|
||||
|
||||
// Answers the two read-backs an apply runs from fields the case sets, so what
|
||||
// the engine does with each answer is what is under test.
|
||||
type tlsReadBackRunner struct {
|
||||
recordingRunner
|
||||
|
||||
ssl string
|
||||
ruleRows int
|
||||
brokenRows int
|
||||
// What pg_reload_conf() answered. Empty is the postmaster having taken the
|
||||
// signal, which is every case but the one about it not having.
|
||||
reload string
|
||||
}
|
||||
|
||||
func (r *tlsReadBackRunner) run(ctx context.Context, c command) (string, error) {
|
||||
switch {
|
||||
case strings.Contains(c.Stdin, "SHOW ssl"):
|
||||
r.calls = append(r.calls, c)
|
||||
return r.ssl + "\n", nil
|
||||
case strings.Contains(c.Stdin, "pg_reload_conf"):
|
||||
r.calls = append(r.calls, c)
|
||||
if r.reload == "" {
|
||||
return "t\n", nil
|
||||
}
|
||||
return r.reload + "\n", nil
|
||||
case strings.Contains(c.Stdin, "pg_hba_file_rules"):
|
||||
r.calls = append(r.calls, c)
|
||||
return fmt.Sprintf("%d|%d\n", r.ruleRows, r.brokenRows), nil
|
||||
}
|
||||
return r.recordingRunner.run(ctx, c)
|
||||
}
|
||||
|
||||
func forceSSLParameter(t *testing.T, value string) []handlers_rds.Parameter {
|
||||
t.Helper()
|
||||
return []handlers_rds.Parameter{
|
||||
{Name: "work_mem", Value: "4096"},
|
||||
{Name: testPostgresEngineMeta(t).TLSEnforcementParameter(), Value: value},
|
||||
}
|
||||
}
|
||||
|
||||
// PostgreSQL has no server setting for this, so the value in the parameter file
|
||||
// is inert: the rule the generated pg_hba includes is the whole of it.
|
||||
func TestPostgresEngine_ApplyParametersWritesTheEnforcementRule(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on", ruleRows: postgresForceSSLRuleCount}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
|
||||
if _, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "1")); err != nil {
|
||||
t.Fatalf("ApplyParameters: %v", err)
|
||||
}
|
||||
rule, err := os.ReadFile(engine.forceSSLRulePath())
|
||||
if err != nil {
|
||||
t.Fatalf("read the enforcement rule: %v", err)
|
||||
}
|
||||
if string(rule) != postgresForceSSLRules {
|
||||
t.Errorf("enforcement rule = %q, want the constant rds-init also writes", rule)
|
||||
}
|
||||
}
|
||||
|
||||
// A snapshot carries hba.d with it, so restoring one taken while enforcing into
|
||||
// a group that turns enforcement off has to take the rule with it.
|
||||
func TestPostgresEngine_ApplyParametersRemovesAStaleEnforcementRule(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on"}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
if err := os.MkdirAll(filepath.Dir(engine.forceSSLRulePath()), 0o700); err != nil {
|
||||
t.Fatalf("create hba.d: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(engine.forceSSLRulePath(), []byte(postgresForceSSLRules), 0o600); err != nil {
|
||||
t.Fatalf("write the rule the restored volume carried: %v", err)
|
||||
}
|
||||
|
||||
if _, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "0")); err != nil {
|
||||
t.Fatalf("ApplyParameters: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(engine.forceSSLRulePath()); !os.IsNotExist(err) {
|
||||
t.Errorf("the stale enforcement rule is still in place (stat err = %v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The ordinary state of an instance whose resolved set predates the parameter,
|
||||
// and the whole of the migration: it begins requiring TLS with no control-plane
|
||||
// work at all.
|
||||
func TestPostgresEngine_ApplyParametersEnforcesOnAnAbsentKey(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on", ruleRows: postgresForceSSLRuleCount}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
|
||||
if _, err := engine.ApplyParameters(t.Context(),
|
||||
[]handlers_rds.Parameter{{Name: "work_mem", Value: "4096"}}); err != nil {
|
||||
t.Fatalf("ApplyParameters: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(engine.forceSSLRulePath()); err != nil {
|
||||
t.Errorf("a set predating the parameter did not enforce (stat err = %v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The resolver canonicalises every boolean, so a value that is neither means the
|
||||
// file was written by something other than the platform. Reading an unparsable
|
||||
// security setting as off is the one choice not open here.
|
||||
func TestPostgresEngine_ApplyParametersRefusesAnUnparsableEnforcementValue(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on"}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
|
||||
_, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "yes"))
|
||||
if err == nil || !strings.Contains(err.Error(), "neither 1 nor 0") {
|
||||
t.Fatalf("ApplyParameters error = %v, want a refusal naming the unreadable value", err)
|
||||
}
|
||||
if _, statErr := os.Stat(engine.params.installedPath()); !os.IsNotExist(statErr) {
|
||||
t.Errorf("the refused set is still installed (stat err = %v)", statErr)
|
||||
}
|
||||
}
|
||||
|
||||
// Without this, a live enable against an engine started with no certificate
|
||||
// would reject every subsequent connection and report the parameter applied.
|
||||
func TestPostgresEngine_ApplyParametersRefusesToEnforceWithoutTLS(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "off"}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
|
||||
_, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "1"))
|
||||
if err == nil || !strings.Contains(err.Error(), "serving without TLS") {
|
||||
t.Fatalf("ApplyParameters error = %v, want a refusal naming the engine's own TLS state", err)
|
||||
}
|
||||
if _, statErr := os.Stat(engine.forceSSLRulePath()); !os.IsNotExist(statErr) {
|
||||
t.Errorf("the enforcement rule was written anyway (stat err = %v)", statErr)
|
||||
}
|
||||
}
|
||||
|
||||
// PostgreSQL discards a pg_hba it cannot parse and keeps the rules it is already
|
||||
// serving, which presents as an apply that succeeded and enforces nothing. The
|
||||
// shape is asserted as well as the errors, because our own derivation writing
|
||||
// the wrong thing is the failure that will actually happen.
|
||||
func TestPostgresEngine_ApplyParametersFailsAnUnverifiableEnforcement(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
value string
|
||||
ruleRows int
|
||||
brokenRows int
|
||||
want string
|
||||
}{
|
||||
{name: "rule missing", value: "1", ruleRows: 0, want: "carries 0 TLS enforcement rules, want 2"},
|
||||
{name: "rule present when it should not be", value: "0", ruleRows: 2, want: "carries 2 TLS enforcement rules, want 0"},
|
||||
{name: "a rule the engine could not parse", value: "1", ruleRows: 2, brokenRows: 1, want: "could not parse"},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on", ruleRows: tc.ruleRows, brokenRows: tc.brokenRows}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
|
||||
_, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, tc.value))
|
||||
if err == nil || !strings.Contains(err.Error(), tc.want) {
|
||||
t.Fatalf("ApplyParameters error = %v, want %q", err, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A rejected set has to take its enforcement with it. Withdrawing one without
|
||||
// the other would leave the instance serving the previous parameters under the
|
||||
// rejected set's TLS posture.
|
||||
func TestPostgresEngine_ApplyParametersWithdrawsBothFilesTogether(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on", ruleRows: postgresForceSSLRuleCount}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
if _, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "1")); err != nil {
|
||||
t.Fatalf("first ApplyParameters: %v", err)
|
||||
}
|
||||
|
||||
// Turning enforcement off, verified against a pg_hba that still carries the
|
||||
// rule — so the apply fails after both files have already been replaced.
|
||||
if _, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "0")); err == nil {
|
||||
t.Fatal("ApplyParameters succeeded against an unverifiable enforcement")
|
||||
}
|
||||
installed, err := os.ReadFile(engine.params.installedPath())
|
||||
if err != nil {
|
||||
t.Fatalf("read the installed parameters: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(installed), "rds.force_ssl = '1'") {
|
||||
t.Errorf("installed parameters = %q, want the previous accepted set restored", installed)
|
||||
}
|
||||
if _, err := os.Stat(engine.forceSSLRulePath()); err != nil {
|
||||
t.Errorf("the enforcement of the previous set was not restored with it (stat err = %v)", err)
|
||||
}
|
||||
// The reload already adopted the rejected files, so putting them back is not
|
||||
// enough on its own.
|
||||
if !strings.Contains(runner.calls[len(runner.calls)-1].Stdin, "pg_reload_conf") {
|
||||
t.Errorf("last statement = %q, want a reload taking the engine off the withdrawn set", runner.calls[len(runner.calls)-1].Stdin)
|
||||
}
|
||||
}
|
||||
|
||||
// A service restart does not re-run rds-init, so a restore stopping at the
|
||||
// parameter file would bring the last known good set up under the enforcement of
|
||||
// the set that was just rejected.
|
||||
func TestPostgresEngine_RestoreLastKnownGoodReDerivesEnforcement(t *testing.T) {
|
||||
runner := &tlsReadBackRunner{ssl: "on", ruleRows: postgresForceSSLRuleCount}
|
||||
engine := newScriptedTestEngine(t, runner.run)
|
||||
|
||||
if _, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "1")); err != nil {
|
||||
t.Fatalf("ApplyParameters(enforcing): %v", err)
|
||||
}
|
||||
if err := engine.RecordServingParameters(t.Context()); err != nil {
|
||||
t.Fatalf("RecordServingParameters: %v", err)
|
||||
}
|
||||
runner.ruleRows = 0
|
||||
if _, err := engine.ApplyParameters(t.Context(), forceSSLParameter(t, "0")); err != nil {
|
||||
t.Fatalf("ApplyParameters(not enforcing): %v", err)
|
||||
}
|
||||
if _, err := os.Stat(engine.forceSSLRulePath()); !os.IsNotExist(err) {
|
||||
t.Fatalf("the enforcement rule survived the set that turned it off (stat err = %v)", err)
|
||||
}
|
||||
|
||||
engine.probe = stubProbe(t, 2, nil)
|
||||
restored, err := engine.RestoreLastKnownGoodParameters(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("RestoreLastKnownGoodParameters: %v", err)
|
||||
}
|
||||
if !restored {
|
||||
t.Fatal("did not restore a set that differs from the last known good")
|
||||
}
|
||||
if _, err := os.Stat(engine.forceSSLRulePath()); err != nil {
|
||||
t.Errorf("the restored set's enforcement was not re-derived (stat err = %v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresEngine_ApplyParametersKeepsRepairSetWhenRestartTimesOut(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
engine := newTestEngine(t, runner.run)
|
||||
engine.probe = stubProbe(t, 2, nil)
|
||||
engine.repairTimeout = 10 * time.Millisecond
|
||||
engine.repairPoll = time.Millisecond
|
||||
|
||||
_, err := engine.ApplyParameters(t.Context(), []handlers_rds.Parameter{{Name: "work_mem", Value: "8192"}})
|
||||
if err == nil || !strings.Contains(err.Error(), "wait for the engine") {
|
||||
t.Fatalf("ApplyParameters error = %v, want the repair wait failure", err)
|
||||
}
|
||||
installed, readErr := os.ReadFile(engine.params.installedPath())
|
||||
if readErr != nil {
|
||||
t.Fatalf("read installed parameters: %v", readErr)
|
||||
}
|
||||
if !strings.Contains(string(installed), "work_mem = '8192'") {
|
||||
t.Errorf("installed parameters = %q, want the checked repair set retained", installed)
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user