mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
dev
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
b5879f44ff |
ADR-0001 package-boundary migration checkpoint (#1190)
* refactor: move telemetry file naming to contract
* docs: record telemetry contract move
* refactor: move aws xml response helpers
* docs: record aws xml helper move
* refactor: move aws tag projections to foundation
* docs: record aws tag projection move
* refactor: move account key to state foundation
* docs: record account key state move
* refactor: move host privilege escalation to runtime
* docs: record host privilege move
* chore: resolve lint findings from package-boundary moves
* refactor: move process supervision helpers to runtime host
* docs: record process supervision move
* refactor: move nbd endpoint helpers to compute runtime
* docs: record nbd endpoint move
* refactor: move image catalogue to operator
* docs: record image catalogue move
* refactor: move resource id generation to aws foundation
* docs: record resource id move
* refactor: move predastore role to runtime roles
* docs: point service manifest at acm domain
* docs: record predastore role move
* refactor: move northstar role to runtime roles
* docs: record northstar role move
* refactor: move awsgw role to runtime roles
* docs: record awsgw role move
* refactor: move error payload helpers to aws errors
* docs: record error payload move
* refactor: move account identifiers to aws identifiers
* docs: record account identifier move
* refactor: move NATS transport to messaging foundation
* docs: record NATS transport move
* refactor: move system instance contract to EC2 domain
* docs: record system instance move
* refactor: move system VPC builder to network domain
* docs: record system VPC move
* refactor: move instance metadata service to EC2 domain
* docs: record guest metadata move and dependency-rule debt
* refactor: move DNS control plane to DNS domain
* docs: record DNS domain move
* refactor: move base zone bootstrap to DNS domain
* docs: record base zone bootstrap move
* refactor: replace SystemAccountID with GlobalAccountID
* docs: record SystemAccountID removal
* refactor: extract CA key-pair loader to foundation pki
* docs: record foundation pki extraction
* refactor: move tenant CA paths to ACM domain
* docs: record tenant CA path move
* refactor: move IPsec certificate handling to network host
* docs: record IPsec certificate move
* refactor: move Ochre serving lifecycle to the Ochre domain
handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.
* docs: record Ochre domain move
* refactor: move Ochre vector store to the Ochre domain
handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.
* docs: record Ochre vector store move
* refactor: move Hugging Face Hub client to the Ochre domain
gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.
* docs: record Hugging Face Hub client move and Ochre gateway debt
* refactor: move EC2 tag mutation helpers to the EC2 domain
utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.
* docs: record EC2 tag mutation move and migrate split finding
* refactor: move network packages under the network domain
Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.
* docs: record network domain move
* refactor: move standalone EC2 resource packages to the EC2 domain
Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.
* docs: record standalone EC2 resource moves
* refactor: move EC2 VPC networking packages to the EC2 domain
Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.
* docs: record EC2 VPC networking moves
* refactor: move EC2 compute and storage packages to the EC2 domain
Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.
* docs: record EC2 compute and storage moves
* refactor: move ECR credential provider to the EKS agent tree
Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.
* docs: record ECR credential provider move
* refactor: move EKS gateway publish helper to the EKS agent tree
Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.
* docs: record EKS gateway publish move
* refactor: move EKS gateway fetch helper to the EKS agent tree
Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.
* docs: record EKS gateway fetch move
* refactor: move EKS webhook cert helper to the EKS agent tree
cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.
* docs: record EKS webhook cert move
* refactor: move EKS konnectivity cert helper to the EKS agent tree
cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.
* docs: record EKS konnectivity cert move
* refactor: move EKS token webhook to the EKS agent tree
cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.
* docs: record EKS token webhook move
* refactor: move ECS agent runtime and credentials packages to the ECS agent tree
* docs: record ECS agent subpackage move
* refactor: move ECS agent implementation to the ECS agent tree
cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.
* docs: record ECS agent move
* refactor: move RDS guest agent to agents/rds/agent
* docs: record RDS guest agent move
* refactor: move installer packages to operator/installer
* docs: record installer move
* refactor: move spx command tree to operator/cli
Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.
* docs: record spx command tree move
* chore: accept known vulnerabilities with an expiry
The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.
* refactor: move EC2 account action adapters to domains/ec2/awsapi
* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi
* refactor: move EC2 key action adapters to domains/ec2/awsapi
* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi
* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi
* docs: record EC2 standalone action adapter moves
* refactor: move EC2 zone action adapters to domains/ec2/awsapi
* refactor: move EC2 vpc action adapters to domains/ec2/awsapi
* refactor: move EC2 eigw action adapters to domains/ec2/awsapi
* refactor: move EC2 eip action adapters to domains/ec2/awsapi
* refactor: move EC2 igw action adapters to domains/ec2/awsapi
* refactor: move EC2 natgw action adapters to domains/ec2/awsapi
* refactor: move EC2 routetable action adapters to domains/ec2/awsapi
* docs: record EC2 VPC networking action adapter moves
* refactor: move EC2 image action adapters to domains/ec2/awsapi
* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi
* refactor: move EC2 volume action adapters to domains/ec2/awsapi
* refactor: move EC2 tags action adapters to domains/ec2/awsapi
* docs: record EC2 storage and tag action adapter moves
* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi
* refactor: move EC2 idempotency adapter to domains/ec2/awsapi
* refactor: move EC2 instance action adapters to domains/ec2/awsapi
* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi
* docs: record EC2 instance action adapter moves
* refactor: move AWS REST-JSON router to ingress/aws/rest
* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit
* docs: record AWS ingress helper moves and tooling conformance
* docs: correct historical lifecycle references
* test: characterize ECR control-plane dispatch before registration seam
* feat: add AWS ingress service registration contract
* docs: record ingress registration phases 0 and 1
* refactor: route registered AWS services through the ingress dispatch registry
The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.
The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.
* docs: record ingress dispatch registry consumption
* refactor: register the ECR control plane from its domain
domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.
Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.
* docs: record ECR control-plane registration
* fix: compose the ECR control plane in the integration harness
The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.
* docs: record ingress pipeline deferral
* refactor: extract AWS error envelope rendering to ingress
ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.
The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.
* docs: record error envelope extraction
* refactor: move AWS service selection into ingress dispatch
Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.
* docs: record ingress service selection move
* test: characterize ACM control-plane dispatch before registration
Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.
* refactor: register ACM control plane from its domain
domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.
* docs: record ACM registration
* docs: record ECS, EKS and RDS source inventory and first-resource ranking
* docs: record EKS control-plane secrets persisted in the cluster record
* chore: drop containerd advisory acceptance now that dev ships v2.4.1
* test: characterize RDS DB subnet group lifecycle before extraction
* docs: record RDS DB subnet group lifecycle contract
* refactor: extract the RDS DB subnet group lifecycle owner
domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.
* docs: record RDS DB subnet group owner extraction
* docs: correct RDS subnet group tagging debt and conformance scope
* test: characterize ECS task definition lifecycle before extraction
* docs: record ECS task definition lifecycle contract
* refactor: extract ECS task definition owner into domains/ecs/taskdefinition
Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.
The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.
List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.
* docs: record ECS task definition owner extraction
* test: characterize EKS access entry lifecycle before extraction
* docs: record EKS access entry lifecycle contract
* refactor: extract EKS access entry owner into domains/eks/access
Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.
Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.
List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.
* docs: record EKS access entry owner extraction
* test: characterize RDS engine catalogue before extraction
* fix: return InvalidParameterValue for a missing RDS master password
ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.
* docs: record RDS engine catalogue boundary and evidence
* refactor: extract RDS engine catalogue into domains/rds/engine
Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.
The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.
The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.
* docs: record RDS engine catalogue owner move
* test: characterize RDS DB parameter group lifecycle before extraction
* docs: record RDS DB parameter group lifecycle contract
* refactor: extract RDS DB parameter group owner
Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.
* docs: record RDS DB parameter group owner extraction
* test: characterize EKS add-on lifecycle and guest wire
Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.
* docs: record EKS add-on lifecycle contract
* fix: require iam:PassRole for EKS add-on service account role
CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.
* docs: record EKS add-on PassRole check
* refactor: extract EKS add-on owner
Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.
* docs: record EKS add-on owner extraction
* refactor: extract EKS add-on guest wire contract
Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.
* docs: record EKS add-on wire contract extraction
* docs: update EKS add-on inventory and remaining edges
* docs: draft EKS add-on lifecycle correction design for review
* chore: run contracts tests in the default test, coverage and race targets
* docs: apply review decisions to EKS add-on lifecycle design
* test: characterize EKS add-on v1 compatibility with added fields
Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.
* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
* docs: draft EKS add-on v2 guest contract design for review
* docs: qualify historical architecture references
* docs: apply review decisions and ADR-0006 to EKS add-on v2 design
* fix(eks): bind internal report publishing to the caller's cluster
PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.
PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.
* fix(eks): bind webhook token review to the caller's cluster
WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.
WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.
* fix(eks): validate add-on serviceAccountRoleArn syntax
CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.
The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.
* docs: record EKS internal-route and role-ARN security corrections
* docs: add EKS add-on release membership and activation case study
* fix: point UI build, CI and lint rules at moved package paths
* docs: record EKS add-on activation review decisions
* refactor: move quota admission to domains/admission/quota
Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.
* docs: reconcile package-boundary record for ADR-0001 close-out
Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.
* test: move KV migration engine tests to foundation/state/migrate
The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.
* refactor: move config and object-store upgrade workflow to operator/upgrade
Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.
* docs: record migrate move to operator/upgrade
* test: enforce ADR-0001 layering with a blocking ratchet
Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.
Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.
* refactor: drop the legacy handlers_ecr import alias
Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.
* chore: mark spf13/pflag as a direct dependency
operator/cli/wiring.go imports pflag directly. Same version; no upgrade.
* docs: record test-package-check and nilaway as unverified gates
* refactor: move the layering test to module-root architecture/
The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.
* fix: update paths left stale by the package moves
The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.
* refactor: move the default-VPC subject into contracts/ec2/v1
utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.
* docs: qualify the migrate split and record admin and Viperblock findings
The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.
* refactor: move operator-only admin code to operator/admin
Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.
foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.
* docs: record operator/admin split in package-boundary migration
* test: pin viperblockd fence, abandon and dirty-marker wire shapes
Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.
* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts
Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.
Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.
* refactor: split the Viperblock EBS adapter out of services/viperblockd
Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.
The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.
Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.
* refactor: move the viperblockd role to runtime/roles/viperblockd
With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.
* docs: record viperblockd contract, adapter and role moves
* test: characterise vpc.* network-realisation routes
Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.
Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).
* feat: add contracts/network/v1 for vpc.* realisation routes
Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.
* refactor: use contracts/network/v1 for vpc.* producers and consumers
Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.
Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.
No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.
* refactor: replace utils/vpcd_event.go with a typed network projection client
Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.
utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.
* refactor: route the remaining fire-and-forget vpc.* producers through projection
CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.
* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils
The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.
utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.
* docs: record the network contract, projection client and utils removal
* test: characterise instance-state persistence, recovery and generation handling
Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.
* test: freeze direct instance-state consumers
Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.
* docs: record ADR-0007 slice 0 instance-state inventory
Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.
* test: narrow the instance-state ratchet to instance-state access
Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.
* docs: record the narrowed instance-state ratchet commit
* docs: design the instance-state transition record and activation gate
* docs: record the instance-state transition-record design commit
* docs: record slice 1a activation-gate decisions
* feat: add instance transition-record codec with format marker
Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.
Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.
No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.
* docs: record slice 1b step 1 codec and awsgw rollout qualification
Records the codec added in
|
||
|
Julian Sommer
|
c81abd1836 | fix: add fipsboot guard to missing services | ||
|
Josh Siviour
|
6cf9ee9d03 |
fix(dhcp): retransmit within a DORA transaction, resolve the chaddr on every path, stop masking IPAM errors (#593)
* fix(dhcp): retransmit within a DORA transaction instead of restarting it nclient4's retryFn re-sends the same packet under the same xid with a doubling read deadline, which is RFC 2131 4.1 retransmission. WithRetry(1) runs that loop exactly once, so the client transmitted a single DISCOVER per attempt and never retransmitted it. socketTimeout then handed that lone transmission the caller's whole budget, so it sat silent for 4s, then 8s, then 16s, then 32s. Manager does retry, on that schedule, but each attempt builds a fresh client and therefore a fresh xid. Every attempt is a new transaction rather than a retransmission of the previous one, so an OFFER answering an earlier exchange is discarded as belonging to an unknown transaction. Send six transmissions per exchange starting from a 1s read, spanning a 63s ladder — longer than the widest window Manager schedules, so an attempt ends on the caller's context rather than on an exhausted ladder. socketTimeout now clamps to the caller's remaining budget instead of consuming it in one silent read. Observed live recovering an exchange whose OFFER arrived on the third transmission, which a single-shot client would have missed. Both allocation sites also stop replacing the allocator's error with a bare InsufficientAddressCapacity. ResolveErrorCode unwraps, so a genuinely exhausted static pool still surfaces that code while a DHCP fault reports as itself. The flat mapping is why every failure on this path read as a capacity problem regardless of cause. * chore(dhcptest): add --broadcast so the probe can mirror vpcd's Acquire path * fix(dhcp): resolve the chaddr on renew and release, not just on acquire Acquire derived a chaddr when the request carried none. Renew and Release read lease.HWAddr straight out of the store and handed it to nclient4, so a lease persisted before the chaddr was carried renewed with all zeros in chaddr and option 61 fell back to its untyped form. The lease then lands upstream with no hardware address against it — the outcome option 61 was fixed to stop, reached by a different route — and because the server sees an unknown client each time, every renewal takes a new lease. resolveHWAddr now serves all three paths and treats all-zeros as unusable rather than merely empty: six zero bytes passes a width check yet is the exact value that produces the unattributable entry. * test(dhcp): cover chaddr resolution and the unmasked allocation errors |
||
|
Josh Siviour
|
291e550e8a |
fix(dhcp): honour the caller's retransmission budget and encode option 61 correctly (#565)
* fix(dhcp): derive nclient4 read deadline from the caller's context
nclient4 races its own read deadline against the caller's context and the
shorter one ends the attempt. Acquire and Renew passed a fixed 5s, so every
window in Manager's 4s/8s/16s/32s retransmission schedule past the first was
truncated to 5s: the client listened for ~19s of its 90s budget and a server
that went quiet for longer was never waited out.
Measured in the sink, the split was total — attempt 1 (jittered under 5s)
always failed with the caller's context deadline, attempts 2 to 4 always
failed with nclient4's ErrNoResponse — and only 27% of clients that lost
attempt 1 were rescued by attempt 2.
Track the context's remaining time instead, keeping the configured value as
the fallback its doc comment already describes.
* fix(dhcp): keep the socket deadline strictly behind the caller's
Setting the nclient4 read deadline to exactly the context's remaining time
makes the two expire together, so a timed-out attempt reports either
context.DeadlineExceeded or nclient4's ErrNoResponse depending on which
goroutine wins. Observed live on a settled bridge: three attempts reported the
context deadline and the fourth reported ErrNoResponse, which reads as a
packet-matching fault rather than the plain timeout it is.
Give the socket a second of grace so ctx.Done() is always what fires.
* fix(dhcp): encode option 61 with its hardware-type byte
The client identifier was sent as a bare string, so the server read the
identifier's first character as the hardware type — "dhcp-gw-lrp-vpc-x"
arrived as type 100 ('d') with the identifier truncated. Since that is not
Ethernet, the server recorded no hardware address, and the upstream lease
table listed our leases with a MAC of all zeroes.
Send the chaddr typed as Ethernet, falling back to a type-0 identifier when
no hardware address is available. RELEASE now encodes identically, since the
server matches a release to a lease by client identifier.
Option 61 no longer carries the readable id, so restore it via hostname and
vendor class, which were specified but never populated. dhcptest builds its
options through the same helper rather than its own copy, which had drifted
into reproducing the broken encoding.
* fix(dhcptest): derive the probe MAC the way vpcd does
The probe carried its own copy of the MAC derivation, still on the 24-bit
02:00:00:xx:xx:xx hash that vpcd moved off when widening it to a SHA-256
prefix. So --spinifex-id leased under a MAC the daemon would never use:
"gateway-wan1" resolved to 02:00:00:7c:b7:b6 here and 02:54:df:71:22:5c in
vpcd, which makes the probe unable to reproduce a daemon lease at all.
Call DeriveMAC instead and drop the local copy.
|
||
|
Julian Sommer
|
b5198d55e2 | chore: enable depguard to block log | ||
|
Josh Siviour
|
fd3dc6b680 |
feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255)
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
|
||
|
tomnewton-mulga
|
28ca689130 |
feat: VFIO GPU passthrough (#188)
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |