mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-16 01:22:00 +00:00
Josh Siviour
b5879f44ff
* refactor: move telemetry file naming to contract
* docs: record telemetry contract move
* refactor: move aws xml response helpers
* docs: record aws xml helper move
* refactor: move aws tag projections to foundation
* docs: record aws tag projection move
* refactor: move account key to state foundation
* docs: record account key state move
* refactor: move host privilege escalation to runtime
* docs: record host privilege move
* chore: resolve lint findings from package-boundary moves
* refactor: move process supervision helpers to runtime host
* docs: record process supervision move
* refactor: move nbd endpoint helpers to compute runtime
* docs: record nbd endpoint move
* refactor: move image catalogue to operator
* docs: record image catalogue move
* refactor: move resource id generation to aws foundation
* docs: record resource id move
* refactor: move predastore role to runtime roles
* docs: point service manifest at acm domain
* docs: record predastore role move
* refactor: move northstar role to runtime roles
* docs: record northstar role move
* refactor: move awsgw role to runtime roles
* docs: record awsgw role move
* refactor: move error payload helpers to aws errors
* docs: record error payload move
* refactor: move account identifiers to aws identifiers
* docs: record account identifier move
* refactor: move NATS transport to messaging foundation
* docs: record NATS transport move
* refactor: move system instance contract to EC2 domain
* docs: record system instance move
* refactor: move system VPC builder to network domain
* docs: record system VPC move
* refactor: move instance metadata service to EC2 domain
* docs: record guest metadata move and dependency-rule debt
* refactor: move DNS control plane to DNS domain
* docs: record DNS domain move
* refactor: move base zone bootstrap to DNS domain
* docs: record base zone bootstrap move
* refactor: replace SystemAccountID with GlobalAccountID
* docs: record SystemAccountID removal
* refactor: extract CA key-pair loader to foundation pki
* docs: record foundation pki extraction
* refactor: move tenant CA paths to ACM domain
* docs: record tenant CA path move
* refactor: move IPsec certificate handling to network host
* docs: record IPsec certificate move
* refactor: move Ochre serving lifecycle to the Ochre domain
handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.
* docs: record Ochre domain move
* refactor: move Ochre vector store to the Ochre domain
handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.
* docs: record Ochre vector store move
* refactor: move Hugging Face Hub client to the Ochre domain
gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.
* docs: record Hugging Face Hub client move and Ochre gateway debt
* refactor: move EC2 tag mutation helpers to the EC2 domain
utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.
* docs: record EC2 tag mutation move and migrate split finding
* refactor: move network packages under the network domain
Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.
* docs: record network domain move
* refactor: move standalone EC2 resource packages to the EC2 domain
Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.
* docs: record standalone EC2 resource moves
* refactor: move EC2 VPC networking packages to the EC2 domain
Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.
* docs: record EC2 VPC networking moves
* refactor: move EC2 compute and storage packages to the EC2 domain
Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.
* docs: record EC2 compute and storage moves
* refactor: move ECR credential provider to the EKS agent tree
Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.
* docs: record ECR credential provider move
* refactor: move EKS gateway publish helper to the EKS agent tree
Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.
* docs: record EKS gateway publish move
* refactor: move EKS gateway fetch helper to the EKS agent tree
Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.
* docs: record EKS gateway fetch move
* refactor: move EKS webhook cert helper to the EKS agent tree
cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.
* docs: record EKS webhook cert move
* refactor: move EKS konnectivity cert helper to the EKS agent tree
cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.
* docs: record EKS konnectivity cert move
* refactor: move EKS token webhook to the EKS agent tree
cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.
* docs: record EKS token webhook move
* refactor: move ECS agent runtime and credentials packages to the ECS agent tree
* docs: record ECS agent subpackage move
* refactor: move ECS agent implementation to the ECS agent tree
cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.
* docs: record ECS agent move
* refactor: move RDS guest agent to agents/rds/agent
* docs: record RDS guest agent move
* refactor: move installer packages to operator/installer
* docs: record installer move
* refactor: move spx command tree to operator/cli
Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.
* docs: record spx command tree move
* chore: accept known vulnerabilities with an expiry
The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.
* refactor: move EC2 account action adapters to domains/ec2/awsapi
* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi
* refactor: move EC2 key action adapters to domains/ec2/awsapi
* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi
* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi
* docs: record EC2 standalone action adapter moves
* refactor: move EC2 zone action adapters to domains/ec2/awsapi
* refactor: move EC2 vpc action adapters to domains/ec2/awsapi
* refactor: move EC2 eigw action adapters to domains/ec2/awsapi
* refactor: move EC2 eip action adapters to domains/ec2/awsapi
* refactor: move EC2 igw action adapters to domains/ec2/awsapi
* refactor: move EC2 natgw action adapters to domains/ec2/awsapi
* refactor: move EC2 routetable action adapters to domains/ec2/awsapi
* docs: record EC2 VPC networking action adapter moves
* refactor: move EC2 image action adapters to domains/ec2/awsapi
* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi
* refactor: move EC2 volume action adapters to domains/ec2/awsapi
* refactor: move EC2 tags action adapters to domains/ec2/awsapi
* docs: record EC2 storage and tag action adapter moves
* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi
* refactor: move EC2 idempotency adapter to domains/ec2/awsapi
* refactor: move EC2 instance action adapters to domains/ec2/awsapi
* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi
* docs: record EC2 instance action adapter moves
* refactor: move AWS REST-JSON router to ingress/aws/rest
* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit
* docs: record AWS ingress helper moves and tooling conformance
* docs: correct historical lifecycle references
* test: characterize ECR control-plane dispatch before registration seam
* feat: add AWS ingress service registration contract
* docs: record ingress registration phases 0 and 1
* refactor: route registered AWS services through the ingress dispatch registry
The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.
The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.
* docs: record ingress dispatch registry consumption
* refactor: register the ECR control plane from its domain
domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.
Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.
* docs: record ECR control-plane registration
* fix: compose the ECR control plane in the integration harness
The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.
* docs: record ingress pipeline deferral
* refactor: extract AWS error envelope rendering to ingress
ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.
The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.
* docs: record error envelope extraction
* refactor: move AWS service selection into ingress dispatch
Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.
* docs: record ingress service selection move
* test: characterize ACM control-plane dispatch before registration
Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.
* refactor: register ACM control plane from its domain
domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.
* docs: record ACM registration
* docs: record ECS, EKS and RDS source inventory and first-resource ranking
* docs: record EKS control-plane secrets persisted in the cluster record
* chore: drop containerd advisory acceptance now that dev ships v2.4.1
* test: characterize RDS DB subnet group lifecycle before extraction
* docs: record RDS DB subnet group lifecycle contract
* refactor: extract the RDS DB subnet group lifecycle owner
domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.
* docs: record RDS DB subnet group owner extraction
* docs: correct RDS subnet group tagging debt and conformance scope
* test: characterize ECS task definition lifecycle before extraction
* docs: record ECS task definition lifecycle contract
* refactor: extract ECS task definition owner into domains/ecs/taskdefinition
Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.
The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.
List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.
* docs: record ECS task definition owner extraction
* test: characterize EKS access entry lifecycle before extraction
* docs: record EKS access entry lifecycle contract
* refactor: extract EKS access entry owner into domains/eks/access
Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.
Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.
List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.
* docs: record EKS access entry owner extraction
* test: characterize RDS engine catalogue before extraction
* fix: return InvalidParameterValue for a missing RDS master password
ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.
* docs: record RDS engine catalogue boundary and evidence
* refactor: extract RDS engine catalogue into domains/rds/engine
Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.
The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.
The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.
* docs: record RDS engine catalogue owner move
* test: characterize RDS DB parameter group lifecycle before extraction
* docs: record RDS DB parameter group lifecycle contract
* refactor: extract RDS DB parameter group owner
Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.
* docs: record RDS DB parameter group owner extraction
* test: characterize EKS add-on lifecycle and guest wire
Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.
* docs: record EKS add-on lifecycle contract
* fix: require iam:PassRole for EKS add-on service account role
CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.
* docs: record EKS add-on PassRole check
* refactor: extract EKS add-on owner
Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.
* docs: record EKS add-on owner extraction
* refactor: extract EKS add-on guest wire contract
Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.
* docs: record EKS add-on wire contract extraction
* docs: update EKS add-on inventory and remaining edges
* docs: draft EKS add-on lifecycle correction design for review
* chore: run contracts tests in the default test, coverage and race targets
* docs: apply review decisions to EKS add-on lifecycle design
* test: characterize EKS add-on v1 compatibility with added fields
Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.
* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
* docs: draft EKS add-on v2 guest contract design for review
* docs: qualify historical architecture references
* docs: apply review decisions and ADR-0006 to EKS add-on v2 design
* fix(eks): bind internal report publishing to the caller's cluster
PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.
PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.
* fix(eks): bind webhook token review to the caller's cluster
WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.
WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.
* fix(eks): validate add-on serviceAccountRoleArn syntax
CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.
The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.
* docs: record EKS internal-route and role-ARN security corrections
* docs: add EKS add-on release membership and activation case study
* fix: point UI build, CI and lint rules at moved package paths
* docs: record EKS add-on activation review decisions
* refactor: move quota admission to domains/admission/quota
Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.
* docs: reconcile package-boundary record for ADR-0001 close-out
Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.
* test: move KV migration engine tests to foundation/state/migrate
The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.
* refactor: move config and object-store upgrade workflow to operator/upgrade
Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.
* docs: record migrate move to operator/upgrade
* test: enforce ADR-0001 layering with a blocking ratchet
Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.
Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.
* refactor: drop the legacy handlers_ecr import alias
Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.
* chore: mark spf13/pflag as a direct dependency
operator/cli/wiring.go imports pflag directly. Same version; no upgrade.
* docs: record test-package-check and nilaway as unverified gates
* refactor: move the layering test to module-root architecture/
The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.
* fix: update paths left stale by the package moves
The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.
* refactor: move the default-VPC subject into contracts/ec2/v1
utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.
* docs: qualify the migrate split and record admin and Viperblock findings
The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.
* refactor: move operator-only admin code to operator/admin
Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.
foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.
* docs: record operator/admin split in package-boundary migration
* test: pin viperblockd fence, abandon and dirty-marker wire shapes
Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.
* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts
Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.
Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.
* refactor: split the Viperblock EBS adapter out of services/viperblockd
Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.
The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.
Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.
* refactor: move the viperblockd role to runtime/roles/viperblockd
With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.
* docs: record viperblockd contract, adapter and role moves
* test: characterise vpc.* network-realisation routes
Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.
Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).
* feat: add contracts/network/v1 for vpc.* realisation routes
Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.
* refactor: use contracts/network/v1 for vpc.* producers and consumers
Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.
Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.
No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.
* refactor: replace utils/vpcd_event.go with a typed network projection client
Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.
utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.
* refactor: route the remaining fire-and-forget vpc.* producers through projection
CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.
* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils
The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.
utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.
* docs: record the network contract, projection client and utils removal
* test: characterise instance-state persistence, recovery and generation handling
Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.
* test: freeze direct instance-state consumers
Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.
* docs: record ADR-0007 slice 0 instance-state inventory
Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.
* test: narrow the instance-state ratchet to instance-state access
Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.
* docs: record the narrowed instance-state ratchet commit
* docs: design the instance-state transition record and activation gate
* docs: record the instance-state transition-record design commit
* docs: record slice 1a activation-gate decisions
* feat: add instance transition-record codec with format marker
Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.
Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.
No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.
* docs: record slice 1b step 1 codec and awsgw rollout qualification
Records the codec added in e62f921d2 as design Section 10 (C1 to C7): its
surface, preservation at every object level, the changed-array limitation,
marker handling, the rejection classes and its capable-writers-only scope. C2
supersedes P1's top-level-only preservation for the codec. G19 records that
old read-only gateway consumers remain safe but DescribeInstanceStatus gains
unconfirmed/degraded visibility only after awsgw upgrades. The migration record
gains the commit row and states that no production path uses the codec yet.
* refactor: move the instance transition-record codec to domains/ec2/instance/state
The codec defines how the canonical persisted instance record evolves, with
its assignment and observation sections, which is EC2 instance ownership rather
than node-local VM supervision. It moves unchanged into its own package and
keeps wrapping the legacy vm.InstanceRecord alias and its spec and status
types, which stay in runtime/compute/vm for now.
The domain-to-runtime import is recorded as new dependency-rule debt with its
removal condition and allow-listed against that entry. The instance-state
ratchet now names the new file as an owner file; its allowlist is unchanged.
* fix: refuse array changes that would drop element extensions
A changed array is replaced whole by the writer's value, which silently lost
any member a newer writer had added inside its elements. Encode now inspects
the array as read: if any element carries a member its typed schema does not
know, at any depth, it returns ErrRecordExtensionConflict naming the array and
the first such member. Arrays without such members are still replaced whole,
unchanged arrays are still kept verbatim, and arrays inside unknown members are
never inspected.
* docs: record the codec move and the array extension-conflict rule
Records 8db9771fb and c617914a1. T1 and Section 10 now name
domains/ec2/instance/state as the eventual owner of the persisted instance
record and its evolution, with vm.InstanceRecord as the temporary legacy type
it wraps. C3 replaces the changed-array limitation with the refusal: a changed
array with element extensions fails with ErrRecordExtensionConflict and is never
silently dropped. P2 stays a conservative rule until finalisation.
* fix: refuse removals that would drop record extension fields
A capable writer may no longer remove or replace a known object, map entry or
array when its stored value carries members the writer's schema does not know.
The check that guarded changed arrays now runs wherever the merge drops or
overwrites a stored value instead of merging into it: a known member deleted or
set to null, a map entry removed or stored under a key the typed decode
rewrote, a value of another JSON kind, and a changed array. Encode returns
ErrRecordExtensionConflict naming the changed container and the first unknown
member. Removal stays allowed when the stored value has no extensions, and a
change merged member by member still keeps unknown members.
* docs: record the uniform extension-conflict rule
Adds design clause C8 for 0d7b4266d: a capable writer may not remove or
replace a known object, map entry or array whose stored value carries unknown
structured extension fields; removal without extensions stays allowed. C3 and
C5 point at it. The migration record gains the commit row, updated subsection
wording and a pointer-bump follow-up deferring make docs-portal to the Mulga
gitlink PR so the portal evaluates the proposed Spinifex revision.
* refactor: inject vpcd's IMDS instance-state readers from the instance state boundary
* docs: cite the vpcd reader injection commit in the migration record
* refactor: move vpcd to runtime/roles/vpcd
* docs: cite the vpcd move commit in the migration record
* refactor: move the ECS agent wire vocabulary to contracts/ecs/v1
Move handlers/ecs/bus to contracts/ecs/v1 (package ecsv1) with no legacy
shim. Subject strings, JSON fields and types are byte-identical; the old-agent
compatibility test moves unchanged and golden pins cover the remaining
message types. The README route table records that only service-reconcile
has a live producer today; the agent reuses these payload types through the
gateway actions. service-interfaces.yaml lists ecs.bus.* under the daemon.
* refactor: move system-image promotion to domains/ec2/image
Move images_promote.go and its tests from admin to domains/ec2/image:
SystemOwnerAlias, PromoteImageOpts/PromoteImageResult, PromoteSystemImage,
GetAMIMetadata and the private snapshot reader. Storage, ordering, errors and
promotion behaviour are unchanged; the daemon, gateway/spx and operator CLI
callers import ec2image. admin now holds only account IDs and key generators.
* docs: record the ECS contract and image-promotion moves and the remaining-roots matrix
* refactor: move the bootstrap account identifiers to foundation/aws/identifiers
DefaultAccountID and DefaultAccountName join GlobalAccountID as the fixed,
platform-reserved identity of the first human-facing bootstrap account.
Names and values are unchanged and stay fixed. Callers in daemon, gateway,
awsgw, accountteardown and the operator CLI use awsidentifiers; admin now
holds only the IAM credential generators.
* docs: cite the bootstrap account identifier move in the migration record
* refactor: inject the STS cluster JWKS lookup from handlers/eks
handlers/sts declares ClusterJWKSResolver and keeps JWKS decoding and
InvalidIdentityToken mapping; handlers/eks implements it as
OIDCJWKSReader beside the bucket and key layout helpers. awsgw and the
integration harness inject the reader. STS no longer imports any EKS
package, including tests, and the handlers/sts -> handlers/eks layering
exception is removed.
* docs: cite the STS JWKS resolver cut in the migration record
* refactor: move the AWS credential-format generators to foundation/aws/credentials and retire admin
GenerateAWSAccessKey and GenerateAWSSecretKey move unchanged to
foundation/aws/credentials as a stateless format primitive. handlers/iam
and operator/cli import it directly. The admin legacy root is deleted
and joins the retired roots so it cannot return.
* docs: cite the admin retirement in the migration record
* revert: remove the uncalled ADR-0007 codec and design stack from the ADR-0001 checkpoint
The instance transition-record codec, its tests and the slice 1a design
had no production caller. They leave this package-boundary checkpoint
and will be reapplied on a dedicated ADR-0007 branch. Slice 0
characterisation and the vpcd IMDS readers in
domains/ec2/instance/state stay.
* docs: cite the ADR-0007 codec revert in the migration record
* fix: declare the ECS action subjects in service-interfaces.yaml
Declaring ecs.bus.* made ecs a managed prefix for manifest-lint, which then
failed on the 34 ECS action subjects the daemon registers. Declare them as
ecs.*, the same wildcard convention as rds.*.
* chore: run manifest-check and manifest-lint in preflight
CI runs both, but preflight did not, so manifest drift could pass local
checks and still fail Build and Test.
* docs: record resumed preflight enforcement and dev merge ports
* fix: restore dev changes dropped by earlier merges
Reapplies the slices.ContainsFunc rewrite in nodeFitsCustomerInstance and the DeleteDBSubnetGroup doc comment, both lost when their files were refactored.
* docs: record accountteardown as a frozen legacy root pending cleanup capabilities
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
1662 lines
75 KiB
Bash
Executable File
1662 lines
75 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
# OVN Compute Node Setup for Spinifex VPC Networking
|
|
#
|
|
# This script bootstraps a compute node for OVN-based VPC networking:
|
|
# 1. Installs OVN/OVS packages (if not present)
|
|
# 2. Enables required services (openvswitch-switch, ovn-controller)
|
|
# 3. Creates br-int with secure fail-mode
|
|
# 4. Configures WAN bridge for public subnet uplink (auto-detected or manual)
|
|
# 5. Configures OVS external_ids for OVN chassis identity
|
|
# 6. Applies sysctl tuning for overlay networking
|
|
#
|
|
# Usage:
|
|
# ./scripts/setup-ovn.sh [options]
|
|
#
|
|
# Options:
|
|
# --management Also start OVN central services (NB DB, SB DB, ovn-northd)
|
|
# --wan-bridge=NAME OVS bridge for WAN traffic (default: auto-detect from default route)
|
|
# --wan-iface=NAME Physical NIC to add to the WAN bridge (use with --wan-bridge)
|
|
# --dhcp Obtain gateway IP via DHCP on the WAN bridge interface
|
|
# --nat-uplink Routed NAT mode: no WAN NIC is bridged. Creates br-ext
|
|
# with a transit veth (spx-nat-host 100.127.0.1/24) and
|
|
# host masquerade rules; VMs reach the WAN over any
|
|
# uplink (ethernet, WiFi, cellular, PPP), and EIPs work
|
|
# once a public pool is configured. This is the only
|
|
# mode that works on a cloud VNIC. Pair with
|
|
# `spx admin init --external-mode=nat`.
|
|
# --mgmt-bridge=NAME OVS bridge for system-instance control plane (default: br-mgmt)
|
|
# --mgmt-cidr=CIDR IPv4 CIDR to assign on the mgmt bridge (default: 10.15.8.1/24)
|
|
# --mgmt-iface=NAME Physical/virtual NIC to enslave to the mgmt bridge (multi-node only)
|
|
# --no-mgmt-bridge Skip mgmt bridge provisioning (for dev-networking hosts)
|
|
# --ovn-remote=ADDR OVN SB DB address; accepts a comma-separated list of
|
|
# SB endpoints for compute nodes pointing at a RAFT
|
|
# cluster (default: tcp:127.0.0.1:6642)
|
|
# --encap-ip=IP Geneve tunnel endpoint IP (default: auto-detect)
|
|
# --lan-addr=IP LAN-plane IP to bind the OVN NB(6641)/SB(6642)
|
|
# client listeners on, in addition to loopback
|
|
# (requires --management). Falls back to
|
|
# --db-cluster-local-addr when omitted, since every
|
|
# clustered node already passes its own LAN IP.
|
|
# --db-cluster-local-addr=IP This DB node's own IP for OVSDB RAFT clustering.
|
|
# Enables clustered NB(6643)/SB(6644) DBs (requires
|
|
# --management). Empty --db-cluster-remote-addr ⇒ create
|
|
# the cluster; set ⇒ join it.
|
|
# --db-cluster-remote-addr=IP An existing cluster DB node's IP to join. Omit
|
|
# on the first (init) DB node that forms the cluster.
|
|
# --recreate-db Destroy and recreate the NB/SB DBs so they can be
|
|
# created in clustered format. Required when converting
|
|
# a node that has already run a standalone ovn-central
|
|
# (which is every node — the ovn-central package starts
|
|
# one on install). DISCARDS ALL LOGICAL NETWORK STATE:
|
|
# safe on a fresh node, destroys every VPC on a live one.
|
|
# --teardown Remove the host wiring this script builds — veth pairs,
|
|
# systemd-networkd persistence, routed-NAT egress rules,
|
|
# the OVS bridges and the ovn-* external_ids — and exit.
|
|
# The way back to a node this script can be re-run on
|
|
# cleanly, because a second run only unpicks the mode it
|
|
# is switching away from. Leaves the NB/SB databases and
|
|
# every VPC in them, /etc/spinifex, and the system-id;
|
|
# erasing those is node-reset.sh and uninstall-spx.sh.
|
|
# --dry-run Print what --teardown would do and change nothing.
|
|
#
|
|
# Cluster Sizing — which DB topology to run:
|
|
#
|
|
# Nodes DB topology Tolerates
|
|
# 1-2 standalone ovn-central on node 1 nothing
|
|
# 3 RAFT across all 3 1 node
|
|
# 4+ RAFT across 3, remainder compute-only 1 node
|
|
#
|
|
# Three nodes is the minimum recommended multi-server deployment.
|
|
#
|
|
# NEVER run RAFT with 2 members. Quorum is a majority, so 2-of-2 tolerates
|
|
# zero failures — and it is strictly worse than standalone, because either
|
|
# node failing loses quorum rather than only node 1. Two-node deployments
|
|
# run standalone and accept the single point of failure.
|
|
#
|
|
# Do not scale DB members past 3. RAFT write latency is bounded by the
|
|
# slowest member of the majority, and a 4th member buys no extra fault
|
|
# tolerance. Nodes beyond the third are compute-only, pointing at all three
|
|
# SB endpoints via the comma-separated --ovn-remote form.
|
|
#
|
|
# Losing ovn-central does not stop the data plane: ovn-controller has already
|
|
# programmed flows into br-int, so running instances keep networking. Only
|
|
# control-plane change stops — new VPCs, instance launches, port bindings.
|
|
#
|
|
# WAN Bridge Auto-Detection:
|
|
# When no --wan-bridge is given, the script checks the default route interface:
|
|
# - If it's an OVS bridge → use it directly for bridge-mappings
|
|
# - If it's a Linux bridge → create OVS br-ext + veth pair to link them
|
|
# (non-destructive, Linux bridge keeps IP/routes, no interruption)
|
|
# - If it's a physical NIC → stop and print guidance (cannot safely move NIC)
|
|
#
|
|
# Examples:
|
|
# # WAN is already on a bridge (tofu-cluster, production):
|
|
# ./scripts/setup-ovn.sh --management
|
|
#
|
|
# # Dedicated WAN NIC (not your SSH NIC — you take responsibility):
|
|
# ./scripts/setup-ovn.sh --management --wan-bridge=br-wan --wan-iface=eth1
|
|
#
|
|
# # Compute node joining an existing cluster:
|
|
# ./scripts/setup-ovn.sh --ovn-remote=tcp:10.0.0.1:6642 --encap-ip=10.0.0.2
|
|
#
|
|
# # DB node 1 — create an OVSDB RAFT cluster (init):
|
|
# ./scripts/setup-ovn.sh --management --db-cluster-local-addr=10.0.0.1
|
|
#
|
|
# # DB nodes 2,3 — join the cluster:
|
|
# ./scripts/setup-ovn.sh --management --db-cluster-local-addr=10.0.0.2 \
|
|
# --db-cluster-remote-addr=10.0.0.1
|
|
#
|
|
# # Compute node pointing at all 3 clustered SB endpoints:
|
|
# ./scripts/setup-ovn.sh --ovn-remote=tcp:10.0.0.1:6642,tcp:10.0.0.2:6642,tcp:10.0.0.3:6642
|
|
#
|
|
# # No WAN bridge (overlay-only, no public subnet):
|
|
# ./scripts/setup-ovn.sh --management --encap-ip=10.0.0.1
|
|
#
|
|
# # Non-bridgeable uplink (WiFi/cellular) — routed NAT:
|
|
# ./scripts/setup-ovn.sh --management --nat-uplink
|
|
#
|
|
# # Standalone management node, NB/SB client listeners also on the LAN plane:
|
|
# ./scripts/setup-ovn.sh --management --lan-addr=10.0.0.1
|
|
|
|
set -e
|
|
|
|
# Defaults
|
|
MANAGEMENT=false
|
|
WAN_BRIDGE=""
|
|
WAN_IFACE=""
|
|
EXTERNAL_DHCP=false
|
|
NAT_UPLINK=false
|
|
MGMT_BRIDGE_ENABLED=true
|
|
MGMT_BRIDGE="br-mgmt"
|
|
MGMT_CIDR="10.15.8.1/24"
|
|
MGMT_IFACE=""
|
|
OVN_REMOTE="tcp:127.0.0.1:6642"
|
|
ENCAP_IP=""
|
|
# LAN-plane IP for the NB/SB client listeners (6641/6642), in addition to
|
|
# loopback. Empty means loopback-only; resolved further down against
|
|
# DB_CLUSTER_LOCAL_ADDR once arguments are parsed.
|
|
LAN_ADDR=""
|
|
# OVSDB RAFT clustering. Empty by default ⇒ single, non-clustered ovn-central
|
|
# (dev box, existing single-node clusters). When LOCAL_ADDR is set, the NB/SB
|
|
# DBs run clustered; REMOTE_ADDR empty ⇒ create the cluster, set ⇒ join it.
|
|
DB_CLUSTER_LOCAL_ADDR=""
|
|
DB_CLUSTER_REMOTE_ADDR=""
|
|
# Recreating the NB/SB DBs discards all logical network state, so it is opt-in.
|
|
RECREATE_DB=false
|
|
OVN_DBDIR="${OVN_DBDIR:-/var/lib/ovn}"
|
|
# NODE_NAME is left empty by default. The chassis-id pin block at Step 4
|
|
# only runs when --node-name=NAME is explicitly given. Passing nothing
|
|
# preserves whatever system-id already lives in OVS (gold-image UUID,
|
|
# bootstrap-install.sh's pre-written nodeN, manual ansible value, etc.).
|
|
# Bootstrap callers that want IPsec-cert-identity matching pin the
|
|
# system-id themselves before / after invoking setup-ovn.sh — they
|
|
# don't need this script to do it for them.
|
|
NODE_NAME=""
|
|
# --teardown removes the host wiring this script builds, so a re-run starts
|
|
# from a clean node instead of compounding whatever the last run left.
|
|
TEARDOWN=false
|
|
DRY_RUN=false
|
|
|
|
# Parse arguments
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--management) MANAGEMENT=true ;;
|
|
--dhcp) EXTERNAL_DHCP=true ;;
|
|
--nat-uplink) NAT_UPLINK=true ;;
|
|
--wan-bridge=*) WAN_BRIDGE="${arg#*=}" ;;
|
|
--wan-iface=*) WAN_IFACE="${arg#*=}" ;;
|
|
--mgmt-bridge=*) MGMT_BRIDGE="${arg#*=}" ;;
|
|
--mgmt-cidr=*) MGMT_CIDR="${arg#*=}" ;;
|
|
--mgmt-iface=*) MGMT_IFACE="${arg#*=}" ;;
|
|
--no-mgmt-bridge) MGMT_BRIDGE_ENABLED=false ;;
|
|
--ovn-remote=*) OVN_REMOTE="${arg#*=}" ;;
|
|
--encap-ip=*) ENCAP_IP="${arg#*=}" ;;
|
|
--lan-addr=*) LAN_ADDR="${arg#*=}" ;;
|
|
--db-cluster-local-addr=*) DB_CLUSTER_LOCAL_ADDR="${arg#*=}" ;;
|
|
--db-cluster-remote-addr=*) DB_CLUSTER_REMOTE_ADDR="${arg#*=}" ;;
|
|
--recreate-db) RECREATE_DB=true ;;
|
|
--node-name=*) NODE_NAME="${arg#*=}" ;;
|
|
--teardown) TEARDOWN=true ;;
|
|
--dry-run) DRY_RUN=true ;;
|
|
--help|-h)
|
|
sed -n '3,/^set -e/{/^set -e/!p}' "$0"
|
|
exit 0
|
|
;;
|
|
*)
|
|
echo "Unknown option: $arg"
|
|
exit 1
|
|
;;
|
|
esac
|
|
done
|
|
|
|
# --- Teardown ---
|
|
#
|
|
# Removes the host wiring this script builds and nothing else. A node that has
|
|
# been set up twice with different --wan-bridge values carries both shapes at
|
|
# once, and the second run cannot unpick the first: it rips down only the mode
|
|
# it is switching away from, and only when it gets far enough to know the mode.
|
|
# This is the way back to a node the script can be run on cleanly.
|
|
#
|
|
# What it deliberately leaves: the NB/SB databases and every logical network in
|
|
# them, /etc/spinifex, the OVS system-id, and the packages. Erasing those is
|
|
# node-reset.sh and uninstall-spx.sh, and a teardown that took the control
|
|
# plane's state with it could not be used to re-run this script.
|
|
run() {
|
|
if [ "$DRY_RUN" = true ]; then
|
|
echo " would run: $*"
|
|
return 0
|
|
fi
|
|
"$@"
|
|
}
|
|
|
|
# True when $1 names an existing Linux (not OVS) bridge.
|
|
is_linux_bridge() {
|
|
[ -d "/sys/class/net/$1/bridge" ]
|
|
}
|
|
|
|
# Prints the bridge $1 is enslaved to, or nothing.
|
|
iface_master() {
|
|
local m
|
|
m=$(ip -o link show "$1" 2>/dev/null | sed -n 's/.* master \([^ ]*\) .*/\1/p')
|
|
printf '%s' "$m"
|
|
}
|
|
|
|
teardown() {
|
|
echo "Tearing down the OVN host wiring built by this script"
|
|
[ "$DRY_RUN" = true ] && echo " (dry run — nothing is changed)"
|
|
echo ""
|
|
|
|
echo "Step 1: Removing veth pairs..."
|
|
# Deleting one end deletes the peer, so the OVS port has to go first or it
|
|
# is left in OVSDB naming a device the kernel no longer has.
|
|
for pair in "veth-wan-ovs veth-wan-br" "spx-nat-ovs spx-nat-host"; do
|
|
set -- $pair
|
|
if command -v ovs-vsctl >/dev/null 2>&1; then
|
|
run sudo ovs-vsctl --if-exists del-port "$1" 2>/dev/null || true
|
|
fi
|
|
if ip link show "$2" >/dev/null 2>&1; then
|
|
run sudo ip link del "$2" 2>/dev/null || true
|
|
echo " removed veth pair: $2 ↔ $1"
|
|
fi
|
|
done
|
|
|
|
echo ""
|
|
echo "Step 2: Removing systemd-networkd persistence..."
|
|
for f in /etc/systemd/network/05-spinifex-ovs-internal.network \
|
|
/etc/systemd/network/10-spinifex-mgmt.network \
|
|
/etc/systemd/network/14-spinifex-br-wan.netdev \
|
|
/etc/systemd/network/15-spinifex-veth-wan.netdev \
|
|
/etc/systemd/network/15-spinifex-veth-wan.network \
|
|
/etc/systemd/network/16-spinifex-veth-wan-ovs.network \
|
|
/etc/systemd/network/17-spinifex-nat.netdev \
|
|
/etc/systemd/network/17-spinifex-nat.network \
|
|
/etc/systemd/network/18-spinifex-nat-ovs.network; do
|
|
if [ -f "$f" ]; then
|
|
run sudo rm -f "$f"
|
|
echo " removed $f"
|
|
fi
|
|
done
|
|
run sudo networkctl reload 2>/dev/null || true
|
|
|
|
echo ""
|
|
echo "Step 3: Removing routed-NAT egress rules..."
|
|
# Drain by comment rather than by rule text: the rules were built from a
|
|
# CIDR and an uplink this run may not know, and deleting by spec would miss
|
|
# any installed against a different uplink.
|
|
if command -v iptables >/dev/null 2>&1; then
|
|
for table_chain in "nat POSTROUTING" "filter FORWARD"; do
|
|
set -- $table_chain
|
|
while iptables -t "$1" -S "$2" 2>/dev/null | grep -q 'spinifex-nat-egress'; do
|
|
rule=$(iptables -t "$1" -S "$2" 2>/dev/null | grep -m1 'spinifex-nat-egress' | sed 's/^-A /-D /')
|
|
[ -n "$rule" ] || break
|
|
# shellcheck disable=SC2086
|
|
run sudo iptables -t "$1" $rule 2>/dev/null || break
|
|
echo " removed $1/$2 rule (spinifex-nat-egress)"
|
|
[ "$DRY_RUN" = true ] && break
|
|
done
|
|
done
|
|
fi
|
|
|
|
echo ""
|
|
echo "Step 4: Removing OVS bridges and external_ids..."
|
|
if command -v ovs-vsctl >/dev/null 2>&1; then
|
|
# br-ext and br-int are this script's; a bridge the operator named with
|
|
# --wan-bridge is removed too, unless it is a Linux bridge, which this
|
|
# script never created and must not delete.
|
|
for br in br-ext br-int "$MGMT_BRIDGE" "$WAN_BRIDGE"; do
|
|
[ -n "$br" ] || continue
|
|
if is_linux_bridge "$br"; then
|
|
echo " left $br alone: it is a Linux bridge, not ours to delete"
|
|
continue
|
|
fi
|
|
if sudo ovs-vsctl br-exists "$br" 2>/dev/null; then
|
|
run sudo ovs-vsctl --if-exists del-br "$br"
|
|
echo " removed OVS bridge: $br"
|
|
fi
|
|
done
|
|
for key in ovn-bridge-mappings ovn-remote ovn-encap-ip ovn-encap-type; do
|
|
run sudo ovs-vsctl remove Open_vSwitch . external_ids "$key" 2>/dev/null || true
|
|
done
|
|
echo " cleared ovn-bridge-mappings, ovn-remote, ovn-encap-ip, ovn-encap-type"
|
|
fi
|
|
|
|
echo ""
|
|
echo "Teardown complete. The NB/SB databases, /etc/spinifex and the OVS"
|
|
echo "system-id are untouched — re-run this script to wire the node again."
|
|
}
|
|
|
|
if [ "$TEARDOWN" = true ]; then
|
|
teardown
|
|
exit 0
|
|
fi
|
|
|
|
# OVSDB RAFT clustering runs the NB/SB DBs, so it only applies to management
|
|
# (DB) nodes. A remote addr without a local addr is meaningless (nothing to
|
|
# join with). Fail loudly rather than silently starting a single-node DB.
|
|
if [ -n "$DB_CLUSTER_LOCAL_ADDR" ] && [ "$MANAGEMENT" != true ]; then
|
|
echo "ERROR: --db-cluster-local-addr requires --management (clustered DBs run on management nodes)"
|
|
exit 1
|
|
fi
|
|
if [ -n "$DB_CLUSTER_REMOTE_ADDR" ] && [ -z "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
echo "ERROR: --db-cluster-remote-addr requires --db-cluster-local-addr"
|
|
exit 1
|
|
fi
|
|
if [ "$RECREATE_DB" = true ] && [ -z "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
echo "ERROR: --recreate-db requires --db-cluster-local-addr (it exists to allow"
|
|
echo " clustered DB creation over an existing standalone DB)"
|
|
exit 1
|
|
fi
|
|
|
|
# Fall back to the RAFT clustering address when no --lan-addr was given:
|
|
# every clustered node already passes --db-cluster-local-addr with its LAN
|
|
# IP, so clusters stay correct even from a caller that predates this flag.
|
|
if [ -z "$LAN_ADDR" ] && [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
LAN_ADDR="$DB_CLUSTER_LOCAL_ADDR"
|
|
fi
|
|
|
|
# --- WAN bridge auto-detection ---
|
|
# Determine the WAN bridge name and how to set it up.
|
|
WAN_BRIDGE_MODE="" # "existing", "veth", "direct", or ""
|
|
LINUX_BRIDGE="" # Set when WAN_BRIDGE_MODE="veth" — the Linux bridge behind the veth pair
|
|
|
|
detect_wan_bridge() {
|
|
# If --wan-bridge was explicitly given, use it
|
|
if [ -n "$WAN_BRIDGE" ]; then
|
|
# A Linux bridge of this name is the veth case, not the OVS-bridge case,
|
|
# and it is checked first because it is the only arm that can silently
|
|
# produce a broken datapath. An OVS bridge cannot take over the name of
|
|
# an existing Linux bridge: ovs-vsctl exits 0, OVSDB accepts the rows,
|
|
# and ofproto then refuses with "could not add network device br-wan to
|
|
# ofproto (File exists)" — visible only in `ovs-vsctl show`. The
|
|
# auto-detect path below has always handled this correctly by building a
|
|
# veth pair; it was simply unreachable whenever the operator named the
|
|
# bridge, which is exactly what they must do when the default route is
|
|
# on a physical NIC.
|
|
if is_linux_bridge "$WAN_BRIDGE"; then
|
|
LINUX_BRIDGE="$WAN_BRIDGE"
|
|
WAN_BRIDGE="br-ext"
|
|
WAN_BRIDGE_MODE="veth"
|
|
echo " $LINUX_BRIDGE is a Linux bridge — linking it to OVS br-ext via a veth pair"
|
|
if [ -n "$WAN_IFACE" ]; then
|
|
echo " ignoring --wan-iface=$WAN_IFACE: $LINUX_BRIDGE already owns its ports"
|
|
WAN_IFACE=""
|
|
fi
|
|
return
|
|
fi
|
|
|
|
if [ -n "$WAN_IFACE" ]; then
|
|
# Enslaving an interface that another bridge already owns fails the
|
|
# same silent way. Refuse rather than build something that reports
|
|
# success and cannot forward.
|
|
local master
|
|
master=$(iface_master "$WAN_IFACE")
|
|
if [ -n "$master" ]; then
|
|
echo "ERROR: $WAN_IFACE is already enslaved to bridge '$master'."
|
|
echo " Adding it to an OVS bridge as well cannot work — ovs-vsctl will"
|
|
echo " exit 0 and the datapath will refuse the device."
|
|
echo " Either detach it first, or point --wan-bridge at '$master' to"
|
|
echo " link that bridge to OVS with a veth pair:"
|
|
echo " $0 --management --wan-bridge=$master"
|
|
exit 1
|
|
fi
|
|
WAN_BRIDGE_MODE="direct"
|
|
elif sudo ovs-vsctl br-exists "$WAN_BRIDGE" 2>/dev/null; then
|
|
WAN_BRIDGE_MODE="existing"
|
|
else
|
|
# Bridge doesn't exist yet and no --wan-iface — create empty OVS bridge
|
|
WAN_BRIDGE_MODE="existing"
|
|
fi
|
|
return
|
|
fi
|
|
|
|
# Auto-detect: find the default route interface
|
|
local default_dev
|
|
default_dev=$(ip -4 route show default 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1)}' | head -1)
|
|
|
|
if [ -z "$default_dev" ]; then
|
|
echo " No default route found — no WAN bridge configured"
|
|
echo " (VMs will not have external connectivity)"
|
|
return
|
|
fi
|
|
|
|
# Check if the default route device is a bridge (Linux or OVS)
|
|
local is_bridge=false
|
|
if ip -d link show "$default_dev" 2>/dev/null | grep -q "bridge"; then
|
|
is_bridge=true
|
|
fi
|
|
if sudo ovs-vsctl br-exists "$default_dev" 2>/dev/null; then
|
|
is_bridge=true
|
|
fi
|
|
|
|
if [ "$is_bridge" = true ]; then
|
|
if sudo ovs-vsctl br-exists "$default_dev" 2>/dev/null; then
|
|
# Already an OVS bridge — use it directly for bridge-mappings
|
|
WAN_BRIDGE="$default_dev"
|
|
WAN_BRIDGE_MODE="existing"
|
|
echo " Auto-detected WAN bridge: $WAN_BRIDGE (OVS bridge, default route)"
|
|
else
|
|
# Linux bridge — link to OVS via veth pair (non-destructive)
|
|
LINUX_BRIDGE="$default_dev"
|
|
WAN_BRIDGE="br-ext"
|
|
WAN_BRIDGE_MODE="veth"
|
|
echo " Auto-detected Linux bridge: $LINUX_BRIDGE (default route)"
|
|
echo " Will create OVS bridge br-ext + veth pair to link them"
|
|
fi
|
|
return
|
|
fi
|
|
|
|
# Default route is a physical NIC — cannot safely move it to OVS
|
|
# because it might be carrying SSH.
|
|
local wan_ip
|
|
wan_ip=$(ip -4 -o addr show "$default_dev" 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)
|
|
|
|
echo ""
|
|
echo "============================================================"
|
|
echo " WAN interface '$default_dev' ($wan_ip) is a physical NIC."
|
|
echo " Cannot auto-create a bridge — this may drop your connection."
|
|
echo ""
|
|
echo " Options:"
|
|
echo ""
|
|
echo " 1. Create a WAN bridge first (e.g. via netplan), then re-run:"
|
|
echo " ./scripts/setup-ovn.sh --management"
|
|
echo ""
|
|
echo " 2. Dedicated WAN NIC (NOT your SSH connection):"
|
|
echo " ./scripts/setup-ovn.sh --management --wan-bridge=br-wan --wan-iface=$default_dev"
|
|
echo ""
|
|
echo " 3. No external networking (overlay-only):"
|
|
echo " ./scripts/setup-ovn.sh --management --encap-ip=$wan_ip"
|
|
echo ""
|
|
echo " 4. Non-bridgeable uplink (WiFi/cellular/PPP) — routed NAT mode"
|
|
echo " (add a public pool for EIPs; without one, VMs get egress only):"
|
|
echo " ./scripts/setup-ovn.sh --management --nat-uplink"
|
|
echo " then: spx admin init --external-mode=nat"
|
|
echo "============================================================"
|
|
echo ""
|
|
exit 1
|
|
}
|
|
|
|
if [ "$NAT_UPLINK" = true ]; then
|
|
# Routed NAT: nothing is bridged, so the uplink type is irrelevant.
|
|
if [ -n "$WAN_BRIDGE" ] || [ -n "$WAN_IFACE" ]; then
|
|
echo "ERROR: --nat-uplink does not take --wan-bridge/--wan-iface (no WAN NIC is bridged)"
|
|
exit 1
|
|
fi
|
|
WAN_BRIDGE="br-ext"
|
|
WAN_BRIDGE_MODE="nat"
|
|
echo " Routed NAT uplink: br-ext + transit veth, host masquerade (no WAN bridge)"
|
|
else
|
|
detect_wan_bridge
|
|
fi
|
|
|
|
# Auto-detect encap IP if not specified
|
|
if [ -z "$ENCAP_IP" ]; then
|
|
# Prefer br-vpc IP if it exists (dedicated VPC data plane)
|
|
if ip -4 addr show br-vpc >/dev/null 2>&1; then
|
|
ENCAP_IP=$(ip -4 -o addr show br-vpc 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)
|
|
if [ -n "$ENCAP_IP" ]; then
|
|
echo "Auto-detected encap IP from br-vpc: $ENCAP_IP"
|
|
fi
|
|
fi
|
|
# Fall back to default route source IP
|
|
if [ -z "$ENCAP_IP" ]; then
|
|
ENCAP_IP=$(ip -4 route get 8.8.8.8 2>/dev/null | awk '/src/{print $7}' | head -1)
|
|
if [ -z "$ENCAP_IP" ]; then
|
|
ENCAP_IP="127.0.0.1"
|
|
fi
|
|
echo "Auto-detected encap IP: $ENCAP_IP"
|
|
fi
|
|
fi
|
|
|
|
echo "=== Spinifex OVN Compute Node Setup ==="
|
|
echo " Management node: $MANAGEMENT"
|
|
if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
if [ -n "$DB_CLUSTER_REMOTE_ADDR" ]; then
|
|
echo " DB RAFT role: join (local=$DB_CLUSTER_LOCAL_ADDR remote=$DB_CLUSTER_REMOTE_ADDR)"
|
|
else
|
|
echo " DB RAFT role: create (local=$DB_CLUSTER_LOCAL_ADDR)"
|
|
fi
|
|
fi
|
|
if [ -n "$WAN_BRIDGE" ]; then
|
|
echo " WAN bridge: $WAN_BRIDGE ($WAN_BRIDGE_MODE)"
|
|
if [ -n "$LINUX_BRIDGE" ]; then
|
|
echo " Linux bridge: $LINUX_BRIDGE (linked via veth pair)"
|
|
fi
|
|
if [ -n "$WAN_IFACE" ]; then
|
|
echo " WAN interface: $WAN_IFACE"
|
|
fi
|
|
else
|
|
echo " WAN bridge: none (overlay-only)"
|
|
fi
|
|
echo " OVN Remote (SB): $OVN_REMOTE"
|
|
echo " Encap IP: $ENCAP_IP"
|
|
echo ""
|
|
|
|
# Packages (openvswitch-switch, ovn-host, openvswitch-ipsec, strongswan-charon,
|
|
# and ovn-central on management nodes) are baked into the gold image via
|
|
# scripts/tofu-cluster/image-builder/scripts/provision.sh. Runtime apt-get
|
|
# was removed to keep CI off the (flaky) apt-cacher-ng path. If a package is
|
|
# missing, the downstream ovs/ovn commands will fail loudly — the fix is to
|
|
# rebuild the gold image, not to re-add apt-get here.
|
|
|
|
# strongswan-charon ships an AppArmor profile for /usr/lib/ipsec/charon that
|
|
# only allows reading from /etc/ipsec.*, /etc/strongswan.*, and a few other
|
|
# fixed paths. ovs-monitor-ipsec writes the per-tunnel strongSwan config with
|
|
# absolute paths to our peer cert + key under /etc/spinifex/ipsec/ AND to the
|
|
# cluster CA at /etc/spinifex/ca.pem, so charon hits "Permission denied" on
|
|
# both load paths and surfaces as `no trusted RSA public key found for
|
|
# '<peer>'` (CA can't be loaded → can't validate peer cert chain → no SAs).
|
|
# The profile includes a 'local' override file expressly for site additions
|
|
# — grant reads on /etc/spinifex/** (covers ca.pem AND ipsec/peer.{pem,key})
|
|
# and reload.
|
|
if [ -d /etc/apparmor.d/local ]; then
|
|
LOCAL_OVERRIDE=/etc/apparmor.d/local/usr.lib.ipsec.charon
|
|
# Always rewrite — re-runs may have an older, narrower rule (e.g. the
|
|
# initial /etc/spinifex/ipsec/** only) that misses /etc/spinifex/ca.pem.
|
|
echo " Adding AppArmor read grant for /etc/spinifex (cert + CA) to charon profile"
|
|
echo "/etc/spinifex/** r," | sudo tee "$LOCAL_OVERRIDE" >/dev/null
|
|
sudo apparmor_parser -r /etc/apparmor.d/usr.lib.ipsec.charon 2>/dev/null || true
|
|
fi
|
|
|
|
# --- Step 2: Enable services ---
|
|
# ovn-ctl consults the RAFT flags only when it creates a database. The
|
|
# ovn-central package starts a standalone ovsdb-server on install, so a
|
|
# standalone-format DB is always already present by the time this script first
|
|
# runs: without this check ovn-ctl serves that DB, silently ignores the cluster
|
|
# flags, and the script reports success on a cluster that was never formed.
|
|
ensure_clustered_db_storage() {
|
|
local db
|
|
local standalone=()
|
|
for db in "$OVN_DBDIR/ovnnb_db.db" "$OVN_DBDIR/ovnsb_db.db"; do
|
|
if [ -f "$db" ] && ! sudo ovsdb-tool db-is-clustered "$db" 2>/dev/null; then
|
|
standalone+=("$db")
|
|
fi
|
|
done
|
|
if [ ${#standalone[@]} -eq 0 ]; then
|
|
return 0
|
|
fi
|
|
|
|
if [ "$RECREATE_DB" != true ]; then
|
|
echo "ERROR: clustered DBs requested, but these are in standalone format:" >&2
|
|
printf ' %s\n' "${standalone[@]}" >&2
|
|
echo "" >&2
|
|
echo " A clustered DB can only be created from scratch, so these must be" >&2
|
|
echo " removed first. That DISCARDS ALL LOGICAL NETWORK STATE — every logical" >&2
|
|
echo " switch, router, port and ACL, and so every VPC on this node." >&2
|
|
echo "" >&2
|
|
echo " A freshly installed node has nothing to lose: re-run with --recreate-db." >&2
|
|
echo " A node running workloads does: do not." >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo " --recreate-db: removing standalone DBs so ovn-ctl can create clustered ones"
|
|
|
|
# ovn-controller must go down with them. Left running, it reconnects to the
|
|
# fresh SB and re-registers under whatever system-id it started with — which
|
|
# on a renamed node is the old one, and that row then holds this node's encap
|
|
# IP so the correctly-named chassis can never commit. Step 5 restarts it.
|
|
sudo systemctl stop ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb ovn-central ovn-controller 2>/dev/null || true
|
|
sudo rm -f "${standalone[@]}"
|
|
}
|
|
|
|
echo ""
|
|
echo "Step 2: Enabling services..."
|
|
|
|
sudo systemctl enable openvswitch-switch
|
|
sudo systemctl start openvswitch-switch
|
|
echo " openvswitch-switch: started"
|
|
|
|
if [ "$MANAGEMENT" = true ]; then
|
|
sudo systemctl enable ovn-central
|
|
|
|
# LAN-plane NB/SB client listener, added to the loopback one set below.
|
|
# Guard is load-bearing: --db-nb-addr defaults to 0.0.0.0, so
|
|
# create-insecure-remote=yes with no LAN_ADDR recreates the wildcard bug.
|
|
OVN_DB_LISTEN_OPTS=""
|
|
if [ -n "$LAN_ADDR" ]; then
|
|
OVN_DB_LISTEN_OPTS="--db-nb-addr=$LAN_ADDR --db-nb-create-insecure-remote=yes --db-sb-addr=$LAN_ADDR --db-sb-create-insecure-remote=yes"
|
|
fi
|
|
|
|
if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
ensure_clustered_db_storage
|
|
|
|
# Clustered NB/SB via native OVSDB RAFT. Both per-DB units source one
|
|
# shared OVN_CTL_OPTS from /etc/default/ovn-central; each run_*_ovsdb
|
|
# consumes only its own --db-{nb,sb}-* flags. RAFT ports default to NB
|
|
# 6643 / SB 6644. ovn-ctl creates the cluster when no remote-addr is
|
|
# given (and no existing .db file), joins when one is.
|
|
OVN_CTL_OPTS="--db-nb-cluster-local-addr=$DB_CLUSTER_LOCAL_ADDR --db-sb-cluster-local-addr=$DB_CLUSTER_LOCAL_ADDR"
|
|
if [ -n "$DB_CLUSTER_REMOTE_ADDR" ]; then
|
|
OVN_CTL_OPTS="$OVN_CTL_OPTS --db-nb-cluster-remote-addr=$DB_CLUSTER_REMOTE_ADDR --db-sb-cluster-remote-addr=$DB_CLUSTER_REMOTE_ADDR"
|
|
echo " ovn-central: joining RAFT cluster (local=$DB_CLUSTER_LOCAL_ADDR remote=$DB_CLUSTER_REMOTE_ADDR)"
|
|
else
|
|
echo " ovn-central: creating RAFT cluster (local=$DB_CLUSTER_LOCAL_ADDR)"
|
|
fi
|
|
|
|
# Point ovn-northd's client NB/SB connections at the full RAFT member
|
|
# list so the active northd follows the leader. Without this it dials the
|
|
# local member only and stops advancing SB_Global.nb_cfg once leadership
|
|
# moves off this node, wedging the ovn-nbctl --wait=hv flows barrier.
|
|
# Derived from OVN_REMOTE (the SB list); the NB list is the same hosts on
|
|
# 6641. Skipped when OVN_REMOTE is the localhost default (caller passed no
|
|
# member list) so a lone standalone DB node is unaffected.
|
|
if [ "$OVN_REMOTE" != "tcp:127.0.0.1:6642" ]; then
|
|
OVN_REMOTE_NB="${OVN_REMOTE//:6642/:6641}"
|
|
OVN_CTL_OPTS="$OVN_CTL_OPTS --ovn-northd-nb-db=$OVN_REMOTE_NB --ovn-northd-sb-db=$OVN_REMOTE"
|
|
echo " ovn-northd: NB=$OVN_REMOTE_NB SB=$OVN_REMOTE (RAFT member list)"
|
|
else
|
|
NORTHD_LOCAL_ONLY=true
|
|
fi
|
|
|
|
if [ -n "$OVN_DB_LISTEN_OPTS" ]; then
|
|
OVN_CTL_OPTS="$OVN_CTL_OPTS $OVN_DB_LISTEN_OPTS"
|
|
echo " NB/SB client listen: 127.0.0.1, $LAN_ADDR"
|
|
else
|
|
echo " NB/SB client listen: 127.0.0.1"
|
|
fi
|
|
|
|
echo "OVN_CTL_OPTS=\"$OVN_CTL_OPTS\"" | sudo tee /etc/default/ovn-central >/dev/null
|
|
echo " wrote /etc/default/ovn-central"
|
|
|
|
# The packaged ovn-northd.service ExecStop runs `ovn-ctl stop_northd`
|
|
# without --ovn-manage-ovsdb=no, so restarting northd also tears down the
|
|
# NB/SB ovsdb-server units. With the split clustered units those DBs are
|
|
# owned by their own units, so override ExecStop to leave them alone —
|
|
# otherwise the restart below races and kills the freshly-started DBs.
|
|
sudo mkdir -p /etc/systemd/system/ovn-northd.service.d
|
|
sudo tee /etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf >/dev/null <<'EOF'
|
|
[Service]
|
|
ExecStop=
|
|
ExecStop=/usr/share/ovn/scripts/ovn-ctl stop_northd --no-monitor --ovn-manage-ovsdb=no
|
|
EOF
|
|
sudo systemctl daemon-reload
|
|
|
|
# The ovn-central aggregator is ExecStart=/bin/true, so restarting it
|
|
# won't restart the children — restart the per-DB units directly to
|
|
# pick up the new OVN_CTL_OPTS.
|
|
sudo systemctl restart ovn-ovsdb-server-nb ovn-ovsdb-server-sb ovn-northd
|
|
echo " ovn-central: started clustered (NB DB + SB DB + ovn-northd)"
|
|
else
|
|
# Always write the file, even when there is nothing to put in it. This
|
|
# branch is how a node that used to be clustered comes back standalone,
|
|
# and the RAFT flags live here rather than in the database — leaving a
|
|
# stale file in place restarts the DB as a cluster member dialling a
|
|
# peer that no longer exists, which looks like a hung leader election.
|
|
echo "OVN_CTL_OPTS=\"$OVN_DB_LISTEN_OPTS\"" | sudo tee /etc/default/ovn-central >/dev/null
|
|
echo " wrote /etc/default/ovn-central"
|
|
sudo systemctl start ovn-central
|
|
|
|
# ovn-central is ExecStart=/bin/true; restarting it does not restart
|
|
# its children, so a re-run with changed options needs the per-DB units
|
|
# restarted directly to pick them up.
|
|
sudo systemctl restart ovn-ovsdb-server-nb ovn-ovsdb-server-sb
|
|
echo " ovn-central: started (NB DB + SB DB + ovn-northd)"
|
|
if [ -n "$LAN_ADDR" ]; then
|
|
echo " NB/SB client listen: 127.0.0.1, $LAN_ADDR"
|
|
else
|
|
echo " NB/SB client listen: 127.0.0.1"
|
|
fi
|
|
fi
|
|
|
|
# A clustered DB has no unix socket to fall back on, and a follower refuses
|
|
# to answer at all unless leader-only is turned off. Without both of these
|
|
# the waits below can never succeed on a clustered node: they burn their
|
|
# full timeout, gate nothing, and the run still continues.
|
|
NBCTL=(ovn-nbctl)
|
|
SBCTL=(ovn-sbctl)
|
|
if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
NBCTL=(ovn-nbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6641" --no-leader-only)
|
|
SBCTL=(ovn-sbctl --db="tcp:$DB_CLUSTER_LOCAL_ADDR:6642" --no-leader-only)
|
|
fi
|
|
|
|
# Wait for OVN NB DB socket to become available
|
|
for i in $(seq 1 15); do
|
|
if sudo "${NBCTL[@]}" --timeout=2 get-connection >/dev/null 2>&1; then
|
|
break
|
|
fi
|
|
echo " Waiting for OVN NB DB... ($i/15)"
|
|
sleep 1
|
|
done
|
|
|
|
# Verify rather than assume: a DB that came up standalone despite the RAFT
|
|
# flags is the exact failure this guards against, and it stays invisible
|
|
# until a node goes down and the cluster turns out not to exist.
|
|
if [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
for db in "$OVN_DBDIR/ovnnb_db.db" "$OVN_DBDIR/ovnsb_db.db"; do
|
|
if ! sudo ovsdb-tool db-is-clustered "$db" 2>/dev/null; then
|
|
echo "ERROR: $db is not in clustered format after startup." >&2
|
|
echo " The RAFT configuration did not take effect." >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
echo " DB storage: clustered (NB + SB verified)"
|
|
fi
|
|
|
|
# An OVSDB RAFT follower does not proxy writes — it answers "not cluster
|
|
# leader; trying another server" — so a northd whose client list names one
|
|
# member works only while it shares a node with the leader. When it does not,
|
|
# NB->SB translation stops dead: nothing is logged above info, every service
|
|
# stays active, and the only evidence is SB_Global.nb_cfg falling behind
|
|
# NB_Global.nb_cfg. Derive the list from the live membership rather than
|
|
# trusting the caller to have passed --ovn-remote.
|
|
if [ "${NORTHD_LOCAL_ONLY:-false}" = true ] && [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
NB_MEMBERS=$(sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl \
|
|
cluster/status OVN_Northbound 2>/dev/null \
|
|
| grep -oE 'at tcp:[0-9.]+:' | sed 's/^at tcp://; s/:$//' | sort -u)
|
|
NB_LIST=""
|
|
SB_LIST=""
|
|
for addr in $NB_MEMBERS; do
|
|
NB_LIST="${NB_LIST:+$NB_LIST,}tcp:$addr:6641"
|
|
SB_LIST="${SB_LIST:+$SB_LIST,}tcp:$addr:6642"
|
|
done
|
|
if [ "$(printf '%s\n' "$NB_MEMBERS" | grep -c .)" -gt 1 ]; then
|
|
sudo sed -i "s|\"\$| --ovn-northd-nb-db=$NB_LIST --ovn-northd-sb-db=$SB_LIST\"|" \
|
|
/etc/default/ovn-central
|
|
sudo systemctl restart ovn-northd
|
|
echo " ovn-northd: NB=$NB_LIST SB=$SB_LIST (derived from RAFT membership)"
|
|
else
|
|
echo " NOTE: this node is the only NB RAFT member, so ovn-northd is pointed at" >&2
|
|
echo " itself. Once the other members join, re-run this script here (or" >&2
|
|
echo " pass --ovn-remote with the full member list) — otherwise northd" >&2
|
|
echo " silently stops translating NB to SB the first time NB leadership" >&2
|
|
echo " moves off this node, and every VPC created after that never" >&2
|
|
echo " attaches its internet gateway." >&2
|
|
fi
|
|
fi
|
|
|
|
# Wait for the Southbound DB to be serving before ovn-controller (Step 5)
|
|
# dials it. On a single node ovn-controller races a fresh SB RAFT election
|
|
# with no join step to absorb the window — attaching before the leader is
|
|
# elected (and before northd writes the gateway logical flows) leaves it with
|
|
# a partial datapath that never reconverges. Gate on SB reachable and, when
|
|
# clustered, an elected leader.
|
|
for i in $(seq 1 30); do
|
|
if sudo "${SBCTL[@]}" --timeout=2 show >/dev/null 2>&1; then
|
|
if [ -z "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
|
break
|
|
fi
|
|
SB_LEADER=$(sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl \
|
|
cluster/status OVN_Southbound 2>/dev/null \
|
|
| awk '/^Leader:/{print $2; exit}')
|
|
if [ -n "$SB_LEADER" ] && [ "$SB_LEADER" != "unknown" ]; then
|
|
break
|
|
fi
|
|
fi
|
|
echo " Waiting for OVN SB DB leader... ($i/30)"
|
|
sleep 1
|
|
done
|
|
|
|
# set-connection replicates through RAFT, so it only runs on the create
|
|
# node. Scoped to loopback, not a wildcard ptcp:6641, because 127.0.0.1
|
|
# is valid on every node; the differing LAN address is set above instead.
|
|
if [ -z "$DB_CLUSTER_REMOTE_ADDR" ]; then
|
|
sudo ovn-nbctl set-connection ptcp:6641:127.0.0.1
|
|
sudo ovn-sbctl set-connection ptcp:6642:127.0.0.1
|
|
if [ -n "$LAN_ADDR" ]; then
|
|
echo " OVN NB DB listening on tcp:127.0.0.1:6641, tcp:$LAN_ADDR:6641"
|
|
echo " OVN SB DB listening on tcp:127.0.0.1:6642, tcp:$LAN_ADDR:6642"
|
|
else
|
|
echo " OVN NB DB listening on tcp:127.0.0.1:6641"
|
|
echo " OVN SB DB listening on tcp:127.0.0.1:6642"
|
|
fi
|
|
fi
|
|
else
|
|
# A compute node must not run a database. The ovn-central package starts a
|
|
# standalone one on install, and nothing here used to stop it, so every
|
|
# node past the quorum kept an unclustered NB/SB that no chassis reports
|
|
# into. It serves nothing, but it answers on the local socket, which is
|
|
# enough for any ovn-nbctl without an explicit --db to confirm against the
|
|
# wrong database — a `--wait=hv` there can never be acknowledged and burns
|
|
# its whole timeout instead.
|
|
#
|
|
# Refuse to demote a node this script previously set up as a management
|
|
# node. --management is the only record of the role and is re-asserted on
|
|
# every run, so a re-run that forgets it silently takes the cluster's
|
|
# database away. The symptom lands three layers off: nothing listens on
|
|
# 6641, and vpcd crash-loops on "connection refused" with no hint that this
|
|
# script caused it. Seen on mulga-poc at 15,061 restarts.
|
|
#
|
|
# The signal is /etc/default/ovn-central, not the presence of a database:
|
|
# the ovn-central *package* creates DBs on install, so every compute node
|
|
# has them and testing for them would refuse every legitimate compute node.
|
|
# Only the --management path below writes OVN_CTL_OPTS to that file.
|
|
if [ -f /etc/default/ovn-central ] && grep -q '^OVN_CTL_OPTS=' /etc/default/ovn-central 2>/dev/null; then
|
|
echo "ERROR: this node was configured as a management node by a previous run"
|
|
echo " (/etc/default/ovn-central holds OVN_CTL_OPTS), but --management was"
|
|
echo " not given. Re-running without it stops and disables ovn-central and"
|
|
echo " takes the control plane down."
|
|
echo ""
|
|
echo " If this is a management node, re-run with --management."
|
|
echo " If it is genuinely being demoted to a compute node:"
|
|
echo " sudo rm -f /etc/default/ovn-central"
|
|
exit 1
|
|
fi
|
|
|
|
if systemctl is-enabled ovn-central >/dev/null 2>&1 ||
|
|
systemctl is-active ovn-central >/dev/null 2>&1; then
|
|
sudo systemctl stop ovn-central ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb 2>/dev/null || true
|
|
sudo systemctl disable ovn-central ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb 2>/dev/null || true
|
|
echo " ovn-central: stopped and disabled (compute node runs no database)"
|
|
fi
|
|
for db in "$OVN_DBDIR/ovnnb_db.db" "$OVN_DBDIR/ovnsb_db.db"; do
|
|
[ -f "$db" ] || continue
|
|
sudo mv "$db" "$db.standalone-disabled"
|
|
echo " moved aside $db (left by the package's standalone ovn-central)"
|
|
done
|
|
fi
|
|
|
|
# --- Step 3: Create and configure br-int ---
|
|
echo ""
|
|
echo "Step 3: Configuring br-int..."
|
|
|
|
sudo ovs-vsctl --may-exist add-br br-int
|
|
sudo ovs-vsctl set Bridge br-int fail-mode=secure
|
|
sudo ovs-vsctl set Bridge br-int other-config:disable-in-band=true
|
|
sudo ip link set br-int up
|
|
echo " br-int: created, fail-mode=secure, up"
|
|
|
|
# Mark OVS internal netdevs Unmanaged for systemd-networkd. Without this,
|
|
# Trixie's networkd takes ownership of any unconfigured iface and may bring
|
|
# br-int/br-ext admin-down after setup-ovn.sh's `ip link set up` (no Match
|
|
# rule => default management => no carrier => link down). OVS dataplane
|
|
# still forwards, but ovn-controller flow programming and any tooling that
|
|
# probes link state misbehave. Unmanaged=yes keeps OVS in sole control.
|
|
OVS_INTERNAL_NET=/etc/systemd/network/05-spinifex-ovs-internal.network
|
|
if [ ! -f "$OVS_INTERNAL_NET" ]; then
|
|
sudo tee "$OVS_INTERNAL_NET" >/dev/null <<'NETWORK'
|
|
[Match]
|
|
Name=br-int br-ext
|
|
|
|
[Link]
|
|
Unmanaged=yes
|
|
NETWORK
|
|
sudo networkctl reload 2>/dev/null || true
|
|
echo " wrote $OVS_INTERNAL_NET (Unmanaged=yes for br-int br-ext)"
|
|
fi
|
|
|
|
# --- Step 3b: Configure WAN bridge for public subnet uplink ---
|
|
if [ -n "$WAN_BRIDGE" ]; then
|
|
echo ""
|
|
echo "Step 3b: Configuring WAN bridge ($WAN_BRIDGE) for public subnet uplink..."
|
|
|
|
# Rip down any stale veth persistence from a previous veth-mode install
|
|
# when switching to any non-veth mode. Idempotent — each command uses
|
|
# --if-exists / 2>/dev/null to tolerate absence. Without this, the
|
|
# veth pair re-materialises on reboot and fights the current mode's
|
|
# Bridge plumbing for the host-to-OVN datapath.
|
|
if [ "$WAN_BRIDGE_MODE" != "veth" ]; then
|
|
sudo rm -f /etc/systemd/network/14-spinifex-br-wan.netdev \
|
|
/etc/systemd/network/15-spinifex-veth-wan.netdev \
|
|
/etc/systemd/network/15-spinifex-veth-wan.network \
|
|
/etc/systemd/network/16-spinifex-veth-wan-ovs.network
|
|
sudo networkctl reload 2>/dev/null || true
|
|
sudo ovs-vsctl --if-exists del-port veth-wan-ovs 2>/dev/null || true
|
|
sudo ip link del veth-wan-br 2>/dev/null || true
|
|
fi
|
|
|
|
# Same ripdown for stale routed-NAT plumbing when switching away from nat.
|
|
if [ "$WAN_BRIDGE_MODE" != "nat" ]; then
|
|
sudo rm -f /etc/systemd/network/17-spinifex-nat.netdev \
|
|
/etc/systemd/network/17-spinifex-nat.network \
|
|
/etc/systemd/network/18-spinifex-nat-ovs.network
|
|
sudo networkctl reload 2>/dev/null || true
|
|
sudo ovs-vsctl --if-exists del-port spx-nat-ovs 2>/dev/null || true
|
|
sudo ip link del spx-nat-host 2>/dev/null || true
|
|
fi
|
|
|
|
case "$WAN_BRIDGE_MODE" in
|
|
existing)
|
|
# Already an OVS bridge (from a previous run or explicit --wan-bridge).
|
|
if ! sudo ovs-vsctl br-exists "$WAN_BRIDGE" 2>/dev/null; then
|
|
sudo ovs-vsctl --may-exist add-br "$WAN_BRIDGE"
|
|
echo " created OVS bridge: $WAN_BRIDGE"
|
|
fi
|
|
sudo ip link set "$WAN_BRIDGE" up
|
|
echo " $WAN_BRIDGE: OVS bridge, up"
|
|
;;
|
|
|
|
veth)
|
|
# Linux bridge detected (e.g. br-wan from cloud-init/netplan).
|
|
# OVN bridge-mappings require an OVS bridge. Rather than converting
|
|
# the Linux bridge (destructive, causes WAN interruption), we create
|
|
# a separate OVS bridge and link them with a veth pair:
|
|
#
|
|
# br-wan (Linux, keeps IP/routes) ←→ veth pair ←→ br-ext (OVS, for OVN)
|
|
#
|
|
# No network interruption. The Linux bridge is untouched.
|
|
|
|
# Create OVS bridge
|
|
if ! sudo ovs-vsctl br-exists "$WAN_BRIDGE" 2>/dev/null; then
|
|
sudo ovs-vsctl --may-exist add-br "$WAN_BRIDGE"
|
|
echo " created OVS bridge: $WAN_BRIDGE"
|
|
fi
|
|
sudo ip link set "$WAN_BRIDGE" up
|
|
|
|
# Create veth pair (idempotent)
|
|
if ! ip link show veth-wan-br >/dev/null 2>&1; then
|
|
sudo ip link add veth-wan-br type veth peer name veth-wan-ovs
|
|
echo " created veth pair: veth-wan-br ↔ veth-wan-ovs"
|
|
else
|
|
echo " veth pair already exists: veth-wan-br ↔ veth-wan-ovs"
|
|
fi
|
|
|
|
# Enslave veth-wan-br to the Linux bridge
|
|
if ! ip link show veth-wan-br 2>/dev/null | grep -q "master $LINUX_BRIDGE"; then
|
|
sudo ip link set veth-wan-br master "$LINUX_BRIDGE"
|
|
echo " veth-wan-br → $LINUX_BRIDGE (Linux bridge)"
|
|
fi
|
|
sudo ip link set veth-wan-br up
|
|
|
|
# Add veth-wan-ovs to the OVS bridge
|
|
if ! sudo ovs-vsctl port-to-br veth-wan-ovs >/dev/null 2>&1; then
|
|
sudo ovs-vsctl --may-exist add-port "$WAN_BRIDGE" veth-wan-ovs
|
|
echo " veth-wan-ovs → $WAN_BRIDGE (OVS bridge)"
|
|
fi
|
|
sudo ip link set veth-wan-ovs up
|
|
|
|
echo " $LINUX_BRIDGE (Linux) ↔ veth pair ↔ $WAN_BRIDGE (OVS)"
|
|
echo " $LINUX_BRIDGE keeps its IP and routes — no interruption"
|
|
|
|
# Persist the veth pair across reboot via systemd-networkd. Veths
|
|
# are kernel-only and vanish on reboot; without persistence vpcd
|
|
# starts with the OVS port pointing at a nonexistent peer and
|
|
# silently falls back to direct mode.
|
|
#
|
|
# networkd's Bridge= directive requires the target bridge to be a
|
|
# known NetDev. On ISO-installed nodes the installer writes
|
|
# 11-spinifex-br-wan.netdev, which the gate below detects and
|
|
# skips this write. On binary-installed nodes where the operator
|
|
# manages br-wan outside networkd (e.g. cloud-init), this file
|
|
# fills the gap so veth-wan-br resolves its bridge on reboot.
|
|
# `Failed to create netdev: File exists` is harmless — networkd
|
|
# matches the existing kernel bridge by name+kind.
|
|
#
|
|
# Gate: skip the write if any .netdev (installer, cloud-init,
|
|
# netplan, manual) already declares this bridge — don't clobber
|
|
# existing networkd config. networkd searches /etc, /run, /usr/lib;
|
|
# check all three. Our own file is excluded so idempotent re-runs
|
|
# still rewrite when needed.
|
|
BR_WAN_NETDEV="/etc/systemd/network/14-spinifex-br-wan.netdev"
|
|
VETH_NETDEV="/etc/systemd/network/15-spinifex-veth-wan.netdev"
|
|
VETH_NETWORK="/etc/systemd/network/15-spinifex-veth-wan.network"
|
|
VETH_OVS_NETWORK="/etc/systemd/network/16-spinifex-veth-wan-ovs.network"
|
|
EXISTING_BR_NETDEV=$(grep -rls --include="*.netdev" -E "^\s*Name=$LINUX_BRIDGE\s*$" \
|
|
/etc/systemd/network /run/systemd/network /usr/lib/systemd/network 2>/dev/null \
|
|
| grep -v "^$BR_WAN_NETDEV$" || true)
|
|
if [ -n "$EXISTING_BR_NETDEV" ]; then
|
|
echo " skipping $BR_WAN_NETDEV — operator-managed NetDev already declares $LINUX_BRIDGE: $EXISTING_BR_NETDEV"
|
|
else
|
|
sudo tee "$BR_WAN_NETDEV" >/dev/null <<NETDEV
|
|
[NetDev]
|
|
Name=$LINUX_BRIDGE
|
|
Kind=bridge
|
|
NETDEV
|
|
fi
|
|
sudo tee "$VETH_NETDEV" >/dev/null <<NETDEV
|
|
[NetDev]
|
|
Name=veth-wan-br
|
|
Kind=veth
|
|
|
|
[Peer]
|
|
Name=veth-wan-ovs
|
|
NETDEV
|
|
sudo tee "$VETH_NETWORK" >/dev/null <<NETWORK
|
|
[Match]
|
|
Name=veth-wan-br
|
|
|
|
[Network]
|
|
Bridge=$LINUX_BRIDGE
|
|
ConfigureWithoutCarrier=yes
|
|
NETWORK
|
|
# Second unit admin-ups the OVS end of the pair. OVS owns the port
|
|
# (enslaved via ovs-vsctl add-port above) but does not flip admin
|
|
# state on external ports — that's networkd's job. Without this,
|
|
# veth-wan-ovs stays DOWN after reboot, peer goes LOWERLAYERDOWN,
|
|
# br-wan loses carrier.
|
|
sudo tee "$VETH_OVS_NETWORK" >/dev/null <<NETWORK
|
|
[Match]
|
|
Name=veth-wan-ovs
|
|
|
|
[Link]
|
|
RequiredForOnline=no
|
|
|
|
[Network]
|
|
ConfigureWithoutCarrier=yes
|
|
NETWORK
|
|
sudo networkctl reload 2>/dev/null || true
|
|
echo " wrote $VETH_NETDEV + $VETH_NETWORK + $VETH_OVS_NETWORK (veth persists on reboot)"
|
|
;;
|
|
|
|
direct)
|
|
# Add WAN NIC directly to OVS bridge. The NIC becomes an OVS slave —
|
|
# its IP (if any) is no longer reachable from the host. The user has
|
|
# confirmed this NIC is NOT their SSH connection.
|
|
if ! ip link show "$WAN_IFACE" >/dev/null 2>&1; then
|
|
echo " ERROR: interface $WAN_IFACE does not exist"
|
|
echo " Available interfaces:"
|
|
ip -o link show | awk -F': ' '{print " " $2}'
|
|
exit 1
|
|
fi
|
|
|
|
sudo ovs-vsctl --may-exist add-br "$WAN_BRIDGE"
|
|
sudo ip link set "$WAN_BRIDGE" up
|
|
|
|
if sudo ovs-vsctl port-to-br "$WAN_IFACE" >/dev/null 2>&1; then
|
|
echo " $WAN_IFACE already on $(sudo ovs-vsctl port-to-br "$WAN_IFACE")"
|
|
else
|
|
sudo ovs-vsctl --may-exist add-port "$WAN_BRIDGE" "$WAN_IFACE"
|
|
echo " added $WAN_IFACE directly to $WAN_BRIDGE"
|
|
fi
|
|
sudo ip link set "$WAN_IFACE" up
|
|
echo " $WAN_BRIDGE: direct bridge on $WAN_IFACE"
|
|
echo " NOTE: $WAN_IFACE is now an OVS port — no host IP on this NIC"
|
|
;;
|
|
|
|
nat)
|
|
# Routed NAT: br-ext carries no WAN NIC. A transit veth pair links
|
|
# it to the host stack (spx-nat-host owns 100.127.0.1/24); the host
|
|
# forwards and masquerades the transit /24 out whatever uplink it
|
|
# has. OVN SNATs each VPC CIDR to its gateway LRP transit IP, so
|
|
# the masquerade rule below is the only host-side NAT state.
|
|
NAT_TRANSIT_CIDR="100.127.0.0/24"
|
|
NAT_TRANSIT_GW_CIDR="100.127.0.1/24"
|
|
# Same on every node. OVN keeps one MAC binding for the transit
|
|
# nexthop cluster-wide, so a random per-node MAC sends every
|
|
# chassis's egress to whichever node seeded it; the pair is
|
|
# node-local, so one address everywhere is unambiguous.
|
|
NAT_TRANSIT_HOST_MAC="02:00:64:7f:00:01"
|
|
|
|
if ! sudo ovs-vsctl br-exists "$WAN_BRIDGE" 2>/dev/null; then
|
|
sudo ovs-vsctl --may-exist add-br "$WAN_BRIDGE"
|
|
echo " created OVS bridge: $WAN_BRIDGE"
|
|
fi
|
|
sudo ip link set "$WAN_BRIDGE" up
|
|
|
|
# Create transit veth pair (idempotent)
|
|
if ! ip link show spx-nat-host >/dev/null 2>&1; then
|
|
sudo ip link add spx-nat-host type veth peer name spx-nat-ovs
|
|
echo " created veth pair: spx-nat-host ↔ spx-nat-ovs"
|
|
else
|
|
echo " veth pair already exists: spx-nat-host ↔ spx-nat-ovs"
|
|
fi
|
|
sudo ip link set spx-nat-host address "$NAT_TRANSIT_HOST_MAC"
|
|
sudo ip addr replace "$NAT_TRANSIT_GW_CIDR" dev spx-nat-host
|
|
|
|
# Add the OVS end to br-ext
|
|
if ! sudo ovs-vsctl port-to-br spx-nat-ovs >/dev/null 2>&1; then
|
|
sudo ovs-vsctl --may-exist add-port "$WAN_BRIDGE" spx-nat-ovs
|
|
echo " spx-nat-ovs → $WAN_BRIDGE (OVS bridge)"
|
|
fi
|
|
sudo ip link set spx-nat-host up
|
|
sudo ip link set spx-nat-ovs up
|
|
echo " host (spx-nat-host $NAT_TRANSIT_GW_CIDR) ↔ veth pair ↔ $WAN_BRIDGE (OVS)"
|
|
|
|
# Persist the veth pair + transit IP across reboot (veths are
|
|
# kernel-only; same rationale as veth mode above).
|
|
NAT_NETDEV="/etc/systemd/network/17-spinifex-nat.netdev"
|
|
NAT_NETWORK="/etc/systemd/network/17-spinifex-nat.network"
|
|
NAT_OVS_NETWORK="/etc/systemd/network/18-spinifex-nat-ovs.network"
|
|
sudo tee "$NAT_NETDEV" >/dev/null <<NETDEV
|
|
[NetDev]
|
|
Name=spx-nat-host
|
|
Kind=veth
|
|
MACAddress=$NAT_TRANSIT_HOST_MAC
|
|
|
|
[Peer]
|
|
Name=spx-nat-ovs
|
|
NETDEV
|
|
sudo tee "$NAT_NETWORK" >/dev/null <<NETWORK
|
|
[Match]
|
|
Name=spx-nat-host
|
|
|
|
[Network]
|
|
Address=$NAT_TRANSIT_GW_CIDR
|
|
ConfigureWithoutCarrier=yes
|
|
NETWORK
|
|
# Admin-up the OVS end after reboot (OVS owns the port but does
|
|
# not flip admin state on external ports — same as veth mode).
|
|
sudo tee "$NAT_OVS_NETWORK" >/dev/null <<NETWORK
|
|
[Match]
|
|
Name=spx-nat-ovs
|
|
|
|
[Link]
|
|
RequiredForOnline=no
|
|
|
|
[Network]
|
|
ConfigureWithoutCarrier=yes
|
|
NETWORK
|
|
sudo networkctl reload 2>/dev/null || true
|
|
echo " wrote $NAT_NETDEV + $NAT_NETWORK + $NAT_OVS_NETWORK (transit veth persists on reboot)"
|
|
|
|
# Kernel egress: masquerade the transit /24 out any uplink and
|
|
# accept forwarded transit traffic. vpcd re-ensures these on every
|
|
# start; installing here too means the wiring is testable before
|
|
# services run.
|
|
#
|
|
# Insert at the head, never append: Ubuntu ships a catch-all REJECT
|
|
# in FORWARD, and an appended ACCEPT sits behind it and never
|
|
# matches. Drain first so an already-appended copy is repositioned
|
|
# rather than left stranded. Every spec carries our comment, so
|
|
# only our own rules are ever deleted.
|
|
nat_egress_rule() { # nat_egress_rule <table> <chain> <spec...>
|
|
local table="$1" chain="$2"; shift 2
|
|
local i
|
|
for i in 1 2 3 4 5 6 7 8; do
|
|
sudo iptables -t "$table" -D "$chain" "$@" 2>/dev/null || break
|
|
done
|
|
sudo iptables -t "$table" -I "$chain" 1 "$@"
|
|
}
|
|
nat_egress_rule nat POSTROUTING -s "$NAT_TRANSIT_CIDR" ! -d "$NAT_TRANSIT_CIDR" \
|
|
-m comment --comment "spinifex-nat-egress" -j MASQUERADE
|
|
nat_egress_rule filter FORWARD -i spx-nat-host -s "$NAT_TRANSIT_CIDR" \
|
|
-m comment --comment "spinifex-nat-egress" -j ACCEPT
|
|
nat_egress_rule filter FORWARD -o spx-nat-host -m conntrack --ctstate RELATED,ESTABLISHED \
|
|
-m comment --comment "spinifex-nat-egress" -j ACCEPT
|
|
echo " installed masquerade + forward rules for $NAT_TRANSIT_CIDR (comment: spinifex-nat-egress)"
|
|
;;
|
|
esac
|
|
|
|
# --- Prove the bridge actually came up ---
|
|
#
|
|
# ovs-vsctl reports on the database, not the datapath. `add-br` and
|
|
# `add-port` exit 0 whenever OVSDB accepts the row, and ofproto can refuse
|
|
# the device afterwards for reasons ovs-vsctl never sees — a name already
|
|
# taken by a Linux bridge, an interface another bridge already owns. The
|
|
# error is recorded per-interface and surfaces only here. Without this scan
|
|
# the script reports a successful setup and vpcd fails minutes later with a
|
|
# message that points somewhere else entirely.
|
|
if [ -n "$WAN_BRIDGE_MODE" ] && command -v ovs-vsctl >/dev/null 2>&1; then
|
|
OVS_ERRORS=$(sudo ovs-vsctl --columns=name,error --format=table list Interface 2>/dev/null \
|
|
| awk 'NR>2 && $0 ~ /could not|error/ && $0 !~ /\[\]/' || true)
|
|
if [ -n "$OVS_ERRORS" ]; then
|
|
echo ""
|
|
echo "ERROR: OVS accepted the configuration but the datapath refused it:"
|
|
echo "$OVS_ERRORS" | sed 's/^/ /'
|
|
echo ""
|
|
echo " The bridge exists in OVSDB and cannot forward. Nothing downstream"
|
|
echo " will work, and vpcd will fail with an unrelated-looking error."
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
# --- DHCP: obtain gateway IP for OVN SNAT ---
|
|
if [ "$EXTERNAL_DHCP" = true ] && [ "$WAN_BRIDGE_MODE" = "nat" ]; then
|
|
echo ""
|
|
echo " Skipping --dhcp: routed NAT mode has a fixed transit gateway (100.127.0.1)"
|
|
elif [ "$EXTERNAL_DHCP" = true ]; then
|
|
echo ""
|
|
echo "Step 3c: Obtaining external gateway IP via DHCP..."
|
|
|
|
# DHCP on the WAN bridge itself (direct/existing/veth).
|
|
DHCP_IFACE="$WAN_BRIDGE"
|
|
|
|
# Run DHCP client to get a lease
|
|
if command -v dhcpcd >/dev/null 2>&1; then
|
|
sudo dhcpcd --waitip=4 --timeout 15 "$DHCP_IFACE" 2>/dev/null || true
|
|
elif command -v dhclient >/dev/null 2>&1; then
|
|
sudo dhclient -1 -timeout 15 "$DHCP_IFACE" 2>/dev/null || true
|
|
else
|
|
echo " WARNING: no DHCP client found (dhcpcd or dhclient)"
|
|
echo " Install dhcpcd-base or isc-dhcp-client, or set gateway_ip manually"
|
|
fi
|
|
|
|
# Read the obtained IP
|
|
DHCP_IP=$(ip -4 addr show dev "$DHCP_IFACE" 2>/dev/null | awk '/inet /{print $2}' | head -1 | cut -d/ -f1)
|
|
if [ -n "$DHCP_IP" ]; then
|
|
echo " DHCP obtained: $DHCP_IP on $DHCP_IFACE"
|
|
|
|
# Write the gateway IP to the spinifex config so vpcd can use it
|
|
CONFIG_DIR="${CONFIG_DIR:-$HOME/spinifex/config}"
|
|
CONFIG_FILE="$CONFIG_DIR/spinifex.toml"
|
|
if [ -f "$CONFIG_FILE" ]; then
|
|
if grep -q "gateway_ip" "$CONFIG_FILE"; then
|
|
sed -i "s/gateway_ip.*/gateway_ip = \"$DHCP_IP\"/" "$CONFIG_FILE"
|
|
else
|
|
sed -i "/^gateway *=.*/a gateway_ip = \"$DHCP_IP\"" "$CONFIG_FILE"
|
|
fi
|
|
echo " Updated $CONFIG_FILE with gateway_ip = $DHCP_IP"
|
|
else
|
|
echo " WARNING: $CONFIG_FILE not found — set gateway_ip manually"
|
|
fi
|
|
else
|
|
echo " WARNING: DHCP failed to obtain IP on $DHCP_IFACE"
|
|
echo " VMs will not have external connectivity until gateway_ip is configured"
|
|
fi
|
|
fi
|
|
fi
|
|
|
|
# --- Step 3d: Management bridge for system-instance control plane ---
|
|
# br-mgmt is an OVS bridge (not L2-learning Linux bridge, not part of OVN
|
|
# overlay). fail-mode=standalone makes it behave like a plain L2 switch
|
|
# without OVN flows; it is excluded from ovn-bridge-mappings so
|
|
# ovn-controller ignores it. System instance TAPs attach via ovs-vsctl
|
|
# add-port (see SetupMgmtTapDevice in daemon/network.go).
|
|
if [ "$MGMT_BRIDGE_ENABLED" = true ]; then
|
|
echo ""
|
|
echo "Step 3d: Configuring management bridge ($MGMT_BRIDGE)..."
|
|
|
|
# --may-exist is idempotent against OVS, not against the kernel. When a
|
|
# Linux bridge already holds the name, OVS creates the bridge record but
|
|
# cannot create the internal netdev, leaving a half-built bridge whose
|
|
# only symptom is an error buried in `ovs-vsctl show`.
|
|
if ip link show "$MGMT_BRIDGE" >/dev/null 2>&1 && \
|
|
! sudo ovs-vsctl br-exists "$MGMT_BRIDGE" 2>/dev/null; then
|
|
echo " ERROR: '$MGMT_BRIDGE' already exists as a non-OVS link"
|
|
ip -d link show "$MGMT_BRIDGE" | head -2
|
|
echo ""
|
|
echo " The management bridge must be an OVS bridge. Either:"
|
|
echo " 1. Point this script elsewhere: --mgmt-bridge=<name>"
|
|
echo " 2. Rename the existing link (a plane bridge belongs on br-wan/br-lan/br-vpc)"
|
|
echo " 3. Skip mgmt provisioning entirely: --no-mgmt-bridge"
|
|
exit 1
|
|
fi
|
|
|
|
sudo ovs-vsctl --may-exist add-br "$MGMT_BRIDGE"
|
|
sudo ovs-vsctl set Bridge "$MGMT_BRIDGE" \
|
|
fail-mode=standalone \
|
|
other-config:disable-in-band=true
|
|
sudo ip link set "$MGMT_BRIDGE" up
|
|
echo " $MGMT_BRIDGE: OVS bridge, fail-mode=standalone, up"
|
|
|
|
# Enslave physical/virtual mgmt NIC when provided (multi-node deployments
|
|
# where the node's mgmt subnet spans multiple hosts).
|
|
if [ -n "$MGMT_IFACE" ]; then
|
|
if ! ip link show "$MGMT_IFACE" >/dev/null 2>&1; then
|
|
echo " ERROR: mgmt interface $MGMT_IFACE does not exist"
|
|
ip -o link show | awk -F': ' '{print " " $2}'
|
|
exit 1
|
|
fi
|
|
|
|
# Drop any existing IP on the NIC — IP belongs on the bridge.
|
|
sudo ip addr flush dev "$MGMT_IFACE" || true
|
|
sudo ovs-vsctl --may-exist add-port "$MGMT_BRIDGE" "$MGMT_IFACE"
|
|
sudo ip link set "$MGMT_IFACE" up
|
|
echo " $MGMT_IFACE: port on $MGMT_BRIDGE"
|
|
fi
|
|
|
|
if [ -n "$MGMT_CIDR" ]; then
|
|
sudo ip addr replace "$MGMT_CIDR" dev "$MGMT_BRIDGE"
|
|
echo " $MGMT_BRIDGE: address $MGMT_CIDR"
|
|
|
|
# Persist the IP across reboots via a systemd-networkd drop-in. The
|
|
# OVS bridge definition itself survives in ovsdb; only the L3 IP
|
|
# needs re-applying on boot.
|
|
MGMT_NETD_UNIT="/etc/systemd/network/10-spinifex-mgmt.network"
|
|
sudo tee "$MGMT_NETD_UNIT" >/dev/null <<NETD
|
|
[Match]
|
|
Name=$MGMT_BRIDGE
|
|
|
|
[Network]
|
|
Address=$MGMT_CIDR
|
|
ConfigureWithoutCarrier=yes
|
|
NETD
|
|
echo " wrote $MGMT_NETD_UNIT (IP persists on reboot via systemd-networkd)"
|
|
fi
|
|
fi
|
|
|
|
# --- Step 4: Configure OVN external_ids ---
|
|
echo ""
|
|
echo "Step 4: Setting OVS external_ids for OVN..."
|
|
|
|
if [ -n "$WAN_BRIDGE" ]; then
|
|
BRIDGE_MAPPINGS="external:${WAN_BRIDGE}"
|
|
sudo ovs-vsctl set Open_vSwitch . \
|
|
external_ids:ovn-remote="$OVN_REMOTE" \
|
|
external_ids:ovn-encap-ip="$ENCAP_IP" \
|
|
external_ids:ovn-encap-type="geneve" \
|
|
external_ids:ovn-bridge-mappings="$BRIDGE_MAPPINGS"
|
|
echo " ovn-bridge-mappings: $BRIDGE_MAPPINGS"
|
|
else
|
|
sudo ovs-vsctl set Open_vSwitch . \
|
|
external_ids:ovn-remote="$OVN_REMOTE" \
|
|
external_ids:ovn-encap-ip="$ENCAP_IP" \
|
|
external_ids:ovn-encap-type="geneve"
|
|
fi
|
|
|
|
# Pin OVS system-id (= OVN chassis-id) to NODE_NAME when --node-name was
|
|
# explicitly given. Reasons to pin:
|
|
# 1. IPsec identity: ovs-monitor-ipsec uses chassis-id as the IKEv2
|
|
# `@<name>` peer identity. Our per-node IPsec peer cert
|
|
# (host.GenerateIPSecPeerCert) carries the cluster node name as CN
|
|
# + dnsName SAN — see spx admin init/join, which take --node NAME and
|
|
# bake NAME into the cert. Leaving chassis-id as the package-generated
|
|
# UUID would cause `received AUTHENTICATION_FAILED` because charon
|
|
# validates `@<UUID>` against a cert dnsName=<NODE_NAME>.
|
|
# 2. Ops legibility: `ovn-sbctl show` lists chassis by name, not UUID.
|
|
#
|
|
# When --node-name is NOT given (CI bootstrap-install.sh today, dev
|
|
# single-node bring-up, ansible roles that don't yet pass it), leave the
|
|
# system-id alone. A hostname fallback used to live here, but on CI
|
|
# single-node it rewrote system-id from the gold-image UUID to a
|
|
# different hostname while the existing SBDB chassis row kept name=UUID;
|
|
# vpcd's discoverChassis then logged "skipping stale local chassis" and
|
|
# refused to start. Preserving the on-disk value is always safe — any
|
|
# caller that needs IPsec cert identity matching pins it themselves.
|
|
if [ -n "$NODE_NAME" ]; then
|
|
OLD_ID=$(sudo ovs-vsctl get Open_vSwitch . external_ids:system-id 2>/dev/null | tr -d '"')
|
|
echo "$NODE_NAME" | sudo tee /etc/openvswitch/system-id.conf >/dev/null
|
|
sudo ovs-vsctl set Open_vSwitch . external_ids:system-id="$NODE_NAME"
|
|
echo " system-id: $NODE_NAME (pinned via --node-name)"
|
|
|
|
# A renamed chassis cannot register while the row it left behind still
|
|
# holds this node's encap IP: ovn-controller loops on "OVNSB commit
|
|
# failed" and never appears in `ovn-sbctl show`. The stale row owns no
|
|
# state worth keeping — ovn-controller rebuilds everything on register —
|
|
# but it must go before Step 5 restarts the controller under the new name.
|
|
if [ -n "$OLD_ID" ] && [ "$OLD_ID" != "$NODE_NAME" ]; then
|
|
if sudo ovn-sbctl --db="$OVN_REMOTE" --timeout=10 chassis-del "$OLD_ID" 2>/dev/null; then
|
|
echo " stale chassis: removed '$OLD_ID' (renamed to $NODE_NAME)"
|
|
fi
|
|
fi
|
|
else
|
|
CURRENT_ID=$(sudo ovs-vsctl get Open_vSwitch . external_ids:system-id 2>/dev/null | tr -d '"')
|
|
echo " system-id: ${CURRENT_ID:-<unset>} (preserved; no --node-name given)"
|
|
fi
|
|
echo " ovn-remote: $OVN_REMOTE"
|
|
echo " ovn-encap-ip: $ENCAP_IP"
|
|
echo " ovn-encap-type: geneve"
|
|
|
|
# --- Step 5: Start ovn-controller ---
|
|
echo ""
|
|
echo "Step 5: Starting ovn-controller..."
|
|
|
|
# Set ovn-controller file log level to WARN so it doesn't spam the log with
|
|
# connection-retry INFO messages ("OVNSB commit failed") when the SB DB
|
|
# isn't running. Uses a systemd ExecStartPost so it persists across restarts.
|
|
OVN_CTRL_OVERRIDE="/etc/systemd/system/ovn-controller.service.d/log-level.conf"
|
|
sudo mkdir -p "$(dirname "$OVN_CTRL_OVERRIDE")"
|
|
sudo tee "$OVN_CTRL_OVERRIDE" >/dev/null <<'OVERRIDE'
|
|
[Service]
|
|
ExecStartPost=/bin/sh -c 'OVS_RUNDIR=/var/run/ovn exec /usr/bin/ovs-appctl -t ovn-controller vlog/set file:warn'
|
|
OVERRIDE
|
|
sudo systemctl daemon-reload
|
|
echo " ovn-controller log level: file:warn (via systemd drop-in)"
|
|
|
|
sudo systemctl restart ovn-controller
|
|
echo " ovn-controller: started"
|
|
|
|
# Force a full datapath recompute once ovn-controller has connected to the SB.
|
|
# On a fresh single-node RAFT the controller can attach mid-election and program
|
|
# a partial datapath (SNAT ct-commit without output/delivery), then never
|
|
# reconverge. Recomputing after the SB connection is up rebuilds all OpenFlow
|
|
# from the converged Southbound. Compute nodes race a remote SB too, so this
|
|
# runs on every node.
|
|
for i in $(seq 1 30); do
|
|
CTRL_STATUS=$(sudo OVS_RUNDIR=/var/run/ovn ovs-appctl -t ovn-controller \
|
|
connection-status 2>/dev/null)
|
|
if [ "$CTRL_STATUS" = "connected" ]; then
|
|
break
|
|
fi
|
|
echo " Waiting for ovn-controller SB connection... ($i/30)"
|
|
sleep 1
|
|
done
|
|
sudo OVS_RUNDIR=/var/run/ovn ovs-appctl -t ovn-controller inc-engine/recompute \
|
|
2>/dev/null || true
|
|
echo " ovn-controller: forced datapath recompute"
|
|
|
|
# --- Step 6: Sysctl tuning ---
|
|
echo ""
|
|
echo "Step 6: Applying sysctl for overlay networking..."
|
|
|
|
sudo tee /etc/sysctl.d/99-spinifex-vpc.conf >/dev/null <<'SYSCTL'
|
|
# Spinifex VPC networking: enable IP forwarding and disable rp_filter
|
|
# for Geneve overlay traffic on OVS bridges.
|
|
net.ipv4.ip_forward=1
|
|
net.ipv4.conf.all.rp_filter=0
|
|
net.ipv4.conf.default.rp_filter=0
|
|
SYSCTL
|
|
sudo sysctl --system -q
|
|
echo " ip_forward=1, rp_filter=0"
|
|
|
|
# --- Step 6b: Ensure data NIC routing for Geneve tunnels ---
|
|
echo ""
|
|
echo "Step 6b: Configuring data NIC routing for Geneve tunnels..."
|
|
|
|
# When management and data NICs share the same subnet (e.g. both on 10.1.0.0/16),
|
|
# the kernel may route Geneve tunnel traffic through the management NIC with the
|
|
# wrong source IP. This causes remote OVS nodes to drop incoming tunnel packets
|
|
# because the source IP doesn't match the configured tunnel remote_ip.
|
|
# Fix: lower the route metric on the data NIC so it's preferred.
|
|
DATA_IFACE=$(ip -o -4 addr show | awk -v ip="$ENCAP_IP" '$0 ~ ip"/" {print $2}')
|
|
if [ -n "$DATA_IFACE" ]; then
|
|
SUBNET=$(ip -o -4 route show dev "$DATA_IFACE" proto kernel scope link | awk '{print $1}' | head -1)
|
|
if [ -n "$SUBNET" ]; then
|
|
sudo ip route replace "$SUBNET" dev "$DATA_IFACE" src "$ENCAP_IP" metric 50
|
|
echo " data route: $SUBNET via $DATA_IFACE src $ENCAP_IP (metric 50)"
|
|
else
|
|
echo " skipped: no kernel route found for $DATA_IFACE"
|
|
fi
|
|
else
|
|
echo " skipped: could not find interface for $ENCAP_IP"
|
|
fi
|
|
|
|
# --- Step 7: Verify Geneve kernel support ---
|
|
echo ""
|
|
echo "Step 7: Verifying Geneve kernel module..."
|
|
|
|
if sudo modprobe geneve 2>/dev/null; then
|
|
echo " geneve module: loaded"
|
|
else
|
|
echo " WARNING: geneve module not available (tunnels may not work)"
|
|
fi
|
|
|
|
# --- Step 8: Grant the spinifex service users access to OVS/OVN ---
|
|
echo ""
|
|
echo "Step 8: Configuring service-user access..."
|
|
|
|
# Group-scoped, never world. spinifex-daemon and spinifex-vpcd both run with
|
|
# Group=spinifex, so 0660 root:spinifex is exactly the reach they need. A
|
|
# world-writable db.sock hands the local datapath — bridges, ports, ovn-remote —
|
|
# to any account on the box, and a world-writable ctl socket lets any account
|
|
# run `ovn-appctl exit`.
|
|
PERMS_HELPER="/usr/local/lib/spinifex/ovs-socket-perms.sh"
|
|
sudo mkdir -p "$(dirname "$PERMS_HELPER")"
|
|
sudo tee "$PERMS_HELPER" >/dev/null <<'HELPER'
|
|
#!/bin/sh
|
|
# Group-own the OVS/OVN control sockets to `spinifex` so the service users reach
|
|
# them without sudo. Driven from ExecStartPost on both openvswitch-switch and
|
|
# ovn-controller: each recreates its own sockets on start, and the ovn-controller
|
|
# ctl socket name embeds the pid, so it is a new file every time.
|
|
#
|
|
# $1 is an optional glob for the caller's own socket, waited on before the sweep.
|
|
set -eu
|
|
GROUP=spinifex
|
|
WAIT_FOR="${1:-}"
|
|
|
|
if ! getent group "$GROUP" >/dev/null 2>&1; then
|
|
exit 0
|
|
fi
|
|
|
|
# Unquoted so the caller's pattern globs. A caller passing nothing sweeps at once.
|
|
wait_target_present() {
|
|
if [ -z "$WAIT_FOR" ]; then
|
|
return 0
|
|
fi
|
|
for s in $WAIT_FOR; do
|
|
if [ -S "$s" ]; then
|
|
return 0
|
|
fi
|
|
done
|
|
return 1
|
|
}
|
|
|
|
# ExecStartPost can outrun socket creation, so poll briefly rather than miss the
|
|
# socket and leave the health probe unable to read connection-status.
|
|
i=0
|
|
while [ "$i" -lt 25 ]; do
|
|
if wait_target_present; then
|
|
break
|
|
fi
|
|
i=$((i + 1))
|
|
sleep 0.2
|
|
done
|
|
|
|
# The dir must be traversable by the group before the sockets inside it matter.
|
|
# Only the group changes; the owner keeps full access whoever it is.
|
|
if [ -d /var/run/ovn ]; then
|
|
chgrp "$GROUP" /var/run/ovn 2>/dev/null || true
|
|
chmod 0750 /var/run/ovn 2>/dev/null || true
|
|
fi
|
|
|
|
# ovn??_db.sock are the NB/SB databases themselves, which ovn-nbctl and
|
|
# ovn-sbctl connect to. Bridge <name>.mgmt sockets are deliberately NOT here:
|
|
# ovs-vswitchd creates one whenever a bridge appears, including bridges spinifex
|
|
# creates at runtime, so ovs-ofctl keeps its sudo grant.
|
|
for s in /var/run/openvswitch/db.sock /var/run/openvswitch/*.ctl \
|
|
/var/run/ovn/*.ctl /var/run/ovn/ovnnb_db.sock /var/run/ovn/ovnsb_db.sock; do
|
|
if [ -S "$s" ]; then
|
|
chgrp "$GROUP" "$s" 2>/dev/null || true
|
|
chmod 0660 "$s" 2>/dev/null || true
|
|
fi
|
|
done
|
|
|
|
# Group-WRITE on the pidfiles, not just read: `ovn-appctl -t <daemon>` resolves
|
|
# the pid-named ctl socket through the pidfile, and OVS opens it O_RDWR to test
|
|
# the fcntl liveness lock. Read-only fails the probe with EACCES. This also
|
|
# tightens them from the 0644 they ship with.
|
|
for p in /var/run/openvswitch/*.pid /var/run/ovn/*.pid; do
|
|
if [ -f "$p" ]; then
|
|
chgrp "$GROUP" "$p" 2>/dev/null || true
|
|
chmod 0660 "$p" 2>/dev/null || true
|
|
fi
|
|
done
|
|
HELPER
|
|
sudo chmod 0755 "$PERMS_HELPER"
|
|
sudo "$PERMS_HELPER"
|
|
echo " OVS/OVN sockets: 0660 root:spinifex (group-scoped, no world access)"
|
|
|
|
# Persist across restarts of both daemons. Rewritten unconditionally: an existing
|
|
# file is the old 0666 override, and skipping would leave that exposure in place.
|
|
# openvswitch-ipsec is included because Step 10 restarts it after this sweep,
|
|
# so its ctl socket would otherwise be the one file left at the shipped mode.
|
|
for unit in openvswitch-switch:/var/run/openvswitch/db.sock \
|
|
"ovn-controller:/var/run/ovn/*.ctl" \
|
|
"openvswitch-ipsec:/var/run/openvswitch/ovs-monitor-ipsec.*.ctl"; do
|
|
UNIT="${unit%%:*}"
|
|
WAIT_GLOB="${unit#*:}"
|
|
OVERRIDE_DIR="/etc/systemd/system/${UNIT}.service.d"
|
|
sudo mkdir -p "$OVERRIDE_DIR"
|
|
# `ExecStartPost=-` so a failure here cannot stop the unit. This drop-in goes
|
|
# into another package's service, and tightening a socket mode is hardening,
|
|
# not a liveness requirement — it must never be the reason openvswitch-switch
|
|
# or ovn-controller will not start. Without the dash, removing the helper
|
|
# (an uninstall, a partial re-run that has not reached this step yet) takes
|
|
# OVS down across the whole host with `Failed at step EXEC`, which reads as
|
|
# an OVS fault and is nothing of the sort.
|
|
sudo tee "$OVERRIDE_DIR/spinifex-perms.conf" >/dev/null <<OVERRIDE
|
|
[Service]
|
|
ExecStartPost=-$PERMS_HELPER "$WAIT_GLOB"
|
|
OVERRIDE
|
|
echo " systemd override: ${UNIT}.service.d/spinifex-perms.conf"
|
|
done
|
|
sudo systemctl daemon-reload
|
|
|
|
# Sudoers rules for spinifex-daemon and spinifex-vpcd are managed by setup.sh
|
|
# (install_sudoers). Skip writing here to avoid conflicts.
|
|
SUDOERS_FILE="/etc/sudoers.d/spinifex-network"
|
|
if [ -f "$SUDOERS_FILE" ]; then
|
|
echo " sudoers rule: already exists ($SUDOERS_FILE, managed by setup.sh)"
|
|
else
|
|
echo " sudoers rule: not found — run setup.sh first, or install manually"
|
|
fi
|
|
|
|
# --- Step 9: Configure OVN log rotation ---
|
|
# The ovn-common package provides /etc/logrotate.d/ovn-common which handles
|
|
# rotation and vlog/reopen. We just add maxsize + rotate to cap disk usage.
|
|
echo ""
|
|
echo "Step 9: Configuring OVN log rotation..."
|
|
|
|
OVN_LOGROTATE="/etc/logrotate.d/ovn-common"
|
|
if [ -f "$OVN_LOGROTATE" ]; then
|
|
if ! grep -q 'maxsize' "$OVN_LOGROTATE"; then
|
|
sudo sed -i '/^\/var\/log\/ovn\/\*\.log {/a\ rotate 5\n maxsize 100M' "$OVN_LOGROTATE"
|
|
echo " added maxsize 100M + rotate 5 to $OVN_LOGROTATE"
|
|
else
|
|
echo " $OVN_LOGROTATE already has maxsize configured"
|
|
fi
|
|
else
|
|
echo " WARNING: $OVN_LOGROTATE not found — install ovn-common package"
|
|
fi
|
|
|
|
# Remove our old custom config if present (superseded by patching ovn-common)
|
|
if [ -f /etc/logrotate.d/ovn-spinifex ]; then
|
|
sudo rm -f /etc/logrotate.d/ovn-spinifex
|
|
echo " removed obsolete /etc/logrotate.d/ovn-spinifex"
|
|
fi
|
|
|
|
# --- Step 10: Enable auto-start on boot ---
|
|
# OVN services should start with the system in production. ovn-controller
|
|
# retries when the SB DB isn't ready; file log level is set to WARN (Step 5)
|
|
# to prevent log spam during those retries.
|
|
echo ""
|
|
echo "Step 10: Enabling OVN auto-start on boot..."
|
|
sudo systemctl enable openvswitch-switch 2>/dev/null || true
|
|
sudo systemctl enable ovn-controller 2>/dev/null || true
|
|
# ovs-monitor-ipsec drives strongSwan from OVS DB cert pointers. The daemon's
|
|
# enableOVNIPSec() flips ipsec_encapsulation=true at runtime and silently drops
|
|
# tunnel traffic if this unit isn't already up — enable at provision time so
|
|
# daemon never needs systemd-write capability (only is-active read).
|
|
sudo systemctl enable openvswitch-ipsec.service 2>/dev/null || true
|
|
# When openvswitch-ipsec is installed before strongswan-starter, the deb
|
|
# post-install starts the service against a missing /usr/sbin/ipsec and
|
|
# leaves it in 'failed' state. Clear the failure and restart unconditionally
|
|
# now that strongswan-starter is present.
|
|
sudo systemctl reset-failed openvswitch-ipsec.service 2>/dev/null || true
|
|
sudo systemctl restart openvswitch-ipsec.service 2>/dev/null || true
|
|
echo " openvswitch-switch: enabled on boot"
|
|
echo " ovn-controller: enabled on boot"
|
|
echo " openvswitch-ipsec: enabled on boot"
|
|
|
|
# --- Step 11: Health check ---
|
|
echo ""
|
|
echo "Step 11: Verifying setup..."
|
|
|
|
OK=true
|
|
|
|
# Check br-int
|
|
if sudo ovs-vsctl br-exists br-int; then
|
|
echo " br-int: OK"
|
|
else
|
|
echo " br-int: FAILED"
|
|
OK=false
|
|
fi
|
|
|
|
# Check WAN bridge (only if configured)
|
|
if [ -n "$WAN_BRIDGE" ]; then
|
|
if sudo ovs-vsctl br-exists "$WAN_BRIDGE"; then
|
|
echo " $WAN_BRIDGE:$(printf '%*s' $((15 - ${#WAN_BRIDGE})) '') OK"
|
|
if [ "$WAN_BRIDGE_MODE" = "veth" ]; then
|
|
if ip link show veth-wan-br >/dev/null 2>&1 && ip link show veth-wan-ovs >/dev/null 2>&1; then
|
|
echo " veth pair: OK (veth-wan-br ↔ veth-wan-ovs)"
|
|
echo " linux bridge: $LINUX_BRIDGE (untouched)"
|
|
else
|
|
echo " veth pair: FAILED (veth-wan-br/veth-wan-ovs not found)"
|
|
OK=false
|
|
fi
|
|
elif [ "$WAN_BRIDGE_MODE" = "direct" ]; then
|
|
if sudo ovs-vsctl port-to-br "$WAN_IFACE" >/dev/null 2>&1; then
|
|
echo " direct bridge: OK ($WAN_IFACE on $WAN_BRIDGE)"
|
|
else
|
|
echo " direct bridge: FAILED ($WAN_IFACE not on $WAN_BRIDGE)"
|
|
OK=false
|
|
fi
|
|
fi
|
|
else
|
|
echo " $WAN_BRIDGE:$(printf '%*s' $((15 - ${#WAN_BRIDGE})) '') FAILED"
|
|
OK=false
|
|
fi
|
|
fi
|
|
|
|
# Check mgmt bridge (only if enabled)
|
|
if [ "$MGMT_BRIDGE_ENABLED" = true ]; then
|
|
if sudo ovs-vsctl br-exists "$MGMT_BRIDGE"; then
|
|
MGMT_IP_ACTUAL=$(ip -4 -o addr show dev "$MGMT_BRIDGE" 2>/dev/null | awk '{print $4}' | head -1)
|
|
echo " $MGMT_BRIDGE:$(printf '%*s' $((15 - ${#MGMT_BRIDGE})) '') OK (${MGMT_IP_ACTUAL:-no IP})"
|
|
else
|
|
echo " $MGMT_BRIDGE:$(printf '%*s' $((15 - ${#MGMT_BRIDGE})) '') FAILED"
|
|
OK=false
|
|
fi
|
|
fi
|
|
|
|
# Check ovn-controller
|
|
if sudo ovs-appctl -t ovn-controller version >/dev/null 2>&1 || systemctl is-active --quiet ovn-controller 2>/dev/null; then
|
|
echo " ovn-controller: OK"
|
|
else
|
|
echo " ovn-controller: FAILED (may still be starting)"
|
|
OK=false
|
|
fi
|
|
|
|
# Check chassis registration (may take a moment)
|
|
if [ "$MANAGEMENT" = true ]; then
|
|
sleep 2
|
|
CHASSIS_COUNT=$(sudo ovn-sbctl show 2>/dev/null | grep -c "Chassis" || true)
|
|
echo " chassis count: $CHASSIS_COUNT"
|
|
fi
|
|
|
|
echo ""
|
|
if [ "$OK" = true ]; then
|
|
echo "=== OVN compute node setup complete ==="
|
|
else
|
|
echo "=== Setup completed with warnings (check above) ==="
|
|
fi
|