mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
dev
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
3415d9f3de |
fix(network): IGW route-table AWS parity + per-subnet egress gating (#270)
* fix(network): IGW default route AWS-parity via per-subnet LR policy
ADR-0006's IGWManager.AttachIGW installed a VPC-router-wide 0.0.0.0/0
static route at attach time, violating AWS semantics: every subnet on
the LR saw egress to the IGW regardless of its route table. Private
subnets that should be IGW-isolated were not.
Move IGW egress lifecycle onto the route-table API. CreateRoute(IGW,
0.0.0.0/0) emits vpc.add-igw-route per associated subnet; the network
subscriber installs an OVN Logical_Router_Policy matching
`inport == "rtr-<subnetID>" && ip4.dst == 0.0.0.0/0` with action
`reroute` via the IGW gateway port. DeleteRoute / DisassociateRouteTable
/ ReplaceRouteTableAssociation tear it down. AssociateRouteTable emits
add events so the terraform ordering (CreateRoute -> Associate) is
honored.
Additions:
- nbdb.LogicalRouterPolicy + Logical_Router_Policy registration
- ovn.Client {Add,Delete,Find,List}LogicalRouterPolicy + live + mock
- policy.RouteManager.{Add,Delete}SubnetEgress with priority constants
- external.IGWManager.{Ensure,Remove}SubnetEgress resolving nexthop +
output port from the gateway state
- subscribers handleAddIGWRoute / handleDeleteIGWRoute on
vpc.{add,delete}-igw-route
- routetable publishIGWRouteEvents / *ForAssociation / *ForSubnet
Test updates:
- network/policy: AddSubnetEgress idempotency, drift-replace, per-subnet
separation, DeleteSubnetEgress
- network/external: AttachIGW must NOT install a router-wide default
route; EnsureSubnetEgress installs the per-subnet policy with the
resolved nexthop (link-local fallback + allocator override paths)
- handlers/ec2/routetable: CreateRoute(IGW), Associate, DeleteRoute,
Disassociate publish events with the expected subnetID + igwID
- ovn/nbdb: table-count + Logical_Router_Policy presence
- subscribers: TestSubscribe_RegistersAllTopics covers the two new topics
* fix(network): NATGW per-subnet egress + DeleteRoute event symmetry
handlers/ec2/routetable:
- DeleteRoute now publishes vpc.delete-nat-gateway per associated subnet
(mirror of the IGW DeleteRoute path) so the subscriber tears down both
the SNAT rule and the per-subnet egress policy.
- publishNatGatewayEvent* signatures carry SubnetId + DestinationCidr so
the subscriber can install the matching Logical_Router_Policy.
network/external:
- IGWManager: EnsureNATGatewaySubnetEgress / RemoveNATGatewaySubnetEgress
install/remove a per-subnet LR policy at priority SubnetEgressPriorityNATGW.
Same nexthop and output port as the IGW priority sibling; lower priority
so an IGW route wins when both apply to the same subnet.
- EnsureSubnetEgress delegates to ensureSubnetEgressAtPriority for shared
validation, lookup, and slog tagging.
network/subscribers:
- handleAddNATGateway: after SNAT install, parse DestinationCidr (defaults
to 0.0.0.0/0) and call EnsureNATGatewaySubnetEgress. Without this the LR
has no route to egress NAT'd packets and private-subnet traffic is
dropped (E2E Phase 8d regression).
- handleDeleteNATGateway: remove the per-subnet policy FIRST, then detach
SNAT, so packets are never routed unmasqueraded after SNAT removal.
- NATGatewayEvent gains SubnetId + DestinationCidr.
Tests:
- policy.AddSubnetEgress validation + drift-replace + propagation paths.
- mock LR-policy methods reject missing routers and tolerate dangling UUIDs.
- subscribers cover NATGW add/delete with invalid CIDR, bad JSON, missing
router, IGW-route bad JSON + invalid CIDR. Full end-to-end add+delete
via NATS asserting policy lifecycle.
- IGW manager covers RemoveSubnetEgress / RemoveNATGatewaySubnetEgress
empty-arg rejection + delete error propagation.
- routetable: DeleteRoute(NATGW) publishes per associated subnet.
* fix(network): exclude VPC CIDR from per-subnet egress reroute predicate
In-VPC traffic (e.g. bastion -> private SSH) was being intercepted by the
NATGW per-subnet egress LR policy on `ip4.dst == 0.0.0.0/0` and forwarded
out the gateway with SNAT, breaking peer-subnet connectivity. Append
`&& ip4.dst != <vpcCIDR>` to the match so in-VPC destinations skip the
reroute and follow the directly-connected route. Apply to both IGW (1000)
and NATGW (900) priorities. CIDR is read from the LR ExternalIDs label
populated at VPC create.
DeleteSubnetEgress takes the same excludes so the match string is
identical on cleanup. igwManager.vpcExcludeCIDRs fetches from the router
each call; lookup failure returns nil and the policy installs without
exclusions (preserves prior behaviour as fallback).
* fix(routetable): publish per-subnet events for implicit main-RT membership
E2E run 26620082197 RCA: subnets with no explicit RT association inherit
the VPC main route table per AWS semantics, but publishIGWRouteEvents and
publishNatGatewayEvents iterated only record.Associations and skipped
Main:true/SubnetId:"" sentinels. Main RT carries exactly one such
sentinel, so CreateRoute(igw)/CreateRoute(natgw) on the main RT fanned
out to ZERO subnets — subscriber never called EnsureSubnetEgress, OVN
LR policy never installed, ping 8.8.8.8 from default-VPC VMs dropped at
the VPC router. Caused TestVPCSubnetE2E, TestNATGateway,
TestMultinodeSpread/NATGatewayInternet to fail.
Add subnetsImplicitlyOnMainRT helper that enumerates VPC subnets
without explicit non-main associations, and fan publish to them when
record.IsMain. Apply to all three publish helpers (IGW, NATGW,
NATGW-delete).
Symmetric fixes for effective-RT changes:
- AssociateRouteTable: subnet leaves implicit main → emit deletes for
main RT's routes before adds for explicit RT
- DisassociateRouteTable: subnet returns to implicit main → emit adds
for main RT's routes after deletes for departing RT
* chore(e2e): capture per-LR lr-route-list/lr-policy-list/lr-nat-list
ovn-nbctl show does not dump static routes or policies. Adds three
per-LR captures so RCA can verify policy/route install state after
the IGW per-subnet refactor.
* chore(e2e): tcpdump br-wan + neigh/fdb/conntrack capture during tests
EIP egress regression on feat/igw-route-table-aws-parity shows SNAT
packets present (n_packets>0) but pings still time out. Post-teardown
ovn.* snapshots can't tell whether replies actually arrived on br-wan
or got dropped at L2 / by upstream router.
start-net-capture.sh: nohup tcpdump on br-wan filtered to 192.168.0.0/23
(EIP pool) + ARP for same range, 200-byte snaplen, 64 MiB cap. Pair:
capture-net-diag.sh stops it via /tmp/spinifex-e2e-tcpdump.pid and copies
wan.pcap into net-diag/.
Also added neigh/bridge-fdb/conntrack dumps to capture-net-diag.sh so
L2/L3 binding state is visible alongside the existing OVN dumps.
Wired both single-VM and multinode workflows to start capture before
the e2e run; teardown step picks up the pcap automatically.
* fix(reconcile): rebuild IGW/NATGW per-subnet egress from KV
Publisher fires vpc.add-igw-route during default-VPC bootstrap before the
subscriber daemon attaches to the topic. NATS core pub/sub drops the
message, leaving the LR without a 0.0.0.0/0 policy and off-subnet egress
broken on the affected VPC for the lifetime of the process. Reproduced
in CI: TestVPCSubnetE2E bastion EIP could ARP and DNS its on-LAN
neighbour but never reach 8.8.8.8.
LoadIntentFromKV now fans every IGW or NAT-gateway RT route out over
its explicit + implicit-main subnets and the reconcile pass invokes
EnsureSubnetEgress / EnsureNATGatewaySubnetEgress for each one. Same
shape as publishIGWRouteEvents so the bootstrap and the runtime event
path produce identical OVN state.
* fix(routes): serialize OVN egress wiring via request-reply
vpc.igw-attach, vpc.add-igw-route and vpc.add-nat-gateway were fire-and-forget
Publish, racing the subscriber's AttachIGW barrier. AttachInternetGateway
returned before vpcd finished wiring the OVN gateway LRP, so a follow-up
CreateRoute(IGW) published an add-igw-route event that the subscriber
processed in a parallel goroutine — installing a Logical_Router_Policy with a
nexthop the LRP didn't yet expose. northd compiled an inert reroute that
never recovered, leaving the bastion's egress traffic blackholed (no packets
dst 8.8.8.8 from the failing VPC's EIP even though local DNS succeeded).
Switch the three add-* publishers to utils.RequestEvent. The subscriber ACKs
once OVN state is committed, so the gateway handler can't issue subsequent
route events against a half-wired router. delete-* paths stay Publish —
teardown ordering doesn't matter.
handleAddNATGateway now calls respond() at every exit so requesters don't
hang. Route-table tests gain an ackSub wrapper (async subscribe + auto-Reply)
so request-based publishers complete during unit tests.
* Revert "fix(routes): serialize OVN egress wiring via request-reply"
This reverts commit bf6f77d5bc8b21f72ed5867175d5a67c010066c8.
* fix(igw): restore default static route in AttachIGW/DetachIGW
OVN 25.03 LR pipeline runs lr_in_ip_routing (table 15) BEFORE lr_in_policy
(table 17). When this branch swapped the per-VPC default static route for
the per-subnet Logical_Router_Policy reroute installed by EnsureSubnetEgress,
external destinations started dropping at routing priority-0 before the
policy table ran. Reproduced locally: ovn-trace stops at table 15 drop, ping
8.8.8.8 from inside a fresh-VPC bastion produced zero packets on br-wan
until `ovn-nbctl lr-route-add <router> 0.0.0.0/0 <wan-gw> <gw-port>` was
issued manually — after which the same ping succeeded immediately. dev still
green because dev calls AddDefaultRoute on AttachIGW.
Restore the original AddDefaultRoute / DeleteDefaultRoute calls; keep the
per-subnet EnsureSubnetEgress machinery intact so future per-subnet egress
gating (DROP policies for subnets whose RT lacks a 0.0.0.0/0 route) can
layer on top. Flip the two TestAttachIGW assertions that pinned the
"AttachIGW must not install a default static route" expectation.
Pairs with revert of bf6f77d5 in the previous commit (request-reply
serialization treated a non-existent race; real bug was missing routing
fallback).
* feat(network): per-subnet egress drop gating for AWS RT parity
VPC LRs carry a router-wide 0.0.0.0/0 default static route while an IGW is
attached (lr_in_ip_routing runs before lr_in_policy on OVN 25.03, so per-
subnet reroute alone is insufficient). Subnets whose effective route table
lacks 0.0.0.0/0 were previously able to egress via that default route,
violating AWS semantics.
Install an LR_Policy DROP at SubnetEgressPriorityDrop=1100 (above IGW=1000
and NATGW=900) for unrouted subnets. Match excludes VPC CIDR, link-local
(169.254/16) and multicast (224/4) so intra-VPC, DHCP/metadata and
multicast still flow.
Wiring:
- policy.RouteManager gains AddSubnetEgressDrop / DeleteSubnetEgressDrop.
- external.IGWManager gains EnsureSubnetEgressDrop / RemoveSubnetEgressDrop.
- New events: SubnetEgressGateEvent / SubnetEgressUngateEvent on topics
vpc.gate-subnet-egress / vpc.ungate-subnet-egress.
- routetable handler publishes gate/ungate alongside existing add/delete
IGW/NATGW route events on every CreateRoute/DeleteRoute/Associate/
Disassociate/Replace mutation, based on subnet's recomputed effective RT.
- reconciler bootstrap path emits DropGates intent for subnets in IGW-
attached VPCs whose effective RT lacks 0.0.0.0/0.
Tests cover primitive idempotency, drift recovery, event publishing per
mutation surface, implicit-main fan-out, and intent computation.
* fix(igw): fan out subnet egress gate decisions on IGW attach/detach
The reconciler computes per-subnet DROP policies for any subnet in an
IGW-attached VPC whose effective route table lacks 0.0.0.0/0, but those
decisions were only re-evaluated by the periodic drift tick (every
5 min). AttachInternetGateway / DetachInternetGateway now publish
gate/ungate events for every subnet in the VPC immediately, closing
the window where a freshly-attached IGW could leave unrouted subnets
unprotected.
Wiring: IGWServiceImpl gains a nil-safe GatePublisher hook satisfied
by RouteTableServiceImpl. Daemon installs the hook once both services
exist. RouteTableServiceImpl.PublishGateDecisionsForVPC enumerates the
VPC's subnets and reuses publishSubnetEgressGateDecision per subnet.
Also fixes the LB e2e suite, which built VPC + IGW + subnet without
ever creating a 0.0.0.0/0 → IGW route — relying on the pre-DROP-gate
behaviour where unrouted subnets still had silent egress via the
router-wide static route. The fixture now installs the explicit
public route on the main RT (AWS-faithful semantics for a public
subnet).
* fix(igw): skip gate fan-out on attach to avoid bootstrap race
AttachInternetGateway fan-out races with the default-VPC bootstrap path
in ensureDefaultVPCInfrastructureFor, which attaches the IGW and
immediately publishes a 0.0.0.0/0 → IGW CreateRoute event. The two
events land on distinct NATS subjects (vpc.gate-subnet-egress vs
vpc.ungate-subnet-egress) with no cross-subject ordering guarantee.
When ungate arrived first the subscriber became a no-op and the
subsequent gate installed a permanent DROP on the default subnet,
killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to the instance public IP).
Detach fan-out is retained — no follow-up CreateRoute fires after
DetachInternetGateway so the path is race-free.
Subnets without a default route still get DROPped during the
reconciler's periodic drift pass (loadSubnetDropGates). The window is
re-opened to ≤5 min on user-driven AttachInternetGateway, which is
the original behaviour pre-mulga-siv-163. A follow-up bead can close
that gap properly by converting the gate/ungate subjects into a
single recompute trigger that re-reads effective RT state.
* fix(vpc): prevent duplicate main route table from bootstrap race
Concurrent multi-node daemons running EnsureDefaultVPC for the admin
account (bootstrap-pinned VpcId) each minted a fresh random rtb-ID and
Put it with IsMain=true. The KV ended up with two main route table
records for the same VPC.
CreateRoute's post-write gate recompute (per-subnet egress drop gating)
then resolved effectiveRouteTable non-deterministically across the two
IsMain=true records; when the orphan (route-less) RT won, the gate
decision became "no default egress" and vpc.gate-subnet-egress fired.
The subscriber installed a Priority-1100 DROP on the default subnet's
egress, killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to 192.168.0.161 in run
26699045536).
Two-part fix:
1. createMainRouteTable scans rtbKV for an existing IsMain=true record
on the same VPC before Put-ing a new one. Race window narrows from
"always duplicates under concurrent bootstrap" to a sub-ms TOCTOU
between scan and Put.
2. effectiveRouteTable / mainRouteTable use a deterministic preferMain
tiebreak when multiple IsMain=true records exist: more routes >
oldest CreatedAt > smaller RouteTableId. Defensive — any duplicate
that slips through (cross-AZ restore, manual KV edit, residual race)
still produces consistent egress gate decisions instead of flapping
on KV scan order.
* fix(policy): atomic ACL replacement for SG mutations
Add ovn.Client.ReplaceACLs — detach + delete existing ACL rows and
attach freshly-created rows in one OVSDB transaction. EnsureSG and
UpdateSG now funnel through ReplaceACLs instead of ClearACLs +
AddACLs, closing the mid-flight window where the port group held
zero ACLs and tenant LSP traffic defaulted to drop (observable on
ICMP and on TCP SYNs that miss conntrack).
Mock client exposes ClearACLCalls / ReplaceACLCalls counters so
tests can assert atomicity. New TestSGManager_UpdateSG_AtomicReplace
fails if SG mutations ever fall back to the split path.
|
||
|
Josh Siviour
|
fd3dc6b680 |
feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255)
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
|