mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
fd3dc6b680
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
aed7b2e (mulga-siv-133), which made handleSubnetCreate call EnsureVPC
idempotently before EnsureSubnet to avoid a NATS publish-order race.
Delete the now-impossible failure case.
* refactor(network/ovn): relocate nbdb model package from services/vpcd
Phase 4 / mulga-siv-125.6 Slice A. Pure import rename, no logic change.
The OVN Northbound DB model (LogicalSwitch, LogicalRouter, DHCPOptions,
etc.) lived under services/vpcd/nbdb/ because that was the original home
of every OVN caller. After Phase 1+2 the L1 client (network/ovn) and all
L3/L5 managers (network/policy, network/external, network/topology,
network/reconcile) became the primary callers — 8 importers in network/*
vs 2 in services/vpcd/. The directory move puts the schema model where
its consumers are, and unblocks deleting services/vpcd/ entirely once
the subscriber/launcher relocations land (Slices B–D).
* refactor(vpcd): delete dead retrofit/reconcileIGW helpers + bridge-mode bleed
Phase 4 / mulga-siv-125.6 Slice B.
Production path no longer touches any of:
- RetrofitAllGatewayPortNetworks / RetrofitAllExternalLocalnetOptions —
the 5-pass startup leftovers that network/reconcile (Phase 2.2) made
redundant. Nothing called them outside tests.
- reconcileIGW + its allocateGatewayLRPIP / gwLrpRange helpers + the
ipv4ToUint32 / uint32ToIPv4 utilities — superseded by
external.IGWManager (Phase 2.4) which carries its own pool logic.
- ensureLocalnetOptions / ensureGatewayPortNetworks /
expectedGatewayPortNetwork — read-before-write retrofit shims for the
same code paths.
- generateMAC — only callers were reconcileIGW + its tests.
useCentralizedNAT (the bridge-mode bleed flagged in plan §2.2) is
replaced by a policy.NATMode value injected via WithNATMode(...). The
mode is resolved once in vpcd.go from host.UplinkMode and threaded into
both the topology-handler lazy managers and the reconciler stack, so
"is this veth?" is no longer a question topology.go can ask. The
TopologyHandler defaults to NATModeDistributed when WithNATMode is
omitted, matching the previous default.
Drops services/vpcd/topology.go from 1220 LOC to 693, deletes ~700 LOC
of obsolete tests (TestGenerateMAC, TestEnsureLocalnetOptions_*,
TestRetrofitAllExternalLocalnetOptions_FixesClearedOptions,
TestReconcileIGW_*, TestGwLrpRange), and frees the nbdb + utils +
net imports in topology.go.
Stale "phase 4 migrates those onto the new managers" comment in vpcd.go
fixed in passing — the subscribers were migrated in Phase 2.6
(mulga-siv-129); only the file-relocation parts of Phase 4 remain.
Stale ensureLocalnetOptions / ensureGatewayPortNetworks references in
network/ovn/{live.go,mock/mock.go} doc comments also cleaned up.
* refactor(network): move NATS VPC subscribers to network/subscribers
Phase 4 Slice C (mulga-siv-125.6). Promotes the NATS event handlers out of
services/vpcd into a new spinifex/network/subscribers package. Subscriber
holds the topology/policy/external managers directly — the lazy *Once
adapter shim (topology_manager.go) is deleted along with vpcd's local
event-struct copies.
launchService now constructs topology.LiveManager eagerly with a DNS
callback derived from external pools (preserving the unscoped-pool lookup
old TopologyHandler.lmDNSServer did), builds every manager up front, then
hands them to subscribers.New + Subscribe. reconcile.New receives
topology.Manager directly instead of through the adapter.
Bridge-mode constants and waitForFlowsHV move into vpcd.go alongside the
launcher; they were left behind as topology.go's last responsibilities.
Handler-layer tests (topology_test.go, security_test.go, integration_test.go)
are dropped: substance is covered by network/topology, network/policy,
network/external manager tests. subscribers/subscribers_test.go adds
focused coverage: missing-dep rejection, topic registration smoke, and
bad-JSON envelope per topic.
services/vpcd is now down to the launcher + the ovn/mock compat aliases
(removed in Slice D when the launcher promotes to cmd/vpcd/).
* refactor(vpcd): relocate services/vpcd → spinifex/vpcd; drop dead aliases
Phase 4 Slice D (mulga-siv-125.6). Final cleanup of the network-redesign
package layout. services/vpcd/ disappears entirely; the launcher remains
an in-process Service impl, just rehomed under spinifex/vpcd/ (no separate
binary — spx vpcd start stays the cobra entrypoint).
- Delete services/vpcd/mock_ovn.go + ovn.go (compat aliases now dead post
Slice C; only callsite was launchService's NewLiveOVNClient, inlined to
ovn.NewLiveClient).
- git-rename services/vpcd/vpcd.go + vpcd_test.go to spinifex/vpcd/.
- Retarget three importers (spinifex/service/service.go +
service_test.go, cmd/spinifex/cmd/service.go) to the new path.
- Fix stale services/vpcd / TopologyHandler comments in network/reconcile/
+ network/external/types.go + spinifex/vpcd/vpcd.go.
- service-interfaces.yaml vpcd path updated.
- Plan doc §13/§14/§16 + §9.2/§19 updated: drop cmd/vpcd/main.go (was
speculative; never built) in favour of in-process Service impl at
spinifex/vpcd/vpcd.go. Plan §16 Created list now lists subscribers/.
Phase 4 plan items 1+2 (remove services/vpcd, entrypoint location)
satisfied. Items 3+4 remain blocked on mulga-siv-125.4 + 125.5 (bundled
join token, plan archive).
* fix(ovn): install openvswitch-ipsec + strongswan explicitly
ovs-monitor-ipsec.service fails to start without an IKE daemon. The
openvswitch-ipsec package only Recommends strongswan, so any apt
config that drops Recommends (or a base image with the recommends
already skipped) leaves the service in a Restart loop with 'IKE
daemon is not installed'. Hoist both packages into setup-ovn.sh so
provisioning is the single source of truth — the bootstrap shim in
mulga is no longer needed.
* test(network): pin ADR-0006 invariants to executable suite
Add static-analysis + behavioural tests that bind every safety and
liveness clause in ADR-0006 (S1-S9, L1-L3) to a named Go test. Failure
output quotes the ADR clause verbatim and points at the offending
file:line so future debugging sessions can't quietly bypass a layer
contract to make a test green.
Static-analysis suite (spinifex/network/invariants/):
- S1 layers_test.go: parses `go list -json`; rejects upward layer
imports and layers pulling from cross-cutter packages
- S2 host_containment_test.go: AST-detects ovs-vsctl / ovn-nbctl /
ovs-ofctl shell-out outside network/host/
- S3 nat_mode_test.go: rejects .UplinkMode() / NATModeFromUplinkMode()
calls outside the init-time resolver locations
- S4 names_test.go: AST-detects OVN object name construction (concat,
fmt.Sprintf, strings.Join) outside topology/names.go
- S5 policy_no_create_test.go: rejects CreateLogical{Switch,Router,...}
and DeleteLogical{...} calls inside network/policy/
- S6 federation_isolation_test.go: rejects TransitSwitch /
TransitRouterPort references and ovn-ic literals outside
network/federation/ (pre-protects the layer before it lands)
- S8 ipsec_test.go: rejects strongswan / xfrm / IKEv2 / libreswan /
openswan imports anywhere in network/
- S9 reconciler_singleton_test.go: rejects Reconcile…Loop and
Reconcile…Pass<n> function names anywhere in network/
- S7 nats_subjects_test.go: scans for `vpc.<verb>` subjects missing
the {azID} segment; calls t.Skip with the gap report so every test
run prints the work-to-do without breaking CI. Flip to t.Fatalf
when the cluster-wide subject rename ships.
Behavioural suite (spinifex/network/reconcile/liveness_test.go):
- L1 reconciler converges: scaled-intent (N=1,3,8) double-Reconcile
property test; asserts second pass is a no-op
- L2 drift bound: structural check that DriftInterval is positive and
≤30 min support window; full fake-clock behavioural test deferred
- L3 federation re-enables: t.Skip stub pointing at the missing
network/federation/ package
Fixes the one production S4 violation the new walker found:
network/external/igw.go was constructing "gw-port-"+vpcID inline.
Adds topology.GatewaySwitchPort(vpcID) helper and retargets both
AttachIGW and DetachIGW callsites through L2.
Plan + per-clause mechanic + phasing in
docs/development/improvements/network-invariants.md.
Refs: mulga-siv-140
* fix(ovn): install strongswan-charon (not strongswan metapackage)
The strongswan metapackage's '\''charon-systemd | strongswan-charon'\''
alternative resolves to charon-systemd by default, which ships
/usr/sbin/charon-systemd but not the legacy /usr/sbin/ipsec helper or
/usr/lib/ipsec/charon binary that ovs-monitor-ipsec --ike-daemon=strongswan
shells out to. Install strongswan-charon explicitly to pull
strongswan-starter (legacy ipsec controller) + the matching charon binary,
matching what openvswitch-ipsec.service expects.
Also reset-failed + restart openvswitch-ipsec.service after enable to
recover from the deb post-install starting the service before its
dependencies finished unpacking.
* chore: add CODEOWNERS — joshsiv-mulga gates network/ + vpcd/
Fall-through reviewer for any change is the maintainer.
network/ and vpcd/ are explicitly listed so the matching branch
protection rule ("Require review from Code Owners") routes those
PRs to joshsiv-mulga. Those trees are governed by ADR-0006
invariant tests (spinifex/network/invariants/); CODEOWNERS adds a
human review gate on top of the mechanical fence.
Refs: mulga-siv-141
* chore: CODEOWNERS fallthrough → @mulgadc/mulgadc-engineers team
Fallthrough reviewer should be the team, not a single maintainer, so
any engineer can approve PRs outside the explicitly-gated paths.
network/ and vpcd/ keep joshsiv-mulga as the required reviewer per
ADR-0006 invariant ownership.
Refs: mulga-siv-141
* feat(ci): add test_run workflow input for -test.run filter
Pass an optional regex through to each suite's '\''go test -test.run'\'' so
dispatches can target a single test (e.g. '\''^TestMultinodeIPSec$'\'') without
running the full multinode suite. Threaded through single, multinode, and
reboot suite invocations; empty input preserves current behaviour.
* fix(daemon): flip NB_Global.ipsec=true on management node
Setting Open_vSwitch.other_config:ipsec_encapsulation=true on each
chassis is necessary but not sufficient for OVN native IPsec — without
NB_Global.ipsec=true, ovn-controller never adds options:remote_name to
the Geneve tunnel ports, so ovs-monitor-ipsec sees no IPsec-enabled
tunnels and the kernel xfrm tables stay empty. After enabling
ipsec_encapsulation locally, also write NB_Global.ipsec=true when the
local ovnnb_db socket is present (management node only); workers skip
silently — one writer is enough for a cluster-wide flag.
Adds a scoped sudoers entry pinning the daemon to that exact ovn-nbctl
invocation so the grant can't be reused for other NB mutations.
* fix(ovn): grant charon AppArmor read on /etc/spinifex/ipsec
The strongswan-charon profile only allows reads from /etc/ipsec.*,
/etc/strongswan.*, and a small fixed allowlist. ovs-monitor-ipsec writes
the per-tunnel strongSwan config with absolute paths to our peer cert +
key under /etc/spinifex/ipsec/, so charon hits 'Permission denied'
loading them (it runs as uid 0 but AppArmor still blocks) and no xfrm
SAs come up. Drop a read grant for /etc/spinifex/ipsec/** into the
profile's local override hook + reload.
* fix(ovn): pin OVS system-id to hostname for IPsec cert identity match
ovs-monitor-ipsec uses chassis-id as the IKEv2 `@<name>` peer identity in
the strongSwan config it generates. Our per-node IPsec peer cert
(admin.GenerateIPSecPeerCert) carries the node name as CN + dnsName SAN.
With chassis-id left at the package-generated UUID, charon validates
`@<UUID>` against a cert with dnsName=<hostname> — mismatch returns
AUTHENTICATION_FAILED and no xfrm SAs are negotiated.
Pin system-id to hostname (which equals the node name on Mulga clusters)
by writing both the on-disk source of truth (/etc/openvswitch/system-id.conf,
read by ovs-ctl on boot) and the running OVS DB value. Writing only one
would drift on the next reboot — the original "do not override" comment
was right about that hazard, just wrong about the resolution.
Cross-link the invariant in admin/ipsec.go: changing the cert identity
without updating setup-ovn.sh (or vice versa) breaks IKE silently.
Refs: mulga-siv-136
* fix(ovn): take --node-name flag so chassis-id matches cluster identity
CI hostnames are bead-themed (env20-casuarina) and don't match the
cluster node name (node1/node2) baked into the IPsec peer cert dnsName
SAN at `spx admin init/join --node NAME`. Pinning system-id to
`hostname -s` produced env20-casuarina; charon then validated
`@node1` against dnsName=env20-casuarina and IKE failed with
AUTHENTICATION_FAILED.
Add a --node-name=NAME flag to setup-ovn.sh, default to hostname for
manual/dev invocations, and update the ansible install role to pass
{{ spinifex_node_name }}. The matching bootstrap-install.sh change
lives in mulga.
Refs: mulga-siv-136
* refactor(daemon): relocate L0 helpers to network/host + widen ADR-0006 S2
Phase 4 Slice E of mulga-siv-125.6. Cleans up spinifex/daemon/network.go
per the audit and widens the S2 invariant to forbid OVS/OVN shell-outs
in spinifex/daemon/ (daemon is an L0 consumer; S2 applies).
Moved to network/host/:
- OVSNetworkPlumber + SetupTap/CleanupTap → host.OVSPlumber (host.NewOVSPlumber)
- GetBridgeIPv4 → host.GetBridgeIPv4
- OVNHealthStatus + CheckOVNHealth → host.OVNHealth + host.HealthStatus
- daemon/ipsec.go OVS/OVN ops → host.SetIPSecCertPaths, host.EnableIPSecEncapsulation, host.SetNBGlobalIPSec
Moved to utils:
- sudoCommand → utils.SudoCommand + utils.SetSudoCommandForTest test seam
(cross-package reassign would trip the reassign linter; helper keeps
the indirection inside utils).
Moved to vm:
- generateMgmtMAC → vm.GenerateMgmtMAC (pairs with vm.GenerateDevMAC)
Deleted (dead):
- SetupComputeNode (replaced by network/host/bridges.go:ensureBridges +
scripts/setup-ovn.sh:572-579)
- EnsureDataRoute
- findInterfaceByIP
- daemon/network.go + dead tests in daemon/network_test.go (duplicates of
vm package tests retained; mgmt MAC test moved to vm/lifecycle_test.go)
S2 invariant (network/invariants/host_containment_test.go) now walks
both spinifex/network/ and spinifex/daemon/, skipping network/host/.
* fix(ovn): broaden AppArmor read grant to /etc/spinifex (covers CA cert)
The earlier override granted reads on /etc/spinifex/ipsec/** only, which
covers the peer cert + key but NOT the cluster CA at /etc/spinifex/ca.pem.
charon then can't load the CA, can't validate the peer cert chain, and
surfaces the failure as `no trusted RSA public key found for '<peer>'`
even though identity matching now succeeds — IKE_AUTH fails with
AUTHENTICATION_FAILED and no xfrm SAs negotiate.
Grant /etc/spinifex/** instead (covers ca.pem + ipsec/peer.{pem,key})
and always rewrite the override so re-runs replace the older narrower
rule.
Refs: mulga-siv-136
* fix(ipsec): symlink cluster CA into charon's default cacerts dir
ovs-monitor-ipsec writes per-conn `cacert=/etc/spinifex/ca.pem` references
in the strongSwan config, but strongSwan 6.0 only loads CAs declared in
standalone `ca` sections or files placed in /etc/ipsec.d/cacerts at
startup. Without this, charon authenticates the local side fine but
rejects every inbound peer cert as `no trusted RSA public key found for
'<peer>'`, AUTH_FAILED, and no xfrm SAs negotiate.
Do the symlink in admin.GenerateIPSecPeerCert (right after writing the
peer cert) — setup-ovn.sh runs before `spx admin init/join` creates
/etc/spinifex/ca.pem, so the wiring can't live there. Use a stage-and-
rename pattern so concurrent test invocations don't race on the symlink.
/etc/ipsec.d/cacerts is on charon's default AppArmor allowlist so no
profile change needed.
Refs: mulga-siv-136
* chore(setup-ovn): drop apt-get install_packages block
OVS/OVN/openvswitch-ipsec/strongswan-charon are baked into the CI gold
image via mulga's image-builder/scripts/provision.sh (commits 116bb05,
a9b3a9c). Keeping a runtime apt-get install path here pulled CI through
apt-cacher-ng on every bootstrap and tripped on intermittent "Message
has been manipulated" InRelease cache corruption (env21/env20 runs).
Drop the install block. Gold-image rebuild is the canonical place to
add new system packages. If a package is missing, downstream ovs/ovn
commands fail loudly during setup-ovn.sh's later steps — clearer signal
than silently re-installing through a flaky proxy.
setup.sh already supports INSTALL_SPINIFEX_SKIP_APT=1 (passed by mulga's
bootstrap-install.sh:237) so this removes the last runtime apt-get call
on the CI provisioning path.
* fix(ovn): gate system-id pin on explicit --node-name
setup-ovn.sh used to fall back to `hostname -s` when --node-name wasn't
passed, and then unconditionally wrote that value to
/etc/openvswitch/system-id.conf + ovs-vsctl external_ids:system-id.
On CI single-node, the gold image already registered a chassis with
name=<UUID>; setup-ovn.sh then rewrote system-id to the bead-themed
hostname (env10-banksia). vpcd's discoverChassis treats any chassis row
whose name != local system-id but hostname == local hostname as a stale
local chassis and skips it, so vpcd reported "no OVN chassis registered
in SBDB" and refused to start, cascading into the e2e + e2e-go failures
on run 26430839508.
Drop the hostname fallback. If --node-name is given, pin both the
on-disk file and the running OVS DB to it (callers that care about
IPsec cert identity matching still get it). If no --node-name is given,
leave the existing system-id alone — that's always safe whether the
value came from a gold-image UUID, bootstrap-install.sh writing nodeN
before this script runs, or a manual ansible value.
* fix(network/reconcile): skip orphan prune on startup; drift owns GC
The vpcd startup reconcile races against the daemon's EnsureDefaultVPC:
the daemon writes SG KV rows + publishes vpc.create-sg on each node in
parallel with vpcd starting up. The leader-gated reconcile can load
intent before any SG row is visible on its local JetStream replica
while subscribers on peer nodes are concurrently creating port groups
from the same vpc.create-sg events. A startup orphan-port-group sweep
then deletes managed sg_* port groups that intent hasn't caught up to.
CI run 26434124790 trace:
node2 reconcile/lock: elected holder=env20-bottlebrush
node2 reconcile: starting intent_sgs=0
node2 reconcile/apply: removed orphan port group pg=sg_161c70ce23e704a2a
node2 reconcile/apply: removed orphan port group pg=sg_a0d078650a3150391
node2 reconcile/apply: removed orphan port group pg=sg_54b564b374e9162bb
Every subsequent RunInstances → vpc.create-port failed with
"port group not found" for ~3 minutes until the 5-minute drift tick
recreated the port groups. Tests give up after 60s; six multinode
suites cascade-fail on InsufficientInstanceCapacity.
Split the reconciler into Reconcile (full prune, drift loop) and
ReconcileApplyOnly (no prune, startup). applySGs takes a pruneOrphans
bool. vpcd startup calls ReconcileApplyOnly; drift loop unchanged.
Legitimate orphans get garbage-collected on the next drift tick — at
worst 5 minutes of lingering rows, no production impact.
Add TestReconcile_ApplyOnlyKeepsOrphanPortGroup to lock the contract.
* fix(ipsec): trigger charon rereadcacerts after CA symlink install
strongSwan scans /etc/ipsec.d/cacerts only at charon startup. The unit
openvswitch-ipsec.service is enabled at provision time and starts before
admin init writes the cluster CA symlink, so charon ends up running with
zero CAs loaded. Every IKE_AUTH then fails with `no trusted RSA public
key found for '<peer>'` and no SAs negotiate — the failure mode exposed
by TestMultinodeIPSec.
Append a stroke rereadcacerts call at the end of
installCAIntoCharonTrustStore so an already-running charon picks up the
freshly-placed CA without a service restart. Failure is swallowed at
debug — a missing/stopped charon at admin-init time is expected and a
later startup will scan cacerts naturally.
* refactor(network/host): move IPSec orchestrator from daemon (ADR-0006 S8)
ADR-0006 S8: "IPSec is OVN-native only. No layer implements custom IKEv2,
XFRM rules, or direct strongSwan management. IPSec SA lifecycle is delegated
entirely to OVN native IPSec and is invisible above L0."
The enableOVNIPSec orchestrator and its ensureOVSMonitorIPSecActive helper
lived in daemon/ipsec.go but composed L0 primitives (SetIPSecCertPaths,
EnableIPSecEncapsulation, SetNBGlobalIPSec) that already lived in
network/host. The orchestrator now lives alongside its callees as
host.EnableOVNIPSec, taking configPath and clusterConfig from the daemon at
the call site. Helper and test-override package vars (systemctlActiveTimeout,
ovnNBSocketPath) move with it; tests are translated to call EnableOVNIPSec
directly rather than via the Daemon struct.
Closes mulga-siv-142
* refactor(daemon): use topology.Port for OVN port names (ADR-0006 S4)
ADR-0006 S4 forbids any layer above L2 from constructing OVN object
names. Three daemon sites formatted "port-" + ENIId inline; route them
through topology.Port instead.
Closes mulga-siv-143
* refactor: use topology.Port for OVN port names in handlers + vm (ADR-0006 S4)
Replace inline "port-"+eniID formatting across five sites (eip, instance x2,
vpc/eni, vm/network_helpers) with topology.Port. OVSIfaceID stays as a thin
wrapper to preserve the OVN binding contract comment and existing call sites.
Closes mulga-siv-144
* test(network/invariants): widen scan roots to daemon/handlers/vm/testutil/utils (ADR-0006)
Widen ADR-0006 invariant scan roots so drift in upstack consumers surfaces
in CI. Backlog of violations in widened roots was drained by
mulga-siv-142/-143/-144; this commit makes the wider scan permanent.
- S2 (host_containment): add vm/ alongside network/ + daemon/
- S3 (nat_mode): add daemon/ alongside network/ + vpcd/
- S4 (names): add daemon/, handlers/, vm/ alongside
network/ + vpcd/; add narrow AWS-ID exemption
(vpc-cidr-assoc-) so AWS resource IDs are not
misclassified as OVN object names
- S6 (federation): add daemon/, handlers/ alongside network/;
convert single-root walk to slice
- S7 (nats_subjects): add daemon/, handlers/, testutil/, utils/ —
stays t.Skip pending cluster-wide subject
rename; gap report now lists 58 subjects
CODEOWNERS: add /spinifex/network/invariants/ rule so widening or
narrowing the scan needs explicit owner review.
Closes mulga-siv-145
Refs: audit-adr-0006.md
* chore(e2e): disable IPsec in pseudo multi-node test
Pseudo multi-node runs three simulated nodes as unprivileged tf-user.
The charon trust-store install in 904b45e writes /etc/ipsec.d/cacerts,
which fails with EACCES under a non-root caller and aborts admin init.
Disable --ipsec for this suite — loopback-only nodes don't exercise
real Geneve encryption anyway. Real-multi-node + single-node e2e
(both run admin init via sudo) continue to cover the IPsec path.
* fix(ovn): atomic Ensure* primitives prevent duplicate logical entities
OVN NB has no unique constraint on Name. The previous
GetLogicalRouter→CreateLogicalRouter sequence in EnsureVPC was a TOCTOU
race: when vpc.create on one node and the defensive vpc.create-subnet
EnsureVPC call on another both observed absence, both inserted, yielding
two routers per VPC. The orphan held DNAT references blocking EIP
detach, and DNAT/flow installation landing on the orphan caused
TestMultinodeGuestSSH SSH timeouts.
Add EnsureLogicalRouter / EnsureLogicalSwitch / EnsurePortGroup to the
libovsdb client. Each builds a single OVSDB transaction combining a
wait-op (Until=equal, Rows=[]) gated on absence of a Name match with the
insert. The server serialises the wait+insert atomically; concurrent
callers see exactly one survivor and the rest fall through to a re-fetch
that returns the canonical row.
Switch all production callers (topology.EnsureVPC / EnsureSubnet /
EnsureSGPortGroup, external.AttachIGW, reconcile.applyVPCs /
applySubnets) to the Ensure* variants. Bare Create* remains available to
the L1 wrapper.
Add invariant test TestS4_OVNNamedCreatesUseEnsure (AST walk) to block
future reintroduction of the bare Create* call sites outside
network/ovn.
Mock implements Ensure* under its existing mutex so the concurrency
tests (50/30/30 goroutines for Router/Switch/PortGroup) prove a single
UUID survives.
Refs: mulga-siv-146
* fix(ovn): wait-op rows must be non-empty; flip to NotEqual predicate
The Ensure* primitives sent a wait-op with `Rows: []ovsdb.Row{}` plus
`Until: ==` (RFC 7047 §5.2.6 absence test). The libovsdb Operation
struct tags Rows as `json:"rows,omitempty"`, so the empty slice was
dropped on the wire and ovsdb-server rejected every transaction with
"Required 'rows' member is missing", breaking reconciler bootstrap and
every downstream create-port lookup ("logical switch X not found").
Flip the predicate to the equivalent `Until: !=` formulation with a
single-row projection containing the Name we want absent:
no row → projection [] != [{name}] → predicate holds
row exists → projection [{name}] == [{name}] → predicate fails
Same atomic guarantee, JSON now well-formed. Mock tests can't catch
this — only a real ovsdb-server validates the envelope.
* chore: remove generateMAC wrapper
* chore: remove all bead / plan references
* fix(network/policy): wait for chassis flows after AddNATGateway
AddEIP fires m.barrier() to block until ovn-northd compiles NB→SB and every
chassis installs the new dnat_and_snat flows; AddNATGateway returned right
after the NB commit. So `aws ec2 create-route … --nat-gateway-id` would
publish vpc.add-nat-gateway, the SNAT row would land in NB, and the API
call would succeed before the gateway chassis had programmed the SNAT flow
for the private subnet.
Phase 8d's 10×5s=50s ping budget races that flow-install latency on slow
CI hosts (last seen run 26455602615 single-node, identical flake on
unmodified dev run 26434711010 — pre-existing, not introduced by Ensure*).
Mirror AddEIP: call m.barrier() after AddNAT returns. Add unit test
TestNATManager_AddNATGateway_FlowsBarrier_Fires covering the new path.
* chore: remove more plan refs
* chore: remove GenerateIPSecPeerCertIfEnabled
* chore: remove dupliate subnetGatewayCIDR function
* chore(network): trim superfluous comments across network/ tree
Collapse multi-paragraph godocs to 1-2 sentences, drop WHAT-restatement
comments, remove section dividers grouping <10 items, and cut prose
narration in tests. ADR clause references, one-line WHY comments for
non-obvious OVN/OVSDB/AWS/NATS semantics, and exported-symbol godoc are
preserved. No behavioural changes.
Packages: invariants, topology, host, external, ovn, reconcile, policy,
subscribers.
* fix(igw): restore attachment State="available" for AWS parity
AWS InternetGatewayAttachment.State is "available" when an IGW is
attached to a VPC, per the AWS SDK doc ("the state is available when
attached to a VPC; otherwise, this value is not returned"). Commit
b7505f8 wrote "attached"/"detached" to align with the reconciler
filter, which inverted AWS parity.
Fix the producer to write "available" again and update the consumer:
- handlers/ec2/igw: AttachInternetGateway and DetachInternetGateway
both persist State="available"; IGWRecord comment narrowed.
- network/reconcile/intent.loadIGWs: filter accepts "available" (was
"attached"); attachment is identified by VpcId != "".
- tests updated: igw service_impl_test, routetable service_impl_test
fixture, reconcile intent_test fixture.
* Update spinifex.toml
* refactor(subscribers): route EIP/NATGW through L5 wrappers (ADR-0006 §S1)
Subscriber previously called policy.NATManager directly for EIP and NAT
Gateway operations, bypassing the L5 wrappers (EIPManager, NATGWManager)
that existed on paper. ADR-0006 §S1 requires every cross-layer call to
pass through the immediate lower neighbor's typed interface; IGW already
followed this pattern.
Wire EIPManager and NATGWManager into the Subscriber struct, construct
them in vpcd alongside IGWManager, and swap the four direct-L3 call
sites (AddEIP, DeleteEIP, AddNATGateway, DeleteNATGateway) onto the
L5 surface (AttachEIP, DetachEIP, AttachNATGateway, DetachNATGateway).
AttachEIP now fires the flows-barrier, giving callers the same
"reachable on return" guarantee IGW already had.
The reconciler still calls policy.NATManager directly; that's a
cross-cutter and a separate concern.
* test(reboot): port 8MB WAL pad workaround from bash run-reboot-e2e.sh
App VMs' user-data writes (unit file + index.html) sit in the on-host
WAL until WriteWALToChunk crosses the 4MB threshold. Host reboot before
that loses them — systemd then refuses to load the zero-byte unit and
the responder never starts. Mirror the bash driver's dd+sync pad so the
WAL flushes pre-reboot. Drop when the upstream WAL fix lands.
* test(e2e): dump OVN/OVS/SG state on Phase 8d NATGW failure
Phase 8d's existing failure path only prints lr-nat-list and
lr-route-list. CI artifacts are not uploaded when run-e2e.sh exits
nonzero, so debugging the NATGW regression requires inline diagnostics
in the journal. Expand the dump to include nb/sb show, chassis bindings,
port_group ACLs, br-int flows filtered to the NATGW IP/private subnet,
host routes, and the private VM's own ip/route/neigh state.
* fix(ovn,reconcile): scope NAT deletes to router; NATGW snat per associated subnet
Two compounding bugs caused Phase 8d NATGW E2E flake (~60% CI failure):
1. ovn.DeleteNAT/DeleteNATByExternalIP searched all NAT rows globally and
only mutated the named router afterwards. With AWS subnet CIDRs repeating
across VPCs, (type, logicalIP/externalIP) is not globally unique — the
delete picked a NAT row owned by a different router, the mutate was a
no-op, and the row delete failed with "remaining reference(s)", leaving
the original router's NAT orphaned. Both calls now resolve the LR first
and filter NAT lookups to lr.NAT UUIDs. Mock client mirrors the scoping.
2. reconcile.loadNATGWs used the NATGW's home (public) subnet CIDR as
SubnetCIDR, producing an snat for the wrong CIDR that collided with the
correct event-published snat. loadNATGWs now walks route tables and
emits one NATGWSpec per (NATGW, associated private subnet) pair,
mirroring publishNatGatewayEventsForAssociation. Intent map keyed by
natgwID|subnetCIDR so multi-subnet NATGWs produce distinct specs.
Tests:
- nat_test.go: cross-router same-logicalIP isolation for DeleteEIP and
DeleteNATGateway
- intent_test.go: NATGW snat uses associated private subnet CIDR, not the
NATGW home subnet; NATGW with no associations produces no spec
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
334 lines
13 KiB
Makefile
334 lines
13 KiB
Makefile
GO_PROJECT_NAME := spx
|
|
SHELL := /bin/bash
|
|
|
|
# Detect architecture for cross-platform support
|
|
ARCH := $(shell uname -m)
|
|
ifeq ($(ARCH),x86_64)
|
|
GO_ARCH := amd64
|
|
AWS_ARCH := x86_64
|
|
# On x86, we can run ARM VMs via emulation
|
|
QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
|
|
else ifeq ($(ARCH),aarch64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
# On ARM, we can run x86 VMs via emulation
|
|
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
|
|
else ifeq ($(ARCH),arm64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
|
|
else
|
|
$(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
|
|
endif
|
|
|
|
# Ask Go whether workspace mode is active
|
|
IN_WORKSPACE := $(shell go env GOWORK)
|
|
|
|
# Use -mod=mod unless Go reports an active workspace path
|
|
ifeq ($(IN_WORKSPACE),)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else ifeq ($(IN_WORKSPACE),off)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else
|
|
GO_BUILD_MOD :=
|
|
endif
|
|
|
|
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
|
|
# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
|
|
# propagates through the filter — otherwise a test failure is swallowed by
|
|
# grep's own (success) exit code and preflight prints "passed" on red.
|
|
ifdef QUIET
|
|
_Q = @
|
|
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
|
|
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
|
|
_SECQ = >
|
|
else
|
|
_Q =
|
|
_COVQ =
|
|
_RACEQ =
|
|
_SECQ = 2>&1 | tee
|
|
endif
|
|
|
|
build: go_build build-installer build-lb-agent
|
|
|
|
# Build spinifex-ui frontend (requires pnpm)
|
|
build-ui:
|
|
@echo -e "\n....Building spinifex-ui frontend...."
|
|
cd spinifex/services/spinifexui/frontend && pnpm build
|
|
|
|
# GO commands
|
|
VERSION ?= $(shell git describe --tags --always --dirty)
|
|
COMMIT ?= $(shell git rev-parse --short HEAD)
|
|
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
|
|
|
|
go_build:
|
|
@echo -e "\n....Building $(GO_PROJECT_NAME)"
|
|
GOFIPS140=v1.0.0 go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
|
|
|
|
build-installer:
|
|
@echo -e "\n....Building spinifex-installer"
|
|
GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
|
|
|
|
build-lb-agent:
|
|
@echo -e "\n....Building lb-agent (static)"
|
|
CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
|
|
|
|
build-system-image: ## Build a system image from manifest (use IMAGE=lb)
|
|
ifndef IMAGE
|
|
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
|
|
endif
|
|
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
|
|
|
|
MICROVM_OUT_DIR := build/microvm
|
|
MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
|
|
MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
|
|
|
|
# Grouped target — script writes both files in one run.
|
|
$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
|
|
./scripts/build-microvm-image.sh
|
|
|
|
# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
|
|
# build-microvm-image stays correctly stale-aware.
|
|
bin/lb-agent:
|
|
$(MAKE) build-lb-agent
|
|
|
|
build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
|
|
|
|
install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
|
|
sudo install -d /usr/share/spinifex/microvm
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
|
|
|
|
go_run:
|
|
@echo -e "\n....Running $(GO_PROJECT_NAME)...."
|
|
$(GOPATH)/bin/$(GO_PROJECT_NAME)
|
|
|
|
# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
|
|
# Use this before committing to catch CI failures locally.
|
|
preflight:
|
|
@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
|
|
@echo -e "\n ✅ Preflight passed — safe to commit."
|
|
|
|
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
|
|
# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
|
|
# infrastructure required, safe to run in CI without a cluster.
|
|
test-harness:
|
|
@echo -e "\n....Running e2e harness unit tests...."
|
|
$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
|
|
|
|
# Validate docs/service-interfaces.yaml. Schema check + cross-reference
|
|
# of services/suites/fixtures + on-disk path existence. Subject content
|
|
# vs source is enforced separately in Bead 5 drift lint.
|
|
manifest-check:
|
|
@echo -e "\n....Checking service-interfaces.yaml...."
|
|
@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
|
|
|
|
# Run unit tests
|
|
test:
|
|
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
|
|
LOG_IGNORE=1 go test -timeout 120s ./spinifex/...
|
|
|
|
# Run unit tests with coverage profile
|
|
COVERPROFILE ?= coverage.out
|
|
test-cover:
|
|
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverpkg=./spinifex/... -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... $(_COVQ)
|
|
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
|
|
|
|
# Run unit tests with race detector
|
|
test-race:
|
|
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
|
|
|
|
# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
|
|
# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
|
|
test-actions:
|
|
@echo -e "\n....Running action tests...."
|
|
LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
|
|
|
|
# Check that new/changed code meets coverage threshold (runs tests first)
|
|
diff-coverage: test-cover
|
|
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
|
|
|
|
bench:
|
|
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
|
|
$(MAKE) easyjson
|
|
LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
|
|
|
|
run:
|
|
$(MAKE) go_build
|
|
$(MAKE) go_run
|
|
|
|
# Fast iteration: build + install binary + restart all services.
|
|
# Microvm artifacts are reinstalled when they already exist on disk — the rule's
|
|
# input timestamps drive a rebuild only if anything actually changed. On a fresh
|
|
# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
|
|
# `make install-microvm` explicitly the first time.
|
|
deploy: build
|
|
sudo install -m 755 bin/spx /usr/local/bin/spx
|
|
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
|
|
$(MAKE) install-microvm; \
|
|
else \
|
|
echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
|
|
fi
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart spinifex.target
|
|
|
|
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
|
|
# Not needed for code-only changes — use deploy for those.
|
|
reinstall:
|
|
scripts/dev-install.sh
|
|
|
|
clean:
|
|
rm -f ./bin/$(GO_PROJECT_NAME)
|
|
rm -rf spinifex/services/spinifexui/frontend/dist
|
|
|
|
install-system:
|
|
@echo -e "\n....Installing system dependencies for $(ARCH)...."
|
|
@echo "QEMU packages: $(QEMU_PACKAGES)"
|
|
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
|
|
-o Dpkg::Options::="--force-confdef" \
|
|
-o Dpkg::Options::="--force-confold" \
|
|
nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
|
|
ovmf qemu-efi-aarch64 \
|
|
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
|
|
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
|
|
ovn-central ovn-host openvswitch-switch dhcpcd-base
|
|
|
|
install-go:
|
|
@echo -e "\n....Installing Go 1.26.3 for $(ARCH) ($(GO_ARCH))...."
|
|
@if [ ! -d "/usr/local/go" ]; then \
|
|
curl -L https://go.dev/dl/go1.26.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
|
|
else \
|
|
echo "Go already installed in /usr/local/go"; \
|
|
fi
|
|
@echo "Go version: $$(go version)"
|
|
|
|
install-aws:
|
|
@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
|
|
@if ! command -v aws >/dev/null 2>&1; then \
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
|
|
unzip -o awscliv2.zip; \
|
|
./aws/install; \
|
|
rm -rf awscliv2.zip aws/; \
|
|
else \
|
|
echo "AWS CLI already installed"; \
|
|
fi
|
|
|
|
quickinstall: install-system install-go install-aws
|
|
@echo -e "\n✅ Quickinstall complete for $(ARCH)."
|
|
@echo " Please ensure /usr/local/go/bin is in your PATH."
|
|
@echo " Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
|
|
|
|
# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
|
|
lint:
|
|
@echo "Running golangci-lint..."
|
|
$(_Q)golangci-lint run ./...
|
|
@echo " golangci-lint ok"
|
|
|
|
# Auto-fix all linter issues that have fixers
|
|
fix:
|
|
golangci-lint run --fix ./...
|
|
|
|
# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
|
|
govulncheck:
|
|
@echo "Running govulncheck..."
|
|
$(_Q)go tool govulncheck ./...
|
|
@echo " govulncheck ok"
|
|
|
|
# Build release tarballs — use distro-ARCH for single arch, distro for both
|
|
distro: distro-amd64 distro-arm64
|
|
@echo ""
|
|
@echo "Distribution tarballs:"
|
|
@ls -lh dist/*.tar.gz
|
|
@echo ""
|
|
@cat dist/*.sha256
|
|
|
|
distro-amd64:
|
|
@echo "Building spinifex $(VERSION) linux/amd64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/amd64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/amd64/ \
|
|
../
|
|
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
|
|
echo "[distro-amd64] staging microvm artifacts into tarball"; \
|
|
mkdir -p dist/amd64/microvm; \
|
|
cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
|
|
else \
|
|
echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
|
|
echo "[distro-amd64] run 'make build-microvm-image' before 'make distro-amd64'"; \
|
|
fi
|
|
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
|
|
|
|
distro-arm64:
|
|
@echo "Building spinifex $(VERSION) linux/arm64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/arm64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/arm64/ \
|
|
../
|
|
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
|
|
|
|
distro-clean:
|
|
rm -rf dist/
|
|
|
|
# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
|
|
# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
|
|
#
|
|
# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
|
|
# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
|
|
# make's own flag and does NOT reach ansible.
|
|
# make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
|
|
EXTRA_VARS ?=
|
|
_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
|
|
|
|
ansible-dev-preflight:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
|
|
|
|
ansible-dev-teardown:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
|
|
|
|
ansible-dev-install:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
|
|
|
|
ansible-dev-reset:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
|
|
|
|
ansible-dev-deploy:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
|
|
|
|
ansible-dev-status:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
|
|
|
|
ansible-dev-logs:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
|
|
|
|
# Snapshot / restore require an explicit -e snapshot_name=<name>. The
|
|
# wrapper passes ANSIBLE_EXTRA through so devs can run
|
|
# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
|
|
ansible-dev-snapshot:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
|
|
|
|
ansible-dev-restore:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
|
|
|
|
ansible-dev-version:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
|
|
|
|
ansible-dev-vpc:
|
|
cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
|
|
|
|
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
|
|
deploy reinstall clean \
|
|
install-system install-go install-aws quickinstall \
|
|
lint fix govulncheck \
|
|
distro distro-amd64 distro-arm64 distro-clean \
|
|
ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy ansible-dev-status ansible-dev-logs ansible-dev-snapshot ansible-dev-restore ansible-dev-version ansible-dev-vpc
|