Add four branch tests for verifyBridgeMode that were missing from the
initial Fix 2 commit:
- direct mode rejects empty external_interface
- direct mode rejects empty dhcp_bind_bridge
- veth mode rejects empty dhcp_bind_bridge
- veth mode fails when veth-wan-br link missing
Lifts diff coverage from 59.5% to 65.6%, meeting the 65% gate.
Make bridge-mode detection loud and fail-start when runtime state does
not match config. Closes D4/D12/D18 of dhcp-vpcd-issues.md.
Changes:
- detectBridgeMode: slog.Info at each decision, slog.Warn on fall-through
(was silent Debug — let the veth persistence bug hide for weeks).
- Extract resolveBridgeConfig so mode/bind-bridge resolution is unit
testable. Empty mode now stays empty so verifyBridgeMode rejects it
with an explicit list of supported values.
- Add verifyBridgeMode: refuses to start when external iface missing
(direct), when external iface not on the OVS bridge (direct), when
veth-wan-ovs missing or attached to the wrong OVS bridge, or when
veth-wan-br missing or on the wrong Linux bridge. References
OvnExternalBridge (= "br-ext") so the sanity check tracks the contract,
not a hardcoded string.
- Wire verifyBridgeMode into launchService. Error is loud and includes
operator guidance ("re-run scripts/setup-ovn.sh").
- Injectable probes (ifaceIsMacvlan, ifaceExists, portToBr,
readLinkMaster) so tests can exercise every branch without touching
real interfaces.
- Tests: 16 new unit tests covering detectBridgeMode, resolveBridgeConfig,
and all verifyBridgeMode branches.
CreateLogicalSwitchPort only seeds options:nat-addresses at first
creation. If a VPC's localnet port was created during a vpcd run that
saw direct mode (e.g. pre-Fix 1 reboot without the veth pair), the port
keeps stale options forever — OVN never GARPs for the external IP and
centralized NAT stays broken.
Add TopologyHandler.ensureLocalnetOptions: read the LSP, diff
options["nat-addresses"] against useCentralizedNAT(), update only on
mismatch. Called from reconcileIGW after CreateLogicalSwitchPort so the
retrofit runs on every reconcile pass (no-op when already correct).
Mock tracks UpdateLogicalSwitchPortCalls so the idempotency test
asserts no write was issued.
setup-ovn.sh's veth branch creates veth-wan-br ↔ veth-wan-ovs but nothing
recreates the kernel interfaces on reboot. OVSDB remembers veth-wan-ovs
as a port on br-ext but the peer is gone — the external path silently
breaks and vpcd falls through to direct mode.
- setup-ovn.sh: write /etc/systemd/network/15-spinifex-veth-wan.{netdev,network}
in the veth branch so systemd-networkd recreates the pair on boot. Add
a single pre-case guard that rips down stale veth persistence + the OVS
port + the kernel veth whenever WAN_BRIDGE_MODE != veth, so mode
transitions are idempotent (D17).
- reset-dev-env.sh: delete the netdev/network units during cleanup so
`make reset` gives a truly clean slate.
- spinifex-vpcd.service: After=systemd-networkd.service openvswitch-switch.service
so vpcd's sanity checks see a populated kernel + OVSDB (D5).
The old wan_bridge name misled operators into pointing the field at
the OVN-side bridge 'br-ext', which never sees LAN DHCP traffic. On
consumer-router LANs (WAN NIC enslaved to a Linux bridge) the vpcd
DHCP client's AF_PACKET bind failed silently, blocking the default
install path.
- Rename Go field WanBridge -> DhcpBindBridge and TOML key wan_bridge
-> dhcp_bind_bridge across config, vpcd service, external IPAM,
daemon pool wiring, admin settings, and spinifex.toml template.
- Rewrite detectedWanBridge() -> detectedDhcpBindBridge(): the
Linux-bridge branch now returns the bridge name verbatim (e.g.
"br-wan") instead of synthesising "br-ext". OVS-vs-Linux probe via
ovs-vsctl is no longer needed for this field.
- Add checkLegacyWanBridgeKey() in cmd/service.go: vpcd fail-starts
if the deprecated 'wan_bridge' TOML key or SPINIFEX_VPCD_WAN_BRIDGE
env var is set. No silent alias, no auto-rewrite (per plan D3/D13).
- Template carries an operator-facing comment warning never to set
the field to 'br-ext'.
- 7 new unit tests for the detector branches and fail-start paths.
Smoke on ben-bottlebrush: spx admin init --force writes
dhcp_bind_bridge matching the live br-wan Linux bridge; VM launched
with --map-public-ip-on-launch reached from LAN; restoring a
legacy-key backup causes vpcd to enter failed state with the
deprecated-key guidance log.
Plan: docs/development/bugs/dhcp-vpcd-issues.md (Fix 4)
Bead: mulga-998.a (unblocker for .b / .c)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)
Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:
- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
/local/instances (placeholder types until daemon-local-autonomy lands),
plus /health for scaffolding validation
Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.
* feat: add DDIL harness fault injection and link profiles
Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:
- harness/fault.go — PartitionNode (per-peer iptables DROP with
orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
RestartDaemonOnly, WaitForMode (polls /local/status for typed
DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
SATCOM, HFData, Flapping constants (unvalidated approximations per
plan §2 table; validation tracking lives in TEST_COVERAGE.md)
All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.
* feat: add DDIL harness witness VM helpers
LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.
* feat: add DDIL harness snapshot and state assertions
TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.
* feat: add DDIL harness reset, retry, and quarantine
ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.
AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.
Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.
bd: mulga-siv-5.5
* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)
Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.
* feat: add DDIL E2E CI workflow (workflow_dispatch only)
Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
siv-8.5 migrated daemon + awsgw but missed the sidecar services and
the spx CLI. vpcd and viperblockd read cfg.NatsHost / cfg.S3Host raw
from spinifex.toml; when BindIP=0.0.0.0 the TLS handshake fails SAN
verification because 0.0.0.0 is not a cert SAN. Same for the spx get
CLI path.
Call sites:
spinifex/services/viperblockd/viperblockd.go:173 (NATS)
spinifex/services/viperblockd/viperblockd.go:407 (S3 s3.S3Config)
spinifex/services/viperblockd/viperblockd.go:533 (S3 nbd.NBDKitConfig)
spinifex/services/vpcd/vpcd.go:277 (NATS)
cmd/spinifex/cmd/get.go:93 (CLI NATS)
admin.DialTarget rewrites 0.0.0.0:N to 127.0.0.1:N — loopback is always
a SAN and in-process sidecars share the host with NATS/predastore.
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway
Convert br-mgmt from an externally-provisioned Linux bridge to an OVS
bridge owned by scripts/setup-ovn.sh, giving every deployment type
(tofu-cluster, production single-node, production multi-node) a single
provisioning path.
setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface /
--no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone
(excluded from ovn-bridge-mappings so ovn-controller ignores it) and
assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot.
SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master`
to `ovs-vsctl add-port` / `del-port` to match the new bridge type.
Add a loopback guard to wireLBAgentConfig's fallback branch: on a
production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge,
the fallback previously injected 127.0.0.1 into LB VM cloud-init —
silently unreachable because the VM's loopback is its own. The guard
leaves GatewayURL empty and the existing "LB agent gateway URL not
configured" error fires instead. Regression covered in
daemon_wire_lb_test.go.
* fix: auto-detect primary IP for admin init --bind default
Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback
only, so system VMs on br-mgmt could never reach it. Auto-detect the
default-route source IPv4 instead, which keeps TLS cert SANs specific
(still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1
explicitly to preserve loopback-only behavior on laptops.
Closes mulga-siv-7.
* test: refresh wire lb loopback-bind comment for siv-7
* fix: dev-install scripts should let admin init auto-detect --bind
Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while
setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with
no listener. Drop the explicit --bind so auto-detect picks the primary IP
and the siv-6 mgmtRouteVia path routes LB traffic to a real listener.
* fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
PR #144 (nats TLS) made every service listen on --bind's specific IP. For
awsgw and predastore this broke localhost-reachability that the UI proxy
(spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on —
latent since #144, surfaced by siv-7's auto-detect default on dev installs
(502 from the UI).
Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen
wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host
daemon → predastore) keep working, LAN clients still reach them via the
LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}}
because peer-advertised values must not resolve to 0.0.0.0/loopback.
predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1
for local-node consumers, so existing callers see a usable connect address.
* revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8)
Clears the deck before layering the listen-vs-advertise split
(mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node
default and introduces an explicit --advertise flag for off-host dial
targets; the reverted approach (auto-detect a specific IP at init,
hardcode 0.0.0.0 in the template, guard against loopback in
wireLBAgentConfig) becomes redundant once that lands.
Reverts:
- bb0ad3a fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
- f20a47b fix: dev-install scripts should let admin init auto-detect --bind
- 49b0b4e test: refresh wire lb loopback-bind comment for siv-7
- b88dfdf fix: auto-detect primary IP for admin init --bind default
Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig
fallback and its regression test — subsumed by the GatewayURL
precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error,
no silent empty-string fallthrough). The br-mgmt OVS provisioning
from 74b6e2d is retained.
Plan: docs/development/bugs/spx-admin-init-bind.md
* feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2)
Introduce the two small address-resolution helpers the listen-vs-
advertise split needs, with full unit coverage. No callers yet —
dial-site migration lands in mulga-siv-8.5.
- DialTarget(listenAddr): rewrites wildcard listen addresses
(0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process
clients dial a target the daemon's cert already has in its SANs.
Specific IPs, loopback, hostnames, and malformed input pass through.
- AdvertiseHost(listenAddr, advertiseIP): picks the host off-host
clients should dial. Wildcard listen → advertiseIP; specific host →
that host. Port is dropped so callers append the protocol-appropriate
port.
Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1
* feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3)
Pure data-model change. Add AdvertiseIP to:
- admin.ConfigSettings (local node, populated by init).
- config.Config (per-node; mapstructure "advertise").
- formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake
carries it between peers.
buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so
joined nodes from older clusters / pre-siv-8 joiners keep working.
No behaviour change yet — templates, CLI flag, and dial-site migration
land in siv-8.4 and siv-8.5.
* feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4)
Introduce the listen-vs-advertise split at the CLI boundary.
admin init/join:
- --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert).
- New --advertise <IP> flag on both init and join; optional, auto-detected
from DetectNetwork().WAN.IP when --bind is wildcard or loopback.
- resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements
precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 +
warning.
- Join resolves advertise before building the JoinRequest so peers record
a reachable dial target (not 0.0.0.0).
- isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator
can now run "init --nodes 3" without typing a specific --bind.
- Init success message prints bind / advertise / loopback three-way split.
Templates:
- spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered
only when AdvertiseIP is set.
- predastore-multinode.toml: iam.nats_url host wrapped in the same
{{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that
predastore.toml already uses, so co-resident predastore dials its local
NATS via loopback instead of 0.0.0.0.
Tests cover the resolveAdvertiseIP matrix, the template advertise field
(present + omitted), and the predastore nats_url loopback shim.
Dial-site migration and GatewayURL rework land in siv-8.5.
* feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5)
Switch every in-process dial site to admin.DialTarget so the wildcard
listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon
→ NATS and → predastore (boot, reconnect, status probe, S3 store, service
manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so
loopback-dial a wildcard-listen service is always valid; the old
0.0.0.0 literal was tripping TLS SAN verification.
Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go):
1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged)
2. br-mgmt present, AWSGW on 0.0.0.0 → br-mgmt IP (unchanged)
3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW)
4. DevNetworking shim → 10.0.2.2 (unchanged)
5. AWSGW on specific IP, no bridge → that IP (unchanged)
6. Else → log error, skip assignment
Kills the pre-siv-8 silent empty-URL failure mode that made
CreateLoadBalancer fail with 'missing system credentials: gatewayURL='
on every fresh single-node install. Supersedes siv-6's loopback guard.
Extend SetupAWSCredentials with a wanIP parameter (reserved for a
future --operator-endpoint flag) and thread AdvertiseIP through
finalizeNodeSetup's three call sites (single-node init, multi-node
init, join). Print 'Bind / Advertise / Loopback' on multi-node init
and join completion to match the single-node summary.
Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3
Internet-facing ALBs allocated their public IP by calling
externalIPAM.AllocateIP() directly from LaunchSystemInstance, bypassing
the EIP service entirely. No EIPRecord landed in the eips KV bucket, the
tag was "auto_assign" (EIP service uses "elastic_ip"), and
DescribeAddresses never returned the IP — so any OpenTofu flow that
expects an AllocateAddress-observable record hung waiting.
LaunchSystemInstance now calls eipService.AllocateAddress +
AssociateAddress when the EIP service is wired (falls back to the old
direct IPAM path otherwise). TerminateSystemInstance releases through
the service so the KV record is cleaned up and vpc.delete-nat is
published. The generic stopInstance teardown is left untouched so
customer RunInstances flows behave identically.
Also refreshes the nginx-alb terraform example to use public subnets
with auto-assigned public IPs (the configuration that actually exercises
this path end-to-end) and tightens the awsgwBindIP loopback guard in
wireLBAgentConfig so the gateway URL is usable by LB agents.
* fix: replace stringly-typed TLS error detection with sentinel errors
Export ErrCACertRead and ErrCACertParse from utils/nats.go so callers can
detect permanent TLS configuration failures with errors.Is instead of
substring matching on error messages. Update daemon.go and awsgw.go retry
loops accordingly, and update tests to use assert.ErrorIs.
Refs: mulga-siv-1
* refactor: extract shared NATS connect-with-retry into utils
daemon.connectNATS() and awsgw.connectNATS() were near-identical retry
loops. Extract to utils.ConnectNATSWithRetry() with functional options
for test configurability.
Replace fragile image-name substring match (-lb-/-alb-) with a
DescribeImages tag filter on spinifex:managed-by=elbv2. Remove the
silent fallback that picks the first image in the store when no LB
image exists — CreateLoadBalancer now returns ServerInternal with a
remediation hint instead of persisting a record in StateFailed.
- daemon: tag filter + error return from the resolver closure
- service_impl: resolver signature is func() (string, error); caches
only successful results so the next call retries after an error
- CreateLoadBalancer gate now requires launcher + networking; AMI is
resolved inside the gate and failures short-circuit before persist
- admin images import: add --name+--file combo (catalog metadata with
a local file source) and a --tag key=value flag for explicit tagging
of file-only imports
- build-system-image.sh + lb.conf: propagate SYSTEM_TAG via --tag so
locally built system AMIs get the right tag even when the catalog
is bypassed
Closes mulga-969
joshsiv-mulga