initPool's first guard inverted: !errors.Is(err, ErrKeyNotFound) was
true when err==nil, so initPool returned immediately when a record
existed and never reached the drift-reconcile or already-initialized
branches. As a result, range_start/range_end/source changes in
spinifex.toml never propagated and nextAvailableExternalIP failed
with "invalid IP range: - " on dhcp -> static transitions.
Restructure to: bail on hard errors, reconcile or no-op when record
exists, fall through to create when ErrKeyNotFound.
Verified on dev cluster: dhcp -> static toml change now reconciles
KV bounds on restart and RunInstances allocates from the new range.
Stale Allocated entries and stale gateway_ip on source change remain
out of scope for this hotfix; tracked under mulga-siv-24 follow-ups.
Symmetric to the openvswitch-switch ordering: ensures ovn-controller
shutdown waits for spinifex-daemon's instance teardown, the correct
general invariant for VM cleanup since logical port programming
flows through OVN. PR1a flagged this as optional follow-up.
Bead: mulga-siv-20.1
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a)
Three minimal edits to close the orphan-OVS-port failure mode that
survives PR1a's systemd ordering fix (host crash, OOM-kill of ovsdb,
operator manually stopping OVS on a live host) and the
terminate-during-pending race that leaks taps when a user aborts a
slow launch.
1. SetupTapDevice (network.go): unconditional `ovs-vsctl --if-exists
del-port br-int <tap>` + separately-gated kernel `ip tuntap del`.
The previous kernel-side-gated cleanup missed orphan OVS ports
because OVS conf.db survives reboot while kernel taps don't.
Rejected `--may-exist add-port`: it would silently keep stale
external_ids:iface-id/attached-mac from a prior launch.
2. stopInstance mgmt-tap branch (daemon.go) + CleanupMgmtTapDevice
(network.go): derive the mgmt tap name from instance.ID rather
than reading instance.MgmtTap, since that field is set only after
SetupMgmtTapDevice returns and a terminate racing mid-launch
would skip cleanup. CleanupMgmtTapDevice gains a kernel-presence
short-circuit so unconditional invocation doesn't log spurious
"Device does not exist" errors for instances that never reached
SetupMgmtTapDevice.
3. LaunchInstance (daemon.go): `launchStillValid` helper re-checks
instance status before StartInstance and TransitionState, in
addition to the entry guard. On detection of shutting-down /
terminated, log Info and return nil — the concurrent terminate
goroutine owns cleanup; this is an expected race outcome, not an
error. Removes the spurious "Failed to transition instance to
running" ERROR that surfaced on every legitimate
terminate-during-pending. Plus the entry path now returns nil
instead of an error for the same reason.
Also lands `scripts/start-then-terminate-pending.sh` (added during
investigation) which reproduces the runtime race for manual
verification.
Bead: mulga-siv-20.2
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1b)
systemd reverses After= on shutdown so daemon now runs CleanupTapDevice
while ovsdb-server still answers. Eliminates orphan OVS port left in
conf.db when host reboots — root trigger for "port already exists" on
post-reboot RunInstances. spinifex-vpcd.service already had this; the
daemon unit was missed.
Also pull openvswitch-switch + ovn-controller into spinifex.target Wants
so the platform brings them up explicitly rather than relying on
implicit ordering from spinifex-vpcd alone.
Bead: mulga-siv-20.1
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a)
Follow-up to d509df7. The sudo PAM audit bug exists in two more vpcd call
sites beyond portToBr: localSystemID() and discoverChassis().
When vpcd.service's AmbientCapabilities trip sudo's PAM, stderr carries
"sudo: unable to send audit message: Operation not permitted". CombinedOutput
merges that into stdout and corrupts the parsed value:
localSystemID → localID="sudo: ...\n\"add0104e-..."
discoverChassis → tries to parse the stderr banner as chassis name/hostname
Downstream effect: discoverChassis skips the real local chassis as "stale",
vpcd falls back to cfg.ChassisNames (e.g. "chassis-josh-banksia") which no
chassis in SBDB actually uses, so every gateway_chassis row points at a
phantom chassis. cr-gw* chassisredirect ports never bind → no proxy-ARP
for EIPs → external ping fails with ARP timeout.
Switch both to Output() + exec.ExitError.Stderr, matching d509df7's pattern.
Found during mulga-998.b reboot drill while verifying external reachability:
chassis discovery log showed `localID="sudo: unable to send audit message...`,
`cr-gw-vpc-* chassis=[]`, and host ping to EIP returned Destination Host
Unreachable. With this fix applied, discoverChassis returns the real system-id
and (after stale gateway_chassis rows are cleared) cr-gw binds, proxy-ARP
responds, and EIP ping succeeds end-to-end.
Note: existing gateway_chassis rows written pre-fix still carry the phantom
chassis_name. Cleanup is one-shot — delete those rows and trigger an IGW
detach+attach, or run reset-dev-env.sh. vpcd does not reconcile existing
rows on startup (potential future retrofit, similar to Fix 3's
RetrofitAllExternalLocalnetOptions).
vpcd.service sets AmbientCapabilities which trips sudo's pam_limits into
emitting 'sudo: unable to send audit message: Operation not permitted'
on stderr. CombinedOutput merged that into stdout and poisoned the
port-to-br result, causing verifyBridgeMode to fail with a bogus bridge
name on service start after reinstall.
Switch to Output() (stdout only) + exec.ExitError.Stderr on error.
ensureLocalnetOptions was only invoked inside reconcileIGW, which is gated
on the external logical switch being absent — so a port whose options were
cleared or drifted from the current bridge mode would never be corrected
on restart. Drill 2 (stale-options recovery) exposed this: clearing
ext-port options and restarting vpcd left the port with options={}.
Changes:
- Extend ensureLocalnetOptions to also restore network_name=external (not
just nat-addresses). A manual `ovn-nbctl clear` wipes both, and the port
is non-functional without network_name.
- Add RetrofitAllExternalLocalnetOptions: walks every OVN logical switch
tagged spinifex:role=external and calls ensureLocalnetOptions on its
ext-port-<vpcId>. OVN is the source of truth; NATS KV records may be
missing or stale.
- Call RetrofitAllExternalLocalnetOptions after ReconcileFromKV during
vpcd startup.
- Tests: network_name restoration in both centralized and direct modes;
RetrofitAll across multiple switches with mixed stale / correct / non-
external state.
Fix 1 persisted the veth pair across reboot via
15-spinifex-veth-wan.{netdev,network}, but only configured
veth-wan-br. Its peer veth-wan-ovs had no .network unit, so
networkd left it unmanaged and admin-DOWN after reboot. That
cascades: peer goes LOWERLAYERDOWN, br-wan loses carrier,
external DHCP/NAT path broken — despite vpcd correctly
detecting mode=veth and the sanity check passing.
OVS does not admin-up external (non-internal) ports — that's
networkd's job. Pre-reboot the setup-ovn.sh 'ip link set
veth-wan-ovs up' made it work, but that doesn't persist.
Write a second unit 16-spinifex-veth-wan-ovs.network with
ConfigureWithoutCarrier=yes so networkd flips admin state
without taking ownership of bridge/address assignment (OVS
retains that). Cleanup paths in the non-veth branch of
setup-ovn.sh and reset-dev-env.sh updated to remove the new
unit too.
Verified live via 'networkctl reload': veth-wan-ovs and
veth-wan-br both UP, br-wan LOWER_UP. Reboot drill per
docs/development/bugs/dhcp-vpcd-issues.md 'Live drill 1'
still required to confirm persistence.
Add four branch tests for verifyBridgeMode that were missing from the
initial Fix 2 commit:
- direct mode rejects empty external_interface
- direct mode rejects empty dhcp_bind_bridge
- veth mode rejects empty dhcp_bind_bridge
- veth mode fails when veth-wan-br link missing
Lifts diff coverage from 59.5% to 65.6%, meeting the 65% gate.
Make bridge-mode detection loud and fail-start when runtime state does
not match config. Closes D4/D12/D18 of dhcp-vpcd-issues.md.
Changes:
- detectBridgeMode: slog.Info at each decision, slog.Warn on fall-through
(was silent Debug — let the veth persistence bug hide for weeks).
- Extract resolveBridgeConfig so mode/bind-bridge resolution is unit
testable. Empty mode now stays empty so verifyBridgeMode rejects it
with an explicit list of supported values.
- Add verifyBridgeMode: refuses to start when external iface missing
(direct), when external iface not on the OVS bridge (direct), when
veth-wan-ovs missing or attached to the wrong OVS bridge, or when
veth-wan-br missing or on the wrong Linux bridge. References
OvnExternalBridge (= "br-ext") so the sanity check tracks the contract,
not a hardcoded string.
- Wire verifyBridgeMode into launchService. Error is loud and includes
operator guidance ("re-run scripts/setup-ovn.sh").
- Injectable probes (ifaceIsMacvlan, ifaceExists, portToBr,
readLinkMaster) so tests can exercise every branch without touching
real interfaces.
- Tests: 16 new unit tests covering detectBridgeMode, resolveBridgeConfig,
and all verifyBridgeMode branches.
CreateLogicalSwitchPort only seeds options:nat-addresses at first
creation. If a VPC's localnet port was created during a vpcd run that
saw direct mode (e.g. pre-Fix 1 reboot without the veth pair), the port
keeps stale options forever — OVN never GARPs for the external IP and
centralized NAT stays broken.
Add TopologyHandler.ensureLocalnetOptions: read the LSP, diff
options["nat-addresses"] against useCentralizedNAT(), update only on
mismatch. Called from reconcileIGW after CreateLogicalSwitchPort so the
retrofit runs on every reconcile pass (no-op when already correct).
Mock tracks UpdateLogicalSwitchPortCalls so the idempotency test
asserts no write was issued.
setup-ovn.sh's veth branch creates veth-wan-br ↔ veth-wan-ovs but nothing
recreates the kernel interfaces on reboot. OVSDB remembers veth-wan-ovs
as a port on br-ext but the peer is gone — the external path silently
breaks and vpcd falls through to direct mode.
- setup-ovn.sh: write /etc/systemd/network/15-spinifex-veth-wan.{netdev,network}
in the veth branch so systemd-networkd recreates the pair on boot. Add
a single pre-case guard that rips down stale veth persistence + the OVS
port + the kernel veth whenever WAN_BRIDGE_MODE != veth, so mode
transitions are idempotent (D17).
- reset-dev-env.sh: delete the netdev/network units during cleanup so
`make reset` gives a truly clean slate.
- spinifex-vpcd.service: After=systemd-networkd.service openvswitch-switch.service
so vpcd's sanity checks see a populated kernel + OVSDB (D5).
The old wan_bridge name misled operators into pointing the field at
the OVN-side bridge 'br-ext', which never sees LAN DHCP traffic. On
consumer-router LANs (WAN NIC enslaved to a Linux bridge) the vpcd
DHCP client's AF_PACKET bind failed silently, blocking the default
install path.
- Rename Go field WanBridge -> DhcpBindBridge and TOML key wan_bridge
-> dhcp_bind_bridge across config, vpcd service, external IPAM,
daemon pool wiring, admin settings, and spinifex.toml template.
- Rewrite detectedWanBridge() -> detectedDhcpBindBridge(): the
Linux-bridge branch now returns the bridge name verbatim (e.g.
"br-wan") instead of synthesising "br-ext". OVS-vs-Linux probe via
ovs-vsctl is no longer needed for this field.
- Add checkLegacyWanBridgeKey() in cmd/service.go: vpcd fail-starts
if the deprecated 'wan_bridge' TOML key or SPINIFEX_VPCD_WAN_BRIDGE
env var is set. No silent alias, no auto-rewrite (per plan D3/D13).
- Template carries an operator-facing comment warning never to set
the field to 'br-ext'.
- 7 new unit tests for the detector branches and fail-start paths.
Smoke on ben-bottlebrush: spx admin init --force writes
dhcp_bind_bridge matching the live br-wan Linux bridge; VM launched
with --map-public-ip-on-launch reached from LAN; restoring a
legacy-key backup causes vpcd to enter failed state with the
deprecated-key guidance log.
Plan: docs/development/bugs/dhcp-vpcd-issues.md (Fix 4)
Bead: mulga-998.a (unblocker for .b / .c)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)
Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:
- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
/local/instances (placeholder types until daemon-local-autonomy lands),
plus /health for scaffolding validation
Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.
* feat: add DDIL harness fault injection and link profiles
Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:
- harness/fault.go — PartitionNode (per-peer iptables DROP with
orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
RestartDaemonOnly, WaitForMode (polls /local/status for typed
DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
SATCOM, HFData, Flapping constants (unvalidated approximations per
plan §2 table; validation tracking lives in TEST_COVERAGE.md)
All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.
* feat: add DDIL harness witness VM helpers
LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.
* feat: add DDIL harness snapshot and state assertions
TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.
* feat: add DDIL harness reset, retry, and quarantine
ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.
AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.
Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.
bd: mulga-siv-5.5
* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)
Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.
* feat: add DDIL E2E CI workflow (workflow_dispatch only)
Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
siv-8.5 migrated daemon + awsgw but missed the sidecar services and
the spx CLI. vpcd and viperblockd read cfg.NatsHost / cfg.S3Host raw
from spinifex.toml; when BindIP=0.0.0.0 the TLS handshake fails SAN
verification because 0.0.0.0 is not a cert SAN. Same for the spx get
CLI path.
Call sites:
spinifex/services/viperblockd/viperblockd.go:173 (NATS)
spinifex/services/viperblockd/viperblockd.go:407 (S3 s3.S3Config)
spinifex/services/viperblockd/viperblockd.go:533 (S3 nbd.NBDKitConfig)
spinifex/services/vpcd/vpcd.go:277 (NATS)
cmd/spinifex/cmd/get.go:93 (CLI NATS)
admin.DialTarget rewrites 0.0.0.0:N to 127.0.0.1:N — loopback is always
a SAN and in-process sidecars share the host with NATS/predastore.
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway
Convert br-mgmt from an externally-provisioned Linux bridge to an OVS
bridge owned by scripts/setup-ovn.sh, giving every deployment type
(tofu-cluster, production single-node, production multi-node) a single
provisioning path.
setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface /
--no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone
(excluded from ovn-bridge-mappings so ovn-controller ignores it) and
assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot.
SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master`
to `ovs-vsctl add-port` / `del-port` to match the new bridge type.
Add a loopback guard to wireLBAgentConfig's fallback branch: on a
production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge,
the fallback previously injected 127.0.0.1 into LB VM cloud-init —
silently unreachable because the VM's loopback is its own. The guard
leaves GatewayURL empty and the existing "LB agent gateway URL not
configured" error fires instead. Regression covered in
daemon_wire_lb_test.go.
* fix: auto-detect primary IP for admin init --bind default
Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback
only, so system VMs on br-mgmt could never reach it. Auto-detect the
default-route source IPv4 instead, which keeps TLS cert SANs specific
(still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1
explicitly to preserve loopback-only behavior on laptops.
Closes mulga-siv-7.
* test: refresh wire lb loopback-bind comment for siv-7
* fix: dev-install scripts should let admin init auto-detect --bind
Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while
setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with
no listener. Drop the explicit --bind so auto-detect picks the primary IP
and the siv-6 mgmtRouteVia path routes LB traffic to a real listener.
* fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
PR #144 (nats TLS) made every service listen on --bind's specific IP. For
awsgw and predastore this broke localhost-reachability that the UI proxy
(spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on —
latent since #144, surfaced by siv-7's auto-detect default on dev installs
(502 from the UI).
Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen
wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host
daemon → predastore) keep working, LAN clients still reach them via the
LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}}
because peer-advertised values must not resolve to 0.0.0.0/loopback.
predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1
for local-node consumers, so existing callers see a usable connect address.
* revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8)
Clears the deck before layering the listen-vs-advertise split
(mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node
default and introduces an explicit --advertise flag for off-host dial
targets; the reverted approach (auto-detect a specific IP at init,
hardcode 0.0.0.0 in the template, guard against loopback in
wireLBAgentConfig) becomes redundant once that lands.
Reverts:
- bb0ad3a fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
- f20a47b fix: dev-install scripts should let admin init auto-detect --bind
- 49b0b4e test: refresh wire lb loopback-bind comment for siv-7
- b88dfdf fix: auto-detect primary IP for admin init --bind default
Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig
fallback and its regression test — subsumed by the GatewayURL
precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error,
no silent empty-string fallthrough). The br-mgmt OVS provisioning
from 74b6e2d is retained.
Plan: docs/development/bugs/spx-admin-init-bind.md
* feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2)
Introduce the two small address-resolution helpers the listen-vs-
advertise split needs, with full unit coverage. No callers yet —
dial-site migration lands in mulga-siv-8.5.
- DialTarget(listenAddr): rewrites wildcard listen addresses
(0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process
clients dial a target the daemon's cert already has in its SANs.
Specific IPs, loopback, hostnames, and malformed input pass through.
- AdvertiseHost(listenAddr, advertiseIP): picks the host off-host
clients should dial. Wildcard listen → advertiseIP; specific host →
that host. Port is dropped so callers append the protocol-appropriate
port.
Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1
* feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3)
Pure data-model change. Add AdvertiseIP to:
- admin.ConfigSettings (local node, populated by init).
- config.Config (per-node; mapstructure "advertise").
- formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake
carries it between peers.
buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so
joined nodes from older clusters / pre-siv-8 joiners keep working.
No behaviour change yet — templates, CLI flag, and dial-site migration
land in siv-8.4 and siv-8.5.
* feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4)
Introduce the listen-vs-advertise split at the CLI boundary.
admin init/join:
- --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert).
- New --advertise <IP> flag on both init and join; optional, auto-detected
from DetectNetwork().WAN.IP when --bind is wildcard or loopback.
- resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements
precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 +
warning.
- Join resolves advertise before building the JoinRequest so peers record
a reachable dial target (not 0.0.0.0).
- isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator
can now run "init --nodes 3" without typing a specific --bind.
- Init success message prints bind / advertise / loopback three-way split.
Templates:
- spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered
only when AdvertiseIP is set.
- predastore-multinode.toml: iam.nats_url host wrapped in the same
{{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that
predastore.toml already uses, so co-resident predastore dials its local
NATS via loopback instead of 0.0.0.0.
Tests cover the resolveAdvertiseIP matrix, the template advertise field
(present + omitted), and the predastore nats_url loopback shim.
Dial-site migration and GatewayURL rework land in siv-8.5.
* feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5)
Switch every in-process dial site to admin.DialTarget so the wildcard
listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon
→ NATS and → predastore (boot, reconnect, status probe, S3 store, service
manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so
loopback-dial a wildcard-listen service is always valid; the old
0.0.0.0 literal was tripping TLS SAN verification.
Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go):
1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged)
2. br-mgmt present, AWSGW on 0.0.0.0 → br-mgmt IP (unchanged)
3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW)
4. DevNetworking shim → 10.0.2.2 (unchanged)
5. AWSGW on specific IP, no bridge → that IP (unchanged)
6. Else → log error, skip assignment
Kills the pre-siv-8 silent empty-URL failure mode that made
CreateLoadBalancer fail with 'missing system credentials: gatewayURL='
on every fresh single-node install. Supersedes siv-6's loopback guard.
Extend SetupAWSCredentials with a wanIP parameter (reserved for a
future --operator-endpoint flag) and thread AdvertiseIP through
finalizeNodeSetup's three call sites (single-node init, multi-node
init, join). Print 'Bind / Advertise / Loopback' on multi-node init
and join completion to match the single-node summary.
Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3
Replaces the sudo-dhcpcd shell-out in ObtainDHCPLease / ReleaseDHCPLease
with NATS request/reply against vpc.dhcp.acquire / vpc.dhcp.release.
The daemon-side IPAM passes its nats.Conn into NewExternalIPAM and
threads hostname (option 12) / vendor-class (option 60) through to
vpcd; ExternalIPAllocation now records LeaseExpiresUnix, DHCPServerID
and HWAddr on DHCP-sourced entries. allocateFromPool is restructured
to acquire the lease once up-front and rely on vpcd's idempotent
Manager for CAS-retry safety; a single release fires only if every
CAS attempt fails. Drops the stdout parser, the per-bridge mutex, and
the parse test.
Systemd: spinifex-daemon loses AF_PACKET, ProtectKernelTunables and
MemoryDenyWriteExecute are restored to =yes; spinifex-vpcd gains
AF_PACKET in RestrictAddressFamilies, AmbientCapabilities=CAP_NET_RAW,
and CAP_NET_ADMIN / CAP_NET_RAW / CAP_NET_BIND_SERVICE in the bounding
set. Net privilege footprint narrows.
See docs/development/bugs/dhcp-allocate-error.md.
Julian Sommer