siv-8.5 migrated daemon + awsgw but missed the sidecar services and
the spx CLI. vpcd and viperblockd read cfg.NatsHost / cfg.S3Host raw
from spinifex.toml; when BindIP=0.0.0.0 the TLS handshake fails SAN
verification because 0.0.0.0 is not a cert SAN. Same for the spx get
CLI path.
Call sites:
spinifex/services/viperblockd/viperblockd.go:173 (NATS)
spinifex/services/viperblockd/viperblockd.go:407 (S3 s3.S3Config)
spinifex/services/viperblockd/viperblockd.go:533 (S3 nbd.NBDKitConfig)
spinifex/services/vpcd/vpcd.go:277 (NATS)
cmd/spinifex/cmd/get.go:93 (CLI NATS)
admin.DialTarget rewrites 0.0.0.0:N to 127.0.0.1:N — loopback is always
a SAN and in-process sidecars share the host with NATS/predastore.
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway
Convert br-mgmt from an externally-provisioned Linux bridge to an OVS
bridge owned by scripts/setup-ovn.sh, giving every deployment type
(tofu-cluster, production single-node, production multi-node) a single
provisioning path.
setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface /
--no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone
(excluded from ovn-bridge-mappings so ovn-controller ignores it) and
assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot.
SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master`
to `ovs-vsctl add-port` / `del-port` to match the new bridge type.
Add a loopback guard to wireLBAgentConfig's fallback branch: on a
production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge,
the fallback previously injected 127.0.0.1 into LB VM cloud-init —
silently unreachable because the VM's loopback is its own. The guard
leaves GatewayURL empty and the existing "LB agent gateway URL not
configured" error fires instead. Regression covered in
daemon_wire_lb_test.go.
* fix: auto-detect primary IP for admin init --bind default
Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback
only, so system VMs on br-mgmt could never reach it. Auto-detect the
default-route source IPv4 instead, which keeps TLS cert SANs specific
(still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1
explicitly to preserve loopback-only behavior on laptops.
Closes mulga-siv-7.
* test: refresh wire lb loopback-bind comment for siv-7
* fix: dev-install scripts should let admin init auto-detect --bind
Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while
setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with
no listener. Drop the explicit --bind so auto-detect picks the primary IP
and the siv-6 mgmtRouteVia path routes LB traffic to a real listener.
* fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
PR #144 (nats TLS) made every service listen on --bind's specific IP. For
awsgw and predastore this broke localhost-reachability that the UI proxy
(spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on —
latent since #144, surfaced by siv-7's auto-detect default on dev installs
(502 from the UI).
Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen
wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host
daemon → predastore) keep working, LAN clients still reach them via the
LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}}
because peer-advertised values must not resolve to 0.0.0.0/loopback.
predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1
for local-node consumers, so existing callers see a usable connect address.
* revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8)
Clears the deck before layering the listen-vs-advertise split
(mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node
default and introduces an explicit --advertise flag for off-host dial
targets; the reverted approach (auto-detect a specific IP at init,
hardcode 0.0.0.0 in the template, guard against loopback in
wireLBAgentConfig) becomes redundant once that lands.
Reverts:
- bb0ad3a fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
- f20a47b fix: dev-install scripts should let admin init auto-detect --bind
- 49b0b4e test: refresh wire lb loopback-bind comment for siv-7
- b88dfdf fix: auto-detect primary IP for admin init --bind default
Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig
fallback and its regression test — subsumed by the GatewayURL
precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error,
no silent empty-string fallthrough). The br-mgmt OVS provisioning
from 74b6e2d is retained.
Plan: docs/development/bugs/spx-admin-init-bind.md
* feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2)
Introduce the two small address-resolution helpers the listen-vs-
advertise split needs, with full unit coverage. No callers yet —
dial-site migration lands in mulga-siv-8.5.
- DialTarget(listenAddr): rewrites wildcard listen addresses
(0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process
clients dial a target the daemon's cert already has in its SANs.
Specific IPs, loopback, hostnames, and malformed input pass through.
- AdvertiseHost(listenAddr, advertiseIP): picks the host off-host
clients should dial. Wildcard listen → advertiseIP; specific host →
that host. Port is dropped so callers append the protocol-appropriate
port.
Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1
* feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3)
Pure data-model change. Add AdvertiseIP to:
- admin.ConfigSettings (local node, populated by init).
- config.Config (per-node; mapstructure "advertise").
- formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake
carries it between peers.
buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so
joined nodes from older clusters / pre-siv-8 joiners keep working.
No behaviour change yet — templates, CLI flag, and dial-site migration
land in siv-8.4 and siv-8.5.
* feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4)
Introduce the listen-vs-advertise split at the CLI boundary.
admin init/join:
- --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert).
- New --advertise <IP> flag on both init and join; optional, auto-detected
from DetectNetwork().WAN.IP when --bind is wildcard or loopback.
- resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements
precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 +
warning.
- Join resolves advertise before building the JoinRequest so peers record
a reachable dial target (not 0.0.0.0).
- isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator
can now run "init --nodes 3" without typing a specific --bind.
- Init success message prints bind / advertise / loopback three-way split.
Templates:
- spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered
only when AdvertiseIP is set.
- predastore-multinode.toml: iam.nats_url host wrapped in the same
{{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that
predastore.toml already uses, so co-resident predastore dials its local
NATS via loopback instead of 0.0.0.0.
Tests cover the resolveAdvertiseIP matrix, the template advertise field
(present + omitted), and the predastore nats_url loopback shim.
Dial-site migration and GatewayURL rework land in siv-8.5.
* feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5)
Switch every in-process dial site to admin.DialTarget so the wildcard
listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon
→ NATS and → predastore (boot, reconnect, status probe, S3 store, service
manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so
loopback-dial a wildcard-listen service is always valid; the old
0.0.0.0 literal was tripping TLS SAN verification.
Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go):
1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged)
2. br-mgmt present, AWSGW on 0.0.0.0 → br-mgmt IP (unchanged)
3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW)
4. DevNetworking shim → 10.0.2.2 (unchanged)
5. AWSGW on specific IP, no bridge → that IP (unchanged)
6. Else → log error, skip assignment
Kills the pre-siv-8 silent empty-URL failure mode that made
CreateLoadBalancer fail with 'missing system credentials: gatewayURL='
on every fresh single-node install. Supersedes siv-6's loopback guard.
Extend SetupAWSCredentials with a wanIP parameter (reserved for a
future --operator-endpoint flag) and thread AdvertiseIP through
finalizeNodeSetup's three call sites (single-node init, multi-node
init, join). Print 'Bind / Advertise / Loopback' on multi-node init
and join completion to match the single-node summary.
Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3
Replaces the sudo-dhcpcd shell-out in ObtainDHCPLease / ReleaseDHCPLease
with NATS request/reply against vpc.dhcp.acquire / vpc.dhcp.release.
The daemon-side IPAM passes its nats.Conn into NewExternalIPAM and
threads hostname (option 12) / vendor-class (option 60) through to
vpcd; ExternalIPAllocation now records LeaseExpiresUnix, DHCPServerID
and HWAddr on DHCP-sourced entries. allocateFromPool is restructured
to acquire the lease once up-front and rely on vpcd's idempotent
Manager for CAS-retry safety; a single release fires only if every
CAS attempt fails. Drops the stdout parser, the per-bridge mutex, and
the parse test.
Systemd: spinifex-daemon loses AF_PACKET, ProtectKernelTunables and
MemoryDenyWriteExecute are restored to =yes; spinifex-vpcd gains
AF_PACKET in RestrictAddressFamilies, AmbientCapabilities=CAP_NET_RAW,
and CAP_NET_ADMIN / CAP_NET_RAW / CAP_NET_BIND_SERVICE in the bounding
set. Net privilege footprint narrows.
See docs/development/bugs/dhcp-allocate-error.md.
Introduces the spinifex/services/vpcd/dhcp package with a Client
interface, typed Lease struct, nclient4-backed implementation, and a
Fake for unit tests. Adds DHCPManager in the vpcd package: subscribes
to vpc.dhcp.acquire / vpc.dhcp.release NATS requests, runs a renewal
goroutine per lease with ±10% jitter on T1/T2, persists leases in the
new spinifex-dhcp-leases KV bucket, and fans out vpc.dhcp.lease-expired
when a lease permanently fails to renew. Wires the Manager into
launchService so it starts on vpcd boot and rehydrates persisted leases
via Bootstrap.
Handler-side wiring (ExternalIPAM → NATS) and the systemd capability
move land separately in mulga-991.
See docs/development/bugs/dhcp-allocate-error.md.
Stop ObtainDHCPLease from erroring on hosts where dhcpcd is already
running as a daemon managing br-wan (e.g. server 2). Add -T (test-only
mode) so the invocation does not collide with the host daemon's
control socket on the interface.
Softens ProtectKernelTunables and MemoryDenyWriteExecute in the daemon
unit to accommodate dhcpcd's runtime requirements; both revert when
the in-process Go DHCP client rewrite lands (mulga-989 / mulga-990 /
mulga-991, see docs/development/bugs/dhcp-allocate-error.md).
Interim: leaves the phantom-lease and release-socket-clobber risks
active until the rewrite replaces this path entirely.
Internet-facing ALBs allocated their public IP by calling
externalIPAM.AllocateIP() directly from LaunchSystemInstance, bypassing
the EIP service entirely. No EIPRecord landed in the eips KV bucket, the
tag was "auto_assign" (EIP service uses "elastic_ip"), and
DescribeAddresses never returned the IP — so any OpenTofu flow that
expects an AllocateAddress-observable record hung waiting.
LaunchSystemInstance now calls eipService.AllocateAddress +
AssociateAddress when the EIP service is wired (falls back to the old
direct IPAM path otherwise). TerminateSystemInstance releases through
the service so the KV record is cleaned up and vpc.delete-nat is
published. The generic stopInstance teardown is left untouched so
customer RunInstances flows behave identically.
Also refreshes the nginx-alb terraform example to use public subnets
with auto-assigned public IPs (the configuration that actually exercises
this path end-to-end) and tightens the awsgwBindIP loopback guard in
wireLBAgentConfig so the gateway URL is usable by LB agents.
* fix: replace stringly-typed TLS error detection with sentinel errors
Export ErrCACertRead and ErrCACertParse from utils/nats.go so callers can
detect permanent TLS configuration failures with errors.Is instead of
substring matching on error messages. Update daemon.go and awsgw.go retry
loops accordingly, and update tests to use assert.ErrorIs.
Refs: mulga-siv-1
* refactor: extract shared NATS connect-with-retry into utils
daemon.connectNATS() and awsgw.connectNATS() were near-identical retry
loops. Extract to utils.ConnectNATSWithRetry() with functional options
for test configurability.
Reset now targets /etc/spinifex, /var/lib/spinifex, /var/log/spinifex,
/run/spinifex (the privilege-separated layout installed by setup.sh) and
drives re-install from a source build via dev-install.sh instead of the
legacy in-tree $HOME/spinifex + start-dev.sh flow.
Capture region, AZ, external_mode, pool range, gateway, and prefix from
the existing spinifex.toml before the wipe so the reset restores the
current topology rather than applying defaults. Fresh boxes fall back to
ap-southeast-2 and a pool derived from the WAN subnet.
Drop the macvlan branch entirely — bridge-only from here on. Operators
on single-NIC hosts must run reset from the console (documented in the
script header).
dev-install.sh gains DEV_INSTALL_SKIP_INIT so reset can own the post-
setup.sh steps (setup-ovn, spx admin init with preserved args, CA
trust, service start, LB image import). Also honors SPINIFEX_REGION /
SPINIFEX_AZ for non-reset callers, and scopes systemctl reset-failed
to spinifex-* so unrelated failed units aren't silently cleared.
Closes mulga-983.
Ben Duncan