gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-22 20:26:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: spx admin init/join listen-vs-advertise split + br-mgmt OVS provisioning (#158)

Browse Source 
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway

Convert br-mgmt from an externally-provisioned Linux bridge to an OVS
bridge owned by scripts/setup-ovn.sh, giving every deployment type
(tofu-cluster, production single-node, production multi-node) a single
provisioning path.

setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface /
--no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone
(excluded from ovn-bridge-mappings so ovn-controller ignores it) and
assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot.

SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master`
to `ovs-vsctl add-port` / `del-port` to match the new bridge type.

Add a loopback guard to wireLBAgentConfig's fallback branch: on a
production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge,
the fallback previously injected 127.0.0.1 into LB VM cloud-init —
silently unreachable because the VM's loopback is its own. The guard
leaves GatewayURL empty and the existing "LB agent gateway URL not
configured" error fires instead. Regression covered in
daemon_wire_lb_test.go.

* fix: auto-detect primary IP for admin init --bind default

Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback
only, so system VMs on br-mgmt could never reach it. Auto-detect the
default-route source IPv4 instead, which keeps TLS cert SANs specific
(still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1
explicitly to preserve loopback-only behavior on laptops.

Closes mulga-siv-7.

* test: refresh wire lb loopback-bind comment for siv-7

* fix: dev-install scripts should let admin init auto-detect --bind

Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while
setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with
no listener. Drop the explicit --bind so auto-detect picks the primary IP
and the siv-6 mgmtRouteVia path routes LB traffic to a real listener.

* fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template

PR #144 (nats TLS) made every service listen on --bind's specific IP. For
awsgw and predastore this broke localhost-reachability that the UI proxy
(spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on —
latent since #144, surfaced by siv-7's auto-detect default on dev installs
(502 from the UI).

Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen
wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host
daemon → predastore) keep working, LAN clients still reach them via the
LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}}
because peer-advertised values must not resolve to 0.0.0.0/loopback.
predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1
for local-node consumers, so existing callers see a usable connect address.

* revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8)

Clears the deck before layering the listen-vs-advertise split
(mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node
default and introduces an explicit --advertise flag for off-host dial
targets; the reverted approach (auto-detect a specific IP at init,
hardcode 0.0.0.0 in the template, guard against loopback in
wireLBAgentConfig) becomes redundant once that lands.

Reverts:
- bb0ad3a fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
- f20a47b fix: dev-install scripts should let admin init auto-detect --bind
- 49b0b4e test: refresh wire lb loopback-bind comment for siv-7
- b88dfdf fix: auto-detect primary IP for admin init --bind default

Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig
fallback and its regression test — subsumed by the GatewayURL
precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error,
no silent empty-string fallthrough). The br-mgmt OVS provisioning
from 74b6e2d is retained.

Plan: docs/development/bugs/spx-admin-init-bind.md

* feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2)

Introduce the two small address-resolution helpers the listen-vs-
advertise split needs, with full unit coverage. No callers yet —
dial-site migration lands in mulga-siv-8.5.

- DialTarget(listenAddr): rewrites wildcard listen addresses
  (0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process
  clients dial a target the daemon's cert already has in its SANs.
  Specific IPs, loopback, hostnames, and malformed input pass through.
- AdvertiseHost(listenAddr, advertiseIP): picks the host off-host
  clients should dial. Wildcard listen → advertiseIP; specific host →
  that host. Port is dropped so callers append the protocol-appropriate
  port.

Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1

* feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3)

Pure data-model change. Add AdvertiseIP to:
- admin.ConfigSettings (local node, populated by init).
- config.Config (per-node; mapstructure "advertise").
- formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake
  carries it between peers.

buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so
joined nodes from older clusters / pre-siv-8 joiners keep working.

No behaviour change yet — templates, CLI flag, and dial-site migration
land in siv-8.4 and siv-8.5.

* feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4)

Introduce the listen-vs-advertise split at the CLI boundary.

admin init/join:
- --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert).
- New --advertise <IP> flag on both init and join; optional, auto-detected
  from DetectNetwork().WAN.IP when --bind is wildcard or loopback.
- resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements
  precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 +
  warning.
- Join resolves advertise before building the JoinRequest so peers record
  a reachable dial target (not 0.0.0.0).
- isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator
  can now run "init --nodes 3" without typing a specific --bind.
- Init success message prints bind / advertise / loopback three-way split.

Templates:
- spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered
  only when AdvertiseIP is set.
- predastore-multinode.toml: iam.nats_url host wrapped in the same
  {{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that
  predastore.toml already uses, so co-resident predastore dials its local
  NATS via loopback instead of 0.0.0.0.

Tests cover the resolveAdvertiseIP matrix, the template advertise field
(present + omitted), and the predastore nats_url loopback shim.

Dial-site migration and GatewayURL rework land in siv-8.5.

* feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5)

Switch every in-process dial site to admin.DialTarget so the wildcard
listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon
→ NATS and → predastore (boot, reconnect, status probe, S3 store, service
manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so
loopback-dial a wildcard-listen service is always valid; the old
0.0.0.0 literal was tripping TLS SAN verification.

Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go):

  1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged)
  2. br-mgmt present, AWSGW on 0.0.0.0     → br-mgmt IP (unchanged)
  3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW)
  4. DevNetworking shim                    → 10.0.2.2 (unchanged)
  5. AWSGW on specific IP, no bridge       → that IP (unchanged)
  6. Else                                  → log error, skip assignment

Kills the pre-siv-8 silent empty-URL failure mode that made
CreateLoadBalancer fail with 'missing system credentials: gatewayURL='
on every fresh single-node install. Supersedes siv-6's loopback guard.

Extend SetupAWSCredentials with a wanIP parameter (reserved for a
future --operator-endpoint flag) and thread AdvertiseIP through
finalizeNodeSetup's three call sites (single-node init, multi-node
init, join). Print 'Bind / Advertise / Loopback' on multi-node init
and join completion to match the single-node summary.

Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3
This commit is contained in:
Josh Siviour
2026-04-23 14:02:12 +10:00
committed by GitHub
parent 3c5e09a937
commit 75fb15be2b
15 changed files with 530 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/setup-ovn.sh
+77 -1
View File
@@ -20,6 +20,10 @@
# --macvlan Create macvlan off --wan-iface instead of moving NIC directly.
# SSH-safe for single-NIC hosts where WAN NIC carries SSH.
# --dhcp Obtain gateway IP via DHCP on the WAN bridge interface
# --mgmt-bridge=NAME OVS bridge for system-instance control plane (default: br-mgmt)
# --mgmt-cidr=CIDR IPv4 CIDR to assign on the mgmt bridge (default: 10.15.8.1/24)
# --mgmt-iface=NAME Physical/virtual NIC to enslave to the mgmt bridge (multi-node only)
# --no-mgmt-bridge Skip mgmt bridge provisioning (for dev-networking hosts)
# --ovn-remote=ADDR OVN SB DB address (default: tcp:127.0.0.1:6642)
# --encap-ip=IP Geneve tunnel endpoint IP (default: auto-detect)
# --chassis-id=ID OVN chassis identifier (default: chassis-$(hostname -s))
@@ -55,6 +59,10 @@ WAN_BRIDGE=""
WAN_IFACE=""
MACVLAN_MODE=false
EXTERNAL_DHCP=false
MGMT_BRIDGE_ENABLED=true
MGMT_BRIDGE="br-mgmt"
MGMT_CIDR="10.15.8.1/24"
MGMT_IFACE=""
OVN_REMOTE="tcp:127.0.0.1:6642"
ENCAP_IP=""
CHASSIS_ID=""
@@ -67,11 +75,15 @@ for arg in "$@"; do
--dhcp) EXTERNAL_DHCP=true ;;
--wan-bridge=*) WAN_BRIDGE="${arg#*=}" ;;
--wan-iface=*) WAN_IFACE="${arg#*=}" ;;
--mgmt-bridge=*) MGMT_BRIDGE="${arg#*=}" ;;
--mgmt-cidr=*) MGMT_CIDR="${arg#*=}" ;;
--mgmt-iface=*) MGMT_IFACE="${arg#*=}" ;;
--no-mgmt-bridge) MGMT_BRIDGE_ENABLED=false ;;
--ovn-remote=*) OVN_REMOTE="${arg#*=}" ;;
--encap-ip=*) ENCAP_IP="${arg#*=}" ;;
--chassis-id=*) CHASSIS_ID="${arg#*=}" ;;
--help|-h)
head -44 "$0" | tail -42
head -50 "$0" | tail -48
exit 0
;;
*)
@@ -451,6 +463,59 @@ if [ -n "$WAN_BRIDGE" ]; then
fi
fi
# --- Step 3d: Management bridge for system-instance control plane ---
# br-mgmt is an OVS bridge (not L2-learning Linux bridge, not part of OVN
# overlay). fail-mode=standalone makes it behave like a plain L2 switch
# without OVN flows; it is excluded from ovn-bridge-mappings so
# ovn-controller ignores it. System instance TAPs attach via ovs-vsctl
# add-port (see SetupMgmtTapDevice in daemon/network.go).
if [ "$MGMT_BRIDGE_ENABLED" = true ]; then
echo ""
echo "Step 3d: Configuring management bridge ($MGMT_BRIDGE)..."
sudo ovs-vsctl --may-exist add-br "$MGMT_BRIDGE"
sudo ovs-vsctl set Bridge "$MGMT_BRIDGE" \
fail-mode=standalone \
other-config:disable-in-band=true
sudo ip link set "$MGMT_BRIDGE" up
echo " $MGMT_BRIDGE: OVS bridge, fail-mode=standalone, up"
# Enslave physical/virtual mgmt NIC when provided (multi-node deployments
# where the node's mgmt subnet spans multiple hosts).
if [ -n "$MGMT_IFACE" ]; then
if ! ip link show "$MGMT_IFACE" >/dev/null 2>&1; then
echo " ERROR: mgmt interface $MGMT_IFACE does not exist"
ip -o link show | awk -F': ' '{print " " $2}'
exit 1
fi
# Drop any existing IP on the NIC — IP belongs on the bridge.
sudo ip addr flush dev "$MGMT_IFACE" || true
sudo ovs-vsctl --may-exist add-port "$MGMT_BRIDGE" "$MGMT_IFACE"
sudo ip link set "$MGMT_IFACE" up
echo " $MGMT_IFACE: port on $MGMT_BRIDGE"
fi
if [ -n "$MGMT_CIDR" ]; then
sudo ip addr replace "$MGMT_CIDR" dev "$MGMT_BRIDGE"
echo " $MGMT_BRIDGE: address $MGMT_CIDR"
# Persist the IP across reboots via a systemd-networkd drop-in. The
# OVS bridge definition itself survives in ovsdb; only the L3 IP
# needs re-applying on boot.
MGMT_NETD_UNIT="/etc/systemd/network/10-spinifex-mgmt.network"
sudo tee "$MGMT_NETD_UNIT" >/dev/null <<NETD
[Match]
Name=$MGMT_BRIDGE
[Network]
Address=$MGMT_CIDR
ConfigureWithoutCarrier=yes
NETD
echo " wrote $MGMT_NETD_UNIT (IP persists on reboot via systemd-networkd)"
fi
fi
# --- Step 4: Configure OVN external_ids ---
echo ""
echo "Step 4: Setting OVS external_ids for OVN..."
@@ -669,6 +734,17 @@ if [ -n "$WAN_BRIDGE" ]; then
fi
fi
# Check mgmt bridge (only if enabled)
if [ "$MGMT_BRIDGE_ENABLED" = true ]; then
if sudo ovs-vsctl br-exists "$MGMT_BRIDGE"; then
MGMT_IP_ACTUAL=$(ip -4 -o addr show dev "$MGMT_BRIDGE" 2>/dev/null | awk '{print $4}' | head -1)
echo " $MGMT_BRIDGE:$(printf '%*s' $((15 - ${#MGMT_BRIDGE})) '') OK (${MGMT_IP_ACTUAL:-no IP})"
else
echo " $MGMT_BRIDGE:$(printf '%*s' $((15 - ${#MGMT_BRIDGE})) '') FAILED"
OK=false
fi
fi
# Check ovn-controller
if sudo ovs-appctl -t ovn-controller version >/dev/null 2>&1 || systemctl is-active --quiet ovn-controller 2>/dev/null; then
echo " ovn-controller: OK"