mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
fix: group-scope the OVS/OVN sockets and retire the vpcd sudoers grants (#609)
* fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh, with a systemd override re-applying it on every openvswitch-switch restart. Write access to db.sock is full control of the local datapath, so this handed the datapath to any account on the box. Both service users run with Group=spinifex, so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller alike — the latter is new, and matters because the ovn-controller ctl socket name embeds the pid and is recreated on every start. That also fixes the health probe reporting ovn-controller as not_running on a healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is granted; as spinifex-daemon sudo demanded a password and the swallowed error read as OVN being down. With the ctl socket group-owned the probe needs no privilege, so health.go moves to a non-escalating Runner rather than the alternative of granting the daemon a root-equivalent ovn-appctl rule to read a status. ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick, so 1+N sudo invocations every tick, each one logging a pam_limits warning about the core rlimit the unit deliberately pins to 0. It now issues a single JSON query per pass, and the sudoers drop-in disables the PAM session for both service users, which also leaves sudo children inheriting the unit's rlimits. An absent iface-id on a patch port is now skipped with a warning instead of aborting the pass: it is the only record of the full ENI, so such a port cannot join the live set either way, and failing the pass stalled reconcile for every healthy port behind it. A transient query failure still aborts. * fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`: it resolves the pid-named socket through the pidfile, and OVS opens that O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which also tightens them from the 0644 they ship with. openvswitch-ipsec joins the override list because setup-ovn.sh restarts it after the sweep runs, leaving its ctl socket as the one file still at the shipped mode. Verified on the dev node: every socket and pidfile is 0660 root:spinifex and survives a restart of all three units, an unprivileged user is refused, and /health reports ovn-controller ok where the sudo probe read not_running. * refactor: stop escalating the OVS/OVN socket clients Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the service user. utils.NeedsPrivilege carries that policy and both command builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate while the socket clients no longer do. Their sudoers grants are removed: each took unrestricted args and all of them accept --log-file=PATH, so every one was a root-equivalent grant handed out to run a query. ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that ovs-vswitchd creates when the bridge appears, including bridges spinifex creates at runtime, so the provisioning sweep cannot group-own it up front. The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl. vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing the policy entirely. It survived only because the grants existed and broke the OVN flows-ready barrier the moment they were removed, so it now delegates to utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another package reintroduces a local copy. Verified on the dev node: an instance launches, IMDS is served, the flows-ready barrier completes, and no sudo denial appears anywhere in the journal. * refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant alone was root-equivalent: sudoers cannot constrain its arguments, so `sudo ip netns exec <ns> /bin/sh` was a root shell for the service account. The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel passes an ambient set to each child on exec, so those tools work as the service user with no escalation at all. Verified on a live node: all sixteen ip subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient set. spinifex-vpcd now has no sudoers rules. NeedsPrivilege had to become capability-aware to do this. spx is one binary shared by both service units, and the daemon holds CAP_SYS_ADMIN and CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express "vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb from /proc/self/status once and matches it against the capability each invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the net.* trees and nothing else. The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which runs as the provisioning user, and the in-guest ecs-agent. TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant cannot be re-added without also explaining why the capability stopped working. * chore: restore routeTree.gen.ts swept in by mistake * fix: probe ovn-controller with ovn-appctl, not ovs-appctl checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl, which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's socket and pidfile are in /var/run/ovn, so the probe could never succeed and logged "ovn-controller.pid: open: No such file or directory" on every check. Not a functional break: the next probe globs the full /var/run/ovn path and succeeds, so the function still returned nil. It cost an exec per check and made a healthy node look like it was failing an OVN liveness probe. ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches first. The glob and systemctl fallbacks stay for older layouts.
This commit is contained in:
Josh Siviour
+94
-25
@@ -941,40 +941,109 @@ else
|
||||
echo " WARNING: geneve module not available (tunnels may not work)"
|
||||
fi
|
||||
|
||||
# --- Step 8: Grant non-root access to OVS/OVN ---
|
||||
# --- Step 8: Grant the spinifex service users access to OVS/OVN ---
|
||||
echo ""
|
||||
echo "Step 8: Configuring non-root access..."
|
||||
echo "Step 8: Configuring service-user access..."
|
||||
|
||||
# Open OVS DB socket so non-root processes can use ovs-vsctl
|
||||
OVS_SOCK="/var/run/openvswitch/db.sock"
|
||||
if [ -S "$OVS_SOCK" ]; then
|
||||
sudo chmod 0666 "$OVS_SOCK"
|
||||
echo " OVS DB socket: opened ($OVS_SOCK)"
|
||||
# Group-scoped, never world. spinifex-daemon and spinifex-vpcd both run with
|
||||
# Group=spinifex, so 0660 root:spinifex is exactly the reach they need. A
|
||||
# world-writable db.sock hands the local datapath — bridges, ports, ovn-remote —
|
||||
# to any account on the box, and a world-writable ctl socket lets any account
|
||||
# run `ovn-appctl exit`.
|
||||
PERMS_HELPER="/usr/local/lib/spinifex/ovs-socket-perms.sh"
|
||||
sudo mkdir -p "$(dirname "$PERMS_HELPER")"
|
||||
sudo tee "$PERMS_HELPER" >/dev/null <<'HELPER'
|
||||
#!/bin/sh
|
||||
# Group-own the OVS/OVN control sockets to `spinifex` so the service users reach
|
||||
# them without sudo. Driven from ExecStartPost on both openvswitch-switch and
|
||||
# ovn-controller: each recreates its own sockets on start, and the ovn-controller
|
||||
# ctl socket name embeds the pid, so it is a new file every time.
|
||||
#
|
||||
# $1 is an optional glob for the caller's own socket, waited on before the sweep.
|
||||
set -eu
|
||||
GROUP=spinifex
|
||||
WAIT_FOR="${1:-}"
|
||||
|
||||
if ! getent group "$GROUP" >/dev/null 2>&1; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Open OVN runtime directory and ctl sockets for ovs-appctl access
|
||||
if [ -d "/var/run/ovn" ]; then
|
||||
sudo chmod 0755 /var/run/ovn
|
||||
sudo chmod 0666 /var/run/ovn/*.ctl 2>/dev/null || true
|
||||
echo " OVN ctl sockets: opened (/var/run/ovn/)"
|
||||
fi
|
||||
if [ -d "/var/run/openvswitch" ]; then
|
||||
sudo chmod 0666 /var/run/openvswitch/*.ctl 2>/dev/null || true
|
||||
# Unquoted so the caller's pattern globs. A caller passing nothing sweeps at once.
|
||||
wait_target_present() {
|
||||
if [ -z "$WAIT_FOR" ]; then
|
||||
return 0
|
||||
fi
|
||||
for s in $WAIT_FOR; do
|
||||
if [ -S "$s" ]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# ExecStartPost can outrun socket creation, so poll briefly rather than miss the
|
||||
# socket and leave the health probe unable to read connection-status.
|
||||
i=0
|
||||
while [ "$i" -lt 25 ]; do
|
||||
if wait_target_present; then
|
||||
break
|
||||
fi
|
||||
i=$((i + 1))
|
||||
sleep 0.2
|
||||
done
|
||||
|
||||
# The dir must be traversable by the group before the sockets inside it matter.
|
||||
# Only the group changes; the owner keeps full access whoever it is.
|
||||
if [ -d /var/run/ovn ]; then
|
||||
chgrp "$GROUP" /var/run/ovn 2>/dev/null || true
|
||||
chmod 0750 /var/run/ovn 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Create persistent systemd override so permissions survive OVS restarts
|
||||
OVERRIDE_DIR="/etc/systemd/system/openvswitch-switch.service.d"
|
||||
if [ ! -f "$OVERRIDE_DIR/spinifex-perms.conf" ]; then
|
||||
# ovn??_db.sock are the NB/SB databases themselves, which ovn-nbctl and
|
||||
# ovn-sbctl connect to. Bridge <name>.mgmt sockets are deliberately NOT here:
|
||||
# ovs-vswitchd creates one whenever a bridge appears, including bridges spinifex
|
||||
# creates at runtime, so ovs-ofctl keeps its sudo grant.
|
||||
for s in /var/run/openvswitch/db.sock /var/run/openvswitch/*.ctl \
|
||||
/var/run/ovn/*.ctl /var/run/ovn/ovnnb_db.sock /var/run/ovn/ovnsb_db.sock; do
|
||||
if [ -S "$s" ]; then
|
||||
chgrp "$GROUP" "$s" 2>/dev/null || true
|
||||
chmod 0660 "$s" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
|
||||
# Group-WRITE on the pidfiles, not just read: `ovn-appctl -t <daemon>` resolves
|
||||
# the pid-named ctl socket through the pidfile, and OVS opens it O_RDWR to test
|
||||
# the fcntl liveness lock. Read-only fails the probe with EACCES. This also
|
||||
# tightens them from the 0644 they ship with.
|
||||
for p in /var/run/openvswitch/*.pid /var/run/ovn/*.pid; do
|
||||
if [ -f "$p" ]; then
|
||||
chgrp "$GROUP" "$p" 2>/dev/null || true
|
||||
chmod 0660 "$p" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
HELPER
|
||||
sudo chmod 0755 "$PERMS_HELPER"
|
||||
sudo "$PERMS_HELPER"
|
||||
echo " OVS/OVN sockets: 0660 root:spinifex (group-scoped, no world access)"
|
||||
|
||||
# Persist across restarts of both daemons. Rewritten unconditionally: an existing
|
||||
# file is the old 0666 override, and skipping would leave that exposure in place.
|
||||
# openvswitch-ipsec is included because Step 10 restarts it after this sweep,
|
||||
# so its ctl socket would otherwise be the one file left at the shipped mode.
|
||||
for unit in openvswitch-switch:/var/run/openvswitch/db.sock \
|
||||
"ovn-controller:/var/run/ovn/*.ctl" \
|
||||
"openvswitch-ipsec:/var/run/openvswitch/ovs-monitor-ipsec.*.ctl"; do
|
||||
UNIT="${unit%%:*}"
|
||||
WAIT_GLOB="${unit#*:}"
|
||||
OVERRIDE_DIR="/etc/systemd/system/${UNIT}.service.d"
|
||||
sudo mkdir -p "$OVERRIDE_DIR"
|
||||
sudo tee "$OVERRIDE_DIR/spinifex-perms.conf" >/dev/null <<'OVERRIDE'
|
||||
sudo tee "$OVERRIDE_DIR/spinifex-perms.conf" >/dev/null <<OVERRIDE
|
||||
[Service]
|
||||
ExecStartPost=/bin/chmod 0666 /var/run/openvswitch/db.sock
|
||||
ExecStartPost=$PERMS_HELPER "$WAIT_GLOB"
|
||||
OVERRIDE
|
||||
sudo systemctl daemon-reload
|
||||
echo " systemd override: created (db.sock permissions persist across restarts)"
|
||||
else
|
||||
echo " systemd override: already exists"
|
||||
fi
|
||||
echo " systemd override: ${UNIT}.service.d/spinifex-perms.conf"
|
||||
done
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Sudoers rules for spinifex-daemon and spinifex-vpcd are managed by setup.sh
|
||||
# (install_sudoers). Skip writing here to avoid conflicts.
|
||||
|
||||
+29
-21
@@ -184,31 +184,39 @@ create_service_users() {
|
||||
install_sudoers() {
|
||||
stage "installing scoped sudoers rules"
|
||||
$SUDO tee /etc/sudoers.d/spinifex-network > /dev/null << 'SUDOERS'
|
||||
# Spinifex daemon: tap devices, OVS bridge management, and DHCP for external IPs
|
||||
# ovs-ofctl installs the per-tap IMDS datapath flows on br-imds; sysctl sets the
|
||||
# per-endpoint rp_filter/accept_local the asymmetric reply path needs.
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /sbin/ip, /usr/sbin/ip
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/ovs-vsctl, /usr/bin/ovs-appctl, /usr/bin/ovs-ofctl
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /usr/sbin/sysctl -qw net.ipv4.conf.*
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /usr/sbin/dhcpcd
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/ovn-nbctl set NB_Global . ipsec=true
|
||||
# No PAM session for the service users. Every sudo call otherwise logs
|
||||
# "pam_limits: Could not set limit for 'core'": the units set LimitCORE=0 and
|
||||
# their CapabilityBoundingSet omits CAP_SYS_RESOURCE, so pam_limits cannot apply
|
||||
# the limits.conf default and warns. It also means sudo children inherit the
|
||||
# unit's rlimits rather than the system defaults, which is the RG-9 intent.
|
||||
Defaults:spinifex-daemon !pam_session
|
||||
|
||||
# Spinifex VPC daemon: OVN and OVS read/write, OVN controller status check and DHCP
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/dhcpcd
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/ovs-vsctl, /usr/bin/ovs-appctl
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/ovn-nbctl, /usr/bin/ovn-sbctl, /usr/bin/ovn-appctl
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/systemctl is-active --quiet ovn-controller
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /sbin/ip, /usr/sbin/ip
|
||||
# Routed-NAT mode: transit masquerade + per-EIP FORWARD accepts, proxy-ARP
|
||||
# delay tune, and gratuitous ARP announcements for host-delivered EIPs.
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/iptables, /sbin/iptables
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/sysctl -w net.ipv4.neigh.*
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/arping, /usr/bin/arping
|
||||
# The OVS/OVN client tools are deliberately absent. They do all their work over
|
||||
# control sockets that setup-ovn.sh group-owns to `spinifex`, so they run as the
|
||||
# service user. Granting them would be worse than pointless: a NOPASSWD rule for
|
||||
# them takes unrestricted args, and ovs-vsctl/ovn-nbctl/ovn-appctl all accept
|
||||
# --log-file=PATH, which writes a root-owned file wherever the caller points it.
|
||||
#
|
||||
# ovs-ofctl is the exception that keeps its grant: it talks to a per-bridge
|
||||
# <bridge>.mgmt socket created by ovs-vswitchd when the bridge appears, including
|
||||
# bridges spinifex creates at runtime, so it cannot be group-owned up front.
|
||||
|
||||
# spinifex-vpcd has no rules at all. Its ip/iptables/sysctl/arping work is done
|
||||
# under the CAP_NET_ADMIN and CAP_NET_RAW its unit grants ambiently, which the
|
||||
# kernel passes to each child on exec. Those grants were also root-equivalent:
|
||||
# `sudo ip netns exec <ns> /bin/sh` is a root shell.
|
||||
|
||||
# Spinifex daemon: tap devices and OpenFlow rules. Its unit holds no
|
||||
# CAP_NET_ADMIN, so unlike vpcd it cannot drop these. ovs-ofctl installs the
|
||||
# per-tap IMDS datapath flows on br-imds; sysctl sets the per-endpoint
|
||||
# rp_filter/accept_local the asymmetric reply path needs.
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /sbin/ip, /usr/sbin/ip
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/ovs-ofctl
|
||||
spinifex-daemon ALL=(root) NOPASSWD: /usr/sbin/sysctl -qw net.ipv4.conf.*
|
||||
SUDOERS
|
||||
$SUDO chmod 0440 /etc/sudoers.d/spinifex-network
|
||||
$SUDO visudo -cf /etc/sudoers.d/spinifex-network || fatal "Invalid sudoers syntax in spinifex-network"
|
||||
info "Scoped sudoers rules installed for spinifex-daemon and spinifex-vpcd"
|
||||
info "Scoped sudoers rules installed for spinifex-daemon (spinifex-vpcd needs none)"
|
||||
}
|
||||
|
||||
# --- Install apt dependencies ---
|
||||
|
||||
Reference in New Issue
Block a user