mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
287487a4f2ffcc493be36fbeec369893c41d61ba
205 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
f877f5edef |
feat(network/external): restore source=dhcp on external pools (#262)
* refactor(network/external): introduce Allocator interface + StaticPoolAllocator
Carve external IP allocation out of handlers/ec2/vpc/external_ipam.go into
network/external/static_pool.go behind a typed Allocator interface. The KV
bucket name, schema, CAS retry count, and gateway-IP reservation semantics
are unchanged so existing static-pool data carries over with no manual
migration.
ExternalIPAM stays in handlers/ec2/vpc (L5 cleanup deferred per plan) but
becomes a thin facade that constructs a StaticPoolAllocator and delegates
Allocate / Release / GetPoolRecord through the new interface. Q4 will plug
DHCPPoolAllocator into the same Allocator slot.
Plan: docs/development/improvements/external-pool-dhcp-source.md (Q1).
* feat(network/external/dhcp): package skeleton — Client + Fake + KV store
New network/external/dhcp/ package, restored from the structure deleted
in Slice C (commit
|
||
|
tomnewton-mulga
|
78d703da78 |
Feat/mig capable (#260)
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit
|
||
|
Josh Siviour
|
fd3dc6b680 |
feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255)
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
|
||
|
Julian Sommer
|
b3d8a7d262 | fix: trim sleep times | ||
|
Julian Sommer
|
0ff911fdbc | fix: change predastore node ips to 127.0.0.1 (#244) | ||
|
Julian Sommer
|
b2dde0a66d | feat: spx admin images remove (#239) | ||
|
Julian Sommer
|
739cf082a9 | feat: rocky images + RHEL support (#238) | ||
|
Julian Sommer
|
20a8a39608 | feat: flip images to UEFI default (#237) | ||
|
Julian Sommer
|
2032e1518f | fix: golangci-lint config | ||
|
Julian Sommer
|
762381adfd | chore: add BootMode field | ||
|
Josh Siviour
|
7eff142c34 |
feat(e2e): bare-metal multi-node cluster support + density-aware guards (#227)
* feat(predastore): support 2-node distributed mode with RS(1,1)
Lower minimum node threshold from 3 to 2. Emit data=1,parity=1 when
exactly 2 nodes configured — effectively mirroring. 3+ nodes continue
to use RS(2,1).
* fix(ec2): route StartInstances back to original node with node-down fallback
ec2.start used a plain queue group, so NATS could route a restart request to
any daemon — including a capacity-constrained NUC that can't fit a second
instance. This caused InsufficientInstanceCapacity in the Phase 10 stop/start
test on 2-node bare-metal clusters where one node has significantly fewer
resources.
Each daemon now also subscribes to ec2.start.{nodeId} (no queue group). The
queue-group handler peeks at the instance ID, looks up LastNode from the
stopped KV, and forwards to ec2.start.{lastNode}. If the original node does
not respond within 5 seconds (down or partitioned), the handler falls back to
starting locally on whichever node picked up the request — preserving the
existing behaviour for node-failure scenarios.
The direct handler (ec2.start.{nodeId}) always starts locally and never
forwards, preventing routing loops.
* fix(ec2): fall back to local start when original node returns InsufficientInstanceCapacity
The forwarding handler was relaying all responses from the targeted node
unconditionally, including InsufficientInstanceCapacity. This meant that if
the NUC (node2) was the LastNode but had no remaining capacity, the error
propagated to the caller rather than falling back to the local node which may
have room.
Now only success and non-capacity errors are relayed; an InsufficientInstance-
Capacity response from the targeted node triggers a local start attempt, giving
any other node in the cluster a chance to absorb the instance.
* test(e2e): make VPC stop/start restart non-fatal on InsufficientInstanceCapacity
Asymmetric hardware (NUC node2) may not have capacity to restart all
VPC instances. IP-persists-while-stopped is still verified and fails
hard; restart verification is skipped with a warning on capacity errors.
* test(e2e): density-aware capacity guard across bare-metal multi-node suite
Sub-3-node bare-metal clusters with asymmetric/undersized nodes (e.g. a
4-vCPU NUC alongside a 16-vCPU box) can fit only one t3.nano on the
smallest node after the daemon's host reserve. Phases that direct or
indirectly land >1 instance on that node hit InsufficientInstanceCapacity
and previously hard-failed the entire suite.
Add is_density_skip helper and skip_test counter, and wrap every
density-sensitive run-instances / start-instances / wait-for-running call
site (phases 3, 5, 7, 10, 11) so capacity exhaustion on sub-3-node
clusters records a SKIP rather than a FAIL. On 3+ node clusters the
guard short-circuits to false, preserving existing failure semantics.
Summary block now prints Skipped count + names so degraded coverage is
visible in CI logs.
Refs: mulga-siv-72
* test(e2e): parameterise lb SSH key + user via PEER_SSH_KEY / PEER_SSH_USER
Allow run-lb-e2e.sh to run against clusters that don't use tf-user with the
tofu-cluster SSH key (e.g. bare-metal nodes which use spinifex@ + bm-ci key).
Defaults preserve existing behaviour on the tofu-cluster CI path.
Refs: mulga-siv-73
|
||
|
joshsiv-mulga
|
3037c349a1 | fix(installer): skip SPINIFEX_JOIN_ADDR validation when SPINIFEX_SKIP_FORMATION=1 | ||
|
joshsiv-mulga
|
fba2103133 |
feat(firstboot): SkipFormation flag — bm-bootstrap owns cluster formation
Add SkipFormation bool to firstboot.Config. When set, the cluster formation block (spx admin init/join) is replaced with a no-op echo, leaving formation to the provisioning controller. Wired through: autoinstall reads SPINIFEX_SKIP_FORMATION=1 from env, install.go passes it to firstboot.Config. spinifex-init.sh parses the kernel cmdline param and exports it. boot-handler.go hardcodes SPINIFEX_SKIP_FORMATION=1 so all bare-metal CI nodes skip formation — bm-bootstrap.sh always runs spx admin init/join with the correct --nodes N from nodes.json, enabling proper multi-node formation. |
||
|
Julian Sommer
|
0dc0daf116 | feat: use FIPS compliant cryptography (#225) | ||
|
joshsiv-mulga
|
7d25b04532 | fix(installer): fire install callback before reboot to clear PXE flag | ||
|
joshsiv-mulga
|
7dd3af3848 |
revert(installer): undo grub-install changes that introduced PXE boot loop
The --no-nvram flag removed the UEFI NVRAM boot entry, causing firmware to fall through to PXE on every reboot. Restore original --recheck behaviour which was confirmed working in manual testing. |
||
|
joshsiv-mulga
|
cc157e97f9 |
fix(installer): skip fs-probe and nvram in grub-install to prevent eMMC scan hang
grub-probe scans all block devices even without --recheck. On N100 mini-PCs with eMMC + NVMe, scanning the eMMC causes grub-install to hang. Also skip NVRAM writes (--no-nvram) since --removable installs to the default EFI path without needing to register a boot entry in UEFI firmware variables. |
||
|
joshsiv-mulga
|
fa5919fc14 |
fix(installer): remove --recheck from grub-install to prevent device scan hang
--recheck triggers grub to rescan all block devices. On N100 mini-PCs with eMMC + NVMe, scanning the eMMC causes grub-install to hang indefinitely. Device maps are generated fresh on each install so --recheck is redundant. |
||
|
joshsiv-mulga
|
f8d025d07e |
fix(installer): disable os-prober to prevent update-grub hang on multi-disk hardware
On N100 mini-PCs with eMMC + NVMe, update-grub's os-prober probes all block devices via the bind-mounted /dev, hanging indefinitely on the eMMC. GRUB_DISABLE_OS_PROBER=true skips os-prober in the chroot. Also apply modernize lint fixes to vm test files (atomic.Int32). |
||
|
tomnewton-mulga
|
43fa6eae59 |
feat: Updated GPU-Passthrough - AMD support, GPU images, /demo folder (#217)
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * installer grub image fixes * made gpu-passthrough more user facing * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * Fix: AMD GPU edge cases * script for running 8x instances on SMCI computer * fix: increase disk space on instances * feat: multi-GPU passthrough support for g7e.12xlarge (2× MI350X) - vm.VM: GPUPCIAddress string → GPUPCIAddresses []string for multi-GPU - instancetypes: add g7e.12xlarge (2 GPUs), GPUCountForType(), AMDmi350x model - instancetypes/generate: use GPUCountForType for GpuInfo.Count and TotalGpuMemoryInMiB - gpu/manager: Release() loops all pool entries for instanceID (not just first) - vm/lifecycle: loop over GPUPCIAddresses, emit gpu0/gpu1/... QEMU devices - daemon/handlers: claim N GPUs in loop for both RunInstances and StartStoppedInstance - daemon/vm_adapters: ReclaimByAddress loops GPUPCIAddresses; ReleaseGPU checks len > 0 - scripts/launch-gpu-fleet: fixed fleet — 6× g7e.4xlarge + 1× g7e.12xlarge * feat: per-instance-type disk size in GPU fleet (300 GB single, 600 GB dual) * fix: multi-gpu instance types * fix: kill QEMU synchronously when QMP handshake fails to free VFIO device When startQEMU succeeds but newQMPClientWithHandshake times out, the instance has not yet been inserted into the manager map so vmMgr.Delete in the rollback path is a no-op. QEMU keeps running and holds the VFIO group open. The GPU pool entry gets freed, the next Claim returns the same PCI address, and every subsequent QEMU fails with 'device or resource busy'. Fix: SIGKILL QEMU and wait for it to exit before returning the QMP error, ensuring the VFIO device is released before the caller recycles the GPU pool slot. * fix: quarantine GPU after QEMU launch failure to prevent re-assignment cascade When QEMU crashes during startup (e.g. pci_irq_handler assertion on a specific GPU), the pool Release frees that address and the next NATS request immediately re-claims the same broken device. Add MarkFailed() to gpu.Manager which sets Available=false for a PCI address after a launch failure, preventing re-assignment until the daemon restarts. Call MarkFailed for each GPU address in both RunInstances and StartStoppedInstance rollback paths after the Release succeeds. * fixed instance fleet launch script * remove known hosts and stagger provisioning * improvements to launch script * more upgrades * single instance gpu instance provisioning * fixed single instance provision * smci computer demo dashboard * fixes to launch script * updated smci dashboard_server * fixed demo dashboard * updated dashboard.py * feat: GPU Passthrough fixes to passthrough process - builds proper AMD or NVIDA ready images, generalised launch script * feat: gpu-passthrough updating image builder to include gpu-ready ubuntu images * fixing image builder process * fixed image builder to remove /tmp/ file before import to free space * fix: Updated COMMANDS.md to reflect new GPU management admin commands * Update docs, update internal launch script to work with new SGs, test tweaks * Updated Docs, fixed internal launch-gpu.sh script, test tweaks * fxing CI race due to GPU load |
||
|
Bryn Mailer
|
50eea38066 |
feat(predastore): aes 256 encryption at rest
Sibling changes in Spinifex to support changes made to Predastore in this PR. |
||
|
Josh Siviour
|
6c54a6790c |
feat(daemon): DDIL Tier 1 — daemon-local-autonomy phase 1 (#204)
* refactor(daemon): extract Describe instance handlers to service (1b-pre phase 1)
Collapse the Describe family in daemon_handlers_instance.go to thin Pattern A
wrappers over a transport-neutral InstanceServiceImpl. This brings the read
path in line with the rest of the codebase and is the prerequisite for 1b's
local HTTP API (mulga-siv-14, daemon-local-autonomy plan).
Extracted: DescribeInstances, DescribeInstanceTypes, DescribeInstanceAttribute,
DescribeStoppedInstances, DescribeTerminatedInstances. Filter helpers
(instanceMatchesFilters, matchTagKey/Value, DescribeInstancesValidFilters,
IsInstanceVisible) move to the instance package alongside the service methods.
InstanceService gains the new methods and two dep interfaces
(ResourceCapacityProvider, StoppedInstanceStore) so the daemon's ResourceManager
and JetStreamManager can plug in without leaking concrete types. NATSInstanceService
returns a clear error for the fan-out describes (DescribeInstances,
DescribeInstanceTypes) since cluster-wide aggregation lives in the gateway's
scatter-gather helpers, not a single-node NATS request.
Behaviour changes: malformed JSON on DescribeInstanceAttribute now responds
with ValidationError (matching every other Pattern A handler) instead of
ServerInternal. Test updated.
Mutators (RunInstances, ModifyInstanceAttribute, StartStoppedInstance,
TerminateStoppedInstance, sub-handlers) remain Pattern B for now; follow-up
phases of mulga-siv-14 will collapse them.
* feat(daemon): local-first instance state persistence (1a)
Make /var/lib/spinifex/state/instance-state.json the source of truth for a
node's VMs; JetStream KV becomes a best-effort cluster cache. The daemon now
survives KV unavailability without losing instance state.
- New local_state.go: atomic write (tmp+fsync+rename), schema_version gate,
missing-file = fresh install, corrupt = refuse start.
- Daemon.WriteState writes the local file under Instances.Mu, then calls
jsManager.WriteStateBestEffort with a 1s timeout. KV failures log and
continue; clients never see "KV down" during partition.
- Daemon.LoadState reads from the local file only — no KV fallback (a fresh
node has empty KV anyway, so silent fallback would mask bugs).
- restoreInstances now returns error and Start() aborts on corrupt local
state rather than silently starting empty.
- jsManager.WriteState no longer self-locks (caller holds Mu); added
WriteStateBestEffort and shared marshalInstanceState helper.
Reconnect-driven KV resync deferred to siv-16 (1c owns the callback wiring).
Bead: mulga-siv-15
Plan: docs/development/feature/daemon-local-state-persistence.md
* fix(daemon): root local state at DataDir not BaseDir
LocalStatePath was falling back to BaseDir when DataDir was empty, putting
the file at /var/lib/spinifex/spinifex/state/instance-state.json (under the
daemon's own subdir) instead of the documented
/var/lib/spinifex/state/instance-state.json (the platform-shared root).
Drop the BaseDir fallback. Empty DataDir resolves to DefaultDataDir
(/var/lib/spinifex). Tests now set DataDir alongside BaseDir on their config
fixtures.
* feat(daemon): graceful NATS disconnect/reconnect handling (1c)
Daemon now flips a `mode` field (cluster|standalone) on NATS
disconnect/reconnect via optional callback hooks added to
utils.ConnectNATS. Reconnect bumps natsRetryCount and triggers 1a's
full local-state push to KV. NATSRequest/NATSScatterGather fail fast
with ErrClusterUnavailable when the connection is down. Gateway
short-circuits NATS-bound requests with a 503 ServiceUnavailable
carrying "cluster unavailable: NATS disconnected — check daemon
/local/status" so AWS clients see the failure mode immediately
instead of hanging on per-call timeouts.
Plan: docs/development/feature/daemon-autonomy-1c-disconnect.md
* fix(gateway): SigV4 middleware short-circuits on NATS disconnect
The 1c gateway.Request() IsConnected check (gateway.go:234) was
unreachable for any authenticated path because SigV4 auth middleware
runs first and calls IAMService.LookupAccessKey over JetStream KV. With
NATS down the lookup hangs ~5s on context.DeadlineExceeded and the
gateway returns 500 InternalError. AWS SDK retries it twice for ~18s of
user-visible latency instead of the <100ms 503 promised by the 1c plan.
Mirror the IsConnected check at the top of SigV4AuthMiddleware (after
credential parse so malformed auth headers still get the right SigV4
error code, before the IAM lookup). The catch-all in gateway.Request()
stays as defence-in-depth for any future unauthed paths.
Coverage debt picked up alongside: 1c utils.NATSRequest /
NATSScatterGather fail-fast paths, daemon Mode/onNATSDisconnect/
onNATSReconnect/localStatePath/Write/LoadState branches, and 1b-pre
InstanceServiceImpl Describe* methods (partial siv-22).
Bead: mulga-siv-23
* feat(daemon): read-only /local/* HTTP API (1b)
Wires GET /local/instances, GET /local/instances/{id}, GET /local/status
onto the existing ClusterManager TLS listener so observers can read this
node's instance state and connectivity mode while NATS is unreachable.
Also fixes a pre-existing race in WriteState: json.Marshal walked shared
*vm.VM pointers from SnapshotMap while concurrent TransitionState calls
mutated VM fields under the manager lock. Marshal now runs inside
vmMgr.View; WriteLocalStateBytes / WriteStateBytesBestEffort accept the
pre-marshalled JSON and commit lock-free. TestTransitionState_Concurrent
Transitions is race-clean across 30 iterations.
Adds Daemon.stateRevision (atomic.Uint64) bumped on every successful
local-state write, surfaced as /local/status.revision so observers can
detect changes without diffing payloads.
* feat(daemon): two-phase startup — survive NATS down (1d)
Split daemon Start() into startLocal() (HTTPS, mgmt bridge, OVS plumber,
OOM, local state load — no NATS) and startCluster() (background, infinite
retry exp backoff cap 60s for NATS connect, then JetStream + services +
restoreInstances + subscribeAll + heartbeat). Daemon never process-exits
on NATS failure; instead it stays up in standalone mode and reconnects in
the background.
Drop ExecStartPre=wait-for-nats.sh from spinifex-daemon.service — local
bootstrap no longer requires NATS to be reachable.
ConnectNATSWithRetry now supports infinite retry (WithMaxWait(0)),
context cancellation (WithContext), per-attempt error callback for the
nats_retry_count operator signal on /local/status, and a configurable
backoff cap (WithMaxRetryDelay).
bead: mulga-siv-17 (closes)
* feat(daemon): Tier 1 invariant guard for cluster-scoped service init (1e-audit)
Make the DDIL §1e-audit invariant explicit: startLocal must not initialise any
NATS-dependent or JetStream-KV-backed handle. Adds assertNoClusterServicesInitialised
as a runtime guard at the end of startLocal that fails fast if a future edit
hoists a cluster-scoped service into Phase 1, plus comment blocks naming the
invariant on both startLocal and startCluster.
Code audit confirms the existing 1d split already keeps every cluster-scoped
bucket out of Phase 1; this change pins the property so it cannot regress
silently. Tier table for the 25 daemon-process KV buckets is finalised in
docs/development/feature/daemon-local-autonomy.md §1e-audit.
bead: mulga-siv-19 (closes)
* feat(daemon): NATS observability hardening (1d-obs)
Closes the residual visibility gap left by the 1d non-blocking startup:
- Extend /local/status with kv_sync_failures, last_kv_sync_at,
last_kv_sync_error so an operator polling the local API sees prolonged
divergence between the local file and JetStream KV.
- Add a KVSyncObserver interface on JetStreamManager so the daemon can
count best-effort write outcomes without coupling the manager to
daemon state.
- Escalate utils.ConnectNATSWithRetry per-attempt log to slog.Error
(rate-limited to once per minute, with disconnected_for context) once
attempt count crosses the natsRetryEscalateAttempt threshold (~30
attempts ≈ ~30 min disconnected at the 60s backoff cap), so a stuck
NATS config surfaces in error logs instead of routine warn noise.
Prometheus metrics export deferred until a system-wide telemetry plan
(likely OTel) lands; the structured /local/status fields plus the
escalated retry log cover the Tier 1 visibility gap.
* feat(daemon): opt-in SPINIFEX_REQUIRE_NATS strict-startup mode (1d-strict)
When SPINIFEX_REQUIRE_NATS=1, startCluster's first NATS connect uses a
bounded ~30s retry and the daemon aborts on timeout. Default behaviour
(env unset) keeps the Tier 1 infinite-retry path so prod / edge nodes
still survive a missing NATS at boot. Restores pre-DDIL fail-fast UX
for dev / test / single-node deploys without flipping the prod default.
mulga-siv-44.
* test(scripts): systemd-level smoke for DDIL phase 1
Bash harness exercising daemon-local-autonomy on a real `make deploy` node:
baseline cluster mode → kill spinifex-nats → assert standalone + retry
counter advance + /local/instances served → restart NATS → assert mode flip
+ KV resync stamp → SPINIFEX_REQUIRE_NATS=1 bounded-exit. Cleanup trap
restores both services. Complements existing unit coverage by validating
real systemd wiring on port 8443.
* fix(test-ddil): daemon ClusterManager binds 4432 not 8443
8443 is predastore. Daemon's HTTPS ClusterManager (where /local/* lives)
binds [nodes.node1.daemon].host = 0.0.0.0:4432 in the deployed config.
Plan doc daemon-local-autonomy.md:112 incorrectly said 8443; actual code
uses d.config.Daemon.Host.
* fix(test-ddil): correct retry-counter site + bypass NATS Wants= for strict test
Two bugs in the v1 script:
1) nats_retry_count assertion ran during disconnect — but the counter
only bumps on disconnect→reconnect *cycles* (daemon.go:321,
onNATSReconnect). Moved assertion to step 3 (restart NATS) where
the reconnect callback fires.
2) Step 4 stopped spinifex-nats then ran 'systemctl restart
spinifex-daemon' — but the daemon unit's Wants=spinifex-nats.service
re-pulled NATS up via the Wants= chain, so the strict-mode 30s
timeout never fired. systemctl mask refuses on /etc-installed
units, and --runtime mask is shadowed by /etc. Block via a runtime
drop-in with ConditionPathExists=! pointing at a sentinel file.
Verified: all 5 checks pass on a single-node make deploy box; daemon
exits 32s after restart with SPINIFEX_REQUIRE_NATS=1 + NATS blocked.
* fix(admin): render data_dir into spinifex.toml so non-root nodes write local state
Pseudo-multinode E2E (rootless tf-user, ~/nodeN data roots) hit
"mkdir /var/lib/spinifex/state: permission denied" on every RunInstances:
Config.DataDir was empty because the template never emitted a data_dir
key, and LocalStatePath fell back to DefaultDataDir (/var/lib/spinifex).
Surfaced after
|
||
|
Julian Sommer
|
29a192766c | configs | ||
|
joshsiv-mulga
|
5f8ee0b5a2 |
feat(installer): persist SPINIFEX_INSTALL_CALLBACK through firstboot reboot
In the PXE path SPINIFEX_INSTALL_CALLBACK arrives on the iPXE kernel cmdline but was never threaded into the firstboot script, so firstboot never curled the controller's /done endpoint. The flag stayed on disk, the node re-entered the install loop on the next PXE boot. Fix: read SPINIFEX_INSTALL_CALLBACK from the live-environment env in toFirstbootConfig and embed a best-effort curl into the generated firstboot script, fired once after the success marker is written. Interactive and USB-stick installs are unaffected (env var absent -> callbackBlock is empty, no curl injected). |
||
|
tomnewton-mulga
|
28ca689130 |
feat: VFIO GPU passthrough (#188)
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Julian Sommer
|
fa071795d0 | fix: use debian 13 | ||
|
tomnewton-mulga
|
85184d45f9 | Merge branch 'dev-test' into dev | ||
|
tomnewton-mulga
|
9d665f9dde | bringing local networking changes onto dev branch | ||
|
Bryn Mailer
|
3c2f57101f |
refactor: lock free write path (#179)
* fix: remove predastore filesystem backend references * fix: remove unused import * chore: bump predastore version * fix: create test buckets explicitly |
||
|
Josh Siviour
|
69404b38ce |
fix(vpcd): public-IP boot race + DHCP DORA hardening (siv-33/34/36/37/38/39) (#177)
* fix(installer): emit kernel cmdline to ttyS0 alongside tty0 (mulga-siv-33)
GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the
installed kernel printed nothing on serial after the GRUB menu. Headless
installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a
black screen forever and had no path to debug boot failures.
Append console=ttyS0,115200 — the last `console=` on the cmdline becomes
the system console, so ttyS0 wins on serial-only hardware while tty0
still receives output on local displays.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(ipam): widen vpc.dhcp.acquire retry budget to 60 s (mulga-siv-34)
ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard
error. On a cold ISO boot, NATS comes up well before vpcd's reconcile
leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological
~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's
dhcp.acquire subscription and fails permanently — d.externalIPAM stays
nil for the daemon's lifetime, daemon_handlers_instance.go silently
skips all auto-public-IP allocation, and instances launch with no
public IP, no dnat_and_snat, and no NAT event ever published.
Bump maxAttempts to 60 with 1 s spacing; expose both as package vars
so tests can shrink the budget for fast timeout-path coverage. Pure
widening of the existing retry loop — no behaviour change on the
success path, only the worst-case wallclock when vpcd never comes up.
The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd
not enabled in the ISO → veth pair vanishes on reboot → vpcd
crashloops) is fixed alongside this. A 60 s budget fits comfortably
above all observed healthy-startup times.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(setup-ovn): declare Linux WAN bridge as NetDev so networkd-managed veth resolves Bridge= on reboot (mulga-siv-33)
The veth-mode reboot persistence drop-ins added in mulga-998.b were
incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with
`Bridge=br-wan`, but never declared br-wan itself to networkd.
br-wan is created by `networking.service` (ifupdown via
`/etc/network/interfaces`, generated by `install.go writeNetworkConfig`),
not networkd. networkd's `Bridge=` directive requires the target to be a
known NetDev — when it isn't, networkd silently logs
`br-wan NetDev could not be found, ignoring assignment` and the veth
ends up with no master. vpcd's bridge-mode sanity check then fails on
every reboot.
Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as
`Kind=bridge`. networkd matches the existing kernel bridge by name+kind
(the `Failed to create netdev: File exists` log is harmless), and
`Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the
IP — networkd just needs the NetDev declaration to enslave the veth.
The write is gated to avoid clobbering operator-managed networkd config:
if any other `.netdev` under `/etc/systemd/network`,
`/run/systemd/network`, or `/usr/lib/systemd/network` already declares
the WAN bridge by name, the write is skipped with a log line pointing
at the operator file. Cleanup hook also removes the new file when
switching away from veth mode.
Verified end-to-end via QEMU repro: clean install + first boot + reboot,
veth-wan-br master persists across reboots, vpcd starts cleanly.
* fix(vpcd/dhcp): enable AF_PACKET promisc on bridge for DORA (mulga-siv-33)
Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and
reply with a unicast OFFER addressed to the synthetic per-tenant chaddr
(e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports
but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set,
so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times
out at 15s ("unable to receive an offer: context deadline exceeded").
Dev/CI didn't hit this:
- CI uses spx_wan_mode=static — no DORA via vpcd.
- QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so
their replies are L2 broadcast which the bridge always passes up.
Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/
Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET
DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket
(per-DORA, ~15s) and refcounted by the kernel — auto-clears on close.
* revert(vpcd/dhcp): drop AF_PACKET promisc wrapper, restore nclient4.New
|
||
|
Julian Sommer
|
3494ea4eb0 | fix: don't log secret key | ||
|
Julian Sommer
|
a3d1d5f7b3 | chore: delete zero-op tests (#169) | ||
|
Ben Duncan
|
94600d7b84 | OVN chassis issues, new reconsile function, reboot fixed (#165) | ||
|
Ben Duncan
|
0434a823cc | Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer | ||
|
Ben Duncan
|
40a42cbb73 | grub branding tweaks | ||
|
joshsiv-mulga
|
c01207b342 |
merge: fix/dhcp-vpcd-issues (mulga-998.b Fixes 1+1b+2+3) into feat/iso-installer
Fixes DHCP/vpcd external-path issues discovered during reboot drill on josh-banksia: - Fix 1: persist veth-wan pair via networkd .netdev/.network - Fix 1b: admin-up veth-wan-ovs via second .network unit (peer stayed LOWERLAYERDOWN) - Fix 2: loud detectBridgeMode + verifyBridgeMode sanity check - Fix 3: retrofit nat-addresses=router + network_name=external on every vpcd startup via RetrofitAllExternalLocalnetOptions Bead: mulga-siv-13 (closed). Plan: docs/development/bugs/dhcp-vpcd-issues.md. Reboot-drill evidence: smoketest 9/9, Drill 2 retrofit verified post-reboot. # Conflicts: # scripts/reset-dev-env.sh # spinifex/handlers/ec2/vpc/external_ipam.go # spinifex/services/vpcd/vpcd.go |
||
|
joshsiv-mulga
|
112738198f |
fix: rename wan_bridge to dhcp_bind_bridge (mulga-998.a)
The old wan_bridge name misled operators into pointing the field at the OVN-side bridge 'br-ext', which never sees LAN DHCP traffic. On consumer-router LANs (WAN NIC enslaved to a Linux bridge) the vpcd DHCP client's AF_PACKET bind failed silently, blocking the default install path. - Rename Go field WanBridge -> DhcpBindBridge and TOML key wan_bridge -> dhcp_bind_bridge across config, vpcd service, external IPAM, daemon pool wiring, admin settings, and spinifex.toml template. - Rewrite detectedWanBridge() -> detectedDhcpBindBridge(): the Linux-bridge branch now returns the bridge name verbatim (e.g. "br-wan") instead of synthesising "br-ext". OVS-vs-Linux probe via ovs-vsctl is no longer needed for this field. - Add checkLegacyWanBridgeKey() in cmd/service.go: vpcd fail-starts if the deprecated 'wan_bridge' TOML key or SPINIFEX_VPCD_WAN_BRIDGE env var is set. No silent alias, no auto-rewrite (per plan D3/D13). - Template carries an operator-facing comment warning never to set the field to 'br-ext'. - 7 new unit tests for the detector branches and fail-start paths. Smoke on ben-bottlebrush: spx admin init --force writes dhcp_bind_bridge matching the live br-wan Linux bridge; VM launched with --map-public-ip-on-launch reached from LAN; restoring a legacy-key backup causes vpcd to enter failed state with the deprecated-key guidance log. Plan: docs/development/bugs/dhcp-vpcd-issues.md (Fix 4) Bead: mulga-998.a (unblocker for .b / .c) |
||
|
Ben Duncan
|
6f0821d02b | Latest merge | ||
|
Josh Siviour
|
604693ea0c |
fix: wrap missed NATS/S3 dial sites with admin.DialTarget (#159)
siv-8.5 migrated daemon + awsgw but missed the sidecar services and the spx CLI. vpcd and viperblockd read cfg.NatsHost / cfg.S3Host raw from spinifex.toml; when BindIP=0.0.0.0 the TLS handshake fails SAN verification because 0.0.0.0 is not a cert SAN. Same for the spx get CLI path. Call sites: spinifex/services/viperblockd/viperblockd.go:173 (NATS) spinifex/services/viperblockd/viperblockd.go:407 (S3 s3.S3Config) spinifex/services/viperblockd/viperblockd.go:533 (S3 nbd.NBDKitConfig) spinifex/services/vpcd/vpcd.go:277 (NATS) cmd/spinifex/cmd/get.go:93 (CLI NATS) admin.DialTarget rewrites 0.0.0.0:N to 127.0.0.1:N — loopback is always a SAN and in-process sidecars share the host with NATS/predastore. |
||
|
Ben Duncan
|
1d38311e07 | main merge | ||
|
Josh Siviour
|
75fb15be2b |
feat: spx admin init/join listen-vs-advertise split + br-mgmt OVS provisioning (#158)
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway Convert br-mgmt from an externally-provisioned Linux bridge to an OVS bridge owned by scripts/setup-ovn.sh, giving every deployment type (tofu-cluster, production single-node, production multi-node) a single provisioning path. setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface / --no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone (excluded from ovn-bridge-mappings so ovn-controller ignores it) and assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot. SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master` to `ovs-vsctl add-port` / `del-port` to match the new bridge type. Add a loopback guard to wireLBAgentConfig's fallback branch: on a production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge, the fallback previously injected 127.0.0.1 into LB VM cloud-init — silently unreachable because the VM's loopback is its own. The guard leaves GatewayURL empty and the existing "LB agent gateway URL not configured" error fires instead. Regression covered in daemon_wire_lb_test.go. * fix: auto-detect primary IP for admin init --bind default Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback only, so system VMs on br-mgmt could never reach it. Auto-detect the default-route source IPv4 instead, which keeps TLS cert SANs specific (still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1 explicitly to preserve loopback-only behavior on laptops. Closes mulga-siv-7. * test: refresh wire lb loopback-bind comment for siv-7 * fix: dev-install scripts should let admin init auto-detect --bind Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with no listener. Drop the explicit --bind so auto-detect picks the primary IP and the siv-6 mgmtRouteVia path routes LB traffic to a real listener. * fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template PR #144 (nats TLS) made every service listen on --bind's specific IP. For awsgw and predastore this broke localhost-reachability that the UI proxy (spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on — latent since #144, surfaced by siv-7's auto-detect default on dev installs (502 from the UI). Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host daemon → predastore) keep working, LAN clients still reach them via the LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}} because peer-advertised values must not resolve to 0.0.0.0/loopback. predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1 for local-node consumers, so existing callers see a usable connect address. * revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8) Clears the deck before layering the listen-vs-advertise split (mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node default and introduces an explicit --advertise flag for off-host dial targets; the reverted approach (auto-detect a specific IP at init, hardcode 0.0.0.0 in the template, guard against loopback in wireLBAgentConfig) becomes redundant once that lands. Reverts: - |
||
|
tomnewton-mulga
|
7cc90f554c | daemon race fix | ||
|
Ben Duncan
|
93a6704944 | Merge remote-tracking branch 'origin/main' into feat/iso-installer | ||
|
Ben Duncan
|
3d1f8340be | ISO tweaks, email prompt on installer, improved network detection | ||
|
Ben Duncan
|
27d7fdd12b | Latest changes re ISO and binary install path | ||
|
tomnewton-mulga
|
6736722533 | Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer | ||
|
tomnewton-mulga
|
9d51800f58 | fixed duplicate IP bug between physical and VM nodes | ||
|
Ben Duncan
|
a209805ffc | Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer | ||
|
Ben Duncan
|
b591c04880 | firstboot tweaks | ||
|
tomnewton-mulga
|
ad3762bdcc | fixing duplicate IP issue |