Commit Graph

484 Commits

Author SHA1 Message Date
Ben Duncan 946429d792 feat: Phase 8c — daemon auto-assign public IP on RunInstances
Wire public IP auto-assignment end-to-end:
- RunInstances checks MapPublicIpOnLaunch on subnet, allocates external
  IP from ExternalIPAM, updates ENI record, publishes vpc.add-nat for
  dnat_and_snat rule, returns PublicIpAddress in response
- TerminateInstances releases public IP back to pool, publishes
  vpc.delete-nat to remove NAT rule before ENI deletion
- DescribeInstances populates PublicIpAddress from VM struct

Add ExternalIPAM initialization in daemon startup from cluster
NetworkConfig. Add GetSubnet and UpdateENIPublicIP helpers. Add
PublicIP/PublicIPPool fields to vm.VM struct. Fix network auto-detect
to recognize br-external as WAN interface.
2026-03-20 06:41:32 +00:00
Ben Duncan a554fe4ecf docs: rewrite single + multi-node install guides
Single-node:
- Add auto-detected network topology + external networking section
- Replace hostfwd SSH with VPC overlay private IP SSH
- Add setup-ovn.sh step with --dhcp flag
- Fix undefined variables ($SPINIFEX_AMI, $SPINIFEX_SUBNET)
- Use bash variable capture for VPC/subnet/AMI IDs
- Add three-tier external networking table
- Fix step headings (## → ###)

Multi-node:
- Move CA certificate trust to Step 5 (before starting services)
- Add --external-bridge --dhcp to all OVN setup commands
- Add AMI import step (was missing entirely)
- Add VPC + subnet creation step with variable capture
- Add SSH via private IP step
- Fix sudo inconsistency (all formation commands use sudo)
- Add external networking section with pool upgrade path
- Add DHCP troubleshooting section
- Add port requirements to overview
2026-03-20 05:58:38 +00:00
Ben Duncan 2af3bf5287 feat: auto-detect network topology + DHCP gateway in admin init
spx admin init now works with zero external networking flags:
- Auto-detects WAN interface, gateway, prefix from ip route
- Auto-detects single-NIC vs multi-NIC mode
- Defaults to nat mode with DHCP for gateway IP
- Prints detected topology with interface roles

setup-ovn.sh gains --dhcp flag: obtains a DHCP lease on the
macvlan/bridge interface for the OVN gateway SNAT address, then
writes gateway_ip into spinifex.toml. Tested on VM: macvlan gets
separate IP from router DHCP, host keeps its original IP.

Three tiers of external networking:
- Zero flags: auto nat+DHCP (VMs get outbound internet)
- --external-pool=start-end: pool mode (VMs get public IPs)
- --no-external: overlay only (no internet for VMs)
2026-03-20 05:45:30 +00:00
Ben Duncan 1f38213d4b feat: add external networking flags to spx admin init
Add --external-mode, --external-iface, --single-nic, --external-pool,
--external-gateway, --external-prefix-len flags to admin init. Config
is now generated with the correct [network] section and per-node vpcd
external_interface/single_nic — no manual spinifex.toml editing needed.

Single-node single-NIC:
  spx admin init --external-mode=pool --external-iface=eth0 --single-nic \
    --external-pool=192.168.1.150-192.168.1.250 --external-gateway=192.168.1.1

Single-node two-NIC:
  spx admin init --external-mode=pool --external-iface=eth1 \
    --external-pool=192.168.1.150-192.168.1.250 --external-gateway=192.168.1.1

Next steps output now shows the correct setup-ovn.sh command with
matching flags.
2026-03-20 05:24:06 +00:00
Ben Duncan 1ee7012ba9 feat: Phase 7 — single-NIC macvlan bridge strategy
Add --single-nic flag to setup-ovn.sh. When used with --external-bridge,
creates a macvlan sub-interface (spx-ext-{iface}) in bridge mode off the
host's NIC instead of adding the NIC directly to br-external. The host
keeps its IP — no migration, SSH-safe.

Strategy selection:
- Dedicated NIC: --external-iface=eth1 (add directly)
- Single NIC:    --external-iface=eth0 --single-nic (macvlan)
- IP migration:  --external-iface=eth0 (legacy, no --single-nic)

Add single_nic bool field to VPCDConfig for per-node config. Different
nodes in a cluster can use different strategies. Unit test verifies
single_nic parsing alongside external_interface.
2026-03-20 05:15:35 +00:00
Ben Duncan b6e99947e1 feat: Phase 5 — unit tests for ACL, security groups, public subnet, EIP
Add 18 unit tests covering:
- ACL match expression compiler (8 tests): TCP port/range, ICMP, UDP
  range, all-protocols, SG-ref source, egress, no-source filter
- Security group Port Group lifecycle (3 tests): create, delete, update
  with ACL generation via NATS events + MockOVNClient
- ModifySubnetAttribute + MapPublicIpOnLaunch (2 tests): default false,
  toggle to true via ModifySubnetAttribute
- Elastic IP lifecycle (5 tests): allocate, allocate from specific pool,
  release, release-while-associated error, describe
2026-03-20 04:35:52 +00:00
Ben Duncan 5481ef4361 feat: Phase 4 — security groups (OVN ACLs + Port Groups)
Add security group CRUD handlers: CreateSecurityGroup, DeleteSecurityGroup,
DescribeSecurityGroups, AuthorizeSecurityGroupIngress/Egress,
RevokeSecurityGroupIngress/Egress. SG records stored in NATS KV.

Add ACL match expression compiler (acl.go) translating AWS SG rules to
OVN ACL match strings for TCP/UDP port/ranges, ICMP, all-protocols, CIDR
filters, and intra-SG references.

Add vpcd security.go handlers for OVN Port Group lifecycle and ACL
management via vpc.create-sg/delete-sg/update-sg NATS topics. Extend
OVNClient interface with CreatePortGroup, DeletePortGroup,
SetPortGroupPorts, AddACL, ClearACLs. Add PortGroup and ACL to nbdb
model.
2026-03-20 04:24:49 +00:00
Ben Duncan ee06105c56 feat: Phase 3 — Elastic IPs (AllocateAddress, AssociateAddress)
Add EIP service with AllocateAddress, ReleaseAddress, AssociateAddress,
DisassociateAddress, and DescribeAddresses. Uses ExternalIPAM for IP
allocation from admin-defined pools. Publishes vpc.add-nat/delete-nat
events for OVN dnat_and_snat rules.

EIP records stored in spinifex-vpc-elastic-ips NATS KV bucket with
account-scoped keys. AssociateAddress replaces auto-assigned public IPs
when present.
2026-03-20 04:21:46 +00:00
Ben Duncan 150bf246a4 feat: Phase 2 — public subnets + auto-assign public IPs
Add MapPublicIpOnLaunch field to SubnetRecord with ModifySubnetAttribute
API handler. Add PublicIpAddress/PublicIpPool fields to ENIRecord for
tracking auto-assigned public IPs.

Add vpc.add-nat / vpc.delete-nat NATS topics and handlers in vpcd
topology for creating distributed dnat_and_snat NAT rules (with
LogicalPort + ExternalMAC for chassis-local NAT processing).

Wire ModifySubnetAttribute through gateway → daemon → VPC service.
2026-03-20 04:19:28 +00:00
Ben Duncan c1cdfc07c3 feat: Phase 1 completion — external IPAM, br-external setup, config template
Add external IPAM module (CAS-based IP allocation from admin-defined
pools in NATS KV). Supports multi-pool with region/AZ scoping, pool
fallback logic, gateway IP reservation, and concurrent allocation via
CAS retries. 12 unit tests covering allocation, release, exhaustion,
CAS conflict, multi-pool, fallback, validation, and idempotent init.

Add --external-bridge flag to setup-ovn.sh: creates OVS bridge
br-external, adds WAN NIC, migrates host IP to bridge, and sets
ovn-bridge-mappings=external:br-external for OVN localnet ports.

Add commented [network] section and external_interface to spinifex.toml
config template for discoverability.
2026-03-20 03:59:47 +00:00
Ben Duncan c6af189638 Merge commit '02931567ac207bd074d8b2efda6e523ab36540d9' into vpc-public-subnet 2026-03-20 03:47:50 +00:00
Julian Sommer 02931567ac release: update go.mod for v0.4.2 v0.4.2 2026-03-20 13:09:48 +10:00
dependabot[bot] c4890c80d3 build(deps): bump fast-xml-parser (#83) 2026-03-20 12:48:53 +11:00
Julian Sommer d2d3e52522 feat: migrate to golangci-lint (#82) 2026-03-20 12:14:42 +11:00
Julian Sommer 42cb6dbed0 fix: add better error message 2026-03-20 11:12:32 +10:00
Julian Sommer 6cdeea3aa8 chore: more tests 2026-03-20 09:27:21 +10:00
Julian Sommer 4884b68b51 feat: version bootstrap file 2026-03-20 09:12:42 +10:00
Julian Sommer 726528bd9d feat: version jetstream buckets 2026-03-20 09:07:40 +10:00
Julian Sommer 593920ce7d feat: E2E test redistribution and dev-setup cleanup
Redistribute E2E tests between single-node and multi-node suites, slim
down pseudo multi-node suite, and remove directory creation from
dev-setup.sh so spx admin init is the single source of truth for
~/spinifex/ setup.
2026-03-20 08:24:15 +10:00
Ben Duncan 5e30d12f76 feat: Phase 1 — external network config + IGW gateway router fix
Add cluster-wide [network] config with external_mode and multi-pool
support (ExternalPool with region/AZ scoping). Per-node vpcd gets
external_interface for WAN NIC name.

Fix handleIGWAttach to use real external IPs from the configured pool
instead of link-local 169.254.0.1/30. Gateway router port now gets
the pool's gateway IP with correct prefix length, SNAT uses the real
external IP, and the default route points to the actual WAN gateway.

Includes pool selection logic (AZ-scoped → region-scoped → unscoped
fallback) and unit tests for config parsing, IGW with external pool,
explicit gateway IP, and pool selection.
2026-03-19 11:34:30 +00:00
Julian Sommer ea8700df99 fix: increase sleep time v0.4.1 2026-03-19 15:02:50 +10:00
Julian Sommer 857cbd75ac fix: use spx service stop instead of shell PID kill in stop-dev.sh 2026-03-19 14:50:50 +10:00
Julian Sommer 0fa38f7986 fix: write credentials to user even when running as sudo 2026-03-19 14:05:48 +10:00
Julian Sommer 1a4d590bf3 fix: nbd log level 2026-03-19 13:45:27 +10:00
Julian Sommer b7678ab7c5 feat: fix silent failures across spinifex (P2 audit H-25 through H-43)
- PID file failures now prevent service start instead of logging and continuing
- Status() methods use PID-based health checks instead of returning empty string
- NewResourceManager returns error instead of silently defaulting to 8GB
- Crash recovery re-allocates resources before relaunch
- restoreInstances stops after failed resource allocation
- Cluster manager bind errors propagated to Start()
- NATS disconnect/reconnect logged at Warn level
- VPC CIDR fallback logs warning about overbroad rule
- vpcd respond() handles json.Marshal failure
- Network cleanup failures logged instead of silently discarded
- fmt.Println replaced with slog.Debug in NATS service
- Signature mismatch no longer logs expected signature
- extractDiskImagePath uses errors.Is for EOF
- validateDiskImagePath checks file command error
- os.Stat uncommented in admin image import
- net.SplitHostPort errors checked
- Gateway response write errors logged
- vpcd OVN NB address uses leader IP on non-primary nodes
- Release workflow parallelized with native arm64 runner
2026-03-19 13:37:41 +10:00
Julian Sommer 11730be685 fix: locate release assets by filename instead of hardcoded path 2026-03-19 12:13:28 +10:00
Julian Sommer a4bc8bee74 fix: pending instance terminate (#80) 2026-03-19 12:47:00 +11:00
Julian Sommer fafc104c30 feat: add release verification job with draft-then-promote flow 2026-03-19 10:52:39 +10:00
dependabot[bot] efa6257850 build(deps): bump google.golang.org/grpc from 1.79.1 to 1.79.3 (#79) 2026-03-19 11:35:06 +11:00
Julian Sommer 7e46f0012b chore: bump more action versions 2026-03-19 10:23:52 +10:00
Julian Sommer 58e6817efe chore: bump docker build action ver 2026-03-19 10:19:02 +10:00
Julian Sommer 71ec2b7ee5 release: update go.mod for v0.4.0 v0.4.0 2026-03-19 08:28:16 +10:00
Julian Sommer 9c430eea07 fix: release workflow 2026-03-19 08:17:30 +10:00
Julian Sommer 1be6c69e51 format 2026-03-19 08:03:23 +10:00
Ben Duncan c523f78403 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-18 05:08:30 +00:00
Ben Duncan 2918a02fac correct spx path 2026-03-18 05:08:15 +00:00
Julian Sommer 8e3210f023 fix 2026-03-18 14:36:12 +10:00
Julian Sommer 6ba9d3dbf0 remove trailing slash 2026-03-18 14:24:52 +10:00
Ben Duncan 27cdfc4653 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-18 04:14:22 +00:00
Ben Duncan adadb5ef4c Fixed noninteractive error on apt get 2026-03-18 04:13:53 +00:00
Julian Sommer d0fc2a2120 last fix 2026-03-18 13:40:27 +10:00
Julian Sommer a09a8efe3c build frontend with new branding 2026-03-18 13:38:47 +10:00
Julian Sommer 7d43d0f782 hopefully last rename 2026-03-18 13:24:45 +10:00
Julian Sommer ba0d8b67ff feat: e2e pipeline speedup (#78) 2026-03-18 13:50:45 +11:00
Ben Duncan 454f381826 Moved to GNU Affero General Public License v3 2026-03-18 02:04:41 +00:00
Ben Duncan b95ff42a2f feat: rename product from Hive to Spinifex
Trademark conflict — bulk rename across entire codebase:
- Go module: github.com/mulgadc/hive → github.com/mulgadc/spinifex
- Binary: hive → spx
- Directories: hive/ → spinifex/, hiveui → spinifexui
- Env vars: HIVE_* → SPINIFEX_*
- Config: hive.toml → spinifex.toml
- Systemd: hive-* → spinifex-*
- NATS: hive-workers → spinifex-workers
- All paths, docs, scripts, CI workflows updated
2026-03-18 01:42:53 +00:00
Ben Duncan 2064c338b3 chore: gofmt formatting after rename 2026-03-18 01:23:11 +00:00
Ben Duncan 4f00cf9e4c refactor: update Dockerfile and build artifacts for spinifex rename
- Dockerfile.distro: update all paths from hive → spinifex
- Binary output name: spx
- Logrotate: logrotate-spinifex
2026-03-18 01:20:27 +00:00
Ben Duncan 48012a4058 refactor: rename all documentation from hive to spinifex
- Rename .claude/rules/hive.md → spinifex.md
- Update CLAUDE.md component description
- Rename doc directories: hive-admin-cli → spinifex-admin-cli, etc.
- Rename doc files: hive-cluster-stats, hive-code-consolidation, etc.
- Bulk update all markdown files across docs/ tree
- Update internal docs: CLUSTER-INSTALL, TESTING, DEVNOTES, BRANCHES
2026-03-18 01:19:05 +00:00
Ben Duncan 55186d95f2 refactor: rename CI/CD workflows from hive to spinifex
- Rename hive.yml → spinifex.yml
- Update workflow names, path triggers, working directories
- Update release.yml tarball references
- Update e2e.yml test paths
- Update ui.yml frontend paths
2026-03-18 01:09:02 +00:00