Compare commits

...

383 Commits

Author SHA1 Message Date
Julian Sommer b7b639084b chore: use ts config files 2026-04-14 14:28:00 +10:00
Julian Sommer 7dcad2ba1d chore: remove unused e2e 2026-04-14 13:59:29 +10:00
Julian Sommer 7cf0783122 fix: set dev tags as prerelease 2026-04-14 13:29:05 +10:00
Josh Siviour 13eae4d40b fix(elbv2): tag-based LB AMI lookup with hard error on miss (#137)
Replace fragile image-name substring match (-lb-/-alb-) with a
DescribeImages tag filter on spinifex:managed-by=elbv2. Remove the
silent fallback that picks the first image in the store when no LB
image exists — CreateLoadBalancer now returns ServerInternal with a
remediation hint instead of persisting a record in StateFailed.

- daemon: tag filter + error return from the resolver closure
- service_impl: resolver signature is func() (string, error); caches
  only successful results so the next call retries after an error
- CreateLoadBalancer gate now requires launcher + networking; AMI is
  resolved inside the gate and failures short-circuit before persist
- admin images import: add --name+--file combo (catalog metadata with
  a local file source) and a --tag key=value flag for explicit tagging
  of file-only imports
- build-system-image.sh + lb.conf: propagate SYSTEM_TAG via --tag so
  locally built system AMIs get the right tag even when the catalog
  is bypassed

Closes mulga-969
2026-04-14 12:52:28 +10:00
Julian Sommer b32111102d fix: README.md 2026-04-14 12:06:33 +10:00
Ben Duncan 71d0f31997 Merge branch 'main' of github.com:mulgadc/spinifex 2026-04-14 11:45:46 +10:00
Ben Duncan f99eb42ca7 Doc tweaks and README for release 2026-04-14 11:45:20 +10:00
Julian Sommer 56cdc158a6 chore: bump viperblock 2026-04-14 11:17:48 +10:00
Julian Sommer a4ae813e38 fix: recursive chmod on shared data dirs so existing image files are group-readable
chmod 0770 without -R only affected the top directory; pre-existing files
(e.g. images imported while owned by another user at mode 0600) stayed
unreadable to the spinifex group, breaking image import on upgrade paths.
2026-04-14 11:05:01 +10:00
Julian Sommer e0118975d1 fix: setup.sh 2026-04-14 10:17:01 +10:00
Julian Sommer 36e3481ca3 fix: enable spinifex.target so services autostart on boot 2026-04-14 10:14:09 +10:00
Julian Sommer 515c313e07 fix: setup.sh newgrp 2026-04-14 10:07:59 +10:00
Julian Sommer 0547bdd1fe feat: loadbalancer frontend (#136) 2026-04-14 09:24:14 +10:00
Julian Sommer e75953f5e2 chore: build ui with latest deps 2026-04-14 09:07:59 +10:00
dependabot[bot] e5a6f3ea60 chore: bump the npm-dependencies group (#134) 2026-04-14 08:59:32 +10:00
dependabot[bot] 36da673ecd chore: bump the github-actions group with 2 updates (#133) 2026-04-14 08:59:01 +10:00
Ben Duncan 96b5ad5740 Minor doc tweaks and permissions 2026-04-13 15:23:18 +10:00
Ben Duncan ebd0a64fe5 chore: bump predastore to v1.1.5 (#135)
Picks up multipart upload wedge fixes from predastore v1.1.5:
- 76fdbf1 fix: pool cleanup must not reap connections with in-flight streams (Bug A)
- 7df9645 fix: add mutex to bufferedWALFile to prevent bufio race (Bug B)
- 9dcd3dd fix: drain shard body before taking wal.mu in handlePUTShard (Bug C)
- 43e20f6 fix: widen QUIC flow-control windows to mitigate multipart upload deadlock

plus regression tests for all three bugs (predastore PR #8).
2026-04-13 13:00:25 +10:00
Josh Siviour db9e93351e fix: nginx-alb + route table NAT GW SNAT event bugs
Bundle of ALB and NAT GW fixes surfaced by the nginx-alb terraform example.

- fix: expose ALB ENI private IPs in DescribeLoadBalancers (mulga-928)
- fix: attach every ENI to multi-subnet ALB VMs (mulga-929)
- fix: nginx-alb example workers reach apt via NAT Gateway (mulga-930)
- fix: accept every ALB attribute key terraform sends (mulga-931)
- fix: publish NAT GW SNAT events on route table (dis)association (mulga-932)
- fix: publish NAT GW SNAT events on route table replace (mulga-933)
- refactor: extract multi-ENI NIC setup into testable helpers

E2E: https://github.com/mulgadc/spinifex/actions/runs/24318441048 (all green)
2026-04-13 09:29:47 +10:00
Josh Siviour c8ca746bf0 feat: ELBv2 attributes API + DescribeTags (#128)
* feat: add attributes map to LB and TG records with defaults

Replace CrossZoneEnabled bool with Attributes map[string]string on
LoadBalancerRecord and TargetGroupRecord. Add DefaultLoadBalancerAttributes()
and DefaultTargetGroupAttributes() functions. ALBs seed cross-zone enabled
at create time.

* feat: add 4 attributes methods to ELBv2Service interface

* feat: implement ELBv2 attributes service methods and NATS wrappers

Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
to service_impl.go with proper ARN validation and default merging.

Add corresponding NATS request wrappers in service_nats.go.

* feat: add gateway handlers for ELBv2 attributes API endpoints

Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
gateway handlers with input validation and action map registration.

* feat: add daemon handlers & NATS subscriptions for elbv2 attributes

Add 4 handler functions in daemon_handlers_elbv2.go and 4 NATS
subscriptions in daemon.go for ModifyTargetGroupAttributes,
DescribeTargetGroupAttributes, ModifyLoadBalancerAttributes, and
DescribeLoadBalancerAttributes.

* fix: add missing accountID authorization check to ELBv2 attribute handlers

* fix: guard nil attributes and log skips in ELBv2 Modify*Attributes

ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes silently
dropped attributes with nil Key or Value, and would panic on a nil
slice element when dereferencing attr.Key. Add an explicit nil-element
guard and log a warning whenever an attribute is skipped, so callers
and operators can see divergence between the submitted payload and the
persisted state.

* fix: sort ELBv2 attribute responses by key for deterministic output

DescribeTargetGroupAttributes and DescribeLoadBalancerAttributes iterated
a map and appended to a slice, producing random attribute order on each
call. Sort keys before building the response so Terraform plans stay
stable and snapshot tests can rely on consistent ordering.

Closes mulga-910

* feat: add ELBv2 DescribeTags handler (mulga-919)

Required by the Terraform AWS provider's post-create refresh on
aws_lb, aws_lb_target_group, and aws_lb_listener. Without it, the
gateway returns InvalidAction and apply fails after the resources
are already created.

Dispatches by ARN resource segment (loadbalancer/targetgroup/listener)
to the existing record stores. Enforces account scoping via per-type
not-found errors to avoid cross-account existence leaks. Listeners
return an empty Tags slice since ListenerRecord doesn't store tags yet
— that's the exact case the provider hits on every untagged listener
refresh. Output is sorted by key for deterministic responses.

* refactor: parameterize DefaultLoadBalancerAttributes by lbType (mulga-915)

Cross-zone default was split across three places: CreateLoadBalancer seeded
{load_balancing.cross_zone.enabled: true} for ALBs, DefaultLoadBalancerAttributes
returned false unconditionally, and tests asserted both sides. Make
DefaultLoadBalancerAttributes(lbType) the single source of truth: ALBs get
"true", NLBs (and any other type) get "false". CreateLoadBalancer no longer
seeds attributes — DescribeLoadBalancerAttributes derives per-type defaults
from lb.Type on read. Updated TestCreateLoadBalancer_CrossZoneAttributes to
assert via the Describe handler instead of poking stored record state.

* perf: direct KV lookup for ELBv2 *ByArn finders (mulga-913)

GetLoadBalancerByArn and GetTargetGroupByArn used to call ListLoadBalancers /
ListTargetGroups and linearly scan, doing one NATS round-trip per record in
the bucket. Terraform hits Describe*Attributes on every plan/refresh — a
20-ALB / 40-TG workspace was doing ~60 calls × ~60 records = ~3600 KV gets
per refresh where ~60 would suffice. The LB/TG ARN's final path segment IS
the KV key used by GetLoadBalancer(lbID)/GetTargetGroup(tgID), so parse the
short ID from the ARN and do an O(1) lookup, verifying the record's stored
ARN matches as defence-in-depth against any stale-key weirdness.

* fix: reject ELBv2 ModifyAttributes calls where every entry is invalid (mulga-911)

When every submitted attribute was rejected by the nil-guard (nil element,
nil Key, or nil Value), Modify{TargetGroup,LoadBalancer}Attributes still
called PutTargetGroup/PutLoadBalancer with an unchanged record and returned
a successful ModifyOutput with empty Attributes. Only a slog.Warn marked
the skip — the caller saw 200 OK and moved on. Return
ErrorInvalidParameterValue when len(input.Attributes) > 0 and every entry
was rejected, so the caller learns that nothing landed.

* fix: reject unknown ELBv2 ModifyAttributes keys with ValidationError (mulga-912)

ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes had no whitelist
check — any caller-supplied key (typo like 'stickness.enabled', deprecated
keys, or cross-product keys sent to the wrong resource) was persisted verbatim
and merrily echoed back by Describe. AWS rejects unknown keys with
ValidationError; match that so Terraform surfaces the typo at plan time.
Validate each key against DefaultTargetGroupAttributes() /
DefaultLoadBalancerAttributes(lb.Type) and return ErrorValidationError on the
first unknown key, logging both the arn and the offending key name.

* perf: skip no-op Put in ELBv2 ModifyAttributes (mulga-914)

The unconditional PutTargetGroup/PutLoadBalancer fired on every Modify call,
even when the caller sent an empty attribute list or when every submitted
value already matched the stored value. Terraform re-applies the same
attribute set on every drift check, so the steady-state path was doing a
pointless NATS/KV round-trip per call and unnecessarily expanding the
read-modify-write race window.

Track a dirty flag per Modify call: compare each submitted key's value
against tg.Attributes/lb.Attributes and only write to the map (and allocate
the map lazily) when it actually differs. Skip the Put entirely if nothing
changed. Added regression tests that capture the KV entry revision before
and after a no-op modify to prove the write was skipped.

* test: assert exact ModifyAttributes echoed key/value pairs (mulga-917)

Both round-trip tests used assert.Len(t, modOut.Attributes, 2) and never
checked that the returned Attributes actually contained the submitted
keys/values. A regression that pointed at the wrong source, dropped Value,
or returned a copy of defaults would have passed the length check
unchallenged. Build a modMap from the echoed response and assert each
expected Key/Value pair explicitly.

* test: cover sequential ModifyAttributes merge and same-key overwrite (mulga-916)

All existing round-trip tests did one Modify then Describe. No test covered:
Modify {A}, Modify {B}, Describe, assert BOTH keys present; or Modify
overwriting the same key on a second call. A future refactor that replaced
tg.Attributes instead of mutating in place would pass every existing test
but silently wipe previous attributes on each Modify — the most likely
real-world ALB/TG bug. Add sequential-merge tests for both TG and LB
handlers that exercise multi-key merge, same-key overwrite, and
cross-survival of unrelated keys.
2026-04-10 21:31:43 +10:00
Josh Siviour 67bdc9d4af fix: nginx-alb terraform example uses private IPs for instances (#131)
The ALB reaches backends over the private VPC network, so the Nginx
instances no longer need public IPs. Drop associate_public_ip_address
and map_public_ip_on_launch, and swap the instance public-IP outputs
for private-IP ones.

The *.elb.spinifex.local DNS name does not resolve from the operator's
host, so document an AWS CLI one-liner (describe-load-balancers with
an AvailabilityZones[].LoadBalancerAddresses[].IpAddress query) that
retrieves the ALB's public IP for curl/browser testing.
2026-04-10 18:54:12 +10:00
Josh Siviour 8d2830481d update alb terraform example (#130) 2026-04-10 16:52:52 +10:00
Josh Siviour 6238988a8f feat: add Terraform nginx-alb example (#129)
Deploys two Nginx EC2 instances behind an Application Load Balancer
on Spinifex, exercising VPC, public subnets, IGW, security group,
key pair + cloud-init, ALB, target group, and HTTP listener end-to-end.

Also gitignore generated *.pem files from tofu local_file outputs.
2026-04-10 15:56:50 +10:00
Julian Sommer 4365fb0958 feat: update guide 2026-04-10 15:05:46 +10:00
Julian Sommer 48c90b795a feat: migration framework (#127) 2026-04-10 14:34:15 +10:00
Julian Sommer 18684d5c8b fix: remove InsecureSkipVerify and normalize 0.0.0.0 connect addresses (#125) 2026-04-10 13:30:13 +10:00
Julian Sommer 708bb1d340 feat: enforce nats authentication (again) (#126) 2026-04-10 12:18:46 +10:00
Julian Sommer 893fbbc243 fix: remediate trivial security audit findings (#124) 2026-04-10 11:42:43 +10:00
Julian Sommer 7723ad4a89 fix: enable perfsprint 2026-04-10 11:36:51 +10:00
Josh Siviour ab95566795 feat: initial ELBv2 implementation (#122)
* feat: add ALB target health reporting via HAProxy stats

Agent-side: periodically query HAProxy stats socket and publish
target health status to NATS (elbv2.alb.<id>.health topic).

Control-plane: healthChecker subscribes to health reports from all
ALB agents and updates target health state in the KV store. Wire
into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup.

E2E: remove health check skip — targets must now become healthy.

* feat: allocate public IP for internet-facing ALB VMs in daemon

Add Scheme field to SystemInstanceInput so the daemon knows whether
an ALB is internet-facing or internal. For internet-facing, allocate
a public IP via externalIPAM and publish vpc.add-nat for the OVN
dnat_and_snat rule. Internal ALBs skip public IP assignment entirely.

* fix: release auto-assigned public IPs on DeleteNetworkInterface

DeleteNetworkInterface previously leaked public IPs and NAT rules.
Now checks if the ENI has a public IP, verifies it's not EIP-owned,
and releases via externalIPAM + publishes vpc.delete-nat. Daemon
injects externalIPAM and EIP KV into VPC service at init time.

Tests cover: auto-assigned IP release with NAT event, EIP-owned IP
skipped, and no-op when externalIPAM is not configured.

* feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770)

In CreateLoadBalancer, when DevNetworking is false (pool mode), require
2+ subnets in different AZs for internet-facing scheme. Dev mode and
internal scheme skip the validation.

* feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771)

Add 9 unit tests covering scheme-dependent ALB behavior:
- DNS name uses internal- prefix for internal scheme (AWS-compatible)
- Internet-facing ALBs include public IP in AvailabilityZones response
- Internal ALBs have no public IP in AZ response
- Launcher receives correct scheme, delete cleans up VM + ENIs

Also implements the code changes these tests exercise:
- AvailZoneInfo.PublicIP field populated from launcher output
- lbRecordToSDK populates LoadBalancerAddresses for internet-facing

* feat: add custom AMI for ALB to images list

* feat: add internet-facing ALB data plane E2E test (mulga-772)

New E2E script that validates internet-facing ALBs get public IPs and are
reachable from a peer cluster node (external validation). Skips in dev mode.

* fix: set ENDPOINT to runner node IP in ALB E2E workflow

* fix: correct paths filter to reference alb-e2e.yml

* fix: alb ami name and workflow step to import the image

* fix: spx path

* fix: remove ALB AMI import from workflow and added to bootstrap.sh

* fix: use two subnets in different AZs for internet-facing ALB E2E

Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have
subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each.

* refactor: remove multi-AZ enforcement for internet-facing ALBs

Team decision: ALBs must work with single-AZ and single-node clusters.
Removes the multi-AZ validation, pool mode helper, associated tests,
and error constant. E2E script reverted to single subnet.

* fix: use correct account ID for ALB ENI public IP assignment

The ENI is created under the caller's account but LaunchSystemInstance
was looking it up with GlobalAccountID (system account), causing the
UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's
account ID via SystemInstanceInput.AccountID.

* fix: add debug logging for ALB public IP assignment failure

* fix: expand debug logging to include peer node and more keywords

* fix: lazy system AMI discovery for ALB VMs

The daemon was caching the system AMI at startup, but the ALB image is
imported after services start (bootstrap step 9). Replace SetSystemAMI
with SetSystemAMIFunc so the AMI is resolved at request time when
CreateLoadBalancer actually needs it.

* fix: resolve ALB VM instance type dynamically from resource manager

The instance type was hardcoded to t3.nano which may not exist on all
clusters. Now the daemon provides a resolver that picks the smallest
available instance type from the resource manager.

* push interim changes

* refactor: rewrite ALB agent from NATS client to HTTP server

Replace NATS subscriptions with an HTTP server on :8405 exposing
POST /config, GET /ping, and GET /health. The daemon will push
config and poll health over HTTP through the VPC network, removing
the agent's direct NATS dependency.

- agent.go: net/http server replaces nats.Conn/Subscriptions
- health.go: remove push-based reportHealth; /health handler
  calls queryHAProxyStats synchronously on each request
- cmd/alb-agent: --nats flag replaced with --listen
- tests: rewritten with httptest instead of NATS test server
- health_checker_test: inline topic strings (HealthTopic removed)

* refactor: replace NATS with HTTP client for daemon-to-ALB agent comms

Daemon now pushes config via POST and polls ping via GET to the ALB
agent's HTTP server on the VPC IP, eliminating the agent's NATS
dependency from the daemon side.

- service_impl: pushConfig uses http.Post, waitForAgentReady uses
  http.Get, both target http://<vpcIP>:8405
- albVMUserData: drop natsURL param (agent no longer needs NATS)
- types: add VPCIP field to LoadBalancerRecord
- daemon: remove SetNATSURL call and 10.0.2.2 translation block
- arn_test: update user data tests for new signature

* refactor: rewrite ALB health checker from NATS subscription to HTTP polling

Health checker now polls each active ALB's GET /health endpoint on a
10s ticker instead of subscribing to a NATS wildcard topic. Removes
the NATS connection dependency from the health checker.

- health_checker: replace nc/sub with httpClient + stopCh polling loop
- newHealthChecker: drop nc param, add agentURLFn for testability
- service_impl: update newHealthChecker call site
- tests: rewrite with httptest, test pollAll/pollOne paths

* refactor: remove NATS wiring from ALB image and make hostfwd dev-only

- alb-setup.sh: remove --nats flag, use --listen=:8405; remove
  placeholder HAProxy frontend on 8405 (agent owns the port now);
  update description and comments
- service_impl: HostfwdPorts (80/443) now only set when DevNetworking
  is enabled — production ALBs are reached via VPC IP directly

* feat: system credentials file + daemon wiring for ALB agent SigV4 auth

Write config/system-credentials.json during admin init/join with the
system access key. Daemon reads it at startup, resolves the gateway URL
from external IPAM (or 10.0.2.2 in dev mode), and wires both into the
ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY,
and ALB_SECRET_KEY into cloud-init for ALB VMs.

* feat: ALB agent gateway actions + daemon handlers (heartbeat, config)

Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB
agent can communicate via the AWS gateway instead of direct HTTP.

- Add agent_types.go with input/output types for both actions
- Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active
  on first contact, updates LastHeartbeat, processes health report
- Add GetALBConfig: returns pre-computed HAProxy config text and hash
- Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores
  config + SHA256 hash on LB record for agent pull)
- Remove waitForAgentReady polling goroutine (heartbeat handles transition)
- Remove HTTP client from ELBv2ServiceImpl (no longer needed)
- Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health
  reports now arrive directly via ALBAgentHeartbeat
- Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord
- Wire NATS subscriptions + daemon handlers for both new actions
- Add gateway handler files and elbv2Actions map entries
- Comprehensive tests for heartbeat, config, and updateStoredConfig

* feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r)

Replace the HTTP server agent with a poll-based client that communicates
with the gateway using SigV4-signed requests. The agent now sends
ALBAgentHeartbeat every 5s with health reports and fetches config via
GetALBConfig when the config hash changes.

* feat: add VPC service route for ALB agent gateway access

ALB VMs need to reach the gateway for heartbeat/config polling, but
VPCs without an IGW have no external routing. Instead of requiring
an IGW, the daemon now publishes a vpc.ensure-service-route event
before launching ALB VMs. vpcd handles this by creating minimal
external connectivity (external switch, localnet port, gateway router
port, chassis bindings) if not already present, then adds a /32 static
route for the gateway IP.

Refactors handleIGWAttach to share the external connectivity setup via
ensureExternalConnectivity helper.

* Revert "feat: add VPC service route for ALB agent gateway access"

This reverts commit d4220d8806.

* feat: add management NIC infrastructure for system instance control plane

Add config, IP allocator, TAP helpers, and VM struct fields for the
management NIC that gives system instances (ALBs etc.) L3 access to the
host control plane via br-mgmt, independent of VPC routing.

- DaemonConfig.MgmtBridge (default "br-mgmt")
- MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart
- generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4
- SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP)
- VM struct: MgmtMAC, MgmtIP, MgmtTap fields

* feat: wire management NIC into QEMU launch and cloud-init config

Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt),
extend generateNetworkConfig to emit mgmt0 stanza with static IP,
and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes
mgmt TAP cleanup on instance stop and failed launch cleanup.

* test: add coverage for ELBv2 service lifecycle and setter methods

* fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP

* fix: generate per-interface cloud-init config when eniMAC present without devMAC

The guard in generateNetworkConfig required both eniMAC and devMAC to
produce per-interface config, falling back to wildcard DHCP otherwise.
System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking
is independent), so the mgmt NIC stanza was never emitted. The wildcard
config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the
management NIC unconfigured and the agent unable to reach the gateway.

* fix: add wildcard support to ENI description filter + add ca-certificates to ALB image

- ENI DescribeNetworkInterfaces description filter did exact string match,
  but AWS supports * wildcards. E2E test uses "ELB *" which didn't match
  "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching.
- ALB image was missing ca-certificates package, so cloud-init ca_certs
  module couldn't add the Spinifex CA to the system trust store. Agent
  TLS handshakes failed with "certificate signed by unknown authority".
- Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP
  where async TerminateSystemInstance goroutine wrote to mock fields
  while the test read them.

* fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance

DeleteLoadBalancer was trying to delete ENIs synchronously, but they were
still attached (in-use) to the running VM. The delete was rejected with
InvalidNetworkInterface.InUse.

Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID
empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs
after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface
to clear the in-use status.

E2E test now polls for ENI cleanup (up to 30s) since VM termination is async.

* fix: detach then delete ENIs in DeleteLoadBalancer

ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface
rejects them. Fix: call DetachENI first to clear in-use status, then
delete. stopInstance tolerates already-deleted ENIs.

* test: add internal ALB dataplane E2E test

Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane
test. Internet-facing coverage is already provided by
run-elbv2-dataplane-internet-facing-e2e.sh.

The internal test validates:
- ALB reaches active via mgmt NIC heartbeat (no IGW dependency)
- Scheme=internal, no public IP, internal- DNS prefix
- Health checks pass via VPC L2 (OVN logical switch)
- Round-robin traffic from a VPC client to the ALB's private IP
- ENI cleanup on ALB deletion

Uses a client VM inside the VPC that curls the ALB private IP and
serves results via HTTP on its public IP for host-side verification.

* feat: add NLB constants and data model changes (mulga-838)

* feat: type-aware validation in CreateLoadBalancer (mulga-839)

Accept Type=network for NLB, reject SecurityGroups for NLB, set
CrossZoneEnabled defaults (false for NLB, true for ALB), and use
/net/ ARN path segment for network load balancers.

* feat: type-aware validation in CreateTargetGroup (mulga-840)

Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check
defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds).
Reject invalid protocols with InvalidParameterValue.

* feat: type-aware validation in CreateListener (mulga-841)

* feat: add NLB listener and protocol compatibility unit tests

Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4
protocols, protocol rejection for wrong LB type, listener-to-target-group
protocol compatibility validation, and default protocol rejection on NLB.

* feat: add TCP HAProxy config generation for NLB

Add TCP mode template and generateNLBConfig function for Network Load
Balancer support. GenerateHAProxyConfig now branches on LB type,
producing mode-tcp configs for NLBs with tcp-check or httpchk health
checks depending on the target group's health check protocol.

* refactor: rename albagent to lbagent for NLB generalization

Rename the ALB-specific agent package and binary to generic LB names
in preparation for NLB support. Both ALB and NLB VMs use the same
agent — it just applies whatever HAProxy config the control plane
generates.

- Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/
- Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig*
- Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat
- Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars
- Rename Makefile targets: build-alb-agent → build-lb-agent
- Update gateway action map, daemon subscriptions, service interface
- Fix pre-existing unused matchFilter function lint error
- Fix awserrors test count for InvalidConfigurationRequest

* feat: add sys.micro system instance type for LB VMs (mulga-845)

Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for
system VMs (load balancers, NAT gateways). Registered in the type map
for allocation but excluded from DescribeInstanceTypes, NATS
subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc
now returns sys.micro instead of scanning for smallest guest type.

* feat: LB image shrink and rename (mulga-846)

Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE
reduced from 512M to 192M. Update all references to use lb-agent binary
and LB_* env vars. Update COMMANDS.md for NLB support.

* chore: rename file

* feat: NLB E2E test and bug fixes (mulga-847)

Add full NLB data plane E2E test (33 assertions, 9 phases):
VPC/subnet setup, TCP target group with health checks, NLB
creation (type=network), TCP listener, agent heartbeat,
target health verification, TCP traffic round-robin, target
deregistration, and full cleanup verification.

Bug fixes found during E2E testing:
- Fix HAProxy stats socket path mismatch: config used alb-/nlb-
  prefix but agent expects lb- prefix, preventing health reports
- Fix system AMI lookup to match new -lb- image name, falling
  back to legacy -alb- name
- Bump LB image size from 192M to 256M (Alpine base is 200M)

* test: validate NLB internal scheme (mulga-857)

NLB with scheme=internal works out of the box — shared code paths
with ALB handle it correctly. Add unit test and E2E coverage:

- Unit test: NLB internal creates with no public IP, internal- DNS prefix
- E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic
- E2E passed 56/56 with perfect round-robin (20/20 requests)

* feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854)

Replace 3 separate LB dataplane scripts with a single consolidated test
that covers all 4 variants (ALB/NLB × internet-facing/internal).

- Share one VPC, subnet, and 2 dual-purpose app instances across all suites
- ALB + NLB provision in parallel within each phase (internet-facing / internal)
- One client VM tests both internal LBs simultaneously
- Replace sleep 100 with LB health check gates
- Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node

* chore: disable LB E2E tests in CI until dev env issue resolved

* fix: read LB agent credentials from spinifex.toml instead of system-credentials.json

The daemon couldn't read system-credentials.json due to systemd
hardening (file owned by root:root 0600, not in ReadOnlyPaths).
Use the same system credentials already loaded from spinifex.toml
via the predastore config section — these are identical keys written
by admin init.

* feat: automate LB image build and import (mulga-861)

- dev-install.sh: build lb-agent and run build-system-image.sh --import
  after services start so the LB image is available out of the box
- build-system-image.sh: add --quiet flag to suppress build output,
  remove hardcoded --config path (uses auto-detected default)
- images.go: rename alb-alpine → lb-alpine to match current naming

* update go version

* add LB e2e tests

* fix: clean LB image build dir and wait for nbd device readiness

Previous CI runs can leave stale state in /tmp/lb-image-build/ causing
e2fsck failures. Clean the build dir before each CI build. Also add a
readiness check after qemu-nbd connect to avoid race conditions.

* fix: serialize nbd access with flock for concurrent CI builds

Single-node and multi-node CI jobs share the same host and both build
the LB system image using /dev/nbd0. Use flock to serialize the nbd
connect/mount/customize/disconnect section so concurrent builds wait
rather than collide.

* fix: move flock to cover entire image build, not just nbd section

Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be
acquired before downloading/copying the image, not just before nbd
connect, otherwise the second job can clobber the first job's files.

* fix: skip LB image build if output is fresh from concurrent CI job

When concurrent CI jobs build the same image, the second job now reuses
the first's output if it's less than 10 minutes old, avoiding a
duplicate download + build cycle.

* fix: filter AMI discovery by name in E2E tests

With the LB Alpine image now imported alongside Ubuntu, the unfiltered
describe-images call picked up the wrong AMI. Filter by the expected
image name so tests use the Ubuntu image for instance lifecycle.

* fix: filter AMI discovery by name in multi-node E2E tests

Same fix as run-e2e.sh — with the LB Alpine image now imported,
the unfiltered describe-images picks the wrong AMI.

* fix: use awsgw bind IP as LB agent gateway URL fallback

When br-mgmt is not configured and dev_networking is off (production
with external pool), the gateway URL was falling through to the
external IPAM pool's GatewayIP — the WAN router, not the Spinifex
node. LB VMs couldn't reach the awsgw, so heartbeats never arrived
and ALBs stayed stuck in provisioning.

Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach
the host node's WAN IP where the gateway listens.

* fix: capture StateReason and daemon logs when LB app instances fail

Instances with user-data terminate immediately in CI. Add
describe-instances StateReason query and daemon log dump to diagnose
whether it's volume_preparation_failed or launch_failed.

* fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning

Three fixes for CI E2E failures:

1. run-e2e.sh: Remove custom AMI from predastore after deleting its
   backing snapshot. Previously the AMI remained discoverable but
   unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey
   on the deleted snapshot.

2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead
   of picking any non-alpine image. Also improve debug log dump to use
   journalctl (binary-install CI uses systemd, not log files).

3. CreateLoadBalancer: Only set provisioning state when a system VM
   was actually launched. Previously a failed launch still set
   provisioning, leaving the LB stuck forever with no agent to
   send heartbeats.

* fix: use AWSGW bind IP as LB agent gateway URL in multi-node

In multi-node deployments, the AWSGW binds to the WAN IP (e.g.
192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as
the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB
agent's heartbeat was refused, leaving internet-facing ALBs stuck in
provisioning.

Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches
it via VPC/external networking. Internal LBs get a host route on the
mgmt NIC so they can reach the AWSGW bind IP through br-mgmt.

* fix: add mgmt route via bootcmd for multi-node internal LBs

Alpine cloud-init does not support v2 routes under ethernets — adding
a routes: block to the network-config breaks all networking in the VM.

Instead, add the host route via a bootcmd in the LB user data. This
runs early in cloud-init and adds `ip route add {awsgw}/32 via
{br-mgmt}` so internal LB agents can reach the AWSGW through the
management NIC when the gateway binds to a specific WAN IP.

Also moves the networking service from boot to default runlevel in the
LB Alpine image to prevent a race with cloud-init.

Verified locally: internal ALB transitions to active with AWSGW bound
to WAN IP (simulating multi-node).

* add LB enable flag to e2e

* test: add elbv2 tests to meet diff coverage threshold

Cover health checker edge cases (invalid JSON, empty servers, zero
thresholds, port-0 fallback) and service_impl paths (mgmt route
cloud-init, protocol validation, missing target group).

* fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat

Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat
can pass the struct directly instead of marshaling to JSON and immediately
unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn
so failures are visible in logs.

* fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867)

Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey,
and systemSecretKey — blank env vars previously caused agent auth-loop with
no daemon-side log. Set LB state to failed (not active) when VM launch fails
so the API reflects the broken data-plane instead of black-holing traffic.

* fix: data race in getSystemAMI/getSystemInstanceType lazy caching

Use sync.Once to guard lazy resolution of systemAMI and
systemInstanceType fields. Concurrent CreateLoadBalancer calls
previously raced on unsynchronized read+write of these cached values.
The setter methods reset the Once so re-resolution works when the
func is set after initial construction.

Also fix pre-existing nosprintfhostport lint in daemon.go.

* fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned

publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now
delegates to utils.PublishEvent (matching the EIP service pattern).

isEIPOwned previously swallowed all KV errors and returned false, which
could release an EIP-owned public IP if the store was temporarily
unavailable. Now returns (bool, error) so the caller skips release on
KV failure instead of assuming no EIP ownership.

* fix: narrow health checker to only scan target groups for reporting LB

handleHealthReportDirect was calling ListTargetGroups() on every heartbeat
(every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow
LB → listeners → TG ARNs and fetch only the attached target groups via
direct key lookups. Falls back to full scan if LBID is missing.

* fix: pass region through to LB agent SigV4 signing (mulga-871)

LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region
from ELBv2 service config → cloud-init LB_REGION env var → lb-agent
--region flag → lbagent.New() → signer.Sign() call.

* fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig

Both endpoints accepted any accountID without verifying the LB belongs
to that account. Now verify accountID matches the LB's AccountID,
with an exception for the system account (000000000000) which the
ALB agent uses for heartbeats and config fetches.

* fix: upgrade silent failure paths from Debug to Warn/Error

- healthChecker target group list failure: Debug → Warn
- HAProxy stats socket error in agent tick(): Debug → Warn
- ENI delete EIP-owned skip: Debug → Info (operational visibility)
- wireLBAgentConfig: warn when no gateway host is resolved
- detectMgmtIP: warn when bridge exists but IP is empty
- stale mgmt TAP cleanup: log errors instead of discarding

* refactor: deduplicate MAC hash function into utils.HashMAC

Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC,
generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper.

* fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps

Previously the full LoadBalancerRecord (including ConfigText) was written
to JetStream KV on every heartbeat (every 5s). Now only persists on state
transitions (provisioning→active) or when LastHeartbeat is older than 60s.

* fix: fail ALB launch when public IP allocation fails for internet-facing scheme

Previously, IPAM allocation failure was logged as a warning and the ALB
VM continued launching without a public IP. The ALB reached active state
but was unreachable from the internet. Now the launch returns an error
and cleans up the partially-created instance, causing the ALB to enter
the failed state.

* fix: retry system AMI resolution when image not yet imported

sync.Once cached empty results permanently — if the daemon started
before the system image was imported, all CreateLoadBalancer calls
silently skipped VM creation. Replace sync.Once with a mutex that
only caches non-empty results, allowing retry on subsequent calls.

Same fix applied to getSystemInstanceType for consistency.

* fix: propagate updateStoredConfig errors to API callers (mulga-884)

updateStoredConfig and updateStoredConfigForTargetGroup now return
errors instead of silently swallowing them. Callers (CreateListener,
DeleteListener, RegisterTargets, DeregisterTargets) return
ErrorServerInternal when config persistence fails, preventing the API
from reporting success while HAProxy serves stale configuration.

* test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885)

* fix: parse HAProxy templates once at package level (mulga-875)

generateALBConfig and generateNLBConfig were calling template.New().Parse()
on every invocation. Move to package-level template.Must for one-time parsing.

* fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886)

When management TAP or IP allocation fails, internal-scheme ALBs now
return an error instead of silently continuing without a management NIC.
Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB
gets stuck in provisioning forever. Internet-facing ALBs retain the
existing degrade-and-continue behavior.

* refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877)

The app/net path segment derivation was duplicated in buildLBArn,
buildListenerArn, and inline in CreateLoadBalancer. Extracted into a
single lbArnPathSegment helper.

* refactor: use aws.StringValue in toHealthReport (mulga-878)

Replace manual nil-check-then-deref with aws.StringValue for cleaner
pointer-to-value conversion in the heartbeat health report builder.

* refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876)

Replace hand-rolled setupTestNATS (~20 lines) with the shared
testutil.StartTestJetStream helper. Simplify return to just *Store
since no caller needed the *nats.Conn.

* fix: address PR #122 review comments (mulga-887)

- Extract NATEvent named type from anonymous struct in publishNATEvent
- Remove unused build-lb-image.sh thin wrapper
- Remove SPX_ENABLE_LB env var from e2e workflow (unused)
- Remove no-op healthChecker start/stop methods
- Export simple ELBv2 fields, remove trivial setter methods
- Delete old e2e scripts replaced by consolidated run-lb-e2e.sh

* fix: check specific error type for ENI deletion on termination (mulga-888)

ENI deletion errors were all silently downgraded to Debug level,
masking genuine failures (KV issues, permissions, ENI stuck in-use)
that cause orphaned ENIs and leaked IPAM addresses. Now only
NotFound is logged at Debug; all other errors log at Error.

* fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890)

GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures,
not just missing devices. Permission errors on /sys/class/net/ were
silently treated as "bridge absent". Now only "no such network
interface" returns nil; other errors propagate to the caller.

* fix: remove backend address from proxy error response (mulga-892)

The reverse proxy error handler included the internal backend
host:port in the XML error body sent to clients. The address is
already logged server-side; the client now sees a generic message.

* fix: release mutex before KV store I/O in health checker (mulga-893)

handleHealthReportDirect held hc.mu across PutTargetGroup calls
(JetStream KV Put — network I/O), serializing concurrent heartbeats
from independent LBs. Now collects changed TGs under the lock and
persists them after releasing it.

* fix: warn when system credentials missing in wireLBAgentConfig (mulga-894)

fix: log DetachENI errors instead of discarding with blank identifier (mulga-895)

* fix: detect veth bridge mode and use centralized NAT for it

The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was
misclassified as direct bridge mode, causing OVN to use distributed NAT.
Distributed NAT hairpin routing fails through the Linux bridge intermediary,
breaking host-to-local-LB connectivity.

Add BridgeModeVeth constant, detect veth-wan-ovs interface in
detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode
and nat-addresses=router for both macvlan and veth modes. isMacvlanMode()
is preserved for macvlan-specific behavior (MAC alignment).

* fix: add retry loop for peer NLB reachability test

With centralized NAT, ARP resolution for NAT external IPs relies on
gratuitous ARPs from the gateway chassis. These take time to propagate
to peer nodes. The peer→NLB test had a single 3s attempt while all
other connectivity tests use retry loops.

* fix: use request-reply for vpc.add-nat to prevent ARP propagation races

publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon
continued before vpcd committed the OVN NAT rule. This caused intermittent
NLB connectivity failures: the LB was reported active before OVN had
the DNAT rule and sent GARPs for the public IP.

Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits
for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E
tests for more reliable TCP probing during ARP resolution.

* fix: clean up stale NAT rules before adding new ones for reused IPs

When a public IP is returned to the external pool and quickly reallocated
(e.g. instance terminated, then LB created), the fire-and-forget
vpc.delete-nat may not have been processed yet. handleAddNAT now removes
any existing dnat_and_snat rule for the same external IP before creating
the new one, preventing duplicate/conflicting OVN NAT rules.

This was causing intermittent NLB connectivity failures in multi-node E2E
where NLB public IPs collided with recently-freed instance IPs.

* fix: clean up stale NAT rules across all VPC routers on IP reuse

The previous fix (0a93081) only removed stale dnat_and_snat rules from
the target VPC's router. When a public IP is freed from one VPC and
reallocated to a different VPC before the fire-and-forget vpc.delete-nat
is processed, the stale rule on the old VPC's router caused duplicate
OVN ARP responses — traffic was black-holed when the host resolved to
the wrong router's MAC.

DeleteAllNATsByExternalIP now searches every logical router for stale
NAT rules matching the external IP and removes them in one transaction.

Also adds host connectivity diagnostics (ARP, routing, OVN NAT state)
to run-lb-e2e.sh so the next CI failure surfaces the root cause of the
persistent NLB-from-host issue.

* fix: run LB inet tests from actual hosting node, not hardcoded node1

The host/peer connectivity tests were hardcoded: "host" = node1 (where
the test script runs), "peer" = node2. But LB VMs land on whichever
node picks up the NATS queue message, so the tests were randomly testing
local vs remote connectivity depending on placement luck.

Now the test:
1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs)
2. After LB creation, discovers which node hosts each LB VM by checking
   for the ENI's tap device
3. Runs "local" tests from the hosting node and "remote" tests from a
   different node

Also updates e2e.yml to pass --nodes instead of --peer.
2026-04-10 10:27:20 +10:00
Julian Sommer b001c3929d Revert "feat: enforce nats authentication (#123)"
This reverts commit 553e7857df.
2026-04-09 16:06:03 +10:00
Julian Sommer 1a2e490ae7 fix: policy matching case insensitive 2026-04-09 15:34:43 +10:00
Julian Sommer 553e7857df feat: enforce nats authentication (#123) 2026-04-09 15:27:00 +10:00
Julian Sommer 7449090c86 fix: enable more linters (#121) 2026-04-09 14:36:22 +10:00
Julian Sommer 26e590716e feat: add systemd hardening directives to all service units (#120) 2026-04-09 12:09:43 +10:00
Julian Sommer e27d909e8d feat: move NBD sockets to /run/spinifex/nbd/ and enable PrivateTmp (#119) 2026-04-09 11:11:54 +10:00
Julian Sommer ccb6ed287d fix: move bootstrap.json to different folder to tighten perms (#118) 2026-04-08 16:12:25 +10:00
Julian Sommer 9a7e914d28 feat: awsgw directory (#117) 2026-04-08 15:19:29 +10:00
Julian Sommer 6026d46ba1 fix: ovn logging 2026-04-08 13:00:06 +10:00
Julian Sommer f0eb28ffc2 fix: remove gateway csp 2026-04-08 12:45:36 +10:00
Julian Sommer d33e6f3d70 chore: go 1.26.2 2026-04-08 11:53:29 +10:00
Julian Sommer 45ea467fcc feat: webserver proxy (#116) 2026-04-08 11:46:43 +10:00
Julian Sommer 820b605c0b fix: require sudo admin init 2026-04-08 09:32:25 +10:00
Julian Sommer a56a2baf1f feat: privilege separation (#114) 2026-04-07 16:22:19 +10:00
Julian Sommer 99f9949bdd fix: enable TLS on cluster manager API (#113) 2026-04-07 10:56:16 +10:00
dependabot[bot] 5598e523d5 chore: bump the npm-dependencies group (#112) 2026-04-07 08:57:04 +10:00
Julian Sommer dda4575888 fix: use binary install for single node e2e (#111) 2026-04-02 16:22:41 +11:00
Ben Duncan e5b6204875 Fix service helper sequence 2026-04-02 13:53:29 +10:00
Ben Duncan 21aa1296c9 systemd tweaks for binary install to correctly function 2026-04-02 13:35:52 +10:00
Julian Sommer ea6b5d770a fix: mask credentials in CLI override log output
Access key and secret key values were printed to stdout when using
--access-key and --secret-key flags. These are captured by systemd
journal and log aggregation, exposing secrets. Log that keys were
overridden without printing the actual values.

Addresses H-5 from security audit.
2026-04-02 12:25:27 +10:00
Julian Sommer f986cf9748 fix: ui cert path 2026-04-02 11:55:32 +10:00
Julian Sommer 11f7c99955 fix: ui cert path 2026-04-02 11:20:02 +10:00
Julian Sommer 1b45c7fc7d fix: nats ip in services 2026-04-02 10:58:39 +10:00
Julian Sommer 868a92bd55 fix: service 2026-04-02 10:35:00 +10:00
Julian Sommer af52d881b9 fix: chown 2026-04-02 10:26:56 +10:00
Julian Sommer c51ea0c29d fix: make download checksum verification mandatory 2026-04-02 09:56:57 +10:00
Julian Sommer 8cf0cc332f feat: remove pw login 2026-04-02 09:51:07 +10:00
Julian Sommer 1d6cefd9d2 feat: extract helpers (#109) 2026-04-02 10:25:01 +11:00
Julian Sommer 0c160d7f45 fix: add request body size limit 2026-04-01 15:36:06 +10:00
Julian Sommer 91e4efb0ef feat: extract tag helper 2026-04-01 15:27:43 +10:00
Julian Sommer f5ec8c3bae feat: formation server HTTPS (#108) 2026-04-01 16:20:28 +11:00
Julian Sommer 8795a7b926 fix: remove dev branch 2026-04-01 13:54:35 +10:00
Julian Sommer 39e9fa6fb8 feat: more filters (#106) 2026-04-01 14:23:04 +11:00
Julian Sommer 853b5bfd32 feat: filterutil migration (#105) 2026-04-01 14:05:21 +11:00
Julian Sommer d7c72e80af feat: filters (#104) 2026-04-01 12:20:56 +11:00
Julian Sommer 475e632404 chore: update COMMANDS.md 2026-03-31 15:53:02 +10:00
Julian Sommer 92e25a9653 chore: bump predastore 2026-03-31 14:11:47 +10:00
Julian Sommer ec525d3a6d add back -fsSL 2026-03-31 13:59:58 +10:00
Julian Sommer 94978f517b actual last 2026-03-31 13:55:48 +10:00
Julian Sommer 4d7239e645 last doc tweaks 2026-03-31 13:50:18 +10:00
Ben Duncan acc8dc1994 Remove macvlan 2026-03-31 13:36:42 +10:00
Julian Sommer a7ed93c6f3 docs: add prerequisite 2026-03-31 13:28:12 +10:00
Julian Sommer 67075c6600 docs: better urls 2026-03-31 13:25:23 +10:00
Ben Duncan ae223cdf15 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-31 13:19:48 +10:00
Ben Duncan b73a630f41 TF updates 2026-03-31 13:19:17 +10:00
Julian Sommer c283100fcf docs: rename source-install to source-installation 2026-03-31 13:16:42 +10:00
Julian Sommer 355decd406 docs: tweaks 2026-03-31 13:10:59 +10:00
Julian Sommer b76953bfb4 doc tweaks 2026-03-31 13:02:21 +10:00
Julian Sommer c9e83380df a 2026-03-31 11:59:07 +10:00
Julian Sommer e421840541 feat: placment group docs 2026-03-31 11:42:29 +10:00
Julian Sommer 406ba9431e fix 2026-03-31 11:39:10 +10:00
Julian Sommer 7882f77a36 add more docs 2026-03-31 11:36:09 +10:00
Julian Sommer f7b99f0c50 fix: COMMANDS.md 2026-03-31 11:29:28 +10:00
Julian Sommer f3ee50e9b8 fix: show deleted nat gateways 2026-03-31 10:49:36 +10:00
Julian Sommer 4f3b54abc2 fix: ModifyInstanceAttribute SourceDestCheck 2026-03-31 10:30:14 +10:00
Julian Sommer a7925d65f2 feat: DescribeAddressesAttribute (#103) 2026-03-31 11:14:27 +11:00
Julian Sommer 0d2e91da37 fix: downgrade node types to 24.12.0 2026-03-31 08:16:45 +10:00
Julian Sommer e2c7a64aff fix: set ui to use dark mode by default 2026-03-31 08:10:59 +10:00
dependabot[bot] 588141cf67 chore: bump the npm-dependencies group across 1 directory with 18 updates (#101) 2026-03-31 09:05:09 +11:00
dependabot[bot] 9e3076727e chore: bump github.com/nats-io/nats.go in the go-dependencies group (#99) 2026-03-31 09:03:19 +11:00
Ben Duncan c1df418c5c fix: parse NODE column correctly from pipe-delimited spx get vms output 2026-03-30 22:16:23 +10:00
Ben Duncan 5e0d819393 test: add frontend unit tests to meet 70% line coverage threshold
Add tests for: back-link, not-found, list-card, header,
delete-confirmation-dialog, use-initial-sidebar hook, admin queries,
and admin-dashboard-panel. Line coverage: 63.98% → 71.12%.
2026-03-30 22:09:14 +10:00
Ben Duncan 6d3b2f7e5d feat: add multi-node NAT GW + spread placement group E2E test
Phase 11 in run-multinode-e2e.sh:
- Create VPC with public + private subnets + IGW
- Launch bastion in public subnet, verify SSH
- Create spread placement group, launch 3 VMs in private subnet
- Validate spread distribution via spx get vms (3 nodes)
- Verify no internet pre-NAT (ping 8.8.8.8 fails from all 3)
- Create NAT GW + EIP + route, poll for OVN SNAT rule
- Verify internet via NAT GW from all 3 nodes (different physical hosts)
- Full cleanup: NAT GW, instances, placement group, VPC resources
2026-03-30 21:54:22 +10:00
Ben Duncan f3c5bce811 feat: admin dashboard, nodes page, S3 service metrics, sidebar branding
Admin Dashboard Panel:
- spinifex service namespace on AWS gateway (GetCallerIdentity,
  GetVersion, GetNodes, GetVMs, GetStorageStatus)
- Admin detection via SigV4-signed GetCallerIdentity
- Dashboard panel: version, license, cluster status, add-ons matrix
- Sidebar branding: Mulga logo + "Spinifex", SVG favicon
- Upgrade button (admin + open-source)
- Documentation link (admin only)

Nodes Page (/nodes, admin only):
- Cluster nodes with services, roles, region, uptime
- Elastic Compute Cloud (EC2) instances with age
- Resource usage per node + instance type capacity table

S3 Service Metrics (/s3/service-metrics, admin only):
- Daemon NATS handler parses predastore TOML (creds stripped)
- Gateway queries DB nodes /status + /health in parallel
- Predastore (S3) Overview: RS encoding, node counts, cluster health
- DB Consensus Nodes: Raft state, leader, term, commit/applied index
- Object Storage Shard Nodes: topology from config

Frontend infrastructure:
- signedFetch utility (SigV4 via @smithy/signature-v4)
- AdminContext + useAdmin hook
- React Query options for admin endpoints
- Full-width sidebar separator above sign out
2026-03-30 17:21:45 +10:00
Ben Duncan 9b8863a182 Revert "feat: add install call-home telemetry to admin init and join"
This reverts commit 6e807f88cf.
2026-03-30 16:35:47 +10:00
Ben Duncan 6e807f88cf feat: add install call-home telemetry to admin init and join
Sends anonymous telemetry to install.mulgadc.com/install during
spx admin init and join. Reports machine_id (/etc/machine-id),
region, az, node count, version, and arch. Fire-and-forget with
5-second timeout — never blocks the user.

Opt out with --no-telemetry flag or SPX_NO_TELEMETRY=1 env var.
2026-03-30 16:34:45 +10:00
Ben Duncan 32f5ec92c0 fix: exclude 127.0.0.1 from SERVICE_IPS when awsgw binds to specific IP 2026-03-30 11:14:49 +10:00
Ben Duncan 6a4b823de7 fix: only test TLS connections on awsgw bind IP, not all interfaces
In single-node mode, cert E2E discovered all host IPs (WAN, mgmt, VPC)
and tried openssl s_client on each. But awsgw only binds to the
configured host IP, not all interfaces. Management (10.11.x) and VPC
(10.x) IPs have no listener on port 9999, causing false failures.

Split NODE_IPS (all IPs for SAN validation) from SERVICE_IPS (IPs
where awsgw listens) parsed from spinifex.toml. Connection-based
tests (3, 4c, 5, 8) now use SERVICE_IPS.
2026-03-30 10:51:29 +10:00
Ben Duncan 91483d632e fix: replace fixed sleeps with retry loops in NAT GW E2E tests
The NAT GW E2E tests used 2-3 second sleeps before verifying SNAT
rule creation, outbound connectivity, and rule removal. This caused
flaky CI failures when OVN took longer to propagate flows to the
datapath (varies by host load and OVN version).

Replace with retry loops:
- SNAT rule check: poll OVN NB every 1s for up to 30s
- Outbound ping: retry up to 10 attempts with 5s backoff (~50s max)
- SNAT removal: poll up to 15 attempts with 2s backoff (~30s max)

Also dump OVN routes alongside NAT rules on failure for debugging.
2026-03-30 10:27:22 +10:00
Ben Duncan 5426dc1bb0 feat: improved certificate authority with auto-discovery, CA download, and cloud-init injection
Auto-discover all node IPs (WAN, mgmt, VPC) and hostname for server
cert SANs — fixes TLS failures when accessing Spinifex via non-localhost
IPs. Adds CA download endpoint (/api/ca.pem) with install instructions
on the login page. Injects CA cert into guest VM cloud-init so EC2
instances trust Spinifex services without --insecure. Adds 'spx admin
cert renew' command for cert regeneration after IP changes.

Includes certificate validation E2E test suite (run-cert-e2e.sh) with
system trust store, SAN, and in-guest TLS verification across single,
pseudo multi-node, and real multi-node modes.

Closes: mulga-777, mulga-778, mulga-779, mulga-780, mulga-781
Plan: docs/development/feature/cert-manager.md
2026-03-28 14:29:07 +10:00
Julian Sommer 15f88caee7 fix: latest frontend build 2026-03-27 15:57:23 +10:00
Julian Sommer 5d194098ca feat: security groups frontend (#97) 2026-03-27 16:56:00 +11:00
Julian Sommer 94a80a3be7 fix: lint issues 2026-03-27 15:13:55 +10:00
Julian Sommer 3f642524a8 feat: vpc create wizard (#96) 2026-03-27 16:07:20 +11:00
Julian Sommer de1340a331 fix: let reset-dev-env.sh auto-detect chassis ID from hostname
Previously hardcoded CHASSIS_ID="chassis-node1" which could mismatch
the OVS system-id from prior setup-ovn.sh runs, creating stale chassis
entries in the SBDB. Now defers to setup-ovn.sh's auto-detect
(chassis-$(hostname -s)) so the name is always consistent.
2026-03-27 12:44:01 +10:00
Julian Sommer ae094cb936 fix: set OutputPort on default route and filter stale chassis
Two stacked bugs prevented SSH to VPC instances:

1. The default route (0.0.0.0/0) was added without OutputPort. For
   DHCP-sourced pools the gateway router port uses link-local addressing
   (169.254.0.1/30) so OVN northd cannot resolve the nexthop (e.g.
   192.168.1.1) to an output port — it silently drops the route from
   the southbound DB.

2. discoverChassis returned all chassis from the SBDB including stale
   entries from previous OVN setups on the same host. When a stale
   chassis got highest gateway priority, the chassisredirect port
   stayed unbound (up: false), so no ARP/NAT traffic was processed.

Fix: add OutputPort to static routes in both handleIGWAttach and
reconcileIGW, and filter stale chassis by cross-referencing the local
OVS system-id with SBDB hostnames.
2026-03-27 12:40:52 +10:00
Julian Sommer c2ab63fdef feat: basic terraform templates 2026-03-27 12:09:49 +10:00
Julian Sommer 04bea48a39 fix: dhcp default behaviour 2026-03-27 11:53:48 +10:00
Julian Sommer a92ea13798 fixes 2026-03-27 11:30:39 +10:00
Julian Sommer 46670ee8af test 2026-03-27 10:54:02 +10:00
Julian Sommer cbc04a2437 fix: chassis names mismatch 2026-03-27 10:26:16 +10:00
Julian Sommer 216842cc1b fix: enable ovn on boot 2026-03-27 09:41:23 +10:00
Julian Sommer 8ea40a1973 fix: test coverage 2026-03-27 09:04:21 +10:00
Julian Sommer 5bfc680a68 fix: elbv2 error handling 2026-03-27 08:45:32 +10:00
Julian Sommer 98d5ec92ed fix: add pre commit hook 2026-03-27 08:28:16 +10:00
Julian Sommer 2244ba56b0 fix: race condition test 2026-03-27 08:09:26 +10:00
Ben Duncan 9e8e944faa fix: pass private IP in vpc.delete-nat event to clean up OVN rules (mulga-764)
Root cause: when terminating instances, the vpc.delete-nat event was
published with an empty logicalIP. vpcd's DeleteNAT looked for a
dnat_and_snat rule with logicalIP="" — found nothing — and the real
rule was left as a stale entry in OVN.

When the IPAM later recycled that public IP for an EIP/NAT Gateway,
OVN had conflicting rules: stale dnat_and_snat AND new snat, both
using the same external IP.

Fix: extract PrivateIpAddress from instance.Instance before publishing
vpc.delete-nat, in both running and stopped termination paths.

Also adds 14 unit tests for NAT Gateway service.
2026-03-27 07:43:53 +10:00
joshsiv-mulga 3129c9eb70 feat: generic system image builder + lint fixes (mulga-766)
Extract build-alb-image.sh into manifest-driven build-system-image.sh:
- alb.conf manifest + alb-setup.sh for ALB-specific config
- build-alb-image.sh kept as thin wrapper for backward compat
- Makefile targets: build-system-image, build-alb-image
- Hardened error handling: curl --fail, e2fsck/resize2fs checks,
  static-link enforcement, rc-update validation, nbd device guard

Also fixes 12 lint issues: errcheck, errchkjson, gosec, gofmt, modernize.
2026-03-26 22:30:19 +10:00
joshsiv-mulga f744aedede feat: ALB/ELBv2 service 2026-03-26 21:38:14 +10:00
Ben Duncan e5801aa6ba fix: NAT Gateway E2E order + CA cert in reset-dev-env (mulga-763)
E2E: associate subnet with route table BEFORE adding the NAT GW route.
CreateRoute publishes SNAT events for associated subnets — if the
association doesn't exist yet, no SNAT rule is created in OVN.

Validated locally: bastion SSH hop works, ping 8.8.8.8 passes with
NAT GW, fails without. All 5 steps confirmed.

reset-dev-env.sh: trust Spinifex CA cert before starting services,
fixing TLS cert verify failures on fresh resets.
2026-03-26 16:23:38 +10:00
Julian Sommer 77a4a40ca6 fix: tf 2026-03-26 15:46:09 +10:00
Julian Sommer ab84905297 fix: test coverage (#93) 2026-03-26 16:30:44 +11:00
Ben Duncan 2a3ada5e93 fix: NAT Gateway E2E — poll for private instance SSH readiness (150s)
The bastion SSH hop failed because the private instance's cloud-init
hadn't completed (SSH daemon not ready). The instance was "running"
from the hypervisor perspective but cloud-init takes ~30-40s.

- Replace 2-attempt retry with 30-attempt poll (5s intervals, 150s max)
- Hard failure with OVN debug output instead of silent skip
- Matches how wait_for_ssh works for public instances
2026-03-26 15:29:48 +10:00
Ben Duncan adfd20dee5 test: NAT Gateway E2E — bastion SSH + ping validation (mulga-763)
Adds Phase 8d to run-e2e.sh with 5-step NAT Gateway validation:

1. Baseline: bastion → private VM → ping 8.8.8.8 (FAIL, no NAT GW)
2. Create NAT GW + EIP + route table + route + associate with private subnet
3. With NAT GW: bastion → private VM → ping 8.8.8.8 (PASS, SNAT active)
4. Delete NAT GW + route + cleanup
5. After deletion: bastion → private VM → ping 8.8.8.8 (FAIL, confirms NAT GW)

Reuses public + private instances from Phase 8b (deferred cleanup).
Bastion SSH approach: CI → public VM (pub IP) → private VM (VPC priv IP).
OVN SNAT rule verification when ovn-nbctl is available.
2026-03-26 14:43:01 +10:00
Ben Duncan 4ffc7785a5 feat: NAT Gateway — CRUD + vpcd SNAT rules + route table integration (mulga-750)
Implements AWS-compatible NAT Gateways for private subnet outbound internet:

API (3 actions):
- CreateNatGateway: validates subnet exists, EIP allocated, stores in KV
- DeleteNatGateway: removes record, publishes delete events for SNAT cleanup
- DescribeNatGateways: filter by ID, vpc-id, state

vpcd SNAT rule management:
- handleAddNATGateway: creates OVN snat rule (subnet CIDR → NAT GW EIP)
- handleDeleteNATGateway: removes OVN snat rule
- Subscribes to vpc.add-nat-gateway / vpc.delete-nat-gateway topics

Route table integration:
- CreateRoute now accepts NatGatewayId target (in addition to GatewayId)
- Validates NAT GW exists and belongs to same VPC
- Publishes vpc.add-nat-gateway events for associated subnets

OVN mechanics:
- Single snat rule per private subnet CIDR (shared by all VMs)
- No policy-based routing needed — SNAT rewrites before route lookup
- Existing 0.0.0.0/0 default route handles all egress
- Connection tracking handles return traffic automatically
2026-03-26 14:43:01 +10:00
Julian Sommer e91b67a7c6 feat: ModifyNetworkInterfaceAttribute 2026-03-26 14:08:28 +10:00
Julian Sommer 051ecbd4ce fix test 2026-03-26 13:54:29 +10:00
Julian Sommer ef68c5bffe feat: DescribeInstanceCreditSpecifications 2026-03-26 13:46:46 +10:00
Julian Sommer 64c7305b37 feat: DescribeInstanceAttribute (#92) 2026-03-26 14:39:45 +11:00
Ben Duncan d025cab50e fix: harden route table error handling (silent-failure-hunter review)
- getRouteTable: distinguish ErrKeyNotFound (→ not found) from other
  errors (→ log + internal error) instead of mapping all to "not found"
- putRouteTable: log underlying error on marshal/write failures
- KV scan loops (allRouteTablesForVPC, DescribeRouteTables,
  DisassociateRouteTable, ReplaceRouteTableAssociation): fail on
  transient NATS errors instead of silently skipping entries. Only
  skip ErrKeyNotFound (deleted between Keys() and Get())
- ReplaceRouteTableAssociation: compensating write if second KV put
  fails — restores association to old table to prevent data loss
- createMainRouteTable: returns error instead of swallowing it
- ensureDefaultIGWRoute: log actual errors at Warn level, not Debug
- E2E test: fail on missing default VPC instead of skipping
2026-03-26 13:33:22 +10:00
Julian Sommer 1c194da915 fix: change "attached" to "available" in igw to match aws + add describe instance types filtering for TF 2026-03-26 12:34:32 +10:00
Julian Sommer 4ad5843ff5 feat: ModifyVpcAttribute + DescribeVpcAttribute (#91) 2026-03-26 13:28:09 +11:00
Ben Duncan 7ea0353d2f test: E2E route table validation in single-node test (mulga-761)
Adds Phase 8c to run-e2e.sh with three steps:

1. Verify default VPC main route table has local + IGW routes
2. Custom route table full lifecycle (create → add route → associate
   → disassociate → delete route → delete table)
3. Error paths: reject local route deletion, reject main table deletion
2026-03-26 12:26:52 +10:00
Ben Duncan f8fc2b2cbb feat: route tables V1 — CRUD, associations, VPC integration (mulga-760)
Implements AWS-compatible route tables for VPC subnets:

Route Table CRUD:
- CreateRouteTable, DeleteRouteTable, DescribeRouteTables
- Main route table auto-created with every VPC (CreateVpc + EnsureDefaultVPC)
- Cannot delete main table or tables with subnet associations

Route CRUD:
- CreateRoute (IGW targets), DeleteRoute, ReplaceRoute
- Immutable local route (VPC CIDR → local) cannot be deleted
- IGW ownership validation (must be attached to same VPC)
- Duplicate destination CIDR rejected

Association CRUD:
- AssociateRouteTable, DisassociateRouteTable, ReplaceRouteTableAssociation
- One explicit association per subnet
- Disassociating reverts subnet to main table

VPC Integration:
- CreateVpc auto-creates main route table with local route
- EnsureDefaultVPC creates main route table for default VPC
- ensureDefaultVPCInfrastructure adds 0.0.0.0/0 → IGW route

DescribeRouteTables filters: vpc-id, route-table-id, association.main,
association.subnet-id, route.destination-cidr-block, route.gateway-id

20 unit tests covering all CRUD operations and error paths.
2026-03-26 12:26:52 +10:00
Julian Sommer 74671a05a5 aws basic 2026-03-26 12:00:38 +10:00
Julian Sommer 90a3da2cc2 fix: daemon nil pointer 2026-03-26 11:59:44 +10:00
Julian Sommer 06a8da8fb5 fix: dependabot 2026-03-26 11:05:04 +10:00
Julian Sommer 7c6f03f0d0 chore: update action versions 2026-03-26 10:45:02 +10:00
Julian Sommer e6b48ce170 fix: docs 2026-03-26 10:29:48 +10:00
Julian Sommer fca9d018fc fix: install guide 2026-03-26 10:02:04 +10:00
Julian Sommer 4a2ae33dd9 fix: add missing assets 2026-03-26 09:56:38 +10:00
Julian Sommer 9e53443f7a fix: decrease shutdown wait time 2026-03-26 08:55:55 +10:00
Julian Sommer bf6b25842f feat: add dependabot auto updates 2026-03-26 08:36:50 +10:00
Ben Duncan 5b442e8eef chore: increase pseudo multi-node E2E timeout to 35m
External networking adds overhead (DHCP, tap setup/teardown) and the
shutdown drain phase needs more time with tap networking cleanup.
2026-03-25 20:54:46 +10:00
Ben Duncan a3c4819c1c fix: enable external networking in pseudo multi-node E2E
The pseudo multi-node test was not passing --external-mode/--external-pool
to spx admin init, so instances never got public IPs. SSH then failed
trying to use hostfwd which doesn't exist with tap networking.

- init_leader_node: pass SPX_EXTERNAL_POOL as --external-mode=pool flags
- e2e.yml: set SPX_EXTERNAL_POOL=192.168.1.220-229 for pseudo test run
2026-03-25 20:26:52 +10:00
Ben Duncan 70199f410a fix: pseudo multi-node E2E uses public IP + nbdkit for node detection
After enabling tap networking (public subnet), QEMU no longer uses
hostfwd bindings. The pseudo multi-node test was still grepping for
hostfwd to detect instance distribution and SSH endpoints.

- check_instance_distribution: count nbdkit root-volume processes per
  node IP instead of hostfwd-bound QEMU processes
- get_instance_node: match QEMU nbd socket → nbdkit host= IP to
  determine which simulated node hosts each instance
- count_instances_per_node: use get_instance_node instead of hostfwd
- SSH connectivity: check PublicIpAddress first (port 22), fall back
  to hostfwd — same pattern as single-node and multi-node E2E
2026-03-25 19:53:00 +10:00
Ben Duncan 7cdf630cf7 feat: propagate network config via formation server to joining nodes (mulga-759)
admin init builds a NetworkConfig from external networking flags and
passes it to the formation server. When joining nodes poll /formation/status,
they receive the cluster-wide [network] section (external mode, pools,
gateway, DNS, bootstrap VPC IDs). Each joining node auto-detects its
own external_interface and wan_bridge locally.

Fixes multi-node clusters having no [network] section in spinifex.toml
on secondary nodes, which caused no public IP assignment and broke
guest SSH in E2E tests.
2026-03-25 17:06:21 +10:00
Ben Duncan fb8c42e934 fix: poll for NAT rule cleanup instead of fixed sleep + hard fail
The dnat_and_snat rule is removed asynchronously via NATS after instance
termination. A 2s sleep was insufficient. Now polls for up to 30s and
downgrades to WARN if the rule persists (non-fatal — cleanup happens
when the VPC is deleted).
2026-03-25 16:35:15 +10:00
Ben Duncan cc46cf746e fix: multi-node E2E SSH via public IP when external networking enabled
Same fix as single-node: detect PublicIpAddress from describe-instances,
SSH via public IP on port 22. Falls back to QEMU hostfwd on the hosting
node when no public IP is available.
2026-03-25 16:05:54 +10:00
Ben Duncan e65f00c86a fix: set TenantID on root volume created by RunInstances (mulga-758)
GenerateVolumes did not set TenantID on the VolumeMetadata for root
volumes cloned from AMIs. After the pre-Phase4 tenant fallback was
removed (ae316db), describe-volumes filters volumes by exact tenantID
match — root volumes with empty TenantID became invisible to the
owning account, breaking E2E Phase 5b volume lifecycle tests.
2026-03-25 16:01:18 +10:00
Ben Duncan 85f51b8084 Merge branch 'vpc-public-subnet' into dev 2026-03-25 15:43:08 +10:00
Ben Duncan 4749a37725 fix: SSH via public IP instead of QEMU hostfwd when external networking enabled
When external_mode is set, dev_networking=false (no QEMU hostfwd NIC).
E2E tests now detect the instance's public IP from describe-instances
and SSH via port 22 on the public IP. Falls back to hostfwd only when
no public IP is available (overlay-only mode).

Applies to Phase 5a-ii, 7b, 7c-pre, and 8b SSH tests.
2026-03-25 15:34:33 +10:00
Ben Duncan 982a7b6b45 feat: SSH via public IP in E2E public subnet test
Verify full networking path: host → public IP → OVN DNAT → VM.
Test outbound internet from VM via OVN SNAT (ping 8.8.8.8).
Falls back to API-only verification if public IP unreachable.
2026-03-25 15:33:21 +10:00
Julian Sommer b81fdb5bc0 chore: frontend deps 2026-03-25 15:26:50 +10:00
Julian Sommer 71e82b22a7 chore: bump deps 2026-03-25 15:24:21 +10:00
Ben Duncan 21c4754336 feat: E2E tests for public/private subnet isolation + remove legacy nat mode
- Add Phase 8b to run-e2e.sh: public subnet instance gets PublicIpAddress
  and dnat_and_snat rule; private subnet instance gets neither; no blanket
  VPC CIDR SNAT (validates mulga-754 fix)
- Remove legacy external_mode="nat" — only "pool" with source=static or
  source=dhcp. Admin init defaults to pool/dhcp when WAN detected.
- Enable dev_networking always (was disabled with external mode, breaking
  E2E SSH tests)
- Update bootstrap.sh + e2e.yml: SPX_EXTERNAL_POOL env var configures
  static IP pool for CI (230-240 single-node, 240-250 multi-node)
- Fix run-vpc-e2e.sh SNAT assertion for mulga-754 (no blanket SNAT)
- Add unit tests for parseDHCPCDLeasedIP, ReconcileFromKV edge cases,
  DHCP pool config validation to meet 60% diff coverage threshold
2026-03-25 15:22:18 +10:00
Julian Sommer ec29cc2745 Revert "fix: remove InsecureSkipVerify (#87)"
This reverts commit a66b1a3d48.
2026-03-25 15:19:48 +10:00
Julian Sommer 5d31f05dba refactor: migrate ObjectStore from ListObjects V1 to V2
Replace all ListObjects (V1) usage with ListObjectsV2 across the
objectstore interface, both implementations, all 10 call sites, and
tests. V2 uses ContinuationToken instead of Marker for pagination
and adds KeyCount to responses. The frontend and Predastore already
speak V2 — this aligns the Go backend.
2026-03-25 14:44:13 +10:00
Julian Sommer 73e5905092 chore: bump viperblock version 2026-03-25 14:17:15 +10:00
Ben Duncan edcc53b947 Merge commit '820f620e34006699de1105859301146aaf491ca5' into vpc-public-subnet
# Conflicts:
#	cmd/spinifex/cmd/admin.go
#	docs/installing-spinifex/README.md
#	scripts/setup.sh
#	spinifex/admin/admin.go
#	spinifex/admin/network_detect.go
#	spinifex/handlers/iam/service_impl.go
2026-03-25 14:16:47 +10:00
Julian Sommer 820f620e34 feat: aws tf template 2026-03-25 13:54:24 +10:00
Ben Duncan 3d73dd180b fix: add dhcpcd-base to system dependencies for DHCP pool mode
DHCP pool mode requires dhcpcd-base (/usr/sbin/dhcpcd) to obtain
leases from the router. Added to Makefile install-system, setup.sh
binary installer, and source-install docs.

Closes: mulga-753
2026-03-25 12:05:21 +10:00
Ben Duncan 7bfb032f90 fix: remove blanket VPC CIDR SNAT from IGW — private subnet isolation (AWS parity)
handleIGWAttach and reconcileIGW created a blanket snat rule for the
entire VPC CIDR, allowing private subnet instances (no public IP) to
route outbound through the IGW. AWS behavior: only instances with
public IPs route through IGW via per-VM dnat_and_snat rules. Private
subnets require a NAT Gateway for outbound internet.

Also fixes pre-existing lint issues in external_ipam_dhcp.go (gofmt,
modernize, nolintlint).

Closes: mulga-754
2026-03-25 12:05:15 +10:00
Julian Sommer 15d6b0aa20 fix: multi node e2e decrease ssh timeout 2026-03-25 11:53:52 +10:00
Julian Sommer a66b1a3d48 fix: remove InsecureSkipVerify (#87) 2026-03-25 12:51:34 +11:00
Ben Duncan cbbd81bc76 feat: DHCP pool mode — public IPs from router DHCP instead of static range
Adds source="dhcp" option to external pool config. When set, ExternalIPAM
obtains public IPs from the upstream router's DHCP server (via dhcpcd on
br-ext) instead of allocating from a static range_start..range_end.

The VM behavior is identical to static pool mode: private VPC IP from OVN
DHCP, public IP via DNAT+SNAT, full DescribeInstances support. The only
difference is where the public IP comes from.

Two separate DHCP conversations:
- Host ↔ Router DHCP (br-ext): "give me a public IP for this VM's NAT"
- VM ↔ OVN DHCP (br-int): "give me my private VPC IP"

Tested: DHCP pool mode on single-node (ben-banksia). VM got public IP
192.168.1.77 from router DHCP, SSH works, outbound internet works.
2026-03-25 10:53:29 +10:00
Julian Sommer 8c4adf08a3 chore: bump predastore to v1.1.2 2026-03-25 10:52:08 +10:00
Julian Sommer c0b89e6d8c fix: path 2026-03-25 10:35:30 +10:00
Ben Duncan dd86805514 docs: add DHCP pool mode to vpc-networking guide
Explains static vs DHCP IP sources in pool mode, the two-DHCP-conversations
architecture, when to use each, and adds a DHCP homelab deployment example.
Also updates bridge setup section for veth pair approach.
2026-03-25 10:30:42 +10:00
Julian Sommer ad1339505d fix: dump vpcd log 2026-03-25 10:09:39 +10:00
Julian Sommer ba7e6a5c67 fix: redirect image import to dev null 2026-03-25 10:04:16 +10:00
Ben Duncan 4d56565c3f feat: veth pair bridge — link Linux bridge to OVS without conversion
setup-ovn.sh now detects whether the default route is on a Linux bridge
or OVS bridge. When it's a Linux bridge (e.g. br-wan from cloud-init),
instead of destructively converting it, the script creates an OVS bridge
(br-ext) and links them with a veth pair. The Linux bridge keeps its IP,
routes, and ports untouched — no WAN interruption.

Also fixes all pre-existing lint issues (gofmt, ineffassign, errchkjson,
gosec, modernize, whitespace) to bring golangci-lint to 0 issues.

Tested: pool mode with static IP range on single-node (ben-banksia).
VM gets public IP 192.168.1.201, SSH works, outbound internet works.
2026-03-25 10:01:37 +10:00
Julian Sommer ae316dba47 fix: remove pre-Phase4 volume tenant fallback
Simplify all 6 tenant-check sites in volume handlers from
`if accountID != "" && tenantID != "" && tenantID != accountID`
to `if tenantID != accountID`. No pre-Phase4 volumes with empty
TenantID exist in any environment — this code was unreachable.
Remove vol-legacy test fixture and empty-accountID-sees-all assertion.
2026-03-25 09:59:42 +10:00
Julian Sommer bdf0522895 feat: admin init auto install cert 2026-03-25 09:06:50 +10:00
dependabot[bot] e347f0a75e chore: bump github.com/nats-io/nats-server/v2 (#86) 2026-03-25 08:14:15 +10:00
Ben Duncan c2992ebe5e fix: reset-dev-env.sh — pass ADMIN_INIT_ARGS to second admin init --force
The second admin init --force was called without external networking
flags, overwriting the pool/nat config from the first call with
auto-detected NAT+DHCP defaults.
2026-03-24 17:09:48 +10:00
Ben Duncan ffca7330c5 fix: skip external IPAM init in NAT mode — only pool mode needs per-VM allocation
In NAT mode, all VMs share a single gateway IP for SNAT. No per-VM
public IPs are allocated. Previously, external IPAM was initialized
for any external_mode != "", causing noisy AllocateIP failures (empty
pool range) on every instance launch.

Now IPAM is only initialized when external_mode == "pool". In NAT mode,
d.externalIPAM stays nil, cleanly skipping the auto-assign public IP
path. SNAT topology is still created by vpcd via the pool's gateway_ip.

Part of mulga-737 (Phase 16).
2026-03-24 16:27:28 +10:00
Ben Duncan a21a275776 docs: update vpc-networking README for br-wan + new setup-ovn.sh flags
Replace all br-external references with br-wan. Update setup-ovn.sh
command examples to use --wan-bridge/--wan-iface/--macvlan instead of
--external-bridge/--external-iface/--direct.

Part of mulga-742.
2026-03-24 16:19:19 +10:00
Ben Duncan 78bc0d989d docs: update install guides + setup.sh for new setup-ovn.sh flags
- All guides: OVN setup before admin init (consistent ordering)
- Single-node + source-install: Replace --external-bridge/--external-iface
  with auto-detection + --wan-bridge/--wan-iface/--macvlan options
- Multi-node: Simplify examples (auto-detect WAN bridge), show override
  options in prose
- setup.sh: Update "next steps" to match single-node guide, add
  --wan-bridge/--macvlan guidance for physical NIC setups

Part of mulga-742.
2026-03-24 16:17:35 +10:00
Ben Duncan 8c6f66670a feat: add wan_bridge config field, update vpcd detection + network_detect
- config.go: Add WanBridge field to VPCDConfig (default "br-wan")
- spinifex.toml template: Add wan_bridge under [nodes.X.vpcd]
- admin.go: Auto-detect WAN bridge name from default route interface,
  add detectedWanBridge helper
- service.go: Wire WanBridge from cluster config to vpcd Config
- vpcd.go: Add WanBridge to Config, log it on startup, update comments
- network_detect.go: Fix isVirtualInterface to accept the default route
  device as a parameter instead of hardcoding br-external. Bridges like
  br-wan are now correctly treated as physical when they carry the
  default route.

Part of mulga-741.
2026-03-24 16:02:22 +10:00
Ben Duncan 1fca43c9f4 refactor: reset-dev-env.sh + stop-dev.sh — use new setup-ovn.sh flags
reset-dev-env.sh: Replace --external-bridge/--external-iface/--direct
with --wan-bridge/--wan-iface/--macvlan. Auto-detect when WAN interface
is already a bridge (tofu-cluster) vs physical NIC.

stop-dev.sh: Replace hardcoded br-external check with dynamic scan of
all OVS bridges — any non-br-int bridge with an IP means OVS is
infrastructure and should not be stopped.

Part of mulga-740.
2026-03-24 15:58:49 +10:00
Ben Duncan 34b50b11fb refactor: setup-ovn.sh — replace br-external with configurable WAN bridge
Replace --external-bridge/--external-iface/--direct with:
- --wan-bridge=NAME: OVS bridge for WAN traffic (auto-detected from
  default route if it's already a bridge)
- --wan-iface=NAME: physical NIC to add to the WAN bridge
- --macvlan: SSH-safe macvlan mode for single-NIC hosts

Auto-detection logic:
- If default route goes through a bridge (br-wan) → use it directly
- If default route is a physical NIC → stop with guidance (never
  auto-move a NIC into OVS — may drop SSH)

Bridge-mappings updated: external:$WAN_BRIDGE (was external:br-external).
OVN logical network name stays "external" (no topology.go change needed).

Encap IP now prefers br-vpc IP when available (dedicated VPC data plane).

Part of mulga-739. Blocks VPC E2E work.
2026-03-24 15:46:49 +10:00
Julian Sommer 8f81ac29a4 chore: update license references 2026-03-24 15:43:11 +10:00
Julian Sommer eefed70658 docs: add iam 2026-03-24 15:16:33 +10:00
Ben Duncan 216ca9e710 feat: Phase 12b + Phase 16 — KV reconcile on startup + NAT mode support
Phase 12b: vpcd now reconciles OVN topology from NATS KV on startup
(Pass 2). After subscribing to topics, it reads all VPCs, subnets,
IGW attachments, and ENIs from KV and ensures matching OVN objects
exist. Handles reboots, OVN DB loss, and missed events. All operations
are idempotent.

Phase 16 prep: Add --gateway-ip flag to admin init for NAT mode
(specifies the OVN gateway router's external IP without DHCP).
Update reset-dev-env.sh to support EXTERNAL_MODE=nat env var for
NAT/DHCP mode testing alongside the default pool mode.
2026-03-24 15:03:04 +10:00
Julian Sommer fefd5ed9e8 chore: update to typescript 6 2026-03-24 14:29:55 +10:00
Julian Sommer d47f2ab4fd fix: a11y 2026-03-24 14:02:26 +10:00
Julian Sommer 58a9d5741f fix: missing ui tests 2026-03-24 13:45:48 +10:00
Julian Sommer 625a78c529 feat: placement groups frontend 2026-03-24 13:39:04 +10:00
Ben Duncan 88baf5e202 feat: direct bridge mode — prefer over macvlan when WAN NIC != SSH NIC
Auto-detect bridge topology: if a dedicated WAN NIC exists (not used for
SSH), add it directly to br-external instead of creating a macvlan.
Direct bridge enables distributed NAT (DNAT on VM's own chassis) and
eliminates macvlan workarounds (MAC alignment, gratuitous ARPs,
centralized NAT bottleneck).

- topology.go: bridge-mode-aware NAT and localnet port options
- vpcd.go: auto-detect bridge mode, skip MAC alignment for direct bridge
- setup-ovn.sh: --direct flag for physical NIC on br-external
- reset-dev-env.sh: SSH NIC detection via SSH_CONNECTION
2026-03-24 01:57:54 +00:00
Julian Sommer 6379a2098a fix: lint errors 2026-03-24 11:55:42 +10:00
Julian Sommer 638828f636 feat: add leaders to get nodes command 2026-03-24 11:50:28 +10:00
Julian Sommer 0fbf4537c0 chore: bump viperblock 2026-03-24 11:24:40 +10:00
Julian Sommer 1b5108d379 chore: bump frontend deps 2026-03-24 10:45:01 +10:00
Julian Sommer bab7b1dea3 fix: enable last rules 2026-03-24 10:28:20 +10:00
Julian Sommer 30b08ca984 fix: use nullish coalescing 2026-03-24 10:20:11 +10:00
Julian Sommer 3467e0cf22 feat: oxlint type checked rules 2026-03-24 10:14:31 +10:00
Julian Sommer 35d0df5b56 feat: placement groups (#85) 2026-03-24 10:52:00 +11:00
Julian Sommer 648b36d672 fix: add agent browser 2026-03-24 09:25:48 +10:00
Ben Duncan d6062ed550 docs: document direct bridge vs macvlan modes for br-external
Direct bridge (WAN NIC added directly to OVS) is preferred when a
dedicated WAN NIC exists. Enables distributed NAT, no MAC workarounds,
host can reach VMs. Macvlan is fallback for single-NIC hosts only.
2026-03-23 11:36:15 +00:00
Ben Duncan 93aaf0f991 fix: single-node E2E — OVN SB cleanup, chassis naming, macvlan NAT, DNS
Three blockers fixed for VPC public subnet E2E on single-node:

1. OVN SB stale data: reset-dev-env.sh now deletes both NB+SB DB files
   instead of individual NB cleanup. Eliminates ovn-controller commit
   failure loop from stale chassis/port binding entries.

2. Chassis name mismatch: reset-dev-env.sh passes --chassis-id=chassis-node1
   to setup-ovn.sh, aligning OVS system-id with vpcd's chassis-{nodeName}.

3. Macvlan unicast filtering: switched to centralized NAT (no external_mac
   on DNAT rules), set nat-addresses=router on localnet ports, and vpcd
   aligns macvlan MAC with OVN gateway router MAC on startup. Macvlan only
   delivers unicast matching its own MAC — this ensures inbound traffic
   from the LAN reaches OVS/OVN.

Also fixed: DNS for VMs. admin init auto-detects host DNS servers via
resolvectl, stores them in spinifex.toml dns_servers field, and vpcd
propagates them to OVN DHCP options. Fallback: 8.8.8.8 + 1.1.1.1.

Validated E2E: inbound ping, SSH, outbound ping, DNS, curl, apt update.
2026-03-23 10:59:08 +00:00
Ben Duncan f34452883c refactor: move bootstrap config to admin init, add IGW ID, eliminate race
- admin init now pre-generates VPC/subnet/IGW IDs and writes [bootstrap]
  to spinifex.toml before services ever start. No race condition.
- EnsureDefaultVPC accepts optional BootstrapIDs to use pre-determined
  IDs instead of generating random ones.
- CreateInternetGatewayWithID allows daemon to use the bootstrap IGW ID.
- Removed writeBootstrapConfig from daemon (no longer needed).
- Fixed bootstrap account_id: uses admin account (000000000001) not
  system account (000000000000).
- reconcileIGW now sets spinifex:igw_id in OVN external IDs.
2026-03-23 06:57:25 +00:00
Ben Duncan c21f3dc188 feat: vpcd OVN reconciliation on startup + default subnet resolution
Phase 11b: RunInstances now auto-resolves the default subnet when no
SubnetId is specified, matching AWS behavior. Previously fell back to
SLIRP networking.

Phase 12: vpcd reconciles OVN topology from [bootstrap] config on startup.
The daemon writes a [bootstrap] section to spinifex.toml with the default
VPC/subnet IDs on first run. vpcd reads this on startup and creates any
missing OVN objects (router, switch, DHCP, IGW topology, SNAT, gateway
chassis). All operations are idempotent — safe on every restart.

This solves the bootstrap ordering problem where admin init creates VPC
records before vpcd is running, so NATS events are missed.

EnsureDefaultVPC now returns DefaultVPCInfo with VPC/subnet IDs.
2026-03-23 06:40:19 +00:00
Ben Duncan daae6ca6b8 refactor: remove single-nic bridge mode, standardize on macvlan
The single-nic IP migration path was fragile and caused connectivity
issues during setup. macvlan is now the only bridge strategy — always
creates spx-ext-{nic} on the WAN NIC. Removed --single-nic flag,
SingleNIC config/detection fields, and updated all docs.
2026-03-23 01:12:22 +00:00
Ben Duncan 32276c7cc7 feat: wire EIP + SG gateway actions, fix gateway chassis scheduling, default VPC infra
Phase 8a: Wire all 5 EIP gateway actions (AllocateAddress, ReleaseAddress,
AssociateAddress, DisassociateAddress, DescribeAddresses) end-to-end through
gateway → NATS → daemon → EIP service.

Phase 8b: Wire all 7 SG gateway actions (Create/Delete/Describe SecurityGroup,
Authorize/Revoke Ingress/Egress) through the same pipeline.

Critical fix: handleIGWAttach was not calling SetGatewayChassis on the gateway
router port. Without this, OVN has no chassis binding for external traffic —
SNAT/default route exist logically but packets have nowhere to go physically.
This was the root cause of outbound routing failure (can't ping 8.8.8.8).

Default VPC now matches AWS behavior: auto-creates IGW (attached),
MapPublicIpOnLaunch=true on default subnet, and a default security group
with intra-SG inbound allow + all outbound allow.

Comprehensive VPC networking docs rewrite covering modes, bridge setup,
configuration, security groups, EIPs, OVN reference, and troubleshooting.
2026-03-21 10:26:59 +00:00
Ben Duncan 30394b735b Removed legacy dir 2026-03-21 09:36:28 +00:00
Ben Duncan a25a4cec34 Merge commit '9b64c1911fc7e1a2442c376fb2cba0f5ece9a913' into vpc-public-subnet 2026-03-20 07:28:15 +00:00
Ben Duncan f6c6660f15 fix: create default VPC for admin account, not just system account
EnsureDefaultVPC was only called for GlobalAccountID (000000000000).
The admin account (000000000001) from admin init never got a default
VPC because seedAdminAccount didn't publish iam.account.created.

Fix: daemon startup now ensures default VPC for both system and admin
accounts. Also publish iam.account.created from seedAdminAccount so
any future account-creation listeners fire correctly.
2026-03-20 07:19:05 +00:00
Ben Duncan 704874c0c7 docs: restructure install guides + new common setup guide
All 3 install guides (binary, multi-node, source) now end at the same
verification point: aws ec2 describe-instance-types. Installation is
complete when that command returns instance types.

New docs/setting-up-your-cluster/README.md is the common guide all 3
install paths point to. Covers: AMI import, SSH key creation, VPC with
public subnet (IGW + modify-subnet-attribute), instance launch with
public IP, SSH via public IP, instance lifecycle management.

Removed SSH-via-hostfwd from all guides — public subnet is the default
connectivity path. Private subnet documented as an alternative.
2026-03-20 07:10:33 +00:00
Ben Duncan 665093343c fix: preserve DNS resolution after IP migration to br-external
When setup-ovn.sh migrates the WAN IP from a physical NIC to
br-external, systemd-resolved loses its DNS association (DNS servers
are per-interface). This breaks hostname resolution for everything
on the host — git, apt, curl, SSH all fail.

Now captures DNS config from the original interface before migration
and re-associates it with br-external via resolvectl. Falls back to
8.8.8.8 if no DNS was configured on the original interface.
2026-03-20 06:57:18 +00:00
Ben Duncan bb9e8db386 fix: stop-dev.sh must not stop OVS when br-external has WAN IP
When setup-ovn.sh migrates the host's WAN IP to br-external, stopping
openvswitch-switch destroys the bridge and the host loses all network
connectivity — unreachable via SSH, requires console reboot.

Now checks if br-external has an IP before stopping OVS. If it does,
skip the OVS stop and only stop ovn-controller + ovn-central. OVS is
infrastructure — bridges must survive service stop/start cycles.

Discovered during 3-node E2E testing: stop-dev.sh killed all 3 nodes'
WAN connectivity simultaneously.
2026-03-20 06:55:23 +00:00
Ben Duncan 0d4a70ef28 fix: remove verbose next-steps from admin init, disable dev_networking with external mode
Admin init output was duplicating install guide steps. Simplified to
just show completion message. Install guides already have all steps.

Set dev_networking=false when external_mode is configured — the hostfwd
NIC adds a confusing second interface inside VMs and is not needed when
public subnets provide WAN-routable IPs.
2026-03-20 06:45:11 +00:00
Ben Duncan 946429d792 feat: Phase 8c — daemon auto-assign public IP on RunInstances
Wire public IP auto-assignment end-to-end:
- RunInstances checks MapPublicIpOnLaunch on subnet, allocates external
  IP from ExternalIPAM, updates ENI record, publishes vpc.add-nat for
  dnat_and_snat rule, returns PublicIpAddress in response
- TerminateInstances releases public IP back to pool, publishes
  vpc.delete-nat to remove NAT rule before ENI deletion
- DescribeInstances populates PublicIpAddress from VM struct

Add ExternalIPAM initialization in daemon startup from cluster
NetworkConfig. Add GetSubnet and UpdateENIPublicIP helpers. Add
PublicIP/PublicIPPool fields to vm.VM struct. Fix network auto-detect
to recognize br-external as WAN interface.
2026-03-20 06:41:32 +00:00
Ben Duncan a554fe4ecf docs: rewrite single + multi-node install guides
Single-node:
- Add auto-detected network topology + external networking section
- Replace hostfwd SSH with VPC overlay private IP SSH
- Add setup-ovn.sh step with --dhcp flag
- Fix undefined variables ($SPINIFEX_AMI, $SPINIFEX_SUBNET)
- Use bash variable capture for VPC/subnet/AMI IDs
- Add three-tier external networking table
- Fix step headings (## → ###)

Multi-node:
- Move CA certificate trust to Step 5 (before starting services)
- Add --external-bridge --dhcp to all OVN setup commands
- Add AMI import step (was missing entirely)
- Add VPC + subnet creation step with variable capture
- Add SSH via private IP step
- Fix sudo inconsistency (all formation commands use sudo)
- Add external networking section with pool upgrade path
- Add DHCP troubleshooting section
- Add port requirements to overview
2026-03-20 05:58:38 +00:00
Ben Duncan 2af3bf5287 feat: auto-detect network topology + DHCP gateway in admin init
spx admin init now works with zero external networking flags:
- Auto-detects WAN interface, gateway, prefix from ip route
- Auto-detects single-NIC vs multi-NIC mode
- Defaults to nat mode with DHCP for gateway IP
- Prints detected topology with interface roles

setup-ovn.sh gains --dhcp flag: obtains a DHCP lease on the
macvlan/bridge interface for the OVN gateway SNAT address, then
writes gateway_ip into spinifex.toml. Tested on VM: macvlan gets
separate IP from router DHCP, host keeps its original IP.

Three tiers of external networking:
- Zero flags: auto nat+DHCP (VMs get outbound internet)
- --external-pool=start-end: pool mode (VMs get public IPs)
- --no-external: overlay only (no internet for VMs)
2026-03-20 05:45:30 +00:00
Ben Duncan 1f38213d4b feat: add external networking flags to spx admin init
Add --external-mode, --external-iface, --single-nic, --external-pool,
--external-gateway, --external-prefix-len flags to admin init. Config
is now generated with the correct [network] section and per-node vpcd
external_interface/single_nic — no manual spinifex.toml editing needed.

Single-node single-NIC:
  spx admin init --external-mode=pool --external-iface=eth0 --single-nic \
    --external-pool=192.168.1.150-192.168.1.250 --external-gateway=192.168.1.1

Single-node two-NIC:
  spx admin init --external-mode=pool --external-iface=eth1 \
    --external-pool=192.168.1.150-192.168.1.250 --external-gateway=192.168.1.1

Next steps output now shows the correct setup-ovn.sh command with
matching flags.
2026-03-20 05:24:06 +00:00
Ben Duncan 1ee7012ba9 feat: Phase 7 — single-NIC macvlan bridge strategy
Add --single-nic flag to setup-ovn.sh. When used with --external-bridge,
creates a macvlan sub-interface (spx-ext-{iface}) in bridge mode off the
host's NIC instead of adding the NIC directly to br-external. The host
keeps its IP — no migration, SSH-safe.

Strategy selection:
- Dedicated NIC: --external-iface=eth1 (add directly)
- Single NIC:    --external-iface=eth0 --single-nic (macvlan)
- IP migration:  --external-iface=eth0 (legacy, no --single-nic)

Add single_nic bool field to VPCDConfig for per-node config. Different
nodes in a cluster can use different strategies. Unit test verifies
single_nic parsing alongside external_interface.
2026-03-20 05:15:35 +00:00
Ben Duncan b6e99947e1 feat: Phase 5 — unit tests for ACL, security groups, public subnet, EIP
Add 18 unit tests covering:
- ACL match expression compiler (8 tests): TCP port/range, ICMP, UDP
  range, all-protocols, SG-ref source, egress, no-source filter
- Security group Port Group lifecycle (3 tests): create, delete, update
  with ACL generation via NATS events + MockOVNClient
- ModifySubnetAttribute + MapPublicIpOnLaunch (2 tests): default false,
  toggle to true via ModifySubnetAttribute
- Elastic IP lifecycle (5 tests): allocate, allocate from specific pool,
  release, release-while-associated error, describe
2026-03-20 04:35:52 +00:00
Ben Duncan 5481ef4361 feat: Phase 4 — security groups (OVN ACLs + Port Groups)
Add security group CRUD handlers: CreateSecurityGroup, DeleteSecurityGroup,
DescribeSecurityGroups, AuthorizeSecurityGroupIngress/Egress,
RevokeSecurityGroupIngress/Egress. SG records stored in NATS KV.

Add ACL match expression compiler (acl.go) translating AWS SG rules to
OVN ACL match strings for TCP/UDP port/ranges, ICMP, all-protocols, CIDR
filters, and intra-SG references.

Add vpcd security.go handlers for OVN Port Group lifecycle and ACL
management via vpc.create-sg/delete-sg/update-sg NATS topics. Extend
OVNClient interface with CreatePortGroup, DeletePortGroup,
SetPortGroupPorts, AddACL, ClearACLs. Add PortGroup and ACL to nbdb
model.
2026-03-20 04:24:49 +00:00
Ben Duncan ee06105c56 feat: Phase 3 — Elastic IPs (AllocateAddress, AssociateAddress)
Add EIP service with AllocateAddress, ReleaseAddress, AssociateAddress,
DisassociateAddress, and DescribeAddresses. Uses ExternalIPAM for IP
allocation from admin-defined pools. Publishes vpc.add-nat/delete-nat
events for OVN dnat_and_snat rules.

EIP records stored in spinifex-vpc-elastic-ips NATS KV bucket with
account-scoped keys. AssociateAddress replaces auto-assigned public IPs
when present.
2026-03-20 04:21:46 +00:00
Ben Duncan 150bf246a4 feat: Phase 2 — public subnets + auto-assign public IPs
Add MapPublicIpOnLaunch field to SubnetRecord with ModifySubnetAttribute
API handler. Add PublicIpAddress/PublicIpPool fields to ENIRecord for
tracking auto-assigned public IPs.

Add vpc.add-nat / vpc.delete-nat NATS topics and handlers in vpcd
topology for creating distributed dnat_and_snat NAT rules (with
LogicalPort + ExternalMAC for chassis-local NAT processing).

Wire ModifySubnetAttribute through gateway → daemon → VPC service.
2026-03-20 04:19:28 +00:00
Julian Sommer 9b64c1911f ci: add pull_request triggers to CI workflows
Enable PR-triggered CI so required status checks can gate merges.
Spinifex runs on all PRs (no path filter) to avoid blocking non-Go PRs.
UI workflow keeps path filters as it's not a required check.
2026-03-20 14:08:12 +10:00
Ben Duncan c1cdfc07c3 feat: Phase 1 completion — external IPAM, br-external setup, config template
Add external IPAM module (CAS-based IP allocation from admin-defined
pools in NATS KV). Supports multi-pool with region/AZ scoping, pool
fallback logic, gateway IP reservation, and concurrent allocation via
CAS retries. 12 unit tests covering allocation, release, exhaustion,
CAS conflict, multi-pool, fallback, validation, and idempotent init.

Add --external-bridge flag to setup-ovn.sh: creates OVS bridge
br-external, adds WAN NIC, migrates host IP to bridge, and sets
ovn-bridge-mappings=external:br-external for OVN localnet ports.

Add commented [network] section and external_interface to spinifex.toml
config template for discoverability.
2026-03-20 03:59:47 +00:00
Ben Duncan c6af189638 Merge commit '02931567ac207bd074d8b2efda6e523ab36540d9' into vpc-public-subnet 2026-03-20 03:47:50 +00:00
Julian Sommer 02931567ac release: update go.mod for v0.4.2 2026-03-20 13:09:48 +10:00
dependabot[bot] c4890c80d3 build(deps): bump fast-xml-parser (#83) 2026-03-20 12:48:53 +11:00
Julian Sommer d2d3e52522 feat: migrate to golangci-lint (#82) 2026-03-20 12:14:42 +11:00
Julian Sommer 42cb6dbed0 fix: add better error message 2026-03-20 11:12:32 +10:00
Julian Sommer 6cdeea3aa8 chore: more tests 2026-03-20 09:27:21 +10:00
Julian Sommer 4884b68b51 feat: version bootstrap file 2026-03-20 09:12:42 +10:00
Julian Sommer 726528bd9d feat: version jetstream buckets 2026-03-20 09:07:40 +10:00
Julian Sommer 593920ce7d feat: E2E test redistribution and dev-setup cleanup
Redistribute E2E tests between single-node and multi-node suites, slim
down pseudo multi-node suite, and remove directory creation from
dev-setup.sh so spx admin init is the single source of truth for
~/spinifex/ setup.
2026-03-20 08:24:15 +10:00
Ben Duncan 5e30d12f76 feat: Phase 1 — external network config + IGW gateway router fix
Add cluster-wide [network] config with external_mode and multi-pool
support (ExternalPool with region/AZ scoping). Per-node vpcd gets
external_interface for WAN NIC name.

Fix handleIGWAttach to use real external IPs from the configured pool
instead of link-local 169.254.0.1/30. Gateway router port now gets
the pool's gateway IP with correct prefix length, SNAT uses the real
external IP, and the default route points to the actual WAN gateway.

Includes pool selection logic (AZ-scoped → region-scoped → unscoped
fallback) and unit tests for config parsing, IGW with external pool,
explicit gateway IP, and pool selection.
2026-03-19 11:34:30 +00:00
Julian Sommer ea8700df99 fix: increase sleep time 2026-03-19 15:02:50 +10:00
Julian Sommer 857cbd75ac fix: use spx service stop instead of shell PID kill in stop-dev.sh 2026-03-19 14:50:50 +10:00
Julian Sommer 0fa38f7986 fix: write credentials to user even when running as sudo 2026-03-19 14:05:48 +10:00
Julian Sommer 1a4d590bf3 fix: nbd log level 2026-03-19 13:45:27 +10:00
Julian Sommer b7678ab7c5 feat: fix silent failures across spinifex (P2 audit H-25 through H-43)
- PID file failures now prevent service start instead of logging and continuing
- Status() methods use PID-based health checks instead of returning empty string
- NewResourceManager returns error instead of silently defaulting to 8GB
- Crash recovery re-allocates resources before relaunch
- restoreInstances stops after failed resource allocation
- Cluster manager bind errors propagated to Start()
- NATS disconnect/reconnect logged at Warn level
- VPC CIDR fallback logs warning about overbroad rule
- vpcd respond() handles json.Marshal failure
- Network cleanup failures logged instead of silently discarded
- fmt.Println replaced with slog.Debug in NATS service
- Signature mismatch no longer logs expected signature
- extractDiskImagePath uses errors.Is for EOF
- validateDiskImagePath checks file command error
- os.Stat uncommented in admin image import
- net.SplitHostPort errors checked
- Gateway response write errors logged
- vpcd OVN NB address uses leader IP on non-primary nodes
- Release workflow parallelized with native arm64 runner
2026-03-19 13:37:41 +10:00
Julian Sommer 11730be685 fix: locate release assets by filename instead of hardcoded path 2026-03-19 12:13:28 +10:00
Julian Sommer a4bc8bee74 fix: pending instance terminate (#80) 2026-03-19 12:47:00 +11:00
Julian Sommer fafc104c30 feat: add release verification job with draft-then-promote flow 2026-03-19 10:52:39 +10:00
dependabot[bot] efa6257850 build(deps): bump google.golang.org/grpc from 1.79.1 to 1.79.3 (#79) 2026-03-19 11:35:06 +11:00
Julian Sommer 7e46f0012b chore: bump more action versions 2026-03-19 10:23:52 +10:00
Julian Sommer 58e6817efe chore: bump docker build action ver 2026-03-19 10:19:02 +10:00
Julian Sommer 71ec2b7ee5 release: update go.mod for v0.4.0 2026-03-19 08:28:16 +10:00
Julian Sommer 9c430eea07 fix: release workflow 2026-03-19 08:17:30 +10:00
Julian Sommer 1be6c69e51 format 2026-03-19 08:03:23 +10:00
Ben Duncan c523f78403 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-18 05:08:30 +00:00
Ben Duncan 2918a02fac correct spx path 2026-03-18 05:08:15 +00:00
Julian Sommer 8e3210f023 fix 2026-03-18 14:36:12 +10:00
Julian Sommer 6ba9d3dbf0 remove trailing slash 2026-03-18 14:24:52 +10:00
Ben Duncan 27cdfc4653 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-18 04:14:22 +00:00
Ben Duncan adadb5ef4c Fixed noninteractive error on apt get 2026-03-18 04:13:53 +00:00
Julian Sommer d0fc2a2120 last fix 2026-03-18 13:40:27 +10:00
Julian Sommer a09a8efe3c build frontend with new branding 2026-03-18 13:38:47 +10:00
Julian Sommer 7d43d0f782 hopefully last rename 2026-03-18 13:24:45 +10:00
Julian Sommer ba0d8b67ff feat: e2e pipeline speedup (#78) 2026-03-18 13:50:45 +11:00
Ben Duncan 454f381826 Moved to GNU Affero General Public License v3 2026-03-18 02:04:41 +00:00
Ben Duncan b95ff42a2f feat: rename product from Hive to Spinifex
Trademark conflict — bulk rename across entire codebase:
- Go module: github.com/mulgadc/hive → github.com/mulgadc/spinifex
- Binary: hive → spx
- Directories: hive/ → spinifex/, hiveui → spinifexui
- Env vars: HIVE_* → SPINIFEX_*
- Config: hive.toml → spinifex.toml
- Systemd: hive-* → spinifex-*
- NATS: hive-workers → spinifex-workers
- All paths, docs, scripts, CI workflows updated
2026-03-18 01:42:53 +00:00
Ben Duncan 2064c338b3 chore: gofmt formatting after rename 2026-03-18 01:23:11 +00:00
Ben Duncan 4f00cf9e4c refactor: update Dockerfile and build artifacts for spinifex rename
- Dockerfile.distro: update all paths from hive → spinifex
- Binary output name: spx
- Logrotate: logrotate-spinifex
2026-03-18 01:20:27 +00:00
Ben Duncan 48012a4058 refactor: rename all documentation from hive to spinifex
- Rename .claude/rules/hive.md → spinifex.md
- Update CLAUDE.md component description
- Rename doc directories: hive-admin-cli → spinifex-admin-cli, etc.
- Rename doc files: hive-cluster-stats, hive-code-consolidation, etc.
- Bulk update all markdown files across docs/ tree
- Update internal docs: CLUSTER-INSTALL, TESTING, DEVNOTES, BRANCHES
2026-03-18 01:19:05 +00:00
Ben Duncan 55186d95f2 refactor: rename CI/CD workflows from hive to spinifex
- Rename hive.yml → spinifex.yml
- Update workflow names, path triggers, working directories
- Update release.yml tarball references
- Update e2e.yml test paths
- Update ui.yml frontend paths
2026-03-18 01:09:02 +00:00
Ben Duncan c11ac04892 refactor: rename frontend UI branding from hive to spinifex
- package.json: name → spinifex-ui
- index.html: page title → Spinifex Control Panel
- sidebar-layout: Mulga Hive → Mulga Spinifex
- auth.ts: localStorage key → spinifex:v1:aws-credentials
- main.tsx: theme storage key → spinifex-ui-theme
- spinifexui_test.go: update service name assertion
2026-03-18 01:06:55 +00:00
Ben Duncan 9e744e5be6 refactor: rename NATS queue group and topics from hive to spinifex
- hive-workers → spinifex-workers queue group (~16 files)
- hive.admin.* → spinifex.admin.* topics
- hive.nodes.* → spinifex.nodes.* topics
- hive.node.* → spinifex.node.* topics
- hive.cluster.* → spinifex.cluster.* topics
- spinifex-nats server_name already updated in Phase 5
2026-03-18 01:06:03 +00:00
Ben Duncan 5d93786995 refactor: rename hive to spinifex in all shell scripts
- Update scripts/: start-dev, stop-dev, dev-setup, setup, setup-ovn, etc.
- Update tests/e2e/: all E2E test scripts
- Update scripts/iac/: rename hive-test.sh → spinifex-test.sh, update TF files
- Update config/: rename hive.service and install script
- Binary refs use spx, env vars use SPINIFEX_, paths use spinifex
2026-03-18 01:04:12 +00:00
Ben Duncan b97a98e2b7 refactor: rename systemd units and logrotate from hive to spinifex
- Rename all build/systemd/hive-* → spinifex-*
- Rename build/logrotate/hive → spinifex
- Update unit contents: descriptions, dependencies, user/group, paths
- Binary path set to /usr/local/bin/spx
2026-03-18 00:53:34 +00:00
Ben Duncan ee0b92c192 refactor: rename config files, paths, and env vars from hive to spinifex
- Rename hive.toml template → spinifex.toml
- Update all default paths: /etc/hive → /etc/spinifex, ~/hive → ~/spinifex
- Rename all HIVE_* env vars → SPINIFEX_* (SetEnvPrefix, BindEnv calls)
- Update nats.conf template server_name
- Update admin.go init/join commands for new paths and names
- Update paths.go, paths_test.go, config.go, config_test.go
2026-03-18 00:47:01 +00:00
Ben Duncan 00f5b6590e refactor: rename CLI binary to spx and update all command names
- Makefile: GO_PROJECT_NAME → spx, update build/test/dist paths
- root.go: Use → spx, update branding to Spinifex
- service.go: hive/hive-ui commands → spinifex/spinifex-ui
- Service name constants: hive → spinifex, hive-ui → spinifex-ui
- Update service_test.go switch cases
2026-03-18 00:42:34 +00:00
Ben Duncan 06804f5775 refactor: rename directory structure hive → spinifex
- git mv hive/ → spinifex/
- git mv cmd/hive/ → cmd/spinifex/
- git mv services/hive/ → services/spinifex/
- git mv services/hiveui/ → services/spinifexui/
- Update package declarations and all internal import paths
- Update service.go switch cases for new service names
2026-03-18 00:37:30 +00:00
Ben Duncan 767ad0bf5c refactor: rename Go module path from hive to spinifex
Bulk rename github.com/mulgadc/hive → github.com/mulgadc/spinifex
across go.mod and all 178 Go files (~411 import occurrences).
2026-03-18 00:29:24 +00:00
Julian Sommer 4152e4422f chore: remove redundant x/net/http2 and automaxprocs deps
ForceAttemptHTTP2 + NextProtos ALPN already handle HTTP/2 on Go 1.22+.
Go 1.25+ runtime is container-aware for GOMAXPROCS by default.
2026-03-18 10:14:52 +10:00
Julian Sommer 48bbe423df fix: complete termination lifecycle for failed instances 2026-03-18 09:53:48 +10:00
Julian Sommer e6b5b3b135 chore: update vite react plugin 2026-03-18 08:54:57 +10:00
Julian Sommer f526972607 chore: go back to ultracite 2026-03-18 08:46:49 +10:00
Julian Sommer ddcd7f622f chore: bump deps 2026-03-18 08:20:18 +10:00
Ben Duncan 2459e5f7ac improved docs 2026-03-16 12:52:14 +00:00
Ben Duncan 120bfa8953 fix: generate go.work in Dockerfile instead of copying gitignored file 2026-03-16 12:36:56 +00:00
Ben Duncan d8d5fd7d2c New doc links 2026-03-16 12:32:44 +00:00
Ben Duncan b857c7982e Merge remote-tracking branch 'origin/dev' 2026-03-16 12:17:57 +00:00
Ben Duncan 965e65df3b Added test coverage 2026-03-16 12:14:46 +00:00
Ben Duncan 4c4683ba0c installer tweaks for sudo 2026-03-16 12:04:06 +00:00
Ben Duncan a458a28406 new binary mode and systemd process 2026-03-16 11:57:52 +00:00
Ben Duncan 2351d4ff81 New doc templates 2026-03-16 22:47:08 +11:00
Julian Sommer e7a3b5b712 fix: update test for explicit error handling 2026-03-13 16:07:49 +10:00
Julian Sommer 36f820c859 fix: error handling (#76) 2026-03-13 17:01:00 +11:00
Julian Sommer c60a691ac1 fix: start/stop race condition 2026-03-13 05:27:28 +00:00
Julian Sommer 21fd5a7081 fix: IAM replication 2026-03-13 04:33:19 +00:00
Julian Sommer 2b455e365e fix: dropped state 2026-03-13 03:53:38 +00:00
Julian Sommer e6a09d371c fix: multi node e2e retry 2026-03-13 03:37:41 +00:00
Julian Sommer 6f3f348287 chore: bump vite plugin react 2026-03-13 02:52:08 +00:00
Julian Sommer 8e8b17c599 chore: migrate to vite8 2026-03-13 02:39:28 +00:00
Julian Sommer 72aa3e7563 feat: migrate to oxlint (#75) 2026-03-13 13:24:01 +11:00
Julian Sommer 096b3380fc feat: default vpc account scoping (#74) 2026-03-13 13:18:58 +11:00
Julian Sommer 663805cd23 fix: race conditions on node failure 2026-03-13 01:53:00 +00:00
Julian Sommer 380ab2a870 fix: replicate all js buckets 2026-03-13 10:48:00 +10:00
Julian Sommer 98c75cbb83 fix: stop-dev.sh 2026-03-13 09:17:41 +10:00
Julian Sommer 3ffe67271f feat: multi node e2e (#73) 2026-03-13 10:11:26 +11:00
Julian Sommer 4466de04d3 fix: ovn logrotate 2026-03-12 08:24:52 +10:00
Julian Sommer fd91b80851 fix: hardcoded predastore nats url 2026-03-11 13:57:49 +10:00
Julian Sommer 0853eb5068 fix: return error from GenerateNATSToken instead of panicking
Same pattern as H-5 — crypto/rand failure caused a panic. Now returns
(string, error) so callers can handle it gracefully.
2026-03-11 12:08:44 +10:00
Julian Sommer 3d8c9f3812 fix: hive panics (#71) 2026-03-11 13:02:31 +11:00
Julian Sommer 2cac5e2c37 fix: cors 2026-03-11 11:07:44 +10:00
Julian Sommer 0ec13f33eb fix: lint 2026-03-11 10:58:03 +10:00
Julian Sommer 7af16b75b9 feat: 80% real frontend coverage 2026-03-11 10:56:42 +10:00
Julian Sommer 40e09f0886 feat: enforce frontend code coverage 2026-03-11 10:25:04 +10:00
Julian Sommer e26cd7fd82 feat: ui tests 2026-03-11 10:14:31 +10:00
Julian Sommer 8f223d35d3 feat: setup vitest 2026-03-11 09:47:46 +10:00
Julian Sommer b0dbc76b22 chore: frontend deps 2026-03-11 09:32:46 +10:00
Julian Sommer 506bced4fa fix: e2e single 2026-03-11 08:43:05 +10:00
Ben Duncan 438ec3c973 Merge branch 'dev' of github.com:mulgadc/hive into dev 2026-03-10 22:13:11 +10:00
Ben Duncan 11a1f18cb0 removed restrict on hostfwd, vpc issue to ssh in 2026-03-10 22:13:00 +10:00
Julian Sommer 1ff08ab80a fix: seperate system account (#70) 2026-03-10 16:30:34 +11:00
Ben Duncan d17eef2ba9 Merge branch 'dev' of github.com:mulgadc/hive into dev 2026-03-10 10:58:04 +10:00
Ben Duncan b1059899a2 dev networking, remote dhcp, due to double route bug 2026-03-10 10:57:55 +10:00
Julian Sommer 17da8b2eb2 fix: update INSTALL.md 2026-03-10 10:49:29 +10:00
Julian Sommer ee3df227a7 feat: reboot instances (#68) 2026-03-10 11:48:58 +11:00
Julian Sommer 015dbbc272 feat: frontend performance 2026-03-10 10:31:59 +10:00
Julian Sommer 404b3d6246 release: update go.mod for v0.2.3 2026-03-10 09:36:58 +10:00
Julian Sommer 021b0982f4 fix: bump go 1.26.1 references and fix gosec G104 in hiveui 2026-03-10 08:33:04 +10:00
Julian Sommer 7b590957e1 fix: bump go 1.26.1 2026-03-10 08:19:38 +10:00
Julian Sommer c9c3cda651 fix: redirect HTTP to HTTPS on same port for hive-ui
Uses a TLS-detecting listener that peeks the first byte of each
connection. TLS handshakes (0x16) proceed normally; plain HTTP
requests get a 301 redirect to the HTTPS URL. This handles
http://localhost:3000 gracefully instead of showing a cryptic error.
2026-03-10 08:14:19 +10:00
Julian Sommer 1efb4c4a3b fix: config 2026-03-06 14:51:21 +10:00
Julian Sommer 05cb4ac108 feat: frontend refactors (#66) 2026-03-06 15:42:01 +11:00
Julian Sommer 6eac4d6598 feat: IAM Frontend (#65) 2026-03-06 15:14:50 +11:00
Julian Sommer f03a2314c6 feat: terminated instances expiry (#63) 2026-03-06 12:51:27 +11:00
Julian Sommer 2c444b1eba chore: install tsgo 2026-03-06 11:46:21 +10:00
Julian Sommer aea2e2c05a ci: setup go work after 2026-03-06 10:41:45 +10:00
Julian Sommer d90bf53eaf fix: log rotate ovn 2026-03-06 10:10:34 +10:00
Julian Sommer 14c52366da fix: decouple gateway from qmp 2026-03-06 10:06:59 +10:00
Julian Sommer 6737c42601 refactor: consolidate NATS/API types into types/ package (phase 1)
Move EBS, cluster, and node message types from config/ to types/.
Remove duplicate NodeDiscoverResponse from gateway and daemon.
Delete dead EC2StartInstances types and tests from config.
Update all imports across source and test files.
2026-03-06 09:56:04 +10:00
Julian Sommer 19988af768 fix: gofmt formatting 2026-03-06 09:48:16 +10:00
Julian Sommer d20fcd2c38 fix merge 2026-03-06 09:45:34 +10:00
Julian Sommer 1c6950b89e release: update go.mod for v0.2.2 2026-03-06 09:45:34 +10:00
Julian Sommer 94ec56bb59 fix: extract shared type 2026-03-06 09:45:34 +10:00
Julian Sommer 0b2e613c6c fix: diff-coverage.sh 2026-03-06 08:22:30 +10:00
Julian Sommer 2d304528b8 chore: bump deps 2026-03-06 08:15:48 +10:00
Julian Sommer 35f2104325 feat: more tests 2026-03-05 15:09:24 +10:00
Julian Sommer c4027a0380 test: add gateway core routing, IAM auth, and handler test coverage
Squash of dev commits: gateway core routing/policy/middleware tests,
IAM auth and handler tests, awserrors tests, respondWithJSON helper,
utils refactor into focused files, and daemon test speedups.
2026-03-05 14:20:00 +10:00
Julian Sommer 77c6d05042 refactor: replace NYTimes/gziphandler with stdlib compress/gzip
Drop the external gziphandler dependency in hiveui. The replacement
uses a simple http.ResponseWriter wrapper that checks Content-Type
against an allowlist before enabling gzip compression. Same behavior,
zero external deps.
2026-03-05 12:53:46 +10:00
Julian Sommer 8af3d2e5fa feat: migrate off fiber (#62) 2026-03-05 12:53:46 +10:00
Julian Sommer 49c5807ef1 refactor: accounts 2026-03-05 11:41:51 +10:00
Julian Sommer 5fad038b87 feat: IAM account scoping (#61) 2026-03-05 09:06:35 +10:00
Julian Sommer 48c0565d47 fix: ovn shut down on stop-dev.sh 2026-03-03 15:12:44 +10:00
Julian Sommer 371193a832 feat: IAM v1 (#59) 2026-03-03 15:23:19 +11:00
Julian Sommer 9d66ef8233 fix: sanitize error codes to prevent leaking internal errors to clients
Add awserrors.ValidErrorCode() to validate error strings against
ErrorLookup before sending to clients. Replace hardcoded
"ValidationError" strings in utils.go with awserrors constants.
2026-02-27 08:49:18 +10:00
Julian Sommer 992111e809 fix: race condition + frontend refetching 2026-02-27 08:14:50 +10:00
Julian Sommer 12c2e2789e fix: skip diff coverage when instrumentable lines below minimum
The MIN_LOC=100 threshold checked total diff lines, not instrumentable
lines. A commit with many non-instrumentable lines (structs, comments)
but only 1-2 instrumentable lines would unfairly fail at 0% coverage.
Add a minimum instrumentable lines threshold (5) so the check skips
when the sample size is too small to be meaningful.
2026-02-26 11:35:22 +10:00
Julian Sommer 3720a5ba90 feat: blockdevicemappings fix (#55) 2026-02-26 11:47:37 +11:00
Julian Sommer de0e049b16 fix: e2e console output 2026-02-26 08:26:34 +10:00
Julian Sommer 6e3fe148af feat: frontend features 2026-02-26 08:22:55 +10:00
Julian Sommer 81b990a238 fix: e2e verify errored instance didnt restart on launch 2026-02-25 11:33:44 +10:00
Julian Sommer 52f2498699 fix: proper recovery checks e2e 2026-02-25 11:16:07 +10:00
Julian Sommer 61e8eb9d91 fix: reset LaunchTime on instance relaunch to prevent watchdog termination
During cluster restart, restoreInstances resets running instances (with
dead QEMU) to pending for relaunch, but didn't reset LaunchTime. The
pending watchdog (5min timeout) saw the stale LaunchTime from the
original launch and immediately marked instances as failed/terminated,
preventing any relaunch from succeeding.
2026-02-25 10:48:16 +10:00
Julian Sommer 1453beca6b Merge remote-tracking branch 'origin/main' into dev 2026-02-25 10:45:45 +10:00
Julian Sommer d0056ee925 fix: daemon-level retry with backoff for NATS/JetStream startup (#52) 2026-02-25 11:43:10 +11:00
Julian Sommer 55486490aa fix: kv stream recovery (#54) 2026-02-25 11:41:28 +11:00
Julian Sommer b370f6336e docs 2026-02-25 10:01:12 +10:00
Julian Sommer 1910347cce fix: make building on start-dev.sh opt in 2026-02-25 09:56:13 +10:00
dependabot[bot] e08b8140a6 chore: bump github.com/gofiber/fiber/v2 from 2.52.11 to 2.52.12 (#51) 2026-02-25 10:36:09 +11:00
dependabot[bot] a7e0cb6972 chore: bump github.com/nats-io/nats-server/v2 (#50) 2026-02-25 10:35:49 +11:00
Julian Sommer 6b9fc7b600 refactor: daemon 2026-02-25 08:22:46 +10:00
Julian Sommer c2e66e1ab4 refactor: extract respondWithError; fix nil panic in RunInstance
Extract duplicated respondWithError closures into a single package-level
function. Guard against nil ImageId in handleEC2RunInstances to prevent
panic under race conditions. Fix diff-coverage base ref for main branch.
2026-02-24 15:02:57 +10:00
Julian Sommer deae613a75 refactor: split daemon_handlers.go into per-domain files
Split the monolithic daemon_handlers.go into focused files:
  - daemon_handlers_account.go
  - daemon_handlers_console.go
  - daemon_handlers_igw.go
  - daemon_handlers_image.go
  - daemon_handlers_instance.go
  - daemon_handlers_key.go
  - daemon_handlers_snapshot.go
  - daemon_handlers_tag.go
  - daemon_handlers_volume.go
  - daemon_handlers_vpc.go

Add tests for ModifyVolume success, TerminateStoppedInstance with volumes,
DescribeInstanceTypes capacity filter, StartStoppedInstance type-not-available,
AttachVolume/DetachVolume validation paths, RunInstances error paths, and
DescribeInstances malformed ID.

Fix diff-coverage.sh to use git rename/copy detection (-M -C) so file
splits are not incorrectly flagged as all-new code.
2026-02-24 14:28:24 +10:00
Julian Sommer 8188707c91 feat: VPC networking v1 — OVN-based virtual networking for EC2 instances 2026-02-24 14:13:57 +10:00
Julian Sommer b9c528ce4c fix frontend vuln 2026-02-24 11:56:17 +10:00
Julian Sommer 5992d3f056 test: general test coverage improvements across hive packages
Coverage improvements:
- awsec2query: parser byte slice, invalid input, locationName fallback (73% → 83%)
- config: EC2StartInstancesResponse.Respond NATS round-trip, NodeBaseDir, ShardWAL
- daemon: 14 NATS delegate handlers, device allocation, canAllocate (count=0 boundary),
  resource management, shutdown ACK/progress, JetStream WriteServiceManifest,
  fix smallestAllocType to score by both vCPU and memory
- gateway: GenerateEC2ErrorResponse XML validation (fixed no-op assertion),
  DiscoverActiveNodes (5 scenarios), ParseAWSQueryArgs, ParseArgsToStruct (41.6% → 52.1%)
- utils: ConnectNATS (success, token, wrong token, bad address), NATSRequest
- vm: ResetNodeLocalState, PIDFile/QMP, NoGraphic, Serial+Console partial configs,
  NetDevs, ARM64, KVM, full config (68.1% → 91.7%)
2026-02-20 15:09:33 +10:00
Julian Sommer 6bc36455fc test coverage improvements: instance, hiveui, health, utils
Total hive coverage: 54.3% → 55.8% (+1.5%)

- instance/service_impl.go: new test file (generateHostname, RunInstance,
  cloud-init template, constructor)
- hiveui.go: new test file (security headers, gzip middleware, New(),
  Status(), Reload())
- daemon/health.go: extend existing tests (handleInstanceCrash core flow,
  first crash time, unknown instance type, maybeRestart scheduling)
- utils/utils.go: extend existing tests (pidPath all branches, RuntimeDir,
  extractDiskImagePath, WritePidFile error path)
- fix: use smallestAllocType in health tests to avoid CI resource failures
2026-02-20 13:46:31 +10:00
Julian Sommer db45c028be test: e2e + unit coverage for ModifyInstanceAttribute and key service
- Add Phase 7b to single-node E2E: stop instance, modify type (nano →
  xlarge), start, SSH in, verify vCPU count (nproc) and memory
  (/proc/meminfo) match the new instance type
- Add Phase 8q negative test: modify-instance-attribute on running
  instance expects InvalidInstanceID.NotFound
- Improve key service integration test coverage (mulga-580)
- Pin viperblock/predastore CI checkout to main (fixes feature branch CI)
- Update TEST_COVERAGE.md
2026-02-20 12:01:03 +10:00
Julian Sommer ff6ca8a057 feat: modify-instance-attribute (#48) 2026-02-20 12:19:20 +11:00
Julian Sommer df0e8a6459 ci: add race detection, fix test race conditions 2026-02-20 09:40:02 +10:00
Julian Sommer 3dc3d41d79 ci: enforce code coverage 2026-02-20 08:58:44 +10:00
Julian Sommer eba702b1f2 test: add 10 viperblockd coverage tests, fix daemon test sleep
viperblockd: add snapshot handler, sync/unmount/delete with VB instance,
invalid JSON, dual-publish, generic topic routing, and auxiliary volume
suffix tests. Coverage 43.1% → 53.6%.

daemon: write clean shutdown marker in restore tests to skip the 3s
crash-recovery sleep (no QEMU runs in tests). Suite 36s → 6s.
2026-02-20 08:42:30 +10:00
Julian Sommer e6ffecafca gofix: exclude stringsbuilder analyzer from modernize checks 2026-02-19 15:15:24 +10:00
Julian Sommer 27a27f660a feat: add go fix check in ci 2026-02-19 14:43:59 +10:00
Julian Sommer b4dcd3ead9 gofix: modernize codebase with go fix analyzers
Apply safe modernizations across the codebase:
- minmax: replace if/else clamping with min/max builtins
- fmtappendf: replace []byte(fmt.Sprintf) with fmt.Appendf
- forvar: remove redundant loop variable re-declarations
- inline: rename reflect.Ptr to reflect.Pointer
- mapsloop: replace explicit map loops with maps package
- omitzero: remove ineffective omitempty on time.Time fields
- rangeint: replace 3-clause for loops with range-over-int
- slicescontains: replace manual loops with slices.Contains
2026-02-19 14:40:16 +10:00
Julian Sommer 43acb57b9f feat: add e2e tests for serial console access and error compliance 2026-02-19 14:23:55 +10:00
Julian Sommer 17c011e9db fix 2026-02-19 13:17:20 +10:00
Julian Sommer a330e35d5f feat: account settings console access (#47) 2026-02-19 13:05:11 +10:00
Julian Sommer a5ff1ca5a7 feat: more awserrors (#46) 2026-02-19 12:15:45 +11:00
Julian Sommer 8573800914 fix: move instances with unavailable type to stopped on recovery (#45) 2026-02-19 10:31:13 +11:00
Julian Sommer cefbcea485 fix: README.md 2026-02-19 08:30:25 +10:00
Julian Sommer 1bf4a9cfab fix: ci branches 2026-02-18 15:05:28 +10:00
1032 changed files with 138749 additions and 28511 deletions
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -e
branch=$(git rev-parse --abbrev-ref HEAD)
# Only enforce preflight on dev and main
if [ "$branch" != "dev" ] && [ "$branch" != "main" ]; then
exit 0
fi
# Only run preflight if Go files were staged
if git diff --cached --name-only | grep -q '\.go$'; then
echo "Go files staged on $branch — running make preflight..."
make preflight
else
echo "No Go files staged, skipping preflight."
fi
+225
View File
@@ -0,0 +1,225 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#161b22"/>
<stop offset="45%" style="stop-color:#1f2028"/>
<stop offset="100%" style="stop-color:#161b22"/>
</linearGradient>
<linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262633"/>
<stop offset="100%" style="stop-color:#7F1D1D14"/>
</linearGradient>
<linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#F87171"/>
<stop offset="50%" style="stop-color:#DC2626"/>
<stop offset="100%" style="stop-color:#7F1D1D"/>
</linearGradient>
</defs>
<rect width="900" height="620" fill="url(#bg)" rx="12"/>
<!-- Title bar -->
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
<text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
<!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
<rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
<text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
<text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
<g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
<rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
<rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
<rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
<rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
<rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
<rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
</g>
<!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
<line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
<text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
<!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
<rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
<text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
<text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
<g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
<rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
<rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
<rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
<rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
<rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
<rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
</g>
<!-- ═══ NATS BUS ═══════════════════════════════════════ -->
<line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
<text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
<text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.* ebs.* s3.* vpc.* iam.*</text>
<text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
<!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
<g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
<line x1="100" y1="306" x2="100" y2="328"/>
<line x1="275" y1="306" x2="275" y2="328"/>
<line x1="450" y1="306" x2="450" y2="328"/>
<line x1="625" y1="306" x2="625" y2="328"/>
<line x1="800" y1="306" x2="800" y2="328"/>
</g>
<!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
<!-- EC2 -->
<g transform="translate(28, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
</g>
<!-- EBS -->
<g transform="translate(204, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
</g>
<!-- S3 -->
<g transform="translate(380, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
</g>
<!-- VPC -->
<g transform="translate(556, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
</g>
<!-- IAM -->
<g transform="translate(732, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
</g>
<!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
<g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
<line x1="100" y1="410" x2="100" y2="440"/>
<line x1="276" y1="410" x2="276" y2="440"/>
<line x1="452" y1="410" x2="452" y2="440"/>
<line x1="628" y1="410" x2="628" y2="440"/>
<line x1="804" y1="410" x2="804" y2="440"/>
</g>
<!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
<rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
<text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
<!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
<!-- QEMU/KVM -->
<g transform="translate(28, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
</g>
<!-- Viperblock -->
<g transform="translate(204, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
</g>
<!-- Predastore -->
<g transform="translate(380, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
</g>
<!-- OVN/OVS -->
<g transform="translate(556, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
</g>
<!-- IAM store -->
<g transform="translate(732, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
</g>
<!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
<g transform="translate(16, 558)">
<rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
<text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
<tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
<tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
</text>
<text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 17 KiB

+154
View File
@@ -0,0 +1,154 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="300" viewBox="0 0 900 300">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a;stop-opacity:1" />
<stop offset="50%" style="stop-color:#0d1117;stop-opacity:1" />
<stop offset="100%" style="stop-color:#0a0a0a;stop-opacity:1" />
</linearGradient>
<radialGradient id="vignette" cx="50%" cy="50%" r="70%">
<stop offset="0%" style="stop-color:#7F1D1D;stop-opacity:0.08"/>
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0"/>
</radialGradient>
<linearGradient id="mulga-grad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.95" />
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.95" />
</linearGradient>
<linearGradient id="rule" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC2626;stop-opacity:0" />
<stop offset="50%" style="stop-color:#F87171;stop-opacity:0.45" />
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0" />
</linearGradient>
<filter id="node-glow">
<feGaussianBlur in="SourceGraphic" stdDeviation="3" />
</filter>
</defs>
<!-- Background -->
<rect width="900" height="300" fill="url(#bg)" rx="12"/>
<rect width="900" height="300" fill="url(#vignette)" rx="12"/>
<!-- Mesh network background (mulgadc style) -->
<g opacity="0.35">
<!-- Connections (drawn first, under nodes) -->
<g stroke="#7F1D1D" stroke-width="0.6" fill="none">
<line x1="80" y1="60" x2="200" y2="40"/>
<line x1="80" y1="60" x2="150" y2="140"/>
<line x1="200" y1="40" x2="330" y2="90"/>
<line x1="150" y1="140" x2="260" y2="180"/>
<line x1="150" y1="140" x2="60" y2="220"/>
<line x1="330" y1="90" x2="260" y2="180"/>
<line x1="260" y1="180" x2="120" y2="260"/>
<line x1="60" y1="220" x2="120" y2="260"/>
<line x1="620" y1="50" x2="760" y2="90"/>
<line x1="620" y1="50" x2="700" y2="160"/>
<line x1="760" y1="90" x2="850" y2="160"/>
<line x1="700" y1="160" x2="820" y2="220"/>
<line x1="850" y1="160" x2="820" y2="220"/>
<line x1="700" y1="160" x2="600" y2="240"/>
<line x1="600" y1="240" x2="820" y2="220"/>
<line x1="760" y1="90" x2="700" y2="160"/>
</g>
<!-- Nodes -->
<g fill="#DC2626">
<circle cx="80" cy="60" r="2.5"/>
<circle cx="200" cy="40" r="2"/>
<circle cx="330" cy="90" r="2.5"/>
<circle cx="150" cy="140" r="3"/>
<circle cx="260" cy="180" r="2"/>
<circle cx="60" cy="220" r="2"/>
<circle cx="120" cy="260" r="2.5"/>
<circle cx="620" cy="50" r="2"/>
<circle cx="760" cy="90" r="3"/>
<circle cx="700" cy="160" r="2.5"/>
<circle cx="850" cy="160" r="2"/>
<circle cx="820" cy="220" r="2.5"/>
<circle cx="600" cy="240" r="2"/>
</g>
<!-- Pulse rings (static "about-to-pulse" look) -->
<g fill="none" stroke="#F87171" stroke-width="0.8" opacity="0.5">
<circle cx="150" cy="140" r="6"/>
<circle cx="760" cy="90" r="6"/>
<circle cx="700" cy="160" r="8"/>
</g>
</g>
<!-- Main emblem: cluster-node stack with NATS bus -->
<g transform="translate(200, 150)">
<!-- Outer glow ring -->
<circle cx="0" cy="0" r="64" fill="none" stroke="url(#mulga-grad)" stroke-width="1.5" opacity="0.35" filter="url(#node-glow)"/>
<!-- Top node -->
<rect x="-24" y="-52" width="48" height="22" rx="3" fill="none" stroke="#DC2626" stroke-width="1.8"/>
<circle cx="-16" cy="-41" r="1.6" fill="#F87171"/>
<circle cx="-8" cy="-41" r="1.6" fill="#F87171"/>
<line x1="0" y1="-41" x2="16" y2="-41" stroke="#F87171" stroke-width="0.9" opacity="0.6"/>
<!-- Middle node (active) -->
<rect x="-30" y="-14" width="60" height="28" rx="3" fill="url(#mulga-grad)" opacity="0.22" stroke="#DC2626" stroke-width="2"/>
<circle cx="-20" cy="0" r="2" fill="#F87171"/>
<circle cx="-10" cy="0" r="2" fill="#F87171"/>
<line x1="-2" y1="0" x2="22" y2="0" stroke="#F87171" stroke-width="1.2" opacity="0.8"/>
<!-- Bottom node -->
<rect x="-24" y="30" width="48" height="22" rx="3" fill="none" stroke="#7F1D1D" stroke-width="1.8"/>
<circle cx="-16" cy="41" r="1.6" fill="#DC2626"/>
<circle cx="-8" cy="41" r="1.6" fill="#DC2626"/>
<line x1="0" y1="41" x2="16" y2="41" stroke="#DC2626" stroke-width="0.9" opacity="0.6"/>
<!-- Radiating mesh ports -->
<g stroke-width="0.9" opacity="0.55">
<line x1="-30" y1="0" x2="-58" y2="-26" stroke="#DC2626"/>
<line x1="-30" y1="0" x2="-64" y2="0" stroke="#DC2626"/>
<line x1="-30" y1="0" x2="-58" y2="26" stroke="#DC2626"/>
<line x1="30" y1="0" x2="58" y2="-26" stroke="#7F1D1D"/>
<line x1="30" y1="0" x2="64" y2="0" stroke="#7F1D1D"/>
<line x1="30" y1="0" x2="58" y2="26" stroke="#7F1D1D"/>
</g>
<g fill="#F87171">
<circle cx="-58" cy="-26" r="2.5"/>
<circle cx="-64" cy="0" r="2.5"/>
<circle cx="-58" cy="26" r="2.5"/>
</g>
<g fill="#DC2626">
<circle cx="58" cy="-26" r="2.5"/>
<circle cx="64" cy="0" r="2.5"/>
<circle cx="58" cy="26" r="2.5"/>
</g>
</g>
<!-- Title -->
<text x="340" y="120" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="56" font-weight="700" fill="#f5f5f5" letter-spacing="-1.5">Spinifex</text>
<!-- Subtitle -->
<text x="340" y="155" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">Open-source AWS-compatible platform</text>
<text x="340" y="180" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">for secure edge deployments</text>
<!-- Rule -->
<rect x="340" y="200" width="440" height="1" fill="url(#rule)" rx="1"/>
<!-- Service pills -->
<g font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" font-weight="500">
<rect x="340" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="365" y="231" fill="#F87171" text-anchor="middle">EC2</text>
<rect x="398" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="423" y="231" fill="#F87171" text-anchor="middle">EBS</text>
<rect x="456" y="215" width="45" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="478" y="231" fill="#F87171" text-anchor="middle">S3</text>
<rect x="509" y="215" width="55" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="536" y="231" fill="#F87171" text-anchor="middle">VPC</text>
<rect x="572" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="597" y="231" fill="#F87171" text-anchor="middle">IAM</text>
<rect x="630" y="215" width="78" height="24" rx="12" fill="#18181b" stroke="#52525b" stroke-width="0.5" stroke-opacity="0.8"/>
<text x="669" y="231" fill="#a1a1aa" text-anchor="middle">Air-Gapped</text>
</g>
<!-- Tagline -->
<text x="340" y="265" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" fill="#71717a" letter-spacing="1.8">YOUR INFRASTRUCTURE · YOUR HARDWARE · YOUR CLOUD</text>
</svg>

After

Width:  |  Height:  |  Size: 7.5 KiB

+175
View File
@@ -0,0 +1,175 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="720" viewBox="0 0 900 720">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#161b22"/>
<stop offset="45%" style="stop-color:#1f2028"/>
<stop offset="100%" style="stop-color:#161b22"/>
</linearGradient>
<linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262633"/>
<stop offset="100%" style="stop-color:#7F1D1D14"/>
</linearGradient>
<linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#F87171"/>
<stop offset="50%" style="stop-color:#DC2626"/>
<stop offset="100%" style="stop-color:#7F1D1D"/>
</linearGradient>
<linearGradient id="daemon-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262622"/>
<stop offset="100%" style="stop-color:#7F1D1D0a"/>
</linearGradient>
</defs>
<rect width="900" height="720" fill="url(#bg)" rx="12"/>
<!-- Title -->
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">HIGH-LEVEL ARCHITECTURE</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">SDK → Gateway → NATS → Daemons → VMs</text>
<!-- ═══ AWS SDK / CLI ══════════════════════════════════ -->
<rect x="16" y="50" width="868" height="76" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
<text x="32" y="72" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="1">AWS SDK / CLI</text>
<text x="32" y="90" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Users interact with Spinifex using standard AWS SDKs and CLIs against a custom endpoint URL</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
<rect x="32" y="98" width="480" height="20" rx="4" fill="#0a0a0a" stroke="#3fb950" stroke-width="0.6" stroke-opacity="0.4"/>
<text x="42" y="112" fill="#7ee787">$ aws ec2 run-instances --endpoint-url https://localhost:9999 ...</text>
</g>
<!-- Connector: HTTPS + SigV4 -->
<line x1="450" y1="126" x2="450" y2="154" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="380" y="128" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
<text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">HTTPS · SigV4 Auth</text>
<!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
<rect x="16" y="154" width="868" height="104" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
<text x="32" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY · port 9999</text>
<text x="876" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a" text-anchor="end">spinifex/services/awsgw/awsgw.go</text>
<!-- Two-column feature bullets -->
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="40" y="200">● TLS termination</text>
<text x="40" y="220">● EC2 Query Protocol parsing</text>
<text x="40" y="240">● XML response generation</text>
<text x="460" y="200">● AWS SigV4 authentication</text>
<text x="460" y="220">● Routes to service handlers</text>
<text x="460" y="240">● NATS client connection</text>
</g>
<!-- Connector: JSON over NATS -->
<line x1="450" y1="258" x2="450" y2="286" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="380" y="260" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
<text x="450" y="274" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">JSON over NATS</text>
<!-- ═══ NATS BROKER ═══════════════════════════════════ -->
<rect x="16" y="286" width="868" height="124" rx="8" fill="#1a0a0a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<rect x="16" y="286" width="868" height="28" rx="8" fill="url(#nats-grad)" opacity="0.9"/>
<text x="32" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS MESSAGE BROKER</text>
<text x="876" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">request/reply · queue groups · fan-out</text>
<!-- Two-column topics/queue groups -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="40" y="332" fill="#F87171" font-weight="600">Topics</text>
<text x="40" y="350" fill="#f5f5f5">ec2.RunInstances</text>
<text x="40" y="368" fill="#f5f5f5">ec2.DescribeInstances</text>
<text x="40" y="386" fill="#f5f5f5">ec2.cmd.{instanceID}</text>
<text x="40" y="404" fill="#f5f5f5">ebs.mount · ebs.unmount</text>
<text x="460" y="332" fill="#F87171" font-weight="600">Delivery mode</text>
<text x="460" y="350" fill="#a1a1aa">spinifex-workers <tspan fill="#71717a">(load-balanced)</tspan></text>
<text x="460" y="368" fill="#a1a1aa">no queue group <tspan fill="#71717a">(fan-out to all nodes)</tspan></text>
<text x="460" y="386" fill="#a1a1aa">per-instance <tspan fill="#71717a">(start / stop / terminate)</tspan></text>
<text x="460" y="404" fill="#a1a1aa">Viperblock request/reply</text>
</g>
<!-- Fan-out connectors to 3 daemons -->
<g stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6" fill="none">
<line x1="450" y1="410" x2="450" y2="430"/>
<line x1="450" y1="430" x2="155" y2="452"/>
<line x1="450" y1="430" x2="450" y2="452"/>
<line x1="450" y1="430" x2="745" y2="452"/>
</g>
<!-- ═══ DAEMON ROW ════════════════════════════════════ -->
<!-- Daemon 1 -->
<g transform="translate(32, 452)">
<rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
<text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 1</text>
<line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="20" y="52">● NATS subscriber</text>
<text x="20" y="70">● Resource manager</text>
<text x="20" y="88">● QEMU/KVM launcher</text>
<text x="20" y="106">● QMP monitoring</text>
</g>
<text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
</g>
<!-- Daemon 2 -->
<g transform="translate(327, 452)">
<rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
<text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 2</text>
<line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="20" y="52">● NATS subscriber</text>
<text x="20" y="70">● Resource manager</text>
<text x="20" y="88">● QEMU/KVM launcher</text>
<text x="20" y="106">● QMP monitoring</text>
</g>
<text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
</g>
<!-- Daemon N -->
<g transform="translate(622, 452)">
<rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
<text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node N</text>
<line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="20" y="52">● NATS subscriber</text>
<text x="20" y="70">● Resource manager</text>
<text x="20" y="88">● QEMU/KVM launcher</text>
<text x="20" y="106">● QMP monitoring</text>
</g>
<text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
</g>
<!-- Daemon → VM connectors -->
<g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
<line x1="155" y1="576" x2="155" y2="610"/>
<line x1="450" y1="576" x2="450" y2="610"/>
<line x1="745" y1="576" x2="745" y2="610"/>
</g>
<!-- ═══ VM BAR + ROW ═══════════════════════════════════ -->
<rect x="16" y="610" width="868" height="22" rx="5" fill="url(#bar-grad)"/>
<text x="450" y="625" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">WORKLOADS · QEMU/KVM virtual machines per node</text>
<!-- VM boxes -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="32" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="155" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
<text x="155" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
<text x="155" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 1 workloads</text>
<rect x="327" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="450" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
<text x="450" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
<text x="450" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 2 workloads</text>
<rect x="622" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="745" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
<text x="745" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
<text x="745" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node N workloads</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 12 KiB

+46
View File
@@ -0,0 +1,46 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="180" fill="url(#bg)" rx="10"/>
<text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">BASTION · only entry point into a private subnet</text>
<!-- WAN -->
<g transform="translate(40, 56)">
<rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
<text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">operator / laptop</text>
</g>
<!-- Arrow: SSH -->
<line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
<rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">SSH</text>
<!-- Bastion -->
<g transform="translate(338, 56)">
<rect width="220" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.3" stroke-opacity="0.6"/>
<text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Bastion</text>
<text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">public subnet</text>
</g>
<!-- Arrow: SSH (private IP) -->
<line x1="558" y1="86" x2="638" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="634,82 646,86 634,90" fill="#a855f7" opacity="0.8"/>
<rect x="552" y="62" width="140" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="622" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">SSH (private IP)</text>
<!-- App server -->
<g transform="translate(656, 56)">
<rect width="220" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
<text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">App Server</text>
<text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">private subnet · no internet</text>
</g>
<text x="450" y="156" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">App server has no route to the internet — all access hops through the bastion.</text>
</svg>

After

Width:  |  Height:  |  Size: 3.1 KiB

+98
View File
@@ -0,0 +1,98 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="480" viewBox="0 0 900 480">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="inet-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb9501f"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f71f"/>
<stop offset="100%" style="stop-color:#a855f708"/>
</linearGradient>
</defs>
<rect width="900" height="480" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · multi-AZ</text>
<!-- Internet -->
<rect x="350" y="54" width="200" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Internet</text>
<!-- connector to IGW -->
<line x1="450" y1="94" x2="450" y2="116" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="446,112 454,112 450,120" fill="#71717a" opacity="0.7"/>
<!-- IGW -->
<rect x="380" y="122" width="140" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">IGW</text>
<text x="450" y="158" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">nginx-alb-igw</text>
<!-- Fan-out to two public subnets -->
<line x1="450" y1="166" x2="450" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="230" y1="184" x2="670" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="230" y1="184" x2="230" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="670" y1="184" x2="670" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<!-- public_a -->
<g transform="translate(108, 202)">
<rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_a</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
<rect x="20" y="70" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="85" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">NAT GW</text>
</g>
<!-- public_b -->
<g transform="translate(548, 202)">
<rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_b</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="56" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="71" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
</g>
<!-- SNAT arrow from public_a NAT → private row -->
<line x1="230" y1="306" x2="230" y2="340" stroke="#F87171" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.65"/>
<polygon points="226,336 234,336 230,344" fill="#F87171" opacity="0.8"/>
<rect x="256" y="314" width="240" height="20" rx="10" fill="#0a0a0a" stroke="#F87171" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="376" y="328" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">SNAT for private subnets (egress)</text>
<!-- private_a -->
<g transform="translate(108, 350)">
<rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_a</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-a</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_1</text>
<text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
</g>
<!-- private_b -->
<g transform="translate(548, 350)">
<rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_b</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-b</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_2</text>
<text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 7.1 KiB

+44
View File
@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="180" fill="url(#bg)" rx="10"/>
<text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">S3 WEBAPP · Flask on EC2 with Predastore backend</text>
<!-- Browser -->
<g transform="translate(40, 56)">
<rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Browser</text>
<text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">end user</text>
</g>
<!-- Arrow: HTTP -->
<line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
<rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">HTTP</text>
<!-- EC2 Instance -->
<g transform="translate(338, 56)">
<rect width="240" height="60" rx="8" fill="#DC26261a" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.65"/>
<text x="120" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">EC2 Instance</text>
<text x="120" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">Flask webapp · port 80</text>
</g>
<!-- Arrow: S3 API (boto3) -->
<line x1="578" y1="86" x2="658" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="654,82 666,86 654,90" fill="#a855f7" opacity="0.8"/>
<rect x="570" y="62" width="144" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="642" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">S3 API (boto3)</text>
<!-- Predastore -->
<g transform="translate(676, 56)">
<rect width="200" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
<text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">Predastore</text>
<text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">S3 store · port 8443</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.9 KiB

@@ -0,0 +1,78 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="330" viewBox="0 0 900 330">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="cluster-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="pool-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262622"/>
<stop offset="100%" style="stop-color:#7F1D1D08"/>
</linearGradient>
<linearGradient id="node-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#58a6ff22"/>
<stop offset="100%" style="stop-color:#58a6ff08"/>
</linearGradient>
</defs>
<rect width="900" height="330" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">CONFIGURATION LEVELS · spinifex.toml</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">three scopes: cluster · pool · node</text>
<!-- File badge -->
<rect x="24" y="50" width="140" height="24" rx="6" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
<text x="94" y="67" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#e4e4e7" text-anchor="middle">spinifex.toml</text>
<!-- spine -->
<line x1="52" y1="74" x2="52" y2="298" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<!-- ═══ Level 1: [network] cluster-wide ═══════════════════ -->
<line x1="52" y1="108" x2="92" y2="108" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<rect x="92" y="86" width="764" height="56" rx="8" fill="url(#cluster-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="108" y="108" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">[network]</text>
<text x="220" y="108" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Cluster-wide · which mode?</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="108" y="128" fill="#a1a1aa">└─</text>
<text x="128" y="128" fill="#7ee787">external_mode</text>
<text x="224" y="128" fill="#a1a1aa">=</text>
<text x="238" y="128" fill="#f5f5f5">"pool"</text>
</g>
<!-- ═══ Level 2: [[network.external_pools]] ═══════════════ -->
<line x1="52" y1="176" x2="92" y2="176" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<rect x="92" y="154" width="764" height="78" rx="8" fill="url(#pool-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="108" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">[[network.external_pools]]</text>
<text x="328" y="176" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">IP ranges · what IPs to use?</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="108" y="196" fill="#a1a1aa">├─</text>
<text x="128" y="196" fill="#FCA5A5">name</text>
<text x="158" y="196" fill="#71717a">·</text>
<text x="166" y="196" fill="#FCA5A5">range_start</text>
<text x="246" y="196" fill="#71717a">·</text>
<text x="254" y="196" fill="#FCA5A5">range_end</text>
<text x="108" y="212" fill="#a1a1aa">├─</text>
<text x="128" y="212" fill="#FCA5A5">gateway</text>
<text x="182" y="212" fill="#71717a">·</text>
<text x="190" y="212" fill="#FCA5A5">prefix_len</text>
<text x="108" y="228" fill="#a1a1aa">└─</text>
<text x="128" y="228" fill="#FCA5A5">region</text>
<text x="174" y="228" fill="#71717a">·</text>
<text x="182" y="228" fill="#FCA5A5">az</text>
<text x="202" y="228" fill="#71717a">(optional)</text>
</g>
<!-- ═══ Level 3: [nodes.NAME.vpcd] ════════════════════════ -->
<line x1="52" y1="268" x2="92" y2="268" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<rect x="92" y="246" width="764" height="56" rx="8" fill="url(#node-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="108" y="268" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700">[nodes.NAME.vpcd]</text>
<text x="266" y="268" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Per-node · which NIC?</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="108" y="288" fill="#a1a1aa">└─</text>
<text x="128" y="288" fill="#8ab4f8">external_interface</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.8 KiB

@@ -0,0 +1,63 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="110" fill="url(#bg)" rx="10"/>
<text x="24" y="22" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DATA PATH · VM to WAN via OVN localnet</text>
<!-- VM TAP -->
<g transform="translate(24, 40)">
<rect width="118" height="48" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="59" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">VM TAP</text>
<text x="59" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">guest interface</text>
</g>
<polygon points="148,60 160,64 148,68" fill="#71717a" opacity="0.7"/>
<line x1="142" y1="64" x2="148" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- br-int -->
<g transform="translate(166, 40)">
<rect width="102" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="51" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-int</text>
<text x="51" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
</g>
<polygon points="274,60 286,64 274,68" fill="#71717a" opacity="0.7"/>
<line x1="268" y1="64" x2="274" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- OVN logical pipeline -->
<g transform="translate(292, 40)">
<rect width="192" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="96" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN logical pipeline</text>
<text x="96" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">switch → router → SNAT/DNAT</text>
</g>
<polygon points="490,60 502,64 490,68" fill="#71717a" opacity="0.7"/>
<line x1="484" y1="64" x2="490" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- localnet -->
<g transform="translate(508, 40)">
<rect width="120" height="48" rx="7" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
<text x="60" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">localnet</text>
<text x="60" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">OVN port type</text>
</g>
<polygon points="634,60 646,64 634,68" fill="#71717a" opacity="0.7"/>
<line x1="628" y1="64" x2="634" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- br-wan -->
<g transform="translate(652, 40)">
<rect width="110" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="55" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
<text x="55" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
</g>
<polygon points="768,60 780,64 768,68" fill="#71717a" opacity="0.7"/>
<line x1="762" y1="64" x2="768" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- WAN -->
<g transform="translate(786, 40)">
<rect width="90" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="45" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
<text x="45" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">uplink</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.1 KiB

@@ -0,0 +1,74 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="360" viewBox="0 0 900 360">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="host-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#58a6ff22"/>
<stop offset="100%" style="stop-color:#58a6ff08"/>
</linearGradient>
<linearGradient id="vm-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
</defs>
<rect width="900" height="360" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">TWO DHCP CONVERSATIONS · completely independent</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">pool mode · static or dhcp source</text>
<!-- ═══ HOST ↔ ROUTER DHCP ══════════════════════════════ -->
<rect x="24" y="54" width="852" height="138" rx="10" fill="url(#host-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="44" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700" letter-spacing="0.8">HOST ↔ ROUTER DHCP</text>
<text x="876" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="end">on br-ext · veth pair → br-wan</text>
<line x1="24" y1="88" x2="876" y2="88" stroke="#58a6ff" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<!-- step 1 -->
<circle cx="54" cy="108" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
<text x="54" y="112" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">1</text>
<text x="74" y="112" fill="#f5f5f5">Spinifex asks router:</text>
<text x="228" y="112" fill="#a1a1aa">"give me a public IP for this VM"</text>
<!-- step 2 -->
<circle cx="54" cy="132" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
<text x="54" y="136" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">2</text>
<text x="74" y="136" fill="#f5f5f5">Router responds:</text>
<text x="196" y="136" fill="#8ab4f8" font-weight="600">192.168.1.75</text>
<!-- step 3 -->
<circle cx="54" cy="156" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
<text x="54" y="160" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">3</text>
<text x="74" y="160" fill="#f5f5f5">Spinifex stores IP, tells OVN:</text>
<text x="274" y="160" fill="#F87171" font-weight="600">dnat_and_snat(192.168.1.75 ↔ 172.31.0.4)</text>
</g>
<text x="44" y="181" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">Only used in DHCP source mode. Static mode picks from range.</text>
<!-- separator -->
<line x1="24" y1="206" x2="876" y2="206" stroke="#27272a" stroke-width="0.8"/>
<text x="450" y="220" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#52525b" text-anchor="middle" letter-spacing="2">INDEPENDENT · no packets cross between</text>
<!-- ═══ VM ↔ OVN DHCP ═══════════════════════════════════ -->
<rect x="24" y="234" width="852" height="110" rx="10" fill="url(#vm-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="44" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="0.8">VM ↔ OVN DHCP</text>
<text x="876" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end">on tap → br-int → OVN logical switch</text>
<line x1="24" y1="268" x2="876" y2="268" stroke="#3fb950" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<circle cx="54" cy="288" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
<text x="54" y="292" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">1</text>
<text x="74" y="292" fill="#f5f5f5">cloud-init asks:</text>
<text x="190" y="292" fill="#a1a1aa">"what's my IP?"</text>
<circle cx="54" cy="312" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
<text x="54" y="316" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">2</text>
<text x="74" y="316" fill="#f5f5f5">OVN DHCP responds:</text>
<text x="226" y="316" fill="#a1a1aa">"you are</text>
<text x="284" y="316" fill="#7ee787" font-weight="600">172.31.0.4</text>
<text x="356" y="316" fill="#a1a1aa">"</text>
</g>
<text x="44" y="336" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">VM never sees 192.168.1.x or the router.</text>
</svg>

After

Width:  |  Height:  |  Size: 5.2 KiB

@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="170" fill="url(#bg)" rx="10"/>
<text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DIRECT BRIDGE MODE · dedicated WAN NIC</text>
<!-- Row 1: Management NIC (independent, no WAN bridge path) -->
<g transform="translate(24, 42)">
<rect width="852" height="44" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="20" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">Management NIC</text>
<text x="20" y="36" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">host IP · SSH · spinifex services · overlay (Geneve)</text>
<text x="840" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end" font-weight="600">independent path</text>
</g>
<!-- Row 2: WAN NIC → br-wan → localnet -->
<g transform="translate(24, 100)">
<rect width="160" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="80" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
<text x="80" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">dedicated physical</text>
</g>
<polygon points="198,118 210,122 198,126" fill="#71717a" opacity="0.7"/>
<line x1="186" y1="122" x2="198" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<g transform="translate(214, 100)">
<rect width="220" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="110" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
<text x="110" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS port (NIC added directly)</text>
</g>
<polygon points="448,118 460,122 448,126" fill="#71717a" opacity="0.7"/>
<line x1="434" y1="122" x2="448" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<g transform="translate(464, 100)">
<rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">OVN localnet</text>
<text x="90" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">external switch port</text>
</g>
<text x="670" y="130" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" font-weight="600">distributed NAT · any MAC · all frames</text>
</svg>

After

Width:  |  Height:  |  Size: 3.1 KiB

+91
View File
@@ -0,0 +1,91 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="440" viewBox="0 0 900 440">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="host-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#27272a88"/>
<stop offset="100%" style="stop-color:#27272a22"/>
</linearGradient>
<linearGradient id="br-int-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f71f"/>
<stop offset="100%" style="stop-color:#a855f708"/>
</linearGradient>
<linearGradient id="br-wan-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#58a6ff1f"/>
<stop offset="100%" style="stop-color:#58a6ff08"/>
</linearGradient>
</defs>
<rect width="900" height="440" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK FLOW · bare-metal host</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">br-int overlay · br-wan WAN uplink</text>
<!-- ═══ Bare-Metal Host container ═══════════════════════ -->
<rect x="40" y="60" width="820" height="284" rx="12" fill="url(#host-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="58" y="84" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" font-weight="700" letter-spacing="1">BARE-METAL HOST</text>
<line x1="40" y1="96" x2="860" y2="96" stroke="#52525b" stroke-opacity="0.5" stroke-width="0.5"/>
<!-- br-int bridge (left) -->
<g transform="translate(80, 122)">
<rect width="260" height="176" rx="10" fill="url(#br-int-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" font-weight="700">br-int</text>
<text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="end">(overlay)</text>
<line x1="20" y1="34" x2="240" y2="34" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="30" y="52" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
<text x="130" y="70" fill="#d8b4fe" text-anchor="middle">tap-eni1</text>
<rect x="30" y="88" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
<text x="130" y="106" fill="#d8b4fe" text-anchor="middle">tap-eni2</text>
<rect x="30" y="124" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
<text x="130" y="142" fill="#d8b4fe" text-anchor="middle">tap-eni3</text>
</g>
</g>
<!-- OVN NAT pipeline arrow in middle -->
<g transform="translate(352, 200)">
<line x1="0" y1="10" x2="148" y2="10" stroke="#DC2626" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="148,5 160,10 148,15" fill="#DC2626" opacity="0.8"/>
<text x="78" y="0" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN</text>
<text x="78" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">NAT pipeline</text>
</g>
<!-- br-wan bridge (right) -->
<g transform="translate(520, 122)">
<rect width="260" height="176" rx="10" fill="url(#br-wan-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" font-weight="700">br-wan</text>
<text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#8ab4f8" text-anchor="end">(WAN)</text>
<line x1="20" y1="34" x2="240" y2="34" stroke="#58a6ff" stroke-opacity="0.3" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="30" y="52" width="200" height="60" rx="5" fill="#58a6ff1a" stroke="#58a6ff" stroke-opacity="0.45" stroke-width="0.8"/>
<text x="130" y="74" fill="#8ab4f8" text-anchor="middle" font-weight="600">spx-ext-{nic}</text>
<text x="130" y="92" fill="#a1a1aa" text-anchor="middle" font-size="10">(macvlan)</text>
</g>
</g>
<!-- macvlan bridge mode label -->
<text x="650" y="314" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">macvlan bridge mode</text>
<!-- WAN NIC inside host, bottom -->
<line x1="650" y1="298" x2="650" y2="320" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<g transform="translate(500, 320)">
<rect width="300" height="20" rx="5" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
<text x="150" y="14" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">WAN NIC · host IP unchanged</text>
</g>
<!-- connector out -->
<line x1="650" y1="344" x2="650" y2="380" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="646,376 654,376 650,384" fill="#71717a" opacity="0.7"/>
<!-- Physical WAN -->
<g transform="translate(500, 388)">
<rect width="300" height="40" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="150" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">Physical WAN</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 5.7 KiB

+57
View File
@@ -0,0 +1,57 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="320" viewBox="0 0 900 320">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="320" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">MACVLAN FALLBACK · single-NIC host</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">WAN NIC == SSH NIC · host IP preserved</text>
<!-- WAN NIC (parent) -->
<g transform="translate(280, 58)">
<rect width="340" height="54" rx="8" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.3" stroke-opacity="0.65"/>
<text x="170" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
<text x="170" y="42" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">host IP unchanged · parent interface</text>
</g>
<!-- Split point -->
<line x1="450" y1="112" x2="450" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<line x1="260" y1="132" x2="640" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<line x1="260" y1="132" x2="260" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<line x1="640" y1="132" x2="640" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- Child 1: host traffic -->
<g transform="translate(120, 148)">
<rect width="280" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">host traffic</text>
<text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SSH · management · overlay</text>
</g>
<!-- Child 2: macvlan -->
<g transform="translate(500, 148)">
<rect width="280" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">spx-ext-{nic}</text>
<text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">macvlan · bridge mode · no IP</text>
</g>
<!-- br-wan -->
<line x1="640" y1="196" x2="640" y2="216" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="636,212 644,212 640,220" fill="#71717a" opacity="0.7"/>
<g transform="translate(500, 220)">
<rect width="280" height="40" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="140" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
</g>
<!-- localnet -->
<line x1="640" y1="260" x2="640" y2="276" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="636,272 644,272 640,280" fill="#71717a" opacity="0.7"/>
<g transform="translate(500, 280)">
<rect width="280" height="30" rx="6" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
<text x="140" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="600">localnet → OVN ext switch</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 3.6 KiB

@@ -0,0 +1,131 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="wan-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="ovs-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#58a6ff22"/>
<stop offset="100%" style="stop-color:#58a6ff0a"/>
</linearGradient>
<linearGradient id="router-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#DC262622"/>
<stop offset="100%" style="stop-color:#7F1D1D0a"/>
</linearGradient>
<linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f722"/>
<stop offset="100%" style="stop-color:#a855f708"/>
</linearGradient>
</defs>
<rect width="900" height="560" fill="url(#bg)" rx="12"/>
<!-- Title -->
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK PATH</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">OVN logical topology · bridge mapping · NAT</text>
<!-- WAN -->
<rect x="300" y="54" width="300" height="40" rx="8" fill="url(#wan-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.55"/>
<text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Physical Network (WAN)</text>
<!-- connector -->
<line x1="450" y1="94" x2="450" y2="112" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- br-wan bridge -->
<rect x="340" y="112" width="220" height="52" rx="8" fill="url(#ovs-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="450" y="134" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
<text x="450" y="152" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">OVS bridge · physical uplink</text>
<!-- connector with bridge-mappings label -->
<line x1="450" y1="164" x2="450" y2="186" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<rect x="316" y="170" width="268" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.5"/>
<text x="450" y="183" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">ovn-bridge-mappings: external:br-wan</text>
<!-- VPC Logical Router (the big box) -->
<rect x="60" y="208" width="780" height="196" rx="10" fill="url(#router-grad)" stroke="#DC2626" stroke-width="1.4" stroke-opacity="0.75"/>
<text x="78" y="232" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" font-weight="700" letter-spacing="1">VPC LOGICAL ROUTER</text>
<line x1="60" y1="242" x2="840" y2="242" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<!-- Router ports row -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="82" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
<text x="142" y="270" fill="#F87171" text-anchor="middle" font-weight="600">Gateway port</text>
<text x="142" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">public IP</text>
<text x="215" y="282" fill="#71717a">─────</text>
<rect x="260" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
<text x="320" y="270" fill="#F87171" text-anchor="middle" font-weight="600">ext switch</text>
<text x="320" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">OVN logical</text>
<text x="393" y="282" fill="#71717a">─────</text>
<rect x="438" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
<text x="498" y="270" fill="#F87171" text-anchor="middle" font-weight="600">localnet port</text>
<text x="498" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">→ br-wan</text>
</g>
<!-- NAT rules + Routes, two columns on right -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
<text x="598" y="260" fill="#F87171" font-weight="600">NAT rules</text>
<text x="598" y="276" fill="#f5f5f5">SNAT:</text>
<text x="644" y="276" fill="#a1a1aa">VPC CIDR → gw IP</text>
<text x="598" y="292" fill="#f5f5f5">dnat_and_snat:</text>
<text x="704" y="292" fill="#a1a1aa">pub ↔ priv</text>
</g>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
<text x="82" y="330" fill="#F87171" font-weight="600">Routes</text>
<text x="82" y="346" fill="#f5f5f5">0.0.0.0/0</text>
<text x="154" y="346" fill="#a1a1aa">→ WAN gateway</text>
</g>
<!-- Subnet branches inside router -->
<line x1="150" y1="362" x2="150" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="150" y1="386" x2="305" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="305" y1="386" x2="305" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="150" y1="386" x2="515" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="515" y1="386" x2="515" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<!-- Subnet A (public) -->
<g transform="translate(215, 404)">
<rect width="180" height="124" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">subnet-a</text>
<text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">(public)</text>
<line x1="16" y1="46" x2="164" y2="46" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.1.0/24</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
<circle cx="40" cy="88" r="2.5" fill="#3fb950"/>
<text x="52" y="92">eni-1</text>
<circle cx="40" cy="108" r="2.5" fill="#3fb950"/>
<text x="52" y="112">eni-2</text>
</g>
</g>
<!-- Subnet B (private) -->
<g transform="translate(425, 404)">
<rect width="180" height="124" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">subnet-b</text>
<text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">(private)</text>
<line x1="16" y1="46" x2="164" y2="46" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.2.0/24</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
<circle cx="40" cy="88" r="2.5" fill="#a855f7"/>
<text x="52" y="92">eni-3</text>
<circle cx="40" cy="108" r="2.5" fill="#a855f7"/>
<text x="52" y="112">eni-4</text>
</g>
</g>
<!-- Footnote -->
<text x="24" y="548" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a">Cross-host traffic uses Geneve tunnels (UDP 6081) over the overlay NIC · ovn-controller programs OpenFlow rules on br-int</text>
</svg>

After

Width:  |  Height:  |  Size: 8.2 KiB

@@ -0,0 +1,56 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="110" fill="url(#bg)" rx="10"/>
<text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PRIVATE SUBNET · no default route</text>
<!-- Instance -->
<g transform="translate(40, 42)">
<rect width="180" height="44" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">Instance</text>
<text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">10.0.2.5</text>
</g>
<!-- arrow -->
<g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
<line x1="220" y1="64" x2="260" y2="64" stroke-dasharray="3,3"/>
</g>
<polygon points="260,60 268,64 260,68" fill="#DC2626" opacity="0.6"/>
<!-- Router -->
<g transform="translate(275, 42)">
<rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">VPC Router</text>
<text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">lookup 0.0.0.0/0</text>
</g>
<!-- arrow -->
<g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
<line x1="455" y1="64" x2="495" y2="64" stroke-dasharray="3,3"/>
</g>
<polygon points="495,60 503,64 495,68" fill="#DC2626" opacity="0.6"/>
<!-- [no default route] -->
<g transform="translate(510, 42)">
<rect width="196" height="44" rx="7" fill="#0a0a0a" stroke="#71717a" stroke-width="1" stroke-opacity="0.7" stroke-dasharray="4,3"/>
<text x="98" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">[no default route]</text>
<text x="98" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle">no IGW attached</text>
</g>
<!-- arrow to dropped -->
<g stroke="#71717a" stroke-width="1.2" opacity="0.6" fill="none">
<line x1="706" y1="64" x2="746" y2="64" stroke-dasharray="3,3"/>
</g>
<polygon points="746,60 754,64 746,68" fill="#71717a" opacity="0.6"/>
<!-- dropped -->
<g transform="translate(760, 42)">
<rect width="116" height="44" rx="7" fill="#27272a" stroke="#52525b" stroke-width="1" stroke-opacity="0.7"/>
<text x="58" y="27" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">dropped</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 3.0 KiB

@@ -0,0 +1,74 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="170" fill="url(#bg)" rx="10"/>
<text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PUBLIC SUBNET · 1:1 NAT managed by OVN</text>
<!-- OUTBOUND row -->
<text x="24" y="52" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700">Outbound</text>
<g transform="translate(112, 38)">
<rect width="180" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">Instance</text>
<text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">10.0.1.5</text>
</g>
<g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="292" y1="58" x2="352" y2="58" stroke-dasharray="3,3"/>
</g>
<polygon points="352,54 360,58 352,62" fill="#3fb950" opacity="0.7"/>
<g transform="translate(368, 38)">
<rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">SNAT</text>
</g>
<g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="498" y1="58" x2="558" y2="58" stroke-dasharray="3,3"/>
</g>
<polygon points="558,54 566,58 558,62" fill="#3fb950" opacity="0.7"/>
<g transform="translate(574, 38)">
<rect width="272" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">exits as 203.0.113.10</text>
<text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">source IP rewritten</text>
</g>
<!-- INBOUND row -->
<text x="24" y="110" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" font-weight="700">Inbound</text>
<g transform="translate(112, 96)">
<rect width="180" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">WAN client</text>
<text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">→ 203.0.113.10</text>
</g>
<g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="292" y1="116" x2="352" y2="116" stroke-dasharray="3,3"/>
</g>
<polygon points="352,112 360,116 352,120" fill="#58a6ff" opacity="0.7"/>
<g transform="translate(368, 96)">
<rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">DNAT</text>
</g>
<g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="498" y1="116" x2="558" y2="116" stroke-dasharray="3,3"/>
</g>
<polygon points="558,112 566,116 558,120" fill="#58a6ff" opacity="0.7"/>
<g transform="translate(574, 96)">
<rect width="272" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">arrives at 10.0.1.5</text>
<text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">dest IP rewritten</text>
</g>
<!-- Footnote -->
<text x="24" y="158" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a">The instance OS only sees its private IP (10.0.1.5) — the public address is managed entirely by OVN's dnat_and_snat rule.</text>
</svg>

After

Width:  |  Height:  |  Size: 4.4 KiB

+144
View File
@@ -0,0 +1,144 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
<defs>
<linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
<stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
</linearGradient>
<linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
<stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
</linearGradient>
<linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
</linearGradient>
<linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
<stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
</linearGradient>
</defs>
<rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
<!-- Subtle mesh dots background -->
<g opacity="0.12" fill="#7F1D1D">
<circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
<circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
<circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
<circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
<circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
<circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
</g>
<!-- Title -->
<text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
<text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
<!-- Layer 4: Applications (top) -->
<g transform="translate(60, 95)">
<rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
<text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
<g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
<rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
<rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
<rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
<rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
<rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
<rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
</g>
</g>
<!-- Down arrow -->
<g transform="translate(450, 188)" opacity="0.45">
<line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
<polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
</g>
<!-- Layer 3: Spinifex (highlighted) -->
<g transform="translate(60, 215)">
<rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
<text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
<text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
<g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
<!-- EC2 -->
<rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
<text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
<text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
<text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
<!-- EBS -->
<rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
<text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
<text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
<text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
<!-- S3 -->
<rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
<text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
<text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
<text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
<!-- VPC -->
<rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
<text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
<text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
<text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
<!-- IAM -->
<rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
<text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
<text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
<text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
<!-- Control -->
<rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
<text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
<text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
<text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
</g>
</g>
<!-- Down arrow -->
<g transform="translate(450, 360)" opacity="0.45">
<line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
<polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
</g>
<!-- Layer 2: OS -->
<g transform="translate(60, 387)">
<rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
<text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
<text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
</g>
<!-- Down arrow -->
<g transform="translate(450, 454)" opacity="0.45">
<line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
<polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
</g>
<!-- Layer 1: Hardware -->
<g transform="translate(60, 481)">
<rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
<text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 9.5 KiB

+52
View File
@@ -0,0 +1,52 @@
version: 2
updates:
- package-ecosystem: gomod
target-branch: main
directory: /
schedule:
interval: weekly
day: monday
timezone: Australia/Sydney
cooldown:
default-days: 1
groups:
go-dependencies:
patterns:
- "*"
- package-ecosystem: github-actions
target-branch: main
directory: /
schedule:
interval: weekly
day: monday
timezone: Australia/Sydney
cooldown:
default-days: 1
ignore:
- dependency-name: "*"
update-types:
- "version-update:semver-minor"
- "version-update:semver-patch"
groups:
github-actions:
patterns:
- "*"
- package-ecosystem: npm
target-branch: main
directory: /spinifex/services/spinifexui/frontend
schedule:
interval: weekly
day: monday
timezone: Australia/Sydney
cooldown:
default-days: 1
ignore:
- dependency-name: "@types/node"
update-types:
- "version-update:semver-major"
groups:
npm-dependencies:
patterns:
- "*"
+230 -48
View File
@@ -1,80 +1,262 @@
name: Hive E2E Test
name: Spinifex E2E Tests
on:
push:
branches:
- main
- e2e
paths:
- "**/*.go"
- go.mod
- go.sum
- Makefile
- tests/e2e/**
- scripts/**
- .github/workflows/e2e.yml
workflow_dispatch:
inputs:
sha:
description: "PR head SHA to report status on"
required: false
permissions:
statuses: write
env:
GIT_BRANCH: ${{ github.ref_name }}
jobs:
build_e2e_image:
name: Build E2E Image
runs-on: self-hosted
e2e_single:
name: Single-Node E2E
runs-on: [self-hosted, ci-single]
timeout-minutes: 30
env:
TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
steps:
- name: Checkout Code
uses: actions/checkout@v6
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
id: app-token
with:
path: hive
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- name: Clone Dependencies
run: ./hive/scripts/clone-deps.sh
- uses: actions/checkout@v6
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Build E2E Image
- name: Build install artifacts
working-directory: mulga/scripts/tofu-cluster
run: |
docker build \
-t hive-e2e:latest \
-f hive/tests/e2e/Dockerfile.e2e \
.
./build-install-artifacts.sh "${GIT_BRANCH}" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-tests.tar"
hive_e2e_test:
name: Hive Single-Node E2E Test
runs-on: self-hosted
needs: build_e2e_image
- name: Provision VM
working-directory: mulga/scripts/tofu-cluster
run: |
export TF_VAR_git_branch="${GIT_BRANCH}"
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap via Binary Install
working-directory: mulga/scripts/tofu-cluster
run: |
SPX_EXTERNAL_POOL="192.168.1.230-192.168.1.240" \
./bootstrap-install.sh env3 \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
steps:
- name: Run E2E Tests
working-directory: mulga/scripts/tofu-cluster
run: |
docker run --privileged --rm \
-v /dev/kvm:/dev/kvm \
--name hive-e2e-test \
hive-e2e:latest
INVENTORY=$(tofu output -json inventory)
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
'cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ./run-lb-e2e.sh'
hive_multinode_e2e_test:
name: Hive Multi-Node E2E Test
runs-on: self-hosted
needs: build_e2e_image
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
e2e_pseudo_multi:
name: Pseudo Multi-Node E2E
runs-on: [self-hosted, ci-pseudo]
timeout-minutes: 30
env:
TOFU_LOG: /tmp/e2e-tofu-pseudo-${{ github.run_id }}.log
steps:
- name: Run Multi-Node E2E Tests
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v6
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Clone and build
working-directory: mulga/scripts/tofu-cluster
run: ./build-artifacts.sh "${GIT_BRANCH}" "${RUNNER_TEMP}/spinifex-artifacts.tar"
- name: Provision VM
working-directory: mulga/scripts/tofu-cluster
run: |
docker run --privileged --rm \
-v /dev/kvm:/dev/kvm \
--cap-add=NET_ADMIN \
--name hive-multinode-e2e-test \
hive-e2e:latest \
./tests/e2e/run-multinode-e2e.sh
export TF_VAR_git_branch="${GIT_BRANCH}"
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
cleanup:
name: Cleanup
runs-on: self-hosted
needs: [hive_e2e_test, hive_multinode_e2e_test]
if: always()
- name: Bootstrap Cluster
working-directory: mulga/scripts/tofu-cluster
run: |
SPX_PREBUILT=true \
SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
./bootstrap.sh env4 >> "$TOFU_LOG" 2>&1
- name: Run E2E Tests
working-directory: mulga/scripts/tofu-cluster
run: |
INVENTORY=$(tofu output -json inventory)
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
'cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL="192.168.1.220-192.168.1.229" ./tests/e2e/run-pseudo-multinode-e2e.sh'
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
e2e_real_multi:
name: Real Multi-Node E2E
runs-on: [self-hosted, ci-multi]
timeout-minutes: 35
env:
TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
steps:
- name: Remove Stale Containers
run: |
docker rm -f hive-e2e-test 2>/dev/null || true
docker rm -f hive-multinode-e2e-test 2>/dev/null || true
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- name: Prune Docker Images
run: docker image prune -f
- uses: actions/create-github-app-token@v3
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v6
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Build Install Artifacts
working-directory: mulga/scripts/tofu-cluster
run: |
./build-install-artifacts.sh "${GIT_BRANCH}" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-tests.tar"
- name: Provision VMs
working-directory: mulga/scripts/tofu-cluster
run: |
export TF_VAR_git_branch="${GIT_BRANCH}"
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap via Binary Install
working-directory: mulga/scripts/tofu-cluster
run: |
SPX_EXTERNAL_POOL="192.168.1.240-192.168.1.250" \
./bootstrap-install.sh env2 \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
- name: Run E2E Tests
working-directory: mulga/scripts/tofu-cluster
run: |
INVENTORY=$(tofu output -json inventory)
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
# Build node IP list
NODE_IPS=""
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
for i in $(seq 0 $((NODE_COUNT - 1))); do
NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
done
NODE2_IP=$(echo "$INVENTORY" | jq -r '.nodes[1].wan_ip')
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
"cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
e2e_status:
name: Report E2E Status
if: always() && inputs.sha
needs: [e2e_single, e2e_pseudo_multi, e2e_real_multi]
runs-on: [self-hosted, ci-single]
timeout-minutes: 5
steps:
- name: Report status to PR
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "${{ needs.e2e_single.result }}" == "success" && \
"${{ needs.e2e_pseudo_multi.result }}" == "success" && \
"${{ needs.e2e_real_multi.result }}" == "success" ]]; then
STATE="success"
DESC="All E2E suites passed"
else
STATE="failure"
DESC="E2E failed: single=${{ needs.e2e_single.result }} pseudo=${{ needs.e2e_pseudo_multi.result }} multi=${{ needs.e2e_real_multi.result }}"
fi
gh api "repos/${{ github.repository }}/statuses/${{ inputs.sha }}" \
-f state="$STATE" \
-f context=e2e \
-f description="$DESC" \
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
-103
View File
@@ -1,103 +0,0 @@
name: Hive Build and Test
on:
push:
branches:
- main
- dev
paths:
- "**/*.go"
- go.mod
- go.sum
- Makefile
- .github/workflows/hive.yml
pull_request:
paths:
- "**/*.go"
- go.mod
- go.sum
- Makefile
- .github/workflows/hive.yml
workflow_dispatch:
jobs:
build_and_test:
name: Hive Build and Test
runs-on: ubuntu-24.04
steps:
- name: Checkout Hive
uses: actions/checkout@v6
with:
path: hive
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
ref: dev
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
ref: dev
path: predastore
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
- name: Build Hive
run: make -C hive build
- name: Run Tests
run: make -C hive test
lint_and_security:
name: Hive Lint and Security
runs-on: ubuntu-24.04
steps:
- name: Checkout Hive
uses: actions/checkout@v6
with:
path: hive
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
ref: dev
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
ref: dev
path: predastore
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
- name: Check Format
run: make -C hive check-format
- name: Go Vet
run: make -C hive vet
- name: Security Checks
run: make -C hive security-check
+194
View File
@@ -0,0 +1,194 @@
name: Release
on:
push:
tags: ["v*"]
permissions:
contents: write
jobs:
distro-amd64:
runs-on: ubuntu-24.04
steps:
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: spinifex
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Build amd64 Tarball
env:
VERSION: ${{ github.ref_name }}
run: cd spinifex && make distro-amd64 VERSION="$VERSION"
- name: Upload amd64 Artifacts
uses: actions/upload-artifact@v7
with:
name: distro-amd64
path: |
spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz
spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz.sha256
spinifex/scripts/setup.sh
distro-arm64:
runs-on: ubuntu-24.04-arm
steps:
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: spinifex
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Build arm64 Tarball
env:
VERSION: ${{ github.ref_name }}
run: cd spinifex && make distro-arm64 VERSION="$VERSION"
- name: Upload arm64 Artifacts
uses: actions/upload-artifact@v7
with:
name: distro-arm64
path: |
spinifex/dist/spinifex-${{ github.ref_name }}-linux-arm64.tar.gz
spinifex/dist/spinifex-${{ github.ref_name }}-linux-arm64.tar.gz.sha256
release:
needs: [distro-amd64, distro-arm64]
runs-on: ubuntu-24.04
steps:
- name: Download amd64 Artifacts
uses: actions/download-artifact@v8
with:
name: distro-amd64
path: artifacts
- name: Download arm64 Artifacts
uses: actions/download-artifact@v8
with:
name: distro-arm64
path: artifacts
- name: Create Draft GitHub Release
uses: softprops/action-gh-release@v3
with:
draft: true
files: |
artifacts/**/*.tar.gz
artifacts/**/*.sha256
artifacts/**/setup.sh
generate_release_notes: true
- name: Upload Tarball for Verify Job
uses: actions/upload-artifact@v7
with:
name: release-assets
path: artifacts/
verify:
needs: release
runs-on: [self-hosted, ci-single]
timeout-minutes: 25
env:
TOFU_LOG: /tmp/release-verify-${{ github.run_id }}.log
TAG: ${{ github.ref_name }}
REPO: ${{ github.repository }}
steps:
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
id: app-token
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v6
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Download Release Assets
uses: actions/download-artifact@v8
with:
name: release-assets
path: release-assets
- name: Provision VM
working-directory: mulga/scripts/tofu-cluster
run: |
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap via Binary Install
working-directory: mulga/scripts/tofu-cluster
run: |
SETUP_SH=$(find "${GITHUB_WORKSPACE}/release-assets" -name "setup.sh" | head -1)
TARBALL=$(find "${GITHUB_WORKSPACE}/release-assets" -name "*-linux-amd64.tar.gz" | head -1)
./bootstrap-install.sh env3 "$SETUP_SH" "$TARBALL"
- name: Promote Release
if: success()
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "$TAG" =~ -(dev|rc|alpha|beta) ]]; then
gh release edit "$TAG" --repo "$REPO" --draft=false --prerelease
echo "Release ${TAG} promoted as prerelease"
else
gh release edit "$TAG" --repo "$REPO" --draft=false
echo "Release ${TAG} promoted to latest"
fi
- name: Mark Release Failed
if: failure()
env:
GH_TOKEN: ${{ github.token }}
run: |
EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
NOTES_FILE=$(mktemp)
printf 'Verification failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
rm -f "$NOTES_FILE"
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
+132
View File
@@ -0,0 +1,132 @@
name: Spinifex Build and Test
on:
push:
branches:
- main
paths:
- "**/*.go"
- go.mod
- go.sum
- Makefile
- .github/workflows/spinifex.yml
pull_request:
branches:
- main
workflow_dispatch:
jobs:
build_and_test:
name: Spinifex Build and Test
runs-on: ubuntu-24.04
steps:
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: spinifex
fetch-depth: 2
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "spinifex/go.sum"
go-version-file: "spinifex/go.mod"
- name: Setup Go Workspace
run: go work init ./spinifex ./viperblock ./predastore
- name: Build Spinifex
run: make -C spinifex build
- name: Run Tests with Coverage
run: make -C spinifex test-cover
- name: Check Diff Coverage
run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
race_detection:
name: Spinifex Race Detection
runs-on: ubuntu-24.04
steps:
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "spinifex/go.sum"
go-version-file: "spinifex/go.mod"
- name: Setup Go Workspace
run: go work init ./spinifex ./viperblock ./predastore
- name: Run Tests with Race Detector
run: make -C spinifex test-race
lint_and_security:
name: Spinifex Lint and Security
runs-on: ubuntu-24.04
steps:
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "spinifex/go.sum"
go-version-file: "spinifex/go.mod"
- name: Setup Go Workspace
run: go work init ./spinifex ./viperblock ./predastore
- name: Lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.11.3
working-directory: spinifex
- name: Govulncheck
run: make -C spinifex govulncheck
+14 -10
View File
@@ -4,13 +4,14 @@ on:
push:
branches:
- main
- ui
paths:
- hive/services/hiveui/frontend/**
- spinifex/services/spinifexui/frontend/**
- .github/workflows/ui.yml
pull_request:
branches:
- main
paths:
- hive/services/hiveui/frontend/**
- spinifex/services/spinifexui/frontend/**
- .github/workflows/ui.yml
workflow_dispatch:
@@ -20,31 +21,34 @@ jobs:
runs-on: ubuntu-24.04
defaults:
run:
working-directory: hive/services/hiveui/frontend
working-directory: spinifex/services/spinifexui/frontend
steps:
- name: Checkout Code
uses: actions/checkout@v6
with:
sparse-checkout: hive/services/hiveui/frontend
sparse-checkout: spinifex/services/spinifexui/frontend
- name: Setup pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v6
with:
package_json_file: hive/services/hiveui/frontend/package.json
package_json_file: spinifex/services/spinifexui/frontend/package.json
- name: Setup Node.js
uses: actions/setup-node@v6
with:
cache: "pnpm"
cache-dependency-path: hive/services/hiveui/frontend/pnpm-lock.yaml
node-version-file: hive/services/hiveui/frontend/.nvmrc
cache-dependency-path: spinifex/services/spinifexui/frontend/pnpm-lock.yaml
node-version-file: spinifex/services/spinifexui/frontend/.nvmrc
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Run Biome Check
- name: Run Check
run: pnpm check
- name: Run Tests with Coverage
run: pnpm test:coverage
- name: Build Frontend
run: pnpm build
+6 -1
View File
@@ -7,12 +7,17 @@
!example.tfvars
scripts/iac/proxmox/clusters/
# Generated SSH private keys from terraform examples (e.g. local_file.nginx_pem)
*.pem
# Go workspace
go.work
go.work.sum
# Build outputs
bin/
dist/
!spinifex/services/spinifexui/frontend/dist/
tmp/
*.test
*.out
@@ -34,8 +39,8 @@ Thumbs.db
docs/INTERNAL.md
docs/PRIVATE.md
docs/DEVNOTES.md
docs/HIVE_DEVELOPMENT_PLAN.md
docs/PLAN.md
docs/TODO.md
docs/development/
docs/internal/
scripts/iac/aws/spinifex-test.pem
+136
View File
@@ -0,0 +1,136 @@
version: "2"
formatters:
enable:
- gofmt
settings:
gofmt:
simplify: false # modernize linter covers gofmt -s simplifications (Go 1.23+)
linters:
enable:
- asasalint # catches pass []any as any in variadic func(...any)
- asciicheck # checks that code does not contain non-ASCII identifiers
- bidichk # checks for dangerous unicode bidi character sequences
- bodyclose # checks whether HTTP response body is closed successfully
- canonicalheader # checks whether net/http.Header uses canonical header keys
- decorder # checks declaration order of types, consts, vars, and funcs
- durationcheck # checks for two durations multiplied together
- embeddedstructfieldcheck # checks embedded sync.Mutex/RWMutex in structs
- errcheck # checks for unchecked errors in type assertions and I/O
- errchkjson # checks types passed to json encoding functions, flags omittable error checks
- errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
- exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
- fatcontext # detects nested contexts in loops
- forcetypeassert # flags unchecked type assertions that can panic
- gocheckcompilerdirectives # validates go compiler directive comments (//go:)
- goprintffuncname # checks that printf-like functions are named with f at the end
- gosec # inspects source code for security problems
- govet # reports suspicious constructs (Printf alignment, shadow, etc.)
- ineffassign # detects assignments to existing variables that are not used
- makezero # finds slice declarations with non-zero initial length followed by append
- mirror # catches wrong mirror patterns in bytes/strings usage
- musttag # checks structs passed to marshal/unmarshal have proper tags
- modernize # modernizes code to use newer Go idioms (go fix analyzers)
- nakedret # finds naked returns in functions
- nilerr # catches code that returns nil even after checking err != nil
- nilnesserr # catches returning a different nil error variable after err != nil check
- nolintlint # reports ill-formed or insufficient nolint directives
- nosprintfhostport # flags fmt.Sprintf host:port — use net.JoinHostPort for IPv6 safety
- perfsprint # detects fmt.Sprintf calls replaceable with faster alternatives
- predeclared # catches shadowing of Go builtins (len, cap, error, etc.)
- reassign # checks that package variables are not reassigned
- recvcheck # checks for receiver type consistency (pointer vs value)
- sloglint # enforces consistent log/slog usage (key-value pairs, no globals)
- staticcheck # extensive static analysis checks
- unconvert # removes unnecessary type conversions
- usestdlibvars # detects hardcoded values that should use stdlib constants
- usetesting # catches os.TempDir() etc. in tests — should use t.TempDir()
- wastedassign # finds assignments to variables that are never subsequently read
- whitespace # detects leading and trailing whitespace in functions
settings:
embeddedstructfieldcheck:
forbid-mutex: true
errcheck:
check-type-assertions: true
exclude-functions:
- (*github.com/nats-io/nats.go.Subscription).Unsubscribe # cleanup-only, errors not actionable
- (*github.com/kdomanski/iso9660.ImageWriter).Cleanup # deferred cleanup, errors not actionable
govet:
enable-all: true
disable:
- fieldalignment
- shadow # too noisy to retrofit — all 50 findings are err re-declarations
- unusedwrite # false positives on struct field assignments before marshal/send
staticcheck:
checks:
- all
- -ST1000 # package comment
- -ST1003 # naming conventions (Id→ID etc.) — too noisy to retrofit
- -ST1012 # error var naming — errname covers this + error type suffixes
- -ST1013 # HTTP status constants — usestdlibvars is broader
- -ST1016 # receiver name consistency (too noisy to retrofit)
- -ST1020 # godoc on exported types — not enforcing yet
- -ST1021 # godoc on exported type groups — not enforcing yet
- -ST1022 # godoc on exported vars/consts — not enforcing yet
- -SA1019 # deprecated usage (handle separately)
- -SA4006 # value overwritten — ineffassign + wastedassign cover this better
- -SA9005 # ineffective break (staticcheck false positives)
gosec:
excludes:
- G104 # unchecked errors — errcheck is stricter
- G116 # bidi chars — bidichk is more thorough
- G117 # use of net/http serve functions (we handle TLS separately)
- G703 # path traversal taint analysis — false positives on config file loading
- G705 # XSS taint analysis — false positives on internal error responses
- G706 # log injection taint analysis — false positives on structured slog usage
modernize:
disable:
- newexpr # don't replace aws.String() with new() — AWS SDK idiom
- stringsbuilder # don't replace small string concatenations with strings.Builder
perfsprint:
concat-loop: false
err-error: false # don't flag err.Error()
errorf: false # don't convert fmt.Errorf to errors.New
strconcat: false
nakedret:
max-func-lines: 0
nolintlint:
require-explanation: true
require-specific: true
reassign:
patterns:
- ".*"
exclusions:
presets:
- std-error-handling
- common-false-positives
rules:
- path: '_test\.go'
linters:
- bodyclose
- errcheck
- errchkjson
- gosec
# spinifex-specific: gosec exclusions
- path: "cmd/"
linters:
- errcheck # CLI glue code — viper/cobra/pterm errors not actionable
- gosec # matches current -exclude-dir=cmd
- text: "G204"
linters: [gosec] # subprocess execution — legitimate in VM orchestration
- text: "G304"
linters: [gosec] # file path from variable — necessary for config loading
+2 -926
View File
@@ -1,927 +1,3 @@
# Hive Installation
# Spinifex Installation
Notes for development environment installation.
## Dependencies
The currently supported operating systems that are validated and tested include:
- Ubuntu 22.04
- Ubuntu 24.04
- Ubuntu 25.10
- Debian 12.13
For the development preview, please use one of the supported versions above.
## Download
Create the base directory for the Hive development environment.
```bash
mkdir -p ~/Development/mulga/
cd ~/Development/mulga/
git clone https://github.com/mulgadc/hive.git
```
### Quick Install
To bootstrap the dependencies of Hive in one simple step (QEMU, Go, AWS CLI):
```bash
sudo make -C hive quickinstall
```
Ensure Go is available in your `PATH` if not previously installed:
```bash
export PATH=$PATH:/usr/local/go/bin/
```
### Manual Install
Alternatively, run the following steps to manually set up the required dependencies.
```bash
sudo add-apt-repository universe
sudo apt install nbdkit nbdkit-plugin-dev pkg-config qemu-system qemu-utils qemu-kvm libvirt-daemon-system libvirt-clients libvirt-dev make gcc unzip xz-utils file
```
Ensure the Go toolkit is installed for version 1.25.7 or higher. Recommended to install the latest directly from [https://go.dev/dl/](https://go.dev/dl/).
Confirm Go is correctly installed, and set in your $PATH.
```bash
go version
```
Hive provides AWS API/SDK layer functionality, which requires the AWS CLI tool to be installed to interface with the system.
```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
```
Confirm awscli version > 2.0 is installed.
[https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
```bash
aws --version
```
## Build
Setup the dev environment and package dependencies on [viperblock](https://github.com/mulgadc/viperblock/) and [predastore](https://github.com/mulgadc/predastore/).
```bash
cd hive
./scripts/clone-deps.sh # Clone viperblock + predastore repositories
./scripts/dev-setup.sh # Setup complete development environment
```
Once complete, confirm `./bin/hive` exists and is executable.
# Single Node Installation
For rapid development and testing, `hive` can be installed locally as a single node instance. Follow the instructions below for a complete working environment.
## Init
When running Hive for the first time, run the init function to create the default directories for data, config files and layout required.
Example single node installation to get started, this will create a new region `ap-southeast-2` and availability zone `ap-southeast-2a` on your local instance:
```bash
./bin/hive admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1
```
## Trust CA Certificate (Required)
Hive generates a local Certificate Authority (CA) during initialization. For AWS CLI and other tools to trust Hive services over HTTPS, you must add the CA certificate to your system's trust store.
```bash
sudo cp ~/hive/config/ca.pem /usr/local/share/ca-certificates/hive-ca.crt
sudo update-ca-certificates
```
Verify the certificate was added:
```bash
ls -la /usr/local/share/ca-certificates/hive-ca.crt
```
Next, set the AWS profile to use `hive` which points to the local environment.
```bash
export AWS_PROFILE=hive
```
## Launch Services
Start the core services for development.
```bash
./scripts/start-dev.sh
```
## Create SSH Key
For first install, create or import an existing key pair which can be used to launch EC2 instances.
### Import Existing Key
Import an existing key pair, replace `~/.ssh/id_rsa.pub` with your specified key.
```bash
aws ec2 import-key-pair --key-name "hive-key" --public-key-material fileb://~/.ssh/id_rsa.pub
```
If no key exists, generate one using `ssh-keygen` and repeat the command above.
```bash
ssh-keygen -t rsa
```
### Create New Key
Alternatively, create a new key pair using the AWS CLI tool and store the JSON output of the AWS SDK using the `jq` command.
```bash
aws ec2 create-key-pair \
--key-name hive-key \
| jq -r '.KeyMaterial | rtrimstr("\n")' > ~/.ssh/hive-key
```
Update permissions for the key for SSH to accept reading the file.
```bash
chmod 600 ~/.ssh/hive-key
```
Next, generate a public key from the specified private key pair.
```bash
ssh-keygen -y -f ~/.ssh/hive-key > ~/.ssh/hive-key.pub
```
### Verify New Key
```bash
aws ec2 describe-key-pairs
```
```json
{
"KeyPairs": [
{
"KeyPairId": "key-3caa0a34f53d12a3",
"KeyType": "ed25519",
"Tags": [],
"CreateTime": "2025-10-28T13:39:23.458000+00:00",
"KeyName": "hive-key",
"KeyFingerprint": "SHA256:/g/A5OkeZeSydz9WUErXYVdCt00b0VbfN6RLn2YVFAY"
}
]
}
```
## Create AMI Template
The Hive CLI tool offers 2 ways of importing an image. You can use automatic image importing, and select from our range of images. Alternatively, provide your own image source file (e.g image.tar.gz, image.gz, image.tar.xz) and the tool will automatically extract and upload the raw OS image as an AMI (after validating the image contains UEFI/BIOS boot capability).
**Note:** When downloading OS images, use platforms that support the `cloud-init` feature to automatically bootstrap when using the Hive EC2 functionality to access SSH and networking services.
### Automatic Image Import
Discover available images to automatically download and install. This will pull the images from the distro official mirror and simplify the process to bootstrap a Hive installation with AMIs for common operating systems.
```bash
./bin/hive admin images list
```
```bash
NAME | DISTRO | VERSION | ARCH | BOOT
alpine-3.22.2-x86_64 | alpine | 3.22.2 | x86_64 | bios
debian-12-arm64 | debian | 12 | arm64 | bios
debian-12-x86_64 | debian | 12 | x86_64 | bios
ubuntu-24.04-arm64 | ubuntu | 24.04 | arm64 | bios
ubuntu-24.04-x86_64 | ubuntu | 24.04 | x86_64 | bios
```
Next, choose the image you would like to import as an AMI. Choose the appropriate `x86_64` or `arm64` architecture depending on your platform.
```bash
./bin/hive admin images import --name debian-12-arm64
```
### Manual Image Import
Using this method you can import any OS disk image. For example, download the Debian 12 image from the repository [https://cloud.debian.org/images/cloud/bookworm/latest/](https://cloud.debian.org/images/cloud/bookworm/latest/).
Download the image:
```bash
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-arm64.tar.xz -O ~/debian-12-genericcloud-arm64.tar.xz
```
Import as an AMI to the backend store:
```bash
./bin/hive admin images import --file ~/debian-12-genericcloud-arm64.tar.xz --arch arm64 --distro debian --version 12
```
### Verify Image Import
Export the AMI.
```bash
export HIVE_AMI="ami-XXX"
```
Next, verify available disk images to confirm the import was successful.
```bash
aws ec2 describe-images --image-ids $HIVE_AMI
```
## Run Instance
Once Hive is successfully installed and bootstrapped with a system AMI and SSH keys, proceed to run an instance.
### Query Instance Types
Depending on the host platform Hive is installed, different compute instances will be available.
Query available instance types and choose an instance type available on your host:
```bash
aws ec2 describe-instance-types
```
```json
{
"InstanceTypes": [
{
"InstanceType": "t3.small",
"CurrentGeneration": true,
"SupportedRootDeviceTypes": [
"ebs"
],
"SupportedVirtualizationTypes": [
"hvm"
],
"Hypervisor": "kvm",
"ProcessorInfo": {
"SupportedArchitectures": [
"x86_64"
]
},
"VCpuInfo": {
"DefaultVCpus": 2
},
"MemoryInfo": {
"SizeInMiB": 2048
},
"BurstablePerformanceSupported": false
},
]
}
```
Export the instance-type:
```sh
export HIVE_INSTANCE="t3.small"
```
### Run Instance
Next, launch a new instance, note `hive-key` is the SSH key specified in the previous stage.
```bash
aws ec2 run-instances \
--image-id $HIVE_AMI \
--instance-type $HIVE_INSTANCE \
--key-name hive-key \
--security-group-ids sg-0123456789abcdef0 \
--subnet-id subnet-6e7f829e \
--count 1
```
A sample response is below from the `RunInstance` request, note the `InstanceId` attribute:
```json
{
"ReservationId": "r-f101157331e261a68",
"OwnerId": "123456789012",
"Instances": [
{
"InstanceId": "i-36765eb0c6609e4d2",
"ImageId": "ami-15c3dcfe607460f15",
"State": {
"Code": 0,
"Name": "pending"
},
"KeyName": "hive-key",
"InstanceType": "t3.micro",
"LaunchTime": "2025-11-12T13:07:47.548000+00:00"
}
]
}
```
Export the instance ID.
```bash
export INSTANCE_ID="i-XXX"
```
### Verify Instance
Next, validate the running instance is ready.
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```
Confirm the `State.Name` attribute is set as `running`.
```json
{
"Reservations": [
{
"ReservationId": "r-f101157331e261a68",
"OwnerId": "123456789012",
"Instances": [
{
"InstanceId": "i-36765eb0c6609e4d2",
"ImageId": "ami-15c3dcfe607460f15",
"State": {
"Code": 16,
"Name": "running"
},
"KeyName": "hive-key",
"InstanceType": "t3.micro",
"LaunchTime": "2025-11-12T13:07:47.548000+00:00"
}
]
}
]
}
```
## SSH Connection (development)
For a Hive development environment (toggled off for production), a local SSH port forwarding will be active to connect directly to the instance, regardless of the VPC and network settings.
Determine the SSH port allocated.
```bash
ps auxw | grep $INSTANCE_ID
```
```bash
qemu-system-x86_64 -daemonize -pidfile /run/user/1000/i-36765eb0c6609e4d.pid -qmp unix:/run/user/1000/qmp-i-36765eb0c6609e4d2.sock,server,nowait -enable-kvm -M ubuntu -serial pty -cpu host -smp 2 -m 1024 -drive file=nbd://127.0.0.1:42653,format=raw,if=none,media=disk,id=os -drive file=nbd://127.0.0.1:44499,format=raw,if=virtio,media=cdrom,id=cloudinit -device virtio-blk-pci,drive=os,bootindex=1 -device virtio-rng-pci -device virtio-net-pci,netdev=net0 -netdev user,id=net0,hostfwd=tcp:127.0.0.1:33683-:22
```
Note the `hostfwd=tcp:127.0.0.1:33683-:22`, in this case the local port `33683` will connect to the new instance.
```bash
ssh -i ~/.ssh/hive-key ec2-user@127.0.0.1 -p 33683
...
Linux hive-vm-36765eb0 6.1.0-40-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.153-1 (2025-09-20) x86_64
ec2-user@hive-vm-36765eb0:~$
```
Congratulations! Your first AMI image is imported, a new EC2 instance launched, and successfully connected via SSH for the configured SSH key, using the OS `cloud-init` procedure.
## Managing Instances
### Stop Instance
To stop a running instance gracefully:
```bash
aws ec2 stop-instances --instance-ids $INSTANCE_ID
```
```json
{
"StoppingInstances": [
{
"InstanceId": "i-36765eb0c6609e4d2",
"CurrentState": {
"Code": 64,
"Name": "stopping"
},
"PreviousState": {
"Code": 16,
"Name": "running"
}
}
]
}
```
Next, confirm the instance has stopped as requested.
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```
```json
{
"Reservations": [
{
"ReservationId": "r-f101157331e261a68",
"OwnerId": "123456789012",
"Instances": [
{
"InstanceId": "i-36765eb0c6609e4d2",
"ImageId": "ami-15c3dcfe607460f15",
"State": {
"Code": 80,
"Name": "stopped"
},
"KeyName": "hive-key",
"InstanceType": "t3.micro",
"LaunchTime": "2025-11-12T13:07:47.548000+00:00"
}
]
}
]
}
```
### Start Instance
To start a previously stopped instance:
```bash
aws ec2 start-instances --instance-ids $INSTANCE_ID
```
```json
{
"StartingInstances": [
{
"InstanceId": "i-36765eb0c6609e4d2",
"CurrentState": {
"Code": 0,
"Name": "pending"
},
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}
```
Next, validate the instance is running as expected.
```bash
aws ec2 describe-instances --instance-ids "i-36765eb0c6609e4d2"
```
```json
{
"Reservations": [
{
"ReservationId": "r-f101157331e261a68",
"OwnerId": "123456789012",
"Instances": [
{
"InstanceId": "i-36765eb0c6609e4d2",
"ImageId": "ami-15c3dcfe607460f15",
"State": {
"Code": 16,
"Name": "running"
},
"KeyName": "hive-key",
"InstanceType": "t3.micro",
"LaunchTime": "2025-11-12T13:07:47.548000+00:00"
}
]
}
]
}
```
### Terminate Instance
To terminate an instance, which will first stop the instance, and on success, remove the EBS volumes and permanently remove the instance data.
```bash
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
```
```json
{
"TerminatingInstances": [
{
"InstanceId": "i-36765eb0c6609e4d2",
"CurrentState": {
"Code": 16,
"Name": "running"
},
"PreviousState": {
"Code": 16,
"Name": "running"
}
}
]
}
```
Next, validate the instance is removed, this may take a few minutes depending on the instance volume size.
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```
On success no data will be returned, since the instance is no longer available.
## UI Management Panel
The Hive Platform also offers a web interface for managing hive services. Simply have the hive server running and go to [https://localhost:3000](https://localhost:3000) in your browser to continue.
**Note:** If you are not viewing the website on the same machine that is running the hive server, you will need to go to [https://localhost:9999](https://localhost:9999) and [https://localhost:8443](https://localhost:8443) and accept the certificates. If you have added the CA to your local machine this is not needed.
# Multi-Node Configuration
Hive is designed for distributed, multi-server deployments. A Hive cluster operates as a fully distributed infrastructure region — similar to an AWS Region — where multiple nodes work together to provide high availability, data durability, and fault tolerance across the platform.
When deploying a multi-node cluster, you define a **region** (e.g., `ap-southeast-2`, `us-east-1`) and **availability zones** to organize your infrastructure. Services are distributed across nodes: NATS provides a clustered message bus for request routing and JetStream replication, Predastore uses Raft consensus with Reed-Solomon erasure coding (RS 2+1) for durable S3-compatible object storage, and the Hive daemon on each node can independently serve AWS API requests. The result is a platform where compute, storage, and API services are replicated — an AMI stored on one node is available from any node, an EC2 instance launched via any gateway is visible cluster-wide, and the loss of a single node does not compromise data or availability.
Cluster formation is automatic. When initializing with `--nodes 3`, the init node starts a formation server and waits for other nodes to join. Once all nodes have registered, each node receives the full cluster topology — credentials, CA certificates, NATS routes, Predastore peer lists — and generates its own configuration files. No manual configuration synchronization is required.
## Choose a Deployment Mode
| | Simulated (Option A) | Real Multi-Server (Option B) |
|---|---|---|
| **Servers** | 1 machine, loopback IPs | 3+ physical/virtual servers |
| **Use case** | Development & testing | Production & staging |
| **Network** | `127.0.0.1/2/3` (no setup) | Real LAN IPs |
| **Data dirs** | `~/node1/`, `~/node2/`, `~/node3/` | `~/hive/` on each server |
Both options use `HIVE_NODE1`, `HIVE_NODE2`, and `HIVE_NODE3` environment variables. Set them in the option you choose, then all subsequent sections (verification, usage, shutdown) reference them.
### Option A: Simulated Multi-Node (Single Server)
For development and testing, a 3-node cluster can be simulated on a single machine using loopback addresses. Linux handles the full `127.0.0.0/8` range natively, so `127.0.0.1`, `127.0.0.2`, and `127.0.0.3` all work without any network configuration. Each node uses a separate data directory to isolate configuration, logs, and state.
#### 1. Build and Set Variables
```bash
make build
export HIVE_NODE1=127.0.0.1
export HIVE_NODE2=127.0.0.2
export HIVE_NODE3=127.0.0.3
```
#### 2. Form the Cluster
The formation process requires running init and join commands concurrently. The init node blocks until all expected nodes have joined.
**Terminal 1 — Initialize the leader node:**
```bash
./bin/hive admin init \
--node node1 \
--nodes 3 \
--bind $HIVE_NODE1 \
--cluster-bind $HIVE_NODE1 \
--port 4432 \
--hive-dir ~/node1/ \
--config-dir ~/node1/config/ \
--region ap-southeast-2 \
--az ap-southeast-2a
```
This starts a formation server on `$HIVE_NODE1:4432` and waits for 2 more nodes to join.
**Terminal 2 — Join node 2 (while init is running):**
```bash
./bin/hive admin join \
--node node2 \
--bind $HIVE_NODE2 \
--cluster-bind $HIVE_NODE2 \
--host $HIVE_NODE1:4432 \
--data-dir ~/node2/ \
--config-dir ~/node2/config/ \
--region ap-southeast-2 \
--az ap-southeast-2a
```
**Terminal 3 — Join node 3 (while init is running):**
```bash
./bin/hive admin join \
--node node3 \
--bind $HIVE_NODE3 \
--cluster-bind $HIVE_NODE3 \
--host $HIVE_NODE1:4432 \
--data-dir ~/node3/ \
--config-dir ~/node3/config/ \
--region ap-southeast-2 \
--az ap-southeast-2a
```
All three processes will exit with a cluster summary once formation is complete.
#### 3. Start Services
Start services on each node (in separate terminals or background):
```bash
HIVE_SKIP_BUILD=true UI=false ./scripts/start-dev.sh ~/node1/
HIVE_SKIP_BUILD=true UI=false ./scripts/start-dev.sh ~/node2/
HIVE_SKIP_BUILD=true UI=false ./scripts/start-dev.sh ~/node3/
```
### Option B: Real Multi-Node (Physical Servers)
For production or production-like deployments across multiple physical servers or VMs.
#### Prerequisites
On **each server**:
1. Install Hive dependencies (see [Dependencies](#dependencies) above)
2. Clone the repository and build:
```bash
mkdir -p ~/Development/mulga/
cd ~/Development/mulga/
git clone https://github.com/mulgadc/hive.git
cd hive
./scripts/clone-deps.sh
make build
```
3. Identify the bind IP for each server:
```bash
ip addr show | grep "inet "
```
#### 1. Set Variables
Export the IPs for your three servers (replace with your actual IPs) on each server:
```bash
export HIVE_NODE1=192.168.1.10
export HIVE_NODE2=192.168.1.11
export HIVE_NODE3=192.168.1.12
```
#### 2. Form the Cluster
**Server 1 — Initialize the cluster:**
```bash
cd ~/Development/mulga/hive
./bin/hive admin init \
--node node1 \
--nodes 3 \
--bind $HIVE_NODE1 \
--cluster-bind $HIVE_NODE1 \
--port 4432 \
--hive-dir ~/hive/ \
--config-dir ~/hive/config/ \
--region us-east-1 \
--az us-east-1a
```
**Server 2 — Join the cluster (while init is running):**
```bash
cd ~/Development/mulga/hive
./bin/hive admin join \
--node node2 \
--bind $HIVE_NODE2 \
--cluster-bind $HIVE_NODE2 \
--host $HIVE_NODE1:4432 \
--data-dir ~/hive/ \
--config-dir ~/hive/config/ \
--region us-east-1 \
--az us-east-1a
```
**Server 3 — Join the cluster (while init is running):**
```bash
cd ~/Development/mulga/hive
./bin/hive admin join \
--node node3 \
--bind $HIVE_NODE3 \
--cluster-bind $HIVE_NODE3 \
--host $HIVE_NODE1:4432 \
--data-dir ~/hive/ \
--config-dir ~/hive/config/ \
--region us-east-1 \
--az us-east-1a
```
All three processes will exit with a cluster summary once formation is complete.
#### 3. Start Services
After formation completes, start services on **all servers**:
```bash
./scripts/start-dev.sh
```
## Verify the Cluster
From any node, view cluster status and resource capacity:
```bash
./bin/hive get nodes
# NAME STATUS IP REGION AZ UPTIME VMs SERVICES
# node1 Ready $HIVE_NODE1 ... ... 2m 0 nats,predastore,viperblock,daemon,...
# node2 Ready $HIVE_NODE2 ... ... 2m 0 nats,predastore,viperblock,daemon
# node3 Ready $HIVE_NODE3 ... ... 2m 0 nats,predastore,viperblock,daemon
./bin/hive top nodes
# NAME CPU (used/total) MEM (used/total) VMs
# node1 0/16 0.0Gi/30.6Gi 0
# node2 0/16 0.0Gi/30.6Gi 0
# node3 0/16 0.0Gi/30.6Gi 0
```
All nodes should show `Ready` status. Use `hive get vms` to see running instances across the cluster.
Check individual daemon health endpoints:
```bash
curl -s http://$HIVE_NODE1:4432/health
curl -s http://$HIVE_NODE2:4432/health
curl -s http://$HIVE_NODE3:4432/health
```
Check NATS cluster routing (from any node):
```bash
grep -i "route\|cluster" ~/node1/logs/nats.log # simulated
grep -i "route\|cluster" ~/hive/logs/nats.log # real multi-server
```
Check Predastore Raft consensus:
```bash
grep -i "leader\|election" ~/node1/logs/predastore.log # simulated
grep -i "leader\|election" ~/hive/logs/predastore.log # real multi-server
```
## Trust CA Certificate
The CA certificate is generated by the init node and distributed to all joining nodes during formation. On **each node** (or the single machine for simulated mode), trust the CA:
```bash
sudo cp ~/hive/config/ca.pem /usr/local/share/ca-certificates/hive-ca.crt
sudo update-ca-certificates
```
For simulated mode, the config directory is under each node's data directory — copy from any node since the CA is the same:
```bash
sudo cp ~/node1/config/ca.pem /usr/local/share/ca-certificates/hive-ca.crt
sudo update-ca-certificates
```
Set the AWS profile:
```bash
export AWS_PROFILE=hive
```
## Using the Cluster
Connect to any node's AWS Gateway:
```bash
AWS_ENDPOINT_URL=https://$HIVE_NODE1:9999 aws ec2 describe-instances
AWS_ENDPOINT_URL=https://$HIVE_NODE2:9999 aws ec2 describe-instances
```
Check logs on each node for debugging. Log locations depend on your deployment mode — `~/node{1,2,3}/logs/` for simulated, `~/hive/logs/` for real multi-server.
## Advanced Configuration
### Service Co-location
Every node that runs EC2 instances (compute) **must** also run the viperblock service. EBS volumes are mounted locally via Unix domain sockets (NBD) — the daemon publishes to a node-specific NATS topic (`ebs.{nodeName}.mount`) which is handled by the viperblock instance on the same server. Nodes without viperblock cannot run EC2 instances.
In the default deployment, every node runs all services (NATS, Predastore, Viperblock, Daemon, Gateway), which satisfies this requirement.
### Dual NIC Configuration
If servers have separate management and cluster network interfaces, use `--cluster-bind` to specify the cluster network IP:
```bash
# Server 1: management=192.168.1.10, cluster=10.0.0.10
./bin/hive admin init \
--node node1 \
--nodes 3 \
--bind 192.168.1.10 \
--cluster-bind 10.0.0.10 \
--port 4432 \
--hive-dir ~/hive/ \
--config-dir ~/hive/config/ \
--region us-east-1 \
--az us-east-1a
```
The `--bind` IP is used for the formation server, daemon, and AWS gateway. The `--cluster-bind` IP is used for NATS cluster routing and Predastore Raft consensus. If `--cluster-bind` is not specified, it defaults to the `--bind` IP.
### IP Aliases (Simulated Mode)
For a more production-like simulation using dedicated IP addresses on a network interface:
```bash
sudo ip addr add 10.11.12.1/24 dev eth0
sudo ip addr add 10.11.12.2/24 dev eth0
sudo ip addr add 10.11.12.3/24 dev eth0
```
Replace `eth0` with your network interface name (e.g., `enp0s3`, `ens33`). Then export these as your `HIVE_NODE` variables instead of the loopback addresses.
To remove the aliases:
```bash
sudo ip addr del 10.11.12.1/24 dev eth0
sudo ip addr del 10.11.12.2/24 dev eth0
sudo ip addr del 10.11.12.3/24 dev eth0
```
## Shutdown
When running in the multi-server cluster mode a graceful shutdown is required to synchronize and coordinate nodes.
On any node, issue the following command to shutdown nodes cluster wide:
```sh
./bin/hive admin cluster shutdown
```
Example output:
```bash
Starting coordinated cluster shutdown (3 nodes)
Phases: gate -> drain -> storage -> persist -> infra
Timeout per phase: 2m0s
[GATE] Sending to 3 node(s)...
[GATE] node1: stopped awsgw, hive-ui
[GATE] node3: stopped awsgw, hive-ui
[GATE] node2: stopped awsgw, hive-ui
[GATE] Complete (3/3 nodes, 2.005s)
[DRAIN] Sending to 3 node(s)...
[DRAIN] node1: 0/0 VMs remaining
[DRAIN] node1: 0/0 VMs remaining
[DRAIN] node2: 0/0 VMs remaining
[DRAIN] node2: 0/0 VMs remaining
[DRAIN] node3: 0/0 VMs remaining
[DRAIN] node3: 0/0 VMs remaining
[DRAIN] node1: OK
[DRAIN] node3: OK
[DRAIN] node2: OK
[DRAIN] Complete (3/3 nodes, 5ms)
[STORAGE] Sending to 3 node(s)...
[STORAGE] node1: stopped viperblock
[STORAGE] node3: stopped viperblock
[STORAGE] node2: stopped viperblock
[STORAGE] Complete (3/3 nodes, 1.09s)
[PERSIST] Sending to 3 node(s)...
[PERSIST] node1: stopped predastore
[PERSIST] node3: stopped predastore
[PERSIST] node2: stopped predastore
[PERSIST] Complete (3/3 nodes, 1.003s)
[INFRA] Sending final shutdown to all nodes...
[INFRA] Complete
Cluster shutdown complete (6.104s)
```
If EC2 instances are running, the stop process will gracefully terminate them, unmount attached EBS volumes (via NBD), and flush the write-ahead-log (WAL) to the S3 server (Predastore). This may take a few minutes depending on instance volume sizes.
To fully remove a simulated cluster's data:
```bash
rm -rf ~/node1/ ~/node2/ ~/node3/
```
For documentation on using and installing Spinifex see [https://docs.mulgadc.com](https://docs.mulgadc.com/)
+629 -169
View File
@@ -1,201 +1,661 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
1. Definitions.
Preamble
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
The precise terms and conditions for copying, distribution and
modification follow.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
TERMS AND CONDITIONS
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
0. Definitions.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
"This License" refers to version 3 of the GNU Affero General Public License.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
A "covered work" means either the unmodified Program or a work based
on the Program.
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
1. Source Code.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
END OF TERMS AND CONDITIONS
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
APPENDIX: How to apply the Apache License to your work.
The Corresponding Source for a work in source code form is that
same work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
2. Basic Permissions.
Copyright [yyyy] [name of copyright owner]
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
http://www.apache.org/licenses/LICENSE-2.0
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.
+121 -73
View File
@@ -1,4 +1,4 @@
GO_PROJECT_NAME := hive
GO_PROJECT_NAME := spx
SHELL := /bin/bash
# Detect architecture for cross-platform support
@@ -33,32 +33,79 @@ else
GO_BUILD_MOD :=
endif
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
ifdef QUIET
_Q = @
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
_SECQ = >
else
_Q =
_COVQ =
_RACEQ =
_SECQ = 2>&1 | tee
endif
build:
$(MAKE) go_build
# Build hive-ui frontend (requires pnpm)
# Build spinifex-ui frontend (requires pnpm)
build-ui:
@echo -e "\n....Building hive-ui frontend...."
cd hive/services/hiveui/frontend && pnpm build
@echo -e "\n....Building spinifex-ui frontend...."
cd spinifex/services/spinifexui/frontend && pnpm build
# GO commands
VERSION ?= $(shell git describe --tags --always --dirty)
COMMIT ?= $(shell git rev-parse --short HEAD)
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
go_build:
@echo -e "\n....Building $(GO_PROJECT_NAME)"
go build $(GO_BUILD_MOD) -ldflags "-s -w" -o ./bin/$(GO_PROJECT_NAME) cmd/hive/main.go
go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
build-lb-agent:
@echo -e "\n....Building lb-agent (static)"
CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
build-system-image: ## Build a system image from manifest (use IMAGE=lb)
ifndef IMAGE
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
endif
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
build-lb-image: ## Build LB Alpine image
$(MAKE) build-system-image IMAGE=lb
go_run:
@echo -e "\n....Running $(GO_PROJECT_NAME)...."
$(GOPATH)/bin/$(GO_PROJECT_NAME)
# Preflight — runs the same checks as GitHub Actions (format + lint + security + tests).
# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
# Use this before committing to catch CI failures locally.
preflight: check-format vet security-check test
preflight:
@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
@echo -e "\n ✅ Preflight passed — safe to commit."
# Run unit tests
test:
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
LOG_IGNORE=1 go test -v -timeout 120s ./...
LOG_IGNORE=1 go test -timeout 120s ./spinifex/...
# Run unit tests with coverage profile
COVERPROFILE ?= coverage.out
test-cover:
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... $(_COVQ)
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
# Run unit tests with race detector
test-race:
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
# Check that new/changed code meets coverage threshold (runs tests first)
diff-coverage: test-cover
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
bench:
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
@@ -69,22 +116,36 @@ run:
$(MAKE) go_build
$(MAKE) go_run
# Fast iteration: build + install binary + restart all services
deploy: build
sudo install -m 755 bin/spx /usr/local/bin/spx
sudo systemctl daemon-reload
sudo systemctl restart spinifex.target
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
# Not needed for code-only changes — use deploy for those.
reinstall:
scripts/dev-install.sh
clean:
rm -f ./bin/$(GO_PROJECT_NAME)
rm -rf hive/services/hiveui/frontend/dist
rm -rf spinifex/services/spinifexui/frontend/dist
install-system:
@echo -e "\n....Installing system dependencies for $(ARCH)...."
@echo "QEMU packages: $(QEMU_PACKAGES)"
apt-get update && sudo apt-get install -y \
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confold" \
nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
ovn-central ovn-host openvswitch-switch dhcpcd-base
install-go:
@echo -e "\n....Installing Go 1.26.0 for $(ARCH) ($(GO_ARCH))...."
@echo -e "\n....Installing Go 1.26.2 for $(ARCH) ($(GO_ARCH))...."
@if [ ! -d "/usr/local/go" ]; then \
curl -L https://go.dev/dl/go1.26.0.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
curl -L https://go.dev/dl/go1.26.2.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
else \
echo "Go already installed in /usr/local/go"; \
fi
@@ -106,72 +167,59 @@ quickinstall: install-system install-go install-aws
@echo " Please ensure /usr/local/go/bin is in your PATH."
@echo " Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
# Format all Go files in place
format:
gofmt -w .
# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
lint:
@echo "Running golangci-lint..."
$(_Q)golangci-lint run ./...
@echo " golangci-lint ok"
# Check that all Go files are formatted (CI-compatible, fails on diff)
check-format:
@echo "Checking gofmt..."
@UNFORMATTED=$$(gofmt -l .); \
if [ -n "$$UNFORMATTED" ]; then \
echo "Files not formatted:"; \
echo "$$UNFORMATTED"; \
echo "Run 'make format' to fix."; \
exit 1; \
fi
@echo " gofmt ok"
# Auto-fix all linter issues that have fixers
fix:
golangci-lint run --fix ./...
# Go vet (fails on issues, matches CI)
vet:
@echo "Running go vet..."
go vet ./...
@echo " go vet ok"
# Security checks — each tool fails the build on findings (matches CI).
# Reports are also saved to tests/ for review.
security-check:
@echo -e "\n....Running security checks for $(GO_PROJECT_NAME)...."
set -o pipefail && go tool govulncheck ./... 2>&1 | tee tests/govulncheck-report.txt
# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
govulncheck:
@echo "Running govulncheck..."
$(_Q)go tool govulncheck ./...
@echo " govulncheck ok"
set -o pipefail && go tool gosec -exclude=G204,G304,G402 -exclude-generated -exclude-dir=cmd ./... 2>&1 | tee tests/gosec-report.txt
@echo " gosec ok"
set -o pipefail && go tool staticcheck -checks="all,-ST1000,-ST1003,-ST1016,-ST1020,-ST1021,-ST1022,-SA1019,-SA9005" ./... 2>&1 | tee tests/staticcheck-report.txt
@echo " staticcheck ok"
# Docker E2E tests (mirrors GitHub Actions e2e.yml)
# Usage: make test-docker # both suites
# make test-docker-single # single-node only
# make test-docker-multi # multi-node only
PARENT_DIR := $(shell cd .. && pwd)
E2E_IMAGE := hive-e2e:latest
# Build release tarballs — use distro-ARCH for single arch, distro for both
distro: distro-amd64 distro-arm64
@echo ""
@echo "Distribution tarballs:"
@ls -lh dist/*.tar.gz
@echo ""
@cat dist/*.sha256
test-docker-build:
@echo -e "\n....Building E2E Docker image...."
@for dep in viperblock predastore; do \
if [ ! -d "$(PARENT_DIR)/$$dep" ]; then \
echo "Missing sibling repo $$dep — running clone-deps.sh"; \
./scripts/clone-deps.sh; \
break; \
fi; \
done
docker build -t $(E2E_IMAGE) -f tests/e2e/Dockerfile.e2e $(PARENT_DIR)
distro-amd64:
@echo "Building spinifex $(VERSION) linux/amd64..."
@mkdir -p dist/
docker buildx build \
--platform linux/amd64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/amd64/ \
../
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
test-docker-single: test-docker-build
@echo -e "\n....Running Single-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm $(E2E_IMAGE)
distro-arm64:
@echo "Building spinifex $(VERSION) linux/arm64..."
@mkdir -p dist/
docker buildx build \
--platform linux/arm64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/arm64/ \
../
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
test-docker-multi: test-docker-build
@echo -e "\n....Running Multi-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm --cap-add=NET_ADMIN $(E2E_IMAGE) ./tests/e2e/run-multinode-e2e.sh
distro-clean:
rm -rf dist/
test-docker: test-docker-build
@echo -e "\n....Running Single-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm $(E2E_IMAGE)
@echo -e "\n....Running Multi-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm --cap-add=NET_ADMIN $(E2E_IMAGE) ./tests/e2e/run-multinode-e2e.sh
.PHONY: build build-ui go_build go_run preflight test bench run clean \
.PHONY: build build-ui build-lb-agent build-system-image build-lb-image go_build go_run preflight test test-cover test-race diff-coverage bench run \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
format check-format vet security-check \
test-docker-build test-docker-single test-docker-multi test-docker
lint fix govulncheck \
distro distro-amd64 distro-arm64 distro-clean
+89 -35
View File
@@ -1,25 +1,55 @@
# Hive: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
<p align="center">
<img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
</p>
(Developer Preview) - Hive developed by [Mulga Defense Corporation](https://mulgadc.com/) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Hive gives you AWS-style workflows on your own hardware.
<p align="center">
<a href="https://go.dev/"><img src="https://img.shields.io/badge/Go-1.22+-00ADD8?style=flat-square&logo=go&logoColor=white" alt="Go"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-3fb950?style=flat-square" alt="License"></a>
<a href="https://docs.mulgadc.com"><img src="https://img.shields.io/badge/Docs-mulgadc.com-DC2626?style=flat-square" alt="Docs"></a>
<a href="https://github.com/mulgadc/spinifex/actions"><img src="https://img.shields.io/badge/CI-e2e-58a6ff?style=flat-square" alt="CI"></a>
</p>
## What is Hive?
<p align="center">
<a href="#what-is-spinifex">What is Spinifex?</a> ·
<a href="#the-platform">Platform</a> ·
<a href="#core-components">Components</a> ·
<a href="#architecture-at-a-glance">Architecture</a> ·
<a href="#installation">Installation</a> ·
<a href="https://docs.mulgadc.com">Docs</a>
</p>
Hive replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
---
It’s designed for developers and operators who need:
# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
## What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
It's designed for developers and operators who need:
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
You can run Hive on a few servers in a rack, a field site, or anywhere centralized cloud services aren’t feasible.
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
## The Platform
<p align="center">
<img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
## Core Components
### Hive (Compute Service – EC2 Alternative)
### Spinifex (Compute Service – EC2 Alternative)
Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU’s QMP interface. Designed to be straightforward and scriptable, Hive lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Hive with underlying instances.
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
@@ -27,7 +57,7 @@ Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs sim
### Viperblock (Block Storage – EBS Alternative)
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It’s built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
@@ -43,53 +73,83 @@ Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs sim
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
## Architecture at a Glance
<p align="center">
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
</p>
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
- Designed for Control – Run on your own terms, whether that’s in a datacenter, remote site, or sensitive environment.
- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
- Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under Apache 2.0. Fork it, extend it, or deploy it as-is.
- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
## Development Setup
## Installation
### For Developers
Installation requires an Ubuntu / Debian system. See the detailed documentation at [https://docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
Get started with Hive development in minutes:
### Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes.
```bash
# Clone the repository
git clone https://github.com/mulgadc/hive.git
cd hive
curl -fsSL https://install.mulgadc.com | bash
# Setup dependencies and development environment
./scripts/clone-deps.sh # Clone viperblock + predastore repositories
./scripts/dev-setup.sh # Setup complete development environment
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
Next, follow the detailed [Installation Guide](./INSTALL.md) to bootstrap a new environment with SSH keys, AMI (OS images) and how to spin up new EC2 instances using Hive.
### Development Setup
For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation
### Component Repositories
Hive coordinates these independent components:
Spinifex coordinates these independent components:
- **[Predastore](https://github.com/mulgadc/predastore)** - S3-compatible object storage
- **[Viperblock](https://github.com/mulgadc/viperblock)** - EBS-compatible block storage
Each component can be developed independently. See component-specific documentation for focused development guides.
## Detailed Installation Method
## Spinifex UI
See [Installation Guide](INSTALL.md) on how to setup a Hive node to get started.
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
Then configure your AWS CLI to point to Hive's endpoints and start launching VMs, buckets, and volumes using familiar commands and Terraform scripts.
<p align="center">
<img src="docs/spinifex-ui.jpg" alt="Spinifex web console — dashboard view" width="900">
</p>
The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
```bash
open https://YOUR_NODE_IP:3000
```
- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
## Development Philosophy
### Built by Engineers, For Engineers
Hive is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
**Real-World Experience:**
@@ -100,7 +160,7 @@ Hive is developed by experienced infrastructure engineers with deep AWS expertis
### AI-Assisted Development
While Hive is architected and implemented by experienced engineers, we leverage **Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
While Spinifex is architected and implemented by experienced engineers, we leverage **Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
**How We Use Claude Code:**
@@ -116,14 +176,8 @@ While Hive is architected and implemented by experienced engineers, we leverage
- **Performance Optimization**: Real-world performance tuning and benchmarking
- **Production Operations**: Deployment strategies and operational procedures
**Development Artifacts:**
- `.claude/CLAUDE.md` - Instructions for Claude Code when working on this codebase
- `HIVE_DEVELOPMENT_PLAN.md` - Comprehensive roadmap and implementation strategy
- Component-specific guidance in `viperblock/CLAUDE.md` and `predastore/CLAUDE.md`
This hybrid approach ensures Hive benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## License
Hive is open source under the Apache 2.0 License. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.
+49
View File
@@ -0,0 +1,49 @@
ARG VERSION=dev
FROM golang:1.26-bookworm AS builder
ARG VERSION
RUN apt-get update && apt-get install -y --no-install-recommends \
nbdkit nbdkit-plugin-dev pkg-config gcc make \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Copy module files for dependency caching
COPY spinifex/go.mod spinifex/go.sum ./spinifex/
COPY predastore/go.mod predastore/go.sum ./predastore/
COPY viperblock/go.mod viperblock/go.sum ./viperblock/
COPY viperblock/nbd/go.mod viperblock/nbd/go.sum ./viperblock/nbd/
COPY viperblock/nbd/libguestfs.org/nbdkit/go.mod ./viperblock/nbd/libguestfs.org/nbdkit/
# Generate go.work (gitignored, so not in checkout)
RUN cd spinifex && go work init . ../predastore ../viperblock ../viperblock/nbd ../viperblock/nbd/libguestfs.org/nbdkit
# Download dependencies (cached layer)
RUN cd spinifex && go mod download
# Copy full source
COPY spinifex/ ./spinifex/
COPY predastore/ ./predastore/
COPY viperblock/ ./viperblock/
# Build spinifex binary (spx)
RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
cd spinifex && go build -ldflags "-s -w \
-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=${VERSION} \
-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=${COMMIT}" \
-o /out/spx cmd/spinifex/main.go
# Build nbdkit viperblock plugin (.so)
RUN cd viperblock && go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
# Copy setup scripts and systemd units
RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true
RUN cp spinifex/scripts/setup-ovn.sh /out/setup-ovn.sh
RUN mkdir -p /out/systemd && cp spinifex/build/systemd/* /out/systemd/
RUN mkdir -p /out/scripts && cp spinifex/build/scripts/* /out/scripts/
RUN cp spinifex/build/logrotate/spinifex /out/logrotate-spinifex
FROM scratch AS export
COPY --from=builder /out/ /
+9
View File
@@ -0,0 +1,9 @@
/var/log/spinifex/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
copytruncate
}
+29
View File
@@ -0,0 +1,29 @@
#!/bin/sh
# Helper script for spinifex-predastore.service
# Detects host, port, backend, and node-id from spinifex.toml
# then execs spx. Falls back to single-node defaults if no config found.
CONF=/etc/spinifex/spinifex.toml
BIND="0.0.0.0:8443"
BACKEND="filesystem"
NODE_ID="0"
if [ -f "$CONF" ]; then
H=$(awk -F'"' '/\[nodes\..*\.predastore\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
if [ -n "$H" ]; then
BIND="$H"
BACKEND="distributed"
fi
N=$(awk -F'= *' '/node_id/{gsub(/ /,"",$2);print $2;exit}' "$CONF")
if [ -n "$N" ] && [ "$N" != "0" ]; then
NODE_ID="$N"
fi
fi
export SPINIFEX_PREDASTORE_HOST="${BIND%%:*}"
export SPINIFEX_PREDASTORE_PORT="${BIND##*:}"
export SPINIFEX_PREDASTORE_BACKEND="$BACKEND"
export SPINIFEX_PREDASTORE_NODE_ID="$NODE_ID"
exec /usr/local/bin/spx service predastore start
+37
View File
@@ -0,0 +1,37 @@
#!/bin/sh
# Wait for NATS to be reachable before starting a dependent service.
# Reads the NATS address from spinifex.toml [nodes.*.nats] section,
# falls back to nats.conf, then to 127.0.0.1:4222.
CONF=/etc/spinifex/spinifex.toml
NATS_CONF=/etc/spinifex/nats/nats.conf
NATS_HOST=""
NATS_PORT="4222"
TIMEOUT=30
# Try spinifex.toml first
if [ -f "$CONF" ]; then
NATS_BIND=$(awk -F'"' '/\[nodes\..*\.nats\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
if [ -n "$NATS_BIND" ]; then
NATS_HOST="${NATS_BIND%%:*}"
NATS_PORT="${NATS_BIND##*:}"
fi
fi
# Fall back to nats.conf
if [ -z "$NATS_HOST" ] && [ -f "$NATS_CONF" ]; then
NATS_HOST=$(grep "^listen:" "$NATS_CONF" | sed 's/listen: *//;s/:.*//')
fi
# Final fallback
NATS_HOST="${NATS_HOST:-127.0.0.1}"
for i in $(seq 1 "$TIMEOUT"); do
if nc -z "$NATS_HOST" "$NATS_PORT" 2>/dev/null; then
exit 0
fi
sleep 1
done
echo "Timed out waiting for NATS at $NATS_HOST:$NATS_PORT" >&2
exit 1
+53
View File
@@ -0,0 +1,53 @@
[Unit]
Description=Spinifex AWS Gateway
PartOf=spinifex.target
After=spinifex-nats.service spinifex-daemon.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-gw
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service awsgw start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/awsgw/
Environment=SPINIFEX_AWSGW_TLS_CERT=/etc/spinifex/server.pem
Environment=SPINIFEX_AWSGW_TLS_KEY=/etc/spinifex/server.key
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key
ReadWritePaths=/var/lib/spinifex/awsgw /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+54
View File
@@ -0,0 +1,54 @@
[Unit]
Description=Spinifex Daemon (VM Orchestration)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-viperblock.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-daemon
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service spinifex start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
ProtectSystem=full
PrivateDevices=no
DevicePolicy=closed
DeviceAllow=/dev/kvm rw
DeviceAllow=/dev/net/tun rw
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key
ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
# AF_PACKET is required by dhcpcd (raw L2 BOOTP before the iface has an IP).
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+48
View File
@@ -0,0 +1,48 @@
[Unit]
Description=Spinifex NATS Message Bus
PartOf=spinifex.target
Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service
[Service]
Type=simple
User=spinifex-nats
Group=spinifex
ExecStart=/usr/local/bin/spx service nats start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/nats/nats.conf
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/nats
ReadWritePaths=/var/lib/spinifex/nats /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+53
View File
@@ -0,0 +1,53 @@
[Unit]
Description=Spinifex Predastore (S3-compatible Object Storage)
PartOf=spinifex.target
After=spinifex-nats.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-storage
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/var/lib/spinifex/predastore-start.sh
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_PREDASTORE_CONFIG_PATH=/etc/spinifex/predastore/predastore.toml
Environment=SPINIFEX_PREDASTORE_BASE_PATH=/var/lib/spinifex/predastore/
Environment=SPINIFEX_PREDASTORE_TLS_CERT=/etc/spinifex/server.pem
Environment=SPINIFEX_PREDASTORE_TLS_KEY=/etc/spinifex/server.key
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/predastore /etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key
ReadWritePaths=/var/lib/spinifex/predastore /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+49
View File
@@ -0,0 +1,49 @@
[Unit]
Description=Spinifex Web UI
PartOf=spinifex.target
After=spinifex-awsgw.service
[Service]
Type=simple
User=spinifex-ui
Group=spinifex
ExecStart=/usr/local/bin/spx service spinifex-ui start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_UI_TLS_CERT=/etc/spinifex/server.pem
Environment=SPINIFEX_UI_TLS_KEY=/etc/spinifex/server.key
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/server.pem /etc/spinifex/server.key
ReadWritePaths=/var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+52
View File
@@ -0,0 +1,52 @@
[Unit]
Description=Spinifex Viperblock (Block Storage)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-predastore.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-viperblock
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service viperblock start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
EnvironmentFile=/etc/spinifex/systemd.env
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/viperblock/
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/systemd.env
ReadWritePaths=/var/lib/spinifex/viperblock /var/log/spinifex /run/spinifex/nbd
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+49
View File
@@ -0,0 +1,49 @@
[Unit]
Description=Spinifex VPC Daemon (OVN Networking)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-daemon.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-vpcd
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service vpcd start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
# Filesystem access — NoNewPrivileges omitted (vpcd uses sudo for ovs-vsctl, ovn-nbctl, systemctl)
ProtectSystem=full
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/spinifex.toml
ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+8
View File
@@ -0,0 +1,8 @@
[Unit]
Description=Spinifex Infrastructure Platform
After=network-online.target
Wants=network-online.target
Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service
[Install]
WantedBy=multi-user.target
File diff suppressed because it is too large Load Diff
-13
View File
@@ -1,13 +0,0 @@
# AWS Gateway configuration file in TOML format.
version = "1.0"
region = "{{.Region}}"
# Enable verbose debugging
debug = false
[[auth]]
access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
-61
View File
@@ -1,61 +0,0 @@
# Domain configuration file in TOML format.
version = "1.0"
epoch = 1
node = "{{.Node}}"
[nodes.{{.Node}}]
node = "{{.Node}}"
host = "{{.BindIP}}"
region = "{{.Region}}"
az = "{{.Az}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
account_id = "{{.AccountID}}"
accesskey = "{{.AccessKey}}"
secretkey = "{{.SecretKey}}"
[nodes.{{.Node}}.daemon]
host = "{{.BindIP}}:{{.Port}}"
tlskey = "config/server.key"
tlscert = "config/server.pem"
[nodes.{{.Node}}.awsgw]
host = "{{.BindIP}}:9999"
tlskey = "config/server.key"
tlscert = "config/server.pem"
debug = true
config = "awsgw/awsgw.toml"
[nodes.{{.Node}}.nats]
host = "{{.BindIP}}:4222"
[nodes.{{.Node}}.nats.acl]
token = "{{.NatsToken}}"
[nodes.{{.Node}}.nats.sub]
[nodes.{{.Node}}.viperblock]
shardwal = false
[nodes.{{.Node}}.predastore]
host = "{{.BindIP}}:8443"
region = "{{.Region}}"
bucket = "predastore"
accesskey = "{{.AccessKey}}"
secretkey = "{{.SecretKey}}"
{{- if gt .PredastoreNodeID 0}}
node_id = {{.PredastoreNodeID}}
{{- end}}
{{- range .RemoteNodes}}
[nodes.{{.Name}}]
node = "{{.Name}}"
host = "{{.Host}}"
region = "{{.Region}}"
az = "{{.AZ}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
{{- end}}
-22
View File
@@ -1,22 +0,0 @@
/*
Copyright © 2025 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
import "github.com/mulgadc/hive/cmd/hive/cmd"
func main() {
cmd.Execute()
}
+90
View File
@@ -0,0 +1,90 @@
// lb-agent runs inside load balancer VMs and manages HAProxy configuration.
//
// It polls the AWS gateway for config updates and reports health via heartbeats.
// All communication uses SigV4-signed HTTP requests.
//
// Usage:
//
// lb-agent --lb-id=lb-xxxxx --gateway=https://192.168.1.33:9999 --access-key=AKIA... --secret-key=...
//
// Flags default to environment variables set by cloud-init:
// LB_LB_ID, LB_GATEWAY_URL, LB_ACCESS_KEY, LB_SECRET_KEY, LB_REGION.
package main
import (
"flag"
"log/slog"
"os"
"os/signal"
"syscall"
"github.com/mulgadc/spinifex/spinifex/lbagent"
)
func main() {
var (
lbID string
gatewayURL string
accessKey string
secretKey string
region string
)
hostname, _ := os.Hostname()
flag.StringVar(&lbID, "lb-id", envOrDefault("LB_LB_ID", hostname), "Load balancer ID")
flag.StringVar(&gatewayURL, "gateway", os.Getenv("LB_GATEWAY_URL"), "Gateway URL (e.g. https://192.168.1.33:9999)")
flag.StringVar(&accessKey, "access-key", os.Getenv("LB_ACCESS_KEY"), "AWS access key ID")
flag.StringVar(&secretKey, "secret-key", os.Getenv("LB_SECRET_KEY"), "AWS secret access key")
flag.StringVar(&region, "region", envOrDefault("LB_REGION", "us-east-1"), "AWS region for SigV4 signing")
flag.Parse()
if lbID == "" {
slog.Error("--lb-id is required (or set LB_LB_ID)")
os.Exit(1)
}
if gatewayURL == "" {
slog.Error("--gateway is required (or set LB_GATEWAY_URL)")
os.Exit(1)
}
if accessKey == "" || secretKey == "" {
slog.Error("--access-key and --secret-key are required (or set LB_ACCESS_KEY / LB_SECRET_KEY)")
os.Exit(1)
}
slog.Info("Starting LB agent", "lbId", lbID, "gateway", gatewayURL)
agent, err := lbagent.New(lbID, gatewayURL, accessKey, secretKey, region)
if err != nil {
slog.Error("Failed to create agent", "err", err)
os.Exit(1)
}
// Start poll loop in a goroutine — it blocks until Stop is called.
errCh := make(chan error, 1)
go func() {
if err := agent.Start(); err != nil {
errCh <- err
}
}()
// Wait for shutdown signal or fatal error.
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
select {
case sig := <-sigCh:
slog.Info("Received signal, shutting down", "signal", sig)
case err := <-errCh:
slog.Error("Agent error", "err", err)
}
agent.Stop()
}
func envOrDefault(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
File diff suppressed because it is too large Load Diff
+92
View File
@@ -0,0 +1,92 @@
package cmd
import (
"bufio"
"fmt"
"os"
"strings"
"github.com/mulgadc/spinifex/spinifex/migrate"
"github.com/spf13/cobra"
)
var upgradeCmd = &cobra.Command{
Use: "upgrade",
Short: "Apply pending config migrations",
Long: `Apply pending configuration file migrations for upgrades between Spinifex versions.
When run interactively (without --yes), shows pending migrations and prompts
for confirmation. When called from setup.sh with --yes, applies immediately.
Operators can skip migrations during install by setting INSTALL_SPINIFEX_SKIP_MIGRATE=1,
then run 'spx admin upgrade' manually to review and apply.`,
Run: runAdminUpgrade,
}
func runAdminUpgrade(cmd *cobra.Command, _ []string) {
configDir, _ := cmd.Root().Flags().GetString("config-dir")
dataDir, _ := cmd.Root().Flags().GetString("spinifex-dir")
yes, _ := cmd.Flags().GetBool("yes")
// Check that the installation exists.
spinifexToml := configDir + "/spinifex.toml"
if _, err := os.Stat(spinifexToml); os.IsNotExist(err) {
fmt.Fprintf(os.Stderr, "No Spinifex installation found at %s\nRun 'spx admin init' first.\n", configDir)
os.Exit(1)
}
// Show current versions.
versions, err := migrate.DefaultRegistry.ConfigVersions(configDir)
if err != nil {
fmt.Fprintf(os.Stderr, "Error reading config versions: %v\n", err)
os.Exit(1)
}
fmt.Println("Reading current config versions...")
for name, v := range versions {
if v == 0 {
fmt.Printf(" %-20s %d (no version marker)\n", name+":", v)
} else {
fmt.Printf(" %-20s %d\n", name+":", v)
}
}
fmt.Println()
// Check for pending migrations.
pending, err := migrate.DefaultRegistry.PendingConfig(configDir)
if err != nil {
fmt.Fprintf(os.Stderr, "Error checking pending migrations: %v\n", err)
os.Exit(1)
}
if len(pending) == 0 {
fmt.Println("No pending config migrations.")
return
}
fmt.Println("Pending config migrations:")
for _, p := range pending {
fmt.Printf(" [config] %s %d → %d: %s\n", p.Target, p.FromVersion, p.ToVersion, p.Description)
}
fmt.Println()
// Prompt for confirmation unless --yes.
if !yes {
fmt.Print("Apply? [y/N] ")
reader := bufio.NewReader(os.Stdin)
answer, _ := reader.ReadString('\n')
answer = strings.TrimSpace(strings.ToLower(answer))
if answer != "y" && answer != "yes" {
fmt.Println("Aborted.")
return
}
}
// Apply migrations.
if err := migrate.DefaultRegistry.RunAllConfig(configDir, dataDir); err != nil {
fmt.Fprintf(os.Stderr, "Migration failed: %v\n", err)
os.Exit(1)
}
fmt.Println("\nDone. Restart services to apply: sudo systemctl restart spinifex.target")
}
@@ -7,7 +7,7 @@ import (
"strings"
"time"
"github.com/mulgadc/hive/hive/daemon"
"github.com/mulgadc/spinifex/spinifex/daemon"
"github.com/nats-io/nats.go"
"github.com/spf13/cobra"
)
@@ -77,7 +77,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
// Execute phases sequentially (except INFRA which is fire-and-forget)
for _, phase := range phases {
topic := "hive.cluster.shutdown." + phase
topic := "spinifex.cluster.shutdown." + phase
req := daemon.ShutdownRequest{
Phase: phase,
Force: force,
@@ -106,7 +106,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
// For DRAIN phase, subscribe to progress updates
var progressSub *nats.Subscription
if phase == "drain" {
progressSub, err = nc.Subscribe("hive.cluster.shutdown.progress", func(msg *nats.Msg) {
progressSub, err = nc.Subscribe("spinifex.cluster.shutdown.progress", func(msg *nats.Msg) {
var progress daemon.ShutdownProgress
if err := json.Unmarshal(msg.Data, &progress); err != nil {
return
+56 -23
View File
@@ -1,17 +1,18 @@
/*
Copyright © 2025 Mulga Defense Corporation
Copyright © 2026 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
http://www.apache.org/licenses/LICENSE-2.0
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package cmd
@@ -19,12 +20,15 @@ import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/utils"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/mulgadc/spinifex/spinifex/types"
"github.com/mulgadc/spinifex/spinifex/utils"
"github.com/nats-io/nats.go"
"github.com/pterm/pterm"
"github.com/spf13/cobra"
@@ -64,8 +68,7 @@ func init() {
func loadConfigAndConnect() (*config.ClusterConfig, *nats.Conn, error) {
cfgPath := viper.GetString("config")
if cfgPath == "" {
homeDir, _ := os.UserHomeDir()
cfgPath = fmt.Sprintf("%s/hive/config/hive.toml", homeDir)
cfgPath = DefaultConfigFile()
}
cfg, err := config.LoadConfig(cfgPath)
@@ -73,6 +76,19 @@ func loadConfigAndConnect() (*config.ClusterConfig, *nats.Conn, error) {
return nil, nil, fmt.Errorf("failed to load config: %w", err)
}
// Backfill BaseDir from config file path when not set in the config.
if nodeConfig, ok := cfg.Nodes[cfg.Node]; ok && nodeConfig.BaseDir == "" {
if isProductionLayout() {
// Production: BaseDir is /var/lib/spinifex (data dir).
// /var/lib/spinifex/config is a symlink to /etc/spinifex.
nodeConfig.BaseDir = DefaultDataDir()
} else {
// Dev: config lives at <baseDir>/config/spinifex.toml, so baseDir is two levels up.
nodeConfig.BaseDir = filepath.Dir(filepath.Dir(cfgPath))
}
cfg.Nodes[cfg.Node] = nodeConfig
}
nodeConfig := cfg.Nodes[cfg.Node]
nc, err := utils.ConnectNATS(nodeConfig.NATS.Host, nodeConfig.NATS.ACL.Token)
if err != nil {
@@ -130,6 +146,20 @@ func formatUptime(seconds int64) string {
return fmt.Sprintf("%dm", mins)
}
func formatRoles(resp types.NodeStatusResponse) string {
var roles []string
if resp.NATSRole != "" {
roles = append(roles, "nats:"+resp.NATSRole)
}
if resp.PredastoreRole != "" {
roles = append(roles, "predastore:"+resp.PredastoreRole)
}
if len(roles) == 0 {
return "-"
}
return strings.Join(roles, ",")
}
func formatMemGB(gb float64) string {
if gb >= 1.0 {
return fmt.Sprintf("%.1fGi", gb)
@@ -146,16 +176,16 @@ func runGetNodes(cmd *cobra.Command, args []string) {
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.status", timeout)
responses, err := collectResponses(nc, "spinifex.node.status", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
// Parse responses into a map by node name
respondedNodes := make(map[string]config.NodeStatusResponse)
respondedNodes := make(map[string]types.NodeStatusResponse)
for _, data := range responses {
var resp config.NodeStatusResponse
var resp types.NodeStatusResponse
if err := json.Unmarshal(data, &resp); err != nil {
continue
}
@@ -164,7 +194,7 @@ func runGetNodes(cmd *cobra.Command, args []string) {
// Build table: union of config-known nodes + NATS responders
tableData := pterm.TableData{
{"NAME", "STATUS", "IP", "REGION", "AZ", "UPTIME", "VMs", "SERVICES"},
{"NAME", "STATUS", "ROLES", "IP", "REGION", "AZ", "UPTIME", "VMs", "SERVICES"},
}
// Collect all node names: config + responded (union)
@@ -187,17 +217,19 @@ func runGetNodes(cmd *cobra.Command, args []string) {
tableData = append(tableData, []string{
resp.Node,
resp.Status,
formatRoles(resp),
resp.Host,
resp.Region,
resp.AZ,
formatUptime(resp.Uptime),
fmt.Sprintf("%d", resp.VMCount),
strconv.Itoa(resp.VMCount),
strings.Join(resp.Services, ","),
})
} else {
tableData = append(tableData, []string{
name,
"NotReady",
"-",
nodeCfg.Host,
nodeCfg.Region,
nodeCfg.AZ,
@@ -220,21 +252,22 @@ func runGetVMs(cmd *cobra.Command, args []string) {
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.vms", timeout)
responses, err := collectResponses(nc, "spinifex.node.vms", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
type vmRow struct {
config.VMInfo
types.VMInfo
Node string
Host string
}
var allVMs []vmRow
for _, data := range responses {
var resp config.NodeVMsResponse
var resp types.NodeVMsResponse
if err := json.Unmarshal(data, &resp); err != nil {
continue
}
@@ -269,7 +302,7 @@ func runGetVMs(cmd *cobra.Command, args []string) {
v.InstanceID,
v.Status,
v.InstanceType,
fmt.Sprintf("%d", v.VCPU),
strconv.Itoa(v.VCPU),
formatMemGB(v.MemoryGB),
v.Node,
v.Host,
+49
View File
@@ -0,0 +1,49 @@
package cmd
import (
"testing"
"github.com/mulgadc/spinifex/spinifex/types"
"github.com/stretchr/testify/assert"
)
func TestFormatRoles(t *testing.T) {
tests := []struct {
name string
resp types.NodeStatusResponse
want string
}{
{
name: "no roles",
resp: types.NodeStatusResponse{},
want: "-",
},
{
name: "nats leader only",
resp: types.NodeStatusResponse{NATSRole: "leader"},
want: "nats:leader",
},
{
name: "predastore follower only",
resp: types.NodeStatusResponse{PredastoreRole: "follower"},
want: "predastore:follower",
},
{
name: "both roles",
resp: types.NodeStatusResponse{NATSRole: "leader", PredastoreRole: "follower"},
want: "nats:leader,predastore:follower",
},
{
name: "both leaders",
resp: types.NodeStatusResponse{NATSRole: "leader", PredastoreRole: "leader"},
want: "nats:leader,predastore:leader",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := formatRoles(tt.resp)
assert.Equal(t, tt.want, got)
})
}
}
+66
View File
@@ -0,0 +1,66 @@
package cmd
import (
"os"
"os/user"
"path/filepath"
)
// DefaultConfigDir returns the default configuration directory.
// Production: /etc/spinifex (when /etc/spinifex exists)
// Development: ~/spinifex/config
func DefaultConfigDir() string {
if isProductionLayout() {
return "/etc/spinifex"
}
return filepath.Join(realUserHomeDir(), "spinifex", "config")
}
// DefaultDataDir returns the default data directory.
// Production: /var/lib/spinifex
// Development: ~/spinifex
func DefaultDataDir() string {
if isProductionLayout() {
return "/var/lib/spinifex"
}
return filepath.Join(realUserHomeDir(), "spinifex")
}
// LogDirFor returns the log directory for a given data directory.
// Production: /var/log/spinifex (matches systemd ReadWritePaths)
// Development: <dataDir>/logs (supports custom per-node data dirs)
func LogDirFor(dataDir string) string {
if isProductionLayout() {
return "/var/log/spinifex"
}
return filepath.Join(dataDir, "logs")
}
// realUserHomeDir returns the home directory of the real (non-sudo) user.
// When running under sudo, SUDO_USER is set to the invoking user — resolve
// their home directory so config/data land in the right place.
func realUserHomeDir() string {
if sudoUser := os.Getenv("SUDO_USER"); sudoUser != "" {
if u, err := user.Lookup(sudoUser); err == nil {
return u.HomeDir
}
}
homeDir, _ := os.UserHomeDir()
return homeDir
}
// DefaultConfigFile returns the default path to spinifex.toml.
func DefaultConfigFile() string {
return filepath.Join(DefaultConfigDir(), "spinifex.toml")
}
// isProductionLayout returns true when running in a production install.
// Detected by: /etc/spinifex directory exists (created by setup.sh during
// binary install). No root check — allows non-root users (e.g. tf-user) to
// run CLI commands like `spx get nodes` without sudo or --config flags.
func isProductionLayout() bool {
if info, err := os.Stat("/etc/spinifex"); err == nil && info.IsDir() {
return true
}
return false
}
+57
View File
@@ -0,0 +1,57 @@
package cmd
import (
"os"
"path/filepath"
"testing"
)
func TestDefaultConfigDir_Dev(t *testing.T) {
// Without /etc/spinifex should get dev paths
if _, err := os.Stat("/etc/spinifex"); err == nil {
t.Skip("test requires /etc/spinifex to not exist")
}
homeDir, _ := os.UserHomeDir()
expected := filepath.Join(homeDir, "spinifex", "config")
got := DefaultConfigDir()
if got != expected {
t.Errorf("DefaultConfigDir() = %q, want %q", got, expected)
}
}
func TestDefaultDataDir_Dev(t *testing.T) {
if _, err := os.Stat("/etc/spinifex"); err == nil {
t.Skip("test requires /etc/spinifex to not exist")
}
homeDir, _ := os.UserHomeDir()
expected := filepath.Join(homeDir, "spinifex")
got := DefaultDataDir()
if got != expected {
t.Errorf("DefaultDataDir() = %q, want %q", got, expected)
}
}
func TestDefaultConfigFile(t *testing.T) {
if _, err := os.Stat("/etc/spinifex"); err == nil {
t.Skip("test requires /etc/spinifex to not exist")
}
got := DefaultConfigFile()
homeDir, _ := os.UserHomeDir()
expected := filepath.Join(homeDir, "spinifex", "config", "spinifex.toml")
if got != expected {
t.Errorf("DefaultConfigFile() = %q, want %q", got, expected)
}
}
func TestIsProductionLayout_NoEtcSpinifex(t *testing.T) {
if _, err := os.Stat("/etc/spinifex"); err == nil {
t.Skip("test requires /etc/spinifex to not exist")
}
if isProductionLayout() {
t.Error("isProductionLayout() = true without /etc/spinifex, want false")
}
}
@@ -1,17 +1,18 @@
/*
Copyright © 2025 Mulga Defense Corporation
Copyright © 2026 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
http://www.apache.org/licenses/LICENSE-2.0
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package cmd
@@ -19,7 +20,7 @@ import (
"fmt"
"os"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@@ -32,20 +33,17 @@ var (
// rootCmd represents the base command when called without any subcommands
var rootCmd = &cobra.Command{
Use: "hive",
Short: "Hive - Open source AWS-compatible platform for secure edge deployments",
Use: "spx",
Short: "Spinifex - Open source AWS-compatible platform for secure edge deployments",
Long: `
__/\\\________/\\\__/\\\\\\\\\\\__/\\\________/\\\__/\\\\\\\\\\\\\\\_
_\/\\\_______\/\\\_\/////\\\///__\/\\\_______\/\\\_\/\\\///////////__
_\/\\\_______\/\\\_____\/\\\_____\//\\\______/\\\__\/\\\_____________
_\/\\\\\\\\\\\\\\\_____\/\\\______\//\\\____/\\\___\/\\\\\\\\\\\_____
_\/\\\/////////\\\_____\/\\\_______\//\\\__/\\\____\/\\\///////______
_\/\\\_______\/\\\_____\/\\\________\//\\\/\\\_____\/\\\_____________
_\/\\\_______\/\\\_____\/\\\_________\//\\\\\______\/\\\_____________
_\/\\\_______\/\\\__/\\\\\\\\\\\______\//\\\_______\/\\\\\\\\\\\\\\\_
_\///________\///__\///////////________\///________\///////////////__
_____ _____ _____ _ _ _____ ______ ________ __
/ ____| __ \_ _| \ | |_ _| ____| ____\ \ / /
| (___ | |__) || | | \| | | | | |__ | |__ \ V /
\___ \| ___/ | | | . ‘ | | | | __| | __| > <
____) | | _| |_| |\ |_| |_| | | |____ / . \
|_____/|_| |_____|_| \_|_____|_| |______/_/ \_\
Hive – Open source AWS-compatible platform for secure edge deployments.
Spinifex – Open source AWS-compatible platform for secure edge deployments.
Run EC2, VPC, S3, and EBS-like services on bare metal with full control.
Built for environments where running in the cloud isn’t an option.
Whether you’re deploying to edge sites, private data-centers, or operating
@@ -67,45 +65,44 @@ func init() {
// Global flags
rootCmd.PersistentFlags().String("config", "", "config file (required)")
viper.BindEnv("config", "HIVE_CONFIG_PATH")
viper.BindEnv("config", "SPINIFEX_CONFIG_PATH")
viper.BindPFlag("config", rootCmd.PersistentFlags().Lookup("config"))
// Authentication (access_key, secret)
rootCmd.PersistentFlags().String("access-key", "", "AWS access key (overrides config file and env)")
viper.BindEnv("access-key", "HIVE_ACCESS_KEY")
viper.BindEnv("access-key", "SPINIFEX_ACCESS_KEY")
viper.BindPFlag("access-key", rootCmd.PersistentFlags().Lookup("access-key"))
rootCmd.PersistentFlags().String("secret-key", "", "AWS secret key (overrides config file and env)")
viper.BindEnv("secret-key", "HIVE_SECRET_KEY")
viper.BindEnv("secret-key", "SPINIFEX_SECRET_KEY")
viper.BindPFlag("secret-key", rootCmd.PersistentFlags().Lookup("secret-key"))
rootCmd.PersistentFlags().String("host", "", "AWS Endpoint (overrides config file and env)")
viper.BindEnv("host", "HIVE_HOST")
viper.BindEnv("host", "SPINIFEX_HOST")
viper.BindPFlag("host", rootCmd.PersistentFlags().Lookup("host"))
// Viperblock config
rootCmd.PersistentFlags().String("base-dir", "", "Viperblock base directory (overrides config file and env)")
viper.BindEnv("base-dir", "HIVE_BASE_DIR")
viper.BindEnv("base-dir", "SPINIFEX_BASE_DIR")
viper.BindPFlag("base-dir", rootCmd.PersistentFlags().Lookup("base-dir"))
// NATS specific flags
rootCmd.PersistentFlags().String("nats-host", "", "NATS server host (overrides config file and env)")
viper.BindEnv("nats-host", "HIVE_NATS_HOST")
viper.BindEnv("nats-host", "SPINIFEX_NATS_HOST")
viper.BindPFlag("nats-host", rootCmd.PersistentFlags().Lookup("nats-host"))
rootCmd.PersistentFlags().String("nats-token", "", "NATS authentication token (overrides config file and env)")
viper.BindEnv("nats-token", "HIVE_NATS_TOKEN")
viper.BindEnv("nats-token", "SPINIFEX_NATS_TOKEN")
viper.BindPFlag("nats-token", rootCmd.PersistentFlags().Lookup("nats-token"))
rootCmd.PersistentFlags().String("nats-subject", "", "NATS subscription subject (overrides config file and env)")
viper.BindEnv("nats-subject", "HIVE_NATS_SUBJECT")
viper.BindEnv("nats-subject", "SPINIFEX_NATS_SUBJECT")
viper.BindPFlag("nats-subject", rootCmd.PersistentFlags().Lookup("nats-subject"))
// Bind flags to viper
//viper.BindPFlag("nats.host", rootCmd.PersistentFlags().Lookup("nats-host"))
//viper.BindPFlag("nats.acl.token", rootCmd.PersistentFlags().Lookup("nats-token"))
//viper.BindPFlag("nats.sub.subject", rootCmd.PersistentFlags().Lookup("nats-subject"))
}
// initConfig reads in config file and ENV variables if set.
@@ -116,7 +113,11 @@ func initConfig() {
appConfig, err = config.LoadConfig(cfgFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: %v\n", err)
fmt.Fprintln(os.Stderr, "Continuing with environment variables and defaults...")
// If a config file was explicitly provided, treat load failure as fatal
if cfgFile != "" {
fmt.Fprintf(os.Stderr, "Error: failed to load config %s: %v\n", cfgFile, err)
os.Exit(1)
}
// No config specified — continue with env/defaults (e.g., spx --help)
}
}
@@ -1,17 +1,18 @@
/*
Copyright © 2025 Mulga Defense Corporation
Copyright © 2026 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
http://www.apache.org/licenses/LICENSE-2.0
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package cmd
@@ -21,20 +22,22 @@ import (
"os"
"strings"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/service"
"github.com/mulgadc/hive/hive/services/hiveui"
"github.com/mulgadc/hive/hive/services/nats"
"github.com/mulgadc/hive/hive/services/predastore"
"github.com/mulgadc/hive/hive/services/viperblockd"
"github.com/mulgadc/predastore/s3"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/mulgadc/spinifex/spinifex/service"
"github.com/mulgadc/spinifex/spinifex/services/awsgw"
"github.com/mulgadc/spinifex/spinifex/services/nats"
"github.com/mulgadc/spinifex/spinifex/services/predastore"
"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
"github.com/mulgadc/spinifex/spinifex/services/viperblockd"
"github.com/mulgadc/spinifex/spinifex/services/vpcd"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var serviceCmd = &cobra.Command{
Use: "service",
Short: "Manage Hive services",
Short: "Manage Spinifex services",
}
var predastoreCmd = &cobra.Command{
@@ -52,9 +55,9 @@ var natsCmd = &cobra.Command{
Short: "Manage the nats service",
}
var hiveCmd = &cobra.Command{
Use: "hive",
Short: "Manage the hive service",
var spinifexCmd = &cobra.Command{
Use: "spinifex",
Short: "Manage the spinifex service",
}
var awsgwCmd = &cobra.Command{
@@ -62,10 +65,15 @@ var awsgwCmd = &cobra.Command{
Short: "Manage the awsgw (AWS gateway) service",
}
var hiveUICmd = &cobra.Command{
Use: "hive-ui",
Aliases: []string{"ui", "hiveui"},
Short: "Manage the hive-ui service",
var vpcdCmd = &cobra.Command{
Use: "vpcd",
Short: "Manage the vpcd (VPC daemon) service",
}
var spinifexUICmd = &cobra.Command{
Use: "spinifex-ui",
Aliases: []string{"ui", "spinifexui"},
Short: "Manage the spinifex-ui service",
}
var predastoreStartCmd = &cobra.Command{
@@ -142,7 +150,10 @@ var predastoreStartCmd = &cobra.Command{
return
}
service.Start()
if _, err := service.Start(); err != nil {
fmt.Println("Error starting predastore service:", err)
os.Exit(1)
}
fmt.Println("Predastore service started", service)
},
@@ -161,7 +172,10 @@ var predastoreStopCmd = &cobra.Command{
return
}
service.Stop()
if err = service.Stop(); err != nil {
fmt.Println("Error stopping predastore service:", err)
os.Exit(1)
}
fmt.Println("Predastore service stopped")
@@ -195,12 +209,11 @@ var viperblockStartCmd = &cobra.Command{
// TODO: Support ENV vars, CLI, otherwise revert to config.LoadConfig()
clusterConfig, err := config.LoadConfig(cfgFile)
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
if err != nil {
fmt.Println("Error loading config file:", err)
return
}
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
natsHost := viper.GetString("nats-host")
@@ -232,13 +245,13 @@ var viperblockStartCmd = &cobra.Command{
accessKey := viper.GetString("access-key")
if accessKey != "" {
fmt.Println("Overwriting access-key:", accessKey)
fmt.Println("Overwriting access-key: ****")
nodeConfig.Predastore.AccessKey = accessKey
}
secretKey := viper.GetString("secret-key")
if secretKey != "" {
fmt.Println("Overwriting secret-key:", secretKey)
fmt.Println("Overwriting secret-key: ****")
nodeConfig.Predastore.SecretKey = secretKey
}
@@ -274,6 +287,7 @@ var viperblockStartCmd = &cobra.Command{
service, err := service.New("viperblock", &viperblockd.Config{
NatsHost: nodeConfig.NATS.Host,
NatsToken: nodeConfig.NATS.ACL.Token,
PluginPath: pluginPath,
S3Host: nodeConfig.Predastore.Host,
Bucket: nodeConfig.Predastore.Bucket,
@@ -314,7 +328,10 @@ var viperblockStopCmd = &cobra.Command{
return
}
service.Stop()
if err = service.Stop(); err != nil {
fmt.Println("Error stopping viperblock service:", err)
os.Exit(1)
}
fmt.Println("Viperblock service stopped")
@@ -358,7 +375,10 @@ var natsStartCmd = &cobra.Command{
return
}
service.Start()
if _, err = service.Start(); err != nil {
fmt.Println("Error starting nats service:", err)
os.Exit(1)
}
fmt.Println("NATS service started")
},
}
@@ -376,7 +396,10 @@ var natsStopCmd = &cobra.Command{
return
}
service.Stop()
if err = service.Stop(); err != nil {
fmt.Println("Error stopping nats service:", err)
os.Exit(1)
}
fmt.Println("Nats service stopped")
},
@@ -390,12 +413,12 @@ var natsStatusCmd = &cobra.Command{
},
}
// Repeat for hive
var hiveStartCmd = &cobra.Command{
// Repeat for spinifex
var spinifexStartCmd = &cobra.Command{
Use: "start",
Short: "Start the hive service",
Short: "Start the spinifex service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting hive service...")
fmt.Println("Starting spinifex service...")
cfgFile := viper.GetString("config")
@@ -406,12 +429,11 @@ var hiveStartCmd = &cobra.Command{
// TODO: Support ENV vars, CLI, otherwise revert to config.LoadConfig()
clusterConfig, err := config.LoadConfig(cfgFile)
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
if err != nil {
fmt.Println("Error loading config file:", err)
return
}
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
// Overwrite defaults (CLI first, config second, env third)
baseDir := viper.GetString("base-dir")
@@ -432,47 +454,53 @@ var hiveStartCmd = &cobra.Command{
// Apply changes back to cluster config
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
svc, err := service.New("hive", clusterConfig)
svc, err := service.New("spinifex", clusterConfig)
if err != nil {
fmt.Println("Error starting hive service:", err)
fmt.Println("Error starting spinifex service:", err)
return
}
// Set config path for cluster manager
if hiveSvc, ok := svc.(interface{ SetConfigPath(string) }); ok {
hiveSvc.SetConfigPath(cfgFile)
if spxSvc, ok := svc.(interface{ SetConfigPath(string) }); ok {
spxSvc.SetConfigPath(cfgFile)
}
svc.Start()
fmt.Println("HIVE service started")
if _, err = svc.Start(); err != nil {
fmt.Println("Error starting spinifex service:", err)
os.Exit(1)
}
fmt.Println("Spinifex service started")
},
}
var hiveStopCmd = &cobra.Command{
var spinifexStopCmd = &cobra.Command{
Use: "stop",
Short: "Stop the hive service",
Short: "Stop the spinifex service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping hive service...")
fmt.Println("Stopping spinifex service...")
service, err := service.New("hive", &config.ClusterConfig{})
service, err := service.New("spinifex", &config.ClusterConfig{})
if err != nil {
fmt.Println("Error stopping hive service:", err)
fmt.Println("Error stopping spinifex service:", err)
return
}
service.Stop()
if err = service.Stop(); err != nil {
fmt.Println("Error stopping spinifex service:", err)
os.Exit(1)
}
fmt.Println("Hive service stopped")
fmt.Println("Spinifex service stopped")
},
}
var hiveStatusCmd = &cobra.Command{
var spinifexStatusCmd = &cobra.Command{
Use: "status",
Short: "Get status of the hive service",
Short: "Get status of the spinifex service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Hive service status: ...")
fmt.Println("Spinifex service status: ...")
},
}
@@ -495,12 +523,11 @@ var awsgwStartCmd = &cobra.Command{
// TODO: Support ENV vars, CLI, otherwise revert to config.LoadConfig()
clusterConfig, err := config.LoadConfig(cfgFile)
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
if err != nil {
fmt.Println("Error loading config file:", err)
return
}
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
// Overwrite defaults (CLI first, config second, env third)
awsgwHost := viper.GetString("host")
@@ -524,7 +551,7 @@ var awsgwStartCmd = &cobra.Command{
baseDir := viper.GetString("base-dir")
if awsgwTlsKey != "" {
if baseDir != "" {
fmt.Println("Overwriting awsgw base-dir to:", baseDir)
nodeConfig.BaseDir = baseDir
}
@@ -532,6 +559,7 @@ var awsgwStartCmd = &cobra.Command{
// Apply changes back to cluster config
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
awsgw.SetBuildInfo(Version, Commit)
service, err := service.New("awsgw", clusterConfig)
if err != nil {
@@ -539,7 +567,10 @@ var awsgwStartCmd = &cobra.Command{
return
}
service.Start()
if _, err = service.Start(); err != nil {
fmt.Println("Error starting awsgw service:", err)
os.Exit(1)
}
fmt.Println("AWSGW service started")
},
}
@@ -557,7 +588,10 @@ var awsgwStopCmd = &cobra.Command{
return
}
service.Stop()
if err = service.Stop(); err != nil {
fmt.Println("Error stopping awsgw service:", err)
os.Exit(1)
}
fmt.Println("AWSGW service stopped")
},
@@ -571,18 +605,18 @@ var awsgwStatusCmd = &cobra.Command{
},
}
var hiveUIStartCmd = &cobra.Command{
var spinifexUIStartCmd = &cobra.Command{
Use: "start",
Short: "Start the hive-ui service",
Short: "Start the spinifex-ui service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting hive-ui service...")
fmt.Println("Starting spinifex-ui service...")
port := viper.GetInt("hive-ui-port")
host := viper.GetString("hive-ui-host")
tlsCert := viper.GetString("hive-ui-tls-cert")
tlsKey := viper.GetString("hive-ui-tls-key")
port := viper.GetInt("spinifex-ui-port")
host := viper.GetString("spinifex-ui-host")
tlsCert := viper.GetString("spinifex-ui-tls-cert")
tlsKey := viper.GetString("spinifex-ui-tls-key")
svc, err := service.New("hive-ui", &hiveui.Config{
svc, err := service.New("spinifex-ui", &spinifexui.Config{
Port: port,
Host: host,
TLSCert: tlsCert,
@@ -590,57 +624,180 @@ var hiveUIStartCmd = &cobra.Command{
})
if err != nil {
fmt.Println("Error starting hive-ui service:", err)
fmt.Println("Error starting spinifex-ui service:", err)
return
}
svc.Start()
fmt.Println("hive-ui service started")
if _, err = svc.Start(); err != nil {
fmt.Println("Error starting spinifex-ui service:", err)
os.Exit(1)
}
fmt.Println("spinifex-ui service started")
},
}
var hiveUIStopCmd = &cobra.Command{
var spinifexUIStopCmd = &cobra.Command{
Use: "stop",
Short: "Stop the hive-ui service",
Short: "Stop the spinifex-ui service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping hive-ui service...")
fmt.Println("Stopping spinifex-ui service...")
svc, err := service.New("hive-ui", &hiveui.Config{})
svc, err := service.New("spinifex-ui", &spinifexui.Config{})
if err != nil {
fmt.Println("Error stopping hive-ui service:", err)
fmt.Println("Error stopping spinifex-ui service:", err)
return
}
svc.Stop()
fmt.Println("hive-ui service stopped")
if err = svc.Stop(); err != nil {
fmt.Println("Error stopping spinifex-ui service:", err)
os.Exit(1)
}
fmt.Println("spinifex-ui service stopped")
},
}
var hiveUIStatusCmd = &cobra.Command{
var spinifexUIStatusCmd = &cobra.Command{
Use: "status",
Short: "Get status of the hive-ui service",
Short: "Get status of the spinifex-ui service",
Run: func(cmd *cobra.Command, args []string) {
svc, err := service.New("hive-ui", &hiveui.Config{})
svc, err := service.New("spinifex-ui", &spinifexui.Config{})
if err != nil {
fmt.Println("Error getting hive-ui service status:", err)
fmt.Println("Error getting spinifex-ui service status:", err)
return
}
status, err := svc.Status()
if err != nil {
fmt.Println("Error getting hive-ui service status:", err)
fmt.Println("Error getting spinifex-ui service status:", err)
return
}
fmt.Println("hive-ui service status:", status)
fmt.Println("spinifex-ui service status:", status)
},
}
var vpcdStartCmd = &cobra.Command{
Use: "start",
Short: "Start the vpcd service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting vpcd service...")
cfgFile := viper.GetString("config")
if cfgFile == "" {
fmt.Println("Config file is not set")
return
}
clusterConfig, err := config.LoadConfig(cfgFile)
if err != nil {
fmt.Println("Error loading config file:", err)
return
}
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
// Map cluster-wide external pools to vpcd config
var extPools []vpcd.ExternalPoolConfig
for _, p := range clusterConfig.Network.ExternalPools {
extPools = append(extPools, vpcd.ExternalPoolConfig{
Name: p.Name,
RangeStart: p.RangeStart,
RangeEnd: p.RangeEnd,
Gateway: p.Gateway,
GatewayIP: p.GatewayIP,
PrefixLen: p.PrefixLen,
DNSServers: p.DNSServers,
Region: p.Region,
AZ: p.AZ,
})
}
// Fallback chassis names derived from config node names. vpcd prefers
// discovering chassis from the OVN Southbound DB at startup; these are
// only used if that discovery fails.
var chassisNames []string
for nodeName := range clusterConfig.Nodes {
chassisNames = append(chassisNames, "chassis-"+nodeName)
}
var bootstrap *vpcd.BootstrapVPC
if clusterConfig.Bootstrap.VpcId != "" {
bootstrap = &vpcd.BootstrapVPC{
AccountID: clusterConfig.Bootstrap.AccountID,
VpcId: clusterConfig.Bootstrap.VpcId,
SubnetId: clusterConfig.Bootstrap.SubnetId,
IgwId: clusterConfig.Bootstrap.IgwId,
Cidr: clusterConfig.Bootstrap.Cidr,
SubnetCidr: clusterConfig.Bootstrap.SubnetCidr,
}
}
baseDir := viper.GetString("base-dir")
if baseDir != "" {
fmt.Println("Overwriting vpcd base-dir to:", baseDir)
nodeConfig.BaseDir = baseDir
}
svc, err := service.New("vpcd", &vpcd.Config{
NatsHost: nodeConfig.NATS.Host,
NatsToken: nodeConfig.NATS.ACL.Token,
OVNNBAddr: nodeConfig.VPCD.OVNNBAddr,
OVNSBAddr: nodeConfig.VPCD.OVNSBAddr,
BaseDir: nodeConfig.BaseDir,
Debug: false,
ExternalMode: clusterConfig.Network.ExternalMode,
ExternalPools: extPools,
ChassisNames: chassisNames,
Bootstrap: bootstrap,
ExternalInterface: nodeConfig.VPCD.ExternalInterface,
WanBridge: nodeConfig.VPCD.WanBridge,
BridgeMode: nodeConfig.VPCD.BridgeMode,
})
if err != nil {
fmt.Println("Error starting vpcd service:", err)
return
}
if _, err = svc.Start(); err != nil {
fmt.Println("Error starting vpcd service:", err)
os.Exit(1)
}
fmt.Println("vpcd service started")
},
}
var vpcdStopCmd = &cobra.Command{
Use: "stop",
Short: "Stop the vpcd service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping vpcd service...")
svc, err := service.New("vpcd", &vpcd.Config{})
if err != nil {
fmt.Println("Error stopping vpcd service:", err)
return
}
if err = svc.Stop(); err != nil {
fmt.Println("Error stopping vpcd service:", err)
os.Exit(1)
}
fmt.Println("vpcd service stopped")
},
}
var vpcdStatusCmd = &cobra.Command{
Use: "status",
Short: "Get status of the vpcd service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("vpcd service status: ...")
},
}
func init() {
viper.SetEnvPrefix("HIVE") // Prefix for environment variables
viper.SetEnvPrefix("SPINIFEX") // Prefix for environment variables
viper.SetEnvKeyReplacer(strings.NewReplacer("-", "_"))
viper.AutomaticEnv() // Read environment variables automatically
@@ -651,47 +808,47 @@ func init() {
// Predastore Port
predastoreCmd.PersistentFlags().Int("port", 8443, "Predastore (S3) port")
viper.BindEnv("port", "HIVE_PREDASTORE_PORT")
viper.BindEnv("port", "SPINIFEX_PREDASTORE_PORT")
viper.BindPFlag("port", predastoreCmd.PersistentFlags().Lookup("port"))
// Predastore Host
predastoreCmd.PersistentFlags().String("host", "0.0.0.0", "Predastore (S3) host")
viper.BindEnv("host", "HIVE_PREDASTORE_HOST")
viper.BindEnv("host", "SPINIFEX_PREDASTORE_HOST")
viper.BindPFlag("host", predastoreCmd.PersistentFlags().Lookup("host"))
// Base path
predastoreCmd.PersistentFlags().String("base-path", "", "Predastore (S3) base path")
viper.BindEnv("base-path", "HIVE_PREDASTORE_BASE_PATH")
viper.BindEnv("base-path", "SPINIFEX_PREDASTORE_BASE_PATH")
viper.BindPFlag("base-path", predastoreCmd.PersistentFlags().Lookup("base-path"))
// Predastore Config Path
predastoreCmd.PersistentFlags().String("config-path", "", "Predastore (S3) config path")
viper.BindEnv("config-path", "HIVE_PREDASTORE_CONFIG_PATH")
viper.BindEnv("config-path", "SPINIFEX_PREDASTORE_CONFIG_PATH")
viper.BindPFlag("config-path", predastoreCmd.PersistentFlags().Lookup("config-path"))
// Predastore Debug
predastoreCmd.PersistentFlags().Bool("debug", false, "Predastore (S3) debug")
viper.BindEnv("debug", "HIVE_PREDASTORE_DEBUG")
viper.BindEnv("debug", "SPINIFEX_PREDASTORE_DEBUG")
viper.BindPFlag("debug", predastoreCmd.PersistentFlags().Lookup("debug"))
// Predastore TLS Cert
predastoreCmd.PersistentFlags().String("tls-cert", "", "Predastore (S3) TLS certificate")
viper.BindEnv("tls-cert", "HIVE_PREDASTORE_TLS_CERT")
viper.BindEnv("tls-cert", "SPINIFEX_PREDASTORE_TLS_CERT")
viper.BindPFlag("tls-cert", predastoreCmd.PersistentFlags().Lookup("tls-cert"))
// Predastore TLS Key
predastoreCmd.PersistentFlags().String("tls-key", "", "Predastore (S3) TLS key")
viper.BindEnv("tls-key", "HIVE_PREDASTORE_TLS_KEY")
viper.BindEnv("tls-key", "SPINIFEX_PREDASTORE_TLS_KEY")
viper.BindPFlag("tls-key", predastoreCmd.PersistentFlags().Lookup("tls-key"))
// Predastore Backend
predastoreCmd.PersistentFlags().String("backend", "distributed", "Predastore (S3) backend")
viper.BindEnv("backend", "HIVE_PREDASTORE_BACKEND")
viper.BindEnv("backend", "SPINIFEX_PREDASTORE_BACKEND")
viper.BindPFlag("backend", predastoreCmd.PersistentFlags().Lookup("backend"))
// Predastore Node ID
predastoreCmd.PersistentFlags().Int("node-id", 0, "Predastore (S3) node ID")
viper.BindEnv("node-id", "HIVE_PREDASTORE_NODE_ID")
viper.BindEnv("node-id", "SPINIFEX_PREDASTORE_NODE_ID")
viper.BindPFlag("node-id", predastoreCmd.PersistentFlags().Lookup("node-id"))
// Predastore CPU Profiling
@@ -710,20 +867,20 @@ func init() {
serviceCmd.AddCommand(viperblockCmd)
viperblockCmd.PersistentFlags().String("s3-host", "0.0.0.0:8443", "Predastore (S3) host URI")
viper.BindEnv("s3-host", "HIVE_VIPERBLOCK_S3_HOST")
viperblockCmd.PersistentFlags().String("s3-host", "", "Predastore (S3) host URI")
viper.BindEnv("s3-host", "SPINIFEX_VIPERBLOCK_S3_HOST")
viper.BindPFlag("s3-host", predastoreCmd.PersistentFlags().Lookup("s3-host"))
viperblockCmd.PersistentFlags().String("s3-bucket", "predastore", "Predastore (S3) bucket")
viper.BindEnv("s3-bucket", "HIVE_VIPERBLOCK_S3_BUCKET")
viper.BindEnv("s3-bucket", "SPINIFEX_VIPERBLOCK_S3_BUCKET")
viper.BindPFlag("s3-bucket", predastoreCmd.PersistentFlags().Lookup("s3-bucket"))
viperblockCmd.PersistentFlags().String("s3-region", "ap-southeast-2", "Predastore (S3) region")
viper.BindEnv("s3-region", "HIVE_VIPERBLOCK_S3_REGION")
viper.BindEnv("s3-region", "SPINIFEX_VIPERBLOCK_S3_REGION")
viper.BindPFlag("s3-region", predastoreCmd.PersistentFlags().Lookup("s3-region"))
viperblockCmd.PersistentFlags().String("plugin-path", "/opt/hive/lib/nbdkit-viperblock-plugin.so", "Pathname to the nbdkit viperblockplugin")
viper.BindEnv("plugin-path", "HIVE_VIPERBLOCK_PLUGIN_PATH")
viperblockCmd.PersistentFlags().String("plugin-path", "/opt/spinifex/lib/nbdkit-viperblock-plugin.so", "Pathname to the nbdkit viperblockplugin")
viper.BindEnv("plugin-path", "SPINIFEX_VIPERBLOCK_PLUGIN_PATH")
viper.BindPFlag("plugin-path", predastoreCmd.PersistentFlags().Lookup("plugin-path"))
viperblockCmd.AddCommand(viperblockStartCmd)
@@ -739,82 +896,88 @@ func init() {
// Add NATS flags
natsCmd.PersistentFlags().Int("port", 4222, "NATS server port")
viper.BindEnv("port", "HIVE_NATS_PORT")
viper.BindEnv("port", "SPINIFEX_NATS_PORT")
viper.BindPFlag("port", natsCmd.PersistentFlags().Lookup("port"))
natsCmd.PersistentFlags().String("host", "0.0.0.0", "NATS server host")
viper.BindEnv("host", "HIVE_NATS_HOST")
viper.BindEnv("host", "SPINIFEX_NATS_HOST")
viper.BindPFlag("host", natsCmd.PersistentFlags().Lookup("host"))
natsCmd.PersistentFlags().Bool("debug", false, "Enable debug logging")
viper.BindEnv("debug", "HIVE_NATS_DEBUG")
viper.BindEnv("debug", "SPINIFEX_NATS_DEBUG")
viper.BindPFlag("debug", natsCmd.PersistentFlags().Lookup("debug"))
natsCmd.PersistentFlags().String("data-dir", "", "NATS data directory")
viper.BindEnv("data-dir", "HIVE_NATS_DATA_DIR")
viper.BindEnv("data-dir", "SPINIFEX_NATS_DATA_DIR")
viper.BindPFlag("data-dir", natsCmd.PersistentFlags().Lookup("data-dir"))
natsCmd.PersistentFlags().Bool("jetstream", false, "Enable JetStream")
viper.BindEnv("jetstream", "HIVE_NATS_JETSTREAM")
viper.BindEnv("jetstream", "SPINIFEX_NATS_JETSTREAM")
viper.BindPFlag("jetstream", natsCmd.PersistentFlags().Lookup("jetstream"))
// Hive
serviceCmd.AddCommand(hiveCmd)
// Spinifex
serviceCmd.AddCommand(spinifexCmd)
hiveCmd.AddCommand(hiveStartCmd)
hiveCmd.AddCommand(hiveStopCmd)
hiveCmd.AddCommand(hiveStatusCmd)
spinifexCmd.AddCommand(spinifexStartCmd)
spinifexCmd.AddCommand(spinifexStopCmd)
spinifexCmd.AddCommand(spinifexStatusCmd)
hiveCmd.PersistentFlags().String("wal-dir", "", "Write-ahead log (WAL) directory. Place on high-speed NVMe disk, or tmpfs for development.")
viper.BindEnv("wal-dir", "HIVE_WAL_DIR")
viper.BindPFlag("wal-dir", hiveCmd.PersistentFlags().Lookup("wal-dir"))
spinifexCmd.PersistentFlags().String("wal-dir", "", "Write-ahead log (WAL) directory. Place on high-speed NVMe disk, or tmpfs for development.")
viper.BindEnv("wal-dir", "SPINIFEX_WAL_DIR")
viper.BindPFlag("wal-dir", spinifexCmd.PersistentFlags().Lookup("wal-dir"))
// AWS GW
serviceCmd.AddCommand(awsgwCmd)
awsgwCmd.PersistentFlags().String("host", "0.0.0.0:9999", "AWS Gateway server host")
viper.BindEnv("host", "HIVE_AWSGW_HOST")
viper.BindEnv("host", "SPINIFEX_AWSGW_HOST")
viper.BindPFlag("host", awsgwCmd.PersistentFlags().Lookup("host"))
// AWS GW TLS Cert
awsgwCmd.PersistentFlags().String("tls-cert", "", "AWS Gateway TLS certificate")
viper.BindEnv("tls-cert", "HIVE_AWSGW_TLS_CERT")
viper.BindEnv("tls-cert", "SPINIFEX_AWSGW_TLS_CERT")
viper.BindPFlag("tls-cert", awsgwCmd.PersistentFlags().Lookup("tls-cert"))
// AWS GW TLS Key
awsgwCmd.PersistentFlags().String("tls-key", "", "AWS Gateway TLS key")
viper.BindEnv("tls-key", "HIVE_AWSGW_TLS_KEY")
viper.BindEnv("tls-key", "SPINIFEX_AWSGW_TLS_KEY")
viper.BindPFlag("tls-key", awsgwCmd.PersistentFlags().Lookup("tls-key"))
awsgwCmd.PersistentFlags().Bool("debug", false, "AWS Gateway Debug")
viper.BindEnv("debug", "HIVE_AWSGW_DEBUG")
viper.BindEnv("debug", "SPINIFEX_AWSGW_DEBUG")
viper.BindPFlag("debug", awsgwCmd.PersistentFlags().Lookup("debug"))
awsgwCmd.AddCommand(awsgwStartCmd)
awsgwCmd.AddCommand(awsgwStopCmd)
awsgwCmd.AddCommand(awsgwStatusCmd)
// hive-ui
serviceCmd.AddCommand(hiveUICmd)
// spinifex-ui
serviceCmd.AddCommand(spinifexUICmd)
hiveUICmd.PersistentFlags().Int("port", 3000, "hive-ui server port")
viper.BindEnv("hive-ui-port", "HIVE_UI_PORT")
viper.BindPFlag("hive-ui-port", hiveUICmd.PersistentFlags().Lookup("port"))
spinifexUICmd.PersistentFlags().Int("port", 3000, "spinifex-ui server port")
viper.BindEnv("spinifex-ui-port", "SPINIFEX_UI_PORT")
viper.BindPFlag("spinifex-ui-port", spinifexUICmd.PersistentFlags().Lookup("port"))
hiveUICmd.PersistentFlags().String("host", "0.0.0.0", "hive-ui server host")
viper.BindEnv("hive-ui-host", "HIVE_UI_HOST")
viper.BindPFlag("hive-ui-host", hiveUICmd.PersistentFlags().Lookup("host"))
spinifexUICmd.PersistentFlags().String("host", "0.0.0.0", "spinifex-ui server host")
viper.BindEnv("spinifex-ui-host", "SPINIFEX_UI_HOST")
viper.BindPFlag("spinifex-ui-host", spinifexUICmd.PersistentFlags().Lookup("host"))
hiveUICmd.PersistentFlags().String("tls-cert", "", "TLS certificate path")
viper.BindEnv("hive-ui-tls-cert", "HIVE_UI_TLS_CERT")
viper.BindPFlag("hive-ui-tls-cert", hiveUICmd.PersistentFlags().Lookup("tls-cert"))
spinifexUICmd.PersistentFlags().String("tls-cert", "", "TLS certificate path")
viper.BindEnv("spinifex-ui-tls-cert", "SPINIFEX_UI_TLS_CERT")
viper.BindPFlag("spinifex-ui-tls-cert", spinifexUICmd.PersistentFlags().Lookup("tls-cert"))
hiveUICmd.PersistentFlags().String("tls-key", "", "TLS key path")
viper.BindEnv("hive-ui-tls-key", "HIVE_UI_TLS_KEY")
viper.BindPFlag("hive-ui-tls-key", hiveUICmd.PersistentFlags().Lookup("tls-key"))
spinifexUICmd.PersistentFlags().String("tls-key", "", "TLS key path")
viper.BindEnv("spinifex-ui-tls-key", "SPINIFEX_UI_TLS_KEY")
viper.BindPFlag("spinifex-ui-tls-key", spinifexUICmd.PersistentFlags().Lookup("tls-key"))
hiveUICmd.AddCommand(hiveUIStartCmd)
hiveUICmd.AddCommand(hiveUIStopCmd)
hiveUICmd.AddCommand(hiveUIStatusCmd)
spinifexUICmd.AddCommand(spinifexUIStartCmd)
spinifexUICmd.AddCommand(spinifexUIStopCmd)
spinifexUICmd.AddCommand(spinifexUIStatusCmd)
// vpcd
serviceCmd.AddCommand(vpcdCmd)
vpcdCmd.AddCommand(vpcdStartCmd)
vpcdCmd.AddCommand(vpcdStopCmd)
vpcdCmd.AddCommand(vpcdStatusCmd)
}
+6
View File
@@ -0,0 +1,6 @@
# AWS Gateway configuration file in TOML format.
version = "1.0"
region = "{{.Region}}"
# Enable verbose debugging
debug = false
@@ -1,33 +1,33 @@
# spinifex-config-version: 1
# NATS Server Configuration
# Generated by Hive admin init
# Generated by Spinifex admin init
# Server configuration
{{ if .BindIP }}
{{- if .BindIP }}
listen: {{.BindIP}}:4222
{{ else }}
{{- else }}
listen: 0.0.0.0:4222
{{ end }}
{{- end }}
{{- if and .ClusterBindIP .ClusterRoutes }}
{{ if and .ClusterBindIP .ClusterRoutes }}
cluster {
name: {{ .ClusterName }}
listen: {{.ClusterBindIP}}:4248
routes = [
{{- range .ClusterRoutes }}
"nats://{{ . }}",
"nats-route://{{$.NatsToken}}@{{ . }}",
{{- end }}
]
}
{{ end }}
{{- end }}
server_name: hive-nats-{{.Node}}
server_name: spinifex-nats-{{.Node}}
# Authorization
authorization {
# token: "{{.NatsToken}}"
token: "{{.NatsToken}}"
}
# JetStream for persistent messaging and KV store
@@ -41,7 +41,7 @@ jetstream {
debug: false
trace: false
logtime: true
log_file: "{{.DataDir}}/logs/nats.log"
log_file: "{{.LogDir}}/nats.log"
# Limits
max_connections: 1000
@@ -49,9 +49,9 @@ max_payload: 1MB
max_pending: 64MB
# Monitoring
{{ if .BindIP }}
{{- if .BindIP }}
http: {{.BindIP}}:8222
{{ else }}
{{- else }}
http_port: 8222
{{ end }}
{{- end }}
@@ -11,7 +11,8 @@ debug = false
[rs]
data = 2
parity = 1
{{range .Nodes}}
{{- range .Nodes}}
[[db]]
id = {{.ID}}
host = "{{.Host}}"
@@ -22,14 +23,16 @@ secret_access_key = "{{$.SecretKey}}"
{{- if eq .ID 1}}
leader = true
{{- end}}
{{end}}
{{- end}}
{{- range .Nodes}}
[[nodes]]
id = {{.ID}}
host = "{{.Host}}"
port = 9991
path = "distributed/nodes/node-{{.ID}}/"
{{end}}
{{- end}}
[[buckets]]
name = "predastore"
region = "{{.Region}}"
@@ -43,3 +46,10 @@ secret_access_key = "{{.SecretKey}}"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
# IAM authentication via NATS KV (enables multi-account S3 access)
[iam]
nats_url = "nats://{{.BindIP}}:4222"
nats_token = "{{.NatsToken}}"
master_key_path = "{{.ConfigDir}}/master.key"
access_keys_bucket = "spinifex-iam-access-keys"
@@ -79,4 +79,11 @@ access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
]
# IAM authentication via NATS KV (enables multi-account S3 access)
[iam]
nats_url = "nats://{{if eq .BindIP "0.0.0.0"}}localhost{{else}}{{.BindIP}}{{end}}:4222"
nats_token = "{{.NatsToken}}"
master_key_path = "{{.ConfigDir}}/master.key"
access_keys_bucket = "spinifex-iam-access-keys"
+110
View File
@@ -0,0 +1,110 @@
# Domain configuration file in TOML format.
version = "1.0"
epoch = 1
node = "{{.Node}}"
{{- if .ExternalMode}}
[network]
external_mode = "{{.ExternalMode}}"
{{- if .ExternalDHCP}}
external_dhcp = true
{{- end}}
[[network.external_pools]]
name = "{{.PoolName}}"
{{- if .PoolSource}}
source = "{{.PoolSource}}"
{{- end}}
{{- if .PoolStart}}
range_start = "{{.PoolStart}}"
range_end = "{{.PoolEnd}}"
{{- end}}
gateway = "{{.PoolGateway}}"
{{- if .PoolGatewayIP}}
gateway_ip = "{{.PoolGatewayIP}}"
{{- end}}
prefix_len = {{if .PoolPrefixLen}}{{.PoolPrefixLen}}{{else}}24{{end}}
{{- if .PoolDNSServers}}
dns_servers = [{{range $i, $s := .PoolDNSServers}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
{{- end}}
{{- if .BootstrapVpcId}}
[bootstrap]
account_id = "{{.BootstrapAccountId}}"
vpc_id = "{{.BootstrapVpcId}}"
subnet_id = "{{.BootstrapSubnetId}}"
igw_id = "{{.BootstrapIgwId}}"
cidr = "{{.BootstrapCidr}}"
subnet_cidr = "{{.BootstrapSubnetCidr}}"
{{- end}}
[nodes.{{.Node}}]
node = "{{.Node}}"
host = "{{.BindIP}}"
region = "{{.Region}}"
az = "{{.Az}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
account_id = "{{.AccountID}}"
[nodes.{{.Node}}.daemon]
host = "{{.BindIP}}:{{.Port}}"
tlskey = "config/server.key"
tlscert = "config/server.pem"
{{- if .ExternalMode}}
dev_networking = false
{{- else}}
dev_networking = true
{{- end}}
[nodes.{{.Node}}.awsgw]
host = "{{.BindIP}}:9999"
tlskey = "config/server.key"
tlscert = "config/server.pem"
debug = true
config = "awsgw/awsgw.toml"
[nodes.{{.Node}}.nats]
host = "{{.BindIP}}:4222"
[nodes.{{.Node}}.nats.acl]
token = "{{.NatsToken}}"
[nodes.{{.Node}}.nats.sub]
[nodes.{{.Node}}.viperblock]
shardwal = false
[nodes.{{.Node}}.vpcd]
ovn_nb_addr = "{{.OVNNBAddr}}"
ovn_sb_addr = "{{.OVNSBAddr}}"
{{- if .ExternalIface}}
external_interface = "{{.ExternalIface}}"
{{- end}}
{{- if .WanBridge}}
wan_bridge = "{{.WanBridge}}"
{{- end}}
[nodes.{{.Node}}.predastore]
host = "{{.BindIP}}:8443"
region = "{{.Region}}"
bucket = "predastore"
accesskey = "{{.AccessKey}}"
secretkey = "{{.SecretKey}}"
{{- if gt .PredastoreNodeID 0}}
node_id = {{.PredastoreNodeID}}
{{- end}}
{{- range .RemoteNodes}}
[nodes.{{.Name}}]
node = "{{.Name}}"
host = "{{.Host}}"
region = "{{.Region}}"
az = "{{.AZ}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
{{- end}}
+19 -17
View File
@@ -1,17 +1,18 @@
/*
Copyright © 2025 Mulga Defense Corporation
Copyright © 2026 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
http://www.apache.org/licenses/LICENSE-2.0
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package cmd
@@ -20,9 +21,10 @@ import (
"fmt"
"os"
"sort"
"strconv"
"time"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/spinifex/spinifex/types"
"github.com/pterm/pterm"
"github.com/spf13/cobra"
)
@@ -56,15 +58,15 @@ func runTopNodes(cmd *cobra.Command, args []string) {
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.status", timeout)
responses, err := collectResponses(nc, "spinifex.node.status", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
respondedNodes := make(map[string]config.NodeStatusResponse)
respondedNodes := make(map[string]types.NodeStatusResponse)
for _, data := range responses {
var resp config.NodeStatusResponse
var resp types.NodeStatusResponse
if err := json.Unmarshal(data, &resp); err != nil {
continue
}
@@ -98,7 +100,7 @@ func runTopNodes(cmd *cobra.Command, args []string) {
resp.Node,
fmt.Sprintf("%d/%d", resp.AllocVCPU, resp.TotalVCPU),
fmt.Sprintf("%s/%s", formatMemGB(resp.AllocMemGB), formatMemGB(resp.TotalMemGB)),
fmt.Sprintf("%d", resp.VMCount),
strconv.Itoa(resp.VMCount),
})
for _, cap := range resp.InstanceTypes {
@@ -145,8 +147,8 @@ func runTopNodes(cmd *cobra.Command, args []string) {
agg := capacityMap[name]
capTable = append(capTable, []string{
name,
fmt.Sprintf("%d", agg.Available),
fmt.Sprintf("%d", agg.VCPU),
strconv.Itoa(agg.Available),
strconv.Itoa(agg.VCPU),
formatMemGB(agg.MemoryGB),
})
}
+27
View File
@@ -0,0 +1,27 @@
package cmd
import (
"fmt"
"runtime"
"github.com/spf13/cobra"
)
// Version variables set via ldflags at build time.
// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0"
var (
Version = "dev"
Commit = "unknown"
)
var versionCmd = &cobra.Command{
Use: "version",
Short: "Print the Spinifex version",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("spinifex %s (%s) %s/%s\n", Version, Commit, runtime.GOOS, runtime.GOARCH)
},
}
func init() {
rootCmd.AddCommand(versionCmd)
}
+29
View File
@@ -0,0 +1,29 @@
package cmd
import (
"testing"
)
func TestVersionVarsExist(t *testing.T) {
// Verify the version variables are set (defaults when not injected via ldflags)
if Version == "" {
t.Error("Version should not be empty")
}
if Commit == "" {
t.Error("Commit should not be empty")
}
}
func TestVersionCommand(t *testing.T) {
// Verify the version command is registered
found := false
for _, cmd := range rootCmd.Commands() {
if cmd.Use == "version" {
found = true
break
}
}
if !found {
t.Error("version command not registered on rootCmd")
}
}
+23
View File
@@ -0,0 +1,23 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package main
import "github.com/mulgadc/spinifex/cmd/spinifex/cmd"
func main() {
cmd.Execute()
}
-7
View File
@@ -1,7 +0,0 @@
#!/bin/sh
# Linux
#genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data
# MacOS
mkisofs -output seed.iso -volid cidata -joliet -rock meta-data user-data
-5
View File
@@ -1,5 +0,0 @@
#!/bin/sh
# Example to launch nbdkit
./server/nbdkit -p 10810 --pidfile /tmp/vb-vol-1-efi.pid ./plugins/golang/examples/ramdisk/nbdkit-goramdisk-plugin.so -v -f size=67108864 volume=vol-50c968e8dbcc74bd2-efi bucket=predastore region=ap-southeast-2 access_key="AKIAIOSFODNN7EXAMPLE" secret_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" base_dir="/tmp/vb/" host="https://127.0.0.1:8443" cache_size=1
-28
View File
@@ -1,28 +0,0 @@
#!/bin/sh
qemu-system-aarch64 \
-nographic \
-M virt,highmem=off \
-accel hvf \
-cpu host \
-smp 4 \
-m 3000 \
-drive file=/Users/benduncan/qemu/QEMU_EFI.img,if=pflash,format=raw \
-device virtio-blk-pci,drive=debian,bootindex=1 \
-drive if=none,media=disk,id=debian,format=raw,file=nbd://192.168.64.5:10809/default \
-drive format=raw,file=nbd://192.168.64.5:10810/default,if=pflash \
-drive file=nbd://192.168.64.5:10811/default,format=raw,if=virtio \
-device qemu-xhci \
-device usb-kbd \
-device usb-tablet \
-device intel-hda \
-device hda-duplex \
-netdev user,id=net0,hostfwd=tcp::2222-:22 -device virtio-net-device,netdev=net0 \
-device virtio-rng-pci
# ./server/nbdkit -p 10809 --pidfile /tmp/vb-vol-1.pid ./plugins/golang/examples/ramdisk/nbdkit-goramdisk-plugin.so -v -f size=4294967296 volume=vol-e660ff0065ad21f74 bucket=predastore region=ap-southeast-2 access_key="AKIAIOSFODNN7EXAMPLE" secret_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" base_dir="/tmp/vb/" host="https://127.0.0.1:8443" cache_size=20
# ./server/nbdkit -p 10810 --pidfile /tmp/vb-vol-1-efi.pid ./plugins/golang/examples/ramdisk/nbdkit-goramdisk-plugin.so -v -f size=67108864 volume=vol-e660ff0065ad21f74-efi bucket=predastore region=ap-southeast-2 access_key="AKIAIOSFODNN7EXAMPLE" secret_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" base_dir="/tmp/vb/" host="https://127.0.0.1:8443" cache_size=1
# ./server/nbdkit -p 10811 --pidfile /tmp/vb-vol-1-cloudinit.pid ./plugins/golang/examples/ramdisk/nbdkit-goramdisk-plugin.so -v -f size=1048576 volume=vol-e660ff0065ad21f74-cloudinit bucket=predastore region=ap-southeast-2 access_key="AKIAIOSFODNN7EXAMPLE" secret_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" base_dir="/tmp/vb/" host="https://127.0.0.1:8443" cache_size=1
+44 -44
View File
@@ -1,71 +1,71 @@
# Hive Systemd Service
# Spinifex Systemd Service
This directory contains the systemd service configuration for running Hive services on Ubuntu and Debian systems.
This directory contains the systemd service configuration for running Spinifex services on Ubuntu and Debian systems.
## Files
- `hive.service` - The systemd service file that manages all Hive services
- `install-hive-service.sh` - Installation script to set up the service
- `spinifex.service` - The systemd service file that manages all Spinifex services
- `install-spinifex-service.sh` - Installation script to set up the service
- `README.md` - This documentation file
## Installation
### Prerequisites
1. Ensure Hive is installed in `/opt/hive/`
2. Ensure the `hive` binary is available at `/opt/hive/bin/hive`
1. Ensure Spinifex is installed in `/opt/spinifex/`
2. Ensure the `spx` binary is available at `/opt/spinifex/bin/spx`
3. Run the installation script as root
### Setup Configuration
```
/opt/hive/bin/hive admin init
/opt/spinifex/bin/spx admin init
```
This will init the Hive environment and create the default configuration files in the home directory, e.g `~/hive/`.
This will init the Spinifex environment and create the default configuration files in the home directory, e.g `~/spinifex/`.
```
/home/username/hive:
/home/username/spinifex:
amis config images logs nats predastore state viperblock volumes
/home/username/hive/amis:
/home/username/spinifex/amis:
/home/username/hive/config:
awsgw hive.toml nats predastore server.key server.pem
/home/username/spinifex/config:
awsgw spinifex.toml nats predastore server.key server.pem
/home/username/hive/config/awsgw:
/home/username/spinifex/config/awsgw:
awsgw.toml
/home/username/hive/config/nats:
/home/username/spinifex/config/nats:
nats.conf
/home/username/hive/config/predastore:
/home/username/spinifex/config/predastore:
predastore.toml
/home/username/hive/images:
/home/username/spinifex/images:
/home/username/hive/logs:
awsgw.log awsgw.pid hive.log hive.pid nats.log nats.pid predastore.log predastore.pid viperblock.log viperblock.pid
/home/username/spinifex/logs:
awsgw.log awsgw.pid spinifex.log spinifex.pid nats.log nats.pid predastore.log predastore.pid viperblock.log viperblock.pid
/home/username/hive/nats:
/home/username/spinifex/nats:
/home/username/hive/predastore:
/home/username/spinifex/predastore:
data
/home/username/hive/predastore/data:
/home/username/spinifex/predastore/data:
predastore
/home/ben/hive/state:
/home/ben/spinifex/state:
/home/ben/hive/viperblock:
/home/ben/spinifex/viperblock:
/home/ben/hive/volumes:
/home/ben/spinifex/volumes:
```
### Quick Installation
```bash
sudo ./config/install-hive-service.sh
sudo ./config/install-spinifex-service.sh
```
### Manual Installation
@@ -74,7 +74,7 @@ If you prefer to install manually:
1. Copy the service file:
```bash
sudo cp config/hive.service /etc/systemd/system/
sudo cp config/spinifex.service /etc/systemd/system/
```
2. Reload systemd:
@@ -84,44 +84,44 @@ If you prefer to install manually:
3. Enable the service:
```bash
sudo systemctl enable hive.service
sudo systemctl enable spinifex.service
```
## Service Management
### Start the service
```bash
sudo systemctl start hive
sudo systemctl start spinifex
```
### Stop the service
```bash
sudo systemctl stop hive
sudo systemctl stop spinifex
```
### Check service status
```bash
sudo systemctl status hive
sudo systemctl status spinifex
```
### Restart the service
```bash
sudo systemctl restart hive
sudo systemctl restart spinifex
```
### View service logs
```bash
sudo journalctl -u hive -f
sudo journalctl -u spinifex -f
```
### Disable auto-start
```bash
sudo systemctl disable hive
sudo systemctl disable spinifex
```
## Service Details
The `hive.service` file:
The `spinifex.service` file:
- Starts services in the correct order: NATS → Predastore → Viperblock
- Includes proper delays between service starts to ensure dependencies are ready
@@ -143,41 +143,41 @@ The service starts the following components in order:
### Check if the service is running
```bash
sudo systemctl is-active hive
sudo systemctl is-active spinifex
```
### View recent logs
```bash
sudo journalctl -u hive --since "1 hour ago"
sudo journalctl -u spinifex --since "1 hour ago"
```
### Check service configuration
```bash
sudo systemctl cat hive
sudo systemctl cat spinifex
```
### Verify binary exists
```bash
ls -la /opt/hive/bin/hive
ls -la /opt/spinifex/bin/spx
```
### Test service commands manually
```bash
/opt/hive/bin/hive service nats start
/opt/hive/bin/hive service predastore start
/opt/hive/bin/hive service viperblock start
/opt/spinifex/bin/spx service nats start
/opt/spinifex/bin/spx service predastore start
/opt/spinifex/bin/spx service viperblock start
```
## Customization
If you need to modify the service configuration:
1. Edit the service file: `sudo nano /etc/systemd/system/hive.service`
1. Edit the service file: `sudo nano /etc/systemd/system/spinifex.service`
2. Reload systemd: `sudo systemctl daemon-reload`
3. Restart the service: `sudo systemctl restart hive`
3. Restart the service: `sudo systemctl restart spinifex`
## Security Notes
- The service runs as root, which is necessary for system-level services
- Security hardening is enabled with `ProtectSystem=strict` and `NoNewPrivileges=true`
- Only specific paths are writable: `/opt/hive`, `/var/log`, `/var/run`
- Only specific paths are writable: `/opt/spinifex`, `/var/log`, `/var/run`
-49
View File
@@ -1,49 +0,0 @@
[Unit]
Description=Hive Services
Documentation=https://github.com/mulgadc/hive
After=network.target
Wants=network.target
[Service]
Type=oneshot
RemainAfterExit=yes
User=root
Group=root
WorkingDirectory=/opt/hive
Environment=PATH=/opt/hive/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Start NATS service
ExecStart=/opt/hive/bin/hive service nats start
ExecStartPost=/bin/sleep 3
# Start Predastore service
ExecStart=/opt/hive/bin/hive service predastore start
ExecStartPost=/bin/sleep 2
# Start Viperblock service
ExecStart=/opt/hive/bin/hive service viperblock start
# Stop services in reverse order
ExecStop=/opt/hive/bin/hive service viperblock stop
ExecStopPost=/bin/sleep 2
ExecStop=/opt/hive/bin/hive service predastore stop
ExecStopPost=/bin/sleep 2
ExecStop=/opt/hive/bin/hive service nats stop
# Restart policy
Restart=on-failure
RestartSec=10
# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/hive /var/log /var/run
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=hive
[Install]
WantedBy=multi-user.target
@@ -1,7 +1,7 @@
#!/bin/bash
# Hive Service Installation Script
# This script installs the hive systemd service on Ubuntu/Debian systems
# Spinifex Service Installation Script
# This script installs the spinifex systemd service on Ubuntu/Debian systems
set -e
@@ -38,7 +38,7 @@ fi
# Get the directory where this script is located
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SERVICE_FILE="$SCRIPT_DIR/hive.service"
SERVICE_FILE="$SCRIPT_DIR/spinifex.service"
# Check if service file exists
if [[ ! -f "$SERVICE_FILE" ]]; then
@@ -46,7 +46,7 @@ if [[ ! -f "$SERVICE_FILE" ]]; then
exit 1
fi
print_status "Installing Hive systemd service..."
print_status "Installing Spinifex systemd service..."
# Copy service file to systemd directory
cp "$SERVICE_FILE" /etc/systemd/system/
@@ -55,20 +55,20 @@ cp "$SERVICE_FILE" /etc/systemd/system/
systemctl daemon-reload
# Enable the service to start on boot
systemctl enable hive.service
systemctl enable spinifex.service
print_status "Service installed successfully!"
print_status "You can now use the following commands:"
echo " systemctl start hive # Start the service"
echo " systemctl stop hive # Stop the service"
echo " systemctl status hive # Check service status"
echo " systemctl restart hive # Restart the service"
echo " journalctl -u hive -f # View service logs"
echo " systemctl start spinifex # Start the service"
echo " systemctl stop spinifex # Stop the service"
echo " systemctl status spinifex # Check service status"
echo " systemctl restart spinifex # Restart the service"
echo " journalctl -u spinifex -f # View service logs"
# Check if hive binary exists
if [[ ! -f "/opt/hive/bin/hive" ]]; then
print_warning "Hive binary not found at /opt/hive/bin/hive"
print_warning "Please ensure hive is installed in /opt/hive before starting the service"
# Check if spinifex binary exists
if [[ ! -f "/opt/spinifex/bin/spinifex" ]]; then
print_warning "Spinifex binary not found at /opt/spinifex/bin/spinifex"
print_warning "Please ensure spinifex is installed in /opt/spinifex before starting the service"
fi
print_status "Installation complete!"
+49
View File
@@ -0,0 +1,49 @@
[Unit]
Description=Spinifex Services
Documentation=https://github.com/mulgadc/spinifex
After=network.target
Wants=network.target
[Service]
Type=oneshot
RemainAfterExit=yes
User=root
Group=root
WorkingDirectory=/opt/spinifex
Environment=PATH=/opt/spinifex/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Start NATS service
ExecStart=/opt/spinifex/bin/spinifex service nats start
ExecStartPost=/bin/sleep 3
# Start Predastore service
ExecStart=/opt/spinifex/bin/spinifex service predastore start
ExecStartPost=/bin/sleep 2
# Start Viperblock service
ExecStart=/opt/spinifex/bin/spinifex service viperblock start
# Stop services in reverse order
ExecStop=/opt/spinifex/bin/spinifex service viperblock stop
ExecStopPost=/bin/sleep 2
ExecStop=/opt/spinifex/bin/spinifex service predastore stop
ExecStopPost=/bin/sleep 2
ExecStop=/opt/spinifex/bin/spinifex service nats stop
# Restart policy
Restart=on-failure
RestartSec=10
# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/spinifex /var/log /var/run
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=spinifex
[Install]
WantedBy=multi-user.target
+400 -120
View File
@@ -1,36 +1,132 @@
# AWS Command Implementation Matrix
# Command Implementation Matrix
## EC2 - Instance Management
## Spinifex Admin CLI
Platform management commands not exposed via the AWS gateway API. These are CLI-only commands.
### Service Management
Service lifecycle commands for starting, stopping, and checking status of all Spinifex cluster services. Each service subcommand supports `start`, `stop`, and `status` operations.
| Command | Flags | Prerequisites | Basic Logic | Status |
|---------|-------|---------------|-------------|--------|
| `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--backend` (distributed/filesystem, default: distributed), `--node-id` (default: 0), `--pprof`, `--pprof-output` | TLS cert/key, base-path, config-path required | Creates predastore service instance with S3-compatible storage backend → starts service | **DONE** |
| `spx service predastore stop` | — | Predastore must be running | Stops the predastore service | **DONE** |
| `spx service predastore status` | — | None | Reports predastore service status | **DONE** |
| `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (default: /opt/spinifex/lib/nbdkit-viperblock-plugin.so) | `--config` required, plugin-path must exist on disk | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | **DONE** |
| `spx service viperblock stop` | — | Viperblock must be running | Stops the viperblock service | **DONE** |
| `spx service viperblock status` | — | None | Reports viperblock service status | **DONE** |
| `spx service nats start` | `--port` (default: 4222), `--host` (default: 0.0.0.0), `--debug`, `--data-dir`, `--jetstream` | None | Starts embedded NATS server with optional JetStream | **DONE** |
| `spx service nats stop` | — | NATS must be running | Stops the NATS service | **DONE** |
| `spx service nats status` | — | None | Reports NATS service status | **DONE** |
| `spx service spinifex start` | `--wal-dir` | `--config` required | Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) | **DONE** |
| `spx service spinifex stop` | — | Spinifex daemon must be running | Stops the spinifex daemon service | **DONE** |
| `spx service spinifex status` | — | None | Reports spinifex daemon service status | **DONE** |
| `spx service awsgw start` | `--host` (default: 0.0.0.0:9999), `--tls-cert`, `--tls-key`, `--debug` | `--config` required | Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS | **DONE** |
| `spx service awsgw stop` | — | AWS gateway must be running | Stops the AWS gateway service | **DONE** |
| `spx service awsgw status` | — | None | Reports AWS gateway service status | **DONE** |
| `spx service spinifex-ui start` | `--port` (default: 3000), `--host` (default: 0.0.0.0), `--tls-cert`, `--tls-key` | None | Starts embedded web UI server serving the React frontend. Aliases: `ui`, `spinifexui` | **DONE** |
| `spx service spinifex-ui stop` | — | spinifex-ui must be running | Stops the spinifex-ui service | **DONE** |
| `spx service spinifex-ui status` | — | None | Reports spinifex-ui service status | **DONE** |
| `spx service vpcd start` | — | `--config` required, OVN/OVS installed | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) | **DONE** |
| `spx service vpcd stop` | — | vpcd must be running | Stops the vpcd service | **DONE** |
| `spx service vpcd status` | — | None | Reports vpcd service status | **DONE** |
### Cluster Inspection
Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.
| Command | Flags | Prerequisites | Basic Logic | Output Columns | Status |
|---------|-------|---------------|-------------|----------------|--------|
| `spx get nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Loads config → publishes to `spinifex.node.status` fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as `NotReady` | NAME, STATUS, IP, REGION, AZ, UPTIME, VMs, SERVICES | **DONE** |
| `spx get vms` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.vms` fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: `spx get instances` | INSTANCE, STATUS, TYPE, VCPU, MEM, NODE, IP, AGE | **DONE** |
### Resource Monitoring
| Command | Flags | Prerequisites | Basic Logic | Output | Status |
|---------|-------|---------------|-------------|--------|--------|
| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
### Cluster Initialization
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--force` | None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` | 1. Init creates all config files<br>2. Root credentials printed once<br>3. master.key is 32 bytes, mode 0600<br>4. bootstrap.json consumed on first start<br>5. `--force` re-initializes existing config<br>6. AWS CLI profile `spx` auto-configured | **DONE** |
| `spx admin join` | `--host` (required), `--node` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Leader node must be running | Connects to leader node → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster<br>2. Missing host (error)<br>3. Missing node name (error) | **DONE** |
### Version
| Command | Flags | Prerequisites | Basic Logic | Status |
|---------|-------|---------------|-------------|--------|
| `spx version` | — | None | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) | **DONE** |
### Cluster Operations
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Cluster must be running | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. | 1. Shutdown running cluster<br>2. All nodes stop cleanly<br>3. Force shutdown with unresponsive nodes<br>4. Dry-run prints plan | **DONE** |
### Certificate Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `spx admin cert renew` | `--extra-ip` (additional IPs for SANs), `--extra-dns` (additional DNS names for SANs) | Existing CA from `spx admin init` | Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. | 1. Renew with auto-detected IPs<br>2. Renew with extra IPs<br>3. Renew with extra DNS names | **DONE** |
### Upgrade Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `spx admin upgrade` | `--yes` (apply without prompting), `--config-dir` (persistent, default: `~/spinifex/config`), `--spinifex-dir` (persistent, default: `~/spinifex`) | Existing installation (`spinifex.toml` must exist in config dir) | Reads current config versions from registry → computes pending config migrations (from→to per target) → prints version summary and pending list → prompts for confirmation unless `--yes` → runs `migrate.DefaultRegistry.RunAllConfig()` to apply migrations to config files. Intended for upgrades between Spinifex versions. Operators can skip migrations during install by setting `INSTALL_SPINIFEX_SKIP_MIGRATE=1`, then run `spx admin upgrade` manually to review and apply. After completion, services must be restarted with `sudo systemctl restart spinifex.target`. Invoked non-interactively by `scripts/setup.sh` with `--yes`. | 1. No installation present (error, suggests `spx admin init`)<br>2. No pending migrations (prints "No pending config migrations")<br>3. Pending migrations with interactive confirmation (y/yes applies, anything else aborts)<br>4. Pending migrations with `--yes` (applies without prompt)<br>5. Migration failure surfaces error and exits non-zero | **DONE** |
### Account Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `spx admin account create` | `--name` | Cluster must be running (NATS), master key must exist | Connects to NATS → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates `admin` user in new account → creates access key for admin → creates AdministratorAccess policy (Action:*, Resource:*) → attaches policy → prints credentials | 1. Create first account (ID 000000000001)<br>2. Sequential IDs (000000000002, etc.)<br>3. Admin user has full access<br>4. Credentials printed once | **DONE** |
| `spx admin account list` | — | Cluster must be running (NATS), master key must exist | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created | 1. List all accounts<br>2. Empty list (only global account) | **DONE** |
### Image Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `spx admin images import` | `--name` | Cluster must be running, image file must exist locally | Imports a local disk image (QCOW2) as an AMI → uploads to Predastore S3 → registers as available AMI | 1. Import valid QCOW2 image<br>2. Image appears in `describe-images` | **DONE** |
| `spx admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
## AWS Commands
### EC2 - Instance Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `run-instances` | `--image-id`, `--instance-type`, `--count` (Min/MaxCount), `--key-name`, `--user-data`, `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination) | `--security-group-ids`, `--subnet-id`, `--placement`, `--tag-specifications`, `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--iam-instance-profile`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | `describe-images` (AMI must exist), `create-key-pair` (optional), VPC/SG (optional) | Gateway parses AWS query → NATS `ec2.runinstances` → daemon creates QEMU/KVM VM with viperblock-backed root volume via NBD → cloud-init injects user-data/keys → returns reservation with instance ID | 1. Launch with valid AMI and key pair<br>2. Launch with invalid AMI ID (error)<br>3. Launch with block device mappings (custom volume size)<br>4. Launch multiple instances (MinCount/MaxCount)<br>5. DryRun returns validation-only<br>6. Invalid instance type returns error | **DONE** |
| `describe-instances` | `--instance-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | Gateway fans out NATS `ec2.DescribeInstances` to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list | 1. Describe all instances (no filter)<br>2. Describe by instance ID<br>3. Describe with filters (e.g. instance-state-name)<br>4. Instance not found returns empty set<br>5. Multi-node aggregation returns instances from all nodes | **DONE** |
| `run-instances` | `--image-id`, `--instance-type`, `--count` (Min/MaxCount), `--key-name`, `--user-data`, `--subnet-id` (auto-creates ENI, assigns private IP), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName only — routes via spread or cluster strategy) | `--security-group-ids`, `--tag-specifications`, `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--iam-instance-profile`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | `describe-images` (AMI must exist), `create-key-pair` (optional), VPC/SG (optional) | Gateway parses AWS query → if Placement.GroupName set, looks up strategy: spread → `distributeInstancesSpread()` (1 instance per node, atomic CAS reservation), cluster → `distributeInstancesCluster()` (pin all to single node); otherwise NATS `ec2.runinstances` → daemon creates QEMU/KVM VM with viperblock-backed root volume via NBD → if SubnetId provided, auto-creates ENI with private IP → cloud-init injects user-data/keys → on termination, removes instance from placement group → returns reservation with instance ID | 1. Launch with valid AMI and key pair<br>2. Launch with invalid AMI ID (error)<br>3. Launch with block device mappings (custom volume size)<br>4. Launch multiple instances (MinCount/MaxCount)<br>5. Launch with subnet-id (auto-creates ENI)<br>6. Invalid instance type returns error<br>7. Launch with spread placement group (1 per node)<br>8. Launch with cluster placement group (all on one node)<br>9. Insufficient capacity for placement group (error) | **DONE** |
| `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:\*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | None | Gateway fans out NATS `ec2.DescribeInstances` to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list. Filters applied per-node before aggregation (reduces payload). Also applies to stopped/terminated instances via `describeInstancesFromKV()`. | 1. Describe all instances (no filter)<br>2. Describe by instance ID<br>3. Describe with filters (e.g. instance-state-name)<br>4. Instance not found returns empty set<br>5. Multi-node aggregation returns instances from all nodes<br>6. Filter by tag<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
| `start-instances` | `--instance-ids` | `--dry-run`, `--force` | `run-instances` (instance must exist in stopped state) | Gateway sends NATS `ec2.cmd.{instance-id}` → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running | 1. Start a stopped instance<br>2. Start already-running instance (error: IncorrectInstanceState)<br>3. Start with invalid instance ID<br>4. Verify volumes re-mount on start | **DONE** |
| `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | `run-instances` (instance must be running) | Gateway sends NATS to target node → daemon issues QMP `system_powerdown` for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped | 1. Graceful stop of running instance<br>2. Force stop (kills QEMU process)<br>3. Stop already-stopped instance (error)<br>4. Verify ~30s heartbeat detection | **DONE** |
| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID | **DONE** |
| `reboot-instances` | — | `--instance-ids`, `--dry-run` | `run-instances` (instance must be running) | Gateway sends NATS to target node → daemon issues QMP `system_reset` → instance reboots without stopping | 1. Reboot running instance<br>2. Reboot stopped instance (error)<br>3. Verify instance stays in running state after reboot | **NOT STARTED** (parser test exists) |
| `reboot-instances` | `--instance-ids` | `--dry-run` | `run-instances` (instance must be running) | Gateway validates instance IDs → sends EC2InstanceCommand with `RebootInstance=true` via NATS `ec2.cmd.{instanceId}` → daemon validates instance is in StateRunning (returns IncorrectInstanceState if stopped) → issues QMP `system_reset` → instance reboots without stopping, stays in running state | 1. Reboot running instance<br>2. Reboot multiple instances<br>3. Reboot stopped instance (error: IncorrectInstanceState)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Verify instance stays in running state after reboot | **DONE** |
| `describe-instance-types` | `--filters` (capacity filter only) | `--instance-types`, `--max-results`, `--next-token`, `--dry-run`, all other filters | None | Gateway fans out NATS `ec2.DescribeInstanceTypes` to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns | 1. List all instance types<br>2. Filter by specific type<br>3. Filter with `capacity=true` shows available slots<br>4. Verify vCPU/memory specs match hardware | **DONE** |
| `modify-instance-attribute` | — | `--instance-id`, `--instance-type`, `--user-data`, `--disable-api-termination`, `--ebs-optimized` | Instance must be stopped | Validate instance stopped → update instance metadata in NATS KV store → changes applied on next start | 1. Change instance type while stopped<br>2. Modify running instance (error)<br>3. Toggle disable-api-termination | **NOT STARTED** |
| `get-console-output` | — | `--instance-id`, `--latest` | Instance must exist | QMP command to read serial console ring buffer → return base64-encoded output with timestamp | 1. Get output from running instance<br>2. Get output from stopped instance<br>3. Verify cloud-init boot logs visible | **NOT STARTED** |
| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data` | `--disable-api-termination`, `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must be stopped (in NATS KV) | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `spinifex-workers` queue group → daemon loads stopped instance from JetStream KV → applies attribute change → writes back to KV → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Modify running instance (error: NotFound — running instances not in KV)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Instance not stopped (error: IncorrectInstanceState)<br>6. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>7. StateReason cleared on type change (recovery from capacity-unavailable)<br>8. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>9. No attribute set (error: InvalidParameterValue)<br>10. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
| `get-console-output` | `--instance-id` | `--latest` (always returns latest), `--dry-run` | Instance must be running on a node | Gateway sends NATS `ec2.{instanceId}.GetConsoleOutput` (per-instance topic, routed to owning node) → daemon reads console log file from disk → returns last 64KB base64-encoded with timestamp. Always available regardless of serial console access setting (matches AWS behavior). | 1. Get output from running instance<br>2. Empty log file returns empty output<br>3. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.DescribeInstanceAttribute` with `spinifex-workers` queue group → daemon checks running instances first (`d.Instances.VMS`), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (`instanceType`, `userData`) return real values; unstored attributes return AWS defaults (`instanceInitiatedShutdownBehavior`=stop, `disableApiTermination`=false, etc.) | 1. Get instanceType from running instance<br>2. Get userData from stopped instance<br>3. Get default disableApiTermination<br>4. Invalid attribute name (error)<br>5. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
| `describe-instance-credit-specifications` | `--instance-ids` | `--filters`, `--max-results`, `--dry-run` | None | Gateway-only stub — returns `CpuCredits: "standard"` for each requested instance ID. No daemon round-trip. T-series credit mode is not persisted. | 1. Get credit spec for T-series instance<br>2. Multiple instance IDs | **DONE** |
| `monitor-instances` | — | `--instance-ids` | Instance must exist | Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state | 1. Enable monitoring<br>2. Verify monitoring state in describe-instances | **NOT STARTED** |
| `unmonitor-instances` | — | `--instance-ids` | Instance must exist | Disable detailed monitoring → revert to basic monitoring → return monitoring state | 1. Disable monitoring<br>2. Verify monitoring state reverts in describe-instances | **NOT STARTED** |
## EC2 - Key Pair Management
### EC2 - Key Pair Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-key-pair` | `--key-name`, `--key-type` (rsa/ed25519) | `--key-format` (pem/ppk), `--tag-specifications`, `--dry-run` | None | NATS `ec2.CreateKeyPair` → daemon generates SSH keypair → stores public key in Predastore S3 (`/bucket/ec2/{name}.pub`) → returns private key material (one-time) | 1. Create RSA key pair<br>2. Create ED25519 key pair<br>3. Duplicate key name (error: InvalidKeyPair.Duplicate)<br>4. Verify key material returned only on creation | **DONE** |
| `describe-key-pairs` | `--key-names`, `--key-pair-ids` | `--filters`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeKeyPairs` → daemon lists public keys from Predastore S3 → returns key names and fingerprints | 1. List all key pairs<br>2. Filter by key name<br>3. Filter by key pair ID<br>4. Non-existent key returns empty | **DONE** |
| `describe-key-pairs` | `--key-names`, `--key-pair-ids`, `--filters` (key-pair-id, key-name, fingerprint, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeKeyPairs` → daemon lists public keys from Predastore S3 → applies filters → returns key names and fingerprints | 1. List all key pairs<br>2. Filter by key name<br>3. Filter by key pair ID<br>4. Non-existent key returns empty<br>5. Unknown filter returns InvalidParameterValue | **DONE** |
| `delete-key-pair` | `--key-name`, `--key-pair-id` | `--dry-run` | Key must exist | NATS `ec2.DeleteKeyPair` → daemon deletes public key from Predastore S3 → returns success | 1. Delete existing key pair<br>2. Delete non-existent key (idempotent, no error)<br>3. Verify key no longer in describe-key-pairs | **DONE** |
| `import-key-pair` | `--key-name`, `--public-key-material` | `--tag-specifications`, `--dry-run` | None | NATS `ec2.ImportKeyPair` → daemon stores provided public key in Predastore S3 → returns key name and fingerprint | 1. Import valid SSH public key<br>2. Import invalid key material (error)<br>3. Import duplicate name (error)<br>4. Verify imported key usable with run-instances | **DONE** |
## EC2 - AMI Image Management
### EC2 - AMI Image Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-images` | `--image-ids` (format validation only) | `--owners`, `--executable-users`, `--filters`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName) | **DONE** |
| `create-image` | — | `--instance-id`, `--name`, `--description`, `--no-reboot`, `--block-device-mappings`, `--tag-specifications` | Instance must exist | Stop instance (unless --no-reboot) → snapshot root volume via viperblock → create AMI metadata in S3 → register as new AMI → restart instance → return ami-ID | 1. Create image from running instance<br>2. Create with --no-reboot<br>3. Invalid instance ID (error)<br>4. Verify new AMI appears in describe-images<br>5. Launch new instance from created AMI | **NOT STARTED** |
| `describe-images` | `--image-ids` (format validation only), `--owners` (self, account ID, alias), `--filters` (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:\*) | `--executable-users`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds, Owners, and Filters → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName)<br>6. Filter by name wildcard<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
| `create-image` | `--instance-id`, `--name`, `--description`, `--tag-specifications` | `--no-reboot`, `--block-device-mappings`, `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.{instanceId}.CreateImage` (per-instance topic) → daemon extracts root volume → snapshots via `ebs.snapshot` NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 (`{amiId}/config.json`) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. | 1. Create image from running instance<br>2. Create image from stopped instance<br>3. Invalid instance ID (error)<br>4. Duplicate AMI name (error)<br>5. Verify new AMI appears in describe-images<br>6. Launch new instance from created AMI | **DONE** |
| `register-image` | — | `--name`, `--description`, `--architecture`, `--root-device-name`, `--virtualization-type`, `--block-device-mappings`, `--boot-mode`, `--ena-support`, `--sriov-net-support`, `--tpm-support`, `--imds-support` | Image data must exist in S3 | Create AMI metadata entry in Predastore → assign ami-ID → set state to available → return ami-ID | 1. Register with valid metadata<br>2. Missing required name (error)<br>3. Verify registered image in describe-images | **NOT STARTED** |
| `deregister-image` | — | `--image-id`, `--dry-run` | AMI must exist, no instances using it | Mark AMI metadata as deregistered in S3 → AMI no longer available for new launches → existing instances unaffected | 1. Deregister existing AMI<br>2. Deregister non-existent AMI (error)<br>3. Verify deregistered AMI not in describe-images<br>4. Existing instances from AMI still run | **NOT STARTED** |
| `copy-image` | — | `--source-image-id`, `--source-region`, `--name`, `--description`, `--encrypted`, `--kms-key-id`, `--client-token`, `--copy-image-tags`, `--tag-specifications` | Source AMI must exist | Copy AMI data between regions/nodes in Predastore → create new AMI metadata with new ami-ID → return new ami-ID | 1. Copy image within same region<br>2. Copy non-existent image (error)<br>3. Verify copied image independent of source | **NOT STARTED** |
@@ -39,30 +135,30 @@
| `modify-image-attribute` | — | `--image-id`, `--launch-permission`, `--description`, `--operation-type`, `--user-ids`, `--user-groups`, `--organization-arns`, `--dry-run` | AMI must exist | Update specific attribute in AMI metadata → persist to S3 | 1. Add launch permission<br>2. Modify description<br>3. Invalid AMI (error) | **NOT STARTED** |
| `reset-image-attribute` | — | `--image-id`, `--attribute`, `--dry-run` | AMI must exist | Reset attribute to default value in AMI metadata | 1. Reset launch permission to owner-only<br>2. Invalid AMI (error) | **NOT STARTED** |
## EC2 - Volume (EBS) Management
### EC2 - Volume (EBS) Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-volumes` | `--volume-ids` (fast-path lookup), `DeleteOnTermination` (from persisted VolumeMetadata) | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeVolumes` → daemon queries viperblock for volume metadata → returns volume list with state, size, attachments, type, DeleteOnTermination flag | 1. List all volumes<br>2. Filter by volume ID<br>3. Filter by attachment state<br>4. Non-existent volume returns empty<br>5. DeleteOnTermination reflects persisted value | **DONE** |
| `describe-volumes` | `--volume-ids` (fast-path lookup), `DeleteOnTermination` (from persisted VolumeMetadata), `--filters` (volume-id, status, size, volume-type, attachment.instance-id, attachment.status, attachment.device, availability-zone, tag:\*) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeVolumes` → daemon queries viperblock for volume metadata → applies filters → returns volume list with state, size, attachments, type, DeleteOnTermination flag | 1. List all volumes<br>2. Filter by volume ID<br>3. Filter by attachment state<br>4. Non-existent volume returns empty<br>5. DeleteOnTermination reflects persisted value<br>6. Filter by status, size, volume-type<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
| `modify-volume` | `--volume-id`, `--size`, `--volume-type`, `--iops` | `--throughput`, `--dry-run`, `--multi-attach-enabled` | Volume must exist | NATS `ec2.ModifyVolume` → daemon sends resize request to viperblock → NBD does not support live resize, instance must be stopped → returns modification state | 1. Increase volume size<br>2. Modify volume type<br>3. Decrease size (error - not supported)<br>4. Modify attached volume (requires stop/start) | **DONE** |
| `create-volume` | `--size`, `--availability-zone`, `--volume-type` (gp3 only) | `--iops`, `--encrypted`, `--snapshot-id`, `--tag-specifications` | Valid AZ configured via `hive init` | Gateway validates input → NATS `ec2.CreateVolume` → daemon generates vol-ID via viperblock → creates empty volume of specified size → persists config.json to Predastore S3 → returns vol-ID with state=available | 1. Create 80GB gp3 volume<br>2. Boundary sizes (1 GiB min, 16384 GiB max)<br>3. Invalid AZ (error)<br>4. Verify volume in describe-volumes<br>5. Unsupported volume type (error - only gp3)<br>6. Size out of range (error) | **DONE** |
| `create-volume` | `--size`, `--availability-zone`, `--volume-type` (gp3 only), `--snapshot-id` (creates volume from snapshot) | `--iops` (hardcoded 3000), `--encrypted` (hardcoded false), `--throughput`, `--tag-specifications` | Valid AZ configured via `spinifex init` | Gateway validates input → NATS `ec2.CreateVolume` → daemon generates vol-ID via viperblock → creates volume (empty or from snapshot) of specified size → persists config.json to Predastore S3 → returns vol-ID with state=available | 1. Create 80GB gp3 volume<br>2. Boundary sizes (1 GiB min, 16384 GiB max)<br>3. Invalid AZ (error)<br>4. Verify volume in describe-volumes<br>5. Unsupported volume type (error - only gp3)<br>6. Size out of range (error)<br>7. Create from snapshot | **DONE** |
| `delete-volume` | `--volume-id` | `--dry-run` | Volume must exist and be detached (state=available) | Gateway validates vol- prefix → NATS `ec2.DeleteVolume` → daemon confirms state=available and no AttachedInstance → NATS `ebs.delete` to viperblockd (stops nbdkit/WAL) → deletes S3 objects under vol-id/, vol-id-efi/, vol-id-cloudinit/ → returns success | 1. Delete detached volume<br>2. Delete attached volume (error: VolumeInUse)<br>3. Delete non-existent volume (error: InvalidVolume.NotFound)<br>4. Verify volume gone from describe-volumes<br>5. Malformed volume ID (error: InvalidVolumeID.Malformed)<br>6. Double delete (idempotent NotFound) | **DONE** |
| `attach-volume` | `--volume-id`, `--instance-id`, `--device` (optional, auto-assigns `/dev/sd[f-p]`) | `--dry-run` | Volume must exist (available), instance must exist (running) | Gateway sends to `ec2.cmd.{instanceId}` → daemon validates volume (Predastore) → `ebs.mount` via NATS (viperblock starts NBD server) → QMP `blockdev-add` (nbd-{volId}) → QMP `device_add` (virtio-blk-pci, vdisk-{volId}) → three-phase rollback on failure → update EBSRequests + BlockDeviceMappings → persist to JetStream + Predastore → respond with VolumeAttachment | 1. Attach volume to running instance<br>2. Auto-assign device name<br>3. Attach already-attached volume (VolumeInUse)<br>4. Attach to non-existent instance (InvalidInstanceID.NotFound)<br>5. Attach to stopped instance (IncorrectInstanceState)<br>6. Volume not found (InvalidVolume.NotFound)<br>7. All device slots full (AttachmentLimitExceeded)<br>8. Volume persists across stop/start | **DONE** |
| `detach-volume` | `--volume-id`, `--instance-id` (optional, resolved via DescribeVolumes), `--device` (optional cross-check), `--force` | `--dry-run` | Volume must be attached, instance must be running | Gateway resolves InstanceId if omitted (via DescribeVolumes) → sends to `ec2.cmd.{instanceId}` → daemon validates (running, attached, not boot/EFI/CloudInit, device match) → three-phase hot-unplug: QMP `device_del` (force continues on failure) → QMP `blockdev-del` (abort if fails, preserves state to prevent double-attach) → `ebs.unmount` via NATS (best-effort) → remove from EBSRequests + BlockDeviceMappings → update volume metadata to available → persist state → respond with VolumeAttachment (state=detaching) | 1. Detach with explicit InstanceId<br>2. Detach without InstanceId (gateway resolution)<br>3. Detach with correct --device cross-check<br>4. Missing VolumeId (InvalidParameterValue)<br>5. Volume not attached (IncorrectState)<br>6. Nonexistent volume (InvalidVolume.NotFound)<br>7. Nonexistent instance (InvalidInstanceID.NotFound)<br>8. Instance not running (IncorrectInstanceState)<br>9. Device mismatch (InvalidParameterValue)<br>10. Boot volume protection (OperationNotPermitted)<br>11. Force flag (continues past device_del failure)<br>12. Volume reusability (re-attach after detach) | **DONE** |
| `describe-volume-status` | `--volume-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates vol- prefix → NATS `ec2.DescribeVolumeStatus` → daemon fetches VolumeConfig from Predastore S3 (parallel for specific IDs, sequential list-all for no IDs) → builds VolumeStatusItem per volume (status=ok, io-enabled=passed, io-performance=not-applicable) → returns InvalidVolume.NotFound for missing explicit IDs → skips internal sub-volumes (-efi, -cloudinit) | 1. List all volume statuses<br>2. Filter by specific volume IDs (fast path)<br>3. Non-existent volume ID returns InvalidVolume.NotFound<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Internal sub-volumes excluded from listing<br>6. Nil/empty input defaults to all volumes | **DONE** |
| `describe-volume-status` | `--volume-ids`, `--filters` (volume-id, volume-status.status, availability-zone) | `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates vol- prefix → NATS `ec2.DescribeVolumeStatus` → daemon fetches VolumeConfig from Predastore S3 (parallel for specific IDs, sequential list-all for no IDs) → applies filters → builds VolumeStatusItem per volume (status=ok, io-enabled=passed, io-performance=not-applicable) → returns InvalidVolume.NotFound for missing explicit IDs → skips internal sub-volumes (-efi, -cloudinit) | 1. List all volume statuses<br>2. Filter by specific volume IDs (fast path)<br>3. Non-existent volume ID returns InvalidVolume.NotFound<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Internal sub-volumes excluded from listing<br>6. Nil/empty input defaults to all volumes<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
| `describe-volumes-modifications` | — | `--volume-ids`, `--filters`, `--max-results` | None | Query pending/completed volume modifications → return modification state, progress, original/target size | 1. Check in-progress modification<br>2. Check completed modification<br>3. No modifications returns empty | **NOT STARTED** |
## EC2 - Snapshot Management
### EC2 - Snapshot Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-snapshot` | `--volume-id`, `--description`, `--tag-specifications` | `--dry-run` | Volume must exist in Predastore | Gateway validates vol- prefix → NATS `ec2.CreateSnapshot` → daemon reads VolumeConfig from Predastore → generates snap-ID via viperblock → stores SnapshotConfig (metadata-only, points to source volume) as completed → returns Snapshot with state=completed, progress=100% | 1. Snapshot from valid volume<br>2. Missing volume ID (InvalidParameterValue)<br>3. Volume not found (InvalidVolume.NotFound)<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Snapshot with description<br>6. Snapshot with tag specifications | **DONE** |
| `create-snapshots` | — | `--instance-specification`, `--description`, `--tag-specifications` | Instance must exist, instance-volume attachment tracking | Create snapshots of all volumes attached to instance → return list of snapshot IDs. Blocked: requires instance-volume attachment tracking to discover which volumes to snapshot. | 1. Snapshot all volumes on instance<br>2. Instance with no volumes | **NOT STARTED** |
| `delete-snapshot` | `--snapshot-id` | `--dry-run` | Snapshot must exist | Gateway validates snap- prefix → NATS `ec2.DeleteSnapshot` → daemon verifies snapshot exists in Predastore → lists and deletes all objects under snapshot prefix → returns success | 1. Delete existing snapshot<br>2. Delete non-existent snapshot (InvalidSnapshot.NotFound)<br>3. Missing snapshot ID (InvalidParameterValue)<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed) | **DONE** |
| `describe-snapshots` | `--snapshot-ids` | `--owner-ids`, `--filters`, `--max-results`, `--dry-run` | None | Gateway validates snap- prefix on IDs → NATS `ec2.DescribeSnapshots` → daemon lists snap- prefixed objects in Predastore → reads SnapshotConfig for each → filters by snapshot ID if specified → returns snapshot list | 1. List all snapshots<br>2. Filter by snapshot ID<br>3. Empty snapshot list<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed) | **DONE** |
| `describe-snapshots` | `--snapshot-ids`, `--filters` (snapshot-id, status, volume-id, volume-size, owner-id, tag:\*) | `--owner-ids`, `--max-results`, `--dry-run` | None | Gateway validates snap- prefix on IDs → NATS `ec2.DescribeSnapshots` → daemon lists snap- prefixed objects in Predastore → reads SnapshotConfig for each → applies filters → returns snapshot list | 1. List all snapshots<br>2. Filter by snapshot ID<br>3. Empty snapshot list<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed)<br>5. Filter by status, volume-id, volume-size<br>6. Unknown filter returns InvalidParameterValue | **DONE** |
| `copy-snapshot` | `--source-snapshot-id`, `--source-region`, `--description` | `--encrypted`, `--dry-run` | Source snapshot must exist | Gateway validates snap- prefix + source region → NATS `ec2.CopySnapshot` → daemon reads source SnapshotConfig → generates new snap-ID → copies metadata (preserves tags, description override) → stores as completed → returns new snapshot ID | 1. Copy within same region<br>2. Copy non-existent snapshot (InvalidSnapshot.NotFound)<br>3. Missing source ID (InvalidParameterValue)<br>4. Missing source region (MissingParameter)<br>5. Copy preserves tags<br>6. Copy with description override | **DONE** |
## EC2 - Tags
### EC2 - Tags
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -70,31 +166,31 @@
| `delete-tags` | `--resources`, `--tags` | `--dry-run` | None | Gateway validates input (resources non-empty) → NATS `ec2.DeleteTags` → daemon removes specified tag keys from resource JSON, or all tags if no keys specified → return success | 1. Delete specific tag by key<br>2. Delete all tags (no tags specified)<br>3. Empty resources list (error: MissingParameter) | **DONE** |
| `describe-tags` | `--filters` (resource-id, resource-type, key, value) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeTags` → daemon lists all `tags/*.json` from Predastore S3 → applies filters (resource-id, resource-type, key, value) → returns tag list with resource type auto-detected from ID prefix | 1. List all tags<br>2. Filter by resource-id<br>3. Filter by resource-type (instance, volume, image, snapshot, vpc, subnet, security-group, route-table, internet-gateway, egress-only-internet-gateway)<br>4. Filter by key<br>5. Filter by value<br>6. Empty result when no tags exist | **DONE** |
## EC2 - Regions & Availability Zones
### EC2 - Regions & Availability Zones
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-regions` | *(returns configured region, endpoint, opt-in status from config — input params ignored)* | `--region-names`, `--filters`, `--all-regions`, `--dry-run` | None | Return configured region from hive init config with Endpoint and OptInStatus → local-only response, no NATS | 1. List all regions<br>2. Filter by region name<br>3. Verify endpoint URL returned<br>4. Verify OptInStatus returned | **DONE** |
| `describe-availability-zones` | *(returns configured AZ, region, zone ID, state, opt-in status from config — input params ignored)* | `--zone-names`, `--filters`, `--all-availability-zones` | None | Return configured AZ from hive init config with zone ID, state, group name, network border group → local-only response, no NATS | 1. List all AZs<br>2. Filter by zone name<br>3. Verify zone state is available<br>4. Verify region name matches config | **DONE** |
| `describe-regions` | *(returns configured region, endpoint, opt-in status from config — input params ignored)* | `--region-names`, `--filters`, `--all-regions`, `--dry-run` | None | Return configured region from spinifex init config with Endpoint and OptInStatus → local-only response, no NATS | 1. List all regions<br>2. Filter by region name<br>3. Verify endpoint URL returned<br>4. Verify OptInStatus returned | **DONE** |
| `describe-availability-zones` | *(returns configured AZ, region, zone ID, state, opt-in status from config — input params ignored)* | `--zone-names`, `--filters`, `--all-availability-zones` | None | Return configured AZ from spinifex init config with zone ID, state, group name, network border group → local-only response, no NATS | 1. List all AZs<br>2. Filter by zone name<br>3. Verify zone state is available<br>4. Verify region name matches config | **DONE** |
## EC2 - Account Attributes
### EC2 - Account Attributes
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-account-attributes` | `--attribute-names` | `--dry-run` | None | Gateway parses input → returns static account attributes: supported-platforms=VPC, default-vpc=none, max-instances=100, vpc-max-security-groups-per-interface=5, max-elastic-ips=5, vpc-max-elastic-ips=20 → local-only response, no NATS | 1. List all account attributes<br>2. Filter by attribute name<br>3. Verify all 6 attributes returned with correct values | **DONE** |
## EC2 - Account Settings (Deferred)
### EC2 - Account Settings (Deferred)
Persistence layer works (NATS JetStream KV), but stored values are not yet enforced by downstream services. Will be completed post-merge.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `enable-ebs-encryption-by-default` | — | `--dry-run` | None | Store encryption default flag in JetStream KV → return `EbsEncryptionByDefault=true` | 1. Enable encryption<br>2. Verify via get | **NOT STARTED** (enforcement pending) |
| `disable-ebs-encryption-by-default` | — | `--dry-run` | None | Clear encryption default flag in JetStream KV → return `EbsEncryptionByDefault=false` | 1. Disable encryption<br>2. Verify via get | **NOT STARTED** (enforcement pending) |
| `get-ebs-encryption-by-default` | — | `--dry-run` | None | Read encryption default flag from JetStream KV → return current state | 1. Get default state<br>2. Verify matches last enable/disable | **NOT STARTED** (enforcement pending) |
| `enable-serial-console-access` | — | `--dry-run` | None | Store serial console access flag in JetStream KV → return enabled=true | 1. Enable access<br>2. Verify via get | **NOT STARTED** (enforcement pending) |
| `disable-serial-console-access` | — | `--dry-run` | None | Clear serial console access flag in JetStream KV → return enabled=false | 1. Disable access | **NOT STARTED** (enforcement pending) |
| `get-serial-console-access-status` | — | `--dry-run` | None | Read serial console access flag from JetStream KV | 1. Get current status | **NOT STARTED** (enforcement pending) |
| `enable-ebs-encryption-by-default` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.EnableEbsEncryptionByDefault` with `spinifex-workers` queue group → daemon stores `EbsEncryptionByDefault=true` in JetStream KV (`spinifex-ec2-account-settings` bucket) → return `EbsEncryptionByDefault=true`. Enforcement deferred: stored value not yet checked by CreateVolume/CreateSnapshot. | 1. Enable encryption<br>2. Verify via get | **STARTED** (enforcement pending) |
| `disable-ebs-encryption-by-default` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.DisableEbsEncryptionByDefault` with `spinifex-workers` queue group → daemon stores `EbsEncryptionByDefault=false` in JetStream KV → return `EbsEncryptionByDefault=false`. | 1. Disable encryption<br>2. Verify via get | **STARTED** (enforcement pending) |
| `get-ebs-encryption-by-default` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.GetEbsEncryptionByDefault` with `spinifex-workers` queue group → daemon reads `EbsEncryptionByDefault` from JetStream KV → return current state. | 1. Get default state<br>2. Verify matches last enable/disable | **STARTED** (enforcement pending) |
| `enable-serial-console-access` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.EnableSerialConsoleAccess` with `spinifex-workers` queue group → daemon stores `SerialConsoleAccessEnabled=true` in JetStream KV (`spinifex-ec2-account-settings` bucket) → return enabled=true. Stored for future interactive serial console support (not currently enforced by any operation). | 1. Enable access<br>2. Verify via get | **STARTED** (enforcement pending) |
| `disable-serial-console-access` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.DisableSerialConsoleAccess` with `spinifex-workers` queue group → daemon stores `SerialConsoleAccessEnabled=false` in JetStream KV → return enabled=false. Stored for future interactive serial console support (not currently enforced by any operation). | 1. Disable access<br>2. Verify via get | **STARTED** (enforcement pending) |
| `get-serial-console-access-status` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.GetSerialConsoleAccessStatus` with `spinifex-workers` queue group → daemon reads `SerialConsoleAccessEnabled` from JetStream KV → return current state. | 1. Get current status<br>2. Verify matches last enable/disable | **DONE** |
| `enable-snapshot-block-public-access` | — | `--state` | None | Store snapshot block public access state in JetStream KV | 1. Enable with block-all-sharing<br>2. Enable with block-new-sharing | **NOT STARTED** (enforcement pending) |
| `disable-snapshot-block-public-access` | — | `--dry-run` | None | Clear snapshot block public access state in JetStream KV | 1. Disable access | **NOT STARTED** (enforcement pending) |
| `get-snapshot-block-public-access-state` | — | `--dry-run` | None | Read snapshot block public access state from JetStream KV | 1. Get current state | **NOT STARTED** (enforcement pending) |
@@ -104,23 +200,27 @@ Persistence layer works (NATS JetStream KV), but stored values are not yet enfor
| `modify-instance-metadata-defaults` | — | `--http-tokens`, `--http-put-response-hop-limit`, `--http-endpoint`, `--instance-metadata-tags` | None | Store instance metadata defaults in JetStream KV | 1. Set httpTokens=required<br>2. Set hop limit | **NOT STARTED** (enforcement pending) |
| `get-instance-metadata-defaults` | — | `--dry-run` | None | Read instance metadata defaults from JetStream KV | 1. Get current defaults | **NOT STARTED** (enforcement pending) |
## EC2 - Egress-Only Internet Gateway
### EC2 - Egress-Only Internet Gateway
Gateway routing and KV CRUD are wired up but no OVN/OVS integration exists. EIGWs are stored in `spinifex-eigw` KV bucket but have no effect on network topology (no vpcd events published, unlike IGW which publishes `vpc.igw-attach`/`vpc.igw-detach`).
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-egress-only-internet-gateway` | — | `--vpc-id`, `--client-token`, `--tag-specifications`, `--dry-run` | VPC must exist | NATS `ec2.CreateEgressOnlyInternetGateway` → daemon generates eigw-ID → creates record in NATS JetStream KV with state=attached → returns EgressOnlyInternetGateway with attachment | 1. Create with valid VpcId<br>2. Missing VpcId (error: MissingParameter)<br>3. Create with inline tags<br>4. Verify ID format eigw-{16hex} | **NOT STARTED** |
| `delete-egress-only-internet-gateway` | — | `--egress-only-internet-gateway-id`, `--dry-run` | EIGW must exist | NATS `ec2.DeleteEgressOnlyInternetGateway` → daemon verifies EIGW exists in KV → deletes from KV → returns ReturnCode=true | 1. Delete existing EIGW<br>2. Delete non-existent EIGW (error)<br>3. Missing ID (error: MissingParameter) | **NOT STARTED** |
| `describe-egress-only-internet-gateways` | — | `--egress-only-internet-gateway-ids`, `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeEgressOnlyInternetGateways` → daemon lists all keys from KV bucket → filters by ID if specified → returns list | 1. Describe all EIGWs<br>2. Filter by specific ID<br>3. Non-existent ID returns empty list | **NOT STARTED** |
| `create-egress-only-internet-gateway` | `--vpc-id`, `--tag-specifications` | `--client-token`, `--dry-run` | VPC must exist | Gateway validates VpcId → NATS `ec2.CreateEgressOnlyInternetGateway` → daemon generates eigw-ID → creates EgressOnlyIGWRecord in `spinifex-eigw` KV bucket with state=attached (always attached at creation, unlike IGW) → returns EgressOnlyInternetGateway with attachment | 1. Create with valid VpcId<br>2. Missing VpcId (error: MissingParameter)<br>3. Create with inline tags<br>4. Verify ID format eigw-{16hex} | **STARTED** (KV only, no OVN) |
| `delete-egress-only-internet-gateway` | `--egress-only-internet-gateway-id` | `--dry-run` | EIGW must exist | NATS `ec2.DeleteEgressOnlyInternetGateway` → daemon verifies EIGW exists in KV → deletes from `spinifex-eigw` KV → returns ReturnCode=true. No detach required (unlike IGW). | 1. Delete existing EIGW<br>2. Delete non-existent EIGW (error)<br>3. Missing ID (error: MissingParameter) | **STARTED** (KV only, no OVN) |
| `describe-egress-only-internet-gateways` | `--egress-only-internet-gateway-ids`, `--filters` (egress-only-internet-gateway-id, tag:\*) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeEgressOnlyInternetGateways` → daemon lists all keys from `spinifex-eigw` KV bucket → filters by ID if specified → applies filters → always includes Attachments array → returns list | 1. Describe all EIGWs<br>2. Filter by specific ID<br>3. Non-existent ID returns empty list<br>4. Unknown filter returns InvalidParameterValue | **STARTED** (KV only, no OVN) |
## EC2 - Placement Groups
### EC2 - Placement Groups
Placement groups are stored in `spinifex-placement-groups` JetStream KV bucket, keyed by `{accountID}.{groupName}`. All mutations use CAS (compare-and-swap) with 5-retry limit for optimistic concurrency. Supported strategies: **spread** (strict 1 instance per node) and **cluster** (all instances pinned to single node). Partition strategy is rejected. Internal NATS operations (ReserveSpreadNodes, FinalizeSpreadInstances, ReleaseSpreadNodes, RemoveInstanceFromPlacementGroup, ReserveClusterNode, FinalizeClusterInstances) support atomic multi-node coordination for RunInstances and TerminateInstances.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-placement-group` | — | `--group-name`, `--strategy` (cluster/spread/partition), `--partition-count`, `--spread-level`, `--tag-specifications` | None | NATS `ec2.CreatePlacementGroup` → daemon stores placement group in JetStream KV with strategy, state=available → return group metadata | 1. Create with cluster strategy<br>2. Create with spread strategy<br>3. Create with partition strategy + partition count<br>4. Duplicate name (error)<br>5. Invalid strategy (error) | **NOT STARTED** |
| `delete-placement-group` | — | `--group-name`, `--dry-run` | Group must exist, no instances in group | NATS `ec2.DeletePlacementGroup` → daemon verifies group exists and is empty → delete from KV → return success | 1. Delete empty group<br>2. Delete group with instances (error)<br>3. Delete non-existent group (error) | **NOT STARTED** |
| `describe-placement-groups` | — | `--group-names`, `--group-ids`, `--filters` | None | NATS `ec2.DescribePlacementGroups` → daemon lists groups from KV → return list with strategy and state | 1. List all groups<br>2. Filter by name<br>3. Filter by strategy | **NOT STARTED** |
| `create-placement-group` | `--group-name`, `--strategy` (spread/cluster) | `--partition-count`, `--spread-level` (hardcoded to "host"), `--tag-specifications`, `--dry-run` | None | Gateway validates GroupName + Strategy → NATS `ec2.CreatePlacementGroup` with `spinifex-workers` queue group → daemon generates pg-ID (pg-{16hex}) → stores PlacementGroupRecord in `spinifex-placement-groups` KV with state=available, SpreadLevel="host" for spread → rejects partition strategy (InvalidParameterValue) → rejects duplicate names (InvalidPlacementGroup.Duplicate) → returns PlacementGroup | 1. Create with spread strategy<br>2. Create with cluster strategy<br>3. Create with partition strategy (error: InvalidParameterValue)<br>4. Duplicate name (error: InvalidPlacementGroup.Duplicate)<br>5. Missing GroupName (error)<br>6. Missing Strategy (error) | **DONE** |
| `delete-placement-group` | `--group-name` | `--dry-run` | Group must exist, no instances in group | NATS `ec2.DeletePlacementGroup` → daemon verifies group exists (InvalidPlacementGroup.Unknown) → counts instances in NodeInstances map → rejects if non-empty (InvalidPlacementGroup.InUse) → deletes from KV → returns success | 1. Delete empty group<br>2. Delete group with instances (error: InvalidPlacementGroup.InUse)<br>3. Delete non-existent group (error: InvalidPlacementGroup.Unknown) | **DONE** |
| `describe-placement-groups` | `--group-names`, `--group-ids`, `--filters` (strategy, state, spread-level, group-name) | `--dry-run` | None | NATS `ec2.DescribePlacementGroups` → daemon lists all keys from `spinifex-placement-groups` KV → filters by GroupNames, GroupIds, or Filters → returns list with Strategy, State, SpreadLevel, GroupId | 1. List all groups<br>2. Filter by name<br>3. Filter by group ID<br>4. Filter by strategy<br>5. Filter by state<br>6. Empty result when no groups exist | **DONE** |
## EC2 - Dedicated Hosts
### EC2 - Dedicated Hosts
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -128,7 +228,7 @@ Persistence layer works (NATS JetStream KV), but stored values are not yet enfor
| `describe-hosts` | — | `--host-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeHosts` → daemon lists hosts from KV → return host list with capacity and instance info | 1. List all hosts<br>2. Filter by host ID<br>3. Filter by instance type | **NOT STARTED** |
| `release-hosts` | — | `--host-ids` | Host must exist, no running instances | NATS `ec2.ReleaseHosts` → daemon verifies no instances on host → delete from KV → return success/failure per host | 1. Release empty host<br>2. Release host with instances (error)<br>3. Release non-existent host (error) | **NOT STARTED** |
## EC2 - IPv4 Pools
### EC2 - IPv4 Pools
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -136,7 +236,7 @@ Persistence layer works (NATS JetStream KV), but stored values are not yet enfor
| `delete-public-ipv4-pool` | — | `--pool-id`, `--dry-run` | Pool must exist and be empty | NATS `ec2.DeletePublicIpv4Pool` → daemon verifies pool empty → delete from KV → return success | 1. Delete empty pool<br>2. Delete pool with addresses (error) | **NOT STARTED** |
| `describe-public-ipv4-pools` | — | `--pool-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribePublicIpv4Pools` → daemon lists pools from KV → return pool list with address counts | 1. List all pools<br>2. Filter by pool ID | **NOT STARTED** |
## EC2 - DHCP Options
### EC2 - DHCP Options
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -145,7 +245,7 @@ Persistence layer works (NATS JetStream KV), but stored values are not yet enfor
| `describe-dhcp-options` | — | `--dhcp-options-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeDhcpOptions` → daemon lists DHCP options from KV → return list with configurations | 1. List all DHCP options<br>2. Filter by ID | **NOT STARTED** |
| `associate-dhcp-options` | — | `--dhcp-options-id`, `--vpc-id`, `--dry-run` | DHCP options and VPC must exist | NATS `ec2.AssociateDhcpOptions` → daemon links DHCP options to VPC → update VPC metadata → return success | 1. Associate with VPC<br>2. Missing DHCP options ID (error)<br>3. Missing VPC ID (error) | **NOT STARTED** |
## EC2 - Capacity Reservations
### EC2 - Capacity Reservations
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -154,71 +254,80 @@ Persistence layer works (NATS JetStream KV), but stored values are not yet enfor
| `describe-capacity-reservations` | — | `--capacity-reservation-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeCapacityReservations` → daemon lists reservations from KV → return list with state and counts | 1. List all reservations<br>2. Filter by ID<br>3. Filter by instance type | **NOT STARTED** |
| `modify-capacity-reservation` | — | `--capacity-reservation-id`, `--instance-count`, `--end-date`, `--end-date-type` | Reservation must exist | NATS `ec2.ModifyCapacityReservation` → daemon updates reservation in KV → return success | 1. Modify instance count<br>2. Modify end date<br>3. Missing reservation ID (error) | **NOT STARTED** |
## EC2 - Elastic IP
### EC2 - Elastic IP
EIP operations are conditionally available — only registered when external IPAM is configured (pool mode). EIPs are stored in `spinifex-eip-allocations` KV bucket. Association publishes `vpc.add-nat`/`vpc.delete-nat` events to vpcd for OVN NAT configuration.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `allocate-address` | — | `--domain` (vpc), `--tag-specifications`, `--dry-run` | None | NATS `ec2.AllocateAddress` → daemon allocates IP from pool → stores in JetStream KV with eipalloc-ID → return AllocationId, PublicIp, Domain | 1. Allocate EIP<br>2. Verify in describe-addresses<br>3. Allocate without --domain uses default | **NOT STARTED** |
| `release-address` | — | `--allocation-id`, `--dry-run` | EIP must exist and be disassociated | NATS `ec2.ReleaseAddress` → daemon verifies not associated → release IP back to pool → delete from KV → return success | 1. Release disassociated EIP<br>2. Release associated EIP (error)<br>3. Release non-existent (error) | **NOT STARTED** |
| `associate-address` | — | `--allocation-id`, `--instance-id` or `--network-interface-id`, `--dry-run` | EIP and target must exist | NATS `ec2.AssociateAddress` → daemon links EIP to instance/ENI → configure OVS DNAT/SNAT rules → return eipassoc-ID | 1. Associate with instance<br>2. Re-associate (moves EIP)<br>3. Associate non-existent EIP (error) | **NOT STARTED** |
| `disassociate-address` | — | `--association-id`, `--dry-run` | Association must exist | NATS `ec2.DisassociateAddress` → daemon unlinks EIP from instance/ENI → remove OVS DNAT/SNAT rules → return success | 1. Disassociate existing<br>2. Disassociate non-existent (error) | **NOT STARTED** |
| `describe-addresses` | — | `--allocation-ids`, `--filters`, `--public-ips` | None | NATS `ec2.DescribeAddresses` → daemon lists EIPs from KV → return list with association info | 1. List all EIPs<br>2. Filter by allocation ID<br>3. Filter by public IP | **NOT STARTED** |
| `allocate-address` | `--public-ipv4-pool` (optional, specific IPAM pool), `--tag-specifications` | `--domain` (hardcoded vpc), `--dry-run` | EIP service must be initialized (external IPAM configured) | Gateway validates input → NATS `ec2.AllocateAddress` with `spinifex-workers` queue group → daemon allocates IP from external IPAM pool (named or best-matching) → creates EIPRecord with state=allocated in `spinifex-eip-allocations` KV → returns AllocationId, PublicIp, Domain=vpc, PublicIpv4Pool | 1. Allocate EIP from default pool<br>2. Allocate from specific pool<br>3. Pool exhausted (InsufficientAddressCapacity)<br>4. With tag specifications<br>5. Verify in describe-addresses | **DONE** |
| `release-address` | `--allocation-id` | `--dry-run` | EIP must exist and be disassociated | Gateway validates AllocationId non-empty → NATS `ec2.ReleaseAddress` → daemon retrieves EIP record from KV → checks state != associated (InvalidAddressLocked if still associated) → releases IP back to IPAM pool → deletes KV record → returns success | 1. Release disassociated EIP<br>2. Release associated EIP (error: InvalidAddressLocked)<br>3. Release non-existent (error: InvalidAllocationID.NotFound)<br>4. Missing AllocationId (error: MissingParameter) | **DONE** |
| `associate-address` | `--allocation-id`, `--network-interface-id` OR `--instance-id`, `--private-ip-address` (optional override) | `--dry-run`, `--allow-reassociation` | EIP and target ENI/instance must exist | Gateway validates AllocationId + one of NetworkInterfaceId/InstanceId required → NATS `ec2.AssociateAddress` → daemon retrieves EIP → resolves target ENI (by ID or instance lookup) → generates AssociationId → updates record state=associated with ENI/instance/VPC/private IP → publishes `vpc.add-nat` event to vpcd (OVN NAT configuration) → returns AssociationId | 1. Associate with instance<br>2. Associate with specific ENI<br>3. Associate with private IP override<br>4. Missing AllocationId (MissingParameter)<br>5. Missing target (MissingParameter)<br>6. Non-existent EIP (InvalidAllocationID.NotFound)<br>7. Non-existent ENI (InvalidNetworkInterfaceID.NotFound) | **DONE** |
| `disassociate-address` | `--association-id` | `--dry-run` | Association must exist | Gateway validates AssociationId non-empty → NATS `ec2.DisassociateAddress` → daemon scans KV to find EIP by AssociationId → publishes `vpc.delete-nat` event to vpcd → clears association fields → reverts state to allocated → updates KV → returns success | 1. Disassociate existing<br>2. Non-existent association (InvalidAssociationID.NotFound)<br>3. Missing AssociationId (MissingParameter) | **DONE** |
| `describe-addresses` | `--allocation-ids`, `--public-ips`, `--filters` (allocation-id, public-ip, instance-id, association-id, domain, tag:\*) | `--dry-run` | None | NATS `ec2.DescribeAddresses` → daemon scans all EIP records in account → filters by AllocationIds and/or PublicIps if specified → applies filters → returns Address list with allocation, association, PublicIp, PrivateIpAddress, NetworkInterfaceId, InstanceId, Domain | 1. List all EIPs<br>2. Filter by allocation ID<br>3. Filter by public IP<br>4. Non-existent AllocationId returns error<br>5. Empty result when no EIPs<br>6. Filter by instance-id, domain<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
| `describe-addresses-attribute` | `--allocation-ids` (optional filter) | `--attribute`, `--dry-run`, `--max-results`, `--next-token` | None | Gateway validates input → NATS `ec2.DescribeAddressesAttribute` with `spinifex-workers` queue group → daemon looks up EIP records by AllocationIds (or scans all account EIPs if none specified) → returns AddressAttribute list with AllocationId and PublicIp per entry. Unlike DescribeAddresses, returns empty list (not error) for non-existent AllocationIds. PtrRecord always nil (reverse-DNS not supported). | 1. List all EIP attributes<br>2. Filter by allocation IDs<br>3. Non-existent AllocationId returns empty (not error)<br>4. Nil input returns error | **DONE** |
## EC2 - NAT Gateway
### EC2 - NAT Gateway
NAT gateways are stored in `spinifex-vpc-nat-gateways` KV bucket. Deleted gateways are moved to `spinifex-vpc-deleted-nat-gateways` with a 1-hour TTL for Terraform polling. SNAT rule coordination uses `vpc.add-nat-gateway` and `vpc.delete-nat-gateway` NATS events (published via route table service on CreateRoute and during DeleteNatGateway respectively).
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-nat-gateway` | — | `--subnet-id`, `--allocation-id`, `--connectivity-type`, `--tag-specifications` | Subnet must exist, EIP must be allocated (for public type) | NATS `ec2.CreateNatGateway` → daemon creates NAT gateway in JetStream KV with nat-ID → configure OVS SNAT rules → state=pending→available → return NatGateway | 1. Create public NAT gateway<br>2. Create private NAT gateway (no allocation-id)<br>3. Invalid subnet (error) | **NOT STARTED** |
| `delete-nat-gateway` | — | `--nat-gateway-id`, `--dry-run` | NAT gateway must exist | NATS `ec2.DeleteNatGateway` → daemon marks as deleting → remove OVS SNAT rules → delete from KV → return NatGatewayId | 1. Delete existing NAT gateway<br>2. Delete non-existent (error) | **NOT STARTED** |
| `describe-nat-gateways` | — | `--nat-gateway-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeNatGateways` → daemon lists NAT gateways from KV → return list with state, subnet, addresses | 1. List all NAT gateways<br>2. Filter by ID<br>3. Filter by VPC ID | **NOT STARTED** |
| `create-nat-gateway` | `--subnet-id`, `--allocation-id` | `--connectivity-type` (public only), `--tag-specifications`, `--dry-run` | Subnet must exist, EIP must be allocated and unassociated | Gateway validates SubnetId + AllocationId (required) → NATS `ec2.CreateNatGateway` with `spinifex-workers` queue group → daemon verifies subnet exists in `spinifex-vpc-subnets` KV → verifies EIP exists in `spinifex-vpc-eips` KV and has no AssociationId (Resource.AlreadyAssociated if in use) → generates nat-ID → stores NatGatewayRecord in `spinifex-vpc-nat-gateways` KV with state=available → returns NatGateway with addresses | 1. Create NAT gateway (happy path)<br>2. Subnet not found (InvalidSubnetID.NotFound)<br>3. EIP not found (InvalidAllocationID.NotFound)<br>4. EIP already associated (Resource.AlreadyAssociated)<br>5. Missing SubnetId or AllocationId (MissingParameter) | **DONE** |
| `delete-nat-gateway` | `--nat-gateway-id` | `--dry-run` | NAT gateway must exist | Gateway validates NatGatewayId (required) → NATS `ec2.DeleteNatGateway` → daemon retrieves record from KV → scans all route tables in VPC for routes targeting this NAT gateway → publishes `vpc.delete-nat-gateway` event per affected subnet (SNAT rule cleanup) → sets state=deleted → stores in `spinifex-vpc-deleted-nat-gateways` with 1-hour TTL → deletes from active KV bucket | 1. Delete existing NAT gateway<br>2. Delete non-existent (InvalidNatGatewayID.NotFound)<br>3. Deleted gateway visible when queried by ID (state=deleted)<br>4. Deleted gateway excluded from general list queries (TTL auto-expire) | **DONE** |
| `describe-nat-gateways` | `--nat-gateway-ids`, `--filters` (vpc-id, state) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeNatGateways` → daemon scans `spinifex-vpc-nat-gateways` KV → applies NatGatewayIds filter and/or vpc-id/state filters → for specific IDs not found in active bucket, checks deleted bucket → returns InvalidNatGatewayID.NotFound if ID never existed → returns list with state, addresses, subnet, VPC | 1. List all NAT gateways<br>2. Filter by specific IDs<br>3. Filter by state<br>4. Filter by vpc-id<br>5. Non-existent ID (InvalidNatGatewayID.NotFound)<br>6. Deleted gateway visible by ID query | **DONE** |
| `assign-private-nat-gateway-address` | — | `--nat-gateway-id`, `--private-ip-addresses` | NAT gateway must exist (private type) | NATS `ec2.AssignPrivateNatGatewayAddress` → daemon assigns private IPs to NAT gateway → return assigned addresses | 1. Assign address<br>2. Missing NAT gateway ID (error) | **NOT STARTED** |
| `associate-nat-gateway-address` | — | `--nat-gateway-id`, `--allocation-ids` | NAT gateway and EIPs must exist | NATS `ec2.AssociateNatGatewayAddress` → daemon associates EIPs with NAT gateway → return association info | 1. Associate EIP<br>2. Missing NAT gateway ID (error) | **NOT STARTED** |
## EC2 - VPC Networking (Core)
### EC2 - VPC Networking (Core)
Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Single AZ for Hive v1.
Requires OVN/OVS (`apt install openvswitch-switch ovn-host`). VPC/Subnet/ENI/SG CRUD stores metadata in NATS KV and publishes events to vpcd (the network topology daemon) which translates to OVN logical switches, ports, routers, and ACLs. Single AZ for Spinifex v1. IPAM uses CAS-based allocation from NATS KV (`spinifex-vpc-ipam` bucket). Security groups stored in `spinifex-vpc-security-groups` KV bucket with optimistic locking on rule updates.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-vpc` | — | `--cidr-block`, `--tag-specifications`, `--instance-tenancy` | None | Create OVS bridge for VPC → assign vpc-ID → store VPC metadata in NATS KV → configure CIDR range → return VPC with state=available | 1. Create VPC with /16 CIDR<br>2. Overlapping CIDR (error)<br>3. Invalid CIDR format (error)<br>4. Verify in describe-vpcs | **NOT STARTED** |
| `delete-vpc` | — | `--vpc-id` | VPC must be empty (no subnets, gateways, instances) | Verify no dependent resources → delete OVS bridge → remove metadata → return success | 1. Delete empty VPC<br>2. Delete VPC with subnets (error: DependencyViolation)<br>3. Delete non-existent VPC (error) | **NOT STARTED** |
| `describe-vpcs` | — | `--vpc-ids`, `--filters`, `--max-results` | None | Query NATS KV for VPC metadata → return VPC list with CIDR, state, tags | 1. List all VPCs<br>2. Filter by VPC ID<br>3. Filter by CIDR block | **NOT STARTED** |
| `modify-vpc-attribute` | — | `--vpc-id`, `--enable-dns-support`, `--enable-dns-hostnames` | VPC must exist | NATS `ec2.ModifyVpcAttribute` → daemon updates VPC attribute in KV → return success | 1. Enable DNS support<br>2. Enable DNS hostnames<br>3. Missing VPC ID (error) | **NOT STARTED** |
| `create-vpc` | `--cidr-block`, `--tag-specifications` | `--instance-tenancy`, `--dry-run` | None | Gateway validates CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS `ec2.CreateVpc` → daemon allocates unique VNI from `spinifex-vpc-vni-counter` KV → generates vpc-ID → stores VPCRecord in `spinifex-vpc-vpcs` KV → publishes `vpc.create` event to vpcd (creates OVN logical switch) → returns VPC with state=available | 1. Create VPC with /16 CIDR<br>2. Invalid CIDR format (error: InvalidVpcRange)<br>3. CIDR prefix outside /16–/28 (error)<br>4. Verify in describe-vpcs<br>5. Tags applied at creation | **DONE** |
| `delete-vpc` | `--vpc-id` | `--dry-run` | VPC must be empty (no subnets) | Gateway validates VpcId → NATS `ec2.DeleteVpc` → daemon checks for dependent subnets in `spinifex-vpc-subnets` KV → returns DependencyViolation if subnets exist → deletes from `spinifex-vpc-vpcs` KV → publishes `vpc.delete` event to vpcd (removes OVN resources) | 1. Delete empty VPC<br>2. Delete VPC with subnets (DependencyViolation)<br>3. Delete non-existent VPC (error) | **DONE** |
| `describe-vpcs` | `--vpc-ids`, `--filters` (vpc-id, state, cidr-block, is-default, owner-id, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeVpcs` → daemon lists all keys from `spinifex-vpc-vpcs` KV → filters by VPC IDs if specified → applies filters → returns error for non-existent requested IDs → returns VPC list with CIDR, state, CIDR associations | 1. List all VPCs<br>2. Filter by VPC ID<br>3. Non-existent VPC ID returns error<br>4. Filter by state, cidr-block, is-default<br>5. Unknown filter returns InvalidParameterValue | **DONE** |
| `modify-vpc-attribute` | `--vpc-id`, `--enable-dns-hostnames`, `--enable-dns-support`, `--enable-network-address-usage-metrics` | `--dry-run` | VPC must exist | Gateway validates VpcId non-empty → NATS `ec2.ModifyVpcAttribute` → daemon retrieves VPC record from `spinifex-vpc-vpcs` KV → updates boolean flags if provided → updates KV with optimistic locking → returns empty output | 1. Enable DNS hostnames<br>2. Enable DNS support<br>3. Enable network address usage metrics<br>4. Missing VPC ID (MissingParameter)<br>5. Non-existent VPC (InvalidVpcID.NotFound) | **DONE** |
| `describe-vpc-attribute` | `--vpc-id`, `--attribute` (enableDnsHostnames, enableDnsSupport, enableNetworkAddressUsageMetrics) | `--dry-run` | VPC must exist | Gateway validates VpcId + Attribute non-empty → NATS `ec2.DescribeVpcAttribute` → daemon retrieves VPC record → returns single attribute as AttributeBooleanValue (one attribute per call, matches AWS behavior) → validates attribute name | 1. Get enableDnsHostnames<br>2. Get enableDnsSupport<br>3. Get enableNetworkAddressUsageMetrics<br>4. Invalid attribute name (InvalidParameterValue)<br>5. Missing VPC ID (MissingParameter)<br>6. Non-existent VPC (InvalidVpcID.NotFound) | **DONE** |
| `associate-vpc-cidr-block` | — | `--vpc-id`, `--cidr-block` | VPC must exist | NATS `ec2.AssociateVpcCidrBlock` → daemon adds secondary CIDR to VPC metadata → return association | 1. Add secondary CIDR<br>2. Overlapping CIDR (error)<br>3. Max CIDR blocks exceeded (error) | **NOT STARTED** |
| `disassociate-vpc-cidr-block` | — | `--association-id` | Association must exist | NATS `ec2.DisassociateVpcCidrBlock` → daemon removes secondary CIDR from VPC → return success | 1. Remove secondary CIDR<br>2. Remove primary CIDR (error) | **NOT STARTED** |
| `create-subnet` | — | `--vpc-id`, `--cidr-block`, `--availability-zone`, `--tag-specifications` | VPC must exist | Validate CIDR within VPC range → create OVS port group with VLAN tag → assign subnet-ID → store metadata | 1. Create subnet within VPC CIDR<br>2. Subnet CIDR outside VPC range (error)<br>3. Overlapping subnet CIDRs (error) | **NOT STARTED** |
| `delete-subnet` | — | `--subnet-id` | Subnet must be empty (no instances) | Verify no instances in subnet → remove OVS port group → delete metadata | 1. Delete empty subnet<br>2. Delete subnet with instances (error) | **NOT STARTED** |
| `describe-subnets` | — | `--subnet-ids`, `--filters`, `--max-results` | None | Query NATS KV for subnet metadata → return subnet list | 1. List all subnets<br>2. Filter by VPC ID<br>3. Filter by AZ | **NOT STARTED** |
| `modify-subnet-attribute` | — | `--subnet-id`, `--map-public-ip-on-launch`, `--assign-ipv6-address-on-creation` | Subnet must exist | NATS `ec2.ModifySubnetAttribute` → daemon updates subnet attribute in KV → return success | 1. Enable auto-assign public IP<br>2. Missing subnet ID (error) | **NOT STARTED** |
| `create-subnet` | `--vpc-id`, `--cidr-block`, `--availability-zone`, `--tag-specifications` | `--dry-run` | VPC must exist | Gateway validates VpcId + CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS `ec2.CreateSubnet` → daemon verifies parent VPC exists → validates subnet CIDR is within VPC CIDR range → checks for overlapping subnets in same VPC → calculates available IPs (total hosts - 5 AWS-reserved: .0/.1/.2/.3/.255) → generates subnet-ID → stores SubnetRecord in `spinifex-vpc-subnets` KV → publishes `vpc.create-subnet` event to vpcd → defaults AZ from config if omitted | 1. Create subnet within VPC CIDR<br>2. Subnet CIDR outside VPC range (error: InvalidSubnetRange)<br>3. Overlapping subnet CIDRs (error: InvalidSubnetConflict)<br>4. CIDR prefix outside /16–/28 (error)<br>5. Missing VPC or CIDR (error: MissingParameter) | **DONE** |
| `delete-subnet` | `--subnet-id` | `--dry-run` | Subnet must exist | NATS `ec2.DeleteSubnet` → daemon verifies subnet exists → deletes from `spinifex-vpc-subnets` KV → publishes `vpc.delete-subnet` event to vpcd | 1. Delete existing subnet<br>2. Delete non-existent subnet (error) | **DONE** |
| `describe-subnets` | `--subnet-ids`, `--filters` (vpc-id, subnet-id, availability-zone, cidr-block, state, default-for-az, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeSubnets` → daemon lists all keys from `spinifex-vpc-subnets` KV → applies filters → filters by subnet IDs if specified → recalculates available IPs from CIDR → returns error for non-existent requested IDs | 1. List all subnets<br>2. Filter by VPC ID<br>3. Filter by subnet ID<br>4. Non-existent subnet returns error<br>5. Filter by availability-zone, cidr-block, state<br>6. Unknown filter returns InvalidParameterValue | **DONE** |
| `modify-subnet-attribute` | `--subnet-id`, `--map-public-ip-on-launch` | `--assign-ipv6-address-on-creation`, `--dry-run` | Subnet must exist | Gateway validates SubnetId non-empty → NATS `ec2.ModifySubnetAttribute` → daemon retrieves Subnet record from `spinifex-vpc-subnets` KV → updates MapPublicIpOnLaunch flag if provided → updates KV with optimistic locking → returns empty output | 1. Enable auto-assign public IP<br>2. Disable auto-assign public IP<br>3. Missing SubnetId (MissingParameter)<br>4. Non-existent subnet (InvalidSubnetID.NotFound) | **DONE** |
| `associate-subnet-cidr-block` | — | `--subnet-id`, `--ipv6-cidr-block` | Subnet must exist | NATS `ec2.AssociateSubnetCidrBlock` → daemon adds IPv6 CIDR to subnet → return association | 1. Add IPv6 CIDR<br>2. Missing subnet ID (error) | **NOT STARTED** |
| `disassociate-subnet-cidr-block` | — | `--association-id` | Association must exist | NATS `ec2.DisassociateSubnetCidrBlock` → daemon removes IPv6 CIDR from subnet → return success | 1. Remove IPv6 CIDR<br>2. Invalid association (error) | **NOT STARTED** |
| `create-security-group` | — | `--group-name`, `--description`, `--vpc-id`, `--tag-specifications` | VPC must exist | Create security group metadata → create default OVS flow rules (deny all inbound, allow all outbound) → assign sg-ID | 1. Create SG in VPC<br>2. Duplicate name in same VPC (error)<br>3. Verify default rules | **NOT STARTED** |
| `delete-security-group` | — | `--group-id` | SG must not be in use by instances | Verify no instances reference SG → remove OVS flow rules → delete metadata | 1. Delete unused SG<br>2. Delete SG in use (error) | **NOT STARTED** |
| `describe-security-groups` | — | `--group-ids`, `--group-names`, `--filters`, `--max-results` | None | Query NATS KV for SG metadata → return SG list with rules | 1. List all SGs<br>2. Filter by VPC ID<br>3. Filter by group name | **NOT STARTED** |
| `authorize-security-group-ingress` | — | `--group-id`, `--protocol`, `--port`, `--cidr`, `--source-group` | SG must exist | Add inbound rule → create OVS OpenFlow rule → persist to metadata | 1. Allow SSH (port 22) from 0.0.0.0/0<br>2. Allow from specific CIDR<br>3. Duplicate rule (idempotent) | **NOT STARTED** |
| `authorize-security-group-egress` | — | `--group-id`, `--protocol`, `--port`, `--cidr` | SG must exist | Add outbound rule → create OVS OpenFlow rule → persist to metadata | 1. Allow HTTPS outbound<br>2. Restrict to specific CIDR | **NOT STARTED** |
| `revoke-security-group-ingress` | — | `--group-id`, `--protocol`, `--port`, `--cidr` | SG must exist, rule must exist | Remove inbound rule → delete OVS OpenFlow rule → update metadata | 1. Revoke existing rule<br>2. Revoke non-existent rule (error) | **NOT STARTED** |
| `revoke-security-group-egress` | — | `--group-id`, `--protocol`, `--port`, `--cidr` | SG must exist, rule must exist | Remove outbound rule → delete OVS OpenFlow rule → update metadata | 1. Revoke existing rule<br>2. Revoke non-existent rule (error) | **NOT STARTED** |
| `create-internet-gateway` | — | `--tag-specifications` | None | Create IGW metadata → assign igw-ID → configure OVS NAT bridge for external access | 1. Create IGW<br>2. Verify in describe-internet-gateways | **NOT STARTED** |
| `attach-internet-gateway` | — | `--internet-gateway-id`, `--vpc-id` | IGW and VPC must exist | Link IGW to VPC → configure OVS flows for internet routing → update metadata | 1. Attach IGW to VPC<br>2. Attach already-attached IGW (error)<br>3. Attach to non-existent VPC (error) | **NOT STARTED** |
| `detach-internet-gateway` | — | `--internet-gateway-id`, `--vpc-id` | IGW must be attached to VPC | Unlink IGW from VPC → remove OVS internet routing flows | 1. Detach attached IGW<br>2. Detach unattached IGW (error) | **NOT STARTED** |
| `delete-internet-gateway` | — | `--internet-gateway-id` | IGW must be detached | Verify IGW detached → remove OVS NAT bridge → delete metadata | 1. Delete detached IGW<br>2. Delete attached IGW (error) | **NOT STARTED** |
| `describe-internet-gateways` | — | `--internet-gateway-ids`, `--filters`, `--max-results` | None | Query NATS KV for IGW metadata → return list with attachment info | 1. List all IGWs<br>2. Filter by attachment VPC | **NOT STARTED** |
| `create-security-group` | `--group-name`, `--description`, `--vpc-id`, `--tag-specifications` | `--dry-run` | VPC must exist | Gateway validates GroupName + VpcId required → NATS `ec2.CreateSecurityGroup` → daemon verifies VPC exists → checks duplicate name in same VPC → generates sg-ID → creates SecurityGroupRecord in `spinifex-vpc-security-groups` KV with default egress rule (allow all outbound 0.0.0.0/0), no ingress rules → publishes `vpc.create-sg` event to vpcd (OVN ACL configuration) → returns GroupId | 1. Create SG in VPC<br>2. Duplicate name in same VPC (error: InvalidGroup.Duplicate)<br>3. Verify default egress rule (allow all)<br>4. Verify no default ingress rules<br>5. Missing GroupName (MissingParameter)<br>6. Non-existent VPC (InvalidVpcID.NotFound)<br>7. With tag specifications | **DONE** |
| `delete-security-group` | `--group-id` | `--dry-run` | SG must exist | Gateway validates GroupId non-empty → NATS `ec2.DeleteSecurityGroup` → daemon retrieves SG from KV → deletes from `spinifex-vpc-security-groups` KV → publishes `vpc.delete-sg` event to vpcd → returns success | 1. Delete existing SG<br>2. Delete non-existent SG (error: InvalidGroup.NotFound)<br>3. Missing GroupId (MissingParameter) | **DONE** |
| `describe-security-groups` | `--group-ids`, `--filters` (vpc-id, group-name, group-id, description, ip-permission.cidr, tag:\*) | `--group-names`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeSecurityGroups` → daemon scans all SG records in account → applies filters (AND logic across filter names, OR across values) → returns SecurityGroup list with ingress/egress rules → returns error for non-existent requested GroupIds | 1. List all SGs<br>2. Filter by VPC ID<br>3. Filter by group name<br>4. Filter by GroupId<br>5. Non-existent GroupId (InvalidGroup.NotFound)<br>6. Empty result when no SGs<br>7. Filter by description, ip-permission.cidr<br>8. Unknown filter returns InvalidParameterValue | **DONE** |
| `authorize-security-group-ingress` | `--group-id`, `--ip-permissions` (IpProtocol, FromPort, ToPort, IpRanges/CidrIp, UserIdGroupPairs/GroupId) | `--dry-run` | SG must exist | Gateway validates GroupId non-empty → NATS `ec2.AuthorizeSecurityGroupIngress` → daemon retrieves SG → converts IpPermissions to SGRule objects → appends to IngressRules → updates KV with optimistic locking (revision check) → publishes `vpc.update-sg` event to vpcd → returns true | 1. Allow SSH (port 22) from 0.0.0.0/0<br>2. Allow from specific CIDR<br>3. Allow from source security group<br>4. Missing GroupId (MissingParameter)<br>5. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
| `authorize-security-group-egress` | `--group-id`, `--ip-permissions` | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.AuthorizeSecurityGroupEgress` → daemon retrieves SG → appends rules to EgressRules → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Allow HTTPS outbound<br>2. Restrict to specific CIDR<br>3. Missing GroupId (MissingParameter)<br>4. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
| `revoke-security-group-ingress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupIngress` → daemon retrieves SG → removes matching rules from IngressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
| `revoke-security-group-egress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupEgress` → daemon retrieves SG → removes matching rules from EgressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
| `create-internet-gateway` | `--tag-specifications` | `--dry-run` | None | NATS `ec2.CreateInternetGateway` → daemon generates igw-ID → creates IGWRecord in `spinifex-igw` KV bucket with state=available (not attached) → returns InternetGateway with empty Attachments | 1. Create IGW<br>2. Verify in describe-internet-gateways<br>3. Tags applied at creation | **DONE** |
| `attach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW and VPC must exist | NATS `ec2.AttachInternetGateway` → daemon validates IGW exists and is not already attached (ResourceAlreadyAssociated) → sets VpcId on record → state=attached → publishes `vpc.igw-attach` event to vpcd (creates OVN external switch, gateway port, SNAT rules) | 1. Attach IGW to VPC<br>2. Attach already-attached IGW (ResourceAlreadyAssociated)<br>3. Non-existent IGW (error)<br>4. Missing params (MissingParameter) | **DONE** |
| `detach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW must be attached to specified VPC | NATS `ec2.DetachInternetGateway` → daemon validates IGW exists and is attached to specified VPC (GatewayNotAttached if wrong VPC) → clears VpcId → state=available → publishes `vpc.igw-detach` event to vpcd (removes OVN resources) | 1. Detach attached IGW<br>2. Detach from wrong VPC (GatewayNotAttached)<br>3. Detach unattached IGW (error) | **DONE** |
| `delete-internet-gateway` | `--internet-gateway-id` | `--dry-run` | IGW must be detached | NATS `ec2.DeleteInternetGateway` → daemon validates IGW exists and VpcId is empty (DependencyViolation if attached) → deletes from `spinifex-igw` KV | 1. Delete detached IGW<br>2. Delete attached IGW (DependencyViolation)<br>3. Delete non-existent (error) | **DONE** |
| `describe-internet-gateways` | `--internet-gateway-ids`, `--filters` (internet-gateway-id, attachment.vpc-id, attachment.state, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeInternetGateways` → daemon lists all keys from `spinifex-igw` KV → filters by IGW IDs if specified → applies filters → returns list with Attachments array (if attached) | 1. List all IGWs<br>2. Filter by IGW ID<br>3. Verify attachment info<br>4. Filter by attachment.vpc-id, attachment.state<br>5. Unknown filter returns InvalidParameterValue | **DONE** |
## EC2 - Route Tables
### EC2 - Route Tables
Route tables are stored in `spinifex-vpc-route-tables` JetStream KV bucket, keyed by `{accountID}.{routeTableId}`. Each route table has a Routes array (with local route pre-populated) and an Associations array. All operations validate VPC consistency. When a route targeting a NAT gateway is created, `vpc.add-nat-gateway` events are published per associated subnet for vpcd SNAT rule creation.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-route-table` | — | `--vpc-id`, `--tag-specifications` | VPC must exist | NATS `ec2.CreateRouteTable` → daemon creates route table in KV with rtb-ID → add default local route for VPC CIDR → return route table | 1. Create route table<br>2. Verify default local route<br>3. Missing VPC ID (error) | **NOT STARTED** |
| `delete-route-table` | — | `--route-table-id`, `--dry-run` | Route table must exist and not be main route table | NATS `ec2.DeleteRouteTable` → daemon verifies not main table → delete from KV → return success | 1. Delete non-main route table<br>2. Delete main route table (error)<br>3. Delete non-existent (error) | **NOT STARTED** |
| `describe-route-tables` | — | `--route-table-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeRouteTables` → daemon lists route tables from KV → return tables with routes | 1. List all route tables<br>2. Filter by VPC | **NOT STARTED** |
| `create-route` | — | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` or `--nat-gateway-id` or `--egress-only-internet-gateway-id` | Route table and target must exist | NATS `ec2.CreateRoute` → daemon adds route entry → configure OVS flow for destination CIDR → update metadata | 1. Add internet route via IGW<br>2. Add route to NAT gateway<br>3. Add route to EIGW<br>4. Conflicting route (error) | **NOT STARTED** |
| `delete-route` | — | `--route-table-id`, `--destination-cidr-block` | Route must exist | NATS `ec2.DeleteRoute` → daemon removes route entry → remove OVS flow → update metadata | 1. Delete existing route<br>2. Delete non-existent route (error)<br>3. Delete local route (error) | **NOT STARTED** |
| `replace-route` | — | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` or `--nat-gateway-id` | Route must exist | NATS `ec2.ReplaceRoute` → daemon replaces route target → update OVS flow → update metadata | 1. Replace IGW route with NAT<br>2. Replace non-existent route (error) | **NOT STARTED** |
| `associate-route-table` | — | `--route-table-id`, `--subnet-id` or `--gateway-id` | Route table and target must exist | NATS `ec2.AssociateRouteTable` → daemon links route table to subnet/gateway → return rtbassoc-ID | 1. Associate with subnet<br>2. Missing route table ID (error) | **NOT STARTED** |
| `disassociate-route-table` | — | `--association-id` | Association must exist, not main table association | NATS `ec2.DisassociateRouteTable` → daemon unlinks route table → return success | 1. Disassociate existing<br>2. Disassociate main table (error) | **NOT STARTED** |
| `create-route-table` | `--vpc-id` | `--tag-specifications`, `--dry-run` | VPC must exist | Gateway validates VpcId (required) → NATS `ec2.CreateRouteTable` with `spinifex-workers` queue group → daemon verifies VPC exists in `spinifex-vpc-vpcs` KV → retrieves VPC CIDR → generates rtb-ID → creates RouteTableRecord with IsMain=false, local route pre-populated (destination=VPC CIDR, gateway="local"), empty associations → stores in `spinifex-vpc-route-tables` KV → returns RouteTable | 1. Create route table with local route (happy path)<br>2. VPC not found (InvalidVpcID.NotFound)<br>3. Missing VpcId (MissingParameter) | **DONE** |
| `delete-route-table` | `--route-table-id` | `--dry-run` | Route table must exist, not main, no subnet associations | Gateway validates RouteTableId (required) → NATS `ec2.DeleteRouteTable` → daemon retrieves record → rejects if IsMain=true (DependencyViolation) → rejects if has non-main subnet associations (DependencyViolation) → deletes from KV → returns success | 1. Delete custom route table<br>2. Delete main route table (DependencyViolation)<br>3. Delete table with subnet associations (DependencyViolation)<br>4. Delete non-existent (InvalidRouteTableID.NotFound) | **DONE** |
| `describe-route-tables` | `--route-table-ids`, `--filters` (vpc-id, route-table-id, association.main, association.route-table-association-id, association.subnet-id, route.destination-cidr-block, route.gateway-id) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeRouteTables` → daemon scans all keys in `spinifex-vpc-route-tables` KV → filters by RouteTableIds if specified → applies all provided filters (AND logic) → returns error for non-existent requested IDs → converts records to EC2 RouteTable objects | 1. List all route tables<br>2. Filter by vpc-id<br>3. Filter by association.main (true/false)<br>4. Non-existent ID (InvalidRouteTableID.NotFound)<br>5. Empty result when no route tables | **DONE** |
| `create-route` | `--route-table-id`, `--destination-cidr-block`, `--gateway-id`, `--nat-gateway-id` | `--egress-only-internet-gateway-id`, `--vpc-peering-connection-id`, `--dry-run` | Route table and target must exist in same VPC | Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS `ec2.CreateRoute` → daemon retrieves route table → checks for duplicate destination (RouteAlreadyExists) → validates exactly one of GatewayId/NatGatewayId provided → if GatewayId: verifies IGW exists in `spinifex-igw` KV and is attached to same VPC → if NatGatewayId: verifies NAT GW exists in `spinifex-vpc-nat-gateways` KV and is in same VPC → for NAT GW routes, publishes `vpc.add-nat-gateway` event per associated subnet → appends RouteRecord (state=active, origin=CreateRoute) → updates KV → returns Return=true | 1. Add IGW route (happy path)<br>2. Duplicate destination (RouteAlreadyExists)<br>3. IGW not found (InvalidInternetGatewayID.NotFound)<br>4. NAT GW not found (InvalidNatGatewayID.NotFound)<br>5. Gateway not in same VPC (InvalidParameterValue)<br>6. Missing RouteTableId or DestinationCidrBlock (MissingParameter) | **DONE** |
| `delete-route` | `--route-table-id`, `--destination-cidr-block` | `--dry-run` | Route must exist | Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS `ec2.DeleteRoute` → daemon retrieves route table → finds matching route → rejects if gateway is "local" (InvalidParameterValue — local routes cannot be deleted) → removes route from Routes array → updates KV → returns success | 1. Delete IGW route (happy path, local route remains)<br>2. Delete local route (InvalidParameterValue)<br>3. Route not found (InvalidRoute.NotFound)<br>4. Missing parameters (MissingParameter) | **DONE** |
| `replace-route` | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` | `--nat-gateway-id`, `--dry-run` | Route must exist, new target must exist in same VPC | Gateway validates RouteTableId + DestinationCidrBlock + GatewayId (required) → NATS `ec2.ReplaceRoute` → daemon retrieves route table → finds matching route → rejects if local route → verifies IGW exists and is attached to same VPC → updates route GatewayId, clears NatGatewayId, sets state=active → updates KV → returns success | 1. Replace route target (IGW swap)<br>2. Replace local route (InvalidParameterValue)<br>3. Route not found (InvalidRoute.NotFound)<br>4. IGW not found (InvalidInternetGatewayID.NotFound) | **DONE** |
| `associate-route-table` | `--route-table-id`, `--subnet-id` | `--gateway-id`, `--dry-run` | Route table and subnet must exist in same VPC | Gateway validates RouteTableId + SubnetId (required) → NATS `ec2.AssociateRouteTable` → daemon retrieves route table → verifies subnet exists and is in same VPC → scans ALL route tables in VPC to check subnet isn't already explicitly associated (Resource.AlreadyAssociated) → generates rtbassoc-ID → appends AssociationRecord (Main=false) → updates KV → returns AssociationId with state=associated | 1. Associate subnet (happy path)<br>2. Duplicate subnet association (Resource.AlreadyAssociated)<br>3. Subnet not found (InvalidSubnetID.NotFound)<br>4. Subnet in different VPC (InvalidParameterValue)<br>5. Missing parameters (MissingParameter) | **DONE** |
| `disassociate-route-table` | `--association-id` | `--dry-run` | Association must exist, not main table association | Gateway validates AssociationId (required) → NATS `ec2.DisassociateRouteTable` → daemon scans ALL route tables in account to find matching AssociationId → rejects if Main=true (InvalidParameterValue — main associations are immutable) → removes association from Associations array → updates KV → returns success | 1. Disassociate subnet (happy path)<br>2. Disassociate main table (InvalidParameterValue)<br>3. Association not found (InvalidAssociationID.NotFound) | **DONE** |
| `replace-route-table-association` | `--association-id`, `--route-table-id` | `--dry-run` | Association and new route table must exist | Gateway validates AssociationId + RouteTableId (required) → NATS `ec2.ReplaceRouteTableAssociation` → daemon verifies new route table exists → scans all route tables to find matching AssociationId → if Main=true: atomically swaps IsMain flag between old and new table → two-phase commit: removes association from old table, creates new AssociationId on new table → compensating write on failure (logs CRITICAL if compensation fails) → returns new AssociationId with state=associated | 1. Move subnet between tables (happy path)<br>2. Replace main table association (swaps IsMain flags)<br>3. Association not found (InvalidAssociationID.NotFound)<br>4. Route table not found (InvalidRouteTableID.NotFound) | **DONE** |
## EC2 - VPC Peering
### EC2 - VPC Peering
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -229,7 +338,7 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `describe-vpc-peering-connections` | — | `--vpc-peering-connection-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeVpcPeeringConnections` → daemon lists peerings from KV → return list with state | 1. List all peerings<br>2. Filter by ID<br>3. Filter by VPC | **NOT STARTED** |
| `modify-vpc-peering-connection-options` | — | `--vpc-peering-connection-id`, `--requester-peering-connection-options`, `--accepter-peering-connection-options` | Peering must be active | NATS `ec2.ModifyVpcPeeringConnectionOptions` → daemon updates peering options in KV → return modified options | 1. Enable DNS resolution<br>2. Missing peering ID (error) | **NOT STARTED** |
## EC2 - VPC Endpoints
### EC2 - VPC Endpoints
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -239,7 +348,7 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `describe-vpc-endpoint-services` | — | `--service-names`, `--filters`, `--max-results` | None | NATS `ec2.DescribeVpcEndpointServices` → daemon returns available services (S3, DynamoDB equivalents) → return service list | 1. List available services<br>2. Filter by service name | **NOT STARTED** |
| `modify-vpc-endpoint` | — | `--vpc-endpoint-id`, `--add-route-table-ids`, `--remove-route-table-ids`, `--add-subnet-ids`, `--remove-subnet-ids` | Endpoint must exist | NATS `ec2.ModifyVpcEndpoint` → daemon updates endpoint route tables/subnets in KV → return success | 1. Add route table<br>2. Remove subnet<br>3. Missing endpoint ID (error) | **NOT STARTED** |
## EC2 - VPN Gateway & Customer Gateway
### EC2 - VPN Gateway & Customer Gateway
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -252,7 +361,7 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `detach-vpn-gateway` | — | `--vpn-gateway-id`, `--vpc-id` | Must be attached | NATS `ec2.DetachVpnGateway` → daemon unlinks VPN gateway from VPC → return success | 1. Detach from VPC<br>2. Not attached (error) | **NOT STARTED** |
| `describe-vpn-gateways` | — | `--vpn-gateway-ids`, `--filters` | None | NATS `ec2.DescribeVpnGateways` → daemon lists from KV → return list with attachment info | 1. List all<br>2. Filter by VPC attachment | **NOT STARTED** |
## EC2 - VPN Connections
### EC2 - VPN Connections
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -262,7 +371,7 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `modify-vpn-connection` | — | `--vpn-connection-id`, `--vpn-gateway-id`, `--customer-gateway-id` | Connection must exist | NATS `ec2.ModifyVpnConnection` → daemon updates connection target in KV → return modified connection | 1. Change VPN gateway<br>2. Missing connection ID (error) | **NOT STARTED** |
| `modify-vpn-connection-options` | — | `--vpn-connection-id`, `--local-ipv4-network-cidr`, `--remote-ipv4-network-cidr` | Connection must exist | NATS `ec2.ModifyVpnConnectionOptions` → daemon updates tunnel options in KV → return modified connection | 1. Modify CIDR ranges<br>2. Missing connection ID (error) | **NOT STARTED** |
## EC2 - Network ACLs
### EC2 - Network ACLs
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -274,7 +383,7 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `replace-network-acl-association` | — | `--association-id`, `--network-acl-id` | Both must exist | NATS `ec2.ReplaceNetworkAclAssociation` → daemon swaps NACL for subnet → return new association ID | 1. Replace association<br>2. Missing association ID (error) | **NOT STARTED** |
| `replace-network-acl-entry` | — | `--network-acl-id`, `--rule-number`, `--protocol`, `--rule-action`, `--cidr-block`, `--ingress`/`--egress`, `--port-range` | Entry must exist | NATS `ec2.ReplaceNetworkAclEntry` → daemon replaces rule in KV → update OVS flow → return success | 1. Replace existing entry<br>2. Replace non-existent entry (error) | **NOT STARTED** |
## EC2 - Prefix Lists
### EC2 - Prefix Lists
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -285,20 +394,22 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `get-managed-prefix-list-entries` | — | `--prefix-list-id`, `--target-version`, `--max-results` | Prefix list must exist | NATS `ec2.GetManagedPrefixListEntries` → daemon reads entries from KV → return entry list | 1. List entries<br>2. List entries at specific version | **NOT STARTED** |
| `get-managed-prefix-list-associations` | — | `--prefix-list-id`, `--max-results` | Prefix list must exist | NATS `ec2.GetManagedPrefixListAssociations` → daemon finds references in SGs/route tables → return association list | 1. List associations<br>2. No associations returns empty | **NOT STARTED** |
## EC2 - Network Interfaces
### EC2 - Network Interfaces
ENIs are managed via `spinifex-vpc-enis` KV bucket with CAS-based IPAM allocation from `spinifex-vpc-ipam`. ENIs are auto-created by `run-instances --subnet-id` and auto-deleted on termination. MAC addresses are deterministically generated from the ENI ID. Attach/detach are internal operations used by instance lifecycle — no standalone attach/detach API yet.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-network-interface` | — | `--subnet-id`, `--description`, `--private-ip-address`, `--groups`, `--tag-specifications` | Subnet must exist | NATS `ec2.CreateNetworkInterface` → daemon creates ENI in KV with eni-ID → assign private IP from subnet pool → return ENI | 1. Create in subnet<br>2. Create with specific private IP<br>3. Missing subnet ID (error) | **NOT STARTED** |
| `delete-network-interface` | — | `--network-interface-id`, `--dry-run` | Must not be attached | NATS `ec2.DeleteNetworkInterface` → daemon verifies detached → release IP → delete from KV → return success | 1. Delete detached ENI<br>2. Delete attached ENI (error) | **NOT STARTED** |
| `describe-network-interfaces` | — | `--network-interface-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeNetworkInterfaces` → daemon lists ENIs from KV → return list with IP/attachment info | 1. List all ENIs<br>2. Filter by subnet<br>3. Filter by attachment | **NOT STARTED** |
| `attach-network-interface` | — | `--network-interface-id`, `--instance-id`, `--device-index` | ENI and instance must exist | NATS `ec2.AttachNetworkInterface` → daemon attaches OVS port to instance QEMU process → configure MAC/IP → return eni-attach-ID | 1. Attach ENI to instance<br>2. Already attached (error)<br>3. Missing device index (error) | **NOT STARTED** |
| `detach-network-interface` | — | `--attachment-id`, `--force` | ENI must be attached | NATS `ec2.DetachNetworkInterface` → daemon detaches OVS port from instance → return success | 1. Detach ENI<br>2. Force detach<br>3. Not attached (error) | **NOT STARTED** |
| `modify-network-interface-attribute` | — | `--network-interface-id`, `--description`, `--source-dest-check`, `--groups` | ENI must exist | NATS `ec2.ModifyNetworkInterfaceAttribute` → daemon updates ENI attributes in KV → return success | 1. Modify description<br>2. Toggle source/dest check<br>3. Update security groups | **NOT STARTED** |
| `create-network-interface` | `--subnet-id`, `--private-ip-address`, `--description`, `--tag-specifications` | `--groups`, `--dry-run` | Subnet must exist | Gateway validates SubnetId (required) → NATS `ec2.CreateNetworkInterface` → daemon verifies subnet exists in `spinifex-vpc-subnets` KV → if no PrivateIpAddress, allocates from IPAM pool (CAS-based, up to 5 retries) → generates eni-ID → generates deterministic MAC from ENI ID → stores ENIRecord in `spinifex-vpc-enis` KV → publishes `vpc.create-port` event to vpcd (creates OVN logical port) → returns NetworkInterface with status=available | 1. Create in subnet (auto-allocate IP)<br>2. Create with specific private IP<br>3. Missing subnet ID (MissingParameter)<br>4. Non-existent subnet (error)<br>5. Tags applied at creation | **DONE** |
| `delete-network-interface` | `--network-interface-id` | `--dry-run` | Must not be attached (status != in-use) | NATS `ec2.DeleteNetworkInterface` → daemon verifies ENI exists and status is not "in-use" (InvalidNetworkInterfaceInUse) → releases IP back to IPAM pool → deletes from `spinifex-vpc-enis` KV → publishes `vpc.delete-port` event to vpcd | 1. Delete detached ENI<br>2. Delete attached ENI (InvalidNetworkInterfaceInUse)<br>3. Delete non-existent (error) | **DONE** |
| `describe-network-interfaces` | `--network-interface-ids`, `--filters` (subnet-id, vpc-id, attachment.instance-id) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeNetworkInterfaces` → daemon lists all keys from `spinifex-vpc-enis` KV → applies filters (subnet-id, vpc-id, attachment.instance-id) → filters by ENI IDs if specified → returns error for non-existent requested IDs | 1. List all ENIs<br>2. Filter by subnet-id<br>3. Filter by vpc-id<br>4. Filter by attachment.instance-id<br>5. Non-existent ENI returns error | **DONE** |
| `attach-network-interface` | — | `--network-interface-id`, `--instance-id`, `--device-index` | ENI and instance must exist | Internal operation used by `run-instances --subnet-id`. Marks ENI as "in-use", sets instance attachment, generates attachment ID. No standalone API endpoint yet. | 1. Attach ENI to instance<br>2. Already attached (error)<br>3. Missing device index (error) | **NOT STARTED** (internal only) |
| `detach-network-interface` | — | `--attachment-id`, `--force` | ENI must be attached | Internal operation used by terminate-instances. Marks ENI as "available", clears attachment info. No standalone API endpoint yet. | 1. Detach ENI<br>2. Force detach<br>3. Not attached (error) | **NOT STARTED** (internal only) |
| `modify-network-interface-attribute` | — | `--network-interface-id`, `--description`, `--groups` | ENI must exist; at least one attribute required | NATS `ec2.ModifyNetworkInterfaceAttribute` → daemon validates ENI exists → updates description and/or security groups in `spinifex-vpc-enis` KV (optimistic locking) → return success | 1. Modify description<br>2. Update security groups<br>3. No attributes (InvalidParameterValue)<br>4. Non-existent ENI (error) | **DONE** |
| `assign-private-ip-addresses` | — | `--network-interface-id`, `--private-ip-addresses` or `--secondary-private-ip-address-count` | ENI must exist | NATS `ec2.AssignPrivateIpAddresses` → daemon allocates IPs from subnet pool → assign to ENI → return assigned IPs | 1. Assign specific IPs<br>2. Assign by count<br>3. Cannot specify both (error) | **NOT STARTED** |
| `unassign-private-ip-addresses` | — | `--network-interface-id`, `--private-ip-addresses` | IPs must be assigned to ENI | NATS `ec2.UnassignPrivateIpAddresses` → daemon releases IPs → return success | 1. Unassign secondary IP<br>2. Unassign primary IP (error) | **NOT STARTED** |
## EC2 - Launch Templates
### EC2 - Launch Templates
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -308,7 +419,7 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `describe-launch-templates` | — | `--launch-template-ids`, `--launch-template-names`, `--filters` | None | NATS `ec2.DescribeLaunchTemplates` → daemon lists templates from KV → return list | 1. List all templates<br>2. Filter by name<br>3. Filter by ID | **NOT STARTED** |
| `describe-launch-template-versions` | — | `--launch-template-id` or `--launch-template-name`, `--versions`, `--min-version`, `--max-version` | Template must exist | NATS `ec2.DescribeLaunchTemplateVersions` → daemon lists versions from KV → return version list with data | 1. List all versions<br>2. List specific version<br>3. Template not found (error) | **NOT STARTED** |
## EC2 - Misc Operations
### EC2 - Misc Operations
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -316,7 +427,7 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `enable-address-transfer` | — | `--allocation-id`, `--transfer-account-id` | EIP must exist | NATS `ec2.EnableAddressTransfer` → daemon stores transfer in JetStream KV with 7-day expiration → return AllocationId and TransferAccountId | 1. Enable transfer<br>2. Verify transfer stored with expiration | **NOT STARTED** |
| `disable-address-transfer` | — | `--allocation-id` | Transfer must exist | NATS `ec2.DisableAddressTransfer` → daemon updates transfer status to disabled in KV → return status | 1. Disable active transfer<br>2. Disable non-existent transfer (error) | **NOT STARTED** |
## EBS Direct API
### EBS Direct API
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
@@ -327,32 +438,201 @@ Requires Open vSwitch (`apt install openvswitch-switch openvswitch-common`). Sin
| `list-snapshot-blocks` | — | `--snapshot-id`, `--max-results`, `--next-token` | Snapshot must exist | List all non-empty blocks in snapshot with tokens and sizes | 1. List blocks of populated snapshot<br>2. List blocks of empty snapshot | **NOT STARTED** |
| `list-changed-blocks` | — | `--second-snapshot-id`, `--first-snapshot-id`, `--max-results` | Snapshots must exist | Compare two snapshots → return list of differing block indexes | 1. Compare parent and child snapshots<br>2. Compare unrelated snapshots | **NOT STARTED** |
## S3 (via Predastore)
### IAM - User Management
Currently S3 operations are handled directly by Predastore. Consider moving control/data plane to Hive format for AWS gateway integration.
All IAM operations are account-scoped. Users, policies, and access keys are namespaced per account. The gateway extracts the account ID from the authenticated access key via `c.Locals("sigv4.accountId")` and passes it to all IAM service methods. Root user (account `000000000000`) bypasses policy evaluation entirely.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `s3 mb` (CreateBucket) | — | `s3://bucket-name` | None | NATS `s3.createbucket` → Predastore creates bucket → return success | 1. Create valid bucket<br>2. Duplicate bucket name (error)<br>3. Invalid bucket name (error) | **NOT STARTED** (Predastore handles directly) |
| `s3 rb` (DeleteBucket) | — | `s3://bucket-name`, `--force` | Bucket must exist | NATS `s3.deletebucket` → verify bucket empty (or force) → Predastore deletes bucket | 1. Delete empty bucket<br>2. Delete non-empty bucket (error)<br>3. Force delete non-empty bucket | **NOT STARTED** |
| `s3 ls` (ListBuckets/Objects) | — | `s3://bucket/prefix`, `--recursive` | None | NATS `s3.listobjects` → Predastore lists objects with prefix → return list | 1. List all buckets<br>2. List objects in bucket<br>3. List with prefix filter | **NOT STARTED** |
| `s3 cp` (PutObject/GetObject) | — | `source dest`, `--recursive`, `--acl` | Bucket must exist | NATS `s3.putobject`/`s3.getobject` → Predastore stores/retrieves with Reed-Solomon encoding | 1. Upload single file<br>2. Download single file<br>3. Recursive upload/download<br>4. Large file (multipart) | **NOT STARTED** |
| `s3 rm` (DeleteObject) | — | `s3://bucket/key`, `--recursive` | Object must exist | NATS `s3.deleteobject` → Predastore deletes object shards | 1. Delete single object<br>2. Recursive delete<br>3. Delete non-existent (idempotent) | **NOT STARTED** |
| `s3 sync` | — | `source dest`, `--delete`, `--exclude`, `--include` | Bucket must exist | Compare source and dest → upload changed/new files → optionally delete removed files | 1. Sync local dir to S3<br>2. Sync with --delete<br>3. Sync with exclude filter | **NOT STARTED** |
| `create-user` | `--user-name`, `--path` | `--tags`, `--permissions-boundary` | None | Gateway validates UserName non-empty → IAMService.CreateUser(accountID, input) → generates AIDA-prefixed UserId → stores in KV at key `{accountID}.{userName}` → ARN includes real account ID → conditional create for race safety | 1. Create user<br>2. Duplicate name in same account (EntityAlreadyExists)<br>3. Same name in different account (succeeds)<br>4. Missing username (InvalidInput)<br>5. Default path is `/` | **DONE** |
| `get-user` | `--user-name` | — | User must exist in caller's account | Gateway validates UserName → IAMService.GetUser(accountID, input) → reads from KV at `{accountID}.{userName}` → returns User with ARN, Path, CreateDate | 1. Get existing user<br>2. Get non-existent user (NoSuchEntity)<br>3. Cannot see user in different account (NoSuchEntity) | **DONE** |
| `list-users` | `--path-prefix` | `--max-items`, `--marker` | None | IAMService.ListUsers(accountID, input) → prefix-scans KV for `{accountID}.*` → filters by path prefix if specified → returns user list scoped to caller's account | 1. List all users in account<br>2. List with path prefix filter<br>3. Empty list<br>4. Does not show other accounts' users | **DONE** |
| `delete-user` | `--user-name` | — | User must exist, no access keys attached | Gateway validates UserName → IAMService.DeleteUser(accountID, input) → checks user has no access keys → deletes from KV | 1. Delete user with no keys<br>2. Delete user with keys attached (DeleteConflict)<br>3. Delete non-existent user (NoSuchEntity) | **DONE** |
## IAM
### IAM - Access Key Management
Access keys are globally unique (keyed by AccessKeyID) but associated with an account. Max 2 keys per user. Secrets encrypted with AES-256-GCM using master key.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-user` | — | `--user-name`, `--path`, `--tags` | None | Store user in JetStream KV → generate AIDA-prefixed UserId → return User with ARN | 1. Create user<br>2. Duplicate name (EntityAlreadyExists)<br>3. Missing username (error) | **NOT STARTED** |
| `get-user` | — | `--user-name` | User must exist | Read user from JetStream KV → return User details | 1. Get existing user<br>2. Get non-existent user (NoSuchEntity) | **NOT STARTED** |
| `list-users` | — | `--path-prefix`, `--max-items`, `--marker` | None | List users from JetStream KV → return user list | 1. List all users<br>2. List with path prefix<br>3. Empty list | **NOT STARTED** |
| `delete-user` | — | `--user-name` | User must exist, no access keys/policies | Remove user from JetStream KV → return success | 1. Delete user with no keys<br>2. Delete user with keys (error)<br>3. Delete non-existent (NoSuchEntity) | **NOT STARTED** |
| `create-access-key` | — | `--user-name` | User must exist | Generate access key ID (AKIA prefix) and secret → store in JetStream KV → return credentials | 1. Create key for user<br>2. Max keys exceeded (error) | **NOT STARTED** (gateway stub exists) |
| `delete-access-key` | — | `--access-key-id`, `--user-name` | Key must exist | Remove access key from JetStream KV → return success | 1. Delete existing key<br>2. Delete non-existent key (error) | **NOT STARTED** |
| `list-access-keys` | — | `--user-name`, `--max-items` | None | Query JetStream KV for user's access keys → return list | 1. List keys for user<br>2. No keys returns empty | **NOT STARTED** |
| `create-policy` | — | `--policy-name`, `--policy-document`, `--path`, `--description`, `--tags` | None | Validate JSON policy document → store in JetStream KV → generate policy ARN → return Policy | 1. Create with valid document<br>2. Invalid JSON (error)<br>3. Duplicate name (EntityAlreadyExists) | **NOT STARTED** |
| `delete-policy` | — | `--policy-arn` | Policy must exist, not attached to any user | Remove policy from JetStream KV → return success | 1. Delete unattached policy<br>2. Delete attached policy (error) | **NOT STARTED** |
| `attach-user-policy` | — | `--user-name`, `--policy-arn` | User and policy must exist | Link policy to user in JetStream KV → return success | 1. Attach policy<br>2. Already attached (idempotent)<br>3. Non-existent user (NoSuchEntity) | **NOT STARTED** |
| `detach-user-policy` | — | `--user-name`, `--policy-arn` | User and policy must exist, policy attached | Unlink policy from user in JetStream KV → return success | 1. Detach attached policy<br>2. Not attached (error) | **NOT STARTED** |
| `list-attached-user-policies` | — | `--user-name`, `--path-prefix`, `--max-items` | User must exist | List policies attached to user from JetStream KV → return policy list | 1. List attached policies<br>2. No policies returns empty | **NOT STARTED** |
| `create-access-key` | `--user-name` | — | User must exist | IAMService.CreateAccessKey(accountID, input) → generates AKIA-prefixed key ID + random secret → encrypts secret with master key → stores in KV (globally unique key ID) → updates user's AccessKeys list → returns plaintext secret (one-time only) | 1. Create key for user (AKIA prefix)<br>2. Max 2 keys per user (LimitExceeded on 3rd)<br>3. Non-existent user (NoSuchEntity)<br>4. Secret only returned on creation | **DONE** |
| `list-access-keys` | `--user-name` | `--max-items`, `--marker` | None | IAMService.ListAccessKeys(accountID, input) → reads user's AccessKeys list → fetches each key's metadata → returns key IDs and status (no secrets) | 1. List keys for user<br>2. No keys returns empty list<br>3. Shows Status (Active/Inactive) but not secrets | **DONE** |
| `delete-access-key` | `--access-key-id`, `--user-name` | — | Key must exist | IAMService.DeleteAccessKey(accountID, input) → removes key from global KV → removes key ID from user's AccessKeys list | 1. Delete existing key<br>2. Delete non-existent key (NoSuchEntity) | **DONE** |
| `update-access-key` | `--access-key-id`, `--user-name`, `--status` (Active/Inactive) | — | Key must exist | IAMService.UpdateAccessKey(input) → reads key from global KV (key ID is globally unique, no accountID needed) → updates Status field → inactive keys rejected at SigV4 auth | 1. Deactivate key (Inactive)<br>2. Reactivate key (Active)<br>3. Non-existent key (NoSuchEntity)<br>4. Inactive key → auth returns InvalidClientTokenId | **DONE** |
### IAM - Policy Management
Policies are account-scoped. KV key is `{accountID}.{policyName}`. Policy documents require `Version: "2012-10-17"` and valid Statement array. Supports wildcard action matching (`ec2:*`, `ec2:Describe*`, `*`). Evaluation order: explicit Deny > explicit Allow > implicit Deny.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-policy` | `--policy-name`, `--policy-document`, `--path`, `--description` | `--tags` | None | Gateway validates PolicyName and PolicyDocument non-empty → IAMService.CreatePolicy(accountID, input) → validates JSON (Version, Statement with Effect/Action/Resource) → generates ANPA-prefixed PolicyId → stores at `{accountID}.{policyName}` → ARN includes real account ID | 1. Create with valid document<br>2. Malformed JSON (MalformedPolicyDocument)<br>3. Missing Effect/Action/Resource (MalformedPolicyDocument)<br>4. Duplicate name in same account (EntityAlreadyExists)<br>5. Same name in different account (succeeds)<br>6. With path and description | **DONE** |
| `get-policy` | `--policy-arn` | — | Policy must exist | Gateway validates PolicyArn → extracts account ID and policy name from ARN → IAMService.GetPolicy(accountID, input) → returns policy metadata (no document) with AttachmentCount | 1. Get existing policy<br>2. Non-existent policy (NoSuchEntity)<br>3. Returns metadata only, not document | **DONE** |
| `get-policy-version` | `--policy-arn`, `--version-id` | — | Policy must exist | Gateway validates PolicyArn and VersionId → IAMService.GetPolicyVersion(accountID, input) → returns PolicyVersion with URL-encoded Document, VersionId, IsDefaultVersion | 1. Get version v1 (returns document)<br>2. Non-existent version (NoSuchEntity)<br>3. Non-existent policy (NoSuchEntity) | **DONE** |
| `list-policies` | — | `--scope`, `--only-attached`, `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListPolicies(accountID, input) → prefix-scans KV for `{accountID}.*` → returns all policies in caller's account | 1. List all policies in account<br>2. Empty list<br>3. Does not show other accounts' policies | **DONE** |
| `delete-policy` | `--policy-arn` | — | Policy must exist, not attached to any user | Gateway validates PolicyArn → IAMService.DeletePolicy(accountID, input) → checks no users have this policy attached (scans account's users) → deletes from KV | 1. Delete unattached policy<br>2. Delete attached policy (DeleteConflict)<br>3. Delete non-existent (NoSuchEntity) | **DONE** |
### IAM - Policy Attachment
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `attach-user-policy` | `--user-name`, `--policy-arn` | — | User and policy must exist in caller's account | Gateway validates both params → IAMService.AttachUserPolicy(accountID, input) → verifies policy exists → adds policy ARN to user's AttachedPolicies list → idempotent (re-attach succeeds without duplicate) | 1. Attach policy to user<br>2. Already attached (idempotent, no error)<br>3. Non-existent user (NoSuchEntity)<br>4. Non-existent policy (NoSuchEntity) | **DONE** |
| `detach-user-policy` | `--user-name`, `--policy-arn` | — | Policy must be attached to user | Gateway validates both params → IAMService.DetachUserPolicy(accountID, input) → removes policy ARN from user's AttachedPolicies list | 1. Detach attached policy<br>2. Policy not attached (NoSuchEntity)<br>3. Non-existent user (NoSuchEntity) | **DONE** |
| `list-attached-user-policies` | `--user-name` | `--path-prefix`, `--max-items`, `--marker` | User must exist | IAMService.ListAttachedUserPolicies(accountID, input) → reads user's AttachedPolicies list → returns policy ARNs and names | 1. List attached policies<br>2. No policies returns empty list | **DONE** |
### IAM - Policy Evaluation
The gateway evaluates IAM policies on every request (except root user bypass). Policy evaluation is internal — not an AWS API.
| Feature | Description | Status |
|---------|-------------|--------|
| Root user bypass | Root user (`000000000000:root`) skips all policy checks | **DONE** |
| Default deny | Users with no attached policies are denied all actions | **DONE** |
| Explicit allow | Matching Allow statement grants access | **DONE** |
| Explicit deny | Deny statements override Allow (deny wins) | **DONE** |
| Wildcard matching | `*` matches all, `ec2:*` matches all EC2, `ec2:Describe*` matches prefix | **DONE** |
| Account-scoped evaluation | Policies resolved from caller's account context | **DONE** |
| EC2 action mapping | All registered EC2 actions mapped to IAM policy strings | **DONE** |
| IAM action mapping | 16 IAM actions mapped to IAM policy strings | **DONE** |
### IAM - Roles
IAM roles are required for instance profiles, cross-account access, and service-linked permissions. Without roles, workloads must use static access keys. Roles should be stored in a `spinifex-iam-roles` KV bucket keyed by `{accountID}.{roleName}`. Trust policies (AssumeRolePolicyDocument) define who can assume the role.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-role` | — | `--role-name`, `--assume-role-policy-document`, `--path`, `--description`, `--max-session-duration`, `--permissions-boundary`, `--tags` | None | IAMService.CreateRole(accountID, input) → validate RoleName + AssumeRolePolicyDocument (required) → validate trust policy JSON → generate AROA-prefixed RoleId → store RoleRecord in `spinifex-iam-roles` KV at `{accountID}.{roleName}` → ARN: `arn:aws:iam::{accountID}:role/{path}{roleName}` → return Role | 1. Create with valid trust policy<br>2. Malformed trust policy (MalformedPolicyDocument)<br>3. Duplicate name (EntityAlreadyExists)<br>4. Missing RoleName (InvalidInput)<br>5. With path and description | **NOT STARTED** |
| `get-role` | — | `--role-name` | Role must exist | IAMService.GetRole(accountID, input) → reads from KV at `{accountID}.{roleName}` → returns Role with ARN, Path, AssumeRolePolicyDocument, CreateDate | 1. Get existing role<br>2. Non-existent role (NoSuchEntity) | **NOT STARTED** |
| `list-roles` | — | `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListRoles(accountID, input) → prefix-scans KV for `{accountID}.*` → filters by path prefix → returns role list scoped to caller's account | 1. List all roles<br>2. List with path prefix<br>3. Empty list<br>4. Account isolation | **NOT STARTED** |
| `delete-role` | — | `--role-name` | Role must exist, no attached policies, no instance profiles | IAMService.DeleteRole(accountID, input) → verify no attached policies (DeleteConflict) → verify not in any instance profile (DeleteConflict) → delete from KV | 1. Delete role with no attachments<br>2. Delete role with policies (DeleteConflict)<br>3. Delete role in instance profile (DeleteConflict)<br>4. Non-existent role (NoSuchEntity) | **NOT STARTED** |
| `update-role` | — | `--role-name`, `--description`, `--max-session-duration` | Role must exist | IAMService.UpdateRole(accountID, input) → update description and/or session duration in KV → return success | 1. Update description<br>2. Update max session duration<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
| `update-assume-role-policy` | — | `--role-name`, `--policy-document` | Role must exist | IAMService.UpdateAssumeRolePolicy(accountID, input) → validate trust policy JSON → update AssumeRolePolicyDocument in KV → return success | 1. Update trust policy<br>2. Malformed policy (MalformedPolicyDocument)<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
| `attach-role-policy` | — | `--role-name`, `--policy-arn` | Role and policy must exist | IAMService.AttachRolePolicy(accountID, input) → verify policy exists → add ARN to role's AttachedPolicies → idempotent | 1. Attach policy<br>2. Already attached (idempotent)<br>3. Non-existent role (NoSuchEntity)<br>4. Non-existent policy (NoSuchEntity) | **NOT STARTED** |
| `detach-role-policy` | — | `--role-name`, `--policy-arn` | Policy must be attached | IAMService.DetachRolePolicy(accountID, input) → remove ARN from role's AttachedPolicies | 1. Detach attached policy<br>2. Not attached (NoSuchEntity)<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
| `list-attached-role-policies` | — | `--role-name`, `--path-prefix`, `--max-items`, `--marker` | Role must exist | IAMService.ListAttachedRolePolicies(accountID, input) → reads role's AttachedPolicies list → returns ARNs and names | 1. List attached policies<br>2. No policies (empty list)<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
### IAM - Instance Profiles
Instance profiles are containers for IAM roles that allow EC2 instances to assume a role and obtain temporary credentials. Required for `run-instances --iam-instance-profile`. Stored in `spinifex-iam-instance-profiles` KV bucket keyed by `{accountID}.{profileName}`.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-instance-profile` | — | `--instance-profile-name`, `--path`, `--tags` | None | IAMService.CreateInstanceProfile(accountID, input) → validate name (required) → generate AIPA-prefixed InstanceProfileId → store in `spinifex-iam-instance-profiles` KV with empty Roles array → ARN: `arn:aws:iam::{accountID}:instance-profile/{path}{name}` → return InstanceProfile | 1. Create profile<br>2. Duplicate name (EntityAlreadyExists)<br>3. Missing name (InvalidInput) | **NOT STARTED** |
| `get-instance-profile` | — | `--instance-profile-name` | Profile must exist | IAMService.GetInstanceProfile(accountID, input) → reads from KV → returns InstanceProfile with Roles array | 1. Get existing profile<br>2. Non-existent (NoSuchEntity) | **NOT STARTED** |
| `list-instance-profiles` | — | `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListInstanceProfiles(accountID, input) → prefix-scans KV → returns list | 1. List all profiles<br>2. Empty list | **NOT STARTED** |
| `list-instance-profiles-for-role` | — | `--role-name`, `--max-items`, `--marker` | Role must exist | IAMService.ListInstanceProfilesForRole(accountID, input) → scan all profiles → filter where Roles contains role name → return list | 1. List profiles for role<br>2. Role not in any profile (empty) | **NOT STARTED** |
| `delete-instance-profile` | — | `--instance-profile-name` | Profile must exist, Roles must be empty | IAMService.DeleteInstanceProfile(accountID, input) → verify Roles empty (DeleteConflict if not) → delete from KV | 1. Delete empty profile<br>2. Delete profile with role (DeleteConflict)<br>3. Non-existent (NoSuchEntity) | **NOT STARTED** |
| `add-role-to-instance-profile` | — | `--instance-profile-name`, `--role-name` | Both must exist, max 1 role per profile | IAMService.AddRoleToInstanceProfile(accountID, input) → verify profile exists → verify role exists → check Roles array is empty (LimitExceeded if role already present) → append role → update KV | 1. Add role to empty profile<br>2. Profile already has role (LimitExceeded)<br>3. Non-existent profile (NoSuchEntity)<br>4. Non-existent role (NoSuchEntity) | **NOT STARTED** |
| `remove-role-from-instance-profile` | — | `--instance-profile-name`, `--role-name` | Role must be in profile | IAMService.RemoveRoleFromInstanceProfile(accountID, input) → remove role from Roles array → update KV | 1. Remove role<br>2. Role not in profile (NoSuchEntity) | **NOT STARTED** |
### IAM - Groups
Groups provide a way to attach policies to multiple users. Stored in `spinifex-iam-groups` KV bucket keyed by `{accountID}.{groupName}`.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-group` | — | `--group-name`, `--path` | None | IAMService.CreateGroup(accountID, input) → validate GroupName (required) → generate AGPA-prefixed GroupId → store in KV → return Group | 1. Create group<br>2. Duplicate name (EntityAlreadyExists)<br>3. Missing name (InvalidInput) | **NOT STARTED** |
| `get-group` | — | `--group-name` | Group must exist | IAMService.GetGroup(accountID, input) → reads group from KV → returns Group with member Users list | 1. Get existing group<br>2. Non-existent (NoSuchEntity) | **NOT STARTED** |
| `list-groups` | — | `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListGroups(accountID, input) → prefix-scans KV → returns list | 1. List all groups<br>2. Empty list | **NOT STARTED** |
| `delete-group` | — | `--group-name` | Group must exist, no members, no attached policies | IAMService.DeleteGroup(accountID, input) → verify no members → verify no attached policies → delete from KV | 1. Delete empty group<br>2. Delete group with members (DeleteConflict)<br>3. Delete group with policies (DeleteConflict)<br>4. Non-existent (NoSuchEntity) | **NOT STARTED** |
| `add-user-to-group` | — | `--group-name`, `--user-name` | Both must exist | IAMService.AddUserToGroup(accountID, input) → verify group + user exist → add user to group's Members list → idempotent | 1. Add user<br>2. Already a member (idempotent)<br>3. Non-existent group (NoSuchEntity)<br>4. Non-existent user (NoSuchEntity) | **NOT STARTED** |
| `remove-user-from-group` | — | `--group-name`, `--user-name` | User must be in group | IAMService.RemoveUserFromGroup(accountID, input) → remove user from Members list → update KV | 1. Remove member<br>2. Not a member (NoSuchEntity) | **NOT STARTED** |
| `list-groups-for-user` | — | `--user-name`, `--max-items`, `--marker` | User must exist | IAMService.ListGroupsForUser(accountID, input) → scan all groups → filter where Members contains user → return list | 1. List groups for user<br>2. User in no groups (empty) | **NOT STARTED** |
| `attach-group-policy` | — | `--group-name`, `--policy-arn` | Group and policy must exist | IAMService.AttachGroupPolicy(accountID, input) → verify policy exists → add ARN to group's AttachedPolicies → idempotent | 1. Attach policy<br>2. Already attached (idempotent)<br>3. Non-existent group (NoSuchEntity) | **NOT STARTED** |
| `detach-group-policy` | — | `--group-name`, `--policy-arn` | Policy must be attached | IAMService.DetachGroupPolicy(accountID, input) → remove ARN from AttachedPolicies | 1. Detach policy<br>2. Not attached (NoSuchEntity) | **NOT STARTED** |
| `list-attached-group-policies` | — | `--group-name`, `--path-prefix`, `--max-items`, `--marker` | Group must exist | IAMService.ListAttachedGroupPolicies(accountID, input) → reads group's AttachedPolicies → returns ARNs and names | 1. List attached policies<br>2. No policies (empty) | **NOT STARTED** |
### STS (Security Token Service)
STS provides temporary credentials and identity verification. `GetCallerIdentity` is called by every AWS SDK/CLI invocation. `AssumeRole` is required once IAM roles are implemented. STS needs its own gateway service endpoint (`sts` service in the `supportedServices` map) with a dedicated action map.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `get-caller-identity` | — | *(no flags)* | Valid SigV4 credentials | Gateway extracts accountID and username from authenticated request → returns Account, Arn (`arn:aws:iam::{accountID}:user/{userName}` or `arn:aws:sts::{accountID}:assumed-role/{roleName}/{sessionName}`), UserId. Note: a Spinifex-custom version exists at `spinifex.GetCallerIdentity` but needs a proper STS endpoint. | 1. Get identity for IAM user<br>2. Get identity for assumed role<br>3. Get identity for root user<br>4. Invalid credentials (AuthFailure) | **NOT STARTED** |
| `assume-role` | — | `--role-arn`, `--role-session-name`, `--duration-seconds`, `--policy`, `--external-id`, `--serial-number`, `--token-code`, `--tags`, `--transitive-tag-keys` | Role must exist, caller must be allowed by trust policy | Gateway validates RoleArn + RoleSessionName (required) → looks up role → evaluates trust policy (AssumeRolePolicyDocument) against caller → generates temporary credentials (AccessKeyId with ASIA prefix, SecretAccessKey, SessionToken) → credentials expire after DurationSeconds (default 3600, max 43200) → return Credentials + AssumedRoleUser (ARN, AssumedRoleId) | 1. Assume role with valid trust policy<br>2. Trust policy denies caller (AccessDenied)<br>3. Non-existent role (NoSuchEntity)<br>4. Expired session token rejected at SigV4<br>5. Custom duration<br>6. Session tags | **NOT STARTED** |
| `get-session-token` | — | `--duration-seconds`, `--serial-number`, `--token-code` | Valid IAM user credentials (not role) | Gateway validates caller is IAM user (not assumed role) → generates temporary credentials (ASIA prefix) → session inherits user's permissions → return Credentials | 1. Get token for IAM user<br>2. Duration within range<br>3. Reject if caller is assumed role | **NOT STARTED** |
### ELBv2 (Elastic Load Balancing v2 — Application & Network Load Balancer)
All ELBv2 resources are stored in the `spinifex-elbv2` JetStream KV bucket. Key format: `lb.{lbId}`, `tg.{tgId}`, `listener.{listenerId}`. The data plane uses a system-managed LB VM running HAProxy, launched automatically during CreateLoadBalancer. HAProxy configuration is pushed via NATS (`elbv2.lb.{lbId}.config` topic) whenever listeners or targets change. Agent readiness is polled via `elbv2.lb.{lbId}.ping`.
#### ELBv2 - Load Balancers
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-load-balancer` | `--name`, `--subnets`, `--security-groups`, `--scheme` (internet-facing/internal), `--tags`, `--ip-address-type` (ipv4 only) | `--type` (hardcoded application), `--customer-owned-ipv4-pool`, `--dry-run` | None (subnets optional) | Gateway validates Name (required) → NATS `elbv2.CreateLoadBalancer` with `spinifex-workers` queue group → daemon validates scheme (internet-facing or internal, InvalidScheme if other) → checks duplicate name in account (DuplicateLoadBalancerName) → generates LB ID → builds DNS name `{name}-{lbId}.{region}.elb.spinifex.local` → if subnets provided, creates one ENI per subnet with `spinifex:managed-by=elbv2` tag (rolls back on failure) → if system AMI available, launches sys.micro LB VM with cloud-init running `/usr/local/bin/lb-agent` → state=provisioning → background goroutine polls agent ping for ~5min → on agent ready: state=active, on timeout: state=failed → if no VM launcher: state=active immediately → persists to KV → returns LoadBalancer with ARN, DNSName, State | 1. Create with name (happy path)<br>2. Internal scheme<br>3. Invalid scheme (InvalidScheme)<br>4. Duplicate name (DuplicateLoadBalancerName)<br>5. Missing name (MissingParameter)<br>6. With tags<br>7. With subnets (ENI creation) | **DONE** |
| `delete-load-balancer` | `--load-balancer-arn` | `--dry-run` | LB must exist | Gateway validates LoadBalancerArn (required) → NATS `elbv2.DeleteLoadBalancer` → daemon verifies LB exists and belongs to account → terminates LB VM (async background) → deletes all listeners for this LB → deletes system-managed ENIs → deletes LB record from KV → returns success | 1. Delete existing LB<br>2. Not found (LoadBalancerNotFound)<br>3. Cascades listener cleanup<br>4. ENI cleanup | **DONE** |
| `describe-load-balancers` | `--load-balancer-arns`, `--names` | `--page-size`, `--marker`, `--dry-run` | None | NATS `elbv2.DescribeLoadBalancers` → daemon lists all LB records from KV → filters by account ID → applies ARN filter and/or Name filter → returns LoadBalancer list with State, DNSName, Scheme, VpcId, AvailabilityZones | 1. List all LBs<br>2. Filter by name<br>3. Account isolation | **DONE** |
#### ELBv2 - Target Groups
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-target-group` | `--name`, `--protocol` (HTTP), `--port` (default 80), `--vpc-id`, `--target-type` (instance only), `--health-check-protocol`, `--health-check-port`, `--health-check-path` (default /), `--health-check-interval-seconds` (default 30), `--health-check-timeout-seconds` (default 5), `--healthy-threshold-count` (default 5), `--unhealthy-threshold-count` (default 2), `--matcher` (HttpCode, default 200), `--tags` | `--health-check-enabled`, `--protocol-version`, `--ip-address-type`, `--dry-run` | None | Gateway validates Name (required) → NATS `elbv2.CreateTargetGroup` → daemon checks duplicate name in same VPC (DuplicateTargetGroupName) → generates TG ID → creates HealthCheckConfig with provided values or defaults → stores record with empty Targets array → returns TargetGroup with ARN | 1. Create with defaults<br>2. Custom health check<br>3. Duplicate name (DuplicateTargetGroupName)<br>4. Missing name (MissingParameter) | **DONE** |
| `delete-target-group` | `--target-group-arn` | `--dry-run` | TG must exist, not referenced by listener | Gateway validates TargetGroupArn (required) → NATS `elbv2.DeleteTargetGroup` → daemon verifies TG exists → scans all listeners to check if any DefaultActions reference this TG (TargetGroupInUse if referenced) → deletes from KV | 1. Delete TG<br>2. TG in use by listener (TargetGroupInUse)<br>3. Not found (TargetGroupNotFound) | **DONE** |
| `describe-target-groups` | `--target-group-arns`, `--names`, `--load-balancer-arn` | `--page-size`, `--marker`, `--dry-run` | None | NATS `elbv2.DescribeTargetGroups` → daemon lists all TG records → filters by account → applies ARN/Name filters → if LoadBalancerArn specified, lists listeners for that LB and includes only TGs referenced by those listeners → returns TargetGroup list | 1. List all TGs<br>2. Filter by LB ARN (via listener lookup)<br>3. Account isolation | **DONE** |
#### ELBv2 - Targets
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `register-targets` | `--target-group-arn`, `--targets` (Id=instance-id, Port=optional override) | `--dry-run` | TG must exist, instances should exist | Gateway validates TargetGroupArn + Targets (required) → NATS `elbv2.RegisterTargets` → daemon fetches TG → deduplicates against existing targets (key: `{id}:{port}`, idempotent) → resolves instance ID → private IP via VPC ENI lookup → adds target with state=initial → persists to KV → calls `pushConfigForTargetGroup()` to regenerate HAProxy config on all LBs using this TG → returns success | 1. Register targets<br>2. Idempotent re-registration<br>3. TG not found (TargetGroupNotFound)<br>4. Missing ARN or targets (MissingParameter) | **DONE** |
| `deregister-targets` | `--target-group-arn`, `--targets` (Id, Port) | `--dry-run` | TG must exist | Gateway validates TargetGroupArn + Targets (required) → NATS `elbv2.DeregisterTargets` → daemon fetches TG → builds removal set (key: `{id}:{port}`) → filters Targets array to keep only non-matching → persists to KV → calls `pushConfigForTargetGroup()` to update HAProxy config → returns success | 1. Deregister targets<br>2. TG not found (TargetGroupNotFound)<br>3. Missing parameters (MissingParameter) | **DONE** |
| `describe-target-health` | `--target-group-arn`, `--targets` (optional filter to specific targets) | `--include` | TG must exist | Gateway validates TargetGroupArn (required) → NATS `elbv2.DescribeTargetHealth` → daemon fetches TG → builds filter set from Targets param (if specified) → for each target, returns TargetHealthDescription with Target (Id, Port) and TargetHealth (State, Description). Note: health state is stored in records, not actively polled — HAProxy runs its own independent health checks. | 1. Describe all targets<br>2. Filter to specific targets<br>3. TG not found (TargetGroupNotFound) | **DONE** |
#### ELBv2 - Listeners
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP, default), `--port` (default 80) | `--ssl-policy`, `--certificates`, `--alpn-policy`, `--mutual-authentication`, `--dry-run` | LB must exist | Gateway validates LoadBalancerArn + DefaultActions (required) → NATS `elbv2.CreateListener` → daemon verifies LB exists → checks for duplicate port on same LB (DuplicateListener) → generates listener ID → builds ARN → stores ListenerRecord in KV → calls `pushConfig(lb)` to regenerate HAProxy config (creates frontend for listener port, backend for target group with health checks) → returns Listener with ARN, Protocol, Port, DefaultActions | 1. Create listener<br>2. Duplicate port (DuplicateListener)<br>3. LB not found (LoadBalancerNotFound)<br>4. Graceful handling when NATS unavailable | **DONE** |
| `delete-listener` | `--listener-arn` | `--dry-run` | Listener must exist | Gateway validates ListenerArn (required) → NATS `elbv2.DeleteListener` → daemon verifies listener exists and belongs to account → deletes from KV → fetches parent LB → calls `pushConfig(lb)` to regenerate HAProxy config without this listener → returns success | 1. Delete listener<br>2. Not found (ListenerNotFound) | **DONE** |
| `describe-listeners` | `--load-balancer-arn`, `--listener-arns` | `--page-size`, `--marker`, `--dry-run` | None | NATS `elbv2.DescribeListeners` → daemon lists listeners → if LoadBalancerArn specified, filters to that LB → applies ARN filter → account isolation → returns Listener list | 1. Filter by LB ARN<br>2. Account isolation | **DONE** |
#### ELBv2 - Tags
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-tags` | `--resource-arns` (loadbalancer, targetgroup, listener) | — | Resource(s) must exist | Gateway validates non-empty ResourceArns → NATS `elbv2.DescribeTags` → daemon parses each ARN's resource segment to dispatch by type → looks up record (LB/TG/listener) → enforces account isolation (cross-account → not-found) → returns TagDescriptions in input order with sorted-by-key Tag list. Listener records don't store tags yet, so they always return an empty Tags slice (matches AWS behaviour for untagged resources). Required by Terraform AWS provider post-create refresh on `aws_lb`, `aws_lb_target_group`, and `aws_lb_listener`. | 1. LB tags round-trip<br>2. TG tags round-trip<br>3. Listener returns empty Tags (no error)<br>4. Multiple ARNs in one call<br>5. Unknown LB/TG/listener (per-type not-found)<br>6. Cross-account ARN (not-found, no leak)<br>7. Invalid/non-ELBv2 ARN (InvalidParameterValue)<br>8. Empty/nil ResourceArns (MissingParameter)<br>9. Untagged LB returns empty Tags | **DONE** |
#### ELBv2 - Not Implemented
| Feature | Description | Priority | Status |
|---------|-------------|----------|--------|
| Listener rules | Path-based and hostname-based routing (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | **NOT STARTED** |
| HTTPS/TLS termination | SSL certificates, SSL policies, ALPN | High | **NOT STARTED** |
| Modify operations | ModifyLoadBalancerAttributes, ModifyTargetGroup, ModifyTargetGroupAttributes, ModifyListener | Medium | **NOT STARTED** |
| Connection draining | Deregistration delay for graceful target removal | Medium | **NOT STARTED** |
| Stickiness | Session affinity / sticky sessions on target groups | Medium | **NOT STARTED** |
| Active health checking | API-driven health state updates (currently HAProxy-only) | Medium | **NOT STARTED** |
| Target types | IP and Lambda target types (only instance supported) | Low | **NOT STARTED** |
| Access logs | S3 access log delivery | Low | **NOT STARTED** |
| WAF integration | AWS WAF association for web application firewall | Low | **NOT STARTED** |
### CloudWatch (Basic Monitoring)
Basic CloudWatch metrics support is required for `monitor-instances`/`unmonitor-instances` to be meaningful. Daemon nodes should publish CPU, disk, and network metrics for running instances. Metrics stored in a time-series-friendly structure in JetStream KV or dedicated NATS subjects.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `put-metric-data` | — | `--namespace`, `--metric-data` (MetricName, Value, Unit, Timestamp, Dimensions) | None | Gateway validates Namespace + MetricData (required) → NATS `cloudwatch.PutMetricData` → daemon stores metric data points in time-series KV (key: `{namespace}.{metricName}.{dimensionHash}`) → return success | 1. Put single metric<br>2. Put with dimensions (InstanceId)<br>3. Missing namespace (MissingParameter) | **NOT STARTED** |
| `get-metric-statistics` | — | `--namespace`, `--metric-name`, `--start-time`, `--end-time`, `--period`, `--statistics` (Average, Sum, Min, Max, SampleCount), `--dimensions` | None | Gateway validates required fields → NATS `cloudwatch.GetMetricStatistics` → daemon queries time-series data → aggregates by period → returns Datapoints with requested statistics | 1. Get CPU utilization over 1 hour<br>2. Get with dimensions (InstanceId)<br>3. No data returns empty | **NOT STARTED** |
| `list-metrics` | — | `--namespace`, `--metric-name`, `--dimensions`, `--recently-active` | None | NATS `cloudwatch.ListMetrics` → daemon scans metric namespaces → returns Metrics list with Namespace, MetricName, Dimensions | 1. List all metrics<br>2. Filter by namespace<br>3. Filter by metric name | **NOT STARTED** |
| `describe-alarms` | — | `--alarm-names`, `--alarm-name-prefix`, `--state-value`, `--action-prefix` | None | NATS `cloudwatch.DescribeAlarms` → daemon lists alarms from KV → return MetricAlarms with State, Threshold, ComparisonOperator | 1. List all alarms<br>2. Filter by state (OK, ALARM, INSUFFICIENT_DATA) | **NOT STARTED** |
| `put-metric-alarm` | — | `--alarm-name`, `--namespace`, `--metric-name`, `--statistic`, `--period`, `--evaluation-periods`, `--threshold`, `--comparison-operator`, `--alarm-actions`, `--dimensions` | None | NATS `cloudwatch.PutMetricAlarm` → daemon stores alarm config in KV → alarm evaluation runs on metric publish → state transitions trigger alarm actions | 1. Create alarm<br>2. Update existing alarm<br>3. Missing required fields | **NOT STARTED** |
| `delete-alarms` | — | `--alarm-names` | Alarms must exist | NATS `cloudwatch.DeleteAlarms` → daemon deletes alarm records from KV → return success | 1. Delete existing alarm<br>2. Non-existent alarm (ResourceNotFound) | **NOT STARTED** |
### Auto Scaling
Auto Scaling manages instance fleets based on scaling policies. Depends on Launch Templates (EC2) and optionally ELBv2 target groups for health-based scaling. Stored in `spinifex-autoscaling` KV bucket.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-auto-scaling-group` | — | `--auto-scaling-group-name`, `--launch-template` (Id/Name, Version), `--min-size`, `--max-size`, `--desired-capacity`, `--vpc-zone-identifier` (subnet IDs), `--target-group-arns`, `--health-check-type` (EC2/ELB), `--health-check-grace-period`, `--tags` | Launch template must exist | NATS `autoscaling.CreateAutoScalingGroup` → daemon validates launch template → stores ASG record in KV → launches DesiredCapacity instances using template → registers with target groups if specified → starts health check loop → return success | 1. Create ASG with launch template<br>2. Scale to desired capacity<br>3. Missing launch template (ValidationError)<br>4. Min > Max (ValidationError) | **NOT STARTED** |
| `update-auto-scaling-group` | — | `--auto-scaling-group-name`, `--min-size`, `--max-size`, `--desired-capacity`, `--launch-template`, `--health-check-type`, `--health-check-grace-period` | ASG must exist | NATS `autoscaling.UpdateAutoScalingGroup` → daemon updates ASG config in KV → if DesiredCapacity changed, scale up/down accordingly → return success | 1. Update desired capacity<br>2. Update launch template<br>3. Non-existent ASG (ValidationError) | **NOT STARTED** |
| `delete-auto-scaling-group` | — | `--auto-scaling-group-name`, `--force-delete` | ASG must exist | NATS `autoscaling.DeleteAutoScalingGroup` → if force: terminate all instances → deregister from target groups → delete from KV → return success. Without force: reject if instances > 0 | 1. Delete empty ASG<br>2. Force delete with instances<br>3. Delete with instances (ResourceInUse) | **NOT STARTED** |
| `describe-auto-scaling-groups` | — | `--auto-scaling-group-names`, `--filters`, `--max-records` | None | NATS `autoscaling.DescribeAutoScalingGroups` → daemon lists ASG records → return list with Instances, DesiredCapacity, MinSize, MaxSize | 1. List all ASGs<br>2. Filter by name | **NOT STARTED** |
| `set-desired-capacity` | — | `--auto-scaling-group-name`, `--desired-capacity`, `--honor-cooldown` | ASG must exist | NATS `autoscaling.SetDesiredCapacity` → daemon updates desired count → scale up (launch) or scale down (terminate oldest) → return success | 1. Scale up<br>2. Scale down<br>3. Exceeds max (ValidationError) | **NOT STARTED** |
| `describe-auto-scaling-instances` | — | `--instance-ids`, `--max-records` | None | NATS `autoscaling.DescribeAutoScalingInstances` → daemon lists all ASG-managed instances → return list with ASG name, lifecycle state, health status | 1. List all ASG instances<br>2. Filter by instance ID | **NOT STARTED** |
| `put-scaling-policy` | — | `--auto-scaling-group-name`, `--policy-name`, `--policy-type` (TargetTrackingScaling, StepScaling, SimpleScaling), `--target-tracking-configuration`, `--scaling-adjustment`, `--cooldown` | ASG must exist | NATS `autoscaling.PutScalingPolicy` → daemon stores policy in KV → links to CloudWatch alarms for metric-based triggers → return PolicyARN | 1. Create target tracking policy<br>2. Create step scaling policy<br>3. Non-existent ASG (ValidationError) | **NOT STARTED** |
| `delete-scaling-policy` | — | `--auto-scaling-group-name`, `--policy-name` | Policy must exist | NATS `autoscaling.DeleteScalingPolicy` → daemon deletes policy → remove CloudWatch alarm links → return success | 1. Delete existing policy<br>2. Non-existent policy | **NOT STARTED** |
+26 -72
View File
@@ -1,64 +1,18 @@
# Hive Architecture
# Spinifex Architecture
Hive is an AWS-compatible infrastructure platform for bare-metal, edge, and on-premise deployments. It provides EC2-compatible VM orchestration backed by QEMU/KVM, with storage provided by companion projects Viperblock (EBS) and Predastore (S3).
Spinifex is an AWS-compatible infrastructure platform for bare-metal, edge, and on-premise deployments. It provides EC2-compatible VM orchestration backed by QEMU/KVM, with storage provided by companion projects Viperblock (EBS) and Predastore (S3).
## High-Level Architecture
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ AWS SDK / CLI │
│ (aws ec2 run-instances --endpoint-url ...) │
└─────────────────────────────────────────────────────────────────────────────┘
│
│ HTTPS (SigV4 Auth)
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ AWS Gateway (Port 9999) │
│ │
│ • TLS termination • AWS SigV4 authentication │
│ • EC2 Query Protocol parsing • Routes to service handlers │
│ • XML response generation • NATS client connection │
│ │
│ File: hive/services/awsgw/awsgw.go │
└─────────────────────────────────────────────────────────────────────────────┘
│
│ JSON over NATS
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ NATS Message Broker │
│ │
│ Topics: Queue Groups: │
│ • ec2.RunInstances • hive-workers (load balanced) │
│ • ec2.DescribeInstances • No queue (fan-out to all nodes) │
│ • ec2.cmd.{instanceID} (per-instance start/stop/terminate) │
│ • ebs.mount / ebs.unmount │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ Hive Daemon │ │ Hive Daemon │ │ Hive Daemon │
│ (Node 1) │ │ (Node 2) │ │ (Node N) │
│ │ │ │ │ │
│ • NATS subscriber │ │ • NATS subscriber │ │ • NATS subscriber │
│ • Resource manager │ │ • Resource manager │ │ • Resource manager │
│ • QEMU/KVM launcher │ │ • QEMU/KVM launcher │ │ • QEMU/KVM launcher │
│ • QMP monitoring │ │ • QMP monitoring │ │ • QMP monitoring │
│ │ │ │ │ │
│ File: daemon.go │ │ File: daemon.go │ │ File: daemon.go │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ QEMU/KVM VMs │ │ QEMU/KVM VMs │ │ QEMU/KVM VMs │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
```
<p align="center">
<img src="../.github/assets/design-architecture.svg" alt="Spinifex high-level architecture — AWS SDK, gateway, NATS broker, per-node daemons, QEMU/KVM VMs" width="900">
</p>
## Request Flow
### 1. AWS SDK Request
Users interact with Hive using standard AWS SDKs or the AWS CLI by pointing to a custom endpoint:
Users interact with Spinifex using standard AWS SDKs or the AWS CLI by pointing to a custom endpoint:
```bash
aws --endpoint-url https://localhost:9999 --no-verify-ssl ec2 run-instances \
@@ -73,7 +27,7 @@ The AWS SDK formats this as an HTTP POST with:
### 2. AWS Gateway
The gateway (`hive/services/awsgw/awsgw.go:57-107`) is the entry point:
The gateway (`spinifex/services/awsgw/awsgw.go:57-107`) is the entry point:
```go
// awsgw.go:74 - Connect to NATS
@@ -89,7 +43,7 @@ gw := gateway.GatewayConfig{
log.Fatal(app.ListenTLS(nodeConfig.AWSGW.Host, nodeConfig.AWSGW.TLSCert, nodeConfig.AWSGW.TLSKey))
```
Request routing (`hive/gateway/gateway.go:125-156`):
Request routing (`spinifex/gateway/gateway.go:125-156`):
1. **Authentication**: SigV4 middleware validates AWS credentials (`gateway.go:104`)
2. **Service Detection**: Extracts service name (ec2, iam, account) from Authorization header (`gateway.go:158-186`)
@@ -109,7 +63,7 @@ case "iam":
### 3. EC2 Handler
The EC2 handler (`hive/gateway/ec2.go`) parses the Action parameter and delegates to specific handlers:
The EC2 handler (`spinifex/gateway/ec2.go`) parses the Action parameter and delegates to specific handlers:
```go
// ec2.go - Action routing
@@ -124,7 +78,7 @@ case "DescribeInstances":
### 4. NATS Messaging
The gateway communicates with daemons via NATS request/response (`hive/handlers/ec2/instance/service_nats.go:24-57`):
The gateway communicates with daemons via NATS request/response (`spinifex/handlers/ec2/instance/service_nats.go:24-57`):
```go
// service_nats.go:24-57
@@ -145,13 +99,13 @@ func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput) (*ec2.R
### 5. Daemon Processing
Daemons (`hive/daemon/daemon.go`) subscribe to NATS topics and handle requests:
Daemons (`spinifex/daemon/daemon.go`) subscribe to NATS topics and handle requests:
```go
// daemon.go:319 - Subscribe with queue group for load balancing
d.natsSubscriptions["ec2.RunInstances"], err = d.natsConn.QueueSubscribe(
"ec2.RunInstances",
"hive-workers", // Queue group - only one daemon handles each request
"spinifex-workers", // Queue group - only one daemon handles each request
d.handleEC2RunInstances,
)
@@ -162,7 +116,7 @@ d.natsSubscriptions["ec2.DescribeInstances"], err = d.natsConn.Subscribe(
)
```
**Queue Groups**: Topics subscribed with a queue group (`hive-workers`) are load-balanced - only one daemon handles each request. Topics without a queue group fan out to all daemons.
**Queue Groups**: Topics subscribed with a queue group (`spinifex-workers`) are load-balanced - only one daemon handles each request. Topics without a queue group fan out to all daemons.
### 6. VM Launch
@@ -225,19 +179,19 @@ type ResourceManager struct {
| Topic | Queue Group | Purpose |
|-------|-------------|---------|
| `ec2.RunInstances` | `hive-workers` | Launch new instances |
| `ec2.RunInstances` | `spinifex-workers` | Launch new instances |
| `ec2.DescribeInstances` | None (fan-out) | Query all nodes for instances |
| `ec2.cmd.{instanceID}` | None (owner only) | Per-instance commands (start/stop/terminate, attach/detach volume) |
| `ec2.CreateKeyPair` | `hive-workers` | Generate SSH keypair |
| `ec2.DescribeKeyPairs` | `hive-workers` | List SSH keypairs |
| `ec2.DescribeImages` | `hive-workers` | List AMIs |
| `ec2.CreateKeyPair` | `spinifex-workers` | Generate SSH keypair |
| `ec2.DescribeKeyPairs` | `spinifex-workers` | List SSH keypairs |
| `ec2.DescribeImages` | `spinifex-workers` | List AMIs |
| `ebs.mount` | - | Mount volume via Viperblock |
| `ebs.unmount` | - | Unmount volume |
| `hive.admin.{node}.health` | None | Node health checks |
| `spinifex.admin.{node}.health` | None | Node health checks |
### Multi-Node Aggregation
For operations that need data from all nodes (like `DescribeInstances`), the gateway uses inbox-based fan-out (`hive/gateway/ec2/instance/DescribeInstances.go`):
For operations that need data from all nodes (like `DescribeInstances`), the gateway uses inbox-based fan-out (`spinifex/gateway/ec2/instance/DescribeInstances.go`):
```go
// DescribeInstances.go:16-111
@@ -288,13 +242,13 @@ Object storage used for:
## Configuration
Cluster configuration (`hive/config/config.go`):
Cluster configuration (`spinifex/config/config.go`):
```go
type ClusterConfig struct {
Epoch uint64 // Incremented on config changes
Node string // This node's identifier
Version string // Hive version
Version string // Spinifex version
Nodes map[string]Config // Configuration for all nodes
}
@@ -312,8 +266,8 @@ type Config struct {
## File Structure
```
hive/
├── hive/
spinifex/
├── spinifex/
│ ├── services/
│ │ └── awsgw/
│ │ └── awsgw.go # AWS Gateway service entry point
@@ -340,7 +294,7 @@ hive/
│ └── qmp/
│ └── qmp.go # QEMU Machine Protocol client
└── cmd/
└── hive/
└── spinifex/
└── main.go # CLI entry point
```
@@ -351,8 +305,8 @@ hive/
1. Start NATS broker
2. Start Predastore (S3)
3. Start Viperblock (EBS)
4. Start Hive Gateway (`hive awsgw`)
5. Start Hive Daemon(s) (`hive daemon`)
4. Start Spinifex Gateway (`spx awsgw`)
5. Start Spinifex Daemon(s) (`spx daemon`)
### Testing with AWS CLI
File diff suppressed because it is too large Load Diff
+3
View File
@@ -0,0 +1,3 @@
# Spinifex Documentation
This directory contains the Spinifex documentation. This is available in individual markdown files, and a generated website to aid navigation and browing documentation at: [https://docs.mulgadc.com](https://docs.mulgadc.com/)
+66
View File
@@ -0,0 +1,66 @@
# Spinifex Tag Conventions
Spinifex uses a small set of reserved tag keys to mark resources that are
owned and managed by the platform itself (as opposed to resources created
directly by customer AWS API calls). The UI uses these tags to hide
system-owned resources from customer-facing listings so operators do not
mistake them for user workloads and accidentally terminate or modify
them.
## `spinifex:managed-by`
Identifies the Spinifex platform component that owns a resource.
| Value | Component | Tagged resource types |
|---------|---------------------------------|---------------------------------------|
| `elbv2` | ALB / HAProxy load balancers | LB VM (`instance`), LB ENI, LB AMI |
The tag is applied by the backend at create/launch time:
- **Instance** — set in
`spinifex/spinifex/daemon/daemon_system_instance.go` when
`LaunchSystemInstance` builds the `RunInstancesInput`, and mirrored onto
the stored `vm.VM.ManagedBy` field for fast lookup.
- **ENI** — set in
`spinifex/spinifex/handlers/elbv2/service_impl.go` when
`CreateLoadBalancer` calls `CreateNetworkInterface`.
- **AMI** — declared on the catalog entry in
`spinifex/spinifex/utils/images.go` and copied into
`AMIMetadata.Tags` by the image importer in
`spinifex/cmd/spinifex/cmd/admin.go`.
Tag keys and values are defined once in `spinifex/spinifex/tags/tags.go`;
new system components should register their value there and reuse the
constants rather than hardcoding strings.
## UI behaviour
The admin **Nodes** page and the **Images** (AMIs) page filter out
resources that carry `spinifex:managed-by`. Engineers who need to see
system resources for debugging can append `?system=1` to the URL — the
filter is bypassed and a hidden-count banner is replaced with the full
listing. Direct navigation to a system AMI detail page without
`?system=1` renders the "Image not found" fallback.
The EC2 **Instances** page does not need UI-side filtering: backend
SigV4 authentication scopes `DescribeInstances` by account ID, and
system instances live under the global system account
(`000000000000`), so customers never see them.
## `spinifex:lb-arn`
Applied to ELBv2-managed ENIs alongside `spinifex:managed-by=elbv2`.
Stores the ARN of the parent load balancer so reconciliation can map an
ENI back to its owner without a separate lookup. Customer-facing
filtering keys off `spinifex:managed-by`; `spinifex:lb-arn` is
informational.
## Adding a new system component
1. Add a new value constant next to `ManagedByELBv2` in
`spinifex/spinifex/tags/tags.go`.
2. Tag every resource the component creates (instance, ENI, AMI,
volume, …) with `spinifex:managed-by=<value>`.
3. Add the value to the table above so operators can recognise it.
4. No frontend changes are required — the UI filters on the key, not
the value.
+55
View File
@@ -0,0 +1,55 @@
---
title: "Creating Accounts"
description: "Create and manage isolated user accounts with their own resources."
category: "Administration"
tags:
- accounts
- iam
- admin
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
---
# Creating Accounts
> Create and manage isolated user accounts with their own resources.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex supports multi-tenant account isolation. Each account gets its own IAM credentials, AWS CLI profile, and isolated resource namespace.
## Instructions
## Create Account
```bash
spx admin account create --name myteam
export AWS_PROFILE=spinifex-myteam
```
## Troubleshooting
## Credentials Not Working
Verify the AWS CLI configuration files exist and contain the correct profile:
```bash
cat ~/.aws/config
cat ~/.aws/credentials
```
Ensure the `AWS_PROFILE` environment variable matches the profile name:
```bash
echo $AWS_PROFILE
export AWS_PROFILE=spinifex-myteam
```
+71
View File
@@ -0,0 +1,71 @@
---
title: "Hybrid Sync"
description: "Synchronize data between Spinifex and AWS when network connectivity is available."
category: "Migration"
tags:
- hybrid
- sync
- s3
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
---
# Hybrid Sync
> Synchronize data between Spinifex and AWS when network connectivity is available.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex's hybrid mode enables bidirectional data synchronization between local infrastructure and AWS cloud services. Ideal for intermittent connectivity environments.
## Instructions
## Prerequisites
Ensure the AWS profile is set:
```bash
export AWS_PROFILE=spinifex
```
## Push Local to Cloud
```bash
aws s3 sync s3://local-bucket/ s3://cloud-bucket/ \
--source-region spinifex --region us-east-1
```
## Pull Cloud to Local
```bash
aws s3 sync s3://cloud-bucket/ s3://local-bucket/ \
--source-region us-east-1 --region spinifex
```
## EBS Volume Backup
```bash
rsync -avz /data/ user@cloud-server:/backup/spinifex-data/
```
## Troubleshooting
## Sync Fails Mid Transfer
S3 sync is idempotent — re-run the same command to resume where it left off:
```bash
aws s3 sync s3://local-bucket/ s3://cloud-bucket/ \
--source-region spinifex --region us-east-1
```
Only changed or missing files will be transferred on subsequent runs.
+538
View File
@@ -0,0 +1,538 @@
---
title: "IAM Users and Policies"
description: "Create IAM users, manage access keys, and control permissions with policies."
category: "Administration"
tags:
- iam
- users
- policies
- access keys
- security
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "AWS IAM CLI Reference"
url: "https://docs.aws.amazon.com/cli/latest/reference/iam/"
- title: "IAM Policy Reference"
url: "https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html"
---
# IAM Users and Policies
> Create IAM users, manage access keys, and control permissions with policies.
## Table of Contents
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [Users](#users)
- [Access Keys](#access-keys)
- [Policies](#policies)
- [Attaching Policies](#attaching-policies)
- [Putting It All Together](#putting-it-all-together)
- [Command Reference](#command-reference)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex implements AWS-compatible IAM with 16 operations covering user management, access key lifecycle, policy CRUD, and policy attachment. All IAM resources are scoped to the account that creates them — users in one account cannot see or modify resources in another.
When you create an account with `spx admin account create`, Spinifex bootstraps a root user with an `AdministratorAccess` policy and writes the credentials to `~/.aws/credentials`. From there, you use the standard AWS CLI to manage additional users and permissions.
**How authentication works:** Every AWS CLI request is signed with SigV4 using an access key pair. The gateway verifies the signature, resolves the caller's account, and evaluates attached policies before routing the request. The root user (account `000000000000`) bypasses policy evaluation entirely.
## Instructions
## Prerequisites
- A running Spinifex cluster with the AWS gateway started
- An account created via `spx admin account create` (see [Creating Accounts](/docs/creating-accounts))
- AWS CLI configured with the account's profile:
```bash
export AWS_PROFILE=spinifex-myteam
```
## Users
### Create a User
```bash
aws iam create-user --user-name alice
```
The response includes the user's ARN, unique ID, and creation date:
```json
{
"User": {
"UserName": "alice",
"UserId": "AIDA1A2B3C4D5E6F7890",
"Arn": "arn:aws:iam::000000000001:user/alice",
"Path": "/",
"CreateDate": "2026-03-24T10:00:00Z"
}
}
```
Use `--path` to organise users into hierarchical groups:
```bash
aws iam create-user --user-name bob --path /developers/
```
### Get a User
```bash
aws iam get-user --user-name alice
```
### List Users
```bash
aws iam list-users
```
Filter by path prefix:
```bash
aws iam list-users --path-prefix /developers/
```
### Delete a User
Before deleting a user, remove all access keys and detach all policies:
```bash
# Remove access keys
aws iam list-access-keys --user-name alice
aws iam delete-access-key --user-name alice --access-key-id AKIA...
# Detach policies
aws iam list-attached-user-policies --user-name alice
aws iam detach-user-policy --user-name alice \
--policy-arn arn:aws:iam::000000000001:policy/MyPolicy
# Now delete
aws iam delete-user --user-name alice
```
## Access Keys
Access keys are how users authenticate with the AWS CLI. Each user can have up to **2 access keys** at a time, allowing key rotation without downtime.
### Create an Access Key
```bash
aws iam create-access-key --user-name alice
```
```json
{
"AccessKey": {
"UserName": "alice",
"AccessKeyId": "AKIA1A2B3C4D5E6F7890ABCD",
"Status": "Active",
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"CreateDate": "2026-03-24T10:05:00Z"
}
}
```
> **The secret access key is only shown once.** Save it immediately. If lost, delete the key and create a new one.
Add the credentials to `~/.aws/credentials` under a new profile and set `AWS_PROFILE` to use them.
### List Access Keys
```bash
aws iam list-access-keys --user-name alice
```
### Rotate Access Keys
Create a second key, update your configuration, then delete the old one:
```bash
# 1. Create new key (while old key still works)
aws iam create-access-key --user-name alice
# 2. Update ~/.aws/credentials with the new key
# 3. Verify the new key works
AWS_PROFILE=spinifex-alice aws sts get-caller-identity
# 4. Delete the old key
aws iam delete-access-key --user-name alice --access-key-id AKIA_OLD_KEY_ID
```
### Deactivate an Access Key
Temporarily disable a key without deleting it:
```bash
aws iam update-access-key --user-name alice \
--access-key-id AKIA1A2B3C4D5E6F7890ABCD \
--status Inactive
```
Reactivate it later:
```bash
aws iam update-access-key --user-name alice \
--access-key-id AKIA1A2B3C4D5E6F7890ABCD \
--status Active
```
## Policies
Policies are JSON documents that define what actions a user can perform on which resources.
### Policy Document Format
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["ec2:RunInstances", "ec2:DescribeInstances"],
"Resource": "*"
}
]
}
```
Each statement has:
- **Effect** — `Allow` or `Deny`
- **Action** — Service actions (e.g. `ec2:RunInstances`, `s3:GetObject`). Supports wildcards: `ec2:*`, `s3:Get*`, or `*` for all actions.
- **Resource** — Target resources. Use `*` for all resources.
**Evaluation order:** An explicit `Deny` always wins. If no statement matches, access is denied by default.
### Create a Policy
Save the policy document to a file, then create the policy:
```bash
cat > /tmp/ec2-readonly.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeImages",
"ec2:DescribeKeyPairs",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs"
],
"Resource": "*"
}
]
}
EOF
aws iam create-policy \
--policy-name EC2ReadOnly \
--policy-document file:///tmp/ec2-readonly.json
```
```json
{
"Policy": {
"PolicyName": "EC2ReadOnly",
"PolicyId": "ANPA1A2B3C4D5E6F7890",
"Arn": "arn:aws:iam::000000000001:policy/EC2ReadOnly",
"Path": "/",
"DefaultVersionId": "v1",
"CreateDate": "2026-03-24T10:10:00Z"
}
}
```
### Common Policy Examples
**Full administrator access:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
```
**S3 read/write (for hybrid sync, backups):**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket",
"s3:DeleteObject"
],
"Resource": "*"
}
]
}
```
**EC2 operator (launch and manage instances, no VPC changes):**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:RunInstances",
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:DescribeInstances",
"ec2:DescribeImages"
],
"Resource": "*"
}
]
}
```
**Deny terminate (attach alongside a broader Allow to prevent accidental termination):**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": "ec2:TerminateInstances",
"Resource": "*"
}
]
}
```
### Get a Policy
```bash
aws iam get-policy \
--policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```
### Get the Policy Document
Use `get-policy-version` with version `v1` to retrieve the actual JSON document:
```bash
aws iam get-policy-version \
--policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly \
--version-id v1
```
### List Policies
```bash
aws iam list-policies
```
### Delete a Policy
A policy must be detached from all users before it can be deleted:
```bash
aws iam delete-policy \
--policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```
## Attaching Policies
Policies have no effect until attached to a user.
### Attach a Policy to a User
```bash
aws iam attach-user-policy --user-name alice \
--policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```
### List a User's Policies
```bash
aws iam list-attached-user-policies --user-name alice
```
### Detach a Policy
```bash
aws iam detach-user-policy --user-name alice \
--policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```
## Putting It All Together
A complete workflow for onboarding a developer with scoped EC2 and S3 access:
```bash
# 1. Create the user
aws iam create-user --user-name dev-carol --path /developers/
# 2. Create an access key
aws iam create-access-key --user-name dev-carol
# Save the AccessKeyId and SecretAccessKey from the output
# 3. Create a scoped policy
cat > /tmp/dev-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EC2Access",
"Effect": "Allow",
"Action": ["ec2:*"],
"Resource": "*"
},
{
"Sid": "S3ReadOnly",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": "*"
},
{
"Sid": "NoTerminate",
"Effect": "Deny",
"Action": "ec2:TerminateInstances",
"Resource": "*"
}
]
}
EOF
aws iam create-policy \
--policy-name DeveloperAccess \
--policy-document file:///tmp/dev-policy.json
# 4. Attach the policy
aws iam attach-user-policy --user-name dev-carol \
--policy-arn arn:aws:iam::000000000001:policy/DeveloperAccess
# 5. Configure the developer's AWS CLI
aws configure --profile spinifex-carol
# Enter the access key and secret from step 2
# 6. Verify
AWS_PROFILE=spinifex-carol aws ec2 describe-instances
AWS_PROFILE=spinifex-carol aws ec2 terminate-instances --instance-ids i-123
# ^ This will be denied by the NoTerminate statement
```
## Command Reference
| Command | Key Parameters | Notes |
| ----------------------------- | -------------------------------------------- | --------------------------------- |
| `create-user` | `--user-name`, `--path` | Path default: `/` |
| `get-user` | `--user-name` | |
| `list-users` | `--path-prefix` | |
| `delete-user` | `--user-name` | Must have no keys or policies |
| `create-access-key` | `--user-name` | Max 2 per user; secret shown once |
| `list-access-keys` | `--user-name` | |
| `delete-access-key` | `--user-name`, `--access-key-id` | |
| `update-access-key` | `--user-name`, `--access-key-id`, `--status` | `Active` or `Inactive` |
| `create-policy` | `--policy-name`, `--policy-document` | Max 6144 bytes |
| `get-policy` | `--policy-arn` | Metadata only |
| `get-policy-version` | `--policy-arn`, `--version-id` | Use `v1`; includes document |
| `list-policies` | | |
| `delete-policy` | `--policy-arn` | Must not be attached |
| `attach-user-policy` | `--user-name`, `--policy-arn` | Idempotent |
| `detach-user-policy` | `--user-name`, `--policy-arn` | |
| `list-attached-user-policies` | `--user-name` | |
## Troubleshooting
### AccessDenied on IAM Commands
The calling user must have IAM permissions. Attach a policy with `iam:*` actions, or use the root account profile:
```bash
export AWS_PROFILE=spinifex-myteam
aws iam list-users
```
### InvalidClientTokenId
The access key is either inactive or does not exist. Check the key status:
```bash
aws iam list-access-keys --user-name alice
```
If the key shows `Inactive`, reactivate it:
```bash
aws iam update-access-key --user-name alice \
--access-key-id AKIA... --status Active
```
If the key was deleted, create a new one.
### DeleteConflict When Deleting a User
The user still has access keys or attached policies. Remove them first:
```bash
# Check for access keys
aws iam list-access-keys --user-name alice
# Check for attached policies
aws iam list-attached-user-policies --user-name alice
```
### DeleteConflict When Deleting a Policy
The policy is still attached to one or more users. Detach it from all users before deleting:
```bash
# Find who has it attached, then detach
aws iam detach-user-policy --user-name alice \
--policy-arn arn:aws:iam::000000000001:policy/MyPolicy
aws iam delete-policy \
--policy-arn arn:aws:iam::000000000001:policy/MyPolicy
```
### LimitExceeded When Creating Access Keys
Each user can have at most 2 access keys. Delete an existing key before creating a new one:
```bash
aws iam list-access-keys --user-name alice
aws iam delete-access-key --user-name alice --access-key-id AKIA_OLD
aws iam create-access-key --user-name alice
```
### MalformedPolicyDocument
The policy JSON is invalid. Check that:
- `Version` is exactly `"2012-10-17"`
- At least one `Statement` exists
- Each statement has `Effect` (`Allow`/`Deny`), `Action`, and `Resource`
- The document is under 6144 bytes
- The JSON is well-formed (no trailing commas, correct quoting)
+131
View File
@@ -0,0 +1,131 @@
---
title: "Air-Gapped Install"
description: "Deploy Spinifex in environments without internet connectivity. Covers offline package caching, USB deployment, and package verification."
category: "Getting Started"
tags:
- install
- air-gapped
- offline
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "apt-cacher-ng"
url: "https://wiki.debian.org/AptCacherNg"
---
# Air-Gapped Installation
> Deploy Spinifex in environments without internet connectivity.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
In air-gapped or disconnected environments, Spinifex can be deployed without internet access. This guide covers preparing offline packages on a connected machine, creating USB deployment media, and installing on the target server with package verification.
**Deployment methods:**
- **apt-cacher-ng** — Local APT cache for Debian/Ubuntu packages
- **Go module cache** — Pre-downloaded Go dependencies
- **USB deployment** — Portable installation media with all dependencies
- **GPG verification** — Cryptographic package integrity checks
## Instructions
## Step 1. Prepare Packages (on a connected machine)
Set up a local APT cache to collect all required packages:
```bash
sudo apt install apt-cacher-ng
sudo systemctl enable --now apt-cacher-ng
```
Pre-download all Spinifex dependencies:
```bash
sudo apt install --download-only nbdkit nbdkit-plugin-dev pkg-config \
qemu-system qemu-utils qemu-kvm libvirt-daemon-system \
libvirt-clients libvirt-dev make gcc unzip xz-utils file \
ovn-central ovn-host openvswitch-switch
```
## Step 2. Create USB Deployment Media
```bash
mkdir -p /media/spinifex-deploy/{apt-packages,go-cache,spinifex-source}
cp /var/cache/apt/archives/*.deb /media/spinifex-deploy/apt-packages/
cp -r ~/go/pkg/mod/cache/ /media/spinifex-deploy/go-cache/
cp -r ~/Development/mulga/spinifex /media/spinifex-deploy/spinifex-source/
```
## Step 3. Verify Package Integrity
Sign and verify packages before transferring to the target:
```bash
gpg --import /media/spinifex-deploy/mulga-signing-key.asc
gpg --verify spinifex-v1.0.tar.gz.sig spinifex-v1.0.tar.gz
sha256sum -c /media/spinifex-deploy/checksums.sha256
```
## Step 4. Install on the Target Server
Mount the USB and install:
```bash
sudo mount /dev/sdb1 /mnt/usb
sudo dpkg -i /mnt/usb/apt-packages/*.deb
cp -r /mnt/usb/spinifex-source ~/Development/mulga/spinifex
cd ~/Development/mulga/spinifex && make build
```
Then follow the [Source Install](/docs/install-source) guide from Step 3 (Setup OVN) onwards.
## Troubleshooting
### Missing Dependencies After dpkg Install
Some packages may have unresolved dependencies. Fix with:
```bash
sudo dpkg -i /mnt/usb/apt-packages/*.deb
sudo apt-get install -f
```
The `-f` flag tells apt to fix broken dependencies using what's available locally.
### GPG Verification Fails
The signing key may not match or the download was corrupted. Re-import the key and verify:
```bash
gpg --import /media/spinifex-deploy/mulga-signing-key.asc
gpg --verify spinifex-v1.0.tar.gz.sig spinifex-v1.0.tar.gz
```
If verification still fails, re-download the package on the connected machine and compare checksums:
```bash
sha256sum spinifex-v1.0.tar.gz
```
### Go Module Cache Errors
The Go module cache may have been incompletely copied. On the connected machine, re-export the full cache:
```bash
cp -r ~/go/pkg/mod/cache/ /media/spinifex-deploy/go-cache/
```
On the target, restore it:
```bash
mkdir -p ~/go/pkg/mod/
cp -r /mnt/usb/go-cache ~/go/pkg/mod/cache
```
+177
View File
@@ -0,0 +1,177 @@
---
title: "Multi-Node Install"
description: "Deploy Spinifex across multiple servers to create an availability zone with high availability, data durability, and fault tolerance."
category: "Getting Started"
tags:
- install
- multi node
- cluster
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "Predastore (S3)"
url: "https://github.com/mulgadc/predastore"
- title: "Viperblock (EBS)"
url: "https://github.com/mulgadc/viperblock"
---
# Multi-Node Installation
> Deploy Spinifex across multiple servers to create an availability zone.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for peers to join, then distributes credentials, CA certificates, and configuration.
**Network requirements:**
- Minimum 1 NIC per server (2 recommended for production)
- UDP port 6081 open between hosts (Geneve tunnels)
- TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
## Instructions
## Step 1. Install Spinifex on Each Server
```bash
curl -fsSL https://install.mulgadc.com | bash
```
## Step 2. Set Node IP Variables
On **each server**, export the management IPs for all nodes:
```bash
export SPINIFEX_NODE1=192.168.1.10
export SPINIFEX_NODE2=192.168.1.11
export SPINIFEX_NODE3=192.168.1.12
export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a
```
## Step 3. Setup OVN Networking
Server 1 runs OVN central and must be set up first. If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it. Otherwise use `--wan-bridge=br-wan --wan-iface=eth1` (dedicated WAN NIC) or `--macvlan --wan-iface=eth0` (single-NIC).
**Server 1:**
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--encap-ip=$SPINIFEX_NODE1
```
**Server 2** (after server 1 is ready):
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
--ovn-remote=tcp:$SPINIFEX_NODE1:6642 \
--encap-ip=$SPINIFEX_NODE2
```
**Server 3** (after server 1 is ready):
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
--ovn-remote=tcp:$SPINIFEX_NODE1:6642 \
--encap-ip=$SPINIFEX_NODE3
```
Verify all 3 chassis registered:
```bash
sudo ovn-sbctl show
```
## Step 4. Form the Cluster
Run init and join concurrently — init blocks until all nodes join.
**Server 1 — Initialize:**
```bash
sudo spx admin init \
--node node1 --nodes 3 \
--bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
--port 4432 --region $AWS_REGION --az $AWS_AZ
```
Save the admin credentials — they will not be shown again.
**Server 2 — Join** (while init is running):
```bash
sudo spx admin join \
--node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
--host $SPINIFEX_NODE1:4432 --region $AWS_REGION --az $AWS_AZ
```
**Server 3 — Join** (while init is running):
```bash
sudo spx admin join \
--node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
--host $SPINIFEX_NODE1:4432 --region $AWS_REGION --az $AWS_AZ
```
## Step 5. Start Services
On **all servers**:
```bash
sudo systemctl start spinifex.target
```
## Step 6. Verify
From any node:
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
If this returns a list of available instance types, your cluster is working.
**Congratulations! Your Spinifex cluster is installed.**
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
## Troubleshooting
### Nodes Not Joining
The init command must still be running when join executes. If init exited, re-run with `--force`.
```bash
curl -sk https://$SPINIFEX_NODE1:4432/health
```
### OVN Chassis Not Registering
```bash
sudo ovn-sbctl show
sudo ss -tlnp | grep 6642
```
### CA Certificate Not Trusted
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
### Cross-Host VMs Cannot Communicate
```bash
sudo ovs-vsctl show | grep -i geneve
sudo ss -ulnp | grep 6081
```
+126
View File
@@ -0,0 +1,126 @@
---
title: "Source Install"
description: "Build Spinifex from source for development, custom builds, or contributing."
category: "Getting Started"
tags:
- install
- source
- development
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "Go Downloads"
url: "https://go.dev/dl/"
---
# Source Installation
> Build Spinifex from source for development, custom builds, or contributing.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
This guide builds Spinifex from source. For production deployments, the [binary installer](/docs/install) is recommended.
**Requirements:**
- Ubuntu 22.04+ / Debian 12+
- Go 1.26.2+
- GCC, make, pkg-config
- QEMU/KVM, OVN/Open vSwitch, AWS CLI v2, dhcpcd-base
## Instructions
## Step 1. Install Dependencies
```bash
mkdir -p ~/Development/mulga && cd ~/Development/mulga
git clone https://github.com/mulgadc/spinifex.git
sudo make -C spinifex quickinstall
export PATH=$PATH:/usr/local/go/bin
```
## Step 2. Clone and Build
```bash
cd spinifex
./scripts/clone-deps.sh
./scripts/dev-setup.sh
```
Confirm `./bin/spx` exists.
## Step 3. Setup OVN
If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it:
```bash
./scripts/setup-ovn.sh --management
```
If your WAN is a physical NIC, choose one:
```bash
# Dedicated WAN NIC (not your SSH connection):
./scripts/setup-ovn.sh --management --wan-bridge=br-wan --wan-iface=eth1
# Single-NIC host (SSH-safe macvlan):
./scripts/setup-ovn.sh --management --macvlan --wan-iface=enp0s3
```
## Step 4. Development initialisation
```bash
./scripts/dev-install.sh
```
Note, this will bootstrap a single node development environment
## Step 5. Start services
```bash
sudo systemctl status spinifex.target
```
Start all services
## Step 6. Verify Installation
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
If this returns a list of available instance types, your installation is working.
**Congratulations! Spinifex is installed from source.**
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
## Troubleshooting
### Go Not Found in PATH
```bash
export PATH=$PATH:/usr/local/go/bin
```
### ./bin/spx Missing After Build
```bash
./scripts/dev-setup.sh
```
### CA Certificate Not Trusted
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
+132
View File
@@ -0,0 +1,132 @@
---
title: "Single-Node Install"
description: "Install Spinifex on a single server using the binary installer."
category: "Getting Started"
tags:
- install
- single node
- quickstart
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "Predastore (S3)"
url: "https://github.com/mulgadc/predastore"
- title: "Viperblock (EBS)"
url: "https://github.com/mulgadc/viperblock"
---
# Installing Spinifex
> Install Spinifex on a single server using the binary installer.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex is an open-source infrastructure platform that brings core AWS services — EC2, VPC, EBS, and S3 — to bare-metal, edge, and on-prem environments.
This guide installs Spinifex on a single server using the binary installer. For multi-server clusters, see [Multi-Node Install](/docs/install-multi-node). To build from source, see [Source Install](/docs/install-source).
**Supported Operating Systems:**
- Ubuntu 22.04 / 24.04 / 25.10
- Debian 12 / 13
**What Gets Installed:**
- Spinifex daemon and CLI
- QEMU/KVM for virtual machine management
- OVN/Open vSwitch for VPC networking
- Predastore (S3-compatible object storage)
- Viperblock (EBS-compatible block storage)
- AWS CLI v2
## Instructions
## Step 1. Install Spinifex
```bash
curl -fsSL https://install.mulgadc.com | bash
```
The installer downloads the Spinifex binary and bootstraps all dependencies (QEMU, OVN/OVS, AWS CLI).
## Step 2. Setup OVN Networking
If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it:
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
```
If your WAN is a physical NIC, choose one:
```bash
# Dedicated WAN NIC (not your SSH connection):
sudo /usr/local/share/spinifex/setup-ovn.sh --management --wan-bridge=br-wan --wan-iface=eth1
```
**Separating VPC tunnel traffic from WAN:** If you want Geneve tunnel traffic (inter-VM east-west for VPC traffic) to use a dedicated interface instead of the WAN IP, add `--encap-ip=<IP>` to specify the tunnel endpoint address:
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management --encap-ip=10.0.0.1
```
This is recommended for production and required in multi-node deployments. See [Multi-Node Install](/docs/install-multi-node) for details.
## Step 3. Initialize
```bash
sudo spx admin init --node node1 --nodes 1
```
This auto-detects your network topology, generates configuration and TLS certificates, installs the CA into the system trust store, and configures AWS CLI credentials. Save the admin credentials printed during init — they will not be shown again.
## Step 4. Start Services
```bash
sudo systemctl start spinifex.target
```
## Step 5. Verify
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
If this returns a list of available instance types, your installation is working.
**Congratulations! Spinifex is installed.**
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
## Troubleshooting
### spx Command Not Found
```bash
export PATH=$PATH:/usr/local/bin
```
### CA Certificate Not Trusted
`sudo spx admin init` installs the CA automatically. If you need to re-install it manually:
```bash
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
### OVN Services Not Starting
```bash
sudo systemctl is-active ovn-controller
journalctl -u ovn-controller --no-pager -n 20
```
+173
View File
@@ -0,0 +1,173 @@
---
title: "Launching Instances"
description: "Launch, manage, and connect to EC2-compatible virtual machines on Spinifex."
category: "Compute"
tags:
- ec2
- instances
- vm
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "AWS EC2 CLI"
url: "https://docs.aws.amazon.com/cli/latest/reference/ec2/"
---
# Launching Instances
> Launch, manage, and connect to EC2-compatible virtual machines on Spinifex.
## Table of Contents
- [Overview](#overview)
- [Launch](#launch)
- [Manage](#manage)
- [Reboot](#reboot)
- [Modify Instance Attributes](#modify-instance-attributes)
- [Console Output](#console-output)
- [Instance Types](#instance-types)
- [SSH (Development)](#ssh-development)
- [Troubleshooting](#troubleshooting)
- [Instance Fails to Boot](#instance-fails-to-boot)
- [Cannot SSH Into Instance](#cannot-ssh-into-instance)
---
## Overview
Spinifex provides EC2-compatible VM management built on QEMU/KVM. Instances support cloud-init, SSH key injection, VPC networking, and standard AWS lifecycle operations.
**Supported operations:**
- `run-instances` — Launch new VMs
- `describe-instances` — Query state
- `stop-instances` / `start-instances` — Lifecycle
- `reboot-instances` — In-place restart (QMP reset)
- `terminate-instances` — Permanent removal
- `modify-instance-attribute` — Change type, user data, or source/dest check
- `get-console-output` — Retrieve serial console log
- `describe-instance-types` — List available instance types
## Instructions
## Prerequisites
Ensure the AWS profile is set:
```bash
export AWS_PROFILE=spinifex
```
## Launch
```bash
aws ec2 run-instances \
--image-id $SPINIFEX_AMI \
--instance-type t3.small \
--key-name spinifex-key
export INSTANCE_ID="i-XXX"
```
## Manage
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
aws ec2 stop-instances --instance-ids $INSTANCE_ID
aws ec2 start-instances --instance-ids $INSTANCE_ID
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
aws ec2 reboot-instances --instance-ids $INSTANCE_ID
```
## Modify Instance Attributes
Change instance type, user data, or source/dest check. Instance type and user data require the instance to be **stopped** first.
### Change Instance Type
```bash
aws ec2 stop-instances --instance-ids $INSTANCE_ID
aws ec2 modify-instance-attribute \
--instance-id $INSTANCE_ID \
--instance-type t3.medium
aws ec2 start-instances --instance-ids $INSTANCE_ID
```
## Console Output
Retrieve the serial console log for a running instance. Output is base64-encoded.
```bash
aws ec2 get-console-output --instance-id $INSTANCE_ID
```
Decode the output:
```bash
aws ec2 get-console-output --instance-id $INSTANCE_ID \
--query 'Output' --output text | base64 -d
```
## Instance Types
List instance types available on the current host. The catalog is generated from the host CPU (Intel, AMD, or ARM) and includes burstable (t-family), general purpose (m-family), compute optimised (c-family), and memory optimised (r-family) types.
```bash
aws ec2 describe-instance-types
```
Filter to a specific type:
```bash
aws ec2 describe-instance-types --instance-types t3.micro t3.small
```
Show capacity (how many of each type can still be launched):
```bash
aws ec2 describe-instance-types \
--filters Name=capacity,Values=true
```
## SSH (Development)
In development mode, find the QEMU port forward and connect via localhost:
```bash
ps auxw | grep $INSTANCE_ID
ssh -i ~/.ssh/spinifex-key ec2-user@127.0.0.1 -p <port>
```
For production SSH via public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster).
## Troubleshooting
### Instance Fails to Boot
Check QEMU logs for the instance and verify the AMI architecture matches your host:
```bash
cat ~/spinifex/logs/spinifex.log
aws ec2 describe-images --image-ids $SPINIFEX_AMI
```
If the AMI is for a different architecture (e.g. arm64 on an x86_64 host), import the correct image:
```bash
spx admin images list
spx admin images import --name debian-12-x86_64
```
### Cannot SSH Into Instance
cloud-init needs time to configure the instance after boot. Wait 30-60 seconds and retry.
Verify the SSH key was specified correctly when launching:
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```
Check the `KeyName` field matches the key you're using to connect.
+108
View File
@@ -0,0 +1,108 @@
---
title: "Moving an AWS Workload to Mulga"
description: "Migrate existing AWS workloads to Spinifex using compatible APIs, SDKs, and Terraform."
category: "Migration"
tags:
- migration
- aws
- terraform
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "AWS CLI Reference"
url: "https://docs.aws.amazon.com/cli/latest/reference/"
- title: "Terraform AWS Provider"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
---
# Moving an AWS Workload to Mulga
> Migrate existing AWS workloads to Spinifex using compatible APIs, SDKs, and Terraform.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex provides drop-in compatibility with AWS APIs, making it possible to migrate existing workloads with minimal changes.
**Supported Services:** VPC, EC2, EBS, S3, IAM
**Compatible Tools:** AWS CLI, AWS SDKs, Terraform, any S3-compatible client
## Instructions
## Prerequisites
Ensure the AWS profile is set:
```bash
export AWS_PROFILE=spinifex
```
## Configure AWS CLI
```bash
aws ec2 describe-instances
aws s3 ls
```
## Terraform
```hcl
provider "aws" {
region = "ap-southeast-2"
access_key = "your-spinifex-access-key"
secret_key = "your-spinifex-secret-key"
endpoints {
ec2 = "https://localhost:9999"
s3 = "https://localhost:8443"
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
}
```
## Hybrid S3 Sync
```bash
aws s3 sync s3://local-bucket/ s3://cloud-bucket/ --source-region spinifex --region us-east-1
```
## Troubleshooting
## Terraform Provider Errors
Ensure all three skip flags are set in your provider configuration:
```hcl
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
```
Without these, Terraform will try to validate credentials and metadata against real AWS endpoints.
## S3 Signature Errors
Spinifex uses AWS Signature V4. Ensure your AWS CLI is version 2.0 or higher:
```bash
aws --version
```
If using an older version, upgrade:
```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install --update
```
+121
View File
@@ -0,0 +1,121 @@
---
title: "Placement Groups"
description: "Create and manage spread and cluster placement groups for hardware-level instance placement control."
category: "Compute"
tags:
- ec2
- placement
- spread
- cluster
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "AWS Placement Groups"
url: "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html"
---
# Placement Groups
> Create and manage spread and cluster placement groups for hardware-level instance placement control.
## Table of Contents
- [Overview](#overview)
- [Create](#create)
- [Launch Into a Placement Group](#launch-into-a-placement-group)
- [Describe](#describe)
- [Delete](#delete)
- [Troubleshooting](#troubleshooting)
- [InsufficientInstanceCapacity Error](#insufficientinstancecapacity-error)
- [Cannot Delete Placement Group](#cannot-delete-placement-group)
---
## Overview
Placement groups control how Spinifex distributes instances across physical hosts. Two strategies are supported:
- **Spread** — One instance per physical host, maximising fault isolation
- **Cluster** — All instances on the same host, minimising latency
The `partition` strategy is not supported.
**Supported operations:**
- `create-placement-group` — Create a new group
- `describe-placement-groups` — Query groups with optional filters
- `delete-placement-group` — Remove an empty group
## Instructions
## Prerequisites
Ensure the AWS profile is set:
```bash
export AWS_PROFILE=spinifex
```
## Create
```bash
aws ec2 create-placement-group \
--group-name my-spread-group \
--strategy spread
```
```bash
aws ec2 create-placement-group \
--group-name my-cluster-group \
--strategy cluster
```
## Launch Into a Placement Group
```bash
aws ec2 run-instances \
--image-id $SPINIFEX_AMI \
--instance-type t3.small \
--key-name spinifex-key \
--placement GroupName=my-spread-group \
--count 3
```
## Describe
```bash
aws ec2 describe-placement-groups
aws ec2 describe-placement-groups --group-names my-spread-group
aws ec2 describe-placement-groups --group-ids pg-abc123
aws ec2 describe-placement-groups \
--filters Name=strategy,Values=spread
```
## Delete
The group must have no running instances before it can be deleted:
```bash
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
aws ec2 delete-placement-group --group-name my-spread-group
```
## Troubleshooting
### InsufficientInstanceCapacity Error
Not enough distinct physical hosts for a spread launch. Reduce `--count` or terminate existing instances to free host slots:
```bash
spx admin nodes list
```
### Cannot Delete Placement Group
The group still has running instances. Terminate them first:
```bash
aws ec2 describe-instances \
--filters Name=placement-group-name,Values=my-spread-group
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
```
+389
View File
@@ -0,0 +1,389 @@
---
title: "Setting Up Your Cluster"
description: "Import an AMI, create a VPC with a public subnet, and launch your first EC2 instance on Spinifex."
category: "Getting Started"
tags:
- setup
- vpc
- ec2
- quickstart
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "AWS EC2 CLI Reference"
url: "https://docs.aws.amazon.com/cli/latest/reference/ec2/"
---
# Setting Up Your Cluster
> Import an AMI, create a VPC with a public subnet, and launch your first instance.
## Table of Contents
- [1. Import an AMI](#1-import-an-ami)
- [2. Create an SSH Key](#2-create-an-ssh-key)
- [3. Create a VPC and Public Subnet](#3-create-a-vpc-and-public-subnet)
- [4. Launch an Instance](#4-launch-an-instance)
- [5. Connect via SSH](#5-connect-via-ssh)
- [6. Managing Instances](#6-managing-instances)
- [7. Launching the Web UI](#7-launching-the-web-ui)
- [Troubleshooting](#troubleshooting)
---
## Overview
This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:
- [Binary Install (Single Node)](/docs/install)
- [Multi-Node Install](/docs/install-multi-node)
- [Source Install](/docs/install-source)
## Instructions
## Prerequisites
Ensure the AWS profile is set:
```bash
export AWS_PROFILE=spinifex
```
## 1. Import an AMI
List available images and import one matching your architecture:
```bash
spx admin images list
```
```
NAME | DISTRO | VERSION | ARCH | BOOT
debian-12-arm64 | debian | 12 | arm64 | bios
debian-12-x86_64 | debian | 12 | x86_64 | bios
ubuntu-24.04-arm64 | ubuntu | 24.04 | arm64 | bios
ubuntu-24.04-x86_64 | ubuntu | 24.04 | x86_64 | bios
```
Import an image:
```bash
spx admin images import --name ubuntu-24.04-x86_64
```
Or import a local image file:
```bash
spx admin images import --file ~/images/ubuntu-24.04.img --distro ubuntu --version 24.04 --arch x86_64
```
Verify the import and note the AMI ID:
```bash
aws ec2 describe-images
```
## 2. Create an SSH Key
### Option A: Import an existing key
```bash
aws ec2 import-key-pair \
--key-name "spinifex-key" \
--public-key-material fileb://~/.ssh/id_rsa.pub
```
### Option B: Create a new key pair
```bash
aws ec2 create-key-pair --key-name spinifex-key \
| jq -r '.KeyMaterial | rtrimstr("\n")' > ~/.ssh/spinifex-key
chmod 600 ~/.ssh/spinifex-key
ssh-keygen -y -f ~/.ssh/spinifex-key > ~/.ssh/spinifex-key.pub
```
Verify:
```bash
aws ec2 describe-key-pairs
```
## 3. Create a VPC and Public Subnet
### Create a VPC
```bash
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
--query 'Vpc.VpcId' --output text)
echo "VPC: $VPC_ID"
```
### Create an Internet Gateway
An Internet Gateway enables instances in public subnets to reach the internet and be reachable from the LAN/WAN.
```bash
IGW_ID=$(aws ec2 create-internet-gateway \
--query 'InternetGateway.InternetGatewayId' --output text)
aws ec2 attach-internet-gateway \
--internet-gateway-id $IGW_ID \
--vpc-id $VPC_ID
echo "IGW: $IGW_ID (attached to $VPC_ID)"
```
### Create a Public Subnet
A public subnet auto-assigns a routable IP to each instance, making it directly reachable from your network.
```bash
SUBNET_ID=$(aws ec2 create-subnet \
--vpc-id $VPC_ID \
--cidr-block 10.200.1.0/24 \
--query 'Subnet.SubnetId' --output text)
# Enable auto-assign public IP
aws ec2 modify-subnet-attribute \
--subnet-id $SUBNET_ID \
--map-public-ip-on-launch
echo "Subnet: $SUBNET_ID (public)"
```
### Verify
```bash
aws ec2 describe-vpcs --vpc-ids $VPC_ID
aws ec2 describe-subnets --subnet-ids $SUBNET_ID
```
## 4. Launch an Instance
Query available instance types for your hardware:
```bash
aws ec2 describe-instance-types
```
Launch an instance in the public subnet, note this will select an instance type with 2 VCPU and 1024MB of memory.
> **Note:** Replace `AMI_NAME` with the previously imported image above with the `ami-` prefix.
```bash
AMI_NAME="ami-ubuntu-24.04-x86_64"
AMI_ID=$(aws ec2 describe-images \
--filters "Name=name,Values=$AMI_NAME" \
--query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
--output text)
INSTANCE_TYPE=$(aws ec2 describe-instance-types \
--query "sort_by(InstanceTypes[?VCpuInfo.DefaultVCpus==\`2\` && MemoryInfo.SizeInMiB>=\`1024\`], &MemoryInfo.SizeInMiB)[0].InstanceType" \
--output text)
INSTANCE_ID=$(aws ec2 run-instances \
--image-id $AMI_ID \
--instance-type $INSTANCE_TYPE \
--key-name spinifex-key \
--subnet-id $SUBNET_ID \
--count 1 \
--query 'Instances[0].InstanceId' --output text)
echo "Instance: $INSTANCE_ID"
```
Wait for the instance to reach `running` state:
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID \
--query 'Reservations[0].Instances[0].[State.Name, PrivateIpAddress, PublicIpAddress]' \
--output text
```
Expected output:
```
running 10.200.1.4 192.168.1.155
```
The instance has both a private IP (VPC overlay) and a public IP (from your external pool, routable on your network).
## 5. Connect via SSH
SSH directly to the instance's public IP:
```bash
PUBLIC_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
ssh -i ~/.ssh/spinifex-key ec2-user@$PUBLIC_IP
```
> **Note:** cloud-init takes 30-60 seconds to configure the instance. If SSH is refused, wait and retry.
Once connected, verify the instance has internet access:
```bash
curl -s http://ifconfig.me
```
This should return the instance's public IP or the gateway's SNAT address.
## 6. Managing Instances
### Stop
```bash
aws ec2 stop-instances --instance-ids $INSTANCE_ID
```
### Start
```bash
aws ec2 start-instances --instance-ids $INSTANCE_ID
```
### Terminate
```bash
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
```
### Console Output
View the instance's serial console log (useful for debugging boot issues):
```bash
aws ec2 get-console-output --instance-id $INSTANCE_ID \
--query 'Output' --output text
```
### Multi-Node: Check Instance Placement
On a multi-node cluster, instances are distributed across nodes:
```bash
spx get vms
```
## 7. Launching the Web UI
Spinifex ships with a built-in web console — an alternative to the AWS CLI, SDKs, and Terraform, analogous to the AWS Management Console. Every action in the UI is a standard AWS SigV4 API call, so the same IAM policies and audit behaviour apply.
![Spinifex Web UI](../spinifex-ui.jpg)
### Open the Console
The UI is served by each node on port `3000` over TLS. Replace `YOUR_NODE_IP` with the address of the node you installed Spinifex on (or `localhost` if you're on the node itself):
```
https://YOUR_NODE_IP:3000
```
### Trust the Self-Signed Certificate (required)
On first load, your browser will show a TLS warning — Spinifex generates a self-signed certificate at install time. This is expected.
1. Accept the warning to reach the login page (exact wording varies by browser — e.g. Chrome: *Advanced → Proceed to ...*, Firefox: *Advanced → Accept the Risk and Continue*).
2. On the login page, click **Download Certificate** and save `spinifex-ca.pem` to your machine.
3. Install the certificate as a **trusted root** on your workstation:
- **macOS:** open `spinifex-ca.pem` in Keychain Access → *System* keychain → set *Trust* to **Always Trust**.
- **Linux:** `sudo cp spinifex-ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt && sudo update-ca-certificates`
- **Windows:** double-click the file → *Install Certificate* → *Local Machine* → *Trusted Root Certification Authorities*.
- **Browser-only (Firefox):** *Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import* and tick *Trust this CA to identify websites*.
4. Restart your browser and reload `https://YOUR_NODE_IP:3000`. The padlock should now show a valid certificate.
> **Why this is required:** the UI logs in by reading your AWS credentials through a trusted TLS channel. Browsers refuse to send credentials over an untrusted connection, so the certificate must be installed as trusted — temporary "Proceed anyway" exceptions won't work for login.
### Log In with AWS Credentials
The console authenticates against the AWS credentials in `~/.aws/credentials` on the node where Spinifex is installed:
```ini
[spinifex]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
```
At the Spinifex login screen, paste the **Access Key ID** and **Secret Access Key** from the `[spinifex]` profile (or whichever profile maps to the IAM user or role you want to use). Additional users and policies can be managed through the UI or via `aws iam` commands — see [IAM Users and Policies](/docs/iam-users-and-policies).
Once logged in, you have browser-based access to every Spinifex feature: launch and manage instances, attach EBS volumes, browse S3 buckets, configure VPCs and security groups, and manage IAM users and keys — all backed by the same AWS-compatible control plane the CLI uses.
## Additional Options
### Private Subnets (No Public IP)
Create a subnet without `--map-public-ip-on-launch`:
```bash
PRIVATE_SUBNET=$(aws ec2 create-subnet \
--vpc-id $VPC_ID \
--cidr-block 10.200.2.0/24 \
--query 'Subnet.SubnetId' --output text)
```
Instances in private subnets get a private IP only. They can still reach the internet via SNAT through the VPC router, but are not directly reachable from your network.
### Multiple Accounts
Create isolated accounts with their own resources:
```bash
spx admin account create --name myteam
export AWS_PROFILE=spinifex-myteam
```
## Troubleshooting
### Instance Stuck in Pending
```bash
cat ~/spinifex/logs/spinifex.log
aws ec2 describe-images
```
### SSH Connection Refused
cloud-init needs 30-60 seconds after boot. Check instance state:
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```
### No Public IP Assigned
Verify the subnet has `MapPublicIpOnLaunch` enabled:
```bash
aws ec2 describe-subnets --subnet-ids $SUBNET_ID
```
If `MapPublicIpOnLaunch` is false:
```bash
aws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID --map-public-ip-on-launch
```
Also verify an Internet Gateway is attached to the VPC:
```bash
aws ec2 describe-internet-gateways
```
### Instance Has No Internet Access
Check the VPC router's NAT rules (from the host):
```bash
sudo ovn-nbctl lr-nat-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)
```
Verify the default route exists:
```bash
sudo ovn-nbctl lr-route-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)
```
+96
View File
@@ -0,0 +1,96 @@
---
title: "Spinifex Admin CLI"
description: "Complete reference for the Spinifex administration CLI. Manage accounts, nodes, VMs, and services."
category: "Administration"
tags:
- cli
- admin
- reference
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
---
# Spinifex Admin CLI
> Complete reference for the Spinifex administration CLI.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
The `spx` binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control.
**Binary location:** `/usr/local/bin/spx` (binary install) or `~/Development/mulga/spinifex/bin/spx` (source install)
All services in the Spinifex platform are managed through this single binary.
## Instructions
## Account Management
```bash
spx admin account create --name myteam
export AWS_PROFILE=spinifex-myteam
```
## Node Management
```bash
spx get nodes
```
```
NAME STATUS IP REGION AZ UPTIME VMs
node1 Ready 127.0.0.1 ap-southeast-2 ap-southeast-2a 2m 0
node2 Ready 127.0.0.2 ap-southeast-2 ap-southeast-2a 2m 0
node3 Ready 127.0.0.3 ap-southeast-2 ap-southeast-2a 2m 0
```
## Monitor Resources
```bash
spx top nodes
```
## Image Management
```bash
spx admin images list
spx admin images import --name debian-12-arm64
```
## Cluster Shutdown
```bash
spx admin cluster shutdown
```
## Troubleshooting
### Permission Denied Running Spinifex
The binary may not be executable. Fix permissions:
```bash
chmod +x /usr/local/bin/spx
```
If you get permission errors during operations, ensure you're running with appropriate privileges. Some OVN and networking commands require `sudo`.
### Services Fail to Start
Check the daemon logs for specific errors:
```bash
ls ~/spinifex/logs/
cat ~/spinifex/logs/spinifex.log
```
Common causes include port conflicts, missing OVN configuration, or untrusted CA certificates.
Binary file not shown.

After

Width:  |  Height:  |  Size: 151 KiB

@@ -0,0 +1,155 @@
---
title: "Bastion with Private Subnet"
description: "Deploy a bastion host to access an isolated private compute instance with no internet connectivity."
category: "Terraform Workbooks"
tags:
- terraform
- bastion
- vpc
- security
- private subnet
- workbook
resources:
- title: "Terraform AWS Provider"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "OpenTofu"
url: "https://opentofu.org/"
---
# Terraform: Bastion with Private Subnet
> Deploy a bastion host to access an isolated private compute instance with no internet connectivity.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Deploy a VPC with public and private subnets where the private subnet has no internet access. A bastion host in the public subnet is the only way to reach instances in the private subnet. This pattern is used for sensitive workloads that must remain isolated from the internet — the private instance cannot make or receive any connections outside the VPC.
**Architecture:**
<p align="center">
<img src="../../../.github/assets/diagrams/tf-bastion.svg" alt="Bastion architecture — WAN reaches bastion in public subnet, SSH hop to app server in private subnet" width="900">
</p>
**What you'll learn:**
- Creating public and private subnets in a VPC
- Isolating compute instances from the internet using route tables and security groups
- Using a bastion host as the sole access point to private resources
- SSH hopping through a bastion to reach private instances
**Use cases:**
- Sensitive data processing that must not have internet egress
- Internal services that should only be reachable within the VPC
- Compliance workloads requiring network isolation
**Prerequisites:**
- Spinifex installed and running (see [Installing Spinifex](/docs/install))
- A Debian 12 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed
- `AWS_PROFILE=spinifex` configured
## Instructions
### Step 1. Get the Template
Clone the Terraform examples from the Spinifex repository:
```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/bastion-private-subnet
```
Or create a `main.tf` file and paste the full configuration below.
<!-- INCLUDE: main.tf lang:hcl -->
### Step 2. Deploy
```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```
### Step 3. Connect
> **Note:** EC2 instances can take 30+ seconds to boot after apply. If SSH is unreachable, wait and retry.
SSH into the bastion:
```bash
ssh -i bastion-demo.pem ec2-user@<bastion_public_ip>
```
From the bastion, SSH to the private instance. The key is pre-installed at `~/.ssh/bastion-demo.pem` via cloud-init:
```bash
ssh -i ~/.ssh/bastion-demo.pem ec2-user@<private_ip>
```
### Step 4. Verify Isolation
From the private instance, confirm there is no internet connectivity:
```bash
# This should time out — the private instance has no route to the internet
curl --connect-timeout 5 https://deb.debian.org || echo "No internet access (expected)"
```
The private instance can only communicate within the VPC (`10.20.0.0/16`). Its security group restricts egress to VPC-internal traffic only.
### Clean Up
```bash
tofu destroy
```
## Troubleshooting
### Cannot SSH to Private Instance
The private instance has no public IP — it is only reachable from the bastion. SSH to the bastion first, then use the pre-installed key to hop:
```bash
ssh -i bastion-demo.pem ec2-user@<bastion_ip>
ssh -i ~/.ssh/bastion-demo.pem ec2-user@<private_ip>
```
If the key is missing on the bastion, check that cloud-init completed successfully:
```bash
sudo journalctl -u cloud-init --no-pager
ls -la ~/.ssh/bastion-demo.pem
```
### Private Instance Can Reach the Internet
If the private instance unexpectedly has internet access, check that its route table has no default route:
```bash
aws ec2 describe-route-tables --profile spinifex
```
The private route table should have no `0.0.0.0/0` route. Also verify the private security group egress is restricted to the VPC CIDR (`10.20.0.0/16`).
### AMI Not Found
Ensure you have imported a Debian 12 image:
```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```
@@ -0,0 +1,346 @@
# Example 2: Bastion Host with Private Subnet
#
# Deploys a VPC with both public and private subnets. A bastion host in the
# public subnet provides SSH access to an isolated instance in the private
# subnet. The private instance has no internet connectivity — ideal for
# sensitive workloads that must remain air-gapped from the internet.
#
# Architecture:
#
# WAN ──SSH──▶ Bastion (public subnet)
# │
# ▼ SSH (private IP)
# App Server (private subnet, no internet)
#
# Usage:
# export AWS_PROFILE=spinifex
# tofu init && tofu apply
#
# After apply:
# # SSH to the bastion
# ssh -i bastion-demo.pem ec2-user@<bastion_ip>
#
# # From the bastion, SSH to the private instance
# # (the key is pre-installed at ~/.ssh/bastion-demo.pem via cloud-init)
# ssh -i ~/.ssh/bastion-demo.pem ec2-user@<private_ip>
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
}
}
}
# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------
variable "region" {
type = string
default = "ap-southeast-2"
}
variable "spinifex_endpoint" {
type = string
default = "https://localhost:9999"
description = "Spinifex AWS gateway endpoint"
}
# ---------------------------------------------------------------------------
# Provider
# ---------------------------------------------------------------------------
provider "aws" {
region = var.region
endpoints {
ec2 = var.spinifex_endpoint
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------
data "aws_availability_zones" "available" {
state = "available"
}
data "aws_ami" "debian12" {
most_recent = true
owners = ["000000000000"]
filter {
name = "name"
values = ["*debian-12*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
filter {
name = "root-device-type"
values = ["ebs"]
}
}
# ---------------------------------------------------------------------------
# SSH Key Pair (shared by bastion and private instances)
# ---------------------------------------------------------------------------
resource "tls_private_key" "bastion" {
algorithm = "ED25519"
}
resource "aws_key_pair" "bastion" {
key_name = "bastion-demo"
public_key = tls_private_key.bastion.public_key_openssh
}
resource "local_file" "bastion_pem" {
filename = "${path.module}/bastion-demo.pem"
content = tls_private_key.bastion.private_key_openssh
file_permission = "0600"
}
# ---------------------------------------------------------------------------
# VPC
# ---------------------------------------------------------------------------
resource "aws_vpc" "main" {
cidr_block = "10.20.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "bastion-demo-vpc"
}
}
# ---------------------------------------------------------------------------
# Internet Gateway — only the public subnet routes through this
# ---------------------------------------------------------------------------
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.main.id
tags = {
Name = "bastion-demo-igw"
}
}
# ---------------------------------------------------------------------------
# Public Subnet — bastion host lives here
# ---------------------------------------------------------------------------
resource "aws_subnet" "public" {
vpc_id = aws_vpc.main.id
cidr_block = "10.20.1.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
map_public_ip_on_launch = true
tags = {
Name = "bastion-demo-public"
}
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
tags = {
Name = "bastion-demo-public-rt"
}
}
resource "aws_route_table_association" "public" {
subnet_id = aws_subnet.public.id
route_table_id = aws_route_table.public.id
}
# ---------------------------------------------------------------------------
# Private Subnet — isolated instances live here (no public IPs, no internet)
# ---------------------------------------------------------------------------
resource "aws_subnet" "private" {
vpc_id = aws_vpc.main.id
cidr_block = "10.20.2.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
map_public_ip_on_launch = false
tags = {
Name = "bastion-demo-private"
}
}
# Private route table — no default route, no internet access
resource "aws_route_table" "private" {
vpc_id = aws_vpc.main.id
tags = {
Name = "bastion-demo-private-rt"
}
}
resource "aws_route_table_association" "private" {
subnet_id = aws_subnet.private.id
route_table_id = aws_route_table.private.id
}
# ---------------------------------------------------------------------------
# Security Groups
# ---------------------------------------------------------------------------
# Bastion: SSH from anywhere
resource "aws_security_group" "bastion" {
name = "bastion-demo-bastion-sg"
description = "Bastion: SSH from WAN"
vpc_id = aws_vpc.main.id
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
description = "All outbound"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "bastion-demo-bastion-sg"
}
}
# Private instances: SSH only from the bastion security group
resource "aws_security_group" "private" {
name = "bastion-demo-private-sg"
description = "Private: SSH from bastion only"
vpc_id = aws_vpc.main.id
ingress {
description = "SSH from bastion"
from_port = 22
to_port = 22
protocol = "tcp"
security_groups = [aws_security_group.bastion.id]
}
egress {
description = "VPC internal only"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["10.20.0.0/16"]
}
tags = {
Name = "bastion-demo-private-sg"
}
}
# ---------------------------------------------------------------------------
# Bastion Host (public subnet)
# ---------------------------------------------------------------------------
resource "aws_instance" "bastion" {
ami = data.aws_ami.debian12.id
instance_type = "t3.small"
subnet_id = aws_subnet.public.id
vpc_security_group_ids = [aws_security_group.bastion.id]
key_name = aws_key_pair.bastion.key_name
associate_public_ip_address = true
# Copy the SSH private key onto the bastion so you can hop to private instances
user_data_base64 = base64encode(<<-USERDATA
#!/bin/bash
set -euo pipefail
mkdir -p /home/ec2-user/.ssh
cat > /home/ec2-user/.ssh/bastion-demo.pem <<'KEY'
${tls_private_key.bastion.private_key_openssh}
KEY
chmod 600 /home/ec2-user/.ssh/bastion-demo.pem
chown -R ec2-user:ec2-user /home/ec2-user/.ssh
USERDATA
)
tags = {
Name = "bastion-demo-bastion"
}
}
# ---------------------------------------------------------------------------
# Private Instance (private subnet — no public IP, no internet)
# ---------------------------------------------------------------------------
resource "aws_instance" "private" {
ami = data.aws_ami.debian12.id
instance_type = "t3.small"
subnet_id = aws_subnet.private.id
vpc_security_group_ids = [aws_security_group.private.id]
key_name = aws_key_pair.bastion.key_name
tags = {
Name = "bastion-demo-private-app"
}
}
# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------
output "note" {
value = "EC2 instances can take 30+ seconds to boot after apply. If SSH is unreachable, wait and retry."
}
output "bastion_public_ip" {
value = aws_instance.bastion.public_ip
}
output "private_instance_ip" {
value = aws_instance.private.private_ip
}
output "ssh_to_bastion" {
description = "SSH to the bastion host"
value = "ssh -i bastion-demo.pem ec2-user@${aws_instance.bastion.public_ip}"
}
output "ssh_to_private_from_bastion" {
description = "From the bastion, SSH to the private instance (key is pre-installed via cloud-init)"
value = "ssh -i ~/.ssh/bastion-demo.pem ec2-user@${aws_instance.private.private_ip}"
}
+231
View File
@@ -0,0 +1,231 @@
---
title: "Nginx Web Server (Load Balanced)"
description: "Deploy a VPC with two private EC2 instances running Nginx, fronted by an internet-facing ALB using Terraform on Spinifex."
category: "Terraform Workbooks"
tags:
- terraform
- nginx
- ec2
- elbv2
- alb
- vpc
- workbook
resources:
- title: "Terraform AWS Provider"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "OpenTofu"
url: "https://opentofu.org/"
---
# Terraform: Nginx Web Servers with ALB
> Deploys two Nginx web server instances behind an Application Load Balancer on Spinifex.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Deploy two Nginx web servers behind an internet-facing Application Load Balancer on Spinifex using Terraform/OpenTofu. This workbook provisions a VPC with public and private subnets, an internet gateway and NAT Gateway, route tables, security group, SSH key pair, an application load balancer (ALB) and two EC2 instances with cloud-init user-data that installs and starts Nginx. Only the ALB is reachable from outside the VPC — the Nginx instances live in the **private subnets** and reach the internet only for cloud-init bootstrapping via the NAT Gateway.
<p align="center">
<img src="../../../.github/assets/diagrams/tf-nginx-alb.svg" alt="Nginx + ALB VPC — IGW, two public subnets with ALB ENIs and NAT GW, two private subnets hosting nginx" width="900">
</p>
**What you'll learn:**
- Configuring the AWS Terraform provider to target Spinifex
- Creating a VPC with public + private subnets, an IGW and a NAT Gateway
- Provisioning a multi-AZ internet-facing ALB fronting private workers
- Provisioning an EC2 instance with cloud-init user-data
- Generating SSH key pairs with the TLS provider
**What gets created**
| Resource | Name | Purpose |
|---|---|---|
| VPC | `nginx-alb-vpc` | Isolated network (10.20.0.0/16) |
| Public Subnets | `nginx-alb-public-a`, `nginx-alb-public-b` | Two AZs hosting the ALB and NAT Gateway |
| Private Subnets | `nginx-alb-private-a`, `nginx-alb-private-b` | Two AZs hosting the Nginx workers |
| Internet Gateway | `nginx-alb-igw` | Routes internet traffic for the public subnets |
| Elastic IP | `nginx-alb-nat-eip` | Public address for the NAT Gateway |
| NAT Gateway | `nginx-alb-nat` | Outbound internet for the private subnets (cloud-init apt bootstrap) |
| Security Group | `nginx-alb-sg` | Allows SSH (22) and HTTP (80) inbound |
| EC2 Instances | `nginx-alb-1`, `nginx-alb-2` | Debian 12 with Nginx via cloud-init (private subnets) |
| ALB | `nginx-alb` | Internet-facing Application Load Balancer on port 80 |
| Target Group | `nginx-alb-tg` | HTTP health-checked group for both instances |
| Listener | HTTP :80 | Forwards traffic to the target group |
**Prerequisites:**
- Spinifex installed and running (see [Installing Spinifex](/docs/install))
- A Debian 12 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed
- `AWS_PROFILE=spinifex` configured
## Instructions
### Step 1. Get the Template
Clone the Terraform examples from the Spinifex repository:
```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/nginx-alb
```
Or create a `main.tf` file and paste the full configuration below.
<!-- INCLUDE: main.tf lang:hcl -->
### Step 2. Install Load Balancer AMI
Next install the load balancer AMI images, which is used as the AMI disk image for launching the load-balancer and a requirement.
```bash
spx admin images import --name lb-alpine-3.21.6-x86_64
```
### Step 3. Deploy
```bash
export AWS_PROFILE=spinifex
tofu init
```
### Step 4. Specify instance and apply
Next, depending on your architecture and CPU/memory requirements you must specify an instance type to launch.
Either specify an instance type directly (e.g Intel)
```bash
# AMD instance
export TF_VAR_instance_type="t3a.small"
# Or, Intel
export TF_VAR_instance_type="t3.small"
```
Or alternatively, using the AWS CLI tool query your instance for available types (e.g Intel, AMD, ARM) that support 2 vCPUs and 1 GB RAM.
```bash
TF_VAR_instance_type=$(aws ec2 describe-instance-types \
--query "sort_by(InstanceTypes[?VCpuInfo.DefaultVCpus==\`2\` && MemoryInfo.SizeInMiB>=\`1024\`], &MemoryInfo.SizeInMiB)[0].InstanceType" \
--output text)
```
Next, apply and launch the template:
```bash
tofu apply
```
### Step 5. Verify
> **Note:** EC2 instances can take 30+ seconds to boot after apply, and the NAT Gateway must be `available` before cloud-init on the workers can reach the apt repository. If the ALB returns 5xx or HTTP is unreachable, wait and retry — the target group health checks need a moment to mark both instances healthy once Nginx has installed.
The ALB is internet-facing, but the DNS name Spinifex returns (`*.elb.spinifex.local`) will not resolve from your host. Fetch the ALB's public IP with the AWS CLI:
```bash
ALB_IP=$(aws elbv2 describe-load-balancers --names nginx-alb \
--query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
--output text)
echo "ALB public IP: $ALB_IP"
```
Then hit the ALB — successive requests should alternate between Server 1 and Server 2:
```bash
curl http://$ALB_IP
curl http://$ALB_IP
```
Open `http://$ALB_IP` in your browser and refresh to see the page alternate content served from each instance.
The Nginx instances themselves only have private IPs (see the `instance_1_private_ip` / `instance_2_private_ip` outputs) and are only reachable from inside the VPC — go through the ALB.
Check target health via AWS CLI:
```bash
TG_ARN=$(aws elbv2 describe-target-groups \
--query 'TargetGroups[0].TargetGroupArn' \
--output text)
echo "Target Group ARN: $TG_ARN"
aws elbv2 describe-target-health --target-group-arn $TG_ARN
```
### Cleanup
```bash
tofu destroy
```
## Troubleshooting
### AMI Not Found
Ensure you have imported a Debian 12 image. Check available AMIs:
```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```
If missing import:
```bash
spx admin images import --name debian-12-x86_64
```
### Provider Connection Refused
Verify Spinifex services are running:
```bash
sudo systemctl status spinifex.target
curl -k https://localhost:9999/
```
### ALB Returns 5xx / Targets Unhealthy
Give the instances a moment to finish cloud-init (Nginx has to install before it can answer health checks). Check target health:
```bash
TG_ARN=$(aws elbv2 describe-target-groups --names nginx-alb-tg \
--query 'TargetGroups[0].TargetGroupArn' --output text)
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
```
If targets stay unhealthy, verify the instances are running:
```bash
aws ec2 describe-instances --profile spinifex
```
If cloud-init on the workers never finished, confirm the NAT Gateway is `available` (the private subnets rely on it for outbound apt access):
```bash
aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'
```
### Nginx Not Responding
The Nginx instances have no public IP, so you can't SSH in directly from your host. If you need to inspect cloud-init logs, launch a small jump host in the same VPC or run commands via the Spinifex console, then:
```bash
ssh -i nginx-alb-demo.pem ec2-user@<instance_private_ip>
sudo journalctl -u cloud-init --no-pager
sudo systemctl status nginx
```
+523
View File
@@ -0,0 +1,523 @@
# Example: Nginx Web Servers with ALB on Spinifex
#
# Deploys a VPC with two public subnets (ALB + NAT Gateway) and two private
# subnets hosting Nginx EC2 instances. An internet-facing Application Load
# Balancer distributes HTTP traffic between the private workers, and the
# NAT Gateway gives them outbound internet access so cloud-init can install
# Nginx from the Debian apt repository.
#
# Demonstrates: VPC, public and private subnets, internet gateway, NAT
# Gateway with Elastic IP, route tables, security group, key pair, cloud-init
# user-data, EC2 instances, ALB, target group, and listener.
#
# Usage:
# cd spinifex/docs/terraform/nginx-alb
# export AWS_PROFILE=spinifex
# tofu init && tofu apply
#
# After apply, fetch the ALB's public IP (the *.elb.spinifex.local DNS
# name does not resolve from your host):
#
# aws elbv2 describe-load-balancers --names nginx-alb \
# --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
# --output text
#
# Then:
# curl http://<alb_public_ip> # Load-balanced Nginx (alternates between instances)
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.65.0, < 5.66.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
}
}
}
# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------
variable "region" {
type = string
default = "ap-southeast-2"
}
variable "instance_type" {
type = string
default = "t3.small"
}
variable "spinifex_endpoint" {
type = string
default = "https://localhost:9999"
description = "Spinifex AWS gateway endpoint"
}
# ---------------------------------------------------------------------------
# Provider — point the AWS provider at Spinifex
# ---------------------------------------------------------------------------
provider "aws" {
region = var.region
endpoints {
ec2 = var.spinifex_endpoint
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
elasticloadbalancingv2 = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------
data "aws_availability_zones" "available" {
state = "available"
}
data "aws_ami" "debian12" {
most_recent = true
owners = ["000000000000"] # Spinifex system images
filter {
name = "name"
values = ["*debian-12*"]
}
}
# ---------------------------------------------------------------------------
# SSH Key Pair
# ---------------------------------------------------------------------------
resource "tls_private_key" "nginx" {
algorithm = "ED25519"
}
resource "aws_key_pair" "nginx" {
key_name = "nginx-alb-demo"
public_key = tls_private_key.nginx.public_key_openssh
}
resource "local_file" "nginx_pem" {
filename = "${path.module}/nginx-alb-demo.pem"
content = tls_private_key.nginx.private_key_openssh
file_permission = "0600"
}
# ---------------------------------------------------------------------------
# VPC
# ---------------------------------------------------------------------------
resource "aws_vpc" "main" {
cidr_block = "10.20.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "nginx-alb-vpc"
}
}
# ---------------------------------------------------------------------------
# Internet Gateway
# ---------------------------------------------------------------------------
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.main.id
tags = {
Name = "nginx-alb-igw"
}
}
# ---------------------------------------------------------------------------
# Public Subnets (two AZs for the ALB and NAT Gateway)
# ---------------------------------------------------------------------------
resource "aws_subnet" "public_a" {
vpc_id = aws_vpc.main.id
cidr_block = "10.20.1.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
tags = {
Name = "nginx-alb-public-a"
}
}
resource "aws_subnet" "public_b" {
vpc_id = aws_vpc.main.id
cidr_block = "10.20.2.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
tags = {
Name = "nginx-alb-public-b"
}
}
# ---------------------------------------------------------------------------
# Private Subnets (two AZs for the Nginx workers)
# ---------------------------------------------------------------------------
resource "aws_subnet" "private_a" {
vpc_id = aws_vpc.main.id
cidr_block = "10.20.11.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
tags = {
Name = "nginx-alb-private-a"
}
}
resource "aws_subnet" "private_b" {
vpc_id = aws_vpc.main.id
cidr_block = "10.20.12.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
tags = {
Name = "nginx-alb-private-b"
}
}
# ---------------------------------------------------------------------------
# NAT Gateway — outbound internet for the private subnets
#
# Background plumbing: the private-subnet workers need outbound connectivity
# during cloud-init so apt-get can install Nginx. A single NAT Gateway in
# public_a provides SNAT for both private subnets via the private route table.
# ---------------------------------------------------------------------------
resource "aws_eip" "nat" {
domain = "vpc"
tags = {
Name = "nginx-alb-nat-eip"
}
}
resource "aws_nat_gateway" "main" {
allocation_id = aws_eip.nat.id
subnet_id = aws_subnet.public_a.id
tags = {
Name = "nginx-alb-nat"
}
depends_on = [aws_internet_gateway.igw]
}
# ---------------------------------------------------------------------------
# Route Tables — public subnets egress via IGW, private subnets via NAT GW
# ---------------------------------------------------------------------------
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
tags = {
Name = "nginx-alb-public-rt"
}
}
resource "aws_route_table_association" "public_a" {
subnet_id = aws_subnet.public_a.id
route_table_id = aws_route_table.public.id
}
resource "aws_route_table_association" "public_b" {
subnet_id = aws_subnet.public_b.id
route_table_id = aws_route_table.public.id
}
resource "aws_route_table" "private" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.main.id
}
tags = {
Name = "nginx-alb-private-rt"
}
}
resource "aws_route_table_association" "private_a" {
subnet_id = aws_subnet.private_a.id
route_table_id = aws_route_table.private.id
}
resource "aws_route_table_association" "private_b" {
subnet_id = aws_subnet.private_b.id
route_table_id = aws_route_table.private.id
}
# ---------------------------------------------------------------------------
# Security Group — SSH + HTTP inbound, all outbound
# ---------------------------------------------------------------------------
resource "aws_security_group" "web" {
name = "nginx-alb-sg"
description = "Allow SSH and HTTP inbound"
vpc_id = aws_vpc.main.id
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "HTTP"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
description = "All outbound"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "nginx-alb-sg"
}
}
# ---------------------------------------------------------------------------
# EC2 Instances — two Nginx servers with distinct landing pages
# ---------------------------------------------------------------------------
resource "aws_instance" "nginx_1" {
ami = data.aws_ami.debian12.id
instance_type = var.instance_type
subnet_id = aws_subnet.private_a.id
vpc_security_group_ids = [aws_security_group.web.id]
key_name = aws_key_pair.nginx.key_name
# Workers pull nginx from apt via the NAT Gateway — creation must wait
# until the NAT Gateway is available so cloud-init can reach the repos.
depends_on = [aws_nat_gateway.main]
user_data_base64 = base64encode(<<-USERDATA
#!/bin/bash
set -euo pipefail
apt-get update -y
apt-get install -y nginx
INSTANCE_ID=$(cat /var/lib/cloud/data/instance-id 2>/dev/null || hostname)
cat > /var/www/html/index.html <<HTML
<!DOCTYPE html>
<html>
<head><title>Spinifex ALB Demo</title></head>
<body style="font-family: sans-serif; max-width: 600px; margin: 80px auto;">
<h1>Hello from Spinifex!</h1>
<p><strong>Instance:</strong> $INSTANCE_ID (Server 1)</p>
<p>This Nginx server is behind an Application Load Balancer.</p>
<hr>
<p><small>Provisioned via cloud-init user-data.</small></p>
</body>
</html>
HTML
systemctl enable nginx
systemctl restart nginx
USERDATA
)
tags = {
Name = "nginx-alb-1"
}
}
resource "aws_instance" "nginx_2" {
ami = data.aws_ami.debian12.id
instance_type = var.instance_type
subnet_id = aws_subnet.private_b.id
vpc_security_group_ids = [aws_security_group.web.id]
key_name = aws_key_pair.nginx.key_name
# Workers pull nginx from apt via the NAT Gateway — creation must wait
# until the NAT Gateway is available so cloud-init can reach the repos.
depends_on = [aws_nat_gateway.main]
user_data_base64 = base64encode(<<-USERDATA
#!/bin/bash
set -euo pipefail
apt-get update -y
apt-get install -y nginx
INSTANCE_ID=$(cat /var/lib/cloud/data/instance-id 2>/dev/null || hostname)
cat > /var/www/html/index.html <<HTML
<!DOCTYPE html>
<html>
<head><title>Spinifex ALB Demo</title></head>
<body style="font-family: sans-serif; max-width: 600px; margin: 80px auto;">
<h1>Hello from Spinifex!</h1>
<p><strong>Instance:</strong> $INSTANCE_ID (Server 2)</p>
<p>This Nginx server is behind an Application Load Balancer.</p>
<hr>
<p><small>Provisioned via cloud-init user-data.</small></p>
</body>
</html>
HTML
systemctl enable nginx
systemctl restart nginx
USERDATA
)
tags = {
Name = "nginx-alb-2"
}
}
# ---------------------------------------------------------------------------
# Application Load Balancer
# ---------------------------------------------------------------------------
resource "aws_lb" "web" {
name = "nginx-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.web.id]
subnets = [aws_subnet.public_a.id, aws_subnet.public_b.id]
tags = {
Name = "nginx-alb"
}
}
# ---------------------------------------------------------------------------
# Target Group — HTTP health-checked on port 80
# ---------------------------------------------------------------------------
resource "aws_lb_target_group" "nginx" {
name = "nginx-alb-tg"
port = 80
protocol = "HTTP"
vpc_id = aws_vpc.main.id
health_check {
path = "/"
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 3
timeout = 5
interval = 10
}
tags = {
Name = "nginx-alb-tg"
}
}
# ---------------------------------------------------------------------------
# Register both instances as targets
# ---------------------------------------------------------------------------
resource "aws_lb_target_group_attachment" "nginx_1" {
target_group_arn = aws_lb_target_group.nginx.arn
target_id = aws_instance.nginx_1.id
port = 80
}
resource "aws_lb_target_group_attachment" "nginx_2" {
target_group_arn = aws_lb_target_group.nginx.arn
target_id = aws_instance.nginx_2.id
port = 80
}
# ---------------------------------------------------------------------------
# Listener — forward HTTP :80 to the target group
# ---------------------------------------------------------------------------
resource "aws_lb_listener" "http" {
load_balancer_arn = aws_lb.web.arn
port = 80
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.nginx.arn
}
}
# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------
output "note" {
value = <<-EOT
EC2 instances can take 30+ seconds to boot after apply — if HTTP is
unreachable, wait and retry.
The Nginx instances have private IPs only. The ALB DNS name ends in
.elb.spinifex.local and will not resolve from your host, so fetch the
ALB's public IP with:
aws elbv2 describe-load-balancers --names nginx-alb \
--query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
--output text
Then: curl http://<that-ip>
EOT
}
output "alb_name" {
value = aws_lb.web.name
}
output "alb_arn" {
value = aws_lb.web.arn
}
output "alb_dns_name" {
value = aws_lb.web.dns_name
}
output "instance_1_id" {
value = aws_instance.nginx_1.id
}
output "instance_1_private_ip" {
value = aws_instance.nginx_1.private_ip
}
output "instance_2_id" {
value = aws_instance.nginx_2.id
}
output "instance_2_private_ip" {
value = aws_instance.nginx_2.private_ip
}
+129
View File
@@ -0,0 +1,129 @@
---
title: "Nginx Web Server"
description: "Deploy a VPC with a public subnet and an EC2 instance running Nginx using Terraform on Spinifex."
category: "Terraform Workbooks"
tags:
- terraform
- nginx
- ec2
- vpc
- workbook
resources:
- title: "Terraform AWS Provider"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "OpenTofu"
url: "https://opentofu.org/"
---
# Terraform: Nginx Web Server
> Deploy a VPC with a public subnet and an EC2 instance running Nginx using Terraform on Spinifex.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Deploy a complete Nginx web server on Spinifex using Terraform/OpenTofu. This workbook provisions a VPC, public subnet, internet gateway, route table, security group, SSH key pair, and an EC2 instance with cloud-init user-data that installs and starts Nginx.
**What you'll learn:**
- Configuring the AWS Terraform provider to target Spinifex
- Creating a VPC with public internet access
- Provisioning an EC2 instance with cloud-init user-data
- Generating SSH key pairs with the TLS provider
**Prerequisites:**
- Spinifex installed and running (see [Installing Spinifex](/docs/install))
- A Debian 12 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed
- `AWS_PROFILE=spinifex` configured
## Instructions
### Step 1. Get the Template
Clone the Terraform examples from the Spinifex repository:
```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/nginx-webserver
```
Or create a `main.tf` file and paste the full configuration below.
<!-- INCLUDE: main.tf lang:hcl -->
### Step 2. Deploy
```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```
### Step 3. Verify
> **Note:** EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry.
After apply completes, use the outputs to test:
```bash
curl http://<public_ip>
ssh -i nginx-demo.pem ec2-user@<public_ip>
```
Open the `web_url` output in your browser to see the Nginx welcome page.
### Clean Up
```bash
tofu destroy
```
## Troubleshooting
### AMI Not Found
Ensure you have imported a Debian 12 image. Check available AMIs:
```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```
### Provider Connection Refused
Verify Spinifex services are running:
```bash
sudo systemctl status spinifex.target
curl -k https://localhost:9999/
```
### SSH Connection Timeout
Check that the security group allows inbound SSH (port 22) and that the instance has a public IP assigned. Verify the instance is running:
```bash
aws ec2 describe-instances --profile spinifex
```
### Nginx Not Responding
SSH into the instance and check cloud-init logs:
```bash
ssh -i nginx-demo.pem ec2-user@<public_ip>
sudo journalctl -u cloud-init --no-pager
sudo systemctl status nginx
```
+286
View File
@@ -0,0 +1,286 @@
# Example 1: Nginx Web Server on Spinifex
#
# Deploys a VPC with a public subnet and an EC2 instance running Nginx.
# Demonstrates: VPC, subnet, internet gateway, route table, security group,
# key pair, cloud-init user-data, and EC2 instance provisioning.
#
# Usage:
# cd spinifex/scripts/iac/aws/examples/01-nginx-webserver
# export AWS_PROFILE=spinifex
# tofu init && tofu apply
#
# After apply:
# curl http://<public_ip> # Nginx welcome page
# ssh -i nginx-demo.pem ec2-user@<public_ip>
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
}
}
}
# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------
variable "region" {
type = string
default = "ap-southeast-2"
}
variable "spinifex_endpoint" {
type = string
default = "https://localhost:9999"
description = "Spinifex AWS gateway endpoint"
}
# ---------------------------------------------------------------------------
# Provider — point the AWS provider at Spinifex
# ---------------------------------------------------------------------------
provider "aws" {
region = var.region
endpoints {
ec2 = var.spinifex_endpoint
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------
data "aws_availability_zones" "available" {
state = "available"
}
data "aws_ami" "debian12" {
most_recent = true
owners = ["000000000000"] # Spinifex system images
filter {
name = "name"
values = ["*debian-12*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
filter {
name = "root-device-type"
values = ["ebs"]
}
}
# ---------------------------------------------------------------------------
# SSH Key Pair
# ---------------------------------------------------------------------------
resource "tls_private_key" "nginx" {
algorithm = "ED25519"
}
resource "aws_key_pair" "nginx" {
key_name = "nginx-demo"
public_key = tls_private_key.nginx.public_key_openssh
}
resource "local_file" "nginx_pem" {
filename = "${path.module}/nginx-demo.pem"
content = tls_private_key.nginx.private_key_openssh
file_permission = "0600"
}
# ---------------------------------------------------------------------------
# VPC
# ---------------------------------------------------------------------------
resource "aws_vpc" "main" {
cidr_block = "10.10.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "nginx-demo-vpc"
}
}
# ---------------------------------------------------------------------------
# Internet Gateway — gives the public subnet a route to the internet
# ---------------------------------------------------------------------------
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.main.id
tags = {
Name = "nginx-demo-igw"
}
}
# ---------------------------------------------------------------------------
# Public Subnet
# ---------------------------------------------------------------------------
resource "aws_subnet" "public" {
vpc_id = aws_vpc.main.id
cidr_block = "10.10.1.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
map_public_ip_on_launch = true
tags = {
Name = "nginx-demo-public"
}
}
# ---------------------------------------------------------------------------
# Route Table — send 0.0.0.0/0 through the internet gateway
# ---------------------------------------------------------------------------
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
tags = {
Name = "nginx-demo-public-rt"
}
}
resource "aws_route_table_association" "public" {
subnet_id = aws_subnet.public.id
route_table_id = aws_route_table.public.id
}
# ---------------------------------------------------------------------------
# Security Group — SSH + HTTP inbound, all outbound
# ---------------------------------------------------------------------------
resource "aws_security_group" "web" {
name = "nginx-demo-sg"
description = "Allow SSH and HTTP inbound"
vpc_id = aws_vpc.main.id
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "HTTP"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
description = "All outbound"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "nginx-demo-sg"
}
}
# ---------------------------------------------------------------------------
# EC2 Instance — Debian 12 with Nginx installed via cloud-init
# ---------------------------------------------------------------------------
resource "aws_instance" "nginx" {
ami = data.aws_ami.debian12.id
instance_type = "t3.small"
subnet_id = aws_subnet.public.id
vpc_security_group_ids = [aws_security_group.web.id]
key_name = aws_key_pair.nginx.key_name
associate_public_ip_address = true
user_data_base64 = base64encode(<<-USERDATA
#!/bin/bash
set -euo pipefail
# Install Nginx
apt-get update -y
apt-get install -y nginx
# Write a custom landing page
cat > /var/www/html/index.html <<'HTML'
<!DOCTYPE html>
<html>
<head><title>Spinifex Demo</title></head>
<body style="font-family: sans-serif; max-width: 600px; margin: 80px auto;">
<h1>Hello from Spinifex!</h1>
<p>This Nginx server was deployed with Terraform on Spinifex infrastructure.</p>
<hr>
<p><small>Instance provisioned via cloud-init user-data.</small></p>
</body>
</html>
HTML
# Ensure Nginx is running
systemctl enable nginx
systemctl restart nginx
USERDATA
)
tags = {
Name = "nginx-demo"
}
}
# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------
output "note" {
value = "EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry."
}
output "instance_id" {
value = aws_instance.nginx.id
}
output "public_ip" {
value = aws_instance.nginx.public_ip
}
output "ssh_command" {
value = "ssh -i nginx-demo.pem ec2-user@${aws_instance.nginx.public_ip}"
}
output "web_url" {
value = "http://${aws_instance.nginx.public_ip}"
}
+159
View File
@@ -0,0 +1,159 @@
---
title: "S3-Backed Web App"
description: "Deploy a Flask file-sharing web application backed by S3 (Predastore) using Terraform on Spinifex."
category: "Terraform Workbooks"
tags:
- terraform
- s3
- predastore
- flask
- webapp
- workbook
resources:
- title: "Terraform AWS Provider"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "Predastore (S3)"
url: "https://github.com/mulgadc/predastore"
- title: "OpenTofu"
url: "https://opentofu.org/"
---
# Terraform: S3-Backed Web App
> Deploy a Flask file-sharing web application backed by S3 (Predastore) using Terraform on Spinifex.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Troubleshooting](#troubleshooting)
---
## Overview
Deploy an EC2 instance running a Flask file-sharing web application backed by S3 (Predastore). Users can upload files through a web form and browse uploaded content — demonstrating Terraform managing both compute and object storage together.
**Architecture:**
<p align="center">
<img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance to Predastore via S3 API" width="900">
</p>
**What you'll learn:**
- Configuring the AWS provider with both Spinifex and Predastore endpoints
- Creating S3 buckets on Predastore via Terraform
- Deploying a Python webapp with cloud-init that talks to S3
- Passing credentials and configuration to instances via user-data
**Prerequisites:**
- Spinifex installed and running (see [Installing Spinifex](/docs/install))
- Predastore running (S3 API on port 8443)
- A Debian 12 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed
- `AWS_PROFILE=spinifex` configured
- The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost
## Instructions
### Step 1. Get the Template
Clone the Terraform examples from the Spinifex repository:
```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/s3-webapp
```
Or create the files manually and paste the full configuration below.
### Step 2. Create terraform.tfvars
Before deploying, create a `terraform.tfvars` with your Predastore credentials. The `predastore_host` must be reachable from inside the VPC — use the host's br-wan or LAN IP, not localhost.
<!-- INCLUDE: terraform.tfvars.example lang:hcl -->
### Step 3. Create main.tf
<!-- INCLUDE: main.tf lang:hcl -->
### Step 4. Deploy
```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```
### Step 5. Test the Application
> **Note:** EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry.
Open the `web_url` output in your browser. You should see the file browser UI. Upload a file and verify it appears in the list.
```bash
# Verify via CLI
curl http://<public_ip>
# Check the S3 bucket directly
aws s3 ls s3://webapp-uploads/ --profile spinifex --endpoint-url https://localhost:8443
```
### Clean Up
```bash
tofu destroy
```
## Troubleshooting
### Predastore Connection Refused from Instance
The EC2 instance cannot reach `localhost` on the host. Set `predastore_host` to the host's br-wan or LAN IP address:
```hcl
predastore_host = "192.168.1.10:8443"
```
### S3 Bucket Creation Fails
Verify Predastore is running and accessible:
```bash
curl -k https://localhost:8443/
aws s3 ls --profile spinifex --endpoint-url https://localhost:8443
```
### Flask App Not Starting
SSH into the instance and check the service:
```bash
ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
sudo systemctl status s3-webapp
sudo journalctl -u s3-webapp --no-pager -n 50
```
### Upload Fails or Files Not Appearing
Check that the S3 credentials in `/opt/webapp/.env` are correct and that the bucket exists:
```bash
ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
cat /opt/webapp/.env
/opt/webapp/venv/bin/python -c "import boto3; print('boto3 OK')"
```
### AMI Not Found
Ensure you have imported a Debian 12 image:
```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```
+445
View File
@@ -0,0 +1,445 @@
# Example 3: S3-Backed Web Application
#
# Deploys an EC2 instance running a simple file-sharing webapp backed by S3
# (Predastore). Users can upload files through a web form and browse uploaded
# content — demonstrating Terraform managing both compute (Spinifex) and
# object storage (Predastore) resources together.
#
# Architecture:
#
# Browser ──HTTP──▶ EC2 Instance (Flask webapp, port 80)
# │
# ▼ S3 API (boto3)
# Predastore (port 8443)
#
# Prerequisites:
# - Spinifex services running (gateway on port 9999)
# - Predastore running (S3 API on port 8443)
# - The EC2 instance must be able to reach the Predastore endpoint.
# Set `predastore_host` to the IP reachable from inside the VPC
# (e.g. the host's br-wan IP, NOT localhost).
#
# Usage:
# cd spinifex/scripts/iac/aws/examples/03-s3-webapp
# export AWS_PROFILE=spinifex
# tofu init && tofu apply
#
# After apply:
# curl http://<public_ip> # File browser UI
# ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
}
}
}
# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------
variable "region" {
type = string
default = "ap-southeast-2"
}
variable "spinifex_endpoint" {
type = string
default = "https://localhost:9999"
description = "Spinifex AWS gateway endpoint (EC2/IAM)"
}
variable "predastore_endpoint" {
type = string
default = "https://localhost:8443"
description = "Predastore S3 endpoint (for Terraform to create buckets)"
}
variable "predastore_host" {
type = string
description = "Predastore host:port reachable from inside the VPC (e.g. 192.168.1.10:8443)"
}
variable "s3_access_key" {
type = string
description = "S3 access key for Predastore"
}
variable "s3_secret_key" {
type = string
sensitive = true
description = "S3 secret key for Predastore"
}
variable "bucket_name" {
type = string
default = "webapp-uploads"
}
# ---------------------------------------------------------------------------
# Provider — EC2 via Spinifex gateway, S3 via Predastore
# ---------------------------------------------------------------------------
provider "aws" {
region = var.region
access_key = var.s3_access_key
secret_key = var.s3_secret_key
endpoints {
ec2 = var.spinifex_endpoint
s3 = var.predastore_endpoint
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
}
s3_use_path_style = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------
data "aws_availability_zones" "available" {
state = "available"
}
data "aws_ami" "debian12" {
most_recent = true
owners = ["000000000000"]
filter {
name = "name"
values = ["*debian-12*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
filter {
name = "root-device-type"
values = ["ebs"]
}
}
# ---------------------------------------------------------------------------
# S3 Bucket (created on Predastore)
# ---------------------------------------------------------------------------
resource "aws_s3_bucket" "uploads" {
bucket = var.bucket_name
}
# ---------------------------------------------------------------------------
# SSH Key Pair
# ---------------------------------------------------------------------------
resource "tls_private_key" "webapp" {
algorithm = "ED25519"
}
resource "aws_key_pair" "webapp" {
key_name = "s3-webapp-demo"
public_key = tls_private_key.webapp.public_key_openssh
}
resource "local_file" "webapp_pem" {
filename = "${path.module}/s3-webapp-demo.pem"
content = tls_private_key.webapp.private_key_openssh
file_permission = "0600"
}
# ---------------------------------------------------------------------------
# VPC + Public Subnet
# ---------------------------------------------------------------------------
resource "aws_vpc" "main" {
cidr_block = "10.30.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "s3-webapp-demo-vpc"
}
}
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.main.id
tags = {
Name = "s3-webapp-demo-igw"
}
}
resource "aws_subnet" "public" {
vpc_id = aws_vpc.main.id
cidr_block = "10.30.1.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
map_public_ip_on_launch = true
tags = {
Name = "s3-webapp-demo-public"
}
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
tags = {
Name = "s3-webapp-demo-public-rt"
}
}
resource "aws_route_table_association" "public" {
subnet_id = aws_subnet.public.id
route_table_id = aws_route_table.public.id
}
# ---------------------------------------------------------------------------
# Security Group — SSH + HTTP inbound, all outbound
# ---------------------------------------------------------------------------
resource "aws_security_group" "webapp" {
name = "s3-webapp-demo-sg"
description = "Allow SSH and HTTP inbound"
vpc_id = aws_vpc.main.id
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "HTTP"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
description = "All outbound"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "s3-webapp-demo-sg"
}
}
# ---------------------------------------------------------------------------
# EC2 Instance — Flask webapp that talks to Predastore S3
# ---------------------------------------------------------------------------
resource "aws_instance" "webapp" {
ami = data.aws_ami.debian12.id
instance_type = "t3.small"
subnet_id = aws_subnet.public.id
vpc_security_group_ids = [aws_security_group.webapp.id]
key_name = aws_key_pair.webapp.key_name
associate_public_ip_address = true
user_data_base64 = base64encode(<<-USERDATA
#!/bin/bash
set -euo pipefail
# Install dependencies
apt-get update -y
apt-get install -y python3-pip python3-venv
# Create app directory and virtualenv
mkdir -p /opt/webapp
python3 -m venv /opt/webapp/venv
/opt/webapp/venv/bin/pip install flask boto3
# Write S3 credentials config
cat > /opt/webapp/.env <<'ENVFILE'
S3_ENDPOINT=https://${var.predastore_host}
S3_BUCKET=${var.bucket_name}
S3_ACCESS_KEY=${var.s3_access_key}
S3_SECRET_KEY=${var.s3_secret_key}
S3_REGION=${var.region}
ENVFILE
# Write the Flask application
cat > /opt/webapp/app.py <<'PYEOF'
import os, io, urllib3
from flask import Flask, request, redirect, url_for, Response
# Suppress TLS warnings for self-signed certs
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# Load env
env = {}
with open("/opt/webapp/.env") as f:
for line in f:
line = line.strip()
if "=" in line and not line.startswith("#"):
k, v = line.split("=", 1)
env[k] = v
import boto3
from botocore.config import Config
s3 = boto3.client(
"s3",
endpoint_url=env["S3_ENDPOINT"],
aws_access_key_id=env["S3_ACCESS_KEY"],
aws_secret_access_key=env["S3_SECRET_KEY"],
region_name=env["S3_REGION"],
verify=False,
config=Config(s3={"addressing_style": "path"}),
)
BUCKET = env["S3_BUCKET"]
app = Flask(__name__)
@app.route("/")
def index():
# List objects in the bucket
try:
resp = s3.list_objects_v2(Bucket=BUCKET)
objects = resp.get("Contents", [])
except Exception as e:
objects = []
rows = ""
for obj in objects:
key = obj["Key"]
size = obj["Size"]
rows += f'<tr><td><a href="/files/{key}">{key}</a></td><td>{size} bytes</td></tr>\n'
return f"""<!DOCTYPE html>
<html>
<head><title>Spinifex S3 File Browser</title></head>
<body style="font-family: sans-serif; max-width: 700px; margin: 40px auto;">
<h1>Spinifex S3 File Browser</h1>
<p>Bucket: <code>{BUCKET}</code></p>
<h2>Upload a File</h2>
<form method="POST" action="/upload" enctype="multipart/form-data">
<input type="file" name="file" required>
<button type="submit">Upload</button>
</form>
<h2>Files</h2>
<table border="1" cellpadding="6" cellspacing="0" style="border-collapse: collapse;">
<tr><th>Key</th><th>Size</th></tr>
{rows if rows else '<tr><td colspan="2">No files yet</td></tr>'}
</table>
<hr>
<p><small>Powered by Spinifex + Predastore</small></p>
</body>
</html>"""
@app.route("/upload", methods=["POST"])
def upload():
f = request.files.get("file")
if not f or not f.filename:
return redirect("/")
s3.put_object(Bucket=BUCKET, Key=f.filename, Body=f.read())
return redirect("/")
@app.route("/files/<path:key>")
def download(key):
try:
obj = s3.get_object(Bucket=BUCKET, Key=key)
return Response(
obj["Body"].read(),
headers={"Content-Disposition": f'inline; filename="{key}"'},
)
except Exception:
return "Not found", 404
if __name__ == "__main__":
app.run(host="0.0.0.0", port=80)
PYEOF
# Create a systemd service so the webapp starts on boot
cat > /etc/systemd/system/s3-webapp.service <<'SVCEOF'
[Unit]
Description=S3 File Browser Webapp
After=network.target
[Service]
Type=simple
ExecStart=/opt/webapp/venv/bin/python /opt/webapp/app.py
WorkingDirectory=/opt/webapp
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
SVCEOF
systemctl daemon-reload
systemctl enable s3-webapp
systemctl start s3-webapp
USERDATA
)
tags = {
Name = "s3-webapp-demo"
}
}
# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------
output "note" {
value = "EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry."
}
output "instance_id" {
value = aws_instance.webapp.id
}
output "public_ip" {
value = aws_instance.webapp.public_ip
}
output "bucket_name" {
value = aws_s3_bucket.uploads.id
}
output "ssh_command" {
value = "ssh -i s3-webapp-demo.pem ec2-user@${aws_instance.webapp.public_ip}"
}
output "web_url" {
value = "http://${aws_instance.webapp.public_ip}"
}
@@ -0,0 +1,14 @@
# Copy this to terraform.tfvars and fill in your values.
#
# The predastore_host must be reachable from INSIDE the VPC guest — not
# localhost. Use the host's br-wan or LAN IP, e.g. "192.168.1.10:8443".
predastore_host = "192.168.1.10:8443"
s3_access_key = "AKIAIOSFODNN7EXAMPLE"
s3_secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
# Optional overrides:
# spinifex_endpoint = "https://localhost:9999"
# predastore_endpoint = "https://localhost:8443"
# bucket_name = "webapp-uploads"
# region = "ap-southeast-2"
+124
View File
@@ -0,0 +1,124 @@
---
title: "Updating Spinifex"
description: "Upgrade an existing Spinifex installation to the latest release."
category: "Getting Started"
tags:
- update
- upgrade
- migrate
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "Single-Node Install"
url: "https://docs.mulgadc.com/docs/install"
---
# Updating Spinifex
> Upgrade an existing Spinifex installation to the latest release.
## Table of Contents
- [Overview](#overview)
- [Instructions](#instructions)
- [Manual Upgrade](#manual-upgrade)
- [Troubleshooting](#troubleshooting)
---
## Overview
Updating Spinifex is the same command used to install it. The installer detects an existing installation, downloads the latest binary and runs any pending configuration migrations before restarting services.
For operators who want to review migrations before they are applied, a manual upgrade path is also supported.
## Instructions
## Step 1. Re-run the Installer
```bash
curl -fsSL https://install.mulgadc.com | bash
```
That's it. The installer will:
1. Download and install the latest Spinifex binary.
2. Reinstall systemd units so new services are picked up.
3. Run any pending configuration migrations automatically (equivalent to `spx admin upgrade --yes`).
4. Restart `spinifex.target` if the services were already running.
## Step 2. Verify
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
If this returns a list of instance types, your upgrade is complete.
## Manual Upgrade
If you prefer to review pending migrations before they are applied, Spinifex supports running `spx admin init` to allow you to verify config file migrations.
## Step 1. Install the New Binary Without Running Migrations
```bash
curl -fsSL https://install.mulgadc.com | INSTALL_SPINIFEX_SKIP_MIGRATE=1 bash
```
The installer will download the new binary and reinstall systemd units, but will **not** apply any configuration migrations.
## Step 2. Review Pending Migrations
```bash
sudo spx admin upgrade
```
The command prints the current version of each config file, the migrations that would be applied, and a `from → to` description for each. It then prompts for confirmation before making any changes. Answer `n` to abort without touching your config.
## Step 3. Apply Migrations
When you are ready, answer `y` at the prompt, or re-run with `--yes` to apply non-interactively:
```bash
sudo spx admin upgrade --yes
```
## Step 4. Restart Services
Migrations modify config files on disk but do not restart running services. Apply the new config with:
```bash
sudo systemctl restart spinifex.target
```
## Troubleshooting
### No Pending Config Migrations
```
No pending config migrations.
```
Your config is already at the latest version. Nothing to do.
### No Spinifex Installation Found
```
No Spinifex installation found at /etc/spinifex
Run 'spx admin init' first.
```
`spx admin upgrade` requires an initialized installation. If this is a fresh host, follow the [Single-Node Install](/docs/install) guide instead.
### Migration Failure
If a migration fails, the installer and `spx admin upgrade` exit non-zero and leave the config in its prior state where possible. Review the error output, then re-run `sudo spx admin upgrade` once the underlying issue is resolved.
### Services Did Not Pick Up New Config
Migrations edit config files on disk but the running daemons continue to use the config they loaded at start-up. Restart with:
```bash
sudo systemctl restart spinifex.target
```

Some files were not shown because too many files have changed in this diff Show More