mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
ab95566795
* feat: add ALB target health reporting via HAProxy stats Agent-side: periodically query HAProxy stats socket and publish target health status to NATS (elbv2.alb.<id>.health topic). Control-plane: healthChecker subscribes to health reports from all ALB agents and updates target health state in the KV store. Wire into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup. E2E: remove health check skip — targets must now become healthy. * feat: allocate public IP for internet-facing ALB VMs in daemon Add Scheme field to SystemInstanceInput so the daemon knows whether an ALB is internet-facing or internal. For internet-facing, allocate a public IP via externalIPAM and publish vpc.add-nat for the OVN dnat_and_snat rule. Internal ALBs skip public IP assignment entirely. * fix: release auto-assigned public IPs on DeleteNetworkInterface DeleteNetworkInterface previously leaked public IPs and NAT rules. Now checks if the ENI has a public IP, verifies it's not EIP-owned, and releases via externalIPAM + publishes vpc.delete-nat. Daemon injects externalIPAM and EIP KV into VPC service at init time. Tests cover: auto-assigned IP release with NAT event, EIP-owned IP skipped, and no-op when externalIPAM is not configured. * feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770) In CreateLoadBalancer, when DevNetworking is false (pool mode), require 2+ subnets in different AZs for internet-facing scheme. Dev mode and internal scheme skip the validation. * feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771) Add 9 unit tests covering scheme-dependent ALB behavior: - DNS name uses internal- prefix for internal scheme (AWS-compatible) - Internet-facing ALBs include public IP in AvailabilityZones response - Internal ALBs have no public IP in AZ response - Launcher receives correct scheme, delete cleans up VM + ENIs Also implements the code changes these tests exercise: - AvailZoneInfo.PublicIP field populated from launcher output - lbRecordToSDK populates LoadBalancerAddresses for internet-facing * feat: add custom AMI for ALB to images list * feat: add internet-facing ALB data plane E2E test (mulga-772) New E2E script that validates internet-facing ALBs get public IPs and are reachable from a peer cluster node (external validation). Skips in dev mode. * fix: set ENDPOINT to runner node IP in ALB E2E workflow * fix: correct paths filter to reference alb-e2e.yml * fix: alb ami name and workflow step to import the image * fix: spx path * fix: remove ALB AMI import from workflow and added to bootstrap.sh * fix: use two subnets in different AZs for internet-facing ALB E2E Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each. * refactor: remove multi-AZ enforcement for internet-facing ALBs Team decision: ALBs must work with single-AZ and single-node clusters. Removes the multi-AZ validation, pool mode helper, associated tests, and error constant. E2E script reverted to single subnet. * fix: use correct account ID for ALB ENI public IP assignment The ENI is created under the caller's account but LaunchSystemInstance was looking it up with GlobalAccountID (system account), causing the UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's account ID via SystemInstanceInput.AccountID. * fix: add debug logging for ALB public IP assignment failure * fix: expand debug logging to include peer node and more keywords * fix: lazy system AMI discovery for ALB VMs The daemon was caching the system AMI at startup, but the ALB image is imported after services start (bootstrap step 9). Replace SetSystemAMI with SetSystemAMIFunc so the AMI is resolved at request time when CreateLoadBalancer actually needs it. * fix: resolve ALB VM instance type dynamically from resource manager The instance type was hardcoded to t3.nano which may not exist on all clusters. Now the daemon provides a resolver that picks the smallest available instance type from the resource manager. * push interim changes * refactor: rewrite ALB agent from NATS client to HTTP server Replace NATS subscriptions with an HTTP server on :8405 exposing POST /config, GET /ping, and GET /health. The daemon will push config and poll health over HTTP through the VPC network, removing the agent's direct NATS dependency. - agent.go: net/http server replaces nats.Conn/Subscriptions - health.go: remove push-based reportHealth; /health handler calls queryHAProxyStats synchronously on each request - cmd/alb-agent: --nats flag replaced with --listen - tests: rewritten with httptest instead of NATS test server - health_checker_test: inline topic strings (HealthTopic removed) * refactor: replace NATS with HTTP client for daemon-to-ALB agent comms Daemon now pushes config via POST and polls ping via GET to the ALB agent's HTTP server on the VPC IP, eliminating the agent's NATS dependency from the daemon side. - service_impl: pushConfig uses http.Post, waitForAgentReady uses http.Get, both target http://<vpcIP>:8405 - albVMUserData: drop natsURL param (agent no longer needs NATS) - types: add VPCIP field to LoadBalancerRecord - daemon: remove SetNATSURL call and 10.0.2.2 translation block - arn_test: update user data tests for new signature * refactor: rewrite ALB health checker from NATS subscription to HTTP polling Health checker now polls each active ALB's GET /health endpoint on a 10s ticker instead of subscribing to a NATS wildcard topic. Removes the NATS connection dependency from the health checker. - health_checker: replace nc/sub with httpClient + stopCh polling loop - newHealthChecker: drop nc param, add agentURLFn for testability - service_impl: update newHealthChecker call site - tests: rewrite with httptest, test pollAll/pollOne paths * refactor: remove NATS wiring from ALB image and make hostfwd dev-only - alb-setup.sh: remove --nats flag, use --listen=:8405; remove placeholder HAProxy frontend on 8405 (agent owns the port now); update description and comments - service_impl: HostfwdPorts (80/443) now only set when DevNetworking is enabled — production ALBs are reached via VPC IP directly * feat: system credentials file + daemon wiring for ALB agent SigV4 auth Write config/system-credentials.json during admin init/join with the system access key. Daemon reads it at startup, resolves the gateway URL from external IPAM (or 10.0.2.2 in dev mode), and wires both into the ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY, and ALB_SECRET_KEY into cloud-init for ALB VMs. * feat: ALB agent gateway actions + daemon handlers (heartbeat, config) Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB agent can communicate via the AWS gateway instead of direct HTTP. - Add agent_types.go with input/output types for both actions - Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active on first contact, updates LastHeartbeat, processes health report - Add GetALBConfig: returns pre-computed HAProxy config text and hash - Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores config + SHA256 hash on LB record for agent pull) - Remove waitForAgentReady polling goroutine (heartbeat handles transition) - Remove HTTP client from ELBv2ServiceImpl (no longer needed) - Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health reports now arrive directly via ALBAgentHeartbeat - Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord - Wire NATS subscriptions + daemon handlers for both new actions - Add gateway handler files and elbv2Actions map entries - Comprehensive tests for heartbeat, config, and updateStoredConfig * feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r) Replace the HTTP server agent with a poll-based client that communicates with the gateway using SigV4-signed requests. The agent now sends ALBAgentHeartbeat every 5s with health reports and fetches config via GetALBConfig when the config hash changes. * feat: add VPC service route for ALB agent gateway access ALB VMs need to reach the gateway for heartbeat/config polling, but VPCs without an IGW have no external routing. Instead of requiring an IGW, the daemon now publishes a vpc.ensure-service-route event before launching ALB VMs. vpcd handles this by creating minimal external connectivity (external switch, localnet port, gateway router port, chassis bindings) if not already present, then adds a /32 static route for the gateway IP. Refactors handleIGWAttach to share the external connectivity setup via ensureExternalConnectivity helper. * Revert "feat: add VPC service route for ALB agent gateway access" This reverts commitd4220d8806. * feat: add management NIC infrastructure for system instance control plane Add config, IP allocator, TAP helpers, and VM struct fields for the management NIC that gives system instances (ALBs etc.) L3 access to the host control plane via br-mgmt, independent of VPC routing. - DaemonConfig.MgmtBridge (default "br-mgmt") - MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart - generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4 - SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP) - VM struct: MgmtMAC, MgmtIP, MgmtTap fields * feat: wire management NIC into QEMU launch and cloud-init config Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt), extend generateNetworkConfig to emit mgmt0 stanza with static IP, and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes mgmt TAP cleanup on instance stop and failed launch cleanup. * test: add coverage for ELBv2 service lifecycle and setter methods * fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP * fix: generate per-interface cloud-init config when eniMAC present without devMAC The guard in generateNetworkConfig required both eniMAC and devMAC to produce per-interface config, falling back to wildcard DHCP otherwise. System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking is independent), so the mgmt NIC stanza was never emitted. The wildcard config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the management NIC unconfigured and the agent unable to reach the gateway. * fix: add wildcard support to ENI description filter + add ca-certificates to ALB image - ENI DescribeNetworkInterfaces description filter did exact string match, but AWS supports * wildcards. E2E test uses "ELB *" which didn't match "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching. - ALB image was missing ca-certificates package, so cloud-init ca_certs module couldn't add the Spinifex CA to the system trust store. Agent TLS handshakes failed with "certificate signed by unknown authority". - Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP where async TerminateSystemInstance goroutine wrote to mock fields while the test read them. * fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance DeleteLoadBalancer was trying to delete ENIs synchronously, but they were still attached (in-use) to the running VM. The delete was rejected with InvalidNetworkInterface.InUse. Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface to clear the in-use status. E2E test now polls for ENI cleanup (up to 30s) since VM termination is async. * fix: detach then delete ENIs in DeleteLoadBalancer ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface rejects them. Fix: call DetachENI first to clear in-use status, then delete. stopInstance tolerates already-deleted ENIs. * test: add internal ALB dataplane E2E test Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane test. Internet-facing coverage is already provided by run-elbv2-dataplane-internet-facing-e2e.sh. The internal test validates: - ALB reaches active via mgmt NIC heartbeat (no IGW dependency) - Scheme=internal, no public IP, internal- DNS prefix - Health checks pass via VPC L2 (OVN logical switch) - Round-robin traffic from a VPC client to the ALB's private IP - ENI cleanup on ALB deletion Uses a client VM inside the VPC that curls the ALB private IP and serves results via HTTP on its public IP for host-side verification. * feat: add NLB constants and data model changes (mulga-838) * feat: type-aware validation in CreateLoadBalancer (mulga-839) Accept Type=network for NLB, reject SecurityGroups for NLB, set CrossZoneEnabled defaults (false for NLB, true for ALB), and use /net/ ARN path segment for network load balancers. * feat: type-aware validation in CreateTargetGroup (mulga-840) Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds). Reject invalid protocols with InvalidParameterValue. * feat: type-aware validation in CreateListener (mulga-841) * feat: add NLB listener and protocol compatibility unit tests Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4 protocols, protocol rejection for wrong LB type, listener-to-target-group protocol compatibility validation, and default protocol rejection on NLB. * feat: add TCP HAProxy config generation for NLB Add TCP mode template and generateNLBConfig function for Network Load Balancer support. GenerateHAProxyConfig now branches on LB type, producing mode-tcp configs for NLBs with tcp-check or httpchk health checks depending on the target group's health check protocol. * refactor: rename albagent to lbagent for NLB generalization Rename the ALB-specific agent package and binary to generic LB names in preparation for NLB support. Both ALB and NLB VMs use the same agent — it just applies whatever HAProxy config the control plane generates. - Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/ - Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig* - Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat - Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars - Rename Makefile targets: build-alb-agent → build-lb-agent - Update gateway action map, daemon subscriptions, service interface - Fix pre-existing unused matchFilter function lint error - Fix awserrors test count for InvalidConfigurationRequest * feat: add sys.micro system instance type for LB VMs (mulga-845) Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for system VMs (load balancers, NAT gateways). Registered in the type map for allocation but excluded from DescribeInstanceTypes, NATS subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc now returns sys.micro instead of scanning for smallest guest type. * feat: LB image shrink and rename (mulga-846) Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE reduced from 512M to 192M. Update all references to use lb-agent binary and LB_* env vars. Update COMMANDS.md for NLB support. * chore: rename file * feat: NLB E2E test and bug fixes (mulga-847) Add full NLB data plane E2E test (33 assertions, 9 phases): VPC/subnet setup, TCP target group with health checks, NLB creation (type=network), TCP listener, agent heartbeat, target health verification, TCP traffic round-robin, target deregistration, and full cleanup verification. Bug fixes found during E2E testing: - Fix HAProxy stats socket path mismatch: config used alb-/nlb- prefix but agent expects lb- prefix, preventing health reports - Fix system AMI lookup to match new -lb- image name, falling back to legacy -alb- name - Bump LB image size from 192M to 256M (Alpine base is 200M) * test: validate NLB internal scheme (mulga-857) NLB with scheme=internal works out of the box — shared code paths with ALB handle it correctly. Add unit test and E2E coverage: - Unit test: NLB internal creates with no public IP, internal- DNS prefix - E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic - E2E passed 56/56 with perfect round-robin (20/20 requests) * feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854) Replace 3 separate LB dataplane scripts with a single consolidated test that covers all 4 variants (ALB/NLB × internet-facing/internal). - Share one VPC, subnet, and 2 dual-purpose app instances across all suites - ALB + NLB provision in parallel within each phase (internet-facing / internal) - One client VM tests both internal LBs simultaneously - Replace sleep 100 with LB health check gates - Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node * chore: disable LB E2E tests in CI until dev env issue resolved * fix: read LB agent credentials from spinifex.toml instead of system-credentials.json The daemon couldn't read system-credentials.json due to systemd hardening (file owned by root:root 0600, not in ReadOnlyPaths). Use the same system credentials already loaded from spinifex.toml via the predastore config section — these are identical keys written by admin init. * feat: automate LB image build and import (mulga-861) - dev-install.sh: build lb-agent and run build-system-image.sh --import after services start so the LB image is available out of the box - build-system-image.sh: add --quiet flag to suppress build output, remove hardcoded --config path (uses auto-detected default) - images.go: rename alb-alpine → lb-alpine to match current naming * update go version * add LB e2e tests * fix: clean LB image build dir and wait for nbd device readiness Previous CI runs can leave stale state in /tmp/lb-image-build/ causing e2fsck failures. Clean the build dir before each CI build. Also add a readiness check after qemu-nbd connect to avoid race conditions. * fix: serialize nbd access with flock for concurrent CI builds Single-node and multi-node CI jobs share the same host and both build the LB system image using /dev/nbd0. Use flock to serialize the nbd connect/mount/customize/disconnect section so concurrent builds wait rather than collide. * fix: move flock to cover entire image build, not just nbd section Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be acquired before downloading/copying the image, not just before nbd connect, otherwise the second job can clobber the first job's files. * fix: skip LB image build if output is fresh from concurrent CI job When concurrent CI jobs build the same image, the second job now reuses the first's output if it's less than 10 minutes old, avoiding a duplicate download + build cycle. * fix: filter AMI discovery by name in E2E tests With the LB Alpine image now imported alongside Ubuntu, the unfiltered describe-images call picked up the wrong AMI. Filter by the expected image name so tests use the Ubuntu image for instance lifecycle. * fix: filter AMI discovery by name in multi-node E2E tests Same fix as run-e2e.sh — with the LB Alpine image now imported, the unfiltered describe-images picks the wrong AMI. * fix: use awsgw bind IP as LB agent gateway URL fallback When br-mgmt is not configured and dev_networking is off (production with external pool), the gateway URL was falling through to the external IPAM pool's GatewayIP — the WAN router, not the Spinifex node. LB VMs couldn't reach the awsgw, so heartbeats never arrived and ALBs stayed stuck in provisioning. Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach the host node's WAN IP where the gateway listens. * fix: capture StateReason and daemon logs when LB app instances fail Instances with user-data terminate immediately in CI. Add describe-instances StateReason query and daemon log dump to diagnose whether it's volume_preparation_failed or launch_failed. * fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning Three fixes for CI E2E failures: 1. run-e2e.sh: Remove custom AMI from predastore after deleting its backing snapshot. Previously the AMI remained discoverable but unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey on the deleted snapshot. 2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead of picking any non-alpine image. Also improve debug log dump to use journalctl (binary-install CI uses systemd, not log files). 3. CreateLoadBalancer: Only set provisioning state when a system VM was actually launched. Previously a failed launch still set provisioning, leaving the LB stuck forever with no agent to send heartbeats. * fix: use AWSGW bind IP as LB agent gateway URL in multi-node In multi-node deployments, the AWSGW binds to the WAN IP (e.g. 192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB agent's heartbeat was refused, leaving internet-facing ALBs stuck in provisioning. Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches it via VPC/external networking. Internal LBs get a host route on the mgmt NIC so they can reach the AWSGW bind IP through br-mgmt. * fix: add mgmt route via bootcmd for multi-node internal LBs Alpine cloud-init does not support v2 routes under ethernets — adding a routes: block to the network-config breaks all networking in the VM. Instead, add the host route via a bootcmd in the LB user data. This runs early in cloud-init and adds `ip route add {awsgw}/32 via {br-mgmt}` so internal LB agents can reach the AWSGW through the management NIC when the gateway binds to a specific WAN IP. Also moves the networking service from boot to default runlevel in the LB Alpine image to prevent a race with cloud-init. Verified locally: internal ALB transitions to active with AWSGW bound to WAN IP (simulating multi-node). * add LB enable flag to e2e * test: add elbv2 tests to meet diff coverage threshold Cover health checker edge cases (invalid JSON, empty servers, zero thresholds, port-0 fallback) and service_impl paths (mgmt route cloud-init, protocol validation, missing target group). * fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat can pass the struct directly instead of marshaling to JSON and immediately unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn so failures are visible in logs. * fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867) Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey, and systemSecretKey — blank env vars previously caused agent auth-loop with no daemon-side log. Set LB state to failed (not active) when VM launch fails so the API reflects the broken data-plane instead of black-holing traffic. * fix: data race in getSystemAMI/getSystemInstanceType lazy caching Use sync.Once to guard lazy resolution of systemAMI and systemInstanceType fields. Concurrent CreateLoadBalancer calls previously raced on unsynchronized read+write of these cached values. The setter methods reset the Once so re-resolution works when the func is set after initial construction. Also fix pre-existing nosprintfhostport lint in daemon.go. * fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now delegates to utils.PublishEvent (matching the EIP service pattern). isEIPOwned previously swallowed all KV errors and returned false, which could release an EIP-owned public IP if the store was temporarily unavailable. Now returns (bool, error) so the caller skips release on KV failure instead of assuming no EIP ownership. * fix: narrow health checker to only scan target groups for reporting LB handleHealthReportDirect was calling ListTargetGroups() on every heartbeat (every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow LB → listeners → TG ARNs and fetch only the attached target groups via direct key lookups. Falls back to full scan if LBID is missing. * fix: pass region through to LB agent SigV4 signing (mulga-871) LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region from ELBv2 service config → cloud-init LB_REGION env var → lb-agent --region flag → lbagent.New() → signer.Sign() call. * fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig Both endpoints accepted any accountID without verifying the LB belongs to that account. Now verify accountID matches the LB's AccountID, with an exception for the system account (000000000000) which the ALB agent uses for heartbeats and config fetches. * fix: upgrade silent failure paths from Debug to Warn/Error - healthChecker target group list failure: Debug → Warn - HAProxy stats socket error in agent tick(): Debug → Warn - ENI delete EIP-owned skip: Debug → Info (operational visibility) - wireLBAgentConfig: warn when no gateway host is resolved - detectMgmtIP: warn when bridge exists but IP is empty - stale mgmt TAP cleanup: log errors instead of discarding * refactor: deduplicate MAC hash function into utils.HashMAC Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC, generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper. * fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps Previously the full LoadBalancerRecord (including ConfigText) was written to JetStream KV on every heartbeat (every 5s). Now only persists on state transitions (provisioning→active) or when LastHeartbeat is older than 60s. * fix: fail ALB launch when public IP allocation fails for internet-facing scheme Previously, IPAM allocation failure was logged as a warning and the ALB VM continued launching without a public IP. The ALB reached active state but was unreachable from the internet. Now the launch returns an error and cleans up the partially-created instance, causing the ALB to enter the failed state. * fix: retry system AMI resolution when image not yet imported sync.Once cached empty results permanently — if the daemon started before the system image was imported, all CreateLoadBalancer calls silently skipped VM creation. Replace sync.Once with a mutex that only caches non-empty results, allowing retry on subsequent calls. Same fix applied to getSystemInstanceType for consistency. * fix: propagate updateStoredConfig errors to API callers (mulga-884) updateStoredConfig and updateStoredConfigForTargetGroup now return errors instead of silently swallowing them. Callers (CreateListener, DeleteListener, RegisterTargets, DeregisterTargets) return ErrorServerInternal when config persistence fails, preventing the API from reporting success while HAProxy serves stale configuration. * test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885) * fix: parse HAProxy templates once at package level (mulga-875) generateALBConfig and generateNLBConfig were calling template.New().Parse() on every invocation. Move to package-level template.Must for one-time parsing. * fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886) When management TAP or IP allocation fails, internal-scheme ALBs now return an error instead of silently continuing without a management NIC. Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB gets stuck in provisioning forever. Internet-facing ALBs retain the existing degrade-and-continue behavior. * refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877) The app/net path segment derivation was duplicated in buildLBArn, buildListenerArn, and inline in CreateLoadBalancer. Extracted into a single lbArnPathSegment helper. * refactor: use aws.StringValue in toHealthReport (mulga-878) Replace manual nil-check-then-deref with aws.StringValue for cleaner pointer-to-value conversion in the heartbeat health report builder. * refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876) Replace hand-rolled setupTestNATS (~20 lines) with the shared testutil.StartTestJetStream helper. Simplify return to just *Store since no caller needed the *nats.Conn. * fix: address PR #122 review comments (mulga-887) - Extract NATEvent named type from anonymous struct in publishNATEvent - Remove unused build-lb-image.sh thin wrapper - Remove SPX_ENABLE_LB env var from e2e workflow (unused) - Remove no-op healthChecker start/stop methods - Export simple ELBv2 fields, remove trivial setter methods - Delete old e2e scripts replaced by consolidated run-lb-e2e.sh * fix: check specific error type for ENI deletion on termination (mulga-888) ENI deletion errors were all silently downgraded to Debug level, masking genuine failures (KV issues, permissions, ENI stuck in-use) that cause orphaned ENIs and leaked IPAM addresses. Now only NotFound is logged at Debug; all other errors log at Error. * fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890) GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures, not just missing devices. Permission errors on /sys/class/net/ were silently treated as "bridge absent". Now only "no such network interface" returns nil; other errors propagate to the caller. * fix: remove backend address from proxy error response (mulga-892) The reverse proxy error handler included the internal backend host:port in the XML error body sent to clients. The address is already logged server-side; the client now sees a generic message. * fix: release mutex before KV store I/O in health checker (mulga-893) handleHealthReportDirect held hc.mu across PutTargetGroup calls (JetStream KV Put — network I/O), serializing concurrent heartbeats from independent LBs. Now collects changed TGs under the lock and persists them after releasing it. * fix: warn when system credentials missing in wireLBAgentConfig (mulga-894) fix: log DetachENI errors instead of discarding with blank identifier (mulga-895) * fix: detect veth bridge mode and use centralized NAT for it The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was misclassified as direct bridge mode, causing OVN to use distributed NAT. Distributed NAT hairpin routing fails through the Linux bridge intermediary, breaking host-to-local-LB connectivity. Add BridgeModeVeth constant, detect veth-wan-ovs interface in detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode and nat-addresses=router for both macvlan and veth modes. isMacvlanMode() is preserved for macvlan-specific behavior (MAC alignment). * fix: add retry loop for peer NLB reachability test With centralized NAT, ARP resolution for NAT external IPs relies on gratuitous ARPs from the gateway chassis. These take time to propagate to peer nodes. The peer→NLB test had a single 3s attempt while all other connectivity tests use retry loops. * fix: use request-reply for vpc.add-nat to prevent ARP propagation races publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon continued before vpcd committed the OVN NAT rule. This caused intermittent NLB connectivity failures: the LB was reported active before OVN had the DNAT rule and sent GARPs for the public IP. Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E tests for more reliable TCP probing during ARP resolution. * fix: clean up stale NAT rules before adding new ones for reused IPs When a public IP is returned to the external pool and quickly reallocated (e.g. instance terminated, then LB created), the fire-and-forget vpc.delete-nat may not have been processed yet. handleAddNAT now removes any existing dnat_and_snat rule for the same external IP before creating the new one, preventing duplicate/conflicting OVN NAT rules. This was causing intermittent NLB connectivity failures in multi-node E2E where NLB public IPs collided with recently-freed instance IPs. * fix: clean up stale NAT rules across all VPC routers on IP reuse The previous fix (0a93081) only removed stale dnat_and_snat rules from the target VPC's router. When a public IP is freed from one VPC and reallocated to a different VPC before the fire-and-forget vpc.delete-nat is processed, the stale rule on the old VPC's router caused duplicate OVN ARP responses — traffic was black-holed when the host resolved to the wrong router's MAC. DeleteAllNATsByExternalIP now searches every logical router for stale NAT rules matching the external IP and removes them in one transaction. Also adds host connectivity diagnostics (ARP, routing, OVN NAT state) to run-lb-e2e.sh so the next CI failure surfaces the root cause of the persistent NLB-from-host issue. * fix: run LB inet tests from actual hosting node, not hardcoded node1 The host/peer connectivity tests were hardcoded: "host" = node1 (where the test script runs), "peer" = node2. But LB VMs land on whichever node picks up the NATS queue message, so the tests were randomly testing local vs remote connectivity depending on placement luck. Now the test: 1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs) 2. After LB creation, discovers which node hosts each LB VM by checking for the ENI's tap device 3. Runs "local" tests from the hosting node and "remote" tests from a different node Also updates e2e.yml to pass --nodes instead of --peer.
226 lines
7.4 KiB
Makefile
226 lines
7.4 KiB
Makefile
GO_PROJECT_NAME := spx
|
|
SHELL := /bin/bash
|
|
|
|
# Detect architecture for cross-platform support
|
|
ARCH := $(shell uname -m)
|
|
ifeq ($(ARCH),x86_64)
|
|
GO_ARCH := amd64
|
|
AWS_ARCH := x86_64
|
|
# On x86, we can run ARM VMs via emulation
|
|
QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
|
|
else ifeq ($(ARCH),aarch64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
# On ARM, we can run x86 VMs via emulation
|
|
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
|
|
else ifeq ($(ARCH),arm64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
|
|
else
|
|
$(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
|
|
endif
|
|
|
|
# Ask Go whether workspace mode is active
|
|
IN_WORKSPACE := $(shell go env GOWORK)
|
|
|
|
# Use -mod=mod unless Go reports an active workspace path
|
|
ifeq ($(IN_WORKSPACE),)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else ifeq ($(IN_WORKSPACE),off)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else
|
|
GO_BUILD_MOD :=
|
|
endif
|
|
|
|
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
|
|
ifdef QUIET
|
|
_Q = @
|
|
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
|
|
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
|
|
_SECQ = >
|
|
else
|
|
_Q =
|
|
_COVQ =
|
|
_RACEQ =
|
|
_SECQ = 2>&1 | tee
|
|
endif
|
|
|
|
build:
|
|
$(MAKE) go_build
|
|
|
|
# Build spinifex-ui frontend (requires pnpm)
|
|
build-ui:
|
|
@echo -e "\n....Building spinifex-ui frontend...."
|
|
cd spinifex/services/spinifexui/frontend && pnpm build
|
|
|
|
# GO commands
|
|
VERSION ?= $(shell git describe --tags --always --dirty)
|
|
COMMIT ?= $(shell git rev-parse --short HEAD)
|
|
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
|
|
|
|
go_build:
|
|
@echo -e "\n....Building $(GO_PROJECT_NAME)"
|
|
go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
|
|
|
|
build-lb-agent:
|
|
@echo -e "\n....Building lb-agent (static)"
|
|
CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
|
|
|
|
build-system-image: ## Build a system image from manifest (use IMAGE=lb)
|
|
ifndef IMAGE
|
|
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
|
|
endif
|
|
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
|
|
|
|
build-lb-image: ## Build LB Alpine image
|
|
$(MAKE) build-system-image IMAGE=lb
|
|
|
|
go_run:
|
|
@echo -e "\n....Running $(GO_PROJECT_NAME)...."
|
|
$(GOPATH)/bin/$(GO_PROJECT_NAME)
|
|
|
|
# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
|
|
# Use this before committing to catch CI failures locally.
|
|
preflight:
|
|
@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
|
|
@echo -e "\n ✅ Preflight passed — safe to commit."
|
|
|
|
# Run unit tests
|
|
test:
|
|
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
|
|
LOG_IGNORE=1 go test -timeout 120s ./spinifex/...
|
|
|
|
# Run unit tests with coverage profile
|
|
COVERPROFILE ?= coverage.out
|
|
test-cover:
|
|
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... $(_COVQ)
|
|
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
|
|
|
|
# Run unit tests with race detector
|
|
test-race:
|
|
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
|
|
|
|
# Check that new/changed code meets coverage threshold (runs tests first)
|
|
diff-coverage: test-cover
|
|
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
|
|
|
|
bench:
|
|
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
|
|
$(MAKE) easyjson
|
|
LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
|
|
|
|
run:
|
|
$(MAKE) go_build
|
|
$(MAKE) go_run
|
|
|
|
# Fast iteration: build + install binary + restart all services
|
|
deploy: build
|
|
sudo install -m 755 bin/spx /usr/local/bin/spx
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart spinifex.target
|
|
|
|
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
|
|
# Not needed for code-only changes — use deploy for those.
|
|
reinstall:
|
|
scripts/dev-install.sh
|
|
|
|
clean:
|
|
rm -f ./bin/$(GO_PROJECT_NAME)
|
|
rm -rf spinifex/services/spinifexui/frontend/dist
|
|
|
|
install-system:
|
|
@echo -e "\n....Installing system dependencies for $(ARCH)...."
|
|
@echo "QEMU packages: $(QEMU_PACKAGES)"
|
|
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
|
|
-o Dpkg::Options::="--force-confdef" \
|
|
-o Dpkg::Options::="--force-confold" \
|
|
nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
|
|
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
|
|
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
|
|
ovn-central ovn-host openvswitch-switch dhcpcd-base
|
|
|
|
install-go:
|
|
@echo -e "\n....Installing Go 1.26.2 for $(ARCH) ($(GO_ARCH))...."
|
|
@if [ ! -d "/usr/local/go" ]; then \
|
|
curl -L https://go.dev/dl/go1.26.2.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
|
|
else \
|
|
echo "Go already installed in /usr/local/go"; \
|
|
fi
|
|
@echo "Go version: $$(go version)"
|
|
|
|
install-aws:
|
|
@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
|
|
@if ! command -v aws >/dev/null 2>&1; then \
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
|
|
unzip -o awscliv2.zip; \
|
|
./aws/install; \
|
|
rm -rf awscliv2.zip aws/; \
|
|
else \
|
|
echo "AWS CLI already installed"; \
|
|
fi
|
|
|
|
quickinstall: install-system install-go install-aws
|
|
@echo -e "\n✅ Quickinstall complete for $(ARCH)."
|
|
@echo " Please ensure /usr/local/go/bin is in your PATH."
|
|
@echo " Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
|
|
|
|
# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
|
|
lint:
|
|
@echo "Running golangci-lint..."
|
|
$(_Q)golangci-lint run ./...
|
|
@echo " golangci-lint ok"
|
|
|
|
# Auto-fix all linter issues that have fixers
|
|
fix:
|
|
golangci-lint run --fix ./...
|
|
|
|
# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
|
|
govulncheck:
|
|
@echo "Running govulncheck..."
|
|
$(_Q)go tool govulncheck ./...
|
|
@echo " govulncheck ok"
|
|
|
|
# Build release tarballs — use distro-ARCH for single arch, distro for both
|
|
distro: distro-amd64 distro-arm64
|
|
@echo ""
|
|
@echo "Distribution tarballs:"
|
|
@ls -lh dist/*.tar.gz
|
|
@echo ""
|
|
@cat dist/*.sha256
|
|
|
|
distro-amd64:
|
|
@echo "Building spinifex $(VERSION) linux/amd64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/amd64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/amd64/ \
|
|
../
|
|
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
|
|
|
|
distro-arm64:
|
|
@echo "Building spinifex $(VERSION) linux/arm64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/arm64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/arm64/ \
|
|
../
|
|
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
|
|
|
|
distro-clean:
|
|
rm -rf dist/
|
|
|
|
.PHONY: build build-ui build-lb-agent build-system-image build-lb-image go_build go_run preflight test test-cover test-race diff-coverage bench run \
|
|
deploy reinstall clean \
|
|
install-system install-go install-aws quickinstall \
|
|
lint fix govulncheck \
|
|
distro distro-amd64 distro-arm64 distro-clean
|