Josh Siviour
c8ca746bf0
* feat: add attributes map to LB and TG records with defaults
Replace CrossZoneEnabled bool with Attributes map[string]string on
LoadBalancerRecord and TargetGroupRecord. Add DefaultLoadBalancerAttributes()
and DefaultTargetGroupAttributes() functions. ALBs seed cross-zone enabled
at create time.
* feat: add 4 attributes methods to ELBv2Service interface
* feat: implement ELBv2 attributes service methods and NATS wrappers
Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
to service_impl.go with proper ARN validation and default merging.
Add corresponding NATS request wrappers in service_nats.go.
* feat: add gateway handlers for ELBv2 attributes API endpoints
Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
gateway handlers with input validation and action map registration.
* feat: add daemon handlers & NATS subscriptions for elbv2 attributes
Add 4 handler functions in daemon_handlers_elbv2.go and 4 NATS
subscriptions in daemon.go for ModifyTargetGroupAttributes,
DescribeTargetGroupAttributes, ModifyLoadBalancerAttributes, and
DescribeLoadBalancerAttributes.
* fix: add missing accountID authorization check to ELBv2 attribute handlers
* fix: guard nil attributes and log skips in ELBv2 Modify*Attributes
ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes silently
dropped attributes with nil Key or Value, and would panic on a nil
slice element when dereferencing attr.Key. Add an explicit nil-element
guard and log a warning whenever an attribute is skipped, so callers
and operators can see divergence between the submitted payload and the
persisted state.
* fix: sort ELBv2 attribute responses by key for deterministic output
DescribeTargetGroupAttributes and DescribeLoadBalancerAttributes iterated
a map and appended to a slice, producing random attribute order on each
call. Sort keys before building the response so Terraform plans stay
stable and snapshot tests can rely on consistent ordering.
Closes mulga-910
* feat: add ELBv2 DescribeTags handler (mulga-919)
Required by the Terraform AWS provider's post-create refresh on
aws_lb, aws_lb_target_group, and aws_lb_listener. Without it, the
gateway returns InvalidAction and apply fails after the resources
are already created.
Dispatches by ARN resource segment (loadbalancer/targetgroup/listener)
to the existing record stores. Enforces account scoping via per-type
not-found errors to avoid cross-account existence leaks. Listeners
return an empty Tags slice since ListenerRecord doesn't store tags yet
— that's the exact case the provider hits on every untagged listener
refresh. Output is sorted by key for deterministic responses.
* refactor: parameterize DefaultLoadBalancerAttributes by lbType (mulga-915)
Cross-zone default was split across three places: CreateLoadBalancer seeded
{load_balancing.cross_zone.enabled: true} for ALBs, DefaultLoadBalancerAttributes
returned false unconditionally, and tests asserted both sides. Make
DefaultLoadBalancerAttributes(lbType) the single source of truth: ALBs get
"true", NLBs (and any other type) get "false". CreateLoadBalancer no longer
seeds attributes — DescribeLoadBalancerAttributes derives per-type defaults
from lb.Type on read. Updated TestCreateLoadBalancer_CrossZoneAttributes to
assert via the Describe handler instead of poking stored record state.
* perf: direct KV lookup for ELBv2 *ByArn finders (mulga-913)
GetLoadBalancerByArn and GetTargetGroupByArn used to call ListLoadBalancers /
ListTargetGroups and linearly scan, doing one NATS round-trip per record in
the bucket. Terraform hits Describe*Attributes on every plan/refresh — a
20-ALB / 40-TG workspace was doing ~60 calls × ~60 records = ~3600 KV gets
per refresh where ~60 would suffice. The LB/TG ARN's final path segment IS
the KV key used by GetLoadBalancer(lbID)/GetTargetGroup(tgID), so parse the
short ID from the ARN and do an O(1) lookup, verifying the record's stored
ARN matches as defence-in-depth against any stale-key weirdness.
* fix: reject ELBv2 ModifyAttributes calls where every entry is invalid (mulga-911)
When every submitted attribute was rejected by the nil-guard (nil element,
nil Key, or nil Value), Modify{TargetGroup,LoadBalancer}Attributes still
called PutTargetGroup/PutLoadBalancer with an unchanged record and returned
a successful ModifyOutput with empty Attributes. Only a slog.Warn marked
the skip — the caller saw 200 OK and moved on. Return
ErrorInvalidParameterValue when len(input.Attributes) > 0 and every entry
was rejected, so the caller learns that nothing landed.
* fix: reject unknown ELBv2 ModifyAttributes keys with ValidationError (mulga-912)
ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes had no whitelist
check — any caller-supplied key (typo like 'stickness.enabled', deprecated
keys, or cross-product keys sent to the wrong resource) was persisted verbatim
and merrily echoed back by Describe. AWS rejects unknown keys with
ValidationError; match that so Terraform surfaces the typo at plan time.
Validate each key against DefaultTargetGroupAttributes() /
DefaultLoadBalancerAttributes(lb.Type) and return ErrorValidationError on the
first unknown key, logging both the arn and the offending key name.
* perf: skip no-op Put in ELBv2 ModifyAttributes (mulga-914)
The unconditional PutTargetGroup/PutLoadBalancer fired on every Modify call,
even when the caller sent an empty attribute list or when every submitted
value already matched the stored value. Terraform re-applies the same
attribute set on every drift check, so the steady-state path was doing a
pointless NATS/KV round-trip per call and unnecessarily expanding the
read-modify-write race window.
Track a dirty flag per Modify call: compare each submitted key's value
against tg.Attributes/lb.Attributes and only write to the map (and allocate
the map lazily) when it actually differs. Skip the Put entirely if nothing
changed. Added regression tests that capture the KV entry revision before
and after a no-op modify to prove the write was skipped.
* test: assert exact ModifyAttributes echoed key/value pairs (mulga-917)
Both round-trip tests used assert.Len(t, modOut.Attributes, 2) and never
checked that the returned Attributes actually contained the submitted
keys/values. A regression that pointed at the wrong source, dropped Value,
or returned a copy of defaults would have passed the length check
unchallenged. Build a modMap from the echoed response and assert each
expected Key/Value pair explicitly.
* test: cover sequential ModifyAttributes merge and same-key overwrite (mulga-916)
All existing round-trip tests did one Modify then Describe. No test covered:
Modify {A}, Modify {B}, Describe, assert BOTH keys present; or Modify
overwriting the same key on a second call. A future refactor that replaced
tg.Attributes instead of mutating in place would pass every existing test
but silently wipe previous attributes on each Modify — the most likely
real-world ALB/TG bug. Add sequential-merge tests for both TG and LB
handlers that exercise multi-key merge, same-key overwrite, and
cross-survival of unrelated keys.
Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
Spinifex developed by Mulga Defense Corporation is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
It’s designed for developers and operators who need:
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren’t feasible.
Core Components
Spinifex (Compute Service – EC2 Alternative)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU’s QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
- Works with standard AWS tooling
Viperblock (Block Storage – EBS Alternative)
Viperblock is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It’s built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- Supports high performance WAL logs using local NVMe drives to reduce IO traffic to S3.
- In memory read/write block cache for blazing performance.
Predastore (Object Storage – S3-Compatible)
Predastore is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Data is chunked and distributed across nodes using Reed-Solomon erasure coding, making it fault-tolerant and ideal for large-scale or low-bandwidth scenarios.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
- Designed for Control – Run on your own terms, whether that’s in a datacenter, remote site, or sensitive environment.
- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
- Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at https://docs.mulgadc.com for maintaining and installing Spinifex.
Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes.
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
Development Setup
For a complete development environment see the Source Install documentation
Component Repositories
Spinifex coordinates these independent components:
- Predastore - S3-compatible object storage
- Viperblock - EBS-compatible block storage
Each component can be developed independently. See component-specific documentation for focused development guides.
Development Philosophy
Built by Engineers, For Engineers
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Real-World Experience:
- Production AWS service development and operations
- Large-scale infrastructure deployment and management
- Edge computing and resource-constrained environments
- Enterprise security and compliance requirements
AI-Assisted Development
While Spinifex is architected and implemented by experienced engineers, we leverage Claude Code (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
How We Use Claude Code:
- Code Generation: Boilerplate AWS API structures and handlers
- Documentation: Comprehensive development guides and API documentation
- Testing: Test case generation and validation scenarios
- Refactoring: Large-scale code restructuring and optimization
What Remains Human-Driven:
- Architecture Decisions: Core system design and scalability choices
- Security Implementation: Authentication, encryption, and threat modeling
- Performance Optimization: Real-world performance tuning and benchmarking
- Production Operations: Deployment strategies and operational procedures
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
License
Spinifex is open source under the GNU Affero General Public License v3.0. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.