gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 09:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
bbff004b456a6c714ac0aac2c80eda589d317d96
spinifex/Makefile
T
Josh Siviour bbff004b45 feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280) 
* feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep

Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first
read/write helpers against the eks-account-{accountID} bucket so later
stages (oidc keypair, NLB wiring, cluster reconciler) can persist state
without re-inventing CAS/sweep logic.

- ClusterMeta struct holds the DescribeCluster source of truth (status,
  endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN).
- ClusterStatus enum mirrors the AWS EKS string set verbatim.
- PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta
  returns ErrClusterNotFound on key absence so the service boundary can
  map to ResourceNotFoundException.
- SetClusterStatus is revision-checked with bounded CAS retries so the
  reconciler and CreateCluster body cannot wedge each other.
- DeleteClusterPrefix sweeps every clusters/{name}/* key; called from
  DeleteCluster after the OIDC private key is zeroized.

Tests cover round-trip, missing/corrupt blob, idempotent status set,
DELETING transition from every prior state, concurrent revision bump
recovery, sibling-cluster preservation, empty-bucket no-op.

Plan: docs/development/feature/eks-6b-create-cluster.md

* feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize

Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body
will call before launching the K3s server VM. Pure crypto helpers — no
daemon, no launcher, no NATS bus.

- GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster
  (matches eks-v1.md Q8), PEM-marshals the private as PKCS8,
  envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM,
  AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and
  publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS
  bytes so CreateCluster can hand them straight to user-data without
  a follow-up KV read.
- JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA
  verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER,
  alg ES256, use sig). Types are defined locally to avoid a
  handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire
  compat is the contract.
- LoadClusterOIDCPrivateKey is the recovery-path read for future
  stages (reconciler restart, restore-snapshot tooling). Validates
  PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else.
- ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of
  the same length, then deletes. DeleteCluster calls this *first* so a
  crash mid-teardown leaves no recoverable key material. Idempotent
  on missing key — supports retry semantics.

Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1
coordinates), per-cluster key uniqueness, round-trip private-key
decrypt matches JWKS public, wrong-master-key fails, missing-key
ErrClusterNotFound mapping, zeroize idempotency, JWKS survives
zeroize (carries no secret material), padCoord left-pad correctness.

Plan: docs/development/feature/eks-6b-create-cluster.md

* feat(eks): cluster SG + NLB ensure/delete helpers

Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account
SG provisioning + per-cluster NLB plumbing the CreateCluster body will
call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure
interface-call helpers — no daemon, no KV, no NATS bus.

- EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg
  and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged
  spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role.
  Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort
  with continue-on-error sweep.
- EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the
  eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md
  Q13), and the TCP :443 forward listener. Returns ClusterNLB
  {LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster
  can persist every ARN in ClusterMeta. Idempotent on cluster name;
  resource-name length validated against the AWS 32-char cap so a too-long
  cluster name fails with a clear error instead of a generic
  InvalidParameterValue downstream.
- RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM
  ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its
  ENI IP; DeleteCluster calls Deregister before VM termination.
- DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already
  cascades listener cleanup inside the ELBv2 impl. LB sweep error takes
  precedence so DeleteCluster surfaces the more visible failure first
  while the TG sweep still runs.

Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests
hand-rolled recorder fakes; the production VPCService + ELBv2Service
satisfy them structurally so no adapter is needed at Stage 6 wire-up.

Tests cover fresh-creates with right names/tags/ports, idempotent reuse,
mixed-existence (control-plane exists, nodegroup created), name-too-long
rejection, empty-inputs rejection, missing-resource no-op on delete, and
first-error-surfaced sweep continuation on delete.

Plan: docs/development/feature/eks-6b-create-cluster.md

* feat(ec2): RunInstances honors pre-created NetworkInterfaceId

Closes the AWS-parity gap where RunInstances always auto-created a
primary ENI even when the caller specified
NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster
launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the
ENI can be tagged/owned distinctly from the launcher's instance; the
existing path silently created a second ENI and left the caller's
unattached.

Changes:
- ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo
  is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status,
  SecurityGroupIDs) so handlers/ec2/instance stays free of an import on
  handlers/ec2/vpc.ENIRecord.
- daemon/vpc_adapter.go gains daemonENICreator.GetENI translating
  VPCServiceImpl.GetENIRecord into ENIInfo.
- PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId
  after the existing subnet/SG lift block. If non-empty, the loop
  dispatches to a new attachPreCreatedENI helper:
    - Looks up the ENI under accountID.
    - Rejects with InvalidNetworkInterface.InUse if Status == "in-use".
    - Calls AttachENI to mark the ENI in-use and bind it to the new VM.
    - Populates instance.ENIId / ENIMac and ec2.Instance network fields
      (PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]).
    - Skips the auto-create + public-IP-auto-assign paths; the caller is
      expected to manage public IP out-of-band.
- Auto-create path is left untouched for every other code path: callers
  that pass SubnetId without a NetworkInterfaceId still get a fresh ENI
  with the existing semantics.

Tests cover the four shapes: happy path (auto-create not called, attach
called once, instance populated), in-use ENI rejected with the canonical
AWS error code, lookup error surfaces verbatim, attach error surfaces
verbatim.

* feat(eks): K3s control-plane VM launcher with cloud-init user-data

LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the
customer-account control-plane ENI in the supplied subnet with the
control-plane SG, renders cloud-init user-data (first-boot env, OIDC
private key, K3s config.yaml with tls-san + service-account args), then
calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing
at the pre-created ENI. RunInstances failures roll back the ENI so the
caller does not leak a customer-account resource.

TerminateK3sServerVM tears the instance and ENI down for DeleteCluster;
missing identifiers are a no-op so retries stay safe.

ENI + VM both live in the customer account (deviation from plan §7 which
specified a SystemAccount-owned VM); cross-account ENI attach is deferred
to a follow-up bead.

* feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease

NATSBootstrap subscribes to the four single-shot subjects the K3s server VM
publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca)
on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are
envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The
published JWKS is cross-checked against the controller-generated keypair by
kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64
via the new SetClusterCertificateAuthority CAS updater.

ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/
{clusterName} (60s TTL, 20s refresh) and drives:
  - CREATING -> ACTIVE once the four bootstrap KV artifacts are present and
    NLB /healthz responds 200.
  - ACTIVE -> probe /healthz on the configured interval, warn on failure.
  - DELETING / FAILED -> exit (DeleteCluster body owns teardown).
Loss of the leader lease (CAS conflict or different holderID) returns
ErrReconcilerLeaseLost so the caller can re-acquire from another node.

Tested with embedded JetStream + a stub HTTPDoer; race-clean.

* feat(eks): OIDC issuer URL builder + reconciler registry

ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/
{clusterName} URL kube-apiserver advertises as service-account-issuer and
that STS / external IRSA validators dereference. Validates URL safety on the
path segments so a hostile cluster name cannot inject scheme/path fragments.

ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines
with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait
for release), and StopAll (graceful daemon shutdown). RunClusterReconciler
is the production SpawnReconcilerFn: constructs a ClusterReconciler,
acquires the leader lease, runs Run in a goroutine. Both helpers consumed
by the upcoming service_impl CreateCluster body.

* feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies

Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB,
Instance launcher, AMI resolver, master key, gateway base URL, region,
holder ID, reconciler registry) and implements the four lifecycle
methods:

- CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer +
  keypair → K3s server VM (cloud-init carries OIDC private PEM) →
  register ENI IP with NLB target group → spawn NATS bootstrap
  subscriber + per-cluster reconciler via the registry.
- DescribeCluster: KV meta read → eks.Cluster shape, with
  ErrClusterNotFound translated to ResourceNotFoundException at the
  service boundary.
- ListClusters: KV key-prefix scan + sorted maxResults/nextToken
  pagination.
- DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop
  reconciler via registry → deregister NLB target + delete NLB →
  terminate K3s VM → delete SGs → delete cluster KV prefix.

Daemon wiring:

- buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key
  best-effort, resolves the gateway base URL via the same precedence the
  LB-agent block applies later, and passes VPCService + ELBv2Service +
  ImageService directly while adapting InstanceServiceImpl via
  daemonEKSInstanceLauncher (RunInstances passes through; Terminate-
  Instances loops vmMgr.Get + StopOrTerminateInstance in-process).
- SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping
  VPCService.GetSubnet.
- Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster
  via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so
  every per-cluster goroutine releases its lease before exit.

Test surface:

- service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers
  the four lifecycle methods when constructed via the back-compat shim
  (CreateCluster/DeleteCluster → ServerInternal, DescribeCluster →
  ResourceNotFound on empty bucket, ListClusters → empty list).
- daemon_handlers_eks_test: drop the four lifecycle subjects from the
  NotImplemented routing loop; handler-side coverage now lives in
  handlers/eks.
- awserrors: add ResourceNotFound (404) for the DescribeCluster path.

* fix(eks): load master.key from config dir not service data dir

eks_deps read master.key from {BaseDir}/config/master.key, but the daemon
systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service
data dir). admin init writes master.key to the shared config dir, so the
daemon looked in a path that only resolved when a {BaseDir}/config symlink
existed — awsgw has one, the daemon's nested data dir does not. Result:
MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster
collapse to ServerInternal.

master.key is a config artifact written next to spinifex.toml. Read it from
filepath.Dir(d.configPath), matching how the daemon already resolves
predastore.toml and TLS certs. No symlink or provisioning change needed.

* fix(eks): register + resolve eks-server AMI via tag, add import --ami-name

CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every
clean install. Two defects:

1. spx admin images import had no way to name a locally-built appliance AMI:
   --name is a catalog lookup (rejects non-catalog values) and the AMI name
   was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran
   'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all
   failed 'Image name not found in available images'.

2. The launcher resolved DescribeImages by name=='eks-server', a name the
   pipeline could never produce.

Add --ami-name to override the registered AMI name on --file imports (fixes
eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the
spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on
multiple matches. Tag-based resolution survives the planned server+agent →
single EKS AMI unify untouched (role chosen per-instance at launch).

build-system-image.sh now passes --ami-name; Makefile gains
import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when
DEV_INSTALL_EKS=1.

* fix(eks): give K3s server VM NATS token + CA so bootstrap can publish

The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback,
unreachable from the VM) and no auth material, while spinifex NATS requires a
shared token over TLS. The VM could not connect to NATS, so it never published
its four one-shot bootstrap messages and the cluster stayed in CREATING.

Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL
(mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys
hierarchy to scope per-principal creds against).

- K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert,
  required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN,
  SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600.
- EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert.
- buildEKSServiceDeps: read CA from config.NATS.CACert; extract
  resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port).
- k3s-first-boot.sh: build nats CLI args with --token + --tlsca.

* fix(eks): retry transient bootstrap persists, fail cluster on permanent error

The NATS bootstrap subscriber tore down on the first persist error and the
cluster sat in CREATING forever. Core-NATS delivers a publication once, so a
transient KV blip during persist was unrecoverable and a malformed payload
(bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster.

- persistWithRetry rides out transient persist failures in-handler (3 attempts,
  200ms backoff) instead of unsubscribing and waiting for a re-publish that
  never comes.
- errBootstrapPermanent classifies validation failures (malformed envelope,
  empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they
  return immediately.
- spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a
  non-context error, so DescribeCluster surfaces the cause and DeleteCluster
  can reclaim resources.
- MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a
  late bootstrap error cannot clobber a concurrent delete or healthy cluster.
- casUpdateMeta extracts the revision-checked RMW loop shared by the meta
  setters.

* fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart

SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the
bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM
still booting) lost the subscriptions; the VM published its four one-shot
core-NATS messages to nobody and the cluster hung in CREATING forever.

- BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are
  still missing from KV. JWKS is excluded: it persists no artifact and
  re-waiting on a once-published subject would deadlock — its cross-check is
  best-effort across a restart.
- RunForKinds subscribes to a subject subset (nil = all four); Run delegates to
  it. subjectHandlers maps each kind to its handler and rejects unknown kinds.
- SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting
  only on the missing artifacts, so a mid-CREATING restart still drives the
  cluster to ACTIVE.

* fix(eks): DeleteCluster must not wipe KV or report success on teardown failure

DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue
then unconditionally swept the per-cluster KV prefix and returned success. A
failed NLB or VM teardown leaked billable resources with no owning record (the
meta holding their ARNs/IDs was erased) and told the operator the cluster was
gone. Zeroize — a stated security guarantee of no recoverable key material —
had its failure ignored too.

- Zeroize, NLB delete, and VM terminate are now blocking: their failures
  accumulate (errors.Join) and abort the call before the KV sweep, leaving the
  cluster DELETING and returning the error so a retry can still reach the ARNs.
- KV prefix sweep runs only after NLB + VM teardown confirm.
- SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning
  record — and target deregister stays best-effort since DeleteClusterNLB tears
  down the whole NLB + target group.

* fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak

CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and
persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any
failure between EnsureClusterNLB and that persist left the NLB + target group +
listener + backing LB VM created but unrecorded; markFailed did no rollback and
DeleteCluster keys teardown off the persisted ARNs, so the resources leaked.

- Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and
  again after LaunchK3sServerVM (records instance + ENI), before the next
  fallible step.
- A failed create now leaves a FAILED meta carrying the resource references, so
  the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM.
- Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete
  reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture
  extracted from the delete-cluster fixture.

* fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens

ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment
path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves —
https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b
cluster's iss claim was rejected with "issuer host must start with oidc.eks",
so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them.

Reconcile on the path-based shape, which is what is actually reachable on
bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from
the IAM OIDC-provider registration (the full issuer string is the registry key)
and the JWKS signature check, not a host pattern — so the parser drops the host
prefix rule and instead pins accountID/clusterName at fixed positions in a
strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted
iss with extra or missing segments still fails closed.

Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss
parses back to the same (accountID, clusterName).

* fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors

A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the
preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently
in use...") for a cluster-exists condition, and because that maps to a
retryable code the SDK's auto-retry surfaced ResourceInUseException instead of
the original failure. FAILED clusters were also a dead state — visible to
neither a clean recreate nor a clear lifecycle.

- Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with
  cluster-scoped messages; CreateCluster's exists check and DescribeCluster /
  DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes.
- CreateCluster reclaims a FAILED cluster: it purges the prior attempt's
  recorded resources and clears KV before recreating, so a retry (including the
  SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces
  rather than recreating on top of orphans.
- Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path
  (blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown
  confirms).
- FAILED clusters remain observable via Describe (status=FAILED) and List — no
  state where create blocks but describe/list show nothing.

* fix(eks): fail CREATING clusters that exceed create timeout

A per-cluster reconciler that never observes a ready control plane (bootstrap
artifacts missing or healthz unreachable) previously stayed CREATING forever.
Add a configurable create timeout (default 15m): once meta.CreatedAt is older
than the deadline and the cluster is still not ready, the reconciler marks it
FAILED with a reason, exiting the loop. Within the deadline it stays CREATING.

* fix(eks): surface ACTIVE cluster health in describe-cluster

The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE
cluster, so describe-cluster kept reporting a dead control plane as healthy.
Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE
reconcile branch (recording the failure reason, clearing on recovery), and map
a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in
describe-cluster.

* test(eks): add plan-mandated CreateCluster/DeleteCluster service cases

service_impl_test only covered shim mode. Add the four cases the plan's Testing
section requires: CONFIG_MAP auth mode rejected (and API accepted), missing
subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim
short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven
by forcing a teardown failure and asserting the OIDC key is already wiped.

* refactor(eks): return OIDC private-key PEM from generator, drop re-read

CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second
KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the
generator already had in hand. Return the plaintext PEM (alongside the JWKS)
from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now
-unused crypto/x509 + encoding/pem imports in service_impl.

* fix(eks): make CreateCluster/DeleteCluster failures diagnosable

Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare
ServerInternal with no request-time log, leaving the client a 'contact AWS
re:Post' message and zero cause. Enumerate the specific missing deps and log
them at ERROR, returning ServiceUnavailable (an operator/config fault, not an
unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a
missing control-plane AMI is logged and surfaced as ServiceUnavailable too.

* fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them

WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json,
which ignores their locationName tags and emits Go PascalCase keys. EKS
clients parse restjson on the lowercase locationName, so every field of
list/describe/create/delete responses was silently dropped (empty endpoint,
null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same
marshaler the SDK uses for restjson bodies — which honors locationName.

Adds wire-shape tests asserting lowercase keys present and PascalCase absent.

* feat(eks): launch K3s server VM via boot-agnostic system-instance path

The EKS K3s control-plane VM launched through the plain tenant RunInstances
path, so it had only its VPC ENI and no management-bridge NIC -- no route to
the daemon's NATS endpoint. It never published its bootstrap subjects and the
cluster stayed CREATING.

Generalize LaunchSystemInstance to a boot-agnostic contract supporting both
system-instance boot styles:

  - BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent
  - BootAMI:    AMI boot + cloud-init seed (q35)    -- EKS K3s server VM

The launch contract moves to a neutral handlers/sysinstance package so EKS can
launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports
the types as aliases. The BootAMI branch mirrors the daemon's
Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record
between Insert and Launch, so the cloud-init network-config renders a static
mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the
on-link address the VM's mgmt NIC can reach), falling back to the gateway-host
resolution when no mgmt bridge exists.

Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC
(10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the
bridge IP. No AMI rebuild required.

* fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge

TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every
TerminateSystemInstance error -- as blocking. Because instance terminate is
async, the immediate ENI delete races it: the first attempt fails
InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases
the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into
purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep
and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING
permanently, leaking its control-plane + nodegroup SGs and KV meta.

Classify the teardown errors instead of blanket-propagating:

  - ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound
    -> idempotent success (the cascade or a prior retry already deleted it);
    must not block the downstream SG + KV sweep.
  - ENI InvalidNetworkInterface.InUse (VM still terminating) and any other
    error -> retryable; surfaced so the cluster stays DELETING and the
    reconciler retries once the instance releases the ENI.
  - Instance already gone -> idempotent. TerminateSystemInstance now returns a
    sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates
    via errors.Is, so a retried delete after the VM drained completes.

Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier
(substring match through %w wrapping) and route the lone ad-hoc strings.Contains
guard in daemon/vm_adapters.go through it.

Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI +
VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak.

* fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key

The controller pre-seeds OIDCJWKSKey at create time, so its presence
proved nothing about the running cluster's actual signing key. A cluster
could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's
published JWKS was ever cross-checked against the generated keypair.

Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after
assertJWKSMatch passes. The reconciler's ACTIVE gate and
BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS.
assertJWKSMatch is map-based and also rejects kty mismatch.

* fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost

startCluster reimplemented the 5-case gateway-host precedence inline,
duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL).
If one drifted, the OIDC issuer host embedded in cluster tokens would
diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch
and cluster-wide token rejection.

startCluster now calls resolveGatewayHost and derives the multi-node
mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost
carries the canonical case enumeration; a table test pins all five
branches plus the case-1 skip guards and the no-host fallthrough.

* fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash

Three issues found while diagnosing why K3s clusters never leave CREATING:

1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in
   default_kernel_opts, so Linux binds /dev/console to the framebuffer and the
   host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and
   k3s-first-boot output were invisible. Reorder so console=ttyS0 is last.

2. k3s-first-boot.initd: command_background was "no" while output_log/error_log
   were set; start-stop-daemon rejects --stdout/--stderr without --background
   ("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and
   the cluster wedged in CREATING. Background it (the publisher waits up to 5min
   on K3s /healthz and must not block the runlevel anyway) and route its output
   to the serial console, which the orchestrator captures (the VM has no login).

3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log
   to the console so the apiserver's own failure is visible host-side.

* fix(eks): seed K3s apiserver a public OIDC key, not the private key

buildK3sUserData pointed both kube-apiserver service-account args at the same
private-key file. --service-account-key-file requires a PUBLIC key — k8s
ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any
valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never
reaches /healthz, first-boot never publishes, and the cluster wedges in
CREATING.

Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and
seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point
service-account-key-file at the public file; keep service-account-signing-key-file
on the private file. Runtime-verified: the apiserver crash is gone and K3s now
starts and schedules system pods.

* feat(eks): egress-only SNAT + DNS for K3s control-plane VM

The K3s server VM (a hidden system instance) could not pull system-pod
images: it had no internet egress and no working resolver, so
coredns/traefik/metrics-server never started and /healthz timed out.

Egress: give the single control-plane VM egress-only internet, confined
to its /32, with no inbound exposure and no customer-visible resource.

- policy/route.go: SystemInstanceEgressPriority (1200, above the drop
  gate) + AddSystemInstanceEgress/Delete — a reroute scoped to
  ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected.
- policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not
  dnat_and_snat, so no inbound path), idempotent on its pool external IP.
- external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the
  gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW
  egress path; ADR-0006 S1/S4/S5 preserved).
- subscribers: vpc.add/delete-system-egress topics + handlers.
- handlers/eks: allocate a hidden (ManagedBy=eks) pool address on
  CreateCluster and publish add; release + publish delete on teardown.
  EIP dep wired through EKSServiceDeps + the daemon.

DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a
dangling symlink, so DHCP-supplied resolvers are never persisted. Write
a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not
write_files — a write_files entry follows the dead symlink, fails, and
aborts the whole block, dropping first-boot.env + the k3s config).

* feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes

Switch the K3s server VM from embedded etcd to embedded SQLite (kine)
for the single-node v1, so CreateCluster reaches a healthy control plane.
etcd's per-commit fsync stalls past the apiserver deadline on viperblock's
deferred-durability block path; SQLite tolerates it.

- k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true
  so the reconciler's anonymous /healthz NLB probe returns 200.
- first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth
  =false) -> authenticated 'kubectl get --raw /readyz'.
- first-boot publish: emit a JSON BootstrapEnvelope for all four subjects
  (token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before.
- nats pub: --stdin -> --force-stdin (nats CLI 0.4.0).
- manifest: add jq (builds the bootstrap-message JSON envelopes).

* feat: EKS reconciler probes control-plane mgmt IP until DNS lands

The host-side cluster reconciler cannot resolve or route to the VPC-internal
NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed
out in CREATING and was marked FAILED even though the control plane was healthy.

Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver
endpoint reachable from the daemon. Capture the mgmt IP from the launch result
(SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta.
ControlPlaneMgmtIP) and build the reconciler health URL as
https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify,
so the mgmt IP need not be in the apiserver cert SAN.

Workaround until authoritative DNS + a host-reachable NLB front-end land, at
which point spawnReconciler reverts to the NLB endpoint.
2026-06-03 22:29:31 +10:00

363 lines
14 KiB
Makefile
Raw Blame History

GO_PROJECT_NAME := spx
SHELL := /bin/bash
# Detect architecture for cross-platform support
ARCH := $(shell uname -m)
ifeq ($(ARCH),x86_64)
GO_ARCH := amd64
AWS_ARCH := x86_64
# On x86, we can run ARM VMs via emulation
QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
else ifeq ($(ARCH),aarch64)
GO_ARCH := arm64
AWS_ARCH := aarch64
# On ARM, we can run x86 VMs via emulation
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
else ifeq ($(ARCH),arm64)
GO_ARCH := arm64
AWS_ARCH := aarch64
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
else
$(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
endif
# Ask Go whether workspace mode is active
IN_WORKSPACE := $(shell go env GOWORK)
# Use -mod=mod unless Go reports an active workspace path
ifeq ($(IN_WORKSPACE),)
GO_BUILD_MOD := -mod=mod
else ifeq ($(IN_WORKSPACE),off)
GO_BUILD_MOD := -mod=mod
else
GO_BUILD_MOD :=
endif
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
# propagates through the filter — otherwise a test failure is swallowed by
# grep's own (success) exit code and preflight prints "passed" on red.
ifdef QUIET
_Q = @
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
_SECQ = >
else
_Q =
_COVQ =
_RACEQ =
_SECQ = 2>&1 | tee
endif
build: go_build build-installer build-lb-agent
# Build spinifex-ui frontend (requires pnpm)
build-ui:
@echo -e "\n....Building spinifex-ui frontend...."
cd spinifex/services/spinifexui/frontend && pnpm build
# GO commands
VERSION ?= $(shell git describe --tags --always --dirty)
COMMIT ?= $(shell git rev-parse --short HEAD)
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
go_build:
@echo -e "\n....Building $(GO_PROJECT_NAME)"
GOFIPS140=v1.0.0 go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
build-installer:
@echo -e "\n....Building spinifex-installer"
GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
build-lb-agent:
@echo -e "\n....Building lb-agent (static)"
CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-server)
ifndef IMAGE
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
endif
@if [ -f scripts/images/$(IMAGE).conf ]; then \
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf $(if $(IMPORT),--import); \
elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf $(if $(IMPORT),--import); \
else \
echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
exit 1; \
fi
build-eks-server-image: ## Build the eks-server AMI (K3s control-plane variant; IMPORT=1 to register it)
$(MAKE) build-system-image IMAGE=eks-server
import-eks-server-image: ## Build + register the eks-server AMI (requires a running cluster)
$(MAKE) build-system-image IMAGE=eks-server IMPORT=1
build-eks-agent-image: ## Build the eks-agent AMI (K3s worker variant)
$(MAKE) build-system-image IMAGE=eks-agent
MICROVM_OUT_DIR := build/microvm
MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
# Grouped target — script writes both files in one run.
$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
./scripts/build-microvm-image.sh
# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
# build-microvm-image stays correctly stale-aware.
bin/lb-agent:
$(MAKE) build-lb-agent
build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
sudo install -d /usr/share/spinifex/microvm
sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
go_run:
@echo -e "\n....Running $(GO_PROJECT_NAME)...."
$(GOPATH)/bin/$(GO_PROJECT_NAME)
# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
# Use this before committing to catch CI failures locally.
preflight:
@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
@echo -e "\n ✅ Preflight passed — safe to commit."
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
# infrastructure required, safe to run in CI without a cluster.
test-harness:
@echo -e "\n....Running e2e harness unit tests...."
$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
# Validate docs/service-interfaces.yaml. Schema check + cross-reference
# of services/suites/fixtures + on-disk path existence. Subject content
# vs source is enforced separately in Bead 5 drift lint.
manifest-check:
@echo -e "\n....Checking service-interfaces.yaml...."
@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
# Run unit tests
test:
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
LOG_IGNORE=1 go test -timeout 120s ./spinifex/...
# Run unit tests with coverage profile
COVERPROFILE ?= coverage.out
test-cover:
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... $(_COVQ)
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
# Run unit tests with race detector
test-race:
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
test-actions:
@echo -e "\n....Running action tests...."
LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
# Check that new/changed code meets coverage threshold (runs tests first)
diff-coverage: test-cover
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
bench:
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
$(MAKE) easyjson
LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
run:
$(MAKE) go_build
$(MAKE) go_run
# Fast iteration: build + install binary + restart all services.
# Microvm artifacts are reinstalled when they already exist on disk — the rule's
# input timestamps drive a rebuild only if anything actually changed. On a fresh
# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
# `make install-microvm` explicitly the first time.
deploy: build
sudo install -m 755 bin/spx /usr/local/bin/spx
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
$(MAKE) install-microvm; \
else \
echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
fi
sudo systemctl daemon-reload
sudo systemctl restart spinifex.target
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
# Not needed for code-only changes — use deploy for those.
reinstall:
scripts/dev-install.sh
clean:
rm -f ./bin/$(GO_PROJECT_NAME)
rm -rf spinifex/services/spinifexui/frontend/dist
install-system:
@echo -e "\n....Installing system dependencies for $(ARCH)...."
@echo "QEMU packages: $(QEMU_PACKAGES)"
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confold" \
nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
ovmf qemu-efi-aarch64 \
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
ovn-central ovn-host openvswitch-switch dhcpcd-base
install-go:
@echo -e "\n....Installing Go 1.26.4 for $(ARCH) ($(GO_ARCH))...."
@if [ ! -d "/usr/local/go" ]; then \
curl -L https://go.dev/dl/go1.26.4.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
else \
echo "Go already installed in /usr/local/go"; \
fi
@echo "Go version: $$(go version)"
install-aws:
@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
@if ! command -v aws >/dev/null 2>&1; then \
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
unzip -o awscliv2.zip; \
./aws/install; \
rm -rf awscliv2.zip aws/; \
else \
echo "AWS CLI already installed"; \
fi
quickinstall: install-system install-go install-aws
@echo -e "\n✅ Quickinstall complete for $(ARCH)."
@echo " Please ensure /usr/local/go/bin is in your PATH."
@echo " Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
lint:
@echo "Running golangci-lint..."
$(_Q)golangci-lint run ./...
@echo " golangci-lint ok"
# Auto-fix all linter issues that have fixers
fix:
golangci-lint run --fix ./...
# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
govulncheck:
@echo "Running govulncheck..."
$(_Q)go tool govulncheck ./...
@echo " govulncheck ok"
# Build release tarballs — use distro-ARCH for single arch, distro for both
distro: distro-amd64 distro-arm64
@echo ""
@echo "Distribution tarballs:"
@ls -lh dist/*.tar.gz
@echo ""
@cat dist/*.sha256
distro-amd64:
@echo "Building spinifex $(VERSION) linux/amd64..."
@mkdir -p dist/
docker buildx build \
--platform linux/amd64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/amd64/ \
../
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
echo "[distro-amd64] staging microvm artifacts into tarball"; \
mkdir -p dist/amd64/microvm; \
cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
else \
echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
echo "[distro-amd64] run 'make build-microvm-image' before 'make distro-amd64'"; \
fi
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
distro-arm64:
@echo "Building spinifex $(VERSION) linux/arm64..."
@mkdir -p dist/
docker buildx build \
--platform linux/arm64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/arm64/ \
../
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
distro-clean:
rm -rf dist/
# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
#
# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
# make's own flag and does NOT reach ansible.
# make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
EXTRA_VARS ?=
_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
ansible-dev-preflight:
cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
ansible-dev-teardown:
cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
ansible-dev-install:
cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
ansible-dev-reset:
cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
ansible-dev-deploy:
cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
ansible-dev-status:
cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
ansible-dev-logs:
cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
# Snapshot / restore require an explicit -e snapshot_name=<name>. The
# wrapper passes ANSIBLE_EXTRA through so devs can run
# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
ansible-dev-snapshot:
cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
ansible-dev-restore:
cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
ansible-dev-version:
cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
ansible-dev-vpc:
cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
# Multi-node cluster bootstrap against a tofu env (env1, env2, bryn, etc).
# Inventory is generated from scripts/tofu-cluster/envs/<env>.tfvars.
# make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
ENV ?=
POOL ?=
ansible-cluster-bootstrap:
@test -n "$(ENV)" || { echo "ENV=<env> required (e.g. env1, env2)"; exit 64; }
cd scripts/ansible && CLUSTER_ENV=$(ENV) ansible-playbook -i inventory/tofu.py playbooks/cluster-bootstrap.yml \
-e cluster_env=$(ENV) \
$(if $(POOL),-e cluster_external_pool=$(POOL),) \
$(_ANSIBLE_EXTRA)
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-server-image import-eks-server-image build-eks-agent-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
lint fix govulncheck \
distro distro-amd64 distro-arm64 distro-clean \
ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy ansible-dev-status ansible-dev-logs ansible-dev-snapshot ansible-dev-restore ansible-dev-version ansible-dev-vpc \
ansible-cluster-bootstrap
Reference in New Issue View Git Blame Copy Permalink