chore(ansible): remove ansible tree, uplift to mulga/ansible/ (#287)

Ansible orchestrates the whole stack (spinifex + predastore + viperblock +
OVN + tofu hosts) and its spec/beads live in mulga. Moving it to top-level
mulga/ansible/ ends the per-change submodule-pointer-bump friction.

- git rm -r scripts/ansible/
- Makefile: drop ansible-dev-* / ansible-cluster-bootstrap targets + .PHONY

The tree lands at mulga/ansible/ (flat copy, Option B). History pivot:
tag feat/uplift-ansible-snapshot-pre-move marks the pre-move HEAD.

Plan: docs/development/improvements/ansible-uplift-to-mulga.md (in mulga).
This commit is contained in:
Josh Siviour
2026-06-04 11:23:08 +10:00
committed by GitHub
parent 61d73d1d4e
commit b0227e7eeb
57 changed files with 1 additions and 3865 deletions
-2
View File
@@ -1,2 +0,0 @@
.creds/
*.retry
-222
View File
@@ -1,222 +0,0 @@
# Ansible Dev Lifecycle (parallel to bash scripts)
Experimental Ansible-based lifecycle for a single-node Spinifex dev host.
Lives alongside the existing `scripts/dev-*.sh` and `scripts/reset-dev-env.sh`;
neither set is removed. Decide which to keep after parity testing.
Plan: `docs/development/improvements/ansible-dev-lifecycle.md` (mulga top-level).
## Prerequisites
Venv lives at sub-repo root (`spinifex/.venv/`) — one venv per sub-repo,
shared by any future python tooling in spinifex. Bootstrap from `spinifex/`:
```
sudo apt install python3.13-venv
python3 -m venv .venv
source .venv/bin/activate
pip install 'ansible>=9'
ansible-galaxy collection install -r scripts/ansible/requirements.yml
```
Then `cd scripts/ansible` to run playbooks — `ansible.cfg` resolves the
inventory and roles relative to CWD.
## Playbooks
| Playbook | Purpose | Bash parity |
|---|---|---|
| `playbooks/dev-preflight.yml` | Dependency + port check | `dev-setup.sh` §checks |
| `playbooks/dev-teardown.yml` | Wipe state only (no reinstall) | `reset-dev-env.sh` §teardown |
| `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
| `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
| `playbooks/dev-deploy.yml` | Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | `make deploy` |
| `playbooks/dev-status.yml` | Read-only health snapshot (units, ports, OVN/OVS, gateway, counts, drift) | none |
| `playbooks/dev-logs.yml` | Dump journald + `/var/log/spinifex/*` + status/networking/versions into a tarball | none |
| `playbooks/dev-snapshot.yml` | Quiesce spinifex+OVN+OVS, tar state paths to a named bundle, restart | none |
| `playbooks/dev-restore.yml` | Stop services, wipe + untar a named bundle, refresh CA, restart | none |
| `playbooks/dev-version.yml` | Build-vs-install drift report (spx, plugin, microvm) + sub-repo HEADs | none |
| `playbooks/dev-vpc.yml` | Idempotent dev VPC fixture (VPC + public subnet + IGW + default route); `-e dev_vpc_state=absent` tears down | partial overlap with `scripts/iac/examples/*` |
Upcoming (not yet implemented):
- `dev-mode-start.yml` / `dev-mode-stop.yml` — parity with `start-dev.sh` / `stop-dev.sh`
(lowest priority — deprecated bash scripts, CI-only callers).
## Invocation
```
ansible-playbook playbooks/dev-preflight.yml
ansible-playbook playbooks/dev-teardown.yml
ansible-playbook playbooks/dev-install.yml
ansible-playbook playbooks/dev-reset.yml
ansible-playbook playbooks/dev-deploy.yml
ansible-playbook playbooks/dev-status.yml
ansible-playbook playbooks/dev-logs.yml
ansible-playbook playbooks/dev-snapshot.yml -e snapshot_name=before-merge
ansible-playbook playbooks/dev-restore.yml -e snapshot_name=before-merge
ansible-playbook playbooks/dev-version.yml
ansible-playbook playbooks/dev-vpc.yml
```
Or via `make` (from `spinifex/`):
```
make ansible-dev-preflight
make ansible-dev-teardown
make ansible-dev-install
make ansible-dev-reset
make ansible-dev-deploy
make ansible-dev-status
make ansible-dev-logs
make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'
make ansible-dev-restore ANSIBLE_EXTRA='-e snapshot_name=before-merge'
make ansible-dev-version
make ansible-dev-vpc
make ansible-dev-vpc ANSIBLE_EXTRA='-e dev_vpc_state=absent'
```
### When to use which
- First-time / clean box → `ansible-dev-install`
- Iterate on Go code, microvm initramfs, lb-agent → `ansible-dev-deploy` (fast)
- Changed systemd units, helper scripts, logrotate, setup.sh → `ansible-dev-reset` (slow, full rebuild)
- Need a clean slate without reinstall → `ansible-dev-teardown`
- "Is my dev box healthy?" → `ansible-dev-status` (read-only, never mutates)
- Filing a bug / capturing the state of a broken box → `ansible-dev-logs` (writes `/tmp/spinifex-logs-<host>-<ts>.tar.gz`; override `-e logs_since=10min` or `-e logs_include_dmesg=false`)
- Checkpoint before a risky op (branch switch, viperblock surgery, ovn restart cycle) → `ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=<name>'`; roll back via `ansible-dev-restore` instead of a full `dev-reset`. Bundles live in `/var/lib/spinifex-snapshots/<name>.tar.gz` (root-owned, survives `dev-teardown`).
- "Do I need to `ansible-dev-deploy`?" → `ansible-dev-version` (drift-only check; faster than `dev-status`)
- Want a known-good VPC/subnet for manual EC2 testing without writing tofu → `ansible-dev-vpc` (smoketest will then find and use it automatically). Tear down with `ANSIBLE_EXTRA='-e dev_vpc_state=absent'`.
## Variable overrides
Defaults in `vars/defaults.yml`. Override per run with `-e`:
```
ansible-playbook playbooks/dev-teardown.yml -e spinifex_wipe_ssh_keys=true
```
Useful overrides:
| Var | Default | What it does |
|---|---|---|
| `spinifex_wipe_legacy_home` | `true` | Remove `$HOME/spinifex` (legacy dev-mode path) |
| `spinifex_wipe_ssh_keys` | `false` | Remove `~/.ssh/spinifex-key*` |
| `spinifex_wipe_aws_creds` | `false` | Remove spinifex profile from `~/.aws/credentials` |
| `teardown_process_wait_seconds` | `30` | Grace period for processes to exit |
| `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
| `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
| `spinifex_external_mode` | `pool` | External networking mode — `pool` or empty (NAT removed) |
| `spinifex_external_no_external` | `false` | `true` → emit `--no-external` and skip external networking |
| `spinifex_external_pool_start` | `""` | **Required** for `pool` mode — first IP of the static WAN pool |
| `spinifex_external_pool_end` | `""` | **Required** for `pool` mode — last IP of the static WAN pool |
| `spinifex_external_gateway` | `""` | WAN gateway (auto-detected when blank) |
| `spinifex_external_prefix_len` | `""` | WAN prefix length (auto-detected when blank) |
| `spinifex_external_gateway_ip` | `""` | OVN router external-IP override (`--gateway-ip`) |
`pool` mode is mandatory unless you set `spinifex_external_no_external=true`.
`spx admin init` rejects the run without a `--external-pool` range, so the
init role asserts `spinifex_external_pool_start/_end` upfront. Either persist
them in `vars/defaults.yml`, or pass them at invocation:
Direct `ansible-playbook` (use `-e` here — this is ansible's flag):
```
ansible-playbook playbooks/dev-install.yml \
-e spinifex_external_pool_start=192.168.1.150 \
-e spinifex_external_pool_end=192.168.1.250
```
Via `make` (use `EXTRA_VARS=` — make eats its own `-e`):
```
make ansible-dev-install \
EXTRA_VARS="spinifex_external_pool_start=192.168.1.150 spinifex_external_pool_end=192.168.1.250"
```
`dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
pool range, gateway, prefix-len and `gateway_ip` from the existing
`/etc/spinifex/spinifex.toml` before teardown and replays them into
`init`. Legacy `external_mode = "nat"` is coerced to `"pool"` with a warn.
Use `-e` to override.
## Helper scripts
`scripts/aws-as-spinifex.sh` and `scripts/spx-as-spinifex.sh` wrap `aws`
and `spx` invocations in `sg spinifex -c '...'` so they run with the
`spinifex` supplementary group active. Both helpers preset
`AWS_PROFILE=spinifex`. All AWS-calling tasks in the roles use these
via `spinifex_aws_cli` / `spinifex_spx_cli` from `vars/defaults.yml`.
Background: `/etc/spinifex/ca.pem` lives under `drwxr-x--- root:spinifex`,
so the AWS CLI needs `spinifex` group membership to read the CA bundle.
Ansible's `become_user` is a no-op when the target user equals the
calling user (no `sudo -u` invoked → no `initgroups(3)` → supplementary
groups inherited from the operator's shell). On a fresh login the
operator's shell would have `spinifex` (added by `setup.sh`); on shells
that predate the group add it does not. The helpers sidestep this
entirely.
## Leak catalog
`roles/teardown/README.md` tracks every known state source. When teardown
misses something, add it there and in `roles/teardown/tasks/main.yml`.
## Multi-node cluster bootstrap
`playbooks/cluster-bootstrap.yml` provisions a 2+ node Spinifex cluster
across a long-running tofu env (`env1`, `env2`, `bryn`, etc). Inventory
is generated dynamically from `scripts/tofu-cluster/envs/<env>.tfvars`
via `inventory/tofu.py` — first node in `nodes = [...]` is the init host,
the rest are joins.
```bash
make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
```
`POOL` is optional; omit for DHCP fallback. Equivalent without make:
```bash
cd scripts/ansible
CLUSTER_ENV=env1 ansible-playbook -i inventory/tofu.py \
playbooks/cluster-bootstrap.yml \
-e cluster_env=env1 \
-e cluster_external_pool=192.168.1.150-192.168.1.159
```
Plays sequence:
1. **runner (localhost)** — `build` role produces `/tmp/spinifex-local.tar.gz`.
2. **all env hosts (parallel)** — copies tarball, stops any dev-layout
services (`~/spinifex/`), runs `preflight` + `install` roles. `setup.sh`
creates `/etc/spinifex`, the `spinifex` user/group, systemd units, and
installs `/usr/local/bin/spx`.
3. **init host only** — `cluster-init` role runs `spx admin init` in the
background, waits for the formation server, captures the join token.
CA is then copied into `/usr/local/share/ca-certificates/spinifex-ca.crt`
**before** `spinifex.target` starts (required — predastore enforces
strict TLS verify against the system trust store).
4. **join hosts (parallel)** — `cluster-join` role runs `spx admin join`
against the formation server, then installs CA, then starts
`spinifex.target`.
5. **all hosts** — polls `https://<ip>:4432/health` until `awsgw=ok`.
6. **init host only** — `cluster-post` role imports the SSH key + Ubuntu
AMI (both idempotent: skip if already present).
Re-running the playbook against an already-bootstrapped env is safe —
`setup.sh`, `spx admin init --force`, CA copy, key/AMI import are all
idempotent.
### Migration from `bootstrap.sh`
`scripts/tofu-cluster/bootstrap.sh` ran services under `~/spinifex/` as
`tf-user` via `start-dev.sh` with no systemd supervision, and installed
the cluster CA *after* services started — predastore would crash with
`x509: certificate signed by unknown authority` on every first boot.
`cluster-bootstrap.yml` replaces that flow with production-layout
(`/etc/spinifex/` + systemd) and correct CA-before-services ordering.
The first run on an env previously bootstrapped by `bootstrap.sh` will
stop the dev-layout services, install the prod layout, and re-init the
cluster from scratch via `spx admin init --force`.
`bootstrap.sh` stays in-tree for CI and pseudo-multinode envs.
-17
View File
@@ -1,17 +0,0 @@
[defaults]
inventory = inventory/localhost.yml
roles_path = roles
stdout_callback = ansible.builtin.default
host_key_checking = False
retry_files_enabled = False
interpreter_python = auto_silent
forks = 10
[inventory]
# inventory/tofu.py is an executable script inventory used by
# cluster-bootstrap.yml — opted into via `-i inventory/tofu.py` on the
# command line; not default to keep dev-* playbooks pointing at localhost.
enable_plugins = host_list, script, auto, yaml, ini, toml
[callback_default]
result_format = yaml
-6
View File
@@ -1,6 +0,0 @@
---
all:
hosts:
localhost:
ansible_connection: local
ansible_python_interpreter: "{{ ansible_playbook_python }}"
-170
View File
@@ -1,170 +0,0 @@
#!/usr/bin/env python3
"""Dynamic ansible inventory generator from tofu-cluster tfvars.
Reads scripts/tofu-cluster/envs/<env>.tfvars (HCL subset) and emits ansible
JSON inventory. First node in the tfvars `nodes = [...]` list is the init
host (forms the cluster); the rest are join hosts.
Usage:
inventory/tofu.py --list --env env1
inventory/tofu.py --host <ip> --env env1
Selecting the env:
CLUSTER_ENV=env1 inventory/tofu.py --list
inventory/tofu.py --list --env env1
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
from pathlib import Path
def _repo_root() -> Path:
here = Path(__file__).resolve()
for parent in here.parents:
if (parent / "go.work").exists() or (parent / ".gitmodules").exists():
return parent
raise SystemExit(f"could not locate mulga repo root from {here}")
def _tfvars_path(env: str) -> Path:
root = _repo_root()
candidate = root / "scripts" / "tofu-cluster" / "envs" / f"{env}.tfvars"
if not candidate.exists():
raise SystemExit(f"tfvars file not found: {candidate}")
return candidate
_STRING_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*"([^"]*)"\s*$')
_NUMBER_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*([0-9]+)\s*$')
_NODE_FIELD_RE = re.compile(r'(\w+)\s*=\s*"([^"]*)"')
def _parse_tfvars(path: Path) -> dict:
"""Minimal tfvars parser — handles the subset used by envs/*.tfvars.
Supports:
key = "string"
key = 12345
nodes = [
{ name = "x", wan_ip = "1.2.3.4", ... },
...
]
"""
out: dict = {"nodes": []}
text = path.read_text()
# Strip line comments
text = re.sub(r"#.*", "", text)
# Pull the nodes block out first, then process remaining lines.
nodes_match = re.search(r"nodes\s*=\s*\[(.+?)\]\s*$", text, re.DOTALL | re.MULTILINE)
if nodes_match:
nodes_body = nodes_match.group(1)
for entry in re.finditer(r"\{([^}]+)\}", nodes_body):
fields = dict(_NODE_FIELD_RE.findall(entry.group(1)))
if "name" in fields and "wan_ip" in fields:
out["nodes"].append(fields)
text = text.replace(nodes_match.group(0), "")
for line in text.splitlines():
if m := _STRING_RE.match(line):
out[m.group(1)] = m.group(2)
elif m := _NUMBER_RE.match(line):
out[m.group(1)] = int(m.group(2))
return out
def _resolve_ssh_key(value: str | None) -> str:
if not value:
return os.path.expanduser("~/.ssh/tf-user-ap-southeast-2")
return os.path.expanduser(value)
def _build_inventory(env: str) -> dict:
data = _parse_tfvars(_tfvars_path(env))
nodes = data.get("nodes") or []
if not nodes:
raise SystemExit(f"{env}.tfvars has no nodes")
ssh_user = data.get("ssh_user", "tf-user")
ssh_key = _resolve_ssh_key(data.get("ssh_public_key_path", "").replace(".pub", ""))
hostvars: dict[str, dict] = {}
all_hosts: list[str] = []
init_hosts: list[str] = []
join_hosts: list[str] = []
for idx, node in enumerate(nodes):
ip = node["wan_ip"]
role = "init" if idx == 0 else "join"
hostvars[ip] = {
"ansible_host": ip,
"ansible_user": ssh_user,
"ansible_ssh_private_key_file": ssh_key,
"ansible_ssh_common_args": "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR",
"spinifex_node_name": node["name"],
"spinifex_node_role": role,
"spinifex_node_index": idx,
"spinifex_wan_ip": ip,
}
all_hosts.append(ip)
(init_hosts if role == "init" else join_hosts).append(ip)
# Group-level vars derived from tfvars (operator can override with -e).
cluster_init_host = init_hosts[0]
group_vars = {
"cluster_env": env,
"cluster_init_host": cluster_init_host,
"cluster_init_ip": cluster_init_host,
"spinifex_node_count": len(nodes),
"spinifex_region": data.get("region", "ap-southeast-2"),
"spinifex_az": data.get("az", "ap-southeast-2a"),
}
if cidr := data.get("env_cidr"):
group_vars["cluster_env_cidr"] = cidr
inventory = {
"_meta": {"hostvars": hostvars},
env: {"hosts": all_hosts, "vars": group_vars},
"cluster": {"children": [env]},
"init": {"hosts": init_hosts},
"join": {"hosts": join_hosts},
}
return inventory
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--list", action="store_true", help="emit full inventory")
parser.add_argument("--host", help="emit hostvars for a single host (ignored — _meta carries them)")
parser.add_argument(
"--env",
default=os.environ.get("CLUSTER_ENV"),
help="tofu env name (matches scripts/tofu-cluster/envs/<env>.tfvars); also via CLUSTER_ENV env var",
)
args = parser.parse_args()
if not args.env:
sys.stderr.write("error: --env required (or CLUSTER_ENV)\n")
return 64
if args.host:
json.dump({}, sys.stdout)
sys.stdout.write("\n")
return 0
inventory = _build_inventory(args.env)
json.dump(inventory, sys.stdout, indent=2)
sys.stdout.write("\n")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,330 +0,0 @@
---
# cluster-bootstrap.yml — multi-node Spinifex cluster bootstrap against a
# tofu-provisioned env. Inventory comes from inventory/tofu.py which parses
# scripts/tofu-cluster/envs/<env>.tfvars.
#
# Usage:
# cd spinifex/scripts/ansible
# ansible-playbook -i inventory/tofu.py playbooks/cluster-bootstrap.yml \
# -e cluster_env=env1 -e cluster_external_pool=192.168.1.150-192.168.1.159
#
# Or via the make wrapper:
# make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
#
# Behaviour:
# - all hosts: preflight + install (setup.sh) — production layout
# - init host: spx admin init (backgrounded) + CA install + spinifex.target
# - join hosts: spx admin join + CA install + spinifex.target (parallel)
# - init host: import ssh key + ubuntu AMI (idempotent)
- name: Cluster bootstrap — build artifact once on runner
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
tasks:
- name: assert cluster_env supplied
ansible.builtin.assert:
that: cluster_env | default('') | length > 0
fail_msg: "Pass -e cluster_env=<env> (matches scripts/tofu-cluster/envs/<env>.tfvars)"
roles:
- build
- name: Cluster bootstrap — install (all nodes, parallel)
hosts: "{{ cluster_env }}"
gather_facts: true
serial: 100%
vars:
install_run_ovn: false # we drive setup-ovn ourselves with role-aware args
cluster_bootstrap_purge: true # wipe stale single-node predastore.toml + raft data
vars_files:
- ../vars/defaults.yml
pre_tasks:
- name: copy build tarball to each node
ansible.builtin.copy:
src: "{{ build_tarball_path }}"
dest: "{{ build_tarball_path }}"
mode: "0644"
- name: stop dev-layout services if running (migration from ~/spinifex/)
become: true
# Bracket trick on the first letter prevents the pattern literal in the
# shell command line from matching the running shell itself (pkill -f
# against literal patterns is otherwise self-killing — rc=-9).
# `spx [a]dmin` catches lingering formation servers from prior bootstrap
# attempts that own port 4432 and steal joins from the new init.
ansible.builtin.shell:
cmd: |
pkill -9 -f "[b]in/spx service" 2>/dev/null || true
pkill -9 -f "spx [a]dmin" 2>/dev/null || true
pkill -9 -f "spinifex-[u]i" 2>/dev/null || true
pkill -9 -f "[n]ats-server" 2>/dev/null || true
pkill -9 -f "[p]redastore" 2>/dev/null || true
pkill -9 -f "[v]iperblock" 2>/dev/null || true
sleep 2
rm -rf "{{ ansible_env.HOME }}/spinifex" 2>/dev/null || true
exit 0
changed_when: false
roles:
- install
# Wipe OVN/OVS state before setup-ovn runs. Without this, re-bootstrapping a
# previously-built cluster leaves stale chassis rows in the SB DB, stale
# external_ids on Open_vSwitch, stale br-int flows + ports, stale
# systemd-networkd unit files, and stale ovs-monitor-ipsec strongSwan certs.
# Symptom: ovn-controller flows reference dead encap endpoints; IPsec SAs
# fail AUTHENTICATION_FAILED; new joins shadow ghosts under the same
# node-name.
- name: Cluster bootstrap — OVN/OVS state reset (all nodes, parallel)
hosts: "{{ cluster_env }}"
gather_facts: false
serial: 100%
vars_files:
- ../vars/defaults.yml
tasks:
- name: reset — stop OVN/OVS services
become: true
ansible.builtin.systemd:
name: "{{ item }}"
state: stopped
loop:
- ovn-controller
- ovn-central
- openvswitch-ipsec
- openvswitch-switch
failed_when: false
- name: reset — delete OVN NB/SB databases
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ ovn_nb_db }}"
- "{{ ovn_sb_db }}"
- name: reset — delete OVS conf.db + system-id (force fresh ovsdb)
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /etc/openvswitch/conf.db
- /etc/openvswitch/.conf.db.~lock~
- /etc/openvswitch/system-id.conf
- name: reset — drop spinifex-managed systemd-networkd unit files
become: true
ansible.builtin.shell: |
rm -f /etc/systemd/network/14-spinifex-*.netdev \
/etc/systemd/network/15-spinifex-*.netdev \
/etc/systemd/network/15-spinifex-*.network \
/etc/systemd/network/16-spinifex-*.network
networkctl reload 2>/dev/null || true
changed_when: true
- name: reset — enumerate stale spinifex veth/tap netdevs
ansible.builtin.shell: |
ip -o link show | awk -F': ' '{print $2}' | cut -d'@' -f1 | \
grep -E '^(veth-wan-br|veth-wan-ovs|spx-ext-|spx-tap-|spx-)' || true
register: reset_links
changed_when: false
- name: reset — delete stale netdevs
become: true
ansible.builtin.command: "ip link delete {{ item }}"
loop: "{{ reset_links.stdout_lines }}"
changed_when: true
failed_when: false
- name: reset — clear ovs-monitor-ipsec strongSwan state
become: true
ansible.builtin.shell: |
rm -rf /var/lib/openvswitch/pki 2>/dev/null || true
rm -f /etc/ipsec.d/certs/ovs-* /etc/ipsec.d/private/ovs-* 2>/dev/null || true
rm -f /etc/ipsec.conf.d/ovs-* 2>/dev/null || true
changed_when: true
- name: reset — start openvswitch-switch with fresh conf.db
become: true
ansible.builtin.systemd:
name: openvswitch-switch
state: started
daemon_reload: true
- name: Cluster bootstrap — OVN central (init host only)
hosts: init
gather_facts: false
vars_files:
- ../vars/defaults.yml
tasks:
- name: setup-ovn --management on init host
become: true
ansible.builtin.command:
cmd: >-
/usr/local/share/spinifex/setup-ovn.sh
--management
--encap-ip={{ spinifex_wan_ip }}
--node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
changed_when: true
- name: Cluster bootstrap — OVN chassis (join hosts, parallel)
hosts: join
gather_facts: false
vars_files:
- ../vars/defaults.yml
tasks:
- name: setup-ovn --ovn-remote on join hosts
become: true
ansible.builtin.command:
cmd: >-
/usr/local/share/spinifex/setup-ovn.sh
--ovn-remote=tcp:{{ cluster_init_host }}:6642
--encap-ip={{ spinifex_wan_ip }}
--node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
changed_when: true
- name: Cluster bootstrap — init host (start formation server)
hosts: init
gather_facts: false
vars_files:
- ../vars/defaults.yml
roles:
- cluster-init
- name: Cluster bootstrap — join hosts run spx admin join (parallel)
hosts: join
gather_facts: false
serial: 100%
vars_files:
- ../vars/defaults.yml
roles:
- cluster-join
- name: Cluster bootstrap — wait for init to finish writing multi-node configs
hosts: init
gather_facts: false
vars_files:
- ../vars/defaults.yml
tasks:
- name: wait for spx admin init async job to complete
become: true
ansible.builtin.async_status:
jid: "{{ cluster_init_async.ansible_job_id }}"
register: cluster_init_done
until: cluster_init_done.finished
retries: 60
delay: 5
- name: dump init log on failure
ansible.builtin.command: tail -100 /tmp/spx-admin-init.log
register: cluster_init_log
changed_when: false
when: cluster_init_done.rc | default(1) != 0
- name: assert init exited cleanly
ansible.builtin.assert:
that: cluster_init_done.rc == 0
fail_msg: |
spx admin init failed (rc={{ cluster_init_done.rc }})
{{ cluster_init_log.stdout | default('(log not captured)') }}
- name: confirm multi-node predastore.toml written
become: true
ansible.builtin.shell: |
if grep -q '127\.0\.0\.1' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null; then
echo "ERROR: predastore.toml still references loopback — multi-node config not written"
exit 1
fi
grep -c '^\[\[nodes\]\]' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null || echo 0
register: predastore_node_count
changed_when: false
- name: Cluster bootstrap — install CA on all nodes (before service start)
hosts: "{{ cluster_env }}"
gather_facts: false
vars_files:
- ../vars/defaults.yml
tasks:
- name: install CA into system trust
become: true
ansible.builtin.copy:
src: "{{ spinifex_ca_src }}"
dest: "{{ spinifex_ca_trust_path }}"
mode: "0644"
remote_src: true
- name: rebuild CA bundle
become: true
ansible.builtin.command: update-ca-certificates
changed_when: true
# spinifex-ui reverse-proxies localhost:9999 (awsgw) and localhost:8443
# (predastore) — hardcoded in spinifex/services/spinifexui/spinifexui.go.
# spx admin init binds those services to BindIP only, so the loopback
# connection fails (BadGateway in UI). Patch to wildcard bind; cert
# already includes 127.0.0.1 in IP SANs.
- name: patch awsgw bind to 0.0.0.0 (UI proxies via localhost)
become: true
ansible.builtin.replace:
path: "{{ spinifex_config_file }}"
regexp: '^(\s*host\s*=\s*")[^"]+(:9999")'
replace: '\g<1>0.0.0.0\g<2>'
- name: patch predastore bind to 0.0.0.0 (UI proxies via localhost)
become: true
ansible.builtin.replace:
path: "{{ spinifex_config_file }}"
regexp: '^(\s*host\s*=\s*")[^"]+(:8443")'
replace: '\g<1>0.0.0.0\g<2>'
- name: Cluster bootstrap — start services (init first, then joins)
hosts: init
gather_facts: false
vars_files:
- ../vars/defaults.yml
tasks:
- name: start spinifex.target on init
become: true
ansible.builtin.systemd:
name: spinifex.target
state: started
- name: Cluster bootstrap — start services on join hosts
hosts: join
gather_facts: false
serial: 100%
vars_files:
- ../vars/defaults.yml
tasks:
- name: start spinifex.target on joiners
become: true
ansible.builtin.systemd:
name: spinifex.target
state: started
- name: Cluster bootstrap — readiness check (all nodes)
hosts: "{{ cluster_env }}"
gather_facts: false
vars_files:
- ../vars/defaults.yml
tasks:
- name: wait for awsgw responsiveness
ansible.builtin.uri:
url: "https://{{ spinifex_wan_ip }}:{{ cluster_formation_port }}/health"
validate_certs: false
return_content: true
register: cluster_health
until:
- cluster_health.status == 200
- cluster_health.json.service_health.awsgw == "ok"
retries: 30
delay: 5
- name: Cluster bootstrap — post-bootstrap fixtures (init host only)
hosts: init
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- cluster-post
-14
View File
@@ -1,14 +0,0 @@
---
# In-place artifact swap: rebuilds + replaces binaries/microvm artifacts on
# an already-installed node, then restarts spinifex.target. Use when iterating
# on code without altering systemd units, users, dirs, or running the full
# smoketest. For first-time or unit/script changes use dev-install / dev-reset.
- name: Dev deploy — build + swap binaries + restart (no setup.sh, no smoketest)
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- build
- deploy
-67
View File
@@ -1,67 +0,0 @@
---
- name: Dev install — build, install, init, smoketest (idempotent)
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
pre_tasks:
- name: preflight
ansible.builtin.include_role:
name: preflight
- name: health — config file present?
ansible.builtin.stat:
path: "{{ spinifex_config_file }}"
register: health_config
- name: health — spinifex.target active?
become: true
ansible.builtin.command: systemctl is-active spinifex.target
register: health_target
changed_when: false
failed_when: false
- name: health — awsgw responds?
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: aws ec2 describe-regions
register: health_gw
changed_when: false
failed_when: false
- name: health — spx binary installed?
ansible.builtin.stat:
path: /usr/local/bin/spx
register: health_spx
- name: health — set already_installed fact
ansible.builtin.set_fact:
spinifex_already_installed: >-
{{ health_config.stat.exists
and health_spx.stat.exists
and (health_target.stdout | trim) == 'active'
and health_gw.rc == 0 }}
- name: health — summary
ansible.builtin.debug:
msg: |
Health:
spx binary: {{ 'present' if health_spx.stat.exists else 'missing' }}
config: {{ 'present' if health_config.stat.exists else 'missing' }}
target: {{ health_target.stdout | trim }}
awsgw: {{ 'ok' if health_gw.rc == 0 else 'not responding' }}
Already installed: {{ spinifex_already_installed }}
{% if spinifex_already_installed %}
→ Skipping build/install/init. Smoketest only.
{% else %}
→ Running full build → install → init → smoketest.
{% endif %}
roles:
- { role: build, when: not (spinifex_already_installed | default(false)) }
- { role: install, when: not (spinifex_already_installed | default(false)) }
- { role: init, when: not (spinifex_already_installed | default(false)) }
- smoketest
-17
View File
@@ -1,17 +0,0 @@
---
# Capture spinifex + ovn/ovs + qemu/nbdkit logs from journald and
# /var/log/spinifex/ into a single tarball for bug reports.
# Tail mode is intentionally not provided — use `journalctl -u 'spinifex-*' -f`
# directly; ansible is the wrong tool for follow.
- name: Dev logs — dump journald + varlog + status into a bundle
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
vars:
logs_since: "1h"
logs_output_dir: "/tmp"
logs_include_dmesg: true
roles:
- logs
@@ -1,8 +0,0 @@
---
- name: Dev preflight — validate host dependencies
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- preflight
-72
View File
@@ -1,72 +0,0 @@
---
- name: Dev reset — capture settings, wipe, rebuild, reinstall, re-init, smoketest
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
pre_tasks:
- name: capture — check existing spinifex.toml
ansible.builtin.stat:
path: "{{ spinifex_config_file }}"
register: reset_config_stat
- name: capture — slurp existing config
ansible.builtin.slurp:
src: "{{ spinifex_config_file }}"
register: reset_config_raw
when: reset_config_stat.stat.exists
- name: capture — decode config
ansible.builtin.set_fact:
reset_config_txt: "{{ reset_config_raw.content | b64decode }}"
when: reset_config_stat.stat.exists
# List-concat-and-take-first pattern: missing regex match falls back to current default.
# `gateway\s*=` deliberately does not match `gateway_ip` / `gateway_lrp_*`.
- name: capture — extract scalars
ansible.builtin.set_fact:
spinifex_region: "{{ (reset_config_txt | regex_findall('region\\s*=\\s*\"([^\"]+)\"') + [spinifex_region])[0] }}"
spinifex_az: "{{ (reset_config_txt | regex_findall('\\baz\\s*=\\s*\"([^\"]+)\"') + [spinifex_az])[0] }}"
spinifex_external_mode: "{{ (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_mode])[0] }}"
spinifex_external_pool_start: "{{ (reset_config_txt | regex_findall('range_start\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_start])[0] }}"
spinifex_external_pool_end: "{{ (reset_config_txt | regex_findall('range_end\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_end])[0] }}"
spinifex_external_gateway: "{{ (reset_config_txt | regex_findall('\\bgateway\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway])[0] }}"
spinifex_external_prefix_len: "{{ (reset_config_txt | regex_findall('prefix_len\\s*=\\s*([0-9]+)') + [spinifex_external_prefix_len | string])[0] }}"
spinifex_external_gateway_ip: "{{ (reset_config_txt | regex_findall('gateway_ip\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway_ip])[0] }}"
when: reset_config_stat.stat.exists
# Legacy: external_mode = "nat" was removed from spx. Coerce to "pool"
# so old configs still reset cleanly; warn so the operator notices.
- name: capture — coerce legacy external_mode=nat to pool
ansible.builtin.set_fact:
spinifex_external_mode: "pool"
when: spinifex_external_mode == 'nat'
- name: capture — warn on legacy nat coercion
ansible.builtin.debug:
msg: "⚠️ Captured external_mode=nat from legacy config; coerced to 'pool' (NAT mode removed from spx admin init)."
when:
- reset_config_stat.stat.exists
- (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"nat\"')) | length > 0
- name: capture — summary
ansible.builtin.debug:
msg: |
{% if reset_config_stat.stat.exists %}
Captured settings from {{ spinifex_config_file }}:
region={{ spinifex_region }}, az={{ spinifex_az }}
external_mode={{ spinifex_external_mode }}
pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
gateway={{ spinifex_external_gateway }}, prefix={{ spinifex_external_prefix_len }}
gateway_ip={{ spinifex_external_gateway_ip }}
{% else %}
No existing config; using defaults from vars/defaults.yml (or -e overrides).
{% endif %}
roles:
- teardown
- build
- install
- init
- smoketest
-14
View File
@@ -1,14 +0,0 @@
---
# Restore a dev-snapshot bundle: stop services, wipe the snapshot path
# set, untar the bundle, refresh CA trust, restart.
#
# Required: -e snapshot_name=<name>
# Optional: -e snapshot_dir=/some/path
- name: Dev restore — wipe + untar snapshot + restart
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- restore
@@ -1,17 +0,0 @@
---
# Quiesce spinifex + ovn + ovs, tar /etc/spinifex + /var/lib/spinifex +
# /var/lib/ovn + /etc/openvswitch/conf.db into a named bundle, restart.
# Use to checkpoint a working state before a risky operation (branch
# switch, viperblock surgery, ovn-controller restart cycle). Roll back
# via dev-restore without a full dev-reset.
#
# Required: -e snapshot_name=<name>
# Optional: -e snapshot_dir=/some/path
- name: Dev snapshot — quiesce + tar state + restart
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- snapshot
-13
View File
@@ -1,13 +0,0 @@
---
# Read-only health snapshot for a Spinifex dev node. Never mutates state.
# Run any time the operator wants to know "is my box healthy?" — pairs with
# dev-preflight (run before install) as the post-install / running-system
# diagnostic.
- name: Dev status — health snapshot (read-only)
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- status
@@ -1,8 +0,0 @@
---
- name: Dev teardown — wipe Spinifex state (no reinstall)
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- teardown
-11
View File
@@ -1,11 +0,0 @@
---
# Detailed build-vs-install drift report. Faster than dev-status (no AWS,
# OVN, or port probing). Answers "do I need to ansible-dev-deploy?".
- name: Dev version — build vs install drift (read-only)
hosts: localhost
gather_facts: false
vars_files:
- ../vars/defaults.yml
roles:
- version
-14
View File
@@ -1,14 +0,0 @@
---
# Idempotent dev-mode VPC fixture: spinifex-dev-vpc + public subnet +
# IGW + default route. Hooks into smoketest's launch path (which finds
# the public subnet automatically). Re-runs are no-ops once converged.
#
# Default state: present. Tear down with -e dev_vpc_state=absent.
- name: Dev VPC — fixture VPC/subnet/IGW/route (idempotent)
hosts: localhost
gather_facts: false
vars_files:
- ../vars/defaults.yml
roles:
- vpc
-6
View File
@@ -1,6 +0,0 @@
---
collections:
- name: community.general
version: ">=8.0.0"
- name: ansible.posix
version: ">=1.5.0"
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: build
description: Build spinifex + viperblock-nbdkit, stage tarball for setup.sh consumption.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,66 +0,0 @@
---
- name: build — spinifex (make build)
ansible.builtin.command:
cmd: make build
chdir: "{{ spinifex_root }}"
changed_when: true
- name: build — viperblock nbdkit plugin (make go_build_nbd)
ansible.builtin.command:
cmd: make go_build_nbd
chdir: "{{ mulga_root }}/viperblock"
changed_when: true
- name: build — detect microVM build script (present only on direct-boot branches)
ansible.builtin.stat:
path: "{{ spinifex_root }}/scripts/build-microvm-image.sh"
register: build_microvm_script
- name: build — microVM kernel + initramfs (direct-boot LBs)
ansible.builtin.command:
cmd: make build-microvm-image
chdir: "{{ spinifex_root }}"
changed_when: true
when: build_microvm_script.stat.exists
- name: build — verify build artifacts present
ansible.builtin.stat:
path: "{{ item }}"
register: build_artifacts
loop:
- "{{ spinifex_root }}/bin/spx"
- "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
- name: build — assert artifacts exist
ansible.builtin.assert:
that: item.stat.exists
fail_msg: "Missing build artifact: {{ item.item }}"
loop: "{{ build_artifacts.results }}"
loop_control:
label: "{{ item.item }}"
- name: build — stage tarball contents (mirrors dev-install.sh)
ansible.builtin.shell: |
set -e
STAGING=$(mktemp -d)
trap 'rm -rf "$STAGING"' EXIT
cp "{{ spinifex_root }}/bin/spx" "$STAGING/spx"
cp "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so" "$STAGING/"
cp "{{ spinifex_root }}/scripts/setup-ovn.sh" "$STAGING/"
mkdir -p "$STAGING/systemd"
cp "{{ spinifex_root }}/build/systemd/"* "$STAGING/systemd/"
mkdir -p "$STAGING/scripts"
cp "{{ spinifex_root }}/build/scripts/"* "$STAGING/scripts/"
cp "{{ spinifex_root }}/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
# microVM artifacts ship only when the branch built them.
if [ -f "{{ spinifex_root }}/build/microvm/vmlinuz" ] && [ -f "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" ]; then
mkdir -p "$STAGING/microvm"
cp "{{ spinifex_root }}/build/microvm/vmlinuz" "$STAGING/microvm/"
cp "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" "$STAGING/microvm/"
fi
tar czf "{{ build_tarball_path }}" -C "$STAGING" .
changed_when: true
- name: build — summary
ansible.builtin.debug:
msg: "Built spx + nbdkit plugin{{ ' + microvm artifacts' if build_microvm_script.stat.exists else '' }}, tarball at {{ build_tarball_path }}"
@@ -1,9 +0,0 @@
---
galaxy_info:
role_name: cluster-init
description: |
Run `spx admin init` on the cluster's init host, backgrounded so the
formation server accepts joins from the cluster-join role. Installs
cluster CA into the system trust store BEFORE starting spinifex.target
so predastore's strict TLS verify is satisfied at first start.
dependencies: []
@@ -1,77 +0,0 @@
---
- name: cluster-init — assert this is the init host
ansible.builtin.assert:
that: spinifex_node_role == "init"
fail_msg: "cluster-init role must only run against hosts in the [init] group"
# vars/defaults.yml sets spinifex_node_{name,count} for single-node dev use
# (precedence 11). Inventory hostvars (precedence 4) lose that fight, so
# we promote the inventory values via set_fact (precedence 15) before any
# task templates `{{ spinifex_node_count }}`.
- name: cluster-init — promote inventory node identity over defaults.yml
ansible.builtin.set_fact:
spinifex_node_name: "{{ hostvars[inventory_hostname]['spinifex_node_name'] }}"
spinifex_node_count: "{{ groups[cluster_env] | length }}"
- name: cluster-init — assemble external network flags
ansible.builtin.set_fact:
cluster_external_flags: >-
{% if cluster_external_pool | length > 0 %}--external-mode=pool --external-pool={{ cluster_external_pool }}{% endif %}
- name: cluster-init — run spx admin init (backgrounded, formation server waits for joins)
become: true
ansible.builtin.shell: |
set -o pipefail
spx admin init --force \
--node {{ spinifex_node_name }} \
--nodes {{ spinifex_node_count }} \
--bind {{ spinifex_wan_ip }} \
--cluster-bind {{ spinifex_wan_ip }} \
--port {{ cluster_formation_port }} \
--region {{ spinifex_region }} \
--az {{ spinifex_az }} \
{{ cluster_external_flags }} \
> /tmp/spx-admin-init.log 2>&1
async: 600
poll: 0
register: cluster_init_async
- name: cluster-init — brief pause so init can fail fast if misconfigured
ansible.builtin.pause:
seconds: 5
- name: cluster-init — check if init exited prematurely (sanity)
ansible.builtin.shell: |
if pgrep -f 'spx admin init' >/dev/null; then
echo "running"
else
tail -50 /tmp/spx-admin-init.log 2>/dev/null
exit 1
fi
register: cluster_init_alive
changed_when: false
failed_when: cluster_init_alive.rc != 0
- name: cluster-init — wait for formation server health
ansible.builtin.uri:
url: "https://{{ spinifex_wan_ip }}:{{ cluster_formation_port }}/formation/health"
validate_certs: false
status_code: 200
register: cluster_init_health
until: cluster_init_health.status == 200
retries: "{{ cluster_formation_timeout // 2 }}"
delay: 2
- name: cluster-init — fetch join token (delegated read for joiners)
become: true
ansible.builtin.slurp:
src: "{{ cluster_join_token_path }}"
register: cluster_join_token_raw
- name: cluster-init — register join token as host fact
ansible.builtin.set_fact:
cluster_join_token: "{{ cluster_join_token_raw.content | b64decode | trim }}"
- name: cluster-init — join token captured; waiting for joins handled by later play
ansible.builtin.debug:
msg: "Formation server up on {{ spinifex_wan_ip }}:{{ cluster_formation_port }}, {{ spinifex_node_count }} nodes expected"
@@ -1,8 +0,0 @@
---
galaxy_info:
role_name: cluster-join
description: |
Run `spx admin join` against the cluster init host's formation server.
Installs cluster CA + starts spinifex.target after join completes
(CA-before-services ordering avoids predastore TLS-verify crashes).
dependencies: []
@@ -1,39 +0,0 @@
---
- name: cluster-join — assert this is a join host
ansible.builtin.assert:
that: spinifex_node_role == "join"
fail_msg: "cluster-join role must only run against hosts in the [join] group"
# Promote inventory hostvars over vars/defaults.yml (see cluster-init).
- name: cluster-join — promote inventory node identity over defaults.yml
ansible.builtin.set_fact:
spinifex_node_name: "{{ hostvars[inventory_hostname]['spinifex_node_name'] }}"
- name: cluster-join — pull join token from init host
ansible.builtin.set_fact:
cluster_join_token: "{{ hostvars[cluster_init_host]['cluster_join_token'] }}"
- name: cluster-join — assert token present
ansible.builtin.assert:
that: cluster_join_token | length > 0
fail_msg: "join token missing — init host did not export cluster_join_token"
- name: cluster-join — run spx admin join (retry up to 5 times)
become: true
ansible.builtin.command:
cmd: >-
spx admin join
--node {{ spinifex_node_name }}
--bind {{ spinifex_wan_ip }}
--cluster-bind {{ spinifex_wan_ip }}
--host {{ cluster_init_host }}:{{ cluster_formation_port }}
--token {{ cluster_join_token }}
--region {{ spinifex_region }}
--az {{ spinifex_az }}
register: cluster_join_result
retries: 5
delay: 3
until: cluster_join_result.rc == 0
# CA install + service start handled by cluster-bootstrap.yml plays that
# run after all joins complete and init has written multi-node configs.
@@ -1,8 +0,0 @@
---
galaxy_info:
role_name: cluster-post
description: |
Post-bootstrap idempotent fixtures on the init host: import SSH key,
import Ubuntu AMI (dual-version detect 24.04/26.04). Skip when
artifacts already exist.
dependencies: []
@@ -1,205 +0,0 @@
---
- name: cluster-post — locate ssh public key for import
ansible.builtin.find:
paths: "{{ ansible_env.HOME }}/.ssh"
patterns:
- spinifex-key.pub
- tf-user-*.pub
file_type: file
register: cluster_post_ssh_pubkey
- name: cluster-post — fail when no public key found
ansible.builtin.assert:
that: cluster_post_ssh_pubkey.files | length > 0
fail_msg: "no public key in ~/.ssh — needed for spinifex-key import"
- name: cluster-post — set pubkey path fact
ansible.builtin.set_fact:
cluster_post_pubkey_path: "{{ cluster_post_ssh_pubkey.files[0].path }}"
- name: cluster-post — check whether spinifex-key already imported
ansible.builtin.command:
cmd: "{{ spinifex_aws_cli }} ec2 describe-key-pairs --key-names spinifex-key"
register: cluster_post_key_check
changed_when: false
failed_when: false
- name: cluster-post — import spinifex-key (if absent)
ansible.builtin.command:
cmd: >-
{{ spinifex_aws_cli }} ec2 import-key-pair
--key-name spinifex-key
--public-key-material fileb://{{ cluster_post_pubkey_path }}
register: cluster_post_key_import
retries: 10
delay: 6
until: cluster_post_key_import.rc == 0
when: cluster_post_key_check.rc != 0
- name: cluster-post — locate ubuntu cloud image (26.04 preferred, 24.04 fallback)
ansible.builtin.stat:
path: "{{ item }}"
register: cluster_post_ubuntu_stat
loop:
- "{{ ansible_env.HOME }}/images/ubuntu-26.04.img"
- "{{ ansible_env.HOME }}/images/ubuntu-24.04.img"
- name: cluster-post — pick first available image
ansible.builtin.set_fact:
cluster_post_ubuntu_image: "{{ (cluster_post_ubuntu_stat.results | selectattr('stat.exists') | map(attribute='item') | list | first) | default('') }}"
- name: cluster-post — fail when no ubuntu image present
ansible.builtin.assert:
that: cluster_post_ubuntu_image | length > 0
fail_msg: "no ubuntu-{24,26}.04.img in ~/images/ — gold image must stage it"
- name: cluster-post — derive ubuntu version from image filename
ansible.builtin.set_fact:
cluster_post_ubuntu_version: "{{ cluster_post_ubuntu_image | regex_search('ubuntu-([0-9]+\\.[0-9]+)\\.img', '\\1') | first }}"
- name: cluster-post — check whether ubuntu AMI already imported
ansible.builtin.command:
cmd: >-
{{ spinifex_aws_cli }} ec2 describe-images
--owners self
--filters Name=name,Values=ubuntu-{{ cluster_post_ubuntu_version }}
register: cluster_post_ami_check
changed_when: false
failed_when: false
- name: cluster-post — import ubuntu AMI (if absent)
ansible.builtin.command:
cmd: >-
{{ spinifex_spx_cli }} admin images import
--file {{ cluster_post_ubuntu_image }}
--arch x86_64
--distro ubuntu
--version {{ cluster_post_ubuntu_version }}
--boot-mode uefi
when:
- cluster_post_ami_check.rc != 0
or "ubuntu-" + cluster_post_ubuntu_version not in cluster_post_ami_check.stdout
- name: cluster-post — slurp final admin creds from ~/.aws/credentials
# Source of truth: SetupAWSCredentials writes the FINAL admin keys here.
# Init log shows obsolete keys (multi-node path silently regenerates them
# after the initial print — see cmd/spinifex/cmd/admin.go writeBootstrapFiles
# called twice in the multi-node flow).
ansible.builtin.slurp:
src: "{{ ansible_env.HOME }}/.aws/credentials"
register: cluster_post_aws_creds_raw
- name: cluster-post — extract admin access/secret keys from spinifex profile
ansible.builtin.set_fact:
cluster_post_admin_access_key: >-
{{ (cluster_post_aws_creds_raw.content | b64decode
| regex_search('\[spinifex\][^\[]*aws_access_key_id\s*=\s*(\S+)', '\1')
| first) | default('') }}
cluster_post_admin_secret_key: >-
{{ (cluster_post_aws_creds_raw.content | b64decode
| regex_search('\[spinifex\][^\[]*aws_secret_access_key\s*=\s*(\S+)', '\1')
| first) | default('') }}
- name: cluster-post — assert creds parsed
ansible.builtin.assert:
that:
- cluster_post_admin_access_key | length > 0
- cluster_post_admin_secret_key | length > 0
fail_msg: "could not parse admin keys from {{ ansible_env.HOME }}/.aws/credentials [spinifex] profile"
- name: cluster-post — ensure local .creds dir exists
delegate_to: localhost
become: false
ansible.builtin.file:
path: "{{ playbook_dir }}/../.creds"
state: directory
mode: "0700"
- name: cluster-post — write creds to operator workstation
delegate_to: localhost
become: false
ansible.builtin.copy:
dest: "{{ playbook_dir }}/../.creds/{{ cluster_env }}.txt"
mode: "0600"
content: |
# Spinifex cluster admin credentials — {{ cluster_env }}
# Generated: {{ ansible_facts['date_time']['iso8601'] }}
# Init host: {{ cluster_init_host }}
#
# UI: https://{{ cluster_init_host }}:3000
# AWS API: https://{{ cluster_init_host }}:4432
# Region: {{ spinifex_region }}
#
# Login (admin user / AWS profile spinifex):
AWS_ACCESS_KEY_ID={{ cluster_post_admin_access_key }}
AWS_SECRET_ACCESS_KEY={{ cluster_post_admin_secret_key }}
# Per-env AWS profile in operator's ~/.aws/credentials so tofu / aws CLI can
# target this cluster via AWS_PROFILE=spinifex-{{ cluster_env }}. blockinfile
# marker keeps re-runs idempotent — replaces the block in place, never
# duplicates. Does not touch the user's existing [spinifex] profile (single-
# node dev). Tofu side: provider "aws" { profile = "spinifex-env1" ... }.
- name: cluster-post — ensure local ~/.aws dir exists
delegate_to: localhost
become: false
ansible.builtin.file:
path: "{{ lookup('env', 'HOME') }}/.aws"
state: directory
mode: "0700"
- name: cluster-post — ensure local ~/.aws/credentials exists
delegate_to: localhost
become: false
ansible.builtin.copy:
dest: "{{ lookup('env', 'HOME') }}/.aws/credentials"
content: ""
force: false
mode: "0600"
- name: cluster-post — upsert spinifex-{{ cluster_env }} profile in local ~/.aws/credentials
delegate_to: localhost
become: false
ansible.builtin.blockinfile:
path: "{{ lookup('env', 'HOME') }}/.aws/credentials"
marker: "# {mark} SPINIFEX-{{ cluster_env | upper }} MANAGED BLOCK"
block: |
[spinifex-{{ cluster_env }}]
aws_access_key_id = {{ cluster_post_admin_access_key }}
aws_secret_access_key = {{ cluster_post_admin_secret_key }}
region = {{ spinifex_region }}
ca_bundle = {{ playbook_dir }}/../.creds/{{ cluster_env }}-ca.pem
endpoint_url = https://{{ cluster_init_host }}:4432
- name: cluster-post — slurp cluster CA from init host
ansible.builtin.slurp:
src: "{{ spinifex_ca_src }}"
register: cluster_post_ca_raw
become: true
- name: cluster-post — write CA pem to operator workstation
delegate_to: localhost
become: false
ansible.builtin.copy:
dest: "{{ playbook_dir }}/../.creds/{{ cluster_env }}-ca.pem"
mode: "0644"
content: "{{ cluster_post_ca_raw.content | b64decode }}"
- name: cluster-post — resolve absolute paths for summary
delegate_to: localhost
become: false
ansible.builtin.command: "readlink -f {{ playbook_dir }}/../.creds"
register: cluster_post_creds_abs
changed_when: false
- name: cluster-post — summary
ansible.builtin.debug:
msg: |
Post-bootstrap done:
spinifex-key: {{ 'imported now' if (cluster_post_key_import.changed | default(false)) else 'already present' }}
ubuntu AMI: ubuntu-{{ cluster_post_ubuntu_version }}
admin creds: {{ cluster_post_creds_abs.stdout }}/{{ cluster_env }}.txt
CA cert: {{ cluster_post_creds_abs.stdout }}/{{ cluster_env }}-ca.pem
AWS profile: spinifex-{{ cluster_env }} (in ~/.aws/credentials)
UI URL: https://{{ cluster_init_host }}:3000
Access Key: {{ cluster_post_admin_access_key }}
Tofu usage: AWS_PROFILE=spinifex-{{ cluster_env }} tofu plan
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: deploy
description: In-place binary + microvm artifact swap for a running Spinifex node (parity with `make deploy`).
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
-144
View File
@@ -1,144 +0,0 @@
---
# In-place artifact swap for an already-installed spinifex node. Mirrors
# `make deploy`: replace binaries + microvm artifacts, restart services.
# Does NOT run setup.sh (no user/dir/systemd-unit changes) or smoketest.
- name: deploy — fail fast if spinifex isn't already installed
ansible.builtin.stat:
path: "{{ spinifex_config_file }}"
register: deploy_installed_stat
failed_when: not deploy_installed_stat.stat.exists
- name: deploy — assert config present
ansible.builtin.assert:
that: deploy_installed_stat.stat.exists
fail_msg: |
{{ spinifex_config_file }} missing — this playbook only updates an
existing install. Run ansible-dev-install or ansible-dev-reset first.
- name: deploy — verify build artifacts present
ansible.builtin.stat:
path: "{{ item }}"
register: deploy_artifacts
loop:
- "{{ spinifex_root }}/bin/spx"
- "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
- name: deploy — assert artifacts exist after build role
ansible.builtin.assert:
that: item.stat.exists
fail_msg: "Build artifact missing: {{ item.item }} — build role should have produced it"
loop: "{{ deploy_artifacts.results }}"
loop_control:
label: "{{ item.item }}"
- name: deploy — detect microvm artifacts (direct-boot branches only)
ansible.builtin.stat:
path: "{{ spinifex_root }}/build/microvm/{{ item }}"
register: deploy_microvm_stat
loop:
- vmlinuz
- initramfs.cpio.gz
- name: deploy — set microvm presence fact
ansible.builtin.set_fact:
deploy_microvm_present: "{{ deploy_microvm_stat.results | map(attribute='stat.exists') | min }}"
- name: deploy — detect nbdkit plugin directory
ansible.builtin.shell: |
set -e
nbdkit --dump-config | awk -F= '/^plugindir=/{print $2; exit}'
register: deploy_plugindir_cmd
changed_when: false
- name: deploy — set plugin path fact
ansible.builtin.set_fact:
deploy_plugindir: "{{ deploy_plugindir_cmd.stdout | trim }}"
- name: deploy — assert plugindir resolved
ansible.builtin.assert:
that: deploy_plugindir | length > 0
fail_msg: "Could not detect nbdkit plugin directory via `nbdkit --dump-config`"
- name: deploy — stop spinifex.target before binary swap
become: true
ansible.builtin.systemd:
name: spinifex.target
state: stopped
failed_when: false
- name: deploy — install spx binary
become: true
ansible.builtin.copy:
src: "{{ spinifex_root }}/bin/spx"
dest: /usr/local/bin/spx
mode: "0755"
owner: root
group: root
- name: deploy — install viperblock nbdkit plugin
become: true
ansible.builtin.copy:
src: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
dest: "{{ deploy_plugindir }}/nbdkit-viperblock-plugin.so"
mode: "0755"
owner: root
group: root
- name: deploy — ensure microvm dir
become: true
ansible.builtin.file:
path: /usr/share/spinifex/microvm
state: directory
mode: "0755"
owner: root
group: root
when: deploy_microvm_present
- name: deploy — install microvm kernel
become: true
ansible.builtin.copy:
src: "{{ spinifex_root }}/build/microvm/vmlinuz"
dest: /usr/share/spinifex/microvm/vmlinuz
mode: "0644"
owner: root
group: root
when: deploy_microvm_present
- name: deploy — install microvm initramfs
become: true
ansible.builtin.copy:
src: "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz"
dest: /usr/share/spinifex/microvm/initramfs.cpio.gz
mode: "0644"
owner: root
group: root
when: deploy_microvm_present
- name: deploy — daemon-reload (covers any unit changes shipped via setup.sh later)
become: true
ansible.builtin.systemd:
daemon_reload: true
- name: deploy — start spinifex.target
become: true
ansible.builtin.systemd:
name: spinifex.target
state: started
- name: deploy — wait for awsgw to respond
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
register: deploy_awsgw
retries: 15
delay: 2
until: deploy_awsgw.rc == 0
changed_when: false
- name: deploy — summary
ansible.builtin.debug:
msg: |
Deployed:
spx → /usr/local/bin/spx
nbdkit plugin → {{ deploy_plugindir }}/nbdkit-viperblock-plugin.so
microvm → {{ '/usr/share/spinifex/microvm/{vmlinuz,initramfs.cpio.gz}' if deploy_microvm_present else 'skipped (not built on this branch)' }}
spinifex.target restarted; awsgw responding.
-7
View File
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: init
description: Run spx admin init, install CA trust, start services, import LB image.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
-90
View File
@@ -1,90 +0,0 @@
---
- name: init — base admin-init flags
ansible.builtin.set_fact:
init_base_flags: >-
--force
--region {{ spinifex_region }}
--az {{ spinifex_az }}
--node {{ spinifex_node_name }}
--nodes {{ spinifex_node_count }}
- name: init — no-external opt-out
ansible.builtin.set_fact:
init_external_flags: "--no-external"
when: spinifex_external_no_external | bool
- name: init — assert pool range supplied when mode=pool
ansible.builtin.assert:
that:
- spinifex_external_pool_start | length > 0
- spinifex_external_pool_end | length > 0
fail_msg: |
spx admin init requires --external-pool when --external-mode=pool.
Set spinifex_external_pool_start / spinifex_external_pool_end in
scripts/ansible/vars/defaults.yml, or pass:
-e spinifex_external_pool_start=192.168.1.150
-e spinifex_external_pool_end=192.168.1.250
Pass -e spinifex_external_no_external=true to skip external networking.
when:
- not (spinifex_external_no_external | bool)
- spinifex_external_mode == 'pool'
- name: init — pool-mode external flags
ansible.builtin.set_fact:
init_external_flags: >-
--external-mode=pool
--external-pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
{% if spinifex_external_gateway | length > 0 %}--external-gateway={{ spinifex_external_gateway }}{% endif %}
{% if spinifex_external_prefix_len | string | length > 0 %}--external-prefix-len={{ spinifex_external_prefix_len }}{% endif %}
{% if spinifex_external_gateway_ip | length > 0 %}--gateway-ip={{ spinifex_external_gateway_ip }}{% endif %}
when:
- not (spinifex_external_no_external | bool)
- spinifex_external_mode == 'pool'
- name: init — assemble full flag string
ansible.builtin.set_fact:
init_all_flags: "{{ init_base_flags }} {{ init_external_flags | default('') }}"
- name: init — run spx admin init
become: true
ansible.builtin.command: "spx admin init {{ init_all_flags }}"
changed_when: true
- name: init — copy CA to system trust
become: true
ansible.builtin.copy:
src: "{{ spinifex_ca_src }}"
dest: "{{ spinifex_ca_trust_path }}"
mode: "0644"
remote_src: true
- name: init — rebuild CA bundle (not --fresh — preserve other CAs)
become: true
ansible.builtin.command: update-ca-certificates
changed_when: true
- name: init — start spinifex.target
become: true
ansible.builtin.systemd:
name: spinifex.target
state: started
# systemctl start returns as soon as units activate; awsgw still wires up
# to NATS + daemons after that. Bash start-dev.sh has internal wait loops;
# systemd path does not. Poll a benign call until gateway is fully ready.
- name: init — wait for AWS gateway readiness
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
register: init_gw_ready
until: init_gw_ready.rc == 0
retries: 30
delay: 2
changed_when: false
- name: init — summary
ansible.builtin.debug:
msg: |
Init complete:
region={{ spinifex_region }}, az={{ spinifex_az }}, node={{ spinifex_node_name }}
external_mode={{ 'disabled' if (spinifex_external_no_external | bool) else spinifex_external_mode }}
spinifex.target: started
CA: {{ spinifex_ca_trust_path }}
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: install
description: Invoke setup.sh and setup-ovn.sh to install Spinifex into production paths.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,79 +0,0 @@
---
- name: install — assert tarball exists
ansible.builtin.stat:
path: "{{ build_tarball_path }}"
register: install_tarball
- name: install — fail if tarball missing
ansible.builtin.assert:
that: install_tarball.stat.exists
fail_msg: "Tarball {{ build_tarball_path }} missing. Run the build role first."
- name: install — stop spinifex.target before reinstall
become: true
ansible.builtin.systemd:
name: spinifex.target
state: stopped
failed_when: false
- name: install — reset failed state (spinifex-* scope)
become: true
ansible.builtin.command: systemctl reset-failed 'spinifex-*'
changed_when: false
failed_when: false
- name: install — scrub stale pid/log files from prior dev-mode installs
become: true
ansible.builtin.shell: |
for dir in {{ spinifex_data_dir }} {{ spinifex_log_dir }} {{ spinifex_etc_dir }}; do
if [ -d "$dir" ]; then
find "$dir" -name '*.pid' -delete 2>/dev/null || true
find "$dir" -name '*.log' -delete 2>/dev/null || true
find "$dir" -name '*.log.*' -delete 2>/dev/null || true
fi
done
changed_when: false
- name: install — remove legacy /var/lib/spinifex/config
become: true
ansible.builtin.file:
path: "{{ spinifex_data_dir }}/config"
state: absent
- name: install — purge stale cluster state (configs + raft + NATS KV) on bootstrap
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ spinifex_etc_dir }}/predastore/predastore.toml"
- "{{ spinifex_etc_dir }}/spinifex.toml"
- "{{ spinifex_data_dir }}/predastore"
- "{{ spinifex_data_dir }}/nats"
- "{{ spinifex_data_dir }}/awsgw"
when: cluster_bootstrap_purge | default(false) | bool
- name: install — run setup.sh (creates users, dirs, systemd units)
become: true
ansible.builtin.command:
cmd: "bash {{ spinifex_root }}/scripts/setup.sh"
environment:
INSTALL_SPINIFEX_TARBALL: "{{ build_tarball_path }}"
INSTALL_SPINIFEX_SKIP_NEWGRP: "1"
changed_when: true
- name: install — cleanup tarball
ansible.builtin.file:
path: "{{ build_tarball_path }}"
state: absent
- name: install — run setup-ovn.sh --management
become: true
ansible.builtin.command:
cmd: "/usr/local/share/spinifex/setup-ovn.sh --management --node-name={{ spinifex_node_name }}"
changed_when: true
when: install_run_ovn | default(true) | bool
- name: install — summary
ansible.builtin.debug:
msg: "Install complete — setup.sh + setup-ovn.sh ran. Services NOT yet started (init role handles that)."
-7
View File
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: logs
description: Dump spinifex + ovn + system logs into a tarball for bug reports.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
-153
View File
@@ -1,153 +0,0 @@
---
# Collect a self-contained log bundle for bug reports. NEVER mutates running
# state — only reads journalctl + /var/log/spinifex/* + status output, then
# tars into logs_output_dir. Errors on missing inputs are non-fatal so a
# half-broken node still produces a bundle.
- name: logs — resolve staging directory
ansible.builtin.set_fact:
logs_stage_dir: "/tmp/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}"
logs_bundle_path: "{{ logs_output_dir }}/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}.tar.gz"
- name: logs — create staging tree
ansible.builtin.file:
path: "{{ logs_stage_dir }}/{{ item }}"
state: directory
mode: "0755"
loop:
- journal
- varlog
- status
- kernel
- name: logs — capture journalctl for spinifex-* units
become: true
ansible.builtin.shell: |
journalctl --no-pager --output=short-iso \
-u 'spinifex-*' --since {{ logs_since | quote }} \
> {{ logs_stage_dir }}/journal/spinifex.log 2>&1 || true
changed_when: true
- name: logs — capture journalctl for ovn-* + openvswitch-switch
become: true
ansible.builtin.shell: |
journalctl --no-pager --output=short-iso \
-u 'ovn-*' -u openvswitch-switch \
--since {{ logs_since | quote }} \
> {{ logs_stage_dir }}/journal/ovn-ovs.log 2>&1 || true
changed_when: true
- name: logs — capture journalctl for nbdkit + qemu (best-effort match)
become: true
ansible.builtin.shell: |
journalctl --no-pager --output=short-iso \
--since {{ logs_since | quote }} \
_COMM=qemu-system-x86_64 + _COMM=qemu-system-aarch64 + _COMM=nbdkit \
> {{ logs_stage_dir }}/journal/qemu-nbdkit.log 2>&1 || true
changed_when: true
- name: logs — copy /var/log/spinifex/*
become: true
ansible.builtin.shell: |
if [ -d {{ spinifex_log_dir | quote }} ]; then
cp -a {{ spinifex_log_dir | quote }}/. {{ logs_stage_dir | quote }}/varlog/ 2>/dev/null || true
fi
changed_when: true
- name: logs — systemctl status snapshot (spinifex-*)
become: true
ansible.builtin.shell: |
systemctl status 'spinifex-*' --no-pager --all \
> {{ logs_stage_dir }}/status/spinifex-units.txt 2>&1 || true
systemctl status ovn-controller ovn-central openvswitch-switch \
--no-pager > {{ logs_stage_dir }}/status/ovn-ovs-units.txt 2>&1 || true
systemctl list-units 'spinifex-*' --all --no-legend --plain \
> {{ logs_stage_dir }}/status/spinifex-list.txt 2>&1 || true
changed_when: true
- name: logs — networking snapshot (ip, ss, ovs)
become: true
ansible.builtin.shell: |
ip -d -s link show > {{ logs_stage_dir }}/status/ip-link.txt 2>&1 || true
ip addr show > {{ logs_stage_dir }}/status/ip-addr.txt 2>&1 || true
ip route show table all > {{ logs_stage_dir }}/status/ip-route.txt 2>&1 || true
ss -tlnp > {{ logs_stage_dir }}/status/ss-tlnp.txt 2>&1 || true
if command -v ovs-vsctl >/dev/null 2>&1; then
ovs-vsctl show > {{ logs_stage_dir }}/status/ovs-show.txt 2>&1 || true
ovs-vsctl list-br > {{ logs_stage_dir }}/status/ovs-br.txt 2>&1 || true
fi
if command -v ovn-nbctl >/dev/null 2>&1; then
ovn-nbctl show > {{ logs_stage_dir }}/status/ovn-nb.txt 2>&1 || true
ovn-sbctl show > {{ logs_stage_dir }}/status/ovn-sb.txt 2>&1 || true
fi
changed_when: true
- name: logs — kernel ring buffer
become: true
ansible.builtin.shell: |
dmesg --time-format=iso > {{ logs_stage_dir }}/kernel/dmesg.txt 2>&1 || true
changed_when: true
when: logs_include_dmesg | bool
- name: logs — capture spx + plugin versions
become: true
ansible.builtin.shell: |
{
echo '## spx version'
/usr/local/bin/spx version 2>&1 | grep -vE '^[0-9]{4}/' || true
echo
echo '## nbdkit plugin sha256'
pdir=$(nbdkit --dump-config 2>/dev/null | awk -F= '/^plugindir=/{print $2; exit}')
[ -n "$pdir" ] && sha256sum "$pdir/nbdkit-viperblock-plugin.so" 2>/dev/null || true
echo
echo '## microvm artifacts'
sha256sum /usr/share/spinifex/microvm/* 2>/dev/null || true
echo
echo '## kernel + os'
uname -a
cat /etc/os-release 2>/dev/null || true
} > {{ logs_stage_dir }}/status/versions.txt 2>&1 || true
changed_when: true
- name: logs — create bundle tarball
become: true
ansible.builtin.shell: |
tar -C {{ logs_stage_dir | dirname | quote }} \
-czf {{ logs_bundle_path | quote }} \
{{ logs_stage_dir | basename | quote }}
changed_when: true
- name: logs — make bundle owned by invoking user
become: true
ansible.builtin.file:
path: "{{ logs_bundle_path }}"
owner: "{{ spinifex_invoking_user }}"
group: "{{ spinifex_invoking_user }}"
mode: "0640"
- name: logs — remove staging tree
become: true
ansible.builtin.file:
path: "{{ logs_stage_dir }}"
state: absent
- name: logs — bundle size
ansible.builtin.stat:
path: "{{ logs_bundle_path }}"
register: logs_bundle_stat
- name: logs — summary
ansible.builtin.debug:
msg: |
Log bundle written:
path : {{ logs_bundle_path }}
size : {{ (logs_bundle_stat.stat.size | int / 1024) | round(1) }} KiB
since : {{ logs_since }}
Contents:
journal/ spinifex-*, ovn-*+openvswitch-switch, qemu/nbdkit
varlog/ copy of {{ spinifex_log_dir }}
status/ systemctl status, ip/ss/ovs/ovn show, versions
kernel/ dmesg{{ '' if logs_include_dmesg else ' (skipped via -e logs_include_dmesg=false)' }}
Attach to bug report. For live tail use:
journalctl -u 'spinifex-*' -f
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: preflight
description: Validate dev-host dependencies for Spinifex (parity with dev-setup.sh).
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,98 +0,0 @@
---
- name: Check required commands
ansible.builtin.command: "which {{ item }}"
register: preflight_required
loop: "{{ spinifex_required_commands }}"
changed_when: false
failed_when: preflight_required.rc != 0
loop_control:
label: "{{ item }}"
- name: Check optional commands
ansible.builtin.command: "which {{ item }}"
register: preflight_optional
loop: "{{ spinifex_optional_commands }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item }}"
- name: Warn on missing optional commands
ansible.builtin.debug:
msg: "WARN: optional command '{{ item.item }}' not found"
loop: "{{ preflight_optional.results }}"
when: item.rc != 0
loop_control:
label: "{{ item.item }}"
- name: Check OVN/OVS commands
ansible.builtin.command: "which {{ item }}"
register: preflight_ovn
loop: "{{ spinifex_ovn_commands }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item }}"
- name: Warn on missing OVN/OVS tools
ansible.builtin.debug:
msg: "WARN: OVN/OVS tool '{{ item.item }}' not found (required for VPC networking). Install: sudo apt-get install ovn-central ovn-host openvswitch-switch"
loop: "{{ preflight_ovn.results }}"
when: item.rc != 0
loop_control:
label: "{{ item.item }}"
- name: Record OVN tool availability
ansible.builtin.set_fact:
preflight_ovs_present: "{{ (preflight_ovn.results | selectattr('item','eq','ovs-vsctl') | selectattr('rc','eq',0) | list | length) > 0 }}"
- name: Check br-int bridge
ansible.builtin.command: ovs-vsctl br-exists br-int
register: preflight_brint
changed_when: false
failed_when: false
when: preflight_ovs_present
- name: Warn if br-int missing
ansible.builtin.debug:
msg: "WARN: br-int missing. Run scripts/setup-ovn.sh --management to initialise OVN networking."
when:
- preflight_ovs_present
- preflight_brint.rc | default(0) != 0
- name: Check sibling component directories
ansible.builtin.stat:
path: "{{ playbook_dir }}/../../../../{{ item }}/go.mod"
register: preflight_components
loop: "{{ spinifex_components }}"
loop_control:
label: "{{ item }}"
- name: Warn on missing component checkouts
ansible.builtin.debug:
msg: "WARN: component '{{ item.item }}' not found at expected sibling path (no go.mod). Run scripts/clone-deps.sh."
loop: "{{ preflight_components.results }}"
when: not item.stat.exists
loop_control:
label: "{{ item.item }}"
- name: Check port availability
ansible.builtin.shell: "ss -tlnH 'sport = :{{ item.port }}' | head -1"
register: preflight_ports
loop: "{{ spinifex_ports }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item.service }} ({{ item.port }})"
- name: Warn on occupied ports
ansible.builtin.debug:
msg: "WARN: port {{ item.item.port }} occupied (needed for {{ item.item.service }})"
loop: "{{ preflight_ports.results }}"
when: item.stdout | length > 0
loop_control:
label: "{{ item.item.service }}"
- name: Preflight summary
ansible.builtin.debug:
msg: "Preflight passed. Required: {{ spinifex_required_commands | join(', ') }}. Warnings above (if any) are non-fatal."
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: restore
description: Restore a spinifex+ovn+ovs snapshot bundle, refresh CA trust, restart.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,140 +0,0 @@
---
# Inverse of the snapshot role: stop services, wipe the same path set,
# untar the named bundle, refresh CA trust (because /etc/spinifex/ca.pem
# came from the snapshot but the system trust store didn't), restart.
- name: restore — require snapshot_name
ansible.builtin.assert:
that:
- snapshot_name is defined
- snapshot_name | length > 0
- snapshot_name is match('^[A-Za-z0-9._-]+$')
fail_msg: "snapshot_name required (alphanumeric/._-): -e snapshot_name=<name>"
- name: restore — resolve bundle path
ansible.builtin.set_fact:
snapshot_bundle_path: "{{ snapshot_dir }}/{{ snapshot_name }}.tar.gz"
- name: restore — assert bundle exists
become: true
ansible.builtin.stat:
path: "{{ snapshot_bundle_path }}"
register: restore_bundle_stat
- name: restore — fail if missing
ansible.builtin.assert:
that: restore_bundle_stat.stat.exists
fail_msg: |
Snapshot bundle missing: {{ snapshot_bundle_path }}.
Available bundles:
ls -lh {{ snapshot_dir }}/*.tar.gz 2>/dev/null
- name: restore — stop services in dependency order
become: true
ansible.builtin.systemd:
name: "{{ item }}"
state: stopped
loop: "{{ snapshot_services_stop_order }}"
failed_when: false
- name: restore — kill lingering QEMU VMs
become: true
ansible.builtin.command: "pkill -x {{ item }}"
loop: "{{ spinifex_qemu_processes }}"
changed_when: false
failed_when: false
- name: restore — wait for QEMU exit
become: true
ansible.builtin.shell: |
set -e
for i in $(seq 1 {{ teardown_process_wait_seconds }}); do
alive=0
for proc in {{ spinifex_qemu_processes | join(' ') }}; do
if pgrep -x "$proc" >/dev/null 2>&1; then alive=1; break; fi
done
if [ "$alive" = "0" ]; then exit 0; fi
sleep 1
done
exit 1
changed_when: false
- name: restore — wipe target paths (only those the snapshot will replace)
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop: "{{ snapshot_paths }}"
- name: restore — recreate parent dirs that tar will refill
become: true
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: root
mode: "0755"
loop:
- "{{ ovn_db_dir }}"
- /etc/openvswitch
- name: restore — extract bundle (preserve permissions/ACLs)
become: true
ansible.builtin.command:
cmd: >-
tar --acls --xattrs -xzpf {{ snapshot_bundle_path | quote }}
-C /
changed_when: true
- name: restore — refresh spinifex CA in system trust
become: true
ansible.builtin.stat:
path: "{{ spinifex_ca_src }}"
register: restore_ca_stat
- name: restore — copy CA back to trust path
become: true
ansible.builtin.copy:
src: "{{ spinifex_ca_src }}"
dest: "{{ spinifex_ca_trust_path }}"
mode: "0644"
remote_src: true
when: restore_ca_stat.stat.exists
- name: restore — rebuild CA bundle
become: true
ansible.builtin.command: update-ca-certificates
changed_when: true
when: restore_ca_stat.stat.exists
- name: restore — daemon-reload (covers any unit file delta in the snapshot)
become: true
ansible.builtin.systemd:
daemon_reload: true
- name: restore — restart services in reverse order
become: true
ansible.builtin.systemd:
name: "{{ item }}"
state: started
loop: "{{ snapshot_services_stop_order | reverse | list }}"
failed_when: false
- name: restore — wait for AWS gateway readiness
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
register: restore_gw_ready
retries: 30
delay: 2
until: restore_gw_ready.rc == 0
changed_when: false
failed_when: false
- name: restore — summary
ansible.builtin.debug:
msg: |
Restore complete:
snapshot : {{ snapshot_name }}
bundle : {{ snapshot_bundle_path }}
paths : {{ snapshot_paths | join(', ') }}
CA : {{ 'refreshed' if restore_ca_stat.stat.exists else 'not in bundle — system trust unchanged' }}
gateway : {{ 'ready' if restore_gw_ready.rc == 0 else 'NOT ready after restart — investigate' }}
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: smoketest
description: Import SSH key, import Ubuntu AMI, launch one smoke-test instance.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,197 +0,0 @@
---
- name: smoketest — resolve invoking user's home
ansible.builtin.getent:
database: passwd
key: "{{ spinifex_invoking_user }}"
- name: smoketest — set user home fact
ansible.builtin.set_fact:
smoketest_user_home: "{{ ansible_facts['getent_passwd'][spinifex_invoking_user][4] }}"
- name: smoketest — generate SSH key if absent
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.command:
cmd: ssh-keygen -t ed25519 -f {{ smoketest_user_home }}/.ssh/spinifex-key -N ""
creates: "{{ smoketest_user_home }}/.ssh/spinifex-key"
- name: smoketest — import SSH key to spinifex
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 import-key-pair
--key-name spinifex-key
--public-key-material fileb://{{ smoketest_user_home }}/.ssh/spinifex-key.pub
register: smoketest_keypair
until: >-
smoketest_keypair.rc == 0 or
'InvalidKeyPair.Duplicate' in (smoketest_keypair.stderr | default(''))
retries: 10
delay: 3
changed_when: "'InvalidKeyPair.Duplicate' not in (smoketest_keypair.stderr | default(''))"
failed_when:
- smoketest_keypair.rc != 0
- "'InvalidKeyPair.Duplicate' not in (smoketest_keypair.stderr | default(''))"
- name: smoketest — detect architecture
ansible.builtin.set_fact:
smoketest_arch: "{{ 'arm64' if ansible_facts['architecture'] in ['aarch64','arm64'] else 'x86_64' }}"
- name: smoketest — resolve local image path
ansible.builtin.set_fact:
smoketest_local_image_path: "{{ smoketest_local_image | expanduser }}"
vars:
smoketest_local_image: "{{ spinifex_smoketest_local_image | replace('~', smoketest_user_home) }}"
- name: smoketest — check for local image
ansible.builtin.stat:
path: "{{ smoketest_local_image_path }}"
register: smoketest_local_img
- name: smoketest — import AMI from local file
ansible.builtin.command: >-
{{ spinifex_spx_cli }} admin images import
--file {{ smoketest_local_image_path }}
--distro {{ spinifex_smoketest_distro }}
--version {{ spinifex_smoketest_version }}
--arch {{ smoketest_arch }}
--boot-mode uefi
register: smoketest_import_local
until: >-
smoketest_import_local.rc == 0 or
'already exists' in (smoketest_import_local.stderr | default(''))
retries: 5
delay: 3
changed_when: smoketest_import_local.rc == 0
failed_when:
- smoketest_import_local.rc != 0
- "'already exists' not in (smoketest_import_local.stderr | default(''))"
when: smoketest_local_img.stat.exists
- name: smoketest — import AMI by name (fallback)
ansible.builtin.command: >-
{{ spinifex_spx_cli }} admin images import
--name {{ spinifex_smoketest_distro }}-{{ spinifex_smoketest_version }}-{{ smoketest_arch }}
register: smoketest_import_remote
until: >-
smoketest_import_remote.rc == 0 or
'already exists' in (smoketest_import_remote.stderr | default(''))
retries: 5
delay: 3
changed_when: smoketest_import_remote.rc == 0
failed_when:
- smoketest_import_remote.rc != 0
- "'already exists' not in (smoketest_import_remote.stderr | default(''))"
when: not smoketest_local_img.stat.exists
- name: smoketest — detect CPU vendor for instance type choice
ansible.builtin.command: grep -q AuthenticAMD /proc/cpuinfo
register: smoketest_amd
changed_when: false
failed_when: false
- name: smoketest — pick instance type
ansible.builtin.set_fact:
smoketest_instance_type: "{{ 't3a.small' if smoketest_amd.rc == 0 else 't3.small' }}"
- name: smoketest — find AMI ID
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-images --query Images[0].ImageId --output text"
register: smoketest_ami
until: smoketest_ami.rc == 0
retries: 10
delay: 3
changed_when: false
- name: smoketest — find public subnet (best-effort)
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-subnets
--filters Name=map-public-ip-on-launch,Values=true
--query Subnets[0].SubnetId
--output text
register: smoketest_subnet_public
changed_when: false
failed_when: false
- name: smoketest — fallback to first subnet (best-effort)
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-subnets --query Subnets[0].SubnetId --output text"
register: smoketest_subnet_any
changed_when: false
failed_when: false
when:
- smoketest_subnet_public.rc != 0 or smoketest_subnet_public.stdout in ['', 'None']
- name: smoketest — resolve subnet id (empty string if none)
ansible.builtin.set_fact:
smoketest_subnet_id: >-
{%- if smoketest_subnet_public.rc == 0 and smoketest_subnet_public.stdout not in ['', 'None'] -%}
{{ smoketest_subnet_public.stdout }}
{%- elif (smoketest_subnet_any.rc | default(1)) == 0 and (smoketest_subnet_any.stdout | default('')) not in ['', 'None'] -%}
{{ smoketest_subnet_any.stdout }}
{%- else -%}{%- endif -%}
- name: smoketest — skip launch if no subnet (fresh install with no user VPC)
ansible.builtin.debug:
msg: |
No subnet found — skipping instance launch.
Services are up, CA is trusted, keypair imported, AMI imported.
Create a VPC+subnet via tofu examples (or `aws ec2 create-vpc` /
`create-subnet`) then re-run if you want a live instance.
when: smoketest_subnet_id | trim == ''
- name: smoketest — launch instance
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 run-instances
--image-id {{ smoketest_ami.stdout }}
--instance-type {{ smoketest_instance_type }}
--key-name spinifex-key
--subnet-id {{ smoketest_subnet_id | trim }}
--count 1
--query Instances[0].InstanceId
--output text
register: smoketest_instance
until: smoketest_instance.rc == 0
retries: 5
delay: 3
changed_when: true
when: smoketest_subnet_id | trim != ''
- name: smoketest — wait for instance running (proves launch path works)
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-instances
--instance-ids {{ smoketest_instance.stdout }}
--query Reservations[0].Instances[0].State.Name
--output text
register: smoketest_instance_state
until: smoketest_instance_state.stdout | trim == 'running'
retries: 30
delay: 2
changed_when: false
failed_when: false
when:
- smoketest_subnet_id | trim != ''
- smoketest_instance.stdout is defined
- smoketest_instance.stdout | trim != ''
- name: smoketest — terminate instance (cleanup)
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 terminate-instances
--instance-ids {{ smoketest_instance.stdout }}
register: smoketest_terminate
until: smoketest_terminate.rc == 0
retries: 5
delay: 3
changed_when: true
failed_when: false
when:
- smoketest_subnet_id | trim != ''
- smoketest_instance.stdout is defined
- smoketest_instance.stdout | trim != ''
- name: smoketest — summary
ansible.builtin.debug:
msg: |
Smoketest complete:
AMI: {{ smoketest_ami.stdout }}
Subnet: {{ smoketest_subnet_id | trim | default('(none — launch skipped)', true) }}
Instance type: {{ smoketest_instance_type }}
Instance ID: {{ smoketest_instance.stdout | default('(skipped)', true) }}
Final state: {{ smoketest_instance_state.stdout | default('(skipped)', true) | trim }}
Terminated: {{ 'yes' if smoketest_terminate.rc | default(1) == 0 else 'no (launch skipped or terminate failed)' }}
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: snapshot
description: Quiesce spinifex + ovn + ovs, tar state paths, restart.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,154 +0,0 @@
---
# Snapshot a quiesced spinifex node: stop spinifex.target → ovn → ovs in order,
# tar /etc/spinifex + /var/lib/spinifex + /var/lib/ovn + /etc/openvswitch/conf.db,
# restart in reverse. Lets devs checkpoint a working state before risky ops
# (branch switch, viperblock surgery, ovn-controller restart cycle) and roll
# back via dev-restore without a full ~10-min dev-reset.
#
# Caveat: /var/lib/spinifex contains viperblock volumes — snapshot size scales
# with VM disks. Default gzip; switch to a faster compressor by hand if needed.
- name: snapshot — require snapshot_name
ansible.builtin.assert:
that:
- snapshot_name is defined
- snapshot_name | length > 0
- snapshot_name is match('^[A-Za-z0-9._-]+$')
fail_msg: "snapshot_name required (alphanumeric/._-): -e snapshot_name=<name>"
- name: snapshot — resolve bundle path
ansible.builtin.set_fact:
snapshot_bundle_path: "{{ snapshot_dir }}/{{ snapshot_name }}.tar.gz"
- name: snapshot — refuse to overwrite existing bundle
ansible.builtin.stat:
path: "{{ snapshot_bundle_path }}"
register: snapshot_existing
- name: snapshot — assert not already present
ansible.builtin.assert:
that: not snapshot_existing.stat.exists
fail_msg: |
Snapshot {{ snapshot_bundle_path }} already exists. Pick another
name, or `sudo rm` the existing bundle first.
- name: snapshot — ensure snapshot dir exists
become: true
ansible.builtin.file:
path: "{{ snapshot_dir }}"
state: directory
owner: root
group: root
mode: "0750"
- name: snapshot — stop services in dependency order
become: true
ansible.builtin.systemd:
name: "{{ item }}"
state: stopped
loop: "{{ snapshot_services_stop_order }}"
failed_when: false
- name: snapshot — kill lingering QEMU VMs
become: true
ansible.builtin.command: "pkill -x {{ item }}"
loop: "{{ spinifex_qemu_processes }}"
changed_when: false
failed_when: false
- name: snapshot — wait for QEMU exit (up to {{ teardown_process_wait_seconds }}s)
become: true
ansible.builtin.shell: |
set -e
for i in $(seq 1 {{ teardown_process_wait_seconds }}); do
alive=0
for proc in {{ spinifex_qemu_processes | join(' ') }}; do
if pgrep -x "$proc" >/dev/null 2>&1; then alive=1; break; fi
done
if [ "$alive" = "0" ]; then exit 0; fi
sleep 1
done
echo "QEMU still running after {{ teardown_process_wait_seconds }}s" >&2
exit 1
changed_when: false
- name: snapshot — verify which paths actually exist (skip missing silently)
ansible.builtin.stat:
path: "{{ item }}"
register: snapshot_path_stats
loop: "{{ snapshot_paths }}"
loop_control:
label: "{{ item }}"
- name: snapshot — assemble list of present paths
ansible.builtin.set_fact:
snapshot_present_paths: >-
{{ snapshot_path_stats.results
| selectattr('stat.exists') | map(attribute='item') | list }}
- name: snapshot — assert at least one path present
ansible.builtin.assert:
that: snapshot_present_paths | length > 0
fail_msg: |
None of the snapshot paths exist:
{{ snapshot_paths | join(', ') }}
Has spinifex ever been installed on this node?
- name: snapshot — create bundle (tar czf with absolute paths)
become: true
ansible.builtin.command:
cmd: >-
tar --acls --xattrs -czpf {{ snapshot_bundle_path | quote }}
{{ snapshot_present_paths | map('quote') | join(' ') }}
changed_when: true
- name: snapshot — record manifest inside the bundle dir for traceability
become: true
ansible.builtin.copy:
dest: "{{ snapshot_dir }}/{{ snapshot_name }}.manifest"
mode: "0640"
content: |
snapshot_name: {{ snapshot_name }}
created_at: {{ ansible_facts['date_time']['iso8601'] }}
hostname: {{ ansible_facts['hostname'] }}
paths:
{% for p in snapshot_present_paths %} - {{ p }}
{% endfor %}
services_quiesced:
{% for s in snapshot_services_stop_order %} - {{ s }}
{% endfor %}
- name: snapshot — restart services in reverse order
become: true
ansible.builtin.systemd:
name: "{{ item }}"
state: started
loop: "{{ snapshot_services_stop_order | reverse | list }}"
failed_when: false
- name: snapshot — wait for AWS gateway readiness
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
register: snapshot_gw_ready
retries: 30
delay: 2
until: snapshot_gw_ready.rc == 0
changed_when: false
failed_when: false
- name: snapshot — bundle size
become: true
ansible.builtin.stat:
path: "{{ snapshot_bundle_path }}"
register: snapshot_bundle_stat
- name: snapshot — summary
ansible.builtin.debug:
msg: |
Snapshot complete:
name : {{ snapshot_name }}
path : {{ snapshot_bundle_path }}
size : {{ (snapshot_bundle_stat.stat.size | int / 1024 / 1024) | round(1) }} MiB
paths : {{ snapshot_present_paths | join(', ') }}
gateway : {{ 'ready' if snapshot_gw_ready.rc == 0 else 'NOT ready after restart — investigate' }}
Restore with: ansible-playbook playbooks/dev-restore.yml -e snapshot_name={{ snapshot_name }}
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: status
description: Read-only health snapshot for a Spinifex dev node (doctor).
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
-259
View File
@@ -1,259 +0,0 @@
---
# Read-only health snapshot. NEVER mutates state. Every probe must use
# `changed_when: false` and `failed_when: false` so the role always reaches
# the summary task even when the node is half-broken — surfacing brokenness
# is the whole point.
- name: status — spinifex.target activation
ansible.builtin.command: systemctl is-active spinifex.target
register: status_target
changed_when: false
failed_when: false
- name: status — list spinifex-* units (active + failed)
ansible.builtin.shell: |
systemctl list-units 'spinifex-*' --all --no-legend --plain \
--output=json 2>/dev/null || \
systemctl list-units 'spinifex-*' --all --no-legend --plain
register: status_units
changed_when: false
failed_when: false
- name: status — failed spinifex-* units (one line per)
ansible.builtin.shell: |
systemctl list-units 'spinifex-*' --state=failed --no-legend --plain \
| awk '{print $1}'
register: status_failed_units
changed_when: false
failed_when: false
- name: status — port listeners (which process holds each spinifex port)
ansible.builtin.shell: |
ss -tlnpH 'sport = :{{ item.port }}' | head -1
register: status_ports
loop: "{{ spinifex_ports }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item.service }} ({{ item.port }})"
- name: status — OVS tooling present?
ansible.builtin.command: which ovs-vsctl
register: status_ovs_which
changed_when: false
failed_when: false
- name: status — OVS bridges
become: true
ansible.builtin.command: ovs-vsctl list-br
register: status_ovs_brs
changed_when: false
failed_when: false
when: status_ovs_which.rc == 0
- name: status — br-int exists?
become: true
ansible.builtin.command: ovs-vsctl br-exists br-int
register: status_brint
changed_when: false
failed_when: false
when: status_ovs_which.rc == 0
- name: status — ovn-controller active?
ansible.builtin.command: systemctl is-active ovn-controller
register: status_ovn_ctrl
changed_when: false
failed_when: false
- name: status — ovn-central active?
ansible.builtin.command: systemctl is-active ovn-central
register: status_ovn_central
changed_when: false
failed_when: false
- name: status — OVN NB/SB DB files present?
become: true
ansible.builtin.stat:
path: "{{ item }}"
register: status_ovn_dbs
loop:
- "{{ ovn_nb_db }}"
- "{{ ovn_sb_db }}"
loop_control:
label: "{{ item }}"
- name: status — spinifex.toml present?
become: true
ansible.builtin.stat:
path: "{{ spinifex_config_file }}"
register: status_config
- name: status — CA trust present?
become: true
ansible.builtin.stat:
path: "{{ spinifex_ca_trust_path }}"
register: status_ca
- name: status — AWS gateway round-trip (describe-regions)
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions --output text"
register: status_awsgw
changed_when: false
failed_when: false
- name: status — installed spx binary
become: true
ansible.builtin.stat:
path: /usr/local/bin/spx
checksum_algorithm: sha256
register: status_spx_installed
- name: status — built spx binary
ansible.builtin.stat:
path: "{{ spinifex_root }}/bin/spx"
checksum_algorithm: sha256
register: status_spx_built
- name: status — spx version (installed)
ansible.builtin.shell: |
/usr/local/bin/spx version 2>/dev/null | grep -vE '^[0-9]{4}/' | head -1
register: status_spx_version
changed_when: false
failed_when: false
when: status_spx_installed.stat.exists
- name: status — installed nbdkit plugin
become: true
ansible.builtin.shell: |
set -e
pdir=$(nbdkit --dump-config | awk -F= '/^plugindir=/{print $2; exit}')
[ -f "$pdir/nbdkit-viperblock-plugin.so" ] && \
sha256sum "$pdir/nbdkit-viperblock-plugin.so" | awk '{print $1}'
register: status_plugin_installed_sha
changed_when: false
failed_when: false
- name: status — built nbdkit plugin
ansible.builtin.stat:
path: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
checksum_algorithm: sha256
register: status_plugin_built
- name: status — microvm artifacts (built)
ansible.builtin.stat:
path: "{{ spinifex_root }}/build/microvm/{{ item }}"
checksum_algorithm: sha256
register: status_microvm_built
loop:
- vmlinuz
- initramfs.cpio.gz
- name: status — microvm artifacts (installed)
become: true
ansible.builtin.stat:
path: /usr/share/spinifex/microvm/{{ item }}
checksum_algorithm: sha256
register: status_microvm_installed
loop:
- vmlinuz
- initramfs.cpio.gz
- name: status — running QEMU VMs
ansible.builtin.shell: pgrep -fa 'qemu-system-' | wc -l
register: status_qemu_count
changed_when: false
failed_when: false
- name: status — count AMIs
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-images --query length(Images) --output text"
register: status_ami_count
changed_when: false
failed_when: false
- name: status — count keypairs
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-key-pairs --query length(KeyPairs) --output text"
register: status_key_count
changed_when: false
failed_when: false
- name: status — count volumes
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-volumes --query length(Volumes) --output text"
register: status_vol_count
changed_when: false
failed_when: false
- name: status — count running instances
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-instances
--filters Name=instance-state-name,Values=running
--query length(Reservations[].Instances[]) --output text
register: status_instance_count
changed_when: false
failed_when: false
- name: status — compute drift facts
ansible.builtin.set_fact:
status_spx_drift: >-
{{ status_spx_installed.stat.exists and status_spx_built.stat.exists and
(status_spx_installed.stat.checksum | default('')) !=
(status_spx_built.stat.checksum | default('')) }}
status_plugin_drift: >-
{{ status_plugin_built.stat.exists and
(status_plugin_installed_sha.stdout | default('') | trim) !=
(status_plugin_built.stat.checksum | default('')) }}
status_microvm_built_present: "{{ status_microvm_built.results
| map(attribute='stat.exists') | min }}"
- name: status — render port lines
ansible.builtin.set_fact:
status_port_lines: >-
{{ status_port_lines | default([]) +
[('%-24s' | format(item.item.service ~ ' (' ~ item.item.port ~ ')'))
~ ': '
~ ('listening' if (item.stdout | length) > 0 else 'free')] }}
loop: "{{ status_ports.results }}"
loop_control:
label: "{{ item.item.service }}"
- name: status — assemble summary
ansible.builtin.set_fact:
status_summary_lines: >-
{{
[
'── Spinifex dev-node status ──────────────────────────────────────────────',
'target : ' ~ (status_target.stdout | trim),
'failed units : ' ~ (status_failed_units.stdout_lines | length)
~ ((' → ' ~ (status_failed_units.stdout_lines | join(', '))) if (status_failed_units.stdout_lines | length) > 0 else ''),
'── Ports ──'
]
+ status_port_lines
+ [
'── Networking ──',
'ovs-vsctl : ' ~ ('present' if status_ovs_which.rc == 0 else 'MISSING'),
'ovs bridges : ' ~ ((status_ovs_brs.stdout_lines | default([]) | join(', ')) or '(none)'),
'br-int : ' ~ ('present' if (status_brint.rc | default(1)) == 0 else 'MISSING'),
'ovn-controller : ' ~ (status_ovn_ctrl.stdout | trim),
'ovn-central : ' ~ (status_ovn_central.stdout | trim),
'ovn NB DB : ' ~ ('present' if status_ovn_dbs.results[0].stat.exists else 'absent'),
'ovn SB DB : ' ~ ('present' if status_ovn_dbs.results[1].stat.exists else 'absent'),
'── Config / trust ──',
'spinifex.toml : ' ~ ('present' if status_config.stat.exists else 'MISSING') ~ ' (' ~ spinifex_config_file ~ ')',
'CA trust : ' ~ ('present' if status_ca.stat.exists else 'MISSING') ~ ' (' ~ spinifex_ca_trust_path ~ ')',
'── AWS gateway ──',
'describe-regions : ' ~ ('OK' if status_awsgw.rc == 0 else 'FAIL (rc=' ~ status_awsgw.rc ~ ')'),
'AMIs : ' ~ (status_ami_count.stdout | default('?') | trim),
'keypairs : ' ~ (status_key_count.stdout | default('?') | trim),
'volumes : ' ~ (status_vol_count.stdout | default('?') | trim),
'running instances: ' ~ (status_instance_count.stdout | default('?') | trim),
'running QEMU : ' ~ (status_qemu_count.stdout | default('?') | trim),
'── Versions / drift ──',
'spx (installed) : ' ~ ((status_spx_version.stdout | default('') | trim) or '(not installed)'),
'spx drift : ' ~ ('YES (built ≠ installed) — run ansible-dev-deploy' if status_spx_drift else 'no'),
'plugin drift : ' ~ ('YES (built ≠ installed) — run ansible-dev-deploy' if status_plugin_drift else 'no'),
'microvm built : ' ~ ('present' if status_microvm_built_present else 'not built on this branch'),
'──────────────────────────────────────────────────────────────────────────'
]
}}
- name: status — summary
ansible.builtin.debug:
msg: "{{ status_summary_lines }}"
-54
View File
@@ -1,54 +0,0 @@
# teardown — Leak Catalog
Every task in this role corresponds to a known state source that survived a
previous reset. When the operator reports "reset didn't clean X", add a new
task and a new row here citing the symptom.
## Initial catalog (from reset-dev-env.sh + reset-dev-env.md target)
| State source | Cleaned by | Symptom if leaked |
|---|---|---|
| `spinifex.target` active | `systemctl stop` | Services keep running, file locks held |
| `spinifex-*` units in failed state | `systemctl reset-failed 'spinifex-*'` | `systemctl status` shows red units, next start refuses |
| Spinifex processes (spx, nats, predastore, etc.) | `pkill` pattern + wait loop | Stale listeners on 4222/8443/9999, start fails with "address in use" |
| QEMU VMs | `pkill qemu-system-*` + wait loop | Zombie VMs hold tap interfaces and viperblock state |
| `/etc/spinifex/` | `file: state=absent` | Stale TOML, stale CA, service users can't read fresh config |
| `/var/lib/spinifex/` | `file: state=absent` | Stale WAL, stale volumes, viperblock mounts fail |
| `/var/log/spinifex/` | `file: state=absent` | Logrotate confusion, permission drift after user changes |
| `/run/spinifex/` | `file: state=absent` | Stale sockets, pidfiles from prior systemd run |
| `$HOME/spinifex/` (legacy) | `file: state=absent` | Old dev-mode state shadowing new install |
| OVS bridges (all) | `ovs-vsctl del-br` loop | `br-int` populated with stale ports → `ovn-controller` commit loop |
| OVS `external_ids` | `ovs-vsctl clear Open_vSwitch .` | Stale chassis entries in SBDB |
| `/var/lib/ovn/ovnnb_db.db`, `ovnsb_db.db` | `file: state=absent` | "OVNSB commit failed, force recompute" loop after reset |
| `veth-wan-br` / `veth-wan-ovs` | `ip link delete` | Pair left dangling after setup-ovn.sh re-run |
| `spx-ext-*` macvlan interfaces | `ip link delete` loop | Stale macvlan on WAN NIC, packet duplication |
| Spinifex CA in trust store | remove + `update-ca-certificates` | TLS clients trust old CA, new cert fails verification |
## Not cleaned by default (opt-in)
| State source | Flag | Why opt-in |
|---|---|---|
| `~/.ssh/spinifex-key*` | `-e spinifex_wipe_ssh_keys=true` | Operator may want key to persist across resets |
| `~/.aws/credentials` spinifex profile | `-e spinifex_wipe_aws_creds=true` | `spx admin init --force` rewrites this block anyway; wipe only on demand |
## Known traps
- **`pgrep -f` self-match.** A literal pattern that matches the pattern
string itself causes `pgrep -f` to match our own shell. Mitigation:
bracket trick (`[b]in/spx` matches `bin/spx` but not `[b]in/spx`).
- **Log followers / ansible wrapper false positives.** `journalctl -f
_SYSTEMD_UNIT=spinifex-*` and ansible's own module shell matches the
service-name patterns in their cmdlines. Mitigation:
`spinifex_process_exclude` filter drops `journalctl|ansible|/bin/sh|pgrep|pkill|tail -[fF]`.
- **SIGTERM survivors.** `spx service spinifex start` (and potentially
other service-supervisor processes) ignore or delay SIGTERM. Mitigation:
kill task escalates SIGTERM → SIGKILL after 5s.
## Adding a new entry
1. Add a task to `tasks/main.yml` in the right section (services / state-dirs /
ovn / ovs / netlink / trust / user-owned).
2. Prefix the task `name:` with the state class (e.g. `netlink:`, `ovn:`).
3. Add a row to the catalog above — symptom is what the operator described.
4. If the leak class is general (e.g. "any interface matching spx-*"),
generalise the task; do not pattern-match on a single instance name.
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: teardown
description: Wipe Spinifex dev state. Central leak-enumeration surface — every state location cleaned here is a known leak source. See README.md for catalog.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,242 +0,0 @@
---
- name: guard — detect node count in spinifex.toml
ansible.builtin.shell: |
set -e
if [ -f {{ spinifex_config_file | quote }} ]; then
grep -cE '^\[nodes\.[^.]+\]' {{ spinifex_config_file | quote }} || echo 0
else
echo 0
fi
register: teardown_node_count
changed_when: false
- name: guard — refuse to run on multi-node cluster
ansible.builtin.assert:
that: teardown_node_count.stdout | int <= 1
fail_msg: >-
Multi-node cluster detected ({{ teardown_node_count.stdout }} nodes in
{{ spinifex_config_file }}). Teardown supports single-node only;
use 'spx admin cluster shutdown' per node.
- name: services — stop spinifex.target
become: true
ansible.builtin.systemd:
name: spinifex.target
state: stopped
failed_when: false
- name: services — reset failed state (spinifex-* scope only)
become: true
ansible.builtin.command: systemctl reset-failed 'spinifex-*'
changed_when: false
failed_when: false
- name: processes — kill spinifex processes (SIGTERM, escalate to SIGKILL after 5s)
become: true
ansible.builtin.shell: |
pids=$(pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" \
| awk '{print $1}')
if [ -n "$pids" ]; then
kill $pids 2>/dev/null || true
sleep 5
pids=$(pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" \
| awk '{print $1}')
if [ -n "$pids" ]; then
kill -9 $pids 2>/dev/null || true
fi
fi
changed_when: false
failed_when: false
- name: processes — wait for exit (up to {{ teardown_process_wait_seconds }}s)
become: true
ansible.builtin.shell: |
for i in $(seq 1 {{ teardown_process_wait_seconds }}); do
pids=$(pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" \
| awk '{print $1}')
if [ -z "$pids" ]; then
exit 0
fi
sleep 1
done
echo "still running after {{ teardown_process_wait_seconds }}s:" >&2
pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" >&2 || true
exit 1
changed_when: false
- name: qemu — kill lingering VMs
become: true
ansible.builtin.command: "pkill -x {{ item }}"
loop: "{{ spinifex_qemu_processes }}"
changed_when: false
failed_when: false
- name: qemu — wait for exit (up to {{ teardown_process_wait_seconds }}s)
become: true
ansible.builtin.shell: |
set -e
for i in $(seq 1 {{ teardown_process_wait_seconds }}); do
alive=0
for proc in {{ spinifex_qemu_processes | join(' ') }}; do
if pgrep -x "$proc" >/dev/null 2>&1; then alive=1; break; fi
done
if [ "$alive" = "0" ]; then exit 0; fi
sleep 1
done
echo "QEMU still running after {{ teardown_process_wait_seconds }}s" >&2
exit 1
changed_when: false
- name: state-dirs — remove production dirs
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ spinifex_etc_dir }}"
- "{{ spinifex_data_dir }}"
- "{{ spinifex_log_dir }}"
- "{{ spinifex_run_dir }}"
- name: state-dirs — remove legacy $HOME/spinifex (opt-in)
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.file:
path: "{{ spinifex_legacy_home_dir }}"
state: absent
when: spinifex_wipe_legacy_home | bool
- name: ovs — detect tooling
ansible.builtin.command: which ovs-vsctl
register: teardown_ovs_which
changed_when: false
failed_when: false
- name: ovs — ensure openvswitch-switch is running for cleanup
become: true
ansible.builtin.systemd:
name: openvswitch-switch
state: started
when: teardown_ovs_which.rc == 0
failed_when: false
- name: ovs — list bridges
become: true
ansible.builtin.command: ovs-vsctl list-br
register: teardown_ovs_brs
changed_when: false
failed_when: false
when: teardown_ovs_which.rc == 0
- name: ovs — delete all bridges
become: true
ansible.builtin.command: "ovs-vsctl --if-exists del-br {{ item }}"
loop: "{{ teardown_ovs_brs.stdout_lines | default([]) }}"
changed_when: true
when: teardown_ovs_which.rc == 0
- name: ovs — clear external_ids
become: true
ansible.builtin.command: ovs-vsctl --if-exists clear Open_vSwitch . external_ids
changed_when: true
failed_when: false
when: teardown_ovs_which.rc == 0
- name: ovs — stop openvswitch-switch after cleanup
become: true
ansible.builtin.systemd:
name: openvswitch-switch
state: stopped
when: teardown_ovs_which.rc == 0
failed_when: false
- name: ovn — stop ovn-central
become: true
ansible.builtin.systemd:
name: ovn-central
state: stopped
failed_when: false
- name: ovn — stop ovn-controller
become: true
ansible.builtin.systemd:
name: ovn-controller
state: stopped
failed_when: false
- name: ovn — delete NB/SB DB files
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ ovn_nb_db }}"
- "{{ ovn_sb_db }}"
- name: netlink — enumerate spinifex-managed veth/tap
ansible.builtin.shell: |
ip -o link show | awk -F': ' '{print $2}' | cut -d'@' -f1 | \
grep -E '^(veth-wan-br|veth-wan-ovs|spx-ext-|spx-tap-|spx-)' || true
register: teardown_links
changed_when: false
- name: netlink — delete enumerated interfaces
become: true
ansible.builtin.command: "ip link delete {{ item }}"
loop: "{{ teardown_links.stdout_lines }}"
changed_when: true
failed_when: false
- name: trust — remove spinifex CA from system store
become: true
ansible.builtin.file:
path: "{{ spinifex_ca_trust_path }}"
state: absent
register: teardown_ca_removed
- name: trust — rebuild CA bundle (without --fresh, preserve other CAs)
become: true
ansible.builtin.command: update-ca-certificates
changed_when: true
when: teardown_ca_removed.changed
- name: user-owned — remove spinifex profile from AWS credentials (opt-in)
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.blockinfile:
path: "~/.aws/credentials"
marker: "# {mark} SPINIFEX MANAGED BLOCK"
block: ""
state: absent
when: spinifex_wipe_aws_creds | bool
failed_when: false
- name: user-owned — remove spinifex SSH key (opt-in)
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.file:
path: "~/.ssh/{{ item }}"
state: absent
loop:
- spinifex-key
- spinifex-key.pub
when: spinifex_wipe_ssh_keys | bool
- name: summary
ansible.builtin.debug:
msg: |
Teardown complete. Catalog: roles/teardown/README.md
systemd : spinifex.target stopped, spinifex-* failed cleared
procs : {{ spinifex_process_pattern }}
qemu : {{ spinifex_qemu_processes | join(', ') }}
dirs : {{ spinifex_etc_dir }}, {{ spinifex_data_dir }}, {{ spinifex_log_dir }}, {{ spinifex_run_dir }}{{ ', ' ~ spinifex_legacy_home_dir if spinifex_wipe_legacy_home else '' }}
ovs : all bridges deleted, external_ids cleared
ovn : NB/SB DB files removed
netlink : {{ teardown_links.stdout_lines | length }} interface(s) removed
trust : {{ spinifex_ca_trust_path }} removed{% if spinifex_wipe_aws_creds %}
aws : credentials block removed{% endif %}{% if spinifex_wipe_ssh_keys %}
ssh : spinifex-key pair removed{% endif %}
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: version
description: Detailed build-vs-install drift report for spinifex artifacts.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -1,161 +0,0 @@
---
# Detailed build-vs-install drift report for the four artifacts that
# `ansible-dev-deploy` swaps: spx, the nbdkit viperblock plugin, and the
# two microvm artifacts (kernel + initramfs). Pure read-only; faster
# than dev-status (no AWS / OVN / port probing).
- name: version — built spx
ansible.builtin.stat:
path: "{{ spinifex_root }}/bin/spx"
checksum_algorithm: sha256
register: ver_spx_built
- name: version — installed spx
become: true
ansible.builtin.stat:
path: /usr/local/bin/spx
checksum_algorithm: sha256
register: ver_spx_installed
- name: version — installed spx version string
ansible.builtin.shell: |
/usr/local/bin/spx version 2>/dev/null | grep -vE '^[0-9]{4}/' | head -1
register: ver_spx_version
changed_when: false
failed_when: false
when: ver_spx_installed.stat.exists
- name: version — built plugin
ansible.builtin.stat:
path: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
checksum_algorithm: sha256
register: ver_plugin_built
- name: version — nbdkit plugin dir
ansible.builtin.shell: |
nbdkit --dump-config 2>/dev/null | awk -F= '/^plugindir=/{print $2; exit}'
register: ver_plugindir
changed_when: false
failed_when: false
- name: version — installed plugin
become: true
ansible.builtin.stat:
path: "{{ ver_plugindir.stdout | trim }}/nbdkit-viperblock-plugin.so"
checksum_algorithm: sha256
register: ver_plugin_installed
when: ver_plugindir.stdout | trim | length > 0
- name: version — built microvm artifacts
ansible.builtin.stat:
path: "{{ spinifex_root }}/build/microvm/{{ item }}"
checksum_algorithm: sha256
register: ver_microvm_built
loop:
- vmlinuz
- initramfs.cpio.gz
- name: version — installed microvm artifacts
become: true
ansible.builtin.stat:
path: /usr/share/spinifex/microvm/{{ item }}
checksum_algorithm: sha256
register: ver_microvm_installed
loop:
- vmlinuz
- initramfs.cpio.gz
- name: version — sub-repo HEAD short-rev (spinifex)
ansible.builtin.command: git -C {{ spinifex_root | quote }} rev-parse --short HEAD
register: ver_spinifex_head
changed_when: false
failed_when: false
- name: version — sub-repo HEAD short-rev (viperblock)
ansible.builtin.command: git -C {{ mulga_root }}/viperblock rev-parse --short HEAD
register: ver_viperblock_head
changed_when: false
failed_when: false
- name: version — sub-repo HEAD short-rev (predastore)
ansible.builtin.command: git -C {{ mulga_root }}/predastore rev-parse --short HEAD
register: ver_predastore_head
changed_when: false
failed_when: false
- name: version — assemble per-artifact rows
ansible.builtin.set_fact:
ver_artifact_rows: >-
{{
[
('%-18s built=%s installed=%s drift=%s' | format(
'spx',
(ver_spx_built.stat.checksum | default('-') | truncate(13, true, ''))
if ver_spx_built.stat.exists else 'MISSING',
(ver_spx_installed.stat.checksum | default('-') | truncate(13, true, ''))
if ver_spx_installed.stat.exists else 'MISSING',
('YES' if ver_spx_built.stat.exists and ver_spx_installed.stat.exists
and ver_spx_built.stat.checksum != ver_spx_installed.stat.checksum
else 'no')
)),
('%-18s built=%s installed=%s drift=%s' | format(
'plugin (nbdkit)',
(ver_plugin_built.stat.checksum | default('-') | truncate(13, true, ''))
if ver_plugin_built.stat.exists else 'MISSING',
(ver_plugin_installed.stat.checksum | default('-') | truncate(13, true, ''))
if (ver_plugin_installed.stat | default({})).get('exists', false) else 'MISSING',
('YES' if ver_plugin_built.stat.exists
and (ver_plugin_installed.stat | default({})).get('exists', false)
and ver_plugin_built.stat.checksum != ver_plugin_installed.stat.checksum
else 'no')
)),
('%-18s built=%s installed=%s drift=%s' | format(
'microvm vmlinuz',
(ver_microvm_built.results[0].stat.checksum | default('-') | truncate(13, true, ''))
if ver_microvm_built.results[0].stat.exists else 'MISSING',
(ver_microvm_installed.results[0].stat.checksum | default('-') | truncate(13, true, ''))
if ver_microvm_installed.results[0].stat.exists else 'MISSING',
('YES' if ver_microvm_built.results[0].stat.exists
and ver_microvm_installed.results[0].stat.exists
and ver_microvm_built.results[0].stat.checksum
!= ver_microvm_installed.results[0].stat.checksum
else 'no')
)),
('%-18s built=%s installed=%s drift=%s' | format(
'microvm initramfs',
(ver_microvm_built.results[1].stat.checksum | default('-') | truncate(13, true, ''))
if ver_microvm_built.results[1].stat.exists else 'MISSING',
(ver_microvm_installed.results[1].stat.checksum | default('-') | truncate(13, true, ''))
if ver_microvm_installed.results[1].stat.exists else 'MISSING',
('YES' if ver_microvm_built.results[1].stat.exists
and ver_microvm_installed.results[1].stat.exists
and ver_microvm_built.results[1].stat.checksum
!= ver_microvm_installed.results[1].stat.checksum
else 'no')
))
]
}}
- name: version — compute any-drift flag
ansible.builtin.set_fact:
ver_any_drift: "{{ 'YES' in (ver_artifact_rows | join(' ')) }}"
- name: version — summary
ansible.builtin.debug:
msg: >-
{{
[
'── Spinifex build vs install ─────────────────────────────────────────────',
'spx (installed): ' ~ ((ver_spx_version.stdout | default('') | trim) or '(not installed)')
]
+ ver_artifact_rows
+ [
'── Sub-repo HEADs ──',
'spinifex : ' ~ ((ver_spinifex_head.stdout | default('') | trim) or '(not a git repo)'),
'viperblock : ' ~ ((ver_viperblock_head.stdout | default('') | trim) or '(missing)'),
'predastore : ' ~ ((ver_predastore_head.stdout | default('') | trim) or '(missing)'),
'──────────────────────────────────────────────────────────────────────────',
('DRIFT detected — run: make ansible-dev-deploy' if ver_any_drift
else 'No drift; built = installed for all artifacts.')
]
}}
-7
View File
@@ -1,7 +0,0 @@
---
galaxy_info:
role_name: vpc
description: Idempotent dev-mode VPC + public subnet + IGW + default route fixture.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
-128
View File
@@ -1,128 +0,0 @@
---
# Tear down the dev VPC fixture. Deletes in reverse dependency order:
# route-table associations, route table, subnet, IGW (detach + delete),
# VPC. Each step is best-effort so a partially-converged fixture still
# converges to "gone" on re-run.
- name: vpc — find VPC by Name tag
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-vpcs
--filters Name=tag:Name,Values={{ dev_vpc_name }}
--query Vpcs[0].VpcId --output text
register: absent_vpc
changed_when: false
- name: vpc — nothing to do
ansible.builtin.debug:
msg: "VPC tagged Name={{ dev_vpc_name }} not found; nothing to delete."
when: absent_vpc.stdout | trim in ['', 'None']
- name: vpc — short-circuit when no VPC
ansible.builtin.meta: end_play
when: absent_vpc.stdout | trim in ['', 'None']
- name: vpc — set VPC id fact
ansible.builtin.set_fact:
absent_vpc_id: "{{ absent_vpc.stdout | trim }}"
- name: vpc — find subnet
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-subnets
--filters Name=vpc-id,Values={{ absent_vpc_id }}
Name=tag:Name,Values={{ dev_vpc_name }}-public
--query Subnets[0].SubnetId --output text
register: absent_subnet
changed_when: false
# Enumerate ALL non-main route tables in the VPC (main RT can't be
# deleted; it goes when the VPC does). Catches orphans from earlier
# runs that landed before the create-tags workaround.
- name: vpc — find all non-main route tables in VPC
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-route-tables
--filters Name=vpc-id,Values={{ absent_vpc_id }}
--query RouteTables[?!(Associations[?Main==`true`])].RouteTableId
--output text
register: absent_rt_list
changed_when: false
- name: vpc — split RT id list
ansible.builtin.set_fact:
absent_rt_ids: "{{ absent_rt_list.stdout.split() | reject('equalto', '') | list }}"
- name: vpc — find all subnet/RT associations to disassociate
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-route-tables
--route-table-ids {{ item }}
--query RouteTables[0].Associations[?SubnetId!=null].RouteTableAssociationId
--output text
register: absent_assocs
loop: "{{ absent_rt_ids }}"
changed_when: false
failed_when: false
- name: vpc — disassociate subnets from route tables
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 disassociate-route-table
--association-id {{ item }}
loop: "{{ absent_assocs.results
| map(attribute='stdout')
| map('default', '')
| map('split')
| flatten
| reject('equalto', '')
| list }}"
changed_when: true
failed_when: false
- name: vpc — delete all non-main route tables
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 delete-route-table
--route-table-id {{ item }}
loop: "{{ absent_rt_ids }}"
changed_when: true
failed_when: false
- name: vpc — delete subnet
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 delete-subnet
--subnet-id {{ absent_subnet.stdout | trim }}
when: absent_subnet.stdout | trim not in ['', 'None']
changed_when: true
failed_when: false
- name: vpc — find IGW
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-internet-gateways
--filters Name=attachment.vpc-id,Values={{ absent_vpc_id }}
--query InternetGateways[0].InternetGatewayId --output text
register: absent_igw
changed_when: false
- name: vpc — detach IGW
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 detach-internet-gateway
--internet-gateway-id {{ absent_igw.stdout | trim }}
--vpc-id {{ absent_vpc_id }}
when: absent_igw.stdout | trim not in ['', 'None']
changed_when: true
failed_when: false
- name: vpc — delete IGW
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 delete-internet-gateway
--internet-gateway-id {{ absent_igw.stdout | trim }}
when: absent_igw.stdout | trim not in ['', 'None']
changed_when: true
failed_when: false
- name: vpc — delete VPC
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 delete-vpc
--vpc-id {{ absent_vpc_id }}
changed_when: true
failed_when: false
- name: vpc — summary (absent)
ansible.builtin.debug:
msg: "Dev VPC {{ dev_vpc_name }} (was {{ absent_vpc_id }}) torn down."
-3
View File
@@ -1,3 +0,0 @@
---
- name: vpc — dispatch by state
ansible.builtin.include_tasks: "{{ dev_vpc_state }}.yml"
-209
View File
@@ -1,209 +0,0 @@
---
# Idempotent create. Every step describes-then-creates: re-running on an
# already-converged VPC produces zero changes. Tag everything with
# Name={{ dev_vpc_name }} so a single tag query finds the whole graph.
- name: vpc — find existing VPC by Name tag
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-vpcs
--filters Name=tag:Name,Values={{ dev_vpc_name }}
--query Vpcs[0].VpcId --output text
register: vpc_existing
changed_when: false
- name: vpc — create VPC if missing
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 create-vpc
--cidr-block {{ dev_vpc_cidr }}
--tag-specifications
ResourceType=vpc,Tags=[{Key=Name,Value={{ dev_vpc_name }}}]
--query Vpc.VpcId --output text
register: vpc_created
when: vpc_existing.stdout | trim in ['', 'None']
changed_when: true
- name: vpc — set VPC id fact
ansible.builtin.set_fact:
dev_vpc_id: >-
{{ vpc_created.stdout | default('') | trim
if (vpc_existing.stdout | trim in ['', 'None'])
else vpc_existing.stdout | trim }}
- name: vpc — assert VPC id resolved
ansible.builtin.assert:
that: dev_vpc_id | length > 0
fail_msg: "VPC id could not be resolved after describe/create"
- name: vpc — find existing IGW attached to this VPC
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-internet-gateways
--filters Name=attachment.vpc-id,Values={{ dev_vpc_id }}
--query InternetGateways[0].InternetGatewayId --output text
register: vpc_igw_existing
changed_when: false
- name: vpc — create IGW if missing
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 create-internet-gateway
--tag-specifications
ResourceType=internet-gateway,Tags=[{Key=Name,Value={{ dev_vpc_name }}-igw}]
--query InternetGateway.InternetGatewayId --output text
register: vpc_igw_created
when: vpc_igw_existing.stdout | trim in ['', 'None']
changed_when: true
- name: vpc — set IGW id fact
ansible.builtin.set_fact:
dev_vpc_igw_id: >-
{{ vpc_igw_created.stdout | default('') | trim
if (vpc_igw_existing.stdout | trim in ['', 'None'])
else vpc_igw_existing.stdout | trim }}
- name: vpc — attach IGW to VPC if needed
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 attach-internet-gateway
--internet-gateway-id {{ dev_vpc_igw_id }}
--vpc-id {{ dev_vpc_id }}
register: vpc_igw_attach
when: vpc_igw_existing.stdout | trim in ['', 'None']
changed_when: true
failed_when:
- vpc_igw_attach.rc | default(0) != 0
- "'Resource.AlreadyAssociated' not in (vpc_igw_attach.stderr | default(''))"
- name: vpc — find existing subnet by Name tag
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-subnets
--filters Name=vpc-id,Values={{ dev_vpc_id }}
Name=tag:Name,Values={{ dev_vpc_name }}-public
--query Subnets[0].SubnetId --output text
register: vpc_subnet_existing
changed_when: false
- name: vpc — create subnet if missing
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 create-subnet
--vpc-id {{ dev_vpc_id }}
--cidr-block {{ dev_vpc_subnet_cidr }}
--availability-zone {{ dev_vpc_subnet_az }}
--tag-specifications
ResourceType=subnet,Tags=[{Key=Name,Value={{ dev_vpc_name }}-public}]
--query Subnet.SubnetId --output text
register: vpc_subnet_created
when: vpc_subnet_existing.stdout | trim in ['', 'None']
changed_when: true
- name: vpc — set subnet id fact
ansible.builtin.set_fact:
dev_vpc_subnet_id: >-
{{ vpc_subnet_created.stdout | default('') | trim
if (vpc_subnet_existing.stdout | trim in ['', 'None'])
else vpc_subnet_existing.stdout | trim }}
- name: vpc — ensure subnet has map-public-ip-on-launch
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 modify-subnet-attribute
--subnet-id {{ dev_vpc_subnet_id }}
--map-public-ip-on-launch
register: vpc_subnet_mapip
changed_when: vpc_subnet_existing.stdout | trim in ['', 'None']
failed_when: false
# Try tag-based lookup first. If the gateway didn't tag a previously
# created RT (spinifex gateway silently drops --tag-specifications on
# create-route-table AND no-ops create-tags for route tables — see
# mulga-siv-94; works correctly for subnets), fall back to "RT
# associated with our subnet" which doesn't need tags.
- name: vpc — find existing route table by Name tag
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-route-tables
--filters Name=vpc-id,Values={{ dev_vpc_id }}
Name=tag:Name,Values={{ dev_vpc_name }}-public
--query RouteTables[0].RouteTableId --output text
register: vpc_rt_existing
changed_when: false
- name: vpc — fallback lookup by subnet association
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-route-tables
--filters Name=association.subnet-id,Values={{ dev_vpc_subnet_id }}
--query RouteTables[0].RouteTableId --output text
register: vpc_rt_byassoc
changed_when: false
when: vpc_rt_existing.stdout | trim in ['', 'None']
- name: vpc — create route table if neither lookup found one
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 create-route-table
--vpc-id {{ dev_vpc_id }}
--query RouteTable.RouteTableId --output text
register: vpc_rt_created
when:
- vpc_rt_existing.stdout | trim in ['', 'None']
- (vpc_rt_byassoc.stdout | default('') | trim) in ['', 'None']
changed_when: true
- name: vpc — set route-table id fact
ansible.builtin.set_fact:
dev_vpc_rt_id: >-
{%- if (vpc_rt_existing.stdout | trim) not in ['', 'None'] -%}
{{ vpc_rt_existing.stdout | trim }}
{%- elif (vpc_rt_byassoc.stdout | default('') | trim) not in ['', 'None'] -%}
{{ vpc_rt_byassoc.stdout | default('') | trim }}
{%- else -%}
{{ vpc_rt_created.stdout | default('') | trim }}
{%- endif -%}
- name: vpc — assert route-table id resolved
ansible.builtin.assert:
that: dev_vpc_rt_id | length > 0
fail_msg: "Route table id could not be resolved after describe/create"
# Spinifex gateway dropped --tag-specifications on the create call;
# apply via create-tags so re-runs can find the RT via tag filter.
- name: vpc — tag route table (workaround for create-tag-specifications gap)
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 create-tags
--resources {{ dev_vpc_rt_id }}
--tags Key=Name,Value={{ dev_vpc_name }}-public
changed_when: false
failed_when: false
- name: vpc — add default route via IGW (ignore RouteAlreadyExists)
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 create-route
--route-table-id {{ dev_vpc_rt_id }}
--destination-cidr-block 0.0.0.0/0
--gateway-id {{ dev_vpc_igw_id }}
register: vpc_route_add
changed_when:
- vpc_route_add.rc == 0
- "'RouteAlreadyExists' not in (vpc_route_add.stderr | default(''))"
failed_when:
- vpc_route_add.rc != 0
- "'RouteAlreadyExists' not in (vpc_route_add.stderr | default(''))"
- name: vpc — associate subnet with route table (ignore already-associated)
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 associate-route-table
--route-table-id {{ dev_vpc_rt_id }}
--subnet-id {{ dev_vpc_subnet_id }}
register: vpc_assoc
changed_when:
- vpc_assoc.rc == 0
- "'Resource.AlreadyAssociated' not in (vpc_assoc.stderr | default(''))"
failed_when:
- vpc_assoc.rc != 0
- "'Resource.AlreadyAssociated' not in (vpc_assoc.stderr | default(''))"
- name: vpc — summary (present)
ansible.builtin.debug:
msg: |
Dev VPC ready:
VPC : {{ dev_vpc_id }} ({{ dev_vpc_cidr }})
IGW : {{ dev_vpc_igw_id }}
Subnet : {{ dev_vpc_subnet_id }} ({{ dev_vpc_subnet_cidr }} @ {{ dev_vpc_subnet_az }}, public)
Route table : {{ dev_vpc_rt_id }}
Smoketest now finds this subnet via its existing
Name=map-public-ip-on-launch,Values=true filter.
@@ -1,27 +0,0 @@
#!/usr/bin/env bash
# Run `aws` CLI inside a shell that has the `spinifex` supplementary group
# active, with AWS_PROFILE=spinifex preset. Works around mulga-siv-92:
# ansible's `become` is a no-op when the target user equals the calling
# user, so `sudo -u` is never invoked and supplementary groups are not
# re-initialised. Without the `spinifex` group, /etc/spinifex/ca.pem (the
# AWS endpoint CA bundle) is unreadable and aws fails with SSL Errno 13.
#
# Usage from ansible tasks:
# ansible.builtin.command:
# cmd: "{{ playbook_dir }}/../scripts/aws-as-spinifex.sh ec2 describe-regions"
#
# All args are forwarded to `aws`. Quoting is preserved via printf %q so
# things like `--query 'length(Images)'` reach aws intact.
set -euo pipefail
if [[ $# -eq 0 ]]; then
echo "usage: $0 <aws-cli-args>" >&2
exit 64
fi
cmd="AWS_PROFILE=spinifex aws"
for arg in "$@"; do
cmd+=" $(printf '%q' "$arg")"
done
exec sg spinifex -c "$cmd"
@@ -1,18 +0,0 @@
#!/usr/bin/env bash
# Run `spx` inside a shell that has the `spinifex` supplementary group
# active. Works around mulga-siv-92 (see aws-as-spinifex.sh for details).
# All args are forwarded to /usr/local/bin/spx via printf %q so flag
# values with spaces or shell metachars survive intact.
set -euo pipefail
if [[ $# -eq 0 ]]; then
echo "usage: $0 <spx-args>" >&2
exit 64
fi
cmd="AWS_PROFILE=spinifex /usr/local/bin/spx"
for arg in "$@"; do
cmd+=" $(printf '%q' "$arg")"
done
exec sg spinifex -c "$cmd"
-137
View File
@@ -1,137 +0,0 @@
---
# Production layout (per spinifex/scripts/setup.sh).
spinifex_etc_dir: /etc/spinifex
spinifex_data_dir: /var/lib/spinifex
spinifex_log_dir: /var/log/spinifex
spinifex_run_dir: /run/spinifex
spinifex_config_file: "{{ spinifex_etc_dir }}/spinifex.toml"
spinifex_ca_trust_path: /usr/local/share/ca-certificates/spinifex-ca.crt
spinifex_ca_src: "{{ spinifex_etc_dir }}/ca.pem"
# Legacy $HOME layout (still present on older dev boxes).
spinifex_legacy_home_dir: "{{ lookup('env','HOME') }}/spinifex"
# OVN state.
ovn_db_dir: /var/lib/ovn
ovn_nb_db: "{{ ovn_db_dir }}/ovnnb_db.db"
ovn_sb_db: "{{ ovn_db_dir }}/ovnsb_db.db"
# Region / init.
spinifex_region: ap-southeast-2
spinifex_az: "{{ spinifex_region }}a"
spinifex_node_name: node1
spinifex_node_count: 1
# External network.
# spx admin init contract (as of network-layer-redesign):
# external_mode : "pool" or "" (NAT path removed)
# external_pool : MANDATORY when mode=pool — set start/end below or pass -e.
# external_gw : auto-detected from default route when blank.
# external_prefix: auto-detected from WAN when blank.
# external_gw_ip : OVN router external-IP override (was nat_gateway_ip).
# no_external : true → emit --no-external (skip external networking).
spinifex_external_mode: pool
spinifex_external_no_external: false
spinifex_external_pool_start: ""
spinifex_external_pool_end: ""
spinifex_external_gateway: ""
spinifex_external_prefix_len: ""
spinifex_external_gateway_ip: ""
# Preflight checks.
spinifex_ports:
- { port: 4222, service: NATS }
- { port: 8443, service: Predastore }
- { port: 9999, service: "Spinifex Gateway" }
spinifex_required_commands:
- go
- make
- nbdkit
spinifex_optional_commands:
- air
spinifex_ovn_commands:
- ovs-vsctl
- ovn-controller
- ovn-nbctl
spinifex_components:
- viperblock
- predastore
# Process patterns for teardown.
# Bracket trick ([b]in/spx etc.) prevents the pattern literal from matching
# our own pgrep/shell cmdline. Exclude filter drops benign log followers and
# ansible's own wrappers that also mention these service names.
spinifex_process_pattern: "([b]in/spx|spinifex-[u]i|nats-[s]erver|[p]redastore|[v]iperblock|[v]pcd)"
spinifex_process_exclude: "(journalctl|ansible|/bin/sh|pgrep|pkill|tail -[fF])"
spinifex_qemu_processes:
- qemu-system-x86_64
- qemu-system-aarch64
# Invoking user (when run under sudo).
spinifex_invoking_user: "{{ lookup('env','SUDO_USER') | default(ansible_facts['user_id'], true) }}"
# Opt-in wipes.
spinifex_wipe_ssh_keys: false
spinifex_wipe_aws_creds: false
spinifex_wipe_legacy_home: true
# Timeouts.
teardown_process_wait_seconds: 30
# Path resolution — playbook_dir is scripts/ansible/playbooks/ at runtime.
spinifex_root: "{{ lookup('pipe', 'readlink -f ' ~ playbook_dir ~ '/../../..') }}"
mulga_root: "{{ lookup('pipe', 'readlink -f ' ~ playbook_dir ~ '/../../../..') }}"
# Wrappers around `aws` and `spx` that re-enter a shell with the
# `spinifex` supplementary group active. Needed because ansible's
# `become_user` is a no-op when target user equals current user, so
# supplementary groups are never re-initialised and /etc/spinifex/ca.pem
# (drwxr-x--- root:spinifex) is unreadable.
spinifex_aws_cli: "{{ playbook_dir }}/../scripts/aws-as-spinifex.sh"
spinifex_spx_cli: "{{ playbook_dir }}/../scripts/spx-as-spinifex.sh"
# dev-vpc fixture. Deterministic VPC + public subnet + IGW + default
# route. Idempotent: re-running detects existing resources by Name tag.
# Used by smoketest's launch path (which looks for any
# map-public-ip-on-launch=true subnet).
dev_vpc_name: spinifex-dev-vpc
dev_vpc_cidr: 10.99.0.0/16
dev_vpc_subnet_cidr: 10.99.1.0/24
dev_vpc_subnet_az: "{{ spinifex_az }}"
dev_vpc_state: present # present | absent
# Build output.
build_tarball_path: /tmp/spinifex-local.tar.gz
# Snapshot / restore. Snapshots quiesce the full spinifex+ovn+ovs stack,
# tar these paths, then restart services. Restore wipes + untars + restarts.
# Default snapshot dir is root-owned so it survives `dev-teardown`
# (teardown only removes the four spinifex_*_dir paths above).
snapshot_dir: /var/lib/spinifex-snapshots
snapshot_paths:
- "{{ spinifex_etc_dir }}"
- "{{ spinifex_data_dir }}"
- "{{ ovn_db_dir }}"
- /etc/openvswitch/conf.db
# Order matters: stop spinifex first (depends on OVN/OVS), then OVN, then OVS.
# Restart in reverse.
snapshot_services_stop_order:
- spinifex.target
- ovn-controller
- ovn-central
- openvswitch-switch
# Smoketest image — parameterised so tofu examples can share the same base.
spinifex_smoketest_distro: debian
spinifex_smoketest_version: "13"
# Local image file path (if present, imported directly; otherwise fall back to --name).
spinifex_smoketest_local_image: "~/images/{{ spinifex_smoketest_distro }}-{{ spinifex_smoketest_version }}.img"
# Multi-node cluster bootstrap (cluster-bootstrap.yml).
# cluster_env, cluster_init_host, spinifex_node_count come from inventory/tofu.py.
cluster_external_pool: "" # "<start>-<end>" or empty for DHCP fallback
cluster_join_token_path: /etc/spinifex/join-token
cluster_formation_port: 4432
cluster_formation_timeout: 60 # seconds (uri retries = value // 2)