feat: Ansible dev lifecycle (parallel to bash scripts) (#178)

Playbooks for preflight / teardown / install / reset covering the same
ground as dev-install.sh, dev-setup.sh, reset-dev-env.sh. Existing bash
scripts untouched — decide which to keep after parity testing.

- roles/teardown + teardown catalog README track every known state
  source; add a task + row when a leak is reported.
- roles/init gates smoketest on awsgw readiness (describe-regions poll).
- roles/smoketest is dup-tolerant, retry-wrapped, auto-terminates the
  launched instance.
- dev-install.yml skips build/install/init when a healthy environment
  is already present (4-signal health check), always runs smoketest.
- dev-reset.yml captures region/az/external-mode scalars from the
  existing spinifex.toml and replays them into init.

Plan: docs/development/improvements/ansible-dev-lifecycle.md (mulga).
Bead: mulga-siv-9.
This commit is contained in:
Josh Siviour
2026-05-01 10:04:37 +10:00
committed by GitHub
parent 69404b38ce
commit b7dea45ee0
24 changed files with 1236 additions and 1 deletions
+1
View File
@@ -31,6 +31,7 @@ Thumbs.db
# Environment
.env
.venv/
# Test reports
*-report.txt
+16 -1
View File
@@ -224,8 +224,23 @@ distro-arm64:
distro-clean:
rm -rf dist/
# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
ansible-dev-preflight:
cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml
ansible-dev-teardown:
cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml
ansible-dev-install:
cd scripts/ansible && ansible-playbook playbooks/dev-install.yml
ansible-dev-reset:
cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-lb-image go_build go_run preflight test test-cover test-race diff-coverage bench run \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
lint fix govulncheck \
distro distro-amd64 distro-arm64 distro-clean
distro distro-amd64 distro-arm64 distro-clean \
ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset
+87
View File
@@ -0,0 +1,87 @@
# Ansible Dev Lifecycle (parallel to bash scripts)
Experimental Ansible-based lifecycle for a single-node Spinifex dev host.
Lives alongside the existing `scripts/dev-*.sh` and `scripts/reset-dev-env.sh`;
neither set is removed. Decide which to keep after parity testing.
Plan: `docs/development/improvements/ansible-dev-lifecycle.md` (mulga top-level).
Bead: `mulga-siv-9`.
## Prerequisites
Venv lives at sub-repo root (`spinifex/.venv/`) — one venv per sub-repo,
shared by any future python tooling in spinifex. Bootstrap from `spinifex/`:
```
sudo apt install python3.13-venv
python3 -m venv .venv
source .venv/bin/activate
pip install 'ansible>=9'
ansible-galaxy collection install -r scripts/ansible/requirements.yml
```
Then `cd scripts/ansible` to run playbooks — `ansible.cfg` resolves the
inventory and roles relative to CWD.
## Playbooks
| Playbook | Purpose | Bash parity |
|---|---|---|
| `playbooks/dev-preflight.yml` | Dependency + port check | `dev-setup.sh` §checks |
| `playbooks/dev-teardown.yml` | Wipe state only (no reinstall) | `reset-dev-env.sh` §teardown |
| `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
| `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
Upcoming (not yet implemented):
- `dev-mode-start.yml` / `dev-mode-stop.yml` — parity with `start-dev.sh` / `stop-dev.sh`
(lowest priority — deprecated bash scripts, CI-only callers).
## Invocation
```
ansible-playbook playbooks/dev-preflight.yml
ansible-playbook playbooks/dev-teardown.yml
ansible-playbook playbooks/dev-install.yml
ansible-playbook playbooks/dev-reset.yml
```
Or via `make` (from `spinifex/`):
```
make ansible-dev-preflight
make ansible-dev-teardown
make ansible-dev-install
make ansible-dev-reset
```
## Variable overrides
Defaults in `vars/defaults.yml`. Override per run with `-e`:
```
ansible-playbook playbooks/dev-teardown.yml -e spinifex_wipe_ssh_keys=true
```
Useful overrides:
| Var | Default | What it does |
|---|---|---|
| `spinifex_wipe_legacy_home` | `true` | Remove `$HOME/spinifex` (legacy dev-mode path) |
| `spinifex_wipe_ssh_keys` | `false` | Remove `~/.ssh/spinifex-key*` |
| `spinifex_wipe_aws_creds` | `false` | Remove spinifex profile from `~/.aws/credentials` |
| `teardown_process_wait_seconds` | `30` | Grace period for processes to exit |
| `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
| `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
| `spinifex_external_mode` | `pool` | `pool` \| `nat` external networking |
| `spinifex_build_lb_image` | `true` | Build + import LB system image after init |
`dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
pool range, gateway, prefix-len, nat gateway_ip from the existing
`/etc/spinifex/spinifex.toml` before teardown and replays them into
`init`. Use `-e` to override.
## Leak catalog
`roles/teardown/README.md` tracks every known state source. When teardown
misses something, add it there and in `roles/teardown/tasks/main.yml`.
+11
View File
@@ -0,0 +1,11 @@
[defaults]
inventory = inventory/localhost.yml
roles_path = roles
stdout_callback = ansible.builtin.default
host_key_checking = False
retry_files_enabled = False
interpreter_python = auto_silent
forks = 1
[callback_default]
result_format = yaml
+6
View File
@@ -0,0 +1,6 @@
---
all:
hosts:
localhost:
ansible_connection: local
ansible_python_interpreter: "{{ ansible_playbook_python }}"
+67
View File
@@ -0,0 +1,67 @@
---
- name: Dev install — build, install, init, smoketest (idempotent)
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
pre_tasks:
- name: preflight
ansible.builtin.include_role:
name: preflight
- name: health — config file present?
ansible.builtin.stat:
path: "{{ spinifex_config_file }}"
register: health_config
- name: health — spinifex.target active?
become: true
ansible.builtin.command: systemctl is-active spinifex.target
register: health_target
changed_when: false
failed_when: false
- name: health — awsgw responds?
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: aws ec2 describe-regions
register: health_gw
changed_when: false
failed_when: false
- name: health — spx binary installed?
ansible.builtin.stat:
path: /usr/local/bin/spx
register: health_spx
- name: health — set already_installed fact
ansible.builtin.set_fact:
spinifex_already_installed: >-
{{ health_config.stat.exists
and health_spx.stat.exists
and (health_target.stdout | trim) == 'active'
and health_gw.rc == 0 }}
- name: health — summary
ansible.builtin.debug:
msg: |
Health:
spx binary: {{ 'present' if health_spx.stat.exists else 'missing' }}
config: {{ 'present' if health_config.stat.exists else 'missing' }}
target: {{ health_target.stdout | trim }}
awsgw: {{ 'ok' if health_gw.rc == 0 else 'not responding' }}
Already installed: {{ spinifex_already_installed }}
{% if spinifex_already_installed %}
→ Skipping build/install/init. Smoketest only.
{% else %}
→ Running full build → install → init → smoketest.
{% endif %}
roles:
- { role: build, when: not (spinifex_already_installed | default(false)) }
- { role: install, when: not (spinifex_already_installed | default(false)) }
- { role: init, when: not (spinifex_already_installed | default(false)) }
- smoketest
@@ -0,0 +1,8 @@
---
- name: Dev preflight — validate host dependencies
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- preflight
+57
View File
@@ -0,0 +1,57 @@
---
- name: Dev reset — capture settings, wipe, rebuild, reinstall, re-init, smoketest
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
pre_tasks:
- name: capture — check existing spinifex.toml
ansible.builtin.stat:
path: "{{ spinifex_config_file }}"
register: reset_config_stat
- name: capture — slurp existing config
ansible.builtin.slurp:
src: "{{ spinifex_config_file }}"
register: reset_config_raw
when: reset_config_stat.stat.exists
- name: capture — decode config
ansible.builtin.set_fact:
reset_config_txt: "{{ reset_config_raw.content | b64decode }}"
when: reset_config_stat.stat.exists
# List-concat-and-take-first pattern: missing regex match falls back to current default.
- name: capture — extract scalars
ansible.builtin.set_fact:
spinifex_region: "{{ (reset_config_txt | regex_findall('region\\s*=\\s*\"([^\"]+)\"') + [spinifex_region])[0] }}"
spinifex_az: "{{ (reset_config_txt | regex_findall('\\baz\\s*=\\s*\"([^\"]+)\"') + [spinifex_az])[0] }}"
spinifex_external_mode: "{{ (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_mode])[0] }}"
spinifex_external_pool_start: "{{ (reset_config_txt | regex_findall('range_start\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_start])[0] }}"
spinifex_external_pool_end: "{{ (reset_config_txt | regex_findall('range_end\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_end])[0] }}"
spinifex_external_gateway: "{{ (reset_config_txt | regex_findall('gateway\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway])[0] }}"
spinifex_external_prefix_len: "{{ (reset_config_txt | regex_findall('prefix_len\\s*=\\s*([0-9]+)') + [spinifex_external_prefix_len | string])[0] }}"
spinifex_nat_gateway_ip: "{{ (reset_config_txt | regex_findall('gateway_ip\\s*=\\s*\"([^\"]+)\"') + [spinifex_nat_gateway_ip])[0] }}"
when: reset_config_stat.stat.exists
- name: capture — summary
ansible.builtin.debug:
msg: |
{% if reset_config_stat.stat.exists %}
Captured settings from {{ spinifex_config_file }}:
region={{ spinifex_region }}, az={{ spinifex_az }}
external_mode={{ spinifex_external_mode }}
pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
gateway={{ spinifex_external_gateway }}, prefix={{ spinifex_external_prefix_len }}
nat_gateway={{ spinifex_nat_gateway_ip }}
{% else %}
No existing config; using defaults from vars/defaults.yml (or -e overrides).
{% endif %}
roles:
- teardown
- build
- install
- init
- smoketest
@@ -0,0 +1,8 @@
---
- name: Dev teardown — wipe Spinifex state (no reinstall)
hosts: localhost
gather_facts: true
vars_files:
- ../vars/defaults.yml
roles:
- teardown
+6
View File
@@ -0,0 +1,6 @@
---
collections:
- name: community.general
version: ">=8.0.0"
- name: ansible.posix
version: ">=1.5.0"
@@ -0,0 +1,7 @@
---
galaxy_info:
role_name: build
description: Build spinifex + viperblock-nbdkit, stage tarball for setup.sh consumption.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -0,0 +1,48 @@
---
- name: build — spinifex (make build)
ansible.builtin.command:
cmd: make build
chdir: "{{ spinifex_root }}"
changed_when: true
- name: build — viperblock nbdkit plugin (make go_build_nbd)
ansible.builtin.command:
cmd: make go_build_nbd
chdir: "{{ mulga_root }}/viperblock"
changed_when: true
- name: build — verify build artifacts present
ansible.builtin.stat:
path: "{{ item }}"
register: build_artifacts
loop:
- "{{ spinifex_root }}/bin/spx"
- "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
- name: build — assert artifacts exist
ansible.builtin.assert:
that: item.stat.exists
fail_msg: "Missing build artifact: {{ item.item }}"
loop: "{{ build_artifacts.results }}"
loop_control:
label: "{{ item.item }}"
- name: build — stage tarball contents (mirrors dev-install.sh)
ansible.builtin.shell: |
set -e
STAGING=$(mktemp -d)
trap 'rm -rf "$STAGING"' EXIT
cp "{{ spinifex_root }}/bin/spx" "$STAGING/spx"
cp "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so" "$STAGING/"
cp "{{ spinifex_root }}/scripts/setup-ovn.sh" "$STAGING/"
mkdir -p "$STAGING/systemd"
cp "{{ spinifex_root }}/build/systemd/"* "$STAGING/systemd/"
mkdir -p "$STAGING/scripts"
cp "{{ spinifex_root }}/build/scripts/"* "$STAGING/scripts/"
cp "{{ spinifex_root }}/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
tar czf "{{ build_tarball_path }}" -C "$STAGING" .
changed_when: true
- name: build — summary
ansible.builtin.debug:
msg: "Built spx + nbdkit plugin, tarball at {{ build_tarball_path }}"
+7
View File
@@ -0,0 +1,7 @@
---
galaxy_info:
role_name: init
description: Run spx admin init, install CA trust, start services, import LB image.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
+98
View File
@@ -0,0 +1,98 @@
---
- name: init — base admin-init flags
ansible.builtin.set_fact:
init_base_flags: >-
--force
--region {{ spinifex_region }}
--az {{ spinifex_az }}
--node {{ spinifex_node_name }}
--nodes {{ spinifex_node_count }}
- name: init — pool-mode external flags
ansible.builtin.set_fact:
init_external_flags: >-
--external-mode=pool
--external-pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
--external-gateway={{ spinifex_external_gateway }}
--external-prefix-len={{ spinifex_external_prefix_len }}
when:
- spinifex_external_mode == 'pool'
- spinifex_external_pool_start | length > 0
- spinifex_external_pool_end | length > 0
- name: init — NAT-mode external flags
ansible.builtin.set_fact:
init_external_flags: >-
--external-mode=nat
--gateway-ip={{ spinifex_nat_gateway_ip }}
--external-gateway={{ spinifex_external_gateway }}
--external-prefix-len={{ spinifex_external_prefix_len }}
when:
- spinifex_external_mode == 'nat'
- spinifex_nat_gateway_ip | length > 0
- name: init — assemble full flag string
ansible.builtin.set_fact:
init_all_flags: "{{ init_base_flags }} {{ init_external_flags | default('') }}"
- name: init — run spx admin init
become: true
ansible.builtin.command: "spx admin init {{ init_all_flags }}"
changed_when: true
- name: init — copy CA to system trust
become: true
ansible.builtin.copy:
src: "{{ spinifex_ca_src }}"
dest: "{{ spinifex_ca_trust_path }}"
mode: "0644"
remote_src: true
- name: init — rebuild CA bundle (not --fresh — preserve other CAs)
become: true
ansible.builtin.command: update-ca-certificates
changed_when: true
- name: init — start spinifex.target
become: true
ansible.builtin.systemd:
name: spinifex.target
state: started
# systemctl start returns as soon as units activate; awsgw still wires up
# to NATS + daemons after that. Bash start-dev.sh has internal wait loops;
# systemd path does not. Poll a benign call until gateway is fully ready.
- name: init — wait for AWS gateway readiness
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: aws ec2 describe-regions
register: init_gw_ready
until: init_gw_ready.rc == 0
retries: 30
delay: 2
changed_when: false
- name: init — build LB agent
ansible.builtin.command:
cmd: make build-lb-agent
chdir: "{{ spinifex_root }}"
changed_when: true
when: spinifex_build_lb_image | bool
- name: init — import LB system image
ansible.builtin.command:
cmd: "{{ spinifex_root }}/scripts/build-system-image.sh {{ spinifex_lb_image_conf }} --import --quiet"
changed_when: true
when: spinifex_build_lb_image | bool
- name: init — summary
ansible.builtin.debug:
msg: |
Init complete:
region={{ spinifex_region }}, az={{ spinifex_az }}, node={{ spinifex_node_name }}
external_mode={{ spinifex_external_mode }}
spinifex.target: started
CA: {{ spinifex_ca_trust_path }}
LB image: {{ 'imported' if spinifex_build_lb_image else 'skipped' }}
@@ -0,0 +1,7 @@
---
galaxy_info:
role_name: install
description: Invoke setup.sh and setup-ovn.sh to install Spinifex into production paths.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -0,0 +1,65 @@
---
- name: install — assert tarball exists
ansible.builtin.stat:
path: "{{ build_tarball_path }}"
register: install_tarball
- name: install — fail if tarball missing
ansible.builtin.assert:
that: install_tarball.stat.exists
fail_msg: "Tarball {{ build_tarball_path }} missing. Run the build role first."
- name: install — stop spinifex.target before reinstall
become: true
ansible.builtin.systemd:
name: spinifex.target
state: stopped
failed_when: false
- name: install — reset failed state (spinifex-* scope)
become: true
ansible.builtin.command: systemctl reset-failed 'spinifex-*'
changed_when: false
failed_when: false
- name: install — scrub stale pid/log files from prior dev-mode installs
become: true
ansible.builtin.shell: |
for dir in {{ spinifex_data_dir }} {{ spinifex_log_dir }} {{ spinifex_etc_dir }}; do
if [ -d "$dir" ]; then
find "$dir" -name '*.pid' -delete 2>/dev/null || true
find "$dir" -name '*.log' -delete 2>/dev/null || true
find "$dir" -name '*.log.*' -delete 2>/dev/null || true
fi
done
changed_when: false
- name: install — remove legacy /var/lib/spinifex/config
become: true
ansible.builtin.file:
path: "{{ spinifex_data_dir }}/config"
state: absent
- name: install — run setup.sh (creates users, dirs, systemd units)
become: true
ansible.builtin.command:
cmd: "bash {{ spinifex_root }}/scripts/setup.sh"
environment:
INSTALL_SPINIFEX_TARBALL: "{{ build_tarball_path }}"
INSTALL_SPINIFEX_SKIP_NEWGRP: "1"
changed_when: true
- name: install — cleanup tarball
ansible.builtin.file:
path: "{{ build_tarball_path }}"
state: absent
- name: install — run setup-ovn.sh --management
become: true
ansible.builtin.command:
cmd: /usr/local/share/spinifex/setup-ovn.sh --management
changed_when: true
- name: install — summary
ansible.builtin.debug:
msg: "Install complete — setup.sh + setup-ovn.sh ran. Services NOT yet started (init role handles that)."
@@ -0,0 +1,7 @@
---
galaxy_info:
role_name: preflight
description: Validate dev-host dependencies for Spinifex (parity with dev-setup.sh).
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -0,0 +1,98 @@
---
- name: Check required commands
ansible.builtin.command: "which {{ item }}"
register: preflight_required
loop: "{{ spinifex_required_commands }}"
changed_when: false
failed_when: preflight_required.rc != 0
loop_control:
label: "{{ item }}"
- name: Check optional commands
ansible.builtin.command: "which {{ item }}"
register: preflight_optional
loop: "{{ spinifex_optional_commands }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item }}"
- name: Warn on missing optional commands
ansible.builtin.debug:
msg: "WARN: optional command '{{ item.item }}' not found"
loop: "{{ preflight_optional.results }}"
when: item.rc != 0
loop_control:
label: "{{ item.item }}"
- name: Check OVN/OVS commands
ansible.builtin.command: "which {{ item }}"
register: preflight_ovn
loop: "{{ spinifex_ovn_commands }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item }}"
- name: Warn on missing OVN/OVS tools
ansible.builtin.debug:
msg: "WARN: OVN/OVS tool '{{ item.item }}' not found (required for VPC networking). Install: sudo apt-get install ovn-central ovn-host openvswitch-switch"
loop: "{{ preflight_ovn.results }}"
when: item.rc != 0
loop_control:
label: "{{ item.item }}"
- name: Record OVN tool availability
ansible.builtin.set_fact:
preflight_ovs_present: "{{ (preflight_ovn.results | selectattr('item','eq','ovs-vsctl') | selectattr('rc','eq',0) | list | length) > 0 }}"
- name: Check br-int bridge
ansible.builtin.command: ovs-vsctl br-exists br-int
register: preflight_brint
changed_when: false
failed_when: false
when: preflight_ovs_present
- name: Warn if br-int missing
ansible.builtin.debug:
msg: "WARN: br-int missing. Run scripts/setup-ovn.sh --management to initialise OVN networking."
when:
- preflight_ovs_present
- preflight_brint.rc | default(0) != 0
- name: Check sibling component directories
ansible.builtin.stat:
path: "{{ playbook_dir }}/../../../../{{ item }}/go.mod"
register: preflight_components
loop: "{{ spinifex_components }}"
loop_control:
label: "{{ item }}"
- name: Warn on missing component checkouts
ansible.builtin.debug:
msg: "WARN: component '{{ item.item }}' not found at expected sibling path (no go.mod). Run scripts/clone-deps.sh."
loop: "{{ preflight_components.results }}"
when: not item.stat.exists
loop_control:
label: "{{ item.item }}"
- name: Check port availability
ansible.builtin.shell: "ss -tlnH 'sport = :{{ item.port }}' | head -1"
register: preflight_ports
loop: "{{ spinifex_ports }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item.service }} ({{ item.port }})"
- name: Warn on occupied ports
ansible.builtin.debug:
msg: "WARN: port {{ item.item.port }} occupied (needed for {{ item.item.service }})"
loop: "{{ preflight_ports.results }}"
when: item.stdout | length > 0
loop_control:
label: "{{ item.item.service }}"
- name: Preflight summary
ansible.builtin.debug:
msg: "Preflight passed. Required: {{ spinifex_required_commands | join(', ') }}. Warnings above (if any) are non-fatal."
@@ -0,0 +1,7 @@
---
galaxy_info:
role_name: smoketest
description: Import SSH key, import Ubuntu AMI, launch one smoke-test instance.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -0,0 +1,232 @@
---
- name: smoketest — resolve invoking user's home
ansible.builtin.getent:
database: passwd
key: "{{ spinifex_invoking_user }}"
- name: smoketest — set user home fact
ansible.builtin.set_fact:
smoketest_user_home: "{{ ansible_facts['getent_passwd'][spinifex_invoking_user][4] }}"
- name: smoketest — generate SSH key if absent
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.command:
cmd: ssh-keygen -t ed25519 -f {{ smoketest_user_home }}/.ssh/spinifex-key -N ""
creates: "{{ smoketest_user_home }}/.ssh/spinifex-key"
- name: smoketest — import SSH key to spinifex
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: >-
aws ec2 import-key-pair
--key-name spinifex-key
--public-key-material fileb://{{ smoketest_user_home }}/.ssh/spinifex-key.pub
register: smoketest_keypair
until: >-
smoketest_keypair.rc == 0 or
'InvalidKeyPair.Duplicate' in (smoketest_keypair.stderr | default(''))
retries: 10
delay: 3
changed_when: "'InvalidKeyPair.Duplicate' not in (smoketest_keypair.stderr | default(''))"
failed_when:
- smoketest_keypair.rc != 0
- "'InvalidKeyPair.Duplicate' not in (smoketest_keypair.stderr | default(''))"
- name: smoketest — detect architecture
ansible.builtin.set_fact:
smoketest_arch: "{{ 'arm64' if ansible_facts['architecture'] in ['aarch64','arm64'] else 'x86_64' }}"
- name: smoketest — resolve local image path
ansible.builtin.set_fact:
smoketest_local_image_path: "{{ smoketest_local_image | expanduser }}"
vars:
smoketest_local_image: "{{ spinifex_smoketest_local_image | replace('~', smoketest_user_home) }}"
- name: smoketest — check for local image
ansible.builtin.stat:
path: "{{ smoketest_local_image_path }}"
register: smoketest_local_img
- name: smoketest — import AMI from local file
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: >-
spx admin images import
--file {{ smoketest_local_image_path }}
--distro {{ spinifex_smoketest_distro }}
--version {{ spinifex_smoketest_version }}
--arch {{ smoketest_arch }}
register: smoketest_import_local
until: >-
smoketest_import_local.rc == 0 or
'already exists' in (smoketest_import_local.stderr | default(''))
retries: 5
delay: 3
changed_when: smoketest_import_local.rc == 0
failed_when:
- smoketest_import_local.rc != 0
- "'already exists' not in (smoketest_import_local.stderr | default(''))"
when: smoketest_local_img.stat.exists
- name: smoketest — import AMI by name (fallback)
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: >-
spx admin images import
--name {{ spinifex_smoketest_distro }}-{{ spinifex_smoketest_version }}-{{ smoketest_arch }}
register: smoketest_import_remote
until: >-
smoketest_import_remote.rc == 0 or
'already exists' in (smoketest_import_remote.stderr | default(''))
retries: 5
delay: 3
changed_when: smoketest_import_remote.rc == 0
failed_when:
- smoketest_import_remote.rc != 0
- "'already exists' not in (smoketest_import_remote.stderr | default(''))"
when: not smoketest_local_img.stat.exists
- name: smoketest — detect CPU vendor for instance type choice
ansible.builtin.command: grep -q AuthenticAMD /proc/cpuinfo
register: smoketest_amd
changed_when: false
failed_when: false
- name: smoketest — pick instance type
ansible.builtin.set_fact:
smoketest_instance_type: "{{ 't3a.small' if smoketest_amd.rc == 0 else 't3.small' }}"
- name: smoketest — find AMI ID
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: aws ec2 describe-images --query "Images[0].ImageId" --output text
register: smoketest_ami
until: smoketest_ami.rc == 0
retries: 10
delay: 3
changed_when: false
- name: smoketest — find public subnet (best-effort)
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: >-
aws ec2 describe-subnets
--filters Name=map-public-ip-on-launch,Values=true
--query Subnets[0].SubnetId
--output text
register: smoketest_subnet_public
changed_when: false
failed_when: false
- name: smoketest — fallback to first subnet (best-effort)
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: aws ec2 describe-subnets --query "Subnets[0].SubnetId" --output text
register: smoketest_subnet_any
changed_when: false
failed_when: false
when:
- smoketest_subnet_public.rc != 0 or smoketest_subnet_public.stdout in ['', 'None']
- name: smoketest — resolve subnet id (empty string if none)
ansible.builtin.set_fact:
smoketest_subnet_id: >-
{%- if smoketest_subnet_public.rc == 0 and smoketest_subnet_public.stdout not in ['', 'None'] -%}
{{ smoketest_subnet_public.stdout }}
{%- elif (smoketest_subnet_any.rc | default(1)) == 0 and (smoketest_subnet_any.stdout | default('')) not in ['', 'None'] -%}
{{ smoketest_subnet_any.stdout }}
{%- else -%}{%- endif -%}
- name: smoketest — skip launch if no subnet (fresh install with no user VPC)
ansible.builtin.debug:
msg: |
No subnet found — skipping instance launch.
Services are up, CA is trusted, keypair imported, AMI imported.
Create a VPC+subnet via tofu examples (or `aws ec2 create-vpc` /
`create-subnet`) then re-run if you want a live instance.
when: smoketest_subnet_id | trim == ''
- name: smoketest — launch instance
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: >-
aws ec2 run-instances
--image-id {{ smoketest_ami.stdout }}
--instance-type {{ smoketest_instance_type }}
--key-name spinifex-key
--subnet-id {{ smoketest_subnet_id | trim }}
--count 1
--query Instances[0].InstanceId
--output text
register: smoketest_instance
until: smoketest_instance.rc == 0
retries: 5
delay: 3
changed_when: true
when: smoketest_subnet_id | trim != ''
- name: smoketest — wait for instance running (proves launch path works)
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: >-
aws ec2 describe-instances
--instance-ids {{ smoketest_instance.stdout }}
--query Reservations[0].Instances[0].State.Name
--output text
register: smoketest_instance_state
until: smoketest_instance_state.stdout | trim == 'running'
retries: 30
delay: 2
changed_when: false
failed_when: false
when:
- smoketest_subnet_id | trim != ''
- smoketest_instance.stdout is defined
- smoketest_instance.stdout | trim != ''
- name: smoketest — terminate instance (cleanup)
become: true
become_user: "{{ spinifex_invoking_user }}"
environment:
AWS_PROFILE: spinifex
ansible.builtin.command: >-
aws ec2 terminate-instances
--instance-ids {{ smoketest_instance.stdout }}
register: smoketest_terminate
until: smoketest_terminate.rc == 0
retries: 5
delay: 3
changed_when: true
failed_when: false
when:
- smoketest_subnet_id | trim != ''
- smoketest_instance.stdout is defined
- smoketest_instance.stdout | trim != ''
- name: smoketest — summary
ansible.builtin.debug:
msg: |
Smoketest complete:
AMI: {{ smoketest_ami.stdout }}
Subnet: {{ smoketest_subnet_id | trim | default('(none — launch skipped)', true) }}
Instance type: {{ smoketest_instance_type }}
Instance ID: {{ smoketest_instance.stdout | default('(skipped)', true) }}
Final state: {{ smoketest_instance_state.stdout | default('(skipped)', true) | trim }}
Terminated: {{ 'yes' if smoketest_terminate.rc | default(1) == 0 else 'no (launch skipped or terminate failed)' }}
+54
View File
@@ -0,0 +1,54 @@
# teardown — Leak Catalog
Every task in this role corresponds to a known state source that survived a
previous reset. When the operator reports "reset didn't clean X", add a new
task and a new row here citing the symptom.
## Initial catalog (from reset-dev-env.sh + reset-dev-env.md target)
| State source | Cleaned by | Symptom if leaked |
|---|---|---|
| `spinifex.target` active | `systemctl stop` | Services keep running, file locks held |
| `spinifex-*` units in failed state | `systemctl reset-failed 'spinifex-*'` | `systemctl status` shows red units, next start refuses |
| Spinifex processes (spx, nats, predastore, etc.) | `pkill` pattern + wait loop | Stale listeners on 4222/8443/9999, start fails with "address in use" |
| QEMU VMs | `pkill qemu-system-*` + wait loop | Zombie VMs hold tap interfaces and viperblock state |
| `/etc/spinifex/` | `file: state=absent` | Stale TOML, stale CA, service users can't read fresh config |
| `/var/lib/spinifex/` | `file: state=absent` | Stale WAL, stale volumes, viperblock mounts fail |
| `/var/log/spinifex/` | `file: state=absent` | Logrotate confusion, permission drift after user changes |
| `/run/spinifex/` | `file: state=absent` | Stale sockets, pidfiles from prior systemd run |
| `$HOME/spinifex/` (legacy) | `file: state=absent` | Old dev-mode state shadowing new install |
| OVS bridges (all) | `ovs-vsctl del-br` loop | `br-int` populated with stale ports → `ovn-controller` commit loop |
| OVS `external_ids` | `ovs-vsctl clear Open_vSwitch .` | Stale chassis entries in SBDB |
| `/var/lib/ovn/ovnnb_db.db`, `ovnsb_db.db` | `file: state=absent` | "OVNSB commit failed, force recompute" loop after reset |
| `veth-wan-br` / `veth-wan-ovs` | `ip link delete` | Pair left dangling after setup-ovn.sh re-run |
| `spx-ext-*` macvlan interfaces | `ip link delete` loop | Stale macvlan on WAN NIC, packet duplication |
| Spinifex CA in trust store | remove + `update-ca-certificates` | TLS clients trust old CA, new cert fails verification |
## Not cleaned by default (opt-in)
| State source | Flag | Why opt-in |
|---|---|---|
| `~/.ssh/spinifex-key*` | `-e spinifex_wipe_ssh_keys=true` | Operator may want key to persist across resets |
| `~/.aws/credentials` spinifex profile | `-e spinifex_wipe_aws_creds=true` | `spx admin init --force` rewrites this block anyway; wipe only on demand |
## Known traps
- **`pgrep -f` self-match.** A literal pattern that matches the pattern
string itself causes `pgrep -f` to match our own shell. Mitigation:
bracket trick (`[b]in/spx` matches `bin/spx` but not `[b]in/spx`).
- **Log followers / ansible wrapper false positives.** `journalctl -f
_SYSTEMD_UNIT=spinifex-*` and ansible's own module shell matches the
service-name patterns in their cmdlines. Mitigation:
`spinifex_process_exclude` filter drops `journalctl|ansible|/bin/sh|pgrep|pkill|tail -[fF]`.
- **SIGTERM survivors.** `spx service spinifex start` (and potentially
other service-supervisor processes) ignore or delay SIGTERM. Mitigation:
kill task escalates SIGTERM → SIGKILL after 5s.
## Adding a new entry
1. Add a task to `tasks/main.yml` in the right section (services / state-dirs /
ovn / ovs / netlink / trust / user-owned).
2. Prefix the task `name:` with the state class (e.g. `netlink:`, `ovn:`).
3. Add a row to the catalog above — symptom is what the operator described.
4. If the leak class is general (e.g. "any interface matching spx-*"),
generalise the task; do not pattern-match on a single instance name.
@@ -0,0 +1,7 @@
---
galaxy_info:
role_name: teardown
description: Wipe Spinifex dev state. Central leak-enumeration surface — every state location cleaned here is a known leak source. See README.md for catalog.
license: BSD-3-Clause
min_ansible_version: "2.14"
dependencies: []
@@ -0,0 +1,242 @@
---
- name: guard — detect node count in spinifex.toml
ansible.builtin.shell: |
set -e
if [ -f {{ spinifex_config_file | quote }} ]; then
grep -cE '^\[nodes\.[^.]+\]' {{ spinifex_config_file | quote }} || echo 0
else
echo 0
fi
register: teardown_node_count
changed_when: false
- name: guard — refuse to run on multi-node cluster
ansible.builtin.assert:
that: teardown_node_count.stdout | int <= 1
fail_msg: >-
Multi-node cluster detected ({{ teardown_node_count.stdout }} nodes in
{{ spinifex_config_file }}). Teardown supports single-node only;
use 'spx admin cluster shutdown' per node.
- name: services — stop spinifex.target
become: true
ansible.builtin.systemd:
name: spinifex.target
state: stopped
failed_when: false
- name: services — reset failed state (spinifex-* scope only)
become: true
ansible.builtin.command: systemctl reset-failed 'spinifex-*'
changed_when: false
failed_when: false
- name: processes — kill spinifex processes (SIGTERM, escalate to SIGKILL after 5s)
become: true
ansible.builtin.shell: |
pids=$(pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" \
| awk '{print $1}')
if [ -n "$pids" ]; then
kill $pids 2>/dev/null || true
sleep 5
pids=$(pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" \
| awk '{print $1}')
if [ -n "$pids" ]; then
kill -9 $pids 2>/dev/null || true
fi
fi
changed_when: false
failed_when: false
- name: processes — wait for exit (up to {{ teardown_process_wait_seconds }}s)
become: true
ansible.builtin.shell: |
for i in $(seq 1 {{ teardown_process_wait_seconds }}); do
pids=$(pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" \
| awk '{print $1}')
if [ -z "$pids" ]; then
exit 0
fi
sleep 1
done
echo "still running after {{ teardown_process_wait_seconds }}s:" >&2
pgrep -af "{{ spinifex_process_pattern }}" \
| grep -vE "{{ spinifex_process_exclude }}" >&2 || true
exit 1
changed_when: false
- name: qemu — kill lingering VMs
become: true
ansible.builtin.command: "pkill -x {{ item }}"
loop: "{{ spinifex_qemu_processes }}"
changed_when: false
failed_when: false
- name: qemu — wait for exit (up to {{ teardown_process_wait_seconds }}s)
become: true
ansible.builtin.shell: |
set -e
for i in $(seq 1 {{ teardown_process_wait_seconds }}); do
alive=0
for proc in {{ spinifex_qemu_processes | join(' ') }}; do
if pgrep -x "$proc" >/dev/null 2>&1; then alive=1; break; fi
done
if [ "$alive" = "0" ]; then exit 0; fi
sleep 1
done
echo "QEMU still running after {{ teardown_process_wait_seconds }}s" >&2
exit 1
changed_when: false
- name: state-dirs — remove production dirs
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ spinifex_etc_dir }}"
- "{{ spinifex_data_dir }}"
- "{{ spinifex_log_dir }}"
- "{{ spinifex_run_dir }}"
- name: state-dirs — remove legacy $HOME/spinifex (opt-in)
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.file:
path: "{{ spinifex_legacy_home_dir }}"
state: absent
when: spinifex_wipe_legacy_home | bool
- name: ovs — detect tooling
ansible.builtin.command: which ovs-vsctl
register: teardown_ovs_which
changed_when: false
failed_when: false
- name: ovs — ensure openvswitch-switch is running for cleanup
become: true
ansible.builtin.systemd:
name: openvswitch-switch
state: started
when: teardown_ovs_which.rc == 0
failed_when: false
- name: ovs — list bridges
become: true
ansible.builtin.command: ovs-vsctl list-br
register: teardown_ovs_brs
changed_when: false
failed_when: false
when: teardown_ovs_which.rc == 0
- name: ovs — delete all bridges
become: true
ansible.builtin.command: "ovs-vsctl --if-exists del-br {{ item }}"
loop: "{{ teardown_ovs_brs.stdout_lines | default([]) }}"
changed_when: true
when: teardown_ovs_which.rc == 0
- name: ovs — clear external_ids
become: true
ansible.builtin.command: ovs-vsctl --if-exists clear Open_vSwitch . external_ids
changed_when: true
failed_when: false
when: teardown_ovs_which.rc == 0
- name: ovs — stop openvswitch-switch after cleanup
become: true
ansible.builtin.systemd:
name: openvswitch-switch
state: stopped
when: teardown_ovs_which.rc == 0
failed_when: false
- name: ovn — stop ovn-central
become: true
ansible.builtin.systemd:
name: ovn-central
state: stopped
failed_when: false
- name: ovn — stop ovn-controller
become: true
ansible.builtin.systemd:
name: ovn-controller
state: stopped
failed_when: false
- name: ovn — delete NB/SB DB files
become: true
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ ovn_nb_db }}"
- "{{ ovn_sb_db }}"
- name: netlink — enumerate spinifex-managed veth/tap
ansible.builtin.shell: |
ip -o link show | awk -F': ' '{print $2}' | cut -d'@' -f1 | \
grep -E '^(veth-wan-br|veth-wan-ovs|spx-ext-|spx-tap-|spx-)' || true
register: teardown_links
changed_when: false
- name: netlink — delete enumerated interfaces
become: true
ansible.builtin.command: "ip link delete {{ item }}"
loop: "{{ teardown_links.stdout_lines }}"
changed_when: true
failed_when: false
- name: trust — remove spinifex CA from system store
become: true
ansible.builtin.file:
path: "{{ spinifex_ca_trust_path }}"
state: absent
register: teardown_ca_removed
- name: trust — rebuild CA bundle (without --fresh, preserve other CAs)
become: true
ansible.builtin.command: update-ca-certificates
changed_when: true
when: teardown_ca_removed.changed
- name: user-owned — remove spinifex profile from AWS credentials (opt-in)
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.blockinfile:
path: "~/.aws/credentials"
marker: "# {mark} SPINIFEX MANAGED BLOCK"
block: ""
state: absent
when: spinifex_wipe_aws_creds | bool
failed_when: false
- name: user-owned — remove spinifex SSH key (opt-in)
become: true
become_user: "{{ spinifex_invoking_user }}"
ansible.builtin.file:
path: "~/.ssh/{{ item }}"
state: absent
loop:
- spinifex-key
- spinifex-key.pub
when: spinifex_wipe_ssh_keys | bool
- name: summary
ansible.builtin.debug:
msg: |
Teardown complete. Catalog: roles/teardown/README.md
systemd : spinifex.target stopped, spinifex-* failed cleared
procs : {{ spinifex_process_pattern }}
qemu : {{ spinifex_qemu_processes | join(', ') }}
dirs : {{ spinifex_etc_dir }}, {{ spinifex_data_dir }}, {{ spinifex_log_dir }}, {{ spinifex_run_dir }}{{ ', ' ~ spinifex_legacy_home_dir if spinifex_wipe_legacy_home else '' }}
ovs : all bridges deleted, external_ids cleared
ovn : NB/SB DB files removed
netlink : {{ teardown_links.stdout_lines | length }} interface(s) removed
trust : {{ spinifex_ca_trust_path }} removed{% if spinifex_wipe_aws_creds %}
aws : credentials block removed{% endif %}{% if spinifex_wipe_ssh_keys %}
ssh : spinifex-key pair removed{% endif %}
+90
View File
@@ -0,0 +1,90 @@
---
# Production layout (per spinifex/scripts/setup.sh).
spinifex_etc_dir: /etc/spinifex
spinifex_data_dir: /var/lib/spinifex
spinifex_log_dir: /var/log/spinifex
spinifex_run_dir: /run/spinifex
spinifex_config_file: "{{ spinifex_etc_dir }}/spinifex.toml"
spinifex_ca_trust_path: /usr/local/share/ca-certificates/spinifex-ca.crt
spinifex_ca_src: "{{ spinifex_etc_dir }}/ca.pem"
# Legacy $HOME layout (still present on older dev boxes).
spinifex_legacy_home_dir: "{{ lookup('env','HOME') }}/spinifex"
# OVN state.
ovn_db_dir: /var/lib/ovn
ovn_nb_db: "{{ ovn_db_dir }}/ovnnb_db.db"
ovn_sb_db: "{{ ovn_db_dir }}/ovnsb_db.db"
# Region / init.
spinifex_region: ap-southeast-2
spinifex_az: "{{ spinifex_region }}a"
spinifex_node_name: node1
spinifex_node_count: 1
# External network.
spinifex_external_mode: pool
spinifex_external_pool_start: ""
spinifex_external_pool_end: ""
spinifex_external_gateway: ""
spinifex_external_prefix_len: ""
spinifex_nat_gateway_ip: ""
# Preflight checks.
spinifex_ports:
- { port: 4222, service: NATS }
- { port: 8443, service: Predastore }
- { port: 9999, service: "Spinifex Gateway" }
spinifex_required_commands:
- go
- make
- nbdkit
spinifex_optional_commands:
- air
spinifex_ovn_commands:
- ovs-vsctl
- ovn-controller
- ovn-nbctl
spinifex_components:
- viperblock
- predastore
# Process patterns for teardown.
# Bracket trick ([b]in/spx etc.) prevents the pattern literal from matching
# our own pgrep/shell cmdline. Exclude filter drops benign log followers and
# ansible's own wrappers that also mention these service names.
spinifex_process_pattern: '([b]in/spx|spinifex-[u]i|nats-[s]erver|[p]redastore|[v]iperblock|[v]pcd)'
spinifex_process_exclude: '(journalctl|ansible|/bin/sh|pgrep|pkill|tail -[fF])'
spinifex_qemu_processes:
- qemu-system-x86_64
- qemu-system-aarch64
# Invoking user (when run under sudo).
spinifex_invoking_user: "{{ lookup('env','SUDO_USER') | default(ansible_facts['user_id'], true) }}"
# Opt-in wipes.
spinifex_wipe_ssh_keys: false
spinifex_wipe_aws_creds: false
spinifex_wipe_legacy_home: true
# Timeouts.
teardown_process_wait_seconds: 30
# Path resolution — playbook_dir is scripts/ansible/playbooks/ at runtime.
spinifex_root: "{{ lookup('pipe', 'readlink -f ' ~ playbook_dir ~ '/../../..') }}"
mulga_root: "{{ lookup('pipe', 'readlink -f ' ~ playbook_dir ~ '/../../../..') }}"
# Build output.
build_tarball_path: /tmp/spinifex-local.tar.gz
# LB system image (built + imported at end of init).
spinifex_build_lb_image: true
spinifex_lb_image_conf: "{{ spinifex_root }}/scripts/images/lb.conf"
# Smoketest image — parameterised so tofu examples can share the same base.
spinifex_smoketest_distro: debian
spinifex_smoketest_version: "12"
# Local image file path (if present, imported directly; otherwise fall back to --name).
spinifex_smoketest_local_image: "~/images/{{ spinifex_smoketest_distro }}-{{ spinifex_smoketest_version }}.img"