mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 03:11:46 +00:00
Josh Siviour
ae6fcfe7a1
* chore(ansible): add dev-status playbook (read-only health doctor)
New `dev-status` role + playbook + Makefile target. Read-only snapshot of
spinifex.target, failed units, port listeners, OVN/OVS state, gateway
round-trip, AMI/keypair/volume/instance counts, and build-vs-install drift
for spx + nbdkit plugin + microvm artifacts. Pairs with dev-preflight as
the post-install / running-system diagnostic.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.1.
* chore(ansible): add dev-logs playbook (bug-report bundle)
New `logs` role + playbook + Makefile target. Captures journalctl for
spinifex-* / ovn-* / openvswitch-switch / qemu / nbdkit, copies
/var/log/spinifex/, snapshots systemctl status + networking (ip/ss/ovs/
ovn show) + spx/plugin/microvm version manifest + dmesg, then bundles
into /tmp/spinifex-logs-<host>-<ts>.tar.gz owned by the invoking user.
Tail mode deliberately omitted — devs should `journalctl -u 'spinifex-*' -f`
directly; ansible captures stdout and is the wrong tool for follow.
Overrides: -e logs_since=10min, -e logs_output_dir=/tmp,
-e logs_include_dmesg=false.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.2.
* chore(ansible): add dev-snapshot / dev-restore (state checkpoint)
New snapshot + restore roles + playbooks + Makefile targets.
dev-snapshot quiesces the full spinifex+OVN+OVS stack in dependency
order (spinifex.target → ovn-controller → ovn-central →
openvswitch-switch), kills lingering QEMU VMs, then tars /etc/spinifex,
/var/lib/spinifex (viperblock volumes included), /var/lib/ovn (NB+SB
DBs), and /etc/openvswitch/conf.db into /var/lib/spinifex-snapshots/
<name>.tar.gz with a sibling .manifest, restarts services in reverse,
and waits for the AWS gateway to come back.
dev-restore inverses: stop services, wipe the same path set, untar the
named bundle, copy /etc/spinifex/ca.pem back into the system trust
store and rebuild the CA bundle (the snapshot doesn't include
/usr/local/share/ca-certificates/spinifex-ca.crt), daemon-reload,
restart in reverse order.
Both refuse to run without -e snapshot_name=<name>, validate the name
matches [A-Za-z0-9._-]+, and produce honest summaries when the gateway
fails to come back (does not mask other ongoing issues).
Tested on dev box: snapshot/restore of 2.4 GiB state (large viperblock
volumes) completes cleanly with no data drift; services all return to
active.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.3.
* fix(ansible): wrap aws/spx calls in sg spinifex to fix group inheritance
Adds scripts/ansible/scripts/aws-as-spinifex.sh and spx-as-spinifex.sh
helpers that re-enter the command under `sg spinifex -c '...'` with
AWS_PROFILE=spinifex preset. Swaps every AWS / spx admin task across
the status, init, deploy, smoketest, snapshot, and restore roles to
use these via spinifex_aws_cli / spinifex_spx_cli vars.
Root cause (mulga-siv-92): ansible's `become_user` is a no-op when the
target user equals the calling user, so sudo never runs and
supplementary groups are inherited from the operator's invoking shell
rather than re-initialised via initgroups(3). On shells that predate
setup.sh's spinifex group add, the spinifex group is missing and
/etc/spinifex/ca.pem (drwxr-x--- root:spinifex, the AWS CLI's
ca_bundle) is unreadable — aws cli fails with SSL Errno 13. The
helpers re-enter the command in a shell that explicitly assumes the
spinifex group via sg(1), bypassing the issue regardless of how
ansible-playbook is invoked.
Side benefit: drops 50 lines of repeated `become: true / become_user /
environment: AWS_PROFILE` boilerplate across the roles.
Closes mulga-siv-92.
* chore(ansible): add dev-version playbook (build vs install drift)
New `version` role + playbook + Makefile target. Compares sha256 of
built artifacts (bin/spx, viperblock/lib/nbdkit-viperblock-plugin.so,
build/microvm/{vmlinuz,initramfs.cpio.gz}) against their installed
counterparts (/usr/local/bin/spx, plugindir, /usr/share/spinifex/
microvm/*). Prints per-artifact drift verdict plus sub-repo HEAD
short-revs (spinifex/viperblock/predastore) for context. Closes with
either "DRIFT detected — run: make ansible-dev-deploy" or "No drift".
Pure read-only. Faster than dev-status (no AWS/OVN/port probing).
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.9.
* chore(ansible): add missing roles/deploy/meta/main.yml
All other roles have meta/main.yml; deploy was the lone outlier.
Consistency fix. Closes mulga-siv-86.10.
* chore(ansible): add dev-vpc playbook (idempotent dev VPC fixture)
New `vpc` role + playbook + Makefile target. Idempotent
present/absent dispatch creates a deterministic spinifex-dev-vpc with
public subnet (10.99.1.0/24, map-public-ip-on-launch), internet
gateway, route table, and default route. Hooks into smoketest's launch
path — smoketest's existing
Name=map-public-ip-on-launch,Values=true filter now picks it up
instead of falling through to the "no subnet → skip launch" branch.
Idempotency: each step describe-then-create, looking up by Name tag
where possible. Re-running on a converged VPC is changed=0, skipped=5
(the create tasks).
Workaround for mulga-siv-94 (gateway silently drops
--tag-specifications and no-ops create-tags for route tables): role
falls back to "RT associated with our subnet" lookup so tag absence
doesn't break idempotency. create-tags task is kept as a no-op-now
forward-looking step for when the gateway gap is closed.
Teardown (-e dev_vpc_state=absent) enumerates ALL non-main route
tables in the VPC (not just the tagged one), so orphans from prior
runs that landed before the fallback was added get cleaned up too.
Tested end-to-end against josh-banksia dev gateway: create / re-run
(idempotent) / teardown all converge cleanly.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.4.
Ansible Dev Lifecycle (parallel to bash scripts)
Experimental Ansible-based lifecycle for a single-node Spinifex dev host.
Lives alongside the existing scripts/dev-*.sh and scripts/reset-dev-env.sh;
neither set is removed. Decide which to keep after parity testing.
Plan: docs/development/improvements/ansible-dev-lifecycle.md (mulga top-level).
Bead: mulga-siv-9.
Prerequisites
Venv lives at sub-repo root (spinifex/.venv/) — one venv per sub-repo,
shared by any future python tooling in spinifex. Bootstrap from spinifex/:
sudo apt install python3.13-venv
python3 -m venv .venv
source .venv/bin/activate
pip install 'ansible>=9'
ansible-galaxy collection install -r scripts/ansible/requirements.yml
Then cd scripts/ansible to run playbooks — ansible.cfg resolves the
inventory and roles relative to CWD.
Playbooks
| Playbook | Purpose | Bash parity |
|---|---|---|
playbooks/dev-preflight.yml |
Dependency + port check | dev-setup.sh §checks |
playbooks/dev-teardown.yml |
Wipe state only (no reinstall) | reset-dev-env.sh §teardown |
playbooks/dev-install.yml |
Build + install + init + smoketest on clean box | dev-install.sh |
playbooks/dev-reset.yml |
Capture settings → teardown → build → install → init → smoketest | reset-dev-env.sh (full) |
playbooks/dev-deploy.yml |
Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | make deploy |
playbooks/dev-status.yml |
Read-only health snapshot (units, ports, OVN/OVS, gateway, counts, drift) | none |
playbooks/dev-logs.yml |
Dump journald + /var/log/spinifex/* + status/networking/versions into a tarball |
none |
playbooks/dev-snapshot.yml |
Quiesce spinifex+OVN+OVS, tar state paths to a named bundle, restart | none |
playbooks/dev-restore.yml |
Stop services, wipe + untar a named bundle, refresh CA, restart | none |
playbooks/dev-version.yml |
Build-vs-install drift report (spx, plugin, microvm) + sub-repo HEADs | none |
playbooks/dev-vpc.yml |
Idempotent dev VPC fixture (VPC + public subnet + IGW + default route); -e dev_vpc_state=absent tears down |
partial overlap with scripts/iac/examples/* |
Upcoming (not yet implemented):
dev-mode-start.yml/dev-mode-stop.yml— parity withstart-dev.sh/stop-dev.sh(lowest priority — deprecated bash scripts, CI-only callers).
Invocation
ansible-playbook playbooks/dev-preflight.yml
ansible-playbook playbooks/dev-teardown.yml
ansible-playbook playbooks/dev-install.yml
ansible-playbook playbooks/dev-reset.yml
ansible-playbook playbooks/dev-deploy.yml
ansible-playbook playbooks/dev-status.yml
ansible-playbook playbooks/dev-logs.yml
ansible-playbook playbooks/dev-snapshot.yml -e snapshot_name=before-merge
ansible-playbook playbooks/dev-restore.yml -e snapshot_name=before-merge
ansible-playbook playbooks/dev-version.yml
ansible-playbook playbooks/dev-vpc.yml
Or via make (from spinifex/):
make ansible-dev-preflight
make ansible-dev-teardown
make ansible-dev-install
make ansible-dev-reset
make ansible-dev-deploy
make ansible-dev-status
make ansible-dev-logs
make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'
make ansible-dev-restore ANSIBLE_EXTRA='-e snapshot_name=before-merge'
make ansible-dev-version
make ansible-dev-vpc
make ansible-dev-vpc ANSIBLE_EXTRA='-e dev_vpc_state=absent'
When to use which
- First-time / clean box →
ansible-dev-install - Iterate on Go code, microvm initramfs, lb-agent →
ansible-dev-deploy(fast) - Changed systemd units, helper scripts, logrotate, setup.sh →
ansible-dev-reset(slow, full rebuild) - Need a clean slate without reinstall →
ansible-dev-teardown - "Is my dev box healthy?" →
ansible-dev-status(read-only, never mutates) - Filing a bug / capturing the state of a broken box →
ansible-dev-logs(writes/tmp/spinifex-logs-<host>-<ts>.tar.gz; override-e logs_since=10minor-e logs_include_dmesg=false) - Checkpoint before a risky op (branch switch, viperblock surgery, ovn restart cycle) →
ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=<name>'; roll back viaansible-dev-restoreinstead of a fulldev-reset. Bundles live in/var/lib/spinifex-snapshots/<name>.tar.gz(root-owned, survivesdev-teardown). - "Do I need to
ansible-dev-deploy?" →ansible-dev-version(drift-only check; faster thandev-status) - Want a known-good VPC/subnet for manual EC2 testing without writing tofu →
ansible-dev-vpc(smoketest will then find and use it automatically). Tear down withANSIBLE_EXTRA='-e dev_vpc_state=absent'.
Variable overrides
Defaults in vars/defaults.yml. Override per run with -e:
ansible-playbook playbooks/dev-teardown.yml -e spinifex_wipe_ssh_keys=true
Useful overrides:
| Var | Default | What it does |
|---|---|---|
spinifex_wipe_legacy_home |
true |
Remove $HOME/spinifex (legacy dev-mode path) |
spinifex_wipe_ssh_keys |
false |
Remove ~/.ssh/spinifex-key* |
spinifex_wipe_aws_creds |
false |
Remove spinifex profile from ~/.aws/credentials |
teardown_process_wait_seconds |
30 |
Grace period for processes to exit |
spinifex_region |
ap-southeast-2 |
Region passed to spx admin init |
spinifex_az |
{{ region }}a |
AZ passed to spx admin init |
spinifex_external_mode |
pool |
pool | nat external networking |
dev-reset.yml automatically captures region, az, external_mode,
pool range, gateway, prefix-len, nat gateway_ip from the existing
/etc/spinifex/spinifex.toml before teardown and replays them into
init. Use -e to override.
Helper scripts
scripts/aws-as-spinifex.sh and scripts/spx-as-spinifex.sh wrap aws
and spx invocations in sg spinifex -c '...' so they run with the
spinifex supplementary group active. Both helpers preset
AWS_PROFILE=spinifex. All AWS-calling tasks in the roles use these
via spinifex_aws_cli / spinifex_spx_cli from vars/defaults.yml.
Background: /etc/spinifex/ca.pem lives under drwxr-x--- root:spinifex,
so the AWS CLI needs spinifex group membership to read the CA bundle.
Ansible's become_user is a no-op when the target user equals the
calling user (no sudo -u invoked → no initgroups(3) → supplementary
groups inherited from the operator's shell). On a fresh login the
operator's shell would have spinifex (added by setup.sh); on shells
that predate the group add it does not. The helpers sidestep this
entirely. See bead mulga-siv-92.
Leak catalog
roles/teardown/README.md tracks every known state source. When teardown
misses something, add it there and in roles/teardown/tasks/main.yml.