Files
spinifex/scripts/ansible/roles/status/tasks/main.yml
T
Josh Siviour ae6fcfe7a1 Ansible dev extensions (#254)
* chore(ansible): add dev-status playbook (read-only health doctor)

New `dev-status` role + playbook + Makefile target. Read-only snapshot of
spinifex.target, failed units, port listeners, OVN/OVS state, gateway
round-trip, AMI/keypair/volume/instance counts, and build-vs-install drift
for spx + nbdkit plugin + microvm artifacts. Pairs with dev-preflight as
the post-install / running-system diagnostic.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.1.

* chore(ansible): add dev-logs playbook (bug-report bundle)

New `logs` role + playbook + Makefile target. Captures journalctl for
spinifex-* / ovn-* / openvswitch-switch / qemu / nbdkit, copies
/var/log/spinifex/, snapshots systemctl status + networking (ip/ss/ovs/
ovn show) + spx/plugin/microvm version manifest + dmesg, then bundles
into /tmp/spinifex-logs-<host>-<ts>.tar.gz owned by the invoking user.

Tail mode deliberately omitted — devs should `journalctl -u 'spinifex-*' -f`
directly; ansible captures stdout and is the wrong tool for follow.

Overrides: -e logs_since=10min, -e logs_output_dir=/tmp,
-e logs_include_dmesg=false.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.2.

* chore(ansible): add dev-snapshot / dev-restore (state checkpoint)

New snapshot + restore roles + playbooks + Makefile targets.

dev-snapshot quiesces the full spinifex+OVN+OVS stack in dependency
order (spinifex.target → ovn-controller → ovn-central →
openvswitch-switch), kills lingering QEMU VMs, then tars /etc/spinifex,
/var/lib/spinifex (viperblock volumes included), /var/lib/ovn (NB+SB
DBs), and /etc/openvswitch/conf.db into /var/lib/spinifex-snapshots/
<name>.tar.gz with a sibling .manifest, restarts services in reverse,
and waits for the AWS gateway to come back.

dev-restore inverses: stop services, wipe the same path set, untar the
named bundle, copy /etc/spinifex/ca.pem back into the system trust
store and rebuild the CA bundle (the snapshot doesn't include
/usr/local/share/ca-certificates/spinifex-ca.crt), daemon-reload,
restart in reverse order.

Both refuse to run without -e snapshot_name=<name>, validate the name
matches [A-Za-z0-9._-]+, and produce honest summaries when the gateway
fails to come back (does not mask other ongoing issues).

Tested on dev box: snapshot/restore of 2.4 GiB state (large viperblock
volumes) completes cleanly with no data drift; services all return to
active.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.3.

* fix(ansible): wrap aws/spx calls in sg spinifex to fix group inheritance

Adds scripts/ansible/scripts/aws-as-spinifex.sh and spx-as-spinifex.sh
helpers that re-enter the command under `sg spinifex -c '...'` with
AWS_PROFILE=spinifex preset. Swaps every AWS / spx admin task across
the status, init, deploy, smoketest, snapshot, and restore roles to
use these via spinifex_aws_cli / spinifex_spx_cli vars.

Root cause (mulga-siv-92): ansible's `become_user` is a no-op when the
target user equals the calling user, so sudo never runs and
supplementary groups are inherited from the operator's invoking shell
rather than re-initialised via initgroups(3). On shells that predate
setup.sh's spinifex group add, the spinifex group is missing and
/etc/spinifex/ca.pem (drwxr-x--- root:spinifex, the AWS CLI's
ca_bundle) is unreadable — aws cli fails with SSL Errno 13. The
helpers re-enter the command in a shell that explicitly assumes the
spinifex group via sg(1), bypassing the issue regardless of how
ansible-playbook is invoked.

Side benefit: drops 50 lines of repeated `become: true / become_user /
environment: AWS_PROFILE` boilerplate across the roles.

Closes mulga-siv-92.

* chore(ansible): add dev-version playbook (build vs install drift)

New `version` role + playbook + Makefile target. Compares sha256 of
built artifacts (bin/spx, viperblock/lib/nbdkit-viperblock-plugin.so,
build/microvm/{vmlinuz,initramfs.cpio.gz}) against their installed
counterparts (/usr/local/bin/spx, plugindir, /usr/share/spinifex/
microvm/*). Prints per-artifact drift verdict plus sub-repo HEAD
short-revs (spinifex/viperblock/predastore) for context. Closes with
either "DRIFT detected — run: make ansible-dev-deploy" or "No drift".

Pure read-only. Faster than dev-status (no AWS/OVN/port probing).

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.9.

* chore(ansible): add missing roles/deploy/meta/main.yml

All other roles have meta/main.yml; deploy was the lone outlier.
Consistency fix. Closes mulga-siv-86.10.

* chore(ansible): add dev-vpc playbook (idempotent dev VPC fixture)

New `vpc` role + playbook + Makefile target. Idempotent
present/absent dispatch creates a deterministic spinifex-dev-vpc with
public subnet (10.99.1.0/24, map-public-ip-on-launch), internet
gateway, route table, and default route. Hooks into smoketest's launch
path — smoketest's existing
Name=map-public-ip-on-launch,Values=true filter now picks it up
instead of falling through to the "no subnet → skip launch" branch.

Idempotency: each step describe-then-create, looking up by Name tag
where possible. Re-running on a converged VPC is changed=0, skipped=5
(the create tasks).

Workaround for mulga-siv-94 (gateway silently drops
--tag-specifications and no-ops create-tags for route tables): role
falls back to "RT associated with our subnet" lookup so tag absence
doesn't break idempotency. create-tags task is kept as a no-op-now
forward-looking step for when the gateway gap is closed.

Teardown (-e dev_vpc_state=absent) enumerates ALL non-main route
tables in the VPC (not just the tagged one), so orphans from prior
runs that landed before the fallback was added get cleaned up too.

Tested end-to-end against josh-banksia dev gateway: create / re-run
(idempotent) / teardown all converge cleanly.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.4.
2026-05-26 16:14:11 +10:00

260 lines
9.6 KiB
YAML

---
# Read-only health snapshot. NEVER mutates state. Every probe must use
# `changed_when: false` and `failed_when: false` so the role always reaches
# the summary task even when the node is half-broken — surfacing brokenness
# is the whole point.
- name: status — spinifex.target activation
ansible.builtin.command: systemctl is-active spinifex.target
register: status_target
changed_when: false
failed_when: false
- name: status — list spinifex-* units (active + failed)
ansible.builtin.shell: |
systemctl list-units 'spinifex-*' --all --no-legend --plain \
--output=json 2>/dev/null || \
systemctl list-units 'spinifex-*' --all --no-legend --plain
register: status_units
changed_when: false
failed_when: false
- name: status — failed spinifex-* units (one line per)
ansible.builtin.shell: |
systemctl list-units 'spinifex-*' --state=failed --no-legend --plain \
| awk '{print $1}'
register: status_failed_units
changed_when: false
failed_when: false
- name: status — port listeners (which process holds each spinifex port)
ansible.builtin.shell: |
ss -tlnpH 'sport = :{{ item.port }}' | head -1
register: status_ports
loop: "{{ spinifex_ports }}"
changed_when: false
failed_when: false
loop_control:
label: "{{ item.service }} ({{ item.port }})"
- name: status — OVS tooling present?
ansible.builtin.command: which ovs-vsctl
register: status_ovs_which
changed_when: false
failed_when: false
- name: status — OVS bridges
become: true
ansible.builtin.command: ovs-vsctl list-br
register: status_ovs_brs
changed_when: false
failed_when: false
when: status_ovs_which.rc == 0
- name: status — br-int exists?
become: true
ansible.builtin.command: ovs-vsctl br-exists br-int
register: status_brint
changed_when: false
failed_when: false
when: status_ovs_which.rc == 0
- name: status — ovn-controller active?
ansible.builtin.command: systemctl is-active ovn-controller
register: status_ovn_ctrl
changed_when: false
failed_when: false
- name: status — ovn-central active?
ansible.builtin.command: systemctl is-active ovn-central
register: status_ovn_central
changed_when: false
failed_when: false
- name: status — OVN NB/SB DB files present?
become: true
ansible.builtin.stat:
path: "{{ item }}"
register: status_ovn_dbs
loop:
- "{{ ovn_nb_db }}"
- "{{ ovn_sb_db }}"
loop_control:
label: "{{ item }}"
- name: status — spinifex.toml present?
become: true
ansible.builtin.stat:
path: "{{ spinifex_config_file }}"
register: status_config
- name: status — CA trust present?
become: true
ansible.builtin.stat:
path: "{{ spinifex_ca_trust_path }}"
register: status_ca
- name: status — AWS gateway round-trip (describe-regions)
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions --output text"
register: status_awsgw
changed_when: false
failed_when: false
- name: status — installed spx binary
become: true
ansible.builtin.stat:
path: /usr/local/bin/spx
checksum_algorithm: sha256
register: status_spx_installed
- name: status — built spx binary
ansible.builtin.stat:
path: "{{ spinifex_root }}/bin/spx"
checksum_algorithm: sha256
register: status_spx_built
- name: status — spx version (installed)
ansible.builtin.shell: |
/usr/local/bin/spx version 2>/dev/null | grep -vE '^[0-9]{4}/' | head -1
register: status_spx_version
changed_when: false
failed_when: false
when: status_spx_installed.stat.exists
- name: status — installed nbdkit plugin
become: true
ansible.builtin.shell: |
set -e
pdir=$(nbdkit --dump-config | awk -F= '/^plugindir=/{print $2; exit}')
[ -f "$pdir/nbdkit-viperblock-plugin.so" ] && \
sha256sum "$pdir/nbdkit-viperblock-plugin.so" | awk '{print $1}'
register: status_plugin_installed_sha
changed_when: false
failed_when: false
- name: status — built nbdkit plugin
ansible.builtin.stat:
path: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
checksum_algorithm: sha256
register: status_plugin_built
- name: status — microvm artifacts (built)
ansible.builtin.stat:
path: "{{ spinifex_root }}/build/microvm/{{ item }}"
checksum_algorithm: sha256
register: status_microvm_built
loop:
- vmlinuz
- initramfs.cpio.gz
- name: status — microvm artifacts (installed)
become: true
ansible.builtin.stat:
path: /usr/share/spinifex/microvm/{{ item }}
checksum_algorithm: sha256
register: status_microvm_installed
loop:
- vmlinuz
- initramfs.cpio.gz
- name: status — running QEMU VMs
ansible.builtin.shell: pgrep -fa 'qemu-system-' | wc -l
register: status_qemu_count
changed_when: false
failed_when: false
- name: status — count AMIs
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-images --query length(Images) --output text"
register: status_ami_count
changed_when: false
failed_when: false
- name: status — count keypairs
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-key-pairs --query length(KeyPairs) --output text"
register: status_key_count
changed_when: false
failed_when: false
- name: status — count volumes
ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-volumes --query length(Volumes) --output text"
register: status_vol_count
changed_when: false
failed_when: false
- name: status — count running instances
ansible.builtin.command: >-
{{ spinifex_aws_cli }} ec2 describe-instances
--filters Name=instance-state-name,Values=running
--query length(Reservations[].Instances[]) --output text
register: status_instance_count
changed_when: false
failed_when: false
- name: status — compute drift facts
ansible.builtin.set_fact:
status_spx_drift: >-
{{ status_spx_installed.stat.exists and status_spx_built.stat.exists and
(status_spx_installed.stat.checksum | default('')) !=
(status_spx_built.stat.checksum | default('')) }}
status_plugin_drift: >-
{{ status_plugin_built.stat.exists and
(status_plugin_installed_sha.stdout | default('') | trim) !=
(status_plugin_built.stat.checksum | default('')) }}
status_microvm_built_present: "{{ status_microvm_built.results
| map(attribute='stat.exists') | min }}"
- name: status — render port lines
ansible.builtin.set_fact:
status_port_lines: >-
{{ status_port_lines | default([]) +
[('%-24s' | format(item.item.service ~ ' (' ~ item.item.port ~ ')'))
~ ': '
~ ('listening' if (item.stdout | length) > 0 else 'free')] }}
loop: "{{ status_ports.results }}"
loop_control:
label: "{{ item.item.service }}"
- name: status — assemble summary
ansible.builtin.set_fact:
status_summary_lines: >-
{{
[
'── Spinifex dev-node status ──────────────────────────────────────────────',
'target : ' ~ (status_target.stdout | trim),
'failed units : ' ~ (status_failed_units.stdout_lines | length)
~ ((' → ' ~ (status_failed_units.stdout_lines | join(', '))) if (status_failed_units.stdout_lines | length) > 0 else ''),
'── Ports ──'
]
+ status_port_lines
+ [
'── Networking ──',
'ovs-vsctl : ' ~ ('present' if status_ovs_which.rc == 0 else 'MISSING'),
'ovs bridges : ' ~ ((status_ovs_brs.stdout_lines | default([]) | join(', ')) or '(none)'),
'br-int : ' ~ ('present' if (status_brint.rc | default(1)) == 0 else 'MISSING'),
'ovn-controller : ' ~ (status_ovn_ctrl.stdout | trim),
'ovn-central : ' ~ (status_ovn_central.stdout | trim),
'ovn NB DB : ' ~ ('present' if status_ovn_dbs.results[0].stat.exists else 'absent'),
'ovn SB DB : ' ~ ('present' if status_ovn_dbs.results[1].stat.exists else 'absent'),
'── Config / trust ──',
'spinifex.toml : ' ~ ('present' if status_config.stat.exists else 'MISSING') ~ ' (' ~ spinifex_config_file ~ ')',
'CA trust : ' ~ ('present' if status_ca.stat.exists else 'MISSING') ~ ' (' ~ spinifex_ca_trust_path ~ ')',
'── AWS gateway ──',
'describe-regions : ' ~ ('OK' if status_awsgw.rc == 0 else 'FAIL (rc=' ~ status_awsgw.rc ~ ')'),
'AMIs : ' ~ (status_ami_count.stdout | default('?') | trim),
'keypairs : ' ~ (status_key_count.stdout | default('?') | trim),
'volumes : ' ~ (status_vol_count.stdout | default('?') | trim),
'running instances: ' ~ (status_instance_count.stdout | default('?') | trim),
'running QEMU : ' ~ (status_qemu_count.stdout | default('?') | trim),
'── Versions / drift ──',
'spx (installed) : ' ~ ((status_spx_version.stdout | default('') | trim) or '(not installed)'),
'spx drift : ' ~ ('YES (built ≠ installed) — run ansible-dev-deploy' if status_spx_drift else 'no'),
'plugin drift : ' ~ ('YES (built ≠ installed) — run ansible-dev-deploy' if status_plugin_drift else 'no'),
'microvm built : ' ~ ('present' if status_microvm_built_present else 'not built on this branch'),
'──────────────────────────────────────────────────────────────────────────'
]
}}
- name: status — summary
ansible.builtin.debug:
msg: "{{ status_summary_lines }}"