Files
spinifex/scripts/ansible/roles/logs/tasks/main.yml
T
Josh Siviour ae6fcfe7a1 Ansible dev extensions (#254)
* chore(ansible): add dev-status playbook (read-only health doctor)

New `dev-status` role + playbook + Makefile target. Read-only snapshot of
spinifex.target, failed units, port listeners, OVN/OVS state, gateway
round-trip, AMI/keypair/volume/instance counts, and build-vs-install drift
for spx + nbdkit plugin + microvm artifacts. Pairs with dev-preflight as
the post-install / running-system diagnostic.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.1.

* chore(ansible): add dev-logs playbook (bug-report bundle)

New `logs` role + playbook + Makefile target. Captures journalctl for
spinifex-* / ovn-* / openvswitch-switch / qemu / nbdkit, copies
/var/log/spinifex/, snapshots systemctl status + networking (ip/ss/ovs/
ovn show) + spx/plugin/microvm version manifest + dmesg, then bundles
into /tmp/spinifex-logs-<host>-<ts>.tar.gz owned by the invoking user.

Tail mode deliberately omitted — devs should `journalctl -u 'spinifex-*' -f`
directly; ansible captures stdout and is the wrong tool for follow.

Overrides: -e logs_since=10min, -e logs_output_dir=/tmp,
-e logs_include_dmesg=false.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.2.

* chore(ansible): add dev-snapshot / dev-restore (state checkpoint)

New snapshot + restore roles + playbooks + Makefile targets.

dev-snapshot quiesces the full spinifex+OVN+OVS stack in dependency
order (spinifex.target → ovn-controller → ovn-central →
openvswitch-switch), kills lingering QEMU VMs, then tars /etc/spinifex,
/var/lib/spinifex (viperblock volumes included), /var/lib/ovn (NB+SB
DBs), and /etc/openvswitch/conf.db into /var/lib/spinifex-snapshots/
<name>.tar.gz with a sibling .manifest, restarts services in reverse,
and waits for the AWS gateway to come back.

dev-restore inverses: stop services, wipe the same path set, untar the
named bundle, copy /etc/spinifex/ca.pem back into the system trust
store and rebuild the CA bundle (the snapshot doesn't include
/usr/local/share/ca-certificates/spinifex-ca.crt), daemon-reload,
restart in reverse order.

Both refuse to run without -e snapshot_name=<name>, validate the name
matches [A-Za-z0-9._-]+, and produce honest summaries when the gateway
fails to come back (does not mask other ongoing issues).

Tested on dev box: snapshot/restore of 2.4 GiB state (large viperblock
volumes) completes cleanly with no data drift; services all return to
active.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.3.

* fix(ansible): wrap aws/spx calls in sg spinifex to fix group inheritance

Adds scripts/ansible/scripts/aws-as-spinifex.sh and spx-as-spinifex.sh
helpers that re-enter the command under `sg spinifex -c '...'` with
AWS_PROFILE=spinifex preset. Swaps every AWS / spx admin task across
the status, init, deploy, smoketest, snapshot, and restore roles to
use these via spinifex_aws_cli / spinifex_spx_cli vars.

Root cause (mulga-siv-92): ansible's `become_user` is a no-op when the
target user equals the calling user, so sudo never runs and
supplementary groups are inherited from the operator's invoking shell
rather than re-initialised via initgroups(3). On shells that predate
setup.sh's spinifex group add, the spinifex group is missing and
/etc/spinifex/ca.pem (drwxr-x--- root:spinifex, the AWS CLI's
ca_bundle) is unreadable — aws cli fails with SSL Errno 13. The
helpers re-enter the command in a shell that explicitly assumes the
spinifex group via sg(1), bypassing the issue regardless of how
ansible-playbook is invoked.

Side benefit: drops 50 lines of repeated `become: true / become_user /
environment: AWS_PROFILE` boilerplate across the roles.

Closes mulga-siv-92.

* chore(ansible): add dev-version playbook (build vs install drift)

New `version` role + playbook + Makefile target. Compares sha256 of
built artifacts (bin/spx, viperblock/lib/nbdkit-viperblock-plugin.so,
build/microvm/{vmlinuz,initramfs.cpio.gz}) against their installed
counterparts (/usr/local/bin/spx, plugindir, /usr/share/spinifex/
microvm/*). Prints per-artifact drift verdict plus sub-repo HEAD
short-revs (spinifex/viperblock/predastore) for context. Closes with
either "DRIFT detected — run: make ansible-dev-deploy" or "No drift".

Pure read-only. Faster than dev-status (no AWS/OVN/port probing).

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.9.

* chore(ansible): add missing roles/deploy/meta/main.yml

All other roles have meta/main.yml; deploy was the lone outlier.
Consistency fix. Closes mulga-siv-86.10.

* chore(ansible): add dev-vpc playbook (idempotent dev VPC fixture)

New `vpc` role + playbook + Makefile target. Idempotent
present/absent dispatch creates a deterministic spinifex-dev-vpc with
public subnet (10.99.1.0/24, map-public-ip-on-launch), internet
gateway, route table, and default route. Hooks into smoketest's launch
path — smoketest's existing
Name=map-public-ip-on-launch,Values=true filter now picks it up
instead of falling through to the "no subnet → skip launch" branch.

Idempotency: each step describe-then-create, looking up by Name tag
where possible. Re-running on a converged VPC is changed=0, skipped=5
(the create tasks).

Workaround for mulga-siv-94 (gateway silently drops
--tag-specifications and no-ops create-tags for route tables): role
falls back to "RT associated with our subnet" lookup so tag absence
doesn't break idempotency. create-tags task is kept as a no-op-now
forward-looking step for when the gateway gap is closed.

Teardown (-e dev_vpc_state=absent) enumerates ALL non-main route
tables in the VPC (not just the tagged one), so orphans from prior
runs that landed before the fallback was added get cleaned up too.

Tested end-to-end against josh-banksia dev gateway: create / re-run
(idempotent) / teardown all converge cleanly.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.4.
2026-05-26 16:14:11 +10:00

154 lines
5.6 KiB
YAML

---
# Collect a self-contained log bundle for bug reports. NEVER mutates running
# state — only reads journalctl + /var/log/spinifex/* + status output, then
# tars into logs_output_dir. Errors on missing inputs are non-fatal so a
# half-broken node still produces a bundle.
- name: logs — resolve staging directory
ansible.builtin.set_fact:
logs_stage_dir: "/tmp/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}"
logs_bundle_path: "{{ logs_output_dir }}/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}.tar.gz"
- name: logs — create staging tree
ansible.builtin.file:
path: "{{ logs_stage_dir }}/{{ item }}"
state: directory
mode: "0755"
loop:
- journal
- varlog
- status
- kernel
- name: logs — capture journalctl for spinifex-* units
become: true
ansible.builtin.shell: |
journalctl --no-pager --output=short-iso \
-u 'spinifex-*' --since {{ logs_since | quote }} \
> {{ logs_stage_dir }}/journal/spinifex.log 2>&1 || true
changed_when: true
- name: logs — capture journalctl for ovn-* + openvswitch-switch
become: true
ansible.builtin.shell: |
journalctl --no-pager --output=short-iso \
-u 'ovn-*' -u openvswitch-switch \
--since {{ logs_since | quote }} \
> {{ logs_stage_dir }}/journal/ovn-ovs.log 2>&1 || true
changed_when: true
- name: logs — capture journalctl for nbdkit + qemu (best-effort match)
become: true
ansible.builtin.shell: |
journalctl --no-pager --output=short-iso \
--since {{ logs_since | quote }} \
_COMM=qemu-system-x86_64 + _COMM=qemu-system-aarch64 + _COMM=nbdkit \
> {{ logs_stage_dir }}/journal/qemu-nbdkit.log 2>&1 || true
changed_when: true
- name: logs — copy /var/log/spinifex/*
become: true
ansible.builtin.shell: |
if [ -d {{ spinifex_log_dir | quote }} ]; then
cp -a {{ spinifex_log_dir | quote }}/. {{ logs_stage_dir | quote }}/varlog/ 2>/dev/null || true
fi
changed_when: true
- name: logs — systemctl status snapshot (spinifex-*)
become: true
ansible.builtin.shell: |
systemctl status 'spinifex-*' --no-pager --all \
> {{ logs_stage_dir }}/status/spinifex-units.txt 2>&1 || true
systemctl status ovn-controller ovn-central openvswitch-switch \
--no-pager > {{ logs_stage_dir }}/status/ovn-ovs-units.txt 2>&1 || true
systemctl list-units 'spinifex-*' --all --no-legend --plain \
> {{ logs_stage_dir }}/status/spinifex-list.txt 2>&1 || true
changed_when: true
- name: logs — networking snapshot (ip, ss, ovs)
become: true
ansible.builtin.shell: |
ip -d -s link show > {{ logs_stage_dir }}/status/ip-link.txt 2>&1 || true
ip addr show > {{ logs_stage_dir }}/status/ip-addr.txt 2>&1 || true
ip route show table all > {{ logs_stage_dir }}/status/ip-route.txt 2>&1 || true
ss -tlnp > {{ logs_stage_dir }}/status/ss-tlnp.txt 2>&1 || true
if command -v ovs-vsctl >/dev/null 2>&1; then
ovs-vsctl show > {{ logs_stage_dir }}/status/ovs-show.txt 2>&1 || true
ovs-vsctl list-br > {{ logs_stage_dir }}/status/ovs-br.txt 2>&1 || true
fi
if command -v ovn-nbctl >/dev/null 2>&1; then
ovn-nbctl show > {{ logs_stage_dir }}/status/ovn-nb.txt 2>&1 || true
ovn-sbctl show > {{ logs_stage_dir }}/status/ovn-sb.txt 2>&1 || true
fi
changed_when: true
- name: logs — kernel ring buffer
become: true
ansible.builtin.shell: |
dmesg --time-format=iso > {{ logs_stage_dir }}/kernel/dmesg.txt 2>&1 || true
changed_when: true
when: logs_include_dmesg | bool
- name: logs — capture spx + plugin versions
become: true
ansible.builtin.shell: |
{
echo '## spx version'
/usr/local/bin/spx version 2>&1 | grep -vE '^[0-9]{4}/' || true
echo
echo '## nbdkit plugin sha256'
pdir=$(nbdkit --dump-config 2>/dev/null | awk -F= '/^plugindir=/{print $2; exit}')
[ -n "$pdir" ] && sha256sum "$pdir/nbdkit-viperblock-plugin.so" 2>/dev/null || true
echo
echo '## microvm artifacts'
sha256sum /usr/share/spinifex/microvm/* 2>/dev/null || true
echo
echo '## kernel + os'
uname -a
cat /etc/os-release 2>/dev/null || true
} > {{ logs_stage_dir }}/status/versions.txt 2>&1 || true
changed_when: true
- name: logs — create bundle tarball
become: true
ansible.builtin.shell: |
tar -C {{ logs_stage_dir | dirname | quote }} \
-czf {{ logs_bundle_path | quote }} \
{{ logs_stage_dir | basename | quote }}
changed_when: true
- name: logs — make bundle owned by invoking user
become: true
ansible.builtin.file:
path: "{{ logs_bundle_path }}"
owner: "{{ spinifex_invoking_user }}"
group: "{{ spinifex_invoking_user }}"
mode: "0640"
- name: logs — remove staging tree
become: true
ansible.builtin.file:
path: "{{ logs_stage_dir }}"
state: absent
- name: logs — bundle size
ansible.builtin.stat:
path: "{{ logs_bundle_path }}"
register: logs_bundle_stat
- name: logs — summary
ansible.builtin.debug:
msg: |
Log bundle written:
path : {{ logs_bundle_path }}
size : {{ (logs_bundle_stat.stat.size | int / 1024) | round(1) }} KiB
since : {{ logs_since }}
Contents:
journal/ spinifex-*, ovn-*+openvswitch-switch, qemu/nbdkit
varlog/ copy of {{ spinifex_log_dir }}
status/ systemctl status, ip/ss/ovs/ovn show, versions
kernel/ dmesg{{ '' if logs_include_dmesg else ' (skipped via -e logs_include_dmesg=false)' }}
Attach to bug report. For live tail use:
journalctl -u 'spinifex-*' -f