gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: record the AMI-metadata upgrade cliff and binary-swap skew (#773)

Browse Source 
AMI metadata moved to ebsmetadata documents and the legacy
ami-<id>/config.json read path was removed rather than migrated, so an
AMI imported before the change is invisible to the control plane after
upgrading. Nothing reports it as pending, because the gap is in stored
data rather than in a config file. Warn about it alongside the existing
object-storage exemption, and say how to check and recover.

Also document the binary-swap trap: replacing spx under a running
target leaves services on the replaced inode, and because handlers are
queue-group workers a skewed node answers a fraction of requests with
the old behaviour, which reads as an intermittent fault.
This commit is contained in:
Josh Siviour
2026-08-12 21:45:46 +10:00
committed by GitHub
parent 4faf6e8024
commit 1e1d00cda1
1 changed files with 45 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin/update/README.md
+45
View File
@@ -35,6 +35,17 @@ For operators who want to review migrations before they are applied, a manual up
> [!WARNING]
> **Object storage is exempt when upgrading from v1.15.0 or earlier to v1.16.0.** Predastore's configuration schema and on-disk layout changed with the object storage cutover in v1.16.0, and no migration converts an installation from before it. `spx admin upgrade` will not report anything pending for `predastore.toml`, and updating the binary over such an installation leaves Predastore unable to start. Those clusters have to be re-initialised from scratch, which discards their stored objects — export anything you need first. Upgrades between v1.16.0 and later releases are unaffected.
> [!WARNING]
> **AMI metadata is exempt when upgrading from v1.16.0 or earlier to the release carrying the EBS-provider decoupling.** AMI metadata moved to `ebsmetadata` documents, and the legacy path that read it from `ami-<id>/config.json` was removed rather than migrated. There is no prefix scan, no fallback and no backfill at daemon start, so an AMI imported before the change becomes invisible to the control plane afterwards: `describe-images --image-ids` answers `InvalidAMIID.NotFound` and launches fail with `AMI has no snapshot ID, cannot perform zero-copy clone`. As with object storage above, `spx admin upgrade` reports nothing pending, because the gap is in stored data rather than in a config file.
>
> **Re-import affected AMIs after upgrading**, using [`spx admin images import`](/docs/admin/spinifex-admin-cli), which writes metadata in the new location. If the source images are no longer available, the installation has to be re-initialised. Verify before you rely on it — an AMI is only healthy if it resolves by ID, not merely if it appears in the list:
>
> ```bash
> aws ec2 describe-images --image-ids <ami-id>
> ```
>
> This warning covers AMI metadata specifically, which is what has been observed. Check any other imported state you depend on before upgrading a cluster you cannot rebuild.
## Instructions
## Step 1. Re-run the Installer
@@ -129,3 +140,37 @@ sudo systemctl restart spinifex.target
```
A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See [Host and Guest Lifecycle](/docs/admin/host-lifecycle) for the full contract.
### A Node Answers Some Requests as if It Were Still on the Old Build
Replacing `/usr/local/bin/spx` while `spinifex.target` is running does **not** move the running services onto the new binary. They keep executing the replaced file's now-unlinked inode until each unit restarts, so a node can serve the old build indefinitely. Only a service that happens to restart for its own reasons picks the new one up, which leaves a node running a mixture.
This is easy to miss, because the request handlers are NATS queue-group workers spread across nodes: one skewed node in three answers roughly one request in three with the old behaviour, which reads as an intermittent fault rather than a broken node.
Check for it with:
```bash
for u in spinifex-daemon spinifex-awsgw spinifex-viperblock spinifex-vpcd spinifex-ui spinifex-predastore; do
pid=$(systemctl show -p MainPID --value "$u")
[ "$pid" != "0" ] && printf '%-22s %s\n' "$u" "$(sudo readlink /proc/$pid/exe)"
done
```
Any line ending `(deleted)` is running a replaced binary. Restart the target to clear it:
```bash
sudo systemctl restart spinifex.target
```
Re-running the installer avoids this entirely — it restarts services after installing. Prefer it over copying a binary onto a live node.
### Instances Fail to Launch After an Upgrade
```
AMI has no snapshot ID, cannot perform zero-copy clone
```
Or `describe-images --image-ids` reports `InvalidAMIID.NotFound` for an AMI that still appears in the unfiltered `describe-images` list. Two different causes produce this, so check them in order:
1. **A node running a replaced binary**, per the previous entry. Suspect this first if the failure is intermittent — the same command succeeding on some attempts and failing on others is characteristic.
2. **AMI metadata predating the EBS-provider decoupling**, per the warning at the top of this page. This is consistent rather than intermittent, and is resolved by re-importing the AMI.