gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-19 10:52:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
dev
spinifex/scripts/terraform/oci-spx/eip-move-test.sh
T
Ben Duncan 8409799cbf fix: default the OCI helper to the DEFAULT profile (#1174) 
* fix: default the OCI helper to the DEFAULT profile

oci_env.py shipped our own internal Oracle compartment profile as the default for
every reader, with a second internal name as a fallback.
A fresh tenancy's config holds only `[DEFAULT]`, so the shipped default was a name
nobody else can have, and the fallback chain could land a deployment in whichever
tenancy a different profile happened to point at.

The profile is now `DEFAULT`, overridable by `OCI_CLI_PROFILE` or `--profile`, and a
named profile the config does not hold is an error rather than a fallback.
`DEFAULT` needs its own existence test, because configparser treats it as the
fallback for every other section and `has_section` denies it exists even when
`oci setup config` wrote nothing else.

`eip-move-test.sh` and the config table in the README carried the same name.
CI reads its credential from the environment rather than from a profile, so nothing
there referenced it.

* docs: look the OCI compartment OCID up by name

The guide asked the reader to paste `ocid1.compartment.oc1..xxx` three times, and
the `oci` CLI it already requires can resolve the name instead.

The lookup needs its own check. `oci iam compartment list --query ... | [0]` exits 0
and prints nothing when the name matches nothing, so an unset `$CID` would otherwise
reach Terraform and fail with an OCI error naming neither the compartment nor the
cause.

Two compartments in one tenancy can share a name, which the listing command shows
and the lookup cannot resolve, so the paste form stays as the answer for that case.
Records why the tenancy root is the wrong value: the step 2.1 policy is scoped to
whatever `$CID` holds.

* Revert "docs: look the OCI compartment OCID up by name"

This reverts commit 5443149cce.

* Reapply "docs: look the OCI compartment OCID up by name"

This reverts commit 1535d94c8a.

* fix: stop validate-topology.sh failing on macOS bash

`tf_principal_var` is empty on every run that does not pass `--instance-principal`,
and bash before 4.4 treats `"${arr[@]}"` on an empty array as an unbound variable
under `set -u`.
macOS ships bash 3.2, so the documented deployment command aborted at the tf_vars
assembly with `tf_principal_var[@]: unbound variable` before building anything.

Assigning the array was not enough, which is why the earlier `HOSTS=()` fix did not
cover this: the error is in the expansion, not the reference.
Both sites now expand only behind a length test, which is valid on every version.

`capture_journals` had the same shape for `HOSTS`, reachable when a run fails before
the apply.

Reproduced and fixed against bash 3.2.57 in a container, which emits the identical
message, and the script now runs to the terraform call on 3.2 for vm-single,
vm-single-principal and `--instance-principal`, including the `--keep` cleanup path
with no hosts.

* docs: record the bash 3.x requirement in the OCI scripts

These three are the scripts the OCI guide tells an operator to run, and macOS ships
bash 3.2, so a bash 4 construct in any of them breaks a deployment rather than a
test.
The constraint has now been rediscovered twice, through declare -A and mapfile, and
then through an empty array expanded under set -u.

Names the three traps at the top of each file so the next change does not have to
find them by failing on someone's Mac.

* fix: stream the workbook suite instead of hiding it in a file

The nine workbooks ran with stdout redirected to `workbooks.log`, so a run wedged
inside one printed nothing until the whole suite finished.
A CI job then sat silent for hours with no way to tell which workbook it was in, or
whether it was working at all.

The output is now teed, so the file the per-workbook table is parsed from is
unchanged while the run is visible live.
Each of the driver's own RUN/PASS/FAIL lines is stamped with elapsed seconds and the
workbook name, which is what identifies the slow one without reading the log.

`pipefail` carries the ssh exit status through the pipe, so a failed suite still
takes the branch that records the table and captures the journals.

* fix: grant the OCI policy the subnet verb AssignPrivateIP requires

An OCI node with a pool-mode external interface formed, reported healthy, and
then refused every launch wanting a public address with a 404
NotAuthorizedOrNotFound from AssignPrivateIP.

AssignPrivateIP calls CreatePrivateIp, which Oracle documents as needing
PRIVATE_IP_CREATE, PRIVATE_IP_ASSIGN, VNIC_ASSIGN and SUBNET_ATTACH, and
DeletePrivateIp as needing SUBNET_DETACH.
The policy granted use vnics, manage private-ips and manage public-ips, and no
subnet permission at all, so SUBNET_ATTACH was never held.
Only the subnets use verb carries it.

Measured on a live node as the instance principal: GetVnic and ListPrivateIps
succeed, GetSubnet and CreatePrivateIp both 404.
Authentication, the dynamic group's matching rule and the compartment were all
correct, so the credential was authentic and unauthorised.
The documented operator policy for the API-key route carried the same three
statements, so neither credential route could allocate.

A node now proves its own credential at pool start by making one read in each
permission class, and logs the missing verb if one fails.
The topology gate waited for "resolved the external VNIC", which comes from the
metadata service and needs no IAM policy, so an unauthorised node passed the
gate and failed an hour later inside a workbook as capacity.
It now waits for the authorisation line and fails immediately on a refusal.

Reads rather than a real assign-and-release round trip: a round trip also fails
on a full subnet, and reporting that as an authorisation fault sends an operator
to IAM for a capacity problem.

* docs: say what a principal run costs in a tenancy with no dynamic group

No principal topology can pass there, and the only previous hint was one clause
about OCI_INSTANCE_PRINCIPAL needing the group to exist.

* test: cover the subnet read and the authorisation verdict

Diff coverage was 39.1% against a 60% gate. Three of the four gaps were real.

GetSubnet on the SDK-backed client had no test, so the request shape was
unverified: addressing the collection rather than one subnet by OCID would
answer from a different permission and report a credential as working when it is
not. Also pins that a sparse body is not an error and that the unauthorised 404
arrives as ErrNotFound, which is what stops the check retrying a verdict that
will never change.

The fake's GetSubnet was exercised only from the ocinet tests, and coverage is
recorded per package under test, so it read as zero-covered.

reportAuthorisation now has a test asserting the exact log text, because
validate-topology.sh greps for it and a rename would quietly turn the gate into
one that never fires.

The six lines still uncovered are inside FromPoolConfig, which needs a live
JetStream to reach. Diff coverage 87.0%.

* fix: refuse a run whose tfvars file contradicts the compartment it was given

Terraform ranks *.auto.tfvars above TF_VAR_*, so a leftover file wins silently
over an operator targeting a different tenancy by environment, and the guide
tells every operator to write one.

Hit for real: a stale terraform.auto.tfvars naming a different tenancy's
compartment and a bare-metal shape overrode TF_VAR_compartment_ocid, and the
plan was for 11 resources in a compartment nobody had chosen. Only a dry run
caught it.

The run now stops and names both values rather than building.

* fix: collect orphaned OCI public IPs and name the limit that was hit

A reserved public IP attached to nothing is on no VNIC, so the reconcile
that exists to find leaked addresses could not see the ones that matter.
It also ran only at startup, while an address is leaked on release and on
teardown. Twenty accumulated in one compartment and exhausted the
tenancy's limit of 50, after which every launch needing a public address
aborted.

Reconcile now lists reserved public IPs in the compartment and collects
those that carry our prefix, are attached to nothing, are named by no
binding and are older than a grace period. All four are needed: an
allocated but unassociated Elastic IP is detached too and is the
customer's to keep, and a young detached address is what another node's
CreatePublicIP looks like until it attaches. A pool with no binding
record is not swept at all, since that is indistinguishable from a store
that cannot be read.

The pass now repeats on a ticker rather than running once at startup.

Both OCI ceilings also resolved to InsufficientAddressCapacity, which
tells an operator the region is out of addresses when the fix is to
release addresses they already hold. AWS separates them, so a tenancy out
of reserved public IPs is now AddressLimitExceeded and a full VNIC is
PrivateIpAddressLimitExceeded.

* feat: run the OCI principal e2e in a tenancy that can hold its dynamic group

vm-single-principal needs the tenancy's dynamic group, and creating one is a
tenancy-root write that a shared test tenancy does not grant us. The topology
could never pass there however correct the code was, so it now takes its own
credential from OCI_PRINCIPAL_*, all six or none. A partial set is refused
before the build rather than at the apply, because it would build one tenancy's
instances against another's compartment. With none set the topology is recorded
SKIPPED instead of spending a provision to fail.

The sweep takes the same credential, or a leftover principal deployment is
destroyed against a tenancy that never held it and reported clean.

Two defects found while proving it:

--dry-run reported FAIL on every topology with a tree source, because the dry
run skips the build and then refuses the tarball it would have produced. The
flag operators are told to run first could not pass.

The dry run truncated its plan to the last 30 lines, which is the outputs block.
That dropped the resource count and every compartment and shape line, so the
one check a dry run exists for could not be made from its output. It now writes
the whole plan and prints the summary.

Removing a deployment is documented as leaving public IPs behind. Spinifex
allocates them on the running node, so they are in no Terraform state and
destroy cannot see them; twenty accumulated and exhausted a tenancy's limit,
which then refused every launch in every compartment.

* fix: canonicalise the Opc-Next-Page header in the pagination test

CI's linter rejects a non-canonical header name and the local preflight does
not, so this passed here and failed there.
2026-10-06 20:56:16 -05:00

237 lines
10 KiB
Bash
Executable File
Raw Permalink Blame History

#!/usr/bin/env bash
# eip-move-test.sh — prove an EIP follows its guest across an OCI node move.
#
# Drives the three-node Terraform OCI cluster through the exact sequence that
# strands an address today: associate an EIP, stop the guest, force it to start
# on a different node, and assert the address still answers. The decisive check
# is not the ping — it is the OCI API's own record of which VNIC carries the
# private half, because a reachable address on the wrong VNIC is luck.
#
# Step 0b first proves the other half of the distinction on the same guest: the
# auto-assigned address it launched with is gone from the OCI API while it is
# stopped, and a different one comes back. That is the billing claim, asserted
# against OCI rather than against our own record of what we think we released.
#
# Read-write on dev-oci only. Never point this at prod or dev-prod.
set -euo pipefail
KEY=${KEY:-$HOME/.ssh/oci-poc.pem}
OCI_BIN=${OCI_BIN:-$HOME/bin/oci}
OCI_PROFILE=${OCI_PROFILE:-DEFAULT}
SUBNET=${SUBNET:-ocid1.subnet.oc1.ap-sydney-1.aaaaaaaaimrhjgsqgrfg7bnka2d3nbx2n5ss7kurcwneon22rkclqf3ypfqa}
# Public address -> node name. The control plane is reached through node01.
declare -A NODE_IP=(
[spinifexnode01]=149.118.68.212
[spinifexnode02]=137.23.19.167
[spinifexnode03]=137.23.14.3
)
API_HOST=${API_HOST:-${NODE_IP[spinifexnode01]}}
say() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
fail() { printf '\033[31mFAIL: %s\033[0m\n' "$*" >&2; FAILED=1; }
ok() { printf '\033[32mok: %s\033[0m\n' "$*"; }
FAILED=0
# Assertions take values rather than an exit status: `cond && ok || fail` runs
# the failure branch whenever ok's own status is non-zero, and passing $? through
# a helper only moves the same trap one line down.
eq() { # eq <actual> <expected> <what>
if [[ $1 == "$2" ]]; then ok "$3 ($1)"; else fail "$3: got '$1', wanted '$2'"; fi
}
unset_or_none() { # unset_or_none <actual> <what>
if [[ -z $1 || $1 == None ]]; then ok "$2"; else fail "$2: got '$1'"; fi
}
is_set() { # is_set <actual> <what>
if [[ -n $1 && $1 != None ]]; then ok "$2"; else fail "$2: got '$1'"; fi
}
on() { # on <ip> <cmd...>
local ip=$1; shift
timeout 60 ssh -i "$KEY" -o StrictHostKeyChecking=no -o BatchMode=yes "ubuntu@$ip" "$@"
}
aws_spx() { # aws_spx <args...> — run the AWS CLI from node01 against the cluster
on "$API_HOST" "export AWS_PROFILE=spinifex AWS_CA_BUNDLE=/etc/spinifex/ca.pem; \
aws --endpoint-url https://localhost:9999 --region ap-southeast-2 $*"
}
# node_of <instance-id> — which node is the guest on right now
node_of() {
on "$API_HOST" "sudo spx get instances --config /etc/spinifex/spinifex.toml 2>/dev/null" \
| awk -v id="$1" '$0 ~ id {print}' | grep -oE 'spinifexnode0[0-9]' | head -1
}
# private_half_of <public-addr> — the OCI private address this EIP rides the
# wire as. Read off the OVN row that stamps the public address, which is the
# one place the pairing is recorded outside the bindings bucket. Guessing from
# the host's /32 routes would be ambiguous: a node hosting the guest carries the
# auto-assigned address's private half on the same VNIC.
private_half_of() {
on "$API_HOST" "NB=\$(sudo grep -oP '^ovn_nb_addr\\s*=\\s*\"\\K[^\"]+' /etc/spinifex/spinifex.toml); \
sudo ovn-nbctl --db=\"\$NB\" --bare --columns=external_ip \
find NAT 'external_ids:spinifex\\:public_ip=\"$1\"'" 2>/dev/null | tr -d '\r' | head -1
}
# reserved_ip_ocid <public-addr> — the OCI public IP object behind an address,
# empty when OCI holds none. This is the object the tenancy is billed for, so
# it — not our own KV — is what answers whether a stop really gave it back.
reserved_ip_ocid() {
"$OCI_BIN" --profile "$OCI_PROFILE" network public-ip get \
--public-ip-address "$1" --query 'data.id' --raw-output 2>/dev/null || true
}
# state_of <instance-id> — the EC2 state name, for polling
state_of() {
aws_spx "ec2 describe-instances --instance-ids $1 \
--query 'Reservations[0].Instances[0].State.Name' --output text" 2>/dev/null | tr -d '\r'
}
# public_ip_of <instance-id> — what DescribeInstances reports, "None" when none
public_ip_of() {
aws_spx "ec2 describe-instances --instance-ids $1 \
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text" 2>/dev/null | tr -d '\r'
}
# is_state <instance-id> <state> — a predicate for wait_for
is_state() { [[ $(state_of "$1") == "$2" ]]; }
# vnic_of_private <addr> — the OCI VNIC OCID currently carrying a private IP
vnic_of_private() {
"$OCI_BIN" --profile "$OCI_PROFILE" network private-ip list \
--subnet-id "$SUBNET" --ip-address "$1" --query 'data[0]."vnic-id"' --raw-output 2>/dev/null
}
# external_vnic_of <node-name> — that node's br-wan (secondary) VNIC OCID
external_vnic_of() {
on "${NODE_IP[$1]}" 'curl -s -H "Authorization: Bearer Oracle" -L \
http://169.254.169.254/opc/v2/vnics/ | python3 -c "
import sys,json
v=json.load(sys.stdin)
print(v[1][\"vnicId\"] if len(v)>1 else \"\")"'
}
reachable() { ping -c4 -W2 "$1" >/dev/null 2>&1; }
wait_for() { # wait_for <seconds> <description> <cmd...>
local secs=$1 desc=$2; shift 2
local deadline=$(( SECONDS + secs ))
while (( SECONDS < deadline )); do
if "$@"; then ok "$desc"; return 0; fi
sleep 5
done
fail "$desc (timed out after ${secs}s)"
return 1
}
INSTANCE=${INSTANCE:-}
ALLOC=""
cleanup() {
[[ -n $ALLOC ]] || return 0
say "cleanup: releasing $ALLOC"
local assoc
assoc=$(aws_spx "ec2 describe-addresses --query \
\"Addresses[?AllocationId=='$ALLOC'].AssociationId\" --output text" 2>/dev/null | tr -d '\r')
[[ -n $assoc && $assoc != None ]] && aws_spx "ec2 disassociate-address --association-id $assoc" >/dev/null 2>&1
sleep 5
aws_spx "ec2 release-address --allocation-id $ALLOC" >/dev/null 2>&1 || true
}
trap cleanup EXIT
[[ -n $INSTANCE ]] || { echo "set INSTANCE=i-... (a running guest with an ENI)" >&2; exit 2; }
say "0. starting state"
ENI=$(aws_spx "ec2 describe-instances --instance-ids $INSTANCE \
--query 'Reservations[0].Instances[0].NetworkInterfaces[0].NetworkInterfaceId' --output text" | tr -d '\r')
AUTO=$(public_ip_of "$INSTANCE")
echo "instance $INSTANCE on $(node_of "$INSTANCE"), eni $ENI, auto-assigned ${AUTO:-<none>}"
# An auto-assigned address is borrowed for as long as the guest runs. On OCI it
# is a reserved public IP object billed to the tenancy, so "released on stop" is
# a billing claim and OCI's own API is the only place it can be proved.
say "0b. an auto-assigned address goes back to OCI on stop"
if [[ -z $AUTO || $AUTO == None ]]; then
echo "skipped: guest has no auto-assigned address"
else
AUTO_OCID=$(reserved_ip_ocid "$AUTO")
is_set "$AUTO_OCID" "OCI holds a public IP object for $AUTO while running"
aws_spx "ec2 stop-instances --instance-ids $INSTANCE" >/dev/null
wait_for 180 "instance stopped" is_state "$INSTANCE" stopped
unset_or_none "$(public_ip_of "$INSTANCE")" "stopped instance reports no PublicIpAddress"
unset_or_none "$(reserved_ip_ocid "$AUTO")" "OCI no longer bills a public IP for $AUTO"
aws_spx "ec2 start-instances --instance-ids $INSTANCE" >/dev/null
wait_for 300 "instance running again" is_state "$INSTANCE" running
RESTARTED=$(public_ip_of "$INSTANCE")
is_set "$RESTARTED" "restarted instance has an auto-assigned address"
if [[ $RESTARTED == "$AUTO" ]]; then
fail "restart returned the same address $AUTO — the stop did not release it"
else
ok "restart returned a different address ($AUTO -> $RESTARTED)"
fi
is_set "$(reserved_ip_ocid "$RESTARTED")" "OCI holds a public IP object for $RESTARTED"
fi
# After the restart above, so the move in step 4 is measured from where the
# guest actually is rather than from where it started the script.
START_NODE=$(node_of "$INSTANCE")
echo "instance now on $START_NODE"
say "1. allocate and associate an EIP"
ALLOC=$(aws_spx "ec2 allocate-address --query AllocationId --output text" | tr -d '\r')
EIP=$(aws_spx "ec2 describe-addresses --allocation-ids $ALLOC --query 'Addresses[0].PublicIp' --output text" | tr -d '\r')
aws_spx "ec2 associate-address --allocation-id $ALLOC --network-interface-id $ENI" >/dev/null
echo "allocated $ALLOC = $EIP"
wait_for 300 "EIP $EIP answers on $START_NODE" reachable "$EIP"
PRIVATE=$(private_half_of "$EIP")
echo "private half: ${PRIVATE:-<unresolved>}"
is_set "$PRIVATE" "the EIP resolves to an OCI private address"
say "2. DescribeInstances must report the EIP, not the auto-assigned address"
eq "$(public_ip_of "$INSTANCE")" "$EIP" "DescribeInstances reports the EIP"
say "3. stop the guest"
aws_spx "ec2 stop-instances --instance-ids $INSTANCE" >/dev/null
wait_for 180 "instance stopped" is_state "$INSTANCE" stopped
unset_or_none "$(public_ip_of "$INSTANCE")" "stopped instance reports no PublicIpAddress"
# The EIP's own object is the customer's and must outlive the stop — the exact
# opposite of what step 0b asserted for the borrowed address.
is_set "$(reserved_ip_ocid "$EIP")" "OCI still holds the reserved public IP for $EIP"
STILL=$(aws_spx "ec2 describe-addresses --allocation-ids $ALLOC --query 'Addresses[0].AssociationId' --output text" | tr -d '\r')
is_set "$STILL" "EIP stays associated across the stop"
say "4. force a start on a different node"
on "${NODE_IP[$START_NODE]}" 'sudo systemctl stop spinifex-daemon'
restore_daemon() { on "${NODE_IP[$START_NODE]}" 'sudo systemctl start spinifex-daemon' || true; }
trap 'restore_daemon; cleanup' EXIT
aws_spx "ec2 start-instances --instance-ids $INSTANCE" >/dev/null
sleep 20
NEW_NODE=$(node_of "$INSTANCE")
echo "now on $NEW_NODE (was $START_NODE)"
[[ $NEW_NODE != "$START_NODE" ]] || { fail "guest did not move; nothing to prove"; exit 1; }
say "5. the decisive checks"
wait_for 240 "EIP $EIP answers after the move to $NEW_NODE" reachable "$EIP"
if [[ -n ${PRIVATE:-} ]]; then
WANT=$(external_vnic_of "$NEW_NODE"); GOT=$(vnic_of_private "$PRIVATE")
eq "$GOT" "$WANT" "OCI carries $PRIVATE on $NEW_NODE's VNIC"
fi
eq "$(public_ip_of "$INSTANCE")" "$EIP" "still reports the EIP after the move"
say "6. restore"
restore_daemon
wait_for 120 "$START_NODE prunes its stale route for $PRIVATE" bash -c \
"! timeout 60 ssh -i $KEY -o StrictHostKeyChecking=no ubuntu@${NODE_IP[$START_NODE]} \
'ip route show dev spx-nat-host' | grep -q '${PRIVATE:-__none__}'"
(( FAILED )) && { echo; echo "RESULT: FAILED"; exit 1; }
echo; echo "RESULT: PASSED"
Reference in New Issue View Git Blame Copy Permalink