Service limits are per tenancy. vm-single-principal now builds in its own, so
the shape the other topologies share can have zero quota there, and the apply
then fails before anything about the instance principal is exercised.
Measured in ours: standard-e6-core-count is 0 available, against 13 for e5 and
10 for standard3. The first principal run died on 400-LimitExceeded for Core
Instance, which reads as a capacity problem and is a shape with no quota.
OCI_PRINCIPAL_SHAPE overrides it for that topology alone and defaults to the
shape every other topology uses, so nothing changes where quota already exists.
* fix: default the OCI helper to the DEFAULT profile
oci_env.py shipped our own internal Oracle compartment profile as the default for
every reader, with a second internal name as a fallback.
A fresh tenancy's config holds only `[DEFAULT]`, so the shipped default was a name
nobody else can have, and the fallback chain could land a deployment in whichever
tenancy a different profile happened to point at.
The profile is now `DEFAULT`, overridable by `OCI_CLI_PROFILE` or `--profile`, and a
named profile the config does not hold is an error rather than a fallback.
`DEFAULT` needs its own existence test, because configparser treats it as the
fallback for every other section and `has_section` denies it exists even when
`oci setup config` wrote nothing else.
`eip-move-test.sh` and the config table in the README carried the same name.
CI reads its credential from the environment rather than from a profile, so nothing
there referenced it.
* docs: look the OCI compartment OCID up by name
The guide asked the reader to paste `ocid1.compartment.oc1..xxx` three times, and
the `oci` CLI it already requires can resolve the name instead.
The lookup needs its own check. `oci iam compartment list --query ... | [0]` exits 0
and prints nothing when the name matches nothing, so an unset `$CID` would otherwise
reach Terraform and fail with an OCI error naming neither the compartment nor the
cause.
Two compartments in one tenancy can share a name, which the listing command shows
and the lookup cannot resolve, so the paste form stays as the answer for that case.
Records why the tenancy root is the wrong value: the step 2.1 policy is scoped to
whatever `$CID` holds.
* Revert "docs: look the OCI compartment OCID up by name"
This reverts commit 5443149cce.
* Reapply "docs: look the OCI compartment OCID up by name"
This reverts commit 1535d94c8a.
* fix: stop validate-topology.sh failing on macOS bash
`tf_principal_var` is empty on every run that does not pass `--instance-principal`,
and bash before 4.4 treats `"${arr[@]}"` on an empty array as an unbound variable
under `set -u`.
macOS ships bash 3.2, so the documented deployment command aborted at the tf_vars
assembly with `tf_principal_var[@]: unbound variable` before building anything.
Assigning the array was not enough, which is why the earlier `HOSTS=()` fix did not
cover this: the error is in the expansion, not the reference.
Both sites now expand only behind a length test, which is valid on every version.
`capture_journals` had the same shape for `HOSTS`, reachable when a run fails before
the apply.
Reproduced and fixed against bash 3.2.57 in a container, which emits the identical
message, and the script now runs to the terraform call on 3.2 for vm-single,
vm-single-principal and `--instance-principal`, including the `--keep` cleanup path
with no hosts.
* docs: record the bash 3.x requirement in the OCI scripts
These three are the scripts the OCI guide tells an operator to run, and macOS ships
bash 3.2, so a bash 4 construct in any of them breaks a deployment rather than a
test.
The constraint has now been rediscovered twice, through declare -A and mapfile, and
then through an empty array expanded under set -u.
Names the three traps at the top of each file so the next change does not have to
find them by failing on someone's Mac.
* fix: stream the workbook suite instead of hiding it in a file
The nine workbooks ran with stdout redirected to `workbooks.log`, so a run wedged
inside one printed nothing until the whole suite finished.
A CI job then sat silent for hours with no way to tell which workbook it was in, or
whether it was working at all.
The output is now teed, so the file the per-workbook table is parsed from is
unchanged while the run is visible live.
Each of the driver's own RUN/PASS/FAIL lines is stamped with elapsed seconds and the
workbook name, which is what identifies the slow one without reading the log.
`pipefail` carries the ssh exit status through the pipe, so a failed suite still
takes the branch that records the table and captures the journals.
* fix: grant the OCI policy the subnet verb AssignPrivateIP requires
An OCI node with a pool-mode external interface formed, reported healthy, and
then refused every launch wanting a public address with a 404
NotAuthorizedOrNotFound from AssignPrivateIP.
AssignPrivateIP calls CreatePrivateIp, which Oracle documents as needing
PRIVATE_IP_CREATE, PRIVATE_IP_ASSIGN, VNIC_ASSIGN and SUBNET_ATTACH, and
DeletePrivateIp as needing SUBNET_DETACH.
The policy granted use vnics, manage private-ips and manage public-ips, and no
subnet permission at all, so SUBNET_ATTACH was never held.
Only the subnets use verb carries it.
Measured on a live node as the instance principal: GetVnic and ListPrivateIps
succeed, GetSubnet and CreatePrivateIp both 404.
Authentication, the dynamic group's matching rule and the compartment were all
correct, so the credential was authentic and unauthorised.
The documented operator policy for the API-key route carried the same three
statements, so neither credential route could allocate.
A node now proves its own credential at pool start by making one read in each
permission class, and logs the missing verb if one fails.
The topology gate waited for "resolved the external VNIC", which comes from the
metadata service and needs no IAM policy, so an unauthorised node passed the
gate and failed an hour later inside a workbook as capacity.
It now waits for the authorisation line and fails immediately on a refusal.
Reads rather than a real assign-and-release round trip: a round trip also fails
on a full subnet, and reporting that as an authorisation fault sends an operator
to IAM for a capacity problem.
* docs: say what a principal run costs in a tenancy with no dynamic group
No principal topology can pass there, and the only previous hint was one clause
about OCI_INSTANCE_PRINCIPAL needing the group to exist.
* test: cover the subnet read and the authorisation verdict
Diff coverage was 39.1% against a 60% gate. Three of the four gaps were real.
GetSubnet on the SDK-backed client had no test, so the request shape was
unverified: addressing the collection rather than one subnet by OCID would
answer from a different permission and report a credential as working when it is
not. Also pins that a sparse body is not an error and that the unauthorised 404
arrives as ErrNotFound, which is what stops the check retrying a verdict that
will never change.
The fake's GetSubnet was exercised only from the ocinet tests, and coverage is
recorded per package under test, so it read as zero-covered.
reportAuthorisation now has a test asserting the exact log text, because
validate-topology.sh greps for it and a rename would quietly turn the gate into
one that never fires.
The six lines still uncovered are inside FromPoolConfig, which needs a live
JetStream to reach. Diff coverage 87.0%.
* fix: refuse a run whose tfvars file contradicts the compartment it was given
Terraform ranks *.auto.tfvars above TF_VAR_*, so a leftover file wins silently
over an operator targeting a different tenancy by environment, and the guide
tells every operator to write one.
Hit for real: a stale terraform.auto.tfvars naming a different tenancy's
compartment and a bare-metal shape overrode TF_VAR_compartment_ocid, and the
plan was for 11 resources in a compartment nobody had chosen. Only a dry run
caught it.
The run now stops and names both values rather than building.
* fix: collect orphaned OCI public IPs and name the limit that was hit
A reserved public IP attached to nothing is on no VNIC, so the reconcile
that exists to find leaked addresses could not see the ones that matter.
It also ran only at startup, while an address is leaked on release and on
teardown. Twenty accumulated in one compartment and exhausted the
tenancy's limit of 50, after which every launch needing a public address
aborted.
Reconcile now lists reserved public IPs in the compartment and collects
those that carry our prefix, are attached to nothing, are named by no
binding and are older than a grace period. All four are needed: an
allocated but unassociated Elastic IP is detached too and is the
customer's to keep, and a young detached address is what another node's
CreatePublicIP looks like until it attaches. A pool with no binding
record is not swept at all, since that is indistinguishable from a store
that cannot be read.
The pass now repeats on a ticker rather than running once at startup.
Both OCI ceilings also resolved to InsufficientAddressCapacity, which
tells an operator the region is out of addresses when the fix is to
release addresses they already hold. AWS separates them, so a tenancy out
of reserved public IPs is now AddressLimitExceeded and a full VNIC is
PrivateIpAddressLimitExceeded.
* feat: run the OCI principal e2e in a tenancy that can hold its dynamic group
vm-single-principal needs the tenancy's dynamic group, and creating one is a
tenancy-root write that a shared test tenancy does not grant us. The topology
could never pass there however correct the code was, so it now takes its own
credential from OCI_PRINCIPAL_*, all six or none. A partial set is refused
before the build rather than at the apply, because it would build one tenancy's
instances against another's compartment. With none set the topology is recorded
SKIPPED instead of spending a provision to fail.
The sweep takes the same credential, or a leftover principal deployment is
destroyed against a tenancy that never held it and reported clean.
Two defects found while proving it:
--dry-run reported FAIL on every topology with a tree source, because the dry
run skips the build and then refuses the tarball it would have produced. The
flag operators are told to run first could not pass.
The dry run truncated its plan to the last 30 lines, which is the outputs block.
That dropped the resource count and every compartment and shape line, so the
one check a dry run exists for could not be made from its output. It now writes
the whole plan and prints the summary.
Removing a deployment is documented as leaving public IPs behind. Spinifex
allocates them on the running node, so they are in no Terraform state and
destroy cannot see them; twenty accumulated and exhausted a tenancy's limit,
which then refused every launch in every compartment.
* fix: canonicalise the Opc-Next-Page header in the pagination test
CI's linter rejects a non-canonical header name and the local preflight does
not, so this passed here and failed there.
* docs(oci): offer both credential routes, and default the guide to the release
Step 2 was one route only, the scoped API key, with the instance principal
mentioned in a TIP at the end of it. It is now "2. OCI Credentials for
Spinifex" with the two routes as 2.1 and 2.2, a table to choose between them,
and the instance principal first because it leaves no key on any node.
The guide also assumed the dev branch throughout. `git switch dev` and
`--channel dev` are now an optional note for installing a pre-release, and the
documented path is the published release.
* test(oci): add a vm-single-principal topology, and cover the credential branch
Instance-principal authentication was broken for a release while the OCI
nightly stayed green, because the run's two axes never crossed: every topology
that built the address allocator authenticated with an API key, and every run
that authenticated as the instance was dispatched with no external pool, so the
allocator was never constructed and the credential never used.
vm-single-principal closes that. It is vm-single with the credential fixed by
the topology rather than by a separate switch, so the name alone decides it, and
pairing it with --no-external-pool is refused rather than run: without the
allocator there is nothing for it to exercise. It joins the nightly's default
list, and naming it alone is the way to test this one thing in a quarter of the
full run.
A unit test covers the same branch without a tenancy. newClient now reaches its
two constructors through package variables a test can swap, which is the only
way to observe the choice off an OCI instance - both real constructors need
either an instance identity or a key file on disk.
* fix(oci): let the tfvars choose the credential, and stop the guide hardcoding paths
The deploy driver passed instance_principal as a -var, which outranks a
tfvars file, so a deployment that set it there had it silently replaced with
off and the nodes went looking for a key. Same defect class as the shape and
node count, and the same fix: the flag is a choice about one run and still
outranks everything, and without it the deployment's own tfvars decides.
The driver now reads instance_principal from terraform.tfvars and
*.auto.tfvars for its pre-apply gates, taking the last assignment as Terraform
does, and checks that guess against a new Terraform output once the apply has
run. They have to agree, because the credential handoff is skipped on the
strength of it and a wrong answer leaves a node with no credential at all.
The guide now carries one export rather than two more steps. SPX_PRINCIPAL is
set in step 2.1, expanded into the tfvars step 4 already writes, and read back
from there in step 5, so an instance-principal deployment runs the same
command as a key-based one.
Getting into the node needed an address copied out of the log. Every topology
now writes its hosts file, not only the multi-node one that hands it to
install-node.sh, and the run prints the ssh and destroy commands when --keep
leaves the infrastructure up. The guide exports TOPOLOGY once, so the paths
read the same for one VM, three, or bare metal.
* fix(oci): print the ssh key path as ~ rather than expanded
The --keep line read ssh -i /home/you/.ssh/oci-spx, which is the invoking
user's home and differs on every machine. Collapsed to ~ when the key is under
$HOME, left alone when it is not.
Admission clamps on live MemAvailable, so on a loaded box running tests in
parallel any daemon test that allocates could see no capacity. TestMain now
disables the live-memory gate; its own tests inject a reader.
admin init and account create wrote a single profile endpoint_url, so aws s3
reached the gateway on :9999, which does not serve S3. Profiles now carry a
services section pointing s3 at the predastore gate, and the INI rewrite keeps
nested settings instead of flattening them.
Servers reply before ending their span, so the client can return before
the span reaches the recorder. SpansByKind now polls until a server span
has ended and fails the test if none does within 5s.
The dev release tagged itself with git describe plus -dev, and describe
resolved to the previous dev tag, so every tag appended to the last until
the tarball name passed the 255-byte filename limit. Describe against
release tags only, here and in the Makefile and bare-metal VERSION.
* fix(oci): run the deploy driver on the bash macOS ships
`validate-topology.sh` died on line 41 with `bm: unbound variable` on a Mac. macOS
ships bash 3.2, where `declare -A` is an indexed assignment and the subscript is
read as an arithmetic expression, so `[bm]` dereferenced an unset variable and
`set -u` ended the run. `mapfile` further down would have failed next.
The guide names macOS as a supported workstation, so the driver now uses neither:
the per-topology defaults are a `case` in a function, and the host list is a read
loop over a captured string. The output is captured before the loop so a failed
`terraform output` fails there rather than silently producing an empty host list.
Verified: all three topologies resolve their shape and node count, and an unknown
topology still names the valid set.
* docs(oci): end the OCI deploy where the install guides end
The deploy command now carries --skip-workload, so the guide stops at a formed
cluster and a working AWS call rather than at a workbook run, and hands off to
Setting Up Your Cluster the way every other install guide does. A step to SSH to
the node was missing between the deploy and the verification call.
Three things this corrects while the section was open. The expected output block
now matches the command it documents, including the --skip-workload line that
ends the run. The --channel dev callout said OCI support was missing from the
deploy driver, when what is missing is an OCI address allocator in a published
release. The verification output was fenced as json while holding a shell
session, which is neither valid json nor the thing being shown.
The note on probing a guest's public address from outside the VCN moves to the
handoff section, since that is now where a guest first gets launched.
`validate-topology.sh` passed the topology's shape and node count as `-var`,
which is the strongest source Terraform reads, so a `compute_shape` or
`node_count` in `terraform.auto.tfvars` was silently discarded. The OCI guide
told the reader to write both into that file and then to run the driver, so a
deployment asking for `VM.Standard.E5.Flex` got `VM.Standard.E6.Flex` and said
nothing about it. Three of the five shapes the guide recommends were unreachable
by the command it gives, including both bare-metal ones other than the shape CI
happens to use.
They now go in as `TF_VAR_compute_shape` and `TF_VAR_node_count`, the weakest
source, so the tfvars file wins and a deployment needs no flag to size itself.
CI is unaffected: `*.auto.tfvars` is gitignored, so a runner's checkout holds no
such file and a named topology still resolves to the same values every run.
`instance_principal` stays a `-var`, because `--instance-principal` is a choice
about the run rather than about the infrastructure's size and a stale tfvars must
not quietly contradict it.
Because the count can now come from the file, the driver no longer presumes it.
The number of addresses in the `hosts_file` output is the authority for the
single-node init path and the readiness gate, a divergence from the topology's
default is logged rather than fatal, and a new `compute_shape` output lets the run
report the shape it built instead of the one it asked for.
In the guide: the tfvars heredoc is unquoted so `$CID` expands, with a `grep` to
confirm it did, and `node_client_cidr_allow_list` is bracketed, since it is a
`list(string)` and the bare CIDR was an HCL syntax error.
* docs(oci): rewrite both OCI guides in the house voice
Seventy-five em dashes across the two guides, with four of them in headings where GitHub and the docs site slugify them differently and a cross-link resolves on neither. Recast as commas, colons and separate sentences.
The silent-credential failure was stated five times as the same triple, "forms, passes every health check, and then refuses every launch". It now reads in full once, under Credentials, and the other four refer back to it instead of echoing the skeleton.
Also broken up: the parallel stack in "Why run Spinifex on OCI", several five-sentence paragraphs, and a handful of clauses that leaned on a dash where a full stop was clearer.
No technical content changed. Every inline-code token is byte-identical before and after, every fenced block is unchanged apart from one comment line, and the four reworded headings carry no inbound anchor links. The docs build reports zero warnings for either page and both render at the same size as before.
* docs(oci): open the install guide with what Spinifex is, as the other install guides do
* docs(oci): ship the credential installer instead of asking for a hook
The guide told the reader to write their own executable to put the OCI API key
on each node, which is work nobody should have to do to install a product, and
the step was unclear about where the scoping actually happens.
`scripts/terraform/oci-spx/spx-oci-config.sh` now does it. It takes
`validate-topology.sh`'s hook contract, `hook <ssh-key> <host>...`, so it drops
straight into `--credential-hook` and is also runnable on its own against an
existing cluster.
It resolves the credential in four steps and takes the first complete one: the
`OCI_SPX_*` environment variables, the `[spinifex]` profile in `~/.oci/config`,
`~/.oci/oci_api_key_spx.pem` with the Terraform profile's identifiers, then the
Terraform credential itself with a warning that it is wider than the allocator
needs. Steps 2 to 4 go through `scripts/oci_env.py`, the same resolver the apply
uses, so the credential on the nodes and the one that built the infrastructure
cannot come from different profiles. The fingerprint is checked against the key
before anything is written, because a mismatched pair is otherwise an OCI 401 at
the first allocation, long after the deploy has reported success.
The node's filename stays `oci_api_key.pem` while the workstation's is
`oci_api_key_spx.pem`: the suffix exists only to keep the Spinifex key off
whatever `[DEFAULT]` uses, and a node has no other OCI key. Nothing reads the
name anyway, since the daemon learns the path from `key_file` inside
`/etc/spinifex/oci/config`.
Key material reaches the nodes over SSH, in the remote shell's stdin rather than
its arguments where `ps` would expose it, and goes into neither user-data nor
Terraform state. `--credential-hook` still accepts any executable with that
argument shape, so a tenancy that keeps credentials in a vault can substitute
its own.
Also in the guide: the policy statements name the compartment by `compartment
id` rather than by name, which is the form that resolves from a compartment the
policy is not attached at; `$CID` is consistently a compartment OCID; and the
table of contents matches the headings again.
* docs(oci): publish both OCI guides on the docs site
The site extracts only four H2 sections and drops everything else, so both guides
would have published nearly empty: the deployment guide matched none of the four,
and the reference doc only its troubleshooting. Insert the section markers the
other published docs already use, keeping every heading at the level it had so no
anchor moves.
The architecture reference becomes oci-architecture/README.md, because a page is
always <slug>/README.md and a flat URL takes the last path segment -- plain
"architecture" would have published at /docs/architecture.
Two headings lose an em dash. GitHub renders it as a double hyphen in the
fragment and the site's slugify drops it entirely, so the two disagreed and the
cross-links resolved on neither.
* docs(oci): the OCI guides sit in their own Cloud Install section
Their category frontmatter named Install, which is the section they are moving out of.
attachedPolicies resolved each attached ARN and, on any error, logged a warning
and skipped it. A transport failure is indistinguishable from a missing record
there, so one timed-out read reported the policy as not attached.
Two paths then read the same field and disagreed. ListAttachedRolePolicies
returned the filtered list, so a client concluded the attachment was gone and
dropped it from its own state without calling DetachRolePolicy, while DeleteRole
checked the unfiltered field and refused for as long as the ARN was recorded.
Nothing could clear it through the API. This is what wedged the ecs-quickstart
workbook's teardown on DeleteRole(ecsInstanceRole) with a 409.
The state being guarded is also unreachable: DeletePolicy refuses while any
principal attaches the policy and fails closed if the attachment scan errors, so
no API sequence produces a principal holding an ARN with no policy record. AWS
cannot return a short list here either -- an attached policy always exists, and
an internal failure is a 5xx.
AWS-managed ARNs are unaffected. They are deliberately stored without a record
and are still reported from the ARN itself, above the lookup.
Closes mulga-tmk5e
* feat: run every Terraform workbook on OCI, and split the guide in two
The OCI topology driver ran the shared driver's default five workbooks, which left ECS and all three EKS variants untested on every platform even though their assertions already existed. It now passes an explicit nine and imports the three appliance images they need — RDS, ECS and EKS each boot their own, found by a spinifex:managed-by tag and never by name.
demo-app stays out because it is not a workbook: it has no .tf at all, being the container image the EKS workbooks' nested workloads modules deploy, so listing it would fail on a missing root module rather than test anything.
The guide is now two documents. oci-integration/README.md is the end-user path — six steps, deploy first, and it closes the three prerequisites nothing documented before: the clone, the ref, and the SSH keypair the driver refuses to start without. architecture.md holds the sizing, the IAM policy, every variable, the three filtering layers and the troubleshooting.
Its firewall advice was also backwards. node_client_cidr_allow_list is an OCI security-list rule over the whole public subnet, and a guest's public address is a secondary private IP on the node's own VNIC, so narrowing it to the operator's address cuts internet ingress to every guest rather than protecting the node. The hardening section now arms the nft policy and narrows SSH through custom.nft, which is the layer that can tell a node's listener from a guest's.
The deploy step names --channel dev rather than a pinned tag. The dev tag string grows with every cut, so a pinned example goes stale within the week.
* fix(oci): the guide could not work on a fresh tenancy, because the profile name was ours
oci_env.py bridges ~/.oci/config into TF_VAR_* — the provider takes the tenancy, user, fingerprint, key and region as Terraform variables and reads no config file itself — but the profile it looked for was the reference tenancy's, and validate-topology.sh passed no way to change it. A fresh account's config holds [DEFAULT], so every deploy would have stopped at "OCI profile 'apacanzset03child03' was not found" before building anything.
The profile is now resolved as --oci-profile, then $OCI_CLI_PROFILE, then the reference tenancy, then DEFAULT. A name passed explicitly and absent is an error listing the profiles present, never a fallback: the failure worth preventing is building a cluster in whichever tenancy happened to be first. The reference tenancy still outranks DEFAULT, so existing runs are unchanged, and CI is untouched because it takes the environment route rather than a profile.
has_profile exists because configparser treats DEFAULT as the fallback for every other section rather than as a section, so has_section denies it exists even when it is the only thing in the file — which made the first version of this fallback do nothing for exactly the case it was written for.
Three documentation corrections found while checking the guide against the code. The IAM policy is ten operations and needs "use vnics", not "read vnics": registering a secondary private IP is an operation on a VNIC, so OCI checks VNIC_ASSIGN, which lives in the use verb, and with read the node forms and looks healthy and every allocation fails NotAuthorizedOrNotFound. These are now the same three statements instance-principal.tf grants, which is the only version of the policy that has been exercised.
The guide also promised a worked example of the credential hook that does not exist and should not: nothing here distributes a credential. It now states the contract instead — the argument shape, the two files, and that a non-zero exit stops the deploy before anything is built. And the dev channel's rate limit is documented with the tag as its remedy, rather than left in the driver's help where a reader will not see it.
* fix(ci): a failing workbook's log never reached the artifact, so only the runner had it
run-tofu-examples-e2e.sh writes its assertion output to the per-topology workbooks.log in the state directory. "Sweep anything left behind" deletes those directories and runs before "Upload the run", so the upload's three oci-state/.validate-* patterns matched nothing and said so to nobody. The first nine-workbook run lost the one thing it was dispatched to find: ecs-quickstart failed on vm-single and the reason existed only on the runner, which does not resolve from a workstation.
A collect step now copies *.log, *.txt and *.tsv into the artifact directory before the sweep, prefixed by topology so two topologies cannot overwrite each other. The workflow already records that "Publish the result tables" sits ahead of the sweep for this reason; the logs were simply never given the same treatment.
The dead patterns are removed rather than left. A path that cannot match is worse than no path, because it reads as protection that is present — which applies to the credential-hook exclusion too: it now lives in the collect step, which is the only thing that writes into the artifact tree, so the file cannot reach an artifact rather than being filtered out of one.
* fix(ci): a state directory kept the last run's logs, and the artifact published them as this run's
Collecting the per-topology logs surfaced the next defect immediately: a vm-single-only run produced an artifact holding bm-workbooks.log and nine vm-multi node logs, because those state directories still held the previous run's output. Anyone reading that artifact would conclude three topologies ran when one did, which is worse than the missing log it replaced.
Two causes, one each side. validate-topology.sh never cleared its own logs, so a topology's directory accumulated them; it now removes *.log, nodes.txt and cloudinit-*.txt as it starts, before the first line of the new run, and never the tfstate, which is how the sweep finds hosts to destroy. The globs are named rather than a find, so what goes is readable at a glance, and :? stops the shell instead of expanding to /*.log if STATE_DIR is ever unset.
That alone does not cover a topology the run skipped, whose directory is never entered. So the collect step now takes only the topologies this run drove, read back from the per-topology log the harness writes rather than from the dispatch input -- a run that names no topology gets the harness's default, and reading that here would be a second copy of the list to disagree with the first.
Also proved by this: ecs-quickstart passes on a freshly built vm-single, 172s, where it failed at 293s as the seventh of nine. The failure is order or load dependent on one 32 GB node, not a property of the workbook.
* fix(ci): a failed workbook run left no node journal, so the cause was unreadable
The workbook log says which API call failed; only the journal says why, and the
nodes are destroyed minutes later. An IAM teardown failure on the nightly came
down to one swallowed warning that existed nowhere else, and by the time the
artifact was read the host it was logged on had been torn down.
Warnings and above first, since that is where a swallowed error surfaces, then a
bounded tail for context. Unbounded, a multi-hour suite's journal would dwarf the
rest of the artifact. The existing state-directory clear keeps these from going
stale between runs, and the collect step already globs *.log.
* feat(oci): separate using an instance principal from creating one
The allocator needs OCI credentials at runtime, and a node with none forms,
passes every health check, then refuses every launch wanting a public address.
Instance principal is the credential route that puts no key material on a node,
but it needs a dynamic group and a policy, and those are tenancy-root resources
the compartment-scoped user the rest of the configuration runs as cannot create.
enable_instance_principal conflated the two, so the better credential route was
only reachable by a tenancy admin. instance_principal now takes off, adopt or
create: adopt selects the credential and creates nothing, because the matching
rule covers the whole compartment and so names no instance to reference.
setup-identity.sh owns the create case, with its own state. validate-topology.sh
destroys its state at the end of every run, so holding tenancy resources there
would let a teardown delete the tenancy's policy.
Also package install-node.sh and smoke-test.sh, so a cluster can be formed from
a curl | bash install with no checkout. smoke-test.sh is not optional: formation
scp's it from its own directory for the post-formation verification.
* feat(oci): take the credential handoff from an operator hook, and widen the workflow name
An API-key deployment needs a credential on each node, and nothing in this
repository can supply one: a key in user-data is readable from instance metadata
for the life of the instance, and this host runs other people's guests. A manual
install does not survive either, because a topology destroys itself on success.
So the harness calls an operator-owned executable after formation and before the
pool, the only window where a node has /etc/spinifex and has not started the
allocator. Instance principal needs no handoff, so the hook is skipped there.
The workflow is renamed for the vendors it will cover rather than the first one.
Nothing is orphaned in the Actions UI: it has never been on the default branch
and has never run, so there is no history attached to the old filename.
* docs(oci): the API-key path needs the config file, not just the PEM
oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node
holding only the key forms and then fails every allocation with no stated cause.
* fix(ci): the runner context is not available in a job-level env block
A workflow referencing it there is rejected before any job starts, which is why
the rename produced a failed run attributed to the push. The reference had never
been validated: a dispatch-only workflow that is never dispatched is never parsed.
* fix(deps): bump containerd to v2.4.1 for GO-2026-6597
The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change.
* fix(e2e): probe guest public addresses from outside the cloud's own network
A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet.
The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there.
Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them.
* ci: read the Go version from go.mod in the release workflows
Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod.
* feat: publish the OCI topology results as tables on the run page
A green tick on one job with 4,000 lines of log behind it is confidence for
whoever wrote the harness and nothing to anybody else. Someone assessing whether
Spinifex works on their cloud should be able to see that it formed and served
guests on a single VM, on three VMs and on bare metal without opening a log.
record now takes <gate> <status> [detail] and writes results.tsv beside the
human block, so a gate's status is a field the script sets rather than a regex
over prose that reads fine and matches wrong. record_workbooks turns the
driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing
branch too: which four passed is most of what the fifth failing means.
e2e-oci.sh --summary-only renders both tables to stdout and to
GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep,
because the sweep destroys the state directories the tables are read from. A
skipped gate gets its own mark, and a topology that never ran says so: the one
way a results table can do real harm is a reader taking an absent row for a
passing one.
* docs(oci): cut the deploy-by-hand appendix, and validate the config for real
The appendix was 444 of 1014 lines restating an install the Terraform already
does, which made the document read as though deploying by hand were a normal
route. It is not: the configuration builds the VCN, the volumes and both VNICs
per node and renders the cloud-init, in the same order every time. What is
genuinely OCI-specific is four things, each of which already has a section, so
the replacement names those and sends everyone else to the single-node and
multi-node install guides.
Verify end to end told the reader to ping a guest's public address from the
host, which cannot work: a VCN does not route its own public addresses back into
itself. That is the finding this branch is built on and the document was
asserting the opposite. It now says where to probe from and why, and carries the
allocator's `resolved the external VNIC` check, which existed only in the cut
appendix and is the one line proving the credentials and compartment are right.
`spx config validate` is not a command -- spx has admin, get, service, top and
version -- so that line could only ever fail and its output and status were both
discarded. It is replaced by parsing the TOML, which is what was wanted: the
remap and the pool append are textual edits, and a broken result would otherwise
surface as every service failing to start with nothing saying why. The remap now
checks its own substitution count for the same reason, since re.sub with no
match rewrites the file unchanged and exits 0.
Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is
per step, so appending from the run as well as the publish step would have put
two copies of every table on the run page.
* fix(oci): check the credential hook before building anything
The hook is a path from a CI variable, and whether it exists on the runner is
knowable before Terraform runs. Checked where it was used, a wrong path cost the
apply, the install and the formation first -- forty minutes and a bare-metal bill
to be told a file is missing. Nothing about the check depends on the apply, so it
moves to the top, where --dry-run reaches it too.
* fix(oci): the summary must say what was built and what was not tested
The first CI run headed its tables "Published release `latest channel`" over a
build of a feature branch, which is the one error in a results table that would
mislead a reader who trusted it. The header now states a tree build as a tree
build, and names how the allocator authenticated.
A run with no external pool says so in the header rather than leaving it to a
SKIPPED row further down. Public-address allocation is the claim a green table is
most likely to be read as making, and it is exactly the claim such a run cannot
support. Both flags are passed to the publish step too, so the summary rendered
after a failure describes the same run as the one that failed.
Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an
external evaluator installs a specific build, and five is about a week of them --
short enough that the tag someone was handed stops resolving while they are still
using it.
* docs(oci): the workbook table is a snapshot, and the nightly is the current answer
It is dated against a build five releases of work ago, and the nightly now runs
the same workbooks on all three topologies and publishes its own table. Say which
one to read rather than letting a stale table be taken as current.
* ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file
A workflow's displayed name is captured when GitHub first registers it and is not
refreshed afterwards. Both OCI entries were registered from a run that could not
be parsed -- a job-level env block referencing runner.temp -- so both list as
their path, which reads as a missing name: key and is not one. The 2026-10-02 run
was green on all three topologies and the entry still read as a path, with
updated_at unchanged from the broken first run, so fixing the file is necessary
and not sufficient.
A new path is a new registration, and its first run reads a valid file. Dropping
-nightly is also more honest: there is no schedule: here, because GitHub
registers one only from the default branch, so a cron could never run dev and a
feature branch as separate nightlies.
* Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file"
The rename makes the workflow undispatchable. `gh workflow run` resolves a
filename against the default branch, and the raw dispatch endpoint returns 404
for a path GitHub has not registered -- checked twice, several minutes apart, so
it is not an indexing delay. The old name works only because it is already
registered, and an unrunnable workflow is a worse problem than a mislabelled one.
The sidebar label needs a fresh registration, which a rename cannot provide from
a feature branch. It corrects itself when the file reaches main, where GitHub
re-reads it; the alternative is deleting the dead registrations' runs, which is an
operator's call.
* fix(ci): never publish the credential hook's output as an artifact
The hook is the operator's own executable, and the one thing known about it is
that it handles a private key. Its stdout and stderr are captured to
credential-hook.log, which the artifact's *.log glob was collecting into a
downloadable archive with a fourteen-day retention.
Excluded rather than trusted to have been quiet. A hook that prints its key once
while someone is debugging it should not thereby publish it.
* fix(oci): the hairpin probe measured nothing on one node, and claimed too much
With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node
connected to itself -- which always succeeds. It reported "present" on vm-single
and bm for that reason and not as a measurement. It now needs two nodes and says
so otherwise.
The claim it carried was also wrong. "A VCN does not reflect its own public
addresses back into itself" was read off a workstation run whose
node_client_cidr_allow_list was narrowed to one office address, which makes a
node's own public address a non-permitted source: the refusal that follows is
identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe
reports the hairpin present between two nodes, so the original reading was
confounded and is retracted.
None of this changes the remedy. A customer reaches a guest from outside the VCN,
which is where the proxy's egress is, and probing from there is correct whichever
of the two causes is in play -- 15 of 15 workbooks pass through it on all three
topologies. What changes is what the documentation may assert: the guide now names
both causes as indistinguishable rather than stating one as fact.
* docs(oci): a reader could not run the first command, because nothing told them to clone
The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory
no step had obtained. It also never mentioned the SSH keypair the driver refuses to
start without, nor that the configuration is not on main or in v1.21.0 yet, so all
three had to be inferred from the deploy failing.
Restructured so the deploy comes first and reads as four steps and one command:
get the configuration, choose one node or three, write the inputs, run it. The
credential decision stays before step 4, because it changes an argument there and a
node without one forms, looks healthy, and then refuses every launch that wants a
public address. Everything else -- why OCI, sizing, the architecture, the IAM
policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under
Reference, unchanged in substance.
Adds what a successful run prints, naming `allocator ready` as the line to look for:
every line above it appears on a node that cannot allocate, and that is the one
failure the run does not announce.
Also drops an empty verdict.txt committed from a local run. The directory the guide
now tells customers to work in should not ship one run's artifacts.
The installer served by install.mulgadc.com is the latest release's, so it
predates the --channel flag the harness was passing and would have died with an
unknown option. Document the same form in both install guides.
* fix: size the volume lease against the stall the host actually produces
A guest lost its disk whenever JetStream writes failed for fifteen seconds,
which is what one NVMe I/O timeout on the node holding the JetStream leader
produces. The budget composed badly rather than any single number being wrong:
a confirmation could already be volumeLeaseRenewInterval old when an outage
began, and validity expired volumeLeaseValidity after it, so an outage covering
[15s, 30s) surrendered a volume no peer had taken. Measured outages are 10-15s,
so the lease was sized at the edge of a routine event. On one cluster thirty
eight renewal timeouts produced six surrenders and four destroyed guests.
Validity rises to 40s and the renewal interval falls to 10s, taking the
tolerated outage from fifteen seconds to thirty. The validity-to-TTL gap is now
named by volumeLeaseAcquireMargin instead of being implied by two constants:
nothing may acquire until the server-side bucket TTL ages the entry out, so the
margin only has to cover scheduling and request latency on this side.
The renewal timeout rises above the stall bound, so a renewal that begins inside
a stall is answered when it clears rather than being certain to fail. That is
not a constant on its own. renewLoop was one goroutine, so a blocking Update
delayed the validity check by the whole timeout: measured on the old loop with
the new timeout, a holder surrendered 51s after its last confirmation, eleven
seconds past validity and six past the server TTL, which is a window where a
peer may already hold the volume. The renewal now runs on its own goroutine with
one call in flight at a time, the loop keeps testing validity underneath it, and
it waits for the renewal before closing done so release still reads a settled
revision.
deviceStallBound documents nvme_core.io_timeout as a property of the host. It is
not configured or applied here, only asserted against, because it describes the
machines rather than the product.
The sizing is a probability, not a guarantee. An outage longer than validity
still surrenders, and the fence still destroys the guest instead of revoking the
export and leaving it paused, which is the change that would make outage length
stop mattering.
* fix: keep a fenced instance out of every automatic relaunch
A fence does not seal, so the volumes it leaves behind are a state nothing has
shown is mountable. Two paths relaunched one anyway. The restore path consults
the restart window, which forgives on a clock, so a fence days old reports a
full budget and the next daemon start remounts it. The recovery reconciler
excludes a storage fault but not a fence, and a fence kills the export instead
of pausing the guest, so the record publishes no I/O error and reads as an
ordinary desired-running instance on a node that has died.
Both now read vm.VolumeFenced off the StateReason the fence already writes. The
refusal is narrower than StateError on purpose: a recovery that failed for its
own reasons is what the restart budget exists for.
Also separates the two storage quantities the lease was sized against. One
stalled I/O lasts deviceStallBound, which is what a renewal timeout has to
exceed; what the lease has to survive is a run of them, now named
jetstreamOutageBound. They were conflated, so raising the device constant would
have licensed a validity past the server TTL.
A surrender can only ever report validity, so the length of an outage was
unobservable. Renewals now count consecutively and report outage_ms once they
recover, which measures the windows the lease survives.
install.mulgadc.com has served a dev channel since it was written -- setup.sh
understands INSTALL_SPINIFEX_CHANNEL=dev and the Worker resolves it to the newest
GitHub prerelease -- but nothing ever produced one, so /download/dev/amd64
returned 404. This is the producer.
A separate workflow rather than conditionals in release.yml. A dev build is a
different artifact, not a release with flags off: amd64 only, no ISO, no system
images, no bare-metal gate, and nothing on a self-hosted runner. The sibling
repositories are taken at dev rather than their default branch, because dev of
spinifex against main of the others is a pairing nobody runs.
The tag is the describe string plus -dev, which release.yml's own convention
already treats as a prerelease, and which makes the commit recoverable from the
version alone. The flag is set at creation and then the published state is
asserted: a prerelease must never become the latest release, because latest is
what an unqualified curl | bash installs.
setup.sh gains --channel and --version so the instruction is a flag rather than
an environment variable, and the OCI harness gains OCI_CHANNEL so a nightly can
test the path the contacts are actually given.
Julian Sommer