3 Commits

Author	SHA1	Message	Date
Ben Duncan	8409799cbf	fix: default the OCI helper to the DEFAULT profile (#1174) ... * fix: default the OCI helper to the DEFAULT profile oci_env.py shipped our own internal Oracle compartment profile as the default for every reader, with a second internal name as a fallback. A fresh tenancy's config holds only `[DEFAULT]`, so the shipped default was a name nobody else can have, and the fallback chain could land a deployment in whichever tenancy a different profile happened to point at. The profile is now `DEFAULT`, overridable by `OCI_CLI_PROFILE` or `--profile`, and a named profile the config does not hold is an error rather than a fallback. `DEFAULT` needs its own existence test, because configparser treats it as the fallback for every other section and `has_section` denies it exists even when `oci setup config` wrote nothing else. `eip-move-test.sh` and the config table in the README carried the same name. CI reads its credential from the environment rather than from a profile, so nothing there referenced it. * docs: look the OCI compartment OCID up by name The guide asked the reader to paste `ocid1.compartment.oc1..xxx` three times, and the `oci` CLI it already requires can resolve the name instead. The lookup needs its own check. `oci iam compartment list --query ... | [0]` exits 0 and prints nothing when the name matches nothing, so an unset `$CID` would otherwise reach Terraform and fail with an OCI error naming neither the compartment nor the cause. Two compartments in one tenancy can share a name, which the listing command shows and the lookup cannot resolve, so the paste form stays as the answer for that case. Records why the tenancy root is the wrong value: the step 2.1 policy is scoped to whatever `$CID` holds. * Revert "docs: look the OCI compartment OCID up by name" This reverts commit 5443149cce. * Reapply "docs: look the OCI compartment OCID up by name" This reverts commit 1535d94c8a. * fix: stop validate-topology.sh failing on macOS bash `tf_principal_var` is empty on every run that does not pass `--instance-principal`, and bash before 4.4 treats `"${arr[@]}"` on an empty array as an unbound variable under `set -u`. macOS ships bash 3.2, so the documented deployment command aborted at the tf_vars assembly with `tf_principal_var[@]: unbound variable` before building anything. Assigning the array was not enough, which is why the earlier `HOSTS=()` fix did not cover this: the error is in the expansion, not the reference. Both sites now expand only behind a length test, which is valid on every version. `capture_journals` had the same shape for `HOSTS`, reachable when a run fails before the apply. Reproduced and fixed against bash 3.2.57 in a container, which emits the identical message, and the script now runs to the terraform call on 3.2 for vm-single, vm-single-principal and `--instance-principal`, including the `--keep` cleanup path with no hosts. * docs: record the bash 3.x requirement in the OCI scripts These three are the scripts the OCI guide tells an operator to run, and macOS ships bash 3.2, so a bash 4 construct in any of them breaks a deployment rather than a test. The constraint has now been rediscovered twice, through declare -A and mapfile, and then through an empty array expanded under set -u. Names the three traps at the top of each file so the next change does not have to find them by failing on someone's Mac. * fix: stream the workbook suite instead of hiding it in a file The nine workbooks ran with stdout redirected to `workbooks.log`, so a run wedged inside one printed nothing until the whole suite finished. A CI job then sat silent for hours with no way to tell which workbook it was in, or whether it was working at all. The output is now teed, so the file the per-workbook table is parsed from is unchanged while the run is visible live. Each of the driver's own RUN/PASS/FAIL lines is stamped with elapsed seconds and the workbook name, which is what identifies the slow one without reading the log. `pipefail` carries the ssh exit status through the pipe, so a failed suite still takes the branch that records the table and captures the journals. * fix: grant the OCI policy the subnet verb AssignPrivateIP requires An OCI node with a pool-mode external interface formed, reported healthy, and then refused every launch wanting a public address with a 404 NotAuthorizedOrNotFound from AssignPrivateIP. AssignPrivateIP calls CreatePrivateIp, which Oracle documents as needing PRIVATE_IP_CREATE, PRIVATE_IP_ASSIGN, VNIC_ASSIGN and SUBNET_ATTACH, and DeletePrivateIp as needing SUBNET_DETACH. The policy granted use vnics, manage private-ips and manage public-ips, and no subnet permission at all, so SUBNET_ATTACH was never held. Only the subnets use verb carries it. Measured on a live node as the instance principal: GetVnic and ListPrivateIps succeed, GetSubnet and CreatePrivateIp both 404. Authentication, the dynamic group's matching rule and the compartment were all correct, so the credential was authentic and unauthorised. The documented operator policy for the API-key route carried the same three statements, so neither credential route could allocate. A node now proves its own credential at pool start by making one read in each permission class, and logs the missing verb if one fails. The topology gate waited for "resolved the external VNIC", which comes from the metadata service and needs no IAM policy, so an unauthorised node passed the gate and failed an hour later inside a workbook as capacity. It now waits for the authorisation line and fails immediately on a refusal. Reads rather than a real assign-and-release round trip: a round trip also fails on a full subnet, and reporting that as an authorisation fault sends an operator to IAM for a capacity problem. * docs: say what a principal run costs in a tenancy with no dynamic group No principal topology can pass there, and the only previous hint was one clause about OCI_INSTANCE_PRINCIPAL needing the group to exist. * test: cover the subnet read and the authorisation verdict Diff coverage was 39.1% against a 60% gate. Three of the four gaps were real. GetSubnet on the SDK-backed client had no test, so the request shape was unverified: addressing the collection rather than one subnet by OCID would answer from a different permission and report a credential as working when it is not. Also pins that a sparse body is not an error and that the unauthorised 404 arrives as ErrNotFound, which is what stops the check retrying a verdict that will never change. The fake's GetSubnet was exercised only from the ocinet tests, and coverage is recorded per package under test, so it read as zero-covered. reportAuthorisation now has a test asserting the exact log text, because validate-topology.sh greps for it and a rename would quietly turn the gate into one that never fires. The six lines still uncovered are inside FromPoolConfig, which needs a live JetStream to reach. Diff coverage 87.0%. * fix: refuse a run whose tfvars file contradicts the compartment it was given Terraform ranks *.auto.tfvars above TF_VAR_*, so a leftover file wins silently over an operator targeting a different tenancy by environment, and the guide tells every operator to write one. Hit for real: a stale terraform.auto.tfvars naming a different tenancy's compartment and a bare-metal shape overrode TF_VAR_compartment_ocid, and the plan was for 11 resources in a compartment nobody had chosen. Only a dry run caught it. The run now stops and names both values rather than building. * fix: collect orphaned OCI public IPs and name the limit that was hit A reserved public IP attached to nothing is on no VNIC, so the reconcile that exists to find leaked addresses could not see the ones that matter. It also ran only at startup, while an address is leaked on release and on teardown. Twenty accumulated in one compartment and exhausted the tenancy's limit of 50, after which every launch needing a public address aborted. Reconcile now lists reserved public IPs in the compartment and collects those that carry our prefix, are attached to nothing, are named by no binding and are older than a grace period. All four are needed: an allocated but unassociated Elastic IP is detached too and is the customer's to keep, and a young detached address is what another node's CreatePublicIP looks like until it attaches. A pool with no binding record is not swept at all, since that is indistinguishable from a store that cannot be read. The pass now repeats on a ticker rather than running once at startup. Both OCI ceilings also resolved to InsufficientAddressCapacity, which tells an operator the region is out of addresses when the fix is to release addresses they already hold. AWS separates them, so a tenancy out of reserved public IPs is now AddressLimitExceeded and a full VNIC is PrivateIpAddressLimitExceeded. * feat: run the OCI principal e2e in a tenancy that can hold its dynamic group vm-single-principal needs the tenancy's dynamic group, and creating one is a tenancy-root write that a shared test tenancy does not grant us. The topology could never pass there however correct the code was, so it now takes its own credential from OCI_PRINCIPAL_*, all six or none. A partial set is refused before the build rather than at the apply, because it would build one tenancy's instances against another's compartment. With none set the topology is recorded SKIPPED instead of spending a provision to fail. The sweep takes the same credential, or a leftover principal deployment is destroyed against a tenancy that never held it and reported clean. Two defects found while proving it: --dry-run reported FAIL on every topology with a tree source, because the dry run skips the build and then refuses the tarball it would have produced. The flag operators are told to run first could not pass. The dry run truncated its plan to the last 30 lines, which is the outputs block. That dropped the resource count and every compartment and shape line, so the one check a dry run exists for could not be made from its output. It now writes the whole plan and prints the summary. Removing a deployment is documented as leaving public IPs behind. Spinifex allocates them on the running node, so they are in no Terraform state and destroy cannot see them; twenty accumulated and exhausted a tenancy's limit, which then refused every launch in every compartment. * fix: canonicalise the Opc-Next-Page header in the pagination test CI's linter rejects a non-canonical header name and the local preflight does not, so this passed here and failed there.	2026-10-06 20:56:16 -05:00
Ben Duncan	c6841ce1f0	feat(oci): separate using an instance principal from creating one (#1157) ... * feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts.	2026-10-04 09:04:55 -05:00
Ben Duncan	ade060f384	fix(oci): make bare metal work, remove the per-node API key, and validate all three topologies (#1146) ... * fix(oci): make a bare-metal node form and mount its data volume Bare metal presents VNICs differently from a VM: the first VNIC on a physical port is the port, and a second one sharing that port is a tagged overlay with no Linux device at all. The external VNIC was landing in that second slot, so no interface ever carried it and the WAN bridge could not be built. Attach it at nic_index 1 on a BM shape, which makes it the first VNIC on the second port and therefore untagged and present at boot, exactly as it is on a VM. Two consequences of that followed. The bridge script picked the first non-primary VNIC from instance metadata, which on bare metal can be one with no interface, so it now selects the VNIC the kernel actually has a device for and waits for that. And OCI makes a bare-metal volume attachment multipath, where the Oracle agent refuses to log in unless multipathd is running -- probing the unit name Debian retired, so on Ubuntu 26.04 it never sees it. Alias the retired name and assert the alias on every poll rather than once, because the first run had it created and then vanish. Finally, the three cloud-init steps become ordered systemd units. runcmd continues past a failed step, so a node whose volume never appeared still opened its service ports with nothing behind them; Requires= makes that impossible and lets a boot that raced the iSCSI login retry on the next one. * feat(oci): authenticate the address allocator as the instance principal The OCI allocator could only authenticate from an API-key config file, so every node needed a private key copied to it and a config file hand-written beside it. A node without them forms, passes every health check, and then refuses every launch that wants a public address with InsufficientAddressCapacity -- the cause appears only in the journal. NewInstancePrincipalClient already existed and was called from nowhere. Wire it in behind a new oci_auth key on the pool, so a node can authenticate with the certificate its own metadata service serves and hold no key material at all. Nothing to copy, nothing to rotate, and nothing in Terraform state. The reason recorded for preferring the config file was that it reads no instance metadata, because Spinifex's own IMDS endpoints claim 169.254.169.254 and take the cloud's metadata service with them. That no longer distinguishes the two: oci_vnic_iface is what the guide recommends and it resolves the VNIC through a metadata lookup already, so such a node depends on the IMDS remap either way. The two constructors' comments also contradicted each other on which was preferred; both now say what they cost. config_file stays the default, because instance principal needs a dynamic group and a policy only a tenancy admin can write. The Terraform creates both behind enable_instance_principal, off by default for the same reason the module creates no other tenancy-root resource, and the policy grants three verbs on vnics, private-ips and public-ips in one compartment rather than anything broader. Naming the method explicitly rather than inferring it is deliberate: a node meant to use its own identity now fails at config load instead of falling back to a key file nobody installed. Pairing oci_auth="instance_principal" with a config file is refused for the same reason, since the certificate is already the credential and the file would be the half silently ignored. Terraform also stages the pool fragment on each node, which under instance principal contains no secret at all. * feat(oci): validate all three topologies from one script, and document both auth routes validate-topology.sh builds one topology from nothing, installs the published release, forms the cluster, configures the pool and the IMDS remap, runs a workbook against it, proves the workbook actually serves traffic, and destroys everything. Three topologies because each breaks differently: bare metal presents VNICs unlike a VM, a single node has no Geneve underlay to get wrong, and only a cluster exercises RAFT, the gateway chassis and cross-node allocation. --topology has no default, deliberately. A command aimed at the wrong topology is the easiest expensive mistake available here, and each keeps its own Terraform state so two can be built from one checkout without either destroying the other's instances. The teardown decides the verdict rather than tidying up after it. A topology or a workbook that cannot be destroyed is half proved, so destroy runs from an EXIT trap and a teardown failure fails the run. It also checks the two things a green apply does not: that vpcd logged resolving the external VNIC, which is the only evidence the credentials work and br-wan matched a real VNIC, and that the load balancer returns 200 -- a workbook can create cleanly and carry no traffic. The guide gains the instance-principal route beside the key file, and loses its claim that bare metal changes nothing else in the document. That claim was wrong in exactly two places, and both are now handled rather than left to the reader: the external VNIC's nic_index and the Oracle agent's multipath unit name. It now also says plainly that the external-address path on bare metal is unproven. * test(oci): let a topology be validated without an OCI credential The external pool is the one phase that needs a key or a dynamic group, and the VM topologies have to be regression-tested regardless. --no-external-pool forms and verifies without it, and implies --skip-workload so the run cannot claim a guest path it never exercised. * test(oci): one entry point that proves a ref works on OCI, and cleans up after itself e2e-oci.sh drives every topology from environment variables alone, so a workstation, a Worker dispatch and e2e-oci-nightly.yml all take the same path rather than drifting into three answers about whether OCI works. Three things it deliberately does not do. It does not test the published release: --distro installs the tarball built from the ref under test, because a nightly that cannot fail on a bad commit is decoration. It does not carry its own workbook assertions, running tests/e2e/run-tofu-examples-e2e.sh over the published workbooks instead, so a workbook that passes on a hypervisor and fails here is a difference in OCI. And it cannot quietly leave cloud resources running: --sweep-only destroys whatever an earlier run's state still holds, and it runs before the topologies, after them, and from an always() step. The credential now reaches Terraform as a value through OCI_PRIVATE_KEY, so no runner writes key material to disk, and oci_env.py refuses --shell on that route because in CI stdout is the job log. Four harness defects the first real runs exposed, each of which had been giving a wrong answer: - oci_env.py announced itself on stdout, corrupting every captured terraform output. That is also why a single node came back as two hosts. - terraform output was passed -var, which it rejects, and a silent fallback then read the default state file instead of the topology's. - grep -c Ready matches NotReady, so a single-node run whose only node was NotReady on 0.0.0.0 reported PASS. The status column is parsed now, and membership is waited for rather than sampled once. - install-node.sh refuses fewer than two hosts, so a single node forms with spx admin init the way the guide documents. Also waits for the dpkg lock: cloud-init being finished does not mean apt is. * fix(oci): resolve mulga's path explicitly, and refuse an unpushed ref Two assumptions in the new harness that a run would not have caught quickly. Deriving mulga's root from spinifex's is right for a submodule checkout and wrong for CI, where the two are siblings under the workspace, so MULGA_ROOT is explicit and the workflow sets it. build-install-artifacts.sh clones each repo from GitHub rather than using the checkout beside it, and falls back to dev when the ref is not there. A nightly on an unpushed branch would then pass or fail on dev's code while reporting the branch. Check the ref is on origin first, and skip the Go suite build: the workbook driver is a shell script, so those binaries prove nothing here. * fix(oci): do not read $? after a command when set -e is on The workbook phase read the driver's exit status on the line after the ssh that produced it, which under set -e is never reached: a failing run aborted before printing the driver's own per-workbook diagnostics, losing them exactly when they are the thing you need. An `if !` keeps them, and names which workbooks passed and which did not. * feat: let an OCI e2e dispatch form without the external allocator A tenancy without the nodes' dynamic group cannot authenticate the allocator, so every topology failed at it rather than reporting what it had proved. The no_external_pool input forms with no allocator and records the allocator and the workbooks as SKIPPED, never PASS. Also rebuilds the dry-run plan message imperatively: it still referenced a variable renamed to WORKBOOKS, and under set -u the substitution died and the text silently vanished from the line. * fix: name the offending input when an OCI credential is unusable A credential from the environment reached the SDK unvalidated, so a secret with a trailing newline failed as an uncaught InvalidConfig traceback naming an SDK config field that nobody set by that name. Strip the scalar values, since a secret store written with echo carries a newline, and translate both InvalidConfig and InvalidPrivateKey into the variable or config key to go and fix. * fix: make the OCI nightly able to build, and stop it leaking a dispatch input into the shell Four defects, found reviewing the workflow rather than running it. The builder clones all five sub-repos from GitHub over https and they are all private, so a hosted runner had no credential and the tree build could never have worked. The app token now covers six repositories and git carries it for our organisation only. A dispatch input was interpolated into a run block, where a quote ends the string and the remainder executes. It goes through the environment now, which is the only place a caller-supplied value belongs. The artifact upload took the whole directory, which meant the distro tarball and the test archive, and would have taken terraform.tfstate had the state directories sat under it. It is a named list now, and it also collects the per-topology logs that were never uploaded at all. * fix: make the OCI nightly's runner a setting, and stop widening the app token The five sub-repos are public, so build-install-artifacts.sh clones them without a credential and the token has no reason to reach them. Only mulga is private, which is what it was scoped to before. The runner is now inputs.runner, then the OCI_E2E_RUNNER variable, then hosted. A per-provider self-hosted agent is a reasonable thing to want -- it lifts the six-hour cap a hosted job carries and keeps a long cloud wait off the hypervisor labels -- and it should not need a workflow edit. Removing the SSH key is load bearing on that path: a self-hosted runner keeps $HOME between jobs, so the key would sit there between nightlies. * fix: install the OCI SDK the harness needs, and keep state out of the checkout Nothing installed the oci Python module, on any runner: validate-topology.sh called bare python3 and oci_env.py imports oci, so every topology would have failed at the first terraform call. The harness now uses a system python3 that has it and otherwise builds its own virtualenv from requirements.txt, which is the same behaviour on a workstation, a hosted runner and a self-hosted one. The state directories move under OCI_STATE_ROOT, defaulting to the script directory. A persistent runner needs them outside the checkout, because actions/checkout wipes untracked files there and would remove the state of a killed run before the sweep could read it, leaving OCI instances running that nothing is able to find. * fix: drop the OCI SDK dependency, so the harness runs on any host python3-venv is not installed on banksia, and the harness needed it: the environment helper re-executed itself from a virtualenv to import the OCI SDK. It used the SDK for two things -- parsing an INI file, which configparser already does, and discovering the tenancy home region, which no Terraform file consumed. Neither justifies a dependency on every host that runs this, so the helper is standard library only now. The credential is checked for shape instead, locally and before any OCI call, which is a better failure than a 401 forty seconds into a run. Two things the removal uncovered. The home region is a fixed property of a tenancy rather than something to rediscover every run, so it is OCI_HOME_REGION. And OCI serves IAM writes from the home region while the dynamic group and policy were using the default provider for the build region, which would have failed when the tenancy applied them: they go through an aliased provider that defaults to the build region, so a tenancy whose home region is where it builds sees no change.	2026-10-01 13:37:05 -05:00