feat(oci): separate using an instance principal from creating one (#1157)

* feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts.
2026-10-15 09:01:31 +00:00 · 2026-10-04 09:04:55 -05:00
parent 70007405c1
commit c6841ce1f0
18 changed files with 969 additions and 857 deletions
@@ -75,7 +75,7 @@ jobs:
      - name: Set up Go
        uses: actions/setup-go@v7
        with:
-          go-version: "1.27.0"
+          go-version-file: spinifex/go.mod

      # The tag carries the describe string so the commit is recoverable from the
      # version alone, and the -dev suffix is what release.yml's own convention
@@ -167,6 +167,10 @@ jobs:
      # Kept rather than unbounded: the Worker reads the first page of releases to
      # resolve the channel, so enough prereleases would eventually push the real
      # ones off it, and each build carries two assets.
+      #
+      # Twenty, not five. A pinned dev tag is how an external evaluator installs a
+      # specific build, and five is about a week of them -- short enough that the tag
+      # someone was given stops resolving while they are still using it.
      - name: Prune old dev prereleases
        if: steps.version.outputs.exists != 'true'
        env:
@@ -174,7 +178,7 @@ jobs:
        run: |
          set -euo pipefail
          gh api "repos/$GITHUB_REPOSITORY/releases?per_page=100" \
-              --jq 'map(select(.prerelease and (.tag_name | endswith("-dev")))) | .[5:] | .[].tag_name' \
+              --jq 'map(select(.prerelease and (.tag_name | endswith("-dev")))) | .[20:] | .[].tag_name' \
              > stale.txt
          while read -r tag; do
              [ -n "$tag" ] || continue
@@ -1,4 +1,4 @@
-name: E2E OCI Nightly
+name: E2E Cloud Vendor Nightly

 # Dispatched by the same Cloudflare Worker that drives e2e-nightly.yml, once per
 # ref, rather than by a `schedule:` here. GitHub registers a schedule only from
@@ -28,6 +28,12 @@ on:
        required: false
        type: boolean
        default: false
+      auth:
+        description: "How nodes reach the OCI API: instance_principal (needs the tenancy dynamic group) or api_key (needs the credential hook)"
+        required: false
+        default: instance_principal
+        type: choice
+        options: [instance_principal, api_key]
      no_external_pool:
        description: "Form without the OCI allocator. Skips every workbook needing a public address."
        required: false
@@ -59,12 +65,17 @@ jobs:
    # Three topologies including a bare-metal build, each with its own install,
    # formation and workbook pass. The sweep still runs if this fires.
    timeout-minutes: 300
-    env:
-      # Outside github.workspace on purpose. actions/checkout runs git clean -ffdx,
-      # which on a persistent runner would delete the Terraform state of a killed
-      # run before the next sweep could read it, leaving instances nothing can find.
-      OCI_STATE_ROOT: ${{ runner.workspace }}/oci-state
    steps:
+      # Set here rather than in a job-level env block: the runner context is not
+      # available at job level, only from a step onwards, and a workflow that
+      # references it there is rejected as invalid before any job starts.
+      #
+      # Outside github.workspace on purpose. The checkout wipes untracked files, so
+      # on a persistent runner it would delete the Terraform state of a killed run
+      # before the next sweep could read it, leaving instances nothing can find.
+      - name: Put the Terraform state outside the checkout
+        run: echo "OCI_STATE_ROOT=$RUNNER_WORKSPACE/oci-state" >> "$GITHUB_ENV"
+
      - uses: actions/checkout@v7
        with:
          path: spinifex
@@ -121,12 +132,20 @@ jobs:
          OCI_TOPOLOGIES: ${{ inputs.topologies }}
          OCI_SOURCE: ${{ inputs.source }}
          OCI_CHANNEL: ${{ inputs.channel }}
-          # 1 unless the dispatch asked to form without the allocator, which is the
-          # only way to run before the tenancy's dynamic group exists. The run then
-          # records the allocator and the workbooks as SKIPPED rather than passing
-          # them, so a green run with this set is not a claim about guest networking.
-          OCI_INSTANCE_PRINCIPAL: ${{ inputs.no_external_pool && '0' || '1' }}
+          # Independent of the pool, deliberately. These were one switch, which made
+          # the combination we actually need -- an API key with the allocator running
+          # -- impossible to ask for, so public-IP allocation went untested.
+          OCI_INSTANCE_PRINCIPAL: ${{ inputs.auth == 'api_key' && '0' || '1' }}
+          # Skipping the pool records the allocator and every public-address workbook
+          # as SKIPPED, so a green run with this set is not a claim about guest
+          # networking. It exists for a formation-only check, not as a way past a
+          # missing credential.
          OCI_NO_EXTERNAL_POOL: ${{ inputs.no_external_pool && '1' || '0' }}
+          # Only an API-key run needs one: an instance principal is the node's own
+          # certificate, so there is nothing to hand over. Unset is not an error, and
+          # the allocator gate then fails rather than passing a node that cannot
+          # authenticate.
+          OCI_CREDENTIAL_HOOK: ${{ inputs.auth == 'api_key' && vars.OCI_CREDENTIAL_HOOK || '' }}
          OCI_ARTIFACT_DIR: ${{ runner.temp }}/e2e-oci
          OCI_KEEP_ON_FAIL: ${{ inputs.keep_on_fail && '1' || '0' }}
          # Through the environment, never interpolated into the script below: a
@@ -141,6 +160,29 @@ jobs:
          ./e2e-oci.sh 2>&1 | tee "$RUNNER_TEMP/e2e-oci.log"
          exit "${PIPESTATUS[0]}"

+      # Before the sweep, which destroys the state the tables are read from, and
+      # always(): a run whose verdict is a failure is exactly the one somebody
+      # wants a table for. The successful path writes the same tables itself, so
+      # this step is for the run that did not get there.
+      #
+      # The markdown lives in the script rather than here. A heredoc in a YAML
+      # string is the one place in this repository nothing can lint or run.
+      - name: Publish the result tables
+        if: always()
+        working-directory: spinifex/scripts/terraform/oci-spx
+        env:
+          OCI_REGION: ${{ vars.OCI_REGION || 'ap-sydney-1' }}
+          OCI_ARTIFACT_DIR: ${{ runner.temp }}/e2e-oci
+          OCI_TOPOLOGIES: ${{ inputs.topologies }}
+          OCI_SOURCE: ${{ inputs.source }}
+          OCI_CHANNEL: ${{ inputs.channel }}
+          SPX_REF: ${{ github.ref_name }}
+          # The same two the run step passes. The header states what was and was not
+          # tested, so a summary rendered without them would claim more than the run.
+          OCI_INSTANCE_PRINCIPAL: ${{ inputs.auth == 'api_key' && '0' || '1' }}
+          OCI_NO_EXTERNAL_POOL: ${{ inputs.no_external_pool && '1' || '0' }}
+        run: ./e2e-oci.sh --summary-only
+
      # always(), and before the artifact upload: an OCI bare-metal host left
      # running by a cancelled or timed-out job is the most expensive way this
      # workflow can fail. The sweep works from the Terraform state files, so it
@@ -178,7 +220,14 @@ jobs:
            ${{ runner.temp }}/e2e-oci.log
            ${{ runner.temp }}/e2e-oci/*.log
            ${{ runner.temp }}/e2e-oci/verdict.txt
+            ${{ runner.temp }}/e2e-oci/summary.md
            ${{ runner.temp }}/e2e-oci-sweep/*.log
            ${{ runner.workspace }}/oci-state/.validate-*/*.log
            ${{ runner.workspace }}/oci-state/.validate-*/*.txt
+            ${{ runner.workspace }}/oci-state/.validate-*/*.tsv
+            # The hook is the operator's own executable and the one thing known
+            # about it is that it handles a private key. Whatever it printed is not
+            # ours to publish in a downloadable artifact, so it is excluded rather
+            # than trusted to have been quiet.
+            !${{ runner.workspace }}/oci-state/.validate-*/credential-hook.log
          retention-days: 14
@@ -46,7 +46,7 @@ jobs:
      - name: Set up Go
        uses: actions/setup-go@v7
        with:
-          go-version: "1.27.0"
+          go-version-file: spinifex/go.mod

      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
        run: cd spinifex && make build-microvm-image
@@ -167,7 +167,7 @@ preflight:
 # reason: between them they install the software, wire OVN, erase the state and
 # remove the install, so a shellcheck warning in one is an outage or a wrong
 # deletion, not a style note.
-NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/node-reset.sh scripts/uninstall-spx.sh
+NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh

 test-build-scripts:
 	@echo -e "\n....Running build/scripts/**/*_test.sh...."
@@ -13,7 +13,7 @@ require (
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
-	github.com/containerd/containerd/v2 v2.4.0
+	github.com/containerd/containerd/v2 v2.4.1
 	github.com/distribution/reference v0.6.0
 	github.com/go-chi/chi/v5 v5.3.2
 	github.com/golang-jwt/jwt/v5 v5.3.1
@@ -111,8 +111,8 @@ github.com/containerd/console v1.0.5 h1:R0ymNeydRqH2DmakFNdmjR2k0t7UPuiOV/N/27/q
 github.com/containerd/console v1.0.5/go.mod h1:YynlIjWYF8myEu6sdkwKIvGQq+cOckRm6So2avqoYAk=
 github.com/containerd/containerd/api v1.12.0 h1:kuQm82SbDrCuO4n7hf2L8zsBtZLuympyq5X/VotfX2A=
 github.com/containerd/containerd/api v1.12.0/go.mod h1:EBcSzoi9Vl18cdODaXUCskf3D2NT8lsSXeZJnU5jIUc=
-github.com/containerd/containerd/v2 v2.4.0 h1:mZLYWkPAgF4tfTjinClrRfvUxmxHxg1PQL7cjrrpTtk=
-github.com/containerd/containerd/v2 v2.4.0/go.mod h1:gHZz+v5y8mGt9grF3ynuaa3r6bXLmghBZWTAftHAUtg=
+github.com/containerd/containerd/v2 v2.4.1 h1:DUx/ZJN7cEu0WuzHClDB+68H/bqMEH5pWoEjf0ae4hc=
+github.com/containerd/containerd/v2 v2.4.1/go.mod h1:vgLdtvl3prFk1d3ZVqQcsDD5Q9IKM+vAIdU54U85w+I=
 github.com/containerd/continuity v0.5.0 h1:7a85HZpCSs+1Zps0Ee3DPSuAWY+0SJM1JNM51nlEVDg=
 github.com/containerd/continuity v0.5.0/go.mod h1:/lNJvtJKUQStBzpVQ1+rasXO1LAWtUQssk28EZvJ3nE=
 github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
@@ -898,6 +898,17 @@ install_files() {
        info "  /usr/local/share/spinifex/setup.sh"
    fi

+    # Formation. Shipping these means a multi-node cluster can be formed from a
+    # `curl | bash` install with no checkout, which is what a cloud operator has.
+    # smoke-test.sh comes too: install-node.sh scp's it from its own directory for
+    # the post-formation verification, so without it formation fails at the last step.
+    for formation in install-node.sh smoke-test.sh; do
+        if [ -f "$EXTRACT_DIR/$formation" ]; then
+            $SUDO install -m 0755 "$EXTRACT_DIR/$formation" "/usr/local/share/spinifex/$formation"
+            info "  /usr/local/share/spinifex/$formation"
+        fi
+    done
+
    # Teardown scripts. An install delivered by `curl | bash` with no way to
    # undo it is a gap in the product, not just in our tooling, and node-reset.sh
    # ships alongside because uninstall-spx.sh delegates its state teardown to it
@@ -10,3 +10,6 @@ terraform.tfvars
 *.pem
 *.log
 .validate-*/
+# Per-run artifacts and the identity state, which is not a topology's to destroy.
+.e2e-oci-*/
+.identity/
@@ -209,14 +209,27 @@ node_client_cidr_allow_list = ["203.0.113.10/32"]

 The allocator needs OCI credentials at runtime, and there are two ways to give it them. A node with neither forms, passes every health check, and then refuses every launch that wants a public address with `InsufficientAddressCapacity` — the cause appears only in the node's journal, so this is worth getting right before first start.

-**Instance principal is the better one, and it is what `enable_instance_principal = true` sets up.** Each node authenticates with the certificate its own metadata service serves, so no key material exists on any node, there is nothing to rotate, and nothing sensitive reaches Terraform state. The cost is a dynamic group and a policy, which are tenancy-root resources — so the apply needs a tenancy-admin principal, which is why it is off by default and why the rest of this configuration deliberately creates nothing at tenancy root. Create them once, by hand or with a privileged principal, and every later node and rebuild inherits them:
+**Instance principal is the better one.** Each node authenticates with the certificate its own metadata service serves, so no key material exists on any node, there is nothing to rotate, and nothing sensitive reaches Terraform state. The cost is a dynamic group and a policy, which are tenancy-root resources — so creating them needs a tenancy-admin principal, which is why the rest of this configuration deliberately creates nothing at tenancy root.
+
+`instance_principal` has three values, because using an instance principal and being allowed to create one are different rights:
+
+| Value | Pool auth | Creates the dynamic group and policy | Credential needed |
+| --- | --- | --- | --- |
+| `off` (default) | API key file | No | Compartment-scoped |
+| `adopt` | `instance_principal` | No — assumes they exist | Compartment-scoped |
+| `create` | `instance_principal` | Yes | **Tenancy admin** |
+
+Create them once per tenancy, then every deployment and rebuild afterwards uses `adopt`:

 ```bash
-python3 scripts/oci_env.py --ssh-public-key-path <pub> -- \
-    terraform apply -var enable_instance_principal=true \
-    -target oci_identity_dynamic_group.nodes -target oci_identity_policy.nodes
+./setup-identity.sh --dry-run    # always first
+./setup-identity.sh
 ```

+`setup-identity.sh` targets only those two resources and keeps them in `.identity/terraform.tfstate`, separate from every topology's state. That separation is load-bearing: `validate-topology.sh` destroys its own state at the end of each run, so holding tenancy resources there would let a nightly teardown delete the tenancy's policy.
+
+**`adopt` references the dynamic group by nothing at all.** Its matching rule is `instance.compartment.id`, so it covers every instance in the compartment and names no OCID — which is why adopting needs no read on an identity resource and no tenancy rights. The cost is that a missing policy is invisible at apply time: the node forms, passes every health check, and then refuses every launch wanting a public address. The allocator gate in `validate-topology.sh` is what catches that, by requiring `resolved the external VNIC` in each node's `spinifex-vpcd` journal.
+
 The policy grants three verbs in one compartment — `use vnics`, `manage private-ips`, `manage public-ips` — which is exactly what allocating an external address does and nothing more.

 **An API key is the fallback**, and the default because it needs nothing from a tenancy admin. Grant that user only the operations Spinifex performs on addresses and nothing else; it is not a tenancy admin, and broader rights widen the blast radius of a node compromise for no benefit. The key then has to be installed on **every** node, by hand, under `/etc/spinifex/oci/` — not a home directory, because the daemon's unit sets `ProtectHome=yes`:
@@ -226,6 +239,22 @@ sudo install -d -o root -g spinifex -m 0750 /etc/spinifex/oci
 sudo install -o root -g spinifex -m 0640 ~/.oci/oci_api_key.pem /etc/spinifex/oci/oci_api_key.pem
 ```

+**The key is half of it.** `oci_config_file` names an ordinary OCI SDK config, and nothing creates that file, so a node with only the PEM forms and then fails every allocation. The profile must be `spinifex`, to match `oci_config_profile`, and the four identifiers are the ones already in the tfvars:
+
+```bash
+sudo tee /etc/spinifex/oci/config >/dev/null <<'CONF'
+[spinifex]
+user=ocid1.user.oc1..<yours>
+fingerprint=<yours>
+tenancy=ocid1.tenancy.oc1..<yours>
+region=ap-sydney-1
+key_file=/etc/spinifex/oci/oci_api_key.pem
+CONF
+sudo chown root:spinifex /etc/spinifex/oci/config && sudo chmod 0640 /etc/spinifex/oci/config
+```
+
+For repeated runs, `validate-topology.sh --credential-hook` does both steps on every node. The hook is an operator-owned executable rather than anything in this repository, because a credential rendered into user-data or Terraform state is readable from instance metadata for the life of the instance, and these hosts run other people's guests. Instance principal needs no hook at all, which is the reason to prefer it wherever a dynamic group can be created.
+
 Either way, Terraform stages the matching pool block at `/etc/spinifex/oci/external-pool.toml` on each node, with `oci_auth` set to match. Append it to `/etc/spinifex/spinifex.toml` after `spx admin init` and restart `spinifex.target`. Under instance principal that file holds no secret at all, which is the point of it.

 ## Validating a topology end to end
@@ -45,6 +45,11 @@ locals {
  # after the boot disk (mulga-poc has oraclevda1 pointing at sdb1).
  data_device = "/dev/oracleoci/oraclevdb"

+  # adopt and create differ only in who creates the dynamic group and policy; a node
+  # authenticates identically either way, because the matching rule covers the whole
+  # compartment and so names no instance.
+  use_instance_principal = var.instance_principal != "off"
+
  # The allocator's pool, staged on each node for spx admin init to pick up. Under
  # instance principal there is no oci_config_file key, because naming one beside a
  # certificate credential is a second source of truth that config load rejects.
@@ -56,7 +61,7 @@ locals {
    "source             = \"oci\"",
    "oci_compartment_id = \"${var.compartment_ocid}\"",
    "oci_vnic_iface     = \"${var.wan_bridge_name}\"",
-    ], var.enable_instance_principal ? [
+    ], local.use_instance_principal ? [
    "oci_auth           = \"instance_principal\"",
    ] : [
    "oci_config_file    = \"/etc/spinifex/oci/config\"",
@@ -9,6 +9,10 @@
 #   SPX_REF           Ref to prove. Default: the spinifex checkout's current branch.
 #   OCI_TOPOLOGIES    Space-separated. Default: "vm-single vm-multi bm".
 #   OCI_SOURCE        tree (build SPX_REF) or release (the published installer).
+#   OCI_VERSION       With OCI_SOURCE=release, install this exact release tag and
+#                     ignore OCI_CHANNEL. Preferred for anything repeatable: a tag
+#                     resolves by redirect, while the dev channel resolves through a
+#                     rate-limited GitHub API call that 404s when it trips.
 #   OCI_CHANNEL       With OCI_SOURCE=release, which published channel to install:
 #                     latest (default) or dev, the newest prerelease. dev is the
 #                     path our cloud contacts are given, so it is worth testing.
@@ -24,6 +28,10 @@
 #                     workbook are then recorded SKIPPED, never PASS, so a green
 #                     run with this set is not a claim about guest networking.
 #   OCI_SSH_PUBLIC_KEY / OCI_SSH_PRIVATE_KEY   Paths. Default ~/.ssh/oci-spx[.pub].
+#   OCI_CREDENTIAL_HOOK     Executable run on each topology after formation and
+#                     before the pool, as "hook <ssh-key> <host>...". An API-key
+#                     deployment installs its credential here; instance principal
+#                     needs none, so it is skipped unless the pool is key-based.
 #   OCI_ARTIFACT_DIR  Where logs and the verdict land. Default ./.e2e-oci-<stamp>.
 #   OCI_KEEP_ON_FAIL  1 to leave a failed topology up for inspection. Off by
 #                     default: an OCI bare-metal host left overnight is expensive.
@@ -42,6 +50,7 @@ MULGA_ROOT="${MULGA_ROOT:-$SPINIFEX_ROOT/..}"
 TOPOLOGIES="${OCI_TOPOLOGIES:-vm-single vm-multi bm}"
 SOURCE="${OCI_SOURCE:-tree}"
 CHANNEL="${OCI_CHANNEL:-latest}"
+INSTALL_VERSION="${OCI_VERSION:-}"
 REF="${SPX_REF:-$(git -C "$SPINIFEX_ROOT" rev-parse --abbrev-ref HEAD)}"
 SSH_PUBLIC_KEY="${OCI_SSH_PUBLIC_KEY:-$HOME/.ssh/oci-spx.pub}"
 SSH_PRIVATE_KEY="${OCI_SSH_PRIVATE_KEY:-$HOME/.ssh/oci-spx}"
@@ -52,6 +61,7 @@ ARTIFACT_DIR="${OCI_ARTIFACT_DIR:-$HERE/.e2e-oci-$(date -u +%Y%m%dT%H%M%SZ)}"
 export OCI_STATE_ROOT="${OCI_STATE_ROOT:-$HERE}"
 STATE_ROOT="$OCI_STATE_ROOT"
 SWEEP_ONLY=0
+SUMMARY_ONLY=0
 DRY_RUN=0

 log() { printf '[e2e-oci] %s\n' "$*"; }
@@ -62,11 +72,13 @@ die() {

 usage() {
    cat >&2 <<EOF
-usage: ${0##*/} [--sweep-only] [--dry-run]
+usage: ${0##*/} [--sweep-only] [--summary-only] [--dry-run]

 Everything else is an environment variable; see the header of this script.

  --sweep-only   Destroy anything an earlier run left behind, then stop.
+  --summary-only Write the result tables from the state files, then stop. For
+                 a job that was cancelled or killed before it got there.
  --dry-run      Print what each topology would do, build nothing.
 EOF
    exit 2
@@ -75,6 +87,7 @@ EOF
 while [ $# -gt 0 ]; do
    case "$1" in
        --sweep-only) SWEEP_ONLY=1; shift ;;
+        --summary-only) SUMMARY_ONLY=1; shift ;;
        --dry-run) DRY_RUN=1; shift ;;
        -h | --help) usage ;;
        *) die "unknown option $1" ;;
@@ -91,6 +104,8 @@ case "$CHANNEL" in
    *) die "OCI_CHANNEL must be latest or dev, got '$CHANNEL'" ;;
 esac
 # Saying release+dev and tree at once is two different artifacts in one verdict.
+[ "$SOURCE" = tree ] && [ -n "$INSTALL_VERSION" ] \
+    && die "OCI_VERSION needs OCI_SOURCE=release: a tree build installs the ref, not a published tag"
 [ "$SOURCE" = tree ] && [ "$CHANNEL" != latest ] \
    && die "OCI_CHANNEL=$CHANNEL needs OCI_SOURCE=release: a tree build installs the ref, not a channel"

@@ -98,6 +113,86 @@ mkdir -p "$ARTIFACT_DIR"
 VERDICT="$ARTIFACT_DIR/verdict.txt"
 : > "$VERDICT"

+# The run page is the evidence, and its readers are not us: someone assessing
+# whether Spinifex works on their cloud should get the answer without opening a
+# log or knowing what this harness is. Markdown on stdout as well as to the step
+# summary, so a workstation run shows the same thing CI publishes.
+mark() {
+    case "$1" in
+        PASS) printf '✅ PASS' ;;
+        FAIL) printf '❌ FAIL' ;;
+        SKIPPED) printf '⏭️ SKIPPED' ;;
+        *) printf 'ℹ️ %s' "$1" ;;
+    esac
+}
+
+summary() {
+    printf '## Spinifex on OCI — %s\n\n' "$REF"
+    # Said, not implied. A tree build and a published release answer different
+    # questions, and a table headed "published release" over a build of somebody's
+    # branch is the one error here that would mislead a reader who trusted it.
+    if [ "$SOURCE" = tree ]; then
+        printf 'Built from this ref, region `%s`.\n\n' "${OCI_REGION:-ap-sydney-1}"
+    else
+        printf 'Published release `%s`, region `%s`.\n\n' \
+            "${INSTALL_VERSION:-$CHANNEL channel}" "${OCI_REGION:-ap-sydney-1}"
+    fi
+
+    # In the header, not left to a SKIPPED row further down. A run with no allocator
+    # is not evidence about public addressing, and that is precisely the claim a
+    # reader of a green table would otherwise take from it.
+    if [ "${OCI_NO_EXTERNAL_POOL:-0}" = 1 ]; then
+        printf '> **No external address pool.** Public-address allocation and every workbook needing a public address were skipped, so nothing here speaks to guest ingress.\n\n'
+    elif [ "${OCI_INSTANCE_PRINCIPAL:-1}" = 1 ]; then
+        printf 'Allocator authenticated as the instance principal.\n\n'
+    else
+        printf 'Allocator authenticated with an API key.\n\n'
+    fi
+
+    local topology results workbooks gate status detail name secs
+    for topology in $TOPOLOGIES; do
+        results="$STATE_ROOT/.validate-$topology/results.tsv"
+        workbooks="$STATE_ROOT/.validate-$topology/workbooks.tsv"
+        printf '### `%s`\n\n' "$topology"
+        # A topology the loop never reached is said so rather than omitted: a
+        # reader takes an absent row for a passing one.
+        if [ ! -s "$results" ]; then
+            printf 'Did not run.\n\n'
+            continue
+        fi
+        printf '| Gate | Result | Detail |\n| --- | --- | --- |\n'
+        while IFS=$'\t' read -r gate status detail; do
+            [ -n "$gate" ] || continue
+            printf '| %s | %s | %s |\n' "$gate" "$(mark "$status")" "$detail"
+        done < "$results"
+        printf '\n'
+        if [ ! -s "$workbooks" ]; then
+            printf 'No published workbooks ran.\n\n'
+            continue
+        fi
+        printf '| Terraform workbook | Result | Duration |\n| --- | --- | --- |\n'
+        while IFS=$'\t' read -r name status secs; do
+            [ -n "$name" ] || continue
+            printf '| `%s` | %s | %ss |\n' "$name" "$(mark "$status")" "$secs"
+        done < "$workbooks"
+        printf '\n'
+    done
+}
+
+# GITHUB_STEP_SUMMARY is per step, so appending from both the run and the dedicated
+# publish step would put two copies of the tables on the run page. One owner: the run
+# writes the file and prints it, and only --summary-only publishes it.
+emit_summary() {
+    summary | tee "$ARTIFACT_DIR/summary.md"
+}
+
+publish_summary() {
+    emit_summary
+    if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+        cat "$ARTIFACT_DIR/summary.md" >> "$GITHUB_STEP_SUMMARY"
+    fi
+}
+
 # A topology destroys itself from its own EXIT trap, so this only catches a run
 # the trap never reached: a killed process, a runner reclaimed mid-job, a reboot.
 # It works from the state files rather than by name filter, so it can only ever
@@ -127,6 +222,14 @@ sys.exit(0 if any(r["instances"] for r in state.get("resources", [])) else 1)
    [ "$found" = 0 ] && log "sweep: nothing left behind"
 }

+# Before the sweep, because the sweep is what destroys the state the tables are
+# read from. This is the path a cancelled job takes, so it has to work on whatever
+# the run got through rather than assuming a complete set.
+if [ "$SUMMARY_ONLY" = 1 ]; then
+    publish_summary
+    exit 0
+fi
+
 # Run the sweep before the topologies as well as after: a leftover cluster from a
 # killed run is both a bill and a second set of resources with our names on it.
 log "artifacts in $ARTIFACT_DIR"
@@ -178,7 +281,14 @@ for topology in $TOPOLOGIES; do
        args+=(--instance-principal)
    fi
    [ -n "$DISTRO_TARBALL" ] && args+=(--distro "$DISTRO_TARBALL" --setup-sh "$ARTIFACT_DIR/setup.sh")
-    [ "$SOURCE" = release ] && args+=(--channel "$CHANNEL")
+    if [ "$SOURCE" = release ]; then
+        if [ -n "$INSTALL_VERSION" ]; then
+            args+=(--version "$INSTALL_VERSION")
+        else
+            args+=(--channel "$CHANNEL")
+        fi
+    fi
+    [ -n "${OCI_CREDENTIAL_HOOK:-}" ] && args+=(--credential-hook "$OCI_CREDENTIAL_HOOK")
    [ "${OCI_KEEP_ON_FAIL:-0}" = 1 ] && args+=(--keep-on-fail)
    [ -n "${WORKBOOKS+x}" ] && args+=(--workbooks "$WORKBOOKS")

@@ -201,8 +311,11 @@ done
 # case that costs money, and the verdict should say so rather than the log.
 sweep

+emit_summary
+
+
 echo
-log "=== $REF on OCI ($SOURCE build$([ "$SOURCE" = release ] && echo ", $CHANNEL channel")) ==="
+log "=== $REF on OCI ($SOURCE build$([ "$SOURCE" = release ] && { [ -n "$INSTALL_VERSION" ] && echo ", $INSTALL_VERSION" || echo ", $CHANNEL channel"; })) ==="
 cat "$VERDICT"
 if [ "$RUN_RC" = 0 ] && ! grep -q FAIL "$VERDICT"; then
    log "PASS"
@@ -3,15 +3,15 @@
 # node forms and passes every health check, then refuses every launch that wants a
 # public address with InsufficientAddressCapacity and no stated cause.
 #
-# Off by default, and deliberately: a dynamic group and a policy are tenancy-root
-# resources, so this needs rights the compartment-scoped user the rest of this
-# configuration runs as does not have and should not be given. Turn it on with a
-# principal that holds them, or create both by hand once -- the OCIDs are stable
-# and nothing here is per-node.
+# Only instance_principal = "create" builds these, and deliberately: a dynamic
+# group and a policy are tenancy-root resources, so creating them needs rights the
+# compartment-scoped user the rest of this configuration runs as does not have and
+# should not be given. Create them once with a tenancy admin, then every later
+# deployment uses "adopt", which references nothing and so needs no such rights.

 resource "oci_identity_dynamic_group" "nodes" {
  provider       = oci.home
-  count          = var.enable_instance_principal ? 1 : 0
+  count          = var.instance_principal == "create" ? 1 : 0
  compartment_id = var.tenancy_ocid
  name           = "${var.deployment_name}-nodes"
  description    = "Spinifex nodes in the ${var.deployment_name} deployment, authenticating as instance principals."
@@ -26,7 +26,7 @@ resource "oci_identity_dynamic_group" "nodes" {
 # these three verbs cover; nothing here grants compute, storage or identity.
 resource "oci_identity_policy" "nodes" {
  provider       = oci.home
-  count          = var.enable_instance_principal ? 1 : 0
+  count          = var.instance_principal == "create" ? 1 : 0
  compartment_id = var.compartment_ocid
  name           = "${var.deployment_name}-nodes-network"
  description    = "Allows Spinifex nodes to manage the private and public IPs backing external addresses."
@@ -0,0 +1,86 @@
+#!/bin/bash
+# setup-identity.sh -- create the dynamic group and policy that let Spinifex nodes
+# authenticate to the OCI API as themselves, so no API key is installed on any node.
+#
+# Run once per tenancy, with a tenancy-admin principal. Every deployment afterwards
+# passes instance_principal = "adopt", which references these by nothing at all: the
+# matching rule covers the whole compartment, so it names no instance and needs no
+# OCID from here.
+#
+# Its own state file, deliberately. validate-topology.sh destroys its state at the
+# end of every run, and these two resources outlive every topology -- holding them
+# in a per-topology state would let a nightly teardown delete the tenancy's policy.
+set -uo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+STATE_DIR="${OCI_STATE_ROOT:-$HERE}/.identity"
+SSH_PUBLIC_KEY="${OCI_SSH_PUBLIC_KEY:-$HOME/.ssh/oci-spx.pub}"
+DESTROY=0
+DRY_RUN=0
+
+log() { printf '[setup-identity] %s\n' "$*"; }
+die() {
+    printf '[setup-identity] ERROR: %s\n' "$*" >&2
+    exit 1
+}
+
+usage() {
+    cat >&2 <<EOF
+usage: ${0##*/} [--dry-run] [--destroy]
+
+Creates the OCI dynamic group and policy for instance-principal authentication.
+Needs a tenancy-admin OCI credential; the compartment-scoped user the rest of this
+configuration runs as cannot create either resource.
+
+  --dry-run   Show the plan, change nothing.
+  --destroy   Remove them again. Every node using "adopt" loses its credential.
+EOF
+    exit 2
+}
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --dry-run) DRY_RUN=1; shift ;;
+        --destroy) DESTROY=1; shift ;;
+        -h | --help) usage ;;
+        *) die "unknown option $1" ;;
+    esac
+done
+
+[ -r "$SSH_PUBLIC_KEY" ] || die "no readable SSH public key at $SSH_PUBLIC_KEY (set OCI_SSH_PUBLIC_KEY)"
+mkdir -p "$STATE_DIR"
+
+# The instance and network resources are not targeted, so nothing else in this
+# configuration is created or read. A tenancy admin running this does not need to
+# hold the compartment rights the rest of it uses.
+targets=(-target oci_identity_dynamic_group.nodes -target oci_identity_policy.nodes)
+
+tf() {
+    python3 "$HERE/scripts/oci_env.py" --ssh-public-key-path "$SSH_PUBLIC_KEY" -- \
+        terraform -chdir="$HERE" "$@"
+}
+
+tf init -input=false -upgrade >/dev/null || die "terraform init failed"
+
+if [ "$DESTROY" = 1 ]; then
+    log "destroying the dynamic group and policy"
+    tf destroy -auto-approve -var 'instance_principal=create' \
+        -state "$STATE_DIR/terraform.tfstate" "${targets[@]}" \
+        || die "destroy failed"
+    log "removed. Any node configured for instance principal can no longer allocate an address."
+    exit 0
+fi
+
+args=(-var 'instance_principal=create' -state "$STATE_DIR/terraform.tfstate" "${targets[@]}")
+
+if [ "$DRY_RUN" = 1 ]; then
+    log "dry run: planning the dynamic group and policy only"
+    tf plan "${args[@]}" || die "plan failed"
+    exit 0
+fi
+
+log "creating the dynamic group and policy (tenancy-root resources)"
+tf apply -auto-approve "${args[@]}" || die "apply failed -- a tenancy-admin principal is required"
+
+log "done. State in $STATE_DIR/terraform.tfstate -- keep it, it is the only record."
+log "Deployments can now use instance_principal = \"adopt\" with a compartment-scoped user."
@@ -21,12 +21,14 @@ SKIP_POOL=0
 DRY_RUN=0
 DESTROY_ONLY=0
 INSTANCE_PRINCIPAL=0
+CREDENTIAL_HOOK="${OCI_CREDENTIAL_HOOK:-}"
 DISTRO=""
 SETUP_SH=""
 # Empty means the driver's own default list. Unset is distinguishable from empty,
 # so --workbooks "" can deliberately mean "run none".
 WORKBOOKS_SET=0
 CHANNEL=latest
+INSTALL_VERSION=""
 WORKBOOKS=""

 # Shapes and counts per topology. Named here rather than passed in, because the
@@ -62,9 +64,16 @@ workbook against it, then destroys everything.
  --instance-principal    Configure the pool with oci_auth="instance_principal"
                          instead of a key file. Needs the dynamic group and
                          policy to exist already; see instance-principal.tf.
+  --credential-hook PATH  Executable run after formation, before the pool, as
+                          "hook <ssh-key> <host>...". Where an API-key deployment
+                          installs its credential. Default \$OCI_CREDENTIAL_HOOK.
  --workbooks LIST        Space-separated workbooks for the published driver to
                          run on the cluster. Empty means run none; omitted means
                          the driver's own default list.
+  --version TAG           Install this exact release tag. Preferred over
+                          --channel dev for anything repeatable: a tag resolves
+                          by redirect, while the dev channel resolves through a
+                          rate-limited GitHub API call that 404s when it trips.
  --distro PATH           Install this distro tarball instead of the published
                          release, so what is proved is the ref it was built from.
  --setup-sh PATH         setup.sh to pair with --distro. Both or neither.
@@ -87,9 +96,11 @@ while [ $# -gt 0 ]; do
        --ssh-private-key) SSH_PRIVATE_KEY="${2:?}"; shift 2 ;;
        --workbooks) WORKBOOKS_SET=1; WORKBOOKS="${2-}"; shift 2 ;;
        --channel) CHANNEL="${2-}"; shift 2 ;;
+        --version) INSTALL_VERSION="${2:?}"; shift 2 ;;
        --distro) DISTRO="${2:?}"; shift 2 ;;
        --setup-sh) SETUP_SH="${2:?}"; shift 2 ;;
        --instance-principal) INSTANCE_PRINCIPAL=1; shift ;;
+        --credential-hook) CREDENTIAL_HOOK="${2:?}"; shift 2 ;;
        --no-external-pool) SKIP_POOL=1; SKIP_WORKLOAD=1; shift ;;
        --skip-workload) SKIP_WORKLOAD=1; shift ;;
        --keep) KEEP=1; shift ;;
@@ -112,6 +123,10 @@ if [ -n "$DISTRO" ] || [ -n "$SETUP_SH" ]; then
    [ -r "$SETUP_SH" ] || die "--setup-sh is not readable: '$SETUP_SH'"
 fi

+# adopt, never create: this script's state is destroyed at the end of every run, and
+# the dynamic group and policy outlive every topology. setup-identity.sh owns them.
+PRINCIPAL_MODE=$([ "$INSTANCE_PRINCIPAL" = 1 ] && echo adopt || echo off)
+
 SHAPE="${TOPO_SHAPE[$TOPOLOGY]}"
 NODES="${TOPO_NODES[$TOPOLOGY]}"
 # Outside the checkout on a persistent runner, because actions/checkout runs
@@ -122,6 +137,12 @@ mkdir -p "$STATE_ROOT"
 STATE_DIR="$STATE_ROOT/.validate-$TOPOLOGY"
 RESULTS="$STATE_DIR/results.txt"

+# Tab-separated siblings of the human block, for the run-page tables. Written here
+# rather than parsed out of the log later: the status of a gate is then a field
+# this script sets, not a regex over prose that reads fine and matches wrong.
+RESULTS_TSV="$STATE_DIR/results.tsv"
+WORKBOOKS_TSV="$STATE_DIR/workbooks.tsv"
+
 # A separate state directory per topology, so two topologies can be built from one
 # checkout without one destroying the other's instances.
 tf() {
@@ -135,7 +156,7 @@ tf_state=(-state "$STATE_DIR/terraform.tfstate")
 tf_vars=(
    -var "compute_shape=$SHAPE"
    -var "node_count=$NODES"
-    -var "enable_instance_principal=$([ "$INSTANCE_PRINCIPAL" = 1 ] && echo true || echo false)"
+    -var "instance_principal=$PRINCIPAL_MODE"
    "${tf_state[@]}"
 )

@@ -148,7 +169,23 @@ ssh_node() {
        -o LogLevel=ERROR -o BatchMode=yes -o ConnectTimeout=15 "ubuntu@$host" "$@"
 }

-record() { printf '%s\n' "$*" >> "$RESULTS"; }
+# record <gate> <PASS|FAIL|SKIPPED|INFO> [detail]
+record() {
+    local gate="$1" status="$2" detail="${3-}"
+    printf '%s: %s%s\n' "$gate" "$status" "${detail:+ $detail}" >> "$RESULTS"
+    printf '%s\t%s\t%s\n' "$gate" "$status" "$detail" >> "$RESULTS_TSV"
+}
+
+# The driver's per-workbook lines, which are go test's own shape because
+# go-junit-report consumes them downstream. A workbook the driver never reached
+# is absent from its log and so absent here, which the summary renders as a row
+# rather than dropping — a missing row reads as a pass.
+record_workbooks() {
+    local log="$1"
+    [ -r "$log" ] || return 0
+    sed -n 's/^--- \(PASS\|FAIL\): TestTofuWorkbook_\([A-Za-z0-9_]*\) (\([0-9]*\)\.[0-9]*s)$/\2\t\1\t\3/p' \
+        "$log" | tr '_' '-' > "$WORKBOOKS_TSV"
+}

 # Teardown is the last assertion, not cleanup: a workbook or a topology that
 # cannot be destroyed is a defect, and it has been one before.
@@ -157,10 +194,10 @@ destroy_topology() {
    log "destroying"
    if tf destroy -auto-approve -no-color "${tf_vars[@]}" > "$STATE_DIR/destroy.log" 2>&1; then
        DESTROY_RC=0
-        record "destroy: PASS"
+        record destroy PASS
    else
        DESTROY_RC=1
-        record "destroy: FAIL (see $STATE_DIR/destroy.log)"
+        record destroy FAIL "see $STATE_DIR/destroy.log"
        log "TEARDOWN FAILED — resources may still be billing. $STATE_DIR/destroy.log"
    fi
 }
@@ -206,9 +243,20 @@ if [ "$DESTROY_ONLY" = 1 ]; then
 fi

 : > "$RESULTS"
+: > "$RESULTS_TSV"
+: > "$WORKBOOKS_TSV"

 log "$SHAPE, $NODES node(s), state in $STATE_DIR"

+# Checked here, not where the hook is run: nothing about it depends on the apply, and
+# the run is otherwise forty minutes and a bare-metal bill from discovering that the
+# path in a CI variable does not exist on this runner.
+if [ "$SKIP_POOL" != 1 ] && [ "$PRINCIPAL_MODE" = off ]; then
+    [ -n "$CREDENTIAL_HOOK" ] \
+        || die "this deployment authenticates with an API key and no credential hook is set, so no node could reach the OCI API; set --credential-hook or OCI_CREDENTIAL_HOOK, or use an instance principal"
+    [ -x "$CREDENTIAL_HOOK" ] || die "credential hook is not executable: $CREDENTIAL_HOOK"
+fi
+
 # The directory's own terraform.tfstate belongs to hand-driven runs, not to a
 # topology, and an instance left in it is both a bill and a name that collides
 # with ours in the console. Say so rather than letting it be a surprise.
@@ -249,7 +297,7 @@ tf init -input=false > "$STATE_DIR/init.log" 2>&1 || die "terraform init failed;
 log "building"
 tf apply -auto-approve -no-color "${tf_vars[@]}" > "$STATE_DIR/apply.log" 2>&1 \
    || die "terraform apply failed; see $STATE_DIR/apply.log"
-record "build: PASS"
+record build PASS

 mapfile -t HOSTS < <(tf output -raw "${tf_state[@]}" hosts_file) \
    || die "could not read the hosts_file output from $STATE_DIR/terraform.tfstate"
@@ -283,7 +331,7 @@ for host in "${HOSTS[@]}"; do
        || die "cloud-init did not converge on $host; see $STATE_DIR/cloudinit-$host.txt"
    log "$host cloud-init: $(head -1 "$STATE_DIR/cloudinit-$host.txt")"
 done
-record "cloud-init units: PASS"
+record "cloud-init units" PASS

 # With --distro the bytes are the ref under test; without it they are the published
 # release, which answers a different question -- what a customer following the
@@ -298,7 +346,7 @@ for host in "${HOSTS[@]}"; do
    # cloud-init being finished does not mean apt is: the apt-daily timers and
    # unattended-upgrades run on their own schedule and hold the dpkg lock, which
    # the installer then fails on. Wait for the lock rather than fight it.
-    ssh_node "$host" "DISTRO_NAME='$(basename "${DISTRO:-}")' SETUP_NAME='$(basename "${SETUP_SH:-}")' CHANNEL='$CHANNEL' bash -s" \
+    ssh_node "$host" "DISTRO_NAME='$(basename "${DISTRO:-}")' SETUP_NAME='$(basename "${SETUP_SH:-}")' CHANNEL='$CHANNEL' INSTALL_VERSION='$INSTALL_VERSION' bash -s" \
        > "$STATE_DIR/install-$host.log" 2>&1 <<'REMOTE' \
        || die "install failed on $host; see $STATE_DIR/install-$host.log"
 set -e
@@ -318,13 +366,23 @@ else
    # The real customer path, including the checksum step a local tarball skips.
    # The environment variable rather than --channel: the installer served by
    # install.mulgadc.com is the latest release's, so it predates the flag.
-    curl -sfL https://install.mulgadc.com | sudo env INSTALL_SPINIFEX_CHANNEL="$CHANNEL" bash
+    #
+    # A tag goes in as VERSION, not CHANNEL. Only the dev channel resolves through
+    # an unauthenticated GitHub API call, which is rate limited per source address
+    # and returns 404 once it trips -- indistinguishable from a missing asset. A
+    # tagged path is a plain redirect, so it does not have that failure mode.
+    if [ -n "$INSTALL_VERSION" ]; then
+        curl -sfL https://install.mulgadc.com | sudo env INSTALL_SPINIFEX_VERSION="$INSTALL_VERSION" bash
+    else
+        curl -sfL https://install.mulgadc.com | sudo env INSTALL_SPINIFEX_CHANNEL="$CHANNEL" bash
+    fi
 fi
 sudo /usr/local/share/spinifex/setup-ovn.sh --management --nat-uplink
 REMOTE
-    log "$host installed: $(ssh_node "$host" 'spx version' 2>&1 | head -1)"
+    INSTALLED_VERSION="$(ssh_node "$host" 'spx version' 2>&1 | head -1)"
+    log "$host installed: $INSTALLED_VERSION"
 done
-record "install: PASS"
+record install PASS "$INSTALLED_VERSION"

 # A single node is its own documented path: install-node.sh refuses fewer than two
 # hosts, because there is nothing to join. Both branches use the flags the guide
@@ -351,7 +409,21 @@ else
        --yes > "$STATE_DIR/form.log" 2>&1 \
        || die "formation failed; see $STATE_DIR/form.log"
 fi
-record "formation: PASS"
+record formation PASS
+
+# Runs after formation and before the pool is configured, which is the only window
+# where a node has /etc/spinifex but has not yet started the allocator. Nothing in
+# this repository knows what it does: an API-key deployment needs a credential on
+# each node, and a credential belongs to the operator, not to a checked-in script.
+# Skipped under instance principal, which needs no handoff at all.
+if [ "$SKIP_POOL" != 1 ] && [ "$PRINCIPAL_MODE" = off ]; then
+    log "running the credential hook"
+    # Arguments, not a file: the hook is told where the nodes are and how to reach
+    # them, and decides for itself what to put there.
+    "$CREDENTIAL_HOOK" "$SSH_PRIVATE_KEY" "${HOSTS[@]}" > "$STATE_DIR/credential-hook.log" 2>&1 \
+        || die "the credential hook failed; see $STATE_DIR/credential-hook.log"
+    log "credential hook ok"
+fi

 # The IMDS remap and the pool are both set-before-first-start, and the remap is the
 # half that is invisible when missing: without it the cloud's metadata service is
@@ -362,45 +434,90 @@ for host in "${HOSTS[@]}"; do
        || die "pool configuration failed on $host; see $STATE_DIR/pool-$host.log"
 set -e
 sudo cp /etc/spinifex/spinifex.toml /etc/spinifex/spinifex.toml.bak-prepool
+# subn, not sub: with no [network] section to match, sub rewrites the file unchanged
+# and exits 0, leaving the remap silently unapplied. That is the half of this that is
+# invisible when missing, so the substitution count is checked rather than assumed.
 grep -q imds_host_meta_ip /etc/spinifex/spinifex.toml || sudo python3 - <<"PY"
-import re
+import re, sys
 path = "/etc/spinifex/spinifex.toml"
-text = open(path).read()
-text = re.sub(r"(?m)^\[network\]$",
-              "[network]\nimds_host_meta_ip = \"169.254.42.254\"\nimds_host_dns_ip  = \"169.254.42.253\"",
-              text, count=1)
+text, n = re.subn(r"(?m)^\[network\]$",
+                  "[network]\nimds_host_meta_ip = \"169.254.42.254\"\nimds_host_dns_ip  = \"169.254.42.253\"",
+                  open(path).read(), count=1)
+if n != 1:
+    sys.exit("no [network] section in %s, so the IMDS remap was not applied" % path)
 open(path, "w").write(text)
 PY
 if [ "${SKIP_POOL:-0}" != 1 ]; then
    grep -q 'name               = "oci-public"' /etc/spinifex/spinifex.toml \
        || sudo tee -a /etc/spinifex/spinifex.toml < /etc/spinifex/oci/external-pool.toml >/dev/null
 fi
-sudo spx config validate --config /etc/spinifex/spinifex.toml 2>/dev/null || true
+# spx has no config subcommand, so the `spx config validate` that used to be here
+# could only ever fail, and was swallowed. Parsing the file is the check that was
+# wanted: both edits above are textual, and a broken result would otherwise surface
+# as every service failing to start with nothing saying why.
+sudo python3 -c 'import sys, tomllib; tomllib.load(open(sys.argv[1], "rb"))' \
+    /etc/spinifex/spinifex.toml \
+    || { echo "spinifex.toml is not valid TOML after the IMDS remap and the pool append"; exit 1; }
 sudo systemctl restart spinifex.target
 REMOTE
 done

+# Everything needed to tell the three allocator faults apart, and deliberately no
+# credential: the config file and the PEM are reported as mode and size only, which
+# distinguishes absent from unreadable without copying either into a log that is
+# attached to a CI run.
+allocator_diagnostics() {
+    local host="$1"
+    ssh_node "$host" '
+        echo "=== spinifex-daemon: the allocator (ocinet) lives here ==="
+        sudo journalctl -u spinifex-daemon --since -20min --no-pager | grep -i "ocinet\|external VNIC\|allocator\|oci" | tail -40
+        echo "=== spinifex-vpcd: consumes the addresses, does not allocate them ==="
+        sudo journalctl -u spinifex-vpcd --since -20min --no-pager | tail -40
+        echo "=== external_pools as configured ==="
+        sudo sed -n "/\[\[network.external_pools\]\]/,\$p" /etc/spinifex/spinifex.toml
+        echo "=== credential files (mode and size, never content) ==="
+        sudo stat -c "%n %A %s bytes owner=%U:%G" \
+            /etc/spinifex/oci/config /etc/spinifex/oci/oci_api_key.pem 2>&1
+        echo "=== the bridge MAC the allocator matches on ==="
+        ip -br link show
+        echo "=== MACs IMDS reports for this instance VNICs ==="
+        curl -sf -H "Authorization: Bearer Oracle" \
+            http://169.254.169.254/opc/v2/vnics/ | tr "," "\n" | grep -i "macAddr\|privateIp" || \
+            echo "IMDS unreachable -- check the remap, Spinifex claims 169.254.169.254 itself"
+    ' 2>&1
+}
+
 # resolved the external VNIC is the line that proves the credential works, the
 # compartment is right and br-wan's MAC matched a real VNIC. A node missing it
 # accepts allocate-address and then fails it.
 if [ "$SKIP_POOL" = 1 ]; then
    log "--no-external-pool: no allocator to check"
-    record "oci allocator: SKIPPED"
+    record "oci allocator" SKIPPED "no external pool configured"
 else
    log "checking the allocator came up"
    for host in "${HOSTS[@]}"; do
        found=0
        for _ in $(seq 1 30); do
-            if ssh_node "$host" "sudo journalctl -u spinifex-vpcd --since -10min --no-pager | grep -q 'resolved the external VNIC'" 2>/dev/null; then
+            # spinifex-daemon, not spinifex-vpcd: the allocator is built in the
+            # daemon, and vpcd only consumes the addresses it hands out. Gating on
+            # vpcd's journal failed a node whose allocator was working.
+            if ssh_node "$host" "sudo journalctl -u spinifex-daemon --since -10min --no-pager | grep -q 'resolved the external VNIC'" 2>/dev/null; then
                found=1
                break
            fi
            sleep 10
        done
-        [ "$found" = 1 ] || die "$host never logged 'resolved the external VNIC'; the OCI allocator is not up, so no guest can get a public address"
+        if [ "$found" != 1 ]; then
+            # Before the teardown, because the teardown is what destroys the only
+            # copy. Three unrelated faults land here -- a credential the SDK would
+            # not load, a bridge MAC matching no VNIC, and a pool vpcd never read --
+            # and they are indistinguishable from the missing log line alone.
+            allocator_diagnostics "$host" > "$STATE_DIR/allocator-$host.log" 2>&1 || true
+            die "$host never logged 'resolved the external VNIC'; the OCI allocator is not up, so no guest can get a public address. Diagnostics: $STATE_DIR/allocator-$host.log"
+        fi
        log "$host allocator ready"
    done
-    record "oci allocator: PASS"
+    record "oci allocator" PASS
 fi

 # Parse the STATUS column, never grep for Ready: NotReady contains it, and that
@@ -421,11 +538,11 @@ for _ in $(seq 1 30); do
 done
 cat "$STATE_DIR/nodes.txt"
 [ "$ready" = "$NODES" ] || die "expected $NODES Ready node(s), got $ready; see $STATE_DIR/nodes.txt"
-record "membership: PASS ($ready Ready)"
+record membership PASS "$ready Ready"

 if [ "$SKIP_WORKLOAD" = 1 ]; then
    log "--skip-workload: stopping before the workbook"
-    record "workbook: SKIPPED"
+    record workbooks SKIPPED "--skip-workload"
    exit 0
 fi

@@ -458,14 +575,58 @@ ssh_node "${HOSTS[0]}" '
 workbook_env="WORKBOOK_DIR=\$HOME/workbooks"
 [ "$WORKBOOKS_SET" = 1 ] && workbook_env="$workbook_env $(printf 'WORKBOOKS=%q' "$WORKBOOKS")"

+# Whether a node can reach a public address inside its own VCN. Informational: the
+# remedy below is right either way, because a customer reaches a guest from outside.
+#
+# Needs two hosts. With one, HOSTS[0] and HOSTS[-1] are the same address and the node
+# connects to itself, which always succeeds and measures nothing — it reported
+# "present" on vm-single and bm for exactly that reason before this guard.
+#
+# A negative result is only interpretable when the security list admits the source,
+# so the record says which it was: narrowing node_client_cidr_allow_list makes a
+# node's own public address a non-permitted source, and the refusal that follows
+# looks identical to a VCN that does not hairpin.
+if [ "${#HOSTS[@]}" -lt 2 ]; then
+    record "vcn hairpin" SKIPPED "needs two nodes; one node can only probe itself"
+elif ssh_node "${HOSTS[0]}" \
+    "bash -c 'exec 3<>/dev/tcp/${HOSTS[${#HOSTS[@]}-1]}/22' 2>/dev/null"; then
+    record "vcn hairpin" INFO "present: a node reached another node's public address"
+else
+    record "vcn hairpin" INFO "not reached, which is a VCN without hairpin or a security list that excludes the source"
+fi
+
+# Remote dynamic forward: the node gets a SOCKS5 proxy on this port whose egress
+# is this host, outside the VCN. It gives the driver the customer's vantage point
+# for public addresses without moving the driver off the node, where its ovn-nbctl,
+# journalctl and spx diagnostics have to run. The subnet's own allow list does not
+# need a new entry: ssh from here already works, and its rules are per-CIDR for
+# every protocol.
+PUBLIC_PROXY_PORT="${PUBLIC_PROXY_PORT:-1080}"
+workbook_env="$workbook_env E2E_PUBLIC_PROXY=127.0.0.1:$PUBLIC_PROXY_PORT"
+
+# ExitOnForwardFailure, so a port already in use stops the run here instead of
+# handing the driver a proxy that is not there. Keepalives because this one
+# connection carries the proxy for the whole suite, and a dropped forward would
+# read as every remaining public address going dark at once.
+#
 # if !, not a $? read after the fact: under set -e a failing ssh never reaches the
 # next line, which is the one that prints the driver's own diagnostics.
-if ssh_node "${HOSTS[0]}" "chmod +x ~/run-tofu-examples-e2e.sh; $workbook_env ~/run-tofu-examples-e2e.sh" \
+if ssh -i "$SSH_PRIVATE_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+    -o LogLevel=ERROR -o BatchMode=yes -o ConnectTimeout=15 \
+    -o ServerAliveInterval=30 -o ServerAliveCountMax=6 \
+    -o ExitOnForwardFailure=yes -R "$PUBLIC_PROXY_PORT" "ubuntu@${HOSTS[0]}" \
+    "chmod +x ~/run-tofu-examples-e2e.sh; $workbook_env ~/run-tofu-examples-e2e.sh" \
    > "$STATE_DIR/workbooks.log" 2>&1; then
+    record_workbooks "$STATE_DIR/workbooks.log"
    passed="$(grep -c '^--- PASS' "$STATE_DIR/workbooks.log" || true)"
    ran="$(grep -c '^=== RUN' "$STATE_DIR/workbooks.log" || true)"
-    record "workbooks: PASS ($passed of $ran)"
+    record workbooks PASS "$passed of $ran"
 else
+    # Before die, so a failed suite still gets its per-workbook table: which four
+    # passed is most of what the fifth failing means.
+    record_workbooks "$STATE_DIR/workbooks.log"
+    failed="$(grep -c '^--- FAIL' "$STATE_DIR/workbooks.log" || true)"
+    record workbooks FAIL "$failed failed, see $STATE_DIR/workbooks.log"
    tail -60 "$STATE_DIR/workbooks.log"
    grep -E '^--- (PASS|FAIL)' "$STATE_DIR/workbooks.log" | sed 's/^/  /' || true
    die "the published workbooks failed on $TOPOLOGY; see $STATE_DIR/workbooks.log"
@@ -54,12 +54,18 @@ variable "deployment_name" {
    error_message = "deployment_name must be 1-15 lowercase alphanumeric characters starting with a letter, because it is used as the VCN dns_label."
  }
 }
-# Needs a tenancy-admin principal, which is why it is opt-in. See
-# instance-principal.tf for what the policy grants and why it is narrow.
-variable "enable_instance_principal" {
-  description = "Create the dynamic group and policy that let nodes authenticate as themselves, so no OCI API key is installed on any node."
-  type        = bool
-  default     = false
+# Three values rather than a bool, because using an instance principal and being
+# allowed to create one are different rights. See instance-principal.tf for what
+# the policy grants and why it is narrow.
+variable "instance_principal" {
+  description = "off: nodes use an API key. adopt: nodes authenticate as themselves against a dynamic group and policy that already exist. create: also create those two, which needs a tenancy-admin principal."
+  type        = string
+  default     = "off"
+
+  validation {
+    condition     = contains(["off", "adopt", "create"], var.instance_principal)
+    error_message = "instance_principal must be off, adopt or create."
+  }
 }

 variable "vcn_cidr" {
@@ -25,10 +25,60 @@ OPENTOFU_VERSION="${OPENTOFU_VERSION:-1.11.5}"
 WAN_IP=$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{print $7; exit}')
 SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5 -o LogLevel=ERROR)

+# A guest's public address, reached from where a customer would reach it. Empty
+# means this host is already outside the cluster's own network, which is true of
+# every hypervisor runner. Set it to host:port for a SOCKS5 proxy whose egress is
+# off-network when the host is not: a cloud may refuse a public address from inside
+# its own network, or admit only listed sources, and either fails a healthy cluster
+# from here however well the datapath works.
+PUBLIC_PROXY="${E2E_PUBLIC_PROXY:-}"
+
 CURRENT_WORKBOOK=""

 log() { echo "[$(date +%H:%M:%S)] $*"; }

+# curl, from outside the cluster's network when PUBLIC_PROXY says it has to be.
+# socks5h, not socks5: the proxy resolves, so a name only the far side knows
+# still works.
+curl_public() {
+    if [ -n "$PUBLIC_PROXY" ]; then
+        curl --proxy "socks5h://${PUBLIC_PROXY}" "$@"
+        return
+    fi
+    curl "$@"
+}
+
+# PUBLIC_PROXY is for public addresses only, so this is deliberately not in
+# SSH_OPTS itself: a hop added to every ssh would also divert the bastion's
+# inner ssh to a private address, which no proxy can reach.
+ssh_public() {
+    if [ -n "$PUBLIC_PROXY" ]; then
+        ssh -o "ProxyCommand=nc -X 5 -x ${PUBLIC_PROXY} %h %p" "$@"
+        return
+    fi
+    ssh "$@"
+}
+
+# Fail here rather than letting every assertion time out in turn: a proxy that
+# is named and does not work is indistinguishable, from the far end, from a
+# cluster that is not serving.
+check_public_proxy() {
+    [ -n "$PUBLIC_PROXY" ] || return 0
+    command -v nc >/dev/null 2>&1 || {
+        sudo apt-get install -y -qq netcat-openbsd >/dev/null 2>&1 || {
+            log "public-address probes need nc (netcat-openbsd) for ssh over SOCKS"
+            return 1
+        }
+    }
+    # Something off-network that is not ours, so the answer is about the proxy
+    # and not about the cluster.
+    if ! curl_public -sf -o /dev/null --max-time 20 https://api.github.com/; then
+        log "public-address probes go through SOCKS ${PUBLIC_PROXY}, which cannot reach the internet"
+        return 1
+    fi
+    log "public-address probes go through SOCKS ${PUBLIC_PROXY} (outside this host's network)"
+}
+
 install_tofu() {
    command -v tofu >/dev/null 2>&1 && return 0
    local arch
@@ -110,7 +160,7 @@ trap cleanup EXIT
 wait_for_ssh() {
    local key="$1" host="$2"
    for _ in $(seq 1 30); do
-        if ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${host}" true 2>/dev/null; then
+        if ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${host}" true 2>/dev/null; then
            return 0
        fi
        sleep 5
@@ -124,7 +174,7 @@ wait_for_http_200() {
    local attempts=$((budget / 5))
    for _ in $(seq 1 "$attempts"); do
        local status
-        status=$(curl -sk -o /dev/null -w '%{http_code}' --max-time 5 "$url" 2>/dev/null || echo "000")
+        status=$(curl_public -sk -o /dev/null -w '%{http_code}' --max-time 5 "$url" 2>/dev/null || echo "000")
        if [ "$status" = "200" ]; then
            return 0
        fi
@@ -163,7 +213,7 @@ assert_bastion_private_subnet() {
    }
    # sshd accepts before cloud-init finishes writing ~/.ssh/bastion-demo.pem.
    # Wait up to 180s for user_data to drop the key before hopping.
-    if ! ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${bastion}" \
+    if ! ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${bastion}" \
        'for _ in $(seq 1 36); do [ -s ~/.ssh/bastion-demo.pem ] && exit 0; sleep 5; done; exit 1'; then
        log "  bastion: ~/.ssh/bastion-demo.pem never appeared (cloud-init stalled?)"
        return 1
@@ -172,8 +222,8 @@ assert_bastion_private_subnet() {

    local attempt
    for attempt in $(seq 1 30); do
-        if ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${bastion}" "${inner_ssh} true" 2>/dev/null; then
-            ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${bastion}" "${inner_ssh} id" | grep -q '^uid='
+        if ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${bastion}" "${inner_ssh} true" 2>/dev/null; then
+            ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${bastion}" "${inner_ssh} id" | grep -q '^uid='
            return $?
        fi
        sleep 5
@@ -251,10 +301,10 @@ assert_rds_quickstart() {

    # cloud-init installs psql and writes .pgpass after sshd starts. Bound the
    # wait so a stalled guest still produces diagnostics within the cell budget.
-    if ! ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${ip}" \
+    if ! ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${ip}" \
        'timeout 300 cloud-init status --wait >/dev/null && command -v psql >/dev/null && test -s "$HOME/.pgpass"'; then
        log "  rds-quickstart: client bootstrap incomplete after 300s"
-        ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${ip}" \
+        ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${ip}" \
            'cloud-init status --long; sudo tail -n 40 /var/log/cloud-init-output.log' 2>&1 | \
            sed "s|^|    |" || true
        return 1
@@ -266,7 +316,7 @@ assert_rds_quickstart() {
        "CREATE TABLE IF NOT EXISTS smoke (id int primary key, note text);" \
        "INSERT INTO smoke VALUES (1, 'nightly') ON CONFLICT (id) DO UPDATE SET note = 'nightly';" \
        "SELECT note FROM smoke WHERE id = 1;" |
-        ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${ip}" \
+        ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${ip}" \
            'bash -lc "psql -v ON_ERROR_STOP=1 -tA"' 2>&1); then
        log "  rds-quickstart: SQL round-trip failed"
        printf '%s\n' "$query_output" | sed 's|^|    |'
@@ -292,7 +342,7 @@ dump_s3_webapp_guest() {
    fi
    chmod 600 "$key" 2>/dev/null || true
    log "--- s3-webapp guest: cloud-init status + output log + unit status ---"
-    ssh "${SSH_OPTS[@]}" -i "$key" "ubuntu@${host}" '
+    ssh_public "${SSH_OPTS[@]}" -i "$key" "ubuntu@${host}" '
        echo "== cloud-init status =="; cloud-init status --long 2>/dev/null || true
        echo "== /var/log/cloud-init-output.log (tail 80) =="
        sudo tail -n 80 /var/log/cloud-init-output.log 2>/dev/null || true
@@ -322,14 +372,14 @@ assert_s3_webapp() {

    tmp=$(mktemp)
    echo "nightly-smoke" > "$tmp"
-    if ! curl -sf -F "file=@${tmp};filename=${sentinel}" "http://${ip}/upload" >/dev/null; then
+    if ! curl_public -sf -F "file=@${tmp};filename=${sentinel}" "http://${ip}/upload" >/dev/null; then
        rm -f "$tmp"
        log "  s3-webapp: upload via app failed (IMDS -> STS -> S3 PutObject?)"
        return 1
    fi
    rm -f "$tmp"

-    curl -sf "http://${ip}/" | grep -q "$sentinel" || return 1
+    curl_public -sf "http://${ip}/" | grep -q "$sentinel" || return 1

    # The bucket carries tags, and a clean plan over them is the gate: the AWS
    # provider reads them through S3 Control at https://{account}.{endpoint},
@@ -659,6 +709,7 @@ run_workbook() {
 # --- Main ---

 install_tofu || { log "tofu install failed"; exit 1; }
+check_public_proxy || { log "the public-address vantage point is unusable" ; exit 1; }
 require_rds_image || { log "required RDS AMI is unavailable"; exit 1; }

 INSTANCE_TYPE=$(detect_instance_type)