gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 09:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(oci): both credential routes in the guide, and a nightly topology that proves the principal one (#1173)

Browse Source 
* docs(oci): offer both credential routes, and default the guide to the release

Step 2 was one route only, the scoped API key, with the instance principal
mentioned in a TIP at the end of it. It is now "2. OCI Credentials for
Spinifex" with the two routes as 2.1 and 2.2, a table to choose between them,
and the instance principal first because it leaves no key on any node.

The guide also assumed the dev branch throughout. `git switch dev` and
`--channel dev` are now an optional note for installing a pre-release, and the
documented path is the published release.

* test(oci): add a vm-single-principal topology, and cover the credential branch

Instance-principal authentication was broken for a release while the OCI
nightly stayed green, because the run's two axes never crossed: every topology
that built the address allocator authenticated with an API key, and every run
that authenticated as the instance was dispatched with no external pool, so the
allocator was never constructed and the credential never used.

vm-single-principal closes that. It is vm-single with the credential fixed by
the topology rather than by a separate switch, so the name alone decides it, and
pairing it with --no-external-pool is refused rather than run: without the
allocator there is nothing for it to exercise. It joins the nightly's default
list, and naming it alone is the way to test this one thing in a quarter of the
full run.

A unit test covers the same branch without a tenancy. newClient now reaches its
two constructors through package variables a test can swap, which is the only
way to observe the choice off an OCI instance - both real constructors need
either an instance identity or a key file on disk.

* fix(oci): let the tfvars choose the credential, and stop the guide hardcoding paths

The deploy driver passed instance_principal as a -var, which outranks a
tfvars file, so a deployment that set it there had it silently replaced with
off and the nodes went looking for a key. Same defect class as the shape and
node count, and the same fix: the flag is a choice about one run and still
outranks everything, and without it the deployment's own tfvars decides.

The driver now reads instance_principal from terraform.tfvars and
*.auto.tfvars for its pre-apply gates, taking the last assignment as Terraform
does, and checks that guess against a new Terraform output once the apply has
run. They have to agree, because the credential handoff is skipped on the
strength of it and a wrong answer leaves a node with no credential at all.

The guide now carries one export rather than two more steps. SPX_PRINCIPAL is
set in step 2.1, expanded into the tfvars step 4 already writes, and read back
from there in step 5, so an instance-principal deployment runs the same
command as a key-based one.

Getting into the node needed an address copied out of the log. Every topology
now writes its hosts file, not only the multi-node one that hands it to
install-node.sh, and the run prints the ssh and destroy commands when --keep
leaves the infrastructure up. The guide exports TOPOLOGY once, so the paths
read the same for one VM, three, or bare metal.

* fix(oci): print the ssh key path as ~ rather than expanded

The --keep line read ssh -i /home/you/.ssh/oci-spx, which is the invoking
user's home and differs on every machine. Collapsed to ~ when the key is under
$HOME, left alone when it is not.
This commit is contained in:
Ben Duncan
2026-10-06 13:15:56 -05:00
committed by GitHub
parent cfc7b76cca
commit 7c56855526
9 changed files with 329 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/e2e-cloudvendor-nightly.yml
+3 -3
View File
@@ -10,7 +10,7 @@ on:
workflow_dispatch:
inputs:
topologies:
description: "Space-separated topologies (blank = vm-single vm-multi bm)"
description: "Space-separated topologies (blank = vm-single vm-multi bm vm-single-principal). One name runs one topology."
required: false
workbooks:
description: "Space-separated workbooks (blank = the driver's default list)"
@@ -29,13 +29,13 @@ on:
type: boolean
default: false
auth:
description: "How nodes reach the OCI API: instance_principal (needs the tenancy dynamic group) or api_key (needs the credential hook)"
description: "How nodes reach the OCI API: instance_principal (needs the tenancy dynamic group) or api_key (needs the credential hook). The vm-single-principal topology always uses instance_principal."
required: false
default: instance_principal
type: choice
options: [instance_principal, api_key]
no_external_pool:
description: "Form without the OCI allocator. Skips every workbook needing a public address."
description: "Form without the OCI allocator. Skips every workbook needing a public address. Not applied to vm-single-principal."
required: false
type: boolean
default: false
docs/oci-integration/README.md
+74 -28
View File
@@ -29,7 +29,9 @@ resources:
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [1. Clone the Spinifex repository](#1-clone-the-spinifex-repository)
- [2. Give the nodes a limited scoped OCI credential](#2-give-the-nodes-a-limited-scoped-oci-credential)
- [2. OCI Credentials for Spinifex](#2-oci-credentials-for-spinifex)
- [2.1 Instance principal (preferred)](#21-instance-principal-preferred)
- [2.2 OCI IAM user with scoped permissions](#22-oci-iam-user-with-scoped-permissions)
- [3. Choose deployment method](#3-choose-deployment-method)
- [4. Write your Terraform inputs](#4-write-your-terraform-inputs)
- [5. Deploy](#5-deploy)
@@ -76,7 +78,7 @@ What sits underneath is in [Architecture and Operations](../oci-architecture/REA
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| **An OCI compartment OCID** | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have |
| **OCI credentials in `~/.oci/config`** | For Terraform, to build the infrastructure. `oci setup config` writes one, or write it by hand |
| **An OCI API key for the nodes** | Separate from the above, and the subject of step 2. Without it a node comes up healthy and still cannot give any guest a public address |
| **A runtime credential for the nodes** | Separate from the above, and the subject of step 2: either an instance principal or a scoped API key. Without one a node comes up healthy and still cannot give any guest a public address |
| **Terraform or OpenTofu, `git`, Python 3** | On your workstation. The Python helper is standard library only, so there is nothing to `pip install` |
Check two quotas before you start, because both refuse at apply time rather than at plan time. **Reserved public IPs** are capped at 50 per region across the whole tenancy, and the **compute limit for your shape** starts at zero for bare metal on many new tenancies. [Quotas](../oci-architecture/README.md#quotas) has the commands.
@@ -127,13 +129,11 @@ To begin the deployment of Spinifex on OCI deploy the repository:
```bash
git clone https://github.com/mulgadc/spinifex.git
cd spinifex
git switch dev
cd scripts/terraform/oci-spx
cd spinifex/scripts/terraform/oci-spx
```
> [!IMPORTANT]
> **`git switch dev` is needed today.** OCI support is not in the current release, so the deploy driver is not on `main` yet. From the next release onward `main` carries it and you can skip that line. This chooses the _Terraform and the driver_ only. Which Spinifex build lands on the nodes is step 5.
> [!NOTE]
> **To deploy the development branch instead of the release**, add `git switch dev` before the `cd`, and pass `--channel dev` in step 5. The two choose different things: the branch chooses the Terraform and the deploy driver, and `--channel` chooses which Spinifex build lands on the nodes. Use the release for anything you intend to keep.
Then make a new SSH keypair the deploy installs on every Spinifex node which you can login with:
@@ -149,11 +149,49 @@ Optionally with more than one profile you can define one using the environment v
export OCI_CLI_PROFILE=customprofile
```
## 2. Give the nodes a limited scoped OCI credential
## 2. OCI Credentials for Spinifex
Spinifex calls the OCI API at runtime to register each guest's public address and related functions for Spinifex to support running within an OCI environment. This step gives Spinifex a credential with a limited permission scope for improved security.
Spinifex calls the OCI API at runtime to register each guest's public address, so every node needs a credential that can manage private and public IPs in your compartment. Without one a node forms and passes every health check, then refuses every launch that asks for a public address.
### Generate the key
There are two ways to provide it, and they differ in who has to authorise them rather than in what Spinifex is allowed to do. Both grant [the same three permissions](../oci-architecture/README.md#the-iam-policy).
| | [2.1 Instance principal](#21-instance-principal-preferred) | [2.2 OCI IAM user](#22-oci-iam-user-with-scoped-permissions) |
| ------------------- | ----------------------------------------------------------- | ------------------------------------------------------------ |
| **Key material** | None. Each node authenticates as itself | A private key on every node, at `/etc/spinifex/oci/` |
| **Who can set up** | A tenancy admin, once per tenancy | Anyone with identity rights in the tenancy |
| **Rotation** | Nothing to rotate | Yours to rotate and redistribute |
| **What you set** | `export SPX_PRINCIPAL=adopt`, read by step 4 | Nothing extra; step 5 installs the key for you |
**Use 2.1 if you are a tenancy admin.** It is the better of the two because there is no key to leak, rotate or forget about, and a replaced node needs no handoff. Use 2.2 when your compartment was allocated to you inside someone else's tenancy, which is the common case and the reason 2.2 exists at all.
Do one of the two, not both, then continue to step 3.
### 2.1 Instance principal (preferred)
Each node authenticates with the certificate its own metadata service serves at `/opc/v2/identity/cert.pem`, so no key is written anywhere. Authorising that takes a dynamic group and a policy, and **both are tenancy-root resources** — a compartment-scoped user cannot create them, and the attempt fails with `404-NotAuthorizedOrNotFound` on `CreateDynamicGroup`, which reads like a misconfiguration and is not one.
Create them once, with a tenancy-admin OCI profile, then set one variable:
```bash
./setup-identity.sh --dry-run # always first
./setup-identity.sh
export SPX_PRINCIPAL=adopt
```
**That is all of it.** Step 4 writes `SPX_PRINCIPAL` into your Terraform inputs and step 5 reads it back from there, so neither step needs anything extra from you. Keep the same shell for both, as with `$CID`.
The dynamic group matches on the compartment, so it covers every node you build there and needs no update when a node is replaced. Its Terraform state goes in `.identity/`, separate from the deployment's — keep that directory, as it is the only record of what was created.
Only `setup-identity.sh` needs tenancy-admin rights. Every deployment afterwards runs as your ordinary compartment user.
To remove the group and policy again, `./setup-identity.sh --destroy`. Every node configured for `adopt` loses its credential the moment you do.
### 2.2 OCI IAM user with scoped permissions
This route creates a dedicated OCI user holding only the three permissions Spinifex needs, and installs its key on each node. It needs identity rights in the tenancy but no tenancy-root rights, so it works in an allocated compartment.
#### Generate the key
```bash
openssl genrsa -out ~/.oci/oci_api_key_spx.pem 2048
@@ -164,7 +202,7 @@ openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key_spx.pem | openssl md5 -c
The last command prints the **fingerprint**, which is how OCI names a key everywhere it refers to one. Keep it; you need it in the config file at the end of this step.
### Create a user and a group for it
#### Create a user and a group for it
Both of these are identity resources, so they are created in the tenancy rather than in your compartment. This step requires the prerequisite `oci` CLI tool to be installed and configured with an API scope that includes creating new user accounts and groups.
@@ -206,7 +244,7 @@ An email will be sent to the designated `$ADMIN_EMAIL` defined, you are required
> [!NOTE]
> **If these fail with `NotAuthorizedOrNotFound`, you do not have identity rights in the tenancy**, which is normal when someone allocated you a compartment inside theirs. Ask whoever owns the tenancy to grant you the required permissions.
### Grant the group its three permissions
#### Grant the group its three permissions
Spinifex calls [ten operations](../oci-architecture/README.md#the-iam-policy), and these three statements are what authorise them. Attach the policy to the compartment you are deploying into using the command below. Replace `$CID` with that compartment's OCID.
@@ -225,7 +263,7 @@ oci iam policy create --compartment-id "$CID" --name SpinifexOperators \
> [!NOTE]
> **If this fails with `NotAuthorizedOrNotFound`, confirm `$CID` is the compartment you are deploying into.** It is a `ocid1.compartment.oc1..` OCID, or your tenancy OCID if you deploy into the root compartment. An identity-domain OCID is a different resource and will not work here.
### Upload the public key to that user
#### Upload the public key to that user
```bash
oci iam user api-key upload --user-id "$USER_OCID" \
@@ -249,7 +287,7 @@ This will output:
}
```
### Write the profile and prove it works
#### Write the profile and prove it works
The Spinifex nodes authenticate with an OCI SDK config under the profile name `spinifex`. Write that profile on your workstation first, because it is both what the nodes get and the only way to test the credential before a deployment depends on it:
@@ -276,7 +314,7 @@ An empty response is a pass, because the permission is what is being tested rath
> [!WARNING]
> **Do not set `OCI_CLI_PROFILE=spinifex`.** That variable chooses the profile **Terraform** builds the infrastructure with, and this one cannot: creating a VCN, a subnet and instances is far outside the three grants above, so the apply fails on the first resource. The nodes find the `spinifex` profile by name without being told.
### Validate deployment
#### Validate deployment
Prior to running the Terraform scripts to deploy Spinifex, verify the OCI configuration and environment is correctly setup:
@@ -298,9 +336,6 @@ Step 5 runs this script for real, over SSH, once the nodes are formed. Those two
**The `credential:` line is the one to check.** It has to name the `[spinifex]` profile. If instead it prints two `WARNING:` lines about falling back to "the credential Terraform builds with", the profile was not found and your nodes would get your full-access credential rather than the scoped one you just made.
> [!TIP]
> **A tenancy admin can skip key files altogether** and authenticate the nodes as the instance itself, which leaves no credential on disk to rotate or leak. It needs a dynamic group and a policy created once at the tenancy root, so it is not available in a compartment someone allocated to you. [Instance principal](../oci-architecture/README.md#an-instance-principal-no-key-material-but-it-needs-a-tenancy-admin) covers it, and you then pass `--instance-principal` in step 5 instead of `--credential-hook`.
## 3. Choose deployment method
Choose how to deploy Spinifex on OCI, either as a single node, or at minimum a three node cluster for added resiliency and redundancy.
@@ -333,6 +368,10 @@ cat > terraform.auto.tfvars <<EOF
compartment_ocid = "$CID"
deployment_name = "spinifex"
# How the nodes authenticate to OCI, from step 2. "off" is an API key, which is
# what step 2.2 installs; "adopt" is the instance principal from step 2.1.
instance_principal = "${SPX_PRINCIPAL:-off}"
# Shape and size.
# Bare metal is preferred for large scale production usage.
compute_shape = "VM.Standard.E5.Flex"
@@ -368,24 +407,24 @@ grep compartment_ocid terraform.auto.tfvars
## 5. Deploy
```bash
export TOPOLOGY=vm-single
./validate-topology.sh \
--topology vm-single \
--channel dev \
--topology "$TOPOLOGY" \
--credential-hook ./spx-oci-config.sh \
--skip-workload \
--keep
```
> [!IMPORTANT]
> **`--channel dev` is needed today.** OCI support is not in a published release yet, so the current release cannot allocate an OCI public address. From the next release onward, drop `--channel dev` and the deploy takes `latest`.
The same command covers both credential routes. It reads `instance_principal` from the tfvars you wrote in step 4, so an instance-principal deployment needs no extra argument and `--credential-hook` is simply ignored.
One command builds the infrastructure, installs Spinifex, forms the cluster, configures OCI public addressing and verifies that the address allocator came up.
| Argument | What to pass |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--topology` | `vm-single` for one VM, `vm-multi` for three, `bm` for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission. It supplies a shape and a node count only where your tfvars is silent |
| `--channel` | **`dev` until the next release ships**, because OCI support is not in the current stable release yet. It resolves to the newest pre-release at the moment you run it. `latest` is the default and becomes the right answer once a release carries OCI support |
| `--credential-hook` | `./spx-oci-config.sh`, which installs the credential from step 2 on every node. With an instance principal, replace it with `--instance-principal` and there is no key to install |
| `--channel` | Nothing, normally. It defaults to `latest`, the published release. Pass `dev` to install the newest pre-release instead, or `--version <tag>` to pin an exact build |
| `--credential-hook` | `./spx-oci-config.sh`, which installs the step 2.2 credential on every node. Not used when your tfvars set `instance_principal`, because there is then no key to install |
| `--skip-workload` | Stop once the cluster is verified. Drop it and the driver also runs the published Terraform workbooks on the node, which launches real guests on public addresses and tears each one down again |
| `--keep` | **This is what makes it a deployment rather than a test.** Without it the driver destroys everything at the end, which is right for CI and wrong here |
@@ -394,7 +433,7 @@ One command builds the infrastructure, installs Spinifex, forms the cluster, con
Add `--dry-run` to print the plan and change nothing.
> [!NOTE]
> **If the install step fails to download, pass a tag instead of the channel.** `dev` is the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited and then 404s. Take the newest tag from the [releases page](https://github.com/mulgadc/spinifex/releases) and swap `--channel dev` for `--version <tag>`, which resolves by redirect and cannot trip the same limit. A tag is also what to use for a repeatable deployment, where you want the build pinned rather than current.
> **If `--channel dev` fails to download, pass a tag instead.** `dev` is the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited and then 404s. Take the newest tag from the [releases page](https://github.com/mulgadc/spinifex/releases) and swap `--channel dev` for `--version <tag>`, which resolves by redirect and cannot trip the same limit. A tag is also what to use for a repeatable deployment, where you want the build pinned rather than current.
## 6. Check it worked
@@ -402,6 +441,7 @@ The stages print as they finish, and all of these have to appear:
```text
[validate-vm-single] building
[validate-vm-single] VM.Standard.E5.Flex, 1 node(s), oci_auth from a key file
[validate-vm-single] waiting for cloud-init on each node
[validate-vm-single] installing Spinifex on each node
[validate-vm-single] initializing the single node
@@ -413,20 +453,26 @@ The stages print as they finish, and all of these have to appear:
[validate-vm-single] cluster membership
[validate-vm-single] --skip-workload: stopping before the workbook
[validate-vm-single] --keep: leaving the infrastructure up, no verdict recorded
[validate-vm-single] ssh in with: ssh -i ~/.ssh/oci-spx ubuntu@150.230.13.131
[validate-vm-single] destroy it with: ./validate-topology.sh --topology vm-single --destroy-only
```
**`allocator ready` is the line to look for.** Every line above it also prints on a node that cannot allocate a public address, and that failure stays silent until a guest asks for one.
`running the credential hook` is `spx-oci-config.sh`, and its own output is captured rather than printed. If the deploy stops there, read `credential-hook.log` in the state directory the driver names at startup — its `credential:` line says which credential went onto the nodes.
`running the credential hook` is `spx-oci-config.sh`, and its own output is captured rather than printed. If the deploy stops there, read `credential-hook.log` in the state directory the driver names at startup — its `credential:` line says which credential went onto the nodes. On the instance-principal route those two lines are absent, because there is no handoff to make.
### Access the Spinifex node
SSH to the node at the address on the `allocator ready` line:
The deploy's last line is the command to get in, so you can paste it straight back. The addresses are also on disk, one per line, which is what to read from a script:
```bash
ssh -i ~/.ssh/oci-spx ubuntu@<node-ip>
cat ".validate-$TOPOLOGY/hosts"
ssh -i ~/.ssh/oci-spx ubuntu@"$(head -1 ".validate-$TOPOLOGY/hosts")"
```
`$TOPOLOGY` is the variable you exported in step 5, so this is the same command whether you deployed one VM, three, or bare metal. The first line is the node the cluster formed on.
Then call the AWS surface with the `[spinifex]` profile that `spx admin init` wrote into `~/.aws/credentials`:
```bash
docs/oci-integration/oci-architecture/README.md
+1 -3
View File
@@ -152,8 +152,6 @@ cd scripts/terraform/oci-spx
Then set `instance_principal = "adopt"` in your tfvars and pass `--instance-principal` to the deploy instead of `--credential-hook`. The dynamic group matches on `instance.compartment.id`, so it covers every node you ever build in that compartment and nothing needs updating when nodes are replaced.
**This path is not yet proved end to end.** Every validated run so far used an API key, because the reference tenancy is an allocated child compartment with no tenancy-root rights. On a tenancy where you are admin it should be the better choice, so treat the first run as the test.
## The IAM policy
Spinifex calls exactly ten operations, all of them in `spinifex/cloud/oci/client.go`. Grant no more than these:
@@ -178,7 +176,7 @@ Allow group SpinifexOperators to manage private-ips in compartment <compartment-
Allow group SpinifexOperators to manage public-ips in compartment <compartment-name>
```
These are the same three statements `instance-principal.tf` grants the dynamic group, which is the only version of this policy that has been exercised. Keep them identical: the two credential paths differ in who is authorised, never in what.
`instance-principal.tf` grants the dynamic group these same three statements, so an instance principal can allocate a public address exactly as an IAM user can. The two credential routes differ in who is authorised, never in what.
> [!IMPORTANT]
> **`use vnics` is required, and `read vnics` is not enough.** Registering a secondary private IP is an operation *on a VNIC*, so OCI checks `VNIC_ASSIGN` as well as `PRIVATE_IP_CREATE`, and `VNIC_ASSIGN` lives in the `use` verb. Grant only `read` and every allocation fails with `NotAuthorizedOrNotFound` on a node that looks entirely healthy, which reads as a wrong compartment OCID rather than a missing verb.
scripts/terraform/oci-spx/e2e-oci.sh
+19 -6
View File
@@ -7,7 +7,9 @@
# Configured entirely by environment variables so a dispatch needs no arguments:
#
# SPX_REF Ref to prove. Default: the spinifex checkout's current branch.
# OCI_TOPOLOGIES Space-separated. Default: "vm-single vm-multi bm".
# OCI_TOPOLOGIES Space-separated. Default: "vm-single vm-multi bm
# vm-single-principal". Set one name to test one thing: a
# single VM topology is roughly a quarter of the full run.
# OCI_SOURCE tree (build SPX_REF) or release (the published installer).
# OCI_VERSION With OCI_SOURCE=release, install this exact release tag and
# ignore OCI_CHANNEL. Preferred for anything repeatable: a tag
@@ -22,11 +24,14 @@
# list. Set to "" to skip the workbook phase entirely.
# OCI_INSTANCE_PRINCIPAL Authenticate the node's allocator as the instance
# principal. Default 1, and it needs the tenancy's dynamic
# group to exist.
# group to exist. The vm-single-principal topology ignores it
# and always authenticates that way, which is its purpose.
# OCI_NO_EXTERNAL_POOL 1 to form with no allocator at all, for a tenancy whose
# dynamic group does not exist yet. The allocator and every
# workbook are then recorded SKIPPED, never PASS, so a green
# run with this set is not a claim about guest networking.
# Not applied to vm-single-principal, which the allocator is
# the entire test of; that pairing is refused rather than run.
# OCI_SSH_PUBLIC_KEY / OCI_SSH_PRIVATE_KEY Paths. Default ~/.ssh/oci-spx[.pub].
# OCI_CREDENTIAL_HOOK Executable run on each topology after formation and
# before the pool, as "hook <ssh-key> <host>...". An API-key
@@ -47,7 +52,7 @@ SPINIFEX_ROOT="$(cd "$HERE/../../.." && pwd)"
# under the workspace. Deriving one from the other is only right in the first case.
MULGA_ROOT="${MULGA_ROOT:-$SPINIFEX_ROOT/..}"
TOPOLOGIES="${OCI_TOPOLOGIES:-vm-single vm-multi bm}"
TOPOLOGIES="${OCI_TOPOLOGIES:-vm-single vm-multi bm vm-single-principal}"
SOURCE="${OCI_SOURCE:-tree}"
CHANNEL="${OCI_CHANNEL:-latest}"
INSTALL_VERSION="${OCI_VERSION:-}"
@@ -141,12 +146,14 @@ summary() {
# In the header, not left to a SKIPPED row further down. A run with no allocator
# is not evidence about public addressing, and that is precisely the claim a
# reader of a green table would otherwise take from it.
# Said per topology rather than once for the run: vm-single-principal pins its
# own credential, so a single claim over a mixed run is wrong for one of them.
if [ "${OCI_NO_EXTERNAL_POOL:-0}" = 1 ]; then
printf '> **No external address pool.** Public-address allocation and every workbook needing a public address were skipped, so nothing here speaks to guest ingress.\n\n'
printf '> **No external address pool.** Public-address allocation and every workbook needing a public address were skipped, so nothing here speaks to guest ingress. `vm-single-principal`, if it ran, is the exception and did build one.\n\n'
elif [ "${OCI_INSTANCE_PRINCIPAL:-1}" = 1 ]; then
printf 'Allocator authenticated as the instance principal.\n\n'
else
printf 'Allocator authenticated with an API key.\n\n'
printf 'Allocator authenticated with an API key, except `vm-single-principal` which authenticates as the instance.\n\n'
fi
local topology results workbooks gate status detail name secs
@@ -275,7 +282,13 @@ for topology in $TOPOLOGIES; do
# Two separate choices. --no-external-pool is the one that forms without any
# allocator, for a tenancy whose dynamic group does not exist yet; it records
# the allocator and the workbooks as SKIPPED rather than passing them.
if [ "${OCI_NO_EXTERNAL_POOL:-0}" = 1 ]; then
#
# vm-single-principal opts out of both: it sets its own credential, and a run
# of it without an allocator would exercise nothing. validate-topology.sh
# refuses that pairing, so skipping it here keeps a mixed run from aborting.
if [ "$topology" = vm-single-principal ]; then
:
elif [ "${OCI_NO_EXTERNAL_POOL:-0}" = 1 ]; then
args+=(--no-external-pool)
elif [ "${OCI_INSTANCE_PRINCIPAL:-1}" = 1 ]; then
args+=(--instance-principal)
scripts/terraform/oci-spx/outputs.tf
+5 -2
View File
@@ -8,9 +8,12 @@ output "security_list_ocids" { value = { public = oci_core_security_list.public.
output "compute_ocids" { value = [for instance in oci_core_instance.mulgadc : instance.id] }
output "data_volume_ocids" { value = [for volume in oci_core_volume.mulgadc_data : volume.id] }
# Read back rather than assumed: the shape can come from a tfvars file, so this is
# the only place that knows what was actually built.
# Read back rather than assumed: both can come from a tfvars file, so this is the
# only place that knows what was actually built. The deploy driver skips the
# credential handoff on instance_principal, so a wrong guess leaves a node with no
# credential at all.
output "compute_shape" { value = var.compute_shape }
output "instance_principal" { value = var.instance_principal }
# What the Spinifex deploy path consumes. The primary VNIC is the host plane, so
# these are the addresses update-nodes.sh and install-node.sh connect to.
scripts/terraform/oci-spx/validate-topology.sh
+82 -10
View File
@@ -33,21 +33,31 @@ WORKBOOKS_SET=0
CHANNEL=latest
INSTALL_VERSION=""
WORKBOOKS=""
# Declared here, not at the apply that fills it: the EXIT trap reads it, and under
# set -u an unset array is an error rather than an empty one.
HOSTS=()
TOPOLOGIES="bm vm-single vm-multi"
TOPOLOGIES="bm vm-single vm-multi vm-single-principal"
# What a topology is worth when nothing says otherwise, so a CI run of a named
# topology is the same every time. Supplied as TF_VAR_, which terraform.auto.tfvars
# outranks, so a deployment sets its shape and node count in that file and no
# flag has to carry them.
# outranks, so a deployment sizes itself in that file and no flag has to carry it.
#
# TOPO_FORCE_PRINCIPAL is the exception: vm-single-principal exists to exercise
# instance-principal authentication, so the credential is the topology rather than
# a separate switch that could be left at the other value.
#
# A case rather than an associative array: macOS ships bash 3.2, where declare -A
# is an indexed assignment and the key is read as an arithmetic variable.
topo_defaults() {
TOPO_FORCE_PRINCIPAL=0
case "$1" in
bm) TOPO_DEFAULT_SHAPE="BM.Standard.E2.64"; TOPO_DEFAULT_NODES=1 ;;
vm-single) TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=1 ;;
vm-multi) TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=3 ;;
vm-single-principal)
TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=1
TOPO_FORCE_PRINCIPAL=1 ;;
*) return 1 ;;
esac
}
@@ -60,18 +70,22 @@ die() {
usage() {
cat >&2 <<EOF
usage: ${0##*/} --topology <bm|vm-single|vm-multi> [options]
usage: ${0##*/} --topology <bm|vm-single|vm-multi|vm-single-principal> [options]
Builds the topology, installs Spinifex, forms the cluster, runs a Terraform
workbook against it, then destroys everything.
--topology <name> Required. No default: a command aimed at the wrong
topology is the easiest expensive mistake here.
vm-single-principal is vm-single authenticating as the
instance, and implies --instance-principal.
--ssh-public-key PATH Public half installed on each node. Default $SSH_PUBLIC_KEY
--ssh-private-key PATH Private half used to reach them. Default $SSH_PRIVATE_KEY
--instance-principal Configure the pool with oci_auth="instance_principal"
instead of a key file. Needs the dynamic group and
policy to exist already; see instance-principal.tf.
Not needed when instance_principal is set in your
tfvars, which this reads; the flag outranks it.
--credential-hook PATH Executable run after formation, before the pool, as
"hook <ssh-key> <host>...". Where an API-key deployment
installs its credential. Default \$OCI_CREDENTIAL_HOOK.
@@ -136,9 +150,44 @@ if [ -n "$DISTRO" ] || [ -n "$SETUP_SH" ]; then
[ -r "$SETUP_SH" ] || die "--setup-sh is not readable: '$SETUP_SH'"
fi
if [ "$TOPO_FORCE_PRINCIPAL" = 1 ]; then
INSTANCE_PRINCIPAL=1
# The pool is the whole point of this topology: with no allocator built, the
# credential is never used and a green run says nothing about it. That is
# exactly how the auth branch stayed broken through passing nightlies.
[ "$SKIP_POOL" != 1 ] \
|| die "--no-external-pool makes $TOPOLOGY meaningless: the allocator is what exercises the instance principal, so a run without it proves nothing"
fi
# Terraform's own order, as far as this script needs it: the last assignment across
# terraform.tfvars then *.auto.tfvars alphabetically is the one that applies. Read
# because the credential-hook gate has to run before the apply spends anything; the
# apply's own output is checked against this afterwards.
tfvars_instance_principal() {
local file mode="" found=""
for file in "$HERE/terraform.tfvars" "$HERE"/*.auto.tfvars; do
[ -r "$file" ] || continue
mode=$(sed -n 's/^[[:space:]]*instance_principal[[:space:]]*=[[:space:]]*"\([a-z]*\)".*/\1/p' \
"$file" | tail -1)
[ -n "$mode" ] && found="$mode"
done
[ -n "$found" ] || return 1
printf '%s\n' "$found"
}
# adopt, never create: this script's state is destroyed at the end of every run, and
# the dynamic group and policy outlive every topology. setup-identity.sh owns them.
PRINCIPAL_MODE=$([ "$INSTANCE_PRINCIPAL" = 1 ] && echo adopt || echo off)
#
# A flag goes in as -var and outranks the tfvars, because it is a choice about this
# run. Without one the deployment's own tfvars decides, so a guide can set the
# credential in the file it already writes and no flag has to carry it.
tf_principal_var=()
if [ "$INSTANCE_PRINCIPAL" = 1 ]; then
PRINCIPAL_MODE=adopt
tf_principal_var=(-var "instance_principal=$PRINCIPAL_MODE")
else
PRINCIPAL_MODE="$(tfvars_instance_principal || echo off)"
fi
# What the apply actually built, filled in from the hosts_file output once it has.
# Until then these are what the topology asked for, which is all there is to say.
@@ -174,10 +223,8 @@ tf() {
# terraform output takes -state but not -var, so the two sets are kept apart. They
# were one set once, and the -var made output fail into the default state file.
tf_state=(-state "$STATE_DIR/terraform.tfstate")
# instance_principal stays a -var, and outranks any tfvars, because --instance-principal
# is a choice about this run rather than about the infrastructure's size.
tf_vars=(
-var "instance_principal=$PRINCIPAL_MODE"
"${tf_principal_var[@]}"
"${tf_state[@]}"
)
@@ -243,6 +290,17 @@ cleanup() {
local rc=$?
if [ "$KEEP" = 1 ]; then
log "--keep: leaving the infrastructure up, no verdict recorded"
# The command to get in, printed rather than left to be assembled from an
# address elsewhere in the log. HOSTS is empty if we never got as far as
# the apply, which is why this is guarded rather than assumed.
if [ "${#HOSTS[@]}" -gt 0 ]; then
# ~ rather than the expanded path, so the line is the same on every
# machine and reads as something a person would type.
local key="$SSH_PRIVATE_KEY"
case "$key" in "$HOME"/*) key="~${key#"$HOME"}" ;; esac
log "ssh in with: ssh -i $key ubuntu@${HOSTS[0]}"
log "destroy it with: $0 --topology $TOPOLOGY --destroy-only"
fi
return
fi
# Only on a failure, and said out loud: the cost of forgetting is an OCI bare
@@ -361,6 +419,11 @@ for host in "${HOSTS[@]}"; do
|| die "the hosts_file output is not an address: '$host' -- something on the wrapper's stdout is in the capture"
done
# Written for every topology, not just the multi-node one that passes it to
# install-node.sh: it is also how an operator gets the addresses afterwards
# without a terraform invocation and the right -state path.
printf '%s\n' "${HOSTS[@]}" > "$STATE_DIR/hosts"
# node_count can come from a tfvars file, so the hosts the apply produced are the
# authority on how many there are. Every later count check reads this.
if [ "${#HOSTS[@]}" != "$NODES" ]; then
@@ -369,8 +432,18 @@ if [ "${#HOSTS[@]}" != "$NODES" ]; then
fi
SHAPE=$(tf output -raw "${tf_state[@]}" compute_shape) \
|| die "could not read the compute_shape output from $STATE_DIR/terraform.tfstate"
# Terraform is the authority on which credential was built, and the reader above is
# a guess made before the apply. They have to agree: the hook is skipped on the
# strength of that guess, and a node left with no credential at all would otherwise
# fail much later as an allocator that cannot resolve its VNIC.
built_principal=$(tf output -raw "${tf_state[@]}" instance_principal) \
|| die "could not read the instance_principal output from $STATE_DIR/terraform.tfstate"
[ "$built_principal" = "$PRINCIPAL_MODE" ] \
|| die "read instance_principal='$PRINCIPAL_MODE' from the tfvars and the apply built '$built_principal'; set it in one place"
log "hosts: ${HOSTS[*]}"
log "$SHAPE, $NODES node(s)"
log "$SHAPE, $NODES node(s), oci_auth $([ "$PRINCIPAL_MODE" = off ] && echo "from a key file" || echo "as the instance principal")"
# cloud-init owns the volume, the ports and br-wan, and all three are units now, so
# the question is whether they converged rather than whether a runcmd happened to
@@ -462,7 +535,6 @@ if [ "$NODES" = 1 ]; then
' > "$STATE_DIR/form.log" 2>&1 \
|| die "single-node init failed; see $STATE_DIR/form.log"
else
printf '%s\n' "${HOSTS[@]}" > "$STATE_DIR/hosts"
log "forming the cluster"
"$REPO_ROOT/scripts/install-node.sh" \
--hosts-file "$STATE_DIR/hosts" \
spinifex/network/external/ocinet/export_test.go
Vendored
+36
View File
@@ -0,0 +1,36 @@
package ocinet
import (
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
"github.com/mulgadc/spinifex/spinifex/network/external"
)
// NewClient is the credential choice a pool makes, which is worth testing on its
// own: both real constructors need an instance or a key file, so the branch is
// otherwise observable only on OCI.
var NewClient = newClient
// SwapClientConstructors replaces both so a test can see which one a pool selects,
// and returns a function restoring them.
func SwapClientConstructors(
principal func() (oci.Client, error),
configFile func(path, profile string) (oci.Client, error),
) func() {
savedPrincipal, savedConfigFile := newInstancePrincipalClient, newConfigFileClient
newInstancePrincipalClient, newConfigFileClient = principal, configFile
return func() {
newInstancePrincipalClient, newConfigFileClient = savedPrincipal, savedConfigFile
}
}
// PoolWithAuth is the smallest source="oci" pool that reaches the credential
// choice, so a test states the auth it means and nothing else.
func PoolWithAuth(auth, configFile, profile string) external.ExternalPoolConfig {
return external.ExternalPoolConfig{
Name: "oci-public",
Source: external.SourceOCI,
OCIAuth: auth,
OCIConfigFile: configFile,
OCIConfigProfile: profile,
}
}
spinifex/network/external/ocinet/wiring.go
Vendored
+10 -2
View File
@@ -79,11 +79,19 @@ func FromPoolConfig(ctx context.Context, js jetstream.JetStream, pool external.E
func newClient(pool external.ExternalPoolConfig) (oci.Client, error) {
if pool.UsesInstancePrincipal() {
slog.Info("ocinet authenticating as the instance principal", "pool", pool.Name)
return oci.NewInstancePrincipalClient()
return newInstancePrincipalClient()
}
return oci.NewConfigFileClient(pool.OCIConfigFile, pool.OCIConfigProfile)
return newConfigFileClient(pool.OCIConfigFile, pool.OCIConfigProfile)
}
// Indirected so a test can assert which credential a pool selects. Both real
// constructors need an instance or a key file, so the branch is otherwise only
// observable on OCI, and it was wrong there for a release without anyone seeing.
var (
newInstancePrincipalClient = oci.NewInstancePrincipalClient
newConfigFileClient = oci.NewConfigFileClient
)
// resolveVNICID turns whichever VNIC key the operator set into an OCID.
func resolveVNICID(ctx context.Context, pool external.ExternalPoolConfig) (string, error) {
if pool.OCIVNICID != "" {
spinifex/network/external/ocinet/wiring_test.go
Vendored
+99
View File
@@ -0,0 +1,99 @@
package ocinet_test
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
"github.com/mulgadc/spinifex/spinifex/network/external"
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
)
// A pool naming oci_auth="instance_principal" must not read a key file. It did
// for a release: the allocator failed with "open ~/.oci/config: no such file or
// directory" on a node that had been given no key on purpose.
func TestNewClientSelectsTheCredentialThePoolNames(t *testing.T) {
for _, tc := range []struct {
name string
auth string
configFile string
profile string
wantPrincipal bool
}{
{
name: "instance principal",
auth: external.OCIAuthInstancePrincipal,
wantPrincipal: true,
},
{
// A pool configured for an instance principal still must not fall
// back to a key file when one happens to be configured as well.
name: "instance principal with a config file also set",
auth: external.OCIAuthInstancePrincipal,
configFile: "/etc/spinifex/oci/config",
profile: "spinifex",
wantPrincipal: true,
},
{
name: "config file",
auth: external.OCIAuthConfigFile,
configFile: "/etc/spinifex/oci/config",
profile: "spinifex",
},
{
// The default, so a pool written before oci_auth existed keeps
// authenticating from its key file.
name: "unset auth defaults to the config file",
configFile: "/etc/spinifex/oci/config",
profile: "spinifex",
},
} {
t.Run(tc.name, func(t *testing.T) {
var principalCalls, configFileCalls int
var gotPath, gotProfile string
restore := ocinet.SwapClientConstructors(
func() (oci.Client, error) {
principalCalls++
return oci.NewFake(), nil
},
func(path, profile string) (oci.Client, error) {
configFileCalls++
gotPath, gotProfile = path, profile
return oci.NewFake(), nil
},
)
defer restore()
client, err := ocinet.NewClient(ocinet.PoolWithAuth(tc.auth, tc.configFile, tc.profile))
require.NoError(t, err)
require.NotNil(t, client)
if tc.wantPrincipal {
assert.Equal(t, 1, principalCalls, "should authenticate as the instance")
assert.Zero(t, configFileCalls, "must not read a key file")
return
}
assert.Equal(t, 1, configFileCalls, "should authenticate from the key file")
assert.Zero(t, principalCalls)
assert.Equal(t, tc.configFile, gotPath, "the pool's oci_config_file should reach the SDK")
assert.Equal(t, tc.profile, gotProfile, "the pool's oci_config_profile should reach the SDK")
})
}
}
// The error has to name the pool: a node carries one OCI pool today and may carry
// several, and "no such file or directory" alone says nothing about which.
func TestNewClientReturnsTheConstructorError(t *testing.T) {
wantErr := errors.New("no such file or directory")
restore := ocinet.SwapClientConstructors(
func() (oci.Client, error) { return nil, errors.New("not on an instance") },
func(string, string) (oci.Client, error) { return nil, wantErr },
)
defer restore()
_, err := ocinet.NewClient(ocinet.PoolWithAuth(external.OCIAuthConfigFile, "/missing", "spinifex"))
require.ErrorIs(t, err, wantErr)
}