diff --git a/docs/oci-integration/README.md b/docs/oci-integration/README.md index a0197cfd9..a35b82874 100644 --- a/docs/oci-integration/README.md +++ b/docs/oci-integration/README.md @@ -22,15 +22,15 @@ resources: # Running Spinifex on Oracle Cloud Infrastructure -> Run the AWS surface — EC2, EBS, S3, VPC — inside your own OCI tenancy, with guests that get real, publicly reachable addresses through OCI's API. +> Run the AWS surface (EC2, EBS, S3, VPC) inside your own OCI tenancy, with guests that get real, publicly reachable addresses through OCI's API. ## Table of Contents - [Overview](#overview) - [Prerequisites](#prerequisites) -- [1. Get the configuration](#1-get-the-configuration) -- [2. Give the nodes an OCI credential](#2-give-the-nodes-an-oci-credential) -- [3. Choose one node or three](#3-choose-one-node-or-three) +- [1. Clone the Spinifex repository](#1-clone-the-spinifex-repository) +- [2. Give the nodes a limited scoped OCI credential](#2-give-the-nodes-a-limited-scoped-oci-credential) +- [3. Choose deployment method](#3-choose-deployment-method) - [4. Write your Terraform inputs](#4-write-your-terraform-inputs) - [5. Deploy](#5-deploy) - [6. Check it worked](#6-check-it-worked) @@ -43,26 +43,86 @@ resources: ## Overview -Spinifex brings core AWS services to hardware you control: a node runs the Spinifex daemon, QEMU/KVM for guests, OVN for VPC networking, [Predastore](https://github.com/mulgadc/predastore) for S3 and [Viperblock](https://github.com/mulgadc/viperblock) for EBS, all behind a SigV4 endpoint the ordinary AWS SDKs and CLI talk to unmodified. +Spinifex is an open-source infrastructure platform that brings core AWS services to bare-metal, edge, and on-prem environments. It serves the EC2, EBS, S3, VPC, IAM and STS APIs over a SigV4 endpoint of your own, so the AWS CLI, the AWS SDKs and the AWS Terraform provider all work against it unmodified. Nothing a tenant has written has to be rewritten to run on it. -**Terraform does the whole deployment** — the VCN, the block volumes, both VNICs per node, the install, the cluster formation and the OCI public-address allocator. Six steps below, one command, about ten minutes for a single node. +This guide installs Spinifex on Oracle Cloud Infrastructure, on one instance or three, with guests that get real public addresses from OCI's own pool. The install is the same product you would put on your own servers, so [Single-Node Install](/docs/install) and [Multi-Node Install](/docs/install-multi-node) describe the same formation, storage and networking. What differs here is the layer underneath, because a VCN is not an Ethernet segment and public addresses come from an API rather than a range you choose. -Everything underneath — the architecture, the IAM policy, the quotas, every variable, the troubleshooting — is in [Architecture and Operations](../oci-architecture/README.md). +**Why run it on OCI:** + +- **Cost.** OCI prices compute and block storage below the large US clouds, and one Spinifex node serves a tenant's whole fleet as QEMU guests on hardware you are billed for once. +- **Egress.** Oracle's outbound-transfer allowance and per-GB rate are far cheaper than the majors, and guest-to-guest traffic rides the private plane, where it is not billed as internet egress at all. +- **Portability.** Your tenants code against Spinifex, not against OCI, so the same workloads move to your own rack or an edge site later as a deployment decision rather than a rewrite. +- **Real public addresses.** Spinifex registers each guest's address with OCI, so a guest is reachable on the internet rather than hidden behind a shared NAT. + +**What gets installed**, on every node: + +- Spinifex daemon and CLI +- QEMU/KVM for guests +- OVN and Open vSwitch for VPC networking +- [Predastore](https://github.com/mulgadc/predastore), S3-compatible object storage +- [Viperblock](https://github.com/mulgadc/viperblock), EBS-compatible block storage +- The OCI public-address allocator, which is the one component specific to this platform + +Nodes run **Ubuntu 26.04**, resolved at plan time as the newest Canonical platform image for your shape and region. Oracle Linux support as the host OS to run Spinifex is under active development. + +**Terraform does the whole deployment:** the VCN, the block volumes, both VNICs per node, the Spinifex install, the cluster formation and the allocator. Six steps below, one command, about ten minutes for a single node. + +What sits underneath is in [Architecture and Operations](../oci-architecture/README.md), which also carries the IAM policy, the quotas, every variable and the troubleshooting. ## Prerequisites -| | | -| --- | --- | -| **An OCI compartment OCID** | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have | -| **OCI credentials in `~/.oci/config`** | For Terraform, to build the infrastructure. `oci setup config` writes one, or write it by hand | -| **An OCI API key for the nodes** | Separate from the above, and step 2. Without it a node forms, looks healthy, and cannot give any guest a public address | -| **Terraform or OpenTofu, `git`, Python 3** | On your workstation. The Python helper is standard library only, so there is nothing to `pip install` | +| | | +| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | +| **An OCI compartment OCID** | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have | +| **OCI credentials in `~/.oci/config`** | For Terraform, to build the infrastructure. `oci setup config` writes one, or write it by hand | +| **An OCI API key for the nodes** | Separate from the above, and the subject of step 2. Without it a node comes up healthy and still cannot give any guest a public address | +| **Terraform or OpenTofu, `git`, Python 3** | On your workstation. The Python helper is standard library only, so there is nothing to `pip install` | -Check two quotas before you start, because both refuse at apply time rather than at plan time: **reserved public IPs** (50 per region, tenancy-wide) and the **compute limit for your shape**, which on a new tenancy is often zero for bare metal. [Quotas](../oci-architecture/README.md#quotas) has the commands. +Check two quotas before you start, because both refuse at apply time rather than at plan time. **Reserved public IPs** are capped at 50 per region across the whole tenancy, and the **compute limit for your shape** starts at zero for bare metal on many new tenancies. [Quotas](../oci-architecture/README.md#quotas) has the commands. + +For the workstation (Linux, MacOS, Windows [WSL]) that will be used to deploy Spinifex on OCI the following prerequisites are required. + +### OCI CLI tool + +Ensure the `oci` CLI tool is pre-installed. Follow the install guide at: [https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm) + +Once installed configure your OCI credentials (~/.oci/config) which will be used in later steps in the installation tutorial. + +`oci setup config` + +Once setup validate your OCI credentials work as expected. `$CID` is the compartment you are deploying into, which is a `ocid1.compartment.oc1..` OCID, or your tenancy OCID if that compartment is the root. The same value is used throughout this guide and again as `compartment_ocid` in step 4. + +```sh +export CID="ocid1.compartment.oc1..xxx" +oci compute shape list --compartment-id $CID +``` + +Expected output to confirm API key works as expected: + +```json +{ + "data": [ + { + "baseline-ocpu-utilizations": null, + "billing-type": "PAID", + ... + } + ] +} +``` + +### Install Terraform / OpenTofu + +The next prerequisite is to install Terraform or OpenTofu to deploy the Spinifex stack on OCI. + +- Terraform - [https://developer.hashicorp.com/terraform/install](https://developer.hashicorp.com/terraform/install) +- OpenTofu - [https://opentofu.org/docs/intro/install/](https://opentofu.org/docs/intro/install/) ## Instructions -## 1. Get the configuration +## 1. Clone the Spinifex repository + +To begin the deployment of Spinifex on OCI deploy the repository: ```bash git clone https://github.com/mulgadc/spinifex.git @@ -72,70 +132,189 @@ cd scripts/terraform/oci-spx ``` > [!IMPORTANT] -> **`git switch dev` is needed today.** OCI support is not in the current release, so the deploy driver is not on `main` yet. From the next release onward `main` carries it and you can skip that line. This chooses the _Terraform and the driver_ only — which Spinifex build lands on the nodes is step 5. +> **`git switch dev` is needed today.** OCI support is not in the current release, so the deploy driver is not on `main` yet. From the next release onward `main` carries it and you can skip that line. This chooses the _Terraform and the driver_ only. Which Spinifex build lands on the nodes is step 5. -Then make the keypair the deploy installs on every node and logs in with. It refuses to start without one, rather than building instances you cannot reach: +Then make a new SSH keypair the deploy installs on every Spinifex node which you can login with: ```bash ssh-keygen -t ed25519 -f ~/.ssh/oci-spx -N '' -C spinifex-oci ``` -Terraform reads your OCI credential from `~/.oci/config`, taking the tenancy, the user, the fingerprint, the key and **the region** from the profile — so the region you deploy into is the profile's, not something you set in step 4. A config holding a single `[DEFAULT]` profile, which is what `oci setup config` writes, needs nothing further. With more than one profile, name the one you mean: +Terraform reads your OCI credential from `~/.oci/config` which is a prerequisites step above, taking the tenancy, the user, the fingerprint, the key and **the region** from the profile. A config holding a single `[DEFAULT]` profile, which is what `oci setup config` writes, needs nothing further. + +Optionally with more than one profile you can define one using the environment variable `OCI_CLI_PROFILE` ```bash -./validate-topology.sh --oci-profile mycorp ... # or export OCI_CLI_PROFILE=mycorp +export OCI_CLI_PROFILE=customprofile ``` -A name you pass that the config does not hold is an error listing the profiles it does hold. It never falls back, because the failure mode worth preventing is building a cluster in whichever tenancy happened to be first. +## 2. Give the nodes a limited scoped OCI credential -## 2. Give the nodes an OCI credential +Spinifex calls the OCI API at runtime to register each guest's public address and related functions for Spinifex to support running within an OCI environment. This step gives Spinifex a credential with a limited permission scope for improved security. -Spinifex calls the OCI API at runtime to register each guest's public address. Create an API key for that, if you do not already have one: +### Generate the key ```bash -openssl genrsa -out ~/.oci/oci_api_key.pem 2048 -chmod 600 ~/.oci/oci_api_key.pem -openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem -openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 -c # the fingerprint +openssl genrsa -out ~/.oci/oci_api_key_spx.pem 2048 +chmod 600 ~/.oci/oci_api_key_spx.pem +openssl rsa -pubout -in ~/.oci/oci_api_key_spx.pem -out ~/.oci/oci_api_key_spx_public.pem +openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key_spx.pem | openssl md5 -c ``` -Upload the public key under **Identity → Users → API Keys**, and grant it the [ten operations Spinifex uses](../oci-architecture/README.md#the-iam-policy) in your compartment — no more. +The last command prints the **fingerprint**, which is how OCI names a key everywhere it refers to one. Keep it; you need it in the config file at the end of this step. -**The deploy installs the credential on each node by calling a small script of yours**, so no key material ever goes into user-data or Terraform state, where it would be readable from instance metadata for the life of the instance. Write one that puts these two files on each host it is given: +### Create a user and a group for it -| Path | Mode | Contents | -| --- | --- | --- | -| `/etc/spinifex/oci/oci_api_key.pem` | `0640 root:spinifex` | The private key | -| `/etc/spinifex/oci/config` | `0640 root:spinifex` | An OCI SDK config, profile **`[spinifex]`**, naming the key by path | +Both of these are identity resources, so they are created in the tenancy rather than in your compartment. This step requires the prerequisite `oci` CLI tool to be installed and configured with an API scope that includes creating new user accounts and groups. -The profile name must be `spinifex`. It is called as `your-hook ...` — the first argument is the private key to reach the nodes with, the rest are the hosts — and it must exit non-zero if any host failed, so a missing credential stops the deploy rather than surfacing later as a launch that cannot get an address. [Credentials](../oci-architecture/README.md#credentials-an-api-key-or-an-instance-principal) has the config file's exact contents. +Replace `admin@yourdomain.com` with your administrator email which will be linked to the new OCI user account. -`chmod +x` it. The deploy checks that before it builds anything, so a hook that is missing or not executable costs you a few seconds rather than forty minutes and a bare-metal bill. +```bash +export ADMIN_EMAIL=admin@yourdomain.com -**Nothing in this repository ships that script, deliberately.** A credential belongs to whoever owns it, so the hook is yours to write and yours to keep. +USER_OCID=$(oci iam user create --name spinifex-allocator \ + --email $ADMIN_EMAIL \ + --description "Spinifex external-address allocator" \ + --query 'data.id' --raw-output) + +GROUP_OCID=$(oci iam group create --name SpinifexOperators \ + --description "May manage the private and public IPs behind Spinifex external addresses" \ + --query 'data.id' --raw-output) + +oci iam group add-user --group-id "$GROUP_OCID" --user-id "$USER_OCID" +``` + +On success the output will include: + +```json +{ + "data": { + "compartment-id": "ocid1.tenancy.oc1..xxx", + "group-id": "ocid1.group.oc1..xxx", + "id": "ocid1.groupmembership.oc1..axxx", + "inactive-status": null, + "lifecycle-state": "ACTIVE", + "time-created": "2026-10-05T19:39:48.478000+00:00", + "user-id": "ocid1.user.oc1..xxx" + } +} +``` + +An email will be sent to the designated `$ADMIN_EMAIL` defined, you are required to click the "Activate Your Account" link to fully activate the new account in OCI. + +> [!NOTE] +> **If these fail with `NotAuthorizedOrNotFound`, you do not have identity rights in the tenancy**, which is normal when someone allocated you a compartment inside theirs. Ask whoever owns the tenancy to grant you the required permissions. + +### Grant the group its three permissions + +Spinifex calls [ten operations](../oci-architecture/README.md#the-iam-policy), and these three statements are what authorise them. Attach the policy to the compartment you are deploying into using the command below. Replace `$CID` with that compartment's OCID. + +```bash +export CID="ocid1.compartment.oc1..xxx" + +oci iam policy create --compartment-id "$CID" --name SpinifexOperators \ + --description "Spinifex external-address allocation" \ + --statements '[ + "Allow group SpinifexOperators to use vnics in compartment id '"$CID"'", + "Allow group SpinifexOperators to manage private-ips in compartment id '"$CID"'", + "Allow group SpinifexOperators to manage public-ips in compartment id '"$CID"'" + ]' +``` + +> [!NOTE] +> **If this fails with `NotAuthorizedOrNotFound`, confirm `$CID` is the compartment you are deploying into.** It is a `ocid1.compartment.oc1..` OCID, or your tenancy OCID if you deploy into the root compartment. An identity-domain OCID is a different resource and will not work here. + +### Upload the public key to that user + +```bash +oci iam user api-key upload --user-id "$USER_OCID" \ + --key-file ~/.oci/oci_api_key_spx_public.pem +``` + +This will output: + +```json +{ + "data": { + "fingerprint": "xxx", + "inactive-status": null, + "key-id": "ocid1.tenancy.oc1..xxx", + "key-value": "-----BEGIN PUBLIC KEY-----xxxx-----END PUBLIC KEY-----", + "lifecycle-state": "ACTIVE", + "time-created": "2026-10-05T19:51:58.843000+00:00", + "user-id": "ocid1.user.oc1..axxx" + }, + "etag": "xxx" +} +``` + +### Write the profile and prove it works + +The Spinifex nodes authenticate with an OCI SDK config under the profile name `spinifex`. Write that profile on your workstation first, because it is both what the nodes get and the only way to test the credential before a deployment depends on it: + +```ini +# ~/.oci/config +[spinifex] +user= +fingerprint= +tenancy= +region= +key_file=~/.oci/oci_api_key_spx.pem +``` + +Then confirm the key, the user, the group and the policy all line up. This call needs exactly the `public-ips` grant above, so it fails in the same way a node would: + +```bash +oci network public-ip list --compartment-id "$CID" --scope REGION --all --profile spinifex +``` + +An empty response is a pass, because the permission is what is being tested rather than the contents. `NotAuthorizedOrNotFound` means the policy is not in force: check that the statements name the compartment by `compartment id`, and that the user is actually in the group. + +**Do this before step 5.** confirm the new OCI account can correctly authenticate and issue the required API calls for Spinifex to function correctly once installed. + +> [!WARNING] +> **Do not set `OCI_CLI_PROFILE=spinifex`.** That variable chooses the profile **Terraform** builds the infrastructure with, and this one cannot: creating a VCN, a subnet and instances is far outside the three grants above, so the apply fails on the first resource. The nodes find the `spinifex` profile by name without being told. + +### Validate deployment + +Prior to running the Terraform scripts to deploy Spinifex, verify the OCI configuration and environment is correctly setup: + +```bash +./spx-oci-config.sh --dry-run ~/.ssh/oci-spx +``` + +Expected output: + +```text +[spx-oci-config] credential: the [spinifex] profile in ~/.oci/config +[spx-oci-config] user: ocid1.user.oc1..aaaa... +[spx-oci-config] fingerprint: a1:b2:... +[spx-oci-config] region: ap-sydney-1 +[spx-oci-config] would write /etc/spinifex/oci/config and /etc/spinifex/oci/oci_api_key.pem as 0640 root:spinifex on: (no hosts given) +``` + +Step 5 runs this script for real, over SSH, once the nodes are formed. Those two files are what the allocator authenticates with, and the `spinifex` profile name in the second one is what it looks for. + +**The `credential:` line is the one to check.** It has to name the `[spinifex]` profile. If instead it prints two `WARNING:` lines about falling back to "the credential Terraform builds with", the profile was not found and your nodes would get your full-access credential rather than the scoped one you just made. > [!TIP] -> If you are a tenancy admin you can skip key files entirely and authenticate the nodes as the instance itself. It needs a dynamic group and a policy created once at the tenancy root — see [instance principal](../oci-architecture/README.md#an-instance-principal-no-key-material-but-it-needs-a-tenancy-admin). The API key path above is the one that works in any tenancy, including a compartment someone allocated to you. +> **A tenancy admin can skip key files altogether** and authenticate the nodes as the instance itself, which leaves no credential on disk to rotate or leak. It needs a dynamic group and a policy created once at the tenancy root, so it is not available in a compartment someone allocated to you. [Instance principal](../oci-architecture/README.md#an-instance-principal-no-key-material-but-it-needs-a-tenancy-admin) covers it, and you then pass `--instance-principal` in step 5 instead of `--credential-hook`. -## 3. Choose one node or three +## 3. Choose deployment method -**This is the only architectural decision, and it is one line in step 4.** +Choose how to deploy Spinifex on OCI, either as a single node, or at minimum a three node cluster for added resiliency and redundancy. -| | Single node | Three nodes | -| --- | --- | --- | -| **`node_count`** | `1` | `3` | -| **Survives losing a node** | No | Yes — NATS keeps quorum, predastore RS(2,1) keeps the data readable, the OVN raft keeps a leader | -| **Public addresses available** | 64 | 192, three VNICs' worth, still bounded by the regional quota | -| **Fault domains** | One | Three, one per node, chosen by Terraform | -| **Use it for** | Evaluation, a lab, an edge site with one box | Anything you would be unhappy to lose | - -Two is not worth taking: it doubles the cost of a single node and gives you a cluster that cannot form a quorum. [Sizing](../oci-architecture/README.md#sizing) covers shapes; bare metal is the recommendation, and `VM.Standard.E6.Flex` is the minimum. - -The pool has no configured size — Spinifex asks OCI for an address when a guest needs one. **Two ceilings bind it and neither is ours**: 64 secondary private IPs per VNIC, which is the 64 above and is not raisable, and the regional reserved-public-IP quota, which is tenancy-wide and shared with everything else you run on OCI. The second is the one you will hit first. +| | Single node | Three nodes | +| ------------------------------ | -------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| **`node_count`** | `1` | `3` | +| **Survives losing a node** | No | Yes. NATS keeps quorum, predastore RS(2,1) keeps the data readable, the OVN raft keeps a leader | +| **Public addresses available** | 64 | 192, three VNICs' worth, still bounded by the regional quota | +| **Fault domains** | One | Three, one per node, chosen by Terraform | +| **Use it for** | Evaluation, a lab, an edge site with one box | Production. Multi-node resiliency | ## 4. Write your Terraform inputs -One file, in the directory you are already in. Only `compartment_ocid` is required — every other variable has a default that builds a working single node: +One file, in the directory you are already in. Only `compartment_ocid` is required, because every other variable has a default that builds a working single node: ```bash cat > terraform.auto.tfvars <<'EOF' @@ -159,7 +338,7 @@ EOF [Every Terraform variable](../oci-architecture/README.md#every-terraform-variable) has the full list. > [!NOTE] -> **Leave `node_client_cidr_allow_list` at its `0.0.0.0/0` default for now.** It is an OCI security-list rule covering the whole public subnet, and your guests' public addresses cross it too — narrowing it to your own address cuts internet access to every guest, not just to the node. [Harden it before production](#harden-it-before-production) is where you lock the node down, in the layer that can tell the two apart. +> **Leave `node_client_cidr_allow_list` at its `0.0.0.0/0` default for now.** It is an OCI security-list rule covering the whole public subnet, and your guests' public addresses cross it too, so narrowing it to your own address cuts internet access to every guest rather than just to the node. [Harden it before production](#harden-it-before-production) locks the node down in the layer that can tell the two apart. ## 5. Deploy @@ -167,23 +346,23 @@ EOF ./validate-topology.sh \ --topology vm-single \ --channel dev \ - --credential-hook ~/.spinifex/oci-credential-hook.sh \ + --credential-hook ./spx-oci-config.sh \ --keep ``` -One command builds the infrastructure, installs Spinifex, forms the cluster, configures OCI public addressing, verifies it, then launches real guests on public addresses and tears those down again. +One command builds the infrastructure, installs Spinifex, forms the cluster, configures OCI public addressing, verifies it, then launches real guests on public addresses and tears those guests down again. -| Argument | What to pass | -| --- | --- | -| `--topology` | `vm-single` for one VM, `vm-multi` for three, `bm` for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission | -| `--channel` | **`dev` until the next release ships**, because OCI support is not in the current stable release yet. It resolves to the newest pre-release at the moment you run it. `latest` is the default and becomes the right answer once a release carries OCI support | -| `--credential-hook` | Your script from step 2. With an instance principal, replace it with `--instance-principal` | -| `--keep` | **This is what makes it a deployment rather than a test.** Without it the driver destroys everything at the end, which is right for CI and not what you want here | +| Argument | What to pass | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--topology` | `vm-single` for one VM, `vm-multi` for three, `bm` for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission | +| `--channel` | **`dev` until the next release ships**, because OCI support is not in the current stable release yet. It resolves to the newest pre-release at the moment you run it. `latest` is the default and becomes the right answer once a release carries OCI support | +| `--credential-hook` | `./spx-oci-config.sh`, which installs the credential from step 2 on every node. With an instance principal, replace it with `--instance-principal` and there is no key to install | +| `--keep` | **This is what makes it a deployment rather than a test.** Without it the driver destroys everything at the end, which is right for CI and wrong here | Add `--skip-workload` to stop once the cluster is verified, without launching the validation guests. Add `--dry-run` to print the plan and change nothing. > [!NOTE] -> **If the install step fails to download, pass a tag instead of the channel.** `dev` is the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited and then 404s. Take the newest tag from the [releases page](https://github.com/mulgadc/spinifex/releases) and swap `--channel dev` for `--version `, which resolves by redirect and cannot trip the same limit. That is also what to use for a repeatable deployment, where you want the build pinned rather than current. +> **If the install step fails to download, pass a tag instead of the channel.** `dev` is the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited and then 404s. Take the newest tag from the [releases page](https://github.com/mulgadc/spinifex/releases) and swap `--channel dev` for `--version `, which resolves by redirect and cannot trip the same limit. A tag is also what to use for a repeatable deployment, where you want the build pinned rather than current. ## 6. Check it worked @@ -202,9 +381,11 @@ The stages print as they finish, and all of these have to appear: [validate-vm-single] --keep: leaving the infrastructure up, no verdict recorded ``` -**`allocator ready` is the line to look for.** Every line above it also appears on a node that cannot allocate a public address, and that failure is otherwise silent until a guest asks for one. +**`allocator ready` is the line to look for.** Every line above it also prints on a node that cannot allocate a public address, and that failure stays silent until a guest asks for one. -Then prove it the way a customer would. On the node, the `[spinifex]` profile that `spx admin init` wrote into `~/.aws/credentials` is the one to use: +`running the credential hook` is `spx-oci-config.sh`, and its own output is captured rather than printed. If the deploy stops there, or if `allocator ready` never arrives, read `credential-hook.log` in the state directory the driver names at startup — its `credential:` line says which credential went onto the nodes. + +Then prove it the way a customer would. On the node, use the `[spinifex]` profile that `spx admin init` wrote into `~/.aws/credentials`: ```bash export AWS_PROFILE=spinifex @@ -224,11 +405,11 @@ ssh -i ubuntu@ hostname ``` > [!IMPORTANT] -> **Probe a public address from outside the VCN.** A node is a bad vantage point for its own network's public addresses, so a timeout from a node is not evidence about the guest. Private addresses are the separate question and do work from inside. +> **Probe a public address from outside the VCN.** A node is a poor vantage point for its own network's public addresses, so a timeout measured from a node says nothing about the guest. Private addresses are a separate question, and those do work from inside. ## Harden it before production -The deployment comes up usable, not locked down. Three layers filter traffic and they are easy to confuse — [Firewalls, in all three layers](../oci-architecture/README.md#firewalls-in-all-three-layers) explains which is which. The short version: +The deployment comes up usable, not locked down. Three layers filter traffic here and the first job is telling them apart, which [Firewalls, in all three layers](../oci-architecture/README.md#firewalls-in-all-three-layers) does in full. The short version: **1. Arm the Spinifex host firewall.** It is installed but not armed on this path. Arming it gives you the proper policy: the public plane open, the cluster plane scoped to peers only, and SSH scoped to a list you own. @@ -237,7 +418,7 @@ The deployment comes up usable, not locked down. Three layers filter traffic and sudo env SETUP_STAGES=firewall /usr/local/share/spinifex/setup.sh --firewall on ``` -**2. Narrow SSH.** It is open to the world until you do — that comes from the Ubuntu image's own rules, not from ours. Edit the one file the installer creates and never overwrites: +**2. Narrow SSH.** It is open to the world until you do, which comes from the Ubuntu image's own rules rather than from ours. Edit the one file the installer creates and never overwrites: ```bash # /etc/spinifex/firewall/custom.nft @@ -251,15 +432,15 @@ sudo /usr/local/lib/spinifex/spinifex-firewall-apply **3. Decide about the console on port 3000.** The S3 gate (8443) and the AWS gateway (9999) are the product's public surface and should stay public. The web console need not be. -**Leave the OCI security list open.** Guest public addresses cross it, so narrowing `node_client_cidr_allow_list` breaks your tenants rather than protecting the node — the host firewall above is the layer that can tell a node's listener from a guest's. +**Leave the OCI security list open.** Guest public addresses cross it, so narrowing `node_client_cidr_allow_list` breaks your tenants instead of protecting the node. The host firewall above is the layer that can tell a node's listener from a guest's. Full detail, including which port serves what and why forming a cluster starts by turning the policy off, is in [Host Firewall](/docs/host-firewall). ## Next steps -**Set up your cluster.** It is running but holds nothing yet — no images, no networks, no instances. [Setting Up Your Cluster](/docs/setting-up-your-cluster) imports an AMI, creates a key pair and a VPC, and launches your first instance. +**Set up your cluster.** It is running but holds nothing yet: no images, no networks, no instances. [Setting Up Your Cluster](/docs/setting-up-your-cluster) imports an AMI, creates a key pair and a VPC, and launches your first instance. -**Run your existing Terraform against it.** Everything above builds infrastructure _on_ OCI with the OCI provider. From here you use the **AWS** provider, unmodified from the registry, pointed at your own cluster — the same `aws_vpc`, `aws_instance`, `aws_db_instance`, `aws_ecs_service` and `aws_eks_cluster` resources a team already has in git. Three things differ, and only three: +**Run your existing Terraform against it.** Everything above builds infrastructure _on_ OCI with the OCI provider. From here you use the **AWS** provider, unmodified from the registry, pointed at your own cluster. Your `aws_vpc`, `aws_instance`, `aws_db_instance`, `aws_ecs_service` and `aws_eks_cluster` resources stay as they are in git. Three things differ, and only three: ```hcl provider "aws" { @@ -280,9 +461,9 @@ provider "aws" { } ``` -> **Use the node's private address, not its public one.** The node certificate carries no SAN for the public address, because that address is never on the wire — OCI NATs it to a private one. A call to `https://:9999` fails TLS verification with _hostname doesn't match_. Run Terraform from a node, or from anything else inside the VCN. +> **Use the node's private address, not its public one.** The node certificate carries no SAN for the public address, because that address is never on the wire: OCI NATs it to a private one. A call to `https://:9999` fails TLS verification with _hostname doesn't match_. Run Terraform from a node, or from anything else inside the VCN. -The [Terraform workbooks](../terraform-workbooks/nginx-alb/README.md) we ship are the worked examples, and `e2e-cloudvendor-nightly` runs them on a single VM, on three VMs and on bare metal, publishing a table per topology on its own run page. Read that for the build you are installing. +The [Terraform workbooks](../terraform-workbooks/nginx-alb/README.md) we ship are the worked examples. `e2e-cloudvendor-nightly` runs them on a single VM, on three VMs and on bare metal, publishing a table per topology on its own run page, so read that for the build you are installing. **Oracle Linux guests.** Four Oracle Linux images are in the catalog, so you can run the distro your Oracle support contract covers: @@ -290,7 +471,7 @@ The [Terraform workbooks](../terraform-workbooks/nginx-alb/README.md) we ship ar sudo spx admin images import --name oracle-10.1-x86_64 --config /etc/spinifex/spinifex.toml ``` -`oracle-10.1-x86_64`, `oracle-10.1-arm64`, `oracle-9.8-x86_64` and `oracle-9.8-arm64`. All four boot **UEFI**, so launch them into a shape that boots UEFI or the failure looks like a hung boot rather than a rejected image. **Log in as `cloud-user`, not `opc`** — these are the generic KVM cloud images from `yum.oracle.com`, and `opc`, `oracle` and `ec2-user` are all refused. The UI's instance detail page shows the right user per AMI. +The four are `oracle-10.1-x86_64`, `oracle-10.1-arm64`, `oracle-9.8-x86_64` and `oracle-9.8-arm64`. All of them boot **UEFI**, so launch them into a shape that boots UEFI or the failure looks like a hung boot rather than a rejected image. **Log in as `cloud-user`, not `opc`.** These are the generic KVM cloud images from `yum.oracle.com`, which refuse `opc`, `oracle` and `ec2-user` alike. The UI's instance detail page shows the right user per AMI. ## Removing the deployment @@ -302,14 +483,14 @@ It destroys whatever that topology's Terraform state holds, so it can only remov ## Troubleshooting -The deploy stops at the first failure and names the log it wrote, all under `.validate-/`. Add `--keep-on-fail` to leave a failed deployment up so you can log in and look — **it keeps billing** until you run `--destroy-only`. +The deploy stops at the first failure and names the log it wrote, all under `.validate-/`. Add `--keep-on-fail` to leave a failed deployment up so you can log in and look. **It keeps billing** until you run `--destroy-only`. -[Troubleshooting](../oci-architecture/README.md#troubleshooting) covers the symptoms worth knowing in advance, and several of them look like a different fault than they are: +[Troubleshooting](../oci-architecture/README.md#troubleshooting) covers the symptoms worth knowing in advance, several of which point at the wrong layer on first reading: -- [If a deploy stage fails](../oci-architecture/README.md#if-a-deploy-stage-fails) — which log holds the answer +- [If a deploy stage fails](../oci-architecture/README.md#if-a-deploy-stage-fails), and which log holds the answer - [A guest's public address is unreachable](../oci-architecture/README.md#a-guests-public-address-is-unreachable) -- [A node that was fine and then was not](../oci-architecture/README.md#a-node-that-was-fine-and-then-was-not) — one command answers all of it -- [Guests cannot reach instance metadata](../oci-architecture/README.md#guests-cannot-reach-instance-metadata) — OCI and AWS guests want the same address -- [Addresses and quotas](../oci-architecture/README.md#addresses-and-quotas) — including why a detached address still bills +- [A node that was fine and then was not](../oci-architecture/README.md#a-node-that-was-fine-and-then-was-not), where one command answers all of it +- [Guests cannot reach instance metadata](../oci-architecture/README.md#guests-cannot-reach-instance-metadata), because OCI and AWS guests want the same address +- [Addresses and quotas](../oci-architecture/README.md#addresses-and-quotas), including why a detached address still bills Installing by hand instead of with Terraform is [Deploying without Terraform](../oci-architecture/README.md#deploying-without-terraform). diff --git a/docs/oci-integration/oci-architecture/README.md b/docs/oci-integration/oci-architecture/README.md index 5b42919ba..05bf7073b 100644 --- a/docs/oci-integration/oci-architecture/README.md +++ b/docs/oci-integration/oci-architecture/README.md @@ -24,7 +24,7 @@ resources: url: "/docs/install-multi-node" --- -# Spinifex on OCI — Architecture and Operations +# Spinifex on OCI: Architecture and Operations > The reference half of the OCI integration. [Spinifex on Oracle Cloud](/docs/oci-integration) is the step-by-step deployment; everything here is what sits underneath it, plus the troubleshooting. @@ -49,17 +49,17 @@ resources: ## Why run Spinifex on OCI -**Lower cost for the same shape of workload.** OCI's compute and block storage are priced well below the large US clouds for equivalent OCPU and IOPS, and Spinifex turns one large instance into a whole EC2/EBS/S3 surface — so a tenant's twenty small instances are twenty QEMU guests on hardware you are billed for once, not twenty billable cloud instances. +**Lower cost for the same shape of workload.** OCI prices compute and block storage well below the large US clouds for equivalent OCPU and IOPS, and Spinifex turns one large instance into a whole EC2/EBS/S3 surface. A tenant's twenty small instances become twenty QEMU guests on hardware you are billed for once, instead of twenty billable cloud instances. -**Egress stops being the line item that decides the architecture.** Oracle publishes a large monthly outbound-transfer allowance per tenancy and a per-GB rate beyond it that is roughly an order of magnitude below the majors — check the current pricing page rather than a number in a document, but the gap is the reason this integration exists. Traffic _between_ guests never leaves the host or, on a cluster, never leaves the VCN: it is Geneve over the private plane, which is not billed as internet egress at all. +**Egress stops being the line item that decides the architecture.** Oracle publishes a large monthly outbound-transfer allowance per tenancy, then charges per GB beyond it at roughly an order of magnitude below the majors. Check the current pricing page rather than a number in a document, but that gap is the reason this integration exists. Traffic _between_ guests never leaves the host, or on a cluster never leaves the VCN: it is Geneve over the private plane, which is not billed as internet egress at all. -**Portability — the workload stops being AWS-shaped and starts being yours.** The API your tenants code against is Spinifex's, so the same Terraform, the same SDK calls and the same AMIs run on OCI today, on bare metal in your own rack tomorrow, and at an edge site after that. Moving off AWS becomes a deployment decision rather than a rewrite, and moving _again_ costs nothing extra. +**The workload stops being AWS-shaped and becomes yours.** Your tenants code against Spinifex's API, so one set of Terraform, SDK calls and AMIs runs on OCI today and on bare metal in your own rack whenever you decide to move it. Leaving AWS becomes a deployment decision rather than a rewrite, and the second move costs nothing extra. -**You keep the parts of OCI that are worth keeping.** Bare-metal shapes, fault domains, block volumes with a chosen performance tier, and real public addresses out of OCI's own pool. Spinifex does not hide the cloud underneath it — it registers each guest's address with OCI so that guest is genuinely reachable on the internet, not behind a shared NAT. +**You keep the parts of OCI worth keeping.** Bare-metal shapes, fault domains, block volumes with a chosen performance tier, and real public addresses out of OCI's own pool. Spinifex registers each guest's address with OCI rather than hiding the cloud underneath it, so the guest is genuinely reachable on the internet instead of sitting behind a shared NAT. ## Sizing -**Bare metal is the recommendation.** `BM.Standard.E2.64` or similar gives you the hardware directly — no nested virtualisation, full NIC performance, and no hypervisor between your guests and the wire. If bare metal is out of scope, a single `VM.Standard.E6.Flex` is the recommendation for testing and development, or three of them for production. +**Bare metal is the recommendation.** `BM.Standard.E2.64` or similar gives you the hardware directly, with no nested virtualisation, full NIC performance, and nothing between your guests and the wire. Where bare metal is out of scope, a single `VM.Standard.E6.Flex` suits testing and development, or three of them production. | | Minimum | Recommended | | --------------- | --------------------- | ----------------------------------------------------------------- | @@ -68,33 +68,33 @@ resources: | **RAM** | 32 GB | 128 GB+ | | **Data volume** | 256 GiB block volume | 1 TB+ NVMe-backed | | **VNICs** | 2 | 2 | -| **Nodes** | 1 | 3 — see [Cluster sizing](/docs/install-multi-node#cluster-sizing) | +| **Nodes** | 1 | 3, see [Cluster sizing](/docs/install-multi-node#cluster-sizing) | ## How it fits together -Everything in [Single-Node Install](/docs/install) and [Multi-Node Install](/docs/install-multi-node) still applies: formation, storage and OVN behave exactly as they do on hardware you own. What changes is the layer underneath — an OCI VCN is not an Ethernet segment, so the external datapath is wired differently, and public addresses come from OCI's API rather than from a range you write in a config file. +Everything in [Single-Node Install](/docs/install) and [Multi-Node Install](/docs/install-multi-node) still applies: formation, storage and OVN behave exactly as they do on hardware you own. The layer underneath is what changes. An OCI VCN is not an Ethernet segment, so the external datapath is wired differently, and public addresses come from OCI's API rather than from a range you write in a config file. -### Single node — the whole AWS surface inside one OCI instance +### Single node: the whole AWS surface inside one OCI instance -One instance runs the control plane and every tenant guest. Customers reach the AWS APIs on the node's own address; their instances reach the internet through addresses registered on the node's second VNIC. +One instance runs the control plane and every tenant guest. Customers reach the AWS APIs on the node's own address, while their instances reach the internet through addresses registered on the node's second VNIC.

- Single node on OCI — the instance inside the VCN public subnet, both VNICs, br-wan, the routed-NAT transit veth, the control plane, tenant VPCs and the iSCSI data volume + Single node on OCI: the instance inside the VCN public subnet, both VNICs, br-wan, the routed-NAT transit veth, the control plane, tenant VPCs and the iSCSI data volume

-Two things in that picture are the whole integration. The guest's private address (`10.0.1.4`) never leaves the host — OVN and the host route both hold it. The address OCI delivers to is a **secondary private IP registered on VNIC 1**, appearing on `br-wan`, and the customer's public address (`150.230.13.131`) is a reserved public IP that OCI 1:1-NATs onto it upstream. `describe-instances` reports the public one; every datapath object holds the private one, which is why the two never appear together on the host. +Two things in that picture are the whole integration. The guest's private address (`10.0.1.4`) never leaves the host, because OVN and the host route both hold it. The address OCI delivers to is a **secondary private IP registered on VNIC 1**, appearing on `br-wan`, and the customer's public address (`150.230.13.131`) is a reserved public IP that OCI 1:1-NATs onto it upstream. `describe-instances` reports the public one while every datapath object holds the private one, which is why the two never appear together on the host. -The third is storage. Every guest disk, every S3 object and the JetStream state all live under `/var/lib/spinifex`, which is an **OCI block volume attached over iSCSI**, not the boot disk. Size it for the guests you intend to run; a boot volume that quietly ends up carrying the stack is the most common way an install on OCI fails later rather than immediately. +The third is storage. Every guest disk, every S3 object and the JetStream state all live under `/var/lib/spinifex`, which is an **OCI block volume attached over iSCSI**, not the boot disk. Size it for the guests you intend to run. A boot volume that quietly ends up carrying the stack is the most common way an install on OCI fails later rather than immediately. -### Three nodes — the same surface, distributed +### Three nodes: the same surface, distributed -Each node registers the addresses for **its own** guests on **its own** second VNIC, and the allocator runs per node, so nothing has to know another node's OCIDs. When a guest moves — a stop/start that lands elsewhere, or a host failure — the node it lands on claims the address's private half from the old VNIC through a single OCI call. The OCID and the public half are untouched, so the customer's address never changes. +Each node registers the addresses for **its own** guests on **its own** second VNIC, and the allocator runs per node, so nothing has to know another node's OCIDs. When a guest moves, whether through a stop/start that lands elsewhere or a host failure, the node it lands on claims the address's private half from the old VNIC with a single OCI call. The OCID and the public half are untouched, so the customer's address never changes.

- Three nodes on OCI — three instances inside the VCN public subnet, one per fault domain, each with its own VNICs, addresses and iSCSI volume, joined east-west by Geneve, NATS, predastore and the OVN raft + Three nodes on OCI: three instances inside the VCN public subnet, one per fault domain, each with its own VNICs, addresses and iSCSI volume, joined east-west by Geneve, NATS, predastore and the OVN raft

-The overlay, the object shards and the OVN databases all cross **VNIC 0**, inside the VCN. Size that plane, not the external one: guest-to-guest traffic between nodes is Geneve over the VCN, and it is the link every distributed layer shares. +The overlay, the object shards and the OVN databases all cross **VNIC 0**, inside the VCN. Size that plane rather than the external one: guest-to-guest traffic between nodes is Geneve over the VCN, and it is the link every distributed layer shares. > [!NOTE] > **The Spinifex region is your own naming and has nothing to do with the OCI region.** `ap-southeast-2` in `spinifex.toml` beside `ap-sydney-1` in OCI is correct and expected. Do not read one as evidence about the other. @@ -103,9 +103,9 @@ The overlay, the object shards and the OVN databases all cross **VNIC 0**, insid The allocator needs OCI credentials at runtime. **A node with none forms, passes every health check, and then refuses every launch that wants a public address** with `InsufficientAddressCapacity`, naming no cause outside its own journal. -### An API key — the documented path +### An API key, the documented path -It needs nothing from a tenancy admin, which also makes it the only option when the compartment belongs to someone else's tenancy. The cost is key material on disk, with a rotation obligation, on every node: +An API key needs nothing from a tenancy admin, which also makes it the only option when the compartment belongs to someone else's tenancy. The cost is key material on disk, with a rotation obligation, on every node: ```bash # On your workstation, not the node. @@ -118,14 +118,14 @@ openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem \ Upload the public key to the user under **Identity → Users → API Keys**, and record the user OCID, the fingerprint and the tenancy OCID. -Each node then needs **two** files, and a node with only the first is the silent failure above — it forms, looks healthy, and cannot allocate: +Each node then needs **two** files, and a node holding only the key is the silent failure described above. | Path | Mode | Contents | | ----------------------------------- | -------------------- | ------------------------------------------------------------------------ | | `/etc/spinifex/oci/oci_api_key.pem` | `0640 root:spinifex` | The private key | | `/etc/spinifex/oci/config` | `0640 root:spinifex` | An ordinary OCI SDK config, profile `[spinifex]`, naming the key by path | -Nothing creates the second one for you, and its profile name must be `spinifex` to match `oci_config_profile` in the pool: +The profile name must be `spinifex` to match `oci_config_profile` in the pool: ```ini [spinifex] @@ -136,11 +136,13 @@ region=ap-sydney-1 key_file=/etc/spinifex/oci/oci_api_key.pem ``` -**Put that in a credential hook and the deploy installs it on every node.** The hook is an executable of yours, run as `hook ...` after formation and before the pool is configured, which is the only window where a node has `/etc/spinifex` but has not started the allocator. It is yours rather than ours deliberately: a credential belongs to whoever owns it, and one rendered into user-data or Terraform state is readable from instance metadata for the life of the instance — on a host that runs other people's guests. +**`spx-oci-config.sh` writes both files on every node, and the deploy calls it for you.** It runs after formation and before the pool is configured, which is the only window where a node has `/etc/spinifex` but has not started the allocator, and it reads the credential from your `[spinifex]` profile, a dedicated key, a set of `OCI_SPX_*` environment variables, or failing all three the credential Terraform itself used — resolved through the same `scripts/oci_env.py` as the apply, so the two cannot disagree. + +**The key reaches the nodes over SSH and goes nowhere else.** It is never rendered into user-data or Terraform state, where it would be readable from instance metadata for the life of the instance, and it is carried in the remote shell's stdin rather than its arguments, where `ps` would expose it to every other tenant on the host. `--credential-hook PATH` still takes any executable called as `hook ...`, so a tenancy that keeps credentials in a vault can substitute its own. ### An instance principal: no key material, but it needs a tenancy admin -The instance certificate is served at `/opc/v2/identity/cert.pem` and the Go SDK authenticates as the instance with no key on disk. It must also be _authorised_, by a dynamic group and a policy, and **both are tenancy-root resources** — `oci_identity_dynamic_group` is created with `compartment_id` set to the tenancy OCID. A compartment-scoped user cannot create them, and the attempt fails with `404-NotAuthorizedOrNotFound` on `CreateDynamicGroup`, which reads like a misconfiguration and is not one. +The instance certificate is served at `/opc/v2/identity/cert.pem` and the Go SDK authenticates as the instance with no key on disk. It must also be _authorised_, by a dynamic group and a policy, and **both are tenancy-root resources**: `oci_identity_dynamic_group` is created with `compartment_id` set to the tenancy OCID. A compartment-scoped user cannot create them, and the attempt fails with `404-NotAuthorizedOrNotFound` on `CreateDynamicGroup`, which reads like a misconfiguration and is not one. ```bash cd scripts/terraform/oci-spx @@ -150,7 +152,7 @@ cd scripts/terraform/oci-spx Then set `instance_principal = "adopt"` in your tfvars and pass `--instance-principal` to the deploy instead of `--credential-hook`. The dynamic group matches on `instance.compartment.id`, so it covers every node you ever build in that compartment and nothing needs updating when nodes are replaced. -**This path is not yet proved end to end.** Every validated run so far used an API key, because the reference tenancy is an allocated child compartment with no tenancy-root rights. On a tenancy where you are admin it should be the better choice; treat the first run as the test. +**This path is not yet proved end to end.** Every validated run so far used an API key, because the reference tenancy is an allocated child compartment with no tenancy-root rights. On a tenancy where you are admin it should be the better choice, so treat the first run as the test. ## The IAM policy @@ -179,11 +181,11 @@ Allow group SpinifexOperators to manage public-ips in compartment [!IMPORTANT] -> **`use vnics` is required, and `read vnics` is not enough.** Registering a secondary private IP is an operation *on a VNIC*, so OCI checks `VNIC_ASSIGN` as well as `PRIVATE_IP_CREATE`, and `VNIC_ASSIGN` lives in the `use` verb. With only `read`, the node forms and looks healthy and every allocation fails `NotAuthorizedOrNotFound` — which reads as a wrong compartment OCID rather than a missing verb. +> **`use vnics` is required, and `read vnics` is not enough.** Registering a secondary private IP is an operation *on a VNIC*, so OCI checks `VNIC_ASSIGN` as well as `PRIVATE_IP_CREATE`, and `VNIC_ASSIGN` lives in the `use` verb. Grant only `read` and every allocation fails with `NotAuthorizedOrNotFound` on a node that looks entirely healthy, which reads as a wrong compartment OCID rather than a missing verb. **Scope it to a compartment, not the tenancy.** `manage public-ips` at tenancy level lets a compromised node consume the whole regional reserved-public-IP quota. -**Do not grant `manage vnics` or `manage instances`.** Spinifex never creates, attaches or deletes a VNIC, and never touches an instance — it only adds addresses to a VNIC that already exists. `use` permits that; `manage` permits VNIC lifecycle, which is strictly more than the integration can use. +**Do not grant `manage vnics` or `manage instances`.** Spinifex never creates, attaches or deletes a VNIC, and never touches an instance. It only adds addresses to a VNIC that already exists, which `use` permits. `manage` also permits VNIC lifecycle, which is strictly more than the integration can use. ## Quotas @@ -192,9 +194,9 @@ Two limits bound how many public addresses a deployment can hand out, and they d | Quota | Default | Scope | Notes | | ---------------------------------- | ------- | ------------------------- | ---------------------------------------------------------------------------------------------- | | **Reserved public IPs** | 50 | Per region, whole tenancy | Shared by every node, so it does **not** grow with node count. This is the wall you hit first. | -| **Secondary private IPs per VNIC** | 64 | Per VNIC, so per node | Not raisable. Three nodes gives you 192 — this is the limit that does scale. | +| **Secondary private IPs per VNIC** | 64 | Per VNIC, so per node | Not raisable. Three nodes gives you 192, so this is the limit that does scale. | -**On a new tenancy, check the compute limits too.** A fresh account commonly carries a service limit of zero for bare-metal shapes and a low one for flex VMs, so `BM.Standard.E2.64` may be unavailable until you request it. That refusal arrives at apply time, not at plan time. +**On a new tenancy, check the compute limits too.** A fresh account commonly carries a service limit of zero for bare-metal shapes and a low one for flex VMs, which can leave `BM.Standard.E2.64` unavailable until you request it. That refusal arrives at apply time, not at plan time. A raise is a support request, not a setting: Console → Governance & Administration → Limits, Quotas and Usage → "Request a service limit increase". Cite the exact limit name from `oci limits definition list`, not a name from documentation. @@ -205,7 +207,7 @@ oci limits resource-availability get --service-name vcn \ --limit-name --compartment-id ``` -**These calls need a grant the node does not have, and must not be given.** Limits and usage are tenancy-level, which cannot be compartment-scoped the way the policy above insists the node's is. Create a **second, read-only principal** for an operator or a reporting job and keep it off every node: +**These calls need a grant the node does not have, and must not be given.** Limits and usage are tenancy-level, which cannot be compartment-scoped the way the policy above insists the node's is. Create a **second, read-only principal** for an operator or a reporting job, and keep it off every node: ```text Allow group SpinifexReporting to read limits in tenancy @@ -216,12 +218,12 @@ The second verb is what Cost Analysis and `oci usage-api usage-summary request-s ## Firewalls, in all three layers -Three independent things filter traffic to an OCI deployment, and they are easy to mistake for one. Nearly every "my port is closed" report on OCI is the wrong layer being adjusted. +Three independent things filter traffic to an OCI deployment, and they are easy to mistake for one. Nearly every "my port is closed" report on OCI turns out to be the wrong layer being adjusted. | Layer | What it guards | Who configures it | | --- | --- | --- | | **OCI security list** | The whole public subnet, including the addresses guests are reached on | `node_client_cidr_allow_list` | -| **The host firewall** | The node's own listeners — console, S3 gate, AWS gateway, SSH, the cluster plane | `node_service_ports`, then the Spinifex nft policy | +| **The host firewall** | The node's own listeners: console, S3 gate, AWS gateway, SSH, the cluster plane | `node_service_ports`, then the Spinifex nft policy | | **OVN security groups** | Each guest, per AWS security-group rules | Your tenants, through the EC2 API | ### The OCI security list is deliberately open, and should stay that way @@ -232,9 +234,9 @@ ingress_security_rules { protocol = "all", source = var.vcn_cidr } # intra- dynamic "ingress_security_rules" { ... source = each allow-list CIDR } # protocol = "all" ``` -A guest's public address is a **secondary private IP on VNIC 1, in the public subnet**, so inbound traffic to a guest crosses this security list. Narrowing `node_client_cidr_allow_list` to your own office `/32` therefore cuts internet ingress to **every guest**, not just to the node — a tenant who opens 443 on their own security group finds it does nothing, with no error anywhere naming the cause. +A guest's public address is a **secondary private IP on VNIC 1, in the public subnet**, so inbound traffic to a guest crosses this security list. Narrowing `node_client_cidr_allow_list` to your own office `/32` therefore cuts internet ingress to **every guest**, not just to the node. A tenant who then opens 443 on their own security group finds it does nothing, with no error anywhere naming the cause. -That is also why the rule is `protocol = "all"` rather than the three service ports: enumerating ports here would mean a Terraform change for every customer port. +That is also why the rule is `protocol = "all"` rather than the three service ports. Enumerating ports here would mean a Terraform change for every customer port. **So leave it at `0.0.0.0/0` for any deployment that serves guests**, and do the restricting in the host firewall, which is the layer that can tell the node's own listeners apart from a guest's. Narrow the allow list only for a private lab with no public guest services. @@ -244,7 +246,7 @@ The separate intra-VCN rule exists so that narrowing the allow list can never cu Terraform's cloud-init runs `open-service-ports.sh`, which opens all protocols from the peer CIDR plus `node_service_ports` (`3000 8443 9999`) to any source, in the Ubuntu image's **iptables**. SSH is already open to the world from the image's own INPUT rule, and nothing in that script narrows it. -The Spinifex nft policy is installed but **not armed** on this path, because the installer arms it only when asked. Arming it is what gives you the proper policy — public plane, peer-scoped cluster plane, and an SSH scope you own: +The Spinifex nft policy is installed but **not armed** on this path, because the installer arms it only when asked. Arming it is what gives you the proper policy: public plane, peer-scoped cluster plane, and an SSH scope you own. ```bash # On each node. Arms the policy the ISO installer arms by default. @@ -254,7 +256,7 @@ sudo env SETUP_STAGES=firewall /usr/local/share/spinifex/setup.sh --firewall on Then narrow SSH in the one file `setup.sh` creates and never overwrites: ```bash -# /etc/spinifex/firewall/custom.nft — yours, left unchanged by upgrades. +# /etc/spinifex/firewall/custom.nft is yours, and upgrades leave it alone. define trusted_ssh_peers = { 203.0.113.10/32 } ``` @@ -263,19 +265,19 @@ sudo nft -c -f /etc/spinifex/firewall/spinifex.nft # check before applying sudo /usr/local/lib/spinifex/spinifex-firewall-apply ``` -What the armed policy allows, and why each port is where it is, is in [Host Firewall](/docs/host-firewall) — including [Narrowing SSH access](/docs/host-firewall#narrowing-ssh-access) and the rule that **forming or expanding a cluster starts by turning the policy off**, because nodes mid-formation are not yet each other's peers. +What the armed policy allows, and why each port is where it is, is in [Host Firewall](/docs/host-firewall), along with [Narrowing SSH access](/docs/host-firewall#narrowing-ssh-access) and the rule that **forming or expanding a cluster starts by turning the policy off**, because nodes mid-formation are not yet each other's peers. Two OCI-specific notes on top of that guide: - **Northstar's DNS on 53 is in the armed policy's public plane, but `node_service_ports` does not include it.** Until the nft policy is armed, a node serving a public zone does not answer on 53 from the internet. -- **Never flush iptables on an OCI node.** The image's `InstanceServices` chain in OUTPUT is what permits iSCSI to `169.254.2.0/24:3260` and the metadata service; clearing it takes the data volume and IMDS with it. +- **Never flush iptables on an OCI node.** The image's `InstanceServices` chain in OUTPUT is what permits iSCSI to `169.254.2.0/24:3260` and the metadata service, so clearing it takes the data volume and IMDS with it. ## What Terraform builds, per node | Resource | Detail | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `oci_core_instance` | One per `node_count`, round-robined across **fault domains** so three nodes land on three sets of racks rather than wherever placement puts them | -| Primary VNIC | Public IP, `hostname_label`, the host plane — SSH, the advertise address, the Geneve endpoint | +| Primary VNIC | Public IP, `hostname_label`, the host plane: SSH, the advertise address, the Geneve endpoint | | `oci_core_vnic_attachment` | A **second VNIC** in the same subnet. Every Spinifex external address becomes a secondary private IP here at runtime | | `oci_core_volume` | A block volume per node, `data_volume_size_in_gbs` at `data_volume_vpus_per_gb` | | `oci_core_volume_attachment` | **`attachment_type = "iscsi"`**, with the Oracle Cloud Agent's Block Volume Management plugin enabled so the node logs the iSCSI session in itself | @@ -283,13 +285,13 @@ Two OCI-specific notes on top of that guide: The resource-by-resource architecture is in [`scripts/terraform/oci-spx`](https://github.com/mulgadc/spinifex/tree/dev/scripts/terraform/oci-spx). -**The data volume is iSCSI, and that is the detail that bites.** OCI presents the volume as an iSCSI target reachable at `169.254.2.2:3260` — attaching it in the API does not put a block device on the host. `is_agent_auto_iscsi_login_enabled` makes the Oracle Cloud Agent do the login, and because it does that _asynchronously_, the device is not there when cloud-init first runs. The mount script waits up to ten minutes for it, formats it **only if `blkid` reports no filesystem** (so a re-run cannot erase a populated volume), and writes an fstab entry by UUID: +**The data volume is iSCSI, and that is the detail that bites.** OCI presents the volume as an iSCSI target reachable at `169.254.2.2:3260`, so attaching it in the API does not put a block device on the host. `is_agent_auto_iscsi_login_enabled` makes the Oracle Cloud Agent do the login, and it does that _asynchronously_, so the device is not there when cloud-init first runs. The mount script waits up to ten minutes for it, formats it **only if `blkid` reports no filesystem** so that a re-run cannot erase a populated volume, and writes an fstab entry by UUID: ```text UUID= /var/lib/spinifex ext4 defaults,_netdev,nofail 0 2 ``` -`_netdev` is what orders the mount after the network, and therefore after `iscsid` has a session. `nofail` is what keeps a missing volume from wedging the boot. Mounting `/var/lib/spinifex` _itself_ rather than somewhere else and symlinking is deliberate: viperblock, predastore and JetStream all land on the volume with no configuration pointing anywhere unusual. +`_netdev` orders the mount after the network, and therefore after `iscsid` has a session. `nofail` keeps a missing volume from wedging the boot. Mounting `/var/lib/spinifex` _itself_ rather than somewhere else and symlinking is deliberate: viperblock, predastore and JetStream all land on the volume with no configuration pointing anywhere unusual. ## Every Terraform variable @@ -297,27 +299,27 @@ All of these have defaults that build a working single node, except the first: | Variable | Default | What it decides | | ----------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `compartment_ocid` | — | **Required.** An OCID, not a name: this deploys into a compartment that already exists, because creating one needs tenancy-root rights the API user must not have | +| `compartment_ocid` | none | **Required.** An OCID, not a name: this deploys into a compartment that already exists, because creating one needs tenancy-root rights the API user must not have | | `deployment_name` | `spinifex` | Prefixes every resource and becomes the VCN `dns_label`, so 1–15 lowercase alphanumerics starting with a letter | | `node_count` | `1` | Nodes, 1–25. `1` for a single node, `3` for a cluster. Each gets its own volume, its own second VNIC and its own fault domain | -| `compute_shape` | `VM.Standard.E6.Flex` | **`BM.*` for bare metal, `VM.*.Flex` for a VM.** A fixed bare-metal shape carries its own size, so `shape_config` is emitted only when the name contains `Flex` — set a BM shape and the two sizing variables below are ignored rather than rejected | +| `compute_shape` | `VM.Standard.E6.Flex` | **`BM.*` for bare metal, `VM.*.Flex` for a VM.** A fixed bare-metal shape carries its own size, so `shape_config` is emitted only when the name contains `Flex`. Set a BM shape and the two sizing variables below are ignored rather than rejected | | `compute_ocpus` | `8` | OCPUs on a flex shape. Validated at ≥ 8 (16 vCPU), the minimum recommended for a node | | `compute_memory_in_gbs` | `32` | GiB on a flex shape. Validated at ≥ 32 | -| `data_volume_size_in_gbs` | `256` | Size of each node's data volume. This is where guest disks, S3 objects and JetStream live — size it for the guests, not for the control plane | +| `data_volume_size_in_gbs` | `256` | Size of each node's data volume. Guest disks, S3 objects and JetStream all live here, so size it for the guests rather than for the control plane | | `data_volume_vpus_per_gb` | `120` | Performance tier, 0–120 VPUs/GB. 120 is Ultra High Performance; 10 is Balanced. Higher is more IOPS and more cost | | `data_mountpoint` | `/var/lib/spinifex` | Where cloud-init mounts it. Change only if you know why | | `vcn_cidr` | `10.200.0.0/22` | VCN address space. A `/22` leaves room for the secondary private IPs Spinifex registers per node | | `public_subnet_cidr` | `10.200.0.0/23` | The subnet every node sits in, **on both VNICs**. Nodes serve the UI, the S3 gate and the AWS gateway directly, so they cannot live in a private subnet | -| `node_client_cidr_allow_list` | `["0.0.0.0/0"]` | OCI security-list ingress, **all protocols**, for the whole public subnet. Guest public addresses cross it too — see [Firewalls](#firewalls-in-all-three-layers) before narrowing it | +| `node_client_cidr_allow_list` | `["0.0.0.0/0"]` | OCI security-list ingress, **all protocols**, for the whole public subnet. Guest public addresses cross it too, so read [Firewalls](#firewalls-in-all-three-layers) before narrowing it | | `node_service_ports` | `[3000, 8443, 9999]` | Console, predastore gate, AWS gateway, opened in the host's iptables by cloud-init. SSH comes from the image's own rule | | `instance_principal` | unset | `"adopt"` to authenticate the allocator as the instance rather than with a key file. Needs the tenancy dynamic group and policy to exist | | `ubuntu_version` | `26.04` | Canonical platform image version. The newest image for the shape and region is resolved at plan time rather than pinned to a regional OCID | | `wan_bridge_name` | `br-wan` | The Linux bridge built over the second VNIC | | `wan_bridge_mtu` | `9000` | The OCI VCN carries 9000 | -**Six more exist and are not yours to set**: `tenancy_ocid`, `user_ocid`, `fingerprint`, `private_key`, `private_key_path` and `region`. `scripts/oci_env.py` fills them as `TF_VAR_*` from the profile in `~/.oci/config` on every `terraform` call the driver makes, so **the region you deploy into is the profile's region** and the credential never reaches a `.tfvars` file. Setting them by hand works and is how a run with no config file at all would have to work, but then two sources disagree about which tenancy this is. +**Six more exist and are not yours to set**: `tenancy_ocid`, `user_ocid`, `fingerprint`, `private_key`, `private_key_path` and `region`. `scripts/oci_env.py` fills them as `TF_VAR_*` from the profile in `~/.oci/config` on every `terraform` call the driver makes, so **the region you deploy into is the profile's region** and the credential never reaches a `.tfvars` file. Setting them by hand works, and is how a run with no config file at all would have to work, but then two sources disagree about which tenancy this is. -The profile is resolved as `--oci-profile`, then `$OCI_CLI_PROFILE`, then the reference tenancy, then `DEFAULT`. A name passed explicitly and absent from the config is an error that lists the profiles present; only an unasked-for name falls through. CI takes a different route entirely — `OCI_TENANCY_OCID`, `OCI_USER_OCID`, `OCI_FINGERPRINT`, `OCI_PRIVATE_KEY` and `OCI_REGION` in the environment, so a runner writes no key to disk — and a partial set there is an error rather than a fallback, because half a credential means a secret failed to reach the job. +The profile is resolved as `--oci-profile`, then `$OCI_CLI_PROFILE`, then the reference tenancy, then `DEFAULT`. A name passed explicitly and absent from the config is an error that lists the profiles present; only an unasked-for name falls through. CI takes a different route entirely, reading `OCI_TENANCY_OCID`, `OCI_USER_OCID`, `OCI_FINGERPRINT`, `OCI_PRIVATE_KEY` and `OCI_REGION` from the environment so a runner writes no key to disk. A partial set there is an error rather than a fallback, because half a credential means a secret failed to reach the job. ## The `oci` CLI @@ -331,23 +333,23 @@ sudo ln -sf ~/.oci-cli-venv/bin/oci /usr/local/bin/oci oci --version ``` -**What the CLI is and is not for.** Spinifex's daemon uses the OCI **Go SDK** for every allocation — the CLI is not on that path. You need it for bootstrap and for operational inspection. Do not build automation that shells out to it during an allocation; allocations run inside a request budget that a forked Python process will not respect. +**What the CLI is and is not for.** Spinifex's daemon uses the OCI **Go SDK** for every allocation, so the CLI is not on that path at all. You need it for bootstrap and for operational inspection. Do not build automation that shells out to it during an allocation: allocations run inside a request budget that a forked Python process will not respect. --- ## Deploying without Terraform -**Terraform automates every step of a normal install, and that is the point of it.** The `oci-spx` configuration builds the VCN, the subnets, the gateways, the block volumes and both VNICs per node, then renders the cloud-init that installs Spinifex and sets up OVN — so the sequence below happens without anyone typing it, in the same order every time, on one node or three. +**Terraform automates every step of a normal install, and that is the point of it.** The `oci-spx` configuration builds the VCN, the subnets, the gateways, the block volumes and both VNICs per node, then renders the cloud-init that installs Spinifex and sets up OVN. The sequence below happens without anyone typing it, in the same order every time, on one node or three. -If you need to install by hand — an existing OCI tenancy you cannot run Terraform against, an unsupported shape, or a stage you are debugging — the install itself is not OCI-specific. Follow the standard guides: +If you do need to install by hand, whether for an existing OCI tenancy you cannot run Terraform against, an unsupported shape, or a stage you are debugging, the install itself is not OCI-specific. Follow the standard guides: - [Installing Spinifex](../../install/install/README.md) for a single node. - [Multi-Node Installation](../../install/install-multi-node/README.md) for a cluster, which covers `spx admin init` on the leader and `spx admin join` on the rest. Four things about OCI are not in those guides, and each has its own section above: -- **The external pool** is `source = "oci"`, which needs the API key or instance principal from [Credentials](#credentials-an-api-key-or-an-instance-principal) and the [IAM policy](#the-iam-policy). Without it a node forms, passes every health check, and then refuses every launch that wants a public address. -- **`br-wan` stays a Linux bridge** owned by netplan, linked to OVS `br-ext` by a veth pair. Pass `setup-ovn.sh --wan-bridge=br-wan` and it detects this itself; the NIC never becomes an OVS port, so the VNIC keeps its MAC identity. See [How it fits together](#how-it-fits-together). +- **The external pool** is `source = "oci"`, which needs the API key or instance principal from [Credentials](#credentials-an-api-key-or-an-instance-principal) and the [IAM policy](#the-iam-policy). Omit it and you get the silent failure that section opens with: a healthy node that cannot hand out an address. +- **`br-wan` stays a Linux bridge** owned by netplan, linked to OVS `br-ext` by a veth pair. Pass `setup-ovn.sh --wan-bridge=br-wan` and it detects this itself. The NIC never becomes an OVS port, so the VNIC keeps its MAC identity. See [How it fits together](#how-it-fits-together). - **The block volume must be mounted with `_netdev`**, because it arrives over iSCSI and the mount is otherwise attempted before a session exists. - **IMDS needs the 169.254 remap** described under [Guests cannot reach instance metadata](#guests-cannot-reach-instance-metadata), since the guest's metadata address collides with OCI's own. @@ -368,38 +370,38 @@ The driver stops at the first failure, names the log, and leaves the rest alone. | Verifying public addressing | `allocator-.log` | | The Terraform workbooks | `workbooks.log`, `workbooks.tsv` | -The allocator log is the one worth knowing about, because public addressing is the part with the most moving pieces. If the allocator never comes up, the driver captures the daemon journal, the pool as configured, the bridge and OCI MAC addresses, and whether each credential file exists and is readable — never the credentials themselves — before anything is torn down. +The allocator log is the one worth knowing about, because public addressing is the part with the most moving pieces. If the allocator never comes up, the driver captures the daemon journal, the pool as configured, the bridge and OCI MAC addresses, and whether each credential file exists and is readable, never the credentials themselves, before anything is torn down. Add `--keep-on-fail` to leave a failed deployment running so you can log in and look. **It keeps billing** until you run `--destroy-only`. ### Addresses and quotas -**Spinifex enforces no limit of its own.** It attempts the allocation and reports what OCI says, so the ceiling you hit is always your tenancy's, never a number baked into the software. A quota is raised with Oracle, not reconfigured here, and there is nothing to restart afterwards. +**Spinifex enforces no limit of its own.** It attempts the allocation and reports what OCI says, so the ceiling you hit is always your tenancy's and never a number baked into the software. A quota is raised with Oracle, not reconfigured here, and there is nothing to restart afterwards. -**`InsufficientAddressCapacity` on `allocate-address` is what exhaustion looks like.** It is the AWS error code for "out of addresses" and Spinifex returns it for either quota, mapped from OCI's own refusal. On a cluster the regional one is the likelier of the two. +**`InsufficientAddressCapacity` on `allocate-address` is what exhaustion looks like.** It is the AWS error code for "out of addresses", and Spinifex returns it for either quota, mapped from OCI's own refusal. On a cluster the regional one is the likelier of the two. -**A detached reserved public IP still bills.** `disassociate-address` returns the address to `AVAILABLE` without deleting it, exactly as an unassociated AWS Elastic IP behaves, and it keeps consuming your regional quota as well as your bill. `release-address` is what frees both. +**A detached reserved public IP still bills.** `disassociate-address` returns the address to `AVAILABLE` without deleting it, exactly as an unassociated AWS Elastic IP behaves, and it keeps consuming your regional quota as well as your bill. `release-address` frees both. -**A stopped instance costs nothing for its auto-assigned address.** Stopping an instance returns its auto-assigned address to the pool and deletes the reserved public IP object behind it, as AWS does; the start that follows creates a new one. An **Elastic IP** is yours and survives the stop untouched — that is the whole distinction between the two, and on OCI it is also the difference between a stopped fleet that bills for addresses and one that does not. +**A stopped instance costs nothing for its auto-assigned address.** Stopping an instance returns its auto-assigned address to the pool and deletes the reserved public IP object behind it, as AWS does, and the start that follows creates a new one. An **Elastic IP** is yours and survives the stop untouched. That is the whole distinction between the two, and on OCI it is also the difference between a stopped fleet that bills for addresses and one that does not. ### A guest's public address is unreachable -**Check where you are probing from first.** A node is a bad vantage point for a public address, and two independent things make it so: the security list may not admit the node's own public address as a source, and a VCN may or may not reflect its public addresses back into itself. Either way a healthy cluster fails the probe. Test from outside the VCN, which is where a customer is. +**Check where you are probing from first.** A node is a poor vantage point for a public address, for two independent reasons: the security list may not admit the node's own public address as a source, and a VCN may or may not reflect its public addresses back into itself. Either one fails the probe on a perfectly healthy cluster. Test from outside the VCN, which is where a customer is. Then, in order: -1. `sudo ovn-nbctl lr-nat-list ` — is there a `dnat_and_snat` row, and does it hold the _private_ address? -2. `ip route get ` — does it route via the transit veth? -3. `sudo iptables -S FORWARD | grep spinifex-eip-ingress` — both directions present? -4. `oci network private-ip list --vnic-id ` — is the private address actually registered with OCI, **and on this node's VNIC**? If not, nothing else matters. +1. `sudo ovn-nbctl lr-nat-list ` for a `dnat_and_snat` row, holding the _private_ address. +2. `ip route get ` to confirm it routes via the transit veth. +3. `sudo iptables -S FORWARD | grep spinifex-eip-ingress` for both directions. +4. `oci network private-ip list --vnic-id ` to confirm the private address is registered with OCI, **and on this node's VNIC**. If it is not, nothing else matters. -**Egress works from one address and not another on the same NIC.** OCI enforces the source address per VNIC, so this means the address is not a registered private-IP object on it. It is never a routing problem on the host. +**Egress works from one address and not another on the same NIC.** OCI enforces the source address per VNIC, so that means the address is not a registered private-IP object on it. It is never a routing problem on the host. -**An address went dark after a guest moved node.** The affinity pass should have claimed it within 15 seconds. Check the new node's daemon journal for `ocinet affinity`, then confirm with step 4 above which VNIC OCI thinks holds the private half. +**An address went dark after a guest moved node.** The affinity pass should have claimed it within 15 seconds. Check the new node's daemon journal for `ocinet affinity`, then use step 4 above to confirm which VNIC OCI thinks holds the private half. ### A node that was fine and then was not -Terraform's cloud-init does the host jobs on each node — the iSCSI volume, `br-wan` and its source routing, the firewall. When one of them did not take, the symptom arrives much later and rarely looks like a networking or storage problem. This one command answers all of it: +Terraform's cloud-init does the host jobs on each node: the iSCSI volume, `br-wan` and its source routing, the firewall. When one of them did not take, the symptom arrives much later and rarely looks like a networking or storage problem. This one command answers all of it: ```bash ssh -i ~/.ssh/oci-spx ubuntu@ ' @@ -417,7 +419,7 @@ Three answers are worth knowing in advance, because each looks wrong and is not: - **`br-wan` holds a `/32` and the default route is on the other interface.** That is how two VNICs share one subnet without the second stealing the first one's traffic, and it is what satisfies OCI's per-VNIC source check. `br-wan`'s address appears in `ip rule` instead of in the default route. - **`br-wan`'s address is not one you chose.** It is the second VNIC's own private IP, assigned by OCI, and not predictable before the apply. -- **`enp1s0` carrying an address is the one real fault here.** The kernel derives an on-link route from it that beats the primary VNIC's, so every packet to another node leaves the wrong VNIC and OCI drops it. The node still forms a cluster and still reports `Ready` — formation uses outbound connections — and it surfaces later as `predastore … the stripe is short one holder` on the first AMI import. `sudo /usr/local/sbin/spinifex-setup-wan-bridge` repairs it in place. +- **`enp1s0` carrying an address is the one real fault here.** The kernel derives an on-link route from it that beats the primary VNIC's, so every packet to another node leaves the wrong VNIC and OCI drops it. The node still forms a cluster and still reports `Ready`, because formation uses outbound connections, and it surfaces later as `predastore … the stripe is short one holder` on the first AMI import. `sudo /usr/local/sbin/spinifex-setup-wan-bridge` repairs it in place. ### The allocator @@ -433,21 +435,21 @@ sudo journalctl -u spinifex-vpcd --since -5m | grep -i ocinet "msg":"OCI allocator ready","pool":"oci-public","collected":0,"stale_bindings":0,"skipped":1 ``` -**`resolved the external VNIC` is the line that matters** — it proves the credentials work, the compartment is right, and `br-wan`'s MAC matched a real VNIC. A node missing it will accept `allocate-address` and fail it. +**`resolved the external VNIC` is the line that matters.** It proves the credentials work, the compartment is right, and `br-wan`'s MAC matched a real VNIC. A node missing it will accept `allocate-address` and then fail it. -On a cold cluster start you may instead see `OCI allocator reconcile failed … nats: no responders available for request` on some nodes. That is vpcd racing JetStream's KV at boot; the startup reconcile is skipped and not retried. It is harmless on a new cluster where nothing has been allocated, and it is tracked — restart `spinifex-vpcd` on that node to run it. +On a cold cluster start you may instead see `OCI allocator reconcile failed … nats: no responders available for request` on some nodes. That is vpcd racing JetStream's KV at boot, where the startup reconcile is skipped and not retried. It is harmless on a new cluster where nothing has been allocated, and it is tracked; restart `spinifex-vpcd` on that node to run it. ### Host -**`RunInstances` returns `ServerInternal`.** Check the daemon journal for the real error — `journalctl -u spinifex-daemon --since -10m`. An OCI allocation failure surfaces this way. +**`RunInstances` returns `ServerInternal`.** Check the daemon journal for the real error with `journalctl -u spinifex-daemon --since -10m`. An OCI allocation failure surfaces this way. -**The stack ends up on the boot disk and the node fills.** The data volume was mounted without `_netdev` and lost the race with `iscsid`. `findmnt /var/lib/spinifex` is the check; `ls` cannot tell the difference. +**The stack ends up on the boot disk and the node fills.** The data volume was mounted without `_netdev` and lost the race with `iscsid`. `findmnt /var/lib/spinifex` is the check, and `ls` cannot tell the difference. ### Guests cannot reach instance metadata -**Guests boot but cloud-init hangs on metadata**, or the host stops answering its own. OCI uses `169.254.169.254` for its instance metadata, and so does every AWS-compatible guest — the host and the guests want the same address. +**Guests boot but cloud-init hangs on metadata**, or the host stops answering its own. OCI uses `169.254.169.254` for its instance metadata, and so does every AWS-compatible guest, so the host and the guests want the same address. -Spinifex resolves this by moving the _host_ side of the per-guest metadata link to a different link-local pair, so guests keep asking `169.254.169.254` exactly as they do on AWS. The deploy sets `imds_host_meta_ip` and `imds_host_dns_ip` for you. **Set both or neither** — a half-configured pair is ignored and the node behaves as if the remap were off, which is what this symptom looks like. +Spinifex resolves this by moving the _host_ side of the per-guest metadata link to a different link-local pair, leaving guests to ask `169.254.169.254` exactly as they do on AWS. The deploy sets `imds_host_meta_ip` and `imds_host_dns_ip` for you. **Set both or neither**, because a half-configured pair is ignored and the node behaves as if the remap were off, which is what this symptom looks like. Check it after your first guest launches: @@ -458,7 +460,7 @@ curl -sH 'Authorization: Bearer Oracle' -o /dev/null -w '%{http_code}\n' \ ip -br addr show | grep ime- # holds 169.254.42.x, not .169.x ``` -`ip route get 169.254.169.254` returning `dev lo` is the signature of a missing remap. The same signature can appear after a guest terminates, from a leftover `ime-` endpoint still holding the address; with the remap configured it is harmless, and removing the stale OVS port clears it. +`ip route get 169.254.169.254` returning `dev lo` is the signature of a missing remap. The same signature can appear after a guest terminates, from a leftover `ime-` endpoint still holding the address; with the remap configured that is harmless, and removing the stale OVS port clears it. ### An OVS problem, when `ovs-vsctl` says nothing is wrong diff --git a/scripts/terraform/oci-spx/README.md b/scripts/terraform/oci-spx/README.md index 1bd925c9e..a56d4f9a8 100644 --- a/scripts/terraform/oci-spx/README.md +++ b/scripts/terraform/oci-spx/README.md @@ -232,28 +232,24 @@ Create them once per tenancy, then every deployment and rebuild afterwards uses The policy grants three verbs in one compartment — `use vnics`, `manage private-ips`, `manage public-ips` — which is exactly what allocating an external address does and nothing more. -**An API key is the fallback**, and the default because it needs nothing from a tenancy admin. Grant that user only the operations Spinifex performs on addresses and nothing else; it is not a tenancy admin, and broader rights widen the blast radius of a node compromise for no benefit. The key then has to be installed on **every** node, by hand, under `/etc/spinifex/oci/` — not a home directory, because the daemon's unit sets `ProtectHome=yes`: +**An API key is the fallback**, and the default because it needs nothing from a tenancy admin. Grant that user only the operations Spinifex performs on addresses and nothing else; it is not a tenancy admin, and broader rights widen the blast radius of a node compromise for no benefit. -```bash -sudo install -d -o root -g spinifex -m 0750 /etc/spinifex/oci -sudo install -o root -g spinifex -m 0640 ~/.oci/oci_api_key.pem /etc/spinifex/oci/oci_api_key.pem -``` +`spx-oci-config.sh` installs it on **every** node, as `/etc/spinifex/oci/{oci_api_key.pem,config}` — not a home directory, because the daemon's unit sets `ProtectHome=yes`. Both files are needed: `oci_config_file` names an ordinary OCI SDK config, so a node holding only the PEM forms and then fails every allocation. The profile it writes is `spinifex`, matching `oci_config_profile`: -**The key is half of it.** `oci_config_file` names an ordinary OCI SDK config, and nothing creates that file, so a node with only the PEM forms and then fails every allocation. The profile must be `spinifex`, to match `oci_config_profile`, and the four identifiers are the ones already in the tfvars: - -```bash -sudo tee /etc/spinifex/oci/config >/dev/null <<'CONF' +```ini [spinifex] user=ocid1.user.oc1.. fingerprint= tenancy=ocid1.tenancy.oc1.. region=ap-sydney-1 key_file=/etc/spinifex/oci/oci_api_key.pem -CONF -sudo chown root:spinifex /etc/spinifex/oci/config && sudo chmod 0640 /etc/spinifex/oci/config ``` -For repeated runs, `validate-topology.sh --credential-hook` does both steps on every node. The hook is an operator-owned executable rather than anything in this repository, because a credential rendered into user-data or Terraform state is readable from instance metadata for the life of the instance, and these hosts run other people's guests. Instance principal needs no hook at all, which is the reason to prefer it wherever a dynamic group can be created. +The script takes `validate-topology.sh`'s hook contract, `hook ...`, so it is both the default thing to pass to `--credential-hook` and runnable on its own against an existing cluster. `--dry-run` resolves and validates the credential without touching a host. + +It resolves the credential in four steps and takes the first complete one: the `OCI_SPX_*` environment variables, the `[spinifex]` profile in `~/.oci/config`, `~/.oci/oci_api_key_spx.pem` with the Terraform profile's identifiers, then the Terraform credential itself with a warning that it is wider than needed. Steps 2 to 4 go through `scripts/oci_env.py`, the same resolver the apply uses, so the node credential and the one that built the infrastructure cannot come from different profiles. The fingerprint is checked against the key before anything is written, because a mismatched pair is otherwise an OCI 401 at the first allocation. + +Credential material reaches the nodes over SSH, in the remote shell's stdin rather than its arguments, and goes into neither user-data nor Terraform state. `--credential-hook` still accepts any executable with that argument shape, so a tenancy holding credentials in a vault can substitute its own. Instance principal needs no hook at all, which is the reason to prefer it wherever a dynamic group can be created. Either way, Terraform stages the matching pool block at `/etc/spinifex/oci/external-pool.toml` on each node, with `oci_auth` set to match. Append it to `/etc/spinifex/spinifex.toml` after `spx admin init` and restart `spinifex.target`. Under instance principal that file holds no secret at all, which is the point of it. diff --git a/scripts/terraform/oci-spx/spx-oci-config.sh b/scripts/terraform/oci-spx/spx-oci-config.sh new file mode 100755 index 000000000..d8b5e49c0 --- /dev/null +++ b/scripts/terraform/oci-spx/spx-oci-config.sh @@ -0,0 +1,264 @@ +#!/usr/bin/env bash +# Installs the OCI credential the external-address allocator authenticates with +# on every node, as /etc/spinifex/oci/{config,oci_api_key.pem}. +# +# Usage: spx-oci-config.sh [--dry-run] ... +# +# The argument shape is validate-topology.sh's credential-hook contract, so this +# drops straight into --credential-hook or $OCI_CREDENTIAL_HOOK. +set -euo pipefail + +HERE=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +SELF="$HERE/$(basename -- "$0")" + +# What the allocator reads, from the pool fragment Terraform stages on each node. +# The config path and the profile name are the contract; the key filename is only +# ever named by key_file inside that config. +REMOTE_DIR=/etc/spinifex/oci +REMOTE_CONFIG="$REMOTE_DIR/config" +REMOTE_PEM="$REMOTE_DIR/oci_api_key.pem" +REMOTE_PROFILE=spinifex + +SPX_KEY="${OCI_SPX_KEY:-$HOME/.oci/oci_api_key_spx.pem}" +SPX_PROFILE="${OCI_SPX_PROFILE:-spinifex}" +SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null + -o LogLevel=ERROR -o BatchMode=yes -o ConnectTimeout=15) +DRY_RUN=0 + +die() { echo "$(basename -- "$0"): $*" >&2; exit 1; } +log() { echo "[spx-oci-config] $*"; } + +usage() { + sed -n '2,8p' "$SELF" | sed 's/^# \{0,1\}//' + cat <<'EOF' + +Options: + --dry-run Resolve and validate the credential, print what would be written, + and touch no host. + -h, --help This text. + +The credential is resolved in this order, and the first one that is complete wins: + + 1. $OCI_SPX_PRIVATE_KEY with $OCI_SPX_USER_OCID, $OCI_SPX_FINGERPRINT, + $OCI_SPX_TENANCY_OCID and $OCI_SPX_REGION. A dedicated narrow credential, + which is the shape a CI secret store holds. A partial set is an error. + 2. The [spinifex] profile in ~/.oci/config, or $OCI_SPX_PROFILE. + 3. ~/.oci/oci_api_key_spx.pem, or $OCI_SPX_KEY, with the user, tenancy and + region of the profile Terraform itself uses. The fingerprint is derived + from the key rather than read from the config. + 4. The credential Terraform builds with. This works and is wider than the + allocator needs, so it warns. + +Rules 2 to 4 resolve the Terraform profile through scripts/oci_env.py, so the +precedence is the same one the apply used and the two cannot disagree. +EOF +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --dry-run) DRY_RUN=1; shift ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + -*) die "unknown option: $1" ;; + *) break ;; + esac +done + +# oci_env.py owns credential resolution for this directory: the environment route +# a runner uses, ~/.oci/config otherwise, a partial set treated as an error, and +# the shape checks. Re-exec under it rather than reimplement any of that here. +if [ -z "${SPX_OCI_ENV_READY:-}" ]; then + [ -x "$HERE/scripts/oci_env.py" ] || [ -r "$HERE/scripts/oci_env.py" ] \ + || die "scripts/oci_env.py was not found next to $SELF" + if [ "$DRY_RUN" = 1 ]; then + exec env SPX_OCI_ENV_READY=1 python3 "$HERE/scripts/oci_env.py" -- "$SELF" --dry-run "$@" + fi + exec env SPX_OCI_ENV_READY=1 python3 "$HERE/scripts/oci_env.py" -- "$SELF" "$@" +fi + +# The same argument shape in both modes, so a --dry-run rehearsal and the real run +# differ by one flag. Only the checks that need a host are skipped. +ssh_key="${1:?usage: $(basename -- "$0") [--dry-run] ...}" +shift +if [ "$DRY_RUN" != 1 ]; then + [ "$#" -gt 0 ] || die "no hosts given" + [ -r "$ssh_key" ] || die "SSH private key is not readable: $ssh_key" +fi + +# A key's fingerprint is a property of the key. Deriving it beats trusting a +# config field, because a stale fingerprint beside a good key is a 401 that names +# neither. +fingerprint_of() { + openssl rsa -pubout -outform DER -in "$1" 2>/dev/null \ + | openssl md5 -c 2>/dev/null | awk '{print $NF}' +} + +require_complete() { + local source="$1" missing="" name + shift + for name in "$@"; do + [ -n "${!name:-}" ] || missing="$missing $name" + done + [ -z "$missing" ] || die "$source is incomplete, missing:$missing" +} + +key_pem="" # the PEM itself, never a path, so nothing is read twice +oci_user="" +oci_fingerprint="" +oci_tenancy="" +oci_region="" +source_name="" + +if [ -n "${OCI_SPX_PRIVATE_KEY:-}${OCI_SPX_USER_OCID:-}${OCI_SPX_FINGERPRINT:-}${OCI_SPX_TENANCY_OCID:-}${OCI_SPX_REGION:-}" ]; then + require_complete "the OCI_SPX_* credential in the environment" \ + OCI_SPX_PRIVATE_KEY OCI_SPX_USER_OCID OCI_SPX_FINGERPRINT \ + OCI_SPX_TENANCY_OCID OCI_SPX_REGION + # Base64 accepted by shape, because a PEM's newlines do not survive every + # secret store, and detected rather than selected by a second variable. + case "$OCI_SPX_PRIVATE_KEY" in + *-----BEGIN*) key_pem="$OCI_SPX_PRIVATE_KEY" ;; + *) key_pem=$(printf '%s' "$OCI_SPX_PRIVATE_KEY" | tr -d ' \n' | base64 -d 2>/dev/null) || true ;; + esac + case "$key_pem" in + *-----BEGIN*) ;; + *) die "OCI_SPX_PRIVATE_KEY is neither a PEM nor a base64-encoded PEM" ;; + esac + oci_user="$OCI_SPX_USER_OCID" + oci_fingerprint="$OCI_SPX_FINGERPRINT" + oci_tenancy="$OCI_SPX_TENANCY_OCID" + oci_region="$OCI_SPX_REGION" + source_name="the OCI_SPX_* environment credential" + +elif [ -r "$HOME/.oci/config" ] && python3 - "$SPX_PROFILE" <<'PY' +import configparser, pathlib, sys +cfg = configparser.RawConfigParser() +cfg.read(pathlib.Path.home() / ".oci" / "config") +sys.exit(0 if cfg.has_section(sys.argv[1]) else 1) +PY +then + # One pipe-separated line read into named variables, rather than eval of + # generated shell: none of these fields can contain a pipe. + profile_fields=$(python3 - "$SPX_PROFILE" <<'PY' +import configparser, pathlib, sys +cfg = configparser.RawConfigParser() +cfg.read(pathlib.Path.home() / ".oci" / "config") +section = dict(cfg[sys.argv[1]]) +fields = ("user", "fingerprint", "tenancy", "region", "key_file") +missing = [f for f in fields if not section.get(f)] +if missing: + sys.exit(f"profile {sys.argv[1]} has no {', '.join(missing)}") +print("|".join(section[f] for f in fields)) +PY +) || die "could not read the [$SPX_PROFILE] profile from ~/.oci/config" + IFS='|' read -r profile_user profile_fingerprint profile_tenancy \ + profile_region profile_key_file < "$pem_tmp" ) +derived=$(fingerprint_of "$pem_tmp") +[ -n "$derived" ] || die "$source_name does not hold a usable RSA private key" +[ "$derived" = "$oci_fingerprint" ] \ + || die "the key from $source_name has fingerprint $derived, but the credential names $oci_fingerprint" + +log "credential: $source_name" +log "user: $oci_user" +log "fingerprint: $oci_fingerprint" +log "region: $oci_region" + +if [ "$DRY_RUN" = 1 ]; then + log "would write $REMOTE_CONFIG and $REMOTE_PEM as 0640 root:spinifex on:${*:-" (no hosts given)"}" + exit 0 +fi + +# base64 so the PEM survives two shells, and carried in the stdin stream rather +# than in the command: an argument is readable from ps on the node for the length +# of the call, on a host that runs other people's guests. +key_b64=$(printf '%s\n' "$key_pem" | base64 | tr -d '\n') + +install_on() { + # Unquoted heredoc on purpose: the key and the four credential fields are + # expanded here, so nothing in it is expanded on the node. + # shellcheck disable=SC2087 + ssh -i "$ssh_key" "${SSH_OPTS[@]}" "ubuntu@$1" bash -s </dev/null \ + || { echo "no spinifex group yet, so this ran before the install" >&2; exit 3; } +sudo install -d -m 0750 -o root -g spinifex $REMOTE_DIR +# install /dev/null first, then write: the file never exists at a wider mode, and +# sudo's umask is not the caller's. +sudo install -m 0640 -o root -g spinifex /dev/null $REMOTE_PEM +base64 -d <<'PEM_B64' | sudo tee $REMOTE_PEM >/dev/null +$key_b64 +PEM_B64 +sudo install -m 0640 -o root -g spinifex /dev/null $REMOTE_CONFIG +sudo tee $REMOTE_CONFIG >/dev/null <<'CONFIG' +[$REMOTE_PROFILE] +user=$oci_user +fingerprint=$oci_fingerprint +tenancy=$oci_tenancy +region=$oci_region +key_file=$REMOTE_PEM +CONFIG +REMOTE +} + +# Every host is attempted and the failures are named. Stopping at the first leaves +# a half-credentialled cluster and a log that does not say which half. +failed="" +for host in "$@"; do + log "installing the OCI credential on $host" + install_on "$host" || failed="$failed $host" +done + +[ -z "$failed" ] || die "failed on:$failed" +log "credential installed on $# host(s)"