From cf17866feda52d29b97f5227373f0a2cf00e8ee1 Mon Sep 17 00:00:00 2001 From: Ben Duncan <5178243+benduncan@users.noreply.github.com> Date: Sun, 4 Oct 2026 15:18:02 -0500 Subject: [PATCH] docs(oci): publish both OCI guides on the docs site (#1165) * docs(oci): publish both OCI guides on the docs site The site extracts only four H2 sections and drops everything else, so both guides would have published nearly empty: the deployment guide matched none of the four, and the reference doc only its troubleshooting. Insert the section markers the other published docs already use, keeping every heading at the level it had so no anchor moves. The architecture reference becomes oci-architecture/README.md, because a page is always /README.md and a flat URL takes the last path segment -- plain "architecture" would have published at /docs/architecture. Two headings lose an em dash. GitHub renders it as a double hyphen in the fragment and the site's slugify drops it entirely, so the two disagreed and the cross-links resolved on neither. * docs(oci): the OCI guides sit in their own Cloud Install section Their category frontmatter named Install, which is the section they are moving out of. --- docs/oci-integration/README.md | 63 ++++++++++--------- .../README.md} | 63 ++++++++++--------- 2 files changed, 68 insertions(+), 58 deletions(-) rename docs/oci-integration/{architecture.md => oci-architecture/README.md} (95%) diff --git a/docs/oci-integration/README.md b/docs/oci-integration/README.md index 7bb0f0dad..a0197cfd9 100644 --- a/docs/oci-integration/README.md +++ b/docs/oci-integration/README.md @@ -1,8 +1,8 @@ --- -title: "Spinifex on Oracle Cloud" +title: "Oracle Cloud Infrastructure (OCI)" seoTitle: "Run Spinifex on Oracle Cloud Infrastructure — Spinifex Docs" -description: "Deploy Spinifex on OCI end to end with Terraform, single node or three, and give guests real public addresses through OCI's own API." -category: "Install" +description: "Deploy Spinifex on Oracle Cloud Infrastructure end to end with Terraform, on one node or three, and give guests real public addresses through OCI's own API." +category: "Cloud Install" tags: - install - oci @@ -11,7 +11,7 @@ tags: - cluster resources: - title: "OCI Architecture and Operations" - url: "/docs/oci-integration/architecture" + url: "/docs/oci-architecture" - title: "Multi-Node Install" url: "/docs/install-multi-node" - title: "Host Firewall" @@ -24,15 +24,10 @@ resources: > Run the AWS surface — EC2, EBS, S3, VPC — inside your own OCI tenancy, with guests that get real, publicly reachable addresses through OCI's API. -Spinifex brings core AWS services to hardware you control: a node runs the Spinifex daemon, QEMU/KVM for guests, OVN for VPC networking, [Predastore](https://github.com/mulgadc/predastore) for S3 and [Viperblock](https://github.com/mulgadc/viperblock) for EBS, all behind a SigV4 endpoint the ordinary AWS SDKs and CLI talk to unmodified. - -**Terraform does the whole deployment** — the VCN, the block volumes, both VNICs per node, the install, the cluster formation and the OCI public-address allocator. Six steps below, one command, about ten minutes for a single node. - -Everything underneath — the architecture, the IAM policy, the quotas, every variable, the troubleshooting — is in [Architecture and Operations](./architecture.md). - ## Table of Contents -- [What you need](#what-you-need) +- [Overview](#overview) +- [Prerequisites](#prerequisites) - [1. Get the configuration](#1-get-the-configuration) - [2. Give the nodes an OCI credential](#2-give-the-nodes-an-oci-credential) - [3. Choose one node or three](#3-choose-one-node-or-three) @@ -42,11 +37,19 @@ Everything underneath — the architecture, the IAM policy, the quotas, every va - [Harden it before production](#harden-it-before-production) - [Next steps](#next-steps) - [Removing the deployment](#removing-the-deployment) -- [If something goes wrong](#if-something-goes-wrong) +- [Troubleshooting](#troubleshooting) --- -## What you need +## Overview + +Spinifex brings core AWS services to hardware you control: a node runs the Spinifex daemon, QEMU/KVM for guests, OVN for VPC networking, [Predastore](https://github.com/mulgadc/predastore) for S3 and [Viperblock](https://github.com/mulgadc/viperblock) for EBS, all behind a SigV4 endpoint the ordinary AWS SDKs and CLI talk to unmodified. + +**Terraform does the whole deployment** — the VCN, the block volumes, both VNICs per node, the install, the cluster formation and the OCI public-address allocator. Six steps below, one command, about ten minutes for a single node. + +Everything underneath — the architecture, the IAM policy, the quotas, every variable, the troubleshooting — is in [Architecture and Operations](../oci-architecture/README.md). + +## Prerequisites | | | | --- | --- | @@ -55,7 +58,9 @@ Everything underneath — the architecture, the IAM policy, the quotas, every va | **An OCI API key for the nodes** | Separate from the above, and step 2. Without it a node forms, looks healthy, and cannot give any guest a public address | | **Terraform or OpenTofu, `git`, Python 3** | On your workstation. The Python helper is standard library only, so there is nothing to `pip install` | -Check two quotas before you start, because both refuse at apply time rather than at plan time: **reserved public IPs** (50 per region, tenancy-wide) and the **compute limit for your shape**, which on a new tenancy is often zero for bare metal. [Quotas](./architecture.md#quotas) has the commands. +Check two quotas before you start, because both refuse at apply time rather than at plan time: **reserved public IPs** (50 per region, tenancy-wide) and the **compute limit for your shape**, which on a new tenancy is often zero for bare metal. [Quotas](../oci-architecture/README.md#quotas) has the commands. + +## Instructions ## 1. Get the configuration @@ -94,7 +99,7 @@ openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pe openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 -c # the fingerprint ``` -Upload the public key under **Identity → Users → API Keys**, and grant it the [ten operations Spinifex uses](./architecture.md#the-iam-policy) in your compartment — no more. +Upload the public key under **Identity → Users → API Keys**, and grant it the [ten operations Spinifex uses](../oci-architecture/README.md#the-iam-policy) in your compartment — no more. **The deploy installs the credential on each node by calling a small script of yours**, so no key material ever goes into user-data or Terraform state, where it would be readable from instance metadata for the life of the instance. Write one that puts these two files on each host it is given: @@ -103,14 +108,14 @@ Upload the public key under **Identity → Users → API Keys**, and grant it th | `/etc/spinifex/oci/oci_api_key.pem` | `0640 root:spinifex` | The private key | | `/etc/spinifex/oci/config` | `0640 root:spinifex` | An OCI SDK config, profile **`[spinifex]`**, naming the key by path | -The profile name must be `spinifex`. It is called as `your-hook ...` — the first argument is the private key to reach the nodes with, the rest are the hosts — and it must exit non-zero if any host failed, so a missing credential stops the deploy rather than surfacing later as a launch that cannot get an address. [Credentials](./architecture.md#credentials--an-api-key-or-an-instance-principal) has the config file's exact contents. +The profile name must be `spinifex`. It is called as `your-hook ...` — the first argument is the private key to reach the nodes with, the rest are the hosts — and it must exit non-zero if any host failed, so a missing credential stops the deploy rather than surfacing later as a launch that cannot get an address. [Credentials](../oci-architecture/README.md#credentials-an-api-key-or-an-instance-principal) has the config file's exact contents. `chmod +x` it. The deploy checks that before it builds anything, so a hook that is missing or not executable costs you a few seconds rather than forty minutes and a bare-metal bill. **Nothing in this repository ships that script, deliberately.** A credential belongs to whoever owns it, so the hook is yours to write and yours to keep. > [!TIP] -> If you are a tenancy admin you can skip key files entirely and authenticate the nodes as the instance itself. It needs a dynamic group and a policy created once at the tenancy root — see [instance principal](./architecture.md#an-instance-principal--no-key-material-but-it-needs-a-tenancy-admin). The API key path above is the one that works in any tenancy, including a compartment someone allocated to you. +> If you are a tenancy admin you can skip key files entirely and authenticate the nodes as the instance itself. It needs a dynamic group and a policy created once at the tenancy root — see [instance principal](../oci-architecture/README.md#an-instance-principal-no-key-material-but-it-needs-a-tenancy-admin). The API key path above is the one that works in any tenancy, including a compartment someone allocated to you. ## 3. Choose one node or three @@ -124,7 +129,7 @@ The profile name must be `spinifex`. It is called as `your-hook | **Fault domains** | One | Three, one per node, chosen by Terraform | | **Use it for** | Evaluation, a lab, an edge site with one box | Anything you would be unhappy to lose | -Two is not worth taking: it doubles the cost of a single node and gives you a cluster that cannot form a quorum. [Sizing](./architecture.md#sizing) covers shapes; bare metal is the recommendation, and `VM.Standard.E6.Flex` is the minimum. +Two is not worth taking: it doubles the cost of a single node and gives you a cluster that cannot form a quorum. [Sizing](../oci-architecture/README.md#sizing) covers shapes; bare metal is the recommendation, and `VM.Standard.E6.Flex` is the minimum. The pool has no configured size — Spinifex asks OCI for an address when a guest needs one. **Two ceilings bind it and neither is ours**: 64 secondary private IPs per VNIC, which is the 64 above and is not raisable, and the regional reserved-public-IP quota, which is tenancy-wide and shared with everything else you run on OCI. The second is the one you will hit first. @@ -151,7 +156,7 @@ data_volume_vpus_per_gb = 120 # 120 = Ultra High Performance EOF ``` -[Every Terraform variable](./architecture.md#every-terraform-variable) has the full list. +[Every Terraform variable](../oci-architecture/README.md#every-terraform-variable) has the full list. > [!NOTE] > **Leave `node_client_cidr_allow_list` at its `0.0.0.0/0` default for now.** It is an OCI security-list rule covering the whole public subnet, and your guests' public addresses cross it too — narrowing it to your own address cuts internet access to every guest, not just to the node. [Harden it before production](#harden-it-before-production) is where you lock the node down, in the layer that can tell the two apart. @@ -223,7 +228,7 @@ ssh -i ubuntu@ hostname ## Harden it before production -The deployment comes up usable, not locked down. Three layers filter traffic and they are easy to confuse — [Firewalls, in all three layers](./architecture.md#firewalls-in-all-three-layers) explains which is which. The short version: +The deployment comes up usable, not locked down. Three layers filter traffic and they are easy to confuse — [Firewalls, in all three layers](../oci-architecture/README.md#firewalls-in-all-three-layers) explains which is which. The short version: **1. Arm the Spinifex host firewall.** It is installed but not armed on this path. Arming it gives you the proper policy: the public plane open, the cluster plane scoped to peers only, and SSH scoped to a list you own. @@ -277,7 +282,7 @@ provider "aws" { > **Use the node's private address, not its public one.** The node certificate carries no SAN for the public address, because that address is never on the wire — OCI NATs it to a private one. A call to `https://:9999` fails TLS verification with _hostname doesn't match_. Run Terraform from a node, or from anything else inside the VCN. -The [Terraform workbooks](../terraform-workbooks/) we ship are the worked examples, and `e2e-cloudvendor-nightly` runs them on a single VM, on three VMs and on bare metal, publishing a table per topology on its own run page. Read that for the build you are installing. +The [Terraform workbooks](../terraform-workbooks/nginx-alb/README.md) we ship are the worked examples, and `e2e-cloudvendor-nightly` runs them on a single VM, on three VMs and on bare metal, publishing a table per topology on its own run page. Read that for the build you are installing. **Oracle Linux guests.** Four Oracle Linux images are in the catalog, so you can run the distro your Oracle support contract covers: @@ -295,16 +300,16 @@ sudo spx admin images import --name oracle-10.1-x86_64 --config /etc/spinifex/sp It destroys whatever that topology's Terraform state holds, so it can only remove what this configuration created. -## If something goes wrong +## Troubleshooting The deploy stops at the first failure and names the log it wrote, all under `.validate-/`. Add `--keep-on-fail` to leave a failed deployment up so you can log in and look — **it keeps billing** until you run `--destroy-only`. -[Troubleshooting](./architecture.md#troubleshooting) covers the symptoms worth knowing in advance, and several of them look like a different fault than they are: +[Troubleshooting](../oci-architecture/README.md#troubleshooting) covers the symptoms worth knowing in advance, and several of them look like a different fault than they are: -- [If a deploy stage fails](./architecture.md#if-a-deploy-stage-fails) — which log holds the answer -- [A guest's public address is unreachable](./architecture.md#a-guests-public-address-is-unreachable) -- [A node that was fine and then was not](./architecture.md#a-node-that-was-fine-and-then-was-not) — one command answers all of it -- [Guests cannot reach instance metadata](./architecture.md#guests-cannot-reach-instance-metadata) — OCI and AWS guests want the same address -- [Addresses and quotas](./architecture.md#addresses-and-quotas) — including why a detached address still bills +- [If a deploy stage fails](../oci-architecture/README.md#if-a-deploy-stage-fails) — which log holds the answer +- [A guest's public address is unreachable](../oci-architecture/README.md#a-guests-public-address-is-unreachable) +- [A node that was fine and then was not](../oci-architecture/README.md#a-node-that-was-fine-and-then-was-not) — one command answers all of it +- [Guests cannot reach instance metadata](../oci-architecture/README.md#guests-cannot-reach-instance-metadata) — OCI and AWS guests want the same address +- [Addresses and quotas](../oci-architecture/README.md#addresses-and-quotas) — including why a detached address still bills -Installing by hand instead of with Terraform is [Deploying without Terraform](./architecture.md#deploying-without-terraform). +Installing by hand instead of with Terraform is [Deploying without Terraform](../oci-architecture/README.md#deploying-without-terraform). diff --git a/docs/oci-integration/architecture.md b/docs/oci-integration/oci-architecture/README.md similarity index 95% rename from docs/oci-integration/architecture.md rename to docs/oci-integration/oci-architecture/README.md index 281b2f705..5b42919ba 100644 --- a/docs/oci-integration/architecture.md +++ b/docs/oci-integration/oci-architecture/README.md @@ -1,8 +1,11 @@ --- -title: "Spinifex on Oracle Cloud — Architecture and Operations" -seoTitle: "Spinifex on OCI: architecture, IAM, quotas and troubleshooting — Spinifex Docs" -description: "How Spinifex is wired on OCI — both VNICs, the iSCSI data volume, the public-address allocator — with the IAM policy, the quotas, every Terraform variable and the troubleshooting." -category: "Install" +title: "OCI Architecture" +seoTitle: "Spinifex on OCI: Architecture and Operations — Spinifex Docs" +description: "How Spinifex is wired on OCI: both VNICs, the iSCSI data volume and the public-address allocator, with the IAM policy, the quotas and the troubleshooting." +category: "Cloud Install" +sections: + - overview + - troubleshooting tags: - install - oci @@ -30,18 +33,20 @@ resources: - [Why run Spinifex on OCI](#why-run-spinifex-on-oci) - [Sizing](#sizing) - [How it fits together](#how-it-fits-together) -- [Credentials — an API key, or an instance principal](#credentials--an-api-key-or-an-instance-principal) +- [Credentials: an API key, or an instance principal](#credentials-an-api-key-or-an-instance-principal) - [The IAM policy](#the-iam-policy) - [Quotas](#quotas) - [Firewalls, in all three layers](#firewalls-in-all-three-layers) - [What Terraform builds, per node](#what-terraform-builds-per-node) - [Every Terraform variable](#every-terraform-variable) - [The `oci` CLI](#the-oci-cli) -- [Troubleshooting](#troubleshooting) - [Deploying without Terraform](#deploying-without-terraform) +- [Troubleshooting](#troubleshooting) --- +## Overview + ## Why run Spinifex on OCI **Lower cost for the same shape of workload.** OCI's compute and block storage are priced well below the large US clouds for equivalent OCPU and IOPS, and Spinifex turns one large instance into a whole EC2/EBS/S3 surface — so a tenant's twenty small instances are twenty QEMU guests on hardware you are billed for once, not twenty billable cloud instances. @@ -74,7 +79,7 @@ Everything in [Single-Node Install](/docs/install) and [Multi-Node Install](/doc One instance runs the control plane and every tenant guest. Customers reach the AWS APIs on the node's own address; their instances reach the internet through addresses registered on the node's second VNIC.

- Single node on OCI — the instance inside the VCN public subnet, both VNICs, br-wan, the routed-NAT transit veth, the control plane, tenant VPCs and the iSCSI data volume + Single node on OCI — the instance inside the VCN public subnet, both VNICs, br-wan, the routed-NAT transit veth, the control plane, tenant VPCs and the iSCSI data volume

Two things in that picture are the whole integration. The guest's private address (`10.0.1.4`) never leaves the host — OVN and the host route both hold it. The address OCI delivers to is a **secondary private IP registered on VNIC 1**, appearing on `br-wan`, and the customer's public address (`150.230.13.131`) is a reserved public IP that OCI 1:1-NATs onto it upstream. `describe-instances` reports the public one; every datapath object holds the private one, which is why the two never appear together on the host. @@ -86,7 +91,7 @@ The third is storage. Every guest disk, every S3 object and the JetStream state Each node registers the addresses for **its own** guests on **its own** second VNIC, and the allocator runs per node, so nothing has to know another node's OCIDs. When a guest moves — a stop/start that lands elsewhere, or a host failure — the node it lands on claims the address's private half from the old VNIC through a single OCI call. The OCID and the public half are untouched, so the customer's address never changes.

- Three nodes on OCI — three instances inside the VCN public subnet, one per fault domain, each with its own VNICs, addresses and iSCSI volume, joined east-west by Geneve, NATS, predastore and the OVN raft + Three nodes on OCI — three instances inside the VCN public subnet, one per fault domain, each with its own VNICs, addresses and iSCSI volume, joined east-west by Geneve, NATS, predastore and the OVN raft

The overlay, the object shards and the OVN databases all cross **VNIC 0**, inside the VCN. Size that plane, not the external one: guest-to-guest traffic between nodes is Geneve over the VCN, and it is the link every distributed layer shares. @@ -94,7 +99,7 @@ The overlay, the object shards and the OVN databases all cross **VNIC 0**, insid > [!NOTE] > **The Spinifex region is your own naming and has nothing to do with the OCI region.** `ap-southeast-2` in `spinifex.toml` beside `ap-sydney-1` in OCI is correct and expected. Do not read one as evidence about the other. -## Credentials — an API key, or an instance principal +## Credentials: an API key, or an instance principal The allocator needs OCI credentials at runtime. **A node with none forms, passes every health check, and then refuses every launch that wants a public address** with `InsufficientAddressCapacity`, naming no cause outside its own journal. @@ -133,7 +138,7 @@ key_file=/etc/spinifex/oci/oci_api_key.pem **Put that in a credential hook and the deploy installs it on every node.** The hook is an executable of yours, run as `hook ...` after formation and before the pool is configured, which is the only window where a node has `/etc/spinifex` but has not started the allocator. It is yours rather than ours deliberately: a credential belongs to whoever owns it, and one rendered into user-data or Terraform state is readable from instance metadata for the life of the instance — on a host that runs other people's guests. -### An instance principal — no key material, but it needs a tenancy admin +### An instance principal: no key material, but it needs a tenancy admin The instance certificate is served at `/opc/v2/identity/cert.pem` and the Go SDK authenticates as the instance with no key on disk. It must also be _authorised_, by a dynamic group and a policy, and **both are tenancy-root resources** — `oci_identity_dynamic_group` is created with `compartment_id` set to the tenancy OCID. A compartment-scoped user cannot create them, and the attempt fails with `404-NotAuthorizedOrNotFound` on `CreateDynamicGroup`, which reads like a misconfiguration and is not one. @@ -276,7 +281,7 @@ Two OCI-specific notes on top of that guide: | `oci_core_volume_attachment` | **`attachment_type = "iscsi"`**, with the Oracle Cloud Agent's Block Volume Management plugin enabled so the node logs the iSCSI session in itself | | cloud-init | Partitions, formats, mounts and `fstab`s the volume; builds `br-wan` over the second VNIC; opens the service ports | -The resource-by-resource architecture is in [`scripts/terraform/oci-spx`](../../scripts/terraform/oci-spx/README.md). +The resource-by-resource architecture is in [`scripts/terraform/oci-spx`](https://github.com/mulgadc/spinifex/tree/dev/scripts/terraform/oci-spx). **The data volume is iSCSI, and that is the detail that bites.** OCI presents the volume as an iSCSI target reachable at `169.254.2.2:3260` — attaching it in the API does not put a block device on the host. `is_agent_auto_iscsi_login_enabled` makes the Oracle Cloud Agent do the login, and because it does that _asynchronously_, the device is not there when cloud-init first runs. The mount script waits up to ten minutes for it, formats it **only if `blkid` reports no filesystem** (so a re-run cannot erase a populated volume), and writes an fstab entry by UUID: @@ -330,6 +335,24 @@ oci --version --- +## Deploying without Terraform + +**Terraform automates every step of a normal install, and that is the point of it.** The `oci-spx` configuration builds the VCN, the subnets, the gateways, the block volumes and both VNICs per node, then renders the cloud-init that installs Spinifex and sets up OVN — so the sequence below happens without anyone typing it, in the same order every time, on one node or three. + +If you need to install by hand — an existing OCI tenancy you cannot run Terraform against, an unsupported shape, or a stage you are debugging — the install itself is not OCI-specific. Follow the standard guides: + +- [Installing Spinifex](../../install/install/README.md) for a single node. +- [Multi-Node Installation](../../install/install-multi-node/README.md) for a cluster, which covers `spx admin init` on the leader and `spx admin join` on the rest. + +Four things about OCI are not in those guides, and each has its own section above: + +- **The external pool** is `source = "oci"`, which needs the API key or instance principal from [Credentials](#credentials-an-api-key-or-an-instance-principal) and the [IAM policy](#the-iam-policy). Without it a node forms, passes every health check, and then refuses every launch that wants a public address. +- **`br-wan` stays a Linux bridge** owned by netplan, linked to OVS `br-ext` by a veth pair. Pass `setup-ovn.sh --wan-bridge=br-wan` and it detects this itself; the NIC never becomes an OVS port, so the VNIC keeps its MAC identity. See [How it fits together](#how-it-fits-together). +- **The block volume must be mounted with `_netdev`**, because it arrives over iSCSI and the mount is otherwise attempted before a session exists. +- **IMDS needs the 169.254 remap** described under [Guests cannot reach instance metadata](#guests-cannot-reach-instance-metadata), since the guest's metadata address collides with OCI's own. + +--- + ## Troubleshooting ### If a deploy stage fails @@ -445,21 +468,3 @@ ip -br addr show | grep ime- # holds 169.254.42.x, not .169.x sudo ovs-vsctl --columns=name,error list Interface sudo journalctl -u ovs-vswitchd --since -10m ``` - ---- - -## Deploying without Terraform - -**Terraform automates every step of a normal install, and that is the point of it.** The `oci-spx` configuration builds the VCN, the subnets, the gateways, the block volumes and both VNICs per node, then renders the cloud-init that installs Spinifex and sets up OVN — so the sequence below happens without anyone typing it, in the same order every time, on one node or three. - -If you need to install by hand — an existing OCI tenancy you cannot run Terraform against, an unsupported shape, or a stage you are debugging — the install itself is not OCI-specific. Follow the standard guides: - -- [Installing Spinifex](../install/install/README.md) for a single node. -- [Multi-Node Installation](../install/install-multi-node/README.md) for a cluster, which covers `spx admin init` on the leader and `spx admin join` on the rest. - -Four things about OCI are not in those guides, and each has its own section above: - -- **The external pool** is `source = "oci"`, which needs the API key or instance principal from [Credentials](#credentials--an-api-key-or-an-instance-principal) and the [IAM policy](#the-iam-policy). Without it a node forms, passes every health check, and then refuses every launch that wants a public address. -- **`br-wan` stays a Linux bridge** owned by netplan, linked to OVS `br-ext` by a veth pair. Pass `setup-ovn.sh --wan-bridge=br-wan` and it detects this itself; the NIC never becomes an OVS port, so the VNIC keeps its MAC identity. See [How it fits together](#how-it-fits-together). -- **The block volume must be mounted with `_netdev`**, because it arrives over iSCSI and the mount is otherwise attempted before a session exists. -- **IMDS needs the 169.254 remap** described under [Guests cannot reach instance metadata](#guests-cannot-reach-instance-metadata), since the guest's metadata address collides with OCI's own.