Ben Duncan
c6841ce1f0
* feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts.
title, seoTitle, description, category, tags, resources
| title | seoTitle | description | category | tags | resources | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Spinifex on Oracle Cloud | Run Spinifex on Oracle Cloud Infrastructure — Spinifex Docs | Deploy Spinifex on OCI end to end with Terraform, single node or three, and give guests real public addresses through OCI's own API. | Install |
|
|
Running Spinifex on Oracle Cloud Infrastructure
Run the AWS surface — EC2, EBS, S3, VPC — inside your own OCI tenancy, with guests that get real, publicly reachable addresses through OCI's API.
Spinifex brings core AWS services to hardware you control: a node runs the Spinifex daemon, QEMU/KVM for guests, OVN for VPC networking, Predastore for S3 and Viperblock for EBS, all behind a SigV4 endpoint the ordinary AWS SDKs and CLI talk to unmodified.
This guide runs that stack on OCI. Terraform does the whole deployment — the VCN, the block volumes, both VNICs per node, the install, the cluster formation and the OCI public-address allocator — so Deploy is four steps and one command, and everything after it is reference. One node is a working install; three is the minimum that survives losing a node. Every command below was run against a live OCI tenancy.
Table of Contents
Deploy
What you need
| On your workstation | Why |
|---|---|
Terraform or OpenTofu, git, Python 3 |
Python is used only by a helper that is standard library only, so there is nothing to pip install |
An SSH keypair at ~/.ssh/oci-spx |
Created in step 1. The deploy refuses to start without it rather than building instances you cannot log in to |
| An OCI compartment OCID | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have |
OCI API credentials in ~/.oci/config |
For Terraform itself, to build the infrastructure |
| A way for the nodes to call OCI | Separate from the above, and the one prerequisite worth reading first: an instance principal or an API key |
Important
That last row is the one that bites. Terraform's credentials build the infrastructure; the nodes need their own credentials at runtime to allocate public addresses. A node with none forms, passes every health check, and then refuses every launch that wants a public address with
InsufficientAddressCapacity, naming no cause outside its own journal. Decide between the two now — it changes one argument in step 4.
1. Get the configuration
The Terraform configuration and the deploy driver live in the Spinifex repository:
git clone https://github.com/mulgadc/spinifex.git
cd spinifex
git switch dev
cd scripts/terraform/oci-spx
# The deploy installs this on every node and logs in with it. Any type OpenSSH
# supports is fine; it must be at this path, or named with --ssh-public-key.
ssh-keygen -t ed25519 -f ~/.ssh/oci-spx -N '' -C spinifex-oci
Note
git switch devis needed until the next release ships. OCI support is not inv1.21.0, so the configuration you are about to run is not onmainyet. That ref decides the Terraform and the driver only — which Spinifex build lands on the nodes is a separate choice, made by--versionin step 4.
2. Choose one node or three
This is the only architectural decision, and it is one line in step 3.
| Single node | Three nodes | |
|---|---|---|
node_count |
1 |
3 |
| Survives losing a node | No | Yes — NATS keeps quorum, predastore RS(2,1) keeps the data readable, the OVN raft keeps a leader |
| Public addresses available | 64 — the per-VNIC secondary private IP limit | 192, three VNICs' worth, still bounded by the regional reserved-public-IP quota |
| Fault domains | One | Three, one per node, chosen by Terraform |
| Use it for | Evaluation, a lab, an edge site with one box | Anything you would be unhappy to lose |
Two is not an option worth taking: it doubles the cost of a single node and gives you a cluster that cannot form a quorum. See Cluster sizing and Sizing for shapes.
3. Write your Terraform inputs
One hand-edited file, in the directory you are already in. Every variable has a default that builds a working single node, so only compartment_ocid is strictly required:
cat > terraform.auto.tfvars <<'EOF'
compartment_ocid = "ocid1.compartment.oc1..aaaa..."
deployment_name = "spinifex"
# Shape and size. Bare metal is preferred; a flex VM is what the quota allows.
compute_shape = "VM.Standard.E6.Flex"
compute_ocpus = 8 # OCI counts an OCPU as a full core
compute_memory_in_gbs = 32
# 1 for a single node, 3 for a cluster. This is the only line that chooses.
node_count = 3
# The block volume behind /var/lib/spinifex, attached over iSCSI.
data_volume_size_in_gbs = 256
data_volume_vpus_per_gb = 120 # 120 = Ultra High Performance
node_client_cidr_allow_list = ["203.0.113.10/32"]
EOF
Add instance_principal = "adopt" if you went that way in Credentials. Every Terraform variable has the full list.
Warning
node_client_cidr_allow_listdefaults to0.0.0.0/0. Nodes sit in a public subnet with public addresses because they serve the UI, the S3 gate and the AWS gateway directly — there is no bastion. Restrict it to your own address before any real use, as the example above does.
4. Run the deploy
One command builds the infrastructure, installs Spinifex, forms the cluster, configures OCI public addressing and verifies the result:
./validate-topology.sh \
--topology vm-single \
--version v1.21.0-70-g1743b602-dev-1-g66bc1172-dev \
--credential-hook ~/.spinifex/oci-credential-hook.sh \
--keep
About ten minutes for a single node, printing each stage as it finishes. The four arguments:
| Argument | What to pass |
|---|---|
--topology |
vm-single for one VM, vm-multi for a three-node cluster, bm for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission |
--version |
The Spinifex release to install on the nodes. Use the tag above until the next release ships, because OCI support is not in the current stable release yet |
--credential-hook |
Only when nodes authenticate with an API key — see Credentials. With an instance principal, replace it with --instance-principal |
--keep |
This is what makes it a deployment rather than a test. Without it the driver destroys everything at the end, which is correct for CI and not what you want here |
Also worth knowing:
--skip-workloadstops after verifying the cluster. Without it the driver launches real guests on public addresses and tears them down again, which is the only check that proves end-to-end connectivity rather than inferring it.--dry-runprints the plan and the per-node steps and changes nothing.--ssh-public-key/--ssh-private-keyif your keypair is not at~/.ssh/oci-spx.
Tip
Prefer
--version <tag>over--channel dev. Both install a prerelease, but--channel devresolves through a GitHub API call that is rate limited per source address, and when it trips it returns a 404 that looks exactly like a missing release. A tag is a plain redirect and has no such failure mode.
What a successful run looks like
The stages, in order, and all of them have to appear:
[validate-vm-single] VM.Standard.E6.Flex, 1 node(s), state in .validate-vm-single
[validate-vm-single] building
[validate-vm-single] hosts: 150.230.13.131
[validate-vm-single] waiting for cloud-init on each node
[validate-vm-single] 150.230.13.131 cloud-init: status: done
[validate-vm-single] installing Spinifex on each node (published release)
[validate-vm-single] 150.230.13.131 installed: spinifex v1.21.0-70-g1743b602-dev
[validate-vm-single] initializing the single node
[validate-vm-single] running the credential hook
[validate-vm-single] configuring the IMDS remap and the external pool
[validate-vm-single] checking the allocator came up
[validate-vm-single] 150.230.13.131 allocator ready
[validate-vm-single] cluster membership
[validate-vm-single] --keep: leaving the infrastructure up, no verdict recorded
allocator ready is the line to look for. Everything above it would also appear on a node that cannot allocate a public address, and that failure is otherwise silent until the first launch. --keep records no verdict by design: a deployment left running has not been proved destroyable, so the driver declines to call it a pass.
If a stage fails
The driver stops at the first failure, names the log, and leaves the rest alone. Logs are all under .validate-<topology>/:
| Stage | Log |
|---|---|
| Building the infrastructure | apply.log |
| Installing Spinifex | install-<host>.log |
| Forming the cluster | form.log |
| Installing the credential, configuring the pool | credential-hook.log, pool-<host>.log |
| Verifying public addressing | allocator-<host>.log |
The last one is the log worth knowing about, because public addressing is the part with the most moving pieces. If the allocator never comes up, the driver captures the daemon journal, the pool as configured, the bridge and OCI MAC addresses, and whether each credential file exists and is readable — never the credentials themselves — before anything is torn down. Troubleshooting reads those.
Add --keep-on-fail to leave a failed deployment running so you can log in and look. It keeps billing until you destroy it.
Removing the deployment
./validate-topology.sh --topology vm-single --destroy-only
It destroys whatever that topology's Terraform state holds, so it can only remove what this configuration created.
Verify end to end
export AWS_PROFILE=spinifex
aws ec2 allocate-address
aws ec2 run-instances --image-id <ami> --instance-type t3.micro --key-name <key> \
--subnet-id <subnet> --associate-public-ip-address
aws ec2 describe-instances --query \
'Reservations[].Instances[].[InstanceId,PrivateIpAddress,PublicIpAddress,State.Name]' \
--output text
Then reach the guest on its public address — from your workstation, not from the node:
ping -c 3 <public-ip>
ssh -i <key> ubuntu@<public-ip> hostname
Important
Probe a public address from outside the VCN, not from a node. Two things make a node a bad vantage point, and they are indistinguishable from each other and from a broken datapath:
node_client_cidr_allow_listmay not admit the node's own public address, and a VCN may not reflect its public addresses back into itself at all. Either way a healthy cluster fails the probe, so a timeout from a node is not evidence about the guest. Private addresses are the separate question and do work from inside.
Confirm the OCI allocator came up, which spx get nodes cannot tell you. Run this on the node:
sudo journalctl -u spinifex-vpcd --since -5m | grep -i ocinet
"msg":"ocinet resolved the external VNIC from its interface","pool":"oci-public","iface":"br-wan",
"vnic_id":"ocid1.vnic.oc1.ap-sydney-1.abzxsljrjwlf...kimuq","private_ip":"10.200.1.31","subnet":"10.200.0.0/23"
"msg":"OCI allocator ready","pool":"oci-public","collected":0,"stale_bindings":0,"skipped":1
resolved the external VNIC is the line that matters — it proves the credentials work, the compartment is right, and br-wan's MAC matched a real VNIC. A node missing it will accept allocate-address and fail it.
Note
On a cold cluster start you may instead see
OCI allocator reconcile failed … nats: no responders available for requeston some nodes. That is vpcd racing JetStream's KV at boot; the startup reconcile is skipped and not retried. It is harmless on a new cluster where nothing has been allocated, and it is tracked — restartspinifex-vpcdon that node to run it. Theresolved the external VNICline above is still the one that decides whether allocation works.
And from inside the guest — note that IMDS is IMDSv2, so it needs a token; an untokened request correctly returns 401:
TOK=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
-H 'X-aws-ec2-metadata-token-ttl-seconds: 120')
curl -s -H "X-aws-ec2-metadata-token: $TOK" \
http://169.254.169.254/latest/meta-data/instance-id
traceroute -n 8.8.8.8
Next steps
Set up your cluster
The cluster is running, but it holds nothing yet — no machine images, no networks, no instances.
Continue to Setting Up Your Cluster to import an AMI, create an SSH key pair, create a VPC with a public subnet, and launch your first instance. It ends by arming the host firewall, which is also where you re-arm it if you turned it off to form the cluster.
Run your existing Terraform against the cluster
This is the part worth pausing on. Everything above builds infrastructure on OCI using the OCI provider. Everything from here uses the AWS provider — unmodified, from the OpenTofu or Terraform registry — pointed at your own cluster. The same aws_vpc, aws_instance, aws_db_instance, aws_ecs_service and aws_eks_cluster resources a team already has in git apply against Spinifex on an OCI tenancy, with no rewriting and no OCI-specific module.
Three things differ from a provider block aimed at AWS, and only three:
provider "aws" {
region = "ap-southeast-2"
access_key = var.access_key
secret_key = var.secret_key
endpoints {
ec2 = var.spinifex_endpoint # https://<node private IP>:9999
s3 = var.predastore_endpoint # https://<node private IP>:8443
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
}
s3_use_path_style = true
skip_metadata_api_check = true
skip_region_validation = true
}
Credentials come from the [spinifex] profile that spx admin init writes into ~/.aws/credentials on node 1.
Use the node's private address, not its OCI reserved public IP. The node certificate carries no SAN for the public address, because that address is never on the wire — OCI NATs it to a private one. An
awsorterraformcall tohttps://<public IP>:9999fails TLS verification with hostname doesn't match. Until that is fixed, run Terraform from a node, or from anything else inside the VCN.
That portability is a claim, so it is measured rather than asserted. Each workbook below is a full apply, a functional assertion against the thing it built, and a destroy, measured on 2026-09-26 against a three-node cluster of VM.Standard.E6.Flex instances in ap-sydney-1 running spinifex v1.20.0-113.
This is a snapshot, and the current answer is the nightly. e2e-cloudvendor-nightly runs the same workbooks on a single VM, on three VMs and on bare metal, and publishes a table per topology on its own run page. Read that for the build you are installing rather than taking this table as current.
| Workbook | What it exercises | Result |
|---|---|---|
nginx-webserver |
VPC, subnet, IGW, security group, key pair, EC2 instance, public IP; HTTP 200 from the internet | Passed (86s) |
bastion-private-subnet |
Public and private subnets, NAT egress, a bastion, SSH hop to a private host | Passed (117s) |
rds-quickstart |
DB subnet group, parameter group, aws_db_instance (PostgreSQL), client instance, connection |
Passed (296s) |
ecs-quickstart |
ECS cluster, task definition, service, container instances, ALB with a healthy target | Passed (166s) |
eks-quickstart |
EKS control plane, managed node group, kubectl against the cluster, nodes Ready |
Passed (259s) |
nginx-alb |
Application Load Balancer across two subnets, two backends, health checks | Passed (123s) |
s3-webapp |
S3 bucket, IAM role and instance profile, IMDS-fetched credentials, upload from the guest | Passed |
demo-app |
Container image for the EKS workbooks, pushed to ECR | Passed |
eks-https-ingress |
Private-subnet workers behind a NAT gateway, LBC addon, ACM certificate, HTTPS Ingress | Passed (252s) |
eks-gitops-argocd |
Private-subnet workers, Argo CD addon, EBS-CSI PersistentVolume, GitOps sync | Passed (225s) |
Oracle Linux as a guest image
Spinifex ships four Oracle Linux entries in its image catalog, so a customer on OCI can run the same distro their Oracle support contract covers. They import exactly like any other AMI — nothing about them is OCI-specific, and they run equally well on bare metal:
| Catalog name | Release | Arch | Kernel |
|---|---|---|---|
oracle-10.1-x86_64 |
Oracle Linux 10.1 | x86_64 | UEK 8 |
oracle-10.1-arm64 |
Oracle Linux 10.1 | arm64 | UEK 8 |
oracle-9.8-x86_64 |
Oracle Linux 9.8 | x86_64 | UEK 7 |
oracle-9.8-arm64 |
Oracle Linux 9.8 | arm64 | UEK 7 |
sudo spx admin images import --name oracle-10.1-x86_64 --config /etc/spinifex/spinifex.toml
aws ec2 describe-images --query 'Images[].[Name,State,BootMode]' --output text
All four boot UEFI, so launch them into a shape that boots UEFI — a BIOS-only instance type will not come up, and the failure looks like a hung boot rather than a rejected image.
The login user is cloud-user, not opc, and this is the one that catches people:
ssh -i path/to/key.pem cloud-user@<public-ip>
opc is the default user on the images Oracle publishes into OCI itself; these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on both releases: opc, oracle and ec2-user are all refused. The Spinifex UI's instance detail page shows the right user per AMI, so read it there rather than guessing from the distro.
Reference
Why run Spinifex on OCI
Lower cost for the same shape of workload. OCI's compute and block storage are priced well below the large US clouds for equivalent OCPU and IOPS, and Spinifex turns one large instance into a whole EC2/EBS/S3 surface — so a tenant's twenty small instances are twenty QEMU guests on hardware you are billed for once, not twenty billable cloud instances.
Egress stops being the line item that decides the architecture. Oracle publishes a large monthly outbound-transfer allowance per tenancy and a per-GB rate beyond it that is roughly an order of magnitude below the majors — check the current pricing page rather than a number in a document, but the gap is the reason this integration exists. Traffic between guests never leaves the host or, on a cluster, never leaves the VCN: it is Geneve over the private plane, which is not billed as internet egress at all.
Portability — the workload stops being AWS-shaped and starts being yours. The API your tenants code against is Spinifex's, so the same Terraform, the same SDK calls and the same AMIs run on OCI today, on bare metal in your own rack tomorrow, and at an edge site after that. Moving off AWS becomes a deployment decision rather than a rewrite, and moving again costs nothing extra.
You keep the parts of OCI that are worth keeping. Bare-metal shapes, fault domains, block volumes with a chosen performance tier, and real public addresses out of OCI's own pool. Spinifex does not hide the cloud underneath it — it registers each guest's address with OCI so that guest is genuinely reachable on the internet, not behind a shared NAT.
Sizing
Bare metal is the recommendation. BM.Standard.E2.64 or similar gives you the hardware directly — no nested virtualisation, full NIC performance, and no hypervisor between your guests and the wire. If bare metal is out of scope, a single VM.Standard.E6.Flex is the recommendation for testing and development, or three of them for production.
| Minimum | Recommended | |
|---|---|---|
| Shape | VM.Standard.E6.Flex |
Bare metal, e.g. BM.Standard.E2.64 |
| OCPU | 8 (16 vCPU) | 32+ |
| RAM | 32 GB | 128 GB+ |
| Data volume | 256 GiB block volume | 1 TB+ NVMe-backed |
| VNICs | 2 | 2 |
| Nodes | 1 | 3 — see Cluster sizing |
How it fits together
Everything in Single-Node Install and Multi-Node Install still applies: formation, storage and OVN behave exactly as they do on hardware you own. What changes is the layer underneath — an OCI VCN is not an Ethernet segment, so the external datapath is wired differently, and public addresses come from OCI's API rather than from a range you write in a config file.
Single node — the whole AWS surface inside one OCI instance. One instance runs the control plane and every tenant guest. Customers reach the AWS APIs on the node's own address; their instances reach the internet through addresses registered on the node's second VNIC.
Two things in that picture are the whole integration. The guest's private address (10.0.1.4) never leaves the host — OVN and the host route both hold it. The address OCI delivers to is a secondary private IP registered on VNIC 1, appearing on br-wan, and the customer's public address (150.230.13.131) is a reserved public IP that OCI 1:1-NATs onto it upstream. describe-instances reports the public one; every datapath object holds the private one, which is why the two never appear together on the host.
The third is storage. Every guest disk, every S3 object and the JetStream state all live under /var/lib/spinifex, which is an OCI block volume attached over iSCSI, not the boot disk. Size it for the guests you intend to run; a boot volume that quietly ends up carrying the stack is the most common way an install on OCI fails later rather than immediately.
Three nodes — the same surface, distributed. Each node registers the addresses for its own guests on its own second VNIC, and the allocator runs per node, so nothing has to know another node's OCIDs. When a guest moves — a stop/start that lands elsewhere, or a host failure — the node it lands on claims the address's private half from the old VNIC through a single OCI call. The OCID and the public half are untouched, so the customer's address never changes.
The overlay, the object shards and the OVN databases all cross VNIC 0, inside the VCN. Size that plane, not the external one: guest-to-guest traffic between nodes is Geneve over the VCN, and it is the link every distributed layer shares.
Note
The Spinifex region is your own naming and has nothing to do with the OCI region.
ap-southeast-2inspinifex.tomlbesideap-sydney-1in OCI is correct and expected. Do not read one as evidence about the other.
Credentials — an instance principal, or an API key
The allocator needs OCI credentials at runtime, and there are two ways to give it them.
An instance principal is the better one, and it is a one-time setup step. The instance certificate is served at /opc/v2/identity/cert.pem and the Go SDK authenticates as the instance with no key material on disk — but it must also be authorised, by a dynamic group and a policy. On the reference tenancy it authenticated without being authorised for anything, which is the silent failure above. Those two resources live at the tenancy root, so creating them needs a tenancy admin once; every deployment and rebuild afterwards needs only a compartment-scoped user:
cd scripts/terraform/oci-spx
./setup-identity.sh --dry-run # always first
./setup-identity.sh # needs a tenancy-admin OCI profile
Then set instance_principal = "adopt" in your tfvars and pass --instance-principal to the deploy. The dynamic group matches on instance.compartment.id, so it covers every node you ever build in that compartment and nothing needs updating when nodes are replaced.
An API key is the fallback, and the default, because it needs nothing from a tenancy admin — which makes it the only option when the compartment belongs to someone else's tenancy. The cost is key material on disk, with a rotation obligation, on every node:
# On your workstation, not the node.
openssl genrsa -out ~/.oci/oci_api_key.pem 2048
chmod 600 ~/.oci/oci_api_key.pem
openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem
openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem \
| openssl md5 -c # this is the fingerprint
Upload the public key to the user under Identity → Users → API Keys, and record the user OCID, the fingerprint and the tenancy OCID.
Each node then needs two files, and a node with only the first is the silent failure — it forms, looks healthy, and cannot allocate:
| Path | Mode | Contents |
|---|---|---|
/etc/spinifex/oci/oci_api_key.pem |
0640 root:spinifex |
The private key |
/etc/spinifex/oci/config |
0640 root:spinifex |
An ordinary OCI SDK config, profile [spinifex], naming the key by path |
Nothing creates the second one for you, and its profile name must be spinifex to match oci_config_profile in the pool:
[spinifex]
user=ocid1.user.oc1..aaaa...
fingerprint=12:34:56:78:90:ab:cd:ef:12:34:56:78:90:ab:cd:ef
tenancy=ocid1.tenancy.oc1..aaaa...
region=ap-sydney-1
key_file=/etc/spinifex/oci/oci_api_key.pem
Put that in a credential hook and the deploy installs it on every node. The hook is an executable of yours, run as hook <ssh-key> <host>... after formation and before the pool is configured, which is the only window where a node has /etc/spinifex but has not started the allocator. It is yours rather than ours deliberately: a credential belongs to whoever owns it, and one rendered into user-data or Terraform state is readable from instance metadata for the life of the instance — on a host that runs other people's guests.
The IAM policy
Spinifex calls exactly nine operations. Grant no more than these:
| Operation | Why |
|---|---|
CreatePrivateIp |
Register the secondary private IP the datapath carries |
DeletePrivateIp |
Release it |
GetPrivateIp |
Poll for AVAILABLE after an asynchronous assign |
ListPrivateIps |
The startup reconcile, and the affinity pass that moves an address between nodes |
UpdatePrivateIp |
Reassign a private IP to this node's VNIC when a guest moves here |
CreatePublicIp |
Create the RESERVED public IP |
DeletePublicIp |
Release it |
GetPublicIp / GetPublicIpByPrivateIpId |
Poll lifecycle state, and reconcile a private IP back to its public half |
UpdatePublicIp |
Attach and detach the public IP from a private IP |
The matching policy, scoped to the one compartment Spinifex allocates in:
Allow group SpinifexOperators to manage private-ips in compartment <compartment-name>
Allow group SpinifexOperators to manage public-ips in compartment <compartment-name>
Allow group SpinifexOperators to read vnics in compartment <compartment-name>
Allow group SpinifexOperators to read subnets in compartment <compartment-name>
Allow group SpinifexOperators to read vcns in compartment <compartment-name>
Scope it to a compartment, not the tenancy. manage public-ips at tenancy level lets a compromised node consume the whole regional reserved-public-IP quota.
Do not grant manage instances or manage vnics. Spinifex never creates, attaches or deletes a VNIC — it only adds addresses to a VNIC that already exists. A grant that allows VNIC lifecycle is strictly more than the integration can use.
Quotas
Two limits bound how many public addresses a deployment can hand out, and they do not scale the same way:
| Quota | Default | Scope | Notes |
|---|---|---|---|
| Reserved public IPs | 50 | Per region, whole tenancy | Shared by every node, so it does not grow with node count. This is the wall you hit first. |
| Secondary private IPs per VNIC | 64 | Per VNIC, so per node | Not raisable. Three nodes gives you 192 — this is the limit that does scale. |
A raise on the first is a support request, not a setting: Console → Governance & Administration → Limits, Quotas and Usage → "Request a service limit increase". Cite the exact limit name from oci limits definition list, not a name from documentation.
oci limits definition list --service-name vcn --compartment-id <tenancy-ocid>
oci limits value list --service-name vcn --compartment-id <tenancy-ocid>
oci limits resource-availability get --service-name vcn \
--limit-name <name-from-above> --compartment-id <tenancy-ocid>
These calls need a grant the node does not have, and must not be given. Limits and usage are tenancy-level, which cannot be compartment-scoped the way the policy above insists the node's is. Create a second, read-only principal for an operator or a reporting job and keep it off every node:
Allow group SpinifexReporting to read limits in tenancy
Allow group SpinifexReporting to read usage-reports in tenancy
The second verb is what Cost Analysis and oci usage-api usage-summary request-summarized-usages read. Adding either to SpinifexOperators would hand every node tenancy-wide read, which is exactly what the node policy refuses.
The oci CLI
The oci CLI is not on the Ubuntu image, and apt has no python3-oci-cli package. Nothing in Deploy needs it; install it when you want to inspect allocations by hand:
sudo apt-get update && sudo apt-get install -y python3-venv
python3 -m venv ~/.oci-cli-venv
~/.oci-cli-venv/bin/pip install --upgrade pip oci-cli
sudo ln -sf ~/.oci-cli-venv/bin/oci /usr/local/bin/oci
oci --version
What the CLI is and is not for. Spinifex's daemon uses the OCI Go SDK for every allocation — the CLI is not on that path. You need it for bootstrap and for operational inspection. Do not build automation that shells out to it during an allocation; allocations run inside a request budget that a forked Python process will not respect.
What Terraform builds, per node
| Resource | Detail |
|---|---|
oci_core_instance |
One per node_count, round-robined across fault domains so three nodes land on three sets of racks rather than wherever placement puts them |
| Primary VNIC | Public IP, hostname_label, the host plane — SSH, the advertise address, the Geneve endpoint |
oci_core_vnic_attachment |
A second VNIC in the same subnet. Every Spinifex external address becomes a secondary private IP here at runtime |
oci_core_volume |
A block volume per node, data_volume_size_in_gbs at data_volume_vpus_per_gb |
oci_core_volume_attachment |
attachment_type = "iscsi", with the Oracle Cloud Agent's Block Volume Management plugin enabled so the node logs the iSCSI session in itself |
| cloud-init | Partitions, formats, mounts and fstabs the volume; builds br-wan over the second VNIC; opens the service ports |
The resource-by-resource architecture is in scripts/terraform/oci-spx.
The data volume is iSCSI, and that is the detail that bites. OCI presents the volume as an iSCSI target reachable at 169.254.2.2:3260 — attaching it in the API does not put a block device on the host. is_agent_auto_iscsi_login_enabled makes the Oracle Cloud Agent do the login, and because it does that asynchronously, the device is not there when cloud-init first runs. The mount script waits up to ten minutes for it, formats it only if blkid reports no filesystem (so a re-run cannot erase a populated volume), and writes an fstab entry by UUID:
UUID=<uuid> /var/lib/spinifex ext4 defaults,_netdev,nofail 0 2
_netdev is what orders the mount after the network, and therefore after iscsid has a session. nofail is what keeps a missing volume from wedging the boot. Mounting /var/lib/spinifex itself rather than somewhere else and symlinking is deliberate: viperblock, predastore and JetStream all land on the volume with no configuration pointing anywhere unusual.
Every Terraform variable
All of these have defaults that build a working single node, except the first:
| Variable | Default | What it decides |
|---|---|---|
compartment_ocid |
— | Required. An OCID, not a name: this deploys into a compartment that already exists, because creating one needs tenancy-root rights the API user must not have |
deployment_name |
spinifex |
Prefixes every resource and becomes the VCN dns_label, so 1–15 lowercase alphanumerics starting with a letter |
node_count |
1 |
Nodes, 1–25. 1 for a single node, 3 for a cluster. Each gets its own volume, its own second VNIC and its own fault domain |
compute_shape |
VM.Standard.E6.Flex |
BM.* for bare metal, VM.*.Flex for a VM. A fixed bare-metal shape carries its own size, so shape_config is emitted only when the name contains Flex — set a BM shape and the two sizing variables below are ignored rather than rejected |
compute_ocpus |
8 |
OCPUs on a flex shape. Validated at ≥ 8 (16 vCPU), the minimum recommended for a node |
compute_memory_in_gbs |
32 |
GiB on a flex shape. Validated at ≥ 32 |
data_volume_size_in_gbs |
256 |
Size of each node's data volume. This is where guest disks, S3 objects and JetStream live — size it for the guests, not for the control plane |
data_volume_vpus_per_gb |
120 |
Performance tier, 0–120 VPUs/GB. 120 is Ultra High Performance; 10 is Balanced. Higher is more IOPS and more cost |
data_mountpoint |
/var/lib/spinifex |
Where cloud-init mounts it. Change only if you know why |
vcn_cidr |
10.200.0.0/22 |
VCN address space. A /22 leaves room for the secondary private IPs Spinifex registers per node |
public_subnet_cidr |
10.200.0.0/23 |
The subnet every node sits in, on both VNICs. Nodes serve the UI, the S3 gate and the AWS gateway directly, so they cannot live in a private subnet |
node_client_cidr_allow_list |
["0.0.0.0/0"] |
Who may reach SSH and the service ports from the internet. Change this |
node_service_ports |
[3000, 8443, 9999] |
Console, predastore gate, AWS gateway. SSH is opened separately |
ubuntu_version |
26.04 |
Canonical platform image version. The newest image for the shape and region is resolved at plan time rather than pinned to a regional OCID |
wan_bridge_name |
br-wan |
The Linux bridge built over the second VNIC |
wan_bridge_mtu |
9000 |
The OCI VCN carries 9000 |
Troubleshooting
Addresses and quotas
Spinifex enforces no limit of its own. It attempts the allocation and reports what OCI says, so the ceiling you hit is always your tenancy's, never a number baked into the software. A quota is raised with Oracle, not reconfigured here, and there is nothing to restart afterwards.
InsufficientAddressCapacity on allocate-address is what exhaustion looks like. It is the AWS error code for "out of addresses" and Spinifex returns it for either quota, mapped from OCI's own refusal. On a cluster the regional one is the likelier of the two.
A detached reserved public IP still bills. disassociate-address returns the address to AVAILABLE without deleting it, exactly as an unassociated AWS Elastic IP behaves, and it keeps consuming your regional quota as well as your bill. release-address is what frees both.
A stopped instance costs nothing for its auto-assigned address. Stopping an instance returns its auto-assigned address to the pool and deletes the reserved public IP object behind it, as AWS does; the start that follows creates a new one. An Elastic IP is yours and survives the stop untouched — that is the whole distinction between the two, and on OCI it is also the difference between a stopped fleet that bills for addresses and one that does not.
Datapath
Instance launches but the public address is unreachable. In order:
sudo ovn-nbctl lr-nat-list <router>— is there adnat_and_snatrow, and does it hold the private address?ip route get <private-addr>— does it route via the transit veth?sudo iptables -S FORWARD | grep spinifex-eip-ingress— both directions present?oci network private-ip list --vnic-id <vnic>— is the private address actually registered with OCI, and on this node's VNIC? If not, nothing else matters.
Egress works from one address and not another on the same NIC. OCI enforces the source address per VNIC, so this means the address is not a registered private-IP object on it. It is never a routing problem on the host.
An address went dark after a guest moved node. The affinity pass should have claimed it within 15 seconds. Check the new node's daemon journal for ocinet affinity, then confirm with step 4 above which VNIC OCI thinks holds the private half.
A node that was fine and then was not
Terraform's cloud-init does the host jobs on each node — the iSCSI volume, br-wan and its source routing, the firewall. When one of them did not take, the symptom arrives much later and rarely looks like a networking or storage problem. This one command answers all of it:
ssh -i ~/.ssh/oci-spx ubuntu@<node-public-ip> '
cloud-init status # done
sudo iscsiadm -m session # a session to 169.254.2.2:3260
findmnt /var/lib/spinifex # mounted by UUID, with _netdev
ip -br addr show br-wan # UP, second VNIC address as a /32
ip -br addr show enp1s0 # the bridge MEMBER holds NO address
ip route show default # still on the PRIMARY interface
ls -l /dev/kvm # present
'
Three answers are worth knowing in advance, because each looks wrong and is not:
br-wanholds a/32and the default route is on the other interface. That is how two VNICs share one subnet without the second stealing the first one's traffic, and it is what satisfies OCI's per-VNIC source check.br-wan's address appears inip ruleinstead of in the default route.br-wan's address is not one you chose. It is the second VNIC's own private IP, assigned by OCI, and not predictable before the apply.enp1s0carrying an address is the one real fault here. The kernel derives an on-link route from it that beats the primary VNIC's, so every packet to another node leaves the wrong VNIC and OCI drops it. The node still forms a cluster and still reportsReady— formation uses outbound connections — and it surfaces later aspredastore … the stripe is short one holderon the first AMI import.sudo /usr/local/sbin/spinifex-setup-wan-bridgerepairs it in place.
Host
RunInstances returns ServerInternal. Check the daemon journal for the real error — journalctl -u spinifex-daemon --since -10m. An OCI allocation failure surfaces this way.
The stack ends up on the boot disk and the node fills. The data volume was mounted without _netdev and lost the race with iscsid. findmnt /var/lib/spinifex is the check; ls cannot tell the difference.
Guests cannot reach instance metadata
Guests boot but cloud-init hangs on metadata, or the host stops answering its own. OCI uses 169.254.169.254 for its instance metadata, and so does every AWS-compatible guest — the host and the guests want the same address.
Spinifex resolves this by moving the host side of the per-guest metadata link to a different link-local pair, so guests keep asking 169.254.169.254 exactly as they do on AWS. The deploy sets imds_host_meta_ip and imds_host_dns_ip for you. Set both or neither — a half-configured pair is ignored and the node behaves as if the remap were off, which is what this symptom looks like.
Check it after your first guest launches:
ip route get 169.254.169.254 # your real NIC, not lo
curl -sH 'Authorization: Bearer Oracle' -o /dev/null -w '%{http_code}\n' \
http://169.254.169.254/opc/v2/instance/ # 200
ip -br addr show | grep ime- # holds 169.254.42.x, not .169.x
ip route get 169.254.169.254 returning dev lo is the signature of a missing remap. The same signature can appear after a guest terminates, from a leftover ime- endpoint still holding the address; with the remap configured it is harmless, and removing the stale OVS port clears it.
Deploying without Terraform
Terraform automates every step of a normal install, and that is the point of it. The oci-spx configuration builds the VCN, the subnets, the gateways, the block volumes and both VNICs per node, then renders the cloud-init that installs Spinifex and sets up OVN — so the sequence below happens without anyone typing it, in the same order every time, on one node or three.
If you need to install by hand — an existing OCI tenancy you cannot run Terraform against, an unsupported shape, or a stage you are debugging — the install itself is not OCI-specific. Follow the standard guides:
- Installing Spinifex for a single node.
- Multi-Node Installation for a cluster, which covers
spx admin initon the leader andspx admin joinon the rest.
Four things about OCI are not in those guides, and each has its own section above:
- The external pool is
source = "oci", which needs the instance principal or API key from Credentials and the IAM policy. Without it a node forms, passes every health check, and then refuses every launch that wants a public address. br-wanstays a Linux bridge owned by netplan, linked to OVSbr-extby a veth pair. Passsetup-ovn.sh --wan-bridge=br-wanand it detects this itself; the NIC never becomes an OVS port, so the VNIC keeps its MAC identity. See How it fits together.- The block volume must be mounted with
_netdev, because it arrives over iSCSI and the mount is otherwise attempted before a session exists. - IMDS needs the 169.254 remap described under Guests cannot reach instance metadata, since the guest's metadata address collides with OCI's own.