Ben Duncan
46c0c44cc2
* feat: run every Terraform workbook on OCI, and split the guide in two The OCI topology driver ran the shared driver's default five workbooks, which left ECS and all three EKS variants untested on every platform even though their assertions already existed. It now passes an explicit nine and imports the three appliance images they need — RDS, ECS and EKS each boot their own, found by a spinifex:managed-by tag and never by name. demo-app stays out because it is not a workbook: it has no .tf at all, being the container image the EKS workbooks' nested workloads modules deploy, so listing it would fail on a missing root module rather than test anything. The guide is now two documents. oci-integration/README.md is the end-user path — six steps, deploy first, and it closes the three prerequisites nothing documented before: the clone, the ref, and the SSH keypair the driver refuses to start without. architecture.md holds the sizing, the IAM policy, every variable, the three filtering layers and the troubleshooting. Its firewall advice was also backwards. node_client_cidr_allow_list is an OCI security-list rule over the whole public subnet, and a guest's public address is a secondary private IP on the node's own VNIC, so narrowing it to the operator's address cuts internet ingress to every guest rather than protecting the node. The hardening section now arms the nft policy and narrows SSH through custom.nft, which is the layer that can tell a node's listener from a guest's. The deploy step names --channel dev rather than a pinned tag. The dev tag string grows with every cut, so a pinned example goes stale within the week. * fix(oci): the guide could not work on a fresh tenancy, because the profile name was ours oci_env.py bridges ~/.oci/config into TF_VAR_* — the provider takes the tenancy, user, fingerprint, key and region as Terraform variables and reads no config file itself — but the profile it looked for was the reference tenancy's, and validate-topology.sh passed no way to change it. A fresh account's config holds [DEFAULT], so every deploy would have stopped at "OCI profile 'apacanzset03child03' was not found" before building anything. The profile is now resolved as --oci-profile, then $OCI_CLI_PROFILE, then the reference tenancy, then DEFAULT. A name passed explicitly and absent is an error listing the profiles present, never a fallback: the failure worth preventing is building a cluster in whichever tenancy happened to be first. The reference tenancy still outranks DEFAULT, so existing runs are unchanged, and CI is untouched because it takes the environment route rather than a profile. has_profile exists because configparser treats DEFAULT as the fallback for every other section rather than as a section, so has_section denies it exists even when it is the only thing in the file — which made the first version of this fallback do nothing for exactly the case it was written for. Three documentation corrections found while checking the guide against the code. The IAM policy is ten operations and needs "use vnics", not "read vnics": registering a secondary private IP is an operation on a VNIC, so OCI checks VNIC_ASSIGN, which lives in the use verb, and with read the node forms and looks healthy and every allocation fails NotAuthorizedOrNotFound. These are now the same three statements instance-principal.tf grants, which is the only version of the policy that has been exercised. The guide also promised a worked example of the credential hook that does not exist and should not: nothing here distributes a credential. It now states the contract instead — the argument shape, the two files, and that a non-zero exit stops the deploy before anything is built. And the dev channel's rate limit is documented with the tag as its remedy, rather than left in the driver's help where a reader will not see it. * fix(ci): a failing workbook's log never reached the artifact, so only the runner had it run-tofu-examples-e2e.sh writes its assertion output to the per-topology workbooks.log in the state directory. "Sweep anything left behind" deletes those directories and runs before "Upload the run", so the upload's three oci-state/.validate-* patterns matched nothing and said so to nobody. The first nine-workbook run lost the one thing it was dispatched to find: ecs-quickstart failed on vm-single and the reason existed only on the runner, which does not resolve from a workstation. A collect step now copies *.log, *.txt and *.tsv into the artifact directory before the sweep, prefixed by topology so two topologies cannot overwrite each other. The workflow already records that "Publish the result tables" sits ahead of the sweep for this reason; the logs were simply never given the same treatment. The dead patterns are removed rather than left. A path that cannot match is worse than no path, because it reads as protection that is present — which applies to the credential-hook exclusion too: it now lives in the collect step, which is the only thing that writes into the artifact tree, so the file cannot reach an artifact rather than being filtered out of one. * fix(ci): a state directory kept the last run's logs, and the artifact published them as this run's Collecting the per-topology logs surfaced the next defect immediately: a vm-single-only run produced an artifact holding bm-workbooks.log and nine vm-multi node logs, because those state directories still held the previous run's output. Anyone reading that artifact would conclude three topologies ran when one did, which is worse than the missing log it replaced. Two causes, one each side. validate-topology.sh never cleared its own logs, so a topology's directory accumulated them; it now removes *.log, nodes.txt and cloudinit-*.txt as it starts, before the first line of the new run, and never the tfstate, which is how the sweep finds hosts to destroy. The globs are named rather than a find, so what goes is readable at a glance, and :? stops the shell instead of expanding to /*.log if STATE_DIR is ever unset. That alone does not cover a topology the run skipped, whose directory is never entered. So the collect step now takes only the topologies this run drove, read back from the per-topology log the harness writes rather than from the dispatch input -- a run that names no topology gets the harness's default, and reading that here would be a second copy of the list to disagree with the first. Also proved by this: ecs-quickstart passes on a freshly built vm-single, 172s, where it failed at 293s as the seventh of nine. The failure is order or load dependent on one 32 GB node, not a property of the workbook. * fix(ci): a failed workbook run left no node journal, so the cause was unreadable The workbook log says which API call failed; only the journal says why, and the nodes are destroyed minutes later. An IAM teardown failure on the nightly came down to one swallowed warning that existed nowhere else, and by the time the artifact was read the host it was logged on had been torn down. Warnings and above first, since that is where a swallowed error surfaces, then a bounded tail for context. Unbounded, a multi-hour suite's journal would dwarf the rest of the artifact. The existing state-directory clear keeps these from going stale between runs, and the collect step already globs *.log.
title, seoTitle, description, category, tags, resources
| title | seoTitle | description | category | tags | resources | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Spinifex on Oracle Cloud | Run Spinifex on Oracle Cloud Infrastructure — Spinifex Docs | Deploy Spinifex on OCI end to end with Terraform, single node or three, and give guests real public addresses through OCI's own API. | Install |
|
|
Running Spinifex on Oracle Cloud Infrastructure
Run the AWS surface — EC2, EBS, S3, VPC — inside your own OCI tenancy, with guests that get real, publicly reachable addresses through OCI's API.
Spinifex brings core AWS services to hardware you control: a node runs the Spinifex daemon, QEMU/KVM for guests, OVN for VPC networking, Predastore for S3 and Viperblock for EBS, all behind a SigV4 endpoint the ordinary AWS SDKs and CLI talk to unmodified.
Terraform does the whole deployment — the VCN, the block volumes, both VNICs per node, the install, the cluster formation and the OCI public-address allocator. Six steps below, one command, about ten minutes for a single node.
Everything underneath — the architecture, the IAM policy, the quotas, every variable, the troubleshooting — is in Architecture and Operations.
Table of Contents
- What you need
- 1. Get the configuration
- 2. Give the nodes an OCI credential
- 3. Choose one node or three
- 4. Write your Terraform inputs
- 5. Deploy
- 6. Check it worked
- Harden it before production
- Next steps
- Removing the deployment
- If something goes wrong
What you need
| An OCI compartment OCID | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have |
OCI credentials in ~/.oci/config |
For Terraform, to build the infrastructure. oci setup config writes one, or write it by hand |
| An OCI API key for the nodes | Separate from the above, and step 2. Without it a node forms, looks healthy, and cannot give any guest a public address |
Terraform or OpenTofu, git, Python 3 |
On your workstation. The Python helper is standard library only, so there is nothing to pip install |
Check two quotas before you start, because both refuse at apply time rather than at plan time: reserved public IPs (50 per region, tenancy-wide) and the compute limit for your shape, which on a new tenancy is often zero for bare metal. Quotas has the commands.
1. Get the configuration
git clone https://github.com/mulgadc/spinifex.git
cd spinifex
git switch dev
cd scripts/terraform/oci-spx
Important
git switch devis needed today. OCI support is not in the current release, so the deploy driver is not onmainyet. From the next release onwardmaincarries it and you can skip that line. This chooses the Terraform and the driver only — which Spinifex build lands on the nodes is step 5.
Then make the keypair the deploy installs on every node and logs in with. It refuses to start without one, rather than building instances you cannot reach:
ssh-keygen -t ed25519 -f ~/.ssh/oci-spx -N '' -C spinifex-oci
Terraform reads your OCI credential from ~/.oci/config, taking the tenancy, the user, the fingerprint, the key and the region from the profile — so the region you deploy into is the profile's, not something you set in step 4. A config holding a single [DEFAULT] profile, which is what oci setup config writes, needs nothing further. With more than one profile, name the one you mean:
./validate-topology.sh --oci-profile mycorp ... # or export OCI_CLI_PROFILE=mycorp
A name you pass that the config does not hold is an error listing the profiles it does hold. It never falls back, because the failure mode worth preventing is building a cluster in whichever tenancy happened to be first.
2. Give the nodes an OCI credential
Spinifex calls the OCI API at runtime to register each guest's public address. Create an API key for that, if you do not already have one:
openssl genrsa -out ~/.oci/oci_api_key.pem 2048
chmod 600 ~/.oci/oci_api_key.pem
openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem
openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 -c # the fingerprint
Upload the public key under Identity → Users → API Keys, and grant it the ten operations Spinifex uses in your compartment — no more.
The deploy installs the credential on each node by calling a small script of yours, so no key material ever goes into user-data or Terraform state, where it would be readable from instance metadata for the life of the instance. Write one that puts these two files on each host it is given:
| Path | Mode | Contents |
|---|---|---|
/etc/spinifex/oci/oci_api_key.pem |
0640 root:spinifex |
The private key |
/etc/spinifex/oci/config |
0640 root:spinifex |
An OCI SDK config, profile [spinifex], naming the key by path |
The profile name must be spinifex. It is called as your-hook <ssh-key> <host>... — the first argument is the private key to reach the nodes with, the rest are the hosts — and it must exit non-zero if any host failed, so a missing credential stops the deploy rather than surfacing later as a launch that cannot get an address. Credentials has the config file's exact contents.
chmod +x it. The deploy checks that before it builds anything, so a hook that is missing or not executable costs you a few seconds rather than forty minutes and a bare-metal bill.
Nothing in this repository ships that script, deliberately. A credential belongs to whoever owns it, so the hook is yours to write and yours to keep.
Tip
If you are a tenancy admin you can skip key files entirely and authenticate the nodes as the instance itself. It needs a dynamic group and a policy created once at the tenancy root — see instance principal. The API key path above is the one that works in any tenancy, including a compartment someone allocated to you.
3. Choose one node or three
This is the only architectural decision, and it is one line in step 4.
| Single node | Three nodes | |
|---|---|---|
node_count |
1 |
3 |
| Survives losing a node | No | Yes — NATS keeps quorum, predastore RS(2,1) keeps the data readable, the OVN raft keeps a leader |
| Public addresses available | 64 | 192, three VNICs' worth, still bounded by the regional quota |
| Fault domains | One | Three, one per node, chosen by Terraform |
| Use it for | Evaluation, a lab, an edge site with one box | Anything you would be unhappy to lose |
Two is not worth taking: it doubles the cost of a single node and gives you a cluster that cannot form a quorum. Sizing covers shapes; bare metal is the recommendation, and VM.Standard.E6.Flex is the minimum.
The pool has no configured size — Spinifex asks OCI for an address when a guest needs one. Two ceilings bind it and neither is ours: 64 secondary private IPs per VNIC, which is the 64 above and is not raisable, and the regional reserved-public-IP quota, which is tenancy-wide and shared with everything else you run on OCI. The second is the one you will hit first.
4. Write your Terraform inputs
One file, in the directory you are already in. Only compartment_ocid is required — every other variable has a default that builds a working single node:
cat > terraform.auto.tfvars <<'EOF'
compartment_ocid = "ocid1.compartment.oc1..aaaa..."
deployment_name = "spinifex"
# Shape and size. Bare metal is preferred; a flex VM is what the quota allows.
compute_shape = "VM.Standard.E6.Flex"
compute_ocpus = 8 # OCI counts an OCPU as a full core
compute_memory_in_gbs = 32
# 1 for a single node, 3 for a cluster. This is the only line that chooses.
node_count = 1
# The block volume behind /var/lib/spinifex, attached over iSCSI.
data_volume_size_in_gbs = 256
data_volume_vpus_per_gb = 120 # 120 = Ultra High Performance
EOF
Every Terraform variable has the full list.
Note
Leave
node_client_cidr_allow_listat its0.0.0.0/0default for now. It is an OCI security-list rule covering the whole public subnet, and your guests' public addresses cross it too — narrowing it to your own address cuts internet access to every guest, not just to the node. Harden it before production is where you lock the node down, in the layer that can tell the two apart.
5. Deploy
./validate-topology.sh \
--topology vm-single \
--channel dev \
--credential-hook ~/.spinifex/oci-credential-hook.sh \
--keep
One command builds the infrastructure, installs Spinifex, forms the cluster, configures OCI public addressing, verifies it, then launches real guests on public addresses and tears those down again.
| Argument | What to pass |
|---|---|
--topology |
vm-single for one VM, vm-multi for three, bm for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission |
--channel |
dev until the next release ships, because OCI support is not in the current stable release yet. It resolves to the newest pre-release at the moment you run it. latest is the default and becomes the right answer once a release carries OCI support |
--credential-hook |
Your script from step 2. With an instance principal, replace it with --instance-principal |
--keep |
This is what makes it a deployment rather than a test. Without it the driver destroys everything at the end, which is right for CI and not what you want here |
Add --skip-workload to stop once the cluster is verified, without launching the validation guests. Add --dry-run to print the plan and change nothing.
Note
If the install step fails to download, pass a tag instead of the channel.
devis the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited and then 404s. Take the newest tag from the releases page and swap--channel devfor--version <tag>, which resolves by redirect and cannot trip the same limit. That is also what to use for a repeatable deployment, where you want the build pinned rather than current.
6. Check it worked
The stages print as they finish, and all of these have to appear:
[validate-vm-single] building
[validate-vm-single] waiting for cloud-init on each node
[validate-vm-single] installing Spinifex on each node
[validate-vm-single] initializing the single node
[validate-vm-single] running the credential hook
[validate-vm-single] configuring the IMDS remap and the external pool
[validate-vm-single] checking the allocator came up
[validate-vm-single] 150.230.13.131 allocator ready
[validate-vm-single] cluster membership
[validate-vm-single] --keep: leaving the infrastructure up, no verdict recorded
allocator ready is the line to look for. Every line above it also appears on a node that cannot allocate a public address, and that failure is otherwise silent until a guest asks for one.
Then prove it the way a customer would. On the node, the [spinifex] profile that spx admin init wrote into ~/.aws/credentials is the one to use:
export AWS_PROFILE=spinifex
aws ec2 allocate-address
aws ec2 run-instances --image-id <ami> --instance-type t3.micro --key-name <key> \
--subnet-id <subnet> --associate-public-ip-address
aws ec2 describe-instances --query \
'Reservations[].Instances[].[InstanceId,PublicIpAddress,State.Name]' --output text
Then reach the guest from your workstation, not from the node:
ping -c 3 <public-ip>
ssh -i <key> ubuntu@<public-ip> hostname
Important
Probe a public address from outside the VCN. A node is a bad vantage point for its own network's public addresses, so a timeout from a node is not evidence about the guest. Private addresses are the separate question and do work from inside.
Harden it before production
The deployment comes up usable, not locked down. Three layers filter traffic and they are easy to confuse — Firewalls, in all three layers explains which is which. The short version:
1. Arm the Spinifex host firewall. It is installed but not armed on this path. Arming it gives you the proper policy: the public plane open, the cluster plane scoped to peers only, and SSH scoped to a list you own.
# On every node.
sudo env SETUP_STAGES=firewall /usr/local/share/spinifex/setup.sh --firewall on
2. Narrow SSH. It is open to the world until you do — that comes from the Ubuntu image's own rules, not from ours. Edit the one file the installer creates and never overwrites:
# /etc/spinifex/firewall/custom.nft
define trusted_ssh_peers = { 203.0.113.10/32 }
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft # check, then apply
sudo /usr/local/lib/spinifex/spinifex-firewall-apply
3. Decide about the console on port 3000. The S3 gate (8443) and the AWS gateway (9999) are the product's public surface and should stay public. The web console need not be.
Leave the OCI security list open. Guest public addresses cross it, so narrowing node_client_cidr_allow_list breaks your tenants rather than protecting the node — the host firewall above is the layer that can tell a node's listener from a guest's.
Full detail, including which port serves what and why forming a cluster starts by turning the policy off, is in Host Firewall.
Next steps
Set up your cluster. It is running but holds nothing yet — no images, no networks, no instances. Setting Up Your Cluster imports an AMI, creates a key pair and a VPC, and launches your first instance.
Run your existing Terraform against it. Everything above builds infrastructure on OCI with the OCI provider. From here you use the AWS provider, unmodified from the registry, pointed at your own cluster — the same aws_vpc, aws_instance, aws_db_instance, aws_ecs_service and aws_eks_cluster resources a team already has in git. Three things differ, and only three:
provider "aws" {
region = "ap-southeast-2"
access_key = var.access_key
secret_key = var.secret_key
endpoints {
ec2 = var.spinifex_endpoint # https://<node private IP>:9999
s3 = var.predastore_endpoint # https://<node private IP>:8443
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
}
s3_use_path_style = true
skip_metadata_api_check = true
skip_region_validation = true
}
Use the node's private address, not its public one. The node certificate carries no SAN for the public address, because that address is never on the wire — OCI NATs it to a private one. A call to
https://<public IP>:9999fails TLS verification with hostname doesn't match. Run Terraform from a node, or from anything else inside the VCN.
The Terraform workbooks we ship are the worked examples, and e2e-cloudvendor-nightly runs them on a single VM, on three VMs and on bare metal, publishing a table per topology on its own run page. Read that for the build you are installing.
Oracle Linux guests. Four Oracle Linux images are in the catalog, so you can run the distro your Oracle support contract covers:
sudo spx admin images import --name oracle-10.1-x86_64 --config /etc/spinifex/spinifex.toml
oracle-10.1-x86_64, oracle-10.1-arm64, oracle-9.8-x86_64 and oracle-9.8-arm64. All four boot UEFI, so launch them into a shape that boots UEFI or the failure looks like a hung boot rather than a rejected image. Log in as cloud-user, not opc — these are the generic KVM cloud images from yum.oracle.com, and opc, oracle and ec2-user are all refused. The UI's instance detail page shows the right user per AMI.
Removing the deployment
./validate-topology.sh --topology vm-single --destroy-only
It destroys whatever that topology's Terraform state holds, so it can only remove what this configuration created.
If something goes wrong
The deploy stops at the first failure and names the log it wrote, all under .validate-<topology>/. Add --keep-on-fail to leave a failed deployment up so you can log in and look — it keeps billing until you run --destroy-only.
Troubleshooting covers the symptoms worth knowing in advance, and several of them look like a different fault than they are:
- If a deploy stage fails — which log holds the answer
- A guest's public address is unreachable
- A node that was fine and then was not — one command answers all of it
- Guests cannot reach instance metadata — OCI and AWS guests want the same address
- Addresses and quotas — including why a detached address still bills
Installing by hand instead of with Terraform is Deploying without Terraform.