* docs(oci): offer both credential routes, and default the guide to the release Step 2 was one route only, the scoped API key, with the instance principal mentioned in a TIP at the end of it. It is now "2. OCI Credentials for Spinifex" with the two routes as 2.1 and 2.2, a table to choose between them, and the instance principal first because it leaves no key on any node. The guide also assumed the dev branch throughout. `git switch dev` and `--channel dev` are now an optional note for installing a pre-release, and the documented path is the published release. * test(oci): add a vm-single-principal topology, and cover the credential branch Instance-principal authentication was broken for a release while the OCI nightly stayed green, because the run's two axes never crossed: every topology that built the address allocator authenticated with an API key, and every run that authenticated as the instance was dispatched with no external pool, so the allocator was never constructed and the credential never used. vm-single-principal closes that. It is vm-single with the credential fixed by the topology rather than by a separate switch, so the name alone decides it, and pairing it with --no-external-pool is refused rather than run: without the allocator there is nothing for it to exercise. It joins the nightly's default list, and naming it alone is the way to test this one thing in a quarter of the full run. A unit test covers the same branch without a tenancy. newClient now reaches its two constructors through package variables a test can swap, which is the only way to observe the choice off an OCI instance - both real constructors need either an instance identity or a key file on disk. * fix(oci): let the tfvars choose the credential, and stop the guide hardcoding paths The deploy driver passed instance_principal as a -var, which outranks a tfvars file, so a deployment that set it there had it silently replaced with off and the nodes went looking for a key. Same defect class as the shape and node count, and the same fix: the flag is a choice about one run and still outranks everything, and without it the deployment's own tfvars decides. The driver now reads instance_principal from terraform.tfvars and *.auto.tfvars for its pre-apply gates, taking the last assignment as Terraform does, and checks that guess against a new Terraform output once the apply has run. They have to agree, because the credential handoff is skipped on the strength of it and a wrong answer leaves a node with no credential at all. The guide now carries one export rather than two more steps. SPX_PRINCIPAL is set in step 2.1, expanded into the tfvars step 4 already writes, and read back from there in step 5, so an instance-principal deployment runs the same command as a key-based one. Getting into the node needed an address copied out of the log. Every topology now writes its hosts file, not only the multi-node one that hands it to install-node.sh, and the run prints the ssh and destroy commands when --keep leaves the infrastructure up. The guide exports TOPOLOGY once, so the paths read the same for one VM, three, or bare metal. * fix(oci): print the ssh key path as ~ rather than expanded The --keep line read ssh -i /home/you/.ssh/oci-spx, which is the invoking user's home and differs on every machine. Collapsed to ~ when the key is under $HOME, left alone when it is not.
title, seoTitle, description, category, tags, resources
| title | seoTitle | description | category | tags | resources | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Oracle Cloud Infrastructure (OCI) | Run Spinifex on Oracle Cloud Infrastructure — Spinifex Docs | Deploy Spinifex on Oracle Cloud Infrastructure end to end with Terraform, on one node or three, and give guests real public addresses through OCI's own API. | Cloud Install |
|
|
Running Spinifex on Oracle Cloud Infrastructure
Run the AWS surface (EC2, EBS, S3, VPC) inside your own OCI tenancy, with guests that get real, publicly reachable addresses through OCI's API.
Table of Contents
- Overview
- Prerequisites
- 1. Clone the Spinifex repository
- 2. OCI Credentials for Spinifex
- 3. Choose deployment method
- 4. Write your Terraform inputs
- 5. Deploy
- 6. Check it worked
- 7. Set up your cluster
- Harden it before production
- Next steps
- Removing the deployment
- Troubleshooting
Overview
Spinifex is an open-source infrastructure platform that brings core AWS services to bare-metal, edge, and on-prem environments. It serves the EC2, EBS, S3, VPC, IAM and STS APIs over a SigV4 endpoint of your own, so the AWS CLI, the AWS SDKs and the AWS Terraform provider all work against it unmodified. Nothing a tenant has written has to be rewritten to run on it.
This guide installs Spinifex on Oracle Cloud Infrastructure, on one instance or three, with guests that get real public addresses from OCI's own pool. The install is the same product you would put on your own servers, so Single-Node Install and Multi-Node Install describe the same formation, storage and networking. What differs here is the layer underneath, because a VCN is not an Ethernet segment and public addresses come from an API rather than a range you choose.
Why run it on OCI:
- Cost. OCI prices compute and block storage below the large US clouds, and one Spinifex node serves a tenant's whole fleet as QEMU guests on hardware you are billed for once.
- Egress. Oracle's outbound-transfer allowance and per-GB rate are far cheaper than the majors, and guest-to-guest traffic rides the private plane, where it is not billed as internet egress at all.
- Portability. Your tenants code against Spinifex, not against OCI, so the same workloads move to your own rack or an edge site later as a deployment decision rather than a rewrite.
- Real public addresses. Spinifex registers each guest's address with OCI, so a guest is reachable on the internet rather than hidden behind a shared NAT.
What gets installed, on every node:
- Spinifex daemon and CLI
- QEMU/KVM for guests
- OVN and Open vSwitch for VPC networking
- Predastore, S3-compatible object storage
- Viperblock, EBS-compatible block storage
- The OCI public-address allocator, which is the one component specific to this platform
Nodes run Ubuntu 26.04, resolved at plan time as the newest Canonical platform image for your shape and region. Oracle Linux support as the host OS to run Spinifex is under active development.
Terraform does the whole deployment: the VCN, the block volumes, both VNICs per node, the Spinifex install, the cluster formation and the allocator. Six steps below, one command, about ten minutes for a single node.
What sits underneath is in Architecture and Operations, which also carries the IAM policy, the quotas, every variable and the troubleshooting.
Prerequisites
| An OCI compartment OCID | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have |
OCI credentials in ~/.oci/config |
For Terraform, to build the infrastructure. oci setup config writes one, or write it by hand |
| A runtime credential for the nodes | Separate from the above, and the subject of step 2: either an instance principal or a scoped API key. Without one a node comes up healthy and still cannot give any guest a public address |
Terraform or OpenTofu, git, Python 3 |
On your workstation. The Python helper is standard library only, so there is nothing to pip install |
Check two quotas before you start, because both refuse at apply time rather than at plan time. Reserved public IPs are capped at 50 per region across the whole tenancy, and the compute limit for your shape starts at zero for bare metal on many new tenancies. Quotas has the commands.
For the workstation (Linux, MacOS, Windows [WSL]) that will be used to deploy Spinifex on OCI the following prerequisites are required.
OCI CLI tool
Ensure the oci CLI tool is pre-installed. Follow the install guide at: https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm
Once installed configure your OCI credentials (~/.oci/config) which will be used in later steps in the installation tutorial.
oci setup config
Once setup validate your OCI credentials work as expected. $CID is the compartment you are deploying into, which is a ocid1.compartment.oc1.. OCID, or your tenancy OCID if that compartment is the root. The same value is used throughout this guide and again as compartment_ocid in step 4.
export CID="ocid1.compartment.oc1..xxx"
oci compute shape list --compartment-id $CID
Expected output to confirm API key works as expected:
{
"data": [
{
"baseline-ocpu-utilizations": null,
"billing-type": "PAID",
...
}
]
}
Install Terraform / OpenTofu
The next prerequisite is to install Terraform or OpenTofu to deploy the Spinifex stack on OCI.
- Terraform - https://developer.hashicorp.com/terraform/install
- OpenTofu - https://opentofu.org/docs/intro/install/
Instructions
1. Clone the Spinifex repository
To begin the deployment of Spinifex on OCI deploy the repository:
git clone https://github.com/mulgadc/spinifex.git
cd spinifex/scripts/terraform/oci-spx
Note
To deploy the development branch instead of the release, add
git switch devbefore thecd, and pass--channel devin step 5. The two choose different things: the branch chooses the Terraform and the deploy driver, and--channelchooses which Spinifex build lands on the nodes. Use the release for anything you intend to keep.
Then make a new SSH keypair the deploy installs on every Spinifex node which you can login with:
ssh-keygen -t ed25519 -f ~/.ssh/oci-spx -N '' -C spinifex-oci
Terraform reads your OCI credential from ~/.oci/config which is a prerequisites step above, taking the tenancy, the user, the fingerprint, the key and the region from the profile. A config holding a single [DEFAULT] profile, which is what oci setup config writes, needs nothing further.
Optionally with more than one profile you can define one using the environment variable OCI_CLI_PROFILE
export OCI_CLI_PROFILE=customprofile
2. OCI Credentials for Spinifex
Spinifex calls the OCI API at runtime to register each guest's public address, so every node needs a credential that can manage private and public IPs in your compartment. Without one a node forms and passes every health check, then refuses every launch that asks for a public address.
There are two ways to provide it, and they differ in who has to authorise them rather than in what Spinifex is allowed to do. Both grant the same three permissions.
| 2.1 Instance principal | 2.2 OCI IAM user | |
|---|---|---|
| Key material | None. Each node authenticates as itself | A private key on every node, at /etc/spinifex/oci/ |
| Who can set up | A tenancy admin, once per tenancy | Anyone with identity rights in the tenancy |
| Rotation | Nothing to rotate | Yours to rotate and redistribute |
| What you set | export SPX_PRINCIPAL=adopt, read by step 4 |
Nothing extra; step 5 installs the key for you |
Use 2.1 if you are a tenancy admin. It is the better of the two because there is no key to leak, rotate or forget about, and a replaced node needs no handoff. Use 2.2 when your compartment was allocated to you inside someone else's tenancy, which is the common case and the reason 2.2 exists at all.
Do one of the two, not both, then continue to step 3.
2.1 Instance principal (preferred)
Each node authenticates with the certificate its own metadata service serves at /opc/v2/identity/cert.pem, so no key is written anywhere. Authorising that takes a dynamic group and a policy, and both are tenancy-root resources — a compartment-scoped user cannot create them, and the attempt fails with 404-NotAuthorizedOrNotFound on CreateDynamicGroup, which reads like a misconfiguration and is not one.
Create them once, with a tenancy-admin OCI profile, then set one variable:
./setup-identity.sh --dry-run # always first
./setup-identity.sh
export SPX_PRINCIPAL=adopt
That is all of it. Step 4 writes SPX_PRINCIPAL into your Terraform inputs and step 5 reads it back from there, so neither step needs anything extra from you. Keep the same shell for both, as with $CID.
The dynamic group matches on the compartment, so it covers every node you build there and needs no update when a node is replaced. Its Terraform state goes in .identity/, separate from the deployment's — keep that directory, as it is the only record of what was created.
Only setup-identity.sh needs tenancy-admin rights. Every deployment afterwards runs as your ordinary compartment user.
To remove the group and policy again, ./setup-identity.sh --destroy. Every node configured for adopt loses its credential the moment you do.
2.2 OCI IAM user with scoped permissions
This route creates a dedicated OCI user holding only the three permissions Spinifex needs, and installs its key on each node. It needs identity rights in the tenancy but no tenancy-root rights, so it works in an allocated compartment.
Generate the key
openssl genrsa -out ~/.oci/oci_api_key_spx.pem 2048
chmod 600 ~/.oci/oci_api_key_spx.pem
openssl rsa -pubout -in ~/.oci/oci_api_key_spx.pem -out ~/.oci/oci_api_key_spx_public.pem
openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key_spx.pem | openssl md5 -c
The last command prints the fingerprint, which is how OCI names a key everywhere it refers to one. Keep it; you need it in the config file at the end of this step.
Create a user and a group for it
Both of these are identity resources, so they are created in the tenancy rather than in your compartment. This step requires the prerequisite oci CLI tool to be installed and configured with an API scope that includes creating new user accounts and groups.
Replace admin@yourdomain.com with your administrator email which will be linked to the new OCI user account.
export ADMIN_EMAIL=admin@yourdomain.com
USER_OCID=$(oci iam user create --name spinifex-allocator \
--email $ADMIN_EMAIL \
--description "Spinifex external-address allocator" \
--query 'data.id' --raw-output)
GROUP_OCID=$(oci iam group create --name SpinifexOperators \
--description "May manage the private and public IPs behind Spinifex external addresses" \
--query 'data.id' --raw-output)
oci iam group add-user --group-id "$GROUP_OCID" --user-id "$USER_OCID"
On success the output will include:
{
"data": {
"compartment-id": "ocid1.tenancy.oc1..xxx",
"group-id": "ocid1.group.oc1..xxx",
"id": "ocid1.groupmembership.oc1..axxx",
"inactive-status": null,
"lifecycle-state": "ACTIVE",
"time-created": "2026-10-05T19:39:48.478000+00:00",
"user-id": "ocid1.user.oc1..xxx"
}
}
An email will be sent to the designated $ADMIN_EMAIL defined, you are required to click the "Activate Your Account" link to fully activate the new account in OCI.
Note
If these fail with
NotAuthorizedOrNotFound, you do not have identity rights in the tenancy, which is normal when someone allocated you a compartment inside theirs. Ask whoever owns the tenancy to grant you the required permissions.
Grant the group its three permissions
Spinifex calls ten operations, and these three statements are what authorise them. Attach the policy to the compartment you are deploying into using the command below. Replace $CID with that compartment's OCID.
export CID="ocid1.compartment.oc1..xxx"
oci iam policy create --compartment-id "$CID" --name SpinifexOperators \
--description "Spinifex external-address allocation" \
--statements '[
"Allow group SpinifexOperators to use vnics in compartment id '"$CID"'",
"Allow group SpinifexOperators to manage private-ips in compartment id '"$CID"'",
"Allow group SpinifexOperators to manage public-ips in compartment id '"$CID"'"
]'
Note
If this fails with
NotAuthorizedOrNotFound, confirm$CIDis the compartment you are deploying into. It is aocid1.compartment.oc1..OCID, or your tenancy OCID if you deploy into the root compartment. An identity-domain OCID is a different resource and will not work here.
Upload the public key to that user
oci iam user api-key upload --user-id "$USER_OCID" \
--key-file ~/.oci/oci_api_key_spx_public.pem
This will output:
{
"data": {
"fingerprint": "xxx",
"inactive-status": null,
"key-id": "ocid1.tenancy.oc1..xxx",
"key-value": "-----BEGIN PUBLIC KEY-----xxxx-----END PUBLIC KEY-----",
"lifecycle-state": "ACTIVE",
"time-created": "2026-10-05T19:51:58.843000+00:00",
"user-id": "ocid1.user.oc1..axxx"
},
"etag": "xxx"
}
Write the profile and prove it works
The Spinifex nodes authenticate with an OCI SDK config under the profile name spinifex. Write that profile on your workstation first, because it is both what the nodes get and the only way to test the credential before a deployment depends on it:
# ~/.oci/config
[spinifex]
user=<the USER_OCID from above>
fingerprint=<the fingerprint from above>
tenancy=<your tenancy OCID, the same one your DEFAULT profile uses>
region=<your region, e.g. ap-sydney-1>
key_file=~/.oci/oci_api_key_spx.pem
Then confirm the key, the user, the group and the policy all line up. This call needs exactly the public-ips grant above, so it fails in the same way a node would:
oci network public-ip list --compartment-id "$CID" --scope REGION --all --profile spinifex
An empty response is a pass, because the permission is what is being tested rather than the contents. NotAuthorizedOrNotFound means the policy is not in force: check that the statements name the compartment by compartment id, and that the user is actually in the group.
Do this before step 5. confirm the new OCI account can correctly authenticate and issue the required API calls for Spinifex to function correctly once installed.
Warning
Do not set
OCI_CLI_PROFILE=spinifex. That variable chooses the profile Terraform builds the infrastructure with, and this one cannot: creating a VCN, a subnet and instances is far outside the three grants above, so the apply fails on the first resource. The nodes find thespinifexprofile by name without being told.
Validate deployment
Prior to running the Terraform scripts to deploy Spinifex, verify the OCI configuration and environment is correctly setup:
./spx-oci-config.sh --dry-run ~/.ssh/oci-spx
Expected output:
[spx-oci-config] credential: the [spinifex] profile in ~/.oci/config
[spx-oci-config] user: ocid1.user.oc1..aaaa...
[spx-oci-config] fingerprint: a1:b2:...
[spx-oci-config] region: ap-sydney-1
[spx-oci-config] would write /etc/spinifex/oci/config and /etc/spinifex/oci/oci_api_key.pem as 0640 root:spinifex on: (no hosts given)
Step 5 runs this script for real, over SSH, once the nodes are formed. Those two files are what the allocator authenticates with, and the spinifex profile name in the second one is what it looks for.
The credential: line is the one to check. It has to name the [spinifex] profile. If instead it prints two WARNING: lines about falling back to "the credential Terraform builds with", the profile was not found and your nodes would get your full-access credential rather than the scoped one you just made.
3. Choose deployment method
Choose how to deploy Spinifex on OCI, either as a single node, or at minimum a three node cluster for added resiliency and redundancy.
| Single node | Three nodes | |
|---|---|---|
node_count |
1 |
3 |
| Survives losing a node | No | Yes. NATS keeps quorum, predastore RS(2,1) keeps the data readable, the OVN raft keeps a leader |
| Public addresses available | 64 | 192, three VNICs' worth, still bounded by the regional quota |
| Fault domains | One | Three, one per node, chosen by Terraform |
| Use it for | Evaluation, a lab, an edge site with one box | Production. Multi-node resiliency |
4. Write your Terraform inputs
One file, in the directory you are already in. Only compartment_ocid is required, because every other variable has a default that builds a working single node:
Recommended compute shapes for a VM deployment:
VM.Standard.E6.Flex- Higher performance, 5th Gen AMD EPYC processorVM.Standard.E5.Flex- Base-line performance, 4th Gen AMD EPYC processor
Recommended bare metal shapes, preferred for large-scale production:
BM.Standard.E5.192- 192 OCPU / 2304 GB RAM. Higher performance, 4th Gen AMD EPYC processorBM.Standard3.64- 64 OCPU / 1024 GB RAM. Baseline, Intel Xeon Platinum 8358BM.Standard.E2.64- 64 OCPU / 512 GB RAM. Legacy, 1st Gen AMD EPYC processor
cat > terraform.auto.tfvars <<EOF
compartment_ocid = "$CID"
deployment_name = "spinifex"
# How the nodes authenticate to OCI, from step 2. "off" is an API key, which is
# what step 2.2 installs; "adopt" is the instance principal from step 2.1.
instance_principal = "${SPX_PRINCIPAL:-off}"
# Shape and size.
# Bare metal is preferred for large scale production usage.
compute_shape = "VM.Standard.E5.Flex"
compute_ocpus = 8 # OCI counts an OCPU as a full core, 1 OCPU = 2vCPU
compute_memory_in_gbs = 32
# 1 for a single node, 3 required for a cluster.
node_count = 1
# The block volume behind /var/lib/spinifex, attached over iSCSI.
data_volume_size_in_gbs = 256
data_volume_vpus_per_gb = 120 # 120 = Ultra High Performance
# Firewall. A list, so it can hold several ranges. Read the note below before
# narrowing it.
node_client_cidr_allow_list = ["0.0.0.0/0"]
EOF
The heredoc delimiter is unquoted, which is what lets $CID expand from the variable you exported in step 2. Check the file rather than assume, because a literal $CID in there fails at plan time with an unhelpful message about an invalid OCID:
grep compartment_ocid terraform.auto.tfvars
This file is what decides the deployment. compute_ocpus and compute_memory_in_gbs size a VM flex shape and are ignored on a fixed bare-metal one, which comes at one fixed size with all available OCPU/memory.
Every Terraform variable has the full list.
Note
Leave
node_client_cidr_allow_listat its0.0.0.0/0default for now. It is an OCI security-list rule covering the whole public subnet, and your guests' public addresses cross it too, so narrowing it to your own address cuts internet access to every guest rather than just to the node. Harden it before production locks the node down in the layer that can tell the two apart.
5. Deploy
export TOPOLOGY=vm-single
./validate-topology.sh \
--topology "$TOPOLOGY" \
--credential-hook ./spx-oci-config.sh \
--skip-workload \
--keep
The same command covers both credential routes. It reads instance_principal from the tfvars you wrote in step 4, so an instance-principal deployment needs no extra argument and --credential-hook is simply ignored.
One command builds the infrastructure, installs Spinifex, forms the cluster, configures OCI public addressing and verifies that the address allocator came up.
| Argument | What to pass |
|---|---|
--topology |
vm-single for one VM, vm-multi for three, bm for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission. It supplies a shape and a node count only where your tfvars is silent |
--channel |
Nothing, normally. It defaults to latest, the published release. Pass dev to install the newest pre-release instead, or --version <tag> to pin an exact build |
--credential-hook |
./spx-oci-config.sh, which installs the step 2.2 credential on every node. Not used when your tfvars set instance_principal, because there is then no key to install |
--skip-workload |
Stop once the cluster is verified. Drop it and the driver also runs the published Terraform workbooks on the node, which launches real guests on public addresses and tears each one down again |
--keep |
This is what makes it a deployment rather than a test. Without it the driver destroys everything at the end, which is right for CI and wrong here |
Your terraform.auto.tfvars outranks the topology, so --topology bm with compute_shape = "BM.Standard.E5.192" in that file deploys on that shape rather than the one CI runs. The run logs the shape and node count it actually built, straight after the host addresses.
Add --dry-run to print the plan and change nothing.
Note
If
--channel devfails to download, pass a tag instead.devis the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited and then 404s. Take the newest tag from the releases page and swap--channel devfor--version <tag>, which resolves by redirect and cannot trip the same limit. A tag is also what to use for a repeatable deployment, where you want the build pinned rather than current.
6. Check it worked
The stages print as they finish, and all of these have to appear:
[validate-vm-single] building
[validate-vm-single] VM.Standard.E5.Flex, 1 node(s), oci_auth from a key file
[validate-vm-single] waiting for cloud-init on each node
[validate-vm-single] installing Spinifex on each node
[validate-vm-single] initializing the single node
[validate-vm-single] running the credential hook
[validate-vm-single] credential hook ok
[validate-vm-single] configuring the IMDS remap and the external pool
[validate-vm-single] checking the allocator came up
[validate-vm-single] 150.230.13.131 allocator ready
[validate-vm-single] cluster membership
[validate-vm-single] --skip-workload: stopping before the workbook
[validate-vm-single] --keep: leaving the infrastructure up, no verdict recorded
[validate-vm-single] ssh in with: ssh -i ~/.ssh/oci-spx ubuntu@150.230.13.131
[validate-vm-single] destroy it with: ./validate-topology.sh --topology vm-single --destroy-only
allocator ready is the line to look for. Every line above it also prints on a node that cannot allocate a public address, and that failure stays silent until a guest asks for one.
running the credential hook is spx-oci-config.sh, and its own output is captured rather than printed. If the deploy stops there, read credential-hook.log in the state directory the driver names at startup — its credential: line says which credential went onto the nodes. On the instance-principal route those two lines are absent, because there is no handoff to make.
Access the Spinifex node
The deploy's last line is the command to get in, so you can paste it straight back. The addresses are also on disk, one per line, which is what to read from a script:
cat ".validate-$TOPOLOGY/hosts"
ssh -i ~/.ssh/oci-spx ubuntu@"$(head -1 ".validate-$TOPOLOGY/hosts")"
$TOPOLOGY is the variable you exported in step 5, so this is the same command whether you deployed one VM, three, or bare metal. The first line is the node the cluster formed on.
Then call the AWS surface with the [spinifex] profile that spx admin init wrote into ~/.aws/credentials:
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
{
"InstanceTypes": [
{
"InstanceType": "c7a.12xlarge",
"CurrentGeneration": true,
...
}
]
}
If this returns a list of available instance types, your installation is working.
Congratulations! Spinifex is installed.
7. Set up your cluster
Spinifex is running, but it holds nothing yet — no machine images, no networks, no instances.
Continue to Setting Up Your Cluster to import an AMI, create an SSH key pair, create a VPC with a public subnet, and launch your first instance.
Important
Probe a guest's public address from your workstation, not from the node. A node is a poor vantage point for its own network's public addresses, so a timeout measured there says nothing about the guest. Private addresses are a separate question, and those do work from inside.
Harden it before production
The deployment comes up usable, not locked down. Three layers filter traffic here and the first job is telling them apart, which Firewalls, in all three layers does in full. The short version:
1. Arm the Spinifex host firewall. It is installed but not armed on this path. Arming it gives you the proper policy: the public plane open, the cluster plane scoped to peers only, and SSH scoped to a list you own.
# On every node.
sudo env SETUP_STAGES=firewall /usr/local/share/spinifex/setup.sh --firewall on
2. Narrow SSH. It is open to the world until you do, which comes from the Ubuntu image's own rules rather than from ours. Edit the one file the installer creates and never overwrites:
# /etc/spinifex/firewall/custom.nft
define trusted_ssh_peers = { 203.0.113.10/32 }
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft # check, then apply
sudo /usr/local/lib/spinifex/spinifex-firewall-apply
3. Decide about the console on port 3000. The S3 gate (8443) and the AWS gateway (9999) are the product's public surface and should stay public. The web console need not be.
Leave the OCI security list open. Guest public addresses cross it, so narrowing node_client_cidr_allow_list breaks your tenants instead of protecting the node. The host firewall above is the layer that can tell a node's listener from a guest's.
Full detail, including which port serves what and why forming a cluster starts by turning the policy off, is in Host Firewall.
Next steps
Set up your cluster. It is running but holds nothing yet: no images, no networks, no instances. Setting Up Your Cluster imports an AMI, creates a key pair and a VPC, and launches your first instance.
Run your existing Terraform against it. Everything above builds infrastructure on OCI with the OCI provider. From here you use the AWS provider, unmodified from the registry, pointed at your own cluster. Your aws_vpc, aws_instance, aws_db_instance, aws_ecs_service and aws_eks_cluster resources stay as they are in git. Three things differ, and only three:
provider "aws" {
region = "ap-southeast-2"
access_key = var.access_key
secret_key = var.secret_key
endpoints {
ec2 = var.spinifex_endpoint # https://<node private IP>:9999
s3 = var.predastore_endpoint # https://<node private IP>:8443
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
}
s3_use_path_style = true
skip_metadata_api_check = true
skip_region_validation = true
}
Use the node's private address, not its public one. The node certificate carries no SAN for the public address, because that address is never on the wire: OCI NATs it to a private one. A call to
https://<public IP>:9999fails TLS verification with hostname doesn't match. Run Terraform from a node, or from anything else inside the VCN.
The Terraform workbooks we ship are the worked examples. e2e-cloudvendor-nightly runs them on a single VM, on three VMs and on bare metal, publishing a table per topology on its own run page, so read that for the build you are installing.
Oracle Linux guests. Four Oracle Linux images are in the catalog, so you can run the distro your Oracle support contract covers:
sudo spx admin images import --name oracle-10.1-x86_64 --config /etc/spinifex/spinifex.toml
The four are oracle-10.1-x86_64, oracle-10.1-arm64, oracle-9.8-x86_64 and oracle-9.8-arm64. All of them boot UEFI, so launch them into a shape that boots UEFI or the failure looks like a hung boot rather than a rejected image. Log in as cloud-user, not opc. These are the generic KVM cloud images from yum.oracle.com, which refuse opc, oracle and ec2-user alike. The UI's instance detail page shows the right user per AMI.
Removing the deployment
./validate-topology.sh --topology vm-single --destroy-only
It destroys whatever that topology's Terraform state holds, so it can only remove what this configuration created.
Troubleshooting
The deploy stops at the first failure and names the log it wrote, all under .validate-<topology>/. Add --keep-on-fail to leave a failed deployment up so you can log in and look. It keeps billing until you run --destroy-only.
Troubleshooting covers the symptoms worth knowing in advance, several of which point at the wrong layer on first reading:
- If a deploy stage fails, and which log holds the answer
- A guest's public address is unreachable
- A node that was fine and then was not, where one command answers all of it
- Guests cannot reach instance metadata, because OCI and AWS guests want the same address
- Addresses and quotas, including why a detached address still bills
Installing by hand instead of with Terraform is Deploying without Terraform.