Ben Duncan
021f439bab
* feat: recover instances automatically from a node that stopped heartbeating A node that dies hard takes its guests with it, and nothing brings them back until an operator brings that node back. Every daemon now runs the same small reconciler: it reads the instance records, selects desired-running instances whose owner is stale, and relaunches them here. The claim is a CAS on Status.LastNode and that is the whole of the coordination. Several survivors race one instance, exactly one revision wins, and the losers find a record naming a live owner and stop considering it. A claimant that dies mid-launch leaves the record naming a node that is itself stale, which is the same candidate the next pass derives again, so there is no lease to expire and nothing to reconcile by hand. Ownership is not permission to write. The volume leases still decide whether the guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt that loses the volume race has touched no tap, no OVS port and no OVN binding. A storage failure is not a host failure and wants the opposite response. A guest paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is refused: the volumes are the same objects in the same store from anywhere, so relaunching moves the pause rather than ending it. A node whose own object store is not answering stands down entirely, which separates a store broken here from a store broken everywhere without either having to be diagnosed. A launch that does not complete backs this node off that instance, doubling to a cap, because nothing on the mount path can yet say which of the two it hit. Recovery is off by default and inert below three nodes. It also waits until it has seen its peers live, or five minutes, so the first node back from a coordinated shutdown does not recover guests from nodes that are still booting. * test: prove instance recovery moves a host failure and refuses a storage failure The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves. Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live. harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down. * fix: make automatic instance recovery a cluster-wide setting Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file. Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it. * test: drive real I/O into the storage-fault case, and freeze the store rather than stop it Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted. The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them. The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three. * test: give instance recovery its own suite and its own nightly cell Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five. Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set. * fix: give a guest long enough to open a volume the object store is slow to serve QEMU opens every drive before its monitor answers a greeting, so the QMP greeting deadline is really a deadline on opening the guest's volumes. At 30s it was shorter than a cold viperblock open costs on a cluster that has lost a node, where every object read is served degraded: measured at 35s with one node of three away. The launch killed the guest five seconds short of ready and then did it again on every retry, which is a recovery that can never complete. A network-backed drive now gets its own deadline for the same reason VFIO already had one — synchronous work before the monitor answers — and the instancerecovery suite is what found it. Also make the suite's predastore thaw best effort per node. It runs after a node has been taken away, where there is no process to signal, and a node it cannot reach must not stop it restoring the rest. * docs: name the two newest cells in the dispatch input The cells input is the operator's index of what a number means, and it stopped at 29 while 30 and 31 were added. * test: prove the guest is writing before freezing the store in the recovery suite The storage-fault test froze the object store as soon as RunInstances reported running, which is when QEMU has launched rather than when the guest has booted. On the nightly cluster the freeze landed while the guest was still in UEFI, so the only drive it could reach was the EFI varstore — and pflash has no werror=stop to pause on, so the guest never paused and the test reported a product failure it had caused itself. The load driver now prints to the serial console after each pass it has written and synced, and the test waits for that before freezing. The console is read through GetConsoleOutput, so this still needs no login and still runs on cells whose guests have no public address. * feat: recover instances automatically with nothing to configure The switch is gone. An instance whose host has failed is down whether or not recovery is enabled, so the only outcome a disabled reconciler can produce is leaving it down — which is the state recovery exists to end, not a safer one to fall back to. Against that, the switch cost a cluster that can be configured into never recovering anything, with no symptom until a node dies. This feature already hit that once: a top-level table parsed into nothing and every node reported recovery off with the setting plainly in the file. What makes recovery inert is now only facts about the cluster — no cluster config, no JetStream, or fewer than three nodes — and each says in the log which one applied. The e2e no longer writes config or restarts daemons: it waits for every node to report the reconciler active on a cluster nothing configured, so reintroducing a switch fails there as well as in the unit tests. * feat: stop retrying a recovery that is never going to work Every failed attempt used to go to the same exponential backoff and stay there. An instance nothing could launch was retried every fifteen minutes forever, still observed-running on a node that is gone, with no account of it anywhere the customer or an operator would look. A failure is now classified and counted. Two classes are read from what the launch actually reports: the node it was taken from still holds a volume lease, which clears on that lease's clock and is retried on a faster one, and this node has no room, which another survivor may. There is no terminal class, because nothing on this path can prove one — the launch collapses everything else into ServerInternal, so a verdict of "never going to work" would be a guess, and the guess that abandons a recoverable instance is the expensive one. The budget bounds the unknown case instead. Every class counts against it, since a class that did not would be the old unbounded retry under a narrower condition. Exhausting it leaves the instance where AWS leaves one whose host failed and could not be brought back: stopped, carrying a StateReason that names the cause and the last error, one StartInstances from running again. Writing the desired state down is also what makes one node's decision cluster-wide, because selection reads desired state — the same property the claim already has. A record another node has since claimed is never touched, so a node exhausting its attempts cannot stop a guest that is now running elsewhere. * fix: give a guest the time for every cost it pays before its monitor answers A GPU instance booting from a network-backed root volume pays both costs: QEMU opens every drive across the object store and then pins the whole of guest RAM, one after the other, all before the monitor answers. The deadline took whichever was larger, so such a guest was given less time than it needs. That the two floors are both 180s is why nothing has been seen yet — the VFIO branch has always returned at least the network-drive deadline. It is the same defect either way, and the symptom when it lands is a GPU instance SIGKILLed on every attempt while its plain sibling launches, which reads as a GPU fault rather than as a deadline. The costs are now added, the plain QEMU startup is counted once rather than replaced, and the cap covers both floors together so it cannot clip a guest before either cost has been waited out. * test: assert the AWS-visible arc of a host failure end to end A customer cannot see which host an instance is on, and AWS never tells them. What they can see is DescribeInstanceStatus, so that is where a recovery has to be provable, and the suite never looked at it. Three frames are now required, in order: running with system-status ok before anything is broken, system-status impaired while the owning node is gone, and ok again once a survivor is running it. The middle frame is the one worth having — the window opens when the owner's heartbeat goes stale, which is strictly before any survivor may claim, and closes when the relaunch registers, so it is bounded rather than lucky. Reporting ok throughout would be the more comfortable lie. The availability zone is compared across the first and last frames as well. A recovery that crossed one would break every placement promise made to the customer, and nothing else in the suite would have noticed. * test: prove a half-healthy node fences itself and is never a second writer The two existing recovery cases take a node away by stopping its services, which is the failure this feature handles least interestingly: a node that is gone has no opinion about who owns its volumes. The one that matters is the node that is still up, still running QEMU, still answering its own local clients, and simply cannot reach consensus. That is the state in which two nodes can each believe they own a volume, and the only thing between that and a corrupted disk is that the partitioned node gives up first, on a clock it can evaluate without reaching anything. So this drops NATS on 4222 and 4248 between one node and its peers and nothing else, then asserts the ordering the safety argument rests on rather than the outcome it produces. A watch samples every node every three seconds for the whole run, including the heal, and fails on any single sample naming two hosts. Two guests writing one volume through one object store is data loss already under way, not a race that resolves, and no point-in-time assertion can see one that appeared between its own samples. The injector takes its own nft table rather than touching inet spinifex_filter, because a test that flushed the node's real firewall would leave it either unfirewalled or misfirewalled with no sign of which, and a separate table at priority -300 heals in one atomic delete. A transient systemd timer removes it independently, so a cancelled workflow does not leave a node cut off from its cluster until somebody notices. It proves SSH still works and proves a peer port is genuinely unreachable before anything is asserted, since a rule set that installed cleanly and dropped nothing would read as the system tolerating a partition. The suite budget goes to 90m and cell 31 to 105m to fit the third case. * fix: move an instance's boot state, volumes, logical port and address with it A record-driven launch assumes the node reading the record is the node that wrote it. On a daemon restart that holds. On a recovery onto a survivor it does not, and five things follow from it — one that stops a moved instance booting at all, and four that stop it running anywhere. A load balancer never comes back from a host failure. Its boot configuration is a set of files under the runtime directory and the record holds their paths, so the survivor restores paths that exist only on the node that died. The hook that rebuilds them was reachable from Restore alone, and the cross-node path goes straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert that are not there. The gateway URL baked into that env names the creating node's management bridge as well, which the rebuild corrects because it rebuilds from whichever node is running it. Firing the hook on both paths fixes both, and a rebuild that fails now refuses the launch: a guest that boots without its network configuration is worse than one that does not boot, because the records would say it recovered. The export outlives the guest and holds the volume lease. A returning node is healthy, so it renews that lease indefinitely and the new owner never acquires: the guest is stopped here and refused there, recovery spends its attempt budget on a volume that was never going to be free, and gives up on an instance that was never unrecoverable. So the export is given up too, over a new node-addressed request that tears it down exactly as the fence does. Deliberately not an unmount — an unmount seals, this node's block map is the stale one, and sealing publishes it over the winner's. Unlike the fence it deletes the lease key rather than leaving it to expire, because the lease is still this node's to hand back. The tap outlives the guest, and the tap is what claims the instance's OVN logical port. The binding names one chassis, so two nodes offering the same iface-id do not split the traffic — they contend, and the address answers from whichever won last. It also blocks the layer above: the external IP's host ingress is pruned on the evidence that the owning guest sits on another chassis, and that stays false while this node is still claiming the port. The management address outlives the guest as a cluster-wide reservation, and the release was unconditional. br-mgmt is one flat L2 segment, so the node an instance left was freeing an address still in use and offering it to the next instance anywhere. Only the node an entry names may release it now, with the takeover recorded on arrival so the node actually running the instance can still give the address back. An entry written before the node field existed names nobody and stays releasable, because there is no other holder to collide with. The fence path already cleaned up the tap, so the release fix repairs that path too. * test: prove a load balancer keeps its name, its address and its traffic when its host dies Takes away the node running an internet-facing ALB and asserts the four things a customer of one actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again without anybody being called, and the backends are healthy again from the recovered load balancer's point of view. The backends are placed around the load balancer rather than beside it. A load balancer lands wherever a node answers its queue group, so its node is discovered and the backends are relaunched until both are elsewhere; one on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve. DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve through the same server, so the claim under test is what this cluster answers for a name rather than what the machine running the tests was configured to forward to. A watch samples every surviving node every five seconds for the whole run, because an answer that went missing or pointed elsewhere for one reconcile pass is the outage, and no single sample reaches it. The returning node then has to give up what it no longer owns: no second copy of the guest, and no tap still claiming the load balancer's logical port, since OVN binds a port to one chassis and a second claimant contends for it rather than splitting the traffic. The last assertion is that the name still serves with that node back in the cluster. * docs: name the new cell in the nightly's cell list * test: stop the disk-load guest from filling the console it is judged by The load generator announced every write, once a second, for as long as the test ran. The console is a ring buffer this suite reads back, so an hour of that pushes out everything explaining a failure, including the kernel's own account of the I/O error the test exists to produce. Announce twenty times, then keep writing quietly. * docs: say what the OCI guide has been proven on, and what a host failure does The guide read as a support statement for OCI rather than as an account of one tested path, promised seven steps and then gave eight, and said nothing about what happens to a guest or its public address when the node under it fails. State the evidence, state that the nightly guards the datapath and not the cloud, and give the failure behaviour as a table with the reason codes a customer will actually see. Also say plainly what the volume lease is: one running guest and a bounded, logged window, not a write barrier the object store enforces. Someone sizing a workload against this should read that before they read the table. * docs: say why the mount classifier refuses to retry a lease another node holds It reads as an omission — a lease held elsewhere is the most transient refusal there is. It is not. This classifier drives the restore path, where the holder is a node that is running the guest, so retrying would relaunch against a live owner. Recovery wants the opposite and reads the refusal itself. * feat: claim a recovered guest's public address in line, and verify it arrived A recovered instance was reported running before anything moved its public address off the node that died, so for up to fifteen seconds the records said recovered and the address answered nothing. The recovery path now runs the affinity pass itself, which is idempotent and is the same pass the ticker runs. Two things the pass could not do before. It abandoned every remaining address on the first API failure, though a throttle or a denial is per-object and the guests behind the others are just as dark. And it trusted the move's own response, which is OCI agreeing to the request rather than OCI having done it, so an accepted move that never landed read as success. It now re-reads what OCI delivers here and reports the difference. An address that will not arrive makes the guest a system-status failure with the time it went dark, the same shape AWS uses for a host-level fault. The guest passes every check it can be asked about itself, which is exactly why reporting it healthy was the worst available answer. * feat: give a recovery attempt one trace across all three of its phases The volumes opening, QEMU starting and its monitor answering already trace themselves, but nothing tied them to the attempt that asked for them: three orphan traces per recovery, no way to ask which phase a slow one spent its time in, and nothing naming the node the guest came from. A span over the attempt makes them its children, and the reconciler's own log lines carry its trace ID because the handler stamps it from the context. A failed attempt records the fault and the attempt count on the span, so the budget running out is readable as a sequence rather than as twelve unrelated warnings. * fix: finish a fence once its writer finally exits A kill that did not take effect was logged, counted and dropped. The export stayed registered and the lease stayed held, which is correct while the writer lives and wrong forever after it dies: a lease a healthy node keeps renewing never expires, so the volume could not start anywhere. The failure is now re-reported every 30 s until it resolves, and the teardown completes as soon as the writer is confirmed gone. Only a delivered kill is waited on — a refused one says nothing about what became of the writer. * test: stop judging a detached stop or terminate on a transient state Both tests read the instance state immediately after the ack, but the transition runs on its own goroutine, so the read races it in both directions: too early sees running, too late sees the final state. The race gate caught the first of those on a loaded runner. * fix: replicate the volume lease and dirty buckets across the cluster Both were created with the default single replica, so both lived on one node. That node holds the answer to "who may write this volume" and the marker every mount reads, and losing it takes volume leasing and mounting down cluster-wide. A partitioned node was fenced correctly and then no survivor could relaunch its guest: every attempt failed on "read dirty mark: no responders available", for ten minutes, until the budget ran out. The replica count is now the cluster's node count, as the gateway's own buckets already do. Existing buckets are raised in place rather than requiring a wipe. The e2e diagnostics that should have said this are fixed too. The fence and the surrender are logged by the storage service, not the daemon, so the assertions read the wrong journal; and Eventuallyf evaluates its message at the call site, so a ten-minute wait was explained by a log that stopped before it began.
title, seoTitle, description, category, tags, resources
| title | seoTitle | description | category | tags | resources | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Spinifex on Oracle Cloud | Run Spinifex on Oracle Cloud Infrastructure — Spinifex Docs | Deploy Spinifex on OCI end to end with Terraform, single node or three, and give guests real public addresses through OCI's own API. | Install |
|
|
Running Spinifex on Oracle Cloud Infrastructure
Run the AWS surface — EC2, EBS, S3, VPC — inside your own OCI tenancy, with guests that get real, publicly reachable addresses through OCI's API.
Table of Contents
- Overview
- Support status
- Why run Spinifex on OCI
- What OCI changes
- How it fits together
- Prerequisites
- Deploying
- Step 1. Set your Terraform inputs
- Step 2. Build the infrastructure
- Step 3. Install Spinifex and set up OVN
- Step 4. Form the cluster
- Step 5. Configure Spinifex for OCI, start and verify
- Step 6. Set up your cluster
- Step 7. Run your existing Terraform against the cluster
- Step 8. Oracle Linux as a guest image
- Verify end to end
- Troubleshooting
- Limits in this version
Overview
Spinifex is an open-source infrastructure platform that brings core AWS services to bare-metal, edge and on-prem environments. A node runs the Spinifex daemon and CLI, QEMU/KVM for guests, OVN/Open vSwitch for VPC networking, Predastore for S3-compatible object storage and Viperblock for EBS-compatible block storage — and serves them all behind a SigV4 endpoint that the ordinary AWS SDKs and CLI talk to unmodified.
This guide runs that same stack on Oracle Cloud Infrastructure. Everything in Single-Node Install and Multi-Node Install still applies: one node is a working install, three is the minimum for a cluster that can lose a node, and formation, storage and OVN behave exactly as they do on hardware you own. What changes is the layer underneath — an OCI VCN is not an Ethernet segment, so the external datapath is wired differently, and public addresses come from OCI's API rather than from a range you write in a config file.
Read this guide end to end and you have a working cluster. Deploying is one sequence of eight steps, from an empty compartment to a node that can launch an instance with a publicly reachable address, and then to the Oracle Linux guest image to put on it. It builds the infrastructure with Terraform, then installs and forms Spinifex the same way Single-Node Install and Multi-Node Install do — those steps are inlined here, so you do not need to read three documents at once. The only choice to make is one node or three, and it is one line in Step 1.
Every command and every output below was run against a live OCI tenancy, on both paths.
Support status
What is written here was proven on one shape, in one region, on one guest OS. Three nodes of VM.Standard.E6.Flex in ap-sydney-1, Ubuntu hosts, routed NAT, OCI reserved public IPs over secondary private IPs, with a single-node pass of the same guide beside it. Nothing about the integration is region-specific or shape-specific by design, but nothing else has been run, so treat another region, a bare-metal shape or a different host image as untested rather than unsupported — and record the shape, region, fault domains, quotas, kernel and spx version when you reproduce it, because that is the only way a later difference can be attributed.
OCI is not in the nightly test matrix. The routed-NAT datapath this integration depends on is guarded nightly, on three nodes, on our own hardware — the behaviour is covered, the cloud underneath it is not. So an OCI-specific regression would be found by the next person to run this guide rather than by CI. That is a real gap and it is tracked; it is not a reason to avoid the integration, but it is a reason not to read "proven" as "continuously proven".
What happens when a node fails
A three-node cluster recovers a guest from a node that has stopped heartbeating, onto a survivor, with no operator action. There is no setting for it: an instance whose host is gone is down either way, and the only thing a switch could buy is leaving it down. It is inert below three nodes, because a single node has nowhere to move a guest to.
The cluster is deliberately slow to believe a node has gone. A daemon takes no recovery action until it has seen every peer live or five minutes have passed, so a coordinated restart does not look like three simultaneous host failures; a node has to be seen stale across two passes fifteen seconds apart; and the guest's volume lease has to lapse before anything else may open it. Expect a recovery to take minutes, not seconds.
| Condition | What the cluster does | What you see |
|---|---|---|
| A node stops heartbeating and a survivor can reach its own object store | A survivor claims the instance and relaunches it | The same instance ID, the same volume and the same private address on a new host. DescribeInstanceStatus reports the move, as it would on EC2 |
| The volume is still leased by the node that had it | Retries every 15–90 seconds until the lease lapses | Recovery takes about a lease TTL longer. The guest never runs in two places |
| A survivor's own object store is not answering | That node stands down and lets the others try | If the object store is broken everywhere, nothing moves and nothing claims to have moved. An instance that no node could have run stays where it is |
| No survivor has capacity | Each survivor tries in its own order, with backoff | After the attempt budget the instance is stopped carrying Server.InsufficientInstanceCapacity, one StartInstances from running |
| Anything else — a torn volume, an OCI API denial, a launch that will not finish | Retries with doubling backoff, bounded at twelve attempts, roughly an hour and a half | The instance is stopped carrying Server.HostRecoveryFailed and the last error. It is never left looping with nothing visible to you |
| A planned whole-cluster restart | The settle window suppresses recovery while nodes return | Guests are not migrated out from under a deploy |
The public address follows the guest, and it keeps its identity while it does. An OCI public address is a reserved public IP object that OCI 1:1-NATs onto a secondary private IP on a VNIC, and the guest only ever sees the private half. Recovery moves the private half between VNICs in the same subnet — one OCI call, server-side, keeping the OCID — so the reserved public IP is never released, never reallocated and never renumbered. The customer's address is the same address before and after. What does change is which node OCI delivers it to, and each node claims its own guests' addresses on a fifteen-second pass, so there is a window after the guest is running in which it is not yet reachable on its public address. Private VPC addresses have no such window: they are node-independent.
What this is not. A fenced guest is stopped locally, and the volume lease is what stops it — it is not a write barrier the storage backend enforces. A node that loses contact with the cluster gives its volume up on a clock it can evaluate without reaching anything, which bounds the exposure to one renewal interval rather than to the lease TTL, and a survivor does not open the volume until the lease has lapsed. But a write the losing node had already issued is not refused by the object store, and the failure to stop the local writer is reported rather than corrected. So the guarantee is "one running guest, and a bounded, logged window" — not "a second writer is impossible". If your workload cannot tolerate that, it wants a replicated datastore rather than a recovered instance.
Why run Spinifex on OCI
Lower cost for the same shape of workload. OCI's compute and block storage are priced well below the large US clouds for equivalent OCPU and IOPS, and Spinifex turns one large instance into a whole EC2/EBS/S3 surface — so a tenant's twenty small instances are twenty QEMU guests on hardware you are billed for once, not twenty billable cloud instances.
Egress stops being the line item that decides the architecture. Oracle publishes a large monthly outbound-transfer allowance per tenancy and a per-GB rate beyond it that is roughly an order of magnitude below the majors — check the current pricing page rather than a number in a document, but the gap is the reason this integration exists. Traffic between guests never leaves the host or, on a cluster, never leaves the VCN: it is Geneve over the private plane, which is not billed as internet egress at all.
Portability — the workload stops being AWS-shaped and starts being yours. The API your tenants code against is Spinifex's, so the same Terraform, the same SDK calls and the same AMIs run on OCI today, on bare metal in your own rack tomorrow, and at an edge site after that. Moving off AWS becomes a deployment decision rather than a rewrite, and moving again costs nothing extra.
You keep the parts of OCI that are worth keeping. Bare-metal shapes, fault domains, block volumes with a chosen performance tier, and real public addresses out of OCI's own pool. Spinifex does not hide the cloud underneath it — it registers each guest's address with OCI so that guest is genuinely reachable on the internet, not behind a shared NAT.
What OCI changes
Important
Read this section before you provision anything. OCI enforces addressing inside its virtual network in a way that makes one of Spinifex's three external modes unusable, and the choice is baked in at
spx admin init. Getting it wrong produces a cluster that looks entirely healthy and drops every guest packet.
Three properties of an OCI virtual network shape everything below.
OCI enforces the source IP per VNIC. A packet leaving the instance with a source address that is not a registered private-IP object on that VNIC is dropped by the hypervisor. Measured, not inferred: from the same NIC with the same MAC, ping -I <registered> 8.8.8.8 gets 0% loss and ping -I <unregistered> 8.8.8.8 gets 100%. So an external address pool cannot be a range you write in a config file — every address has to be registered with OCI before it works, and registering it is exactly what this integration does.
Pinging the VCN router (
10.200.0.1) proves nothing. It answers from any source address at ~0.08 ms because the hypervisor replies locally. Always test egress past the subnet.
A VNIC accepts one MAC; it does not learn MACs. Every inbound frame arrives addressed to the VNIC's own MAC, whatever address the packet is for. Spinifex's default external datapath advertises each address by ARPing from a per-VPC router MAC onto a shared L2 segment, which needs a segment that learns MACs. OCI has none. Enforcement sits in the hypervisor on VM shapes and in the SmartNIC on bare metal, so there is no guest-side workaround.
A public IP is a second object, and it never appears on the wire. An OCI public address is a RESERVED public IP that OCI 1:1-NATs upstream onto a secondary private IP on the VNIC. The instance only ever sees the private half. Spinifex handles this internally — AWS-facing APIs report the public address, the datapath uses the private one — but it explains why ip addr on the host never shows a customer's public IP, and why that is correct rather than a fault.
The consequence: routed mode is mandatory
| External mode | What goes on the uplink | On OCI |
|---|---|---|
pool (UplinkModePhysical, UplinkModeVeth) |
Per-VPC gateway MAC plus a per-rule external MAC, ARPed onto the segment | Does not work. Egress is dropped (wrong source MAC); ingress never reaches OVN (the frame is addressed to the VNIC MAC, which no router port owns). |
nat (UplinkModeRouted) |
Nothing. The host forwards over an RFC 6598 transit veth and sets no external MAC | Works. One MAC identity, and registered source addresses. |
Spinifex refuses source = "oci" on a pool unless the node is in nat mode, and the refusal names this reason. See VPC Networking → External modes for what each mode does off OCI.
Sizing
Bare metal is the recommendation. BM.Standard.E2.64 or similar gives you the hardware directly — no nested virtualisation, full NIC performance, and no hypervisor between your guests and the wire.
| Minimum | Recommended | |
|---|---|---|
| Shape | VM.Standard.E6.Flex |
Bare metal, e.g. BM.Standard.E2.64 |
| OCPU | 8 (16 vCPU) | 32+ |
| RAM | 32 GB | 128 GB+ |
| Data volume | 256 GiB block volume | 1 TB+ NVMe-backed |
| VNICs | 2 | 2 |
| Nodes | 1 | 3 — see Cluster sizing |
Verify nested virtualisation before anything else on a VM shape. Oracle's nested-KVM guidance names VM.Standard3.Flex (Intel) and VM.Standard.E5.Flex (AMD), and states Ampere shapes do not support it. VM.Standard.E6.Flex is not in that list but was measured working on 2026-09-24 (EPYC 9J45, /dev/kvm present, svm on all threads, kvm_amd nested=1). Treat an unlisted shape as unproven until you have checked:
# If /dev/kvm is absent, stop — change shape, do not continue.
ls -l /dev/kvm
grep -oE 'vmx|svm' /proc/cpuinfo | sort -u
cat /sys/module/kvm_amd/parameters/nested 2>/dev/null \
|| cat /sys/module/kvm_intel/parameters/nested
OCI has no Debian image. Ubuntu LTS is the tested path — Ubuntu 26.04 LTS on kernel 7.0.0-1009-oracle is what this integration was built and proven against. Anything in the tooling that assumes Debian is a known gap, not a supported configuration.
How it fits together
Single node — the whole AWS surface inside one OCI instance
One bare-metal instance runs the control plane and every tenant guest. Customers reach the AWS APIs on the node's own address; their instances reach the internet through addresses registered on the node's second VNIC.
Two things in that picture are the whole integration. The guest's private address (10.0.1.4) never leaves the host — OVN and the host route both hold it. The address OCI delivers to is a secondary private IP registered on VNIC 1, appearing on br-wan, and the customer's public address (150.230.13.131) is a reserved public IP that OCI 1:1-NATs onto it upstream. describe-instances reports the public one; every datapath object holds the private one, which is why the two never appear together on the host.
The third is storage. Every guest disk, every S3 object and the JetStream state all live under /var/lib/spinifex, which is an OCI block volume attached over iSCSI, not the boot disk. Size it for the guests you intend to run; a boot volume that quietly ends up carrying the stack is the most common way an install on OCI fails later rather than immediately.
Three nodes — the same surface, distributed
Each node registers the addresses for its own guests on its own second VNIC, and the allocator runs per node, so nothing has to know another node's OCIDs. When a guest moves — a stop/start that lands elsewhere, or a host failure — the node it lands on claims the address's private half from the old VNIC through a single OCI call. The OCID and the public half are untouched, so the customer's address never changes.
The overlay, the object shards and the OVN databases all cross VNIC 0, inside the VCN. Size that plane, not the external one: guest-to-guest traffic between nodes is Geneve over the VCN, and it is the link every distributed layer shares.
Note
The Spinifex region is your own naming and has nothing to do with the OCI region.
ap-southeast-2inspinifex.tomlbesideap-sydney-1in OCI is correct and expected. Do not read one as evidence about the other.
Prerequisites
Both paths need the same three things: an OCI compartment you can write to, an API key for Spinifex to allocate addresses with, and enough quota to hand out the addresses you plan to use.
Credentials — v1 uses an operator-provisioned API key
Instance principal is the better design and is not what v1 uses. The instance certificate is served at /opc/v2/identity/cert.pem and the Go SDK can authenticate as the instance with no key material on disk — but it still needs a dynamic group and a policy authorising it, and on the reference tenancy it authenticated without being authorised for anything. v1 requires an operator-provisioned API key, deliberately, to keep the setup a single documented path.
# On your workstation, not the node.
openssl genrsa -out ~/.oci/oci_api_key.pem 2048
chmod 600 ~/.oci/oci_api_key.pem
openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem
openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem \
| openssl md5 -c # this is the fingerprint
Upload the public key to the user under Identity → Users → API Keys, and record the user OCID, the fingerprint and the tenancy OCID.
The IAM policy
Spinifex calls exactly nine operations. Grant no more than these:
| Operation | Why |
|---|---|
CreatePrivateIp |
Register the secondary private IP the datapath carries |
DeletePrivateIp |
Release it |
GetPrivateIp |
Poll for AVAILABLE after an asynchronous assign |
ListPrivateIps |
The startup reconcile, and the affinity pass that moves an address between nodes |
UpdatePrivateIp |
Reassign a private IP to this node's VNIC when a guest moves here |
CreatePublicIp |
Create the RESERVED public IP |
DeletePublicIp |
Release it |
GetPublicIp / GetPublicIpByPrivateIpId |
Poll lifecycle state, and reconcile a private IP back to its public half |
UpdatePublicIp |
Attach and detach the public IP from a private IP |
The matching policy, scoped to the one compartment Spinifex allocates in:
Allow group SpinifexOperators to manage private-ips in compartment <compartment-name>
Allow group SpinifexOperators to manage public-ips in compartment <compartment-name>
Allow group SpinifexOperators to read vnics in compartment <compartment-name>
Allow group SpinifexOperators to read subnets in compartment <compartment-name>
Allow group SpinifexOperators to read vcns in compartment <compartment-name>
Scope it to a compartment, not the tenancy. manage public-ips at tenancy level lets a compromised node consume the whole regional reserved-public-IP quota.
Do not grant manage instances or manage vnics. Spinifex never creates, attaches or deletes a VNIC — it only adds addresses to a VNIC that already exists. A grant that allows VNIC lifecycle is strictly more than the integration can use.
Quotas to check before you size a deployment
Two limits bound how many public addresses a deployment can hand out, and they do not scale the same way:
| Quota | Default | Scope | Notes |
|---|---|---|---|
| Reserved public IPs | 50 | Per region, whole tenancy | Shared by every node, so it does not grow with node count. This is the wall you hit first. |
| Secondary private IPs per VNIC | 64 | Per VNIC, so per node | Not raisable. Three nodes gives you 192 — this is the limit that does scale. |
A raise on the first is a support request, not a setting: Console → Governance & Administration → Limits, Quotas and Usage → "Request a service limit increase". Cite the exact limit name from oci limits definition list, not a name from documentation.
oci limits definition list --service-name vcn --compartment-id <tenancy-ocid>
oci limits value list --service-name vcn --compartment-id <tenancy-ocid>
oci limits resource-availability get --service-name vcn \
--limit-name <name-from-above> --compartment-id <tenancy-ocid>
These calls need a grant the node does not have, and must not be given. Limits and usage are tenancy-level, which cannot be compartment-scoped the way the policy above insists the node's is. Create a second, read-only principal for an operator or a reporting job and keep it off every node:
Allow group SpinifexReporting to read limits in tenancy
Allow group SpinifexReporting to read usage-reports in tenancy
The second verb is what Cost Analysis and oci usage-api usage-summary request-summarized-usages read. Adding either to SpinifexOperators would hand every node tenancy-wide read, which is exactly what the node policy refuses.
The oci CLI
The oci CLI is not on the Ubuntu image, and apt has no python3-oci-cli package. Install it into a virtualenv on your workstation, and on the node if you want to inspect allocations there:
sudo apt-get update && sudo apt-get install -y python3-venv
python3 -m venv ~/.oci-cli-venv
~/.oci-cli-venv/bin/pip install --upgrade pip oci-cli
sudo ln -sf ~/.oci-cli-venv/bin/oci /usr/local/bin/oci
oci --version
What the CLI is and is not for. Spinifex's daemon uses the OCI Go SDK for every allocation — the CLI is not on that path. You need it for bootstrap and for operational inspection. Do not build automation that shells out to it during an allocation; allocations run inside a request budget that a forked Python process will not respect.
Deploying
Terraform builds the infrastructure; the standard Spinifex installer builds the node. That split is deliberate — formation on OCI is the same path prod and bare metal take, and diverging it for one cloud would mean two formation paths to keep correct.
There is one sequence of eight steps below, and the only thing that differs between a single node and a cluster is node_count in Step 1 and which variant of Steps 4 and 5 you follow. Decide now:
| Single node | Three nodes | |
|---|---|---|
node_count |
1 |
3 |
| Survives losing a node | No | Yes — NATS keeps quorum, predastore RS(2,1) keeps the data readable, the OVN raft keeps a leader |
| Public addresses available | 64 — the per-VNIC secondary private IP limit | 192, three VNICs' worth, still bounded by the regional reserved-public-IP quota |
| Fault domains | One | Three, one per node, chosen by Terraform |
| Use it for | Evaluation, a lab, an edge site with one box | Anything you would be unhappy to lose |
Two is not an option worth taking: it doubles the cost of a single node and gives you a cluster that cannot form a quorum. See Cluster sizing.
The configuration lives in scripts/terraform/oci-spx, which has the resource-by-resource architecture.
What Terraform builds, per node
| Resource | Detail |
|---|---|
oci_core_instance |
One per node_count, round-robined across fault domains so three nodes land on three sets of racks rather than wherever placement puts them |
| Primary VNIC | Public IP, hostname_label, the host plane — SSH, the advertise address, the Geneve endpoint |
oci_core_vnic_attachment |
A second VNIC in the same subnet. Every Spinifex external address becomes a secondary private IP here at runtime |
oci_core_volume |
A block volume per node, data_volume_size_in_gbs at data_volume_vpus_per_gb |
oci_core_volume_attachment |
attachment_type = "iscsi", with the Oracle Cloud Agent's Block Volume Management plugin enabled so the node logs the iSCSI session in itself |
| cloud-init | Partitions, formats, mounts and fstabs the volume; builds br-wan over the second VNIC; opens the service ports |
The data volume is iSCSI, and that is the detail that bites. OCI presents the volume as an iSCSI target reachable at 169.254.2.2:3260 — attaching it in the API does not put a block device on the host. is_agent_auto_iscsi_login_enabled makes the Oracle Cloud Agent do the login, and because it does that asynchronously, the device is not there when cloud-init first runs. The mount script waits up to ten minutes for it, formats it only if blkid reports no filesystem (so a re-run cannot erase a populated volume), and writes an fstab entry by UUID:
UUID=<uuid> /var/lib/spinifex ext4 defaults,_netdev,nofail 0 2
_netdev is what orders the mount after the network, and therefore after iscsid has a session. nofail is what keeps a missing volume from wedging the boot. Mounting /var/lib/spinifex itself rather than somewhere else and symlinking is deliberate: viperblock, predastore and JetStream all land on the volume with no configuration pointing anywhere unusual.
Step 1. Set your Terraform inputs
Prerequisites: Terraform, Python 3, OCI credentials in ~/.oci/config, and an SSH public key. The Python helper builds and uses the configuration's own .venv.
cd scripts/terraform/oci-spx
cat > terraform.auto.tfvars <<'EOF'
compartment_ocid = "ocid1.compartment.oc1..aaaa..."
deployment_name = "spinifex"
# Shape and size. Bare metal is preferred; a flex VM is what the quota allows.
compute_shape = "VM.Standard.E6.Flex"
compute_ocpus = 8 # OCI counts an OCPU as a full core
compute_memory_in_gbs = 32
# 1 for a single node, 3 for a cluster. This is the only line that chooses.
node_count = 3
# The block volume behind /var/lib/spinifex, attached over iSCSI.
data_volume_size_in_gbs = 256
data_volume_vpus_per_gb = 120 # 120 = Ultra High Performance
node_client_cidr_allow_list = ["203.0.113.10/32"]
EOF
The variables worth knowing, all of which have defaults that build a working single node:
| Variable | Default | What it decides |
|---|---|---|
compartment_ocid |
— | Required. An OCID, not a name: this deploys into a compartment that already exists, because creating one needs tenancy-root rights the API user must not have |
deployment_name |
spinifex |
Prefixes every resource and becomes the VCN dns_label, so 1–15 lowercase alphanumerics starting with a letter |
node_count |
1 |
Nodes, 1–25. 1 for a single node, 3 for a cluster. Each gets its own volume, its own second VNIC and its own fault domain |
compute_shape |
VM.Standard.E6.Flex |
BM.* for bare metal, VM.*.Flex for a VM. A fixed bare-metal shape carries its own size, so shape_config is emitted only when the name contains Flex — set a BM shape and the two sizing variables below are ignored rather than rejected |
compute_ocpus |
8 |
OCPUs on a flex shape. Validated at ≥ 8 (16 vCPU), the minimum recommended for a node |
compute_memory_in_gbs |
32 |
GiB on a flex shape. Validated at ≥ 32 |
data_volume_size_in_gbs |
256 |
Size of each node's data volume. This is where guest disks, S3 objects and JetStream live — size it for the guests, not for the control plane |
data_volume_vpus_per_gb |
120 |
Performance tier, 0–120 VPUs/GB. 120 is Ultra High Performance; 10 is Balanced. Higher is more IOPS and more cost |
data_mountpoint |
/var/lib/spinifex |
Where cloud-init mounts it. Change only if you know why |
vcn_cidr |
10.200.0.0/22 |
VCN address space. A /22 leaves room for the secondary private IPs Spinifex registers per node |
public_subnet_cidr |
10.200.0.0/23 |
The subnet every node sits in, on both VNICs. Nodes serve the UI, the S3 gate and the AWS gateway directly, so they cannot live in a private subnet |
node_client_cidr_allow_list |
["0.0.0.0/0"] |
Who may reach SSH and the service ports from the internet. Change this |
node_service_ports |
[3000, 8443, 9999] |
Console, predastore gate, AWS gateway. SSH is opened separately |
ubuntu_version |
26.04 |
Canonical platform image version. The newest image for the shape and region is resolved at plan time rather than pinned to a regional OCID |
wan_bridge_name |
br-wan |
The Linux bridge built over the second VNIC |
wan_bridge_mtu |
9000 |
The OCI VCN carries 9000 |
Warning
node_client_cidr_allow_listdefaults to0.0.0.0/0. Nodes sit in a public subnet with public addresses because they serve the UI, the S3 gate and the AWS gateway directly — there is no bastion. Restrict this before any real use.
Note
Bare metal changes nothing else in this document. Set
compute_shape = "BM.Standard.E2.64"and the same config builds it: the two flex sizing variables stop applying, the shape's own CPU and memory take over, and every other step — the volume, the second VNIC,br-wan, the install — is identical. Nested virtualisation stops being a question, because there is no nesting.
Step 2. Build the infrastructure
KEY=~/.ssh/oci-spx.pub
python3 scripts/oci_env.py --ssh-public-key-path "$KEY" -- terraform init
python3 scripts/oci_env.py --ssh-public-key-path "$KEY" -- terraform plan -out=oci-spx.tfplan
Read the saved plan, then apply that exact plan rather than re-planning:
python3 scripts/oci_env.py --ssh-public-key-path "$KEY" -- terraform apply oci-spx.tfplan
The apply ends with the addresses you need for every remaining step. With node_count = 3 it prints twenty-two resources and this:
Apply complete! Resources: 22 added, 0 changed, 0 destroyed.
Outputs:
compartment_name = "mulgadc"
compartment_ocid = "ocid1.compartment.oc1..aaaaaaaa6nkk...ztta"
hosts_file = <<EOT
192.9.186.97
137.23.20.12
192.9.162.14
EOT
nodes = [
{
"fault_domain" = "FAULT-DOMAIN-1"
"name" = "spinifex-node-01"
"private_ip" = "10.200.1.249"
"public_ip" = "192.9.186.97"
},
{
"fault_domain" = "FAULT-DOMAIN-2"
"name" = "spinifex-node-02"
"private_ip" = "10.200.0.201"
"public_ip" = "137.23.20.12"
},
{
"fault_domain" = "FAULT-DOMAIN-3"
"name" = "spinifex-node-03"
"private_ip" = "10.200.1.79"
"public_ip" = "192.9.162.14"
},
]
subnet_ocids = {
"private" = "ocid1.subnet.oc1.ap-sydney-1.aaaaaaaayqu4...ssgq"
"public" = "ocid1.subnet.oc1.ap-sydney-1.aaaaaaaagptz...tefq"
}
vcn_ocid = "ocid1.vcn.oc1.ap-sydney-1.amaaaaaa6eq5...4mcq"
With node_count = 1 it is fourteen resources and one entry:
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
nodes = [
{
"fault_domain" = "FAULT-DOMAIN-1"
"name" = "spinifex-node-01"
"private_ip" = "10.200.0.252"
"public_ip" = "161.33.226.245"
},
]
hosts_file = "161.33.226.245"
Read nodes twice, because the two addresses in each entry are used for different things and confusing them wastes an afternoon:
public_ipis how you reach the node — SSH, the console on:3000, the AWS gateway on:9999. It is on the primary VNIC and Terraform assigned it.private_ipis what goes in--bind,--cluster-bind,--encap-ipand--db-cluster-*-addrin Steps 4 and 5. Every cluster flag takes the private address, because the mesh runs inside the VCN and never over the public one.fault_domainis not cosmetic. Three nodes in three fault domains is what makes a three-node cluster survive a rack, and it is chosen here, not later.
terraform output reprints all of this at any time, and terraform output -json nodes is the machine-readable form.
Warning
Never commit the state file, a plan file, or key material.
.gitignorecoversterraform.tfstate*,*.tfplan,*.auto.tfvars,terraform.tfvarsand*.pem.
Step 3. Install Spinifex and set up OVN
Spinifex installs the same way here as anywhere else. Run this on every node:
curl -fsSL https://install.mulgadc.com | bash
That puts the binary, the systemd units and the helper scripts in place and starts nothing. Then wire OVN, which is where the single-node and multi-node paths differ.
Two flags are the same on every node and on both paths, and both are decisions you cannot change afterwards without re-doing this step:
--managementis load bearing. Omitting it takes the compute-node branch, which stops and disablesovn-central— silently, leaving a node with no record of what it was meant to be.--nat-uplink, not--wan-bridge=br-wan. The two are mutually exclusive, and only--nat-uplinkcreates thespx-nattransit veth that routed mode runs on.--wan-bridgetakes the veth-to-br-extbranch and deletesspx-naton the way, after whichhost.Routed.EnsureUplinkPortrefuses with "not on OVS — run setup-ovn.sh --nat-uplink".
That does not make br-wan redundant. It is still doing two jobs, neither of which involves OVS:
- It names the VNIC for the OCI allocator.
oci_vnic_iface = "br-wan"in Step 6's pool config is resolved to a VNIC OCID by MAC, on the node it runs on — which is what lets one identical config file be correct on all three nodes. - It holds the addresses. Every external address Spinifex registers becomes a secondary private IP on that VNIC, appears on
br-wan, and gets its own source-routing rule into table 200. The guests' own traffic reaches it throughspx-natand the host'sspinifex-nat-egressmasquerade.
So a correct OCI node has br-wan as a Linux bridge with no OVS port at all, and br-ext as an OVS bridge whose ports are spx-nat-ovs and the patch to br-int. That asymmetry is the design, not a half-finished install.
If you suspect an OVS problem on this host, read
ovs-vsctl --columns=name,error list Interfaceand the vswitchd log. Never trustovs-vsctl's exit status — it exits 0 even when the datapath rejected the device.
Single node
sudo /usr/local/share/spinifex/setup-ovn.sh --management --nat-uplink
Three nodes
Servers 1 to 3 run a clustered OVN database, so VPC networking survives losing any one of them. Node 1 creates the cluster and must finish first; nodes 2 and 3 then join it. --recreate-db appears in all three because ovn-central starts a standalone database when the package installs, and a clustered one can only be created from scratch.
Every address below is a private IP from the Terraform nodes output.
# Run on every node, with the values from your own apply.
export SPINIFEX_NODE1=10.200.1.249
export SPINIFEX_NODE2=10.200.0.201
export SPINIFEX_NODE3=10.200.1.79
export SPINIFEX_OVN_REMOTE=tcp:$SPINIFEX_NODE1:6642,tcp:$SPINIFEX_NODE2:6642,tcp:$SPINIFEX_NODE3:6642
--ovn-remote is the whole member list, and it is the same on all three nodes — do not trim it to the peers. It is what ovn-northd dials, and an OVSDB RAFT follower does not forward writes: it answers not cluster leader, trying another server. A northd pointed at one member therefore works only while it happens to share a node with the database leader, and stops translating Northbound intent into Southbound flows the moment leadership moves. Nothing about that is visible from systemctl — every service stays active — and the symptom appears much later as a terraform apply hung on aws_internet_gateway.igw: Still creating... forever.
Node 1 — create the cluster:
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management --nat-uplink \
--db-cluster-local-addr=$SPINIFEX_NODE1 \
--ovn-remote=$SPINIFEX_OVN_REMOTE \
--recreate-db \
--encap-ip=$SPINIFEX_NODE1
Nodes 2 and 3 — join it, after node 1 reports === OVN compute node setup complete ===:
# on node 2
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management --nat-uplink \
--db-cluster-local-addr=$SPINIFEX_NODE2 \
--db-cluster-remote-addr=$SPINIFEX_NODE1 \
--ovn-remote=$SPINIFEX_OVN_REMOTE \
--recreate-db \
--encap-ip=$SPINIFEX_NODE2
# on node 3
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management --nat-uplink \
--db-cluster-local-addr=$SPINIFEX_NODE3 \
--db-cluster-remote-addr=$SPINIFEX_NODE1 \
--ovn-remote=$SPINIFEX_OVN_REMOTE \
--recreate-db \
--encap-ip=$SPINIFEX_NODE3
Each run ends with its own verification block. Node 1 reports chassis count: 1; nodes 2 and 3 report 0, because a chassis is registered by ovn-controller a moment later and the script checks immediately. That is not a failure — confirm the real answer from node 1 once all three are done:
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovn-sbctl show
Name: OVN_Northbound
Cluster ID: 7c7d (7c7d6d29-8243-4f82-8b4e-2848dbd5942e)
Server ID: 5d95 (5d952726-6ab5-4188-8eb9-b70acf37c374)
Address: tcp:10.200.1.249:6643
Status: cluster member
Role: leader
Term: 1
Status: cluster member with one leader, and a Chassis entry in ovn-sbctl show for each of the three. If cluster/status says the database is standalone, --db-cluster-local-addr did not reach it — re-run that node.
Then confirm northd is actually translating, which cluster/status cannot tell you. Run this on any node:
echo "NB $(sudo ovn-nbctl --no-leader-only get NB_Global . nb_cfg) / SB $(sudo ovn-sbctl --no-leader-only get SB_Global . nb_cfg)"
The two numbers must be equal, or within one of each other on a busy cluster. A gap that does not close within a few seconds means the active ovn-northd cannot write to the Northbound leader, and /var/log/ovn/ovn-northd.log will be looping on clustered database server is not cluster leader; trying another server. The fix is the --ovn-remote list above.
Step 4. Form the cluster
Everything here is Spinifex's own formation, unchanged from bare metal except for two flags that OCI requires:
--external-mode=natis mandatory. See the consequence.poolmode produces a cluster that passes every health check and drops every guest packet.--ipsec=false:openvswitch-ipsecis masked on the OCI Ubuntu image whilespx admin initdefaults the flag to true. A cluster formed with it on will not bring its overlay up.
Both are read from node 1 only. Joining nodes inherit the external mode, the transit pool and the IPsec posture, so they are chosen once.
Single node
sudo spx admin init --node node1 --nodes 1 \
--region ap-southeast-2 --az ap-southeast-2a \
--external-mode=nat --ipsec=false
It finishes in a few seconds:
📡 External networking: nat (routed; no public pool configured yet)
Transit: 100.127.0.0/24 via spx-nat-host (host masquerades out any uplink)
✅ Created: spinifex.toml
🔧 Configuring AWS credentials... Profile: spinifex
✅ Host DNS: spx3.net + compute.internal -> 10.200.0.252:53 (northstar)
🎉 Spinifex initialization complete!
Advertise IP: 10.200.0.252
"no public pool configured yet" is expected — the OCI pool is Step 6.
Three nodes
Init and join run concurrently: init blocks until every node has joined. Start node 1, take the token from its output, then run both joins while it is still waiting.
--force is in every command so the sequence is identical whichever way you installed. On a node Terraform just built there is nothing to lose either way; on a node that has been in service, joining discards its master key and orphans every volume sealed under it, and --force is the confirmation for that.
Node 1 — initialize:
sudo spx admin init --force \
--node node1 --nodes 3 \
--bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
--port 4432 --region ap-southeast-2 --az ap-southeast-2a \
--external-mode=nat --ipsec=false
It generates the keys and the CA, then stops and waits, printing the token:
📡 Formation server started on 10.200.1.249:4432
Waiting for 2 more node(s) to join...
Token expires in 30m0s
Other nodes should run:
sudo spx admin join --host 10.200.1.249:4432 --token spx_join_YHRL6y_yVP0E8Tc1 --node <name> --bind <ip>
Take the token from that output, but run the commands below rather than the line it prints — they add --force and --cluster-bind.
Nodes 2 and 3 — join, while init is still waiting:
# on node 2
sudo spx admin join --force \
--node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init> \
--region ap-southeast-2 --az ap-southeast-2a
# on node 3
sudo spx admin join --force \
--node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init> \
--region ap-southeast-2 --az ap-southeast-2a
Each join answers with the membership it now sees:
🎉 Node successfully joined cluster!
Cluster: spinifex (3 nodes)
Bind: 10.200.0.201 Advertise: 10.200.0.201 Loopback: 127.0.0.1
Nodes:
- node2 (bind=10.200.0.201 advertise=10.200.0.201)
- node3 (bind=10.200.1.79 advertise=10.200.1.79)
- node1 (bind=10.200.1.249 advertise=10.200.1.249)
and node 1 unblocks:
🎉 Cluster formation complete!
Cluster: spinifex (3 nodes)
Region: ap-southeast-2
Confirm the joiners inherited the OCI-critical settings before going on — this is cheap and catches a node that formed against the wrong leader:
sudo grep -E 'external_mode|ipsec_enabled' /etc/spinifex/spinifex.toml
ipsec_enabled = false
external_mode = "nat"
The join token expires 30 minutes after init; --token-ttl 2h if provisioning is slower than that.
Step 5. Configure Spinifex for OCI, start and verify
This is the step that is genuinely OCI-specific, and it is identical on one node or three. Do all of it on every node before starting anything.
Discover the OCIDs
# Instance, compartment and both VNICs, straight from the instance metadata.
curl -sH 'Authorization: Bearer Oracle' http://169.254.169.254/opc/v2/instance/ \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print("compartment:", d["compartmentId"]); print("instance:", d["id"])'
curl -sH 'Authorization: Bearer Oracle' http://169.254.169.254/opc/v2/vnics/ \
| python3 -c 'import json,sys
for v in json.load(sys.stdin):
print(v["macAddr"], v["vnicId"], v["subnetCidrBlock"], v.get("privateIp"))'
compartment: ocid1.compartment.oc1..aaaaaaaa6nkk...ztta
instance: ocid1.instance.oc1.ap-sydney-1.anzxsljr6eq5...tkxdq
02:00:17:01:9C:22 ocid1.vnic.oc1.ap-sydney-1.abzxsljrnat7...y6vw5q 10.200.0.0/23 10.200.0.252
02:00:17:04:1A:57 ocid1.vnic.oc1.ap-sydney-1.abzxsljrjd7z...czgvua 10.200.0.0/23 10.200.1.233
You only need the compartment OCID for the config below — the VNIC is resolved at runtime. Match a VNIC by MAC against ip -br link show br-wan if you want to confirm which is which; Oracle's VNIC ordering is not a documented contract, so "the second one" is not a selector.
Install the credentials on the node
The daemon cannot read ~/.oci/. Its systemd unit sets ProtectHome=yes, so a config under any home directory is invisible to it however the permissions read. Credentials go under /etc/spinifex/, on every node:
sudo install -d -o root -g spinifex -m 0750 /etc/spinifex/oci
sudo install -o root -g spinifex -m 0640 ~/.oci/oci_api_key.pem /etc/spinifex/oci/oci_api_key.pem
sudo tee /etc/spinifex/oci/config >/dev/null <<'EOF'
[spinifex]
user=ocid1.user.oc1..aaaa...
fingerprint=aa:bb:cc:...
key_file=/etc/spinifex/oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaa...
region=ap-sydney-1
EOF
sudo chown root:spinifex /etc/spinifex/oci/config
sudo chmod 0640 /etc/spinifex/oci/config
drwxr-x--- 2 root spinifex 4096 Sep 26 03:48 .
-rw-r----- 1 root spinifex 303 Sep 26 03:48 config
-rw-r----- 1 root spinifex 1715 Sep 26 03:48 oci_api_key.pem
The profile name — spinifex here — is what oci_config_profile names below. Keep a copy at ~/.oci/config too if you want the oci CLI on the node; that is a different consumer and it is not on any Spinifex code path.
Configure the OCI pool and the IMDS remap
Both go in /etc/spinifex/spinifex.toml, on every node, with the same text on each — nothing here is per-node:
[network]
external_mode = "nat"
imds_host_meta_ip = "169.254.42.254"
imds_host_dns_ip = "169.254.42.253"
# Leave the nat-transit pool init wrote exactly as it is, and add this one.
[[network.external_pools]]
name = "oci-public"
source = "oci"
oci_compartment_id = "ocid1.compartment.oc1..aaaa..."
oci_vnic_iface = "br-wan"
oci_config_file = "/etc/spinifex/oci/config"
oci_config_profile = "spinifex"
dns_servers = ["169.254.169.253"]
Notes on the keys:
- Exactly one of
oci_vnic_idandoci_vnic_iface, never both — the config is rejected if you set both or neither. Two that disagreed would send allocations to a VNIC the datapath is not on, and OCI would drop the traffic without a word. oci_vnic_ifaceis the right choice on a cluster, because it resolves through instance metadata by MAC on the node it runs on. Naming eitherbr-wanor the physical interface resolves to the same VNIC. Useoci_vnic_idonly when you want to pin a specific VNIC on a single node.oci_compartment_idis the only other required key.oci_subnet_idis optional and defaults to the VNIC's own subnet; set it only when you want private IPs from a different subnet.oci_config_filedefaults to~/.oci/configandoci_config_profiletoDEFAULT— on a node both need setting, because the daemon cannot read a home directory.oci_public_ip_pooltakes a BYOIP pool OCID. Accepted today so BYOIP is a config change later rather than a code change; see Limits.range_start,range_end,gw_lrp_range_*,bind_bridgeanddhcp_macare rejected on an OCI pool. OCI owns the addresses; a range you wrote would be fiction.- Leave the
nat-transitpool alone. In routed mode the per-VPC gateway router addresses come from the RFC 6598 transit range, not from OCI — only public addresses handed to guests consume an OCI address. Fifty VPCs and three Elastic IPs cost three reserved public IPs, not fifty-three. - The IMDS pair is required on OCI, and it is set both keys or neither — a half-configured pair is ignored and the node behaves as if the remap were off. The next section is why it exists;
169.254.42.254/169.254.42.253is the tested choice.
Start and verify
On every node:
sudo systemctl start spinifex.target
Then, from any one of them:
sudo spx get nodes
NAME | STATUS | ROLES | IP | REGION | AZ | UPTIME | VMs | SERVICES
node1 | Ready | nats:follower | 10.200.1.249 | ap-southeast-2 | ap-southeast-2a | 0m | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node2 | Ready | nats:follower | 10.200.0.201 | ap-southeast-2 | ap-southeast-2a | 0m | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node3 | Ready | nats:leader | 10.200.1.79 | ap-southeast-2 | ap-southeast-2a | 0m | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
Every node listed, all Ready, exactly one nats:leader, and the same SERVICES on each. spx top nodes shows pooled capacity and what the cluster can launch right now; if it looks like one server rather than three, the others never joined.
Confirm the OCI allocator came up, which spx get nodes cannot tell you:
sudo journalctl -u spinifex-vpcd --since -5m | grep -i ocinet
"msg":"ocinet resolved the external VNIC from its interface","pool":"oci-public","iface":"br-wan",
"vnic_id":"ocid1.vnic.oc1.ap-sydney-1.abzxsljrjwlf...kimuq","private_ip":"10.200.1.31","subnet":"10.200.0.0/23"
"msg":"OCI allocator ready","pool":"oci-public","collected":0,"stale_bindings":0,"skipped":1
resolved the external VNIC is the line that matters — it proves the credentials work, the compartment is right, and br-wan's MAC matched a real VNIC. A node missing it will accept allocate-address and fail it.
Note
On a cold cluster start you may instead see
OCI allocator reconcile failed … nats: no responders available for requeston some nodes. That is vpcd racing JetStream's KV at boot; the startup reconcile is skipped and not retried. It is harmless on a new cluster where nothing has been allocated, and it is tracked — restartspinifex-vpcdon that node to run it. Theresolved the external VNICline above is still the one that decides whether allocation works.
Step 6. Set up your cluster
The cluster is running, but it holds nothing yet — no machine images, no networks, no instances.
Continue to Setting Up Your Cluster to import an AMI, create an SSH key pair, create a VPC with a public subnet, and launch your first instance. It ends by arming the host firewall, which is also where you re-arm it if you turned it off to form the cluster.
Step 7. Run your existing Terraform against the cluster
This is the part worth pausing on. Everything above builds infrastructure on OCI using the OCI provider. Everything from here uses the AWS provider — unmodified, from the OpenTofu or Terraform registry — pointed at your own cluster. The same aws_vpc, aws_instance, aws_db_instance, aws_ecs_service and aws_eks_cluster resources a team already has in git apply against Spinifex on an OCI tenancy, with no rewriting and no OCI-specific module.
That is the claim, so it is measured rather than asserted. What follows is a run of the workbooks we ship against a three-node Spinifex cluster on OCI VMs.
Pointing the provider at your cluster
Three things differ from a provider block aimed at AWS, and only three:
provider "aws" {
region = "ap-southeast-2"
access_key = var.access_key
secret_key = var.secret_key
endpoints {
ec2 = var.spinifex_endpoint # https://<node private IP>:9999
s3 = var.predastore_endpoint # https://<node private IP>:8443
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
}
s3_use_path_style = true
skip_metadata_api_check = true
skip_region_validation = true
}
Credentials come from the [spinifex] profile that spx admin init writes into ~/.aws/credentials on node 1.
Use the node's private address, not its OCI reserved public IP. The node certificate carries no SAN for the public address, because that address is never on the wire — OCI NATs it to a private one. An
awsorterraformcall tohttps://<public IP>:9999fails TLS verification with hostname doesn't match. Until that is fixed, run Terraform from a node, or from anything else inside the VCN.
What we tested, and what happened
Measured on 2026-09-26 against a three-node cluster of VM.Standard.E6.Flex instances in ap-sydney-1, running spinifex v1.20.0-113. Each workbook is a full apply, a functional assertion against the thing it built, and a destroy.
| Workbook | What it exercises | Result |
|---|---|---|
nginx-webserver |
VPC, subnet, IGW, security group, key pair, EC2 instance, public IP; HTTP 200 from the internet | Passed (86s) |
bastion-private-subnet |
Public and private subnets, NAT egress, a bastion, SSH hop to a private host | Passed (117s) |
rds-quickstart |
DB subnet group, parameter group, aws_db_instance (PostgreSQL), client instance, connection |
Passed (296s) |
ecs-quickstart |
ECS cluster, task definition, service, container instances, ALB with a healthy target | Passed (166s) |
eks-quickstart |
EKS control plane, managed node group, kubectl against the cluster, nodes Ready |
Passed (259s) |
nginx-alb |
Application Load Balancer across two subnets, two backends, health checks | Passed (123s) |
s3-webapp |
S3 bucket, IAM role and instance profile, IMDS-fetched credentials, upload from the guest | Failed — see below |
demo-app |
Container image for the EKS workbooks, pushed to ECR | Passed |
eks-https-ingress |
Private-subnet workers behind a NAT gateway, LBC addon, ACM certificate, HTTPS Ingress | Passed (252s) |
eks-gitops-argocd |
Private-subnet workers, Argo CD addon, EBS-CSI PersistentVolume, GitOps sync | Passed (225s) |
Nine of the ten pass. RDS, ECS and EKS are the answer to the question this section exists to ask: an EKS control plane and a managed node group come up, kubectl get nodes reports them Ready, a PostgreSQL instance accepts connections, and an ECS service runs behind a load balancer with a healthy target. None of it knows it is running on someone else's cloud.
The last two are the harder network. Their workers sit in private subnets and reach the cluster endpoint and ECR through a NAT gateway, so a worker registering Ready is the proof that the whole NAT path works — which on OCI it did not until three defects in the gateway's address handling were fixed. If those two pass and eks-quickstart also passes, NAT egress is sound.
s3-webapp fails on the read-back rather than the create: the AWS provider issues an S3 Control ListTagsForResource for the bucket, an endpoint nothing serves, and the SDK builds its hostname by prefixing the account ID onto the host — which cannot resolve against an IP. The bucket, the IAM role and the instance are all created correctly first. This is not OCI-specific — it fails the same way on bare metal — and it is a known defect we are fixing. Until then, either pin the AWS provider to 5.x or set skip_requesting_account_id = true in the provider block, and buckets apply cleanly.
Two things bit during the run that are worth knowing before you hit them, neither of which is a reason the workbooks do not work:
- A
spinifex-daemonrestart while a node group is creating is terminal for that node group —NodeCreationFailure: create interrupted by daemon restart, with no retry. - A
terraform destroyimmediately followed by aterraform applyof the same EKS cluster name can fail withResourceInUseExceptionwhileDescribeClusterandListClustersboth report the cluster gone, so there is nothing to wait on. Leave a minute between the two.
Step 8. Oracle Linux as a guest image
Spinifex ships four Oracle Linux entries in its image catalog, so a customer on OCI can run the same distro their Oracle support contract covers. They import exactly like any other AMI — nothing about them is OCI-specific, and they run equally well on bare metal:
| Catalog name | Release | Arch | Kernel |
|---|---|---|---|
oracle-10.1-x86_64 |
Oracle Linux 10.1 | x86_64 | UEK 8 |
oracle-10.1-arm64 |
Oracle Linux 10.1 | arm64 | UEK 8 |
oracle-9.8-x86_64 |
Oracle Linux 9.8 | x86_64 | UEK 7 |
oracle-9.8-arm64 |
Oracle Linux 9.8 | arm64 | UEK 7 |
sudo spx admin images import --name oracle-10.1-x86_64 --config /etc/spinifex/spinifex.toml
aws ec2 describe-images --query 'Images[].[Name,State,BootMode]' --output text
All four boot UEFI, so launch them into a shape that boots UEFI — a BIOS-only instance type will not come up, and the failure looks like a hung boot rather than a rejected image.
The login user is cloud-user, not opc
ssh -i path/to/key.pem cloud-user@<public-ip>
This is the one that catches people. opc is the default user on the images Oracle publishes into OCI itself; these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on both releases: opc, oracle and ec2-user are all refused. The Spinifex UI's instance detail page shows the right user per AMI, so read it there rather than guessing from the distro.
Why these four pin a checksum digest inline
Every other catalog entry verifies against the publisher's sums file. Oracle ships none — the digests are published as HTML on yum.oracle.com/oracle-linux-templates.html — so these four carry ChecksumDigest in spinifex/utils/images.go instead. That is safe only because each URL names an immutable build (b291, b293, b178, b182) and Oracle never moves one. A new point release is a new URL and a new digest, never an edit to an existing entry.
On arm64 Oracle publishes a -kvm-cloud- build alongside a plain -kvm- one. The catalog takes -kvm-cloud-: it is the one with cloud-init, and so the one that matches x86_64's -kvm-. The plain arm64 build imports fine and then has no datasource, which surfaces as an instance with no key and no metadata rather than as an import error.
SELinux is enforcing
Both releases ship SELinux: enforcing, unlike the Debian and Ubuntu images. That is the upstream default and Spinifex does not change it. It does not affect networking, IMDS or cloud-init — all verified working — but a workload that has only ever run on Debian may meet it for the first time here.
Verify end to end
export AWS_PROFILE=<your-profile>
aws ec2 allocate-address
aws ec2 run-instances --image-id <ami> --instance-type t3.micro --key-name <key> \
--subnet-id <subnet> --associate-public-ip-address
aws ec2 describe-instances --query \
'Reservations[].Instances[].[InstanceId,PrivateIpAddress,PublicIpAddress,State.Name]' \
--output text
Then, from the host:
ping -c 3 <public-ip>
ssh -i <key> ubuntu@<public-ip> hostname
And from inside the guest — note that IMDS is IMDSv2, so it needs a token; an untokened request correctly returns 401:
TOK=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
-H 'X-aws-ec2-metadata-token-ttl-seconds: 120')
curl -s -H "X-aws-ec2-metadata-token: $TOK" \
http://169.254.169.254/latest/meta-data/instance-id
traceroute -n 8.8.8.8
A healthy topology is three hops before OCI's network: the OVN router port, the transit gateway, then upstream.
On a cluster, prove an address survives a move. scripts/terraform/oci-spx/eip-move-test.sh does the whole sequence — it asserts that an auto-assigned address is returned to OCI on stop and a different one comes back, then associates an Elastic IP, forces the guest to start on another node, and checks OCI's own record of which VNIC carries the private half:
INSTANCE=i-... ./scripts/terraform/oci-spx/eip-move-test.sh
What a correct node looks like
Worth knowing so the asymmetry does not read as a fault:
sudo ovn-nbctl lr-nat-list <logical-router> # dnat_and_snat holds the PRIVATE address
aws ec2 describe-instances ... # reports the PUBLIC address
ip route show | grep spx-nat-host # route to the PRIVATE address
These disagreeing is the design. The public address is 1:1-NATed by OCI upstream and never touches the wire, so every datapath object holds the private half while every AWS-facing record holds the public half. If OVN showed the public address, the rule would be matching something that never arrives.
Troubleshooting
Addresses and quotas
Spinifex enforces no limit of its own. It attempts the allocation and reports what OCI says, so the ceiling you hit is always your tenancy's, never a number baked into the software. A quota is raised with Oracle, not reconfigured here, and there is nothing to restart afterwards.
InsufficientAddressCapacity on allocate-address is what exhaustion looks like. It is the AWS error code for "out of addresses" and Spinifex returns it for either quota, mapped from OCI's own refusal. On a cluster the regional one is the likelier of the two.
A detached reserved public IP still bills. disassociate-address returns the address to AVAILABLE without deleting it, exactly as an unassociated AWS Elastic IP behaves, and it keeps consuming your regional quota as well as your bill. release-address is what frees both.
A stopped instance costs nothing for its auto-assigned address. Stopping an instance returns its auto-assigned address to the pool and deletes the reserved public IP object behind it, as AWS does; the start that follows creates a new one. An Elastic IP is yours and survives the stop untouched — that is the whole distinction between the two, and on OCI it is also the difference between a stopped fleet that bills for addresses and one that does not.
Datapath
Instance launches but the public address is unreachable. In order:
sudo ovn-nbctl lr-nat-list <router>— is there adnat_and_snatrow, and does it hold the private address?ip route get <private-addr>— does it route via the transit veth?sudo iptables -S FORWARD | grep spinifex-eip-ingress— both directions present?oci network private-ip list --vnic-id <vnic>— is the private address actually registered with OCI, and on this node's VNIC? If not, nothing else matters.
Egress works from one address and not another on the same NIC. That is source-IP enforcement, and it means the address is not a registered private-IP object on the VNIC. It is never a routing problem on the host.
An address went dark after a guest moved node. The affinity pass should have claimed it within 15 seconds. Check the new node's daemon journal for ocinet affinity, then confirm with step 4 above which VNIC OCI thinks holds the private half.
A node that was fine and then was not
Terraform's cloud-init does the host jobs on each node — the iSCSI volume, br-wan and its source routing, the firewall. When one of them did not take, the symptom arrives much later and rarely looks like a networking or storage problem. This one command answers all of it:
ssh -i ~/.ssh/oci-spx ubuntu@<node-public-ip> '
cloud-init status # done
sudo iscsiadm -m session # a session to 169.254.2.2:3260
findmnt /var/lib/spinifex # mounted by UUID, with _netdev
ip -br addr show br-wan # UP, second VNIC address as a /32
ip -br addr show enp1s0 # the bridge MEMBER holds NO address
ip route show default # still on the PRIMARY interface
ls -l /dev/kvm # present
'
Three answers are worth knowing in advance, because each looks wrong and is not:
br-wanholds a/32and the default route is on the other interface. That is how two VNICs share one subnet without the second stealing the first one's traffic, and it is what satisfies OCI's per-VNIC source check.br-wan's address appears inip ruleinstead of in the default route.br-wan's address is not one you chose. It is the second VNIC's own private IP, assigned by OCI, and not predictable before the apply.enp1s0carrying an address is the one real fault here. The kernel derives an on-link route from it that beats the primary VNIC's, so every packet to another node leaves the wrong VNIC and OCI drops it. The node still forms a cluster and still reportsReady— formation uses outbound connections — and it surfaces later aspredastore … the stripe is short one holderon the first AMI import.sudo /usr/local/sbin/spinifex-setup-wan-bridgerepairs it in place.
Host
RunInstances returns ServerInternal. Check the daemon journal for the real error — journalctl -u spinifex-daemon --since -10m. An OCI allocation failure surfaces this way.
The stack ends up on the boot disk and the node fills. The data volume was mounted without _netdev and lost the race with iscsid. findmnt /var/lib/spinifex is the check; ls cannot tell the difference.
Guests cannot reach instance metadata
Guests boot but cloud-init hangs on metadata, or the host stops answering its own. OCI uses 169.254.169.254 for its instance metadata, and so does every AWS-compatible guest — the host and the guests want the same address.
Spinifex resolves this by moving the host side of the per-guest metadata link to a different link-local pair, so guests keep asking 169.254.169.254 exactly as they do on AWS. Step 5 sets imds_host_meta_ip and imds_host_dns_ip for you. Set both or neither — a half-configured pair is ignored and the node behaves as if the remap were off, which is what this symptom looks like.
Check it after your first guest launches:
ip route get 169.254.169.254 # your real NIC, not lo
curl -sH 'Authorization: Bearer Oracle' -o /dev/null -w '%{http_code}\n' \
http://169.254.169.254/opc/v2/instance/ # 200
ip -br addr show | grep ime- # holds 169.254.42.x, not .169.x
ip route get 169.254.169.254 returning dev lo is the signature of a missing remap. The same signature can appear after a guest terminates, from a leftover ime- endpoint still holding the address; with the remap configured it is harmless, and removing the stale OVS port clears it.
Limits in this version
- Operator-provisioned API key, not instance principal. The key is on each node's disk and its rotation is your responsibility.
- One VNIC per pool. A pool cannot grow past the per-VNIC secondary private IP limit of 64. Three nodes give you three pools' worth; a single node is capped at 64 addresses.
- IPv4 only. IPv6 on OCI is materially simpler — no pools, prefixes assign directly to VCNs — but is not implemented.
- No BYOIP. The integration accepts a public IP pool OCID in config so BYOIP becomes a config change rather than a code change, but the RIR validation and Oracle's own validation window both precede any use of it.
- OCI Block Volumes are not an EBS provider. They back
/var/lib/spinifexand therefore viperblock, which captures most of the performance benefit, but there is no native OCI block provider. - A recovered guest is briefly unreachable on its public address. The address is never renumbered, but the node it is delivered to is corrected by a background pass every fifteen seconds rather than by the recovery itself. Private VPC addresses are unaffected. Support status has the detail.
- Terraform builds infrastructure, not Spinifex. Formation is still the standard install path, by design: it is the same path prod and bare metal take, and diverging it for one cloud would mean two formation paths to keep correct.
What is no longer a limit
Recorded because earlier versions of this guide said otherwise:
- Multi-node works. The
--nodes=1cap on routed mode is gone, three-node formation and guest-to-guest traffic are proven on OCI, and the routed-NAT datapath is guarded nightly on three nodes — on our own hardware, not on OCI. See Support status. - Public addresses are not pinned to one node. Each node allocates on its own VNIC, and an affinity pass reassigns an address's private half to whichever node the guest is on — one OCI call, keeping the OCID, so the customer's public address never changes. No node needs another node's VNIC OCID, and
oci_vnic_ifaceis correct on every node. - Addresses are released when an instance stops. An auto-assigned address goes back to the pool on stop and the reserved public IP object is deleted, so a stopped fleet stops costing you addresses.