mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-21 19:56:33 +00:00
feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108)
* feat: ship uninstall-spx.sh and fix the OCI bring-up path
Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the
software where node-reset.sh clears state and leaves the node installed. An
operator who installed with `curl ... | bash` had no teardown path at all.
State teardown is delegated to node-reset.sh rather than reimplemented, and
setup.sh now ships both to /usr/local/share/spinifex so they are reachable
without a checkout.
It refuses on a multi-node cluster and --yes does not override that: a flag
meaning "don't ask me" must not also mean "ignore a safety property".
/var/lib/spinifex and every apt package are kept unless asked for, and both
opt-ins confirm separately even under --yes.
Proved by a full uninstall/reinstall round trip on dev-oci, which found two
bugs neither prod nor dev-prod can reach, because both are ISO-installed:
node-reset.sh exited 2, silently, at its last step and after the wipe.
setup.sh creates the spinifex *group* and the spinifex-* service users but
never a *user* named spinifex, so `getent passwd spinifex` exits 2 and
pipefail carried it into set -e.
The uninstaller broke openvswitch-switch on its first run by removing the
socket-perms helper while leaving the drop-in referencing it, failing the
unit at step EXEC with status 203. Fixed on both sides: the uninstaller
removes drop-ins it wrote into other units' namespaces and restarts anything
it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can
never stop another package's daemon.
setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux
bridge it links it to OVS with a veth pair instead of enslaving the NIC, so
the VNIC keeps its MAC identity and netplan keeps owning the addresses. It
now fails on OVS datapath errors instead of reporting success, and the
--management demotion guard keys on /etc/default/ovn-central rather than on
database presence, which the ovn-central package creates on every node.
Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts
along with shellcheck over the four scripts that ship to every node.
* feat: allocate external IPs from the Oracle Cloud API
An OCI VNIC drops any packet whose source is not a registered private IP
object on it, so an external pool cannot be range math there — the address
has to be created through the provider before it exists at all. Add
source = "oci" and the allocator behind it.
One allocation is two OCI objects: a secondary private IP that rides the
wire, and a RESERVED public IP attached to it that the internet reaches.
Allocate returns the public address because that is what AWS semantics are
about, and records the pair so the host plumbing and release path can find
the private half.
The OCI objects are created before the binding is written, and dropped
after it on release — both orders chosen so a crash leaves a discoverable
leak rather than a record that lies. Reconcile squares the two on startup
and is required, not optional: a leaked reserved public IP bills and holds
one of the 50 per-region slots with nothing referencing it. It never
touches an object without our display-name prefix, because the reference
host carries operator-created secondaries on the same VNIC.
Resolve the VNIC from an interface name by MAC through IMDS rather than by
Oracle's VNIC ordering, which is not a documented contract.
Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring
is not a second single-purpose installer.
* fix: refuse source = "oci" outside routed mode, and cover the SDK path
An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound
frame to it regardless of which of its addresses the packet is for.
Measured on the reference host: an internet source reaches 10.200.0.144 as
router-MAC to VNIC-MAC, the public address never appears on the wire at
all, and every off-subnet destination resolves to the same next hop.
Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on
the uplink and ARPs for the address, which policy.go:50 justifies with "the
Linux bridge learns the per-rule external MAC on the veth port like any
other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode
guest's egress is dropped and its ingress never reaches OVN. Routed mode
keeps the address on the host, which already wears the VNIC's MAC.
Pinning every LRP to that MAC does not rescue pool mode: they share one
external logical switch and OVN resolves inbound frames by an L2 lookup on
it, so duplicate MACs work for one VPC and silently misroute past that.
The refusal names the MAC reason, because a refusal that only says "use nat
mode" reads as arbitrary and gets worked around.
Also test the SDK request shapes against a local endpoint. Two of them are
wrong in ways reading does not catch: pagination stopping early reports a
partial VNIC to reconcile, which then drops live bindings, and a nil
privateIpId on detach is omitted from the body entirely, so OCI reads the
update as a no-op, returns 200, and leaves the address attached and billing.
* fix: install host firewall rules where a distro's catch-all reject cannot precede them
Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD.
Two Spinifex datapaths were silently dead behind it on any such host.
EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules
landed after the REJECT and never matched: 24113 packets on the reject, 0 on
both accepts, and guests had no egress. The comment named the case it did not
handle, anticipating a FORWARD policy of DROP — which an append does survive —
rather than an explicit REJECT rule, which it cannot. Insert at the head, and
drain existing copies first, because a rule an earlier build appended still
satisfies -C and would otherwise stay stranded across an upgrade.
Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254
and .253 per instance on the ime- devices, so guest requests are locally
delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init
spent 490s timing out and reported the endpoint as disabled. With one wildcarded
accept the same image boots in 18.6s and finds its datasource.
The natuplink e2e check had the same blind spot the code did, asserting -C and
therefore presence but never position. It now asserts the accepts precede the
first reject.
* feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata
An OCI external address is two objects, not one: a RESERVED public IP that
OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP
on the VNIC that the wire actually carries. Every AWS-facing record holds the
public half; every datapath object must hold the private one.
The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP,
PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile
rebuilds the rule set from AWS-facing intent on every pass — so a correctly
mapped rule installed further upstream is swept on the next reconcile and
replaced with one matching an address that never arrives. The seam is a
DatapathResolver defaulting to identity, so every non-OCI deployment is
unchanged. A resolve failure is fatal: falling back to the public address
would install a rule that reads correctly in ovn-nbctl and blackholes.
ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no
credentials, because vpcd is a different process from the daemon that owns
the allocator and the bindings record is all either would consult.
Two live bugs the in-memory test store was hiding, both found by making the
fake report ErrNotFound the way the real store does:
- KVStore.Mutate could not create, so every pool failed its own first
allocation after OCI had already created and billed the pair.
- Reconcile treated a missing record as a failure, aborting the one pass
that exists to find an object created before its binding was written.
Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's
own metadata service lives — the first guest launch made it a local address
and took the host's metadata and DNS with it. The datapath is OpenFlow
steered, so the guest-facing addresses stay put and only the host-side
endpoint address moves, with a PREROUTING DNAT on ime-+ translating and
conntrack un-NATing the reply. Off by default; identity when unset.
The IMDS firewall rules move to vpcd. They are wildcarded over every
endpoint rather than per-launch work, and spinifex-daemon holds no iptables
grant and no CAP_NET_ADMIN by design, so attempting them aborted every
launch.
* fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package
The error string pointed at DefaultHostBindAddrs, which does not exist — a
caller following it would have looked for a symbol that was never written.
nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries
the //test:in-package marker the package check asks for.
* docs: index what each e2e suite proves
Written because a datapath comment was read as an end-to-end behavioural claim
and the suite that already answered it was found only after the wrong answer
had been given. TestVPCEgressPaths/NATBaseline is the authority on private
subnet egress, and nothing pointed at it.
* fix: source-route an EIP's reply out the VNIC that holds it
An EIP's reply carries the EIP as its source. On a host that pins its
uplink with a policy rule, the reply took the main table instead and left
by the wrong interface — which a cloud VNIC drops, because the address is
registered on a different one. Ingress reached the guest and nothing came
back.
Every bind now mirrors the uplink's rule onto the EIP. The selector is the
rule rather than the interface: both OCI VNICs sit in one subnet, so "the
interface owning the EIP's subnet" names two of them, and two candidate
rules is a refusal rather than a guess. A host with no source routing gets
no rule and no change.
* docs: name the setup-ovn recipe routed mode actually needs on OCI
The OCI guide told operators to run `--wan-bridge=br-wan`, which is
mutually exclusive with `--nat-uplink` and deletes the spx-nat transit
veth routed mode runs on. The node then refuses to bring the uplink up,
with an error naming the flag the guide should have used.
Several messages also claimed routed mode is outbound-only with no
public IPs. That stopped being true when the routed EIP path landed, and
it reads as "what you are about to do cannot work" during the one init
the OCI path documents.
* feat: let routed NAT form a multi-node cluster, and carry every pool through formation
Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no
technical justification -- it was a v1 scope decision -- but routed NAT is the
only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become
"Spinifex on a cloud host is single-node".
It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms,
all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but
every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only
the gateway chassis reaches its own guests. init now says so on stderr rather
than refusing, because the refusal made the defects unreachable and therefore
unfixable. Distributing the EIP datapath is the actual fix.
Lifting it exposed a second defect that stops formation outright. NetworkConfig
held nine flattened Pool* scalars and the init node set PoolName = "wan"
unconditionally; routed NAT renders two pools and the one that matters --
nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a
single pool named "wan" holding the transit gateway and the detected WAN prefix,
and the leader re-rendered its own config from the same struct, so all three
nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData
gains json tags because it is on the wire now rather than a template input.
* feat(install-node): add --external-mode so a cluster can be formed in routed NAT
install-node.sh could not express routed NAT at all. It required
--external-pool/--external-gateway/--external-prefix-len and passed
--external-mode=pool to every spx admin init unconditionally, so the only
supported way to form a routed-NAT cluster was by hand.
--external-mode nat refuses the three pool flags rather than ignoring them, and
threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check
rejects a physical NIC during formation, which is correct in pool mode and
wrong in routed mode, where nothing is bridged.
* fix(ocinet): stop each node deleting its peers' OCI addresses on startup
Reconcile compares the binding record against the private IPs on one VNIC. The
record is one key per pool and therefore cluster-wide, so on more than one node
every peer's binding looks exactly like a binding whose objects OCI has lost.
Each node dropped them, and direction 1 on the owning node then collected the
now-unclaimed objects as leaks -- deleting the reserved public IP and private IP
of a running instance.
Observed on the three-node OCI cluster: node2's startup pass logged "collected a
leaked address" for an address it had allocated ten minutes earlier and "dropped
a stale binding" for one living on node3's VNIC.
The damage did not stop at the OCI objects. With the binding gone,
Lookup.DatapathIP returns the public address unchanged, so the OVN
dnat_and_snat row carries the public address instead of the private half. Guest
egress then leaves the VNIC sourced from a reserved public IP, which OCI drops
because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute
no-ops too, since a public address shares no subnet with any host address. One
destroyed binding takes out both directions.
Binding.VNICID already exists for exactly this ("how a node that comes back from
the dead tells mine from moved on without me") and Reconcile never consulted it.
A binding is now only dropped when it names this node's VNIC; one naming no VNIC
cannot be attributed and is left alone rather than guessed at.
* fix(host): find the EIP's source-routing table when the uplink is held as a /32
uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding
a host address configured with a prefix that contains it. On OCI both VNICs sit
in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts
the subnet's on-link route in the main table at metric 0 and takes the whole
subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was
ever found, no source rule was installed, and the reply left by the primary
VNIC where OCI dropped it for carrying an unregistered source.
The address is not the only record of who owns the subnet. The rule names a
table, and that table holds the on-link route. Consulted as a fallback, only
when the address prefixes matched nobody, so a host that already resolved this
the direct way keeps its existing answer -- prod and dev-prod are unaffected.
Only scope-link routes count. A default route covers every address and would
make every source-routing table an equally good answer.
* docs(oci): correct the multi-node story and name the oci_vnic_id constraint
The nat-mode warning said EIPs and default egress were both broken on more
than one node. Default egress is fine: delete a guest's dnat_and_snat row and
it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the
public-IP path was broken, and the three defects behind it are now fixed.
What remains is not a bug but the shape of the datapath: routed NAT centralises
the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other
nodes cannot reach it, and OCI drops any frame whose source is not registered on
the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway
chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator
writing the pool will read it, and in §9, where the cost is that a gateway
change kills every public IP until the pools are repointed.
The single-node cap itself is gone from the doc: a three-node cluster has been
formed and measured, and the warning proceeds rather than refusing.
* feat(nat): distribute the routed-NAT EIP path so it works on more than one node
Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is
a localnet, so the transit segment a /32 route pointed into existed only on
that one node. Measured on three nodes: all three claimed all three EIPs, the
nodes disagreed on the gateway LRP, and only guests on the gateway chassis were
reachable.
The mechanism to fix it is not new. Pool mode already distributes EIPs by
putting external_mac and logical_port on the dnat_and_snat row and letting
OVN's ARP responder answer for the external IP on the external segment. Routed
mode has an external segment too, the spx-nat veth pair, and it being per node
is the distribution: only the node running the instance can answer ARP on it.
So a routed rule is now distributed on the same terms as a pool one, and its
host route becomes on-link on the transit veth instead of via a gateway LRP
that may be on another chassis. bindHostEIP stops demanding that address for a
distributed rule, which is what previously failed the bind on every node but
one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS
rather than the Southbound DB because the question is about this host.
The gateway LRP keeps its transit-pool address. Default egress routes through
it and works across three nodes today, so making it link-local as pool mode
does would break a working path to fix one distribution has already fixed.
Pool and static are unmoved, and a test asserts that against the same input
rather than leaving it to inspection. A rule with no MAC stays centralised, so
a cluster part-way through an upgrade is never in a state neither shape handles.
Beads: mulga-opyl7, mulga-la0pe
* fix(nat): make routed-EIP host plumbing work on every node
Three guests on three OCI nodes, one guest per node: only one was
reachable. Two defects, both invisible on a single node.
The transit veth took a random MAC per node, but OVN holds one MAC
binding for the transit nexthop across the whole cluster — so every
chassis addressed its egress to whichever node seeded the binding, and
the other kernels dropped it at L2. The pair is node-local, so one
locally-administered address everywhere is unambiguous; setup-ovn.sh
pins it at creation and in the netdev unit, and EnsureUplinkPort repairs
a veth made before that.
Host routes and proxy-ARP also rode the leader-gated reconcile pass, so
a node that never won the lease never plumbed its own guests' EIPs.
BindHostEIPs does that half on its own loop on every node; Bind already
leaves alone an EIP whose ENI is elsewhere, so two nodes never contend.
An unresolvable address skips the sweep rather than shrinking it.
Pool and static are untouched: the new pass is a no-op outside routed
mode, and a test asserts it.
Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on
a node that is already tangled. It deliberately spares the NB/SB
databases, /etc/spinifex, the system-id and the packages, and refuses to
del-br a Linux bridge.
* fix(nat): assert the transit MAC where vpcd actually runs it
EnsureUplinkPort is never called in production — host.Wiring is
constructed nowhere — so the repair landed in a path the deployed
daemon does not take, and three nodes came back with three different
MACs. vpcd re-asserts it beside the NAT egress rules instead, which is
the routed-NAT branch that does run on every start.
* fix(vpcd): make uplink-mode detection say what it saw, and check it
detectBridgeMode logged only the branch it took, so a node detected as
veth when it was wired for nat looked identical to one that was right.
It now reports both probes every time, errors when a node carries both
uplink veths (set up twice, and --teardown is the way out), and errors
when external_mode and the wiring disagree — the datapath follows the
wiring, so the config silently loses that argument. An explicitly
configured bridge_mode gets the same cross-check, since nothing else
ever compared it against the wiring.
The vpc-networking doc now opens nat with "start with pool unless one
of these applies" and gains a section on nat across more than one node;
the admin warning for --external-mode=nat on two or more nodes says
what was verified by hand and what is still uncovered.
* test(nat): prove routed NAT delivers each node's own guests, on a cluster
natuplink ran single-node only, which is exactly where the two defects
that broke three nodes are invisible. It now admits ModeMultinode, and
phase 9 takes a second shape there: a guest per node in a spread
placement group, and for each one — its public IP plumbed by the node
running it and by no other node, the route on-link with proxy-ARP
answered there, a dnat_and_snat row carrying external_mac and
logical_port, a TCP handshake in from another node, and the guest's own
console verdict for egress. Every node's transit veth is checked
against the one cluster-wide MAC before any guest boots.
Single-node phase 9 gains the same row and on-link assertions, so the
shape is held on both topologies.
Two gates come from what three OCI nodes actually do: every node holds
a gateway_chassis row for every VPC (the active one is the highest
priority) and every node installs the VPC ingress route, so only phase
7 needs gating — it runs where the gateway port and the guest are both
local, and logs which one is missing otherwise.
Cell 20 is real-multi x nat with its own nightly step, leaving the
permutation cells' suite set alone.
* test(nat): run the VPC egress paths in nat mode, not pool mode only
The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on
external_mode = "pool", with a comment saying nat clusters have no
EIP/NAT-GW support. They have had EIPs since the routed path landed, so
the gate now asks whether the cluster can allocate public addresses at
all — pool, or nat with a non-transit pool configured.
The nat-single cell runs TestVPCEgressPaths beside natuplink, which is
the only nightly coverage of a private subnet's isolation and its NAT
gateway in routed mode. natuplink never creates a private subnet.
* docs(config): list nat among external_mode values
* fix(ci): renumber the multi-node nat cell off the bare-metal job
Cell 20 is already the bare-metal single-node job, gated on the same
cells input but declared in the workflow rather than e2e-cells.json.
A matrix entry with the same number runs two unrelated jobs under one
name, and naming 20 in a dispatch selects both.
* feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests
Oracle publishes its qcow2 checksums inline on an HTML page and ships no
sums file, so the four new entries carry the expected digest in the catalog
rather than a URL to fetch one from. Images gains ChecksumDigest for that,
and the catalog import path verifies against it and says which of the two
sources it used, so a recorded digest never reads as a fetched sums file.
Pinning is safe here and only here: each URL names an immutable build
(b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build
alongside the plain -kvm- one; the cloud build is the cloud-init image and
is the one that matches x86_64's -kvm-.
TestImageCatalogReachable HEADs every URL in the catalog from the general
single-node suite. These artifacts live on hosts nobody here controls, so
an upstream that retires a build breaks `spx admin images import` for a
customer and nothing else in the suite would notice. It also asserts that
every entry sets exactly one checksum source, and that no pinned entry
points at a "latest" URL whose bytes would change under it.
* fix(formation): carry bridge_mode onto the leader and every joiner
applyNetworkConfig copied external_mode, external_interface and the pools
out of the formation NetworkConfig but never bridge_mode, so a multi-node
cluster wrote a spinifex.toml without the key while single-node init wrote
it. Every node then fell back to autodetecting a mode that init had pinned.
Nightly cell 30 caught it: TestNATUplink failed at the config phase on a
spinifex.toml that did not contain bridge_mode = "nat". Deriving it from
the mode rather than adding a NetworkConfig field keeps the leader and the
joiners on the same rule single-node init already uses.
* feat(ui): show how to ssh into an instance, and which user to use
An instance detail page gave the addresses and the key pair name but never
the login user, which is the one part of the command nobody can derive. It
belongs to the AMI, not to Spinifex: nothing in the launch path creates or
renames an account, so stock cloud-init applies the distro's own
default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is
ubuntu, and guessing from the distro name gets two of those three wrong.
The card resolves the user from the AMI's name and description, and an
unrecognised image yields a placeholder rather than a guess, because a
wrong username in a copy-paste command is worse than an honest gap. A
private-only instance is called out as reachable from inside the VPC only,
rather than presented as a command that will simply time out.
* fix(nat): plumb host state for a NAT gateway's public address
AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT
node the host is what delivers a public address, so an egressing private guest
left with a source the host held no route back to, and the masquerade rule —
which matches the transit 100.127.0.0/24 alone — never saw the packet. The
gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD
accepts an EIP gets.
hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway
address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs
deliberately: pruneOrphanEIPs and addLive read that set as the complete list of
addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so
widening it would stop the sweep reclaiming a stale row on an address later
reused as a gateway's.
AddNATGateway also binds directly, on both the create and idempotent-skip
paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP
pass — and so a reboot that keeps the OVN row still replays the volatile half.
Documents NAT gateways in nat mode in vpc-networking, and says at the head of
that section that nat is a supported mode covered by nightly cells 19 and 30.
* docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user
Four catalog entries, all UEFI. The login user is the one that catches
people: opc is the default on the images Oracle publishes into OCI itself,
while these are the generic KVM cloud images from yum.oracle.com, whose
cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle
and ec2-user are all refused.
Also records why these four pin a checksum digest inline (Oracle ships no sums
file, and each URL names an immutable build), why arm64 takes the -kvm-cloud-
build, and that SELinux ships enforcing.
* fix(reconcile): stop reading an unreadable intent as an empty one
A wedged JetStream KV returned no keys for the ENI bucket while writes were
still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts
read every guest LSP as an orphan against it, and every guest on the node lost
DHCP and IMDS while the API kept answering and the instances stayed "running".
The existing re-read guard did not help: it re-reads through the same path.
Three defences, because no single one covers every shape of the fault.
A failed kv.Get now fails the pass instead of dropping the record. Dropping one
VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads
as permission to delete every guest port in that VPC — one failed read, same
outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that
is a poison record rather than an unhealthy store, and failing on it would
wedge reconcile forever.
The port sweep now asks local OVS which taps are plugged in here and never
deletes a port that has one. A running guest's port cannot be an orphan
whatever intent says, and OVS is the one liveness signal that does not come
through the KV read path the sweep is otherwise entirely driven by. Wired from
vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the
sweep rather than deciding on intent alone.
Each sweep also refuses outright when the keep set it would delete against
holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That
is the case local taps cannot cover, because the drift sweep is leader-gated
and lists NB cluster-wide, so the leader holds no tap for another node's guest.
The cost is real: a genuinely orphaned row now survives until one live record
of its class exists. Leaking a stale row is the recoverable half.
Three existing prune tests drove their fixtures to a wholly empty intent, which
is now the refusal case; they keep one live record of the class so they still
prove the prune. The Get-failure path has no unit test — there is no
jetstream.JetStream fake in the tree and an embedded server cannot be made to
list a key it will then refuse.
* docs(oci): separate the two address quotas, and the principal that can read them
The regional reserved-public-IP ceiling is tenancy-wide and does not grow with
node count, so a cluster hits it before the per-VNIC 64, which does. Records
the commands that read both, and that a detached reserved public IP still
bills.
Reading limits or usage needs a tenancy-level grant, which cannot be
compartment-scoped the way the node's policy is. That is a second, read-only
principal kept off every node, not an addition to SpinifexOperators.
* docs(oci): give quotas their own troubleshooting section, and say we enforce none
Spinifex hardcodes no address ceiling: it attempts the allocation and maps
OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with
Oracle and nothing here is reconfigured or restarted — which is what an
operator needs to know before it bites, not after.
Separates the two quotas by how they scale: the regional reserved-public-IP
default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and
is not.
* fix(oci): follow a guest to its new node when its EIP moves
OCI delivers a public address to whichever VNIC holds the private IP behind
it, and nothing ever moved that private IP. Binding.VNICID was written once at
allocate time, as the VNIC of whichever queue-group daemon happened to serve
the AWS request, and never updated — so an address landed on the right node
only by chance, and a guest that stopped and started elsewhere went dark while
every OVN row, host route and firewall rule on the new node was correct.
ClaimLocalAddresses closes the gap from the node that should hold the address.
Local OVS decides ownership, the same authority the host EIP binder already
trusts for the same question, and UpdatePrivateIp with a new VnicId does the
move in one server-side call. The private IP OCID survives it, so the reserved
public IP attached to it follows without being touched and the customer's
address never changes.
A loop per node, not leader-gated: the question is about this host's guests,
and a node that never wins the reconcile lease would otherwise never claim the
addresses of the instances it is running.
* fix(oci): record which ENI an EIP is associated with
An EIP is allocated bare and attached by a later AssociateAddress, so the OCI
binding written at allocate time named no ENI and nothing ever filled it in.
The address then stayed on whichever node happened to serve the allocation:
the affinity pass could follow an auto-assigned address, which is allocated
with its ENI, but never an EIP — the case customers actually use.
BindOwner is on an optional OwnerTracker interface rather than Allocator,
because only OCI has a notion of an address being delivered to one node. A
static or DHCP address is on the wire everywhere its node can reach, those
allocators do not implement it, and ExternalIPAM treats a pool that does not
as a success with nothing to do.
The same gap made the release owner guard inert for EIPs: it compares against
a binding ENI that was always empty, so a stale teardown could free an address
already handed to another instance.
* fix(ec2): report the address an instance is actually reachable on
Three faults kept DescribeInstances answering with the address launch assigned,
whatever happened afterwards.
The launch-time stamp lives on the stored ec2.Instance, and the stopped path
only declined to add an address rather than removing one, so a stopped instance
went on advertising an address it had released and that may already belong to
someone else. The projection now clears it, and the NIC association with it,
working on its copy so a restart can still report the address.
The running path filled the public address only when it was nil, which the
stamp never is, so an EIP could not be reported however correctly it was
recorded. The runtime address now wins.
And nothing told the rest of the system about an association: AssociateAddress
wrote neither the ENI record, which is what IMDS and the network reconcile
read, nor the instance record, which is what a describe answers from. The ENI
write is direct; the instance write is a fan-out, because the queue-group
worker that serves AssociateAddress is rarely the node running the instance.
* fix(net): take the host EIP plumbing off a node its guest has left
Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept
anything still wanted cluster-wide. A guest that moved therefore left the old
node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with
nothing that would ever remove them. On a shared segment that node goes on
answering ARP for an address another node is now serving.
The two halves now ask the same question by the same authority, in the loop
that already runs per node. An unreadable local OVS is not a yes: tearing down
a live address on a guess would fire everywhere at once, so it keeps the
binding and the next pass tries again. A binder with no chassis test behaves
exactly as before, and a centralised EIP is exempt because it hairpins through
the gateway chassis and every node plumbs it legitimately.
* fix(net): refuse to free an Elastic IP on behalf of a departing interface
Releasing an instance's public address on teardown releases it by address,
scoped to the ENI holding it. Since an association now writes the EIP onto the
instance record, that address is the EIP — so terminating an instance gave a
customer's Elastic IP back to the pool while describe-addresses still reported
it as theirs, free to be handed to another guest.
The existing ownership guard cannot catch this. It only fires when the stored
allocation already names an ENI, and an EIP is allocated bare and associated
later, so the stored ENI is empty and the guard passes straight through.
The distinction that holds is who is asking. ReleaseAddress is the address's own
owner and names no interface; instance teardown names the interface it is tearing
down. So a release that names an owner may free an auto-assigned address and
never an EIP, which is enforced in each of the three allocators.
DHCP needed the scope carried on the wire, since it releases by address and
resolves the lease afterwards. Absent on an older caller, which reads as false
and keeps the previous behaviour.
Worst on the two providers where it is not recoverable: on DHCP the lease is the
only thing holding the address and the server promises nothing about handing it
back, and on OCI the address is a reserved public IP object that the release
destroys outright.
* test(e2e): prove an Elastic IP survives a stop and start
Instance teardown releases the public address by value, and since the
association writes the EIP onto the instance record, that release aims
straight at the customer's Elastic IP. The allocator guards now refuse it,
but nothing end to end proved the address comes back on a restart.
The step goes inside EIPFlip, which already runs last on the shared public
guest for the same reason a stop is safe there: the address the guest
returns on is the allocation's, not the pool's. It runs on the static
cells and the DHCP ones alike, and on DHCP it doubles as the only outside
view of the lease still renewing while the guest is down.
* feat: release an auto-assigned public address on stop, take a new one on start
AWS lends an instance a public address for as long as it runs and gives it a
different one on the next start. Spinifex kept the address across a stop, which
holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a
lease nothing is using.
A stop now offers the address back through the instance cleaner, which refuses
anything an Elastic IP is behind — the EIP association record answers that, the
same question an ENI delete already asks — and records on the VM that the start
owes it a new one. A host drain keeps the address: Restore relaunches the guest
on the next boot, and changing its address under it is an outage nobody asked
for.
The start allocates before the launch, so an exhausted pool refuses the start
rather than booting a guest the customer cannot reach, adopts an Elastic IP
associated while the instance was stopped instead of assigning a second address
beside it, and hands the address back if the launch then fails.
e2e: a StopStartAddress stage on the public guest asserts a stopped instance
reports no address and comes back on a different one from a static pool, and a
StopStartKeepsTheEIP step asserts the opposite for an Elastic IP.
* docs: restructure the OCI guide around two deployment paths, with diagrams
The guide had grown into a numbered wall with no shape: no frontmatter, no
contents, and a reader with an OCI account had no way to tell where to start.
It now follows the multi-node install template — overview, prerequisites, then
the deployment path — and splits that path in two: Terraform for one node or
three, which is the preferred route, and a manual single-node walkthrough for
an instance that already exists. Both meet at one shared configuration section
rather than repeating it.
Two ASCII diagrams carry what the prose could not: one OCI instance with the
whole AWS surface inside it, and three nodes with the same surface distributed,
showing which plane each VNIC carries and how an address follows a guest that
moves.
The limits section was materially out of date and said so in the places that
mattered most. Multi-node is no longer experimental, public addresses are no
longer pinned to one node — each allocates on its own VNIC and the affinity
pass moves the private half to wherever the guest is — and an auto-assigned
address is now returned to OCI on stop, so a stopped fleet stops paying for
addresses. What changed is recorded rather than quietly edited.
The Terraform configuration moves in from the monorepo as
scripts/terraform/oci-spx, so the guide points at something in this repository.
Also marks the two new lifecycle test files in-package, with the reason each
one needs to be: both substitute unexported dependencies.
* docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan
Replaces the two ASCII diagrams with SVGs in the house style, one for a
single node and one for three. Renames the two deployment sections to
Terraform deployment and Manual deployment, documents what the Terraform
builds per node and every variable worth setting, and explains that the
data volume is an iSCSI attachment rather than a disk.
The manual path now covers the iSCSI login, the fstab entry and the
netplan source routing that Terraform writes, and states what br-wan is
for on a routed-NAT node: it names the VNIC for the OCI allocator and
holds the registered addresses, but is never bridged into OVS.
Opens with what Spinifex is and why OCI is worth running it on.
* fix: make gateway convergence work on a multi-node routed-NAT cluster
Three defects each stopped a multi-node cluster from ever attaching an
internet gateway, and all three produced the same uninformative symptom:
`terraform apply` sitting on `aws_internet_gateway.igw: Still creating...`
forever while every service reported healthy.
ovn-northd was given only its local OVSDB socket as a client connection.
A RAFT follower does not forward writes -- it answers "not cluster leader,
trying another server" -- so with one entry in the list, northd stops
translating Northbound intent into Southbound flows the moment database
leadership moves to another node. Measured on the OCI cluster:
SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no
chassisredirect port for any VPC created after leadership moved.
setup-ovn.sh wrote the member list only when the caller passed
--ovn-remote, which only install-node.sh did; it now derives the list from
the live RAFT membership otherwise, and says so when it finds a single
member, because that node needs a second pass once the others join.
The gateway datapath probe ran wherever the reconcile lease landed rather
than on the chassis owning the port. Both probes leave from the local
uplink, and under routed NAT the transit subnet exists identically on every
node, so a remote probe reported a healthy gateway as dead and aimed a
repair at a datapath the node does not own. GatewayPortLocal gates it.
On OCI the probe then ARPed the tenant-visible public IP, which the cloud
NATs and which appears on no interface anywhere. The reconciler now maps
the address through the same DatapathResolver the NAT rules are built from.
Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC
delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed
the destroy and stranded the VPC.
The OCI guide gains a Terraform section recording which shipped workbooks
were validated against a three-node OCI cluster and which are still open.
* docs: record the validated Terraform workbook results for the three-node OCI cluster
Six of the seven workbooks run against the OCI cluster pass, including the
three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the
clean eks-quickstart re-run, and names the two lifecycle gaps that bit during
the run so an operator recognises them rather than reading them as a broken
workbook.
* fix: route the NAT gateway's SNAT source through the datapath resolver
AddNATGateway was the one NAT path that did not map its external address to
the one that rides the wire. Where the public half of an allocation is NAT'd by
the cloud it is on no interface anywhere, so every private-subnet guest egressed
with a source that was dropped, while the gateway reported available and DNS
kept working because the VPC resolver is host-local.
The unmapped address also failed the host EIP sweep, which works in resolved
addresses, so the bind was pruned as foreign on the next pass and reinstalled on
the one after. The public half now survives as an external ID instead.
Also restructures the OCI guide around numbered steps through to running
Terraform, and moves the cloud-init and metadata sections into troubleshooting
where they are read when something is wrong rather than followed when it is not.
* fix: place a NAT gateway's OCI address on the node that forwards for it
OCI drops a packet whose source is not a registered private IP on the VNIC it
leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis.
Two independent placements decided those and nothing reconciled them: the
address was created by whichever worker served AllocateAddress, while the
chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster
that made private-subnet egress a coin flip, and the same Terraform passed or
died depending on which node answered.
The affinity pass already solves this for guests, by asking local OVS where the
tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which
is exactly and only this address. The equivalent local fact is the VPC's
chassisredirect binding, so ownership now has two answers rather than one, and
an address that is neither a guest's nor a gateway's is still left alone.
The VPC is recorded from the reconcile pass rather than from CreateNatGateway,
because the answer is the gateway chassis and reconcile is what decides it.
* fix: plumb a NAT gateway's host ingress only on its gateway chassis
A NAT gateway's address has no ENI, so neither the bind's locality test nor
the prune's could ever fire for it and every node installed its /32 route into
OVN. Latent on bare metal, where pool addresses are disjoint from the node
subnet; on a cloud that hands out the address from the nodes' own subnet it
outranks the connected route and swallows node-to-node traffic.
Give that address the same second authority the ocinet affinity pass has: the
VPC's chassisredirect binding, read from the local ovn-controller. An
unanswerable question still leaves the binding alone, so a single node whose
gateway port is not yet claimed behaves exactly as before.
The orphan EIP prune built its live set from ENI and user EIPs alone while also
sweeping host ingress, so every drift pass tore down the ingress of every live
NAT gateway and left it dark until the next host EIP tick.
* fix: ask the gateway chassis only about a NAT gateway's address
gatewayElsewhere skipped the NAT gateway marker it was named for and answered
for every spec, so on a node that is not the VPC's gateway chassis every EIP —
including a guest's, whose locality is its own port — was left unplumbed. Two
of three nodes bound nothing.
* test: assert the two private-subnet EKS workbooks in the tofu harness
eks-https-ingress and eks-gitops-argocd put their workers in private subnets
behind a NAT gateway, so a worker only registers Ready if its egress to the
cluster endpoint and to ECR survives that path. That makes them the workbooks
that catch a NAT gateway fault, and neither had an assertion, so neither could
be run at all — run_workbook calls assert_<name> by construction.
Both share eks-quickstart's readiness check, extracted unchanged. They also
name the worker size node_instance_type rather than instance_type, and tofu
rejects a -var for an undeclared variable outright, so the name is read from
the config instead of kept in a list here.
* test: count agents, not every node, when asserting EKS workers Ready
The control plane is three K3s servers, so an unfiltered node count reaches a
one-worker cluster's desired size before any worker joins — the assertion
passed on exactly the failure it exists to catch, reporting "4/1 workers
Ready". Filtering on the control-plane role label makes it 1/1, which is the
number that says a node-group worker registered.
Matters most for the two private-subnet workbooks, where the worker joining is
the whole proof that egress through the NAT gateway works.
* docs: record the two private-subnet EKS workbooks passing on OCI
This commit is contained in:
Ben Duncan
@@ -0,0 +1,80 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="836" viewBox="0 0 1020 836" role="img" aria-label="Spinifex on a single OCI instance: the instance sits inside the VCN public subnet 10.200.0.0/23; its primary VNIC carries the default route and the cluster mesh, its secondary VNIC is the br-wan bridge holding registered secondary private IPs paired with reserved public IPs; guests reach them through the spx-nat transit veth, and an iSCSI block volume backs /var/lib/spinifex.">
|
||||
<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
|
||||
<rect x="0" y="0" width="1020" height="836" rx="20" fill="#14181D" stroke="#2A2F36" stroke-width="1"/>
|
||||
<text x="28" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ON OCI — SINGLE NODE</text>
|
||||
<text x="992" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">one instance · routed NAT · real public addresses</text>
|
||||
<text x="28" y="72" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">One OCI instance, single Spinifex node</text>
|
||||
<rect x="28" y="92" width="964" height="76" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="48" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">INTERNET</text>
|
||||
<text x="972" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">AWS SDKs, SSH and guest egress all arrive here</text>
|
||||
<rect x="46" y="126" width="272.7" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="182.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli / Terraform / boto3 → :9999</text>
|
||||
<rect x="327.7" y="126" width="177.2" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="416.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">ssh → guest public IP</text>
|
||||
<rect x="513.92" y="126" width="218.1" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="623.0" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">guest egress ← same address</text>
|
||||
<line x1="510.0" y1="170" x2="510.0" y2="186" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
|
||||
<rect x="28" y="188" width="964" height="582" rx="16" fill="#EE4B12" fill-opacity="0.022" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.45" stroke-dasharray="6 5"/>
|
||||
<text x="50" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2">OCI VCN 10.200.0.0/16 · PUBLIC SUBNET 10.200.0.0/23</text>
|
||||
<text x="970" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">virtual router 10.200.0.1 · internet gateway</text>
|
||||
<rect x="48" y="232" width="924" height="520" rx="14" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
|
||||
<text x="68" y="266" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">BM.Standard.E2.64</text>
|
||||
<text x="260" y="266" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">spinifex-node-01 · FAULT-DOMAIN-1</text>
|
||||
<text x="952" y="266" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">one host = one Spinifex region</text>
|
||||
<line x1="68" y1="282" x2="952" y2="282" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
|
||||
<rect x="68" y="298" width="434.0" height="108" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="88" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">VNIC 0 — PRIMARY</text>
|
||||
<text x="482.0" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">enp0s9</text>
|
||||
<text x="88" y="356" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">10.200.0.252/23 · holds the default route</text>
|
||||
<text x="88" y="378" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">advertise IP · ovn-encap-ip · NATS · API :9999</text>
|
||||
<rect x="518.0" y="298" width="434.0" height="108" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="538.0" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">VNIC 1 — SECONDARY</text>
|
||||
<text x="932.0" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">br-wan (Linux bridge)</text>
|
||||
<text x="538.0" y="356" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">10.200.1.233/32 + ip rule → table 200</text>
|
||||
<text x="538.0" y="378" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">secondary private IPs ⇄ reserved public IPs</text>
|
||||
<line x1="735.0" y1="406" x2="735.0" y2="422" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
|
||||
<rect x="68" y="424" width="884" height="40" rx="11" fill="#1E242B" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
|
||||
<text x="88" y="449" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#EE4B12" font-weight="600" text-anchor="start">spx-nat-host ⇄ spx-nat-ovs</text>
|
||||
<text x="296" y="449" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">transit 100.127.0.0/24 · nft chain spinifex-nat-egress masquerades out br-wan</text>
|
||||
<line x1="510.0" y1="464" x2="510.0" y2="480" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
|
||||
<rect x="68" y="482" width="884" height="100" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="88" y="512" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">SPINIFEX CONTROL PLANE</text>
|
||||
<text x="932" y="512" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">every service is one binary: spx service <name></text>
|
||||
<rect x="86" y="530" width="109.0" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="140.5" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw :9999</text>
|
||||
<rect x="204.01999999999998" y="530" width="74.9" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="241.5" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
|
||||
<rect x="287.94" y="530" width="61.3" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="318.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
|
||||
<rect x="358.22" y="530" width="61.3" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="388.9" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">NATS</text>
|
||||
<rect x="428.5" y="530" width="102.2" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="479.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
|
||||
<rect x="539.7" y="530" width="102.2" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="590.8" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
|
||||
<rect x="650.9000000000001" y="530" width="95.4" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="698.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">northstar</text>
|
||||
<rect x="755.2800000000001" y="530" width="88.6" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="799.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">ui :3000</text>
|
||||
<text x="86" y="568" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">SigV4 in, NATS in the middle, QEMU/KVM · OVN/OVS · Predastore · Viperblock underneath</text>
|
||||
<line x1="248" y1="582" x2="248" y2="598" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
|
||||
<line x1="772" y1="582" x2="772" y2="598" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
|
||||
<rect x="68" y="600" width="532" height="130" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="88" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">TENANT VPCs — OVN br-int</text>
|
||||
<text x="580" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">geneve-local</text>
|
||||
<rect x="86" y="644" width="496" height="34" rx="9" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="100" y="666" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">vpc-a 10.0.0.0/16</text>
|
||||
<text x="310" y="666" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-… 10.0.1.4 → 150.230.13.131</text>
|
||||
<rect x="86" y="686" width="496" height="34" rx="9" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="100" y="708" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">vpc-b 172.31.0.0/16</text>
|
||||
<text x="310" y="708" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-… 172.31.4.9 · private, egress only</text>
|
||||
<rect x="616" y="600" width="336" height="130" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="636" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">BLOCK VOLUME — iSCSI</text>
|
||||
<text x="932" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">256 GB+</text>
|
||||
<text x="636" y="658" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">169.254.2.2:3260 → /dev/sdb</text>
|
||||
<text x="636" y="680" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">/var/lib/spinifex</text>
|
||||
<text x="636" y="700" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11.5" fill="#5B636C" font-weight="400" text-anchor="start">mounted by UUID, defaults,_netdev,nofail</text>
|
||||
<text x="636" y="718" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11.5" fill="#5B636C" font-weight="400" text-anchor="start">viperblock · predastore · JetStream</text>
|
||||
<text x="28" y="796" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">A guest's public address is a real OCI reserved public IP, attached to a secondary private IP on br-wan.</text>
|
||||
<text x="28" y="814" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">Inbound and outbound are therefore the same address, and OCI's per-VNIC source check passes.</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 15 KiB |
@@ -0,0 +1,116 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="796" viewBox="0 0 1020 796" role="img" aria-label="Spinifex across three OCI instances: all three sit inside the VCN public subnet 10.200.0.0/23, one per fault domain. Each carries a primary VNIC for the cluster mesh and a br-wan secondary VNIC holding its reserved public addresses, its own iSCSI block volume, and its share of the guests. Geneve, NATS JetStream, predastore erasure coding and the OVN raft join them east-west.">
|
||||
<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
|
||||
<rect x="0" y="0" width="1020" height="796" rx="20" fill="#14181D" stroke="#2A2F36" stroke-width="1"/>
|
||||
<text x="28" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ON OCI — THREE NODES</text>
|
||||
<text x="992" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">three fault domains · one region · one API endpoint</text>
|
||||
<text x="28" y="72" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">Three OCI instances, redundant multi-node install</text>
|
||||
<rect x="28" y="92" width="964" height="76" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="48" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">INTERNET</text>
|
||||
<text x="972" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">one endpoint, whichever node answers</text>
|
||||
<rect x="46" y="126" width="272.7" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="182.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli / Terraform / boto3 → :9999</text>
|
||||
<rect x="327.7" y="126" width="177.2" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="416.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">ssh → guest public IP</text>
|
||||
<rect x="513.92" y="126" width="218.1" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="623.0" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">guest egress ← same address</text>
|
||||
<line x1="510.0" y1="170" x2="510.0" y2="186" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
|
||||
<rect x="28" y="188" width="964" height="464" rx="16" fill="#EE4B12" fill-opacity="0.022" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.45" stroke-dasharray="6 5"/>
|
||||
<text x="50" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2">OCI VCN 10.200.0.0/16 · PUBLIC SUBNET 10.200.0.0/23</text>
|
||||
<text x="970" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">virtual router 10.200.0.1 · internet gateway</text>
|
||||
<rect x="48.0" y="232" width="297.3" height="272" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
|
||||
<text x="66.0" y="262" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="800" text-anchor="start">spinifex-node-01</text>
|
||||
<text x="327.3333333333333" y="262" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.85">FAULT-DOMAIN-1</text>
|
||||
<text x="66.0" y="280" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="start">BM.Standard.E2.64</text>
|
||||
<line x1="66.0" y1="292" x2="327.3333333333333" y2="292" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
|
||||
<text x="66.0" y="314" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 0 10.200.0.252</text>
|
||||
<text x="66.0" y="330" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">mesh · advertise · encap</text>
|
||||
<text x="66.0" y="352" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 1 br-wan 10.200.1.233/32</text>
|
||||
<text x="66.0" y="368" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">public IP pairs · table 200</text>
|
||||
<rect x="66.0" y="382" width="51.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="91.5" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw</text>
|
||||
<rect x="123.0" y="382" width="57.2" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="151.6" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
|
||||
<rect x="186.2" y="382" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="208.6" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
|
||||
<rect x="66.0" y="408" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="88.4" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">nats</text>
|
||||
<rect x="116.8" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="157.8" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
|
||||
<rect x="204.8" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="245.8" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
|
||||
<rect x="66.0" y="438" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="76.0" y="454" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-23d2… 10.0.1.4 → 150.230.13.131</text>
|
||||
<rect x="66.0" y="466" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="76.0" y="482" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">iSCSI /dev/sdb → /var/lib/spinifex</text>
|
||||
<line x1="345.3333333333333" y1="364" x2="361.3333333333333" y2="364" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.5" stroke-dasharray="2 4"/>
|
||||
<rect x="361.3333333333333" y="232" width="297.3" height="272" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
|
||||
<text x="379.3333333333333" y="262" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="800" text-anchor="start">spinifex-node-02</text>
|
||||
<text x="640.6666666666666" y="262" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.85">FAULT-DOMAIN-2</text>
|
||||
<text x="379.3333333333333" y="280" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="start">BM.Standard.E2.64</text>
|
||||
<line x1="379.3333333333333" y1="292" x2="640.6666666666666" y2="292" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
|
||||
<text x="379.3333333333333" y="314" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 0 10.200.0.118</text>
|
||||
<text x="379.3333333333333" y="330" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">mesh · advertise · encap</text>
|
||||
<text x="379.3333333333333" y="352" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 1 br-wan 10.200.1.96/32</text>
|
||||
<text x="379.3333333333333" y="368" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">public IP pairs · table 200</text>
|
||||
<rect x="379.3333333333333" y="382" width="51.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="404.8" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw</text>
|
||||
<rect x="436.3333333333333" y="382" width="57.2" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="464.9" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
|
||||
<rect x="499.5333333333333" y="382" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="521.9" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
|
||||
<rect x="379.3333333333333" y="408" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="401.7" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">nats</text>
|
||||
<rect x="430.1333333333333" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="471.1" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
|
||||
<rect x="518.1333333333333" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="559.1" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
|
||||
<rect x="379.3333333333333" y="438" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="389.3333333333333" y="454" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-6b91… 10.0.1.9 → 137.23.9.125</text>
|
||||
<rect x="379.3333333333333" y="466" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="389.3333333333333" y="482" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">iSCSI /dev/sdb → /var/lib/spinifex</text>
|
||||
<line x1="658.6666666666666" y1="364" x2="674.6666666666666" y2="364" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.5" stroke-dasharray="2 4"/>
|
||||
<rect x="674.6666666666666" y="232" width="297.3" height="272" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
|
||||
<text x="692.6666666666666" y="262" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="800" text-anchor="start">spinifex-node-03</text>
|
||||
<text x="954.0" y="262" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.85">FAULT-DOMAIN-3</text>
|
||||
<text x="692.6666666666666" y="280" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="start">BM.Standard.E2.64</text>
|
||||
<line x1="692.6666666666666" y1="292" x2="954.0" y2="292" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
|
||||
<text x="692.6666666666666" y="314" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 0 10.200.0.41</text>
|
||||
<text x="692.6666666666666" y="330" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">mesh · advertise · encap</text>
|
||||
<text x="692.6666666666666" y="352" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 1 br-wan 10.200.1.12/32</text>
|
||||
<text x="692.6666666666666" y="368" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">public IP pairs · table 200</text>
|
||||
<rect x="692.6666666666666" y="382" width="51.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="718.2" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw</text>
|
||||
<rect x="749.6666666666666" y="382" width="57.2" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="778.3" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
|
||||
<rect x="812.8666666666667" y="382" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="835.3" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
|
||||
<rect x="692.6666666666666" y="408" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="715.1" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">nats</text>
|
||||
<rect x="743.4666666666666" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="784.5" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
|
||||
<rect x="831.4666666666666" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="872.5" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
|
||||
<rect x="692.6666666666666" y="438" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="702.6666666666666" y="454" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-0ae4… 10.0.2.6 → 150.230.4.88</text>
|
||||
<rect x="692.6666666666666" y="466" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="702.6666666666666" y="482" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">iSCSI /dev/sdb → /var/lib/spinifex</text>
|
||||
<rect x="48" y="524" width="924" height="104" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
|
||||
<text x="68" y="554" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">EAST–WEST, INSIDE THE VCN</text>
|
||||
<text x="952" y="554" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">no public address is involved</text>
|
||||
<rect x="66" y="574" width="136.3" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="134.2" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">Geneve UDP 6081</text>
|
||||
<rect x="211.3" y="574" width="149.9" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="286.3" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">NATS JetStream R3</text>
|
||||
<rect x="370.24" y="574" width="156.8" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="448.6" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore RS(2,1)</text>
|
||||
<rect x="536.0" y="574" width="163.6" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="617.8" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">OVN Northbound raft</text>
|
||||
<rect x="708.58" y="574" width="68.1" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
|
||||
<text x="742.6" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">OVSDB</text>
|
||||
<text x="66" y="612" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">Guest-to-guest traffic rides Geneve between the nodes' primary VNICs — it never leaves the subnet and never touches br-wan.</text>
|
||||
<rect x="28" y="668" width="964" height="46" rx="12" fill="#EE4B12" fill-opacity="0.07" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.55"/>
|
||||
<text x="48" y="688" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13" fill="#F5F3EE" font-weight="700" text-anchor="start">A guest that moves takes its address with it.</text>
|
||||
<text x="48" y="705" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">The reserved public IP follows the secondary private IP to whichever node's br-wan the guest lands on — the tenant sees no change.</text>
|
||||
<text x="28" y="756" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">Three nodes is the smallest cluster that tolerates losing one: NATS JetStream keeps quorum, predastore RS(2,1)</text>
|
||||
<text x="28" y="774" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">keeps the data readable, and the OVN Northbound raft keeps a leader. One node per fault domain is what makes that real.</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 21 KiB |
@@ -143,5 +143,16 @@
|
||||
"extra": "",
|
||||
"suite": "storagefault",
|
||||
"timeout_minutes": 100
|
||||
},
|
||||
{
|
||||
"cell": 30,
|
||||
"install": "binary",
|
||||
"topo": "real-multi",
|
||||
"net": "nat",
|
||||
"host": "debian-13",
|
||||
"arch": "x86_64",
|
||||
"runner": "ci-multi",
|
||||
"extra": "nat-multi",
|
||||
"suite": ""
|
||||
}
|
||||
]
|
||||
|
||||
@@ -10,7 +10,7 @@ on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
cells:
|
||||
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection"
|
||||
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes"
|
||||
required: false
|
||||
|
||||
permissions:
|
||||
@@ -278,8 +278,31 @@ jobs:
|
||||
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
|
||||
if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
|
||||
|
||||
# natuplink proves a guest with a public IP reaches the internet. The two
|
||||
# surfaces it never touches are a private subnet's isolation and its NAT
|
||||
# gateway, and TestVPCEgressPaths is the authority on both — it skipped in
|
||||
# nat mode until the fixture learned that routed NAT carries EIPs too.
|
||||
- name: Run VPC Egress Paths in NAT mode
|
||||
if: matrix.extra == 'nat-single'
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
INVENTORY=$(tofu output -json inventory)
|
||||
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
||||
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
||||
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
||||
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
|
||||
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
||||
echo '::group::suite single/TestVPCEgressPaths'; \
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-vpcegress-nat.start'; \
|
||||
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
||||
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
||||
./single.test -test.v -test.timeout=30m -test.run '^TestVPCEgressPaths\$' \
|
||||
2>&1 | tee '${ARTIFACT_DIR}/test-vpcegress-nat.log'; \
|
||||
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
|
||||
if [ \$rc -ne 0 ]; then echo 'vpcegress-nat: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
|
||||
|
||||
- name: Run Multi Node E2E
|
||||
if: matrix.topo == 'real-multi' && matrix.suite == ''
|
||||
if: matrix.topo == 'real-multi' && matrix.suite == '' && matrix.extra == ''
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -314,6 +337,40 @@ jobs:
|
||||
if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
|
||||
done; exit \$FAIL"
|
||||
|
||||
# The same suite as the nat-single cell, run against three nodes: there it
|
||||
# proves routed NAT works at all, here that each node delivers its own
|
||||
# guests' public IPs and no others — which one node cannot show.
|
||||
- name: Run NAT Uplink Multi E2E
|
||||
if: matrix.extra == 'nat-multi'
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
INVENTORY=$(tofu output -json inventory)
|
||||
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
||||
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
||||
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
||||
|
||||
NODES_CSV=""
|
||||
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
|
||||
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
||||
ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
|
||||
NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
|
||||
done
|
||||
|
||||
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
|
||||
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
||||
if [ ! -f natuplink.test ]; then echo 'natuplink.test not in tests tar (branch predates suite) — skipping'; exit 0; fi; \
|
||||
echo '::group::suite natuplink'; \
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-natuplink.start'; \
|
||||
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
|
||||
SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
|
||||
SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
|
||||
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
||||
./natuplink.test -test.v -test.timeout=30m \
|
||||
2>&1 | tee '${ARTIFACT_DIR}/test-natuplink.log'; \
|
||||
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
|
||||
if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
|
||||
|
||||
# The dedicated suite cells only. Resolved once here rather than in each
|
||||
# step below, which would repeat the same three lines five times.
|
||||
- name: Resolve VM SSH details
|
||||
|
||||
@@ -160,14 +160,27 @@ preflight:
|
||||
# Shell suites + shellcheck for build/scripts/, the systemd-unit helpers that
|
||||
# ship on every node (unlike scripts/images/, kept in preflight: a wrong
|
||||
# restart decision here is an outage, not asset churn).
|
||||
#
|
||||
# NODE_SCRIPTS are the four that setup.sh installs into
|
||||
# /usr/local/share/spinifex on every node. They are covered here for the same
|
||||
# reason: between them they install the software, wire OVN, erase the state and
|
||||
# remove the install, so a shellcheck warning in one is an outage or a wrong
|
||||
# deletion, not a style note.
|
||||
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/node-reset.sh scripts/uninstall-spx.sh
|
||||
|
||||
test-build-scripts:
|
||||
@echo -e "\n....Running build/scripts/**/*_test.sh...."
|
||||
@for t in $$(find build/scripts -name '*_test.sh' | sort); do \
|
||||
echo "-- $$t"; \
|
||||
sh "$$t" || exit 1; \
|
||||
done
|
||||
@echo -e "\n....Running shellcheck over build/scripts/**/*.sh...."
|
||||
shellcheck -S warning $$(find build/scripts -name '*.sh' | sort)
|
||||
@echo -e "\n....Running scripts/*_test.sh...."
|
||||
@for t in $$(find scripts -maxdepth 1 -name '*_test.sh' | sort); do \
|
||||
echo "-- $$t"; \
|
||||
sh "$$t" || exit 1; \
|
||||
done
|
||||
@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
|
||||
shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
|
||||
@echo " test-build-scripts ok"
|
||||
|
||||
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
|
||||
|
||||
+40
-27
@@ -496,6 +496,10 @@ func validateChecksumFlags(checksumPath string, checksumSet bool, localFile stri
|
||||
return nil
|
||||
}
|
||||
|
||||
// catalogPinnedDigestSource records that the expected digest came from this
|
||||
// binary's catalog, so a recorded digest never reads as a fetched sums file.
|
||||
const catalogPinnedDigestSource = "catalog:pinned"
|
||||
|
||||
// importSource is the file an import is about to extract, and what it is to be verified against.
|
||||
type importSource struct {
|
||||
imageFile, imageName, localFile string
|
||||
@@ -511,10 +515,23 @@ func (src importSource) resolveDigest(out, errOut io.Writer) (ebsmetadata.ImageD
|
||||
digest := ebsmetadata.ImageDigest{Filename: filepath.Base(src.imageFile)}
|
||||
switch sourceDigestMode(src.localFile, src.checksumPath, src.skipVerify) {
|
||||
case ebsmetadata.DigestCatalog:
|
||||
if src.image.Checksum == "" || src.image.ChecksumType == "" {
|
||||
fmt.Fprintf(errOut, "Catalog entry %q is missing Checksum/ChecksumType; refusing import.\n", src.imageName)
|
||||
if src.image.ChecksumType == "" || (src.image.Checksum == "" && src.image.ChecksumDigest == "") {
|
||||
fmt.Fprintf(errOut, "Catalog entry %q is missing Checksum/ChecksumDigest/ChecksumType; refusing import.\n", src.imageName)
|
||||
return digest, errors.New("catalog entry has no checksum")
|
||||
}
|
||||
// A pinned digest is checked against this file rather than a sums file
|
||||
// fetched from the same host that served the image.
|
||||
if src.image.ChecksumDigest != "" {
|
||||
actual, err := utils.VerifyImageDigest(src.imageFile, src.image.ChecksumType, src.image.ChecksumDigest)
|
||||
if err != nil {
|
||||
printChecksumError(errOut, src.imageFile, src.imageName, src.image, err)
|
||||
return digest, err
|
||||
}
|
||||
fmt.Fprintf(out, "✅ Verified image checksum (%s) against the pinned catalog digest\n", src.image.ChecksumType)
|
||||
digest.Algorithm, digest.Value, digest.Source = src.image.ChecksumType, actual, catalogPinnedDigestSource
|
||||
digest.Verification = ebsmetadata.DigestCatalog
|
||||
return digest, nil
|
||||
}
|
||||
actual, err := utils.VerifyImageChecksum(src.imageFile, src.image.Checksum, src.image.ChecksumType)
|
||||
if err != nil {
|
||||
printChecksumError(errOut, src.imageFile, src.imageName, src.image, err)
|
||||
@@ -1283,7 +1300,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
|
||||
detected, err := admin.DetectNetwork()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "⚠️ Network auto-detection failed: %v\n", err)
|
||||
fmt.Fprintf(os.Stderr, " Use --external-mode=nat for outbound-only VMs on a non-bridgeable uplink, or specify --external-* flags manually.\n")
|
||||
fmt.Fprintf(os.Stderr, " Use --external-mode=nat on a non-bridgeable uplink, or specify --external-* flags manually.\n")
|
||||
} else {
|
||||
detectedNet = detected
|
||||
|
||||
@@ -1321,7 +1338,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
|
||||
if externalMode == "" && !cmd.Flags().Changed("external-mode") {
|
||||
if isNonBridgeableUplink(detected.WAN.Name) {
|
||||
fmt.Fprintf(os.Stderr, "\n❌ Detected WAN interface %s cannot be bridged (WiFi/cellular/PPP).\n", detected.WAN.Name)
|
||||
fmt.Fprintf(os.Stderr, " Use routed NAT mode instead (outbound-only VM networking):\n")
|
||||
fmt.Fprintf(os.Stderr, " Use routed NAT mode instead:\n")
|
||||
fmt.Fprintf(os.Stderr, " ./scripts/setup-ovn.sh --management --nat-uplink\n")
|
||||
fmt.Fprintf(os.Stderr, " spx admin init --external-mode=nat\n")
|
||||
os.Exit(1)
|
||||
@@ -1343,9 +1360,14 @@ func runAdminInit(cmd *cobra.Command, args []string) {
|
||||
// the transit segment claims externalGateway below.
|
||||
natPublicGateway := externalGateway
|
||||
if externalMode == "nat" {
|
||||
// Verified on three OCI nodes 2026-09-26: each node delivers its own
|
||||
// instances' public IPs, inbound and outbound. Still flagged because no
|
||||
// nightly cell covers multi-node nat yet.
|
||||
if nodes >= 2 {
|
||||
fmt.Fprintf(os.Stderr, "❌ Error: --external-mode=nat is single-node only (v1); use --nodes=1\n")
|
||||
os.Exit(1)
|
||||
fmt.Fprintf(os.Stderr, "⚠️ --external-mode=nat on %d nodes is NEW.\n", nodes)
|
||||
fmt.Fprintf(os.Stderr, " Public IPs are delivered by the node running the instance; default egress stays on the VPC's gateway chassis.\n")
|
||||
fmt.Fprintf(os.Stderr, " Verified by hand on three nodes, but not yet covered by a nightly cell.\n")
|
||||
fmt.Fprintf(os.Stderr, " Check with: ip route show dev spx-nat-host — exactly one node should hold each public address.\n")
|
||||
}
|
||||
if !natPublicPool && (externalBindBridge != "" || gatewayIP != "") {
|
||||
fmt.Fprintf(os.Stderr, "❌ Error: --external-bind-bridge/--gateway-ip require --external-pool or --external-source in --external-mode=nat\n")
|
||||
@@ -1675,14 +1697,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
|
||||
bootstrapSubnetId := utils.GenerateResourceID("subnet")
|
||||
bootstrapIgwId := utils.GenerateResourceID("igw")
|
||||
networkConfig.ExternalMode = externalMode
|
||||
networkConfig.PoolName = "wan"
|
||||
networkConfig.PoolSource = externalSource
|
||||
networkConfig.PoolBindBridge = externalBindBridge
|
||||
networkConfig.PoolStart = poolStart
|
||||
networkConfig.PoolEnd = poolEnd
|
||||
networkConfig.PoolGateway = externalGateway
|
||||
networkConfig.PoolGatewayIP = gatewayIP
|
||||
networkConfig.PoolPrefixLen = externalPrefixLen
|
||||
networkConfig.Pools = externalPools
|
||||
networkConfig.PoolDNSServers = dnsServers
|
||||
networkConfig.BootstrapAccountId = admin.DefaultAccountID()
|
||||
networkConfig.BootstrapVpcId = bootstrapVpcId
|
||||
@@ -1835,7 +1850,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
|
||||
fmt.Printf(" Public pool: dhcp via %s\n", externalBindBridge)
|
||||
}
|
||||
} else {
|
||||
fmt.Printf("\n📡 External networking: nat (routed, outbound-only — no public IPs/EIPs)\n")
|
||||
fmt.Printf("\n📡 External networking: nat (routed; no public pool configured yet)\n")
|
||||
}
|
||||
fmt.Printf(" Transit: %s via %s (host masquerades out any uplink)\n", host.NATTransitCIDR, host.NATTransitHostEnd)
|
||||
fmt.Printf(" Host setup: ./scripts/setup-ovn.sh --nat-uplink (run before starting services)\n")
|
||||
@@ -3199,18 +3214,11 @@ func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkCon
|
||||
settings.IPSecEnabled = nc.IPSecEnabled
|
||||
settings.ExternalMode = nc.ExternalMode
|
||||
settings.PoolDNSServers = nc.PoolDNSServers
|
||||
// Derived from the mode rather than carried in NetworkConfig, so the leader
|
||||
// and every joiner pin the same value from the same rule as single-node init.
|
||||
settings.BridgeMode = bridgeModeFor(nc.ExternalMode)
|
||||
if nc.ExternalMode != "" {
|
||||
settings.Pools = []admin.PoolData{{
|
||||
Name: nc.PoolName,
|
||||
Source: nc.PoolSource,
|
||||
BindBridge: nc.PoolBindBridge,
|
||||
Start: nc.PoolStart,
|
||||
End: nc.PoolEnd,
|
||||
Gateway: nc.PoolGateway,
|
||||
GatewayIP: nc.PoolGatewayIP,
|
||||
PrefixLen: nc.PoolPrefixLen,
|
||||
DNSServers: nc.PoolDNSServers,
|
||||
}}
|
||||
settings.Pools = nc.Pools
|
||||
}
|
||||
|
||||
settings.BootstrapAccountId = nc.BootstrapAccountId
|
||||
@@ -3911,7 +3919,12 @@ func resolveIfaceIP(iface string) string {
|
||||
func printChecksumError(w io.Writer, imageFile, imageName string, image utils.Images, err error) {
|
||||
fmt.Fprintf(w, "Image integrity verification failed: %v\n", err)
|
||||
fmt.Fprintf(w, " file: %s\n", imageFile)
|
||||
fmt.Fprintf(w, " source: %s\n", image.Checksum)
|
||||
if image.ChecksumDigest != "" {
|
||||
fmt.Fprintf(w, " source: %s (%s)\n", catalogPinnedDigestSource, image.ChecksumDigest)
|
||||
fmt.Fprintf(w, " url: %s\n", image.URL)
|
||||
} else {
|
||||
fmt.Fprintf(w, " source: %s\n", image.Checksum)
|
||||
}
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "The cached file was left in place. To re-download and retry:")
|
||||
fmt.Fprintf(w, " spx admin images import --name %s --force\n", imageName)
|
||||
|
||||
@@ -738,21 +738,45 @@ func TestSpinifexTomlTemplate_ExternalPoolStaticSource(t *testing.T) {
|
||||
assert.Contains(t, content, `gateway = "192.168.1.1"`)
|
||||
}
|
||||
|
||||
// Formation joiners must receive PoolBindBridge from the init node so the
|
||||
// Formation joiners must receive BindBridge from the init node so the
|
||||
// cluster-wide config reaches every chassis intact.
|
||||
func TestApplyNetworkConfig_PropagatesPoolBindBridge(t *testing.T) {
|
||||
settings := &admin.ConfigSettings{}
|
||||
nc := &formation.NetworkConfig{
|
||||
ExternalMode: "pool",
|
||||
PoolName: "wan",
|
||||
PoolSource: "dhcp",
|
||||
PoolBindBridge: "br-wan",
|
||||
PoolPrefixLen: 24,
|
||||
ExternalMode: "pool",
|
||||
Pools: []admin.PoolData{{
|
||||
Name: "wan", Source: "dhcp", BindBridge: "br-wan", PrefixLen: 24,
|
||||
}},
|
||||
}
|
||||
applyNetworkConfig(settings, nc)
|
||||
require.Len(t, settings.Pools, 1)
|
||||
assert.Equal(t, "dhcp", settings.Pools[0].Source)
|
||||
assert.Equal(t, "br-wan", settings.Pools[0].BindBridge)
|
||||
assert.Empty(t, settings.BridgeMode, "bridged modes stay auto-detected, only nat is pinned")
|
||||
}
|
||||
|
||||
// Routed NAT renders two pools, and the transit one carries a gateway-LRP range
|
||||
// that pool mode has no equivalent for. A joiner that received only the first,
|
||||
// or lost the range, would allocate VPC gateway addresses from nowhere.
|
||||
func TestApplyNetworkConfig_PropagatesEveryPool(t *testing.T) {
|
||||
settings := &admin.ConfigSettings{}
|
||||
nc := &formation.NetworkConfig{
|
||||
ExternalMode: "nat",
|
||||
Pools: []admin.PoolData{
|
||||
{Name: "nat-transit", Gateway: "100.127.0.1", PrefixLen: 24,
|
||||
GwLrpRangeStart: "100.127.0.16", GwLrpRangeEnd: "100.127.0.254"},
|
||||
{Name: "wan", Source: "static", Start: "192.168.1.150", End: "192.168.1.200",
|
||||
Gateway: "192.168.1.1", PrefixLen: 24},
|
||||
},
|
||||
}
|
||||
applyNetworkConfig(settings, nc)
|
||||
require.Len(t, settings.Pools, 2)
|
||||
assert.Equal(t, "nat", settings.BridgeMode,
|
||||
"a joiner that loses bridge_mode falls back to autodetection for a mode init pinned")
|
||||
assert.Equal(t, "nat-transit", settings.Pools[0].Name)
|
||||
assert.Equal(t, "100.127.0.16", settings.Pools[0].GwLrpRangeStart)
|
||||
assert.Equal(t, "100.127.0.254", settings.Pools[0].GwLrpRangeEnd)
|
||||
assert.Equal(t, "wan", settings.Pools[1].Name)
|
||||
}
|
||||
|
||||
func renderSingleNodePredastore(t *testing.T, settings admin.ConfigSettings) string {
|
||||
@@ -1405,7 +1429,36 @@ func TestImportSourceResolveDigest(t *testing.T) {
|
||||
var out, errOut bytes.Buffer
|
||||
_, err := importSource{imageFile: img, imageName: "debian-13-x86_64"}.resolveDigest(&out, &errOut)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, errOut.String(), "missing Checksum/ChecksumType")
|
||||
assert.Contains(t, errOut.String(), "missing Checksum/ChecksumDigest/ChecksumType")
|
||||
})
|
||||
|
||||
// Oracle publishes its digests inline on a web page and ships no sums file,
|
||||
// so a pinned digest is the only way its entries can import verified.
|
||||
t.Run("catalog pinned digest verifies without fetching a sums file", func(t *testing.T) {
|
||||
var out, errOut bytes.Buffer
|
||||
got, err := importSource{
|
||||
imageFile: img, imageName: "oracle-10.1-x86_64",
|
||||
image: utils.Images{ChecksumType: "sha256", ChecksumDigest: imgHex},
|
||||
}.resolveDigest(&out, &errOut)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, ebsmetadata.ImageDigest{
|
||||
Algorithm: "sha256", Value: imgHex, Source: catalogPinnedDigestSource,
|
||||
Verification: ebsmetadata.DigestCatalog, Filename: "image.raw",
|
||||
}, got)
|
||||
assert.Contains(t, out.String(), "pinned catalog digest")
|
||||
})
|
||||
|
||||
t.Run("catalog pinned digest mismatch is refused and names the url", func(t *testing.T) {
|
||||
var out, errOut bytes.Buffer
|
||||
wrong := strings.Repeat("a", 64)
|
||||
_, err := importSource{
|
||||
imageFile: img, imageName: "oracle-10.1-x86_64",
|
||||
image: utils.Images{ChecksumType: "sha256", ChecksumDigest: wrong, URL: "https://example.invalid/ol.qcow2"},
|
||||
}.resolveDigest(&out, &errOut)
|
||||
require.ErrorIs(t, err, utils.ErrChecksumMismatch)
|
||||
assert.Contains(t, errOut.String(), catalogPinnedDigestSource)
|
||||
assert.Contains(t, errOut.String(), "https://example.invalid/ol.qcow2")
|
||||
assert.FileExists(t, img, "the downloaded file must be left in place")
|
||||
})
|
||||
|
||||
t.Run("unreadable image fails the unverified hash", func(t *testing.T) {
|
||||
|
||||
@@ -1032,6 +1032,8 @@ var vpcdStartCmd = &cobra.Command{
|
||||
UnderlayMTU: clusterConfig.Network.UnderlayMTU,
|
||||
BlockedWANPorts: clusterConfig.Network.ResolvedBlockedWANPorts(),
|
||||
EgressBlockExemptVPCs: clusterConfig.Network.EgressBlockExemptVPCs,
|
||||
IMDSHostMetaIP: clusterConfig.Network.IMDSHostMetaIP,
|
||||
IMDSHostDNSIP: clusterConfig.Network.IMDSHostDNSIP,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("create vpcd service: %w", err)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: "Setting Up Your Cluster"
|
||||
seoTitle: "Set Up Your First Spinifex Cluster — Spinifex Docs"
|
||||
description: "Import an AMI, create an SSH key pair and a VPC with a public subnet, then launch your first EC2 instance and connect to it on a fresh Spinifex cluster."
|
||||
description: "Import an AMI and the RDS, ECS and EKS base images, create an SSH key pair and a VPC with a public subnet, then launch your first EC2 instance on a fresh Spinifex cluster."
|
||||
category: "Admin"
|
||||
tags:
|
||||
- setup
|
||||
@@ -17,11 +17,12 @@ resources:
|
||||
|
||||
# Setting Up Your Cluster
|
||||
|
||||
> Import an AMI, create a VPC with a public subnet, and launch your first instance.
|
||||
> Import an AMI and the system base images, create a VPC with a public subnet, and launch your first instance.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [1. Import an AMI](#1-import-an-ami)
|
||||
- [Option C: Import the system base images](#option-c-import-the-system-base-images)
|
||||
- [2. Create an SSH Key](#2-create-an-ssh-key)
|
||||
- [3. Create a VPC and Public Subnet](#3-create-a-vpc-and-public-subnet)
|
||||
- [4. Launch an Instance](#4-launch-an-instance)
|
||||
@@ -35,7 +36,7 @@ resources:
|
||||
|
||||
## Overview
|
||||
|
||||
This guide walks through the first steps on a freshly installed cluster: importing an AMI, creating an SSH key and a VPC with a public subnet, launching an instance, and connecting to it.
|
||||
This guide walks through the first steps on a freshly installed cluster: importing an AMI and the system base images RDS, ECS and EKS need, creating an SSH key and a VPC with a public subnet, launching an instance, and connecting to it.
|
||||
|
||||
## Instructions
|
||||
|
||||
@@ -82,6 +83,41 @@ Verify the import and note the AMI ID:
|
||||
AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
|
||||
```
|
||||
|
||||
### Option C: Import the system base images
|
||||
|
||||
**RDS, ECS and EKS each launch from an image of their own, and none of them is imported for you.** Until these are in the catalog, `CreateDBInstance`, `CreateCluster` and `CreateNodegroup` all fail at the point they go looking for a node image — so a cluster without them has those three services present in the API and unusable in practice. Import them once, now, and the install is functional from the start.
|
||||
|
||||
Run these one at a time rather than in parallel, so predastore is not taking four concurrent uploads:
|
||||
|
||||
```bash
|
||||
for img in spinifex-rds-postgres spinifex-rds-mariadb spinifex-ecs-node spinifex-eks-node; do
|
||||
sudo spx admin images import --name "$img" --config /etc/spinifex/spinifex.toml
|
||||
done
|
||||
```
|
||||
|
||||
| Image | Serves | Contents |
|
||||
| --- | --- | --- |
|
||||
| `spinifex-rds-postgres` | RDS, `--engine postgres` | Alpine 3.24.1 + PostgreSQL 18 + `rds-init` |
|
||||
| `spinifex-rds-mariadb` | RDS, `--engine mariadb` | Alpine 3.24.1 + MariaDB 11.8 + `rds-init` |
|
||||
| `spinifex-ecs-node` | ECS container instances | Alpine 3.21.7 + containerd + ecs-agent |
|
||||
| `spinifex-eks-node` | EKS control plane and nodegroups | Alpine 3.21.7 + K3s v1.32.5 + eks-token-webhook |
|
||||
|
||||
All four boot **BIOS**, unlike the distro images, which boot UEFI. That is a property of how they are built and nothing you need to pass.
|
||||
|
||||
If you run GPU workloads, add `spinifex-ecs-node-gpu` and `spinifex-eks-node-gpu` as well — Ubuntu 26.04 with the NVIDIA driver, selected automatically for a GPU instance type. They boot UEFI and they are large, so import them only if you have GPUs.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **These register under generic names, and that is correct.** `describe-images` shows `spinifex-eks-node` and `spinifex-ecs-node` both as `ami-alpine-3.21.7-x86_64`, and the two RDS images both as `ami-alpine-3.24.1-x86_64`. Nothing resolves them by name: ECS and EKS find theirs by the `spinifex:managed-by=ecs|eks` tag, and RDS by `spinifex:managed-by=rds` plus the `engine` and `engine-version` tags. **Verify the tags, not the names** — four rows with duplicate-looking names is what success looks like:
|
||||
>
|
||||
> ```bash
|
||||
> aws ec2 describe-images \
|
||||
> --filters 'Name=tag-key,Values=spinifex:managed-by' \
|
||||
> --query 'Images[].[ImageId,State,Tags[?Key==`spinifex:managed-by`].Value|[0],Tags[?Key==`engine`].Value|[0]]' \
|
||||
> --output text
|
||||
> ```
|
||||
|
||||
`spx admin images list --config /etc/spinifex/spinifex.toml` on a node shows the whole catalog, including the images not imported here.
|
||||
|
||||
## 2. Create an SSH Key
|
||||
|
||||
### Option A: Import an existing key
|
||||
|
||||
@@ -161,6 +161,12 @@ Both support the same AWS features: public subnets, Elastic IPs, security groups
|
||||
|
||||
## `nat` — Shared SNAT (Simple)
|
||||
|
||||
**`nat` is a supported mode, not a fallback.** It is covered by the nightly e2e suite (cells 19 and 30, single- and multi-node) and it is the only mode a cloud-hosted node can run, so the OCI deployment depends on it. Read "specialised" below as "fewer deployments need it", never as "less finished".
|
||||
|
||||
**Start with `pool` unless one of the reasons below applies.** Pool mode is the default, is what most of this document describes, and is what the AWS feature set was built against. `nat` is the specialised mode: it exists for uplinks that cannot be bridged, and it reaches AWS parity through host-side plumbing rather than by putting VM addresses on the wire.
|
||||
|
||||
Reach for `nat` when the uplink refuses to carry anything but its own MAC and its own addresses — WiFi and WWAN, and **any cloud-hosted node**, where the VNIC drops a frame whose source address is not registered to it. On such a host `nat` is not a preference, it is the only mode that works at all; pool mode's localnet needs to announce VM addresses by ARP and a cloud VNIC will not carry them. See [oci-integration](../../oci-integration/README.md).
|
||||
|
||||
All VMs share a single external IP for outbound SNAT. By default there are no public IPs, no Elastic IPs, and no inbound from WAN — all subnets behave as private subnets with internet access. On routed-NAT nodes, adding a public pool restores full public IP parity (see below).
|
||||
|
||||
> **Limitation (routed-NAT v1):** System instances (ECS/EKS/load-balancer
|
||||
@@ -244,6 +250,24 @@ gateway = "192.168.1.1"
|
||||
prefix_len = 24
|
||||
```
|
||||
|
||||
### NAT gateways in NAT mode
|
||||
|
||||
`CreateNatGateway` works here too, and draws its public address from the same public pool an EIP does. On the OVN side it is an `snat` row on the VPC router rewriting the private subnet's CIDR to that address — not the `dnat_and_snat` an EIP gets — so it is centralised on the VPC's gateway chassis by construction and there is no port or MAC to bind it to.
|
||||
|
||||
**That address needs the same host plumbing as an EIP**, and on a routed-NAT node it is the host that delivers it: the `/32` route steering it into OVN, the proxy-ARP entry answering for it on the uplink, and the FORWARD accepts. Without them an egressing private instance leaves with a source address the host holds no route back to, while the masquerade rule — which matches the transit `100.127.0.0/24` alone — never sees the packet. Spinifex binds it when the gateway is created and re-asserts it on the host EIP pass, so no operator step is involved; `ip route show dev spx-nat-host` should list the gateway's public address alongside any EIPs.
|
||||
|
||||
### NAT mode on more than one node
|
||||
|
||||
`spx admin init --external-mode=nat --nodes N` works for N > 1, and each public IP is delivered by the node running its instance rather than by one gateway node. Three things make that true, and all three are automatic:
|
||||
|
||||
- The `dnat_and_snat` row carries `external_mac` and `logical_port`, so OVN processes it on the instance's own chassis — the same mechanism pool mode has always used. It is safe here because `ext-shared` is a localnet, so the transit veth is node-local and only the node running the instance can answer for the address on it.
|
||||
- Each node installs a host `/32` for its own instances' addresses and leaves the rest alone, asking local OVS which ENIs are plugged in here. Run `ip route show dev spx-nat-host` on each node: exactly one node should hold each public address.
|
||||
- Every node's `spx-nat-host` carries the same MAC, `02:00:64:7f:00:01`. OVN keeps one MAC binding for the transit next hop across the whole cluster, so a per-node address would send every chassis's return traffic to whichever node wrote the binding last. `setup-ovn.sh` sets it and vpcd re-asserts it on every start.
|
||||
|
||||
Default egress — an instance with no public IP — is still centralised on the VPC's gateway chassis, which is unchanged and is what the gateway LRP's transit address is for.
|
||||
|
||||
**On a cloud host the address must also be registered on that node's VNIC.** The allocator runs per node and registers on the local VNIC, which is the right one exactly because the traffic now leaves there. `oci_vnic_iface` is the setting; `oci_vnic_id` is only needed to pin a pool to a specific VNIC and should not be set for this.
|
||||
|
||||
**Caveat — reaching an EIP from the spinifex host itself.** Host-sourced traffic enters OVN from the transit net, which is exempt from NAT (that is what makes the jumpbox pattern work) — so a host connection to an EIP that carries the transit source IP would skip DNAT. Spinifex stamps the EIP route with the uplink's LAN IP as source to avoid this, but if no uplink address can be determined, connect to the instance's **private IP** from the host instead. Other machines on the LAN are unaffected.
|
||||
|
||||
## Disabled (Empty/Omitted)
|
||||
@@ -263,6 +287,9 @@ VPC networking is overlay-only. No external connectivity. Instances can only com
|
||||
| DescribeInstances shows public IP | Yes | Yes | With public pool | No |
|
||||
| Admin must reserve IP range | Yes | No | Only static pool | No |
|
||||
| Needs router DHCP | No | Yes | Optional | No |
|
||||
| Works on a non-bridgeable uplink | No | No | Yes | n/a |
|
||||
| NAT gateways | Yes | Yes | With public pool | No |
|
||||
| More than one node | Yes | Yes | Yes | Yes |
|
||||
|
||||
If you start with `nat` and later need public subnets: on a bridgeable uplink switch to `pool` and define a range (or use `source = "dhcp"`); on a routed-NAT node just add a public pool alongside `nat-transit` — no data migration needed.
|
||||
|
||||
@@ -460,8 +487,8 @@ external_mode = "pool" # "pool", "nat", or "" (disabled)
|
||||
|
||||
| Value | Behavior |
|
||||
| -------------- | ----------------------------------------------------------------- |
|
||||
| `"pool"` | Full public networking — public subnets, auto-assign, Elastic IPs |
|
||||
| `"nat"` | Outbound-only SNAT — all VMs share one external IP |
|
||||
| `"pool"` | **Default.** Full public networking — public subnets, auto-assign, Elastic IPs |
|
||||
| `"nat"` | Outbound-only SNAT — all VMs share one external IP. Add a public pool for full parity. For uplinks that cannot be bridged, including every cloud host |
|
||||
| `""` / omitted | Overlay-only — no external connectivity |
|
||||
|
||||
## IP Pools: network.external_pools
|
||||
|
||||
@@ -17,18 +17,18 @@ tags:
|
||||
|
||||
## Overview
|
||||
|
||||
The platform serves **391 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
|
||||
The platform serves **389 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
|
||||
|
||||
| Service | Operations |
|
||||
|---|---:|
|
||||
| [ACM](/coverage/acm) | 9 |
|
||||
| [EC2](/coverage/ec2) | 126 |
|
||||
| [EC2](/coverage/ec2) | 127 |
|
||||
| [ECR](/coverage/ecr) | 22 |
|
||||
| [ECS](/coverage/ecs) | 31 |
|
||||
| [EKS](/coverage/eks) | 36 |
|
||||
| [ELBv2](/coverage/elbv2) | 34 |
|
||||
| [IAM](/coverage/iam) | 76 |
|
||||
| [RDS](/coverage/rds) | 26 |
|
||||
| [S3](/coverage/s3) | 26 |
|
||||
| [S3](/coverage/s3) | 23 |
|
||||
| [STS](/coverage/sts) | 5 |
|
||||
| **Total** | **391** |
|
||||
| **Total** | **389** |
|
||||
|
||||
@@ -778,6 +778,7 @@
|
||||
"DescribeVolumes",
|
||||
"DescribeVolumesModifications",
|
||||
"DescribeVpcAttribute",
|
||||
"DescribeVpcEndpoints",
|
||||
"DescribeVpcs",
|
||||
"DetachInternetGateway",
|
||||
"DetachNetworkInterface",
|
||||
@@ -906,6 +907,7 @@
|
||||
"DescribeVolumes",
|
||||
"DescribeVolumesModifications",
|
||||
"DescribeVpcAttribute",
|
||||
"DescribeVpcEndpoints",
|
||||
"DescribeVpcs",
|
||||
"DetachInternetGateway",
|
||||
"DetachNetworkInterface",
|
||||
@@ -1240,7 +1242,6 @@
|
||||
"DescribeVpcEndpointServiceConfigurations",
|
||||
"DescribeVpcEndpointServicePermissions",
|
||||
"DescribeVpcEndpointServices",
|
||||
"DescribeVpcEndpoints",
|
||||
"DescribeVpcPeeringConnections",
|
||||
"DescribeVpnConnections",
|
||||
"DescribeVpnGateways",
|
||||
@@ -3048,18 +3049,15 @@
|
||||
"DeleteObjects",
|
||||
"GetBucketLocation",
|
||||
"GetBucketTagging",
|
||||
"GetBucketVersioning",
|
||||
"GetObject",
|
||||
"HeadBucket",
|
||||
"HeadObject",
|
||||
"ListBuckets",
|
||||
"ListMultipartUploads",
|
||||
"ListObjectVersions",
|
||||
"ListObjects",
|
||||
"ListObjectsV2",
|
||||
"ListParts",
|
||||
"PutBucketTagging",
|
||||
"PutBucketVersioning",
|
||||
"PutObject",
|
||||
"UploadPart",
|
||||
"UploadPartCopy"
|
||||
@@ -3076,18 +3074,15 @@
|
||||
"DeleteObjects",
|
||||
"GetBucketLocation",
|
||||
"GetBucketTagging",
|
||||
"GetBucketVersioning",
|
||||
"GetObject",
|
||||
"HeadBucket",
|
||||
"HeadObject",
|
||||
"ListBuckets",
|
||||
"ListMultipartUploads",
|
||||
"ListObjectVersions",
|
||||
"ListObjects",
|
||||
"ListObjectsV2",
|
||||
"ListParts",
|
||||
"PutBucketTagging",
|
||||
"PutBucketVersioning",
|
||||
"PutObject",
|
||||
"UploadPart",
|
||||
"UploadPartCopy"
|
||||
@@ -3127,6 +3122,7 @@
|
||||
"GetBucketPolicyStatus",
|
||||
"GetBucketReplication",
|
||||
"GetBucketRequestPayment",
|
||||
"GetBucketVersioning",
|
||||
"GetBucketWebsite",
|
||||
"GetObjectAcl",
|
||||
"GetObjectAttributes",
|
||||
@@ -3141,6 +3137,7 @@
|
||||
"ListBucketInventoryConfigurations",
|
||||
"ListBucketMetricsConfigurations",
|
||||
"ListDirectoryBuckets",
|
||||
"ListObjectVersions",
|
||||
"PutBucketAccelerateConfiguration",
|
||||
"PutBucketAcl",
|
||||
"PutBucketAnalyticsConfiguration",
|
||||
@@ -3158,6 +3155,7 @@
|
||||
"PutBucketPolicy",
|
||||
"PutBucketReplication",
|
||||
"PutBucketRequestPayment",
|
||||
"PutBucketVersioning",
|
||||
"PutBucketWebsite",
|
||||
"PutObjectAcl",
|
||||
"PutObjectLegalHold",
|
||||
|
||||
@@ -18,7 +18,7 @@ tags:
|
||||
|
||||
## Overview
|
||||
|
||||
Spinifex implements **126 operations** in the EC2 `2016-11-15` API model.
|
||||
Spinifex implements **127 operations** in the EC2 `2016-11-15` API model.
|
||||
|
||||
### Spot Instances Are a Mock
|
||||
|
||||
@@ -110,6 +110,7 @@ Spot Instance Requests are a mock over the on-demand `RunInstances` path. A requ
|
||||
| `DescribeVolumes` |
|
||||
| `DescribeVolumesModifications` |
|
||||
| `DescribeVpcAttribute` |
|
||||
| `DescribeVpcEndpoints` |
|
||||
| `DescribeVpcs` |
|
||||
| `DetachInternetGateway` |
|
||||
| `DetachNetworkInterface` |
|
||||
|
||||
@@ -21,7 +21,7 @@ Pinned model source: `aws-sdk-go v1.55.8`.
|
||||
|
||||
## ec2
|
||||
|
||||
`2016-11-15` — 126 of 625 modelled operations implemented (20.2%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
|
||||
`2016-11-15` — 127 of 625 modelled operations implemented (20.3%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
|
||||
|
||||
### Not implemented
|
||||
|
||||
@@ -287,7 +287,6 @@ Pinned model source: `aws-sdk-go v1.55.8`.
|
||||
- `DescribeVpcEndpointServiceConfigurations`
|
||||
- `DescribeVpcEndpointServicePermissions`
|
||||
- `DescribeVpcEndpointServices`
|
||||
- `DescribeVpcEndpoints`
|
||||
- `DescribeVpcPeeringConnections`
|
||||
- `DescribeVpnConnections`
|
||||
- `DescribeVpnGateways`
|
||||
@@ -970,7 +969,7 @@ option-groups: Option groups configure engine add-ons for engines this platform
|
||||
|
||||
## s3
|
||||
|
||||
`2006-03-01` — 26 of 99 modelled operations implemented (26.3%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model.
|
||||
`2006-03-01` — 23 of 99 modelled operations implemented (23.2%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model.
|
||||
|
||||
### Not implemented
|
||||
|
||||
@@ -1003,6 +1002,7 @@ option-groups: Option groups configure engine add-ons for engines this platform
|
||||
- `GetBucketPolicy`
|
||||
- `GetBucketPolicyStatus`
|
||||
- `GetBucketReplication`
|
||||
- `GetBucketVersioning`
|
||||
- `GetBucketWebsite`
|
||||
- `GetObjectAcl`
|
||||
- `GetObjectAttributes`
|
||||
@@ -1017,6 +1017,7 @@ option-groups: Option groups configure engine add-ons for engines this platform
|
||||
- `ListBucketInventoryConfigurations`
|
||||
- `ListBucketMetricsConfigurations`
|
||||
- `ListDirectoryBuckets`
|
||||
- `ListObjectVersions`
|
||||
- `PutBucketAcl`
|
||||
- `PutBucketAnalyticsConfiguration`
|
||||
- `PutBucketCors`
|
||||
@@ -1032,6 +1033,7 @@ option-groups: Option groups configure engine add-ons for engines this platform
|
||||
- `PutBucketOwnershipControls`
|
||||
- `PutBucketPolicy`
|
||||
- `PutBucketReplication`
|
||||
- `PutBucketVersioning`
|
||||
- `PutBucketWebsite`
|
||||
- `PutObjectAcl`
|
||||
- `PutObjectLegalHold`
|
||||
|
||||
@@ -18,7 +18,7 @@ tags:
|
||||
|
||||
## Overview
|
||||
|
||||
Predastore implements **26 operations** in the S3 `2006-03-01` API model.
|
||||
Predastore implements **23 operations** in the S3 `2006-03-01` API model.
|
||||
|
||||
### Predastore serves this endpoint
|
||||
|
||||
@@ -45,18 +45,15 @@ That behaviour is measured separately, against the `ceph/s3-tests` suite Ceph RG
|
||||
| `DeleteObjects` |
|
||||
| `GetBucketLocation` |
|
||||
| `GetBucketTagging` |
|
||||
| `GetBucketVersioning` |
|
||||
| `GetObject` |
|
||||
| `HeadBucket` |
|
||||
| `HeadObject` |
|
||||
| `ListBuckets` |
|
||||
| `ListMultipartUploads` |
|
||||
| `ListObjectVersions` |
|
||||
| `ListObjects` |
|
||||
| `ListObjectsV2` |
|
||||
| `ListParts` |
|
||||
| `PutBucketTagging` |
|
||||
| `PutBucketVersioning` |
|
||||
| `PutObject` |
|
||||
| `UploadPart` |
|
||||
| `UploadPartCopy` |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -86,6 +86,9 @@ services:
|
||||
# Cluster coordination subjects.
|
||||
- "ec2.ReserveClusterNode"
|
||||
- "ec2.FinalizeClusterInstances"
|
||||
# An EIP association reaches whichever node holds the instance, which is
|
||||
# rarely the node that served the request (fan-out, no queue group).
|
||||
- "ec2.eni.public-ip-changed"
|
||||
- "spinifex.node.status"
|
||||
- "spinifex.node.vms"
|
||||
- "spinifex.nodes.discover"
|
||||
@@ -168,6 +171,8 @@ services:
|
||||
- "ebs.config.*"
|
||||
# → predastore (S3 / AMI / keypair store)
|
||||
- "s3.*"
|
||||
# → every daemon, so the one holding the instance reports the new address
|
||||
- "ec2.eni.public-ip-changed"
|
||||
# IAM lifecycle event
|
||||
- "iam.account.created"
|
||||
# → the in-guest rds-agent: set-password and the like, published on the
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# What each E2E suite proves
|
||||
|
||||
**This index exists to answer "does anything already test that?" before someone reasons it out from the code.** Comments describe intent when written and can go stale; these suites run against a live cluster and are the current answer on behaviour.
|
||||
|
||||
Suites live in `tests/e2e/<name>/` and build to `tests/e2e/_bin/<name>.test`. Every file carries `//go:build e2e`, so a default build skips them. The sets that decide what runs where are in `.github/scripts/suite-sets.sh`, and the nightly permutation matrix is `.github/e2e-cells.json`.
|
||||
|
||||
| Set | Suites |
|
||||
| --- | --- |
|
||||
| `E2E_SUITES_SINGLE` | `single iam cert eks ecs storagegrowth partialblock rds quota storagefault` |
|
||||
| `E2E_SUITES_MULTI` | `multinode lb cert quota storagefault` |
|
||||
| `E2E_SUITES_NIGHTLY_SINGLE` | `single cert iam` |
|
||||
| `E2E_SUITES_NIGHTLY_MULTI` | `multinode cert lb` |
|
||||
|
||||
The nightly permutation sets are deliberately narrower than the full ones: those cells have a ~35 minute budget and exist to prove every install / network / host-OS combination boots and serves. `eks`, `ecs`, `rds` and `storagefault` each get a dedicated cell instead.
|
||||
|
||||
## The suites
|
||||
|
||||
### `single` — the broad single-node EC2/EBS/VPC surface
|
||||
|
||||
The largest suite, 25 top-level tests. Notable entry points:
|
||||
|
||||
- **`TestVPCEgressPaths`** — **the authority on subnet egress semantics.** One VPC with a public and a private subnet, sharing two guests across four stages: `PublicSubnetEgress` (IGW egress works), `RouteBeforeSubnet` (a regression guard for an IGW route installed on the main route table before the subnet exists), the NAT Gateway rounds, and `EIPFlip` (associating an Elastic IP onto a running instance).
|
||||
- **`NATBaseline` hard-fails if a private-subnet guest reaches the internet with no NAT Gateway.** `NATGatewayUp` then proves egress appears and `NATGatewayDown` that it disappears. `SPINIFEX_VPCEGRESS_NAT_ROUNDS` repeats the up/down cycle for a soak run.
|
||||
- **This is AWS parity for private subnets.** It ran in pool mode only until 2026-09-26: the fixture gated it on `external_mode = "pool"`, on the stale premise that nat clusters have no EIPs. They do when a public pool is configured, so the gate is now "can this cluster allocate public addresses" and the nat-single cell runs this test beside `natuplink`, which never creates a private subnet.
|
||||
- `TestSGPolicyDatapath`, `TestSGReachabilityPolicy` — security groups as an actual datapath, not just API state.
|
||||
- `TestWANEgressSMTPBlock` — egress port blocking.
|
||||
- `TestInstanceMetadata` — IMDS from inside a guest.
|
||||
- `TestStopStart`, `TestEarlyRebootLiveness`, `TestGuestChurnDurability`, `TestSnapshotLifecycle`, `TestSnapshotBackedLaunch`, `TestCreateImage`, `TestVolumeLifecycle`, `TestSpotInstanceLifecycle`, `TestLaunchTemplateBoot`.
|
||||
- `TestNegativeErrorPaths` — the `8a`–`8h` AWS error-shape cases (malformed AMI ID, invalid instance type, volume in use, detach-root forbidden, and so on).
|
||||
|
||||
### `natuplink` — routed NAT (`external_mode = "nat"`), one node or three
|
||||
|
||||
Runs **on** a node, shelling out to `ip`/`iptables`/`ovn-nbctl` locally. Skips unless `spinifex.toml` says `external_mode = "nat"`. Nine sequential phases: host wiring (transit veth, `ip_forward`, NAT egress rules), config, the EIP surface (enabled or disabled depending on whether a public pool exists), default-subnet public IP mapping, OVN gateway and SNAT on the transit net, instance egress **proven via the serial console**, host-to-guest Tier 1 ingress, a second VPC getting a unique transit gateway IP, and the EIP ingress lifecycle.
|
||||
|
||||
**Phase 9 has two shapes.** On one node it is the EIP lifecycle: allocate, associate, host delivery plumbing, a TCP handshake to the address, a vpcd restart that must replay the plumbing, and disassociate tearing it down. On a cluster it is the distributed lane instead — a guest on every node, each public IP plumbed by the node running its guest and by **no other**, every transit veth carrying the one cluster-wide MAC, each `dnat_and_snat` row carrying `external_mac` and `logical_port`, inbound proven by a handshake and outbound by each guest's own console. **That lane is the regression guard for the two defects that made routed NAT work on one node only**, and neither is visible to a single-node run.
|
||||
|
||||
Phases 7 and 8 are gated on a cluster by what the datapath can actually do: the transit segment is a localnet, so host-to-guest Tier 1 ingress only exists where the VPC's gateway router port and the guest are both on this host. The VPC ingress route itself is installed on every node, so phase 8 checks it unconditionally.
|
||||
|
||||
Two nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
|
||||
|
||||
### `multinode` — behaviour that only exists on more than one node
|
||||
|
||||
`VPCSetup`, `SpansMultipleNodes`, `SpreadPlacement`, `EveryRunningInstanceReported`, `BastionSSH`, and its own NAT Gateway lane: `PreNATIsolation`, `NATGatewayInternet`, `NATCleanupOrdering`.
|
||||
|
||||
### `iam` — everything IAM and STS
|
||||
|
||||
Policy evaluation against live requests rather than unit-level: a deny applying to the next request without a new credential, resource-scoped grants naming one key pair, `aws:SourceIp` seeing the real client address, assumed-role sessions carrying role policies, and revocation stopping one key alone.
|
||||
|
||||
### `cert` — the cluster CA and TLS identity
|
||||
|
||||
CA installed and downloadable, fingerprint matches config, hostnames present in DNS SANs, OpenSSL verification, `curl` without a CA flag, and a shared issuer across nodes.
|
||||
|
||||
### `lb` — load balancers (multi-node)
|
||||
|
||||
Internal and internet-facing ALB and NLB, HTTPS listeners, listener rules, `ModifyListener`, and NLB UDP.
|
||||
|
||||
### `eks`, `ecs`, `ecr` — container services
|
||||
|
||||
`eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.
|
||||
|
||||
### `rds` — managed databases
|
||||
|
||||
The create/describe/connect path against a live cluster: a DB VM that actually boots, TLS enforced with plaintext refused, clients connecting from the same and another subnet, class changes replacing the VM while keeping overrides, deferred parameter-group attachment windows, and delete-while-creating leaving nothing behind.
|
||||
|
||||
### Storage and durability
|
||||
|
||||
- **`storagefault`** — what a guest does when its storage backend stops answering, and what stop/start does when the seal failed. **The claim is about corruption, not availability.** SIGSTOPs a cluster-wide service, so it always runs last and in its own cell. Three-node, because the RS(2,1) shard floor and cross-node takeover after a failed seal do not exist on one node.
|
||||
- **`partialblock`** — concurrent sub-block writes to the same block surviving a cold reread through NBD from a real guest.
|
||||
- **`storagegrowth`** — volume growth and image pull. Requires `external_mode=pool`; skips on a nat env, which has no EIP and therefore no guest SSH.
|
||||
- **`reboot`** — a slow guest waited for and keeping what it wrote, and a wedged guest reset once its budget expires. Proves a real reboot by requiring a *new* boot ID.
|
||||
- **`diskperf`** — guest disk performance gate, pinned fio profile on a dedicated blank volume. Also requires `external_mode=pool` for guest SSH.
|
||||
|
||||
### Other
|
||||
|
||||
- **`quota`** — every enforced dimension refuses with the same `quotaExceeded` code; a launch is charged for every instance it asks for; one instance wider than the cap is refused.
|
||||
- **`gpu`** — device visible in the guest, driver access, launch.
|
||||
- **`bedrock`** — `ListFoundationModels`/`GetFoundationModel` against a running cluster. Wire-format parity is unit-tested, not here.
|
||||
- **`ddil`** — Denied, Disrupted, Intermittent, Limited network scenarios.
|
||||
- **`teardown`** — sweeps leftovers.
|
||||
- **`harness`** — shared primitives, not a suite. `harness/vpc_egress.go` carries the VPC/subnet/IGW/NAT-gateway fixtures the egress scenarios build on.
|
||||
- **`manifest-check`, `manifest-lint`** — manifest validation, not cluster behaviour.
|
||||
|
||||
## Keeping this honest
|
||||
|
||||
Add a row when a suite is added, and correct one when a suite's claim changes. An index that has drifted is worse than none, because it will be trusted. If a suite's package comment and this file disagree, check the test bodies — both are prose and either can be stale.
|
||||
@@ -160,6 +160,7 @@ require (
|
||||
github.com/nats-io/nuid v1.0.1 // indirect
|
||||
github.com/opencontainers/go-digest v1.0.0 // indirect
|
||||
github.com/opencontainers/image-spec v1.1.1 // indirect
|
||||
github.com/oracle/oci-go-sdk/v65 v65.126.0 // indirect
|
||||
github.com/pierrec/lz4/v4 v4.1.18 // indirect
|
||||
github.com/pkg/errors v0.9.1 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
|
||||
|
||||
@@ -392,6 +392,8 @@ github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJw
|
||||
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
|
||||
github.com/opencontainers/runtime-spec v1.3.0 h1:YZupQUdctfhpZy3TM39nN9Ika5CBWT5diQ8ibYCRkxg=
|
||||
github.com/opencontainers/runtime-spec v1.3.0/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0=
|
||||
github.com/oracle/oci-go-sdk/v65 v65.126.0 h1:RuV0MEcLOOgNOBadYbbkUQriCK4Gm5348F/GdWvYPcI=
|
||||
github.com/oracle/oci-go-sdk/v65 v65.126.0/go.mod h1:Pzy+BpgkDesvGZXEHgslwhIYobHCPHg6wRta1mWnlqQ=
|
||||
github.com/ovn-kubernetes/libovsdb v0.8.1 h1:M2J8bcJt5mXCom0HqzfEtuHkT80CTSQRcYG7acT8gf4=
|
||||
github.com/ovn-kubernetes/libovsdb v0.8.1/go.mod h1:ZlnHLzagmLOSvyd9qfxBIZp6wOSOw0IsRsc+6lNUGbU=
|
||||
github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
|
||||
|
||||
@@ -12,6 +12,19 @@ Node installation, dev-environment lifecycle, guest image builds, and verificati
|
||||
| `dev-install.sh` | Full local dev setup through the production installer: builds from source, packs a tarball, runs `setup.sh`, initialises the cluster, and starts services under systemd. |
|
||||
| `clone-deps.sh` | Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
|
||||
|
||||
## Teardown
|
||||
|
||||
Both ship to `/usr/local/share/spinifex/` on every node, so they are available on a host installed with `curl … | bash` and not only from a checkout. They differ in what they take away:
|
||||
|
||||
| Script | What it does |
|
||||
|---|---|
|
||||
| `node-reset.sh` | Clears **state** and leaves the node installed: stops services, destroys guests, wipes `/etc/spinifex`, `/var/lib/spinifex`, `/var/log/spinifex` and the OVN databases, and removes the OVS bridges. The node is then ready for a fresh `spx admin init`. `--keep-data` preserves `/var/lib/spinifex`; `SPX_RESET_CONFIRM=reset` runs it unattended. |
|
||||
| `uninstall-spx.sh` | Removes the **software** — the inverse of `setup.sh`. Delegates the state teardown above to `node-reset.sh` rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the `systemd-networkd` units, the host firewall policy and its loaded nft table, and the Spinifex CA from the host trust store. Verifies afterwards and exits non-zero naming anything that survived. |
|
||||
|
||||
`uninstall-spx.sh` keeps `/var/lib/spinifex` and every apt package by default — uninstalling software should not destroy data, and a `curl … | bash` install lands on hosts that were already using openvswitch, libvirt and chrony. `--purge-data` and `--purge-deps` opt into each, and both confirm separately even under `--yes` (`SPX_PURGE_CONFIRM=destroy` for unattended data removal). `--dry-run` prints the whole plan and touches nothing.
|
||||
|
||||
**It refuses on a multi-node cluster, and `--yes` does not override that.** Uninstalling one node of a running cluster strands its guests and leaves the survivors reconciling against a chassis that will never answer. Run `spx admin cluster shutdown` first, then the uninstaller on each node.
|
||||
|
||||
## Dev environment
|
||||
|
||||
| Script | What it does |
|
||||
|
||||
@@ -29,6 +29,8 @@ trap 'rm -rf "$STAGING"' EXIT
|
||||
cp "$PROJECT_ROOT/bin/spx" "$STAGING/spx"
|
||||
cp "$MULGA_ROOT/viperblock/lib/nbdkit-viperblock-plugin.so" "$STAGING/"
|
||||
cp "$PROJECT_ROOT/scripts/setup-ovn.sh" "$STAGING/"
|
||||
cp "$PROJECT_ROOT/scripts/node-reset.sh" "$STAGING/"
|
||||
cp "$PROJECT_ROOT/scripts/uninstall-spx.sh" "$STAGING/"
|
||||
mkdir -p "$STAGING/systemd"
|
||||
cp "$PROJECT_ROOT/build/systemd/"* "$STAGING/systemd/"
|
||||
mkdir -p "$STAGING/scripts"
|
||||
|
||||
+66
-21
@@ -18,9 +18,15 @@
|
||||
# The first host is the init node; the rest join it.
|
||||
#
|
||||
# Options:
|
||||
# --external-pool A-B Public IP range for instances (required)
|
||||
# --external-gateway IP Gateway for that range (required)
|
||||
# --external-prefix-len N Prefix length of the range's subnet (required)
|
||||
# --external-mode M pool (default) or nat. pool bridges a WAN NIC and
|
||||
# needs the three flags below. nat routes through the
|
||||
# spx-nat veth and takes none of them — it is the only
|
||||
# mode a cloud VNIC can carry, and its public
|
||||
# addresses come from a provider pool added to
|
||||
# spinifex.toml, not from this script.
|
||||
# --external-pool A-B Public IP range for instances (required in pool mode)
|
||||
# --external-gateway IP Gateway for that range (required in pool mode)
|
||||
# --external-prefix-len N Prefix length of the range's subnet (required in pool mode)
|
||||
# --external-iface NAME WAN NIC for br-external (default: auto-detected)
|
||||
# --node-names A,B,C Cluster node names (default: node1..nodeN)
|
||||
# --lan-bridge NAME Bridge carrying internal cluster traffic
|
||||
@@ -67,6 +73,7 @@ SSH_USER="${SSH_USER:-spinifex}"
|
||||
IDENTITY=""
|
||||
HOSTS=()
|
||||
NODE_NAMES=()
|
||||
EXT_MODE="pool"
|
||||
EXT_POOL=""
|
||||
EXT_GATEWAY=""
|
||||
EXT_PREFIX=""
|
||||
@@ -111,6 +118,7 @@ FIREWALL_PEERS="/etc/spinifex/firewall/peers.nft"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--external-mode) EXT_MODE="$2"; shift 2 ;;
|
||||
--external-pool) EXT_POOL="$2"; shift 2 ;;
|
||||
--external-gateway) EXT_GATEWAY="$2"; shift 2 ;;
|
||||
--external-prefix-len) EXT_PREFIX="$2"; shift 2 ;;
|
||||
@@ -138,7 +146,7 @@ while [[ $# -gt 0 ]]; do
|
||||
--no-firewall) MANAGE_FIREWALL=false; shift ;;
|
||||
--yes|-y) ASSUME_YES=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) sed -n '2,61p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
-h|--help) sed -n '2,67p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
-*) echo "ERROR: unknown option: $1" >&2; exit 2 ;;
|
||||
*) HOSTS+=("$1"); shift ;;
|
||||
esac
|
||||
@@ -164,13 +172,30 @@ if [ "$N" -eq 2 ]; then
|
||||
log " for a fault-tolerant deployment."
|
||||
fi
|
||||
|
||||
for required in "--external-pool:$EXT_POOL" "--external-gateway:$EXT_GATEWAY" "--external-prefix-len:$EXT_PREFIX"; do
|
||||
[ -n "${required#*:}" ] || fail "${required%%:*} is required.
|
||||
case "$EXT_MODE" in
|
||||
pool|nat) ;;
|
||||
*) fail "--external-mode must be pool or nat, got: $EXT_MODE" ;;
|
||||
esac
|
||||
|
||||
# Routed mode's own addresses are the fixed transit /24 the veth carries, and any
|
||||
# public addresses come from a pool in spinifex.toml — a provider pool on a cloud
|
||||
# host. There is nothing for these three flags to describe, so passing them is a
|
||||
# misunderstanding worth naming rather than ignoring.
|
||||
if [ "$EXT_MODE" = "nat" ]; then
|
||||
for unused in "--external-pool:$EXT_POOL" "--external-gateway:$EXT_GATEWAY" "--external-prefix-len:$EXT_PREFIX"; do
|
||||
[ -z "${unused#*:}" ] || fail "${unused%%:*} is a pool-mode flag and --external-mode=nat ignores it.
|
||||
Routed mode addresses instances from the transit /24; public addresses come
|
||||
from a [[network.external_pools]] entry in spinifex.toml."
|
||||
done
|
||||
else
|
||||
for required in "--external-pool:$EXT_POOL" "--external-gateway:$EXT_GATEWAY" "--external-prefix-len:$EXT_PREFIX"; do
|
||||
[ -n "${required#*:}" ] || fail "${required%%:*} is required in --external-mode=pool.
|
||||
Instances get their public addresses from this range, and there is no safe
|
||||
default to guess. Example:
|
||||
--external-pool 216.218.163.101-216.218.163.110 \\
|
||||
--external-gateway 216.218.163.97 --external-prefix-len 27"
|
||||
done
|
||||
done
|
||||
fi
|
||||
|
||||
case "$IPSEC" in
|
||||
on) IPSEC_ENABLED=true ;;
|
||||
@@ -178,9 +203,11 @@ case "$IPSEC" in
|
||||
*) fail "--ipsec must be on or off, got: $IPSEC" ;;
|
||||
esac
|
||||
|
||||
[[ "$EXT_POOL" =~ ^[0-9.]+-[0-9.]+$ ]] || fail "--external-pool must be START-END, got: $EXT_POOL"
|
||||
[[ "$EXT_PREFIX" =~ ^[0-9]+$ ]] && [ "$EXT_PREFIX" -ge 1 ] && [ "$EXT_PREFIX" -le 32 ] ||
|
||||
fail "--external-prefix-len must be 1-32, got: $EXT_PREFIX"
|
||||
if [ "$EXT_MODE" = "pool" ]; then
|
||||
[[ "$EXT_POOL" =~ ^[0-9.]+-[0-9.]+$ ]] || fail "--external-pool must be START-END, got: $EXT_POOL"
|
||||
[[ "$EXT_PREFIX" =~ ^[0-9]+$ ]] && [ "$EXT_PREFIX" -ge 1 ] && [ "$EXT_PREFIX" -le 32 ] ||
|
||||
fail "--external-prefix-len must be 1-32, got: $EXT_PREFIX"
|
||||
fi
|
||||
|
||||
AZ="${AZ:-${REGION}a}"
|
||||
|
||||
@@ -337,7 +364,11 @@ for i in $(seq 0 $((N - 1))); do
|
||||
"$role" "${HOSTS[$i]}" "${NODE_NAMES[$i]}" "${WAN_IPS[$i]}" "${LAN_IPS[$i]}" "${VPC_IPS[$i]}"
|
||||
done
|
||||
echo ""
|
||||
echo " Instance pool: $EXT_POOL via $EXT_GATEWAY/$EXT_PREFIX"
|
||||
if [ "$EXT_MODE" = "nat" ]; then
|
||||
echo " External: routed nat — transit /24 on spx-nat, public addresses from spinifex.toml"
|
||||
else
|
||||
echo " Instance pool: $EXT_POOL via $EXT_GATEWAY/$EXT_PREFIX"
|
||||
fi
|
||||
echo ""
|
||||
if $WIPE; then
|
||||
echo " --wipe: this DESTROYS, on ALL $N hosts INCLUDING ${HOSTS[0]}:"
|
||||
@@ -505,6 +536,12 @@ done
|
||||
# --recreate-db means no stale chassis row survives to be orphaned by it.
|
||||
log "building the OVN database ($([ "$DB_NODES" -eq 3 ] && echo "RAFT across 3" || echo standalone))"
|
||||
|
||||
# Without this, setup-ovn.sh auto-detects the uplink, finds a physical NIC it
|
||||
# refuses to bridge, and stops with a menu. Routed mode wants none of that: the
|
||||
# transit veth is the uplink, and --nat-uplink is what builds and keeps it.
|
||||
OVN_UPLINK_ARGS=""
|
||||
[ "$EXT_MODE" = "nat" ] && OVN_UPLINK_ARGS="--nat-uplink"
|
||||
|
||||
if [ "$DB_NODES" -eq 3 ]; then
|
||||
# Computed before the database nodes are built, not after, because they need
|
||||
# it too. setup-ovn.sh writes ovn-northd's NB/SB connections from this list
|
||||
@@ -520,7 +557,7 @@ if [ "$DB_NODES" -eq 3 ]; then
|
||||
--db-cluster-local-addr=${LAN_IPS[0]} \
|
||||
--lan-addr=${LAN_IPS[0]} \
|
||||
--ovn-remote=$OVN_REMOTE \
|
||||
--recreate-db \
|
||||
--recreate-db $OVN_UPLINK_ARGS \
|
||||
--encap-ip=${VPC_IPS[0]}" || fail "${HOSTS[0]}: could not create the OVN database cluster"
|
||||
log " ${HOSTS[0]} created the database cluster"
|
||||
|
||||
@@ -531,14 +568,14 @@ if [ "$DB_NODES" -eq 3 ]; then
|
||||
--db-cluster-remote-addr=${LAN_IPS[0]} \
|
||||
--lan-addr=${LAN_IPS[$i]} \
|
||||
--ovn-remote=$OVN_REMOTE \
|
||||
--recreate-db \
|
||||
--recreate-db $OVN_UPLINK_ARGS \
|
||||
--encap-ip=${VPC_IPS[$i]}" || fail "${HOSTS[$i]}: could not join the OVN database cluster"
|
||||
log " ${HOSTS[$i]} joined the database cluster"
|
||||
done
|
||||
else
|
||||
on "${HOSTS[0]}" "sudo $SETUP_OVN --management \
|
||||
--node-name=${NODE_NAMES[0]} \
|
||||
--lan-addr=${LAN_IPS[0]} \
|
||||
--lan-addr=${LAN_IPS[0]} $OVN_UPLINK_ARGS \
|
||||
--encap-ip=${VPC_IPS[0]}" || fail "${HOSTS[0]}: setup-ovn.sh failed"
|
||||
log " ${HOSTS[0]} is running a standalone database"
|
||||
|
||||
@@ -549,7 +586,7 @@ fi
|
||||
for i in $(seq "$DB_NODES" $((N - 1))); do
|
||||
on "${HOSTS[$i]}" "sudo $SETUP_OVN \
|
||||
--node-name=${NODE_NAMES[$i]} \
|
||||
--ovn-remote=$OVN_REMOTE \
|
||||
--ovn-remote=$OVN_REMOTE $OVN_UPLINK_ARGS \
|
||||
--encap-ip=${VPC_IPS[$i]}" || fail "${HOSTS[$i]}: setup-ovn.sh failed"
|
||||
log " ${HOSTS[$i]} attached as a compute node"
|
||||
done
|
||||
@@ -574,12 +611,16 @@ init_args=(
|
||||
--region "$REGION"
|
||||
--az "$AZ"
|
||||
--token-ttl "$TOKEN_TTL"
|
||||
--external-mode=pool
|
||||
--external-source=static
|
||||
--external-pool="$EXT_POOL"
|
||||
--external-gateway="$EXT_GATEWAY"
|
||||
--external-prefix-len="$EXT_PREFIX"
|
||||
--external-mode="$EXT_MODE"
|
||||
)
|
||||
if [ "$EXT_MODE" = "pool" ]; then
|
||||
init_args+=(
|
||||
--external-source=static
|
||||
--external-pool="$EXT_POOL"
|
||||
--external-gateway="$EXT_GATEWAY"
|
||||
--external-prefix-len="$EXT_PREFIX"
|
||||
)
|
||||
fi
|
||||
[ -n "$EXT_IFACE" ] && init_args+=(--external-iface="$EXT_IFACE")
|
||||
[ -n "$EMAIL" ] && init_args+=(--email="$EMAIL")
|
||||
|
||||
@@ -931,4 +972,8 @@ if $RUN_SMOKE; then
|
||||
fi
|
||||
|
||||
echo ""
|
||||
log "cluster formed: $N nodes, pool $EXT_POOL"
|
||||
if [ "$EXT_MODE" = "nat" ]; then
|
||||
log "cluster formed: $N nodes, routed nat"
|
||||
else
|
||||
log "cluster formed: $N nodes, pool $EXT_POOL"
|
||||
fi
|
||||
|
||||
+10
-2
@@ -358,8 +358,16 @@ fi
|
||||
# tree holding its own keys and configs. It is not user data — it is wreckage
|
||||
# from a misdetected layout — and leaving it invites a later dev-mode run to
|
||||
# adopt it.
|
||||
DEV_ROOT="$(getent passwd spinifex | cut -d: -f6)/spinifex"
|
||||
if [ -n "${DEV_ROOT#/spinifex}" ] && [ -d "$DEV_ROOT" ]; then
|
||||
#
|
||||
# `|| true` is load bearing. setup.sh creates the *group* spinifex and the eight
|
||||
# spinifex-* service users, never a user named spinifex — that account comes from
|
||||
# the ISO's interactive login. So on any curl|bash-installed node `getent passwd
|
||||
# spinifex` exits 2, and under `set -euo pipefail` that took the whole script
|
||||
# down at the last step, after the wipe had already happened. Prod and dev-prod
|
||||
# are ISO-installed, which is why it stayed hidden.
|
||||
DEV_HOME="$(getent passwd spinifex | cut -d: -f6 || true)"
|
||||
DEV_ROOT="${DEV_HOME}/spinifex"
|
||||
if [ -n "$DEV_HOME" ] && [ -d "$DEV_ROOT" ]; then
|
||||
log "removing the dev-layout fallback tree at $DEV_ROOT"
|
||||
run sudo rm -rf "$DEV_ROOT"
|
||||
fi
|
||||
|
||||
+294
-17
@@ -20,8 +20,10 @@
|
||||
# --dhcp Obtain gateway IP via DHCP on the WAN bridge interface
|
||||
# --nat-uplink Routed NAT mode: no WAN NIC is bridged. Creates br-ext
|
||||
# with a transit veth (spx-nat-host 100.127.0.1/24) and
|
||||
# host masquerade rules; VMs get outbound-only WAN over
|
||||
# any uplink (ethernet, WiFi, cellular, PPP). Pair with
|
||||
# host masquerade rules; VMs reach the WAN over any
|
||||
# uplink (ethernet, WiFi, cellular, PPP), and EIPs work
|
||||
# once a public pool is configured. This is the only
|
||||
# mode that works on a cloud VNIC. Pair with
|
||||
# `spx admin init --external-mode=nat`.
|
||||
# --mgmt-bridge=NAME OVS bridge for system-instance control plane (default: br-mgmt)
|
||||
# --mgmt-cidr=CIDR IPv4 CIDR to assign on the mgmt bridge (default: 10.15.8.1/24)
|
||||
@@ -48,6 +50,15 @@
|
||||
# (which is every node — the ovn-central package starts
|
||||
# one on install). DISCARDS ALL LOGICAL NETWORK STATE:
|
||||
# safe on a fresh node, destroys every VPC on a live one.
|
||||
# --teardown Remove the host wiring this script builds — veth pairs,
|
||||
# systemd-networkd persistence, routed-NAT egress rules,
|
||||
# the OVS bridges and the ovn-* external_ids — and exit.
|
||||
# The way back to a node this script can be re-run on
|
||||
# cleanly, because a second run only unpicks the mode it
|
||||
# is switching away from. Leaves the NB/SB databases and
|
||||
# every VPC in them, /etc/spinifex, and the system-id;
|
||||
# erasing those is node-reset.sh and uninstall-spx.sh.
|
||||
# --dry-run Print what --teardown would do and change nothing.
|
||||
#
|
||||
# Cluster Sizing — which DB topology to run:
|
||||
#
|
||||
@@ -102,7 +113,7 @@
|
||||
# # No WAN bridge (overlay-only, no public subnet):
|
||||
# ./scripts/setup-ovn.sh --management --encap-ip=10.0.0.1
|
||||
#
|
||||
# # Non-bridgeable uplink (WiFi/cellular) — routed NAT, outbound-only VMs:
|
||||
# # Non-bridgeable uplink (WiFi/cellular) — routed NAT:
|
||||
# ./scripts/setup-ovn.sh --management --nat-uplink
|
||||
#
|
||||
# # Standalone management node, NB/SB client listeners also on the LAN plane:
|
||||
@@ -142,6 +153,10 @@ OVN_DBDIR="${OVN_DBDIR:-/var/lib/ovn}"
|
||||
# system-id themselves before / after invoking setup-ovn.sh — they
|
||||
# don't need this script to do it for them.
|
||||
NODE_NAME=""
|
||||
# --teardown removes the host wiring this script builds, so a re-run starts
|
||||
# from a clean node instead of compounding whatever the last run left.
|
||||
TEARDOWN=false
|
||||
DRY_RUN=false
|
||||
|
||||
# Parse arguments
|
||||
for arg in "$@"; do
|
||||
@@ -162,6 +177,8 @@ for arg in "$@"; do
|
||||
--db-cluster-remote-addr=*) DB_CLUSTER_REMOTE_ADDR="${arg#*=}" ;;
|
||||
--recreate-db) RECREATE_DB=true ;;
|
||||
--node-name=*) NODE_NAME="${arg#*=}" ;;
|
||||
--teardown) TEARDOWN=true ;;
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
--help|-h)
|
||||
sed -n '3,/^set -e/{/^set -e/!p}' "$0"
|
||||
exit 0
|
||||
@@ -173,6 +190,127 @@ for arg in "$@"; do
|
||||
esac
|
||||
done
|
||||
|
||||
# --- Teardown ---
|
||||
#
|
||||
# Removes the host wiring this script builds and nothing else. A node that has
|
||||
# been set up twice with different --wan-bridge values carries both shapes at
|
||||
# once, and the second run cannot unpick the first: it rips down only the mode
|
||||
# it is switching away from, and only when it gets far enough to know the mode.
|
||||
# This is the way back to a node the script can be run on cleanly.
|
||||
#
|
||||
# What it deliberately leaves: the NB/SB databases and every logical network in
|
||||
# them, /etc/spinifex, the OVS system-id, and the packages. Erasing those is
|
||||
# node-reset.sh and uninstall-spx.sh, and a teardown that took the control
|
||||
# plane's state with it could not be used to re-run this script.
|
||||
run() {
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo " would run: $*"
|
||||
return 0
|
||||
fi
|
||||
"$@"
|
||||
}
|
||||
|
||||
# True when $1 names an existing Linux (not OVS) bridge.
|
||||
is_linux_bridge() {
|
||||
[ -d "/sys/class/net/$1/bridge" ]
|
||||
}
|
||||
|
||||
# Prints the bridge $1 is enslaved to, or nothing.
|
||||
iface_master() {
|
||||
local m
|
||||
m=$(ip -o link show "$1" 2>/dev/null | sed -n 's/.* master \([^ ]*\) .*/\1/p')
|
||||
printf '%s' "$m"
|
||||
}
|
||||
|
||||
teardown() {
|
||||
echo "Tearing down the OVN host wiring built by this script"
|
||||
[ "$DRY_RUN" = true ] && echo " (dry run — nothing is changed)"
|
||||
echo ""
|
||||
|
||||
echo "Step 1: Removing veth pairs..."
|
||||
# Deleting one end deletes the peer, so the OVS port has to go first or it
|
||||
# is left in OVSDB naming a device the kernel no longer has.
|
||||
for pair in "veth-wan-ovs veth-wan-br" "spx-nat-ovs spx-nat-host"; do
|
||||
set -- $pair
|
||||
if command -v ovs-vsctl >/dev/null 2>&1; then
|
||||
run sudo ovs-vsctl --if-exists del-port "$1" 2>/dev/null || true
|
||||
fi
|
||||
if ip link show "$2" >/dev/null 2>&1; then
|
||||
run sudo ip link del "$2" 2>/dev/null || true
|
||||
echo " removed veth pair: $2 ↔ $1"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Step 2: Removing systemd-networkd persistence..."
|
||||
for f in /etc/systemd/network/05-spinifex-ovs-internal.network \
|
||||
/etc/systemd/network/10-spinifex-mgmt.network \
|
||||
/etc/systemd/network/14-spinifex-br-wan.netdev \
|
||||
/etc/systemd/network/15-spinifex-veth-wan.netdev \
|
||||
/etc/systemd/network/15-spinifex-veth-wan.network \
|
||||
/etc/systemd/network/16-spinifex-veth-wan-ovs.network \
|
||||
/etc/systemd/network/17-spinifex-nat.netdev \
|
||||
/etc/systemd/network/17-spinifex-nat.network \
|
||||
/etc/systemd/network/18-spinifex-nat-ovs.network; do
|
||||
if [ -f "$f" ]; then
|
||||
run sudo rm -f "$f"
|
||||
echo " removed $f"
|
||||
fi
|
||||
done
|
||||
run sudo networkctl reload 2>/dev/null || true
|
||||
|
||||
echo ""
|
||||
echo "Step 3: Removing routed-NAT egress rules..."
|
||||
# Drain by comment rather than by rule text: the rules were built from a
|
||||
# CIDR and an uplink this run may not know, and deleting by spec would miss
|
||||
# any installed against a different uplink.
|
||||
if command -v iptables >/dev/null 2>&1; then
|
||||
for table_chain in "nat POSTROUTING" "filter FORWARD"; do
|
||||
set -- $table_chain
|
||||
while iptables -t "$1" -S "$2" 2>/dev/null | grep -q 'spinifex-nat-egress'; do
|
||||
rule=$(iptables -t "$1" -S "$2" 2>/dev/null | grep -m1 'spinifex-nat-egress' | sed 's/^-A /-D /')
|
||||
[ -n "$rule" ] || break
|
||||
# shellcheck disable=SC2086
|
||||
run sudo iptables -t "$1" $rule 2>/dev/null || break
|
||||
echo " removed $1/$2 rule (spinifex-nat-egress)"
|
||||
[ "$DRY_RUN" = true ] && break
|
||||
done
|
||||
done
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Step 4: Removing OVS bridges and external_ids..."
|
||||
if command -v ovs-vsctl >/dev/null 2>&1; then
|
||||
# br-ext and br-int are this script's; a bridge the operator named with
|
||||
# --wan-bridge is removed too, unless it is a Linux bridge, which this
|
||||
# script never created and must not delete.
|
||||
for br in br-ext br-int "$MGMT_BRIDGE" "$WAN_BRIDGE"; do
|
||||
[ -n "$br" ] || continue
|
||||
if is_linux_bridge "$br"; then
|
||||
echo " left $br alone: it is a Linux bridge, not ours to delete"
|
||||
continue
|
||||
fi
|
||||
if sudo ovs-vsctl br-exists "$br" 2>/dev/null; then
|
||||
run sudo ovs-vsctl --if-exists del-br "$br"
|
||||
echo " removed OVS bridge: $br"
|
||||
fi
|
||||
done
|
||||
for key in ovn-bridge-mappings ovn-remote ovn-encap-ip ovn-encap-type; do
|
||||
run sudo ovs-vsctl remove Open_vSwitch . external_ids "$key" 2>/dev/null || true
|
||||
done
|
||||
echo " cleared ovn-bridge-mappings, ovn-remote, ovn-encap-ip, ovn-encap-type"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Teardown complete. The NB/SB databases, /etc/spinifex and the OVS"
|
||||
echo "system-id are untouched — re-run this script to wire the node again."
|
||||
}
|
||||
|
||||
if [ "$TEARDOWN" = true ]; then
|
||||
teardown
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# OVSDB RAFT clustering runs the NB/SB DBs, so it only applies to management
|
||||
# (DB) nodes. A remote addr without a local addr is meaningless (nothing to
|
||||
# join with). Fail loudly rather than silently starting a single-node DB.
|
||||
@@ -205,7 +343,43 @@ LINUX_BRIDGE="" # Set when WAN_BRIDGE_MODE="veth" — the Linux bridge behin
|
||||
detect_wan_bridge() {
|
||||
# If --wan-bridge was explicitly given, use it
|
||||
if [ -n "$WAN_BRIDGE" ]; then
|
||||
# A Linux bridge of this name is the veth case, not the OVS-bridge case,
|
||||
# and it is checked first because it is the only arm that can silently
|
||||
# produce a broken datapath. An OVS bridge cannot take over the name of
|
||||
# an existing Linux bridge: ovs-vsctl exits 0, OVSDB accepts the rows,
|
||||
# and ofproto then refuses with "could not add network device br-wan to
|
||||
# ofproto (File exists)" — visible only in `ovs-vsctl show`. The
|
||||
# auto-detect path below has always handled this correctly by building a
|
||||
# veth pair; it was simply unreachable whenever the operator named the
|
||||
# bridge, which is exactly what they must do when the default route is
|
||||
# on a physical NIC.
|
||||
if is_linux_bridge "$WAN_BRIDGE"; then
|
||||
LINUX_BRIDGE="$WAN_BRIDGE"
|
||||
WAN_BRIDGE="br-ext"
|
||||
WAN_BRIDGE_MODE="veth"
|
||||
echo " $LINUX_BRIDGE is a Linux bridge — linking it to OVS br-ext via a veth pair"
|
||||
if [ -n "$WAN_IFACE" ]; then
|
||||
echo " ignoring --wan-iface=$WAN_IFACE: $LINUX_BRIDGE already owns its ports"
|
||||
WAN_IFACE=""
|
||||
fi
|
||||
return
|
||||
fi
|
||||
|
||||
if [ -n "$WAN_IFACE" ]; then
|
||||
# Enslaving an interface that another bridge already owns fails the
|
||||
# same silent way. Refuse rather than build something that reports
|
||||
# success and cannot forward.
|
||||
local master
|
||||
master=$(iface_master "$WAN_IFACE")
|
||||
if [ -n "$master" ]; then
|
||||
echo "ERROR: $WAN_IFACE is already enslaved to bridge '$master'."
|
||||
echo " Adding it to an OVS bridge as well cannot work — ovs-vsctl will"
|
||||
echo " exit 0 and the datapath will refuse the device."
|
||||
echo " Either detach it first, or point --wan-bridge at '$master' to"
|
||||
echo " link that bridge to OVS with a veth pair:"
|
||||
echo " $0 --management --wan-bridge=$master"
|
||||
exit 1
|
||||
fi
|
||||
WAN_BRIDGE_MODE="direct"
|
||||
elif sudo ovs-vsctl br-exists "$WAN_BRIDGE" 2>/dev/null; then
|
||||
WAN_BRIDGE_MODE="existing"
|
||||
@@ -274,7 +448,7 @@ detect_wan_bridge() {
|
||||
echo " ./scripts/setup-ovn.sh --management --encap-ip=$wan_ip"
|
||||
echo ""
|
||||
echo " 4. Non-bridgeable uplink (WiFi/cellular/PPP) — routed NAT mode"
|
||||
echo " (VMs get outbound-only internet, no public IPs):"
|
||||
echo " (add a public pool for EIPs; without one, VMs get egress only):"
|
||||
echo " ./scripts/setup-ovn.sh --management --nat-uplink"
|
||||
echo " then: spx admin init --external-mode=nat"
|
||||
echo "============================================================"
|
||||
@@ -450,6 +624,8 @@ if [ "$MANAGEMENT" = true ]; then
|
||||
OVN_REMOTE_NB="${OVN_REMOTE//:6642/:6641}"
|
||||
OVN_CTL_OPTS="$OVN_CTL_OPTS --ovn-northd-nb-db=$OVN_REMOTE_NB --ovn-northd-sb-db=$OVN_REMOTE"
|
||||
echo " ovn-northd: NB=$OVN_REMOTE_NB SB=$OVN_REMOTE (RAFT member list)"
|
||||
else
|
||||
NORTHD_LOCAL_ONLY=true
|
||||
fi
|
||||
|
||||
if [ -n "$OVN_DB_LISTEN_OPTS" ]; then
|
||||
@@ -536,6 +712,38 @@ EOF
|
||||
echo " DB storage: clustered (NB + SB verified)"
|
||||
fi
|
||||
|
||||
# An OVSDB RAFT follower does not proxy writes — it answers "not cluster
|
||||
# leader; trying another server" — so a northd whose client list names one
|
||||
# member works only while it shares a node with the leader. When it does not,
|
||||
# NB->SB translation stops dead: nothing is logged above info, every service
|
||||
# stays active, and the only evidence is SB_Global.nb_cfg falling behind
|
||||
# NB_Global.nb_cfg. Derive the list from the live membership rather than
|
||||
# trusting the caller to have passed --ovn-remote.
|
||||
if [ "${NORTHD_LOCAL_ONLY:-false}" = true ] && [ -n "$DB_CLUSTER_LOCAL_ADDR" ]; then
|
||||
NB_MEMBERS=$(sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl \
|
||||
cluster/status OVN_Northbound 2>/dev/null \
|
||||
| grep -oE 'at tcp:[0-9.]+:' | sed 's/^at tcp://; s/:$//' | sort -u)
|
||||
NB_LIST=""
|
||||
SB_LIST=""
|
||||
for addr in $NB_MEMBERS; do
|
||||
NB_LIST="${NB_LIST:+$NB_LIST,}tcp:$addr:6641"
|
||||
SB_LIST="${SB_LIST:+$SB_LIST,}tcp:$addr:6642"
|
||||
done
|
||||
if [ "$(printf '%s\n' "$NB_MEMBERS" | grep -c .)" -gt 1 ]; then
|
||||
sudo sed -i "s|\"\$| --ovn-northd-nb-db=$NB_LIST --ovn-northd-sb-db=$SB_LIST\"|" \
|
||||
/etc/default/ovn-central
|
||||
sudo systemctl restart ovn-northd
|
||||
echo " ovn-northd: NB=$NB_LIST SB=$SB_LIST (derived from RAFT membership)"
|
||||
else
|
||||
echo " NOTE: this node is the only NB RAFT member, so ovn-northd is pointed at" >&2
|
||||
echo " itself. Once the other members join, re-run this script here (or" >&2
|
||||
echo " pass --ovn-remote with the full member list) — otherwise northd" >&2
|
||||
echo " silently stops translating NB to SB the first time NB leadership" >&2
|
||||
echo " moves off this node, and every VPC created after that never" >&2
|
||||
echo " attaches its internet gateway." >&2
|
||||
fi
|
||||
fi
|
||||
|
||||
# Wait for the Southbound DB to be serving before ovn-controller (Step 5)
|
||||
# dials it. On a single node ovn-controller races a fresh SB RAFT election
|
||||
# with no join step to absorb the window — attaching before the leader is
|
||||
@@ -580,6 +788,30 @@ else
|
||||
# enough for any ovn-nbctl without an explicit --db to confirm against the
|
||||
# wrong database — a `--wait=hv` there can never be acknowledged and burns
|
||||
# its whole timeout instead.
|
||||
#
|
||||
# Refuse to demote a node this script previously set up as a management
|
||||
# node. --management is the only record of the role and is re-asserted on
|
||||
# every run, so a re-run that forgets it silently takes the cluster's
|
||||
# database away. The symptom lands three layers off: nothing listens on
|
||||
# 6641, and vpcd crash-loops on "connection refused" with no hint that this
|
||||
# script caused it. Seen on mulga-poc at 15,061 restarts.
|
||||
#
|
||||
# The signal is /etc/default/ovn-central, not the presence of a database:
|
||||
# the ovn-central *package* creates DBs on install, so every compute node
|
||||
# has them and testing for them would refuse every legitimate compute node.
|
||||
# Only the --management path below writes OVN_CTL_OPTS to that file.
|
||||
if [ -f /etc/default/ovn-central ] && grep -q '^OVN_CTL_OPTS=' /etc/default/ovn-central 2>/dev/null; then
|
||||
echo "ERROR: this node was configured as a management node by a previous run"
|
||||
echo " (/etc/default/ovn-central holds OVN_CTL_OPTS), but --management was"
|
||||
echo " not given. Re-running without it stops and disables ovn-central and"
|
||||
echo " takes the control plane down."
|
||||
echo ""
|
||||
echo " If this is a management node, re-run with --management."
|
||||
echo " If it is genuinely being demoted to a compute node:"
|
||||
echo " sudo rm -f /etc/default/ovn-central"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if systemctl is-enabled ovn-central >/dev/null 2>&1 ||
|
||||
systemctl is-active ovn-central >/dev/null 2>&1; then
|
||||
sudo systemctl stop ovn-central ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb 2>/dev/null || true
|
||||
@@ -808,6 +1040,11 @@ NETWORK
|
||||
# the masquerade rule below is the only host-side NAT state.
|
||||
NAT_TRANSIT_CIDR="100.127.0.0/24"
|
||||
NAT_TRANSIT_GW_CIDR="100.127.0.1/24"
|
||||
# Same on every node. OVN keeps one MAC binding for the transit
|
||||
# nexthop cluster-wide, so a random per-node MAC sends every
|
||||
# chassis's egress to whichever node seeded it; the pair is
|
||||
# node-local, so one address everywhere is unambiguous.
|
||||
NAT_TRANSIT_HOST_MAC="02:00:64:7f:00:01"
|
||||
|
||||
if ! sudo ovs-vsctl br-exists "$WAN_BRIDGE" 2>/dev/null; then
|
||||
sudo ovs-vsctl --may-exist add-br "$WAN_BRIDGE"
|
||||
@@ -822,6 +1059,7 @@ NETWORK
|
||||
else
|
||||
echo " veth pair already exists: spx-nat-host ↔ spx-nat-ovs"
|
||||
fi
|
||||
sudo ip link set spx-nat-host address "$NAT_TRANSIT_HOST_MAC"
|
||||
sudo ip addr replace "$NAT_TRANSIT_GW_CIDR" dev spx-nat-host
|
||||
|
||||
# Add the OVS end to br-ext
|
||||
@@ -842,6 +1080,7 @@ NETWORK
|
||||
[NetDev]
|
||||
Name=spx-nat-host
|
||||
Kind=veth
|
||||
MACAddress=$NAT_TRANSIT_HOST_MAC
|
||||
|
||||
[Peer]
|
||||
Name=spx-nat-ovs
|
||||
@@ -870,25 +1109,56 @@ NETWORK
|
||||
echo " wrote $NAT_NETDEV + $NAT_NETWORK + $NAT_OVS_NETWORK (transit veth persists on reboot)"
|
||||
|
||||
# Kernel egress: masquerade the transit /24 out any uplink and
|
||||
# accept forwarded transit traffic even under FORWARD-policy DROP.
|
||||
# vpcd re-ensures these on every start; installing here too means
|
||||
# the wiring is testable before services run.
|
||||
sudo iptables -t nat -C POSTROUTING -s "$NAT_TRANSIT_CIDR" ! -d "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j MASQUERADE 2>/dev/null || \
|
||||
sudo iptables -t nat -A POSTROUTING -s "$NAT_TRANSIT_CIDR" ! -d "$NAT_TRANSIT_CIDR" \
|
||||
# accept forwarded transit traffic. vpcd re-ensures these on every
|
||||
# start; installing here too means the wiring is testable before
|
||||
# services run.
|
||||
#
|
||||
# Insert at the head, never append: Ubuntu ships a catch-all REJECT
|
||||
# in FORWARD, and an appended ACCEPT sits behind it and never
|
||||
# matches. Drain first so an already-appended copy is repositioned
|
||||
# rather than left stranded. Every spec carries our comment, so
|
||||
# only our own rules are ever deleted.
|
||||
nat_egress_rule() { # nat_egress_rule <table> <chain> <spec...>
|
||||
local table="$1" chain="$2"; shift 2
|
||||
local i
|
||||
for i in 1 2 3 4 5 6 7 8; do
|
||||
sudo iptables -t "$table" -D "$chain" "$@" 2>/dev/null || break
|
||||
done
|
||||
sudo iptables -t "$table" -I "$chain" 1 "$@"
|
||||
}
|
||||
nat_egress_rule nat POSTROUTING -s "$NAT_TRANSIT_CIDR" ! -d "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j MASQUERADE
|
||||
sudo iptables -C FORWARD -i spx-nat-host -s "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT 2>/dev/null || \
|
||||
sudo iptables -A FORWARD -i spx-nat-host -s "$NAT_TRANSIT_CIDR" \
|
||||
nat_egress_rule filter FORWARD -i spx-nat-host -s "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT
|
||||
sudo iptables -C FORWARD -o spx-nat-host -m conntrack --ctstate RELATED,ESTABLISHED \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT 2>/dev/null || \
|
||||
sudo iptables -A FORWARD -o spx-nat-host -m conntrack --ctstate RELATED,ESTABLISHED \
|
||||
nat_egress_rule filter FORWARD -o spx-nat-host -m conntrack --ctstate RELATED,ESTABLISHED \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT
|
||||
echo " installed masquerade + forward rules for $NAT_TRANSIT_CIDR (comment: spinifex-nat-egress)"
|
||||
;;
|
||||
esac
|
||||
|
||||
# --- Prove the bridge actually came up ---
|
||||
#
|
||||
# ovs-vsctl reports on the database, not the datapath. `add-br` and
|
||||
# `add-port` exit 0 whenever OVSDB accepts the row, and ofproto can refuse
|
||||
# the device afterwards for reasons ovs-vsctl never sees — a name already
|
||||
# taken by a Linux bridge, an interface another bridge already owns. The
|
||||
# error is recorded per-interface and surfaces only here. Without this scan
|
||||
# the script reports a successful setup and vpcd fails minutes later with a
|
||||
# message that points somewhere else entirely.
|
||||
if [ -n "$WAN_BRIDGE_MODE" ] && command -v ovs-vsctl >/dev/null 2>&1; then
|
||||
OVS_ERRORS=$(sudo ovs-vsctl --columns=name,error --format=table list Interface 2>/dev/null \
|
||||
| awk 'NR>2 && $0 ~ /could not|error/ && $0 !~ /\[\]/' || true)
|
||||
if [ -n "$OVS_ERRORS" ]; then
|
||||
echo ""
|
||||
echo "ERROR: OVS accepted the configuration but the datapath refused it:"
|
||||
echo "$OVS_ERRORS" | sed 's/^/ /'
|
||||
echo ""
|
||||
echo " The bridge exists in OVSDB and cannot forward. Nothing downstream"
|
||||
echo " will work, and vpcd will fail with an unrelated-looking error."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- DHCP: obtain gateway IP for OVN SNAT ---
|
||||
if [ "$EXTERNAL_DHCP" = true ] && [ "$WAN_BRIDGE_MODE" = "nat" ]; then
|
||||
echo ""
|
||||
@@ -1246,9 +1516,16 @@ for unit in openvswitch-switch:/var/run/openvswitch/db.sock \
|
||||
WAIT_GLOB="${unit#*:}"
|
||||
OVERRIDE_DIR="/etc/systemd/system/${UNIT}.service.d"
|
||||
sudo mkdir -p "$OVERRIDE_DIR"
|
||||
# `ExecStartPost=-` so a failure here cannot stop the unit. This drop-in goes
|
||||
# into another package's service, and tightening a socket mode is hardening,
|
||||
# not a liveness requirement — it must never be the reason openvswitch-switch
|
||||
# or ovn-controller will not start. Without the dash, removing the helper
|
||||
# (an uninstall, a partial re-run that has not reached this step yet) takes
|
||||
# OVS down across the whole host with `Failed at step EXEC`, which reads as
|
||||
# an OVS fault and is nothing of the sort.
|
||||
sudo tee "$OVERRIDE_DIR/spinifex-perms.conf" >/dev/null <<OVERRIDE
|
||||
[Service]
|
||||
ExecStartPost=$PERMS_HELPER "$WAIT_GLOB"
|
||||
ExecStartPost=-$PERMS_HELPER "$WAIT_GLOB"
|
||||
OVERRIDE
|
||||
echo " systemd override: ${UNIT}.service.d/spinifex-perms.conf"
|
||||
done
|
||||
|
||||
Executable
+140
@@ -0,0 +1,140 @@
|
||||
#!/bin/sh
|
||||
# setup-ovn_test.sh — tests for the teardown path and its documentation.
|
||||
#
|
||||
# Only --teardown --dry-run and --help are exercised. Both return before the
|
||||
# script changes anything, which matters more here than usual: the script under
|
||||
# test wires the node's datapath, and a test that ran the real teardown would
|
||||
# delete the bridges of the machine running CI.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
SETUP="$SCRIPT_DIR/setup-ovn.sh"
|
||||
TMP=$(mktemp -d)
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
fails=0
|
||||
pass() { echo " ok - $1"; }
|
||||
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
|
||||
|
||||
echo "-- setup-ovn.sh"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Help.
|
||||
# --------------------------------------------------------------------------
|
||||
help_out=$("$SETUP" --help 2>&1) && rc=0 || rc=$?
|
||||
[ "$rc" = 0 ] && pass "--help exits 0" || fail "--help exited $rc, want 0"
|
||||
|
||||
for flag in --teardown --dry-run --management --nat-uplink; do
|
||||
case "$help_out" in
|
||||
*"$flag"*) pass "--help documents $flag" ;;
|
||||
*) fail "--help does not document $flag" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --teardown is the flag an operator reaches for when a node is already
|
||||
# tangled, so the help has to say what it keeps. Somebody who reads it as
|
||||
# "removes everything" will not run it, and will hand-edit OVS instead.
|
||||
case "$help_out" in
|
||||
*"NB/SB databases"*) pass "--help says what --teardown preserves" ;;
|
||||
*) fail "--help does not say what --teardown preserves" ;;
|
||||
esac
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Teardown, dry run. Every mutation must go through run(), so a dry run can
|
||||
# report the whole plan without performing any of it.
|
||||
# --------------------------------------------------------------------------
|
||||
out=$("$SETUP" --teardown --dry-run 2>&1) && rc=0 || rc=$?
|
||||
[ "$rc" = 0 ] && pass "--teardown --dry-run exits 0" || fail "--teardown --dry-run exited $rc, want 0"
|
||||
|
||||
case "$out" in
|
||||
*"nothing is changed"*) pass "dry run says it changes nothing" ;;
|
||||
*) fail "dry run does not announce itself" ;;
|
||||
esac
|
||||
|
||||
# The four groups the teardown exists to remove. A silently skipped group is
|
||||
# the failure mode that leaves a node still tangled after a clean-looking run.
|
||||
for step in "Removing veth pairs" "Removing systemd-networkd persistence" \
|
||||
"Removing routed-NAT egress rules" "Removing OVS bridges"; do
|
||||
case "$out" in
|
||||
*"$step"*) pass "dry run covers: $step" ;;
|
||||
*) fail "dry run skips: $step" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# The OVS port goes before the veth: deleting one end of a pair deletes the
|
||||
# peer, and an OVS port left naming a device the kernel no longer has is
|
||||
# exactly the state a re-run cannot unpick.
|
||||
port_line=$(printf '%s\n' "$out" | grep -n 'del-port veth-wan-ovs' | head -1 | cut -d: -f1 || true)
|
||||
link_line=$(printf '%s\n' "$out" | grep -n 'ip link del veth-wan-br' | head -1 | cut -d: -f1 || true)
|
||||
if [ -n "$port_line" ] && [ -n "$link_line" ] && [ "$port_line" -lt "$link_line" ]; then
|
||||
pass "the OVS port is removed before its veth"
|
||||
else
|
||||
fail "veth teardown order is wrong (port=$port_line link=$link_line)"
|
||||
fi
|
||||
|
||||
# A dry run that reported a real mutation would be the worst kind of bug here.
|
||||
if printf '%s\n' "$out" | grep -E '^[[:space:]]+(sudo|ip|ovs-vsctl|iptables) ' >/dev/null 2>&1; then
|
||||
fail "dry run printed a bare command, not a 'would run:' line"
|
||||
else
|
||||
pass "every dry-run mutation is reported, not executed"
|
||||
fi
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Static checks on the teardown body.
|
||||
# --------------------------------------------------------------------------
|
||||
# Comments and echoed prose name the things the teardown deliberately spares,
|
||||
# so match against the commands alone or every such sentence reads as a hit.
|
||||
body=$(sed -n '/^teardown() {/,/^}/p' "$SETUP")
|
||||
body_cmds=$(printf '%s\n' "$body" | grep -vE '^[[:space:]]*(#|echo )' || true)
|
||||
|
||||
# Deleting a Linux bridge would take the host's own addressing with it: on a
|
||||
# cloud node br-wan is a netplan-owned Linux bridge holding the VNIC's address
|
||||
# and the default route, and this script never created it.
|
||||
case "$body" in
|
||||
*is_linux_bridge*) pass "teardown refuses to delete a Linux bridge" ;;
|
||||
*) fail "teardown does not check for a Linux bridge before del-br" ;;
|
||||
esac
|
||||
|
||||
# The databases are the control plane's state. A teardown that took them could
|
||||
# not be used for its actual purpose, which is re-running this script.
|
||||
for forbidden in "ovnnb_db" "ovnsb_db" "/etc/spinifex" "rm -rf"; do
|
||||
case "$body_cmds" in
|
||||
*"$forbidden"*) fail "teardown touches $forbidden" ;;
|
||||
*) pass "teardown leaves $forbidden alone" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The transit veth MAC.
|
||||
# --------------------------------------------------------------------------
|
||||
# OVN keeps one MAC binding for the transit nexthop across the cluster, so a
|
||||
# random per-node MAC sends every chassis's egress to whichever node seeded it.
|
||||
# The address must therefore be a literal the script sets, and it must survive a
|
||||
# reboot or the node goes dark the next time systemd-networkd recreates the pair.
|
||||
nat_mode_body=$(sed -n '/^ nat)/,/^ ;;/p' "$SETUP")
|
||||
|
||||
case "$nat_mode_body" in
|
||||
*'ip link set spx-nat-host address'*) pass "nat mode pins the transit host MAC" ;;
|
||||
*) fail "nat mode does not set a MAC on spx-nat-host" ;;
|
||||
esac
|
||||
|
||||
case "$nat_mode_body" in
|
||||
*'MACAddress=$NAT_TRANSIT_HOST_MAC'*) pass "the netdev unit persists the transit MAC" ;;
|
||||
*) fail "the netdev unit does not persist the transit MAC" ;;
|
||||
esac
|
||||
|
||||
# The script and the Go constant must agree: vpcd repairs a veth made before
|
||||
# this existed, and a disagreement would have the two fighting every pass.
|
||||
script_mac=$(printf '%s\n' "$nat_mode_body" | sed -n 's/.*NAT_TRANSIT_HOST_MAC="\([^"]*\)".*/\1/p' | head -1)
|
||||
go_mac=$(sed -n 's/.*NATTransitHostMAC = "\([^"]*\)".*/\1/p' "$SCRIPT_DIR/../spinifex/network/host/routed.go" | head -1)
|
||||
if [ -n "$script_mac" ] && [ "$script_mac" = "$go_mac" ]; then
|
||||
pass "the transit MAC matches host.NATTransitHostMAC"
|
||||
else
|
||||
fail "transit MAC disagrees: script=$script_mac go=$go_mac"
|
||||
fi
|
||||
|
||||
if [ "$fails" -ne 0 ]; then
|
||||
echo " setup-ovn.sh: $fails failure(s)"
|
||||
exit 1
|
||||
fi
|
||||
echo " setup-ovn.sh ok"
|
||||
@@ -33,6 +33,7 @@ IPSEC_STATE_HELPER="/usr/local/lib/spinifex/spinifex-set-ipsec-state"
|
||||
# the only component that knows every node's resolved planes.
|
||||
FIREWALL_DIR="/etc/spinifex/firewall"
|
||||
FIREWALL_RULES="${FIREWALL_DIR}/spinifex.nft"
|
||||
# shellcheck disable=SC2034 # part of the firewall dir layout; written by install-node.sh
|
||||
FIREWALL_PEERS="${FIREWALL_DIR}/peers.nft"
|
||||
FIREWALL_LOCAL="${FIREWALL_DIR}/local.nft"
|
||||
FIREWALL_OPEN="${FIREWALL_DIR}/open-ports.nft"
|
||||
@@ -896,6 +897,17 @@ install_files() {
|
||||
info " /usr/local/share/spinifex/setup.sh"
|
||||
fi
|
||||
|
||||
# Teardown scripts. An install delivered by `curl | bash` with no way to
|
||||
# undo it is a gap in the product, not just in our tooling, and node-reset.sh
|
||||
# ships alongside because uninstall-spx.sh delegates its state teardown to it
|
||||
# rather than reimplementing the QEMU and JetStream waits.
|
||||
for teardown in node-reset.sh uninstall-spx.sh; do
|
||||
if [ -f "$EXTRACT_DIR/$teardown" ]; then
|
||||
$SUDO install -m 0755 "$EXTRACT_DIR/$teardown" "/usr/local/share/spinifex/$teardown"
|
||||
info " /usr/local/share/spinifex/$teardown"
|
||||
fi
|
||||
done
|
||||
|
||||
# microVM kernel + initramfs
|
||||
$SUDO install -d /usr/share/spinifex/microvm
|
||||
if [ -d "$EXTRACT_DIR/microvm" ]; then
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
.terraform/
|
||||
.venv/
|
||||
__pycache__/
|
||||
terraform.tfstate
|
||||
terraform.tfstate.*
|
||||
*.tfplan
|
||||
*.auto.tfvars
|
||||
crash.log
|
||||
terraform.tfvars
|
||||
*.pem
|
||||
@@ -0,0 +1,244 @@
|
||||
# oci-spx — Terraform for Spinifex on OCI
|
||||
|
||||
Terraform configuration that builds the OCI infrastructure a Spinifex cluster runs on: a VCN with its gateways and subnets, one to N nodes with two VNICs each, and a data volume per node. No DRG, remote peering, local peering gateway, RZG, or other external network attachment is created.
|
||||
|
||||
**This is a local fork of [`aszynkow/oci_mulgadc`](https://github.com/aszynkow/oci_mulgadc), modified for Spinifex.** It builds infrastructure only; Spinifex itself is installed afterwards by the existing deploy path (`update-nodes.sh`, `install-node.sh`), the same scripts that serve prod and dev-prod. The operator guide that wraps it is [`docs/oci-integration`](../../../docs/oci-integration/README.md); the plan is `docs/development/feature/oci-terraform-provisioning.md` in the mulga monorepo.
|
||||
|
||||
## What This Repository Adds
|
||||
|
||||
**It deploys into a compartment that already exists and creates no identity resources.** `compartment_ocid` is required. Upstream created a compartment, an IAM group and a root policy; those need tenancy-root rights, which is the opposite of the permission boundary this deployment argues for.
|
||||
|
||||
Every resource is prefixed with `deployment_name` (default `spinifex`), so nothing here can collide with or be mistaken for something built by hand in the same compartment. The value becomes the VCN `dns_label`, so it is validated as 1-15 lowercase alphanumerics.
|
||||
|
||||
- `spinifex-vcn` (`10.200.0.0/22`) with public and private subnets, IGW, NAT Gateway, Service Gateway, and separate public/private route tables.
|
||||
- A public-subnet security list opening the whole VCN to itself and, separately, `node_client_cidr_allow_list` to everything — guest policy belongs to the customer's AWS security groups, not to OCI.
|
||||
- A counted `VM.Standard.E6.Flex` node deployment using the latest compatible Canonical Ubuntu 26.04 image, with one data volume per node.
|
||||
- A second VNIC per node in the same public subnet, which is where every Spinifex external address is registered at runtime.
|
||||
|
||||
## How this differs from upstream
|
||||
|
||||
| | Upstream | Here |
|
||||
| --- | --- | --- |
|
||||
| Shape | `BM.Standard.E2.64` | `VM.Standard.E6.Flex`, 8 OCPU / 32 GB |
|
||||
| Node placement | Private subnet, no public IP | **Public subnet, public IP required** |
|
||||
| Bastion | Standard Bastion in the private subnet | Removed |
|
||||
| Compartment | Created, with an IAM group and root policy | **Existing**, by OCID; no identity resources |
|
||||
| Vault | Imports the SSH private key as a secret | Removed |
|
||||
| Resource names | `mulgadc-*` | `${deployment_name}-*`, default `spinifex-*` |
|
||||
| VNICs per node | 1 | 2 |
|
||||
| VCN | `/16` | `/22` |
|
||||
| Data volume | 20 TiB | 256 GiB |
|
||||
|
||||
## Quick Start
|
||||
|
||||
Prerequisites: Terraform, Python 3, OCI credentials in `~/.oci/config`, and the public and private halves of the SSH key. The Python helper creates and uses this repository's isolated `.venv` and installs [requirements.txt](requirements.txt).
|
||||
|
||||
```bash
|
||||
cd scripts/terraform/oci-spx
|
||||
|
||||
echo 'compartment_ocid = "<ocid of an existing compartment>"' > terraform.auto.tfvars
|
||||
|
||||
python3 scripts/oci_env.py --ssh-public-key-path <path_to_public_ssh_key> -- terraform init
|
||||
|
||||
python3 scripts/oci_env.py --ssh-public-key-path <path_to_public_ssh_key> -- terraform plan -out=mulgadc.tfplan
|
||||
```
|
||||
|
||||
Review the saved plan, then apply that exact plan:
|
||||
|
||||
```bash
|
||||
python3 scripts/oci_env.py --ssh-public-key-path <path_to_public_ssh_key> -- terraform apply mulgadc.tfplan
|
||||
```
|
||||
|
||||
## Config and State
|
||||
|
||||
| Item | Purpose |
|
||||
| --- | --- |
|
||||
| `scripts/oci_env.py` | Loads OCI profile inputs, discovers the tenancy home region, exports Terraform variables, and runs Terraform in the repository `.venv`. |
|
||||
| `requirements.txt` | Python dependency set for the environment helper. |
|
||||
| `terraform.tfvars.example` | Example non-secret Terraform inputs. Copy it to untracked `terraform.auto.tfvars` for local overrides. |
|
||||
| `.venv/` | Repository-local virtual environment; generated locally and not committed. |
|
||||
| `terraform.tfstate` | Terraform state; generated locally unless a remote backend is configured. Do not commit it. |
|
||||
|
||||
The helper defaults to OCI profile `apacanzset03child03`. If it is absent and `apacanzset03child3` is locally configured, the helper explicitly falls back to that profile. Override either setting when needed:
|
||||
|
||||
```bash
|
||||
python3 scripts/oci_env.py --profile my-profile --region ap-sydney-1 -- terraform plan
|
||||
```
|
||||
|
||||
`ssh_public_key_path` is what installs your key on each node. **`ssh_private_key_path` is unused** — the vault that imported it was removed, so Terraform never reads a private key and none can reach the state file. The helper still exports it, which is why the variable is still declared.
|
||||
|
||||
> [!WARNING]
|
||||
> Never commit the state file, a plan file, or key material. `.gitignore` covers `terraform.tfstate*`, `*.tfplan`, `*.auto.tfvars`, `terraform.tfvars` and `*.pem`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
compartment["Existing compartment<br/>var.compartment_ocid"]
|
||||
|
||||
subgraph region["Selected workload region"]
|
||||
subgraph vcn["spinifex-vcn · 10.200.0.0/22"]
|
||||
igw["Internet Gateway<br/>spinifex-igw"]
|
||||
nat["NAT Gateway<br/>spinifex-nat"]
|
||||
sgw["Service Gateway<br/>spinifex-sgw"]
|
||||
public_rt["Public route table<br/>0.0.0.0/0 → IGW"]
|
||||
private_rt["Private route table<br/>0.0.0.0/0 → NAT<br/>Oracle Services → SGW"]
|
||||
public_subnet["Public subnet<br/>10.200.0.0/23"]
|
||||
private_subnet["Private subnet · unused<br/>10.200.2.0/24"]
|
||||
public_sl["spinifex-public-sl<br/>all from VCN · 22/3000/8443/9999 + ping from internet"]
|
||||
private_sl["spinifex-private-sl<br/>SSH 22 + ping from VCN only"]
|
||||
|
||||
public_subnet --> public_rt --> igw
|
||||
private_subnet --> private_rt
|
||||
private_rt --> nat
|
||||
private_rt --> sgw
|
||||
public_sl --> public_subnet
|
||||
private_sl --> private_subnet
|
||||
end
|
||||
|
||||
subgraph node_set["Counted node deployment · node_count (default 1)"]
|
||||
node["VM.Standard.E6.Flex[n]<br/>8 OCPU / 32 GB · Ubuntu 26.04"]
|
||||
vnic1["Primary VNIC<br/>SSH · advertise · Geneve"]
|
||||
vnic2["Secondary VNIC<br/>Spinifex external addresses"]
|
||||
volume["256-GiB data volume[n]<br/>iSCSI → /var/lib/spinifex"]
|
||||
node --> vnic1
|
||||
node --> vnic2
|
||||
node --> volume
|
||||
end
|
||||
public_subnet --> vnic1
|
||||
public_subnet --> vnic2
|
||||
end
|
||||
|
||||
compartment --> vcn
|
||||
```
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```text
|
||||
.
|
||||
├── identity.tf # Lookup of the existing compartment
|
||||
├── network.tf # VCN, gateways, routes, subnets, and security lists
|
||||
├── compute.tf # Counted nodes, second VNICs, volumes, and iSCSI attachments
|
||||
├── cloud-init/ # user_data template and the data-volume mount script
|
||||
├── provider.tf # Single-region provider
|
||||
├── variables.tf # Inputs and validations
|
||||
├── outputs.tf # OCIDs and policy outputs
|
||||
├── versions.tf # Terraform and provider version constraints
|
||||
├── terraform.tfvars.example
|
||||
├── requirements.txt
|
||||
└── scripts/oci_env.py # Python environment and Terraform command helper
|
||||
```
|
||||
|
||||
## Identity and regions
|
||||
|
||||
**Everything is created in one region, in a compartment that already exists.** There is no aliased home-region provider and no identity resource, because creating a compartment, a group or a root policy needs tenancy-root rights. A profile scoped to a compartment fails those calls with `404-NotAuthorizedOrNotFound` against the home-region identity endpoint, which reads like a missing resource rather than a missing permission.
|
||||
|
||||
`region` defaults to `ap-sydney-1`. Spinifex's own region is a separate setting that follows AWS naming and defaults to `ap-southeast-2`; seeing both is correct, not a mismatch.
|
||||
|
||||
## Sizing
|
||||
|
||||
`node_count` defaults to `1` and supports `1` through `25`. Each index produces a matching instance, second VNIC, data volume, and attachment; names include the index, for example `spinifex-node-01`.
|
||||
|
||||
**8 OCPU (16 vCPU) and 32 GB RAM is the minimum recommended for a Spinifex node**, and is what the variables default to. OCI counts an OCPU as a full core, so 8 OCPU is 16 threads. The control plane, predastore, viperblock, NATS and OVN all share that before a single guest boots.
|
||||
|
||||
**Bare metal is the preferred shape** — the whole host, no noisy neighbour, and nested virtualisation that is not a guest of a guest. Until the bare-metal quota in this tenancy is raised we deploy on `VM.Standard.E6.Flex`, which is known-good: nested KVM works on it, measured on `mulga-poc`.
|
||||
|
||||
`shape_config` sizes a flex shape and is **rejected on a fixed bare-metal shape**, so `compute.tf` emits it only when the shape name contains `Flex`. Setting `compute_shape` to a `BM.*` shape is therefore a one-variable change, and `compute_ocpus` and `compute_memory_in_gbs` are then ignored.
|
||||
|
||||
Each data volume defaults to `256` GiB at `120` VPUs/GB, the Ultra High Performance tier. It is a separate data volume: the boot volume comes from the image's default size and is not affected by `data_volume_size_in_gbs`.
|
||||
|
||||
## The data volume and cloud-init
|
||||
|
||||
`cloud-init/mount-data-volume.sh` runs once per boot from `user_data` and leaves the volume mounted at `data_mountpoint`, which defaults to `/var/lib/spinifex`. It mounts Spinifex's data directory itself rather than `/mnt/something` with a redirect, so it is one mount and no Spinifex config points anywhere unusual.
|
||||
|
||||
It is safe to re-run. **It formats only a device with no filesystem on it**, and treats a `blkid` failure it does not recognise as fatal rather than assuming the device is blank.
|
||||
|
||||
**`_netdev` is load bearing.** Without it the mount is attempted before `iscsid` has a session, and the boot either hangs or silently lands the whole stack on the small boot disk. That is what happened on `mulga-poc` before its fstab entry was corrected, and it was only caught because a benchmark looked wrong. The acceptance test is therefore a **reboot**, not a successful first boot — `findmnt /var/lib/spinifex`, never `ls`.
|
||||
|
||||
The script mounts the device by name and never lets `mount(8)` resolve the mount point through fstab: the fstab entry exists for the next boot, not the current one.
|
||||
|
||||
`device` is set on the attachment deliberately. A volume attached without one is named after the boot disk — `mulga-poc`'s symlink is `/dev/oracleoci/oraclevda1 → /dev/sdb1` — whereas asking for the path gives a stable `/dev/oracleoci/oraclevdb`, identical on every node.
|
||||
|
||||
## The WAN bridge
|
||||
|
||||
`cloud-init/setup-wan-bridge.sh` builds `br-wan`, the Linux bridge carrying Spinifex's external datapath, over the **secondary** VNIC: NIC enslaved, bridge MAC cloned from the VNIC, MTU 9000, and table-200 source routing.
|
||||
|
||||
**Source routing is not optional on OCI.** OCI enforces the source address per VNIC, so a reply leaving through the primary from a secondary address is dropped on the wire. Test egress past the subnet — pinging the VCN router proves nothing, because the hypervisor answers it locally.
|
||||
|
||||
**It writes `/etc/netplan/60-spinifex-wan.yaml` and never edits `50-cloud-init.yaml`.** cloud-init owns that file and regenerates it, so edits there are lost. netplan merges every file in the directory, which is why a separate higher-numbered file is the supported way to extend a cloud-init-managed network.
|
||||
|
||||
**The VNIC's MAC and address come from IMDS at runtime, not from Terraform.** They cannot come from Terraform: the second VNIC is attached after the instance reaches RUNNING, so when `user_data` is written it does not exist yet. The script reads `/opc/v2/vnics/` and takes the first entry whose MAC is not that of the interface holding the default route.
|
||||
|
||||
`spinifex-wan-bridge.service` carries it, with `ConditionPathExists=!` on the netplan file: a boot where the VNIC arrived late retries, and once configured the unit stops running because netplan owns the config. Seeing it `inactive` after a reboot is correct, not a failure.
|
||||
|
||||
## The host firewall
|
||||
|
||||
**The Ubuntu OCI image closes the ports a node serves.** Its INPUT chain ends in `REJECT --reject-with icmp-host-prohibited` with only `22`, ICMP, `lo` and ESTABLISHED above it, so `3000`, `8443` and `9999` are unreachable on a fresh instance. `cloud-init/open-service-ports.sh` inserts them **above** the REJECT — appending would put them where they can never match — and persists with `netfilter-persistent save`.
|
||||
|
||||
> [!WARNING]
|
||||
> **Never flush iptables on an OCI node.** The image's `InstanceServices` chain in OUTPUT is what permits iSCSI to `169.254.2.0/24:3260` and the metadata service. `iptables -F` takes `/var/lib/spinifex` and IMDS with it, and it surfaces later as a mount problem rather than a firewall one.
|
||||
|
||||
FORWARD is left alone too: Spinifex adds its own `spinifex-nat-egress` rules there.
|
||||
|
||||
`node_service_ports` drives the host firewall only, and that is the whole of what it drives. The OCI security list is open, deliberately — see below.
|
||||
|
||||
### Three layers, and only one of them is the customer's
|
||||
|
||||
| Layer | Guards | Who sets it |
|
||||
| --- | --- | --- |
|
||||
| OCI security list | The subnet | **Open.** Narrow `node_client_cidr_allow_list` to restrict who reaches the deployment at all |
|
||||
| OVN security groups | Each guest | The customer, through the AWS API, exactly as on EC2 |
|
||||
| Host firewall (`open-service-ports.sh`) | The node's own listeners | `node_service_ports` |
|
||||
|
||||
**The OCI security list is open on purpose.** A guest's firewall is its AWS security group, and a customer who opens `443` on theirs expects `443` to work. They cannot see an OCI security list, let alone edit one. Enumerating guest ports there would mean a Terraform change per customer port, and the failure would present as "my security group does nothing" — among the hardest network faults to trace. The intra-VCN rule is kept separate from the internet rule so that narrowing the latter can never cut Geneve, OVN, NATS or predastore between nodes.
|
||||
|
||||
## Network and Access
|
||||
|
||||
**Spinifex nodes sit in the public subnet with public IPs, and this is a requirement rather than a convenience.** A node serves the UI on `3000`, the predastore gate on `8443` and the AWS gateway on `9999` to operators and customers directly; a node in a private subnet cannot do its job. There is no bastion.
|
||||
|
||||
Each node gets two VNICs in that same subnet. The primary carries the host plane — SSH, the node's advertise address, and the Geneve overlay — and the secondary is where the OCI allocator registers every Spinifex external address as a secondary private IP. `VM.Standard.E6.Flex` allows two VNIC attachments in total, so that is the whole budget; Oracle documents up to 64 secondary private IPv4 objects per VNIC, which is the per-node ceiling on Elastic IPs.
|
||||
|
||||
`skip_source_dest_check` is deliberately left off. Routed NAT masquerades to the VNIC's own address and an Elastic IP's private half is itself a registered address on the VNIC, so nothing Spinifex sends is a foreign source.
|
||||
|
||||
The private subnet is **unused today** and kept only so it exists: Spinifex guests live on the OVN overlay and never occupy an OCI subnet.
|
||||
|
||||
`node_client_cidr_allow_list` controls who may reach a node's SSH and service ports from the internet, and defaults to `0.0.0.0/0`. Restrict it before production use:
|
||||
|
||||
```hcl
|
||||
node_client_cidr_allow_list = ["203.0.113.10/32"]
|
||||
```
|
||||
|
||||
## The API user this seeds
|
||||
|
||||
The OCI provider integration needs credentials on each node so Spinifex can allocate addresses at runtime. **Grant that user only the operations Spinifex performs on addresses** — create/get/list/delete private IPs, create/get/update/delete public IPs — and nothing else. It is not a tenancy admin. Broader rights widen the blast radius of a node compromise for no benefit.
|
||||
|
||||
## Common Commands
|
||||
|
||||
Show the current outputs:
|
||||
|
||||
```bash
|
||||
terraform output
|
||||
```
|
||||
|
||||
Run a plan through the Python helper:
|
||||
|
||||
```bash
|
||||
python3 scripts/oci_env.py --ssh-public-key-path <path_to_public_ssh_key> -- terraform plan
|
||||
```
|
||||
|
||||
Export the helper's variables into the current shell instead of wrapping every command:
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/oci_env.py --shell --ssh-public-key-path <path_to_public_ssh_key>)"
|
||||
terraform plan
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Missing SSH-path error:** pass `--ssh-public-key-path` to `oci_env.py`. The private-key option is optional and unused.
|
||||
- **`404-NotAuthorizedOrNotFound` on a create:** this is a permission error wearing a not-found error's clothes. Check the profile can write to `compartment_ocid`.
|
||||
- **Additional node does not have a data volume:** keep `node_count` within `1`–`25`; the instance, volume, and attachment use the same count.
|
||||
- **Need to access a node:** SSH to its public IP with the key from `ssh_public_key_path`. `terraform output nodes` lists the names and addresses; `terraform output -raw hosts_file` writes the form `update-nodes.sh --hosts-file` accepts.
|
||||
- **`shape_config` rejected on apply:** the shape is a fixed bare-metal one. Remove the size variables from your `.auto.tfvars`; they do not apply.
|
||||
- **Second VNIC refused:** `VM.Standard.E6.Flex` allows two VNIC attachments and both are used here. A shape with a lower `max-vnic-attachments` cannot run Spinifex.
|
||||
- **`/var/lib/spinifex` is not mounted after boot:** read `/var/log/cloud-init-output.log` for the `[spinifex-data-volume]` lines. If it says the device never appeared, the iSCSI session is the problem — check `iscsiadm -m session` and that the Block Volume Management agent plugin is enabled. Re-running `/usr/local/sbin/spinifex-mount-data-volume /dev/oracleoci/oraclevdb /var/lib/spinifex` is safe.
|
||||
@@ -0,0 +1,94 @@
|
||||
#!/bin/bash
|
||||
# Prepares the OCI block volume that backs Spinifex's data directory, and makes
|
||||
# the mount survive a reboot. Safe to re-run: it never formats a device that
|
||||
# already holds a filesystem.
|
||||
set -euo pipefail
|
||||
|
||||
DEVICE="${1:?device path required}"
|
||||
MOUNTPOINT="${2:?mount point required}"
|
||||
FSLABEL=spinifex-data
|
||||
|
||||
# Overridable so the fstab logic can be exercised against a scratch file rather
|
||||
# than the host's real one. Nothing in production sets it.
|
||||
FSTAB="${FSTAB:-/etc/fstab}"
|
||||
|
||||
log() { printf '[spinifex-data-volume] %s\n' "$*"; }
|
||||
die() {
|
||||
log "ERROR: $*"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# The Oracle Cloud Agent logs the iSCSI session in asynchronously, so the device
|
||||
# is not present when cloud-init first runs. Wait for it instead of racing it.
|
||||
wait_for_device() {
|
||||
local i
|
||||
for i in $(seq 1 120); do
|
||||
if [ -b "$DEVICE" ]; then
|
||||
log "$DEVICE appeared after $((i * 5))s"
|
||||
return 0
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
die "$DEVICE did not appear within 600s; check the iSCSI session with 'iscsiadm -m session'"
|
||||
}
|
||||
|
||||
# blkid reporting a TYPE is the only thing standing between a re-run and an erased
|
||||
# data volume, so a failure to read the device is fatal rather than "assume blank".
|
||||
format_if_blank() {
|
||||
local fstype rc=0
|
||||
fstype="$(blkid -o value -s TYPE "$DEVICE" 2>/dev/null)" || rc=$?
|
||||
|
||||
# blkid exits 2 for an unrecognised device, which is the genuinely blank case.
|
||||
if [ "$rc" -ne 0 ] && [ "$rc" -ne 2 ]; then
|
||||
die "blkid failed on $DEVICE with status $rc; refusing to format"
|
||||
fi
|
||||
|
||||
if [ -n "$fstype" ]; then
|
||||
log "$DEVICE already holds a $fstype filesystem; leaving it alone"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log "no filesystem on $DEVICE; creating ext4"
|
||||
mkfs.ext4 -q -L "$FSLABEL" "$DEVICE"
|
||||
}
|
||||
|
||||
# _netdev is load bearing. Without it the mount is attempted before iscsid has a
|
||||
# session and the boot either hangs or silently lands the stack on the boot disk.
|
||||
write_fstab_entry() {
|
||||
local uuid entry
|
||||
uuid="$(blkid -o value -s UUID "$DEVICE")" || die "no UUID on $DEVICE after formatting"
|
||||
entry="UUID=$uuid $MOUNTPOINT ext4 defaults,_netdev,nofail 0 2"
|
||||
|
||||
if grep -qF "$entry" "$FSTAB"; then
|
||||
log "fstab already carries the entry for $uuid"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Drop any earlier entry for this mount point, so a re-created volume with a
|
||||
# new UUID does not leave a stale line behind it.
|
||||
sed -i.bak -E "\%^[^#][^[:space:]]*[[:space:]]+${MOUNTPOINT}[[:space:]]%d" "$FSTAB"
|
||||
printf '%s\n' "$entry" >> "$FSTAB"
|
||||
log "added $entry"
|
||||
}
|
||||
|
||||
main() {
|
||||
wait_for_device
|
||||
format_if_blank
|
||||
mkdir -p "$MOUNTPOINT"
|
||||
write_fstab_entry
|
||||
|
||||
systemctl daemon-reload
|
||||
|
||||
# Mount the device by name rather than letting mount(8) look the mount point up
|
||||
# in fstab: the fstab entry exists for the next boot, not for this one.
|
||||
if mountpoint -q "$MOUNTPOINT"; then
|
||||
log "$MOUNTPOINT is already mounted"
|
||||
else
|
||||
mount -o defaults,_netdev,nofail "$DEVICE" "$MOUNTPOINT"
|
||||
fi
|
||||
|
||||
findmnt --noheadings --output SOURCE,TARGET,FSTYPE,OPTIONS "$MOUNTPOINT" \
|
||||
|| die "$MOUNTPOINT is not mounted after mount(8) reported success"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/bin/bash
|
||||
# Opens what a Spinifex node serves: every protocol to its cluster peers, and the
|
||||
# listed TCP ports to everyone else. The Ubuntu OCI image ships an INPUT chain
|
||||
# that ends in REJECT with only 22, ICMP, lo and ESTABLISHED above it, so the
|
||||
# console, the predastore gate and the AWS gateway are closed by default.
|
||||
#
|
||||
# Usage: spinifex-open-service-ports <peer-cidr> <port> [port...]
|
||||
set -euo pipefail
|
||||
|
||||
log() { printf '[spinifex-service-ports] %s\n' "$*"; }
|
||||
die() {
|
||||
log "ERROR: $*"
|
||||
exit 1
|
||||
}
|
||||
|
||||
[ "$#" -gt 1 ] || die "usage: $(basename "$0") <peer-cidr> <port> [port...]"
|
||||
|
||||
PEER_CIDR="$1"
|
||||
shift
|
||||
|
||||
# Never flush. The image's InstanceServices chain in OUTPUT is what permits iSCSI
|
||||
# to 169.254.2.0/24:3260 and the metadata service; clearing it takes the data
|
||||
# volume and IMDS with it. FORWARD is left alone too -- Spinifex manages it.
|
||||
insert_position() {
|
||||
iptables -L INPUT --line-numbers -n \
|
||||
| awk '$2 == "REJECT" || $2 == "DROP" { print $1; exit }'
|
||||
}
|
||||
|
||||
open_port() {
|
||||
local port="$1" pos
|
||||
if iptables -C INPUT -p tcp -m state --state NEW -m tcp --dport "$port" -j ACCEPT 2>/dev/null; then
|
||||
log "port $port is already open"
|
||||
return 0
|
||||
fi
|
||||
|
||||
pos="$(insert_position)"
|
||||
if [ -n "$pos" ]; then
|
||||
# Appending would land after the REJECT, where the rule can never match.
|
||||
iptables -I INPUT "$pos" -p tcp -m state --state NEW -m tcp --dport "$port" -j ACCEPT
|
||||
log "opened port $port at INPUT position $pos"
|
||||
else
|
||||
iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport "$port" -j ACCEPT
|
||||
log "opened port $port (appended; INPUT has no terminal REJECT)"
|
||||
fi
|
||||
}
|
||||
|
||||
# A cluster talks on far more than the ports customers reach: the formation
|
||||
# server, OVN's NB/SB and raft, NATS, predastore, viperblock and Geneve. Listing
|
||||
# them would be a list to keep in step with the code, and the subnet holds only
|
||||
# our own nodes, so the peer CIDR is the boundary -- exactly as the OCI security
|
||||
# list treats it. Narrowing the internet rule can then never cut the cluster.
|
||||
open_peers() {
|
||||
if iptables -C INPUT -s "$PEER_CIDR" -j ACCEPT 2>/dev/null; then
|
||||
log "peer CIDR $PEER_CIDR is already open"
|
||||
return 0
|
||||
fi
|
||||
iptables -I INPUT 1 -s "$PEER_CIDR" -m comment --comment spinifex-peers -j ACCEPT
|
||||
log "opened all protocols from $PEER_CIDR at INPUT position 1"
|
||||
}
|
||||
|
||||
main() {
|
||||
local port
|
||||
open_peers
|
||||
for port in "$@"; do
|
||||
open_port "$port"
|
||||
done
|
||||
|
||||
netfilter-persistent save >/dev/null || die "netfilter-persistent save failed; rules would not survive a reboot"
|
||||
log "saved to /etc/iptables/rules.v4"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+215
@@ -0,0 +1,215 @@
|
||||
#!/bin/bash
|
||||
# Builds the Linux bridge that carries Spinifex's external datapath over the
|
||||
# secondary OCI VNIC. Discovers that VNIC from the instance metadata service,
|
||||
# because Terraform attaches it after the instance is already RUNNING and its
|
||||
# MAC and address are therefore unknown when user-data is written.
|
||||
set -euo pipefail
|
||||
|
||||
BRIDGE="${BRIDGE:-br-wan}"
|
||||
MTU="${MTU:-9000}"
|
||||
IMDS="${IMDS:-http://169.254.169.254/opc/v2}"
|
||||
|
||||
# A file of our own. 50-cloud-init.yaml belongs to cloud-init, so anything added
|
||||
# there is lost when it re-renders. netplan merges every file in the directory,
|
||||
# which is also why strip_cloud_init_secondary below has work to do.
|
||||
NETPLAN_FILE="${NETPLAN_FILE:-/etc/netplan/60-spinifex-wan.yaml}"
|
||||
|
||||
log() { printf '[spinifex-wan-bridge] %s\n' "$*"; }
|
||||
die() {
|
||||
log "ERROR: $*"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# The VNIC holding the default route is the host plane and must be left alone.
|
||||
primary_mac() {
|
||||
local dev
|
||||
dev="$(ip -4 route show default | awk '{for (i=1;i<NF;i++) if ($i=="dev") print $(i+1); exit}')"
|
||||
[ -n "$dev" ] || die "no default route, so the primary VNIC cannot be identified"
|
||||
cat "/sys/class/net/$dev/address"
|
||||
}
|
||||
|
||||
# Prints shell assignments for the first VNIC that is not the primary, or exits 1.
|
||||
imds_secondary() {
|
||||
curl -sf --max-time 10 -H 'Authorization: Bearer Oracle' "$IMDS/vnics/" |
|
||||
python3 -c '
|
||||
import ipaddress, json, sys
|
||||
|
||||
primary = sys.argv[1].lower()
|
||||
for vnic in json.load(sys.stdin):
|
||||
if vnic["macAddr"].lower() == primary:
|
||||
continue
|
||||
net = ipaddress.ip_network(vnic["subnetCidrBlock"])
|
||||
print("WAN_MAC=%s" % vnic["macAddr"].lower())
|
||||
print("WAN_IP=%s" % vnic["privateIp"])
|
||||
print("WAN_PREFIX=%d" % net.prefixlen)
|
||||
print("WAN_CIDR=%s" % vnic["subnetCidrBlock"])
|
||||
print("WAN_ROUTER=%s" % vnic["virtualRouterIp"])
|
||||
sys.exit(0)
|
||||
sys.exit(1)
|
||||
' "$1"
|
||||
}
|
||||
|
||||
wait_for_secondary() {
|
||||
local primary i out
|
||||
primary="$(primary_mac)"
|
||||
log "primary VNIC MAC is $primary"
|
||||
|
||||
for i in $(seq 1 60); do
|
||||
if out="$(imds_secondary "$primary")" && [ -n "$out" ]; then
|
||||
eval "$out"
|
||||
log "secondary VNIC after $((i * 5))s: $WAN_MAC $WAN_IP/$WAN_PREFIX via $WAN_ROUTER"
|
||||
return 0
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
die "no secondary VNIC appeared within 300s; check the VNIC attachment in OCI"
|
||||
}
|
||||
|
||||
# The physical interface for a MAC. Matching on the address rather than trusting a
|
||||
# name means a rename between boots cannot silently bridge the wrong interface.
|
||||
#
|
||||
# Bridges are skipped deliberately: once this has run, $BRIDGE wears the same
|
||||
# cloned MAC as its member, and a plain match finds the bridge and tries to
|
||||
# enslave it to itself.
|
||||
iface_for_mac() {
|
||||
local mac="$1" dev name
|
||||
for dev in /sys/class/net/*; do
|
||||
name="$(basename "$dev")"
|
||||
[ -e "$dev/address" ] || continue
|
||||
[ -d "$dev/bridge" ] && continue
|
||||
[ "$name" = "$BRIDGE" ] && continue
|
||||
if [ "$(cat "$dev/address")" = "$mac" ]; then
|
||||
printf '%s\n' "$name"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Source routing is not optional on OCI: it enforces the source address per VNIC,
|
||||
# so a reply leaving through the primary from a secondary address is dropped.
|
||||
write_netplan() {
|
||||
local iface="$1"
|
||||
cat > "$NETPLAN_FILE" <<EOF
|
||||
# Generated by spinifex-setup-wan-bridge. Do not edit; regenerate instead.
|
||||
network:
|
||||
version: 2
|
||||
ethernets:
|
||||
# Keyed by name, not by a macaddress match. The bridge clones the member's MAC,
|
||||
# so once it exists a match finds two devices and netplan refuses the interface
|
||||
# as non-unique. The script re-resolves the name from the MAC on every run and
|
||||
# rewrites this file if it ever changes, which covers the rename case instead.
|
||||
${iface}:
|
||||
dhcp4: false
|
||||
dhcp6: false
|
||||
mtu: ${MTU}
|
||||
bridges:
|
||||
${BRIDGE}:
|
||||
interfaces: [${iface}]
|
||||
macaddress: "${WAN_MAC}"
|
||||
mtu: ${MTU}
|
||||
dhcp4: false
|
||||
dhcp6: false
|
||||
# /32, not /${WAN_PREFIX}. Both VNICs sit in one subnet, so the on-link
|
||||
# route the kernel derives from the address would land in the main table at
|
||||
# metric 0 and take the whole subnet off the primary VNIC. Replies to the
|
||||
# primary's own address then leave by br-wan and OCI drops them, because it
|
||||
# enforces the source address per VNIC. Table 200 carries the on-link route.
|
||||
addresses:
|
||||
- ${WAN_IP}/32
|
||||
routes:
|
||||
- to: ${WAN_CIDR}
|
||||
scope: link
|
||||
table: 200
|
||||
- to: default
|
||||
via: ${WAN_ROUTER}
|
||||
table: 200
|
||||
routing-policy:
|
||||
- from: ${WAN_IP}/32
|
||||
table: 200
|
||||
priority: 1000
|
||||
EOF
|
||||
chmod 0600 "$NETPLAN_FILE"
|
||||
}
|
||||
|
||||
# cloud-init renders 50-cloud-init.yaml from the datasource's network config, so
|
||||
# whether the secondary VNIC appears there depends on whether it was attached when
|
||||
# that ran — a race Terraform loses on some nodes and wins on others. When it does
|
||||
# appear it carries the VNIC address as a /PREFIX, and netplan merges both files:
|
||||
# the bridge gets its /32 and the member keeps the prefixed address, whose metric-0
|
||||
# on-link route takes the whole subnet off the primary VNIC. Every packet between
|
||||
# nodes then leaves the wrong VNIC and OCI drops it. The stanza also matches on the
|
||||
# MAC the bridge later clones, so once the bridge exists netplan refuses the whole
|
||||
# directory with "Cannot find unique matching interface".
|
||||
#
|
||||
# Overriding it from 60- cannot work for the same reason, so remove it, and stop
|
||||
# cloud-init re-rendering the file on a later boot. The primary VNIC's own stanza
|
||||
# stays exactly as cloud-init wrote it.
|
||||
STRIPPED=0
|
||||
|
||||
strip_cloud_init_secondary() {
|
||||
local file=/etc/netplan/50-cloud-init.yaml
|
||||
[ -f "$file" ] || return 0
|
||||
|
||||
local removed
|
||||
removed="$(MAC="$WAN_MAC" python3 - "$file" <<'PY'
|
||||
import os, sys, yaml
|
||||
|
||||
path, mac = sys.argv[1], os.environ["MAC"].lower()
|
||||
with open(path) as fh:
|
||||
cfg = yaml.safe_load(fh) or {}
|
||||
|
||||
eths = cfg.get("network", {}).get("ethernets", {})
|
||||
drop = [n for n, e in eths.items()
|
||||
if str((e or {}).get("match", {}).get("macaddress", "")).lower() == mac]
|
||||
if not drop:
|
||||
sys.exit(0)
|
||||
|
||||
for name in drop:
|
||||
del eths[name]
|
||||
with open(path, "w") as fh:
|
||||
yaml.safe_dump(cfg, fh, default_flow_style=False, sort_keys=False)
|
||||
print(", ".join(drop))
|
||||
PY
|
||||
)" || die "could not edit $file"
|
||||
|
||||
if [ -n "$removed" ]; then
|
||||
log "removed the secondary VNIC stanza for $removed from $file"
|
||||
STRIPPED=1
|
||||
fi
|
||||
|
||||
printf 'network: {config: disabled}\n' > /etc/cloud/cloud.cfg.d/99-spinifex-network.cfg
|
||||
chmod 0644 /etc/cloud/cloud.cfg.d/99-spinifex-network.cfg
|
||||
}
|
||||
|
||||
main() {
|
||||
wait_for_secondary
|
||||
|
||||
local iface
|
||||
iface="$(iface_for_mac "$WAN_MAC")" || die "no interface carries $WAN_MAC"
|
||||
log "bridging $iface into $BRIDGE"
|
||||
|
||||
strip_cloud_init_secondary
|
||||
|
||||
local previous=""
|
||||
[ -f "$NETPLAN_FILE" ] && previous="$(cat "$NETPLAN_FILE")"
|
||||
write_netplan "$iface"
|
||||
|
||||
if [ "$STRIPPED" = 0 ] && [ "$previous" = "$(cat "$NETPLAN_FILE")" ] && ip link show "$BRIDGE" >/dev/null 2>&1; then
|
||||
log "$BRIDGE is already configured and unchanged"
|
||||
return 0
|
||||
fi
|
||||
|
||||
netplan apply
|
||||
for _ in $(seq 1 30); do
|
||||
ip -4 addr show "$BRIDGE" 2>/dev/null | grep -q "inet ${WAN_IP}/" && break
|
||||
sleep 2
|
||||
done
|
||||
|
||||
ip -br addr show "$BRIDGE" || die "$BRIDGE did not come up"
|
||||
ip -4 addr show "$BRIDGE" | grep -q "inet ${WAN_IP}/" \
|
||||
|| die "$BRIDGE is up but does not hold ${WAN_IP}"
|
||||
log "table 200: $(ip route show table 200 | tr '\n' ';')"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,50 @@
|
||||
#cloud-config
|
||||
# Brings the node to the state a bare-metal Spinifex node is in after imaging.
|
||||
# Installing Spinifex itself is the deploy path's job, not cloud-init's.
|
||||
|
||||
write_files:
|
||||
- path: /usr/local/sbin/spinifex-mount-data-volume
|
||||
owner: root:root
|
||||
permissions: "0750"
|
||||
content: |
|
||||
${indent(6, mount_script)}
|
||||
|
||||
- path: /usr/local/sbin/spinifex-setup-wan-bridge
|
||||
owner: root:root
|
||||
permissions: "0750"
|
||||
content: |
|
||||
${indent(6, wan_bridge_script)}
|
||||
|
||||
- path: /usr/local/sbin/spinifex-open-service-ports
|
||||
owner: root:root
|
||||
permissions: "0750"
|
||||
content: |
|
||||
${indent(6, service_ports_script)}
|
||||
|
||||
# The secondary VNIC is attached after the instance reaches RUNNING, so it can
|
||||
# arrive late. This unit retries on the next boot if the first attempt missed it,
|
||||
# and stops running once the netplan file it writes exists.
|
||||
- path: /etc/systemd/system/spinifex-wan-bridge.service
|
||||
owner: root:root
|
||||
permissions: "0644"
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Spinifex WAN bridge over the secondary OCI VNIC
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
ConditionPathExists=!/etc/netplan/60-spinifex-wan.yaml
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
Environment=BRIDGE=${wan_bridge_name}
|
||||
Environment=MTU=${wan_bridge_mtu}
|
||||
ExecStart=/usr/local/sbin/spinifex-setup-wan-bridge
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
runcmd:
|
||||
- [/usr/local/sbin/spinifex-mount-data-volume, "${data_device}", "${data_mountpoint}"]
|
||||
- [/usr/local/sbin/spinifex-open-service-ports, "${peer_cidr}", ${service_ports}]
|
||||
- [systemctl, enable, --now, spinifex-wan-bridge.service]
|
||||
@@ -0,0 +1,139 @@
|
||||
data "oci_identity_availability_domains" "available" {
|
||||
compartment_id = var.tenancy_ocid
|
||||
}
|
||||
|
||||
# ap-sydney-1 has exactly one availability domain, so spreading nodes across ADs
|
||||
# is not an option there and fault domains are the whole of the hardware
|
||||
# separation available. Each one is a different set of racks, power and network
|
||||
# inside the AD, which is what a Spinifex node needs from its neighbours.
|
||||
data "oci_identity_fault_domains" "available" {
|
||||
compartment_id = var.compartment_ocid
|
||||
availability_domain = local.compute_availability_domain
|
||||
}
|
||||
|
||||
# Uses the newest Ubuntu 26.04 platform image that OCI advertises for this
|
||||
# shape and region at plan time, rather than pinning a regional image OCID.
|
||||
data "oci_core_images" "ubuntu_2604" {
|
||||
compartment_id = var.tenancy_ocid
|
||||
operating_system = "Canonical Ubuntu"
|
||||
operating_system_version = var.ubuntu_version
|
||||
shape = var.compute_shape
|
||||
sort_by = "TIMECREATED"
|
||||
sort_order = "DESC"
|
||||
}
|
||||
|
||||
locals {
|
||||
compute_availability_domain = data.oci_identity_availability_domains.available.availability_domains[0].name
|
||||
ubuntu_image_id = data.oci_core_images.ubuntu_2604.images[0].id
|
||||
|
||||
# shape_config sizes a flex shape and is rejected outright on a fixed bare-metal
|
||||
# one, so it is emitted only when the shape name says Flex.
|
||||
shape_is_flex = strcontains(var.compute_shape, "Flex")
|
||||
|
||||
# Round-robin, so three nodes land on three fault domains rather than wherever
|
||||
# OCI's placement happens to put them. Left to itself OCI spreads on a
|
||||
# best-effort basis and says nothing about the result, which is not something
|
||||
# a storage cluster should take on trust.
|
||||
fault_domains = data.oci_identity_fault_domains.available.fault_domains[*].name
|
||||
|
||||
# Every node has exactly one data volume, so every node names it the same. Asking
|
||||
# for the path is what makes it stable: a volume attached without one is named
|
||||
# after the boot disk (mulga-poc has oraclevda1 pointing at sdb1).
|
||||
data_device = "/dev/oracleoci/oraclevdb"
|
||||
|
||||
# Identical on every node, so a change to it replaces all of them together.
|
||||
user_data = templatefile("${path.module}/cloud-init/user-data.yaml.tftpl", {
|
||||
data_device = local.data_device
|
||||
data_mountpoint = var.data_mountpoint
|
||||
peer_cidr = var.vcn_cidr
|
||||
service_ports = join(", ", [for p in var.node_service_ports : "\"${p}\""])
|
||||
wan_bridge_name = var.wan_bridge_name
|
||||
wan_bridge_mtu = var.wan_bridge_mtu
|
||||
mount_script = file("${path.module}/cloud-init/mount-data-volume.sh")
|
||||
wan_bridge_script = file("${path.module}/cloud-init/setup-wan-bridge.sh")
|
||||
service_ports_script = file("${path.module}/cloud-init/open-service-ports.sh")
|
||||
})
|
||||
}
|
||||
|
||||
resource "oci_core_instance" "mulgadc" {
|
||||
count = var.node_count
|
||||
availability_domain = local.compute_availability_domain
|
||||
fault_domain = local.fault_domains[count.index % length(local.fault_domains)]
|
||||
compartment_id = var.compartment_ocid
|
||||
display_name = format("%s-node-%02d", var.deployment_name, count.index + 1)
|
||||
shape = var.compute_shape
|
||||
|
||||
dynamic "shape_config" {
|
||||
for_each = local.shape_is_flex ? [1] : []
|
||||
content {
|
||||
ocpus = var.compute_ocpus
|
||||
memory_in_gbs = var.compute_memory_in_gbs
|
||||
}
|
||||
}
|
||||
|
||||
metadata = {
|
||||
ssh_authorized_keys = file(var.ssh_public_key_path)
|
||||
user_data = base64encode(local.user_data)
|
||||
}
|
||||
|
||||
# Required for automatic UHP iSCSI multipath setup and login.
|
||||
agent_config {
|
||||
plugins_config {
|
||||
name = "Block Volume Management"
|
||||
desired_state = "ENABLED"
|
||||
}
|
||||
}
|
||||
|
||||
# The host plane: SSH, the node's advertise address, and the Geneve overlay.
|
||||
# Public, because a node serves the UI, predastore and the AWS gateway itself.
|
||||
create_vnic_details {
|
||||
subnet_id = oci_core_subnet.public.id
|
||||
assign_public_ip = true
|
||||
hostname_label = format("%snode%02d", var.deployment_name, count.index + 1)
|
||||
}
|
||||
|
||||
source_details {
|
||||
source_type = "image"
|
||||
source_id = local.ubuntu_image_id
|
||||
}
|
||||
}
|
||||
|
||||
# Every Spinifex external address lives here as a secondary private IP, written by
|
||||
# the OCI allocator at runtime. Same subnet as the primary, which mulga-poc proves
|
||||
# works. E6.Flex allows two VNICs total, so this is the whole remaining budget.
|
||||
resource "oci_core_vnic_attachment" "mulgadc_external" {
|
||||
count = var.node_count
|
||||
instance_id = oci_core_instance.mulgadc[count.index].id
|
||||
|
||||
create_vnic_details {
|
||||
subnet_id = oci_core_subnet.public.id
|
||||
display_name = format("%s-node-%02d-external", var.deployment_name, count.index + 1)
|
||||
assign_public_ip = true
|
||||
|
||||
# Deliberately left at the default. Routed NAT masquerades to this VNIC's own
|
||||
# address and an EIP's private half is a registered IP here, so nothing we
|
||||
# send is a foreign source and skipping the check would buy nothing.
|
||||
skip_source_dest_check = false
|
||||
}
|
||||
}
|
||||
|
||||
# Backs /var/lib/spinifex, so viperblock, predastore and JetStream all land on it.
|
||||
# OCI presents the device; partitioning, formatting and mounting remain an OS task.
|
||||
resource "oci_core_volume" "mulgadc_data" {
|
||||
count = var.node_count
|
||||
availability_domain = local.compute_availability_domain
|
||||
compartment_id = var.compartment_ocid
|
||||
display_name = format("%s-node-%02d-data", var.deployment_name, count.index + 1)
|
||||
size_in_gbs = var.data_volume_size_in_gbs
|
||||
vpus_per_gb = var.data_volume_vpus_per_gb
|
||||
}
|
||||
|
||||
resource "oci_core_volume_attachment" "mulgadc_data" {
|
||||
count = var.node_count
|
||||
attachment_type = "iscsi"
|
||||
# Required by OCI for Ultra High Performance iSCSI volume attachments.
|
||||
device = local.data_device
|
||||
is_agent_auto_iscsi_login_enabled = true
|
||||
instance_id = oci_core_instance.mulgadc[count.index].id
|
||||
volume_id = oci_core_volume.mulgadc_data[count.index].id
|
||||
}
|
||||
Executable
+236
@@ -0,0 +1,236 @@
|
||||
#!/usr/bin/env bash
|
||||
# eip-move-test.sh — prove an EIP follows its guest across an OCI node move.
|
||||
#
|
||||
# Drives the three-node Terraform OCI cluster through the exact sequence that
|
||||
# strands an address today: associate an EIP, stop the guest, force it to start
|
||||
# on a different node, and assert the address still answers. The decisive check
|
||||
# is not the ping — it is the OCI API's own record of which VNIC carries the
|
||||
# private half, because a reachable address on the wrong VNIC is luck.
|
||||
#
|
||||
# Step 0b first proves the other half of the distinction on the same guest: the
|
||||
# auto-assigned address it launched with is gone from the OCI API while it is
|
||||
# stopped, and a different one comes back. That is the billing claim, asserted
|
||||
# against OCI rather than against our own record of what we think we released.
|
||||
#
|
||||
# Read-write on dev-oci only. Never point this at prod or dev-prod.
|
||||
set -euo pipefail
|
||||
|
||||
KEY=${KEY:-$HOME/.ssh/oci-poc.pem}
|
||||
OCI_BIN=${OCI_BIN:-$HOME/bin/oci}
|
||||
OCI_PROFILE=${OCI_PROFILE:-apacanzset03child03}
|
||||
SUBNET=${SUBNET:-ocid1.subnet.oc1.ap-sydney-1.aaaaaaaaimrhjgsqgrfg7bnka2d3nbx2n5ss7kurcwneon22rkclqf3ypfqa}
|
||||
|
||||
# Public address -> node name. The control plane is reached through node01.
|
||||
declare -A NODE_IP=(
|
||||
[spinifexnode01]=149.118.68.212
|
||||
[spinifexnode02]=137.23.19.167
|
||||
[spinifexnode03]=137.23.14.3
|
||||
)
|
||||
API_HOST=${API_HOST:-${NODE_IP[spinifexnode01]}}
|
||||
|
||||
say() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
|
||||
fail() { printf '\033[31mFAIL: %s\033[0m\n' "$*" >&2; FAILED=1; }
|
||||
ok() { printf '\033[32mok: %s\033[0m\n' "$*"; }
|
||||
FAILED=0
|
||||
|
||||
# Assertions take values rather than an exit status: `cond && ok || fail` runs
|
||||
# the failure branch whenever ok's own status is non-zero, and passing $? through
|
||||
# a helper only moves the same trap one line down.
|
||||
eq() { # eq <actual> <expected> <what>
|
||||
if [[ $1 == "$2" ]]; then ok "$3 ($1)"; else fail "$3: got '$1', wanted '$2'"; fi
|
||||
}
|
||||
unset_or_none() { # unset_or_none <actual> <what>
|
||||
if [[ -z $1 || $1 == None ]]; then ok "$2"; else fail "$2: got '$1'"; fi
|
||||
}
|
||||
is_set() { # is_set <actual> <what>
|
||||
if [[ -n $1 && $1 != None ]]; then ok "$2"; else fail "$2: got '$1'"; fi
|
||||
}
|
||||
|
||||
on() { # on <ip> <cmd...>
|
||||
local ip=$1; shift
|
||||
timeout 60 ssh -i "$KEY" -o StrictHostKeyChecking=no -o BatchMode=yes "ubuntu@$ip" "$@"
|
||||
}
|
||||
|
||||
aws_spx() { # aws_spx <args...> — run the AWS CLI from node01 against the cluster
|
||||
on "$API_HOST" "export AWS_PROFILE=spinifex AWS_CA_BUNDLE=/etc/spinifex/ca.pem; \
|
||||
aws --endpoint-url https://localhost:9999 --region ap-southeast-2 $*"
|
||||
}
|
||||
|
||||
# node_of <instance-id> — which node is the guest on right now
|
||||
node_of() {
|
||||
on "$API_HOST" "sudo spx get instances --config /etc/spinifex/spinifex.toml 2>/dev/null" \
|
||||
| awk -v id="$1" '$0 ~ id {print}' | grep -oE 'spinifexnode0[0-9]' | head -1
|
||||
}
|
||||
|
||||
# private_half_of <public-addr> — the OCI private address this EIP rides the
|
||||
# wire as. Read off the OVN row that stamps the public address, which is the
|
||||
# one place the pairing is recorded outside the bindings bucket. Guessing from
|
||||
# the host's /32 routes would be ambiguous: a node hosting the guest carries the
|
||||
# auto-assigned address's private half on the same VNIC.
|
||||
private_half_of() {
|
||||
on "$API_HOST" "NB=\$(sudo grep -oP '^ovn_nb_addr\\s*=\\s*\"\\K[^\"]+' /etc/spinifex/spinifex.toml); \
|
||||
sudo ovn-nbctl --db=\"\$NB\" --bare --columns=external_ip \
|
||||
find NAT 'external_ids:spinifex\\:public_ip=\"$1\"'" 2>/dev/null | tr -d '\r' | head -1
|
||||
}
|
||||
|
||||
# reserved_ip_ocid <public-addr> — the OCI public IP object behind an address,
|
||||
# empty when OCI holds none. This is the object the tenancy is billed for, so
|
||||
# it — not our own KV — is what answers whether a stop really gave it back.
|
||||
reserved_ip_ocid() {
|
||||
"$OCI_BIN" --profile "$OCI_PROFILE" network public-ip get \
|
||||
--public-ip-address "$1" --query 'data.id' --raw-output 2>/dev/null || true
|
||||
}
|
||||
|
||||
# state_of <instance-id> — the EC2 state name, for polling
|
||||
state_of() {
|
||||
aws_spx "ec2 describe-instances --instance-ids $1 \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text" 2>/dev/null | tr -d '\r'
|
||||
}
|
||||
|
||||
# public_ip_of <instance-id> — what DescribeInstances reports, "None" when none
|
||||
public_ip_of() {
|
||||
aws_spx "ec2 describe-instances --instance-ids $1 \
|
||||
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text" 2>/dev/null | tr -d '\r'
|
||||
}
|
||||
|
||||
# is_state <instance-id> <state> — a predicate for wait_for
|
||||
is_state() { [[ $(state_of "$1") == "$2" ]]; }
|
||||
|
||||
# vnic_of_private <addr> — the OCI VNIC OCID currently carrying a private IP
|
||||
vnic_of_private() {
|
||||
"$OCI_BIN" --profile "$OCI_PROFILE" network private-ip list \
|
||||
--subnet-id "$SUBNET" --ip-address "$1" --query 'data[0]."vnic-id"' --raw-output 2>/dev/null
|
||||
}
|
||||
|
||||
# external_vnic_of <node-name> — that node's br-wan (secondary) VNIC OCID
|
||||
external_vnic_of() {
|
||||
on "${NODE_IP[$1]}" 'curl -s -H "Authorization: Bearer Oracle" -L \
|
||||
http://169.254.169.254/opc/v2/vnics/ | python3 -c "
|
||||
import sys,json
|
||||
v=json.load(sys.stdin)
|
||||
print(v[1][\"vnicId\"] if len(v)>1 else \"\")"'
|
||||
}
|
||||
|
||||
reachable() { ping -c4 -W2 "$1" >/dev/null 2>&1; }
|
||||
|
||||
wait_for() { # wait_for <seconds> <description> <cmd...>
|
||||
local secs=$1 desc=$2; shift 2
|
||||
local deadline=$(( SECONDS + secs ))
|
||||
while (( SECONDS < deadline )); do
|
||||
if "$@"; then ok "$desc"; return 0; fi
|
||||
sleep 5
|
||||
done
|
||||
fail "$desc (timed out after ${secs}s)"
|
||||
return 1
|
||||
}
|
||||
|
||||
INSTANCE=${INSTANCE:-}
|
||||
ALLOC=""
|
||||
cleanup() {
|
||||
[[ -n $ALLOC ]] || return 0
|
||||
say "cleanup: releasing $ALLOC"
|
||||
local assoc
|
||||
assoc=$(aws_spx "ec2 describe-addresses --query \
|
||||
\"Addresses[?AllocationId=='$ALLOC'].AssociationId\" --output text" 2>/dev/null | tr -d '\r')
|
||||
[[ -n $assoc && $assoc != None ]] && aws_spx "ec2 disassociate-address --association-id $assoc" >/dev/null 2>&1
|
||||
sleep 5
|
||||
aws_spx "ec2 release-address --allocation-id $ALLOC" >/dev/null 2>&1 || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
[[ -n $INSTANCE ]] || { echo "set INSTANCE=i-... (a running guest with an ENI)" >&2; exit 2; }
|
||||
|
||||
say "0. starting state"
|
||||
ENI=$(aws_spx "ec2 describe-instances --instance-ids $INSTANCE \
|
||||
--query 'Reservations[0].Instances[0].NetworkInterfaces[0].NetworkInterfaceId' --output text" | tr -d '\r')
|
||||
AUTO=$(public_ip_of "$INSTANCE")
|
||||
echo "instance $INSTANCE on $(node_of "$INSTANCE"), eni $ENI, auto-assigned ${AUTO:-<none>}"
|
||||
|
||||
# An auto-assigned address is borrowed for as long as the guest runs. On OCI it
|
||||
# is a reserved public IP object billed to the tenancy, so "released on stop" is
|
||||
# a billing claim and OCI's own API is the only place it can be proved.
|
||||
say "0b. an auto-assigned address goes back to OCI on stop"
|
||||
if [[ -z $AUTO || $AUTO == None ]]; then
|
||||
echo "skipped: guest has no auto-assigned address"
|
||||
else
|
||||
AUTO_OCID=$(reserved_ip_ocid "$AUTO")
|
||||
is_set "$AUTO_OCID" "OCI holds a public IP object for $AUTO while running"
|
||||
|
||||
aws_spx "ec2 stop-instances --instance-ids $INSTANCE" >/dev/null
|
||||
wait_for 180 "instance stopped" is_state "$INSTANCE" stopped
|
||||
unset_or_none "$(public_ip_of "$INSTANCE")" "stopped instance reports no PublicIpAddress"
|
||||
unset_or_none "$(reserved_ip_ocid "$AUTO")" "OCI no longer bills a public IP for $AUTO"
|
||||
|
||||
aws_spx "ec2 start-instances --instance-ids $INSTANCE" >/dev/null
|
||||
wait_for 300 "instance running again" is_state "$INSTANCE" running
|
||||
RESTARTED=$(public_ip_of "$INSTANCE")
|
||||
is_set "$RESTARTED" "restarted instance has an auto-assigned address"
|
||||
if [[ $RESTARTED == "$AUTO" ]]; then
|
||||
fail "restart returned the same address $AUTO — the stop did not release it"
|
||||
else
|
||||
ok "restart returned a different address ($AUTO -> $RESTARTED)"
|
||||
fi
|
||||
is_set "$(reserved_ip_ocid "$RESTARTED")" "OCI holds a public IP object for $RESTARTED"
|
||||
fi
|
||||
|
||||
# After the restart above, so the move in step 4 is measured from where the
|
||||
# guest actually is rather than from where it started the script.
|
||||
START_NODE=$(node_of "$INSTANCE")
|
||||
echo "instance now on $START_NODE"
|
||||
|
||||
say "1. allocate and associate an EIP"
|
||||
ALLOC=$(aws_spx "ec2 allocate-address --query AllocationId --output text" | tr -d '\r')
|
||||
EIP=$(aws_spx "ec2 describe-addresses --allocation-ids $ALLOC --query 'Addresses[0].PublicIp' --output text" | tr -d '\r')
|
||||
aws_spx "ec2 associate-address --allocation-id $ALLOC --network-interface-id $ENI" >/dev/null
|
||||
echo "allocated $ALLOC = $EIP"
|
||||
wait_for 300 "EIP $EIP answers on $START_NODE" reachable "$EIP"
|
||||
|
||||
PRIVATE=$(private_half_of "$EIP")
|
||||
echo "private half: ${PRIVATE:-<unresolved>}"
|
||||
is_set "$PRIVATE" "the EIP resolves to an OCI private address"
|
||||
|
||||
say "2. DescribeInstances must report the EIP, not the auto-assigned address"
|
||||
eq "$(public_ip_of "$INSTANCE")" "$EIP" "DescribeInstances reports the EIP"
|
||||
|
||||
say "3. stop the guest"
|
||||
aws_spx "ec2 stop-instances --instance-ids $INSTANCE" >/dev/null
|
||||
wait_for 180 "instance stopped" is_state "$INSTANCE" stopped
|
||||
|
||||
unset_or_none "$(public_ip_of "$INSTANCE")" "stopped instance reports no PublicIpAddress"
|
||||
|
||||
# The EIP's own object is the customer's and must outlive the stop — the exact
|
||||
# opposite of what step 0b asserted for the borrowed address.
|
||||
is_set "$(reserved_ip_ocid "$EIP")" "OCI still holds the reserved public IP for $EIP"
|
||||
|
||||
STILL=$(aws_spx "ec2 describe-addresses --allocation-ids $ALLOC --query 'Addresses[0].AssociationId' --output text" | tr -d '\r')
|
||||
is_set "$STILL" "EIP stays associated across the stop"
|
||||
|
||||
say "4. force a start on a different node"
|
||||
on "${NODE_IP[$START_NODE]}" 'sudo systemctl stop spinifex-daemon'
|
||||
restore_daemon() { on "${NODE_IP[$START_NODE]}" 'sudo systemctl start spinifex-daemon' || true; }
|
||||
trap 'restore_daemon; cleanup' EXIT
|
||||
|
||||
aws_spx "ec2 start-instances --instance-ids $INSTANCE" >/dev/null
|
||||
sleep 20
|
||||
NEW_NODE=$(node_of "$INSTANCE")
|
||||
echo "now on $NEW_NODE (was $START_NODE)"
|
||||
[[ $NEW_NODE != "$START_NODE" ]] || { fail "guest did not move; nothing to prove"; exit 1; }
|
||||
|
||||
say "5. the decisive checks"
|
||||
wait_for 240 "EIP $EIP answers after the move to $NEW_NODE" reachable "$EIP"
|
||||
|
||||
if [[ -n ${PRIVATE:-} ]]; then
|
||||
WANT=$(external_vnic_of "$NEW_NODE"); GOT=$(vnic_of_private "$PRIVATE")
|
||||
eq "$GOT" "$WANT" "OCI carries $PRIVATE on $NEW_NODE's VNIC"
|
||||
fi
|
||||
|
||||
eq "$(public_ip_of "$INSTANCE")" "$EIP" "still reports the EIP after the move"
|
||||
|
||||
say "6. restore"
|
||||
restore_daemon
|
||||
wait_for 120 "$START_NODE prunes its stale route for $PRIVATE" bash -c \
|
||||
"! timeout 60 ssh -i $KEY -o StrictHostKeyChecking=no ubuntu@${NODE_IP[$START_NODE]} \
|
||||
'ip route show dev spx-nat-host' | grep -q '${PRIVATE:-__none__}'"
|
||||
|
||||
(( FAILED )) && { echo; echo "RESULT: FAILED"; exit 1; }
|
||||
echo; echo "RESULT: PASSED"
|
||||
@@ -0,0 +1,7 @@
|
||||
# Upstream created the compartment, an IAM group and a root policy. We do not:
|
||||
# those are tenancy-root resources, and the API user Spinifex runs as is scoped to
|
||||
# a compartment on purpose. Deploying into an existing compartment is the whole
|
||||
# point -- creating one needs rights no node should ever hold.
|
||||
data "oci_identity_compartment" "mulgadc" {
|
||||
id = var.compartment_ocid
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
data "oci_core_services" "oracle_services_network" {}
|
||||
|
||||
locals {
|
||||
all_oracle_services = one([for s in data.oci_core_services.oracle_services_network.services : s if strcontains(s.name, "Services In Oracle Services Network")])
|
||||
}
|
||||
|
||||
resource "oci_core_vcn" "mulgadc" {
|
||||
compartment_id = var.compartment_ocid
|
||||
display_name = "${var.deployment_name}-vcn"
|
||||
dns_label = var.deployment_name
|
||||
cidr_blocks = [var.vcn_cidr]
|
||||
}
|
||||
resource "oci_core_internet_gateway" "mulgadc" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-igw"
|
||||
enabled = true
|
||||
}
|
||||
resource "oci_core_nat_gateway" "mulgadc" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-nat"
|
||||
}
|
||||
resource "oci_core_service_gateway" "mulgadc" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-sgw"
|
||||
services { service_id = local.all_oracle_services.id }
|
||||
}
|
||||
resource "oci_core_route_table" "public" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-public-rt"
|
||||
route_rules {
|
||||
network_entity_id = oci_core_internet_gateway.mulgadc.id
|
||||
destination = "0.0.0.0/0"
|
||||
destination_type = "CIDR_BLOCK"
|
||||
}
|
||||
}
|
||||
resource "oci_core_route_table" "private" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-private-rt"
|
||||
route_rules {
|
||||
network_entity_id = oci_core_nat_gateway.mulgadc.id
|
||||
destination = "0.0.0.0/0"
|
||||
destination_type = "CIDR_BLOCK"
|
||||
}
|
||||
route_rules {
|
||||
network_entity_id = oci_core_service_gateway.mulgadc.id
|
||||
destination = local.all_oracle_services.cidr_block
|
||||
destination_type = "SERVICE_CIDR_BLOCK"
|
||||
}
|
||||
}
|
||||
|
||||
# Private-subnet ingress is limited to traffic originating inside this VCN.
|
||||
# Egress remains open so private workloads can use the NAT and Service Gateway.
|
||||
resource "oci_core_security_list" "private" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-private-sl"
|
||||
|
||||
ingress_security_rules {
|
||||
protocol = "6" # TCP
|
||||
source = var.vcn_cidr
|
||||
|
||||
tcp_options {
|
||||
min = 22
|
||||
max = 22
|
||||
}
|
||||
}
|
||||
|
||||
ingress_security_rules {
|
||||
protocol = "1" # ICMP
|
||||
source = var.vcn_cidr
|
||||
|
||||
icmp_options {
|
||||
type = 8 # Echo request (ping)
|
||||
code = 0
|
||||
}
|
||||
}
|
||||
|
||||
egress_security_rules {
|
||||
protocol = "all"
|
||||
destination = "0.0.0.0/0"
|
||||
}
|
||||
}
|
||||
# Deliberately permissive, and that is the design rather than a shortcut.
|
||||
#
|
||||
# A Spinifex guest's firewall is its AWS security group, enforced by OVN on the
|
||||
# node. A customer who opens 443 on their own security group expects 443 to
|
||||
# work, and they have no way to know an OCI security list exists, let alone to
|
||||
# edit one. Enumerating ports here would mean every customer port needed a
|
||||
# Terraform change, and the failure would present as "my security group does
|
||||
# nothing" — which is the hardest kind of network fault to trace.
|
||||
#
|
||||
# So the layers are: OCI security list open, OVN security groups enforce the
|
||||
# guest policy, and the host firewall (cloud-init/open-service-ports.sh) guards
|
||||
# the node's own listeners. Narrow `node_client_cidr_allow_list` to restrict
|
||||
# who can reach the deployment at all; the intra-VCN rule is separate so that
|
||||
# narrowing it can never cut Geneve, OVN, NATS or predastore between nodes.
|
||||
resource "oci_core_security_list" "public" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-public-sl"
|
||||
|
||||
ingress_security_rules {
|
||||
protocol = "all"
|
||||
source = var.vcn_cidr
|
||||
}
|
||||
|
||||
dynamic "ingress_security_rules" {
|
||||
for_each = var.node_client_cidr_allow_list
|
||||
content {
|
||||
protocol = "all"
|
||||
source = ingress_security_rules.value
|
||||
}
|
||||
}
|
||||
|
||||
egress_security_rules {
|
||||
protocol = "all"
|
||||
destination = "0.0.0.0/0"
|
||||
}
|
||||
}
|
||||
|
||||
resource "oci_core_subnet" "public" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-public-subnet"
|
||||
dns_label = "public"
|
||||
cidr_block = var.public_subnet_cidr
|
||||
route_table_id = oci_core_route_table.public.id
|
||||
prohibit_public_ip_on_vnic = false
|
||||
security_list_ids = [oci_core_security_list.public.id]
|
||||
}
|
||||
resource "oci_core_subnet" "private" {
|
||||
compartment_id = var.compartment_ocid
|
||||
vcn_id = oci_core_vcn.mulgadc.id
|
||||
display_name = "${var.deployment_name}-private-subnet"
|
||||
dns_label = "private"
|
||||
cidr_block = var.private_subnet_cidr
|
||||
route_table_id = oci_core_route_table.private.id
|
||||
prohibit_public_ip_on_vnic = true
|
||||
# Retain the VCN default list and add the private workload rules.
|
||||
security_list_ids = [oci_core_vcn.mulgadc.default_security_list_id, oci_core_security_list.private.id]
|
||||
}
|
||||
|
||||
# Deliberately absent: DRG, remote peering connection, local peering gateway, and RZG attachment.
|
||||
@@ -0,0 +1,26 @@
|
||||
output "compartment_name" { value = data.oci_identity_compartment.mulgadc.name }
|
||||
output "compartment_ocid" { value = var.compartment_ocid }
|
||||
output "vcn_ocid" { value = oci_core_vcn.mulgadc.id }
|
||||
output "subnet_ocids" { value = { public = oci_core_subnet.public.id, private = oci_core_subnet.private.id } }
|
||||
output "gateway_ocids" { value = { internet_gateway = oci_core_internet_gateway.mulgadc.id, nat_gateway = oci_core_nat_gateway.mulgadc.id, service_gateway = oci_core_service_gateway.mulgadc.id } }
|
||||
output "route_table_ocids" { value = { public = oci_core_route_table.public.id, private = oci_core_route_table.private.id } }
|
||||
output "security_list_ocids" { value = { public = oci_core_security_list.public.id, private = oci_core_security_list.private.id } }
|
||||
output "compute_ocids" { value = [for instance in oci_core_instance.mulgadc : instance.id] }
|
||||
output "data_volume_ocids" { value = [for volume in oci_core_volume.mulgadc_data : volume.id] }
|
||||
|
||||
# What the Spinifex deploy path consumes. The primary VNIC is the host plane, so
|
||||
# these are the addresses update-nodes.sh and install-node.sh connect to.
|
||||
output "nodes" {
|
||||
value = [for instance in oci_core_instance.mulgadc : {
|
||||
name = instance.display_name
|
||||
public_ip = instance.public_ip
|
||||
private_ip = instance.private_ip
|
||||
fault_domain = instance.fault_domain
|
||||
}]
|
||||
}
|
||||
|
||||
# Ready for `update-nodes.sh --hosts-file`, so the handoff between Terraform and
|
||||
# the deploy scripts is a file rather than a copy-paste.
|
||||
output "hosts_file" {
|
||||
value = join("\n", [for instance in oci_core_instance.mulgadc : instance.public_ip])
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
provider "oci" {
|
||||
tenancy_ocid = var.tenancy_ocid
|
||||
user_ocid = var.user_ocid
|
||||
fingerprint = var.fingerprint
|
||||
private_key_path = var.private_key_path
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# The aliased home-region provider is gone with the identity resources it served.
|
||||
# Everything here now lives in one region, in an existing compartment.
|
||||
@@ -0,0 +1,2 @@
|
||||
# Dependencies used only by scripts/oci_env.py in its isolated .venv.
|
||||
oci>=2.160.0,<3.0.0
|
||||
Executable
+157
@@ -0,0 +1,157 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prepare OCI-profile Terraform environment variables in this repo's .venv.
|
||||
|
||||
Examples:
|
||||
eval "$(python3 scripts/oci_env.py --shell)"
|
||||
python3 scripts/oci_env.py -- terraform plan -out=mulgadc.tfplan
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import configparser
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import venv
|
||||
|
||||
|
||||
REPOSITORY = Path(__file__).resolve().parent.parent
|
||||
VENV_PYTHON = REPOSITORY / ".venv" / "bin" / "python"
|
||||
REQUIREMENTS = REPOSITORY / "requirements.txt"
|
||||
REQUESTED_PROFILE = "apacanzset03child03"
|
||||
FALLBACK_PROFILE = "apacanzset03child3"
|
||||
|
||||
|
||||
def ensure_venv() -> None:
|
||||
"""Create, populate, then re-execute from this repository's virtualenv."""
|
||||
if Path(sys.executable).resolve() == VENV_PYTHON.resolve():
|
||||
return
|
||||
if not VENV_PYTHON.exists():
|
||||
venv.EnvBuilder(with_pip=True).create(VENV_PYTHON.parent.parent)
|
||||
subprocess.check_call([str(VENV_PYTHON), "-m", "pip", "install", "--quiet", "-r", str(REQUIREMENTS)])
|
||||
os.execv(str(VENV_PYTHON), [str(VENV_PYTHON), str(Path(__file__).resolve()), *sys.argv[1:]])
|
||||
|
||||
|
||||
def load_profile(config_path: Path, requested_profile: str) -> tuple[str, dict[str, str]]:
|
||||
parser = configparser.RawConfigParser()
|
||||
parser.read(config_path)
|
||||
profile = requested_profile
|
||||
if not parser.has_section(profile):
|
||||
if profile == REQUESTED_PROFILE and parser.has_section(FALLBACK_PROFILE):
|
||||
print(
|
||||
f"Profile {REQUESTED_PROFILE} not found; using {FALLBACK_PROFILE}.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
profile = FALLBACK_PROFILE
|
||||
else:
|
||||
raise ValueError(f"OCI profile {profile!r} was not found in {config_path}")
|
||||
|
||||
# Validate through the OCI SDK installed in the dedicated virtualenv.
|
||||
import oci
|
||||
|
||||
values = oci.config.from_file(str(config_path), profile)
|
||||
required = ("tenancy", "user", "fingerprint", "key_file", "region")
|
||||
missing = [key for key in required if not values.get(key)]
|
||||
if missing:
|
||||
raise ValueError(f"Missing OCI profile values: {', '.join(missing)}")
|
||||
return profile, values
|
||||
|
||||
|
||||
def tenancy_home_region(values: dict[str, str]) -> str:
|
||||
"""Read the home region from OCI rather than assuming the profile region."""
|
||||
import oci
|
||||
|
||||
subscriptions = oci.identity.IdentityClient(values).list_region_subscriptions(values["tenancy"]).data
|
||||
home = next((item.region_name for item in subscriptions if item.is_home_region), None)
|
||||
if not home:
|
||||
raise ValueError("OCI did not return a tenancy home region")
|
||||
return home
|
||||
|
||||
|
||||
def terraform_environment(
|
||||
profile: str,
|
||||
values: dict[str, str],
|
||||
region_override: str | None,
|
||||
ssh_public_key_path: str | None,
|
||||
ssh_private_key_path: str | None,
|
||||
) -> dict[str, str]:
|
||||
environment = {
|
||||
"OCI_CLI_PROFILE": profile,
|
||||
"TF_VAR_tenancy_ocid": values["tenancy"],
|
||||
"TF_VAR_user_ocid": values["user"],
|
||||
"TF_VAR_fingerprint": values["fingerprint"],
|
||||
"TF_VAR_private_key_path": values["key_file"],
|
||||
"TF_VAR_region": region_override or values["region"],
|
||||
"TF_VAR_home_region": tenancy_home_region(values),
|
||||
}
|
||||
if ssh_public_key_path:
|
||||
key_path = Path(ssh_public_key_path).expanduser().resolve()
|
||||
if not key_path.is_file():
|
||||
raise FileNotFoundError(f"SSH public key was not found at {key_path}")
|
||||
environment["TF_VAR_ssh_public_key_path"] = str(key_path)
|
||||
if ssh_private_key_path:
|
||||
key_path = Path(ssh_private_key_path).expanduser().resolve()
|
||||
if not key_path.is_file():
|
||||
raise FileNotFoundError(f"SSH private key was not found at {key_path}")
|
||||
environment["TF_VAR_ssh_private_key_path"] = str(key_path)
|
||||
return environment
|
||||
|
||||
|
||||
def main() -> int:
|
||||
arg_parser = argparse.ArgumentParser()
|
||||
arg_parser.add_argument("--profile", default=REQUESTED_PROFILE)
|
||||
arg_parser.add_argument("--region")
|
||||
arg_parser.add_argument(
|
||||
"--ssh-public-key-path",
|
||||
help="Optional public-key path exported as TF_VAR_ssh_public_key_path.",
|
||||
)
|
||||
arg_parser.add_argument(
|
||||
"--ssh-private-key-path",
|
||||
help="Optional private-key path exported as TF_VAR_ssh_private_key_path.",
|
||||
)
|
||||
arg_parser.add_argument("--shell", action="store_true", help="Print shell exports for eval.")
|
||||
arg_parser.add_argument("command", nargs=argparse.REMAINDER, help="Command to run with the environment; prefix with --.")
|
||||
args = arg_parser.parse_args()
|
||||
if args.shell and args.command:
|
||||
arg_parser.error("--shell and a command cannot be used together")
|
||||
|
||||
ensure_venv()
|
||||
config_path = Path(os.environ.get("OCI_CONFIG_FILE", Path.home() / ".oci" / "config"))
|
||||
if not config_path.is_file():
|
||||
raise FileNotFoundError(f"OCI configuration was not found at {config_path}")
|
||||
profile, values = load_profile(config_path, args.profile)
|
||||
environment = terraform_environment(
|
||||
profile,
|
||||
values,
|
||||
args.region,
|
||||
args.ssh_public_key_path,
|
||||
args.ssh_private_key_path,
|
||||
)
|
||||
|
||||
if args.shell:
|
||||
for key, value in environment.items():
|
||||
print(f"export {key}={shlex.quote(value)}")
|
||||
return 0
|
||||
|
||||
if not args.command:
|
||||
print(f"Loaded OCI profile {profile} for region {environment['TF_VAR_region']}; virtualenv: {VENV_PYTHON.parent.parent}")
|
||||
return 0
|
||||
|
||||
command = args.command[1:] if args.command[0] == "--" else args.command
|
||||
if not command:
|
||||
raise ValueError("A command is required after --")
|
||||
child_environment = os.environ.copy()
|
||||
child_environment.update(environment)
|
||||
print(f"Running with OCI profile {profile} in {environment['TF_VAR_region']}: {' '.join(map(shlex.quote, command))}")
|
||||
return subprocess.call(command, cwd=REPOSITORY, env=child_environment)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (FileNotFoundError, ValueError, subprocess.CalledProcessError) as error:
|
||||
print(f"Error: {error}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,12 @@
|
||||
# Example non-secret overrides. Copy to the untracked terraform.auto.tfvars.
|
||||
|
||||
# Restrict who may reach a node's SSH and Spinifex service ports from the internet.
|
||||
node_client_cidr_allow_list = ["203.0.113.10/32"]
|
||||
|
||||
# 1 for the single-node replica, 3 for a cluster.
|
||||
node_count = 1
|
||||
|
||||
# Bare metal is preferred; this is what we test on until the quota is raised.
|
||||
compute_shape = "VM.Standard.E6.Flex"
|
||||
compute_ocpus = 8
|
||||
compute_memory_in_gbs = 32
|
||||
@@ -0,0 +1,167 @@
|
||||
variable "tenancy_ocid" { type = string }
|
||||
variable "user_ocid" { type = string }
|
||||
variable "fingerprint" { type = string }
|
||||
variable "private_key_path" { type = string }
|
||||
# OCI's own region name. Spinifex's region is a separate setting that follows AWS
|
||||
# naming, so ap-sydney-1 here beside ap-southeast-2 in spinifex.toml is correct.
|
||||
variable "region" {
|
||||
type = string
|
||||
default = "ap-sydney-1"
|
||||
}
|
||||
# Unused. Kept declared because scripts/oci_env.py discovers and exports it, and
|
||||
# there are no identity resources left that need the home-region endpoint.
|
||||
variable "home_region" {
|
||||
description = "Unused. The tenancy home region, exported by the Python environment tool."
|
||||
type = string
|
||||
default = null
|
||||
nullable = true
|
||||
}
|
||||
# Required, and deliberately an OCID rather than a name. This deploys into a
|
||||
# compartment that already exists; creating one needs tenancy-root rights that the
|
||||
# API user Spinifex runs as must not have, and a name lookup can match a deleted one.
|
||||
variable "compartment_ocid" {
|
||||
description = "OCID of the existing compartment to deploy into."
|
||||
type = string
|
||||
}
|
||||
|
||||
# Prefixes every resource this config owns, so nothing it creates can be confused
|
||||
# with -- or collide with -- anything built by hand in the same compartment. The
|
||||
# VCN dns_label takes this value directly, so keep it short and alphanumeric.
|
||||
variable "deployment_name" {
|
||||
description = "Name prefix for every resource this configuration creates."
|
||||
type = string
|
||||
default = "spinifex"
|
||||
|
||||
validation {
|
||||
condition = can(regex("^[a-z][a-z0-9]{0,14}$", var.deployment_name))
|
||||
error_message = "deployment_name must be 1-15 lowercase alphanumeric characters starting with a letter, because it is used as the VCN dns_label."
|
||||
}
|
||||
}
|
||||
variable "vcn_cidr" {
|
||||
description = "VCN address space. A /22 leaves room for the per-node secondary private IPs Spinifex allocates for external addresses."
|
||||
type = string
|
||||
default = "10.200.0.0/22"
|
||||
}
|
||||
variable "public_subnet_cidr" {
|
||||
description = "Subnet every Spinifex node sits in, on both of its VNICs. Nodes serve the UI, predastore and the AWS gateway directly, so they cannot live in a private subnet."
|
||||
type = string
|
||||
default = "10.200.0.0/23"
|
||||
}
|
||||
variable "private_subnet_cidr" {
|
||||
description = "Unused by Spinifex today; guests live on the OVN overlay, not in an OCI subnet. Kept so the subnet exists if something later needs it."
|
||||
type = string
|
||||
default = "10.200.2.0/24"
|
||||
}
|
||||
|
||||
variable "compute_shape" {
|
||||
description = "Compute shape for each Spinifex node. Bare metal is preferred; VM.Standard.E6.Flex is what we test on until the bare-metal quota is raised."
|
||||
type = string
|
||||
default = "VM.Standard.E6.Flex"
|
||||
}
|
||||
|
||||
# A flex shape carries no size of its own, so these are not tuning knobs -- without
|
||||
# them the shape is unbuildable. Both are the minimum recommended for a node.
|
||||
variable "compute_ocpus" {
|
||||
description = "OCPUs per node on a flex shape. OCI counts an OCPU as a full core, so 8 OCPU is 16 vCPU. Ignored on a fixed bare-metal shape."
|
||||
type = number
|
||||
default = 8
|
||||
|
||||
validation {
|
||||
condition = var.compute_ocpus >= 8
|
||||
error_message = "compute_ocpus must be at least 8 (16 vCPU): the minimum recommended for a Spinifex node."
|
||||
}
|
||||
}
|
||||
|
||||
variable "compute_memory_in_gbs" {
|
||||
description = "Memory per node in GiB on a flex shape. Ignored on a fixed bare-metal shape."
|
||||
type = number
|
||||
default = 32
|
||||
|
||||
validation {
|
||||
condition = var.compute_memory_in_gbs >= 32
|
||||
error_message = "compute_memory_in_gbs must be at least 32: the minimum recommended for a Spinifex node."
|
||||
}
|
||||
}
|
||||
|
||||
variable "node_count" {
|
||||
description = "Number of Spinifex nodes, each with its own data volume. 1 for the single-node replica, 3 for a cluster."
|
||||
type = number
|
||||
default = 1
|
||||
|
||||
validation {
|
||||
condition = var.node_count >= 1 && var.node_count <= 25 && floor(var.node_count) == var.node_count
|
||||
error_message = "node_count must be a whole number between 1 and 25."
|
||||
}
|
||||
}
|
||||
|
||||
variable "ubuntu_version" {
|
||||
description = "Canonical Ubuntu platform image version."
|
||||
type = string
|
||||
default = "26.04"
|
||||
}
|
||||
|
||||
variable "data_volume_size_in_gbs" {
|
||||
description = "Size in GiB of the data block volume that backs /var/lib/spinifex on each node."
|
||||
type = number
|
||||
default = 256
|
||||
}
|
||||
|
||||
variable "data_mountpoint" {
|
||||
description = "Where cloud-init mounts the data volume. Spinifex's data directory itself, so one mount rather than a mount plus a redirect, and no config points anywhere unusual."
|
||||
type = string
|
||||
default = "/var/lib/spinifex"
|
||||
}
|
||||
|
||||
variable "data_volume_vpus_per_gb" {
|
||||
description = "Performance tier for each node's data volume in VPUs per GB."
|
||||
type = number
|
||||
default = 120
|
||||
|
||||
validation {
|
||||
condition = var.data_volume_vpus_per_gb >= 0 && var.data_volume_vpus_per_gb <= 120
|
||||
error_message = "data_volume_vpus_per_gb must be between 0 and 120."
|
||||
}
|
||||
}
|
||||
|
||||
variable "ssh_public_key_path" {
|
||||
description = "Absolute path to the SSH public key installed on the Ubuntu instance."
|
||||
type = string
|
||||
default = null
|
||||
nullable = true
|
||||
}
|
||||
|
||||
# Nothing consumes this. It stays declared because scripts/oci_env.py exports it,
|
||||
# and an undeclared TF_VAR_ is easier to misread as a broken helper than as unused.
|
||||
variable "ssh_private_key_path" {
|
||||
description = "Unused. The vault that imported the private key was removed; no resource needs it."
|
||||
type = string
|
||||
default = null
|
||||
nullable = true
|
||||
}
|
||||
|
||||
variable "node_client_cidr_allow_list" {
|
||||
description = "CIDR blocks permitted to reach a node's SSH and Spinifex service ports from the internet."
|
||||
type = list(string)
|
||||
default = ["0.0.0.0/0"]
|
||||
}
|
||||
|
||||
# The host firewall only. The OCI security list is deliberately open, because a
|
||||
# guest's policy is its AWS security group and OVN enforces that on the node —
|
||||
# see the comment on oci_core_security_list.public.
|
||||
variable "node_service_ports" {
|
||||
description = "TCP ports the node itself serves: console, predastore gate, AWS gateway. SSH is always opened separately."
|
||||
type = list(number)
|
||||
default = [3000, 8443, 9999]
|
||||
}
|
||||
|
||||
variable "wan_bridge_name" {
|
||||
description = "Linux bridge built over the secondary VNIC to carry the external datapath."
|
||||
type = string
|
||||
default = "br-wan"
|
||||
}
|
||||
|
||||
variable "wan_bridge_mtu" {
|
||||
description = "MTU for the WAN bridge and its member interface. The OCI VCN carries 9000."
|
||||
type = number
|
||||
default = 9000
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers { oci = { source = "oracle/oci", version = ">= 6.0.0" } }
|
||||
}
|
||||
Executable
+533
@@ -0,0 +1,533 @@
|
||||
#!/bin/bash
|
||||
# uninstall-spx.sh — remove a Spinifex install from this node.
|
||||
#
|
||||
# The inverse of setup.sh. Where node-reset.sh clears *state* and leaves the
|
||||
# node installed and ready to re-init, this removes the *software*: the binary,
|
||||
# the units, the service users, the helper scripts and the configuration setup.sh
|
||||
# wrote. State teardown is delegated to node-reset.sh rather than reimplemented.
|
||||
#
|
||||
# Usage:
|
||||
# sudo /usr/local/share/spinifex/uninstall-spx.sh [options]
|
||||
#
|
||||
# Options:
|
||||
# --purge-data Also remove /var/lib/spinifex — volumes, S3 objects, JetStream.
|
||||
# Off by default: uninstalling software should not destroy data.
|
||||
# --purge-deps Also apt-remove the packages setup.sh installed. Names them and
|
||||
# confirms first, even under --yes. Off by default because a
|
||||
# curl|bash install lands on hosts that were already using them.
|
||||
# --yes Skip the confirmation prompt. Does NOT cover --purge-data,
|
||||
# --purge-deps, or the multi-node refusal.
|
||||
# --dry-run Print the whole plan, touch nothing, exit 0.
|
||||
#
|
||||
# Environment:
|
||||
# SPX_PURGE_CONFIRM=destroy Unattended confirmation for --purge-data only.
|
||||
# SPX_ETC_DIR Override /etc/spinifex. Exists so the multi-node
|
||||
# refusal can be tested against a fixture config.
|
||||
#
|
||||
# WHAT THIS REMOVES
|
||||
#
|
||||
# /usr/local/bin/spx, the nbdkit plugin, /usr/local/{share,lib}/spinifex,
|
||||
# /usr/share/spinifex, every spinifex systemd unit and slice, the eight
|
||||
# spinifex-* service users, the sudoers grant, the sysctl/logrotate/chrony/udev
|
||||
# drop-ins, the host firewall policy and its loaded nft table, the OVS bridges
|
||||
# and OVN databases, and the Spinifex CA from the host trust store.
|
||||
#
|
||||
# WHAT THIS LEAVES ALONE
|
||||
#
|
||||
# /var/lib/spinifex unless --purge-data. Every apt package unless --purge-deps.
|
||||
# /swapfile, which setup.sh may have created but cannot be told apart from one
|
||||
# the host already had. Anything the operator configured around us: netplan,
|
||||
# /etc/iptables/rules.v4, hand-written bridges.
|
||||
set -euo pipefail
|
||||
|
||||
ETC_DIR="${SPX_ETC_DIR:-/etc/spinifex}"
|
||||
DATA_DIR=/var/lib/spinifex
|
||||
LOG_DIR=/var/log/spinifex
|
||||
RUN_DIR=/run/spinifex
|
||||
CONFIG_FILE="$ETC_DIR/spinifex.toml"
|
||||
SHARE_DIR=/usr/local/share/spinifex
|
||||
NODE_RESET="$SHARE_DIR/node-reset.sh"
|
||||
|
||||
PURGE_DATA=false
|
||||
PURGE_DEPS=false
|
||||
ASSUME_YES=false
|
||||
DRY_RUN=false
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--purge-data) PURGE_DATA=true ;;
|
||||
--purge-deps) PURGE_DEPS=true ;;
|
||||
--yes | -y) ASSUME_YES=true ;;
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
-h | --help)
|
||||
# Header comment block, minus the shebang. Taken up to the first
|
||||
# non-comment line rather than by line number, which drifts.
|
||||
sed -n '2,/^[^#]/p' "$0" | sed '/^[^#]/d; s/^# \?//'
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: unknown option: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
log() { echo "[uninstall-spx] $*"; }
|
||||
run() {
|
||||
if $DRY_RUN; then
|
||||
echo " would run: $*"
|
||||
return 0
|
||||
fi
|
||||
"$@"
|
||||
}
|
||||
# Paths are removed through one helper so --dry-run reports them uniformly and
|
||||
# an absent path is never an error. Uninstalling a partial install is the
|
||||
# ordinary case, not an exception.
|
||||
#
|
||||
# The existence test goes through sudo, not a bare [ -e ]. /etc/sudoers.d is
|
||||
# 0750 root:root, so an unprivileged test reports "absent" for a file that is
|
||||
# plainly there and the dry run then under-reports what it is about to delete —
|
||||
# which is the one thing a dry run must never do.
|
||||
exists() { sudo test -e "$1" || sudo test -L "$1"; }
|
||||
rm_path() {
|
||||
for p in "$@"; do
|
||||
exists "$p" || continue
|
||||
if $DRY_RUN; then
|
||||
echo " would remove: $p"
|
||||
else
|
||||
sudo rm -rf "$p"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Refuse on a multi-node cluster that is still up.
|
||||
#
|
||||
# Uninstalling one node of a running cluster strands its guests and leaves the
|
||||
# survivors reconciling against a chassis that will never answer. This is a
|
||||
# structural check, not a prompt, so --yes does not cover it: a flag meaning
|
||||
# "don't ask me" must not also mean "ignore a safety property".
|
||||
# ---------------------------------------------------------------------------
|
||||
NODES=()
|
||||
if [ -r "$CONFIG_FILE" ]; then
|
||||
mapfile -t NODES < <(sudo grep -oP '^\[nodes\.\K[^.\]]+' "$CONFIG_FILE" 2>/dev/null | sort -u || true)
|
||||
fi
|
||||
|
||||
if [ "${#NODES[@]}" -gt 1 ]; then
|
||||
cat >&2 <<EOF
|
||||
ERROR: this node is configured as part of a ${#NODES[@]}-node cluster:
|
||||
${NODES[*]}
|
||||
|
||||
Uninstalling one node of a running cluster strands its guests and leaves the
|
||||
other nodes reconciling against a chassis that will never answer. Take the
|
||||
cluster down first:
|
||||
|
||||
spx admin cluster shutdown
|
||||
|
||||
then run this on each node. --yes does not override this check.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Report what is at stake, then confirm.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "on $(hostname): removing the Spinifex install"
|
||||
if command -v spx >/dev/null 2>&1; then
|
||||
log " version: $(spx version 2>/dev/null | head -1 || echo unknown)"
|
||||
fi
|
||||
if [ -d "$DATA_DIR" ]; then
|
||||
size=$(sudo du -sh "$DATA_DIR" 2>/dev/null | cut -f1 || true)
|
||||
instances=$(sudo find "$DATA_DIR/instances" -maxdepth 1 -mindepth 1 -type d 2>/dev/null | wc -l || true)
|
||||
volumes=$(sudo find "$DATA_DIR/volumes" -maxdepth 1 -mindepth 1 2>/dev/null | wc -l || true)
|
||||
if $PURGE_DATA; then
|
||||
log " --purge-data: DESTROYING $instances instance(s), $volumes volume(s), ${size:-unknown} under $DATA_DIR"
|
||||
else
|
||||
log " keeping $DATA_DIR ($instances instance(s), $volumes volume(s), ${size:-unknown}) — pass --purge-data to remove it"
|
||||
fi
|
||||
fi
|
||||
$PURGE_DEPS && log " --purge-deps: apt packages will be removed (confirmed separately below)"
|
||||
|
||||
if ! $ASSUME_YES && ! $DRY_RUN; then
|
||||
read -r -p "Remove the Spinifex install from this node? Type 'uninstall' to continue: " reply
|
||||
[ "$reply" = "uninstall" ] || { log "aborted"; exit 1; }
|
||||
fi
|
||||
|
||||
# --purge-data is a second decision with a different blast radius, so it takes
|
||||
# its own confirmation even when --yes was given for the uninstall itself.
|
||||
#
|
||||
# SPX_PURGE_CONFIRM=destroy is the unattended escape, matching node-reset.sh's
|
||||
# SPX_RESET_CONFIRM and install-node.sh's SPX_WIPE_CONFIRM. It is deliberately a
|
||||
# separate variable from --yes: a caller that wants no prompts still has to name
|
||||
# the data destruction specifically, so no existing automation acquires it by
|
||||
# adding --yes.
|
||||
if $PURGE_DATA && ! $DRY_RUN; then
|
||||
if [ "${SPX_PURGE_CONFIRM:-}" = "destroy" ]; then
|
||||
log "SPX_PURGE_CONFIRM=destroy: proceeding without a prompt"
|
||||
else
|
||||
read -r -p "Also destroy every volume and object under $DATA_DIR? Type 'destroy' to continue: " reply
|
||||
[ "$reply" = "destroy" ] || { log "aborted"; exit 1; }
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Stop everything, and confirm it stopped.
|
||||
#
|
||||
# `systemctl stop spinifex.target` returns once the target is inactive, which is
|
||||
# not the same as its services being stopped: spinifex-shutdown.service is
|
||||
# ordered After= the storage services, so their stop jobs queue behind its
|
||||
# ExecStop. Removing the binary out from under a running service leaves it
|
||||
# executing an unlinked image, and removing the users out from under one is
|
||||
# worse. node-reset.sh and update-nodes.sh both carry this same wait.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "stopping services"
|
||||
run sudo systemctl stop spinifex.target 2>/dev/null || true
|
||||
if ! $DRY_RUN; then
|
||||
for unit in spinifex-predastore spinifex-viperblock spinifex-nats; do
|
||||
if systemctl is-active --quiet "$unit"; then
|
||||
sudo systemctl stop "$unit" || true
|
||||
fi
|
||||
done
|
||||
elapsed=0
|
||||
while [ -n "$(pgrep -x spx || true)" ]; do
|
||||
if [ "$elapsed" -ge 120 ]; then
|
||||
echo "ERROR: spx still running after 120s:" >&2
|
||||
pgrep -ax spx >&2 || true
|
||||
echo " Removing the install under a live service leaves it running an" >&2
|
||||
echo " unlinked binary. Stop them manually and re-run." >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
elapsed=$((elapsed + 2))
|
||||
done
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. Delegate state teardown to node-reset.sh.
|
||||
#
|
||||
# It handles the guests, the OVS bridges, the OVN databases and the JetStream
|
||||
# store, and it carries correctness that is invisible in its code: QEMU matched
|
||||
# as ^qemu-system because comm truncates at 15 characters, a wait for QEMU to
|
||||
# exit rather than for the signal, and the firewall-mode stash that stops an
|
||||
# interrupted run leaving the node unarmed. A second copy of that would drift,
|
||||
# and the failure mode of the drift is data loss.
|
||||
#
|
||||
# --keep-data unless we are purging: node-reset's default is to wipe, ours is
|
||||
# not, and the flag is how the two defaults are reconciled.
|
||||
# ---------------------------------------------------------------------------
|
||||
if [ -x "$NODE_RESET" ]; then
|
||||
log "clearing node state via $NODE_RESET"
|
||||
reset_args=(--yes)
|
||||
$PURGE_DATA || reset_args+=(--keep-data)
|
||||
$DRY_RUN && reset_args+=(--dry-run)
|
||||
if ! sudo "$NODE_RESET" "${reset_args[@]}"; then
|
||||
echo "ERROR: node-reset.sh failed; stopping before removing the install." >&2
|
||||
echo " The node still has its software, so it can be repaired and retried." >&2
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
log "NOTE: $NODE_RESET is not installed — skipping the state teardown"
|
||||
log " guests, OVS bridges and OVN databases are left as they are"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. Unload the host firewall before deleting the files behind it.
|
||||
#
|
||||
# The nft ruleset lives in the kernel; removing spinifex.nft does not unload it.
|
||||
# A node that uninstalls Spinifex and keeps a default-deny input policy with no
|
||||
# file behind it is unreachable after the next reboot and unexplainable to
|
||||
# whoever inherits it.
|
||||
#
|
||||
# Only our own table and our own marked rules, never a global flush: setup.sh
|
||||
# creates exactly `table inet spinifex_filter`, and vpcd writes iptables rules
|
||||
# carrying the comments below. Anything else on this host belongs to someone
|
||||
# else.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "unloading the host firewall policy"
|
||||
if command -v nft >/dev/null 2>&1; then
|
||||
if sudo nft list table inet spinifex_filter >/dev/null 2>&1; then
|
||||
run sudo nft delete table inet spinifex_filter
|
||||
else
|
||||
log " no inet spinifex_filter table loaded"
|
||||
fi
|
||||
fi
|
||||
if command -v iptables-save >/dev/null 2>&1 && ! $DRY_RUN; then
|
||||
for marker in spinifex-nat-egress spinifex-eip-ingress spinifex-imds spinifex-imds-remap; do
|
||||
while read -r table rule; do
|
||||
[ -n "$rule" ] || continue
|
||||
# shellcheck disable=SC2086
|
||||
sudo iptables -t "$table" -D $rule 2>/dev/null || true
|
||||
done < <(sudo iptables-save | awk -v m="$marker" '
|
||||
/^\*/ { t = substr($0, 2) }
|
||||
$0 ~ m && /^-A/ { sub(/^-A /, ""); print t, $0 }')
|
||||
done
|
||||
elif $DRY_RUN; then
|
||||
echo " would remove iptables rules commented spinifex-nat-egress / spinifex-eip-ingress / spinifex-imds / spinifex-imds-remap"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6. systemd units.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "removing systemd units"
|
||||
if ! $DRY_RUN; then
|
||||
sudo systemctl disable --now spinifex.target 2>/dev/null || true
|
||||
for u in /etc/systemd/system/spinifex-*.service /etc/systemd/system/spinifex-*.timer; do
|
||||
[ -e "$u" ] || continue
|
||||
sudo systemctl disable --now "$(basename "$u")" 2>/dev/null || true
|
||||
done
|
||||
fi
|
||||
rm_path /etc/systemd/system/spinifex-*.service \
|
||||
/etc/systemd/system/spinifex-*.timer \
|
||||
/etc/systemd/system/spinifex.target \
|
||||
/etc/systemd/system/spinifex.slice \
|
||||
/etc/systemd/system/spinifex-system.slice \
|
||||
/etc/systemd/system/spinifex-guests.slice \
|
||||
/etc/systemd/system/system.slice.d/spinifex-reserve.conf
|
||||
|
||||
# Drop-ins we wrote into *other people's* units. These are the dangerous ones:
|
||||
# they are not in a spinifex-named path, so a namespace glob misses them, and
|
||||
# each holds an ExecStartPost pointing at a helper under /usr/local/lib/spinifex
|
||||
# that the step below removes. Left behind, openvswitch-switch fails to start
|
||||
# with `Failed at step EXEC ... ovs-socket-perms.sh: No such file or directory`
|
||||
# — the uninstall breaks OVS for whatever else on the host was using it. Found
|
||||
# exactly that way on the first dev-oci round trip.
|
||||
log "removing drop-ins written into other units"
|
||||
rm_path /etc/systemd/system/openvswitch-switch.service.d/spinifex-perms.conf \
|
||||
/etc/systemd/system/ovn-controller.service.d/spinifex-perms.conf \
|
||||
/etc/systemd/system/ovn-controller.service.d/log-level.conf \
|
||||
/etc/systemd/system/openvswitch-ipsec.service.d/spinifex-perms.conf \
|
||||
/etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf
|
||||
# Only if we emptied them — another package may own the same override dir.
|
||||
for d in openvswitch-switch ovn-controller openvswitch-ipsec ovn-northd; do
|
||||
dir="/etc/systemd/system/${d}.service.d"
|
||||
if sudo test -d "$dir" && [ -z "$(sudo ls -A "$dir" 2>/dev/null)" ]; then
|
||||
rm_path "$dir"
|
||||
fi
|
||||
done
|
||||
|
||||
run sudo systemctl daemon-reload
|
||||
run sudo systemctl reset-failed 2>/dev/null || true
|
||||
|
||||
# The units we just un-broke may be sitting in a failed state from an earlier
|
||||
# attempt. Restarting them here is not housekeeping: openvswitch-switch failing
|
||||
# is how the operator discovers the drop-in problem, and leaving it failed after
|
||||
# an uninstall that claims success is the same silent failure one step later.
|
||||
if ! $DRY_RUN; then
|
||||
for u in openvswitch-switch ovn-controller; do
|
||||
if systemctl is-failed --quiet "$u" 2>/dev/null; then
|
||||
log " restarting $u, which our drop-in had failed"
|
||||
sudo systemctl reset-failed "$u" 2>/dev/null || true
|
||||
sudo systemctl restart "$u" 2>/dev/null || \
|
||||
log " NOTE: $u did not restart — check it by hand"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# strongswan's charon profile: a site-override file setup-ovn.sh writes whole
|
||||
# (not appends to) granting reads on /etc/spinifex, which is about to not exist.
|
||||
if sudo test -f /etc/apparmor.d/local/usr.lib.ipsec.charon; then
|
||||
log "removing the AppArmor grant for /etc/spinifex"
|
||||
rm_path /etc/apparmor.d/local/usr.lib.ipsec.charon
|
||||
run sudo apparmor_parser -r /etc/apparmor.d/usr.lib.ipsec.charon 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 7. systemd-networkd units written by setup-ovn.sh.
|
||||
#
|
||||
# Without these the veth and transit-veth pairs are recreated on the next boot,
|
||||
# long after the software that used them is gone. Removed by glob because the
|
||||
# veth-wan set (15/16) and the routed-NAT set (17/18) are both ours and a
|
||||
# hardcoded list has already missed one of them once.
|
||||
# ---------------------------------------------------------------------------
|
||||
if compgen -G "/etc/systemd/network/1[5-8]-spinifex-*" >/dev/null; then
|
||||
log "removing systemd-networkd units"
|
||||
rm_path /etc/systemd/network/1[5-8]-spinifex-*
|
||||
run sudo networkctl reload 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 8. Installed files.
|
||||
#
|
||||
# The nbdkit plugin does not live under any spinifex-owned path — it goes into
|
||||
# nbdkit's own plugin directory — so it is the one file here that is easy to
|
||||
# leave behind. Ask nbdkit where that is rather than guessing, and fall back to
|
||||
# the multiarch paths setup.sh uses when nbdkit has already been removed.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "removing installed files"
|
||||
PLUGINDIR=$(nbdkit --dump-config 2>/dev/null | sed -n 's/^plugindir=//p' || true)
|
||||
if [ -z "$PLUGINDIR" ]; then
|
||||
for d in /usr/lib/x86_64-linux-gnu/nbdkit/plugins /usr/lib/aarch64-linux-gnu/nbdkit/plugins; do
|
||||
[ -d "$d" ] && PLUGINDIR="$d" && break
|
||||
done
|
||||
fi
|
||||
[ -n "$PLUGINDIR" ] && rm_path "$PLUGINDIR/nbdkit-viperblock-plugin.so"
|
||||
|
||||
rm_path /usr/local/bin/spx \
|
||||
/usr/local/share/spinifex \
|
||||
/usr/local/lib/spinifex \
|
||||
/usr/share/spinifex
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 9. Drop-ins. Globbed, not listed: 99-spinifex-vpc.conf is written at runtime
|
||||
# by the daemon through spinifex-set-endpoint-sysctl and does not appear
|
||||
# anywhere in setup.sh, so an inventory built by reading the installer alone
|
||||
# leaves behind the drop-in that changes forwarding and rp_filter.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "removing configuration drop-ins"
|
||||
rm_path /etc/sysctl.d/*spinifex* \
|
||||
/etc/sudoers.d/spinifex-network \
|
||||
/etc/chrony/conf.d/spinifex.conf \
|
||||
/etc/logrotate.d/spinifex \
|
||||
/etc/udev/rules.d/*spinifex*
|
||||
run sudo sysctl --system >/dev/null 2>&1 || true
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 10. Spinifex directories.
|
||||
#
|
||||
# node-reset.sh emptied these of files; what is left is the structure setup.sh
|
||||
# built, which is install state and goes now. /var/lib/spinifex only under
|
||||
# --purge-data — and even then, only if it is not a mountpoint, because on an
|
||||
# ISO-installed node several of these are separate ZFS datasets and a
|
||||
# mountpoint cannot be removed.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "removing spinifex directories"
|
||||
rm_path "$ETC_DIR" "$LOG_DIR" "$RUN_DIR"
|
||||
if $PURGE_DATA; then
|
||||
if mountpoint -q "$DATA_DIR" 2>/dev/null; then
|
||||
log "NOTE: $DATA_DIR is a mountpoint — emptying it rather than removing it"
|
||||
if ! $DRY_RUN; then
|
||||
sudo find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + 2>/dev/null || true
|
||||
else
|
||||
echo " would empty: $DATA_DIR"
|
||||
fi
|
||||
else
|
||||
rm_path "$DATA_DIR"
|
||||
fi
|
||||
else
|
||||
log " keeping $DATA_DIR"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 11. Service users and groups.
|
||||
#
|
||||
# After the directories, because a user's home directory is under them and
|
||||
# userdel warns about a missing home. Order here is cosmetic, not correctness.
|
||||
# ---------------------------------------------------------------------------
|
||||
log "removing service users and groups"
|
||||
for svc in nats gw daemon storage northstar viperblock vpcd ui; do
|
||||
user="spinifex-${svc}"
|
||||
if id -u "$user" >/dev/null 2>&1; then
|
||||
run sudo userdel "$user" 2>/dev/null || log " NOTE: could not remove user $user"
|
||||
fi
|
||||
done
|
||||
for grp in spinifex-viperblock spinifex; do
|
||||
if getent group "$grp" >/dev/null 2>&1; then
|
||||
run sudo groupdel "$grp" 2>/dev/null || log " NOTE: could not remove group $grp (still has members?)"
|
||||
fi
|
||||
done
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 12. The CA in the host trust store. Leaving it means the host trusts a CA
|
||||
# whose key no longer exists anywhere.
|
||||
# ---------------------------------------------------------------------------
|
||||
if [ -f /usr/local/share/ca-certificates/spinifex-ca.crt ]; then
|
||||
log "removing the Spinifex CA from the host trust store"
|
||||
rm_path /usr/local/share/ca-certificates/spinifex-ca.crt
|
||||
run sudo update-ca-certificates >/dev/null 2>&1 || true
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 13. Swap is reported, never removed.
|
||||
#
|
||||
# setup_swap creates an 8G /swapfile with the fstab line `/swapfile none swap sw
|
||||
# 0 0` — which is byte-identical to the line a host would already have if it
|
||||
# provisioned its own swap the ordinary way. There is no marker, so there is no
|
||||
# way to tell ours from theirs, and removing a host's swap because we might have
|
||||
# made it is not a trade worth taking. Say it is there and let the operator
|
||||
# decide.
|
||||
# ---------------------------------------------------------------------------
|
||||
if [ -f /swapfile ] && grep -q '^/swapfile ' /etc/fstab 2>/dev/null; then
|
||||
log "NOTE: /swapfile is left in place. setup.sh may have created it, but the"
|
||||
log " fstab line is indistinguishable from a host's own. To remove it:"
|
||||
log " sudo swapoff /swapfile && sudo rm -f /swapfile"
|
||||
log " then delete its line from /etc/fstab"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 14. apt packages, only when asked, and only after naming them.
|
||||
#
|
||||
# This is the one irreversible thing in the script. On a dedicated node these
|
||||
# are ours; on the host a curl|bash install actually lands on they are someone
|
||||
# else's, and openvswitch/libvirt/qemu/chrony going away underneath a running
|
||||
# workload is a far worse outcome than a few megabytes left behind. Confirmed
|
||||
# separately even under --yes, and never autoremove, which is unbounded.
|
||||
# ---------------------------------------------------------------------------
|
||||
if $PURGE_DEPS; then
|
||||
PACKAGES="nbdkit qemu-utils gdisk ovmf less libvirt-daemon-system libvirt-clients
|
||||
pciutils jq ethtool netcat-openbsd unzip xz-utils
|
||||
ovn-central ovn-host openvswitch-switch openvswitch-ipsec strongswan-charon
|
||||
chrony nftables"
|
||||
log "--purge-deps: these packages would be removed:"
|
||||
# shellcheck disable=SC2086
|
||||
for p in $PACKAGES; do echo " $p"; done
|
||||
log " NOT removed: curl, wget, file, iproute2, dhcpcd-base, qemu-system-* — too"
|
||||
log " widely depended on to remove from a host we do not own."
|
||||
if $DRY_RUN; then
|
||||
echo " would run: apt-get remove -y <the above>"
|
||||
else
|
||||
read -r -p "Remove these packages? Type 'remove' to continue: " reply
|
||||
if [ "$reply" = "remove" ]; then
|
||||
# shellcheck disable=SC2086
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get remove -y $PACKAGES || \
|
||||
log "NOTE: apt-get remove reported errors; check the output above"
|
||||
else
|
||||
log " skipped"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 15. Verify. An uninstaller that reports success over a surviving unit file is
|
||||
# worse than one that refuses, because the next install silently inherits it.
|
||||
# ---------------------------------------------------------------------------
|
||||
if ! $DRY_RUN; then
|
||||
log "verifying"
|
||||
LEFT=()
|
||||
for p in /usr/local/bin/spx /usr/local/share/spinifex /usr/local/lib/spinifex \
|
||||
/usr/share/spinifex "$ETC_DIR" "$LOG_DIR" \
|
||||
/etc/sudoers.d/spinifex-network /etc/logrotate.d/spinifex \
|
||||
/etc/chrony/conf.d/spinifex.conf; do
|
||||
exists "$p" && LEFT+=("$p")
|
||||
done
|
||||
for p in /etc/systemd/system/spinifex-*.service /etc/systemd/system/spinifex*.slice \
|
||||
/etc/systemd/system/spinifex.target /etc/sysctl.d/*spinifex* \
|
||||
/etc/systemd/network/1[5-8]-spinifex-* \
|
||||
/etc/systemd/system/*.service.d/spinifex-perms.conf \
|
||||
/etc/systemd/system/ovn-northd.service.d/no-manage-ovsdb.conf \
|
||||
/etc/systemd/system/ovn-controller.service.d/log-level.conf; do
|
||||
exists "$p" && LEFT+=("$p")
|
||||
done
|
||||
# A unit we broke and could not restart is a failed uninstall, not a wart.
|
||||
for u in openvswitch-switch ovn-controller; do
|
||||
systemctl is-failed --quiet "$u" 2>/dev/null && LEFT+=("$u.service is failed")
|
||||
done
|
||||
for svc in nats gw daemon storage northstar viperblock vpcd ui; do
|
||||
id -u "spinifex-${svc}" >/dev/null 2>&1 && LEFT+=("user spinifex-${svc}")
|
||||
done
|
||||
if command -v nft >/dev/null 2>&1 && sudo nft list table inet spinifex_filter >/dev/null 2>&1; then
|
||||
LEFT+=("nft table inet spinifex_filter")
|
||||
fi
|
||||
$PURGE_DATA && [ -e "$DATA_DIR" ] && ! mountpoint -q "$DATA_DIR" 2>/dev/null && LEFT+=("$DATA_DIR")
|
||||
|
||||
if [ "${#LEFT[@]}" -gt 0 ]; then
|
||||
echo "ERROR: the uninstall did not complete. Still present:" >&2
|
||||
printf ' %s\n' "${LEFT[@]}" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
log "done — Spinifex is removed from this node"
|
||||
$PURGE_DATA || { [ -d "$DATA_DIR" ] && log " $DATA_DIR was kept (--purge-data removes it)"; }
|
||||
$PURGE_DEPS || log " apt dependencies were kept (--purge-deps removes them)"
|
||||
exit 0
|
||||
Executable
+156
@@ -0,0 +1,156 @@
|
||||
#!/bin/sh
|
||||
# uninstall-spx_test.sh — tests for the shipped uninstaller.
|
||||
#
|
||||
# Everything here runs against the refusal and help paths, which return before
|
||||
# the script touches the host. Nothing in this file may reach a path that
|
||||
# removes anything: the script under test is the one that deletes the install,
|
||||
# so a test that ran it for real would uninstall the machine running CI.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
UNINSTALL="$SCRIPT_DIR/uninstall-spx.sh"
|
||||
TMP=$(mktemp -d)
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
fails=0
|
||||
pass() { echo " ok - $1"; }
|
||||
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
|
||||
|
||||
check() { # check <description> <condition-result>
|
||||
if [ "$2" = 0 ]; then pass "$1"; else fail "$1"; fi
|
||||
}
|
||||
|
||||
echo "-- uninstall-spx.sh"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Help and argument parsing.
|
||||
# --------------------------------------------------------------------------
|
||||
help_out=$("$UNINSTALL" --help 2>&1) && rc=0 || rc=$?
|
||||
check "--help exits 0" "$rc"
|
||||
|
||||
# Every flag the parser accepts must appear in the help text. This is what
|
||||
# catches a flag added to the case arms and never documented, which on a
|
||||
# destructive script is how an operator finds --purge-data by accident.
|
||||
for flag in --purge-data --purge-deps --yes --dry-run; do
|
||||
case "$help_out" in
|
||||
*"$flag"*) pass "--help documents $flag" ;;
|
||||
*) fail "--help does not document $flag" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# The header is printed up to the first non-comment line, so a truncated block
|
||||
# means the range drifted. The last paragraph is the one that used to be cut.
|
||||
case "$help_out" in
|
||||
*"netplan"*) pass "--help prints the whole header" ;;
|
||||
*) fail "--help output is truncated before the end of the header" ;;
|
||||
esac
|
||||
|
||||
case "$help_out" in
|
||||
*'#'*) fail "--help leaves comment markers in the output" ;;
|
||||
*) pass "--help strips comment markers" ;;
|
||||
esac
|
||||
|
||||
"$UNINSTALL" --not-a-real-flag >"$TMP/out" 2>"$TMP/err" && rc=0 || rc=$?
|
||||
[ "$rc" = 1 ] && pass "unknown option exits 1" || fail "unknown option exited $rc, want 1"
|
||||
grep -q "unknown option" "$TMP/err" \
|
||||
&& pass "unknown option explains itself on stderr" \
|
||||
|| fail "unknown option wrote nothing useful to stderr"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The multi-node refusal.
|
||||
#
|
||||
# Uninstalling one node of a running cluster strands its guests, so this is a
|
||||
# structural refusal rather than a prompt. The property under test is that
|
||||
# --yes does not cover it — a flag meaning "don't ask me" must not also mean
|
||||
# "ignore a safety property".
|
||||
# --------------------------------------------------------------------------
|
||||
mkcfg() { # mkcfg <dir> <node>...
|
||||
dir="$1"
|
||||
shift
|
||||
mkdir -p "$dir"
|
||||
{
|
||||
echo '[network]'
|
||||
for n in "$@"; do
|
||||
echo "[nodes.$n]"
|
||||
echo "advertise = \"10.0.0.1\""
|
||||
echo "[nodes.$n.daemon]"
|
||||
done
|
||||
} >"$dir/spinifex.toml"
|
||||
}
|
||||
|
||||
mkcfg "$TMP/three" node1 node2 node3
|
||||
mkcfg "$TMP/one" node1
|
||||
|
||||
for args in "" "--yes" "--yes --dry-run"; do
|
||||
# shellcheck disable=SC2086
|
||||
SPX_ETC_DIR="$TMP/three" "$UNINSTALL" $args >"$TMP/out" 2>"$TMP/err" && rc=0 || rc=$?
|
||||
label=${args:-"(no flags)"}
|
||||
[ "$rc" = 1 ] \
|
||||
&& pass "three-node config refuses with $label" \
|
||||
|| fail "three-node config with $label exited $rc, want 1"
|
||||
grep -q "3-node cluster" "$TMP/err" \
|
||||
&& pass "refusal names the node count for $label" \
|
||||
|| fail "refusal for $label did not name the node count"
|
||||
grep -q "spx admin cluster shutdown" "$TMP/err" \
|
||||
&& pass "refusal for $label says how to proceed" \
|
||||
|| fail "refusal for $label does not say how to proceed"
|
||||
done
|
||||
|
||||
grep -q "node1 node2 node3" "$TMP/err" \
|
||||
&& pass "refusal lists the configured nodes" \
|
||||
|| fail "refusal does not list the configured nodes"
|
||||
|
||||
# A single-node config must get past the refusal. Asserted by what it is NOT:
|
||||
# reaching the refusal message. --dry-run keeps it from touching the host, and
|
||||
# the exit status is left alone because the dry run's later steps depend on
|
||||
# whatever is installed on the machine running this test.
|
||||
SPX_ETC_DIR="$TMP/one" "$UNINSTALL" --dry-run >"$TMP/out" 2>"$TMP/err" || true
|
||||
grep -q "node cluster" "$TMP/err" \
|
||||
&& fail "single-node config was refused as multi-node" \
|
||||
|| pass "single-node config is not refused"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The dry run's promise.
|
||||
#
|
||||
# --dry-run must print the plan and touch nothing, which holds only while every
|
||||
# mutating command goes through run() or rm_path(). A bare `sudo rm` or `sudo
|
||||
# systemctl` added later would ignore the flag and delete for real during what
|
||||
# the operator asked to be a rehearsal. Guarded blocks are allowed, so lines
|
||||
# inside an `if ! $DRY_RUN` are not the concern — an unguarded one is.
|
||||
# --------------------------------------------------------------------------
|
||||
stray=$(awk '
|
||||
# The helpers themselves are where the guarded mutation is meant to live.
|
||||
/^(run|rm_path)\(\) \{/ { helper = 1 }
|
||||
helper { if (/^\}$/) helper = 0; next }
|
||||
/if ! \$DRY_RUN/ || /\$DRY_RUN && / { guard = 1 }
|
||||
/^fi$/ || /^else$/ { guard = 0 }
|
||||
guard { next }
|
||||
/^[[:space:]]*#/ { next }
|
||||
/^[[:space:]]*(run|exists|rm_path)[[:space:]]/ { next }
|
||||
/^[[:space:]]*sudo[[:space:]]+(rm|userdel|groupdel|systemctl[[:space:]]+(disable|stop|restart))/ {
|
||||
printf "%d: %s\n", NR, $0
|
||||
}
|
||||
' "$UNINSTALL")
|
||||
|
||||
if [ -n "$stray" ]; then
|
||||
fail "mutating commands outside run()/rm_path() and outside a \$DRY_RUN guard:"
|
||||
echo "$stray" | sed 's/^/ /'
|
||||
else
|
||||
pass "every mutation goes through run()/rm_path() or a \$DRY_RUN guard"
|
||||
fi
|
||||
|
||||
# The nft teardown must name our own table. A global flush would take the
|
||||
# host's other rules with it.
|
||||
grep -q 'nft delete table inet spinifex_filter' "$UNINSTALL" \
|
||||
&& pass "nft teardown deletes only inet spinifex_filter" \
|
||||
|| fail "nft teardown does not delete inet spinifex_filter by name"
|
||||
grep -qE 'nft[[:space:]]+(-[a-z][[:space:]]+)*flush ruleset' "$UNINSTALL" \
|
||||
&& fail "nft teardown flushes the whole ruleset" \
|
||||
|| pass "nft teardown never flushes the whole ruleset"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
if [ "$fails" -gt 0 ]; then
|
||||
echo " $fails check(s) failed"
|
||||
exit 1
|
||||
fi
|
||||
echo " uninstall-spx.sh ok"
|
||||
+14
-12
@@ -148,21 +148,23 @@ type ConfigSettings struct {
|
||||
}
|
||||
|
||||
// PoolData is one [[network.external_pools]] block rendered into spinifex.toml.
|
||||
// The json tags are load bearing: formation sends these to every joiner, so a
|
||||
// renamed field silently drops a pool rather than failing to compile.
|
||||
type PoolData struct {
|
||||
Name string // Pool name (e.g., "wan", "nat-transit")
|
||||
Source string // IP source: "static" or "dhcp"
|
||||
BindBridge string // Linux bridge / interface for upstream DORA (source=dhcp only)
|
||||
Start string // First IP in range (static only)
|
||||
End string // Last IP in range (static only)
|
||||
Gateway string // WAN gateway IP
|
||||
GatewayIP string // Explicit SNAT IP (overrides default of first IP in range)
|
||||
PrefixLen int // Subnet prefix length (default 24)
|
||||
DNSServers []string // DNS servers for VM DHCP (auto-detected from host)
|
||||
DHCPMAC string // DHCP client MAC strategy: "derived" (default) or "interface"
|
||||
Name string `json:"name"` // Pool name (e.g., "wan", "nat-transit")
|
||||
Source string `json:"source,omitempty"` // IP source: "static" or "dhcp"
|
||||
BindBridge string `json:"bind_bridge,omitempty"` // Linux bridge / interface for upstream DORA (source=dhcp only)
|
||||
Start string `json:"start,omitempty"` // First IP in range (static only)
|
||||
End string `json:"end,omitempty"` // Last IP in range (static only)
|
||||
Gateway string `json:"gateway,omitempty"` // WAN gateway IP
|
||||
GatewayIP string `json:"gateway_ip,omitempty"` // Explicit SNAT IP (overrides default of first IP in range)
|
||||
PrefixLen int `json:"prefix_len,omitempty"` // Subnet prefix length (default 24)
|
||||
DNSServers []string `json:"dns_servers,omitempty"` // DNS servers for VM DHCP (auto-detected from host)
|
||||
DHCPMAC string `json:"dhcp_mac,omitempty"` // DHCP client MAC strategy: "derived" (default) or "interface"
|
||||
// GwLrpRangeStart/End reserve gateway-LRP IPs for OVN routers. When empty
|
||||
// the allocator auto-derives the top 16 host IPs of the pool subnet.
|
||||
GwLrpRangeStart string
|
||||
GwLrpRangeEnd string
|
||||
GwLrpRangeStart string `json:"gw_lrp_range_start,omitempty"`
|
||||
GwLrpRangeEnd string `json:"gw_lrp_range_end,omitempty"`
|
||||
}
|
||||
|
||||
// PredastoreNodeConfig describes a single Predastore node for multi-node config generation.
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
package oci_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/rsa"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/oracle/oci-go-sdk/v65/common"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
)
|
||||
|
||||
// capture records what the SDK actually put on the wire, which is the point of
|
||||
// these tests: the request shape is the part of client.go that can be wrong in
|
||||
// a way no amount of reading catches.
|
||||
type capture struct {
|
||||
method string
|
||||
path string
|
||||
query url.Values
|
||||
body map[string]any
|
||||
}
|
||||
|
||||
// testClient wires the real SDK-backed client to a handler. The signing key is
|
||||
// generated per test — the SDK refuses to build a client without one, and a
|
||||
// throwaway key is enough because nothing verifies the signature here.
|
||||
func testClient(t *testing.T, h http.HandlerFunc) (oci.Client, *[]capture) {
|
||||
t.Helper()
|
||||
var seen []capture
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
c := capture{method: r.Method, path: r.URL.Path, query: r.URL.Query()}
|
||||
if b, _ := io.ReadAll(r.Body); len(b) > 0 {
|
||||
_ = json.Unmarshal(b, &c.body)
|
||||
}
|
||||
seen = append(seen, c)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
h(w, r)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
key, err := rsa.GenerateKey(rand.Reader, 2048)
|
||||
require.NoError(t, err)
|
||||
pemKey := pem.EncodeToMemory(&pem.Block{
|
||||
Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key),
|
||||
})
|
||||
provider := common.NewRawConfigurationProvider(
|
||||
"ocid1.tenancy.oc1..t", "ocid1.user.oc1..u", "ap-sydney-1",
|
||||
"aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99", string(pemKey), nil)
|
||||
|
||||
c, err := oci.NewTestClient(provider, strings.TrimPrefix(srv.URL, "https://"), srv.Client())
|
||||
require.NoError(t, err)
|
||||
return c, &seen
|
||||
}
|
||||
|
||||
// OCI pages list responses and a VNIC can hold 64 addresses. A client that
|
||||
// ignored opc-next-page would report a partial VNIC to reconcile, which then
|
||||
// treats the addresses it did not see as absent — and drops live bindings.
|
||||
func TestListPrivateIPsFollowsEveryPage(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Query().Get("page") {
|
||||
case "":
|
||||
w.Header().Set("opc-next-page", "page2")
|
||||
_, _ = w.Write([]byte(`[{"id":"p1","ipAddress":"10.200.0.100","vnicId":"v1"}]`))
|
||||
case "page2":
|
||||
w.Header().Set("opc-next-page", "page3")
|
||||
_, _ = w.Write([]byte(`[{"id":"p2","ipAddress":"10.200.0.101","vnicId":"v1"}]`))
|
||||
default:
|
||||
_, _ = w.Write([]byte(`[{"id":"p3","ipAddress":"10.200.0.102","vnicId":"v1"}]`))
|
||||
}
|
||||
})
|
||||
|
||||
got, err := c.ListPrivateIPs(context.Background(), "v1")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 3, "pagination stopped early — reconcile would treat the rest as absent")
|
||||
assert.Equal(t, "10.200.0.102", got[2].Address.String())
|
||||
assert.Len(t, *seen, 3)
|
||||
assert.Equal(t, "v1", (*seen)[0].query.Get("vnicId"))
|
||||
}
|
||||
|
||||
func TestListPrivateIPsStopsOnASinglePage(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`[{"id":"p1","ipAddress":"10.200.0.100","vnicId":"v1"}]`))
|
||||
})
|
||||
|
||||
got, err := c.ListPrivateIPs(context.Background(), "v1")
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, got, 1)
|
||||
assert.Len(t, *seen, 1, "a response with no next page must not be re-fetched")
|
||||
}
|
||||
|
||||
// The bug this guards against is silent: a nil *string is omitted from the JSON
|
||||
// body, so OCI would read the update as "change nothing", return 200, and leave
|
||||
// the public IP attached. The address would keep billing and keep pointing at a
|
||||
// released instance.
|
||||
func TestDetachPublicIPSendsAnExplicitEmptyPrivateIP(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"id":"q1","ipAddress":"203.0.113.10","lifecycleState":"UNASSIGNED"}`))
|
||||
})
|
||||
|
||||
got, err := c.DetachPublicIP(context.Background(), "q1")
|
||||
require.NoError(t, err)
|
||||
assert.False(t, got.IsAssigned())
|
||||
|
||||
require.Len(t, *seen, 1)
|
||||
body := (*seen)[0].body
|
||||
raw, present := body["privateIpId"]
|
||||
require.True(t, present, "privateIpId was omitted, so OCI reads the update as a no-op and the address stays attached")
|
||||
assert.Empty(t, raw)
|
||||
}
|
||||
|
||||
func TestAttachPublicIPNamesTheTargetPrivateIP(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"id":"q1","privateIpId":"p2","lifecycleState":"ASSIGNED"}`))
|
||||
})
|
||||
|
||||
got, err := c.AttachPublicIP(context.Background(), "q1", "p2")
|
||||
require.NoError(t, err)
|
||||
assert.True(t, got.IsAssigned())
|
||||
assert.Equal(t, "p2", (*seen)[0].body["privateIpId"])
|
||||
}
|
||||
|
||||
// An EIP has to outlive its association, so the create must ask for RESERVED.
|
||||
// An EPHEMERAL public IP would vanish when the instance released it, which is
|
||||
// not what AWS AllocateAddress promises.
|
||||
func TestCreatePublicIPAsksForAReservedLifetime(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"id":"q1","ipAddress":"203.0.113.10","lifetime":"RESERVED","lifecycleState":"ASSIGNING"}`))
|
||||
})
|
||||
|
||||
got, err := c.CreatePublicIP(context.Background(), "comp1", "p1", "spinifex-eipalloc-1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, oci.LifetimeReserved, got.Lifetime)
|
||||
assert.False(t, got.IsAssigned(), "a create returns ASSIGNING; the caller must poll")
|
||||
|
||||
body := (*seen)[0].body
|
||||
assert.Equal(t, "RESERVED", body["lifetime"])
|
||||
assert.Equal(t, "comp1", body["compartmentId"])
|
||||
assert.Equal(t, "p1", body["privateIpId"])
|
||||
}
|
||||
|
||||
// OCI picks the address unless we name one, and the allocator relies on that:
|
||||
// naming one would mean guessing an address OCI may already have handed out.
|
||||
func TestAssignPrivateIPOmitsTheAddressWhenWeDoNotChooseIt(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"id":"p1","ipAddress":"10.200.0.183","vnicId":"v1"}`))
|
||||
})
|
||||
|
||||
got, err := c.AssignPrivateIP(context.Background(), "v1", netip.Addr{}, "spinifex-eipalloc-1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "10.200.0.183", got.Address.String())
|
||||
|
||||
body := (*seen)[0].body
|
||||
assert.Equal(t, "v1", body["vnicId"])
|
||||
assert.Equal(t, "spinifex-eipalloc-1", body["displayName"])
|
||||
_, present := body["ipAddress"]
|
||||
assert.False(t, present, "an omitted address is how OCI is asked to choose")
|
||||
}
|
||||
|
||||
// A 404 from the real transport has to reach the caller as ErrNotFound, or
|
||||
// reconcile cannot tell "already gone" from "the API is broken" and would stop
|
||||
// mid-pass rather than finishing the cleanup.
|
||||
func TestServiceErrorsFromTheWireClassify(t *testing.T) {
|
||||
c, _ := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
_, _ = w.Write([]byte(`{"code":"NotAuthorizedOrNotFound","message":"Authorization failed or requested resource not found."}`))
|
||||
})
|
||||
|
||||
_, err := c.GetPublicIP(context.Background(), "q1")
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, oci.ErrNotFound)
|
||||
}
|
||||
|
||||
func TestQuotaErrorsFromTheWireClassify(t *testing.T) {
|
||||
c, _ := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
_, _ = w.Write([]byte(`{"code":"LimitExceeded","message":"public IP limit reached"}`))
|
||||
})
|
||||
|
||||
_, err := c.CreatePublicIP(context.Background(), "comp1", "p1", "spinifex-1")
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, oci.ErrLimitExceeded)
|
||||
}
|
||||
|
||||
func TestDeleteAndUnassignReachTheRightRoutes(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
})
|
||||
|
||||
require.NoError(t, c.DeletePublicIP(context.Background(), "q1"))
|
||||
require.NoError(t, c.UnassignPrivateIP(context.Background(), "p1"))
|
||||
|
||||
require.Len(t, *seen, 2)
|
||||
assert.Equal(t, http.MethodDelete, (*seen)[0].method)
|
||||
assert.Contains(t, (*seen)[0].path, "q1")
|
||||
assert.Equal(t, http.MethodDelete, (*seen)[1].method)
|
||||
assert.Contains(t, (*seen)[1].path, "p1")
|
||||
}
|
||||
|
||||
func TestGetPublicIPByPrivateIPIDAsksByPrivateIP(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"id":"q1","privateIpId":"p1","lifecycleState":"ASSIGNED"}`))
|
||||
})
|
||||
|
||||
got, err := c.GetPublicIPByPrivateIPID(context.Background(), "p1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "q1", got.ID)
|
||||
assert.Equal(t, "p1", (*seen)[0].body["privateIpId"])
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
package oci
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/oracle/oci-go-sdk/v65/common"
|
||||
"github.com/oracle/oci-go-sdk/v65/common/auth"
|
||||
"github.com/oracle/oci-go-sdk/v65/core"
|
||||
)
|
||||
|
||||
// Client is the narrow surface the allocator needs. It is expressed in this
|
||||
// package's own types rather than the SDK's so the SDK stays confined to this
|
||||
// file and the fake in fake.go is a map rather than a mock framework.
|
||||
type Client interface {
|
||||
// AssignPrivateIP creates a secondary private IP on vnicID. addr may be
|
||||
// the zero Addr to let OCI choose from the subnet, which is the normal
|
||||
// case — we do not pick addresses, OCI does.
|
||||
AssignPrivateIP(ctx context.Context, vnicID string, addr netip.Addr, displayName string) (PrivateIP, error)
|
||||
// MovePrivateIP reassigns an existing secondary private IP to vnicID,
|
||||
// which must be in the same subnet. One call, applied server-side, and the
|
||||
// OCID survives it — so the reserved public IP attached to this private IP
|
||||
// stays attached and the customer's address never changes. Detaching and
|
||||
// re-creating would lose both.
|
||||
MovePrivateIP(ctx context.Context, privateIPID, vnicID string) (PrivateIP, error)
|
||||
// UnassignPrivateIP deletes a secondary private IP. Deleting a primary is
|
||||
// refused by OCI, which is the behaviour we want.
|
||||
UnassignPrivateIP(ctx context.Context, privateIPID string) error
|
||||
GetPrivateIP(ctx context.Context, privateIPID string) (PrivateIP, error)
|
||||
ListPrivateIPs(ctx context.Context, vnicID string) ([]PrivateIP, error)
|
||||
|
||||
// CreatePublicIP creates a RESERVED public IP in compartmentID and, when
|
||||
// privateIPID is non-empty, attaches it in the same call.
|
||||
CreatePublicIP(ctx context.Context, compartmentID, privateIPID, displayName string) (PublicIP, error)
|
||||
// AttachPublicIP moves an existing reserved public IP onto privateIPID.
|
||||
AttachPublicIP(ctx context.Context, publicIPID, privateIPID string) (PublicIP, error)
|
||||
// DetachPublicIP unassigns without deleting, so the address is kept.
|
||||
DetachPublicIP(ctx context.Context, publicIPID string) (PublicIP, error)
|
||||
DeletePublicIP(ctx context.Context, publicIPID string) error
|
||||
GetPublicIP(ctx context.Context, publicIPID string) (PublicIP, error)
|
||||
// GetPublicIPByPrivateIPID returns the public IP attached to a private IP,
|
||||
// or ErrNotFound when it has none.
|
||||
GetPublicIPByPrivateIPID(ctx context.Context, privateIPID string) (PublicIP, error)
|
||||
}
|
||||
|
||||
// apiClient is the real implementation over the OCI Go SDK.
|
||||
type apiClient struct {
|
||||
net core.VirtualNetworkClient
|
||||
}
|
||||
|
||||
var _ Client = (*apiClient)(nil)
|
||||
|
||||
// NewInstancePrincipalClient authenticates as the instance itself, using the
|
||||
// certificate the metadata service serves at /opc/v2/identity/cert.pem. No key
|
||||
// material on disk and nothing to rotate, which is why this is the only
|
||||
// constructor offered for the daemon path — a node that can reach its own
|
||||
// metadata service is already proving it is the instance it claims to be.
|
||||
func NewInstancePrincipalClient() (Client, error) {
|
||||
provider, err := auth.InstancePrincipalConfigurationProvider()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("oci instance principal: %w", err)
|
||||
}
|
||||
c, err := core.NewVirtualNetworkClientWithConfigurationProvider(provider)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("oci virtual network client: %w", err)
|
||||
}
|
||||
return &apiClient{net: c}, nil
|
||||
}
|
||||
|
||||
// DefaultConfigFile is where the OCI CLI and SDK both look for API-key
|
||||
// credentials, and where a v1 OCI deployment is expected to provide them.
|
||||
const DefaultConfigFile = "~/.oci/config"
|
||||
|
||||
// DefaultConfigProfile is the profile name used when none is configured. The
|
||||
// CLI's own default, so a config file written by `oci setup config` works
|
||||
// unmodified.
|
||||
const DefaultConfigProfile = "DEFAULT"
|
||||
|
||||
// NewConfigFileClient authenticates from an API-key config file — the same
|
||||
// ~/.oci/config the oci CLI uses.
|
||||
//
|
||||
// This is the v1 path, deliberately in preference to instance principal. It
|
||||
// needs no dynamic group and no IAM policy written by a tenancy admin, and it
|
||||
// never reads the instance metadata service, which matters because Spinifex's
|
||||
// own per-instance IMDS endpoints claim 169.254.169.254 on the host and take
|
||||
// the cloud's own metadata service with it.
|
||||
func NewConfigFileClient(path, profile string) (Client, error) {
|
||||
if path == "" {
|
||||
path = DefaultConfigFile
|
||||
}
|
||||
if profile == "" {
|
||||
profile = DefaultConfigProfile
|
||||
}
|
||||
expanded, err := expandHome(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
provider, err := common.ConfigurationProviderFromFileWithProfile(expanded, profile, "")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("oci config file %s (profile %s): %w", expanded, profile, err)
|
||||
}
|
||||
return NewClientWithProvider(provider)
|
||||
}
|
||||
|
||||
// expandHome resolves a leading ~ so operators can configure the conventional
|
||||
// path rather than an absolute one that differs per service user.
|
||||
func expandHome(path string) (string, error) {
|
||||
if !strings.HasPrefix(path, "~") {
|
||||
return path, nil
|
||||
}
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("oci config path %q: %w", path, err)
|
||||
}
|
||||
return filepath.Join(home, strings.TrimPrefix(path, "~")), nil
|
||||
}
|
||||
|
||||
// NewClientWithProvider builds a client from an explicit configuration
|
||||
// provider, for an operator running off a config file rather than on an
|
||||
// instance.
|
||||
func NewClientWithProvider(p common.ConfigurationProvider) (Client, error) {
|
||||
c, err := core.NewVirtualNetworkClientWithConfigurationProvider(p)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("oci virtual network client: %w", err)
|
||||
}
|
||||
return &apiClient{net: c}, nil
|
||||
}
|
||||
|
||||
func (c *apiClient) AssignPrivateIP(ctx context.Context, vnicID string, addr netip.Addr, displayName string) (PrivateIP, error) {
|
||||
details := core.CreatePrivateIpDetails{VnicId: &vnicID}
|
||||
if displayName != "" {
|
||||
details.DisplayName = &displayName
|
||||
}
|
||||
if addr.IsValid() {
|
||||
s := addr.String()
|
||||
details.IpAddress = &s
|
||||
}
|
||||
resp, err := c.net.CreatePrivateIp(ctx, core.CreatePrivateIpRequest{CreatePrivateIpDetails: details})
|
||||
if err != nil {
|
||||
return PrivateIP{}, wrap("AssignPrivateIP", err)
|
||||
}
|
||||
return toPrivateIP(resp.PrivateIp)
|
||||
}
|
||||
|
||||
func (c *apiClient) MovePrivateIP(ctx context.Context, privateIPID, vnicID string) (PrivateIP, error) {
|
||||
resp, err := c.net.UpdatePrivateIp(ctx, core.UpdatePrivateIpRequest{
|
||||
PrivateIpId: &privateIPID,
|
||||
UpdatePrivateIpDetails: core.UpdatePrivateIpDetails{VnicId: &vnicID},
|
||||
})
|
||||
if err != nil {
|
||||
return PrivateIP{}, wrap("MovePrivateIP", err)
|
||||
}
|
||||
return toPrivateIP(resp.PrivateIp)
|
||||
}
|
||||
|
||||
func (c *apiClient) UnassignPrivateIP(ctx context.Context, privateIPID string) error {
|
||||
_, err := c.net.DeletePrivateIp(ctx, core.DeletePrivateIpRequest{PrivateIpId: &privateIPID})
|
||||
return wrap("UnassignPrivateIP", err)
|
||||
}
|
||||
|
||||
func (c *apiClient) GetPrivateIP(ctx context.Context, privateIPID string) (PrivateIP, error) {
|
||||
resp, err := c.net.GetPrivateIp(ctx, core.GetPrivateIpRequest{PrivateIpId: &privateIPID})
|
||||
if err != nil {
|
||||
return PrivateIP{}, wrap("GetPrivateIP", err)
|
||||
}
|
||||
return toPrivateIP(resp.PrivateIp)
|
||||
}
|
||||
|
||||
func (c *apiClient) ListPrivateIPs(ctx context.Context, vnicID string) ([]PrivateIP, error) {
|
||||
var out []PrivateIP
|
||||
var page *string
|
||||
for {
|
||||
resp, err := c.net.ListPrivateIps(ctx, core.ListPrivateIpsRequest{VnicId: &vnicID, Page: page})
|
||||
if err != nil {
|
||||
return nil, wrap("ListPrivateIPs", err)
|
||||
}
|
||||
for _, item := range resp.Items {
|
||||
ip, err := toPrivateIP(item)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, ip)
|
||||
}
|
||||
if resp.OpcNextPage == nil {
|
||||
return out, nil
|
||||
}
|
||||
page = resp.OpcNextPage
|
||||
}
|
||||
}
|
||||
|
||||
func (c *apiClient) CreatePublicIP(ctx context.Context, compartmentID, privateIPID, displayName string) (PublicIP, error) {
|
||||
details := core.CreatePublicIpDetails{
|
||||
CompartmentId: &compartmentID,
|
||||
Lifetime: core.CreatePublicIpDetailsLifetimeReserved,
|
||||
}
|
||||
if displayName != "" {
|
||||
details.DisplayName = &displayName
|
||||
}
|
||||
if privateIPID != "" {
|
||||
details.PrivateIpId = &privateIPID
|
||||
}
|
||||
resp, err := c.net.CreatePublicIp(ctx, core.CreatePublicIpRequest{CreatePublicIpDetails: details})
|
||||
if err != nil {
|
||||
return PublicIP{}, wrap("CreatePublicIP", err)
|
||||
}
|
||||
return toPublicIP(resp.PublicIp)
|
||||
}
|
||||
|
||||
func (c *apiClient) AttachPublicIP(ctx context.Context, publicIPID, privateIPID string) (PublicIP, error) {
|
||||
resp, err := c.net.UpdatePublicIp(ctx, core.UpdatePublicIpRequest{
|
||||
PublicIpId: &publicIPID,
|
||||
UpdatePublicIpDetails: core.UpdatePublicIpDetails{PrivateIpId: &privateIPID},
|
||||
})
|
||||
if err != nil {
|
||||
return PublicIP{}, wrap("AttachPublicIP", err)
|
||||
}
|
||||
return toPublicIP(resp.PublicIp)
|
||||
}
|
||||
|
||||
// DetachPublicIP clears privateIpId. The SDK omits a nil pointer, so the empty
|
||||
// string is sent explicitly — that is what OCI reads as "unassign", and a nil
|
||||
// here would be a no-op update that reports success while changing nothing.
|
||||
func (c *apiClient) DetachPublicIP(ctx context.Context, publicIPID string) (PublicIP, error) {
|
||||
empty := ""
|
||||
resp, err := c.net.UpdatePublicIp(ctx, core.UpdatePublicIpRequest{
|
||||
PublicIpId: &publicIPID,
|
||||
UpdatePublicIpDetails: core.UpdatePublicIpDetails{PrivateIpId: &empty},
|
||||
})
|
||||
if err != nil {
|
||||
return PublicIP{}, wrap("DetachPublicIP", err)
|
||||
}
|
||||
return toPublicIP(resp.PublicIp)
|
||||
}
|
||||
|
||||
func (c *apiClient) DeletePublicIP(ctx context.Context, publicIPID string) error {
|
||||
_, err := c.net.DeletePublicIp(ctx, core.DeletePublicIpRequest{PublicIpId: &publicIPID})
|
||||
return wrap("DeletePublicIP", err)
|
||||
}
|
||||
|
||||
func (c *apiClient) GetPublicIP(ctx context.Context, publicIPID string) (PublicIP, error) {
|
||||
resp, err := c.net.GetPublicIp(ctx, core.GetPublicIpRequest{PublicIpId: &publicIPID})
|
||||
if err != nil {
|
||||
return PublicIP{}, wrap("GetPublicIP", err)
|
||||
}
|
||||
return toPublicIP(resp.PublicIp)
|
||||
}
|
||||
|
||||
func (c *apiClient) GetPublicIPByPrivateIPID(ctx context.Context, privateIPID string) (PublicIP, error) {
|
||||
resp, err := c.net.GetPublicIpByPrivateIpId(ctx, core.GetPublicIpByPrivateIpIdRequest{
|
||||
GetPublicIpByPrivateIpIdDetails: core.GetPublicIpByPrivateIpIdDetails{PrivateIpId: &privateIPID},
|
||||
})
|
||||
if err != nil {
|
||||
return PublicIP{}, wrap("GetPublicIPByPrivateIPID", err)
|
||||
}
|
||||
return toPublicIP(resp.PublicIp)
|
||||
}
|
||||
|
||||
func toPrivateIP(p core.PrivateIp) (PrivateIP, error) {
|
||||
out := PrivateIP{
|
||||
ID: deref(p.Id),
|
||||
VNICID: deref(p.VnicId),
|
||||
SubnetID: deref(p.SubnetId),
|
||||
DisplayName: deref(p.DisplayName),
|
||||
IsPrimary: p.IsPrimary != nil && *p.IsPrimary,
|
||||
}
|
||||
if s := deref(p.IpAddress); s != "" {
|
||||
addr, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
return PrivateIP{}, fmt.Errorf("oci: parse private ip %q: %w", s, err)
|
||||
}
|
||||
out.Address = addr
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func toPublicIP(p core.PublicIp) (PublicIP, error) {
|
||||
out := PublicIP{
|
||||
ID: deref(p.Id),
|
||||
PrivateIPID: deref(p.PrivateIpId),
|
||||
DisplayName: deref(p.DisplayName),
|
||||
Lifetime: string(p.Lifetime),
|
||||
LifecycleState: string(p.LifecycleState),
|
||||
}
|
||||
if s := deref(p.IpAddress); s != "" {
|
||||
addr, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
return PublicIP{}, fmt.Errorf("oci: parse public ip %q: %w", s, err)
|
||||
}
|
||||
out.Address = addr
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func deref(s *string) string {
|
||||
if s == nil {
|
||||
return ""
|
||||
}
|
||||
return *s
|
||||
}
|
||||
|
||||
// wrap classifies an SDK error against the three sentinels. The HTTP status is
|
||||
// the reliable signal; the service code is carried for the message but not
|
||||
// matched on, because OCI's code strings differ per service and per endpoint
|
||||
// while the statuses do not.
|
||||
func wrap(op string, err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
svc, ok := common.IsServiceError(err)
|
||||
if !ok {
|
||||
return fmt.Errorf("oci %s: %w", op, err)
|
||||
}
|
||||
api := &APIError{
|
||||
Op: op,
|
||||
Code: svc.GetCode(),
|
||||
Message: svc.GetMessage(),
|
||||
StatusCode: svc.GetHTTPStatusCode(),
|
||||
RequestID: svc.GetOpcRequestID(),
|
||||
}
|
||||
switch {
|
||||
case svc.GetHTTPStatusCode() == 404:
|
||||
api.kind = ErrNotFound
|
||||
case svc.GetHTTPStatusCode() == 409, svc.GetHTTPStatusCode() == 412:
|
||||
api.kind = ErrConflict
|
||||
case svc.GetHTTPStatusCode() == 400 && svc.GetCode() == "LimitExceeded":
|
||||
api.kind = ErrLimitExceeded
|
||||
case svc.GetHTTPStatusCode() == 429:
|
||||
api.kind = ErrLimitExceeded
|
||||
}
|
||||
return api
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package oci_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/netip"
|
||||
"testing"
|
||||
|
||||
ocisdk "github.com/oracle/oci-go-sdk/v65/common"
|
||||
"github.com/oracle/oci-go-sdk/v65/core"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
)
|
||||
|
||||
// serviceError is an SDK service error. common.IsServiceError matches on the
|
||||
// interface, so a local type is enough and no HTTP round trip is needed.
|
||||
type serviceError struct {
|
||||
code string
|
||||
message string
|
||||
status int
|
||||
reqID string
|
||||
}
|
||||
|
||||
func (e serviceError) Error() string { return e.message }
|
||||
func (e serviceError) GetHTTPStatusCode() int { return e.status }
|
||||
func (e serviceError) GetMessage() string { return e.message }
|
||||
func (e serviceError) GetCode() string { return e.code }
|
||||
func (e serviceError) GetOpcRequestID() string { return e.reqID }
|
||||
|
||||
var _ ocisdk.ServiceError = serviceError{}
|
||||
|
||||
// The classification is control flow, not cosmetics: a quota refusal has to
|
||||
// reach the customer as InsufficientAddressCapacity, a 404 has to let reconcile
|
||||
// treat an object as already gone, and a conflict has to trigger a re-read
|
||||
// rather than a blind retry.
|
||||
func TestWrapClassifiesTheStatusesThatChangeBehaviour(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
err serviceError
|
||||
want error
|
||||
reason string
|
||||
}{
|
||||
{"404 is not found", serviceError{status: 404, code: "NotAuthorizedOrNotFound"}, oci.ErrNotFound,
|
||||
"OCI returns 404 for both missing and unauthorised, and either way the object is not ours to use"},
|
||||
{"409 is a conflict", serviceError{status: 409, code: "Conflict"}, oci.ErrConflict, ""},
|
||||
{"412 is a conflict", serviceError{status: 412, code: "PreconditionFailed"}, oci.ErrConflict,
|
||||
"an etag mismatch means another node changed it first"},
|
||||
{"400 LimitExceeded is a quota", serviceError{status: 400, code: "LimitExceeded"}, oci.ErrLimitExceeded,
|
||||
"64 private IPs per VNIC and 50 public IPs per region arrive as this"},
|
||||
{"429 is a quota", serviceError{status: 429, code: "TooManyRequests"}, oci.ErrLimitExceeded, ""},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
err := oci.Wrap("TestOp", tc.err)
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, tc.want, tc.reason)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A 400 that is not a quota must not read as one, or a malformed request would
|
||||
// be reported to the customer as the region being full.
|
||||
func TestWrapLeavesAnUnclassifiedStatusUnmatched(t *testing.T) {
|
||||
err := oci.Wrap("TestOp", serviceError{status: 400, code: "InvalidParameter"})
|
||||
require.Error(t, err)
|
||||
assert.NotErrorIs(t, err, oci.ErrLimitExceeded)
|
||||
assert.NotErrorIs(t, err, oci.ErrNotFound)
|
||||
assert.NotErrorIs(t, err, oci.ErrConflict)
|
||||
}
|
||||
|
||||
func TestWrapPassesANonServiceErrorThrough(t *testing.T) {
|
||||
sentinel := errors.New("dial tcp: connection refused")
|
||||
err := oci.Wrap("TestOp", sentinel)
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, sentinel, "a transport failure must stay unwrappable to its cause")
|
||||
assert.Contains(t, err.Error(), "TestOp")
|
||||
}
|
||||
|
||||
func TestWrapOfNilIsNil(t *testing.T) {
|
||||
assert.NoError(t, oci.Wrap("TestOp", nil))
|
||||
}
|
||||
|
||||
// The request ID is what Oracle support asks for first, so it has to survive
|
||||
// into the message rather than only into a struct field nobody prints.
|
||||
func TestAPIErrorMessageCarriesTheRequestID(t *testing.T) {
|
||||
err := oci.Wrap("CreatePublicIp", serviceError{
|
||||
status: 400, code: "LimitExceeded", message: "public IP limit reached", reqID: "ABC123",
|
||||
})
|
||||
assert.Contains(t, err.Error(), "ABC123")
|
||||
assert.Contains(t, err.Error(), "public IP limit reached")
|
||||
assert.Contains(t, err.Error(), "CreatePublicIp")
|
||||
assert.Contains(t, err.Error(), "400")
|
||||
}
|
||||
|
||||
// Every field on an SDK response is a pointer, and OCI omits the ones it has
|
||||
// no value for. A converter that dereferenced blind would panic on exactly the
|
||||
// responses that matter most — an ASSIGNING public IP has no address yet.
|
||||
func TestConvertersToleratePointerFieldsOCIOmits(t *testing.T) {
|
||||
priv, err := oci.ToPrivateIP(core.PrivateIp{})
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, priv.ID)
|
||||
assert.False(t, priv.Address.IsValid(), "an absent address must be the zero Addr, not a parse error")
|
||||
assert.False(t, priv.IsPrimary)
|
||||
|
||||
pub, err := oci.ToPublicIP(core.PublicIp{})
|
||||
require.NoError(t, err)
|
||||
assert.False(t, pub.Address.IsValid())
|
||||
assert.False(t, pub.IsAssigned())
|
||||
}
|
||||
|
||||
func TestConvertersCarryTheFieldsTheAllocatorReadsBack(t *testing.T) {
|
||||
priv, err := oci.ToPrivateIP(core.PrivateIp{
|
||||
Id: strptr("ocid1.privateip.oc1..p1"),
|
||||
IpAddress: strptr("10.200.0.183"),
|
||||
VnicId: strptr("ocid1.vnic.oc1..v1"),
|
||||
SubnetId: strptr("ocid1.subnet.oc1..s1"),
|
||||
DisplayName: strptr("spinifex-eipalloc-1"),
|
||||
IsPrimary: boolptr(true),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, netip.MustParseAddr("10.200.0.183"), priv.Address)
|
||||
assert.Equal(t, "ocid1.vnic.oc1..v1", priv.VNICID)
|
||||
assert.True(t, priv.IsPrimary)
|
||||
|
||||
pub, err := oci.ToPublicIP(core.PublicIp{
|
||||
Id: strptr("ocid1.publicip.oc1..q1"),
|
||||
IpAddress: strptr("161.33.227.97"),
|
||||
PrivateIpId: strptr("ocid1.privateip.oc1..p1"),
|
||||
Lifetime: core.PublicIpLifetimeReserved,
|
||||
LifecycleState: core.PublicIpLifecycleStateAssigned,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, netip.MustParseAddr("161.33.227.97"), pub.Address)
|
||||
assert.Equal(t, oci.LifetimeReserved, pub.Lifetime)
|
||||
assert.True(t, pub.IsAssigned(), "an ASSIGNED public IP is the only one safe to hand out")
|
||||
}
|
||||
|
||||
// A malformed address has to be an error, not a silently zero Addr that the
|
||||
// allocator would then record as a customer's public IP.
|
||||
func TestConvertersRefuseAnUnparseableAddress(t *testing.T) {
|
||||
_, err := oci.ToPrivateIP(core.PrivateIp{IpAddress: strptr("not-an-ip")})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "not-an-ip")
|
||||
|
||||
_, err = oci.ToPublicIP(core.PublicIp{IpAddress: strptr("999.1.1.1")})
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestDerefOfNilIsEmpty(t *testing.T) {
|
||||
assert.Empty(t, oci.Deref(nil))
|
||||
assert.Equal(t, "x", oci.Deref(strptr("x")))
|
||||
}
|
||||
|
||||
func strptr(s string) *string { return &s }
|
||||
func boolptr(b bool) *bool { return &b }
|
||||
@@ -0,0 +1,40 @@
|
||||
package oci
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/oracle/oci-go-sdk/v65/common"
|
||||
"github.com/oracle/oci-go-sdk/v65/core"
|
||||
)
|
||||
|
||||
// The MAC-matching half of VNIC resolution is deliberately unexported — it is
|
||||
// an implementation detail of ResolveVNICByInterface — but it is also the part
|
||||
// worth testing directly, because the alternative is a test that depends on the
|
||||
// host having an OCI NIC. Exported here for the external test package.
|
||||
var (
|
||||
NormalizeMAC = normalizeMAC
|
||||
MatchVNICByMAC = matchVNICByMAC
|
||||
)
|
||||
|
||||
// The error classifier and the SDK-to-our-types converters carry the logic in
|
||||
// client.go that is worth testing; the rest of that file is SDK plumbing that
|
||||
// only a live endpoint exercises.
|
||||
var (
|
||||
Wrap = wrap
|
||||
ToPrivateIP = toPrivateIP
|
||||
ToPublicIP = toPublicIP
|
||||
Deref = deref
|
||||
)
|
||||
|
||||
// NewTestClient points the real SDK-backed client at a local endpoint, so the
|
||||
// request shapes this package builds — pagination, and the explicit empty
|
||||
// string that DetachPublicIP depends on — are testable without a tenancy.
|
||||
func NewTestClient(provider common.ConfigurationProvider, endpoint string, hc *http.Client) (Client, error) {
|
||||
c, err := core.NewVirtualNetworkClientWithConfigurationProvider(provider)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.Host = endpoint
|
||||
c.HTTPClient = hc
|
||||
return &apiClient{net: c}, nil
|
||||
}
|
||||
@@ -0,0 +1,334 @@
|
||||
package oci
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Fake is an in-memory Client for tests. It is in the production package
|
||||
// rather than a _test file because the allocator package needs it too, and it
|
||||
// models the two behaviours that actually bite: OCI picks the address, and
|
||||
// assignment is asynchronous so a create is not immediately ASSIGNED.
|
||||
type Fake struct {
|
||||
mu sync.Mutex
|
||||
|
||||
// NextAddr supplies successive private addresses when the caller does not
|
||||
// name one, standing in for OCI's own choice from the subnet.
|
||||
NextAddr netip.Addr
|
||||
// NextPublic supplies successive public addresses.
|
||||
NextPublic netip.Addr
|
||||
|
||||
// AssignAfter is how many Get calls a public IP spends in ASSIGNING before
|
||||
// reporting ASSIGNED. Zero means assigned immediately.
|
||||
AssignAfter int
|
||||
|
||||
// FailWith, when non-nil, is returned by the next call to the named
|
||||
// operation and then cleared. Keyed by the Op strings used in APIError.
|
||||
FailWith map[string]error
|
||||
|
||||
// PrivateLimit caps private IPs per VNIC; 0 means unlimited. OCI's real
|
||||
// cap is 64.
|
||||
PrivateLimit int
|
||||
|
||||
privateIPs map[string]PrivateIP // by OCID
|
||||
publicIPs map[string]PublicIP // by OCID
|
||||
pending map[string]int // public OCID -> Gets remaining before ASSIGNED
|
||||
seq int
|
||||
}
|
||||
|
||||
var _ Client = (*Fake)(nil)
|
||||
|
||||
// NewFake returns a Fake seeded with sensible address counters.
|
||||
func NewFake() *Fake {
|
||||
return &Fake{
|
||||
NextAddr: netip.MustParseAddr("10.200.0.100"),
|
||||
NextPublic: netip.MustParseAddr("203.0.113.10"),
|
||||
FailWith: map[string]error{},
|
||||
privateIPs: map[string]PrivateIP{},
|
||||
publicIPs: map[string]PublicIP{},
|
||||
pending: map[string]int{},
|
||||
}
|
||||
}
|
||||
|
||||
// SeedPrivateIP inserts a private IP directly, for reconcile tests that need
|
||||
// OCI to already hold state our KV does not know about.
|
||||
func (f *Fake) SeedPrivateIP(p PrivateIP) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.privateIPs[p.ID] = p
|
||||
}
|
||||
|
||||
// SeedPublicIP inserts a public IP directly.
|
||||
func (f *Fake) SeedPublicIP(p PublicIP) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.publicIPs[p.ID] = p
|
||||
}
|
||||
|
||||
// PrivateIPs returns a snapshot, so a test can assert nothing leaked.
|
||||
func (f *Fake) PrivateIPs() []PrivateIP {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
out := make([]PrivateIP, 0, len(f.privateIPs))
|
||||
for _, p := range f.privateIPs {
|
||||
out = append(out, p)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// PublicIPs returns a snapshot.
|
||||
func (f *Fake) PublicIPs() []PublicIP {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
out := make([]PublicIP, 0, len(f.publicIPs))
|
||||
for _, p := range f.publicIPs {
|
||||
out = append(out, p)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// FailOp stages err for the next call to op. Locked, so a test may stage one
|
||||
// against a Fake that a loop under test is already calling.
|
||||
func (f *Fake) FailOp(op string, err error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.FailWith[op] = err
|
||||
}
|
||||
|
||||
// take returns and clears a staged failure for op.
|
||||
func (f *Fake) take(op string) error {
|
||||
if err, ok := f.FailWith[op]; ok {
|
||||
delete(f.FailWith, op)
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fake) next() int { f.seq++; return f.seq }
|
||||
|
||||
func (f *Fake) AssignPrivateIP(_ context.Context, vnicID string, addr netip.Addr, displayName string) (PrivateIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("AssignPrivateIP"); err != nil {
|
||||
return PrivateIP{}, err
|
||||
}
|
||||
if f.PrivateLimit > 0 {
|
||||
n := 0
|
||||
for _, p := range f.privateIPs {
|
||||
if p.VNICID == vnicID {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n >= f.PrivateLimit {
|
||||
return PrivateIP{}, &APIError{
|
||||
Op: "AssignPrivateIP", Code: "LimitExceeded", StatusCode: 400,
|
||||
Message: fmt.Sprintf("vnic %s already has %d private IPs", vnicID, n),
|
||||
kind: ErrLimitExceeded,
|
||||
}
|
||||
}
|
||||
}
|
||||
if !addr.IsValid() {
|
||||
addr = f.NextAddr
|
||||
f.NextAddr = f.NextAddr.Next()
|
||||
}
|
||||
for _, p := range f.privateIPs {
|
||||
if p.Address == addr {
|
||||
return PrivateIP{}, &APIError{
|
||||
Op: "AssignPrivateIP", Code: "Conflict", StatusCode: 409,
|
||||
Message: fmt.Sprintf("address %s already assigned", addr), kind: ErrConflict,
|
||||
}
|
||||
}
|
||||
}
|
||||
p := PrivateIP{
|
||||
ID: fmt.Sprintf("ocid1.privateip.oc1..fake%03d", f.next()),
|
||||
Address: addr,
|
||||
VNICID: vnicID,
|
||||
DisplayName: displayName,
|
||||
}
|
||||
f.privateIPs[p.ID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (f *Fake) MovePrivateIP(_ context.Context, privateIPID, vnicID string) (PrivateIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("MovePrivateIP"); err != nil {
|
||||
return PrivateIP{}, err
|
||||
}
|
||||
p, ok := f.privateIPs[privateIPID]
|
||||
if !ok {
|
||||
return PrivateIP{}, &APIError{Op: "MovePrivateIP", StatusCode: 404, Message: "no such private ip", kind: ErrNotFound}
|
||||
}
|
||||
if p.IsPrimary {
|
||||
return PrivateIP{}, &APIError{
|
||||
Op: "MovePrivateIP", StatusCode: 409, kind: ErrConflict,
|
||||
Message: "cannot reassign the primary private ip",
|
||||
}
|
||||
}
|
||||
// The OCID and the address survive the move, which is the whole point: an
|
||||
// attached public IP references this object, so it follows without being
|
||||
// touched.
|
||||
p.VNICID = vnicID
|
||||
f.privateIPs[privateIPID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (f *Fake) UnassignPrivateIP(_ context.Context, privateIPID string) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("UnassignPrivateIP"); err != nil {
|
||||
return err
|
||||
}
|
||||
p, ok := f.privateIPs[privateIPID]
|
||||
if !ok {
|
||||
return &APIError{Op: "UnassignPrivateIP", StatusCode: 404, Message: "no such private ip", kind: ErrNotFound}
|
||||
}
|
||||
if p.IsPrimary {
|
||||
return &APIError{Op: "UnassignPrivateIP", StatusCode: 409, Message: "cannot delete the primary private ip", kind: ErrConflict}
|
||||
}
|
||||
delete(f.privateIPs, privateIPID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fake) GetPrivateIP(_ context.Context, privateIPID string) (PrivateIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("GetPrivateIP"); err != nil {
|
||||
return PrivateIP{}, err
|
||||
}
|
||||
p, ok := f.privateIPs[privateIPID]
|
||||
if !ok {
|
||||
return PrivateIP{}, &APIError{Op: "GetPrivateIP", StatusCode: 404, Message: "no such private ip", kind: ErrNotFound}
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (f *Fake) ListPrivateIPs(_ context.Context, vnicID string) ([]PrivateIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("ListPrivateIPs"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []PrivateIP
|
||||
for _, p := range f.privateIPs {
|
||||
if p.VNICID == vnicID {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (f *Fake) CreatePublicIP(_ context.Context, compartmentID, privateIPID, displayName string) (PublicIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("CreatePublicIP"); err != nil {
|
||||
return PublicIP{}, err
|
||||
}
|
||||
_ = compartmentID
|
||||
addr := f.NextPublic
|
||||
f.NextPublic = f.NextPublic.Next()
|
||||
p := PublicIP{
|
||||
ID: fmt.Sprintf("ocid1.publicip.oc1..fake%03d", f.next()),
|
||||
Address: addr,
|
||||
PrivateIPID: privateIPID,
|
||||
DisplayName: displayName,
|
||||
Lifetime: LifetimeReserved,
|
||||
LifecycleState: LifecycleStateAssigned,
|
||||
}
|
||||
if privateIPID == "" {
|
||||
p.LifecycleState = LifecycleStateAvailable
|
||||
} else if f.AssignAfter > 0 {
|
||||
p.LifecycleState = LifecycleStateAssigning
|
||||
f.pending[p.ID] = f.AssignAfter
|
||||
}
|
||||
f.publicIPs[p.ID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (f *Fake) AttachPublicIP(_ context.Context, publicIPID, privateIPID string) (PublicIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("AttachPublicIP"); err != nil {
|
||||
return PublicIP{}, err
|
||||
}
|
||||
p, ok := f.publicIPs[publicIPID]
|
||||
if !ok {
|
||||
return PublicIP{}, &APIError{Op: "AttachPublicIP", StatusCode: 404, Message: "no such public ip", kind: ErrNotFound}
|
||||
}
|
||||
p.PrivateIPID = privateIPID
|
||||
p.LifecycleState = LifecycleStateAssigned
|
||||
if f.AssignAfter > 0 {
|
||||
p.LifecycleState = LifecycleStateAssigning
|
||||
f.pending[p.ID] = f.AssignAfter
|
||||
}
|
||||
f.publicIPs[publicIPID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (f *Fake) DetachPublicIP(_ context.Context, publicIPID string) (PublicIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("DetachPublicIP"); err != nil {
|
||||
return PublicIP{}, err
|
||||
}
|
||||
p, ok := f.publicIPs[publicIPID]
|
||||
if !ok {
|
||||
return PublicIP{}, &APIError{Op: "DetachPublicIP", StatusCode: 404, Message: "no such public ip", kind: ErrNotFound}
|
||||
}
|
||||
p.PrivateIPID = ""
|
||||
p.LifecycleState = LifecycleStateAvailable
|
||||
f.publicIPs[publicIPID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (f *Fake) DeletePublicIP(_ context.Context, publicIPID string) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("DeletePublicIP"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, ok := f.publicIPs[publicIPID]; !ok {
|
||||
return &APIError{Op: "DeletePublicIP", StatusCode: 404, Message: "no such public ip", kind: ErrNotFound}
|
||||
}
|
||||
delete(f.publicIPs, publicIPID)
|
||||
delete(f.pending, publicIPID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fake) GetPublicIP(_ context.Context, publicIPID string) (PublicIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("GetPublicIP"); err != nil {
|
||||
return PublicIP{}, err
|
||||
}
|
||||
p, ok := f.publicIPs[publicIPID]
|
||||
if !ok {
|
||||
return PublicIP{}, &APIError{Op: "GetPublicIP", StatusCode: 404, Message: "no such public ip", kind: ErrNotFound}
|
||||
}
|
||||
if n, pendingNow := f.pending[publicIPID]; pendingNow {
|
||||
if n <= 1 {
|
||||
delete(f.pending, publicIPID)
|
||||
p.LifecycleState = LifecycleStateAssigned
|
||||
f.publicIPs[publicIPID] = p
|
||||
} else {
|
||||
f.pending[publicIPID] = n - 1
|
||||
}
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (f *Fake) GetPublicIPByPrivateIPID(_ context.Context, privateIPID string) (PublicIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("GetPublicIPByPrivateIPID"); err != nil {
|
||||
return PublicIP{}, err
|
||||
}
|
||||
for _, p := range f.publicIPs {
|
||||
if p.PrivateIPID == privateIPID {
|
||||
return p, nil
|
||||
}
|
||||
}
|
||||
return PublicIP{}, &APIError{Op: "GetPublicIPByPrivateIPID", StatusCode: 404, Message: "no public ip for private ip", kind: ErrNotFound}
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
package oci_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/netip"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
)
|
||||
|
||||
// The Fake is what every allocator test is measured against, so a Fake that
|
||||
// models OCI wrongly makes those tests agree on a fiction. These check the
|
||||
// behaviours the allocator actually depends on.
|
||||
|
||||
func TestFakeAssignsSuccessiveAddressesLikeOCIChoosingThem(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
// The caller passes the zero Addr: OCI picks, not us. That asymmetry with
|
||||
// a static pool is the whole reason this package exists.
|
||||
a, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
b, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-2")
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, "10.200.0.100", a.Address.String())
|
||||
assert.Equal(t, "10.200.0.101", b.Address.String())
|
||||
assert.NotEqual(t, a.ID, b.ID, "each assignment is a distinct OCI object")
|
||||
assert.Equal(t, "vnic1", a.VNICID)
|
||||
}
|
||||
|
||||
func TestFakeHonoursACallerChosenAddress(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
got, err := f.AssignPrivateIP(ctx, "vnic1", netip.MustParseAddr("10.200.0.183"), "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "10.200.0.183", got.Address.String())
|
||||
}
|
||||
|
||||
// OCI caps private IPs at 64 per VNIC and the allocator has to surface that as
|
||||
// InsufficientAddressCapacity, so the Fake must refuse with the sentinel rather
|
||||
// than with a generic error.
|
||||
func TestFakeRefusesPastThePrivateIPLimit(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
f.PrivateLimit = 1
|
||||
|
||||
_, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-2")
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, oci.ErrLimitExceeded)
|
||||
}
|
||||
|
||||
// A create returns ASSIGNING and the address is not reachable until it settles.
|
||||
// An allocator that returned on the first response would hand out a dead
|
||||
// address, which is why the poll ladder exists.
|
||||
func TestFakeModelsAsynchronousAssignment(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
f.AssignAfter = 2
|
||||
|
||||
priv, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
pub, err := f.CreatePublicIP(ctx, "comp1", priv.ID, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
assert.False(t, pub.IsAssigned(), "a fresh public IP must not claim to be assigned")
|
||||
|
||||
first, err := f.GetPublicIP(ctx, pub.ID)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, first.IsAssigned())
|
||||
|
||||
second, err := f.GetPublicIP(ctx, pub.ID)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, second.IsAssigned(), "it must settle after the configured number of polls")
|
||||
}
|
||||
|
||||
func TestFakeCreatesReservedPublicIPsAttachedToTheirPrivateIP(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
priv, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
pub, err := f.CreatePublicIP(ctx, "comp1", priv.ID, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, oci.LifetimeReserved, pub.Lifetime,
|
||||
"an EIP must outlive its association, which is what RESERVED means")
|
||||
assert.Equal(t, priv.ID, pub.PrivateIPID)
|
||||
assert.Equal(t, "203.0.113.10", pub.Address.String())
|
||||
|
||||
found, err := f.GetPublicIPByPrivateIPID(ctx, priv.ID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, pub.ID, found.ID)
|
||||
}
|
||||
|
||||
func TestFakeReportsNotFoundForObjectsThatAreGone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
_, err := f.GetPrivateIP(ctx, "ocid1.privateip.oc1..missing")
|
||||
assert.ErrorIs(t, err, oci.ErrNotFound)
|
||||
|
||||
_, err = f.GetPublicIP(ctx, "ocid1.publicip.oc1..missing")
|
||||
assert.ErrorIs(t, err, oci.ErrNotFound)
|
||||
|
||||
_, err = f.GetPublicIPByPrivateIPID(ctx, "ocid1.privateip.oc1..missing")
|
||||
assert.ErrorIs(t, err, oci.ErrNotFound,
|
||||
"reconcile distinguishes a leaked private IP with no public IP from an API failure by this")
|
||||
}
|
||||
|
||||
func TestFakeDetachLeavesThePublicIPButUnassigned(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
priv, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
pub, err := f.CreatePublicIP(ctx, "comp1", priv.ID, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := f.DetachPublicIP(ctx, pub.ID)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, got.PrivateIPID)
|
||||
assert.False(t, got.IsAssigned())
|
||||
assert.Len(t, f.PublicIPs(), 1, "detach unassigns; only delete removes the object and stops the bill")
|
||||
}
|
||||
|
||||
func TestFakeAttachRepointsAPublicIP(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
first, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
second, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-2")
|
||||
require.NoError(t, err)
|
||||
pub, err := f.CreatePublicIP(ctx, "comp1", first.ID, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := f.AttachPublicIP(ctx, pub.ID, second.ID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, second.ID, got.PrivateIPID)
|
||||
}
|
||||
|
||||
func TestFakeListsOnlyTheNamedVNICsAddresses(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
_, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
_, err = f.AssignPrivateIP(ctx, "vnic2", netip.Addr{}, "spinifex-2")
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := f.ListPrivateIPs(ctx, "vnic1")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "vnic1", got[0].VNICID)
|
||||
}
|
||||
|
||||
func TestFakeUnassignAndDeleteRemoveTheObjects(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
priv, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
pub, err := f.CreatePublicIP(ctx, "comp1", priv.ID, "spinifex-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, f.DeletePublicIP(ctx, pub.ID))
|
||||
require.NoError(t, f.UnassignPrivateIP(ctx, priv.ID))
|
||||
assert.Empty(t, f.PublicIPs())
|
||||
assert.Empty(t, f.PrivateIPs())
|
||||
|
||||
assert.ErrorIs(t, f.UnassignPrivateIP(ctx, priv.ID), oci.ErrNotFound)
|
||||
assert.ErrorIs(t, f.DeletePublicIP(ctx, pub.ID), oci.ErrNotFound)
|
||||
}
|
||||
|
||||
// FailWith is how the rollback tests force a failure at an exact step, so it
|
||||
// has to fire once and then clear — otherwise the cleanup call it triggers
|
||||
// fails too and the test measures the wrong thing.
|
||||
func TestFakeFailWithFiresOnceThenClears(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
boom := errors.New("boom")
|
||||
f.FailWith["AssignPrivateIP"] = boom
|
||||
|
||||
_, err := f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
assert.ErrorIs(t, err, boom)
|
||||
|
||||
_, err = f.AssignPrivateIP(ctx, "vnic1", netip.Addr{}, "spinifex-1")
|
||||
assert.NoError(t, err, "a second call must succeed or rollback paths cannot be tested")
|
||||
}
|
||||
|
||||
func TestFakeSeedHelpersExposeStateReconcileTestsNeed(t *testing.T) {
|
||||
f := oci.NewFake()
|
||||
f.SeedPrivateIP(oci.PrivateIP{ID: "p1", VNICID: "vnic1", DisplayName: "vnic-1", IsPrimary: true})
|
||||
f.SeedPublicIP(oci.PublicIP{ID: "q1", PrivateIPID: "p1"})
|
||||
|
||||
assert.Len(t, f.PrivateIPs(), 1)
|
||||
assert.Len(t, f.PublicIPs(), 1)
|
||||
assert.True(t, f.PrivateIPs()[0].IsPrimary)
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
package oci
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// IMDSBaseURL is the OCI instance metadata service. v2 requires the bearer
|
||||
// header below; v1 is unauthenticated and disabled on current images.
|
||||
const IMDSBaseURL = "http://169.254.169.254/opc/v2"
|
||||
|
||||
// imdsTimeout bounds a metadata read. The service is link-local and answers in
|
||||
// single-digit milliseconds, so a slow read means it is not there — which is
|
||||
// the answer a non-OCI host needs, and it should not take the startup path
|
||||
// down with it.
|
||||
const imdsTimeout = 5 * time.Second
|
||||
|
||||
// VNICMetadata is one entry of /opc/v2/vnics/. Only the fields that identify
|
||||
// the VNIC are decoded; the rest of the document is not our business.
|
||||
type VNICMetadata struct {
|
||||
VNICID string `json:"vnicId"`
|
||||
MACAddr string `json:"macAddr"`
|
||||
PrivateIP string `json:"privateIp"`
|
||||
SubnetCIDRBlock string `json:"subnetCidrBlock"`
|
||||
VirtualRouterIP string `json:"virtualRouterIp"`
|
||||
VLANTag int `json:"vlanTag"`
|
||||
}
|
||||
|
||||
// InstanceMetadata is the subset of /opc/v2/instance/ the allocator needs.
|
||||
type InstanceMetadata struct {
|
||||
ID string `json:"id"`
|
||||
CompartmentID string `json:"compartmentId"`
|
||||
Region string `json:"region"`
|
||||
CanonicalRegionName string `json:"canonicalRegionName"`
|
||||
AvailabilityDomain string `json:"availabilityDomain"`
|
||||
Shape string `json:"shape"`
|
||||
}
|
||||
|
||||
// MetadataClient reads the instance metadata service.
|
||||
type MetadataClient struct {
|
||||
BaseURL string
|
||||
HTTP *http.Client
|
||||
}
|
||||
|
||||
// NewMetadataClient returns a client for the link-local metadata service.
|
||||
func NewMetadataClient() *MetadataClient {
|
||||
return &MetadataClient{
|
||||
BaseURL: IMDSBaseURL,
|
||||
HTTP: &http.Client{Timeout: imdsTimeout},
|
||||
}
|
||||
}
|
||||
|
||||
// VNICs returns the VNICs attached to this instance, in Oracle's order.
|
||||
func (c *MetadataClient) VNICs(ctx context.Context) ([]VNICMetadata, error) {
|
||||
var out []VNICMetadata
|
||||
if err := c.get(ctx, "/vnics/", &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Instance returns this instance's own metadata document.
|
||||
func (c *MetadataClient) Instance(ctx context.Context) (InstanceMetadata, error) {
|
||||
var out InstanceMetadata
|
||||
if err := c.get(ctx, "/instance/", &out); err != nil {
|
||||
return InstanceMetadata{}, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *MetadataClient) get(ctx context.Context, path string, out any) error {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.BaseURL+path, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("oci imds %s: %w", path, err)
|
||||
}
|
||||
// v2 refuses without this exact header; it is a confused-deputy guard, not
|
||||
// a credential, and Oracle documents the literal string.
|
||||
req.Header.Set("Authorization", "Bearer Oracle")
|
||||
resp, err := c.HTTP.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("oci imds %s: %w", path, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("oci imds %s: http %d", path, resp.StatusCode)
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(out); err != nil {
|
||||
return fmt.Errorf("oci imds %s: decode: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ResolveVNICByInterface maps a host interface name to the VNIC OCID carrying
|
||||
// its MAC. Matching on MAC rather than on Oracle's VNIC order is what makes
|
||||
// this safe: the order is not a documented contract, and on the reference host
|
||||
// the external VNIC is the second entry only by chance.
|
||||
//
|
||||
// A MAC-cloned Linux bridge resolves to the same VNIC as the NIC inside it,
|
||||
// which is the intended answer — on OCI the external datapath is a bridge that
|
||||
// clones the VNIC's MAC, so an operator naming either gets the same VNIC.
|
||||
func ResolveVNICByInterface(ctx context.Context, md *MetadataClient, iface string) (VNICMetadata, error) {
|
||||
link, err := net.InterfaceByName(iface)
|
||||
if err != nil {
|
||||
return VNICMetadata{}, fmt.Errorf("oci resolve vnic: interface %q: %w", iface, err)
|
||||
}
|
||||
mac := normalizeMAC(link.HardwareAddr.String())
|
||||
if mac == "" {
|
||||
return VNICMetadata{}, fmt.Errorf("oci resolve vnic: interface %q has no MAC", iface)
|
||||
}
|
||||
|
||||
vnics, err := md.VNICs(ctx)
|
||||
if err != nil {
|
||||
return VNICMetadata{}, err
|
||||
}
|
||||
return matchVNICByMAC(vnics, mac, iface)
|
||||
}
|
||||
|
||||
// matchVNICByMAC is the matching half of ResolveVNICByInterface, split out so
|
||||
// it can be tested without depending on the host having an OCI NIC.
|
||||
func matchVNICByMAC(vnics []VNICMetadata, mac, iface string) (VNICMetadata, error) {
|
||||
for _, v := range vnics {
|
||||
// IMDS reports the MAC uppercase and the kernel reports it lowercase,
|
||||
// so a byte comparison of the two strings never matches.
|
||||
if normalizeMAC(v.MACAddr) == normalizeMAC(mac) {
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
attached := make([]string, 0, len(vnics))
|
||||
for _, v := range vnics {
|
||||
attached = append(attached, v.MACAddr)
|
||||
}
|
||||
return VNICMetadata{}, fmt.Errorf("oci resolve vnic: no VNIC has interface %q's MAC %s (attached: %s)",
|
||||
iface, mac, strings.Join(attached, ", "))
|
||||
}
|
||||
|
||||
func normalizeMAC(s string) string { return strings.ToLower(strings.TrimSpace(s)) }
|
||||
@@ -0,0 +1,127 @@
|
||||
package oci_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// The document shape is taken verbatim from mulga-poc, uppercase MACs and all.
|
||||
const vnicsDoc = `[
|
||||
{"macAddr":"02:00:17:00:EF:07","privateIp":"10.200.0.144","subnetCidrBlock":"10.200.0.0/24",
|
||||
"virtualRouterIp":"10.200.0.1","vlanTag":119,
|
||||
"vnicId":"ocid1.vnic.oc1.ap-sydney-1.abzxsljrs4pqgaicbsgasbozqrjh3bocqvqpsu5gda2a2aseijmow67t7nsq"},
|
||||
{"macAddr":"02:00:17:01:7E:71","privateIp":"10.200.0.29","subnetCidrBlock":"10.200.0.0/24",
|
||||
"virtualRouterIp":"10.200.0.1","vlanTag":1603,
|
||||
"vnicId":"ocid1.vnic.oc1.ap-sydney-1.abzxsljrygicziuh2jktbre2ftvqxe2yh7lby2rrsqcbu6irus2sg73ajj5a"}
|
||||
]`
|
||||
|
||||
const instanceDoc = `{
|
||||
"id":"ocid1.instance.oc1.ap-sydney-1.anzxsljr6eq57gicdlugsqfidppe356dhslyar7pq762qpgkdv2loyrv7uyq",
|
||||
"compartmentId":"ocid1.compartment.oc1..aaaaaaaa6nkkqia2q3z2gjmqhqpq66t4k42keqob5wphgvtrfwlkfbr3ztta",
|
||||
"region":"ap-sydney-1","canonicalRegionName":"ap-sydney-1",
|
||||
"availabilityDomain":"RgrR:AP-SYDNEY-1-AD-1","shape":"VM.Standard.E6.Flex"
|
||||
}`
|
||||
|
||||
func testMetadata(t *testing.T) (*oci.MetadataClient, *[]string) {
|
||||
t.Helper()
|
||||
var auth []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
auth = append(auth, r.Header.Get("Authorization"))
|
||||
switch r.URL.Path {
|
||||
case "/vnics/":
|
||||
_, _ = w.Write([]byte(vnicsDoc))
|
||||
case "/instance/":
|
||||
_, _ = w.Write([]byte(instanceDoc))
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return &oci.MetadataClient{BaseURL: srv.URL, HTTP: srv.Client()}, &auth
|
||||
}
|
||||
|
||||
func TestVNICsDecodesTheMetadataDocument(t *testing.T) {
|
||||
md, auth := testMetadata(t)
|
||||
|
||||
vnics, err := md.VNICs(context.Background())
|
||||
require.NoError(t, err)
|
||||
require.Len(t, vnics, 2)
|
||||
|
||||
assert.Equal(t, "10.200.0.29", vnics[1].PrivateIP)
|
||||
assert.Equal(t, 1603, vnics[1].VLANTag)
|
||||
assert.Contains(t, vnics[1].VNICID, "ocid1.vnic.oc1.ap-sydney-1.")
|
||||
// IMDS v2 refuses the request without the literal bearer header.
|
||||
assert.Equal(t, []string{"Bearer Oracle"}, *auth)
|
||||
}
|
||||
|
||||
func TestInstanceCarriesTheCompartment(t *testing.T) {
|
||||
md, _ := testMetadata(t)
|
||||
|
||||
inst, err := md.Instance(context.Background())
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ocid1.compartment.oc1..aaaaaaaa6nkkqia2q3z2gjmqhqpq66t4k42keqob5wphgvtrfwlkfbr3ztta", inst.CompartmentID)
|
||||
assert.Equal(t, "ap-sydney-1", inst.CanonicalRegionName)
|
||||
}
|
||||
|
||||
// IMDS reports MACs uppercase and the kernel reports them lowercase, so a byte
|
||||
// comparison of the two never matches and every resolve would fail.
|
||||
func TestMACMatchingIsCaseInsensitive(t *testing.T) {
|
||||
assert.Equal(t, oci.NormalizeMAC("02:00:17:01:7E:71"), oci.NormalizeMAC("02:00:17:01:7e:71"))
|
||||
assert.NotEqual(t, oci.NormalizeMAC("02:00:17:01:7E:71"), oci.NormalizeMAC("02:00:17:00:ef:07"))
|
||||
}
|
||||
|
||||
// A metadata read that fails has to say which document and why: on a non-OCI
|
||||
// host this is the error an operator sees, and "404" alone explains nothing.
|
||||
func TestMetadataErrorsCarryThePathAndStatus(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
md := &oci.MetadataClient{BaseURL: srv.URL, HTTP: srv.Client()}
|
||||
|
||||
_, err := md.VNICs(context.Background())
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "/vnics/")
|
||||
assert.Contains(t, err.Error(), "404")
|
||||
}
|
||||
|
||||
// A MAC-cloned bridge (br-wan carries enp1s0's MAC on the reference host) must
|
||||
// resolve to the same VNIC as the NIC inside it — that is the whole reason
|
||||
// matching is on MAC rather than on Oracle's VNIC ordering, which is not a
|
||||
// documented contract and puts the external VNIC second only by chance.
|
||||
func TestMatchVNICResolvesTheClonedMACToTheExternalVNIC(t *testing.T) {
|
||||
md, _ := testMetadata(t)
|
||||
vnics, err := md.VNICs(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := oci.MatchVNICByMAC(vnics, "02:00:17:01:7e:71", "br-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "10.200.0.29", got.PrivateIP)
|
||||
assert.Equal(t, vnics[1].VNICID, got.VNICID)
|
||||
}
|
||||
|
||||
func TestMatchVNICNamesTheAttachedMACsWhenItCannotMatch(t *testing.T) {
|
||||
md, _ := testMetadata(t)
|
||||
vnics, err := md.VNICs(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = oci.MatchVNICByMAC(vnics, "aa:bb:cc:dd:ee:ff", "eth9")
|
||||
require.Error(t, err)
|
||||
// The operator's next move is comparing MACs, so the error has to show them.
|
||||
assert.Contains(t, err.Error(), "eth9")
|
||||
assert.Contains(t, err.Error(), "02:00:17:01:7E:71")
|
||||
}
|
||||
|
||||
func TestResolveVNICRejectsAnUnknownInterface(t *testing.T) {
|
||||
md, _ := testMetadata(t)
|
||||
|
||||
_, err := oci.ResolveVNICByInterface(context.Background(), md, "definitely-not-a-nic")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "definitely-not-a-nic")
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
// Package oci wraps the subset of the Oracle Cloud API that Spinifex needs to
|
||||
// hand out publicly reachable addresses on an OCI instance.
|
||||
//
|
||||
// OCI enforces the source IP per VNIC: a packet leaving the instance with a
|
||||
// source address that is not a registered private IP object on that VNIC is
|
||||
// dropped by the hypervisor, whatever the MAC. Measured on mulga-poc — from one
|
||||
// NIC with one MAC, an unregistered address gets 100% loss past the subnet
|
||||
// while registered ones get none. So an external address is not ours to invent
|
||||
// from range math; it has to be created through this API first.
|
||||
//
|
||||
// The unit of a publicly reachable address here is a pair: a secondary private
|
||||
// IP on the VNIC, which is what travels on the wire and what the host
|
||||
// configures on the interface, and a RESERVED public IP attached to it, which
|
||||
// is what the internet sees and what AWS semantics call the Elastic IP. OCI
|
||||
// 1:1 NATs between them. RESERVED rather than EPHEMERAL because an AWS EIP
|
||||
// outlives its association — an ephemeral public IP dies with the private IP
|
||||
// it was created against, which would change the customer's address on every
|
||||
// re-association.
|
||||
package oci
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
)
|
||||
|
||||
// PrivateIP is a VCN private IP object on a VNIC. The OCID, not the address,
|
||||
// is the handle every other call takes.
|
||||
type PrivateIP struct {
|
||||
ID string
|
||||
Address netip.Addr
|
||||
VNICID string
|
||||
SubnetID string
|
||||
DisplayName string
|
||||
IsPrimary bool
|
||||
}
|
||||
|
||||
// PublicIP is an OCI public IP object. PrivateIPID is empty when the address
|
||||
// is reserved but not currently attached to anything — the state an allocated
|
||||
// but unassociated AWS EIP maps onto.
|
||||
type PublicIP struct {
|
||||
ID string
|
||||
Address netip.Addr
|
||||
PrivateIPID string
|
||||
DisplayName string
|
||||
Lifetime string
|
||||
LifecycleState string
|
||||
}
|
||||
|
||||
// IsAssigned reports whether the public IP has finished provisioning and is
|
||||
// attached. OCI assignment is asynchronous, so a create returns before this is
|
||||
// true and callers poll.
|
||||
func (p PublicIP) IsAssigned() bool { return p.LifecycleState == LifecycleStateAssigned }
|
||||
|
||||
// Public IP lifecycle states, as returned by the API.
|
||||
const (
|
||||
LifecycleStateProvisioning = "PROVISIONING"
|
||||
LifecycleStateAvailable = "AVAILABLE"
|
||||
LifecycleStateAssigning = "ASSIGNING"
|
||||
LifecycleStateAssigned = "ASSIGNED"
|
||||
LifecycleStateUnassigning = "UNASSIGNING"
|
||||
LifecycleStateUnassigned = "UNASSIGNED"
|
||||
LifecycleStateTerminating = "TERMINATING"
|
||||
LifecycleStateTerminated = "TERMINATED"
|
||||
)
|
||||
|
||||
// Public IP lifetimes.
|
||||
const (
|
||||
LifetimeReserved = "RESERVED"
|
||||
LifetimeEphemeral = "EPHEMERAL"
|
||||
)
|
||||
|
||||
// The three failures that need distinct handling upstream. Everything else is
|
||||
// returned wrapped and untyped, because a caller that cannot act differently on
|
||||
// an error does not benefit from being able to name it.
|
||||
var (
|
||||
// ErrLimitExceeded is a quota refusal: 64 private IPs per VNIC, 50 public
|
||||
// IPs per region. The allocator maps this to InsufficientAddressCapacity so
|
||||
// it reaches the customer as the AWS error that means the same thing.
|
||||
ErrLimitExceeded = errors.New("oci: service limit exceeded")
|
||||
|
||||
// ErrNotFound is a 404. Release paths treat it as success — an address that
|
||||
// is already gone is the state the caller wanted.
|
||||
ErrNotFound = errors.New("oci: resource not found")
|
||||
|
||||
// ErrConflict is a 409 or a 412 etag mismatch: another node changed the
|
||||
// resource first. The caller re-reads rather than retrying blind.
|
||||
ErrConflict = errors.New("oci: conflicting state")
|
||||
)
|
||||
|
||||
// APIError carries the OCI service error alongside one of the sentinels above,
|
||||
// so errors.Is works for control flow while the message keeps the request ID
|
||||
// that Oracle support asks for.
|
||||
type APIError struct {
|
||||
Op string
|
||||
Code string
|
||||
Message string
|
||||
StatusCode int
|
||||
RequestID string
|
||||
kind error
|
||||
}
|
||||
|
||||
func (e *APIError) Error() string {
|
||||
return fmt.Sprintf("oci %s: %s (%s, http %d, opc-request-id %s)",
|
||||
e.Op, e.Message, e.Code, e.StatusCode, e.RequestID)
|
||||
}
|
||||
|
||||
// Unwrap returns the sentinel so errors.Is(err, ErrLimitExceeded) matches.
|
||||
func (e *APIError) Unwrap() error { return e.kind }
|
||||
@@ -65,7 +65,7 @@ type AWSConfig struct {
|
||||
// ExternalPool defines a range of routable IPs that Spinifex manages for public subnets.
|
||||
type ExternalPool struct {
|
||||
Name string `mapstructure:"name"` // Pool identifier (e.g., "wan", "dc1-primary")
|
||||
Source string `mapstructure:"source"` // IP source: "static" (default) or "dhcp"
|
||||
Source string `mapstructure:"source"` // IP source: "static" (default), "dhcp" or "oci"
|
||||
BindBridge string `mapstructure:"bind_bridge"` // Linux bridge for DHCP DORA (source=dhcp only)
|
||||
DHCPMAC string `mapstructure:"dhcp_mac"` // DHCP client MAC strategy: "derived" (default) or "interface" (source=dhcp only)
|
||||
RangeStart string `mapstructure:"range_start"` // First IP in range (static source only)
|
||||
@@ -80,6 +80,18 @@ type ExternalPool struct {
|
||||
// Must NOT overlap [RangeStart, RangeEnd] — link-local 169.254/16 is rejected by upstream routers.
|
||||
GwLrpRangeStart string `mapstructure:"gw_lrp_range_start"`
|
||||
GwLrpRangeEnd string `mapstructure:"gw_lrp_range_end"`
|
||||
|
||||
// OCI identifies the VNIC and compartment the provider allocates against
|
||||
// (source=oci only). An OCI VNIC drops any source address that is not a
|
||||
// registered private IP object on it, so the addresses are created through
|
||||
// the provider rather than computed from a range.
|
||||
OCICompartmentID string `mapstructure:"oci_compartment_id"` // Compartment OCID the public IPs are created in
|
||||
OCIVNICID string `mapstructure:"oci_vnic_id"` // VNIC OCID carrying the addresses
|
||||
OCIVNICIface string `mapstructure:"oci_vnic_iface"` // Host interface to resolve the VNIC OCID from, instead of oci_vnic_id
|
||||
OCISubnetID string `mapstructure:"oci_subnet_id"` // Subnet OCID the private IPs come from (optional; defaults to the VNIC's)
|
||||
OCIPublicIPPool string `mapstructure:"oci_public_ip_pool"` // Public IP pool OCID for BYOIP (optional)
|
||||
OCIConfigFile string `mapstructure:"oci_config_file"` // API-key config file (optional; defaults to ~/.oci/config)
|
||||
OCIConfigProfile string `mapstructure:"oci_config_profile"` // Profile within that file (optional; defaults to DEFAULT)
|
||||
}
|
||||
|
||||
// DefaultUnderlayMTU is the standard Ethernet payload, and the assumption a
|
||||
@@ -88,7 +100,7 @@ const DefaultUnderlayMTU = 1500
|
||||
|
||||
// NetworkConfig holds cluster-wide external network settings.
|
||||
type NetworkConfig struct {
|
||||
ExternalMode string `mapstructure:"external_mode"` // "pool" or "" (disabled)
|
||||
ExternalMode string `mapstructure:"external_mode"` // "pool", "nat" or "" (disabled)
|
||||
ExternalPools []ExternalPool `mapstructure:"external_pools"` // One or more IP pools
|
||||
// IPSecEnabled toggles OVN native IPsec (AES-256-GCM) on every node. Default true; disable only for trusted lab topologies.
|
||||
IPSecEnabled bool `mapstructure:"ipsec_enabled"`
|
||||
@@ -114,6 +126,16 @@ type NetworkConfig struct {
|
||||
// operator workaround for a tenant with a legitimate need until per-account
|
||||
// exceptions exist.
|
||||
EgressBlockExemptVPCs []string `mapstructure:"egress_block_exempt_vpcs"`
|
||||
// IMDSHostMetaIP and IMDSHostDNSIP move the IMDS endpoint's own addresses
|
||||
// off 169.254.169.254 / .253, for a node that needs those for itself.
|
||||
// A cloud guest does — they are its metadata service and its resolver — and
|
||||
// an endpoint /32 shadows the route to both, so the node loses DNS and its
|
||||
// cloud API for as long as any guest runs. Guests are unaffected: they keep
|
||||
// addressing the standard pair and a DNAT on the endpoint rewrites it.
|
||||
//
|
||||
// Set both or neither. Pick addresses the host has no route to.
|
||||
IMDSHostMetaIP string `mapstructure:"imds_host_meta_ip"`
|
||||
IMDSHostDNSIP string `mapstructure:"imds_host_dns_ip"`
|
||||
}
|
||||
|
||||
// DefaultBlockedWANPorts mirrors AWS's out-of-the-box outbound mail block:
|
||||
@@ -632,6 +654,9 @@ func validateClusterConfig(cc *ClusterConfig) error {
|
||||
}
|
||||
var ranges []poolRange
|
||||
for _, p := range cc.Network.ExternalPools {
|
||||
if p.Source != "oci" && (p.OCICompartmentID != "" || p.OCIVNICID != "" || p.OCIVNICIface != "" || p.OCISubnetID != "" || p.OCIPublicIPPool != "" || p.OCIConfigFile != "" || p.OCIConfigProfile != "") {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: oci_* keys are only valid with source=\"oci\"", p.Name)
|
||||
}
|
||||
switch p.DHCPMAC {
|
||||
case "", "derived", "interface":
|
||||
default:
|
||||
@@ -656,8 +681,39 @@ func validateClusterConfig(cc *ClusterConfig) error {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: gw_lrp_range_start/gw_lrp_range_end not allowed with source=\"dhcp\" (gateway LRP IP is DORA'd per VPC)", p.Name)
|
||||
}
|
||||
continue
|
||||
case "oci":
|
||||
if p.OCICompartmentID == "" {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: source=\"oci\" requires oci_compartment_id", p.Name)
|
||||
}
|
||||
// One or the other, never both: an OCID and an interface name that
|
||||
// disagree would send allocations to a VNIC the datapath is not on,
|
||||
// and the addresses would be silently unreachable.
|
||||
if (p.OCIVNICID == "") == (p.OCIVNICIface == "") {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: source=\"oci\" requires exactly one of oci_vnic_id or oci_vnic_iface", p.Name)
|
||||
}
|
||||
if p.BindBridge != "" || p.DHCPMAC != "" {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: bind_bridge/dhcp_mac are only valid with source=\"dhcp\"", p.Name)
|
||||
}
|
||||
if p.RangeStart != "" || p.RangeEnd != "" {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: range_start/range_end not allowed with source=\"oci\" (OCI picks the address)", p.Name)
|
||||
}
|
||||
if p.GwLrpRangeStart != "" || p.GwLrpRangeEnd != "" {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: gw_lrp_range_start/gw_lrp_range_end not allowed with source=\"oci\" (an OCI address per VPC gateway would exhaust the 50-per-region public IP quota)", p.Name)
|
||||
}
|
||||
// Pool mode puts a per-VPC gateway MAC and a per-NAT-rule external
|
||||
// MAC on the uplink and ARPs for the address. An OCI VNIC accepts
|
||||
// exactly one MAC, its own, and delivers every inbound frame to it
|
||||
// regardless of which of its addresses the packet is for — so a
|
||||
// pool-mode guest's egress is dropped and its ingress never reaches
|
||||
// OVN. Routed mode keeps the address on the host, where it already
|
||||
// wears the VNIC's MAC. Refusing here costs a config error; not
|
||||
// refusing costs a day of tcpdump.
|
||||
if cc.Network.ExternalMode != "nat" {
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: source=\"oci\" requires [network] external_mode = \"nat\" (routed); OCI VNICs accept only their own MAC, so pool mode's per-VPC gateway MACs are dropped by the provider", p.Name)
|
||||
}
|
||||
continue
|
||||
default:
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: source=%q unsupported; use \"static\" or \"dhcp\"", p.Name, p.Source)
|
||||
return fmt.Errorf("config: [[network.external_pools]] %q: source=%q unsupported; use \"static\", \"dhcp\" or \"oci\"", p.Name, p.Source)
|
||||
}
|
||||
if p.RangeStart == "" || p.RangeEnd == "" {
|
||||
continue
|
||||
|
||||
@@ -1178,3 +1178,157 @@ func TestResolvedBlockedWANPorts(t *testing.T) {
|
||||
custom := []int{25, 2525}
|
||||
assert.Equal(t, custom, NetworkConfig{BlockedPortsWAN: &custom}.ResolvedBlockedWANPorts())
|
||||
}
|
||||
|
||||
// ociPoolTOML builds a config whose OCI pool body is the caller's, so each
|
||||
// rejection test differs only by the thing it is testing.
|
||||
func ociPoolTOML(externalMode, poolBody string) string {
|
||||
return `
|
||||
node = "n1"
|
||||
|
||||
[network]
|
||||
external_mode = "` + externalMode + `"
|
||||
|
||||
[[network.external_pools]]
|
||||
name = "wan"
|
||||
` + poolBody + `
|
||||
|
||||
[nodes.n1]
|
||||
region = "ap-southeast-2"
|
||||
`
|
||||
}
|
||||
|
||||
func loadOCIPool(t *testing.T, externalMode, poolBody string) (*ClusterConfig, error) {
|
||||
t.Helper()
|
||||
resetViper(t)
|
||||
path := filepath.Join(t.TempDir(), "spinifex.toml")
|
||||
require.NoError(t, os.WriteFile(path, []byte(ociPoolTOML(externalMode, poolBody)), 0600))
|
||||
return LoadConfig(path)
|
||||
}
|
||||
|
||||
func TestLoadConfig_NetworkPoolOCISourceAccepted(t *testing.T) {
|
||||
cfg, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
oci_vnic_iface = "br-wan"
|
||||
gateway = "10.200.0.1"
|
||||
prefix_len = 24
|
||||
`)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, cfg.Network.ExternalPools, 1)
|
||||
p := cfg.Network.ExternalPools[0]
|
||||
assert.Equal(t, "oci", p.Source)
|
||||
assert.Equal(t, "ocid1.compartment.oc1..c1", p.OCICompartmentID)
|
||||
assert.Equal(t, "br-wan", p.OCIVNICIface)
|
||||
}
|
||||
|
||||
func TestLoadConfig_NetworkPoolOCIAcceptsAVNICOCID(t *testing.T) {
|
||||
cfg, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
oci_vnic_id = "ocid1.vnic.oc1.ap-sydney-1.v1"
|
||||
`)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ocid1.vnic.oc1.ap-sydney-1.v1", cfg.Network.ExternalPools[0].OCIVNICID)
|
||||
}
|
||||
|
||||
func TestLoadConfig_NetworkPoolOCIRequiresCompartment(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_vnic_iface = "br-wan"
|
||||
`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "oci_compartment_id")
|
||||
}
|
||||
|
||||
// An OCID and an interface name that disagree would send allocations to a VNIC
|
||||
// the datapath is not on, and OCI would drop the traffic silently.
|
||||
func TestLoadConfig_NetworkPoolOCIRejectsBothVNICKeys(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
oci_vnic_id = "ocid1.vnic.oc1.ap-sydney-1.v1"
|
||||
oci_vnic_iface = "br-wan"
|
||||
`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "exactly one of oci_vnic_id or oci_vnic_iface")
|
||||
}
|
||||
|
||||
func TestLoadConfig_NetworkPoolOCIRejectsNeitherVNICKey(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "exactly one of oci_vnic_id or oci_vnic_iface")
|
||||
}
|
||||
|
||||
func TestLoadConfig_NetworkPoolOCIRejectsRange(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
oci_vnic_iface = "br-wan"
|
||||
range_start = "10.200.0.100"
|
||||
range_end = "10.200.0.110"
|
||||
`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "range_start")
|
||||
}
|
||||
|
||||
// One OCI address per VPC gateway would exhaust the 50-per-region public IP
|
||||
// quota after fifty VPCs, for addresses no customer ever sees.
|
||||
func TestLoadConfig_NetworkPoolOCIRejectsGwLrpRange(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
oci_vnic_iface = "br-wan"
|
||||
gw_lrp_range_start = "10.200.0.200"
|
||||
gw_lrp_range_end = "10.200.0.210"
|
||||
`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "gw_lrp_range_start")
|
||||
}
|
||||
|
||||
func TestLoadConfig_NetworkPoolOCIRejectsDHCPKeys(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "nat", `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
oci_vnic_iface = "br-wan"
|
||||
bind_bridge = "br-wan"
|
||||
`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "bind_bridge")
|
||||
}
|
||||
|
||||
// Pool mode puts a per-VPC gateway MAC on the uplink and ARPs for the address.
|
||||
// An OCI VNIC accepts only its own MAC, so a pool-mode guest's egress is
|
||||
// dropped and its ingress never reaches OVN — silently, and only once a guest
|
||||
// boots. Refusing at config time is the whole point.
|
||||
func TestLoadConfig_NetworkPoolOCIRequiresRoutedMode(t *testing.T) {
|
||||
for _, mode := range []string{"", "pool"} {
|
||||
_, err := loadOCIPool(t, mode, `
|
||||
source = "oci"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
oci_vnic_iface = "br-wan"
|
||||
`)
|
||||
require.Error(t, err, "external_mode=%q must be refused for source=oci", mode)
|
||||
assert.Contains(t, err.Error(), "external_mode")
|
||||
assert.Contains(t, err.Error(), "MAC", "the refusal must say why, or it reads as arbitrary")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfig_OCIKeysRejectedOnANonOCIPool(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "pool", `
|
||||
source = "static"
|
||||
oci_compartment_id = "ocid1.compartment.oc1..c1"
|
||||
range_start = "10.200.0.100"
|
||||
range_end = "10.200.0.110"
|
||||
`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "oci_* keys are only valid")
|
||||
}
|
||||
|
||||
func TestLoadConfig_UnknownSourceErrorNamesOCI(t *testing.T) {
|
||||
_, err := loadOCIPool(t, "pool", `source = "magic"`)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), `"oci"`)
|
||||
}
|
||||
|
||||
@@ -65,6 +65,7 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/kvutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/host"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/otelsetup"
|
||||
@@ -1268,6 +1269,9 @@ func (d *Daemon) subscribeAll() error {
|
||||
natsSub{"ec2.DisassociateAddress", handleNATSRequest(d.node, d.eipService.DisassociateAddress), "spinifex-workers"},
|
||||
natsSub{"ec2.DescribeAddresses", handleNATSRequest(d.node, d.eipService.DescribeAddresses), "spinifex-workers"},
|
||||
natsSub{"ec2.DescribeAddressesAttribute", handleNATSRequest(d.node, d.eipService.DescribeAddressesAttribute), "spinifex-workers"},
|
||||
// Fan-out, no queue group: the association has to reach the node running
|
||||
// the instance, which is rarely the one that served the request.
|
||||
natsSub{handlers_ec2_eip.SubjectENIPublicIPChanged, d.handleENIPublicIPChanged, ""},
|
||||
// vpcd holds the leases, but the records naming those addresses live
|
||||
// here, so the reconcile request flows daemon-ward.
|
||||
natsSub{dhcp.TopicLeaseChanged, d.handleDHCPLeaseChanged, "spinifex-workers"},
|
||||
@@ -1358,6 +1362,12 @@ func (d *Daemon) startLocal() error {
|
||||
}
|
||||
}
|
||||
|
||||
// Before any tap is attached: the endpoint addresses and vpcd's binds have
|
||||
// to agree, and both read the same config keys.
|
||||
if d.clusterConfig != nil {
|
||||
host.SetIMDSHostAddrs(d.clusterConfig.Network.IMDSHostMetaIP, d.clusterConfig.Network.IMDSHostDNSIP)
|
||||
}
|
||||
|
||||
// Initialise OVS network plumber (no NATS dep).
|
||||
if d.networkPlumber == nil {
|
||||
d.networkPlumber = host.NewOVSPlumber()
|
||||
@@ -1436,6 +1446,14 @@ func (d *Daemon) externalPoolConfigs() (pools []external.ExternalPoolConfig, any
|
||||
AZ: p.AZ,
|
||||
GwLrpRangeStart: p.GwLrpRangeStart,
|
||||
GwLrpRangeEnd: p.GwLrpRangeEnd,
|
||||
|
||||
OCICompartmentID: p.OCICompartmentID,
|
||||
OCIVNICID: p.OCIVNICID,
|
||||
OCIVNICIface: p.OCIVNICIface,
|
||||
OCISubnetID: p.OCISubnetID,
|
||||
OCIPublicIPPool: p.OCIPublicIPPool,
|
||||
OCIConfigFile: p.OCIConfigFile,
|
||||
OCIConfigProfile: p.OCIConfigProfile,
|
||||
})
|
||||
if p.Source == "dhcp" {
|
||||
anyDHCP = true
|
||||
@@ -1444,6 +1462,79 @@ func (d *Daemon) externalPoolConfigs() (pools []external.ExternalPoolConfig, any
|
||||
return pools, anyDHCP
|
||||
}
|
||||
|
||||
// ovnSBAddr is this node's OVN Southbound address, empty when unconfigured,
|
||||
// which leaves the local socket.
|
||||
func (d *Daemon) ovnSBAddr() string {
|
||||
if d.clusterConfig == nil {
|
||||
return ""
|
||||
}
|
||||
return d.clusterConfig.Nodes[d.clusterConfig.Node].VPCD.OVNSBAddr
|
||||
}
|
||||
|
||||
// installOCIAllocators builds one OCI allocator per source="oci" pool and
|
||||
// reconciles it against OCI before it serves anything. The reconcile is on the
|
||||
// startup path on purpose: Allocate creates OCI objects before writing its
|
||||
// record, so a crash in between leaves a reserved public IP that bills and
|
||||
// holds one of the 50 per-region slots with nothing referencing it, and this
|
||||
// pass is the only thing that ever finds it.
|
||||
func (d *Daemon) installOCIAllocators(ipam *handlers_ec2_vpc.ExternalIPAM, js jetstream.JetStream) error {
|
||||
for _, p := range ipam.PoolsWithSource(external.SourceOCI) {
|
||||
alloc, err := ocinet.FromPoolConfig(d.ctx, js, p, d.ovnSBAddr())
|
||||
if err != nil {
|
||||
return fmt.Errorf("build OCI allocator for pool %q: %w", p.Name, err)
|
||||
}
|
||||
if err := ipam.InstallAllocator(p.Name, alloc); err != nil {
|
||||
return err
|
||||
}
|
||||
// A reconcile failure is not fatal: it leaks money, not correctness,
|
||||
// and refusing to start would take EIPs down over a transient API
|
||||
// error. It is logged at Error so it is never silently skipped.
|
||||
res, err := alloc.Reconcile(d.ctx)
|
||||
if err != nil {
|
||||
slog.Error("OCI allocator reconcile failed; leaked addresses may be billing",
|
||||
"pool", p.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
slog.Info("OCI allocator ready", "pool", p.Name,
|
||||
"collected", len(res.Collected), "stale_bindings", len(res.Stale), "skipped", res.Skipped)
|
||||
go d.runOCIAffinityLoop(alloc, p.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ociAffinityInterval is how often a node checks that OCI delivers its guests'
|
||||
// addresses to it. A guest that has just started elsewhere is dark until the
|
||||
// pass runs, so it is short relative to a boot — the address is back before the
|
||||
// guest has finished coming up.
|
||||
const ociAffinityInterval = 15 * time.Second
|
||||
|
||||
// runOCIAffinityLoop keeps OCI's idea of where an address lives in step with
|
||||
// where its guest actually runs. Every node runs its own, like the host EIP
|
||||
// loop and for the same reason: the question is about this host's guests, and a
|
||||
// node that never wins the reconcile lease would otherwise never claim the
|
||||
// addresses of the instances it is running.
|
||||
func (d *Daemon) runOCIAffinityLoop(alloc *ocinet.PoolAllocator, poolName string) {
|
||||
ticker := time.NewTicker(ociAffinityInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-d.ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
res, err := alloc.ClaimLocalAddresses(d.ctx)
|
||||
if err != nil {
|
||||
slog.Error("OCI address affinity pass failed; a guest here may be unreachable on its public address",
|
||||
"pool", poolName, "err", err)
|
||||
continue
|
||||
}
|
||||
if len(res.Claimed) > 0 {
|
||||
slog.Info("OCI addresses moved to this node", "pool", poolName,
|
||||
"claimed", res.Claimed, "local_bindings", res.Local)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// hasPublicIPPools reports whether the cluster can allocate routable public
|
||||
// IPs: pool mode always, nat mode only with a public pool beside the transit.
|
||||
func (d *Daemon) hasPublicIPPools() bool {
|
||||
@@ -1691,6 +1782,9 @@ func (d *Daemon) startCluster() error {
|
||||
return nil, fmt.Errorf("enable DHCP allocator: %w", dhcpErr)
|
||||
}
|
||||
}
|
||||
if ociErr := d.installOCIAllocators(ipam, js); ociErr != nil {
|
||||
return nil, ociErr
|
||||
}
|
||||
return ipam, nil
|
||||
})
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package daemon
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
handlers_ec2_eip "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eip"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
)
|
||||
|
||||
// handleENIPublicIPChanged applies an EIP association to the instance record on
|
||||
// whichever node is running it. AssociateAddress is served by a queue-group
|
||||
// worker that is rarely that node, so without this the address was written to
|
||||
// the EIP and ENI records while the instance kept reporting whatever launch
|
||||
// assigned it — the stale address survived the association and every describe.
|
||||
//
|
||||
// Every node receives this and all but one ignore it: UpdateState reports the
|
||||
// instance absent, which is the normal case, not a failure.
|
||||
func (d *Daemon) handleENIPublicIPChanged(msg *nats.Msg) string {
|
||||
var evt handlers_ec2_eip.ENIPublicIPChanged
|
||||
if err := json.Unmarshal(msg.Data, &evt); err != nil {
|
||||
slog.Error("public-IP change: bad event", "err", err)
|
||||
return outcomeError
|
||||
}
|
||||
if evt.InstanceID == "" {
|
||||
return outcomeSuccess
|
||||
}
|
||||
|
||||
found, err := d.vmMgr.UpdateAndPersist(evt.InstanceID, func(v *vm.VM) bool {
|
||||
if v.PublicIP == evt.PublicIP {
|
||||
return false
|
||||
}
|
||||
v.PublicIP = evt.PublicIP
|
||||
v.PublicIPPool = evt.PoolName
|
||||
return true
|
||||
})
|
||||
if err != nil {
|
||||
slog.Error("public-IP change: failed to persist the new address",
|
||||
"instanceId", evt.InstanceID, "publicIp", evt.PublicIP, "err", err)
|
||||
return outcomeError
|
||||
}
|
||||
if found {
|
||||
slog.Info("public-IP change applied to a local instance",
|
||||
"instanceId", evt.InstanceID, "eniId", evt.ENIID, "publicIp", evt.PublicIP)
|
||||
}
|
||||
return outcomeSuccess
|
||||
}
|
||||
@@ -768,7 +768,9 @@ func (a *instanceCleanerAdapter) CleanupMgmtNetwork(instance *vm.VM) {
|
||||
|
||||
// ReleasePublicIP publishes vpc.delete-nat for the OVN dnat_and_snat rule
|
||||
// and releases the public IP back to the external IPAM pool. No-op when
|
||||
// the instance has no public IP.
|
||||
// the instance has no public IP. Terminate calls it for whatever the instance
|
||||
// holds; the stop path calls it only once it has established the address is
|
||||
// not an Elastic IP.
|
||||
func (a *instanceCleanerAdapter) ReleasePublicIP(instance *vm.VM) error {
|
||||
if instance.PublicIP == "" || instance.PublicIPPool == "" || a.d.externalIPAM == nil {
|
||||
return nil
|
||||
@@ -800,15 +802,59 @@ func (a *instanceCleanerAdapter) ReleasePublicIP(instance *vm.VM) error {
|
||||
"instanceId", instance.ID, "err", err)
|
||||
return nil
|
||||
}
|
||||
slog.Warn("Failed to release public IP on termination",
|
||||
slog.Warn("Failed to release public IP",
|
||||
"ip", instance.PublicIP, "pool", instance.PublicIPPool, "err", err)
|
||||
return err
|
||||
}
|
||||
slog.Info("Released public IP on termination",
|
||||
slog.Info("Released public IP",
|
||||
"ip", instance.PublicIP, "instanceId", instance.ID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// ReleaseAutoAssignedPublicIP returns an auto-assigned address to its pool when
|
||||
// the instance stops, as AWS does. Three things can be holding the address and
|
||||
// only one of them may be reclaimed here: an EIP allocated through the EIP
|
||||
// service (PublicIPAllocID), an EIP associated to this ENI after the fact, or
|
||||
// the address the launch borrowed. The first two are the customer's and stay.
|
||||
func (a *instanceCleanerAdapter) ReleaseAutoAssignedPublicIP(instance *vm.VM) (bool, error) {
|
||||
if instance.PublicIP == "" || instance.PublicIPPool == "" || a.d.externalIPAM == nil {
|
||||
return false, nil
|
||||
}
|
||||
if instance.PublicIPAllocID != "" {
|
||||
return false, nil
|
||||
}
|
||||
ctx := context.Background()
|
||||
if instance.ENIId != "" && a.d.vpcService != nil {
|
||||
owned, err := a.d.vpcService.ENIHasEIP(ctx, instance.AccountID, instance.ENIId)
|
||||
if err != nil {
|
||||
// Keeping the address costs a pool slot until terminate; releasing one
|
||||
// we could not prove is ours destroys a customer's Elastic IP.
|
||||
return false, fmt.Errorf("check EIP association for %s: %w", instance.ENIId, err)
|
||||
}
|
||||
if owned {
|
||||
slog.Info("Stop kept the public address — it belongs to an Elastic IP",
|
||||
"instanceId", instance.ID, "ip", instance.PublicIP, "eni", instance.ENIId)
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
if err := a.ReleasePublicIP(instance); err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// The reconciler rebuilds the NAT rule from the ENI record, so leaving the
|
||||
// address on it puts back the rule the release just took down.
|
||||
if instance.ENIId != "" && a.d.vpcService != nil {
|
||||
if err := a.d.vpcService.UpdateENIPublicIP(instance.AccountID, instance.ENIId, "", ""); err != nil {
|
||||
slog.Warn("Failed to clear ENI public IP on stop",
|
||||
"instanceId", instance.ID, "eni", instance.ENIId, "err", err)
|
||||
}
|
||||
}
|
||||
slog.Info("Released auto-assigned public IP on stop",
|
||||
"instanceId", instance.ID, "ip", instance.PublicIP, "pool", instance.PublicIPPool)
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// DetachAndDeleteENI detaches the auto-created ENI from the instance and
|
||||
// deletes it via the VPC service. NotFound is tolerated. It then sweeps every
|
||||
// other ENI the KV records against this instance — post-launch attaches
|
||||
|
||||
@@ -54,6 +54,8 @@ func (a *daemonENICreator) GetENI(_ context.Context, accountID, eniID string) (*
|
||||
MacAddress: rec.MacAddress,
|
||||
Status: rec.Status,
|
||||
SecurityGroupIDs: rec.SecurityGroupIds,
|
||||
PublicIpAddress: rec.PublicIpAddress,
|
||||
PublicIpPool: rec.PublicIpPool,
|
||||
DeleteOnTermination: rec.DeleteOnTermination == nil ||
|
||||
*rec.DeleteOnTermination,
|
||||
}, nil
|
||||
@@ -75,6 +77,10 @@ func (a *daemonENICreator) UpdateENIPublicIP(_ context.Context, accountID, eniID
|
||||
return a.d.vpcService.UpdateENIPublicIP(accountID, eniID, publicIP, poolName)
|
||||
}
|
||||
|
||||
func (a *daemonENICreator) ENIHasEIP(ctx context.Context, accountID, eniID string) (bool, error) {
|
||||
return a.d.vpcService.ENIHasEIP(ctx, accountID, eniID)
|
||||
}
|
||||
|
||||
func (a *daemonENICreator) ListInstanceENIs(_ context.Context, accountID, instanceID string) ([]handlers_ec2_instance.ENIInfo, error) {
|
||||
records, err := a.d.vpcService.ListInstanceENIs(accountID, instanceID)
|
||||
if err != nil {
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/bluebottle/pkg/tlsconfig"
|
||||
"github.com/mulgadc/spinifex/spinifex/admin"
|
||||
)
|
||||
|
||||
// NodeInfo describes a node participating in cluster formation.
|
||||
@@ -63,16 +64,14 @@ type StatusResponse struct {
|
||||
// NetworkConfig holds the cluster-wide external networking configuration
|
||||
// propagated from the init node to joining nodes during formation.
|
||||
type NetworkConfig struct {
|
||||
ExternalMode string `json:"external_mode"`
|
||||
PoolName string `json:"pool_name"`
|
||||
PoolSource string `json:"pool_source,omitempty"`
|
||||
PoolBindBridge string `json:"pool_bind_bridge,omitempty"`
|
||||
PoolStart string `json:"pool_start,omitempty"`
|
||||
PoolEnd string `json:"pool_end,omitempty"`
|
||||
PoolGateway string `json:"pool_gateway"`
|
||||
PoolGatewayIP string `json:"pool_gateway_ip,omitempty"`
|
||||
PoolPrefixLen int `json:"pool_prefix_len"`
|
||||
PoolDNSServers []string `json:"pool_dns_servers,omitempty"`
|
||||
ExternalMode string `json:"external_mode"`
|
||||
|
||||
// Pools is every [[network.external_pools]] entry the init node rendered, in
|
||||
// order. A list rather than one flattened pool because routed NAT has two —
|
||||
// the transit segment and any public pool — and the transit pool carries a
|
||||
// gateway-LRP range that has no equivalent in pool mode.
|
||||
Pools []admin.PoolData `json:"pools,omitempty"`
|
||||
PoolDNSServers []string `json:"pool_dns_servers,omitempty"`
|
||||
|
||||
// IPSecEnabled propagates the cluster-wide intra-AZ IPsec toggle so the
|
||||
// joining node provisions strongSwan and flips OVS ipsec_encapsulation to
|
||||
|
||||
@@ -482,6 +482,14 @@ var ec2Actions = map[string]ec2Action{
|
||||
"DescribeVpcAttribute": ec2Handler(func(ctx context.Context, input *ec2.DescribeVpcAttributeInput, gw *GatewayConfig, accountID string) (any, error) {
|
||||
return gateway_ec2_vpc.DescribeVpcAttribute(ctx, input, gw.NATSConn, accountID)
|
||||
}),
|
||||
// No VPC endpoint can exist here, because CreateVpcEndpoint is not served, so
|
||||
// the empty list is the true answer rather than a stand-in for one. It has to
|
||||
// be answered all the same: the Terraform AWS provider calls this on every VPC
|
||||
// delete to sweep GuardDuty-managed endpoints, and an InvalidAction there fails
|
||||
// the destroy and strands the VPC.
|
||||
"DescribeVpcEndpoints": ec2Handler(func(_ context.Context, _ *ec2.DescribeVpcEndpointsInput, _ *GatewayConfig, _ string) (any, error) {
|
||||
return ec2.DescribeVpcEndpointsOutput{VpcEndpoints: []*ec2.VpcEndpoint{}}, nil
|
||||
}),
|
||||
"CreateSubnet": ec2Handler(func(ctx context.Context, input *ec2.CreateSubnetInput, gw *GatewayConfig, accountID string) (any, error) {
|
||||
if err := gw.Quota.EnforceSubnets(ctx, gw.NATSConn, accountID, 1); err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -295,6 +295,7 @@ var ec2Scopes = map[string][]*resourceScope{
|
||||
"DescribeVolumes": unscoped,
|
||||
"DescribeVolumesModifications": unscoped,
|
||||
"DescribeVpcAttribute": unscoped,
|
||||
"DescribeVpcEndpoints": unscoped,
|
||||
"DescribeVpcs": unscoped,
|
||||
}
|
||||
|
||||
|
||||
@@ -40,6 +40,21 @@ type EIPServiceImpl struct {
|
||||
allocSF singleflight.Group
|
||||
}
|
||||
|
||||
// SubjectENIPublicIPChanged carries an EIP association to every daemon, because
|
||||
// the node that serves AssociateAddress is load-balanced and is rarely the one
|
||||
// running the instance. Fan-out with no queue group: the node holding the
|
||||
// instance is the only one that acts, and every other ignores it.
|
||||
const SubjectENIPublicIPChanged = "ec2.eni.public-ip-changed"
|
||||
|
||||
// ENIPublicIPChanged is the payload on SubjectENIPublicIPChanged. An empty
|
||||
// PublicIP is a disassociate.
|
||||
type ENIPublicIPChanged struct {
|
||||
InstanceID string `json:"instance_id"`
|
||||
ENIID string `json:"eni_id"`
|
||||
PublicIP string `json:"public_ip"`
|
||||
PoolName string `json:"pool_name,omitempty"`
|
||||
}
|
||||
|
||||
// natEvent is the payload published to vpc.add-nat / vpc.delete-nat topics.
|
||||
type natEvent struct {
|
||||
VpcId string `json:"vpc_id"`
|
||||
@@ -258,6 +273,10 @@ func (s *EIPServiceImpl) AssociateAddress(ctx context.Context, input *ec2.Associ
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
s.bindPoolOwner(ctx, record.PoolName, record.PublicIp, eniID)
|
||||
s.stampENIPublicIP(ctx, accountID, eniID, record.PublicIp, record.PoolName)
|
||||
s.announcePublicIP(ctx, instanceID, eniID, record.PublicIp, record.PoolName)
|
||||
|
||||
// Publish vpc.add-nat event (fire-and-forget).
|
||||
s.publishNATEvent("vpc.add-nat", vpcID, record.PublicIp, privateIP, eniID, macAddr)
|
||||
|
||||
@@ -299,6 +318,8 @@ func (s *EIPServiceImpl) DisassociateAddress(ctx context.Context, input *ec2.Dis
|
||||
s.publishNATEvent("vpc.delete-nat", record.VpcId, record.PublicIp, record.PrivateIp, record.ENIId, macAddr)
|
||||
}
|
||||
|
||||
poolName, publicIP := record.PoolName, record.PublicIp
|
||||
eniID, instanceID := record.ENIId, record.InstanceId
|
||||
clearAssociation(record)
|
||||
|
||||
data, err := json.Marshal(record)
|
||||
@@ -308,12 +329,84 @@ func (s *EIPServiceImpl) DisassociateAddress(ctx context.Context, input *ec2.Dis
|
||||
if _, err := s.eipKV.Update(ctx, key, data, revision); err != nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
s.bindPoolOwner(ctx, poolName, publicIP, "")
|
||||
s.stampENIPublicIP(ctx, accountID, eniID, "", "")
|
||||
s.announcePublicIP(ctx, instanceID, eniID, "", "")
|
||||
|
||||
slog.InfoContext(ctx, "DisassociateAddress completed", "associationId", associationID, "accountID", accountID)
|
||||
|
||||
return &ec2.DisassociateAddressOutput{}, nil
|
||||
}
|
||||
|
||||
// bindPoolOwner tells the pool which ENI now holds publicIP, empty for none.
|
||||
// Most pools have nothing to do with it; a pool whose addresses are delivered
|
||||
// to one node needs it, because an EIP is allocated with no ENI and this is the
|
||||
// only point the owner is ever known.
|
||||
//
|
||||
// Logged rather than returned: the association itself has already been written
|
||||
// and is what AWS answers on, so failing the call here would report an error for
|
||||
// an association that did happen. The pool's own reconcile converges it.
|
||||
func (s *EIPServiceImpl) bindPoolOwner(ctx context.Context, poolName, publicIP, eniID string) {
|
||||
if poolName == "" || publicIP == "" {
|
||||
return
|
||||
}
|
||||
if err := s.externalIPAM.BindOwner(ctx, poolName, publicIP, eniID); err != nil {
|
||||
slog.WarnContext(ctx, "failed to record the EIP owner with its pool",
|
||||
"pool", poolName, "publicIp", publicIP, "eniId", eniID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// announcePublicIP tells every daemon that eniID's public address changed, so
|
||||
// the one running the instance can report it. Best-effort: the EIP and ENI
|
||||
// records are already written and are what the next reconcile reads, so a lost
|
||||
// announcement costs a stale DescribeInstances until then, not correctness.
|
||||
func (s *EIPServiceImpl) announcePublicIP(ctx context.Context, instanceID, eniID, publicIP, poolName string) {
|
||||
if s.natsConn == nil || instanceID == "" {
|
||||
return
|
||||
}
|
||||
payload, err := json.Marshal(ENIPublicIPChanged{
|
||||
InstanceID: instanceID, ENIID: eniID, PublicIP: publicIP, PoolName: poolName,
|
||||
})
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "failed to encode the public-IP change announcement",
|
||||
"instanceId", instanceID, "err", err)
|
||||
return
|
||||
}
|
||||
if err := s.natsConn.Publish(SubjectENIPublicIPChanged, payload); err != nil {
|
||||
slog.WarnContext(ctx, "failed to announce the public-IP change",
|
||||
"instanceId", instanceID, "eniId", eniID, "publicIp", publicIP, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// eniPublicIPStamper is the ENI-record write the EIP service needs. Narrowed to
|
||||
// this one method rather than widening VPCService, which every other consumer
|
||||
// would then have to implement.
|
||||
type eniPublicIPStamper interface {
|
||||
UpdateENIPublicIP(accountID, eniID, publicIP, poolName string) error
|
||||
}
|
||||
|
||||
// stampENIPublicIP records the address on the ENI, empty to clear it. The ENI
|
||||
// record is what IMDS serves and what the network reconcile reads, so without
|
||||
// this an associated EIP was visible only on the EIP record and the interface
|
||||
// still advertised whatever launch assigned it.
|
||||
//
|
||||
// Logged rather than returned, like bindPoolOwner: the association is already
|
||||
// written and answered on, so failing here would report an error for something
|
||||
// that did happen.
|
||||
func (s *EIPServiceImpl) stampENIPublicIP(ctx context.Context, accountID, eniID, publicIP, poolName string) {
|
||||
if eniID == "" {
|
||||
return
|
||||
}
|
||||
stamper, ok := s.vpcService.(eniPublicIPStamper)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := stamper.UpdateENIPublicIP(accountID, eniID, publicIP, poolName); err != nil {
|
||||
slog.WarnContext(ctx, "failed to stamp the public IP on the ENI record",
|
||||
"eniId", eniID, "publicIp", publicIP, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// clearAssociation reverts a record to the unassociated "allocated" state. The
|
||||
// allocation itself survives: disassociating never hands the address back.
|
||||
func clearAssociation(record *EIPRecord) {
|
||||
@@ -352,7 +445,7 @@ func (s *EIPServiceImpl) DisassociateByENI(ctx context.Context, accountID, eniID
|
||||
// as part of tearing that ENI down, so the interface may already be gone.
|
||||
s.publishNATEvent("vpc.delete-nat", record.VpcId, record.PublicIp, record.PrivateIp, eniID, record.MacAddress)
|
||||
|
||||
publicIP, allocationID := record.PublicIp, record.AllocationId
|
||||
publicIP, allocationID, poolName := record.PublicIp, record.AllocationId, record.PoolName
|
||||
clearAssociation(record)
|
||||
|
||||
data, err := json.Marshal(record)
|
||||
@@ -362,6 +455,8 @@ func (s *EIPServiceImpl) DisassociateByENI(ctx context.Context, accountID, eniID
|
||||
if _, err := s.eipKV.Update(ctx, m.key, data, m.revision); err != nil {
|
||||
return false, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
s.bindPoolOwner(ctx, poolName, publicIP, "")
|
||||
// No ENI stamp to clear: this runs while that ENI is being deleted.
|
||||
|
||||
slog.InfoContext(ctx, "DisassociateByENI completed",
|
||||
"eniId", eniID, "publicIp", publicIP, "allocationId", allocationID, "accountID", accountID)
|
||||
|
||||
@@ -52,8 +52,18 @@ func ProjectInstance(v *vm.VM, cfg InstanceProjection) (inst *ec2.Instance, stat
|
||||
// Public IP and its derived DNS names exist only while the instance runs; a
|
||||
// stopped instance has released the public IP, so the KV path leaves them
|
||||
// unset. Mirrors the records the control-plane writer publishes to northstar.
|
||||
if !cfg.IncludeRuntimeNetwork {
|
||||
// Cleared, not merely left alone: launch stamps the auto-assigned address
|
||||
// onto the stored instance, so a stopped instance kept reporting an
|
||||
// address that had been released and may already belong to someone else.
|
||||
instanceCopy.PublicIpAddress = nil
|
||||
instanceCopy.PublicDnsName = nil
|
||||
}
|
||||
if cfg.IncludeRuntimeNetwork {
|
||||
if v.PublicIP != "" && instanceCopy.PublicIpAddress == nil {
|
||||
// The runtime address wins over the launch stamp. Associating an EIP
|
||||
// replaces what an instance is reachable on, and filling only a nil field
|
||||
// could never express that — the stamp is always present.
|
||||
if v.PublicIP != "" {
|
||||
instanceCopy.PublicIpAddress = aws.String(v.PublicIP)
|
||||
}
|
||||
|
||||
@@ -172,7 +182,12 @@ func ProjectInstance(v *vm.VM, cfg InstanceProjection) (inst *ec2.Instance, stat
|
||||
// force-new, so leaving Association unset reports false for an instance
|
||||
// that has a public IP and every plan proposes a destroy/recreate.
|
||||
isPrimaryNIC := nicCopy.Attachment != nil && nicCopy.Attachment.DeviceIndex != nil && *nicCopy.Attachment.DeviceIndex == 0
|
||||
if cfg.IncludeRuntimeNetwork && isPrimaryNIC && v.PublicIP != "" {
|
||||
switch {
|
||||
case !cfg.IncludeRuntimeNetwork:
|
||||
// Same reason the instance-level address is cleared: the stored
|
||||
// association is the launch-time one and outlives the address.
|
||||
nicCopy.Association = nil
|
||||
case isPrimaryNIC && v.PublicIP != "":
|
||||
nicCopy.Association = &ec2.InstanceNetworkInterfaceAssociation{
|
||||
PublicIp: aws.String(v.PublicIP),
|
||||
PublicDnsName: aws.String(publicDNS),
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
//test:in-package — extends projection_test.go's in-package fullVM fixture and
|
||||
// projects through the unexported field set it builds.
|
||||
|
||||
package handlers_ec2_instance
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// stampedVM is a launched instance: the auto-assigned address is written onto
|
||||
// the stored ec2.Instance at launch and stays there for the instance's life.
|
||||
func stampedVM(status vm.InstanceState, runtimePublicIP string) *vm.VM {
|
||||
v := fullVM(status)
|
||||
v.PublicIP = runtimePublicIP
|
||||
v.Instance.PublicIpAddress = aws.String("203.0.113.7")
|
||||
v.Instance.PublicDnsName = aws.String("ec2-203-0-113-7.example.com")
|
||||
v.Instance.NetworkInterfaces = []*ec2.InstanceNetworkInterface{{
|
||||
NetworkInterfaceId: aws.String("eni-1"),
|
||||
Attachment: &ec2.InstanceNetworkInterfaceAttachment{DeviceIndex: aws.Int64(0)},
|
||||
Association: &ec2.InstanceNetworkInterfaceAssociation{
|
||||
PublicIp: aws.String("203.0.113.7"),
|
||||
},
|
||||
}}
|
||||
return v
|
||||
}
|
||||
|
||||
// AWS releases an auto-assigned address when an instance stops. Ours kept
|
||||
// reporting it, because the launch stamp lives on the stored instance and the
|
||||
// stopped path only declined to add an address — it never removed one.
|
||||
func TestStoppedInstanceReportsNoPublicAddress(t *testing.T) {
|
||||
v := stampedVM(vm.StateStopped, "")
|
||||
|
||||
got, _ := ProjectInstance(v, InstanceProjection{
|
||||
Region: "us-west-1", IncludeRuntimeNetwork: false,
|
||||
FallbackStateCode: 80, FallbackStateName: "stopped",
|
||||
})
|
||||
|
||||
assert.Nil(t, got.PublicIpAddress, "a stopped instance still advertises a released address")
|
||||
assert.Nil(t, got.PublicDnsName)
|
||||
require.Len(t, got.NetworkInterfaces, 1)
|
||||
assert.Nil(t, got.NetworkInterfaces[0].Association,
|
||||
"the primary NIC still advertises the released association")
|
||||
}
|
||||
|
||||
// Clearing the projection must not reach into the stored instance, or the
|
||||
// address is gone for good and a restart cannot report it again.
|
||||
func TestClearingTheStoppedAddressLeavesTheStoredInstanceIntact(t *testing.T) {
|
||||
v := stampedVM(vm.StateStopped, "")
|
||||
|
||||
_, _ = ProjectInstance(v, InstanceProjection{
|
||||
Region: "us-west-1", IncludeRuntimeNetwork: false,
|
||||
FallbackStateCode: 80, FallbackStateName: "stopped",
|
||||
})
|
||||
|
||||
assert.Equal(t, "203.0.113.7", aws.StringValue(v.Instance.PublicIpAddress))
|
||||
require.Len(t, v.Instance.NetworkInterfaces, 1)
|
||||
require.NotNil(t, v.Instance.NetworkInterfaces[0].Association)
|
||||
}
|
||||
|
||||
// Associating an EIP replaces what the instance is reachable on. The old
|
||||
// projection only filled a nil field, and the launch stamp is never nil, so the
|
||||
// EIP could not be reported however correctly it was recorded.
|
||||
func TestTheRuntimeAddressWinsOverTheLaunchStamp(t *testing.T) {
|
||||
v := stampedVM(vm.StateRunning, "198.51.100.42")
|
||||
|
||||
got, _ := ProjectInstance(v, InstanceProjection{
|
||||
Region: "us-west-1", IncludeRuntimeNetwork: true,
|
||||
FallbackStateCode: 16, FallbackStateName: "running",
|
||||
})
|
||||
|
||||
assert.Equal(t, "198.51.100.42", aws.StringValue(got.PublicIpAddress))
|
||||
require.Len(t, got.NetworkInterfaces, 1)
|
||||
require.NotNil(t, got.NetworkInterfaces[0].Association)
|
||||
assert.Equal(t, "198.51.100.42", aws.StringValue(got.NetworkInterfaces[0].Association.PublicIp))
|
||||
}
|
||||
|
||||
// With no EIP the auto-assigned address is still the answer, so the stamp must
|
||||
// survive a running projection.
|
||||
func TestTheLaunchStampStandsWhenThereIsNoRuntimeAddress(t *testing.T) {
|
||||
v := stampedVM(vm.StateRunning, "")
|
||||
|
||||
got, _ := ProjectInstance(v, InstanceProjection{
|
||||
Region: "us-west-1", IncludeRuntimeNetwork: true,
|
||||
FallbackStateCode: 16, FallbackStateName: "running",
|
||||
})
|
||||
|
||||
assert.Equal(t, "203.0.113.7", aws.StringValue(got.PublicIpAddress))
|
||||
}
|
||||
@@ -188,6 +188,11 @@ type ENIInfo struct {
|
||||
MacAddress string
|
||||
Status string
|
||||
SecurityGroupIDs []string
|
||||
// PublicIpAddress / PublicIpPool are the address the interface is currently
|
||||
// reachable on. The record outlives a stop, so a start reads them to find an
|
||||
// Elastic IP associated while the instance was down.
|
||||
PublicIpAddress string
|
||||
PublicIpPool string
|
||||
// DeleteOnTermination mirrors the stored ENIRecord field, defaulted true
|
||||
// when unset. Read by the terminate sweep to decide detach-only vs delete.
|
||||
DeleteOnTermination bool
|
||||
@@ -205,6 +210,11 @@ type ENICreator interface {
|
||||
AttachENI(ctx context.Context, accountID, eniID, instanceID string, deviceIndex int64) (string, error)
|
||||
DetachENI(ctx context.Context, accountID, eniID string) error
|
||||
UpdateENIPublicIP(ctx context.Context, accountID, eniID, publicIP, poolName string) error
|
||||
// ENIHasEIP reports whether an Elastic IP is associated with the ENI. A
|
||||
// start must not auto-assign an address beside one the customer already
|
||||
// owns: the EIP is what the instance is reachable on, and a second address
|
||||
// is one they never asked for and would be billed for on some providers.
|
||||
ENIHasEIP(ctx context.Context, accountID, eniID string) (bool, error)
|
||||
// ListInstanceENIs returns every ENI currently attached to instanceID.
|
||||
// Used by the terminate sweep for post-launch attachments the launch-time
|
||||
// ENIId scalar on vm.VM never carries.
|
||||
|
||||
@@ -2658,8 +2658,29 @@ func (s *InstanceServiceImpl) StartStoppedInstance(ctx context.Context, input *S
|
||||
"count", len(attachments), "attachments", attachments)
|
||||
}
|
||||
|
||||
// Before Run, so a pool with nothing left refuses the start instead of
|
||||
// booting a guest the customer cannot reach.
|
||||
addressAllocated, err := s.reassignAutoAssignedPublicIP(ctx, instance)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "StartStoppedInstance: public IP allocation failed — refusing the start",
|
||||
"instanceId", input.InstanceID, "err", err)
|
||||
if gpuClaimed {
|
||||
if relErr := s.gpuClaimer.Release(instance.ID); relErr != nil {
|
||||
slog.ErrorContext(ctx, "StartStoppedInstance: GPU release failed after address allocation failure",
|
||||
"instanceId", input.InstanceID, "err", relErr)
|
||||
}
|
||||
}
|
||||
s.resourceMgr.Deallocate(instanceType)
|
||||
s.vmMgr.Delete(instance.ID)
|
||||
s.restoreClaimedStoppedInstance(ctx, instance)
|
||||
return nil, errors.New(awserrors.ValidErrorCodeFromError(err))
|
||||
}
|
||||
|
||||
if err := s.vmMgr.Run(ctx, instance); err != nil {
|
||||
slog.ErrorContext(ctx, "StartStoppedInstance: vmMgr.Run failed", "instanceId", input.InstanceID, "err", err)
|
||||
if addressAllocated {
|
||||
s.rollbackStartPublicIP(ctx, instance)
|
||||
}
|
||||
if gpuClaimed {
|
||||
if relErr := s.gpuClaimer.Release(instance.ID); relErr != nil {
|
||||
slog.ErrorContext(ctx, "StartStoppedInstance: GPU release failed after launch failure",
|
||||
@@ -2881,6 +2902,110 @@ func (s *InstanceServiceImpl) allocatePublicIP(ctx context.Context, eniID, insta
|
||||
return s.ipAllocator.AllocateIP(ctx, region, az, handlers_ec2_vpc.PurposeENIPublic, "", eniID, instanceID)
|
||||
}
|
||||
|
||||
// reassignAutoAssignedPublicIP gives a starting instance the fresh address AWS
|
||||
// would. The one it had went back to its pool when it stopped, so an address is
|
||||
// taken here and stamped everywhere the datapath reads it.
|
||||
//
|
||||
// It does nothing unless the stop actually released one, which is what keeps an
|
||||
// instance that never had a public address, one holding an Elastic IP, and one
|
||||
// stopped by a build that predates this from being handed an address on start.
|
||||
// An exhausted pool is returned as an error so the caller refuses the start
|
||||
// rather than booting a guest the customer cannot reach.
|
||||
//
|
||||
// allocated reports whether an address came out of the pool here, which is the
|
||||
// only case a later failure may hand back: an Elastic IP adopted below is the
|
||||
// customer's and a rollback must not touch it.
|
||||
func (s *InstanceServiceImpl) reassignAutoAssignedPublicIP(ctx context.Context, instance *vm.VM) (allocated bool, err error) {
|
||||
if !instance.AutoAssignPublicIP || instance.PublicIP != "" || instance.ENIId == "" {
|
||||
return false, nil
|
||||
}
|
||||
// The flag is cleared only once the instance is holding an address, so a
|
||||
// refused start stays owed one and the retry does not bring it up with none.
|
||||
eni, err := s.eniInfoForStart(ctx, instance)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// An Elastic IP associated while the instance was stopped is what it comes
|
||||
// back on. Auto-assigning here would hand the customer a second address they
|
||||
// never asked for, so adopt the EIP instead and take nothing from the pool.
|
||||
if s.eniCreator != nil {
|
||||
owned, eipErr := s.eniCreator.ENIHasEIP(ctx, instance.AccountID, instance.ENIId)
|
||||
if eipErr != nil {
|
||||
return false, fmt.Errorf("check EIP association for %s: %w", instance.ENIId, eipErr)
|
||||
}
|
||||
if owned {
|
||||
instance.PublicIP = eni.PublicIpAddress
|
||||
instance.PublicIPPool = eni.PublicIpPool
|
||||
instance.AutoAssignPublicIP = false
|
||||
slog.InfoContext(ctx, "StartStoppedInstance: instance starts on the Elastic IP associated to its interface",
|
||||
"instanceId", instance.ID, "eniId", instance.ENIId, "publicIp", instance.PublicIP)
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
publicIP, poolName, err := s.allocatePublicIP(ctx, instance.ENIId, instance.ID)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("allocate public IP for %s: %w", instance.ID, err)
|
||||
}
|
||||
if updateErr := s.eniCreator.UpdateENIPublicIP(ctx, instance.AccountID, instance.ENIId, publicIP, poolName); updateErr != nil {
|
||||
// The reconciler builds the NAT rule from this record, so an address the
|
||||
// instance holds but the record does not is an address nothing routes.
|
||||
slog.WarnContext(ctx, "StartStoppedInstance: failed to update ENI with public IP",
|
||||
"instanceId", instance.ID, "eniId", instance.ENIId, "err", updateErr)
|
||||
}
|
||||
if natErr := utils.AddNAT(s.natsConn, eni.VpcID, publicIP, eni.PrivateIpAddress,
|
||||
topology.Port(instance.ENIId), eni.MacAddress); natErr != nil {
|
||||
// Neutralise before releasing in case the timeout committed the rule.
|
||||
utils.PublishNATEvent(s.natsConn, "vpc.delete-nat", eni.VpcID, publicIP, eni.PrivateIpAddress,
|
||||
topology.Port(instance.ENIId), eni.MacAddress)
|
||||
s.releaseAutoAssignedPublicIP(ctx, instance.AccountID, instance.ENIId, publicIP, poolName)
|
||||
return false, natErr
|
||||
}
|
||||
|
||||
instance.PublicIP = publicIP
|
||||
instance.PublicIPPool = poolName
|
||||
instance.AutoAssignPublicIP = false
|
||||
slog.InfoContext(ctx, "StartStoppedInstance: auto-assigned a new public IP",
|
||||
"instanceId", instance.ID, "publicIp", publicIP, "pool", poolName)
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// eniInfoForStart reads the interface a start is bringing back up. The stored
|
||||
// instance carries the launch-time copy, which a cross-node start and an EIP
|
||||
// associated while stopped both outdate, so the record is read fresh.
|
||||
func (s *InstanceServiceImpl) eniInfoForStart(ctx context.Context, instance *vm.VM) (*ENIInfo, error) {
|
||||
if s.eniCreator == nil {
|
||||
return nil, fmt.Errorf("no ENI service on this node: %w", errors.New(awserrors.ErrorServerInternal))
|
||||
}
|
||||
eni, err := s.eniCreator.GetENI(ctx, instance.AccountID, instance.ENIId)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read ENI %s: %w", instance.ENIId, err)
|
||||
}
|
||||
return eni, nil
|
||||
}
|
||||
|
||||
// rollbackStartPublicIP returns the address a start took when the launch that
|
||||
// followed failed, so a refused start does not cost the pool a slot per attempt.
|
||||
// The instance goes back to the stopped store still marked for re-assignment.
|
||||
func (s *InstanceServiceImpl) rollbackStartPublicIP(ctx context.Context, instance *vm.VM) {
|
||||
if instance.PublicIP == "" || instance.PublicIPPool == "" {
|
||||
return
|
||||
}
|
||||
vpcID := ""
|
||||
privateIP := ""
|
||||
if instance.Instance != nil {
|
||||
vpcID = aws.StringValue(instance.Instance.VpcId)
|
||||
privateIP = aws.StringValue(instance.Instance.PrivateIpAddress)
|
||||
}
|
||||
utils.PublishNATEvent(s.natsConn, "vpc.delete-nat", vpcID, instance.PublicIP, privateIP,
|
||||
topology.Port(instance.ENIId), instance.ENIMac)
|
||||
s.releaseAutoAssignedPublicIP(ctx, instance.AccountID, instance.ENIId, instance.PublicIP, instance.PublicIPPool)
|
||||
instance.PublicIP = ""
|
||||
instance.PublicIPPool = ""
|
||||
instance.AutoAssignPublicIP = true
|
||||
}
|
||||
|
||||
// rollbackAutoAssignedPublicIP unwinds a failed auto-assign: clears the ENI
|
||||
// public IP record, releases the IPAM lease, then detaches and deletes the ENI.
|
||||
func (s *InstanceServiceImpl) rollbackAutoAssignedPublicIP(ctx context.Context, accountID, instanceID, eniID, publicIP, poolName string) {
|
||||
|
||||
@@ -3497,6 +3497,12 @@ type fakeENICreator struct {
|
||||
detachErr error // returned by every DetachENI call when set
|
||||
instanceENIs map[string][]ENIInfo // keyed by instanceID, for ListInstanceENIs
|
||||
listENIsErr error
|
||||
eniHasEIP bool // what ENIHasEIP reports for every ENI
|
||||
eniHasEIPErr error // when set, ENIHasEIP fails instead
|
||||
}
|
||||
|
||||
func (f *fakeENICreator) ENIHasEIP(_ context.Context, _, _ string) (bool, error) {
|
||||
return f.eniHasEIP, f.eniHasEIPErr
|
||||
}
|
||||
|
||||
func (f *fakeENICreator) GetDefaultSubnet(_ context.Context, _ string) (*SubnetInfo, error) {
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
// Builds an InstanceServiceImpl field by field so the allocator, the releaser
|
||||
// and the ENI creator are all fakes, which is the only way to see what a start
|
||||
// did to the address before the launch it then fails.
|
||||
//
|
||||
//test:in-package — every dependency it substitutes is an unexported field.
|
||||
package handlers_ec2_instance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
vmmock "github.com/mulgadc/spinifex/spinifex/vm/mock"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// refusingMounter fails the launch at its first real step, which is the
|
||||
// earliest point past the address work these tests are about. Standing up a
|
||||
// launchable guest would prove nothing extra and would need QEMU.
|
||||
type refusingMounter struct{}
|
||||
|
||||
func (refusingMounter) Mount(context.Context, *vm.VM) error { return errors.New("no volumes here") }
|
||||
func (refusingMounter) Unmount(context.Context, *vm.VM) error { return nil }
|
||||
func (refusingMounter) MountOne(context.Context, string, *types.EBSRequest) error {
|
||||
return nil
|
||||
}
|
||||
func (refusingMounter) UnmountOne(context.Context, string, types.EBSRequest) error { return nil }
|
||||
|
||||
// startFixture builds a stopped instance whose address went back to the pool,
|
||||
// wired to fakes for everything the re-assignment touches. vmMgr.Run always
|
||||
// fails in this harness, so every case asserts what the start did to the
|
||||
// address before the launch, plus whatever the failure hands back.
|
||||
type startFixture struct {
|
||||
svc *InstanceServiceImpl
|
||||
store *vmmock.StateStore
|
||||
ipam *fakeIPAllocator
|
||||
rel *fakePublicIPReleaser
|
||||
eni *fakeENICreator
|
||||
}
|
||||
|
||||
func newStartFixture(t *testing.T, stopped *vm.VM) *startFixture {
|
||||
t.Helper()
|
||||
t.Setenv("XDG_RUNTIME_DIR", t.TempDir())
|
||||
|
||||
store := &vmmock.StateStore{Stopped: map[string]*vm.VM{stopped.ID: stopped}}
|
||||
prov := &fakeResourceCapacityProvider{
|
||||
instanceTypes: map[string]*ec2.InstanceTypeInfo{
|
||||
"t3.micro": {InstanceType: aws.String("t3.micro")},
|
||||
},
|
||||
}
|
||||
eniCreator := &fakeENICreator{getENIByID: map[string]*ENIInfo{
|
||||
"eni-1": {
|
||||
NetworkInterfaceID: "eni-1",
|
||||
VpcID: "vpc-1",
|
||||
PrivateIpAddress: "10.0.0.5",
|
||||
MacAddress: "02:00:00:00:00:01",
|
||||
},
|
||||
}}
|
||||
f := &startFixture{
|
||||
store: store,
|
||||
ipam: &fakeIPAllocator{publicIP: "203.0.113.20", poolName: "wan"},
|
||||
rel: &fakePublicIPReleaser{},
|
||||
eni: eniCreator,
|
||||
}
|
||||
vmMgr := vm.NewManager()
|
||||
vmMgr.SetDeps(vm.Deps{VolumeMounter: refusingMounter{}})
|
||||
f.svc = &InstanceServiceImpl{
|
||||
stoppedStore: store,
|
||||
resourceMgr: prov,
|
||||
vmMgr: vmMgr,
|
||||
ipAllocator: f.ipam,
|
||||
ipReleaser: f.rel,
|
||||
eniCreator: eniCreator,
|
||||
natsConn: embeddedNATS(t),
|
||||
}
|
||||
|
||||
for _, topic := range []string{"vpc.add-nat", "vpc.delete-nat"} {
|
||||
sub, err := f.svc.natsConn.Subscribe(topic, func(msg *nats.Msg) {
|
||||
_ = msg.Respond([]byte(`{"success":true}`))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func (f *startFixture) start(t *testing.T, id string) error {
|
||||
t.Helper()
|
||||
_, err := f.svc.StartStoppedInstance(context.Background(), &StartStoppedInstanceInput{InstanceID: id}, "acc")
|
||||
return err
|
||||
}
|
||||
|
||||
// releasedOnStop is the shape the stop path leaves behind: no address, and the
|
||||
// marker saying one is owed.
|
||||
func releasedOnStop(id string) *vm.VM {
|
||||
return &vm.VM{
|
||||
ID: id,
|
||||
Status: vm.StateStopped,
|
||||
AccountID: "acc",
|
||||
InstanceType: "t3.micro",
|
||||
ENIId: "eni-1",
|
||||
AutoAssignPublicIP: true,
|
||||
Instance: &ec2.Instance{VpcId: aws.String("vpc-1"), PrivateIpAddress: aws.String("10.0.0.5")},
|
||||
}
|
||||
}
|
||||
|
||||
// AWS gives a restarted instance a different address than the one it stopped
|
||||
// with, so a start has to take a new one rather than assume the old one is
|
||||
// still there.
|
||||
func TestAStartTakesANewAddressForAnInstanceThatReleasedOneOnStop(t *testing.T) {
|
||||
f := newStartFixture(t, releasedOnStop("i-restart"))
|
||||
|
||||
require.Error(t, f.start(t, "i-restart"), "the harness has no runnable VM")
|
||||
|
||||
assert.Equal(t, 1, f.ipam.calls, "start did not allocate a replacement address")
|
||||
assert.Equal(t, 1, f.eni.updateCalls-f.eni.clearCalls,
|
||||
"the ENI record never learned the new address, so the reconciler would build no NAT rule")
|
||||
}
|
||||
|
||||
// An instance that never had a public address must not be handed one by a
|
||||
// start: a private-subnet guest is private on purpose.
|
||||
func TestAStartLeavesAnInstanceThatNeverHadAnAddressAlone(t *testing.T) {
|
||||
stopped := releasedOnStop("i-private")
|
||||
stopped.AutoAssignPublicIP = false
|
||||
f := newStartFixture(t, stopped)
|
||||
|
||||
require.Error(t, f.start(t, "i-private"))
|
||||
|
||||
assert.Zero(t, f.ipam.calls, "start assigned a public address to an instance that had none")
|
||||
}
|
||||
|
||||
// An Elastic IP associated while the instance was stopped is what it comes back
|
||||
// on. Auto-assigning here would hand the customer a second address they never
|
||||
// asked for and, on OCI, would bill them for it.
|
||||
func TestAStartAdoptsAnElasticIPRatherThanAssigningBesideIt(t *testing.T) {
|
||||
f := newStartFixture(t, releasedOnStop("i-eip"))
|
||||
f.eni.eniHasEIP = true
|
||||
f.eni.getENIByID["eni-1"].PublicIpAddress = "198.51.100.7"
|
||||
f.eni.getENIByID["eni-1"].PublicIpPool = "wan"
|
||||
|
||||
require.Error(t, f.start(t, "i-eip"))
|
||||
|
||||
assert.Zero(t, f.ipam.calls, "start assigned a second address beside an Elastic IP")
|
||||
restored := f.store.WroteStopped["i-eip"]
|
||||
require.NotNil(t, restored)
|
||||
assert.Equal(t, "198.51.100.7", restored.PublicIP,
|
||||
"the instance does not report the Elastic IP it is reachable on")
|
||||
}
|
||||
|
||||
// An exhausted pool is the AWS answer to a start that cannot be given an
|
||||
// address, and the instance has to stay stopped and startable rather than be
|
||||
// lost by the claim that already removed it.
|
||||
func TestAStartAgainstAnExhaustedPoolIsRefusedAndTheInstanceSurvives(t *testing.T) {
|
||||
f := newStartFixture(t, releasedOnStop("i-exhausted"))
|
||||
f.ipam.err = errors.New(awserrors.ErrorInsufficientAddressCapacity)
|
||||
|
||||
err := f.start(t, "i-exhausted")
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInsufficientAddressCapacity, err.Error())
|
||||
restored := f.store.WroteStopped["i-exhausted"]
|
||||
require.NotNil(t, restored, "the claim removed the instance and nothing put it back")
|
||||
assert.Equal(t, vm.StateStopped, restored.Status)
|
||||
assert.True(t, restored.AutoAssignPublicIP,
|
||||
"a retry would start the instance with no address at all")
|
||||
}
|
||||
|
||||
// A launch that fails after the address was taken must hand it back, or every
|
||||
// refused start costs the pool a slot.
|
||||
func TestAFailedLaunchReturnsTheAddressTheStartTook(t *testing.T) {
|
||||
f := newStartFixture(t, releasedOnStop("i-launch-fail"))
|
||||
|
||||
require.Error(t, f.start(t, "i-launch-fail"))
|
||||
|
||||
assert.Equal(t, []string{"203.0.113.20"}, f.rel.released,
|
||||
"the address the start took was not returned when the launch failed")
|
||||
restored := f.store.WroteStopped["i-launch-fail"]
|
||||
require.NotNil(t, restored)
|
||||
assert.Empty(t, restored.PublicIP, "the restored record holds an address the pool has back")
|
||||
assert.True(t, restored.AutoAssignPublicIP, "the next start would bring the instance up with no address")
|
||||
}
|
||||
|
||||
// An adopted Elastic IP is not the start's to give back, so a failed launch
|
||||
// must leave it exactly where it is.
|
||||
func TestAFailedLaunchDoesNotReturnAnAdoptedElasticIP(t *testing.T) {
|
||||
f := newStartFixture(t, releasedOnStop("i-eip-fail"))
|
||||
f.eni.eniHasEIP = true
|
||||
f.eni.getENIByID["eni-1"].PublicIpAddress = "198.51.100.7"
|
||||
f.eni.getENIByID["eni-1"].PublicIpPool = "wan"
|
||||
|
||||
require.Error(t, f.start(t, "i-eip-fail"))
|
||||
|
||||
assert.Empty(t, f.rel.released,
|
||||
"a failed launch gave a customer's Elastic IP back to the pool")
|
||||
}
|
||||
@@ -1122,6 +1122,14 @@ func (s *VPCServiceImpl) validateSGAttachment(ctx context.Context, accountID str
|
||||
return nil
|
||||
}
|
||||
|
||||
// ENIHasEIP reports whether an Elastic IP is associated with eniID. It is the
|
||||
// one question three different moments in an instance's life have to ask: an
|
||||
// ENI delete may not return an EIP-backed address to the pool, a stop may not
|
||||
// reclaim one, and a start must not auto-assign a second address beside one.
|
||||
func (s *VPCServiceImpl) ENIHasEIP(ctx context.Context, accountID, eniID string) (bool, error) {
|
||||
return s.isEIPOwned(ctx, eniID, accountID)
|
||||
}
|
||||
|
||||
// isEIPOwned checks whether the given ENI's public IP is owned by an Elastic IP.
|
||||
// Returns (true, nil) if an EIP record references this ENI, (false, nil) if none
|
||||
// match, or (false, err) if the KV store could not be read.
|
||||
|
||||
@@ -27,17 +27,19 @@ const (
|
||||
)
|
||||
|
||||
// ExternalIPAM is the AWS-facing entry point for external IP allocation,
|
||||
// dispatching to StaticPoolAllocator or dhcp.DHCPPoolAllocator per pool name.
|
||||
// dispatching per pool name to the static allocator or to whatever the wiring
|
||||
// installed for that pool.
|
||||
type ExternalIPAM struct {
|
||||
kv jetstream.KeyValue
|
||||
pools []external.ExternalPoolConfig
|
||||
static *external.StaticPoolAllocator
|
||||
perPool map[string]external.Allocator // dhcp overrides; static pools fall through to `static`
|
||||
perPool map[string]external.Allocator // installed overrides; static pools fall through to `static`
|
||||
}
|
||||
|
||||
// NewExternalIPAM creates a new ExternalIPAM. Static pools wire through
|
||||
// external.StaticPoolAllocator; DHCP-sourced pools wait for EnableDHCP
|
||||
// to install the per-pool dhcp.DHCPPoolAllocator.
|
||||
// external.StaticPoolAllocator; every other source waits for the wiring to
|
||||
// call InstallAllocator, because their allocators need dependencies this
|
||||
// package must not construct.
|
||||
func NewExternalIPAM(ctx context.Context, js jetstream.JetStream, pools []external.ExternalPoolConfig) (*ExternalIPAM, error) {
|
||||
staticPools := filterStatic(pools)
|
||||
var (
|
||||
@@ -63,6 +65,40 @@ func NewExternalIPAMWithKV(kv jetstream.KeyValue, pools []external.ExternalPoolC
|
||||
return &ExternalIPAM{kv: kv, pools: pools, static: alloc, perPool: map[string]external.Allocator{}}
|
||||
}
|
||||
|
||||
// InstallAllocator routes one pool to a non-static allocator. Every source
|
||||
// other than static needs a dependency this package must not construct — a
|
||||
// NATS client for DHCP, an authenticated API client for OCI — so the wiring
|
||||
// hands the built allocator in rather than naming the source here.
|
||||
// Idempotent: a repeated call replaces the entry.
|
||||
func (m *ExternalIPAM) InstallAllocator(poolName string, a external.Allocator) error {
|
||||
if m == nil {
|
||||
return errors.New("ExternalIPAM InstallAllocator: nil IPAM")
|
||||
}
|
||||
if poolName == "" {
|
||||
return errors.New("ExternalIPAM InstallAllocator: empty pool name")
|
||||
}
|
||||
if a == nil {
|
||||
return fmt.Errorf("ExternalIPAM InstallAllocator: nil allocator for pool %q", poolName)
|
||||
}
|
||||
m.perPool[poolName] = a
|
||||
return nil
|
||||
}
|
||||
|
||||
// PoolsWithSource returns the configured pools using the given source, so the
|
||||
// wiring can build one allocator per pool without reaching into m.pools.
|
||||
func (m *ExternalIPAM) PoolsWithSource(source string) []external.ExternalPoolConfig {
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
var out []external.ExternalPoolConfig
|
||||
for _, p := range m.pools {
|
||||
if p.Source == source {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// EnableDHCP installs a DHCPPoolAllocator for every pool with
|
||||
// Source="dhcp". client is the daemon-side NATS wrapper that fans out
|
||||
// to vpcd. Idempotent — repeated calls overwrite existing dhcp entries.
|
||||
@@ -70,11 +106,10 @@ func (m *ExternalIPAM) EnableDHCP(client *dhcp.NATSClient) error {
|
||||
if client == nil {
|
||||
return errors.New("ExternalIPAM EnableDHCP: nil NATSClient")
|
||||
}
|
||||
for _, p := range m.pools {
|
||||
if p.Source != external.SourceDHCP {
|
||||
continue
|
||||
for _, p := range m.PoolsWithSource(external.SourceDHCP) {
|
||||
if err := m.InstallAllocator(p.Name, dhcp.NewDHCPPoolAllocator(client, p)); err != nil {
|
||||
return err
|
||||
}
|
||||
m.perPool[p.Name] = dhcp.NewDHCPPoolAllocator(client, p)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -136,6 +171,33 @@ func (m *ExternalIPAM) ReleaseIP(ctx context.Context, poolName, ip, ownerENIID s
|
||||
return alloc.Release(ctx, poolName, addr, ownerENIID)
|
||||
}
|
||||
|
||||
// BindOwner tells a pool which ENI an already-allocated address is attached to,
|
||||
// for the allocators that need it. An EIP is allocated bare and associated
|
||||
// later, so this is the only point at which an OCI pool can learn where the
|
||||
// address has to be delivered. An empty eniID records the disassociate.
|
||||
//
|
||||
// A pool whose allocator does not track owners is a success with nothing to do:
|
||||
// for a static or DHCP address the association lives in the EIP record alone,
|
||||
// and failing here would break EIPs on every non-OCI pool.
|
||||
func (m *ExternalIPAM) BindOwner(ctx context.Context, poolName, ip, eniID string) error {
|
||||
if m == nil {
|
||||
return errors.New("external IPAM unavailable on this node")
|
||||
}
|
||||
addr, err := netip.ParseAddr(ip)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse owner bind IP %q: %w", ip, err)
|
||||
}
|
||||
alloc, err := m.allocatorFor(poolName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tracker, ok := alloc.(external.OwnerTracker)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return tracker.BindOwner(ctx, poolName, addr, eniID)
|
||||
}
|
||||
|
||||
// GetPoolRecord returns the current IPAM record for a pool. DHCP-sourced
|
||||
// pools have no static record — the per-AZ lease bucket is authoritative.
|
||||
func (m *ExternalIPAM) GetPoolRecord(poolName string) (*ExternalIPAMRecord, error) {
|
||||
@@ -154,7 +216,7 @@ func (m *ExternalIPAM) allocatorFor(poolName string) (external.Allocator, error)
|
||||
return a, nil
|
||||
}
|
||||
if m.static == nil {
|
||||
return nil, fmt.Errorf("no allocator for pool %q (static disabled, dhcp not enabled)", poolName)
|
||||
return nil, fmt.Errorf("no allocator for pool %q (static disabled and no allocator installed for it)", poolName)
|
||||
}
|
||||
return m.static, nil
|
||||
}
|
||||
|
||||
@@ -8,6 +8,36 @@ const MetaDataServerIP = "169.254.169.254"
|
||||
// the same demux flows). Guests receive it as their DHCP nameserver.
|
||||
const VPCDNSServerIP = "169.254.169.253"
|
||||
|
||||
// HostBindAddrs are the host-side addresses a per-tap responder binds. They
|
||||
// are the guest-facing addresses everywhere the host has no use for them
|
||||
// itself, which is every bare-metal deployment.
|
||||
//
|
||||
// A cloud host is the exception: 169.254.169.254 is its own metadata service
|
||||
// and 169.254.169.253 its own resolver, and a /32 on an ime- endpoint shadows
|
||||
// both — so the node loses DNS and its cloud API the moment a guest starts.
|
||||
// Binding elsewhere and DNAT'ing the guest's packets keeps the guest's view
|
||||
// at 169.254.169.254 while leaving the host's routes alone.
|
||||
type HostBindAddrs struct {
|
||||
Meta string
|
||||
DNS string
|
||||
}
|
||||
|
||||
// NewHostBindAddrs takes the configured pair, falling back to the guest-facing
|
||||
// addresses when either is unset — half a remap would bind one address the
|
||||
// guest can reach and one the host has already taken.
|
||||
func NewHostBindAddrs(meta, dns string) HostBindAddrs {
|
||||
if meta == "" || dns == "" {
|
||||
return HostBindAddrs{Meta: MetaDataServerIP, DNS: VPCDNSServerIP}
|
||||
}
|
||||
return HostBindAddrs{Meta: meta, DNS: dns}
|
||||
}
|
||||
|
||||
// Remapped reports whether the host serves these on addresses other than the
|
||||
// ones the guest addresses, and so needs the DNAT that restores the fiction.
|
||||
func (a HostBindAddrs) Remapped() bool {
|
||||
return a.Meta != MetaDataServerIP || a.DNS != VPCDNSServerIP
|
||||
}
|
||||
|
||||
// pinnedVersion is the dated IMDS API version advertised by GET /.
|
||||
const pinnedVersion = "2021-07-15"
|
||||
|
||||
|
||||
@@ -305,23 +305,25 @@ func (tb *tokenBucket) allow() bool {
|
||||
return tb.limiter.AllowN(tb.now(), 1)
|
||||
}
|
||||
|
||||
// bindTapDNS opens the DNS shim sockets — UDP and TCP 169.254.169.253:53 on the
|
||||
// tap's endpoint via SO_BINDTODEVICE. Like the .254 HTTP bind, no netns is
|
||||
// involved: the endpoint owns .253, and the per-tap demux flow already steers
|
||||
// guest queries here, so binding it lights up the reserved co-tenant slot.
|
||||
func bindTapDNS(ctx context.Context, endpoint string) (net.PacketConn, net.Listener, error) {
|
||||
lc := net.ListenConfig{Control: bindToDeviceControl(endpoint)}
|
||||
addr := net.JoinHostPort(VPCDNSServerIP, "53")
|
||||
pc, err := lc.ListenPacket(ctx, "udp4", addr)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
// tapDNSBinder opens the DNS shim sockets — UDP and TCP bindIP:53 on the tap's
|
||||
// endpoint via SO_BINDTODEVICE. Like the HTTP bind, no netns is involved: the
|
||||
// endpoint owns the address, and the per-tap demux flow already steers guest
|
||||
// queries here, so binding it lights up the reserved co-tenant slot.
|
||||
func tapDNSBinder(bindIP string) dnsListenFunc {
|
||||
return func(ctx context.Context, endpoint string) (net.PacketConn, net.Listener, error) {
|
||||
lc := net.ListenConfig{Control: bindToDeviceControl(endpoint)}
|
||||
addr := net.JoinHostPort(bindIP, "53")
|
||||
pc, err := lc.ListenPacket(ctx, "udp4", addr)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
ln, err := lc.Listen(ctx, "tcp4", addr)
|
||||
if err != nil {
|
||||
_ = pc.Close()
|
||||
return nil, nil, err
|
||||
}
|
||||
return pc, ln, nil
|
||||
}
|
||||
ln, err := lc.Listen(ctx, "tcp4", addr)
|
||||
if err != nil {
|
||||
_ = pc.Close()
|
||||
return nil, nil, err
|
||||
}
|
||||
return pc, ln, nil
|
||||
}
|
||||
|
||||
// servfail rewrites a query header in place into a minimal SERVFAIL response, so
|
||||
|
||||
@@ -95,7 +95,10 @@ type IMDSServiceImpl struct {
|
||||
// nil, degrading that source to a miss rather than failing IMDS to start —
|
||||
// the caller (vpcd) owns their construction and availability.
|
||||
// ctx bounds the bucket opens only; each served request carries its own.
|
||||
func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer, iamSvc profileLookup, pubKeys publicKeyLookup, localState localStateReader, records recordLoader, listTaps listTapsFunc, baseDomain, internalDomain, servicesDomain, caCertPath string, resolverIPs []string) (*IMDSServiceImpl, error) {
|
||||
func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer, iamSvc profileLookup, pubKeys publicKeyLookup, localState localStateReader, records recordLoader, listTaps listTapsFunc, baseDomain, internalDomain, servicesDomain, caCertPath string, resolverIPs []string, bind HostBindAddrs) (*IMDSServiceImpl, error) {
|
||||
if bind.Meta == "" || bind.DNS == "" {
|
||||
return nil, errors.New("IMDS host bind addresses required (use NewHostBindAddrs)")
|
||||
}
|
||||
if natsConn == nil {
|
||||
return nil, errors.New("nil NATS connection")
|
||||
}
|
||||
@@ -173,14 +176,19 @@ func NewIMDSServiceImpl(ctx context.Context, natsConn *nats.Conn, sts stsAssumer
|
||||
}
|
||||
// Each per-tap responder serves the shared mux, threading its tap's ENI
|
||||
// identity into every request via BaseContext.
|
||||
svc.tapResp = newTapResponderManager(svc.httpHandler(), svc.resolver.resolveENIByID, bindTapListener)
|
||||
svc.tapResp = newTapResponderManager(svc.httpHandler(), svc.resolver.resolveENIByID, tapListenerBinder(bind.Meta))
|
||||
if len(resolverIPs) > 0 {
|
||||
targets := make([]string, 0, len(resolverIPs))
|
||||
for _, ip := range resolverIPs {
|
||||
targets = append(targets, net.JoinHostPort(ip, northstarResolverPort))
|
||||
}
|
||||
svc.tapResp.enableDNS(bindTapDNS, newDNSForwarder(targets))
|
||||
slog.Info("IMDS: VPC DNS shim enabled", "addr", VPCDNSServerIP, "backends", targets)
|
||||
svc.tapResp.enableDNS(tapDNSBinder(bind.DNS), newDNSForwarder(targets))
|
||||
slog.Info("IMDS: VPC DNS shim enabled", "addr", bind.DNS, "backends", targets)
|
||||
}
|
||||
if bind.Remapped() {
|
||||
slog.Info("IMDS: serving on remapped host addresses; guests still address the standard ones",
|
||||
"meta_bind", bind.Meta, "dns_bind", bind.DNS,
|
||||
"guest_meta", MetaDataServerIP, "guest_dns", VPCDNSServerIP)
|
||||
}
|
||||
|
||||
slog.Info("IMDS service initialized", "eni_bucket", kvBucketENIs)
|
||||
|
||||
@@ -328,12 +328,15 @@ func deviceIfindex(dev string) (int, error) {
|
||||
return iface.Index, nil
|
||||
}
|
||||
|
||||
// bindTapListener opens 169.254.169.254:80 on the tap's endpoint via SO_BINDTODEVICE
|
||||
// — the per-tap serving socket. No netns/CAP_SYS_ADMIN: the endpoint is in the root
|
||||
// netns on br-imds and owns .254, so the bind targets the address the guest does.
|
||||
func bindTapListener(ctx context.Context, endpoint string) (net.Listener, error) {
|
||||
lc := net.ListenConfig{Control: bindToDeviceControl(endpoint)}
|
||||
return lc.Listen(ctx, "tcp4", net.JoinHostPort(MetaDataServerIP, "80"))
|
||||
// tapListenerBinder opens bindIP:80 on the tap's endpoint via SO_BINDTODEVICE
|
||||
// — the per-tap serving socket. No netns/CAP_SYS_ADMIN: the endpoint is in the
|
||||
// root netns on br-imds and owns the address, so the bind targets what the
|
||||
// guest addresses, or what the host's DNAT rewrites it to.
|
||||
func tapListenerBinder(bindIP string) tapListenFunc {
|
||||
return func(ctx context.Context, endpoint string) (net.Listener, error) {
|
||||
lc := net.ListenConfig{Control: bindToDeviceControl(endpoint)}
|
||||
return lc.Listen(ctx, "tcp4", net.JoinHostPort(bindIP, "80"))
|
||||
}
|
||||
}
|
||||
|
||||
// bindToDeviceControl returns a ListenConfig.Control that scopes the socket to
|
||||
|
||||
+14
@@ -16,6 +16,20 @@ type AllocateRequest struct {
|
||||
InstanceID string
|
||||
}
|
||||
|
||||
// OwnerTracker is an Allocator that also needs to hear about associations made
|
||||
// after the address was allocated. An EIP is allocated with no ENI and attached
|
||||
// later, so an allocator whose behaviour depends on the owner cannot learn it
|
||||
// from Allocate alone.
|
||||
//
|
||||
// Only OCI needs this: the address is delivered to one VNIC, so the owning ENI
|
||||
// is what decides which node that must be. A static or DHCP address is on the
|
||||
// wire everywhere its node can reach, and those allocators do not implement it.
|
||||
type OwnerTracker interface {
|
||||
// BindOwner records eniID as the current owner of ip in poolName. An empty
|
||||
// eniID clears the owner, which is what a disassociate leaves behind.
|
||||
BindOwner(ctx context.Context, poolName string, ip netip.Addr, eniID string) error
|
||||
}
|
||||
|
||||
// Allocator hands out a single external IP per AWS identity from a named
|
||||
// pool. Implementations: StaticPoolAllocator (range math + KV CAS) and
|
||||
// DHCPPoolAllocator (RFC 2131 DORA via vpcd, Q4).
|
||||
|
||||
+10
@@ -598,6 +598,16 @@ func (m *Manager) handleReleaseMsg(msg *nats.Msg) {
|
||||
entry, err := m.store.LookupByIP(ctx, req.PoolName, req.IP)
|
||||
switch {
|
||||
case err == nil:
|
||||
// An EIP's lease is the only thing holding the address — a DHCP
|
||||
// server promises nothing about handing it back — so a release made
|
||||
// for a departing interface must leave it renewing. Only the
|
||||
// address's own owner, which names no interface, may free it.
|
||||
if req.OwnerScoped && entry.Purpose == PurposeEIP {
|
||||
slog.Info("dhcp manager: release skip — address belongs to an Elastic IP allocation",
|
||||
"pool", req.PoolName, "ip", req.IP, "client_id", entry.Lease.ClientID)
|
||||
_ = msg.Respond(emptyReleaseReply)
|
||||
return
|
||||
}
|
||||
clientID = entry.Lease.ClientID
|
||||
case errors.Is(err, jetstream.ErrKeyNotFound):
|
||||
// Answering SUCCESS here told the caller the address was freed when
|
||||
|
||||
+4
-2
@@ -101,11 +101,13 @@ func (c *NATSClient) RequestRelease(ctx context.Context, clientID string) error
|
||||
// RequestReleaseByIP releases the lease that holds (poolName, ip).
|
||||
// Used by DHCPPoolAllocator.Release, whose external.Allocator signature
|
||||
// only carries the IP — vpcd looks up the client-id from the lease store.
|
||||
func (c *NATSClient) RequestReleaseByIP(ctx context.Context, poolName, ip string) error {
|
||||
// ownerScoped marks a release made for a departing interface rather than by the
|
||||
// address's owner, which vpcd refuses for an EIP.
|
||||
func (c *NATSClient) RequestReleaseByIP(ctx context.Context, poolName, ip string, ownerScoped bool) error {
|
||||
if ip == "" {
|
||||
return errors.New("dhcp NATSClient: ip required")
|
||||
}
|
||||
return c.requestRelease(ctx, releaseWireRequest{PoolName: poolName, IP: ip})
|
||||
return c.requestRelease(ctx, releaseWireRequest{PoolName: poolName, IP: ip, OwnerScoped: ownerScoped})
|
||||
}
|
||||
|
||||
func (c *NATSClient) requestRelease(ctx context.Context, req releaseWireRequest) error {
|
||||
|
||||
+6
-5
@@ -83,10 +83,11 @@ func (a *DHCPPoolAllocator) Allocate(ctx context.Context, req external.AllocateR
|
||||
}
|
||||
|
||||
// Release issues vpc.dhcp.release for ip. Errors if poolName doesn't match
|
||||
// this allocator's pool to prevent cross-pool release. ownerENIID is unused:
|
||||
// DHCP leases are keyed by client-ID, not ENI, so ownership scoping does not
|
||||
// apply.
|
||||
func (a *DHCPPoolAllocator) Release(ctx context.Context, poolName string, ip netip.Addr, _ string) error {
|
||||
// this allocator's pool to prevent cross-pool release. ownerENIID is not a
|
||||
// lease key — leases are keyed by client-ID — but its presence still carries
|
||||
// meaning: it marks a release made for a departing interface, which vpcd
|
||||
// refuses for an EIP.
|
||||
func (a *DHCPPoolAllocator) Release(ctx context.Context, poolName string, ip netip.Addr, ownerENIID string) error {
|
||||
if a == nil || a.client == nil {
|
||||
return errors.New("dhcp pool allocator: nil client")
|
||||
}
|
||||
@@ -96,7 +97,7 @@ func (a *DHCPPoolAllocator) Release(ctx context.Context, poolName string, ip net
|
||||
if !ip.IsValid() {
|
||||
return errors.New("dhcp pool allocator: invalid ip")
|
||||
}
|
||||
return a.client.RequestReleaseByIP(ctx, a.pool.Name, ip.String())
|
||||
return a.client.RequestReleaseByIP(ctx, a.pool.Name, ip.String(), ownerENIID != "")
|
||||
}
|
||||
|
||||
func (a *DHCPPoolAllocator) hwAddrFor(clientID string) net.HardwareAddr {
|
||||
|
||||
+1
-1
@@ -141,7 +141,7 @@ func TestDHCPPoolAllocatorReleaseUncodedErrorIsNotTerminal(t *testing.T) {
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
|
||||
client := dhcp.NewNATSClient(nc, 3*time.Second)
|
||||
err = client.RequestReleaseByIP(context.Background(), "wan", "198.51.100.99")
|
||||
err = client.RequestReleaseByIP(context.Background(), "wan", "198.51.100.99", false)
|
||||
require.Error(t, err)
|
||||
assert.NotErrorIs(t, err, dhcp.ErrLeaseNotTracked)
|
||||
assert.Contains(t, err.Error(), "upstream unreachable")
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
package dhcp_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// seedLeaseFor puts a live lease in the store under purpose and returns the
|
||||
// address it holds, so a release can be aimed at it by IP the way vpcd is.
|
||||
func seedLeaseFor(t *testing.T, fake *dhcp.Fake, store *dhcp.Store, purpose, clientID, ip string) string {
|
||||
t.Helper()
|
||||
hw, err := net.ParseMAC("02:00:00:aa:bb:cc")
|
||||
require.NoError(t, err)
|
||||
lease, err := fake.Acquire(context.Background(), dhcp.AcquireRequest{
|
||||
Bridge: "br-wan", ClientID: clientID, HWAddr: hw,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
lease.IP = net.ParseIP(ip).To4()
|
||||
lease.LeaseDuration = time.Hour
|
||||
require.NoError(t, store.Put(t.Context(), dhcp.Entry{
|
||||
Purpose: purpose, PoolName: "wan", Lease: lease,
|
||||
}))
|
||||
return ip
|
||||
}
|
||||
|
||||
// Instance teardown releases by address, and after an associate that address is
|
||||
// the EIP. On a DHCP pool the lease is the only thing holding it — the server
|
||||
// promises nothing about handing it back — so this release has to be refused.
|
||||
func TestAnOwnerScopedReleaseLeavesAnEIPLeaseAlone(t *testing.T) {
|
||||
fake := dhcp.NewFake()
|
||||
mgr, store, nc := newTestManager(t, "az1", fake, time.Now)
|
||||
ip := seedLeaseFor(t, fake, store, dhcp.PurposeEIP, "eipalloc-1", "192.0.2.50")
|
||||
|
||||
_, err := mgr.Subscribe(nc)
|
||||
require.NoError(t, err)
|
||||
client := dhcp.NewNATSClient(nc, 3*time.Second)
|
||||
|
||||
require.NoError(t, client.RequestReleaseByIP(t.Context(), "wan", ip, true))
|
||||
|
||||
assert.Zero(t, fake.ReleaseCount(),
|
||||
"terminating an instance sent DHCPRELEASE for a customer's Elastic IP")
|
||||
_, err = store.Get(t.Context(), "eipalloc-1")
|
||||
assert.NoError(t, err, "the EIP's lease was deleted, so nothing is renewing it")
|
||||
}
|
||||
|
||||
// ReleaseAddress is the address's own owner and names no interface, so it must
|
||||
// still be able to give the EIP up.
|
||||
func TestAnUnscopedReleaseStillFreesAnEIPLease(t *testing.T) {
|
||||
fake := dhcp.NewFake()
|
||||
mgr, store, nc := newTestManager(t, "az1", fake, time.Now)
|
||||
ip := seedLeaseFor(t, fake, store, dhcp.PurposeEIP, "eipalloc-1", "192.0.2.50")
|
||||
|
||||
_, err := mgr.Subscribe(nc)
|
||||
require.NoError(t, err)
|
||||
client := dhcp.NewNATSClient(nc, 3*time.Second)
|
||||
|
||||
require.NoError(t, client.RequestReleaseByIP(t.Context(), "wan", ip, false))
|
||||
|
||||
assert.Equal(t, 1, fake.ReleaseCount(), "ReleaseAddress could not free its own address")
|
||||
}
|
||||
|
||||
// The refusal must not reach the auto-assigned address, which is exactly what
|
||||
// instance teardown is there to reclaim.
|
||||
func TestAnOwnerScopedReleaseStillFreesAnAutoAssignedLease(t *testing.T) {
|
||||
fake := dhcp.NewFake()
|
||||
mgr, store, nc := newTestManager(t, "az1", fake, time.Now)
|
||||
ip := seedLeaseFor(t, fake, store, dhcp.PurposeENIPublic, "eni-1", "192.0.2.51")
|
||||
|
||||
_, err := mgr.Subscribe(nc)
|
||||
require.NoError(t, err)
|
||||
client := dhcp.NewNATSClient(nc, 3*time.Second)
|
||||
|
||||
require.NoError(t, client.RequestReleaseByIP(t.Context(), "wan", ip, true))
|
||||
|
||||
assert.Equal(t, 1, fake.ReleaseCount(), "an auto-assigned lease leaked on teardown")
|
||||
_, err = store.Get(t.Context(), "eni-1")
|
||||
assert.Error(t, err, "the lease record outlived the address")
|
||||
}
|
||||
+6
@@ -111,6 +111,12 @@ type releaseWireRequest struct {
|
||||
ClientID string `json:"client_id,omitempty"`
|
||||
PoolName string `json:"pool_name,omitempty"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
// OwnerScoped marks a release made on behalf of an interface that is going
|
||||
// away, rather than one made by the address's own owner. Such a release must
|
||||
// never free an EIP: the lease is the only thing holding the address, and a
|
||||
// DHCP server gives no guarantee of handing it back. Absent on an older
|
||||
// caller, which reads as false and keeps the previous behaviour.
|
||||
OwnerScoped bool `json:"owner_scoped,omitempty"`
|
||||
}
|
||||
|
||||
// Code is machine-readable and lets callers branch on a condition without
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
//test:in-package — reads PoolRecord.Allocated and the unexported wanPool /
|
||||
// newStaticAllocator fixtures from static_pool_test.go to assert on the stored
|
||||
// allocation rather than on a return value.
|
||||
|
||||
package external
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// An EIP is allocated with no ENI and associated later, so the ownership guard
|
||||
// — which only fires when the stored allocation already names an ENI — cannot
|
||||
// protect it. The purpose has to.
|
||||
func TestAnOwnerScopedReleaseWillNotFreeAnEIP(t *testing.T) {
|
||||
a := newStaticAllocator(t, []ExternalPoolConfig{wanPool()})
|
||||
ctx := t.Context()
|
||||
|
||||
eip, err := a.Allocate(ctx, AllocateRequest{
|
||||
PoolName: "wan", Purpose: "eip", AllocationID: "eipalloc-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
// What instance teardown does: release the address the instance is holding,
|
||||
// scoped to the ENI it is holding it on. After an associate, that address is
|
||||
// the EIP.
|
||||
require.NoError(t, a.Release(ctx, "wan", eip, "eni-1"))
|
||||
|
||||
rec, err := a.GetPoolRecord(ctx, "wan")
|
||||
require.NoError(t, err)
|
||||
alloc, stillHeld := rec.Allocated[eip.String()]
|
||||
require.True(t, stillHeld,
|
||||
"terminating an instance gave a customer's Elastic IP back to the pool")
|
||||
assert.Equal(t, "eipalloc-1", alloc.AllocationID)
|
||||
}
|
||||
|
||||
// ReleaseAddress is the only caller allowed to free an EIP, and it names no
|
||||
// owner precisely because there is no interface to scope to.
|
||||
func TestAnUnscopedReleaseStillFreesAnEIP(t *testing.T) {
|
||||
a := newStaticAllocator(t, []ExternalPoolConfig{wanPool()})
|
||||
ctx := t.Context()
|
||||
|
||||
eip, err := a.Allocate(ctx, AllocateRequest{
|
||||
PoolName: "wan", Purpose: "eip", AllocationID: "eipalloc-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, a.Release(ctx, "wan", eip, ""))
|
||||
|
||||
rec, err := a.GetPoolRecord(ctx, "wan")
|
||||
require.NoError(t, err)
|
||||
_, stillHeld := rec.Allocated[eip.String()]
|
||||
assert.False(t, stillHeld, "ReleaseAddress could not free the address it owns")
|
||||
}
|
||||
|
||||
// The guard must not reach the auto-assigned address, which is exactly what
|
||||
// instance teardown is supposed to reclaim.
|
||||
func TestAnOwnerScopedReleaseStillFreesAnAutoAssignedAddress(t *testing.T) {
|
||||
a := newStaticAllocator(t, []ExternalPoolConfig{wanPool()})
|
||||
ctx := t.Context()
|
||||
|
||||
ip, err := a.Allocate(ctx, AllocateRequest{
|
||||
PoolName: "wan", Purpose: "eni-public", ENIID: "eni-1", InstanceID: "i-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, a.Release(ctx, "wan", ip, "eni-1"))
|
||||
|
||||
rec, err := a.GetPoolRecord(ctx, "wan")
|
||||
require.NoError(t, err)
|
||||
_, stillHeld := rec.Allocated[ip.String()]
|
||||
assert.False(t, stillHeld, "an auto-assigned address leaked on teardown")
|
||||
}
|
||||
+183
@@ -0,0 +1,183 @@
|
||||
package ocinet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/topology"
|
||||
)
|
||||
|
||||
// ClaimResult reports what an affinity pass moved, so a caller can log it and a
|
||||
// test can assert on it without reading the fake's internals.
|
||||
type ClaimResult struct {
|
||||
// Claimed is the public addresses whose private half this node took over.
|
||||
Claimed []string
|
||||
// Local is how many bindings this node already owned, moved or not.
|
||||
Local int
|
||||
}
|
||||
|
||||
// ClaimLocalAddresses makes OCI agree with where the guests actually are.
|
||||
//
|
||||
// An OCI address is delivered to whichever VNIC carries its private half, and
|
||||
// nothing about allocating one consults the node the instance will run on: the
|
||||
// AWS request is served by an arbitrary queue-group worker, so the private IP
|
||||
// lands on that worker's VNIC. It is then never moved. A guest that starts on a
|
||||
// different node than it stopped on — which stop/start does routinely, since the
|
||||
// start is only *preferentially* routed back to LastNode — keeps a correct OVN
|
||||
// rule and a correct host route on its new node while OCI keeps delivering to
|
||||
// the old one. The address goes dark and stays dark.
|
||||
//
|
||||
// The fix is one call: OCI reassigns a secondary private IP between VNICs in
|
||||
// the same subnet server-side, keeping the OCID, so the reserved public IP
|
||||
// attached to it follows without being touched and the customer's address never
|
||||
// changes. There is no window in which the address belongs to neither node, and
|
||||
// so nothing for the losing node to be told — it drops its now-unwanted host
|
||||
// route on its own next pass, exactly as it would for a released address.
|
||||
//
|
||||
// Ownership is decided by local OVS, not by the record: a tap plugged in here
|
||||
// is a guest running here, and it is the same authority the host EIP binder
|
||||
// already uses to decide which addresses this node plumbs. Asking OCI which
|
||||
// private IPs are on our own VNIC costs one call per pass and is authoritative,
|
||||
// so a record that has drifted — an address moved in the console — is repaired
|
||||
// rather than believed.
|
||||
//
|
||||
// A NAT gateway's address has no tap to look for, and the same reasoning gives
|
||||
// the same answer from a different local fact: it is a logical router's SNAT
|
||||
// source, so it belongs on the VPC's gateway chassis, and whether that is this
|
||||
// node is something the chassisredirect port binding says locally.
|
||||
func (a *PoolAllocator) ClaimLocalAddresses(ctx context.Context) (ClaimResult, error) {
|
||||
var res ClaimResult
|
||||
if a.localPorts == nil && a.localGateway == nil {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
var plugged map[string]struct{}
|
||||
if a.localPorts != nil {
|
||||
var err error
|
||||
plugged, err = a.localPorts(ctx)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("ocinet affinity: list local OVS ports: %w", err)
|
||||
}
|
||||
}
|
||||
// No taps and no gateway is a node carrying nothing, which owns no
|
||||
// addresses. It is also what a broken OVS looks like, so there is nothing to
|
||||
// claim either way and the pass costs no API call.
|
||||
if len(plugged) == 0 && a.localGateway == nil {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
rec, err := a.store.Get(ctx, a.cfg.Pool.Name)
|
||||
if err != nil {
|
||||
if errors.Is(err, kvstore.ErrNotFound) {
|
||||
return res, nil
|
||||
}
|
||||
return res, fmt.Errorf("ocinet affinity: read bindings: %w", err)
|
||||
}
|
||||
|
||||
mine := make(map[string]struct{})
|
||||
for key, b := range rec.Bindings {
|
||||
if b.PrivateIPID == "" {
|
||||
continue
|
||||
}
|
||||
local, err := a.ownsBinding(ctx, b, plugged)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
if !local {
|
||||
continue
|
||||
}
|
||||
res.Local++
|
||||
mine[key] = struct{}{}
|
||||
}
|
||||
if len(mine) == 0 {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// One call, not one per binding: this is the authoritative answer to "what
|
||||
// does OCI already deliver to me", and every binding is judged against it.
|
||||
held, err := a.heldPrivateIPIDs(ctx)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
|
||||
for key := range mine {
|
||||
b := rec.Bindings[key]
|
||||
if _, ok := held[b.PrivateIPID]; ok {
|
||||
continue
|
||||
}
|
||||
moved, err := a.client.MovePrivateIP(ctx, b.PrivateIPID, a.cfg.VNICID)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("ocinet affinity: move %s (%s) to %s: %w",
|
||||
b.PrivateAddr, b.PrivateIPID, a.cfg.VNICID, err)
|
||||
}
|
||||
slog.WarnContext(ctx, "ocinet claimed an address whose datapath runs here",
|
||||
"pool", a.cfg.Pool.Name, "public_ip", key, "private_ip", b.PrivateAddr,
|
||||
"eni_id", b.ENIID, "gateway_vpc_id", b.GatewayVPCID,
|
||||
"from_vnic", b.VNICID, "to_vnic", moved.VNICID)
|
||||
|
||||
if err := a.recordVNIC(ctx, key, b.PrivateIPID, moved.VNICID); err != nil {
|
||||
return res, err
|
||||
}
|
||||
res.Claimed = append(res.Claimed, key)
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// ownsBinding answers whether this node is where b's traffic is handled, and so
|
||||
// whose VNIC OCI has to deliver it to.
|
||||
//
|
||||
// A guest address belongs where its tap is plugged. A NAT gateway's belongs on
|
||||
// the VPC's gateway chassis: it is the SNAT source for a logical router, so the
|
||||
// node that forwards is the node OCI must see it leave from. An address that is
|
||||
// neither — a bare EIP nobody has associated yet — has no node, and leaving it
|
||||
// where it was allocated is correct until something claims it.
|
||||
func (a *PoolAllocator) ownsBinding(ctx context.Context, b Binding, plugged map[string]struct{}) (bool, error) {
|
||||
if b.ENIID != "" {
|
||||
_, local := plugged[topology.Port(b.ENIID)]
|
||||
return local, nil
|
||||
}
|
||||
if b.GatewayVPCID == "" || a.localGateway == nil {
|
||||
return false, nil
|
||||
}
|
||||
local, err := a.localGateway(ctx, b.GatewayVPCID)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("ocinet affinity: gateway chassis for %s: %w", b.GatewayVPCID, err)
|
||||
}
|
||||
return local, nil
|
||||
}
|
||||
|
||||
// heldPrivateIPIDs is the set of private IP OCIDs OCI currently carries on this
|
||||
// node's external VNIC.
|
||||
func (a *PoolAllocator) heldPrivateIPIDs(ctx context.Context) (map[string]struct{}, error) {
|
||||
live, err := a.client.ListPrivateIPs(ctx, a.cfg.VNICID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ocinet affinity: list private ips on %s: %w", a.cfg.VNICID, err)
|
||||
}
|
||||
held := make(map[string]struct{}, len(live))
|
||||
for _, p := range live {
|
||||
held[p.ID] = struct{}{}
|
||||
}
|
||||
return held, nil
|
||||
}
|
||||
|
||||
// recordVNIC writes the new owner back under CAS. The move already happened, so
|
||||
// a binding that changed underneath us is re-read rather than overwritten: only
|
||||
// the VNIC is ours to update, and only while it still names the object we moved.
|
||||
func (a *PoolAllocator) recordVNIC(ctx context.Context, key, privateIPID, vnicID string) error {
|
||||
err := a.store.Mutate(ctx, a.cfg.Pool.Name, func(r *Record) (bool, error) {
|
||||
b, ok := r.Bindings[key]
|
||||
if !ok || b.PrivateIPID != privateIPID || b.VNICID == vnicID {
|
||||
return false, nil
|
||||
}
|
||||
b.VNICID = vnicID
|
||||
r.Bindings[key] = b
|
||||
return true, nil
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("ocinet affinity: record new VNIC for %s: %w", key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+315
@@ -0,0 +1,315 @@
|
||||
package ocinet_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/netip"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
|
||||
)
|
||||
|
||||
const (
|
||||
thisVNIC = "ocid1.vnic.oc1..vnic1"
|
||||
otherVNIC = "ocid1.vnic.oc1..vnic2"
|
||||
)
|
||||
|
||||
// newAffinityAllocator builds an allocator whose VNIC is thisVNIC and whose
|
||||
// idea of "running here" is the caller's port set.
|
||||
func newAffinityAllocator(t *testing.T, fake *oci.Fake, local ...string) (*ocinet.PoolAllocator, *ocinet.MemStore) {
|
||||
t.Helper()
|
||||
ports := make(map[string]struct{}, len(local))
|
||||
for _, p := range local {
|
||||
ports[p] = struct{}{}
|
||||
}
|
||||
store := ocinet.NewMemStore()
|
||||
a, err := ocinet.New(fake, store, ocinet.Config{
|
||||
Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI},
|
||||
VNICID: thisVNIC,
|
||||
CompartmentID: "ocid1.compartment.oc1..comp1",
|
||||
Schedule: []time.Duration{time.Millisecond},
|
||||
Budget: time.Second,
|
||||
Sleep: func(context.Context, time.Duration) error { return nil },
|
||||
LocalPorts: func(context.Context) (map[string]struct{}, error) { return ports, nil },
|
||||
})
|
||||
require.NoError(t, err)
|
||||
return a, store
|
||||
}
|
||||
|
||||
// seedForeignBinding puts an allocated pair on otherVNIC and records it, which
|
||||
// is the state after a guest has been started on a node other than the one that
|
||||
// served its AllocateAddress.
|
||||
func seedForeignBinding(t *testing.T, fake *oci.Fake, store *ocinet.MemStore, eniID string) (pubAddr string, privID string) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
priv, err := fake.AssignPrivateIP(ctx, otherVNIC, netip.Addr{}, ocinet.DisplayNamePrefix+eniID)
|
||||
require.NoError(t, err)
|
||||
pub, err := fake.CreatePublicIP(ctx, "ocid1.compartment.oc1..comp1", priv.ID, ocinet.DisplayNamePrefix+eniID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, store.Mutate(ctx, "oci-wan", func(r *ocinet.Record) (bool, error) {
|
||||
r.Bindings[pub.Address.String()] = ocinet.Binding{
|
||||
PublicIPID: pub.ID, PrivateIPID: priv.ID, PrivateAddr: priv.Address.String(),
|
||||
ENIID: eniID, VNICID: otherVNIC,
|
||||
}
|
||||
return true, nil
|
||||
}))
|
||||
return pub.Address.String(), priv.ID
|
||||
}
|
||||
|
||||
// The bug this pass exists for: the guest runs here, OCI delivers elsewhere, so
|
||||
// the address is dark until the private half is moved onto this node's VNIC.
|
||||
func TestClaimMovesAnAddressWhoseGuestRunsHere(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newAffinityAllocator(t, fake, "port-eni-1")
|
||||
pubAddr, privID := seedForeignBinding(t, fake, store, "eni-1")
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{pubAddr}, res.Claimed)
|
||||
|
||||
got, err := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, thisVNIC, got.VNICID, "OCI still delivers this address to the old node")
|
||||
|
||||
// The record must agree, or the startup reconcile reads the address as a
|
||||
// peer's and leaves a real leak of ours alone forever.
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, thisVNIC, rec.Bindings[pubAddr].VNICID)
|
||||
}
|
||||
|
||||
// The reserved public IP is the customer's address. A move that re-created the
|
||||
// private IP would drop it; reassigning keeps the OCID, so the pairing holds.
|
||||
func TestClaimKeepsThePublicAddressAndItsAttachment(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newAffinityAllocator(t, fake, "port-eni-1")
|
||||
pubAddr, privID := seedForeignBinding(t, fake, store, "eni-1")
|
||||
|
||||
_, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
pubs := fake.PublicIPs()
|
||||
require.Len(t, pubs, 1, "the reserved public IP must survive the move")
|
||||
assert.Equal(t, pubAddr, pubs[0].Address.String(), "the customer's address changed")
|
||||
assert.Equal(t, privID, pubs[0].PrivateIPID, "the public IP came detached from its private half")
|
||||
}
|
||||
|
||||
// A node must never take an address whose guest is somewhere else. Getting this
|
||||
// wrong is worse than the bug: it steals a working address from a live guest.
|
||||
func TestClaimLeavesAnAddressWhoseGuestRunsElsewhere(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
// No local ports at all beyond an unrelated one.
|
||||
a, store := newAffinityAllocator(t, fake, "port-eni-somebody-else")
|
||||
_, privID := seedForeignBinding(t, fake, store, "eni-1")
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Claimed)
|
||||
assert.Zero(t, res.Local)
|
||||
|
||||
got, err := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, otherVNIC, got.VNICID, "took an address belonging to another node's guest")
|
||||
}
|
||||
|
||||
// An allocated-but-unassociated EIP has no guest, so no node owns it. Claiming
|
||||
// it would drag every free address onto whichever node swept last.
|
||||
func TestClaimIgnoresAnUnassociatedAddress(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newAffinityAllocator(t, fake, "port-eni-1")
|
||||
_, privID := seedForeignBinding(t, fake, store, "")
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Claimed)
|
||||
|
||||
got, err := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, otherVNIC, got.VNICID)
|
||||
}
|
||||
|
||||
// Steady state is every pass on every node, so it must cost nothing once the
|
||||
// addresses are where they belong.
|
||||
func TestClaimMakesNoMoveCallWhenTheAddressIsAlreadyOurs(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, _ := newAffinityAllocator(t, fake, "port-eni-1")
|
||||
|
||||
_, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", ENIID: "eni-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Any MovePrivateIP at all now is a needless write against a live address.
|
||||
fake.FailOp("MovePrivateIP", errors.New("must not be called"))
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Claimed)
|
||||
assert.Equal(t, 1, res.Local, "the binding is this node's and should be counted")
|
||||
}
|
||||
|
||||
// OVS is the only signal for "is this mine". A pass that guessed when it could
|
||||
// not read it would either strand addresses or steal them.
|
||||
func TestClaimFailsThePassWhenLocalOVSCannotBeRead(t *testing.T) {
|
||||
store := ocinet.NewMemStore()
|
||||
fake := oci.NewFake()
|
||||
a, err := ocinet.New(fake, store, ocinet.Config{
|
||||
Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI},
|
||||
VNICID: thisVNIC,
|
||||
CompartmentID: "ocid1.compartment.oc1..comp1",
|
||||
LocalPorts: func(context.Context) (map[string]struct{}, error) {
|
||||
return nil, errors.New("ovs-vsctl: connection refused")
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = a.ClaimLocalAddresses(context.Background())
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "list local OVS ports")
|
||||
}
|
||||
|
||||
// A single-node cluster and the allocator's own unit tests wire no port source;
|
||||
// that must be a no-op rather than a nil dereference or a claim of everything.
|
||||
func TestClaimIsANoOpWithoutALocalPortSource(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Claimed)
|
||||
assert.Zero(t, res.Local)
|
||||
}
|
||||
|
||||
// The move succeeded against OCI before the record was written, so a failure to
|
||||
// write it back must surface — silently keeping the old VNIC would have the
|
||||
// startup reconcile treat this node's own address as a peer's.
|
||||
func TestClaimReportsAFailureToRecordTheNewOwner(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newAffinityAllocator(t, fake, "port-eni-1")
|
||||
seedForeignBinding(t, fake, store, "eni-1")
|
||||
|
||||
store.FailMutate(errors.New("kv unavailable"))
|
||||
_, err := a.ClaimLocalAddresses(ctx)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "record new VNIC")
|
||||
}
|
||||
|
||||
// newGatewayAffinityAllocator builds an allocator with no guests at all and a
|
||||
// fixed answer to "am I the gateway chassis for this VPC".
|
||||
func newGatewayAffinityAllocator(t *testing.T, fake *oci.Fake, local bool, err error) (*ocinet.PoolAllocator, *ocinet.MemStore) {
|
||||
t.Helper()
|
||||
store := ocinet.NewMemStore()
|
||||
a, aerr := ocinet.New(fake, store, ocinet.Config{
|
||||
Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI},
|
||||
VNICID: thisVNIC,
|
||||
CompartmentID: "ocid1.compartment.oc1..comp1",
|
||||
Schedule: []time.Duration{time.Millisecond},
|
||||
Budget: time.Second,
|
||||
Sleep: func(context.Context, time.Duration) error { return nil },
|
||||
LocalPorts: func(context.Context) (map[string]struct{}, error) { return nil, nil },
|
||||
LocalGateway: func(context.Context, string) (bool, error) { return local, err },
|
||||
})
|
||||
require.NoError(t, aerr)
|
||||
return a, store
|
||||
}
|
||||
|
||||
// seedGatewayBinding puts an allocated pair on otherVNIC with no ENI and a VPC,
|
||||
// which is a NAT gateway's address created by a node that does not forward it.
|
||||
func seedGatewayBinding(t *testing.T, fake *oci.Fake, store *ocinet.MemStore, vpcID string) (pubAddr string, privID string) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
priv, err := fake.AssignPrivateIP(ctx, otherVNIC, netip.Addr{}, ocinet.DisplayNamePrefix+vpcID)
|
||||
require.NoError(t, err)
|
||||
pub, err := fake.CreatePublicIP(ctx, "ocid1.compartment.oc1..comp1", priv.ID, ocinet.DisplayNamePrefix+vpcID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, store.Mutate(ctx, "oci-wan", func(r *ocinet.Record) (bool, error) {
|
||||
r.Bindings[pub.Address.String()] = ocinet.Binding{
|
||||
PublicIPID: pub.ID, PrivateIPID: priv.ID, PrivateAddr: priv.Address.String(),
|
||||
GatewayVPCID: vpcID, VNICID: otherVNIC,
|
||||
}
|
||||
return true, nil
|
||||
}))
|
||||
return pub.Address.String(), priv.ID
|
||||
}
|
||||
|
||||
// The NAT gateway case: no guest, no ENI, and the address still has to be on
|
||||
// the VNIC of the node that forwards for the VPC or OCI drops every packet.
|
||||
func TestClaimMovesANATGatewayAddressToItsGatewayChassis(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newGatewayAffinityAllocator(t, fake, true, nil)
|
||||
pubAddr, privID := seedGatewayBinding(t, fake, store, "vpc-1")
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{pubAddr}, res.Claimed)
|
||||
|
||||
got, err := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, thisVNIC, got.VNICID, "the SNAT source is not on the forwarding node's VNIC")
|
||||
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, thisVNIC, rec.Bindings[pubAddr].VNICID)
|
||||
}
|
||||
|
||||
// Taking a gateway address from the node that forwards for it is the same theft
|
||||
// as taking a guest's, and breaks every private subnet in that VPC.
|
||||
func TestClaimLeavesANATGatewayAddressWhoseChassisIsElsewhere(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newGatewayAffinityAllocator(t, fake, false, nil)
|
||||
_, privID := seedGatewayBinding(t, fake, store, "vpc-1")
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Claimed)
|
||||
|
||||
got, err := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, otherVNIC, got.VNICID)
|
||||
}
|
||||
|
||||
// An unreadable chassis binding is not "not mine": answering no would leave a
|
||||
// live gateway dark, so the pass fails and the next one asks again.
|
||||
func TestClaimFailsWhenTheGatewayChassisCannotBeRead(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newGatewayAffinityAllocator(t, fake, false, errors.New("ovn-sbctl: connection refused"))
|
||||
_, privID := seedGatewayBinding(t, fake, store, "vpc-1")
|
||||
|
||||
_, err := a.ClaimLocalAddresses(ctx)
|
||||
require.Error(t, err)
|
||||
|
||||
got, gerr := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, gerr)
|
||||
assert.Equal(t, otherVNIC, got.VNICID, "a failed pass must not have moved anything")
|
||||
}
|
||||
|
||||
// A bare EIP nobody has associated still has no node, and the gateway rule must
|
||||
// not change that — claiming one would strand it on whichever node ran first.
|
||||
func TestClaimStillLeavesAnUnassociatedEIPAlone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newGatewayAffinityAllocator(t, fake, true, nil)
|
||||
_, privID := seedGatewayBinding(t, fake, store, "")
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Claimed)
|
||||
|
||||
got, gerr := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, gerr)
|
||||
assert.Equal(t, otherVNIC, got.VNICID)
|
||||
}
|
||||
+497
@@ -0,0 +1,497 @@
|
||||
// Package ocinet implements external.Allocator against the Oracle Cloud API.
|
||||
//
|
||||
// On bare metal an external pool is range math: the pool owns 72.52.77.233-254
|
||||
// and handing one out is arithmetic plus a CAS. On OCI it is not. The
|
||||
// hypervisor drops any packet whose source is not a registered private IP
|
||||
// object on the sending VNIC, so an address only exists once OCI has been asked
|
||||
// to create it. That makes allocation a round trip to a cloud API rather than a
|
||||
// local computation, which is the whole reason this package exists alongside
|
||||
// StaticPoolAllocator rather than replacing it.
|
||||
//
|
||||
// One allocation is two OCI objects. A secondary private IP on the VNIC is what
|
||||
// rides the wire; a RESERVED public IP attached to it is what the internet
|
||||
// reaches and what AWS calls the Elastic IP. Allocate returns the public
|
||||
// address, because that is the address AWS semantics are about — the private
|
||||
// half is carried in the binding record, and Lookup is how vpcd finds it when
|
||||
// it programs the OVN NAT rule and the host ingress route.
|
||||
//
|
||||
// The host does not configure the private address on an interface. Owning it
|
||||
// would make the host terminate the guest's traffic — an SSH to the public
|
||||
// address would reach the node, not the instance. It is routed instead: a /32
|
||||
// via the VPC gateway LRP over the transit veth, which is what EnsureEIPIngress
|
||||
// already installs for a routed-mode EIP.
|
||||
package ocinet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"net/netip"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
)
|
||||
|
||||
// DisplayNamePrefix marks every OCI object this package creates. Reconcile only
|
||||
// ever deletes objects carrying it, so an operator's own secondary addresses on
|
||||
// the same VNIC — mulga-poc has two — are never candidates for cleanup. A
|
||||
// prefix rather than a freeform tag because it is visible in the OCI console,
|
||||
// where whoever is wondering what created an address will actually look.
|
||||
const DisplayNamePrefix = "spinifex-"
|
||||
|
||||
// purposeEIP duplicates handlers/ec2/vpc.PurposeEIP so the allocator can tell
|
||||
// an Elastic IP from an auto-assigned address without importing handlers.
|
||||
const purposeEIP = "eip"
|
||||
|
||||
// assignSchedule is the poll ladder for an address to reach ASSIGNED. OCI
|
||||
// assignment is asynchronous: CreatePublicIp returns ASSIGNING and the address
|
||||
// is not reachable until it settles. Shaped like the DHCP manager's DORA ladder
|
||||
// and bounded by the same reasoning — allocation sits inside the AWS client's
|
||||
// read timeout, so the budget has to leave room for the reply.
|
||||
var assignSchedule = []time.Duration{500 * time.Millisecond, time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second}
|
||||
|
||||
// assignBudget caps the wallclock for one Allocate. Sized against the same 60s
|
||||
// botocore read timeout that gives dhcp.defaultAcquireBudget its 45s.
|
||||
const assignBudget = 45 * time.Second
|
||||
|
||||
// assignJitter is the ± window on each poll so concurrent allocations across
|
||||
// nodes do not synchronise onto the same API second.
|
||||
const assignJitter = 200 * time.Millisecond
|
||||
|
||||
// Binding is one allocated external address: the pair of OCI objects behind it
|
||||
// plus the AWS identity that claimed it. Keyed in the record by the public
|
||||
// address, mirroring StaticPoolAllocator.Allocated, so Release is a direct
|
||||
// lookup and the ownership guard reads the same way.
|
||||
type Binding struct {
|
||||
PublicIPID string `json:"public_ip_id"`
|
||||
PrivateIPID string `json:"private_ip_id"`
|
||||
// PrivateAddr is what the host configures on the VNIC and what inbound
|
||||
// packets actually carry — OCI NATs the public address to it before the
|
||||
// instance ever sees the packet, so the DNAT rule matches this, not the
|
||||
// public address.
|
||||
PrivateAddr string `json:"private_addr"`
|
||||
Purpose string `json:"purpose,omitempty"`
|
||||
AllocationID string `json:"allocation_id,omitempty"`
|
||||
ENIID string `json:"eni_id,omitempty"`
|
||||
InstanceID string `json:"instance_id,omitempty"`
|
||||
// GatewayVPCID names the VPC whose gateway chassis must carry this address,
|
||||
// set when a NAT gateway consumes it. Such an address is never attached to
|
||||
// an ENI, so it is the only thing that says which node it belongs on.
|
||||
GatewayVPCID string `json:"gateway_vpc_id,omitempty"`
|
||||
// VNICID is the VNIC currently carrying the private IP, and so the node
|
||||
// the address lives on. A private IP can be reassigned to another VNIC in
|
||||
// the same subnet without losing its public IP, which is how an address
|
||||
// follows an instance to a surviving node — and this field is how a node
|
||||
// that comes back from the dead tells "mine" from "moved on without me"
|
||||
// before it reasserts host plumbing for an address it no longer holds.
|
||||
VNICID string `json:"vnic_id,omitempty"`
|
||||
}
|
||||
|
||||
// Config wires one pool to one VNIC in one compartment.
|
||||
type Config struct {
|
||||
// Pool is the external pool this allocator serves. Source must be "oci".
|
||||
Pool external.ExternalPoolConfig
|
||||
// VNICID is the OCID of the VNIC that carries external traffic. On the
|
||||
// reference topology that is the second VNIC, the one in br-wan.
|
||||
VNICID string
|
||||
// CompartmentID is where reserved public IPs are created.
|
||||
CompartmentID string
|
||||
|
||||
// LocalPorts returns the OVN logical ports with a live tap on this host.
|
||||
// It is how ClaimLocalAddresses knows which addresses belong here, and the
|
||||
// same authority the host EIP binder uses for the same question. Nil
|
||||
// disables the affinity pass, which is correct for a single-node cluster
|
||||
// and for tests that only exercise allocate and release.
|
||||
LocalPorts func(ctx context.Context) (map[string]struct{}, error)
|
||||
|
||||
// LocalGateway reports whether this host is the gateway chassis for a VPC.
|
||||
// It is the same question LocalPorts answers for a guest, asked of a NAT
|
||||
// gateway, which has no tap to look for. Nil leaves gateway addresses where
|
||||
// they were allocated, which is correct on a single node.
|
||||
LocalGateway func(ctx context.Context, vpcID string) (bool, error)
|
||||
|
||||
// Schedule and Budget override the poll ladder; zero values take the
|
||||
// package defaults. Tests set them to keep runtime down.
|
||||
Schedule []time.Duration
|
||||
Budget time.Duration
|
||||
// Sleep is the delay function, so tests do not wait.
|
||||
Sleep func(context.Context, time.Duration) error
|
||||
}
|
||||
|
||||
// Store persists bindings. Implemented over JetStream KV in store.go; the
|
||||
// interface exists so the allocator's tests need no NATS.
|
||||
type Store interface {
|
||||
// Mutate applies fn to the pool's binding set under CAS, retrying on
|
||||
// contention. fn reports whether it changed anything.
|
||||
Mutate(ctx context.Context, poolName string, fn func(*Record) (bool, error)) error
|
||||
// Get reads the current record.
|
||||
Get(ctx context.Context, poolName string) (Record, error)
|
||||
}
|
||||
|
||||
// Record is the per-pool binding set.
|
||||
type Record struct {
|
||||
Bindings map[string]Binding `json:"bindings"`
|
||||
}
|
||||
|
||||
// PoolAllocator implements external.Allocator against OCI.
|
||||
type PoolAllocator struct {
|
||||
client oci.Client
|
||||
store Store
|
||||
cfg Config
|
||||
|
||||
schedule []time.Duration
|
||||
budget time.Duration
|
||||
sleep func(context.Context, time.Duration) error
|
||||
localPorts func(context.Context) (map[string]struct{}, error)
|
||||
localGateway func(context.Context, string) (bool, error)
|
||||
}
|
||||
|
||||
var (
|
||||
_ external.Allocator = (*PoolAllocator)(nil)
|
||||
_ external.OwnerTracker = (*PoolAllocator)(nil)
|
||||
)
|
||||
|
||||
// New constructs an allocator bound to a single pool.
|
||||
func New(client oci.Client, store Store, cfg Config) (*PoolAllocator, error) {
|
||||
if client == nil {
|
||||
return nil, errors.New("ocinet: nil OCI client")
|
||||
}
|
||||
if store == nil {
|
||||
return nil, errors.New("ocinet: nil store")
|
||||
}
|
||||
if cfg.VNICID == "" {
|
||||
return nil, fmt.Errorf("ocinet: pool %q missing vnic_id", cfg.Pool.Name)
|
||||
}
|
||||
if cfg.CompartmentID == "" {
|
||||
return nil, fmt.Errorf("ocinet: pool %q missing compartment_id", cfg.Pool.Name)
|
||||
}
|
||||
a := &PoolAllocator{
|
||||
client: client,
|
||||
store: store,
|
||||
cfg: cfg,
|
||||
schedule: cfg.Schedule,
|
||||
budget: cfg.Budget,
|
||||
sleep: cfg.Sleep,
|
||||
localPorts: cfg.LocalPorts,
|
||||
localGateway: cfg.LocalGateway,
|
||||
}
|
||||
if len(a.schedule) == 0 {
|
||||
a.schedule = assignSchedule
|
||||
}
|
||||
if a.budget == 0 {
|
||||
a.budget = assignBudget
|
||||
}
|
||||
if a.sleep == nil {
|
||||
a.sleep = sleepCtx
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// Pool returns the pool this allocator manages.
|
||||
func (a *PoolAllocator) Pool() external.ExternalPoolConfig { return a.cfg.Pool }
|
||||
|
||||
// Allocate creates a secondary private IP on the VNIC, attaches a reserved
|
||||
// public IP to it, waits for the assignment to settle, and records the pair.
|
||||
//
|
||||
// The OCI objects are created before the record is written, deliberately. The
|
||||
// inverse order would let a crash leave a record naming addresses that do not
|
||||
// exist, which nothing can detect — whereas a crash here leaves real objects
|
||||
// carrying our display-name prefix and no record, which Reconcile finds by
|
||||
// listing the VNIC. A leak that is discoverable beats a record that lies.
|
||||
func (a *PoolAllocator) Allocate(ctx context.Context, req external.AllocateRequest) (netip.Addr, error) {
|
||||
if req.PoolName != "" && req.PoolName != a.cfg.Pool.Name {
|
||||
return netip.Addr{}, fmt.Errorf("ocinet: pool mismatch (got %q, bound to %q)", req.PoolName, a.cfg.Pool.Name)
|
||||
}
|
||||
|
||||
name := displayName(req)
|
||||
priv, err := a.client.AssignPrivateIP(ctx, a.cfg.VNICID, netip.Addr{}, name)
|
||||
if err != nil {
|
||||
return netip.Addr{}, capacityErr(fmt.Errorf("ocinet: assign private ip: %w", err), err)
|
||||
}
|
||||
|
||||
pub, err := a.client.CreatePublicIP(ctx, a.cfg.CompartmentID, priv.ID, name)
|
||||
if err != nil {
|
||||
// Give the private IP back rather than leaving it for Reconcile: we are
|
||||
// still here and we know exactly what we just made.
|
||||
a.undoPrivate(ctx, priv.ID)
|
||||
return netip.Addr{}, capacityErr(fmt.Errorf("ocinet: create public ip: %w", err), err)
|
||||
}
|
||||
|
||||
pub, err = a.waitAssigned(ctx, pub)
|
||||
if err != nil {
|
||||
a.undoPublic(ctx, pub.ID)
|
||||
a.undoPrivate(ctx, priv.ID)
|
||||
return netip.Addr{}, err
|
||||
}
|
||||
if !pub.Address.IsValid() {
|
||||
a.undoPublic(ctx, pub.ID)
|
||||
a.undoPrivate(ctx, priv.ID)
|
||||
return netip.Addr{}, fmt.Errorf("ocinet: public ip %s has no address", pub.ID)
|
||||
}
|
||||
|
||||
key := pub.Address.String()
|
||||
binding := Binding{
|
||||
PublicIPID: pub.ID,
|
||||
PrivateIPID: priv.ID,
|
||||
PrivateAddr: priv.Address.String(),
|
||||
Purpose: req.Purpose,
|
||||
AllocationID: req.AllocationID,
|
||||
ENIID: req.ENIID,
|
||||
InstanceID: req.InstanceID,
|
||||
VNICID: priv.VNICID,
|
||||
}
|
||||
err = a.store.Mutate(ctx, a.cfg.Pool.Name, func(rec *Record) (bool, error) {
|
||||
if rec.Bindings == nil {
|
||||
rec.Bindings = map[string]Binding{}
|
||||
}
|
||||
rec.Bindings[key] = binding
|
||||
return true, nil
|
||||
})
|
||||
if err != nil {
|
||||
a.undoPublic(ctx, pub.ID)
|
||||
a.undoPrivate(ctx, priv.ID)
|
||||
return netip.Addr{}, fmt.Errorf("ocinet: record binding for %s: %w", key, err)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "ocinet allocated external IP",
|
||||
"pool", a.cfg.Pool.Name, "public_ip", key, "private_ip", binding.PrivateAddr,
|
||||
"purpose", req.Purpose, "public_ip_id", pub.ID, "private_ip_id", priv.ID)
|
||||
return pub.Address, nil
|
||||
}
|
||||
|
||||
// Release deletes both OCI objects and drops the binding. ownerENIID scopes the
|
||||
// release exactly as StaticPoolAllocator.Release does: external addresses are
|
||||
// recycled and teardown sweeps re-emit releases, so a release naming an owner
|
||||
// that no longer holds the binding is a no-op rather than a double-free of an
|
||||
// address already handed to another instance.
|
||||
//
|
||||
// The record is dropped first here, the opposite of Allocate, and for the same
|
||||
// reason: whichever half survives a crash should be the discoverable one. A
|
||||
// binding removed with the objects still present is a leak Reconcile collects;
|
||||
// objects removed with the binding still present would be a record that lies.
|
||||
func (a *PoolAllocator) Release(ctx context.Context, poolName string, ip netip.Addr, ownerENIID string) error {
|
||||
if poolName != "" && poolName != a.cfg.Pool.Name {
|
||||
return fmt.Errorf("ocinet: pool mismatch (got %q, bound to %q)", poolName, a.cfg.Pool.Name)
|
||||
}
|
||||
if !ip.IsValid() {
|
||||
return errors.New("ocinet: invalid ip")
|
||||
}
|
||||
key := ip.String()
|
||||
|
||||
var released Binding
|
||||
var found bool
|
||||
err := a.store.Mutate(ctx, a.cfg.Pool.Name, func(rec *Record) (bool, error) {
|
||||
// Reset per attempt: a CAS retry that finds the binding already gone
|
||||
// must not inherit the previous attempt's verdict.
|
||||
released, found = Binding{}, false
|
||||
|
||||
b, ok := rec.Bindings[key]
|
||||
if !ok {
|
||||
if ownerENIID != "" {
|
||||
return false, nil
|
||||
}
|
||||
return false, fmt.Errorf("ocinet: %s not allocated in pool %s", key, a.cfg.Pool.Name)
|
||||
}
|
||||
if ownerENIID != "" && b.ENIID != "" && b.ENIID != ownerENIID {
|
||||
slog.InfoContext(ctx, "ocinet release skip — address reassigned to a different ENI (stale release)",
|
||||
"pool", a.cfg.Pool.Name, "public_ip", key,
|
||||
"stale_owner_eni", ownerENIID, "current_owner_eni", b.ENIID)
|
||||
return false, nil
|
||||
}
|
||||
// Only an unscoped release — ReleaseAddress, which names no interface —
|
||||
// may free an EIP. Here the address is a reserved public IP object, so
|
||||
// freeing one on instance teardown destroys it at the provider and the
|
||||
// customer cannot get it back.
|
||||
// AllocationID is set only for an EIP, so it stands in when a caller
|
||||
// allocated without naming a purpose.
|
||||
if ownerENIID != "" && (b.Purpose == purposeEIP || b.AllocationID != "") {
|
||||
slog.InfoContext(ctx, "ocinet release skip — address belongs to an Elastic IP allocation",
|
||||
"pool", a.cfg.Pool.Name, "public_ip", key,
|
||||
"allocation_id", b.AllocationID, "owner_eni", ownerENIID)
|
||||
return false, nil
|
||||
}
|
||||
delete(rec.Bindings, key)
|
||||
released, found = b, true
|
||||
return true, nil
|
||||
})
|
||||
if err != nil || !found {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := a.deletePair(ctx, released.PublicIPID, released.PrivateIPID); err != nil {
|
||||
return err
|
||||
}
|
||||
slog.InfoContext(ctx, "ocinet released external IP",
|
||||
"pool", a.cfg.Pool.Name, "public_ip", key, "private_ip", released.PrivateAddr)
|
||||
return nil
|
||||
}
|
||||
|
||||
// BindOwner records which ENI an already-allocated address is attached to.
|
||||
//
|
||||
// Allocate learns the owner only when one exists at the time — true for an
|
||||
// auto-assigned address, never for an EIP, which is allocated bare and attached
|
||||
// by a later AssociateAddress. Without this the binding of every EIP names no
|
||||
// ENI, so ClaimLocalAddresses cannot tell which node should hold it and the
|
||||
// address stays wherever the allocating node happened to put it.
|
||||
//
|
||||
// An empty eniID clears the owner, which is the state a disassociate leaves: the
|
||||
// address is still allocated and still billing, but no guest answers on it and
|
||||
// no node should claim it.
|
||||
func (a *PoolAllocator) BindOwner(ctx context.Context, poolName string, ip netip.Addr, eniID string) error {
|
||||
if poolName != "" && poolName != a.cfg.Pool.Name {
|
||||
return fmt.Errorf("ocinet: pool mismatch (got %q, bound to %q)", poolName, a.cfg.Pool.Name)
|
||||
}
|
||||
if !ip.IsValid() {
|
||||
return errors.New("ocinet: invalid ip")
|
||||
}
|
||||
key := ip.String()
|
||||
return a.store.Mutate(ctx, a.cfg.Pool.Name, func(rec *Record) (bool, error) {
|
||||
b, ok := rec.Bindings[key]
|
||||
// An address this pool never allocated is not ours to annotate, and an
|
||||
// unchanged owner is not worth a KV write on every re-association.
|
||||
if !ok || b.ENIID == eniID {
|
||||
return false, nil
|
||||
}
|
||||
b.ENIID = eniID
|
||||
rec.Bindings[key] = b
|
||||
return true, nil
|
||||
})
|
||||
}
|
||||
|
||||
// BindingFor returns the OCI pair behind an allocated public address. The host
|
||||
// plumbing needs the private half: inbound packets arrive carrying it, not the
|
||||
// public address, because OCI has already NATed by the time the instance sees
|
||||
// them.
|
||||
func (a *PoolAllocator) BindingFor(ctx context.Context, ip netip.Addr) (Binding, bool, error) {
|
||||
rec, err := a.store.Get(ctx, a.cfg.Pool.Name)
|
||||
if errors.Is(err, kvstore.ErrNotFound) {
|
||||
return Binding{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return Binding{}, false, err
|
||||
}
|
||||
b, ok := rec.Bindings[ip.String()]
|
||||
return b, ok, nil
|
||||
}
|
||||
|
||||
// waitAssigned polls until the public IP reports ASSIGNED, or the budget runs
|
||||
// out. A create that has already settled returns without a single extra call.
|
||||
func (a *PoolAllocator) waitAssigned(ctx context.Context, pub oci.PublicIP) (oci.PublicIP, error) {
|
||||
if pub.IsAssigned() {
|
||||
return pub, nil
|
||||
}
|
||||
deadline := time.Now().Add(a.budget)
|
||||
for i := 0; ; i++ {
|
||||
wait := a.schedule[min(i, len(a.schedule)-1)]
|
||||
if j := jitter(assignJitter); wait+j > 0 {
|
||||
wait += j
|
||||
}
|
||||
if time.Now().Add(wait).After(deadline) {
|
||||
return pub, fmt.Errorf("ocinet: public ip %s still %s after %s",
|
||||
pub.ID, pub.LifecycleState, a.budget)
|
||||
}
|
||||
if err := a.sleep(ctx, wait); err != nil {
|
||||
return pub, err
|
||||
}
|
||||
got, err := a.client.GetPublicIP(ctx, pub.ID)
|
||||
if err != nil {
|
||||
return pub, fmt.Errorf("ocinet: poll public ip %s: %w", pub.ID, err)
|
||||
}
|
||||
if got.IsAssigned() {
|
||||
return got, nil
|
||||
}
|
||||
pub = got
|
||||
}
|
||||
}
|
||||
|
||||
// deletePair removes the public IP then the private IP. Public first: deleting
|
||||
// a private IP that still has a public attached is refused by OCI, and the
|
||||
// reverse order would strand the reserved address with nothing pointing at it.
|
||||
// A 404 on either is success — the address is already in the state we wanted.
|
||||
func (a *PoolAllocator) deletePair(ctx context.Context, publicIPID, privateIPID string) error {
|
||||
if publicIPID != "" {
|
||||
if err := a.client.DeletePublicIP(ctx, publicIPID); err != nil && !errors.Is(err, oci.ErrNotFound) {
|
||||
return fmt.Errorf("ocinet: delete public ip %s: %w", publicIPID, err)
|
||||
}
|
||||
}
|
||||
if privateIPID != "" {
|
||||
if err := a.client.UnassignPrivateIP(ctx, privateIPID); err != nil && !errors.Is(err, oci.ErrNotFound) {
|
||||
return fmt.Errorf("ocinet: unassign private ip %s: %w", privateIPID, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// undoPrivate and undoPublic roll back a partial Allocate. Failures are logged
|
||||
// and swallowed: the caller is already returning an error, and the object is
|
||||
// left carrying our prefix so Reconcile collects it on the next pass. Turning a
|
||||
// cleanup failure into the reported error would hide why the allocation failed.
|
||||
func (a *PoolAllocator) undoPrivate(ctx context.Context, privateIPID string) {
|
||||
if err := a.client.UnassignPrivateIP(ctx, privateIPID); err != nil && !errors.Is(err, oci.ErrNotFound) {
|
||||
slog.WarnContext(ctx, "ocinet: could not roll back private ip; reconcile will collect it",
|
||||
"private_ip_id", privateIPID, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *PoolAllocator) undoPublic(ctx context.Context, publicIPID string) {
|
||||
if publicIPID == "" {
|
||||
return
|
||||
}
|
||||
if err := a.client.DeletePublicIP(ctx, publicIPID); err != nil && !errors.Is(err, oci.ErrNotFound) {
|
||||
slog.WarnContext(ctx, "ocinet: could not roll back public ip; reconcile will collect it",
|
||||
"public_ip_id", publicIPID, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// capacityErr maps an OCI quota refusal onto the AWS error that means the same
|
||||
// thing, so a customer who has hit the 64-per-VNIC or 50-per-region ceiling is
|
||||
// told they are out of addresses rather than shown an Oracle error code.
|
||||
func capacityErr(wrapped, raw error) error {
|
||||
if errors.Is(raw, oci.ErrLimitExceeded) {
|
||||
return fmt.Errorf("%w: %w", errors.New(awserrors.ErrorInsufficientAddressCapacity), wrapped)
|
||||
}
|
||||
return wrapped
|
||||
}
|
||||
|
||||
// displayName marks the object as ours and carries the AWS identity, so the OCI
|
||||
// console shows what claimed the address.
|
||||
func displayName(req external.AllocateRequest) string {
|
||||
switch {
|
||||
case req.AllocationID != "":
|
||||
return DisplayNamePrefix + req.AllocationID
|
||||
case req.ENIID != "":
|
||||
return DisplayNamePrefix + req.ENIID
|
||||
case req.InstanceID != "":
|
||||
return DisplayNamePrefix + req.InstanceID
|
||||
default:
|
||||
return DisplayNamePrefix + "unclaimed"
|
||||
}
|
||||
}
|
||||
|
||||
func jitter(span time.Duration) time.Duration {
|
||||
if span <= 0 {
|
||||
return 0
|
||||
}
|
||||
// Spreading the poll ladder, not generating a secret: math/rand is the
|
||||
// right tool and crypto/rand would be cargo cult here.
|
||||
return time.Duration(rand.Int64N(int64(2*span))) - span //nolint:gosec // jitter, not a secret
|
||||
}
|
||||
|
||||
func sleepCtx(ctx context.Context, d time.Duration) error {
|
||||
if d <= 0 {
|
||||
return nil
|
||||
}
|
||||
t := time.NewTimer(d)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-t.C:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
+227
@@ -0,0 +1,227 @@
|
||||
package ocinet_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/netip"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
|
||||
)
|
||||
|
||||
// newTestAllocator wires a fake OCI and an in-memory store, with a poll ladder
|
||||
// that does not sleep so the async-assignment tests stay fast.
|
||||
func newTestAllocator(t *testing.T, fake *oci.Fake) (*ocinet.PoolAllocator, *ocinet.MemStore) {
|
||||
t.Helper()
|
||||
store := ocinet.NewMemStore()
|
||||
a, err := ocinet.New(fake, store, ocinet.Config{
|
||||
Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI},
|
||||
VNICID: "ocid1.vnic.oc1..vnic1",
|
||||
CompartmentID: "ocid1.compartment.oc1..comp1",
|
||||
Schedule: []time.Duration{time.Millisecond},
|
||||
Budget: time.Second,
|
||||
Sleep: func(context.Context, time.Duration) error { return nil },
|
||||
})
|
||||
require.NoError(t, err)
|
||||
return a, store
|
||||
}
|
||||
|
||||
func TestAllocateCreatesThePairAndReturnsThePublicAddress(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
got, err := a.Allocate(ctx, external.AllocateRequest{
|
||||
PoolName: "oci-wan", Purpose: "eip", AllocationID: "eipalloc-1", ENIID: "eni-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
// The public address is what AWS semantics are about, so that is what
|
||||
// Allocate returns — not the private one that rides the wire.
|
||||
assert.Equal(t, "203.0.113.10", got.String())
|
||||
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
b, ok := rec.Bindings[got.String()]
|
||||
require.True(t, ok, "binding not recorded under the public address")
|
||||
assert.Equal(t, "10.200.0.100", b.PrivateAddr, "private half must be recorded for the host plumbing")
|
||||
assert.Equal(t, "eni-1", b.ENIID)
|
||||
assert.NotEmpty(t, b.PublicIPID)
|
||||
assert.NotEmpty(t, b.PrivateIPID)
|
||||
|
||||
require.Len(t, fake.PrivateIPs(), 1)
|
||||
require.Len(t, fake.PublicIPs(), 1)
|
||||
assert.Equal(t, ocinet.DisplayNamePrefix+"eipalloc-1", fake.PrivateIPs()[0].DisplayName,
|
||||
"objects must carry the prefix or reconcile cannot tell them from the operator's")
|
||||
}
|
||||
|
||||
func TestAllocateWaitsOutAsyncAssignment(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
fake.AssignAfter = 2 // ASSIGNING until the second Get
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
got, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
assert.True(t, got.IsValid())
|
||||
|
||||
pubs := fake.PublicIPs()
|
||||
require.Len(t, pubs, 1)
|
||||
assert.Equal(t, oci.LifecycleStateAssigned, pubs[0].LifecycleState,
|
||||
"Allocate must not return an address that is still provisioning")
|
||||
}
|
||||
|
||||
func TestAllocateSurfacesQuotaExhaustionAsInsufficientAddressCapacity(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
fake.PrivateLimit = 1
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
_, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-2"})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorInsufficientAddressCapacity,
|
||||
"an OCI quota refusal must reach the customer as the AWS error that means the same thing")
|
||||
}
|
||||
|
||||
// A failure after the private IP exists must not leave it behind: we are still
|
||||
// running and know exactly what we just created, so rolling back beats leaving
|
||||
// a billable object for the next reconcile pass.
|
||||
func TestAllocateRollsBackThePrivateIPWhenThePublicIPFails(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
fake.FailWith["CreatePublicIP"] = errors.New("boom")
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
_, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.Error(t, err)
|
||||
|
||||
assert.Empty(t, fake.PrivateIPs(), "private IP was not rolled back")
|
||||
assert.Empty(t, fake.PublicIPs())
|
||||
_, err = store.Get(ctx, "oci-wan")
|
||||
require.ErrorIs(t, err, kvstore.ErrNotFound, "a failed allocate must leave no record at all")
|
||||
}
|
||||
|
||||
func TestReleaseDeletesBothObjectsAndDropsTheBinding(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
ip, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1", ENIID: "eni-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Unscoped, as ReleaseAddress is: an owner-scoped release may not free an
|
||||
// EIP, which TestAnOwnerScopedReleaseWillNotFreeAnEIPPair covers.
|
||||
require.NoError(t, a.Release(ctx, "oci-wan", ip, ""))
|
||||
|
||||
assert.Empty(t, fake.PrivateIPs(), "private IP outlived its release")
|
||||
assert.Empty(t, fake.PublicIPs(), "reserved public IP outlived its release — this one bills")
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, rec.Bindings)
|
||||
}
|
||||
|
||||
// External addresses are recycled and teardown sweeps re-emit releases, so a
|
||||
// release naming an owner that no longer holds the binding must be a no-op
|
||||
// rather than a double-free of an address already handed to another instance.
|
||||
func TestReleaseWithAStaleOwnerIsANoOp(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
// Auto-assigned, so the stale-owner guard is what decides this and not the
|
||||
// EIP refusal next to it.
|
||||
ip, err := a.Allocate(ctx, external.AllocateRequest{
|
||||
PoolName: "oci-wan", Purpose: "eni-public", ENIID: "eni-current", InstanceID: "i-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, a.Release(ctx, "oci-wan", ip, "eni-previous"))
|
||||
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, rec.Bindings, 1, "a stale release freed an address belonging to a live ENI")
|
||||
assert.Len(t, fake.PublicIPs(), 1)
|
||||
}
|
||||
|
||||
func TestReleaseOfAnUnknownAddressIsANoOpWhenOwnerScoped(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, _ := newTestAllocator(t, oci.NewFake())
|
||||
|
||||
// Owner-scoped: a duplicated teardown for an address already freed.
|
||||
require.NoError(t, a.Release(ctx, "oci-wan", netip.MustParseAddr("203.0.113.99"), "eni-1"))
|
||||
// Unscoped: nothing claims to know better, so this is a real error.
|
||||
require.Error(t, a.Release(ctx, "oci-wan", netip.MustParseAddr("203.0.113.99"), ""))
|
||||
}
|
||||
|
||||
func TestPoolMismatchIsRefused(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, _ := newTestAllocator(t, oci.NewFake())
|
||||
|
||||
_, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "other-pool", AllocationID: "eipalloc-1"})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "pool mismatch")
|
||||
|
||||
err = a.Release(ctx, "other-pool", netip.MustParseAddr("203.0.113.10"), "")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "pool mismatch")
|
||||
}
|
||||
|
||||
func TestNewRejectsIncompleteConfig(t *testing.T) {
|
||||
store := ocinet.NewMemStore()
|
||||
fake := oci.NewFake()
|
||||
|
||||
_, err := ocinet.New(nil, store, ocinet.Config{VNICID: "v", CompartmentID: "c"})
|
||||
require.Error(t, err)
|
||||
|
||||
_, err = ocinet.New(fake, nil, ocinet.Config{VNICID: "v", CompartmentID: "c"})
|
||||
require.Error(t, err)
|
||||
|
||||
_, err = ocinet.New(fake, store, ocinet.Config{CompartmentID: "c"})
|
||||
require.ErrorContains(t, err, "vnic_id")
|
||||
|
||||
_, err = ocinet.New(fake, store, ocinet.Config{VNICID: "v"})
|
||||
require.ErrorContains(t, err, "compartment_id")
|
||||
}
|
||||
|
||||
func TestBindingForReturnsThePrivateHalf(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, _ := newTestAllocator(t, oci.NewFake())
|
||||
|
||||
ip, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
b, ok, err := a.BindingFor(ctx, ip)
|
||||
require.NoError(t, err)
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, "10.200.0.100", b.PrivateAddr)
|
||||
|
||||
_, ok, err = a.BindingFor(ctx, netip.MustParseAddr("203.0.113.200"))
|
||||
require.NoError(t, err)
|
||||
assert.False(t, ok)
|
||||
}
|
||||
|
||||
// The VNIC on the binding is what lets a node returning from a failure tell an
|
||||
// address that is still its own from one that has moved to a surviving node,
|
||||
// before it reasserts host plumbing for an address it no longer holds.
|
||||
func TestAllocateRecordsTheOwningVNIC(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, store := newTestAllocator(t, oci.NewFake())
|
||||
|
||||
ip, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ocid1.vnic.oc1..vnic1", rec.Bindings[ip.String()].VNICID)
|
||||
}
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
package ocinet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
)
|
||||
|
||||
// Lookup answers "what address does this external IP ride the wire as" from the
|
||||
// bindings store alone. No OCI client and no credentials, because the process
|
||||
// that needs the answer is vpcd: it programs the OVN NAT rule and the host
|
||||
// ingress route, and both must match the private half of the pair while every
|
||||
// AWS-facing record holds the public half.
|
||||
//
|
||||
// Reading KV rather than asking the allocator is not duplication of state —
|
||||
// vpcd is a separate process from the daemon that owns the allocator, and the
|
||||
// bindings record is the only thing either of them would consult.
|
||||
type Lookup struct {
|
||||
store Store
|
||||
pools []string
|
||||
}
|
||||
|
||||
// NewLookup binds a read-only resolver to the named OCI pools.
|
||||
func NewLookup(store Store, pools []string) *Lookup {
|
||||
return &Lookup{store: store, pools: append([]string(nil), pools...)}
|
||||
}
|
||||
|
||||
// DatapathIP maps externalIP to its on-wire private address, or returns it
|
||||
// unchanged when no OCI pool holds a binding for it — every address on a
|
||||
// static or DHCP pool is its own datapath address, and vpcd cannot tell which
|
||||
// pool an address came from without asking.
|
||||
//
|
||||
// A store read failure is an error rather than a pass-through. Falling back to
|
||||
// the public address would install a NAT rule matching an address that never
|
||||
// appears on the wire, which is indistinguishable from a working rule until
|
||||
// the instance turns out to be unreachable.
|
||||
func (l *Lookup) DatapathIP(ctx context.Context, externalIP string) (string, error) {
|
||||
if l == nil || externalIP == "" {
|
||||
return externalIP, nil
|
||||
}
|
||||
for _, pool := range l.pools {
|
||||
rec, err := l.store.Get(ctx, pool)
|
||||
if errors.Is(err, kvstore.ErrNotFound) {
|
||||
// A pool that has never allocated holds no bindings, which is a
|
||||
// miss on this pool rather than a reason to fail the lookup.
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("ocinet: read bindings for pool %q: %w", pool, err)
|
||||
}
|
||||
b, ok := rec.Bindings[externalIP]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if _, err := netip.ParseAddr(b.PrivateAddr); err != nil {
|
||||
return "", fmt.Errorf("ocinet: binding for %s in pool %q has unparseable private addr %q: %w",
|
||||
externalIP, pool, b.PrivateAddr, err)
|
||||
}
|
||||
return b.PrivateAddr, nil
|
||||
}
|
||||
return externalIP, nil
|
||||
}
|
||||
|
||||
// BindGateway records vpcID as the VPC whose NAT gateway answers on externalIP,
|
||||
// so the affinity pass knows which node the address belongs on.
|
||||
//
|
||||
// A NAT gateway's address is the one external address never attached to an ENI:
|
||||
// it is a logical router's SNAT source, not a guest's. Without this the affinity
|
||||
// pass has nothing to place it by and leaves it on whichever node served
|
||||
// AllocateAddress, which OCI then refuses to let it leave from anywhere else.
|
||||
//
|
||||
// Written from the reconcile pass rather than from CreateNatGateway because the
|
||||
// answer is the gateway chassis, which reconcile is what decides. Idempotent:
|
||||
// an unchanged VPC is not a write, so the steady state costs one read.
|
||||
func (l *Lookup) BindGateway(ctx context.Context, externalIP, vpcID string) error {
|
||||
if l == nil || externalIP == "" {
|
||||
return nil
|
||||
}
|
||||
for _, pool := range l.pools {
|
||||
err := l.store.Mutate(ctx, pool, func(rec *Record) (bool, error) {
|
||||
b, ok := rec.Bindings[externalIP]
|
||||
if !ok || b.GatewayVPCID == vpcID {
|
||||
return false, nil
|
||||
}
|
||||
b.GatewayVPCID = vpcID
|
||||
rec.Bindings[externalIP] = b
|
||||
return true, nil
|
||||
})
|
||||
if errors.Is(err, kvstore.ErrNotFound) {
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("ocinet: record gateway VPC for %s in pool %q: %w", externalIP, pool, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
package ocinet_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
|
||||
)
|
||||
|
||||
func seedBinding(t *testing.T, s *ocinet.MemStore, pool, public, private string) {
|
||||
t.Helper()
|
||||
require.NoError(t, s.Mutate(context.Background(), pool, func(rec *ocinet.Record) (bool, error) {
|
||||
rec.Bindings[public] = ocinet.Binding{PrivateAddr: private}
|
||||
return true, nil
|
||||
}))
|
||||
}
|
||||
|
||||
func TestLookupReturnsTheOnWirePrivateAddress(t *testing.T) {
|
||||
store := ocinet.NewMemStore()
|
||||
seedBinding(t, store, "oci-pool", "203.0.113.9", "10.200.0.183")
|
||||
|
||||
got, err := ocinet.NewLookup(store, []string{"oci-pool"}).DatapathIP(context.Background(), "203.0.113.9")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "10.200.0.183", got)
|
||||
}
|
||||
|
||||
// vpcd cannot tell which pool an address came from, so an address no OCI pool
|
||||
// knows about is its own datapath address — every static and DHCP one is.
|
||||
func TestLookupPassesThroughAnUnboundAddress(t *testing.T) {
|
||||
store := ocinet.NewMemStore()
|
||||
seedBinding(t, store, "oci-pool", "203.0.113.9", "10.200.0.183")
|
||||
|
||||
got, err := ocinet.NewLookup(store, []string{"oci-pool"}).DatapathIP(context.Background(), "72.52.77.240")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "72.52.77.240", got)
|
||||
}
|
||||
|
||||
func TestLookupSearchesEveryConfiguredPool(t *testing.T) {
|
||||
store := ocinet.NewMemStore()
|
||||
seedBinding(t, store, "pool-b", "203.0.113.9", "10.200.0.183")
|
||||
|
||||
got, err := ocinet.NewLookup(store, []string{"pool-a", "pool-b"}).DatapathIP(context.Background(), "203.0.113.9")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "10.200.0.183", got)
|
||||
}
|
||||
|
||||
// Passing the public address through on a read failure installs a rule that
|
||||
// looks healthy and blackholes the instance.
|
||||
func TestLookupStoreFailureIsAnError(t *testing.T) {
|
||||
got, err := ocinet.NewLookup(failingStore{}, []string{"oci-pool"}).DatapathIP(context.Background(), "203.0.113.9")
|
||||
require.Error(t, err)
|
||||
assert.Empty(t, got)
|
||||
}
|
||||
|
||||
type failingStore struct{}
|
||||
|
||||
func (failingStore) Mutate(context.Context, string, func(*ocinet.Record) (bool, error)) error {
|
||||
return errors.New("unavailable")
|
||||
}
|
||||
|
||||
func (failingStore) Get(context.Context, string) (ocinet.Record, error) {
|
||||
return ocinet.Record{}, errors.New("unavailable")
|
||||
}
|
||||
|
||||
// The affinity pass places a NAT gateway's address by the VPC it serves, and
|
||||
// this is the only writer of that field.
|
||||
func TestBindGatewayRecordsTheVPCOnTheBinding(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := ocinet.NewMemStore()
|
||||
seedBinding(t, store, "oci-pool", "203.0.113.9", "10.200.0.183")
|
||||
|
||||
require.NoError(t, ocinet.NewLookup(store, []string{"oci-pool"}).BindGateway(ctx, "203.0.113.9", "vpc-1"))
|
||||
|
||||
rec, err := store.Get(ctx, "oci-pool")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "vpc-1", rec.Bindings["203.0.113.9"].GatewayVPCID)
|
||||
assert.Equal(t, "10.200.0.183", rec.Bindings["203.0.113.9"].PrivateAddr, "the pair must survive the annotation")
|
||||
}
|
||||
|
||||
// Deleting the gateway hands the address back, and a stale VPC would keep the
|
||||
// affinity pass moving it to a chassis nothing forwards from.
|
||||
func TestBindGatewayClearsTheVPC(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := ocinet.NewMemStore()
|
||||
seedBinding(t, store, "oci-pool", "203.0.113.9", "10.200.0.183")
|
||||
l := ocinet.NewLookup(store, []string{"oci-pool"})
|
||||
require.NoError(t, l.BindGateway(ctx, "203.0.113.9", "vpc-1"))
|
||||
|
||||
require.NoError(t, l.BindGateway(ctx, "203.0.113.9", ""))
|
||||
|
||||
rec, err := store.Get(ctx, "oci-pool")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, rec.Bindings["203.0.113.9"].GatewayVPCID)
|
||||
}
|
||||
|
||||
// Every node runs this every pass, so an address no OCI pool holds must not be
|
||||
// an error — that is every static and DHCP address on a mixed cluster.
|
||||
func TestBindGatewayIgnoresAnUnboundAddress(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := ocinet.NewMemStore()
|
||||
seedBinding(t, store, "oci-pool", "203.0.113.9", "10.200.0.183")
|
||||
|
||||
require.NoError(t, ocinet.NewLookup(store, []string{"oci-pool"}).BindGateway(ctx, "72.52.77.240", "vpc-1"))
|
||||
}
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
package ocinet_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/netip"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
|
||||
)
|
||||
|
||||
// allocateBare is an EIP as AWS makes one: an address with no interface behind
|
||||
// it yet. Everything about the owner has to arrive later.
|
||||
func allocateBare(t *testing.T, a *ocinet.PoolAllocator) netip.Addr {
|
||||
t.Helper()
|
||||
ip, err := a.Allocate(context.Background(), external.AllocateRequest{
|
||||
PoolName: "oci-wan", Purpose: "eip", AllocationID: "eipalloc-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
return ip
|
||||
}
|
||||
|
||||
func ownerOf(t *testing.T, store *ocinet.MemStore, ip netip.Addr) string {
|
||||
t.Helper()
|
||||
rec, err := store.Get(context.Background(), "oci-wan")
|
||||
require.NoError(t, err)
|
||||
return rec.Bindings[ip.String()].ENIID
|
||||
}
|
||||
|
||||
// Without this the binding of every EIP names no ENI, and the affinity pass
|
||||
// cannot tell which node the address has to be delivered to.
|
||||
func TestBindOwnerRecordsTheENIAnEIPWasAssociatedWith(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, store := newTestAllocator(t, oci.NewFake())
|
||||
ip := allocateBare(t, a)
|
||||
assert.Empty(t, ownerOf(t, store, ip), "a bare allocation should name no owner")
|
||||
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", ip, "eni-1"))
|
||||
assert.Equal(t, "eni-1", ownerOf(t, store, ip))
|
||||
}
|
||||
|
||||
// A disassociated address is still allocated and still billing, but no guest
|
||||
// answers on it, so no node should claim it.
|
||||
func TestBindOwnerClearsTheOwnerOnDisassociate(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, store := newTestAllocator(t, oci.NewFake())
|
||||
ip := allocateBare(t, a)
|
||||
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", ip, "eni-1"))
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", ip, ""))
|
||||
assert.Empty(t, ownerOf(t, store, ip))
|
||||
}
|
||||
|
||||
// A re-association to the same ENI is the steady state of every reconcile, so
|
||||
// it must not churn the record.
|
||||
func TestBindOwnerDoesNotWriteWhenTheOwnerIsUnchanged(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, store := newTestAllocator(t, oci.NewFake())
|
||||
ip := allocateBare(t, a)
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", ip, "eni-1"))
|
||||
|
||||
before := store.Writes()
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", ip, "eni-1"))
|
||||
assert.Equal(t, before, store.Writes(), "an unchanged owner cost a KV write")
|
||||
}
|
||||
|
||||
// An address this pool never handed out is not ours to annotate — inventing a
|
||||
// binding for it would have Reconcile collect an address belonging to nobody.
|
||||
func TestBindOwnerIgnoresAnAddressThisPoolNeverAllocated(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, store := newTestAllocator(t, oci.NewFake())
|
||||
allocateBare(t, a)
|
||||
|
||||
stranger := netip.MustParseAddr("198.51.100.7")
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", stranger, "eni-9"))
|
||||
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.NotContains(t, rec.Bindings, stranger.String())
|
||||
}
|
||||
|
||||
func TestBindOwnerRefusesAnotherPoolsName(t *testing.T) {
|
||||
a, _ := newTestAllocator(t, oci.NewFake())
|
||||
ip := allocateBare(t, a)
|
||||
|
||||
err := a.BindOwner(context.Background(), "some-other-pool", ip, "eni-1")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "pool mismatch")
|
||||
}
|
||||
|
||||
// The whole point of recording the owner: an EIP associated to a guest running
|
||||
// here is then claimable, which it never was before.
|
||||
func TestAnEIPBecomesClaimableOnceItsOwnerIsRecorded(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newAffinityAllocator(t, fake, "port-eni-1")
|
||||
pubAddr, privID := seedForeignBinding(t, fake, store, "")
|
||||
|
||||
res, err := a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, res.Claimed, "an unassociated address has no owning node")
|
||||
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", netip.MustParseAddr(pubAddr), "eni-1"))
|
||||
|
||||
res, err = a.ClaimLocalAddresses(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{pubAddr}, res.Claimed)
|
||||
|
||||
got, err := fake.GetPrivateIP(ctx, privID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, thisVNIC, got.VNICID, "OCI still delivers the EIP to another node")
|
||||
}
|
||||
|
||||
// On OCI the address is a reserved public IP object, so an owner-scoped release
|
||||
// of one does not merely return it to a pool — it destroys it at the provider
|
||||
// and the customer cannot get it back.
|
||||
func TestAnOwnerScopedReleaseWillNotFreeAnEIPPair(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
ip, err := a.Allocate(ctx, external.AllocateRequest{
|
||||
PoolName: "oci-wan", Purpose: "eip", AllocationID: "eipalloc-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, a.BindOwner(ctx, "oci-wan", ip, "eni-1"))
|
||||
|
||||
// What instance teardown does once the EIP is what the instance holds.
|
||||
require.NoError(t, a.Release(ctx, "oci-wan", ip, "eni-1"))
|
||||
|
||||
assert.Len(t, fake.PublicIPs(), 1,
|
||||
"terminating an instance destroyed a customer's reserved public IP")
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, rec.Bindings, 1)
|
||||
}
|
||||
+153
@@ -0,0 +1,153 @@
|
||||
package ocinet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
)
|
||||
|
||||
// ReconcileResult reports what a pass found, so a caller can log it and a test
|
||||
// can assert on it without reading the fake's internals.
|
||||
type ReconcileResult struct {
|
||||
// Collected is the OCIDs of private IPs deleted as leaks.
|
||||
Collected []string
|
||||
// Stale is the binding keys dropped because OCI no longer has the objects.
|
||||
Stale []string
|
||||
// Skipped is the private IPs left alone because they are not ours.
|
||||
Skipped int
|
||||
}
|
||||
|
||||
// Reconcile squares our record against OCI, in both directions. It is required
|
||||
// on startup rather than optional: Allocate creates OCI objects before writing
|
||||
// the record, so a crash in between leaves a reserved public IP that costs
|
||||
// money and holds one of the 50 per-region slots, with nothing referencing it.
|
||||
// Nothing else will ever find that address.
|
||||
//
|
||||
// The two directions fix different faults:
|
||||
//
|
||||
// - An object on the VNIC carrying our prefix with no binding is a leak from
|
||||
// an interrupted Allocate. Delete it.
|
||||
// - A binding naming objects OCI no longer has is a stale record from an
|
||||
// interrupted Release, or from someone deleting the address in the console.
|
||||
// Drop it, so the address is not reported to a customer as theirs.
|
||||
//
|
||||
// **It never touches an object without DisplayNamePrefix.** The reference host
|
||||
// carries operator-created secondary addresses on the same VNIC, and a
|
||||
// reconcile that collected those would take the node's own networking down. The
|
||||
// prefix is the entire safety property here, so an object whose name we cannot
|
||||
// read is skipped, not guessed at.
|
||||
func (a *PoolAllocator) Reconcile(ctx context.Context) (ReconcileResult, error) {
|
||||
var res ReconcileResult
|
||||
|
||||
live, err := a.client.ListPrivateIPs(ctx, a.cfg.VNICID)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("ocinet reconcile: list private ips on %s: %w", a.cfg.VNICID, err)
|
||||
}
|
||||
// A pool that has never allocated has no record, which is an empty binding
|
||||
// set rather than a failure — and the reconcile still has work to do, since
|
||||
// the leak this pass exists to find is precisely an OCI object created
|
||||
// before its binding was written.
|
||||
rec, err := a.store.Get(ctx, a.cfg.Pool.Name)
|
||||
if err != nil && !errors.Is(err, kvstore.ErrNotFound) {
|
||||
return res, fmt.Errorf("ocinet reconcile: read bindings: %w", err)
|
||||
}
|
||||
|
||||
claimed := make(map[string]struct{}, len(rec.Bindings))
|
||||
for _, b := range rec.Bindings {
|
||||
claimed[b.PrivateIPID] = struct{}{}
|
||||
}
|
||||
|
||||
// Direction 1: OCI has objects we do not claim.
|
||||
for _, p := range live {
|
||||
if p.IsPrimary || !strings.HasPrefix(p.DisplayName, DisplayNamePrefix) {
|
||||
res.Skipped++
|
||||
continue
|
||||
}
|
||||
if _, ok := claimed[p.ID]; ok {
|
||||
continue
|
||||
}
|
||||
publicIPID := ""
|
||||
if pub, err := a.client.GetPublicIPByPrivateIPID(ctx, p.ID); err == nil {
|
||||
publicIPID = pub.ID
|
||||
} else if !errors.Is(err, oci.ErrNotFound) {
|
||||
return res, fmt.Errorf("ocinet reconcile: look up public ip for %s: %w", p.ID, err)
|
||||
}
|
||||
if err := a.deletePair(ctx, publicIPID, p.ID); err != nil {
|
||||
return res, fmt.Errorf("ocinet reconcile: collect leaked %s: %w", p.ID, err)
|
||||
}
|
||||
slog.WarnContext(ctx, "ocinet reconcile collected a leaked address",
|
||||
"pool", a.cfg.Pool.Name, "private_ip_id", p.ID, "private_ip", p.Address.String(),
|
||||
"public_ip_id", publicIPID, "display_name", p.DisplayName)
|
||||
res.Collected = append(res.Collected, p.ID)
|
||||
}
|
||||
|
||||
// Direction 2: we claim objects OCI does not have.
|
||||
byID := make(map[string]struct{}, len(live))
|
||||
for _, p := range live {
|
||||
byID[p.ID] = struct{}{}
|
||||
}
|
||||
// Only this node's own bindings are this node's to judge. The record is one
|
||||
// key per pool and therefore cluster-wide, while `live` is one VNIC — so a
|
||||
// binding held by another node looks exactly like a binding whose objects
|
||||
// OCI has lost. Dropping those makes every node delete its peers' addresses
|
||||
// on startup, and direction 1 on the owning node then collects the now
|
||||
// unclaimed objects as leaks: a live address is destroyed under a running
|
||||
// instance. A binding naming no VNIC cannot be attributed, so it is left
|
||||
// alone rather than guessed at.
|
||||
var stale []string
|
||||
var foreign int
|
||||
for key, b := range rec.Bindings {
|
||||
if _, ok := byID[b.PrivateIPID]; ok {
|
||||
continue
|
||||
}
|
||||
if b.VNICID != a.cfg.VNICID {
|
||||
foreign++
|
||||
continue
|
||||
}
|
||||
stale = append(stale, key)
|
||||
}
|
||||
if foreign > 0 {
|
||||
slog.DebugContext(ctx, "ocinet reconcile left other nodes' bindings alone",
|
||||
"pool", a.cfg.Pool.Name, "vnic_id", a.cfg.VNICID, "foreign", foreign)
|
||||
}
|
||||
if len(stale) > 0 {
|
||||
err = a.store.Mutate(ctx, a.cfg.Pool.Name, func(r *Record) (bool, error) {
|
||||
changed := false
|
||||
for _, key := range stale {
|
||||
// Re-check under CAS: another node may have re-allocated this
|
||||
// address between our list and this mutation, in which case the
|
||||
// binding is live again and dropping it would strand a real one.
|
||||
b, ok := r.Bindings[key]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if _, still := byID[b.PrivateIPID]; still {
|
||||
continue
|
||||
}
|
||||
// Re-check ownership too: the re-allocation may have been to
|
||||
// another node's VNIC, which is a live address elsewhere.
|
||||
if b.VNICID != a.cfg.VNICID {
|
||||
continue
|
||||
}
|
||||
delete(r.Bindings, key)
|
||||
changed = true
|
||||
}
|
||||
return changed, nil
|
||||
})
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("ocinet reconcile: drop stale bindings: %w", err)
|
||||
}
|
||||
for _, key := range stale {
|
||||
slog.WarnContext(ctx, "ocinet reconcile dropped a stale binding",
|
||||
"pool", a.cfg.Pool.Name, "public_ip", key)
|
||||
}
|
||||
res.Stale = stale
|
||||
}
|
||||
|
||||
return res, nil
|
||||
}
|
||||
+229
@@ -0,0 +1,229 @@
|
||||
package ocinet_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/netip"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
|
||||
)
|
||||
|
||||
// A crash between creating the OCI objects and writing the binding leaves a
|
||||
// reserved public IP that bills and holds one of the 50 per-region slots, with
|
||||
// nothing referencing it. Reconcile is the only thing that can ever find it.
|
||||
func TestReconcileCollectsALeakedPairFromAnInterruptedAllocate(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
// Simulate the crash: the objects exist and carry our prefix, but the
|
||||
// binding was never written.
|
||||
priv, err := fake.AssignPrivateIP(ctx, "ocid1.vnic.oc1..vnic1", netip.Addr{}, ocinet.DisplayNamePrefix+"eipalloc-lost")
|
||||
require.NoError(t, err)
|
||||
_, err = fake.CreatePublicIP(ctx, "ocid1.compartment.oc1..comp1", priv.ID, ocinet.DisplayNamePrefix+"eipalloc-lost")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, fake.PublicIPs(), 1)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, []string{priv.ID}, res.Collected)
|
||||
assert.Empty(t, fake.PrivateIPs(), "leaked private IP was not collected")
|
||||
assert.Empty(t, fake.PublicIPs(), "leaked reserved public IP was not collected — this one bills")
|
||||
|
||||
// The leak this pass collects is an OCI pair created before its binding was
|
||||
// ever written, so the pool still holds no record afterwards.
|
||||
_, err = store.Get(ctx, "oci-wan")
|
||||
require.ErrorIs(t, err, kvstore.ErrNotFound)
|
||||
}
|
||||
|
||||
// The safety property of the whole pass. mulga-poc carries operator-created
|
||||
// secondary addresses on the same VNIC that Spinifex allocates from, and a
|
||||
// reconcile that collected those would take the node's own networking down.
|
||||
func TestReconcileNeverTouchesAddressesThatAreNotOurs(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
// The operator's own: the VNIC's primary, and a hand-made secondary.
|
||||
fake.SeedPrivateIP(oci.PrivateIP{
|
||||
ID: "ocid1.privateip.oc1..primary", Address: netip.MustParseAddr("10.200.0.29"),
|
||||
VNICID: "ocid1.vnic.oc1..vnic1", DisplayName: "vnic-1", IsPrimary: true,
|
||||
})
|
||||
fake.SeedPrivateIP(oci.PrivateIP{
|
||||
ID: "ocid1.privateip.oc1..operator", Address: netip.MustParseAddr("10.200.0.81"),
|
||||
VNICID: "ocid1.vnic.oc1..vnic1", DisplayName: "vnic-v3",
|
||||
})
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Collected, "reconcile collected an address it did not create")
|
||||
assert.Equal(t, 2, res.Skipped)
|
||||
assert.Len(t, fake.PrivateIPs(), 2, "operator addresses must survive a reconcile pass")
|
||||
}
|
||||
|
||||
// An interrupted Release, or someone deleting the address in the OCI console,
|
||||
// leaves a binding naming objects that no longer exist. Left in place it would
|
||||
// be reported to a customer as their address.
|
||||
func TestReconcileDropsABindingWhoseObjectsAreGone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
ip, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Remove the objects behind our back.
|
||||
for _, p := range fake.PrivateIPs() {
|
||||
require.NoError(t, fake.UnassignPrivateIP(ctx, p.ID))
|
||||
}
|
||||
for _, p := range fake.PublicIPs() {
|
||||
require.NoError(t, fake.DeletePublicIP(ctx, p.ID))
|
||||
}
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, []string{ip.String()}, res.Stale)
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, rec.Bindings, "stale binding survived reconcile")
|
||||
}
|
||||
|
||||
func TestReconcileLeavesAHealthyAllocationAlone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, store := newTestAllocator(t, fake)
|
||||
|
||||
ip, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Collected)
|
||||
assert.Empty(t, res.Stale)
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, rec.Bindings, ip.String())
|
||||
assert.Len(t, fake.PrivateIPs(), 1)
|
||||
assert.Len(t, fake.PublicIPs(), 1)
|
||||
}
|
||||
|
||||
// A leaked private IP that never got a public IP attached — the crash window
|
||||
// between the two creates — must still be collected.
|
||||
func TestReconcileCollectsALeakedPrivateIPWithNoPublicIP(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
priv, err := fake.AssignPrivateIP(ctx, "ocid1.vnic.oc1..vnic1", netip.Addr{}, ocinet.DisplayNamePrefix+"eipalloc-half")
|
||||
require.NoError(t, err)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, []string{priv.ID}, res.Collected)
|
||||
assert.Empty(t, fake.PrivateIPs())
|
||||
}
|
||||
|
||||
// A pool that has never allocated has no record. That is an empty binding set,
|
||||
// not a failure — and the pass still has to run, since the leak it exists to
|
||||
// find is an OCI object created before its binding was ever written.
|
||||
func TestReconcileOnAPoolThatHasNeverAllocated(t *testing.T) {
|
||||
fake := oci.NewFake()
|
||||
alloc, store := newTestAllocator(t, fake)
|
||||
|
||||
if _, err := store.Get(context.Background(), "oci-wan"); err == nil {
|
||||
t.Fatal("precondition: expected no record for a pool that never allocated")
|
||||
}
|
||||
|
||||
res, err := alloc.Reconcile(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Reconcile on an empty pool: %v", err)
|
||||
}
|
||||
if len(res.Collected) != 0 || len(res.Stale) != 0 {
|
||||
t.Errorf("expected nothing to collect: %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
// newPeerAllocator builds a second node's allocator over the same store: one
|
||||
// KV record per pool, one VNIC per node, which is the shape a multi-node
|
||||
// cluster actually has.
|
||||
func newPeerAllocator(t *testing.T, fake *oci.Fake, store ocinet.Store, vnicID string) *ocinet.PoolAllocator {
|
||||
t.Helper()
|
||||
a, err := ocinet.New(fake, store, ocinet.Config{
|
||||
Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI},
|
||||
VNICID: vnicID,
|
||||
CompartmentID: "ocid1.compartment.oc1..comp1",
|
||||
Schedule: []time.Duration{time.Millisecond},
|
||||
Budget: time.Second,
|
||||
Sleep: func(context.Context, time.Duration) error { return nil },
|
||||
})
|
||||
require.NoError(t, err)
|
||||
return a
|
||||
}
|
||||
|
||||
// The record is one key per pool and so cluster-wide, while the live list is
|
||||
// one VNIC. A peer's binding therefore looks exactly like a binding whose
|
||||
// objects OCI has lost, and dropping it makes every node delete its peers'
|
||||
// addresses on startup — after which direction 1 on the owning node collects
|
||||
// the now-unclaimed objects as leaks and a running instance loses its address.
|
||||
func TestReconcileLeavesAnotherNodesBindingAlone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
node1, store := newTestAllocator(t, fake)
|
||||
node2 := newPeerAllocator(t, fake, store, "ocid1.vnic.oc1..vnic2")
|
||||
|
||||
ip, err := node1.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
res, err := node2.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Stale, "node2 dropped node1's binding")
|
||||
assert.Empty(t, res.Collected, "node2 collected node1's address")
|
||||
|
||||
rec, err := store.Get(ctx, "oci-wan")
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, rec.Bindings, ip.String(), "node1's binding did not survive node2's reconcile")
|
||||
|
||||
// The destructive half: with the binding gone, node1's own next pass would
|
||||
// see an unclaimed object carrying our prefix and delete it.
|
||||
res, err = node1.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Collected, "node1 collected its own live address as a leak")
|
||||
assert.Len(t, fake.PublicIPs(), 1, "the reserved public IP of a running instance was deleted")
|
||||
}
|
||||
|
||||
// A node must still drop its own stale bindings, or an interrupted Release
|
||||
// leaves an address reported to a customer as theirs forever.
|
||||
func TestReconcileStillDropsItsOwnStaleBinding(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
node1, store := newTestAllocator(t, fake)
|
||||
node2 := newPeerAllocator(t, fake, store, "ocid1.vnic.oc1..vnic2")
|
||||
|
||||
ip, err := node1.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.NoError(t, err)
|
||||
for _, p := range fake.PrivateIPs() {
|
||||
require.NoError(t, fake.UnassignPrivateIP(ctx, p.ID))
|
||||
}
|
||||
|
||||
// node2 is not the owner, so it must not be the one to decide.
|
||||
res, err := node2.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, res.Stale)
|
||||
|
||||
res, err = node1.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{ip.String()}, res.Stale)
|
||||
}
|
||||
+158
@@ -0,0 +1,158 @@
|
||||
package ocinet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"maps"
|
||||
"sync"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
const (
|
||||
// KVBucketOCIBindings holds the public-address → OCI object-pair mapping.
|
||||
// A bucket of its own rather than a corner of the static-pool bucket:
|
||||
// nothing about it is range math, its records are keyed differently, and
|
||||
// mixing the two would make the static allocator's migrations answerable
|
||||
// for a schema it knows nothing about.
|
||||
KVBucketOCIBindings = "spinifex-oci-bindings"
|
||||
// KVBucketOCIBindingsHistory matches the static pool bucket.
|
||||
KVBucketOCIBindingsHistory = 5
|
||||
// kvCASRetries bounds contention on one pool key.
|
||||
kvCASRetries = 10
|
||||
)
|
||||
|
||||
func bucketConfig() kvstore.Config {
|
||||
return kvstore.Config{
|
||||
Name: KVBucketOCIBindings,
|
||||
History: KVBucketOCIBindingsHistory,
|
||||
Attempts: kvCASRetries,
|
||||
Missing: "ocinet: no JetStream client configured",
|
||||
Exhausted: func(key string, attempts int) error {
|
||||
return fmt.Errorf("ocinet: pool %s contended after %d attempts", key, attempts)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// KVStore is the JetStream-backed Store.
|
||||
type KVStore struct {
|
||||
store *kvstore.Store[Record]
|
||||
}
|
||||
|
||||
var _ Store = (*KVStore)(nil)
|
||||
|
||||
// NewKVStore creates the bindings bucket if it is missing.
|
||||
func NewKVStore(js jetstream.JetStream) *KVStore {
|
||||
return &KVStore{store: kvstore.New[Record](js, bucketConfig())}
|
||||
}
|
||||
|
||||
// NewKVStoreOver builds a store over an already-open bucket.
|
||||
func NewKVStoreOver(kv jetstream.KeyValue) *KVStore {
|
||||
return &KVStore{store: kvstore.Over[Record](nil, kv, bucketConfig())}
|
||||
}
|
||||
|
||||
// Mutate implements Store. Upsert rather than Mutate: the first allocation on
|
||||
// a pool is what creates its record, and Mutate reports ErrNotFound for an
|
||||
// absent key — so every pool would fail its own first allocation, after OCI
|
||||
// had already created and billed the pair.
|
||||
func (s *KVStore) Mutate(ctx context.Context, poolName string, fn func(*Record) (bool, error)) error {
|
||||
return s.store.Upsert(ctx, poolName, func(rec *Record) (bool, error) {
|
||||
// A record written before the field existed, or a freshly created one,
|
||||
// decodes to a nil map that the callers below would panic assigning to.
|
||||
if rec.Bindings == nil {
|
||||
rec.Bindings = map[string]Binding{}
|
||||
}
|
||||
return fn(rec)
|
||||
})
|
||||
}
|
||||
|
||||
// Get implements Store. A pool with no record yet reports kvstore.ErrNotFound,
|
||||
// which callers read as an empty binding set — surfacing it rather than hiding
|
||||
// it keeps a bucket that has gone missing distinguishable from a pool that has
|
||||
// simply never allocated.
|
||||
func (s *KVStore) Get(ctx context.Context, poolName string) (Record, error) {
|
||||
rec, _, err := s.store.Get(ctx, poolName)
|
||||
if err != nil {
|
||||
return Record{}, err
|
||||
}
|
||||
if rec == nil {
|
||||
return Record{Bindings: map[string]Binding{}}, nil
|
||||
}
|
||||
if rec.Bindings == nil {
|
||||
rec.Bindings = map[string]Binding{}
|
||||
}
|
||||
return *rec, nil
|
||||
}
|
||||
|
||||
// MemStore is an in-memory Store for tests. Concurrency-safe so a test can
|
||||
// exercise the CAS-retry paths without NATS.
|
||||
type MemStore struct {
|
||||
mu sync.Mutex
|
||||
records map[string]Record
|
||||
// mutateErr, when set, fails every Mutate. Lets a test drive the paths
|
||||
// where OCI has already been changed and only the record write fails.
|
||||
mutateErr error
|
||||
// writes counts committed mutations, so a test can assert that a pass which
|
||||
// changes nothing costs no write.
|
||||
writes int
|
||||
}
|
||||
|
||||
var _ Store = (*MemStore)(nil)
|
||||
|
||||
// NewMemStore returns an empty in-memory store.
|
||||
func NewMemStore() *MemStore { return &MemStore{records: map[string]Record{}} }
|
||||
|
||||
// FailMutate makes every subsequent Mutate return err.
|
||||
func (m *MemStore) FailMutate(err error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.mutateErr = err
|
||||
}
|
||||
|
||||
// Mutate implements Store.
|
||||
func (m *MemStore) Mutate(_ context.Context, poolName string, fn func(*Record) (bool, error)) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.mutateErr != nil {
|
||||
return m.mutateErr
|
||||
}
|
||||
rec := m.copyLocked(poolName)
|
||||
changed, err := fn(&rec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
m.records[poolName] = rec
|
||||
m.writes++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Writes returns how many mutations have committed.
|
||||
func (m *MemStore) Writes() int {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.writes
|
||||
}
|
||||
|
||||
// Get implements Store, including the not-found error the KV store returns for
|
||||
// a pool that has never allocated. Returning an empty record instead made the
|
||||
// fake disagree with the real store on the one case every pool starts in.
|
||||
func (m *MemStore) Get(_ context.Context, poolName string) (Record, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if _, ok := m.records[poolName]; !ok {
|
||||
return Record{}, fmt.Errorf("%w: %s", kvstore.ErrNotFound, poolName)
|
||||
}
|
||||
return m.copyLocked(poolName), nil
|
||||
}
|
||||
|
||||
// copyLocked returns a deep-enough copy that a caller mutating the returned
|
||||
// record cannot reach into the stored one — the failure a map would otherwise
|
||||
// hand out by reference, making a rolled-back Mutate look like it committed.
|
||||
func (m *MemStore) copyLocked(poolName string) Record {
|
||||
rec := Record{Bindings: map[string]Binding{}}
|
||||
maps.Copy(rec.Bindings, m.records[poolName].Bindings)
|
||||
return rec
|
||||
}
|
||||
+94
@@ -0,0 +1,94 @@
|
||||
package ocinet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/host"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/topology"
|
||||
)
|
||||
|
||||
// FromPoolConfig builds a live allocator for one source="oci" pool: API-key
|
||||
// auth from an ~/.oci/config profile, the VNIC resolved from the host interface
|
||||
// when the config names one rather than an OCID, and the JetStream-backed
|
||||
// bindings store.
|
||||
//
|
||||
// v1 authenticates from the config file rather than as an instance principal.
|
||||
// It needs no dynamic group and no IAM policy from a tenancy admin, and it
|
||||
// reads no instance metadata — which matters, because Spinifex's own IMDS
|
||||
// endpoints claim 169.254.169.254 on the host and take the cloud's metadata
|
||||
// service down with it. Configure oci_vnic_id rather than oci_vnic_iface for
|
||||
// the same reason: resolving by interface is a metadata lookup.
|
||||
//
|
||||
// It does not reconcile. The caller decides when that pass runs, because it
|
||||
// deletes OCI objects and a startup path that does so before the store is
|
||||
// readable would collect live addresses.
|
||||
// sbAddr is the OVN Southbound address the gateway-chassis question is asked
|
||||
// of; empty uses the local socket.
|
||||
func FromPoolConfig(ctx context.Context, js jetstream.JetStream, pool external.ExternalPoolConfig, sbAddr string) (*PoolAllocator, error) {
|
||||
if !pool.IsOCI() {
|
||||
return nil, fmt.Errorf("ocinet: pool %q has source %q, not %q", pool.Name, pool.Source, external.SourceOCI)
|
||||
}
|
||||
|
||||
client, err := oci.NewConfigFileClient(pool.OCIConfigFile, pool.OCIConfigProfile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ocinet: pool %q: %w", pool.Name, err)
|
||||
}
|
||||
|
||||
vnicID, err := resolveVNICID(ctx, pool)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
compartmentID := pool.OCICompartmentID
|
||||
if compartmentID == "" {
|
||||
// Config validation requires the key, so this is the belt to that
|
||||
// brace rather than a supported configuration.
|
||||
inst, mdErr := oci.NewMetadataClient().Instance(ctx)
|
||||
if mdErr != nil {
|
||||
return nil, fmt.Errorf("ocinet: pool %q: no oci_compartment_id and metadata lookup failed: %w", pool.Name, mdErr)
|
||||
}
|
||||
compartmentID = inst.CompartmentID
|
||||
}
|
||||
|
||||
return New(client, NewKVStore(js), Config{
|
||||
Pool: pool,
|
||||
VNICID: vnicID,
|
||||
CompartmentID: compartmentID,
|
||||
// Local OVS decides which addresses this node should hold, so the
|
||||
// affinity pass asks the host rather than any shared record.
|
||||
LocalPorts: func(ctx context.Context) (map[string]struct{}, error) {
|
||||
return host.ListLocalPorts(ctx, host.NewExecRunner())
|
||||
},
|
||||
// The same question for an address with no guest behind it: a NAT
|
||||
// gateway's belongs wherever its VPC's chassisredirect port is claimed,
|
||||
// which the local ovn-controller's own binding answers.
|
||||
LocalGateway: func(ctx context.Context, vpcID string) (bool, error) {
|
||||
return host.NewGatewayClaimProber(sbAddr).
|
||||
GatewayPortLocal(ctx, topology.GatewayChassisRedirectPort(vpcID))
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// resolveVNICID turns whichever VNIC key the operator set into an OCID.
|
||||
func resolveVNICID(ctx context.Context, pool external.ExternalPoolConfig) (string, error) {
|
||||
if pool.OCIVNICID != "" {
|
||||
return pool.OCIVNICID, nil
|
||||
}
|
||||
if pool.OCIVNICIface == "" {
|
||||
return "", fmt.Errorf("ocinet: pool %q sets neither oci_vnic_id nor oci_vnic_iface", pool.Name)
|
||||
}
|
||||
vnic, err := oci.ResolveVNICByInterface(ctx, oci.NewMetadataClient(), pool.OCIVNICIface)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("ocinet: pool %q: %w", pool.Name, err)
|
||||
}
|
||||
slog.InfoContext(ctx, "ocinet resolved the external VNIC from its interface",
|
||||
"pool", pool.Name, "iface", pool.OCIVNICIface, "vnic_id", vnic.VNICID,
|
||||
"private_ip", vnic.PrivateIP, "subnet", vnic.SubnetCIDRBlock)
|
||||
return vnic.VNICID, nil
|
||||
}
|
||||
+17
@@ -31,6 +31,10 @@ const (
|
||||
// allocator can reserve the gateway slot without importing handlers.
|
||||
// The string value is frozen by migration 002 — do not change it.
|
||||
purposeIGWLRP = "igw-lrp"
|
||||
|
||||
// purposeEIP duplicates handlers/ec2/vpc.PurposeEIP, for the same reason
|
||||
// and under the same freeze.
|
||||
purposeEIP = "eip"
|
||||
)
|
||||
|
||||
// ExternalIPAllocation describes how an external IP is being used.
|
||||
@@ -185,6 +189,19 @@ func (a *StaticPoolAllocator) Release(ctx context.Context, poolName string, ip n
|
||||
if alloc.Purpose == purposeIGWLRP {
|
||||
return false, fmt.Errorf("cannot release gateway IP %s in pool %s", target, poolName)
|
||||
}
|
||||
// An EIP outlives every interface it is ever associated with, so only an
|
||||
// unscoped release — ReleaseAddress, which has no interface to name — may
|
||||
// free one. The ownership guard above cannot cover this: an EIP is
|
||||
// allocated bare and associated later, so the stored ENI is empty and the
|
||||
// guard never fires. Without this, terminating an instance hands the
|
||||
// customer's Elastic IP back to the pool.
|
||||
// AllocationID is set only for an EIP, so it stands in when a caller
|
||||
// allocated without naming a purpose.
|
||||
if ownerENIID != "" && (alloc.Purpose == purposeEIP || alloc.AllocationID != "") {
|
||||
slog.InfoContext(ctx, "external IPAM release skip — address belongs to an Elastic IP allocation",
|
||||
"pool", poolName, "ip", target, "allocation_id", alloc.AllocationID, "owner_eni", ownerENIID)
|
||||
return false, nil
|
||||
}
|
||||
|
||||
delete(record.Allocated, target)
|
||||
released = true
|
||||
|
||||
Vendored
+23
@@ -24,6 +24,19 @@ type ExternalPoolConfig struct {
|
||||
AZ string
|
||||
GwLrpRangeStart string
|
||||
GwLrpRangeEnd string
|
||||
// OCICompartmentID, OCIVNICID and OCIVNICIface identify where the provider
|
||||
// creates addresses (Source="oci"). Exactly one of the VNIC fields is set;
|
||||
// the interface name is resolved to an OCID against IMDS at startup.
|
||||
OCICompartmentID string
|
||||
OCIVNICID string
|
||||
OCIVNICIface string
|
||||
OCISubnetID string
|
||||
OCIPublicIPPool string
|
||||
// OCIConfigFile and OCIConfigProfile select the API-key credentials.
|
||||
// v1 authenticates from a config file rather than as an instance
|
||||
// principal: no dynamic group, no IAM policy, and no metadata read.
|
||||
OCIConfigFile string
|
||||
OCIConfigProfile string
|
||||
}
|
||||
|
||||
// IsDHCP reports whether the pool sources IPs from an upstream DHCP server.
|
||||
@@ -31,11 +44,21 @@ func (p *ExternalPoolConfig) IsDHCP() bool {
|
||||
return p != nil && p.Source == SourceDHCP
|
||||
}
|
||||
|
||||
// IsOCI reports whether the pool sources IPs from the Oracle Cloud API.
|
||||
func (p *ExternalPoolConfig) IsOCI() bool {
|
||||
return p != nil && p.Source == SourceOCI
|
||||
}
|
||||
|
||||
const (
|
||||
// SourceStatic is the default pool source (inline range math, KV-backed).
|
||||
SourceStatic = "static"
|
||||
// SourceDHCP delegates allocation to vpcd's DHCPManager.
|
||||
SourceDHCP = "dhcp"
|
||||
// SourceOCI delegates allocation to the Oracle Cloud API. An OCI VNIC
|
||||
// drops any source address that is not a registered private IP object on
|
||||
// it, so the addresses have to be created through the provider rather than
|
||||
// computed from a range.
|
||||
SourceOCI = "oci"
|
||||
|
||||
// DHCPMACDerived leases with deterministic per-client-id 02:xx MACs.
|
||||
DHCPMACDerived = "derived"
|
||||
|
||||
@@ -74,32 +74,55 @@ func eipForwardRules(eip string) [][]string {
|
||||
}
|
||||
}
|
||||
|
||||
// EnsureEIPIngress makes an EIP reachable from the uplink LAN in routed-NAT
|
||||
// mode: a /32 host route steers the EIP into OVN via the VPC gateway LRP's
|
||||
// transit IP, a proxy-ARP neighbor entry answers ARP for the EIP on the
|
||||
// uplink (L3 only — same MAC, so WiFi-safe), and per-EIP FORWARD accepts
|
||||
// cover DROP-policy hosts. The route carries `src <uplink-IP>` so host-local
|
||||
// traffic to the EIP still DNATs (host source would match the exempt set).
|
||||
// Idempotent; call on every EIP bind and reconcile pass.
|
||||
func EnsureEIPIngress(ctx context.Context, r Runner, eip, gwLrpIP, poolGateway, uplinkHint string) error {
|
||||
if eip == "" {
|
||||
return fmt.Errorf("EnsureEIPIngress: eip required")
|
||||
}
|
||||
if gwLrpIP == "" {
|
||||
return fmt.Errorf("EnsureEIPIngress: gwLrpIP required")
|
||||
}
|
||||
// EIPIngress is one EIP's host-side plumbing request in routed-NAT mode.
|
||||
type EIPIngress struct {
|
||||
// EIP is the address as it appears on the transit segment.
|
||||
EIP string
|
||||
|
||||
uplink, srcIP, err := resolveUplink(ctx, r, poolGateway, uplinkHint)
|
||||
// GwLrpIP is the VPC gateway LRP's transit address, the next hop for a
|
||||
// centralised rule. Empty means the rule is distributed: OVN answers ARP
|
||||
// for the EIP on this node's own transit veth with the instance's ENI MAC,
|
||||
// so the route is on-link and no VPC-wide address is involved.
|
||||
GwLrpIP string
|
||||
|
||||
// PoolGateway and UplinkHint locate the LAN-facing interface that has to
|
||||
// proxy-ARP for the EIP. Static pools route toward the gateway; dhcp pools
|
||||
// DORA on the uplink itself and name it directly.
|
||||
PoolGateway string
|
||||
UplinkHint string
|
||||
}
|
||||
|
||||
// EnsureEIPIngress makes an EIP reachable from the uplink LAN in routed-NAT
|
||||
// mode: a /32 host route steers the EIP into OVN across the transit veth, a
|
||||
// proxy-ARP neighbor entry answers ARP for the EIP on the uplink (L3 only —
|
||||
// same MAC, so WiFi-safe), and per-EIP FORWARD accepts cover DROP-policy
|
||||
// hosts. The route carries `src <uplink-IP>` so host-local traffic to the EIP
|
||||
// still DNATs (host source would match the exempt set).
|
||||
// Idempotent; call on every EIP bind and reconcile pass.
|
||||
func EnsureEIPIngress(ctx context.Context, r Runner, in EIPIngress) error {
|
||||
if in.EIP == "" {
|
||||
return fmt.Errorf("EnsureEIPIngress: EIP required")
|
||||
}
|
||||
eip := in.EIP
|
||||
|
||||
uplink, srcIP, err := resolveUplink(ctx, r, in.PoolGateway, in.UplinkHint)
|
||||
if err != nil {
|
||||
return fmt.Errorf("EnsureEIPIngress %s: %w", eip, err)
|
||||
}
|
||||
|
||||
routeArgs := []string{"route", "replace", eip + "/32", "via", gwLrpIP, "dev", NATTransitHostEnd}
|
||||
// A distributed rule is answered on this node's transit segment, so the
|
||||
// EIP is on-link and naming a next hop would send it to a gateway LRP that
|
||||
// may be on another chassis entirely. A centralised rule still hops there.
|
||||
routeArgs := []string{"route", "replace", eip + "/32"}
|
||||
if in.GwLrpIP != "" {
|
||||
routeArgs = append(routeArgs, "via", in.GwLrpIP)
|
||||
}
|
||||
routeArgs = append(routeArgs, "dev", NATTransitHostEnd)
|
||||
if srcIP != "" {
|
||||
routeArgs = append(routeArgs, "src", srcIP)
|
||||
}
|
||||
if out, err := r.Run(ctx, "ip", routeArgs...); err != nil {
|
||||
return fmt.Errorf("install EIP route %s via %s: %s: %w", eip, gwLrpIP, string(out), err)
|
||||
return fmt.Errorf("install EIP route %s (next hop %q): %s: %w", eip, in.GwLrpIP, string(out), err)
|
||||
}
|
||||
|
||||
if uplink != "" {
|
||||
@@ -116,16 +139,30 @@ func EnsureEIPIngress(ctx context.Context, r Runner, eip, gwLrpIP, poolGateway,
|
||||
}
|
||||
}
|
||||
|
||||
// Insert at the head, never append: a distro's catch-all REJECT in FORWARD
|
||||
// (Oracle's cloud image ships one) precedes anything appended, so the EIP
|
||||
// is reachable on the wire and dropped by the host. Probe first rather than
|
||||
// drain-and-reinsert as the egress rules do — this runs on every bind and
|
||||
// reconcile, and re-inserting would blip a live EIP.
|
||||
for _, spec := range eipForwardRules(eip) {
|
||||
if _, err := r.Run(ctx, "iptables", natRuleArgs("-C", "filter", "FORWARD", spec)...); err == nil {
|
||||
continue
|
||||
}
|
||||
if out, err := r.Run(ctx, "iptables", natRuleArgs("-A", "filter", "FORWARD", spec)...); err != nil {
|
||||
if out, err := r.Run(ctx, "iptables", natInsertArgs("filter", "FORWARD", spec)...); err != nil {
|
||||
return fmt.Errorf("install EIP FORWARD rule for %s: %s: %w", eip, string(out), err)
|
||||
}
|
||||
}
|
||||
|
||||
slog.Info("EIP ingress installed", "eip", eip, "gw_lrp_ip", gwLrpIP, "uplink", uplink, "src", srcIP)
|
||||
// A reply leaves with the EIP as its source, so on a host that source-routes
|
||||
// its uplink the EIP needs the same rule. Without it the reply takes the
|
||||
// default route, and a cloud VNIC that enforces its registered addresses
|
||||
// drops it — ingress reaches the guest and nothing comes back.
|
||||
if err := EnsureEIPSourceRoute(ctx, r, eip); err != nil {
|
||||
return fmt.Errorf("EnsureEIPIngress %s: %w", eip, err)
|
||||
}
|
||||
|
||||
slog.Info("EIP ingress installed", "eip", eip, "gw_lrp_ip", in.GwLrpIP,
|
||||
"distributed", in.GwLrpIP == "", "uplink", uplink, "src", srcIP)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -186,6 +223,10 @@ func RemoveEIPIngress(ctx context.Context, r Runner, eip, poolGateway, uplinkHin
|
||||
}
|
||||
}
|
||||
|
||||
if err := RemoveEIPSourceRoute(ctx, r, eip); err != nil {
|
||||
slog.Debug("host: EIP source route not removed", "eip", eip, "err", err)
|
||||
}
|
||||
|
||||
slog.Info("EIP ingress removed", "eip", eip)
|
||||
return nil
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user