Compare commits

..

5 Commits

Author SHA1 Message Date
brynmailer-mdc 221f323e13 docs: say the identity policy grants ocibsd by default 2026-11-13 20:26:54 +10:00
brynmailer-mdc f709226197 test: run the OCI harness test under sh, as the shell suites do 2026-11-13 16:02:10 +10:00
brynmailer-mdc d4c1cc3bda docs: describe ocibsd on every OCI instance and an ocibsd-only nightly 2026-11-13 15:58:06 +10:00
brynmailer-mdc 861254ea52 test: run every OCI nightly topology on ocibsd
The -ocibsd twins are retired to --destroy-only and every topology, bm included, detects ocibsd; --ebs-provider viperblockd keeps a one-off viperblock run. Every node attaches its own data volume over iSCSI with agent login whatever the provider, and the Terraform and identity defaults are ocibsd.
2026-11-13 15:58:06 +10:00
brynmailer-mdc cbe22d84b0 feat: select ocibsd on every OCI instance, bare metal included
Detection and setup.sh now treat any shape the OCI metadata service answers as an OCI instance, so a BM.* node installs the initiator, stages the ocibsd files and gets ocibsd at init. ocibsd's attachments ask the Oracle Cloud Agent not to log them in, and a bare-metal boot session survives the stale-session logout.
2026-11-13 15:58:06 +10:00
21 changed files with 419 additions and 356 deletions
@@ -10,7 +10,7 @@ on:
workflow_dispatch:
inputs:
topologies:
description: "Space-separated topologies (blank = vm-single vm-multi bm vm-single-principal vm-single-ocibsd vm-multi-ocibsd). One name runs one topology."
description: "Space-separated topologies (blank = vm-single vm-multi bm vm-single-principal), each on ocibsd. One name runs one topology."
required: false
workbooks:
description: "Space-separated workbooks (blank = the driver's default list)"
@@ -238,7 +238,7 @@ jobs:
# always(), and before the artifact upload: an OCI bare-metal host left
# running by a cancelled or timed-out job is the most expensive way this
# workflow can fail. It destroys by state file, then deletes OCI volumes and
# backups by the cluster tag each ocibsd topology recorded.
# backups by the cluster tag each topology recorded.
- name: Sweep anything left behind
if: always() && !inputs.keep_on_fail
working-directory: spinifex/scripts/terraform/oci-spx
+1 -1
View File
@@ -302,7 +302,7 @@ NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh sc
# suite rather than the first, and -Otarget keeps each suite's output together.
SHELL_SUITES := $(addprefix shell-suite/,$(sort $(shell find build/scripts -name '*_test.sh'; \
find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh'; \
find scripts/terraform/latitude-spx .github/scripts -maxdepth 1 -name '*_test.sh')))
find scripts/terraform/latitude-spx scripts/terraform/oci-spx .github/scripts -maxdepth 1 -name '*_test.sh')))
.PHONY: $(SHELL_SUITES)
$(SHELL_SUITES): shell-suite/%:
@t=$*; log=$$(mktemp); trap 'rm -f "$$log"' EXIT; $(_RUNSH)
+11 -8
View File
@@ -777,7 +777,7 @@ oci limits resource-availability get --service-name vcn \
## EBS on OCI Block Volumes
On an OCI virtual machine, EBS volumes can be backed by OCI Block Volumes through the `ocibsd` provider instead of Viperblock. Each EBS volume is one OCI block volume, attached over iSCSI to the node that runs the instance using it.
On an OCI instance, virtual machine or bare metal, EBS volumes are backed by OCI Block Volumes through the `ocibsd` provider instead of Viperblock. Each EBS volume is one OCI block volume, attached over iSCSI to the node that runs the instance using it.
### How the provider is chosen
@@ -785,9 +785,9 @@ There is no flag. `spx admin init` takes the provider from the first of these th
1. The existing `/etc/spinifex/spinifex.toml`, whose provider the node keeps; a node table that omits it is `viperblockd`.
2. The `SPINIFEX_EBS_PROVIDER` environment variable: `viperblockd`, `qemunbd` or `ocibsd`.
3. The OCI instance metadata service (IMDS v2), read with a one-second timeout. An OCI VM shape selects `ocibsd`; a `BM.*` shape, or no answer, selects `viperblockd`.
3. The OCI instance metadata service (IMDS v2), read with a one-second timeout. Any OCI shape, `VM.*` or `BM.*`, selects `ocibsd`; no answer selects `viperblockd`.
Init prints its choice and where it came from, for example `EBS provider: ocibsd (detected: OCI virtual machine <shape>)`. Joining nodes take the provider from formation rather than choosing their own. The choice is written once for every node and cannot be changed without wiping and re-forming the cluster.
Init prints its choice and where it came from, for example `EBS provider: ocibsd (detected: OCI instance <shape>)`. Joining nodes take the provider from formation rather than choosing their own. The choice is written once for every node and cannot be changed without wiping and re-forming the cluster.
For `ocibsd`, init fills the compartment and availability domain from instance metadata unless the existing config names them, generates a cluster ID that tags every OCI volume and backup the cluster makes, and defaults `auth` to `instance_principal` when the existing config names no credential. It then reads the compartment's volumes with that credential, and stops with an error naming the needed policy, `manage volume-family` and `use instances`, if the read is refused. It never falls back to Viperblock.
@@ -804,15 +804,18 @@ cluster_id = "<generated-cluster-id>"
auth = "instance_principal"
```
On an OCI VM, `setup.sh` installs the iSCSI initiator (`open-iscsi` with apt, `iscsi-initiator-utils` with dnf) and stages the `ocibsd` files under `/usr/local/share/spinifex/ocibsd`. When the provider resolves to `ocibsd`, init and join install the `spinifex-ocibsd.service` unit, its udev rule and a `spinifex-daemon` drop-in, mask `spinifex-viperblock.service`, and enable `spinifex-ocibsd.service`, so exactly one service answers for EBS on the node.
On an OCI instance, `setup.sh` installs the iSCSI initiator (`open-iscsi` with apt, `iscsi-initiator-utils` with dnf) and stages the `ocibsd` files under `/usr/local/share/spinifex/ocibsd`. When the provider resolves to `ocibsd`, init and join install the `spinifex-ocibsd.service` unit, its udev rule and a `spinifex-daemon` drop-in, mask `spinifex-viperblock.service`, and enable `spinifex-ocibsd.service`, so exactly one service answers for EBS on the node.
To keep Viperblock on an OCI VM, name it before init:
On bare metal the instance boots from an iSCSI boot volume, so the node already holds a session to an `iqn.2015-02.oracle.boot` target; `ocibsd` only ever logs out sessions to its own cluster's volumes and leaves that one alone.
`ocibsd` attaches its volumes with the Oracle Cloud Agent's iSCSI login turned off, so the agent's Block Volume Management plugin can stay enabled for the instance's other attachments.
To keep Viperblock on an OCI instance, name it before init:
```bash
sudo env SPINIFEX_EBS_PROVIDER=viperblockd spx admin init <your usual flags>
```
`validate-topology.sh` does this for every topology except `vm-single-ocibsd` and `vm-multi-ocibsd`, which rely on detection and need `--distro`.
`validate-topology.sh` does this only with `--ebs-provider viperblockd`; by default every topology relies on detection and needs `--distro`. Build such nodes with `ebs_provider = "viperblockd"` in your tfvars, since the Terraform default is `ocibsd`.
### One availability domain per cluster
@@ -827,7 +830,7 @@ Allow dynamic-group <dynamic-group-name> to manage volume-family in compartment
Allow dynamic-group <dynamic-group-name> to use instances in compartment id <compartment-ocid>
```
`manage volume-family` covers volumes, backups and attachments, and `use instances` covers attaching them to the node. The instance-principal policy that `--setup-instance-principal` on an `ocibsd` topology (or `setup-identity.sh --ebs-provider ocibsd`) creates grants both, for a Dynamic Group matching `ALL {instance.compartment.id = '<compartment-ocid>'}`. Built for `viperblockd`, the default, it grants neither, and because the policy is shared by every deployment in the compartment, re-running `setup-identity.sh` without the flag removes them.
`manage volume-family` covers volumes, backups and attachments, and `use instances` covers attaching them to the node. The instance-principal policy that `--setup-instance-principal` or `setup-identity.sh` creates grants both by default, for a Dynamic Group matching `ALL {instance.compartment.id = '<compartment-ocid>'}`. Built with `--ebs-provider viperblockd` it grants neither, and because the policy is shared by every deployment in the compartment, that run removes them. A policy created while `viperblockd` was the default lacks them too, so `spx admin init` refuses the instance principal until `setup-identity.sh` is run again.
For an API key, grant the same two verbs to the key user's group, alongside the four from [section 2.2](#grant-the-group-its-four-permissions):
@@ -836,7 +839,7 @@ Allow group <group-name> to manage volume-family in compartment id <compartment-
Allow group <group-name> to use instances in compartment id <compartment-ocid>
```
`terraform output -json api_key_policy_statements` prints all six for the group named by `api_key_group_name` (default `spinifex-nodes`) when `ebs_provider = "ocibsd"`, and only the four address grants otherwise. Because init defaults to the instance principal, an API-key node needs its credential named in `spinifex.toml` before init, which init then keeps:
`terraform output -json api_key_policy_statements` prints all six for the group named by `api_key_group_name` (default `spinifex-nodes`) when `ebs_provider = "ocibsd"`, the default, and only the four address grants with `ebs_provider = "viperblockd"`. Because init defaults to the instance principal, an API-key node needs its credential named in `spinifex.toml` before init, which init then keeps:
```toml
[nodes.<node>.ebs]
+14 -16
View File
@@ -1393,22 +1393,20 @@ install_apt_deps() {
fi
}
# is_oci_vm is spx admin init's detection: the OCI metadata service answers
# within a second with a shape that is not bare metal. Asked once per run,
# because off OCI each ask costs the full second.
is_oci_vm() {
if [ -z "${OCI_VM:-}" ]; then
# is_oci_instance is spx admin init's detection: the OCI metadata service
# answers within a second with a shape, virtual machine or bare metal. Asked
# once per run, because off OCI each ask costs the full second.
is_oci_instance() {
if [ -z "${OCI_INSTANCE:-}" ]; then
local shape=""
OCI_VM=no
OCI_INSTANCE=no
if command -v curl >/dev/null 2>&1 \
&& shape=$(curl -sf -m 1 -H 'Authorization: Bearer Oracle' "$OCI_IMDS_SHAPE_URL" 2>/dev/null); then
case "$shape" in
"" | BM.*) ;;
*) OCI_VM=yes ;;
esac
&& shape=$(curl -sf -m 1 -H 'Authorization: Bearer Oracle' "$OCI_IMDS_SHAPE_URL" 2>/dev/null) \
&& [ -n "$shape" ]; then
OCI_INSTANCE=yes
fi
fi
[ "$OCI_VM" = yes ]
[ "$OCI_INSTANCE" = yes ]
}
# node_runs_ocibsd reports whether this node's existing config names ocibsd in
@@ -1429,8 +1427,8 @@ node_runs_ocibsd() {
# The initiator ocibsd drives, on the only host it can back EBS on.
install_iscsi_initiator() {
is_oci_vm || return 0
info "OCI virtual machine detected, installing the iSCSI initiator..."
is_oci_instance || return 0
info "OCI instance detected, installing the iSCSI initiator..."
case "$1" in
apt) DEBIAN_FRONTEND=noninteractive $SUDO apt-get install -y -qq open-iscsi > /dev/null ;;
dnf) $SUDO dnf install -y -q iscsi-initiator-utils > /dev/null ;;
@@ -2190,14 +2188,14 @@ install_udev() {
info "udev rules installed"
}
# install_ocibsd stages the tarball's ocibsd/ on an OCI virtual machine, where
# install_ocibsd stages the tarball's ocibsd/ on an OCI instance, where
# spx admin init or join installs it once the provider resolves to ocibsd. On
# a node whose config already runs ocibsd it also reinstalls the staged files.
install_ocibsd() {
stage "staging the ocibsd files"
local ocibsd_node=false f
node_runs_ocibsd && ocibsd_node=true
if ! $ocibsd_node && ! is_oci_vm; then
if ! $ocibsd_node && ! is_oci_instance; then
return 0
fi
if [ ! -d "$EXTRACT_DIR/ocibsd" ]; then
+20 -6
View File
@@ -1,6 +1,6 @@
#!/bin/sh
# setup_ocibsd_test.sh — setup.sh's systemd and udev stages never install the
# tarball's ocibsd/ files; install_ocibsd stages them on an OCI VM, and
# tarball's ocibsd/ files; install_ocibsd stages them on an OCI instance, and
# reinstalls them on upgrade only where the node's config already runs ocibsd.
# A stub SUDO records every privileged command and runs the file ones under a
# scratch root, so what each case leaves on disk is checked as well.
@@ -44,7 +44,7 @@ exec "${args[@]}"
EOF
chmod +x "$TMP/sudo"
# run_setup <case> <oci_vm yes|no> <config body or -> <functions...>
# run_setup <case> <oci_instance yes|no> <config body or -> <functions...>
# runs the named setup.sh functions in a fresh root for one case.
run_setup() {
name="$1" vm="$2" body="$3"
@@ -56,7 +56,7 @@ run_setup() {
[ "$body" = - ] || printf '%s\n' "$body" > "$ROOT/etc/spinifex/spinifex.toml"
export ROOT CALLS
OUT=$(INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '
. "$1"; SUDO="$2"; EXTRACT_DIR="$3"; OCI_VM="$4"; SPINIFEX_TOML="$ROOT/etc/spinifex/spinifex.toml"
. "$1"; SUDO="$2"; EXTRACT_DIR="$3"; OCI_INSTANCE="$4"; SPINIFEX_TOML="$ROOT/etc/spinifex/spinifex.toml"
shift 4
for f in "$@"; do $f; done' _ "$SETUP" "$TMP/sudo" "$EXTRACT" "$vm" "$@" 2>&1) \
|| fail "$name: $* failed: $OUT"
@@ -91,7 +91,7 @@ staged_ok() {
done
}
echo "-- a fresh install on an OCI VM stages ocibsd/ and installs none of it"
echo "-- a fresh install on an OCI instance stages ocibsd/ and installs none of it"
run_setup fresh yes - install_systemd install_udev install_ocibsd
if staged_ok; then pass "the three files are staged"; else fail "the stage does not match the tarball: $(ls -R "$ROOT/$STAGED" 2>&1)"; fi
[ "$(installed_count)" = 0 ] && pass "nothing is installed" || fail "installed $(installed_count) ocibsd file(s) on a node init has not resolved"
@@ -131,7 +131,7 @@ echo "-- an ocibsd node is refreshed even when the metadata service does not ans
run_setup upgrade-no-imds no "$OCIBSD_TOML" install_ocibsd
[ "$(installed_count)" = 3 ] && pass "the three files are installed" || fail "installed $(installed_count) of 3"
echo "-- an upgrade of a viperblock node on an OCI VM installs none of them"
echo "-- an upgrade of a viperblock node on an OCI instance installs none of them"
run_setup viperblock yes '[nodes.node1.ebs]
provider = "viperblockd"' install_systemd install_udev install_ocibsd
[ "$(installed_count)" = 0 ] && pass "nothing is installed" || fail "installed $(installed_count) ocibsd file(s) on a viperblock node"
@@ -142,7 +142,7 @@ echo "-- off OCI nothing is staged"
run_setup off-oci no '[nodes.node1.daemon]' install_ocibsd
[ ! -e "$ROOT/$STAGED" ] && pass "no stage" || fail "staged ocibsd/ off OCI"
echo "-- the initiator is installed on an OCI VM only"
echo "-- the initiator is installed on an OCI instance only"
run_setup initiator-apt yes - "install_iscsi_initiator apt"
grep -q 'apt-get install -y -qq open-iscsi' "$CALLS" && grep -q '^systemctl enable --now iscsid.socket$' "$CALLS" \
&& pass "apt installs open-iscsi and enables iscsid.socket" || fail "apt: $(cat "$CALLS")"
@@ -151,6 +151,20 @@ grep -q 'dnf install -y -q iscsi-initiator-utils' "$CALLS" && pass "dnf installs
run_setup initiator-off no - "install_iscsi_initiator apt"
[ -s "$CALLS" ] && fail "installed an initiator off OCI: $(cat "$CALLS")" || pass "no initiator off OCI"
echo "-- detection takes any shape the metadata service answers, bare metal included"
mkdir -p "$TMP/bin"
printf '#!/bin/sh\n[ -n "$SHAPE" ] || exit 22\nprintf %%s "$SHAPE"\n' > "$TMP/bin/curl"
chmod +x "$TMP/bin/curl"
detect() {
SHAPE="$1" PATH="$TMP/bin:$PATH" INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '
. "$1"; unset OCI_INSTANCE
if is_oci_instance; then echo yes; else echo no; fi' _ "$SETUP" 2>/dev/null
}
for shape in VM.Standard.E6.Flex BM.Standard.E2.64; do
[ "$(detect "$shape")" = yes ] && pass "$shape is an OCI instance" || fail "$shape was not detected"
done
[ "$(detect "")" = no ] && pass "no answer is not an OCI instance" || fail "detected an OCI instance with no metadata answer"
if [ "$fails" -gt 0 ]; then
echo "$fails failure(s)"
exit 1
+7 -5
View File
@@ -230,7 +230,7 @@ Create them once per tenancy, then every deployment and rebuild afterwards uses
**`adopt` references the dynamic group by nothing at all.** Its matching rule is `instance.compartment.id`, so it covers every instance in the compartment and names no OCID — which is why adopting needs no read on an identity resource and no tenancy rights. The cost is that a missing policy is invisible at apply time: the node forms, passes every health check, and then refuses every launch wanting a public address. The allocator gate in `validate-topology.sh` is what catches that, by requiring `ocinet credential authorised to allocate` in each node's `spinifex-daemon` journal.
The policy grants `use vnics`, `manage private-ips`, `manage public-ips` and `use subnets` in one compartment for the allocator, and with `ebs_provider = "ocibsd"` also `manage volume-family` and `use instances` for the `ocibsd` EBS provider. `setup-identity.sh --ebs-provider ocibsd` creates it with them; the policy is shared by every deployment in the compartment, so running it without the flag removes them.
The policy grants `use vnics`, `manage private-ips`, `manage public-ips` and `use subnets` in one compartment for the allocator, and with `ebs_provider = "ocibsd"`, the default, also `manage volume-family` and `use instances` for the `ocibsd` EBS provider. `setup-identity.sh` creates it with them unless given `--ebs-provider viperblockd`; the policy is shared by every deployment in the compartment, so a `viperblockd` run removes them.
The first four are exactly what allocating an external address does and nothing more; the last two are what creating, attaching, backing up and deleting a block volume does.
`use subnets` looks unrelated and is not: `CreatePrivateIp` is checked against `SUBNET_ATTACH` and `CreatePrivateIp` is how an address is registered, so a policy without it authorises nothing and every allocation returns a 404.
@@ -275,17 +275,19 @@ The count that later gates read is the number of addresses the `hosts_file` outp
**A clean teardown can still leave public IPs behind, and they count against the tenancy.** The addresses Spinifex allocates for guests are created by the running node, so they are in no Terraform state and `destroy` neither sees nor removes them. The node's own reconcile collects detached ones every ten minutes, which is exactly the sweep a destroy takes away. Twenty accumulated in `spxbm` over four runs and exhausted the tenancy's 50 reserved-public-IP limit, at which point every later launch in every compartment failed. [Check for leftover public IPs afterwards](../../../docs/oci-integration/README.md#check-for-leftover-public-ips-afterwards) has the query and the rule for reading it.
### The ocibsd topologies
### EBS on OCI Block Volumes
`vm-single-ocibsd` and `vm-multi-ocibsd` are `vm-single` and `vm-multi` with EBS backed by OCI Block Volumes. They build with `ebs_provider = "ocibsd"`, which disables the Block Volume Management plugin and attaches each node's data volume paravirtualized, because every iSCSI session on the node is `ocibsd`'s to manage. They need `--distro`, because a published release predates the `ocibsd` files that `setup.sh` stages, and an external pool, because an API-key run gives `ocibsd` the pool's credential.
Every topology backs EBS with OCI Block Volumes through `ocibsd`, `bm` included. It needs `--distro`, because a published release predates the `ocibsd` files that `setup.sh` stages, and an external pool, because an API-key run gives `ocibsd` the pool's credential. `e2e-oci.sh` records a topology `SKIPPED` without either.
The ocibsd topologies name no EBS provider. `spx admin init` detects that it is on an OCI virtual machine, reads the compartment, availability domain and region from the instance metadata, and generates the cluster's `cluster_id`. Every other topology sets `SPINIFEX_EBS_PROVIDER=viperblockd`, so its viperblock baseline is not replaced by detection on OCI. The run checks the `EBS provider:` line init prints: `detected` on the ocibsd topologies under the instance principal, and `from SPINIFEX_EBS_PROVIDER` on the rest.
No topology names an EBS provider. `spx admin init` detects that it is on an OCI instance, reads the compartment, availability domain and region from the instance metadata, and generates the cluster's `cluster_id`. The run checks the `EBS provider:` line init prints: `detected` under the instance principal. `--ebs-provider viperblockd` on `validate-topology.sh`, or `OCI_EBS_PROVIDER=viperblockd` for `e2e-oci.sh`, runs a one-off viperblock check instead, telling init through `SPINIFEX_EBS_PROVIDER`; no default run uses it. `vm-single-ocibsd` and `vm-multi-ocibsd` are accepted only with `--destroy-only`, so the sweep can still destroy state an earlier run left under those names.
`ebs_provider` decides only the policy grants. Every node attaches its own data volume over iSCSI with the Block Volume Management plugin logging it in, whatever the provider or shape: bare metal attaches no other way, and `ocibsd`'s own attachments ask the agent not to log them in. `terraform test` in this directory plans that against a mocked provider.
Under the instance principal, `ocibsd` uses the instance's own certificate. An API-key run runs the credential hook before formation instead, and seeds each node's `spinifex.toml` with `provider = "ocibsd"` and the hook's `/etc/spinifex/oci/config`, which init keeps as an existing config, so its `EBS provider:` line reads `from the existing config`.
Every volume and backup `ocibsd` creates is tagged `spinifex-cluster=<cluster_id>`, and none of them is in any Terraform state, so the run records the ID in `.validate-<topology>/ocibsd-clusters` before any volume exists.
The `ebs provider` gate then checks the units, the viperblock mask, the daemon's device drop-in, and that the compartment and availability domain init detected are the ones the state reports on every node, imports an AMI onto OCI and boots from it, and drives stop and start, attach and detach, snapshot and restore with a guest-written marker read back over the serial console. On `vm-multi-ocibsd` it also starts the stopped instance on another node. It ends by terminating everything and polling OCI until nothing tagged with the instance's volumes or snapshot is left.
The `ebs provider` gate then checks the units, the viperblock mask, the daemon's device drop-in, and that the compartment and availability domain init detected are the ones the state reports on every node, imports an AMI onto OCI and boots from it, and drives stop and start, attach and detach, snapshot and restore with a guest-written marker read back over the serial console. On `vm-multi` it also starts the stopped instance on another node. It ends by terminating everything and polling OCI until nothing tagged with the instance's volumes or snapshot is left.
`e2e-oci.sh` deletes everything tagged with each recorded `cluster_id` once that topology's nodes are destroyed, including from `--sweep-only`, and fails the run if anything tagged remains. A standalone `validate-topology.sh` run does not sweep, so follow it with `./e2e-oci.sh --sweep-only`.
+7 -9
View File
@@ -45,10 +45,6 @@ locals {
# after the boot disk (mulga-poc has oraclevda1 pointing at sdb1).
data_device = "/dev/oracleoci/oraclevdb"
# The Block Volume Management plugin logs iSCSI sessions in and out by itself,
# and on an ocibsd node those sessions are ocibsd's to manage. With the plugin
# off, the node's data volume attaches paravirtualized, which needs no login.
ocibsd_nodes = var.ebs_provider == "ocibsd"
# adopt and create differ only in who creates the dynamic group and policy; a node
# authenticates identically either way, because the matching rule covers the whole
@@ -111,12 +107,12 @@ resource "oci_core_instance" "mulgadc" {
user_data = base64encode(local.user_data)
}
# Required for automatic UHP iSCSI multipath setup and login, and disabled on
# an ocibsd node, where ocibsd owns every iSCSI session (see ocibsd_nodes).
# Logs in the data volume below, the one attachment that asks it to; ocibsd's
# attachments do not, so the plugin leaves their sessions to ocibsd.
agent_config {
plugins_config {
name = "Block Volume Management"
desired_state = local.ocibsd_nodes ? "DISABLED" : "ENABLED"
desired_state = "ENABLED"
}
}
@@ -171,10 +167,12 @@ resource "oci_core_volume" "mulgadc_data" {
resource "oci_core_volume_attachment" "mulgadc_data" {
count = var.node_count
attachment_type = local.ocibsd_nodes ? "paravirtualized" : "iscsi"
# iSCSI on every node and provider: bare metal attaches no other way, and
# ebs_provider must not change how a deployment's own data volume attaches.
attachment_type = "iscsi"
# Required by OCI for Ultra High Performance iSCSI volume attachments.
device = local.data_device
is_agent_auto_iscsi_login_enabled = local.ocibsd_nodes ? null : true
is_agent_auto_iscsi_login_enabled = true
instance_id = oci_core_instance.mulgadc[count.index].id
volume_id = oci_core_volume.mulgadc_data[count.index].id
}
+10 -8
View File
@@ -8,10 +8,11 @@
#
# SPX_REF Ref to prove. Default: the spinifex checkout's current branch.
# OCI_TOPOLOGIES Space-separated. Default: "vm-single vm-multi bm
# vm-single-principal vm-single-ocibsd vm-multi-ocibsd". Set
# one name to test one thing. The two ocibsd topologies back
# EBS with OCI Block Volumes, need OCI_SOURCE=tree and an
# external pool, and are recorded SKIPPED without them.
# vm-single-principal". Set one name to test one thing.
# Every topology backs EBS with OCI Block Volumes (ocibsd),
# needs OCI_SOURCE=tree and an external pool, and is
# recorded SKIPPED without them.
# OCI_EBS_PROVIDER viperblockd for a one-off viperblock run; blank is ocibsd.
# OCI_SOURCE tree (build SPX_REF) or release (the published installer).
# OCI_VERSION With OCI_SOURCE=release, install this exact release tag and
# ignore OCI_CHANNEL. Preferred for anything repeatable: a tag
@@ -84,7 +85,8 @@ SPINIFEX_ROOT="$(cd "$HERE/../../.." && pwd)"
# under the workspace. Deriving one from the other is only right in the first case.
MULGA_ROOT="${MULGA_ROOT:-$SPINIFEX_ROOT/..}"
TOPOLOGIES="${OCI_TOPOLOGIES:-vm-single vm-multi bm vm-single-principal vm-single-ocibsd vm-multi-ocibsd}"
TOPOLOGIES="${OCI_TOPOLOGIES:-vm-single vm-multi bm vm-single-principal}"
EBS_PROVIDER="${OCI_EBS_PROVIDER:-ocibsd}"
SOURCE="${OCI_SOURCE:-tree}"
CHANNEL="${OCI_CHANNEL:-latest}"
INSTALL_VERSION="${OCI_VERSION:-}"
@@ -436,8 +438,8 @@ for topology in $TOPOLOGIES; do
fi
# ocibsd's files ship in the distro this checkout builds, and its volumes
# need the credential the pool uses, so neither can be stood in for.
case "$topology" in
*-ocibsd)
case "$EBS_PROVIDER" in
ocibsd)
if [ "$SOURCE" != tree ]; then
log "$topology: SKIPPED -- the ocibsd files ship only with a tree build"
echo "$topology: SKIPPED (OCI_SOURCE=$SOURCE)" >> "$VERDICT"
@@ -450,7 +452,7 @@ for topology in $TOPOLOGIES; do
fi
;;
esac
args=(--topology "$topology" --ssh-public-key "$SSH_PUBLIC_KEY" --ssh-private-key "$SSH_PRIVATE_KEY")
args=(--topology "$topology" --ebs-provider "$EBS_PROVIDER" --ssh-public-key "$SSH_PUBLIC_KEY" --ssh-private-key "$SSH_PRIVATE_KEY")
[ "$DRY_RUN" = 1 ] && args+=(--dry-run)
# Two separate choices. --no-external-pool is the one that forms without any
# allocator, for a tenancy whose dynamic group does not exist yet; it records
+66
View File
@@ -0,0 +1,66 @@
#!/bin/sh
# harness_test.sh — the OCI harness's defaults without a tenancy:
# setup-identity.sh grants ocibsd unless told otherwise, and validate-topology.sh
# accepts the retired -ocibsd topology names only to destroy their state.
set -u
HERE=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
fails=0
pass() { echo " ok - $1"; }
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
echo "-- setup-identity.sh passes ebs_provider=ocibsd when --ebs-provider is omitted"
# A copy beside a stub oci_env.py, which the script runs Terraform through.
mkdir -p "$TMP/identity/scripts"
cp "$HERE/setup-identity.sh" "$TMP/identity/"
printf '%s\n' 'import sys' 'open(sys.argv[0] + ".calls", "a").write(" ".join(sys.argv[1:]) + "\n")' \
> "$TMP/identity/scripts/oci_env.py"
touch "$TMP/key.pub"
run_identity() {
: > "$TMP/identity/scripts/oci_env.py.calls"
OCI_SSH_PUBLIC_KEY="$TMP/key.pub" OCI_STATE_ROOT="$TMP/identity" \
bash "$TMP/identity/setup-identity.sh" --dry-run "$@" > "$TMP/identity/out" 2>&1
}
plan_call() { grep ' plan ' "$TMP/identity/scripts/oci_env.py.calls"; }
if run_identity && plan_call | grep -q -- '-var ebs_provider=ocibsd '; then
pass "the plan grants ocibsd by default"
else
fail "default plan: $(cat "$TMP/identity/out") $(plan_call)"
fi
if run_identity --ebs-provider viperblockd && plan_call | grep -q -- '-var ebs_provider=viperblockd '; then
pass "--ebs-provider viperblockd is passed through"
else
fail "viperblockd plan: $(cat "$TMP/identity/out") $(plan_call)"
fi
echo "-- validate-topology.sh takes a retired topology name only with --destroy-only"
validate() {
OCI_STATE_ROOT="$TMP/state" bash "$HERE/validate-topology.sh" \
--ssh-public-key "$TMP/absent.pub" "$@" 2>&1
}
for name in vm-single-ocibsd vm-multi-ocibsd; do
out=$(validate --topology "$name")
case "$out" in
*"accepted only with --destroy-only; run ${name%-ocibsd}"*) pass "$name alone is refused, naming ${name%-ocibsd}" ;;
*) fail "$name alone: $out" ;;
esac
# Topology validation is what it must get past; the absent key stops it next.
out=$(validate --topology "$name" --destroy-only)
case "$out" in
*"no readable SSH public key"*) pass "$name --destroy-only gets past topology validation" ;;
*) fail "$name --destroy-only: $out" ;;
esac
done
out=$(validate --topology vm-single --ebs-provider qemunbd)
case "$out" in
*"--ebs-provider must be ocibsd or viperblockd"*) pass "an unknown --ebs-provider is refused" ;;
*) fail "--ebs-provider qemunbd: $out" ;;
esac
if [ "$fails" -gt 0 ]; then
echo "$fails failure(s)"
exit 1
fi
+5 -5
View File
@@ -21,7 +21,7 @@ STATE_DIR="${OCI_STATE_ROOT:-$HERE}/.identity"
SSH_PUBLIC_KEY="${OCI_SSH_PUBLIC_KEY:-$HOME/.ssh/oci-spx.pub}"
DESTROY=0
DRY_RUN=0
EBS_PROVIDER=viperblockd
EBS_PROVIDER=ocibsd
log() { printf '[setup-identity] %s\n' "$*"; }
die() {
@@ -31,7 +31,7 @@ die() {
usage() {
cat >&2 <<EOF
usage: ${0##*/} [--dry-run] [--destroy] [--ebs-provider viperblockd|ocibsd]
usage: ${0##*/} [--dry-run] [--destroy] [--ebs-provider ocibsd|viperblockd]
Creates the OCI dynamic group and policy for instance-principal authentication.
Needs a tenancy-admin OCI credential; the compartment-scoped user the rest of this
@@ -39,9 +39,9 @@ configuration runs as cannot create either resource.
--dry-run Show the plan, change nothing.
--destroy Remove them again. Every node using "adopt" loses its credential.
--ebs-provider ocibsd also grants manage volume-family and use instances. The
policy is shared by every deployment in the compartment, so a
later run without it removes those grants. Default viperblockd.
--ebs-provider ocibsd (default) also grants manage volume-family and use
instances. The policy is shared by every deployment in the
compartment, so a later viperblockd run removes those grants.
EOF
exit 2
}
@@ -0,0 +1,120 @@
# What ebs_provider decides, planned against a mocked oci provider:
# terraform init && terraform test, from scripts/terraform/oci-spx.
# It grants the ocibsd volume rights and nothing else; a node's own data
# volume attaches the same way whatever the provider or shape.
mock_provider "oci" {}
mock_provider "oci" {
alias = "home"
}
override_data {
target = data.oci_identity_availability_domains.available
values = { availability_domains = [{ name = "AD-1" }] }
}
override_data {
target = data.oci_core_images.ubuntu_2604
values = { images = [{ id = "ocid1.image.oc1..test" }] }
}
override_data {
target = data.oci_identity_fault_domains.available
values = { fault_domains = [{ name = "FAULT-DOMAIN-1" }] }
}
override_data {
target = data.oci_core_services.oracle_services_network
values = { services = [{ id = "ocid1.service.oc1..test", name = "All Services In Oracle Services Network", cidr_block = "all-services" }] }
}
variables {
tenancy_ocid = "ocid1.tenancy.oc1..test"
user_ocid = "ocid1.user.oc1..test"
fingerprint = "00:00"
compartment_ocid = "ocid1.compartment.oc1..test"
ssh_public_key_path = "tests/test-key.pub"
private_key = "not a key: the provider is mocked"
}
run "ebs_provider_defaults_to_ocibsd_grants" {
command = plan
assert {
condition = output.ebs_provider == "ocibsd"
error_message = "ebs_provider must default to ocibsd, which spx admin init detects on any OCI instance"
}
assert {
condition = alltrue([
for grant in ["manage volume-family", "use instances"] :
anytrue([for s in output.api_key_policy_statements : strcontains(s, " to ${grant} in ")])
])
error_message = "the default policy must carry the ocibsd volume grants"
}
}
run "viperblockd_drops_only_the_volume_grants" {
command = plan
variables {
ebs_provider = "viperblockd"
}
assert {
condition = !anytrue([for s in output.api_key_policy_statements : strcontains(s, "volume-family")])
error_message = "a viperblockd deployment is granted no OCI volume"
}
}
run "data_volume_attaches_over_iscsi_with_agent_login" {
command = plan
variables {
ebs_provider = "ocibsd"
compute_shape = "BM.Standard.E2.64"
}
assert {
condition = oci_core_volume_attachment.mulgadc_data[0].attachment_type == "iscsi" && oci_core_volume_attachment.mulgadc_data[0].is_agent_auto_iscsi_login_enabled
error_message = "bare metal attaches only over iSCSI, with the agent logging the data volume in"
}
assert {
condition = one([for p in oci_core_instance.mulgadc[0].agent_config[0].plugins_config : p.desired_state if p.name == "Block Volume Management"]) == "ENABLED"
error_message = "the Block Volume Management plugin must run to log the data volume in"
}
}
run "viperblockd_vm_attaches_its_data_volume_identically" {
command = plan
variables {
ebs_provider = "viperblockd"
}
assert {
condition = oci_core_volume_attachment.mulgadc_data[0].attachment_type == "iscsi" && oci_core_volume_attachment.mulgadc_data[0].is_agent_auto_iscsi_login_enabled
error_message = "ebs_provider must not change how the data volume attaches"
}
assert {
condition = one([for p in oci_core_instance.mulgadc[0].agent_config[0].plugins_config : p.desired_state if p.name == "Block Volume Management"]) == "ENABLED"
error_message = "ebs_provider must not change the Block Volume Management plugin"
}
}
run "ocibsd_vm_attaches_its_data_volume_identically" {
command = plan
variables {
ebs_provider = "ocibsd"
}
assert {
condition = oci_core_volume_attachment.mulgadc_data[0].attachment_type == "iscsi" && oci_core_volume_attachment.mulgadc_data[0].is_agent_auto_iscsi_login_enabled
error_message = "an ocibsd VM must attach its data volume as a viperblockd one does"
}
assert {
condition = one([for p in oci_core_instance.mulgadc[0].agent_config[0].plugins_config : p.desired_state if p.name == "Block Volume Management"]) == "ENABLED"
error_message = "an ocibsd VM must run the plugin as a viperblockd one does"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFakeKeyForTerraformTestsOnly0000000000000 test
+36 -24
View File
@@ -49,7 +49,8 @@ WORKBOOKS=""
# set -u an unset array is an error rather than an empty one.
HOSTS=()
TOPOLOGIES="bm vm-single vm-multi vm-single-principal vm-single-ocibsd vm-multi-ocibsd"
TOPOLOGIES="bm vm-single vm-multi vm-single-principal"
EBS_PROVIDER=""
# What a topology is worth when nothing says otherwise, so a CI run of a named
# topology is the same every time. Supplied as TF_VAR_, which terraform.auto.tfvars
@@ -62,13 +63,13 @@ TOPOLOGIES="bm vm-single vm-multi vm-single-principal vm-single-ocibsd vm-multi-
# A case rather than an associative array: macOS ships bash 3.2, where declare -A
# is an indexed assignment and the key is read as an arithmetic variable.
#
# TOPO_EBS_PROVIDER is what backs EBS. The -ocibsd topologies are their viperblock
# twins on OCI Block Volumes, so a difference between a pair is the provider's.
# They set no provider and rely on init's detection on an OCI VM; every other
# topology names viperblockd in SPINIFEX_EBS_PROVIDER to keep its baseline.
# TOPO_EBS_PROVIDER is what backs EBS: ocibsd, which init detects on any OCI
# instance, unless --ebs-provider viperblockd names it for a one-off check.
# TOPO_LEGACY marks a retired name kept so the sweep can destroy its state.
topo_defaults() {
TOPO_FORCE_PRINCIPAL=0
TOPO_EBS_PROVIDER=viperblockd
TOPO_EBS_PROVIDER=ocibsd
TOPO_LEGACY=0
case "$1" in
bm) TOPO_DEFAULT_SHAPE="BM.Standard.E2.64"; TOPO_DEFAULT_NODES=1 ;;
vm-single) TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=1 ;;
@@ -81,9 +82,9 @@ topo_defaults() {
TOPO_DEFAULT_NODES=1
TOPO_FORCE_PRINCIPAL=1 ;;
vm-single-ocibsd)
TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=1; TOPO_EBS_PROVIDER=ocibsd ;;
TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=1; TOPO_LEGACY=1 ;;
vm-multi-ocibsd)
TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=3; TOPO_EBS_PROVIDER=ocibsd ;;
TOPO_DEFAULT_SHAPE="VM.Standard.E6.Flex"; TOPO_DEFAULT_NODES=3; TOPO_LEGACY=1 ;;
*) return 1 ;;
esac
}
@@ -96,7 +97,7 @@ die() {
usage() {
cat >&2 <<EOF
usage: ${0##*/} --topology <bm|vm-single|vm-multi|vm-single-principal|vm-single-ocibsd|vm-multi-ocibsd> [options]
usage: ${0##*/} --topology <bm|vm-single|vm-multi|vm-single-principal> [options]
Builds a topology, or installs on existing OCI hosts, then installs Spinifex,
forms the cluster, and verifies the OCI address allocator.
@@ -105,13 +106,16 @@ forms the cluster, and verifies the OCI address allocator.
topology is the easiest expensive mistake here.
vm-single-principal is vm-single authenticating as the
instance, and implies --instance-principal.
vm-single-ocibsd and vm-multi-ocibsd set no EBS
provider, so init detects ocibsd on the OCI VM, and
add the ebs provider gate; the others set
SPINIFEX_EBS_PROVIDER=viperblockd. The ocibsd pair
need --distro. Their OCI volumes and backups are
in no Terraform state: the destroy here leaves any a
failed run made, and e2e-oci.sh sweeps them by tag.
Every topology backs EBS with ocibsd, which init
detects on the OCI instance, runs the ebs provider
gate, and needs --distro. Its OCI volumes and
backups are in no Terraform state: the destroy here
leaves any a failed run made, and e2e-oci.sh sweeps
them by tag. vm-single-ocibsd and vm-multi-ocibsd are
accepted with --destroy-only, for earlier state.
--ebs-provider NAME ocibsd (default) or viperblockd, which init is then
told through SPINIFEX_EBS_PROVIDER. For a one-off
viperblock check; no default run uses it.
--ssh-public-key PATH Public half installed on each node. Default $SSH_PUBLIC_KEY
--ssh-private-key PATH Private half used to reach them. Default $SSH_PRIVATE_KEY
--instance-principal Configure the pool with oci_auth="instance_principal"
@@ -178,6 +182,7 @@ EOF
while [ $# -gt 0 ]; do
case "$1" in
--topology) TOPOLOGY="${2:?}"; shift 2 ;;
--ebs-provider) EBS_PROVIDER="${2:?}"; shift 2 ;;
--ssh-public-key) SSH_PUBLIC_KEY="${2:?}"; shift 2 ;;
--ssh-private-key) SSH_PRIVATE_KEY="${2:?}"; shift 2 ;;
--workbooks) WORKBOOKS_SET=1; WORKBOOKS="${2-}"; shift 2 ;;
@@ -208,6 +213,13 @@ done
[ -n "$TOPOLOGY" ] || usage
topo_defaults "$TOPOLOGY" || die "unknown topology $TOPOLOGY; one of: $TOPOLOGIES"
[ "$TOPO_LEGACY" = 0 ] || [ "$DESTROY_ONLY" = 1 ] \
|| die "$TOPOLOGY is accepted only with --destroy-only; run ${TOPOLOGY%-ocibsd}, which backs EBS with ocibsd"
case "$EBS_PROVIDER" in
"") ;;
ocibsd | viperblockd) TOPO_EBS_PROVIDER="$EBS_PROVIDER" ;;
*) die "--ebs-provider must be ocibsd or viperblockd, not '$EBS_PROVIDER'" ;;
esac
[ -r "$SSH_PUBLIC_KEY" ] || die "no readable SSH public key at $SSH_PUBLIC_KEY"
[ -r "$SSH_PRIVATE_KEY" ] || die "no readable SSH private key at $SSH_PRIVATE_KEY"
# Both or neither: a distro with the published setup.sh installs one ref's bytes
@@ -860,9 +872,9 @@ done
record install PASS "$INSTALLED_VERSION"
# spx admin init's EBS choice, written to every node once (ADR-0018:S3). The ocibsd
# topologies set nothing, so init detects ocibsd on the OCI VM; the rest name
# viperblockd, which keeps the viperblock baseline on OCI.
# spx admin init's EBS choice, written to every node once (ADR-0018:S3). An
# ocibsd run sets nothing, so init detects ocibsd on the OCI instance; a
# viperblockd run names it, since detection would choose ocibsd.
INIT_EBS_PROVIDER=""
[ "$TOPO_EBS_PROVIDER" = ocibsd ] || INIT_EBS_PROVIDER=viperblockd
@@ -928,7 +940,7 @@ else
ssh-keygen -R "$host" >/dev/null 2>&1 || true
done
# env -u first, so a SPINIFEX_EBS_PROVIDER in this shell cannot stand in for
# detection on the ocibsd topologies.
# detection on an ocibsd run.
env -u SPINIFEX_EBS_PROVIDER ${INIT_EBS_PROVIDER:+SPINIFEX_EBS_PROVIDER=$INIT_EBS_PROVIDER} \
"$REPO_ROOT/scripts/install-node.sh" \
--hosts-file "$STATE_DIR/hosts" \
@@ -951,11 +963,11 @@ init_ebs_line() {
ssh_node "${HOSTS[0]}" "sudo grep -m1 '^EBS provider: ' /var/log/spx-init.log"
fi
}
# The provider has to be the topology's for the reason the topology names: the
# ocibsd pair detected on the VM, the rest told viperblockd by the environment,
# so a baseline flipped by detection or an ocibsd run that was configured fails.
# The provider has to be the run's for the reason the run gives: ocibsd detected
# on the instance or kept from the seeded config, viperblockd from the
# environment, so a choice that came from anywhere else fails.
case "$TOPO_EBS_PROVIDER:$API_KEY_EBS" in
ocibsd:0) want_ebs='EBS provider: ocibsd (detected: OCI virtual machine ' ;;
ocibsd:0) want_ebs='EBS provider: ocibsd (detected: OCI instance ' ;;
ocibsd:1) want_ebs='EBS provider: ocibsd (from the existing config)' ;;
*) want_ebs='EBS provider: viperblockd (from SPINIFEX_EBS_PROVIDER)' ;;
esac
+5 -5
View File
@@ -68,13 +68,13 @@ variable "instance_principal" {
}
}
# What backs EBS on these nodes. ocibsd attaches an OCI block volume per EBS
# volume at runtime, so the Oracle Cloud Agent's Block Volume Management plugin
# is disabled and the node's own data volume attaches paravirtualized instead.
# What backs EBS on these nodes, which decides only the policy grants. ocibsd,
# the default, is what spx admin init detects on any OCI instance; viperblockd
# also needs SPINIFEX_EBS_PROVIDER=viperblockd at init.
variable "ebs_provider" {
description = "EBS provider the nodes are installed with: viperblockd, or ocibsd for OCI Block Volumes."
description = "EBS provider the nodes are installed with: ocibsd for OCI Block Volumes, or viperblockd."
type = string
default = "viperblockd"
default = "ocibsd"
validation {
condition = contains(["viperblockd", "ocibsd"], var.ebs_provider)
+9 -10
View File
@@ -9,7 +9,6 @@ import (
"net/http"
"os"
"path/filepath"
"strings"
"time"
toml "github.com/pelletier/go-toml/v2"
@@ -136,10 +135,10 @@ func readExistingEBS(tomlPath, node string) (*existingEBS, error) {
return &n.EBS, nil
}
// isOCIVM reports whether md is an OCI virtual machine; bare-metal shapes
// stay on viperblockd until a bare-metal iSCSI run is proven.
func isOCIVM(md oci.InstanceMetadata) bool {
return md.Shape != "" && !strings.HasPrefix(md.Shape, "BM.")
// isOCIInstance reports whether md describes an OCI instance, virtual
// machine or bare metal.
func isOCIInstance(md oci.InstanceMetadata) bool {
return md.Shape != ""
}
// detect reads the OCI instance metadata within imdsDetectTimeout; ok is
@@ -170,10 +169,10 @@ func (r ebsResolver) resolveProvider(ctx context.Context, existing *existingEBS)
switch {
case !ok:
return nodeEBS{EBSConfig: formation.EBSConfig{Provider: config.EBSProviderViperblockd}, Source: ebsFromDetection, Detail: "no OCI instance metadata"}, nil
case !isOCIVM(md):
return nodeEBS{EBSConfig: formation.EBSConfig{Provider: config.EBSProviderViperblockd}, Source: ebsFromDetection, Detail: "OCI bare metal " + md.Shape}, nil
case !isOCIInstance(md):
return nodeEBS{EBSConfig: formation.EBSConfig{Provider: config.EBSProviderViperblockd}, Source: ebsFromDetection, Detail: "OCI instance metadata names no shape"}, nil
}
return nodeEBS{EBSConfig: formation.EBSConfig{Provider: config.EBSProviderOCIBSD}, Source: ebsFromDetection, Detail: "OCI virtual machine " + md.Shape}, &md
return nodeEBS{EBSConfig: formation.EBSConfig{Provider: config.EBSProviderOCIBSD}, Source: ebsFromDetection, Detail: "OCI instance " + md.Shape}, &md
}
// fillOCIBSD completes an ocibsd choice: the existing config's values win,
@@ -305,7 +304,7 @@ func generateClusterID() (string, error) {
}
// ocibsdStageDir is where setup.sh stages the tarball's ocibsd/ files on an
// OCI virtual machine.
// OCI instance.
const ocibsdStageDir = "/usr/local/share/spinifex/ocibsd"
// ocibsdFiles maps each staged file to where activation installs it.
@@ -363,7 +362,7 @@ func activateEBSProvider(ctx context.Context, e nodeEBS, a ocibsdActivation) err
func installStaged(src, dst string) error {
data, err := os.ReadFile(src)
if err != nil {
return fmt.Errorf("activate ocibsd: read the staged %s (setup.sh stages it on an OCI virtual machine): %w", src, err)
return fmt.Errorf("activate ocibsd: read the staged %s (setup.sh stages it on an OCI instance): %w", src, err)
}
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { //nolint:gosec // systemd and udev directories are world-readable
return fmt.Errorf("activate ocibsd: %w", err)
+5 -3
View File
@@ -91,7 +91,9 @@ func TestResolveInitEBS_Precedence(t *testing.T) {
{name: "an existing config's provider wins over the environment", existing: "[nodes.node1.ebs]\nprovider = \"qemunbd\"\n", env: config.EBSProviderViperblockd, shape: "VM.Standard.E6.Flex", want: config.EBSProviderQEMUNBD, source: ebsFromConfig},
{name: "the environment wins over detection", env: config.EBSProviderViperblockd, shape: "VM.Standard.E6.Flex", want: config.EBSProviderViperblockd, source: ebsFromEnv},
{name: "detection on an OCI virtual machine", shape: "VM.Standard.E6.Flex", want: config.EBSProviderOCIBSD, source: ebsFromDetection},
{name: "detection on an OCI bare-metal shape", shape: "BM.Standard.E2.64", want: config.EBSProviderViperblockd, source: ebsFromDetection},
{name: "detection on an OCI bare-metal shape", shape: "BM.Standard.E2.64", want: config.EBSProviderOCIBSD, source: ebsFromDetection},
{name: "an existing config's provider wins over detection on bare metal", existing: "[nodes.node1.ebs]\nprovider = \"viperblockd\"\n", shape: "BM.Standard.E2.64", want: config.EBSProviderViperblockd, source: ebsFromConfig},
{name: "the environment wins over detection on bare metal", env: config.EBSProviderViperblockd, shape: "BM.Standard.E2.64", want: config.EBSProviderViperblockd, source: ebsFromEnv},
{name: "detection off OCI", want: config.EBSProviderViperblockd, source: ebsFromDetection},
{name: "another node's table is not this node's config", existing: "[nodes.node2.ebs]\nprovider = \"qemunbd\"\n", want: config.EBSProviderViperblockd, source: ebsFromDetection},
}
@@ -136,7 +138,7 @@ func TestResolveInitEBS_DetectionFillsTheOCITable(t *testing.T) {
assert.Regexp(t, `^[0-9a-f]{32}$`, got.ClusterID, "128 random bits in hex")
assert.Equal(t, 1, f.imdsReads, "detection's read also fills the table")
assert.Equal(t, []string{"instance_principal||"}, f.clientFor, "the credential is checked as instance principal")
assert.Equal(t, "EBS provider: ocibsd (detected: OCI virtual machine VM.Standard.E6.Flex)", got.describe())
assert.Equal(t, "EBS provider: ocibsd (detected: OCI instance VM.Standard.E6.Flex)", got.describe())
again, err := resolveInitEBS(context.Background(), absentConfig(t), "node1", f.resolver())
require.NoError(t, err)
@@ -340,7 +342,7 @@ func TestActivateEBSProvider_MissingStage(t *testing.T) {
runner := &fakeRunner{}
e := nodeEBS{EBSConfig: formation.EBSConfig{Provider: config.EBSProviderOCIBSD}}
err := activateEBSProvider(context.Background(), e, ocibsdActivation{Root: t.TempDir(), Stage: filepath.Join(t.TempDir(), "absent"), Runner: runner})
require.ErrorContains(t, err, "setup.sh stages it on an OCI virtual machine")
require.ErrorContains(t, err, "setup.sh stages it on an OCI instance")
assert.Empty(t, runner.calls, "nothing is enabled without its files")
}
+4 -1
View File
@@ -322,7 +322,7 @@ func (c *blockAPIClient) DeleteVolume(ctx context.Context, volumeID string) erro
}
func (c *blockAPIClient) AttachVolume(ctx context.Context, in AttachVolumeInput) (VolumeAttachment, error) {
notShareable := false
notShareable, noAgentLogin := false, false
resp, err := c.compute.AttachVolume(ctx, core.AttachVolumeRequest{
AttachVolumeDetails: core.AttachIScsiVolumeDetails{
InstanceId: &in.InstanceID,
@@ -330,6 +330,9 @@ func (c *blockAPIClient) AttachVolume(ctx context.Context, in AttachVolumeInput)
DisplayName: optional(in.DisplayName),
IsReadOnly: &in.ReadOnly,
IsShareable: &notShareable,
// The Oracle Cloud Agent logs in and out only attachments that set
// this, and these sessions are the provider's alone to manage.
IsAgentAutoIscsiLoginEnabled: &noAgentLogin,
},
})
if err != nil {
@@ -141,6 +141,7 @@ func TestBlockAttachVolumeIsISCSIAndNotShareable(t *testing.T) {
assert.Equal(t, "iscsi", body["type"])
assert.Equal(t, true, body["isReadOnly"])
assert.Equal(t, false, body["isShareable"])
assert.Equal(t, false, body["isAgentAutoIscsiLoginEnabled"], "the Oracle Cloud Agent must not log these sessions in or out")
assert.Equal(t, "i1", body["instanceId"])
assert.Equal(t, "v1", body["volumeId"])
}
@@ -114,6 +114,23 @@ func TestStartLogsOutOnlyThisClustersDetachedSessions(t *testing.T) {
}
}
// An OCI bare-metal host boots from a boot volume over iSCSI, so its session
// is open before the provider starts and must survive every stale-session pass.
func TestStartKeepsTheBareMetalBootSessionBesideAStaleOne(t *testing.T) {
fake := oci.NewBlockFake()
stale := ociTarget("detached")
seedAttachment(fake, "att-old", instanceOf("node-1"), seedVolume(t, fake, testCluster), oci.AttachmentDetached, stale)
boot := ocibsd.Target{IQN: "iqn.2015-02.oracle.boot:uefi", IPv4: "169.254.0.2", Port: 3260}
initiator := ocibsd.NewFakeInitiator()
initiator.SeedSession(boot)
initiator.SeedSession(stale)
p := newNode(t, fake, initiator)
require.NoError(t, p.Start(context.Background()))
assert.Equal(t, []string{"logout " + stale.IQN}, initiator.Events())
assert.Equal(t, []ocibsd.Target{boot}, initiator.Open())
}
func TestPublishWaitsForTheStaleSessionLogout(t *testing.T) {
fake := oci.NewBlockFake()
stale := ociTarget("stale")
@@ -1,92 +0,0 @@
//go:build e2e
package rds
import (
"testing"
"time"
"github.com/stretchr/testify/assert"
)
func TestRotationOverlapped(t *testing.T) {
sent := time.Date(2026, 11, 13, 3, 47, 2, 0, time.UTC)
at := func(offset time.Duration, status string) statusSample {
return statusSample{sent: sent.Add(offset), status: status}
}
cases := map[string]struct {
samples []statusSample
want bool
}{
"a describe sent after the rotation reads backing-up": {
samples: []statusSample{at(-50*time.Millisecond, dbInstanceBackingUp), at(20*time.Millisecond, dbInstanceBackingUp)},
want: true,
},
"a describe sent before the rotation is not evidence however late it returned": {
samples: []statusSample{at(-time.Millisecond, dbInstanceBackingUp)},
want: false,
},
"the first describe after the rotation reads available: a late send": {
samples: []statusSample{
at(-50*time.Millisecond, dbInstanceBackingUp),
at(10*time.Millisecond, "available"),
at(60*time.Millisecond, dbInstanceBackingUp),
},
want: false,
},
"no describe was sent after the rotation": {
samples: []statusSample{at(-50*time.Millisecond, dbInstanceBackingUp)},
want: false,
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
assert.Equal(t, tc.want, rotationOverlapped(tc.samples, sent))
})
}
}
func TestClassifyOverlapAttempt(t *testing.T) {
sent := time.Date(2026, 11, 13, 3, 47, 2, 0, time.UTC)
held := []statusSample{{sent: sent.Add(-50 * time.Millisecond), status: dbInstanceBackingUp},
{sent: sent.Add(20 * time.Millisecond), status: dbInstanceBackingUp}}
released := []statusSample{{sent: sent.Add(-50 * time.Millisecond), status: dbInstanceBackingUp},
{sent: sent.Add(20 * time.Millisecond), status: "available"}}
neverSeen := []statusSample{{sent: sent, status: "available"}, {sent: sent.Add(time.Second), status: "available"}}
assert.Equal(t, overlapped, classifyOverlapAttempt(held, &sent))
assert.Equal(t, lateSend, classifyOverlapAttempt(released, &sent))
assert.Equal(t, missedStatus, classifyOverlapAttempt(neverSeen, nil))
}
func TestRetryUntilOverlap(t *testing.T) {
scripted := func(results ...overlapAttempt) (func(int) overlapAttempt, *[]int) {
var called []int
return func(n int) overlapAttempt {
called = append(called, n)
return results[n-1]
}, &called
}
t.Run("a missed status and a late send each open another window", func(t *testing.T) {
attempt, called := scripted(missedStatus, lateSend, overlapped)
n, ok := retryUntilOverlap(3, attempt)
assert.True(t, ok)
assert.Equal(t, 3, n)
assert.Equal(t, []int{1, 2, 3}, *called)
})
t.Run("three misses end with none", func(t *testing.T) {
attempt, called := scripted(missedStatus, lateSend, missedStatus)
_, ok := retryUntilOverlap(3, attempt)
assert.False(t, ok)
assert.Equal(t, []int{1, 2, 3}, *called)
})
t.Run("an overlap on the first window opens no other", func(t *testing.T) {
attempt, called := scripted(overlapped)
n, ok := retryUntilOverlap(3, attempt)
assert.True(t, ok)
assert.Equal(t, 1, n)
assert.Equal(t, []int{1}, *called)
})
}
+78 -161
View File
@@ -18,56 +18,30 @@ import (
)
const (
// The status an instance holds for the length of a snapshot, from before the
// quiesce to after the release.
// The status an instance holds for the length of a snapshot, from the quiesce
// to the release.
dbInstanceBackingUp = "backing-up"
// How often the watcher samples the status. The hold has lasted 0.3 to 1.3s,
// so this sees it with time left to send a rotation into it.
backingUpSampleEvery = 50 * time.Millisecond
// How often the watcher samples the status. The hold lasts about a second, so
// the harness's 10s status interval would step straight over it.
backingUpSampleEvery = 100 * time.Millisecond
// One window: the reconciler's pass that fires it, the backup itself, and the
// minute boundary a later attempt has to wait for.
backupOverlapAttemptTimeout = 5 * time.Minute
// The status is sampled, so a short hold can still slip between two samples.
// Each further attempt opens a fresh window.
// The hold is short and the rotations are discrete, so one backup can finish
// between two of them. Each further attempt opens a fresh window.
maxBackupOverlapAttempts = 3
// How often an attempt still waiting for backing-up looks for a snapshot that
// landed without the watcher seeing the status.
backupLandedCheckEvery = time.Second
// A probe table only this check writes to, so its DDL and commit say nothing
// about any row the rest of the test reads.
backupProbeTable = "e2e_backup_probe"
)
// How one window went for the rotation sent into it.
type overlapAttempt int
const (
overlapped overlapAttempt = iota
// The backup landed and no sample read backing-up, so nothing was sent.
missedStatus
// A rotation was sent, but no describe sent after it read backing-up.
lateSend
)
func (a overlapAttempt) String() string {
switch a {
case overlapped:
return "a rotation overlapped the backup"
case missedStatus:
return "the backup landed without the watcher seeing backing-up"
default:
return "the rotation was not seen inside backing-up"
}
}
// Sends a rotation as soon as the instance reports backing-up for an automated
// backup, until one is seen inside it. conn is kept on whichever password is
// live, so later subtests connect with it.
// Rotates the master password back and forth through an automated backup until
// one rotation is seen in flight while the instance reports backing-up. conn is
// kept on whichever password is live, so later subtests connect with it.
func rotateDuringAutomatedBackup(t *testing.T, f *Fixture, client harness.SSHTarget, id string, conn *harness.MariaDBConn) {
t.Helper()
// Put back once done, so no later subtest can be interrupted by a retry of
@@ -75,116 +49,40 @@ func rotateDuringAutomatedBackup(t *testing.T, f *Fixture, client harness.SSHTar
t.Cleanup(func() { pinBackupWindowClearOfTheRun(t, f, id) })
var lastOpened time.Time
_, ok := retryUntilOverlap(maxBackupOverlapAttempts, func(attempt int) overlapAttempt {
result := rotateIntoOneBackup(t, f, client, id, conn, &lastOpened)
if result != overlapped {
t.Logf("attempt %d: %s of %s; opening another window", attempt, result, id)
for attempt := 1; attempt <= maxBackupOverlapAttempts; attempt++ {
before, err := automatedSnapshots(f, id)
require.NoError(t, err)
beforeIDs := automatedSnapshotIDs(before)
lastOpened = openBackupWindowNow(t, f, id, lastOpened)
watch := watchStatus(f, id)
overlapped := rotateUntilTheBackupLands(t, f, client, id, conn, beforeIDs, watch)
watch.stop()
if err := watch.err(); err != nil {
t.Logf("the status watcher missed samples; the last describe failed with: %v", err)
}
return result
})
if !ok {
t.Fatalf("no rotation overlapped an automated backup of %s in %d windows", id, maxBackupOverlapAttempts)
harness.WaitForDBInstanceAvailable(t, f.AWS, id, harness.WithPoll(2*time.Second),
harness.WithTimeout(2*time.Minute))
assertTheEngineIsReleased(t, client, *conn)
if overlapped {
return
}
t.Logf("attempt %d: the backup of %s finished between two rotations; opening another window", attempt, id)
}
t.Fatalf("no rotation overlapped an automated backup of %s in %d windows", id, maxBackupOverlapAttempts)
}
// Runs attempts until one overlaps, and reports which one did.
func retryUntilOverlap(attempts int, attempt func(int) overlapAttempt) (int, bool) {
for n := 1; n <= attempts; n++ {
if attempt(n) == overlapped {
return n, true
}
}
return 0, false
}
// Opens a window, sends one rotation once the instance reports backing-up, and
// waits for the backup to land before saying how the rotation fell.
func rotateIntoOneBackup(t *testing.T, f *Fixture, client harness.SSHTarget, id string,
conn *harness.MariaDBConn, lastOpened *time.Time) overlapAttempt {
t.Helper()
before, err := automatedSnapshots(f, id)
require.NoError(t, err)
beforeIDs := automatedSnapshotIDs(before)
watch := watchStatus(f, id)
*lastOpened = openBackupWindowNow(t, f, id, *lastOpened)
deadline := time.Now().Add(backupOverlapAttemptTimeout)
var sent *time.Time
nextLandedCheck := time.Now().Add(backupLandedCheckEvery)
for {
require.True(t, time.Now().Before(deadline),
"no automated backup of %s finished within %s of its window opening", id, backupOverlapAttemptTimeout)
if watch.latest() == dbInstanceBackingUp {
issued, err := rotateOnce(t, f, client, id, conn)
sent = &issued
t.Logf("rotation sent at %s while %s reported backing-up (err=%v)", issued.Format(time.RFC3339Nano), id, err)
break
}
if time.Now().After(nextLandedCheck) {
if newAutomatedSnapshot(t, f, id, beforeIDs) {
break
}
nextLandedCheck = time.Now().Add(backupLandedCheckEvery)
}
time.Sleep(5 * time.Millisecond)
}
for watch.latest() == dbInstanceBackingUp || !newAutomatedSnapshot(t, f, id, beforeIDs) {
require.True(t, time.Now().Before(deadline),
"no automated backup of %s finished within %s of its window opening", id, backupOverlapAttemptTimeout)
time.Sleep(backupLandedCheckEvery)
}
watch.stop()
if err := watch.err(); err != nil {
t.Logf("the status watcher missed samples; the last describe failed with: %v", err)
}
harness.WaitForDBInstanceAvailable(t, f.AWS, id, harness.WithPoll(2*time.Second),
harness.WithTimeout(2*time.Minute))
assertTheEngineIsReleased(t, client, *conn)
result := classifyOverlapAttempt(watch.all(), sent)
if result == overlapped {
t.Logf("rotation in flight from %s overlapped the backup", sent.Format(time.RFC3339Nano))
}
return result
}
// Classifies a window from its samples and the rotation's send time, nil when
// no rotation was sent.
func classifyOverlapAttempt(samples []statusSample, sent *time.Time) overlapAttempt {
switch {
case sent == nil:
return missedStatus
case rotationOverlapped(samples, *sent):
return overlapped
default:
return lateSend
}
}
// The rotation is only sent once a sample reads backing-up, and the status
// holds it once per backup, so the first describe sent at or after the rotation
// reading it shows the status was held when the rotation went out.
func rotationOverlapped(samples []statusSample, sent time.Time) bool {
for _, s := range samples {
if !s.sent.Before(sent) {
return s.status == dbInstanceBackingUp
}
}
return false
}
// Opens the backup window on the current minute. When the previous attempt's
// window opened on this one, it first waits for the next minute: a window fires
// once, and the write that opens it is what wakes the reconciler to fire it.
// Opens the backup window on the current minute, or on the next one when the
// previous attempt's window already opened on this one: a window fires once,
// and only for a backup older than its opening.
func openBackupWindowNow(t *testing.T, f *Fixture, id string, previous time.Time) time.Time {
t.Helper()
if next := previous.Add(time.Minute); time.Now().Before(next) {
time.Sleep(time.Until(next))
}
opens := time.Now().UTC().Truncate(time.Minute)
if !opens.After(previous) {
opens = previous.Add(time.Minute)
}
window := dailyWindowAt(opens, 30*time.Minute)
harness.Step(t, "Opening the backup window of %q at %s", id, window)
_, err := f.AWS.RDS.ModifyDBInstance(&rds.ModifyDBInstanceInput{
@@ -196,22 +94,39 @@ func openBackupWindowNow(t *testing.T, f *Fixture, id string, previous time.Time
return opens
}
// Rotates the master password to whichever of the two it is not on, checks the
// rotation is whole, and returns when it was sent.
func rotateOnce(t *testing.T, f *Fixture, client harness.SSHTarget, id string, conn *harness.MariaDBConn) (time.Time, error) {
// Rotates until the backup has produced its snapshot and the instance has left
// backing-up, and reports whether any rotation was in flight while it was in it.
func rotateUntilTheBackupLands(t *testing.T, f *Fixture, client harness.SSHTarget, id string,
conn *harness.MariaDBConn, beforeIDs []string, watch *statusWatch) bool {
t.Helper()
next := mariaRotatedPassword
if conn.Password == mariaRotatedPassword {
next = mariaBackupRotationPassword
deadline := time.Now().Add(backupOverlapAttemptTimeout)
overlapped := false
for {
require.True(t, time.Now().Before(deadline),
"no automated backup of %s finished within %s of its window opening", id, backupOverlapAttemptTimeout)
next := mariaRotatedPassword
if conn.Password == mariaRotatedPassword {
next = mariaBackupRotationPassword
}
issued := time.Now()
_, err := f.AWS.RDS.ModifyDBInstance(&rds.ModifyDBInstanceInput{
DBInstanceIdentifier: aws.String(id),
MasterUserPassword: aws.String(next),
ApplyImmediately: aws.Bool(true),
})
answered := time.Now()
if watch.sawBetween(dbInstanceBackingUp, issued, answered) {
overlapped = true
t.Logf("rotation in flight %s-%s overlapped the backup (err=%v)",
issued.Format(time.RFC3339Nano), answered.Format(time.RFC3339Nano), err)
}
assertRotationIsWhole(t, client, conn, next, err)
if watch.latest() != dbInstanceBackingUp && newAutomatedSnapshot(t, f, id, beforeIDs) {
return overlapped
}
}
issued := time.Now()
_, err := f.AWS.RDS.ModifyDBInstance(&rds.ModifyDBInstanceInput{
DBInstanceIdentifier: aws.String(id),
MasterUserPassword: aws.String(next),
ApplyImmediately: aws.Bool(true),
})
assertRotationIsWhole(t, client, conn, next, err)
return issued, err
}
// An accepted rotation leaves only the new password working; a refused one
@@ -270,7 +185,7 @@ func pinBackupWindowClearOfTheRun(t *testing.T, f *Fixture, id string) {
}
// Samples an instance's status on its own goroutine, fast enough to see a hold
// that lasts well under a second.
// that lasts about a second.
type statusWatch struct {
cancel context.CancelFunc
done chan struct{}
@@ -280,10 +195,8 @@ type statusWatch struct {
lastErr error
}
// sent is when the describe went out, so a reply that was already in flight
// when something else happened is never read as coming after it.
type statusSample struct {
sent time.Time
at time.Time
status string
}
@@ -295,13 +208,12 @@ func watchStatus(f *Fixture, id string) *statusWatch {
ticker := time.NewTicker(backingUpSampleEvery)
defer ticker.Stop()
for {
sent := time.Now()
instance, err := harness.DescribeDBInstance(f.AWS, id)
w.mu.Lock()
if err != nil {
w.lastErr = err
} else {
w.samples = append(w.samples, statusSample{sent: sent, status: aws.StringValue(instance.DBInstanceStatus)})
w.samples = append(w.samples, statusSample{at: time.Now(), status: aws.StringValue(instance.DBInstanceStatus)})
}
w.mu.Unlock()
select {
@@ -334,8 +246,13 @@ func (w *statusWatch) latest() string {
return w.samples[len(w.samples)-1].status
}
func (w *statusWatch) all() []statusSample {
func (w *statusWatch) sawBetween(status string, from, to time.Time) bool {
w.mu.Lock()
defer w.mu.Unlock()
return slices.Clone(w.samples)
for _, s := range w.samples {
if s.status == status && !s.at.Before(from) && !s.at.After(to) {
return true
}
}
return false
}