* fix: size the volume lease against the stall the host actually produces
A guest lost its disk whenever JetStream writes failed for fifteen seconds,
which is what one NVMe I/O timeout on the node holding the JetStream leader
produces. The budget composed badly rather than any single number being wrong:
a confirmation could already be volumeLeaseRenewInterval old when an outage
began, and validity expired volumeLeaseValidity after it, so an outage covering
[15s, 30s) surrendered a volume no peer had taken. Measured outages are 10-15s,
so the lease was sized at the edge of a routine event. On one cluster thirty
eight renewal timeouts produced six surrenders and four destroyed guests.
Validity rises to 40s and the renewal interval falls to 10s, taking the
tolerated outage from fifteen seconds to thirty. The validity-to-TTL gap is now
named by volumeLeaseAcquireMargin instead of being implied by two constants:
nothing may acquire until the server-side bucket TTL ages the entry out, so the
margin only has to cover scheduling and request latency on this side.
The renewal timeout rises above the stall bound, so a renewal that begins inside
a stall is answered when it clears rather than being certain to fail. That is
not a constant on its own. renewLoop was one goroutine, so a blocking Update
delayed the validity check by the whole timeout: measured on the old loop with
the new timeout, a holder surrendered 51s after its last confirmation, eleven
seconds past validity and six past the server TTL, which is a window where a
peer may already hold the volume. The renewal now runs on its own goroutine with
one call in flight at a time, the loop keeps testing validity underneath it, and
it waits for the renewal before closing done so release still reads a settled
revision.
deviceStallBound documents nvme_core.io_timeout as a property of the host. It is
not configured or applied here, only asserted against, because it describes the
machines rather than the product.
The sizing is a probability, not a guarantee. An outage longer than validity
still surrenders, and the fence still destroys the guest instead of revoking the
export and leaving it paused, which is the change that would make outage length
stop mattering.
* fix: keep a fenced instance out of every automatic relaunch
A fence does not seal, so the volumes it leaves behind are a state nothing has
shown is mountable. Two paths relaunched one anyway. The restore path consults
the restart window, which forgives on a clock, so a fence days old reports a
full budget and the next daemon start remounts it. The recovery reconciler
excludes a storage fault but not a fence, and a fence kills the export instead
of pausing the guest, so the record publishes no I/O error and reads as an
ordinary desired-running instance on a node that has died.
Both now read vm.VolumeFenced off the StateReason the fence already writes. The
refusal is narrower than StateError on purpose: a recovery that failed for its
own reasons is what the restart budget exists for.
Also separates the two storage quantities the lease was sized against. One
stalled I/O lasts deviceStallBound, which is what a renewal timeout has to
exceed; what the lease has to survive is a run of them, now named
jetstreamOutageBound. They were conflated, so raising the device constant would
have licensed a validity past the server TTL.
A surrender can only ever report validity, so the length of an outage was
unobservable. Renewals now count consecutively and report outage_ms once they
recover, which measures the windows the lease survives.
install.mulgadc.com has served a dev channel since it was written -- setup.sh
understands INSTALL_SPINIFEX_CHANNEL=dev and the Worker resolves it to the newest
GitHub prerelease -- but nothing ever produced one, so /download/dev/amd64
returned 404. This is the producer.
A separate workflow rather than conditionals in release.yml. A dev build is a
different artifact, not a release with flags off: amd64 only, no ISO, no system
images, no bare-metal gate, and nothing on a self-hosted runner. The sibling
repositories are taken at dev rather than their default branch, because dev of
spinifex against main of the others is a pairing nobody runs.
The tag is the describe string plus -dev, which release.yml's own convention
already treats as a prerelease, and which makes the commit recoverable from the
version alone. The flag is set at creation and then the published state is
asserted: a prerelease must never become the latest release, because latest is
what an unqualified curl | bash installs.
setup.sh gains --channel and --version so the instruction is a flag rather than
an environment variable, and the OCI harness gains OCI_CHANNEL so a nightly can
test the path the contacts are actually given.
* fix(oci): make a bare-metal node form and mount its data volume
Bare metal presents VNICs differently from a VM: the first VNIC on a physical
port is the port, and a second one sharing that port is a tagged overlay with no
Linux device at all. The external VNIC was landing in that second slot, so no
interface ever carried it and the WAN bridge could not be built. Attach it at
nic_index 1 on a BM shape, which makes it the first VNIC on the second port and
therefore untagged and present at boot, exactly as it is on a VM.
Two consequences of that followed. The bridge script picked the first non-primary
VNIC from instance metadata, which on bare metal can be one with no interface, so
it now selects the VNIC the kernel actually has a device for and waits for that.
And OCI makes a bare-metal volume attachment multipath, where the Oracle agent
refuses to log in unless multipathd is running -- probing the unit name Debian
retired, so on Ubuntu 26.04 it never sees it. Alias the retired name and assert
the alias on every poll rather than once, because the first run had it created and
then vanish.
Finally, the three cloud-init steps become ordered systemd units. runcmd
continues past a failed step, so a node whose volume never appeared still opened
its service ports with nothing behind them; Requires= makes that impossible and
lets a boot that raced the iSCSI login retry on the next one.
* feat(oci): authenticate the address allocator as the instance principal
The OCI allocator could only authenticate from an API-key config file, so every
node needed a private key copied to it and a config file hand-written beside it.
A node without them forms, passes every health check, and then refuses every
launch that wants a public address with InsufficientAddressCapacity -- the cause
appears only in the journal.
NewInstancePrincipalClient already existed and was called from nowhere. Wire it
in behind a new oci_auth key on the pool, so a node can authenticate with the
certificate its own metadata service serves and hold no key material at all.
Nothing to copy, nothing to rotate, and nothing in Terraform state.
The reason recorded for preferring the config file was that it reads no instance
metadata, because Spinifex's own IMDS endpoints claim 169.254.169.254 and take
the cloud's metadata service with them. That no longer distinguishes the two:
oci_vnic_iface is what the guide recommends and it resolves the VNIC through a
metadata lookup already, so such a node depends on the IMDS remap either way.
The two constructors' comments also contradicted each other on which was
preferred; both now say what they cost.
config_file stays the default, because instance principal needs a dynamic group
and a policy only a tenancy admin can write. The Terraform creates both behind
enable_instance_principal, off by default for the same reason the module creates
no other tenancy-root resource, and the policy grants three verbs on vnics,
private-ips and public-ips in one compartment rather than anything broader.
Naming the method explicitly rather than inferring it is deliberate: a node meant
to use its own identity now fails at config load instead of falling back to a key
file nobody installed. Pairing oci_auth="instance_principal" with a config file
is refused for the same reason, since the certificate is already the credential
and the file would be the half silently ignored.
Terraform also stages the pool fragment on each node, which under instance
principal contains no secret at all.
* feat(oci): validate all three topologies from one script, and document both auth routes
validate-topology.sh builds one topology from nothing, installs the published
release, forms the cluster, configures the pool and the IMDS remap, runs a
workbook against it, proves the workbook actually serves traffic, and destroys
everything. Three topologies because each breaks differently: bare metal presents
VNICs unlike a VM, a single node has no Geneve underlay to get wrong, and only a
cluster exercises RAFT, the gateway chassis and cross-node allocation.
--topology has no default, deliberately. A command aimed at the wrong topology is
the easiest expensive mistake available here, and each keeps its own Terraform
state so two can be built from one checkout without either destroying the other's
instances.
The teardown decides the verdict rather than tidying up after it. A topology or a
workbook that cannot be destroyed is half proved, so destroy runs from an EXIT
trap and a teardown failure fails the run. It also checks the two things a green
apply does not: that vpcd logged resolving the external VNIC, which is the only
evidence the credentials work and br-wan matched a real VNIC, and that the load
balancer returns 200 -- a workbook can create cleanly and carry no traffic.
The guide gains the instance-principal route beside the key file, and loses its
claim that bare metal changes nothing else in the document. That claim was wrong
in exactly two places, and both are now handled rather than left to the reader:
the external VNIC's nic_index and the Oracle agent's multipath unit name. It now
also says plainly that the external-address path on bare metal is unproven.
* test(oci): let a topology be validated without an OCI credential
The external pool is the one phase that needs a key or a dynamic group, and
the VM topologies have to be regression-tested regardless. --no-external-pool
forms and verifies without it, and implies --skip-workload so the run cannot
claim a guest path it never exercised.
* test(oci): one entry point that proves a ref works on OCI, and cleans up after itself
e2e-oci.sh drives every topology from environment variables alone, so a
workstation, a Worker dispatch and e2e-oci-nightly.yml all take the same path
rather than drifting into three answers about whether OCI works.
Three things it deliberately does not do. It does not test the published
release: --distro installs the tarball built from the ref under test, because a
nightly that cannot fail on a bad commit is decoration. It does not carry its
own workbook assertions, running tests/e2e/run-tofu-examples-e2e.sh over the
published workbooks instead, so a workbook that passes on a hypervisor and fails
here is a difference in OCI. And it cannot quietly leave cloud resources
running: --sweep-only destroys whatever an earlier run's state still holds, and
it runs before the topologies, after them, and from an always() step.
The credential now reaches Terraform as a value through OCI_PRIVATE_KEY, so no
runner writes key material to disk, and oci_env.py refuses --shell on that route
because in CI stdout is the job log.
Four harness defects the first real runs exposed, each of which had been giving
a wrong answer:
- oci_env.py announced itself on stdout, corrupting every captured
terraform output. That is also why a single node came back as two hosts.
- terraform output was passed -var, which it rejects, and a silent fallback
then read the default state file instead of the topology's.
- grep -c Ready matches NotReady, so a single-node run whose only node was
NotReady on 0.0.0.0 reported PASS. The status column is parsed now, and
membership is waited for rather than sampled once.
- install-node.sh refuses fewer than two hosts, so a single node forms with
spx admin init the way the guide documents.
Also waits for the dpkg lock: cloud-init being finished does not mean apt is.
* fix(oci): resolve mulga's path explicitly, and refuse an unpushed ref
Two assumptions in the new harness that a run would not have caught quickly.
Deriving mulga's root from spinifex's is right for a submodule checkout and
wrong for CI, where the two are siblings under the workspace, so MULGA_ROOT is
explicit and the workflow sets it.
build-install-artifacts.sh clones each repo from GitHub rather than using the
checkout beside it, and falls back to dev when the ref is not there. A nightly
on an unpushed branch would then pass or fail on dev's code while reporting the
branch. Check the ref is on origin first, and skip the Go suite build: the
workbook driver is a shell script, so those binaries prove nothing here.
* fix(oci): do not read $? after a command when set -e is on
The workbook phase read the driver's exit status on the line after the ssh that
produced it, which under set -e is never reached: a failing run aborted before
printing the driver's own per-workbook diagnostics, losing them exactly when
they are the thing you need. An `if !` keeps them, and names which workbooks
passed and which did not.
* feat: let an OCI e2e dispatch form without the external allocator
A tenancy without the nodes' dynamic group cannot authenticate the
allocator, so every topology failed at it rather than reporting what it
had proved. The no_external_pool input forms with no allocator and
records the allocator and the workbooks as SKIPPED, never PASS.
Also rebuilds the dry-run plan message imperatively: it still referenced
a variable renamed to WORKBOOKS, and under set -u the substitution died
and the text silently vanished from the line.
* fix: name the offending input when an OCI credential is unusable
A credential from the environment reached the SDK unvalidated, so a
secret with a trailing newline failed as an uncaught InvalidConfig
traceback naming an SDK config field that nobody set by that name.
Strip the scalar values, since a secret store written with echo carries
a newline, and translate both InvalidConfig and InvalidPrivateKey into
the variable or config key to go and fix.
* fix: make the OCI nightly able to build, and stop it leaking a dispatch input into the shell
Four defects, found reviewing the workflow rather than running it.
The builder clones all five sub-repos from GitHub over https and they are
all private, so a hosted runner had no credential and the tree build
could never have worked. The app token now covers six repositories and
git carries it for our organisation only.
A dispatch input was interpolated into a run block, where a quote ends
the string and the remainder executes. It goes through the environment
now, which is the only place a caller-supplied value belongs.
The artifact upload took the whole directory, which meant the distro
tarball and the test archive, and would have taken terraform.tfstate had
the state directories sat under it. It is a named list now, and it also
collects the per-topology logs that were never uploaded at all.
* fix: make the OCI nightly's runner a setting, and stop widening the app token
The five sub-repos are public, so build-install-artifacts.sh clones them
without a credential and the token has no reason to reach them. Only
mulga is private, which is what it was scoped to before.
The runner is now inputs.runner, then the OCI_E2E_RUNNER variable, then
hosted. A per-provider self-hosted agent is a reasonable thing to want --
it lifts the six-hour cap a hosted job carries and keeps a long cloud
wait off the hypervisor labels -- and it should not need a workflow edit.
Removing the SSH key is load bearing on that path: a self-hosted runner
keeps $HOME between jobs, so the key would sit there between nightlies.
* fix: install the OCI SDK the harness needs, and keep state out of the checkout
Nothing installed the oci Python module, on any runner: validate-topology.sh
called bare python3 and oci_env.py imports oci, so every topology would have
failed at the first terraform call. The harness now uses a system python3 that
has it and otherwise builds its own virtualenv from requirements.txt, which is
the same behaviour on a workstation, a hosted runner and a self-hosted one.
The state directories move under OCI_STATE_ROOT, defaulting to the script
directory. A persistent runner needs them outside the checkout, because
actions/checkout wipes untracked files there and would remove the state of a
killed run before the sweep could read it, leaving OCI instances running that
nothing is able to find.
* fix: drop the OCI SDK dependency, so the harness runs on any host
python3-venv is not installed on banksia, and the harness needed it: the
environment helper re-executed itself from a virtualenv to import the OCI
SDK. It used the SDK for two things -- parsing an INI file, which
configparser already does, and discovering the tenancy home region, which
no Terraform file consumed. Neither justifies a dependency on every host
that runs this, so the helper is standard library only now.
The credential is checked for shape instead, locally and before any OCI
call, which is a better failure than a 401 forty seconds into a run.
Two things the removal uncovered. The home region is a fixed property of
a tenancy rather than something to rediscover every run, so it is
OCI_HOME_REGION. And OCI serves IAM writes from the home region while the
dynamic group and policy were using the default provider for the build
region, which would have failed when the tenancy applied them: they go
through an aliased provider that defaults to the build region, so a
tenancy whose home region is where it builds sees no change.
DeleteInstanceProfile now deletes via kvutil.DeleteIf at the revision it
checked, so a role attached after the RoleName check sends the delete back
through the DeleteConflict guard instead of removing the profile.
ListInstanceProfiles logs and skips a profile whose attached role no longer
exists rather than failing the whole account listing.
* docs: add work-in-progress Exoscale integration guide with stack diagrams
* feat: source=exoscale external pool allocating Elastic IPs through the exo CLI
* docs: rewrite the Exoscale guide as a short end-user install, and test the exo fake, store and wiring
Fuzz QueryParamsToStruct over query bodies decoded into EC2, ELBv2 and RDS
inputs: no panic, list entries bounded by MaxSliceLen and the input size, and
an entry past the first gap never changes the parse. make fuzz runs every
Fuzz* target for FUZZTIME each.
bodyscope lower-cased names with strings.ToLower and decoded into a map, while
handlers decode with encoding/json. A non-ASCII name that encoding/json folds
onto an ASCII field (repoſitoryName), or a repeated name whose later value is
null or an object, let the gate authorize one resource while the handler acted
on another. Parse now decodes with encoding/json/v2, which refuses repeated
names, and refuses non-ASCII names; both yield ErrAmbiguousBody.
FuzzGateMatchesHandler checks, for every identifier the ECR, ECS, EKS, ACM and
Bedrock resolvers read, that the gate and a handler-shaped decode agree.
Removes directives for linters that are not enabled (noctx, exhaustive,
deadcode) and gosec suppressions already covered by config exclusions.
Tags ochrevector QueryResult and Filter with their existing field names
so musttag passes without changing the NATS wire format, and bounds the
RDS window slot conversion instead of suppressing G115.
Peer /health and predastore readiness probes skipped TLS verification.
Both now trust the cluster CA (nats.cacert) that issues every node's server
cert. A node without the CA stops peer probing, leaving peers reachable,
and reports predastore not ready.
Ben Duncan