* fix: drain guests on a plain target stop, not just restart
`systemctl is-system-running` cannot tell a plain `systemctl stop
spinifex.target` apart from the stop half of a restart -- both report
`running`, and the old `--only-if-host-stopping` gate skipped the drain
for both. That left guests running with no QMP shutdown and no volume
unmount while predastore/viperblock stopped underneath them, so a real
stop discarded any writes still in flight.
Add a second signal: `systemctl list-jobs` shows a pending START job for
spinifex.target during a restart (the target's stop leg resolves
instantly, since it owns no processes) or a pending RESTART job for
spinifex-shutdown.service itself when that unit is restarted directly
(PartOf propagates target-to-member only). Renamed the flag to
`--unless-restarting` to match: drain on a real shutdown or a plain
stop, skip only when one of those jobs shows a restart is already
queued. Any step systemctl can't answer fails toward draining, and a
skipped drain now warns loudly (naming each instance still running) so
an operator sees the risk in the journal instead of discovering it via
`ps`. `--only-if-host-stopping` stays as a deprecated, still-functional
alias so a partial upgrade between unit file and binary doesn't break.
* fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss
A drain that fails at GATE is indistinguishable from a skipped drain from
the guest's perspective: it keeps running while storage goes away
underneath it. Stopping spinifex.target shortly after a restart hit exactly
this: the daemon had not finished re-subscribing to NATS, so the ACK
request came back with zero subscribers and NATS bounced it back near-
instantly ("no responders"), which is why the failure logged in about a
second despite --timeout=120s. collectLocalShutdownACK now retries with
backoff bounded by the phase timeout instead of failing on the first bounce,
and an exhausted retry budget logs at ERROR and names the guests still
running via a severity-aware refactor of warnIfGuestsLeftRunning, so a
failed drain is unmistakable in the journal rather than silent.
* fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget
KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but
shutdownVolumes reaped them one at a time and TimeoutStopSec was never
re-derived for that change. With several mounted volumes the serial loop
blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and
whichever nbdkit hadn't been reaped yet was orphaned.
Fan the per-volume utils.KillProcess calls out concurrently so the reap
costs one worst-case nbdkit, not N of them summed, and raise
TimeoutStopSec to 150s to strictly exceed that single-volume worst case
(120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an
invariant asserting the two stay in that relationship, and a test proving
the reap is actually concurrent (fails against the old serial loop).
* docs: document the host and guest lifecycle contract
States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.
make test-integration cats the AWS model conformance report immediately after
the test run, but the report is written by the test binary itself. Go's test
cache means an unchanged package reports "ok (cached)" without executing, so
no report is written and the cat fails the target with exit 1.
Any branch whose changes do not invalidate that package's cache hits it, which
is why it fires on unrelated branches -- a docs-only branch cannot break a build
yet failed the same way.
-count=1 forces the run. That also keeps the conformance gate honest: a cached
result would mean the check never ran against this commit. Making the cat
tolerant instead would let the gate silently no-op.
The bare-checkout build on the CI runner resolves these from go.mod alone,
where the tagged v1.16.0 releases still carry the pre-bluebottle masterkey
and otelsetup layouts.
otelsetup/otelsetup.go and otelsetup/slog.go were a near-copy of the shared
bootstrap. Entrypoints now call bluebottle directly, passing
WithEnvironmentFallback for SPINIFEX_CI_ENV; the package keeps only the HTTP
middleware and request metrics. Init reattaches the OTLP bridge to a default
logger installed before it ran, so the local addFanoutHandler is no longer
needed.
The bridge is now level-gated, so services no longer ship Debug records to
the sink while running at Info.
Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather
than predastore, so the control plane no longer depends on the S3 server for
shared auth and policy code.
setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare
set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every
interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB
control plane was reachable from the public interface. Write access to the
Southbound database is control of the whole tenant network fabric.
ovsdb-server takes remotes from two additive sources: the Connection table,
which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That
split is what the fix uses. Loopback goes in the replicated row, because
127.0.0.1 is correct on every node, so replication is harmless and local
clients keep working. The lan address goes on the per-node command line via
--db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so
each node binds its own address without pushing it to its peers.
A new --lan-addr flag carries that address, falling back to
--db-cluster-local-addr so clustered nodes stay correct without caller changes.
The standalone branch now writes /etc/default/ovn-central too, since a two-node
install still has a remote ovn-controller that needs the lan listener. The
guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting
create-insecure-remote without a resolved address would recreate the bug.
install-node.sh passes the lan address on both branches and warns when the lan
plane has collapsed onto the public address, since the listener then lands on
the public interface and a firewall is the only remaining control.
* feat: render predastore templates with the cluster and S3 planes split
The templates set one bind_addr per host, which put raft and blob wherever
S3 was published. A host now binds the cluster plane on the address peers
dial and the gate carries its own bind_addr for the S3 API, so the installer
can hand S3 the WAN without exposing replication with it.
The single-node template also drops to one node of each role. Extra blob
nodes only hash-partition keys across more badger indexes on the same drive,
and extra meta replicas add fsyncs for a quorum that dies with the machine.
* docs: regenerate the AWS model operation coverage table
* fix: repin predastore to the merged dev commit
Rework the frontmatter titles and descriptions for the five Identity
docs so each leads with the AWS concept term users actually search, and
add contextual in-prose links between them. Only the roles and instance
profiles page currently earns search impressions, so it now links to all
four siblings and each sibling links back into the cluster.
The branch imports predastore/pkg/auth, predastore/pkg/ratelimit and the
root predastore package, none of which exist in the pinned v1.15.0. Every
build that goes through the go.work workspace resolved those from the
sibling checkout and passed; the e2e binary build on the CI runner uses a
bare checkout governed by go.mod alone, and failed to resolve them.
The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
The fixture wrote a config the strict decoder rejects and built its cluster
through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and
serves under Run. Loading the real file means the fixture trips over the same
validation an install does. Its data directory holds node-<id> directly, and it
drains on cancellation rather than being abandoned at exit.
The runtime tests drove buildRuntime, serve and the old on-disk layout, all of
which predastore owns now, so they go. What replaces them covers what spinifex
still decides: the flag merge and its precedence, a host id or gate node the
config does not have, and the pid file behind Stop and Status.
Two fixes to earlier commits in this series: the host id conversion moves behind
its own guard, and the node status probes use WaitGroup.Go.
Hosts carry their nodes under [[host.node]] now, and a node's address is its
host's addr plus its own port rather than a single host address it shared. The
gate belongs to neither storage table, so it is matched explicitly and an
unknown role still warns.
The reported vocabulary follows predastore's: db_nodes and shard_nodes become
meta_nodes and blob_nodes through the daemon, the gateway and the UI. Bucket
type is gone from predastore entirely, so the field goes with it.
The facade owns every node of a host, the S3 gate among them, so the service no
longer builds a cluster runtime or a gateway of its own. Waiting for a leader,
draining the gateway and the shutdown grace all moved behind Run; pprof and the
base path have no equivalent and are gone.
spinifex.toml stays authoritative for the S3 endpoint: the host entry is
settled from the service flags before Run, and the gate is found by role rather
than by position. A host id is now required, so init writes one for single-node
installs too, where the topology was previously left implicit.
BasePath survives as the pid directory alone. Stop and Status now read the same
value start writes, which they did not before.
The decoder is strict now, so every key the reorganised predastore no longer
reads is a startup failure rather than an ignored line. Both templates move to
nested [[host.node]] tables with the gate, blob and meta roles, portless
bind_addr and addr, singular [[bucket]], and the host TLS and encryption-key
paths the schema requires. version resets to 1 with the format.
A gate node is what serves the S3 API, so a host without one answers nothing.
Ports are unique within a host rather than across the cluster: gate 8443, blob
from 6660, meta from 7660. Colocated nodes bind no socket and still need one.
The daemon dialled a /status endpoint on each meta node to report which held
raft leadership. That endpoint lived in predastore's s3db package; after the
transport collapse the meta nodes speak rpc over QUIC and the in-process pipe
and expose no HTTP surface, so the probe dialled a dead port.
NodeStatusResponse.PredastoreRole is retained and left empty. Every consumer
already guards on the empty string, so the CLI and the UI simply omit the
badge, and restoring the probe later needs no contract change.
The migration rewrote a v4 topology into v5 and recovered raft state through
predastore's clusterrun package, which no longer exists. The on-disk layout it
relocated data into has moved too: node directories are now node-<id> under the
host root, with no host-N level. Nothing migrates a pre-cutover install; it is
re-initialised from scratch.
node-reset.sh keeps directories by design, so it now removes the predastore
cluster tree outright rather than leaving the old skeleton standing.
* feat: add guardrail CRUD store and ARN support to bedrock gateway
Stage 1 of Ochre Guardrails: control-plane store for CreateGuardrail,
CreateGuardrailVersion, GetGuardrail, ListGuardrails, UpdateGuardrail,
and DeleteGuardrail, backed by a JetStream KV bucket keyed per account.
Records keep the full policy config so later stages can build the
filter engine and inference enforcement on top without re-deriving it.
* feat: add deterministic guardrail filter engine and ApplyGuardrail op
Adds the pure wordPolicy/sensitiveInformationPolicy filter (regex-detectable
PII plus a starter profanity list), scaffolds content/topic/contextualGrounding
as NONE, and exposes it all through the bedrock-runtime ApplyGuardrail route.
* feat: enforce guardrails on Converse and ConverseStream
Wire GuardrailConfig enforcement into Router.Converse and
ConverseStream via a shared enforceGuardrail helper (resolve, load,
apply the deterministic filter engine). INPUT blocks skip the
backend entirely; OUTPUT blocks or ANONYMIZE redactions rewrite the
completion before it reaches the caller. ConverseStream buffers
delta text and assesses it at block/message-stop boundaries rather
than streaming unassessed model output. Both self-host and
Anthropic-direct backends are covered automatically since the
enforcement sits above Router's per-backend dispatch. Requests
without a GuardrailConfig are unaffected. An unresolvable or foreign
guardrail fails closed with ResourceNotFoundException.
* feat: enforce guardrails on InvokeModel and InvokeModelWithResponseStream
Adds the Invoke half of Ochre guardrails (plan §5), reusing the Stage 3
enforceGuardrail helper and Stage 2 filter engine unchanged. Guardrail
selection comes from the X-Amzn-Bedrock-Guardrail{Identifier,Version}
headers. Since InvokeModel bodies are family-native/opaque at the
Router, text is extracted/reinjected per family (llama self-host,
anthropic) in the new guardrail_invoke.go. INPUT is checked before the
backend is ever called (fail closed on an unresolved guardrail, no
backend call); OUTPUT is checked after, handling block/anonymize.
InvokeModelWithResponseStream buffers the inner stream and assesses
once at end-of-stream via a guardrailInvokeStreamSource decorator,
mirroring Stage 3's guardrailStreamSource.
* fix: route ApplyGuardrail through bedrock-runtime and use guardrail-specific RNF text
The service discriminator only promoted /model/... paths from bedrock to
bedrock-runtime, so ApplyGuardrail's singular /guardrail/.../apply route fell
through to the control-plane handler and was rejected. Extend the
discriminator to also promote /guardrail/..., which can't collide with the
plural /guardrails control-plane CRUD routes.
Also give guardrail-not-found paths their own ResourceNotFoundException
message instead of the generic "could not resolve the foundation model"
wording shared with model-lookup failures.
CI builds spinifex in a workspace over viperblock and predastore only, so
northstar resolves from the proxy. Its v1.15.0 log sanitiser predates the
attribute.Value migration and does not compile against otel/log v0.21.0.
The log API moved bodies and attributes onto attribute.Value in v0.21.0, so the
otelslog bridge and the sibling log sanitisers have to move in the same window.
No spinifex source change is needed; the workspace resolves the migrated
predastore and viperblock.
* feat: parameterise bedrock endpoint create/delete by account and pinned
Widens EnsureEndpointInput/DescribeEndpointInput/DeleteEndpointInput
with AccountID and (Ensure-only) Pinned, and keys the daemon's
Ensure/Describe/Delete on the resolved account instead of the
hardcoded GlobalAccountID. An empty AccountID still resolves to
GlobalAccountID and Pinned defaults to false, so every existing
caller (resolver, reaper, evictForCapacity, the NATS gateway client)
is unchanged. This is stage 1 of provisioned throughput: it reuses
the existing launch/capacity/readiness machinery for a dedicated,
account-scoped, pinned endpoint without adding a parallel surface.
* feat(bedrock): add provisioned-throughput CRUD and PT ARN
Implements CreateProvisionedModelThroughput, Get, List, Update, and
Delete over a new bedrock-provisioned KV store, plus the
provisioned-model ARN format/parse. Create rejects provider-hosted
models and requests a pinned, account-scoped endpoint from the daemon;
Get derives status live from the pinned endpoint's state; Update
honours name changes only and rejects a served-model swap; Delete
tears down the pinned endpoint before removing the record.
EndpointProvisioner is declared in gateway_bedrock with primitive-typed
methods (handlers_bedrock already imports gateway_bedrock, so it can't
be imported back for its EndpointService interface directly); a small
adapter in handlers_bedrock bridges the two, always setting Pinned:true
and the caller's real account on every Ensure.
Does not yet wire a PT ARN into the Converse/InvokeModel inference
path (next stage).
* feat(bedrock): accept a provisioned-throughput ARN on Converse/InvokeModel
Translates a PT ARN before resolve: the gateway inference entry parses it,
loads the bedrock-provisioned record, and resolves against the pinned
endpoint for the commitment's own account rather than the GlobalAccountID
shorthand every bare modelId uses. Adds EndpointResolver.EndpointForAccount
(gateway-side interface, implemented by DynamicEndpointResolver) so the seam
stays import-cycle-safe, and threads it through vllm.go/invoke_llama.go's
resolve call sites. A foreign-account or unknown PT ARN reads as
ResourceNotFoundException, same as any other unresolvable model identifier.
* fix(bedrock): accept a provisioned-throughput ARN on streaming inference
ConverseStream and InvokeModelWithResponseStream built their router/adapter
with a nil ProvisionedStore and the GlobalAccountID-shorthand provider, so a
PT ARN was silently never translated on the streaming path even though the
non-stream Converse/InvokeModel siblings already handled it. Thread the
gateway's provisioned store through both entry points and give
Router.ConverseStream / InvokeStreamRouter.InvokeModelWithResponseStream the
same translate-before-resolve treatment Converse/InvokeModel already have.
* fix(bedrock): surface pinned endpoints in the operator CLI
Service.List only ever enumerated the GlobalAccountID key prefix, and
'endpoint describe' always looked up GlobalAccountID too, so a pinned,
account-scoped endpoint held a GPU while reading ABSENT and missing from
list. List now scans every account's records, and describe gains an
optional --account flag (default empty, preserving GlobalAccountID
resolution) to look up a specific account's endpoint. list/describe both
render the endpoint's account and pinned status.
* fix(ochre): self-heal weights stage when the recorded snapshot is gone
runStageWeights's idempotency check trusted a matching source URI without
verifying the snapshot it names still exists, so a host rebuild or volume
GC that lost the snapshot left stage permanently short-circuited on a
useless KV record until an operator ran 'weights remove' first. Inject a
weightsSnapshotChecker, verify liveness before the no-op return, and fall
through to re-materialise when the snapshot is gone. 'weights list' uses
the same checker to mark dangling entries so an operator can see which
staged models can actually serve without waiting for endpoint launch to
fail.
* docs: note weights stage self-heal and list STATUS column
Adds resolveServingAMI test cases: single match, multiple matches
selected deterministically by newest CreationDate independent of
describe-order, zero matches, and nil/empty ImageId entries skipped.
A binary-swap deploy (make deploy or a manual spx copy) ships a new binary
whose host-config prerequisites — the privileged helper scripts and their
sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship.
The node looks healthy and the gap surfaces later as a cryptic launch
failure or a silent security regression.
Add a preflight package that checks the host against what this build
expects, comparing asset content, not existence, so a stale-but-present
asset is caught too. The managed helpers install from quoted heredocs, so
their on-disk bytes equal the heredoc body verbatim; the binary carries
the canonical sha256s and compares.
- spinifex/preflight: CheckHost / CheckHostAt, content hash of the two
managed helpers and a NOPASSWD grant-line check for the sysctl helper.
- Generated canonicalHashes plus a drift guard that re-extracts the
heredocs and fails if the committed manifest no longer matches.
- Three surfaces: spx admin preflight, a make deploy gate
(SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning.
Josh Siviour