* feat: ochre-vector list-jobs subject
Adds ochre.vector.listJobs to the daemon-side VectorService surface (both
the in-process and NATS-forwarding implementations) so a caller can list
every ingestion job for an account rather than looking one up by id.
Widens jobGetter to expose JobStore.List, registers the subject on the
daemon's post-connect subscription table, and updates the CLI test
double to satisfy the widened VectorService interface.
Also carries the jackc/puddle/v2 indirect dependency go.sum was missing.
* feat: bedrock-agent knowledge-base + data-source CRUD
Adds the emulated AWS bedrock-agent control-plane gateway service:
KnowledgeBase/DataSource CRUD, StartIngestionJob/GetIngestionJob/
ListIngestionJobs, backed by new gateway-owned KBStore/DataSourceStore
JetStream KV persistence (mirroring registry.go's CAS idiom) and the
existing ochre-vector daemon surface for the bound vector index.
One knowledge base binds exactly one vector index. embeddingModelArn
maps to EmbeddingModel/Dimension; storageConfiguration and roleArn are
accepted and stored verbatim but never acted on. CreateDataSource maps
S3 + fixed-size chunking config onto a SourceSpec, converting AWS's
maxTokens/overlapPercentage into internal ChunkSize/ChunkOverlap.
DeleteKnowledgeBase cascades to every data source under it before
dropping the bound index. Wires BedrockAgentKB/BedrockAgentDataSources/
BedrockAgentVector into the gateway and registers bedrock-agent in the
supported-services and dispatch tables.
Runtime (Retrieve/RetrieveAndGenerate) is out of scope here.
* test: cover bedrock-agent routing and KB/DataSource store edges
Adds end-to-end HTTP tests for BedrockAgent_Request's route table and
per-route handler closures (all eleven operations, plus unknown-route/
missing-account/nil-dependency error paths), KBStore/DataSourceStore
PurgeAll and no-JetStream-client error paths, and ListJobs coverage
(vectorService, NATS round trip, per-account scoping) in the existing
ochre-vector API test suite.
* feat: bedrock-agent-runtime retrieve + retrieve-and-generate
Adds the emulated bedrock-agent-runtime data plane: Retrieve resolves a
knowledge base to its bound vector index and runs a similarity query;
RetrieveAndGenerate runs the same retrieval, renders a RAG prompt (default or
caller-supplied template), and calls Converse in-process for generation,
returning citations built from the retrieved chunks and a session id.
Recovers the full RetrievalFilter operator set from the raw request body
since the vendored aws-sdk-go RetrievalFilter type only carries andAll/orAll,
and fixes GetService never routing real signed bedrock-agent traffic (only
bedrock-runtime and bedrock had discriminator rules; bedrock-agent's own
knowledgebases paths had none).
* fix: bedrock-agent KB delete idempotency + ingestion-job ownership
CreateKnowledgeBase: a rollback DeleteIndex failure after a kb.Create
claim failure is now logged at Error level with the orphaned index id
instead of a bare Warn, while the caller still sees kb.Create's own
error unchanged.
DeleteKnowledgeBase: a bound index that DeleteIndex reports already
gone (ErrIndexNotFound) no longer aborts before the KB record itself is
removed, so a KB never dangles pointing at an index that no longer
exists. Any other DeleteIndex error still aborts the cascade.
GetIngestionJob: now verifies the addressed dataSourceId owns the job
(Bucket+Prefix match against the data source's own SourceSpec, same
best-effort check ListIngestionJobs already used), not just the
knowledge base's bound index -- closing a path where a job could be
read through the wrong data source id.
* fix: default bedrock-agent-runtime retrieve to 5 results (AWS parity)
* fix: thread DataSourceID through ingestion jobs
JobRecord and IngestRequest now carry an exact DataSourceID instead of
relying on Bucket+Prefix as a best-effort proxy. StartIngestionJob
stamps it from the DataSource record; GetIngestionJob and
ListIngestionJobs filter on it exactly, treating a job with no
DataSourceID as belonging to no bedrock-agent data source.
* fix: purge KB/DataSource records on appliance teardown
Appliance.Teardown purged the index registry and job store but left
the gateway-owned KB/DataSource KV records dangling, pointing at a
deleted index. WithKBStores attaches those stores the same way
WithStores does for registry/jobs, built before Connect so teardown
still works on a broken appliance, and Teardown purges them too.
* feat: add /v1/embeddings gateway adapter
* feat: Ochre vector store pgvector backend, KV index registry, orchestration
Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl
with per-account schema/role isolation enforced via SET LOCAL ROLE +
search_path inside every transaction, the KV index registry
(CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile
orchestration tying them together with crash-safe rollback and
re-entrant recovery of stuck records.
* feat: singleton appliance-ensure state machine for ochre vector store
Adds the encrypted control-plane credential store and re-entrant
Ensure state machine for the platform pgvector appliance (D2): a
fixed-key KV claim mirroring Registry.Reserve elects exactly one
winner to launch the appliance via an injected ApplianceLauncher,
losers wait or resume a stale crashed provision, and the master
password is generated once, stored only AES-256-GCM encrypted, and
decrypted solely inside Connect to build a pgxBackend.
* feat: Ochre vector store async ingestion (chunker, job store, worker)
Adds Stage 3: a deterministic recursive chunk splitter, a per-account
ingestion job-record store mirroring the index registry, the
transactional per-document replace on the backend, and the
IngestService worker that lists a predastore source, chunks and
embeds each object, and replaces its rows idempotently.
* feat: Ochre vector store similarity query and metadata filter grammar
Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW
scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that
compiles to parameterized SQL over jsonb, closing off arbitrary SQL by
allowlisting jsonb keys and always binding values as params. Also
fixes RunJob reporting READY on a job that ingested nothing regardless
of failure cause, and stamps SourceSpec.Metadata onto ingested rows so
the new filters have data to match against.
* feat: Ochre vector store request/response API, VectorService, and CLI
Adds the daemon-facing VectorService interface (index CRUD, ingest,
describe job, query), its server implementation over the existing
registry/backend/embedder, a NATS-forwarding client, and the
`spx admin ochre vector` operator CLI. Query resolves and embeds
against each index's own pinned model; ingest stamps the same
model/dimension onto every source spec server-side. accountID is
always taken from the caller context, never the request payload.
Daemon subscription wiring and manifest registration are a later
change; this is pure request/response plumbing with no construction
into daemon.go or awsgw.go.
* feat: wire Ochre vector store VectorService into the daemon
Constructs the platform Postgres appliance and VectorService behind a new
OchreVector config section that defaults off, so existing daemons are
unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the
appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six
ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern,
and degrades to a disabled (nil) service on any boot-time appliance failure
rather than blocking daemon startup.
* fix: fix Ochre vector store boot ordering and self-register subjects
startOchreVector ran before subscribeAll, so its appliance launch fired
rds.CreateDBInstance/DescribeDBInstances requests with no responder yet
registered, timing out the launch on a first-enable boot. It also ran
synchronously, blocking daemon startup up to the appliance launch timeout.
Move it to a background goroutine started after subscribeAll returns, and
have it register its own ochre.vector.* subjects once the VectorService is
actually constructed, via a new registerNatsSubs helper shared with
subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since
this background registration can now race the shutdown path's
unsubscribe-all sweep.
* feat: bake pgvector into the shared rds-postgres image
* fix: reach the ochre appliance over a routed daemon port on its ENI IP
The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net
hostname the control-plane host can neither resolve nor route to, so a healthy
appliance was never reachable and the vector store disabled itself.
Resolve the appliance's customer-ENI private IP read-only via
ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial
that IP for the DSN instead of the hostname, and give the daemon its own routed
ENI in the appliance's subnet bound to an OVS host port so the IP is reachable.
Retry appliance startup with bounded backoff instead of hard-disabling on the
first transient.
* fix: install pgvector at bootstrap for the Ochre vector appliance
The daemon connects to the appliance as its NOSUPERUSER master, and the
image's pgvector is not a trusted extension, so CREATE EXTENSION vector
failed with permission denied. rds-init now installs it via the bootstrap
superuser, gated on the appliance's fixed master name so tenant instances
are unaffected.
* fix: grant the Ochre appliance master CREATE on its database
The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER
master, which does not own the postgres database, so bootstrap now grants it
CREATE on the database alongside the pgvector install. Both are gated to the
appliance master, so tenant instances are unaffected.
* fix: make the Ochre appliance's pgvector reachable under per-account isolation
The daemon connects as a NOSUPERUSER master and runs every account operation
under SET ROLE with a search_path pinned to the account schema. Two gaps blocked
that against a real non-superuser Postgres: the master could not SET ROLE into
the roles it creates, and the account role could not see the vector type.
Bootstrap now grants the appliance master createrole_self_grant and installs
pgvector into a read-only extensions schema granted to PUBLIC; the backend
carries that schema on each account search_path so the vector type resolves
without weakening per-account write isolation. All appliance-scoped.
* fix: prune stale VPC host port before installing the daemon appliance port
A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI
mints a fresh vhp- port at an address a superseded port still holds. The old
port survives at the OS/OVS layer even after its ENI record is wiped, so two
interfaces claim one address and the kernel can route to the dead one
(EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the
target address before installing.
* fix: gate VM restore's stale-socket kill on backing-store readiness
On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an
orphan. While predastore/viperblock is still coming up after a restart those
sockets are transiently unreachable, so a healthy VM (the Ochre appliance was
the observed victim) was destroyed and its RDS record flipped to failed. The
kill now only fires when the backing store is confirmed up; otherwise the live
QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store
is unchanged.
* fix: drive ochre ingest jobs from a daemon-side sweep scheduler
* feat: add operator teardown verb for the ochre platform appliance
Adds Appliance.Teardown to remove the singleton platform Postgres
appliance (RDS instance, host port, KV record), an admin-only NATS
subject and daemon handler to drive it, and `spx admin ochre appliance
teardown --confirm` to invoke it without exposing the action through
the tenant awsgw surface.
* fix: register ochre appliance teardown before Connect so it survives a broken appliance
* fix: purge ochre index and job metadata on appliance teardown
* test: mark ochre in-package test files for the external-test gate
* fix(e2e): resolve the listener inventory without a Go toolchain on the target node
tests/e2e/multinode/listener_invariant_test.go resolved
docs/security/network-connections/README.md at runtime via `go env
GOMOD`, which works under `go test` but not inside a compiled .test
binary shipped to a node with no Go toolchain. Embed the doc in a new
docs/security/network-connections package instead, so the content is
compiled into the binary and every caller parses bytes rather than a
resolved path. RepoRoot and DocPath are now unused and removed.
* docs: trim the embed package comment to the three-line standard
* test: add the listener invariant static guard and e2e runtime check
Adds both halves the plan called for: a static test in
spinifex/network/invariants that scans install scripts and config
templates for a bare OVSDB ptcp remote, a wildcard bind on a
Cluster/Encap port, or a bind port absent from the inventory, and an
e2e test in tests/e2e/multinode that asserts the same on live ss
-tulnp output. Both read
docs/security/network-connections/README.md "## 1. Inbound
Listeners" as their fixture through a shared, non-test parser package
(spinifex/network/listenerinventory), so a wildcard exception has to
be declared in the doc's own prose rather than hardcoded in either
test.
* fix: make the listener wildcard exception fail closed on an explicit marker
The prior heuristic granted an exception for any unnegated "wildcard"
or "0.0.0.0" mention within 40 characters of the port's row. That
fails open: a future row phrased as "avoids the wildcard" or with a
negation further upstream than the window would silently exempt
itself. Replace it with a single exact literal phrase, "binds the
wildcard by design", that a row must contain to grant the exception —
no other wording counts, so an unrecognized mention denies rather
than grants. Update the two rows that need it (5300, 500/4500) and
document the marker's role in the inventory's own prose.
* docs: record that the Geneve tunnel socket binds the wildcard by design
Overwriting /usr/local/bin/spx while spinifex.target is running does not move
the services onto it. Each keeps executing the replaced inode until it
restarts, so a node serves a mixture of two builds, and because the handlers
are NATS queue-group workers the skew shows up as roughly one request in three
behaving like the old build — an intermittent fault rather than a broken node.
The ansible deploy path already detects and remediates this, but only there:
an out-of-band binary copy leaves nothing to report it, and on env19 that
state went unnoticed for a full day. spx admin preflight now reports it, so an
operator has a command to run and a script has a non-zero exit to gate on.
Detection follows the deploy path's: a (deleted) marker on /proc/<pid>/exe, a
target path that no longer resolves, or a path that resolves to a different
inode than the one being executed.
* fix(shutdown): scope a local drain to its own node and stop counting failed phases as done
A node's spinifex-shutdown.service publishes its GATE and DRAIN requests to
the shared spinifex.cluster.shutdown.* subjects, which every daemon subscribes
to as a fan-out. The node filter only decided which ACK the stopping node
waited for, so by the time it applied, every other node had already stopped
its gateway, UI and VPC daemon and powered down its guests. Stopping one node
took down every guest in the cluster.
Requests now carry the node they apply to, and a daemon drops one addressed
elsewhere without ACKing. A cluster shutdown leaves the field empty, so it
still reaches every node.
The STORAGE phase also ran 'pkill -f nbdkit', which matches every nbdkit on
the host: another node's backends on a shared host, and a live guest's backend
that survived a daemon restart under KillMode=process. Cleanup now sweeps this
node's own nbdkit-vol-*.pid files and signals a PID only while /proc still
agrees it is nbdkit, so a recycled PID is never hit.
Finally, the coordinator counted errored ACKs toward its per-phase quorum, so
a node that failed to drain reported N/N and the run advanced into STORAGE and
PERSIST with guests still running. Only clean ACKs count now, and any errored
one aborts the phase unless --force.
* test(shutdown): cover the phase gate, every handler's target check and the real proc paths
Diff coverage came in at 52.9% against a 60% threshold. The gaps were the
per-phase target checks beyond GATE, the real procComm and signalProcess
bodies that every cleanup test stubs out, and the coordinator's abort branch,
which sat inside runClusterShutdown where no unit test could reach it.
The abort decision moves into shutdownPhaseOutcome, a pure function returning
the completed count and the message to print, so the rule that an errored node
does not count toward the quorum is testable on its own rather than only
reachable through a full cluster shutdown.
Image import shelled out to virt-customize to install the deployment CA into
an imported image's trust store. The step was best-effort and skipped whenever
virt-customize was absent, which is the default: nothing in provisioning
installs guestfs-tools on a node. When it did run it mutated the image after
its checksum had been verified, it could not handle Windows guests, and it tied
CA rotation to re-importing every affected AMI.
Nothing the stack launches depended on it. ECS container instances, EKS k3s
servers and BootDirect microVMs all receive the gateway CA at launch via
user-data or fw_cfg, and boot itself needs no CA because IMDS is plain HTTP.
The only uncovered case is a customer instance whose own workload calls a
Spinifex HTTPS endpoint, which needs a pull mechanism rather than an image
mutation.
The IMDS responder now serves the CA at /spinifex/ca.pem, outside the /latest
tree so the AWS-compatible surface stays byte-identical to EC2's. The route is
token-free because a CA certificate is public material and the console already
serves it unauthenticated; the X-Forwarded-For SSRF guard still applies. The
PEM is cached and re-read on modtime change, so rotating ca.pem on a node
reaches guests without restarting vpcd.
* feat: reconcile systemd unit files on upgrade
Unit files were written once at install time by install_systemd() in
scripts/setup.sh and never re-asserted. A node upgraded by swapping the
spx binary kept whatever units it was installed with, so any fix that
lives in a unit rather than in code never reached it.
This is not theoretical. A node whose spinifex-viperblock.service
predates KillMode=process defaults to KillMode=control-group, so on a
target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs
every nbdkit while guest QEMU survives in the daemon's own cgroup: the
guest keeps running with no storage path.
Units are now stamped with a spinifex-unit-version marker, embedded in
the binary via a generated file, and reconciled by spx admin upgrade.
Missing or version-stale units are replaced after a timestamped backup;
a unit at the current version whose content differs is reported as
operator-modified and never overwritten. Drop-ins under <unit>.d are
untouched, so systemctl edit overrides survive.
Nothing is restarted. A daemon-reload is enough for KillMode and
TimeoutStopSec to apply to the next stop, so drift is corrected without
disturbing a running guest.
Adds --dry-run, --units-only and --skip-units to spx admin upgrade.
* test: cover systemd reconcile error paths and the admin upgrade CLI flow
Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate.
Adds tests for reconcile.go's error/edge paths (unreadable unit, backup
and write failures, daemon-reload failure, malformed version markers,
downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly
via an injectable exit and a decision matrix extracted into
validateUpgradeFlags/upgradeSummary for testability.
AMI metadata moved to ebsmetadata documents and the legacy
ami-<id>/config.json read path was removed rather than migrated, so an
AMI imported before the change is invisible to the control plane after
upgrading. Nothing reports it as pending, because the gap is in stored
data rather than in a config file. Warn about it alongside the existing
object-storage exemption, and say how to check and recover.
Also document the binary-swap trap: replacing spx under a running
target leaves services on the replaced inode, and because handlers are
queue-group workers a skewed node answers a fraction of requests with
the old behaviour, which reads as an intermittent fault.
The inventory was missing northstar (53 on the advertise address, 5300 on the
wildcard and dialled cross-node by every node's per-instance DNS shim), charon
(500/4500 plus ESP), Geneve 6081, the per-instance IMDS and VPC-DNS listeners,
and chronyd. Section 6 asks the operator to compare the table against ss -tulnp,
which was not possible while six listeners were absent.
Two classifications were wrong. 8443 was Cluster, but predastore binds the S3
gate to 0.0.0.0 deliberately because S3 is a public plane. 4432 was External,
but it binds the node's --bind address and is transient.
The scope vocabulary predated the three-plane model, so it could not express the
encap plane or record that a node with fewer NICs collapses the planes onto its
public address. Cluster and Encap now state intent rather than a guarantee.
Replaces the reference nftables ruleset. The previous one had no conntrack,
loopback, ICMP or ime-* rules and ended in a blanket drop, so an operator who
applied it lost guest metadata, guest DNS, all egress return traffic and their
own SSH session. The replacement is input-only in a dedicated table, because
nftables verdicts are per-table and a default-deny forward chain would override
the accepts vpcd writes to ip filter and break every routed-NAT instance.
* fix: drain guests on a plain target stop, not just restart
`systemctl is-system-running` cannot tell a plain `systemctl stop
spinifex.target` apart from the stop half of a restart -- both report
`running`, and the old `--only-if-host-stopping` gate skipped the drain
for both. That left guests running with no QMP shutdown and no volume
unmount while predastore/viperblock stopped underneath them, so a real
stop discarded any writes still in flight.
Add a second signal: `systemctl list-jobs` shows a pending START job for
spinifex.target during a restart (the target's stop leg resolves
instantly, since it owns no processes) or a pending RESTART job for
spinifex-shutdown.service itself when that unit is restarted directly
(PartOf propagates target-to-member only). Renamed the flag to
`--unless-restarting` to match: drain on a real shutdown or a plain
stop, skip only when one of those jobs shows a restart is already
queued. Any step systemctl can't answer fails toward draining, and a
skipped drain now warns loudly (naming each instance still running) so
an operator sees the risk in the journal instead of discovering it via
`ps`. `--only-if-host-stopping` stays as a deprecated, still-functional
alias so a partial upgrade between unit file and binary doesn't break.
* fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss
A drain that fails at GATE is indistinguishable from a skipped drain from
the guest's perspective: it keeps running while storage goes away
underneath it. Stopping spinifex.target shortly after a restart hit exactly
this: the daemon had not finished re-subscribing to NATS, so the ACK
request came back with zero subscribers and NATS bounced it back near-
instantly ("no responders"), which is why the failure logged in about a
second despite --timeout=120s. collectLocalShutdownACK now retries with
backoff bounded by the phase timeout instead of failing on the first bounce,
and an exhausted retry budget logs at ERROR and names the guests still
running via a severity-aware refactor of warnIfGuestsLeftRunning, so a
failed drain is unmistakable in the journal rather than silent.
* fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget
KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but
shutdownVolumes reaped them one at a time and TimeoutStopSec was never
re-derived for that change. With several mounted volumes the serial loop
blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and
whichever nbdkit hadn't been reaped yet was orphaned.
Fan the per-volume utils.KillProcess calls out concurrently so the reap
costs one worst-case nbdkit, not N of them summed, and raise
TimeoutStopSec to 150s to strictly exceed that single-volume worst case
(120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an
invariant asserting the two stay in that relationship, and a test proving
the reap is actually concurrent (fails against the old serial loop).
* docs: document the host and guest lifecycle contract
States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.
setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare
set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every
interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB
control plane was reachable from the public interface. Write access to the
Southbound database is control of the whole tenant network fabric.
ovsdb-server takes remotes from two additive sources: the Connection table,
which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That
split is what the fix uses. Loopback goes in the replicated row, because
127.0.0.1 is correct on every node, so replication is harmless and local
clients keep working. The lan address goes on the per-node command line via
--db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so
each node binds its own address without pushing it to its peers.
A new --lan-addr flag carries that address, falling back to
--db-cluster-local-addr so clustered nodes stay correct without caller changes.
The standalone branch now writes /etc/default/ovn-central too, since a two-node
install still has a remote ovn-controller that needs the lan listener. The
guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting
create-insecure-remote without a resolved address would recreate the bug.
install-node.sh passes the lan address on both branches and warns when the lan
plane has collapsed onto the public address, since the listener then lands on
the public interface and a firewall is the only remaining control.
* feat: render predastore templates with the cluster and S3 planes split
The templates set one bind_addr per host, which put raft and blob wherever
S3 was published. A host now binds the cluster plane on the address peers
dial and the gate carries its own bind_addr for the S3 API, so the installer
can hand S3 the WAN without exposing replication with it.
The single-node template also drops to one node of each role. Extra blob
nodes only hash-partition keys across more badger indexes on the same drive,
and extra meta replicas add fsyncs for a quorum that dies with the machine.
* docs: regenerate the AWS model operation coverage table
* fix: repin predastore to the merged dev commit
Rework the frontmatter titles and descriptions for the five Identity
docs so each leads with the AWS concept term users actually search, and
add contextual in-prose links between them. Only the roles and instance
profiles page currently earns search impressions, so it now links to all
four siblings and each sibling links back into the cluster.
The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
The fixture wrote a config the strict decoder rejects and built its cluster
through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and
serves under Run. Loading the real file means the fixture trips over the same
validation an install does. Its data directory holds node-<id> directly, and it
drains on cancellation rather than being abandoned at exit.
The runtime tests drove buildRuntime, serve and the old on-disk layout, all of
which predastore owns now, so they go. What replaces them covers what spinifex
still decides: the flag merge and its precedence, a host id or gate node the
config does not have, and the pid file behind Stop and Status.
Two fixes to earlier commits in this series: the host id conversion moves behind
its own guard, and the node status probes use WaitGroup.Go.
* fix(ochre): self-heal weights stage when the recorded snapshot is gone
runStageWeights's idempotency check trusted a matching source URI without
verifying the snapshot it names still exists, so a host rebuild or volume
GC that lost the snapshot left stage permanently short-circuited on a
useless KV record until an operator ran 'weights remove' first. Inject a
weightsSnapshotChecker, verify liveness before the no-op return, and fall
through to re-materialise when the snapshot is gone. 'weights list' uses
the same checker to mark dangling entries so an operator can see which
staged models can actually serve without waiting for endpoint launch to
fail.
* docs: note weights stage self-heal and list STATUS column
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
* feat: add spx admin ochre endpoint ensure/describe/list/delete
The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in
production calls their NATS client, so the serving-endpoint lifecycle could
not be exercised at all in a running system. This gives an operator a direct
way to drive it.
ensure --wait polls describe until READY and reports the elapsed time, since
the daemon replies STARTING as soon as it claims the model and the launch
outlives the request. That is the measured cold start.
Also corrects the weights stage tokenizer rule in COMMANDS.md, which still
described both tokenizer files as required rather than either.
* test: cover the ochre endpoint Run wrappers' connect and exit paths
* fix: treat an empty endpoint bucket as no endpoints, not an error
ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound,
so 'ochre endpoint list' failed on a cluster that had never launched one.
* feat: add spx admin ochre weights stage/list/remove
Give operators a supported path to stage a self-hosted model's
weights: point at a predastore S3 prefix holding ordinary Hugging
Face files, and stage materialises them into a viperblock volume,
snapshots it, and records the source URI and snapshot ID in the
bedrock-weights KV bucket.
stage refuses before downloading anything if the model is not a
self-host catalog entry, or if the prefix is missing any required
file. Re-staging the same source is a no-op; a different source
replaces the entry and reports the previous snapshot for separate
reclamation.
WeightsStore gains GetWeights and a source-URI field on its record
to support this. LookupServingSpec (catalog.go) is reused unchanged
to validate --model-id.
* fix: unify ServingSpec after rebase onto dev's daemon-launcher spec
Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec
and LookupServingSpec, textually duplicating the ochre weights stage
CLI's own versions in catalog.go. Collapse both into one type carrying
ModelID and VLLMArgs, and one three-value LookupServingSpec so callers
distinguish an unknown model (found=false) from a provider-tier one
(selfHost=false). Update the daemon launcher's call site accordingly.
* test: cover ochre weights stage/list/remove command behaviour
Splits runOchreWeightsStage/List/Remove into testable decision functions
(runStageWeights, listWeightsOutput, removeWeights) separate from cobra and
NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps
swappable so the full command flow can be driven against an embedded
JetStream server instead of live infrastructure. Also relaxes the tokenizer
file requirement to accept either tokenizer.json or tokenizer.model, since
canonical Llama 3.x repos ship only the former at the prefix root.
* fix: resolve mkfs.ext4 from sbin when it is absent from PATH
e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so
weights staging failed on an unprivileged operator's otherwise fine host.
* fix: size the weights volume to a whole GiB so the tail block fits
viperblock rejects a write whose block overruns VolumeSize, so importing an
image of unaligned length failed on its last block. Rounding up also makes
the volume's byte size and its manifest SizeGiB describe the same volume.
Name the volume in the import error so a part-written import is traceable.
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two
providers there, so apt refuses to resolve it; the token was redundant anyway
since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs
the qemu-system-* binaries directly.
sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it
implements no pam_session Defaults setting, and forbids a wildcard inside a
command argument. pam_session is now emitted only for classic sudo, and the
sysctl grant names a fixed-verb helper instead of a wildcard.
The helper is narrower than the rule it replaces: it accepts one interface, a
key restricted to rp_filter or accept_local, and a value restricted to 0 or 1,
where the old grant permitted any net.ipv4.conf key at any value. Granting
sysctl with an unconstrained argument was rejected as root-equivalent.
Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime
package list is resolved against each supported suite's real indices in CI.
Spinifex rejected HttpTokens=optional and served 401 to every untokened
GET, which is stricter than AWS, whose own default is optional. That made
Windows guests impossible: cloudbase-init is the only realistic Windows
bootstrap agent and its EC2Service has no IMDSv2 token support in any
released version, so a Windows instance never received its hostname,
injected password or user-data.
HttpTokens is now settable at launch and via ModifyInstanceMetadataOptions,
accepting optional or required. The launch default stays required, so an
instance that does not ask for IMDSv1 never gets it. Every other
metadata-option field keeps its existing rejection.
The untokened path is reachable without credentials, so the per-instance
lookup is cached with a short TTL; without that a guest could drive
unbounded DescribeInstances fan-outs. A valid token short-circuits before
the lookup, so compliant IMDSv2 clients are unaffected.
* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes
Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on
CreateVolume, persists it to VolumeMetadata, and surfaces it via
DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped
in mulga-siv-380. Pins viperblock to the commit that adds the
VolumeMetadata.Throughput field.
Zero Throughput on a read volume means it predates this field; it is
omitted from the response rather than surfaced as a misleading 0
MiB/s, matching the existing IOPS convention.
* chore: pin viperblock to the merged gp3 Throughput commit
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
* fix(dhcp): release stranded leases and label them with the owning node
Two paths dropped a lease from KV without returning it to the upstream
server, so a source="dhcp" pool leaked addresses permanently.
Manager.Start deleted an expired entry outright. Dropping the entry alone
abandons the binding rather than returning it, and a server that ages
leases by liveness holds the address for as long as anything answers ARP
for it. Adoption now sends a bounded, best-effort RELEASE first, skipping
leases with no stored offer/ack because nclient4 cannot rebuild those.
Release-by-IP answered SUCCESS when the IP was absent from KV, telling
the caller the address was freed when nothing went on the wire. It now
returns an error. Every ReleaseIP call site logs and continues, so this
surfaces the strand without wedging teardown.
Leases were also unattributable upstream: chaddr is a derived HashMAC and
option 61 carries that same MAC, so no field identified the host that took
the lease. Option 12 is now "<node>-<client-id>" and option 60 is
"mulga-spinifex/<node>", giving the lease table a per-lease identity and a
per-host column to group on. HashMAC is untouched — encoding the node
there would cut per-node entropy and reverse a documented decision.
Observed live: 20 office-LAN addresses held by one node's orphaned OVN
gateway LRPs, with no lease records and no renewals.
* fix: rebind the datapath when a DHCP lease moves to a new address
A lease re-issued on a different IP was persisted to KV with no notification
of any kind. doAcquire replaced the lease after an expiry and doRenew accepted
whatever a rebind answered with, while the gateway LRP's address is written
once at attach and never revisited. Each such event burned one address
permanently: the old one squatted by OVN's ARP responder with no renewer and
no DHCPRELEASE, the new one renewed forever attached to nothing.
Route both paths through applyLease, which releases the superseded address and
invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that
moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg
so the VPC SNAT and host ingress route follow, and waits on the flows barrier.
EIP, ENI-public and NATGW addresses are recorded outside vpcd in
DescribeAddresses and DescribeInstances, so the hook reports those rather than
repointing a datapath the control plane no longer agrees with. The release half
still applies to every purpose, so the leak itself is closed either way.
A failed rebind does not fail the lease loop: the new address is held
regardless, and dropping it would strand both.
* fix: reconcile API-visible records when their DHCP lease moves
An EIP or auto-assigned public IP whose lease was re-issued elsewhere left
DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had
already released. The datapath kept the old dnat_and_snat too, so the address
was dead on both sides while the upstream server was free to hand it to
someone else.
vpcd cannot fix this alone: it holds the lease, but the records naming those
addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and
the daemon serves, dispatching to the service owning the moved address.
RebindPublicIP and RebindENIPublicIP commit the new external IP through
vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP,
so one commit is enough — and only then move the record, so a NAT failure never
leaves a record advertising an address that does not route.
Both refuse to move a record naming neither address, so a stale request cannot
overwrite an allocation the lease does not own, and both are idempotent under
duplicate delivery.
A failed or unanswered reconcile propagates back to the lease loop rather than
reading as success. vpc.dhcp.lease-expired, listed in the service manifest but
never implemented, is replaced by the subject that now exists.
* fix: dedupe retried AllocateAddress so one call takes one address
EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can
outlast the AWS client read timeout, so every SDK-level retry minted a
fresh allocation ID and drew another upstream DHCP lease. The caller kept
one EIP and the rest stayed leased with no record naming them, so nothing
ever released them.
Retries are now identified by the SDK's amz-sdk-invocation-id, which is
stable across attempts of one call. The gateway lifts it off the request,
NATSRequest forwards it as a header, and the daemon puts it back on the
context, so the EIP service sees the same token every attempt. singleflight
joins a retry to the call still in flight — the common case — and a TTL'd
KV record answers retries that arrive after the first call completed.
* fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones
Three leaks that all end the same way: an address stays leased upstream with
nothing renewing it on purpose and nothing that will ever release it.
Manager.Start deleted any lease that expired while vpcd was down, after
releasing it. The resource it was taken for still existed, so the datapath kept
forwarding on an address the upstream server now considers free — a
duplicate-IP conflict waiting for the server to reissue it — and nothing ever
re-acquired. Expired leases now get a renewal loop like any other, which
re-DORAs on its first pass; the existing IP-change hook rebinds the datapath
when the new address differs. The reaffirm RENEW is skipped for these, since a
lease the server has already aged out will only fail it.
Nothing reconciled leases against the resources that justify them, so a
resource deleted out from under its lease left the address renewed forever.
ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check
whether the EIP, ENI or VPC still exists, and releases the lease when it does
not. A gateway lease also has a router port to remove, so DetachIGW runs
before the address goes back — releasing it while OVN still answers ARP for it
invites the same conflict. Anything the daemon cannot resolve is reported
unknown and kept: an unreachable daemon must never read as a fleet-wide
deletion.
The acquire ladder ran to ~62s, just past the 60s read timeout botocore
defaults to, which is what turned one AllocateAddress into two leases. It now
totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the
outcome instead of giving up and retrying. This is a mitigation, not a fix, for
callers that send no retry token: botocore emits no amz-sdk-invocation-id, so
the dedupe added earlier cannot see their retries at all.
Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj.
* feat: add spx get dhcp-leases so held external addresses are visible
The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV
bucket with no read path, so working out which addresses were held, by what,
and whether the owner still existed meant writing a NATS client or reading
journals. That was the main obstacle to diagnosing every leak in this series.
The command lists each lease with its purpose, VPC, address, MAC, holding node
and remaining time, and resolves the owner over vpc.dhcp.owner-check so an
orphan is visible as such. --orphans narrows the output to leaks;
--skip-owner-check drops the lookup when only the addresses matter.
Fixes mulga-655iu.
* fix(vpcd): reconcile gateway ports onto their lease address at startup
A gateway LRP configured from a static pool keeps its address when the pool is
switched to source="dhcp", so the VPC forwards on an address nothing leases
while the lease it holds goes unused. No lease event ever fires to correct it.
vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the
address the lease actually holds. RebindGatewayIP is idempotent, so a port
already in agreement costs one OVSDB read.
* test: cover dhcp lease owner-check handlers and the lease CLI
The owner-check path decides whether the reaper releases an address, so every
inconclusive branch is asserted to report unknown rather than gone.
* fix(dhcp): make the outcome of a stale lease release observable
A stale RELEASE was logged only when the transport failed. A lease with no raw
offer/ack was skipped in silence, and a RELEASE the server ignored looked
identical to one it honoured, so an address left bound upstream could not be
traced to either cause.
The skip now warns, and a successful release logs the chaddr it used, which is
what a server matches a RELEASE against.
* fix(dhcp): take the RELEASE chaddr from the lease being released
A server matches a RELEASE to a binding on chaddr, and the release path
re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed
since, that sends an address the binding never had: the server drops the RELEASE
and says nothing, so the address stays held with no renewer.
The chaddr now comes from the lease record, which carries what took the lease
out, falling back to live resolution only when the record has none.
Josh Siviour